2023-07-29T15:41:02.123 INFO:root:teuthology version: 0.0.1.dev172+g6114016 2023-07-29T15:41:02.124 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830 --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/orig.config.yaml 2023-07-29T15:41:02.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-29T15:41:02.285 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830 branch: reef-release description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7356830' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: reef-release ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef-release sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7230 sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: reef-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi039.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 61140164d2579c30ffff3970615e0f82e40c6470 timestamp: 2023-07-29_14:02:18 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-29T15:41:02.286 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa; will attempt to use it 2023-07-29T15:41:02.287 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks 2023-07-29T15:41:02.288 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-29T15:41:02.289 INFO:teuthology.task.internal:Checking packages... 2023-07-29T15:41:02.329 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '0484097b0572e96f42cf6402cbe0ac8dcb046577' 2023-07-29T15:41:02.330 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-29T15:41:02.331 INFO:teuthology.packaging:ref: None 2023-07-29T15:41:02.331 INFO:teuthology.packaging:tag: None 2023-07-29T15:41:02.332 INFO:teuthology.packaging:branch: reef-release 2023-07-29T15:41:02.332 INFO:teuthology.packaging:sha1: 0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T15:41:02.332 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef-release 2023-07-29T15:41:02.680 INFO:teuthology.task.internal:Found packages for ceph version 18.1.3-11.g0484097b 2023-07-29T15:41:02.680 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-29T15:41:02.686 INFO:teuthology.task.internal:no buildpackages task found 2023-07-29T15:41:02.686 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-29T15:41:02.702 INFO:teuthology.task.internal:Saving configuration 2023-07-29T15:41:02.722 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-29T15:41:02.732 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T15:41:02.761 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 15:34:56.813796', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-29T15:41:02.788 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 15:34:56.814850', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-29T15:41:02.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 15:34:56.812591', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-29T15:41:02.813 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-29T15:41:02.819 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-29T15:41:02.820 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-29T15:41:02.820 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-29T15:41:02.820 INFO:teuthology.run_tasks:Running task console_log... 2023-07-29T15:41:02.942 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0320b4d040>, signals=[15]) 2023-07-29T15:41:02.943 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-29T15:41:02.951 INFO:teuthology.task.internal:Opening connections... 2023-07-29T15:41:02.952 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T15:41:02.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:41:03.026 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2023-07-29T15:41:03.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:41:03.095 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2023-07-29T15:41:03.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:41:03.163 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-29T15:41:03.171 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-07-29T15:41:03.193 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-07-29T15:41:03.195 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-07-29T15:41:03.253 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux" 2023-07-29T15:41:03.253 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)" 2023-07-29T15:41:03.253 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-29T15:41:03.254 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/" 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-29T15:41:03.255 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-29T15:41:03.256 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-07-29T15:41:03.284 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2023-07-29T15:41:03.305 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2023-07-29T15:41:03.305 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2023-07-29T15:41:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Red Hat Enterprise Linux" 2023-07-29T15:41:03.365 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8.6 (Ootpa)" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:ID="rhel" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="fedora" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8.6" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2023-07-29T15:41:03.366 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.redhat.com/" 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-29T15:41:03.367 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-29T15:41:03.368 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-29T15:41:03.368 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2023-07-29T15:41:03.393 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2023-07-29T15:41:03.413 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2023-07-29T15:41:03.414 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2023-07-29T15:41:03.473 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="Red Hat Enterprise Linux" 2023-07-29T15:41:03.473 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="8.6 (Ootpa)" 2023-07-29T15:41:03.474 INFO:teuthology.orchestra.run.smithi156.stdout:ID="rhel" 2023-07-29T15:41:03.474 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="fedora" 2023-07-29T15:41:03.474 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="8.6" 2023-07-29T15:41:03.474 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el8" 2023-07-29T15:41:03.475 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-29T15:41:03.475 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2023-07-29T15:41:03.475 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-29T15:41:03.475 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://www.redhat.com/" 2023-07-29T15:41:03.476 INFO:teuthology.orchestra.run.smithi156.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-29T15:41:03.476 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-29T15:41:03.476 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:41:03.476 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-29T15:41:03.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-29T15:41:03.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-29T15:41:03.477 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-29T15:41:03.478 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2023-07-29T15:41:03.501 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-29T15:41:03.514 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-29T15:41:03.557 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-29T15:41:03.557 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-07-29T15:41:03.560 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2023-07-29T15:41:03.563 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2023-07-29T15:41:03.581 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-29T15:41:03.588 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-29T15:41:03.588 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-07-29T15:41:03.619 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2023-07-29T15:41:03.623 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2023-07-29T15:41:03.657 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-29T15:41:03.746 INFO:teuthology.run_tasks:Running task kernel... 2023-07-29T15:41:03.763 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-07-29T15:41:03.764 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-07-29T15:41:03.764 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-07-29T15:41:03.765 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-29T15:41:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T15:41:03.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2023-07-29T15:41:04.221 DEBUG:teuthology.task.kernel:sha1 for client.0 is 8e3722cc016410c8de834520a714aab9f006b92a 2023-07-29T15:41:04.221 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "8e3722cc016410c8de834520a714aab9f006b92a"... 2023-07-29T15:41:04.221 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-07-29T15:41:04.241 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-29T15:41:04.241 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-372.9.1.el8.x86_64 vs 8e3722cc016410c8de834520a714aab9f006b92a 2023-07-29T15:41:04.242 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-07-29T15:41:04.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum upgrade -y linux-firmware 2023-07-29T15:41:05.472 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-07-29T15:41:05.472 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-07-29T15:41:06.897 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-07-29T15:41:07.764 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2023-07-29T15:41:11.373 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 438 kB/s | 24 kB 00:00 2023-07-29T15:41:12.174 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-07-29T15:41:12.175 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-07-29T15:41:12.175 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-07-29T15:41:12.299 INFO:teuthology.task.kernel:Downloading kernel 8e3722cc016410c8de834520a714aab9f006b92a on client.0... 2023-07-29T15:41:12.300 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=8e3722cc016410c8de834520a714aab9f006b92a 2023-07-29T15:41:12.539 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/8e3722cc016410c8de834520a714aab9f006b92a/centos/8/flavors/default/x86_64/ 2023-07-29T15:41:12.539 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc1_g8e3722cc0164-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/8e3722cc016410c8de834520a714aab9f006b92a/centos/8/flavors/default/x86_64/ --input-file=- 2023-07-29T15:41:12.541 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi031.front.sepia.ceph.com to complete... 2023-07-29T15:41:28.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29 15:41:28 URL:https://4.chacra.ceph.com/r/kernel/testing/8e3722cc016410c8de834520a714aab9f006b92a/centos/8/flavors/default/x86_64/kernel-6.5.0_rc1_g8e3722cc0164-1.x86_64.rpm [380683860/380683860] -> "/tmp/kernel.x86_64.rpm" [1] 2023-07-29T15:41:28.809 INFO:teuthology.orchestra.run.smithi031.stderr:FINISHED --2023-07-29 15:41:28-- 2023-07-29T15:41:28.809 INFO:teuthology.orchestra.run.smithi031.stderr:Total wall clock time: 16s 2023-07-29T15:41:28.810 INFO:teuthology.orchestra.run.smithi031.stderr:Downloaded: 1 files, 363M in 16s (22.8 MB/s) 2023-07-29T15:41:28.811 INFO:teuthology.task.kernel:Installing kernel 8e3722cc016410c8de834520a714aab9f006b92a on client.0... 2023-07-29T15:41:28.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-07-29T15:41:30.467 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying... ######################################## 2023-07-29T15:41:30.517 INFO:teuthology.orchestra.run.smithi031.stdout:Preparing... ######################################## 2023-07-29T15:41:30.607 INFO:teuthology.orchestra.run.smithi031.stdout:Updating / installing... 2023-07-29T15:42:11.220 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-6.5.0_rc1_g8e3722cc0164-1 ######################################## 2023-07-29T15:42:38.802 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-07-29T15:42:38.803 DEBUG:teuthology.orchestra.run.smithi031:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-07-29T15:42:38.856 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/System.map-6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:38.856 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/config-6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:38.856 INFO:teuthology.orchestra.run.smithi031.stdout:/boot/vmlinuz-6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-07-29T15:42:38.857 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-07-29T15:42:38.858 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-07-29T15:42:38.858 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-07-29T15:42:38.858 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-07-29T15:42:38.858 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-07-29T15:42:38.858 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-07-29T15:42:38.858 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/crc32c-intel.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/serpent-avx2.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-07-29T15:42:38.859 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kernel 2023-07-29T15:42:38.860 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kernel/cpu 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kernel/cpu/mce 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kvm 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kvm/kvm-amd.ko 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kvm/kvm-intel.ko 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/arch/x86/kvm/kvm.ko 2023-07-29T15:42:38.861 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/ansi_cprng.ko 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/anubis.ko 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/arc4.ko 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx/async_memcpy.ko 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx/async_pq.ko 2023-07-29T15:42:38.862 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx/async_raid6_recov.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx/async_tx.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx/async_xor.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/async_tx/raid6test.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/authenc.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/authencesn.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/blake2b_generic.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/blowfish_common.ko 2023-07-29T15:42:38.863 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/blowfish_generic.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/camellia_generic.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/cast5_generic.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/cast6_generic.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/cast_common.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/ccm.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/cmac.ko 2023-07-29T15:42:38.864 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/crc32_generic.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/crypto_engine.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/crypto_user.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/cts.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/des_generic.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/dh_generic.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/echainiv.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/essiv.ko 2023-07-29T15:42:38.865 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/fcrypt.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/khazad.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/lz4.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/lz4hc.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/md4.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/michael_mic.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/pcbc.ko 2023-07-29T15:42:38.866 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/pcrypt.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/rmd160.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/seed.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/serpent_generic.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/tcrypt.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/tea.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/twofish_common.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/twofish_generic.ko 2023-07-29T15:42:38.867 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/vmac.ko 2023-07-29T15:42:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/wp512.ko 2023-07-29T15:42:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/xcbc.ko 2023-07-29T15:42:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/xor.ko 2023-07-29T15:42:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/crypto/xxhash_generic.ko 2023-07-29T15:42:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers 2023-07-29T15:42:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi 2023-07-29T15:42:38.869 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/acpi_ipmi.ko 2023-07-29T15:42:38.869 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/acpi_pad.ko 2023-07-29T15:42:38.869 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/custom_method.ko 2023-07-29T15:42:38.869 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/ec_sys.ko 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/platform_profile.ko 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/sbs.ko 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/sbshc.ko 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/acpi/video.ko 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/acard-ahci.ko 2023-07-29T15:42:38.870 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/ahci_platform.ko 2023-07-29T15:42:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/ata_generic.ko 2023-07-29T15:42:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/libahci_platform.ko 2023-07-29T15:42:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_acpi.ko 2023-07-29T15:42:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_ali.ko 2023-07-29T15:42:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_amd.ko 2023-07-29T15:42:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_artop.ko 2023-07-29T15:42:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_atiixp.ko 2023-07-29T15:42:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_atp867x.ko 2023-07-29T15:42:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_cmd640.ko 2023-07-29T15:42:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_cmd64x.ko 2023-07-29T15:42:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_cypress.ko 2023-07-29T15:42:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_efar.ko 2023-07-29T15:42:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_hpt366.ko 2023-07-29T15:42:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_hpt37x.ko 2023-07-29T15:42:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_hpt3x2n.ko 2023-07-29T15:42:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_hpt3x3.ko 2023-07-29T15:42:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_it8213.ko 2023-07-29T15:42:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_it821x.ko 2023-07-29T15:42:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_jmicron.ko 2023-07-29T15:42:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_marvell.ko 2023-07-29T15:42:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_mpiix.ko 2023-07-29T15:42:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_netcell.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_ninja32.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_ns87410.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_ns87415.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_oldpiix.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_opti.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_optidma.ko 2023-07-29T15:42:38.876 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_pcmcia.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_pdc2027x.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_piccolo.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_rdc.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_sch.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_serverworks.ko 2023-07-29T15:42:38.877 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_sil680.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_sis.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_sl82c105.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_triflex.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pata_via.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/pdc_adma.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_inic162x.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_mv.ko 2023-07-29T15:42:38.878 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_nv.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_promise.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_qstor.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_sil.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_sil24.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_sis.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_svw.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_sx4.ko 2023-07-29T15:42:38.879 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_uli.ko 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_via.ko 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ata/sata_vsc.ko 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/auxdisplay 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/auxdisplay/ks0108.ko 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/base 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/base/regmap 2023-07-29T15:42:38.880 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/base/regmap/regmap-i2c.ko 2023-07-29T15:42:38.881 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/base/regmap/regmap-mmio.ko 2023-07-29T15:42:38.881 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/bcma 2023-07-29T15:42:38.881 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/bcma/bcma.ko 2023-07-29T15:42:38.881 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block 2023-07-29T15:42:38.881 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/aoe 2023-07-29T15:42:38.882 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/aoe/aoe.ko 2023-07-29T15:42:38.882 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/brd.ko 2023-07-29T15:42:38.882 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/drbd 2023-07-29T15:42:38.882 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/drbd/drbd.ko 2023-07-29T15:42:38.882 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/floppy.ko 2023-07-29T15:42:38.882 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/loop.ko 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/mtip32xx 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/nbd.ko 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/null_blk 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/null_blk/null_blk.ko 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/pktcdvd.ko 2023-07-29T15:42:38.883 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/rbd.ko 2023-07-29T15:42:38.884 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/virtio_blk.ko 2023-07-29T15:42:38.884 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/xen-blkback 2023-07-29T15:42:38.884 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-07-29T15:42:38.884 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/xen-blkfront.ko 2023-07-29T15:42:38.884 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/zram 2023-07-29T15:42:38.884 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/block/zram/zram.ko 2023-07-29T15:42:38.885 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char 2023-07-29T15:42:38.885 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hangcheck-timer.ko 2023-07-29T15:42:38.885 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hw_random 2023-07-29T15:42:38.886 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hw_random/amd-rng.ko 2023-07-29T15:42:38.886 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hw_random/intel-rng.ko 2023-07-29T15:42:38.886 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-07-29T15:42:38.886 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hw_random/via-rng.ko 2023-07-29T15:42:38.887 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/hw_random/virtio-rng.ko 2023-07-29T15:42:38.887 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi 2023-07-29T15:42:38.887 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-07-29T15:42:38.887 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-07-29T15:42:38.888 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-07-29T15:42:38.888 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi/ipmi_si.ko 2023-07-29T15:42:38.888 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-07-29T15:42:38.889 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-07-29T15:42:38.889 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/lp.ko 2023-07-29T15:42:38.889 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/mwave 2023-07-29T15:42:38.889 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/mwave/mwave.ko 2023-07-29T15:42:38.890 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/ppdev.ko 2023-07-29T15:42:38.890 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tlclk.ko 2023-07-29T15:42:38.891 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm 2023-07-29T15:42:38.891 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm/tpm.ko 2023-07-29T15:42:38.891 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm/tpm_atmel.ko 2023-07-29T15:42:38.891 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm/tpm_infineon.ko 2023-07-29T15:42:38.891 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm/tpm_nsc.ko 2023-07-29T15:42:38.891 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm/tpm_tis.ko 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/uv_mmtimer.ko 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/char/virtio_console.ko 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-07-29T15:42:38.892 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq/p4-clockmod.ko 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq/powernow-k8.ko 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/cpufreq/speedstep-lib.ko 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/ccp 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/ccp/ccp.ko 2023-07-29T15:42:38.893 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/intel 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/intel/qat 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/intel/qat/qat_common 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/padlock-aes.ko 2023-07-29T15:42:38.894 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/padlock-sha.ko 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/virtio 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dca 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dca/dca.ko 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/devfreq 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dma 2023-07-29T15:42:38.895 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dma/dw 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dma/dw/dw_dmac.ko 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dma/ioat 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/dma/ioat/ioatdma.ko 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/amd64_edac.ko 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/e752x_edac.ko 2023-07-29T15:42:38.896 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/edac_mce_amd.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i3000_edac.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i3200_edac.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i5100_edac.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i5400_edac.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i7300_edac.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i7core_edac.ko 2023-07-29T15:42:38.897 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/i82975x_edac.ko 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/ie31200_edac.ko 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/sb_edac.ko 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/edac/x38_edac.ko 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/firmware 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/firmware/edd.ko 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/firmware/iscsi_ibft.ko 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/gpio 2023-07-29T15:42:38.898 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/gpio/gpio-ich.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/gpio/gpio-viperboard.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-appleir.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-aureal.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-axff.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-dr.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-elecom.ko 2023-07-29T15:42:38.899 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-elo.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-emsff.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-gaff.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-gt683r.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-gyration.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-holtek-kbd.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-holtek-mouse.ko 2023-07-29T15:42:38.900 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-holtekff.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-hyperv.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-icade.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-keytouch.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-kye.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-lcpower.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-led.ko 2023-07-29T15:42:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-lenovo.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-logitech-dj.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-multitouch.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-ortek.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-penmount.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-petalynx.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-picolcd.ko 2023-07-29T15:42:38.902 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-pl.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-plantronics.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-primax.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-rmi.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-arvo.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-common.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-isku.ko 2023-07-29T15:42:38.903 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-kone.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-konepure.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-lua.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-pyra.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-ryos.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat-savu.ko 2023-07-29T15:42:38.904 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-roccat.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-saitek.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-samsung.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-sensor-hub.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-sjoy.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-sony.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-speedlink.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-steelseries.ko 2023-07-29T15:42:38.905 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-sunplus.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-thrustmaster.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-tivo.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-tmff.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-topseed.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-twinhan.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-uclogic.ko 2023-07-29T15:42:38.906 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-waltop.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-wiimote.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-xinmo.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-zpff.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/hid-zydacron.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/uhid.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hid/wacom.ko 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hv 2023-07-29T15:42:38.907 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hv/hv_balloon.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hv/hv_utils.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hv/hv_vmbus.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/abituguru.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/abituguru3.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/acpi_power_meter.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ad7414.ko 2023-07-29T15:42:38.908 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ad7418.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adc128d818.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adm1025.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adm1026.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adm1029.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adm1031.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adm9240.ko 2023-07-29T15:42:38.909 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ads7828.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adt7410.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adt7411.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adt7462.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adt7470.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adt7475.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/adt7x10.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/amc6821.ko 2023-07-29T15:42:38.910 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/applesmc.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/asb100.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/asc7621.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/asus_atk0110.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/atxp1.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/coretemp.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-07-29T15:42:38.911 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/dme1737.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ds1621.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ds620.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/emc1403.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/emc6w201.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/f71805f.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/f71882fg.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/f75375s.ko 2023-07-29T15:42:38.912 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/fam15h_power.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/fschmd.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/g760a.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/g762.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/gl518sm.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/gl520sm.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/hwmon-vid.ko 2023-07-29T15:42:38.913 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/i5500_temp.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/i5k_amb.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ibmaem.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ibmpex.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ina209.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ina2xx.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/it87.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/k10temp.ko 2023-07-29T15:42:38.914 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/k8temp.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lineage-pem.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm63.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm73.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm75.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm77.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm78.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm80.ko 2023-07-29T15:42:38.915 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm83.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm85.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm87.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm90.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm92.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm93.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm95234.ko 2023-07-29T15:42:38.916 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm95241.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/lm95245.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc2945.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc4151.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc4215.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc4222.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc4245.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc4260.ko 2023-07-29T15:42:38.917 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/ltc4261.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max16065.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max1619.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max1668.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max197.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max6639.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max6650.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/max6697.ko 2023-07-29T15:42:38.918 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/mcp3021.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/nct6683.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/nct6775-core.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/nct6775.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/nct7802.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pc87360.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pc87427.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pcf8591.ko 2023-07-29T15:42:38.919 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/max16064.ko 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/max34440.ko 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/max8688.ko 2023-07-29T15:42:38.920 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/powr1220.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/sch5627.ko 2023-07-29T15:42:38.921 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/sch5636.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/sch56xx-common.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/sht15.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/sht21.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/shtc1.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/sis5595.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/smsc47b397.ko 2023-07-29T15:42:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/smsc47m1.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/smsc47m192.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/thmc50.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/tmp102.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/tmp103.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/tmp401.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/tmp421.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/via-cputemp.ko 2023-07-29T15:42:38.923 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/via686a.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/vt1211.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/vt8231.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83627ehf.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83627hf.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83781d.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83791d.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83792d.ko 2023-07-29T15:42:38.924 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83793.ko 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83795.ko 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83l785ts.ko 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/hwmon/w83l786ng.ko 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/algos 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-07-29T15:42:38.925 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-07-29T15:42:38.926 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-i801.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-isch.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-07-29T15:42:38.927 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-parport.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-07-29T15:42:38.928 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-via.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/i2c-dev.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/i2c-mux.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/i2c-smbus.ko 2023-07-29T15:42:38.929 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/i2c/i2c-stub.ko 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/ib_cm.ko 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/ib_core.ko 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/ib_umad.ko 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-07-29T15:42:38.930 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/iw_cm.ko 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/rdma_cm.ko 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/cxgb4 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/mlx4 2023-07-29T15:42:38.931 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/mlx5 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/mthca 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/ipoib 2023-07-29T15:42:38.932 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-07-29T15:42:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/iser 2023-07-29T15:42:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-07-29T15:42:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/isert 2023-07-29T15:42:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-07-29T15:42:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/srp 2023-07-29T15:42:38.933 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/srpt 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/gameport 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/gameport/fm801-gp.ko 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/gameport/gameport.ko 2023-07-29T15:42:38.934 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/gameport/lightning.ko 2023-07-29T15:42:38.935 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/gameport/ns558.ko 2023-07-29T15:42:38.935 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/joydev.ko 2023-07-29T15:42:38.935 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/keyboard 2023-07-29T15:42:38.935 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/keyboard/gpio_keys.ko 2023-07-29T15:42:38.935 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse 2023-07-29T15:42:38.935 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/appletouch.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/bcm5974.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/cyapatp.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/elan_i2c.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/sermouse.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/synaptics_usb.ko 2023-07-29T15:42:38.936 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/mouse/vsxxxaa.ko 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/rmi4 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/rmi4/rmi_core.ko 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/serio 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/serio/altera_ps2.ko 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/serio/arc_ps2.ko 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-07-29T15:42:38.937 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/serio/serio_raw.ko 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/input/sparse-keymap.ko 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/iommu 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/iommu/amd 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/iommu/amd/iommu_v2.ko 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/leds-blinkm.ko 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/leds-lm3530.ko 2023-07-29T15:42:38.938 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/leds-lp3944.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/leds-ss4200.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-07-29T15:42:38.939 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/bcache 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/bcache/bcache.ko 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-bio-prison.ko 2023-07-29T15:42:38.940 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-cache-smq.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-cache.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-crypt.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-delay.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-flakey.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-log-userspace.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-multipath.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-queue-length.ko 2023-07-29T15:42:38.941 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-raid.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-round-robin.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-service-time.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-switch.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-thin-pool.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/dm-verity.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/faulty.ko 2023-07-29T15:42:38.942 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/linear.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/multipath.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/persistent-data 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/raid0.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/raid1.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/raid10.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/md/raid456.ko 2023-07-29T15:42:38.943 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptbase.ko 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptctl.ko 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptfc.ko 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptlan.ko 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptsas.ko 2023-07-29T15:42:38.944 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptscsih.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/message/fusion/mptspi.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/lpc_ich.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/lpc_sch.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/mfd-core.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/sm501.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/viperboard.ko 2023-07-29T15:42:38.945 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/vx855.ko 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/mfd/wl1273-core.ko 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/altera-stapl 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/apds9802als.ko 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/apds990x.ko 2023-07-29T15:42:38.946 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/bh1770glc.ko 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/cb710 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/cb710/cb710.ko 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/eeprom 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/eeprom/at24.ko 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/eeprom/eeprom.ko 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/eeprom/max6875.ko 2023-07-29T15:42:38.947 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/enclosure.ko 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/hpilo.ko 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/isl29003.ko 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/isl29020.ko 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/lis3lv02d 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-07-29T15:42:38.948 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/mei 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/mei/mei-me.ko 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/mei/mei-txe.ko 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/mei/mei.ko 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/sgi-gru 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/sgi-gru/gru.ko 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/sgi-xp 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/sgi-xp/xp.ko 2023-07-29T15:42:38.949 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/sgi-xp/xpc.ko 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/tifm_7xx1.ko 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/tifm_core.ko 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/tsl2550.ko 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/vmw_balloon.ko 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/vmw_vmci 2023-07-29T15:42:38.950 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/bonding 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/bonding/bonding.ko 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa/b53 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa/b53/b53_common.ko 2023-07-29T15:42:38.951 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa/bcm-sf2.ko 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa/mv88e6060.ko 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa/mv88e6xxx 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/dummy.ko 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/eql.ko 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet 2023-07-29T15:42:38.952 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/3com 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/8390 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/8390/8390.ko 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-07-29T15:42:38.953 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/adaptec 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/agere 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/agere/et131x.ko 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/alteon 2023-07-29T15:42:38.954 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/altera 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/amd 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros 2023-07-29T15:42:38.955 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/alx 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atl1c 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atl1e 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atlx 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-07-29T15:42:38.956 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-07-29T15:42:38.957 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/genet 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/brocade 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/brocade/bna 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb 2023-07-29T15:42:38.958 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-07-29T15:42:38.959 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/cisco 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/cisco/enic 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-07-29T15:42:38.960 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dlink 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-07-29T15:42:38.961 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/dnet.ko 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/emulex 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/emulex/benet 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/ethoc.ko 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/fealnx.ko 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel 2023-07-29T15:42:38.962 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/e100.ko 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/e1000 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/e1000e 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/fm10k 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/i40e 2023-07-29T15:42:38.963 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/iavf 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/igb 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/igbvf 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/ixgbe 2023-07-29T15:42:38.964 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/ixgbevf 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/jme.ko 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/marvell 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/marvell/skge.ko 2023-07-29T15:42:38.965 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox/mlx4 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox/mlx5 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-07-29T15:42:38.966 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/micrel 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/myricom 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/myricom/myri10ge 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/natsemi 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-07-29T15:42:38.967 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/neterion 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/nvidia 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/packetengines 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-07-29T15:42:38.968 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/qlogic 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/qlogic/netxen 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/rdc 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-07-29T15:42:38.969 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/realtek 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/realtek/atp.ko 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/rocker 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sfc 2023-07-29T15:42:38.970 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/silan 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sis 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sis/sis190.ko 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sis/sis900.ko 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/smsc 2023-07-29T15:42:38.971 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/stmicro 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/stmicro/stmmac 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-07-29T15:42:38.972 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sun 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sun/cassini.ko 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sun/niu.ko 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sun/sungem.ko 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/tehuti 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/ti 2023-07-29T15:42:38.973 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/ti/tlan.ko 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/via 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/wiznet 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-07-29T15:42:38.974 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/xircom 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/geneve.ko 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/hyperv 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ieee802154 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ieee802154/fakelb.ko 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ifb.ko 2023-07-29T15:42:38.975 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ipvlan 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ipvlan/ipvlan.ko 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/macvlan.ko 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/macvtap.ko 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/mdio 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/mdio.ko 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-07-29T15:42:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/mdio/mdio-regmap.ko 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/mii.ko 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/net_failover.ko 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/netconsole.ko 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/nlmon.ko 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/pcs 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/pcs/pcs-lynx.ko 2023-07-29T15:42:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/amd.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/bcm7xxx.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/bcm87xx.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/broadcom.ko 2023-07-29T15:42:38.978 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/cicada.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/davicom.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/dp83640.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/et1011c.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/icplus.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/lxt.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/marvell.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/micrel.ko 2023-07-29T15:42:38.979 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/national.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/phylink.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/qsemi.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/realtek.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/smsc.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/ste10Xp.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/phy/vitesse.ko 2023-07-29T15:42:38.980 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/bsd_comp.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/ppp_async.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/ppp_deflate.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/ppp_generic.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/ppp_mppe.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/ppp_synctty.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/pppoe.ko 2023-07-29T15:42:38.981 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/pppox.ko 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/ppp/pptp.ko 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/slip 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/slip/slhc.ko 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/slip/slip.ko 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/sungem_phy.ko 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/tap.ko 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team 2023-07-29T15:42:38.982 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team/team.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team/team_mode_activebackup.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team/team_mode_broadcast.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team/team_mode_random.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/tun.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/veth.ko 2023-07-29T15:42:38.983 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/virtio_net.ko 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/vmxnet3 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/vxlan 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/vxlan/vxlan.ko 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/xen-netback 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/xen-netback/xen-netback.ko 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/net/xen-netfront.ko 2023-07-29T15:42:38.984 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ntb 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ntb/ntb.ko 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/nvme 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/nvme/host 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/nvme/host/nvme-core.ko 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/nvme/host/nvme.ko 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/parport 2023-07-29T15:42:38.985 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/parport/parport.ko 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/parport/parport_cs.ko 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/parport/parport_pc.ko 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/parport/parport_serial.ko 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pci 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pci/hotplug 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pci/pcie 2023-07-29T15:42:38.986 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pci/pcie/aer_inject.ko 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pci/xen-pcifront.ko 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pcmcia 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pcmcia/i82092.ko 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pcmcia/pd6729.ko 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pcmcia/yenta_socket.ko 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pinctrl 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pinctrl/intel 2023-07-29T15:42:38.987 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/acer-wmi.ko 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/acerhdf.ko 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/apple-gmux.ko 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/asus-laptop.ko 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-07-29T15:42:38.988 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/asus-wmi.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/classmate-laptop.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/hdaps.ko 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/intel 2023-07-29T15:42:38.989 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/intel_ips.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/msi-wmi.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/mxm-wmi.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/samsung-laptop.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/samsung-q10.ko 2023-07-29T15:42:38.990 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/topstar-laptop.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/toshiba_haps.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/wmi-bmof.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/platform/x86/wmi.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/powercap 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/powercap/intel_rapl_common.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/powercap/intel_rapl_msr.ko 2023-07-29T15:42:38.991 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pps 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pps/clients 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pps/clients/pps-gpio.ko 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pps/clients/pps-ldisc.ko 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pps/clients/pps_parport.ko 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/pps/pps_core.ko 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ptp 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ptp/ptp.ko 2023-07-29T15:42:38.992 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ptp/ptp_kvm.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-bq32k.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-bq4802.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1286.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1307.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1374.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1511.ko 2023-07-29T15:42:38.993 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1553.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1672.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds1742.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds2404.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-ds3232.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-em3027.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-fm3130.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-isl12022.ko 2023-07-29T15:42:38.994 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-isl1208.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-m41t80.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-m48t35.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-m48t59.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-max6900.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-msm6242.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-pcf2127.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-pcf85063.ko 2023-07-29T15:42:38.995 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-pcf8523.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-pcf8563.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-pcf8583.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-rp5c01.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-rs5c372.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-rx8025.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-rx8581.ko 2023-07-29T15:42:38.996 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/rtc/rtc-x1205.ko 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/3w-9xxx.ko 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/3w-sas.ko 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/3w-xxxx.ko 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/BusLogic.ko 2023-07-29T15:42:38.997 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/a100u2w.ko 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/aacraid 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/aacraid/aacraid.ko 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/advansys.ko 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/aic7xxx 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/am53c974.ko 2023-07-29T15:42:38.998 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/arcmsr 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/atp870u.ko 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/be2iscsi 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/bfa 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/bfa/bfa.ko 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/bnx2fc 2023-07-29T15:42:38.999 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/bnx2i 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/ch.ko 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/csiostor 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/csiostor/csiostor.ko 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/cxgbi 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/cxgbi/cxgb3i 2023-07-29T15:42:39.000 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/cxgbi/cxgb4i 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/dc395x.ko 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/device_handler 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-07-29T15:42:39.001 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/dmx3191d.ko 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/esas2r 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/esas2r/esas2r.ko 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/esp_scsi.ko 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/fcoe 2023-07-29T15:42:39.002 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/fcoe/fcoe.ko 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/fnic 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/fnic/fnic.ko 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/hpsa.ko 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/hptiop.ko 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/hv_storvsc.ko 2023-07-29T15:42:39.003 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/initio.ko 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/ipr.ko 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/ips.ko 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/isci 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/isci/isci.ko 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/iscsi_tcp.ko 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/libfc 2023-07-29T15:42:39.004 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/libfc/libfc.ko 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/libiscsi.ko 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/libiscsi_tcp.ko 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/libsas 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/libsas/libsas.ko 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/lpfc 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/lpfc/lpfc.ko 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/megaraid 2023-07-29T15:42:39.005 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/megaraid.ko 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/mpt3sas 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/mvsas 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/mvsas/mvsas.ko 2023-07-29T15:42:39.006 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/mvumi.ko 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/pm8001 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/pmcraid.ko 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/qla1280.ko 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/qla2xxx 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-07-29T15:42:39.007 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/qla4xxx 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/raid_class.ko 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/scsi_debug.ko 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/scsi_transport_fc.ko 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/scsi_transport_sas.ko 2023-07-29T15:42:39.008 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/scsi_transport_spi.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/scsi_transport_srp.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/ses.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/st.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/stex.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/sym53c8xx_2 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/virtio_scsi.ko 2023-07-29T15:42:39.009 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/vmw_pvscsi.ko 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/scsi/wd719x.ko 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ssb 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ssb/ssb.ko 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/iscsi 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/loopback 2023-07-29T15:42:39.010 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/loopback/tcm_loop.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/target_core_file.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/target_core_iblock.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/target_core_mod.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/target_core_pscsi.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/target_core_user.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/tcm_fc 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-07-29T15:42:39.011 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-07-29T15:42:39.012 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-07-29T15:42:39.013 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/ipwireless 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/n_gsm.ko 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/n_hdlc.ko 2023-07-29T15:42:39.014 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/nozomi.ko 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/serial 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/serial/8250 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/serial/arc_uart.ko 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/serial/jsm 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/serial/jsm/jsm.ko 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/tty/synclink_gt.ko 2023-07-29T15:42:39.015 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ufs 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ufs/core 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ufs/core/ufshcd-core.ko 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ufs/host 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/uio 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/uio/uio.ko 2023-07-29T15:42:39.016 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/uio/uio_aec.ko 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/uio/uio_cif.ko 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/uio/uio_pci_generic.ko 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/uio/uio_sercos3.ko 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/class 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/class/cdc-acm.ko 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/class/cdc-wdm.ko 2023-07-29T15:42:39.017 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/class/usblp.ko 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/class/usbtmc.ko 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/host 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/host/sl811-hcd.ko 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/image 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/image/mdc800.ko 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/image/microtek.ko 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc 2023-07-29T15:42:39.018 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/adutux.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/appledisplay.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/emi26.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/emi62.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/ezusb.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/idmouse.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/iowarrior.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/isight_firmware.ko 2023-07-29T15:42:39.019 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/ldusb.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/legousbtower.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/sisusbvga 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/trancevibrator.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/usb3503.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/usblcd.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/usbsevseg.ko 2023-07-29T15:42:39.020 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/uss720.ko 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/misc/yurex.ko 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/phy 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/phy/phy-generic.ko 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/aircable.ko 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ark3116.ko 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/belkin_sa.ko 2023-07-29T15:42:39.021 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ch341.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/cp210x.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/cyberjack.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/cypress_m8.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/digi_acceleport.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/empeg.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ftdi_sio.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/garmin_gps.ko 2023-07-29T15:42:39.022 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/io_edgeport.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/io_ti.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ipaq.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ipw.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ir-usb.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/keyspan.ko 2023-07-29T15:42:39.023 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/keyspan_pda.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/kl5kusb105.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/kobil_sct.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/mct_u232.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/mos7720.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/mos7840.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/navman.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/omninet.ko 2023-07-29T15:42:39.024 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/opticon.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/option.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/oti6858.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/pl2303.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/qcaux.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/qcserial.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/quatech2.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/safe_serial.ko 2023-07-29T15:42:39.025 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/sierra.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/spcp8x5.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ssu100.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/symbolserial.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/usb_debug.ko 2023-07-29T15:42:39.026 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/usb_wwan.ko 2023-07-29T15:42:39.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/visor.ko 2023-07-29T15:42:39.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/whiteheat.ko 2023-07-29T15:42:39.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/serial/xsens_mt.ko 2023-07-29T15:42:39.027 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/uas.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-alauda.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-cypress.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-datafab.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-freecom.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-isd200.ko 2023-07-29T15:42:39.028 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-karma.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-onetouch.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-realtek.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-sddr09.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-sddr55.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/ums-usbat.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/storage/usb-storage.ko 2023-07-29T15:42:39.029 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/usbip 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/usbip/usbip-core.ko 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/usbip/usbip-host.ko 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vfio 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vfio/pci 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-07-29T15:42:39.030 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vfio/pci/vfio-pci.ko 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vfio/vfio.ko 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vhost 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vhost/vhost.ko 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vhost/vhost_iotlb.ko 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vhost/vhost_net.ko 2023-07-29T15:42:39.031 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/vhost/vhost_scsi.ko 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/video 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/video/backlight 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/video/backlight/apple_bl.ko 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/video/backlight/lcd.ko 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/video/backlight/platform_lcd.ko 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio.ko 2023-07-29T15:42:39.032 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio_balloon.ko 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio_mmio.ko 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio_pci.ko 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/virtio/virtio_ring.ko 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/masters 2023-07-29T15:42:39.033 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/masters/ds2482.ko 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/masters/ds2490.ko 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-07-29T15:42:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_smem.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/slaves/w1_therm.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/w1/wire.ko 2023-07-29T15:42:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/alim1535_wdt.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/alim7101_wdt.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/f71808e_wdt.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/hpwdt.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/i6300esb.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/iTCO_wdt.ko 2023-07-29T15:42:39.036 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/ib700wdt.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/ibmasr.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/it8712f_wdt.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/it87_wdt.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/machzwd.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/nv_tco.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/pcwd_pci.ko 2023-07-29T15:42:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/pcwd_usb.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/sch311x_wdt.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/softdog.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/sp5100_tco.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/via_wdt.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/w83877f_wdt.ko 2023-07-29T15:42:39.038 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/w83977f_wdt.ko 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/wdt_pci.ko 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/watchdog/xen_wdt.ko 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-acpi-processor.ko 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-evtchn.ko 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-gntalloc.ko 2023-07-29T15:42:39.039 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-gntdev.ko 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-pciback 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xen-privcmd.ko 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xenfs 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/drivers/xen/xenfs/xenfs.ko 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/9p 2023-07-29T15:42:39.040 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/9p/9p.ko 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/affs 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/affs/affs.ko 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/befs 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/befs/befs.ko 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/binfmt_misc.ko 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/btrfs 2023-07-29T15:42:39.041 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/btrfs/btrfs.ko 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/cachefiles 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/cachefiles/cachefiles.ko 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ceph 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ceph/ceph.ko 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/coda 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/coda/coda.ko 2023-07-29T15:42:39.042 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/cramfs 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/cramfs/cramfs.ko 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/dlm 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/dlm/dlm.ko 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ecryptfs 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ecryptfs/ecryptfs.ko 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/f2fs 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/f2fs/f2fs.ko 2023-07-29T15:42:39.043 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fat 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fat/fat.ko 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fat/msdos.ko 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fat/vfat.ko 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fscache 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fscache/fscache.ko 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fuse 2023-07-29T15:42:39.044 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fuse/cuse.ko 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/fuse/fuse.ko 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/gfs2 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/gfs2/gfs2.ko 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/hfs 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/hfs/hfs.ko 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/hfsplus 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/hfsplus/hfsplus.ko 2023-07-29T15:42:39.045 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/isofs 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/isofs/isofs.ko 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/jfs 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/jfs/jfs.ko 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/lockd 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/lockd/lockd.ko 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/minix 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/minix/minix.ko 2023-07-29T15:42:39.046 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/netfs 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/netfs/netfs.ko 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/blocklayout 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/filelayout 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/flexfilelayout 2023-07-29T15:42:39.047 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/nfs.ko 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/nfsv3.ko 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs/nfsv4.ko 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs_common 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs_common/grace.ko 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfs_common/nfs_acl.ko 2023-07-29T15:42:39.048 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfsd 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nfsd/nfsd.ko 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nilfs2 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nilfs2/nilfs2.ko 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-celtic.ko 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-centeuro.ko 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-croatian.ko 2023-07-29T15:42:39.049 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-cyrillic.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-gaelic.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-greek.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-iceland.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-inuit.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-roman.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-romanian.ko 2023-07-29T15:42:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/mac-turkish.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp1250.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp1251.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp1255.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp737.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp775.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp850.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp852.ko 2023-07-29T15:42:39.051 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp855.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp857.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp860.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp861.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp862.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp863.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp864.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp865.ko 2023-07-29T15:42:39.052 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp866.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp869.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp874.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp932.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp936.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp949.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_cp950.ko 2023-07-29T15:42:39.053 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_euc-jp.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-1.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-13.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-14.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-15.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-2.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-3.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-4.ko 2023-07-29T15:42:39.054 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-5.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-6.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-7.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_iso8859-9.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_koi8-r.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_koi8-ru.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_koi8-u.ko 2023-07-29T15:42:39.055 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/nls/nls_utf8.ko 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/cluster 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/dlm 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/dlmfs 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-07-29T15:42:39.056 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/ocfs2.ko 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/overlayfs 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/overlayfs/overlay.ko 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/pstore 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/pstore/ramoops.ko 2023-07-29T15:42:39.057 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/reiserfs 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/reiserfs/reiserfs.ko 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/romfs 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/romfs/romfs.ko 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/smb 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/smb/client 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/smb/client/cifs.ko 2023-07-29T15:42:39.058 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/smb/common 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/smb/common/cifs_arc4.ko 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/smb/common/cifs_md4.ko 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/squashfs 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/squashfs/squashfs.ko 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/sysv 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/sysv/sysv.ko 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/udf 2023-07-29T15:42:39.059 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/udf/udf.ko 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ufs 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/ufs/ufs.ko 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/xfs 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/fs/xfs/xfs.ko 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/kernel 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/kernel/rcu 2023-07-29T15:42:39.060 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/kernel/rcu/rcutorture.ko 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/kernel/torture.ko 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/kernel/trace 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/asn1_encoder.ko 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/crc-itu-t.ko 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/crc8.ko 2023-07-29T15:42:39.061 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/crypto 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/crypto/libarc4.ko 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/crypto/libdes.ko 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/libcrc32c.ko 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/lru_cache.ko 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/lz4 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/lz4/lz4_compress.ko 2023-07-29T15:42:39.062 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/lz4/lz4hc_compress.ko 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/math 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/math/cordic.ko 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/raid6 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/raid6/raid6_pq.ko 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/reed_solomon 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/reed_solomon/reed_solomon.ko 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/ts_bm.ko 2023-07-29T15:42:39.063 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/ts_fsm.ko 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/ts_kmp.ko 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/zstd 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/lib/zstd/zstd_compress.ko 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/mm 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/mm/hwpoison-inject.ko 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan 2023-07-29T15:42:39.064 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/6lowpan.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_dest.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_fragment.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_hop.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_ipv6.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_mobility.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_routing.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/6lowpan/nhc_udp.ko 2023-07-29T15:42:39.065 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/802 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/802/garp.ko 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/802/mrp.ko 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/802/p8022.ko 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/802/psnap.ko 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/802/stp.ko 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/8021q 2023-07-29T15:42:39.066 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/8021q/8021q.ko 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/9p 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/9p/9pnet.ko 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/9p/9pnet_fd.ko 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/9p/9pnet_rdma.ko 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/9p/9pnet_virtio.ko 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/batman-adv 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/batman-adv/batman-adv.ko 2023-07-29T15:42:39.067 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/br_netfilter.ko 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/bridge.ko 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_among.ko 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_arp.ko 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-07-29T15:42:39.068 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_ip.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_limit.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_log.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_mark.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-07-29T15:42:39.069 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_snat.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_stp.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-07-29T15:42:39.070 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/ebtables.ko 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ceph 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ceph/libceph.ko 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/core 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/core/failover.ko 2023-07-29T15:42:39.071 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/core/pktgen.ko 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dccp 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dccp/dccp.ko 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dccp/dccp_diag.ko 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dccp/dccp_ipv4.ko 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dccp/dccp_ipv6.ko 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dns_resolver 2023-07-29T15:42:39.072 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dns_resolver/dns_resolver.ko 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dsa 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dsa/dsa_core.ko 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dsa/tag_brcm.ko 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dsa/tag_dsa.ko 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dsa/tag_none.ko 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/dsa/tag_trailer.ko 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ieee802154 2023-07-29T15:42:39.073 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ieee802154/6lowpan 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ieee802154/ieee802154.ko 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ieee802154/ieee802154_socket.ko 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/ah4.ko 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/esp4.ko 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/fou.ko 2023-07-29T15:42:39.074 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/gre.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/inet_diag.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/ip_gre.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/ip_tunnel.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/ip_vti.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/ipcomp.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/ipip.ko 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter 2023-07-29T15:42:39.075 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/arp_tables.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-07-29T15:42:39.076 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/iptable_security.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-07-29T15:42:39.077 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_bic.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_dctcp.ko 2023-07-29T15:42:39.078 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_diag.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_highspeed.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_htcp.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_hybla.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_illinois.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_lp.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_scalable.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_vegas.ko 2023-07-29T15:42:39.079 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_veno.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_westwood.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tcp_yeah.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/tunnel4.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/udp_diag.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/udp_tunnel.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv4/xfrm4_tunnel.ko 2023-07-29T15:42:39.080 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/ah6.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/esp6.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/fou6.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/ip6_tunnel.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/ip6_vti.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/ipcomp6.ko 2023-07-29T15:42:39.081 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-07-29T15:42:39.082 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-07-29T15:42:39.083 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-07-29T15:42:39.084 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/sit.ko 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/tunnel6.ko 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/ipv6/xfrm6_tunnel.ko 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/key 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/key/af_key.ko 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_core.ko 2023-07-29T15:42:39.085 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_debugfs.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_eth.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_ip.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_ip6.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_netlink.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/l2tp/l2tp_ppp.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/llc 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/llc/llc.ko 2023-07-29T15:42:39.086 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/mac802154 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/mac802154/mac802154.ko 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/mpls 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/mpls/mpls_gso.ko 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set.ko 2023-07-29T15:42:39.087 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-07-29T15:42:39.088 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs 2023-07-29T15:42:39.089 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-07-29T15:42:39.090 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conncount.ko 2023-07-29T15:42:39.091 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_h323.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_irc.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-07-29T15:42:39.092 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_sane.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_sip.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_dup_netdev.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_flow_table.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_flow_table_inet.ko 2023-07-29T15:42:39.093 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_log_syslog.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_nat.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_nat_amanda.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_nat_ftp.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_nat_irc.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_nat_sip.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_nat_tftp.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_synproxy_core.ko 2023-07-29T15:42:39.094 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nf_tables.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nfnetlink.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nfnetlink_acct.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nfnetlink_log.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nfnetlink_osf.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nfnetlink_queue.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_chain_nat.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_compat.ko 2023-07-29T15:42:39.095 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_ct.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_dup_netdev.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_fib.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_fwd_netdev.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_hash.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_limit.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_log.ko 2023-07-29T15:42:39.096 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_masq.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_nat.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_numgen.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_queue.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_quota.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_redir.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_reject.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_reject_inet.ko 2023-07-29T15:42:39.097 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_socket.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/nft_tunnel.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_AUDIT.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_CHECKSUM.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_CLASSIFY.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_CT.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_DSCP.ko 2023-07-29T15:42:39.098 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_HL.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_HMARK.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_IDLETIMER.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_LED.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_LOG.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_MASQUERADE.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_NETMAP.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_NFLOG.ko 2023-07-29T15:42:39.099 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_NFQUEUE.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_RATEEST.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_REDIRECT.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_SECMARK.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_TCPMSS.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_TEE.ko 2023-07-29T15:42:39.100 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_TPROXY.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_TRACE.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_addrtype.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_bpf.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_cgroup.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_cluster.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_comment.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_connbytes.ko 2023-07-29T15:42:39.101 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_connlabel.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_connlimit.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_connmark.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_conntrack.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_cpu.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_dccp.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_devgroup.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_dscp.ko 2023-07-29T15:42:39.102 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_ecn.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_esp.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_hashlimit.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_helper.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_hl.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_ipcomp.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_iprange.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_ipvs.ko 2023-07-29T15:42:39.103 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_l2tp.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_length.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_limit.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_mac.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_mark.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_multiport.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_nat.ko 2023-07-29T15:42:39.104 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_nfacct.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_osf.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_owner.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_physdev.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_pkttype.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_policy.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_quota.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_rateest.ko 2023-07-29T15:42:39.105 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_realm.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_recent.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_sctp.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_set.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_socket.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_state.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_statistic.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_string.ko 2023-07-29T15:42:39.106 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_tcpmss.ko 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_time.ko 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netfilter/xt_u32.ko 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netlink 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/netlink/netlink_diag.ko 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/nsh 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/nsh/nsh.ko 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/openvswitch 2023-07-29T15:42:39.107 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/openvswitch/openvswitch.ko 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/openvswitch/vport-geneve.ko 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/openvswitch/vport-gre.ko 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/openvswitch/vport-vxlan.ko 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/packet 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/packet/af_packet_diag.ko 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/rds 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/rds/rds.ko 2023-07-29T15:42:39.108 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/rds/rds_rdma.ko 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/rds/rds_tcp.ko 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_csum.ko 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_gact.ko 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_ipt.ko 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_mirred.ko 2023-07-29T15:42:39.109 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_nat.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_pedit.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_police.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_simple.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_skbedit.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/act_vlan.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/cls_basic.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/cls_bpf.ko 2023-07-29T15:42:39.110 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/cls_flow.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/cls_fw.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/cls_route.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/cls_u32.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/em_cmp.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/em_ipset.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/em_meta.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/em_nbyte.ko 2023-07-29T15:42:39.111 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/em_text.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/em_u32.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_choke.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_codel.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_drr.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_fq.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_gred.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_hfsc.ko 2023-07-29T15:42:39.112 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_hhf.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_htb.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_ingress.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_mqprio.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_mqprio_lib.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_multiq.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_netem.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_pie.ko 2023-07-29T15:42:39.113 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_plug.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_prio.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_qfq.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_red.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_sfb.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_sfq.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_tbf.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sched/sch_teql.ko 2023-07-29T15:42:39.114 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sctp 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sctp/sctp.ko 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sctp/sctp_diag.ko 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc/auth_gss 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc/sunrpc.ko 2023-07-29T15:42:39.115 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc/xprtrdma 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/tipc 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/tipc/diag.ko 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/tipc/tipc.ko 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/unix 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/unix/unix_diag.ko 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/vmw_vsock 2023-07-29T15:42:39.116 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/vmw_vsock/vsock.ko 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/vmw_vsock/vsock_diag.ko 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/vmw_vsock/vsock_loopback.ko 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/xfrm 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/net/xfrm/xfrm_ipcomp.ko 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/security 2023-07-29T15:42:39.117 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/security/keys 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/security/keys/encrypted-keys 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/security/keys/trusted-keys 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/security/keys/trusted-keys/trusted.ko 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/virt 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/virt/lib 2023-07-29T15:42:39.118 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/kernel/virt/lib/irqbypass.ko 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.alias 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.alias.bin 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.builtin 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.builtin.bin 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.builtin.modinfo 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.dep 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.dep.bin 2023-07-29T15:42:39.119 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.devname 2023-07-29T15:42:39.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.order 2023-07-29T15:42:39.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.softdep 2023-07-29T15:42:39.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.symbols 2023-07-29T15:42:39.120 INFO:teuthology.orchestra.run.smithi031.stdout:/lib/modules/6.5.0-rc1-g8e3722cc0164/modules.symbols.bin 2023-07-29T15:42:39.121 DEBUG:teuthology.task.kernel:get_image_version: 6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:39.121 DEBUG:teuthology.orchestra.run.smithi031:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-07-29T15:42:39.150 INFO:teuthology.orchestra.run.smithi031.stdout:postinstall scriptlet (using /bin/sh): 2023-07-29T15:42:39.150 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.5.0-rc1-g8e3722cc0164 -a -r /boot/System.map-6.5.0-rc1-g8e3722cc0164 ]; then 2023-07-29T15:42:39.150 INFO:teuthology.orchestra.run.smithi031.stdout:cp /boot/vmlinuz-6.5.0-rc1-g8e3722cc0164 /boot/.vmlinuz-6.5.0-rc1-g8e3722cc0164-rpm 2023-07-29T15:42:39.150 INFO:teuthology.orchestra.run.smithi031.stdout:cp /boot/System.map-6.5.0-rc1-g8e3722cc0164 /boot/.System.map-6.5.0-rc1-g8e3722cc0164-rpm 2023-07-29T15:42:39.150 INFO:teuthology.orchestra.run.smithi031.stdout:rm -f /boot/vmlinuz-6.5.0-rc1-g8e3722cc0164 /boot/System.map-6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:39.150 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/installkernel 6.5.0-rc1-g8e3722cc0164 /boot/.vmlinuz-6.5.0-rc1-g8e3722cc0164-rpm /boot/.System.map-6.5.0-rc1-g8e3722cc0164-rpm 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:rm -f /boot/.vmlinuz-6.5.0-rc1-g8e3722cc0164-rpm /boot/.System.map-6.5.0-rc1-g8e3722cc0164-rpm 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:preuninstall scriptlet (using /bin/sh): 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:new-kernel-pkg --remove 6.5.0-rc1-g8e3722cc0164 --rminitrd --initrdfile=/boot/initramfs-6.5.0-rc1-g8e3722cc0164.img 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-07-29T15:42:39.151 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-install remove 6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:39.152 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2023-07-29T15:42:39.152 INFO:teuthology.orchestra.run.smithi031.stdout:postuninstall scriptlet (using /bin/sh): 2023-07-29T15:42:39.152 INFO:teuthology.orchestra.run.smithi031.stdout:if [ -x /sbin/update-bootloader ]; then 2023-07-29T15:42:39.152 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/update-bootloader --remove 6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:39.152 INFO:teuthology.orchestra.run.smithi031.stdout:fi 2023-07-29T15:42:39.153 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-07-29T15:42:39.233 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-07-29T15:42:39.233 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-07-29T15:42:39.233 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.02 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 123.el8 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Thu 14 Jul 2022 06:22:19 PM UTC 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:Group : System Environment/Base 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 9265450 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-07-29T15:42:39.234 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Relocations : (not relocatable) 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : Red Hat, Inc. 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : Red Hat, Inc. 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-07-29T15:42:39.235 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-07-29T15:42:39.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:42:39.237 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-07-29T15:42:39.237 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-07-29T15:42:39.238 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:39.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-07-29T15:42:41.187 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-07-29T15:42:43.095 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-07-29T15:42:43.097 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-07-29T15:42:43.114 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.M9fea7K86r 2023-07-29T15:42:43.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.M9fea7K86r 2023-07-29T15:42:43.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.M9fea7K86r 2023-07-29T15:42:43.336 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.M9fea7K86r is 6KB 2023-07-29T15:42:43.358 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.M9fea7K86r 2023-07-29T15:42:43.375 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-07-29T15:42:43.448 INFO:teuthology.orchestra.run.smithi031.stdout:5a84457b7b52416e95d47d02e2e4a60f-0-rescue.conf 2023-07-29T15:42:43.448 INFO:teuthology.orchestra.run.smithi031.stdout:5a84457b7b52416e95d47d02e2e4a60f-4.18.0-372.9.1.el8.x86_64.conf 2023-07-29T15:42:43.448 INFO:teuthology.orchestra.run.smithi031.stdout:5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc1-g8e3722cc0164.conf 2023-07-29T15:42:43.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 5a84457b7b52416e95d47d02e2e4a60f-6.5.0-rc1-g8e3722cc0164 2023-07-29T15:42:43.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-07-29T15:43:13.597 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-07-29T15:43:13.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:43:13.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:43:31.952 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-07-29T15:43:40.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:43:40.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:43:59.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-07-29T15:44:11.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:44:11.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:44:14.420 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.31 2023-07-29T15:44:29.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:44:29.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:44:29.768 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-07-29T15:44:30.312 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:44:30.312 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.5.0-rc1-g8e3722cc0164"... 2023-07-29T15:44:30.313 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-07-29T15:44:30.355 INFO:teuthology.orchestra.run.smithi031.stdout:6.5.0-rc1-g8e3722cc0164 2023-07-29T15:44:30.355 DEBUG:teuthology.task.kernel:current kernel version is 6.5.0-rc1-g8e3722cc0164 vs 6.5.0-rc1-g8e3722cc0164 2023-07-29T15:44:30.355 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-07-29T15:44:30.355 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-07-29T15:44:31.356 DEBUG:teuthology.parallel:result is None 2023-07-29T15:44:31.356 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-29T15:44:31.363 INFO:teuthology.task.internal:Creating test directory... 2023-07-29T15:44:31.363 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-29T15:44:31.368 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-29T15:44:31.370 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-29T15:44:31.410 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-29T15:44:31.417 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-29T15:44:31.428 INFO:teuthology.task.internal:Creating archive directory... 2023-07-29T15:44:31.429 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-29T15:44:31.433 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-29T15:44:31.435 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-29T15:44:31.488 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-29T15:44:31.495 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-29T15:44:31.496 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-29T15:44:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T15:44:31.528 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-29T15:44:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T15:44:31.547 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-29T15:44:31.551 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-29T15:44:31.569 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-29T15:44:31.609 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T15:44:31.625 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T15:44:31.626 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T15:44:31.643 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T15:44:31.687 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T15:44:31.722 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-29T15:44:31.725 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-29T15:44:31.732 INFO:teuthology.task.internal:Configuring sudo... 2023-07-29T15:44:31.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-29T15:44:31.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-29T15:44:31.738 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-29T15:44:31.802 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-29T15:44:31.811 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-29T15:44:31.811 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-29T15:44:31.816 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-29T15:44:31.818 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-29T15:44:31.853 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T15:44:31.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T15:44:32.037 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T15:44:32.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T15:44:32.191 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T15:44:32.191 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-29T15:44:32.260 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T15:44:32.305 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T15:44:32.374 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T15:44:32.424 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T15:44:32.466 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T15:44:32.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-29T15:44:32.536 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T15:44:32.586 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-29T15:44:32.636 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T15:44:32.725 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-29T15:44:32.768 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T15:44:32.769 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-29T15:44:32.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-07-29T15:44:32.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2023-07-29T15:44:32.867 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2023-07-29T15:44:32.936 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T15:44:32.977 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T15:44:32.980 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T15:44:33.406 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-29T15:44:33.415 INFO:teuthology.task.internal:Starting timer... 2023-07-29T15:44:33.415 INFO:teuthology.run_tasks:Running task pcp... 2023-07-29T15:44:33.443 INFO:teuthology.run_tasks:Running task selinux... 2023-07-29T15:44:33.452 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-29T15:44:33.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-07-29T15:44:33.576 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-07-29T15:44:33.580 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2023-07-29T15:44:33.654 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2023-07-29T15:44:33.656 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2023-07-29T15:44:33.706 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2023-07-29T15:44:33.708 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-29T15:44:33.708 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-07-29T15:44:33.757 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-07-29T15:44:33.758 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2023-07-29T15:44:33.780 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2023-07-29T15:44:33.781 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2023-07-29T15:44:33.805 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2023-07-29T15:44:33.806 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2023-07-29T15:44:33.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T15:44:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T15:44:33.884 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T15:44:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T15:44:33.914 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T15:44:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T15:44:33.942 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-29T15:44:33.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-07-29T15:44:34.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2023-07-29T15:44:34.040 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2023-07-29T15:44:34.066 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-29T15:44:34.076 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-29T15:44:34.085 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-29T15:44:34.215 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-29T15:44:34.234 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-29T15:44:34.236 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi039.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-29T15:56:23.706 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2023-07-29T15:56:23.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:56:23.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:56:23.797 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-07-29T15:56:23.911 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-07-29T15:56:23.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-07-29T15:56:23.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:56:23.982 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-07-29T15:56:24.055 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-07-29T15:56:24.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-07-29T15:56:24.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-29T15:56:24.125 DEBUG:teuthology.orchestra.run.smithi156:> true 2023-07-29T15:56:24.153 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2023-07-29T15:56:24.154 INFO:teuthology.run_tasks:Running task clock... 2023-07-29T15:56:24.165 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-29T15:56:24.165 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-29T15:56:24.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T15:56:24.170 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-29T15:56:24.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T15:56:24.173 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-29T15:56:24.173 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T15:56:24.210 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-29T15:56:24.227 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-29T15:56:24.228 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-29T15:56:24.242 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-29T15:56:24.243 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-29T15:56:24.279 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-29T15:56:24.296 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2023-07-29T15:56:24.309 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2023-07-29T15:56:24.315 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2023-07-29T15:56:24.325 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-29T15:56:24.328 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2023-07-29T15:56:24.340 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-29T15:56:24.343 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-29T15:56:24.357 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-29T15:56:24.374 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-07-29T15:56:24.404 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-07-29T15:56:24.415 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-07-29T15:56:24.418 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T15:56:24.419 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-07-29T15:56:24.419 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.419 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.419 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.419 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.426 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-07-29T15:56:24.429 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T15:56:24.429 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-07-29T15:56:24.430 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.430 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.430 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.430 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.439 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-29T15:56:24.474 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-29T15:56:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-07-29T15:56:24.578 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T15:56:24.578 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-07-29T15:56:24.578 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.578 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.578 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.578 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T15:56:24.579 INFO:teuthology.run_tasks:Running task pexec... 2023-07-29T15:56:24.590 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-29T15:56:24.590 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T15:56:24.590 DEBUG:teuthology.orchestra.run.smithi031:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-29T15:56:24.591 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi039.front.sepia.ceph.com 2023-07-29T15:56:24.591 DEBUG:teuthology.orchestra.run.smithi039:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-29T15:56:24.592 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi156.front.sepia.ceph.com 2023-07-29T15:56:24.592 DEBUG:teuthology.orchestra.run.smithi156:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-29T15:56:24.965 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:24.984 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:25.107 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:25.774 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:50 ago on Sat 29 Jul 2023 03:51:35 PM UTC. 2023-07-29T15:56:25.862 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:23 ago on Sat 29 Jul 2023 03:54:02 PM UTC. 2023-07-29T15:56:26.085 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:48 ago on Sat 29 Jul 2023 03:54:37 PM UTC. 2023-07-29T15:56:27.495 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-07-29T15:56:27.496 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout:Resetting modules: 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-07-29T15:56:27.497 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2023-07-29T15:56:27.498 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:27.836 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-07-29T15:56:27.837 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-07-29T15:56:27.837 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-07-29T15:56:27.837 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-07-29T15:56:27.837 INFO:teuthology.orchestra.run.smithi039.stdout:Resetting modules: 2023-07-29T15:56:27.838 INFO:teuthology.orchestra.run.smithi039.stdout: container-tools 2023-07-29T15:56:27.838 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:27.838 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-07-29T15:56:27.838 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-07-29T15:56:27.838 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:Resetting modules: 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout: container-tools 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:28.098 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-07-29T15:56:28.099 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-07-29T15:56:28.099 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:28.465 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:53 ago on Sat 29 Jul 2023 03:51:35 PM UTC. 2023-07-29T15:56:28.867 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:26 ago on Sat 29 Jul 2023 03:54:02 PM UTC. 2023-07-29T15:56:29.367 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:51 ago on Sat 29 Jul 2023 03:54:37 PM UTC. 2023-07-29T15:56:30.728 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-07-29T15:56:31.231 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:31.240 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-07-29T15:56:31.738 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:31.850 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-07-29T15:56:32.031 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:04:56 ago on Sat 29 Jul 2023 03:51:35 PM UTC. 2023-07-29T15:56:32.476 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:32.579 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:30 ago on Sat 29 Jul 2023 03:54:02 PM UTC. 2023-07-29T15:56:33.488 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:55 ago on Sat 29 Jul 2023 03:54:37 PM UTC. 2023-07-29T15:56:33.767 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-07-29T15:56:33.860 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-07-29T15:56:33.861 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-07-29T15:56:33.861 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-07-29T15:56:33.861 INFO:teuthology.orchestra.run.smithi156.stdout:Installing group/module packages: 2023-07-29T15:56:33.861 INFO:teuthology.orchestra.run.smithi156.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-29T15:56:33.861 INFO:teuthology.orchestra.run.smithi156.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-29T15:56:33.862 INFO:teuthology.orchestra.run.smithi156.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-29T15:56:33.862 INFO:teuthology.orchestra.run.smithi156.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-29T15:56:33.862 INFO:teuthology.orchestra.run.smithi156.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-29T15:56:33.862 INFO:teuthology.orchestra.run.smithi156.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout:Installing module profiles: 2023-07-29T15:56:33.863 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools/common 2023-07-29T15:56:33.864 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:33.864 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling module streams: 2023-07-29T15:56:33.864 INFO:teuthology.orchestra.run.smithi156.stdout: container-tools rhel8 2023-07-29T15:56:33.864 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:33.864 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-07-29T15:56:33.865 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================================ 2023-07-29T15:56:33.865 INFO:teuthology.orchestra.run.smithi156.stdout:Install 10 Packages 2023-07-29T15:56:33.865 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:33.865 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 18 M 2023-07-29T15:56:33.866 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 65 M 2023-07-29T15:56:33.866 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-07-29T15:56:33.928 INFO:teuthology.orchestra.run.smithi156.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2023-07-29T15:56:33.945 INFO:teuthology.orchestra.run.smithi156.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 305 kB/s | 25 kB 00:00 2023-07-29T15:56:33.962 INFO:teuthology.orchestra.run.smithi156.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 414 kB/s | 41 kB 00:00 2023-07-29T15:56:33.979 INFO:teuthology.orchestra.run.smithi156.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 980 kB/s | 49 kB 00:00 2023-07-29T15:56:34.013 INFO:teuthology.orchestra.run.smithi156.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 7.6 MB/s | 524 kB 00:00 2023-07-29T15:56:34.030 INFO:teuthology.orchestra.run.smithi156.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.9 MB/s | 149 kB 00:00 2023-07-29T15:56:34.055 INFO:teuthology.orchestra.run.smithi156.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 5.0 MB/s | 213 kB 00:00 2023-07-29T15:56:34.164 INFO:teuthology.orchestra.run.smithi156.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 20 MB/s | 2.2 MB 00:00 2023-07-29T15:56:34.297 INFO:teuthology.orchestra.run.smithi156.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 26 MB/s | 7.0 MB 00:00 2023-07-29T15:56:34.389 INFO:teuthology.orchestra.run.smithi156.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 19 MB/s | 8.0 MB 00:00 2023-07-29T15:56:34.390 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-07-29T15:56:34.390 INFO:teuthology.orchestra.run.smithi156.stdout:Total 35 MB/s | 18 MB 00:00 2023-07-29T15:56:34.552 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-07-29T15:56:34.592 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-07-29T15:56:34.611 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-07-29T15:56:34.611 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-07-29T15:56:34.654 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================================ 2023-07-29T15:56:34.654 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================================ 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout:Installing group/module packages: 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-29T15:56:34.655 INFO:teuthology.orchestra.run.smithi039.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-29T15:56:34.656 INFO:teuthology.orchestra.run.smithi039.stdout:Installing module profiles: 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout: container-tools/common 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling module streams: 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout: container-tools rhel8 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================================ 2023-07-29T15:56:34.657 INFO:teuthology.orchestra.run.smithi039.stdout:Install 10 Packages 2023-07-29T15:56:34.658 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:34.658 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 18 M 2023-07-29T15:56:34.658 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 65 M 2023-07-29T15:56:34.658 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-07-29T15:56:34.772 INFO:teuthology.orchestra.run.smithi039.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 355 kB/s | 41 kB 00:00 2023-07-29T15:56:34.794 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-07-29T15:56:34.798 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-07-29T15:56:34.798 INFO:teuthology.orchestra.run.smithi039.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 176 kB/s | 25 kB 00:00 2023-07-29T15:56:34.823 INFO:teuthology.orchestra.run.smithi039.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 976 kB/s | 49 kB 00:00 2023-07-29T15:56:34.840 INFO:teuthology.orchestra.run.smithi039.stdout:(4/10): python3-pyxdg-0.25-16.el8.noarch.rpm 512 kB/s | 94 kB 00:00 2023-07-29T15:56:34.874 INFO:teuthology.orchestra.run.smithi039.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.8 MB/s | 524 kB 00:00 2023-07-29T15:56:34.899 INFO:teuthology.orchestra.run.smithi039.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-29T15:56:34.916 INFO:teuthology.orchestra.run.smithi039.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 5.0 MB/s | 213 kB 00:00 2023-07-29T15:56:35.008 INFO:teuthology.orchestra.run.smithi039.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-29T15:56:35.125 INFO:teuthology.orchestra.run.smithi039.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 31 MB/s | 7.0 MB 00:00 2023-07-29T15:56:35.241 INFO:teuthology.orchestra.run.smithi039.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 19 MB/s | 8.0 MB 00:00 2023-07-29T15:56:35.242 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-07-29T15:56:35.242 INFO:teuthology.orchestra.run.smithi039.stdout:Total 31 MB/s | 18 MB 00:00 2023-07-29T15:56:35.299 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-07-29T15:56:35.393 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-07-29T15:56:35.450 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-07-29T15:56:35.471 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-07-29T15:56:35.471 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-07-29T15:56:35.499 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================================ 2023-07-29T15:56:35.499 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-07-29T15:56:35.499 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================================ 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout:Installing group/module packages: 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-29T15:56:35.500 INFO:teuthology.orchestra.run.smithi031.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout:Installing module profiles: 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: container-tools/common 2023-07-29T15:56:35.501 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling module streams: 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: container-tools rhel8 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================================ 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout:Install 10 Packages 2023-07-29T15:56:35.502 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:35.503 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 18 M 2023-07-29T15:56:35.503 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 65 M 2023-07-29T15:56:35.503 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-07-29T15:56:35.519 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-29T15:56:35.587 INFO:teuthology.orchestra.run.smithi031.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2023-07-29T15:56:35.612 INFO:teuthology.orchestra.run.smithi031.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 227 kB/s | 25 kB 00:00 2023-07-29T15:56:35.637 INFO:teuthology.orchestra.run.smithi031.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 303 kB/s | 41 kB 00:00 2023-07-29T15:56:35.663 INFO:teuthology.orchestra.run.smithi031.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 655 kB/s | 49 kB 00:00 2023-07-29T15:56:35.663 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-07-29T15:56:35.668 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-07-29T15:56:35.696 INFO:teuthology.orchestra.run.smithi031.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-29T15:56:35.707 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-29T15:56:35.722 INFO:teuthology.orchestra.run.smithi031.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-29T15:56:35.755 INFO:teuthology.orchestra.run.smithi031.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 3.6 MB/s | 213 kB 00:00 2023-07-29T15:56:35.881 INFO:teuthology.orchestra.run.smithi031.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 18 MB/s | 2.2 MB 00:00 2023-07-29T15:56:35.893 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-29T15:56:36.039 INFO:teuthology.orchestra.run.smithi031.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 22 MB/s | 7.0 MB 00:00 2023-07-29T15:56:36.095 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-29T15:56:36.131 INFO:teuthology.orchestra.run.smithi031.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 16 MB/s | 8.0 MB 00:00 2023-07-29T15:56:36.132 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-07-29T15:56:36.132 INFO:teuthology.orchestra.run.smithi031.stdout:Total 29 MB/s | 18 MB 00:00 2023-07-29T15:56:36.219 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-07-29T15:56:36.444 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-29T15:56:36.452 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-29T15:56:36.471 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-07-29T15:56:36.495 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-07-29T15:56:36.496 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-07-29T15:56:36.616 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-29T15:56:36.855 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-29T15:56:37.019 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-07-29T15:56:37.024 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-07-29T15:56:37.049 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-29T15:56:37.207 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-29T15:56:37.456 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-29T15:56:37.576 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-07-29T15:56:37.768 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-29T15:56:37.953 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-29T15:56:38.107 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-29T15:56:38.174 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-29T15:56:38.298 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-29T15:56:38.336 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-29T15:56:38.398 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-29T15:56:38.505 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-29T15:56:38.791 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-29T15:56:38.824 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-29T15:56:39.097 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-29T15:56:39.097 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-29T15:56:39.097 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-29T15:56:39.097 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-29T15:56:39.097 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-29T15:56:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-29T15:56:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-29T15:56:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-29T15:56:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-29T15:56:39.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-29T15:56:39.237 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-29T15:56:39.465 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-29T15:56:39.664 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-29T15:56:39.676 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-29T15:56:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-29T15:56:40.100 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-29T15:56:40.100 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-29T15:56:40.100 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-29T15:56:40.100 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-29T15:56:40.100 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-29T15:56:40.101 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-29T15:56:40.101 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-29T15:56:40.101 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-29T15:56:40.101 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-29T15:56:40.101 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-29T15:56:40.252 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-29T15:56:40.253 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:05:04 ago on Sat 29 Jul 2023 03:51:35 PM UTC. 2023-07-29T15:56:40.693 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-29T15:56:40.941 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-29T15:56:41.118 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-29T15:56:41.291 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-29T15:56:41.341 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-29T15:56:41.341 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:38 ago on Sat 29 Jul 2023 03:54:02 PM UTC. 2023-07-29T15:56:41.745 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-29T15:56:41.745 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-29T15:56:41.745 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-29T15:56:41.745 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-29T15:56:41.745 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-29T15:56:41.745 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-29T15:56:41.746 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-29T15:56:41.746 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-29T15:56:41.746 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-29T15:56:41.747 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-29T15:56:43.004 INFO:teuthology.orchestra.run.smithi156.stdout:Installed products updated. 2023-07-29T15:56:43.200 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-29T15:56:43.200 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:05 ago on Sat 29 Jul 2023 03:54:37 PM UTC. 2023-07-29T15:56:43.236 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:43.236 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-07-29T15:56:43.236 INFO:teuthology.orchestra.run.smithi156.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:43.236 INFO:teuthology.orchestra.run.smithi156.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-29T15:56:43.237 INFO:teuthology.orchestra.run.smithi156.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-29T15:56:43.238 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:56:43.238 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-07-29T15:56:43.444 DEBUG:teuthology.parallel:result is None 2023-07-29T15:56:43.994 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-07-29T15:56:44.246 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:44.246 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-07-29T15:56:44.246 INFO:teuthology.orchestra.run.smithi039.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:44.246 INFO:teuthology.orchestra.run.smithi039.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-29T15:56:44.246 INFO:teuthology.orchestra.run.smithi039.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-29T15:56:44.247 INFO:teuthology.orchestra.run.smithi039.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-29T15:56:44.248 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:56:44.248 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-07-29T15:56:44.437 DEBUG:teuthology.parallel:result is None 2023-07-29T15:56:45.979 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-07-29T15:56:46.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:46.207 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-29T15:56:46.208 INFO:teuthology.orchestra.run.smithi031.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:46.209 INFO:teuthology.orchestra.run.smithi031.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-29T15:56:46.209 INFO:teuthology.orchestra.run.smithi031.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-29T15:56:46.209 INFO:teuthology.orchestra.run.smithi031.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-29T15:56:46.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T15:56:46.209 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-07-29T15:56:46.424 DEBUG:teuthology.parallel:result is None 2023-07-29T15:56:46.424 INFO:teuthology.run_tasks:Running task install... 2023-07-29T15:56:46.432 DEBUG:teuthology.task.install:project ceph 2023-07-29T15:56:46.433 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'}} 2023-07-29T15:56:46.433 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577'} 2023-07-29T15:56:46.433 INFO:teuthology.task.install:Using flavor: default 2023-07-29T15:56:46.440 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-29T15:56:46.440 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-29T15:56:46.441 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'tag': None, 'wait_for_package': False} 2023-07-29T15:56:46.441 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T15:56:46.444 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'tag': None, 'wait_for_package': False} 2023-07-29T15:56:46.444 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T15:56:46.445 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'tag': None, 'wait_for_package': False} 2023-07-29T15:56:46.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T15:56:46.674 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/ 2023-07-29T15:56:46.674 INFO:teuthology.task.install.rpm:Package version is 18.1.3-11.g0484097b 2023-07-29T15:56:46.678 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/ 2023-07-29T15:56:46.679 INFO:teuthology.task.install.rpm:Package version is 18.1.3-11.g0484097b 2023-07-29T15:56:46.682 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/ 2023-07-29T15:56:46.682 INFO:teuthology.task.install.rpm:Package version is 18.1.3-11.g0484097b 2023-07-29T15:56:46.853 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-29T15:56:46.854 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T15:56:46.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-29T15:56:46.863 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-29T15:56:46.863 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T15:56:46.864 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-29T15:56:46.867 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-29T15:56:46.867 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T15:56:46.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-29T15:56:46.883 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-29T15:56:46.883 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0484097b0572e96f42cf6402cbe0ac8dcb046577/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-29T15:56:46.895 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-29T15:56:46.896 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0484097b0572e96f42cf6402cbe0ac8dcb046577/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-29T15:56:46.939 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-29T15:56:46.939 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0484097b0572e96f42cf6402cbe0ac8dcb046577/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-29T15:56:46.952 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-29T15:56:46.963 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-29T15:56:47.030 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-29T15:56:47.041 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-29T15:56:47.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-29T15:56:47.097 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2023-07-29T15:56:47.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2023-07-29T15:56:47.108 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2023-07-29T15:56:47.109 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2023-07-29T15:56:47.146 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-29T15:56:47.214 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-07-29T15:56:47.217 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-07-29T15:56:47.503 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:47.519 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:47.700 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:47.808 INFO:teuthology.orchestra.run.smithi039.stdout:44 files removed 2023-07-29T15:56:47.823 INFO:teuthology.orchestra.run.smithi156.stdout:44 files removed 2023-07-29T15:56:47.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-29T15:56:47.878 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-29T15:56:48.047 INFO:teuthology.orchestra.run.smithi031.stdout:44 files removed 2023-07-29T15:56:48.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-29T15:56:48.225 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:48.227 INFO:teuthology.orchestra.run.smithi156.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:48.580 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-07-29T15:56:49.098 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 190 kB/s | 80 kB 00:00 2023-07-29T15:56:49.103 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 193 kB/s | 80 kB 00:00 2023-07-29T15:56:49.541 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 168 kB/s | 80 kB 00:00 2023-07-29T15:56:49.558 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 22 kB/s | 9.4 kB 00:00 2023-07-29T15:56:49.562 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 22 kB/s | 9.4 kB 00:00 2023-07-29T15:56:50.141 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-07-29T15:56:50.147 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 2.7 kB/s | 1.5 kB 00:00 2023-07-29T15:56:50.160 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 2.6 kB/s | 1.5 kB 00:00 2023-07-29T15:56:50.498 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-07-29T15:56:50.528 INFO:teuthology.orchestra.run.smithi156.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-29T15:56:50.650 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 3.1 kB/s | 1.5 kB 00:00 2023-07-29T15:56:50.926 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2023-07-29T15:56:52.258 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2023-07-29T15:56:54.726 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 16 MB 00:04 2023-07-29T15:56:55.946 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 417 kB/s | 24 kB 00:00 2023-07-29T15:56:56.864 INFO:teuthology.orchestra.run.smithi156.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-07-29T15:56:58.450 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 453 kB/s | 24 kB 00:00 2023-07-29T15:56:59.444 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 52 MB/s | 47 MB 00:00 2023-07-29T15:57:05.215 INFO:teuthology.orchestra.run.smithi156.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 47 MB 00:01 2023-07-29T15:57:08.003 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 58 MB/s | 53 MB 00:00 2023-07-29T15:57:16.097 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-29T15:57:16.098 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-29T15:57:16.112 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-29T15:57:16.113 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-29T15:57:16.114 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-29T15:57:16.115 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-29T15:57:16.118 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-29T15:57:16.121 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-29T15:57:16.122 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-29T15:57:16.255 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2023-07-29T15:57:16.267 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-07-29T15:57:16.268 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2023-07-29T15:57:16.268 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-07-29T15:57:16.268 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2023-07-29T15:57:16.269 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-29T15:57:16.269 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-29T15:57:16.269 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:18.1.3-11.g0484097b.el8 ceph 6.4 k 2023-07-29T15:57:16.270 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:18.1.3-11.g0484097b.el8 ceph 5.4 M 2023-07-29T15:57:16.270 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:18.1.3-11.g0484097b.el8 ceph 847 k 2023-07-29T15:57:16.270 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:18.1.3-11.g0484097b.el8 ceph 165 k 2023-07-29T15:57:16.270 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:18.1.3-11.g0484097b.el8 ceph 1.5 M 2023-07-29T15:57:16.271 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 127 k 2023-07-29T15:57:16.271 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 1.7 M 2023-07-29T15:57:16.271 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 7.4 M 2023-07-29T15:57:16.272 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 47 k 2023-07-29T15:57:16.272 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:18.1.3-11.g0484097b.el8 ceph 13 M 2023-07-29T15:57:16.272 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:18.1.3-11.g0484097b.el8 ceph 47 M 2023-07-29T15:57:16.273 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 258 k 2023-07-29T15:57:16.273 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 160 k 2023-07-29T15:57:16.273 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:18.1.3-11.g0484097b.el8 ceph 257 k 2023-07-29T15:57:16.273 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 18 k 2023-07-29T15:57:16.274 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-29T15:57:16.274 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-29T15:57:16.274 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-29T15:57:16.275 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:18.1.3-11.g0484097b.el8 ceph 30 k 2023-07-29T15:57:16.275 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 769 k 2023-07-29T15:57:16.275 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:18.1.3-11.g0484097b.el8 ceph 139 k 2023-07-29T15:57:16.275 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-29T15:57:16.276 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-29T15:57:16.276 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-29T15:57:16.276 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-29T15:57:16.277 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-29T15:57:16.277 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-29T15:57:16.277 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:18.1.3-11.g0484097b.el8 ceph 190 k 2023-07-29T15:57:16.277 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:18.1.3-11.g0484097b.el8 ceph 371 k 2023-07-29T15:57:16.278 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:18.1.3-11.g0484097b.el8 ceph 364 k 2023-07-29T15:57:16.278 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:18.1.3-11.g0484097b.el8 ceph 107 k 2023-07-29T15:57:16.278 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:18.1.3-11.g0484097b.el8 ceph 94 k 2023-07-29T15:57:16.279 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.4 M 2023-07-29T15:57:16.279 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:18.1.3-11.g0484097b.el8 ceph 179 k 2023-07-29T15:57:16.279 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-29T15:57:16.280 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2023-07-29T15:57:16.280 INFO:teuthology.orchestra.run.smithi156.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-29T15:57:16.280 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.7 M 2023-07-29T15:57:16.281 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.6 M 2023-07-29T15:57:16.281 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2023-07-29T15:57:16.281 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-29T15:57:16.282 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:18.1.3-11.g0484097b.el8 ceph 22 M 2023-07-29T15:57:16.282 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 23 k 2023-07-29T15:57:16.282 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:18.1.3-11.g0484097b.el8 ceph 2.3 M 2023-07-29T15:57:16.283 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 240 k 2023-07-29T15:57:16.283 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:18.1.3-11.g0484097b.el8 ceph 4.6 M 2023-07-29T15:57:16.283 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:18.1.3-11.g0484097b.el8 ceph 18 M 2023-07-29T15:57:16.284 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 15 k 2023-07-29T15:57:16.284 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:18.1.3-11.g0484097b.el8 ceph 24 k 2023-07-29T15:57:16.284 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-29T15:57:16.284 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-29T15:57:16.285 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-29T15:57:16.285 INFO:teuthology.orchestra.run.smithi156.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-29T15:57:16.285 INFO:teuthology.orchestra.run.smithi156.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-29T15:57:16.285 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:18.1.3-11.g0484097b.el8 ceph 137 k 2023-07-29T15:57:16.286 INFO:teuthology.orchestra.run.smithi156.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-29T15:57:16.286 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-29T15:57:16.286 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-29T15:57:16.287 INFO:teuthology.orchestra.run.smithi156.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-29T15:57:16.287 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-29T15:57:16.287 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-29T15:57:16.287 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:18.1.3-11.g0484097b.el8 ceph 474 k 2023-07-29T15:57:16.287 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-29T15:57:16.287 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 6.5 M 2023-07-29T15:57:16.288 INFO:teuthology.orchestra.run.smithi156.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-29T15:57:16.288 INFO:teuthology.orchestra.run.smithi156.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-29T15:57:16.288 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-29T15:57:16.288 INFO:teuthology.orchestra.run.smithi156.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-29T15:57:16.288 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-29T15:57:16.288 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-29T15:57:16.289 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-29T15:57:16.290 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-29T15:57:16.291 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-29T15:57:16.291 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:18.1.3-11.g0484097b.el8 ceph 45 k 2023-07-29T15:57:16.291 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:18.1.3-11.g0484097b.el8 ceph 119 k 2023-07-29T15:57:16.291 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-29T15:57:16.291 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-29T15:57:16.292 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-29T15:57:16.292 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-29T15:57:16.292 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-29T15:57:16.292 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-29T15:57:16.292 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-29T15:57:16.292 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-29T15:57:16.293 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-29T15:57:16.293 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-29T15:57:16.293 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-29T15:57:16.293 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-29T15:57:16.294 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-29T15:57:16.294 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-29T15:57:16.294 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-29T15:57:16.294 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-29T15:57:16.294 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-29T15:57:16.294 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-29T15:57:16.295 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-29T15:57:16.296 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-29T15:57:16.296 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-29T15:57:16.296 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-29T15:57:16.296 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-29T15:57:16.296 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-29T15:57:16.297 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-29T15:57:16.297 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-07-29T15:57:16.297 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-29T15:57:16.297 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-29T15:57:16.297 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-29T15:57:16.298 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-29T15:57:16.298 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-29T15:57:16.298 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-29T15:57:16.298 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-29T15:57:16.298 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-29T15:57:16.298 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-29T15:57:16.299 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling module streams: 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql 10 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout:=========================================================================================================================== 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout:Install 128 Packages 2023-07-29T15:57:16.300 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 3 Packages 2023-07-29T15:57:16.301 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:57:16.301 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 203 M 2023-07-29T15:57:16.302 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2023-07-29T15:57:16.766 INFO:teuthology.orchestra.run.smithi156.stdout:(1/131): ceph-18.1.3-11.g0484097b.el8.x86_64.rp 27 kB/s | 6.4 kB 00:00 2023-07-29T15:57:17.383 INFO:teuthology.orchestra.run.smithi156.stdout:(2/131): ceph-fuse-18.1.3-11.g0484097b.el8.x86_ 1.3 MB/s | 847 kB 00:00 2023-07-29T15:57:17.541 INFO:teuthology.orchestra.run.smithi156.stdout:(3/131): ceph-immutable-object-cache-18.1.3-11. 1.0 MB/s | 165 kB 00:00 2023-07-29T15:57:18.709 INFO:teuthology.orchestra.run.smithi156.stdout:(4/131): ceph-base-18.1.3-11.g0484097b.el8.x86_ 2.5 MB/s | 5.4 MB 00:02 2023-07-29T15:57:18.843 INFO:teuthology.orchestra.run.smithi156.stdout:(5/131): ceph-mds-18.1.3-11.g0484097b.el8.x86_6 1.8 MB/s | 2.3 MB 00:01 2023-07-29T15:57:19.359 INFO:teuthology.orchestra.run.smithi156.stdout:(6/131): ceph-mgr-18.1.3-11.g0484097b.el8.x86_6 2.4 MB/s | 1.5 MB 00:00 2023-07-29T15:57:19.962 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-29T15:57:19.963 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-29T15:57:19.977 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-29T15:57:19.978 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-29T15:57:19.980 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-29T15:57:19.981 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-29T15:57:19.983 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-29T15:57:19.986 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-29T15:57:19.987 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-29T15:57:20.131 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-07-29T15:57:20.144 INFO:teuthology.orchestra.run.smithi039.stdout:=========================================================================================================================== 2023-07-29T15:57:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-07-29T15:57:20.145 INFO:teuthology.orchestra.run.smithi039.stdout:=========================================================================================================================== 2023-07-29T15:57:20.145 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-07-29T15:57:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-29T15:57:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:18.1.3-11.g0484097b.el8 ceph 6.4 k 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:18.1.3-11.g0484097b.el8 ceph 5.4 M 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:18.1.3-11.g0484097b.el8 ceph 847 k 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:18.1.3-11.g0484097b.el8 ceph 165 k 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:18.1.3-11.g0484097b.el8 ceph 1.5 M 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 127 k 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 1.7 M 2023-07-29T15:57:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 7.4 M 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 47 k 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:18.1.3-11.g0484097b.el8 ceph 13 M 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:18.1.3-11.g0484097b.el8 ceph 47 M 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 258 k 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 160 k 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:18.1.3-11.g0484097b.el8 ceph 257 k 2023-07-29T15:57:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 18 k 2023-07-29T15:57:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-29T15:57:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-29T15:57:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-29T15:57:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:18.1.3-11.g0484097b.el8 ceph 30 k 2023-07-29T15:57:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 769 k 2023-07-29T15:57:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:18.1.3-11.g0484097b.el8 ceph 139 k 2023-07-29T15:57:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-29T15:57:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-29T15:57:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-29T15:57:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-29T15:57:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-29T15:57:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-29T15:57:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:18.1.3-11.g0484097b.el8 ceph 190 k 2023-07-29T15:57:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:18.1.3-11.g0484097b.el8 ceph 371 k 2023-07-29T15:57:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:18.1.3-11.g0484097b.el8 ceph 364 k 2023-07-29T15:57:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:18.1.3-11.g0484097b.el8 ceph 107 k 2023-07-29T15:57:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:18.1.3-11.g0484097b.el8 ceph 94 k 2023-07-29T15:57:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.4 M 2023-07-29T15:57:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:18.1.3-11.g0484097b.el8 ceph 179 k 2023-07-29T15:57:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-29T15:57:20.153 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2023-07-29T15:57:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-29T15:57:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.7 M 2023-07-29T15:57:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.6 M 2023-07-29T15:57:20.153 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-07-29T15:57:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-29T15:57:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:18.1.3-11.g0484097b.el8 ceph 22 M 2023-07-29T15:57:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 23 k 2023-07-29T15:57:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:18.1.3-11.g0484097b.el8 ceph 2.3 M 2023-07-29T15:57:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 240 k 2023-07-29T15:57:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:18.1.3-11.g0484097b.el8 ceph 4.6 M 2023-07-29T15:57:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:18.1.3-11.g0484097b.el8 ceph 18 M 2023-07-29T15:57:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 15 k 2023-07-29T15:57:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:18.1.3-11.g0484097b.el8 ceph 24 k 2023-07-29T15:57:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-29T15:57:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-29T15:57:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-29T15:57:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-29T15:57:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-29T15:57:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:18.1.3-11.g0484097b.el8 ceph 137 k 2023-07-29T15:57:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-29T15:57:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-29T15:57:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:18.1.3-11.g0484097b.el8 ceph 474 k 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 6.5 M 2023-07-29T15:57:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-29T15:57:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-29T15:57:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:18.1.3-11.g0484097b.el8 ceph 45 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:18.1.3-11.g0484097b.el8 ceph 119 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-29T15:57:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-29T15:57:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-29T15:57:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-29T15:57:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-29T15:57:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-29T15:57:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-29T15:57:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-29T15:57:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-29T15:57:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-29T15:57:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-29T15:57:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-29T15:57:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-29T15:57:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-29T15:57:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-29T15:57:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-29T15:57:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-29T15:57:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-29T15:57:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-29T15:57:20.169 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling module streams: 2023-07-29T15:57:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql 10 2023-07-29T15:57:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:57:20.170 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-07-29T15:57:20.171 INFO:teuthology.orchestra.run.smithi039.stdout:=========================================================================================================================== 2023-07-29T15:57:20.171 INFO:teuthology.orchestra.run.smithi039.stdout:Install 128 Packages 2023-07-29T15:57:20.171 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 3 Packages 2023-07-29T15:57:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:57:20.173 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 203 M 2023-07-29T15:57:20.173 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-07-29T15:57:20.471 INFO:teuthology.orchestra.run.smithi039.stdout:(1/131): ceph-18.1.3-11.g0484097b.el8.x86_64.rp 23 kB/s | 6.4 kB 00:00 2023-07-29T15:57:20.860 INFO:teuthology.orchestra.run.smithi156.stdout:(7/131): ceph-mon-18.1.3-11.g0484097b.el8.x86_6 2.3 MB/s | 4.6 MB 00:02 2023-07-29T15:57:21.038 INFO:teuthology.orchestra.run.smithi039.stdout:(2/131): ceph-fuse-18.1.3-11.g0484097b.el8.x86_ 1.5 MB/s | 847 kB 00:00 2023-07-29T15:57:21.205 INFO:teuthology.orchestra.run.smithi039.stdout:(3/131): ceph-immutable-object-cache-18.1.3-11. 994 kB/s | 165 kB 00:00 2023-07-29T15:57:22.314 INFO:teuthology.orchestra.run.smithi039.stdout:(4/131): ceph-mds-18.1.3-11.g0484097b.el8.x86_6 2.1 MB/s | 2.3 MB 00:01 2023-07-29T15:57:22.731 INFO:teuthology.orchestra.run.smithi039.stdout:(5/131): ceph-base-18.1.3-11.g0484097b.el8.x86_ 2.1 MB/s | 5.4 MB 00:02 2023-07-29T15:57:22.973 INFO:teuthology.orchestra.run.smithi039.stdout:(6/131): ceph-mgr-18.1.3-11.g0484097b.el8.x86_6 2.4 MB/s | 1.5 MB 00:00 2023-07-29T15:57:23.952 INFO:teuthology.orchestra.run.smithi156.stdout:(8/131): ceph-common-18.1.3-11.g0484097b.el8.x8 2.9 MB/s | 22 MB 00:07 2023-07-29T15:57:24.035 INFO:teuthology.orchestra.run.smithi156.stdout:(9/131): ceph-selinux-18.1.3-11.g0484097b.el8.x 293 kB/s | 24 kB 00:00 2023-07-29T15:57:24.382 INFO:teuthology.orchestra.run.smithi039.stdout:(7/131): ceph-mon-18.1.3-11.g0484097b.el8.x86_6 2.8 MB/s | 4.6 MB 00:01 2023-07-29T15:57:25.602 INFO:teuthology.orchestra.run.smithi156.stdout:(10/131): ceph-osd-18.1.3-11.g0484097b.el8.x86_ 2.8 MB/s | 18 MB 00:06 2023-07-29T15:57:25.752 INFO:teuthology.orchestra.run.smithi156.stdout:(11/131): cephfs-mirror-18.1.3-11.g0484097b.el8 1.7 MB/s | 257 kB 00:00 2023-07-29T15:57:25.894 INFO:teuthology.orchestra.run.smithi156.stdout:(12/131): libcephfs-devel-18.1.3-11.g0484097b.e 215 kB/s | 30 kB 00:00 2023-07-29T15:57:26.028 INFO:teuthology.orchestra.run.smithi156.stdout:(13/131): ceph-radosgw-18.1.3-11.g0484097b.el8. 2.5 MB/s | 13 MB 00:05 2023-07-29T15:57:26.103 INFO:teuthology.orchestra.run.smithi156.stdout:(14/131): libcephsqlite-18.1.3-11.g0484097b.el8 1.8 MB/s | 137 kB 00:00 2023-07-29T15:57:26.178 INFO:teuthology.orchestra.run.smithi156.stdout:(15/131): librados-devel-18.1.3-11.g0484097b.el 1.8 MB/s | 139 kB 00:00 2023-07-29T15:57:26.204 INFO:teuthology.orchestra.run.smithi156.stdout:(16/131): libcephfs2-18.1.3-11.g0484097b.el8.x8 2.4 MB/s | 769 kB 00:00 2023-07-29T15:57:26.346 INFO:teuthology.orchestra.run.smithi156.stdout:(17/131): libradosstriper1-18.1.3-11.g0484097b. 2.8 MB/s | 474 kB 00:00 2023-07-29T15:57:26.412 INFO:teuthology.orchestra.run.smithi156.stdout:(18/131): python3-ceph-argparse-18.1.3-11.g0484 673 kB/s | 45 kB 00:00 2023-07-29T15:57:26.488 INFO:teuthology.orchestra.run.smithi156.stdout:(19/131): python3-ceph-common-18.1.3-11.g048409 1.5 MB/s | 119 kB 00:00 2023-07-29T15:57:26.613 INFO:teuthology.orchestra.run.smithi156.stdout:(20/131): python3-cephfs-18.1.3-11.g0484097b.el 1.5 MB/s | 190 kB 00:00 2023-07-29T15:57:26.813 INFO:teuthology.orchestra.run.smithi156.stdout:(21/131): python3-rados-18.1.3-11.g0484097b.el8 1.8 MB/s | 371 kB 00:00 2023-07-29T15:57:26.980 INFO:teuthology.orchestra.run.smithi156.stdout:(22/131): python3-rbd-18.1.3-11.g0484097b.el8.x 2.1 MB/s | 364 kB 00:00 2023-07-29T15:57:27.047 INFO:teuthology.orchestra.run.smithi156.stdout:(23/131): python3-rgw-18.1.3-11.g0484097b.el8.x 1.6 MB/s | 107 kB 00:00 2023-07-29T15:57:27.122 INFO:teuthology.orchestra.run.smithi156.stdout:(24/131): rbd-fuse-18.1.3-11.g0484097b.el8.x86_ 1.2 MB/s | 94 kB 00:00 2023-07-29T15:57:27.881 INFO:teuthology.orchestra.run.smithi039.stdout:(8/131): ceph-radosgw-18.1.3-11.g0484097b.el8.x 3.7 MB/s | 13 MB 00:03 2023-07-29T15:57:27.948 INFO:teuthology.orchestra.run.smithi039.stdout:(9/131): ceph-selinux-18.1.3-11.g0484097b.el8.x 365 kB/s | 24 kB 00:00 2023-07-29T15:57:27.973 INFO:teuthology.orchestra.run.smithi156.stdout:(25/131): librgw2-18.1.3-11.g0484097b.el8.x86_6 3.7 MB/s | 6.5 MB 00:01 2023-07-29T15:57:28.048 INFO:teuthology.orchestra.run.smithi156.stdout:(26/131): rbd-nbd-18.1.3-11.g0484097b.el8.x86_6 2.3 MB/s | 179 kB 00:00 2023-07-29T15:57:28.107 INFO:teuthology.orchestra.run.smithi156.stdout:(27/131): ceph-grafana-dashboards-18.1.3-11.g04 395 kB/s | 23 kB 00:00 2023-07-29T15:57:28.173 INFO:teuthology.orchestra.run.smithi156.stdout:(28/131): ceph-mgr-cephadm-18.1.3-11.g0484097b. 1.9 MB/s | 127 kB 00:00 2023-07-29T15:57:28.265 INFO:teuthology.orchestra.run.smithi156.stdout:(29/131): rbd-mirror-18.1.3-11.g0484097b.el8.x8 3.0 MB/s | 3.4 MB 00:01 2023-07-29T15:57:28.290 INFO:teuthology.orchestra.run.smithi039.stdout:(10/131): ceph-common-18.1.3-11.g0484097b.el8.x 2.7 MB/s | 22 MB 00:08 2023-07-29T15:57:28.432 INFO:teuthology.orchestra.run.smithi039.stdout:(11/131): cephfs-mirror-18.1.3-11.g0484097b.el8 1.8 MB/s | 257 kB 00:00 2023-07-29T15:57:28.507 INFO:teuthology.orchestra.run.smithi039.stdout:(12/131): libcephfs-devel-18.1.3-11.g0484097b.e 407 kB/s | 30 kB 00:00 2023-07-29T15:57:28.666 INFO:teuthology.orchestra.run.smithi156.stdout:(30/131): ceph-mgr-dashboard-18.1.3-11.g0484097 3.4 MB/s | 1.7 MB 00:00 2023-07-29T15:57:28.766 INFO:teuthology.orchestra.run.smithi156.stdout:(31/131): ceph-mgr-modules-core-18.1.3-11.g0484 2.3 MB/s | 240 kB 00:00 2023-07-29T15:57:28.817 INFO:teuthology.orchestra.run.smithi156.stdout:(32/131): ceph-mgr-rook-18.1.3-11.g0484097b.el8 950 kB/s | 47 kB 00:00 2023-07-29T15:57:28.841 INFO:teuthology.orchestra.run.smithi039.stdout:(13/131): ceph-osd-18.1.3-11.g0484097b.el8.x86_ 3.0 MB/s | 18 MB 00:05 2023-07-29T15:57:28.867 INFO:teuthology.orchestra.run.smithi156.stdout:(33/131): ceph-prometheus-alerts-18.1.3-11.g048 292 kB/s | 15 kB 00:00 2023-07-29T15:57:28.883 INFO:teuthology.orchestra.run.smithi039.stdout:(14/131): libcephfs2-18.1.3-11.g0484097b.el8.x8 2.0 MB/s | 769 kB 00:00 2023-07-29T15:57:28.958 INFO:teuthology.orchestra.run.smithi039.stdout:(15/131): librados-devel-18.1.3-11.g0484097b.el 1.8 MB/s | 139 kB 00:00 2023-07-29T15:57:28.983 INFO:teuthology.orchestra.run.smithi039.stdout:(16/131): libcephsqlite-18.1.3-11.g0484097b.el8 967 kB/s | 137 kB 00:00 2023-07-29T15:57:28.992 INFO:teuthology.orchestra.run.smithi156.stdout:(34/131): ceph-volume-18.1.3-11.g0484097b.el8.n 2.0 MB/s | 258 kB 00:00 2023-07-29T15:57:29.067 INFO:teuthology.orchestra.run.smithi156.stdout:(35/131): cephadm-18.1.3-11.g0484097b.el8.noarc 2.1 MB/s | 160 kB 00:00 2023-07-29T15:57:29.142 INFO:teuthology.orchestra.run.smithi039.stdout:(17/131): libradosstriper1-18.1.3-11.g0484097b. 2.5 MB/s | 474 kB 00:00 2023-07-29T15:57:29.151 INFO:teuthology.orchestra.run.smithi156.stdout:(36/131): cephfs-top-18.1.3-11.g0484097b.el8.no 215 kB/s | 18 kB 00:00 2023-07-29T15:57:29.218 INFO:teuthology.orchestra.run.smithi039.stdout:(18/131): python3-ceph-argparse-18.1.3-11.g0484 596 kB/s | 45 kB 00:00 2023-07-29T15:57:29.292 INFO:teuthology.orchestra.run.smithi039.stdout:(19/131): python3-ceph-common-18.1.3-11.g048409 1.5 MB/s | 119 kB 00:00 2023-07-29T15:57:29.426 INFO:teuthology.orchestra.run.smithi039.stdout:(20/131): python3-cephfs-18.1.3-11.g0484097b.el 1.4 MB/s | 190 kB 00:00 2023-07-29T15:57:29.493 INFO:teuthology.orchestra.run.smithi156.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 897 kB/s | 306 kB 00:00 2023-07-29T15:57:29.576 INFO:teuthology.orchestra.run.smithi156.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 707 kB/s | 59 kB 00:00 2023-07-29T15:57:29.609 INFO:teuthology.orchestra.run.smithi039.stdout:(21/131): python3-rados-18.1.3-11.g0484097b.el8 2.0 MB/s | 371 kB 00:00 2023-07-29T15:57:29.644 INFO:teuthology.orchestra.run.smithi156.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-07-29T15:57:29.761 INFO:teuthology.orchestra.run.smithi156.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.0 MB/s | 476 kB 00:00 2023-07-29T15:57:29.768 INFO:teuthology.orchestra.run.smithi039.stdout:(22/131): python3-rbd-18.1.3-11.g0484097b.el8.x 2.2 MB/s | 364 kB 00:00 2023-07-29T15:57:29.819 INFO:teuthology.orchestra.run.smithi156.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 749 kB/s | 44 kB 00:00 2023-07-29T15:57:29.886 INFO:teuthology.orchestra.run.smithi156.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-07-29T15:57:29.945 INFO:teuthology.orchestra.run.smithi156.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 566 kB/s | 33 kB 00:00 2023-07-29T15:57:30.012 INFO:teuthology.orchestra.run.smithi156.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 244 kB/s | 16 kB 00:00 2023-07-29T15:57:30.030 INFO:teuthology.orchestra.run.smithi039.stdout:(23/131): python3-rgw-18.1.3-11.g0484097b.el8.x 407 kB/s | 107 kB 00:00 2023-07-29T15:57:30.079 INFO:teuthology.orchestra.run.smithi156.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2023-07-29T15:57:30.162 INFO:teuthology.orchestra.run.smithi156.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2023-07-29T15:57:30.211 INFO:teuthology.orchestra.run.smithi039.stdout:(24/131): rbd-fuse-18.1.3-11.g0484097b.el8.x86_ 523 kB/s | 94 kB 00:00 2023-07-29T15:57:30.263 INFO:teuthology.orchestra.run.smithi156.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 813 kB/s | 81 kB 00:00 2023-07-29T15:57:30.338 INFO:teuthology.orchestra.run.smithi156.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 148 kB/s | 11 kB 00:00 2023-07-29T15:57:30.413 INFO:teuthology.orchestra.run.smithi156.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2023-07-29T15:57:30.547 INFO:teuthology.orchestra.run.smithi156.stdout:(50/131): ceph-mgr-diskprediction-local-18.1.3- 3.2 MB/s | 7.4 MB 00:02 2023-07-29T15:57:30.589 INFO:teuthology.orchestra.run.smithi156.stdout:(51/131): python3-kubernetes-11.0.0-6.el8.noarc 6.7 MB/s | 1.2 MB 00:00 2023-07-29T15:57:30.681 INFO:teuthology.orchestra.run.smithi156.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 640 kB/s | 59 kB 00:00 2023-07-29T15:57:30.748 INFO:teuthology.orchestra.run.smithi156.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 903 kB/s | 60 kB 00:00 2023-07-29T15:57:30.790 INFO:teuthology.orchestra.run.smithi156.stdout:(54/131): python3-logutils-0.3.5-11.el8.noarch. 201 kB/s | 49 kB 00:00 2023-07-29T15:57:30.840 INFO:teuthology.orchestra.run.smithi156.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-07-29T15:57:30.874 INFO:teuthology.orchestra.run.smithi156.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-07-29T15:57:30.894 INFO:teuthology.orchestra.run.smithi039.stdout:(25/131): librgw2-18.1.3-11.g0484097b.el8.x86_6 3.4 MB/s | 6.5 MB 00:01 2023-07-29T15:57:30.978 INFO:teuthology.orchestra.run.smithi039.stdout:(26/131): rbd-nbd-18.1.3-11.g0484097b.el8.x86_6 2.1 MB/s | 179 kB 00:00 2023-07-29T15:57:31.049 INFO:teuthology.orchestra.run.smithi156.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 161 kB/s | 34 kB 00:00 2023-07-29T15:57:31.061 INFO:teuthology.orchestra.run.smithi039.stdout:(27/131): ceph-grafana-dashboards-18.1.3-11.g04 277 kB/s | 23 kB 00:00 2023-07-29T15:57:31.141 INFO:teuthology.orchestra.run.smithi156.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 735 kB/s | 196 kB 00:00 2023-07-29T15:57:31.145 INFO:teuthology.orchestra.run.smithi039.stdout:(28/131): ceph-mgr-cephadm-18.1.3-11.g0484097b. 1.5 MB/s | 127 kB 00:00 2023-07-29T15:57:31.167 INFO:teuthology.orchestra.run.smithi156.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 549 kB/s | 64 kB 00:00 2023-07-29T15:57:31.234 INFO:teuthology.orchestra.run.smithi156.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 203 kB/s | 19 kB 00:00 2023-07-29T15:57:31.259 INFO:teuthology.orchestra.run.smithi156.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 260 kB/s | 24 kB 00:00 2023-07-29T15:57:31.326 INFO:teuthology.orchestra.run.smithi156.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 318 kB/s | 29 kB 00:00 2023-07-29T15:57:31.351 INFO:teuthology.orchestra.run.smithi156.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 299 kB/s | 27 kB 00:00 2023-07-29T15:57:31.451 INFO:teuthology.orchestra.run.smithi156.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.5 MB/s | 251 kB 00:00 2023-07-29T15:57:31.521 INFO:teuthology.orchestra.run.smithi039.stdout:(29/131): rbd-mirror-18.1.3-11.g0484097b.el8.x8 2.6 MB/s | 3.4 MB 00:01 2023-07-29T15:57:31.535 INFO:teuthology.orchestra.run.smithi156.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.2 MB/s | 255 kB 00:00 2023-07-29T15:57:31.560 INFO:teuthology.orchestra.run.smithi156.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 564 kB/s | 61 kB 00:00 2023-07-29T15:57:31.611 INFO:teuthology.orchestra.run.smithi156.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-07-29T15:57:31.636 INFO:teuthology.orchestra.run.smithi156.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 305 kB/s | 23 kB 00:00 2023-07-29T15:57:31.692 INFO:teuthology.orchestra.run.smithi156.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 70 kB 00:00 2023-07-29T15:57:31.704 INFO:teuthology.orchestra.run.smithi039.stdout:(30/131): ceph-mgr-dashboard-18.1.3-11.g0484097 3.0 MB/s | 1.7 MB 00:00 2023-07-29T15:57:31.753 INFO:teuthology.orchestra.run.smithi156.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 775 kB/s | 47 kB 00:00 2023-07-29T15:57:31.795 INFO:teuthology.orchestra.run.smithi156.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-29T15:57:31.837 INFO:teuthology.orchestra.run.smithi156.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-29T15:57:31.863 INFO:teuthology.orchestra.run.smithi039.stdout:(31/131): ceph-mgr-modules-core-18.1.3-11.g0484 1.5 MB/s | 240 kB 00:00 2023-07-29T15:57:31.879 INFO:teuthology.orchestra.run.smithi156.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 6.5 MB/s | 1.7 MB 00:00 2023-07-29T15:57:31.904 INFO:teuthology.orchestra.run.smithi156.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2023-07-29T15:57:31.938 INFO:teuthology.orchestra.run.smithi039.stdout:(32/131): ceph-mgr-rook-18.1.3-11.g0484097b.el8 633 kB/s | 47 kB 00:00 2023-07-29T15:57:31.947 INFO:teuthology.orchestra.run.smithi156.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 720 kB/s | 48 kB 00:00 2023-07-29T15:57:31.972 INFO:teuthology.orchestra.run.smithi156.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.4 MB/s | 87 kB 00:00 2023-07-29T15:57:31.988 INFO:teuthology.orchestra.run.smithi156.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-07-29T15:57:32.013 INFO:teuthology.orchestra.run.smithi039.stdout:(33/131): ceph-prometheus-alerts-18.1.3-11.g048 194 kB/s | 15 kB 00:00 2023-07-29T15:57:32.014 INFO:teuthology.orchestra.run.smithi156.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.5 MB/s | 109 kB 00:00 2023-07-29T15:57:32.039 INFO:teuthology.orchestra.run.smithi156.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-07-29T15:57:32.081 INFO:teuthology.orchestra.run.smithi156.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-07-29T15:57:32.098 INFO:teuthology.orchestra.run.smithi156.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 319 kB/s | 19 kB 00:00 2023-07-29T15:57:32.132 INFO:teuthology.orchestra.run.smithi156.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-07-29T15:57:32.165 INFO:teuthology.orchestra.run.smithi156.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 8.2 MB/s | 559 kB 00:00 2023-07-29T15:57:32.172 INFO:teuthology.orchestra.run.smithi039.stdout:(34/131): ceph-volume-18.1.3-11.g0484097b.el8.n 1.6 MB/s | 258 kB 00:00 2023-07-29T15:57:32.192 INFO:teuthology.orchestra.run.smithi156.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 304 kB/s | 18 kB 00:00 2023-07-29T15:57:32.233 INFO:teuthology.orchestra.run.smithi156.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 8.8 MB/s | 605 kB 00:00 2023-07-29T15:57:32.258 INFO:teuthology.orchestra.run.smithi156.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-07-29T15:57:32.272 INFO:teuthology.orchestra.run.smithi039.stdout:(35/131): cephadm-18.1.3-11.g0484097b.el8.noarc 1.6 MB/s | 160 kB 00:00 2023-07-29T15:57:32.300 INFO:teuthology.orchestra.run.smithi156.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-29T15:57:32.350 INFO:teuthology.orchestra.run.smithi156.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 20 MB/s | 2.3 MB 00:00 2023-07-29T15:57:32.364 INFO:teuthology.orchestra.run.smithi039.stdout:(36/131): cephfs-top-18.1.3-11.g0484097b.el8.no 196 kB/s | 18 kB 00:00 2023-07-29T15:57:32.375 INFO:teuthology.orchestra.run.smithi156.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-07-29T15:57:32.401 INFO:teuthology.orchestra.run.smithi156.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-07-29T15:57:32.426 INFO:teuthology.orchestra.run.smithi156.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.9 MB/s | 98 kB 00:00 2023-07-29T15:57:32.451 INFO:teuthology.orchestra.run.smithi156.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-07-29T15:57:32.476 INFO:teuthology.orchestra.run.smithi156.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 770 kB/s | 39 kB 00:00 2023-07-29T15:57:32.510 INFO:teuthology.orchestra.run.smithi156.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 11 MB/s | 644 kB 00:00 2023-07-29T15:57:32.536 INFO:teuthology.orchestra.run.smithi156.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.9 MB/s | 171 kB 00:00 2023-07-29T15:57:32.553 INFO:teuthology.orchestra.run.smithi156.stdout:(96/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-29T15:57:32.595 INFO:teuthology.orchestra.run.smithi156.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-29T15:57:32.628 INFO:teuthology.orchestra.run.smithi156.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-07-29T15:57:32.670 INFO:teuthology.orchestra.run.smithi156.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-29T15:57:32.712 INFO:teuthology.orchestra.run.smithi156.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-29T15:57:32.746 INFO:teuthology.orchestra.run.smithi156.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 1.1 MB/s | 39 kB 00:00 2023-07-29T15:57:32.796 INFO:teuthology.orchestra.run.smithi156.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-07-29T15:57:32.838 INFO:teuthology.orchestra.run.smithi156.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 13 MB/s | 538 kB 00:00 2023-07-29T15:57:32.904 INFO:teuthology.orchestra.run.smithi156.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-07-29T15:57:32.947 INFO:teuthology.orchestra.run.smithi156.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.3 MB/s | 187 kB 00:00 2023-07-29T15:57:33.081 INFO:teuthology.orchestra.run.smithi156.stdout:(106/131): python3-scipy-1.0.0-21.module+el8.5. 25 MB/s | 14 MB 00:00 2023-07-29T15:57:33.114 INFO:teuthology.orchestra.run.smithi156.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 4.2 MB/s | 713 kB 00:00 2023-07-29T15:57:33.148 INFO:teuthology.orchestra.run.smithi156.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 796 kB/s | 27 kB 00:00 2023-07-29T15:57:33.190 INFO:teuthology.orchestra.run.smithi156.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-29T15:57:33.223 INFO:teuthology.orchestra.run.smithi156.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-07-29T15:57:33.290 INFO:teuthology.orchestra.run.smithi156.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 23 MB/s | 4.8 MB 00:00 2023-07-29T15:57:33.306 INFO:teuthology.orchestra.run.smithi039.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 325 kB/s | 306 kB 00:00 2023-07-29T15:57:33.341 INFO:teuthology.orchestra.run.smithi156.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-29T15:57:33.383 INFO:teuthology.orchestra.run.smithi156.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-29T15:57:33.439 INFO:teuthology.orchestra.run.smithi039.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 442 kB/s | 59 kB 00:00 2023-07-29T15:57:33.443 INFO:teuthology.orchestra.run.smithi156.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2023-07-29T15:57:33.467 INFO:teuthology.orchestra.run.smithi156.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 764 kB/s | 64 kB 00:00 2023-07-29T15:57:33.492 INFO:teuthology.orchestra.run.smithi156.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-07-29T15:57:33.509 INFO:teuthology.orchestra.run.smithi156.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 477 kB/s | 20 kB 00:00 2023-07-29T15:57:33.534 INFO:teuthology.orchestra.run.smithi156.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2023-07-29T15:57:33.568 INFO:teuthology.orchestra.run.smithi156.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-29T15:57:33.573 INFO:teuthology.orchestra.run.smithi039.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 563 kB/s | 75 kB 00:00 2023-07-29T15:57:33.610 INFO:teuthology.orchestra.run.smithi156.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-07-29T15:57:33.702 INFO:teuthology.orchestra.run.smithi156.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 29 MB/s | 4.9 MB 00:00 2023-07-29T15:57:33.744 INFO:teuthology.orchestra.run.smithi156.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-07-29T15:57:33.819 INFO:teuthology.orchestra.run.smithi156.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-07-29T15:57:33.844 INFO:teuthology.orchestra.run.smithi156.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.4 MB/s | 354 kB 00:00 2023-07-29T15:57:33.903 INFO:teuthology.orchestra.run.smithi156.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-07-29T15:57:33.995 INFO:teuthology.orchestra.run.smithi156.stdout:(126/131): postgresql-server-10.21-2.module+el8 29 MB/s | 5.1 MB 00:00 2023-07-29T15:57:34.037 INFO:teuthology.orchestra.run.smithi156.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-29T15:57:34.157 INFO:teuthology.orchestra.run.smithi039.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 816 kB/s | 476 kB 00:00 2023-07-29T15:57:34.282 INFO:teuthology.orchestra.run.smithi039.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2023-07-29T15:57:34.566 INFO:teuthology.orchestra.run.smithi039.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 654 kB/s | 185 kB 00:00 2023-07-29T15:57:34.699 INFO:teuthology.orchestra.run.smithi039.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 247 kB/s | 33 kB 00:00 2023-07-29T15:57:34.825 INFO:teuthology.orchestra.run.smithi039.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2023-07-29T15:57:35.092 INFO:teuthology.orchestra.run.smithi039.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 647 kB/s | 173 kB 00:00 2023-07-29T15:57:35.634 INFO:teuthology.orchestra.run.smithi039.stdout:(46/131): ceph-mgr-diskprediction-local-18.1.3- 1.8 MB/s | 7.4 MB 00:04 2023-07-29T15:57:35.637 INFO:teuthology.orchestra.run.smithi156.stdout:(128/131): librbd1-18.1.3-11.g0484097b.el8.x86_ 2.2 MB/s | 3.6 MB 00:01 2023-07-29T15:57:35.695 INFO:teuthology.orchestra.run.smithi156.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-07-29T15:57:35.754 INFO:teuthology.orchestra.run.smithi156.stdout:(130/131): librados2-18.1.3-11.g0484097b.el8.x8 2.1 MB/s | 3.7 MB 00:01 2023-07-29T15:57:35.792 INFO:teuthology.orchestra.run.smithi039.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 549 kB/s | 384 kB 00:00 2023-07-29T15:57:35.934 INFO:teuthology.orchestra.run.smithi039.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 81 kB/s | 11 kB 00:00 2023-07-29T15:57:36.076 INFO:teuthology.orchestra.run.smithi039.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 125 kB/s | 18 kB 00:00 2023-07-29T15:57:36.301 INFO:teuthology.orchestra.run.smithi039.stdout:(50/131): python3-google-auth-1.1.1-10.el8.noar 122 kB/s | 81 kB 00:00 2023-07-29T15:57:36.427 INFO:teuthology.orchestra.run.smithi039.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 389 kB/s | 49 kB 00:00 2023-07-29T15:57:36.569 INFO:teuthology.orchestra.run.smithi039.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 415 kB/s | 59 kB 00:00 2023-07-29T15:57:36.844 INFO:teuthology.orchestra.run.smithi039.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 219 kB/s | 60 kB 00:00 2023-07-29T15:57:37.253 INFO:teuthology.orchestra.run.smithi039.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 1.0 MB/s | 1.2 MB 00:01 2023-07-29T15:57:37.311 INFO:teuthology.orchestra.run.smithi039.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 606 kB/s | 283 kB 00:00 2023-07-29T15:57:37.403 INFO:teuthology.orchestra.run.smithi039.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 104 kB/s | 16 kB 00:00 2023-07-29T15:57:37.462 INFO:teuthology.orchestra.run.smithi039.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 224 kB/s | 34 kB 00:00 2023-07-29T15:57:37.604 INFO:teuthology.orchestra.run.smithi039.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 453 kB/s | 64 kB 00:00 2023-07-29T15:57:37.937 INFO:teuthology.orchestra.run.smithi039.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 56 kB/s | 19 kB 00:00 2023-07-29T15:57:37.988 INFO:teuthology.orchestra.run.smithi039.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 336 kB/s | 196 kB 00:00 2023-07-29T15:57:38.159 INFO:teuthology.orchestra.run.smithi039.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 171 kB/s | 29 kB 00:00 2023-07-29T15:57:38.197 INFO:teuthology.orchestra.run.smithi039.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 92 kB/s | 24 kB 00:00 2023-07-29T15:57:38.288 INFO:teuthology.orchestra.run.smithi039.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 212 kB/s | 27 kB 00:00 2023-07-29T15:57:38.639 INFO:teuthology.orchestra.run.smithi039.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 718 kB/s | 251 kB 00:00 2023-07-29T15:57:38.747 INFO:teuthology.orchestra.run.smithi039.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 566 kB/s | 61 kB 00:00 2023-07-29T15:57:38.823 INFO:teuthology.orchestra.run.smithi039.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 408 kB/s | 255 kB 00:00 2023-07-29T15:57:38.864 INFO:teuthology.orchestra.run.smithi039.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 730 kB/s | 85 kB 00:00 2023-07-29T15:57:38.973 INFO:teuthology.orchestra.run.smithi039.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 153 kB/s | 23 kB 00:00 2023-07-29T15:57:38.999 INFO:teuthology.orchestra.run.smithi039.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2023-07-29T15:57:39.324 INFO:teuthology.orchestra.run.smithi039.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 42 MB/s | 14 MB 00:00 2023-07-29T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.9 MB/s | 110 kB 00:00 2023-07-29T15:57:39.416 INFO:teuthology.orchestra.run.smithi039.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-07-29T15:57:39.458 INFO:teuthology.orchestra.run.smithi039.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-29T15:57:39.500 INFO:teuthology.orchestra.run.smithi039.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-29T15:57:39.533 INFO:teuthology.orchestra.run.smithi039.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 1.1 MB/s | 39 kB 00:00 2023-07-29T15:57:39.592 INFO:teuthology.orchestra.run.smithi039.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2023-07-29T15:57:39.642 INFO:teuthology.orchestra.run.smithi039.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 11 MB/s | 538 kB 00:00 2023-07-29T15:57:39.684 INFO:teuthology.orchestra.run.smithi039.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-29T15:57:39.718 INFO:teuthology.orchestra.run.smithi039.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2023-07-29T15:57:39.802 INFO:teuthology.orchestra.run.smithi039.stdout:(80/131): automake-1.16.1-7.el8.noarch.rpm 8.3 MB/s | 713 kB 00:00 2023-07-29T15:57:39.936 INFO:teuthology.orchestra.run.smithi039.stdout:(81/131): python3-babel-2.5.1-7.el8.noarch.rpm 36 MB/s | 4.8 MB 00:00 2023-07-29T15:57:39.978 INFO:teuthology.orchestra.run.smithi039.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 638 kB/s | 27 kB 00:00 2023-07-29T15:57:40.020 INFO:teuthology.orchestra.run.smithi039.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-29T15:57:40.061 INFO:teuthology.orchestra.run.smithi039.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-07-29T15:57:40.203 INFO:teuthology.orchestra.run.smithi039.stdout:(85/131): python3-numpy-1.14.3-10.el8.x86_64.rp 26 MB/s | 3.7 MB 00:00 2023-07-29T15:57:40.245 INFO:teuthology.orchestra.run.smithi039.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-07-29T15:57:40.287 INFO:teuthology.orchestra.run.smithi039.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-07-29T15:57:40.329 INFO:teuthology.orchestra.run.smithi039.stdout:(88/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-07-29T15:57:40.371 INFO:teuthology.orchestra.run.smithi039.stdout:(89/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2023-07-29T15:57:40.405 INFO:teuthology.orchestra.run.smithi039.stdout:(90/131): python3-jsonpointer-1.10-11.el8.noarc 599 kB/s | 20 kB 00:00 2023-07-29T15:57:40.438 INFO:teuthology.orchestra.run.smithi039.stdout:(91/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 3.1 MB/s | 103 kB 00:00 2023-07-29T15:57:40.489 INFO:teuthology.orchestra.run.smithi039.stdout:(92/131): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2023-07-29T15:57:40.631 INFO:teuthology.orchestra.run.smithi039.stdout:(93/131): openblas-threads-0.3.15-3.el8.x86_64. 34 MB/s | 4.9 MB 00:00 2023-07-29T15:57:40.673 INFO:teuthology.orchestra.run.smithi039.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-07-29T15:57:40.765 INFO:teuthology.orchestra.run.smithi039.stdout:(95/131): thrift-0.13.0-2.el8.x86_64.rpm 935 kB/s | 1.7 MB 00:01 2023-07-29T15:57:40.848 INFO:teuthology.orchestra.run.smithi039.stdout:(96/131): openblas-0.3.15-3.el8.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-07-29T15:57:40.873 INFO:teuthology.orchestra.run.smithi039.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.8 MB/s | 202 kB 00:00 2023-07-29T15:57:40.899 INFO:teuthology.orchestra.run.smithi039.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-07-29T15:57:40.949 INFO:teuthology.orchestra.run.smithi039.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 16 MB/s | 808 kB 00:00 2023-07-29T15:57:41.024 INFO:teuthology.orchestra.run.smithi039.stdout:(100/131): postgresql-10.21-2.module+el8.6.0+15 20 MB/s | 1.5 MB 00:00 2023-07-29T15:57:41.141 INFO:teuthology.orchestra.run.smithi039.stdout:(101/131): postgresql-server-10.21-2.module+el8 19 MB/s | 5.1 MB 00:00 2023-07-29T15:57:41.158 INFO:teuthology.orchestra.run.smithi039.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 350 kB/s | 47 kB 00:00 2023-07-29T15:57:41.184 INFO:teuthology.orchestra.run.smithi039.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 4.7 MB/s | 200 kB 00:00 2023-07-29T15:57:41.201 INFO:teuthology.orchestra.run.smithi039.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-29T15:57:41.226 INFO:teuthology.orchestra.run.smithi039.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2023-07-29T15:57:41.243 INFO:teuthology.orchestra.run.smithi039.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-07-29T15:57:41.268 INFO:teuthology.orchestra.run.smithi039.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-29T15:57:41.293 INFO:teuthology.orchestra.run.smithi039.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 845 kB/s | 43 kB 00:00 2023-07-29T15:57:41.319 INFO:teuthology.orchestra.run.smithi039.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-07-29T15:57:41.344 INFO:teuthology.orchestra.run.smithi039.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-07-29T15:57:41.403 INFO:teuthology.orchestra.run.smithi039.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 6.5 MB/s | 560 kB 00:00 2023-07-29T15:57:41.428 INFO:teuthology.orchestra.run.smithi039.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 224 kB/s | 19 kB 00:00 2023-07-29T15:57:41.453 INFO:teuthology.orchestra.run.smithi039.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-07-29T15:57:41.488 INFO:teuthology.orchestra.run.smithi039.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-07-29T15:57:41.512 INFO:teuthology.orchestra.run.smithi039.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 304 kB/s | 18 kB 00:00 2023-07-29T15:57:41.546 INFO:teuthology.orchestra.run.smithi039.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-07-29T15:57:41.571 INFO:teuthology.orchestra.run.smithi039.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-07-29T15:57:41.605 INFO:teuthology.orchestra.run.smithi039.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 2.1 MB/s | 72 kB 00:00 2023-07-29T15:57:41.663 INFO:teuthology.orchestra.run.smithi039.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 20 MB/s | 2.3 MB 00:00 2023-07-29T15:57:41.690 INFO:teuthology.orchestra.run.smithi039.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 989 kB/s | 83 kB 00:00 2023-07-29T15:57:41.714 INFO:teuthology.orchestra.run.smithi039.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-07-29T15:57:41.739 INFO:teuthology.orchestra.run.smithi039.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-29T15:57:41.797 INFO:teuthology.orchestra.run.smithi039.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 693 kB/s | 58 kB 00:00 2023-07-29T15:57:41.823 INFO:teuthology.orchestra.run.smithi039.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 461 kB/s | 39 kB 00:00 2023-07-29T15:57:41.865 INFO:teuthology.orchestra.run.smithi039.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.3 MB/s | 644 kB 00:00 2023-07-29T15:57:41.890 INFO:teuthology.orchestra.run.smithi039.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.5 MB/s | 171 kB 00:00 2023-07-29T15:57:41.907 INFO:teuthology.orchestra.run.smithi039.stdout:(127/131): NetworkManager-initscripts-updown-1. 3.2 MB/s | 138 kB 00:00 2023-07-29T15:57:43.049 INFO:teuthology.orchestra.run.smithi039.stdout:(128/131): librados2-18.1.3-11.g0484097b.el8.x8 3.2 MB/s | 3.7 MB 00:01 2023-07-29T15:57:43.108 INFO:teuthology.orchestra.run.smithi039.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-07-29T15:57:44.008 INFO:teuthology.orchestra.run.smithi039.stdout:(130/131): librbd1-18.1.3-11.g0484097b.el8.x86_ 1.7 MB/s | 3.6 MB 00:02 2023-07-29T15:57:48.645 INFO:teuthology.orchestra.run.smithi156.stdout:(131/131): ceph-test-18.1.3-11.g0484097b.el8.x8 1.9 MB/s | 47 MB 00:24 2023-07-29T15:57:48.648 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2023-07-29T15:57:48.648 INFO:teuthology.orchestra.run.smithi156.stdout:Total 6.3 MB/s | 203 MB 00:32 2023-07-29T15:57:49.417 INFO:teuthology.orchestra.run.smithi039.stdout:(131/131): ceph-test-18.1.3-11.g0484097b.el8.x8 2.2 MB/s | 47 MB 00:21 2023-07-29T15:57:49.421 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-07-29T15:57:49.421 INFO:teuthology.orchestra.run.smithi039.stdout:Total 6.9 MB/s | 203 MB 00:29 2023-07-29T15:57:49.458 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2023-07-29T15:57:49.578 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2023-07-29T15:57:49.578 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2023-07-29T15:57:50.268 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-07-29T15:57:50.402 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-07-29T15:57:50.402 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-07-29T15:57:50.677 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2023-07-29T15:57:50.682 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2023-07-29T15:57:51.586 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-07-29T15:57:51.591 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-07-29T15:57:55.441 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2023-07-29T15:57:55.614 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-29T15:57:55.805 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-29T15:57:55.957 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-29T15:57:56.085 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-29T15:57:56.282 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-29T15:57:56.648 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-07-29T15:57:56.684 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-29T15:57:56.839 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-29T15:57:56.882 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:18.1.3-11.g0484097b.el8.x86_64 5/134 2023-07-29T15:57:57.022 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-29T15:57:57.275 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-29T15:57:57.288 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:18.1.3-11.g0484097b.el8.x86_64 5/134 2023-07-29T15:57:57.449 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T15:57:57.449 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-29T15:57:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-29T15:57:57.577 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T15:57:57.791 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T15:57:57.791 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-29T15:57:57.874 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T15:57:58.029 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T15:57:58.191 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-29T15:57:58.211 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-29T15:57:58.394 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-29T15:57:58.466 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T15:57:58.610 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-29T15:57:58.896 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T15:57:58.996 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-29T15:57:59.091 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T15:57:59.219 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-29T15:57:59.219 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T15:57:59.357 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T15:57:59.416 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T15:57:59.447 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T15:57:59.515 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T15:57:59.598 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 15/134 2023-07-29T15:57:59.742 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 9/134 2023-07-29T15:57:59.763 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 15/134 2023-07-29T15:57:59.960 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-29T15:57:59.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-29T15:58:00.146 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-29T15:58:00.220 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-29T15:58:00.322 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-29T15:58:00.465 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-29T15:58:00.583 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-29T15:58:00.639 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-29T15:58:00.740 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-29T15:58:00.878 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-29T15:58:01.016 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T15:58:01.112 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T15:58:01.274 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 15/134 2023-07-29T15:58:01.353 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-29T15:58:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 15/134 2023-07-29T15:58:01.498 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-29T15:58:01.507 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-29T15:58:01.629 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-29T15:58:01.799 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-29T15:58:02.105 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-29T15:58:02.281 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-29T15:58:02.454 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-29T15:58:02.649 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-29T15:58:02.990 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-29T15:58:03.230 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-29T15:58:03.285 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-29T15:58:03.462 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-29T15:58:03.474 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-29T15:58:03.732 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-29T15:58:03.896 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:18.1.3-11.g0484097b.el8.x8 26/134 2023-07-29T15:58:04.025 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:18.1.3-11.g0484097b.el8. 27/134 2023-07-29T15:58:04.232 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-29T15:58:04.235 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 28/134 2023-07-29T15:58:04.439 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-29T15:58:04.514 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-29T15:58:04.756 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-29T15:58:05.011 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-29T15:58:05.205 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-29T15:58:05.459 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-29T15:58:05.686 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-29T15:58:05.905 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:18.1.3-11.g0484097b.el8.x8 26/134 2023-07-29T15:58:06.044 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:18.1.3-11.g0484097b.el8. 27/134 2023-07-29T15:58:06.199 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-29T15:58:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 28/134 2023-07-29T15:58:07.218 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-29T15:58:07.346 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-29T15:58:07.520 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-29T15:58:07.629 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-29T15:58:07.766 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-29T15:58:07.927 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-29T15:58:07.996 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-29T15:58:08.123 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-29T15:58:08.233 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-07-29T15:58:08.424 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-29T15:58:08.577 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 36/134 2023-07-29T15:58:08.721 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 37/134 2023-07-29T15:58:08.854 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-29T15:58:08.990 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-29T15:58:09.147 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-29T15:58:09.225 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-29T15:58:09.434 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-29T15:58:09.650 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-29T15:58:09.814 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-29T15:58:09.894 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-29T15:58:10.013 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-29T15:58:10.092 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-29T15:58:10.251 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-29T15:58:10.368 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-29T15:58:10.412 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-29T15:58:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-07-29T15:58:10.632 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-29T15:58:10.791 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-29T15:58:10.902 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-29T15:58:10.933 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-29T15:58:11.112 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 36/134 2023-07-29T15:58:11.119 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-29T15:58:11.269 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 37/134 2023-07-29T15:58:11.317 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-29T15:58:11.464 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-29T15:58:11.650 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-29T15:58:11.672 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-29T15:58:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-29T15:58:11.990 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-29T15:58:12.011 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-29T15:58:12.208 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-29T15:58:12.233 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-29T15:58:12.417 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-29T15:58:12.458 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-29T15:58:12.876 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-29T15:58:12.878 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-29T15:58:13.213 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-29T15:58:13.261 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-29T15:58:13.413 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-29T15:58:13.493 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-29T15:58:13.598 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-29T15:58:13.700 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-29T15:58:13.826 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-29T15:58:13.983 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-29T15:58:13.988 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-29T15:58:14.166 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-29T15:58:14.325 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-29T15:58:14.469 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-29T15:58:14.547 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-29T15:58:14.727 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-29T15:58:14.765 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-29T15:58:14.967 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-29T15:58:14.995 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-29T15:58:15.189 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-29T15:58:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-29T15:58:15.348 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-29T15:58:15.394 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-29T15:58:15.548 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-29T15:58:15.738 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-29T15:58:15.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-29T15:58:16.030 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-29T15:58:16.069 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-29T15:58:16.217 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-29T15:58:16.273 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-29T15:58:16.356 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-29T15:58:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-29T15:58:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-29T15:58:16.730 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-29T15:58:16.782 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-29T15:58:16.916 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-29T15:58:16.969 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-29T15:58:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-29T15:58:17.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-29T15:58:17.214 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-29T15:58:17.285 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-29T15:58:17.478 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-29T15:58:17.526 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-29T15:58:17.735 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-29T15:58:17.758 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-29T15:58:17.888 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-29T15:58:17.960 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-29T15:58:17.979 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-29T15:58:18.219 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-29T15:58:18.219 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-29T15:58:18.424 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-29T15:58:18.437 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-29T15:58:18.589 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-29T15:58:18.621 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-29T15:58:18.821 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-29T15:58:18.831 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-29T15:58:19.111 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-29T15:58:19.363 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-29T15:58:19.536 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-29T15:58:19.734 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-29T15:58:19.926 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-29T15:58:19.927 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-29T15:58:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-29T15:58:20.224 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-29T15:58:20.337 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-29T15:58:20.503 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-29T15:58:20.567 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-29T15:58:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-29T15:58:20.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-29T15:58:20.998 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-29T15:58:21.058 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-29T15:58:21.246 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-29T15:58:21.268 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-29T15:58:21.461 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-29T15:58:21.785 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-29T15:58:22.016 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-29T15:58:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-29T15:58:22.203 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:18.1.3-11.g0484097b.el8.x86_64 92/134 2023-07-29T15:58:22.229 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-29T15:58:22.284 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:18.1.3-11.g0484097b.el8.x86_64 92/134 2023-07-29T15:58:22.471 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 93/134 2023-07-29T15:58:22.507 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-29T15:58:23.722 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-29T15:58:24.019 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-29T15:58:24.281 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-29T15:58:24.639 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-29T15:58:24.919 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-29T15:58:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-29T15:58:25.953 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-29T15:58:26.029 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T15:58:26.154 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:18.1.3-11.g0484097b.el8.x86_64 92/134 2023-07-29T15:58:26.235 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:18.1.3-11.g0484097b.el8.x86_64 92/134 2023-07-29T15:58:26.237 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T15:58:26.421 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 93/134 2023-07-29T15:58:26.818 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T15:58:27.110 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 95/134 2023-07-29T15:58:27.195 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 95/134 2023-07-29T15:58:27.328 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 96/134 2023-07-29T15:58:30.352 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T15:58:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T15:58:31.216 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T15:58:31.515 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 95/134 2023-07-29T15:58:31.587 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 95/134 2023-07-29T15:58:31.737 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 96/134 2023-07-29T15:58:46.317 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 96/134 2023-07-29T15:58:46.318 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2023-07-29T15:58:46.318 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2023-07-29T15:58:46.318 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-07-29T15:58:46.318 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /run 2023-07-29T15:58:46.318 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2023-07-29T15:58:46.318 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2023-07-29T15:58:46.319 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2023-07-29T15:58:46.319 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2023-07-29T15:58:46.319 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /dev 2023-07-29T15:58:46.319 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:58:46.658 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 97/134 2023-07-29T15:58:48.237 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 97/134 2023-07-29T15:58:48.432 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 98/134 2023-07-29T15:58:48.650 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 98/134 2023-07-29T15:58:48.842 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 99/134 2023-07-29T15:58:49.296 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 99/134 2023-07-29T15:58:49.450 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 100/134 2023-07-29T15:58:49.467 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 100/134 2023-07-29T15:58:51.097 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el8.noarch 101/134 2023-07-29T15:58:51.276 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:18.1.3-11.g0484097b.el8.noarch 101/134 2023-07-29T15:58:51.354 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 96/134 2023-07-29T15:58:51.355 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2023-07-29T15:58:51.355 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2023-07-29T15:58:51.355 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-07-29T15:58:51.355 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /run 2023-07-29T15:58:51.356 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2023-07-29T15:58:51.356 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2023-07-29T15:58:51.356 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2023-07-29T15:58:51.356 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2023-07-29T15:58:51.356 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-07-29T15:58:51.356 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:58:51.414 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8 102/134 2023-07-29T15:58:51.622 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el 103/134 2023-07-29T15:58:51.726 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 97/134 2023-07-29T15:58:51.808 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 104/134 2023-07-29T15:58:52.019 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 104/134 2023-07-29T15:58:52.298 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 105/134 2023-07-29T15:58:53.257 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 105/134 2023-07-29T15:58:53.413 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 97/134 2023-07-29T15:58:53.424 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 106/134 2023-07-29T15:58:53.515 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 106/134 2023-07-29T15:58:53.621 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 98/134 2023-07-29T15:58:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 98/134 2023-07-29T15:58:53.867 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8. 107/134 2023-07-29T15:58:54.059 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 99/134 2023-07-29T15:58:54.065 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 108/134 2023-07-29T15:58:54.100 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 108/134 2023-07-29T15:58:54.241 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 109/134 2023-07-29T15:58:54.251 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 109/134 2023-07-29T15:58:54.550 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 99/134 2023-07-29T15:58:54.734 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 100/134 2023-07-29T15:58:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 100/134 2023-07-29T15:58:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:18.1.3-11.g0484097b.el8.x86_64 110/134 2023-07-29T15:58:55.710 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 111/134 2023-07-29T15:58:55.744 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 111/134 2023-07-29T15:58:55.882 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:18.1.3-11.g0484097 112/134 2023-07-29T15:58:55.924 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.3-11.g0484097 112/134 2023-07-29T15:58:56.083 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 113/134 2023-07-29T15:58:56.429 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 113/134 2023-07-29T15:58:56.545 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el8.noarch 101/134 2023-07-29T15:58:56.590 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 114/134 2023-07-29T15:58:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:18.1.3-11.g0484097b.el8.noarch 101/134 2023-07-29T15:58:56.882 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8 102/134 2023-07-29T15:58:57.083 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el 103/134 2023-07-29T15:58:57.275 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 104/134 2023-07-29T15:58:57.494 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 104/134 2023-07-29T15:58:57.811 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 105/134 2023-07-29T15:58:58.835 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 105/134 2023-07-29T15:58:59.016 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 106/134 2023-07-29T15:58:59.113 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 106/134 2023-07-29T15:58:59.515 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8. 107/134 2023-07-29T15:58:59.787 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 108/134 2023-07-29T15:58:59.824 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 108/134 2023-07-29T15:58:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 109/134 2023-07-29T15:59:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 109/134 2023-07-29T15:59:01.021 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 114/134 2023-07-29T15:59:01.350 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:18.1.3-11.g0484097b.el8.x86_64 110/134 2023-07-29T15:59:01.540 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 111/134 2023-07-29T15:59:01.575 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 111/134 2023-07-29T15:59:01.766 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:18.1.3-11.g0484097 112/134 2023-07-29T15:59:01.810 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.3-11.g0484097 112/134 2023-07-29T15:59:01.872 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 115/134 2023-07-29T15:59:01.966 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 113/134 2023-07-29T15:59:02.171 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-29T15:59:02.331 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 113/134 2023-07-29T15:59:02.342 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 117/134 2023-07-29T15:59:02.524 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 114/134 2023-07-29T15:59:02.585 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-29T15:59:02.748 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-29T15:59:02.938 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-29T15:59:03.169 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-29T15:59:03.343 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 121/134 2023-07-29T15:59:03.482 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-29T15:59:05.577 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T15:59:05.814 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T15:59:05.832 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T15:59:05.990 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 124/134 2023-07-29T15:59:06.272 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 125/134 2023-07-29T15:59:06.598 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 126/134 2023-07-29T15:59:06.984 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-29T15:59:07.197 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-29T15:59:07.237 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 114/134 2023-07-29T15:59:07.276 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-29T15:59:07.472 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-29T15:59:07.484 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-29T15:59:07.553 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 115/134 2023-07-29T15:59:07.630 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-29T15:59:07.638 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-29T15:59:07.748 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-29T15:59:07.766 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-29T15:59:07.766 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T15:59:07.960 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-29T15:59:07.967 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T15:59:07.967 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-29T15:59:08.186 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-29T15:59:08.396 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T15:59:08.902 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T15:59:08.903 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:18.1.3-11.g0484097b.el8.x86_64 1/134 2023-07-29T15:59:08.903 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 2/134 2023-07-29T15:59:08.903 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 3/134 2023-07-29T15:59:08.904 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 4/134 2023-07-29T15:59:08.904 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:18.1.3-11.g0484097 5/134 2023-07-29T15:59:08.904 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T15:59:08.904 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T15:59:08.904 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T15:59:08.904 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 9/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 10/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 11/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 12/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 13/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 15/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 16/134 2023-07-29T15:59:08.905 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 17/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 18/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:18.1.3-11.g0484097b.el8.x86_64 19/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:18.1.3-11.g0484097b.el8. 20/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:18.1.3-11.g0484097b.el8.x8 21/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 22/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 23/134 2023-07-29T15:59:08.906 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 24/134 2023-07-29T15:59:08.907 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 25/134 2023-07-29T15:59:08.907 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 26/134 2023-07-29T15:59:08.907 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 27/134 2023-07-29T15:59:08.907 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 28/134 2023-07-29T15:59:08.907 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el 29/134 2023-07-29T15:59:08.908 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 30/134 2023-07-29T15:59:08.914 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 31/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 32/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8. 33/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 34/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8 35/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 36/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:18.1.3-11.g0484097b.el8.noarch 37/134 2023-07-29T15:59:08.915 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 38/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-29T15:59:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-29T15:59:08.917 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-29T15:59:08.917 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-29T15:59:08.917 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-29T15:59:08.917 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-29T15:59:08.917 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-29T15:59:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-29T15:59:08.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-29T15:59:08.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-29T15:59:08.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-07-29T15:59:08.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-29T15:59:08.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-29T15:59:08.919 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-29T15:59:08.920 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-29T15:59:08.921 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-29T15:59:08.922 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-29T15:59:08.922 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-29T15:59:08.922 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-29T15:59:08.922 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-29T15:59:08.922 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-29T15:59:08.923 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-29T15:59:08.924 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-29T15:59:08.924 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-29T15:59:08.924 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-29T15:59:08.924 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-29T15:59:08.924 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-29T15:59:08.924 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-29T15:59:08.925 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-29T15:59:08.926 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-29T15:59:08.927 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-29T15:59:08.928 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-29T15:59:08.928 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-29T15:59:08.928 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-29T15:59:08.928 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-29T15:59:08.928 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:18.1.3-11.g0484097b.el8.x86_64 129/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-29T15:59:08.929 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:18.1.3-11.g0484097b.el8.x86_64 131/134 2023-07-29T15:59:08.930 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T15:59:08.930 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-29T15:59:08.930 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-29T15:59:09.259 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-29T15:59:09.460 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 119/134 2023-07-29T15:59:09.628 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-29T15:59:09.853 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 121/134 2023-07-29T15:59:10.008 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-29T15:59:11.340 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T15:59:11.341 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:02:07 ago on Sat 29 Jul 2023 03:57:04 PM UTC. 2023-07-29T15:59:12.313 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T15:59:12.598 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T15:59:12.616 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T15:59:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 124/134 2023-07-29T15:59:13.082 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 125/134 2023-07-29T15:59:13.402 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 126/134 2023-07-29T15:59:13.683 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-29T15:59:13.799 INFO:teuthology.orchestra.run.smithi156.stdout:Installed products updated. 2023-07-29T15:59:13.848 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-29T15:59:13.856 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-29T15:59:14.017 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:59:14.017 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2023-07-29T15:59:14.017 INFO:teuthology.orchestra.run.smithi156.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-29T15:59:14.017 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.017 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.1-7.el8.noarch 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.0.4-10.el8.x86_64 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.018 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.019 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.020 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.1-9.el8.x86_64 2023-07-29T15:59:14.021 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-29T15:59:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-29T15:59:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-3.el8.x86_64 2023-07-29T15:59:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-29T15:59:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-29T15:59:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-29T15:59:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: libpq-13.5-1.el8.x86_64 2023-07-29T15:59:14.023 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-29T15:59:14.024 INFO:teuthology.orchestra.run.smithi156.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-29T15:59:14.025 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-29T15:59:14.025 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-29T15:59:14.025 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-29T15:59:14.026 INFO:teuthology.orchestra.run.smithi156.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-29T15:59:14.026 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-29T15:59:14.026 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-29T15:59:14.026 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-29T15:59:14.026 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-29T15:59:14.026 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-29T15:59:14.027 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-29T15:59:14.028 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-29T15:59:14.028 INFO:teuthology.orchestra.run.smithi156.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-29T15:59:14.028 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-29T15:59:14.028 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.028 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.029 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.029 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-29T15:59:14.029 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-29T15:59:14.030 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-29T15:59:14.030 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-29T15:59:14.030 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-29T15:59:14.030 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-29T15:59:14.031 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-29T15:59:14.031 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-29T15:59:14.031 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-29T15:59:14.031 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-29T15:59:14.032 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-29T15:59:14.032 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-29T15:59:14.032 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-29T15:59:14.033 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-29T15:59:14.034 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-29T15:59:14.034 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-29T15:59:14.034 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-29T15:59:14.035 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-29T15:59:14.035 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-29T15:59:14.036 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-29T15:59:14.036 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-29T15:59:14.037 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-29T15:59:14.037 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-2.6-1.el8.noarch 2023-07-29T15:59:14.037 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-29T15:59:14.037 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-29T15:59:14.038 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-29T15:59:14.038 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-29T15:59:14.038 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-29T15:59:14.039 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-29T15:59:14.039 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.039 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.039 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-29T15:59:14.039 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el8.noarch 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-29T15:59:14.040 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-29T15:59:14.041 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-29T15:59:14.042 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-29T15:59:14.043 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-29T15:59:14.043 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-29T15:59:14.043 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T15:59:14.043 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2023-07-29T15:59:14.220 DEBUG:teuthology.parallel:result is None 2023-07-29T15:59:14.247 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-29T15:59:14.334 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-29T15:59:14.514 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-29T15:59:14.527 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-29T15:59:14.527 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T15:59:14.742 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T15:59:14.743 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-29T15:59:15.047 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-29T15:59:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T15:59:15.770 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T15:59:15.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:18.1.3-11.g0484097b.el8.x86_64 1/134 2023-07-29T15:59:15.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 2/134 2023-07-29T15:59:15.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 3/134 2023-07-29T15:59:15.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 4/134 2023-07-29T15:59:15.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:18.1.3-11.g0484097 5/134 2023-07-29T15:59:15.771 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 9/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 10/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 11/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 12/134 2023-07-29T15:59:15.772 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 13/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 15/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 16/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 17/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 18/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:18.1.3-11.g0484097b.el8.x86_64 19/134 2023-07-29T15:59:15.773 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:18.1.3-11.g0484097b.el8. 20/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:18.1.3-11.g0484097b.el8.x8 21/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 22/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 23/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 24/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 25/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 26/134 2023-07-29T15:59:15.774 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 27/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 28/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el 29/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 30/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 31/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 32/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8. 33/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 34/134 2023-07-29T15:59:15.775 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8 35/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 36/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:18.1.3-11.g0484097b.el8.noarch 37/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 38/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-29T15:59:15.776 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-29T15:59:15.777 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-29T15:59:15.777 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-29T15:59:15.777 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-29T15:59:15.777 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-29T15:59:15.777 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-29T15:59:15.777 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-29T15:59:15.778 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-29T15:59:15.779 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-29T15:59:15.780 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-29T15:59:15.781 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-29T15:59:15.782 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-29T15:59:15.783 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-29T15:59:15.784 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-29T15:59:15.785 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-29T15:59:15.785 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-29T15:59:15.785 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-29T15:59:15.785 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-29T15:59:15.785 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-29T15:59:15.785 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-29T15:59:15.786 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-29T15:59:15.787 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:18.1.3-11.g0484097b.el8.x86_64 129/134 2023-07-29T15:59:15.788 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-29T15:59:15.789 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:18.1.3-11.g0484097b.el8.x86_64 131/134 2023-07-29T15:59:15.789 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T15:59:15.789 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-29T15:59:18.333 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T15:59:18.333 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:11 ago on Sat 29 Jul 2023 03:57:07 PM UTC. 2023-07-29T15:59:20.936 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-07-29T15:59:21.173 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:59:21.173 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.1-7.el8.noarch 2023-07-29T15:59:21.174 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.0.4-10.el8.x86_64 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.175 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.176 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.177 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.1-9.el8.x86_64 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.6-3.el8.x86_64 2023-07-29T15:59:21.178 INFO:teuthology.orchestra.run.smithi039.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-29T15:59:21.179 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-29T15:59:21.180 INFO:teuthology.orchestra.run.smithi039.stdout: libpq-13.5-1.el8.x86_64 2023-07-29T15:59:21.180 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-29T15:59:21.180 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-29T15:59:21.181 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-29T15:59:21.182 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T15:59:21.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.184 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-29T15:59:21.185 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-29T15:59:21.186 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-2.6-1.el8.noarch 2023-07-29T15:59:21.187 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.188 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el8.noarch 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-29T15:59:21.189 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-29T15:59:21.190 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-29T15:59:21.191 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-29T15:59:21.192 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-29T15:59:21.192 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-07-29T15:59:21.192 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-07-29T15:59:21.363 DEBUG:teuthology.parallel:result is None 2023-07-29T16:01:27.184 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 58 kB/s | 16 MB 04:36 2023-07-29T16:01:31.450 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 363 kB/s | 24 kB 00:00 2023-07-29T16:01:32.569 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 47 MB 00:01 2023-07-29T16:01:41.595 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 53 MB 00:01 2023-07-29T16:01:53.981 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-29T16:01:53.982 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-29T16:01:53.998 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-29T16:01:53.999 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-29T16:01:54.000 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-29T16:01:54.001 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-29T16:01:54.004 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-29T16:01:54.007 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-29T16:01:54.009 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-29T16:01:54.158 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-07-29T16:01:54.172 INFO:teuthology.orchestra.run.smithi031.stdout:=========================================================================================================================== 2023-07-29T16:01:54.172 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout:=========================================================================================================================== 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.1.3-11.g0484097b.el8 ceph 6.4 k 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.1.3-11.g0484097b.el8 ceph 5.4 M 2023-07-29T16:01:54.173 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:18.1.3-11.g0484097b.el8 ceph 847 k 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.1.3-11.g0484097b.el8 ceph 165 k 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.1.3-11.g0484097b.el8 ceph 1.5 M 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 127 k 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 1.7 M 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 7.4 M 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 47 k 2023-07-29T16:01:54.174 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.1.3-11.g0484097b.el8 ceph 13 M 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.1.3-11.g0484097b.el8 ceph 47 M 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 258 k 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 160 k 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:18.1.3-11.g0484097b.el8 ceph 257 k 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 18 k 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-29T16:01:54.175 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.1.3-11.g0484097b.el8 ceph 30 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 769 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.1.3-11.g0484097b.el8 ceph 139 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-29T16:01:54.176 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.1.3-11.g0484097b.el8 ceph 190 k 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.1.3-11.g0484097b.el8 ceph 371 k 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.1.3-11.g0484097b.el8 ceph 364 k 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.1.3-11.g0484097b.el8 ceph 107 k 2023-07-29T16:01:54.177 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.1.3-11.g0484097b.el8 ceph 94 k 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.4 M 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.1.3-11.g0484097b.el8 ceph 179 k 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.7 M 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.1.3-11.g0484097b.el8 ceph 3.6 M 2023-07-29T16:01:54.178 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.1.3-11.g0484097b.el8 ceph 22 M 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 23 k 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.1.3-11.g0484097b.el8 ceph 2.3 M 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 240 k 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.1.3-11.g0484097b.el8 ceph 4.6 M 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.1.3-11.g0484097b.el8 ceph 18 M 2023-07-29T16:01:54.179 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.1.3-11.g0484097b.el8 ceph-noarch 15 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.1.3-11.g0484097b.el8 ceph 24 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.1.3-11.g0484097b.el8 ceph 137 k 2023-07-29T16:01:54.180 INFO:teuthology.orchestra.run.smithi031.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.1.3-11.g0484097b.el8 ceph 474 k 2023-07-29T16:01:54.181 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.1.3-11.g0484097b.el8 ceph 6.5 M 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-29T16:01:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-29T16:01:54.183 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:18.1.3-11.g0484097b.el8 ceph 45 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.1.3-11.g0484097b.el8 ceph 119 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-29T16:01:54.184 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-29T16:01:54.185 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-29T16:01:54.186 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-29T16:01:54.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-29T16:01:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-29T16:01:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-29T16:01:54.188 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-29T16:01:54.192 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-29T16:01:54.192 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-29T16:01:54.193 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-29T16:01:54.194 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-29T16:01:54.194 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-29T16:01:54.195 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-29T16:01:54.195 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-29T16:01:54.195 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-29T16:01:54.195 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-29T16:01:54.196 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-29T16:01:54.196 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-29T16:01:54.196 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-29T16:01:54.196 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-29T16:01:54.200 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-29T16:01:54.201 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling module streams: 2023-07-29T16:01:54.201 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql 10 2023-07-29T16:01:54.201 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:01:54.201 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-07-29T16:01:54.202 INFO:teuthology.orchestra.run.smithi031.stdout:=========================================================================================================================== 2023-07-29T16:01:54.202 INFO:teuthology.orchestra.run.smithi031.stdout:Install 128 Packages 2023-07-29T16:01:54.202 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 3 Packages 2023-07-29T16:01:54.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:01:54.203 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 203 M 2023-07-29T16:01:54.203 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-07-29T16:01:54.639 INFO:teuthology.orchestra.run.smithi031.stdout:(1/131): ceph-18.1.3-11.g0484097b.el8.x86_64.rp 15 kB/s | 6.4 kB 00:00 2023-07-29T16:01:55.546 INFO:teuthology.orchestra.run.smithi031.stdout:(2/131): ceph-fuse-18.1.3-11.g0484097b.el8.x86_ 933 kB/s | 847 kB 00:00 2023-07-29T16:01:55.672 INFO:teuthology.orchestra.run.smithi031.stdout:(3/131): ceph-immutable-object-cache-18.1.3-11. 1.3 MB/s | 165 kB 00:00 2023-07-29T16:01:56.789 INFO:teuthology.orchestra.run.smithi031.stdout:(4/131): ceph-base-18.1.3-11.g0484097b.el8.x86_ 2.1 MB/s | 5.4 MB 00:02 2023-07-29T16:01:56.839 INFO:teuthology.orchestra.run.smithi031.stdout:(5/131): ceph-mds-18.1.3-11.g0484097b.el8.x86_6 2.0 MB/s | 2.3 MB 00:01 2023-07-29T16:01:57.222 INFO:teuthology.orchestra.run.smithi031.stdout:(6/131): ceph-mgr-18.1.3-11.g0484097b.el8.x86_6 3.6 MB/s | 1.5 MB 00:00 2023-07-29T16:01:58.156 INFO:teuthology.orchestra.run.smithi031.stdout:(7/131): ceph-mon-18.1.3-11.g0484097b.el8.x86_6 3.5 MB/s | 4.6 MB 00:01 2023-07-29T16:01:59.681 INFO:teuthology.orchestra.run.smithi031.stdout:(8/131): ceph-osd-18.1.3-11.g0484097b.el8.x86_6 7.2 MB/s | 18 MB 00:02 2023-07-29T16:01:59.831 INFO:teuthology.orchestra.run.smithi031.stdout:(9/131): ceph-selinux-18.1.3-11.g0484097b.el8.x 162 kB/s | 24 kB 00:00 2023-07-29T16:02:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:(10/131): ceph-radosgw-18.1.3-11.g0484097b.el8. 6.3 MB/s | 13 MB 00:02 2023-07-29T16:02:00.390 INFO:teuthology.orchestra.run.smithi031.stdout:(11/131): cephfs-mirror-18.1.3-11.g0484097b.el8 1.6 MB/s | 257 kB 00:00 2023-07-29T16:02:00.558 INFO:teuthology.orchestra.run.smithi031.stdout:(12/131): libcephfs-devel-18.1.3-11.g0484097b.e 183 kB/s | 30 kB 00:00 2023-07-29T16:02:00.740 INFO:teuthology.orchestra.run.smithi031.stdout:(13/131): libcephfs2-18.1.3-11.g0484097b.el8.x8 4.1 MB/s | 769 kB 00:00 2023-07-29T16:02:00.907 INFO:teuthology.orchestra.run.smithi031.stdout:(14/131): libcephsqlite-18.1.3-11.g0484097b.el8 823 kB/s | 137 kB 00:00 2023-07-29T16:02:01.066 INFO:teuthology.orchestra.run.smithi031.stdout:(15/131): librados-devel-18.1.3-11.g0484097b.el 879 kB/s | 139 kB 00:00 2023-07-29T16:02:01.225 INFO:teuthology.orchestra.run.smithi031.stdout:(16/131): libradosstriper1-18.1.3-11.g0484097b. 2.9 MB/s | 474 kB 00:00 2023-07-29T16:02:01.783 INFO:teuthology.orchestra.run.smithi031.stdout:(17/131): ceph-common-18.1.3-11.g0484097b.el8.x 2.9 MB/s | 22 MB 00:07 2023-07-29T16:02:01.959 INFO:teuthology.orchestra.run.smithi031.stdout:(18/131): python3-ceph-argparse-18.1.3-11.g0484 255 kB/s | 45 kB 00:00 2023-07-29T16:02:02.150 INFO:teuthology.orchestra.run.smithi031.stdout:(19/131): python3-ceph-common-18.1.3-11.g048409 620 kB/s | 119 kB 00:00 2023-07-29T16:02:02.251 INFO:teuthology.orchestra.run.smithi031.stdout:(20/131): librgw2-18.1.3-11.g0484097b.el8.x86_6 6.4 MB/s | 6.5 MB 00:01 2023-07-29T16:02:02.310 INFO:teuthology.orchestra.run.smithi031.stdout:(21/131): python3-cephfs-18.1.3-11.g0484097b.el 1.2 MB/s | 190 kB 00:00 2023-07-29T16:02:02.427 INFO:teuthology.orchestra.run.smithi031.stdout:(22/131): python3-rados-18.1.3-11.g0484097b.el8 2.1 MB/s | 371 kB 00:00 2023-07-29T16:02:02.602 INFO:teuthology.orchestra.run.smithi031.stdout:(23/131): python3-rgw-18.1.3-11.g0484097b.el8.x 610 kB/s | 107 kB 00:00 2023-07-29T16:02:02.635 INFO:teuthology.orchestra.run.smithi031.stdout:(24/131): python3-rbd-18.1.3-11.g0484097b.el8.x 1.1 MB/s | 364 kB 00:00 2023-07-29T16:02:02.769 INFO:teuthology.orchestra.run.smithi031.stdout:(25/131): rbd-fuse-18.1.3-11.g0484097b.el8.x86_ 566 kB/s | 94 kB 00:00 2023-07-29T16:02:02.936 INFO:teuthology.orchestra.run.smithi031.stdout:(26/131): rbd-nbd-18.1.3-11.g0484097b.el8.x86_6 1.0 MB/s | 179 kB 00:00 2023-07-29T16:02:03.128 INFO:teuthology.orchestra.run.smithi031.stdout:(27/131): ceph-grafana-dashboards-18.1.3-11.g04 120 kB/s | 23 kB 00:00 2023-07-29T16:02:03.270 INFO:teuthology.orchestra.run.smithi031.stdout:(28/131): ceph-mgr-cephadm-18.1.3-11.g0484097b. 900 kB/s | 127 kB 00:00 2023-07-29T16:02:03.587 INFO:teuthology.orchestra.run.smithi031.stdout:(29/131): ceph-mgr-dashboard-18.1.3-11.g0484097 5.4 MB/s | 1.7 MB 00:00 2023-07-29T16:02:03.754 INFO:teuthology.orchestra.run.smithi031.stdout:(30/131): rbd-mirror-18.1.3-11.g0484097b.el8.x8 3.1 MB/s | 3.4 MB 00:01 2023-07-29T16:02:03.938 INFO:teuthology.orchestra.run.smithi031.stdout:(31/131): ceph-mgr-modules-core-18.1.3-11.g0484 1.3 MB/s | 240 kB 00:00 2023-07-29T16:02:04.104 INFO:teuthology.orchestra.run.smithi031.stdout:(32/131): ceph-mgr-rook-18.1.3-11.g0484097b.el8 285 kB/s | 47 kB 00:00 2023-07-29T16:02:04.246 INFO:teuthology.orchestra.run.smithi031.stdout:(33/131): ceph-prometheus-alerts-18.1.3-11.g048 102 kB/s | 15 kB 00:00 2023-07-29T16:02:04.788 INFO:teuthology.orchestra.run.smithi031.stdout:(34/131): ceph-test-18.1.3-11.g0484097b.el8.x86 9.4 MB/s | 47 MB 00:04 2023-07-29T16:02:04.822 INFO:teuthology.orchestra.run.smithi031.stdout:(35/131): ceph-volume-18.1.3-11.g0484097b.el8.n 449 kB/s | 258 kB 00:00 2023-07-29T16:02:04.930 INFO:teuthology.orchestra.run.smithi031.stdout:(36/131): ceph-mgr-diskprediction-local-18.1.3- 5.5 MB/s | 7.4 MB 00:01 2023-07-29T16:02:04.947 INFO:teuthology.orchestra.run.smithi031.stdout:(37/131): cephfs-top-18.1.3-11.g0484097b.el8.no 143 kB/s | 18 kB 00:00 2023-07-29T16:02:05.064 INFO:teuthology.orchestra.run.smithi031.stdout:(38/131): cephadm-18.1.3-11.g0484097b.el8.noarc 581 kB/s | 160 kB 00:00 2023-07-29T16:02:05.698 INFO:teuthology.orchestra.run.smithi031.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 78 kB/s | 59 kB 00:00 2023-07-29T16:02:05.807 INFO:teuthology.orchestra.run.smithi031.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 350 kB/s | 306 kB 00:00 2023-07-29T16:02:05.957 INFO:teuthology.orchestra.run.smithi031.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 291 kB/s | 44 kB 00:00 2023-07-29T16:02:06.065 INFO:teuthology.orchestra.run.smithi031.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 76 kB/s | 75 kB 00:00 2023-07-29T16:02:06.107 INFO:teuthology.orchestra.run.smithi031.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-07-29T16:02:06.266 INFO:teuthology.orchestra.run.smithi031.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 103 kB/s | 16 kB 00:00 2023-07-29T16:02:06.349 INFO:teuthology.orchestra.run.smithi031.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 116 kB/s | 33 kB 00:00 2023-07-29T16:02:06.425 INFO:teuthology.orchestra.run.smithi031.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-07-29T16:02:06.575 INFO:teuthology.orchestra.run.smithi031.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 542 kB/s | 81 kB 00:00 2023-07-29T16:02:06.727 INFO:teuthology.orchestra.run.smithi031.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2023-07-29T16:02:06.875 INFO:teuthology.orchestra.run.smithi031.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 117 kB/s | 18 kB 00:00 2023-07-29T16:02:07.375 INFO:teuthology.orchestra.run.smithi031.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 2.3 MB/s | 1.2 MB 00:00 2023-07-29T16:02:07.509 INFO:teuthology.orchestra.run.smithi031.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 366 kB/s | 49 kB 00:00 2023-07-29T16:02:07.651 INFO:teuthology.orchestra.run.smithi031.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 415 kB/s | 59 kB 00:00 2023-07-29T16:02:07.802 INFO:teuthology.orchestra.run.smithi031.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 402 kB/s | 60 kB 00:00 2023-07-29T16:02:07.943 INFO:teuthology.orchestra.run.smithi031.stdout:(54/131): python3-asyncssh-2.7.0-2.el8.noarch.r 212 kB/s | 476 kB 00:02 2023-07-29T16:02:08.085 INFO:teuthology.orchestra.run.smithi031.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 998 kB/s | 283 kB 00:00 2023-07-29T16:02:08.435 INFO:teuthology.orchestra.run.smithi031.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 93 kB/s | 16 kB 00:00 2023-07-29T16:02:08.435 INFO:teuthology.orchestra.run.smithi031.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 192 kB/s | 34 kB 00:00 2023-07-29T16:02:08.470 INFO:teuthology.orchestra.run.smithi031.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 308 kB/s | 64 kB 00:00 2023-07-29T16:02:08.495 INFO:teuthology.orchestra.run.smithi031.stdout:(59/131): python3-cherrypy-18.4.0-1.el8.noarch. 179 kB/s | 384 kB 00:02 2023-07-29T16:02:08.620 INFO:teuthology.orchestra.run.smithi031.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-07-29T16:02:08.712 INFO:teuthology.orchestra.run.smithi031.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 319 kB/s | 29 kB 00:00 2023-07-29T16:02:08.771 INFO:teuthology.orchestra.run.smithi031.stdout:(62/131): python3-simplegeneric-0.8.1-17.el8.no 62 kB/s | 19 kB 00:00 2023-07-29T16:02:08.829 INFO:teuthology.orchestra.run.smithi031.stdout:(63/131): python3-routes-2.4.1-12.el8.noarch.rp 273 kB/s | 196 kB 00:00 2023-07-29T16:02:08.855 INFO:teuthology.orchestra.run.smithi031.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 194 kB/s | 27 kB 00:00 2023-07-29T16:02:08.971 INFO:teuthology.orchestra.run.smithi031.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.2 MB/s | 255 kB 00:00 2023-07-29T16:02:09.096 INFO:teuthology.orchestra.run.smithi031.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 686 kB/s | 85 kB 00:00 2023-07-29T16:02:09.122 INFO:teuthology.orchestra.run.smithi031.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 230 kB/s | 61 kB 00:00 2023-07-29T16:02:09.206 INFO:teuthology.orchestra.run.smithi031.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 211 kB/s | 23 kB 00:00 2023-07-29T16:02:09.297 INFO:teuthology.orchestra.run.smithi031.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 762 kB/s | 70 kB 00:00 2023-07-29T16:02:09.697 INFO:teuthology.orchestra.run.smithi031.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 34 MB/s | 14 MB 00:00 2023-07-29T16:02:09.956 INFO:teuthology.orchestra.run.smithi031.stdout:(71/131): python3-webob-1.8.5-1.el8.1.noarch.rp 228 kB/s | 251 kB 00:01 2023-07-29T16:02:09.973 INFO:teuthology.orchestra.run.smithi031.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 400 kB/s | 110 kB 00:00 2023-07-29T16:02:10.032 INFO:teuthology.orchestra.run.smithi031.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 723 kB/s | 54 kB 00:00 2023-07-29T16:02:10.074 INFO:teuthology.orchestra.run.smithi031.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.8 MB/s | 259 kB 00:00 2023-07-29T16:02:10.107 INFO:teuthology.orchestra.run.smithi031.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-07-29T16:02:10.133 INFO:teuthology.orchestra.run.smithi031.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 662 kB/s | 39 kB 00:00 2023-07-29T16:02:10.174 INFO:teuthology.orchestra.run.smithi031.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-07-29T16:02:10.208 INFO:teuthology.orchestra.run.smithi031.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.0 MB/s | 538 kB 00:00 2023-07-29T16:02:10.233 INFO:teuthology.orchestra.run.smithi031.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-07-29T16:02:10.259 INFO:teuthology.orchestra.run.smithi031.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-07-29T16:02:10.309 INFO:teuthology.orchestra.run.smithi031.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 9.3 MB/s | 713 kB 00:00 2023-07-29T16:02:10.359 INFO:teuthology.orchestra.run.smithi031.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 535 kB/s | 27 kB 00:00 2023-07-29T16:02:10.418 INFO:teuthology.orchestra.run.smithi031.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 5.4 MB/s | 320 kB 00:00 2023-07-29T16:02:10.501 INFO:teuthology.orchestra.run.smithi031.stdout:(84/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-29T16:02:10.526 INFO:teuthology.orchestra.run.smithi031.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 405 kB/s | 44 kB 00:00 2023-07-29T16:02:10.586 INFO:teuthology.orchestra.run.smithi031.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 7.6 MB/s | 457 kB 00:00 2023-07-29T16:02:10.661 INFO:teuthology.orchestra.run.smithi031.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 23 MB/s | 3.7 MB 00:00 2023-07-29T16:02:10.686 INFO:teuthology.orchestra.run.smithi031.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 2.2 MB/s | 225 kB 00:00 2023-07-29T16:02:10.712 INFO:teuthology.orchestra.run.smithi031.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-07-29T16:02:10.737 INFO:teuthology.orchestra.run.smithi031.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-07-29T16:02:10.762 INFO:teuthology.orchestra.run.smithi031.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 400 kB/s | 20 kB 00:00 2023-07-29T16:02:10.787 INFO:teuthology.orchestra.run.smithi031.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-07-29T16:02:10.829 INFO:teuthology.orchestra.run.smithi031.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-07-29T16:02:10.879 INFO:teuthology.orchestra.run.smithi031.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-07-29T16:02:10.997 INFO:teuthology.orchestra.run.smithi031.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 23 MB/s | 4.9 MB 00:00 2023-07-29T16:02:11.047 INFO:teuthology.orchestra.run.smithi031.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2023-07-29T16:02:11.130 INFO:teuthology.orchestra.run.smithi031.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2023-07-29T16:02:11.164 INFO:teuthology.orchestra.run.smithi031.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.0 MB/s | 354 kB 00:00 2023-07-29T16:02:11.239 INFO:teuthology.orchestra.run.smithi031.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 10 MB/s | 808 kB 00:00 2023-07-29T16:02:11.348 INFO:teuthology.orchestra.run.smithi031.stdout:(100/131): postgresql-server-10.21-2.module+el8 23 MB/s | 5.1 MB 00:00 2023-07-29T16:02:11.390 INFO:teuthology.orchestra.run.smithi031.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 10 MB/s | 1.5 MB 00:00 2023-07-29T16:02:11.407 INFO:teuthology.orchestra.run.smithi031.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 800 kB/s | 47 kB 00:00 2023-07-29T16:02:11.432 INFO:teuthology.orchestra.run.smithi031.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 4.7 MB/s | 200 kB 00:00 2023-07-29T16:02:11.449 INFO:teuthology.orchestra.run.smithi031.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-29T16:02:11.476 INFO:teuthology.orchestra.run.smithi031.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-29T16:02:11.499 INFO:teuthology.orchestra.run.smithi031.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 958 kB/s | 48 kB 00:00 2023-07-29T16:02:11.525 INFO:teuthology.orchestra.run.smithi031.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-07-29T16:02:11.550 INFO:teuthology.orchestra.run.smithi031.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 852 kB/s | 43 kB 00:00 2023-07-29T16:02:11.600 INFO:teuthology.orchestra.run.smithi031.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.4 MB/s | 109 kB 00:00 2023-07-29T16:02:11.625 INFO:teuthology.orchestra.run.smithi031.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2023-07-29T16:02:11.659 INFO:teuthology.orchestra.run.smithi031.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-07-29T16:02:11.684 INFO:teuthology.orchestra.run.smithi031.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 318 kB/s | 19 kB 00:00 2023-07-29T16:02:11.710 INFO:teuthology.orchestra.run.smithi031.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-07-29T16:02:11.743 INFO:teuthology.orchestra.run.smithi031.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.4 MB/s | 559 kB 00:00 2023-07-29T16:02:11.769 INFO:teuthology.orchestra.run.smithi031.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 305 kB/s | 18 kB 00:00 2023-07-29T16:02:11.811 INFO:teuthology.orchestra.run.smithi031.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 8.8 MB/s | 605 kB 00:00 2023-07-29T16:02:11.838 INFO:teuthology.orchestra.run.smithi031.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-07-29T16:02:11.878 INFO:teuthology.orchestra.run.smithi031.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.7 MB/s | 72 kB 00:00 2023-07-29T16:02:11.912 INFO:teuthology.orchestra.run.smithi031.stdout:(119/131): fuse-2.9.7-15.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-07-29T16:02:11.970 INFO:teuthology.orchestra.run.smithi031.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 15 MB/s | 2.3 MB 00:00 2023-07-29T16:02:11.997 INFO:teuthology.orchestra.run.smithi031.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 718 kB/s | 60 kB 00:00 2023-07-29T16:02:12.021 INFO:teuthology.orchestra.run.smithi031.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-29T16:02:12.046 INFO:teuthology.orchestra.run.smithi031.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-07-29T16:02:12.071 INFO:teuthology.orchestra.run.smithi031.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 768 kB/s | 39 kB 00:00 2023-07-29T16:02:12.108 INFO:teuthology.orchestra.run.smithi031.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-07-29T16:02:12.130 INFO:teuthology.orchestra.run.smithi031.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-07-29T16:02:12.156 INFO:teuthology.orchestra.run.smithi031.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-07-29T16:02:12.881 INFO:teuthology.orchestra.run.smithi031.stdout:(128/131): librados2-18.1.3-11.g0484097b.el8.x8 4.9 MB/s | 3.7 MB 00:00 2023-07-29T16:02:12.940 INFO:teuthology.orchestra.run.smithi031.stdout:(129/131): librbd1-18.1.3-11.g0484097b.el8.x86_ 4.5 MB/s | 3.6 MB 00:00 2023-07-29T16:02:12.981 INFO:teuthology.orchestra.run.smithi031.stdout:(130/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 7.8 MB/s | 798 kB 00:00 2023-07-29T16:02:13.048 INFO:teuthology.orchestra.run.smithi031.stdout:(131/131): thrift-0.13.0-2.el8.x86_64.rpm 452 kB/s | 1.7 MB 00:03 2023-07-29T16:02:13.052 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-07-29T16:02:13.053 INFO:teuthology.orchestra.run.smithi031.stdout:Total 11 MB/s | 203 MB 00:18 2023-07-29T16:02:14.734 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-07-29T16:02:14.882 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-07-29T16:02:14.882 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-07-29T16:02:16.452 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-07-29T16:02:16.456 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-07-29T16:02:21.568 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-07-29T16:02:21.748 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-29T16:02:21.921 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-29T16:02:22.145 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-29T16:02:22.299 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-29T16:02:22.468 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-29T16:02:22.667 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-29T16:02:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-29T16:02:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T16:02:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T16:02:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T16:02:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T16:02:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T16:02:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T16:02:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 9/134 2023-07-29T16:02:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-29T16:02:25.331 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-29T16:02:25.775 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-29T16:02:26.004 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-29T16:02:26.132 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T16:02:26.258 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T16:02:26.390 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 15/134 2023-07-29T16:02:26.487 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 15/134 2023-07-29T16:02:26.615 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-29T16:02:26.770 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-29T16:02:26.949 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-29T16:02:27.693 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-29T16:02:27.849 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-29T16:02:28.574 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-29T16:02:28.748 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-29T16:02:29.644 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-29T16:02:29.906 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-29T16:02:30.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-29T16:02:30.389 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-29T16:02:30.584 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-29T16:02:30.821 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-29T16:02:31.064 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-29T16:02:31.239 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.1.3-11.g0484097b.el8.x8 26/134 2023-07-29T16:02:31.386 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.1.3-11.g0484097b.el8. 27/134 2023-07-29T16:02:32.496 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 28/134 2023-07-29T16:02:32.856 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-29T16:02:33.127 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-29T16:02:33.396 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-29T16:02:33.597 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-29T16:02:35.843 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-29T16:02:36.166 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-29T16:02:36.396 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-07-29T16:02:36.598 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-29T16:02:36.797 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 36/134 2023-07-29T16:02:36.994 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 37/134 2023-07-29T16:02:37.150 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-29T16:02:37.326 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-29T16:02:37.525 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-29T16:02:37.679 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-29T16:02:37.916 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-29T16:02:38.132 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-29T16:02:38.551 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-29T16:02:38.831 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-29T16:02:39.038 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-29T16:02:39.272 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-29T16:02:39.543 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-29T16:02:39.877 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-29T16:02:40.074 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-29T16:02:40.251 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-29T16:02:40.469 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-29T16:02:40.656 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-29T16:02:40.854 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-29T16:02:41.234 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-29T16:02:41.536 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-29T16:02:41.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-29T16:02:42.013 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-29T16:02:42.181 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-29T16:02:42.401 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-29T16:02:42.580 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-29T16:02:42.672 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-29T16:02:42.936 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-29T16:02:43.153 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-29T16:02:43.304 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-29T16:02:43.409 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-29T16:02:43.626 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-29T16:02:43.783 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-29T16:02:43.965 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-29T16:02:44.140 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-29T16:02:44.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-29T16:02:44.630 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-29T16:02:44.776 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-29T16:02:44.918 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-29T16:02:45.113 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-29T16:02:45.322 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-29T16:02:45.547 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-29T16:02:45.758 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-29T16:02:45.904 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-29T16:02:46.144 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-29T16:02:46.379 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-29T16:02:46.570 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-29T16:02:46.880 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-29T16:02:47.114 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-29T16:02:47.283 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-29T16:02:47.510 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-29T16:02:48.937 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-29T16:02:49.227 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-29T16:02:49.478 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-29T16:02:49.856 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-29T16:02:50.166 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-29T16:02:50.375 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-29T16:02:51.242 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-29T16:02:51.394 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.1.3-11.g0484097b.el8.x86_64 92/134 2023-07-29T16:02:51.485 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.1.3-11.g0484097b.el8.x86_64 92/134 2023-07-29T16:02:51.696 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 93/134 2023-07-29T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T16:02:56.862 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T16:02:57.607 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 94/134 2023-07-29T16:02:57.927 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 95/134 2023-07-29T16:02:58.062 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 95/134 2023-07-29T16:02:58.210 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 96/134 2023-07-29T16:03:19.567 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 96/134 2023-07-29T16:03:19.567 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-07-29T16:03:19.567 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /run 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-07-29T16:03:19.568 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-07-29T16:03:19.569 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:03:19.966 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 97/134 2023-07-29T16:03:21.812 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 97/134 2023-07-29T16:03:22.015 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 98/134 2023-07-29T16:03:22.297 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 98/134 2023-07-29T16:03:22.466 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 99/134 2023-07-29T16:03:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 99/134 2023-07-29T16:03:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 100/134 2023-07-29T16:03:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 100/134 2023-07-29T16:03:26.024 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.1.3-11.g0484097b.el8.noarch 101/134 2023-07-29T16:03:26.184 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.1.3-11.g0484097b.el8.noarch 101/134 2023-07-29T16:03:26.298 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8 102/134 2023-07-29T16:03:26.487 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el 103/134 2023-07-29T16:03:26.681 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 104/134 2023-07-29T16:03:26.953 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 104/134 2023-07-29T16:03:27.238 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 105/134 2023-07-29T16:03:28.430 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 105/134 2023-07-29T16:03:28.617 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 106/134 2023-07-29T16:03:28.752 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 106/134 2023-07-29T16:03:29.148 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8. 107/134 2023-07-29T16:03:29.364 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 108/134 2023-07-29T16:03:29.431 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 108/134 2023-07-29T16:03:29.624 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 109/134 2023-07-29T16:03:29.652 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 109/134 2023-07-29T16:03:31.096 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.1.3-11.g0484097b.el8.x86_64 110/134 2023-07-29T16:03:31.298 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 111/134 2023-07-29T16:03:31.360 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 111/134 2023-07-29T16:03:31.506 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.1.3-11.g0484097 112/134 2023-07-29T16:03:31.579 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.1.3-11.g0484097 112/134 2023-07-29T16:03:31.747 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 113/134 2023-07-29T16:03:32.181 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 113/134 2023-07-29T16:03:32.331 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 114/134 2023-07-29T16:03:37.530 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 114/134 2023-07-29T16:03:37.835 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 115/134 2023-07-29T16:03:38.029 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-29T16:03:38.302 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-29T16:03:39.611 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-29T16:03:39.915 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-29T16:03:40.094 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 119/134 2023-07-29T16:03:40.276 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-29T16:03:40.485 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 121/134 2023-07-29T16:03:40.638 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-29T16:03:44.030 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T16:03:44.337 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T16:03:44.377 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-29T16:03:44.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 124/134 2023-07-29T16:03:44.756 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 125/134 2023-07-29T16:03:45.108 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 126/134 2023-07-29T16:03:45.380 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-29T16:03:45.553 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-29T16:03:45.577 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-29T16:03:45.818 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-29T16:03:45.978 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-29T16:03:46.103 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-29T16:03:46.311 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-29T16:03:46.340 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-29T16:03:46.340 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T16:03:46.543 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T16:03:46.543 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-29T16:03:46.785 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-29T16:03:46.996 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T16:03:47.279 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T16:03:47.280 INFO:teuthology.orchestra.run.smithi031.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-07-29T16:03:47.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:03:47.940 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.1.3-11.g0484097b.el8.x86_64 1/134 2023-07-29T16:03:47.941 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 2/134 2023-07-29T16:03:47.941 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 3/134 2023-07-29T16:03:47.941 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 4/134 2023-07-29T16:03:47.941 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.1.3-11.g0484097 5/134 2023-07-29T16:03:47.941 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 6/134 2023-07-29T16:03:47.941 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 7/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 8/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 9/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 10/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 11/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 12/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 13/134 2023-07-29T16:03:47.942 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 14/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 15/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 16/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 17/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_6 18/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.1.3-11.g0484097b.el8.x86_64 19/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.1.3-11.g0484097b.el8. 20/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.1.3-11.g0484097b.el8.x8 21/134 2023-07-29T16:03:47.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 22/134 2023-07-29T16:03:47.944 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 23/134 2023-07-29T16:03:47.944 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 24/134 2023-07-29T16:03:47.944 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 25/134 2023-07-29T16:03:47.944 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 26/134 2023-07-29T16:03:47.944 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 27/134 2023-07-29T16:03:47.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 28/134 2023-07-29T16:03:47.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el 29/134 2023-07-29T16:03:47.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarc 30/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noa 31/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.1.3-11.g04840 32/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8. 33/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 34/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8 35/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 36/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.1.3-11.g0484097b.el8.noarch 37/134 2023-07-29T16:03:47.953 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 38/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-29T16:03:47.954 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-29T16:03:47.955 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-29T16:03:47.956 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-29T16:03:47.957 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-29T16:03:47.958 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-29T16:03:47.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-29T16:03:47.960 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-29T16:03:47.960 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-29T16:03:47.960 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-29T16:03:47.960 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-29T16:03:47.961 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-29T16:03:47.961 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-29T16:03:47.962 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-29T16:03:47.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-29T16:03:47.964 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-29T16:03:47.965 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-29T16:03:47.966 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-29T16:03:47.967 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-29T16:03:47.967 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.1.3-11.g0484097b.el8.x86_64 129/134 2023-07-29T16:03:47.967 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-29T16:03:47.967 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.1.3-11.g0484097b.el8.x86_64 131/134 2023-07-29T16:03:47.968 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-29T16:03:47.968 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-29T16:03:50.608 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-29T16:03:50.608 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:09 ago on Sat 29 Jul 2023 04:01:41 PM UTC. 2023-07-29T16:03:53.452 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-07-29T16:03:53.695 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:03:53.695 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-07-29T16:03:53.695 INFO:teuthology.orchestra.run.smithi031.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-29T16:03:53.695 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.1-7.el8.noarch 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.0.4-10.el8.x86_64 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.696 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.697 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.698 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.699 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.699 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.699 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.699 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.699 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.699 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:18.1.3-11.g0484097b.el8.noarch 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.1-9.el8.x86_64 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-29T16:03:53.700 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-3.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-29T16:03:53.701 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-29T16:03:53.702 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-29T16:03:53.702 INFO:teuthology.orchestra.run.smithi031.stdout: libpq-13.5-1.el8.x86_64 2023-07-29T16:03:53.702 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-29T16:03:53.702 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-29T16:03:53.703 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-29T16:03:53.704 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-29T16:03:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-29T16:03:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-29T16:03:53.705 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-29T16:03:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-29T16:03:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-29T16:03:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-29T16:03:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-29T16:03:53.706 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-29T16:03:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-29T16:03:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-29T16:03:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-29T16:03:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-29T16:03:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-29T16:03:53.707 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T16:03:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T16:03:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-29T16:03:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-29T16:03:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-29T16:03:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-29T16:03:53.708 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-29T16:03:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-29T16:03:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-29T16:03:53.709 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-29T16:03:53.710 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-29T16:03:53.710 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-29T16:03:53.710 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-29T16:03:53.710 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-29T16:03:53.710 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-29T16:03:53.710 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-29T16:03:53.711 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-29T16:03:53.711 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-29T16:03:53.711 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-29T16:03:53.711 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-29T16:03:53.711 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-29T16:03:53.712 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-29T16:03:53.712 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-29T16:03:53.712 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-29T16:03:53.713 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-29T16:03:53.713 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-29T16:03:53.713 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-29T16:03:53.713 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-29T16:03:53.714 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-29T16:03:53.714 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-29T16:03:53.714 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-2.6-1.el8.noarch 2023-07-29T16:03:53.714 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-29T16:03:53.714 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-29T16:03:53.715 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-29T16:03:53.715 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-29T16:03:53.715 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-29T16:03:53.715 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-29T16:03:53.716 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.716 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.716 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-29T16:03:53.716 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-29T16:03:53.716 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.717 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-29T16:03:53.717 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el8.noarch 2023-07-29T16:03:53.717 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-29T16:03:53.717 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-29T16:03:53.718 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-29T16:03:53.718 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-29T16:03:53.718 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-29T16:03:53.718 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-29T16:03:53.718 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-29T16:03:53.719 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-29T16:03:53.719 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-29T16:03:53.719 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-29T16:03:53.719 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-29T16:03:53.719 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.720 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.720 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.1.3-11.g0484097b.el8.x86_64 2023-07-29T16:03:53.720 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-29T16:03:53.720 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-29T16:03:53.721 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-29T16:03:53.721 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-29T16:03:53.721 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-29T16:03:53.722 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-29T16:03:53.722 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:03:53.722 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-07-29T16:03:53.998 DEBUG:teuthology.parallel:result is None 2023-07-29T16:03:53.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:03:54.240 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-29T16:03:54.301 INFO:teuthology.orchestra.run.smithi031.stdout:18.1.3-11.g0484097b.el8 2023-07-29T16:03:54.301 INFO:teuthology.packaging:The installed version of ceph is 18.1.3-11.g0484097b.el8 2023-07-29T16:03:54.302 INFO:teuthology.task.install:The correct ceph version 18.1.3-11.g0484097b is installed. 2023-07-29T16:03:54.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:03:54.557 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-29T16:03:54.587 INFO:teuthology.orchestra.run.smithi039.stdout:18.1.3-11.g0484097b.el8 2023-07-29T16:03:54.588 INFO:teuthology.packaging:The installed version of ceph is 18.1.3-11.g0484097b.el8 2023-07-29T16:03:54.589 INFO:teuthology.task.install:The correct ceph version 18.1.3-11.g0484097b is installed. 2023-07-29T16:03:54.589 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:03:54.892 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-29T16:03:54.922 INFO:teuthology.orchestra.run.smithi156.stdout:18.1.3-11.g0484097b.el8 2023-07-29T16:03:54.923 INFO:teuthology.packaging:The installed version of ceph is 18.1.3-11.g0484097b.el8 2023-07-29T16:03:54.923 INFO:teuthology.task.install:The correct ceph version 18.1.3-11.g0484097b is installed. 2023-07-29T16:03:54.923 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-29T16:03:54.924 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:03:54.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-29T16:03:55.021 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:03:55.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-29T16:03:55.071 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:03:55.072 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-29T16:03:55.119 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-29T16:03:55.120 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:03:55.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-07-29T16:03:55.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-29T16:03:55.302 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:03:55.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2023-07-29T16:03:55.336 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-29T16:03:55.409 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:03:55.411 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2023-07-29T16:03:55.445 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-29T16:03:55.515 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-29T16:03:55.516 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:03:55.516 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-29T16:03:55.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-29T16:03:55.693 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:03:55.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-29T16:03:55.726 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-29T16:03:55.799 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:03:55.800 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-29T16:03:55.833 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-29T16:03:55.904 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-29T16:03:55.905 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:03:55.905 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-07-29T16:03:55.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-29T16:03:56.083 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:03:56.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2023-07-29T16:03:56.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-29T16:03:56.189 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:03:56.189 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2023-07-29T16:03:56.222 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-29T16:03:56.295 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-29T16:03:56.393 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-29T16:03:56.393 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:03:56.404 INFO:tasks.cephadm:Cluster fsid is 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:03:56.405 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-29T16:03:56.406 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.39', 'mon.c': '172.21.15.156'} 2023-07-29T16:03:56.406 INFO:tasks.cephadm:First mon is mon.a on smithi031 2023-07-29T16:03:56.407 INFO:tasks.cephadm:First mgr is x 2023-07-29T16:03:56.407 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-29T16:03:56.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2023-07-29T16:03:56.482 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname $(hostname -s) 2023-07-29T16:03:56.516 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2023-07-29T16:03:56.550 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-29T16:03:56.550 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:03:56.879 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71755/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.1.3-11-g0484097b', 'node_name': '172.21.4.66+confusa04', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.1.3-11.g0484097b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-28 18:59:48.218207', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'reef-release', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '0484097b0572e96f42cf6402cbe0ac8dcb046577', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71755/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.1.3-11-g0484097b', 'node_name': '172.21.5.31+adami01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.1.3-11.g0484097b'}, 'url': 'https://3.chacra.ceph.com/r/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-28 17:52:43.029547', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'reef-release', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2023-07-29T16:03:57.220 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef-release, sha1 0484097b0572e96f42cf6402cbe0ac8dcb046577 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:03:57.221 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/x86_64/flavors/default/cephadm 2023-07-29T16:03:57.221 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:57.873 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648889 Jul 29 16:03 /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:57.873 DEBUG:teuthology.orchestra.run.smithi039:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:58.474 INFO:teuthology.orchestra.run.smithi039.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648889 Jul 29 16:03 /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:58.475 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef-release/0484097b0572e96f42cf6402cbe0ac8dcb046577/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:59.107 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-rw-r--. 1 ubuntu ubuntu 648889 Jul 29 16:03 /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:59.107 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:59.162 DEBUG:teuthology.orchestra.run.smithi039:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:59.184 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-29T16:03:59.241 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 on all hosts... 2023-07-29T16:03:59.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 pull 2023-07-29T16:03:59.247 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 pull 2023-07-29T16:03:59.250 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 pull 2023-07-29T16:04:00.460 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577... 2023-07-29T16:04:00.775 INFO:teuthology.orchestra.run.smithi039.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577... 2023-07-29T16:04:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577... 2023-07-29T16:04:33.255 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-07-29T16:04:33.255 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)", 2023-07-29T16:04:33.256 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "b662b4d80c863e5542d125e87b927ab2388ab446e399f162e66c45ecfa3d181c", 2023-07-29T16:04:33.256 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2023-07-29T16:04:33.256 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dec2dadc08a2f5d927a831d583c48fe98cee7ad6a4f1b5f6644d3d31ec97903f" 2023-07-29T16:04:33.256 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2023-07-29T16:04:33.256 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-07-29T16:04:33.910 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-07-29T16:04:33.910 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)", 2023-07-29T16:04:33.911 INFO:teuthology.orchestra.run.smithi039.stdout: "image_id": "b662b4d80c863e5542d125e87b927ab2388ab446e399f162e66c45ecfa3d181c", 2023-07-29T16:04:33.911 INFO:teuthology.orchestra.run.smithi039.stdout: "repo_digests": [ 2023-07-29T16:04:33.911 INFO:teuthology.orchestra.run.smithi039.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dec2dadc08a2f5d927a831d583c48fe98cee7ad6a4f1b5f6644d3d31ec97903f" 2023-07-29T16:04:33.911 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2023-07-29T16:04:33.911 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-07-29T16:04:36.253 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:04:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc)", 2023-07-29T16:04:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "b662b4d80c863e5542d125e87b927ab2388ab446e399f162e66c45ecfa3d181c", 2023-07-29T16:04:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2023-07-29T16:04:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dec2dadc08a2f5d927a831d583c48fe98cee7ad6a4f1b5f6644d3d31ec97903f" 2023-07-29T16:04:36.254 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:04:36.254 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:04:36.279 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2023-07-29T16:04:36.352 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph 2023-07-29T16:04:36.391 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2023-07-29T16:04:36.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2023-07-29T16:04:36.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /etc/ceph 2023-07-29T16:04:36.529 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2023-07-29T16:04:36.560 INFO:tasks.cephadm:Writing seed config... 2023-07-29T16:04:36.562 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-29T16:04:36.563 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-29T16:04:36.563 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-29T16:04:36.563 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-29T16:04:36.563 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-29T16:04:36.563 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-29T16:04:36.564 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-29T16:04:36.564 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-29T16:04:36.564 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-29T16:04:36.564 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-29T16:04:36.565 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-29T16:04:36.565 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-29T16:04:36.565 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-29T16:04:36.565 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-29T16:04:36.565 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-29T16:04:36.565 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-29T16:04:36.566 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-29T16:04:36.566 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-29T16:04:36.566 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-29T16:04:36.566 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-29T16:04:36.566 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-29T16:04:36.567 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-29T16:04:36.567 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-29T16:04:36.567 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-29T16:04:36.567 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-29T16:04:36.567 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-29T16:04:36.568 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-29T16:04:36.568 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-07-29T16:04:36.568 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-07-29T16:04:36.568 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-07-29T16:04:36.569 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-29T16:04:36.569 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-29T16:04:36.569 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-07-29T16:04:36.569 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-07-29T16:04:36.569 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-29T16:04:36.570 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-29T16:04:36.570 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-07-29T16:04:36.570 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-29T16:04:36.570 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-29T16:04:36.570 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-29T16:04:36.571 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-29T16:04:36.571 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-29T16:04:36.571 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-29T16:04:36.571 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-29T16:04:36.572 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-29T16:04:36.573 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:04:36.573 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-29T16:04:36.615 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 858a7894-2e29-11ee-9b36-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-29T16:04:36.616 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a.service 2023-07-29T16:04:36.661 DEBUG:teuthology.orchestra.run.smithi031:mgr.x> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x.service 2023-07-29T16:04:36.708 INFO:tasks.cephadm:Bootstrapping... 2023-07-29T16:04:36.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 -v bootstrap --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:04:36.729 INFO:journalctl@ceph.mon.a.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:04:36.777 INFO:journalctl@ceph.mgr.x.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:04:36.933 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-07-29T16:04:36.934 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577', '-v', 'bootstrap', '--fsid', '858a7894-2e29-11ee-9b36-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2023-07-29T16:04:37.107 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.1.1 2023-07-29T16:04:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-29T16:04:37.108 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2023-07-29T16:04:37.279 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.1.1 2023-07-29T16:04:37.279 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2023-07-29T16:04:37.279 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2023-07-29T16:04:37.294 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-29T16:04:37.294 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-29T16:04:37.307 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-29T16:04:37.307 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-07-29T16:04:37.320 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2023-07-29T16:04:37.334 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2023-07-29T16:04:37.334 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2023-07-29T16:04:37.334 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2023-07-29T16:04:37.504 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.1.1 2023-07-29T16:04:37.505 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-29T16:04:37.505 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2023-07-29T16:04:37.505 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2023-07-29T16:04:37.518 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-29T16:04:37.519 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-29T16:04:37.531 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-29T16:04:37.532 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-07-29T16:04:37.545 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2023-07-29T16:04:37.558 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2023-07-29T16:04:37.559 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2023-07-29T16:04:37.559 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2023-07-29T16:04:37.559 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:37.559 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 139846282284672 on /run/cephadm/858a7894-2e29-11ee-9b36-001a4aab830c.lock 2023-07-29T16:04:37.560 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 139846282284672 acquired on /run/cephadm/858a7894-2e29-11ee-9b36-001a4aab830c.lock 2023-07-29T16:04:37.560 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2023-07-29T16:04:37.560 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2023-07-29T16:04:37.560 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2023-07-29T16:04:37.568 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-07-29T16:04:37.568 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2023-07-29T16:04:37.575 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-29T16:04:37.575 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-07-29T16:04:37.575 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1774sec hoplimit 64 pref medium 2023-07-29T16:04:37.581 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-29T16:04:37.582 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-29T16:04:37.582 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-29T16:04:37.582 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-07-29T16:04:37.582 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2023-07-29T16:04:37.582 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-29T16:04:37.583 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2023-07-29T16:04:37.584 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2023-07-29T16:04:37.584 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-29T16:04:37.584 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-29T16:04:37.586 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577... 2023-07-29T16:04:38.059 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout b662b4d80c863e5542d125e87b927ab2388ab446e399f162e66c45ecfa3d181c 2023-07-29T16:04:38.060 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577... 2023-07-29T16:04:38.060 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2023-07-29T16:04:38.060 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:627172e6abcbc400141664c55d2e89df2a5a8509ccbdfb49953e7179f9bbb120 2023-07-29T16:04:38.061 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-29T16:04:38.061 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:b662b4d80c863e5542d125e87b927ab2388ab446e399f162e66c45ecfa3d181c 2023-07-29T16:04:38.061 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-29T16:04:38.061 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Storing signatures 2023-07-29T16:04:39.034 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc) 2023-07-29T16:04:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc) 2023-07-29T16:04:39.035 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2023-07-29T16:04:40.211 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2023-07-29T16:04:40.212 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2023-07-29T16:04:41.086 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQCYOMVkdUMSLRAADxmbVgwqWzrjqWjkbTPSFQ== 2023-07-29T16:04:41.865 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQCZOMVkqwYhHBAAPOc69djuGAS+08DWQepb9A== 2023-07-29T16:04:42.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQCaOMVkhQ80DxAAah4tWv2Re3jIDEn6tL8NOA== 2023-07-29T16:04:42.643 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2023-07-29T16:04:43.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-29T16:04:43.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-29T16:04:43.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:43.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-29T16:04:43.419 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-29T16:04:43.419 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-07-29T16:04:43.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:43.420 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-29T16:04:43.420 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:04:43.420 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2023-07-29T16:04:44.300 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2023-07-29T16:04:44.544 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-29T16:04:44.762 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-29T16:04:44.993 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-858a7894-2e29-11ee-9b36-001a4aab830c.target → /etc/systemd/system/ceph-858a7894-2e29-11ee-9b36-001a4aab830c.target. 2023-07-29T16:04:44.993 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-858a7894-2e29-11ee-9b36-001a4aab830c.target → /etc/systemd/system/ceph-858a7894-2e29-11ee-9b36-001a4aab830c.target. 2023-07-29T16:04:45.296 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:04:45.297 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a.service: Unit ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a.service not loaded. 2023-07-29T16:04:45.500 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-858a7894-2e29-11ee-9b36-001a4aab830c.target.wants/ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-858a7894-2e29-11ee-9b36-001a4aab830c@.service. 2023-07-29T16:04:45.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:45 smithi031 systemd[1]: Starting Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:04:46.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:46 smithi031 podman[94964]: 2023-07-29T16:04:46.415 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-29T16:04:46.416 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-07-29T16:04:46.433 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-29T16:04:46.434 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-07-29T16:04:46.434 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-07-29T16:04:46.434 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2023-07-29T16:04:46.435 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2023-07-29T16:04:46.436 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2023-07-29T16:04:46.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:46 smithi031 bash[94964]: 4414d0ec55ba7a92c1c9a18b49b38ad5fe344f9b9cbedbf230837ca4768677ee 2023-07-29T16:04:46.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:46 smithi031 systemd[1]: Started Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:04:46.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:46 smithi031 ceph-mon[95004]: mkfs 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:46.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:46 smithi031 ceph-mon[95004]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-29T16:04:47.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:47 smithi031 ceph-mon[95004]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-29T16:04:47.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:47 smithi031 ceph-mon[95004]: monmap e1: 1 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} 2023-07-29T16:04:47.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:47 smithi031 ceph-mon[95004]: fsmap 2023-07-29T16:04:47.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:47 smithi031 ceph-mon[95004]: osdmap e1: 0 total, 0 up, 0 in 2023-07-29T16:04:47.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:47 smithi031 ceph-mon[95004]: mgrmap e1: no daemons active 2023-07-29T16:04:47.582 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:47 smithi031 ceph-mon[95004]: from='client.? 172.21.15.31:0/3928750751' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-29T16:04:47.622 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2023-07-29T16:04:47.622 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:47.622 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-29T16:04:47.622 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2023-07-29T16:04:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.755204s) 2023-07-29T16:04:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-29T16:04:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-29T16:04:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:47.623 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2023-07-29T16:04:47.624 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-29T16:04:47.624 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-29T16:04:47.624 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-29T16:04:47.624 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2023-07-29T16:04:47.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:47.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.084+0000 7fe969472700 1 Processor -- start 2023-07-29T16:04:47.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.084+0000 7fe969472700 1 -- start start 2023-07-29T16:04:47.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.084+0000 7fe969472700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe9640791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:47.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.084+0000 7fe969472700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9640796f0 con 0x7fe96407acb0 2023-07-29T16:04:47.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.085+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe9640791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:47.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.085+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe9640791b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44578/0 (socket says 172.21.15.31:44578) 2023-07-29T16:04:47.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.085+0000 7fe962ffd700 1 -- 172.21.15.31:0/3307212947 learned_addr learned my addr 172.21.15.31:0/3307212947 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:47.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.085+0000 7fe962ffd700 1 -- 172.21.15.31:0/3307212947 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe964079830 con 0x7fe96407acb0 2023-07-29T16:04:47.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.086+0000 7fe962ffd700 1 --2- 172.21.15.31:0/3307212947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe9640791b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9540098a0 tx=0x7fe954009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fefcb719085fb0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:47.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.086+0000 7fe961ffb700 1 -- 172.21.15.31:0/3307212947 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe954004160 con 0x7fe96407acb0 2023-07-29T16:04:47.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.086+0000 7fe961ffb700 1 -- 172.21.15.31:0/3307212947 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9540042c0 con 0x7fe96407acb0 2023-07-29T16:04:47.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.086+0000 7fe961ffb700 1 -- 172.21.15.31:0/3307212947 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9540045a0 con 0x7fe96407acb0 2023-07-29T16:04:47.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.086+0000 7fe969472700 1 -- 172.21.15.31:0/3307212947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 msgr2=0x7fe9640791b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:47.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.086+0000 7fe969472700 1 --2- 172.21.15.31:0/3307212947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe9640791b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9540098a0 tx=0x7fe954009bb0 comp rx=0 tx=0).stop 2023-07-29T16:04:47.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.087+0000 7fe969472700 1 -- 172.21.15.31:0/3307212947 shutdown_connections 2023-07-29T16:04:47.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.087+0000 7fe969472700 1 --2- 172.21.15.31:0/3307212947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe9640791b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:47.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.087+0000 7fe969472700 1 -- 172.21.15.31:0/3307212947 >> 172.21.15.31:0/3307212947 conn(0x7fe9641011d0 msgr2=0x7fe964103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:47.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.087+0000 7fe969472700 1 -- 172.21.15.31:0/3307212947 shutdown_connections 2023-07-29T16:04:47.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.087+0000 7fe969472700 1 -- 172.21.15.31:0/3307212947 wait complete. 2023-07-29T16:04:47.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.088+0000 7fe969472700 1 Processor -- start 2023-07-29T16:04:47.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.088+0000 7fe969472700 1 -- start start 2023-07-29T16:04:47.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.088+0000 7fe969472700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe96419b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:47.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.088+0000 7fe969472700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419b780 con 0x7fe96407acb0 2023-07-29T16:04:47.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.089+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe96419b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.089+0000 7fe962ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe96419b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44592/0 (socket says 172.21.15.31:44592) 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.089+0000 7fe962ffd700 1 -- 172.21.15.31:0/3928750751 learned_addr learned my addr 172.21.15.31:0/3928750751 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.089+0000 7fe962ffd700 1 -- 172.21.15.31:0/3928750751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe954009550 con 0x7fe96407acb0 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.089+0000 7fe962ffd700 1 --2- 172.21.15.31:0/3928750751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe96419b240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe96407a220 tx=0x7fe954004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe954003da0 con 0x7fe96407acb0 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe954011630 con 0x7fe96407acb0 2023-07-29T16:04:47.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe954011910 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96419b980 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96419be20 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe95401c460 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.090+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe95401a830 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.091+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9641950f0 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.094+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fe95401a9f0 con 0x7fe96407acb0 2023-07-29T16:04:47.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.171+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe96402cc30 con 0x7fe96407acb0 2023-07-29T16:04:47.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.171+0000 7fe94bfff700 1 -- 172.21.15.31:0/3928750751 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fe95401ab90 con 0x7fe96407acb0 2023-07-29T16:04:47.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 msgr2=0x7fe96419b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:47.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 --2- 172.21.15.31:0/3928750751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe96419b240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe96407a220 tx=0x7fe954004750 comp rx=0 tx=0).stop 2023-07-29T16:04:47.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 shutdown_connections 2023-07-29T16:04:47.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 --2- 172.21.15.31:0/3928750751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe96407acb0 0x7fe96419b240 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:47.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 >> 172.21.15.31:0/3928750751 conn(0x7fe9641011d0 msgr2=0x7fe964102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:47.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 shutdown_connections 2023-07-29T16:04:47.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:47.172+0000 7fe969472700 1 -- 172.21.15.31:0/3928750751 wait complete. 2023-07-29T16:04:47.635 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2023-07-29T16:04:47.635 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2023-07-29T16:04:48.595 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2023-07-29T16:04:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:48.605 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2023-07-29T16:04:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-29T16:04:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-29T16:04:48.606 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-29T16:04:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-29T16:04:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-29T16:04:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-29T16:04:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:48.607 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2023-07-29T16:04:48.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-29T16:04:48.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-29T16:04:48.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-29T16:04:48.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-29T16:04:48.608 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-29T16:04:48.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.136+0000 7f21bfd85700 1 Processor -- start 2023-07-29T16:04:48.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.136+0000 7f21bfd85700 1 -- start start 2023-07-29T16:04:48.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.137+0000 7f21bfd85700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b80791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:48.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.137+0000 7f21bfd85700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b8079730 con 0x7f21b807acf0 2023-07-29T16:04:48.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.137+0000 7f21bdb21700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b80791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:48.609 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.137+0000 7f21bdb21700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b80791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44604/0 (socket says 172.21.15.31:44604) 2023-07-29T16:04:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.137+0000 7f21bdb21700 1 -- 172.21.15.31:0/702926146 learned_addr learned my addr 172.21.15.31:0/702926146 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.137+0000 7f21bdb21700 1 -- 172.21.15.31:0/702926146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b8079870 con 0x7f21b807acf0 2023-07-29T16:04:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.138+0000 7f21bdb21700 1 --2- 172.21.15.31:0/702926146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b80791f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21b40098a0 tx=0x7f21b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd6282f627aba18 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.138+0000 7f21bcb1f700 1 -- 172.21.15.31:0/702926146 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21b4004160 con 0x7f21b807acf0 2023-07-29T16:04:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.138+0000 7f21bcb1f700 1 -- 172.21.15.31:0/702926146 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b40042c0 con 0x7f21b807acf0 2023-07-29T16:04:48.610 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.138+0000 7f21bcb1f700 1 -- 172.21.15.31:0/702926146 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21b40045a0 con 0x7f21b807acf0 2023-07-29T16:04:48.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.138+0000 7f21bfd85700 1 -- 172.21.15.31:0/702926146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 msgr2=0x7f21b80791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:48.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.138+0000 7f21bfd85700 1 --2- 172.21.15.31:0/702926146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b80791f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f21b40098a0 tx=0x7f21b4009bb0 comp rx=0 tx=0).stop 2023-07-29T16:04:48.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.139+0000 7f21bfd85700 1 -- 172.21.15.31:0/702926146 shutdown_connections 2023-07-29T16:04:48.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.139+0000 7f21bfd85700 1 --2- 172.21.15.31:0/702926146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b80791f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:48.611 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.139+0000 7f21bfd85700 1 -- 172.21.15.31:0/702926146 >> 172.21.15.31:0/702926146 conn(0x7f21b81011d0 msgr2=0x7f21b8103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:48.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.139+0000 7f21bfd85700 1 -- 172.21.15.31:0/702926146 shutdown_connections 2023-07-29T16:04:48.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.139+0000 7f21bfd85700 1 -- 172.21.15.31:0/702926146 wait complete. 2023-07-29T16:04:48.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.140+0000 7f21bfd85700 1 Processor -- start 2023-07-29T16:04:48.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.140+0000 7f21bfd85700 1 -- start start 2023-07-29T16:04:48.612 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.140+0000 7f21bfd85700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b819b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:48.613 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.140+0000 7f21bfd85700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b819b7c0 con 0x7f21b807acf0 2023-07-29T16:04:48.613 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.140+0000 7f21bdb21700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b819b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:48.613 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21bdb21700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b819b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44610/0 (socket says 172.21.15.31:44610) 2023-07-29T16:04:48.613 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21bdb21700 1 -- 172.21.15.31:0/1109029976 learned_addr learned my addr 172.21.15.31:0/1109029976 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:48.613 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21bdb21700 1 -- 172.21.15.31:0/1109029976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b4009550 con 0x7f21b807acf0 2023-07-29T16:04:48.613 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21bdb21700 1 --2- 172.21.15.31:0/1109029976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b819b280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21b4004000 tx=0x7f21b4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:48.614 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21b4011630 con 0x7f21b807acf0 2023-07-29T16:04:48.614 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b4011790 con 0x7f21b807acf0 2023-07-29T16:04:48.614 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f21b4011a30 con 0x7f21b807acf0 2023-07-29T16:04:48.614 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.141+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b819b9c0 con 0x7f21b807acf0 2023-07-29T16:04:48.614 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.142+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b819be60 con 0x7f21b807acf0 2023-07-29T16:04:48.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.142+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f21b4022070 con 0x7f21b807acf0 2023-07-29T16:04:48.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.142+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f21b401c450 con 0x7f21b807acf0 2023-07-29T16:04:48.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.143+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b81950b0 con 0x7f21b807acf0 2023-07-29T16:04:48.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.145+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f21b401b450 con 0x7f21b807acf0 2023-07-29T16:04:48.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.218+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f21b8062380 con 0x7f21b807acf0 2023-07-29T16:04:48.615 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.222+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21b401b5f0 con 0x7f21b807acf0 2023-07-29T16:04:48.616 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.223+0000 7f21aeffd700 1 -- 172.21.15.31:0/1109029976 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f21b40375d0 con 0x7f21b807acf0 2023-07-29T16:04:48.616 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 msgr2=0x7f21b819b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:48.616 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 --2- 172.21.15.31:0/1109029976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b819b280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f21b4004000 tx=0x7f21b4004750 comp rx=0 tx=0).stop 2023-07-29T16:04:48.616 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 shutdown_connections 2023-07-29T16:04:48.617 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 --2- 172.21.15.31:0/1109029976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21b807acf0 0x7f21b819b280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:48.617 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 >> 172.21.15.31:0/1109029976 conn(0x7f21b81011d0 msgr2=0x7f21b8102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:48.617 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 shutdown_connections 2023-07-29T16:04:48.617 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:48.224+0000 7f21bfd85700 1 -- 172.21.15.31:0/1109029976 wait complete. 2023-07-29T16:04:48.617 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2023-07-29T16:04:48.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:48 smithi031 ceph-mon[95004]: from='client.? 172.21.15.31:0/1109029976' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-29T16:04:48.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:48 smithi031 ceph-mon[95004]: from='client.? 172.21.15.31:0/1109029976' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-29T16:04:49.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.120+0000 7f9f0a4c1700 1 Processor -- start 2023-07-29T16:04:49.583 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.120+0000 7f9f0a4c1700 1 -- start start 2023-07-29T16:04:49.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.120+0000 7f9f0a4c1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f04079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:49.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.120+0000 7f9f0a4c1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04079750 con 0x7f9f0407ad10 2023-07-29T16:04:49.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.121+0000 7f9f03fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f04079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:49.584 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.121+0000 7f9f03fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f04079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44618/0 (socket says 172.21.15.31:44618) 2023-07-29T16:04:49.585 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.121+0000 7f9f03fff700 1 -- 172.21.15.31:0/903612893 learned_addr learned my addr 172.21.15.31:0/903612893 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:49.585 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.121+0000 7f9f03fff700 1 -- 172.21.15.31:0/903612893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f04079890 con 0x7f9f0407ad10 2023-07-29T16:04:49.586 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.121+0000 7f9f03fff700 1 --2- 172.21.15.31:0/903612893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f04079210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40098a0 tx=0x7f9ef4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ba9aeff93c529d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:49.586 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f02ffd700 1 -- 172.21.15.31:0/903612893 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ef4004160 con 0x7f9f0407ad10 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f02ffd700 1 -- 172.21.15.31:0/903612893 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ef40042c0 con 0x7f9f0407ad10 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f02ffd700 1 -- 172.21.15.31:0/903612893 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ef4011a00 con 0x7f9f0407ad10 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/903612893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 msgr2=0x7f9f04079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f0a4c1700 1 --2- 172.21.15.31:0/903612893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f04079210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40098a0 tx=0x7f9ef4009bb0 comp rx=0 tx=0).stop 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/903612893 shutdown_connections 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f0a4c1700 1 --2- 172.21.15.31:0/903612893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f04079210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:49.587 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/903612893 >> 172.21.15.31:0/903612893 conn(0x7f9f041011d0 msgr2=0x7f9f04103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:49.588 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.122+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/903612893 shutdown_connections 2023-07-29T16:04:49.588 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.123+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/903612893 wait complete. 2023-07-29T16:04:49.589 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.123+0000 7f9f0a4c1700 1 Processor -- start 2023-07-29T16:04:49.589 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.124+0000 7f9f0a4c1700 1 -- start start 2023-07-29T16:04:49.589 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.124+0000 7f9f0a4c1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f0419ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:49.589 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.124+0000 7f9f0a4c1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0419bf60 con 0x7f9f0407ad10 2023-07-29T16:04:49.589 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.124+0000 7f9f03fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f0419ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:49.589 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.124+0000 7f9f03fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f0419ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44630/0 (socket says 172.21.15.31:44630) 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.124+0000 7f9f03fff700 1 -- 172.21.15.31:0/3481541632 learned_addr learned my addr 172.21.15.31:0/3481541632 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f03fff700 1 -- 172.21.15.31:0/3481541632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4009550 con 0x7f9f0407ad10 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f03fff700 1 --2- 172.21.15.31:0/3481541632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f0419ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4006af0 tx=0x7f9ef4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ef4003a40 con 0x7f9f0407ad10 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ef4003ba0 con 0x7f9f0407ad10 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0419c160 con 0x7f9f0407ad10 2023-07-29T16:04:49.590 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ef401a700 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0419c600 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.125+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9ef401ac30 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.126+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9ef4019700 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.127+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f04062380 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.130+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f9ef4024ad0 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.203+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9f0419f2e0 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.204+0000 7f9f017fa700 1 -- 172.21.15.31:0/3481541632 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f9ef4024c70 con 0x7f9f0407ad10 2023-07-29T16:04:49.591 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.204+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 msgr2=0x7f9f0419ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:49.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.204+0000 7f9f0a4c1700 1 --2- 172.21.15.31:0/3481541632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f0419ba20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4006af0 tx=0x7f9ef4011ad0 comp rx=0 tx=0).stop 2023-07-29T16:04:49.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.205+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 shutdown_connections 2023-07-29T16:04:49.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.205+0000 7f9f0a4c1700 1 --2- 172.21.15.31:0/3481541632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0407ad10 0x7f9f0419ba20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:49.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.205+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 >> 172.21.15.31:0/3481541632 conn(0x7f9f041011d0 msgr2=0x7f9f04101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:49.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.205+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 shutdown_connections 2023-07-29T16:04:49.592 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:49.205+0000 7f9f0a4c1700 1 -- 172.21.15.31:0/3481541632 wait complete. 2023-07-29T16:04:49.593 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2023-07-29T16:04:49.599 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:49 smithi031 ceph-mon[95004]: from='client.? 172.21.15.31:0/3481541632' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:04:49.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:49 smithi031 systemd[1]: Stopping Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:04:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:50 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[94980]: 2023-07-29T16:04:50.044+0000 7f46d8456700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:04:50.329 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:50 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[94980]: 2023-07-29T16:04:50.044+0000 7f46d8456700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-29T16:04:50.598 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:50 smithi031 bash[95341]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a 2023-07-29T16:04:51.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:50 smithi031 systemd[1]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a.service: Succeeded. 2023-07-29T16:04:51.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:50 smithi031 systemd[1]: Stopped Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:04:51.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:50 smithi031 systemd[1]: Starting Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:04:51.590 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 podman[95464]: 2023-07-29T16:04:51.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 bash[95464]: 717b6f4839e4ad3900c19af2bc550003fa8be2d22d8059cc2127d53abcd9a0af 2023-07-29T16:04:51.591 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: set uid:gid to 167:167 (ceph:ceph) 2023-07-29T16:04:51.607 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc), process ceph-mon, pid 7 2023-07-29T16:04:51.607 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: pidfile_write: ignore empty --pid-file 2023-07-29T16:04:51.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: load: jerasure load: lrc 2023-07-29T16:04:51.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: RocksDB version: 7.9.2 2023-07-29T16:04:51.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Git sha 0 2023-07-29T16:04:51.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Compile date 2023-07-28 16:56:26 2023-07-29T16:04:51.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: DB SUMMARY 2023-07-29T16:04:51.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: DB Session ID: BRKQ8RKKG9G90DRGCKXV 2023-07-29T16:04:51.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: CURRENT file: CURRENT 2023-07-29T16:04:51.609 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: IDENTITY file: IDENTITY 2023-07-29T16:04:51.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-29T16:04:51.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-29T16:04:51.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99093 ; 2023-07-29T16:04:51.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.error_if_exists: 0 2023-07-29T16:04:51.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.create_if_missing: 0 2023-07-29T16:04:51.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.paranoid_checks: 1 2023-07-29T16:04:51.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-29T16:04:51.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-29T16:04:51.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-29T16:04:51.612 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.env: 0x557d6abfb720 2023-07-29T16:04:51.612 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.fs: PosixFileSystem 2023-07-29T16:04:51.612 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.info_log: 0x557d6c23f420 2023-07-29T16:04:51.612 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_file_opening_threads: 16 2023-07-29T16:04:51.612 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.statistics: (nil) 2023-07-29T16:04:51.613 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.use_fsync: 0 2023-07-29T16:04:51.613 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_log_file_size: 0 2023-07-29T16:04:51.613 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T16:04:51.613 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T16:04:51.613 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.keep_log_file_num: 1000 2023-07-29T16:04:51.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.recycle_log_file_num: 0 2023-07-29T16:04:51.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.allow_fallocate: 1 2023-07-29T16:04:51.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.allow_mmap_reads: 0 2023-07-29T16:04:51.614 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.allow_mmap_writes: 0 2023-07-29T16:04:51.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.use_direct_reads: 0 2023-07-29T16:04:51.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T16:04:51.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.create_missing_column_families: 0 2023-07-29T16:04:51.615 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.db_log_dir: 2023-07-29T16:04:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.wal_dir: 2023-07-29T16:04:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T16:04:51.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T16:04:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T16:04:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T16:04:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T16:04:51.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T16:04:51.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.advise_random_on_open: 1 2023-07-29T16:04:51.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.db_write_buffer_size: 0 2023-07-29T16:04:51.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.write_buffer_manager: 0x557d6b4d05a0 2023-07-29T16:04:51.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T16:04:51.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T16:04:51.618 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T16:04:51.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.rate_limiter: (nil) 2023-07-29T16:04:51.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T16:04:51.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.wal_recovery_mode: 2 2023-07-29T16:04:51.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.enable_thread_tracking: 0 2023-07-29T16:04:51.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.enable_pipelined_write: 0 2023-07-29T16:04:51.619 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.unordered_write: 0 2023-07-29T16:04:51.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T16:04:51.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T16:04:51.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T16:04:51.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T16:04:51.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.row_cache: None 2023-07-29T16:04:51.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.wal_filter: None 2023-07-29T16:04:51.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T16:04:51.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.allow_ingest_behind: 0 2023-07-29T16:04:51.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.two_write_queues: 0 2023-07-29T16:04:51.622 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.manual_wal_flush: 0 2023-07-29T16:04:51.622 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.wal_compression: 0 2023-07-29T16:04:51.622 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.atomic_flush: 0 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.log_readahead_size: 0 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.best_efforts_recovery: 0 2023-07-29T16:04:51.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-29T16:04:51.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-29T16:04:51.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.allow_data_in_errors: 0 2023-07-29T16:04:51.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.db_host_id: __hostname__ 2023-07-29T16:04:51.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-29T16:04:51.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_background_jobs: 2 2023-07-29T16:04:51.625 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_background_compactions: -1 2023-07-29T16:04:51.625 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_subcompactions: 1 2023-07-29T16:04:51.625 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T16:04:51.625 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T16:04:51.625 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T16:04:51.625 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_total_wal_size: 0 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_open_files: -1 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bytes_per_sync: 0 2023-07-29T16:04:51.626 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T16:04:51.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T16:04:51.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_readahead_size: 0 2023-07-29T16:04:51.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_background_flushes: -1 2023-07-29T16:04:51.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Compression algorithms supported: 2023-07-29T16:04:51.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T16:04:51.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kZSTD supported: 0 2023-07-29T16:04:51.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kXpressCompression supported: 0 2023-07-29T16:04:51.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kLZ4HCCompression supported: 1 2023-07-29T16:04:51.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kZlibCompression supported: 1 2023-07-29T16:04:51.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kSnappyCompression supported: 1 2023-07-29T16:04:51.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kLZ4Compression supported: 1 2023-07-29T16:04:51.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: kBZip2Compression supported: 0 2023-07-29T16:04:51.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T16:04:51.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-29T16:04:51.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-29T16:04:51.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-29T16:04:51.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T16:04:51.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.merge_operator: 2023-07-29T16:04:51.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_filter: None 2023-07-29T16:04:51.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_filter_factory: None 2023-07-29T16:04:51.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.sst_partitioner_factory: None 2023-07-29T16:04:51.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T16:04:51.632 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T16:04:51.660 INFO:teuthology.orchestra.run.smithi031.stdout:Setting mon public_network to 172.21.0.0/20 2023-07-29T16:04:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557d6c23f520) 2023-07-29T16:04:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2023-07-29T16:04:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T16:04:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T16:04:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2023-07-29T16:04:51.911 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x557d6b5511f0 2023-07-29T16:04:51.912 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2023-07-29T16:04:51.913 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2023-07-29T16:04:51.914 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2023-07-29T16:04:51.915 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.write_buffer_size: 33554432 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_write_buffer_number: 2 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression: NoCompression 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression: Disabled 2023-07-29T16:04:51.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.prefix_extractor: nullptr 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.num_levels: 7 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T16:04:51.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-29T16:04:51.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T16:04:51.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.level: 32767 2023-07-29T16:04:51.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.strategy: 0 2023-07-29T16:04:51.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T16:04:51.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:04:51.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-29T16:04:51.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-29T16:04:51.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.enabled: false 2023-07-29T16:04:51.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-29T16:04:51.921 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T16:04:51.921 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T16:04:51.922 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T16:04:51.922 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.target_file_size_base: 67108864 2023-07-29T16:04:51.922 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T16:04:51.922 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T16:04:51.923 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T16:04:51.923 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T16:04:51.923 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T16:04:51.923 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T16:04:51.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T16:04:51.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T16:04:51.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T16:04:51.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T16:04:51.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T16:04:51.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T16:04:51.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T16:04:51.926 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-29T16:04:51.926 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.arena_block_size: 1048576 2023-07-29T16:04:51.927 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T16:04:51.927 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T16:04:51.927 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.disable_auto_compactions: 0 2023-07-29T16:04:51.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T16:04:51.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T16:04:51.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T16:04:51.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T16:04:51.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T16:04:51.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T16:04:51.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T16:04:51.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T16:04:51.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T16:04:51.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T16:04:51.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.table_properties_collectors: 2023-07-29T16:04:51.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.inplace_update_support: 0 2023-07-29T16:04:51.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T16:04:51.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T16:04:51.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T16:04:51.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T16:04:51.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.bloom_locality: 0 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.max_successive_merges: 0 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.paranoid_file_checks: 0 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.force_consistency_checks: 1 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.report_bg_io_stats: 0 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.ttl: 2592000 2023-07-29T16:04:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T16:04:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-29T16:04:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-29T16:04:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.enable_blob_files: false 2023-07-29T16:04:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.min_blob_size: 0 2023-07-29T16:04:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.blob_file_size: 268435456 2023-07-29T16:04:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.blob_file_starting_level: 0 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-29T16:04:51.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 331de5d9-e075-4b58-a6e1-c638d0568eb1 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646691605523, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646691607273, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 92310, "index_size": 284, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13747, "raw_average_key_size": 49, "raw_value_size": 84807, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1690646691, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "331de5d9-e075-4b58-a6e1-c638d0568eb1", "db_session_id": "BRKQ8RKKG9G90DRGCKXV", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646691607409, "job": 1, "event": "recovery_finished"} 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-29T16:04:51.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557d6b5ee000 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: DB pointer 0x557d6b5da000 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T16:04:51.938 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2023-07-29T16:04:51.939 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 93.94 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 79.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 93.94 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 79.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 79.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T16:04:51.940 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 79.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:04:51.941 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:04:51.942 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:04:51.942 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:04:51.942 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:04:51.942 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 14.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:04:51.942 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 14.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x557d6b5511f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 2.3e-05 secs_since: 0 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:51.943 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???) e1 preinit fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).mds e1 new map 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).mds e1 print_map 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2023-07-29T16:04:51.944 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-07-29T16:04:51.945 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2023-07-29T16:04:51.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-29T16:04:51.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:04:51.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:04:51.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:04:51.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-29T16:04:51.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expand map: {default=false} 2023-07-29T16:04:51.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta from 'false' to 'false' 2023-07-29T16:04:51.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expanded map: {default=false} 2023-07-29T16:04:51.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expand map: {default=info} 2023-07-29T16:04:51.946 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta from 'info' to 'info' 2023-07-29T16:04:51.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expanded map: {default=info} 2023-07-29T16:04:51.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expand map: {default=daemon} 2023-07-29T16:04:51.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-29T16:04:51.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expanded map: {default=daemon} 2023-07-29T16:04:51.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expand map: {default=debug} 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta from 'debug' to 'debug' 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: expand_channel_meta expanded map: {default=debug} 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 systemd[1]: Started Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: monmap e1: 1 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: fsmap 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: osdmap e1: 0 total, 0 up, 0 in 2023-07-29T16:04:51.948 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:51 smithi031 ceph-mon[95504]: mgrmap e1: no daemons active 2023-07-29T16:04:52.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.182+0000 7fec1c399700 1 Processor -- start 2023-07-29T16:04:52.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.183+0000 7fec1c399700 1 -- start start 2023-07-29T16:04:52.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.183+0000 7fec1c399700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec14079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:52.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.183+0000 7fec1c399700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14079750 con 0x7fec1407ad10 2023-07-29T16:04:52.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.184+0000 7fec1a135700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec14079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:52.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.184+0000 7fec1a135700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec14079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35742/0 (socket says 172.21.15.31:35742) 2023-07-29T16:04:52.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.184+0000 7fec1a135700 1 -- 172.21.15.31:0/3210173115 learned_addr learned my addr 172.21.15.31:0/3210173115 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:52.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.184+0000 7fec1a135700 1 -- 172.21.15.31:0/3210173115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec14079890 con 0x7fec1407ad10 2023-07-29T16:04:52.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.184+0000 7fec1a135700 1 --2- 172.21.15.31:0/3210173115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec14079210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fec100098a0 tx=0x7fec10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39630935c45809c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:52.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec19133700 1 -- 172.21.15.31:0/3210173115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec10004160 con 0x7fec1407ad10 2023-07-29T16:04:52.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec19133700 1 -- 172.21.15.31:0/3210173115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec100042c0 con 0x7fec1407ad10 2023-07-29T16:04:52.646 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec19133700 1 -- 172.21.15.31:0/3210173115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec10011a00 con 0x7fec1407ad10 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec1c399700 1 -- 172.21.15.31:0/3210173115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 msgr2=0x7fec14079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec1c399700 1 --2- 172.21.15.31:0/3210173115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec14079210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fec100098a0 tx=0x7fec10009bb0 comp rx=0 tx=0).stop 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec1c399700 1 -- 172.21.15.31:0/3210173115 shutdown_connections 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec1c399700 1 --2- 172.21.15.31:0/3210173115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec14079210 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.185+0000 7fec1c399700 1 -- 172.21.15.31:0/3210173115 >> 172.21.15.31:0/3210173115 conn(0x7fec141011d0 msgr2=0x7fec14103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.186+0000 7fec1c399700 1 -- 172.21.15.31:0/3210173115 shutdown_connections 2023-07-29T16:04:52.647 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.186+0000 7fec1c399700 1 -- 172.21.15.31:0/3210173115 wait complete. 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1c399700 1 Processor -- start 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1c399700 1 -- start start 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1c399700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec1419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1c399700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1419bfa0 con 0x7fec1407ad10 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1a135700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec1419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1a135700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec1419ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35744/0 (socket says 172.21.15.31:35744) 2023-07-29T16:04:52.648 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.187+0000 7fec1a135700 1 -- 172.21.15.31:0/1180578941 learned_addr learned my addr 172.21.15.31:0/1180578941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec1a135700 1 -- 172.21.15.31:0/1180578941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec10009550 con 0x7fec1407ad10 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec1a135700 1 --2- 172.21.15.31:0/1180578941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec1419ba60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fec10006af0 tx=0x7fec10011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec10003a40 con 0x7fec1407ad10 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1419c1a0 con 0x7fec1407ad10 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec10003ba0 con 0x7fec1407ad10 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fec1001a700 con 0x7fec1407ad10 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.188+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1419c640 con 0x7fec1407ad10 2023-07-29T16:04:52.649 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.189+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fec1001ac80 con 0x7fec1407ad10 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.189+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fec10019700 con 0x7fec1407ad10 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.189+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec14062380 con 0x7fec1407ad10 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.192+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7fec10024ad0 con 0x7fec1407ad10 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.266+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fec1419f390 con 0x7fec1407ad10 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.269+0000 7fec0b7fe700 1 -- 172.21.15.31:0/1180578941 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fec10024c70 con 0x7fec1407ad10 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 msgr2=0x7fec1419ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:52.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 --2- 172.21.15.31:0/1180578941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec1419ba60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fec10006af0 tx=0x7fec10011ad0 comp rx=0 tx=0).stop 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 shutdown_connections 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 --2- 172.21.15.31:0/1180578941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec1407ad10 0x7fec1419ba60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 >> 172.21.15.31:0/1180578941 conn(0x7fec141011d0 msgr2=0x7fec14101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 shutdown_connections 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:52.270+0000 7fec1c399700 1 -- 172.21.15.31:0/1180578941 wait complete. 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:04:52.651 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2023-07-29T16:04:52.652 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 9283 ... 2023-07-29T16:04:52.652 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 8765 ... 2023-07-29T16:04:52.943 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x 2023-07-29T16:04:52.943 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x.service: Unit ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x.service not loaded. 2023-07-29T16:04:53.159 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-858a7894-2e29-11ee-9b36-001a4aab830c.target.wants/ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-858a7894-2e29-11ee-9b36-001a4aab830c@.service. 2023-07-29T16:04:53.264 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:53 smithi031 systemd[1]: Starting Ceph mgr.x for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:04:53.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:53 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1180578941' entity='client.admin' 2023-07-29T16:04:53.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:53 smithi031 podman[95734]: 2023-07-29T16:04:53.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:53 smithi031 bash[95734]: 9a4241da149dcf649366b2683783925846d58b0e5efae75913e997b2e78913f6 2023-07-29T16:04:53.972 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-29T16:04:53.972 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-07-29T16:04:53.990 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-29T16:04:53.990 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-07-29T16:04:53.990 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-07-29T16:04:53.990 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2023-07-29T16:04:54.007 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-29T16:04:54.008 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-07-29T16:04:54.024 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-29T16:04:54.025 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-07-29T16:04:54.025 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-07-29T16:04:54.025 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-29T16:04:54.026 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2023-07-29T16:04:54.026 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2023-07-29T16:04:54.202 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:53 smithi031 systemd[1]: Started Ceph mgr.x for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:04:54.202 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:54.094+0000 7f604721e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T16:04:54.203 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:54.206+0000 7f604721e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T16:04:54.456 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:54.332+0000 7f604721e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T16:04:54.796 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:54 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3352227493' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:04:54.797 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:54.460+0000 7f604721e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T16:04:54.797 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:54.694+0000 7f604721e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T16:04:55.021 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:55.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:04:55.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:04:55.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-07-29T16:04:55.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-29T16:04:55.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-29T16:04:55.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-29T16:04:55.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-29T16:04:55.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-29T16:04:55.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-07-29T16:04:55.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:04:55.036 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-29T16:04:55.036 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-07-29T16:04:55.036 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:04:55.036 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2023-07-29T16:04:55.037 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-29T16:04:55.037 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:55.037 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-29T16:04:55.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-29T16:04:55.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-29T16:04:55.038 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:55.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-29T16:04:55.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-29T16:04:55.039 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-29T16:04:55.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-29T16:04:55.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-29T16:04:55.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-29T16:04:55.040 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-29T16:04:55.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-29T16:04:55.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-29T16:04:55.041 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-29T16:04:55.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-29T16:04:55.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-29T16:04:55.042 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-29T16:04:55.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-29T16:04:55.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-29T16:04:55.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.043 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-29T16:04:55.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:55.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-29T16:04:55.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-29T16:04:55.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-29T16:04:55.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-29T16:04:55.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-29T16:04:55.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-29T16:04:55.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-29T16:04:55.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-29T16:04:55.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-07-29T16:04:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:04:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:04:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-29T16:04:55.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:55.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-07-29T16:04:46.418212+0000", 2023-07-29T16:04:55.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:04:55.049 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:55.049 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-29T16:04:55.049 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:04:55.049 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.572+0000 7f8464c04700 1 Processor -- start 2023-07-29T16:04:55.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.572+0000 7f8464c04700 1 -- start start 2023-07-29T16:04:55.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.573+0000 7f8464c04700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f84600791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:55.050 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.573+0000 7f8464c04700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8460079730 con 0x7f846007acf0 2023-07-29T16:04:55.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.573+0000 7f845e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f84600791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:55.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.573+0000 7f845e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f84600791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35772/0 (socket says 172.21.15.31:35772) 2023-07-29T16:04:55.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.573+0000 7f845e59c700 1 -- 172.21.15.31:0/1537528436 learned_addr learned my addr 172.21.15.31:0/1537528436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:55.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.574+0000 7f845e59c700 1 -- 172.21.15.31:0/1537528436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8460079870 con 0x7f846007acf0 2023-07-29T16:04:55.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.574+0000 7f845e59c700 1 --2- 172.21.15.31:0/1537528436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f84600791f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f84500098a0 tx=0x7f8450009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d87cf1e686b68b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:55.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.574+0000 7f845d59a700 1 -- 172.21.15.31:0/1537528436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8450004160 con 0x7f846007acf0 2023-07-29T16:04:55.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.574+0000 7f845d59a700 1 -- 172.21.15.31:0/1537528436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84500042c0 con 0x7f846007acf0 2023-07-29T16:04:55.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.574+0000 7f845d59a700 1 -- 172.21.15.31:0/1537528436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8450011a00 con 0x7f846007acf0 2023-07-29T16:04:55.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 -- 172.21.15.31:0/1537528436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 msgr2=0x7f84600791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:55.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 --2- 172.21.15.31:0/1537528436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f84600791f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f84500098a0 tx=0x7f8450009bb0 comp rx=0 tx=0).stop 2023-07-29T16:04:55.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 -- 172.21.15.31:0/1537528436 shutdown_connections 2023-07-29T16:04:55.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 --2- 172.21.15.31:0/1537528436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f84600791f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:55.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 -- 172.21.15.31:0/1537528436 >> 172.21.15.31:0/1537528436 conn(0x7f84601011d0 msgr2=0x7f8460103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:55.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 -- 172.21.15.31:0/1537528436 shutdown_connections 2023-07-29T16:04:55.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.575+0000 7f8464c04700 1 -- 172.21.15.31:0/1537528436 wait complete. 2023-07-29T16:04:55.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.576+0000 7f8464c04700 1 Processor -- start 2023-07-29T16:04:55.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.576+0000 7f8464c04700 1 -- start start 2023-07-29T16:04:55.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f8464c04700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f846019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:55.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f8464c04700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846019bf80 con 0x7f846007acf0 2023-07-29T16:04:55.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f845e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f846019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:55.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f845e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f846019ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35784/0 (socket says 172.21.15.31:35784) 2023-07-29T16:04:55.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f845e59c700 1 -- 172.21.15.31:0/3352227493 learned_addr learned my addr 172.21.15.31:0/3352227493 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:55.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f845e59c700 1 -- 172.21.15.31:0/3352227493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450009550 con 0x7f846007acf0 2023-07-29T16:04:55.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.577+0000 7f845e59c700 1 --2- 172.21.15.31:0/3352227493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f846019ba40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8450006b20 tx=0x7f8450011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:55.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8450003b70 con 0x7f846007acf0 2023-07-29T16:04:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8450003cd0 con 0x7f846007acf0 2023-07-29T16:04:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84500247c0 con 0x7f846007acf0 2023-07-29T16:04:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846019c180 con 0x7f846007acf0 2023-07-29T16:04:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846019c620 con 0x7f846007acf0 2023-07-29T16:04:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8450024c80 con 0x7f846007acf0 2023-07-29T16:04:55.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.578+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f84500193c0 con 0x7f846007acf0 2023-07-29T16:04:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.579+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8460062380 con 0x7f846007acf0 2023-07-29T16:04:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.582+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f845001ab40 con 0x7f846007acf0 2023-07-29T16:04:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.658+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f846019efe0 con 0x7f846007acf0 2023-07-29T16:04:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.659+0000 7f844f7fe700 1 -- 172.21.15.31:0/3352227493 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f845001ace0 con 0x7f846007acf0 2023-07-29T16:04:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 msgr2=0x7f846019ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 --2- 172.21.15.31:0/3352227493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f846019ba40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8450006b20 tx=0x7f8450011ad0 comp rx=0 tx=0).stop 2023-07-29T16:04:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 shutdown_connections 2023-07-29T16:04:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 --2- 172.21.15.31:0/3352227493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f846007acf0 0x7f846019ba40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:55.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 >> 172.21.15.31:0/3352227493 conn(0x7f84601011d0 msgr2=0x7f8460101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:55.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 shutdown_connections 2023-07-29T16:04:55.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:54.660+0000 7f8464c04700 1 -- 172.21.15.31:0/3352227493 wait complete. 2023-07-29T16:04:55.060 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2023-07-29T16:04:55.407 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:55.126+0000 7f604721e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T16:04:55.408 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:55.245+0000 7f604721e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T16:04:55.408 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:55.412+0000 7f604721e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T16:04:55.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:55.540+0000 7f604721e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T16:04:56.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:56.106+0000 7f604721e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T16:04:57.180 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:56.924+0000 7f604721e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T16:04:57.180 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:57 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:57.143+0000 7f604721e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T16:04:57.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:04:57 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2074187296' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:04:58.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-29T16:04:58.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-29T16:04:58.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-29T16:04:58.114 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-07-29T16:04:58.114 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:04:58.114 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-29T16:04:58.114 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-07-29T16:04:58.114 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:04:58.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-07-29T16:04:58.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-29T16:04:58.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-29T16:04:58.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-29T16:04:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-29T16:04:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-29T16:04:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-29T16:04:58.124 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-29T16:04:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-29T16:04:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-29T16:04:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-29T16:04:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-29T16:04:58.125 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-29T16:04:58.126 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-29T16:04:58.127 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-29T16:04:58.128 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-07-29T16:04:46.418212+0000", 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.568+0000 7f0cddc64700 1 Processor -- start 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.568+0000 7f0cddc64700 1 -- start start 2023-07-29T16:04:58.129 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.568+0000 7f0cddc64700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd80791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.568+0000 7f0cddc64700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80796f0 con 0x7f0cd807acb0 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.569+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd80791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.569+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd80791b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35786/0 (socket says 172.21.15.31:35786) 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.569+0000 7f0cd77fe700 1 -- 172.21.15.31:0/775605909 learned_addr learned my addr 172.21.15.31:0/775605909 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.569+0000 7f0cd77fe700 1 -- 172.21.15.31:0/775605909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8079830 con 0x7f0cd807acb0 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.570+0000 7f0cd77fe700 1 --2- 172.21.15.31:0/775605909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd80791b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80098a0 tx=0x7f0cc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddb4440554f1c6c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:58.130 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.570+0000 7f0cd67fc700 1 -- 172.21.15.31:0/775605909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc8004160 con 0x7f0cd807acb0 2023-07-29T16:04:58.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.570+0000 7f0cd67fc700 1 -- 172.21.15.31:0/775605909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cc80042c0 con 0x7f0cd807acb0 2023-07-29T16:04:58.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.570+0000 7f0cd67fc700 1 -- 172.21.15.31:0/775605909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc8011a00 con 0x7f0cd807acb0 2023-07-29T16:04:58.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 -- 172.21.15.31:0/775605909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 msgr2=0x7f0cd80791b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:58.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 --2- 172.21.15.31:0/775605909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd80791b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80098a0 tx=0x7f0cc8009bb0 comp rx=0 tx=0).stop 2023-07-29T16:04:58.131 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 -- 172.21.15.31:0/775605909 shutdown_connections 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 --2- 172.21.15.31:0/775605909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd80791b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 -- 172.21.15.31:0/775605909 >> 172.21.15.31:0/775605909 conn(0x7f0cd81011d0 msgr2=0x7f0cd8103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 -- 172.21.15.31:0/775605909 shutdown_connections 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.571+0000 7f0cddc64700 1 -- 172.21.15.31:0/775605909 wait complete. 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.572+0000 7f0cddc64700 1 Processor -- start 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.572+0000 7f0cddc64700 1 -- start start 2023-07-29T16:04:58.132 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.572+0000 7f0cddc64700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.572+0000 7f0cddc64700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd819bf50 con 0x7f0cd807acb0 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.573+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd819ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.573+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd819ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35798/0 (socket says 172.21.15.31:35798) 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.573+0000 7f0cd77fe700 1 -- 172.21.15.31:0/2074187296 learned_addr learned my addr 172.21.15.31:0/2074187296 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.573+0000 7f0cd77fe700 1 -- 172.21.15.31:0/2074187296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8009550 con 0x7f0cd807acb0 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.573+0000 7f0cd77fe700 1 --2- 172.21.15.31:0/2074187296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd819ba10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8006af0 tx=0x7f0cc8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:04:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc8003a40 con 0x7f0cd807acb0 2023-07-29T16:04:58.134 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cc8003ba0 con 0x7f0cd807acb0 2023-07-29T16:04:58.134 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc801a700 con 0x7f0cd807acb0 2023-07-29T16:04:58.134 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd819c150 con 0x7f0cd807acb0 2023-07-29T16:04:58.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd819c5f0 con 0x7f0cd807acb0 2023-07-29T16:04:58.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0cc801ac30 con 0x7f0cd807acb0 2023-07-29T16:04:58.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.574+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0cc8019700 con 0x7f0cd807acb0 2023-07-29T16:04:58.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.575+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8062380 con 0x7f0cd807acb0 2023-07-29T16:04:58.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.577+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f0cc8024ad0 con 0x7f0cd807acb0 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.655+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0cd819f2d0 con 0x7f0cd807acb0 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.656+0000 7f0cd4ff9700 1 -- 172.21.15.31:0/2074187296 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0cc8024c70 con 0x7f0cd807acb0 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 msgr2=0x7f0cd819ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 --2- 172.21.15.31:0/2074187296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd819ba10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8006af0 tx=0x7f0cc8011ad0 comp rx=0 tx=0).stop 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 shutdown_connections 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 --2- 172.21.15.31:0/2074187296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cd807acb0 0x7f0cd819ba10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:04:58.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 >> 172.21.15.31:0/2074187296 conn(0x7f0cd81011d0 msgr2=0x7f0cd8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:04:58.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 shutdown_connections 2023-07-29T16:04:58.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:04:57.657+0000 7f0cddc64700 1 -- 172.21.15.31:0/2074187296 wait complete. 2023-07-29T16:04:58.137 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2023-07-29T16:04:58.297 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:58.174+0000 7f604721e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:04:58.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:58.299+0000 7f604721e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T16:04:58.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:58.528+0000 7f604721e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:04:59.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:58.902+0000 7f604721e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T16:04:59.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:59.185+0000 7f604721e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T16:04:59.680 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:59.387+0000 7f604721e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T16:05:00.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:04:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:04:59.740+0000 7f604721e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T16:05:01.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:00 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/226918341' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:05:01.180 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:00.715+0000 7f604721e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T16:05:01.180 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:00.904+0000 7f604721e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T16:05:01.608 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:01.160+0000 7f604721e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T16:05:01.608 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:01.325+0000 7f604721e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T16:05:01.930 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:01.612+0000 7f604721e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T16:05:03.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:02.743+0000 7f604721e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T16:05:03.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: Activating manager daemon x 2023-07-29T16:05:03.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: mgrmap e2: x(active, starting, since 0.00421315s) 2023-07-29T16:05:03.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T16:05:03.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T16:05:03.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T16:05:03.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:05:03.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T16:05:03.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: Manager daemon x is now available 2023-07-29T16:05:03.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:05:03.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' 2023-07-29T16:05:03.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T16:05:03.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' 2023-07-29T16:05:03.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:02 smithi031 ceph-mon[95504]: from='mgr.14100 172.21.15.31:0/113333017' entity='mgr.x' 2023-07-29T16:05:05.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:04 smithi031 ceph-mon[95504]: mgrmap e3: x(active, since 1.01003s) 2023-07-29T16:05:06.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:05 smithi031 ceph-mon[95504]: mgrmap e4: x(active, since 2s) 2023-07-29T16:05:08.014 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:05:08.014 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.015 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:05:08.016 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-29T16:05:08.017 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:08.018 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-29T16:05:08.018 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-29T16:05:08.018 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-29T16:05:08.024 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-29T16:05:08.024 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-29T16:05:08.025 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:08.026 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-29T16:05:08.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:08.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-07-29T16:04:46.418212+0000", 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.651+0000 7f928879b700 1 Processor -- start 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.651+0000 7f928879b700 1 -- start start 2023-07-29T16:05:08.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.652+0000 7f928879b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f92800791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.652+0000 7f928879b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9280079730 con 0x7f928007acf0 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.652+0000 7f9286537700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f92800791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.652+0000 7f9286537700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f92800791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53356/0 (socket says 172.21.15.31:53356) 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.652+0000 7f9286537700 1 -- 172.21.15.31:0/645521575 learned_addr learned my addr 172.21.15.31:0/645521575 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.653+0000 7f9286537700 1 -- 172.21.15.31:0/645521575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280079870 con 0x7f928007acf0 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.653+0000 7f9286537700 1 --2- 172.21.15.31:0/645521575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f92800791f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f927c0098a0 tx=0x7f927c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72adab4e04de2c6c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.653+0000 7f9285535700 1 -- 172.21.15.31:0/645521575 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f927c004160 con 0x7f928007acf0 2023-07-29T16:05:08.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.653+0000 7f9285535700 1 -- 172.21.15.31:0/645521575 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f927c0042c0 con 0x7f928007acf0 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.653+0000 7f9285535700 1 -- 172.21.15.31:0/645521575 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f927c011a00 con 0x7f928007acf0 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 -- 172.21.15.31:0/645521575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 msgr2=0x7f92800791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 --2- 172.21.15.31:0/645521575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f92800791f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f927c0098a0 tx=0x7f927c009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 -- 172.21.15.31:0/645521575 shutdown_connections 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 --2- 172.21.15.31:0/645521575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f92800791f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 -- 172.21.15.31:0/645521575 >> 172.21.15.31:0/645521575 conn(0x7f92801011d0 msgr2=0x7f9280103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:08.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 -- 172.21.15.31:0/645521575 shutdown_connections 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.654+0000 7f928879b700 1 -- 172.21.15.31:0/645521575 wait complete. 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.655+0000 7f928879b700 1 Processor -- start 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.655+0000 7f928879b700 1 -- start start 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.655+0000 7f928879b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f928019ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.655+0000 7f928879b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928019bf80 con 0x7f928007acf0 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.656+0000 7f9286537700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f928019ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.656+0000 7f9286537700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f928019ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53364/0 (socket says 172.21.15.31:53364) 2023-07-29T16:05:08.032 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.656+0000 7f9286537700 1 -- 172.21.15.31:0/226918341 learned_addr learned my addr 172.21.15.31:0/226918341 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.656+0000 7f9286537700 1 -- 172.21.15.31:0/226918341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c009550 con 0x7f928007acf0 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.656+0000 7f9286537700 1 --2- 172.21.15.31:0/226918341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f928019ba40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f927c006b20 tx=0x7f927c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f927c003b70 con 0x7f928007acf0 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f927c003cd0 con 0x7f928007acf0 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f928019c180 con 0x7f928007acf0 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f927c0247c0 con 0x7f928007acf0 2023-07-29T16:05:08.033 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f928019c620 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f927c024d30 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.657+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f927c019450 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.658+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9280062380 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.660+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72422 (secure 0 0 0) 0x7f927c01ab40 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.738+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f928019f300 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.738+0000 7f92777fe700 1 -- 172.21.15.31:0/226918341 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f927c01ace0 con 0x7f928007acf0 2023-07-29T16:05:08.034 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 msgr2=0x7f928019ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 --2- 172.21.15.31:0/226918341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f928019ba40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f927c006b20 tx=0x7f927c011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 shutdown_connections 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 --2- 172.21.15.31:0/226918341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f928007acf0 0x7f928019ba40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 >> 172.21.15.31:0/226918341 conn(0x7f92801011d0 msgr2=0x7f9280101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 shutdown_connections 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:00.740+0000 7f928879b700 1 -- 172.21.15.31:0/226918341 wait complete. 2023-07-29T16:05:08.035 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (3/15)... 2023-07-29T16:05:11.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1332522390' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-29T16:05:11.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:05:11.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:05:11.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:05:11.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-07-29T16:05:11.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-29T16:05:11.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-29T16:05:11.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 19, 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-29T16:05:11.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-29T16:05:11.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:11.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-29T16:05:11.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-29T16:05:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-29T16:05:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-29T16:05:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-29T16:05:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-29T16:05:11.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-29T16:05:11.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-07-29T16:05:11.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-07-29T16:04:46.418212+0000", 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-07-29T16:05:11.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.565+0000 7f732ce1f700 1 Processor -- start 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.565+0000 7f732ce1f700 1 -- start start 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.566+0000 7f732ce1f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f7328105f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.566+0000 7f732ce1f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73281064d0 con 0x7f7328105b80 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.566+0000 7f732659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f7328105f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.566+0000 7f732659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f7328105f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52448/0 (socket says 172.21.15.31:52448) 2023-07-29T16:05:11.324 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.566+0000 7f732659c700 1 -- 172.21.15.31:0/3013999834 learned_addr learned my addr 172.21.15.31:0/3013999834 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.567+0000 7f732659c700 1 -- 172.21.15.31:0/3013999834 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328106610 con 0x7f7328105b80 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.567+0000 7f732659c700 1 --2- 172.21.15.31:0/3013999834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f7328105f90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f73180098a0 tx=0x7f7318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87a673376350066 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.567+0000 7f732559a700 1 -- 172.21.15.31:0/3013999834 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7318004160 con 0x7f7328105b80 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.567+0000 7f732559a700 1 -- 172.21.15.31:0/3013999834 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73180042c0 con 0x7f7328105b80 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.567+0000 7f732559a700 1 -- 172.21.15.31:0/3013999834 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7318011a00 con 0x7f7328105b80 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 -- 172.21.15.31:0/3013999834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 msgr2=0x7f7328105f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:11.325 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 --2- 172.21.15.31:0/3013999834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f7328105f90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f73180098a0 tx=0x7f7318009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 -- 172.21.15.31:0/3013999834 shutdown_connections 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 --2- 172.21.15.31:0/3013999834 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f7328105f90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 -- 172.21.15.31:0/3013999834 >> 172.21.15.31:0/3013999834 conn(0x7f73281011d0 msgr2=0x7f7328103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 -- 172.21.15.31:0/3013999834 shutdown_connections 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.568+0000 7f732ce1f700 1 -- 172.21.15.31:0/3013999834 wait complete. 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.569+0000 7f732ce1f700 1 Processor -- start 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.569+0000 7f732ce1f700 1 -- start start 2023-07-29T16:05:11.326 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.569+0000 7f732ce1f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f732819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.570+0000 7f732ce1f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732819bf30 con 0x7f7328105b80 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.570+0000 7f732659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f732819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.570+0000 7f732659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f732819b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52456/0 (socket says 172.21.15.31:52456) 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.570+0000 7f732659c700 1 -- 172.21.15.31:0/1332522390 learned_addr learned my addr 172.21.15.31:0/1332522390 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.570+0000 7f732659c700 1 -- 172.21.15.31:0/1332522390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7318009550 con 0x7f7328105b80 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.570+0000 7f732659c700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f732819b9f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7318009fb0 tx=0x7f7318011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:11.327 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.571+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7318003930 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.571+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7318003a90 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.571+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732819c130 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.571+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7318019610 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.571+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732819c5d0 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.572+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) v1 ==== 45058+0+0 (secure 0 0 0) 0x7f731801a9d0 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.572+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7328062380 con 0x7f7328105b80 2023-07-29T16:05:11.328 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.572+0000 7f73177fe700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f7310038330 0x7f731003a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.572+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f731804bfb0 con 0x7f7328105b80 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.573+0000 7f7325d9b700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f7310038330 0x7f731003a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.573+0000 7f7325d9b700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f7310038330 0x7f731003a7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f731c006fd0 tx=0x7f731c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.578+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7318023d20 con 0x7f7328105b80 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.868+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7328107a40 con 0x7f7328105b80 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.868+0000 7f73177fe700 1 -- 172.21.15.31:0/1332522390 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f7318023b40 con 0x7f7328105b80 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f7310038330 msgr2=0x7f731003a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:11.329 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f7310038330 0x7f731003a7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f731c006fd0 tx=0x7f731c006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 msgr2=0x7f732819b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f732819b9f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7318009fb0 tx=0x7f7318011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 shutdown_connections 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f7310038330 0x7f731003a7e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 --2- 172.21.15.31:0/1332522390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328105b80 0x7f732819b9f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 >> 172.21.15.31:0/1332522390 conn(0x7f73281011d0 msgr2=0x7f7328101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 shutdown_connections 2023-07-29T16:05:11.330 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:10.871+0000 7f732ce1f700 1 -- 172.21.15.31:0/1332522390 wait complete. 2023-07-29T16:05:11.331 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2023-07-29T16:05:12.493 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:05:12.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2023-07-29T16:05:12.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:05:12.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-29T16:05:12.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-29T16:05:12.508 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2023-07-29T16:05:12.509 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-29T16:05:12.510 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-29T16:05:12.510 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-29T16:05:12.510 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-29T16:05:12.510 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-29T16:05:12.512 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.856+0000 7f5272adf700 1 Processor -- start 2023-07-29T16:05:12.512 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.856+0000 7f5272adf700 1 -- start start 2023-07-29T16:05:12.512 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.856+0000 7f5272adf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:12.512 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.856+0000 7f5272adf700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c079750 con 0x7f526c07ad10 2023-07-29T16:05:12.513 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.857+0000 7f527087b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:12.513 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.857+0000 7f527087b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52466/0 (socket says 172.21.15.31:52466) 2023-07-29T16:05:12.513 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.857+0000 7f527087b700 1 -- 172.21.15.31:0/3150454667 learned_addr learned my addr 172.21.15.31:0/3150454667 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:12.514 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.857+0000 7f527087b700 1 -- 172.21.15.31:0/3150454667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c079890 con 0x7f526c07ad10 2023-07-29T16:05:12.514 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.857+0000 7f527087b700 1 --2- 172.21.15.31:0/3150454667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c079210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f52680098a0 tx=0x7f5268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef1fecbc7566c5d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:12.514 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f52637fe700 1 -- 172.21.15.31:0/3150454667 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5268004160 con 0x7f526c07ad10 2023-07-29T16:05:12.514 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f52637fe700 1 -- 172.21.15.31:0/3150454667 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52680042c0 con 0x7f526c07ad10 2023-07-29T16:05:12.515 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f52637fe700 1 -- 172.21.15.31:0/3150454667 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5268011a00 con 0x7f526c07ad10 2023-07-29T16:05:12.515 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 -- 172.21.15.31:0/3150454667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 msgr2=0x7f526c079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:12.515 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 --2- 172.21.15.31:0/3150454667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c079210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f52680098a0 tx=0x7f5268009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:12.516 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 -- 172.21.15.31:0/3150454667 shutdown_connections 2023-07-29T16:05:12.517 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 --2- 172.21.15.31:0/3150454667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c079210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:12.517 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 -- 172.21.15.31:0/3150454667 >> 172.21.15.31:0/3150454667 conn(0x7f526c1011d0 msgr2=0x7f526c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:12.517 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 -- 172.21.15.31:0/3150454667 shutdown_connections 2023-07-29T16:05:12.517 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.858+0000 7f5272adf700 1 -- 172.21.15.31:0/3150454667 wait complete. 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.859+0000 7f5272adf700 1 Processor -- start 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f5272adf700 1 -- start start 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f5272adf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c19b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f5272adf700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526c19bef0 con 0x7f526c07ad10 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f527087b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c19b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f527087b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c19b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52476/0 (socket says 172.21.15.31:52476) 2023-07-29T16:05:12.518 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f527087b700 1 -- 172.21.15.31:0/302594638 learned_addr learned my addr 172.21.15.31:0/302594638 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.860+0000 7f527087b700 1 -- 172.21.15.31:0/302594638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5268009550 con 0x7f526c07ad10 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.861+0000 7f527087b700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c19b9b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5268009fb0 tx=0x7f5268011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.861+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5268003930 con 0x7f526c07ad10 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.861+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5268003a90 con 0x7f526c07ad10 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.861+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52680195e0 con 0x7f526c07ad10 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.861+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c19c0f0 con 0x7f526c07ad10 2023-07-29T16:05:12.519 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.861+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526c19c590 con 0x7f526c07ad10 2023-07-29T16:05:12.520 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.862+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) v1 ==== 45058+0+0 (secure 0 0 0) 0x7f526801a9d0 con 0x7f526c07ad10 2023-07-29T16:05:12.520 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.862+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526c10bea0 con 0x7f526c07ad10 2023-07-29T16:05:12.520 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.862+0000 7f5261ffb700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f5254038380 0x7f525403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:12.520 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.863+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f526804bf60 con 0x7f526c07ad10 2023-07-29T16:05:12.520 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.863+0000 7f5263fff700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f5254038380 0x7f525403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:12.520 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.863+0000 7f5263fff700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f5254038380 0x7f525403a830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f525c006fd0 tx=0x7f525c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:11.868+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5268023b40 con 0x7f526c07ad10 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.072+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f526c10c0c0 con 0x7f526c07ad10 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.073+0000 7f5261ffb700 1 -- 172.21.15.31:0/302594638 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f5268023b40 con 0x7f526c07ad10 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f5254038380 msgr2=0x7f525403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f5254038380 0x7f525403a830 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f525c006fd0 tx=0x7f525c006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 msgr2=0x7f526c19b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c19b9b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5268009fb0 tx=0x7f5268011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:12.522 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 shutdown_connections 2023-07-29T16:05:12.522 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f5254038380 0x7f525403a830 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:12.522 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 --2- 172.21.15.31:0/302594638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f526c07ad10 0x7f526c19b9b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:12.522 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.075+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 >> 172.21.15.31:0/302594638 conn(0x7f526c1011d0 msgr2=0x7f526c101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:12.522 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.076+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 shutdown_connections 2023-07-29T16:05:12.522 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:12.076+0000 7f5272adf700 1 -- 172.21.15.31:0/302594638 wait complete. 2023-07-29T16:05:12.758 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2023-07-29T16:05:12.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:12 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/302594638' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-29T16:05:14.146 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:13 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2118966230' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-29T16:05:14.146 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:13 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: ignoring --setuser ceph since I am not root 2023-07-29T16:05:14.146 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:13 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: ignoring --setgroup ceph since I am not root 2023-07-29T16:05:14.246 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.541+0000 7f0da63bc700 1 Processor -- start 2023-07-29T16:05:14.246 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.541+0000 7f0da63bc700 1 -- start start 2023-07-29T16:05:14.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.541+0000 7f0da63bc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da0105f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:14.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.541+0000 7f0da63bc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da01064d0 con 0x7f0da0105b80 2023-07-29T16:05:14.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.542+0000 7f0d9ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da0105f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:14.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.542+0000 7f0d9ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da0105f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52482/0 (socket says 172.21.15.31:52482) 2023-07-29T16:05:14.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.542+0000 7f0d9ffff700 1 -- 172.21.15.31:0/1913611412 learned_addr learned my addr 172.21.15.31:0/1913611412 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:14.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.542+0000 7f0d9ffff700 1 -- 172.21.15.31:0/1913611412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da0106610 con 0x7f0da0105b80 2023-07-29T16:05:14.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.542+0000 7f0d9ffff700 1 --2- 172.21.15.31:0/1913611412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da0105f90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d900098a0 tx=0x7f0d90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7551b99ebb9dc58c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:14.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0d9effd700 1 -- 172.21.15.31:0/1913611412 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d90004160 con 0x7f0da0105b80 2023-07-29T16:05:14.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0d9effd700 1 -- 172.21.15.31:0/1913611412 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d900042c0 con 0x7f0da0105b80 2023-07-29T16:05:14.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0d9effd700 1 -- 172.21.15.31:0/1913611412 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d90011a00 con 0x7f0da0105b80 2023-07-29T16:05:14.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0da63bc700 1 -- 172.21.15.31:0/1913611412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 msgr2=0x7f0da0105f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:14.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0da63bc700 1 --2- 172.21.15.31:0/1913611412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da0105f90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0d900098a0 tx=0x7f0d90009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:14.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0da63bc700 1 -- 172.21.15.31:0/1913611412 shutdown_connections 2023-07-29T16:05:14.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0da63bc700 1 --2- 172.21.15.31:0/1913611412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da0105f90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:14.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.543+0000 7f0da63bc700 1 -- 172.21.15.31:0/1913611412 >> 172.21.15.31:0/1913611412 conn(0x7f0da01011d0 msgr2=0x7f0da0103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:14.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.544+0000 7f0da63bc700 1 -- 172.21.15.31:0/1913611412 shutdown_connections 2023-07-29T16:05:14.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.544+0000 7f0da63bc700 1 -- 172.21.15.31:0/1913611412 wait complete. 2023-07-29T16:05:14.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.545+0000 7f0da63bc700 1 Processor -- start 2023-07-29T16:05:14.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.545+0000 7f0da63bc700 1 -- start start 2023-07-29T16:05:14.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.546+0000 7f0da63bc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:14.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.546+0000 7f0da63bc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da019bf30 con 0x7f0da0105b80 2023-07-29T16:05:14.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.546+0000 7f0d9ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:14.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.546+0000 7f0d9ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52492/0 (socket says 172.21.15.31:52492) 2023-07-29T16:05:14.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.546+0000 7f0d9ffff700 1 -- 172.21.15.31:0/2118966230 learned_addr learned my addr 172.21.15.31:0/2118966230 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:14.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.546+0000 7f0d9ffff700 1 -- 172.21.15.31:0/2118966230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d90009550 con 0x7f0da0105b80 2023-07-29T16:05:14.253 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.547+0000 7f0d9ffff700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da019b9f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d90006af0 tx=0x7f0d90011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:14.253 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.547+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d90003a40 con 0x7f0da0105b80 2023-07-29T16:05:14.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.547+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d90003ba0 con 0x7f0da0105b80 2023-07-29T16:05:14.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.547+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da019c130 con 0x7f0da0105b80 2023-07-29T16:05:14.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.547+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d9001a6e0 con 0x7f0da0105b80 2023-07-29T16:05:14.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.547+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da019c5d0 con 0x7f0da0105b80 2023-07-29T16:05:14.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.548+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) v1 ==== 45058+0+0 (secure 0 0 0) 0x7f0d90024430 con 0x7f0da0105b80 2023-07-29T16:05:14.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.548+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da0062380 con 0x7f0da0105b80 2023-07-29T16:05:14.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.549+0000 7f0d9d7fa700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f0d880383d0 0x7f0d8803a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:14.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.549+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d9004c170 con 0x7f0da0105b80 2023-07-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.549+0000 7f0d9f7fe700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f0d880383d0 0x7f0d8803a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.549+0000 7f0d9f7fe700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f0d880383d0 0x7f0d8803a880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d94006fd0 tx=0x7f0d94006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.554+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0d900246e0 con 0x7f0da0105b80 2023-07-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.817+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f0da0107a40 con 0x7f0da0105b80 2023-07-29T16:05:14.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.871+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 5) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f0d9002d8c0 con 0x7f0da0105b80 2023-07-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.873+0000 7f0d9d7fa700 1 -- 172.21.15.31:0/2118966230 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f0d90024b10 con 0x7f0da0105b80 2023-07-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f0d880383d0 msgr2=0x7f0d8803a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f0d880383d0 0x7f0d8803a880 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d94006fd0 tx=0x7f0d94006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 msgr2=0x7f0da019b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:14.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da019b9f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0d90006af0 tx=0x7f0d90011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 shutdown_connections 2023-07-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f0d880383d0 0x7f0d8803a880 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 --2- 172.21.15.31:0/2118966230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0da0105b80 0x7f0da019b9f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 >> 172.21.15.31:0/2118966230 conn(0x7f0da01011d0 msgr2=0x7f0da0101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.875+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 shutdown_connections 2023-07-29T16:05:14.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:13.876+0000 7f0da63bc700 1 -- 172.21.15.31:0/2118966230 wait complete. 2023-07-29T16:05:14.396 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:14.149+0000 7f853324e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T16:05:14.396 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:14.260+0000 7f853324e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T16:05:14.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:14.400+0000 7f853324e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T16:05:14.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:14.526+0000 7f853324e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T16:05:15.099 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:14.759+0000 7f853324e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T16:05:15.099 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2118966230' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-29T16:05:15.100 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:14 smithi031 ceph-mon[95504]: mgrmap e5: x(active, since 11s) 2023-07-29T16:05:15.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:15.172+0000 7f853324e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T16:05:15.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:15.297+0000 7f853324e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T16:05:15.654 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:05:15.654 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.799+0000 7f3c6753b700 1 Processor -- start 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.799+0000 7f3c6753b700 1 -- start start 2023-07-29T16:05:15.655 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.800+0000 7f3c6753b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c60108260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:15.656 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.800+0000 7f3c6753b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c601087a0 con 0x7f3c60107e50 2023-07-29T16:05:15.656 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.800+0000 7f3c652d7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c60108260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:15.656 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.800+0000 7f3c652d7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c60108260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52514/0 (socket says 172.21.15.31:52514) 2023-07-29T16:05:15.657 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.800+0000 7f3c652d7700 1 -- 172.21.15.31:0/437345499 learned_addr learned my addr 172.21.15.31:0/437345499 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:15.657 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.801+0000 7f3c652d7700 1 -- 172.21.15.31:0/437345499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c601088e0 con 0x7f3c60107e50 2023-07-29T16:05:15.657 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.801+0000 7f3c652d7700 1 --2- 172.21.15.31:0/437345499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c60108260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0098a0 tx=0x7f3c5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8014e96e398112b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:15.657 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.801+0000 7f3c57fff700 1 -- 172.21.15.31:0/437345499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c5c004160 con 0x7f3c60107e50 2023-07-29T16:05:15.657 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.801+0000 7f3c57fff700 1 -- 172.21.15.31:0/437345499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c5c0042c0 con 0x7f3c60107e50 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.801+0000 7f3c57fff700 1 -- 172.21.15.31:0/437345499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c5c011a00 con 0x7f3c60107e50 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 -- 172.21.15.31:0/437345499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 msgr2=0x7f3c60108260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 --2- 172.21.15.31:0/437345499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c60108260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c0098a0 tx=0x7f3c5c009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 -- 172.21.15.31:0/437345499 shutdown_connections 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 --2- 172.21.15.31:0/437345499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c60108260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 -- 172.21.15.31:0/437345499 >> 172.21.15.31:0/437345499 conn(0x7f3c60103580 msgr2=0x7f3c60105960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:15.658 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 -- 172.21.15.31:0/437345499 shutdown_connections 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.802+0000 7f3c6753b700 1 -- 172.21.15.31:0/437345499 wait complete. 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.803+0000 7f3c6753b700 1 Processor -- start 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.803+0000 7f3c6753b700 1 -- start start 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.803+0000 7f3c6753b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c6019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.803+0000 7f3c6753b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6019bf30 con 0x7f3c60107e50 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.804+0000 7f3c652d7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c6019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:15.659 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.804+0000 7f3c652d7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c6019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52516/0 (socket says 172.21.15.31:52516) 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.804+0000 7f3c652d7700 1 -- 172.21.15.31:0/319568448 learned_addr learned my addr 172.21.15.31:0/319568448 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.804+0000 7f3c652d7700 1 -- 172.21.15.31:0/319568448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c009550 con 0x7f3c60107e50 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.804+0000 7f3c652d7700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c6019b9f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c006af0 tx=0x7f3c5c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.805+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c5c003a40 con 0x7f3c60107e50 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.805+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c5c003ba0 con 0x7f3c60107e50 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.805+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6019c130 con 0x7f3c60107e50 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.805+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c5c01a6e0 con 0x7f3c60107e50 2023-07-29T16:05:15.660 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.805+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6019c5d0 con 0x7f3c60107e50 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.806+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c60062380 con 0x7f3c60107e50 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.806+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f3c5c024430 con 0x7f3c60107e50 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.807+0000 7f3c567fc700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 0x7f3c4c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.807+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3c5c04bfe0 con 0x7f3c60107e50 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.807+0000 7f3c64ad6700 1 -- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 msgr2=0x7f3c4c03a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.807+0000 7f3c64ad6700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 0x7f3c4c03a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-29T16:05:15.661 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:14.812+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3c5c018350 con 0x7f3c60107e50 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.008+0000 7f3c64ad6700 1 -- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 msgr2=0x7f3c4c03a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.008+0000 7f3c64ad6700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 0x7f3c4c03a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.078+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3c601088e0 con 0x7f3c60107e50 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.079+0000 7f3c567fc700 1 -- 172.21.15.31:0/319568448 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f3c601088e0 con 0x7f3c60107e50 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 msgr2=0x7f3c4c03a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 0x7f3c4c03a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:15.662 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 msgr2=0x7f3c6019b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:15.663 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c6019b9f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c006af0 tx=0x7f3c5c011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:15.663 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 shutdown_connections 2023-07-29T16:05:15.663 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7f3c4c038420 0x7f3c4c03a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:15.664 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 --2- 172.21.15.31:0/319568448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c60107e50 0x7f3c6019b9f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:15.664 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.081+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 >> 172.21.15.31:0/319568448 conn(0x7f3c60103580 msgr2=0x7f3c601052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:15.664 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.082+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 shutdown_connections 2023-07-29T16:05:15.664 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:15.082+0000 7f3c6753b700 1 -- 172.21.15.31:0/319568448 wait complete. 2023-07-29T16:05:15.664 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2023-07-29T16:05:15.664 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2023-07-29T16:05:15.874 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:15.463+0000 7f853324e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T16:05:15.874 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:15.592+0000 7f853324e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T16:05:16.170 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:15 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/319568448' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-29T16:05:16.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:16 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:16.174+0000 7f853324e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T16:05:17.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:16 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:16.948+0000 7f853324e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T16:05:17.430 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:17 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:17.158+0000 7f853324e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T16:05:18.430 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:18.122+0000 7f853324e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:05:18.430 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:18.245+0000 7f853324e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T16:05:18.836 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:18.472+0000 7f853324e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:05:19.119 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:18.841+0000 7f853324e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T16:05:19.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:19.122+0000 7f853324e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T16:05:19.430 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:19.319+0000 7f853324e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T16:05:19.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:19.668+0000 7f853324e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T16:05:20.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:20.618+0000 7f853324e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T16:05:20.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:20.796+0000 7f853324e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T16:05:21.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:21.046+0000 7f853324e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T16:05:21.430 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:21.197+0000 7f853324e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T16:05:21.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:21.477+0000 7f853324e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T16:05:22.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:22.569+0000 7f853324e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T16:05:22.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: Active manager daemon x restarted 2023-07-29T16:05:22.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: Activating manager daemon x 2023-07-29T16:05:22.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: osdmap e2: 0 total, 0 up, 0 in 2023-07-29T16:05:22.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: mgrmap e6: x(active, starting, since 0.00427311s) 2023-07-29T16:05:22.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:05:22.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T16:05:22.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T16:05:22.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T16:05:22.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T16:05:22.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: Manager daemon x is now available 2023-07-29T16:05:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:22 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:23.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: Found migration_current of "None". Setting to last migration. 2023-07-29T16:05:23.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:23.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:05:23.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T16:05:23.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:23.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:23.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:23 smithi031 ceph-mon[95504]: mgrmap e7: x(active, since 1.00712s) 2023-07-29T16:05:24.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:05:24.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-29T16:05:24.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-29T16:05:24.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:05:24.051 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.241+0000 7fafbe74b700 1 Processor -- start 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.241+0000 7fafbe74b700 1 -- start start 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.242+0000 7fafbe74b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb80791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.242+0000 7fafbe74b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8079710 con 0x7fafb807acd0 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.242+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb80791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.242+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb80791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52528/0 (socket says 172.21.15.31:52528) 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.242+0000 7fafb7fff700 1 -- 172.21.15.31:0/372654019 learned_addr learned my addr 172.21.15.31:0/372654019 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:24.052 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.242+0000 7fafb7fff700 1 -- 172.21.15.31:0/372654019 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb8079850 con 0x7fafb807acd0 2023-07-29T16:05:24.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.243+0000 7fafb7fff700 1 --2- 172.21.15.31:0/372654019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb80791d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fafa80098a0 tx=0x7fafa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7994106d69368d1b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:24.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.243+0000 7fafb6ffd700 1 -- 172.21.15.31:0/372654019 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa8004160 con 0x7fafb807acd0 2023-07-29T16:05:24.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.243+0000 7fafb6ffd700 1 -- 172.21.15.31:0/372654019 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafa80042c0 con 0x7fafb807acd0 2023-07-29T16:05:24.053 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.243+0000 7fafb6ffd700 1 -- 172.21.15.31:0/372654019 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa8011a00 con 0x7fafb807acd0 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 -- 172.21.15.31:0/372654019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 msgr2=0x7fafb80791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 --2- 172.21.15.31:0/372654019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb80791d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fafa80098a0 tx=0x7fafa8009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 -- 172.21.15.31:0/372654019 shutdown_connections 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 --2- 172.21.15.31:0/372654019 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb80791d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 -- 172.21.15.31:0/372654019 >> 172.21.15.31:0/372654019 conn(0x7fafb81011d0 msgr2=0x7fafb8103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 -- 172.21.15.31:0/372654019 shutdown_connections 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.244+0000 7fafbe74b700 1 -- 172.21.15.31:0/372654019 wait complete. 2023-07-29T16:05:24.054 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.245+0000 7fafbe74b700 1 Processor -- start 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.245+0000 7fafbe74b700 1 -- start start 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.245+0000 7fafbe74b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb819ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.245+0000 7fafbe74b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb819bf50 con 0x7fafb807acd0 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.246+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb819ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.246+0000 7fafb7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb819ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52536/0 (socket says 172.21.15.31:52536) 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.246+0000 7fafb7fff700 1 -- 172.21.15.31:0/3254070213 learned_addr learned my addr 172.21.15.31:0/3254070213 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:24.055 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.246+0000 7fafb7fff700 1 -- 172.21.15.31:0/3254070213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8009550 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.246+0000 7fafb7fff700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb819ba10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafa8006af0 tx=0x7fafa8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.247+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa8003a40 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.247+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafa8003ba0 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.247+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb819c150 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.247+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa801a700 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.247+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb819c5f0 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.248+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) v1 ==== 45069+0+0 (secure 0 0 0) 0x7fafa801a930 con 0x7fafb807acd0 2023-07-29T16:05:24.056 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.248+0000 7fafb57fa700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.248+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 --> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafa003afe0 con 0x7fafa0038420 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.248+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fafa804c240 con 0x7fafb807acd0 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.249+0000 7fafb77fe700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.249+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.450+0000 7fafb77fe700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.450+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-29T16:05:24.057 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.851+0000 7fafb77fe700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:16.851+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:17.652+0000 7fafb77fe700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:17.652+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:19.253+0000 7fafb77fe700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:19.253+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:22.456+0000 7fafb77fe700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/76880357 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:22.456+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-07-29T16:05:24.058 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:22.575+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 6) v1 ==== 44836+0+0 (secure 0 0 0) 0x7fafa802d910 con 0x7fafb807acd0 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:22.575+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 msgr2=0x7fafa003a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:22.575+0000 7fafb57fa700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.577+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 7) v1 ==== 44963+0+0 (secure 0 0 0) 0x7fafa804dc00 con 0x7fafb807acd0 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.578+0000 7fafb57fa700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fafa003bcf0 0x7fafa003e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.578+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafa003afe0 con 0x7fafa003bcf0 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.578+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fafa003bcf0 0x7fafa003e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:24.059 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.580+0000 7fafb77fe700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fafa003bcf0 0x7fafa003e0d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fafac003a10 tx=0x7fafac0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.581+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fafa003afe0 con 0x7fafa003bcf0 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.587+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fafb819cd50 con 0x7fafa003bcf0 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.587+0000 7fafb57fa700 1 -- 172.21.15.31:0/3254070213 <== mgr.14120 v2:172.21.15.31:6800/1090542383 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fafb819cd50 con 0x7fafa003bcf0 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fafa003bcf0 msgr2=0x7fafa003e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fafa003bcf0 0x7fafa003e0d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fafac003a10 tx=0x7fafac0092b0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 msgr2=0x7fafb819ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb819ba10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafa8006af0 tx=0x7fafa8011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.060 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 shutdown_connections 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fafa003bcf0 0x7fafa003e0d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb807acd0 0x7fafb819ba10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 --2- 172.21.15.31:0/3254070213 >> [v2:172.21.15.31:6800/76880357,v1:172.21.15.31:6801/76880357] conn(0x7fafa0038420 0x7fafa003a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 >> 172.21.15.31:0/3254070213 conn(0x7fafb81011d0 msgr2=0x7fafb8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 shutdown_connections 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:23.588+0000 7fafbe74b700 1 -- 172.21.15.31:0/3254070213 wait complete. 2023-07-29T16:05:24.061 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2023-07-29T16:05:24.062 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2023-07-29T16:05:24.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:24 smithi031 ceph-mon[95504]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-29T16:05:24.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:24 smithi031 ceph-mon[95504]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-29T16:05:24.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:24 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:23] ENGINE Bus STARTING 2023-07-29T16:05:24.916 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:24 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:23] ENGINE Serving on https://172.21.15.31:7150 2023-07-29T16:05:24.917 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:24 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:25.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.533+0000 7f490a1f7700 1 Processor -- start 2023-07-29T16:05:25.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.533+0000 7f490a1f7700 1 -- start start 2023-07-29T16:05:25.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.533+0000 7f490a1f7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f4904079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:25.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.533+0000 7f490a1f7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4904079750 con 0x7f490407ad10 2023-07-29T16:05:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.534+0000 7f49037fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f4904079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.534+0000 7f49037fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f4904079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35724/0 (socket says 172.21.15.31:35724) 2023-07-29T16:05:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.534+0000 7f49037fe700 1 -- 172.21.15.31:0/2645993235 learned_addr learned my addr 172.21.15.31:0/2645993235 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.534+0000 7f49037fe700 1 -- 172.21.15.31:0/2645993235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904079890 con 0x7f490407ad10 2023-07-29T16:05:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.534+0000 7f49037fe700 1 --2- 172.21.15.31:0/2645993235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f4904079210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48f40098a0 tx=0x7f48f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2323aa6969e16e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:25.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f49027fc700 1 -- 172.21.15.31:0/2645993235 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f4004160 con 0x7f490407ad10 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f49027fc700 1 -- 172.21.15.31:0/2645993235 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48f40042c0 con 0x7f490407ad10 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f49027fc700 1 -- 172.21.15.31:0/2645993235 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f4011a00 con 0x7f490407ad10 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f490a1f7700 1 -- 172.21.15.31:0/2645993235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 msgr2=0x7f4904079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f490a1f7700 1 --2- 172.21.15.31:0/2645993235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f4904079210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48f40098a0 tx=0x7f48f4009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f490a1f7700 1 -- 172.21.15.31:0/2645993235 shutdown_connections 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f490a1f7700 1 --2- 172.21.15.31:0/2645993235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f4904079210 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:25.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.535+0000 7f490a1f7700 1 -- 172.21.15.31:0/2645993235 >> 172.21.15.31:0/2645993235 conn(0x7f49041011d0 msgr2=0x7f4904103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.536+0000 7f490a1f7700 1 -- 172.21.15.31:0/2645993235 shutdown_connections 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.536+0000 7f490a1f7700 1 -- 172.21.15.31:0/2645993235 wait complete. 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f490a1f7700 1 Processor -- start 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f490a1f7700 1 -- start start 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f490a1f7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f490419ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f490a1f7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419bfa0 con 0x7f490407ad10 2023-07-29T16:05:25.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f49037fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f490419ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f49037fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f490419ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35726/0 (socket says 172.21.15.31:35726) 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.537+0000 7f49037fe700 1 -- 172.21.15.31:0/3272708037 learned_addr learned my addr 172.21.15.31:0/3272708037 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f49037fe700 1 -- 172.21.15.31:0/3272708037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f4009550 con 0x7f490407ad10 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f49037fe700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f490419ba60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006af0 tx=0x7f48f4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f4003a40 con 0x7f490407ad10 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48f4003ba0 con 0x7f490407ad10 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48f401a700 con 0x7f490407ad10 2023-07-29T16:05:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490419c1a0 con 0x7f490407ad10 2023-07-29T16:05:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.538+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490419c640 con 0x7f490407ad10 2023-07-29T16:05:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.539+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) v1 ==== 44963+0+0 (secure 0 0 0) 0x7f48f4024ad0 con 0x7f490407ad10 2023-07-29T16:05:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.539+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490410bea0 con 0x7f490407ad10 2023-07-29T16:05:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.540+0000 7f4900ff9700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f48ec0382f0 0x7f48ec03a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.540+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f48f404c070 con 0x7f490407ad10 2023-07-29T16:05:25.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.540+0000 7f4902ffd700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f48ec0382f0 0x7f48ec03a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:25.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.541+0000 7f4902ffd700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f48ec0382f0 0x7f48ec03a7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f48f8006fd0 tx=0x7f48f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:25.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.545+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f48f407e720 con 0x7f490407ad10 2023-07-29T16:05:25.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.630+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f48f402d910 con 0x7f490407ad10 2023-07-29T16:05:25.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.769+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4904102640 con 0x7f48ec0382f0 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.778+0000 7f4900ff9700 1 -- 172.21.15.31:0/3272708037 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4904102640 con 0x7f48ec0382f0 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.780+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f48ec0382f0 msgr2=0x7f48ec03a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.780+0000 7f490a1f7700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f48ec0382f0 0x7f48ec03a7a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f48f8006fd0 tx=0x7f48f8006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.780+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 msgr2=0x7f490419ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.780+0000 7f490a1f7700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f490419ba60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006af0 tx=0x7f48f4011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.781+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 shutdown_connections 2023-07-29T16:05:25.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.781+0000 7f490a1f7700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f48ec0382f0 0x7f48ec03a7a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:25.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.781+0000 7f490a1f7700 1 --2- 172.21.15.31:0/3272708037 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490407ad10 0x7f490419ba60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:25.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.781+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 >> 172.21.15.31:0/3272708037 conn(0x7f49041011d0 msgr2=0x7f4904101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:25.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.781+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 shutdown_connections 2023-07-29T16:05:25.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:24.781+0000 7f490a1f7700 1 -- 172.21.15.31:0/3272708037 wait complete. 2023-07-29T16:05:25.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:25 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:24] ENGINE Serving on http://172.21.15.31:8765 2023-07-29T16:05:25.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:25 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:24] ENGINE Bus STARTED 2023-07-29T16:05:25.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:25 smithi031 ceph-mon[95504]: mgrmap e8: x(active, since 2s) 2023-07-29T16:05:25.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:25 smithi031 ceph-mon[95504]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:25.882 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:25 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:25.882 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:25 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:26.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-29T16:05:26.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.640+0000 7f15bf383700 1 Processor -- start 2023-07-29T16:05:26.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.640+0000 7f15bf383700 1 -- start start 2023-07-29T16:05:26.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.640+0000 7f15bf383700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b80791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:26.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.640+0000 7f15bf383700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b8079730 con 0x7f15b807acf0 2023-07-29T16:05:26.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.641+0000 7f15bd11f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b80791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:26.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.641+0000 7f15bd11f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b80791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35736/0 (socket says 172.21.15.31:35736) 2023-07-29T16:05:26.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.641+0000 7f15bd11f700 1 -- 172.21.15.31:0/1400179319 learned_addr learned my addr 172.21.15.31:0/1400179319 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.641+0000 7f15bd11f700 1 -- 172.21.15.31:0/1400179319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8079870 con 0x7f15b807acf0 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.641+0000 7f15bd11f700 1 --2- 172.21.15.31:0/1400179319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b80791f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f15b40098a0 tx=0x7f15b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90dbb91acf3eb2f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.641+0000 7f15affff700 1 -- 172.21.15.31:0/1400179319 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b4004160 con 0x7f15b807acf0 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15affff700 1 -- 172.21.15.31:0/1400179319 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15b40042c0 con 0x7f15b807acf0 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15affff700 1 -- 172.21.15.31:0/1400179319 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b4011a00 con 0x7f15b807acf0 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 -- 172.21.15.31:0/1400179319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 msgr2=0x7f15b80791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:26.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 --2- 172.21.15.31:0/1400179319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b80791f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f15b40098a0 tx=0x7f15b4009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 -- 172.21.15.31:0/1400179319 shutdown_connections 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 --2- 172.21.15.31:0/1400179319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b80791f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 -- 172.21.15.31:0/1400179319 >> 172.21.15.31:0/1400179319 conn(0x7f15b81011d0 msgr2=0x7f15b8103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 -- 172.21.15.31:0/1400179319 shutdown_connections 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.642+0000 7f15bf383700 1 -- 172.21.15.31:0/1400179319 wait complete. 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.643+0000 7f15bf383700 1 Processor -- start 2023-07-29T16:05:26.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bf383700 1 -- start start 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bf383700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b819ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bf383700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b819bf80 con 0x7f15b807acf0 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bd11f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b819ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bd11f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b819ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35750/0 (socket says 172.21.15.31:35750) 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bd11f700 1 -- 172.21.15.31:0/1997557680 learned_addr learned my addr 172.21.15.31:0/1997557680 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.644+0000 7f15bd11f700 1 -- 172.21.15.31:0/1997557680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b4009550 con 0x7f15b807acf0 2023-07-29T16:05:26.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.645+0000 7f15bd11f700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b819ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009fb0 tx=0x7f15b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.645+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b4003b70 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.645+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15b4003cd0 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.645+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b819c180 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.645+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b40247a0 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.645+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b819c620 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.646+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f15b401a430 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.646+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b810bec0 con 0x7f15b807acf0 2023-07-29T16:05:26.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.647+0000 7f15ae7fc700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f15a4038380 0x7f15a403a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.647+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f15b404c440 con 0x7f15b807acf0 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.647+0000 7f15bc91e700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f15a4038380 0x7f15a403a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.647+0000 7f15bc91e700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f15a4038380 0x7f15a403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15a8006fd0 tx=0x7f15a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.652+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f15b4018b40 con 0x7f15b807acf0 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.859+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f15b8102620 con 0x7f15a4038380 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.860+0000 7f15ae7fc700 1 -- 172.21.15.31:0/1997557680 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f15b8102620 con 0x7f15a4038380 2023-07-29T16:05:26.321 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f15a4038380 msgr2=0x7f15a403a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f15a4038380 0x7f15a403a830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15a8006fd0 tx=0x7f15a8006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 msgr2=0x7f15b819ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b819ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009fb0 tx=0x7f15b4011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 shutdown_connections 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f15a4038380 0x7f15a403a830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 --2- 172.21.15.31:0/1997557680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15b807acf0 0x7f15b819ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:26.322 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 >> 172.21.15.31:0/1997557680 conn(0x7f15b81011d0 msgr2=0x7f15b8101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:26.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.862+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 shutdown_connections 2023-07-29T16:05:26.323 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:25.863+0000 7f15bf383700 1 -- 172.21.15.31:0/1997557680 wait complete. 2023-07-29T16:05:26.323 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2023-07-29T16:05:26.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:26 smithi031 ceph-mon[95504]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:28.091 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.812+0000 7f8a617f9700 1 Processor -- start 2023-07-29T16:05:28.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.813+0000 7f8a617f9700 1 -- start start 2023-07-29T16:05:28.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.813+0000 7f8a617f9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c0791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:28.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.813+0000 7f8a617f9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c079710 con 0x7f8a5c07acd0 2023-07-29T16:05:28.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.813+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c0791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:28.092 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.813+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c0791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35754/0 (socket says 172.21.15.31:35754) 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.813+0000 7f8a5affd700 1 -- 172.21.15.31:0/3540673109 learned_addr learned my addr 172.21.15.31:0/3540673109 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.814+0000 7f8a5affd700 1 -- 172.21.15.31:0/3540673109 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c079850 con 0x7f8a5c07acd0 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.814+0000 7f8a5affd700 1 --2- 172.21.15.31:0/3540673109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c0791d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0098a0 tx=0x7f8a4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1643ac1a90a1a7a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.814+0000 7f8a59ffb700 1 -- 172.21.15.31:0/3540673109 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c004160 con 0x7f8a5c07acd0 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.814+0000 7f8a59ffb700 1 -- 172.21.15.31:0/3540673109 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a4c0042c0 con 0x7f8a5c07acd0 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.814+0000 7f8a59ffb700 1 -- 172.21.15.31:0/3540673109 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c011a00 con 0x7f8a5c07acd0 2023-07-29T16:05:28.093 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 -- 172.21.15.31:0/3540673109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 msgr2=0x7f8a5c0791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 --2- 172.21.15.31:0/3540673109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c0791d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0098a0 tx=0x7f8a4c009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 -- 172.21.15.31:0/3540673109 shutdown_connections 2023-07-29T16:05:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 --2- 172.21.15.31:0/3540673109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c0791d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 -- 172.21.15.31:0/3540673109 >> 172.21.15.31:0/3540673109 conn(0x7f8a5c1011d0 msgr2=0x7f8a5c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 -- 172.21.15.31:0/3540673109 shutdown_connections 2023-07-29T16:05:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.815+0000 7f8a617f9700 1 -- 172.21.15.31:0/3540673109 wait complete. 2023-07-29T16:05:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.816+0000 7f8a617f9700 1 Processor -- start 2023-07-29T16:05:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.816+0000 7f8a617f9700 1 -- start start 2023-07-29T16:05:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.816+0000 7f8a617f9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.816+0000 7f8a617f9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19bf50 con 0x7f8a5c07acd0 2023-07-29T16:05:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a5affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c19ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35760/0 (socket says 172.21.15.31:35760) 2023-07-29T16:05:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a5affd700 1 -- 172.21.15.31:0/4250203361 learned_addr learned my addr 172.21.15.31:0/4250203361 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:28.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a5affd700 1 -- 172.21.15.31:0/4250203361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c009550 con 0x7f8a5c07acd0 2023-07-29T16:05:28.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a5affd700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c19ba10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c006af0 tx=0x7f8a4c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:28.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c003a40 con 0x7f8a5c07acd0 2023-07-29T16:05:28.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a4c003ba0 con 0x7f8a5c07acd0 2023-07-29T16:05:28.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a4c01a700 con 0x7f8a5c07acd0 2023-07-29T16:05:28.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.817+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c19c150 con 0x7f8a5c07acd0 2023-07-29T16:05:28.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.818+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c19c5f0 con 0x7f8a5c07acd0 2023-07-29T16:05:28.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.819+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f8a4c024ad0 con 0x7f8a5c07acd0 2023-07-29T16:05:28.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.819+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c10bee0 con 0x7f8a5c07acd0 2023-07-29T16:05:28.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.819+0000 7f8a43fff700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f8a440383d0 0x7f8a4403a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:28.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.819+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8a4c04c2f0 con 0x7f8a5c07acd0 2023-07-29T16:05:28.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.819+0000 7f8a5a7fc700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f8a440383d0 0x7f8a4403a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:28.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.820+0000 7f8a5a7fc700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f8a440383d0 0x7f8a4403a880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8a50006fd0 tx=0x7f8a50006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:28.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:26.824+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8a4c018b80 con 0x7f8a5c07acd0 2023-07-29T16:05:28.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.033+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a5c102630 con 0x7f8a440383d0 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.696+0000 7f8a43fff700 1 -- 172.21.15.31:0/4250203361 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8a5c102630 con 0x7f8a440383d0 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.698+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f8a440383d0 msgr2=0x7f8a4403a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.698+0000 7f8a617f9700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f8a440383d0 0x7f8a4403a880 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8a50006fd0 tx=0x7f8a50006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.698+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 msgr2=0x7f8a5c19ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.698+0000 7f8a617f9700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c19ba10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c006af0 tx=0x7f8a4c011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.699+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 shutdown_connections 2023-07-29T16:05:28.102 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.699+0000 7f8a617f9700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f8a440383d0 0x7f8a4403a880 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:28.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.699+0000 7f8a617f9700 1 --2- 172.21.15.31:0/4250203361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a5c07acd0 0x7f8a5c19ba10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:28.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.699+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 >> 172.21.15.31:0/4250203361 conn(0x7f8a5c1011d0 msgr2=0x7f8a5c101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:28.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.699+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 shutdown_connections 2023-07-29T16:05:28.103 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:27.699+0000 7f8a617f9700 1 -- 172.21.15.31:0/4250203361 wait complete. 2023-07-29T16:05:28.111 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: Generating public/private rsa key pair. 2023-07-29T16:05:28.111 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: Your identification has been saved in /tmp/tmp0jtluwey/key. 2023-07-29T16:05:28.112 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: Your public key has been saved in /tmp/tmp0jtluwey/key.pub. 2023-07-29T16:05:28.112 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: The key fingerprint is: 2023-07-29T16:05:28.112 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: SHA256:nnKv2qOzfvmO5kfClHQQj5pHBzOmT2UKCxhWDZjn7wA ceph-858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:05:28.112 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: The key's randomart image is: 2023-07-29T16:05:28.113 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: +---[RSA 3072]----+ 2023-07-29T16:05:28.113 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | o*+o. Boo | 2023-07-29T16:05:28.113 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | .+ ...=.X. | 2023-07-29T16:05:28.113 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | o o.=oo | 2023-07-29T16:05:28.113 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | E . *o. | 2023-07-29T16:05:28.113 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | . .ooS | 2023-07-29T16:05:28.114 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | . .oo.. | 2023-07-29T16:05:28.114 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | o. ++ | 2023-07-29T16:05:28.114 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | o+=o. | 2023-07-29T16:05:28.114 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: | .=O=*+ | 2023-07-29T16:05:28.114 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: +----[SHA256]-----+ 2023-07-29T16:05:28.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:28 smithi031 ceph-mon[95504]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:28.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:28 smithi031 ceph-mon[95504]: Generating ssh key... 2023-07-29T16:05:28.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:28 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:28.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:28 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:29.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:05:29.178 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.570+0000 7f9cec54e700 1 Processor -- start 2023-07-29T16:05:29.178 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.570+0000 7f9cec54e700 1 -- start start 2023-07-29T16:05:29.179 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.571+0000 7f9cec54e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce40791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:29.179 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.571+0000 7f9cec54e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40796f0 con 0x7f9ce407acb0 2023-07-29T16:05:29.180 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.571+0000 7f9cea2ea700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce40791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:29.180 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.571+0000 7f9cea2ea700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce40791b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35762/0 (socket says 172.21.15.31:35762) 2023-07-29T16:05:29.180 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.571+0000 7f9cea2ea700 1 -- 172.21.15.31:0/1772953062 learned_addr learned my addr 172.21.15.31:0/1772953062 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:29.181 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.572+0000 7f9cea2ea700 1 -- 172.21.15.31:0/1772953062 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4079830 con 0x7f9ce407acb0 2023-07-29T16:05:29.181 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.572+0000 7f9cea2ea700 1 --2- 172.21.15.31:0/1772953062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce40791b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00098a0 tx=0x7f9ce0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f4bdee49f370dc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:29.182 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.572+0000 7f9ce92e8700 1 -- 172.21.15.31:0/1772953062 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ce0004160 con 0x7f9ce407acb0 2023-07-29T16:05:29.182 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.572+0000 7f9ce92e8700 1 -- 172.21.15.31:0/1772953062 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ce00042c0 con 0x7f9ce407acb0 2023-07-29T16:05:29.182 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.572+0000 7f9ce92e8700 1 -- 172.21.15.31:0/1772953062 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ce0011a00 con 0x7f9ce407acb0 2023-07-29T16:05:29.183 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 -- 172.21.15.31:0/1772953062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 msgr2=0x7f9ce40791b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:29.183 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 --2- 172.21.15.31:0/1772953062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce40791b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00098a0 tx=0x7f9ce0009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 -- 172.21.15.31:0/1772953062 shutdown_connections 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 --2- 172.21.15.31:0/1772953062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce40791b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 -- 172.21.15.31:0/1772953062 >> 172.21.15.31:0/1772953062 conn(0x7f9ce41011d0 msgr2=0x7f9ce4103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 -- 172.21.15.31:0/1772953062 shutdown_connections 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.573+0000 7f9cec54e700 1 -- 172.21.15.31:0/1772953062 wait complete. 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.574+0000 7f9cec54e700 1 Processor -- start 2023-07-29T16:05:29.184 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.574+0000 7f9cec54e700 1 -- start start 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.574+0000 7f9cec54e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.575+0000 7f9cec54e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce419bf50 con 0x7f9ce407acb0 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.575+0000 7f9cea2ea700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.575+0000 7f9cea2ea700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35778/0 (socket says 172.21.15.31:35778) 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.575+0000 7f9cea2ea700 1 -- 172.21.15.31:0/821122059 learned_addr learned my addr 172.21.15.31:0/821122059 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.575+0000 7f9cea2ea700 1 -- 172.21.15.31:0/821122059 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce0009550 con 0x7f9ce407acb0 2023-07-29T16:05:29.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.575+0000 7f9cea2ea700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce419ba10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0006af0 tx=0x7f9ce0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.576+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ce0003a40 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.576+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce419c150 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.576+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ce0003ba0 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.576+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ce001a700 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.576+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce419c5f0 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.577+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f9ce0024ad0 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.577+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce410bea0 con 0x7f9ce407acb0 2023-07-29T16:05:29.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.578+0000 7f9cdb7fe700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9cd00383d0 0x7f9cd003a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.578+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9ce004c2b0 con 0x7f9ce407acb0 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.578+0000 7f9ce9ae9700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9cd00383d0 0x7f9cd003a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.578+0000 7f9ce9ae9700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9cd00383d0 0x7f9cd003a880 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4006fd0 tx=0x7f9cd4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.582+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9ce0018b80 con 0x7f9ce407acb0 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.791+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ce4102630 con 0x7f9cd00383d0 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.792+0000 7f9cdb7fe700 1 -- 172.21.15.31:0/821122059 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f9ce4102630 con 0x7f9cd00383d0 2023-07-29T16:05:29.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9cd00383d0 msgr2=0x7f9cd003a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9cd00383d0 0x7f9cd003a880 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4006fd0 tx=0x7f9cd4006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 msgr2=0x7f9ce419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce419ba10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0006af0 tx=0x7f9ce0011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 shutdown_connections 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9cd00383d0 0x7f9cd003a880 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 --2- 172.21.15.31:0/821122059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ce407acb0 0x7f9ce419ba10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:29.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 >> 172.21.15.31:0/821122059 conn(0x7f9ce41011d0 msgr2=0x7f9ce4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:29.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.794+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 shutdown_connections 2023-07-29T16:05:29.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:28.795+0000 7f9cec54e700 1 -- 172.21.15.31:0/821122059 wait complete. 2023-07-29T16:05:29.189 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-29T16:05:29.189 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2023-07-29T16:05:29.189 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2023-07-29T16:05:29.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:29 smithi031 ceph-mon[95504]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:30.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:30 smithi031 ceph-mon[95504]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:32.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:31 smithi031 ceph-mon[95504]: Deploying cephadm binary to smithi031 2023-07-29T16:05:33.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2023-07-29T16:05:33.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.707+0000 7efc22eaf700 1 Processor -- start 2023-07-29T16:05:33.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.707+0000 7efc22eaf700 1 -- start start 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.708+0000 7efc22eaf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.708+0000 7efc22eaf700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c079750 con 0x7efc1c07ad10 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.708+0000 7efc20c4b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.708+0000 7efc20c4b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35794/0 (socket says 172.21.15.31:35794) 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.708+0000 7efc20c4b700 1 -- 172.21.15.31:0/286829520 learned_addr learned my addr 172.21.15.31:0/286829520 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.708+0000 7efc20c4b700 1 -- 172.21.15.31:0/286829520 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c079890 con 0x7efc1c07ad10 2023-07-29T16:05:33.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.709+0000 7efc20c4b700 1 --2- 172.21.15.31:0/286829520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c079210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc100098a0 tx=0x7efc10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e736f1e7f62a98 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:33.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.709+0000 7efc1b7fe700 1 -- 172.21.15.31:0/286829520 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc10004160 con 0x7efc1c07ad10 2023-07-29T16:05:33.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.709+0000 7efc1b7fe700 1 -- 172.21.15.31:0/286829520 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc100042c0 con 0x7efc1c07ad10 2023-07-29T16:05:33.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.709+0000 7efc1b7fe700 1 -- 172.21.15.31:0/286829520 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc10011a00 con 0x7efc1c07ad10 2023-07-29T16:05:33.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 -- 172.21.15.31:0/286829520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 msgr2=0x7efc1c079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:33.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 --2- 172.21.15.31:0/286829520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c079210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc100098a0 tx=0x7efc10009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:33.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 -- 172.21.15.31:0/286829520 shutdown_connections 2023-07-29T16:05:33.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 --2- 172.21.15.31:0/286829520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c079210 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:33.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 -- 172.21.15.31:0/286829520 >> 172.21.15.31:0/286829520 conn(0x7efc1c1011d0 msgr2=0x7efc1c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 -- 172.21.15.31:0/286829520 shutdown_connections 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.710+0000 7efc22eaf700 1 -- 172.21.15.31:0/286829520 wait complete. 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.711+0000 7efc22eaf700 1 Processor -- start 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.711+0000 7efc22eaf700 1 -- start start 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.711+0000 7efc22eaf700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.711+0000 7efc22eaf700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c19bfa0 con 0x7efc1c07ad10 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.711+0000 7efc20c4b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:33.477 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc20c4b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c19ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35800/0 (socket says 172.21.15.31:35800) 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc20c4b700 1 -- 172.21.15.31:0/2783508652 learned_addr learned my addr 172.21.15.31:0/2783508652 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc20c4b700 1 -- 172.21.15.31:0/2783508652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10009550 con 0x7efc1c07ad10 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc20c4b700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c19ba60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc10006af0 tx=0x7efc10011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc10003a40 con 0x7efc1c07ad10 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c19c1a0 con 0x7efc1c07ad10 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc10003ba0 con 0x7efc1c07ad10 2023-07-29T16:05:33.478 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.712+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc1001a700 con 0x7efc1c07ad10 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.713+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c19c640 con 0x7efc1c07ad10 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.714+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c10bea0 con 0x7efc1c07ad10 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.714+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7efc10024ad0 con 0x7efc1c07ad10 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.714+0000 7efc19ffb700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7efc0c038420 0x7efc0c03a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.714+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efc1004c270 con 0x7efc1c07ad10 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.715+0000 7efc1bfff700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7efc0c038420 0x7efc0c03a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:33.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.715+0000 7efc1bfff700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7efc0c038420 0x7efc0c03a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc08006fd0 tx=0x7efc08006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:33.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.719+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efc10018b80 con 0x7efc1c07ad10 2023-07-29T16:05:33.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:29.932+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}) v1 -- 0x7efc1c102640 con 0x7efc0c038420 2023-07-29T16:05:33.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.101+0000 7efc19ffb700 1 -- 172.21.15.31:0/2783508652 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7efc1c102640 con 0x7efc0c038420 2023-07-29T16:05:33.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7efc0c038420 msgr2=0x7efc0c03a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:33.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7efc0c038420 0x7efc0c03a8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efc08006fd0 tx=0x7efc08006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 msgr2=0x7efc1c19ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c19ba60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc10006af0 tx=0x7efc10011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 shutdown_connections 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7efc0c038420 0x7efc0c03a8d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 --2- 172.21.15.31:0/2783508652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc1c07ad10 0x7efc1c19ba60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 >> 172.21.15.31:0/2783508652 conn(0x7efc1c1011d0 msgr2=0x7efc1c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:33.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 shutdown_connections 2023-07-29T16:05:33.482 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:33.104+0000 7efc22eaf700 1 -- 172.21.15.31:0/2783508652 wait complete. 2023-07-29T16:05:33.482 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2023-07-29T16:05:34.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:34 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:34.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:34 smithi031 ceph-mon[95504]: Added host smithi031 2023-07-29T16:05:34.385 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:34 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:34.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-29T16:05:34.950 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.012+0000 7f9e9d807700 1 Processor -- start 2023-07-29T16:05:34.951 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.012+0000 7f9e9d807700 1 -- start start 2023-07-29T16:05:34.951 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.013+0000 7f9e9d807700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e98079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:34.951 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.013+0000 7f9e9d807700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98079750 con 0x7f9e9807ad10 2023-07-29T16:05:34.951 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.013+0000 7f9e96ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e98079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:34.951 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.013+0000 7f9e96ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e98079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38446/0 (socket says 172.21.15.31:38446) 2023-07-29T16:05:34.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.013+0000 7f9e96ffd700 1 -- 172.21.15.31:0/2061135959 learned_addr learned my addr 172.21.15.31:0/2061135959 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:34.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.014+0000 7f9e96ffd700 1 -- 172.21.15.31:0/2061135959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e98079890 con 0x7f9e9807ad10 2023-07-29T16:05:34.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.014+0000 7f9e96ffd700 1 --2- 172.21.15.31:0/2061135959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e98079210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098a0 tx=0x7f9e88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ac678852d12e3a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:34.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.014+0000 7f9e95ffb700 1 -- 172.21.15.31:0/2061135959 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e88004160 con 0x7f9e9807ad10 2023-07-29T16:05:34.952 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.014+0000 7f9e95ffb700 1 -- 172.21.15.31:0/2061135959 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e880042c0 con 0x7f9e9807ad10 2023-07-29T16:05:34.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.014+0000 7f9e95ffb700 1 -- 172.21.15.31:0/2061135959 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e88011a00 con 0x7f9e9807ad10 2023-07-29T16:05:34.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 -- 172.21.15.31:0/2061135959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 msgr2=0x7f9e98079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:34.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 --2- 172.21.15.31:0/2061135959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e98079210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9e880098a0 tx=0x7f9e88009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:34.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 -- 172.21.15.31:0/2061135959 shutdown_connections 2023-07-29T16:05:34.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 --2- 172.21.15.31:0/2061135959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e98079210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:34.953 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 -- 172.21.15.31:0/2061135959 >> 172.21.15.31:0/2061135959 conn(0x7f9e981011d0 msgr2=0x7f9e98103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 -- 172.21.15.31:0/2061135959 shutdown_connections 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.015+0000 7f9e9d807700 1 -- 172.21.15.31:0/2061135959 wait complete. 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.016+0000 7f9e9d807700 1 Processor -- start 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.016+0000 7f9e9d807700 1 -- start start 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.016+0000 7f9e9d807700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e9819ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.016+0000 7f9e9d807700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9819bfa0 con 0x7f9e9807ad10 2023-07-29T16:05:34.954 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e96ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e9819ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e96ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e9819ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38450/0 (socket says 172.21.15.31:38450) 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e96ffd700 1 -- 172.21.15.31:0/3933998007 learned_addr learned my addr 172.21.15.31:0/3933998007 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e96ffd700 1 -- 172.21.15.31:0/3933998007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88009550 con 0x7f9e9807ad10 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e96ffd700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e9819ba60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9e88006af0 tx=0x7f9e88011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e88003a40 con 0x7f9e9807ad10 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e88003ba0 con 0x7f9e9807ad10 2023-07-29T16:05:34.955 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9e8801a700 con 0x7f9e9807ad10 2023-07-29T16:05:34.956 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.017+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e9819c1a0 con 0x7f9e9807ad10 2023-07-29T16:05:34.956 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.018+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9819c640 con 0x7f9e9807ad10 2023-07-29T16:05:34.956 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.019+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f9e88024ad0 con 0x7f9e9807ad10 2023-07-29T16:05:34.956 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.019+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e9810bea0 con 0x7f9e9807ad10 2023-07-29T16:05:34.956 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.019+0000 7f9e9c805700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9e80038420 0x7f9e8003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:34.957 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.019+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9e8804c1b0 con 0x7f9e9807ad10 2023-07-29T16:05:34.957 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.020+0000 7f9e967fc700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9e80038420 0x7f9e8003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:34.957 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.020+0000 7f9e967fc700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9e80038420 0x7f9e8003a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c006fd0 tx=0x7f9e8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:34.957 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.025+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9e88018b80 con 0x7f9e9807ad10 2023-07-29T16:05:34.957 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.258+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9e98102640 con 0x7f9e80038420 2023-07-29T16:05:34.957 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.267+0000 7f9e9c805700 1 -- 172.21.15.31:0/3933998007 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9e98102640 con 0x7f9e80038420 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9e80038420 msgr2=0x7f9e8003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9e80038420 0x7f9e8003a8d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c006fd0 tx=0x7f9e8c006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 msgr2=0x7f9e9819ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e9819ba60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9e88006af0 tx=0x7f9e88011ad0 comp rx=0 tx=0).stop 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 shutdown_connections 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9e80038420 0x7f9e8003a8d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:34.958 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 --2- 172.21.15.31:0/3933998007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e9807ad10 0x7f9e9819ba60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:34.959 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.269+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 >> 172.21.15.31:0/3933998007 conn(0x7f9e981011d0 msgr2=0x7f9e98101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:34.959 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.270+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 shutdown_connections 2023-07-29T16:05:34.959 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:34.270+0000 7f9e9d807700 1 -- 172.21.15.31:0/3933998007 wait complete. 2023-07-29T16:05:34.959 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2023-07-29T16:05:35.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:35 smithi031 ceph-mon[95504]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:35 smithi031 ceph-mon[95504]: Saving service mon spec with placement count:5 2023-07-29T16:05:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:35 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:35 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:36.134 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-29T16:05:36.134 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.503+0000 7fcc9b505700 1 Processor -- start 2023-07-29T16:05:36.134 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.503+0000 7fcc9b505700 1 -- start start 2023-07-29T16:05:36.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.503+0000 7fcc9b505700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc940791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:36.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.503+0000 7fcc9b505700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94079730 con 0x7fcc9407acf0 2023-07-29T16:05:36.135 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.504+0000 7fcc992a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc940791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:36.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.504+0000 7fcc992a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc940791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38452/0 (socket says 172.21.15.31:38452) 2023-07-29T16:05:36.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.504+0000 7fcc992a1700 1 -- 172.21.15.31:0/4163042723 learned_addr learned my addr 172.21.15.31:0/4163042723 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:36.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.504+0000 7fcc992a1700 1 -- 172.21.15.31:0/4163042723 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc94079870 con 0x7fcc9407acf0 2023-07-29T16:05:36.136 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.504+0000 7fcc992a1700 1 --2- 172.21.15.31:0/4163042723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc940791f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc900098a0 tx=0x7fcc90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd76af0d7fe65887 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:36.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc8bfff700 1 -- 172.21.15.31:0/4163042723 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc90004160 con 0x7fcc9407acf0 2023-07-29T16:05:36.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc8bfff700 1 -- 172.21.15.31:0/4163042723 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc900042c0 con 0x7fcc9407acf0 2023-07-29T16:05:36.137 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc8bfff700 1 -- 172.21.15.31:0/4163042723 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc900119b0 con 0x7fcc9407acf0 2023-07-29T16:05:36.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 -- 172.21.15.31:0/4163042723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 msgr2=0x7fcc940791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:36.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 --2- 172.21.15.31:0/4163042723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc940791f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc900098a0 tx=0x7fcc90009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:36.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 -- 172.21.15.31:0/4163042723 shutdown_connections 2023-07-29T16:05:36.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 --2- 172.21.15.31:0/4163042723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc940791f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:36.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 -- 172.21.15.31:0/4163042723 >> 172.21.15.31:0/4163042723 conn(0x7fcc941011d0 msgr2=0x7fcc941035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:36.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 -- 172.21.15.31:0/4163042723 shutdown_connections 2023-07-29T16:05:36.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.505+0000 7fcc9b505700 1 -- 172.21.15.31:0/4163042723 wait complete. 2023-07-29T16:05:36.139 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.506+0000 7fcc9b505700 1 Processor -- start 2023-07-29T16:05:36.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc9b505700 1 -- start start 2023-07-29T16:05:36.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc9b505700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc9419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:36.140 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc9b505700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9419bef0 con 0x7fcc9407acf0 2023-07-29T16:05:36.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc992a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc9419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:36.141 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc992a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc9419b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38462/0 (socket says 172.21.15.31:38462) 2023-07-29T16:05:36.142 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc992a1700 1 -- 172.21.15.31:0/1743085311 learned_addr learned my addr 172.21.15.31:0/1743085311 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:36.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.507+0000 7fcc992a1700 1 -- 172.21.15.31:0/1743085311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc90009550 con 0x7fcc9407acf0 2023-07-29T16:05:36.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.508+0000 7fcc992a1700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc9419b9b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc90003e60 tx=0x7fcc90011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:36.143 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.508+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc90003b20 con 0x7fcc9407acf0 2023-07-29T16:05:36.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.508+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc90003c80 con 0x7fcc9407acf0 2023-07-29T16:05:36.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.508+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcc90024760 con 0x7fcc9407acf0 2023-07-29T16:05:36.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.508+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9419c0f0 con 0x7fcc9407acf0 2023-07-29T16:05:36.144 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.508+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc9419c590 con 0x7fcc9407acf0 2023-07-29T16:05:36.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.509+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7fcc9001ab10 con 0x7fcc9407acf0 2023-07-29T16:05:36.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.509+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc9410be90 con 0x7fcc9407acf0 2023-07-29T16:05:36.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.510+0000 7fcc8a7fc700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fcc80038420 0x7fcc8003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:36.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.510+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcc9004c510 con 0x7fcc9407acf0 2023-07-29T16:05:36.145 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.510+0000 7fcc98aa0700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fcc80038420 0x7fcc8003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:36.146 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.510+0000 7fcc98aa0700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fcc80038420 0x7fcc8003a8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc84006fd0 tx=0x7fcc84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:36.146 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.514+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcc90018920 con 0x7fcc9407acf0 2023-07-29T16:05:36.147 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.744+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcc9419f370 con 0x7fcc80038420 2023-07-29T16:05:36.147 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.754+0000 7fcc8a7fc700 1 -- 172.21.15.31:0/1743085311 <== mgr.14120 v2:172.21.15.31:6800/1090542383 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcc9419f370 con 0x7fcc80038420 2023-07-29T16:05:36.147 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fcc80038420 msgr2=0x7fcc8003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:36.148 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fcc80038420 0x7fcc8003a8d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc84006fd0 tx=0x7fcc84006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:36.148 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 msgr2=0x7fcc9419b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:36.149 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc9419b9b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcc90003e60 tx=0x7fcc90011a30 comp rx=0 tx=0).stop 2023-07-29T16:05:36.149 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 shutdown_connections 2023-07-29T16:05:36.149 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fcc80038420 0x7fcc8003a8d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:36.149 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 --2- 172.21.15.31:0/1743085311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc9407acf0 0x7fcc9419b9b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:36.150 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 >> 172.21.15.31:0/1743085311 conn(0x7fcc941011d0 msgr2=0x7fcc94101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:36.150 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.756+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 shutdown_connections 2023-07-29T16:05:36.150 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:35.757+0000 7fcc9b505700 1 -- 172.21.15.31:0/1743085311 wait complete. 2023-07-29T16:05:36.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:36 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:36.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:36 smithi031 ceph-mon[95504]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:36.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:36 smithi031 ceph-mon[95504]: Saving service mgr spec with placement count:2 2023-07-29T16:05:36.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:36 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.674+0000 7fd980025700 1 Processor -- start 2023-07-29T16:05:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.674+0000 7fd980025700 1 -- start start 2023-07-29T16:05:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.674+0000 7fd980025700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd978105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.674+0000 7fd980025700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9781064e0 con 0x7fd978105b90 2023-07-29T16:05:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.675+0000 7fd97ddc1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd978105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:37.310 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.675+0000 7fd97ddc1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd978105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38476/0 (socket says 172.21.15.31:38476) 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.675+0000 7fd97ddc1700 1 -- 172.21.15.31:0/2221301571 learned_addr learned my addr 172.21.15.31:0/2221301571 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.675+0000 7fd97ddc1700 1 -- 172.21.15.31:0/2221301571 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978106620 con 0x7fd978105b90 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.675+0000 7fd97ddc1700 1 --2- 172.21.15.31:0/2221301571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd978105fa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd9740098a0 tx=0x7fd974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee88c09f6e1d136b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.675+0000 7fd97cdbf700 1 -- 172.21.15.31:0/2221301571 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd974004160 con 0x7fd978105b90 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd97cdbf700 1 -- 172.21.15.31:0/2221301571 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9740042c0 con 0x7fd978105b90 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd97cdbf700 1 -- 172.21.15.31:0/2221301571 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd9740119b0 con 0x7fd978105b90 2023-07-29T16:05:37.311 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 -- 172.21.15.31:0/2221301571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 msgr2=0x7fd978105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 --2- 172.21.15.31:0/2221301571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd978105fa0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd9740098a0 tx=0x7fd974009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 -- 172.21.15.31:0/2221301571 shutdown_connections 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 --2- 172.21.15.31:0/2221301571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd978105fa0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 -- 172.21.15.31:0/2221301571 >> 172.21.15.31:0/2221301571 conn(0x7fd978101200 msgr2=0x7fd978103610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 -- 172.21.15.31:0/2221301571 shutdown_connections 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.676+0000 7fd980025700 1 -- 172.21.15.31:0/2221301571 wait complete. 2023-07-29T16:05:37.312 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.677+0000 7fd980025700 1 Processor -- start 2023-07-29T16:05:37.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd980025700 1 -- start start 2023-07-29T16:05:37.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd980025700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd97819b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:37.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd980025700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97819bf20 con 0x7fd978105b90 2023-07-29T16:05:37.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd97ddc1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd97819b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:37.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd97ddc1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd97819b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38486/0 (socket says 172.21.15.31:38486) 2023-07-29T16:05:37.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd97ddc1700 1 -- 172.21.15.31:0/1485490160 learned_addr learned my addr 172.21.15.31:0/1485490160 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:37.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.678+0000 7fd97ddc1700 1 -- 172.21.15.31:0/1485490160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974009550 con 0x7fd978105b90 2023-07-29T16:05:37.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.679+0000 7fd97ddc1700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd97819b9e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd974011980 tx=0x7fd974011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:37.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.679+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd974003a60 con 0x7fd978105b90 2023-07-29T16:05:37.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.679+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd974003bc0 con 0x7fd978105b90 2023-07-29T16:05:37.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.679+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd97401a670 con 0x7fd978105b90 2023-07-29T16:05:37.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.679+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97819c120 con 0x7fd978105b90 2023-07-29T16:05:37.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.679+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97819c5c0 con 0x7fd978105b90 2023-07-29T16:05:37.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.680+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7fd974024a00 con 0x7fd978105b90 2023-07-29T16:05:37.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.680+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd978062380 con 0x7fd978105b90 2023-07-29T16:05:37.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.681+0000 7fd96effd700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fd964038420 0x7fd96403a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:37.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.681+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd97404c390 con 0x7fd978105b90 2023-07-29T16:05:37.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.681+0000 7fd97d5c0700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fd964038420 0x7fd96403a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:37.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.681+0000 7fd97d5c0700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fd964038420 0x7fd96403a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd968006fd0 tx=0x7fd968006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:37.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.686+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd974024440 con 0x7fd978105b90 2023-07-29T16:05:37.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.901+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd978107a90 con 0x7fd978105b90 2023-07-29T16:05:37.318 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.905+0000 7fd96effd700 1 -- 172.21.15.31:0/1485490160 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd974018350 con 0x7fd978105b90 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.907+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fd964038420 msgr2=0x7fd96403a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.907+0000 7fd980025700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fd964038420 0x7fd96403a8d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd968006fd0 tx=0x7fd968006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.907+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 msgr2=0x7fd97819b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.907+0000 7fd980025700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd97819b9e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd974011980 tx=0x7fd974011a60 comp rx=0 tx=0).stop 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.907+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 shutdown_connections 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.908+0000 7fd980025700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fd964038420 0x7fd96403a8d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:37.319 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.908+0000 7fd980025700 1 --2- 172.21.15.31:0/1485490160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd978105b90 0x7fd97819b9e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:37.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.908+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 >> 172.21.15.31:0/1485490160 conn(0x7fd978101200 msgr2=0x7fd978101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:37.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.908+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 shutdown_connections 2023-07-29T16:05:37.320 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:36.908+0000 7fd980025700 1 -- 172.21.15.31:0/1485490160 wait complete. 2023-07-29T16:05:38.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:37 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1485490160' entity='client.admin' 2023-07-29T16:05:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.031+0000 7fda1bfd7700 1 Processor -- start 2023-07-29T16:05:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.032+0000 7fda1bfd7700 1 -- start start 2023-07-29T16:05:38.699 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.032+0000 7fda1bfd7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda140791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.032+0000 7fda1bfd7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda14079710 con 0x7fda1407acd0 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.032+0000 7fda19d73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda140791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.033+0000 7fda19d73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda140791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38488/0 (socket says 172.21.15.31:38488) 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.033+0000 7fda19d73700 1 -- 172.21.15.31:0/3200528417 learned_addr learned my addr 172.21.15.31:0/3200528417 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.033+0000 7fda19d73700 1 -- 172.21.15.31:0/3200528417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14079850 con 0x7fda1407acd0 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.033+0000 7fda19d73700 1 --2- 172.21.15.31:0/3200528417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda140791d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fda100098a0 tx=0x7fda10009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cadb5ca2e71db7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:38.700 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.033+0000 7fda18d71700 1 -- 172.21.15.31:0/3200528417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda10004160 con 0x7fda1407acd0 2023-07-29T16:05:38.701 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda18d71700 1 -- 172.21.15.31:0/3200528417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda100042c0 con 0x7fda1407acd0 2023-07-29T16:05:38.705 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda18d71700 1 -- 172.21.15.31:0/3200528417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda100119b0 con 0x7fda1407acd0 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3200528417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 msgr2=0x7fda140791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda1bfd7700 1 --2- 172.21.15.31:0/3200528417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda140791d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fda100098a0 tx=0x7fda10009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3200528417 shutdown_connections 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda1bfd7700 1 --2- 172.21.15.31:0/3200528417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda140791d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.034+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3200528417 >> 172.21.15.31:0/3200528417 conn(0x7fda141011d0 msgr2=0x7fda14103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.035+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3200528417 shutdown_connections 2023-07-29T16:05:38.706 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.035+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3200528417 wait complete. 2023-07-29T16:05:38.707 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.036+0000 7fda1bfd7700 1 Processor -- start 2023-07-29T16:05:38.707 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.036+0000 7fda1bfd7700 1 -- start start 2023-07-29T16:05:38.707 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.037+0000 7fda1bfd7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda1419ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.037+0000 7fda1bfd7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1419bf60 con 0x7fda1407acd0 2023-07-29T16:05:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.037+0000 7fda19d73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda1419ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.037+0000 7fda19d73700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda1419ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38502/0 (socket says 172.21.15.31:38502) 2023-07-29T16:05:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.037+0000 7fda19d73700 1 -- 172.21.15.31:0/3979639780 learned_addr learned my addr 172.21.15.31:0/3979639780 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.037+0000 7fda19d73700 1 -- 172.21.15.31:0/3979639780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda10009550 con 0x7fda1407acd0 2023-07-29T16:05:38.709 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.038+0000 7fda19d73700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda1419ba20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fda10000c00 tx=0x7fda10011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.038+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda100039f0 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.038+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda10003b50 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.038+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1419c160 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.038+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fda1001a780 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.038+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1419c600 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.039+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7fda1001a9f0 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.040+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1410bf30 con 0x7fda1407acd0 2023-07-29T16:05:38.710 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.040+0000 7fda0affd700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fda00038420 0x7fda0003a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:38.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.040+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fda1004c230 con 0x7fda1407acd0 2023-07-29T16:05:38.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.040+0000 7fda19572700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fda00038420 0x7fda0003a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:38.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.041+0000 7fda19572700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fda00038420 0x7fda0003a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fda04006fd0 tx=0x7fda04006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:38.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.046+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fda1001aca0 con 0x7fda1407acd0 2023-07-29T16:05:38.711 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.264+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fda1410c120 con 0x7fda1407acd0 2023-07-29T16:05:38.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.270+0000 7fda0affd700 1 -- 172.21.15.31:0/3979639780 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fda1002c350 con 0x7fda1407acd0 2023-07-29T16:05:38.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fda00038420 msgr2=0x7fda0003a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:38.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fda00038420 0x7fda0003a8d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fda04006fd0 tx=0x7fda04006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:38.712 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 msgr2=0x7fda1419ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda1419ba20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fda10000c00 tx=0x7fda10011a30 comp rx=0 tx=0).stop 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 shutdown_connections 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fda00038420 0x7fda0003a8d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 --2- 172.21.15.31:0/3979639780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda1407acd0 0x7fda1419ba20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 >> 172.21.15.31:0/3979639780 conn(0x7fda141011d0 msgr2=0x7fda14101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 shutdown_connections 2023-07-29T16:05:38.713 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:38.273+0000 7fda1bfd7700 1 -- 172.21.15.31:0/3979639780 wait complete. 2023-07-29T16:05:38.714 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2023-07-29T16:05:38.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:38 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:38.961 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:38 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3979639780' entity='client.admin' 2023-07-29T16:05:40.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-mon[95504]: from='mgr.14120 172.21.15.31:0/3624522615' entity='mgr.x' 2023-07-29T16:05:40.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1357023417' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-29T16:05:40.612 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: ignoring --setuser ceph since I am not root 2023-07-29T16:05:40.612 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: ignoring --setgroup ceph since I am not root 2023-07-29T16:05:40.896 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:40.711+0000 7f34a74f3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T16:05:40.897 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:40.820+0000 7f34a74f3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T16:05:40.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.243+0000 7f95a32e9700 1 Processor -- start 2023-07-29T16:05:40.991 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.244+0000 7f95a32e9700 1 -- start start 2023-07-29T16:05:40.991 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.244+0000 7f95a32e9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c079210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:40.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.244+0000 7f95a32e9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c079750 con 0x7f959c07ad10 2023-07-29T16:05:40.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.244+0000 7f95a1085700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c079210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:40.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f95a1085700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c079210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38510/0 (socket says 172.21.15.31:38510) 2023-07-29T16:05:40.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f95a1085700 1 -- 172.21.15.31:0/4141461703 learned_addr learned my addr 172.21.15.31:0/4141461703 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:40.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f95a1085700 1 -- 172.21.15.31:0/4141461703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c079890 con 0x7f959c07ad10 2023-07-29T16:05:40.992 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f95a1085700 1 --2- 172.21.15.31:0/4141461703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c079210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95980098a0 tx=0x7f9598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbff751310c3a02b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f9593fff700 1 -- 172.21.15.31:0/4141461703 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9598004160 con 0x7f959c07ad10 2023-07-29T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f9593fff700 1 -- 172.21.15.31:0/4141461703 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95980042c0 con 0x7f959c07ad10 2023-07-29T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.245+0000 7f9593fff700 1 -- 172.21.15.31:0/4141461703 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95980119b0 con 0x7f959c07ad10 2023-07-29T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 -- 172.21.15.31:0/4141461703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 msgr2=0x7f959c079210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 --2- 172.21.15.31:0/4141461703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c079210 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f95980098a0 tx=0x7f9598009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:40.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 -- 172.21.15.31:0/4141461703 shutdown_connections 2023-07-29T16:05:40.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 --2- 172.21.15.31:0/4141461703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c079210 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:40.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 -- 172.21.15.31:0/4141461703 >> 172.21.15.31:0/4141461703 conn(0x7f959c1011d0 msgr2=0x7f959c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:40.994 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 -- 172.21.15.31:0/4141461703 shutdown_connections 2023-07-29T16:05:40.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.246+0000 7f95a32e9700 1 -- 172.21.15.31:0/4141461703 wait complete. 2023-07-29T16:05:40.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.247+0000 7f95a32e9700 1 Processor -- start 2023-07-29T16:05:40.995 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.247+0000 7f95a32e9700 1 -- start start 2023-07-29T16:05:40.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a32e9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:40.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a32e9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c19bfb0 con 0x7f959c07ad10 2023-07-29T16:05:40.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a1085700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:40.996 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a1085700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c19ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38520/0 (socket says 172.21.15.31:38520) 2023-07-29T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a1085700 1 -- 172.21.15.31:0/1357023417 learned_addr learned my addr 172.21.15.31:0/1357023417 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a1085700 1 -- 172.21.15.31:0/1357023417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009550 con 0x7f959c07ad10 2023-07-29T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.248+0000 7f95a1085700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c19ba70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f9598011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:40.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.249+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9598003ac0 con 0x7f959c07ad10 2023-07-29T16:05:40.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.249+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959c19c1b0 con 0x7f959c07ad10 2023-07-29T16:05:40.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.249+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9598003c20 con 0x7f959c07ad10 2023-07-29T16:05:40.998 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.249+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f959801a720 con 0x7f959c07ad10 2023-07-29T16:05:40.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.249+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959c19c650 con 0x7f959c07ad10 2023-07-29T16:05:40.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.250+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) v1 ==== 45069+0+0 (secure 0 0 0) 0x7f9598024a90 con 0x7f959c07ad10 2023-07-29T16:05:40.999 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.250+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f959c10bef0 con 0x7f959c07ad10 2023-07-29T16:05:41.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.250+0000 7f95927fc700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9588038420 0x7f958803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:41.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.251+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f959804c3c0 con 0x7f959c07ad10 2023-07-29T16:05:41.000 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.251+0000 7f95a0884700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9588038420 0x7f958803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:41.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.251+0000 7f95a0884700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9588038420 0x7f958803a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f958c006fd0 tx=0x7f958c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:41.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.256+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9598024440 con 0x7f959c07ad10 2023-07-29T16:05:41.003 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:39.526+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f959c10c110 con 0x7f959c07ad10 2023-07-29T16:05:41.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.360+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 9) v1 ==== 45082+0+0 (secure 0 0 0) 0x7f959802d910 con 0x7f959c07ad10 2023-07-29T16:05:41.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.362+0000 7f95927fc700 1 -- 172.21.15.31:0/1357023417 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f959800b850 con 0x7f959c07ad10 2023-07-29T16:05:41.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9588038420 msgr2=0x7f958803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:41.004 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9588038420 0x7f958803a8d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f958c006fd0 tx=0x7f958c006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 msgr2=0x7f959c19ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c19ba70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9598000c00 tx=0x7f9598011a80 comp rx=0 tx=0).stop 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 shutdown_connections 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f9588038420 0x7f958803a8d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.366+0000 7f95a32e9700 1 --2- 172.21.15.31:0/1357023417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f959c07ad10 0x7f959c19ba70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.367+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 >> 172.21.15.31:0/1357023417 conn(0x7f959c1011d0 msgr2=0x7f959c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:41.005 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.367+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 shutdown_connections 2023-07-29T16:05:41.006 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:40.367+0000 7f95a32e9700 1 -- 172.21.15.31:0/1357023417 wait complete. 2023-07-29T16:05:41.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:40 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:40.950+0000 7f34a74f3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T16:05:41.180 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:41 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:41.091+0000 7f34a74f3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T16:05:41.679 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:41 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1357023417' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-29T16:05:41.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:41 smithi031 ceph-mon[95504]: mgrmap e9: x(active, since 17s) 2023-07-29T16:05:41.680 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:41 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:41.321+0000 7f34a74f3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T16:05:42.029 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:41 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:41.744+0000 7f34a74f3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T16:05:42.029 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:41 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:41.864+0000 7f34a74f3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T16:05:42.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:05:42.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-29T16:05:42.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-29T16:05:42.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.734+0000 7f7d36f3b700 1 Processor -- start 2023-07-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.734+0000 7f7d36f3b700 1 -- start start 2023-07-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.734+0000 7f7d36f3b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d300791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:42.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.734+0000 7f7d36f3b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30079730 con 0x7f7d3007acf0 2023-07-29T16:05:42.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.735+0000 7f7d34cd7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d300791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:42.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.735+0000 7f7d34cd7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d300791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57326/0 (socket says 172.21.15.31:57326) 2023-07-29T16:05:42.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.735+0000 7f7d34cd7700 1 -- 172.21.15.31:0/1131670032 learned_addr learned my addr 172.21.15.31:0/1131670032 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.735+0000 7f7d34cd7700 1 -- 172.21.15.31:0/1131670032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d30079870 con 0x7f7d3007acf0 2023-07-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.736+0000 7f7d34cd7700 1 --2- 172.21.15.31:0/1131670032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d300791f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d240098a0 tx=0x7f7d24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689217787c3f25d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.736+0000 7f7d2f7fe700 1 -- 172.21.15.31:0/1131670032 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d24004160 con 0x7f7d3007acf0 2023-07-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.736+0000 7f7d2f7fe700 1 -- 172.21.15.31:0/1131670032 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d240042c0 con 0x7f7d3007acf0 2023-07-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.736+0000 7f7d2f7fe700 1 -- 172.21.15.31:0/1131670032 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d240119b0 con 0x7f7d3007acf0 2023-07-29T16:05:42.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.736+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1131670032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 msgr2=0x7f7d300791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.736+0000 7f7d36f3b700 1 --2- 172.21.15.31:0/1131670032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d300791f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7d240098a0 tx=0x7f7d24009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.737+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1131670032 shutdown_connections 2023-07-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.737+0000 7f7d36f3b700 1 --2- 172.21.15.31:0/1131670032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d300791f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:42.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.737+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1131670032 >> 172.21.15.31:0/1131670032 conn(0x7f7d301011d0 msgr2=0x7f7d30103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.737+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1131670032 shutdown_connections 2023-07-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.737+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1131670032 wait complete. 2023-07-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d36f3b700 1 Processor -- start 2023-07-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d36f3b700 1 -- start start 2023-07-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d36f3b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d3019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:42.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d36f3b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3019bf90 con 0x7f7d3007acf0 2023-07-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d34cd7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d3019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d34cd7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d3019ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57342/0 (socket says 172.21.15.31:57342) 2023-07-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.738+0000 7f7d34cd7700 1 -- 172.21.15.31:0/1222667789 learned_addr learned my addr 172.21.15.31:0/1222667789 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.739+0000 7f7d34cd7700 1 -- 172.21.15.31:0/1222667789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d24009550 con 0x7f7d3007acf0 2023-07-29T16:05:42.426 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.739+0000 7f7d34cd7700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d3019ba50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d24003e60 tx=0x7f7d24011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:42.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.739+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d24003b20 con 0x7f7d3007acf0 2023-07-29T16:05:42.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.739+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d24003c80 con 0x7f7d3007acf0 2023-07-29T16:05:42.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.739+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d24024760 con 0x7f7d3007acf0 2023-07-29T16:05:42.429 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.739+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3019c190 con 0x7f7d3007acf0 2023-07-29T16:05:42.430 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.740+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3019c630 con 0x7f7d3007acf0 2023-07-29T16:05:42.430 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.741+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) v1 ==== 45082+0+0 (secure 0 0 0) 0x7f7d2401ab10 con 0x7f7d3007acf0 2023-07-29T16:05:42.430 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.741+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3010bf10 con 0x7f7d3007acf0 2023-07-29T16:05:42.430 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.741+0000 7f7d2dffb700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 0x7f7d2003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:42.430 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.742+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7d2404c490 con 0x7f7d3007acf0 2023-07-29T16:05:42.431 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.742+0000 7f7d2ffff700 1 -- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 msgr2=0x7f7d2003a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:42.431 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.742+0000 7f7d2ffff700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 0x7f7d2003a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-29T16:05:42.431 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.747+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7d24018920 con 0x7f7d3007acf0 2023-07-29T16:05:42.431 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.943+0000 7f7d2ffff700 1 -- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 msgr2=0x7f7d2003a920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:42.431 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:41.943+0000 7f7d2ffff700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 0x7f7d2003a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-29T16:05:42.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.010+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7d3010c130 con 0x7f7d3007acf0 2023-07-29T16:05:42.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.010+0000 7f7d2dffb700 1 -- 172.21.15.31:0/1222667789 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f7d24018350 con 0x7f7d3007acf0 2023-07-29T16:05:42.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 msgr2=0x7f7d2003a920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:05:42.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 0x7f7d2003a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:42.432 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 msgr2=0x7f7d3019ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:42.433 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:42.034+0000 7f34a74f3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T16:05:42.433 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:42.169+0000 7f34a74f3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T16:05:42.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d3019ba50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d24003e60 tx=0x7f7d24011a30 comp rx=0 tx=0).stop 2023-07-29T16:05:42.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 shutdown_connections 2023-07-29T16:05:42.434 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7f7d20038470 0x7f7d2003a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:42.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 --2- 172.21.15.31:0/1222667789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d3007acf0 0x7f7d3019ba50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:42.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 >> 172.21.15.31:0/1222667789 conn(0x7f7d301011d0 msgr2=0x7f7d30101f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:42.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 shutdown_connections 2023-07-29T16:05:42.435 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.013+0000 7f7d36f3b700 1 -- 172.21.15.31:0/1222667789 wait complete. 2023-07-29T16:05:42.435 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2023-07-29T16:05:42.435 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2023-07-29T16:05:42.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:42 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1222667789' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-29T16:05:42.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:42.745+0000 7f34a74f3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T16:05:43.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:43 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:43.524+0000 7f34a74f3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T16:05:43.930 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:43 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:43.737+0000 7f34a74f3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T16:05:45.056 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:44 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:44.706+0000 7f34a74f3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:05:45.056 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:44 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:44.830+0000 7f34a74f3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T16:05:45.427 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:45 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:45.060+0000 7f34a74f3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:05:45.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:45 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:45.433+0000 7f34a74f3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T16:05:46.179 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:45 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:45.717+0000 7f34a74f3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T16:05:46.180 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:45 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:45.915+0000 7f34a74f3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T16:05:46.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:46 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:46.268+0000 7f34a74f3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T16:05:47.655 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:47 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:47.227+0000 7f34a74f3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T16:05:47.656 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:47 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:47.407+0000 7f34a74f3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T16:05:47.929 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:47 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:47.660+0000 7f34a74f3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T16:05:47.930 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:47 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:47.812+0000 7f34a74f3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T16:05:48.429 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:48 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:48.095+0000 7f34a74f3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T16:05:49.679 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:05:49.189+0000 7f34a74f3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: Active manager daemon x restarted 2023-07-29T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: Activating manager daemon x 2023-07-29T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: osdmap e3: 0 total, 0 up, 0 in 2023-07-29T16:05:49.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: mgrmap e10: x(active, starting, since 0.00390002s) 2023-07-29T16:05:49.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:05:49.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-29T16:05:49.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-29T16:05:49.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-29T16:05:49.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-29T16:05:49.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: Manager daemon x is now available 2023-07-29T16:05:49.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:49.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:05:50.619 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-07-29T16:05:50.619 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-29T16:05:50.619 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-29T16:05:50.619 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-07-29T16:05:50.619 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.941+0000 7fce6830c700 1 Processor -- start 2023-07-29T16:05:50.624 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.941+0000 7fce6830c700 1 -- start start 2023-07-29T16:05:50.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.942+0000 7fce6830c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce60105f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:50.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.942+0000 7fce6830c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce601064d0 con 0x7fce60105b80 2023-07-29T16:05:50.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.942+0000 7fce660a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce60105f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:50.625 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.942+0000 7fce660a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce60105f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57346/0 (socket says 172.21.15.31:57346) 2023-07-29T16:05:50.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.942+0000 7fce660a8700 1 -- 172.21.15.31:0/509170992 learned_addr learned my addr 172.21.15.31:0/509170992 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:50.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.942+0000 7fce660a8700 1 -- 172.21.15.31:0/509170992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce60106610 con 0x7fce60105b80 2023-07-29T16:05:50.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.943+0000 7fce660a8700 1 --2- 172.21.15.31:0/509170992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce60105f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0098a0 tx=0x7fce5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8da58228a502b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:50.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.943+0000 7fce650a6700 1 -- 172.21.15.31:0/509170992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c004160 con 0x7fce60105b80 2023-07-29T16:05:50.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.943+0000 7fce650a6700 1 -- 172.21.15.31:0/509170992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce5c0042c0 con 0x7fce60105b80 2023-07-29T16:05:50.626 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.943+0000 7fce650a6700 1 -- 172.21.15.31:0/509170992 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c0119b0 con 0x7fce60105b80 2023-07-29T16:05:50.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.943+0000 7fce6830c700 1 -- 172.21.15.31:0/509170992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 msgr2=0x7fce60105f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:50.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.943+0000 7fce6830c700 1 --2- 172.21.15.31:0/509170992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce60105f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0098a0 tx=0x7fce5c009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.627 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.944+0000 7fce6830c700 1 -- 172.21.15.31:0/509170992 shutdown_connections 2023-07-29T16:05:50.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.944+0000 7fce6830c700 1 --2- 172.21.15.31:0/509170992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce60105f90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.944+0000 7fce6830c700 1 -- 172.21.15.31:0/509170992 >> 172.21.15.31:0/509170992 conn(0x7fce601011d0 msgr2=0x7fce60103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:50.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.944+0000 7fce6830c700 1 -- 172.21.15.31:0/509170992 shutdown_connections 2023-07-29T16:05:50.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.944+0000 7fce6830c700 1 -- 172.21.15.31:0/509170992 wait complete. 2023-07-29T16:05:50.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.945+0000 7fce6830c700 1 Processor -- start 2023-07-29T16:05:50.628 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.945+0000 7fce6830c700 1 -- start start 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.945+0000 7fce6830c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce6019b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.945+0000 7fce6830c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6019bee0 con 0x7fce60105b80 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.946+0000 7fce660a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce6019b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.946+0000 7fce660a8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce6019b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57358/0 (socket says 172.21.15.31:57358) 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.946+0000 7fce660a8700 1 -- 172.21.15.31:0/1530552919 learned_addr learned my addr 172.21.15.31:0/1530552919 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.946+0000 7fce660a8700 1 -- 172.21.15.31:0/1530552919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c009550 con 0x7fce60105b80 2023-07-29T16:05:50.629 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.946+0000 7fce660a8700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce6019b9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fce5c003e60 tx=0x7fce5c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.947+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c003970 con 0x7fce60105b80 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.947+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6019c0e0 con 0x7fce60105b80 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.947+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce5c003ad0 con 0x7fce60105b80 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.947+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce5c0195e0 con 0x7fce60105b80 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.947+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6019c580 con 0x7fce60105b80 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.948+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) v1 ==== 45082+0+0 (secure 0 0 0) 0x7fce5c0198a0 con 0x7fce60105b80 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.949+0000 7fce577fe700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:50.630 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.949+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 --> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce4c03af90 con 0x7fce4c0383d0 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.949+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce5c04c460 con 0x7fce60105b80 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.949+0000 7fce658a7700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:42.949+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:43.150+0000 7fce658a7700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:43.150+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:43.551+0000 7fce658a7700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:50.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:43.551+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:44.352+0000 7fce658a7700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:44.352+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:45.953+0000 7fce658a7700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:45.954+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:49.157+0000 7fce658a7700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/1090542383 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:49.157+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:49.194+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 10) v1 ==== 44849+0+0 (secure 0 0 0) 0x7fce5c019ba0 con 0x7fce60105b80 2023-07-29T16:05:50.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:49.194+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 msgr2=0x7fce4c03a880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:49.194+0000 7fce577fe700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.197+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 11) v1 ==== 44976+0+0 (secure 0 0 0) 0x7fce5c04d9d0 con 0x7fce60105b80 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.197+0000 7fce577fe700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce4c03be30 0x7fce4c03e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.197+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce4c03af90 con 0x7fce4c03be30 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.198+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce4c03be30 0x7fce4c03e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.199+0000 7fce658a7700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce4c03be30 0x7fce4c03e210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fce50003a10 tx=0x7fce500092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:50.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.200+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fce4c03af90 con 0x7fce4c03be30 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.206+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fce6019cce0 con 0x7fce4c03be30 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.206+0000 7fce577fe700 1 -- 172.21.15.31:0/1530552919 <== mgr.14152 v2:172.21.15.31:6800/4163929785 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fce6019cce0 con 0x7fce4c03be30 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce4c03be30 msgr2=0x7fce4c03e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce4c03be30 0x7fce4c03e210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fce50003a10 tx=0x7fce500092b0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 msgr2=0x7fce6019b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce6019b9a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fce5c003e60 tx=0x7fce5c011aa0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 shutdown_connections 2023-07-29T16:05:50.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce4c03be30 0x7fce4c03e210 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce60105b80 0x7fce6019b9a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 --2- 172.21.15.31:0/1530552919 >> [v2:172.21.15.31:6800/1090542383,v1:172.21.15.31:6801/1090542383] conn(0x7fce4c0383d0 0x7fce4c03a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 >> 172.21.15.31:0/1530552919 conn(0x7fce601011d0 msgr2=0x7fce60101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 shutdown_connections 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:50.207+0000 7fce6830c700 1 -- 172.21.15.31:0/1530552919 wait complete. 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2023-07-29T16:05:50.635 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2023-07-29T16:05:50.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-29T16:05:50.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-29T16:05:50.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:50 smithi031 ceph-mon[95504]: mgrmap e11: x(active, since 1.00795s) 2023-07-29T16:05:51.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:51 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:49] ENGINE Bus STARTING 2023-07-29T16:05:51.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:51 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:49] ENGINE Serving on https://172.21.15.31:7150 2023-07-29T16:05:51.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:51 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:49] ENGINE Serving on http://172.21.15.31:8765 2023-07-29T16:05:51.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:51 smithi031 ceph-mon[95504]: [29/Jul/2023:16:05:49] ENGINE Bus STARTED 2023-07-29T16:05:51.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:51 smithi031 ceph-mon[95504]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-29T16:05:51.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:51 smithi031 ceph-mon[95504]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-29T16:05:52.108 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.289+0000 7f1a80cf4700 1 Processor -- start 2023-07-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.289+0000 7f1a80cf4700 1 -- start start 2023-07-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.290+0000 7f1a80cf4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c0791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.290+0000 7f1a80cf4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c079710 con 0x7f1a7c07acd0 2023-07-29T16:05:52.109 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.290+0000 7f1a7a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c0791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:52.110 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.290+0000 7f1a7a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c0791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42976/0 (socket says 172.21.15.31:42976) 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.290+0000 7f1a7a59c700 1 -- 172.21.15.31:0/667281595 learned_addr learned my addr 172.21.15.31:0/667281595 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.290+0000 7f1a7a59c700 1 -- 172.21.15.31:0/667281595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c079850 con 0x7f1a7c07acd0 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.291+0000 7f1a7a59c700 1 --2- 172.21.15.31:0/667281595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c0791d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0098a0 tx=0x7f1a6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a5b33e549f37f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.291+0000 7f1a7959a700 1 -- 172.21.15.31:0/667281595 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c004160 con 0x7f1a7c07acd0 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.291+0000 7f1a7959a700 1 -- 172.21.15.31:0/667281595 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a6c0042c0 con 0x7f1a7c07acd0 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.291+0000 7f1a7959a700 1 -- 172.21.15.31:0/667281595 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c0119b0 con 0x7f1a7c07acd0 2023-07-29T16:05:52.111 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 -- 172.21.15.31:0/667281595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 msgr2=0x7f1a7c0791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 --2- 172.21.15.31:0/667281595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c0791d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c0098a0 tx=0x7f1a6c009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 -- 172.21.15.31:0/667281595 shutdown_connections 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 --2- 172.21.15.31:0/667281595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c0791d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 -- 172.21.15.31:0/667281595 >> 172.21.15.31:0/667281595 conn(0x7f1a7c1011d0 msgr2=0x7f1a7c103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 -- 172.21.15.31:0/667281595 shutdown_connections 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.292+0000 7f1a80cf4700 1 -- 172.21.15.31:0/667281595 wait complete. 2023-07-29T16:05:52.112 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.293+0000 7f1a80cf4700 1 Processor -- start 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.293+0000 7f1a80cf4700 1 -- start start 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a80cf4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a80cf4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c19bf60 con 0x7f1a7c07acd0 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a7a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a7a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42990/0 (socket says 172.21.15.31:42990) 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a7a59c700 1 -- 172.21.15.31:0/2782179793 learned_addr learned my addr 172.21.15.31:0/2782179793 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:52.113 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a7a59c700 1 -- 172.21.15.31:0/2782179793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c009550 con 0x7f1a7c07acd0 2023-07-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.294+0000 7f1a7a59c700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c19ba20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000c00 tx=0x7f1a6c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.295+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c0039f0 con 0x7f1a7c07acd0 2023-07-29T16:05:52.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.295+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a6c003b50 con 0x7f1a7c07acd0 2023-07-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.295+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c19c160 con 0x7f1a7c07acd0 2023-07-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.295+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1a6c01a780 con 0x7f1a7c07acd0 2023-07-29T16:05:52.116 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.295+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c19c600 con 0x7f1a7c07acd0 2023-07-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.296+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) v1 ==== 45082+0+0 (secure 0 0 0) 0x7f1a6c024440 con 0x7f1a7c07acd0 2023-07-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.296+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c10bf30 con 0x7f1a7c07acd0 2023-07-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.296+0000 7f1a6b7fe700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1a64038470 0x7f1a6403a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.297+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1a6c04c480 con 0x7f1a7c07acd0 2023-07-29T16:05:52.118 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.297+0000 7f1a79d9b700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1a64038470 0x7f1a6403a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.297+0000 7f1a79d9b700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1a64038470 0x7f1a6403a920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.302+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1a6c0189a0 con 0x7f1a7c07acd0 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.539+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a7c19f3e0 con 0x7f1a64038470 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.635+0000 7f1a6b7fe700 1 -- 172.21.15.31:0/2782179793 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1a7c19f3e0 con 0x7f1a64038470 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1a64038470 msgr2=0x7f1a6403a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1a64038470 0x7f1a6403a920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1a70006fd0 tx=0x7f1a70006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:52.119 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 msgr2=0x7f1a7c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c19ba20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000c00 tx=0x7f1a6c011a30 comp rx=0 tx=0).stop 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 shutdown_connections 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1a64038470 0x7f1a6403a920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 --2- 172.21.15.31:0/2782179793 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7c07acd0 0x7f1a7c19ba20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 >> 172.21.15.31:0/2782179793 conn(0x7f1a7c1011d0 msgr2=0x7f1a7c101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 shutdown_connections 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:51.638+0000 7f1a80cf4700 1 -- 172.21.15.31:0/2782179793 wait complete. 2023-07-29T16:05:52.120 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2023-07-29T16:05:52.375 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:52 smithi031 ceph-mon[95504]: mgrmap e12: x(active, since 2s) 2023-07-29T16:05:52.375 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:05:52.375 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:05:52.376 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:05:53.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:53 smithi031 ceph-mon[95504]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:53.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:05:53.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:05:53.494 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3WrQYc0Xb3yaNGVAis7iMefQfyHWDa8Iy8pAjfW4T4N8NazmZHEYu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1690646753, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-29T16:05:53.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.653+0000 7f28e91dc700 1 Processor -- start 2023-07-29T16:05:53.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.653+0000 7f28e91dc700 1 -- start start 2023-07-29T16:05:53.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e91dc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e4105fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:53.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e91dc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e41064e0 con 0x7f28e4105b90 2023-07-29T16:05:53.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e4105fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:53.495 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e4105fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42994/0 (socket says 172.21.15.31:42994) 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e2d9d700 1 -- 172.21.15.31:0/2182693112 learned_addr learned my addr 172.21.15.31:0/2182693112 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e2d9d700 1 -- 172.21.15.31:0/2182693112 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e4106620 con 0x7f28e4105b90 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.654+0000 7f28e2d9d700 1 --2- 172.21.15.31:0/2182693112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e4105fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28d40098a0 tx=0x7f28d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2aa45eba7037336 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e1d9b700 1 -- 172.21.15.31:0/2182693112 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28d4004160 con 0x7f28e4105b90 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e1d9b700 1 -- 172.21.15.31:0/2182693112 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28d40042c0 con 0x7f28e4105b90 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e1d9b700 1 -- 172.21.15.31:0/2182693112 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28d40119b0 con 0x7f28e4105b90 2023-07-29T16:05:53.496 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e91dc700 1 -- 172.21.15.31:0/2182693112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 msgr2=0x7f28e4105fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:53.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e91dc700 1 --2- 172.21.15.31:0/2182693112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e4105fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28d40098a0 tx=0x7f28d4009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:53.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e91dc700 1 -- 172.21.15.31:0/2182693112 shutdown_connections 2023-07-29T16:05:53.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e91dc700 1 --2- 172.21.15.31:0/2182693112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e4105fa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:53.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.655+0000 7f28e91dc700 1 -- 172.21.15.31:0/2182693112 >> 172.21.15.31:0/2182693112 conn(0x7f28e4101200 msgr2=0x7f28e4103610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:53.497 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.656+0000 7f28e91dc700 1 -- 172.21.15.31:0/2182693112 shutdown_connections 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.656+0000 7f28e91dc700 1 -- 172.21.15.31:0/2182693112 wait complete. 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e91dc700 1 Processor -- start 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e91dc700 1 -- start start 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e91dc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e91dc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e419bf50 con 0x7f28e4105b90 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e419ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43000/0 (socket says 172.21.15.31:43000) 2023-07-29T16:05:53.498 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.657+0000 7f28e2d9d700 1 -- 172.21.15.31:0/3890295291 learned_addr learned my addr 172.21.15.31:0/3890295291 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:53.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28e2d9d700 1 -- 172.21.15.31:0/3890295291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4009550 con 0x7f28e4105b90 2023-07-29T16:05:53.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28e2d9d700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e419ba10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f28d4011980 tx=0x7f28d4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:53.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28d4003a60 con 0x7f28e4105b90 2023-07-29T16:05:53.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28d4003bc0 con 0x7f28e4105b90 2023-07-29T16:05:53.499 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e419c150 con 0x7f28e4105b90 2023-07-29T16:05:53.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28d401a670 con 0x7f28e4105b90 2023-07-29T16:05:53.500 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.658+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e419c5f0 con 0x7f28e4105b90 2023-07-29T16:05:53.501 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.660+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) v1 ==== 45082+0+0 (secure 0 0 0) 0x7f28d4024a00 con 0x7f28e4105b90 2023-07-29T16:05:53.502 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.660+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e4062380 con 0x7f28e4105b90 2023-07-29T16:05:53.502 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.660+0000 7f28cbfff700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28cc038470 0x7f28cc03a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:53.502 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.660+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f28d404c520 con 0x7f28e4105b90 2023-07-29T16:05:53.502 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.661+0000 7f28e259c700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28cc038470 0x7f28cc03a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:53.502 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.661+0000 7f28e259c700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28cc038470 0x7f28cc03a920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28d8006fd0 tx=0x7f28d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:53.502 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.666+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f28d4024440 con 0x7f28e4105b90 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:52.896+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f28e40621a0 con 0x7f28cc038470 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.147+0000 7f28cbfff700 1 -- 172.21.15.31:0/3890295291 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f28e40621a0 con 0x7f28cc038470 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28cc038470 msgr2=0x7f28cc03a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28cc038470 0x7f28cc03a920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28d8006fd0 tx=0x7f28d8006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 msgr2=0x7f28e419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e419ba10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f28d4011980 tx=0x7f28d4011a60 comp rx=0 tx=0).stop 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 shutdown_connections 2023-07-29T16:05:53.503 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28cc038470 0x7f28cc03a920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:53.504 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 --2- 172.21.15.31:0/3890295291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28e4105b90 0x7f28e419ba10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:53.504 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 >> 172.21.15.31:0/3890295291 conn(0x7f28e4101200 msgr2=0x7f28e4101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:53.504 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 shutdown_connections 2023-07-29T16:05:53.504 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:53.150+0000 7f28e91dc700 1 -- 172.21.15.31:0/3890295291 wait complete. 2023-07-29T16:05:53.504 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2023-07-29T16:05:54.583 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:54 smithi031 ceph-mon[95504]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:05:54.878 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2023-07-29T16:05:54.879 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.032+0000 7f4cbd7a2700 1 Processor -- start 2023-07-29T16:05:54.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.032+0000 7f4cbd7a2700 1 -- start start 2023-07-29T16:05:54.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.032+0000 7f4cbd7a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb80791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:54.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.032+0000 7f4cbd7a2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb8079730 con 0x7f4cb807acf0 2023-07-29T16:05:54.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.033+0000 7f4cb6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb80791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:54.880 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.033+0000 7f4cb6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb80791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43014/0 (socket says 172.21.15.31:43014) 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.033+0000 7f4cb6ffd700 1 -- 172.21.15.31:0/289207134 learned_addr learned my addr 172.21.15.31:0/289207134 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.033+0000 7f4cb6ffd700 1 -- 172.21.15.31:0/289207134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb8079870 con 0x7f4cb807acf0 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.034+0000 7f4cb6ffd700 1 --2- 172.21.15.31:0/289207134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb80791f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80098a0 tx=0x7f4ca8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7c1ea67d9374f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.034+0000 7f4cb5ffb700 1 -- 172.21.15.31:0/289207134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca8004160 con 0x7f4cb807acf0 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.034+0000 7f4cb5ffb700 1 -- 172.21.15.31:0/289207134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ca80042c0 con 0x7f4cb807acf0 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.034+0000 7f4cb5ffb700 1 -- 172.21.15.31:0/289207134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca80119b0 con 0x7f4cb807acf0 2023-07-29T16:05:54.881 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.034+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/289207134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 msgr2=0x7f4cb80791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:54.882 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.034+0000 7f4cbd7a2700 1 --2- 172.21.15.31:0/289207134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb80791f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80098a0 tx=0x7f4ca8009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:54.882 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.035+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/289207134 shutdown_connections 2023-07-29T16:05:54.882 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.035+0000 7f4cbd7a2700 1 --2- 172.21.15.31:0/289207134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb80791f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:54.882 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.035+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/289207134 >> 172.21.15.31:0/289207134 conn(0x7f4cb81011d0 msgr2=0x7f4cb8103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.035+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/289207134 shutdown_connections 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.035+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/289207134 wait complete. 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cbd7a2700 1 Processor -- start 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cbd7a2700 1 -- start start 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cbd7a2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb819ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cbd7a2700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb819bf70 con 0x7f4cb807acf0 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cb6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb819ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:54.883 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cb6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb819ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43028/0 (socket says 172.21.15.31:43028) 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.036+0000 7f4cb6ffd700 1 -- 172.21.15.31:0/190476741 learned_addr learned my addr 172.21.15.31:0/190476741 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4cb6ffd700 1 -- 172.21.15.31:0/190476741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca8009550 con 0x7f4cb807acf0 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4cb6ffd700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb819ba30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8003e60 tx=0x7f4ca8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca8003b20 con 0x7f4cb807acf0 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ca8003c80 con 0x7f4cb807acf0 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb819c170 con 0x7f4cb807acf0 2023-07-29T16:05:54.884 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4ca8024760 con 0x7f4cb807acf0 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.037+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb819c610 con 0x7f4cb807acf0 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.038+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) v1 ==== 45082+0+0 (secure 0 0 0) 0x7f4ca801a430 con 0x7f4cb807acf0 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.038+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb810bf10 con 0x7f4cb807acf0 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.039+0000 7f4c9ffff700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ca00384c0 0x7f4ca003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.039+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4ca804c750 con 0x7f4cb807acf0 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.039+0000 7f4cb67fc700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ca00384c0 0x7f4ca003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:54.885 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.040+0000 7f4cb67fc700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ca00384c0 0x7f4ca003a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006fd0 tx=0x7f4cac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:54.886 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.044+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4ca8018b40 con 0x7f4cb807acf0 2023-07-29T16:05:54.886 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.284+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4cb819f450 con 0x7f4cb807acf0 2023-07-29T16:05:54.886 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.284+0000 7f4c9ffff700 1 -- 172.21.15.31:0/190476741 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4ca8018b40 con 0x7f4cb807acf0 2023-07-29T16:05:54.886 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ca00384c0 msgr2=0x7f4ca003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:54.886 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ca00384c0 0x7f4ca003a970 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4cac006fd0 tx=0x7f4cac006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 msgr2=0x7f4cb819ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb819ba30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8003e60 tx=0x7f4ca8011a30 comp rx=0 tx=0).stop 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 shutdown_connections 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ca00384c0 0x7f4ca003a970 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 --2- 172.21.15.31:0/190476741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4cb807acf0 0x7f4cb819ba30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 >> 172.21.15.31:0/190476741 conn(0x7f4cb81011d0 msgr2=0x7f4cb8101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:54.887 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.287+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 shutdown_connections 2023-07-29T16:05:54.888 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:54.288+0000 7f4cbd7a2700 1 -- 172.21.15.31:0/190476741 wait complete. 2023-07-29T16:05:54.896 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-29T16:05:54.897 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-07-29T16:05:54.912 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-29T16:05:54.913 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-07-29T16:05:54.913 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-07-29T16:05:54.913 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-29T16:05:54.914 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2023-07-29T16:05:54.915 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:54.915 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031.front.sepia.ceph.com:8443/ 2023-07-29T16:05:54.915 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2023-07-29T16:05:54.915 INFO:teuthology.orchestra.run.smithi031.stdout: Password: t0cvugqwss 2023-07-29T16:05:54.915 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:54.916 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config directory 2023-07-29T16:05:54.916 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling autotune for osd_memory_target 2023-07-29T16:05:55.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:55 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/190476741' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-29T16:05:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.627+0000 7f78970fb700 1 Processor -- start 2023-07-29T16:05:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.628+0000 7f78970fb700 1 -- start start 2023-07-29T16:05:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.628+0000 7f78970fb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f78900791d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.628+0000 7f78970fb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7890079710 con 0x7f789007acd0 2023-07-29T16:05:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.628+0000 7f7894e97700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f78900791d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:56.222 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.628+0000 7f7894e97700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f78900791d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43042/0 (socket says 172.21.15.31:43042) 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.628+0000 7f7894e97700 1 -- 172.21.15.31:0/2978615474 learned_addr learned my addr 172.21.15.31:0/2978615474 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.629+0000 7f7894e97700 1 -- 172.21.15.31:0/2978615474 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7890079850 con 0x7f789007acd0 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.629+0000 7f7894e97700 1 --2- 172.21.15.31:0/2978615474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f78900791d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f78840098a0 tx=0x7f7884009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592a14a7cebd529f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.629+0000 7f788f7fe700 1 -- 172.21.15.31:0/2978615474 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7884004160 con 0x7f789007acd0 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.629+0000 7f788f7fe700 1 -- 172.21.15.31:0/2978615474 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78840042c0 con 0x7f789007acd0 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.629+0000 7f788f7fe700 1 -- 172.21.15.31:0/2978615474 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78840119b0 con 0x7f789007acd0 2023-07-29T16:05:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 -- 172.21.15.31:0/2978615474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 msgr2=0x7f78900791d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 --2- 172.21.15.31:0/2978615474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f78900791d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f78840098a0 tx=0x7f7884009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 -- 172.21.15.31:0/2978615474 shutdown_connections 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 --2- 172.21.15.31:0/2978615474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f78900791d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 -- 172.21.15.31:0/2978615474 >> 172.21.15.31:0/2978615474 conn(0x7f78901011d0 msgr2=0x7f7890103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 -- 172.21.15.31:0/2978615474 shutdown_connections 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.630+0000 7f78970fb700 1 -- 172.21.15.31:0/2978615474 wait complete. 2023-07-29T16:05:56.224 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.631+0000 7f78970fb700 1 Processor -- start 2023-07-29T16:05:56.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.631+0000 7f78970fb700 1 -- start start 2023-07-29T16:05:56.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.631+0000 7f78970fb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f789019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:56.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.631+0000 7f78970fb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789019bf60 con 0x7f789007acd0 2023-07-29T16:05:56.225 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.632+0000 7f7894e97700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f789019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:56.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.632+0000 7f7894e97700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f789019ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43058/0 (socket says 172.21.15.31:43058) 2023-07-29T16:05:56.227 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.632+0000 7f7894e97700 1 -- 172.21.15.31:0/709924782 learned_addr learned my addr 172.21.15.31:0/709924782 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.632+0000 7f7894e97700 1 -- 172.21.15.31:0/709924782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884009550 con 0x7f789007acd0 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.632+0000 7f7894e97700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f789019ba20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7884003e60 tx=0x7f7884011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.633+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7884003970 con 0x7f789007acd0 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.633+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789019c160 con 0x7f789007acd0 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.633+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7884003ad0 con 0x7f789007acd0 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.633+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f78840195e0 con 0x7f789007acd0 2023-07-29T16:05:56.228 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.633+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789019c600 con 0x7f789007acd0 2023-07-29T16:05:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.634+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f78840198a0 con 0x7f789007acd0 2023-07-29T16:05:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.634+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789010bf30 con 0x7f789007acd0 2023-07-29T16:05:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.634+0000 7f788dffb700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7880038470 0x7f788003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.634+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f788404c5e0 con 0x7f789007acd0 2023-07-29T16:05:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.635+0000 7f788ffff700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7880038470 0x7f788003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:56.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.635+0000 7f788ffff700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7880038470 0x7f788003a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f787c006fd0 tx=0x7f787c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:56.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.639+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7884003c40 con 0x7f789007acd0 2023-07-29T16:05:56.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.845+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f789010c120 con 0x7f789007acd0 2023-07-29T16:05:56.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.846+0000 7f788dffb700 1 -- 172.21.15.31:0/709924782 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f7884023b40 con 0x7f789007acd0 2023-07-29T16:05:56.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7880038470 msgr2=0x7f788003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:56.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7880038470 0x7f788003a920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f787c006fd0 tx=0x7f787c006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:56.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 msgr2=0x7f789019ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:56.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f789019ba20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7884003e60 tx=0x7f7884011aa0 comp rx=0 tx=0).stop 2023-07-29T16:05:56.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 shutdown_connections 2023-07-29T16:05:56.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7880038470 0x7f788003a920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:56.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 --2- 172.21.15.31:0/709924782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f789007acd0 0x7f789019ba20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:56.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.848+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 >> 172.21.15.31:0/709924782 conn(0x7f78901011d0 msgr2=0x7f7890101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:56.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.849+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 shutdown_connections 2023-07-29T16:05:56.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:55.849+0000 7f78970fb700 1 -- 172.21.15.31:0/709924782 wait complete. 2023-07-29T16:05:56.519 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:56 smithi031 ceph-mon[95504]: mgrmap e13: x(active, since 6s) 2023-07-29T16:05:57.410 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.727+0000 7efe361d7700 1 Processor -- start 2023-07-29T16:05:57.410 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.728+0000 7efe361d7700 1 -- start start 2023-07-29T16:05:57.411 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.728+0000 7efe361d7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe300791f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:57.411 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.728+0000 7efe361d7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe30079730 con 0x7efe3007acf0 2023-07-29T16:05:57.411 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.728+0000 7efe2f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe300791f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:57.412 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.728+0000 7efe2f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe300791f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43074/0 (socket says 172.21.15.31:43074) 2023-07-29T16:05:57.412 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.728+0000 7efe2f7fe700 1 -- 172.21.15.31:0/880482489 learned_addr learned my addr 172.21.15.31:0/880482489 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:57.412 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.729+0000 7efe2f7fe700 1 -- 172.21.15.31:0/880482489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe30079870 con 0x7efe3007acf0 2023-07-29T16:05:57.412 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.729+0000 7efe2f7fe700 1 --2- 172.21.15.31:0/880482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe300791f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe200098a0 tx=0x7efe20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf04cce1cc3e442b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:57.413 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.729+0000 7efe2e7fc700 1 -- 172.21.15.31:0/880482489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe20004160 con 0x7efe3007acf0 2023-07-29T16:05:57.413 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.729+0000 7efe2e7fc700 1 -- 172.21.15.31:0/880482489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe200042c0 con 0x7efe3007acf0 2023-07-29T16:05:57.413 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.729+0000 7efe2e7fc700 1 -- 172.21.15.31:0/880482489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe200119b0 con 0x7efe3007acf0 2023-07-29T16:05:57.413 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 -- 172.21.15.31:0/880482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 msgr2=0x7efe300791f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 --2- 172.21.15.31:0/880482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe300791f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe200098a0 tx=0x7efe20009bb0 comp rx=0 tx=0).stop 2023-07-29T16:05:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 -- 172.21.15.31:0/880482489 shutdown_connections 2023-07-29T16:05:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 --2- 172.21.15.31:0/880482489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe300791f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:57.415 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 -- 172.21.15.31:0/880482489 >> 172.21.15.31:0/880482489 conn(0x7efe301011d0 msgr2=0x7efe30103600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:57.415 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 -- 172.21.15.31:0/880482489 shutdown_connections 2023-07-29T16:05:57.415 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.730+0000 7efe361d7700 1 -- 172.21.15.31:0/880482489 wait complete. 2023-07-29T16:05:57.415 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.731+0000 7efe361d7700 1 Processor -- start 2023-07-29T16:05:57.416 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.731+0000 7efe361d7700 1 -- start start 2023-07-29T16:05:57.416 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe361d7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe3019ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:57.416 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe361d7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3019bf70 con 0x7efe3007acf0 2023-07-29T16:05:57.417 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe2f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe3019ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:57.417 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe2f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe3019ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43084/0 (socket says 172.21.15.31:43084) 2023-07-29T16:05:57.417 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe2f7fe700 1 -- 172.21.15.31:0/2552898937 learned_addr learned my addr 172.21.15.31:0/2552898937 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:05:57.417 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe2f7fe700 1 -- 172.21.15.31:0/2552898937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20009550 con 0x7efe3007acf0 2023-07-29T16:05:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.732+0000 7efe2f7fe700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe3019ba30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe20003e60 tx=0x7efe20011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.733+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe20003b20 con 0x7efe3007acf0 2023-07-29T16:05:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.733+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe20003c80 con 0x7efe3007acf0 2023-07-29T16:05:57.418 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.733+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3019c170 con 0x7efe3007acf0 2023-07-29T16:05:57.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.733+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe20024760 con 0x7efe3007acf0 2023-07-29T16:05:57.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.733+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3019c610 con 0x7efe3007acf0 2023-07-29T16:05:57.419 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.734+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7efe2001ab10 con 0x7efe3007acf0 2023-07-29T16:05:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.734+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe3010bf10 con 0x7efe3007acf0 2023-07-29T16:05:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.734+0000 7efe2cff9700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe180384c0 0x7efe1803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:05:57.420 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.734+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efe2004c710 con 0x7efe3007acf0 2023-07-29T16:05:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.735+0000 7efe2effd700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe180384c0 0x7efe1803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:05:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.735+0000 7efe2effd700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe180384c0 0x7efe1803a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe24006fd0 tx=0x7efe24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:05:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:56.740+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efe2001a430 con 0x7efe3007acf0 2023-07-29T16:05:57.421 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.057+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7efe3010c130 con 0x7efe3007acf0 2023-07-29T16:05:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.061+0000 7efe2cff9700 1 -- 172.21.15.31:0/2552898937 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7efe20018b40 con 0x7efe3007acf0 2023-07-29T16:05:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe180384c0 msgr2=0x7efe1803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:57.422 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe180384c0 0x7efe1803a970 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe24006fd0 tx=0x7efe24006e40 comp rx=0 tx=0).stop 2023-07-29T16:05:57.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 msgr2=0x7efe3019ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:05:57.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe3019ba30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe20003e60 tx=0x7efe20011a30 comp rx=0 tx=0).stop 2023-07-29T16:05:57.423 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 shutdown_connections 2023-07-29T16:05:57.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe180384c0 0x7efe1803a970 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:57.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 --2- 172.21.15.31:0/2552898937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efe3007acf0 0x7efe3019ba30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:05:57.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 >> 172.21.15.31:0/2552898937 conn(0x7efe301011d0 msgr2=0x7efe30101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:05:57.424 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 shutdown_connections 2023-07-29T16:05:57.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2023-07-29T16:05:57.063+0000 7efe361d7700 1 -- 172.21.15.31:0/2552898937 wait complete. 2023-07-29T16:05:57.425 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-29T16:05:57.425 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-29T16:05:57.426 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.426 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:05:57.426 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.426 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2023-07-29T16:05:57.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.427 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-29T16:05:57.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.428 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-29T16:05:57.428 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.428 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2023-07-29T16:05:57.428 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.429 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2023-07-29T16:05:57.429 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.429 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-29T16:05:57.429 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:05:57.430 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2023-07-29T16:05:57.467 INFO:tasks.cephadm:Fetching config... 2023-07-29T16:05:57.468 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:05:57.469 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-29T16:05:57.514 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-29T16:05:57.515 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:05:57.515 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-29T16:05:57.602 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-29T16:05:57.603 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:05:57.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-29T16:05:57.714 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-29T16:05:57.715 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:05:57.715 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-29T16:05:57.802 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-29T16:05:57.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-29T16:05:57.945 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:05:57.978 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-29T16:05:58.029 INFO:teuthology.orchestra.run.smithi039.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:05:58.043 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-29T16:05:58.090 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTU3Ny3ZoTSk280RGl3Ly+ET3K6ucsq1vc9DxK8/sjzzki2QQ/jYO0BUhryh/INH8P3DjMjssnByecSB+F5+lO9B6OsmM/JzUJ4koiR823gT1mEcvFELt6pJ3jtH/IWCzeedo7/IgVeRm6GntYLnumLMa4fTIM7FkYzcZ0in/6OgeRUwkzfHmcRiBQC8RuXjOCLYmsobCUxDCq+F2q023nx3D+N2r+pWCqqcVVb6j8/dYcz5jVg9KTgOd2Oi31OEB2NxLhw6G+iaen92A6EQSIqt8oW72gVOaqv1+sbEztXQ4kUFZI+9inBpnmf75kFCJDa6sS60iyxXVid/JzNfJYOP42hx+xBF1tTI1/q+9AGl4GKUKwtao9ryGNBwLE2921HmjQ1QDSBG7MkdImrfm/nPYuJSAmsWYM5xQlm+mytE0IwbDr+KQdTp51bmT/EJDNTUCcb51tCSeeZVHG0aoknppBwXsBovjIKrZ4Al7DsUTaLbOM1oevdKVcXIfEO7M= ceph-858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:05:58.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-29T16:05:58.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:05:58 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2552898937' entity='client.admin' 2023-07-29T16:05:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:00.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:00.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:00.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:00.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:00.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:00.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 -- 172.21.15.31:0/2153520458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 msgr2=0x7f9b480731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 --2- 172.21.15.31:0/2153520458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b480731e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009910 tx=0x7f9b44009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 -- 172.21.15.31:0/2153520458 shutdown_connections 2023-07-29T16:06:00.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 --2- 172.21.15.31:0/2153520458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b480731e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 -- 172.21.15.31:0/2153520458 >> 172.21.15.31:0/2153520458 conn(0x7f9b480fb3d0 msgr2=0x7f9b480fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 -- 172.21.15.31:0/2153520458 shutdown_connections 2023-07-29T16:06:00.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.451+0000 7f9b507aa700 1 -- 172.21.15.31:0/2153520458 wait complete. 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.452+0000 7f9b507aa700 1 Processor -- start 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.452+0000 7f9b507aa700 1 -- start start 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.452+0000 7f9b507aa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b48197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.453+0000 7f9b507aa700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48197690 con 0x7f9b48074d80 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.453+0000 7f9b4e546700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b48197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.453+0000 7f9b4e546700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b48197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36144/0 (socket says 172.21.15.31:36144) 2023-07-29T16:06:00.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.453+0000 7f9b4e546700 1 -- 172.21.15.31:0/545180718 learned_addr learned my addr 172.21.15.31:0/545180718 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:00.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.453+0000 7f9b4e546700 1 -- 172.21.15.31:0/545180718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b440095f0 con 0x7f9b48074d80 2023-07-29T16:06:00.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.453+0000 7f9b4e546700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b48197150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9b44005f50 tx=0x7f9b440057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:00.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.454+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b4401d070 con 0x7f9b48074d80 2023-07-29T16:06:00.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.454+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b44022470 con 0x7f9b48074d80 2023-07-29T16:06:00.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.454+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9b4400f5f0 con 0x7f9b48074d80 2023-07-29T16:06:00.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.454+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b48197890 con 0x7f9b48074d80 2023-07-29T16:06:00.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.454+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b48197d30 con 0x7f9b48074d80 2023-07-29T16:06:00.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.455+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f9b44022850 con 0x7f9b48074d80 2023-07-29T16:06:00.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.455+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b48190bf0 con 0x7f9b48074d80 2023-07-29T16:06:00.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.456+0000 7f9b3f7fe700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9b340384c0 0x7f9b3403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:00.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.456+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9b4404c520 con 0x7f9b48074d80 2023-07-29T16:06:00.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.456+0000 7f9b4dd45700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9b340384c0 0x7f9b3403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:00.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.457+0000 7f9b4dd45700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9b340384c0 0x7f9b3403a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b38006fd0 tx=0x7f9b38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:00.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.461+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9b4402bba0 con 0x7f9b48074d80 2023-07-29T16:06:00.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.682+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f9b4802cc30 con 0x7f9b48074d80 2023-07-29T16:06:00.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.685+0000 7f9b3f7fe700 1 -- 172.21.15.31:0/545180718 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9b44027030 con 0x7f9b48074d80 2023-07-29T16:06:00.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9b340384c0 msgr2=0x7f9b3403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:00.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9b340384c0 0x7f9b3403a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b38006fd0 tx=0x7f9b38006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:00.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 msgr2=0x7f9b48197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:00.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b48197150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9b44005f50 tx=0x7f9b440057c0 comp rx=0 tx=0).stop 2023-07-29T16:06:00.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 shutdown_connections 2023-07-29T16:06:00.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9b340384c0 0x7f9b3403a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:00.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 --2- 172.21.15.31:0/545180718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9b48074d80 0x7f9b48197150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:00.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 >> 172.21.15.31:0/545180718 conn(0x7f9b480fb3d0 msgr2=0x7f9b480fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:00.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 shutdown_connections 2023-07-29T16:06:00.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:00.688+0000 7f9b507aa700 1 -- 172.21.15.31:0/545180718 wait complete. 2023-07-29T16:06:01.230 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-29T16:06:01.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-29T16:06:01.965 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:02.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/545180718' entity='client.admin' 2023-07-29T16:06:02.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:02.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:03.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.355+0000 7f7556bda700 1 -- 172.21.15.31:0/1227766887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 msgr2=0x7f7550100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:03.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.355+0000 7f7556bda700 1 --2- 172.21.15.31:0/1227766887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550100230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7544009910 tx=0x7f7544009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.356+0000 7f7556bda700 1 -- 172.21.15.31:0/1227766887 shutdown_connections 2023-07-29T16:06:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.356+0000 7f7556bda700 1 --2- 172.21.15.31:0/1227766887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550100230 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.356+0000 7f7556bda700 1 -- 172.21.15.31:0/1227766887 >> 172.21.15.31:0/1227766887 conn(0x7f75500fb3d0 msgr2=0x7f75500fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.356+0000 7f7556bda700 1 -- 172.21.15.31:0/1227766887 shutdown_connections 2023-07-29T16:06:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.356+0000 7f7556bda700 1 -- 172.21.15.31:0/1227766887 wait complete. 2023-07-29T16:06:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.357+0000 7f7556bda700 1 Processor -- start 2023-07-29T16:06:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.357+0000 7f7556bda700 1 -- start start 2023-07-29T16:06:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.358+0000 7f7556bda700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.358+0000 7f7556bda700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75501954b0 con 0x7f75500ffe20 2023-07-29T16:06:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.358+0000 7f7554976700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.358+0000 7f7554976700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550194f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36154/0 (socket says 172.21.15.31:36154) 2023-07-29T16:06:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.358+0000 7f7554976700 1 -- 172.21.15.31:0/2393129197 learned_addr learned my addr 172.21.15.31:0/2393129197 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.358+0000 7f7554976700 1 -- 172.21.15.31:0/2393129197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440095f0 con 0x7f75500ffe20 2023-07-29T16:06:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.359+0000 7f7554976700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550194f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7544005f50 tx=0x7f75440057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.359+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f754401d070 con 0x7f75500ffe20 2023-07-29T16:06:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.359+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7544022470 con 0x7f75500ffe20 2023-07-29T16:06:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.359+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f754400f5f0 con 0x7f75500ffe20 2023-07-29T16:06:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.359+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75501956b0 con 0x7f75500ffe20 2023-07-29T16:06:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.359+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7550195b50 con 0x7f75500ffe20 2023-07-29T16:06:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.360+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f7544022850 con 0x7f75500ffe20 2023-07-29T16:06:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.360+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f755018e980 con 0x7f75500ffe20 2023-07-29T16:06:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.361+0000 7f754dffb700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75400384c0 0x7f754003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.361+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f754404c3c0 con 0x7f75500ffe20 2023-07-29T16:06:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.361+0000 7f754ffff700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75400384c0 0x7f754003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.361+0000 7f754ffff700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75400384c0 0x7f754003a970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f753c006fd0 tx=0x7f753c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.366+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f754400ba90 con 0x7f75500ffe20 2023-07-29T16:06:03.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.579+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7550061190 con 0x7f75400384c0 2023-07-29T16:06:03.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.582+0000 7f754dffb700 1 -- 172.21.15.31:0/2393129197 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7550061190 con 0x7f75400384c0 2023-07-29T16:06:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75400384c0 msgr2=0x7f754003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75400384c0 0x7f754003a970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f753c006fd0 tx=0x7f753c006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 msgr2=0x7f7550194f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550194f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7544005f50 tx=0x7f75440057c0 comp rx=0 tx=0).stop 2023-07-29T16:06:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 shutdown_connections 2023-07-29T16:06:03.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75400384c0 0x7f754003a970 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:03.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 --2- 172.21.15.31:0/2393129197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75500ffe20 0x7f7550194f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:03.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 >> 172.21.15.31:0/2393129197 conn(0x7f75500fb3d0 msgr2=0x7f75500fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:03.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 shutdown_connections 2023-07-29T16:06:03.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:03.585+0000 7f7556bda700 1 -- 172.21.15.31:0/2393129197 wait complete. 2023-07-29T16:06:04.147 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi039 2023-07-29T16:06:04.147 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:06:04.148 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/etc/ceph/ceph.conf 2023-07-29T16:06:04.169 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:06:04.169 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:04.231 INFO:tasks.cephadm:Adding host smithi039 to orchestrator... 2023-07-29T16:06:04.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch host add smithi039 2023-07-29T16:06:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: Updating smithi031:/etc/ceph/ceph.conf 2023-07-29T16:06:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:04 smithi031 ceph-mon[95504]: Updating smithi031:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:05.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:05 smithi031 ceph-mon[95504]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:05.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:05 smithi031 ceph-mon[95504]: Updating smithi031:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:06.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.534+0000 7f67de9d9700 1 -- 172.21.15.31:0/1259739280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 msgr2=0x7f67d8100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:06.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.534+0000 7f67de9d9700 1 --2- 172.21.15.31:0/1259739280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8100460 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f67c8009910 tx=0x7f67c8009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:06.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.534+0000 7f67de9d9700 1 -- 172.21.15.31:0/1259739280 shutdown_connections 2023-07-29T16:06:06.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.534+0000 7f67de9d9700 1 --2- 172.21.15.31:0/1259739280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8100460 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:06.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.534+0000 7f67de9d9700 1 -- 172.21.15.31:0/1259739280 >> 172.21.15.31:0/1259739280 conn(0x7f67d80fb970 msgr2=0x7f67d80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:06.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.535+0000 7f67de9d9700 1 -- 172.21.15.31:0/1259739280 shutdown_connections 2023-07-29T16:06:06.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.535+0000 7f67de9d9700 1 -- 172.21.15.31:0/1259739280 wait complete. 2023-07-29T16:06:06.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67de9d9700 1 Processor -- start 2023-07-29T16:06:06.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67de9d9700 1 -- start start 2023-07-29T16:06:06.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67de9d9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67de9d9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8074fd0 con 0x7f67d8100090 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8073050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36172/0 (socket says 172.21.15.31:36172) 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.536+0000 7f67d7fff700 1 -- 172.21.15.31:0/26859115 learned_addr learned my addr 172.21.15.31:0/26859115 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67d7fff700 1 -- 172.21.15.31:0/26859115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c80095f0 con 0x7f67d8100090 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67d7fff700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8073050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f67c8006010 tx=0x7f67c8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:06.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67c801d070 con 0x7f67d8100090 2023-07-29T16:06:06.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67c800bc10 con 0x7f67d8100090 2023-07-29T16:06:06.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d8073590 con 0x7f67d8100090 2023-07-29T16:06:06.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67c8003910 con 0x7f67d8100090 2023-07-29T16:06:06.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.537+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d8073a10 con 0x7f67d8100090 2023-07-29T16:06:06.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.538+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f67c8003b80 con 0x7f67d8100090 2023-07-29T16:06:06.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.538+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d8108310 con 0x7f67d8100090 2023-07-29T16:06:06.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.539+0000 7f67d57fa700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67c00384c0 0x7f67c003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:06.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.539+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f67c804c070 con 0x7f67d8100090 2023-07-29T16:06:06.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.539+0000 7f67d77fe700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67c00384c0 0x7f67c003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:06.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.540+0000 7f67d77fe700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67c00384c0 0x7f67c003a970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f67cc006fd0 tx=0x7f67cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:06.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.544+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f67c800ba70 con 0x7f67d8100090 2023-07-29T16:06:06.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:06.756+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi039", "target": ["mon-mgr", ""]}) v1 -- 0x7f67d8073d10 con 0x7f67c00384c0 2023-07-29T16:06:07.179 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:07.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:07.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:08.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:07 smithi031 ceph-mon[95504]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi039", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:08.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:07 smithi031 ceph-mon[95504]: Deploying cephadm binary to smithi039 2023-07-29T16:06:09.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.123+0000 7f67d57fa700 1 -- 172.21.15.31:0/26859115 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f67d8073d10 con 0x7f67c00384c0 2023-07-29T16:06:09.120 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi039' with addr '172.21.15.39' 2023-07-29T16:06:09.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67c00384c0 msgr2=0x7f67c003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:09.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67c00384c0 0x7f67c003a970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f67cc006fd0 tx=0x7f67cc006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:09.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 msgr2=0x7f67d8073050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:09.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8073050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f67c8006010 tx=0x7f67c8004ff0 comp rx=0 tx=0).stop 2023-07-29T16:06:09.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 shutdown_connections 2023-07-29T16:06:09.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67c00384c0 0x7f67c003a970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:09.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 --2- 172.21.15.31:0/26859115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67d8100090 0x7f67d8073050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:09.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 >> 172.21.15.31:0/26859115 conn(0x7f67d80fb970 msgr2=0x7f67d8104270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:09.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 shutdown_connections 2023-07-29T16:06:09.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:09.126+0000 7f67de9d9700 1 -- 172.21.15.31:0/26859115 wait complete. 2023-07-29T16:06:09.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch host ls --format=json 2023-07-29T16:06:10.289 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:10.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:10.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:10 smithi031 ceph-mon[95504]: Added host smithi039 2023-07-29T16:06:10.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:10.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:10 smithi031 ceph-mon[95504]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:10.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:11.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:11 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:12.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2819533697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 msgr2=0x7f8bf4101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 --2- 172.21.15.31:0/2819533697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4101300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009910 tx=0x7f8bf0009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2819533697 shutdown_connections 2023-07-29T16:06:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 --2- 172.21.15.31:0/2819533697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4101300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2819533697 >> 172.21.15.31:0/2819533697 conn(0x7f8bf4074af0 msgr2=0x7f8bf4074ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2819533697 shutdown_connections 2023-07-29T16:06:12.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.068+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2819533697 wait complete. 2023-07-29T16:06:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.069+0000 7f8bfbdef700 1 Processor -- start 2023-07-29T16:06:12.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.070+0000 7f8bfbdef700 1 -- start start 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.070+0000 7f8bfbdef700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.070+0000 7f8bfbdef700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4197a90 con 0x7f8bf4100f30 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.070+0000 7f8bf9b8b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.070+0000 7f8bf9b8b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58830/0 (socket says 172.21.15.31:58830) 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.070+0000 7f8bf9b8b700 1 -- 172.21.15.31:0/2732030713 learned_addr learned my addr 172.21.15.31:0/2732030713 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8bf9b8b700 1 -- 172.21.15.31:0/2732030713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf00095f0 con 0x7f8bf4100f30 2023-07-29T16:06:12.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8bf9b8b700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4197550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006010 tx=0x7f8bf0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:12.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8bf001d070 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bf00044a0 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf4197c90 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8bf0003a70 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.071+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf41980b0 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.072+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f8bf0003c60 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.072+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4062380 con 0x7f8bf4100f30 2023-07-29T16:06:12.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.073+0000 7f8beaffd700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8be00384c0 0x7f8be003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.073+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8bf004c180 con 0x7f8bf4100f30 2023-07-29T16:06:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.073+0000 7f8bf938a700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8be00384c0 0x7f8be003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:12.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.074+0000 7f8bf938a700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8be00384c0 0x7f8be003a970 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8be4006fd0 tx=0x7f8be4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:12.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.078+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8bf0005c00 con 0x7f8bf4100f30 2023-07-29T16:06:12.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.287+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8bf4107260 con 0x7f8be00384c0 2023-07-29T16:06:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.288+0000 7f8beaffd700 1 -- 172.21.15.31:0/2732030713 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f8bf4107260 con 0x7f8be00384c0 2023-07-29T16:06:12.284 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:06:12.284 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}] 2023-07-29T16:06:12.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8be00384c0 msgr2=0x7f8be003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:12.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8be00384c0 0x7f8be003a970 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8be4006fd0 tx=0x7f8be4006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:12.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 msgr2=0x7f8bf4197550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:12.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4197550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0006010 tx=0x7f8bf0004ff0 comp rx=0 tx=0).stop 2023-07-29T16:06:12.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 shutdown_connections 2023-07-29T16:06:12.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8be00384c0 0x7f8be003a970 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:12.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 --2- 172.21.15.31:0/2732030713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bf4100f30 0x7f8bf4197550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:12.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 >> 172.21.15.31:0/2732030713 conn(0x7f8bf4074af0 msgr2=0x7f8bf4106b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:12.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 shutdown_connections 2023-07-29T16:06:12.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:12.291+0000 7f8bfbdef700 1 -- 172.21.15.31:0/2732030713 wait complete. 2023-07-29T16:06:12.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:12 smithi031 ceph-mon[95504]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:12.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:12 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:12.667 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2023-07-29T16:06:12.668 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:06:12.668 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2023-07-29T16:06:12.689 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:06:12.690 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:12.748 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2023-07-29T16:06:12.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch host add smithi156 2023-07-29T16:06:13.475 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:13.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:13 smithi031 ceph-mon[95504]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:06:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 -- 172.21.15.31:0/3251561875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 msgr2=0x7f1520102680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 --2- 172.21.15.31:0/3251561875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520102680 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1510009910 tx=0x7f1510009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:14.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 -- 172.21.15.31:0/3251561875 shutdown_connections 2023-07-29T16:06:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 --2- 172.21.15.31:0/3251561875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520102680 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 -- 172.21.15.31:0/3251561875 >> 172.21.15.31:0/3251561875 conn(0x7f15200fdb70 msgr2=0x7f15200fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:14.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 -- 172.21.15.31:0/3251561875 shutdown_connections 2023-07-29T16:06:14.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.895+0000 7f1524e62700 1 -- 172.21.15.31:0/3251561875 wait complete. 2023-07-29T16:06:14.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.896+0000 7f1524e62700 1 Processor -- start 2023-07-29T16:06:14.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.897+0000 7f1524e62700 1 -- start start 2023-07-29T16:06:14.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.897+0000 7f1524e62700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:14.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.897+0000 7f1524e62700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1520197a60 con 0x7f15201022b0 2023-07-29T16:06:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.897+0000 7f151e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.897+0000 7f151e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58848/0 (socket says 172.21.15.31:58848) 2023-07-29T16:06:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.897+0000 7f151e59c700 1 -- 172.21.15.31:0/1851950229 learned_addr learned my addr 172.21.15.31:0/1851950229 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:14.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f151e59c700 1 -- 172.21.15.31:0/1851950229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100095f0 con 0x7f15201022b0 2023-07-29T16:06:14.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f151e59c700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520197520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1510004f40 tx=0x7f1510005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:14.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f151001d070 con 0x7f15201022b0 2023-07-29T16:06:14.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1510004370 con 0x7f15201022b0 2023-07-29T16:06:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1520197c60 con 0x7f15201022b0 2023-07-29T16:06:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15100039b0 con 0x7f15201022b0 2023-07-29T16:06:14.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.898+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1520198080 con 0x7f15201022b0 2023-07-29T16:06:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.899+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f1510003c20 con 0x7f15201022b0 2023-07-29T16:06:14.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.899+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1520062380 con 0x7f15201022b0 2023-07-29T16:06:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.900+0000 7f150f7fe700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15080384c0 0x7f150803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.900+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f151004c0b0 con 0x7f15201022b0 2023-07-29T16:06:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.900+0000 7f151dd9b700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15080384c0 0x7f150803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:14.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.901+0000 7f151dd9b700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15080384c0 0x7f150803a970 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1514006fd0 tx=0x7f1514006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:14.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:14.905+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f151007e720 con 0x7f15201022b0 2023-07-29T16:06:14.929 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:14.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:14.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:14.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:15.119+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) v1 -- 0x7f152010a050 con 0x7f15080384c0 2023-07-29T16:06:15.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:15 smithi031 ceph-mon[95504]: Updating smithi039:/etc/ceph/ceph.conf 2023-07-29T16:06:15.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:15 smithi031 ceph-mon[95504]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:15.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:15.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:15.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:16.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:16 smithi031 ceph-mon[95504]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:16.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:16 smithi031 ceph-mon[95504]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:16.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:16 smithi031 ceph-mon[95504]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:16.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:16 smithi031 ceph-mon[95504]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:17.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.418+0000 7f150f7fe700 1 -- 172.21.15.31:0/1851950229 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f152010a050 con 0x7f15080384c0 2023-07-29T16:06:17.415 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi156' with addr '172.21.15.156' 2023-07-29T16:06:17.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15080384c0 msgr2=0x7f150803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:17.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15080384c0 0x7f150803a970 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1514006fd0 tx=0x7f1514006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:17.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 msgr2=0x7f1520197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:17.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520197520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1510004f40 tx=0x7f1510005020 comp rx=0 tx=0).stop 2023-07-29T16:06:17.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 shutdown_connections 2023-07-29T16:06:17.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15080384c0 0x7f150803a970 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:17.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 --2- 172.21.15.31:0/1851950229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15201022b0 0x7f1520197520 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:17.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 >> 172.21.15.31:0/1851950229 conn(0x7f15200fdb70 msgr2=0x7f1520106490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:17.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 shutdown_connections 2023-07-29T16:06:17.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:17.421+0000 7f1524e62700 1 -- 172.21.15.31:0/1851950229 wait complete. 2023-07-29T16:06:17.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:17 smithi031 ceph-mon[95504]: Deploying cephadm binary to smithi156 2023-07-29T16:06:17.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:17.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:17.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch host ls --format=json 2023-07-29T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:18.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:18 smithi031 ceph-mon[95504]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:18.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:18 smithi031 ceph-mon[95504]: Added host smithi156 2023-07-29T16:06:18.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:18 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:18.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:18 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 -- 172.21.15.31:0/3816356565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 msgr2=0x7fafb8100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 --2- 172.21.15.31:0/3816356565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8100460 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fafac009910 tx=0x7fafac009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 -- 172.21.15.31:0/3816356565 shutdown_connections 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 --2- 172.21.15.31:0/3816356565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8100460 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 -- 172.21.15.31:0/3816356565 >> 172.21.15.31:0/3816356565 conn(0x7fafb80fb970 msgr2=0x7fafb80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 -- 172.21.15.31:0/3816356565 shutdown_connections 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.184+0000 7fafbef7f700 1 -- 172.21.15.31:0/3816356565 wait complete. 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.185+0000 7fafbef7f700 1 Processor -- start 2023-07-29T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.186+0000 7fafbef7f700 1 -- start start 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.186+0000 7fafbef7f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8195300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.186+0000 7fafbef7f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb8195840 con 0x7fafb8100090 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.186+0000 7fafbcd1b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8195300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.186+0000 7fafbcd1b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8195300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40568/0 (socket says 172.21.15.31:40568) 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.186+0000 7fafbcd1b700 1 -- 172.21.15.31:0/1232388120 learned_addr learned my addr 172.21.15.31:0/1232388120 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafbcd1b700 1 -- 172.21.15.31:0/1232388120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0095f0 con 0x7fafb8100090 2023-07-29T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafbcd1b700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8195300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafac006010 tx=0x7fafac004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:20.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafac01d070 con 0x7fafb8100090 2023-07-29T16:06:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafac00bc10 con 0x7fafb8100090 2023-07-29T16:06:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb8195a40 con 0x7fafb8100090 2023-07-29T16:06:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafac003910 con 0x7fafb8100090 2023-07-29T16:06:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.187+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb8195e60 con 0x7fafb8100090 2023-07-29T16:06:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.188+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7fafac00bd80 con 0x7fafb8100090 2023-07-29T16:06:20.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.188+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb8062380 con 0x7fafb8100090 2023-07-29T16:06:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.189+0000 7fafb5ffb700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafa80384c0 0x7fafa803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.189+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fafac04c080 con 0x7fafb8100090 2023-07-29T16:06:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.189+0000 7fafb7fff700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafa80384c0 0x7fafa803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:20.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.189+0000 7fafb7fff700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafa80384c0 0x7fafa803a970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fafa4006fd0 tx=0x7fafa4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:20.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.194+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fafac029360 con 0x7fafb8100090 2023-07-29T16:06:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.409+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafb8107e30 con 0x7fafa80384c0 2023-07-29T16:06:20.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.410+0000 7fafb5ffb700 1 -- 172.21.15.31:0/1232388120 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7fafb8107e30 con 0x7fafa80384c0 2023-07-29T16:06:20.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:06:20.408 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2023-07-29T16:06:20.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafa80384c0 msgr2=0x7fafa803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:20.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafa80384c0 0x7fafa803a970 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fafa4006fd0 tx=0x7fafa4006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:20.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 msgr2=0x7fafb8195300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:20.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8195300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafac006010 tx=0x7fafac004ff0 comp rx=0 tx=0).stop 2023-07-29T16:06:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 shutdown_connections 2023-07-29T16:06:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafa80384c0 0x7fafa803a970 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 --2- 172.21.15.31:0/1232388120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafb8100090 0x7fafb8195300 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 >> 172.21.15.31:0/1232388120 conn(0x7fafb80fb970 msgr2=0x7fafb8104270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 shutdown_connections 2023-07-29T16:06:20.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:20.412+0000 7fafbef7f700 1 -- 172.21.15.31:0/1232388120 wait complete. 2023-07-29T16:06:20.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:20 smithi031 ceph-mon[95504]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:20.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:20.869 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-29T16:06:20.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd crush tunables default 2023-07-29T16:06:21.595 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:21.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:21 smithi031 ceph-mon[95504]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:06:22.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:22.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:22.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:22.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:22.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:22.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:22.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:22.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.135+0000 7fefd3d18700 1 -- 172.21.15.31:0/4068096004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 msgr2=0x7fefcc100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.135+0000 7fefd3d18700 1 --2- 172.21.15.31:0/4068096004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc100460 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fefc8009910 tx=0x7fefc8009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.135+0000 7fefd3d18700 1 -- 172.21.15.31:0/4068096004 shutdown_connections 2023-07-29T16:06:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.135+0000 7fefd3d18700 1 --2- 172.21.15.31:0/4068096004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc100460 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.135+0000 7fefd3d18700 1 -- 172.21.15.31:0/4068096004 >> 172.21.15.31:0/4068096004 conn(0x7fefcc0fb970 msgr2=0x7fefcc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.136+0000 7fefd3d18700 1 -- 172.21.15.31:0/4068096004 shutdown_connections 2023-07-29T16:06:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.136+0000 7fefd3d18700 1 -- 172.21.15.31:0/4068096004 wait complete. 2023-07-29T16:06:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.136+0000 7fefd3d18700 1 Processor -- start 2023-07-29T16:06:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.137+0000 7fefd3d18700 1 -- start start 2023-07-29T16:06:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.137+0000 7fefd3d18700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.137+0000 7fefd3d18700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefcc197b30 con 0x7fefcc100090 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.137+0000 7fefd1ab4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.137+0000 7fefd1ab4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc1975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40590/0 (socket says 172.21.15.31:40590) 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.137+0000 7fefd1ab4700 1 -- 172.21.15.31:0/1574238713 learned_addr learned my addr 172.21.15.31:0/1574238713 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefd1ab4700 1 -- 172.21.15.31:0/1574238713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc80095f0 con 0x7fefcc100090 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefd1ab4700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc1975f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fefc8004f40 tx=0x7fefc8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefc801d070 con 0x7fefcc100090 2023-07-29T16:06:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefcc197d30 con 0x7fefcc100090 2023-07-29T16:06:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fefc8004370 con 0x7fefcc100090 2023-07-29T16:06:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefc80039b0 con 0x7fefcc100090 2023-07-29T16:06:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.138+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefcc198150 con 0x7fefcc100090 2023-07-29T16:06:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.139+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefcc062380 con 0x7fefcc100090 2023-07-29T16:06:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.139+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7fefc8003c70 con 0x7fefcc100090 2023-07-29T16:06:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.140+0000 7fefc2ffd700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fefb8038470 0x7fefb803a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.140+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fefc804c100 con 0x7fefcc100090 2023-07-29T16:06:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.140+0000 7fefd12b3700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fefb8038470 0x7fefb803a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.141+0000 7fefd12b3700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fefb8038470 0x7fefb803a920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fefbc006fd0 tx=0x7fefbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:23.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.146+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fefc802a980 con 0x7fefcc100090 2023-07-29T16:06:23.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:23.361+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fefcc19bfa0 con 0x7fefcc100090 2023-07-29T16:06:23.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:23 smithi031 ceph-mon[95504]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-29T16:06:23.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:23 smithi031 ceph-mon[95504]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:23.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:23 smithi031 ceph-mon[95504]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:23.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:23.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:23.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.344+0000 7fefc2ffd700 1 -- 172.21.15.31:0/1574238713 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fefc8026070 con 0x7fefcc100090 2023-07-29T16:06:24.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fefb8038470 msgr2=0x7fefb803a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:24.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fefb8038470 0x7fefb803a920 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fefbc006fd0 tx=0x7fefbc006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 msgr2=0x7fefcc1975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc1975f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fefc8004f40 tx=0x7fefc8005020 comp rx=0 tx=0).stop 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 shutdown_connections 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fefb8038470 0x7fefb803a920 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 --2- 172.21.15.31:0/1574238713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefcc100090 0x7fefcc1975f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 >> 172.21.15.31:0/1574238713 conn(0x7fefcc0fb970 msgr2=0x7fefcc1039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:24.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 shutdown_connections 2023-07-29T16:06:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:24.347+0000 7fefd3d18700 1 -- 172.21.15.31:0/1574238713 wait complete. 2023-07-29T16:06:24.344 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-07-29T16:06:24.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:24 smithi031 ceph-mon[95504]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:24.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:24 smithi031 ceph-mon[95504]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:24 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1574238713' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-29T16:06:24.793 INFO:tasks.cephadm:Adding mon.a on smithi031 2023-07-29T16:06:24.794 INFO:tasks.cephadm:Adding mon.b on smithi039 2023-07-29T16:06:24.794 INFO:tasks.cephadm:Adding mon.c on smithi156 2023-07-29T16:06:24.794 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c' 2023-07-29T16:06:25.382 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:25.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:25 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1574238713' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-29T16:06:25.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:25 smithi031 ceph-mon[95504]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.600+0000 7f782da2a700 1 -- 172.21.15.156:0/3234749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 msgr2=0x7f7828100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.600+0000 7f782da2a700 1 --2- 172.21.15.156:0/3234749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828100170 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f781c009910 tx=0x7f781c009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.600+0000 7f782da2a700 1 -- 172.21.15.156:0/3234749352 shutdown_connections 2023-07-29T16:06:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.600+0000 7f782da2a700 1 --2- 172.21.15.156:0/3234749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828100170 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:26.597 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.600+0000 7f782da2a700 1 -- 172.21.15.156:0/3234749352 >> 172.21.15.156:0/3234749352 conn(0x7f7828075750 msgr2=0x7f7828075b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.601+0000 7f782da2a700 1 -- 172.21.15.156:0/3234749352 shutdown_connections 2023-07-29T16:06:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.601+0000 7f782da2a700 1 -- 172.21.15.156:0/3234749352 wait complete. 2023-07-29T16:06:26.598 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.601+0000 7f782da2a700 1 Processor -- start 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.601+0000 7f782da2a700 1 -- start start 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.601+0000 7f782da2a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.601+0000 7f782da2a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7828195860 con 0x7f78280ffda0 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.602+0000 7f7826ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.602+0000 7f7826ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.156:36756/0 (socket says 172.21.15.156:36756) 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.602+0000 7f7826ffd700 1 -- 172.21.15.156:0/2252490652 learned_addr learned my addr 172.21.15.156:0/2252490652 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:06:26.600 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.602+0000 7f7826ffd700 1 -- 172.21.15.156:0/2252490652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c0095f0 con 0x7f78280ffda0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.603+0000 7f7826ffd700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828195320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f781c004f40 tx=0x7f781c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.603+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f781c01d070 con 0x7f78280ffda0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.603+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7828195a60 con 0x7f78280ffda0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.603+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f781c00be20 con 0x7f78280ffda0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.603+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f781c022d00 con 0x7f78280ffda0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.603+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7828195e80 con 0x7f78280ffda0 2023-07-29T16:06:26.601 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.604+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f781c00f460 con 0x7f78280ffda0 2023-07-29T16:06:26.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.604+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7828062380 con 0x7f78280ffda0 2023-07-29T16:06:26.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.604+0000 7f782ca28700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7810038470 0x7f781003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:26.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.604+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f781c04c540 con 0x7f78280ffda0 2023-07-29T16:06:26.602 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.604+0000 7f78267fc700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7810038470 0x7f781003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:26.603 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.607+0000 7f78267fc700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7810038470 0x7f781003a920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7818006fd0 tx=0x7f7818006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:26.605 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.610+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f781c02a770 con 0x7f78280ffda0 2023-07-29T16:06:26.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:26 smithi031 ceph-mon[95504]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:26.796 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.800+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f78280fde20 con 0x7f7810038470 2023-07-29T16:06:26.803 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.808+0000 7f782ca28700 1 -- 172.21.15.156:0/2252490652 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f78280fde20 con 0x7f7810038470 2023-07-29T16:06:26.804 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mon update... 2023-07-29T16:06:26.805 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.809+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7810038470 msgr2=0x7f781003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:26.805 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.809+0000 7f782da2a700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7810038470 0x7f781003a920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7818006fd0 tx=0x7f7818006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:26.806 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.809+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 msgr2=0x7f7828195320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:26.806 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.809+0000 7f782da2a700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828195320 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f781c004f40 tx=0x7f781c005020 comp rx=0 tx=0).stop 2023-07-29T16:06:26.806 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.809+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 shutdown_connections 2023-07-29T16:06:26.806 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.809+0000 7f782da2a700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7810038470 0x7f781003a920 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:26.806 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.810+0000 7f782da2a700 1 --2- 172.21.15.156:0/2252490652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78280ffda0 0x7f7828195320 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:26.806 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.810+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 >> 172.21.15.156:0/2252490652 conn(0x7f7828075750 msgr2=0x7f78280fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:26.807 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.810+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 shutdown_connections 2023-07-29T16:06:26.807 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:26.810+0000 7f782da2a700 1 -- 172.21.15.156:0/2252490652 wait complete. 2023-07-29T16:06:27.152 DEBUG:teuthology.orchestra.run.smithi039:mon.b> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.b.service 2023-07-29T16:06:27.155 DEBUG:teuthology.orchestra.run.smithi156:mon.c> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.c.service 2023-07-29T16:06:27.158 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-29T16:06:27.158 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph mon dump -f json 2023-07-29T16:06:27.190 INFO:journalctl@ceph.mon.c.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:06:27.193 INFO:journalctl@ceph.mon.b.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:06:28.021 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:28.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:28.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c;count:3 2023-07-29T16:06:28.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: Deploying daemon mon.c on smithi156 2023-07-29T16:06:28.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:27 smithi031 ceph-mon[95504]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.776+0000 7fd16e6bc700 1 -- 172.21.15.156:0/842250514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 msgr2=0x7fd1681013c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.776+0000 7fd16e6bc700 1 --2- 172.21.15.156:0/842250514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1681013c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd158009910 tx=0x7fd158009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.776+0000 7fd16e6bc700 1 -- 172.21.15.156:0/842250514 shutdown_connections 2023-07-29T16:06:29.773 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.777+0000 7fd16e6bc700 1 --2- 172.21.15.156:0/842250514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1681013c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:29.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.777+0000 7fd16e6bc700 1 -- 172.21.15.156:0/842250514 >> 172.21.15.156:0/842250514 conn(0x7fd1680fa850 msgr2=0x7fd1680fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:29.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.777+0000 7fd16e6bc700 1 -- 172.21.15.156:0/842250514 shutdown_connections 2023-07-29T16:06:29.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.777+0000 7fd16e6bc700 1 -- 172.21.15.156:0/842250514 wait complete. 2023-07-29T16:06:29.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.777+0000 7fd16e6bc700 1 Processor -- start 2023-07-29T16:06:29.774 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.777+0000 7fd16e6bc700 1 -- start start 2023-07-29T16:06:29.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.778+0000 7fd16e6bc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1680fec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:29.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.778+0000 7fd16e6bc700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168100bd0 con 0x7fd168100ff0 2023-07-29T16:06:29.775 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.778+0000 7fd167fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1680fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.778+0000 7fd167fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1680fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.156:36762/0 (socket says 172.21.15.156:36762) 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.778+0000 7fd167fff700 1 -- 172.21.15.156:0/1429842959 learned_addr learned my addr 172.21.15.156:0/1429842959 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.778+0000 7fd167fff700 1 -- 172.21.15.156:0/1429842959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1580095f0 con 0x7fd168100ff0 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.779+0000 7fd167fff700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1680fec50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd158005850 tx=0x7fd158004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.779+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd15801d070 con 0x7fd168100ff0 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.779+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1680ff190 con 0x7fd168100ff0 2023-07-29T16:06:29.776 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.779+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd158004370 con 0x7fd168100ff0 2023-07-29T16:06:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.779+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd158003ab0 con 0x7fd168100ff0 2023-07-29T16:06:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.779+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1680ff610 con 0x7fd168100ff0 2023-07-29T16:06:29.777 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.780+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7fd15800f460 con 0x7fd168100ff0 2023-07-29T16:06:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.780+0000 7fd1657fa700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd150038470 0x7fd15003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.780+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd15804c150 con 0x7fd168100ff0 2023-07-29T16:06:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.780+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd168107500 con 0x7fd168100ff0 2023-07-29T16:06:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.781+0000 7fd1677fe700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd150038470 0x7fd15003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:29.778 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.781+0000 7fd1677fe700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd150038470 0x7fd15003a920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006fd0 tx=0x7fd15c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:29.782 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:29.786+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd158026070 con 0x7fd168100ff0 2023-07-29T16:06:29.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:29 smithi156 systemd[1]: Starting Ceph mon.c for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:06:30.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.063+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd1680ff850 con 0x7fd168100ff0 2023-07-29T16:06:30.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.064+0000 7fd1657fa700 1 -- 172.21.15.156:0/1429842959 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fd158029770 con 0x7fd168100ff0 2023-07-29T16:06:30.061 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T16:06:30.061 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":1,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","modified":"2023-07-29T16:04:43.061127Z","created":"2023-07-29T16:04:43.061127Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-29T16:06:30.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.066+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd150038470 msgr2=0x7fd15003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:30.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.066+0000 7fd16e6bc700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd150038470 0x7fd15003a920 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006fd0 tx=0x7fd15c006e40 comp rx=0 tx=0).stop 2023-07-29T16:06:30.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.066+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 msgr2=0x7fd1680fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:30.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.066+0000 7fd16e6bc700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1680fec50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd158005850 tx=0x7fd158004ff0 comp rx=0 tx=0).stop 2023-07-29T16:06:30.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.067+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 shutdown_connections 2023-07-29T16:06:30.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.067+0000 7fd16e6bc700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd150038470 0x7fd15003a920 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:30.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.067+0000 7fd16e6bc700 1 --2- 172.21.15.156:0/1429842959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd168100ff0 0x7fd1680fec50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:30.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.067+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 >> 172.21.15.156:0/1429842959 conn(0x7fd1680fa850 msgr2=0x7fd168104640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:30.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.067+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 shutdown_connections 2023-07-29T16:06:30.064 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:30.067+0000 7fd16e6bc700 1 -- 172.21.15.156:0/1429842959 wait complete. 2023-07-29T16:06:30.065 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 1 2023-07-29T16:06:30.198 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:29 smithi156 podman[129192]: 2023-07-29T16:06:30.199 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 bash[129192]: 46964b3f8e61fcb620561ced332560cca45106e2eb772fa857c00490565dc2a7 2023-07-29T16:06:30.199 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: set uid:gid to 167:167 (ceph:ceph) 2023-07-29T16:06:30.199 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc), process ceph-mon, pid 7 2023-07-29T16:06:30.199 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pidfile_write: ignore empty --pid-file 2023-07-29T16:06:30.199 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: load: jerasure load: lrc 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: RocksDB version: 7.9.2 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Git sha 0 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Compile date 2023-07-28 16:56:26 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: DB SUMMARY 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: DB Session ID: LTY4ZE5GMVDORVYF3QEZ 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: CURRENT file: CURRENT 2023-07-29T16:06:30.200 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: IDENTITY file: IDENTITY 2023-07-29T16:06:30.201 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-29T16:06:30.201 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-29T16:06:30.201 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-29T16:06:30.202 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.error_if_exists: 0 2023-07-29T16:06:30.202 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.create_if_missing: 0 2023-07-29T16:06:30.202 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.paranoid_checks: 1 2023-07-29T16:06:30.202 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.env: 0x55ccc97cf720 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.fs: PosixFileSystem 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.info_log: 0x55cccbc51420 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_file_opening_threads: 16 2023-07-29T16:06:30.203 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.statistics: (nil) 2023-07-29T16:06:30.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.use_fsync: 0 2023-07-29T16:06:30.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_log_file_size: 0 2023-07-29T16:06:30.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T16:06:30.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T16:06:30.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.keep_log_file_num: 1000 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.recycle_log_file_num: 0 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.allow_fallocate: 1 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.allow_mmap_reads: 0 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.allow_mmap_writes: 0 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.use_direct_reads: 0 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.create_missing_column_families: 0 2023-07-29T16:06:30.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.db_log_dir: 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.wal_dir: 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T16:06:30.206 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.advise_random_on_open: 1 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.db_write_buffer_size: 0 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.write_buffer_manager: 0x55cccaee25a0 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.rate_limiter: (nil) 2023-07-29T16:06:30.207 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.wal_recovery_mode: 2 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.enable_thread_tracking: 0 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.enable_pipelined_write: 0 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.unordered_write: 0 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T16:06:30.208 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.row_cache: None 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.wal_filter: None 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.allow_ingest_behind: 0 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.two_write_queues: 0 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.manual_wal_flush: 0 2023-07-29T16:06:30.209 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.wal_compression: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.atomic_flush: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.log_readahead_size: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.best_efforts_recovery: 0 2023-07-29T16:06:30.210 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-29T16:06:30.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-29T16:06:30.211 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.allow_data_in_errors: 0 2023-07-29T16:06:30.213 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.db_host_id: __hostname__ 2023-07-29T16:06:30.213 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-29T16:06:30.213 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_background_jobs: 2 2023-07-29T16:06:30.213 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_background_compactions: -1 2023-07-29T16:06:30.213 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_subcompactions: 1 2023-07-29T16:06:30.213 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_total_wal_size: 0 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T16:06:30.214 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_open_files: -1 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bytes_per_sync: 0 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_readahead_size: 0 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_background_flushes: -1 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Compression algorithms supported: 2023-07-29T16:06:30.215 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kZSTD supported: 0 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kXpressCompression supported: 0 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kLZ4HCCompression supported: 1 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kZlibCompression supported: 1 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kSnappyCompression supported: 1 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kLZ4Compression supported: 1 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: kBZip2Compression supported: 0 2023-07-29T16:06:30.216 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.merge_operator: 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_filter: None 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_filter_factory: None 2023-07-29T16:06:30.217 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.sst_partitioner_factory: None 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cccbc51520) 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks: 1 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T16:06:30.218 INFO:journalctl@ceph.mon.c.smithi156.stdout: pin_top_level_index_and_filter: 1 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_type: 0 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_index_type: 0 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_shortening: 1 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: checksum: 4 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: no_block_cache: 0 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache: 0x55cccaf631f0 2023-07-29T16:06:30.219 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_name: BinnedLRUCache 2023-07-29T16:06:30.220 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_options: 2023-07-29T16:06:30.220 INFO:journalctl@ceph.mon.c.smithi156.stdout: capacity : 536870912 2023-07-29T16:06:30.220 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_shard_bits : 4 2023-07-29T16:06:30.220 INFO:journalctl@ceph.mon.c.smithi156.stdout: strict_capacity_limit : 0 2023-07-29T16:06:30.220 INFO:journalctl@ceph.mon.c.smithi156.stdout: high_pri_pool_ratio: 0.000 2023-07-29T16:06:30.220 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_cache_compressed: (nil) 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: persistent_cache: (nil) 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size: 4096 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_size_deviation: 10 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_restart_interval: 16 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: index_block_restart_interval: 1 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: metadata_block_size: 4096 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: partition_filters: 0 2023-07-29T16:06:30.221 INFO:journalctl@ceph.mon.c.smithi156.stdout: use_delta_encoding: 1 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: filter_policy: bloomfilter 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: whole_key_filtering: 1 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: verify_compression: 0 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: read_amp_bytes_per_bit: 0 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: format_version: 5 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_index_compression: 1 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: block_align: 0 2023-07-29T16:06:30.222 INFO:journalctl@ceph.mon.c.smithi156.stdout: max_auto_readahead_size: 262144 2023-07-29T16:06:30.223 INFO:journalctl@ceph.mon.c.smithi156.stdout: prepopulate_block_cache: 0 2023-07-29T16:06:30.223 INFO:journalctl@ceph.mon.c.smithi156.stdout: initial_auto_readahead_size: 8192 2023-07-29T16:06:30.223 INFO:journalctl@ceph.mon.c.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2023-07-29T16:06:30.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.write_buffer_size: 33554432 2023-07-29T16:06:30.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_write_buffer_number: 2 2023-07-29T16:06:30.223 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression: NoCompression 2023-07-29T16:06:30.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression: Disabled 2023-07-29T16:06:30.224 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.prefix_extractor: nullptr 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.num_levels: 7 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T16:06:30.225 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-29T16:06:30.226 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.level: 32767 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.strategy: 0 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-29T16:06:30.227 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.enabled: false 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.target_file_size_base: 67108864 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T16:06:30.228 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T16:06:30.229 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T16:06:30.230 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T16:06:30.230 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T16:06:30.230 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T16:06:30.230 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.arena_block_size: 1048576 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.disable_auto_compactions: 0 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T16:06:30.231 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T16:06:30.232 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.table_properties_collectors: 2023-07-29T16:06:30.233 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.inplace_update_support: 0 2023-07-29T16:06:30.233 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T16:06:30.233 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T16:06:30.233 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T16:06:30.233 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.bloom_locality: 0 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.max_successive_merges: 0 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.paranoid_file_checks: 0 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.force_consistency_checks: 1 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.report_bg_io_stats: 0 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.ttl: 2592000 2023-07-29T16:06:30.234 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.enable_blob_files: false 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.min_blob_size: 0 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.blob_file_size: 268435456 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-29T16:06:30.235 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.blob_file_starting_level: 0 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-29T16:06:30.236 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6862adbe-a851-4853-87ba-7ceda84d6bf7 2023-07-29T16:06:30.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646790184371, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-29T16:06:30.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-29T16:06:30.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646790184777, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1690646790, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6862adbe-a851-4853-87ba-7ceda84d6bf7", "db_session_id": "LTY4ZE5GMVDORVYF3QEZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-29T16:06:30.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646790184844, "job": 1, "event": "recovery_finished"} 2023-07-29T16:06:30.237 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55cccb000000 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: DB pointer 0x55cccafec000 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** DB Stats ** 2023-07-29T16:06:30.238 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-29T16:06:30.239 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2023-07-29T16:06:30.240 INFO:journalctl@ceph.mon.c.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T16:06:30.240 INFO:journalctl@ceph.mon.c.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-29T16:06:30.240 INFO:journalctl@ceph.mon.c.smithi156.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** Compaction Stats [default] ** 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:30.241 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:06:30.242 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache BinnedLRUCache@0x55cccaf631f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-29T16:06:30.243 INFO:journalctl@ceph.mon.c.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: using public_addr v2:172.21.15.156:0/0 -> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: starting mon.c rank -1 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(???) e0 preinit fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).mds e1 new map 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-29T16:06:30.244 INFO:journalctl@ceph.mon.c.smithi156.stdout: e1 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout: legacy client fscid: -1 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout: 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout: No filesystems configured 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-29T16:06:30.245 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:06:30.246 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.247 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi039:/etc/ceph/ceph.conf 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.248 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Deploying cephadm binary to smithi156 2023-07-29T16:06:30.249 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:30.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Added host smithi156 2023-07-29T16:06:30.250 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:06:30.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.251 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:30.252 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.253 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1574238713' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1574238713' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c;count:3 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.254 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:30.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:30.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:30.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:30.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:30.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:30.255 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: Deploying daemon mon.c on smithi156 2023-07-29T16:06:30.256 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:30.256 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-29T16:06:30.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 ceph-mon[129245]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-29T16:06:30.510 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:30 smithi156 systemd[1]: Started Ceph mon.c for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:06:31.512 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-29T16:06:31.512 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph mon dump -f json 2023-07-29T16:06:32.066 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:06:35.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:06:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: mon.a calling monitor election 2023-07-29T16:06:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:35.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: mon.c calling monitor election 2023-07-29T16:06:35.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:35.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-29T16:06:35.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:35.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: fsmap 2023-07-29T16:06:35.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:35.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: mgrmap e13: x(active, since 46s) 2023-07-29T16:06:35.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: overall HEALTH_OK 2023-07-29T16:06:35.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:35.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:35.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:35.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:35.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:06:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: mon.a calling monitor election 2023-07-29T16:06:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: mon.c calling monitor election 2023-07-29T16:06:35.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:35.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:35.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-29T16:06:35.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:35.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: fsmap 2023-07-29T16:06:35.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:35.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: mgrmap e13: x(active, since 46s) 2023-07-29T16:06:35.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: overall HEALTH_OK 2023-07-29T16:06:35.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:35.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:35.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:35.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:35.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:36.216 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 -- 172.21.15.156:0/911434893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 msgr2=0x7fcba0007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:36.217 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/911434893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcba0007c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcbb80fed80 tx=0x7fcbac00feb0 comp rx=0 tx=0).stop 2023-07-29T16:06:36.217 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 -- 172.21.15.156:0/911434893 shutdown_connections 2023-07-29T16:06:36.217 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/911434893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcba0007c50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:36.217 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 -- 172.21.15.156:0/911434893 >> 172.21.15.156:0/911434893 conn(0x7fcbb80fa970 msgr2=0x7fcbb80fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:36.217 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 -- 172.21.15.156:0/911434893 shutdown_connections 2023-07-29T16:06:36.217 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.220+0000 7fcbbe36e700 1 -- 172.21.15.156:0/911434893 wait complete. 2023-07-29T16:06:36.218 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.221+0000 7fcbbe36e700 1 Processor -- start 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.222+0000 7fcbbe36e700 1 -- start start 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.222+0000 7fcbbe36e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcbb8193250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.222+0000 7fcbbe36e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.222+0000 7fcbbe36e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb8193d10 con 0x7fcba00057d0 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.222+0000 7fcbbe36e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb8193e80 con 0x7fcbb8193790 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:36.219 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40414/0 (socket says 172.21.15.156:40414) 2023-07-29T16:06:36.220 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 -- 172.21.15.156:0/3020620709 learned_addr learned my addr 172.21.15.156:0/3020620709 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:06:36.220 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 -- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 msgr2=0x7fcbb8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-29T16:06:36.220 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 -- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 msgr2=0x7fcbb8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-29T16:06:36.220 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-29T16:06:36.220 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb77fe700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-29T16:06:36.220 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb7fff700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcbb8193250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:36.221 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb7fff700 1 -- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 msgr2=0x7fcbb8197c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:06:36.221 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb7fff700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:36.222 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.223+0000 7fcbb7fff700 1 -- 172.21.15.156:0/3020620709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac00f900 con 0x7fcba00057d0 2023-07-29T16:06:36.222 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.224+0000 7fcbb7fff700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcbb8193250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcbac011510 tx=0x7fcbac005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:36.222 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.224+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fcbac013b00 con 0x7fcba00057d0 2023-07-29T16:06:36.222 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.224+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb81981a0 con 0x7fcba00057d0 2023-07-29T16:06:36.222 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.224+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbac0044f0 con 0x7fcba00057d0 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.224+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fcbac003b50 con 0x7fcba00057d0 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.224+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb81986a0 con 0x7fcba00057d0 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.225+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7fcbac011590 con 0x7fcba00057d0 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.225+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb8062380 con 0x7fcba00057d0 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.225+0000 7fcbb57fa700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcba4038250 0x7fcba403a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.225+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcbac04d830 con 0x7fcba00057d0 2023-07-29T16:06:36.223 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.226+0000 7fcbb77fe700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcba4038250 0x7fcba403a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:36.224 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.226+0000 7fcbb77fe700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcba4038250 0x7fcba403a700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009180 tx=0x7fcba8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:36.226 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.231+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcbac0109c0 con 0x7fcba00057d0 2023-07-29T16:06:36.489 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.493+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbb8194680 con 0x7fcba00057d0 2023-07-29T16:06:36.490 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.494+0000 7fcbb57fa700 1 -- 172.21.15.156:0/3020620709 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fcbac0103d0 con 0x7fcba00057d0 2023-07-29T16:06:36.490 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T16:06:36.490 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","modified":"2023-07-29T16:06:30.211125Z","created":"2023-07-29T16:04:43.061127Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-29T16:06:36.492 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcba4038250 msgr2=0x7fcba403a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:36.492 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcba4038250 0x7fcba403a700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcba8009180 tx=0x7fcba8008040 comp rx=0 tx=0).stop 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 msgr2=0x7fcbb8193250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcbb8193250 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcbac011510 tx=0x7fcbac005250 comp rx=0 tx=0).stop 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 shutdown_connections 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcba4038250 0x7fcba403a700 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcba00057d0 0x7fcbb8193250 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 --2- 172.21.15.156:0/3020620709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbb8193790 0x7fcbb8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:36.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 >> 172.21.15.156:0/3020620709 conn(0x7fcbb80fa970 msgr2=0x7fcbb80fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:36.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 shutdown_connections 2023-07-29T16:06:36.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:36.496+0000 7fcbbe36e700 1 -- 172.21.15.156:0/3020620709 wait complete. 2023-07-29T16:06:36.494 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 2 2023-07-29T16:06:36.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:36 smithi156 ceph-mon[129245]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:36.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:36 smithi156 ceph-mon[129245]: Deploying daemon mon.b on smithi039 2023-07-29T16:06:36.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:36.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:36 smithi031 ceph-mon[95504]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:36.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:36 smithi031 ceph-mon[95504]: Deploying daemon mon.b on smithi039 2023-07-29T16:06:36.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:37.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:37 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/3020620709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:06:37.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:37 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/3020620709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:06:37.680 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:06:37 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:06:37.210+0000 7f3460e7c700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-07-29T16:06:37.914 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-29T16:06:37.914 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph mon dump -f json 2023-07-29T16:06:38.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:37 smithi039 systemd[1]: Starting Ceph mon.b for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:06:38.339 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 podman[128818]: 2023-07-29T16:06:38.339 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: set uid:gid to 167:167 (ceph:ceph) 2023-07-29T16:06:38.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: ceph version 18.1.3-11-g0484097b (0484097b0572e96f42cf6402cbe0ac8dcb046577) reef (rc), process ceph-mon, pid 7 2023-07-29T16:06:38.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 bash[128818]: 4c01eaf5f917fb7194e4892e9cfb14fd597cbff23df9d6d59b267594e7201f59 2023-07-29T16:06:38.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pidfile_write: ignore empty --pid-file 2023-07-29T16:06:38.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: load: jerasure load: lrc 2023-07-29T16:06:38.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: RocksDB version: 7.9.2 2023-07-29T16:06:38.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Git sha 0 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Compile date 2023-07-28 16:56:26 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: DB SUMMARY 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: DB Session ID: TB409PIWT8GXTBRWT4KP 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: CURRENT file: CURRENT 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: IDENTITY file: IDENTITY 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-29T16:06:38.341 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-29T16:06:38.342 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-29T16:06:38.342 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.error_if_exists: 0 2023-07-29T16:06:38.342 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.create_if_missing: 0 2023-07-29T16:06:38.342 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.paranoid_checks: 1 2023-07-29T16:06:38.342 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-29T16:06:38.343 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-29T16:06:38.343 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-29T16:06:38.343 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.env: 0x5622b8731720 2023-07-29T16:06:38.343 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.fs: PosixFileSystem 2023-07-29T16:06:38.343 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.info_log: 0x5622baecb420 2023-07-29T16:06:38.344 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_file_opening_threads: 16 2023-07-29T16:06:38.344 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.statistics: (nil) 2023-07-29T16:06:38.344 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.use_fsync: 0 2023-07-29T16:06:38.344 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_log_file_size: 0 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.keep_log_file_num: 1000 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.recycle_log_file_num: 0 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.allow_fallocate: 1 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.allow_mmap_reads: 0 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.allow_mmap_writes: 0 2023-07-29T16:06:38.345 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.use_direct_reads: 0 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.create_missing_column_families: 0 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.db_log_dir: 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.wal_dir: 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-29T16:06:38.346 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.advise_random_on_open: 1 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.db_write_buffer_size: 0 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.write_buffer_manager: 0x5622ba15c5a0 2023-07-29T16:06:38.347 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.rate_limiter: (nil) 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.wal_recovery_mode: 2 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.enable_thread_tracking: 0 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.enable_pipelined_write: 0 2023-07-29T16:06:38.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.unordered_write: 0 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.row_cache: None 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.wal_filter: None 2023-07-29T16:06:38.349 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.allow_ingest_behind: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.two_write_queues: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.manual_wal_flush: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.wal_compression: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.atomic_flush: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-29T16:06:38.350 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-29T16:06:38.351 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.log_readahead_size: 0 2023-07-29T16:06:38.351 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-29T16:06:38.351 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.best_efforts_recovery: 0 2023-07-29T16:06:38.351 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-29T16:06:38.351 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-29T16:06:38.351 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.allow_data_in_errors: 0 2023-07-29T16:06:38.353 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.db_host_id: __hostname__ 2023-07-29T16:06:38.353 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_background_jobs: 2 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_background_compactions: -1 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_subcompactions: 1 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-29T16:06:38.354 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_total_wal_size: 0 2023-07-29T16:06:38.355 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-29T16:06:38.355 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-29T16:06:38.355 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-29T16:06:38.355 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-29T16:06:38.355 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_open_files: -1 2023-07-29T16:06:38.355 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bytes_per_sync: 0 2023-07-29T16:06:38.356 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-29T16:06:38.356 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-29T16:06:38.356 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_readahead_size: 0 2023-07-29T16:06:38.356 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_background_flushes: -1 2023-07-29T16:06:38.356 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Compression algorithms supported: 2023-07-29T16:06:38.356 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kZSTD supported: 0 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kXpressCompression supported: 0 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kLZ4HCCompression supported: 1 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kZlibCompression supported: 1 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kSnappyCompression supported: 1 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kLZ4Compression supported: 1 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: kBZip2Compression supported: 0 2023-07-29T16:06:38.357 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.merge_operator: 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_filter: None 2023-07-29T16:06:38.358 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_filter_factory: None 2023-07-29T16:06:38.359 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.sst_partitioner_factory: None 2023-07-29T16:06:38.359 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-29T16:06:38.359 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-29T16:06:38.359 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5622baecb520) 2023-07-29T16:06:38.360 INFO:journalctl@ceph.mon.b.smithi039.stdout: cache_index_and_filter_blocks: 1 2023-07-29T16:06:38.360 INFO:journalctl@ceph.mon.b.smithi039.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-29T16:06:38.360 INFO:journalctl@ceph.mon.b.smithi039.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-29T16:06:38.360 INFO:journalctl@ceph.mon.b.smithi039.stdout: pin_top_level_index_and_filter: 1 2023-07-29T16:06:38.360 INFO:journalctl@ceph.mon.b.smithi039.stdout: index_type: 0 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: data_block_index_type: 0 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: index_shortening: 1 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: checksum: 4 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: no_block_cache: 0 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_cache: 0x5622ba1dd350 2023-07-29T16:06:38.361 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_cache_name: BinnedLRUCache 2023-07-29T16:06:38.362 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_cache_options: 2023-07-29T16:06:38.362 INFO:journalctl@ceph.mon.b.smithi039.stdout: capacity : 536870912 2023-07-29T16:06:38.362 INFO:journalctl@ceph.mon.b.smithi039.stdout: num_shard_bits : 4 2023-07-29T16:06:38.362 INFO:journalctl@ceph.mon.b.smithi039.stdout: strict_capacity_limit : 0 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: high_pri_pool_ratio: 0.000 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_cache_compressed: (nil) 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: persistent_cache: (nil) 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_size: 4096 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_size_deviation: 10 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_restart_interval: 16 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: index_block_restart_interval: 1 2023-07-29T16:06:38.363 INFO:journalctl@ceph.mon.b.smithi039.stdout: metadata_block_size: 4096 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: partition_filters: 0 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: use_delta_encoding: 1 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: filter_policy: bloomfilter 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: whole_key_filtering: 1 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: verify_compression: 0 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: read_amp_bytes_per_bit: 0 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: format_version: 5 2023-07-29T16:06:38.364 INFO:journalctl@ceph.mon.b.smithi039.stdout: enable_index_compression: 1 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout: block_align: 0 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout: max_auto_readahead_size: 262144 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout: prepopulate_block_cache: 0 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout: initial_auto_readahead_size: 8192 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout: num_file_reads_for_auto_readahead: 2 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.write_buffer_size: 33554432 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_write_buffer_number: 2 2023-07-29T16:06:38.365 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression: NoCompression 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression: Disabled 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.prefix_extractor: nullptr 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.num_levels: 7 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-29T16:06:38.366 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-29T16:06:38.367 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-29T16:06:38.367 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-29T16:06:38.367 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-29T16:06:38.367 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-29T16:06:38.368 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.level: 32767 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.strategy: 0 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.enabled: false 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-29T16:06:38.369 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.target_file_size_base: 67108864 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-29T16:06:38.370 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-29T16:06:38.371 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.arena_block_size: 1048576 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.disable_auto_compactions: 0 2023-07-29T16:06:38.372 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-29T16:06:38.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.table_properties_collectors: 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.inplace_update_support: 0 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-29T16:06:38.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.bloom_locality: 0 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.max_successive_merges: 0 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.paranoid_file_checks: 0 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.force_consistency_checks: 1 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.report_bg_io_stats: 0 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.ttl: 2592000 2023-07-29T16:06:38.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.enable_blob_files: false 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.min_blob_size: 0 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.blob_file_size: 268435456 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-29T16:06:38.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-29T16:06:38.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-29T16:06:38.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-29T16:06:38.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-29T16:06:38.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.blob_file_starting_level: 0 2023-07-29T16:06:38.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-29T16:06:38.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-29T16:06:38.378 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-29T16:06:38.378 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b2543d0a-965f-458e-9811-2867235da771 2023-07-29T16:06:38.378 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646798282981, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-29T16:06:38.378 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-29T16:06:38.378 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646798283394, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1690646798, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b2543d0a-965f-458e-9811-2867235da771", "db_session_id": "TB409PIWT8GXTBRWT4KP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-29T16:06:38.378 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: EVENT_LOG_v1 {"time_micros": 1690646798283459, "job": 1, "event": "recovery_finished"} 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5622ba27a000 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: DB pointer 0x5622ba266000 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-29T16:06:38.379 INFO:journalctl@ceph.mon.b.smithi039.stdout: ** DB Stats ** 2023-07-29T16:06:38.380 INFO:journalctl@ceph.mon.b.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:06:38.380 INFO:journalctl@ceph.mon.b.smithi039.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-29T16:06:38.380 INFO:journalctl@ceph.mon.b.smithi039.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:06:38.380 INFO:journalctl@ceph.mon.b.smithi039.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:06:38.380 INFO:journalctl@ceph.mon.b.smithi039.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-29T16:06:38.380 INFO:journalctl@ceph.mon.b.smithi039.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: ** Compaction Stats [default] ** 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:38.381 INFO:journalctl@ceph.mon.b.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: ** Compaction Stats [default] ** 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-29T16:06:38.382 INFO:journalctl@ceph.mon.b.smithi039.stdout: 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-29T16:06:38.383 INFO:journalctl@ceph.mon.b.smithi039.stdout: Cumulative compaction: 0.00 GB write, 0.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout: Interval compaction: 0.00 GB write, 0.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout: Block cache BinnedLRUCache@0x5622ba1dd350#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout: 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-29T16:06:38.384 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: using public_addr v2:172.21.15.39:0/0 -> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: starting mon.b rank -1 at public addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] at bind addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(???) e0 preinit fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).mds e1 new map 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout: e1 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T16:06:38.385 INFO:journalctl@ceph.mon.b.smithi039.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout: legacy client fscid: -1 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout: 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout: No filesystems configured 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-29T16:06:38.386 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.387 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.388 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi039:/etc/ceph/ceph.conf 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.389 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Deploying cephadm binary to smithi156 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.390 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Added host smithi156 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.391 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:06:38.392 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.393 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.client.admin.keyring 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1574238713' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1574238713' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.394 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c;count:3 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.395 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Deploying daemon mon.c on smithi156 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.a calling monitor election 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.396 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.c calling monitor election 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.397 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: fsmap 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mgrmap e13: x(active, since 46s) 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: overall HEALTH_OK 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.398 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: Deploying daemon mon.b on smithi039 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/3020620709' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.399 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-29T16:06:38.400 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 ceph-mon[128858]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-29T16:06:38.400 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:38 smithi039 systemd[1]: Started Ceph mon.b for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:06:38.469 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:06:38.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:38 smithi156 ceph-mon[129245]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:38.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:38 smithi031 ceph-mon[95504]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:43.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: mon.c calling monitor election 2023-07-29T16:06:43.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: mon.a calling monitor election 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: fsmap 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: mgrmap e13: x(active, since 54s) 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: mon.b (rank 2) addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] is down (out of quorum) 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:43.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:43.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: mon.c calling monitor election 2023-07-29T16:06:43.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:06:43.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: mon.a calling monitor election 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: fsmap 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: mgrmap e13: x(active, since 54s) 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: mon.b (rank 2) addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] is down (out of quorum) 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:44.055 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/2303764288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d500035c0 msgr2=0x7f6d50005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:44.055 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/2303764288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d500035c0 0x7f6d50005a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6d640ffea0 tx=0x7f6d5400fe60 comp rx=0 tx=0).stop 2023-07-29T16:06:44.055 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/2303764288 shutdown_connections 2023-07-29T16:06:44.056 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/2303764288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d500035c0 0x7f6d50005a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.056 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/2303764288 >> 172.21.15.156:0/2303764288 conn(0x7f6d640fba50 msgr2=0x7f6d640fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:44.056 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/2303764288 shutdown_connections 2023-07-29T16:06:44.056 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.060+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/2303764288 wait complete. 2023-07-29T16:06:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 Processor -- start 2023-07-29T16:06:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 -- start start 2023-07-29T16:06:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d64195b60 0x7f6d64199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d6419a510 0x7f6d6419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d64196230 con 0x7f6d6419a510 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641a2280 con 0x7f6d64195b60 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6bc5c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d641a23c0 con 0x7f6d500035c0 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40436/0 (socket says 172.21.15.156:40436) 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 -- 172.21.15.156:0/1255076795 learned_addr learned my addr 172.21.15.156:0/1255076795 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 -- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 msgr2=0x7f6d64195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-29T16:06:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 -- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 msgr2=0x7f6d64195620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d699f8700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.061+0000 7f6d6a1f9700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d6419a510 0x7f6d6419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d691f7700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d64195b60 0x7f6d64199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6a1f9700 1 -- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 msgr2=0x7f6d64195620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6a1f9700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6a1f9700 1 -- 172.21.15.156:0/1255076795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d64195b60 msgr2=0x7f6d64199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6a1f9700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d64195b60 0x7f6d64199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6a1f9700 1 -- 172.21.15.156:0/1255076795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5400f8b0 con 0x7f6d6419a510 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6a1f9700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d6419a510 0x7f6d6419a9b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00bd80 tx=0x7f6d5c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c00cd10 con 0x7f6d6419a510 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d641a2670 con 0x7f6d6419a510 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d5c00ce70 con 0x7f6d6419a510 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c012880 con 0x7f6d6419a510 2023-07-29T16:06:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.062+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d641a2bc0 con 0x7f6d6419a510 2023-07-29T16:06:44.062 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.063+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f6d5c014440 con 0x7f6d6419a510 2023-07-29T16:06:44.062 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.063+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6418fad0 con 0x7f6d6419a510 2023-07-29T16:06:44.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.063+0000 7f6d5affd700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6d480388f0 0x7f6d4803ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:44.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.063+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6d5c04a8e0 con 0x7f6d6419a510 2023-07-29T16:06:44.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.064+0000 7f6d699f8700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6d480388f0 0x7f6d4803ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:44.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.064+0000 7f6d699f8700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6d480388f0 0x7f6d4803ada0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6d54020040 tx=0x7f6d54001e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:44.063 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.069+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6d5c0146f0 con 0x7f6d6419a510 2023-07-29T16:06:44.358 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.363+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d64062380 con 0x7f6d6419a510 2023-07-29T16:06:44.358 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.364+0000 7f6d5affd700 1 -- 172.21.15.156:0/1255076795 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7f6d5c019030 con 0x7f6d6419a510 2023-07-29T16:06:44.359 INFO:teuthology.orchestra.run.smithi156.stdout: 2023-07-29T16:06:44.359 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","modified":"2023-07-29T16:06:38.310145Z","created":"2023-07-29T16:04:43.061127Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-29T16:06:44.361 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6d480388f0 msgr2=0x7f6d4803ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:44.361 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6d480388f0 0x7f6d4803ada0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6d54020040 tx=0x7f6d54001e70 comp rx=0 tx=0).stop 2023-07-29T16:06:44.361 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d6419a510 msgr2=0x7f6d6419a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:44.361 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d6419a510 0x7f6d6419a9b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00bd80 tx=0x7f6d5c00be60 comp rx=0 tx=0).stop 2023-07-29T16:06:44.361 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 shutdown_connections 2023-07-29T16:06:44.361 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6d480388f0 0x7f6d4803ada0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.362 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d6419a510 0x7f6d6419a9b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.362 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d500035c0 0x7f6d64195620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.362 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 --2- 172.21.15.156:0/1255076795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d64195b60 0x7f6d64199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:44.362 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 >> 172.21.15.156:0/1255076795 conn(0x7f6d640fba50 msgr2=0x7f6d640fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:44.362 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 shutdown_connections 2023-07-29T16:06:44.362 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:44.366+0000 7f6d6bc5c700 1 -- 172.21.15.156:0/1255076795 wait complete. 2023-07-29T16:06:44.363 INFO:teuthology.orchestra.run.smithi156.stderr:dumped monmap epoch 3 2023-07-29T16:06:44.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:44 smithi031 ceph-mon[95504]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:44.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:44.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:44.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:44.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:44.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:44 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/1255076795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:06:44.732 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:44 smithi156 ceph-mon[129245]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:44.732 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:44.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:44.733 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:44.734 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:44.734 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:44 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/1255076795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:06:44.735 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-29T16:06:44.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph config generate-minimal-conf 2023-07-29T16:06:45.496 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:45.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: mon.c calling monitor election 2023-07-29T16:06:45.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:06:45.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: mon.a calling monitor election 2023-07-29T16:06:45.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:06:45.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:45.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-29T16:06:45.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:45.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: fsmap 2023-07-29T16:06:45.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:45.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: mgrmap e13: x(active, since 54s) 2023-07-29T16:06:45.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-29T16:06:45.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-29T16:06:45.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-29T16:06:45.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: mon.b (rank 2) addr [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] is down (out of quorum) 2023-07-29T16:06:45.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:45.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:45.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:45.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:45.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:45.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:45.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:45.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:45.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:45 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/1255076795' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-29T16:06:46.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: mon.b calling monitor election 2023-07-29T16:06:46.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Updating smithi031:/etc/ceph/ceph.conf 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Updating smithi039:/etc/ceph/ceph.conf 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: mon.b calling monitor election 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: mon.c calling monitor election 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: mon.a calling monitor election 2023-07-29T16:06:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: fsmap 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: mgrmap e13: x(active, since 56s) 2023-07-29T16:06:46.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-29T16:06:46.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: Cluster is now healthy 2023-07-29T16:06:46.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: overall HEALTH_OK 2023-07-29T16:06:46.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: mon.b calling monitor election 2023-07-29T16:06:46.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Updating smithi031:/etc/ceph/ceph.conf 2023-07-29T16:06:46.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Updating smithi039:/etc/ceph/ceph.conf 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: mon.b calling monitor election 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: mon.c calling monitor election 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: mon.a calling monitor election 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-29T16:06:46.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: fsmap 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: mgrmap e13: x(active, since 56s) 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: Cluster is now healthy 2023-07-29T16:06:46.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: overall HEALTH_OK 2023-07-29T16:06:46.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: mon.b calling monitor election 2023-07-29T16:06:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Updating smithi031:/etc/ceph/ceph.conf 2023-07-29T16:06:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Updating smithi039:/etc/ceph/ceph.conf 2023-07-29T16:06:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Updating smithi156:/etc/ceph/ceph.conf 2023-07-29T16:06:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: mon.b calling monitor election 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: mon.c calling monitor election 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: mon.a calling monitor election 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Updating smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Updating smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0],c=[v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0]} removed_ranks: {} 2023-07-29T16:06:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: fsmap 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: osdmap e4: 0 total, 0 up, 0 in 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: mgrmap e13: x(active, since 56s) 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: Cluster is now healthy 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: overall HEALTH_OK 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:46.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.036+0000 7ff70f677700 1 -- 172.21.15.31:0/1839347456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100090 msgr2=0x7ff708100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:47.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.036+0000 7ff70f677700 1 --2- 172.21.15.31:0/1839347456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100090 0x7ff708100460 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff704009910 tx=0x7ff704009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:47.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.037+0000 7ff70f677700 1 -- 172.21.15.31:0/1839347456 shutdown_connections 2023-07-29T16:06:47.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.037+0000 7ff70f677700 1 --2- 172.21.15.31:0/1839347456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff708100090 0x7ff708100460 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.037+0000 7ff70f677700 1 -- 172.21.15.31:0/1839347456 >> 172.21.15.31:0/1839347456 conn(0x7ff7080fb970 msgr2=0x7ff7080fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:47.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.037+0000 7ff70f677700 1 -- 172.21.15.31:0/1839347456 shutdown_connections 2023-07-29T16:06:47.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.037+0000 7ff70f677700 1 -- 172.21.15.31:0/1839347456 wait complete. 2023-07-29T16:06:47.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 Processor -- start 2023-07-29T16:06:47.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 -- start start 2023-07-29T16:06:47.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff708100090 0x7ff70819d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 0x7ff708197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff708197a50 0x7ff708197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70819e010 con 0x7ff70819d970 2023-07-29T16:06:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081a4300 con 0x7ff708197a50 2023-07-29T16:06:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.038+0000 7ff70f677700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081a4440 con 0x7ff708100090 2023-07-29T16:06:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 0x7ff708197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 0x7ff708197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50782/0 (socket says 172.21.15.31:50782) 2023-07-29T16:06:47.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 -- 172.21.15.31:0/685147801 learned_addr learned my addr 172.21.15.31:0/685147801 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:47.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70dc14700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff708197a50 0x7ff708197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70d413700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff708100090 0x7ff70819d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 -- 172.21.15.31:0/685147801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff708100090 msgr2=0x7ff70819d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff708100090 0x7ff70819d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 -- 172.21.15.31:0/685147801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff708197a50 msgr2=0x7ff708197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff708197a50 0x7ff708197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 -- 172.21.15.31:0/685147801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7040095f0 con 0x7ff70819d970 2023-07-29T16:06:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70d413700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff708100090 0x7ff70819d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:06:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.039+0000 7ff70cc12700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 0x7ff708197510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800b780 tx=0x7ff6f800ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.040+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8004230 con 0x7ff70819d970 2023-07-29T16:06:47.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.040+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6f8004390 con 0x7ff70819d970 2023-07-29T16:06:47.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.040+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7081a4720 con 0x7ff70819d970 2023-07-29T16:06:47.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.040+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8011ad0 con 0x7ff70819d970 2023-07-29T16:06:47.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.040+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7081a4c70 con 0x7ff70819d970 2023-07-29T16:06:47.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.041+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7ff6f8011c30 con 0x7ff70819d970 2023-07-29T16:06:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.041+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff708062380 con 0x7ff70819d970 2023-07-29T16:06:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.041+0000 7ff6fe7fc700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff6f4038940 0x7ff6f403adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.041+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff6f804a860 con 0x7ff70819d970 2023-07-29T16:06:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.042+0000 7ff70d413700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff6f4038940 0x7ff6f403adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:47.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.042+0000 7ff70d413700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff6f4038940 0x7ff6f403adf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff704006010 tx=0x7ff70400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:47.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.047+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff6f801a400 con 0x7ff70819d970 2023-07-29T16:06:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.257+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff7081a4f90 con 0x7ff70819d970 2023-07-29T16:06:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.257+0000 7ff6fe7fc700 1 -- 172.21.15.31:0/685147801 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7ff6f8018350 con 0x7ff70819d970 2023-07-29T16:06:47.253 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:06:47.253 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-07-29T16:06:47.253 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:06:47.254 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] 2023-07-29T16:06:47.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff6f4038940 msgr2=0x7ff6f403adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:47.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff6f4038940 0x7ff6f403adf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff704006010 tx=0x7ff70400b540 comp rx=0 tx=0).stop 2023-07-29T16:06:47.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 msgr2=0x7ff708197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 0x7ff708197510 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800b780 tx=0x7ff6f800ba90 comp rx=0 tx=0).stop 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 shutdown_connections 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff6f4038940 0x7ff6f403adf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff708197a50 0x7ff708197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff708100090 0x7ff70819d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 --2- 172.21.15.31:0/685147801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff70819d970 0x7ff708197510 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 >> 172.21.15.31:0/685147801 conn(0x7ff7080fb970 msgr2=0x7ff7081039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:47.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 shutdown_connections 2023-07-29T16:06:47.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:47.260+0000 7ff70f677700 1 -- 172.21.15.31:0/685147801 wait complete. 2023-07-29T16:06:47.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: Updating smithi031:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: Reconfiguring mon.a (unknown last config time)... 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:47.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:47.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:47.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:47 smithi031 ceph-mon[95504]: Reconfiguring daemon mon.a on smithi031 2023-07-29T16:06:47.521 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: Updating smithi031:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: Reconfiguring mon.a (unknown last config time)... 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:47.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:47.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:47.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:47 smithi039 ceph-mon[128858]: Reconfiguring daemon mon.a on smithi031 2023-07-29T16:06:47.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: Updating smithi031:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/config/ceph.conf 2023-07-29T16:06:47.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:47.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:06:47.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: Reconfiguring mon.a (unknown last config time)... 2023-07-29T16:06:47.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:47.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:47.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:47.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:47 smithi156 ceph-mon[129245]: Reconfiguring daemon mon.a on smithi031 2023-07-29T16:06:48.188 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-29T16:06:48.188 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:06:48.188 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2023-07-29T16:06:48.263 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:06:48.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:48.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:48.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/685147801' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:48.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:48.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:48.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:48.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:48.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:48.373 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:06:48.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.conf 2023-07-29T16:06:48.404 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:06:48.405 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:48.473 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:06:48.473 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2023-07-29T16:06:48.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:48.504 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/685147801' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:48.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:48.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:48.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:48.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:48.505 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:48.509 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:06:48.510 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-29T16:06:48.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:48.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/685147801' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:48.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:48.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:48.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:48.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:48.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:48.582 INFO:tasks.cephadm:Adding mgr.x on smithi031 2023-07-29T16:06:48.582 INFO:tasks.cephadm:Adding mgr.y on smithi039 2023-07-29T16:06:48.583 INFO:tasks.cephadm:Adding mgr.z on smithi156 2023-07-29T16:06:48.583 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch apply mgr '3;smithi031=x;smithi039=y;smithi156=z' 2023-07-29T16:06:49.175 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:06:49.521 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:49 smithi039 ceph-mon[128858]: Reconfiguring mon.b (monmap changed)... 2023-07-29T16:06:49.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:49 smithi039 ceph-mon[128858]: Reconfiguring daemon mon.b on smithi039 2023-07-29T16:06:49.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:49 smithi156 ceph-mon[129245]: Reconfiguring mon.b (monmap changed)... 2023-07-29T16:06:49.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:49 smithi156 ceph-mon[129245]: Reconfiguring daemon mon.b on smithi039 2023-07-29T16:06:49.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:49 smithi031 ceph-mon[95504]: Reconfiguring mon.b (monmap changed)... 2023-07-29T16:06:49.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:49 smithi031 ceph-mon[95504]: Reconfiguring daemon mon.b on smithi039 2023-07-29T16:06:50.492 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 -- 172.21.15.156:0/4035258294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41301009d0 msgr2=0x7f4130102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 --2- 172.21.15.156:0/4035258294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41301009d0 0x7f4130102db0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4124009910 tx=0x7f4124009c20 comp rx=0 tx=0).stop 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 -- 172.21.15.156:0/4035258294 shutdown_connections 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 --2- 172.21.15.156:0/4035258294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41301009d0 0x7f4130102db0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 -- 172.21.15.156:0/4035258294 >> 172.21.15.156:0/4035258294 conn(0x7f41300fa3f0 msgr2=0x7f41300fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 -- 172.21.15.156:0/4035258294 shutdown_connections 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 -- 172.21.15.156:0/4035258294 wait complete. 2023-07-29T16:06:50.493 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 Processor -- start 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.491+0000 7f413593d700 1 -- start start 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 0x7f41301973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4130197910 0x7f413019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4130197d80 0x7f413019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413019f690 con 0x7f4130197d80 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413019f800 con 0x7f4130197910 2023-07-29T16:06:50.494 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413019f9c0 con 0x7f41301009d0 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 0x7f41301973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 0x7f41301973d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40446/0 (socket says 172.21.15.156:40446) 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 -- 172.21.15.156:0/238873530 learned_addr learned my addr 172.21.15.156:0/238873530 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412f7fe700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4130197d80 0x7f413019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412e7fc700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4130197910 0x7f413019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 -- 172.21.15.156:0/238873530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4130197910 msgr2=0x7f413019c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4130197910 0x7f413019c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.495 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 -- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4130197d80 msgr2=0x7f413019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4130197d80 0x7f413019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 -- 172.21.15.156:0/238873530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240095f0 con 0x7f41301009d0 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f412effd700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 0x7f41301973d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4124004690 tx=0x7f4124004cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412401d070 con 0x7f41301009d0 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413019fc40 con 0x7f41301009d0 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f412400be20 con 0x7f41301009d0 2023-07-29T16:06:50.496 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.492+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412400f460 con 0x7f41301009d0 2023-07-29T16:06:50.497 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.493+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41301a00e0 con 0x7f41301009d0 2023-07-29T16:06:50.497 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.493+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f412400f5c0 con 0x7f41301009d0 2023-07-29T16:06:50.497 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.493+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4130190ec0 con 0x7f41301009d0 2023-07-29T16:06:50.497 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.493+0000 7f413493b700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41180388f0 0x7f411803ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:50.498 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.493+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f412404c290 con 0x7f41301009d0 2023-07-29T16:06:50.498 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.494+0000 7f412e7fc700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41180388f0 0x7f411803ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:50.498 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.494+0000 7f412e7fc700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41180388f0 0x7f411803ada0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4120009850 tx=0x7f4120008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:50.498 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.498+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4124027030 con 0x7f41301009d0 2023-07-29T16:06:50.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:50.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:50.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: Reconfiguring mon.c (monmap changed)... 2023-07-29T16:06:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:50 smithi156 ceph-mon[129245]: Reconfiguring daemon mon.c on smithi156 2023-07-29T16:06:50.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:50.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:50.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:50.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: Reconfiguring mon.c (monmap changed)... 2023-07-29T16:06:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:50 smithi031 ceph-mon[95504]: Reconfiguring daemon mon.c on smithi156 2023-07-29T16:06:50.695 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.700+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi039=y;smithi156=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f4130061220 con 0x7f41180388f0 2023-07-29T16:06:50.702 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.707+0000 7f413493b700 1 -- 172.21.15.156:0/238873530 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4130061220 con 0x7f41180388f0 2023-07-29T16:06:50.702 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mgr update... 2023-07-29T16:06:50.704 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41180388f0 msgr2=0x7f411803ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:50.704 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41180388f0 0x7f411803ada0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4120009850 tx=0x7f4120008040 comp rx=0 tx=0).stop 2023-07-29T16:06:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 msgr2=0x7f41301973d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 0x7f41301973d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4124004690 tx=0x7f4124004cb0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 shutdown_connections 2023-07-29T16:06:50.705 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41180388f0 0x7f411803ada0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.708 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4130197d80 0x7f413019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.709 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41301009d0 0x7f41301973d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.709 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 --2- 172.21.15.156:0/238873530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4130197910 0x7f413019c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:50.710 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 >> 172.21.15.156:0/238873530 conn(0x7f41300fa3f0 msgr2=0x7f41300fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:50.710 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.709+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 shutdown_connections 2023-07-29T16:06:50.710 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:06:50.710+0000 7f413593d700 1 -- 172.21.15.156:0/238873530 wait complete. 2023-07-29T16:06:50.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:50.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:50.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:50.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: Reconfiguring mon.c (monmap changed)... 2023-07-29T16:06:50.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-29T16:06:50.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-29T16:06:50.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:50.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:50 smithi039 ceph-mon[128858]: Reconfiguring daemon mon.c on smithi156 2023-07-29T16:06:51.137 DEBUG:teuthology.orchestra.run.smithi039:mgr.y> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.y.service 2023-07-29T16:06:51.140 DEBUG:teuthology.orchestra.run.smithi156:mgr.z> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.z.service 2023-07-29T16:06:51.141 INFO:tasks.cephadm:Deploying OSDs... 2023-07-29T16:06:51.142 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:06:51.142 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-07-29T16:06:51.172 INFO:journalctl@ceph.mgr.z.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:06:51.175 INFO:journalctl@ceph.mgr.y.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:06:51.192 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-29T16:06:51.193 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-07-29T16:06:51.279 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-29T16:06:51.280 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:51.280 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 449 Links: 1 2023-07-29T16:06:51.280 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:51.280 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:51.280 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-07-29 16:06:00.320844108 +0000 2023-07-29T16:06:51.280 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-07-29 15:55:42.606807385 +0000 2023-07-29T16:06:51.281 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-07-29 15:55:42.606807385 +0000 2023-07-29T16:06:51.281 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-07-29 15:55:42.577807855 +0000 2023-07-29T16:06:51.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-29T16:06:51.350 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-07-29T16:06:51.350 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-07-29T16:06:51.350 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000306576 s, 1.7 MB/s 2023-07-29T16:06:51.353 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-29T16:06:51.398 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-07-29T16:06:51.483 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-29T16:06:51.483 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:51.484 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 455 Links: 1 2023-07-29T16:06:51.484 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:51.484 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:51.485 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-07-29 16:06:00.320844108 +0000 2023-07-29T16:06:51.485 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-07-29 15:55:43.437793904 +0000 2023-07-29T16:06:51.485 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-07-29 15:55:43.437793904 +0000 2023-07-29T16:06:51.485 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-07-29 15:55:43.410794342 +0000 2023-07-29T16:06:51.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-29T16:06:51.589 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-07-29T16:06:51.590 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-07-29T16:06:51.590 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000288625 s, 1.8 MB/s 2023-07-29T16:06:51.592 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-29T16:06:51.638 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-07-29T16:06:51.723 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-29T16:06:51.723 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:51.723 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 461 Links: 1 2023-07-29T16:06:51.724 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:51.724 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:51.724 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-07-29 16:06:00.320844108 +0000 2023-07-29T16:06:51.724 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-07-29 15:55:44.273780343 +0000 2023-07-29T16:06:51.724 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-07-29 15:55:44.273780343 +0000 2023-07-29T16:06:51.724 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-07-29 15:55:44.246780781 +0000 2023-07-29T16:06:51.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-29T16:06:51.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi039=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:51.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: Saving service mgr spec with placement smithi031=x;smithi039=y;smithi156=z;count:3 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:51.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:51.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:51.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:06:51.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:51.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:51.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: Deploying daemon mgr.z on smithi156 2023-07-29T16:06:51.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:51 smithi156 ceph-mon[129245]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:51.829 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-07-29T16:06:51.829 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-07-29T16:06:51.829 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000324818 s, 1.6 MB/s 2023-07-29T16:06:51.832 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-29T16:06:51.878 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-07-29T16:06:51.962 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-29T16:06:51.962 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:51.962 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 467 Links: 1 2023-07-29T16:06:51.962 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:51.963 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:51.963 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-07-29 16:06:00.320844108 +0000 2023-07-29T16:06:51.963 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-07-29 15:55:45.107766814 +0000 2023-07-29T16:06:51.963 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-07-29 15:55:45.107766814 +0000 2023-07-29T16:06:51.963 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-07-29 15:55:45.079767268 +0000 2023-07-29T16:06:51.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-29T16:06:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi039=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: Saving service mgr spec with placement smithi031=x;smithi039=y;smithi156=z;count:3 2023-07-29T16:06:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:06:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:52.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:52.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: Deploying daemon mgr.z on smithi156 2023-07-29T16:06:52.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:51 smithi039 ceph-mon[128858]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:52.061 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi039=y;smithi156=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:06:52.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: Saving service mgr spec with placement smithi031=x;smithi039=y;smithi156=z;count:3 2023-07-29T16:06:52.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.062 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:52.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:52.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:52.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:52.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:52.063 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:06:52.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:52.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:52.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: Deploying daemon mgr.z on smithi156 2023-07-29T16:06:52.064 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:51 smithi031 ceph-mon[95504]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:52.068 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-07-29T16:06:52.068 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-07-29T16:06:52.068 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000277973 s, 1.8 MB/s 2023-07-29T16:06:52.070 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-29T16:06:52.115 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-07-29T16:06:52.115 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2023-07-29T16:06:52.138 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-29T16:06:52.138 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2023-07-29T16:06:52.199 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-29T16:06:52.199 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 375304 Links: 1 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-07-29 16:06:37.706163883 +0000 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-07-29 15:54:56.619068436 +0000 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-07-29 15:54:56.619068436 +0000 2023-07-29T16:06:52.200 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-07-29T16:06:52.201 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-29T16:06:52.271 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-07-29T16:06:52.272 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-07-29T16:06:52.272 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000385596 s, 1.3 MB/s 2023-07-29T16:06:52.273 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-29T16:06:52.334 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2023-07-29T16:06:52.393 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-29T16:06:52.394 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:52.394 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 367457 Links: 1 2023-07-29T16:06:52.394 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:52.394 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:52.395 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-07-29 16:06:37.706163883 +0000 2023-07-29T16:06:52.395 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-07-29 15:54:57.164057099 +0000 2023-07-29T16:06:52.395 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-07-29 15:54:57.164057099 +0000 2023-07-29T16:06:52.395 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-07-29T16:06:52.395 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-29T16:06:52.462 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-07-29T16:06:52.463 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-07-29T16:06:52.463 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000272781 s, 1.9 MB/s 2023-07-29T16:06:52.464 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-29T16:06:52.524 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2023-07-29T16:06:52.568 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:52 smithi156 systemd[1]: Starting Ceph mgr.z for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:06:52.583 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-29T16:06:52.583 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:52.583 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 377461 Links: 1 2023-07-29T16:06:52.583 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:52.583 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:52.584 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-07-29 16:06:37.706163883 +0000 2023-07-29T16:06:52.584 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-07-29 15:54:57.695046053 +0000 2023-07-29T16:06:52.584 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-07-29 15:54:57.695046053 +0000 2023-07-29T16:06:52.584 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-07-29T16:06:52.584 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-29T16:06:52.650 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-07-29T16:06:52.651 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-07-29T16:06:52.651 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000311645 s, 1.6 MB/s 2023-07-29T16:06:52.652 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-29T16:06:52.711 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2023-07-29T16:06:52.770 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-29T16:06:52.770 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:52.770 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 381525 Links: 1 2023-07-29T16:06:52.770 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:52.770 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:52.770 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-07-29 16:06:37.707163862 +0000 2023-07-29T16:06:52.771 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-07-29 15:54:58.233034862 +0000 2023-07-29T16:06:52.771 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-07-29 15:54:58.233034862 +0000 2023-07-29T16:06:52.771 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-07-29T16:06:52.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-29T16:06:52.838 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-07-29T16:06:52.838 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-07-29T16:06:52.838 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000296572 s, 1.7 MB/s 2023-07-29T16:06:52.839 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-29T16:06:52.884 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:52 smithi156 podman[131102]: 2023-07-29T16:06:52.898 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2023-07-29T16:06:52.898 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2023-07-29T16:06:52.916 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-29T16:06:52.916 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2023-07-29T16:06:52.973 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 373252 Links: 1 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-29 16:06:52.293069115 +0000 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-29 15:52:28.922702784 +0000 2023-07-29T16:06:52.974 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-29 15:52:28.922702784 +0000 2023-07-29T16:06:52.975 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-29T16:06:52.975 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-29T16:06:53.043 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-29T16:06:53.043 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-29T16:06:53.043 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000101752 s, 5.0 MB/s 2023-07-29T16:06:53.045 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-29T16:06:53.102 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2023-07-29T16:06:53.161 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-29T16:06:53.162 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:53.162 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 378507 Links: 1 2023-07-29T16:06:53.162 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:53.163 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:53.164 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-29 16:06:52.293069115 +0000 2023-07-29T16:06:53.164 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-29 15:52:29.434695059 +0000 2023-07-29T16:06:53.164 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-29 15:52:29.434695059 +0000 2023-07-29T16:06:53.165 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-29T16:06:53.165 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-29T16:06:53.190 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:52 smithi156 ceph-mgr[131142]: -- 172.21.15.156:0/2723735268 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d794d91e0 con 0x561d794aec00 2023-07-29T16:06:53.191 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:52 smithi156 bash[131102]: 705bc654b2dfb88d57f6b6f396c2f1c8f8f860dd2a4311a187ca35e7f9e9f8c1 2023-07-29T16:06:53.191 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:52 smithi156 systemd[1]: Started Ceph mgr.z for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:06:53.191 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.018+0000 7fb3f60f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T16:06:53.191 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.102+0000 7fb3f60f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T16:06:53.191 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.195+0000 7fb3f60f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T16:06:53.216 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-29T16:06:53.217 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-29T16:06:53.217 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000113678 s, 4.5 MB/s 2023-07-29T16:06:53.218 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-29T16:06:53.277 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2023-07-29T16:06:53.335 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 381219 Links: 1 2023-07-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-29 16:06:52.293069115 +0000 2023-07-29T16:06:53.336 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-29 15:52:29.940687423 +0000 2023-07-29T16:06:53.337 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-29 15:52:29.940687423 +0000 2023-07-29T16:06:53.337 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-29T16:06:53.337 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-29T16:06:53.404 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-29T16:06:53.404 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-29T16:06:53.404 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.00013827 s, 3.7 MB/s 2023-07-29T16:06:53.405 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-29T16:06:53.463 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2023-07-29T16:06:53.473 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.296+0000 7fb3f60f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T16:06:53.488 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 6h/6d Inode: 377743 Links: 1 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2023-07-29 16:06:52.293069115 +0000 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2023-07-29 15:52:30.449679743 +0000 2023-07-29T16:06:53.489 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2023-07-29 15:52:30.449679743 +0000 2023-07-29T16:06:53.490 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: - 2023-07-29T16:06:53.490 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-29T16:06:53.554 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2023-07-29T16:06:53.554 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2023-07-29T16:06:53.554 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000136056 s, 3.8 MB/s 2023-07-29T16:06:53.556 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-29T16:06:53.613 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2023-07-29T16:06:53.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-29T16:06:53.790 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.477+0000 7fb3f60f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T16:06:54.068 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.795+0000 7fb3f60f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T16:06:54.069 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:53.884+0000 7fb3f60f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T16:06:54.069 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:54.012+0000 7fb3f60f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T16:06:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:54.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:06:54.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:54.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:54.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: Deploying daemon mgr.y on smithi039 2023-07-29T16:06:54.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:53 smithi156 ceph-mon[129245]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:54.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:54.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:06:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: Deploying daemon mgr.y on smithi039 2023-07-29T16:06:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:53 smithi039 ceph-mon[128858]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:54.348 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:54.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:54.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:54.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-29T16:06:54.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:54.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:54.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: Deploying daemon mgr.y on smithi039 2023-07-29T16:06:54.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:53 smithi031 ceph-mon[95504]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:54.520 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:54.107+0000 7fb3f60f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T16:06:54.738 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:54 smithi039 systemd[1]: Starting Ceph mgr.y for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:06:54.818 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:54.524+0000 7fb3f60f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T16:06:55.021 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:54 smithi039 podman[130296]: 2023-07-29T16:06:55.022 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:54 smithi039 ceph-mgr[130336]: -- 172.21.15.39:0/3349209260 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5626da81b1e0 con 0x5626da7f1000 2023-07-29T16:06:55.022 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:54 smithi039 bash[130296]: 639ccd0eec01b0055a18d23a2b9a6c4abdb71a1a613a5c7af4fa339bd13d8ac5 2023-07-29T16:06:55.325 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:55 smithi039 systemd[1]: Started Ceph mgr.y for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:06:55.326 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:55.125+0000 7f956f4af200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-29T16:06:55.326 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:55.220+0000 7f956f4af200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-29T16:06:55.326 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:55.329+0000 7f956f4af200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-29T16:06:55.568 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:55.098+0000 7fb3f60f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T16:06:55.569 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:55.269+0000 7fb3f60f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T16:06:55.652 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:55.443+0000 7f956f4af200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-29T16:06:56.021 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:55.655+0000 7f956f4af200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-29T16:06:56.318 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:56.068+0000 7fb3f60f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:06:56.318 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:56.178+0000 7fb3f60f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T16:06:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-mon[129245]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:56.337 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.338 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.338 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.338 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.339 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-mon[128858]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:56.339 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:56.339 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:56.066+0000 7f956f4af200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-29T16:06:56.339 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:56.178+0000 7f956f4af200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-29T16:06:56.340 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:56.341+0000 7f956f4af200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-29T16:06:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-29T16:06:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-29T16:06:56.362 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2023-07-29T16:06:56.363 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2023-07-29T16:06:56.363 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0548563 s, 191 MB/s 2023-07-29T16:06:56.363 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2023-07-29T16:06:56.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:56.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:56 smithi031 ceph-mon[95504]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:56.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:06:56.656 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:56.355+0000 7fb3f60f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:06:56.656 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:56.660+0000 7fb3f60f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T16:06:56.771 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:56.459+0000 7f956f4af200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-29T16:06:56.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2023-07-29T16:06:57.047 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:56.904+0000 7fb3f60f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T16:06:57.047 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:57.051+0000 7fb3f60f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T16:06:57.271 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:57.004+0000 7f956f4af200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-29T16:06:57.568 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:57.315+0000 7fb3f60f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T16:06:57.577 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:06:58.021 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:57.703+0000 7f956f4af200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-29T16:06:58.022 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:57.894+0000 7f956f4af200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-29T16:06:58.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: Reconfiguring mgr.x (unknown last config time)... 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:58.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:58.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: Reconfiguring daemon mgr.x on smithi031 2023-07-29T16:06:58.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:57 smithi039 ceph-mon[128858]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:58.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:58.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:58.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: Reconfiguring mgr.x (unknown last config time)... 2023-07-29T16:06:58.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:58.127 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:58.128 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:58.128 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: Reconfiguring daemon mgr.x on smithi031 2023-07-29T16:06:58.128 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:57 smithi031 ceph-mon[95504]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:58.318 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:58.018+0000 7fb3f60f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T16:06:58.318 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:58.148+0000 7fb3f60f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T16:06:58.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:58.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: Reconfiguring mgr.x (unknown last config time)... 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: Reconfiguring daemon mgr.x on smithi031 2023-07-29T16:06:58.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:57 smithi156 ceph-mon[129245]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:06:58.660 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:58.337+0000 7fb3f60f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T16:06:58.660 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:58.449+0000 7fb3f60f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T16:06:59.068 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:58.664+0000 7fb3f60f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T16:06:59.178 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:58 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:58.851+0000 7f956f4af200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:06:59.178 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:58 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:58.962+0000 7f956f4af200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-29T16:06:59.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 -- 172.21.15.31:0/546073663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57380fec40 msgr2=0x7f5738101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:59.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 --2- 172.21.15.31:0/546073663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57380fec40 0x7f5738101060 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f57280098f0 tx=0x7f5728009c00 comp rx=0 tx=0).stop 2023-07-29T16:06:59.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 -- 172.21.15.31:0/546073663 shutdown_connections 2023-07-29T16:06:59.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 --2- 172.21.15.31:0/546073663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5738103f60 0x7f57381043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:59.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 --2- 172.21.15.31:0/546073663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57380fec40 0x7f5738101060 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:59.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 --2- 172.21.15.31:0/546073663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57381015a0 0x7f5738103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:59.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 -- 172.21.15.31:0/546073663 >> 172.21.15.31:0/546073663 conn(0x7f57380fa8f0 msgr2=0x7f57380fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:06:59.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 -- 172.21.15.31:0/546073663 shutdown_connections 2023-07-29T16:06:59.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.348+0000 7f573e002700 1 -- 172.21.15.31:0/546073663 wait complete. 2023-07-29T16:06:59.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.349+0000 7f573e002700 1 Processor -- start 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 -- start start 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 0x7f573819c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57381015a0 0x7f573819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5738103f60 0x7f57381a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381a4280 con 0x7f57381015a0 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381a43c0 con 0x7f57380fec40 2023-07-29T16:06:59.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f573e002700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57381a4550 con 0x7f5738103f60 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f57377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 0x7f573819c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f57377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 0x7f573819c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34282/0 (socket says 172.21.15.31:34282) 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f57377fe700 1 -- 172.21.15.31:0/1731990722 learned_addr learned my addr 172.21.15.31:0/1731990722 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f5736ffd700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57381015a0 0x7f573819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.350+0000 7f5737fff700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5738103f60 0x7f57381a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f57377fe700 1 -- 172.21.15.31:0/1731990722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5738103f60 msgr2=0x7f57381a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:59.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f57377fe700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5738103f60 0x7f57381a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f57377fe700 1 -- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57381015a0 msgr2=0x7f573819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f57377fe700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57381015a0 0x7f573819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f57377fe700 1 -- 172.21.15.31:0/1731990722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730009d00 con 0x7f57380fec40 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f5737fff700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5738103f60 0x7f57381a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f57377fe700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 0x7f573819c6f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5728011040 tx=0x7f572800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572801c070 con 0x7f57380fec40 2023-07-29T16:06:59.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f572800f7d0 con 0x7f57380fec40 2023-07-29T16:06:59.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5728009520 con 0x7f57380fec40 2023-07-29T16:06:59.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728004a90 con 0x7f57380fec40 2023-07-29T16:06:59.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.351+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57381a4a50 con 0x7f57380fec40 2023-07-29T16:06:59.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.352+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) v1 ==== 45125+0+0 (secure 0 0 0) 0x7f5728020070 con 0x7f57380fec40 2023-07-29T16:06:59.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.352+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57380fc460 con 0x7f57380fec40 2023-07-29T16:06:59.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.353+0000 7f5734ff9700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57200386a0 0x7f572003ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:06:59.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.353+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f572804bc90 con 0x7f57380fec40 2023-07-29T16:06:59.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.353+0000 7f5736ffd700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57200386a0 0x7f572003ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:06:59.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.354+0000 7f5736ffd700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57200386a0 0x7f572003ab50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f572c009c60 tx=0x7f572c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:06:59.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.358+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f572800edb0 con 0x7f57380fec40 2023-07-29T16:06:59.522 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:59.181+0000 7f956f4af200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-29T16:06:59.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:06:59.574+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5738061190 con 0x7f57200386a0 2023-07-29T16:06:59.784 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:59.531+0000 7f956f4af200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-29T16:06:59.818 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z[131118]: 2023-07-29T16:06:59.532+0000 7fb3f60f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T16:07:00.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: Standby manager daemon z started 2023-07-29T16:07:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-29T16:07:00.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:07:00.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-29T16:07:00.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:07:00.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:00.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:00.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:06:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:00.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:00.267+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 14) v1 ==== 89771+0+0 (secure 0 0 0) 0x7f5728005040 con 0x7f57380fec40 2023-07-29T16:07:00.271 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:59.787+0000 7f956f4af200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-29T16:07:00.272 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:06:59.969+0000 7f956f4af200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-29T16:07:00.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:00.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:00.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:00.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:00.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: Standby manager daemon z started 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:00.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:06:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:00.318 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:00.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:00.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:00.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:00.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:00.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: Standby manager daemon z started 2023-07-29T16:07:00.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-29T16:07:00.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:07:00.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-29T16:07:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.156:0/3091209584' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:07:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:06:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:00.771 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:00 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:00.295+0000 7f956f4af200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-29T16:07:01.512 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:01.116+0000 7f956f4af200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-29T16:07:01.513 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:01.275+0000 7f956f4af200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-29T16:07:01.513 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-mon[128858]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:01.513 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-mon[128858]: mgrmap e14: x(active, since 71s), standbys: z 2023-07-29T16:07:01.513 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-29T16:07:01.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:01 smithi031 ceph-mon[95504]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:01.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:01 smithi031 ceph-mon[95504]: mgrmap e14: x(active, since 71s), standbys: z 2023-07-29T16:07:01.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-29T16:07:01.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:01 smithi156 ceph-mon[129245]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:01 smithi156 ceph-mon[129245]: mgrmap e14: x(active, since 71s), standbys: z 2023-07-29T16:07:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-29T16:07:01.771 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:01.517+0000 7f956f4af200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-29T16:07:01.772 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:01.653+0000 7f956f4af200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-29T16:07:02.266 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:01.922+0000 7f956f4af200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-29T16:07:02.521 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:02 smithi039 ceph-mon[128858]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:02.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:02 smithi156 ceph-mon[129245]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:02.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:02 smithi031 ceph-mon[95504]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:03.271 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:07:02 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y[130312]: 2023-07-29T16:07:02.961+0000 7f956f4af200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-29T16:07:03.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2850405647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f598978-c50a-4143-a36d-091f24731a98"}]: dispatch 2023-07-29T16:07:03.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2850405647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f598978-c50a-4143-a36d-091f24731a98"}]': finished 2023-07-29T16:07:03.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: osdmap e5: 1 total, 0 up, 1 in 2023-07-29T16:07:03.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:03.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: Standby manager daemon y started 2023-07-29T16:07:03.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T16:07:03.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:07:03.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T16:07:03.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:07:03.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/162882588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:03.595+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mon.2 v2:172.21.15.39:3300/0 8 ==== mgrmap(e 15) v1 ==== 134417+0+0 (secure 0 0 0) 0x7f572800d340 con 0x7f57380fec40 2023-07-29T16:07:03.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2850405647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f598978-c50a-4143-a36d-091f24731a98"}]: dispatch 2023-07-29T16:07:03.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2850405647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f598978-c50a-4143-a36d-091f24731a98"}]': finished 2023-07-29T16:07:03.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: osdmap e5: 1 total, 0 up, 1 in 2023-07-29T16:07:03.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:03.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: Standby manager daemon y started 2023-07-29T16:07:03.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T16:07:03.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:07:03.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T16:07:03.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:07:03.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/162882588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:03.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2850405647' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f598978-c50a-4143-a36d-091f24731a98"}]: dispatch 2023-07-29T16:07:03.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2850405647' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f598978-c50a-4143-a36d-091f24731a98"}]': finished 2023-07-29T16:07:03.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: osdmap e5: 1 total, 0 up, 1 in 2023-07-29T16:07:03.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:03.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: Standby manager daemon y started 2023-07-29T16:07:03.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-29T16:07:03.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-29T16:07:03.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-29T16:07:03.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='mgr.? 172.21.15.39:0/788456962' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-29T16:07:03.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/162882588' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:04.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:04 smithi156 ceph-mon[129245]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:04.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:04 smithi156 ceph-mon[129245]: mgrmap e15: x(active, since 74s), standbys: z, y 2023-07-29T16:07:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T16:07:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:04 smithi031 ceph-mon[95504]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:04 smithi031 ceph-mon[95504]: mgrmap e15: x(active, since 74s), standbys: z, y 2023-07-29T16:07:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T16:07:05.021 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:04 smithi039 ceph-mon[128858]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:05.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:04 smithi039 ceph-mon[128858]: mgrmap e15: x(active, since 74s), standbys: z, y 2023-07-29T16:07:05.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-29T16:07:05.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:05 smithi156 ceph-mon[129245]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:05.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:05 smithi031 ceph-mon[95504]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:06.021 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:05 smithi039 ceph-mon[128858]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:07.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:07 smithi031 ceph-mon[95504]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:08.021 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:07 smithi039 ceph-mon[128858]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:08.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:07 smithi156 ceph-mon[129245]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:10.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:10 smithi031 ceph-mon[95504]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:10.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T16:07:10.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:10.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:10 smithi039 ceph-mon[128858]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:10.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T16:07:10.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:10.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:10 smithi156 ceph-mon[129245]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:10.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-29T16:07:10.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:11.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:11 smithi031 ceph-mon[95504]: Deploying daemon osd.0 on smithi031 2023-07-29T16:07:11.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:11 smithi039 ceph-mon[128858]: Deploying daemon osd.0 on smithi031 2023-07-29T16:07:11.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:11 smithi156 ceph-mon[129245]: Deploying daemon osd.0 on smithi031 2023-07-29T16:07:12.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:12 smithi039 ceph-mon[128858]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:12.540 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:12 smithi031 ceph-mon[95504]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:12.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:12 smithi156 ceph-mon[129245]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:14.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:14 smithi031 ceph-mon[95504]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:14.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:14 smithi156 ceph-mon[129245]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:14.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:14 smithi039 ceph-mon[128858]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:16.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:16 smithi031 ceph-mon[95504]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:16.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:16.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:16.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:16.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:16 smithi156 ceph-mon[129245]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:16.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:16 smithi039 ceph-mon[128858]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:16.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:17.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:17 smithi031 ceph-mon[95504]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:18.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:17 smithi039 ceph-mon[128858]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:18.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:17 smithi156 ceph-mon[129245]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:19.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.691+0000 7f5734ff9700 1 -- 172.21.15.31:0/1731990722 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5738061190 con 0x7f57200386a0 2023-07-29T16:07:19.688 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2023-07-29T16:07:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57200386a0 msgr2=0x7f572003ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57200386a0 0x7f572003ab50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f572c009c60 tx=0x7f572c009260 comp rx=0 tx=0).stop 2023-07-29T16:07:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 msgr2=0x7f573819c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 0x7f573819c6f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5728011040 tx=0x7f572800b790 comp rx=0 tx=0).stop 2023-07-29T16:07:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 shutdown_connections 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57200386a0 0x7f572003ab50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5738103f60 0x7f57381a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57380fec40 0x7f573819c6f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 --2- 172.21.15.31:0/1731990722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57381015a0 0x7f573819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 >> 172.21.15.31:0/1731990722 conn(0x7f57380fa8f0 msgr2=0x7f57381022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.693+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 shutdown_connections 2023-07-29T16:07:19.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:07:19.694+0000 7f573e002700 1 -- 172.21.15.31:0/1731990722 wait complete. 2023-07-29T16:07:19.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:19.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:19.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:19.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:19.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:19.847 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:19 smithi031 ceph-mon[95504]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:20.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:19 smithi039 ceph-mon[128858]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:20.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:19 smithi156 ceph-mon[129245]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:20.151 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.0.service 2023-07-29T16:07:20.155 INFO:tasks.cephadm:Deploying osd.1 on smithi039 with /dev/vg_nvme/lv_4... 2023-07-29T16:07:20.156 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-29T16:07:20.230 INFO:journalctl@ceph.osd.0.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:07:20.724 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:07:20.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:20.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:20 smithi031 ceph-mon[95504]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T16:07:20.964 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.965 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:20.965 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:20.965 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:20 smithi039 ceph-mon[128858]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T16:07:21.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:21.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:21.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:21.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:20 smithi156 ceph-mon[129245]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-29T16:07:21.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:21 smithi039 ceph-mon[128858]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T16:07:21.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:21 smithi039 ceph-mon[128858]: osdmap e6: 1 total, 0 up, 1 in 2023-07-29T16:07:21.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:21.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:21 smithi039 ceph-mon[128858]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:07:21.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:21 smithi039 ceph-mon[128858]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:22.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:21 smithi156 ceph-mon[129245]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T16:07:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:21 smithi156 ceph-mon[129245]: osdmap e6: 1 total, 0 up, 1 in 2023-07-29T16:07:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:21 smithi156 ceph-mon[129245]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:07:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:21 smithi156 ceph-mon[129245]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:22.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-mon[95504]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-29T16:07:22.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-mon[95504]: osdmap e6: 1 total, 0 up, 1 in 2023-07-29T16:07:22.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:22.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-mon[95504]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:07:22.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-mon[95504]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:22.182 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-0[108638]: 2023-07-29T16:07:21.689+0000 7f3a11c89700 -1 osd.0 0 waiting for initial osdmap 2023-07-29T16:07:22.182 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:07:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-0[108638]: 2023-07-29T16:07:21.694+0000 7f3a0c27b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:07:22.270 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-29T16:07:22.270 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-29T16:07:22.270 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-07-29T16:07:22.271 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-07-29T16:07:22.271 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0068259 s, 1.5 GB/s 2023-07-29T16:07:22.271 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-07-29T16:07:22.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_4 2023-07-29T16:07:22.837 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:22 smithi031 ceph-mon[95504]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:07:22.837 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:22 smithi031 ceph-mon[95504]: osdmap e7: 1 total, 0 up, 1 in 2023-07-29T16:07:22.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:22.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:23.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:22 smithi039 ceph-mon[128858]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:07:23.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:22 smithi039 ceph-mon[128858]: osdmap e7: 1 total, 0 up, 1 in 2023-07-29T16:07:23.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:23.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:23.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:22 smithi156 ceph-mon[129245]: from='osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:07:23.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:22 smithi156 ceph-mon[129245]: osdmap e7: 1 total, 0 up, 1 in 2023-07-29T16:07:23.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:23.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:23.230 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:07:23.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:07:23.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:07:23.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] boot 2023-07-29T16:07:23.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: osdmap e8: 1 total, 1 up, 1 in 2023-07-29T16:07:23.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:23.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi031 2023-07-29T16:07:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:07:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 17272M 2023-07-29T16:07:23.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:23.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:23.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:23.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:23.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:23 smithi039 ceph-mon[128858]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:24.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:07:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:07:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] boot 2023-07-29T16:07:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: osdmap e8: 1 total, 1 up, 1 in 2023-07-29T16:07:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi031 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 17272M 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:24.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:23 smithi156 ceph-mon[129245]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:24.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:07:24.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: osd.0 [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] boot 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: osdmap e8: 1 total, 1 up, 1 in 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi031 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:07:24.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 17272M 2023-07-29T16:07:24.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:24.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:24.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:24.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:23 smithi031 ceph-mon[95504]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-29T16:07:24.552 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 -- 172.21.15.39:0/2673651894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 msgr2=0x7fcc84101b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:24.552 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/2673651894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84101b80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc780099c0 tx=0x7fcc78009cd0 comp rx=0 tx=0).stop 2023-07-29T16:07:24.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 -- 172.21.15.39:0/2673651894 shutdown_connections 2023-07-29T16:07:24.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/2673651894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc841020c0 0x7fcc84108980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:24.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/2673651894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc840fe6b0 0x7fcc840feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:24.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/2673651894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84101b80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:24.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 -- 172.21.15.39:0/2673651894 >> 172.21.15.39:0/2673651894 conn(0x7fcc840fa0a0 msgr2=0x7fcc840fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:07:24.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 -- 172.21.15.39:0/2673651894 shutdown_connections 2023-07-29T16:07:24.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.556+0000 7fcc8a54d700 1 -- 172.21.15.39:0/2673651894 wait complete. 2023-07-29T16:07:24.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 Processor -- start 2023-07-29T16:07:24.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 -- start start 2023-07-29T16:07:24.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc840fe6b0 0x7fcc84196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:24.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:24.557 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc841020c0 0x7fcc8419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8419dcc0 con 0x7fcc841020c0 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8419de00 con 0x7fcc84101730 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.557+0000 7fcc8a54d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8419df90 con 0x7fcc840fe6b0 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54984/0 (socket says 172.21.15.39:54984) 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 -- 172.21.15.39:0/1055643441 learned_addr learned my addr 172.21.15.39:0/1055643441 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc83fff700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc840fe6b0 0x7fcc84196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:24.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc88aea700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc841020c0 0x7fcc8419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:24.559 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 -- 172.21.15.39:0/1055643441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc840fe6b0 msgr2=0x7fcc84196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:24.559 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc840fe6b0 0x7fcc84196130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:24.559 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 -- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc841020c0 msgr2=0x7fcc8419d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:24.559 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc841020c0 0x7fcc8419d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:24.559 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 -- 172.21.15.39:0/1055643441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780095f0 con 0x7fcc84101730 2023-07-29T16:07:24.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc88aea700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc841020c0 0x7fcc8419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:07:24.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc837fe700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84196670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009990 tx=0x7fcc78004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:07:24.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7801d070 con 0x7fcc84101730 2023-07-29T16:07:24.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8419e210 con 0x7fcc84101730 2023-07-29T16:07:24.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc78004090 con 0x7fcc84101730 2023-07-29T16:07:24.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78004a00 con 0x7fcc84101730 2023-07-29T16:07:24.561 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.558+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc8419e6b0 con 0x7fcc84101730 2023-07-29T16:07:24.561 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.559+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc84107ae0 con 0x7fcc84101730 2023-07-29T16:07:24.561 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.560+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 15) v1 ==== 134417+0+0 (secure 0 0 0) 0x7fcc78003bd0 con 0x7fcc84101730 2023-07-29T16:07:24.561 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.561+0000 7fcc817fa700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc6c0a0780 0x7fcc6c0a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:24.561 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.561+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7fcc780cafa0 con 0x7fcc84101730 2023-07-29T16:07:24.562 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.561+0000 7fcc83fff700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc6c0a0780 0x7fcc6c0a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:24.562 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.562+0000 7fcc83fff700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc6c0a0780 0x7fcc6c0a2c30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcc84068550 tx=0x7fcc74009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:07:24.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.565+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcc780964a0 con 0x7fcc84101730 2023-07-29T16:07:24.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:24.772+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc84061190 con 0x7fcc6c0a0780 2023-07-29T16:07:25.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:24 smithi039 ceph-mon[128858]: osdmap e9: 1 total, 1 up, 1 in 2023-07-29T16:07:25.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:24 smithi156 ceph-mon[129245]: osdmap e9: 1 total, 1 up, 1 in 2023-07-29T16:07:25.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:24 smithi031 ceph-mon[95504]: osdmap e9: 1 total, 1 up, 1 in 2023-07-29T16:07:25.873 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:25 smithi039 ceph-mon[128858]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:25.874 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:25.874 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:25.874 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:25.874 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:25 smithi039 ceph-mon[128858]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:26.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:25 smithi156 ceph-mon[129245]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:26.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:26.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:26.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:26.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:25 smithi156 ceph-mon[129245]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:26.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:25 smithi031 ceph-mon[95504]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:26.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:26.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:26.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:26.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:25 smithi031 ceph-mon[95504]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:27.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:27 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/700055874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]: dispatch 2023-07-29T16:07:27.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:27 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]: dispatch 2023-07-29T16:07:27.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:27 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]': finished 2023-07-29T16:07:27.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:27 smithi031 ceph-mon[95504]: osdmap e10: 2 total, 1 up, 2 in 2023-07-29T16:07:27.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:27.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:27 smithi031 ceph-mon[95504]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:28.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:27 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/700055874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]: dispatch 2023-07-29T16:07:28.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:27 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]: dispatch 2023-07-29T16:07:28.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:27 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]': finished 2023-07-29T16:07:28.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:27 smithi039 ceph-mon[128858]: osdmap e10: 2 total, 1 up, 2 in 2023-07-29T16:07:28.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:28.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:27 smithi039 ceph-mon[128858]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:28.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:27 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/700055874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]: dispatch 2023-07-29T16:07:28.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:27 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]: dispatch 2023-07-29T16:07:28.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:27 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ceb36a3-18a7-4af6-ae87-03d42276da3c"}]': finished 2023-07-29T16:07:28.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:27 smithi156 ceph-mon[129245]: osdmap e10: 2 total, 1 up, 2 in 2023-07-29T16:07:28.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:28.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:27 smithi156 ceph-mon[129245]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:29.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:28 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/1816620617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:29.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:28 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/1816620617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:29.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:28 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/1816620617' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:30.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:29 smithi039 ceph-mon[128858]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:30.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:29 smithi156 ceph-mon[129245]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:30.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:29 smithi031 ceph-mon[95504]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:32.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:32 smithi039 ceph-mon[128858]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:32.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:32 smithi156 ceph-mon[129245]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:32.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:32 smithi031 ceph-mon[95504]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:34.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:34 smithi039 ceph-mon[128858]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:34.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:34 smithi156 ceph-mon[129245]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:34.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:34 smithi031 ceph-mon[95504]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:36.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:36 smithi156 ceph-mon[129245]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T16:07:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:36 smithi156 ceph-mon[129245]: Deploying daemon osd.1 on smithi039 2023-07-29T16:07:36.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:36 smithi031 ceph-mon[95504]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:36.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T16:07:36.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:36.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:36 smithi031 ceph-mon[95504]: Deploying daemon osd.1 on smithi039 2023-07-29T16:07:36.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:36 smithi039 ceph-mon[128858]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:36.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-29T16:07:36.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:36.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:36 smithi039 ceph-mon[128858]: Deploying daemon osd.1 on smithi039 2023-07-29T16:07:37.923 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:37 smithi039 ceph-mon[128858]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:37.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:37 smithi031 ceph-mon[95504]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:38.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:37 smithi156 ceph-mon[129245]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:40.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:40 smithi039 ceph-mon[128858]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:40.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:40 smithi156 ceph-mon[129245]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:40.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:40.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:40.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:40 smithi031 ceph-mon[95504]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:40.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:40.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:41.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:41 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:41.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:41 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:41.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:41 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:42.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:42 smithi156 ceph-mon[129245]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:42.596 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:42 smithi039 ceph-mon[128858]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:42.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:42 smithi031 ceph-mon[95504]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:43.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:43.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:43.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.643 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 1 on host 'smithi039' 2023-07-29T16:07:43.644 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.646+0000 7fcc817fa700 1 -- 172.21.15.39:0/1055643441 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcc84061190 con 0x7fcc6c0a0780 2023-07-29T16:07:43.645 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc6c0a0780 msgr2=0x7fcc6c0a2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:43.645 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc6c0a0780 0x7fcc6c0a2c30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcc84068550 tx=0x7fcc74009190 comp rx=0 tx=0).stop 2023-07-29T16:07:43.645 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 msgr2=0x7fcc84196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:43.645 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84196670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc78009990 tx=0x7fcc78004730 comp rx=0 tx=0).stop 2023-07-29T16:07:43.645 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 shutdown_connections 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc6c0a0780 0x7fcc6c0a2c30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc841020c0 0x7fcc8419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc840fe6b0 0x7fcc84196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 --2- 172.21.15.39:0/1055643441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc84101730 0x7fcc84196670 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 >> 172.21.15.39:0/1055643441 conn(0x7fcc840fa0a0 msgr2=0x7fcc84104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 shutdown_connections 2023-07-29T16:07:43.646 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:07:43.648+0000 7fcc8a54d700 1 -- 172.21.15.39:0/1055643441 wait complete. 2023-07-29T16:07:43.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.129 DEBUG:teuthology.orchestra.run.smithi039:osd.1> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.1.service 2023-07-29T16:07:44.131 INFO:tasks.cephadm:Deploying osd.2 on smithi156 with /dev/vg_nvme/lv_4... 2023-07-29T16:07:44.132 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-29T16:07:44.164 INFO:journalctl@ceph.osd.1.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:07:44.319 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1[133963]: 2023-07-29T16:07:44.170+0000 7fd2b9cfd640 -1 osd.1 0 log_to_monitors true 2023-07-29T16:07:44.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:44 smithi156 ceph-mon[129245]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:44 smithi156 ceph-mon[129245]: from='osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T16:07:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:44 smithi156 ceph-mon[129245]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T16:07:44.637 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-mon[128858]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:44.637 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.638 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.638 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:44.638 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-mon[128858]: from='osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T16:07:44.638 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:44 smithi039 ceph-mon[128858]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T16:07:44.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:44 smithi031 ceph-mon[95504]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:44.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:44.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:07:44.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:44 smithi031 ceph-mon[95504]: from='osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T16:07:44.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:44 smithi031 ceph-mon[95504]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-29T16:07:44.694 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:07:45.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:45 smithi031 ceph-mon[95504]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T16:07:45.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:45 smithi031 ceph-mon[95504]: osdmap e11: 2 total, 1 up, 2 in 2023-07-29T16:07:45.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:45.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:45 smithi031 ceph-mon[95504]: from='osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:07:45.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:45 smithi031 ceph-mon[95504]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:07:45.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:45 smithi031 ceph-mon[95504]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:46.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-mon[128858]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T16:07:46.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-mon[128858]: osdmap e11: 2 total, 1 up, 2 in 2023-07-29T16:07:46.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:46.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-mon[128858]: from='osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:07:46.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-mon[128858]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:07:46.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-mon[128858]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:46.024 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1[133963]: 2023-07-29T16:07:45.627+0000 7fd2aeb74700 -1 osd.1 0 waiting for initial osdmap 2023-07-29T16:07:46.024 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:07:45 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1[133963]: 2023-07-29T16:07:45.631+0000 7fd2ab16a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:07:46.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:45 smithi156 ceph-mon[129245]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-29T16:07:46.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:45 smithi156 ceph-mon[129245]: osdmap e11: 2 total, 1 up, 2 in 2023-07-29T16:07:46.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:46.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:45 smithi156 ceph-mon[129245]: from='osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:07:46.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:45 smithi156 ceph-mon[129245]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:07:46.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:45 smithi156 ceph-mon[129245]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:46.144 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-29T16:07:46.145 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-29T16:07:46.145 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-29T16:07:46.145 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-29T16:07:46.145 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123162 s, 851 MB/s 2023-07-29T16:07:46.145 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-29T16:07:46.534 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2023-07-29T16:07:46.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:07:46.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: osdmap e12: 2 total, 1 up, 2 in 2023-07-29T16:07:46.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:46.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: osdmap e12: 2 total, 1 up, 2 in 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:07:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:46.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:46.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:46.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:07:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: osdmap e12: 2 total, 1 up, 2 in 2023-07-29T16:07:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:07:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:07:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:07:47.090 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:07:47.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:07:47.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:07:47.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi039 2023-07-29T16:07:47.857 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 17168M 2023-07-29T16:07:47.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] boot 2023-07-29T16:07:47.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: osdmap e13: 2 total, 2 up, 2 in 2023-07-29T16:07:47.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:47.858 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:47 smithi156 ceph-mon[129245]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:47.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi039 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 17168M 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] boot 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: osdmap e13: 2 total, 2 up, 2 in 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:47 smithi031 ceph-mon[95504]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:48.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:07:48.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:07:48.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi039 2023-07-29T16:07:48.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 17168M 2023-07-29T16:07:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: osd.1 [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] boot 2023-07-29T16:07:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: osdmap e13: 2 total, 2 up, 2 in 2023-07-29T16:07:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-29T16:07:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:47 smithi039 ceph-mon[128858]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-29T16:07:48.249 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 -- 172.21.15.156:0/3945893662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e241004c0 msgr2=0x7f5e24100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:48.250 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3945893662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e241004c0 0x7f5e24100910 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009990 tx=0x7f5e20009ca0 comp rx=0 tx=0).stop 2023-07-29T16:07:48.250 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 -- 172.21.15.156:0/3945893662 shutdown_connections 2023-07-29T16:07:48.250 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3945893662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e24100e50 0x7f5e24107710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:48.250 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3945893662 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e240ff2c0 0x7f5e240ff6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:48.250 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3945893662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e241004c0 0x7f5e24100910 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 -- 172.21.15.156:0/3945893662 >> 172.21.15.156:0/3945893662 conn(0x7f5e240fa8f0 msgr2=0x7f5e240fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 -- 172.21.15.156:0/3945893662 shutdown_connections 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.252+0000 7f5e2c420700 1 -- 172.21.15.156:0/3945893662 wait complete. 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 Processor -- start 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 -- start start 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e240ff2c0 0x7f5e24069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:48.251 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e241004c0 0x7f5e2419af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:48.252 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 0x7f5e2419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:48.252 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2419dd20 con 0x7f5e240ff2c0 2023-07-29T16:07:48.252 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2419de90 con 0x7f5e241004c0 2023-07-29T16:07:48.252 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2c420700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2419e050 con 0x7f5e24100e50 2023-07-29T16:07:48.252 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 0x7f5e2419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:48.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 0x7f5e2419d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40482/0 (socket says 172.21.15.156:40482) 2023-07-29T16:07:48.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 -- 172.21.15.156:0/3245257951 learned_addr learned my addr 172.21.15.156:0/3245257951 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:07:48.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a1bc700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e240ff2c0 0x7f5e24069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:48.253 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e299bb700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e241004c0 0x7f5e2419af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:48.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 -- 172.21.15.156:0/3245257951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e241004c0 msgr2=0x7f5e2419af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:48.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e241004c0 0x7f5e2419af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:48.254 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 -- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e240ff2c0 msgr2=0x7f5e24069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:07:48.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e240ff2c0 0x7f5e24069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:07:48.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e2a9bd700 1 -- 172.21.15.156:0/3245257951 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e200095f0 con 0x7f5e24100e50 2023-07-29T16:07:48.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.253+0000 7f5e299bb700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e241004c0 0x7f5e2419af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:07:48.255 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.254+0000 7f5e2a9bd700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 0x7f5e2419d580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b970 tx=0x7f5e1c00bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:07:48.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.254+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00c9b0 con 0x7f5e24100e50 2023-07-29T16:07:48.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.254+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e1c014440 con 0x7f5e24100e50 2023-07-29T16:07:48.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.254+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2419e330 con 0x7f5e24100e50 2023-07-29T16:07:48.256 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.254+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0075d0 con 0x7f5e24100e50 2023-07-29T16:07:48.257 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.254+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e2419e8b0 con 0x7f5e24100e50 2023-07-29T16:07:48.258 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.255+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24106850 con 0x7f5e24100e50 2023-07-29T16:07:48.258 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.256+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) v1 ==== 134417+0+0 (secure 0 0 0) 0x7f5e1c0145b0 con 0x7f5e24100e50 2023-07-29T16:07:48.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.256+0000 7f5e1b7fe700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e100a0850 0x7f5e100a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:07:48.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.256+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f5e1c006e40 con 0x7f5e24100e50 2023-07-29T16:07:48.259 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.257+0000 7f5e2a1bc700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e100a0850 0x7f5e100a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:07:48.260 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.257+0000 7f5e2a1bc700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e100a0850 0x7f5e100a2d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5e14006fd0 tx=0x7f5e14008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:07:48.260 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.261+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5e1c04e720 con 0x7f5e24100e50 2023-07-29T16:07:48.449 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:07:48.454+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e24061190 con 0x7f5e100a0850 2023-07-29T16:07:48.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:48 smithi031 ceph-mon[95504]: osdmap e14: 2 total, 2 up, 2 in 2023-07-29T16:07:48.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:48 smithi031 ceph-mon[95504]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:48.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:48.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:48.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:49.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:48 smithi039 ceph-mon[128858]: osdmap e14: 2 total, 2 up, 2 in 2023-07-29T16:07:49.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:48 smithi039 ceph-mon[128858]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:49.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:49.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:49.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:49.068 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:48 smithi156 ceph-mon[129245]: osdmap e14: 2 total, 2 up, 2 in 2023-07-29T16:07:49.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:48 smithi156 ceph-mon[129245]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:07:49.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:07:49.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:07:49.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:49.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:49 smithi156 ceph-mon[129245]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:49.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:49 smithi031 ceph-mon[95504]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:50.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:49 smithi039 ceph-mon[128858]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:51.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:50 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/2735323552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]: dispatch 2023-07-29T16:07:51.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:50 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]: dispatch 2023-07-29T16:07:51.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:50 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]': finished 2023-07-29T16:07:51.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:50 smithi156 ceph-mon[129245]: osdmap e15: 3 total, 2 up, 3 in 2023-07-29T16:07:51.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:07:51.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:50 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/2735323552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]: dispatch 2023-07-29T16:07:51.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:50 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]: dispatch 2023-07-29T16:07:51.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:50 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]': finished 2023-07-29T16:07:51.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:50 smithi031 ceph-mon[95504]: osdmap e15: 3 total, 2 up, 3 in 2023-07-29T16:07:51.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:07:51.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:50 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/2735323552' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]: dispatch 2023-07-29T16:07:51.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:50 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]: dispatch 2023-07-29T16:07:51.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:50 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b70de0c9-39ca-4d04-bbe3-532e576fd586"}]': finished 2023-07-29T16:07:51.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:50 smithi039 ceph-mon[128858]: osdmap e15: 3 total, 2 up, 3 in 2023-07-29T16:07:51.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:07:52.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:51 smithi156 ceph-mon[129245]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:52.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:51 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/3581114702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:52.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:51 smithi031 ceph-mon[95504]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:52.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:51 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/3581114702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:52.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:51 smithi039 ceph-mon[128858]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:52.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:51 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/3581114702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:07:54.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:54 smithi039 ceph-mon[128858]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:54.525 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:54 smithi156 ceph-mon[129245]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:54.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:54 smithi031 ceph-mon[95504]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:56.396 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:56 smithi156 ceph-mon[129245]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:56.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:56 smithi039 ceph-mon[128858]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:56.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:56 smithi031 ceph-mon[95504]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:57.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:57 smithi031 ceph-mon[95504]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:57.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T16:07:57.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:57.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:07:57 smithi031 ceph-mon[95504]: Deploying daemon osd.2 on smithi156 2023-07-29T16:07:58.011 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:57 smithi156 ceph-mon[129245]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:58.011 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T16:07:58.011 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:58.011 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:07:57 smithi156 ceph-mon[129245]: Deploying daemon osd.2 on smithi156 2023-07-29T16:07:58.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:57 smithi039 ceph-mon[128858]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:07:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-29T16:07:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:07:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:07:57 smithi039 ceph-mon[128858]: Deploying daemon osd.2 on smithi156 2023-07-29T16:08:00.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:00 smithi039 ceph-mon[128858]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:00.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:00 smithi156 ceph-mon[129245]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:00.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:00 smithi031 ceph-mon[95504]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:02.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:02 smithi156 ceph-mon[129245]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:02.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:02.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:02.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:02.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:02 smithi031 ceph-mon[95504]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:02.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:02.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:02.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:02.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:02 smithi039 ceph-mon[128858]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:02.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:02.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:02.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:04.164 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.168+0000 7f5e1b7fe700 1 -- 172.21.15.156:0/3245257951 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5e24061190 con 0x7f5e100a0850 2023-07-29T16:08:04.165 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 2 on host 'smithi156' 2023-07-29T16:08:04.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e100a0850 msgr2=0x7f5e100a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:04.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e100a0850 0x7f5e100a2d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5e14006fd0 tx=0x7f5e14008040 comp rx=0 tx=0).stop 2023-07-29T16:08:04.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 msgr2=0x7f5e2419d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:04.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 0x7f5e2419d580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00b970 tx=0x7f5e1c00bd30 comp rx=0 tx=0).stop 2023-07-29T16:08:04.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 shutdown_connections 2023-07-29T16:08:04.167 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e100a0850 0x7f5e100a2d00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:04.168 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e24100e50 0x7f5e2419d580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:04.168 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e240ff2c0 0x7f5e24069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:04.168 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 --2- 172.21.15.156:0/3245257951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e241004c0 0x7f5e2419af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:04.168 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 >> 172.21.15.156:0/3245257951 conn(0x7f5e240fa8f0 msgr2=0x7f5e24103990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:08:04.168 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 shutdown_connections 2023-07-29T16:08:04.168 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:04.170+0000 7f5e2c420700 1 -- 172.21.15.156:0/3245257951 wait complete. 2023-07-29T16:08:04.574 DEBUG:teuthology.orchestra.run.smithi156:osd.2> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.2.service 2023-07-29T16:08:04.577 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_3... 2023-07-29T16:08:04.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-29T16:08:04.611 INFO:journalctl@ceph.osd.2.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:08:04.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:04.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:04.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:04.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:04.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:08:05.569 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2[135103]: 2023-07-29T16:08:05.369+0000 7feb95c93640 -1 osd.2 0 log_to_monitors true 2023-07-29T16:08:05.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:05 smithi156 ceph-mon[129245]: from='osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T16:08:05.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:05 smithi156 ceph-mon[129245]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T16:08:05.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:05 smithi039 ceph-mon[128858]: from='osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T16:08:05.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:05 smithi039 ceph-mon[128858]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T16:08:05.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:05 smithi031 ceph-mon[95504]: from='osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T16:08:05.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:05 smithi031 ceph-mon[95504]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-29T16:08:06.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:06 smithi031 ceph-mon[95504]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:06 smithi031 ceph-mon[95504]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T16:08:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:06 smithi031 ceph-mon[95504]: osdmap e16: 3 total, 2 up, 3 in 2023-07-29T16:08:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:06 smithi031 ceph-mon[95504]: from='osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:08:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:06 smithi031 ceph-mon[95504]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:08:06.702 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2[135103]: 2023-07-29T16:08:06.500+0000 7feb8ab0a700 -1 osd.2 0 waiting for initial osdmap 2023-07-29T16:08:06.703 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2[135103]: 2023-07-29T16:08:06.504+0000 7feb830f8700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:08:06.703 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-mon[129245]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:06.703 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-mon[129245]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T16:08:06.703 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-mon[129245]: osdmap e16: 3 total, 2 up, 3 in 2023-07-29T16:08:06.704 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:06.704 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-mon[129245]: from='osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:08:06.704 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:06 smithi156 ceph-mon[129245]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:08:06.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:06 smithi039 ceph-mon[128858]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:06.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:06 smithi039 ceph-mon[128858]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-29T16:08:06.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:06 smithi039 ceph-mon[128858]: osdmap e16: 3 total, 2 up, 3 in 2023-07-29T16:08:06.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:06 smithi039 ceph-mon[128858]: from='osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:08:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:06 smithi039 ceph-mon[128858]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:08:07.164 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-29T16:08:07.164 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-29T16:08:07.165 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2023-07-29T16:08:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2023-07-29T16:08:07.165 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0400871 s, 262 MB/s 2023-07-29T16:08:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2023-07-29T16:08:07.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2023-07-29T16:08:07.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:08:07.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: osdmap e17: 3 total, 2 up, 3 in 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:08:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: osdmap e17: 3 total, 2 up, 3 in 2023-07-29T16:08:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:08:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: osdmap e17: 3 total, 2 up, 3 in 2023-07-29T16:08:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:08:08.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi156 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 17168M 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] boot 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: osdmap e18: 3 total, 3 up, 3 in 2023-07-29T16:08:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:08.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:08:08.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:08:08.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi156 2023-07-29T16:08:08.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 17168M 2023-07-29T16:08:08.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:08.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] boot 2023-07-29T16:08:08.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: osdmap e18: 3 total, 3 up, 3 in 2023-07-29T16:08:08.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:08.818 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi156 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 17168M 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: osd.2 [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] boot 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: osdmap e18: 3 total, 3 up, 3 in 2023-07-29T16:08:08.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-29T16:08:09.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.708+0000 7f389e34c700 1 -- 172.21.15.31:0/2187206723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 msgr2=0x7f3898073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.708+0000 7f389e34c700 1 --2- 172.21.15.31:0/2187206723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898073c20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f388c009990 tx=0x7f388c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.708+0000 7f389e34c700 1 -- 172.21.15.31:0/2187206723 shutdown_connections 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.708+0000 7f389e34c700 1 --2- 172.21.15.31:0/2187206723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 0x7f389810a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.709+0000 7f389e34c700 1 --2- 172.21.15.31:0/2187206723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3898074d80 0x7f38980731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.709+0000 7f389e34c700 1 --2- 172.21.15.31:0/2187206723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898073c20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.709+0000 7f389e34c700 1 -- 172.21.15.31:0/2187206723 >> 172.21.15.31:0/2187206723 conn(0x7f38980fb9e0 msgr2=0x7f38980fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:08:09.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.709+0000 7f389e34c700 1 -- 172.21.15.31:0/2187206723 shutdown_connections 2023-07-29T16:08:09.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.709+0000 7f389e34c700 1 -- 172.21.15.31:0/2187206723 wait complete. 2023-07-29T16:08:09.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 Processor -- start 2023-07-29T16:08:09.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 -- start start 2023-07-29T16:08:09.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:09.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3898074d80 0x7f3898198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:09.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 0x7f389819f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389819fe90 con 0x7f38981082a0 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389819ffd0 con 0x7f38980737b0 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.710+0000 7f389e34c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38981a0160 con 0x7f3898074d80 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56594/0 (socket says 172.21.15.31:56594) 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 -- 172.21.15.31:0/752307633 learned_addr learned my addr 172.21.15.31:0/752307633 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f38977fe700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3898074d80 0x7f3898198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:09.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f389c8e9700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 0x7f389819f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 -- 172.21.15.31:0/752307633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3898074d80 msgr2=0x7f3898198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3898074d80 0x7f3898198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 -- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 msgr2=0x7f389819f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 0x7f389819f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 -- 172.21.15.31:0/752307633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0095f0 con 0x7f38980737b0 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f389c8e9700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 0x7f389819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:08:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f3897fff700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898198300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f388800e9a0 tx=0x7f388800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:08:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.711+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888009790 con 0x7f38980737b0 2023-07-29T16:08:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.712+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f388800cbd0 con 0x7f38980737b0 2023-07-29T16:08:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.712+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38981a0440 con 0x7f38980737b0 2023-07-29T16:08:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.712+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888018680 con 0x7f38980737b0 2023-07-29T16:08:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.712+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38981a0990 con 0x7f38980737b0 2023-07-29T16:08:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.713+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3898106950 con 0x7f38980737b0 2023-07-29T16:08:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.715+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 15) v1 ==== 134417+0+0 (secure 0 0 0) 0x7f388800cd40 con 0x7f38980737b0 2023-07-29T16:08:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.716+0000 7f38957fa700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38800a0780 0x7f38800a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.716+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f3888014070 con 0x7f38980737b0 2023-07-29T16:08:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.716+0000 7f38977fe700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38800a0780 0x7f38800a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:09.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.717+0000 7f38977fe700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38800a0780 0x7f38800a2c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f388c00b2e0 tx=0x7f388c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:08:09.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.719+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3888095940 con 0x7f38980737b0 2023-07-29T16:08:09.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:09.934+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3898061190 con 0x7f38800a0780 2023-07-29T16:08:10.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:09 smithi031 ceph-mon[95504]: osdmap e19: 3 total, 3 up, 3 in 2023-07-29T16:08:10.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:09 smithi031 ceph-mon[95504]: pgmap v79: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:10.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:09 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:08:10.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:09 smithi039 ceph-mon[128858]: osdmap e19: 3 total, 3 up, 3 in 2023-07-29T16:08:10.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:09 smithi039 ceph-mon[128858]: pgmap v79: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:10.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:09 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:08:10.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:09 smithi156 ceph-mon[129245]: osdmap e19: 3 total, 3 up, 3 in 2023-07-29T16:08:10.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:09 smithi156 ceph-mon[129245]: pgmap v79: 0 pgs: ; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:10.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:09 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:08:11.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-29T16:08:11.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: osdmap e20: 3 total, 3 up, 3 in 2023-07-29T16:08:11.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:08:11.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:11.147 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:11.148 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:11.148 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:11.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-29T16:08:11.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: osdmap e20: 3 total, 3 up, 3 in 2023-07-29T16:08:11.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:08:11.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:11.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:11.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:11.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:11.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-29T16:08:11.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: osdmap e20: 3 total, 3 up, 3 in 2023-07-29T16:08:11.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:08:11.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:11.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:11.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:11.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:11.877 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:11 smithi031 sudo[111870]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-29T16:08:11.877 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:11 smithi031 sudo[111870]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:11.877 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:11 smithi031 sudo[111870]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:11.877 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:11 smithi031 sudo[111870]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:11.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:11.883+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3888095d10 con 0x7f38980737b0 2023-07-29T16:08:12.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:11 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T16:08:12.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:11 smithi031 ceph-mon[95504]: osdmap e21: 3 total, 3 up, 3 in 2023-07-29T16:08:12.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:11 smithi031 ceph-mon[95504]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:12.181 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:12 smithi031 sudo[111924]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-29T16:08:12.181 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:12 smithi031 sudo[111924]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:12.181 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:12 smithi031 sudo[111924]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:12.182 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:08:12 smithi031 sudo[111924]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:12.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:11 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T16:08:12.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:11 smithi039 ceph-mon[128858]: osdmap e21: 3 total, 3 up, 3 in 2023-07-29T16:08:12.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:11 smithi039 ceph-mon[128858]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:12.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:11 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-29T16:08:12.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:11 smithi156 ceph-mon[129245]: osdmap e21: 3 total, 3 up, 3 in 2023-07-29T16:08:12.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:11 smithi156 ceph-mon[129245]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 79 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:12 smithi039 sudo[136177]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-29T16:08:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:12 smithi039 sudo[136177]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:12 smithi039 sudo[136177]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:12.875 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:12 smithi039 sudo[136177]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:13.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:12 smithi031 ceph-mon[95504]: osdmap e22: 3 total, 3 up, 3 in 2023-07-29T16:08:13.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:12 smithi031 ceph-mon[95504]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-29T16:08:13.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:12 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:13.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:12 smithi039 ceph-mon[128858]: osdmap e22: 3 total, 3 up, 3 in 2023-07-29T16:08:13.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:12 smithi039 ceph-mon[128858]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-29T16:08:13.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:12 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:13.273 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:13 smithi039 sudo[136180]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-29T16:08:13.273 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:13 smithi039 sudo[136180]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:13.273 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:13 smithi039 sudo[136180]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:13.273 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:08:13 smithi039 sudo[136180]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:13.294 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:12 smithi156 ceph-mon[129245]: osdmap e22: 3 total, 3 up, 3 in 2023-07-29T16:08:13.295 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:12 smithi156 ceph-mon[129245]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-29T16:08:13.295 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:12 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:13.548 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137263]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-29T16:08:13.549 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:13.549 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137263]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:13.549 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137263]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:13.818 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137266]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-29T16:08:13.819 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137266]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:13.819 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137266]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:13.819 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:08:13 smithi156 sudo[137266]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:14.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112022]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-29T16:08:14.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112022]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:14.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112022]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:14.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112022]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:14.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112240]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-29T16:08:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112240]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 sudo[112240]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:14.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/589108722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de564104-d127-432d-b638-ab1547a64d6f"}]: dispatch 2023-07-29T16:08:14.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/589108722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de564104-d127-432d-b638-ab1547a64d6f"}]': finished 2023-07-29T16:08:14.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: osdmap e23: 4 total, 3 up, 4 in 2023-07-29T16:08:14.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:14 smithi031 ceph-mon[95504]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:14.934 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/589108722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de564104-d127-432d-b638-ab1547a64d6f"}]: dispatch 2023-07-29T16:08:14.934 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/589108722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de564104-d127-432d-b638-ab1547a64d6f"}]': finished 2023-07-29T16:08:14.934 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: osdmap e23: 4 total, 3 up, 4 in 2023-07-29T16:08:14.935 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:14.935 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:14.935 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:14.935 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:14.935 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 ceph-mon[129245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:14.935 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 sudo[137269]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-29T16:08:14.936 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 sudo[137269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:14.936 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 sudo[137269]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:14.936 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:14 smithi156 sudo[137269]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:15.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/589108722' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "de564104-d127-432d-b638-ab1547a64d6f"}]: dispatch 2023-07-29T16:08:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/589108722' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "de564104-d127-432d-b638-ab1547a64d6f"}]': finished 2023-07-29T16:08:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: osdmap e23: 4 total, 3 up, 4 in 2023-07-29T16:08:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:15.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:14 smithi039 ceph-mon[128858]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:15.318 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:15 smithi156 sudo[137272]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-29T16:08:15.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:15 smithi156 sudo[137272]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:15.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:15 smithi156 sudo[137272]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:15.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:15 smithi156 sudo[137272]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:16.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:15 smithi039 sudo[136183]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-29T16:08:16.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:15 smithi039 sudo[136183]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:16.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:15 smithi039 sudo[136183]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:16.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:15 smithi039 sudo[136183]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:16.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2778383384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:08:16.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:16.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:16.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:16.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:16 smithi031 ceph-mon[95504]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:16.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 sudo[136186]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-29T16:08:16.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 sudo[136186]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-29T16:08:16.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 sudo[136186]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 sudo[136186]: pam_unix(sudo:session): session closed for user root 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2778383384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:16 smithi039 ceph-mon[128858]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2778383384' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:08:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-29T16:08:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-29T16:08:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:16 smithi156 ceph-mon[129245]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:17.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:17 smithi031 ceph-mon[95504]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:17.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:17 smithi031 ceph-mon[95504]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:17.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:17 smithi039 ceph-mon[128858]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:17.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:17 smithi039 ceph-mon[128858]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:17.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:17 smithi156 ceph-mon[129245]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-29T16:08:17.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:17 smithi156 ceph-mon[129245]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-29T16:08:18.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:18 smithi031 ceph-mon[95504]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:18.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:18 smithi039 ceph-mon[128858]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:18.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:18 smithi156 ceph-mon[129245]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:20.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:20 smithi031 ceph-mon[95504]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-07-29T16:08:20.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:20 smithi156 ceph-mon[129245]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-07-29T16:08:20.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:20 smithi039 ceph-mon[128858]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-07-29T16:08:21.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T16:08:21.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:21.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:21 smithi156 ceph-mon[129245]: Deploying daemon osd.3 on smithi031 2023-07-29T16:08:21.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T16:08:21.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:21.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:21 smithi031 ceph-mon[95504]: Deploying daemon osd.3 on smithi031 2023-07-29T16:08:21.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-29T16:08:21.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:21.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:21 smithi039 ceph-mon[128858]: Deploying daemon osd.3 on smithi031 2023-07-29T16:08:22.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:22 smithi156 ceph-mon[129245]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-07-29T16:08:22.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:22 smithi031 ceph-mon[95504]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-07-29T16:08:22.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:22 smithi039 ceph-mon[128858]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.2 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-07-29T16:08:24.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:24 smithi156 ceph-mon[129245]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-07-29T16:08:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:24 smithi031 ceph-mon[95504]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-07-29T16:08:24.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:24 smithi039 ceph-mon[128858]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-07-29T16:08:26.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:26 smithi031 ceph-mon[95504]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:26.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:26 smithi156 ceph-mon[129245]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:26.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:26 smithi039 ceph-mon[128858]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:27.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:27.721 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:27.721 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:27.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:27.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:27.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:27.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:27.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:27.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:28.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:28 smithi039 ceph-mon[128858]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:28.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:28 smithi156 ceph-mon[129245]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:28.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:28 smithi031 ceph-mon[95504]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:29.897 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2023-07-29T16:08:29.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.901+0000 7f38957fa700 1 -- 172.21.15.31:0/752307633 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3898061190 con 0x7f38800a0780 2023-07-29T16:08:29.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.903+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38800a0780 msgr2=0x7f38800a2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:29.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.903+0000 7f389e34c700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38800a0780 0x7f38800a2c30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f388c00b2e0 tx=0x7f388c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:08:29.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.903+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 msgr2=0x7f3898198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:29.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.903+0000 7f389e34c700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898198300 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f388800e9a0 tx=0x7f388800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:08:29.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 shutdown_connections 2023-07-29T16:08:29.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38800a0780 0x7f38800a2c30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:29.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38981082a0 0x7f389819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38980737b0 0x7f3898198300 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 --2- 172.21.15.31:0/752307633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3898074d80 0x7f3898198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 >> 172.21.15.31:0/752307633 conn(0x7f38980fb9e0 msgr2=0x7f38980fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:08:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 shutdown_connections 2023-07-29T16:08:29.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:08:29.904+0000 7f389e34c700 1 -- 172.21.15.31:0/752307633 wait complete. 2023-07-29T16:08:30.273 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.273 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.274 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:30.342 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.3.service 2023-07-29T16:08:30.347 INFO:tasks.cephadm:Deploying osd.4 on smithi039 with /dev/vg_nvme/lv_3... 2023-07-29T16:08:30.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-29T16:08:30.380 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.380 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.381 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:30.436 INFO:journalctl@ceph.osd.3.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:08:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:30.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:30.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-07-29T16:08:30.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:30.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:30.681 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:08:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3[114237]: 2023-07-29T16:08:30.455+0000 7f33c6493640 -1 osd.3 0 log_to_monitors true 2023-07-29T16:08:30.917 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:08:31.357 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:31 smithi031 ceph-mon[95504]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T16:08:31.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:31 smithi039 ceph-mon[128858]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T16:08:31.568 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:31 smithi156 ceph-mon[129245]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-29T16:08:32.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-mon[95504]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T16:08:32.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-mon[95504]: osdmap e24: 4 total, 3 up, 4 in 2023-07-29T16:08:32.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:32.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-mon[95504]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:08:32.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-mon[95504]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:32.432 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3[114237]: 2023-07-29T16:08:32.171+0000 7f33bb30a700 -1 osd.3 0 waiting for initial osdmap 2023-07-29T16:08:32.432 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:08:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3[114237]: 2023-07-29T16:08:32.179+0000 7f33b7900700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:08:32.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:32 smithi039 ceph-mon[128858]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T16:08:32.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:32 smithi039 ceph-mon[128858]: osdmap e24: 4 total, 3 up, 4 in 2023-07-29T16:08:32.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:32 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:32.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:32 smithi039 ceph-mon[128858]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:08:32.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:32 smithi039 ceph-mon[128858]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:32.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:32 smithi156 ceph-mon[129245]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-29T16:08:32.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:32 smithi156 ceph-mon[129245]: osdmap e24: 4 total, 3 up, 4 in 2023-07-29T16:08:32.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:32 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:32.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:32 smithi156 ceph-mon[129245]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:08:32.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:32 smithi156 ceph-mon[129245]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:32.674 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-29T16:08:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-29T16:08:32.675 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-07-29T16:08:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-07-29T16:08:32.675 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00783371 s, 1.3 GB/s 2023-07-29T16:08:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-07-29T16:08:33.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_3 2023-07-29T16:08:33.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:33 smithi031 ceph-mon[95504]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:08:33.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:33 smithi031 ceph-mon[95504]: osdmap e25: 4 total, 3 up, 4 in 2023-07-29T16:08:33.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:33 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:33.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:33 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:33.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:33 smithi039 ceph-mon[128858]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:08:33.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:33 smithi039 ceph-mon[128858]: osdmap e25: 4 total, 3 up, 4 in 2023-07-29T16:08:33.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:33 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:33.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:33 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:33 smithi156 ceph-mon[129245]: from='osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:33 smithi156 ceph-mon[129245]: osdmap e25: 4 total, 3 up, 4 in 2023-07-29T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:33 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:33.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:33 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:33.665 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:08:34.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:08:34.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:08:34.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:34.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] boot 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: osdmap e26: 4 total, 4 up, 4 in 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi031 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 8636M 2023-07-29T16:08:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.441 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:08:34.441 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] boot 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: osdmap e26: 4 total, 4 up, 4 in 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi031 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.442 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 8636M 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:34.443 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:08:34.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:08:34.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:34.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: osd.3 [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] boot 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: osdmap e26: 4 total, 4 up, 4 in 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi031 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:34.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:34.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 8636M 2023-07-29T16:08:34.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:34.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:34.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:34.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 -- 172.21.15.39:0/2057727335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 msgr2=0x7f6094100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:34.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 --2- 172.21.15.39:0/2057727335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094100820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6090009990 tx=0x7f6090009ca0 comp rx=0 tx=0).stop 2023-07-29T16:08:34.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 -- 172.21.15.39:0/2057727335 shutdown_connections 2023-07-29T16:08:34.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 --2- 172.21.15.39:0/2057727335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6094101fa0 0x7f6094108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:34.963 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 --2- 172.21.15.39:0/2057727335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094100820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:34.963 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 --2- 172.21.15.39:0/2057727335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6094101610 0x7f6094101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:34.963 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 -- 172.21.15.39:0/2057727335 >> 172.21.15.39:0/2057727335 conn(0x7f60940fba80 msgr2=0x7f60940fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:08:34.963 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 -- 172.21.15.39:0/2057727335 shutdown_connections 2023-07-29T16:08:34.963 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.961+0000 7f609c742700 1 -- 172.21.15.39:0/2057727335 wait complete. 2023-07-29T16:08:34.963 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 Processor -- start 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 -- start start 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6094101610 0x7f6094196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6094101fa0 0x7f609419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609419dca0 con 0x7f6094101610 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609419dde0 con 0x7f6094100410 2023-07-29T16:08:34.964 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609c742700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609419df70 con 0x7f6094101fa0 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55064/0 (socket says 172.21.15.39:55064) 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 -- 172.21.15.39:0/802573835 learned_addr learned my addr 172.21.15.39:0/802573835 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f6099cdd700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6094101610 0x7f6094196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609acdf700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6094101fa0 0x7f609419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 -- 172.21.15.39:0/802573835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6094101fa0 msgr2=0x7f609419d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:34.965 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6094101fa0 0x7f609419d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 -- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6094101610 msgr2=0x7f6094196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6094101610 0x7f6094196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 -- 172.21.15.39:0/802573835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60900095f0 con 0x7f6094100410 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.962+0000 7f609a4de700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094196110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f60900095c0 tx=0x7f609000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.963+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609001d070 con 0x7f6094100410 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.963+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609419e1f0 con 0x7f6094100410 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.963+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f609000fb80 con 0x7f6094100410 2023-07-29T16:08:34.966 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.963+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090017650 con 0x7f6094100410 2023-07-29T16:08:34.967 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.963+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609419e660 con 0x7f6094100410 2023-07-29T16:08:34.967 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.964+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609410bbd0 con 0x7f6094100410 2023-07-29T16:08:34.967 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.966+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6090022470 con 0x7f6094100410 2023-07-29T16:08:34.968 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.966+0000 7f608b7fe700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60800a0790 0x7f60800a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:34.968 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.966+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f60900cc5a0 con 0x7f6094100410 2023-07-29T16:08:34.968 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.966+0000 7f6099cdd700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60800a0790 0x7f60800a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:34.968 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.966+0000 7f6099cdd700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60800a0790 0x7f60800a2c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6084005fd0 tx=0x7f608400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:08:34.968 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:34.970+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6090097230 con 0x7f6094100410 2023-07-29T16:08:35.173 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:35.176+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6094061190 con 0x7f60800a0790 2023-07-29T16:08:35.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:35 smithi039 ceph-mon[128858]: osdmap e27: 4 total, 4 up, 4 in 2023-07-29T16:08:35.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:35.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:35.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:35 smithi156 ceph-mon[129245]: osdmap e27: 4 total, 4 up, 4 in 2023-07-29T16:08:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:35.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:35.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:35 smithi031 ceph-mon[95504]: osdmap e27: 4 total, 4 up, 4 in 2023-07-29T16:08:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:36.480 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:36 smithi039 ceph-mon[128858]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:36.480 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:36 smithi039 ceph-mon[128858]: osdmap e28: 4 total, 4 up, 4 in 2023-07-29T16:08:36.480 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:36 smithi039 ceph-mon[128858]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:36 smithi156 ceph-mon[129245]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:36 smithi156 ceph-mon[129245]: osdmap e28: 4 total, 4 up, 4 in 2023-07-29T16:08:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:36 smithi156 ceph-mon[129245]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:36.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:36 smithi031 ceph-mon[95504]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:36.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:36 smithi031 ceph-mon[95504]: osdmap e28: 4 total, 4 up, 4 in 2023-07-29T16:08:36.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:36 smithi031 ceph-mon[95504]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:37.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:37 smithi039 ceph-mon[128858]: osdmap e29: 4 total, 4 up, 4 in 2023-07-29T16:08:37.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:37 smithi156 ceph-mon[129245]: osdmap e29: 4 total, 4 up, 4 in 2023-07-29T16:08:37.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:37 smithi031 ceph-mon[95504]: osdmap e29: 4 total, 4 up, 4 in 2023-07-29T16:08:38.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: osdmap e30: 4 total, 4 up, 4 in 2023-07-29T16:08:38.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:38.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/2841100223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]: dispatch 2023-07-29T16:08:38.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]: dispatch 2023-07-29T16:08:38.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]': finished 2023-07-29T16:08:38.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: osdmap e31: 5 total, 4 up, 5 in 2023-07-29T16:08:38.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:38.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: osdmap e30: 4 total, 4 up, 4 in 2023-07-29T16:08:38.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:38.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/2841100223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]: dispatch 2023-07-29T16:08:38.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]: dispatch 2023-07-29T16:08:38.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]': finished 2023-07-29T16:08:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: osdmap e31: 5 total, 4 up, 5 in 2023-07-29T16:08:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:38 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:38.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: osdmap e30: 4 total, 4 up, 4 in 2023-07-29T16:08:38.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:38.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/2841100223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]: dispatch 2023-07-29T16:08:38.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]: dispatch 2023-07-29T16:08:38.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8c0cb642-c0c5-4349-8c45-584df086b71b"}]': finished 2023-07-29T16:08:38.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: osdmap e31: 5 total, 4 up, 5 in 2023-07-29T16:08:38.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:38 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:39.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:39 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/29826199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:08:39.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:39 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/29826199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:08:40.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:39 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/29826199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:08:40.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:40 smithi156 ceph-mon[129245]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-07-29T16:08:40.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:40 smithi031 ceph-mon[95504]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-07-29T16:08:41.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:40 smithi039 ceph-mon[128858]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2023-07-29T16:08:42.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:42 smithi156 ceph-mon[129245]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:08:42.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:42 smithi031 ceph-mon[95504]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:08:43.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:42 smithi039 ceph-mon[128858]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:08:44.797 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:44 smithi039 ceph-mon[128858]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-29T16:08:44.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:44 smithi156 ceph-mon[129245]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-29T16:08:44.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:44 smithi031 ceph-mon[95504]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2023-07-29T16:08:45.930 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:45 smithi031 ceph-mon[95504]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-29T16:08:46.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:45 smithi039 ceph-mon[128858]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-29T16:08:46.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:45 smithi156 ceph-mon[129245]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-29T16:08:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T16:08:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:46.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:46 smithi031 ceph-mon[95504]: Deploying daemon osd.4 on smithi039 2023-07-29T16:08:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T16:08:47.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:46 smithi039 ceph-mon[128858]: Deploying daemon osd.4 on smithi039 2023-07-29T16:08:47.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-29T16:08:47.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:47.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:46 smithi156 ceph-mon[129245]: Deploying daemon osd.4 on smithi039 2023-07-29T16:08:47.892 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:47 smithi039 ceph-mon[128858]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:08:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:47 smithi031 ceph-mon[95504]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:08:48.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:47 smithi156 ceph-mon[129245]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:08:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:50 smithi156 ceph-mon[129245]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-29T16:08:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:50.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:50.604 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:50 smithi039 ceph-mon[128858]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-29T16:08:50.605 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:50.605 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:50.606 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:50.680 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:50 smithi031 ceph-mon[95504]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 50 KiB/s, 0 objects/s recovering 2023-07-29T16:08:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:50.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:52.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:52 smithi039 ceph-mon[128858]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:52.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:52.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:52.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:52 smithi156 ceph-mon[129245]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:52.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:52.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:52.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:52 smithi031 ceph-mon[95504]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:53.093 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 4 on host 'smithi039' 2023-07-29T16:08:53.094 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.097+0000 7f608b7fe700 1 -- 172.21.15.39:0/802573835 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6094061190 con 0x7f60800a0790 2023-07-29T16:08:53.095 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60800a0790 msgr2=0x7f60800a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60800a0790 0x7f60800a2c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6084005fd0 tx=0x7f608400a560 comp rx=0 tx=0).stop 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 msgr2=0x7f6094196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094196110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f60900095c0 tx=0x7f609000f690 comp rx=0 tx=0).stop 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 shutdown_connections 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60800a0790 0x7f60800a2c40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6094101fa0 0x7f609419d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:53.096 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6094100410 0x7f6094196110 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:53.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 --2- 172.21.15.39:0/802573835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6094101610 0x7f6094196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:53.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 >> 172.21.15.39:0/802573835 conn(0x7f60940fba80 msgr2=0x7f60940fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:08:53.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 shutdown_connections 2023-07-29T16:08:53.097 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:08:53.099+0000 7f609c742700 1 -- 172.21.15.39:0/802573835 wait complete. 2023-07-29T16:08:53.560 DEBUG:teuthology.orchestra.run.smithi039:osd.4> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.4.service 2023-07-29T16:08:53.563 INFO:tasks.cephadm:Deploying osd.5 on smithi156 with /dev/vg_nvme/lv_3... 2023-07-29T16:08:53.563 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-29T16:08:53.592 INFO:journalctl@ceph.osd.4.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:08:54.124 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:08:54.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:54 smithi039 ceph-mon[128858]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:54.273 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:08:54 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:08:54.132+0000 7f120285b640 -1 osd.4 0 log_to_monitors true 2023-07-29T16:08:54.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:54.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:54.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:54.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:54 smithi156 ceph-mon[129245]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:54.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:54.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:08:54.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:54.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:54 smithi031 ceph-mon[95504]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:55.138 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:55 smithi156 ceph-mon[129245]: from='osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T16:08:55.138 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:55 smithi156 ceph-mon[129245]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T16:08:55.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:55 smithi039 ceph-mon[128858]: from='osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T16:08:55.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:55 smithi039 ceph-mon[128858]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T16:08:55.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:55 smithi031 ceph-mon[95504]: from='osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T16:08:55.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:55 smithi031 ceph-mon[95504]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-29T16:08:55.595 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-29T16:08:55.595 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-29T16:08:55.595 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-29T16:08:55.595 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-29T16:08:55.595 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0082722 s, 1.3 GB/s 2023-07-29T16:08:55.596 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-29T16:08:56.067 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2023-07-29T16:08:56.272 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:08:56.083+0000 7f11f76d2700 -1 osd.4 0 waiting for initial osdmap 2023-07-29T16:08:56.273 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:08:56.091+0000 7f11f24c5700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:08:56.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T16:08:56.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: osdmap e32: 5 total, 4 up, 5 in 2023-07-29T16:08:56.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:56.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:08:56.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:56.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:56.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T16:08:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: osdmap e32: 5 total, 4 up, 5 in 2023-07-29T16:08:56.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:56.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:56.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:56.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:56.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-29T16:08:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: osdmap e32: 5 total, 4 up, 5 in 2023-07-29T16:08:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:08:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:08:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:08:56.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:08:56.631 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:08:57.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:57 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi039 2023-07-29T16:08:57.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:57 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 8584M 2023-07-29T16:08:57.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:57 smithi031 ceph-mon[95504]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:08:57.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:57 smithi031 ceph-mon[95504]: osdmap e33: 5 total, 4 up, 5 in 2023-07-29T16:08:57.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:57.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:57.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:57 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi039 2023-07-29T16:08:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:57 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 8584M 2023-07-29T16:08:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:57 smithi039 ceph-mon[128858]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:08:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:57 smithi039 ceph-mon[128858]: osdmap e33: 5 total, 4 up, 5 in 2023-07-29T16:08:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:57.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:57 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi039 2023-07-29T16:08:57.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:57 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 8584M 2023-07-29T16:08:57.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:57 smithi156 ceph-mon[129245]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:08:57.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:57 smithi156 ceph-mon[129245]: osdmap e33: 5 total, 4 up, 5 in 2023-07-29T16:08:57.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:57.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:57.837 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 -- 172.21.15.156:0/1736436741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 msgr2=0x7faa6c1000a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:57.837 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 --2- 172.21.15.156:0/1736436741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c1000a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7faa60009990 tx=0x7faa60009ca0 comp rx=0 tx=0).stop 2023-07-29T16:08:57.838 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 -- 172.21.15.156:0/1736436741 shutdown_connections 2023-07-29T16:08:57.838 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 --2- 172.21.15.156:0/1736436741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 0x7faa6c106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:57.838 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 --2- 172.21.15.156:0/1736436741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c0fea50 0x7faa6c0fee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:57.838 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 --2- 172.21.15.156:0/1736436741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c1000a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:57.838 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 -- 172.21.15.156:0/1736436741 >> 172.21.15.156:0/1736436741 conn(0x7faa6c0fa0a0 msgr2=0x7faa6c0fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:08:57.839 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 -- 172.21.15.156:0/1736436741 shutdown_connections 2023-07-29T16:08:57.839 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.841+0000 7faa73918700 1 -- 172.21.15.156:0/1736436741 wait complete. 2023-07-29T16:08:57.841 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 Processor -- start 2023-07-29T16:08:57.841 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 -- start start 2023-07-29T16:08:57.842 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c0fea50 0x7faa6c1029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:57.842 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c101030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:57.842 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 0x7faa6c101570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:57.842 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c101cf0 con 0x7faa6c0fea50 2023-07-29T16:08:57.842 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c101e60 con 0x7faa6c1005e0 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa73918700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c1a01d0 con 0x7faa6c0ffc50 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c101030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c101030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40560/0 (socket says 172.21.15.156:40560) 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 -- 172.21.15.156:0/1333515484 learned_addr learned my addr 172.21.15.156:0/1333515484 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa71eb5700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 0x7faa6c101570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 -- 172.21.15.156:0/1333515484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 msgr2=0x7faa6c101570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:57.843 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 0x7faa6c101570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 -- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c0fea50 msgr2=0x7faa6c1029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c0fea50 0x7faa6c1029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa70eb3700 1 -- 172.21.15.156:0/1333515484 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600095f0 con 0x7faa6c0ffc50 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.842+0000 7faa71eb5700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 0x7faa6c101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.843+0000 7faa70eb3700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c101030 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa60000c00 tx=0x7faa6000f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.843+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6001d070 con 0x7faa6c0ffc50 2023-07-29T16:08:57.844 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.843+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c1a0450 con 0x7faa6c0ffc50 2023-07-29T16:08:57.845 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.843+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa6000fc00 con 0x7faa6c0ffc50 2023-07-29T16:08:57.845 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.843+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa60017650 con 0x7faa6c0ffc50 2023-07-29T16:08:57.845 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.843+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c1a0940 con 0x7faa6c0ffc50 2023-07-29T16:08:57.845 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.844+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c10a2b0 con 0x7faa6c0ffc50 2023-07-29T16:08:57.845 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.845+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7faa60022470 con 0x7faa6c0ffc50 2023-07-29T16:08:57.845 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.845+0000 7faa5e7fc700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faa580a0790 0x7faa580a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:08:57.846 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.846+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7faa600cbf30 con 0x7faa6c0ffc50 2023-07-29T16:08:57.846 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.846+0000 7faa716b4700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faa580a0790 0x7faa580a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:08:57.846 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.846+0000 7faa716b4700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faa580a0790 0x7faa580a2c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0ffab0 tx=0x7faa68006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:08:57.846 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:57.849+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7faa60096ad0 con 0x7faa6c0ffc50 2023-07-29T16:08:58.039 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:08:58.043+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7faa6c1a0f80 con 0x7faa580a0790 2023-07-29T16:08:58.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:08:58.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:08:58.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:58.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] boot 2023-07-29T16:08:58.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: osdmap e34: 5 total, 5 up, 5 in 2023-07-29T16:08:58.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:58.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:58.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:58.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:58.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:58.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:08:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:08:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] boot 2023-07-29T16:08:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: osdmap e34: 5 total, 5 up, 5 in 2023-07-29T16:08:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:58.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:58.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:58.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:58.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:08:58.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: osd.4 [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] boot 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: osdmap e34: 5 total, 5 up, 5 in 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:08:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:08:59.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:59 smithi031 ceph-mon[95504]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:59.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:08:59 smithi031 ceph-mon[95504]: osdmap e35: 5 total, 5 up, 5 in 2023-07-29T16:08:59.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:59 smithi039 ceph-mon[128858]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:59.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:08:59 smithi039 ceph-mon[128858]: osdmap e35: 5 total, 5 up, 5 in 2023-07-29T16:08:59.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:59 smithi156 ceph-mon[129245]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:08:59.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:08:59 smithi156 ceph-mon[129245]: osdmap e35: 5 total, 5 up, 5 in 2023-07-29T16:09:00.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:00 smithi031 ceph-mon[95504]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:00.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:00 smithi039 ceph-mon[128858]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:00.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:00 smithi156 ceph-mon[129245]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:01.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/2717311327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]: dispatch 2023-07-29T16:09:01.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:01 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]: dispatch 2023-07-29T16:09:01.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:01 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]': finished 2023-07-29T16:09:01.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:01 smithi031 ceph-mon[95504]: osdmap e36: 6 total, 5 up, 6 in 2023-07-29T16:09:01.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:01.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/935004537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:01.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/2717311327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]: dispatch 2023-07-29T16:09:01.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:01 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]: dispatch 2023-07-29T16:09:01.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:01 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]': finished 2023-07-29T16:09:01.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:01 smithi039 ceph-mon[128858]: osdmap e36: 6 total, 5 up, 6 in 2023-07-29T16:09:01.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:01.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/935004537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/2717311327' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]: dispatch 2023-07-29T16:09:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:01 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]: dispatch 2023-07-29T16:09:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:01 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df"}]': finished 2023-07-29T16:09:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:01 smithi156 ceph-mon[129245]: osdmap e36: 6 total, 5 up, 6 in 2023-07-29T16:09:01.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:01.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/935004537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:02.430 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:02 smithi031 ceph-mon[95504]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:02.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:02 smithi039 ceph-mon[128858]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:02.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:02 smithi156 ceph-mon[129245]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:04.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:04 smithi156 ceph-mon[129245]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:04.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:04 smithi031 ceph-mon[95504]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:04.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:04 smithi039 ceph-mon[128858]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:06.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:06 smithi156 ceph-mon[129245]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:06 smithi031 ceph-mon[95504]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:06.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:06 smithi039 ceph-mon[128858]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T16:09:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:07 smithi031 ceph-mon[95504]: Deploying daemon osd.5 on smithi156 2023-07-29T16:09:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:07 smithi031 ceph-mon[95504]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:08.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T16:09:08.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:08.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:07 smithi039 ceph-mon[128858]: Deploying daemon osd.5 on smithi156 2023-07-29T16:09:08.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:07 smithi039 ceph-mon[128858]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:08.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-29T16:09:08.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:08.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:07 smithi156 ceph-mon[129245]: Deploying daemon osd.5 on smithi156 2023-07-29T16:09:08.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:07 smithi156 ceph-mon[129245]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:10.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:10 smithi156 ceph-mon[129245]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:10.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:10 smithi031 ceph-mon[95504]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:10.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:10 smithi039 ceph-mon[128858]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:12.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:11 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:12.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:11 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:12.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:11 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:12.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:11 smithi156 ceph-mon[129245]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:12.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:11 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:12.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:11 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:12.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:11 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:12.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:11 smithi031 ceph-mon[95504]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:12.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:11 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:12.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:11 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:12.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:11 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:12.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:11 smithi039 ceph-mon[128858]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:13.555 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 5 on host 'smithi156' 2023-07-29T16:09:13.556 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.558+0000 7faa5e7fc700 1 -- 172.21.15.156:0/1333515484 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faa6c1a0f80 con 0x7faa580a0790 2023-07-29T16:09:13.558 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faa580a0790 msgr2=0x7faa580a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:13.558 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faa580a0790 0x7faa580a2c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0ffab0 tx=0x7faa68006d20 comp rx=0 tx=0).stop 2023-07-29T16:09:13.559 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 msgr2=0x7faa6c101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:13.559 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c101030 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa60000c00 tx=0x7faa6000f710 comp rx=0 tx=0).stop 2023-07-29T16:09:13.559 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 shutdown_connections 2023-07-29T16:09:13.559 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faa580a0790 0x7faa580a2c40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:13.560 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa6c1005e0 0x7faa6c101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:13.560 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa6c0fea50 0x7faa6c1029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:13.560 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 --2- 172.21.15.156:0/1333515484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa6c0ffc50 0x7faa6c101030 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa60000c00 tx=0x7faa6000f710 comp rx=0 tx=0).stop 2023-07-29T16:09:13.561 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 >> 172.21.15.156:0/1333515484 conn(0x7faa6c0fa0a0 msgr2=0x7faa6c0fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:09:13.561 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 shutdown_connections 2023-07-29T16:09:13.561 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:09:13.561+0000 7faa73918700 1 -- 172.21.15.156:0/1333515484 wait complete. 2023-07-29T16:09:13.916 DEBUG:teuthology.orchestra.run.smithi156:osd.5> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.5.service 2023-07-29T16:09:13.918 INFO:tasks.cephadm:Deploying osd.6 on smithi031 with /dev/vg_nvme/lv_2... 2023-07-29T16:09:13.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-29T16:09:13.950 INFO:journalctl@ceph.osd.5.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:09:14.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:14.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:14.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:14.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:14 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:14.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:14 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:14.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:14.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:14.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:14.675 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:09:15.229 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:09:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:09:14.907+0000 7fb50d7ef640 -1 osd.5 0 log_to_monitors true 2023-07-29T16:09:15.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:15 smithi039 ceph-mon[128858]: from='osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T16:09:15.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:15 smithi039 ceph-mon[128858]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T16:09:15.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:15 smithi156 ceph-mon[129245]: from='osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T16:09:15.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:15 smithi156 ceph-mon[129245]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T16:09:15.605 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:15 smithi031 ceph-mon[95504]: from='osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T16:09:15.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:15 smithi031 ceph-mon[95504]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-29T16:09:16.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T16:09:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: osdmap e37: 6 total, 5 up, 6 in 2023-07-29T16:09:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:09:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:09:16.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:16.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.569 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:09:16.278+0000 7fb502666700 -1 osd.5 0 waiting for initial osdmap 2023-07-29T16:09:16.569 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:09:16.287+0000 7fb4fec5c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: osdmap e37: 6 total, 5 up, 6 in 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:09:16.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:16.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:16.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: osdmap e37: 6 total, 5 up, 6 in 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:16.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:16.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:16.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:16.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-29T16:09:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-29T16:09:16.737 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2023-07-29T16:09:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2023-07-29T16:09:16.737 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0303418 s, 346 MB/s 2023-07-29T16:09:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2023-07-29T16:09:17.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2023-07-29T16:09:17.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi156 2023-07-29T16:09:17.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 8584M 2023-07-29T16:09:17.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:09:17.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: osdmap e38: 6 total, 5 up, 6 in 2023-07-29T16:09:17.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] boot 2023-07-29T16:09:17.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: osdmap e39: 6 total, 6 up, 6 in 2023-07-29T16:09:17.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi156 2023-07-29T16:09:17.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 8584M 2023-07-29T16:09:17.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:09:17.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: osdmap e38: 6 total, 5 up, 6 in 2023-07-29T16:09:17.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] boot 2023-07-29T16:09:17.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: osdmap e39: 6 total, 6 up, 6 in 2023-07-29T16:09:17.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi156 2023-07-29T16:09:17.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 8584M 2023-07-29T16:09:17.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:09:17.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: osdmap e38: 6 total, 5 up, 6 in 2023-07-29T16:09:17.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: osd.5 [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] boot 2023-07-29T16:09:17.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: osdmap e39: 6 total, 6 up, 6 in 2023-07-29T16:09:17.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-29T16:09:17.993 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:09:18.520 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:18 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:09:18.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:18 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:09:18.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:18 smithi031 ceph-mon[95504]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:18.530 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:18 smithi031 ceph-mon[95504]: osdmap e40: 6 total, 6 up, 6 in 2023-07-29T16:09:18.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:18 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:09:18.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:18 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:09:18.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:18 smithi156 ceph-mon[129245]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:18.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:18 smithi156 ceph-mon[129245]: osdmap e40: 6 total, 6 up, 6 in 2023-07-29T16:09:18.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:18 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:09:18.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:18 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:09:18.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:18 smithi039 ceph-mon[128858]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-29T16:09:18.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:18 smithi039 ceph-mon[128858]: osdmap e40: 6 total, 6 up, 6 in 2023-07-29T16:09:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 -- 172.21.15.31:0/3429850122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c78101f60 msgr2=0x7f8c78108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3429850122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c78101f60 0x7f8c78108820 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c7000b0a0 tx=0x7f8c7000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:09:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 -- 172.21.15.31:0/3429850122 shutdown_connections 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3429850122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c78101f60 0x7f8c78108820 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3429850122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c781007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3429850122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c781015d0 0x7f8c78101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 -- 172.21.15.31:0/3429850122 >> 172.21.15.31:0/3429850122 conn(0x7f8c780fb9e0 msgr2=0x7f8c780fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 -- 172.21.15.31:0/3429850122 shutdown_connections 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.422+0000 7f8c7e767700 1 -- 172.21.15.31:0/3429850122 wait complete. 2023-07-29T16:09:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.423+0000 7f8c7e767700 1 Processor -- start 2023-07-29T16:09:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.423+0000 7f8c7e767700 1 -- start start 2023-07-29T16:09:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7e767700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c78198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7e767700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c781015d0 0x7f8c78198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7e767700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c78101f60 0x7f8c7819f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7e767700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7819ffc0 con 0x7f8c781015d0 2023-07-29T16:09:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7e767700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c781a0100 con 0x7f8c781003d0 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7e767700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c781a0290 con 0x7f8c78101f60 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c77fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c78198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c77fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c78198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56120/0 (socket says 172.21.15.31:56120) 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c77fff700 1 -- 172.21.15.31:0/3059732278 learned_addr learned my addr 172.21.15.31:0/3059732278 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c777fe700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c781015d0 0x7f8c78198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c7cd04700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c78101f60 0x7f8c7819f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:19.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c777fe700 1 -- 172.21.15.31:0/3059732278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c78101f60 msgr2=0x7f8c7819f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c777fe700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c78101f60 0x7f8c7819f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c777fe700 1 -- 172.21.15.31:0/3059732278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 msgr2=0x7f8c78198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c777fe700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c78198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c777fe700 1 -- 172.21.15.31:0/3059732278 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c70009d00 con 0x7f8c781015d0 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.424+0000 7f8c77fff700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c78198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.425+0000 7f8c777fe700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c781015d0 0x7f8c78198970 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00d980 tx=0x7f8c6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.425+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00f890 con 0x7f8c781015d0 2023-07-29T16:09:19.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.425+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c6c010460 con 0x7f8c781015d0 2023-07-29T16:09:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.425+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c781a0570 con 0x7f8c781015d0 2023-07-29T16:09:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.425+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c00e610 con 0x7f8c781015d0 2023-07-29T16:09:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.425+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c781a0ac0 con 0x7f8c781015d0 2023-07-29T16:09:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.426+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c78192aa0 con 0x7f8c781015d0 2023-07-29T16:09:19.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.429+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8c6c010890 con 0x7f8c781015d0 2023-07-29T16:09:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.430+0000 7f8c757fa700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8c600a0810 0x7f8c600a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.430+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f8c6c0ca5a0 con 0x7f8c781015d0 2023-07-29T16:09:19.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.430+0000 7f8c77fff700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8c600a0810 0x7f8c600a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:19.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.431+0000 7f8c77fff700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8c600a0810 0x7f8c600a2cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c68007fd0 tx=0x7f8c68007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:09:19.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.433+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8c6c094ec0 con 0x7f8c781015d0 2023-07-29T16:09:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:19.649+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c78061190 con 0x7f8c600a0810 2023-07-29T16:09:20.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:19 smithi039 ceph-mon[128858]: osdmap e41: 6 total, 6 up, 6 in 2023-07-29T16:09:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:19 smithi039 ceph-mon[128858]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:09:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:09:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:19 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:19 smithi156 ceph-mon[129245]: osdmap e41: 6 total, 6 up, 6 in 2023-07-29T16:09:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:19 smithi156 ceph-mon[129245]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:09:20.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:09:20.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:19 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:20.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:19 smithi031 ceph-mon[95504]: osdmap e41: 6 total, 6 up, 6 in 2023-07-29T16:09:20.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:19 smithi031 ceph-mon[95504]: pgmap v136: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:20.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:09:20.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:09:20.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:19 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:20.969 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:20 smithi031 ceph-mon[95504]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:09:20.970 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:20 smithi031 ceph-mon[95504]: osdmap e42: 6 total, 6 up, 6 in 2023-07-29T16:09:21.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:20 smithi039 ceph-mon[128858]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:09:21.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:20 smithi039 ceph-mon[128858]: osdmap e42: 6 total, 6 up, 6 in 2023-07-29T16:09:21.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:20 smithi156 ceph-mon[129245]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:09:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:20 smithi156 ceph-mon[129245]: osdmap e42: 6 total, 6 up, 6 in 2023-07-29T16:09:21.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:21 smithi031 ceph-mon[95504]: osdmap e43: 6 total, 6 up, 6 in 2023-07-29T16:09:21.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:21 smithi031 ceph-mon[95504]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:22.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:21 smithi039 ceph-mon[128858]: osdmap e43: 6 total, 6 up, 6 in 2023-07-29T16:09:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:21 smithi039 ceph-mon[128858]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:21 smithi156 ceph-mon[129245]: osdmap e43: 6 total, 6 up, 6 in 2023-07-29T16:09:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:21 smithi156 ceph-mon[129245]: pgmap v139: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:23.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2298833695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e93cbf91-6144-4ac7-8bb6-1e964c5945d9"}]: dispatch 2023-07-29T16:09:23.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2298833695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e93cbf91-6144-4ac7-8bb6-1e964c5945d9"}]': finished 2023-07-29T16:09:23.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:22 smithi039 ceph-mon[128858]: osdmap e44: 7 total, 6 up, 7 in 2023-07-29T16:09:23.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:23.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2298833695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e93cbf91-6144-4ac7-8bb6-1e964c5945d9"}]: dispatch 2023-07-29T16:09:23.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2298833695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e93cbf91-6144-4ac7-8bb6-1e964c5945d9"}]': finished 2023-07-29T16:09:23.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:22 smithi156 ceph-mon[129245]: osdmap e44: 7 total, 6 up, 7 in 2023-07-29T16:09:23.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:23.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2298833695' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e93cbf91-6144-4ac7-8bb6-1e964c5945d9"}]: dispatch 2023-07-29T16:09:23.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2298833695' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e93cbf91-6144-4ac7-8bb6-1e964c5945d9"}]': finished 2023-07-29T16:09:23.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:22 smithi031 ceph-mon[95504]: osdmap e44: 7 total, 6 up, 7 in 2023-07-29T16:09:23.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:23 smithi156 ceph-mon[129245]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2023-07-29T16:09:24.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1716033571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:23 smithi031 ceph-mon[95504]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2023-07-29T16:09:24.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1716033571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:24.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:23 smithi039 ceph-mon[128858]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2023-07-29T16:09:24.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1716033571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:26.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:26 smithi156 ceph-mon[129245]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:09:26.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:26 smithi031 ceph-mon[95504]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:09:26.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:26 smithi039 ceph-mon[128858]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:09:28.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:27 smithi039 ceph-mon[128858]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-29T16:09:28.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:27 smithi156 ceph-mon[129245]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-29T16:09:28.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:27 smithi031 ceph-mon[95504]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-29T16:09:30.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:30 smithi031 ceph-mon[95504]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-29T16:09:30.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T16:09:30.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:30 smithi156 ceph-mon[129245]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-29T16:09:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T16:09:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:30.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:30 smithi039 ceph-mon[128858]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-29T16:09:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-29T16:09:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:31.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:31 smithi156 ceph-mon[129245]: Deploying daemon osd.6 on smithi031 2023-07-29T16:09:31.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:31 smithi031 ceph-mon[95504]: Deploying daemon osd.6 on smithi031 2023-07-29T16:09:31.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:31 smithi039 ceph-mon[128858]: Deploying daemon osd.6 on smithi031 2023-07-29T16:09:32.559 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:32 smithi031 ceph-mon[95504]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:09:32.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:32 smithi156 ceph-mon[129245]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:09:32.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:32 smithi039 ceph-mon[128858]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:09:34.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:34 smithi031 ceph-mon[95504]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2023-07-29T16:09:34.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:34 smithi156 ceph-mon[129245]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2023-07-29T16:09:34.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:34 smithi039 ceph-mon[128858]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 54 KiB/s, 0 objects/s recovering 2023-07-29T16:09:36.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:36 smithi031 ceph-mon[95504]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:36.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:36.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:36.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:36 smithi156 ceph-mon[129245]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:36.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:36 smithi039 ceph-mon[128858]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:36.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:36.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:36.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:37.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:37 smithi031 ceph-mon[95504]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:38.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:37 smithi039 ceph-mon[128858]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:38.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:37 smithi156 ceph-mon[129245]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:39.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.608+0000 7f8c757fa700 1 -- 172.21.15.31:0/3059732278 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8c78061190 con 0x7f8c600a0810 2023-07-29T16:09:39.604 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 6 on host 'smithi031' 2023-07-29T16:09:39.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8c600a0810 msgr2=0x7f8c600a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:39.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8c600a0810 0x7f8c600a2cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8c68007fd0 tx=0x7f8c68007480 comp rx=0 tx=0).stop 2023-07-29T16:09:39.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c781015d0 msgr2=0x7f8c78198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:39.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c781015d0 0x7f8c78198970 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00d980 tx=0x7f8c6c00dc90 comp rx=0 tx=0).stop 2023-07-29T16:09:39.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 shutdown_connections 2023-07-29T16:09:39.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8c600a0810 0x7f8c600a2cc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c78101f60 0x7f8c7819f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8c781003d0 0x7f8c78198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 --2- 172.21.15.31:0/3059732278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8c781015d0 0x7f8c78198970 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.611+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 >> 172.21.15.31:0/3059732278 conn(0x7f8c780fb9e0 msgr2=0x7f8c78106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:09:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.612+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 shutdown_connections 2023-07-29T16:09:39.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:09:39.612+0000 7f8c7e767700 1 -- 172.21.15.31:0/3059732278 wait complete. 2023-07-29T16:09:40.018 DEBUG:teuthology.orchestra.run.smithi031:osd.6> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.6.service 2023-07-29T16:09:40.022 INFO:tasks.cephadm:Deploying osd.7 on smithi039 with /dev/vg_nvme/lv_2... 2023-07-29T16:09:40.022 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-29T16:09:40.104 INFO:journalctl@ceph.osd.6.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:09:40.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:40.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:40.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:40.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:40.431 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:09:40.124+0000 7fae1d29f640 -1 osd.6 0 log_to_monitors true 2023-07-29T16:09:40.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:40.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:40.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:40.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:40.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:40.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:40.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:09:40.606 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:09:41.238 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:41 smithi039 ceph-mon[128858]: from='osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T16:09:41.238 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:41 smithi039 ceph-mon[128858]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T16:09:41.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:41 smithi156 ceph-mon[129245]: from='osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T16:09:41.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:41 smithi156 ceph-mon[129245]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T16:09:41.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:41 smithi031 ceph-mon[95504]: from='osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T16:09:41.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:41 smithi031 ceph-mon[95504]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-29T16:09:42.300 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-29T16:09:42.300 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-29T16:09:42.300 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-07-29T16:09:42.300 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-07-29T16:09:42.301 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010436 s, 1.0 GB/s 2023-07-29T16:09:42.301 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-07-29T16:09:42.339 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:42 smithi039 ceph-mon[128858]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T16:09:42.339 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:42 smithi039 ceph-mon[128858]: osdmap e45: 7 total, 6 up, 7 in 2023-07-29T16:09:42.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:42.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:42 smithi039 ceph-mon[128858]: from='osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:09:42.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:42 smithi039 ceph-mon[128858]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:09:42.340 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:42 smithi039 ceph-mon[128858]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:42.419 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:09:42.098+0000 7fae13919700 -1 osd.6 0 waiting for initial osdmap 2023-07-29T16:09:42.419 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:09:42.113+0000 7fae0df0b700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:09:42.419 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-mon[95504]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T16:09:42.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-mon[95504]: osdmap e45: 7 total, 6 up, 7 in 2023-07-29T16:09:42.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:42.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-mon[95504]: from='osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:09:42.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-mon[95504]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:09:42.420 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:42 smithi031 ceph-mon[95504]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:42.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:42 smithi156 ceph-mon[129245]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-29T16:09:42.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:42 smithi156 ceph-mon[129245]: osdmap e45: 7 total, 6 up, 7 in 2023-07-29T16:09:42.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:42.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:42 smithi156 ceph-mon[129245]: from='osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:09:42.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:42 smithi156 ceph-mon[129245]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:09:42.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:42 smithi156 ceph-mon[129245]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:42.769 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_2 2023-07-29T16:09:43.096 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:09:43.328 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:09:43.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: osdmap e46: 7 total, 6 up, 7 in 2023-07-29T16:09:43.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:43.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:43.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:43.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.519 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:09:43.519 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: osdmap e46: 7 total, 6 up, 7 in 2023-07-29T16:09:43.519 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:43.519 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.520 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:43.521 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:43.521 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: osdmap e46: 7 total, 6 up, 7 in 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:09:43.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:43.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:43.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:09:43.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:09:44.373 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:09:44.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:09:44.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi031 2023-07-29T16:09:44.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 5757M 2023-07-29T16:09:44.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] boot 2023-07-29T16:09:44.374 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: osdmap e47: 7 total, 7 up, 7 in 2023-07-29T16:09:44.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:44.375 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:44 smithi039 ceph-mon[128858]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:44.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:09:44.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:09:44.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi031 2023-07-29T16:09:44.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 5757M 2023-07-29T16:09:44.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] boot 2023-07-29T16:09:44.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: osdmap e47: 7 total, 7 up, 7 in 2023-07-29T16:09:44.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:44.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:44 smithi031 ceph-mon[95504]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:44.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.548+0000 7f4625917700 1 -- 172.21.15.39:0/2442956244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 msgr2=0x7f4620100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:44.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 --2- 172.21.15.39:0/2442956244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f4620100870 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4610009990 tx=0x7f4610009ca0 comp rx=0 tx=0).stop 2023-07-29T16:09:44.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 -- 172.21.15.39:0/2442956244 shutdown_connections 2023-07-29T16:09:44.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 --2- 172.21.15.39:0/2442956244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101ff0 0x7f46201088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:44.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 --2- 172.21.15.39:0/2442956244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f4620100870 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:44.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 --2- 172.21.15.39:0/2442956244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101660 0x7f4620101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:44.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 -- 172.21.15.39:0/2442956244 >> 172.21.15.39:0/2442956244 conn(0x7f46200fba90 msgr2=0x7f46200fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:09:44.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 -- 172.21.15.39:0/2442956244 shutdown_connections 2023-07-29T16:09:44.546 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 -- 172.21.15.39:0/2442956244 wait complete. 2023-07-29T16:09:44.549 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.549+0000 7f4625917700 1 Processor -- start 2023-07-29T16:09:44.549 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 -- start start 2023-07-29T16:09:44.550 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f46201961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:44.550 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101660 0x7f4620196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:44.550 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101ff0 0x7f462019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:44.550 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462019dd50 con 0x7f4620101660 2023-07-29T16:09:44.551 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462019de90 con 0x7f4620100460 2023-07-29T16:09:44.551 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f4625917700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462019e020 con 0x7f4620101ff0 2023-07-29T16:09:44.551 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f46201961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:44.552 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f46201961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55176/0 (socket says 172.21.15.39:55176) 2023-07-29T16:09:44.552 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 -- 172.21.15.39:0/3364572057 learned_addr learned my addr 172.21.15.39:0/3364572057 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-07-29T16:09:44.552 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461f7fe700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101ff0 0x7f462019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:44.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461e7fc700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101660 0x7f4620196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:44.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 -- 172.21.15.39:0/3364572057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101ff0 msgr2=0x7f462019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:44.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101ff0 0x7f462019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:44.553 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 -- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101660 msgr2=0x7f4620196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:09:44.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101660 0x7f4620196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:09:44.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461effd700 1 -- 172.21.15.39:0/3364572057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46100095f0 con 0x7f4620100460 2023-07-29T16:09:44.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461f7fe700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101ff0 0x7f462019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:09:44.555 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.550+0000 7f461e7fc700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101660 0x7f4620196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:09:44.555 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.551+0000 7f461effd700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f46201961c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46100095c0 tx=0x7f461000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:09:44.555 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.551+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461001d070 con 0x7f4620100460 2023-07-29T16:09:44.555 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.551+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462019e2a0 con 0x7f4620100460 2023-07-29T16:09:44.555 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.551+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f461000fb80 con 0x7f4620100460 2023-07-29T16:09:44.556 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.551+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4610017650 con 0x7f4620100460 2023-07-29T16:09:44.556 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.551+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462019e6b0 con 0x7f4620100460 2023-07-29T16:09:44.556 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.552+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620107b00 con 0x7f4620100460 2023-07-29T16:09:44.556 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.553+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4610022470 con 0x7f4620100460 2023-07-29T16:09:44.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.553+0000 7f4624915700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46080a0790 0x7f46080a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:09:44.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.553+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f46100cc560 con 0x7f4620100460 2023-07-29T16:09:44.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.554+0000 7f461e7fc700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46080a0790 0x7f46080a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:09:44.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.554+0000 7f461e7fc700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46080a0790 0x7f46080a2c40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f461400bcf0 tx=0x7f461400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:09:44.558 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.557+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4610096da0 con 0x7f4620100460 2023-07-29T16:09:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:09:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:09:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi031 2023-07-29T16:09:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 5757M 2023-07-29T16:09:44.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: osd.6 [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] boot 2023-07-29T16:09:44.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: osdmap e47: 7 total, 7 up, 7 in 2023-07-29T16:09:44.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-29T16:09:44.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:44 smithi156 ceph-mon[129245]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-29T16:09:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:09:44.761+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4620061190 con 0x7f46080a0790 2023-07-29T16:09:45.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:45 smithi031 ceph-mon[95504]: osdmap e48: 7 total, 7 up, 7 in 2023-07-29T16:09:45.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:09:45.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:09:45.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:45.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:45 smithi039 ceph-mon[128858]: osdmap e48: 7 total, 7 up, 7 in 2023-07-29T16:09:45.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:09:45.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:09:45.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:45.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:45 smithi156 ceph-mon[129245]: osdmap e48: 7 total, 7 up, 7 in 2023-07-29T16:09:45.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:09:45.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:09:45.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:46.376 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:46 smithi039 ceph-mon[128858]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:09:46.377 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:46 smithi039 ceph-mon[128858]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:46 smithi031 ceph-mon[95504]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:09:46.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:46 smithi031 ceph-mon[95504]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:46 smithi156 ceph-mon[129245]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:09:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:46 smithi156 ceph-mon[129245]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:47 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/2560115357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]: dispatch 2023-07-29T16:09:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:47 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]: dispatch 2023-07-29T16:09:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:47 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]': finished 2023-07-29T16:09:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:47 smithi039 ceph-mon[128858]: osdmap e49: 8 total, 7 up, 8 in 2023-07-29T16:09:48.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:09:48.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:47 smithi039 ceph-mon[128858]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:48.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:47 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/2560115357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]: dispatch 2023-07-29T16:09:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:47 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]: dispatch 2023-07-29T16:09:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:47 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]': finished 2023-07-29T16:09:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:47 smithi156 ceph-mon[129245]: osdmap e49: 8 total, 7 up, 8 in 2023-07-29T16:09:48.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:09:48.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:47 smithi156 ceph-mon[129245]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:48.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:47 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/2560115357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]: dispatch 2023-07-29T16:09:48.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:47 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]: dispatch 2023-07-29T16:09:48.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:47 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "78eb79f8-f2ac-4b63-a6d7-431e91959528"}]': finished 2023-07-29T16:09:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:47 smithi031 ceph-mon[95504]: osdmap e49: 8 total, 7 up, 8 in 2023-07-29T16:09:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:09:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:47 smithi031 ceph-mon[95504]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:49.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:48 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/2608405497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:49.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:48 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/2608405497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:49.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:48 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/2608405497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:09:50.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:49 smithi039 ceph-mon[128858]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:50.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:49 smithi156 ceph-mon[129245]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:50.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:49 smithi031 ceph-mon[95504]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:52.476 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:52 smithi039 ceph-mon[128858]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:52.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:52 smithi156 ceph-mon[129245]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:52.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:52 smithi031 ceph-mon[95504]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:54.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:54 smithi039 ceph-mon[128858]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:54.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:54 smithi156 ceph-mon[129245]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:54.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:54 smithi031 ceph-mon[95504]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:55.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:55 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T16:09:55.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:55 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:55.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:55 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T16:09:55.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:55 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:55.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:55 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-29T16:09:55.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:55 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:09:56.522 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:56 smithi039 ceph-mon[128858]: Deploying daemon osd.7 on smithi039 2023-07-29T16:09:56.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:56 smithi039 ceph-mon[128858]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:56.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:56 smithi156 ceph-mon[129245]: Deploying daemon osd.7 on smithi039 2023-07-29T16:09:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:56 smithi156 ceph-mon[129245]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:56.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:56 smithi031 ceph-mon[95504]: Deploying daemon osd.7 on smithi039 2023-07-29T16:09:56.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:56 smithi031 ceph-mon[95504]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:09:57 smithi039 ceph-mon[128858]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:58.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:09:57 smithi156 ceph-mon[129245]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:09:58.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:09:57 smithi031 ceph-mon[95504]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:00.549 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:00 smithi039 ceph-mon[128858]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:00.549 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:00.550 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:00.550 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:00.550 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:00 smithi039 ceph-mon[128858]: overall HEALTH_OK 2023-07-29T16:10:00.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:00 smithi156 ceph-mon[129245]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:00.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:00.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:00.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:00.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:00 smithi156 ceph-mon[129245]: overall HEALTH_OK 2023-07-29T16:10:00.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:00 smithi031 ceph-mon[95504]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:00.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:00.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:00.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:00.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:00 smithi031 ceph-mon[95504]: overall HEALTH_OK 2023-07-29T16:10:02.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:02 smithi156 ceph-mon[129245]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:02.585 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:02 smithi039 ceph-mon[128858]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:02.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:02 smithi031 ceph-mon[95504]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:02.878 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.880+0000 7f4624915700 1 -- 172.21.15.39:0/3364572057 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4620061190 con 0x7f46080a0790 2023-07-29T16:10:02.878 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 7 on host 'smithi039' 2023-07-29T16:10:02.880 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46080a0790 msgr2=0x7f46080a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:02.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46080a0790 0x7f46080a2c40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f461400bcf0 tx=0x7f461400b2c0 comp rx=0 tx=0).stop 2023-07-29T16:10:02.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 msgr2=0x7f46201961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:02.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f46201961c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f46100095c0 tx=0x7f461000f690 comp rx=0 tx=0).stop 2023-07-29T16:10:02.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 shutdown_connections 2023-07-29T16:10:02.881 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46080a0790 0x7f46080a2c40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4620101ff0 0x7f462019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4620100460 0x7f46201961c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 --2- 172.21.15.39:0/3364572057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4620101660 0x7f4620196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 >> 172.21.15.39:0/3364572057 conn(0x7f46200fba90 msgr2=0x7f4620104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:10:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 shutdown_connections 2023-07-29T16:10:02.882 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:02.883+0000 7f4625917700 1 -- 172.21.15.39:0/3364572057 wait complete. 2023-07-29T16:10:03.317 DEBUG:teuthology.orchestra.run.smithi039:osd.7> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.7.service 2023-07-29T16:10:03.319 INFO:tasks.cephadm:Deploying osd.8 on smithi156 with /dev/vg_nvme/lv_2... 2023-07-29T16:10:03.319 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-29T16:10:03.350 INFO:journalctl@ceph.osd.7.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:10:03.627 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.627 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.628 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:03.628 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:03.628 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.628 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.628 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.628 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:03.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:03.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:03.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:03.884 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:10:03.891 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:10:03 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:10:03.629+0000 7f4ce6186640 -1 osd.7 0 log_to_monitors true 2023-07-29T16:10:03.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:03.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:03.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:03.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:04.692 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:04 smithi039 ceph-mon[128858]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:04.692 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:04 smithi039 ceph-mon[128858]: from='osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T16:10:04.692 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:04 smithi039 ceph-mon[128858]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T16:10:04.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:04 smithi156 ceph-mon[129245]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:04 smithi156 ceph-mon[129245]: from='osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T16:10:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:04 smithi156 ceph-mon[129245]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T16:10:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:04 smithi031 ceph-mon[95504]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:04.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:04 smithi031 ceph-mon[95504]: from='osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T16:10:04.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:04 smithi031 ceph-mon[95504]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-29T16:10:05.373 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-29T16:10:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-29T16:10:05.374 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-29T16:10:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-29T16:10:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00848187 s, 1.2 GB/s 2023-07-29T16:10:05.374 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-29T16:10:05.523 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:10:05.497+0000 7f4cdaffd700 -1 osd.7 0 waiting for initial osdmap 2023-07-29T16:10:05.523 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:10:05.509+0000 7f4cd5df0700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:10:05.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-mon[128858]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T16:10:05.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-mon[128858]: osdmap e50: 8 total, 7 up, 8 in 2023-07-29T16:10:05.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:05.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-mon[128858]: from='osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:10:05.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:05 smithi039 ceph-mon[128858]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:10:05.725 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:05 smithi156 ceph-mon[129245]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T16:10:05.726 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:05 smithi156 ceph-mon[129245]: osdmap e50: 8 total, 7 up, 8 in 2023-07-29T16:10:05.726 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:05.726 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:05 smithi156 ceph-mon[129245]: from='osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:10:05.726 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:05 smithi156 ceph-mon[129245]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:10:05.727 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2023-07-29T16:10:05.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:05 smithi031 ceph-mon[95504]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-29T16:10:05.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:05 smithi031 ceph-mon[95504]: osdmap e50: 8 total, 7 up, 8 in 2023-07-29T16:10:05.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:05.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:05 smithi031 ceph-mon[95504]: from='osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:10:05.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:05 smithi031 ceph-mon[95504]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:10:06.281 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:10:06.755 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:06.755 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:10:06.755 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: osdmap e51: 8 total, 7 up, 8 in 2023-07-29T16:10:06.755 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:06.755 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.756 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:06.757 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:06.757 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:10:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: osdmap e51: 8 total, 7 up, 8 in 2023-07-29T16:10:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-29T16:10:06.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:10:06.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: osdmap e51: 8 total, 7 up, 8 in 2023-07-29T16:10:06.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:06.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:06.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:07.455 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.458+0000 7f48385f7700 1 -- 172.21.15.156:0/1907943729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48300fe700 msgr2=0x7f48300feb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:07.455 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.458+0000 7f48385f7700 1 --2- 172.21.15.156:0/1907943729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48300fe700 0x7f48300feb10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f482c009990 tx=0x7f482c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:10:07.455 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- 172.21.15.156:0/1907943729 shutdown_connections 2023-07-29T16:10:07.455 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 --2- 172.21.15.156:0/1907943729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4830102110 0x7f48301089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:07.456 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 --2- 172.21.15.156:0/1907943729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48300fe700 0x7f48300feb10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:07.456 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 --2- 172.21.15.156:0/1907943729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4830101780 0x7f4830101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:07.456 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- 172.21.15.156:0/1907943729 >> 172.21.15.156:0/1907943729 conn(0x7f48300fa0d0 msgr2=0x7f48300fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:10:07.456 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- 172.21.15.156:0/1907943729 shutdown_connections 2023-07-29T16:10:07.456 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- 172.21.15.156:0/1907943729 wait complete. 2023-07-29T16:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 Processor -- start 2023-07-29T16:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- start start 2023-07-29T16:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48300fe700 0x7f48301010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4830101780 0x7f48300ff730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 0x7f48300ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:07.458 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48301003f0 con 0x7f48300fe700 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4830100560 con 0x7f4830101780 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.459+0000 7f48385f7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48301a0170 con 0x7f4830102110 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 0x7f48300ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 0x7f48300ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40674/0 (socket says 172.21.15.156:40674) 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 -- 172.21.15.156:0/1127396581 learned_addr learned my addr 172.21.15.156:0/1127396581 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4835b92700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4830101780 0x7f48300ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:07.459 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836393700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48300fe700 0x7f48301010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 -- 172.21.15.156:0/1127396581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4830101780 msgr2=0x7f48300ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4830101780 0x7f48300ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 -- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48300fe700 msgr2=0x7f48301010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48300fe700 0x7f48301010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 -- 172.21.15.156:0/1127396581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c0095f0 con 0x7f4830102110 2023-07-29T16:10:07.460 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f4836b94700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 0x7f48300ffc70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f482800b970 tx=0x7f482800bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482800c930 con 0x7f4830102110 2023-07-29T16:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4828014440 con 0x7f4830102110 2023-07-29T16:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48280075d0 con 0x7f4830102110 2023-07-29T16:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48301a0450 con 0x7f4830102110 2023-07-29T16:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.460+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48301a0900 con 0x7f4830102110 2023-07-29T16:10:07.461 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.461+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483010bc70 con 0x7f4830102110 2023-07-29T16:10:07.462 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.462+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f48280145b0 con 0x7f4830102110 2023-07-29T16:10:07.462 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.463+0000 7f48237fe700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f481c0a0890 0x7f481c0a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:07.462 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.463+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f4828006e40 con 0x7f4830102110 2023-07-29T16:10:07.462 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.463+0000 7f4836393700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f481c0a0890 0x7f481c0a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:07.463 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.464+0000 7f4836393700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f481c0a0890 0x7f481c0a2d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f482c006010 tx=0x7f482c003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:10:07.463 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.467+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f482804e720 con 0x7f4830102110 2023-07-29T16:10:07.659 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:07.663+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4830061190 con 0x7f481c0a0890 2023-07-29T16:10:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:10:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:10:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi039 2023-07-29T16:10:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 5722M 2023-07-29T16:10:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] boot 2023-07-29T16:10:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: osdmap e52: 8 total, 8 up, 8 in 2023-07-29T16:10:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:07 smithi039 ceph-mon[128858]: osdmap e53: 8 total, 8 up, 8 in 2023-07-29T16:10:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:10:07.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:10:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi039 2023-07-29T16:10:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 5722M 2023-07-29T16:10:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] boot 2023-07-29T16:10:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: osdmap e52: 8 total, 8 up, 8 in 2023-07-29T16:10:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:07 smithi156 ceph-mon[129245]: osdmap e53: 8 total, 8 up, 8 in 2023-07-29T16:10:07.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi039 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 5722M 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: osd.7 [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] boot 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: osdmap e52: 8 total, 8 up, 8 in 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-29T16:10:07.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:07 smithi031 ceph-mon[95504]: osdmap e53: 8 total, 8 up, 8 in 2023-07-29T16:10:08.731 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:08 smithi156 ceph-mon[129245]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:08.732 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:08.732 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:08.732 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:08.732 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:08 smithi156 ceph-mon[129245]: osdmap e54: 8 total, 8 up, 8 in 2023-07-29T16:10:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:08 smithi039 ceph-mon[128858]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:08 smithi039 ceph-mon[128858]: osdmap e54: 8 total, 8 up, 8 in 2023-07-29T16:10:08.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:08 smithi031 ceph-mon[95504]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:08.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:08.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:08.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:08.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:08 smithi031 ceph-mon[95504]: osdmap e54: 8 total, 8 up, 8 in 2023-07-29T16:10:09.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:09 smithi156 ceph-mon[129245]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:09.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:09 smithi156 ceph-mon[129245]: osdmap e55: 8 total, 8 up, 8 in 2023-07-29T16:10:09.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:09 smithi039 ceph-mon[128858]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:09.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:09 smithi039 ceph-mon[128858]: osdmap e55: 8 total, 8 up, 8 in 2023-07-29T16:10:09.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:09 smithi031 ceph-mon[95504]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:09.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:09 smithi031 ceph-mon[95504]: osdmap e55: 8 total, 8 up, 8 in 2023-07-29T16:10:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: osdmap e56: 8 total, 8 up, 8 in 2023-07-29T16:10:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/3610991800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]: dispatch 2023-07-29T16:10:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]: dispatch 2023-07-29T16:10:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]': finished 2023-07-29T16:10:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: osdmap e57: 9 total, 8 up, 9 in 2023-07-29T16:10:10.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:10.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:10.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: osdmap e56: 8 total, 8 up, 8 in 2023-07-29T16:10:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/3610991800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]: dispatch 2023-07-29T16:10:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]: dispatch 2023-07-29T16:10:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]': finished 2023-07-29T16:10:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: osdmap e57: 9 total, 8 up, 9 in 2023-07-29T16:10:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:10.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:10.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: osdmap e56: 8 total, 8 up, 8 in 2023-07-29T16:10:10.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/3610991800' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]: dispatch 2023-07-29T16:10:10.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]: dispatch 2023-07-29T16:10:10.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fead396d-f980-4122-8ec0-cbb58dd236fe"}]': finished 2023-07-29T16:10:10.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: osdmap e57: 9 total, 8 up, 9 in 2023-07-29T16:10:10.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:11.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:11 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/2111591485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:11.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:11 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/2111591485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:11.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:11 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/2111591485' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:12.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:12 smithi039 ceph-mon[128858]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:12.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:12 smithi156 ceph-mon[129245]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:12.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:12 smithi031 ceph-mon[95504]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:14.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:14 smithi039 ceph-mon[128858]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:14.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:14 smithi156 ceph-mon[129245]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:14 smithi031 ceph-mon[95504]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:16 smithi039 ceph-mon[128858]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-29T16:10:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:16.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:16 smithi156 ceph-mon[129245]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-29T16:10:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:16.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:16 smithi031 ceph-mon[95504]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:16.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-29T16:10:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:17.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:17 smithi039 ceph-mon[128858]: Deploying daemon osd.8 on smithi156 2023-07-29T16:10:17.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:17 smithi156 ceph-mon[129245]: Deploying daemon osd.8 on smithi156 2023-07-29T16:10:17.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:17 smithi031 ceph-mon[95504]: Deploying daemon osd.8 on smithi156 2023-07-29T16:10:18.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:18 smithi156 ceph-mon[129245]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:18.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:18 smithi031 ceph-mon[95504]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:19.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:18 smithi039 ceph-mon[128858]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:20.717 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:20 smithi156 ceph-mon[129245]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:20.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:20 smithi031 ceph-mon[95504]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:21.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:20 smithi039 ceph-mon[128858]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:21.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:21.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:21.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:21.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:21 smithi156 ceph-mon[129245]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:21 smithi039 ceph-mon[128858]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:22.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:22.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:22.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:22.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:21 smithi031 ceph-mon[95504]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:23.475 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.478+0000 7f48237fe700 1 -- 172.21.15.156:0/1127396581 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4830061190 con 0x7f481c0a0890 2023-07-29T16:10:23.475 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 8 on host 'smithi156' 2023-07-29T16:10:23.477 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f481c0a0890 msgr2=0x7f481c0a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:23.478 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f481c0a0890 0x7f481c0a2d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f482c006010 tx=0x7f482c003680 comp rx=0 tx=0).stop 2023-07-29T16:10:23.478 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 msgr2=0x7f48300ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:23.478 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 0x7f48300ffc70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f482800b970 tx=0x7f482800bc80 comp rx=0 tx=0).stop 2023-07-29T16:10:23.478 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 shutdown_connections 2023-07-29T16:10:23.478 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f481c0a0890 0x7f481c0a2d40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:23.478 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4830102110 0x7f48300ffc70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:23.479 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48300fe700 0x7f48301010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:23.479 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 --2- 172.21.15.156:0/1127396581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4830101780 0x7f48300ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:23.479 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.480+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 >> 172.21.15.156:0/1127396581 conn(0x7f48300fa0d0 msgr2=0x7f48300fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:10:23.479 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.481+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 shutdown_connections 2023-07-29T16:10:23.479 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:10:23.481+0000 7f48385f7700 1 -- 172.21.15.156:0/1127396581 wait complete. 2023-07-29T16:10:23.880 DEBUG:teuthology.orchestra.run.smithi156:osd.8> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.8.service 2023-07-29T16:10:23.882 INFO:tasks.cephadm:Deploying osd.9 on smithi031 with /dev/vg_nvme/lv_1... 2023-07-29T16:10:23.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-29T16:10:23.914 INFO:journalctl@ceph.osd.8.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:10:24.626 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:10:24.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:24 smithi156 ceph-mon[129245]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:24.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.662 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:24 smithi031 ceph-mon[95504]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:24 smithi039 ceph-mon[128858]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:25.069 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:10:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:10:24.663+0000 7fe8890e7640 -1 osd.8 0 log_to_monitors true 2023-07-29T16:10:25.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:25 smithi156 ceph-mon[129245]: from='osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-29T16:10:25.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:25 smithi156 ceph-mon[129245]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-29T16:10:25.769 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:25 smithi031 ceph-mon[95504]: from='osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-29T16:10:25.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:25 smithi031 ceph-mon[95504]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-29T16:10:25.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:25 smithi039 ceph-mon[128858]: from='osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-29T16:10:25.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:25 smithi039 ceph-mon[128858]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-29T16:10:26.528 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-29T16:10:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-29T16:10:26.529 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2023-07-29T16:10:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2023-07-29T16:10:26.529 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0319179 s, 329 MB/s 2023-07-29T16:10:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2023-07-29T16:10:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-29T16:10:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: osdmap e58: 9 total, 8 up, 9 in 2023-07-29T16:10:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.819 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:10:26.462+0000 7fe87df5e700 -1 osd.8 0 waiting for initial osdmap 2023-07-29T16:10:26.819 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:10:26.475+0000 7fe87a554700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:10:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-29T16:10:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: osdmap e58: 9 total, 8 up, 9 in 2023-07-29T16:10:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:10:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.905 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:26.905 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: osdmap e58: 9 total, 8 up, 9 in 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:26.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:26.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:26.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:26.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2023-07-29T16:10:27.642 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:10:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi156 2023-07-29T16:10:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 5722M 2023-07-29T16:10:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:10:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: osdmap e59: 9 total, 8 up, 9 in 2023-07-29T16:10:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] boot 2023-07-29T16:10:27.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: osdmap e60: 9 total, 9 up, 9 in 2023-07-29T16:10:27.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi156 2023-07-29T16:10:27.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 5722M 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: osdmap e59: 9 total, 8 up, 9 in 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] boot 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: osdmap e60: 9 total, 9 up, 9 in 2023-07-29T16:10:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi156 2023-07-29T16:10:27.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 5722M 2023-07-29T16:10:27.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:10:27.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: osdmap e59: 9 total, 8 up, 9 in 2023-07-29T16:10:27.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:27.861 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: osd.8 [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] boot 2023-07-29T16:10:27.861 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: osdmap e60: 9 total, 9 up, 9 in 2023-07-29T16:10:27.861 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-29T16:10:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:28 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:10:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:28 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:10:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:28 smithi039 ceph-mon[128858]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:28 smithi039 ceph-mon[128858]: osdmap e61: 9 total, 9 up, 9 in 2023-07-29T16:10:28.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:28 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:10:28.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:28 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:10:28.775 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:28 smithi031 ceph-mon[95504]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:28.776 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:28 smithi031 ceph-mon[95504]: osdmap e61: 9 total, 9 up, 9 in 2023-07-29T16:10:28.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:28 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:10:28.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:28 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:10:28.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:28 smithi156 ceph-mon[129245]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-29T16:10:28.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:28 smithi156 ceph-mon[129245]: osdmap e61: 9 total, 9 up, 9 in 2023-07-29T16:10:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1087983480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 msgr2=0x7f3dd4101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1087983480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 0x7f3dd4101a20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009990 tx=0x7f3dc4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:10:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1087983480 shutdown_connections 2023-07-29T16:10:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1087983480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd4101f60 0x7f3dd4108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1087983480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dd41003d0 0x7f3dd41007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1087983480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 0x7f3dd4101a20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1087983480 >> 172.21.15.31:0/1087983480 conn(0x7f3dd40fb9e0 msgr2=0x7f3dd40fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:10:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1087983480 shutdown_connections 2023-07-29T16:10:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.029+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1087983480 wait complete. 2023-07-29T16:10:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.030+0000 7f3ddb2b3700 1 Processor -- start 2023-07-29T16:10:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 -- start start 2023-07-29T16:10:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 0x7f3dd41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 0x7f3dd4198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dd4101f60 0x7f3dd419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd419fe60 con 0x7f3dd41003d0 2023-07-29T16:10:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd419ffa0 con 0x7f3dd41015d0 2023-07-29T16:10:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3ddb2b3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd41a0130 con 0x7f3dd4101f60 2023-07-29T16:10:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3dd904f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 0x7f3dd41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3dd904f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 0x7f3dd41982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52974/0 (socket says 172.21.15.31:52974) 2023-07-29T16:10:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3dd904f700 1 -- 172.21.15.31:0/1984583246 learned_addr learned my addr 172.21.15.31:0/1984583246 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:10:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3dd884e700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 0x7f3dd4198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.031+0000 7f3dd9850700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dd4101f60 0x7f3dd419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dd904f700 1 -- 172.21.15.31:0/1984583246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dd4101f60 msgr2=0x7f3dd419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dd904f700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dd4101f60 0x7f3dd419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dd904f700 1 -- 172.21.15.31:0/1984583246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 msgr2=0x7f3dd4198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dd904f700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 0x7f3dd4198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dd904f700 1 -- 172.21.15.31:0/1984583246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc40095f0 con 0x7f3dd41003d0 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dd904f700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 0x7f3dd41982d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000e9a0 tx=0x7f3dd000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0009790 con 0x7f3dd41003d0 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dd000c810 con 0x7f3dd41003d0 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd41a0410 con 0x7f3dd41003d0 2023-07-29T16:10:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0003c70 con 0x7f3dd41003d0 2023-07-29T16:10:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.032+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd41a0910 con 0x7f3dd41003d0 2023-07-29T16:10:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.033+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd4192a60 con 0x7f3dd41003d0 2023-07-29T16:10:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.036+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3dd000cc80 con 0x7f3dd41003d0 2023-07-29T16:10:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.037+0000 7f3dca7fc700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3dc00a0790 0x7f3dc00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.037+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f3dd0014070 con 0x7f3dd41003d0 2023-07-29T16:10:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.037+0000 7f3dd884e700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3dc00a0790 0x7f3dc00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.038+0000 7f3dd884e700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3dc00a0790 0x7f3dc00a2c40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400b680 tx=0x7f3dc4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:10:29.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.039+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3dd0095880 con 0x7f3dd41003d0 2023-07-29T16:10:29.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:29.252+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dd4061190 con 0x7f3dc00a0790 2023-07-29T16:10:30.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:29 smithi039 ceph-mon[128858]: osdmap e62: 9 total, 9 up, 9 in 2023-07-29T16:10:30.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:29 smithi039 ceph-mon[128858]: pgmap v192: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:30.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:29 smithi039 ceph-mon[128858]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:30.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:30.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:30.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:30.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:29 smithi156 ceph-mon[129245]: osdmap e62: 9 total, 9 up, 9 in 2023-07-29T16:10:30.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:29 smithi156 ceph-mon[129245]: pgmap v192: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:30.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:29 smithi156 ceph-mon[129245]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:30.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:30.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:30.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:30.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:29 smithi031 ceph-mon[95504]: osdmap e62: 9 total, 9 up, 9 in 2023-07-29T16:10:30.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:29 smithi031 ceph-mon[95504]: pgmap v192: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:30.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:29 smithi031 ceph-mon[95504]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:30.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:30.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:30.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:30.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:30 smithi031 ceph-mon[95504]: osdmap e63: 9 total, 9 up, 9 in 2023-07-29T16:10:31.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:30 smithi039 ceph-mon[128858]: osdmap e63: 9 total, 9 up, 9 in 2023-07-29T16:10:31.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:30 smithi156 ceph-mon[129245]: osdmap e63: 9 total, 9 up, 9 in 2023-07-29T16:10:32.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:31 smithi039 ceph-mon[128858]: osdmap e64: 9 total, 9 up, 9 in 2023-07-29T16:10:32.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:31 smithi039 ceph-mon[128858]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:32.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:31 smithi156 ceph-mon[129245]: osdmap e64: 9 total, 9 up, 9 in 2023-07-29T16:10:32.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:31 smithi156 ceph-mon[129245]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:32.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:31 smithi031 ceph-mon[95504]: osdmap e64: 9 total, 9 up, 9 in 2023-07-29T16:10:32.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:31 smithi031 ceph-mon[95504]: pgmap v195: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:33.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:32 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3607085931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4646b9f8-76b7-4985-b448-0323ecbe272a"}]: dispatch 2023-07-29T16:10:33.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:32 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3607085931' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4646b9f8-76b7-4985-b448-0323ecbe272a"}]': finished 2023-07-29T16:10:33.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:32 smithi039 ceph-mon[128858]: osdmap e65: 10 total, 9 up, 10 in 2023-07-29T16:10:33.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:32 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:33.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:32 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3607085931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4646b9f8-76b7-4985-b448-0323ecbe272a"}]: dispatch 2023-07-29T16:10:33.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:32 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3607085931' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4646b9f8-76b7-4985-b448-0323ecbe272a"}]': finished 2023-07-29T16:10:33.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:32 smithi156 ceph-mon[129245]: osdmap e65: 10 total, 9 up, 10 in 2023-07-29T16:10:33.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:32 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:33.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:32 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3607085931' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4646b9f8-76b7-4985-b448-0323ecbe272a"}]: dispatch 2023-07-29T16:10:33.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:32 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3607085931' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4646b9f8-76b7-4985-b448-0323ecbe272a"}]': finished 2023-07-29T16:10:33.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:32 smithi031 ceph-mon[95504]: osdmap e65: 10 total, 9 up, 10 in 2023-07-29T16:10:33.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:32 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:34.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:33 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1124105710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:34.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:33 smithi039 ceph-mon[128858]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2023-07-29T16:10:34.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:33 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1124105710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:34.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:33 smithi156 ceph-mon[129245]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2023-07-29T16:10:34.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:33 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1124105710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:34.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:33 smithi031 ceph-mon[95504]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 127 KiB/s, 0 objects/s recovering 2023-07-29T16:10:36.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:36 smithi031 ceph-mon[95504]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:10:36.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:36 smithi156 ceph-mon[129245]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:10:36.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:36 smithi039 ceph-mon[128858]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-29T16:10:37.975 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:37 smithi031 ceph-mon[95504]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-29T16:10:38.022 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:37 smithi039 ceph-mon[128858]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-29T16:10:38.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:37 smithi156 ceph-mon[129245]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-29T16:10:40.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:40 smithi156 ceph-mon[129245]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-29T16:10:40.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-29T16:10:40.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:40.587 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:40 smithi031 ceph-mon[95504]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-29T16:10:40.587 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-29T16:10:40.587 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:40.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:40 smithi039 ceph-mon[128858]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-07-29T16:10:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-29T16:10:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:41.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:41 smithi156 ceph-mon[129245]: Deploying daemon osd.9 on smithi031 2023-07-29T16:10:41.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:41 smithi031 ceph-mon[95504]: Deploying daemon osd.9 on smithi031 2023-07-29T16:10:41.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:41 smithi039 ceph-mon[128858]: Deploying daemon osd.9 on smithi031 2023-07-29T16:10:42.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:42 smithi156 ceph-mon[129245]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:10:42.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:42 smithi031 ceph-mon[95504]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:10:42.772 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:42 smithi039 ceph-mon[128858]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-29T16:10:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:44 smithi156 ceph-mon[129245]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:44.592 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:44 smithi031 ceph-mon[95504]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:44.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:44 smithi039 ceph-mon[128858]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:46 smithi156 ceph-mon[129245]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:46 smithi031 ceph-mon[95504]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:46 smithi039 ceph-mon[128858]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:47.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:47 smithi031 ceph-mon[95504]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:47 smithi039 ceph-mon[128858]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:48.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:47 smithi156 ceph-mon[129245]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:49.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.268+0000 7f3dca7fc700 1 -- 172.21.15.31:0/1984583246 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3dd4061190 con 0x7f3dc00a0790 2023-07-29T16:10:49.265 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 9 on host 'smithi031' 2023-07-29T16:10:49.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3dc00a0790 msgr2=0x7f3dc00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:49.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3dc00a0790 0x7f3dc00a2c40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3dc400b680 tx=0x7f3dc4005fd0 comp rx=0 tx=0).stop 2023-07-29T16:10:49.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 msgr2=0x7f3dd41982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:49.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 0x7f3dd41982d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000e9a0 tx=0x7f3dd000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:10:49.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 shutdown_connections 2023-07-29T16:10:49.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3dc00a0790 0x7f3dc00a2c40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dd4101f60 0x7f3dd419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dd41003d0 0x7f3dd41982d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 --2- 172.21.15.31:0/1984583246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dd41015d0 0x7f3dd4198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 >> 172.21.15.31:0/1984583246 conn(0x7f3dd40fb9e0 msgr2=0x7f3dd4106f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:10:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 shutdown_connections 2023-07-29T16:10:49.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:10:49.271+0000 7f3ddb2b3700 1 -- 172.21.15.31:0/1984583246 wait complete. 2023-07-29T16:10:49.773 DEBUG:teuthology.orchestra.run.smithi031:osd.9> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.9.service 2023-07-29T16:10:49.777 INFO:tasks.cephadm:Deploying osd.10 on smithi039 with /dev/vg_nvme/lv_1... 2023-07-29T16:10:49.777 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-29T16:10:49.867 INFO:journalctl@ceph.osd.9.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:10:50.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:50.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:50.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:50.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:49 smithi039 ceph-mon[128858]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:50.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:50.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:49 smithi156 ceph-mon[129245]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:50.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:10:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:49 smithi031 ceph-mon[95504]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:50.353 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:10:50.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:50 smithi031 ceph-mon[95504]: from='osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-29T16:10:50.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:50 smithi031 ceph-mon[95504]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-29T16:10:51.003 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:50 smithi039 ceph-mon[128858]: from='osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-29T16:10:51.003 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:50 smithi039 ceph-mon[128858]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-29T16:10:51.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:50 smithi156 ceph-mon[129245]: from='osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-29T16:10:51.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:50 smithi156 ceph-mon[129245]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-29T16:10:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:51 smithi039 ceph-mon[128858]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-29T16:10:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:51 smithi039 ceph-mon[128858]: osdmap e66: 10 total, 9 up, 10 in 2023-07-29T16:10:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:52.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:51 smithi039 ceph-mon[128858]: from='osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:10:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:51 smithi039 ceph-mon[128858]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:10:52.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:51 smithi039 ceph-mon[128858]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:52.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:51 smithi156 ceph-mon[129245]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-29T16:10:52.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:51 smithi156 ceph-mon[129245]: osdmap e66: 10 total, 9 up, 10 in 2023-07-29T16:10:52.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:52.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:51 smithi156 ceph-mon[129245]: from='osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:10:52.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:51 smithi156 ceph-mon[129245]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:10:52.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:51 smithi156 ceph-mon[129245]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:52.107 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-29T16:10:52.107 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-29T16:10:52.108 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2023-07-29T16:10:52.108 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2023-07-29T16:10:52.108 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0150086 s, 699 MB/s 2023-07-29T16:10:52.108 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2023-07-29T16:10:52.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-mon[95504]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-29T16:10:52.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-mon[95504]: osdmap e66: 10 total, 9 up, 10 in 2023-07-29T16:10:52.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:52.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-mon[95504]: from='osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:10:52.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-mon[95504]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-07-29T16:10:52.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-mon[95504]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:52.135 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:10:51.786+0000 7feaf6e6e700 -1 osd.9 0 waiting for initial osdmap 2023-07-29T16:10:52.135 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:10:51 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:10:51.804+0000 7feaf1460700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:10:52.501 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi039:vg_nvme/lv_1 2023-07-29T16:10:53.062 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:10:53.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: osdmap e67: 10 total, 9 up, 10 in 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi031 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:53.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:10:53.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: osdmap e67: 10 total, 9 up, 10 in 2023-07-29T16:10:53.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi031 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:53.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:53.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-07-29T16:10:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: osdmap e67: 10 total, 9 up, 10 in 2023-07-29T16:10:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi031 2023-07-29T16:10:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:53.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:53.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:10:53.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:10:54.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:10:54.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:10:54.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 4318M 2023-07-29T16:10:54.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] boot 2023-07-29T16:10:54.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: osdmap e68: 10 total, 10 up, 10 in 2023-07-29T16:10:54.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:54.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:53 smithi039 ceph-mon[128858]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:54.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:10:54.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:10:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 4318M 2023-07-29T16:10:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] boot 2023-07-29T16:10:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: osdmap e68: 10 total, 10 up, 10 in 2023-07-29T16:10:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:54.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:53 smithi156 ceph-mon[129245]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:54.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:10:54.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:10:54.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 4318M 2023-07-29T16:10:54.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: osd.9 [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] boot 2023-07-29T16:10:54.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: osdmap e68: 10 total, 10 up, 10 in 2023-07-29T16:10:54.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-29T16:10:54.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:53 smithi031 ceph-mon[95504]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-29T16:10:54.318 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 -- 172.21.15.39:0/2387946300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 msgr2=0x7fa1d4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/2387946300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d4100d60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009990 tx=0x7fa1c4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 -- 172.21.15.39:0/2387946300 shutdown_connections 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/2387946300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1d40ffed0 0x7fa1d4107780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/2387946300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d4100d60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/2387946300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1d40ff520 0x7fa1d40ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 -- 172.21.15.39:0/2387946300 >> 172.21.15.39:0/2387946300 conn(0x7fa1d40fa0a0 msgr2=0x7fa1d40fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 -- 172.21.15.39:0/2387946300 shutdown_connections 2023-07-29T16:10:54.319 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.320+0000 7fa1d9e48700 1 -- 172.21.15.39:0/2387946300 wait complete. 2023-07-29T16:10:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 Processor -- start 2023-07-29T16:10:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 -- start start 2023-07-29T16:10:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1d40ff520 0x7fa1d4196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:54.321 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1d40ffed0 0x7fa1d4196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d419dce0 con 0x7fa1d40ff520 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d419de20 con 0x7fa1d4100950 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.321+0000 7fa1d9e48700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d419dfb0 con 0x7fa1d40ffed0 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d419d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55328/0 (socket says 172.21.15.39:55328) 2023-07-29T16:10:54.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 -- 172.21.15.39:0/4047420052 learned_addr learned my addr 172.21.15.39:0/4047420052 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d37fe700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1d40ff520 0x7fa1d4196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d2ffd700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1d40ffed0 0x7fa1d4196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 -- 172.21.15.39:0/4047420052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1d40ffed0 msgr2=0x7fa1d4196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1d40ffed0 0x7fa1d4196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 -- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1d40ff520 msgr2=0x7fa1d4196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1d40ff520 0x7fa1d4196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:10:54.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 -- 172.21.15.39:0/4047420052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40095f0 con 0x7fa1d4100950 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d3fff700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d419d710 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc007ee0 tx=0x7fa1cc00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc017070 con 0x7fa1d4100950 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d419e290 con 0x7fa1d4100950 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1cc00f040 con 0x7fa1d4100950 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc005740 con 0x7fa1d4100950 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.322+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d419e740 con 0x7fa1d4100950 2023-07-29T16:10:54.324 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.323+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d410bc10 con 0x7fa1d4100950 2023-07-29T16:10:54.325 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.325+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa1cc00dc60 con 0x7fa1d4100950 2023-07-29T16:10:54.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.325+0000 7fa1d0ff9700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1bc0a0790 0x7fa1bc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:10:54.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.326+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fa1cc0cb010 con 0x7fa1d4100950 2023-07-29T16:10:54.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.326+0000 7fa1d37fe700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1bc0a0790 0x7fa1bc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:10:54.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.327+0000 7fa1d37fe700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1bc0a0790 0x7fa1bc0a2c40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b5c0 tx=0x7fa1c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:10:54.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.329+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa1cc0952b0 con 0x7fa1d4100950 2023-07-29T16:10:54.533 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:10:54.536+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1d4061190 con 0x7fa1bc0a0790 2023-07-29T16:10:55.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:54 smithi156 ceph-mon[129245]: osdmap e69: 10 total, 10 up, 10 in 2023-07-29T16:10:55.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:54 smithi156 ceph-mon[129245]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:55.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:55.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:55.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:55.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:54 smithi031 ceph-mon[95504]: osdmap e69: 10 total, 10 up, 10 in 2023-07-29T16:10:55.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:54 smithi031 ceph-mon[95504]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:55.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:55.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:55.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:55.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:54 smithi039 ceph-mon[128858]: osdmap e69: 10 total, 10 up, 10 in 2023-07-29T16:10:55.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:54 smithi039 ceph-mon[128858]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:10:55.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:10:55.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:10:55.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:10:56.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:55 smithi156 ceph-mon[129245]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:10:56.119 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:55 smithi039 ceph-mon[128858]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:10:56.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:55 smithi031 ceph-mon[95504]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:10:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:57 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/2124449413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]: dispatch 2023-07-29T16:10:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:57 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]: dispatch 2023-07-29T16:10:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:57 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]': finished 2023-07-29T16:10:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:57 smithi039 ceph-mon[128858]: osdmap e70: 11 total, 10 up, 11 in 2023-07-29T16:10:58.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:10:58.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:57 smithi039 ceph-mon[128858]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:10:58.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:57 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/2124449413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]: dispatch 2023-07-29T16:10:58.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:57 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]: dispatch 2023-07-29T16:10:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:57 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]': finished 2023-07-29T16:10:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:57 smithi156 ceph-mon[129245]: osdmap e70: 11 total, 10 up, 11 in 2023-07-29T16:10:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:10:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:57 smithi156 ceph-mon[129245]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:10:58.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:57 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/2124449413' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]: dispatch 2023-07-29T16:10:58.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:57 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]: dispatch 2023-07-29T16:10:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:57 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ced3e9f-39cb-4112-bd93-5a36a3949bd7"}]': finished 2023-07-29T16:10:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:57 smithi031 ceph-mon[95504]: osdmap e70: 11 total, 10 up, 11 in 2023-07-29T16:10:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:10:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:57 smithi031 ceph-mon[95504]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:10:59.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:58 smithi156 ceph-mon[129245]: from='client.? 172.21.15.39:0/2988966545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:59.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:58 smithi031 ceph-mon[95504]: from='client.? 172.21.15.39:0/2988966545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:10:59.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:58 smithi039 ceph-mon[128858]: from='client.? 172.21.15.39:0/2988966545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:11:00.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:10:59 smithi156 ceph-mon[129245]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:00.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:10:59 smithi031 ceph-mon[95504]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:00.272 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:10:59 smithi039 ceph-mon[128858]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:02.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:02 smithi039 ceph-mon[128858]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:02.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:02 smithi156 ceph-mon[129245]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:02.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:02 smithi031 ceph-mon[95504]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:04.386 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:04 smithi039 ceph-mon[128858]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:04.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:04 smithi156 ceph-mon[129245]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:04.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:04 smithi031 ceph-mon[95504]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:05.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-29T16:11:05.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:05.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-29T16:11:05.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:05.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-29T16:11:05.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:06.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:06 smithi039 ceph-mon[128858]: Deploying daemon osd.10 on smithi039 2023-07-29T16:11:06.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:06 smithi039 ceph-mon[128858]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:06 smithi156 ceph-mon[129245]: Deploying daemon osd.10 on smithi039 2023-07-29T16:11:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:06 smithi156 ceph-mon[129245]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:06.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:06 smithi031 ceph-mon[95504]: Deploying daemon osd.10 on smithi039 2023-07-29T16:11:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:06 smithi031 ceph-mon[95504]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:08.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:07 smithi039 ceph-mon[128858]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:08.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:07 smithi156 ceph-mon[129245]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:08.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:07 smithi031 ceph-mon[95504]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:10.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:10 smithi039 ceph-mon[128858]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:10.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:10.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:10.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:10.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:10 smithi156 ceph-mon[129245]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:10.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:10.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:10.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:10.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:10 smithi031 ceph-mon[95504]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:10.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:10.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:10.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:12.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:12 smithi039 ceph-mon[128858]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:12.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:12 smithi156 ceph-mon[129245]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:12.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.606+0000 7fa1d0ff9700 1 -- 172.21.15.39:0/4047420052 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fa1d4061190 con 0x7fa1bc0a0790 2023-07-29T16:11:12.603 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 10 on host 'smithi039' 2023-07-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1bc0a0790 msgr2=0x7fa1bc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1bc0a0790 0x7fa1bc0a2c40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400b5c0 tx=0x7fa1c4005fd0 comp rx=0 tx=0).stop 2023-07-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 msgr2=0x7fa1d419d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:12.607 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d419d710 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc007ee0 tx=0x7fa1cc00d3b0 comp rx=0 tx=0).stop 2023-07-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 shutdown_connections 2023-07-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1bc0a0790 0x7fa1bc0a2c40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1d4100950 0x7fa1d419d710 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1d40ff520 0x7fa1d4196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 --2- 172.21.15.39:0/4047420052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1d40ffed0 0x7fa1d4196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.610+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 >> 172.21.15.39:0/4047420052 conn(0x7fa1d40fa0a0 msgr2=0x7fa1d40fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.611+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 shutdown_connections 2023-07-29T16:11:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2023-07-29T16:11:12.611+0000 7fa1d9e48700 1 -- 172.21.15.39:0/4047420052 wait complete. 2023-07-29T16:11:12.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:12 smithi031 ceph-mon[95504]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:13.072 DEBUG:teuthology.orchestra.run.smithi039:osd.10> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.10.service 2023-07-29T16:11:13.074 INFO:tasks.cephadm:Deploying osd.11 on smithi156 with /dev/vg_nvme/lv_1... 2023-07-29T16:11:13.074 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-29T16:11:13.107 INFO:journalctl@ceph.osd.10.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:11:13.636 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:11:13.841 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.842 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:13 smithi156 ceph-mon[129245]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:13.891 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.891 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.891 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.892 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.892 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:13 smithi039 ceph-mon[128858]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:13.892 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:11:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:11:13.661+0000 7f3d837f3640 -1 osd.10 0 log_to_monitors true 2023-07-29T16:11:13.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:13.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:13 smithi031 ceph-mon[95504]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:14.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:14 smithi031 ceph-mon[95504]: from='osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-29T16:11:14.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:14 smithi031 ceph-mon[95504]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-29T16:11:15.014 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:14 smithi039 ceph-mon[128858]: from='osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-29T16:11:15.014 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:14 smithi039 ceph-mon[128858]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-29T16:11:15.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:14 smithi156 ceph-mon[129245]: from='osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-29T16:11:15.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:14 smithi156 ceph-mon[129245]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-29T16:11:15.100 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-29T16:11:15.100 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-29T16:11:15.100 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2023-07-29T16:11:15.101 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2023-07-29T16:11:15.101 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00829128 s, 1.3 GB/s 2023-07-29T16:11:15.101 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2023-07-29T16:11:15.581 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2023-07-29T16:11:15.683 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:11:15.670+0000 7f3d7866a700 -1 osd.10 0 waiting for initial osdmap 2023-07-29T16:11:15.684 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-29T16:11:15.684 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: osdmap e71: 11 total, 10 up, 11 in 2023-07-29T16:11:15.684 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:15.684 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:11:15.684 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:11:15.685 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:15.685 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi039 2023-07-29T16:11:15.685 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.685 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.685 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.685 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 4292M 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:15.686 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-29T16:11:15.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: osdmap e71: 11 total, 10 up, 11 in 2023-07-29T16:11:15.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:15.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:11:15.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:11:15.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:15.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi039 2023-07-29T16:11:15.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 4292M 2023-07-29T16:11:15.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:15.828 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:15.828 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-29T16:11:15.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: osdmap e71: 11 total, 10 up, 11 in 2023-07-29T16:11:15.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:15.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:11:15.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi039 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:15.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 4292M 2023-07-29T16:11:15.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:15.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:15.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:15.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:16.023 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:11:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:11:15.686+0000 7f3d7345d700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:11:16.138 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:11:16.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:11:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:11:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:11:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: osdmap e72: 11 total, 10 up, 11 in 2023-07-29T16:11:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:17.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:11:17.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:11:17.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:11:17.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: osdmap e72: 11 total, 10 up, 11 in 2023-07-29T16:11:17.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:17.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:17.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:17.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:17.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:17.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:17.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:11:17.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:11:17.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2023-07-29T16:11:17.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: osdmap e72: 11 total, 10 up, 11 in 2023-07-29T16:11:17.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:17.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:17.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:17.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:17.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:17.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:17.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.321+0000 7f9701538700 1 -- 172.21.15.156:0/3069329000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100590 msgr2=0x7f96fc1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:17.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.321+0000 7f9701538700 1 --2- 172.21.15.156:0/3069329000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100590 0x7f96fc1009e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f96f0009990 tx=0x7f96f0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:11:17.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 -- 172.21.15.156:0/3069329000 shutdown_connections 2023-07-29T16:11:17.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 --2- 172.21.15.156:0/3069329000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc100f20 0x7f96fc1077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:17.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 --2- 172.21.15.156:0/3069329000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc0ff390 0x7f96fc0ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:17.320 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 --2- 172.21.15.156:0/3069329000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100590 0x7f96fc1009e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:17.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 -- 172.21.15.156:0/3069329000 >> 172.21.15.156:0/3069329000 conn(0x7f96fc0fa9e0 msgr2=0x7f96fc0fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:17.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 -- 172.21.15.156:0/3069329000 shutdown_connections 2023-07-29T16:11:17.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 -- 172.21.15.156:0/3069329000 wait complete. 2023-07-29T16:11:17.321 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.322+0000 7f9701538700 1 Processor -- start 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 -- start start 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc0ff390 0x7f96fc069d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc100590 0x7f96fc19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 0x7f96fc19d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc19dc50 con 0x7f96fc0ff390 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc19ddc0 con 0x7f96fc100590 2023-07-29T16:11:17.323 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f9701538700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96fc19df80 con 0x7f96fc100f20 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 0x7f96fc19d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 0x7f96fc19d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40818/0 (socket says 172.21.15.156:40818) 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 -- 172.21.15.156:0/931851582 learned_addr learned my addr 172.21.15.156:0/931851582 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fa7fc700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc100590 0x7f96fc19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96faffd700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc0ff390 0x7f96fc069d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 -- 172.21.15.156:0/931851582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc100590 msgr2=0x7f96fc19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:17.324 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc100590 0x7f96fc19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 -- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc0ff390 msgr2=0x7f96fc069d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc0ff390 0x7f96fc069d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 -- 172.21.15.156:0/931851582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00095f0 con 0x7f96fc100f20 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96faffd700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc0ff390 0x7f96fc069d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.323+0000 7f96fb7fe700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 0x7f96fc19d4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96f400ada0 tx=0x7f96f400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.324+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f400f070 con 0x7f96fc100f20 2023-07-29T16:11:17.325 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.324+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96fc19e260 con 0x7f96fc100f20 2023-07-29T16:11:17.326 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.324+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96f4007b30 con 0x7f96fc100f20 2023-07-29T16:11:17.326 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.324+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f40053e0 con 0x7f96fc100f20 2023-07-29T16:11:17.326 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.324+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96fc19e760 con 0x7f96fc100f20 2023-07-29T16:11:17.326 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.325+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96fc10ab60 con 0x7f96fc100f20 2023-07-29T16:11:17.327 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.326+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f96f400c980 con 0x7f96fc100f20 2023-07-29T16:11:17.327 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.326+0000 7f96e3fff700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96e40a0790 0x7f96e40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:17.327 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.327+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f96f40ca9b0 con 0x7f96fc100f20 2023-07-29T16:11:17.327 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.327+0000 7f96faffd700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96e40a0790 0x7f96e40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:17.327 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.327+0000 7f96faffd700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96e40a0790 0x7f96e40a2c40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96ec005950 tx=0x7f96ec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:17.328 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.331+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f96f4094b10 con 0x7f96fc100f20 2023-07-29T16:11:17.521 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:17.523+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f96fc061190 con 0x7f96e40a0790 2023-07-29T16:11:17.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] boot 2023-07-29T16:11:17.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: osdmap e73: 11 total, 11 up, 11 in 2023-07-29T16:11:17.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:17.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: osdmap e74: 11 total, 11 up, 11 in 2023-07-29T16:11:17.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:17.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:11:17.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:11:17.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:11:17.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:18.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] boot 2023-07-29T16:11:18.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: osdmap e73: 11 total, 11 up, 11 in 2023-07-29T16:11:18.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: osdmap e74: 11 total, 11 up, 11 in 2023-07-29T16:11:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:11:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:11:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:11:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:18.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: osd.10 [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] boot 2023-07-29T16:11:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: osdmap e73: 11 total, 11 up, 11 in 2023-07-29T16:11:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-29T16:11:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: osdmap e74: 11 total, 11 up, 11 in 2023-07-29T16:11:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: pgmap v228: 1 pgs: 1 peering; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-29T16:11:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:11:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-29T16:11:18.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-29T16:11:18.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:19.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:18 smithi039 ceph-mon[128858]: osdmap e75: 11 total, 11 up, 11 in 2023-07-29T16:11:19.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:18 smithi156 ceph-mon[129245]: osdmap e75: 11 total, 11 up, 11 in 2023-07-29T16:11:19.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:18 smithi031 ceph-mon[95504]: osdmap e75: 11 total, 11 up, 11 in 2023-07-29T16:11:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:19 smithi039 ceph-mon[128858]: osdmap e76: 11 total, 11 up, 11 in 2023-07-29T16:11:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:19 smithi039 ceph-mon[128858]: pgmap v231: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:20.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:19 smithi156 ceph-mon[129245]: osdmap e76: 11 total, 11 up, 11 in 2023-07-29T16:11:20.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:19 smithi156 ceph-mon[129245]: pgmap v231: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:20.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:19 smithi031 ceph-mon[95504]: osdmap e76: 11 total, 11 up, 11 in 2023-07-29T16:11:20.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:19 smithi031 ceph-mon[95504]: pgmap v231: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:21.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: osdmap e77: 11 total, 11 up, 11 in 2023-07-29T16:11:21.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/1265730923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]: dispatch 2023-07-29T16:11:21.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]: dispatch 2023-07-29T16:11:21.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]': finished 2023-07-29T16:11:21.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: osdmap e78: 12 total, 11 up, 12 in 2023-07-29T16:11:21.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:21.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:20 smithi039 ceph-mon[128858]: from='client.? 172.21.15.156:0/2154482709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:11:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: osdmap e77: 11 total, 11 up, 11 in 2023-07-29T16:11:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/1265730923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]: dispatch 2023-07-29T16:11:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]: dispatch 2023-07-29T16:11:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]': finished 2023-07-29T16:11:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: osdmap e78: 12 total, 11 up, 12 in 2023-07-29T16:11:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:21.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:20 smithi156 ceph-mon[129245]: from='client.? 172.21.15.156:0/2154482709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:11:21.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: osdmap e77: 11 total, 11 up, 11 in 2023-07-29T16:11:21.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/1265730923' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]: dispatch 2023-07-29T16:11:21.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]: dispatch 2023-07-29T16:11:21.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19"}]': finished 2023-07-29T16:11:21.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: osdmap e78: 12 total, 11 up, 12 in 2023-07-29T16:11:21.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:21.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:20 smithi031 ceph-mon[95504]: from='client.? 172.21.15.156:0/2154482709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-29T16:11:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:21 smithi039 ceph-mon[128858]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:22.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:21 smithi156 ceph-mon[129245]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:22.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:21 smithi031 ceph-mon[95504]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:24.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:24 smithi156 ceph-mon[129245]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-29T16:11:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:24 smithi031 ceph-mon[95504]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-29T16:11:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:24 smithi039 ceph-mon[128858]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 104 KiB/s, 0 objects/s recovering 2023-07-29T16:11:26.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:26 smithi156 ceph-mon[129245]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2023-07-29T16:11:26.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:26 smithi031 ceph-mon[95504]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2023-07-29T16:11:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:26 smithi039 ceph-mon[128858]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 88 KiB/s, 0 objects/s recovering 2023-07-29T16:11:27.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-29T16:11:27.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:27.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:27 smithi156 ceph-mon[129245]: Deploying daemon osd.11 on smithi156 2023-07-29T16:11:27.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-29T16:11:27.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:27.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:27 smithi031 ceph-mon[95504]: Deploying daemon osd.11 on smithi156 2023-07-29T16:11:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-29T16:11:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:27.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:27 smithi039 ceph-mon[128858]: Deploying daemon osd.11 on smithi156 2023-07-29T16:11:28.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:28 smithi156 ceph-mon[129245]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-29T16:11:28.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:28 smithi031 ceph-mon[95504]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-29T16:11:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:28 smithi039 ceph-mon[128858]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-29T16:11:30.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:30 smithi156 ceph-mon[129245]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-29T16:11:30.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:30 smithi031 ceph-mon[95504]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-29T16:11:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:30 smithi039 ceph-mon[128858]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-29T16:11:31.675 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:31.676 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:31.676 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:31.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:31.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:31.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:31.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:31.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:31.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:32.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:32 smithi156 ceph-mon[129245]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-29T16:11:32.931 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:32 smithi031 ceph-mon[95504]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-29T16:11:33.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:32 smithi039 ceph-mon[128858]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-07-29T16:11:33.288 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.291+0000 7f96e3fff700 1 -- 172.21.15.156:0/931851582 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f96fc061190 con 0x7f96e40a0790 2023-07-29T16:11:33.289 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 11 on host 'smithi156' 2023-07-29T16:11:33.290 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96e40a0790 msgr2=0x7f96e40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:33.291 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96e40a0790 0x7f96e40a2c40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f96ec005950 tx=0x7f96ec0058e0 comp rx=0 tx=0).stop 2023-07-29T16:11:33.291 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 msgr2=0x7f96fc19d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:33.291 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 0x7f96fc19d4b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96f400ada0 tx=0x7f96f400c330 comp rx=0 tx=0).stop 2023-07-29T16:11:33.291 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 shutdown_connections 2023-07-29T16:11:33.291 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96e40a0790 0x7f96e40a2c40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:33.291 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96fc100f20 0x7f96fc19d4b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:33.292 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96fc0ff390 0x7f96fc069d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:33.292 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 --2- 172.21.15.156:0/931851582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96fc100590 0x7f96fc19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:33.292 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 >> 172.21.15.156:0/931851582 conn(0x7f96fc0fa9e0 msgr2=0x7f96fc0fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:33.292 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 shutdown_connections 2023-07-29T16:11:33.292 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:33.293+0000 7f9701538700 1 -- 172.21.15.156:0/931851582 wait complete. 2023-07-29T16:11:33.737 DEBUG:teuthology.orchestra.run.smithi156:osd.11> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.11.service 2023-07-29T16:11:33.739 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-29T16:11:33.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd stat -f json 2023-07-29T16:11:33.773 INFO:journalctl@ceph.osd.11.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:11:34.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.427 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:34.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:34.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-29T16:11:34.428 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:34.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-29T16:11:34.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:11:34.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:34.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:34.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:34.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-29T16:11:34.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:34.797 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:11:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:11:34.515+0000 7fc6d062a640 -1 osd.11 0 log_to_monitors true 2023-07-29T16:11:35.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:35 smithi031 ceph-mon[95504]: from='osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-29T16:11:35.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:35 smithi031 ceph-mon[95504]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-29T16:11:35.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:35 smithi039 ceph-mon[128858]: from='osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-29T16:11:35.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:35 smithi039 ceph-mon[128858]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-29T16:11:35.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:35 smithi156 ceph-mon[129245]: from='osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-29T16:11:35.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:35 smithi156 ceph-mon[129245]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-29T16:11:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 -- 172.21.15.31:0/3511606891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01003d0 msgr2=0x7f14e01007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 --2- 172.21.15.31:0/3511606891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01003d0 0x7f14e01007e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009990 tx=0x7f14d0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:11:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 -- 172.21.15.31:0/3511606891 shutdown_connections 2023-07-29T16:11:35.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 --2- 172.21.15.31:0/3511606891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14e0101f60 0x7f14e0108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 --2- 172.21.15.31:0/3511606891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01003d0 0x7f14e01007e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 --2- 172.21.15.31:0/3511606891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e01015d0 0x7f14e0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 -- 172.21.15.31:0/3511606891 >> 172.21.15.31:0/3511606891 conn(0x7f14e00fb9e0 msgr2=0x7f14e00fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 -- 172.21.15.31:0/3511606891 shutdown_connections 2023-07-29T16:11:35.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.866+0000 7f14e59af700 1 -- 172.21.15.31:0/3511606891 wait complete. 2023-07-29T16:11:35.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 Processor -- start 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 -- start start 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14e01003d0 0x7f14e0198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01015d0 0x7f14e0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e0101f60 0x7f14e019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e019fec0 con 0x7f14e01015d0 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e01a0000 con 0x7f14e01003d0 2023-07-29T16:11:35.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.868+0000 7f14e59af700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e01a0190 con 0x7f14e0101f60 2023-07-29T16:11:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14df7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e0101f60 0x7f14e019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01015d0 0x7f14e0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14df7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e0101f60 0x7f14e019f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54964/0 (socket says 172.21.15.31:54964) 2023-07-29T16:11:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14df7fe700 1 -- 172.21.15.31:0/2114617234 learned_addr learned my addr 172.21.15.31:0/2114617234 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:35.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14deffd700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14e01003d0 0x7f14e0198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:35.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 -- 172.21.15.31:0/2114617234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e0101f60 msgr2=0x7f14e019f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:35.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e0101f60 0x7f14e019f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:35.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 -- 172.21.15.31:0/2114617234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14e01003d0 msgr2=0x7f14e0198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14e01003d0 0x7f14e0198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 -- 172.21.15.31:0/2114617234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d00095f0 con 0x7f14e01015d0 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.869+0000 7f14de7fc700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01015d0 0x7f14e0198870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f14d400ea20 tx=0x7f14d400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.870+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d400cbd0 con 0x7f14e01015d0 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.870+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14d400ee60 con 0x7f14e01015d0 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.870+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4018680 con 0x7f14e01015d0 2023-07-29T16:11:35.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.870+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e01a0470 con 0x7f14e01015d0 2023-07-29T16:11:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.870+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e01a0920 con 0x7f14e01015d0 2023-07-29T16:11:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.871+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e0192aa0 con 0x7f14e01015d0 2023-07-29T16:11:35.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.873+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f14d4010430 con 0x7f14e01015d0 2023-07-29T16:11:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.874+0000 7f14e49ad700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14c80a0810 0x7f14c80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:35.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.874+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f14d4014070 con 0x7f14e01015d0 2023-07-29T16:11:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.875+0000 7f14deffd700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14c80a0810 0x7f14c80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:35.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.875+0000 7f14deffd700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14c80a0810 0x7f14c80a2cc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14d000b2e0 tx=0x7f14d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:35.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:35.878+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f14d4051720 con 0x7f14e01015d0 2023-07-29T16:11:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.090+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f14e002cc30 con 0x7f14e01015d0 2023-07-29T16:11:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.091+0000 7f14e49ad700 1 -- 172.21.15.31:0/2114617234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f14d4095700 con 0x7f14e01015d0 2023-07-29T16:11:36.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14c80a0810 msgr2=0x7f14c80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14c80a0810 0x7f14c80a2cc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14d000b2e0 tx=0x7f14d0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01015d0 msgr2=0x7f14e0198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01015d0 0x7f14e0198870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f14d400ea20 tx=0x7f14d400ed30 comp rx=0 tx=0).stop 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 shutdown_connections 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14c80a0810 0x7f14c80a2cc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14e0101f60 0x7f14e019f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14e01003d0 0x7f14e0198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 --2- 172.21.15.31:0/2114617234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14e01015d0 0x7f14e0198870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.093+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 >> 172.21.15.31:0/2114617234 conn(0x7f14e00fb9e0 msgr2=0x7f14e0106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.094+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 shutdown_connections 2023-07-29T16:11:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:36.094+0000 7f14e59af700 1 -- 172.21.15.31:0/2114617234 wait complete. 2023-07-29T16:11:36.414 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: osdmap e79: 12 total, 11 up, 12 in 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.415 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:36.416 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:36.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.417 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:36 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2114617234' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T16:11:36.457 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1690647076,"num_in_osds":12,"osd_in_since":1690647079,"num_remapped_pgs":0} 2023-07-29T16:11:36.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:36.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-29T16:11:36.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: osdmap e79: 12 total, 11 up, 12 in 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:36 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2114617234' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T16:11:36.570 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:11:36.275+0000 7fc6c54a1700 -1 osd.11 0 waiting for initial osdmap 2023-07-29T16:11:36.570 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:11:36.293+0000 7fc6bda8f700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-29T16:11:36.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:36.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-29T16:11:36.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: osdmap e79: 12 total, 11 up, 12 in 2023-07-29T16:11:36.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:11:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:36 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2114617234' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T16:11:37.458 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd stat -f json 2023-07-29T16:11:37.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: Detected new or changed devices on smithi156 2023-07-29T16:11:37.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 4292M 2023-07-29T16:11:37.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:11:37.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: osdmap e80: 12 total, 11 up, 12 in 2023-07-29T16:11:37.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: Detected new or changed devices on smithi156 2023-07-29T16:11:37.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 4292M 2023-07-29T16:11:37.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:11:37.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: osdmap e80: 12 total, 11 up, 12 in 2023-07-29T16:11:37.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: Detected new or changed devices on smithi156 2023-07-29T16:11:37.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 4292M 2023-07-29T16:11:37.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2023-07-29T16:11:37.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: osdmap e80: 12 total, 11 up, 12 in 2023-07-29T16:11:37.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:37.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:38.198 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:11:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:38 smithi156 ceph-mon[129245]: purged_snaps scrub starts 2023-07-29T16:11:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:38 smithi156 ceph-mon[129245]: purged_snaps scrub ok 2023-07-29T16:11:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:38 smithi156 ceph-mon[129245]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:38 smithi156 ceph-mon[129245]: osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] boot 2023-07-29T16:11:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:38 smithi156 ceph-mon[129245]: osdmap e81: 12 total, 12 up, 12 in 2023-07-29T16:11:38.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:38 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:38.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:38 smithi031 ceph-mon[95504]: purged_snaps scrub starts 2023-07-29T16:11:38.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:38 smithi031 ceph-mon[95504]: purged_snaps scrub ok 2023-07-29T16:11:38.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:38 smithi031 ceph-mon[95504]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:38.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:38 smithi031 ceph-mon[95504]: osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] boot 2023-07-29T16:11:38.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:38 smithi031 ceph-mon[95504]: osdmap e81: 12 total, 12 up, 12 in 2023-07-29T16:11:38.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:38 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:38.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:38 smithi039 ceph-mon[128858]: purged_snaps scrub starts 2023-07-29T16:11:38.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:38 smithi039 ceph-mon[128858]: purged_snaps scrub ok 2023-07-29T16:11:38.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:38 smithi039 ceph-mon[128858]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-29T16:11:38.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:38 smithi039 ceph-mon[128858]: osd.11 [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] boot 2023-07-29T16:11:38.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:38 smithi039 ceph-mon[128858]: osdmap e81: 12 total, 12 up, 12 in 2023-07-29T16:11:38.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-29T16:11:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 -- 172.21.15.31:0/1624617862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa374101f60 msgr2=0x7fa374108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 --2- 172.21.15.31:0/1624617862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa374101f60 0x7fa374108820 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00b0a0 tx=0x7fa36c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 -- 172.21.15.31:0/1624617862 shutdown_connections 2023-07-29T16:11:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 --2- 172.21.15.31:0/1624617862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa374101f60 0x7fa374108820 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 --2- 172.21.15.31:0/1624617862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3741003d0 0x7fa3741007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 --2- 172.21.15.31:0/1624617862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3741015d0 0x7fa374101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 -- 172.21.15.31:0/1624617862 >> 172.21.15.31:0/1624617862 conn(0x7fa3740fb9e0 msgr2=0x7fa3740fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 -- 172.21.15.31:0/1624617862 shutdown_connections 2023-07-29T16:11:39.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.564+0000 7fa379d4b700 1 -- 172.21.15.31:0/1624617862 wait complete. 2023-07-29T16:11:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 Processor -- start 2023-07-29T16:11:39.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 -- start start 2023-07-29T16:11:39.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 0x7fa374198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:39.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3741015d0 0x7fa374198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:39.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa374101f60 0x7fa37419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:39.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa37419ff90 con 0x7fa3741003d0 2023-07-29T16:11:39.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741a00d0 con 0x7fa3741015d0 2023-07-29T16:11:39.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.565+0000 7fa379d4b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3741a0260 con 0x7fa374101f60 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa3737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 0x7fa374198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa3737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 0x7fa374198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42874/0 (socket says 172.21.15.31:42874) 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa3737fe700 1 -- 172.21.15.31:0/3182230438 learned_addr learned my addr 172.21.15.31:0/3182230438 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa374101f60 0x7fa37419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa372ffd700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3741015d0 0x7fa374198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 -- 172.21.15.31:0/3182230438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3741015d0 msgr2=0x7fa374198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:39.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3741015d0 0x7fa374198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 -- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 msgr2=0x7fa374198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 0x7fa374198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 -- 172.21.15.31:0/3182230438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c009d00 con 0x7fa374101f60 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa3737fe700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 0x7fa374198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.566+0000 7fa373fff700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa374101f60 0x7fa37419f9c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00b070 tx=0x7fa36c007930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.567+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c00e040 con 0x7fa374101f60 2023-07-29T16:11:39.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.567+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa36c004c50 con 0x7fa374101f60 2023-07-29T16:11:39.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.567+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3741a04e0 con 0x7fa374101f60 2023-07-29T16:11:39.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.567+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36c0052a0 con 0x7fa374101f60 2023-07-29T16:11:39.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.567+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3741a0900 con 0x7fa374101f60 2023-07-29T16:11:39.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.568+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa374192ac0 con 0x7fa374101f60 2023-07-29T16:11:39.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.570+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa36c01d470 con 0x7fa374101f60 2023-07-29T16:11:39.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.571+0000 7fa370ff9700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa35c0a0790 0x7fa35c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:39.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.571+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa36c052080 con 0x7fa374101f60 2023-07-29T16:11:39.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.571+0000 7fa3737fe700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa35c0a0790 0x7fa35c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:39.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.572+0000 7fa3737fe700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa35c0a0790 0x7fa35c0a2c40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa364005950 tx=0x7fa364009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:39.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:39 smithi156 ceph-mon[129245]: osdmap e82: 12 total, 12 up, 12 in 2023-07-29T16:11:39.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.573+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa36c097070 con 0x7fa374101f60 2023-07-29T16:11:39.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:39 smithi031 ceph-mon[95504]: osdmap e82: 12 total, 12 up, 12 in 2023-07-29T16:11:39.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:39 smithi039 ceph-mon[128858]: osdmap e82: 12 total, 12 up, 12 in 2023-07-29T16:11:39.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.779+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa37402cc30 con 0x7fa374101f60 2023-07-29T16:11:39.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.780+0000 7fa370ff9700 1 -- 172.21.15.31:0/3182230438 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) v1 ==== 74+0+133 (secure 0 0 0) 0x7fa36c09a690 con 0x7fa374101f60 2023-07-29T16:11:39.777 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:39.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.782+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa35c0a0790 msgr2=0x7fa35c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:39.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.782+0000 7fa379d4b700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa35c0a0790 0x7fa35c0a2c40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa364005950 tx=0x7fa364009260 comp rx=0 tx=0).stop 2023-07-29T16:11:39.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.782+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa374101f60 msgr2=0x7fa37419f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:39.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.782+0000 7fa379d4b700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa374101f60 0x7fa37419f9c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00b070 tx=0x7fa36c007930 comp rx=0 tx=0).stop 2023-07-29T16:11:39.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 shutdown_connections 2023-07-29T16:11:39.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa35c0a0790 0x7fa35c0a2c40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa374101f60 0x7fa37419f9c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3741003d0 0x7fa374198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 --2- 172.21.15.31:0/3182230438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa3741015d0 0x7fa374198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 >> 172.21.15.31:0/3182230438 conn(0x7fa3740fb9e0 msgr2=0x7fa374106eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 shutdown_connections 2023-07-29T16:11:39.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:39.783+0000 7fa379d4b700 1 -- 172.21.15.31:0/3182230438 wait complete. 2023-07-29T16:11:40.280 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1690647097,"num_in_osds":12,"osd_in_since":1690647079,"num_remapped_pgs":0} 2023-07-29T16:11:40.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd dump --format=json 2023-07-29T16:11:40.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:40 smithi156 ceph-mon[129245]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:40.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:40 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3182230438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T16:11:40.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:40 smithi031 ceph-mon[95504]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:40.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:40 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3182230438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T16:11:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:40 smithi039 ceph-mon[128858]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:40 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3182230438' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-29T16:11:41.013 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:11:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 -- 172.21.15.31:0/2629730469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 msgr2=0x7f54c410a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 --2- 172.21.15.31:0/2629730469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c410a440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009990 tx=0x7f54b0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 -- 172.21.15.31:0/2629730469 shutdown_connections 2023-07-29T16:11:42.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 --2- 172.21.15.31:0/2629730469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c410a980 0x7f54c410cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 --2- 172.21.15.31:0/2629730469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54c4101630 0x7f54c4101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 --2- 172.21.15.31:0/2629730469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c410a440 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 -- 172.21.15.31:0/2629730469 >> 172.21.15.31:0/2629730469 conn(0x7f54c40763b0 msgr2=0x7f54c40767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 -- 172.21.15.31:0/2629730469 shutdown_connections 2023-07-29T16:11:42.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.415+0000 7f54cad09700 1 -- 172.21.15.31:0/2629730469 wait complete. 2023-07-29T16:11:42.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.416+0000 7f54cad09700 1 Processor -- start 2023-07-29T16:11:42.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.416+0000 7f54cad09700 1 -- start start 2023-07-29T16:11:42.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54cad09700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c4101630 0x7f54c41986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:42.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54cad09700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c4198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:42.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54cad09700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54c410a980 0x7f54c419bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54cad09700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c419c5c0 con 0x7f54c4101f40 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54cad09700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c419c730 con 0x7f54c4101630 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54cad09700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c419c8f0 con 0x7f54c410a980 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c4198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c8aa5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c4101630 0x7f54c41986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c4198c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53670/0 (socket says 172.21.15.31:53670) 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c3fff700 1 -- 172.21.15.31:0/2679151156 learned_addr learned my addr 172.21.15.31:0/2679151156 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:42.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c92a6700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54c410a980 0x7f54c419bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:42.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c3fff700 1 -- 172.21.15.31:0/2679151156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54c410a980 msgr2=0x7f54c419bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:42.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.417+0000 7f54c3fff700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54c410a980 0x7f54c419bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c3fff700 1 -- 172.21.15.31:0/2679151156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c4101630 msgr2=0x7f54c41986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c3fff700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c4101630 0x7f54c41986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c3fff700 1 -- 172.21.15.31:0/2679151156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b00095f0 con 0x7f54c4101f40 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c3fff700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c4198c00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005910 tx=0x7f54b0004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b001d070 con 0x7f54c4101f40 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c419cb70 con 0x7f54c4101f40 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54b0022470 con 0x7f54c4101f40 2023-07-29T16:11:42.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b0017650 con 0x7f54c4101f40 2023-07-29T16:11:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.418+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c41a5520 con 0x7f54c4101f40 2023-07-29T16:11:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.419+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c4107f00 con 0x7f54c4101f40 2023-07-29T16:11:42.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.421+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f54b00225e0 con 0x7f54c4101f40 2023-07-29T16:11:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.423+0000 7f54c1ffb700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54b40a0790 0x7f54b40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:42.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.423+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f54b00cc840 con 0x7f54c4101f40 2023-07-29T16:11:42.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.423+0000 7f54c8aa5700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54b40a0790 0x7f54b40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:42.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.423+0000 7f54c8aa5700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54b40a0790 0x7f54b40a2c40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009c60 tx=0x7f54b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:42.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.427+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f54b005c720 con 0x7f54c4101f40 2023-07-29T16:11:42.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:42 smithi031 ceph-mon[95504]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:42.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:42 smithi156 ceph-mon[129245]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:42.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.632+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54c41078e0 con 0x7f54c4101f40 2023-07-29T16:11:42.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.633+0000 7f54c1ffb700 1 -- 172.21.15.31:0/2679151156 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18393 (secure 0 0 0) 0x7f54b0096770 con 0x7f54c4101f40 2023-07-29T16:11:42.630 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:42.630 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:11:38.287390+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:11:42.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.635+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54b40a0790 msgr2=0x7f54b40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:42.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.635+0000 7f54cad09700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54b40a0790 0x7f54b40a2c40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f54b8009c60 tx=0x7f54b8009260 comp rx=0 tx=0).stop 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 msgr2=0x7f54c4198c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c4198c00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f54b0005910 tx=0x7f54b0004710 comp rx=0 tx=0).stop 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 shutdown_connections 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54b40a0790 0x7f54b40a2c40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54c410a980 0x7f54c419bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54c4101630 0x7f54c41986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 --2- 172.21.15.31:0/2679151156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54c4101f40 0x7f54c4198c00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:42.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 >> 172.21.15.31:0/2679151156 conn(0x7f54c40763b0 msgr2=0x7f54c40ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:42.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 shutdown_connections 2023-07-29T16:11:42.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:42.636+0000 7f54cad09700 1 -- 172.21.15.31:0/2679151156 wait complete. 2023-07-29T16:11:42.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:42 smithi039 ceph-mon[128858]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:43.092 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:11:43.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-29T16:11:43.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:43 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2679151156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:11:43.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:43 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2679151156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:11:43.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:43 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2679151156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:11:43.825 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:11:44.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:44 smithi031 ceph-mon[95504]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:44.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:44 smithi156 ceph-mon[129245]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:44.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:44 smithi039 ceph-mon[128858]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 -- 172.21.15.31:0/1063663439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 msgr2=0x7ff37810d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 --2- 172.21.15.31:0/1063663439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff37810d260 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff37000b0d0 tx=0x7ff37000b3e0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 -- 172.21.15.31:0/1063663439 shutdown_connections 2023-07-29T16:11:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 --2- 172.21.15.31:0/1063663439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff37810d260 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 --2- 172.21.15.31:0/1063663439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 0x7ff378100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 --2- 172.21.15.31:0/1063663439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff378100fe0 0x7ff378104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 -- 172.21.15.31:0/1063663439 >> 172.21.15.31:0/1063663439 conn(0x7ff3780fbea0 msgr2=0x7ff3780fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 -- 172.21.15.31:0/1063663439 shutdown_connections 2023-07-29T16:11:45.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.250+0000 7ff37f209700 1 -- 172.21.15.31:0/1063663439 wait complete. 2023-07-29T16:11:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.251+0000 7ff37f209700 1 Processor -- start 2023-07-29T16:11:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 -- start start 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 0x7ff3781966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff378100fe0 0x7ff378196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff378199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37819a5e0 con 0x7ff3781056e0 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37819a750 con 0x7ff378100640 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37f209700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37819a910 con 0x7ff378100fe0 2023-07-29T16:11:45.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37cfa5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 0x7ff3781966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37d7a6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff378199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37cfa5700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 0x7ff3781966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35620/0 (socket says 172.21.15.31:35620) 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff37cfa5700 1 -- 172.21.15.31:0/295176436 learned_addr learned my addr 172.21.15.31:0/295176436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.252+0000 7ff377fff700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff378100fe0 0x7ff378196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37d7a6700 1 -- 172.21.15.31:0/295176436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff378100fe0 msgr2=0x7ff378196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37d7a6700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff378100fe0 0x7ff378196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37d7a6700 1 -- 172.21.15.31:0/295176436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 msgr2=0x7ff3781966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37d7a6700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 0x7ff3781966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37d7a6700 1 -- 172.21.15.31:0/295176436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370009d00 con 0x7ff3781056e0 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37d7a6700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff378199fe0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff370009fd0 tx=0x7ff3700077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37000e040 con 0x7ff3781056e0 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37000bcb0 con 0x7ff3781056e0 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37819ab90 con 0x7ff3781056e0 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.253+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff370005840 con 0x7ff3781056e0 2023-07-29T16:11:45.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.254+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3781a3540 con 0x7ff3781056e0 2023-07-29T16:11:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.254+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37804ea50 con 0x7ff3781056e0 2023-07-29T16:11:45.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.257+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff370004a50 con 0x7ff3781056e0 2023-07-29T16:11:45.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.259+0000 7ff375ffb700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3680a0790 0x7ff3680a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.259+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff3700cce70 con 0x7ff3781056e0 2023-07-29T16:11:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.259+0000 7ff37cfa5700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3680a0790 0x7ff3680a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.260+0000 7ff37cfa5700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3680a0790 0x7ff3680a2c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009d20 tx=0x7ff36c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:45.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.261+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff370096da0 con 0x7ff3781056e0 2023-07-29T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.467+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff3781a36d0 con 0x7ff3781056e0 2023-07-29T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.467+0000 7ff375ffb700 1 -- 172.21.15.31:0/295176436 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff37009a3c0 con 0x7ff3781056e0 2023-07-29T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3680a0790 msgr2=0x7ff3680a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3680a0790 0x7ff3680a2c40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009d20 tx=0x7ff36c009260 comp rx=0 tx=0).stop 2023-07-29T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 msgr2=0x7ff378199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff378199fe0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff370009fd0 tx=0x7ff3700077f0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 shutdown_connections 2023-07-29T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3680a0790 0x7ff3680a2c40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff3781056e0 0x7ff378199fe0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff378100640 0x7ff3781966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 --2- 172.21.15.31:0/295176436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff378100fe0 0x7ff378196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 >> 172.21.15.31:0/295176436 conn(0x7ff3780fbea0 msgr2=0x7ff3780fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 shutdown_connections 2023-07-29T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:45.470+0000 7ff37f209700 1 -- 172.21.15.31:0/295176436 wait complete. 2023-07-29T16:11:45.907 INFO:tasks.cephadm:Adding mds.a on smithi031 2023-07-29T16:11:45.908 INFO:tasks.cephadm:Adding mds.d on smithi031 2023-07-29T16:11:45.908 INFO:tasks.cephadm:Adding mds.g on smithi031 2023-07-29T16:11:45.908 INFO:tasks.cephadm:Adding mds.j on smithi031 2023-07-29T16:11:45.908 INFO:tasks.cephadm:Adding mds.b on smithi039 2023-07-29T16:11:45.909 INFO:tasks.cephadm:Adding mds.e on smithi039 2023-07-29T16:11:45.909 INFO:tasks.cephadm:Adding mds.h on smithi039 2023-07-29T16:11:45.909 INFO:tasks.cephadm:Adding mds.k on smithi039 2023-07-29T16:11:45.909 INFO:tasks.cephadm:Adding mds.c on smithi156 2023-07-29T16:11:45.909 INFO:tasks.cephadm:Adding mds.f on smithi156 2023-07-29T16:11:45.910 INFO:tasks.cephadm:Adding mds.i on smithi156 2023-07-29T16:11:45.910 INFO:tasks.cephadm:Adding mds.l on smithi156 2023-07-29T16:11:45.910 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph orch apply mds all '12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l' 2023-07-29T16:11:46.478 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:11:46.569 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:46 smithi156 ceph-mon[129245]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:46.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:46 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/295176436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:11:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:46 smithi031 ceph-mon[95504]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:46 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/295176436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:11:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:46 smithi039 ceph-mon[128858]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:46 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/295176436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:11:47.679 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 -- 172.21.15.156:0/3340003796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 msgr2=0x7f5eb41084f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:47.680 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/3340003796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb41084f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5eac00b0a0 tx=0x7f5eac00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.680 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 -- 172.21.15.156:0/3340003796 shutdown_connections 2023-07-29T16:11:47.680 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/3340003796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb41084f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.680 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/3340003796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5eb4100e50 0x7f5eb4103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.680 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/3340003796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4103770 0x7f5eb4105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.680 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 -- 172.21.15.156:0/3340003796 >> 172.21.15.156:0/3340003796 conn(0x7f5eb40fa8f0 msgr2=0x7f5eb40fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 -- 172.21.15.156:0/3340003796 shutdown_connections 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 -- 172.21.15.156:0/3340003796 wait complete. 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.680+0000 7f5eb95b6700 1 Processor -- start 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 -- start start 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4100e50 0x7f5eb4196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5eb4103770 0x7f5eb4196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb419d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:47.681 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb419ddd0 con 0x7f5eb4100e50 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb419df10 con 0x7f5eb4103770 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb95b6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb419e0a0 con 0x7f5eb4106090 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb419d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb419d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:40944/0 (socket says 172.21.15.156:40944) 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 -- 172.21.15.156:0/2356982467 learned_addr learned my addr 172.21.15.156:0/2356982467 (peer_addr_for_me v2:172.21.15.156:0/0) 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb27fc700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5eb4103770 0x7f5eb4196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:47.682 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb2ffd700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4100e50 0x7f5eb4196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 -- 172.21.15.156:0/2356982467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5eb4103770 msgr2=0x7f5eb4196780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5eb4103770 0x7f5eb4196780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 -- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4100e50 msgr2=0x7f5eb4196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4100e50 0x7f5eb4196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 -- 172.21.15.156:0/2356982467 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac009d00 con 0x7f5eb4106090 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb2ffd700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4100e50 0x7f5eb4196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.681+0000 7f5eb37fe700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb419d800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5eac00b070 tx=0x7f5eac004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:47.683 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.682+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac00e040 con 0x7f5eb4106090 2023-07-29T16:11:47.684 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.682+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5eac009440 con 0x7f5eb4106090 2023-07-29T16:11:47.684 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.682+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb419e320 con 0x7f5eb4106090 2023-07-29T16:11:47.684 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.682+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eac004020 con 0x7f5eb4106090 2023-07-29T16:11:47.684 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.682+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb419e810 con 0x7f5eb4106090 2023-07-29T16:11:47.685 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.683+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb40fc460 con 0x7f5eb4106090 2023-07-29T16:11:47.685 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.684+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5eac01b660 con 0x7f5eb4106090 2023-07-29T16:11:47.685 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.684+0000 7f5e9bfff700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e9c0a0790 0x7f5e9c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:47.685 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.684+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5eac0cccc0 con 0x7f5eb4106090 2023-07-29T16:11:47.685 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.684+0000 7f5eb2ffd700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e9c0a0790 0x7f5e9c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:47.686 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.685+0000 7f5eb2ffd700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e9c0a0790 0x7f5e9c0a2c40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005950 tx=0x7f5ea8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:47.686 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.688+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5eac096c90 con 0x7f5eb4106090 2023-07-29T16:11:47.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:47 smithi156 ceph-mon[129245]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:47.880 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.883+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f5eb4061390 con 0x7f5e9c0a0790 2023-07-29T16:11:47.889 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.891+0000 7f5e9bfff700 1 -- 172.21.15.156:0/2356982467 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f5eb4061390 con 0x7f5e9c0a0790 2023-07-29T16:11:47.889 INFO:teuthology.orchestra.run.smithi156.stdout:Scheduled mds.all update... 2023-07-29T16:11:47.891 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e9c0a0790 msgr2=0x7f5e9c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:47.891 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e9c0a0790 0x7f5e9c0a2c40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8005950 tx=0x7f5ea8005f50 comp rx=0 tx=0).stop 2023-07-29T16:11:47.891 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 msgr2=0x7f5eb419d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb419d800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5eac00b070 tx=0x7f5eac004d00 comp rx=0 tx=0).stop 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 shutdown_connections 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e9c0a0790 0x7f5e9c0a2c40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5eb4106090 0x7f5eb419d800 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5eb4100e50 0x7f5eb4196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 --2- 172.21.15.156:0/2356982467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5eb4103770 0x7f5eb4196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:47.892 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 >> 172.21.15.156:0/2356982467 conn(0x7f5eb40fa8f0 msgr2=0x7f5eb4101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:47.893 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 shutdown_connections 2023-07-29T16:11:47.893 INFO:teuthology.orchestra.run.smithi156.stderr:2023-07-29T16:11:47.894+0000 7f5eb95b6700 1 -- 172.21.15.156:0/2356982467 wait complete. 2023-07-29T16:11:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:47 smithi039 ceph-mon[128858]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:47 smithi031 ceph-mon[95504]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:48.335 DEBUG:teuthology.orchestra.run.smithi031:mds.a> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.a.service 2023-07-29T16:11:48.340 DEBUG:teuthology.orchestra.run.smithi031:mds.d> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.d.service 2023-07-29T16:11:48.385 DEBUG:teuthology.orchestra.run.smithi031:mds.g> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.g.service 2023-07-29T16:11:48.432 DEBUG:teuthology.orchestra.run.smithi031:mds.j> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.j.service 2023-07-29T16:11:48.433 INFO:journalctl@ceph.mds.a.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:11:48.464 INFO:journalctl@ceph.mds.d.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:11:48.467 DEBUG:teuthology.orchestra.run.smithi039:mds.b> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.b.service 2023-07-29T16:11:48.470 DEBUG:teuthology.orchestra.run.smithi039:mds.e> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.e.service 2023-07-29T16:11:48.501 INFO:journalctl@ceph.mds.b.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:11:48.502 DEBUG:teuthology.orchestra.run.smithi039:mds.h> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.h.service 2023-07-29T16:11:48.510 INFO:journalctl@ceph.mds.g.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:11:48.528 INFO:journalctl@ceph.mds.e.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:11:48.529 DEBUG:teuthology.orchestra.run.smithi039:mds.k> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.k.service 2023-07-29T16:11:48.541 INFO:journalctl@ceph.mds.j.smithi031.stdout:-- Logs begin at Sat 2023-07-29 15:44:00 UTC. -- 2023-07-29T16:11:48.556 INFO:journalctl@ceph.mds.h.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:11:48.556 DEBUG:teuthology.orchestra.run.smithi156:mds.c> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.c.service 2023-07-29T16:11:48.558 DEBUG:teuthology.orchestra.run.smithi156:mds.f> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.f.service 2023-07-29T16:11:48.582 INFO:journalctl@ceph.mds.k.smithi039.stdout:-- Logs begin at Sat 2023-07-29 15:40:13 UTC. -- 2023-07-29T16:11:48.587 INFO:journalctl@ceph.mds.c.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:11:48.588 DEBUG:teuthology.orchestra.run.smithi156:mds.i> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.i.service 2023-07-29T16:11:48.614 INFO:journalctl@ceph.mds.f.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:11:48.615 DEBUG:teuthology.orchestra.run.smithi156:mds.l> sudo journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.l.service 2023-07-29T16:11:48.640 INFO:journalctl@ceph.mds.i.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:11:48.641 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-29T16:11:48.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:11:48.666 INFO:journalctl@ceph.mds.l.smithi156.stdout:-- Logs begin at Sat 2023-07-29 15:39:18 UTC. -- 2023-07-29T16:11:48.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.857+0000 7f6f461ff700 1 -- 172.21.15.31:0/544945275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40108200 msgr2=0x7f6f4010a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.857+0000 7f6f461ff700 1 --2- 172.21.15.31:0/544945275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40108200 0x7f6f4010a680 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b0a0 tx=0x7f6f3800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:11:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 -- 172.21.15.31:0/544945275 shutdown_connections 2023-07-29T16:11:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 --2- 172.21.15.31:0/544945275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40108200 0x7f6f4010a680 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 --2- 172.21.15.31:0/544945275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40074d90 0x7f6f400731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 --2- 172.21.15.31:0/544945275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 -- 172.21.15.31:0/544945275 >> 172.21.15.31:0/544945275 conn(0x7f6f400fb920 msgr2=0x7f6f400fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 -- 172.21.15.31:0/544945275 shutdown_connections 2023-07-29T16:11:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.858+0000 7f6f461ff700 1 -- 172.21.15.31:0/544945275 wait complete. 2023-07-29T16:11:48.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 Processor -- start 2023-07-29T16:11:48.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 -- start start 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40074d90 0x7f6f401988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40108200 0x7f6f4019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4019fee0 con 0x7f6f40108200 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401a0020 con 0x7f6f40074d90 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f461ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401a01b0 con 0x7f6f400737b0 2023-07-29T16:11:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f3f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f3effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40074d90 0x7f6f401988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f3f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42830/0 (socket says 172.21.15.31:42830) 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f3f7fe700 1 -- 172.21.15.31:0/3256790977 learned_addr learned my addr 172.21.15.31:0/3256790977 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.859+0000 7f6f3ffff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40108200 0x7f6f4019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3effd700 1 -- 172.21.15.31:0/3256790977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 msgr2=0x7f6f40198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3effd700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3effd700 1 -- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40108200 msgr2=0x7f6f4019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3effd700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40108200 0x7f6f4019f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3effd700 1 -- 172.21.15.31:0/3256790977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009d00 con 0x7f6f40074d90 2023-07-29T16:11:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3ffff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40108200 0x7f6f4019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3f7fe700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3effd700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40074d90 0x7f6f401988a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e9c0 tx=0x7f6f3400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34004220 con 0x7f6f40074d90 2023-07-29T16:11:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f34004380 con 0x7f6f40074d90 2023-07-29T16:11:48.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f401a0490 con 0x7f6f40074d90 2023-07-29T16:11:48.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.860+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34003be0 con 0x7f6f40074d90 2023-07-29T16:11:48.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.861+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f401a09f0 con 0x7f6f40074d90 2023-07-29T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.862+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f40106fc0 con 0x7f6f40074d90 2023-07-29T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.864+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6f34003d40 con 0x7f6f40074d90 2023-07-29T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.865+0000 7f6f3cff9700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0860 0x7f6f280a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.865+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f34014070 con 0x7f6f40074d90 2023-07-29T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.865+0000 7f6f3f7fe700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0860 0x7f6f280a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:48.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.866+0000 7f6f3f7fe700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0860 0x7f6f280a2d10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f30005950 tx=0x7f6f30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:48.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:48.867+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6f34095a40 con 0x7f6f40074d90 2023-07-29T16:11:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.075+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f4002cc30 con 0x7f6f40074d90 2023-07-29T16:11:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.077+0000 7f6f3cff9700 1 -- 172.21.15.31:0/3256790977 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18393 (secure 0 0 0) 0x7f6f3401f340 con 0x7f6f40074d90 2023-07-29T16:11:49.074 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:49.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:11:38.287390+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:11:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0860 msgr2=0x7f6f280a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0860 0x7f6f280a2d10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6f30005950 tx=0x7f6f30009260 comp rx=0 tx=0).stop 2023-07-29T16:11:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40074d90 msgr2=0x7f6f401988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40074d90 0x7f6f401988a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e9c0 tx=0x7f6f3400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 shutdown_connections 2023-07-29T16:11:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0860 0x7f6f280a2d10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40108200 0x7f6f4019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f400737b0 0x7f6f40198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 --2- 172.21.15.31:0/3256790977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40074d90 0x7f6f401988a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 >> 172.21.15.31:0/3256790977 conn(0x7f6f400fb920 msgr2=0x7f6f40104100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 shutdown_connections 2023-07-29T16:11:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.079+0000 7f6f461ff700 1 -- 172.21.15.31:0/3256790977 wait complete. 2023-07-29T16:11:49.093 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:49.094 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:49.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:49.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:49.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:48 smithi031 ceph-mon[95504]: Deploying daemon mds.l on smithi156 2023-07-29T16:11:49.096 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:11:49.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:11:49.263 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:11:49.264 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-29T16:11:49.264 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:49.264 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:49.264 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:49.264 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:49.264 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:49.265 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:49.265 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:49.265 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:49.265 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:48 smithi156 ceph-mon[129245]: Deploying daemon mds.l on smithi156 2023-07-29T16:11:49.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:11:49.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-29T16:11:49.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:49.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:48 smithi039 ceph-mon[128858]: Deploying daemon mds.l on smithi156 2023-07-29T16:11:49.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 -- 172.21.15.31:0/246075075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 msgr2=0x7fbb5010a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 --2- 172.21.15.31:0/246075075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 0x7fbb5010a660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800b0a0 tx=0x7fbb4800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 -- 172.21.15.31:0/246075075 shutdown_connections 2023-07-29T16:11:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 --2- 172.21.15.31:0/246075075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 0x7fbb5010a660 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 --2- 172.21.15.31:0/246075075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb50074d90 0x7fbb500731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 --2- 172.21.15.31:0/246075075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb500737b0 0x7fbb50073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 -- 172.21.15.31:0/246075075 >> 172.21.15.31:0/246075075 conn(0x7fbb500fb920 msgr2=0x7fbb500fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.307+0000 7fbb57fff700 1 -- 172.21.15.31:0/246075075 shutdown_connections 2023-07-29T16:11:49.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.308+0000 7fbb57fff700 1 -- 172.21.15.31:0/246075075 wait complete. 2023-07-29T16:11:49.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.308+0000 7fbb57fff700 1 Processor -- start 2023-07-29T16:11:49.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 -- start start 2023-07-29T16:11:49.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb500737b0 0x7fbb501983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 0x7fbb50198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 0x7fbb5019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5019ff70 con 0x7fbb500737b0 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb501a00b0 con 0x7fbb50074d90 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb57fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb501a0240 con 0x7fbb501081e0 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb55d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb500737b0 0x7fbb501983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb5559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 0x7fbb50198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb5659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 0x7fbb5019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb5559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 0x7fbb50198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35658/0 (socket says 172.21.15.31:35658) 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.309+0000 7fbb5559a700 1 -- 172.21.15.31:0/3384584500 learned_addr learned my addr 172.21.15.31:0/3384584500 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:49.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb5559a700 1 -- 172.21.15.31:0/3384584500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 msgr2=0x7fbb5019f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb5559a700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 0x7fbb5019f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb5559a700 1 -- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb500737b0 msgr2=0x7fbb501983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb5559a700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb500737b0 0x7fbb501983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb5559a700 1 -- 172.21.15.31:0/3384584500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb48009d00 con 0x7fbb50074d90 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb55d9b700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb500737b0 0x7fbb501983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb5559a700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 0x7fbb50198930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000e970 tx=0x7fbb4000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:49.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40009790 con 0x7fbb50074d90 2023-07-29T16:11:49.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb501a0520 con 0x7fbb50074d90 2023-07-29T16:11:49.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb4000c7e0 con 0x7fbb50074d90 2023-07-29T16:11:49.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb40003c70 con 0x7fbb50074d90 2023-07-29T16:11:49.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.310+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb501a0a30 con 0x7fbb50074d90 2023-07-29T16:11:49.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.311+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb50106890 con 0x7fbb50074d90 2023-07-29T16:11:49.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.313+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbb4000cc50 con 0x7fbb50074d90 2023-07-29T16:11:49.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.314+0000 7fbb46ffd700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb3c0a0790 0x7fbb3c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.314+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbb40014070 con 0x7fbb50074d90 2023-07-29T16:11:49.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.315+0000 7fbb55d9b700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb3c0a0790 0x7fbb3c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.315+0000 7fbb55d9b700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb3c0a0790 0x7fbb3c0a2c40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbb50075160 tx=0x7fbb4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:49.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.318+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbb400957b0 con 0x7fbb50074d90 2023-07-29T16:11:49.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.527+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb50061990 con 0x7fbb50074d90 2023-07-29T16:11:49.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.527+0000 7fbb46ffd700 1 -- 172.21.15.31:0/3384584500 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb40003dd0 con 0x7fbb50074d90 2023-07-29T16:11:49.524 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:11:49.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb3c0a0790 msgr2=0x7fbb3c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb3c0a0790 0x7fbb3c0a2c40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbb50075160 tx=0x7fbb4c009260 comp rx=0 tx=0).stop 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 msgr2=0x7fbb50198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 0x7fbb50198930 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000e970 tx=0x7fbb4000ec80 comp rx=0 tx=0).stop 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 shutdown_connections 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb3c0a0790 0x7fbb3c0a2c40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb501081e0 0x7fbb5019f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb500737b0 0x7fbb501983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 --2- 172.21.15.31:0/3384584500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb50074d90 0x7fbb50198930 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 >> 172.21.15.31:0/3384584500 conn(0x7fbb500fb920 msgr2=0x7fbb500fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 shutdown_connections 2023-07-29T16:11:49.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.530+0000 7fbb57fff700 1 -- 172.21.15.31:0/3384584500 wait complete. 2023-07-29T16:11:49.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:11:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 -- 172.21.15.31:0/3917140848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101ec0 msgr2=0x7f8958108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 --2- 172.21.15.31:0/3917140848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101ec0 0x7f8958108780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8950009f90 tx=0x7f895000b330 comp rx=0 tx=0).stop 2023-07-29T16:11:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 -- 172.21.15.31:0/3917140848 shutdown_connections 2023-07-29T16:11:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 --2- 172.21.15.31:0/3917140848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101ec0 0x7f8958108780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 --2- 172.21.15.31:0/3917140848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8958100340 0x7f8958100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 --2- 172.21.15.31:0/3917140848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101530 0x7f8958101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 -- 172.21.15.31:0/3917140848 >> 172.21.15.31:0/3917140848 conn(0x7f89580fb920 msgr2=0x7f89580fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 -- 172.21.15.31:0/3917140848 shutdown_connections 2023-07-29T16:11:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.751+0000 7f895f7ff700 1 -- 172.21.15.31:0/3917140848 wait complete. 2023-07-29T16:11:49.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 Processor -- start 2023-07-29T16:11:49.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 -- start start 2023-07-29T16:11:49.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8958100340 0x7f8958198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101530 0x7f8958198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 0x7f895819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819fed0 con 0x7f8958101530 2023-07-29T16:11:49.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89581a0010 con 0x7f8958100340 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.752+0000 7f895f7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89581a01a0 con 0x7f8958101ec0 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895cd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101530 0x7f8958198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 0x7f895819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 0x7f895819f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42874/0 (socket says 172.21.15.31:42874) 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 -- 172.21.15.31:0/2473500239 learned_addr learned my addr 172.21.15.31:0/2473500239 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895d59b700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8958100340 0x7f8958198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 -- 172.21.15.31:0/2473500239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8958100340 msgr2=0x7f8958198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8958100340 0x7f8958198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 -- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101530 msgr2=0x7f8958198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101530 0x7f8958198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 -- 172.21.15.31:0/2473500239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89480095f0 con 0x7f8958101ec0 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895cd9a700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101530 0x7f8958198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.753+0000 7f895dd9c700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 0x7f895819f900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8950007930 tx=0x7f8950004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:49.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.754+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895000e040 con 0x7f8958101ec0 2023-07-29T16:11:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.754+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8950004e80 con 0x7f8958101ec0 2023-07-29T16:11:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.754+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8950009c40 con 0x7f8958101ec0 2023-07-29T16:11:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.754+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89500052a0 con 0x7f8958101ec0 2023-07-29T16:11:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.754+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89581a0710 con 0x7f8958101ec0 2023-07-29T16:11:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.755+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895810bb60 con 0x7f8958101ec0 2023-07-29T16:11:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.757+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f895001d470 con 0x7f8958101ec0 2023-07-29T16:11:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.758+0000 7f894e7fc700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89440a0860 0x7f89440a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.758+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f89500ccf40 con 0x7f8958101ec0 2023-07-29T16:11:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.758+0000 7f895d59b700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89440a0860 0x7f89440a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.759+0000 7f895d59b700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89440a0860 0x7f89440a2d10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8958101390 tx=0x7f8954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:49.761+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8950096e70 con 0x7f8958101ec0 2023-07-29T16:11:49.882 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:11:49 smithi156 systemd[1]: Starting Ceph mds.l for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:11:50.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.015+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89581a0e10 con 0x7f8958101ec0 2023-07-29T16:11:50.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.015+0000 7f894e7fc700 1 -- 172.21.15.31:0/2473500239 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f89581a0e10 con 0x7f8958101ec0 2023-07-29T16:11:50.012 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:50.013 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-29T16:11:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89440a0860 msgr2=0x7f89440a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89440a0860 0x7f89440a2d10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8958101390 tx=0x7f8954009260 comp rx=0 tx=0).stop 2023-07-29T16:11:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 msgr2=0x7f895819f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 0x7f895819f900 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8950007930 tx=0x7f8950004c60 comp rx=0 tx=0).stop 2023-07-29T16:11:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 shutdown_connections 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89440a0860 0x7f89440a2d10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8958101ec0 0x7f895819f900 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8958100340 0x7f8958198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 --2- 172.21.15.31:0/2473500239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8958101530 0x7f8958198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 >> 172.21.15.31:0/2473500239 conn(0x7f89580fb920 msgr2=0x7f89580fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 shutdown_connections 2023-07-29T16:11:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:50.018+0000 7f895f7ff700 1 -- 172.21.15.31:0/2473500239 wait complete. 2023-07-29T16:11:50.017 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 1 2023-07-29T16:11:50.179 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:49 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3256790977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:11:50.180 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:49 smithi156 ceph-mon[129245]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:50.180 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:49 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3384584500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:11:50.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:49 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3256790977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:11:50.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:49 smithi031 ceph-mon[95504]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:50.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:49 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3384584500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:11:50.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:49 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3256790977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:11:50.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:49 smithi039 ceph-mon[128858]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:50.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:49 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3384584500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:11:50.430 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:11:50 smithi156 podman[154491]: 2023-07-29T16:11:50.430 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mds[154531]: -- 172.21.15.156:0/59498003 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d4c1bb1e0 con 0x563d4c190800 2023-07-29T16:11:50.430 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:11:50 smithi156 bash[154491]: a6000ae264c495b2314182d40bb3477e2e14d0c85a8500e23034bf3a76a28f66 2023-07-29T16:11:50.431 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-l[154507]: starting mds.l at 2023-07-29T16:11:50.819 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:11:50 smithi156 systemd[1]: Started Ceph mds.l for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:11:51.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2473500239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:51.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6832/2085573271,v1:172.21.15.156:6833/2085573271] up:boot 2023-07-29T16:11:51.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: fsmap 1 up:standby 2023-07-29T16:11:51.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-29T16:11:51.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:51.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:51.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:51.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:50 smithi031 ceph-mon[95504]: Deploying daemon mds.k on smithi039 2023-07-29T16:11:51.258 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2473500239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6832/2085573271,v1:172.21.15.156:6833/2085573271] up:boot 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: fsmap 1 up:standby 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.259 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:51.260 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:51.260 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:51.260 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:50 smithi039 ceph-mon[128858]: Deploying daemon mds.k on smithi039 2023-07-29T16:11:51.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2473500239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:51.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6832/2085573271,v1:172.21.15.156:6833/2085573271] up:boot 2023-07-29T16:11:51.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: fsmap 1 up:standby 2023-07-29T16:11:51.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-29T16:11:51.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:51.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:51.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:51.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:51.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:50 smithi156 ceph-mon[129245]: Deploying daemon mds.k on smithi039 2023-07-29T16:11:52.034 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:11:52.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:51 smithi031 ceph-mon[95504]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:52.197 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:51 smithi039 ceph-mon[128858]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:52.197 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:11:52 smithi039 systemd[1]: Starting Ceph mds.k for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:11:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 -- 172.21.15.31:0/1359100430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 msgr2=0x7fde78101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/1359100430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78101980 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde6c009990 tx=0x7fde6c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 -- 172.21.15.31:0/1359100430 shutdown_connections 2023-07-29T16:11:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/1359100430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde78101ec0 0x7fde78108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/1359100430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde78100340 0x7fde78100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/1359100430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78101980 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 -- 172.21.15.31:0/1359100430 >> 172.21.15.31:0/1359100430 conn(0x7fde780fb920 msgr2=0x7fde780fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 -- 172.21.15.31:0/1359100430 shutdown_connections 2023-07-29T16:11:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.252+0000 7fde7d7ff700 1 -- 172.21.15.31:0/1359100430 wait complete. 2023-07-29T16:11:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.253+0000 7fde7d7ff700 1 Processor -- start 2023-07-29T16:11:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.253+0000 7fde7d7ff700 1 -- start start 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde7d7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde78100340 0x7fde78074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde7d7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde7d7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde78101ec0 0x7fde78073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde7d7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78073e20 con 0x7fde78101ec0 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde7d7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78073f90 con 0x7fde78101530 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde7d7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde781a00c0 con 0x7fde78100340 2023-07-29T16:11:52.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45968/0 (socket says 172.21.15.31:45968) 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 -- 172.21.15.31:0/3805182704 learned_addr learned my addr 172.21.15.31:0/3805182704 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde777fe700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde78101ec0 0x7fde78073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde76ffd700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde78100340 0x7fde78074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 -- 172.21.15.31:0/3805182704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde78100340 msgr2=0x7fde78074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde78100340 0x7fde78074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 -- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde78101ec0 msgr2=0x7fde78073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde78101ec0 0x7fde78073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.254+0000 7fde767fc700 1 -- 172.21.15.31:0/3805182704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0095f0 con 0x7fde78101530 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.255+0000 7fde767fc700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78073140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde6c004760 tx=0x7fde6c004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.255+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c01d070 con 0x7fde78101530 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.255+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde781a0340 con 0x7fde78101530 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.255+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde6c004830 con 0x7fde78101530 2023-07-29T16:11:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.255+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6c005350 con 0x7fde78101530 2023-07-29T16:11:52.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.255+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde781a0700 con 0x7fde78101530 2023-07-29T16:11:52.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.256+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde781078a0 con 0x7fde78101530 2023-07-29T16:11:52.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.258+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fde6c005510 con 0x7fde78101530 2023-07-29T16:11:52.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.259+0000 7fde5ffff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde600a07e0 0x7fde600a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:52.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.259+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fde6c0cc890 con 0x7fde78101530 2023-07-29T16:11:52.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.259+0000 7fde76ffd700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde600a07e0 0x7fde600a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:52.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.260+0000 7fde76ffd700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde600a07e0 0x7fde600a2c90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fde78101390 tx=0x7fde68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:52.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.261+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fde6c0967c0 con 0x7fde78101530 2023-07-29T16:11:52.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:51 smithi156 ceph-mon[129245]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:52.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.513+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde7802cc30 con 0x7fde78101530 2023-07-29T16:11:52.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.514+0000 7fde5ffff700 1 -- 172.21.15.31:0/3805182704 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fde6c099de0 con 0x7fde78101530 2023-07-29T16:11:52.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:52.511 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-29T16:11:52.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde600a07e0 msgr2=0x7fde600a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:52.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde600a07e0 0x7fde600a2c90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fde78101390 tx=0x7fde68009260 comp rx=0 tx=0).stop 2023-07-29T16:11:52.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 msgr2=0x7fde78073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:52.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78073140 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde6c004760 tx=0x7fde6c004790 comp rx=0 tx=0).stop 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 shutdown_connections 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde600a07e0 0x7fde600a2c90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde78101ec0 0x7fde78073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde78100340 0x7fde78074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 --2- 172.21.15.31:0/3805182704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde78101530 0x7fde78073140 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 >> 172.21.15.31:0/3805182704 conn(0x7fde780fb920 msgr2=0x7fde781049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:52.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.516+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 shutdown_connections 2023-07-29T16:11:52.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:52.517+0000 7fde7d7ff700 1 -- 172.21.15.31:0/3805182704 wait complete. 2023-07-29T16:11:52.515 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2023-07-29T16:11:52.773 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:11:52 smithi039 podman[153244]: 2023-07-29T16:11:53.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:52 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3805182704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:53.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:52 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3805182704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:53.273 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:11:52 smithi039 ceph-mds[153284]: -- 172.21.15.39:0/3125729475 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568a23ab1e0 con 0x5568a2380c00 2023-07-29T16:11:53.273 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:11:52 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-k[153260]: starting mds.k at 2023-07-29T16:11:53.274 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:11:52 smithi039 bash[153244]: b47fa95684201a26d0e4abe001772028368ef35df90cf29e429f5ff5d2f3a158 2023-07-29T16:11:53.274 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:11:52 smithi039 systemd[1]: Started Ceph mds.k for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:11:53.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:52 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3805182704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6832/3785261631,v1:172.21.15.39:6833/3785261631] up:boot 2023-07-29T16:11:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: fsmap 2 up:standby 2023-07-29T16:11:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-29T16:11:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.086 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:54.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:54.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:54.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: Deploying daemon mds.j on smithi031 2023-07-29T16:11:54.087 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:53 smithi031 ceph-mon[95504]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6832/3785261631,v1:172.21.15.39:6833/3785261631] up:boot 2023-07-29T16:11:54.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: fsmap 2 up:standby 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:54.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:54.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: Deploying daemon mds.j on smithi031 2023-07-29T16:11:54.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:53 smithi039 ceph-mon[128858]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:54.319 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6832/3785261631,v1:172.21.15.39:6833/3785261631] up:boot 2023-07-29T16:11:54.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: fsmap 2 up:standby 2023-07-29T16:11:54.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-29T16:11:54.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:54.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:54.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:54.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:54.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: Deploying daemon mds.j on smithi031 2023-07-29T16:11:54.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:53 smithi156 ceph-mon[129245]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:54.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:11:54.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 -- 172.21.15.31:0/610075525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a0074d90 msgr2=0x7ff4a00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:54.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/610075525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a0074d90 0x7ff4a00731e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff490009990 tx=0x7ff490009ca0 comp rx=0 tx=0).stop 2023-07-29T16:11:54.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 -- 172.21.15.31:0/610075525 shutdown_connections 2023-07-29T16:11:54.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/610075525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0108200 0x7ff4a010a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:54.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/610075525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a0074d90 0x7ff4a00731e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:54.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/610075525 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a00737b0 0x7ff4a0073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:54.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 -- 172.21.15.31:0/610075525 >> 172.21.15.31:0/610075525 conn(0x7ff4a00fb920 msgr2=0x7ff4a00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:54.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 -- 172.21.15.31:0/610075525 shutdown_connections 2023-07-29T16:11:54.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.780+0000 7ff4a65ff700 1 -- 172.21.15.31:0/610075525 wait complete. 2023-07-29T16:11:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.781+0000 7ff4a65ff700 1 Processor -- start 2023-07-29T16:11:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.781+0000 7ff4a65ff700 1 -- start start 2023-07-29T16:11:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a65ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 0x7ff4a01982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a65ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0074d90 0x7ff4a01987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a65ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a0108200 0x7ff4a019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:54.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a65ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a019fe30 con 0x7ff4a00737b0 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a65ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a019ff70 con 0x7ff4a0074d90 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a65ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01a0100 con 0x7ff4a0108200 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 0x7ff4a01982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 0x7ff4a01982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49080/0 (socket says 172.21.15.31:49080) 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 -- 172.21.15.31:0/2591130512 learned_addr learned my addr 172.21.15.31:0/2591130512 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a4b9c700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a0108200 0x7ff4a019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49f7fe700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0074d90 0x7ff4a01987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:54.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 -- 172.21.15.31:0/2591130512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a0108200 msgr2=0x7ff4a019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a0108200 0x7ff4a019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 -- 172.21.15.31:0/2591130512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0074d90 msgr2=0x7ff4a01987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0074d90 0x7ff4a01987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 -- 172.21.15.31:0/2591130512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4900095f0 con 0x7ff4a00737b0 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff4a4b9c700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a0108200 0x7ff4a019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49f7fe700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0074d90 0x7ff4a01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:11:54.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.782+0000 7ff49ffff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 0x7ff4a01982b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff4900098e0 tx=0x7ff490005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.783+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49001d070 con 0x7ff4a00737b0 2023-07-29T16:11:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.783+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a01a0380 con 0x7ff4a00737b0 2023-07-29T16:11:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.783+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff490004ca0 con 0x7ff4a00737b0 2023-07-29T16:11:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.783+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4900055f0 con 0x7ff4a00737b0 2023-07-29T16:11:54.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.783+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a01a0750 con 0x7ff4a00737b0 2023-07-29T16:11:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.784+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a01068b0 con 0x7ff4a00737b0 2023-07-29T16:11:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.786+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff49000fc70 con 0x7ff4a00737b0 2023-07-29T16:11:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.787+0000 7ff49d7fa700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4880a07e0 0x7ff4880a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.787+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff4900cce70 con 0x7ff4a00737b0 2023-07-29T16:11:54.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.787+0000 7ff49f7fe700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4880a07e0 0x7ff4880a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:54.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.788+0000 7ff49f7fe700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4880a07e0 0x7ff4880a2c90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0074ab0 tx=0x7ff494009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:54.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:54.789+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff490096da0 con 0x7ff4a00737b0 2023-07-29T16:11:55.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.059+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4a002d0c0 con 0x7ff4a00737b0 2023-07-29T16:11:55.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.060+0000 7ff49d7fa700 1 -- 172.21.15.31:0/2591130512 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2017 (secure 0 0 0) 0x7ff49002f800 con 0x7ff4a00737b0 2023-07-29T16:11:55.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:55.058 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-29T16:11:55.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4880a07e0 msgr2=0x7ff4880a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4880a07e0 0x7ff4880a2c90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0074ab0 tx=0x7ff494009260 comp rx=0 tx=0).stop 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 msgr2=0x7ff4a01982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 0x7ff4a01982b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff4900098e0 tx=0x7ff490005ef0 comp rx=0 tx=0).stop 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 shutdown_connections 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4880a07e0 0x7ff4880a2c90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4a0108200 0x7ff4a019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:55.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff4a00737b0 0x7ff4a01982b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:55.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 --2- 172.21.15.31:0/2591130512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a0074d90 0x7ff4a01987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:55.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 >> 172.21.15.31:0/2591130512 conn(0x7ff4a00fb920 msgr2=0x7ff4a00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:55.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 shutdown_connections 2023-07-29T16:11:55.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:55.063+0000 7ff4a65ff700 1 -- 172.21.15.31:0/2591130512 wait complete. 2023-07-29T16:11:55.062 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2023-07-29T16:11:55.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:55 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2591130512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:55.182 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:11:54 smithi031 systemd[1]: Starting Ceph mds.j for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:11:55.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:55 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2591130512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:55.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:55 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2591130512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:55.665 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:11:55 smithi031 podman[129373]: 2023-07-29T16:11:55.665 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:11:55 smithi031 bash[129373]: ab1451c0064e53adc34a7288f1dbef6513ffe880b6e93603b77c26cb093a464b 2023-07-29T16:11:55.665 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:11:55 smithi031 ceph-mds[129413]: -- 172.21.15.31:0/1198709534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fc60d311e0 con 0x55fc60d06c00 2023-07-29T16:11:55.665 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:11:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-j[129389]: starting mds.j at 2023-07-29T16:11:55.931 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:11:55 smithi031 systemd[1]: Started Ceph mds.j for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:11:56.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:56.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:56.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:56.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:56.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:56.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:56.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:11:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:11:57.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:11:57.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:57 smithi031 ceph-mon[95504]: Deploying daemon mds.d on smithi031 2023-07-29T16:11:57.249 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:57 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.31:6834/2459121100,v1:172.21.15.31:6835/2459121100] up:boot 2023-07-29T16:11:57.249 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:57 smithi031 ceph-mon[95504]: fsmap 3 up:standby 2023-07-29T16:11:57.249 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-29T16:11:57.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/535511011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb64074d90 msgr2=0x7fdb640731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:57.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/535511011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb64074d90 0x7fdb640731e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdb60009990 tx=0x7fdb60009ca0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/535511011 shutdown_connections 2023-07-29T16:11:57.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/535511011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb6410c910 0x7fdb6410ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/535511011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb64074d90 0x7fdb640731e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/535511011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb640737b0 0x7fdb64073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/535511011 >> 172.21.15.31:0/535511011 conn(0x7fdb640fb920 msgr2=0x7fdb640fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:57.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.321+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/535511011 shutdown_connections 2023-07-29T16:11:57.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.322+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/535511011 wait complete. 2023-07-29T16:11:57.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 Processor -- start 2023-07-29T16:11:57.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 -- start start 2023-07-29T16:11:57.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb640737b0 0x7fdb6419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:57.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 0x7fdb6419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6410c910 0x7fdb641a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb641a4200 con 0x7fdb6410c910 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb641a4340 con 0x7fdb64074d90 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.323+0000 7fdb6b9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb641a44d0 con 0x7fdb640737b0 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb69f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6410c910 0x7fdb641a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb68f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 0x7fdb6419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb69f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6410c910 0x7fdb641a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49122/0 (socket says 172.21.15.31:49122) 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb69f9c700 1 -- 172.21.15.31:0/3637803891 learned_addr learned my addr 172.21.15.31:0/3637803891 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb68f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 0x7fdb6419cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46028/0 (socket says 172.21.15.31:46028) 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb640737b0 0x7fdb6419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 -- 172.21.15.31:0/3637803891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 msgr2=0x7fdb6419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 0x7fdb6419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 -- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6410c910 msgr2=0x7fdb641a3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6410c910 0x7fdb641a3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 -- 172.21.15.31:0/3637803891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb600095f0 con 0x7fdb640737b0 2023-07-29T16:11:57.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb68f9a700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 0x7fdb6419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:57.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.324+0000 7fdb6979b700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb640737b0 0x7fdb6419c680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb600098e0 tx=0x7fdb60004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:57.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.325+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb6001d070 con 0x7fdb640737b0 2023-07-29T16:11:57.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.325+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb600054e0 con 0x7fdb640737b0 2023-07-29T16:11:57.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.325+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb60004bd0 con 0x7fdb640737b0 2023-07-29T16:11:57.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.325+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb641a4750 con 0x7fdb640737b0 2023-07-29T16:11:57.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.325+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb641a4b20 con 0x7fdb640737b0 2023-07-29T16:11:57.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.326+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb64102460 con 0x7fdb640737b0 2023-07-29T16:11:57.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.328+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdb6000fb90 con 0x7fdb640737b0 2023-07-29T16:11:57.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.329+0000 7fdb5a7fc700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb500a0860 0x7fdb500a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:57.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.330+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdb600cc940 con 0x7fdb640737b0 2023-07-29T16:11:57.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.330+0000 7fdb68f9a700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb500a0860 0x7fdb500a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:57.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.330+0000 7fdb68f9a700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb500a0860 0x7fdb500a2d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb64074ab0 tx=0x7fdb54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.332+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdb60096870 con 0x7fdb640737b0 2023-07-29T16:11:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:57 smithi039 ceph-mon[128858]: Deploying daemon mds.d on smithi031 2023-07-29T16:11:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:57 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.31:6834/2459121100,v1:172.21.15.31:6835/2459121100] up:boot 2023-07-29T16:11:57.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:57 smithi039 ceph-mon[128858]: fsmap 3 up:standby 2023-07-29T16:11:57.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-29T16:11:57.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:57 smithi156 ceph-mon[129245]: Deploying daemon mds.d on smithi031 2023-07-29T16:11:57.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:57 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.31:6834/2459121100,v1:172.21.15.31:6835/2459121100] up:boot 2023-07-29T16:11:57.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:57 smithi156 ceph-mon[129245]: fsmap 3 up:standby 2023-07-29T16:11:57.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-29T16:11:57.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.605+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb6402cc30 con 0x7fdb640737b0 2023-07-29T16:11:57.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.606+0000 7fdb5a7fc700 1 -- 172.21.15.31:0/3637803891 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2774 (secure 0 0 0) 0x7fdb60099e90 con 0x7fdb640737b0 2023-07-29T16:11:57.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:11:57.604 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-29T16:11:57.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb500a0860 msgr2=0x7fdb500a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:57.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb500a0860 0x7fdb500a2d10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb64074ab0 tx=0x7fdb54009260 comp rx=0 tx=0).stop 2023-07-29T16:11:57.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb640737b0 msgr2=0x7fdb6419c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:57.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb640737b0 0x7fdb6419c680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdb600098e0 tx=0x7fdb60004730 comp rx=0 tx=0).stop 2023-07-29T16:11:57.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 shutdown_connections 2023-07-29T16:11:57.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb500a0860 0x7fdb500a2d10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb6410c910 0x7fdb641a3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb640737b0 0x7fdb6419c680 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 --2- 172.21.15.31:0/3637803891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb64074d90 0x7fdb6419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:57.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 >> 172.21.15.31:0/3637803891 conn(0x7fdb640fb920 msgr2=0x7fdb640fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:57.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 shutdown_connections 2023-07-29T16:11:57.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:57.609+0000 7fdb6b9ff700 1 -- 172.21.15.31:0/3637803891 wait complete. 2023-07-29T16:11:57.609 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 4 2023-07-29T16:11:58.135 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:11:57 smithi031 systemd[1]: Starting Ceph mds.d for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:11:58.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:58 smithi031 ceph-mon[95504]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:58.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:58 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3637803891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:58 smithi039 ceph-mon[128858]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:58.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:58 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3637803891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:58 smithi156 ceph-mon[129245]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:11:58.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:58 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3637803891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:11:58.681 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:11:58 smithi031 podman[129768]: 2023-07-29T16:11:59.181 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:11:58 smithi031 bash[129768]: dd850115b7796e567be86d4a56ab15f85f8bd62aee0c2fe062924e48d28ee910 2023-07-29T16:11:59.182 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:11:58 smithi031 ceph-mds[129809]: -- 172.21.15.31:0/4248186854 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5633f2bed1e0 con 0x5633f2bc2800 2023-07-29T16:11:59.182 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:11:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-d[129784]: starting mds.d at 2023-07-29T16:11:59.182 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:11:58 smithi031 systemd[1]: Started Ceph mds.d for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:11:59.632 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:11:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 -- 172.21.15.31:0/1413186143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 msgr2=0x7f4794101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 --2- 172.21.15.31:0/1413186143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794101980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47880098f0 tx=0x7f4788009c00 comp rx=0 tx=0).stop 2023-07-29T16:11:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 -- 172.21.15.31:0/1413186143 shutdown_connections 2023-07-29T16:11:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 --2- 172.21.15.31:0/1413186143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794101ec0 0x7f4794108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 --2- 172.21.15.31:0/1413186143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794100340 0x7f4794100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 --2- 172.21.15.31:0/1413186143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794101980 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 -- 172.21.15.31:0/1413186143 >> 172.21.15.31:0/1413186143 conn(0x7f47940fb920 msgr2=0x7f47940fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:11:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 -- 172.21.15.31:0/1413186143 shutdown_connections 2023-07-29T16:11:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.856+0000 7f47991ff700 1 -- 172.21.15.31:0/1413186143 wait complete. 2023-07-29T16:11:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.857+0000 7f47991ff700 1 Processor -- start 2023-07-29T16:11:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 -- start start 2023-07-29T16:11:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794100340 0x7f4794074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 0x7f4794073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:59.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4794073e20 con 0x7f4794101ec0 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4794073f90 con 0x7f4794101530 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f47991ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47941a0170 con 0x7f4794100340 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479359e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 0x7f4794073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479359e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 0x7f4794073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54862/0 (socket says 172.21.15.31:54862) 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479359e700 1 -- 172.21.15.31:0/745215655 learned_addr learned my addr 172.21.15.31:0/745215655 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479259c700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f4792d9d700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794100340 0x7f4794074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479259c700 1 -- 172.21.15.31:0/745215655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794100340 msgr2=0x7f4794074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479259c700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794100340 0x7f4794074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.858+0000 7f479259c700 1 -- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 msgr2=0x7f4794073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f479259c700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 0x7f4794073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f479259c700 1 -- 172.21.15.31:0/745215655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784009520 con 0x7f4794101530 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f4792d9d700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794100340 0x7f4794074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:11:59.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f479359e700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 0x7f4794073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:11:59.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f479259c700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794073140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4788009230 tx=0x7f478800b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:59.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478801c070 con 0x7f4794101530 2023-07-29T16:11:59.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f478800f7f0 con 0x7f4794101530 2023-07-29T16:11:59.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478c009d00 con 0x7f4794101530 2023-07-29T16:11:59.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788003ac0 con 0x7f4794101530 2023-07-29T16:11:59.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.859+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4788009520 con 0x7f4794101530 2023-07-29T16:11:59.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.860+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4794107920 con 0x7f4794101530 2023-07-29T16:11:59.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.862+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4788020030 con 0x7f4794101530 2023-07-29T16:11:59.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.863+0000 7f477bfff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f477c0a0860 0x7f477c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:11:59.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.863+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f47880cc850 con 0x7f4794101530 2023-07-29T16:11:59.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.864+0000 7f4792d9d700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f477c0a0860 0x7f477c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:11:59.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.864+0000 7f4792d9d700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f477c0a0860 0x7f477c0a2d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4794101390 tx=0x7f4784009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:11:59.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:11:59.866+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4788096780 con 0x7f4794101530 2023-07-29T16:12:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: Deploying daemon mds.e on smithi039 2023-07-29T16:12:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.31:6836/3256727678,v1:172.21.15.31:6837/3256727678] up:boot 2023-07-29T16:12:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: fsmap 4 up:standby 2023-07-29T16:12:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-29T16:12:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:11:59 smithi156 ceph-mon[129245]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:00.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.120+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f479402cc30 con 0x7f4794101530 2023-07-29T16:12:00.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.121+0000 7f477bfff700 1 -- 172.21.15.31:0/745215655 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3531 (secure 0 0 0) 0x7f4788099da0 con 0x7f4794101530 2023-07-29T16:12:00.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:00.119 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[]} 2023-07-29T16:12:00.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f477c0a0860 msgr2=0x7f477c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:00.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f477c0a0860 0x7f477c0a2d10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4794101390 tx=0x7f4784009260 comp rx=0 tx=0).stop 2023-07-29T16:12:00.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 msgr2=0x7f4794073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:00.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794073140 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4788009230 tx=0x7f478800b730 comp rx=0 tx=0).stop 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 shutdown_connections 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f477c0a0860 0x7f477c0a2d10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4794101ec0 0x7f4794073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4794100340 0x7f4794074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 --2- 172.21.15.31:0/745215655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4794101530 0x7f4794073140 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.123+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 >> 172.21.15.31:0/745215655 conn(0x7f47940fb920 msgr2=0x7f4794104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.124+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 shutdown_connections 2023-07-29T16:12:00.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:00.124+0000 7f47991ff700 1 -- 172.21.15.31:0/745215655 wait complete. 2023-07-29T16:12:00.123 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 5 2023-07-29T16:12:00.138 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: Deploying daemon mds.e on smithi039 2023-07-29T16:12:00.139 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.31:6836/3256727678,v1:172.21.15.31:6837/3256727678] up:boot 2023-07-29T16:12:00.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: fsmap 4 up:standby 2023-07-29T16:12:00.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-29T16:12:00.140 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:11:59 smithi031 ceph-mon[95504]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:00.140 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.140 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: Deploying daemon mds.e on smithi039 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.31:6836/3256727678,v1:172.21.15.31:6837/3256727678] up:boot 2023-07-29T16:12:00.141 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: fsmap 4 up:standby 2023-07-29T16:12:00.142 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-29T16:12:00.142 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:11:59 smithi039 ceph-mon[128858]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:00.524 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:12:00 smithi039 systemd[1]: Starting Ceph mds.e for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:00.831 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:12:00 smithi039 podman[153586]: 2023-07-29T16:12:00.831 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:00 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/745215655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:01.069 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:00 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/745215655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:01.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:00 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/745215655' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:01.273 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:12:00 smithi039 ceph-mds[153626]: -- 172.21.15.39:0/2700027686 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa144b71e0 con 0x55fa1448c800 2023-07-29T16:12:01.273 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:12:00 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e[153602]: starting mds.e at 2023-07-29T16:12:01.273 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:12:00 smithi039 bash[153586]: 50f94357d46fe7e9c21ef04d2e4c8a5985dbb31aa776bf0bad25130dfa743750 2023-07-29T16:12:01.274 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:12:00 smithi039 systemd[1]: Started Ceph mds.e for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:02.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:02.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:02.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:02.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:02.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: Deploying daemon mds.f on smithi156 2023-07-29T16:12:02.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:02.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6834/2848213481,v1:172.21.15.39:6835/2848213481] up:boot 2023-07-29T16:12:02.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: fsmap 5 up:standby 2023-07-29T16:12:02.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-29T16:12:02.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: Deploying daemon mds.f on smithi156 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6834/2848213481,v1:172.21.15.39:6835/2848213481] up:boot 2023-07-29T16:12:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: fsmap 5 up:standby 2023-07-29T16:12:02.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-29T16:12:02.299 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.299 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.299 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:02.299 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:02.299 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:02.299 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:02.300 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: Deploying daemon mds.f on smithi156 2023-07-29T16:12:02.300 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:02.300 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6834/2848213481,v1:172.21.15.39:6835/2848213481] up:boot 2023-07-29T16:12:02.300 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: fsmap 5 up:standby 2023-07-29T16:12:02.300 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-29T16:12:02.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.361+0000 7f3d953ff700 1 -- 172.21.15.31:0/4210247317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90101ec0 msgr2=0x7f3d90108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.361+0000 7f3d953ff700 1 --2- 172.21.15.31:0/4210247317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90101ec0 0x7f3d90108780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d8800b0a0 tx=0x7f3d8800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 -- 172.21.15.31:0/4210247317 shutdown_connections 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 --2- 172.21.15.31:0/4210247317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90101ec0 0x7f3d90108780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 --2- 172.21.15.31:0/4210247317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90100340 0x7f3d90100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 --2- 172.21.15.31:0/4210247317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101530 0x7f3d90101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 -- 172.21.15.31:0/4210247317 >> 172.21.15.31:0/4210247317 conn(0x7f3d900fb920 msgr2=0x7f3d900fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:02.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 -- 172.21.15.31:0/4210247317 shutdown_connections 2023-07-29T16:12:02.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.362+0000 7f3d953ff700 1 -- 172.21.15.31:0/4210247317 wait complete. 2023-07-29T16:12:02.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 Processor -- start 2023-07-29T16:12:02.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 -- start start 2023-07-29T16:12:02.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 0x7f3d90074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:02.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 0x7f3d90073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101ec0 0x7f3d90073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90073e20 con 0x7f3d90100340 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d90073f90 con 0x7f3d90101ec0 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.363+0000 7f3d953ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901a01d0 con 0x7f3d90101530 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 0x7f3d90074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 0x7f3d90073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:02.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 0x7f3d90074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54878/0 (socket says 172.21.15.31:54878) 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 0x7f3d90073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37276/0 (socket says 172.21.15.31:37276) 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8effd700 1 -- 172.21.15.31:0/118543379 learned_addr learned my addr 172.21.15.31:0/118543379 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8f7fe700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101ec0 0x7f3d90073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 -- 172.21.15.31:0/118543379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101ec0 msgr2=0x7f3d90073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101ec0 0x7f3d90073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 -- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 msgr2=0x7f3d90074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:02.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 0x7f3d90074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 -- 172.21.15.31:0/118543379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d88009d00 con 0x7f3d90101530 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8f7fe700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101ec0 0x7f3d90073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8effd700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 0x7f3d90074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d8e7fc700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 0x7f3d90073140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400e9f0 tx=0x7f3d8400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8400eee0 con 0x7f3d90101530 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.364+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d901a04b0 con 0x7f3d90101530 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.365+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d8400cbd0 con 0x7f3d90101530 2023-07-29T16:12:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.365+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84018680 con 0x7f3d90101530 2023-07-29T16:12:02.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.365+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d901a0a10 con 0x7f3d90101530 2023-07-29T16:12:02.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.365+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d901077d0 con 0x7f3d90101530 2023-07-29T16:12:02.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.368+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3d8400cd40 con 0x7f3d90101530 2023-07-29T16:12:02.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.369+0000 7f3d77fff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d780a07e0 0x7f3d780a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:02.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.369+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3d84014070 con 0x7f3d90101530 2023-07-29T16:12:02.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.369+0000 7f3d8effd700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d780a07e0 0x7f3d780a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:02.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.370+0000 7f3d8effd700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d780a07e0 0x7f3d780a2c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3d80005950 tx=0x7f3d80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:02.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.375+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3d84095ad0 con 0x7f3d90101530 2023-07-29T16:12:02.570 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:12:02 smithi156 systemd[1]: Starting Ceph mds.f for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:02.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.627+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d9002cc30 con 0x7f3d90101530 2023-07-29T16:12:02.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.627+0000 7f3d77fff700 1 -- 172.21.15.31:0/118543379 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4288 (secure 0 0 0) 0x7f3d840990f0 con 0x7f3d90101530 2023-07-29T16:12:02.626 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:02.626 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-29T16:12:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d780a07e0 msgr2=0x7f3d780a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d780a07e0 0x7f3d780a2c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3d80005950 tx=0x7f3d80009260 comp rx=0 tx=0).stop 2023-07-29T16:12:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 msgr2=0x7f3d90073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 0x7f3d90073140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400e9f0 tx=0x7f3d8400ed00 comp rx=0 tx=0).stop 2023-07-29T16:12:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 shutdown_connections 2023-07-29T16:12:02.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d780a07e0 0x7f3d780a2c90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d90101ec0 0x7f3d90073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d90100340 0x7f3d90074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 --2- 172.21.15.31:0/118543379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d90101530 0x7f3d90073140 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 >> 172.21.15.31:0/118543379 conn(0x7f3d900fb920 msgr2=0x7f3d90104910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 shutdown_connections 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:02.630+0000 7f3d953ff700 1 -- 172.21.15.31:0/118543379 wait complete. 2023-07-29T16:12:02.629 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 6 2023-07-29T16:12:02.935 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:12:02 smithi156 podman[154830]: 2023-07-29T16:12:02.935 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:12:02 smithi156 ceph-mds[154870]: -- 172.21.15.156:0/3616550510 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55abad8271e0 con 0x55abad7fc800 2023-07-29T16:12:02.935 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:12:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f[154846]: starting mds.f at 2023-07-29T16:12:03.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/118543379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:03.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/118543379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:03.320 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:12:02 smithi156 bash[154830]: c41de773f07e7bd0c4fd9feb5d076642188d63adafebe35a5178e55977ee8eee 2023-07-29T16:12:03.320 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:12:03 smithi156 systemd[1]: Started Ceph mds.f for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:03.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/118543379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:04.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6834/1642153321,v1:172.21.15.156:6835/1642153321] up:boot 2023-07-29T16:12:04.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: fsmap 6 up:standby 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:04.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: Deploying daemon mds.c on smithi156 2023-07-29T16:12:04.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:04 smithi039 ceph-mon[128858]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:04.312 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6834/1642153321,v1:172.21.15.156:6835/1642153321] up:boot 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: fsmap 6 up:standby 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:04.313 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:04.314 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:04.314 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: Deploying daemon mds.c on smithi156 2023-07-29T16:12:04.314 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:04 smithi156 ceph-mon[129245]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:04.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6834/1642153321,v1:172.21.15.156:6835/1642153321] up:boot 2023-07-29T16:12:04.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: fsmap 6 up:standby 2023-07-29T16:12:04.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-29T16:12:04.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:04.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:04.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:04.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:04.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: Deploying daemon mds.c on smithi156 2023-07-29T16:12:04.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:04 smithi031 ceph-mon[95504]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:04.647 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:04.843 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:12:04 smithi156 systemd[1]: Starting Ceph mds.c for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 -- 172.21.15.31:0/525572349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac100340 msgr2=0x7efeac100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:04.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 --2- 172.21.15.31:0/525572349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac100340 0x7efeac100750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efea8009990 tx=0x7efea8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 -- 172.21.15.31:0/525572349 shutdown_connections 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 --2- 172.21.15.31:0/525572349 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeac101ec0 0x7efeac108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 --2- 172.21.15.31:0/525572349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac100340 0x7efeac100750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 --2- 172.21.15.31:0/525572349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeac101530 0x7efeac101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.863+0000 7efeb47ff700 1 -- 172.21.15.31:0/525572349 >> 172.21.15.31:0/525572349 conn(0x7efeac0fb920 msgr2=0x7efeac0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.864+0000 7efeb47ff700 1 -- 172.21.15.31:0/525572349 shutdown_connections 2023-07-29T16:12:04.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.864+0000 7efeb47ff700 1 -- 172.21.15.31:0/525572349 wait complete. 2023-07-29T16:12:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 Processor -- start 2023-07-29T16:12:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 -- start start 2023-07-29T16:12:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeac100340 0x7efeac198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 0x7efeac1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeac101ec0 0x7efeac19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac19fdf0 con 0x7efeac101ec0 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac19ff30 con 0x7efeac100340 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb47ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeac1a00c0 con 0x7efeac101530 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb1d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 0x7efeac1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb2d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeac101ec0 0x7efeac19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb1d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 0x7efeac1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37284/0 (socket says 172.21.15.31:37284) 2023-07-29T16:12:04.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb1d9a700 1 -- 172.21.15.31:0/1126292778 learned_addr learned my addr 172.21.15.31:0/1126292778 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.865+0000 7efeb259b700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeac100340 0x7efeac198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb1d9a700 1 -- 172.21.15.31:0/1126292778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeac100340 msgr2=0x7efeac198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb1d9a700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeac100340 0x7efeac198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb1d9a700 1 -- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeac101ec0 msgr2=0x7efeac19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb1d9a700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeac101ec0 0x7efeac19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb1d9a700 1 -- 172.21.15.31:0/1126292778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea4009c40 con 0x7efeac101530 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb1d9a700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 0x7efeac1987b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d950 tx=0x7efe9c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:04.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c009930 con 0x7efeac101530 2023-07-29T16:12:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe9c010460 con 0x7efeac101530 2023-07-29T16:12:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe9c00f620 con 0x7efeac101530 2023-07-29T16:12:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea80095f0 con 0x7efeac101530 2023-07-29T16:12:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.866+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeac1a0720 con 0x7efeac101530 2023-07-29T16:12:04.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.867+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeac1929e0 con 0x7efeac101530 2023-07-29T16:12:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.869+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7efe9c0108d0 con 0x7efeac101530 2023-07-29T16:12:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.871+0000 7efea37fe700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe980a0860 0x7efe980a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:04.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.871+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efe9c0cb380 con 0x7efeac101530 2023-07-29T16:12:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.871+0000 7efeb259b700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe980a0860 0x7efe980a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:04.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.872+0000 7efeb259b700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe980a0860 0x7efe980a2d10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efea800b2e0 tx=0x7efea8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:04.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:04.872+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efe9c095220 con 0x7efeac101530 2023-07-29T16:12:05.093 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:12:04 smithi156 podman[155170]: 2023-07-29T16:12:05.093 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:12:05 smithi156 ceph-mds[155210]: -- 172.21.15.156:0/2836314103 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ae453cf1e0 con 0x55ae453a5000 2023-07-29T16:12:05.094 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:12:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: starting mds.c at 2023-07-29T16:12:05.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.128+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efeac02cc30 con 0x7efeac101530 2023-07-29T16:12:05.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.129+0000 7efea37fe700 1 -- 172.21.15.31:0/1126292778 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5048 (secure 0 0 0) 0x7efe9c016090 con 0x7efeac101530 2023-07-29T16:12:05.127 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:05.127 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-29T16:12:05.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe980a0860 msgr2=0x7efe980a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:05.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe980a0860 0x7efe980a2d10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efea800b2e0 tx=0x7efea8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 msgr2=0x7efeac1987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 0x7efeac1987b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d950 tx=0x7efe9c00dc60 comp rx=0 tx=0).stop 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 shutdown_connections 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efe980a0860 0x7efe980a2d10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efeac101ec0 0x7efeac19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeac100340 0x7efeac198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 --2- 172.21.15.31:0/1126292778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efeac101530 0x7efeac1987b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.131+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 >> 172.21.15.31:0/1126292778 conn(0x7efeac0fb920 msgr2=0x7efeac106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.132+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 shutdown_connections 2023-07-29T16:12:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:05.132+0000 7efeb47ff700 1 -- 172.21.15.31:0/1126292778 wait complete. 2023-07-29T16:12:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 7 2023-07-29T16:12:05.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:05 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1126292778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:05.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:05 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6836/809205544,v1:172.21.15.156:6837/809205544] up:boot 2023-07-29T16:12:05.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:05 smithi031 ceph-mon[95504]: fsmap 7 up:standby 2023-07-29T16:12:05.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:12:05.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:05 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1126292778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:05.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:05 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6836/809205544,v1:172.21.15.156:6837/809205544] up:boot 2023-07-29T16:12:05.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:05 smithi039 ceph-mon[128858]: fsmap 7 up:standby 2023-07-29T16:12:05.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:12:05.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:05 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1126292778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:05.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:05 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6836/809205544,v1:172.21.15.156:6837/809205544] up:boot 2023-07-29T16:12:05.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:05 smithi156 ceph-mon[129245]: fsmap 7 up:standby 2023-07-29T16:12:05.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:12:05.571 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:12:05 smithi156 bash[155170]: a21ec13297e651a19f17d578c41cb9d19f93677ce988092a0b250f69bed72ee2 2023-07-29T16:12:05.571 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:12:05 smithi156 systemd[1]: Started Ceph mds.c for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:06.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:06.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:06.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:06.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: Deploying daemon mds.h on smithi039 2023-07-29T16:12:06.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:06 smithi039 ceph-mon[128858]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:06.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:06.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: Deploying daemon mds.h on smithi039 2023-07-29T16:12:06.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:06 smithi156 ceph-mon[129245]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:06.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: Deploying daemon mds.h on smithi039 2023-07-29T16:12:06.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:06 smithi031 ceph-mon[95504]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:06.975 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:12:06 smithi039 systemd[1]: Starting Ceph mds.h for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:07.150 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:07.273 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:12:06 smithi039 podman[153927]: 2023-07-29T16:12:07.274 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:12:07 smithi039 ceph-mds[153967]: -- 172.21.15.39:0/2979638250 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55779f8551e0 con 0x55779f82a800 2023-07-29T16:12:07.274 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:12:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h[153943]: starting mds.h at 2023-07-29T16:12:07.274 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:12:07 smithi039 bash[153927]: 835ee9ef48d756f2197df3782c960e7b74433b1f1a827889466e7e73bdc38d7c 2023-07-29T16:12:07.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 -- 172.21.15.31:0/3440038208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101530 msgr2=0x7f4d70101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:07.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 --2- 172.21.15.31:0/3440038208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101530 0x7f4d70101980 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009990 tx=0x7f4d60009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 -- 172.21.15.31:0/3440038208 shutdown_connections 2023-07-29T16:12:07.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 --2- 172.21.15.31:0/3440038208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d70101ec0 0x7f4d70108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 --2- 172.21.15.31:0/3440038208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d70100340 0x7f4d70100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 --2- 172.21.15.31:0/3440038208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101530 0x7f4d70101980 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 -- 172.21.15.31:0/3440038208 >> 172.21.15.31:0/3440038208 conn(0x7f4d700fb920 msgr2=0x7f4d700fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:07.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 -- 172.21.15.31:0/3440038208 shutdown_connections 2023-07-29T16:12:07.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.371+0000 7f4d77dff700 1 -- 172.21.15.31:0/3440038208 wait complete. 2023-07-29T16:12:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.372+0000 7f4d77dff700 1 Processor -- start 2023-07-29T16:12:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 -- start start 2023-07-29T16:12:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d70100340 0x7f4d701960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d70101530 0x7f4d701965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 0x7f4d7019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:07.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7019dc30 con 0x7f4d70101ec0 2023-07-29T16:12:07.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7019dd70 con 0x7f4d70101530 2023-07-29T16:12:07.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.373+0000 7f4d77dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7019df00 con 0x7f4d70100340 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 0x7f4d7019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d70101530 0x7f4d701965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 0x7f4d7019d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54920/0 (socket says 172.21.15.31:54920) 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d75b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d70100340 0x7f4d701960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 -- 172.21.15.31:0/2164350177 learned_addr learned my addr 172.21.15.31:0/2164350177 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 -- 172.21.15.31:0/2164350177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d70100340 msgr2=0x7f4d701960b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d70100340 0x7f4d701960b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 -- 172.21.15.31:0/2164350177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d70101530 msgr2=0x7f4d701965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d70101530 0x7f4d701965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 -- 172.21.15.31:0/2164350177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600095f0 con 0x7f4d70101ec0 2023-07-29T16:12:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.374+0000 7f4d7639c700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 0x7f4d7019d660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4d68007ee0 tx=0x7f4d6800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.377+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6800dd20 con 0x7f4d70101ec0 2023-07-29T16:12:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.377+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7019e1e0 con 0x7f4d70101ec0 2023-07-29T16:12:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.377+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d6800f040 con 0x7f4d70101ec0 2023-07-29T16:12:07.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.377+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d680048a0 con 0x7f4d70101ec0 2023-07-29T16:12:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.377+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7019e740 con 0x7f4d70101ec0 2023-07-29T16:12:07.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.379+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70107870 con 0x7f4d70101ec0 2023-07-29T16:12:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.381+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4d68003a00 con 0x7f4d70101ec0 2023-07-29T16:12:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.382+0000 7f4d66ffd700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4d5c0a0860 0x7f4d5c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.382+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4d680cad30 con 0x7f4d70101ec0 2023-07-29T16:12:07.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.383+0000 7f4d75b9b700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4d5c0a0860 0x7f4d5c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:07.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.383+0000 7f4d75b9b700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4d5c0a0860 0x7f4d5c0a2d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4d70101390 tx=0x7f4d6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:07.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.388+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4d6804e720 con 0x7f4d70101ec0 2023-07-29T16:12:07.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.653+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d7002cc30 con 0x7f4d70101ec0 2023-07-29T16:12:07.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.654+0000 7f4d66ffd700 1 -- 172.21.15.31:0/2164350177 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5805 (secure 0 0 0) 0x7f4d68094c70 con 0x7f4d70101ec0 2023-07-29T16:12:07.652 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:07.652 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-29T16:12:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4d5c0a0860 msgr2=0x7f4d5c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4d5c0a0860 0x7f4d5c0a2d10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4d70101390 tx=0x7f4d6c009260 comp rx=0 tx=0).stop 2023-07-29T16:12:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 msgr2=0x7f4d7019d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 0x7f4d7019d660 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4d68007ee0 tx=0x7f4d6800d3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 shutdown_connections 2023-07-29T16:12:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4d5c0a0860 0x7f4d5c0a2d10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d70101ec0 0x7f4d7019d660 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d70100340 0x7f4d701960b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 --2- 172.21.15.31:0/2164350177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d70101530 0x7f4d701965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 >> 172.21.15.31:0/2164350177 conn(0x7f4d700fb920 msgr2=0x7f4d701049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 shutdown_connections 2023-07-29T16:12:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:07.656+0000 7f4d77dff700 1 -- 172.21.15.31:0/2164350177 wait complete. 2023-07-29T16:12:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2023-07-29T16:12:07.773 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:12:07 smithi039 systemd[1]: Started Ceph mds.h for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:08.663 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:08.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:08.664 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:08.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:08.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: Deploying daemon mds.i on smithi156 2023-07-29T16:12:08.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2164350177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:08.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6836/1297587845,v1:172.21.15.39:6837/1297587845] up:boot 2023-07-29T16:12:08.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: fsmap 8 up:standby 2023-07-29T16:12:08.665 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:08.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:08.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: Deploying daemon mds.i on smithi156 2023-07-29T16:12:08.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2164350177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:08.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6836/1297587845,v1:172.21.15.39:6837/1297587845] up:boot 2023-07-29T16:12:08.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: fsmap 8 up:standby 2023-07-29T16:12:08.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-29T16:12:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: Deploying daemon mds.i on smithi156 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2164350177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:08.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6836/1297587845,v1:172.21.15.39:6837/1297587845] up:boot 2023-07-29T16:12:08.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: fsmap 8 up:standby 2023-07-29T16:12:08.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-29T16:12:09.070 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:12:08 smithi156 systemd[1]: Starting Ceph mds.i for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:09.492 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:12:09 smithi156 podman[155508]: 2023-07-29T16:12:09.492 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:12:09 smithi156 ceph-mds[155548]: -- 172.21.15.156:0/1796418747 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56152abfd1e0 con 0x56152abd3000 2023-07-29T16:12:09.492 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:12:09 smithi156 bash[155508]: e6a72ed9a4c60be6ec3ab7b2171056def55565ce042c400370f50b63861a142f 2023-07-29T16:12:09.492 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:12:09 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: starting mds.i at 2023-07-29T16:12:09.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:09.820 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:12:09 smithi156 systemd[1]: Started Ceph mds.i for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 -- 172.21.15.31:0/1066438002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a81036b0 msgr2=0x7f24a8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1066438002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a81036b0 0x7f24a8105a90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f24980098f0 tx=0x7f2498009c00 comp rx=0 tx=0).stop 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 -- 172.21.15.31:0/1066438002 shutdown_connections 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1066438002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a8105fd0 0x7f24a8108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1066438002 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1066438002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a81036b0 0x7f24a8105a90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.885+0000 7f24b01ff700 1 -- 172.21.15.31:0/1066438002 >> 172.21.15.31:0/1066438002 conn(0x7f24a80fa810 msgr2=0x7f24a80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.886+0000 7f24b01ff700 1 -- 172.21.15.31:0/1066438002 shutdown_connections 2023-07-29T16:12:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.886+0000 7f24b01ff700 1 -- 172.21.15.31:0/1066438002 wait complete. 2023-07-29T16:12:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.887+0000 7f24b01ff700 1 Processor -- start 2023-07-29T16:12:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.887+0000 7f24b01ff700 1 -- start start 2023-07-29T16:12:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24b01ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24b01ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a81036b0 0x7f24a81965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24b01ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a8105fd0 0x7f24a819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24b01ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819dc00 con 0x7f24a8105fd0 2023-07-29T16:12:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24b01ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819dd40 con 0x7f24a8100d90 2023-07-29T16:12:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24b01ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a819ded0 con 0x7f24a81036b0 2023-07-29T16:12:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24adf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24ae79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a8105fd0 0x7f24a819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24adf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48432/0 (socket says 172.21.15.31:48432) 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24adf9b700 1 -- 172.21.15.31:0/1172921787 learned_addr learned my addr 172.21.15.31:0/1172921787 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24ad79a700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a81036b0 0x7f24a81965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24ad79a700 1 -- 172.21.15.31:0/1172921787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 msgr2=0x7f24a8196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24ad79a700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24ad79a700 1 -- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a8105fd0 msgr2=0x7f24a819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.888+0000 7f24ad79a700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a8105fd0 0x7f24a819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f24ad79a700 1 -- 172.21.15.31:0/1172921787 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2498009520 con 0x7f24a81036b0 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f24adf9b700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f24ad79a700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a81036b0 0x7f24a81965c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2498009230 tx=0x7f249800b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249801c070 con 0x7f24a81036b0 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f249800f820 con 0x7f24a81036b0 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a819e150 con 0x7f24a81036b0 2023-07-29T16:12:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498003a70 con 0x7f24a81036b0 2023-07-29T16:12:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.889+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a819e570 con 0x7f24a81036b0 2023-07-29T16:12:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.890+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a80fc380 con 0x7f24a81036b0 2023-07-29T16:12:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.892+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2498020030 con 0x7f24a81036b0 2023-07-29T16:12:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.893+0000 7f249effd700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24940a07e0 0x7f24940a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.893+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24980cc560 con 0x7f24a81036b0 2023-07-29T16:12:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.893+0000 7f24adf9b700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24940a07e0 0x7f24940a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.894+0000 7f24adf9b700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24940a07e0 0x7f24940a2c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f24a40104e0 tx=0x7f24a4007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:09.897+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2498096490 con 0x7f24a81036b0 2023-07-29T16:12:10.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.156+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a802cc30 con 0x7f24a81036b0 2023-07-29T16:12:10.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.157+0000 7f249effd700 1 -- 172.21.15.31:0/1172921787 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6562 (secure 0 0 0) 0x7f2498099ab0 con 0x7f24a81036b0 2023-07-29T16:12:10.155 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:10.155 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-29T16:12:10.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.159+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24940a07e0 msgr2=0x7f24940a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:10.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.159+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24940a07e0 0x7f24940a2c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f24a40104e0 tx=0x7f24a4007480 comp rx=0 tx=0).stop 2023-07-29T16:12:10.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.159+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a81036b0 msgr2=0x7f24a81965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.159+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a81036b0 0x7f24a81965c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2498009230 tx=0x7f249800b790 comp rx=0 tx=0).stop 2023-07-29T16:12:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 shutdown_connections 2023-07-29T16:12:10.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24940a07e0 0x7f24940a2c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a8105fd0 0x7f24a819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24a8100d90 0x7f24a8196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 --2- 172.21.15.31:0/1172921787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24a81036b0 0x7f24a81965c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 >> 172.21.15.31:0/1172921787 conn(0x7f24a80fa810 msgr2=0x7f24a81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 shutdown_connections 2023-07-29T16:12:10.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:10.160+0000 7f24b01ff700 1 -- 172.21.15.31:0/1172921787 wait complete. 2023-07-29T16:12:10.160 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 9 2023-07-29T16:12:10.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:10.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: Deploying daemon mds.g on smithi031 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1172921787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:10.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:boot 2023-07-29T16:12:10.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: fsmap 9 up:standby 2023-07-29T16:12:10.773 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-29T16:12:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: Deploying daemon mds.g on smithi031 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1172921787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:boot 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: fsmap 9 up:standby 2023-07-29T16:12:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-29T16:12:11.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:11.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:11.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: Deploying daemon mds.g on smithi031 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1172921787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:11.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:boot 2023-07-29T16:12:11.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: fsmap 9 up:standby 2023-07-29T16:12:11.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-29T16:12:11.932 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:12:11 smithi031 systemd[1]: Starting Ceph mds.g for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:12.178 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:12.380 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:12:12 smithi031 podman[130349]: 2023-07-29T16:12:12.380 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:12:12 smithi031 bash[130349]: e54d61e97c204ded71b21c5f939558122df78db2ea8736cf5a11a16bdc6f879d 2023-07-29T16:12:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/471029226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 msgr2=0x7f6a48100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/471029226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48100750 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6a380098f0 tx=0x7f6a38009c00 comp rx=0 tx=0).stop 2023-07-29T16:12:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/471029226 shutdown_connections 2023-07-29T16:12:12.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/471029226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a48101ec0 0x7f6a48108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/471029226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48100750 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/471029226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101530 0x7f6a48101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/471029226 >> 172.21.15.31:0/471029226 conn(0x7f6a480fb920 msgr2=0x7f6a480fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/471029226 shutdown_connections 2023-07-29T16:12:12.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.403+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/471029226 wait complete. 2023-07-29T16:12:12.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 Processor -- start 2023-07-29T16:12:12.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 -- start start 2023-07-29T16:12:12.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a48101530 0x7f6a48196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101ec0 0x7f6a4819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4819dc80 con 0x7f6a48100340 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4819ddc0 con 0x7f6a48101ec0 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a4e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4819df50 con 0x7f6a48101530 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.405+0000 7f6a47fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:12.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48196100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53816/0 (socket says 172.21.15.31:53816) 2023-07-29T16:12:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 -- 172.21.15.31:0/2785244445 learned_addr learned my addr 172.21.15.31:0/2785244445 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a477fe700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a48101530 0x7f6a48196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:12.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a4cb9c700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101ec0 0x7f6a4819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 -- 172.21.15.31:0/2785244445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a48101530 msgr2=0x7f6a48196640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a48101530 0x7f6a48196640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 -- 172.21.15.31:0/2785244445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101ec0 msgr2=0x7f6a4819d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101ec0 0x7f6a4819d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 -- 172.21.15.31:0/2785244445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38009520 con 0x7f6a48100340 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a4cb9c700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101ec0 0x7f6a4819d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a47fff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48196100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6a38005e90 tx=0x7f6a3800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3801c070 con 0x7f6a48100340 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a3800f7f0 con 0x7f6a48100340 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4819e1d0 con 0x7f6a48100340 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.406+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a38003b00 con 0x7f6a48100340 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.407+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4819e5f0 con 0x7f6a48100340 2023-07-29T16:12:12.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.408+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a48107920 con 0x7f6a48100340 2023-07-29T16:12:12.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.409+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6a38020030 con 0x7f6a48100340 2023-07-29T16:12:12.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.412+0000 7f6a457fa700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a300a07e0 0x7f6a300a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.412+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6a380cc610 con 0x7f6a48100340 2023-07-29T16:12:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.412+0000 7f6a477fe700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a300a07e0 0x7f6a300a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:12.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.412+0000 7f6a477fe700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a300a07e0 0x7f6a300a2c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009c60 tx=0x7f6a3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:12.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.417+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6a38096540 con 0x7f6a48100340 2023-07-29T16:12:12.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.679+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a4802cc30 con 0x7f6a48100340 2023-07-29T16:12:12.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.680+0000 7f6a457fa700 1 -- 172.21.15.31:0/2785244445 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7324 (secure 0 0 0) 0x7f6a38099b60 con 0x7f6a48100340 2023-07-29T16:12:12.678 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:12.678 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-29T16:12:12.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a300a07e0 msgr2=0x7f6a300a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:12.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a300a07e0 0x7f6a300a2c90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009c60 tx=0x7f6a3c009260 comp rx=0 tx=0).stop 2023-07-29T16:12:12.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 msgr2=0x7f6a48196100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:12.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48196100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6a38005e90 tx=0x7f6a3800b730 comp rx=0 tx=0).stop 2023-07-29T16:12:12.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 shutdown_connections 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a300a07e0 0x7f6a300a2c90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a48101ec0 0x7f6a4819d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a48100340 0x7f6a48196100 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 --2- 172.21.15.31:0/2785244445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a48101530 0x7f6a48196640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 >> 172.21.15.31:0/2785244445 conn(0x7f6a480fb920 msgr2=0x7f6a48104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 shutdown_connections 2023-07-29T16:12:12.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:12.682+0000 7f6a4e5ff700 1 -- 172.21.15.31:0/2785244445 wait complete. 2023-07-29T16:12:12.682 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 10 2023-07-29T16:12:12.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:12 smithi031 ceph-mon[95504]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:12.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:12 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:12.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:12 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:12.683 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:12:12 smithi031 ceph-mds[130412]: -- 172.21.15.31:0/1657902571 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567b01f71e0 con 0x5567b01cc800 2023-07-29T16:12:12.683 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:12:12 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g[130372]: starting mds.g at 2023-07-29T16:12:12.683 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:12:12 smithi031 systemd[1]: Started Ceph mds.g for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:12.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:12 smithi156 ceph-mon[129245]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:12.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:12 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:12.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:12 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:13.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:12 smithi039 ceph-mon[128858]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:13.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:12 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:13.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:12 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:13.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:13.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:13.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:13.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:13.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: Deploying daemon mds.a on smithi031 2023-07-29T16:12:13.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2785244445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:13.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] up:boot 2023-07-29T16:12:13.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: fsmap 10 up:standby 2023-07-29T16:12:13.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-29T16:12:13.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:13.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:13.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:13.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: Deploying daemon mds.a on smithi031 2023-07-29T16:12:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2785244445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] up:boot 2023-07-29T16:12:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: fsmap 10 up:standby 2023-07-29T16:12:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-29T16:12:14.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:14.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: Deploying daemon mds.a on smithi031 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2785244445' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] up:boot 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: fsmap 10 up:standby 2023-07-29T16:12:14.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-29T16:12:14.607 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:14 smithi031 ceph-mon[95504]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:14.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:14.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:14 smithi156 ceph-mon[129245]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:14.932 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:12:14 smithi031 systemd[1]: Starting Ceph mds.a for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:14.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 -- 172.21.15.31:0/186752157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 msgr2=0x7f98fc100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:14.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 --2- 172.21.15.31:0/186752157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 0x7f98fc100760 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009990 tx=0x7f98f0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:14.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 -- 172.21.15.31:0/186752157 shutdown_connections 2023-07-29T16:12:14.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 --2- 172.21.15.31:0/186752157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc101ed0 0x7f98fc108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:14.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 --2- 172.21.15.31:0/186752157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 0x7f98fc100760 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:14.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 --2- 172.21.15.31:0/186752157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101540 0x7f98fc101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:14.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 -- 172.21.15.31:0/186752157 >> 172.21.15.31:0/186752157 conn(0x7f98fc0fb950 msgr2=0x7f98fc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:14.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 -- 172.21.15.31:0/186752157 shutdown_connections 2023-07-29T16:12:14.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.937+0000 7f99031ff700 1 -- 172.21.15.31:0/186752157 wait complete. 2023-07-29T16:12:14.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 Processor -- start 2023-07-29T16:12:14.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 -- start start 2023-07-29T16:12:14.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 0x7f98fc198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:14.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc101540 0x7f98fc198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 0x7f98fc19f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc19fea0 con 0x7f98fc101ed0 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc19ffe0 con 0x7f98fc100350 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.939+0000 7f99031ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc1a0170 con 0x7f98fc101540 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 0x7f98fc19f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 0x7f98fc19f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53830/0 (socket says 172.21.15.31:53830) 2023-07-29T16:12:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 -- 172.21.15.31:0/2538600386 learned_addr learned my addr 172.21.15.31:0/2538600386 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:14.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f98fbfff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc101540 0x7f98fc198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:14.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f9900f9b700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 0x7f98fc198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 -- 172.21.15.31:0/2538600386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc101540 msgr2=0x7f98fc198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc101540 0x7f98fc198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 -- 172.21.15.31:0/2538600386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 msgr2=0x7f98fc198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 0x7f98fc198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 -- 172.21.15.31:0/2538600386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8009520 con 0x7f98fc101ed0 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f990179c700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 0x7f98fc19f8d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f98f400ef30 tx=0x7f98f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:14.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.940+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4018070 con 0x7f98fc101ed0 2023-07-29T16:12:14.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.941+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98f400f040 con 0x7f98fc101ed0 2023-07-29T16:12:14.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.941+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f00095f0 con 0x7f98fc101ed0 2023-07-29T16:12:14.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.941+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f400cd00 con 0x7f98fc101ed0 2023-07-29T16:12:14.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.941+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc1a0770 con 0x7f98fc101ed0 2023-07-29T16:12:14.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.942+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc192a30 con 0x7f98fc101ed0 2023-07-29T16:12:14.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.944+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f98f4003a00 con 0x7f98fc101ed0 2023-07-29T16:12:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.946+0000 7f98f9ffb700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98ec0a0860 0x7f98ec0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:14.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.947+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f98f40caff0 con 0x7f98fc101ed0 2023-07-29T16:12:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.947+0000 7f9900f9b700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98ec0a0860 0x7f98ec0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:14.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.947+0000 7f9900f9b700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98ec0a0860 0x7f98ec0a2d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b2e0 tx=0x7f98f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:14.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:14.949+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f98f4094f60 con 0x7f98fc101ed0 2023-07-29T16:12:15.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:14 smithi039 ceph-mon[128858]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:15.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.230+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98fc02cc30 con 0x7f98fc101ed0 2023-07-29T16:12:15.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.231+0000 7f98f9ffb700 1 -- 172.21.15.31:0/2538600386 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8082 (secure 0 0 0) 0x7f98f4098580 con 0x7f98fc101ed0 2023-07-29T16:12:15.229 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24551,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[]} 2023-07-29T16:12:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.233+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98ec0a0860 msgr2=0x7f98ec0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:15.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.233+0000 7f99031ff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98ec0a0860 0x7f98ec0a2d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b2e0 tx=0x7f98f0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.233+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 msgr2=0x7f98fc19f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.233+0000 7f99031ff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 0x7f98fc19f8d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f98f400ef30 tx=0x7f98f400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 shutdown_connections 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98ec0a0860 0x7f98ec0a2d10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98fc101ed0 0x7f98fc19f8d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98fc100350 0x7f98fc198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:15.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 --2- 172.21.15.31:0/2538600386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98fc101540 0x7f98fc198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 >> 172.21.15.31:0/2538600386 conn(0x7f98fc0fb950 msgr2=0x7f98fc106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 shutdown_connections 2023-07-29T16:12:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:15.234+0000 7f99031ff700 1 -- 172.21.15.31:0/2538600386 wait complete. 2023-07-29T16:12:15.233 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2023-07-29T16:12:15.350 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:12:15 smithi031 podman[130760]: 2023-07-29T16:12:15.351 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:12:15 smithi031 bash[130760]: 9fd0935710a6226d63e8096baa444efa33bada83fb841a85249b6d1a5352a684 2023-07-29T16:12:15.351 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mds[130821]: -- 172.21.15.31:0/3712625894 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564d5f8ff1e0 con 0x564d5f8d4c00 2023-07-29T16:12:15.682 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-a[130794]: starting mds.a at 2023-07-29T16:12:15.682 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:12:15 smithi031 systemd[1]: Started Ceph mds.a for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:15.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2538600386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:15.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:15.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:15.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:15.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:15.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:15.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:15 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:15.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2538600386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:15.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:15.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:15.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:15.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:15.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:15.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:15 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:16.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2538600386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:16.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:16.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:16.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:16.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-29T16:12:16.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-29T16:12:16.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:15 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:16 smithi156 ceph-mon[129245]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:16 smithi156 ceph-mon[129245]: Deploying daemon mds.b on smithi039 2023-07-29T16:12:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:16 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.31:6840/3775768915,v1:172.21.15.31:6841/3775768915] up:boot 2023-07-29T16:12:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:16 smithi156 ceph-mon[129245]: fsmap 11 up:standby 2023-07-29T16:12:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-29T16:12:16.821 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:16 smithi039 ceph-mon[128858]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:16.821 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:16 smithi039 ceph-mon[128858]: Deploying daemon mds.b on smithi039 2023-07-29T16:12:16.821 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:16 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.31:6840/3775768915,v1:172.21.15.31:6841/3775768915] up:boot 2023-07-29T16:12:16.821 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:16 smithi039 ceph-mon[128858]: fsmap 11 up:standby 2023-07-29T16:12:16.822 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-29T16:12:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:16 smithi031 ceph-mon[95504]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:16 smithi031 ceph-mon[95504]: Deploying daemon mds.b on smithi039 2023-07-29T16:12:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:16 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.31:6840/3775768915,v1:172.21.15.31:6841/3775768915] up:boot 2023-07-29T16:12:16.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:16 smithi031 ceph-mon[95504]: fsmap 11 up:standby 2023-07-29T16:12:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-29T16:12:17.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:17.272 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:16 smithi039 systemd[1]: Starting Ceph mds.b for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:12:17.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/539028438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 msgr2=0x7fe3b8101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:17.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/539028438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8101980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009990 tx=0x7fe3a8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/539028438 shutdown_connections 2023-07-29T16:12:17.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/539028438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 0x7fe3b8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/539028438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 0x7fe3b8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/539028438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8101980 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/539028438 >> 172.21.15.31:0/539028438 conn(0x7fe3b80fb920 msgr2=0x7fe3b80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:17.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/539028438 shutdown_connections 2023-07-29T16:12:17.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.472+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/539028438 wait complete. 2023-07-29T16:12:17.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.473+0000 7fe3bf3ff700 1 Processor -- start 2023-07-29T16:12:17.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.473+0000 7fe3bf3ff700 1 -- start start 2023-07-29T16:12:17.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bf3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 0x7fe3b8074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:17.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bf3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:17.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bf3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 0x7fe3b8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bf3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b8073e20 con 0x7fe3b8100340 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bf3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b8073f90 con 0x7fe3b8101530 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bf3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b81a00c0 con 0x7fe3b8101ec0 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 0x7fe3b8074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bc99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bc99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48504/0 (socket says 172.21.15.31:48504) 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bc99a700 1 -- 172.21.15.31:0/1694899066 learned_addr learned my addr 172.21.15.31:0/1694899066 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:17.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd99c700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 0x7fe3b8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 -- 172.21.15.31:0/1694899066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 msgr2=0x7fe3b8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 0x7fe3b8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 -- 172.21.15.31:0/1694899066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 msgr2=0x7fe3b8073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 -- 172.21.15.31:0/1694899066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b4009520 con 0x7fe3b8100340 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd99c700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 0x7fe3b8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.474+0000 7fe3bd19b700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 0x7fe3b8074b00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400e9c0 tx=0x7fe3b400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:17.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.475+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400cb70 con 0x7fe3b8100340 2023-07-29T16:12:17.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.475+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3b400ccd0 con 0x7fe3b8100340 2023-07-29T16:12:17.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.475+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a80095f0 con 0x7fe3b8100340 2023-07-29T16:12:17.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.475+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4018680 con 0x7fe3b8100340 2023-07-29T16:12:17.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.475+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b81a0720 con 0x7fe3b8100340 2023-07-29T16:12:17.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.476+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b8107890 con 0x7fe3b8100340 2023-07-29T16:12:17.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.478+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe3b4010430 con 0x7fe3b8100340 2023-07-29T16:12:17.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.479+0000 7fe3ae7fc700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3a40a07e0 0x7fe3a40a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:17.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.480+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe3b4014070 con 0x7fe3b8100340 2023-07-29T16:12:17.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.480+0000 7fe3bc99a700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3a40a07e0 0x7fe3a40a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:17.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.480+0000 7fe3bc99a700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3a40a07e0 0x7fe3a40a2c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800b2e0 tx=0x7fe3a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:17.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.484+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe3b4095a40 con 0x7fe3b8100340 2023-07-29T16:12:17.523 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:17 smithi039 podman[154267]: 2023-07-29T16:12:17.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.757+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3b802ce90 con 0x7fe3b8100340 2023-07-29T16:12:17.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.758+0000 7fe3ae7fc700 1 -- 172.21.15.31:0/1694899066 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9598 (secure 0 0 0) 0x7fe3b4099060 con 0x7fe3b8100340 2023-07-29T16:12:17.756 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:17.757 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24530,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24550,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24551,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24581,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-29T16:12:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3a40a07e0 msgr2=0x7fe3a40a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3a40a07e0 0x7fe3a40a2c90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe3a800b2e0 tx=0x7fe3a8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 msgr2=0x7fe3b8074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 0x7fe3b8074b00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400e9c0 tx=0x7fe3b400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 shutdown_connections 2023-07-29T16:12:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3a40a07e0 0x7fe3a40a2c90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3b8101ec0 0x7fe3b8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe3b8100340 0x7fe3b8074b00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 --2- 172.21.15.31:0/1694899066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3b8101530 0x7fe3b8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 >> 172.21.15.31:0/1694899066 conn(0x7fe3b80fb920 msgr2=0x7fe3b81049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 shutdown_connections 2023-07-29T16:12:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.761+0000 7fe3bf3ff700 1 -- 172.21.15.31:0/1694899066 wait complete. 2023-07-29T16:12:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2023-07-29T16:12:17.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:12:17.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 -- 172.21.15.31:0/3484359181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c102560 msgr2=0x7f6f9c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:17.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/3484359181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c102560 0x7f6f9c102970 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6f98009990 tx=0x7f6f98009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 -- 172.21.15.31:0/3484359181 shutdown_connections 2023-07-29T16:12:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/3484359181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c1040e0 0x7f6f9c10a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/3484359181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c102560 0x7f6f9c102970 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/3484359181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f9c103750 0x7f6f9c103ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 -- 172.21.15.31:0/3484359181 >> 172.21.15.31:0/3484359181 conn(0x7f6f9c0fdba0 msgr2=0x7f6f9c0fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.998+0000 7f6fa45ff700 1 -- 172.21.15.31:0/3484359181 shutdown_connections 2023-07-29T16:12:17.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:17.999+0000 7f6fa45ff700 1 -- 172.21.15.31:0/3484359181 wait complete. 2023-07-29T16:12:17.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 Processor -- start 2023-07-29T16:12:17.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 -- start start 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f9c102560 0x7f6f9c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 0x7f6f9c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c1040e0 0x7f6f9c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c19fe30 con 0x7f6f9c1040e0 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c19ff70 con 0x7f6f9c103750 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa45ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c1a0100 con 0x7f6f9c102560 2023-07-29T16:12:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa239b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f9c102560 0x7f6f9c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa1b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 0x7f6f9c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa1b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 0x7f6f9c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48520/0 (socket says 172.21.15.31:48520) 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.000+0000 7f6fa1b9a700 1 -- 172.21.15.31:0/148380745 learned_addr learned my addr 172.21.15.31:0/148380745 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa2b9c700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c1040e0 0x7f6f9c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa1b9a700 1 -- 172.21.15.31:0/148380745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f9c102560 msgr2=0x7f6f9c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa1b9a700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f9c102560 0x7f6f9c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa1b9a700 1 -- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c1040e0 msgr2=0x7f6f9c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa1b9a700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c1040e0 0x7f6f9c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa1b9a700 1 -- 172.21.15.31:0/148380745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94009c40 con 0x7f6f9c103750 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa1b9a700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 0x7f6f9c1987f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00e9a0 tx=0x7f6f8c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c009790 con 0x7f6f9c103750 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f8c00cbd0 con 0x7f6f9c103750 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c018680 con 0x7f6f9c103750 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.001+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f980095f0 con 0x7f6f9c103750 2023-07-29T16:12:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.002+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c1a0710 con 0x7f6f9c103750 2023-07-29T16:12:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.003+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c109aa0 con 0x7f6f9c103750 2023-07-29T16:12:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.005+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6f8c00cd40 con 0x7f6f9c103750 2023-07-29T16:12:18.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.006+0000 7f6f937fe700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f880a07e0 0x7f6f880a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.006+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f8c014070 con 0x7f6f9c103750 2023-07-29T16:12:18.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.006+0000 7f6fa239b700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f880a07e0 0x7f6f880a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.007+0000 7f6fa239b700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f880a07e0 0x7f6f880a2c90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800b2e0 tx=0x7f6f98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:18.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.008+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6f8c0959f0 con 0x7f6f9c103750 2023-07-29T16:12:18.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mds[154307]: -- 172.21.15.39:0/1373881827 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb6fd211e0 con 0x55fb6fcf7000 2023-07-29T16:12:18.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: starting mds.b at 2023-07-29T16:12:18.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:17 smithi039 bash[154267]: 4f70964c16455301579b551726a4d3c26c248064f6ee6df7a0258ae58ffc250c 2023-07-29T16:12:18.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:17 smithi039 systemd[1]: Started Ceph mds.b for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:12:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mon[128858]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:17 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:17 smithi156 ceph-mon[129245]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:17 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:17 smithi031 ceph-mon[95504]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:17 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:18.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.231+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f9c02cc30 con 0x7f6f9c103750 2023-07-29T16:12:18.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.232+0000 7f6f937fe700 1 -- 172.21.15.31:0/148380745 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18393 (secure 0 0 0) 0x7f6f8c099010 con 0x7f6f9c103750 2023-07-29T16:12:18.230 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:18.230 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":82,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:11:38.287390+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:12:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f880a07e0 msgr2=0x7f6f880a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f880a07e0 0x7f6f880a2c90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6f9800b2e0 tx=0x7f6f98005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 msgr2=0x7f6f9c1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 0x7f6f9c1987f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00e9a0 tx=0x7f6f8c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 shutdown_connections 2023-07-29T16:12:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f880a07e0 0x7f6f880a2c90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f9c1040e0 0x7f6f9c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f9c102560 0x7f6f9c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 --2- 172.21.15.31:0/148380745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f9c103750 0x7f6f9c1987f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 >> 172.21.15.31:0/148380745 conn(0x7f6f9c0fdba0 msgr2=0x7f6f9c106be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 shutdown_connections 2023-07-29T16:12:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.235+0000 7f6fa45ff700 1 -- 172.21.15.31:0/148380745 wait complete. 2023-07-29T16:12:18.252 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:12:18.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:12:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 -- 172.21.15.31:0/3863399701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 msgr2=0x7f25e8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/3863399701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e8108770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f25e000b0a0 tx=0x7f25e000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 -- 172.21.15.31:0/3863399701 shutdown_connections 2023-07-29T16:12:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/3863399701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e8108770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/3863399701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/3863399701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 0x7f25e8101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 -- 172.21.15.31:0/3863399701 >> 172.21.15.31:0/3863399701 conn(0x7f25e80fb970 msgr2=0x7f25e80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 -- 172.21.15.31:0/3863399701 shutdown_connections 2023-07-29T16:12:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.479+0000 7f25ee7ff700 1 -- 172.21.15.31:0/3863399701 wait complete. 2023-07-29T16:12:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 Processor -- start 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 -- start start 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 0x7f25e8196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e819d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e819dad0 con 0x7f25e8100330 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e819dc10 con 0x7f25e8101520 2023-07-29T16:12:18.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.481+0000 7f25ee7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e819dda0 con 0x7f25e8101eb0 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53900/0 (socket says 172.21.15.31:53900) 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 -- 172.21.15.31:0/1187693277 learned_addr learned my addr 172.21.15.31:0/1187693277 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e77fe700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 0x7f25e8196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25ecd9c700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e819d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 -- 172.21.15.31:0/1187693277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 msgr2=0x7f25e819d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e819d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 -- 172.21.15.31:0/1187693277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 msgr2=0x7f25e8196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 0x7f25e8196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 -- 172.21.15.31:0/1187693277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e0009d00 con 0x7f25e8100330 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25ecd9c700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e819d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e77fe700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 0x7f25e8196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e7fff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8195f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f25d800e970 tx=0x7f25d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.482+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8009790 con 0x7f25e8100330 2023-07-29T16:12:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.483+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e819e080 con 0x7f25e8100330 2023-07-29T16:12:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.483+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25d800c7e0 con 0x7f25e8100330 2023-07-29T16:12:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.483+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8003c70 con 0x7f25e8100330 2023-07-29T16:12:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.483+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e819e5e0 con 0x7f25e8100330 2023-07-29T16:12:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.484+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e810bac0 con 0x7f25e8100330 2023-07-29T16:12:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.486+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f25d800cc50 con 0x7f25e8100330 2023-07-29T16:12:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.487+0000 7f25e57fa700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f25d00a0810 0x7f25d00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.487+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25d8014070 con 0x7f25e8100330 2023-07-29T16:12:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.487+0000 7f25e77fe700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f25d00a0810 0x7f25d00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.488+0000 7f25e77fe700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f25d00a0810 0x7f25d00a2cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f25dc011490 tx=0x7f25dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.491+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f25d8095740 con 0x7f25e8100330 2023-07-29T16:12:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.709+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25e81071b0 con 0x7f25e8100330 2023-07-29T16:12:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.709+0000 7f25e57fa700 1 -- 172.21.15.31:0/1187693277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25d8003dd0 con 0x7f25e8100330 2023-07-29T16:12:18.708 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:12:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f25d00a0810 msgr2=0x7f25d00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f25d00a0810 0x7f25d00a2cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f25dc011490 tx=0x7f25dc009260 comp rx=0 tx=0).stop 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 msgr2=0x7f25e8195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8195f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f25d800e970 tx=0x7f25d800ec80 comp rx=0 tx=0).stop 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 shutdown_connections 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f25d00a0810 0x7f25d00a2cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25e8101eb0 0x7f25e819d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25e8100330 0x7f25e8195f50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 --2- 172.21.15.31:0/1187693277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25e8101520 0x7f25e8196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 >> 172.21.15.31:0/1187693277 conn(0x7f25e80fb970 msgr2=0x7f25e80fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 shutdown_connections 2023-07-29T16:12:18.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.712+0000 7f25ee7ff700 1 -- 172.21.15.31:0/1187693277 wait complete. 2023-07-29T16:12:18.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-29T16:12:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 -- 172.21.15.31:0/2884615984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101530 msgr2=0x7f53dc101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 --2- 172.21.15.31:0/2884615984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101530 0x7f53dc101980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009990 tx=0x7f53cc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 -- 172.21.15.31:0/2884615984 shutdown_connections 2023-07-29T16:12:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 --2- 172.21.15.31:0/2884615984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53dc101ec0 0x7f53dc108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 --2- 172.21.15.31:0/2884615984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 --2- 172.21.15.31:0/2884615984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101530 0x7f53dc101980 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 -- 172.21.15.31:0/2884615984 >> 172.21.15.31:0/2884615984 conn(0x7f53dc0fb920 msgr2=0x7f53dc0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 -- 172.21.15.31:0/2884615984 shutdown_connections 2023-07-29T16:12:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.944+0000 7f53e3bff700 1 -- 172.21.15.31:0/2884615984 wait complete. 2023-07-29T16:12:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 Processor -- start 2023-07-29T16:12:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 -- start start 2023-07-29T16:12:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc1063d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53dc101530 0x7f53dc106910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101ec0 0x7f53dc102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc103700 con 0x7f53dc101530 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc103870 con 0x7f53dc100340 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.945+0000 7f53e3bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc103a30 con 0x7f53dc101ec0 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc1063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e119a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53dc101530 0x7f53dc106910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc1063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48558/0 (socket says 172.21.15.31:48558) 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 -- 172.21.15.31:0/3423223596 learned_addr learned my addr 172.21.15.31:0/3423223596 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e219c700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101ec0 0x7f53dc102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 -- 172.21.15.31:0/3423223596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101ec0 msgr2=0x7f53dc102f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101ec0 0x7f53dc102f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 -- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53dc101530 msgr2=0x7f53dc106910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53dc101530 0x7f53dc106910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 -- 172.21.15.31:0/3423223596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53cc0095f0 con 0x7f53dc100340 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e219c700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101ec0 0x7f53dc102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.946+0000 7f53e199b700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc1063d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e970 tx=0x7f53d800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.947+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8009790 con 0x7f53dc100340 2023-07-29T16:12:18.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.947+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53d800eec0 con 0x7f53dc100340 2023-07-29T16:12:18.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.947+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53dc1a47a0 con 0x7f53dc100340 2023-07-29T16:12:18.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.947+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8003cf0 con 0x7f53dc100340 2023-07-29T16:12:18.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.947+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53dc1a4c80 con 0x7f53dc100340 2023-07-29T16:12:18.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.948+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53dc10bb00 con 0x7f53dc100340 2023-07-29T16:12:18.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.950+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f53d800ccd0 con 0x7f53dc100340 2023-07-29T16:12:18.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.951+0000 7f53d2ffd700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a07e0 0x7f53c80a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:18.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.951+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f53d8014070 con 0x7f53dc100340 2023-07-29T16:12:18.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.952+0000 7f53e119a700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a07e0 0x7f53c80a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.952+0000 7f53e119a700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a07e0 0x7f53c80a2c90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00b2e0 tx=0x7f53cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:18.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:18.953+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f53d8095a70 con 0x7f53dc100340 2023-07-29T16:12:19.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] up:boot 2023-07-29T16:12:19.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: fsmap 12 up:standby 2023-07-29T16:12:19.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-29T16:12:19.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1694899066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/148380745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:18 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1187693277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] up:boot 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: fsmap 12 up:standby 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1694899066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/148380745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:18 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1187693277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:12:19.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.174+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f53dc10bcf0 con 0x7f53dc100340 2023-07-29T16:12:19.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.174+0000 7f53d2ffd700 1 -- 172.21.15.31:0/3423223596 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f53d8099090 con 0x7f53dc100340 2023-07-29T16:12:19.172 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:19.172 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2023-07-29T16:12:19.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:12:19.173 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2023-07-29T16:12:19.173 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2023-07-29T16:12:19.173 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:12:19.173 INFO:teuthology.orchestra.run.smithi031.stdout:] 2023-07-29T16:12:19.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a07e0 msgr2=0x7f53c80a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:19.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a07e0 0x7f53c80a2c90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00b2e0 tx=0x7f53cc005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 msgr2=0x7f53dc1063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:19.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc1063d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e970 tx=0x7f53d800ec80 comp rx=0 tx=0).stop 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 shutdown_connections 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a07e0 0x7f53c80a2c90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53dc101ec0 0x7f53dc102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53dc100340 0x7f53dc1063d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 --2- 172.21.15.31:0/3423223596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53dc101530 0x7f53dc106910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 >> 172.21.15.31:0/3423223596 conn(0x7f53dc0fb920 msgr2=0x7f53dc0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:19.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 shutdown_connections 2023-07-29T16:12:19.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.177+0000 7f53e3bff700 1 -- 172.21.15.31:0/3423223596 wait complete. 2023-07-29T16:12:19.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] up:boot 2023-07-29T16:12:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: fsmap 12 up:standby 2023-07-29T16:12:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-29T16:12:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1694899066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/148380745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:18 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1187693277' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:12:19.194 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-29T16:12:19.195 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-29T16:12:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/1692039397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b80737b0 msgr2=0x7fa9b8073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/1692039397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b80737b0 0x7fa9b8073c20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009990 tx=0x7fa9a4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/1692039397 shutdown_connections 2023-07-29T16:12:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/1692039397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b810c8f0 0x7fa9b810ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/1692039397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 0x7fa9b80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/1692039397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b80737b0 0x7fa9b8073c20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/1692039397 >> 172.21.15.31:0/1692039397 conn(0x7fa9b80fb920 msgr2=0x7fa9b80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/1692039397 shutdown_connections 2023-07-29T16:12:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.415+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/1692039397 wait complete. 2023-07-29T16:12:19.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.416+0000 7fa9bf1ff700 1 Processor -- start 2023-07-29T16:12:19.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 -- start start 2023-07-29T16:12:19.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 0x7fa9b819c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 0x7fa9b819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 0x7fa9b81a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b81a4290 con 0x7fa9b8074d90 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b81a43d0 con 0x7fa9b810c8f0 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bf1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b81a4560 con 0x7fa9b80737b0 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9b7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 0x7fa9b819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bd79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 0x7fa9b81a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:19.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bcf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 0x7fa9b819c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bd79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 0x7fa9b81a3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48568/0 (socket says 172.21.15.31:48568) 2023-07-29T16:12:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.417+0000 7fa9bcf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 0x7fa9b819c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58676/0 (socket says 172.21.15.31:58676) 2023-07-29T16:12:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 -- 172.21.15.31:0/3012479131 learned_addr learned my addr 172.21.15.31:0/3012479131 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 -- 172.21.15.31:0/3012479131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 msgr2=0x7fa9b819c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 0x7fa9b819c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 -- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 msgr2=0x7fa9b819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 0x7fa9b819cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 -- 172.21.15.31:0/3012479131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ac009520 con 0x7fa9b810c8f0 2023-07-29T16:12:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9b7fff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 0x7fa9b819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bcf9b700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 0x7fa9b819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9bd79c700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 0x7fa9b81a3cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000ef30 tx=0x7fa9b000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.418+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0018070 con 0x7fa9b810c8f0 2023-07-29T16:12:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.419+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9b000f040 con 0x7fa9b810c8f0 2023-07-29T16:12:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.419+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a40095f0 con 0x7fa9b810c8f0 2023-07-29T16:12:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.419+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b000cd00 con 0x7fa9b810c8f0 2023-07-29T16:12:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.419+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b81a4bb0 con 0x7fa9b810c8f0 2023-07-29T16:12:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.420+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b8102440 con 0x7fa9b810c8f0 2023-07-29T16:12:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.422+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa9b0003a00 con 0x7fa9b810c8f0 2023-07-29T16:12:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.424+0000 7fa9b5ffb700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9a80a0810 0x7fa9a80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.424+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa9b00cb190 con 0x7fa9b810c8f0 2023-07-29T16:12:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.424+0000 7fa9bcf9b700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9a80a0810 0x7fa9a80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.424+0000 7fa9bcf9b700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9a80a0810 0x7fa9a80a2cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009fd0 tx=0x7fa9ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.425+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa9b0095150 con 0x7fa9b810c8f0 2023-07-29T16:12:19.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:19.665+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa9b802cc30 con 0x7fa9b810c8f0 2023-07-29T16:12:20.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.288+0000 7fa9b5ffb700 1 -- 172.21.15.31:0/3012479131 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa9b0098770 con 0x7fa9b810c8f0 2023-07-29T16:12:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9a80a0810 msgr2=0x7fa9a80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9a80a0810 0x7fa9a80a2cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac009fd0 tx=0x7fa9ac009260 comp rx=0 tx=0).stop 2023-07-29T16:12:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 msgr2=0x7fa9b81a3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 0x7fa9b81a3cc0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000ef30 tx=0x7fa9b000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 shutdown_connections 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9a80a0810 0x7fa9a80a2cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9b810c8f0 0x7fa9b81a3cc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b80737b0 0x7fa9b819c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 --2- 172.21.15.31:0/3012479131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9b8074d90 0x7fa9b819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 >> 172.21.15.31:0/3012479131 conn(0x7fa9b80fb920 msgr2=0x7fa9b80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 shutdown_connections 2023-07-29T16:12:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.292+0000 7fa9bf1ff700 1 -- 172.21.15.31:0/3012479131 wait complete. 2023-07-29T16:12:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2023-07-29T16:12:20.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-29T16:12:20.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:20 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3423223596' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-29T16:12:20.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:20 smithi039 ceph-mon[128858]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:20.523 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:20.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:20 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3012479131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-29T16:12:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 -- 172.21.15.31:0/3265395425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 msgr2=0x7f5428100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 --2- 172.21.15.31:0/3265395425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428100740 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5418009990 tx=0x7f5418009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 -- 172.21.15.31:0/3265395425 shutdown_connections 2023-07-29T16:12:20.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 --2- 172.21.15.31:0/3265395425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428101eb0 0x7f5428108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 --2- 172.21.15.31:0/3265395425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428100740 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 --2- 172.21.15.31:0/3265395425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5428101520 0x7f5428101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 -- 172.21.15.31:0/3265395425 >> 172.21.15.31:0/3265395425 conn(0x7f54280fb970 msgr2=0x7f54280fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 -- 172.21.15.31:0/3265395425 shutdown_connections 2023-07-29T16:12:20.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.553+0000 7f542e9ff700 1 -- 172.21.15.31:0/3265395425 wait complete. 2023-07-29T16:12:20.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.554+0000 7f542e9ff700 1 Processor -- start 2023-07-29T16:12:20.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.554+0000 7f542e9ff700 1 -- start start 2023-07-29T16:12:20.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.554+0000 7f542e9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.554+0000 7f542e9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5428101520 0x7f54281987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f542e9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428101eb0 0x7f542819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f542e9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542819fde0 con 0x7f5428101520 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f542e9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f542819ff20 con 0x7f5428101eb0 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f542e9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54281a00b0 con 0x7f5428100330 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:20.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57536/0 (socket says 172.21.15.31:57536) 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 -- 172.21.15.31:0/336016519 learned_addr learned my addr 172.21.15.31:0/336016519 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f54277fe700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5428101520 0x7f54281987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f542cf9c700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428101eb0 0x7f542819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 -- 172.21.15.31:0/336016519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428101eb0 msgr2=0x7f542819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428101eb0 0x7f542819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 -- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5428101520 msgr2=0x7f54281987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5428101520 0x7f54281987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 -- 172.21.15.31:0/336016519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54180095f0 con 0x7f5428100330 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.555+0000 7f5427fff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428198260 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5418005720 tx=0x7f5418009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.556+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541801d070 con 0x7f5428100330 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.556+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54281a0330 con 0x7f5428100330 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.556+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5418021410 con 0x7f5428100330 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.556+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418017650 con 0x7f5428100330 2023-07-29T16:12:20.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.556+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54281a0800 con 0x7f5428100330 2023-07-29T16:12:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.557+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54281929d0 con 0x7f5428100330 2023-07-29T16:12:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.559+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5418021580 con 0x7f5428100330 2023-07-29T16:12:20.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.560+0000 7f54257fa700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54100a0790 0x7f54100a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:20.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.561+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f54180ccca0 con 0x7f5428100330 2023-07-29T16:12:20.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.561+0000 7f54277fe700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54100a0790 0x7f54100a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:20.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.561+0000 7f54277fe700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54100a0790 0x7f54100a2c40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f541c009c60 tx=0x7f541c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:20.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.562+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5418096a60 con 0x7f5428100330 2023-07-29T16:12:20.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:20 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3423223596' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-29T16:12:20.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:20 smithi156 ceph-mon[129245]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:20.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:20.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:20 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3012479131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-29T16:12:20.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:20 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3423223596' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-29T16:12:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:20 smithi031 ceph-mon[95504]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:20 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3012479131' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-29T16:12:20.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:20.782+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f542802d0a0 con 0x7f5428100330 2023-07-29T16:12:21.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.332+0000 7f54257fa700 1 -- 172.21.15.31:0/336016519 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7f541809a080 con 0x7f5428100330 2023-07-29T16:12:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.344+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54100a0790 msgr2=0x7f54100a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.344+0000 7f542e9ff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54100a0790 0x7f54100a2c40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f541c009c60 tx=0x7f541c009260 comp rx=0 tx=0).stop 2023-07-29T16:12:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.344+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 msgr2=0x7f5428198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.344+0000 7f542e9ff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428198260 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5418005720 tx=0x7f5418009580 comp rx=0 tx=0).stop 2023-07-29T16:12:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.348+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 shutdown_connections 2023-07-29T16:12:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.348+0000 7f542e9ff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54100a0790 0x7f54100a2c40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.348+0000 7f542e9ff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5428101eb0 0x7f542819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.348+0000 7f542e9ff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5428100330 0x7f5428198260 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.348+0000 7f542e9ff700 1 --2- 172.21.15.31:0/336016519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5428101520 0x7f54281987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.348+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 >> 172.21.15.31:0/336016519 conn(0x7f54280fb970 msgr2=0x7f5428106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.351+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 shutdown_connections 2023-07-29T16:12:21.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.351+0000 7f542e9ff700 1 -- 172.21.15.31:0/336016519 wait complete. 2023-07-29T16:12:21.352 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2023-07-29T16:12:21.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-29T16:12:21.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-29T16:12:21.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-29T16:12:21.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: osdmap e83: 12 total, 12 up, 12 in 2023-07-29T16:12:21.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:21 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.599+0000 7fbd59bff700 1 -- 172.21.15.31:0/1413104356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 msgr2=0x7fbd54103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.599+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1413104356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd54103b90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009990 tx=0x7fbd3c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.599+0000 7fbd59bff700 1 -- 172.21.15.31:0/1413104356 shutdown_connections 2023-07-29T16:12:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.599+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1413104356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd541040d0 0x7fbd5410a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.599+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1413104356 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd54102550 0x7fbd54102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.600+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1413104356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd54103b90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.600+0000 7fbd59bff700 1 -- 172.21.15.31:0/1413104356 >> 172.21.15.31:0/1413104356 conn(0x7fbd540fdb90 msgr2=0x7fbd540fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:21.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.600+0000 7fbd59bff700 1 -- 172.21.15.31:0/1413104356 shutdown_connections 2023-07-29T16:12:21.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.600+0000 7fbd59bff700 1 -- 172.21.15.31:0/1413104356 wait complete. 2023-07-29T16:12:21.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.601+0000 7fbd59bff700 1 Processor -- start 2023-07-29T16:12:21.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.601+0000 7fbd59bff700 1 -- start start 2023-07-29T16:12:21.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd59bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 0x7fbd54198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd59bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd541987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd59bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd541040d0 0x7fbd5419f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd59bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5419fe10 con 0x7fbd54102550 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd59bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5419ff50 con 0x7fbd54103740 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd59bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd541a00e0 con 0x7fbd541040d0 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd537fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 0x7fbd54198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd4bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd541987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd537fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 0x7fbd54198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39760/0 (socket says 172.21.15.31:39760) 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd537fe700 1 -- 172.21.15.31:0/1357446809 learned_addr learned my addr 172.21.15.31:0/1357446809 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.602+0000 7fbd53fff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd541040d0 0x7fbd5419f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd4bfff700 1 -- 172.21.15.31:0/1357446809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd541040d0 msgr2=0x7fbd5419f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd4bfff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd541040d0 0x7fbd5419f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd4bfff700 1 -- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 msgr2=0x7fbd54198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd4bfff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 0x7fbd54198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd4bfff700 1 -- 172.21.15.31:0/1357446809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c0095f0 con 0x7fbd54103740 2023-07-29T16:12:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd537fe700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 0x7fbd54198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd4bfff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd541987d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0098e0 tx=0x7fbd3c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:21.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c01d070 con 0x7fbd54103740 2023-07-29T16:12:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd541a0360 con 0x7fbd54103740 2023-07-29T16:12:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd3c021410 con 0x7fbd54103740 2023-07-29T16:12:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.603+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c017650 con 0x7fbd54103740 2023-07-29T16:12:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.604+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd541a07e0 con 0x7fbd54103740 2023-07-29T16:12:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.605+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd54109430 con 0x7fbd54103740 2023-07-29T16:12:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.606+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbd3c021580 con 0x7fbd54103740 2023-07-29T16:12:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.608+0000 7fbd517fa700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbd400a0790 0x7fbd400a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:21.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.608+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fbd3c0cce40 con 0x7fbd54103740 2023-07-29T16:12:21.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.608+0000 7fbd537fe700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbd400a0790 0x7fbd400a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:21.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.609+0000 7fbd537fe700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbd400a0790 0x7fbd400a2c40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005950 tx=0x7fbd44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.610+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbd3c096a80 con 0x7fbd54103740 2023-07-29T16:12:21.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-29T16:12:21.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-29T16:12:21.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: osdmap e83: 12 total, 12 up, 12 in 2023-07-29T16:12:21.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:21 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-29T16:12:21.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-29T16:12:21.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: osdmap e83: 12 total, 12 up, 12 in 2023-07-29T16:12:21.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:21 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:21.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.909+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fbd5402cdf0 con 0x7fbd54103740 2023-07-29T16:12:21.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.926+0000 7fbd517fa700 1 -- 172.21.15.31:0/1357446809 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fbd3c0260c0 con 0x7fbd54103740 2023-07-29T16:12:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbd400a0790 msgr2=0x7fbd400a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbd400a0790 0x7fbd400a2c40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005950 tx=0x7fbd44009260 comp rx=0 tx=0).stop 2023-07-29T16:12:21.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 msgr2=0x7fbd541987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd541987d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0098e0 tx=0x7fbd3c005740 comp rx=0 tx=0).stop 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 shutdown_connections 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbd400a0790 0x7fbd400a2c40 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005950 tx=0x7fbd44009260 comp rx=0 tx=0).stop 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd541040d0 0x7fbd5419f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd54102550 0x7fbd54198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 --2- 172.21.15.31:0/1357446809 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd54103740 0x7fbd541987d0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0098e0 tx=0x7fbd3c005740 comp rx=0 tx=0).stop 2023-07-29T16:12:21.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.934+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 >> 172.21.15.31:0/1357446809 conn(0x7fbd540fdb90 msgr2=0x7fbd540ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:21.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.936+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 shutdown_connections 2023-07-29T16:12:21.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:21.939+0000 7fbd59bff700 1 -- 172.21.15.31:0/1357446809 wait complete. 2023-07-29T16:12:21.951 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2023-07-29T16:12:21.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:12:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 -- 172.21.15.31:0/1215301188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 msgr2=0x7fd210108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 --2- 172.21.15.31:0/1215301188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 0x7fd210108670 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd204009990 tx=0x7fd204009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 -- 172.21.15.31:0/1215301188 shutdown_connections 2023-07-29T16:12:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 --2- 172.21.15.31:0/1215301188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd210071e50 0x7fd2101110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 --2- 172.21.15.31:0/1215301188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 0x7fd210108670 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 --2- 172.21.15.31:0/1215301188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd210071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 -- 172.21.15.31:0/1215301188 >> 172.21.15.31:0/1215301188 conn(0x7fd21006d280 msgr2=0x7fd21006f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 -- 172.21.15.31:0/1215301188 shutdown_connections 2023-07-29T16:12:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.180+0000 7fd216dff700 1 -- 172.21.15.31:0/1215301188 wait complete. 2023-07-29T16:12:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.182+0000 7fd216dff700 1 Processor -- start 2023-07-29T16:12:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.182+0000 7fd216dff700 1 -- start start 2023-07-29T16:12:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.182+0000 7fd216dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd21010a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.182+0000 7fd216dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd210071e50 0x7fd210109020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.182+0000 7fd216dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 0x7fd210109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd216dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210109d00 con 0x7fd2100714a0 2023-07-29T16:12:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd216dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210109e70 con 0x7fd210108260 2023-07-29T16:12:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd216dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2101a8870 con 0x7fd210071e50 2023-07-29T16:12:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd21010a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd21010a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39782/0 (socket says 172.21.15.31:39782) 2023-07-29T16:12:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 -- 172.21.15.31:0/834655641 learned_addr learned my addr 172.21.15.31:0/834655641 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd21539c700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 0x7fd210109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd20ffff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd210071e50 0x7fd210109020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 -- 172.21.15.31:0/834655641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd210071e50 msgr2=0x7fd210109020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd210071e50 0x7fd210109020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 -- 172.21.15.31:0/834655641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 msgr2=0x7fd210109560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 0x7fd210109560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.183+0000 7fd214b9b700 1 -- 172.21.15.31:0/834655641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2040095f0 con 0x7fd2100714a0 2023-07-29T16:12:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.184+0000 7fd214b9b700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd21010a9e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd20400b680 tx=0x7fd2040057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.184+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd20401d070 con 0x7fd2100714a0 2023-07-29T16:12:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.184+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd204022470 con 0x7fd2100714a0 2023-07-29T16:12:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.184+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2101a8af0 con 0x7fd2100714a0 2023-07-29T16:12:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.184+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd204017650 con 0x7fd2100714a0 2023-07-29T16:12:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.184+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2101a8f10 con 0x7fd2100714a0 2023-07-29T16:12:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.185+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd21010fb10 con 0x7fd2100714a0 2023-07-29T16:12:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.187+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd2040225e0 con 0x7fd2100714a0 2023-07-29T16:12:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.188+0000 7fd20dffb700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2000a07e0 0x7fd2000a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.189+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd2040cceb0 con 0x7fd2100714a0 2023-07-29T16:12:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.189+0000 7fd20ffff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2000a07e0 0x7fd2000a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.189+0000 7fd20ffff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2000a07e0 0x7fd2000a2c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009c60 tx=0x7fd1fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:22.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.193+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd204096a90 con 0x7fd2100714a0 2023-07-29T16:12:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.417+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd21002cc30 con 0x7fd2100714a0 2023-07-29T16:12:22.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.418+0000 7fd20dffb700 1 -- 172.21.15.31:0/834655641 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22103 (secure 0 0 0) 0x7fd204027090 con 0x7fd2100714a0 2023-07-29T16:12:22.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:22.417 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":85,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:21.911951+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:12:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2000a07e0 msgr2=0x7fd2000a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2000a07e0 0x7fd2000a2c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc009c60 tx=0x7fd1fc009260 comp rx=0 tx=0).stop 2023-07-29T16:12:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 msgr2=0x7fd21010a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd21010a9e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd20400b680 tx=0x7fd2040057b0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 shutdown_connections 2023-07-29T16:12:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2000a07e0 0x7fd2000a2c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd210108260 0x7fd210109560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd2100714a0 0x7fd21010a9e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 --2- 172.21.15.31:0/834655641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd210071e50 0x7fd210109020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 >> 172.21.15.31:0/834655641 conn(0x7fd21006d280 msgr2=0x7fd2100704b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:22.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 shutdown_connections 2023-07-29T16:12:22.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.421+0000 7fd216dff700 1 -- 172.21.15.31:0/834655641 wait complete. 2023-07-29T16:12:22.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:12:22.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/336016519' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-29T16:12:22.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-29T16:12:22.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: pgmap v269: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:22.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-29T16:12:22.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: osdmap e84: 12 total, 12 up, 12 in 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:12:22.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1357446809' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-29T16:12:22.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-29T16:12:22.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-29T16:12:22.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: osdmap e85: 12 total, 12 up, 12 in 2023-07-29T16:12:22.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-29T16:12:22.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-29T16:12:22.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:22 smithi156 ceph-mon[129245]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-29T16:12:22.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.648+0000 7f98337ff700 1 -- 172.21.15.31:0/3054110590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101ec0 msgr2=0x7f982c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.648+0000 7f98337ff700 1 --2- 172.21.15.31:0/3054110590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101ec0 0x7f982c108780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f982400b0a0 tx=0x7f982400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 -- 172.21.15.31:0/3054110590 shutdown_connections 2023-07-29T16:12:22.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 --2- 172.21.15.31:0/3054110590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101ec0 0x7f982c108780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 --2- 172.21.15.31:0/3054110590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 --2- 172.21.15.31:0/3054110590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f982c101530 0x7f982c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 -- 172.21.15.31:0/3054110590 >> 172.21.15.31:0/3054110590 conn(0x7f982c0fb920 msgr2=0x7f982c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:22.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 -- 172.21.15.31:0/3054110590 shutdown_connections 2023-07-29T16:12:22.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.649+0000 7f98337ff700 1 -- 172.21.15.31:0/3054110590 wait complete. 2023-07-29T16:12:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.650+0000 7f98337ff700 1 Processor -- start 2023-07-29T16:12:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.650+0000 7f98337ff700 1 -- start start 2023-07-29T16:12:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.650+0000 7f98337ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.650+0000 7f98337ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101530 0x7f982c1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f98337ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f982c101ec0 0x7f982c19f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f98337ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c19fef0 con 0x7f982c101530 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f98337ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1a0030 con 0x7f982c100340 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f98337ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1a01c0 con 0x7f982c101ec0 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58688/0 (socket says 172.21.15.31:58688) 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 -- 172.21.15.31:0/1857578476 learned_addr learned my addr 172.21.15.31:0/1857578476 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f9830d9a700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101530 0x7f982c1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f9831d9c700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f982c101ec0 0x7f982c19f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 -- 172.21.15.31:0/1857578476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f982c101ec0 msgr2=0x7f982c19f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f982c101ec0 0x7f982c19f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 -- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101530 msgr2=0x7f982c1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101530 0x7f982c1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 -- 172.21.15.31:0/1857578476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828009520 con 0x7f982c100340 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.651+0000 7f983159b700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c198370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f982800ea50 tx=0x7f982800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:22.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.652+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800cbd0 con 0x7f982c100340 2023-07-29T16:12:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.652+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9824009d00 con 0x7f982c100340 2023-07-29T16:12:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.652+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f982800cd30 con 0x7f982c100340 2023-07-29T16:12:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.652+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828018680 con 0x7f982c100340 2023-07-29T16:12:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.652+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c1a0820 con 0x7f982c100340 2023-07-29T16:12:22.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.653+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c10bb60 con 0x7f982c100340 2023-07-29T16:12:22.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.656+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9828010430 con 0x7f982c100340 2023-07-29T16:12:22.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.658+0000 7f98227fc700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98180a07e0 0x7f98180a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:22.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.658+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9828014070 con 0x7f982c100340 2023-07-29T16:12:22.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.659+0000 7f9830d9a700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98180a07e0 0x7f98180a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:22.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.659+0000 7f9830d9a700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98180a07e0 0x7f98180a2c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f981c009c60 tx=0x7f981c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:22.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.662+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f98280d0070 con 0x7f982c100340 2023-07-29T16:12:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/336016519' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-29T16:12:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-29T16:12:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: pgmap v269: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-29T16:12:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: osdmap e84: 12 total, 12 up, 12 in 2023-07-29T16:12:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1357446809' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-29T16:12:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-29T16:12:22.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-29T16:12:22.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: osdmap e85: 12 total, 12 up, 12 in 2023-07-29T16:12:22.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-29T16:12:22.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-29T16:12:22.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:22 smithi031 ceph-mon[95504]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-29T16:12:22.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/336016519' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-29T16:12:22.823 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: pgmap v269: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: osdmap e84: 12 total, 12 up, 12 in 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:22.824 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1357446809' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: osdmap e85: 12 total, 12 up, 12 in 2023-07-29T16:12:22.825 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-29T16:12:22.826 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-29T16:12:22.826 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:22 smithi039 ceph-mon[128858]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-29T16:12:22.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.884+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f982c10bcf0 con 0x7f982c100340 2023-07-29T16:12:22.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.885+0000 7f98227fc700 1 -- 172.21.15.31:0/1857578476 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22103 (secure 0 0 0) 0x7f9828095ac0 con 0x7f982c100340 2023-07-29T16:12:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:22.884 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":85,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:21.911951+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:12:22.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98180a07e0 msgr2=0x7f98180a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98180a07e0 0x7f98180a2c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f981c009c60 tx=0x7f981c009260 comp rx=0 tx=0).stop 2023-07-29T16:12:22.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 msgr2=0x7f982c198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:22.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c198370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f982800ea50 tx=0x7f982800ed60 comp rx=0 tx=0).stop 2023-07-29T16:12:22.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 shutdown_connections 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98180a07e0 0x7f98180a2c90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f982c101ec0 0x7f982c19f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f982c100340 0x7f982c198370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 --2- 172.21.15.31:0/1857578476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f982c101530 0x7f982c1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 >> 172.21.15.31:0/1857578476 conn(0x7f982c0fb920 msgr2=0x7f982c106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 shutdown_connections 2023-07-29T16:12:22.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:22.888+0000 7f98337ff700 1 -- 172.21.15.31:0/1857578476 wait complete. 2023-07-29T16:12:22.909 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-29T16:12:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 -- 172.21.15.31:0/1400895023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 msgr2=0x7f6498100640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 --2- 172.21.15.31:0/1400895023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498100640 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f648c009990 tx=0x7f648c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 -- 172.21.15.31:0/1400895023 shutdown_connections 2023-07-29T16:12:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 --2- 172.21.15.31:0/1400895023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6498101db0 0x7f6498108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 --2- 172.21.15.31:0/1400895023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498100640 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 --2- 172.21.15.31:0/1400895023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 0x7f6498101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 -- 172.21.15.31:0/1400895023 >> 172.21.15.31:0/1400895023 conn(0x7f64980fb850 msgr2=0x7f64980fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 -- 172.21.15.31:0/1400895023 shutdown_connections 2023-07-29T16:12:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.133+0000 7f649dd9d700 1 -- 172.21.15.31:0/1400895023 wait complete. 2023-07-29T16:12:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.134+0000 7f649dd9d700 1 Processor -- start 2023-07-29T16:12:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.134+0000 7f649dd9d700 1 -- start start 2023-07-29T16:12:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.134+0000 7f649dd9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.134+0000 7f649dd9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 0x7f6498198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649dd9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6498101db0 0x7f649819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649dd9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649819fb70 con 0x7f6498100230 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649dd9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649819fcb0 con 0x7f6498101420 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649dd9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649819fe40 con 0x7f6498101db0 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39808/0 (socket says 172.21.15.31:39808) 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649cd9b700 1 -- 172.21.15.31:0/2670122238 learned_addr learned my addr 172.21.15.31:0/2670122238 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:23.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 0x7f6498198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f649d59c700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6498101db0 0x7f649819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 -- 172.21.15.31:0/2670122238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6498101db0 msgr2=0x7f649819f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6498101db0 0x7f649819f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 -- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 msgr2=0x7f6498197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:23.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 -- 172.21.15.31:0/2670122238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6484009520 con 0x7f6498101420 2023-07-29T16:12:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.135+0000 7f6497fff700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 0x7f6498198530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f648400e9f0 tx=0x7f648400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.136+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400cb70 con 0x7f6498101420 2023-07-29T16:12:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.136+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c0095f0 con 0x7f6498101420 2023-07-29T16:12:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.136+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f648400ee90 con 0x7f6498101420 2023-07-29T16:12:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.136+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484003cf0 con 0x7f6498101420 2023-07-29T16:12:23.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.136+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64981a04f0 con 0x7f6498101420 2023-07-29T16:12:23.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.137+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649810ba50 con 0x7f6498101420 2023-07-29T16:12:23.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.139+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f648400ccd0 con 0x7f6498101420 2023-07-29T16:12:23.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.140+0000 7f6495ffb700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64880a0830 0x7f64880a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:23.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.140+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6484014070 con 0x7f6498101420 2023-07-29T16:12:23.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.140+0000 7f649cd9b700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64880a0830 0x7f64880a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:23.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.141+0000 7f649cd9b700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64880a0830 0x7f64880a2ce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f648c00b2e0 tx=0x7f648c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:23.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.144+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f64840958b0 con 0x7f6498101420 2023-07-29T16:12:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:23.402+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f649810bbe0 con 0x7f6498101420 2023-07-29T16:12:23.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/834655641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:23.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1857578476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:23.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:23 smithi156 ceph-mon[129245]: osdmap e86: 12 total, 12 up, 12 in 2023-07-29T16:12:23.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:23 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] up:active 2023-07-29T16:12:23.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:23 smithi156 ceph-mon[129245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:23.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:23 smithi156 ceph-mon[129245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:23.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/834655641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1857578476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:23 smithi031 ceph-mon[95504]: osdmap e86: 12 total, 12 up, 12 in 2023-07-29T16:12:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:23 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] up:active 2023-07-29T16:12:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:23 smithi031 ceph-mon[95504]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:23 smithi031 ceph-mon[95504]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:23.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/834655641' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:23.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1857578476' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:23 smithi039 ceph-mon[128858]: osdmap e86: 12 total, 12 up, 12 in 2023-07-29T16:12:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:23 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] up:active 2023-07-29T16:12:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:23 smithi039 ceph-mon[128858]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:23 smithi039 ceph-mon[128858]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.326+0000 7f6495ffb700 1 -- 172.21.15.31:0/2670122238 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6484098ed0 con 0x7f6498101420 2023-07-29T16:12:24.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64880a0830 msgr2=0x7f64880a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64880a0830 0x7f64880a2ce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f648c00b2e0 tx=0x7f648c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 msgr2=0x7f6498198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 0x7f6498198530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f648400e9f0 tx=0x7f648400ed00 comp rx=0 tx=0).stop 2023-07-29T16:12:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 shutdown_connections 2023-07-29T16:12:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64880a0830 0x7f64880a2ce0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6498101db0 0x7f649819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6498100230 0x7f6498197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 --2- 172.21.15.31:0/2670122238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6498101420 0x7f6498198530 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 >> 172.21.15.31:0/2670122238 conn(0x7f64980fb850 msgr2=0x7f64980fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 shutdown_connections 2023-07-29T16:12:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.329+0000 7f649dd9d700 1 -- 172.21.15.31:0/2670122238 wait complete. 2023-07-29T16:12:24.346 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-29T16:12:24.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-07-29T16:12:24.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:24 smithi156 ceph-mon[129245]: pgmap v273: 97 pgs: 8 creating+peering, 46 unknown, 43 active+clean; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 8 op/s 2023-07-29T16:12:24.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:24 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2670122238' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-29T16:12:24.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:24 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-29T16:12:24.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.576+0000 7f9c861ff700 1 -- 172.21.15.31:0/3779331468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 msgr2=0x7f9c80108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.576+0000 7f9c861ff700 1 --2- 172.21.15.31:0/3779331468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c80108770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9c7800b0a0 tx=0x7f9c7800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 -- 172.21.15.31:0/3779331468 shutdown_connections 2023-07-29T16:12:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 --2- 172.21.15.31:0/3779331468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c80108770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 --2- 172.21.15.31:0/3779331468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80100330 0x7f9c80100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 --2- 172.21.15.31:0/3779331468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80101520 0x7f9c80101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 -- 172.21.15.31:0/3779331468 >> 172.21.15.31:0/3779331468 conn(0x7f9c800fb970 msgr2=0x7f9c800fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 -- 172.21.15.31:0/3779331468 shutdown_connections 2023-07-29T16:12:24.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.577+0000 7f9c861ff700 1 -- 172.21.15.31:0/3779331468 wait complete. 2023-07-29T16:12:24.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 Processor -- start 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 -- start start 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 0x7f9c80195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80101520 0x7f9c801964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c8019d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8019dae0 con 0x7f9c80100330 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8019dc20 con 0x7f9c80101520 2023-07-29T16:12:24.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.578+0000 7f9c861ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c8019ddb0 con 0x7f9c80101eb0 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 0x7f9c80195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80101520 0x7f9c801964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80101520 0x7f9c801964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58716/0 (socket says 172.21.15.31:58716) 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 0x7f9c80195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39824/0 (socket says 172.21.15.31:39824) 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7effd700 1 -- 172.21.15.31:0/1154372586 learned_addr learned my addr 172.21.15.31:0/1154372586 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c8019d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:24.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 -- 172.21.15.31:0/1154372586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80101520 msgr2=0x7f9c801964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:24.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80101520 0x7f9c801964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 -- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 msgr2=0x7f9c80195f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:24.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 0x7f9c80195f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:24.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 -- 172.21.15.31:0/1154372586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c78009d00 con 0x7f9c80101eb0 2023-07-29T16:12:24.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7f7fe700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 0x7f9c80195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:24.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.579+0000 7f9c7ffff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c8019d510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009fd0 tx=0x7f9c7800bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:24.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.580+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7800e040 con 0x7f9c80101eb0 2023-07-29T16:12:24.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.580+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c780094c0 con 0x7f9c80101eb0 2023-07-29T16:12:24.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.580+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8019e030 con 0x7f9c80101eb0 2023-07-29T16:12:24.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.580+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c780052a0 con 0x7f9c80101eb0 2023-07-29T16:12:24.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.580+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8019e4b0 con 0x7f9c80101eb0 2023-07-29T16:12:24.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.581+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c801071c0 con 0x7f9c80101eb0 2023-07-29T16:12:24.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.584+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9c7801d470 con 0x7f9c80101eb0 2023-07-29T16:12:24.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.585+0000 7f9c7cff9700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9c680a0810 0x7f9c680a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:24.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.586+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9c780cd390 con 0x7f9c80101eb0 2023-07-29T16:12:24.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.586+0000 7f9c7f7fe700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9c680a0810 0x7f9c680a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:24.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.586+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9c78096f70 con 0x7f9c80101eb0 2023-07-29T16:12:24.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.586+0000 7f9c7f7fe700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9c680a0810 0x7f9c680a2cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c80101380 tx=0x7f9c70009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:24.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:24 smithi031 ceph-mon[95504]: pgmap v273: 97 pgs: 8 creating+peering, 46 unknown, 43 active+clean; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 8 op/s 2023-07-29T16:12:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:24 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2670122238' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-29T16:12:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:24 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-29T16:12:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:24 smithi039 ceph-mon[128858]: pgmap v273: 97 pgs: 8 creating+peering, 46 unknown, 43 active+clean; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 8 op/s 2023-07-29T16:12:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:24 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2670122238' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-29T16:12:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:24 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-29T16:12:24.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:24.851+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f9c8002cdc0 con 0x7f9c80101eb0 2023-07-29T16:12:25.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.459+0000 7f9c7cff9700 1 -- 172.21.15.31:0/1154372586 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9c7809a590 con 0x7f9c80101eb0 2023-07-29T16:12:25.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9c680a0810 msgr2=0x7f9c680a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:25.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9c680a0810 0x7f9c680a2cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c80101380 tx=0x7f9c70009260 comp rx=0 tx=0).stop 2023-07-29T16:12:25.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 msgr2=0x7f9c8019d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:25.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c8019d510 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009fd0 tx=0x7f9c7800bce0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 shutdown_connections 2023-07-29T16:12:25.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9c680a0810 0x7f9c680a2cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9c80101eb0 0x7f9c8019d510 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9c80100330 0x7f9c80195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 --2- 172.21.15.31:0/1154372586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9c80101520 0x7f9c801964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 >> 172.21.15.31:0/1154372586 conn(0x7f9c800fb970 msgr2=0x7f9c800fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:25.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 shutdown_connections 2023-07-29T16:12:25.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.462+0000 7f9c861ff700 1 -- 172.21.15.31:0/1154372586 wait complete. 2023-07-29T16:12:25.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-29T16:12:25.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-29T16:12:25.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:25.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1154372586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-29T16:12:25.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:25 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-29T16:12:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 -- 172.21.15.31:0/2638050667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 msgr2=0x7f01b410a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:25.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 --2- 172.21.15.31:0/2638050667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b410a670 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f01ac00b0a0 tx=0x7f01ac00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 -- 172.21.15.31:0/2638050667 shutdown_connections 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 --2- 172.21.15.31:0/2638050667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b410a670 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 --2- 172.21.15.31:0/2638050667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 0x7f01b40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 --2- 172.21.15.31:0/2638050667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 0x7f01b4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 -- 172.21.15.31:0/2638050667 >> 172.21.15.31:0/2638050667 conn(0x7f01b40fb970 msgr2=0x7f01b40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.705+0000 7f01b89ff700 1 -- 172.21.15.31:0/2638050667 shutdown_connections 2023-07-29T16:12:25.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.706+0000 7f01b89ff700 1 -- 172.21.15.31:0/2638050667 wait complete. 2023-07-29T16:12:25.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.707+0000 7f01b89ff700 1 Processor -- start 2023-07-29T16:12:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.707+0000 7f01b89ff700 1 -- start start 2023-07-29T16:12:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.707+0000 7f01b89ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 0x7f01b41983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.707+0000 7f01b89ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 0x7f01b4198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.707+0000 7f01b89ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b419f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b89ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b419ff40 con 0x7f01b41081f0 2023-07-29T16:12:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b89ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b41a0080 con 0x7f01b4074d90 2023-07-29T16:12:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b89ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b41a0210 con 0x7f01b40737b0 2023-07-29T16:12:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 0x7f01b41983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b419f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b419f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39850/0 (socket says 172.21.15.31:39850) 2023-07-29T16:12:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 -- 172.21.15.31:0/3638152266 learned_addr learned my addr 172.21.15.31:0/3638152266 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b1d9b700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 0x7f01b4198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:25.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 -- 172.21.15.31:0/3638152266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 msgr2=0x7f01b41983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 0x7f01b41983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 -- 172.21.15.31:0/3638152266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 msgr2=0x7f01b4198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 0x7f01b4198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b2d9d700 1 -- 172.21.15.31:0/3638152266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac009d00 con 0x7f01b41081f0 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b259c700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 0x7f01b41983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.708+0000 7f01b1d9b700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 0x7f01b4198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:25.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.709+0000 7f01b2d9d700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b419f970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009fd0 tx=0x7f01ac00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.709+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac00e040 con 0x7f01b41081f0 2023-07-29T16:12:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.709+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b41a0490 con 0x7f01b41081f0 2023-07-29T16:12:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.709+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01ac0094a0 con 0x7f01b41081f0 2023-07-29T16:12:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.709+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01ac01b6c0 con 0x7f01b41081f0 2023-07-29T16:12:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.709+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b41a08b0 con 0x7f01b41081f0 2023-07-29T16:12:25.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.710+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b41068a0 con 0x7f01b41081f0 2023-07-29T16:12:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.713+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f01ac01d470 con 0x7f01b41081f0 2023-07-29T16:12:25.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.715+0000 7f01a37fe700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f019c0a0790 0x7f019c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.715+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f01ac0ccb70 con 0x7f01b41081f0 2023-07-29T16:12:25.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.715+0000 7f01b259c700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f019c0a0790 0x7f019c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:25.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.716+0000 7f01b259c700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f019c0a0790 0x7f019c0a2c40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f01b4075160 tx=0x7f01a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:25.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.719+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f01ac096750 con 0x7f01b41081f0 2023-07-29T16:12:25.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-29T16:12:25.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:25.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1154372586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-29T16:12:25.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:25 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-29T16:12:25.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-29T16:12:25.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1154372586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-29T16:12:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:25 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-29T16:12:25.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:25.978+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f01b402cc30 con 0x7f01b41081f0 2023-07-29T16:12:26.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.476+0000 7f01a37fe700 1 -- 172.21.15.31:0/3638152266 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f01ac099d70 con 0x7f01b41081f0 2023-07-29T16:12:26.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.479+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f019c0a0790 msgr2=0x7f019c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:26.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.479+0000 7f01b89ff700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f019c0a0790 0x7f019c0a2c40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f01b4075160 tx=0x7f01a4009260 comp rx=0 tx=0).stop 2023-07-29T16:12:26.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.479+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 msgr2=0x7f01b419f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:26.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.479+0000 7f01b89ff700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b419f970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009fd0 tx=0x7f01ac00bce0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 shutdown_connections 2023-07-29T16:12:26.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f019c0a0790 0x7f019c0a2c40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01b41081f0 0x7f01b419f970 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01b40737b0 0x7f01b41983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 --2- 172.21.15.31:0/3638152266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01b4074d90 0x7f01b4198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 >> 172.21.15.31:0/3638152266 conn(0x7f01b40fb970 msgr2=0x7f01b410a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 shutdown_connections 2023-07-29T16:12:26.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.480+0000 7f01b89ff700 1 -- 172.21.15.31:0/3638152266 wait complete. 2023-07-29T16:12:26.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-29T16:12:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: pgmap v274: 97 pgs: 8 creating+peering, 89 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 8 op/s 2023-07-29T16:12:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-29T16:12:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-29T16:12:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-29T16:12:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: daemon mds.g is now active in filesystem cephfs as rank 1 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:26 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3638152266' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-29T16:12:26.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 -- 172.21.15.31:0/35693867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314101ea0 msgr2=0x7fa314108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:26.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 --2- 172.21.15.31:0/35693867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314101ea0 0x7fa314108760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00b0a0 tx=0x7fa30c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 -- 172.21.15.31:0/35693867 shutdown_connections 2023-07-29T16:12:26.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 --2- 172.21.15.31:0/35693867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314101ea0 0x7fa314108760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 --2- 172.21.15.31:0/35693867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314100320 0x7fa314100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 --2- 172.21.15.31:0/35693867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa314101510 0x7fa314101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 -- 172.21.15.31:0/35693867 >> 172.21.15.31:0/35693867 conn(0x7fa3140fb920 msgr2=0x7fa3140fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:26.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 -- 172.21.15.31:0/35693867 shutdown_connections 2023-07-29T16:12:26.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.718+0000 7fa3189ff700 1 -- 172.21.15.31:0/35693867 wait complete. 2023-07-29T16:12:26.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 Processor -- start 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 -- start start 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314100320 0x7fa314198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa314101510 0x7fa3141986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 0x7fa31419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31419fd10 con 0x7fa314101ea0 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31419fe50 con 0x7fa314101510 2023-07-29T16:12:26.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.719+0000 7fa3189ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31419ffe0 con 0x7fa314100320 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 0x7fa31419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa31259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314100320 0x7fa314198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 0x7fa31419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39880/0 (socket says 172.21.15.31:39880) 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 -- 172.21.15.31:0/2784099071 learned_addr learned my addr 172.21.15.31:0/2784099071 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa311d9b700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa314101510 0x7fa3141986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 -- 172.21.15.31:0/2784099071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314100320 msgr2=0x7fa314198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314100320 0x7fa314198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 -- 172.21.15.31:0/2784099071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa314101510 msgr2=0x7fa3141986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa314101510 0x7fa3141986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 -- 172.21.15.31:0/2784099071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c009d00 con 0x7fa314101ea0 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa31259c700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314100320 0x7fa314198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.720+0000 7fa312d9d700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 0x7fa31419f740 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009850 tx=0x7fa30c00bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.721+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00e040 con 0x7fa314101ea0 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.721+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa30c003e10 con 0x7fa314101ea0 2023-07-29T16:12:26.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.721+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3141a0260 con 0x7fa314101ea0 2023-07-29T16:12:26.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.721+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c01b6c0 con 0x7fa314101ea0 2023-07-29T16:12:26.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.721+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3141a0630 con 0x7fa314101ea0 2023-07-29T16:12:26.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.722+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa314192950 con 0x7fa314101ea0 2023-07-29T16:12:26.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.724+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa30c01d470 con 0x7fa314101ea0 2023-07-29T16:12:26.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.726+0000 7fa3037fe700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2fc0a0790 0x7fa2fc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:26.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.726+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa30c0cce70 con 0x7fa314101ea0 2023-07-29T16:12:26.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.726+0000 7fa31259c700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2fc0a0790 0x7fa2fc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:26.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.727+0000 7fa31259c700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2fc0a0790 0x7fa2fc0a2c40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa314101370 tx=0x7fa304009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:26.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.730+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa30c096a50 con 0x7fa314101ea0 2023-07-29T16:12:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: pgmap v274: 97 pgs: 8 creating+peering, 89 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 8 op/s 2023-07-29T16:12:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-29T16:12:26.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: daemon mds.g is now active in filesystem cephfs as rank 1 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:12:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:26 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3638152266' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-29T16:12:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: pgmap v274: 97 pgs: 8 creating+peering, 89 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s wr, 8 op/s 2023-07-29T16:12:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-29T16:12:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-29T16:12:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: daemon mds.g is now active in filesystem cephfs as rank 1 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:12:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:12:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:26 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3638152266' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-29T16:12:26.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:26.991+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fa31402cc30 con 0x7fa314101ea0 2023-07-29T16:12:27.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.749+0000 7fa3037fe700 1 -- 172.21.15.31:0/2784099071 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa30c09a070 con 0x7fa314101ea0 2023-07-29T16:12:27.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2fc0a0790 msgr2=0x7fa2fc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:27.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2fc0a0790 0x7fa2fc0a2c40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa314101370 tx=0x7fa304009260 comp rx=0 tx=0).stop 2023-07-29T16:12:27.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 msgr2=0x7fa31419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:27.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 0x7fa31419f740 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009850 tx=0x7fa30c00bce0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 shutdown_connections 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2fc0a0790 0x7fa2fc0a2c40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa314101ea0 0x7fa31419f740 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa314100320 0x7fa314198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 --2- 172.21.15.31:0/2784099071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa314101510 0x7fa3141986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 >> 172.21.15.31:0/2784099071 conn(0x7fa3140fb920 msgr2=0x7fa314106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:27.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 shutdown_connections 2023-07-29T16:12:27.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.752+0000 7fa3189ff700 1 -- 172.21.15.31:0/2784099071 wait complete. 2023-07-29T16:12:27.752 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] up:active 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3638152266' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: Cluster is now healthy 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2023-07-29T16:12:27.753 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-07-29T16:12:27.754 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:active 2023-07-29T16:12:27.754 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:12:27.754 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2784099071' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-29T16:12:27.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] up:active 2023-07-29T16:12:27.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3638152266' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-29T16:12:27.770 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: Cluster is now healthy 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:active 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:12:27.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:27 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2784099071' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-29T16:12:27.772 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-29T16:12:27.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-29T16:12:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] up:active 2023-07-29T16:12:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3638152266' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-29T16:12:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-29T16:12:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-29T16:12:27.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: Cluster is now healthy 2023-07-29T16:12:27.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2023-07-29T16:12:27.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2023-07-29T16:12:27.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-07-29T16:12:27.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:active 2023-07-29T16:12:27.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:12:27.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:27 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2784099071' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-29T16:12:27.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 -- 172.21.15.31:0/1559621148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101510 msgr2=0x7ff824101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:27.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/1559621148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101510 0x7ff824101960 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009990 tx=0x7ff80c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 -- 172.21.15.31:0/1559621148 shutdown_connections 2023-07-29T16:12:27.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/1559621148 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824101ea0 0x7ff824108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/1559621148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824100320 0x7ff824100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/1559621148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101510 0x7ff824101960 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 -- 172.21.15.31:0/1559621148 >> 172.21.15.31:0/1559621148 conn(0x7ff8240fb920 msgr2=0x7ff8240fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:27.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 -- 172.21.15.31:0/1559621148 shutdown_connections 2023-07-29T16:12:27.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.990+0000 7ff82a5ff700 1 -- 172.21.15.31:0/1559621148 wait complete. 2023-07-29T16:12:27.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 Processor -- start 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 -- start start 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824100320 0x7ff8241982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824101510 0x7ff824198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 0x7ff82419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82419fe50 con 0x7ff824101510 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82419ff90 con 0x7ff824101ea0 2023-07-29T16:12:27.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.991+0000 7ff82a5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8241a0120 con 0x7ff824100320 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 0x7ff82419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff81bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824101510 0x7ff824198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 0x7ff82419f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58768/0 (socket says 172.21.15.31:58768) 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 -- 172.21.15.31:0/3227873378 learned_addr learned my addr 172.21.15.31:0/3227873378 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff823fff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824100320 0x7ff8241982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 -- 172.21.15.31:0/3227873378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824100320 msgr2=0x7ff8241982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824100320 0x7ff8241982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 -- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824101510 msgr2=0x7ff824198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824101510 0x7ff824198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 -- 172.21.15.31:0/3227873378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0095f0 con 0x7ff824101ea0 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff823fff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824100320 0x7ff8241982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff81bfff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824101510 0x7ff824198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.992+0000 7ff828b9c700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 0x7ff82419f880 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00ef30 tx=0x7ff81c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.993+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c018070 con 0x7ff824101ea0 2023-07-29T16:12:27.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.993+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff81c00f040 con 0x7ff824101ea0 2023-07-29T16:12:27.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.993+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8241a03a0 con 0x7ff824101ea0 2023-07-29T16:12:27.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.993+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81c00cdb0 con 0x7ff824101ea0 2023-07-29T16:12:27.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.993+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8241a0900 con 0x7ff824101ea0 2023-07-29T16:12:27.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.994+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82410bb50 con 0x7ff824101ea0 2023-07-29T16:12:27.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.996+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff81c003a00 con 0x7ff824101ea0 2023-07-29T16:12:27.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.997+0000 7ff821ffb700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8100a0810 0x7ff8100a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:27.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.997+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff81c0cbda0 con 0x7ff824101ea0 2023-07-29T16:12:27.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.997+0000 7ff823fff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8100a0810 0x7ff8100a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:27.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.998+0000 7ff823fff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8100a0810 0x7ff8100a2cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff824101370 tx=0x7ff814009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:27.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:27.999+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff81c095a20 con 0x7ff824101ea0 2023-07-29T16:12:28.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: 2023-07-29T16:12:27.753+0000 7f1f37572700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-29T16:12:28.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: 2023-07-29T16:12:27.753+0000 7f1f37572700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-29T16:12:28.023 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:12:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: 2023-07-29T16:12:27.753+0000 7f1f37572700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-07-29T16:12:28.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.243+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7ff824107e50 con 0x7ff8100a0810 2023-07-29T16:12:28.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.296+0000 7ff821ffb700 1 -- 172.21.15.31:0/3227873378 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff824107e50 con 0x7ff8100a0810 2023-07-29T16:12:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8100a0810 msgr2=0x7ff8100a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8100a0810 0x7ff8100a2cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff824101370 tx=0x7ff814009260 comp rx=0 tx=0).stop 2023-07-29T16:12:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 msgr2=0x7ff82419f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 0x7ff82419f880 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00ef30 tx=0x7ff81c00c3a0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 shutdown_connections 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8100a0810 0x7ff8100a2cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff824101ea0 0x7ff82419f880 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff824100320 0x7ff8241982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 --2- 172.21.15.31:0/3227873378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff824101510 0x7ff824198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 >> 172.21.15.31:0/3227873378 conn(0x7ff8240fb920 msgr2=0x7ff8240fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 shutdown_connections 2023-07-29T16:12:28.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.299+0000 7ff82a5ff700 1 -- 172.21.15.31:0/3227873378 wait complete. 2023-07-29T16:12:28.315 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-29T16:12:28.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 -- 172.21.15.31:0/1611738735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54100330 msgr2=0x7f2e54100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 --2- 172.21.15.31:0/1611738735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54100330 0x7f2e54100740 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e440098f0 tx=0x7f2e44009c00 comp rx=0 tx=0).stop 2023-07-29T16:12:28.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 -- 172.21.15.31:0/1611738735 shutdown_connections 2023-07-29T16:12:28.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 --2- 172.21.15.31:0/1611738735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54101eb0 0x7f2e54108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 --2- 172.21.15.31:0/1611738735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54100330 0x7f2e54100740 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 --2- 172.21.15.31:0/1611738735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e54101520 0x7f2e54101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 -- 172.21.15.31:0/1611738735 >> 172.21.15.31:0/1611738735 conn(0x7f2e540fb970 msgr2=0x7f2e540fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 -- 172.21.15.31:0/1611738735 shutdown_connections 2023-07-29T16:12:28.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.525+0000 7f2e599ff700 1 -- 172.21.15.31:0/1611738735 wait complete. 2023-07-29T16:12:28.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.526+0000 7f2e599ff700 1 Processor -- start 2023-07-29T16:12:28.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.526+0000 7f2e599ff700 1 -- start start 2023-07-29T16:12:28.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e599ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 0x7f2e5419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:28.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e599ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e54101520 0x7f2e5419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:28.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e599ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54101eb0 0x7f2e541a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:28.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e599ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e541a4240 con 0x7f2e54100330 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e599ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e541a4380 con 0x7f2e54101eb0 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e599ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e541a4510 con 0x7f2e54101520 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 0x7f2e5419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 0x7f2e5419c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39942/0 (socket says 172.21.15.31:39942) 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 -- 172.21.15.31:0/2954644739 learned_addr learned my addr 172.21.15.31:0/2954644739 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e527fc700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e54101520 0x7f2e5419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e537fe700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54101eb0 0x7f2e541a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 -- 172.21.15.31:0/2954644739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e54101520 msgr2=0x7f2e5419cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e54101520 0x7f2e5419cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 -- 172.21.15.31:0/2954644739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54101eb0 msgr2=0x7f2e541a3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54101eb0 0x7f2e541a3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e52ffd700 1 -- 172.21.15.31:0/2954644739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c009d00 con 0x7f2e54100330 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.527+0000 7f2e537fe700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54101eb0 0x7f2e541a3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.528+0000 7f2e52ffd700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 0x7f2e5419c6c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2e44000c00 tx=0x7f2e4400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:28.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.528+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4401c070 con 0x7f2e54100330 2023-07-29T16:12:28.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.528+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e44009520 con 0x7f2e54100330 2023-07-29T16:12:28.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.528+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e4400f7b0 con 0x7f2e54100330 2023-07-29T16:12:28.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.528+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44003a00 con 0x7f2e54100330 2023-07-29T16:12:28.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.528+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e541a4a20 con 0x7f2e54100330 2023-07-29T16:12:28.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.529+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e54067ad0 con 0x7f2e54100330 2023-07-29T16:12:28.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.531+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2e44020030 con 0x7f2e54100330 2023-07-29T16:12:28.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.532+0000 7f2e589fd700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2e3c0a0770 0x7f2e3c0a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:28.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.533+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2e440cc9d0 con 0x7f2e54100330 2023-07-29T16:12:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.533+0000 7f2e527fc700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2e3c0a0770 0x7f2e3c0a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:28.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.533+0000 7f2e527fc700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2e3c0a0770 0x7f2e3c0a2c20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009c60 tx=0x7f2e48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:28.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.537+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2e440965b0 con 0x7f2e54100330 2023-07-29T16:12:28.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-mon[95504]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 8 op/s 2023-07-29T16:12:28.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2784099071' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-29T16:12:28.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:12:28.682 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:12:28.249+0000 7f340a45a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-29T16:12:28.682 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:12:28.249+0000 7f340a45a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-29T16:12:28.682 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:12:28.249+0000 7f340a45a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-29T16:12:28.683 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:12:28.249+0000 7f340a45a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-29T16:12:28.683 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:12:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:12:28.249+0000 7f340a45a700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-29T16:12:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:28 smithi039 ceph-mon[128858]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 8 op/s 2023-07-29T16:12:28.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:28 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2784099071' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-29T16:12:28.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:28 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:12:28.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.781+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e54061190 con 0x7f2e3c0a0770 2023-07-29T16:12:28.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.786+0000 7f2e589fd700 1 -- 172.21.15.31:0/2954644739 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f2e54061190 con 0x7f2e3c0a0770 2023-07-29T16:12:28.782 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/sv_0/b1bcdf03-ee47-42d0-9d57-0128f8823201 2023-07-29T16:12:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2e3c0a0770 msgr2=0x7f2e3c0a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2e3c0a0770 0x7f2e3c0a2c20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009c60 tx=0x7f2e48009260 comp rx=0 tx=0).stop 2023-07-29T16:12:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 msgr2=0x7f2e5419c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 0x7f2e5419c6c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2e44000c00 tx=0x7f2e4400b730 comp rx=0 tx=0).stop 2023-07-29T16:12:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 shutdown_connections 2023-07-29T16:12:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2e3c0a0770 0x7f2e3c0a2c20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e54101eb0 0x7f2e541a3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2e54100330 0x7f2e5419c6c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 --2- 172.21.15.31:0/2954644739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e54101520 0x7f2e5419cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 >> 172.21.15.31:0/2954644739 conn(0x7f2e540fb970 msgr2=0x7f2e540fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 shutdown_connections 2023-07-29T16:12:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:28.788+0000 7f2e599ff700 1 -- 172.21.15.31:0/2954644739 wait complete. 2023-07-29T16:12:28.803 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-29T16:12:28.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:28 smithi156 ceph-mon[129245]: pgmap v275: 97 pgs: 97 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 8 op/s 2023-07-29T16:12:28.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:28 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2784099071' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-29T16:12:28.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:28 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:12:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 -- 172.21.15.31:0/1338004620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10100330 msgr2=0x7fda10100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 --2- 172.21.15.31:0/1338004620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10100330 0x7fda10100740 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fda0c009990 tx=0x7fda0c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 -- 172.21.15.31:0/1338004620 shutdown_connections 2023-07-29T16:12:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 --2- 172.21.15.31:0/1338004620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda10101eb0 0x7fda10108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 --2- 172.21.15.31:0/1338004620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10100330 0x7fda10100740 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 --2- 172.21.15.31:0/1338004620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10101520 0x7fda10101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 -- 172.21.15.31:0/1338004620 >> 172.21.15.31:0/1338004620 conn(0x7fda100fb970 msgr2=0x7fda100fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 -- 172.21.15.31:0/1338004620 shutdown_connections 2023-07-29T16:12:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.014+0000 7fda181ff700 1 -- 172.21.15.31:0/1338004620 wait complete. 2023-07-29T16:12:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.015+0000 7fda181ff700 1 Processor -- start 2023-07-29T16:12:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 -- start start 2023-07-29T16:12:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 0x7fda10074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10101520 0x7fda10073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda10101eb0 0x7fda10073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda10073e20 con 0x7fda10100330 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda10073f90 con 0x7fda10101520 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda181ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda101a00f0 con 0x7fda10101eb0 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda15f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 0x7fda10074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda15f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 0x7fda10074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39950/0 (socket says 172.21.15.31:39950) 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda15f9b700 1 -- 172.21.15.31:0/891352806 learned_addr learned my addr 172.21.15.31:0/891352806 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda1579a700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10101520 0x7fda10073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.016+0000 7fda1679c700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda10101eb0 0x7fda10073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda1579a700 1 -- 172.21.15.31:0/891352806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda10101eb0 msgr2=0x7fda10073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda1579a700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda10101eb0 0x7fda10073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda1579a700 1 -- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 msgr2=0x7fda10074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda1579a700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 0x7fda10074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda1579a700 1 -- 172.21.15.31:0/891352806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00009520 con 0x7fda10101520 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda15f9b700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 0x7fda10074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda1579a700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10101520 0x7fda10073140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e9f0 tx=0x7fda0000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0000cb70 con 0x7fda10101520 2023-07-29T16:12:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda0000ee90 con 0x7fda10101520 2023-07-29T16:12:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c0095f0 con 0x7fda10101520 2023-07-29T16:12:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda00003cf0 con 0x7fda10101520 2023-07-29T16:12:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.017+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda101a0750 con 0x7fda10101520 2023-07-29T16:12:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.018+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda101078d0 con 0x7fda10101520 2023-07-29T16:12:29.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.020+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fda0000ccd0 con 0x7fda10101520 2023-07-29T16:12:29.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.022+0000 7fda06ffd700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9fc0a0790 0x7fd9fc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.022+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fda00014070 con 0x7fda10101520 2023-07-29T16:12:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.022+0000 7fda15f9b700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9fc0a0790 0x7fd9fc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.023+0000 7fda15f9b700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9fc0a0790 0x7fd9fc0a2c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00b2e0 tx=0x7fda0c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.024+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fda00095a00 con 0x7fda10101520 2023-07-29T16:12:29.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.264+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fda10061190 con 0x7fd9fc0a0790 2023-07-29T16:12:29.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.294+0000 7fda06ffd700 1 -- 172.21.15.31:0/891352806 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fda10061190 con 0x7fd9fc0a0790 2023-07-29T16:12:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9fc0a0790 msgr2=0x7fd9fc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9fc0a0790 0x7fd9fc0a2c40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00b2e0 tx=0x7fda0c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10101520 msgr2=0x7fda10073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10101520 0x7fda10073140 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda0000e9f0 tx=0x7fda0000ed00 comp rx=0 tx=0).stop 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 shutdown_connections 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9fc0a0790 0x7fd9fc0a2c40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda10101eb0 0x7fda10073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda10100330 0x7fda10074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 --2- 172.21.15.31:0/891352806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda10101520 0x7fda10073140 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 >> 172.21.15.31:0/891352806 conn(0x7fda100fb970 msgr2=0x7fda10104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 shutdown_connections 2023-07-29T16:12:29.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.296+0000 7fda181ff700 1 -- 172.21.15.31:0/891352806 wait complete. 2023-07-29T16:12:29.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-29T16:12:29.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 -- 172.21.15.31:0/3264482993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 msgr2=0x7f44d0100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3264482993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 0x7f44d0100740 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f44cc009990 tx=0x7f44cc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 -- 172.21.15.31:0/3264482993 shutdown_connections 2023-07-29T16:12:29.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3264482993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101eb0 0x7f44d0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3264482993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 0x7f44d0100740 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3264482993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0101520 0x7f44d0101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 -- 172.21.15.31:0/3264482993 >> 172.21.15.31:0/3264482993 conn(0x7f44d00fb970 msgr2=0x7f44d00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:29.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 -- 172.21.15.31:0/3264482993 shutdown_connections 2023-07-29T16:12:29.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.528+0000 7f44d81ff700 1 -- 172.21.15.31:0/3264482993 wait complete. 2023-07-29T16:12:29.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.529+0000 7f44d81ff700 1 Processor -- start 2023-07-29T16:12:29.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.529+0000 7f44d81ff700 1 -- start start 2023-07-29T16:12:29.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.529+0000 7f44d81ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 0x7f44d0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d81ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 0x7f44d0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d81ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0101eb0 0x7f44d019f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d81ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d019fec0 con 0x7f44d0101520 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d81ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01a0000 con 0x7f44d0100330 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d81ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d01a0190 con 0x7f44d0101eb0 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 0x7f44d0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 0x7f44d0198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39956/0 (socket says 172.21.15.31:39956) 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 -- 172.21.15.31:0/3075083591 learned_addr learned my addr 172.21.15.31:0/3075083591 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d5f9b700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 0x7f44d0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d679c700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0101eb0 0x7f44d019f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 -- 172.21.15.31:0/3075083591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0101eb0 msgr2=0x7f44d019f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0101eb0 0x7f44d019f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 -- 172.21.15.31:0/3075083591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 msgr2=0x7f44d0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 0x7f44d0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 -- 172.21.15.31:0/3075083591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8009c40 con 0x7f44d0101520 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.530+0000 7f44d579a700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 0x7f44d0198880 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e9a0 tx=0x7f44c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:29.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.531+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0009790 con 0x7f44d0101520 2023-07-29T16:12:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.531+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44c000ce80 con 0x7f44d0101520 2023-07-29T16:12:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.531+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44cc0095f0 con 0x7f44d0101520 2023-07-29T16:12:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.531+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0018680 con 0x7f44d0101520 2023-07-29T16:12:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.531+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d01a0840 con 0x7f44d0101520 2023-07-29T16:12:29.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.532+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d0192a10 con 0x7f44d0101520 2023-07-29T16:12:29.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.534+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f44c0010430 con 0x7f44d0101520 2023-07-29T16:12:29.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.536+0000 7f44c6ffd700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44bc0a08a0 0x7f44bc0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:29.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.537+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f44c0014070 con 0x7f44d0101520 2023-07-29T16:12:29.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.537+0000 7f44d5f9b700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44bc0a08a0 0x7f44bc0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:29.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.537+0000 7f44d5f9b700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44bc0a08a0 0x7f44bc0a2d50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00b2e0 tx=0x7f44cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:29.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.541+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f44c0051720 con 0x7f44d0101520 2023-07-29T16:12:29.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:29 smithi031 ceph-mon[95504]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:29.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:29 smithi039 ceph-mon[128858]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:29.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.787+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d0061190 con 0x7f44bc0a08a0 2023-07-29T16:12:29.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.791+0000 7f44c6ffd700 1 -- 172.21.15.31:0/3075083591 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f44d0061190 con 0x7f44bc0a08a0 2023-07-29T16:12:29.788 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902 2023-07-29T16:12:29.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.793+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44bc0a08a0 msgr2=0x7f44bc0a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.793+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44bc0a08a0 0x7f44bc0a2d50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00b2e0 tx=0x7f44cc005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.793+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 msgr2=0x7f44d0198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.793+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 0x7f44d0198880 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e9a0 tx=0x7f44c000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.793+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 shutdown_connections 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.793+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44bc0a08a0 0x7f44bc0a2d50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.794+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44d0101eb0 0x7f44d019f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.794+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44d0100330 0x7f44d0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.794+0000 7f44d81ff700 1 --2- 172.21.15.31:0/3075083591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44d0101520 0x7f44d0198880 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.794+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 >> 172.21.15.31:0/3075083591 conn(0x7f44d00fb970 msgr2=0x7f44d0106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.794+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 shutdown_connections 2023-07-29T16:12:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:29.794+0000 7f44d81ff700 1 -- 172.21.15.31:0/3075083591 wait complete. 2023-07-29T16:12:29.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:12:29.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:29 smithi156 ceph-mon[129245]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 -- 172.21.15.31:0/2767663314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284101ec0 msgr2=0x7ff284108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/2767663314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284101ec0 0x7ff284108780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff27c00b0a0 tx=0x7ff27c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 -- 172.21.15.31:0/2767663314 shutdown_connections 2023-07-29T16:12:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/2767663314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284101ec0 0x7ff284108780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/2767663314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff284100340 0x7ff284100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/2767663314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101530 0x7ff284101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.018+0000 7ff28b3ff700 1 -- 172.21.15.31:0/2767663314 >> 172.21.15.31:0/2767663314 conn(0x7ff2840fb920 msgr2=0x7ff2840fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:30.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.019+0000 7ff28b3ff700 1 -- 172.21.15.31:0/2767663314 shutdown_connections 2023-07-29T16:12:30.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.019+0000 7ff28b3ff700 1 -- 172.21.15.31:0/2767663314 wait complete. 2023-07-29T16:12:30.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.019+0000 7ff28b3ff700 1 Processor -- start 2023-07-29T16:12:30.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 -- start start 2023-07-29T16:12:30.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 0x7ff284074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff284101530 0x7ff284073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101ec0 0x7ff284073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff284073e20 con 0x7ff284100340 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff284073f90 con 0x7ff284101ec0 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28b3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2841a01a0 con 0x7ff284101530 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28919b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 0x7ff284074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28919b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 0x7ff284074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42032/0 (socket says 172.21.15.31:42032) 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28919b700 1 -- 172.21.15.31:0/3797317140 learned_addr learned my addr 172.21.15.31:0/3797317140 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28999c700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101ec0 0x7ff284073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.020+0000 7ff28899a700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff284101530 0x7ff284073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28919b700 1 -- 172.21.15.31:0/3797317140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff284101530 msgr2=0x7ff284073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28919b700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff284101530 0x7ff284073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28919b700 1 -- 172.21.15.31:0/3797317140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101ec0 msgr2=0x7ff284073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28919b700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101ec0 0x7ff284073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28919b700 1 -- 172.21.15.31:0/3797317140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c009d00 con 0x7ff284100340 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28999c700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101ec0 0x7ff284073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28919b700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 0x7ff284074b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff28000e970 tx=0x7ff28000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280009790 con 0x7ff284100340 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2841a0480 con 0x7ff284100340 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff28000c7e0 con 0x7ff284100340 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff280003c70 con 0x7ff284100340 2023-07-29T16:12:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.021+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2841a0960 con 0x7ff284100340 2023-07-29T16:12:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.022+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff284107800 con 0x7ff284100340 2023-07-29T16:12:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.024+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7ff28000cc50 con 0x7ff284100340 2023-07-29T16:12:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.026+0000 7ff27a7fc700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff2700a0860 0x7ff2700a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.026+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff280014070 con 0x7ff284100340 2023-07-29T16:12:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.026+0000 7ff28899a700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff2700a0860 0x7ff2700a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.026+0000 7ff28899a700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff2700a0860 0x7ff2700a2d10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff274009c60 tx=0x7ff274009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.030+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff2800957c0 con 0x7ff284100340 2023-07-29T16:12:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.286+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff28402cec0 con 0x7ff284100340 2023-07-29T16:12:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.287+0000 7ff27a7fc700 1 -- 172.21.15.31:0/3797317140 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7ff280003dd0 con 0x7ff284100340 2023-07-29T16:12:30.284 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:30.284 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:12:30.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff2700a0860 msgr2=0x7ff2700a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff2700a0860 0x7ff2700a2d10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff274009c60 tx=0x7ff274009260 comp rx=0 tx=0).stop 2023-07-29T16:12:30.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 msgr2=0x7ff284074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 0x7ff284074b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff28000e970 tx=0x7ff28000ec80 comp rx=0 tx=0).stop 2023-07-29T16:12:30.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 shutdown_connections 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff2700a0860 0x7ff2700a2d10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff284101ec0 0x7ff284073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff284100340 0x7ff284074b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 --2- 172.21.15.31:0/3797317140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff284101530 0x7ff284073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 >> 172.21.15.31:0/3797317140 conn(0x7ff2840fb920 msgr2=0x7ff284104940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 shutdown_connections 2023-07-29T16:12:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.289+0000 7ff28b3ff700 1 -- 172.21.15.31:0/3797317140 wait complete. 2023-07-29T16:12:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:12:30.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:12:30.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 -- 172.21.15.31:0/730744217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4100340 msgr2=0x7f33a4100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/730744217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4100340 0x7f33a4100750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009990 tx=0x7f33a0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 -- 172.21.15.31:0/730744217 shutdown_connections 2023-07-29T16:12:30.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/730744217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33a4101ec0 0x7f33a4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/730744217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4100340 0x7f33a4100750 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/730744217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a4101530 0x7f33a4101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 -- 172.21.15.31:0/730744217 >> 172.21.15.31:0/730744217 conn(0x7f33a40fb920 msgr2=0x7f33a40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:30.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 -- 172.21.15.31:0/730744217 shutdown_connections 2023-07-29T16:12:30.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.513+0000 7f33ab7ff700 1 -- 172.21.15.31:0/730744217 wait complete. 2023-07-29T16:12:30.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.514+0000 7f33ab7ff700 1 Processor -- start 2023-07-29T16:12:30.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.514+0000 7f33ab7ff700 1 -- start start 2023-07-29T16:12:30.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33ab7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a4100340 0x7f33a4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33ab7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 0x7f33a4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33ab7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33a4101ec0 0x7f33a4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33ab7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4073e20 con 0x7f33a4101530 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33ab7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4073f90 con 0x7f33a4100340 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33ab7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a41a00e0 con 0x7f33a4101ec0 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a959b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a4100340 0x7f33a4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 0x7f33a4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a9d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33a4101ec0 0x7f33a4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 0x7f33a4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42054/0 (socket says 172.21.15.31:42054) 2023-07-29T16:12:30.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 -- 172.21.15.31:0/465539526 learned_addr learned my addr 172.21.15.31:0/465539526 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 -- 172.21.15.31:0/465539526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33a4101ec0 msgr2=0x7f33a4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33a4101ec0 0x7f33a4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 -- 172.21.15.31:0/465539526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a4100340 msgr2=0x7f33a4074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a4100340 0x7f33a4074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.515+0000 7f33a8d9a700 1 -- 172.21.15.31:0/465539526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a00095f0 con 0x7f33a4101530 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.516+0000 7f33a8d9a700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 0x7f33a4073140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f339400e9a0 tx=0x7f339400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:30.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.516+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394009790 con 0x7f33a4101530 2023-07-29T16:12:30.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.516+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3394010430 con 0x7f33a4101530 2023-07-29T16:12:30.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.516+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394018680 con 0x7f33a4101530 2023-07-29T16:12:30.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.516+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a41a03c0 con 0x7f33a4101530 2023-07-29T16:12:30.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.516+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a41a0880 con 0x7f33a4101530 2023-07-29T16:12:30.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.517+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a410bb60 con 0x7f33a4101530 2023-07-29T16:12:30.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.519+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f33940105a0 con 0x7f33a4101530 2023-07-29T16:12:30.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.520+0000 7f339a7fc700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33900a08e0 0x7f33900a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.520+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3394014070 con 0x7f33a4101530 2023-07-29T16:12:30.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.520+0000 7f33a959b700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33900a08e0 0x7f33900a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.521+0000 7f33a959b700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33900a08e0 0x7f33900a2d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33a000b2e0 tx=0x7f33a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.524+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3394051720 con 0x7f33a4101530 2023-07-29T16:12:30.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:30 smithi031 ceph-mon[95504]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:30 smithi031 ceph-mon[95504]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:30 smithi031 ceph-mon[95504]: pgmap v276: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4.8 KiB/s wr, 19 op/s 2023-07-29T16:12:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:30 smithi031 ceph-mon[95504]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-29T16:12:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:30 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3797317140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:30.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.735+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33a41071e0 con 0x7f33a4101530 2023-07-29T16:12:30.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.736+0000 7f339a7fc700 1 -- 172.21.15.31:0/465539526 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22090 (secure 0 0 0) 0x7f33940957a0 con 0x7f33a4101530 2023-07-29T16:12:30.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:30.734 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:22.914308+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:12:30.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.738+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33900a08e0 msgr2=0x7f33900a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.738+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33900a08e0 0x7f33900a2d90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f33a000b2e0 tx=0x7f33a0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 msgr2=0x7f33a4073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 0x7f33a4073140 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f339400e9a0 tx=0x7f339400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 shutdown_connections 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33900a08e0 0x7f33900a2d90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33a4101ec0 0x7f33a4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33a4100340 0x7f33a4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 --2- 172.21.15.31:0/465539526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33a4101530 0x7f33a4073140 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 >> 172.21.15.31:0/465539526 conn(0x7f33a40fb920 msgr2=0x7f33a40fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:30.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 shutdown_connections 2023-07-29T16:12:30.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.739+0000 7f33ab7ff700 1 -- 172.21.15.31:0/465539526 wait complete. 2023-07-29T16:12:30.754 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-29T16:12:30.755 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-29T16:12:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:30 smithi039 ceph-mon[128858]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:30 smithi039 ceph-mon[128858]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:30 smithi039 ceph-mon[128858]: pgmap v276: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4.8 KiB/s wr, 19 op/s 2023-07-29T16:12:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:30 smithi039 ceph-mon[128858]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-29T16:12:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:30 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3797317140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:30 smithi156 ceph-mon[129245]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:30 smithi156 ceph-mon[129245]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:30 smithi156 ceph-mon[129245]: pgmap v276: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4.8 KiB/s wr, 19 op/s 2023-07-29T16:12:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:30 smithi156 ceph-mon[129245]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-29T16:12:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:30 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3797317140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:12:30.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 -- 172.21.15.31:0/419926506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba8074d90 msgr2=0x7f7ba80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 --2- 172.21.15.31:0/419926506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba8074d90 0x7f7ba80731e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009990 tx=0x7f7b98009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 -- 172.21.15.31:0/419926506 shutdown_connections 2023-07-29T16:12:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 --2- 172.21.15.31:0/419926506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 0x7f7ba810a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 --2- 172.21.15.31:0/419926506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba8074d90 0x7f7ba80731e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 --2- 172.21.15.31:0/419926506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba80737b0 0x7f7ba8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 -- 172.21.15.31:0/419926506 >> 172.21.15.31:0/419926506 conn(0x7f7ba80fb920 msgr2=0x7f7ba80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:30.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 -- 172.21.15.31:0/419926506 shutdown_connections 2023-07-29T16:12:30.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.962+0000 7f7bacfff700 1 -- 172.21.15.31:0/419926506 wait complete. 2023-07-29T16:12:30.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 Processor -- start 2023-07-29T16:12:30.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 -- start start 2023-07-29T16:12:30.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 0x7f7ba81982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba8074d90 0x7f7ba81987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 0x7f7ba819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba819fe20 con 0x7f7ba80737b0 2023-07-29T16:12:30.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba819ff60 con 0x7f7ba8108200 2023-07-29T16:12:30.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7bacfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba81a00f0 con 0x7f7ba8074d90 2023-07-29T16:12:30.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7ba659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 0x7f7ba81982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7ba659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 0x7f7ba81982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42084/0 (socket says 172.21.15.31:42084) 2023-07-29T16:12:30.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7ba5d9b700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba8074d90 0x7f7ba81987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.963+0000 7f7ba659c700 1 -- 172.21.15.31:0/758392255 learned_addr learned my addr 172.21.15.31:0/758392255 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba6d9d700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 0x7f7ba819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba659c700 1 -- 172.21.15.31:0/758392255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba8074d90 msgr2=0x7f7ba81987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba659c700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba8074d90 0x7f7ba81987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba659c700 1 -- 172.21.15.31:0/758392255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 msgr2=0x7f7ba819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba659c700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 0x7f7ba819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:30.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba659c700 1 -- 172.21.15.31:0/758392255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980095f0 con 0x7f7ba80737b0 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba6d9d700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 0x7f7ba819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba5d9b700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba8074d90 0x7f7ba81987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7ba659c700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 0x7f7ba81982a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7b980098e0 tx=0x7f7b98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9801d070 con 0x7f7ba80737b0 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b980054f0 con 0x7f7ba80737b0 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba81a0370 con 0x7f7ba80737b0 2023-07-29T16:12:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.964+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98004ca0 con 0x7f7ba80737b0 2023-07-29T16:12:30.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.965+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba81a0790 con 0x7f7ba80737b0 2023-07-29T16:12:30.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.966+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba81068b0 con 0x7f7ba80737b0 2023-07-29T16:12:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.967+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f7b9800fc70 con 0x7f7ba80737b0 2023-07-29T16:12:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.969+0000 7f7b977fe700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b900a08e0 0x7f7b900a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.969+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7b980cce30 con 0x7f7ba80737b0 2023-07-29T16:12:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.969+0000 7f7ba5d9b700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b900a08e0 0x7f7b900a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.970+0000 7f7ba5d9b700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b900a08e0 0x7f7b900a2d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8074ab0 tx=0x7f7b9c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:30.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:30.972+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7b9805a720 con 0x7f7ba80737b0 2023-07-29T16:12:31.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.180+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7ba8061190 con 0x7f7b900a08e0 2023-07-29T16:12:31.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.193+0000 7f7b977fe700 1 -- 172.21.15.31:0/758392255 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335746 (secure 0 0 0) 0x7f7ba8061190 con 0x7f7b900a08e0 2023-07-29T16:12:31.194 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:31.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b900a08e0 msgr2=0x7f7b900a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b900a08e0 0x7f7b900a2d90 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8074ab0 tx=0x7f7b9c007400 comp rx=0 tx=0).stop 2023-07-29T16:12:31.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 msgr2=0x7f7ba81982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 0x7f7ba81982a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7b980098e0 tx=0x7f7b98004730 comp rx=0 tx=0).stop 2023-07-29T16:12:31.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 shutdown_connections 2023-07-29T16:12:31.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b900a08e0 0x7f7b900a2d90 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ba8108200 0x7f7ba819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ba80737b0 0x7f7ba81982a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 --2- 172.21.15.31:0/758392255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ba8074d90 0x7f7ba81987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 >> 172.21.15.31:0/758392255 conn(0x7f7ba80fb920 msgr2=0x7f7ba80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:31.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 shutdown_connections 2023-07-29T16:12:31.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.197+0000 7f7bacfff700 1 -- 172.21.15.31:0/758392255 wait complete. 2023-07-29T16:12:31.198 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:12:31.213 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2023-07-29T16:12:29.272136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595446,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":155,"num_read_kb":134,"num_write":305,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336188,"kb_used_data":12764,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124605636,"statfs":{"total":1151940427776,"available":1151596171264,"internally_reserved":0,"allocated":13070336,"data_stored":7080967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,7],"upper_bound":8},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5078,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":29,"num_read_kb":25,"num_write":124,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.944144"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928127+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:12:22.928127+0000","last_peered":"2023-07-29T16:12:22.928127+0000","last_clean":"2023-07-29T16:12:22.928127+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:12:22.928127+0000","last_undegraded":"2023-07-29T16:12:22.928127+0000","last_fullsized":"2023-07-29T16:12:22.928127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715831+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:12:23.715831+0000","last_peered":"2023-07-29T16:12:23.715831+0000","last_clean":"2023-07-29T16:12:23.715831+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:12:23.715831+0000","last_undegraded":"2023-07-29T16:12:23.715831+0000","last_fullsized":"2023-07-29T16:12:23.715831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715823+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:12:23.715823+0000","last_peered":"2023-07-29T16:12:23.715823+0000","last_clean":"2023-07-29T16:12:23.715823+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:12:23.715823+0000","last_undegraded":"2023-07-29T16:12:23.715823+0000","last_fullsized":"2023-07-29T16:12:23.715823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927143+0000","last_change":"2023-07-29T16:12:21.943788+0000","last_active":"2023-07-29T16:12:22.927143+0000","last_peered":"2023-07-29T16:12:22.927143+0000","last_clean":"2023-07-29T16:12:22.927143+0000","last_became_active":"2023-07-29T16:12:21.943292+0000","last_became_peered":"2023-07-29T16:12:21.943292+0000","last_unstale":"2023-07-29T16:12:22.927143+0000","last_undegraded":"2023-07-29T16:12:22.927143+0000","last_fullsized":"2023-07-29T16:12:22.927143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:34:18.994175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464359+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:12:22.464359+0000","last_peered":"2023-07-29T16:12:22.464359+0000","last_clean":"2023-07-29T16:12:22.464359+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:12:22.464359+0000","last_undegraded":"2023-07-29T16:12:22.464359+0000","last_fullsized":"2023-07-29T16:12:22.464359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674941+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:12:22.674941+0000","last_peered":"2023-07-29T16:12:22.674941+0000","last_clean":"2023-07-29T16:12:22.674941+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:12:22.674941+0000","last_undegraded":"2023-07-29T16:12:22.674941+0000","last_fullsized":"2023-07-29T16:12:22.674941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.946618+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:12:21.946618+0000","last_peered":"2023-07-29T16:12:21.946618+0000","last_clean":"2023-07-29T16:12:21.946618+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:12:21.946618+0000","last_undegraded":"2023-07-29T16:12:21.946618+0000","last_fullsized":"2023-07-29T16:12:21.946618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918534+0000","last_change":"2023-07-29T16:12:21.941714+0000","last_active":"2023-07-29T16:12:22.918534+0000","last_peered":"2023-07-29T16:12:22.918534+0000","last_clean":"2023-07-29T16:12:22.918534+0000","last_became_active":"2023-07-29T16:12:21.941403+0000","last_became_peered":"2023-07-29T16:12:21.941403+0000","last_unstale":"2023-07-29T16:12:22.918534+0000","last_undegraded":"2023-07-29T16:12:22.918534+0000","last_fullsized":"2023-07-29T16:12:22.918534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:07:09.470922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652666+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:12:22.652666+0000","last_peered":"2023-07-29T16:12:22.652666+0000","last_clean":"2023-07-29T16:12:22.652666+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:12:22.652666+0000","last_undegraded":"2023-07-29T16:12:22.652666+0000","last_fullsized":"2023-07-29T16:12:22.652666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674987+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:12:22.674987+0000","last_peered":"2023-07-29T16:12:22.674987+0000","last_clean":"2023-07-29T16:12:22.674987+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:12:22.674987+0000","last_undegraded":"2023-07-29T16:12:22.674987+0000","last_fullsized":"2023-07-29T16:12:22.674987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.948821+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:12:21.948821+0000","last_peered":"2023-07-29T16:12:21.948821+0000","last_clean":"2023-07-29T16:12:21.948821+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:12:21.948821+0000","last_undegraded":"2023-07-29T16:12:21.948821+0000","last_fullsized":"2023-07-29T16:12:21.948821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927028+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:12:22.927028+0000","last_peered":"2023-07-29T16:12:22.927028+0000","last_clean":"2023-07-29T16:12:22.927028+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:12:22.927028+0000","last_undegraded":"2023-07-29T16:12:22.927028+0000","last_fullsized":"2023-07-29T16:12:22.927028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841081+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:12:23.841081+0000","last_peered":"2023-07-29T16:12:23.841081+0000","last_clean":"2023-07-29T16:12:23.841081+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:12:23.841081+0000","last_undegraded":"2023-07-29T16:12:23.841081+0000","last_fullsized":"2023-07-29T16:12:23.841081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715803+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:12:23.715803+0000","last_peered":"2023-07-29T16:12:23.715803+0000","last_clean":"2023-07-29T16:12:23.715803+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:12:23.715803+0000","last_undegraded":"2023-07-29T16:12:23.715803+0000","last_fullsized":"2023-07-29T16:12:23.715803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715784+0000","last_change":"2023-07-29T16:12:21.940409+0000","last_active":"2023-07-29T16:12:23.715784+0000","last_peered":"2023-07-29T16:12:23.715784+0000","last_clean":"2023-07-29T16:12:23.715784+0000","last_became_active":"2023-07-29T16:12:21.939995+0000","last_became_peered":"2023-07-29T16:12:21.939995+0000","last_unstale":"2023-07-29T16:12:23.715784+0000","last_undegraded":"2023-07-29T16:12:23.715784+0000","last_fullsized":"2023-07-29T16:12:23.715784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:38:45.973435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652508+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:12:22.652508+0000","last_peered":"2023-07-29T16:12:22.652508+0000","last_clean":"2023-07-29T16:12:22.652508+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:12:22.652508+0000","last_undegraded":"2023-07-29T16:12:22.652508+0000","last_fullsized":"2023-07-29T16:12:22.652508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928213+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:12:22.928213+0000","last_peered":"2023-07-29T16:12:22.928213+0000","last_clean":"2023-07-29T16:12:22.928213+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:12:22.928213+0000","last_undegraded":"2023-07-29T16:12:22.928213+0000","last_fullsized":"2023-07-29T16:12:22.928213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.792116+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:12:23.792116+0000","last_peered":"2023-07-29T16:12:23.792116+0000","last_clean":"2023-07-29T16:12:23.792116+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:12:23.792116+0000","last_undegraded":"2023-07-29T16:12:23.792116+0000","last_fullsized":"2023-07-29T16:12:23.792116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514860+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:12:23.514860+0000","last_peered":"2023-07-29T16:12:23.514860+0000","last_clean":"2023-07-29T16:12:23.514860+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:12:23.514860+0000","last_undegraded":"2023-07-29T16:12:23.514860+0000","last_fullsized":"2023-07-29T16:12:23.514860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493366+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:12:26.493366+0000","last_peered":"2023-07-29T16:12:26.493366+0000","last_clean":"2023-07-29T16:12:26.493366+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:12:26.493366+0000","last_undegraded":"2023-07-29T16:12:26.493366+0000","last_fullsized":"2023-07-29T16:12:26.493366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918853+0000","last_change":"2023-07-29T16:12:21.930788+0000","last_active":"2023-07-29T16:12:22.918853+0000","last_peered":"2023-07-29T16:12:22.918853+0000","last_clean":"2023-07-29T16:12:22.918853+0000","last_became_active":"2023-07-29T16:12:21.930448+0000","last_became_peered":"2023-07-29T16:12:21.930448+0000","last_unstale":"2023-07-29T16:12:22.918853+0000","last_undegraded":"2023-07-29T16:12:22.918853+0000","last_fullsized":"2023-07-29T16:12:22.918853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:28:51.735231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:24.190099+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:12:24.190099+0000","last_peered":"2023-07-29T16:12:24.190099+0000","last_clean":"2023-07-29T16:12:24.190099+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:12:24.190099+0000","last_undegraded":"2023-07-29T16:12:24.190099+0000","last_fullsized":"2023-07-29T16:12:24.190099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715872+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:12:23.715872+0000","last_peered":"2023-07-29T16:12:23.715872+0000","last_clean":"2023-07-29T16:12:23.715872+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:12:23.715872+0000","last_undegraded":"2023-07-29T16:12:23.715872+0000","last_fullsized":"2023-07-29T16:12:23.715872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.491458+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:12:26.491458+0000","last_peered":"2023-07-29T16:12:26.491458+0000","last_clean":"2023-07-29T16:12:26.491458+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:12:26.491458+0000","last_undegraded":"2023-07-29T16:12:26.491458+0000","last_fullsized":"2023-07-29T16:12:26.491458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652606+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:12:22.652606+0000","last_peered":"2023-07-29T16:12:22.652606+0000","last_clean":"2023-07-29T16:12:22.652606+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:12:22.652606+0000","last_undegraded":"2023-07-29T16:12:22.652606+0000","last_fullsized":"2023-07-29T16:12:22.652606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464392+0000","last_change":"2023-07-29T16:12:21.348621+0000","last_active":"2023-07-29T16:12:22.464392+0000","last_peered":"2023-07-29T16:12:22.464392+0000","last_clean":"2023-07-29T16:12:22.464392+0000","last_became_active":"2023-07-29T16:12:21.348302+0000","last_became_peered":"2023-07-29T16:12:21.348302+0000","last_unstale":"2023-07-29T16:12:22.464392+0000","last_undegraded":"2023-07-29T16:12:22.464392+0000","last_fullsized":"2023-07-29T16:12:22.464392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:21:32.671912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290898+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:12:22.290898+0000","last_peered":"2023-07-29T16:12:22.290898+0000","last_clean":"2023-07-29T16:12:22.290898+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:12:22.290898+0000","last_undegraded":"2023-07-29T16:12:22.290898+0000","last_fullsized":"2023-07-29T16:12:22.290898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.743054+0000","last_change":"2023-07-29T16:12:21.345470+0000","last_active":"2023-07-29T16:12:26.743054+0000","last_peered":"2023-07-29T16:12:26.743054+0000","last_clean":"2023-07-29T16:12:26.743054+0000","last_became_active":"2023-07-29T16:12:21.344787+0000","last_became_peered":"2023-07-29T16:12:21.344787+0000","last_unstale":"2023-07-29T16:12:26.743054+0000","last_undegraded":"2023-07-29T16:12:26.743054+0000","last_fullsized":"2023-07-29T16:12:26.743054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:31:19.451396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464348+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:12:22.464348+0000","last_peered":"2023-07-29T16:12:22.464348+0000","last_clean":"2023-07-29T16:12:22.464348+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:12:22.464348+0000","last_undegraded":"2023-07-29T16:12:22.464348+0000","last_fullsized":"2023-07-29T16:12:22.464348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.492930+0000","last_change":"2023-07-29T16:12:21.346661+0000","last_active":"2023-07-29T16:12:26.492930+0000","last_peered":"2023-07-29T16:12:26.492930+0000","last_clean":"2023-07-29T16:12:26.492930+0000","last_became_active":"2023-07-29T16:12:21.346202+0000","last_became_peered":"2023-07-29T16:12:21.346202+0000","last_unstale":"2023-07-29T16:12:26.492930+0000","last_undegraded":"2023-07-29T16:12:26.492930+0000","last_fullsized":"2023-07-29T16:12:26.492930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:11:26.080842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290874+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:12:22.290874+0000","last_peered":"2023-07-29T16:12:22.290874+0000","last_clean":"2023-07-29T16:12:22.290874+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:12:22.290874+0000","last_undegraded":"2023-07-29T16:12:22.290874+0000","last_fullsized":"2023-07-29T16:12:22.290874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.492569+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:12:26.492569+0000","last_peered":"2023-07-29T16:12:26.492569+0000","last_clean":"2023-07-29T16:12:26.492569+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:12:26.492569+0000","last_undegraded":"2023-07-29T16:12:26.492569+0000","last_fullsized":"2023-07-29T16:12:26.492569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918628+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:12:22.918628+0000","last_peered":"2023-07-29T16:12:22.918628+0000","last_clean":"2023-07-29T16:12:22.918628+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:12:22.918628+0000","last_undegraded":"2023-07-29T16:12:22.918628+0000","last_fullsized":"2023-07-29T16:12:22.918628+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.917948+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:12:22.917948+0000","last_peered":"2023-07-29T16:12:22.917948+0000","last_clean":"2023-07-29T16:12:22.917948+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:12:22.917948+0000","last_undegraded":"2023-07-29T16:12:22.917948+0000","last_fullsized":"2023-07-29T16:12:22.917948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.792052+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:12:23.792052+0000","last_peered":"2023-07-29T16:12:23.792052+0000","last_clean":"2023-07-29T16:12:23.792052+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:12:23.792052+0000","last_undegraded":"2023-07-29T16:12:23.792052+0000","last_fullsized":"2023-07-29T16:12:23.792052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476200+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:12:25.476200+0000","last_peered":"2023-07-29T16:12:25.476200+0000","last_clean":"2023-07-29T16:12:25.476200+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:12:25.476200+0000","last_undegraded":"2023-07-29T16:12:25.476200+0000","last_fullsized":"2023-07-29T16:12:25.476200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652649+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:12:22.652649+0000","last_peered":"2023-07-29T16:12:22.652649+0000","last_clean":"2023-07-29T16:12:22.652649+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:12:22.652649+0000","last_undegraded":"2023-07-29T16:12:22.652649+0000","last_fullsized":"2023-07-29T16:12:22.652649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.474802+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:25.474802+0000","last_peered":"2023-07-29T16:12:25.474802+0000","last_clean":"2023-07-29T16:12:25.474802+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:12:25.474802+0000","last_undegraded":"2023-07-29T16:12:25.474802+0000","last_fullsized":"2023-07-29T16:12:25.474802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715863+0000","last_change":"2023-07-29T16:12:21.940714+0000","last_active":"2023-07-29T16:12:23.715863+0000","last_peered":"2023-07-29T16:12:23.715863+0000","last_clean":"2023-07-29T16:12:23.715863+0000","last_became_active":"2023-07-29T16:12:21.940414+0000","last_became_peered":"2023-07-29T16:12:21.940414+0000","last_unstale":"2023-07-29T16:12:23.715863+0000","last_undegraded":"2023-07-29T16:12:23.715863+0000","last_fullsized":"2023-07-29T16:12:23.715863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:53:12.841824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476490+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:12:25.476490+0000","last_peered":"2023-07-29T16:12:25.476490+0000","last_clean":"2023-07-29T16:12:25.476490+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:12:25.476490+0000","last_undegraded":"2023-07-29T16:12:25.476490+0000","last_fullsized":"2023-07-29T16:12:25.476490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652626+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:12:22.652626+0000","last_peered":"2023-07-29T16:12:22.652626+0000","last_clean":"2023-07-29T16:12:22.652626+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:12:22.652626+0000","last_undegraded":"2023-07-29T16:12:22.652626+0000","last_fullsized":"2023-07-29T16:12:22.652626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.743706+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:12:26.743706+0000","last_peered":"2023-07-29T16:12:26.743706+0000","last_clean":"2023-07-29T16:12:26.743706+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:12:26.743706+0000","last_undegraded":"2023-07-29T16:12:26.743706+0000","last_fullsized":"2023-07-29T16:12:26.743706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464407+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:12:22.464407+0000","last_peered":"2023-07-29T16:12:22.464407+0000","last_clean":"2023-07-29T16:12:22.464407+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:12:22.464407+0000","last_undegraded":"2023-07-29T16:12:22.464407+0000","last_fullsized":"2023-07-29T16:12:22.464407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493696+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:12:26.493696+0000","last_peered":"2023-07-29T16:12:26.493696+0000","last_clean":"2023-07-29T16:12:26.493696+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:12:26.493696+0000","last_undegraded":"2023-07-29T16:12:26.493696+0000","last_fullsized":"2023-07-29T16:12:26.493696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841072+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:12:23.841072+0000","last_peered":"2023-07-29T16:12:23.841072+0000","last_clean":"2023-07-29T16:12:23.841072+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:12:23.841072+0000","last_undegraded":"2023-07-29T16:12:23.841072+0000","last_fullsized":"2023-07-29T16:12:23.841072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476121+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:12:25.476121+0000","last_peered":"2023-07-29T16:12:25.476121+0000","last_clean":"2023-07-29T16:12:25.476121+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:12:25.476121+0000","last_undegraded":"2023-07-29T16:12:25.476121+0000","last_fullsized":"2023-07-29T16:12:25.476121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.950273+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:12:21.950273+0000","last_peered":"2023-07-29T16:12:21.950273+0000","last_clean":"2023-07-29T16:12:21.950273+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:12:21.950273+0000","last_undegraded":"2023-07-29T16:12:21.950273+0000","last_fullsized":"2023-07-29T16:12:21.950273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.745293+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:12:26.745293+0000","last_peered":"2023-07-29T16:12:26.745293+0000","last_clean":"2023-07-29T16:12:26.745293+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:12:26.745293+0000","last_undegraded":"2023-07-29T16:12:26.745293+0000","last_fullsized":"2023-07-29T16:12:26.745293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.950230+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:12:21.950230+0000","last_peered":"2023-07-29T16:12:21.950230+0000","last_clean":"2023-07-29T16:12:21.950230+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:12:21.950230+0000","last_undegraded":"2023-07-29T16:12:21.950230+0000","last_fullsized":"2023-07-29T16:12:21.950230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493657+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:12:26.493657+0000","last_peered":"2023-07-29T16:12:26.493657+0000","last_clean":"2023-07-29T16:12:26.493657+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:12:26.493657+0000","last_undegraded":"2023-07-29T16:12:26.493657+0000","last_fullsized":"2023-07-29T16:12:26.493657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290842+0000","last_change":"2023-07-29T16:12:21.938419+0000","last_active":"2023-07-29T16:12:22.290842+0000","last_peered":"2023-07-29T16:12:22.290842+0000","last_clean":"2023-07-29T16:12:22.290842+0000","last_became_active":"2023-07-29T16:12:21.938029+0000","last_became_peered":"2023-07-29T16:12:21.938029+0000","last_unstale":"2023-07-29T16:12:22.290842+0000","last_undegraded":"2023-07-29T16:12:22.290842+0000","last_fullsized":"2023-07-29T16:12:22.290842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:31:54.166660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841376+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:12:23.841376+0000","last_peered":"2023-07-29T16:12:23.841376+0000","last_clean":"2023-07-29T16:12:23.841376+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:12:23.841376+0000","last_undegraded":"2023-07-29T16:12:23.841376+0000","last_fullsized":"2023-07-29T16:12:23.841376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928108+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:12:22.928108+0000","last_peered":"2023-07-29T16:12:22.928108+0000","last_clean":"2023-07-29T16:12:22.928108+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:12:22.928108+0000","last_undegraded":"2023-07-29T16:12:22.928108+0000","last_fullsized":"2023-07-29T16:12:22.928108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.792086+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:12:23.792086+0000","last_peered":"2023-07-29T16:12:23.792086+0000","last_clean":"2023-07-29T16:12:23.792086+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:12:23.792086+0000","last_undegraded":"2023-07-29T16:12:23.792086+0000","last_fullsized":"2023-07-29T16:12:23.792086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674977+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:12:22.674977+0000","last_peered":"2023-07-29T16:12:22.674977+0000","last_clean":"2023-07-29T16:12:22.674977+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:12:22.674977+0000","last_undegraded":"2023-07-29T16:12:22.674977+0000","last_fullsized":"2023-07-29T16:12:22.674977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841427+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:12:23.841427+0000","last_peered":"2023-07-29T16:12:23.841427+0000","last_clean":"2023-07-29T16:12:23.841427+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:12:23.841427+0000","last_undegraded":"2023-07-29T16:12:23.841427+0000","last_fullsized":"2023-07-29T16:12:23.841427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674954+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:12:22.674954+0000","last_peered":"2023-07-29T16:12:22.674954+0000","last_clean":"2023-07-29T16:12:22.674954+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:12:22.674954+0000","last_undegraded":"2023-07-29T16:12:22.674954+0000","last_fullsized":"2023-07-29T16:12:22.674954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290829+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:12:22.290829+0000","last_peered":"2023-07-29T16:12:22.290829+0000","last_clean":"2023-07-29T16:12:22.290829+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:12:22.290829+0000","last_undegraded":"2023-07-29T16:12:22.290829+0000","last_fullsized":"2023-07-29T16:12:22.290829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:27.746173+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:12:27.746173+0000","last_peered":"2023-07-29T16:12:27.746173+0000","last_clean":"2023-07-29T16:12:27.746173+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:12:27.746173+0000","last_undegraded":"2023-07-29T16:12:27.746173+0000","last_fullsized":"2023-07-29T16:12:27.746173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.919145+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:12:22.919145+0000","last_peered":"2023-07-29T16:12:22.919145+0000","last_clean":"2023-07-29T16:12:22.919145+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:12:22.919145+0000","last_undegraded":"2023-07-29T16:12:22.919145+0000","last_fullsized":"2023-07-29T16:12:22.919145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.740890+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:26.740890+0000","last_peered":"2023-07-29T16:12:26.740890+0000","last_clean":"2023-07-29T16:12:26.740890+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:12:26.740890+0000","last_undegraded":"2023-07-29T16:12:26.740890+0000","last_fullsized":"2023-07-29T16:12:26.740890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":328,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.933093+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:12:21.933093+0000","last_peered":"2023-07-29T16:12:21.933093+0000","last_clean":"2023-07-29T16:12:21.933093+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:12:21.933093+0000","last_undegraded":"2023-07-29T16:12:21.933093+0000","last_fullsized":"2023-07-29T16:12:21.933093+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.842089+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:12:23.842089+0000","last_peered":"2023-07-29T16:12:23.842089+0000","last_clean":"2023-07-29T16:12:23.842089+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:12:23.842089+0000","last_undegraded":"2023-07-29T16:12:23.842089+0000","last_fullsized":"2023-07-29T16:12:23.842089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.489683+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:12:26.489683+0000","last_peered":"2023-07-29T16:12:26.489683+0000","last_clean":"2023-07-29T16:12:26.489683+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:12:26.489683+0000","last_undegraded":"2023-07-29T16:12:26.489683+0000","last_fullsized":"2023-07-29T16:12:26.489683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476161+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:12:25.476161+0000","last_peered":"2023-07-29T16:12:25.476161+0000","last_clean":"2023-07-29T16:12:25.476161+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:12:25.476161+0000","last_undegraded":"2023-07-29T16:12:25.476161+0000","last_fullsized":"2023-07-29T16:12:25.476161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.842163+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:12:23.842163+0000","last_peered":"2023-07-29T16:12:23.842163+0000","last_clean":"2023-07-29T16:12:23.842163+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:12:23.842163+0000","last_undegraded":"2023-07-29T16:12:23.842163+0000","last_fullsized":"2023-07-29T16:12:23.842163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493893+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:12:26.493893+0000","last_peered":"2023-07-29T16:12:26.493893+0000","last_clean":"2023-07-29T16:12:26.493893+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:12:26.493893+0000","last_undegraded":"2023-07-29T16:12:26.493893+0000","last_fullsized":"2023-07-29T16:12:26.493893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.919165+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:12:22.919165+0000","last_peered":"2023-07-29T16:12:22.919165+0000","last_clean":"2023-07-29T16:12:22.919165+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:12:22.919165+0000","last_undegraded":"2023-07-29T16:12:22.919165+0000","last_fullsized":"2023-07-29T16:12:22.919165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.472891+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:12:25.472891+0000","last_peered":"2023-07-29T16:12:25.472891+0000","last_clean":"2023-07-29T16:12:25.472891+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:12:25.472891+0000","last_undegraded":"2023-07-29T16:12:25.472891+0000","last_fullsized":"2023-07-29T16:12:25.472891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715843+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:12:23.715843+0000","last_peered":"2023-07-29T16:12:23.715843+0000","last_clean":"2023-07-29T16:12:23.715843+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:12:23.715843+0000","last_undegraded":"2023-07-29T16:12:23.715843+0000","last_fullsized":"2023-07-29T16:12:23.715843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476339+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:12:25.476339+0000","last_peered":"2023-07-29T16:12:25.476339+0000","last_clean":"2023-07-29T16:12:25.476339+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:12:25.476339+0000","last_undegraded":"2023-07-29T16:12:25.476339+0000","last_fullsized":"2023-07-29T16:12:25.476339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464335+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:12:22.464335+0000","last_peered":"2023-07-29T16:12:22.464335+0000","last_clean":"2023-07-29T16:12:22.464335+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:12:22.464335+0000","last_undegraded":"2023-07-29T16:12:22.464335+0000","last_fullsized":"2023-07-29T16:12:22.464335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.475756+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:12:25.475756+0000","last_peered":"2023-07-29T16:12:25.475756+0000","last_clean":"2023-07-29T16:12:25.475756+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:12:25.475756+0000","last_undegraded":"2023-07-29T16:12:25.475756+0000","last_fullsized":"2023-07-29T16:12:25.475756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927924+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:12:22.927924+0000","last_peered":"2023-07-29T16:12:22.927924+0000","last_clean":"2023-07-29T16:12:22.927924+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:12:22.927924+0000","last_undegraded":"2023-07-29T16:12:22.927924+0000","last_fullsized":"2023-07-29T16:12:22.927924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.840782+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:12:23.840782+0000","last_peered":"2023-07-29T16:12:23.840782+0000","last_clean":"2023-07-29T16:12:23.840782+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:12:23.840782+0000","last_undegraded":"2023-07-29T16:12:23.840782+0000","last_fullsized":"2023-07-29T16:12:23.840782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918824+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:12:22.918824+0000","last_peered":"2023-07-29T16:12:22.918824+0000","last_clean":"2023-07-29T16:12:22.918824+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:12:22.918824+0000","last_undegraded":"2023-07-29T16:12:22.918824+0000","last_fullsized":"2023-07-29T16:12:22.918824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.474110+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:12:25.474110+0000","last_peered":"2023-07-29T16:12:25.474110+0000","last_clean":"2023-07-29T16:12:25.474110+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:12:25.474110+0000","last_undegraded":"2023-07-29T16:12:25.474110+0000","last_fullsized":"2023-07-29T16:12:25.474110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.951714+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:12:21.951714+0000","last_peered":"2023-07-29T16:12:21.951714+0000","last_clean":"2023-07-29T16:12:21.951714+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:12:21.951714+0000","last_undegraded":"2023-07-29T16:12:21.951714+0000","last_fullsized":"2023-07-29T16:12:21.951714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.492838+0000","last_change":"2023-07-29T16:12:21.346949+0000","last_active":"2023-07-29T16:12:26.492838+0000","last_peered":"2023-07-29T16:12:26.492838+0000","last_clean":"2023-07-29T16:12:26.492838+0000","last_became_active":"2023-07-29T16:12:21.346598+0000","last_became_peered":"2023-07-29T16:12:21.346598+0000","last_unstale":"2023-07-29T16:12:26.492838+0000","last_undegraded":"2023-07-29T16:12:26.492838+0000","last_fullsized":"2023-07-29T16:12:26.492838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:54:03.817342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.947524+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:12:21.947524+0000","last_peered":"2023-07-29T16:12:21.947524+0000","last_clean":"2023-07-29T16:12:21.947524+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:12:21.947524+0000","last_undegraded":"2023-07-29T16:12:21.947524+0000","last_fullsized":"2023-07-29T16:12:21.947524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918258+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:12:22.918258+0000","last_peered":"2023-07-29T16:12:22.918258+0000","last_clean":"2023-07-29T16:12:22.918258+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:12:22.918258+0000","last_undegraded":"2023-07-29T16:12:22.918258+0000","last_fullsized":"2023-07-29T16:12:22.918258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.947514+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:12:21.947514+0000","last_peered":"2023-07-29T16:12:21.947514+0000","last_clean":"2023-07-29T16:12:21.947514+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:12:21.947514+0000","last_undegraded":"2023-07-29T16:12:21.947514+0000","last_fullsized":"2023-07-29T16:12:21.947514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841165+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:12:23.841165+0000","last_peered":"2023-07-29T16:12:23.841165+0000","last_clean":"2023-07-29T16:12:23.841165+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:12:23.841165+0000","last_undegraded":"2023-07-29T16:12:23.841165+0000","last_fullsized":"2023-07-29T16:12:23.841165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.945580+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:12:21.945580+0000","last_peered":"2023-07-29T16:12:21.945580+0000","last_clean":"2023-07-29T16:12:21.945580+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:12:21.945580+0000","last_undegraded":"2023-07-29T16:12:21.945580+0000","last_fullsized":"2023-07-29T16:12:21.945580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927995+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:12:22.927995+0000","last_peered":"2023-07-29T16:12:22.927995+0000","last_clean":"2023-07-29T16:12:22.927995+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:12:22.927995+0000","last_undegraded":"2023-07-29T16:12:22.927995+0000","last_fullsized":"2023-07-29T16:12:22.927995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514793+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:12:23.514793+0000","last_peered":"2023-07-29T16:12:23.514793+0000","last_clean":"2023-07-29T16:12:23.514793+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:12:23.514793+0000","last_undegraded":"2023-07-29T16:12:23.514793+0000","last_fullsized":"2023-07-29T16:12:23.514793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514812+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:12:23.514812+0000","last_peered":"2023-07-29T16:12:23.514812+0000","last_clean":"2023-07-29T16:12:23.514812+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:12:23.514812+0000","last_undegraded":"2023-07-29T16:12:23.514812+0000","last_fullsized":"2023-07-29T16:12:23.514812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715813+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:12:23.715813+0000","last_peered":"2023-07-29T16:12:23.715813+0000","last_clean":"2023-07-29T16:12:23.715813+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:12:23.715813+0000","last_undegraded":"2023-07-29T16:12:23.715813+0000","last_fullsized":"2023-07-29T16:12:23.715813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.842217+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:12:23.842217+0000","last_peered":"2023-07-29T16:12:23.842217+0000","last_clean":"2023-07-29T16:12:23.842217+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:12:23.842217+0000","last_undegraded":"2023-07-29T16:12:23.842217+0000","last_fullsized":"2023-07-29T16:12:23.842217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928128+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:12:22.928128+0000","last_peered":"2023-07-29T16:12:22.928128+0000","last_clean":"2023-07-29T16:12:22.928128+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:12:22.928128+0000","last_undegraded":"2023-07-29T16:12:22.928128+0000","last_fullsized":"2023-07-29T16:12:22.928128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514874+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:12:23.514874+0000","last_peered":"2023-07-29T16:12:23.514874+0000","last_clean":"2023-07-29T16:12:23.514874+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:12:23.514874+0000","last_undegraded":"2023-07-29T16:12:23.514874+0000","last_fullsized":"2023-07-29T16:12:23.514874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.919056+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:12:22.919056+0000","last_peered":"2023-07-29T16:12:22.919056+0000","last_clean":"2023-07-29T16:12:22.919056+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:12:22.919056+0000","last_undegraded":"2023-07-29T16:12:22.919056+0000","last_fullsized":"2023-07-29T16:12:22.919056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290818+0000","last_change":"2023-07-29T16:12:21.935192+0000","last_active":"2023-07-29T16:12:22.290818+0000","last_peered":"2023-07-29T16:12:22.290818+0000","last_clean":"2023-07-29T16:12:22.290818+0000","last_became_active":"2023-07-29T16:12:21.934906+0000","last_became_peered":"2023-07-29T16:12:21.934906+0000","last_unstale":"2023-07-29T16:12:22.290818+0000","last_undegraded":"2023-07-29T16:12:22.290818+0000","last_fullsized":"2023-07-29T16:12:22.290818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:34:20.853432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918740+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:12:22.918740+0000","last_peered":"2023-07-29T16:12:22.918740+0000","last_clean":"2023-07-29T16:12:22.918740+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:12:22.918740+0000","last_undegraded":"2023-07-29T16:12:22.918740+0000","last_fullsized":"2023-07-29T16:12:22.918740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652585+0000","last_change":"2023-07-29T16:12:21.950611+0000","last_active":"2023-07-29T16:12:22.652585+0000","last_peered":"2023-07-29T16:12:22.652585+0000","last_clean":"2023-07-29T16:12:22.652585+0000","last_became_active":"2023-07-29T16:12:21.950060+0000","last_became_peered":"2023-07-29T16:12:21.950060+0000","last_unstale":"2023-07-29T16:12:22.652585+0000","last_undegraded":"2023-07-29T16:12:22.652585+0000","last_fullsized":"2023-07-29T16:12:22.652585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:09.788878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841250+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:12:23.841250+0000","last_peered":"2023-07-29T16:12:23.841250+0000","last_clean":"2023-07-29T16:12:23.841250+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:12:23.841250+0000","last_undegraded":"2023-07-29T16:12:23.841250+0000","last_fullsized":"2023-07-29T16:12:23.841250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927993+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:12:22.927993+0000","last_peered":"2023-07-29T16:12:22.927993+0000","last_clean":"2023-07-29T16:12:22.927993+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:12:22.927993+0000","last_undegraded":"2023-07-29T16:12:22.927993+0000","last_fullsized":"2023-07-29T16:12:22.927993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5078,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":29,"num_read_kb":25,"num_write":124,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":32983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":925696,"data_stored":441942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":10,"up_from":73,"seq":313532612625,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1556480,"data_stored":1034541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55600000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.505},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.63},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.72999999999999998}]},{"osd":9,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.76100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":9,"up_from":68,"seq":292057776149,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":942080,"data_stored":442178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":0.71899999999999997}]},{"osd":1,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.501},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.65600000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82399999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.88100000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.623},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.69199999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.96999999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.93999999999999995}]}]},{"osd":8,"up_from":60,"seq":257698037785,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":937984,"data_stored":441767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,7],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":2.0680000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":1.2250000000000001}]},{"osd":1,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.75700000000000001}]},{"osd":2,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":1.052},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":1.5489999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":1.7250000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.94699999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":2.1760000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":1.1639999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.67000000000000004}]},{"osd":9,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":1.2749999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.85199999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":7,"up_from":52,"seq":223338299422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1523712,"data_stored":1032553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.39600000000000002},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.437,"15min":0.43099999999999999},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70999999999999996,"5min":0.755,"15min":0.755},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.497},{"interface":"front","average":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.46800000000000003}]},{"osd":4,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.437},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52700000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.38800000000000001}]},{"osd":5,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.69199999999999995,"15min":0.66400000000000003},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.2300000000000004,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64000000000000001},"min":{"1min":0.53200000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.58799999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.73699999999999999,"15min":0.70399999999999996},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.56899999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:11:27 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53300000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.624},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.60799999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":925696,"data_stored":441450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.64000000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.51300000000000001,"15min":0.501},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.83799999999999997}]},{"osd":1,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.91600000000000004,"5min":1.044,"15min":1.044},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.67500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.92500000000000004,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.92700000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.89800000000000002,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.81999999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76500000000000001,"15min":0.76000000000000001},"min":{"1min":0.63300000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.98499999999999999,"5min":1.111,"15min":1.111},"last":0.72599999999999998}]},{"osd":7,"last update":"Sat Jul 29 16:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.80400000000000005}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.76800000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.873},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.89800000000000002}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.93200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":933888,"data_stored":442924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.49199999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76200000000000001,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.53400000000000003,"15min":0.498},"min":{"1min":0.42199999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.54400000000000004,"15min":0.52500000000000002},"min":{"1min":0.40600000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.54100000000000004,"15min":0.51100000000000001},"min":{"1min":0.39600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.42399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.55100000000000005},"min":{"1min":0.433,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.55700000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:03 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.438,"15min":0.41799999999999998},"min":{"1min":0.28299999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.48199999999999998,"15min":0.46200000000000002},"min":{"1min":0.318,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77400000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.56899999999999995},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.629,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.84599999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:11:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64200000000000002}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.60499999999999998,"15min":0.58499999999999996},"min":{"1min":0.45000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.65900000000000003,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.624}]},{"osd":8,"last update":"Sat Jul 29 16:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.73799999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.751},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.78800000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":950272,"data_stored":442838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.44500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81499999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.56999999999999995,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.63}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88300000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86099999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.55400000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.372,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":0.876,"15min":0.876},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.48399999999999999,"15min":0.47199999999999998},"min":{"1min":0.41099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77800000000000002,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.80300000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61399999999999999,"15min":0.58499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.57899999999999996,"15min":0.54400000000000004},"min":{"1min":0.47799999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68700000000000006}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92200000000000004,"5min":1.111,"15min":1.111},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65900000000000003,"15min":0.64200000000000002},"min":{"1min":0.52900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.82899999999999996,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:11:49 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.754},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73399999999999999,"15min":0.72999999999999998},"min":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.439}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.82799999999999996}]},{"osd":9,"last update":"Sat Jul 29 16:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.81599999999999995}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":444226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.55800000000000005},"min":{"1min":0.41899999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.95299999999999996,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.59899999999999998,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.71699999999999997}]},{"osd":1,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.613,"5min":3.613,"15min":3.613},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.81699999999999995}]},{"osd":3,"last update":"Sat Jul 29 16:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.441,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.878,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55900000000000005,"15min":0.54300000000000004},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47299999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.626,"15min":0.59499999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.6019999999999999,"5min":3.6019999999999999,"15min":3.6019999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47699999999999998,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:11:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.042,"5min":0.70999999999999996,"15min":0.65400000000000003},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.1989999999999998,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.624}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.73999999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:11:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.84099999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.91900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":937984,"data_stored":441436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.42699999999999999,"15min":0.39700000000000002},"min":{"1min":0.371,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.374},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.434}]},{"osd":1,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.58699999999999997,"15min":0.56699999999999995},"min":{"1min":0.44700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.52200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.65500000000000003,"15min":0.61599999999999999},"min":{"1min":0.58499999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63300000000000001,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.65500000000000003}]},{"osd":4,"last update":"Sat Jul 29 16:12:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.624,"15min":0.59999999999999998},"min":{"1min":0.46600000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86299999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.622},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67100000000000004,"15min":0.65900000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.73499999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70599999999999996,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.165,"5min":1.343,"15min":1.343},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71899999999999997,"15min":0.69899999999999995},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.71399999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:11:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69699999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.67500000000000004,"15min":0.65500000000000003},"min":{"1min":0.54300000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84199999999999997,"15min":0.82999999999999996},"min":{"1min":0.71999999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.83299999999999996}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.014},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.0089999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.78600000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.75},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.88900000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888109,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":933888,"data_stored":441956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.48799999999999999},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56699999999999995,"15min":0.55600000000000005},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84199999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.71999999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.48099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.751},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.0600000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.70499999999999996}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.48299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.80100000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:11:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.628,"15min":0.62},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88200000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.67600000000000005,"15min":0.66000000000000003},"min":{"1min":0.56799999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.67300000000000004}]},{"osd":7,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.69199999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:11:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.85199999999999998}]},{"osd":9,"last update":"Sat Jul 29 16:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.82699999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:12:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.88400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1544192,"data_stored":1033156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.71899999999999997}]},{"osd":2,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40600000000000003,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29799999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70799999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81299999999999994,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72899999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.41699999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.754,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.76200000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.627,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.65300000000000002,"15min":0.64600000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.626},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79300000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.89500000000000002}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.80400000000000005}]},{"osd":9,"last update":"Sat Jul 29 16:11:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.878},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.91700000000000004}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.86599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":20615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:12:31.222 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-29T16:12:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 -- 172.21.15.31:0/3123957020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 msgr2=0x7fef6c101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 --2- 172.21.15.31:0/3123957020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c101980 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fef58009990 tx=0x7fef58009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 -- 172.21.15.31:0/3123957020 shutdown_connections 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 --2- 172.21.15.31:0/3123957020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fef6c101ec0 0x7fef6c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 --2- 172.21.15.31:0/3123957020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef6c100340 0x7fef6c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 --2- 172.21.15.31:0/3123957020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c101980 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 -- 172.21.15.31:0/3123957020 >> 172.21.15.31:0/3123957020 conn(0x7fef6c0fb920 msgr2=0x7fef6c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 -- 172.21.15.31:0/3123957020 shutdown_connections 2023-07-29T16:12:31.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.460+0000 7fef72bff700 1 -- 172.21.15.31:0/3123957020 wait complete. 2023-07-29T16:12:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.461+0000 7fef72bff700 1 Processor -- start 2023-07-29T16:12:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.461+0000 7fef72bff700 1 -- start start 2023-07-29T16:12:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.461+0000 7fef72bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef6c100340 0x7fef6c074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef72bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef72bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fef6c101ec0 0x7fef6c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef72bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c073e20 con 0x7fef6c101530 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef72bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c073f90 con 0x7fef6c101ec0 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef72bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c1a0120 con 0x7fef6c100340 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42098/0 (socket says 172.21.15.31:42098) 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 -- 172.21.15.31:0/615159645 learned_addr learned my addr 172.21.15.31:0/615159645 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef7099b700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef6c100340 0x7fef6c074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef7119c700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fef6c101ec0 0x7fef6c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 -- 172.21.15.31:0/615159645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef6c100340 msgr2=0x7fef6c074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef6c100340 0x7fef6c074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 -- 172.21.15.31:0/615159645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fef6c101ec0 msgr2=0x7fef6c073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fef6c101ec0 0x7fef6c073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 -- 172.21.15.31:0/615159645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580095f0 con 0x7fef6c101530 2023-07-29T16:12:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.462+0000 7fef6bfff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c073140 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fef5800b620 tx=0x7fef58005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.463+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5801d070 con 0x7fef6c101530 2023-07-29T16:12:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.463+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6c1a03a0 con 0x7fef6c101530 2023-07-29T16:12:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.463+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef58022470 con 0x7fef6c101530 2023-07-29T16:12:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.463+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef58017650 con 0x7fef6c101530 2023-07-29T16:12:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.463+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6c1a0820 con 0x7fef6c101530 2023-07-29T16:12:31.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.464+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6c107880 con 0x7fef6c101530 2023-07-29T16:12:31.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.466+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fef580225e0 con 0x7fef6c101530 2023-07-29T16:12:31.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.468+0000 7fef69ffb700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fef5c0a0860 0x7fef5c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.468+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fef580ccf40 con 0x7fef6c101530 2023-07-29T16:12:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.468+0000 7fef7099b700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fef5c0a0860 0x7fef5c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.468+0000 7fef7099b700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fef5c0a0860 0x7fef5c0a2d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fef6c101390 tx=0x7fef60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:31.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.471+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fef58096ac0 con 0x7fef6c101530 2023-07-29T16:12:31.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.678+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef6c061190 con 0x7fef5c0a0860 2023-07-29T16:12:31.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.688+0000 7fef69ffb700 1 -- 172.21.15.31:0/615159645 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335746 (secure 0 0 0) 0x7fef6c061190 con 0x7fef5c0a0860 2023-07-29T16:12:31.689 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:31.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fef5c0a0860 msgr2=0x7fef5c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fef5c0a0860 0x7fef5c0a2d10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fef6c101390 tx=0x7fef60009260 comp rx=0 tx=0).stop 2023-07-29T16:12:31.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 msgr2=0x7fef6c073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:31.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c073140 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fef5800b620 tx=0x7fef58005740 comp rx=0 tx=0).stop 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 shutdown_connections 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fef5c0a0860 0x7fef5c0a2d10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fef6c101ec0 0x7fef6c073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef6c100340 0x7fef6c074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 --2- 172.21.15.31:0/615159645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef6c101530 0x7fef6c073140 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.693+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 >> 172.21.15.31:0/615159645 conn(0x7fef6c0fb920 msgr2=0x7fef6c1049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:31.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.694+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 shutdown_connections 2023-07-29T16:12:31.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:31.694+0000 7fef72bff700 1 -- 172.21.15.31:0/615159645 wait complete. 2023-07-29T16:12:31.693 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:12:31.709 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2023-07-29T16:12:31.272611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595446,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":155,"num_read_kb":134,"num_write":305,"num_write_kb":2976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":245,"ondisk_log_size":245,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336188,"kb_used_data":12764,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124605636,"statfs":{"total":1151940427776,"available":1151596171264,"internally_reserved":0,"allocated":13070336,"data_stored":7080967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,7],"upper_bound":8},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5078,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":29,"num_read_kb":25,"num_write":124,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.357455"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928127+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:12:22.928127+0000","last_peered":"2023-07-29T16:12:22.928127+0000","last_clean":"2023-07-29T16:12:22.928127+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:12:22.928127+0000","last_undegraded":"2023-07-29T16:12:22.928127+0000","last_fullsized":"2023-07-29T16:12:22.928127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715831+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:12:23.715831+0000","last_peered":"2023-07-29T16:12:23.715831+0000","last_clean":"2023-07-29T16:12:23.715831+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:12:23.715831+0000","last_undegraded":"2023-07-29T16:12:23.715831+0000","last_fullsized":"2023-07-29T16:12:23.715831+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715823+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:12:23.715823+0000","last_peered":"2023-07-29T16:12:23.715823+0000","last_clean":"2023-07-29T16:12:23.715823+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:12:23.715823+0000","last_undegraded":"2023-07-29T16:12:23.715823+0000","last_fullsized":"2023-07-29T16:12:23.715823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927143+0000","last_change":"2023-07-29T16:12:21.943788+0000","last_active":"2023-07-29T16:12:22.927143+0000","last_peered":"2023-07-29T16:12:22.927143+0000","last_clean":"2023-07-29T16:12:22.927143+0000","last_became_active":"2023-07-29T16:12:21.943292+0000","last_became_peered":"2023-07-29T16:12:21.943292+0000","last_unstale":"2023-07-29T16:12:22.927143+0000","last_undegraded":"2023-07-29T16:12:22.927143+0000","last_fullsized":"2023-07-29T16:12:22.927143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:34:18.994175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464359+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:12:22.464359+0000","last_peered":"2023-07-29T16:12:22.464359+0000","last_clean":"2023-07-29T16:12:22.464359+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:12:22.464359+0000","last_undegraded":"2023-07-29T16:12:22.464359+0000","last_fullsized":"2023-07-29T16:12:22.464359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674941+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:12:22.674941+0000","last_peered":"2023-07-29T16:12:22.674941+0000","last_clean":"2023-07-29T16:12:22.674941+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:12:22.674941+0000","last_undegraded":"2023-07-29T16:12:22.674941+0000","last_fullsized":"2023-07-29T16:12:22.674941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.946618+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:12:21.946618+0000","last_peered":"2023-07-29T16:12:21.946618+0000","last_clean":"2023-07-29T16:12:21.946618+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:12:21.946618+0000","last_undegraded":"2023-07-29T16:12:21.946618+0000","last_fullsized":"2023-07-29T16:12:21.946618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918534+0000","last_change":"2023-07-29T16:12:21.941714+0000","last_active":"2023-07-29T16:12:22.918534+0000","last_peered":"2023-07-29T16:12:22.918534+0000","last_clean":"2023-07-29T16:12:22.918534+0000","last_became_active":"2023-07-29T16:12:21.941403+0000","last_became_peered":"2023-07-29T16:12:21.941403+0000","last_unstale":"2023-07-29T16:12:22.918534+0000","last_undegraded":"2023-07-29T16:12:22.918534+0000","last_fullsized":"2023-07-29T16:12:22.918534+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:07:09.470922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652666+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:12:22.652666+0000","last_peered":"2023-07-29T16:12:22.652666+0000","last_clean":"2023-07-29T16:12:22.652666+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:12:22.652666+0000","last_undegraded":"2023-07-29T16:12:22.652666+0000","last_fullsized":"2023-07-29T16:12:22.652666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674987+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:12:22.674987+0000","last_peered":"2023-07-29T16:12:22.674987+0000","last_clean":"2023-07-29T16:12:22.674987+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:12:22.674987+0000","last_undegraded":"2023-07-29T16:12:22.674987+0000","last_fullsized":"2023-07-29T16:12:22.674987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.948821+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:12:21.948821+0000","last_peered":"2023-07-29T16:12:21.948821+0000","last_clean":"2023-07-29T16:12:21.948821+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:12:21.948821+0000","last_undegraded":"2023-07-29T16:12:21.948821+0000","last_fullsized":"2023-07-29T16:12:21.948821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927028+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:12:22.927028+0000","last_peered":"2023-07-29T16:12:22.927028+0000","last_clean":"2023-07-29T16:12:22.927028+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:12:22.927028+0000","last_undegraded":"2023-07-29T16:12:22.927028+0000","last_fullsized":"2023-07-29T16:12:22.927028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841081+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:12:23.841081+0000","last_peered":"2023-07-29T16:12:23.841081+0000","last_clean":"2023-07-29T16:12:23.841081+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:12:23.841081+0000","last_undegraded":"2023-07-29T16:12:23.841081+0000","last_fullsized":"2023-07-29T16:12:23.841081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715803+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:12:23.715803+0000","last_peered":"2023-07-29T16:12:23.715803+0000","last_clean":"2023-07-29T16:12:23.715803+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:12:23.715803+0000","last_undegraded":"2023-07-29T16:12:23.715803+0000","last_fullsized":"2023-07-29T16:12:23.715803+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715784+0000","last_change":"2023-07-29T16:12:21.940409+0000","last_active":"2023-07-29T16:12:23.715784+0000","last_peered":"2023-07-29T16:12:23.715784+0000","last_clean":"2023-07-29T16:12:23.715784+0000","last_became_active":"2023-07-29T16:12:21.939995+0000","last_became_peered":"2023-07-29T16:12:21.939995+0000","last_unstale":"2023-07-29T16:12:23.715784+0000","last_undegraded":"2023-07-29T16:12:23.715784+0000","last_fullsized":"2023-07-29T16:12:23.715784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:38:45.973435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652508+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:12:22.652508+0000","last_peered":"2023-07-29T16:12:22.652508+0000","last_clean":"2023-07-29T16:12:22.652508+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:12:22.652508+0000","last_undegraded":"2023-07-29T16:12:22.652508+0000","last_fullsized":"2023-07-29T16:12:22.652508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928213+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:12:22.928213+0000","last_peered":"2023-07-29T16:12:22.928213+0000","last_clean":"2023-07-29T16:12:22.928213+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:12:22.928213+0000","last_undegraded":"2023-07-29T16:12:22.928213+0000","last_fullsized":"2023-07-29T16:12:22.928213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.792116+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:12:23.792116+0000","last_peered":"2023-07-29T16:12:23.792116+0000","last_clean":"2023-07-29T16:12:23.792116+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:12:23.792116+0000","last_undegraded":"2023-07-29T16:12:23.792116+0000","last_fullsized":"2023-07-29T16:12:23.792116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514860+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:12:23.514860+0000","last_peered":"2023-07-29T16:12:23.514860+0000","last_clean":"2023-07-29T16:12:23.514860+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:12:23.514860+0000","last_undegraded":"2023-07-29T16:12:23.514860+0000","last_fullsized":"2023-07-29T16:12:23.514860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493366+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:12:26.493366+0000","last_peered":"2023-07-29T16:12:26.493366+0000","last_clean":"2023-07-29T16:12:26.493366+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:12:26.493366+0000","last_undegraded":"2023-07-29T16:12:26.493366+0000","last_fullsized":"2023-07-29T16:12:26.493366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918853+0000","last_change":"2023-07-29T16:12:21.930788+0000","last_active":"2023-07-29T16:12:22.918853+0000","last_peered":"2023-07-29T16:12:22.918853+0000","last_clean":"2023-07-29T16:12:22.918853+0000","last_became_active":"2023-07-29T16:12:21.930448+0000","last_became_peered":"2023-07-29T16:12:21.930448+0000","last_unstale":"2023-07-29T16:12:22.918853+0000","last_undegraded":"2023-07-29T16:12:22.918853+0000","last_fullsized":"2023-07-29T16:12:22.918853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:28:51.735231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:24.190099+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:12:24.190099+0000","last_peered":"2023-07-29T16:12:24.190099+0000","last_clean":"2023-07-29T16:12:24.190099+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:12:24.190099+0000","last_undegraded":"2023-07-29T16:12:24.190099+0000","last_fullsized":"2023-07-29T16:12:24.190099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715872+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:12:23.715872+0000","last_peered":"2023-07-29T16:12:23.715872+0000","last_clean":"2023-07-29T16:12:23.715872+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:12:23.715872+0000","last_undegraded":"2023-07-29T16:12:23.715872+0000","last_fullsized":"2023-07-29T16:12:23.715872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.491458+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:12:26.491458+0000","last_peered":"2023-07-29T16:12:26.491458+0000","last_clean":"2023-07-29T16:12:26.491458+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:12:26.491458+0000","last_undegraded":"2023-07-29T16:12:26.491458+0000","last_fullsized":"2023-07-29T16:12:26.491458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652606+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:12:22.652606+0000","last_peered":"2023-07-29T16:12:22.652606+0000","last_clean":"2023-07-29T16:12:22.652606+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:12:22.652606+0000","last_undegraded":"2023-07-29T16:12:22.652606+0000","last_fullsized":"2023-07-29T16:12:22.652606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464392+0000","last_change":"2023-07-29T16:12:21.348621+0000","last_active":"2023-07-29T16:12:22.464392+0000","last_peered":"2023-07-29T16:12:22.464392+0000","last_clean":"2023-07-29T16:12:22.464392+0000","last_became_active":"2023-07-29T16:12:21.348302+0000","last_became_peered":"2023-07-29T16:12:21.348302+0000","last_unstale":"2023-07-29T16:12:22.464392+0000","last_undegraded":"2023-07-29T16:12:22.464392+0000","last_fullsized":"2023-07-29T16:12:22.464392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:21:32.671912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290898+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:12:22.290898+0000","last_peered":"2023-07-29T16:12:22.290898+0000","last_clean":"2023-07-29T16:12:22.290898+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:12:22.290898+0000","last_undegraded":"2023-07-29T16:12:22.290898+0000","last_fullsized":"2023-07-29T16:12:22.290898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.743054+0000","last_change":"2023-07-29T16:12:21.345470+0000","last_active":"2023-07-29T16:12:26.743054+0000","last_peered":"2023-07-29T16:12:26.743054+0000","last_clean":"2023-07-29T16:12:26.743054+0000","last_became_active":"2023-07-29T16:12:21.344787+0000","last_became_peered":"2023-07-29T16:12:21.344787+0000","last_unstale":"2023-07-29T16:12:26.743054+0000","last_undegraded":"2023-07-29T16:12:26.743054+0000","last_fullsized":"2023-07-29T16:12:26.743054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:31:19.451396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464348+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:12:22.464348+0000","last_peered":"2023-07-29T16:12:22.464348+0000","last_clean":"2023-07-29T16:12:22.464348+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:12:22.464348+0000","last_undegraded":"2023-07-29T16:12:22.464348+0000","last_fullsized":"2023-07-29T16:12:22.464348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.492930+0000","last_change":"2023-07-29T16:12:21.346661+0000","last_active":"2023-07-29T16:12:26.492930+0000","last_peered":"2023-07-29T16:12:26.492930+0000","last_clean":"2023-07-29T16:12:26.492930+0000","last_became_active":"2023-07-29T16:12:21.346202+0000","last_became_peered":"2023-07-29T16:12:21.346202+0000","last_unstale":"2023-07-29T16:12:26.492930+0000","last_undegraded":"2023-07-29T16:12:26.492930+0000","last_fullsized":"2023-07-29T16:12:26.492930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:11:26.080842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290874+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:12:22.290874+0000","last_peered":"2023-07-29T16:12:22.290874+0000","last_clean":"2023-07-29T16:12:22.290874+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:12:22.290874+0000","last_undegraded":"2023-07-29T16:12:22.290874+0000","last_fullsized":"2023-07-29T16:12:22.290874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.492569+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:12:26.492569+0000","last_peered":"2023-07-29T16:12:26.492569+0000","last_clean":"2023-07-29T16:12:26.492569+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:12:26.492569+0000","last_undegraded":"2023-07-29T16:12:26.492569+0000","last_fullsized":"2023-07-29T16:12:26.492569+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918628+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:12:22.918628+0000","last_peered":"2023-07-29T16:12:22.918628+0000","last_clean":"2023-07-29T16:12:22.918628+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:12:22.918628+0000","last_undegraded":"2023-07-29T16:12:22.918628+0000","last_fullsized":"2023-07-29T16:12:22.918628+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.917948+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:12:22.917948+0000","last_peered":"2023-07-29T16:12:22.917948+0000","last_clean":"2023-07-29T16:12:22.917948+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:12:22.917948+0000","last_undegraded":"2023-07-29T16:12:22.917948+0000","last_fullsized":"2023-07-29T16:12:22.917948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.792052+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:12:23.792052+0000","last_peered":"2023-07-29T16:12:23.792052+0000","last_clean":"2023-07-29T16:12:23.792052+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:12:23.792052+0000","last_undegraded":"2023-07-29T16:12:23.792052+0000","last_fullsized":"2023-07-29T16:12:23.792052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476200+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:12:25.476200+0000","last_peered":"2023-07-29T16:12:25.476200+0000","last_clean":"2023-07-29T16:12:25.476200+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:12:25.476200+0000","last_undegraded":"2023-07-29T16:12:25.476200+0000","last_fullsized":"2023-07-29T16:12:25.476200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652649+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:12:22.652649+0000","last_peered":"2023-07-29T16:12:22.652649+0000","last_clean":"2023-07-29T16:12:22.652649+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:12:22.652649+0000","last_undegraded":"2023-07-29T16:12:22.652649+0000","last_fullsized":"2023-07-29T16:12:22.652649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.474802+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:25.474802+0000","last_peered":"2023-07-29T16:12:25.474802+0000","last_clean":"2023-07-29T16:12:25.474802+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:12:25.474802+0000","last_undegraded":"2023-07-29T16:12:25.474802+0000","last_fullsized":"2023-07-29T16:12:25.474802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715863+0000","last_change":"2023-07-29T16:12:21.940714+0000","last_active":"2023-07-29T16:12:23.715863+0000","last_peered":"2023-07-29T16:12:23.715863+0000","last_clean":"2023-07-29T16:12:23.715863+0000","last_became_active":"2023-07-29T16:12:21.940414+0000","last_became_peered":"2023-07-29T16:12:21.940414+0000","last_unstale":"2023-07-29T16:12:23.715863+0000","last_undegraded":"2023-07-29T16:12:23.715863+0000","last_fullsized":"2023-07-29T16:12:23.715863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:53:12.841824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476490+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:12:25.476490+0000","last_peered":"2023-07-29T16:12:25.476490+0000","last_clean":"2023-07-29T16:12:25.476490+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:12:25.476490+0000","last_undegraded":"2023-07-29T16:12:25.476490+0000","last_fullsized":"2023-07-29T16:12:25.476490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652626+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:12:22.652626+0000","last_peered":"2023-07-29T16:12:22.652626+0000","last_clean":"2023-07-29T16:12:22.652626+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:12:22.652626+0000","last_undegraded":"2023-07-29T16:12:22.652626+0000","last_fullsized":"2023-07-29T16:12:22.652626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.743706+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:12:26.743706+0000","last_peered":"2023-07-29T16:12:26.743706+0000","last_clean":"2023-07-29T16:12:26.743706+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:12:26.743706+0000","last_undegraded":"2023-07-29T16:12:26.743706+0000","last_fullsized":"2023-07-29T16:12:26.743706+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464407+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:12:22.464407+0000","last_peered":"2023-07-29T16:12:22.464407+0000","last_clean":"2023-07-29T16:12:22.464407+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:12:22.464407+0000","last_undegraded":"2023-07-29T16:12:22.464407+0000","last_fullsized":"2023-07-29T16:12:22.464407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493696+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:12:26.493696+0000","last_peered":"2023-07-29T16:12:26.493696+0000","last_clean":"2023-07-29T16:12:26.493696+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:12:26.493696+0000","last_undegraded":"2023-07-29T16:12:26.493696+0000","last_fullsized":"2023-07-29T16:12:26.493696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841072+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:12:23.841072+0000","last_peered":"2023-07-29T16:12:23.841072+0000","last_clean":"2023-07-29T16:12:23.841072+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:12:23.841072+0000","last_undegraded":"2023-07-29T16:12:23.841072+0000","last_fullsized":"2023-07-29T16:12:23.841072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476121+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:12:25.476121+0000","last_peered":"2023-07-29T16:12:25.476121+0000","last_clean":"2023-07-29T16:12:25.476121+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:12:25.476121+0000","last_undegraded":"2023-07-29T16:12:25.476121+0000","last_fullsized":"2023-07-29T16:12:25.476121+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.950273+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:12:21.950273+0000","last_peered":"2023-07-29T16:12:21.950273+0000","last_clean":"2023-07-29T16:12:21.950273+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:12:21.950273+0000","last_undegraded":"2023-07-29T16:12:21.950273+0000","last_fullsized":"2023-07-29T16:12:21.950273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.745293+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:12:26.745293+0000","last_peered":"2023-07-29T16:12:26.745293+0000","last_clean":"2023-07-29T16:12:26.745293+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:12:26.745293+0000","last_undegraded":"2023-07-29T16:12:26.745293+0000","last_fullsized":"2023-07-29T16:12:26.745293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.950230+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:12:21.950230+0000","last_peered":"2023-07-29T16:12:21.950230+0000","last_clean":"2023-07-29T16:12:21.950230+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:12:21.950230+0000","last_undegraded":"2023-07-29T16:12:21.950230+0000","last_fullsized":"2023-07-29T16:12:21.950230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493657+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:12:26.493657+0000","last_peered":"2023-07-29T16:12:26.493657+0000","last_clean":"2023-07-29T16:12:26.493657+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:12:26.493657+0000","last_undegraded":"2023-07-29T16:12:26.493657+0000","last_fullsized":"2023-07-29T16:12:26.493657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290842+0000","last_change":"2023-07-29T16:12:21.938419+0000","last_active":"2023-07-29T16:12:22.290842+0000","last_peered":"2023-07-29T16:12:22.290842+0000","last_clean":"2023-07-29T16:12:22.290842+0000","last_became_active":"2023-07-29T16:12:21.938029+0000","last_became_peered":"2023-07-29T16:12:21.938029+0000","last_unstale":"2023-07-29T16:12:22.290842+0000","last_undegraded":"2023-07-29T16:12:22.290842+0000","last_fullsized":"2023-07-29T16:12:22.290842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:31:54.166660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841376+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:12:23.841376+0000","last_peered":"2023-07-29T16:12:23.841376+0000","last_clean":"2023-07-29T16:12:23.841376+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:12:23.841376+0000","last_undegraded":"2023-07-29T16:12:23.841376+0000","last_fullsized":"2023-07-29T16:12:23.841376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928108+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:12:22.928108+0000","last_peered":"2023-07-29T16:12:22.928108+0000","last_clean":"2023-07-29T16:12:22.928108+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:12:22.928108+0000","last_undegraded":"2023-07-29T16:12:22.928108+0000","last_fullsized":"2023-07-29T16:12:22.928108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.792086+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:12:23.792086+0000","last_peered":"2023-07-29T16:12:23.792086+0000","last_clean":"2023-07-29T16:12:23.792086+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:12:23.792086+0000","last_undegraded":"2023-07-29T16:12:23.792086+0000","last_fullsized":"2023-07-29T16:12:23.792086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674977+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:12:22.674977+0000","last_peered":"2023-07-29T16:12:22.674977+0000","last_clean":"2023-07-29T16:12:22.674977+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:12:22.674977+0000","last_undegraded":"2023-07-29T16:12:22.674977+0000","last_fullsized":"2023-07-29T16:12:22.674977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841427+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:12:23.841427+0000","last_peered":"2023-07-29T16:12:23.841427+0000","last_clean":"2023-07-29T16:12:23.841427+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:12:23.841427+0000","last_undegraded":"2023-07-29T16:12:23.841427+0000","last_fullsized":"2023-07-29T16:12:23.841427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.674954+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:12:22.674954+0000","last_peered":"2023-07-29T16:12:22.674954+0000","last_clean":"2023-07-29T16:12:22.674954+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:12:22.674954+0000","last_undegraded":"2023-07-29T16:12:22.674954+0000","last_fullsized":"2023-07-29T16:12:22.674954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290829+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:12:22.290829+0000","last_peered":"2023-07-29T16:12:22.290829+0000","last_clean":"2023-07-29T16:12:22.290829+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:12:22.290829+0000","last_undegraded":"2023-07-29T16:12:22.290829+0000","last_fullsized":"2023-07-29T16:12:22.290829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:27.746173+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:12:27.746173+0000","last_peered":"2023-07-29T16:12:27.746173+0000","last_clean":"2023-07-29T16:12:27.746173+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:12:27.746173+0000","last_undegraded":"2023-07-29T16:12:27.746173+0000","last_fullsized":"2023-07-29T16:12:27.746173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.919145+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:12:22.919145+0000","last_peered":"2023-07-29T16:12:22.919145+0000","last_clean":"2023-07-29T16:12:22.919145+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:12:22.919145+0000","last_undegraded":"2023-07-29T16:12:22.919145+0000","last_fullsized":"2023-07-29T16:12:22.919145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.740890+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:26.740890+0000","last_peered":"2023-07-29T16:12:26.740890+0000","last_clean":"2023-07-29T16:12:26.740890+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:12:26.740890+0000","last_undegraded":"2023-07-29T16:12:26.740890+0000","last_fullsized":"2023-07-29T16:12:26.740890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":328,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.933093+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:12:21.933093+0000","last_peered":"2023-07-29T16:12:21.933093+0000","last_clean":"2023-07-29T16:12:21.933093+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:12:21.933093+0000","last_undegraded":"2023-07-29T16:12:21.933093+0000","last_fullsized":"2023-07-29T16:12:21.933093+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.842089+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:12:23.842089+0000","last_peered":"2023-07-29T16:12:23.842089+0000","last_clean":"2023-07-29T16:12:23.842089+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:12:23.842089+0000","last_undegraded":"2023-07-29T16:12:23.842089+0000","last_fullsized":"2023-07-29T16:12:23.842089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.489683+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:12:26.489683+0000","last_peered":"2023-07-29T16:12:26.489683+0000","last_clean":"2023-07-29T16:12:26.489683+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:12:26.489683+0000","last_undegraded":"2023-07-29T16:12:26.489683+0000","last_fullsized":"2023-07-29T16:12:26.489683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476161+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:12:25.476161+0000","last_peered":"2023-07-29T16:12:25.476161+0000","last_clean":"2023-07-29T16:12:25.476161+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:12:25.476161+0000","last_undegraded":"2023-07-29T16:12:25.476161+0000","last_fullsized":"2023-07-29T16:12:25.476161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.842163+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:12:23.842163+0000","last_peered":"2023-07-29T16:12:23.842163+0000","last_clean":"2023-07-29T16:12:23.842163+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:12:23.842163+0000","last_undegraded":"2023-07-29T16:12:23.842163+0000","last_fullsized":"2023-07-29T16:12:23.842163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.493893+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:12:26.493893+0000","last_peered":"2023-07-29T16:12:26.493893+0000","last_clean":"2023-07-29T16:12:26.493893+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:12:26.493893+0000","last_undegraded":"2023-07-29T16:12:26.493893+0000","last_fullsized":"2023-07-29T16:12:26.493893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.919165+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:12:22.919165+0000","last_peered":"2023-07-29T16:12:22.919165+0000","last_clean":"2023-07-29T16:12:22.919165+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:12:22.919165+0000","last_undegraded":"2023-07-29T16:12:22.919165+0000","last_fullsized":"2023-07-29T16:12:22.919165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.472891+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:12:25.472891+0000","last_peered":"2023-07-29T16:12:25.472891+0000","last_clean":"2023-07-29T16:12:25.472891+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:12:25.472891+0000","last_undegraded":"2023-07-29T16:12:25.472891+0000","last_fullsized":"2023-07-29T16:12:25.472891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715843+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:12:23.715843+0000","last_peered":"2023-07-29T16:12:23.715843+0000","last_clean":"2023-07-29T16:12:23.715843+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:12:23.715843+0000","last_undegraded":"2023-07-29T16:12:23.715843+0000","last_fullsized":"2023-07-29T16:12:23.715843+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.476339+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:12:25.476339+0000","last_peered":"2023-07-29T16:12:25.476339+0000","last_clean":"2023-07-29T16:12:25.476339+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:12:25.476339+0000","last_undegraded":"2023-07-29T16:12:25.476339+0000","last_fullsized":"2023-07-29T16:12:25.476339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.464335+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:12:22.464335+0000","last_peered":"2023-07-29T16:12:22.464335+0000","last_clean":"2023-07-29T16:12:22.464335+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:12:22.464335+0000","last_undegraded":"2023-07-29T16:12:22.464335+0000","last_fullsized":"2023-07-29T16:12:22.464335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.475756+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:12:25.475756+0000","last_peered":"2023-07-29T16:12:25.475756+0000","last_clean":"2023-07-29T16:12:25.475756+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:12:25.475756+0000","last_undegraded":"2023-07-29T16:12:25.475756+0000","last_fullsized":"2023-07-29T16:12:25.475756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927924+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:12:22.927924+0000","last_peered":"2023-07-29T16:12:22.927924+0000","last_clean":"2023-07-29T16:12:22.927924+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:12:22.927924+0000","last_undegraded":"2023-07-29T16:12:22.927924+0000","last_fullsized":"2023-07-29T16:12:22.927924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.840782+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:12:23.840782+0000","last_peered":"2023-07-29T16:12:23.840782+0000","last_clean":"2023-07-29T16:12:23.840782+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:12:23.840782+0000","last_undegraded":"2023-07-29T16:12:23.840782+0000","last_fullsized":"2023-07-29T16:12:23.840782+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918824+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:12:22.918824+0000","last_peered":"2023-07-29T16:12:22.918824+0000","last_clean":"2023-07-29T16:12:22.918824+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:12:22.918824+0000","last_undegraded":"2023-07-29T16:12:22.918824+0000","last_fullsized":"2023-07-29T16:12:22.918824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:25.474110+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:12:25.474110+0000","last_peered":"2023-07-29T16:12:25.474110+0000","last_clean":"2023-07-29T16:12:25.474110+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:12:25.474110+0000","last_undegraded":"2023-07-29T16:12:25.474110+0000","last_fullsized":"2023-07-29T16:12:25.474110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.951714+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:12:21.951714+0000","last_peered":"2023-07-29T16:12:21.951714+0000","last_clean":"2023-07-29T16:12:21.951714+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:12:21.951714+0000","last_undegraded":"2023-07-29T16:12:21.951714+0000","last_fullsized":"2023-07-29T16:12:21.951714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:26.492838+0000","last_change":"2023-07-29T16:12:21.346949+0000","last_active":"2023-07-29T16:12:26.492838+0000","last_peered":"2023-07-29T16:12:26.492838+0000","last_clean":"2023-07-29T16:12:26.492838+0000","last_became_active":"2023-07-29T16:12:21.346598+0000","last_became_peered":"2023-07-29T16:12:21.346598+0000","last_unstale":"2023-07-29T16:12:26.492838+0000","last_undegraded":"2023-07-29T16:12:26.492838+0000","last_fullsized":"2023-07-29T16:12:26.492838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:54:03.817342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.947524+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:12:21.947524+0000","last_peered":"2023-07-29T16:12:21.947524+0000","last_clean":"2023-07-29T16:12:21.947524+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:12:21.947524+0000","last_undegraded":"2023-07-29T16:12:21.947524+0000","last_fullsized":"2023-07-29T16:12:21.947524+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918258+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:12:22.918258+0000","last_peered":"2023-07-29T16:12:22.918258+0000","last_clean":"2023-07-29T16:12:22.918258+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:12:22.918258+0000","last_undegraded":"2023-07-29T16:12:22.918258+0000","last_fullsized":"2023-07-29T16:12:22.918258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.947514+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:12:21.947514+0000","last_peered":"2023-07-29T16:12:21.947514+0000","last_clean":"2023-07-29T16:12:21.947514+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:12:21.947514+0000","last_undegraded":"2023-07-29T16:12:21.947514+0000","last_fullsized":"2023-07-29T16:12:21.947514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841165+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:12:23.841165+0000","last_peered":"2023-07-29T16:12:23.841165+0000","last_clean":"2023-07-29T16:12:23.841165+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:12:23.841165+0000","last_undegraded":"2023-07-29T16:12:23.841165+0000","last_fullsized":"2023-07-29T16:12:23.841165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:21.945580+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:12:21.945580+0000","last_peered":"2023-07-29T16:12:21.945580+0000","last_clean":"2023-07-29T16:12:21.945580+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:12:21.945580+0000","last_undegraded":"2023-07-29T16:12:21.945580+0000","last_fullsized":"2023-07-29T16:12:21.945580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927995+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:12:22.927995+0000","last_peered":"2023-07-29T16:12:22.927995+0000","last_clean":"2023-07-29T16:12:22.927995+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:12:22.927995+0000","last_undegraded":"2023-07-29T16:12:22.927995+0000","last_fullsized":"2023-07-29T16:12:22.927995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514793+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:12:23.514793+0000","last_peered":"2023-07-29T16:12:23.514793+0000","last_clean":"2023-07-29T16:12:23.514793+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:12:23.514793+0000","last_undegraded":"2023-07-29T16:12:23.514793+0000","last_fullsized":"2023-07-29T16:12:23.514793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514812+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:12:23.514812+0000","last_peered":"2023-07-29T16:12:23.514812+0000","last_clean":"2023-07-29T16:12:23.514812+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:12:23.514812+0000","last_undegraded":"2023-07-29T16:12:23.514812+0000","last_fullsized":"2023-07-29T16:12:23.514812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.715813+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:12:23.715813+0000","last_peered":"2023-07-29T16:12:23.715813+0000","last_clean":"2023-07-29T16:12:23.715813+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:12:23.715813+0000","last_undegraded":"2023-07-29T16:12:23.715813+0000","last_fullsized":"2023-07-29T16:12:23.715813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.842217+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:12:23.842217+0000","last_peered":"2023-07-29T16:12:23.842217+0000","last_clean":"2023-07-29T16:12:23.842217+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:12:23.842217+0000","last_undegraded":"2023-07-29T16:12:23.842217+0000","last_fullsized":"2023-07-29T16:12:23.842217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.928128+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:12:22.928128+0000","last_peered":"2023-07-29T16:12:22.928128+0000","last_clean":"2023-07-29T16:12:22.928128+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:12:22.928128+0000","last_undegraded":"2023-07-29T16:12:22.928128+0000","last_fullsized":"2023-07-29T16:12:22.928128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.514874+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:12:23.514874+0000","last_peered":"2023-07-29T16:12:23.514874+0000","last_clean":"2023-07-29T16:12:23.514874+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:12:23.514874+0000","last_undegraded":"2023-07-29T16:12:23.514874+0000","last_fullsized":"2023-07-29T16:12:23.514874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.919056+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:12:22.919056+0000","last_peered":"2023-07-29T16:12:22.919056+0000","last_clean":"2023-07-29T16:12:22.919056+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:12:22.919056+0000","last_undegraded":"2023-07-29T16:12:22.919056+0000","last_fullsized":"2023-07-29T16:12:22.919056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.290818+0000","last_change":"2023-07-29T16:12:21.935192+0000","last_active":"2023-07-29T16:12:22.290818+0000","last_peered":"2023-07-29T16:12:22.290818+0000","last_clean":"2023-07-29T16:12:22.290818+0000","last_became_active":"2023-07-29T16:12:21.934906+0000","last_became_peered":"2023-07-29T16:12:21.934906+0000","last_unstale":"2023-07-29T16:12:22.290818+0000","last_undegraded":"2023-07-29T16:12:22.290818+0000","last_fullsized":"2023-07-29T16:12:22.290818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:34:20.853432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.918740+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:12:22.918740+0000","last_peered":"2023-07-29T16:12:22.918740+0000","last_clean":"2023-07-29T16:12:22.918740+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:12:22.918740+0000","last_undegraded":"2023-07-29T16:12:22.918740+0000","last_fullsized":"2023-07-29T16:12:22.918740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.652585+0000","last_change":"2023-07-29T16:12:21.950611+0000","last_active":"2023-07-29T16:12:22.652585+0000","last_peered":"2023-07-29T16:12:22.652585+0000","last_clean":"2023-07-29T16:12:22.652585+0000","last_became_active":"2023-07-29T16:12:21.950060+0000","last_became_peered":"2023-07-29T16:12:21.950060+0000","last_unstale":"2023-07-29T16:12:22.652585+0000","last_undegraded":"2023-07-29T16:12:22.652585+0000","last_fullsized":"2023-07-29T16:12:22.652585+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:09.788878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:23.841250+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:12:23.841250+0000","last_peered":"2023-07-29T16:12:23.841250+0000","last_clean":"2023-07-29T16:12:23.841250+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:12:23.841250+0000","last_undegraded":"2023-07-29T16:12:23.841250+0000","last_fullsized":"2023-07-29T16:12:23.841250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-29T16:12:22.927993+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:12:22.927993+0000","last_peered":"2023-07-29T16:12:22.927993+0000","last_clean":"2023-07-29T16:12:22.927993+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:12:22.927993+0000","last_undegraded":"2023-07-29T16:12:22.927993+0000","last_fullsized":"2023-07-29T16:12:22.927993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5078,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":29,"num_read_kb":25,"num_write":124,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":32983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108,"ondisk_log_size":108,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":925696,"data_stored":441942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":10,"up_from":73,"seq":313532612625,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1556480,"data_stored":1034541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55600000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.505},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.63},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.72999999999999998}]},{"osd":9,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.76100000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]},{"osd":9,"up_from":68,"seq":292057776149,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":920,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":942080,"data_stored":442178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":0.71899999999999997}]},{"osd":1,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.501},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.65600000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.82399999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.4390000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.88100000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.623},{"interface":"front","average":{"1min":0.879,"5min":0.879,"15min":0.879},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.839,"5min":1.839,"15min":1.839},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.69199999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.96999999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":0.93999999999999995}]}]},{"osd":8,"up_from":60,"seq":257698037785,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":937984,"data_stored":441767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,7],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":2.0680000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":1.2250000000000001}]},{"osd":1,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.75700000000000001}]},{"osd":2,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":1.052},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":1.5489999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":1.7250000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.94699999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51800000000000002}]},{"osd":6,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":2.1760000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":1.1639999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:11:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.67000000000000004}]},{"osd":9,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":1.2749999999999999},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.85199999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]}]},{"osd":7,"up_from":52,"seq":223338299422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28440,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716712,"statfs":{"total":95995035648,"available":95965913088,"internally_reserved":0,"allocated":1523712,"data_stored":1032553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.39600000000000002},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.437,"15min":0.43099999999999999},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70999999999999996,"5min":0.755,"15min":0.755},"last":0.34100000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.497},{"interface":"front","average":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51500000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.46800000000000003}]},{"osd":4,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.437},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52700000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.38800000000000001}]},{"osd":5,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.69199999999999995,"15min":0.66400000000000003},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.2300000000000004,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64000000000000001},"min":{"1min":0.53200000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.58799999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.73699999999999999,"15min":0.70399999999999996},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.56899999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:11:27 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53300000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:11:56 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.624},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.60799999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55900000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462947,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":904,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":925696,"data_stored":441450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48099999999999998,"15min":0.47799999999999998},"min":{"1min":0.36199999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.64000000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.51300000000000001,"15min":0.501},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.83799999999999997}]},{"osd":1,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67200000000000004,"15min":0.67400000000000004},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.69899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.91600000000000004,"5min":1.044,"15min":1.044},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.67500000000000004,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.92500000000000004,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.65500000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.61199999999999999,"15min":0.60199999999999998},"min":{"1min":0.40000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.92700000000000005,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.65900000000000003,"15min":0.65100000000000002},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.70599999999999996,"15min":0.70399999999999996},"min":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.89800000000000002,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.81999999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.76500000000000001,"15min":0.76000000000000001},"min":{"1min":0.63300000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.78400000000000003,"15min":0.77700000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.98499999999999999,"5min":1.111,"15min":1.111},"last":0.72599999999999998}]},{"osd":7,"last update":"Sat Jul 29 16:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.80400000000000005}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.76800000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.873},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.89800000000000002}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.93200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":933888,"data_stored":442924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.52600000000000002,"15min":0.49199999999999999},"min":{"1min":0.39900000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.76200000000000001,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.53400000000000003,"15min":0.498},"min":{"1min":0.42199999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.54400000000000004,"15min":0.52500000000000002},"min":{"1min":0.40600000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.54100000000000004,"15min":0.51100000000000001},"min":{"1min":0.39600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.76300000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.55800000000000005,"15min":0.54100000000000004},"min":{"1min":0.42399999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.56999999999999995,"15min":0.55100000000000005},"min":{"1min":0.433,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82499999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.55700000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:03 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.438,"15min":0.41799999999999998},"min":{"1min":0.28299999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.48199999999999998,"15min":0.46200000000000002},"min":{"1min":0.318,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77400000000000002,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.53300000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.56899999999999995},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.629,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.84599999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:11:44 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.67900000000000005,"15min":0.67100000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64200000000000002}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.60499999999999998,"15min":0.58499999999999996},"min":{"1min":0.45000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.65900000000000003,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.624}]},{"osd":8,"last update":"Sat Jul 29 16:11:33 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.73799999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.751},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.78800000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":928,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":950272,"data_stored":442838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:11:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.44500000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81499999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.56999999999999995,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.63}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88300000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86099999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.55400000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.372,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":0.876,"15min":0.876},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.48399999999999999,"15min":0.47199999999999998},"min":{"1min":0.41099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77800000000000002,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.80300000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:02 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.61399999999999999,"15min":0.58499999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.57899999999999996,"15min":0.54400000000000004},"min":{"1min":0.47799999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68700000000000006}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92200000000000004,"5min":1.111,"15min":1.111},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65900000000000003,"15min":0.64200000000000002},"min":{"1min":0.52900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.82899999999999996,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.84499999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:11:49 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76000000000000001,"15min":0.754},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.72799999999999998}]},{"osd":7,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73399999999999999,"15min":0.72999999999999998},"min":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.439}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.82799999999999996}]},{"osd":9,"last update":"Sat Jul 29 16:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.81599999999999995}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92400000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":444226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.55800000000000005},"min":{"1min":0.41899999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.95299999999999996,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.59899999999999998,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.71699999999999997}]},{"osd":1,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.613,"5min":3.613,"15min":3.613},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.81699999999999995}]},{"osd":3,"last update":"Sat Jul 29 16:11:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.441,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.878,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55900000000000005,"15min":0.54300000000000004},"min":{"1min":0.40100000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.47299999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.626,"15min":0.59499999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.6019999999999999,"5min":3.6019999999999999,"15min":3.6019999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47699999999999998,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:11:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66500000000000004,"15min":0.65400000000000003},"min":{"1min":0.44700000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.042,"5min":0.70999999999999996,"15min":0.65400000000000003},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.1989999999999998,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.624}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.73999999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:11:54 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.84099999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.91900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":916,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":937984,"data_stored":441436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.42699999999999999,"15min":0.39700000000000002},"min":{"1min":0.371,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.374},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.45400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.434}]},{"osd":1,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.58699999999999997,"15min":0.56699999999999995},"min":{"1min":0.44700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.59299999999999997,"15min":0.56200000000000006},"min":{"1min":0.52200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68999999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:11:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.65500000000000003,"15min":0.61599999999999999},"min":{"1min":0.58499999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.63300000000000001,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.65500000000000003}]},{"osd":4,"last update":"Sat Jul 29 16:12:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.624,"15min":0.59999999999999998},"min":{"1min":0.46600000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.86299999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.622},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67100000000000004,"15min":0.65900000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.73499999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70599999999999996,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.165,"5min":1.343,"15min":1.343},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71899999999999997,"15min":0.69899999999999995},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.71399999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:11:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69699999999999995,"15min":0.68100000000000005},"min":{"1min":0.54000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.67500000000000004,"15min":0.65500000000000003},"min":{"1min":0.54300000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84199999999999997,"15min":0.82999999999999996},"min":{"1min":0.71999999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.83299999999999996}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.014},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.0089999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:11:53 2023","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.476,"5min":1.476,"15min":1.476},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.78600000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.75},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.88900000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888109,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":933888,"data_stored":441956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.48799999999999999},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45300000000000001}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.56699999999999995,"15min":0.55600000000000005},"min":{"1min":0.46300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84199999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.71999999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.48099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.751},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.0600000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.70499999999999996}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.48299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.80100000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:11:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.628,"15min":0.62},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88200000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.67600000000000005,"15min":0.66000000000000003},"min":{"1min":0.56799999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.67300000000000004}]},{"osd":7,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.69199999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:11:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.85199999999999998}]},{"osd":9,"last update":"Sat Jul 29 16:11:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.82699999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:12:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.88400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28460,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716692,"statfs":{"total":95995035648,"available":95965892608,"internally_reserved":0,"allocated":1544192,"data_stored":1033156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.55000000000000004}]},{"osd":1,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.71899999999999997}]},{"osd":2,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40600000000000003,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29799999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70799999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81299999999999994,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.72899999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.41699999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.754,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.76200000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:11:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.627,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.65300000000000002,"15min":0.64600000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.626},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79300000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.89500000000000002}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.80400000000000005}]},{"osd":9,"last update":"Sat Jul 29 16:11:55 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.878},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.91700000000000004}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.86599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":20615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:12:31.719 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-29T16:12:31.719 INFO:tasks.cephadm:Setting up client nodes... 2023-07-29T16:12:31.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-29T16:12:31.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:31 smithi039 ceph-mon[128858]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:31.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:31 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/465539526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:31 smithi031 ceph-mon[95504]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:31.810 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:31 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/465539526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:31.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:31 smithi156 ceph-mon[129245]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:12:31.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:31 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/465539526' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:32.475 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:32.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:32 smithi031 ceph-mon[95504]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:12:32.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:32 smithi031 ceph-mon[95504]: pgmap v277: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.1 KiB/s wr, 16 op/s 2023-07-29T16:12:32.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:32 smithi039 ceph-mon[128858]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:12:32.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:32 smithi039 ceph-mon[128858]: pgmap v277: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.1 KiB/s wr, 16 op/s 2023-07-29T16:12:32.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:32 smithi156 ceph-mon[129245]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:12:32.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:32 smithi156 ceph-mon[129245]: pgmap v277: 97 pgs: 97 active+clean; 581 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4.1 KiB/s wr, 16 op/s 2023-07-29T16:12:33.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:33 smithi031 ceph-mon[95504]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:12:33.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:33 smithi039 ceph-mon[128858]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:12:33.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:33 smithi156 ceph-mon[129245]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:12:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.014+0000 7fcdaa975700 1 -- 172.21.15.31:0/33077359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41015f0 msgr2=0x7fcda4101a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.014+0000 7fcdaa975700 1 --2- 172.21.15.31:0/33077359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41015f0 0x7fcda4101a40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009990 tx=0x7fcd98009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 -- 172.21.15.31:0/33077359 shutdown_connections 2023-07-29T16:12:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 --2- 172.21.15.31:0/33077359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcda4101f80 0x7fcda4108840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 --2- 172.21.15.31:0/33077359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcda41003f0 0x7fcda4100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 --2- 172.21.15.31:0/33077359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41015f0 0x7fcda4101a40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 -- 172.21.15.31:0/33077359 >> 172.21.15.31:0/33077359 conn(0x7fcda40fba40 msgr2=0x7fcda40fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 -- 172.21.15.31:0/33077359 shutdown_connections 2023-07-29T16:12:34.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.015+0000 7fcdaa975700 1 -- 172.21.15.31:0/33077359 wait complete. 2023-07-29T16:12:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 Processor -- start 2023-07-29T16:12:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 -- start start 2023-07-29T16:12:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 0x7fcda4196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcda41015f0 0x7fcda4196660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:34.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcda4101f80 0x7fcda419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda419dcb0 con 0x7fcda41003f0 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda419ddf0 con 0x7fcda41015f0 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.016+0000 7fcdaa975700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda419df80 con 0x7fcda4101f80 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 0x7fcda4196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 0x7fcda4196120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42114/0 (socket says 172.21.15.31:42114) 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 -- 172.21.15.31:0/1379872751 learned_addr learned my addr 172.21.15.31:0/1379872751 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:34.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda8f12700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcda4101f80 0x7fcda419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda37fe700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcda41015f0 0x7fcda4196660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 -- 172.21.15.31:0/1379872751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcda4101f80 msgr2=0x7fcda419d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcda4101f80 0x7fcda419d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 -- 172.21.15.31:0/1379872751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcda41015f0 msgr2=0x7fcda4196660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcda41015f0 0x7fcda4196660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 -- 172.21.15.31:0/1379872751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd980095f0 con 0x7fcda41003f0 2023-07-29T16:12:34.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda3fff700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 0x7fcda4196120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400e9a0 tx=0x7fcd9400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94009790 con 0x7fcda41003f0 2023-07-29T16:12:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.017+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd9400c810 con 0x7fcda41003f0 2023-07-29T16:12:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.018+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda419e260 con 0x7fcda41003f0 2023-07-29T16:12:34.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.018+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94003c70 con 0x7fcda41003f0 2023-07-29T16:12:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.018+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda419e760 con 0x7fcda41003f0 2023-07-29T16:12:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.019+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda410bc10 con 0x7fcda41003f0 2023-07-29T16:12:34.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.021+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fcd9400cc80 con 0x7fcda41003f0 2023-07-29T16:12:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.022+0000 7fcda17fa700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd8c0a0820 0x7fcd8c0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.022+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcd94014070 con 0x7fcda41003f0 2023-07-29T16:12:34.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.022+0000 7fcda37fe700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd8c0a0820 0x7fcd8c0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:34.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.023+0000 7fcda37fe700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd8c0a0820 0x7fcd8c0a2cd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800b2e0 tx=0x7fcd98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.026+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcd940957b0 con 0x7fcda41003f0 2023-07-29T16:12:34.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.316+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fcda4107300 con 0x7fcda41003f0 2023-07-29T16:12:34.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.320+0000 7fcda17fa700 1 -- 172.21.15.31:0/1379872751 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fcd94003dd0 con 0x7fcda41003f0 2023-07-29T16:12:34.317 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-07-29T16:12:34.318 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQByOsVkaGLkEhAAmWiTULs0a/9LPrOJ8rMgnw== 2023-07-29T16:12:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd8c0a0820 msgr2=0x7fcd8c0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd8c0a0820 0x7fcd8c0a2cd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800b2e0 tx=0x7fcd98005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 msgr2=0x7fcda4196120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:34.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 0x7fcda4196120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcd9400e9a0 tx=0x7fcd9400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 shutdown_connections 2023-07-29T16:12:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd8c0a0820 0x7fcd8c0a2cd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcda4101f80 0x7fcda419d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcda41003f0 0x7fcda4196120 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 --2- 172.21.15.31:0/1379872751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcda41015f0 0x7fcda4196660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:34.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 >> 172.21.15.31:0/1379872751 conn(0x7fcda40fba40 msgr2=0x7fcda40fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:34.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 shutdown_connections 2023-07-29T16:12:34.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:34.323+0000 7fcdaa975700 1 -- 172.21.15.31:0/1379872751 wait complete. 2023-07-29T16:12:34.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:34 smithi031 ceph-mon[95504]: pgmap v278: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.4 KiB/s wr, 16 op/s 2023-07-29T16:12:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:34 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1379872751' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T16:12:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:34 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1379872751' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T16:12:34.755 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:12:34.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-29T16:12:34.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-29T16:12:34.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:34 smithi039 ceph-mon[128858]: pgmap v278: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.4 KiB/s wr, 16 op/s 2023-07-29T16:12:34.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:34 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1379872751' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T16:12:34.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:34 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1379872751' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T16:12:34.819 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:34 smithi156 ceph-mon[129245]: pgmap v278: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 7.4 KiB/s wr, 16 op/s 2023-07-29T16:12:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:34 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1379872751' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-29T16:12:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:34 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1379872751' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-29T16:12:34.858 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-29T16:12:34.859 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-29T16:12:34.859 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph mgr dump --format=json 2023-07-29T16:12:35.634 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:36.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:36 smithi039 ceph-mon[128858]: pgmap v279: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2023-07-29T16:12:36.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:36 smithi156 ceph-mon[129245]: pgmap v279: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2023-07-29T16:12:36.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:36 smithi031 ceph-mon[95504]: pgmap v279: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2023-07-29T16:12:37.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 -- 172.21.15.31:0/1860609953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100fe0 msgr2=0x7fade0104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:37.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 --2- 172.21.15.31:0/1860609953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100fe0 0x7fade0104fb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fadd40099c0 tx=0x7fadd4009cd0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 -- 172.21.15.31:0/1860609953 shutdown_connections 2023-07-29T16:12:37.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 --2- 172.21.15.31:0/1860609953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fade01056e0 0x7fade010d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 --2- 172.21.15.31:0/1860609953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100640 0x7fade0100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 --2- 172.21.15.31:0/1860609953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100fe0 0x7fade0104fb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 -- 172.21.15.31:0/1860609953 >> 172.21.15.31:0/1860609953 conn(0x7fade00fbea0 msgr2=0x7fade00fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:37.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 -- 172.21.15.31:0/1860609953 shutdown_connections 2023-07-29T16:12:37.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.130+0000 7fade4a41700 1 -- 172.21.15.31:0/1860609953 wait complete. 2023-07-29T16:12:37.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.131+0000 7fade4a41700 1 Processor -- start 2023-07-29T16:12:37.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.131+0000 7fade4a41700 1 -- start start 2023-07-29T16:12:37.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fade4a41700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 0x7fade0196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:37.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fade4a41700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100fe0 0x7fade01969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fade4a41700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fade01056e0 0x7fade019ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fade4a41700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade019b440 con 0x7fade0100640 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fade4a41700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade019b5b0 con 0x7fade0100fe0 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fade4a41700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade019b770 con 0x7fade01056e0 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadde59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 0x7fade0196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100fe0 0x7fade01969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:37.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadde59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 0x7fade0196480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42126/0 (socket says 172.21.15.31:42126) 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadde59c700 1 -- 172.21.15.31:0/2930869776 learned_addr learned my addr 172.21.15.31:0/2930869776 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadded9d700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fade01056e0 0x7fade019ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadddd9b700 1 -- 172.21.15.31:0/2930869776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fade01056e0 msgr2=0x7fade019ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadddd9b700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fade01056e0 0x7fade019ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadddd9b700 1 -- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 msgr2=0x7fade0196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadddd9b700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 0x7fade0196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadddd9b700 1 -- 172.21.15.31:0/2930869776 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd40095f0 con 0x7fade0100fe0 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.132+0000 7fadde59c700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 0x7fade0196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.133+0000 7fadddd9b700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100fe0 0x7fade01969c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fadd400b580 tx=0x7fadd400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.133+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd401d070 con 0x7fade0100fe0 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.133+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade019b9f0 con 0x7fade0100fe0 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.133+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadd4022470 con 0x7fade0100fe0 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.133+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd4017650 con 0x7fade0100fe0 2023-07-29T16:12:37.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.133+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01a32b0 con 0x7fade0100fe0 2023-07-29T16:12:37.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.134+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade004ea50 con 0x7fade0100fe0 2023-07-29T16:12:37.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.136+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fadd4022890 con 0x7fade0100fe0 2023-07-29T16:12:37.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.137+0000 7fadcf7fe700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fadc80a0820 0x7fadc80a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:37.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.137+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fadd40ccb10 con 0x7fade0100fe0 2023-07-29T16:12:37.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.138+0000 7fadde59c700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fadc80a0820 0x7fadc80a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:37.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.138+0000 7fadde59c700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fadc80a0820 0x7fadc80a2cd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fadd0011550 tx=0x7fadd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:37.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.140+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fadd4096590 con 0x7fade0100fe0 2023-07-29T16:12:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.402+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fade01a3440 con 0x7fade0100fe0 2023-07-29T16:12:37.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.407+0000 7fadcf7fe700 1 -- 172.21.15.31:0/2930869776 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+258554 (secure 0 0 0) 0x7fadd4027090 con 0x7fade0100fe0 2023-07-29T16:12:37.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fadc80a0820 msgr2=0x7fadc80a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fadc80a0820 0x7fadc80a2cd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fadd0011550 tx=0x7fadd0009260 comp rx=0 tx=0).stop 2023-07-29T16:12:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100fe0 msgr2=0x7fade01969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100fe0 0x7fade01969c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fadd400b580 tx=0x7fadd400f690 comp rx=0 tx=0).stop 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 shutdown_connections 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fadc80a0820 0x7fadc80a2cd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fade01056e0 0x7fade019ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fade0100640 0x7fade0196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 --2- 172.21.15.31:0/2930869776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fade0100fe0 0x7fade01969c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 >> 172.21.15.31:0/2930869776 conn(0x7fade00fbea0 msgr2=0x7fade00fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:37.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 shutdown_connections 2023-07-29T16:12:37.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:37.412+0000 7fade4a41700 1 -- 172.21.15.31:0/2930869776 wait complete. 2023-07-29T16:12:37.810 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":4163929785},{"type":"v1","addr":"172.21.15.31:6801","nonce":4163929785}]},"active_addr":"172.21.15.31:6801/4163929785","active_change":"2023-07-29T16:05:49.191983+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2384937690}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2401285902}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1792039579}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3868017039}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":4286930225}]}]} 2023-07-29T16:12:37.817 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-29T16:12:37.817 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-29T16:12:37.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd dump --format=json 2023-07-29T16:12:38.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:37 smithi039 ceph-mon[128858]: pgmap v280: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2023-07-29T16:12:38.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:37 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2930869776' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-29T16:12:38.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:37 smithi156 ceph-mon[129245]: pgmap v280: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2023-07-29T16:12:38.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:37 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2930869776' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-29T16:12:38.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:37 smithi031 ceph-mon[95504]: pgmap v280: 97 pgs: 97 active+clean; 621 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 5.8 KiB/s wr, 13 op/s 2023-07-29T16:12:38.198 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:37 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2930869776' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-29T16:12:38.552 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:40.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 -- 172.21.15.31:0/832549919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100640 msgr2=0x7fb270100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:40.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 --2- 172.21.15.31:0/832549919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100640 0x7fb270100a10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb26c0099c0 tx=0x7fb26c009cd0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 -- 172.21.15.31:0/832549919 shutdown_connections 2023-07-29T16:12:40.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 --2- 172.21.15.31:0/832549919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 0x7fb27010d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 --2- 172.21.15.31:0/832549919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100640 0x7fb270100a10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 --2- 172.21.15.31:0/832549919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb270100fe0 0x7fb270104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 -- 172.21.15.31:0/832549919 >> 172.21.15.31:0/832549919 conn(0x7fb2700fbea0 msgr2=0x7fb2700fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:40.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 -- 172.21.15.31:0/832549919 shutdown_connections 2023-07-29T16:12:40.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.053+0000 7fb277406700 1 -- 172.21.15.31:0/832549919 wait complete. 2023-07-29T16:12:40.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.054+0000 7fb277406700 1 Processor -- start 2023-07-29T16:12:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 -- start start 2023-07-29T16:12:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb270100640 0x7fb270073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 0x7fb270073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 0x7fb270073b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2701a4910 con 0x7fb270100fe0 2023-07-29T16:12:40.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2701a4a80 con 0x7fb2701056e0 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb277406700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2701a4c40 con 0x7fb270100640 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb2749a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 0x7fb270073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb2759a3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 0x7fb270073b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb2749a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 0x7fb270073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58250/0 (socket says 172.21.15.31:58250) 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb2749a1700 1 -- 172.21.15.31:0/2220151269 learned_addr learned my addr 172.21.15.31:0/2220151269 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.055+0000 7fb2751a2700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb270100640 0x7fb270073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:40.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2749a1700 1 -- 172.21.15.31:0/2220151269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb270100640 msgr2=0x7fb270073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2749a1700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb270100640 0x7fb270073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2749a1700 1 -- 172.21.15.31:0/2220151269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 msgr2=0x7fb270073b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2749a1700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 0x7fb270073b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2749a1700 1 -- 172.21.15.31:0/2220151269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c0095f0 con 0x7fb270100fe0 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2759a3700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 0x7fb270073b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2749a1700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 0x7fb270073590 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb26000e970 tx=0x7fb26000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260009790 con 0x7fb270100fe0 2023-07-29T16:12:40.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb26000c7e0 con 0x7fb270100fe0 2023-07-29T16:12:40.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2701a4f20 con 0x7fb270100fe0 2023-07-29T16:12:40.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2600054e0 con 0x7fb270100fe0 2023-07-29T16:12:40.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.056+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2701a5390 con 0x7fb270100fe0 2023-07-29T16:12:40.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.057+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27004ea50 con 0x7fb270100fe0 2023-07-29T16:12:40.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.059+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fb26000c950 con 0x7fb270100fe0 2023-07-29T16:12:40.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.062+0000 7fb2667fc700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb25c0a0820 0x7fb25c0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:40.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.062+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb260014070 con 0x7fb270100fe0 2023-07-29T16:12:40.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.062+0000 7fb2751a2700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb25c0a0820 0x7fb25c0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:40.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.063+0000 7fb2751a2700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb25c0a0820 0x7fb25c0a2cd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb26c000c00 tx=0x7fb26c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:40.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.066+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb260095850 con 0x7fb270100fe0 2023-07-29T16:12:40.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.272+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb270074580 con 0x7fb270100fe0 2023-07-29T16:12:40.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.274+0000 7fb2667fc700 1 -- 172.21.15.31:0/2220151269 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22090 (secure 0 0 0) 0x7fb260098e70 con 0x7fb270100fe0 2023-07-29T16:12:40.272 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:40.272 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:22.914308+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:12:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb25c0a0820 msgr2=0x7fb25c0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb25c0a0820 0x7fb25c0a2cd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb26c000c00 tx=0x7fb26c019040 comp rx=0 tx=0).stop 2023-07-29T16:12:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 msgr2=0x7fb270073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 0x7fb270073590 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb26000e970 tx=0x7fb26000ec80 comp rx=0 tx=0).stop 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 shutdown_connections 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb25c0a0820 0x7fb25c0a2cd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2701056e0 0x7fb270073b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb270100640 0x7fb270073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 --2- 172.21.15.31:0/2220151269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb270100fe0 0x7fb270073590 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:40.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 >> 172.21.15.31:0/2220151269 conn(0x7fb2700fbea0 msgr2=0x7fb270103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 shutdown_connections 2023-07-29T16:12:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:40.277+0000 7fb277406700 1 -- 172.21.15.31:0/2220151269 wait complete. 2023-07-29T16:12:40.650 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:40 smithi031 ceph-mon[95504]: pgmap v281: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2023-07-29T16:12:40.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:40 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2220151269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:40.732 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-29T16:12:40.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd dump --format=json 2023-07-29T16:12:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:40 smithi039 ceph-mon[128858]: pgmap v281: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2023-07-29T16:12:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:40 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2220151269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:40.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:40 smithi156 ceph-mon[129245]: pgmap v281: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.2 KiB/s wr, 16 op/s 2023-07-29T16:12:40.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:40 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2220151269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:41.466 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:42.594 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:42 smithi031 ceph-mon[95504]: pgmap v282: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-07-29T16:12:42.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:42 smithi039 ceph-mon[128858]: pgmap v282: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-07-29T16:12:42.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:42 smithi156 ceph-mon[129245]: pgmap v282: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-07-29T16:12:42.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 -- 172.21.15.31:0/1855887491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8020100fe0 msgr2=0x7f8020104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1855887491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8020100fe0 0x7f8020104fb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f800c0099c0 tx=0x7f800c009cd0 comp rx=0 tx=0).stop 2023-07-29T16:12:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 -- 172.21.15.31:0/1855887491 shutdown_connections 2023-07-29T16:12:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1855887491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80201056e0 0x7f802010d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1855887491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100640 0x7f8020100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1855887491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8020100fe0 0x7f8020104fb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:42.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 -- 172.21.15.31:0/1855887491 >> 172.21.15.31:0/1855887491 conn(0x7f80200fbea0 msgr2=0x7f80200fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:42.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 -- 172.21.15.31:0/1855887491 shutdown_connections 2023-07-29T16:12:42.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.862+0000 7f8026e7d700 1 -- 172.21.15.31:0/1855887491 wait complete. 2023-07-29T16:12:42.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.863+0000 7f8026e7d700 1 Processor -- start 2023-07-29T16:12:42.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.863+0000 7f8026e7d700 1 -- start start 2023-07-29T16:12:42.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8026e7d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8020100640 0x7f8020073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8026e7d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 0x7f8020073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8026e7d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80201056e0 0x7f8020073b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8026e7d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80201a4940 con 0x7f80201056e0 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8026e7d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80201a4ab0 con 0x7f8020100fe0 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8026e7d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80201a4c70 con 0x7f8020100640 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f801ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 0x7f8020073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:42.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f8024c19700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8020100640 0x7f8020073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f801ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 0x7f8020073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60188/0 (socket says 172.21.15.31:60188) 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f801ffff700 1 -- 172.21.15.31:0/1657680518 learned_addr learned my addr 172.21.15.31:0/1657680518 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.864+0000 7f802541a700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80201056e0 0x7f8020073b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801ffff700 1 -- 172.21.15.31:0/1657680518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8020100640 msgr2=0x7f8020073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801ffff700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8020100640 0x7f8020073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801ffff700 1 -- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80201056e0 msgr2=0x7f8020073b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:42.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801ffff700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80201056e0 0x7f8020073b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801ffff700 1 -- 172.21.15.31:0/1657680518 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0095f0 con 0x7f8020100fe0 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f802541a700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80201056e0 0x7f8020073b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801ffff700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 0x7f8020073590 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f800c0098e0 tx=0x7f800c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c01d070 con 0x7f8020100fe0 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80201a4ef0 con 0x7f8020100fe0 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f800c022470 con 0x7f8020100fe0 2023-07-29T16:12:42.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.865+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c017650 con 0x7f8020100fe0 2023-07-29T16:12:42.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.866+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80201a5300 con 0x7f8020100fe0 2023-07-29T16:12:42.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.867+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802004ea50 con 0x7f8020100fe0 2023-07-29T16:12:42.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.869+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f800c022850 con 0x7f8020100fe0 2023-07-29T16:12:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.870+0000 7f801dffb700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f80100a0820 0x7f80100a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.870+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f800c0ccad0 con 0x7f8020100fe0 2023-07-29T16:12:42.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.870+0000 7f8024c19700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f80100a0820 0x7f80100a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:42.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.871+0000 7f8024c19700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f80100a0820 0x7f80100a2cd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8014009c60 tx=0x7f8014009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:42.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:42.872+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f800c096660 con 0x7f8020100fe0 2023-07-29T16:12:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.080+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80201a55f0 con 0x7f8020100fe0 2023-07-29T16:12:43.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.082+0000 7f801dffb700 1 -- 172.21.15.31:0/1657680518 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22090 (secure 0 0 0) 0x7f800c099c80 con 0x7f8020100fe0 2023-07-29T16:12:43.080 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:12:43.080 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":86,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:22.914308+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:12:43.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f80100a0820 msgr2=0x7f80100a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:43.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f80100a0820 0x7f80100a2cd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8014009c60 tx=0x7f8014009260 comp rx=0 tx=0).stop 2023-07-29T16:12:43.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 msgr2=0x7f8020073590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:43.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 0x7f8020073590 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f800c0098e0 tx=0x7f800c00f690 comp rx=0 tx=0).stop 2023-07-29T16:12:43.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 shutdown_connections 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f80100a0820 0x7f80100a2cd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80201056e0 0x7f8020073b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8020100640 0x7f8020073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 --2- 172.21.15.31:0/1657680518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8020100fe0 0x7f8020073590 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 >> 172.21.15.31:0/1657680518 conn(0x7f80200fbea0 msgr2=0x7f80200fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 shutdown_connections 2023-07-29T16:12:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:43.085+0000 7f8026e7d700 1 -- 172.21.15.31:0/1657680518 wait complete. 2023-07-29T16:12:43.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:43 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1657680518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:43.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-29T16:12:43.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-29T16:12:43.545 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-29T16:12:43.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-29T16:12:43.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-29T16:12:43.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-29T16:12:43.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-29T16:12:43.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-29T16:12:43.548 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-29T16:12:43.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-29T16:12:43.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-29T16:12:43.550 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-29T16:12:43.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:43 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1657680518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:43 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1657680518' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:12:44.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:44 smithi031 ceph-mon[95504]: pgmap v283: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-07-29T16:12:44.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:44 smithi039 ceph-mon[128858]: pgmap v283: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-07-29T16:12:44.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:44 smithi156 ceph-mon[129245]: pgmap v283: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.2 KiB/s wr, 9 op/s 2023-07-29T16:12:45.079 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:45.701 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:46.055 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:46.348 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:46.583 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:46.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:46 smithi031 ceph-mon[95504]: pgmap v284: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-07-29T16:12:46.704 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:46 smithi039 ceph-mon[128858]: pgmap v284: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-07-29T16:12:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:46 smithi156 ceph-mon[129245]: pgmap v284: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-07-29T16:12:47.019 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:47.189 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:47.189 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.222+0000 7f894d685700 1 -- 172.21.15.31:0/383879230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89481056e0 msgr2=0x7f894810d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.222+0000 7f894d685700 1 --2- 172.21.15.31:0/383879230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89481056e0 0x7f894810d260 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f894000b0d0 tx=0x7f894000b3e0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 -- 172.21.15.31:0/383879230 shutdown_connections 2023-07-29T16:12:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 --2- 172.21.15.31:0/383879230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89481056e0 0x7f894810d260 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 --2- 172.21.15.31:0/383879230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 0x7f8948100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 --2- 172.21.15.31:0/383879230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8948100fe0 0x7f8948104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 -- 172.21.15.31:0/383879230 >> 172.21.15.31:0/383879230 conn(0x7f89480fbea0 msgr2=0x7f89480fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 -- 172.21.15.31:0/383879230 shutdown_connections 2023-07-29T16:12:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.223+0000 7f894d685700 1 -- 172.21.15.31:0/383879230 wait complete. 2023-07-29T16:12:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.224+0000 7f894d685700 1 Processor -- start 2023-07-29T16:12:47.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.224+0000 7f894d685700 1 -- start start 2023-07-29T16:12:47.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f894d685700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 0x7f894819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f894d685700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8948100fe0 0x7f894819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f894d685700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89481056e0 0x7f89481986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f894d685700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8948198d30 con 0x7f8948100fe0 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f894d685700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8948198ea0 con 0x7f8948100640 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f894d685700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8948199060 con 0x7f89481056e0 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f8946ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 0x7f894819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:47.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f8946ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 0x7f894819e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60204/0 (socket says 172.21.15.31:60204) 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f8946ffd700 1 -- 172.21.15.31:0/3199975384 learned_addr learned my addr 172.21.15.31:0/3199975384 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f89467fc700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8948100fe0 0x7f894819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.225+0000 7f89477fe700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89481056e0 0x7f89481986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f8946ffd700 1 -- 172.21.15.31:0/3199975384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89481056e0 msgr2=0x7f89481986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f8946ffd700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89481056e0 0x7f89481986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f8946ffd700 1 -- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8948100fe0 msgr2=0x7f894819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f8946ffd700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8948100fe0 0x7f894819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f8946ffd700 1 -- 172.21.15.31:0/3199975384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8940009d00 con 0x7f8948100640 2023-07-29T16:12:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f89467fc700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8948100fe0 0x7f894819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f8946ffd700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 0x7f894819e640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f893800e9a0 tx=0x7f893800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938009790 con 0x7f8948100640 2023-07-29T16:12:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8948199340 con 0x7f8948100640 2023-07-29T16:12:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f893800cbd0 con 0x7f8948100640 2023-07-29T16:12:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8938018680 con 0x7f8948100640 2023-07-29T16:12:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.226+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89481a5630 con 0x7f8948100640 2023-07-29T16:12:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.227+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f89481105a0 con 0x7f8948100640 2023-07-29T16:12:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.230+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f893800cd40 con 0x7f8948100640 2023-07-29T16:12:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.232+0000 7f892ffff700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89300a0820 0x7f89300a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.232+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8938014070 con 0x7f8948100640 2023-07-29T16:12:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.232+0000 7f89467fc700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89300a0820 0x7f89300a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.232+0000 7f892ffff700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] conn(0x7f89300a9ff0 0x7f89300ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.232+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 --> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f89300acab0 con 0x7f89300a9ff0 2023-07-29T16:12:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.232+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89380cc950 con 0x7f8948100640 2023-07-29T16:12:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.233+0000 7f89467fc700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89300a0820 0x7f89300a2cd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8948199b30 tx=0x7f893c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.233+0000 7f89477fe700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] conn(0x7f89300a9ff0 0x7f89300ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:47.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.234+0000 7f89477fe700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] conn(0x7f89300a9ff0 0x7f89300ac400 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:47.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.235+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== osd.4 v2:172.21.15.39:6808/1826445248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f89300acab0 con 0x7f89300a9ff0 2023-07-29T16:12:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:47.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.261+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 --> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f894804ea50 con 0x7f89300a9ff0 2023-07-29T16:12:47.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.262+0000 7f892ffff700 1 -- 172.21.15.31:0/3199975384 <== osd.4 v2:172.21.15.39:6808/1826445248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f894804ea50 con 0x7f89300a9ff0 2023-07-29T16:12:47.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] conn(0x7f89300a9ff0 msgr2=0x7f89300ac400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:47.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] conn(0x7f89300a9ff0 0x7f89300ac400 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:47.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89300a0820 msgr2=0x7f89300a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:47.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89300a0820 0x7f89300a2cd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8948199b30 tx=0x7f893c009260 comp rx=0 tx=0).stop 2023-07-29T16:12:47.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 msgr2=0x7f894819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:47.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 --2- 172.21.15.31:0/3199975384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8948100640 0x7f894819e640 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f893800e9a0 tx=0x7f893800ed60 comp rx=0 tx=0).stop 2023-07-29T16:12:47.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f89477fe700 1 -- 172.21.15.31:0/3199975384 reap_dead start 2023-07-29T16:12:47.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.264+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 shutdown_connections 2023-07-29T16:12:47.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.265+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 >> 172.21.15.31:0/3199975384 conn(0x7f89480fbea0 msgr2=0x7f89480fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:47.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.265+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 shutdown_connections 2023-07-29T16:12:47.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:47.265+0000 7f894d685700 1 -- 172.21.15.31:0/3199975384 wait complete. 2023-07-29T16:12:47.367 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:47.368 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:47.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:47 smithi031 ceph-mon[95504]: pgmap v285: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-07-29T16:12:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:47 smithi039 ceph-mon[128858]: pgmap v285: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-07-29T16:12:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:47 smithi156 ceph-mon[129245]: pgmap v285: 97 pgs: 97 active+clean; 634 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-07-29T16:12:49.394 INFO:teuthology.orchestra.run.smithi031.stdout:146028888113 2023-07-29T16:12:49.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-29T16:12:49.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-07-29T16:12:49.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2023-07-29T16:12:49.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2023-07-29T16:12:49.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2023-07-29T16:12:49.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]: dispatch 2023-07-29T16:12:49.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2023-07-29T16:12:49.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]: dispatch 2023-07-29T16:12:49.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-07-29T16:12:49.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-07-29T16:12:49.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-07-29T16:12:49.579 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-07-29T16:12:49.579 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2023-07-29T16:12:49.580 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2023-07-29T16:12:49.580 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2023-07-29T16:12:49.580 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]: dispatch 2023-07-29T16:12:49.580 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2023-07-29T16:12:49.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]: dispatch 2023-07-29T16:12:49.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-07-29T16:12:49.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-07-29T16:12:49.581 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-07-29T16:12:49.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]: dispatch 2023-07-29T16:12:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-07-29T16:12:49.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-07-29T16:12:49.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2023-07-29T16:12:50.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: pgmap v287: 97 pgs: 97 active+clean; 637 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-07-29T16:12:50.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-07-29T16:12:50.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2023-07-29T16:12:50.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2023-07-29T16:12:50.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2023-07-29T16:12:50.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]': finished 2023-07-29T16:12:50.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2023-07-29T16:12:50.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]': finished 2023-07-29T16:12:50.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-07-29T16:12:50.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-07-29T16:12:50.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-07-29T16:12:50.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: osdmap e87: 12 total, 12 up, 12 in 2023-07-29T16:12:50.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:50 smithi031 ceph-mon[95504]: osdmap e88: 12 total, 12 up, 12 in 2023-07-29T16:12:50.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: pgmap v287: 97 pgs: 97 active+clean; 637 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-07-29T16:12:50.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-07-29T16:12:50.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2023-07-29T16:12:50.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-07-29T16:12:50.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-07-29T16:12:50.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: osdmap e87: 12 total, 12 up, 12 in 2023-07-29T16:12:50.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:50 smithi156 ceph-mon[129245]: osdmap e88: 12 total, 12 up, 12 in 2023-07-29T16:12:50.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: pgmap v287: 97 pgs: 97 active+clean; 637 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-07-29T16:12:50.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]': finished 2023-07-29T16:12:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-07-29T16:12:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-07-29T16:12:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2023-07-29T16:12:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: osdmap e87: 12 total, 12 up, 12 in 2023-07-29T16:12:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:50 smithi039 ceph-mon[128858]: osdmap e88: 12 total, 12 up, 12 in 2023-07-29T16:12:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.366+0000 7f1e159c7700 1 -- 172.21.15.31:0/2315336283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e100ffd30 msgr2=0x7f1e10104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.366+0000 7f1e159c7700 1 --2- 172.21.15.31:0/2315336283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e100ffd30 0x7f1e10104210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1df8009990 tx=0x7f1df8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 -- 172.21.15.31:0/2315336283 shutdown_connections 2023-07-29T16:12:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 --2- 172.21.15.31:0/2315336283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e10104750 0x7f1e10104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 --2- 172.21.15.31:0/2315336283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e100ff420 0x7f1e100ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 --2- 172.21.15.31:0/2315336283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e100ffd30 0x7f1e10104210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 -- 172.21.15.31:0/2315336283 >> 172.21.15.31:0/2315336283 conn(0x7f1e100763b0 msgr2=0x7f1e100767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 -- 172.21.15.31:0/2315336283 shutdown_connections 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.367+0000 7f1e159c7700 1 -- 172.21.15.31:0/2315336283 wait complete. 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.368+0000 7f1e159c7700 1 Processor -- start 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.368+0000 7f1e159c7700 1 -- start start 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e159c7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e100ff420 0x7f1e1010c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e159c7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e100ffd30 0x7f1e1010c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e159c7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e10104750 0x7f1e101b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e159c7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101b1590 con 0x7f1e10104750 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e159c7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101b1700 con 0x7f1e100ffd30 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e159c7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e101b18c0 con 0x7f1e100ff420 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e0effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e100ff420 0x7f1e1010c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e0effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e100ff420 0x7f1e1010c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44098/0 (socket says 172.21.15.31:44098) 2023-07-29T16:12:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e0effd700 1 -- 172.21.15.31:0/1132993310 learned_addr learned my addr 172.21.15.31:0/1132993310 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.369+0000 7f1e0f7fe700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e10104750 0x7f1e101b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e07fff700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e100ffd30 0x7f1e1010c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e0effd700 1 -- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e100ffd30 msgr2=0x7f1e1010c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e0effd700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e100ffd30 0x7f1e1010c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e0effd700 1 -- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e10104750 msgr2=0x7f1e101b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e0effd700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e10104750 0x7f1e101b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e0effd700 1 -- 172.21.15.31:0/1132993310 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00009c40 con 0x7f1e100ff420 2023-07-29T16:12:50.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e0effd700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e100ff420 0x7f1e1010c2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000c370 tx=0x7f1e0000c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:50.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00017070 con 0x7f1e100ff420 2023-07-29T16:12:50.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.370+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df80095f0 con 0x7f1e100ff420 2023-07-29T16:12:50.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.371+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1e0000f040 con 0x7f1e100ff420 2023-07-29T16:12:50.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.371+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e101b1f00 con 0x7f1e100ff420 2023-07-29T16:12:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.371+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0000e050 con 0x7f1e100ff420 2023-07-29T16:12:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.372+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1df0000ff0 con 0x7f1e100ff420 2023-07-29T16:12:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.375+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f1e00003a00 con 0x7f1e100ff420 2023-07-29T16:12:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.377+0000 7f1e149c5700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1dfc0a08a0 0x7f1dfc0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.377+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f1e000cbc00 con 0x7f1e100ff420 2023-07-29T16:12:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.378+0000 7f1e07fff700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1dfc0a08a0 0x7f1dfc0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.378+0000 7f1e149c5700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] conn(0x7f1dfc0aa4d0 0x7f1dfc0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.378+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 --> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1dfc0ad030 con 0x7f1dfc0aa4d0 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.378+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e000cbf70 con 0x7f1e100ff420 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.378+0000 7f1e0f7fe700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] conn(0x7f1dfc0aa4d0 0x7f1dfc0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.378+0000 7f1e07fff700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1dfc0a08a0 0x7f1dfc0a2d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1df800b2e0 tx=0x7f1df8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.379+0000 7f1e0f7fe700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] conn(0x7f1dfc0aa4d0 0x7f1dfc0ac960 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.380+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== osd.1 v2:172.21.15.39:6800/701098917 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1dfc0ad030 con 0x7f1dfc0aa4d0 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.411+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 --> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1df0002d70 con 0x7f1dfc0aa4d0 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.411+0000 7f1e149c5700 1 -- 172.21.15.31:0/1132993310 <== osd.1 v2:172.21.15.39:6800/701098917 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1df0002d70 con 0x7f1dfc0aa4d0 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] conn(0x7f1dfc0aa4d0 msgr2=0x7f1dfc0ac960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] conn(0x7f1dfc0aa4d0 0x7f1dfc0ac960 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1dfc0a08a0 msgr2=0x7f1dfc0a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1dfc0a08a0 0x7f1dfc0a2d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1df800b2e0 tx=0x7f1df8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e100ff420 msgr2=0x7f1e1010c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 --2- 172.21.15.31:0/1132993310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e100ff420 0x7f1e1010c2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e0000c370 tx=0x7f1e0000c680 comp rx=0 tx=0).stop 2023-07-29T16:12:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e0f7fe700 1 -- 172.21.15.31:0/1132993310 reap_dead start 2023-07-29T16:12:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 shutdown_connections 2023-07-29T16:12:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 >> 172.21.15.31:0/1132993310 conn(0x7f1e100763b0 msgr2=0x7f1e10102230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 shutdown_connections 2023-07-29T16:12:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:50.412+0000 7f1e159c7700 1 -- 172.21.15.31:0/1132993310 wait complete. 2023-07-29T16:12:52.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 -- 172.21.15.31:0/2635648593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101f40 msgr2=0x7f505c10a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 --2- 172.21.15.31:0/2635648593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101f40 0x7f505c10a440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f50500098f0 tx=0x7f5050009c00 comp rx=0 tx=0).stop 2023-07-29T16:12:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 -- 172.21.15.31:0/2635648593 shutdown_connections 2023-07-29T16:12:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 --2- 172.21.15.31:0/2635648593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f505c10a980 0x7f505c10cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 --2- 172.21.15.31:0/2635648593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f505c101630 0x7f505c101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 --2- 172.21.15.31:0/2635648593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101f40 0x7f505c10a440 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 -- 172.21.15.31:0/2635648593 >> 172.21.15.31:0/2635648593 conn(0x7f505c0763b0 msgr2=0x7f505c0767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:52.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 -- 172.21.15.31:0/2635648593 shutdown_connections 2023-07-29T16:12:52.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.021+0000 7f5060e32700 1 -- 172.21.15.31:0/2635648593 wait complete. 2023-07-29T16:12:52.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.022+0000 7f5060e32700 1 Processor -- start 2023-07-29T16:12:52.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.022+0000 7f5060e32700 1 -- start start 2023-07-29T16:12:52.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f5060e32700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101630 0x7f505c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f5060e32700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f505c101f40 0x7f505c198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f5060e32700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f505c10a980 0x7f505c19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f5060e32700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c19c610 con 0x7f505c101f40 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f5060e32700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c19c780 con 0x7f505c10a980 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f5060e32700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c19c940 con 0x7f505c101630 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f505a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101630 0x7f505c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f505a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101630 0x7f505c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44114/0 (socket says 172.21.15.31:44114) 2023-07-29T16:12:52.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.023+0000 7f505a59c700 1 -- 172.21.15.31:0/2088580672 learned_addr learned my addr 172.21.15.31:0/2088580672 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f5059d9b700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f505c101f40 0x7f505c198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f505a59c700 1 -- 172.21.15.31:0/2088580672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f505c10a980 msgr2=0x7f505c19c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f505a59c700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f505c10a980 0x7f505c19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f505a59c700 1 -- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f505c101f40 msgr2=0x7f505c198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f505a59c700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f505c101f40 0x7f505c198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f505a59c700 1 -- 172.21.15.31:0/2088580672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050009520 con 0x7f505c101630 2023-07-29T16:12:52.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f5059d9b700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f505c101f40 0x7f505c198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:52.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.024+0000 7f505a59c700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101630 0x7f505c198710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f504c00e9a0 tx=0x7f504c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.025+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c009790 con 0x7f505c101630 2023-07-29T16:12:52.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.025+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f504c00cb50 con 0x7f505c101630 2023-07-29T16:12:52.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.025+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c018680 con 0x7f505c101630 2023-07-29T16:12:52.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.025+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505c19cc20 con 0x7f505c101630 2023-07-29T16:12:52.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.025+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505c1a5570 con 0x7f505c101630 2023-07-29T16:12:52.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.026+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f505c04ea50 con 0x7f505c101630 2023-07-29T16:12:52.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.028+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f504c00ccc0 con 0x7f505c101630 2023-07-29T16:12:52.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.030+0000 7f504b7fe700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50440a08a0 0x7f50440a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.030+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f504c014070 con 0x7f505c101630 2023-07-29T16:12:52.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.030+0000 7f5059d9b700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50440a08a0 0x7f50440a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.030+0000 7f504b7fe700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] conn(0x7f50440aa4d0 0x7f50440ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.031+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 --> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50440ad030 con 0x7f50440aa4d0 2023-07-29T16:12:52.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.031+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f504c044ea0 con 0x7f505c101630 2023-07-29T16:12:52.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.031+0000 7f505ad9d700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] conn(0x7f50440aa4d0 0x7f50440ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.031+0000 7f5059d9b700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50440a08a0 0x7f50440a2d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f505c199e30 tx=0x7f5050019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.032+0000 7f505ad9d700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] conn(0x7f50440aa4d0 0x7f50440ac960 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.033+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== osd.2 v2:172.21.15.156:6800/3425187698 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f50440ad030 con 0x7f50440aa4d0 2023-07-29T16:12:52.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.051+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 --> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f505c1105c0 con 0x7f50440aa4d0 2023-07-29T16:12:52.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.052+0000 7f504b7fe700 1 -- 172.21.15.31:0/2088580672 <== osd.2 v2:172.21.15.156:6800/3425187698 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5054004bb0 con 0x7f50440aa4d0 2023-07-29T16:12:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.052+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] conn(0x7f50440aa4d0 msgr2=0x7f50440ac960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.052+0000 7f5060e32700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] conn(0x7f50440aa4d0 0x7f50440ac960 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50440a08a0 msgr2=0x7f50440a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50440a08a0 0x7f50440a2d50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f505c199e30 tx=0x7f5050019040 comp rx=0 tx=0).stop 2023-07-29T16:12:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101630 msgr2=0x7f505c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 --2- 172.21.15.31:0/2088580672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f505c101630 0x7f505c198710 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f504c00e9a0 tx=0x7f504c00ed60 comp rx=0 tx=0).stop 2023-07-29T16:12:52.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f505ad9d700 1 -- 172.21.15.31:0/2088580672 reap_dead start 2023-07-29T16:12:52.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 shutdown_connections 2023-07-29T16:12:52.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 >> 172.21.15.31:0/2088580672 conn(0x7f505c0763b0 msgr2=0x7f505c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:52.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 shutdown_connections 2023-07-29T16:12:52.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.053+0000 7f5060e32700 1 -- 172.21.15.31:0/2088580672 wait complete. 2023-07-29T16:12:52.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:52 smithi031 ceph-mon[95504]: pgmap v289: 97 pgs: 97 active+clean; 637 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 383 B/s wr, 4 op/s 2023-07-29T16:12:52.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.492+0000 7f90c2680700 1 -- 172.21.15.31:0/845615855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc100fe0 msgr2=0x7f90bc104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.492+0000 7f90c2680700 1 --2- 172.21.15.31:0/845615855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc100fe0 0x7f90bc104fb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90b00099c0 tx=0x7f90b0009cd0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.492+0000 7f90c2680700 1 -- 172.21.15.31:0/845615855 shutdown_connections 2023-07-29T16:12:52.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.492+0000 7f90c2680700 1 --2- 172.21.15.31:0/845615855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc1056e0 0x7f90bc10d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.492+0000 7f90c2680700 1 --2- 172.21.15.31:0/845615855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90bc100640 0x7f90bc100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.493+0000 7f90c2680700 1 --2- 172.21.15.31:0/845615855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc100fe0 0x7f90bc104fb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.493+0000 7f90c2680700 1 -- 172.21.15.31:0/845615855 >> 172.21.15.31:0/845615855 conn(0x7f90bc0fbea0 msgr2=0x7f90bc0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:52.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.493+0000 7f90c2680700 1 -- 172.21.15.31:0/845615855 shutdown_connections 2023-07-29T16:12:52.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.493+0000 7f90c2680700 1 -- 172.21.15.31:0/845615855 wait complete. 2023-07-29T16:12:52.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 Processor -- start 2023-07-29T16:12:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 -- start start 2023-07-29T16:12:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc100640 0x7f90bc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90bc100fe0 0x7f90bc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc1056e0 0x7f90bc19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc19c5d0 con 0x7f90bc1056e0 2023-07-29T16:12:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc19c740 con 0x7f90bc100640 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.494+0000 7f90c2680700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc19c900 con 0x7f90bc100fe0 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc100640 0x7f90bc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc100640 0x7f90bc1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37716/0 (socket says 172.21.15.31:37716) 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 -- 172.21.15.31:0/127330442 learned_addr learned my addr 172.21.15.31:0/127330442 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bb7fe700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90bc100fe0 0x7f90bc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90c0c1d700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc1056e0 0x7f90bc19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 -- 172.21.15.31:0/127330442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90bc100fe0 msgr2=0x7f90bc198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90bc100fe0 0x7f90bc198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 -- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc1056e0 msgr2=0x7f90bc19bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc1056e0 0x7f90bc19bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 -- 172.21.15.31:0/127330442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00095f0 con 0x7f90bc100640 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90c0c1d700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90bc1056e0 0x7f90bc19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.495+0000 7f90bbfff700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc100640 0x7f90bc1986d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00e9a0 tx=0x7f90ac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.496+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac009790 con 0x7f90bc100640 2023-07-29T16:12:52.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.496+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90ac00cbd0 con 0x7f90bc100640 2023-07-29T16:12:52.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.496+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc19cbe0 con 0x7f90bc100640 2023-07-29T16:12:52.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.496+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac018680 con 0x7f90bc100640 2023-07-29T16:12:52.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.496+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc1a5530 con 0x7f90bc100640 2023-07-29T16:12:52.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.497+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f90bc04ea50 con 0x7f90bc100640 2023-07-29T16:12:52.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.499+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f90ac00cd40 con 0x7f90bc100640 2023-07-29T16:12:52.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.500+0000 7f90b97fa700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90a40a0820 0x7f90a40a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.500+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f90ac014070 con 0x7f90bc100640 2023-07-29T16:12:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.500+0000 7f90bb7fe700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90a40a0820 0x7f90a40a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.501+0000 7f90b97fa700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] conn(0x7f90a40aa450 0x7f90a40ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.501+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 --> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90a40acfb0 con 0x7f90a40aa450 2023-07-29T16:12:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.501+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90ac0cc750 con 0x7f90bc100640 2023-07-29T16:12:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.501+0000 7f90c0c1d700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] conn(0x7f90a40aa450 0x7f90a40ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.501+0000 7f90bb7fe700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90a40a0820 0x7f90a40a2cd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f90b00098e0 tx=0x7f90b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.501+0000 7f90c0c1d700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] conn(0x7f90a40aa450 0x7f90a40ac8e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.502+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== osd.0 v2:172.21.15.31:6802/2308009899 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f90a40acfb0 con 0x7f90a40aa450 2023-07-29T16:12:52.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.519+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 --> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f90bc1105d0 con 0x7f90a40aa450 2023-07-29T16:12:52.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.519+0000 7f90b97fa700 1 -- 172.21.15.31:0/127330442 <== osd.0 v2:172.21.15.31:6802/2308009899 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f90b400f040 con 0x7f90a40aa450 2023-07-29T16:12:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.519+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] conn(0x7f90a40aa450 msgr2=0x7f90a40ac8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.519+0000 7f90c2680700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] conn(0x7f90a40aa450 0x7f90a40ac8e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.519+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90a40a0820 msgr2=0x7f90a40a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.519+0000 7f90c2680700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90a40a0820 0x7f90a40a2cd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f90b00098e0 tx=0x7f90b0000bc0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc100640 msgr2=0x7f90bc1986d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c2680700 1 --2- 172.21.15.31:0/127330442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90bc100640 0x7f90bc1986d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00e9a0 tx=0x7f90ac00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c0c1d700 1 -- 172.21.15.31:0/127330442 reap_dead start 2023-07-29T16:12:52.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 shutdown_connections 2023-07-29T16:12:52.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 >> 172.21.15.31:0/127330442 conn(0x7f90bc0fbea0 msgr2=0x7f90bc0fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:52.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 shutdown_connections 2023-07-29T16:12:52.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.520+0000 7f90c2680700 1 -- 172.21.15.31:0/127330442 wait complete. 2023-07-29T16:12:52.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:52 smithi156 ceph-mon[129245]: pgmap v289: 97 pgs: 97 active+clean; 637 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 383 B/s wr, 4 op/s 2023-07-29T16:12:52.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:52 smithi039 ceph-mon[128858]: pgmap v289: 97 pgs: 97 active+clean; 637 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 383 B/s wr, 4 op/s 2023-07-29T16:12:52.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/1344717107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 msgr2=0x7fcdf0100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/1344717107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 0x7fcdf0100a10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009920 tx=0x7fcdec009c30 comp rx=0 tx=0).stop 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/1344717107 shutdown_connections 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/1344717107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf01056e0 0x7fcdf010d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/1344717107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 0x7fcdf0100a10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/1344717107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf0100fe0 0x7fcdf0104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/1344717107 >> 172.21.15.31:0/1344717107 conn(0x7fcdf00fbea0 msgr2=0x7fcdf00fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/1344717107 shutdown_connections 2023-07-29T16:12:52.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.843+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/1344717107 wait complete. 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.845+0000 7fcdf7ce0700 1 Processor -- start 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.845+0000 7fcdf7ce0700 1 -- start start 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.845+0000 7fcdf7ce0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 0x7fcdf019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.845+0000 7fcdf7ce0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf0100fe0 0x7fcdf019eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf7ce0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf01056e0 0x7fcdf0198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf7ce0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf0198cb0 con 0x7fcdf0100fe0 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf7ce0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf0198e20 con 0x7fcdf01056e0 2023-07-29T16:12:52.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf7ce0700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf0198fe0 con 0x7fcdf0100640 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf5a7c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 0x7fcdf019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf01056e0 0x7fcdf0198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf01056e0 0x7fcdf0198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37736/0 (socket says 172.21.15.31:37736) 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 -- 172.21.15.31:0/2973614876 learned_addr learned my addr 172.21.15.31:0/2973614876 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf527b700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf0100fe0 0x7fcdf019eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 -- 172.21.15.31:0/2973614876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 msgr2=0x7fcdf019e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcdf0100640 0x7fcdf019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 -- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf0100fe0 msgr2=0x7fcdf019eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf0100fe0 0x7fcdf019eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.846+0000 7fcdf627d700 1 -- 172.21.15.31:0/2973614876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009520 con 0x7fcdf01056e0 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.847+0000 7fcdf627d700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf01056e0 0x7fcdf0198640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcde800ef30 tx=0x7fcde800c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.847+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8018070 con 0x7fcdf01056e0 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.847+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcde800f040 con 0x7fcdf01056e0 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.847+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde8003dc0 con 0x7fcdf01056e0 2023-07-29T16:12:52.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.847+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf0199260 con 0x7fcdf01056e0 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.847+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf01a55b0 con 0x7fcdf01056e0 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.850+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcdd4000ff0 con 0x7fcdf01056e0 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.851+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fcde800caa0 con 0x7fcdf01056e0 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.853+0000 7fcde6ffd700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcddc0a08a0 0x7fcddc0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.853+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fcde80cb690 con 0x7fcdf01056e0 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.853+0000 7fcde6ffd700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] conn(0x7fcddc0aa4d0 0x7fcddc0ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:52.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.853+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 --> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcddc0ad030 con 0x7fcddc0aa4d0 2023-07-29T16:12:52.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.853+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcde80cdc70 con 0x7fcdf01056e0 2023-07-29T16:12:52.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.854+0000 7fcdf5a7c700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcddc0a08a0 0x7fcddc0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.855+0000 7fcdf527b700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] conn(0x7fcddc0aa4d0 0x7fcddc0ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:52.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.855+0000 7fcdf5a7c700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcddc0a08a0 0x7fcddc0a2d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0094f0 tx=0x7fcdec009440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.856+0000 7fcdf527b700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] conn(0x7fcddc0aa4d0 0x7fcddc0ac960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.857+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== osd.3 v2:172.21.15.31:6810/1855156941 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fcddc0ad030 con 0x7fcddc0aa4d0 2023-07-29T16:12:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.875+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 --> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcdd4002d70 con 0x7fcddc0aa4d0 2023-07-29T16:12:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.875+0000 7fcde6ffd700 1 -- 172.21.15.31:0/2973614876 <== osd.3 v2:172.21.15.31:6810/1855156941 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcdd4002d70 con 0x7fcddc0aa4d0 2023-07-29T16:12:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] conn(0x7fcddc0aa4d0 msgr2=0x7fcddc0ac960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] conn(0x7fcddc0aa4d0 0x7fcddc0ac960 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:52.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcddc0a08a0 msgr2=0x7fcddc0a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcddc0a08a0 0x7fcddc0a2d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0094f0 tx=0x7fcdec009440 comp rx=0 tx=0).stop 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf01056e0 msgr2=0x7fcdf0198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 --2- 172.21.15.31:0/2973614876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcdf01056e0 0x7fcdf0198640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcde800ef30 tx=0x7fcde800c450 comp rx=0 tx=0).stop 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf627d700 1 -- 172.21.15.31:0/2973614876 reap_dead start 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 shutdown_connections 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 >> 172.21.15.31:0/2973614876 conn(0x7fcdf00fbea0 msgr2=0x7fcdf00fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 shutdown_connections 2023-07-29T16:12:52.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:52.876+0000 7fcdf7ce0700 1 -- 172.21.15.31:0/2973614876 wait complete. 2023-07-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 -- 172.21.15.31:0/1272001946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 msgr2=0x7f4690110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 --2- 172.21.15.31:0/1272001946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 0x7f4690110eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4680009990 tx=0x7f4680009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 -- 172.21.15.31:0/1272001946 shutdown_connections 2023-07-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 --2- 172.21.15.31:0/1272001946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46901113f0 0x7f46901137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 --2- 172.21.15.31:0/1272001946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46900730f0 0x7f46900734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 --2- 172.21.15.31:0/1272001946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 0x7f4690110eb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 -- 172.21.15.31:0/1272001946 >> 172.21.15.31:0/1272001946 conn(0x7f46900fbea0 msgr2=0x7f46900fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 -- 172.21.15.31:0/1272001946 shutdown_connections 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.322+0000 7f469832c700 1 -- 172.21.15.31:0/1272001946 wait complete. 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.323+0000 7f469832c700 1 Processor -- start 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.323+0000 7f469832c700 1 -- start start 2023-07-29T16:12:53.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.323+0000 7f469832c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46900730f0 0x7f46901a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f469832c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 0x7f46901a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f469832c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46901113f0 0x7f469019c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f469832c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469019cf90 con 0x7f4690073a00 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f469832c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469019d100 con 0x7f46901113f0 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f469832c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469019d2c0 con 0x7f46900730f0 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46900730f0 0x7f46901a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46900730f0 0x7f46901a28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44164/0 (socket says 172.21.15.31:44164) 2023-07-29T16:12:53.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46968c9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46901113f0 0x7f469019c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 -- 172.21.15.31:0/1468848831 learned_addr learned my addr 172.21.15.31:0/1468848831 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46958c7700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 0x7f46901a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 -- 172.21.15.31:0/1468848831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46901113f0 msgr2=0x7f469019c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46901113f0 0x7f469019c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 -- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 msgr2=0x7f46901a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:53.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4690073a00 0x7f46901a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46960c8700 1 -- 172.21.15.31:0/1468848831 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46800095f0 con 0x7f46900730f0 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.324+0000 7f46968c9700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46901113f0 0x7f469019c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.325+0000 7f46960c8700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46900730f0 0x7f46901a28a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f468c00e9a0 tx=0x7f468c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.325+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c009790 con 0x7f46900730f0 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.325+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f468c010430 con 0x7f46900730f0 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.325+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469019d5a0 con 0x7f46900730f0 2023-07-29T16:12:53.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.325+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c018680 con 0x7f46900730f0 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.325+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46901a9890 con 0x7f46900730f0 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.326+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f469004ea50 con 0x7f46900730f0 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.328+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f468c0105a0 con 0x7f46900730f0 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.329+0000 7f46877fe700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f467c0a0820 0x7f467c0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.329+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f468c014070 con 0x7f46900730f0 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46958c7700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f467c0a0820 0x7f467c0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:53.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46877fe700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] conn(0x7f467c0aa450 0x7f467c0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 --> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f467c0acfb0 con 0x7f467c0aa450 2023-07-29T16:12:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f468c0cc0c0 con 0x7f46900730f0 2023-07-29T16:12:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46968c9700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] conn(0x7f467c0aa450 0x7f467c0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46958c7700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f467c0a0820 0x7f467c0a2cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4690112b00 tx=0x7f4680005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.330+0000 7f46968c9700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] conn(0x7f467c0aa450 0x7f467c0ac8e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:53.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.331+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== osd.9 v2:172.21.15.31:6826/3232146044 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f467c0acfb0 con 0x7f467c0aa450 2023-07-29T16:12:53.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.349+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 --> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46900613d0 con 0x7f467c0aa450 2023-07-29T16:12:53.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f46877fe700 1 -- 172.21.15.31:0/1468848831 <== osd.9 v2:172.21.15.31:6826/3232146044 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f46900613d0 con 0x7f467c0aa450 2023-07-29T16:12:53.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] conn(0x7f467c0aa450 msgr2=0x7f467c0ac8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:53.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] conn(0x7f467c0aa450 0x7f467c0ac8e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f467c0a0820 msgr2=0x7f467c0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:53.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f467c0a0820 0x7f467c0a2cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4690112b00 tx=0x7f4680005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46900730f0 msgr2=0x7f46901a28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:53.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 --2- 172.21.15.31:0/1468848831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46900730f0 0x7f46901a28a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f468c00e9a0 tx=0x7f468c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:12:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f46968c9700 1 -- 172.21.15.31:0/1468848831 reap_dead start 2023-07-29T16:12:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 shutdown_connections 2023-07-29T16:12:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 >> 172.21.15.31:0/1468848831 conn(0x7f46900fbea0 msgr2=0x7f46900fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.350+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 shutdown_connections 2023-07-29T16:12:53.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:53.351+0000 7f469832c700 1 -- 172.21.15.31:0/1468848831 wait complete. 2023-07-29T16:12:53.405 INFO:teuthology.orchestra.run.smithi031.stdout:77309411386 2023-07-29T16:12:53.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-29T16:12:54.348 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:54 smithi031 ceph-mon[95504]: pgmap v290: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:54.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 -- 172.21.15.31:0/1284549076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34111320 msgr2=0x7f7d34113700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:54.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 --2- 172.21.15.31:0/1284549076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34111320 0x7f7d34113700 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009f90 tx=0x7f7d2c00b330 comp rx=0 tx=0).stop 2023-07-29T16:12:54.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 -- 172.21.15.31:0/1284549076 shutdown_connections 2023-07-29T16:12:54.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 --2- 172.21.15.31:0/1284549076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34111320 0x7f7d34113700 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 --2- 172.21.15.31:0/1284549076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d34075280 0x7f7d34075650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 --2- 172.21.15.31:0/1284549076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34075b90 0x7f7d34110de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 -- 172.21.15.31:0/1284549076 >> 172.21.15.31:0/1284549076 conn(0x7f7d340fe180 msgr2=0x7f7d34100590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.475+0000 7f7d38bad700 1 -- 172.21.15.31:0/1284549076 shutdown_connections 2023-07-29T16:12:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.476+0000 7f7d38bad700 1 -- 172.21.15.31:0/1284549076 wait complete. 2023-07-29T16:12:54.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.477+0000 7f7d38bad700 1 Processor -- start 2023-07-29T16:12:54.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 -- start start 2023-07-29T16:12:54.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34075280 0x7f7d341a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d34075b90 0x7f7d341a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34111320 0x7f7d3419ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3419d0b0 con 0x7f7d34075280 2023-07-29T16:12:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3419d220 con 0x7f7d34075b90 2023-07-29T16:12:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.478+0000 7f7d38bad700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3419d3e0 con 0x7f7d34111320 2023-07-29T16:12:54.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34075280 0x7f7d341a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34075280 0x7f7d341a29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44954/0 (socket says 172.21.15.31:44954) 2023-07-29T16:12:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d32d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34111320 0x7f7d3419ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 -- 172.21.15.31:0/2388500878 learned_addr learned my addr 172.21.15.31:0/2388500878 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d31d9b700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d34075b90 0x7f7d341a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:54.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 -- 172.21.15.31:0/2388500878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34111320 msgr2=0x7f7d3419ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d34111320 0x7f7d3419ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 -- 172.21.15.31:0/2388500878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d34075b90 msgr2=0x7f7d341a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7d34075b90 0x7f7d341a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.479+0000 7f7d3259c700 1 -- 172.21.15.31:0/2388500878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c009c40 con 0x7f7d34075280 2023-07-29T16:12:54.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.480+0000 7f7d3259c700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34075280 0x7f7d341a29c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400e9c0 tx=0x7f7d2400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:54.479 INFO:teuthology.orchestra.run.smithi031.stdout:111669149751 2023-07-29T16:12:54.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-29T16:12:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.480+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24004a30 con 0x7f7d34075280 2023-07-29T16:12:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.480+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d2400cc80 con 0x7f7d34075280 2023-07-29T16:12:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.480+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240054e0 con 0x7f7d34075280 2023-07-29T16:12:54.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.480+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3419d6c0 con 0x7f7d34075280 2023-07-29T16:12:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.480+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d341a99b0 con 0x7f7d34075280 2023-07-29T16:12:54.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.482+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d3404ea50 con 0x7f7d34075280 2023-07-29T16:12:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.486+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f7d24004b90 con 0x7f7d34075280 2023-07-29T16:12:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.487+0000 7f7d237fe700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7d1c0a08a0 0x7f7d1c0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.487+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f7d24014070 con 0x7f7d34075280 2023-07-29T16:12:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.487+0000 7f7d31d9b700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7d1c0a08a0 0x7f7d1c0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.488+0000 7f7d237fe700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] conn(0x7f7d1c0aa4c0 0x7f7d1c0ac950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.488+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 --> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d1c0ad020 con 0x7f7d1c0aa4c0 2023-07-29T16:12:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.488+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d240954f0 con 0x7f7d34075280 2023-07-29T16:12:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.488+0000 7f7d32d9d700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] conn(0x7f7d1c0aa4c0 0x7f7d1c0ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.488+0000 7f7d31d9b700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7d1c0a08a0 0x7f7d1c0a2d50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d34077d40 tx=0x7f7d2800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.489+0000 7f7d32d9d700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] conn(0x7f7d1c0aa4c0 0x7f7d1c0ac950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.490+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== osd.6 v2:172.21.15.31:6818/2073594680 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7d1c0ad020 con 0x7f7d1c0aa4c0 2023-07-29T16:12:54.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.512+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 --> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7d34113410 con 0x7f7d1c0aa4c0 2023-07-29T16:12:54.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d237fe700 1 -- 172.21.15.31:0/2388500878 <== osd.6 v2:172.21.15.31:6818/2073594680 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7d34113410 con 0x7f7d1c0aa4c0 2023-07-29T16:12:54.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] conn(0x7f7d1c0aa4c0 msgr2=0x7f7d1c0ac950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:54.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] conn(0x7f7d1c0aa4c0 0x7f7d1c0ac950 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7d1c0a08a0 msgr2=0x7f7d1c0a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:54.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7d1c0a08a0 0x7f7d1c0a2d50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d34077d40 tx=0x7f7d2800b410 comp rx=0 tx=0).stop 2023-07-29T16:12:54.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34075280 msgr2=0x7f7d341a29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:54.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 --2- 172.21.15.31:0/2388500878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d34075280 0x7f7d341a29c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400e9c0 tx=0x7f7d2400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:12:54.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d32d9d700 1 -- 172.21.15.31:0/2388500878 reap_dead start 2023-07-29T16:12:54.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 shutdown_connections 2023-07-29T16:12:54.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.514+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 >> 172.21.15.31:0/2388500878 conn(0x7f7d340fe180 msgr2=0x7f7d34100560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.515+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 shutdown_connections 2023-07-29T16:12:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:54.515+0000 7f7d38bad700 1 -- 172.21.15.31:0/2388500878 wait complete. 2023-07-29T16:12:54.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:54 smithi156 ceph-mon[129245]: pgmap v290: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:54.666 INFO:teuthology.orchestra.run.smithi031.stdout:55834574911 2023-07-29T16:12:54.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-29T16:12:54.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:54 smithi039 ceph-mon[128858]: pgmap v290: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 383 B/s wr, 6 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:292057776154 2023-07-29T16:12:54.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-29T16:12:55.229 INFO:teuthology.orchestra.run.smithi031.stdout:34359738436 2023-07-29T16:12:55.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-29T16:12:55.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 -- 172.21.15.31:0/998629871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 msgr2=0x7f898c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 --2- 172.21.15.31:0/998629871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 0x7f898c101a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8980009990 tx=0x7f8980009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 -- 172.21.15.31:0/998629871 shutdown_connections 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 --2- 172.21.15.31:0/998629871 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f898c10a980 0x7f898c10cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 --2- 172.21.15.31:0/998629871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 0x7f898c101a00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 --2- 172.21.15.31:0/998629871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c101f40 0x7f898c10a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 -- 172.21.15.31:0/998629871 >> 172.21.15.31:0/998629871 conn(0x7f898c0763b0 msgr2=0x7f898c0767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 -- 172.21.15.31:0/998629871 shutdown_connections 2023-07-29T16:12:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.372+0000 7f8992ad4700 1 -- 172.21.15.31:0/998629871 wait complete. 2023-07-29T16:12:55.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.374+0000 7f8992ad4700 1 Processor -- start 2023-07-29T16:12:55.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.374+0000 7f8992ad4700 1 -- start start 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8992ad4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 0x7f898c1a2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8992ad4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f898c101f40 0x7f898c1a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8992ad4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c10a980 0x7f898c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8992ad4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c19cf80 con 0x7f898c101f40 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8992ad4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c19d0f0 con 0x7f898c101630 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8992ad4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898c19d2b0 con 0x7f898c10a980 2023-07-29T16:12:55.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8991071700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c10a980 0x7f898c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8991071700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c10a980 0x7f898c19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44202/0 (socket says 172.21.15.31:44202) 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.375+0000 7f8991071700 1 -- 172.21.15.31:0/4073577916 learned_addr learned my addr 172.21.15.31:0/4073577916 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8990870700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 0x7f898c1a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f898bfff700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f898c101f40 0x7f898c1a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8991071700 1 -- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 msgr2=0x7f898c1a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8991071700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f898c101630 0x7f898c1a2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8991071700 1 -- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f898c101f40 msgr2=0x7f898c1a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8991071700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f898c101f40 0x7f898c1a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8991071700 1 -- 172.21.15.31:0/4073577916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8984009c40 con 0x7f898c10a980 2023-07-29T16:12:55.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f898bfff700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f898c101f40 0x7f898c1a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:12:55.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.376+0000 7f8991071700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c10a980 0x7f898c19c910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f898400c0f0 tx=0x7f898400c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.377+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898400cb60 con 0x7f898c10a980 2023-07-29T16:12:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.377+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f898400e030 con 0x7f898c10a980 2023-07-29T16:12:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.377+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984004810 con 0x7f898c10a980 2023-07-29T16:12:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.377+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89800095f0 con 0x7f898c10a980 2023-07-29T16:12:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.377+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898c194010 con 0x7f898c10a980 2023-07-29T16:12:55.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.379+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f898c04ea50 con 0x7f898c10a980 2023-07-29T16:12:55.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.381+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f8984003a00 con 0x7f898c10a980 2023-07-29T16:12:55.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.382+0000 7f8989ffb700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f897c0a0880 0x7f897c0a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.383+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f89840cbb50 con 0x7f898c10a980 2023-07-29T16:12:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.383+0000 7f8989ffb700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] conn(0x7f897c0aa4b0 0x7f897c0ac940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.383+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 --> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f897c0ad010 con 0x7f897c0aa4b0 2023-07-29T16:12:55.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.383+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89840d02c0 con 0x7f898c10a980 2023-07-29T16:12:55.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.383+0000 7f8990870700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f897c0a0880 0x7f897c0a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.384+0000 7f898bfff700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] conn(0x7f897c0aa4b0 0x7f897c0ac940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.384+0000 7f8990870700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f897c0a0880 0x7f897c0a2d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f898000b2e0 tx=0x7f8980005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.385+0000 7f898bfff700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] conn(0x7f897c0aa4b0 0x7f897c0ac940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.386+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== osd.7 v2:172.21.15.39:6816/4224022326 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f897c0ad010 con 0x7f897c0aa4b0 2023-07-29T16:12:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.404+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 --> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f898c0613d0 con 0x7f897c0aa4b0 2023-07-29T16:12:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.404+0000 7f8989ffb700 1 -- 172.21.15.31:0/4073577916 <== osd.7 v2:172.21.15.39:6816/4224022326 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f898c0613d0 con 0x7f897c0aa4b0 2023-07-29T16:12:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] conn(0x7f897c0aa4b0 msgr2=0x7f897c0ac940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] conn(0x7f897c0aa4b0 0x7f897c0ac940 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f897c0a0880 msgr2=0x7f897c0a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f897c0a0880 0x7f897c0a2d30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f898000b2e0 tx=0x7f8980005fb0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c10a980 msgr2=0x7f898c19c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 --2- 172.21.15.31:0/4073577916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f898c10a980 0x7f898c19c910 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f898400c0f0 tx=0x7f898400c4b0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8991071700 1 -- 172.21.15.31:0/4073577916 reap_dead start 2023-07-29T16:12:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 shutdown_connections 2023-07-29T16:12:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 >> 172.21.15.31:0/4073577916 conn(0x7f898c0763b0 msgr2=0x7f898c102370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 shutdown_connections 2023-07-29T16:12:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.405+0000 7f8992ad4700 1 -- 172.21.15.31:0/4073577916 wait complete. 2023-07-29T16:12:55.459 INFO:teuthology.orchestra.run.smithi031.stdout:201863462953 2023-07-29T16:12:55.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-29T16:12:55.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 -- 172.21.15.31:0/1892240236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ff420 msgr2=0x7f83640ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 --2- 172.21.15.31:0/1892240236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ff420 0x7f83640ff7f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f835400b0a0 tx=0x7f835400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 -- 172.21.15.31:0/1892240236 shutdown_connections 2023-07-29T16:12:55.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 --2- 172.21.15.31:0/1892240236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8364104750 0x7f8364104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 --2- 172.21.15.31:0/1892240236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ff420 0x7f83640ff7f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 --2- 172.21.15.31:0/1892240236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ffd30 0x7f8364104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.589+0000 7f836896c700 1 -- 172.21.15.31:0/1892240236 >> 172.21.15.31:0/1892240236 conn(0x7f83640763b0 msgr2=0x7f83640767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.590+0000 7f836896c700 1 -- 172.21.15.31:0/1892240236 shutdown_connections 2023-07-29T16:12:55.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.590+0000 7f836896c700 1 -- 172.21.15.31:0/1892240236 wait complete. 2023-07-29T16:12:55.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.591+0000 7f836896c700 1 Processor -- start 2023-07-29T16:12:55.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.591+0000 7f836896c700 1 -- start start 2023-07-29T16:12:55.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.591+0000 7f836896c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ff420 0x7f836410c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836896c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ffd30 0x7f836410c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836896c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8364104750 0x7f83641b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836896c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83641b1650 con 0x7f83640ff420 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836896c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836406a7a0 con 0x7f8364104750 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836896c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836406a960 con 0x7f83640ffd30 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ff420 0x7f836410c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ff420 0x7f836410c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44978/0 (socket says 172.21.15.31:44978) 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f836259c700 1 -- 172.21.15.31:0/1941214538 learned_addr learned my addr 172.21.15.31:0/1941214538 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f8359d9b700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ffd30 0x7f836410c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.592+0000 7f8362d9d700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8364104750 0x7f83641b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f836259c700 1 -- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ffd30 msgr2=0x7f836410c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f836259c700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ffd30 0x7f836410c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f836259c700 1 -- 172.21.15.31:0/1941214538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8364104750 msgr2=0x7f83641b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f836259c700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8364104750 0x7f83641b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f836259c700 1 -- 172.21.15.31:0/1941214538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8354009d00 con 0x7f83640ff420 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f8359d9b700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83640ffd30 0x7f836410c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.593+0000 7f8362d9d700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8364104750 0x7f83641b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.594+0000 7f836259c700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ff420 0x7f836410c3b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8354006070 tx=0x7f8354007b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.594+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835400e040 con 0x7f83640ff420 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.594+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8354005530 con 0x7f83640ff420 2023-07-29T16:12:55.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.594+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8354012b70 con 0x7f83640ff420 2023-07-29T16:12:55.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.594+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836406abe0 con 0x7f83640ff420 2023-07-29T16:12:55.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.594+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836406b080 con 0x7f83640ff420 2023-07-29T16:12:55.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.596+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8364116bf0 con 0x7f83640ff420 2023-07-29T16:12:55.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.598+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f83540056a0 con 0x7f83640ff420 2023-07-29T16:12:55.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.600+0000 7f835bfff700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83500a0920 0x7f83500a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.600+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f83540cd4a0 con 0x7f83640ff420 2023-07-29T16:12:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.601+0000 7f8359d9b700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83500a0920 0x7f83500a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.601+0000 7f835bfff700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] conn(0x7f83500aa550 0x7f83500ac9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.601+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 --> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83500ad0b0 con 0x7f83500aa550 2023-07-29T16:12:55.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.601+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83540cd850 con 0x7f83640ff420 2023-07-29T16:12:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.602+0000 7f8362d9d700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] conn(0x7f83500aa550 0x7f83500ac9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.602+0000 7f8359d9b700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83500a0920 0x7f83500a2dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f836410d9d0 tx=0x7f834c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.602+0000 7f8362d9d700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] conn(0x7f83500aa550 0x7f83500ac9e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.603+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== osd.5 v2:172.21.15.156:6808/1995637959 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f83500ad0b0 con 0x7f83500aa550 2023-07-29T16:12:55.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.625+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 --> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f836404ea50 con 0x7f83500aa550 2023-07-29T16:12:55.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f835bfff700 1 -- 172.21.15.31:0/1941214538 <== osd.5 v2:172.21.15.156:6808/1995637959 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f836404ea50 con 0x7f83500aa550 2023-07-29T16:12:55.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] conn(0x7f83500aa550 msgr2=0x7f83500ac9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] conn(0x7f83500aa550 0x7f83500ac9e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83500a0920 msgr2=0x7f83500a2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83500a0920 0x7f83500a2dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f836410d9d0 tx=0x7f834c00b410 comp rx=0 tx=0).stop 2023-07-29T16:12:55.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ff420 msgr2=0x7f836410c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 --2- 172.21.15.31:0/1941214538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83640ff420 0x7f836410c3b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8354006070 tx=0x7f8354007b60 comp rx=0 tx=0).stop 2023-07-29T16:12:55.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f8362d9d700 1 -- 172.21.15.31:0/1941214538 reap_dead start 2023-07-29T16:12:55.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 shutdown_connections 2023-07-29T16:12:55.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.626+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 >> 172.21.15.31:0/1941214538 conn(0x7f83640763b0 msgr2=0x7f8364102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.627+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 shutdown_connections 2023-07-29T16:12:55.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.627+0000 7f836896c700 1 -- 172.21.15.31:0/1941214538 wait complete. 2023-07-29T16:12:55.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.725+0000 7ff87d49a700 1 -- 172.21.15.31:0/2660010101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8781084a0 msgr2=0x7ff878110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.725+0000 7ff87d49a700 1 --2- 172.21.15.31:0/2660010101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8781084a0 0x7ff878110c00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009990 tx=0x7ff86c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 -- 172.21.15.31:0/2660010101 shutdown_connections 2023-07-29T16:12:55.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 --2- 172.21.15.31:0/2660010101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff878111140 0x7ff878113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 --2- 172.21.15.31:0/2660010101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff878107b90 0x7ff878107f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 --2- 172.21.15.31:0/2660010101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8781084a0 0x7ff878110c00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 -- 172.21.15.31:0/2660010101 >> 172.21.15.31:0/2660010101 conn(0x7ff8780761f0 msgr2=0x7ff878078600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 -- 172.21.15.31:0/2660010101 shutdown_connections 2023-07-29T16:12:55.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.726+0000 7ff87d49a700 1 -- 172.21.15.31:0/2660010101 wait complete. 2023-07-29T16:12:55.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.727+0000 7ff87d49a700 1 Processor -- start 2023-07-29T16:12:55.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.728+0000 7ff87d49a700 1 -- start start 2023-07-29T16:12:55.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.728+0000 7ff87d49a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff878107b90 0x7ff8781a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.728+0000 7ff87d49a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8781084a0 0x7ff8781a2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff87d49a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff878111140 0x7ff87819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff87d49a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87819d110 con 0x7ff8781084a0 2023-07-29T16:12:55.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff87d49a700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87819d280 con 0x7ff878107b90 2023-07-29T16:12:55.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff87d49a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87819d440 con 0x7ff878111140 2023-07-29T16:12:55.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8781084a0 0x7ff8781a2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8781084a0 0x7ff8781a2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44992/0 (socket says 172.21.15.31:44992) 2023-07-29T16:12:55.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 -- 172.21.15.31:0/4073641700 learned_addr learned my addr 172.21.15.31:0/4073641700 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:55.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8777fe700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff878111140 0x7ff87819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff876ffd700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff878107b90 0x7ff8781a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 -- 172.21.15.31:0/4073641700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff878111140 msgr2=0x7ff87819caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff878111140 0x7ff87819caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 -- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff878107b90 msgr2=0x7ff8781a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.729+0000 7ff8767fc700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff878107b90 0x7ff8781a2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.730+0000 7ff8767fc700 1 -- 172.21.15.31:0/4073641700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c0095f0 con 0x7ff8781084a0 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.730+0000 7ff8767fc700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8781084a0 0x7ff8781a2f60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0098e0 tx=0x7ff86c003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.730+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c01d070 con 0x7ff8781084a0 2023-07-29T16:12:55.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.730+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff86c005580 con 0x7ff8781084a0 2023-07-29T16:12:55.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.730+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c00fdf0 con 0x7ff8781084a0 2023-07-29T16:12:55.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.730+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87819d6c0 con 0x7ff8781084a0 2023-07-29T16:12:55.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.731+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8781a9a10 con 0x7ff8781084a0 2023-07-29T16:12:55.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.732+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff87804ea50 con 0x7ff8781084a0 2023-07-29T16:12:55.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.736+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7ff86c00f4f0 con 0x7ff8781084a0 2023-07-29T16:12:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.738+0000 7ff85ffff700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8600a08a0 0x7ff8600a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.738+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff86c0ccfc0 con 0x7ff8781084a0 2023-07-29T16:12:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.739+0000 7ff85ffff700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] conn(0x7ff8600aa4d0 0x7ff8600ac960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.739+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 --> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff8600ad030 con 0x7ff8600aa4d0 2023-07-29T16:12:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.739+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff86c0cd330 con 0x7ff8781084a0 2023-07-29T16:12:55.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.739+0000 7ff876ffd700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8600a08a0 0x7ff8600a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.740+0000 7ff8777fe700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] conn(0x7ff8600aa4d0 0x7ff8600ac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.740+0000 7ff876ffd700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8600a08a0 0x7ff8600a2d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff868006fd0 tx=0x7ff868009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.741+0000 7ff8777fe700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] conn(0x7ff8600aa4d0 0x7ff8600ac960 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.742+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== osd.10 v2:172.21.15.39:6824/2960119201 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff8600ad030 con 0x7ff8600aa4d0 2023-07-29T16:12:55.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.769+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 --> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff8780770b0 con 0x7ff8600aa4d0 2023-07-29T16:12:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.770+0000 7ff85ffff700 1 -- 172.21.15.31:0/4073641700 <== osd.10 v2:172.21.15.39:6824/2960119201 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff8780770b0 con 0x7ff8600aa4d0 2023-07-29T16:12:55.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.771+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] conn(0x7ff8600aa4d0 msgr2=0x7ff8600ac960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.771+0000 7ff87d49a700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] conn(0x7ff8600aa4d0 0x7ff8600ac960 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8600a08a0 msgr2=0x7ff8600a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8600a08a0 0x7ff8600a2d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff868006fd0 tx=0x7ff868009190 comp rx=0 tx=0).stop 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8781084a0 msgr2=0x7ff8781a2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 --2- 172.21.15.31:0/4073641700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8781084a0 0x7ff8781a2f60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff86c0098e0 tx=0x7ff86c003c60 comp rx=0 tx=0).stop 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff8777fe700 1 -- 172.21.15.31:0/4073641700 reap_dead start 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 shutdown_connections 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 >> 172.21.15.31:0/4073641700 conn(0x7ff8780761f0 msgr2=0x7ff878076760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 shutdown_connections 2023-07-29T16:12:55.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.772+0000 7ff87d49a700 1 -- 172.21.15.31:0/4073641700 wait complete. 2023-07-29T16:12:55.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3965282243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c100fe0 msgr2=0x7fc79c104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3965282243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c100fe0 0x7fc79c104fb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc790009990 tx=0x7fc790009ca0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3965282243 shutdown_connections 2023-07-29T16:12:55.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3965282243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc79c1056e0 0x7fc79c10d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3965282243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79c100640 0x7fc79c100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3965282243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c100fe0 0x7fc79c104fb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.984+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3965282243 >> 172.21.15.31:0/3965282243 conn(0x7fc79c0fbea0 msgr2=0x7fc79c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:55.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.985+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3965282243 shutdown_connections 2023-07-29T16:12:55.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.985+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3965282243 wait complete. 2023-07-29T16:12:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 Processor -- start 2023-07-29T16:12:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 -- start start 2023-07-29T16:12:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79c100640 0x7fc79c073050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc79c100fe0 0x7fc79c073590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c1056e0 0x7fc79c073b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1a48f0 con 0x7fc79c100fe0 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1a4a60 con 0x7fc79c1056e0 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc7a1d42700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79c1a4c20 con 0x7fc79c100640 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc79affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc79c100fe0 0x7fc79c073590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc79affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc79c100fe0 0x7fc79c073590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45018/0 (socket says 172.21.15.31:45018) 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc79affd700 1 -- 172.21.15.31:0/3279254319 learned_addr learned my addr 172.21.15.31:0/3279254319 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.986+0000 7fc79bfff700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c1056e0 0x7fc79c073b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79b7fe700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79c100640 0x7fc79c073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79bfff700 1 -- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79c100640 msgr2=0x7fc79c073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79bfff700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc79c100640 0x7fc79c073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79bfff700 1 -- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc79c100fe0 msgr2=0x7fc79c073590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:55.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79bfff700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc79c100fe0 0x7fc79c073590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:55.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79bfff700 1 -- 172.21.15.31:0/3279254319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900095f0 con 0x7fc79c1056e0 2023-07-29T16:12:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc79bfff700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c1056e0 0x7fc79c073b10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc79400efc0 tx=0x7fc79400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc794018070 con 0x7fc79c1056e0 2023-07-29T16:12:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc79400f040 con 0x7fc79c1056e0 2023-07-29T16:12:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.987+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79c1a4ea0 con 0x7fc79c1056e0 2023-07-29T16:12:55.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.988+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79400cd40 con 0x7fc79c1056e0 2023-07-29T16:12:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.988+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79c1a5320 con 0x7fc79c1056e0 2023-07-29T16:12:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.989+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc79c110540 con 0x7fc79c1056e0 2023-07-29T16:12:55.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.991+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fc794007500 con 0x7fc79c1056e0 2023-07-29T16:12:55.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.992+0000 7fc798ff9700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7840a0820 0x7fc7840a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.992+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fc7940cc0d0 con 0x7fc79c1056e0 2023-07-29T16:12:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.993+0000 7fc798ff9700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] conn(0x7fc7840aa450 0x7fc7840ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.993+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 --> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7840acfb0 con 0x7fc7840aa450 2023-07-29T16:12:55.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.993+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7940cc440 con 0x7fc79c1056e0 2023-07-29T16:12:55.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.993+0000 7fc79b7fe700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7840a0820 0x7fc7840a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.993+0000 7fc79affd700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] conn(0x7fc7840aa450 0x7fc7840ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:55.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.995+0000 7fc79b7fe700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7840a0820 0x7fc7840a2cd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009c60 tx=0x7fc78c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.995+0000 7fc79affd700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] conn(0x7fc7840aa450 0x7fc7840ac8e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:55.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:55.996+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== osd.11 v2:172.21.15.156:6824/2124313764 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc7840acfb0 con 0x7fc7840aa450 2023-07-29T16:12:56.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.013+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 --> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc79c04ea50 con 0x7fc7840aa450 2023-07-29T16:12:56.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc798ff9700 1 -- 172.21.15.31:0/3279254319 <== osd.11 v2:172.21.15.156:6824/2124313764 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc79c04ea50 con 0x7fc7840aa450 2023-07-29T16:12:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] conn(0x7fc7840aa450 msgr2=0x7fc7840ac8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] conn(0x7fc7840aa450 0x7fc7840ac8e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7840a0820 msgr2=0x7fc7840a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7840a0820 0x7fc7840a2cd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009c60 tx=0x7fc78c009260 comp rx=0 tx=0).stop 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c1056e0 msgr2=0x7fc79c073b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 --2- 172.21.15.31:0/3279254319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc79c1056e0 0x7fc79c073b10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc79400efc0 tx=0x7fc79400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc79bfff700 1 -- 172.21.15.31:0/3279254319 reap_dead start 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 shutdown_connections 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 >> 172.21.15.31:0/3279254319 conn(0x7fc79c0fbea0 msgr2=0x7fc79c0fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:56.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 shutdown_connections 2023-07-29T16:12:56.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.014+0000 7fc7a1d42700 1 -- 172.21.15.31:0/3279254319 wait complete. 2023-07-29T16:12:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.115+0000 7f5b792ce700 1 -- 172.21.15.31:0/3923328477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b741045f0 msgr2=0x7f5b7410c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.115+0000 7f5b792ce700 1 --2- 172.21.15.31:0/3923328477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b741045f0 0x7f5b7410c170 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00b0a0 tx=0x7f5b6c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 -- 172.21.15.31:0/3923328477 shutdown_connections 2023-07-29T16:12:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 --2- 172.21.15.31:0/3923328477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b741045f0 0x7f5b7410c170 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 --2- 172.21.15.31:0/3923328477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ff550 0x7f5b740ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 --2- 172.21.15.31:0/3923328477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b740ffef0 0x7f5b74103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 -- 172.21.15.31:0/3923328477 >> 172.21.15.31:0/3923328477 conn(0x7f5b740763b0 msgr2=0x7f5b740767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:56.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 -- 172.21.15.31:0/3923328477 shutdown_connections 2023-07-29T16:12:56.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.116+0000 7f5b792ce700 1 -- 172.21.15.31:0/3923328477 wait complete. 2023-07-29T16:12:56.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.117+0000 7f5b792ce700 1 Processor -- start 2023-07-29T16:12:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.117+0000 7f5b792ce700 1 -- start start 2023-07-29T16:12:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b792ce700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b740ff550 0x7f5b7410ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b792ce700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ffef0 0x7f5b7410cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b792ce700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b741045f0 0x7f5b741a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b792ce700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7410d8b0 con 0x7f5b740ffef0 2023-07-29T16:12:56.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b792ce700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b741a4d50 con 0x7f5b740ff550 2023-07-29T16:12:56.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b792ce700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b741a4ec0 con 0x7f5b741045f0 2023-07-29T16:12:56.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ffef0 0x7f5b7410cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:56.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ffef0 0x7f5b7410cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45036/0 (socket says 172.21.15.31:45036) 2023-07-29T16:12:56.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 -- 172.21.15.31:0/4119115709 learned_addr learned my addr 172.21.15.31:0/4119115709 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:12:56.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b737fe700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b741045f0 0x7f5b741a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:56.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b72ffd700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b740ff550 0x7f5b7410ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:56.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 -- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b741045f0 msgr2=0x7f5b741a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b741045f0 0x7f5b741a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 -- 172.21.15.31:0/4119115709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b740ff550 msgr2=0x7f5b7410ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b740ff550 0x7f5b7410ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.118+0000 7f5b727fc700 1 -- 172.21.15.31:0/4119115709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c009d00 con 0x7f5b740ffef0 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.119+0000 7f5b737fe700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b741045f0 0x7f5b741a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.119+0000 7f5b72ffd700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b740ff550 0x7f5b7410ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.119+0000 7f5b727fc700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ffef0 0x7f5b7410cfc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800e970 tx=0x7f5b6800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:56.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.119+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68009790 con 0x7f5b740ffef0 2023-07-29T16:12:56.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.119+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b6800c7e0 con 0x7f5b740ffef0 2023-07-29T16:12:56.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.119+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b680054e0 con 0x7f5b740ffef0 2023-07-29T16:12:56.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.120+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b741a5140 con 0x7f5b740ffef0 2023-07-29T16:12:56.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.120+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b741a55f0 con 0x7f5b740ffef0 2023-07-29T16:12:56.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.121+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5b74110620 con 0x7f5b740ffef0 2023-07-29T16:12:56.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.123+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f5b6800c950 con 0x7f5b740ffef0 2023-07-29T16:12:56.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.124+0000 7f5b5bfff700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b5c0a0970 0x7f5b5c0a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:56.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5b68014070 con 0x7f5b740ffef0 2023-07-29T16:12:56.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b72ffd700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b5c0a0970 0x7f5b5c0a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:56.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b5bfff700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] conn(0x7f5b5c0aa5a0 0x7f5b5c0aca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:12:56.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 --> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b5c0ad100 con 0x7f5b5c0aa5a0 2023-07-29T16:12:56.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b680cc410 con 0x7f5b740ffef0 2023-07-29T16:12:56.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b72ffd700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b5c0a0970 0x7f5b5c0a2e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b64011490 tx=0x7f5b64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:56.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.125+0000 7f5b737fe700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] conn(0x7f5b5c0aa5a0 0x7f5b5c0aca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:12:56.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.126+0000 7f5b737fe700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] conn(0x7f5b5c0aa5a0 0x7f5b5c0aca30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:12:56.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.127+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== osd.8 v2:172.21.15.156:6816/2938271220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5b5c0ad100 con 0x7f5b5c0aa5a0 2023-07-29T16:12:56.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.147+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 --> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5b7404ea50 con 0x7f5b5c0aa5a0 2023-07-29T16:12:56.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.147+0000 7f5b5bfff700 1 -- 172.21.15.31:0/4119115709 <== osd.8 v2:172.21.15.156:6816/2938271220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5b7404ea50 con 0x7f5b5c0aa5a0 2023-07-29T16:12:56.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] conn(0x7f5b5c0aa5a0 msgr2=0x7f5b5c0aca30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] conn(0x7f5b5c0aa5a0 0x7f5b5c0aca30 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:12:56.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b5c0a0970 msgr2=0x7f5b5c0a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b5c0a0970 0x7f5b5c0a2e20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5b64011490 tx=0x7f5b64009260 comp rx=0 tx=0).stop 2023-07-29T16:12:56.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ffef0 msgr2=0x7f5b7410cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:12:56.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 --2- 172.21.15.31:0/4119115709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b740ffef0 0x7f5b7410cfc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5b6800e970 tx=0x7f5b6800ec80 comp rx=0 tx=0).stop 2023-07-29T16:12:56.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b737fe700 1 -- 172.21.15.31:0/4119115709 reap_dead start 2023-07-29T16:12:56.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 shutdown_connections 2023-07-29T16:12:56.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 >> 172.21.15.31:0/4119115709 conn(0x7f5b740763b0 msgr2=0x7f5b740fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:12:56.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.148+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 shutdown_connections 2023-07-29T16:12:56.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:12:56.149+0000 7f5b792ce700 1 -- 172.21.15.31:0/4119115709 wait complete. 2023-07-29T16:12:56.525 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:56 smithi031 ceph-mon[95504]: pgmap v291: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:56.538 INFO:teuthology.orchestra.run.smithi031.stdout:313532612631 2023-07-29T16:12:56.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-29T16:12:56.570 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:56 smithi156 ceph-mon[129245]: pgmap v291: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:56.735 INFO:teuthology.orchestra.run.smithi031.stdout:347892350994 2023-07-29T16:12:56.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-29T16:12:56.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:56 smithi039 ceph-mon[128858]: pgmap v291: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 383 B/s wr, 7 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:57.066 INFO:teuthology.orchestra.run.smithi031.stdout:167503724590 2023-07-29T16:12:57.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-29T16:12:57.243 INFO:teuthology.orchestra.run.smithi031.stdout:257698037791 2023-07-29T16:12:57.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-29T16:12:57.564 INFO:teuthology.orchestra.run.smithi031.stdout:223338299428 2023-07-29T16:12:57.565 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-29T16:12:57.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:12:57 smithi031 ceph-mon[95504]: pgmap v292: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:57.952 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:57.995 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:58.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:12:57 smithi039 ceph-mon[128858]: pgmap v292: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:12:57 smithi156 ceph-mon[129245]: pgmap v292: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 72 B/s, 0 objects/s recovering 2023-07-29T16:12:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:58.341 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:59.445 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:59.450 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:12:59.700 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:00.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 -- 172.21.15.31:0/382155857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a581056e0 msgr2=0x7f0a5810d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:00.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/382155857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a581056e0 0x7f0a5810d260 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0a5000b0d0 tx=0x7f0a5000b3e0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 -- 172.21.15.31:0/382155857 shutdown_connections 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/382155857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a581056e0 0x7f0a5810d260 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/382155857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a58100640 0x7f0a58100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/382155857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a58100fe0 0x7f0a58104fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 -- 172.21.15.31:0/382155857 >> 172.21.15.31:0/382155857 conn(0x7f0a580fbea0 msgr2=0x7f0a580fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 -- 172.21.15.31:0/382155857 shutdown_connections 2023-07-29T16:13:00.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.100+0000 7f0a5f9af700 1 -- 172.21.15.31:0/382155857 wait complete. 2023-07-29T16:13:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.101+0000 7f0a5f9af700 1 Processor -- start 2023-07-29T16:13:00.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.101+0000 7f0a5f9af700 1 -- start start 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5f9af700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a58100640 0x7f0a5819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5f9af700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 0x7f0a5819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5f9af700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a581056e0 0x7f0a581986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5f9af700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58198d30 con 0x7f0a58100fe0 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5f9af700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58198ea0 con 0x7f0a581056e0 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5f9af700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58199060 con 0x7f0a58100640 2023-07-29T16:13:00.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5cf4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 0x7f0a5819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5cf4a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 0x7f0a5819eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36566/0 (socket says 172.21.15.31:36566) 2023-07-29T16:13:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5cf4a700 1 -- 172.21.15.31:0/3569123736 learned_addr learned my addr 172.21.15.31:0/3569123736 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5d74b700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a58100640 0x7f0a5819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:00.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.102+0000 7f0a5df4c700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a581056e0 0x7f0a581986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5cf4a700 1 -- 172.21.15.31:0/3569123736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a58100640 msgr2=0x7f0a5819e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5cf4a700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a58100640 0x7f0a5819e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5cf4a700 1 -- 172.21.15.31:0/3569123736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a581056e0 msgr2=0x7f0a581986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:00.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5cf4a700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a581056e0 0x7f0a581986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5cf4a700 1 -- 172.21.15.31:0/3569123736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50009d00 con 0x7f0a58100fe0 2023-07-29T16:13:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5df4c700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a581056e0 0x7f0a581986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5cf4a700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 0x7f0a5819eb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e970 tx=0x7f0a4800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48009790 con 0x7f0a58100fe0 2023-07-29T16:13:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a58199340 con 0x7f0a58100fe0 2023-07-29T16:13:00.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a4800c7e0 con 0x7f0a58100fe0 2023-07-29T16:13:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a480054e0 con 0x7f0a58100fe0 2023-07-29T16:13:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.103+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a581a5630 con 0x7f0a58100fe0 2023-07-29T16:13:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.105+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5804ea50 con 0x7f0a58100fe0 2023-07-29T16:13:00.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.106+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f0a4800c950 con 0x7f0a58100fe0 2023-07-29T16:13:00.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.108+0000 7f0a4e7fc700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a440a0820 0x7f0a440a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:00.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.108+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f0a48014070 con 0x7f0a58100fe0 2023-07-29T16:13:00.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.108+0000 7f0a5d74b700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a440a0820 0x7f0a440a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:00.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.109+0000 7f0a5d74b700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a440a0820 0x7f0a440a2cd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009c60 tx=0x7f0a54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:00.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.112+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0a48095800 con 0x7f0a58100fe0 2023-07-29T16:13:00.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.346+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0a581a5880 con 0x7f0a58100fe0 2023-07-29T16:13:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.347+0000 7f0a4e7fc700 1 -- 172.21.15.31:0/3569123736 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0a48098e20 con 0x7f0a58100fe0 2023-07-29T16:13:00.345 INFO:teuthology.orchestra.run.smithi031.stdout:146028888116 2023-07-29T16:13:00.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a440a0820 msgr2=0x7f0a440a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:00.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a440a0820 0x7f0a440a2cd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0a54009c60 tx=0x7f0a54009260 comp rx=0 tx=0).stop 2023-07-29T16:13:00.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 msgr2=0x7f0a5819eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:00.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 0x7f0a5819eb80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800e970 tx=0x7f0a4800ec80 comp rx=0 tx=0).stop 2023-07-29T16:13:00.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 shutdown_connections 2023-07-29T16:13:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a440a0820 0x7f0a440a2cd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a581056e0 0x7f0a581986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a58100640 0x7f0a5819e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 --2- 172.21.15.31:0/3569123736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a58100fe0 0x7f0a5819eb80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 >> 172.21.15.31:0/3569123736 conn(0x7f0a580fbea0 msgr2=0x7f0a5810b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:00.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 shutdown_connections 2023-07-29T16:13:00.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:00.350+0000 7f0a5f9af700 1 -- 172.21.15.31:0/3569123736 wait complete. 2023-07-29T16:13:00.560 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:00.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:00 smithi031 ceph-mon[95504]: pgmap v293: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 58 B/s, 0 objects/s recovering 2023-07-29T16:13:00.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:00 smithi039 ceph-mon[128858]: pgmap v293: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 58 B/s, 0 objects/s recovering 2023-07-29T16:13:00.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:00 smithi156 ceph-mon[129245]: pgmap v293: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s; 58 B/s, 0 objects/s recovering 2023-07-29T16:13:01.035 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:01.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3569123736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-29T16:13:01.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3569123736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-29T16:13:01.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3569123736' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-29T16:13:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 -- 172.21.15.31:0/679227938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 msgr2=0x7fcaf410f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 --2- 172.21.15.31:0/679227938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf410f2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00b0d0 tx=0x7fcaec00b3e0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 -- 172.21.15.31:0/679227938 shutdown_connections 2023-07-29T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 --2- 172.21.15.31:0/679227938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf410f2c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 --2- 172.21.15.31:0/679227938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf41026a0 0x7fcaf4102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 --2- 172.21.15.31:0/679227938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaf4103040 0x7fcaf4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 -- 172.21.15.31:0/679227938 >> 172.21.15.31:0/679227938 conn(0x7fcaf40fdf40 msgr2=0x7fcaf4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 -- 172.21.15.31:0/679227938 shutdown_connections 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.856+0000 7fcaf3fff700 1 -- 172.21.15.31:0/679227938 wait complete. 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.857+0000 7fcaf3fff700 1 Processor -- start 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.857+0000 7fcaf3fff700 1 -- start start 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaf41026a0 0x7fcaf4198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 0x7fcaf4198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf3fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf419d620 con 0x7fcaf4107740 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf3fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf419d790 con 0x7fcaf4103040 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf3fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf419d950 con 0x7fcaf41026a0 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf27fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 0x7fcaf4198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf27fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 0x7fcaf4198ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42950/0 (socket says 172.21.15.31:42950) 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf27fc700 1 -- 172.21.15.31:0/291864833 learned_addr learned my addr 172.21.15.31:0/291864833 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf2ffd700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaf41026a0 0x7fcaf4198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf37fe700 1 -- 172.21.15.31:0/291864833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaf41026a0 msgr2=0x7fcaf4198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf37fe700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaf41026a0 0x7fcaf4198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf37fe700 1 -- 172.21.15.31:0/291864833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 msgr2=0x7fcaf4198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf37fe700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 0x7fcaf4198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf37fe700 1 -- 172.21.15.31:0/291864833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4009520 con 0x7fcaf4107740 2023-07-29T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.858+0000 7fcaf27fc700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 0x7fcaf4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.859+0000 7fcaf37fe700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf419d010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009fd0 tx=0x7fcaec0077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.859+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec00e040 con 0x7fcaf4107740 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.859+0000 7fcaf3fff700 1 -- 172.21.15.31:0/291864833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaec009d00 con 0x7fcaf4107740 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.859+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcaec00bcb0 con 0x7fcaf4107740 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.859+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec005840 con 0x7fcaf4107740 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.859+0000 7fcaf3fff700 1 -- 172.21.15.31:0/291864833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf41a52c0 con 0x7fcaf4107740 2023-07-29T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.860+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf404f2a0 con 0x7fcaf4107740 2023-07-29T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.862+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fcaec004a50 con 0x7fcaf4107740 2023-07-29T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.863+0000 7fcadbfff700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcadc0a0820 0x7fcadc0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.864+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fcaec0cd480 con 0x7fcaf4107740 2023-07-29T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.864+0000 7fcaf2ffd700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcadc0a0820 0x7fcadc0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.865+0000 7fcaf2ffd700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcadc0a0820 0x7fcadc0a2cd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009df0 tx=0x7fcae4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:01.867+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcaec096dd0 con 0x7fcaf4107740 2023-07-29T16:13:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.131+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fcaf41999c0 con 0x7fcaf4107740 2023-07-29T16:13:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.131+0000 7fcadbfff700 1 -- 172.21.15.31:0/291864833 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcaec021020 con 0x7fcaf4107740 2023-07-29T16:13:02.129 INFO:teuthology.orchestra.run.smithi031.stdout:77309411388 2023-07-29T16:13:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcadc0a0820 msgr2=0x7fcadc0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcadc0a0820 0x7fcadc0a2cd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009df0 tx=0x7fcae4009260 comp rx=0 tx=0).stop 2023-07-29T16:13:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 msgr2=0x7fcaf419d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf419d010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009fd0 tx=0x7fcaec0077f0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 shutdown_connections 2023-07-29T16:13:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcadc0a0820 0x7fcadc0a2cd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaf4107740 0x7fcaf419d010 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaf41026a0 0x7fcaf4198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 --2- 172.21.15.31:0/291864833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaf4103040 0x7fcaf4198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 >> 172.21.15.31:0/291864833 conn(0x7fcaf40fdf40 msgr2=0x7fcaf40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 shutdown_connections 2023-07-29T16:13:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.134+0000 7fcad9ffb700 1 -- 172.21.15.31:0/291864833 wait complete. 2023-07-29T16:13:02.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:02 smithi031 ceph-mon[95504]: pgmap v294: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 52 B/s, 0 objects/s recovering 2023-07-29T16:13:02.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:02 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/291864833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-29T16:13:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:02.644 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888113 got 146028888116 for osd.4 2023-07-29T16:13:02.644 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:02.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:02 smithi039 ceph-mon[128858]: pgmap v294: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 52 B/s, 0 objects/s recovering 2023-07-29T16:13:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:02 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/291864833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-29T16:13:02.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 -- 172.21.15.31:0/1542035195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e01026a0 msgr2=0x7f22e0102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 --2- 172.21.15.31:0/1542035195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e01026a0 0x7f22e0102a70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f22d00099c0 tx=0x7f22d0009cd0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 -- 172.21.15.31:0/1542035195 shutdown_connections 2023-07-29T16:13:02.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 --2- 172.21.15.31:0/1542035195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e0107740 0x7f22e010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 --2- 172.21.15.31:0/1542035195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e01026a0 0x7f22e0102a70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 --2- 172.21.15.31:0/1542035195 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e0103040 0x7f22e0107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.821+0000 7f22dffff700 1 -- 172.21.15.31:0/1542035195 >> 172.21.15.31:0/1542035195 conn(0x7f22e00fdf60 msgr2=0x7f22e0100370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.822+0000 7f22dffff700 1 -- 172.21.15.31:0/1542035195 shutdown_connections 2023-07-29T16:13:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.822+0000 7f22dffff700 1 -- 172.21.15.31:0/1542035195 wait complete. 2023-07-29T16:13:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:02 smithi156 ceph-mon[129245]: pgmap v294: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 52 B/s, 0 objects/s recovering 2023-07-29T16:13:02.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:02 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/291864833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-29T16:13:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.823+0000 7f22dffff700 1 Processor -- start 2023-07-29T16:13:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.823+0000 7f22dffff700 1 -- start start 2023-07-29T16:13:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22dffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 0x7f22e010ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22dffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e0103040 0x7f22e010bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22dffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e0107740 0x7f22e010c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22dffff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010cae0 con 0x7f22e0103040 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22dffff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010cc50 con 0x7f22e01026a0 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22dffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e010ce10 con 0x7f22e0107740 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22deffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 0x7f22e010ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22deffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 0x7f22e010ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42962/0 (socket says 172.21.15.31:42962) 2023-07-29T16:13:02.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22deffd700 1 -- 172.21.15.31:0/3957480046 learned_addr learned my addr 172.21.15.31:0/3957480046 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:02.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22de7fc700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e0103040 0x7f22e010bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.824+0000 7f22df7fe700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e0107740 0x7f22e010c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22deffd700 1 -- 172.21.15.31:0/3957480046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e0107740 msgr2=0x7f22e010c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22deffd700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e0107740 0x7f22e010c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22deffd700 1 -- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e0103040 msgr2=0x7f22e010bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22deffd700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e0103040 0x7f22e010bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22deffd700 1 -- 172.21.15.31:0/3957480046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d00095f0 con 0x7f22e01026a0 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22de7fc700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e0103040 0x7f22e010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:02.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22deffd700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 0x7f22e010ef40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22d000c010 tx=0x7f22d0009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d001c070 con 0x7f22e01026a0 2023-07-29T16:13:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e01a9340 con 0x7f22e01026a0 2023-07-29T16:13:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22d000fdf0 con 0x7f22e01026a0 2023-07-29T16:13:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d0022d90 con 0x7f22e01026a0 2023-07-29T16:13:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.825+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e01a97b0 con 0x7f22e01026a0 2023-07-29T16:13:02.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.827+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e0076930 con 0x7f22e01026a0 2023-07-29T16:13:02.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.829+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f22d0020070 con 0x7f22e01026a0 2023-07-29T16:13:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.830+0000 7f22c7fff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22c80a08a0 0x7f22c80a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.831+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f22d00ccab0 con 0x7f22e01026a0 2023-07-29T16:13:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.831+0000 7f22de7fc700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22c80a08a0 0x7f22c80a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:02.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.832+0000 7f22de7fc700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22c80a08a0 0x7f22c80a2d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f22e010d750 tx=0x7f22d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:02.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:02.832+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f22d0096490 con 0x7f22e01026a0 2023-07-29T16:13:03.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 -- 172.21.15.31:0/17461178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 msgr2=0x7f054010c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 --2- 172.21.15.31:0/17461178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f054010c830 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0534009990 tx=0x7f0534009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 -- 172.21.15.31:0/17461178 shutdown_connections 2023-07-29T16:13:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 --2- 172.21.15.31:0/17461178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f054010f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 --2- 172.21.15.31:0/17461178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05400ff1c0 0x7f05400ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 --2- 172.21.15.31:0/17461178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f054010c830 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 -- 172.21.15.31:0/17461178 >> 172.21.15.31:0/17461178 conn(0x7f05400763b0 msgr2=0x7f05400767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:03.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 -- 172.21.15.31:0/17461178 shutdown_connections 2023-07-29T16:13:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.012+0000 7f05453ae700 1 -- 172.21.15.31:0/17461178 wait complete. 2023-07-29T16:13:03.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.014+0000 7f05453ae700 1 Processor -- start 2023-07-29T16:13:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.014+0000 7f05453ae700 1 -- start start 2023-07-29T16:13:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f05453ae700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05400ff1c0 0x7f0540105960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f05453ae700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f0540105ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f05453ae700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f05401013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f05453ae700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0540101ba0 con 0x7f05400ffad0 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f05453ae700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0540101d10 con 0x7f054010cd70 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f05453ae700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0540101ed0 con 0x7f05400ff1c0 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f053f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f05401013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f053e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f0540105ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f053f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f05401013d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42976/0 (socket says 172.21.15.31:42976) 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f053f7fe700 1 -- 172.21.15.31:0/1014775370 learned_addr learned my addr 172.21.15.31:0/1014775370 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:03.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.015+0000 7f053e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f0540105ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36622/0 (socket says 172.21.15.31:36622) 2023-07-29T16:13:03.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.016+0000 7f053f7fe700 1 -- 172.21.15.31:0/1014775370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05400ff1c0 msgr2=0x7f0540105960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:13:03.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.016+0000 7f053f7fe700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05400ff1c0 0x7f0540105960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.016+0000 7f053f7fe700 1 -- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 msgr2=0x7f0540105ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.016+0000 7f053f7fe700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f0540105ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.016+0000 7f053f7fe700 1 -- 172.21.15.31:0/1014775370 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340095f0 con 0x7f054010cd70 2023-07-29T16:13:03.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.016+0000 7f053e7fc700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f0540105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:03.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.017+0000 7f053f7fe700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f05401013d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f053800ef90 tx=0x7f053800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:03.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.017+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538018070 con 0x7f054010cd70 2023-07-29T16:13:03.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.017+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f053800f040 con 0x7f054010cd70 2023-07-29T16:13:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.017+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f053800cd70 con 0x7f054010cd70 2023-07-29T16:13:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.017+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0540102150 con 0x7f054010cd70 2023-07-29T16:13:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.017+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054010b920 con 0x7f054010cd70 2023-07-29T16:13:03.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.019+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052c005320 con 0x7f054010cd70 2023-07-29T16:13:03.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.021+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f0538004ad0 con 0x7f054010cd70 2023-07-29T16:13:03.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.024+0000 7f0527fff700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05280a0820 0x7f05280a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.024+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f05380cb760 con 0x7f054010cd70 2023-07-29T16:13:03.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.024+0000 7f053effd700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05280a0820 0x7f05280a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.025+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f05380950b0 con 0x7f054010cd70 2023-07-29T16:13:03.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.025+0000 7f053effd700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05280a0820 0x7f05280a2cd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0530009af0 tx=0x7f0530009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:03.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.084+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f22e010d490 con 0x7f22e01026a0 2023-07-29T16:13:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.085+0000 7f22c7fff700 1 -- 172.21.15.31:0/3957480046 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f22d0027020 con 0x7f22e01026a0 2023-07-29T16:13:03.083 INFO:teuthology.orchestra.run.smithi031.stdout:111669149753 2023-07-29T16:13:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22c80a08a0 msgr2=0x7f22c80a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22c80a08a0 0x7f22c80a2d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f22e010d750 tx=0x7f22d4009260 comp rx=0 tx=0).stop 2023-07-29T16:13:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 msgr2=0x7f22e010ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 0x7f22e010ef40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f22d000c010 tx=0x7f22d0009580 comp rx=0 tx=0).stop 2023-07-29T16:13:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 shutdown_connections 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22c80a08a0 0x7f22c80a2d50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22e0107740 0x7f22e010c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f22e01026a0 0x7f22e010ef40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 --2- 172.21.15.31:0/3957480046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22e0103040 0x7f22e010bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 >> 172.21.15.31:0/3957480046 conn(0x7f22e00fdf60 msgr2=0x7f22e00ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 shutdown_connections 2023-07-29T16:13:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.088+0000 7f22dffff700 1 -- 172.21.15.31:0/3957480046 wait complete. 2023-07-29T16:13:03.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 -- 172.21.15.31:0/1215802094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 msgr2=0x7f73ac10f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/1215802094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac10f000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b0a0 tx=0x7f73a400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 -- 172.21.15.31:0/1215802094 shutdown_connections 2023-07-29T16:13:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/1215802094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac10f000 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/1215802094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73ac0730f0 0x7f73ac0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/1215802094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac073a00 0x7f73ac10c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.166+0000 7f73b2bf8700 1 -- 172.21.15.31:0/1215802094 >> 172.21.15.31:0/1215802094 conn(0x7f73ac0fbea0 msgr2=0x7f73ac0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.167+0000 7f73b2bf8700 1 -- 172.21.15.31:0/1215802094 shutdown_connections 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.167+0000 7f73b2bf8700 1 -- 172.21.15.31:0/1215802094 wait complete. 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 Processor -- start 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 -- start start 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac0730f0 0x7f73ac1986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73ac073a00 0x7f73ac198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac19bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac19c650 con 0x7f73ac10cc20 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac19c7c0 con 0x7f73ac0730f0 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b2bf8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ac19c980 con 0x7f73ac073a00 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b0994700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac0730f0 0x7f73ac1986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b0994700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac0730f0 0x7f73ac1986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43002/0 (socket says 172.21.15.31:43002) 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b0994700 1 -- 172.21.15.31:0/3496503337 learned_addr learned my addr 172.21.15.31:0/3496503337 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73b1195700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac19bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.168+0000 7f73abfff700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73ac073a00 0x7f73ac198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b1195700 1 -- 172.21.15.31:0/3496503337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73ac073a00 msgr2=0x7f73ac198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b1195700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73ac073a00 0x7f73ac198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b1195700 1 -- 172.21.15.31:0/3496503337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac0730f0 msgr2=0x7f73ac1986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b1195700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac0730f0 0x7f73ac1986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b1195700 1 -- 172.21.15.31:0/3496503337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a4009d00 con 0x7f73ac10cc20 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b1195700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac19bfc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b070 tx=0x7f73a40078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a400e040 con 0x7f73ac10cc20 2023-07-29T16:13:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73a401d470 con 0x7f73ac10cc20 2023-07-29T16:13:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ac19cc00 con 0x7f73ac10cc20 2023-07-29T16:13:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.169+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a40052a0 con 0x7f73ac10cc20 2023-07-29T16:13:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.170+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ac1a5520 con 0x7f73ac10cc20 2023-07-29T16:13:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.171+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ac107540 con 0x7f73ac10cc20 2023-07-29T16:13:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.174+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f73a4004a50 con 0x7f73ac10cc20 2023-07-29T16:13:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.176+0000 7f73a9ffb700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f739c0a0820 0x7f739c0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.176+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f73a40cd050 con 0x7f73ac10cc20 2023-07-29T16:13:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.176+0000 7f73b0994700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f739c0a0820 0x7f739c0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.177+0000 7f73b0994700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f739c0a0820 0x7f739c0a2cd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009d20 tx=0x7f73a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.184+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f73a4096a00 con 0x7f73ac10cc20 2023-07-29T16:13:03.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.270+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f052c005190 con 0x7f054010cd70 2023-07-29T16:13:03.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.270+0000 7f0527fff700 1 -- 172.21.15.31:0/1014775370 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05380986d0 con 0x7f054010cd70 2023-07-29T16:13:03.269 INFO:teuthology.orchestra.run.smithi031.stdout:55834574913 2023-07-29T16:13:03.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05280a0820 msgr2=0x7f05280a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05280a0820 0x7f05280a2cd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0530009af0 tx=0x7f0530009260 comp rx=0 tx=0).stop 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 msgr2=0x7f05401013d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f05401013d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f053800ef90 tx=0x7f053800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 shutdown_connections 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05280a0820 0x7f05280a2cd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f054010cd70 0x7f05401013d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05400ff1c0 0x7f0540105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 --2- 172.21.15.31:0/1014775370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05400ffad0 0x7f0540105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 >> 172.21.15.31:0/1014775370 conn(0x7f05400763b0 msgr2=0x7f05400fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:03.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 shutdown_connections 2023-07-29T16:13:03.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.273+0000 7f05453ae700 1 -- 172.21.15.31:0/1014775370 wait complete. 2023-07-29T16:13:03.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.422+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f73ac04ea50 con 0x7f73ac10cc20 2023-07-29T16:13:03.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.422+0000 7f73a9ffb700 1 -- 172.21.15.31:0/3496503337 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f73a401d5e0 con 0x7f73ac10cc20 2023-07-29T16:13:03.421 INFO:teuthology.orchestra.run.smithi031.stdout:292057776156 2023-07-29T16:13:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f739c0a0820 msgr2=0x7f739c0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f739c0a0820 0x7f739c0a2cd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009d20 tx=0x7f73a0009260 comp rx=0 tx=0).stop 2023-07-29T16:13:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 msgr2=0x7f73ac19bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac19bfc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b070 tx=0x7f73a40078e0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 shutdown_connections 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f739c0a0820 0x7f739c0a2cd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73ac10cc20 0x7f73ac19bfc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f73ac0730f0 0x7f73ac1986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 --2- 172.21.15.31:0/3496503337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73ac073a00 0x7f73ac198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 >> 172.21.15.31:0/3496503337 conn(0x7f73ac0fbea0 msgr2=0x7f73ac104680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 shutdown_connections 2023-07-29T16:13:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:03.426+0000 7f73b2bf8700 1 -- 172.21.15.31:0/3496503337 wait complete. 2023-07-29T16:13:03.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3957480046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-29T16:13:03.518 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1014775370' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-29T16:13:03.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3957480046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-29T16:13:03.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1014775370' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-29T16:13:03.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3957480046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-29T16:13:03.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1014775370' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-29T16:13:04.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149751 got 111669149753 for osd.3 2023-07-29T16:13:04.097 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:04.293 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776154 got 292057776156 for osd.9 2023-07-29T16:13:04.293 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:04.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:04 smithi031 ceph-mon[95504]: pgmap v295: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 48 B/s, 0 objects/s recovering 2023-07-29T16:13:04.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:04 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3496503337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-29T16:13:04.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:04 smithi039 ceph-mon[128858]: pgmap v295: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 48 B/s, 0 objects/s recovering 2023-07-29T16:13:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:04 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3496503337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-29T16:13:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:04 smithi156 ceph-mon[129245]: pgmap v295: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 48 B/s, 0 objects/s recovering 2023-07-29T16:13:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:04 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3496503337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-29T16:13:05.056 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574911 got 55834574913 for osd.1 2023-07-29T16:13:05.057 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.346+0000 7f0d9c172700 1 -- 172.21.15.31:0/3159576778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 msgr2=0x7f0d94101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.346+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3159576778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 0x7f0d94101a00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009990 tx=0x7f0d90009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 -- 172.21.15.31:0/3159576778 shutdown_connections 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3159576778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 0x7f0d9410cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3159576778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 0x7f0d94101a00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3159576778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d9410a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 -- 172.21.15.31:0/3159576778 >> 172.21.15.31:0/3159576778 conn(0x7f0d940763b0 msgr2=0x7f0d940767b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 -- 172.21.15.31:0/3159576778 shutdown_connections 2023-07-29T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.347+0000 7f0d9c172700 1 -- 172.21.15.31:0/3159576778 wait complete. 2023-07-29T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 Processor -- start 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 -- start start 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 0x7f0d941964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d94196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 0x7f0d9419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9419b480 con 0x7f0d94101630 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9419b5f0 con 0x7f0d94101f40 2023-07-29T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.348+0000 7f0d9c172700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9419b7b0 con 0x7f0d9410a980 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9970d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d94196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9970d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d94196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43014/0 (socket says 172.21.15.31:43014) 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9970d700 1 -- 172.21.15.31:0/3784210780 learned_addr learned my addr 172.21.15.31:0/3784210780 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9a70f700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 0x7f0d9419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d99f0e700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 0x7f0d941964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9a70f700 1 -- 172.21.15.31:0/3784210780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 msgr2=0x7f0d94196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9a70f700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d94196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9a70f700 1 -- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 msgr2=0x7f0d941964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9a70f700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 0x7f0d941964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9a70f700 1 -- 172.21.15.31:0/3784210780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900095f0 con 0x7f0d9410a980 2023-07-29T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.349+0000 7f0d9970d700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d94196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.350+0000 7f0d9a70f700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 0x7f0d9419ae70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00efc0 tx=0x7f0d8c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.350+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c018070 con 0x7f0d9410a980 2023-07-29T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.350+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d8c00f040 con 0x7f0d9410a980 2023-07-29T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.350+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c00cc80 con 0x7f0d9410a980 2023-07-29T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.350+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9419ba30 con 0x7f0d9410a980 2023-07-29T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.350+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d940ff390 con 0x7f0d9410a980 2023-07-29T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.351+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9404ea50 con 0x7f0d9410a980 2023-07-29T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.353+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f0d8c003a00 con 0x7f0d9410a980 2023-07-29T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.356+0000 7f0d8affd700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d800a08a0 0x7f0d800a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.356+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f0d8c0cb400 con 0x7f0d9410a980 2023-07-29T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.356+0000 7f0d99f0e700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d800a08a0 0x7f0d800a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.357+0000 7f0d99f0e700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d800a08a0 0x7f0d800a2d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000b2e0 tx=0x7f0d90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.358+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0d8c094dd0 con 0x7f0d9410a980 2023-07-29T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:05.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411386 got 77309411388 for osd.2 2023-07-29T16:13:05.538 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.618+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0d9419bc20 con 0x7f0d9410a980 2023-07-29T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.619+0000 7f0d8affd700 1 -- 172.21.15.31:0/3784210780 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0d8c0983f0 con 0x7f0d9410a980 2023-07-29T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout:201863462955 2023-07-29T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.621+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d800a08a0 msgr2=0x7f0d800a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.621+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d800a08a0 0x7f0d800a2d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d9000b2e0 tx=0x7f0d90005fd0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.621+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 msgr2=0x7f0d9419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.621+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 0x7f0d9419ae70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00efc0 tx=0x7f0d8c00c3a0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 shutdown_connections 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d800a08a0 0x7f0d800a2d50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d9410a980 0x7f0d9419ae70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d94101630 0x7f0d941964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 --2- 172.21.15.31:0/3784210780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d94101f40 0x7f0d94196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 >> 172.21.15.31:0/3784210780 conn(0x7f0d940763b0 msgr2=0x7f0d9410b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 shutdown_connections 2023-07-29T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.622+0000 7f0d9c172700 1 -- 172.21.15.31:0/3784210780 wait complete. 2023-07-29T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.921+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/3054280177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 msgr2=0x7fcc5810c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.921+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/3054280177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 0x7fcc5810c6f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009990 tx=0x7fcc4c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/3054280177 shutdown_connections 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/3054280177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5810f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/3054280177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc580730f0 0x7fcc580734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/3054280177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 0x7fcc5810c6f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/3054280177 >> 172.21.15.31:0/3054280177 conn(0x7fcc580fbed0 msgr2=0x7fcc580fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/3054280177 shutdown_connections 2023-07-29T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.922+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/3054280177 wait complete. 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 Processor -- start 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 -- start start 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc580730f0 0x7fcc581985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 0x7fcc58198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5819cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5819d640 con 0x7fcc5810cc30 2023-07-29T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5819d7b0 con 0x7fcc58073a00 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.923+0000 7fcc5cdb7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5819d970 con 0x7fcc580730f0 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5819cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5819cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36678/0 (socket says 172.21.15.31:36678) 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 -- 172.21.15.31:0/319382891 learned_addr learned my addr 172.21.15.31:0/319382891 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc5659c700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc580730f0 0x7fcc581985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc55d9b700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 0x7fcc58198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 -- 172.21.15.31:0/319382891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc580730f0 msgr2=0x7fcc581985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc580730f0 0x7fcc581985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 -- 172.21.15.31:0/319382891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 msgr2=0x7fcc58198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 0x7fcc58198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 -- 172.21.15.31:0/319382891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c0095f0 con 0x7fcc5810cc30 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc56d9d700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5819cfa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000ef30 tx=0x7fcc5000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.924+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50018070 con 0x7fcc5810cc30 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.925+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc5000f040 con 0x7fcc5810cc30 2023-07-29T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.925+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5819dbf0 con 0x7fcc5810cc30 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.925+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000cd00 con 0x7fcc5810cc30 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.925+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc581a5420 con 0x7fcc5810cc30 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.926+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5804ea50 con 0x7fcc5810cc30 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.928+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fcc50003a00 con 0x7fcc5810cc30 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.930+0000 7fcc477fe700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc400a0820 0x7fcc400a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.930+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fcc500cb6c0 con 0x7fcc5810cc30 2023-07-29T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.930+0000 7fcc5659c700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc400a0820 0x7fcc400a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.931+0000 7fcc5659c700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc400a0820 0x7fcc400a2cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009c60 tx=0x7fcc48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:05.935+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcc50095010 con 0x7fcc5810cc30 2023-07-29T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.175+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fcc581a56e0 con 0x7fcc5810cc30 2023-07-29T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.175+0000 7fcc477fe700 1 -- 172.21.15.31:0/319382891 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fcc50098630 con 0x7fcc5810cc30 2023-07-29T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout:313532612633 2023-07-29T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc400a0820 msgr2=0x7fcc400a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc400a0820 0x7fcc400a2cd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009c60 tx=0x7fcc48009260 comp rx=0 tx=0).stop 2023-07-29T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 msgr2=0x7fcc5819cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5819cfa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000ef30 tx=0x7fcc5000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 shutdown_connections 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc400a0820 0x7fcc400a2cd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc5810cc30 0x7fcc5819cfa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc580730f0 0x7fcc581985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 --2- 172.21.15.31:0/319382891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc58073a00 0x7fcc58198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 >> 172.21.15.31:0/319382891 conn(0x7fcc580fbed0 msgr2=0x7fcc5810ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 shutdown_connections 2023-07-29T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.178+0000 7fcc5cdb7700 1 -- 172.21.15.31:0/319382891 wait complete. 2023-07-29T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.404+0000 7f7b93a14700 1 -- 172.21.15.31:0/3357625145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c073a00 msgr2=0x7f7b8c110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 --2- 172.21.15.31:0/3357625145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c073a00 0x7f7b8c110eb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009990 tx=0x7f7b7c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 -- 172.21.15.31:0/3357625145 shutdown_connections 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 --2- 172.21.15.31:0/3357625145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c1113f0 0x7f7b8c1137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 --2- 172.21.15.31:0/3357625145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c0730f0 0x7f7b8c0734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 --2- 172.21.15.31:0/3357625145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c073a00 0x7f7b8c110eb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 -- 172.21.15.31:0/3357625145 >> 172.21.15.31:0/3357625145 conn(0x7f7b8c0fbea0 msgr2=0x7f7b8c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 -- 172.21.15.31:0/3357625145 shutdown_connections 2023-07-29T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.405+0000 7f7b93a14700 1 -- 172.21.15.31:0/3357625145 wait complete. 2023-07-29T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.406+0000 7f7b93a14700 1 Processor -- start 2023-07-29T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.406+0000 7f7b93a14700 1 -- start start 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b93a14700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c0730f0 0x7f7b8c1a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b93a14700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c073a00 0x7f7b8c1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b93a14700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c1113f0 0x7f7b8c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b93a14700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c19cfd0 con 0x7f7b8c1113f0 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b93a14700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c19d140 con 0x7f7b8c073a00 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b93a14700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c19d300 con 0x7f7b8c0730f0 2023-07-29T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b91fb1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c1113f0 0x7f7b8c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.407+0000 7f7b90faf700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c073a00 0x7f7b8c1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b91fb1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c1113f0 0x7f7b8c19c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36718/0 (socket says 172.21.15.31:36718) 2023-07-29T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b91fb1700 1 -- 172.21.15.31:0/1947433733 learned_addr learned my addr 172.21.15.31:0/1947433733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b917b0700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c0730f0 0x7f7b8c1a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b90faf700 1 -- 172.21.15.31:0/1947433733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c0730f0 msgr2=0x7f7b8c1a28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b90faf700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c0730f0 0x7f7b8c1a28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b90faf700 1 -- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c1113f0 msgr2=0x7f7b8c19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b90faf700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c1113f0 0x7f7b8c19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b90faf700 1 -- 172.21.15.31:0/1947433733 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b88009520 con 0x7f7b8c073a00 2023-07-29T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.408+0000 7f7b90faf700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c073a00 0x7f7b8c1a2e20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0098e0 tx=0x7f7b7c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.409+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c01d070 con 0x7f7b8c073a00 2023-07-29T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.409+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c0095f0 con 0x7f7b8c073a00 2023-07-29T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.409+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c1a9700 con 0x7f7b8c073a00 2023-07-29T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.409+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b7c005030 con 0x7f7b8c073a00 2023-07-29T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.409+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c004ab0 con 0x7f7b8c073a00 2023-07-29T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.410+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c1038d0 con 0x7f7b8c073a00 2023-07-29T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.412+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f7b7c005430 con 0x7f7b8c073a00 2023-07-29T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.413+0000 7f7b827fc700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b780a0820 0x7f7b780a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.413+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f7b7c04ce40 con 0x7f7b8c073a00 2023-07-29T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.413+0000 7f7b917b0700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b780a0820 0x7f7b780a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.414+0000 7f7b917b0700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b780a0820 0x7f7b780a2cd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009df0 tx=0x7f7b88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.415+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7b7c096970 con 0x7f7b8c073a00 2023-07-29T16:13:06.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:06 smithi031 ceph-mon[95504]: pgmap v296: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:06.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:06 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3784210780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-29T16:13:06.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:06 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/319382891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-29T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.288+0000 7f97b6be0700 1 -- 172.21.15.31:0/1073671850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 msgr2=0x7f97b0110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.288+0000 7f97b6be0700 1 --2- 172.21.15.31:0/1073671850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b0110eb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f979c0098f0 tx=0x7f979c009c00 comp rx=0 tx=0).stop 2023-07-29T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 -- 172.21.15.31:0/1073671850 shutdown_connections 2023-07-29T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 --2- 172.21.15.31:0/1073671850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97b01113f0 0x7f97b01137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 --2- 172.21.15.31:0/1073671850 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b00730f0 0x7f97b00734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 --2- 172.21.15.31:0/1073671850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b0110eb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 -- 172.21.15.31:0/1073671850 >> 172.21.15.31:0/1073671850 conn(0x7f97b00fbea0 msgr2=0x7f97b00fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 -- 172.21.15.31:0/1073671850 shutdown_connections 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.289+0000 7f97b6be0700 1 -- 172.21.15.31:0/1073671850 wait complete. 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.290+0000 7f97b6be0700 1 Processor -- start 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.290+0000 7f97b6be0700 1 -- start start 2023-07-29T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97b6be0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b00730f0 0x7f97b01a2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97b6be0700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b01a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97b6be0700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97b01113f0 0x7f97b019c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97b6be0700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b019d010 con 0x7f97b0073a00 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97b6be0700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b019d180 con 0x7f97b01113f0 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97b6be0700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b019d340 con 0x7f97b00730f0 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b01a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b01a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36696/0 (socket says 172.21.15.31:36696) 2023-07-29T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.291+0000 7f97affff700 1 -- 172.21.15.31:0/897386337 learned_addr learned my addr 172.21.15.31:0/897386337 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97b517d700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97b01113f0 0x7f97b019c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97b497c700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b00730f0 0x7f97b01a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97affff700 1 -- 172.21.15.31:0/897386337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b00730f0 msgr2=0x7f97b01a2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97affff700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b00730f0 0x7f97b01a2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97affff700 1 -- 172.21.15.31:0/897386337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97b01113f0 msgr2=0x7f97b019c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97affff700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97b01113f0 0x7f97b019c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97affff700 1 -- 172.21.15.31:0/897386337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c009520 con 0x7f97b0073a00 2023-07-29T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.292+0000 7f97affff700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b01a2e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f979c009810 tx=0x7f979c00fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.293+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c01c070 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.293+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f979c005040 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.293+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c0049a0 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.293+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b019d5c0 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.293+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b01a9910 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.294+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b01038d0 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.296+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f979c020070 con 0x7f97b0073a00 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.297+0000 7f97adffb700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a00a08a0 0x7f97a00a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.297+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f979c0ccc90 con 0x7f97b0073a00 2023-07-29T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout:347892350996 2023-07-29T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.298+0000 7f97b497c700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a00a08a0 0x7f97a00a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.299+0000 7f97b497c700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a00a08a0 0x7f97a00a2d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f97a4005fd0 tx=0x7f97a4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.300+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f979c0965e0 con 0x7f97b0073a00 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.528+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f97b004ea50 con 0x7f97b0073a00 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.529+0000 7f97adffb700 1 -- 172.21.15.31:0/897386337 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f979c027090 con 0x7f97b0073a00 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a00a08a0 msgr2=0x7f97a00a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a00a08a0 0x7f97a00a2d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f97a4005fd0 tx=0x7f97a4009310 comp rx=0 tx=0).stop 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 msgr2=0x7f97b01a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b01a2e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f979c009810 tx=0x7f979c00fa50 comp rx=0 tx=0).stop 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 shutdown_connections 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a00a08a0 0x7f97a00a2d50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97b01113f0 0x7f97b019c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97b00730f0 0x7f97b01a2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 --2- 172.21.15.31:0/897386337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97b0073a00 0x7f97b01a2e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 >> 172.21.15.31:0/897386337 conn(0x7f97b00fbea0 msgr2=0x7f97b00fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 shutdown_connections 2023-07-29T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.531+0000 7f97b6be0700 1 -- 172.21.15.31:0/897386337 wait complete. 2023-07-29T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.629+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f7b8c04ea50 con 0x7f7b8c073a00 2023-07-29T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.630+0000 7f7b827fc700 1 -- 172.21.15.31:0/1947433733 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7b7c099f90 con 0x7f7b8c073a00 2023-07-29T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout:257698037793 2023-07-29T16:13:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b780a0820 msgr2=0x7f7b780a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b780a0820 0x7f7b780a2cd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009df0 tx=0x7f7b88009260 comp rx=0 tx=0).stop 2023-07-29T16:13:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c073a00 msgr2=0x7f7b8c1a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:06.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c073a00 0x7f7b8c1a2e20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c0098e0 tx=0x7f7b7c004730 comp rx=0 tx=0).stop 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 shutdown_connections 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7b780a0820 0x7f7b780a2cd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7b8c1113f0 0x7f7b8c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b8c0730f0 0x7f7b8c1a28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 --2- 172.21.15.31:0/1947433733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b8c073a00 0x7f7b8c1a2e20 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 >> 172.21.15.31:0/1947433733 conn(0x7f7b8c0fbea0 msgr2=0x7f7b8c0fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:06.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 shutdown_connections 2023-07-29T16:13:06.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:06.633+0000 7f7b93a14700 1 -- 172.21.15.31:0/1947433733 wait complete. 2023-07-29T16:13:06.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:06 smithi039 ceph-mon[128858]: pgmap v296: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:06 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3784210780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-29T16:13:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:06 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/319382891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-29T16:13:06.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:06 smithi156 ceph-mon[129245]: pgmap v296: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:06.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:06 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3784210780' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-29T16:13:06.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:06 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/319382891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-29T16:13:07.105 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612631 got 313532612633 for osd.10 2023-07-29T16:13:07.106 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:07.177 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350994 got 347892350996 for osd.11 2023-07-29T16:13:07.178 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:07.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462953 got 201863462955 for osd.6 2023-07-29T16:13:07.187 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:07.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:07 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/897386337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-29T16:13:07.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:07 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1947433733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-29T16:13:07.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:07 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/897386337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-29T16:13:07.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:07 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1947433733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-29T16:13:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:07 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/897386337' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-29T16:13:07.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:07 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1947433733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-29T16:13:08.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 -- 172.21.15.31:0/3409183141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8480739a0 msgr2=0x7ff84810c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/3409183141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8480739a0 0x7ff84810c580 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff834009990 tx=0x7ff834009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 -- 172.21.15.31:0/3409183141 shutdown_connections 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/3409183141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff84810cac0 0x7ff84810eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/3409183141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff848073090 0x7ff848073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/3409183141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8480739a0 0x7ff84810c580 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 -- 172.21.15.31:0/3409183141 >> 172.21.15.31:0/3409183141 conn(0x7ff8480fbda0 msgr2=0x7ff8480fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:08.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 -- 172.21.15.31:0/3409183141 shutdown_connections 2023-07-29T16:13:08.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.283+0000 7ff84dfdb700 1 -- 172.21.15.31:0/3409183141 wait complete. 2023-07-29T16:13:08.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.284+0000 7ff84dfdb700 1 Processor -- start 2023-07-29T16:13:08.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.284+0000 7ff84dfdb700 1 -- start start 2023-07-29T16:13:08.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84dfdb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 0x7ff848198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84dfdb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8480739a0 0x7ff848198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84dfdb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84810cac0 0x7ff84819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84dfdb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84819d530 con 0x7ff848073090 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84dfdb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84819d6a0 con 0x7ff84810cac0 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84dfdb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84819d860 con 0x7ff8480739a0 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84cfd9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 0x7ff848198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84cfd9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 0x7ff848198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36742/0 (socket says 172.21.15.31:36742) 2023-07-29T16:13:08.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84cfd9700 1 -- 172.21.15.31:0/833101556 learned_addr learned my addr 172.21.15.31:0/833101556 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff84d7da700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84810cac0 0x7ff84819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.285+0000 7ff847fff700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8480739a0 0x7ff848198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84cfd9700 1 -- 172.21.15.31:0/833101556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8480739a0 msgr2=0x7ff848198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84cfd9700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8480739a0 0x7ff848198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84cfd9700 1 -- 172.21.15.31:0/833101556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84810cac0 msgr2=0x7ff84819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84cfd9700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84810cac0 0x7ff84819cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84cfd9700 1 -- 172.21.15.31:0/833101556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8340095f0 con 0x7ff848073090 2023-07-29T16:13:08.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84cfd9700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 0x7ff848198570 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00e9a0 tx=0x7ff83c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:08.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c009790 con 0x7ff848073090 2023-07-29T16:13:08.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84819db40 con 0x7ff848073090 2023-07-29T16:13:08.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff83c010430 con 0x7ff848073090 2023-07-29T16:13:08.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c018680 con 0x7ff848073090 2023-07-29T16:13:08.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.286+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8481a53a0 con 0x7ff848073090 2023-07-29T16:13:08.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.287+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84804f2a0 con 0x7ff848073090 2023-07-29T16:13:08.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.289+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7ff83c0105a0 con 0x7ff848073090 2023-07-29T16:13:08.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.291+0000 7ff845ffb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8380a0820 0x7ff8380a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.291+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff83c014070 con 0x7ff848073090 2023-07-29T16:13:08.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.291+0000 7ff847fff700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8380a0820 0x7ff8380a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.292+0000 7ff847fff700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8380a0820 0x7ff8380a2cd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff848199b90 tx=0x7ff834005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:08.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.293+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff83c0959e0 con 0x7ff848073090 2023-07-29T16:13:08.431 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037791 got 257698037793 for osd.8 2023-07-29T16:13:08.431 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:08.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.517+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7ff8481998d0 con 0x7ff848073090 2023-07-29T16:13:08.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.518+0000 7ff845ffb700 1 -- 172.21.15.31:0/833101556 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff83c026020 con 0x7ff848073090 2023-07-29T16:13:08.516 INFO:teuthology.orchestra.run.smithi031.stdout:34359738439 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.520+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8380a0820 msgr2=0x7ff8380a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.520+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8380a0820 0x7ff8380a2cd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff848199b90 tx=0x7ff834005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 msgr2=0x7ff848198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 0x7ff848198570 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff83c00e9a0 tx=0x7ff83c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 shutdown_connections 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8380a0820 0x7ff8380a2cd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff84810cac0 0x7ff84819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff848073090 0x7ff848198570 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 --2- 172.21.15.31:0/833101556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8480739a0 0x7ff848198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 >> 172.21.15.31:0/833101556 conn(0x7ff8480fbda0 msgr2=0x7ff84810ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:08.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 shutdown_connections 2023-07-29T16:13:08.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.521+0000 7ff84dfdb700 1 -- 172.21.15.31:0/833101556 wait complete. 2023-07-29T16:13:08.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:08 smithi031 ceph-mon[95504]: pgmap v297: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 -- 172.21.15.31:0/2900423540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff44073070 msgr2=0x7eff44073440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 --2- 172.21.15.31:0/2900423540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff44073070 0x7eff44073440 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7eff34009990 tx=0x7eff34009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 -- 172.21.15.31:0/2900423540 shutdown_connections 2023-07-29T16:13:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 --2- 172.21.15.31:0/2900423540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff4410cca0 0x7eff4410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 --2- 172.21.15.31:0/2900423540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff44073070 0x7eff44073440 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 --2- 172.21.15.31:0/2900423540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff44073980 0x7eff4410c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 -- 172.21.15.31:0/2900423540 >> 172.21.15.31:0/2900423540 conn(0x7eff440fbea0 msgr2=0x7eff440fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 -- 172.21.15.31:0/2900423540 shutdown_connections 2023-07-29T16:13:08.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.740+0000 7eff48d5d700 1 -- 172.21.15.31:0/2900423540 wait complete. 2023-07-29T16:13:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.741+0000 7eff48d5d700 1 Processor -- start 2023-07-29T16:13:08.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 -- start start 2023-07-29T16:13:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 0x7eff44198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff44073980 0x7eff44198ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff4410cca0 0x7eff4419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419d620 con 0x7eff4410cca0 2023-07-29T16:13:08.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419d790 con 0x7eff44073980 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.742+0000 7eff48d5d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4419d950 con 0x7eff44073070 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 0x7eff44198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 0x7eff44198660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:33064/0 (socket says 172.21.15.31:33064) 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 -- 172.21.15.31:0/3576612153 learned_addr learned my addr 172.21.15.31:0/3576612153 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff42d9d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff4410cca0 0x7eff4419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff41d9b700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff44073980 0x7eff44198ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 -- 172.21.15.31:0/3576612153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff44073980 msgr2=0x7eff44198ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff44073980 0x7eff44198ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 -- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff4410cca0 msgr2=0x7eff4419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff4410cca0 0x7eff4419d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 -- 172.21.15.31:0/3576612153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340095f0 con 0x7eff44073070 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.743+0000 7eff4259c700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 0x7eff44198660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff34000c00 tx=0x7eff34005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.744+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3401d070 con 0x7eff44073070 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.744+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4419dbd0 con 0x7eff44073070 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.744+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff34022470 con 0x7eff44073070 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.744+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34017650 con 0x7eff44073070 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.744+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff441a5490 con 0x7eff44073070 2023-07-29T16:13:08.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.745+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4404ea50 con 0x7eff44073070 2023-07-29T16:13:08.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.747+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7eff340225e0 con 0x7eff44073070 2023-07-29T16:13:08.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.749+0000 7eff337fe700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff2c0a0820 0x7eff2c0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.749+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7eff340cd210 con 0x7eff44073070 2023-07-29T16:13:08.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.749+0000 7eff41d9b700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff2c0a0820 0x7eff2c0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.750+0000 7eff41d9b700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff2c0a0820 0x7eff2c0a2cd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff44199c80 tx=0x7eff38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.750+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7eff34096b60 con 0x7eff44073070 2023-07-29T16:13:08.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:08 smithi039 ceph-mon[128858]: pgmap v297: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:08.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:08 smithi156 ceph-mon[129245]: pgmap v297: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 -- 172.21.15.31:0/2367241935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8900730f0 msgr2=0x7ff8900734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 --2- 172.21.15.31:0/2367241935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8900730f0 0x7ff8900734c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff880009920 tx=0x7ff880009c30 comp rx=0 tx=0).stop 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 -- 172.21.15.31:0/2367241935 shutdown_connections 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 --2- 172.21.15.31:0/2367241935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8901113f0 0x7ff8901137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 --2- 172.21.15.31:0/2367241935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8900730f0 0x7ff8900734c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 --2- 172.21.15.31:0/2367241935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff890073a00 0x7ff890110eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 -- 172.21.15.31:0/2367241935 >> 172.21.15.31:0/2367241935 conn(0x7ff8900fbea0 msgr2=0x7ff8900fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 -- 172.21.15.31:0/2367241935 shutdown_connections 2023-07-29T16:13:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.851+0000 7ff895ea3700 1 -- 172.21.15.31:0/2367241935 wait complete. 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.852+0000 7ff895ea3700 1 Processor -- start 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.852+0000 7ff895ea3700 1 -- start start 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff895ea3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 0x7ff8901a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff895ea3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff890073a00 0x7ff8901a2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff895ea3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8901113f0 0x7ff89019c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff895ea3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89019d040 con 0x7ff8900730f0 2023-07-29T16:13:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff895ea3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89019d1b0 con 0x7ff8901113f0 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff895ea3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89019d370 con 0x7ff890073a00 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff88f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 0x7ff8901a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff88f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 0x7ff8901a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36778/0 (socket says 172.21.15.31:36778) 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff88ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8901113f0 0x7ff89019c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff88f7fe700 1 -- 172.21.15.31:0/1449957498 learned_addr learned my addr 172.21.15.31:0/1449957498 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff88effd700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff890073a00 0x7ff8901a2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.853+0000 7ff88f7fe700 1 -- 172.21.15.31:0/1449957498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff890073a00 msgr2=0x7ff8901a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88f7fe700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff890073a00 0x7ff8901a2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88f7fe700 1 -- 172.21.15.31:0/1449957498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8901113f0 msgr2=0x7ff89019c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88f7fe700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8901113f0 0x7ff89019c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88f7fe700 1 -- 172.21.15.31:0/1449957498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff888009d00 con 0x7ff8900730f0 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88ffff700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8901113f0 0x7ff89019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88f7fe700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 0x7ff8901a2950 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff880009230 tx=0x7ff88000f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88001c070 con 0x7ff8900730f0 2023-07-29T16:13:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff88000bb20 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff880009520 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800049c0 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.854+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901a9770 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.855+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8901038d0 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.859+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7ff880020070 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.860+0000 7ff88cff9700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8780a08a0 0x7ff8780a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.860+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff8800ccc10 con 0x7ff8900730f0 2023-07-29T16:13:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.860+0000 7ff88effd700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8780a08a0 0x7ff8780a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.861+0000 7ff88effd700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8780a08a0 0x7ff8780a2d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff890112ba0 tx=0x7ff884009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.862+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff880096560 con 0x7ff8900730f0 2023-07-29T16:13:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.967+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7eff441a5620 con 0x7eff44073070 2023-07-29T16:13:08.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.967+0000 7eff337fe700 1 -- 172.21.15.31:0/3576612153 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7eff34027090 con 0x7eff44073070 2023-07-29T16:13:08.966 INFO:teuthology.orchestra.run.smithi031.stdout:167503724592 2023-07-29T16:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff2c0a0820 msgr2=0x7eff2c0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff2c0a0820 0x7eff2c0a2cd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7eff44199c80 tx=0x7eff38009260 comp rx=0 tx=0).stop 2023-07-29T16:13:08.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 msgr2=0x7eff44198660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 0x7eff44198660 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff34000c00 tx=0x7eff34005740 comp rx=0 tx=0).stop 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 shutdown_connections 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff2c0a0820 0x7eff2c0a2cd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff4410cca0 0x7eff4419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff44073070 0x7eff44198660 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 --2- 172.21.15.31:0/3576612153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff44073980 0x7eff44198ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:08.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 >> 172.21.15.31:0/3576612153 conn(0x7eff440fbea0 msgr2=0x7eff4410ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:08.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 shutdown_connections 2023-07-29T16:13:08.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:08.970+0000 7eff48d5d700 1 -- 172.21.15.31:0/3576612153 wait complete. 2023-07-29T16:13:09.027 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738436 got 34359738439 for osd.0 2023-07-29T16:13:09.028 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:09.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.084+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff89004ea50 con 0x7ff8900730f0 2023-07-29T16:13:09.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.085+0000 7ff88cff9700 1 -- 172.21.15.31:0/1449957498 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff880027090 con 0x7ff8900730f0 2023-07-29T16:13:09.083 INFO:teuthology.orchestra.run.smithi031.stdout:223338299431 2023-07-29T16:13:09.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8780a08a0 msgr2=0x7ff8780a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:09.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8780a08a0 0x7ff8780a2d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff890112ba0 tx=0x7ff884009260 comp rx=0 tx=0).stop 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 msgr2=0x7ff8901a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 0x7ff8901a2950 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff880009230 tx=0x7ff88000f9e0 comp rx=0 tx=0).stop 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 shutdown_connections 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8780a08a0 0x7ff8780a2d50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8901113f0 0x7ff89019c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8900730f0 0x7ff8901a2950 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:09.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 --2- 172.21.15.31:0/1449957498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff890073a00 0x7ff8901a2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.087+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 >> 172.21.15.31:0/1449957498 conn(0x7ff8900fbea0 msgr2=0x7ff8900fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.088+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 shutdown_connections 2023-07-29T16:13:09.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:09.088+0000 7ff895ea3700 1 -- 172.21.15.31:0/1449957498 wait complete. 2023-07-29T16:13:09.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724590 got 167503724592 for osd.5 2023-07-29T16:13:09.425 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:09.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:09 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/833101556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-29T16:13:09.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:09 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3576612153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-29T16:13:09.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:09 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1449957498' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-29T16:13:09.707 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299428 got 223338299431 for osd.7 2023-07-29T16:13:09.707 DEBUG:teuthology.parallel:result is None 2023-07-29T16:13:09.707 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-29T16:13:09.708 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph pg dump --format=json 2023-07-29T16:13:09.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:09 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/833101556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-29T16:13:09.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:09 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3576612153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-29T16:13:09.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:09 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1449957498' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-29T16:13:09.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:09 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/833101556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-29T16:13:09.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:09 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3576612153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-29T16:13:09.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:09 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1449957498' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-29T16:13:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:10.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:10 smithi031 ceph-mon[95504]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-07-29T16:13:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:10 smithi039 ceph-mon[128858]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-07-29T16:13:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:10 smithi156 ceph-mon[129245]: pgmap v298: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-07-29T16:13:11.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/1933566140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 msgr2=0x7f5e740734c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:11.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/1933566140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e740734c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009990 tx=0x7f5e70009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/1933566140 shutdown_connections 2023-07-29T16:13:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/1933566140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e7410cc20 0x7f5e7410f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/1933566140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e740734c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/1933566140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e74073a00 0x7f5e7410c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.914+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/1933566140 >> 172.21.15.31:0/1933566140 conn(0x7f5e740fbea0 msgr2=0x7f5e740fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.915+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/1933566140 shutdown_connections 2023-07-29T16:13:11.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.915+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/1933566140 wait complete. 2023-07-29T16:13:11.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 Processor -- start 2023-07-29T16:13:11.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 -- start start 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e741985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e74073a00 0x7f5e74198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e7410cc20 0x7f5e7419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7419d630 con 0x7f5e740730f0 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7419d7a0 con 0x7f5e7410cc20 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7c6bb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7419d960 con 0x7f5e74073a00 2023-07-29T16:13:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7a457700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e741985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7a457700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e741985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44884/0 (socket says 172.21.15.31:44884) 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.916+0000 7f5e7a457700 1 -- 172.21.15.31:0/4055411592 learned_addr learned my addr 172.21.15.31:0/4055411592 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e79c56700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e74073a00 0x7f5e74198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7ac58700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e7410cc20 0x7f5e7419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7a457700 1 -- 172.21.15.31:0/4055411592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e74073a00 msgr2=0x7f5e74198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7a457700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e74073a00 0x7f5e74198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7a457700 1 -- 172.21.15.31:0/4055411592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e7410cc20 msgr2=0x7f5e7419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7a457700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e7410cc20 0x7f5e7419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7a457700 1 -- 172.21.15.31:0/4055411592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e700095f0 con 0x7f5e740730f0 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7a457700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e741985e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e70000c00 tx=0x7f5e70004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7001d070 con 0x7f5e740730f0 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e70021410 con 0x7f5e740730f0 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7419dbe0 con 0x7f5e740730f0 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.917+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70017650 con 0x7f5e740730f0 2023-07-29T16:13:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.918+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e741a5410 con 0x7f5e740730f0 2023-07-29T16:13:11.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.919+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e74107540 con 0x7f5e740730f0 2023-07-29T16:13:11.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.920+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f5e70021580 con 0x7f5e740730f0 2023-07-29T16:13:11.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.922+0000 7f5e6b7fe700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e600a08a0 0x7f5e600a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:11.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.922+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f5e700ccf00 con 0x7f5e740730f0 2023-07-29T16:13:11.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.922+0000 7f5e79c56700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e600a08a0 0x7f5e600a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:11.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.923+0000 7f5e79c56700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e600a08a0 0x7f5e600a2d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5e74199c00 tx=0x7f5e64009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:11.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:11.926+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5e70096850 con 0x7f5e740730f0 2023-07-29T16:13:12.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.128+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e74061190 con 0x7f5e600a08a0 2023-07-29T16:13:12.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.139+0000 7f5e6b7fe700 1 -- 172.21.15.31:0/4055411592 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+339676 (secure 0 0 0) 0x7f5e74061190 con 0x7f5e600a08a0 2023-07-29T16:13:12.143 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:12.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.144+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e600a08a0 msgr2=0x7f5e600a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.144+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e600a08a0 0x7f5e600a2d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5e74199c00 tx=0x7f5e64009260 comp rx=0 tx=0).stop 2023-07-29T16:13:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.144+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 msgr2=0x7f5e741985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.144+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e741985e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e70000c00 tx=0x7f5e70004790 comp rx=0 tx=0).stop 2023-07-29T16:13:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 shutdown_connections 2023-07-29T16:13:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e600a08a0 0x7f5e600a2d50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:12.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e7410cc20 0x7f5e7419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e740730f0 0x7f5e741985e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 --2- 172.21.15.31:0/4055411592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e74073a00 0x7f5e74198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 >> 172.21.15.31:0/4055411592 conn(0x7f5e740fbea0 msgr2=0x7f5e74104680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 shutdown_connections 2023-07-29T16:13:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:12.145+0000 7f5e7c6bb700 1 -- 172.21.15.31:0/4055411592 wait complete. 2023-07-29T16:13:12.146 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:13:12.431 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:12 smithi031 ceph-mon[95504]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:12.521 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2023-07-29T16:13:11.282896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651806,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":401,"num_read_kb":311,"num_write":331,"num_write_kb":3041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":269,"ondisk_log_size":269,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336752,"kb_used_data":13328,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124605072,"statfs":{"total":1151940427776,"available":1151595593728,"internally_reserved":0,"allocated":13647872,"data_stored":7461352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003179"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283337+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:12:49.283337+0000","last_peered":"2023-07-29T16:12:49.283337+0000","last_clean":"2023-07-29T16:12:49.283337+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:12:49.283337+0000","last_undegraded":"2023-07-29T16:12:49.283337+0000","last_fullsized":"2023-07-29T16:12:49.283337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285673+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:12:49.285673+0000","last_peered":"2023-07-29T16:12:49.285673+0000","last_clean":"2023-07-29T16:12:49.285673+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:12:49.285673+0000","last_undegraded":"2023-07-29T16:12:49.285673+0000","last_fullsized":"2023-07-29T16:12:49.285673+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285737+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:12:49.285737+0000","last_peered":"2023-07-29T16:12:49.285737+0000","last_clean":"2023-07-29T16:12:49.285737+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:12:49.285737+0000","last_undegraded":"2023-07-29T16:12:49.285737+0000","last_fullsized":"2023-07-29T16:12:49.285737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.287846+0000","last_change":"2023-07-29T16:12:50.286750+0000","last_active":"2023-07-29T16:12:50.287846+0000","last_peered":"2023-07-29T16:12:50.287846+0000","last_clean":"2023-07-29T16:12:50.287846+0000","last_became_active":"2023-07-29T16:12:50.286341+0000","last_became_peered":"2023-07-29T16:12:50.286341+0000","last_unstale":"2023-07-29T16:12:50.287846+0000","last_undegraded":"2023-07-29T16:12:50.287846+0000","last_fullsized":"2023-07-29T16:12:50.287846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.547598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277064+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:12:49.277064+0000","last_peered":"2023-07-29T16:12:49.277064+0000","last_clean":"2023-07-29T16:12:49.277064+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:12:49.277064+0000","last_undegraded":"2023-07-29T16:12:49.277064+0000","last_fullsized":"2023-07-29T16:12:49.277064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286583+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:12:49.286583+0000","last_peered":"2023-07-29T16:12:49.286583+0000","last_clean":"2023-07-29T16:12:49.286583+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:12:49.286583+0000","last_undegraded":"2023-07-29T16:12:49.286583+0000","last_fullsized":"2023-07-29T16:12:49.286583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283740+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:12:49.283740+0000","last_peered":"2023-07-29T16:12:49.283740+0000","last_clean":"2023-07-29T16:12:49.283740+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:12:49.283740+0000","last_undegraded":"2023-07-29T16:12:49.283740+0000","last_fullsized":"2023-07-29T16:12:49.283740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.336835+0000","last_change":"2023-07-29T16:12:50.335631+0000","last_active":"2023-07-29T16:12:50.336835+0000","last_peered":"2023-07-29T16:12:50.336835+0000","last_clean":"2023-07-29T16:12:50.336835+0000","last_became_active":"2023-07-29T16:12:50.335213+0000","last_became_peered":"2023-07-29T16:12:50.335213+0000","last_unstale":"2023-07-29T16:12:50.336835+0000","last_undegraded":"2023-07-29T16:12:50.336835+0000","last_fullsized":"2023-07-29T16:12:50.336835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:04:33.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283218+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:12:49.283218+0000","last_peered":"2023-07-29T16:12:49.283218+0000","last_clean":"2023-07-29T16:12:49.283218+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:12:49.283218+0000","last_undegraded":"2023-07-29T16:12:49.283218+0000","last_fullsized":"2023-07-29T16:12:49.283218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.287042+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:12:49.287042+0000","last_peered":"2023-07-29T16:12:49.287042+0000","last_clean":"2023-07-29T16:12:49.287042+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:12:49.287042+0000","last_undegraded":"2023-07-29T16:12:49.287042+0000","last_fullsized":"2023-07-29T16:12:49.287042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284085+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:12:49.284085+0000","last_peered":"2023-07-29T16:12:49.284085+0000","last_clean":"2023-07-29T16:12:49.284085+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:12:49.284085+0000","last_undegraded":"2023-07-29T16:12:49.284085+0000","last_fullsized":"2023-07-29T16:12:49.284085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285721+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:12:49.285721+0000","last_peered":"2023-07-29T16:12:49.285721+0000","last_clean":"2023-07-29T16:12:49.285721+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:12:49.285721+0000","last_undegraded":"2023-07-29T16:12:49.285721+0000","last_fullsized":"2023-07-29T16:12:49.285721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279667+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:12:49.279667+0000","last_peered":"2023-07-29T16:12:49.279667+0000","last_clean":"2023-07-29T16:12:49.279667+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:12:49.279667+0000","last_undegraded":"2023-07-29T16:12:49.279667+0000","last_fullsized":"2023-07-29T16:12:49.279667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285455+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:12:49.285455+0000","last_peered":"2023-07-29T16:12:49.285455+0000","last_clean":"2023-07-29T16:12:49.285455+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:12:49.285455+0000","last_undegraded":"2023-07-29T16:12:49.285455+0000","last_fullsized":"2023-07-29T16:12:49.285455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.291287+0000","last_change":"2023-07-29T16:12:50.291287+0000","last_active":"2023-07-29T16:12:50.291287+0000","last_peered":"2023-07-29T16:12:50.291287+0000","last_clean":"2023-07-29T16:12:50.291287+0000","last_became_active":"2023-07-29T16:12:50.290878+0000","last_became_peered":"2023-07-29T16:12:50.290878+0000","last_unstale":"2023-07-29T16:12:50.291287+0000","last_undegraded":"2023-07-29T16:12:50.291287+0000","last_fullsized":"2023-07-29T16:12:50.291287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:59:20.470299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282596+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:12:49.282596+0000","last_peered":"2023-07-29T16:12:49.282596+0000","last_clean":"2023-07-29T16:12:49.282596+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:12:49.282596+0000","last_undegraded":"2023-07-29T16:12:49.282596+0000","last_fullsized":"2023-07-29T16:12:49.282596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283576+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:12:49.283576+0000","last_peered":"2023-07-29T16:12:49.283576+0000","last_clean":"2023-07-29T16:12:49.283576+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:12:49.283576+0000","last_undegraded":"2023-07-29T16:12:49.283576+0000","last_fullsized":"2023-07-29T16:12:49.283576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.280984+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:12:49.280984+0000","last_peered":"2023-07-29T16:12:49.280984+0000","last_clean":"2023-07-29T16:12:49.280984+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:12:49.280984+0000","last_undegraded":"2023-07-29T16:12:49.280984+0000","last_fullsized":"2023-07-29T16:12:49.280984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283694+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:12:49.283694+0000","last_peered":"2023-07-29T16:12:49.283694+0000","last_clean":"2023-07-29T16:12:49.283694+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:12:49.283694+0000","last_undegraded":"2023-07-29T16:12:49.283694+0000","last_fullsized":"2023-07-29T16:12:49.283694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281332+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:12:49.281332+0000","last_peered":"2023-07-29T16:12:49.281332+0000","last_clean":"2023-07-29T16:12:49.281332+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:12:49.281332+0000","last_undegraded":"2023-07-29T16:12:49.281332+0000","last_fullsized":"2023-07-29T16:12:49.281332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.286738+0000","last_change":"2023-07-29T16:12:50.286738+0000","last_active":"2023-07-29T16:12:50.286738+0000","last_peered":"2023-07-29T16:12:50.286738+0000","last_clean":"2023-07-29T16:12:50.286738+0000","last_became_active":"2023-07-29T16:12:50.286379+0000","last_became_peered":"2023-07-29T16:12:50.286379+0000","last_unstale":"2023-07-29T16:12:50.286738+0000","last_undegraded":"2023-07-29T16:12:50.286738+0000","last_fullsized":"2023-07-29T16:12:50.286738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:50:03.338896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277631+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:12:49.277631+0000","last_peered":"2023-07-29T16:12:49.277631+0000","last_clean":"2023-07-29T16:12:49.277631+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:12:49.277631+0000","last_undegraded":"2023-07-29T16:12:49.277631+0000","last_fullsized":"2023-07-29T16:12:49.277631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286330+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:12:49.286330+0000","last_peered":"2023-07-29T16:12:49.286330+0000","last_clean":"2023-07-29T16:12:49.286330+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:12:49.286330+0000","last_undegraded":"2023-07-29T16:12:49.286330+0000","last_fullsized":"2023-07-29T16:12:49.286330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277495+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:12:49.277495+0000","last_peered":"2023-07-29T16:12:49.277495+0000","last_clean":"2023-07-29T16:12:49.277495+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:12:49.277495+0000","last_undegraded":"2023-07-29T16:12:49.277495+0000","last_fullsized":"2023-07-29T16:12:49.277495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284356+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:12:49.284356+0000","last_peered":"2023-07-29T16:12:49.284356+0000","last_clean":"2023-07-29T16:12:49.284356+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:12:49.284356+0000","last_undegraded":"2023-07-29T16:12:49.284356+0000","last_fullsized":"2023-07-29T16:12:49.284356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.288609+0000","last_change":"2023-07-29T16:12:50.288144+0000","last_active":"2023-07-29T16:12:50.288609+0000","last_peered":"2023-07-29T16:12:50.288609+0000","last_clean":"2023-07-29T16:12:50.288609+0000","last_became_active":"2023-07-29T16:12:50.283877+0000","last_became_peered":"2023-07-29T16:12:50.283877+0000","last_unstale":"2023-07-29T16:12:50.288609+0000","last_undegraded":"2023-07-29T16:12:50.288609+0000","last_fullsized":"2023-07-29T16:12:50.288609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T04:00:21.950054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281436+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:12:49.281436+0000","last_peered":"2023-07-29T16:12:49.281436+0000","last_clean":"2023-07-29T16:12:49.281436+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:12:49.281436+0000","last_undegraded":"2023-07-29T16:12:49.281436+0000","last_fullsized":"2023-07-29T16:12:49.281436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:05.803548+0000","last_change":"2023-07-29T16:12:21.345470+0000","last_active":"2023-07-29T16:13:05.803548+0000","last_peered":"2023-07-29T16:13:05.803548+0000","last_clean":"2023-07-29T16:13:05.803548+0000","last_became_active":"2023-07-29T16:12:21.344787+0000","last_became_peered":"2023-07-29T16:12:21.344787+0000","last_unstale":"2023-07-29T16:13:05.803548+0000","last_undegraded":"2023-07-29T16:13:05.803548+0000","last_fullsized":"2023-07-29T16:13:05.803548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:31:19.451396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":961,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277165+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:12:49.277165+0000","last_peered":"2023-07-29T16:12:49.277165+0000","last_clean":"2023-07-29T16:12:49.277165+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:12:49.277165+0000","last_undegraded":"2023-07-29T16:12:49.277165+0000","last_fullsized":"2023-07-29T16:12:49.277165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286488+0000","last_change":"2023-07-29T16:12:21.346661+0000","last_active":"2023-07-29T16:12:49.286488+0000","last_peered":"2023-07-29T16:12:49.286488+0000","last_clean":"2023-07-29T16:12:49.286488+0000","last_became_active":"2023-07-29T16:12:21.346202+0000","last_became_peered":"2023-07-29T16:12:21.346202+0000","last_unstale":"2023-07-29T16:12:49.286488+0000","last_undegraded":"2023-07-29T16:12:49.286488+0000","last_fullsized":"2023-07-29T16:12:49.286488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:11:26.080842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.280433+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:12:49.280433+0000","last_peered":"2023-07-29T16:12:49.280433+0000","last_clean":"2023-07-29T16:12:49.280433+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:12:49.280433+0000","last_undegraded":"2023-07-29T16:12:49.280433+0000","last_fullsized":"2023-07-29T16:12:49.280433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279332+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:12:49.279332+0000","last_peered":"2023-07-29T16:12:49.279332+0000","last_clean":"2023-07-29T16:12:49.279332+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:12:49.279332+0000","last_undegraded":"2023-07-29T16:12:49.279332+0000","last_fullsized":"2023-07-29T16:12:49.279332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277299+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:12:49.277299+0000","last_peered":"2023-07-29T16:12:49.277299+0000","last_clean":"2023-07-29T16:12:49.277299+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:12:49.277299+0000","last_undegraded":"2023-07-29T16:12:49.277299+0000","last_fullsized":"2023-07-29T16:12:49.277299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281515+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:12:49.281515+0000","last_peered":"2023-07-29T16:12:49.281515+0000","last_clean":"2023-07-29T16:12:49.281515+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:12:49.281515+0000","last_undegraded":"2023-07-29T16:12:49.281515+0000","last_fullsized":"2023-07-29T16:12:49.281515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281239+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:12:49.281239+0000","last_peered":"2023-07-29T16:12:49.281239+0000","last_clean":"2023-07-29T16:12:49.281239+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:12:49.281239+0000","last_undegraded":"2023-07-29T16:12:49.281239+0000","last_fullsized":"2023-07-29T16:12:49.281239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":58,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:05.803082+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:13:05.803082+0000","last_peered":"2023-07-29T16:13:05.803082+0000","last_clean":"2023-07-29T16:13:05.803082+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:13:05.803082+0000","last_undegraded":"2023-07-29T16:13:05.803082+0000","last_fullsized":"2023-07-29T16:13:05.803082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283174+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:12:49.283174+0000","last_peered":"2023-07-29T16:12:49.283174+0000","last_clean":"2023-07-29T16:12:49.283174+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:12:49.283174+0000","last_undegraded":"2023-07-29T16:12:49.283174+0000","last_fullsized":"2023-07-29T16:12:49.283174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284244+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:49.284244+0000","last_peered":"2023-07-29T16:12:49.284244+0000","last_clean":"2023-07-29T16:12:49.284244+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:12:49.284244+0000","last_undegraded":"2023-07-29T16:12:49.284244+0000","last_fullsized":"2023-07-29T16:12:49.284244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.336088+0000","last_change":"2023-07-29T16:12:50.336088+0000","last_active":"2023-07-29T16:12:50.336088+0000","last_peered":"2023-07-29T16:12:50.336088+0000","last_clean":"2023-07-29T16:12:50.336088+0000","last_became_active":"2023-07-29T16:12:50.335715+0000","last_became_peered":"2023-07-29T16:12:50.335715+0000","last_unstale":"2023-07-29T16:12:50.336088+0000","last_undegraded":"2023-07-29T16:12:50.336088+0000","last_fullsized":"2023-07-29T16:12:50.336088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:53:33.332020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281524+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:12:49.281524+0000","last_peered":"2023-07-29T16:12:49.281524+0000","last_clean":"2023-07-29T16:12:49.281524+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:12:49.281524+0000","last_undegraded":"2023-07-29T16:12:49.281524+0000","last_fullsized":"2023-07-29T16:12:49.281524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284467+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:12:49.284467+0000","last_peered":"2023-07-29T16:12:49.284467+0000","last_clean":"2023-07-29T16:12:49.284467+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:12:49.284467+0000","last_undegraded":"2023-07-29T16:12:49.284467+0000","last_fullsized":"2023-07-29T16:12:49.284467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'25","reported_seq":128,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:06.804494+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:13:06.804494+0000","last_peered":"2023-07-29T16:13:06.804494+0000","last_clean":"2023-07-29T16:13:06.804494+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:13:06.804494+0000","last_undegraded":"2023-07-29T16:13:06.804494+0000","last_fullsized":"2023-07-29T16:13:06.804494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56904,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":99,"num_write":26,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277327+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:12:49.277327+0000","last_peered":"2023-07-29T16:12:49.277327+0000","last_clean":"2023-07-29T16:12:49.277327+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:12:49.277327+0000","last_undegraded":"2023-07-29T16:12:49.277327+0000","last_fullsized":"2023-07-29T16:12:49.277327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282012+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:12:49.282012+0000","last_peered":"2023-07-29T16:12:49.282012+0000","last_clean":"2023-07-29T16:12:49.282012+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:12:49.282012+0000","last_undegraded":"2023-07-29T16:12:49.282012+0000","last_fullsized":"2023-07-29T16:12:49.282012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279477+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:12:49.279477+0000","last_peered":"2023-07-29T16:12:49.279477+0000","last_clean":"2023-07-29T16:12:49.279477+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:12:49.279477+0000","last_undegraded":"2023-07-29T16:12:49.279477+0000","last_fullsized":"2023-07-29T16:12:49.279477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281031+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:12:49.281031+0000","last_peered":"2023-07-29T16:12:49.281031+0000","last_clean":"2023-07-29T16:12:49.281031+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:12:49.281031+0000","last_undegraded":"2023-07-29T16:12:49.281031+0000","last_fullsized":"2023-07-29T16:12:49.281031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285126+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:12:49.285126+0000","last_peered":"2023-07-29T16:12:49.285126+0000","last_clean":"2023-07-29T16:12:49.285126+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:12:49.285126+0000","last_undegraded":"2023-07-29T16:12:49.285126+0000","last_fullsized":"2023-07-29T16:12:49.285126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:06.804972+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:13:06.804972+0000","last_peered":"2023-07-29T16:13:06.804972+0000","last_clean":"2023-07-29T16:13:06.804972+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:13:06.804972+0000","last_undegraded":"2023-07-29T16:13:06.804972+0000","last_fullsized":"2023-07-29T16:13:06.804972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":927,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284048+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:12:49.284048+0000","last_peered":"2023-07-29T16:12:49.284048+0000","last_clean":"2023-07-29T16:12:49.284048+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:12:49.284048+0000","last_undegraded":"2023-07-29T16:12:49.284048+0000","last_fullsized":"2023-07-29T16:12:49.284048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277033+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:12:49.277033+0000","last_peered":"2023-07-29T16:12:49.277033+0000","last_clean":"2023-07-29T16:12:49.277033+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:12:49.277033+0000","last_undegraded":"2023-07-29T16:12:49.277033+0000","last_fullsized":"2023-07-29T16:12:49.277033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.286421+0000","last_change":"2023-07-29T16:12:50.286421+0000","last_active":"2023-07-29T16:12:50.286421+0000","last_peered":"2023-07-29T16:12:50.286421+0000","last_clean":"2023-07-29T16:12:50.286421+0000","last_became_active":"2023-07-29T16:12:50.286075+0000","last_became_peered":"2023-07-29T16:12:50.286075+0000","last_unstale":"2023-07-29T16:12:50.286421+0000","last_undegraded":"2023-07-29T16:12:50.286421+0000","last_fullsized":"2023-07-29T16:12:50.286421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:24:16.006080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277521+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:12:49.277521+0000","last_peered":"2023-07-29T16:12:49.277521+0000","last_clean":"2023-07-29T16:12:49.277521+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:12:49.277521+0000","last_undegraded":"2023-07-29T16:12:49.277521+0000","last_fullsized":"2023-07-29T16:12:49.277521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283162+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:12:49.283162+0000","last_peered":"2023-07-29T16:12:49.283162+0000","last_clean":"2023-07-29T16:12:49.283162+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:12:49.283162+0000","last_undegraded":"2023-07-29T16:12:49.283162+0000","last_fullsized":"2023-07-29T16:12:49.283162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281619+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:12:49.281619+0000","last_peered":"2023-07-29T16:12:49.281619+0000","last_clean":"2023-07-29T16:12:49.281619+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:12:49.281619+0000","last_undegraded":"2023-07-29T16:12:49.281619+0000","last_fullsized":"2023-07-29T16:12:49.281619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286195+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:12:49.286195+0000","last_peered":"2023-07-29T16:12:49.286195+0000","last_clean":"2023-07-29T16:12:49.286195+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:12:49.286195+0000","last_undegraded":"2023-07-29T16:12:49.286195+0000","last_fullsized":"2023-07-29T16:12:49.286195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285373+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:12:49.285373+0000","last_peered":"2023-07-29T16:12:49.285373+0000","last_clean":"2023-07-29T16:12:49.285373+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:12:49.285373+0000","last_undegraded":"2023-07-29T16:12:49.285373+0000","last_fullsized":"2023-07-29T16:12:49.285373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286791+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:12:49.286791+0000","last_peered":"2023-07-29T16:12:49.286791+0000","last_clean":"2023-07-29T16:12:49.286791+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:12:49.286791+0000","last_undegraded":"2023-07-29T16:12:49.286791+0000","last_fullsized":"2023-07-29T16:12:49.286791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282175+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:12:49.282175+0000","last_peered":"2023-07-29T16:12:49.282175+0000","last_clean":"2023-07-29T16:12:49.282175+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:12:49.282175+0000","last_undegraded":"2023-07-29T16:12:49.282175+0000","last_fullsized":"2023-07-29T16:12:49.282175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:07.805881+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:13:07.805881+0000","last_peered":"2023-07-29T16:13:07.805881+0000","last_clean":"2023-07-29T16:13:07.805881+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:13:07.805881+0000","last_undegraded":"2023-07-29T16:13:07.805881+0000","last_fullsized":"2023-07-29T16:13:07.805881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277297+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:12:49.277297+0000","last_peered":"2023-07-29T16:12:49.277297+0000","last_clean":"2023-07-29T16:12:49.277297+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:12:49.277297+0000","last_undegraded":"2023-07-29T16:12:49.277297+0000","last_fullsized":"2023-07-29T16:12:49.277297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285943+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:49.285943+0000","last_peered":"2023-07-29T16:12:49.285943+0000","last_clean":"2023-07-29T16:12:49.285943+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:12:49.285943+0000","last_undegraded":"2023-07-29T16:12:49.285943+0000","last_fullsized":"2023-07-29T16:12:49.285943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":330,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283747+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:12:49.283747+0000","last_peered":"2023-07-29T16:12:49.283747+0000","last_clean":"2023-07-29T16:12:49.283747+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:12:49.283747+0000","last_undegraded":"2023-07-29T16:12:49.283747+0000","last_fullsized":"2023-07-29T16:12:49.283747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279220+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:12:49.279220+0000","last_peered":"2023-07-29T16:12:49.279220+0000","last_clean":"2023-07-29T16:12:49.279220+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:12:49.279220+0000","last_undegraded":"2023-07-29T16:12:49.279220+0000","last_fullsized":"2023-07-29T16:12:49.279220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283611+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:12:49.283611+0000","last_peered":"2023-07-29T16:12:49.283611+0000","last_clean":"2023-07-29T16:12:49.283611+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:12:49.283611+0000","last_undegraded":"2023-07-29T16:12:49.283611+0000","last_fullsized":"2023-07-29T16:12:49.283611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277415+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:12:49.277415+0000","last_peered":"2023-07-29T16:12:49.277415+0000","last_clean":"2023-07-29T16:12:49.277415+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:12:49.277415+0000","last_undegraded":"2023-07-29T16:12:49.277415+0000","last_fullsized":"2023-07-29T16:12:49.277415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279734+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:12:49.279734+0000","last_peered":"2023-07-29T16:12:49.279734+0000","last_clean":"2023-07-29T16:12:49.279734+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:12:49.279734+0000","last_undegraded":"2023-07-29T16:12:49.279734+0000","last_fullsized":"2023-07-29T16:12:49.279734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.280957+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:12:49.280957+0000","last_peered":"2023-07-29T16:12:49.280957+0000","last_clean":"2023-07-29T16:12:49.280957+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:12:49.280957+0000","last_undegraded":"2023-07-29T16:12:49.280957+0000","last_fullsized":"2023-07-29T16:12:49.280957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277434+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:12:49.277434+0000","last_peered":"2023-07-29T16:12:49.277434+0000","last_clean":"2023-07-29T16:12:49.277434+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:12:49.277434+0000","last_undegraded":"2023-07-29T16:12:49.277434+0000","last_fullsized":"2023-07-29T16:12:49.277434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286080+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:12:49.286080+0000","last_peered":"2023-07-29T16:12:49.286080+0000","last_clean":"2023-07-29T16:12:49.286080+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:12:49.286080+0000","last_undegraded":"2023-07-29T16:12:49.286080+0000","last_fullsized":"2023-07-29T16:12:49.286080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285774+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:12:49.285774+0000","last_peered":"2023-07-29T16:12:49.285774+0000","last_clean":"2023-07-29T16:12:49.285774+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:12:49.285774+0000","last_undegraded":"2023-07-29T16:12:49.285774+0000","last_fullsized":"2023-07-29T16:12:49.285774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282799+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:12:49.282799+0000","last_peered":"2023-07-29T16:12:49.282799+0000","last_clean":"2023-07-29T16:12:49.282799+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:12:49.282799+0000","last_undegraded":"2023-07-29T16:12:49.282799+0000","last_fullsized":"2023-07-29T16:12:49.282799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277085+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:12:49.277085+0000","last_peered":"2023-07-29T16:12:49.277085+0000","last_clean":"2023-07-29T16:12:49.277085+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:12:49.277085+0000","last_undegraded":"2023-07-29T16:12:49.277085+0000","last_fullsized":"2023-07-29T16:12:49.277085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.276759+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:12:49.276759+0000","last_peered":"2023-07-29T16:12:49.276759+0000","last_clean":"2023-07-29T16:12:49.276759+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:12:49.276759+0000","last_undegraded":"2023-07-29T16:12:49.276759+0000","last_fullsized":"2023-07-29T16:12:49.276759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281902+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:12:49.281902+0000","last_peered":"2023-07-29T16:12:49.281902+0000","last_clean":"2023-07-29T16:12:49.281902+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:12:49.281902+0000","last_undegraded":"2023-07-29T16:12:49.281902+0000","last_fullsized":"2023-07-29T16:12:49.281902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281775+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:12:49.281775+0000","last_peered":"2023-07-29T16:12:49.281775+0000","last_clean":"2023-07-29T16:12:49.281775+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:12:49.281775+0000","last_undegraded":"2023-07-29T16:12:49.281775+0000","last_fullsized":"2023-07-29T16:12:49.281775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277542+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:12:49.277542+0000","last_peered":"2023-07-29T16:12:49.277542+0000","last_clean":"2023-07-29T16:12:49.277542+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:12:49.277542+0000","last_undegraded":"2023-07-29T16:12:49.277542+0000","last_fullsized":"2023-07-29T16:12:49.277542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277168+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:12:49.277168+0000","last_peered":"2023-07-29T16:12:49.277168+0000","last_clean":"2023-07-29T16:12:49.277168+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:12:49.277168+0000","last_undegraded":"2023-07-29T16:12:49.277168+0000","last_fullsized":"2023-07-29T16:12:49.277168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285221+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:12:49.285221+0000","last_peered":"2023-07-29T16:12:49.285221+0000","last_clean":"2023-07-29T16:12:49.285221+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:12:49.285221+0000","last_undegraded":"2023-07-29T16:12:49.285221+0000","last_fullsized":"2023-07-29T16:12:49.285221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.297055+0000","last_change":"2023-07-29T16:12:50.296260+0000","last_active":"2023-07-29T16:12:50.297055+0000","last_peered":"2023-07-29T16:12:50.297055+0000","last_clean":"2023-07-29T16:12:50.297055+0000","last_became_active":"2023-07-29T16:12:50.290108+0000","last_became_peered":"2023-07-29T16:12:50.290108+0000","last_unstale":"2023-07-29T16:12:50.297055+0000","last_undegraded":"2023-07-29T16:12:50.297055+0000","last_fullsized":"2023-07-29T16:12:50.297055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:52:17.142674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284664+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:12:49.284664+0000","last_peered":"2023-07-29T16:12:49.284664+0000","last_clean":"2023-07-29T16:12:49.284664+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:12:49.284664+0000","last_undegraded":"2023-07-29T16:12:49.284664+0000","last_fullsized":"2023-07-29T16:12:49.284664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281333+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:12:49.281333+0000","last_peered":"2023-07-29T16:12:49.281333+0000","last_clean":"2023-07-29T16:12:49.281333+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:12:49.281333+0000","last_undegraded":"2023-07-29T16:12:49.281333+0000","last_fullsized":"2023-07-29T16:12:49.281333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284522+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:12:49.284522+0000","last_peered":"2023-07-29T16:12:49.284522+0000","last_clean":"2023-07-29T16:12:49.284522+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:12:49.284522+0000","last_undegraded":"2023-07-29T16:12:49.284522+0000","last_fullsized":"2023-07-29T16:12:49.284522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279954+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:12:49.279954+0000","last_peered":"2023-07-29T16:12:49.279954+0000","last_clean":"2023-07-29T16:12:49.279954+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:12:49.279954+0000","last_undegraded":"2023-07-29T16:12:49.279954+0000","last_fullsized":"2023-07-29T16:12:49.279954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284407+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:12:49.284407+0000","last_peered":"2023-07-29T16:12:49.284407+0000","last_clean":"2023-07-29T16:12:49.284407+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:12:49.284407+0000","last_undegraded":"2023-07-29T16:12:49.284407+0000","last_fullsized":"2023-07-29T16:12:49.284407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282011+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:12:49.282011+0000","last_peered":"2023-07-29T16:12:49.282011+0000","last_clean":"2023-07-29T16:12:49.282011+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:12:49.282011+0000","last_undegraded":"2023-07-29T16:12:49.282011+0000","last_fullsized":"2023-07-29T16:12:49.282011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281568+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:12:49.281568+0000","last_peered":"2023-07-29T16:12:49.281568+0000","last_clean":"2023-07-29T16:12:49.281568+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:12:49.281568+0000","last_undegraded":"2023-07-29T16:12:49.281568+0000","last_fullsized":"2023-07-29T16:12:49.281568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281677+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:12:49.281677+0000","last_peered":"2023-07-29T16:12:49.281677+0000","last_clean":"2023-07-29T16:12:49.281677+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:12:49.281677+0000","last_undegraded":"2023-07-29T16:12:49.281677+0000","last_fullsized":"2023-07-29T16:12:49.281677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285072+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:12:49.285072+0000","last_peered":"2023-07-29T16:12:49.285072+0000","last_clean":"2023-07-29T16:12:49.285072+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:12:49.285072+0000","last_undegraded":"2023-07-29T16:12:49.285072+0000","last_fullsized":"2023-07-29T16:12:49.285072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279855+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:12:49.279855+0000","last_peered":"2023-07-29T16:12:49.279855+0000","last_clean":"2023-07-29T16:12:49.279855+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:12:49.279855+0000","last_undegraded":"2023-07-29T16:12:49.279855+0000","last_fullsized":"2023-07-29T16:12:49.279855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283348+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:12:49.283348+0000","last_peered":"2023-07-29T16:12:49.283348+0000","last_clean":"2023-07-29T16:12:49.283348+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:12:49.283348+0000","last_undegraded":"2023-07-29T16:12:49.283348+0000","last_fullsized":"2023-07-29T16:12:49.283348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283730+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:12:49.283730+0000","last_peered":"2023-07-29T16:12:49.283730+0000","last_clean":"2023-07-29T16:12:49.283730+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:12:49.283730+0000","last_undegraded":"2023-07-29T16:12:49.283730+0000","last_fullsized":"2023-07-29T16:12:49.283730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277336+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:12:49.277336+0000","last_peered":"2023-07-29T16:12:49.277336+0000","last_clean":"2023-07-29T16:12:49.277336+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:12:49.277336+0000","last_undegraded":"2023-07-29T16:12:49.277336+0000","last_fullsized":"2023-07-29T16:12:49.277336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.286557+0000","last_change":"2023-07-29T16:12:50.285320+0000","last_active":"2023-07-29T16:12:50.286557+0000","last_peered":"2023-07-29T16:12:50.286557+0000","last_clean":"2023-07-29T16:12:50.286557+0000","last_became_active":"2023-07-29T16:12:50.284666+0000","last_became_peered":"2023-07-29T16:12:50.284666+0000","last_unstale":"2023-07-29T16:12:50.286557+0000","last_undegraded":"2023-07-29T16:12:50.286557+0000","last_fullsized":"2023-07-29T16:12:50.286557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:19:09.408935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.276777+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:12:49.276777+0000","last_peered":"2023-07-29T16:12:49.276777+0000","last_clean":"2023-07-29T16:12:49.276777+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:12:49.276777+0000","last_undegraded":"2023-07-29T16:12:49.276777+0000","last_fullsized":"2023-07-29T16:12:49.276777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.337354+0000","last_change":"2023-07-29T16:12:50.335842+0000","last_active":"2023-07-29T16:12:50.337354+0000","last_peered":"2023-07-29T16:12:50.337354+0000","last_clean":"2023-07-29T16:12:50.337354+0000","last_became_active":"2023-07-29T16:12:50.335527+0000","last_became_peered":"2023-07-29T16:12:50.335527+0000","last_unstale":"2023-07-29T16:12:50.337354+0000","last_undegraded":"2023-07-29T16:12:50.337354+0000","last_fullsized":"2023-07-29T16:12:50.337354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:35:18.062454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279716+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:12:49.279716+0000","last_peered":"2023-07-29T16:12:49.279716+0000","last_clean":"2023-07-29T16:12:49.279716+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:12:49.279716+0000","last_undegraded":"2023-07-29T16:12:49.279716+0000","last_fullsized":"2023-07-29T16:12:49.279716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281176+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:12:49.281176+0000","last_peered":"2023-07-29T16:12:49.281176+0000","last_clean":"2023-07-29T16:12:49.281176+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:12:49.281176+0000","last_undegraded":"2023-07-29T16:12:49.281176+0000","last_fullsized":"2023-07-29T16:12:49.281176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61192,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":275,"num_read_kb":202,"num_write":148,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350997,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":459516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48999999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.439}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.621}]},{"osd":5,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.66200000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.751}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":9,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.76700000000000002}]},{"osd":10,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.77900000000000003}]}]},{"osd":10,"up_from":73,"seq":313532612634,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1646592,"data_stored":1107965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.71799999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68999999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.746},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.76300000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.81999999999999995}]},{"osd":9,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.879},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.85999999999999999}]},{"osd":11,"last update":"Sat Jul 29 16:12:39 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.90400000000000003}]}]},{"osd":9,"up_from":68,"seq":292057776158,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":974848,"data_stored":459752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.53400000000000003,"15min":0.54900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.71699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71999999999999997,"5min":1.728,"15min":1.728},"last":1.077}]},{"osd":1,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0029999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.6679999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":1.304}]},{"osd":2,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81399999999999995,"5min":1.125,"15min":1.125},"last":1.6930000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.435,"15min":0.435},"max":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":1.2310000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.80600000000000005,"15min":0.83399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92100000000000004,"5min":1.974,"15min":1.974},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.80200000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.5820000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.91500000000000004,"15min":0.94599999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95899999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.80800000000000005}]},{"osd":5,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94699999999999995,"15min":0.97599999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0600000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.7430000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.92800000000000005,"15min":0.96299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95699999999999996,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.607},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89000000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.2649999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.109,"5min":1.839,"15min":1.839},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.94299999999999995,"15min":0.97099999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0569999999999999,"5min":1.47,"15min":1.47},"last":1.722}]},{"osd":8,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.91700000000000004,"15min":0.94199999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0169999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":1.284},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.004,"15min":1.032},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1220000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.63400000000000001}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.766}]}]},{"osd":8,"up_from":60,"seq":257698037794,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":970752,"data_stored":459341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.32200000000000001,"5min":0.318,"15min":0.318},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.38},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.45600000000000002}]},{"osd":3,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":6.9240000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.63,"15min":0.63200000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.48999999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.751,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.621},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.54200000000000004}]},{"osd":5,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64300000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.502}]},{"osd":6,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.73399999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73299999999999998,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.73399999999999999,"15min":0.748},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.76800000000000002,"5min":1,"15min":1},"last":0.66600000000000004}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.751,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.64100000000000001}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":7,"up_from":52,"seq":223338299431,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1528,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1564672,"data_stored":1051564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.39600000000000002},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.437,"15min":0.43099999999999999},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70999999999999996,"5min":0.755,"15min":0.755},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52700000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.69199999999999995,"15min":0.66400000000000003},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.2300000000000004,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64000000000000001},"min":{"1min":0.53200000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.75600000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.73699999999999999,"15min":0.70399999999999996},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.53200000000000003}]},{"osd":8,"last update":"Sat Jul 29 16:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.91300000000000003,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.70899999999999996,"15min":0.70099999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.70799999999999996}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.62},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77400000000000002,"15min":0.77200000000000002},"min":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.80800000000000005}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.84699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":962560,"data_stored":459147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.502,"15min":0.48499999999999999},"min":{"1min":0.34000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.54600000000000004,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.59999999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.69299999999999995,"15min":0.67100000000000004},"min":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.63800000000000001,"15min":0.60999999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.67200000000000004,"15min":0.65600000000000003},"min":{"1min":0.47999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73999999999999999,"15min":0.71599999999999997},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.0269999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.68200000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.745},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":1.1240000000000001}]},{"osd":8,"last update":"Sat Jul 29 16:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.55100000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0429999999999999,"5min":1.151,"15min":1.151},"last":1.214},{"interface":"front","average":{"1min":0.87,"5min":0.86499999999999999,"15min":0.86399999999999999},"min":{"1min":0.70099999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":1.224}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95199999999999996,"15min":0.94999999999999996},"min":{"1min":0.63800000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.58699999999999997,"5min":0.503,"15min":0.503},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.155}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.256},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.093}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":1019904,"data_stored":515838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55400000000000005,"15min":0.501},"min":{"1min":0.377,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.505},"min":{"1min":0.372,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.73999999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.54400000000000004,"15min":0.52500000000000002},"min":{"1min":0.40600000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.54100000000000004,"15min":0.51100000000000001},"min":{"1min":0.39600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.748}]},{"osd":3,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.47899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58399999999999996,"15min":0.55600000000000005},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81200000000000006,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.62},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.501,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51000000000000001}]},{"osd":5,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.56899999999999995},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.629,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.85099999999999998}]},{"osd":6,"last update":"Sat Jul 29 16:12:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.69099999999999995,"15min":0.67500000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95599999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67700000000000005,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.82499999999999996}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.60499999999999998,"15min":0.58499999999999996},"min":{"1min":0.45000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.65900000000000003,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69799999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.746,"15min":0.747},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76100000000000001,"15min":0.75800000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.78000000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85199999999999998,"15min":0.85699999999999998},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.98799999999999999,"5min":1.161,"15min":1.161},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":1.175,"15min":1.175},"last":0.92600000000000005}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.80600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":983040,"data_stored":460412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56599999999999995,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59199999999999997,"15min":0.54500000000000004},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88300000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86099999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.84999999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.372,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":0.876,"15min":0.876},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.48399999999999999,"15min":0.47199999999999998},"min":{"1min":0.41099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77800000000000002,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.75}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85999999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92200000000000004,"5min":1.111,"15min":1.111},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65900000000000003,"15min":0.64200000000000002},"min":{"1min":0.52900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.82899999999999996,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78900000000000003}]},{"osd":6,"last update":"Sat Jul 29 16:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.70899999999999996,"15min":0.71199999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.099,"5min":1.266,"15min":1.266},"last":0.505},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.747},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.036,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.435}]},{"osd":7,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73399999999999999,"15min":0.72999999999999998},"min":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Jul 29 16:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.86699999999999999,"15min":0.89100000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.90200000000000002,"5min":1.335,"15min":1.335},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.88300000000000001,"15min":0.90300000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.96099999999999997,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.76900000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.86499999999999999,"15min":0.88300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.079,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.94499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.014,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.81699999999999995}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.93100000000000005}]},{"osd":11,"last update":"Sat Jul 29 16:12:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.873}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":1040384,"data_stored":516945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.55800000000000005},"min":{"1min":0.41899999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.95299999999999996,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.59899999999999998,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.613,"5min":3.613,"15min":3.613},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.74299999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56999999999999995,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.753,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.54500000000000004},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78500000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64500000000000002}]},{"osd":4,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.626,"15min":0.59499999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.6019999999999999,"5min":3.6019999999999999,"15min":3.6019999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47699999999999998,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.626}]},{"osd":6,"last update":"Sat Jul 29 16:12:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89200000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.70499999999999996,"15min":0.65300000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.877,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.75700000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.93300000000000005,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64100000000000001,"15min":0.64900000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68000000000000005}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76300000000000001,"15min":0.76500000000000001},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":0.997},"last":0.82599999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":974848,"data_stored":459133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.40300000000000002},"min":{"1min":0.34100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.625,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.438},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Sat Jul 29 16:12:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60599999999999998,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.56799999999999995},"min":{"1min":0.443,"5min":0.372,"15min":0.372},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:12:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.621},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.874,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":1.296,"15min":1.296},"last":0.66200000000000003}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64400000000000002,"15min":0.60699999999999998},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88400000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.53400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98099999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70599999999999996,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.165,"5min":1.343,"15min":1.343},"last":0.624},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71899999999999997,"15min":0.69899999999999995},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:12:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70599999999999996,"15min":0.68400000000000005},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0620000000000001,"5min":1.276,"15min":1.276},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.371,"15min":0.371},"max":{"1min":0.91800000000000004,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.74399999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84199999999999997,"15min":0.82999999999999996},"min":{"1min":0.71999999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.63500000000000001}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.093},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.022}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.90300000000000002,"15min":0.92300000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.091,"5min":1.476,"15min":1.476},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.90100000000000002,"15min":0.91800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0509999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.84999999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":970752,"data_stored":460499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.48799999999999999},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.41499999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.56599999999999995},"min":{"1min":0.50700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.65100000000000002}]},{"osd":3,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60099999999999998,"15min":0.55600000000000005},"min":{"1min":0.45100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93300000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.436},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.46899999999999997}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.48299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63900000000000001,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.621},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68899999999999995,"15min":0.66400000000000003},"min":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.018},"last":0.78700000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.5},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.55900000000000005}]},{"osd":8,"last update":"Sat Jul 29 16:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0109999999999999,"5min":1.026,"15min":1.026},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.748},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0189999999999999,"5min":1.069,"15min":1.069},"last":0.72199999999999998}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79500000000000004,"15min":0.80000000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.04,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.59099999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:12:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.63400000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1581056,"data_stored":1051240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.437}]},{"osd":1,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.77200000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40600000000000003,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29799999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70799999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81299999999999994,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.48699999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.41699999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.754,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.622}]},{"osd":6,"last update":"Sat Jul 29 16:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:12:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.626},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79300000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.78700000000000003}]},{"osd":8,"last update":"Sat Jul 29 16:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60299999999999998,"15min":0.60499999999999998},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52600000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.68400000000000005}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.82399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:13:12.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph pg dump --format=json 2023-07-29T16:13:12.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:12 smithi039 ceph-mon[128858]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:12.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:12 smithi156 ceph-mon[129245]: pgmap v299: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:13.263 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:13.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:13 smithi031 ceph-mon[95504]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:13:13.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:13 smithi039 ceph-mon[128858]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:13:13.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:13 smithi156 ceph-mon[129245]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:13:14.474 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:14 smithi031 ceph-mon[95504]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:14.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 -- 172.21.15.31:0/2752936122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100e20 msgr2=0x7fa88c104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 --2- 172.21.15.31:0/2752936122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100e20 0x7fa88c104df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0099c0 tx=0x7fa87c009cd0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 -- 172.21.15.31:0/2752936122 shutdown_connections 2023-07-29T16:13:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 --2- 172.21.15.31:0/2752936122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa88c105520 0x7fa88c10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 --2- 172.21.15.31:0/2752936122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c100480 0x7fa88c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 --2- 172.21.15.31:0/2752936122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100e20 0x7fa88c104df0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 -- 172.21.15.31:0/2752936122 >> 172.21.15.31:0/2752936122 conn(0x7fa88c0fbda0 msgr2=0x7fa88c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:14.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 -- 172.21.15.31:0/2752936122 shutdown_connections 2023-07-29T16:13:14.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.729+0000 7fa8930f7700 1 -- 172.21.15.31:0/2752936122 wait complete. 2023-07-29T16:13:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.730+0000 7fa8930f7700 1 Processor -- start 2023-07-29T16:13:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 -- start start 2023-07-29T16:13:14.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 0x7fa88c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa88c100e20 0x7fa88c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c105520 0x7fa88c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c19b3a0 con 0x7fa88c100480 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c19b510 con 0x7fa88c105520 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8930f7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c19b6d0 con 0x7fa88c100e20 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8920f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 0x7fa88c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:14.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8920f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 0x7fa88c1963e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44906/0 (socket says 172.21.15.31:44906) 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8920f5700 1 -- 172.21.15.31:0/336978941 learned_addr learned my addr 172.21.15.31:0/336978941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8918f4700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa88c100e20 0x7fa88c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.731+0000 7fa8928f6700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c105520 0x7fa88c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8920f5700 1 -- 172.21.15.31:0/336978941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa88c100e20 msgr2=0x7fa88c196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8920f5700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa88c100e20 0x7fa88c196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8920f5700 1 -- 172.21.15.31:0/336978941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c105520 msgr2=0x7fa88c19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:14.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8920f5700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c105520 0x7fa88c19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8920f5700 1 -- 172.21.15.31:0/336978941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c0095f0 con 0x7fa88c100480 2023-07-29T16:13:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8928f6700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c105520 0x7fa88c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8920f5700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 0x7fa88c1963e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa88800d980 tx=0x7fa88800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:14.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888004220 con 0x7fa88c100480 2023-07-29T16:13:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c19b9b0 con 0x7fa88c100480 2023-07-29T16:13:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa888007640 con 0x7fa88c100480 2023-07-29T16:13:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.732+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888005610 con 0x7fa88c100480 2023-07-29T16:13:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.733+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88c1a3210 con 0x7fa88c100480 2023-07-29T16:13:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.734+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88c04f2a0 con 0x7fa88c100480 2023-07-29T16:13:14.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.736+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fa888004750 con 0x7fa88c100480 2023-07-29T16:13:14.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.737+0000 7fa8837fe700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8780a0820 0x7fa8780a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:14.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.737+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fa8880cbae0 con 0x7fa88c100480 2023-07-29T16:13:14.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.737+0000 7fa8918f4700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8780a0820 0x7fa8780a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:14.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.738+0000 7fa8918f4700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8780a0820 0x7fa8780a2cd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0098e0 tx=0x7fa87c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:14.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.741+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa8880953a0 con 0x7fa88c100480 2023-07-29T16:13:14.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:14 smithi039 ceph-mon[128858]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:14.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:14 smithi156 ceph-mon[129245]: pgmap v300: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:14.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.940+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa88c0fcfc0 con 0x7fa8780a0820 2023-07-29T16:13:14.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.951+0000 7fa8837fe700 1 -- 172.21.15.31:0/336978941 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+339636 (secure 0 0 0) 0x7fa88c0fcfc0 con 0x7fa8780a0820 2023-07-29T16:13:14.954 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:14.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.955+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8780a0820 msgr2=0x7fa8780a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:14.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.955+0000 7fa8930f7700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8780a0820 0x7fa8780a2cd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0098e0 tx=0x7fa87c000bc0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.955+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 msgr2=0x7fa88c1963e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:14.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.955+0000 7fa8930f7700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 0x7fa88c1963e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa88800d980 tx=0x7fa88800dc90 comp rx=0 tx=0).stop 2023-07-29T16:13:14.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 shutdown_connections 2023-07-29T16:13:14.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8780a0820 0x7fa8780a2cd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa88c105520 0x7fa88c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa88c100480 0x7fa88c1963e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 --2- 172.21.15.31:0/336978941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa88c100e20 0x7fa88c196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:14.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 >> 172.21.15.31:0/336978941 conn(0x7fa88c0fbda0 msgr2=0x7fa88c0fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:14.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 shutdown_connections 2023-07-29T16:13:14.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:14.956+0000 7fa8930f7700 1 -- 172.21.15.31:0/336978941 wait complete. 2023-07-29T16:13:14.957 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:13:15.345 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2023-07-29T16:13:13.283274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651806,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":426,"num_read_kb":321,"num_write":331,"num_write_kb":3041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1181318,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":269,"ondisk_log_size":269,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336752,"kb_used_data":13328,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124605072,"statfs":{"total":1151940427776,"available":1151595593728,"internally_reserved":0,"allocated":13647872,"data_stored":7461352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002952"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283337+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:12:49.283337+0000","last_peered":"2023-07-29T16:12:49.283337+0000","last_clean":"2023-07-29T16:12:49.283337+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:12:49.283337+0000","last_undegraded":"2023-07-29T16:12:49.283337+0000","last_fullsized":"2023-07-29T16:12:49.283337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285673+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:12:49.285673+0000","last_peered":"2023-07-29T16:12:49.285673+0000","last_clean":"2023-07-29T16:12:49.285673+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:12:49.285673+0000","last_undegraded":"2023-07-29T16:12:49.285673+0000","last_fullsized":"2023-07-29T16:12:49.285673+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285737+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:12:49.285737+0000","last_peered":"2023-07-29T16:12:49.285737+0000","last_clean":"2023-07-29T16:12:49.285737+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:12:49.285737+0000","last_undegraded":"2023-07-29T16:12:49.285737+0000","last_fullsized":"2023-07-29T16:12:49.285737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.287846+0000","last_change":"2023-07-29T16:12:50.286750+0000","last_active":"2023-07-29T16:12:50.287846+0000","last_peered":"2023-07-29T16:12:50.287846+0000","last_clean":"2023-07-29T16:12:50.287846+0000","last_became_active":"2023-07-29T16:12:50.286341+0000","last_became_peered":"2023-07-29T16:12:50.286341+0000","last_unstale":"2023-07-29T16:12:50.287846+0000","last_undegraded":"2023-07-29T16:12:50.287846+0000","last_fullsized":"2023-07-29T16:12:50.287846+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.547598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277064+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:12:49.277064+0000","last_peered":"2023-07-29T16:12:49.277064+0000","last_clean":"2023-07-29T16:12:49.277064+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:12:49.277064+0000","last_undegraded":"2023-07-29T16:12:49.277064+0000","last_fullsized":"2023-07-29T16:12:49.277064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286583+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:12:49.286583+0000","last_peered":"2023-07-29T16:12:49.286583+0000","last_clean":"2023-07-29T16:12:49.286583+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:12:49.286583+0000","last_undegraded":"2023-07-29T16:12:49.286583+0000","last_fullsized":"2023-07-29T16:12:49.286583+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283740+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:12:49.283740+0000","last_peered":"2023-07-29T16:12:49.283740+0000","last_clean":"2023-07-29T16:12:49.283740+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:12:49.283740+0000","last_undegraded":"2023-07-29T16:12:49.283740+0000","last_fullsized":"2023-07-29T16:12:49.283740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.336835+0000","last_change":"2023-07-29T16:12:50.335631+0000","last_active":"2023-07-29T16:12:50.336835+0000","last_peered":"2023-07-29T16:12:50.336835+0000","last_clean":"2023-07-29T16:12:50.336835+0000","last_became_active":"2023-07-29T16:12:50.335213+0000","last_became_peered":"2023-07-29T16:12:50.335213+0000","last_unstale":"2023-07-29T16:12:50.336835+0000","last_undegraded":"2023-07-29T16:12:50.336835+0000","last_fullsized":"2023-07-29T16:12:50.336835+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:04:33.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283218+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:12:49.283218+0000","last_peered":"2023-07-29T16:12:49.283218+0000","last_clean":"2023-07-29T16:12:49.283218+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:12:49.283218+0000","last_undegraded":"2023-07-29T16:12:49.283218+0000","last_fullsized":"2023-07-29T16:12:49.283218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.287042+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:12:49.287042+0000","last_peered":"2023-07-29T16:12:49.287042+0000","last_clean":"2023-07-29T16:12:49.287042+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:12:49.287042+0000","last_undegraded":"2023-07-29T16:12:49.287042+0000","last_fullsized":"2023-07-29T16:12:49.287042+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284085+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:12:49.284085+0000","last_peered":"2023-07-29T16:12:49.284085+0000","last_clean":"2023-07-29T16:12:49.284085+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:12:49.284085+0000","last_undegraded":"2023-07-29T16:12:49.284085+0000","last_fullsized":"2023-07-29T16:12:49.284085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285721+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:12:49.285721+0000","last_peered":"2023-07-29T16:12:49.285721+0000","last_clean":"2023-07-29T16:12:49.285721+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:12:49.285721+0000","last_undegraded":"2023-07-29T16:12:49.285721+0000","last_fullsized":"2023-07-29T16:12:49.285721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279667+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:12:49.279667+0000","last_peered":"2023-07-29T16:12:49.279667+0000","last_clean":"2023-07-29T16:12:49.279667+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:12:49.279667+0000","last_undegraded":"2023-07-29T16:12:49.279667+0000","last_fullsized":"2023-07-29T16:12:49.279667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285455+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:12:49.285455+0000","last_peered":"2023-07-29T16:12:49.285455+0000","last_clean":"2023-07-29T16:12:49.285455+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:12:49.285455+0000","last_undegraded":"2023-07-29T16:12:49.285455+0000","last_fullsized":"2023-07-29T16:12:49.285455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.291287+0000","last_change":"2023-07-29T16:12:50.291287+0000","last_active":"2023-07-29T16:12:50.291287+0000","last_peered":"2023-07-29T16:12:50.291287+0000","last_clean":"2023-07-29T16:12:50.291287+0000","last_became_active":"2023-07-29T16:12:50.290878+0000","last_became_peered":"2023-07-29T16:12:50.290878+0000","last_unstale":"2023-07-29T16:12:50.291287+0000","last_undegraded":"2023-07-29T16:12:50.291287+0000","last_fullsized":"2023-07-29T16:12:50.291287+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:59:20.470299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282596+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:12:49.282596+0000","last_peered":"2023-07-29T16:12:49.282596+0000","last_clean":"2023-07-29T16:12:49.282596+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:12:49.282596+0000","last_undegraded":"2023-07-29T16:12:49.282596+0000","last_fullsized":"2023-07-29T16:12:49.282596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283576+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:12:49.283576+0000","last_peered":"2023-07-29T16:12:49.283576+0000","last_clean":"2023-07-29T16:12:49.283576+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:12:49.283576+0000","last_undegraded":"2023-07-29T16:12:49.283576+0000","last_fullsized":"2023-07-29T16:12:49.283576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.280984+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:12:49.280984+0000","last_peered":"2023-07-29T16:12:49.280984+0000","last_clean":"2023-07-29T16:12:49.280984+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:12:49.280984+0000","last_undegraded":"2023-07-29T16:12:49.280984+0000","last_fullsized":"2023-07-29T16:12:49.280984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283694+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:12:49.283694+0000","last_peered":"2023-07-29T16:12:49.283694+0000","last_clean":"2023-07-29T16:12:49.283694+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:12:49.283694+0000","last_undegraded":"2023-07-29T16:12:49.283694+0000","last_fullsized":"2023-07-29T16:12:49.283694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281332+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:12:49.281332+0000","last_peered":"2023-07-29T16:12:49.281332+0000","last_clean":"2023-07-29T16:12:49.281332+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:12:49.281332+0000","last_undegraded":"2023-07-29T16:12:49.281332+0000","last_fullsized":"2023-07-29T16:12:49.281332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.286738+0000","last_change":"2023-07-29T16:12:50.286738+0000","last_active":"2023-07-29T16:12:50.286738+0000","last_peered":"2023-07-29T16:12:50.286738+0000","last_clean":"2023-07-29T16:12:50.286738+0000","last_became_active":"2023-07-29T16:12:50.286379+0000","last_became_peered":"2023-07-29T16:12:50.286379+0000","last_unstale":"2023-07-29T16:12:50.286738+0000","last_undegraded":"2023-07-29T16:12:50.286738+0000","last_fullsized":"2023-07-29T16:12:50.286738+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:50:03.338896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277631+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:12:49.277631+0000","last_peered":"2023-07-29T16:12:49.277631+0000","last_clean":"2023-07-29T16:12:49.277631+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:12:49.277631+0000","last_undegraded":"2023-07-29T16:12:49.277631+0000","last_fullsized":"2023-07-29T16:12:49.277631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286330+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:12:49.286330+0000","last_peered":"2023-07-29T16:12:49.286330+0000","last_clean":"2023-07-29T16:12:49.286330+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:12:49.286330+0000","last_undegraded":"2023-07-29T16:12:49.286330+0000","last_fullsized":"2023-07-29T16:12:49.286330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277495+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:12:49.277495+0000","last_peered":"2023-07-29T16:12:49.277495+0000","last_clean":"2023-07-29T16:12:49.277495+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:12:49.277495+0000","last_undegraded":"2023-07-29T16:12:49.277495+0000","last_fullsized":"2023-07-29T16:12:49.277495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284356+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:12:49.284356+0000","last_peered":"2023-07-29T16:12:49.284356+0000","last_clean":"2023-07-29T16:12:49.284356+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:12:49.284356+0000","last_undegraded":"2023-07-29T16:12:49.284356+0000","last_fullsized":"2023-07-29T16:12:49.284356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.288609+0000","last_change":"2023-07-29T16:12:50.288144+0000","last_active":"2023-07-29T16:12:50.288609+0000","last_peered":"2023-07-29T16:12:50.288609+0000","last_clean":"2023-07-29T16:12:50.288609+0000","last_became_active":"2023-07-29T16:12:50.283877+0000","last_became_peered":"2023-07-29T16:12:50.283877+0000","last_unstale":"2023-07-29T16:12:50.288609+0000","last_undegraded":"2023-07-29T16:12:50.288609+0000","last_fullsized":"2023-07-29T16:12:50.288609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T04:00:21.950054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281436+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:12:49.281436+0000","last_peered":"2023-07-29T16:12:49.281436+0000","last_clean":"2023-07-29T16:12:49.281436+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:12:49.281436+0000","last_undegraded":"2023-07-29T16:12:49.281436+0000","last_fullsized":"2023-07-29T16:12:49.281436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:10.811242+0000","last_change":"2023-07-29T16:12:21.345470+0000","last_active":"2023-07-29T16:13:10.811242+0000","last_peered":"2023-07-29T16:13:10.811242+0000","last_clean":"2023-07-29T16:13:10.811242+0000","last_became_active":"2023-07-29T16:12:21.344787+0000","last_became_peered":"2023-07-29T16:12:21.344787+0000","last_unstale":"2023-07-29T16:13:10.811242+0000","last_undegraded":"2023-07-29T16:13:10.811242+0000","last_fullsized":"2023-07-29T16:13:10.811242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:31:19.451396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":961,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277165+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:12:49.277165+0000","last_peered":"2023-07-29T16:12:49.277165+0000","last_clean":"2023-07-29T16:12:49.277165+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:12:49.277165+0000","last_undegraded":"2023-07-29T16:12:49.277165+0000","last_fullsized":"2023-07-29T16:12:49.277165+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286488+0000","last_change":"2023-07-29T16:12:21.346661+0000","last_active":"2023-07-29T16:12:49.286488+0000","last_peered":"2023-07-29T16:12:49.286488+0000","last_clean":"2023-07-29T16:12:49.286488+0000","last_became_active":"2023-07-29T16:12:21.346202+0000","last_became_peered":"2023-07-29T16:12:21.346202+0000","last_unstale":"2023-07-29T16:12:49.286488+0000","last_undegraded":"2023-07-29T16:12:49.286488+0000","last_fullsized":"2023-07-29T16:12:49.286488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:11:26.080842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.280433+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:12:49.280433+0000","last_peered":"2023-07-29T16:12:49.280433+0000","last_clean":"2023-07-29T16:12:49.280433+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:12:49.280433+0000","last_undegraded":"2023-07-29T16:12:49.280433+0000","last_fullsized":"2023-07-29T16:12:49.280433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279332+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:12:49.279332+0000","last_peered":"2023-07-29T16:12:49.279332+0000","last_clean":"2023-07-29T16:12:49.279332+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:12:49.279332+0000","last_undegraded":"2023-07-29T16:12:49.279332+0000","last_fullsized":"2023-07-29T16:12:49.279332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277299+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:12:49.277299+0000","last_peered":"2023-07-29T16:12:49.277299+0000","last_clean":"2023-07-29T16:12:49.277299+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:12:49.277299+0000","last_undegraded":"2023-07-29T16:12:49.277299+0000","last_fullsized":"2023-07-29T16:12:49.277299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281515+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:12:49.281515+0000","last_peered":"2023-07-29T16:12:49.281515+0000","last_clean":"2023-07-29T16:12:49.281515+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:12:49.281515+0000","last_undegraded":"2023-07-29T16:12:49.281515+0000","last_fullsized":"2023-07-29T16:12:49.281515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281239+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:12:49.281239+0000","last_peered":"2023-07-29T16:12:49.281239+0000","last_clean":"2023-07-29T16:12:49.281239+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:12:49.281239+0000","last_undegraded":"2023-07-29T16:12:49.281239+0000","last_fullsized":"2023-07-29T16:12:49.281239+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:10.810713+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:13:10.810713+0000","last_peered":"2023-07-29T16:13:10.810713+0000","last_clean":"2023-07-29T16:13:10.810713+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:13:10.810713+0000","last_undegraded":"2023-07-29T16:13:10.810713+0000","last_fullsized":"2023-07-29T16:13:10.810713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283174+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:12:49.283174+0000","last_peered":"2023-07-29T16:12:49.283174+0000","last_clean":"2023-07-29T16:12:49.283174+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:12:49.283174+0000","last_undegraded":"2023-07-29T16:12:49.283174+0000","last_fullsized":"2023-07-29T16:12:49.283174+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284244+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:49.284244+0000","last_peered":"2023-07-29T16:12:49.284244+0000","last_clean":"2023-07-29T16:12:49.284244+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:12:49.284244+0000","last_undegraded":"2023-07-29T16:12:49.284244+0000","last_fullsized":"2023-07-29T16:12:49.284244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.336088+0000","last_change":"2023-07-29T16:12:50.336088+0000","last_active":"2023-07-29T16:12:50.336088+0000","last_peered":"2023-07-29T16:12:50.336088+0000","last_clean":"2023-07-29T16:12:50.336088+0000","last_became_active":"2023-07-29T16:12:50.335715+0000","last_became_peered":"2023-07-29T16:12:50.335715+0000","last_unstale":"2023-07-29T16:12:50.336088+0000","last_undegraded":"2023-07-29T16:12:50.336088+0000","last_fullsized":"2023-07-29T16:12:50.336088+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:53:33.332020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281524+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:12:49.281524+0000","last_peered":"2023-07-29T16:12:49.281524+0000","last_clean":"2023-07-29T16:12:49.281524+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:12:49.281524+0000","last_undegraded":"2023-07-29T16:12:49.281524+0000","last_fullsized":"2023-07-29T16:12:49.281524+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284467+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:12:49.284467+0000","last_peered":"2023-07-29T16:12:49.284467+0000","last_clean":"2023-07-29T16:12:49.284467+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:12:49.284467+0000","last_undegraded":"2023-07-29T16:12:49.284467+0000","last_fullsized":"2023-07-29T16:12:49.284467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'25","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:11.812318+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:13:11.812318+0000","last_peered":"2023-07-29T16:13:11.812318+0000","last_clean":"2023-07-29T16:13:11.812318+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:13:11.812318+0000","last_undegraded":"2023-07-29T16:13:11.812318+0000","last_fullsized":"2023-07-29T16:13:11.812318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56904,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":104,"num_write":26,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277327+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:12:49.277327+0000","last_peered":"2023-07-29T16:12:49.277327+0000","last_clean":"2023-07-29T16:12:49.277327+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:12:49.277327+0000","last_undegraded":"2023-07-29T16:12:49.277327+0000","last_fullsized":"2023-07-29T16:12:49.277327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282012+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:12:49.282012+0000","last_peered":"2023-07-29T16:12:49.282012+0000","last_clean":"2023-07-29T16:12:49.282012+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:12:49.282012+0000","last_undegraded":"2023-07-29T16:12:49.282012+0000","last_fullsized":"2023-07-29T16:12:49.282012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279477+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:12:49.279477+0000","last_peered":"2023-07-29T16:12:49.279477+0000","last_clean":"2023-07-29T16:12:49.279477+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:12:49.279477+0000","last_undegraded":"2023-07-29T16:12:49.279477+0000","last_fullsized":"2023-07-29T16:12:49.279477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281031+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:12:49.281031+0000","last_peered":"2023-07-29T16:12:49.281031+0000","last_clean":"2023-07-29T16:12:49.281031+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:12:49.281031+0000","last_undegraded":"2023-07-29T16:12:49.281031+0000","last_fullsized":"2023-07-29T16:12:49.281031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285126+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:12:49.285126+0000","last_peered":"2023-07-29T16:12:49.285126+0000","last_clean":"2023-07-29T16:12:49.285126+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:12:49.285126+0000","last_undegraded":"2023-07-29T16:12:49.285126+0000","last_fullsized":"2023-07-29T16:12:49.285126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":68,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:11.812330+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:13:11.812330+0000","last_peered":"2023-07-29T16:13:11.812330+0000","last_clean":"2023-07-29T16:13:11.812330+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:13:11.812330+0000","last_undegraded":"2023-07-29T16:13:11.812330+0000","last_fullsized":"2023-07-29T16:13:11.812330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":927,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284048+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:12:49.284048+0000","last_peered":"2023-07-29T16:12:49.284048+0000","last_clean":"2023-07-29T16:12:49.284048+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:12:49.284048+0000","last_undegraded":"2023-07-29T16:12:49.284048+0000","last_fullsized":"2023-07-29T16:12:49.284048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277033+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:12:49.277033+0000","last_peered":"2023-07-29T16:12:49.277033+0000","last_clean":"2023-07-29T16:12:49.277033+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:12:49.277033+0000","last_undegraded":"2023-07-29T16:12:49.277033+0000","last_fullsized":"2023-07-29T16:12:49.277033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.286421+0000","last_change":"2023-07-29T16:12:50.286421+0000","last_active":"2023-07-29T16:12:50.286421+0000","last_peered":"2023-07-29T16:12:50.286421+0000","last_clean":"2023-07-29T16:12:50.286421+0000","last_became_active":"2023-07-29T16:12:50.286075+0000","last_became_peered":"2023-07-29T16:12:50.286075+0000","last_unstale":"2023-07-29T16:12:50.286421+0000","last_undegraded":"2023-07-29T16:12:50.286421+0000","last_fullsized":"2023-07-29T16:12:50.286421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:24:16.006080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277521+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:12:49.277521+0000","last_peered":"2023-07-29T16:12:49.277521+0000","last_clean":"2023-07-29T16:12:49.277521+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:12:49.277521+0000","last_undegraded":"2023-07-29T16:12:49.277521+0000","last_fullsized":"2023-07-29T16:12:49.277521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283162+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:12:49.283162+0000","last_peered":"2023-07-29T16:12:49.283162+0000","last_clean":"2023-07-29T16:12:49.283162+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:12:49.283162+0000","last_undegraded":"2023-07-29T16:12:49.283162+0000","last_fullsized":"2023-07-29T16:12:49.283162+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281619+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:12:49.281619+0000","last_peered":"2023-07-29T16:12:49.281619+0000","last_clean":"2023-07-29T16:12:49.281619+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:12:49.281619+0000","last_undegraded":"2023-07-29T16:12:49.281619+0000","last_fullsized":"2023-07-29T16:12:49.281619+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286195+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:12:49.286195+0000","last_peered":"2023-07-29T16:12:49.286195+0000","last_clean":"2023-07-29T16:12:49.286195+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:12:49.286195+0000","last_undegraded":"2023-07-29T16:12:49.286195+0000","last_fullsized":"2023-07-29T16:12:49.286195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285373+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:12:49.285373+0000","last_peered":"2023-07-29T16:12:49.285373+0000","last_clean":"2023-07-29T16:12:49.285373+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:12:49.285373+0000","last_undegraded":"2023-07-29T16:12:49.285373+0000","last_fullsized":"2023-07-29T16:12:49.285373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286791+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:12:49.286791+0000","last_peered":"2023-07-29T16:12:49.286791+0000","last_clean":"2023-07-29T16:12:49.286791+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:12:49.286791+0000","last_undegraded":"2023-07-29T16:12:49.286791+0000","last_fullsized":"2023-07-29T16:12:49.286791+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282175+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:12:49.282175+0000","last_peered":"2023-07-29T16:12:49.282175+0000","last_clean":"2023-07-29T16:12:49.282175+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:12:49.282175+0000","last_undegraded":"2023-07-29T16:12:49.282175+0000","last_fullsized":"2023-07-29T16:12:49.282175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:13:07.805881+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:13:07.805881+0000","last_peered":"2023-07-29T16:13:07.805881+0000","last_clean":"2023-07-29T16:13:07.805881+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:13:07.805881+0000","last_undegraded":"2023-07-29T16:13:07.805881+0000","last_fullsized":"2023-07-29T16:13:07.805881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277297+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:12:49.277297+0000","last_peered":"2023-07-29T16:12:49.277297+0000","last_clean":"2023-07-29T16:12:49.277297+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:12:49.277297+0000","last_undegraded":"2023-07-29T16:12:49.277297+0000","last_fullsized":"2023-07-29T16:12:49.277297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285943+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:12:49.285943+0000","last_peered":"2023-07-29T16:12:49.285943+0000","last_clean":"2023-07-29T16:12:49.285943+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:12:49.285943+0000","last_undegraded":"2023-07-29T16:12:49.285943+0000","last_fullsized":"2023-07-29T16:12:49.285943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":330,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283747+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:12:49.283747+0000","last_peered":"2023-07-29T16:12:49.283747+0000","last_clean":"2023-07-29T16:12:49.283747+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:12:49.283747+0000","last_undegraded":"2023-07-29T16:12:49.283747+0000","last_fullsized":"2023-07-29T16:12:49.283747+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279220+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:12:49.279220+0000","last_peered":"2023-07-29T16:12:49.279220+0000","last_clean":"2023-07-29T16:12:49.279220+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:12:49.279220+0000","last_undegraded":"2023-07-29T16:12:49.279220+0000","last_fullsized":"2023-07-29T16:12:49.279220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283611+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:12:49.283611+0000","last_peered":"2023-07-29T16:12:49.283611+0000","last_clean":"2023-07-29T16:12:49.283611+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:12:49.283611+0000","last_undegraded":"2023-07-29T16:12:49.283611+0000","last_fullsized":"2023-07-29T16:12:49.283611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277415+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:12:49.277415+0000","last_peered":"2023-07-29T16:12:49.277415+0000","last_clean":"2023-07-29T16:12:49.277415+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:12:49.277415+0000","last_undegraded":"2023-07-29T16:12:49.277415+0000","last_fullsized":"2023-07-29T16:12:49.277415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279734+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:12:49.279734+0000","last_peered":"2023-07-29T16:12:49.279734+0000","last_clean":"2023-07-29T16:12:49.279734+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:12:49.279734+0000","last_undegraded":"2023-07-29T16:12:49.279734+0000","last_fullsized":"2023-07-29T16:12:49.279734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'5","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.280957+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:12:49.280957+0000","last_peered":"2023-07-29T16:12:49.280957+0000","last_clean":"2023-07-29T16:12:49.280957+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:12:49.280957+0000","last_undegraded":"2023-07-29T16:12:49.280957+0000","last_fullsized":"2023-07-29T16:12:49.280957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277434+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:12:49.277434+0000","last_peered":"2023-07-29T16:12:49.277434+0000","last_clean":"2023-07-29T16:12:49.277434+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:12:49.277434+0000","last_undegraded":"2023-07-29T16:12:49.277434+0000","last_fullsized":"2023-07-29T16:12:49.277434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.286080+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:12:49.286080+0000","last_peered":"2023-07-29T16:12:49.286080+0000","last_clean":"2023-07-29T16:12:49.286080+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:12:49.286080+0000","last_undegraded":"2023-07-29T16:12:49.286080+0000","last_fullsized":"2023-07-29T16:12:49.286080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285774+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:12:49.285774+0000","last_peered":"2023-07-29T16:12:49.285774+0000","last_clean":"2023-07-29T16:12:49.285774+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:12:49.285774+0000","last_undegraded":"2023-07-29T16:12:49.285774+0000","last_fullsized":"2023-07-29T16:12:49.285774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282799+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:12:49.282799+0000","last_peered":"2023-07-29T16:12:49.282799+0000","last_clean":"2023-07-29T16:12:49.282799+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:12:49.282799+0000","last_undegraded":"2023-07-29T16:12:49.282799+0000","last_fullsized":"2023-07-29T16:12:49.282799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277085+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:12:49.277085+0000","last_peered":"2023-07-29T16:12:49.277085+0000","last_clean":"2023-07-29T16:12:49.277085+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:12:49.277085+0000","last_undegraded":"2023-07-29T16:12:49.277085+0000","last_fullsized":"2023-07-29T16:12:49.277085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.276759+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:12:49.276759+0000","last_peered":"2023-07-29T16:12:49.276759+0000","last_clean":"2023-07-29T16:12:49.276759+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:12:49.276759+0000","last_undegraded":"2023-07-29T16:12:49.276759+0000","last_fullsized":"2023-07-29T16:12:49.276759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281902+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:12:49.281902+0000","last_peered":"2023-07-29T16:12:49.281902+0000","last_clean":"2023-07-29T16:12:49.281902+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:12:49.281902+0000","last_undegraded":"2023-07-29T16:12:49.281902+0000","last_fullsized":"2023-07-29T16:12:49.281902+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281775+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:12:49.281775+0000","last_peered":"2023-07-29T16:12:49.281775+0000","last_clean":"2023-07-29T16:12:49.281775+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:12:49.281775+0000","last_undegraded":"2023-07-29T16:12:49.281775+0000","last_fullsized":"2023-07-29T16:12:49.281775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277542+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:12:49.277542+0000","last_peered":"2023-07-29T16:12:49.277542+0000","last_clean":"2023-07-29T16:12:49.277542+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:12:49.277542+0000","last_undegraded":"2023-07-29T16:12:49.277542+0000","last_fullsized":"2023-07-29T16:12:49.277542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277168+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:12:49.277168+0000","last_peered":"2023-07-29T16:12:49.277168+0000","last_clean":"2023-07-29T16:12:49.277168+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:12:49.277168+0000","last_undegraded":"2023-07-29T16:12:49.277168+0000","last_fullsized":"2023-07-29T16:12:49.277168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285221+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:12:49.285221+0000","last_peered":"2023-07-29T16:12:49.285221+0000","last_clean":"2023-07-29T16:12:49.285221+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:12:49.285221+0000","last_undegraded":"2023-07-29T16:12:49.285221+0000","last_fullsized":"2023-07-29T16:12:49.285221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.297055+0000","last_change":"2023-07-29T16:12:50.296260+0000","last_active":"2023-07-29T16:12:50.297055+0000","last_peered":"2023-07-29T16:12:50.297055+0000","last_clean":"2023-07-29T16:12:50.297055+0000","last_became_active":"2023-07-29T16:12:50.290108+0000","last_became_peered":"2023-07-29T16:12:50.290108+0000","last_unstale":"2023-07-29T16:12:50.297055+0000","last_undegraded":"2023-07-29T16:12:50.297055+0000","last_fullsized":"2023-07-29T16:12:50.297055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:52:17.142674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284664+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:12:49.284664+0000","last_peered":"2023-07-29T16:12:49.284664+0000","last_clean":"2023-07-29T16:12:49.284664+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:12:49.284664+0000","last_undegraded":"2023-07-29T16:12:49.284664+0000","last_fullsized":"2023-07-29T16:12:49.284664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281333+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:12:49.281333+0000","last_peered":"2023-07-29T16:12:49.281333+0000","last_clean":"2023-07-29T16:12:49.281333+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:12:49.281333+0000","last_undegraded":"2023-07-29T16:12:49.281333+0000","last_fullsized":"2023-07-29T16:12:49.281333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284522+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:12:49.284522+0000","last_peered":"2023-07-29T16:12:49.284522+0000","last_clean":"2023-07-29T16:12:49.284522+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:12:49.284522+0000","last_undegraded":"2023-07-29T16:12:49.284522+0000","last_fullsized":"2023-07-29T16:12:49.284522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279954+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:12:49.279954+0000","last_peered":"2023-07-29T16:12:49.279954+0000","last_clean":"2023-07-29T16:12:49.279954+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:12:49.279954+0000","last_undegraded":"2023-07-29T16:12:49.279954+0000","last_fullsized":"2023-07-29T16:12:49.279954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.284407+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:12:49.284407+0000","last_peered":"2023-07-29T16:12:49.284407+0000","last_clean":"2023-07-29T16:12:49.284407+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:12:49.284407+0000","last_undegraded":"2023-07-29T16:12:49.284407+0000","last_fullsized":"2023-07-29T16:12:49.284407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.282011+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:12:49.282011+0000","last_peered":"2023-07-29T16:12:49.282011+0000","last_clean":"2023-07-29T16:12:49.282011+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:12:49.282011+0000","last_undegraded":"2023-07-29T16:12:49.282011+0000","last_fullsized":"2023-07-29T16:12:49.282011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281568+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:12:49.281568+0000","last_peered":"2023-07-29T16:12:49.281568+0000","last_clean":"2023-07-29T16:12:49.281568+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:12:49.281568+0000","last_undegraded":"2023-07-29T16:12:49.281568+0000","last_fullsized":"2023-07-29T16:12:49.281568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281677+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:12:49.281677+0000","last_peered":"2023-07-29T16:12:49.281677+0000","last_clean":"2023-07-29T16:12:49.281677+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:12:49.281677+0000","last_undegraded":"2023-07-29T16:12:49.281677+0000","last_fullsized":"2023-07-29T16:12:49.281677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.285072+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:12:49.285072+0000","last_peered":"2023-07-29T16:12:49.285072+0000","last_clean":"2023-07-29T16:12:49.285072+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:12:49.285072+0000","last_undegraded":"2023-07-29T16:12:49.285072+0000","last_fullsized":"2023-07-29T16:12:49.285072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279855+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:12:49.279855+0000","last_peered":"2023-07-29T16:12:49.279855+0000","last_clean":"2023-07-29T16:12:49.279855+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:12:49.279855+0000","last_undegraded":"2023-07-29T16:12:49.279855+0000","last_fullsized":"2023-07-29T16:12:49.279855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283348+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:12:49.283348+0000","last_peered":"2023-07-29T16:12:49.283348+0000","last_clean":"2023-07-29T16:12:49.283348+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:12:49.283348+0000","last_undegraded":"2023-07-29T16:12:49.283348+0000","last_fullsized":"2023-07-29T16:12:49.283348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.283730+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:12:49.283730+0000","last_peered":"2023-07-29T16:12:49.283730+0000","last_clean":"2023-07-29T16:12:49.283730+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:12:49.283730+0000","last_undegraded":"2023-07-29T16:12:49.283730+0000","last_fullsized":"2023-07-29T16:12:49.283730+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.277336+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:12:49.277336+0000","last_peered":"2023-07-29T16:12:49.277336+0000","last_clean":"2023-07-29T16:12:49.277336+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:12:49.277336+0000","last_undegraded":"2023-07-29T16:12:49.277336+0000","last_fullsized":"2023-07-29T16:12:49.277336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.286557+0000","last_change":"2023-07-29T16:12:50.285320+0000","last_active":"2023-07-29T16:12:50.286557+0000","last_peered":"2023-07-29T16:12:50.286557+0000","last_clean":"2023-07-29T16:12:50.286557+0000","last_became_active":"2023-07-29T16:12:50.284666+0000","last_became_peered":"2023-07-29T16:12:50.284666+0000","last_unstale":"2023-07-29T16:12:50.286557+0000","last_undegraded":"2023-07-29T16:12:50.286557+0000","last_fullsized":"2023-07-29T16:12:50.286557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:19:09.408935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.276777+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:12:49.276777+0000","last_peered":"2023-07-29T16:12:49.276777+0000","last_clean":"2023-07-29T16:12:49.276777+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:12:49.276777+0000","last_undegraded":"2023-07-29T16:12:49.276777+0000","last_fullsized":"2023-07-29T16:12:49.276777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-29T16:12:50.337354+0000","last_change":"2023-07-29T16:12:50.335842+0000","last_active":"2023-07-29T16:12:50.337354+0000","last_peered":"2023-07-29T16:12:50.337354+0000","last_clean":"2023-07-29T16:12:50.337354+0000","last_became_active":"2023-07-29T16:12:50.335527+0000","last_became_peered":"2023-07-29T16:12:50.335527+0000","last_unstale":"2023-07-29T16:12:50.337354+0000","last_undegraded":"2023-07-29T16:12:50.337354+0000","last_fullsized":"2023-07-29T16:12:50.337354+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:35:18.062454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.279716+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:12:49.279716+0000","last_peered":"2023-07-29T16:12:49.279716+0000","last_clean":"2023-07-29T16:12:49.279716+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:12:49.279716+0000","last_undegraded":"2023-07-29T16:12:49.279716+0000","last_fullsized":"2023-07-29T16:12:49.279716+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-29T16:12:49.281176+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:12:49.281176+0000","last_peered":"2023-07-29T16:12:49.281176+0000","last_clean":"2023-07-29T16:12:49.281176+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:12:49.281176+0000","last_undegraded":"2023-07-29T16:12:49.281176+0000","last_fullsized":"2023-07-29T16:12:49.281176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61192,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":300,"num_read_kb":212,"num_write":148,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350998,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":958464,"data_stored":459516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60599999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.375},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.755},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.67100000000000004}]},{"osd":5,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.49399999999999999}]},{"osd":6,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.64000000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":9,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.913,"5min":1.913,"15min":1.913},"last":0.624},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.82699999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.867,"5min":1.867,"15min":1.867},"last":0.65700000000000003}]}]},{"osd":10,"up_from":73,"seq":313532612635,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1608,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1646592,"data_stored":1107965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.63800000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.374}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.433}]},{"osd":6,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.77400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.319,"5min":1.319,"15min":1.319},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.494,"5min":1.494,"15min":1.494},"last":0.44400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.5369999999999999,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.80600000000000005}]},{"osd":11,"last update":"Sat Jul 29 16:12:39 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.68799999999999994}]}]},{"osd":9,"up_from":68,"seq":292057776158,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":974848,"data_stored":459752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.53400000000000003,"15min":0.54900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.71699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71999999999999997,"5min":1.728,"15min":1.728},"last":1.077}]},{"osd":1,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0029999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.6679999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":1.304}]},{"osd":2,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81399999999999995,"5min":1.125,"15min":1.125},"last":1.6930000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.435,"15min":0.435},"max":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":1.2310000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.80600000000000005,"15min":0.83399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92100000000000004,"5min":1.974,"15min":1.974},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.80200000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.5820000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.91500000000000004,"15min":0.94599999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95899999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.80800000000000005}]},{"osd":5,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94699999999999995,"15min":0.97599999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0600000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":1.7430000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.92800000000000005,"15min":0.96299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95699999999999996,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":1.607},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89000000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":1.2649999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.109,"5min":1.839,"15min":1.839},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.94299999999999995,"15min":0.97099999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0569999999999999,"5min":1.47,"15min":1.47},"last":1.722}]},{"osd":8,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.91700000000000004,"15min":0.94199999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0169999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":1.284},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.004,"15min":1.032},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1220000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.63400000000000001}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"last":1.766}]}]},{"osd":8,"up_from":60,"seq":257698037795,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":970752,"data_stored":459341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":2.0680000000000001,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.46700000000000003,"15min":0.47499999999999998},"min":{"1min":0.32200000000000001,"5min":0.318,"15min":0.318},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.72899999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.625,"15min":0.629},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.63,"15min":0.63200000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.77000000000000002}]},{"osd":4,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.751,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.85899999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.58499999999999996,"15min":0.60299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.64300000000000002,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.871}]},{"osd":6,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.73399999999999999,"15min":0.74399999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":2.1760000000000002,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.74399999999999999,"15min":0.75600000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.93300000000000005}]},{"osd":7,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.73299999999999998,"15min":0.73999999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.73399999999999999,"15min":0.748},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.76800000000000002,"5min":1,"15min":1},"last":1.048}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.751,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.97499999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":1.0669999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":7,"up_from":52,"seq":223338299432,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1528,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1564672,"data_stored":1051564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.39600000000000002},"min":{"1min":0.28499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.498},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.437,"15min":0.43099999999999999},"min":{"1min":0.30299999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70999999999999996,"5min":0.755,"15min":0.755},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.627,"5min":0.61299999999999999,"15min":0.61099999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.80600000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64500000000000002,"15min":0.64600000000000002},"min":{"1min":0.39200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68000000000000005}]},{"osd":4,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.71999999999999997,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.52700000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54800000000000004}]},{"osd":5,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.69199999999999995,"15min":0.66400000000000003},"min":{"1min":0.50900000000000001,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.2300000000000004,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.64700000000000002,"15min":0.64000000000000001},"min":{"1min":0.53200000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.85299999999999998}]},{"osd":6,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.499,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.73699999999999999,"15min":0.70399999999999996},"min":{"1min":0.47199999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":4.2060000000000004,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.69799999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.71599999999999997,"15min":0.71799999999999997},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.91300000000000003,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.70899999999999996,"15min":0.70099999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.91600000000000004}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.871},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77400000000000002,"15min":0.77200000000000002},"min":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.93999999999999995}]},{"osd":10,"last update":"Sat Jul 29 16:12:16 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.89700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]}]},{"osd":6,"up_from":47,"seq":201863462956,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":962560,"data_stored":459147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.502,"15min":0.48499999999999999},"min":{"1min":0.34000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.54600000000000004,"15min":0.51200000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.61899999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.61499999999999999,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.71399999999999997,"15min":0.69399999999999995},"min":{"1min":0.59999999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.69299999999999995,"15min":0.67100000000000004},"min":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.72299999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.63800000000000001,"15min":0.60999999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.67200000000000004,"15min":0.65600000000000003},"min":{"1min":0.47999999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.63400000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.71699999999999997,"15min":0.70799999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73999999999999999,"15min":0.71599999999999997},"min":{"1min":0.48499999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.0269999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:50 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.76300000000000001,"15min":0.75900000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.68200000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":1.1890000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:12:11 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.752,"15min":0.75},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.745},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.77700000000000002,"15min":0.76600000000000001},"min":{"1min":0.52900000000000003,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":1.1240000000000001}]},{"osd":8,"last update":"Sat Jul 29 16:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83299999999999996,"15min":0.83899999999999997},"min":{"1min":0.55100000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0429999999999999,"5min":1.151,"15min":1.151},"last":1.214},{"interface":"front","average":{"1min":0.87,"5min":0.86499999999999999,"15min":0.86399999999999999},"min":{"1min":0.70099999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":1.224}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95199999999999996,"15min":0.94999999999999996},"min":{"1min":0.63800000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.165,"5min":2.165,"15min":2.165},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.94599999999999995,"15min":0.95299999999999996},"min":{"1min":0.58699999999999997,"5min":0.503,"15min":0.503},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":1.155}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.256},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":1.093}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":1019904,"data_stored":515838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.55400000000000005,"15min":0.501},"min":{"1min":0.377,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.55400000000000005,"15min":0.505},"min":{"1min":0.372,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.54400000000000004,"15min":0.52500000000000002},"min":{"1min":0.40600000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.54100000000000004,"15min":0.51100000000000001},"min":{"1min":0.39600000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69899999999999995}]},{"osd":3,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.47899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58399999999999996,"15min":0.55600000000000005},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81200000000000006,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.501,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.36899999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.56899999999999995},"min":{"1min":0.41999999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.629,"15min":0.60099999999999998},"min":{"1min":0.48399999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Jul 29 16:12:45 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.69099999999999995,"15min":0.67500000000000004},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95599999999999996,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.67700000000000005,"15min":0.66600000000000004},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.874,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57099999999999995}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.60499999999999998,"15min":0.58499999999999996},"min":{"1min":0.45000000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.65900000000000003,"15min":0.63600000000000001},"min":{"1min":0.49399999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.60099999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.746,"15min":0.747},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.63},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.76100000000000001,"15min":0.75800000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.48899999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:12:53 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85199999999999998,"15min":0.85699999999999998},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.98799999999999999,"5min":1.161,"15min":1.161},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.85499999999999998,"15min":0.86099999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.995,"5min":1.175,"15min":1.175},"last":0.71799999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.64500000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":983040,"data_stored":460412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56599999999999995,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59199999999999997,"15min":0.54500000000000004},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.77800000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88300000000000001,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.61699999999999999,"15min":0.59599999999999997},"min":{"1min":0.42099999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86099999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.84999999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:10:41 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.372,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.74399999999999999,"5min":0.876,"15min":0.876},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.48399999999999999,"15min":0.47199999999999998},"min":{"1min":0.41099999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77800000000000002,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.75}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85999999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.83399999999999996}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.67000000000000004,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92200000000000004,"5min":1.111,"15min":1.111},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65900000000000003,"15min":0.64200000000000002},"min":{"1min":0.52900000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.82899999999999996,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78900000000000003}]},{"osd":6,"last update":"Sat Jul 29 16:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.70899999999999996,"15min":0.71199999999999997},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.099,"5min":1.266,"15min":1.266},"last":0.505},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.73799999999999999,"15min":0.747},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.036,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.435}]},{"osd":7,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73399999999999999,"15min":0.72999999999999998},"min":{"1min":0.58099999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.81699999999999995,"15min":0.81000000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.73599999999999999}]},{"osd":8,"last update":"Sat Jul 29 16:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.86699999999999999,"15min":0.89100000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.90200000000000002,"5min":1.335,"15min":1.335},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.88300000000000001,"15min":0.90300000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.96099999999999997,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.76900000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.86499999999999999,"15min":0.88300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.079,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.94499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.014,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.81699999999999995}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.212,"5min":1.212,"15min":1.212},"last":0.93100000000000005}]},{"osd":11,"last update":"Sat Jul 29 16:12:40 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.873}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":1040384,"data_stored":516945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56699999999999995,"15min":0.55800000000000005},"min":{"1min":0.41899999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.95299999999999996,"5min":2.3050000000000002,"15min":2.3050000000000002},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.59899999999999998,"15min":0.55100000000000005},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":7.2169999999999996,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.40400000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":3.613,"5min":3.613,"15min":3.613},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.36299999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.74299999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:41 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56999999999999995,"15min":0.53600000000000003},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.753,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56499999999999995,"15min":0.54500000000000004},"min":{"1min":0.40799999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78500000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.64500000000000002}]},{"osd":4,"last update":"Sat Jul 29 16:12:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.626,"15min":0.59499999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":3.6019999999999999,"5min":3.6019999999999999,"15min":3.6019999999999999},"last":0.72599999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.439},"min":{"1min":0.29699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.47699999999999998,"15min":0.46700000000000003},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.626}]},{"osd":6,"last update":"Sat Jul 29 16:12:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89200000000000002,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.70499999999999996,"15min":0.65300000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.877,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.75700000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.93300000000000005,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.64100000000000001,"15min":0.64900000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.68000000000000005}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76300000000000001,"15min":0.76500000000000001},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.91200000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95999999999999996,"5min":0.997,"15min":0.997},"last":0.82599999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80400000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":974848,"data_stored":459133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:38 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.44400000000000001,"15min":0.40300000000000002},"min":{"1min":0.34100000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.625,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.47599999999999998,"15min":0.438},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Sat Jul 29 16:12:38 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.60599999999999998,"15min":0.57299999999999995},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90100000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.61099999999999999,"15min":0.56799999999999995},"min":{"1min":0.443,"5min":0.372,"15min":0.372},"max":{"1min":0.75,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Jul 29 16:12:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67100000000000004,"15min":0.621},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.874,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":1.296,"15min":1.296},"last":0.66200000000000003}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64400000000000002,"15min":0.60699999999999998},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88400000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.53400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98099999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Jul 29 16:12:22 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70599999999999996,"15min":0.69899999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.165,"5min":1.343,"15min":1.343},"last":0.624},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71899999999999997,"15min":0.69899999999999995},"min":{"1min":0.51000000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.67800000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:12:45 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70599999999999996,"15min":0.68400000000000005},"min":{"1min":0.45400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.0620000000000001,"5min":1.276,"15min":1.276},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.371,"15min":0.371},"max":{"1min":0.91800000000000004,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.74399999999999999}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.55400000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.323,"5min":1.323,"15min":1.323},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.84199999999999997,"15min":0.82999999999999996},"min":{"1min":0.71999999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.63500000000000001}]},{"osd":8,"last update":"Sat Jul 29 16:11:29 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.093},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":1.022}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.90300000000000002,"15min":0.92300000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.091,"5min":1.476,"15min":1.476},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.90100000000000002,"15min":0.91800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0509999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.84999999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:12:17 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5349999999999999,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":970752,"data_stored":460499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.52400000000000002,"15min":0.48799999999999999},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.54800000000000004,"15min":0.51700000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.53900000000000003}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.56599999999999995},"min":{"1min":0.50700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.72899999999999998}]},{"osd":3,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60099999999999998,"15min":0.55600000000000005},"min":{"1min":0.45100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93300000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Jul 29 16:12:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.60299999999999998,"15min":0.58399999999999996},"min":{"1min":0.48299999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60799999999999998,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.86099999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71899999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.63900000000000001,"15min":0.623},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68899999999999995,"15min":0.66400000000000003},"min":{"1min":0.52200000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.99099999999999999,"5min":1.018,"15min":1.018},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.58699999999999997},"min":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91700000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.628},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0129999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.502}]},{"osd":8,"last update":"Sat Jul 29 16:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.67100000000000004,"15min":0.66900000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.0109999999999999,"5min":1.026,"15min":1.026},"last":0.622},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.751,"15min":0.748},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.0189999999999999,"5min":1.069,"15min":1.069},"last":0.66800000000000004}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79500000000000004,"15min":0.80000000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.04,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.86799999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:12:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.748}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1581056,"data_stored":1051240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55000000000000004,"15min":0.53500000000000003},"min":{"1min":0.39400000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.79100000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.753,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53200000000000003}]},{"osd":1,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51800000000000002,"15min":0.50600000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69999999999999996,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54200000000000004,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.75900000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.63900000000000001}]},{"osd":2,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40600000000000003,"15min":0.40100000000000002},"min":{"1min":0.27500000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.29799999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.70799999999999996,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.68899999999999995}]},{"osd":3,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81299999999999994,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.70399999999999996}]},{"osd":4,"last update":"Sat Jul 29 16:12:23 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.41699999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.754,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.83699999999999997}]},{"osd":6,"last update":"Sat Jul 29 16:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.66300000000000003,"15min":0.64900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.81100000000000005}]},{"osd":7,"last update":"Sat Jul 29 16:12:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.626},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.79300000000000004,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.872},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63900000000000001,"15min":0.64000000000000001},"min":{"1min":0.49099999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.82699999999999996}]},{"osd":8,"last update":"Sat Jul 29 16:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60299999999999998,"15min":0.60499999999999998},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.754},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.745}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.90700000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:12:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.85899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:13:15.353 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-29T16:13:15.353 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-29T16:13:15.354 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-29T16:13:15.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph health --format=json 2023-07-29T16:13:16.119 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:16 smithi031 ceph-mon[95504]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:13:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:16 smithi031 ceph-mon[95504]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:16 smithi039 ceph-mon[128858]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:13:16.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:16 smithi039 ceph-mon[128858]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:16 smithi156 ceph-mon[129245]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:13:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:16 smithi156 ceph-mon[129245]: pgmap v301: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.578+0000 7f14bd62f700 1 -- 172.21.15.31:0/1670552603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 msgr2=0x7f14b8104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.578+0000 7f14bd62f700 1 --2- 172.21.15.31:0/1670552603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 0x7f14b8104f70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0099c0 tx=0x7f14ac009cd0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 -- 172.21.15.31:0/1670552603 shutdown_connections 2023-07-29T16:13:17.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 --2- 172.21.15.31:0/1670552603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b81056a0 0x7f14b810d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 --2- 172.21.15.31:0/1670552603 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b8100600 0x7f14b81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 --2- 172.21.15.31:0/1670552603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 0x7f14b8104f70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 -- 172.21.15.31:0/1670552603 >> 172.21.15.31:0/1670552603 conn(0x7f14b80fbec0 msgr2=0x7f14b80fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:17.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 -- 172.21.15.31:0/1670552603 shutdown_connections 2023-07-29T16:13:17.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.579+0000 7f14bd62f700 1 -- 172.21.15.31:0/1670552603 wait complete. 2023-07-29T16:13:17.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 Processor -- start 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 -- start start 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b8100600 0x7f14b81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 0x7f14b8198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 0x7f14b819bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b819c5b0 con 0x7f14b81056a0 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b819c720 con 0x7f14b8100600 2023-07-29T16:13:17.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14bd62f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b819c8e0 con 0x7f14b8100fa0 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.580+0000 7f14b77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 0x7f14b819bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 0x7f14b819bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44926/0 (socket says 172.21.15.31:44926) 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b8100600 0x7f14b81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b77fe700 1 -- 172.21.15.31:0/2695289378 learned_addr learned my addr 172.21.15.31:0/2695289378 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b67fc700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 0x7f14b8198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 -- 172.21.15.31:0/2695289378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 msgr2=0x7f14b8198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:17.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 0x7f14b8198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 -- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 msgr2=0x7f14b819bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 0x7f14b819bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 -- 172.21.15.31:0/2695289378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a8009520 con 0x7f14b8100600 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b77fe700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 0x7f14b819bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14b6ffd700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b8100600 0x7f14b81986b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f14a800e9f0 tx=0x7f14a800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a800cb70 con 0x7f14b8100600 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac0095f0 con 0x7f14b8100600 2023-07-29T16:13:17.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a800ee90 con 0x7f14b8100600 2023-07-29T16:13:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.581+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a8003cf0 con 0x7f14b8100600 2023-07-29T16:13:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.582+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b81a5340 con 0x7f14b8100600 2023-07-29T16:13:17.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.583+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b804ea50 con 0x7f14b8100600 2023-07-29T16:13:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.585+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f14a800ccd0 con 0x7f14b8100600 2023-07-29T16:13:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.586+0000 7f149ffff700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14a00a08a0 0x7f14a00a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.586+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f14a8014070 con 0x7f14b8100600 2023-07-29T16:13:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.586+0000 7f14b67fc700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14a00a08a0 0x7f14a00a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:17.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.587+0000 7f14b67fc700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14a00a08a0 0x7f14a00a2d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009fd0 tx=0x7f14ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:17.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.588+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f14a8095990 con 0x7f14b8100600 2023-07-29T16:13:17.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.839+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f14b8199a10 con 0x7f14b8100600 2023-07-29T16:13:17.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.840+0000 7f149ffff700 1 -- 172.21.15.31:0/2695289378 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f14a8098fb0 con 0x7f14b8100600 2023-07-29T16:13:17.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:17.838 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-29T16:13:17.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14a00a08a0 msgr2=0x7f14a00a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14a00a08a0 0x7f14a00a2d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f14ac009fd0 tx=0x7f14ac005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b8100600 msgr2=0x7f14b81986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b8100600 0x7f14b81986b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f14a800e9f0 tx=0x7f14a800ed00 comp rx=0 tx=0).stop 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 shutdown_connections 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14a00a08a0 0x7f14a00a2d50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14b81056a0 0x7f14b819bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14b8100600 0x7f14b81986b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 --2- 172.21.15.31:0/2695289378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14b8100fa0 0x7f14b8198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.842+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 >> 172.21.15.31:0/2695289378 conn(0x7f14b80fbec0 msgr2=0x7f14b80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.843+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 shutdown_connections 2023-07-29T16:13:17.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:17.843+0000 7f14bd62f700 1 -- 172.21.15.31:0/2695289378 wait complete. 2023-07-29T16:13:18.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:17 smithi039 ceph-mon[128858]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:18.031 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:17 smithi031 ceph-mon[95504]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:17 smithi156 ceph-mon[129245]: pgmap v302: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:18.304 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-29T16:13:18.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 -- 172.21.15.31:0/4257730703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754100340 msgr2=0x7f5754100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 --2- 172.21.15.31:0/4257730703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754100340 0x7f5754100750 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5744009990 tx=0x7f5744009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 -- 172.21.15.31:0/4257730703 shutdown_connections 2023-07-29T16:13:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 --2- 172.21.15.31:0/4257730703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5754101ec0 0x7f5754108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 --2- 172.21.15.31:0/4257730703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754100340 0x7f5754100750 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 --2- 172.21.15.31:0/4257730703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5754101530 0x7f5754101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 -- 172.21.15.31:0/4257730703 >> 172.21.15.31:0/4257730703 conn(0x7f57540fb920 msgr2=0x7f57540fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 -- 172.21.15.31:0/4257730703 shutdown_connections 2023-07-29T16:13:18.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.516+0000 7f57591ff700 1 -- 172.21.15.31:0/4257730703 wait complete. 2023-07-29T16:13:18.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.517+0000 7f57591ff700 1 Processor -- start 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.517+0000 7f57591ff700 1 -- start start 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.517+0000 7f57591ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5754100340 0x7f5754074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.517+0000 7f57591ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 0x7f5754073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f57591ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5754101ec0 0x7f5754073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f57591ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5754073e20 con 0x7f5754101530 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f57591ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5754073f90 con 0x7f5754100340 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f57591ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57541a00c0 con 0x7f5754101ec0 2023-07-29T16:13:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 0x7f5754073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 0x7f5754073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44956/0 (socket says 172.21.15.31:44956) 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 -- 172.21.15.31:0/1321041218 learned_addr learned my addr 172.21.15.31:0/1321041218 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f575359e700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5754101ec0 0x7f5754073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f5752d9d700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5754100340 0x7f5754074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 -- 172.21.15.31:0/1321041218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5754101ec0 msgr2=0x7f5754073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5754101ec0 0x7f5754073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 -- 172.21.15.31:0/1321041218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5754100340 msgr2=0x7f5754074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5754100340 0x7f5754074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 -- 172.21.15.31:0/1321041218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57440095f0 con 0x7f5754101530 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.518+0000 7f574bfff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 0x7f5754073140 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e9a0 tx=0x7f573c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.519+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c009790 con 0x7f5754101530 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.519+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f573c010430 con 0x7f5754101530 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.519+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57541a03a0 con 0x7f5754101530 2023-07-29T16:13:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.519+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c018680 con 0x7f5754101530 2023-07-29T16:13:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.519+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57541a0900 con 0x7f5754101530 2023-07-29T16:13:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.520+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5754107830 con 0x7f5754101530 2023-07-29T16:13:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.522+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f573c0105a0 con 0x7f5754101530 2023-07-29T16:13:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.523+0000 7f5750d99700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57400a08e0 0x7f57400a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.523+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f573c014070 con 0x7f5754101530 2023-07-29T16:13:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.524+0000 7f5752d9d700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57400a08e0 0x7f57400a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.524+0000 7f5752d9d700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57400a08e0 0x7f57400a2d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f574400b2e0 tx=0x7f5744005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.527+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f573c051720 con 0x7f5754101530 2023-07-29T16:13:18.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.735+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f575402cc30 con 0x7f5754101530 2023-07-29T16:13:18.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.736+0000 7f5750d99700 1 -- 172.21.15.31:0/1321041218 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22618 (secure 0 0 0) 0x7f573c0957a0 con 0x7f5754101530 2023-07-29T16:13:18.735 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:18.735 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:50.274238+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:18.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57400a08e0 msgr2=0x7f57400a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57400a08e0 0x7f57400a2d90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f574400b2e0 tx=0x7f5744005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 msgr2=0x7f5754073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 0x7f5754073140 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f573c00e9a0 tx=0x7f573c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 shutdown_connections 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57400a08e0 0x7f57400a2d90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5754101ec0 0x7f5754073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5754100340 0x7f5754074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 --2- 172.21.15.31:0/1321041218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5754101530 0x7f5754073140 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 >> 172.21.15.31:0/1321041218 conn(0x7f57540fb920 msgr2=0x7f5754104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:18.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 shutdown_connections 2023-07-29T16:13:18.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.739+0000 7f57591ff700 1 -- 172.21.15.31:0/1321041218 wait complete. 2023-07-29T16:13:18.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:13:18.757 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:13:18.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 -- 172.21.15.31:0/1180838459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda780737b0 msgr2=0x7fda78073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 --2- 172.21.15.31:0/1180838459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda780737b0 0x7fda78073c20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fda68009990 tx=0x7fda68009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 -- 172.21.15.31:0/1180838459 shutdown_connections 2023-07-29T16:13:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 --2- 172.21.15.31:0/1180838459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda7810c900 0x7fda7810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 --2- 172.21.15.31:0/1180838459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda78074d90 0x7fda780731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 --2- 172.21.15.31:0/1180838459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda780737b0 0x7fda78073c20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 -- 172.21.15.31:0/1180838459 >> 172.21.15.31:0/1180838459 conn(0x7fda780fb970 msgr2=0x7fda780fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:18.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.965+0000 7fda801ff700 1 -- 172.21.15.31:0/1180838459 shutdown_connections 2023-07-29T16:13:18.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.966+0000 7fda801ff700 1 -- 172.21.15.31:0/1180838459 wait complete. 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.966+0000 7fda801ff700 1 Processor -- start 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 -- start start 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 0x7fda7819c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda78074d90 0x7fda7819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda7810c900 0x7fda781a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781a4280 con 0x7fda780737b0 2023-07-29T16:13:18.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781a43c0 con 0x7fda7810c900 2023-07-29T16:13:18.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda801ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda781a4550 con 0x7fda78074d90 2023-07-29T16:13:18.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda7df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 0x7fda7819c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda7df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 0x7fda7819c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44984/0 (socket says 172.21.15.31:44984) 2023-07-29T16:13:18.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda7df9b700 1 -- 172.21.15.31:0/2112574765 learned_addr learned my addr 172.21.15.31:0/2112574765 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:18.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda7e79c700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda7810c900 0x7fda781a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.967+0000 7fda7d79a700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda78074d90 0x7fda7819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda7df9b700 1 -- 172.21.15.31:0/2112574765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda78074d90 msgr2=0x7fda7819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda7df9b700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda78074d90 0x7fda7819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda7df9b700 1 -- 172.21.15.31:0/2112574765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda7810c900 msgr2=0x7fda781a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda7df9b700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda7810c900 0x7fda781a3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda7df9b700 1 -- 172.21.15.31:0/2112574765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74009520 con 0x7fda780737b0 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda7df9b700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 0x7fda7819c700 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fda7400ea30 tx=0x7fda7400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74009820 con 0x7fda780737b0 2023-07-29T16:13:18.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda740044e0 con 0x7fda780737b0 2023-07-29T16:13:18.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda680095f0 con 0x7fda780737b0 2023-07-29T16:13:18.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74003dc0 con 0x7fda780737b0 2023-07-29T16:13:18.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.968+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda781a4bb0 con 0x7fda780737b0 2023-07-29T16:13:18.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.969+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda78102450 con 0x7fda780737b0 2023-07-29T16:13:18.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.971+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fda74004000 con 0x7fda780737b0 2023-07-29T16:13:18.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.973+0000 7fda6effd700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fda640a0820 0x7fda640a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:18.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.974+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fda74014070 con 0x7fda780737b0 2023-07-29T16:13:18.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.974+0000 7fda7d79a700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fda640a0820 0x7fda640a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:18.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.974+0000 7fda7d79a700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fda640a0820 0x7fda640a2cd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fda6800b2e0 tx=0x7fda68005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:18.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:18.977+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fda740959e0 con 0x7fda780737b0 2023-07-29T16:13:19.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:18 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2695289378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-29T16:13:19.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:18 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1321041218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:18 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2695289378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-29T16:13:19.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:18 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1321041218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:18 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2695289378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-29T16:13:19.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:18 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1321041218' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.186+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda7802cea0 con 0x7fda780737b0 2023-07-29T16:13:19.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.187+0000 7fda6effd700 1 -- 172.21.15.31:0/2112574765 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda74016330 con 0x7fda780737b0 2023-07-29T16:13:19.185 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:13:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fda640a0820 msgr2=0x7fda640a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fda640a0820 0x7fda640a2cd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fda6800b2e0 tx=0x7fda68005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 msgr2=0x7fda7819c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 0x7fda7819c700 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fda7400ea30 tx=0x7fda7400ed40 comp rx=0 tx=0).stop 2023-07-29T16:13:19.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 shutdown_connections 2023-07-29T16:13:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fda640a0820 0x7fda640a2cd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fda7810c900 0x7fda781a3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda780737b0 0x7fda7819c700 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 --2- 172.21.15.31:0/2112574765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda78074d90 0x7fda7819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 >> 172.21.15.31:0/2112574765 conn(0x7fda780fb970 msgr2=0x7fda780fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:19.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 shutdown_connections 2023-07-29T16:13:19.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.189+0000 7fda801ff700 1 -- 172.21.15.31:0/2112574765 wait complete. 2023-07-29T16:13:19.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:13:19.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 -- 172.21.15.31:0/713215719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc101510 msgr2=0x7f33cc101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 --2- 172.21.15.31:0/713215719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc101510 0x7f33cc101960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f33bc009990 tx=0x7f33bc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 -- 172.21.15.31:0/713215719 shutdown_connections 2023-07-29T16:13:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 --2- 172.21.15.31:0/713215719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33cc101ea0 0x7f33cc108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 --2- 172.21.15.31:0/713215719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33cc100320 0x7f33cc100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 --2- 172.21.15.31:0/713215719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc101510 0x7f33cc101960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 -- 172.21.15.31:0/713215719 >> 172.21.15.31:0/713215719 conn(0x7f33cc0fb920 msgr2=0x7f33cc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 -- 172.21.15.31:0/713215719 shutdown_connections 2023-07-29T16:13:19.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.414+0000 7f33d37ff700 1 -- 172.21.15.31:0/713215719 wait complete. 2023-07-29T16:13:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 Processor -- start 2023-07-29T16:13:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 -- start start 2023-07-29T16:13:19.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 0x7f33cc198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33cc101510 0x7f33cc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33cc101ea0 0x7f33cc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc19fe10 con 0x7f33cc100320 2023-07-29T16:13:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc19ff50 con 0x7f33cc101ea0 2023-07-29T16:13:19.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d37ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc1a00e0 con 0x7f33cc101510 2023-07-29T16:13:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.415+0000 7f33d159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 0x7f33cc198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 0x7f33cc198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44998/0 (socket says 172.21.15.31:44998) 2023-07-29T16:13:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 -- 172.21.15.31:0/846005396 learned_addr learned my addr 172.21.15.31:0/846005396 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:19.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d1d9c700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33cc101ea0 0x7f33cc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d0d9a700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33cc101510 0x7f33cc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 -- 172.21.15.31:0/846005396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33cc101510 msgr2=0x7f33cc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33cc101510 0x7f33cc1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 -- 172.21.15.31:0/846005396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33cc101ea0 msgr2=0x7f33cc19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33cc101ea0 0x7f33cc19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 -- 172.21.15.31:0/846005396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc0095f0 con 0x7f33cc100320 2023-07-29T16:13:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d159b700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 0x7f33cc198290 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f33c800e970 tx=0x7f33c800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8009790 con 0x7f33cc100320 2023-07-29T16:13:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33c800c7e0 con 0x7f33cc100320 2023-07-29T16:13:19.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c8003c70 con 0x7f33cc100320 2023-07-29T16:13:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.416+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc1a03c0 con 0x7f33cc100320 2023-07-29T16:13:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.417+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc1a0880 con 0x7f33cc100320 2023-07-29T16:13:19.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.418+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc192a10 con 0x7f33cc100320 2023-07-29T16:13:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.419+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f33c800cc50 con 0x7f33cc100320 2023-07-29T16:13:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.421+0000 7f33c27fc700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33b80a08a0 0x7f33b80a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.421+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f33c8014070 con 0x7f33cc100320 2023-07-29T16:13:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.421+0000 7f33d0d9a700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33b80a08a0 0x7f33b80a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.422+0000 7f33d0d9a700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33b80a08a0 0x7f33b80a2d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b2e0 tx=0x7f33bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:19.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.425+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f33c80d2080 con 0x7f33cc100320 2023-07-29T16:13:19.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.633+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33cc02cee0 con 0x7f33cc100320 2023-07-29T16:13:19.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.633+0000 7f33c27fc700 1 -- 172.21.15.31:0/846005396 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33c8095740 con 0x7f33cc100320 2023-07-29T16:13:19.631 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:13:19.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33b80a08a0 msgr2=0x7f33b80a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33b80a08a0 0x7f33b80a2d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b2e0 tx=0x7f33bc005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 msgr2=0x7f33cc198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 0x7f33cc198290 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f33c800e970 tx=0x7f33c800ec80 comp rx=0 tx=0).stop 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 shutdown_connections 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f33b80a08a0 0x7f33b80a2d50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33cc101ea0 0x7f33cc19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33cc100320 0x7f33cc198290 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 --2- 172.21.15.31:0/846005396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33cc101510 0x7f33cc1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.635+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 >> 172.21.15.31:0/846005396 conn(0x7f33cc0fb920 msgr2=0x7f33cc106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:19.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.636+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 shutdown_connections 2023-07-29T16:13:19.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.636+0000 7f33d37ff700 1 -- 172.21.15.31:0/846005396 wait complete. 2023-07-29T16:13:19.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:13:19.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 -- 172.21.15.31:0/1543513900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530108220 msgr2=0x7fa53010a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 --2- 172.21.15.31:0/1543513900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530108220 0x7fa53010a6a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa528009f90 tx=0x7fa52800b330 comp rx=0 tx=0).stop 2023-07-29T16:13:19.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 -- 172.21.15.31:0/1543513900 shutdown_connections 2023-07-29T16:13:19.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 --2- 172.21.15.31:0/1543513900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530108220 0x7fa53010a6a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 --2- 172.21.15.31:0/1543513900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 0x7fa530073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 --2- 172.21.15.31:0/1543513900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa530073750 0x7fa530073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 -- 172.21.15.31:0/1543513900 >> 172.21.15.31:0/1543513900 conn(0x7fa5300fb9b0 msgr2=0x7fa5300fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:19.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 -- 172.21.15.31:0/1543513900 shutdown_connections 2023-07-29T16:13:19.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.862+0000 7fa5383ff700 1 -- 172.21.15.31:0/1543513900 wait complete. 2023-07-29T16:13:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.863+0000 7fa5383ff700 1 Processor -- start 2023-07-29T16:13:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.863+0000 7fa5383ff700 1 -- start start 2023-07-29T16:13:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa5383ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 0x7fa530198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa5383ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 0x7fa5301988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa5383ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa530108220 0x7fa53019f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa5383ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53019ff00 con 0x7fa530108220 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa5383ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301a0040 con 0x7fa530073750 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa5383ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5301a01d0 con 0x7fa530074d30 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 0x7fa530198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 0x7fa530198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50790/0 (socket says 172.21.15.31:50790) 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 -- 172.21.15.31:0/2552027404 learned_addr learned my addr 172.21.15.31:0/2552027404 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53699c700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa530108220 0x7fa53019f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53599a700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 0x7fa5301988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 -- 172.21.15.31:0/2552027404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 msgr2=0x7fa5301988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 0x7fa5301988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 -- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa530108220 msgr2=0x7fa53019f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa530108220 0x7fa53019f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 -- 172.21.15.31:0/2552027404 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa528009c40 con 0x7fa530073750 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53599a700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 0x7fa5301988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.864+0000 7fa53619b700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 0x7fa530198380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00e9a0 tx=0x7fa52c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:19.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.865+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c009790 con 0x7fa530073750 2023-07-29T16:13:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.865+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa52c00cbd0 con 0x7fa530073750 2023-07-29T16:13:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.865+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5301a04b0 con 0x7fa530073750 2023-07-29T16:13:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.865+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c018680 con 0x7fa530073750 2023-07-29T16:13:19.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.865+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5301a0a10 con 0x7fa530073750 2023-07-29T16:13:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.866+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa530106960 con 0x7fa530073750 2023-07-29T16:13:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.868+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fa52c00cd40 con 0x7fa530073750 2023-07-29T16:13:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.869+0000 7fa5277fe700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa51c0a08a0 0x7fa51c0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.869+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fa52c014070 con 0x7fa530073750 2023-07-29T16:13:19.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.870+0000 7fa53599a700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa51c0a08a0 0x7fa51c0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:19.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.870+0000 7fa53599a700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa51c0a08a0 0x7fa51c0a2d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa520005fd0 tx=0x7fa520009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:19.872+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa52c0958f0 con 0x7fa530073750 2023-07-29T16:13:19.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:19 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2112574765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:19.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:19 smithi031 ceph-mon[95504]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:19.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:19 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/846005396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:20.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:19 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2112574765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:20.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:19 smithi039 ceph-mon[128858]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:20.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:19 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/846005396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:20.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:19 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2112574765' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:20.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:19 smithi156 ceph-mon[129245]: pgmap v303: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:20.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:19 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/846005396' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:20.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.082+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa53002cdc0 con 0x7fa530073750 2023-07-29T16:13:20.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.084+0000 7fa5277fe700 1 -- 172.21.15.31:0/2552027404 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa52c098f10 con 0x7fa530073750 2023-07-29T16:13:20.081 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:13:20.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa51c0a08a0 msgr2=0x7fa51c0a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa51c0a08a0 0x7fa51c0a2d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa520005fd0 tx=0x7fa520009310 comp rx=0 tx=0).stop 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 msgr2=0x7fa530198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 0x7fa530198380 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa52c00e9a0 tx=0x7fa52c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 shutdown_connections 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa51c0a08a0 0x7fa51c0a2d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa530108220 0x7fa53019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa530073750 0x7fa530198380 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 --2- 172.21.15.31:0/2552027404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa530074d30 0x7fa5301988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.085+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 >> 172.21.15.31:0/2552027404 conn(0x7fa5300fb9b0 msgr2=0x7fa5300fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.086+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 shutdown_connections 2023-07-29T16:13:20.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.086+0000 7fa5383ff700 1 -- 172.21.15.31:0/2552027404 wait complete. 2023-07-29T16:13:20.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:13:20.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 -- 172.21.15.31:0/1056024289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c101540 msgr2=0x7f3a7c101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 --2- 172.21.15.31:0/1056024289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c101540 0x7f3a7c101990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009990 tx=0x7f3a70009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 -- 172.21.15.31:0/1056024289 shutdown_connections 2023-07-29T16:13:20.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 --2- 172.21.15.31:0/1056024289 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c101ed0 0x7f3a7c108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 --2- 172.21.15.31:0/1056024289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c100350 0x7f3a7c100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 --2- 172.21.15.31:0/1056024289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c101540 0x7f3a7c101990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 -- 172.21.15.31:0/1056024289 >> 172.21.15.31:0/1056024289 conn(0x7f3a7c0fb950 msgr2=0x7f3a7c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:20.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 -- 172.21.15.31:0/1056024289 shutdown_connections 2023-07-29T16:13:20.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.308+0000 7f3a829ff700 1 -- 172.21.15.31:0/1056024289 wait complete. 2023-07-29T16:13:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.309+0000 7f3a829ff700 1 Processor -- start 2023-07-29T16:13:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 -- start start 2023-07-29T16:13:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 0x7f3a7c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c101540 0x7f3a7c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c101ed0 0x7f3a7c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c19dbc0 con 0x7f3a7c100350 2023-07-29T16:13:20.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c19dd00 con 0x7f3a7c101540 2023-07-29T16:13:20.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a829ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a7c19de90 con 0x7f3a7c101ed0 2023-07-29T16:13:20.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a7bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 0x7f3a7c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a7bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 0x7f3a7c196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39212/0 (socket says 172.21.15.31:39212) 2023-07-29T16:13:20.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a7bfff700 1 -- 172.21.15.31:0/3392106466 learned_addr learned my addr 172.21.15.31:0/3392106466 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:20.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.310+0000 7f3a7b7fe700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c101540 0x7f3a7c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a80f9c700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c101ed0 0x7f3a7c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a7bfff700 1 -- 172.21.15.31:0/3392106466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c101ed0 msgr2=0x7f3a7c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a7bfff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c101ed0 0x7f3a7c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a7bfff700 1 -- 172.21.15.31:0/3392106466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c101540 msgr2=0x7f3a7c196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a7bfff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c101540 0x7f3a7c196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a7bfff700 1 -- 172.21.15.31:0/3392106466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a700095f0 con 0x7f3a7c100350 2023-07-29T16:13:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a80f9c700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c101ed0 0x7f3a7c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a7bfff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 0x7f3a7c196040 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d980 tx=0x7f3a6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:20.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00b950 con 0x7f3a7c100350 2023-07-29T16:13:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a6c007620 con 0x7f3a7c100350 2023-07-29T16:13:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a7c19e170 con 0x7f3a7c100350 2023-07-29T16:13:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00f3c0 con 0x7f3a7c100350 2023-07-29T16:13:20.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.311+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7c19e6d0 con 0x7f3a7c100350 2023-07-29T16:13:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.312+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a7c10bb80 con 0x7f3a7c100350 2023-07-29T16:13:20.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.314+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f3a6c010640 con 0x7f3a7c100350 2023-07-29T16:13:20.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.316+0000 7f3a797fa700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a640a08e0 0x7f3a640a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.316+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f3a6c0cba80 con 0x7f3a7c100350 2023-07-29T16:13:20.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.316+0000 7f3a7b7fe700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a640a08e0 0x7f3a640a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.317+0000 7f3a7b7fe700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a640a08e0 0x7f3a640a2d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000b2e0 tx=0x7f3a70005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:20.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.320+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3a6c095340 con 0x7f3a7c100350 2023-07-29T16:13:20.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.575+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a7c1071b0 con 0x7f3a7c100350 2023-07-29T16:13:20.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.576+0000 7f3a797fa700 1 -- 172.21.15.31:0/3392106466 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f3a6c016070 con 0x7f3a7c100350 2023-07-29T16:13:20.574 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:20.575 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:13:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.578+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a640a08e0 msgr2=0x7f3a640a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.578+0000 7f3a829ff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a640a08e0 0x7f3a640a2d90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000b2e0 tx=0x7f3a70005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 msgr2=0x7f3a7c196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 0x7f3a7c196040 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d980 tx=0x7f3a6c00dc90 comp rx=0 tx=0).stop 2023-07-29T16:13:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 shutdown_connections 2023-07-29T16:13:20.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a640a08e0 0x7f3a640a2d90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a7c101ed0 0x7f3a7c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a7c100350 0x7f3a7c196040 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 --2- 172.21.15.31:0/3392106466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c101540 0x7f3a7c196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 >> 172.21.15.31:0/3392106466 conn(0x7f3a7c0fb950 msgr2=0x7f3a7c0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 shutdown_connections 2023-07-29T16:13:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.579+0000 7f3a829ff700 1 -- 172.21.15.31:0/3392106466 wait complete. 2023-07-29T16:13:20.579 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:13:20.596 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/1360761737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 msgr2=0x7f6a2810a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/1360761737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2810a650 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000b0a0 tx=0x7f6a2000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/1360761737 shutdown_connections 2023-07-29T16:13:20.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/1360761737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2810a650 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/1360761737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 0x7f6a28073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/1360761737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a28073750 0x7f6a28073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/1360761737 >> 172.21.15.31:0/1360761737 conn(0x7f6a280fb920 msgr2=0x7f6a280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/1360761737 shutdown_connections 2023-07-29T16:13:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.801+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/1360761737 wait complete. 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 Processor -- start 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 -- start start 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a28073750 0x7f6a28198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 0x7f6a28198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2819fec0 con 0x7f6a281081d0 2023-07-29T16:13:20.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a281a0000 con 0x7f6a28073750 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.802+0000 7f6a2f7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a281a0190 con 0x7f6a28074d30 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2cd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 0x7f6a28198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2cd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 0x7f6a28198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41306/0 (socket says 172.21.15.31:41306) 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2cd9a700 1 -- 172.21.15.31:0/965813808 learned_addr learned my addr 172.21.15.31:0/965813808 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2d59b700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a28073750 0x7f6a28198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 -- 172.21.15.31:0/965813808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 msgr2=0x7f6a28198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 0x7f6a28198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 -- 172.21.15.31:0/965813808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a28073750 msgr2=0x7f6a28198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a28073750 0x7f6a28198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 -- 172.21.15.31:0/965813808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20009d00 con 0x7f6a281081d0 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a2dd9c700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2819f8f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009fd0 tx=0x7f6a20007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.803+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2000e040 con 0x7f6a281081d0 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.804+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a281a0410 con 0x7f6a281081d0 2023-07-29T16:13:20.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.804+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a20004c50 con 0x7f6a281081d0 2023-07-29T16:13:20.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.804+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001b6c0 con 0x7f6a281081d0 2023-07-29T16:13:20.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.804+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a281a0830 con 0x7f6a281081d0 2023-07-29T16:13:20.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.805+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a28106910 con 0x7f6a281081d0 2023-07-29T16:13:20.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.807+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f6a2001d470 con 0x7f6a281081d0 2023-07-29T16:13:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.808+0000 7f6a1e7fc700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a140a0860 0x7f6a140a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.808+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f6a200ccea0 con 0x7f6a281081d0 2023-07-29T16:13:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.808+0000 7f6a2d59b700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a140a0860 0x7f6a140a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:20.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.809+0000 7f6a2d59b700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a140a0860 0x7f6a140a2d10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009c60 tx=0x7f6a24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:20.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:20.812+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6a200967f0 con 0x7f6a281081d0 2023-07-29T16:13:20.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:20 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2552027404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:20.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:20 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3392106466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:21.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.022+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a2802cc30 con 0x7f6a281081d0 2023-07-29T16:13:21.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.023+0000 7f6a1e7fc700 1 -- 172.21.15.31:0/965813808 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22618 (secure 0 0 0) 0x7f6a20099e10 con 0x7f6a281081d0 2023-07-29T16:13:21.022 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:21.022 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:50.274238+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:21.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:20 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2552027404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:21.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:20 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3392106466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:21.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a140a0860 msgr2=0x7f6a140a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a140a0860 0x7f6a140a2d10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009c60 tx=0x7f6a24009260 comp rx=0 tx=0).stop 2023-07-29T16:13:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 msgr2=0x7f6a2819f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2819f8f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009fd0 tx=0x7f6a20007930 comp rx=0 tx=0).stop 2023-07-29T16:13:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 shutdown_connections 2023-07-29T16:13:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6a140a0860 0x7f6a140a2d10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a281081d0 0x7f6a2819f8f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a28073750 0x7f6a28198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 --2- 172.21.15.31:0/965813808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a28074d30 0x7f6a28198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 >> 172.21.15.31:0/965813808 conn(0x7f6a280fb920 msgr2=0x7f6a2810a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 shutdown_connections 2023-07-29T16:13:21.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.026+0000 7f6a2f7ff700 1 -- 172.21.15.31:0/965813808 wait complete. 2023-07-29T16:13:21.043 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:13:21.043 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:13:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:20 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2552027404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:21.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:20 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3392106466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:21.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 -- 172.21.15.31:0/1767885336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 msgr2=0x7ff87c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 --2- 172.21.15.31:0/1767885336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c0731e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff86c009990 tx=0x7ff86c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 -- 172.21.15.31:0/1767885336 shutdown_connections 2023-07-29T16:13:21.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 --2- 172.21.15.31:0/1767885336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c10c8f0 0x7ff87c10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 --2- 172.21.15.31:0/1767885336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c0731e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 --2- 172.21.15.31:0/1767885336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87c0737b0 0x7ff87c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 -- 172.21.15.31:0/1767885336 >> 172.21.15.31:0/1767885336 conn(0x7ff87c0fb920 msgr2=0x7ff87c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:21.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 -- 172.21.15.31:0/1767885336 shutdown_connections 2023-07-29T16:13:21.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.252+0000 7ff880dff700 1 -- 172.21.15.31:0/1767885336 wait complete. 2023-07-29T16:13:21.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 Processor -- start 2023-07-29T16:13:21.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 -- start start 2023-07-29T16:13:21.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 0x7ff87c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87c10c8f0 0x7ff87c1a3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1a4270 con 0x7ff87c074d90 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1a43b0 con 0x7ff87c10c8f0 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.253+0000 7ff880dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c1a4540 con 0x7ff87c0737b0 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff87a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 0x7ff87c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff87a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 0x7ff87c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41314/0 (socket says 172.21.15.31:41314) 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff87a59c700 1 -- 172.21.15.31:0/1337704757 learned_addr learned my addr 172.21.15.31:0/1337704757 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff87ad9d700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87c10c8f0 0x7ff87c1a3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 -- 172.21.15.31:0/1337704757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 msgr2=0x7ff87c19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 0x7ff87c19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 -- 172.21.15.31:0/1337704757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87c10c8f0 msgr2=0x7ff87c1a3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87c10c8f0 0x7ff87c1a3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 -- 172.21.15.31:0/1337704757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c0095f0 con 0x7ff87c074d90 2023-07-29T16:13:21.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff87a59c700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 0x7ff87c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:21.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.254+0000 7ff873fff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c19cc30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff86400e9c0 tx=0x7ff86400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:21.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.255+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864004220 con 0x7ff87c074d90 2023-07-29T16:13:21.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.255+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff864004380 con 0x7ff87c074d90 2023-07-29T16:13:21.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.255+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87c1a4820 con 0x7ff87c074d90 2023-07-29T16:13:21.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.255+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff864003be0 con 0x7ff87c074d90 2023-07-29T16:13:21.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.255+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c1a4d80 con 0x7ff87c074d90 2023-07-29T16:13:21.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.256+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff87c102440 con 0x7ff87c074d90 2023-07-29T16:13:21.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.258+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7ff864003d40 con 0x7ff87c074d90 2023-07-29T16:13:21.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.259+0000 7ff8737fe700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8680a08a0 0x7ff8680a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.259+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7ff864014070 con 0x7ff87c074d90 2023-07-29T16:13:21.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.259+0000 7ff87a59c700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8680a08a0 0x7ff8680a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.260+0000 7ff87a59c700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8680a08a0 0x7ff8680a2d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00b2e0 tx=0x7ff86c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:21.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.263+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff864095840 con 0x7ff87c074d90 2023-07-29T16:13:21.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.471+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff87c02ce10 con 0x7ff87c074d90 2023-07-29T16:13:21.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.471+0000 7ff8737fe700 1 -- 172.21.15.31:0/1337704757 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff864098e60 con 0x7ff87c074d90 2023-07-29T16:13:21.469 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:13:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8680a08a0 msgr2=0x7ff8680a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8680a08a0 0x7ff8680a2d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00b2e0 tx=0x7ff86c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 msgr2=0x7ff87c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c19cc30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff86400e9c0 tx=0x7ff86400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 shutdown_connections 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8680a08a0 0x7ff8680a2d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff87c10c8f0 0x7ff87c1a3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff87c0737b0 0x7ff87c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 --2- 172.21.15.31:0/1337704757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c074d90 0x7ff87c19cc30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 >> 172.21.15.31:0/1337704757 conn(0x7ff87c0fb920 msgr2=0x7ff87c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 shutdown_connections 2023-07-29T16:13:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.474+0000 7ff880dff700 1 -- 172.21.15.31:0/1337704757 wait complete. 2023-07-29T16:13:21.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:13:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 -- 172.21.15.31:0/150234586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654100320 msgr2=0x7f1654100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 --2- 172.21.15.31:0/150234586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654100320 0x7f1654100730 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1650009990 tx=0x7f1650009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 -- 172.21.15.31:0/150234586 shutdown_connections 2023-07-29T16:13:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 --2- 172.21.15.31:0/150234586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1654101ea0 0x7f1654108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 --2- 172.21.15.31:0/150234586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654100320 0x7f1654100730 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 --2- 172.21.15.31:0/150234586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1654101510 0x7f1654101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 -- 172.21.15.31:0/150234586 >> 172.21.15.31:0/150234586 conn(0x7f16540fb920 msgr2=0x7f16540fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 -- 172.21.15.31:0/150234586 shutdown_connections 2023-07-29T16:13:21.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.699+0000 7f165b5ff700 1 -- 172.21.15.31:0/150234586 wait complete. 2023-07-29T16:13:21.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.700+0000 7f165b5ff700 1 Processor -- start 2023-07-29T16:13:21.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.700+0000 7f165b5ff700 1 -- start start 2023-07-29T16:13:21.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.700+0000 7f165b5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1654100320 0x7f1654074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.700+0000 7f165b5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 0x7f1654073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.700+0000 7f165b5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1654101ea0 0x7f1654073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f165b5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654073e20 con 0x7f1654101510 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f165b5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654073f90 con 0x7f1654100320 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f165b5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16541a0110 con 0x7f1654101ea0 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 0x7f1654073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 0x7f1654073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39250/0 (socket says 172.21.15.31:39250) 2023-07-29T16:13:21.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 -- 172.21.15.31:0/2107413364 learned_addr learned my addr 172.21.15.31:0/2107413364 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1659b9c700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1654101ea0 0x7f1654073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f165939b700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1654100320 0x7f1654074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 -- 172.21.15.31:0/2107413364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1654101ea0 msgr2=0x7f1654073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1654101ea0 0x7f1654073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 -- 172.21.15.31:0/2107413364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1654100320 msgr2=0x7f1654074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1654100320 0x7f1654074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 -- 172.21.15.31:0/2107413364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16500095f0 con 0x7f1654101510 2023-07-29T16:13:21.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.701+0000 7f1658b9a700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 0x7f1654073140 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f164400d980 tx=0x7f164400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:21.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.702+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644004220 con 0x7f1654101510 2023-07-29T16:13:21.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.702+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1644007640 con 0x7f1654101510 2023-07-29T16:13:21.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.702+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644005690 con 0x7f1654101510 2023-07-29T16:13:21.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.702+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16541a03f0 con 0x7f1654101510 2023-07-29T16:13:21.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.702+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16541a08b0 con 0x7f1654101510 2023-07-29T16:13:21.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.703+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16541072a0 con 0x7f1654101510 2023-07-29T16:13:21.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.705+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f1644004750 con 0x7f1654101510 2023-07-29T16:13:21.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.706+0000 7f164a7fc700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16400a08a0 0x7f16400a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:21.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.706+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f16440cb7e0 con 0x7f1654101510 2023-07-29T16:13:21.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.706+0000 7f165939b700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16400a08a0 0x7f16400a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:21.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.707+0000 7f165939b700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16400a08a0 0x7f16400a2d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f165000b2e0 tx=0x7f1650005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:21.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.710+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f16440950a0 con 0x7f1654101510 2023-07-29T16:13:21.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.921+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f165402ce80 con 0x7f1654101510 2023-07-29T16:13:21.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.922+0000 7f164a7fc700 1 -- 172.21.15.31:0/2107413364 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1644016020 con 0x7f1654101510 2023-07-29T16:13:21.920 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:13:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16400a08a0 msgr2=0x7f16400a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16400a08a0 0x7f16400a2d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f165000b2e0 tx=0x7f1650005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 msgr2=0x7f1654073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 0x7f1654073140 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f164400d980 tx=0x7f164400dc90 comp rx=0 tx=0).stop 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 shutdown_connections 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16400a08a0 0x7f16400a2d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1654101ea0 0x7f1654073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1654100320 0x7f1654074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 --2- 172.21.15.31:0/2107413364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1654101510 0x7f1654073140 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:21.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 >> 172.21.15.31:0/2107413364 conn(0x7f16540fb920 msgr2=0x7f16540fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:21.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 shutdown_connections 2023-07-29T16:13:21.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:21.924+0000 7f165b5ff700 1 -- 172.21.15.31:0/2107413364 wait complete. 2023-07-29T16:13:21.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:13:22.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:21 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/965813808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:22.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:21 smithi039 ceph-mon[128858]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:22.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:21 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1337704757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:22.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:21 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/965813808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:22.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:21 smithi156 ceph-mon[129245]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:22.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:21 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1337704757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 -- 172.21.15.31:0/1231365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101ea0 msgr2=0x7fe54c108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 --2- 172.21.15.31:0/1231365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101ea0 0x7fe54c108760 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe54400b0a0 tx=0x7fe54400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 -- 172.21.15.31:0/1231365296 shutdown_connections 2023-07-29T16:13:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 --2- 172.21.15.31:0/1231365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101ea0 0x7fe54c108760 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 --2- 172.21.15.31:0/1231365296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe54c100320 0x7fe54c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 --2- 172.21.15.31:0/1231365296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c101510 0x7fe54c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 -- 172.21.15.31:0/1231365296 >> 172.21.15.31:0/1231365296 conn(0x7fe54c0fb920 msgr2=0x7fe54c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:22.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 -- 172.21.15.31:0/1231365296 shutdown_connections 2023-07-29T16:13:22.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.148+0000 7fe553fff700 1 -- 172.21.15.31:0/1231365296 wait complete. 2023-07-29T16:13:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.150+0000 7fe553fff700 1 Processor -- start 2023-07-29T16:13:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.150+0000 7fe553fff700 1 -- start start 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.150+0000 7fe553fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c100320 0x7fe54c1961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.150+0000 7fe553fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101510 0x7fe54c196730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.150+0000 7fe553fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe54c101ea0 0x7fe54c19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe553fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19dd70 con 0x7fe54c101510 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe553fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19deb0 con 0x7fe54c101ea0 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe553fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c19e040 con 0x7fe54c100320 2023-07-29T16:13:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe551d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c100320 0x7fe54c1961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101510 0x7fe54c196730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe551d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c100320 0x7fe54c1961f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41364/0 (socket says 172.21.15.31:41364) 2023-07-29T16:13:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe551d9b700 1 -- 172.21.15.31:0/2296012083 learned_addr learned my addr 172.21.15.31:0/2296012083 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55259c700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe54c101ea0 0x7fe54c19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 -- 172.21.15.31:0/2296012083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c100320 msgr2=0x7fe54c1961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c100320 0x7fe54c1961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 -- 172.21.15.31:0/2296012083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe54c101ea0 msgr2=0x7fe54c19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe54c101ea0 0x7fe54c19d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 -- 172.21.15.31:0/2296012083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe544009d00 con 0x7fe54c101510 2023-07-29T16:13:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.151+0000 7fe55159a700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101510 0x7fe54c196730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00d750 tx=0x7fe53c00da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:22.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.152+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c004020 con 0x7fe54c101510 2023-07-29T16:13:22.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.152+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe53c004830 con 0x7fe54c101510 2023-07-29T16:13:22.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.152+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19e320 con 0x7fe54c101510 2023-07-29T16:13:22.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.152+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c005520 con 0x7fe54c101510 2023-07-29T16:13:22.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.152+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c19e880 con 0x7fe54c101510 2023-07-29T16:13:22.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.153+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c107870 con 0x7fe54c101510 2023-07-29T16:13:22.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.155+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fe53c004ca0 con 0x7fe54c101510 2023-07-29T16:13:22.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.156+0000 7fe542ffd700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5380a08a0 0x7fe5380a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.156+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fe53c0cb820 con 0x7fe54c101510 2023-07-29T16:13:22.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.157+0000 7fe551d9b700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5380a08a0 0x7fe5380a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.157+0000 7fe551d9b700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5380a08a0 0x7fe5380a2d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe548005950 tx=0x7fe548009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:22.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.159+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe53c0950e0 con 0x7fe54c101510 2023-07-29T16:13:22.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:21 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/965813808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:22.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:21 smithi031 ceph-mon[95504]: pgmap v304: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:22.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:21 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1337704757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:22.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.367+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe54c02cdc0 con 0x7fe54c101510 2023-07-29T16:13:22.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.367+0000 7fe542ffd700 1 -- 172.21.15.31:0/2296012083 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe53c014070 con 0x7fe54c101510 2023-07-29T16:13:22.365 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:13:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.369+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5380a08a0 msgr2=0x7fe5380a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.369+0000 7fe553fff700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5380a08a0 0x7fe5380a2d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe548005950 tx=0x7fe548009260 comp rx=0 tx=0).stop 2023-07-29T16:13:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.369+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101510 msgr2=0x7fe54c196730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.369+0000 7fe553fff700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101510 0x7fe54c196730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe53c00d750 tx=0x7fe53c00da60 comp rx=0 tx=0).stop 2023-07-29T16:13:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 shutdown_connections 2023-07-29T16:13:22.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5380a08a0 0x7fe5380a2d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe54c101ea0 0x7fe54c19d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe54c100320 0x7fe54c1961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 --2- 172.21.15.31:0/2296012083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe54c101510 0x7fe54c196730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 >> 172.21.15.31:0/2296012083 conn(0x7fe54c0fb920 msgr2=0x7fe54c1049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 shutdown_connections 2023-07-29T16:13:22.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.370+0000 7fe553fff700 1 -- 172.21.15.31:0/2296012083 wait complete. 2023-07-29T16:13:22.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:13:22.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 -- 172.21.15.31:0/2360756431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 msgr2=0x7f4a2c101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2360756431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c101980 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009990 tx=0x7f4a1c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 -- 172.21.15.31:0/2360756431 shutdown_connections 2023-07-29T16:13:22.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2360756431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a2c101ec0 0x7f4a2c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2360756431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 0x7f4a2c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2360756431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c101980 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 -- 172.21.15.31:0/2360756431 >> 172.21.15.31:0/2360756431 conn(0x7f4a2c0fb920 msgr2=0x7f4a2c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:22.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 -- 172.21.15.31:0/2360756431 shutdown_connections 2023-07-29T16:13:22.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.592+0000 7f4a33fff700 1 -- 172.21.15.31:0/2360756431 wait complete. 2023-07-29T16:13:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 Processor -- start 2023-07-29T16:13:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 -- start start 2023-07-29T16:13:22.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 0x7f4a2c196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a2c101ec0 0x7f4a2c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c19dbd0 con 0x7f4a2c101530 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c19dd10 con 0x7f4a2c100340 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.593+0000 7f4a33fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2c19dea0 con 0x7f4a2c101ec0 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39284/0 (socket says 172.21.15.31:39284) 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 -- 172.21.15.31:0/2964632268 learned_addr learned my addr 172.21.15.31:0/2964632268 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a31d9b700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 0x7f4a2c196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3259c700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a2c101ec0 0x7f4a2c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 -- 172.21.15.31:0/2964632268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a2c101ec0 msgr2=0x7f4a2c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a2c101ec0 0x7f4a2c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 -- 172.21.15.31:0/2964632268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 msgr2=0x7f4a2c196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 0x7f4a2c196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 -- 172.21.15.31:0/2964632268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c0095f0 con 0x7f4a2c101530 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a31d9b700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 0x7f4a2c196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.594+0000 7f4a3159a700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c196590 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00c010 tx=0x7f4a1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.595+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c01d070 con 0x7f4a2c101530 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.595+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a1c022470 con 0x7f4a2c101530 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.595+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c017650 con 0x7f4a2c101530 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.595+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c19e120 con 0x7f4a2c101530 2023-07-29T16:13:22.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.595+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c19e4f0 con 0x7f4a2c101530 2023-07-29T16:13:22.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.596+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c10bb70 con 0x7f4a2c101530 2023-07-29T16:13:22.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.598+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f4a1c0225e0 con 0x7f4a2c101530 2023-07-29T16:13:22.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.601+0000 7f4a22ffd700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a180a08e0 0x7f4a180a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:22.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.601+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f4a1c0ccfc0 con 0x7f4a2c101530 2023-07-29T16:13:22.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.601+0000 7f4a31d9b700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a180a08e0 0x7f4a180a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:22.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.601+0000 7f4a31d9b700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a180a08e0 0x7f4a180a2d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c101390 tx=0x7f4a28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:22.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.604+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4a1c096910 con 0x7f4a2c101530 2023-07-29T16:13:22.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.855+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a2c1071a0 con 0x7f4a2c101530 2023-07-29T16:13:22.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.856+0000 7f4a22ffd700 1 -- 172.21.15.31:0/2964632268 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f4a1c099f30 con 0x7f4a2c101530 2023-07-29T16:13:22.854 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:22.855 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:13:22.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a180a08e0 msgr2=0x7f4a180a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a180a08e0 0x7f4a180a2d90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c101390 tx=0x7f4a28009260 comp rx=0 tx=0).stop 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 msgr2=0x7f4a2c196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c196590 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00c010 tx=0x7f4a1c004730 comp rx=0 tx=0).stop 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 shutdown_connections 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a180a08e0 0x7f4a180a2d90 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a2c101ec0 0x7f4a2c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a2c100340 0x7f4a2c196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 --2- 172.21.15.31:0/2964632268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a2c101530 0x7f4a2c196590 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:22.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 >> 172.21.15.31:0/2964632268 conn(0x7f4a2c0fb920 msgr2=0x7f4a2c0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:22.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 shutdown_connections 2023-07-29T16:13:22.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:22.858+0000 7f4a33fff700 1 -- 172.21.15.31:0/2964632268 wait complete. 2023-07-29T16:13:22.858 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:13:22.875 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:23.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2107413364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:23.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2296012083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:23.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2107413364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:23.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2296012083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:23.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 -- 172.21.15.31:0/411847820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 msgr2=0x7fd3f0101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/411847820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 0x7fd3f0101980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009990 tx=0x7fd3e4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 -- 172.21.15.31:0/411847820 shutdown_connections 2023-07-29T16:13:23.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/411847820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0101ec0 0x7fd3f0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/411847820 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3f0100340 0x7fd3f0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/411847820 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 0x7fd3f0101980 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 -- 172.21.15.31:0/411847820 >> 172.21.15.31:0/411847820 conn(0x7fd3f00fb920 msgr2=0x7fd3f00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:23.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 -- 172.21.15.31:0/411847820 shutdown_connections 2023-07-29T16:13:23.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.081+0000 7fd3f4dff700 1 -- 172.21.15.31:0/411847820 wait complete. 2023-07-29T16:13:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 Processor -- start 2023-07-29T16:13:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 -- start start 2023-07-29T16:13:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 0x7fd3f0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 0x7fd3f0198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3f0101ec0 0x7fd3f019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f019fdd0 con 0x7fd3f0101530 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f019ff10 con 0x7fd3f0100340 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.082+0000 7fd3f4dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01a00a0 con 0x7fd3f0101ec0 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 0x7fd3f0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3edd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 0x7fd3f0198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3eed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3f0101ec0 0x7fd3f019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 0x7fd3f0198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50894/0 (socket says 172.21.15.31:50894) 2023-07-29T16:13:23.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 -- 172.21.15.31:0/4184147354 learned_addr learned my addr 172.21.15.31:0/4184147354 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 -- 172.21.15.31:0/4184147354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3f0101ec0 msgr2=0x7fd3f019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3f0101ec0 0x7fd3f019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 -- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 msgr2=0x7fd3f0198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 0x7fd3f0198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 -- 172.21.15.31:0/4184147354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e40095f0 con 0x7fd3f0100340 2023-07-29T16:13:23.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3ee59c700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 0x7fd3f0198250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000e970 tx=0x7fd3e000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.083+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0009790 con 0x7fd3f0100340 2023-07-29T16:13:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.084+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f01a0380 con 0x7fd3f0100340 2023-07-29T16:13:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.084+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3e000c7e0 con 0x7fd3f0100340 2023-07-29T16:13:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.084+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0003c70 con 0x7fd3f0100340 2023-07-29T16:13:23.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.084+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f01a0890 con 0x7fd3f0100340 2023-07-29T16:13:23.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.085+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f010bb70 con 0x7fd3f0100340 2023-07-29T16:13:23.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.087+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fd3e000cc50 con 0x7fd3f0100340 2023-07-29T16:13:23.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.088+0000 7fd3df7fe700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3d80a0860 0x7fd3d80a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.088+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fd3e0014070 con 0x7fd3f0100340 2023-07-29T16:13:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.088+0000 7fd3edd9b700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3d80a0860 0x7fd3d80a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.089+0000 7fd3edd9b700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3d80a0860 0x7fd3d80a2d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b2e0 tx=0x7fd3e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.092+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd3e00957c0 con 0x7fd3f0100340 2023-07-29T16:13:23.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2107413364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:23.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2296012083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:23.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.302+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3f01a0f30 con 0x7fd3f0100340 2023-07-29T16:13:23.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.304+0000 7fd3df7fe700 1 -- 172.21.15.31:0/4184147354 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22618 (secure 0 0 0) 0x7fd3e0003dd0 con 0x7fd3f0100340 2023-07-29T16:13:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:23.302 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:50.274238+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:23.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3d80a0860 msgr2=0x7fd3d80a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3d80a0860 0x7fd3d80a2d10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b2e0 tx=0x7fd3e4005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 msgr2=0x7fd3f0198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 0x7fd3f0198250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000e970 tx=0x7fd3e000ec80 comp rx=0 tx=0).stop 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 shutdown_connections 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3d80a0860 0x7fd3d80a2d10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3f0101ec0 0x7fd3f019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3f0100340 0x7fd3f0198250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 --2- 172.21.15.31:0/4184147354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3f0101530 0x7fd3f0198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.306+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 >> 172.21.15.31:0/4184147354 conn(0x7fd3f00fb920 msgr2=0x7fd3f0106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:23.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.307+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 shutdown_connections 2023-07-29T16:13:23.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.307+0000 7fd3f4dff700 1 -- 172.21.15.31:0/4184147354 wait complete. 2023-07-29T16:13:23.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:13:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.531+0000 7f739cbff700 1 -- 172.21.15.31:0/580052279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398101530 msgr2=0x7f7398101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.531+0000 7f739cbff700 1 --2- 172.21.15.31:0/580052279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398101530 0x7f7398101980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f738c009990 tx=0x7f738c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 -- 172.21.15.31:0/580052279 shutdown_connections 2023-07-29T16:13:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 --2- 172.21.15.31:0/580052279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7398101ec0 0x7f7398108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 --2- 172.21.15.31:0/580052279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398100340 0x7f7398100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 --2- 172.21.15.31:0/580052279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398101530 0x7f7398101980 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 -- 172.21.15.31:0/580052279 >> 172.21.15.31:0/580052279 conn(0x7f73980fb920 msgr2=0x7f73980fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 -- 172.21.15.31:0/580052279 shutdown_connections 2023-07-29T16:13:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.532+0000 7f739cbff700 1 -- 172.21.15.31:0/580052279 wait complete. 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 Processor -- start 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 -- start start 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 0x7f73981060c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398101530 0x7f7398106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7398101ec0 0x7f7398102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73981033f0 con 0x7f7398100340 2023-07-29T16:13:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398103560 con 0x7f7398101530 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739cbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398103720 con 0x7f7398101ec0 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 0x7f73981060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 0x7f73981060c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39322/0 (socket says 172.21.15.31:39322) 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f739659c700 1 -- 172.21.15.31:0/3548819090 learned_addr learned my addr 172.21.15.31:0/3548819090 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.533+0000 7f7395d9b700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398101530 0x7f7398106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f7396d9d700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7398101ec0 0x7f7398102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739659c700 1 -- 172.21.15.31:0/3548819090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7398101ec0 msgr2=0x7f7398102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739659c700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7398101ec0 0x7f7398102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739659c700 1 -- 172.21.15.31:0/3548819090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398101530 msgr2=0x7f7398106600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739659c700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398101530 0x7f7398106600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739659c700 1 -- 172.21.15.31:0/3548819090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0095f0 con 0x7f7398100340 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f7395d9b700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398101530 0x7f7398106600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739659c700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 0x7f73981060c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f738800e970 tx=0x7f738800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388009790 con 0x7f7398100340 2023-07-29T16:13:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73880044e0 con 0x7f7398100340 2023-07-29T16:13:23.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73981a47e0 con 0x7f7398100340 2023-07-29T16:13:23.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388003e30 con 0x7f7398100340 2023-07-29T16:13:23.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.534+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73981a4d10 con 0x7f7398100340 2023-07-29T16:13:23.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.535+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739810bb70 con 0x7f7398100340 2023-07-29T16:13:23.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.537+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f7388003f90 con 0x7f7398100340 2023-07-29T16:13:23.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.539+0000 7f73877fe700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f73800a08e0 0x7f73800a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:23.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.539+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f7388014070 con 0x7f7398100340 2023-07-29T16:13:23.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.539+0000 7f7395d9b700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f73800a08e0 0x7f73800a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:23.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.539+0000 7f7395d9b700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f73800a08e0 0x7f73800a2d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f738c00b2e0 tx=0x7f738c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:23.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.542+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f7388095920 con 0x7f7398100340 2023-07-29T16:13:23.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.800+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f739810bd90 con 0x7f7398100340 2023-07-29T16:13:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.801+0000 7f73877fe700 1 -- 172.21.15.31:0/3548819090 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f7388098f40 con 0x7f7398100340 2023-07-29T16:13:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:23.799 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:13:23.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.803+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f73800a08e0 msgr2=0x7f73800a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.803+0000 7f739cbff700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f73800a08e0 0x7f73800a2d90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f738c00b2e0 tx=0x7f738c005fd0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.803+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 msgr2=0x7f73981060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.803+0000 7f739cbff700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 0x7f73981060c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f738800e970 tx=0x7f738800ed30 comp rx=0 tx=0).stop 2023-07-29T16:13:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 shutdown_connections 2023-07-29T16:13:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f73800a08e0 0x7f73800a2d90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7398101ec0 0x7f7398102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7398100340 0x7f73981060c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 --2- 172.21.15.31:0/3548819090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7398101530 0x7f7398106600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:23.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 >> 172.21.15.31:0/3548819090 conn(0x7f73980fb920 msgr2=0x7f7398106dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:23.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 shutdown_connections 2023-07-29T16:13:23.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:23.804+0000 7f739cbff700 1 -- 172.21.15.31:0/3548819090 wait complete. 2023-07-29T16:13:23.803 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:13:23.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-29T16:12:21.912364+0000', 'modified': '2023-07-29T16:12:27.740070+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24581, 'mds_1': 24551, 'mds_2': 24550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/1642153321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 1642153321}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 1642153321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.39:6837/1297587845', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6836', 'nonce': 1297587845}, {'type': 'v1', 'addr': '172.21.15.39:6837', 'nonce': 1297587845}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/809205544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 809205544}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 809205544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.156:6839/2888902152', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 2888902152}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 2888902152}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24551': {'gid': 24551, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.31:6839/3329463675', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 3329463675}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 3329463675}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24581': {'gid': 24581, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6839/2751828361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6838', 'nonce': 2751828361}, {'type': 'v1', 'addr': '172.21.15.39:6839', 'nonce': 2751828361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-29T16:13:23.821 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-29T16:13:23.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-29T16:13:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.026+0000 7f789adff700 1 -- 172.21.15.31:0/16623196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 msgr2=0x7f7894100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.026+0000 7f789adff700 1 --2- 172.21.15.31:0/16623196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894100740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7888009990 tx=0x7f7888009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 -- 172.21.15.31:0/16623196 shutdown_connections 2023-07-29T16:13:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 --2- 172.21.15.31:0/16623196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7894101eb0 0x7f7894108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 --2- 172.21.15.31:0/16623196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894100740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 --2- 172.21.15.31:0/16623196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7894101520 0x7f7894101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 -- 172.21.15.31:0/16623196 >> 172.21.15.31:0/16623196 conn(0x7f78940fb970 msgr2=0x7f78940fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 -- 172.21.15.31:0/16623196 shutdown_connections 2023-07-29T16:13:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.027+0000 7f789adff700 1 -- 172.21.15.31:0/16623196 wait complete. 2023-07-29T16:13:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 Processor -- start 2023-07-29T16:13:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 -- start start 2023-07-29T16:13:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7894101520 0x7f78941965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7894101eb0 0x7f789419d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789419dc00 con 0x7f7894100330 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789419dd40 con 0x7f7894101eb0 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f789adff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789419ded0 con 0x7f7894101520 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39350/0 (socket says 172.21.15.31:39350) 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.028+0000 7f7898b9b700 1 -- 172.21.15.31:0/891153941 learned_addr learned my addr 172.21.15.31:0/891153941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f789939c700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7894101eb0 0x7f789419d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7893fff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7894101520 0x7f78941965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7898b9b700 1 -- 172.21.15.31:0/891153941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7894101520 msgr2=0x7f78941965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7898b9b700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7894101520 0x7f78941965c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7898b9b700 1 -- 172.21.15.31:0/891153941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7894101eb0 msgr2=0x7f789419d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7898b9b700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7894101eb0 0x7f789419d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7898b9b700 1 -- 172.21.15.31:0/891153941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7880009520 con 0x7f7894100330 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7898b9b700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894196080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7888005740 tx=0x7f7888009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7891ffb700 1 -- 172.21.15.31:0/891153941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788801d070 con 0x7f7894100330 2023-07-29T16:13:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7891ffb700 1 -- 172.21.15.31:0/891153941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7888021410 con 0x7f7894100330 2023-07-29T16:13:24.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78880095f0 con 0x7f7894100330 2023-07-29T16:13:24.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f7891ffb700 1 -- 172.21.15.31:0/891153941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888017650 con 0x7f7894100330 2023-07-29T16:13:24.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.029+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789419e3e0 con 0x7f7894100330 2023-07-29T16:13:24.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.033+0000 7f7891ffb700 1 -- 172.21.15.31:0/891153941 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f7888021810 con 0x7f7894100330 2023-07-29T16:13:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.034+0000 7f78ae721b80 1 Processor -- start 2023-07-29T16:13:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.034+0000 7f7891ffb700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f78840a0820 0x7f78840a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.034+0000 7f7891ffb700 1 -- 172.21.15.31:0/891153941 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f78880cd170 con 0x7f7894100330 2023-07-29T16:13:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.034+0000 7f78ae721b80 1 -- start start 2023-07-29T16:13:24.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.034+0000 7f789939c700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f78840a0820 0x7f78840a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.034+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c1fa0 0x5579966c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966c3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f78ae721b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579966159c0 con 0x5579966c36b0 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f78ae721b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557996613380 con 0x5579966c1fa0 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f78ae721b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55799660ec70 con 0x5579966c28f0 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f789939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c1fa0 0x5579966c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f789939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39364/0 (socket says 172.21.15.31:39364) 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f789939c700 1 -- 172.21.15.31:0/927390180 learned_addr learned my addr 172.21.15.31:0/927390180 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7893fff700 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966c3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f789939c700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f78840a0820 0x7f78840a2cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f788c0093f0 tx=0x7f788c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7898b9b700 1 -- 172.21.15.31:0/927390180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 msgr2=0x5579966c3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7898b9b700 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966c3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7898b9b700 1 -- 172.21.15.31:0/927390180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 msgr2=0x5579966c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7898b9b700 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.035+0000 7f7898b9b700 1 -- 172.21.15.31:0/927390180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55799643cd90 con 0x5579966c1fa0 2023-07-29T16:13:24.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.036+0000 7f7898b9b700 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c1fa0 0x5579966c23b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f78880caaa0 tx=0x7f7888096bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b29a43549128e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.036+0000 7f787effd700 1 -- 172.21.15.31:0/927390180 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788809a780 con 0x5579966c1fa0 2023-07-29T16:13:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.036+0000 7f787effd700 1 -- 172.21.15.31:0/927390180 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f788809e7b0 con 0x5579966c1fa0 2023-07-29T16:13:24.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.036+0000 7f787effd700 1 -- 172.21.15.31:0/927390180 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788809be40 con 0x5579966c1fa0 2023-07-29T16:13:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.036+0000 7f78ae721b80 1 -- 172.21.15.31:0/927390180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c1fa0 msgr2=0x5579966c23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.036+0000 7f78ae721b80 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c1fa0 0x5579966c23b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f78880caaa0 tx=0x7f7888096bf0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 -- 172.21.15.31:0/927390180 shutdown_connections 2023-07-29T16:13:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c1fa0 0x5579966c23b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 --2- 172.21.15.31:0/927390180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966c3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 -- 172.21.15.31:0/927390180 >> 172.21.15.31:0/927390180 conn(0x5579966b3390 msgr2=0x5579966b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 -- 172.21.15.31:0/927390180 shutdown_connections 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 -- 172.21.15.31:0/927390180 wait complete. 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 -- start start 2023-07-29T16:13:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.037+0000 7f78ae721b80 1 Processor -- start 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c36b0 0x5579966d2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579965ab4c0 con 0x5579966c1fa0 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579965ab160 con 0x5579966c36b0 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579964928d0 con 0x5579966c28f0 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f78ae721b80 10 client.0 _ll_register_callbacks cb 0x5579965b45d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966cb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39370/0 (socket says 172.21.15.31:39370) 2023-07-29T16:13:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f7898b9b700 1 -- 172.21.15.31:0/2155653832 learned_addr learned my addr 172.21.15.31:0/2155653832 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f789939c700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c36b0 0x5579966d2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f7893fff700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.038+0000 7f7898b9b700 1 -- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 msgr2=0x5579966cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966cbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7898b9b700 1 -- 172.21.15.31:0/2155653832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c36b0 msgr2=0x5579966d2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c36b0 0x5579966d2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7898b9b700 1 -- 172.21.15.31:0/2155653832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789419e3e0 con 0x5579966c1fa0 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f789939c700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c36b0 0x5579966d2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966cb880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7888096c80 tx=0x7f7888005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f26d4bd201d215 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7873fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7873fff700 1 -- 172.21.15.31:0/2155653832 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888099c70 con 0x5579966c1fa0 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7873fff700 1 -- 172.21.15.31:0/2155653832 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f7888099dd0 con 0x5579966c1fa0 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557996639cc0 con 0x5579966c1fa0 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7873fff700 1 -- 172.21.15.31:0/2155653832 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78880c4bd0 con 0x5579966c1fa0 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f78ae721b80 10 client.14985 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557996425c10 con 0x5579966c1fa0 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.039+0000 7f7873fff700 1 -- 172.21.15.31:0/2155653832 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557996639cc0 con 0x5579966c1fa0 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f7873fff700 1 -- 172.21.15.31:0/2155653832 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f7888096570 con 0x5579966c1fa0 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f78ae721b80 10 client.14985 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f78ae721b80 10 client.14985 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f78ae721b80 20 client.14985 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5579966d6570 0x5579966d5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f78ae721b80 4 client.14985 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55799668cd20 con 0x5579966d6570 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.040+0000 7f7893fff700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5579966d6570 0x5579966d5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.041+0000 7f7893fff700 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5579966d6570 0x5579966d5d30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32c1da49debdaec1 server_cookie=e69f953dd3835af9 in_seq=0 out_seq=0 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.041+0000 7f7873fff700 10 client.14985 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f7873fff700 1 -- 172.21.15.31:0/2155653832 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55799668cd20 con 0x5579966d6570 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f7873fff700 10 client.14985 handle_command_reply: tid=0 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 client.14985 shutdown 2023-07-29T16:13:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 msgr2=0x5579966cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966cb880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7888096c80 tx=0x7f7888005820 comp rx=0 tx=0).stop 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 shutdown_connections 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5579966d6570 0x5579966d5d30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c36b0 0x5579966d2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966cb880 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2155653832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c28f0 0x5579966cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 >> 172.21.15.31:0/2155653832 conn(0x5579966b3390 msgr2=0x5579966b8010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f7873fff700 0 client.14985 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:24.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 shutdown_connections 2023-07-29T16:13:24.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 1 -- 172.21.15.31:0/2155653832 wait complete. 2023-07-29T16:13:24.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.042+0000 7f78ae721b80 20 client.14985 trim_cache size 0 max 16384 2023-07-29T16:13:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.060+0000 7f78ae721b80 1 Processor -- start 2023-07-29T16:13:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.060+0000 7f78ae721b80 1 -- start start 2023-07-29T16:13:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.060+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c1fa0 0x5579966c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.060+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f78ae721b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55799668cd20 con 0x5579966c36b0 2023-07-29T16:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f78ae721b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557996639cc0 con 0x5579966c28f0 2023-07-29T16:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f78ae721b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579965d90b0 con 0x5579966c1fa0 2023-07-29T16:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39384/0 (socket says 172.21.15.31:39384) 2023-07-29T16:13:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c1fa0 0x5579966c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 -- 172.21.15.31:0/2777329082 learned_addr learned my addr 172.21.15.31:0/2777329082 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f7893fff700 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 -- 172.21.15.31:0/2777329082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c1fa0 msgr2=0x5579966c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c1fa0 0x5579966c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 -- 172.21.15.31:0/2777329082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 msgr2=0x5579966df2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 -- 172.21.15.31:0/2777329082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557996463200 con 0x5579966c36b0 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.061+0000 7f789939c700 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c65f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f788c017b50 tx=0x7f788c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3416b2b02c006f9a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f787f7fe700 1 -- 172.21.15.31:0/2777329082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c00e050 con 0x5579966c36b0 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f787f7fe700 1 -- 172.21.15.31:0/2777329082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f788c00f040 con 0x5579966c36b0 2023-07-29T16:13:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f787f7fe700 1 -- 172.21.15.31:0/2777329082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c024d00 con 0x5579966c36b0 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 -- 172.21.15.31:0/2777329082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 msgr2=0x5579966c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c65f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f788c017b50 tx=0x7f788c01b000 comp rx=0 tx=0).stop 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 -- 172.21.15.31:0/2777329082 shutdown_connections 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c36b0 0x5579966c65f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c1fa0 0x5579966c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2777329082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 -- 172.21.15.31:0/2777329082 >> 172.21.15.31:0/2777329082 conn(0x5579966b3390 msgr2=0x5579966ca5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 -- 172.21.15.31:0/2777329082 shutdown_connections 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.062+0000 7f78ae721b80 1 -- 172.21.15.31:0/2777329082 wait complete. 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 -- start start 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 Processor -- start 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c36b0 0x557996728170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557996728880 con 0x5579966c1fa0 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579967289f0 con 0x5579966c28f0 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557996728bb0 con 0x5579966c36b0 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.063+0000 7f78ae721b80 10 client.0 _ll_register_callbacks cb 0x5579965b45d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966b8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39390/0 (socket says 172.21.15.31:39390) 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 -- 172.21.15.31:0/2306276375 learned_addr learned my addr 172.21.15.31:0/2306276375 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f789939c700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c36b0 0x557996728170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7893fff700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 -- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c36b0 msgr2=0x557996728170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c36b0 0x557996728170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 -- 172.21.15.31:0/2306276375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 msgr2=0x5579966df2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 -- 172.21.15.31:0/2306276375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c017800 con 0x5579966c1fa0 2023-07-29T16:13:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7893fff700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7898b9b700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966b8b10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7888098ad0 tx=0x7f7888096ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5645f3b6bd27c9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7873fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7873fff700 1 -- 172.21.15.31:0/2306276375 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7888009290 con 0x5579966c1fa0 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f7873fff700 1 -- 172.21.15.31:0/2306276375 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f78880a26b0 con 0x5579966c1fa0 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.064+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f788809e7b0 con 0x5579966c1fa0 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f7873fff700 1 -- 172.21.15.31:0/2306276375 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7888096570 con 0x5579966c1fa0 2023-07-29T16:13:24.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 10 client.14991 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55799643b260 con 0x5579966c1fa0 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f7873fff700 1 -- 172.21.15.31:0/2306276375 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78880a6070 con 0x5579966c1fa0 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f7873fff700 1 -- 172.21.15.31:0/2306276375 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f7888099d90 con 0x5579966c1fa0 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 10 client.14991 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 10 client.14991 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 20 client.14991 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55799672d390 0x557996729e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 4 client.14991 mds_command: new command op to 24550 tid=0[{"prefix": "status"}] 2023-07-29T16:13:24.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.065+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557996607cf0 con 0x55799672d390 2023-07-29T16:13:24.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f7893fff700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55799672d390 0x557996729e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2964632268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:24.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:23 smithi156 ceph-mon[129245]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:24.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/4184147354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:24.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f7893fff700 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55799672d390 0x557996729e00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8358829642a06c3 server_cookie=76b6295007cda323 in_seq=0 out_seq=0 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f7873fff700 10 client.14991 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f7873fff700 1 -- 172.21.15.31:0/2306276375 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x557996607cf0 con 0x55799672d390 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f7873fff700 10 client.14991 handle_command_reply: tid=0 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f78ae721b80 1 client.14991 shutdown 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 msgr2=0x5579966b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.066+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966b8b10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7888098ad0 tx=0x7f7888096ac0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 shutdown_connections 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5579966c36b0 0x557996728170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5579966c1fa0 0x5579966b8b10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55799672d390 0x557996729e00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 --2- 172.21.15.31:0/2306276375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5579966c28f0 0x5579966df2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 >> 172.21.15.31:0/2306276375 conn(0x5579966b3390 msgr2=0x5579966c9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f7873fff700 0 client.14991 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:24.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 shutdown_connections 2023-07-29T16:13:24.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 1 -- 172.21.15.31:0/2306276375 wait complete. 2023-07-29T16:13:24.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f78ae721b80 20 client.14991 trim_cache size 0 max 16384 2023-07-29T16:13:24.075 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:13:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:13:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2023-07-29T16:13:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24550, 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 57.585475655000003, 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 88, 2023-07-29T16:13:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 86, 2023-07-29T16:13:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 74.644971302000002 2023-07-29T16:13:24.077 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:13:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f78840a0820 msgr2=0x7f78840a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f78840a0820 0x7f78840a2cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f788c0093f0 tx=0x7f788c009920 comp rx=0 tx=0).stop 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 msgr2=0x7f7894196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894196080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7888005740 tx=0x7f7888009510 comp rx=0 tx=0).stop 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 shutdown_connections 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f78840a0820 0x7f78840a2cd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7894101eb0 0x7f789419d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7894100330 0x7f7894196080 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 --2- 172.21.15.31:0/891153941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7894101520 0x7f78941965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.067+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 >> 172.21.15.31:0/891153941 conn(0x7f78940fb970 msgr2=0x7f78940fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.068+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 shutdown_connections 2023-07-29T16:13:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.068+0000 7f789adff700 1 -- 172.21.15.31:0/891153941 wait complete. 2023-07-29T16:13:24.081 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2964632268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:24.081 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:23 smithi031 ceph-mon[95504]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:24.082 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/4184147354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:24.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2023-07-29T16:13:24.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2964632268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:24.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:23 smithi039 ceph-mon[128858]: pgmap v305: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:24.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/4184147354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:24.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 -- 172.21.15.31:0/2449909610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 msgr2=0x7f965810a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 --2- 172.21.15.31:0/2449909610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965810a660 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f965000b0a0 tx=0x7f965000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 -- 172.21.15.31:0/2449909610 shutdown_connections 2023-07-29T16:13:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 --2- 172.21.15.31:0/2449909610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965810a660 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 --2- 172.21.15.31:0/2449909610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9658074d90 0x7f96580731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 --2- 172.21.15.31:0/2449909610 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96580737b0 0x7f9658073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 -- 172.21.15.31:0/2449909610 >> 172.21.15.31:0/2449909610 conn(0x7f96580fb920 msgr2=0x7f96580fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 -- 172.21.15.31:0/2449909610 shutdown_connections 2023-07-29T16:13:24.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.288+0000 7f965efff700 1 -- 172.21.15.31:0/2449909610 wait complete. 2023-07-29T16:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.289+0000 7f965efff700 1 Processor -- start 2023-07-29T16:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 -- start start 2023-07-29T16:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96580737b0 0x7f9658198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9658074d90 0x7f96581988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965819f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965819fef0 con 0x7f96581081e0 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96581a0030 con 0x7f9658074d90 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96581a01c0 con 0x7f96580737b0 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965819f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965819f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39416/0 (socket says 172.21.15.31:39416) 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965d59c700 1 -- 172.21.15.31:0/4130729784 learned_addr learned my addr 172.21.15.31:0/4130729784 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f9657fff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9658074d90 0x7f96581988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.290+0000 7f965cd9b700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96580737b0 0x7f9658198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965d59c700 1 -- 172.21.15.31:0/4130729784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96580737b0 msgr2=0x7f9658198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965d59c700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96580737b0 0x7f9658198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965d59c700 1 -- 172.21.15.31:0/4130729784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9658074d90 msgr2=0x7f96581988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965d59c700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9658074d90 0x7f96581988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965d59c700 1 -- 172.21.15.31:0/4130729784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9650009d00 con 0x7f96581081e0 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965d59c700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965819f920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9650009fd0 tx=0x7f965000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f9655ffb700 1 -- 172.21.15.31:0/4130729784 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965000e040 con 0x7f96581081e0 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f9655ffb700 1 -- 172.21.15.31:0/4130729784 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96500094a0 con 0x7f96581081e0 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f9655ffb700 1 -- 172.21.15.31:0/4130729784 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965001b6c0 con 0x7f96581081e0 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96581a0440 con 0x7f96581081e0 2023-07-29T16:13:24.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.291+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96581a0810 con 0x7f96581081e0 2023-07-29T16:13:24.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.294+0000 7f9655ffb700 1 -- 172.21.15.31:0/4130729784 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f965001d470 con 0x7f96581081e0 2023-07-29T16:13:24.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 Processor -- start 2023-07-29T16:13:24.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f9655ffb700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96480a08a0 0x7f96480a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 -- start start 2023-07-29T16:13:24.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f9655ffb700 1 -- 172.21.15.31:0/4130729784 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f96500ccdc0 con 0x7f96581081e0 2023-07-29T16:13:24.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f965cd9b700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96480a08a0 0x7f96480a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf0199c0 con 0x5624cf0c1b90 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf017380 con 0x5624cf0c32a0 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f967290eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf012c70 con 0x5624cf0c24e0 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f965cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f965cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39432/0 (socket says 172.21.15.31:39432) 2023-07-29T16:13:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f965cd9b700 1 -- 172.21.15.31:0/1433328420 learned_addr learned my addr 172.21.15.31:0/1433328420 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.296+0000 7f9657fff700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965d59c700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96480a08a0 0x7f96480a2d50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f964c013650 tx=0x7f964c013a10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 -- 172.21.15.31:0/1433328420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 msgr2=0x5624cf0c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0c2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 -- 172.21.15.31:0/1433328420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 msgr2=0x5624cf0c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 -- 172.21.15.31:0/1433328420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624cee40d90 con 0x5624cf0c1b90 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965d59c700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f9657fff700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f965cd9b700 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c1fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f964c009c00 tx=0x7f964c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b010b5ebf1ef1c02 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f9642ffd700 1 -- 172.21.15.31:0/1433328420 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c020e60 con 0x5624cf0c1b90 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.297+0000 7f9642ffd700 1 -- 172.21.15.31:0/1433328420 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f964c0044f0 con 0x5624cf0c1b90 2023-07-29T16:13:24.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f9642ffd700 1 -- 172.21.15.31:0/1433328420 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c004ea0 con 0x5624cf0c1b90 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 -- 172.21.15.31:0/1433328420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 msgr2=0x5624cf0c1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c1fa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f964c009c00 tx=0x7f964c020000 comp rx=0 tx=0).stop 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 -- 172.21.15.31:0/1433328420 shutdown_connections 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c1fa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 --2- 172.21.15.31:0/1433328420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 -- 172.21.15.31:0/1433328420 >> 172.21.15.31:0/1433328420 conn(0x5624cf0b7390 msgr2=0x5624cf0bc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 -- 172.21.15.31:0/1433328420 shutdown_connections 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.298+0000 7f967290eb80 1 -- 172.21.15.31:0/1433328420 wait complete. 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.299+0000 7f967290eb80 1 -- start start 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.299+0000 7f967290eb80 1 Processor -- start 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c1b90 0x5624cf0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 0x5624cf0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0d2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cefaf4c0 con 0x5624cf0c24e0 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cefaf160 con 0x5624cf0c32a0 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cee968d0 con 0x5624cf0c1b90 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f967290eb80 10 client.0 _ll_register_callbacks cb 0x5624cefb85d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f9657fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 0x5624cf0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f9657fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 0x5624cf0cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39434/0 (socket says 172.21.15.31:39434) 2023-07-29T16:13:24.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f9657fff700 1 -- 172.21.15.31:0/1728616984 learned_addr learned my addr 172.21.15.31:0/1728616984 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f965d59c700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0d2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.300+0000 7f965cd9b700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c1b90 0x5624cf0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9657fff700 1 -- 172.21.15.31:0/1728616984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c1b90 msgr2=0x5624cf0cb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9657fff700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c1b90 0x5624cf0cb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9657fff700 1 -- 172.21.15.31:0/1728616984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 msgr2=0x5624cf0d2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9657fff700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0d2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9657fff700 1 -- 172.21.15.31:0/1728616984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96440098e0 con 0x5624cf0c24e0 2023-07-29T16:13:24.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f965d59c700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0d2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9657fff700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 0x5624cf0cbd70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9644012960 tx=0x7f9644012c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da2002896ea7de8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9637fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9637fff700 1 -- 172.21.15.31:0/1728616984 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644017070 con 0x5624cf0c24e0 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9637fff700 1 -- 172.21.15.31:0/1728616984 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f9644012190 con 0x5624cf0c24e0 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624cf03dcc0 con 0x5624cf0c24e0 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9637fff700 1 -- 172.21.15.31:0/1728616984 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f964400f7d0 con 0x5624cf0c24e0 2023-07-29T16:13:24.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f967290eb80 10 client.15003 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f96581a0810 con 0x5624cf0c24e0 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.301+0000 7f9637fff700 1 -- 172.21.15.31:0/1728616984 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5624cf03dcc0 con 0x5624cf0c24e0 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f9637fff700 1 -- 172.21.15.31:0/1728616984 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9644024020 con 0x5624cf0c24e0 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f967290eb80 10 client.15003 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f967290eb80 10 client.15003 resolve_mds: resolved name 'g' to daemon mds.g 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f967290eb80 20 client.15003 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:24.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f967290eb80 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf0d6b10 0x5624cf0d5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f967290eb80 4 client.15003 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:13:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624cf090d20 con 0x5624cf0d6b10 2023-07-29T16:13:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.302+0000 7f965cd9b700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf0d6b10 0x5624cf0d5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.303+0000 7f9637fff700 10 client.15003 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.303+0000 7f965cd9b700 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf0d6b10 0x5624cf0d5ec0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa2f9da22f718b3d server_cookie=db50af272ef7495f in_seq=0 out_seq=0 2023-07-29T16:13:24.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.303+0000 7f9637fff700 1 -- 172.21.15.31:0/1728616984 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5624cf090d20 con 0x5624cf0d6b10 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.303+0000 7f9637fff700 10 client.15003 handle_command_reply: tid=0 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 client.15003 shutdown 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 msgr2=0x5624cf0cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 0x5624cf0cbd70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9644012960 tx=0x7f9644012c70 comp rx=0 tx=0).stop 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 shutdown_connections 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf0d6b10 0x5624cf0d5ec0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c32a0 0x5624cf0d2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c1b90 0x5624cf0cb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f9637fff700 0 client.15003 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 --2- 172.21.15.31:0/1728616984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c24e0 0x5624cf0cbd70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 >> 172.21.15.31:0/1728616984 conn(0x5624cf0b7390 msgr2=0x5624cf0bc010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 shutdown_connections 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 1 -- 172.21.15.31:0/1728616984 wait complete. 2023-07-29T16:13:24.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.304+0000 7f967290eb80 20 client.15003 trim_cache size 0 max 16384 2023-07-29T16:13:24.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 Processor -- start 2023-07-29T16:13:24.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 -- start start 2023-07-29T16:13:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c24e0 0x5624cf0df860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c32a0 0x5624cf0bc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cefdd0b0 con 0x5624cf0c1b90 2023-07-29T16:13:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf017cf0 con 0x5624cf0c24e0 2023-07-29T16:13:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.322+0000 7f967290eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ceea2de0 con 0x5624cf0c32a0 2023-07-29T16:13:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39442/0 (socket says 172.21.15.31:39442) 2023-07-29T16:13:24.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 -- 172.21.15.31:0/4107776513 learned_addr learned my addr 172.21.15.31:0/4107776513 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f9657fff700 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c24e0 0x5624cf0df860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965d59c700 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c32a0 0x5624cf0bc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 -- 172.21.15.31:0/4107776513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c32a0 msgr2=0x5624cf0bc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c32a0 0x5624cf0bc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 -- 172.21.15.31:0/4107776513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c24e0 msgr2=0x5624cf0df860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c24e0 0x5624cf0df860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 -- 172.21.15.31:0/4107776513 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c020540 con 0x5624cf0c1b90 2023-07-29T16:13:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f965cd9b700 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c96b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f964c009900 tx=0x7f964c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e8ff53ce02da05 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f96437fe700 1 -- 172.21.15.31:0/4107776513 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c020de0 con 0x5624cf0c1b90 2023-07-29T16:13:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f96437fe700 1 -- 172.21.15.31:0/4107776513 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f964c005500 con 0x5624cf0c1b90 2023-07-29T16:13:24.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.323+0000 7f96437fe700 1 -- 172.21.15.31:0/4107776513 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c034c00 con 0x5624cf0c1b90 2023-07-29T16:13:24.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 -- 172.21.15.31:0/4107776513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 msgr2=0x5624cf0c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c96b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f964c009900 tx=0x7f964c009260 comp rx=0 tx=0).stop 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 -- 172.21.15.31:0/4107776513 shutdown_connections 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c32a0 0x5624cf0bc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c1b90 0x5624cf0c96b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 --2- 172.21.15.31:0/4107776513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c24e0 0x5624cf0df860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 -- 172.21.15.31:0/4107776513 >> 172.21.15.31:0/4107776513 conn(0x5624cf0b7390 msgr2=0x5624cf0bd640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 -- 172.21.15.31:0/4107776513 shutdown_connections 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.324+0000 7f967290eb80 1 -- 172.21.15.31:0/4107776513 wait complete. 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 -- start start 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 Processor -- start 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c1b90 0x5624cf0daad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0db010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 0x5624cf128660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf0db740 con 0x5624cf0c32a0 2023-07-29T16:13:24.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf128bd0 con 0x5624cf0c1b90 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624cf128d90 con 0x5624cf0c24e0 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.325+0000 7f967290eb80 10 client.0 _ll_register_callbacks cb 0x5624cefb85d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 0x5624cf128660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 0x5624cf128660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39456/0 (socket says 172.21.15.31:39456) 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 -- 172.21.15.31:0/299148588 learned_addr learned my addr 172.21.15.31:0/299148588 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965cd9b700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c1b90 0x5624cf0daad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f9657fff700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0db010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 -- 172.21.15.31:0/299148588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 msgr2=0x5624cf0db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0db010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 -- 172.21.15.31:0/299148588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c1b90 msgr2=0x5624cf0daad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c1b90 0x5624cf0daad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 -- 172.21.15.31:0/299148588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624cee40d90 con 0x5624cf0c32a0 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f965d59c700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 0x5624cf128660 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9650009cd0 tx=0x7f96500092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172aee572a600536 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f9637fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:24.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f9637fff700 1 -- 172.21.15.31:0/299148588 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f965009a830 con 0x5624cf0c32a0 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.326+0000 7f9637fff700 1 -- 172.21.15.31:0/299148588 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f965009e400 con 0x5624cf0c32a0 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f964c0044f0 con 0x5624cf0c32a0 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f9637fff700 1 -- 172.21.15.31:0/299148588 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96500a2790 con 0x5624cf0c32a0 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 10 client.15009 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5624cee29c10 con 0x5624cf0c32a0 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f9637fff700 1 -- 172.21.15.31:0/299148588 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f964c0044f0 con 0x5624cf0c32a0 2023-07-29T16:13:24.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f9637fff700 1 -- 172.21.15.31:0/299148588 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f96500b2d20 con 0x5624cf0c32a0 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 10 client.15009 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 10 client.15009 resolve_mds: resolved name 'g' to daemon mds.g 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 20 client.15009 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf12da40 0x5624cf12a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 4 client.15009 mds_command: new command op to 24551 tid=0[{"prefix": "status"}] 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5624cf00bcf0 con 0x5624cf12da40 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.327+0000 7f965cd9b700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf12da40 0x5624cf12a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f965cd9b700 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf12da40 0x5624cf12a3c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=456e292ef0ec4267 server_cookie=6b9d1b9a808bc0ca in_seq=0 out_seq=0 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f9637fff700 10 client.15009 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f9637fff700 1 -- 172.21.15.31:0/299148588 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5624cf00bcf0 con 0x5624cf12da40 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f9637fff700 10 client.15009 handle_command_reply: tid=0 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 client.15009 shutdown 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 msgr2=0x5624cf128660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 0x5624cf128660 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9650009cd0 tx=0x7f96500092e0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 shutdown_connections 2023-07-29T16:13:24.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5624cf12da40 0x5624cf12a3c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624cf0c32a0 0x5624cf128660 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f967290eb80 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624cf0c1b90 0x5624cf0daad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.328+0000 7f9637fff700 0 client.15009 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f967290eb80 1 --2- 172.21.15.31:0/299148588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624cf0c24e0 0x5624cf0db010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 >> 172.21.15.31:0/299148588 conn(0x5624cf0b7390 msgr2=0x5624cf0d5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 shutdown_connections 2023-07-29T16:13:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f967290eb80 1 -- 172.21.15.31:0/299148588 wait complete. 2023-07-29T16:13:24.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f967290eb80 20 client.15009 trim_cache size 0 max 16384 2023-07-29T16:13:24.336 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:13:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:13:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2023-07-29T16:13:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24551, 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 58.864428279999998, 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 88, 2023-07-29T16:13:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 86, 2023-07-29T16:13:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 71.941149314 2023-07-29T16:13:24.338 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:13:24.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96480a08a0 msgr2=0x7f96480a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96480a08a0 0x7f96480a2d50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f964c013650 tx=0x7f964c013a10 comp rx=0 tx=0).stop 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 msgr2=0x7f965819f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965819f920 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9650009fd0 tx=0x7f965000bce0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 shutdown_connections 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96480a08a0 0x7f96480a2d50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96581081e0 0x7f965819f920 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96580737b0 0x7f9658198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 --2- 172.21.15.31:0/4130729784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9658074d90 0x7f96581988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.329+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 >> 172.21.15.31:0/4130729784 conn(0x7f96580fb920 msgr2=0x7f96580fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.330+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 shutdown_connections 2023-07-29T16:13:24.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.330+0000 7f965efff700 1 -- 172.21.15.31:0/4130729784 wait complete. 2023-07-29T16:13:24.345 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-29T16:13:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 -- 172.21.15.31:0/3804706463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 msgr2=0x7f53d0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3804706463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 0x7f53d0108770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009f90 tx=0x7f53c800b330 comp rx=0 tx=0).stop 2023-07-29T16:13:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 -- 172.21.15.31:0/3804706463 shutdown_connections 2023-07-29T16:13:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3804706463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 0x7f53d0108770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3804706463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d0100330 0x7f53d0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3804706463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 -- 172.21.15.31:0/3804706463 >> 172.21.15.31:0/3804706463 conn(0x7f53d00fb970 msgr2=0x7f53d00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.551+0000 7f53d7fff700 1 -- 172.21.15.31:0/3804706463 shutdown_connections 2023-07-29T16:13:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.552+0000 7f53d7fff700 1 -- 172.21.15.31:0/3804706463 wait complete. 2023-07-29T16:13:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.552+0000 7f53d7fff700 1 Processor -- start 2023-07-29T16:13:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 -- start start 2023-07-29T16:13:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d0100330 0x7f53d0074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 0x7f53d0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0073e20 con 0x7f53d0101520 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d0073f90 con 0x7f53d0101eb0 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d7fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d01a01c0 con 0x7f53d0100330 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39468/0 (socket says 172.21.15.31:39468) 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d559a700 1 -- 172.21.15.31:0/3266283378 learned_addr learned my addr 172.21.15.31:0/3266283378 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d659c700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 0x7f53d0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.553+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d0100330 0x7f53d0074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d559a700 1 -- 172.21.15.31:0/3266283378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d0100330 msgr2=0x7f53d0074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d559a700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d0100330 0x7f53d0074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d559a700 1 -- 172.21.15.31:0/3266283378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 msgr2=0x7f53d0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d559a700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 0x7f53d0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d559a700 1 -- 172.21.15.31:0/3266283378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c8009c40 con 0x7f53d0101520 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d559a700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0073140 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f53c000e9c0 tx=0x7f53c000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53c6ffd700 1 -- 172.21.15.31:0/3266283378 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c0004a30 con 0x7f53d0101520 2023-07-29T16:13:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53c6ffd700 1 -- 172.21.15.31:0/3266283378 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53c000cc80 con 0x7f53d0101520 2023-07-29T16:13:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53c6ffd700 1 -- 172.21.15.31:0/3266283378 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00054e0 con 0x7f53d0101520 2023-07-29T16:13:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d01a04a0 con 0x7f53d0101520 2023-07-29T16:13:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.554+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d01a0960 con 0x7f53d0101520 2023-07-29T16:13:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.557+0000 7f53c6ffd700 1 -- 172.21.15.31:0/3266283378 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f53c0004b90 con 0x7f53d0101520 2023-07-29T16:13:24.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 Processor -- start 2023-07-29T16:13:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 -- start start 2023-07-29T16:13:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53c6ffd700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53bc0a0820 0x7f53bc0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc61fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53c6ffd700 1 -- 172.21.15.31:0/3266283378 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f53c0014070 con 0x7f53d0101520 2023-07-29T16:13:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc62d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc636f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bbb99c0 con 0x559e6bc624e0 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bbb7380 con 0x559e6bc632a0 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53eba07b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bbb2c70 con 0x559e6bc61b90 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.559+0000 7f53d659c700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53bc0a0820 0x7f53bc0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc61fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc636f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc61fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41552/0 (socket says 172.21.15.31:41552) 2023-07-29T16:13:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 -- 172.21.15.31:0/3378636333 learned_addr learned my addr 172.21.15.31:0/3378636333 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d559a700 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc62d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 -- 172.21.15.31:0/3378636333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 msgr2=0x559e6bc636f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc636f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 -- 172.21.15.31:0/3378636333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 msgr2=0x559e6bc62d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc62d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 -- 172.21.15.31:0/3378636333 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e6b9e0d90 con 0x559e6bc61b90 2023-07-29T16:13:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d659c700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53bc0a0820 0x7f53bc0a2cd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53d0075160 tx=0x7f53c8007420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.560+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc61fa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009d30 tx=0x7f53cc011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1948e7239d5a97ff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53bbfff700 1 -- 172.21.15.31:0/3378636333 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc004220 con 0x559e6bc61b90 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53bbfff700 1 -- 172.21.15.31:0/3378636333 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53cc004380 con 0x559e6bc61b90 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53bbfff700 1 -- 172.21.15.31:0/3378636333 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc0197a0 con 0x559e6bc61b90 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 -- 172.21.15.31:0/3378636333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 msgr2=0x559e6bc61fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc61fa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009d30 tx=0x7f53cc011720 comp rx=0 tx=0).stop 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 -- 172.21.15.31:0/3378636333 shutdown_connections 2023-07-29T16:13:24.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc636f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc61fa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 --2- 172.21.15.31:0/3378636333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc62d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.561+0000 7f53eba07b80 1 -- 172.21.15.31:0/3378636333 >> 172.21.15.31:0/3378636333 conn(0x559e6bc57390 msgr2=0x559e6bc5c940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.562+0000 7f53eba07b80 1 -- 172.21.15.31:0/3378636333 shutdown_connections 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.562+0000 7f53eba07b80 1 -- 172.21.15.31:0/3378636333 wait complete. 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.562+0000 7f53eba07b80 1 -- start start 2023-07-29T16:13:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.562+0000 7f53eba07b80 1 Processor -- start 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc61b90 0x559e6bc6b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc6bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc632a0 0x559e6bc72cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bb4f4c0 con 0x559e6bc624e0 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bb4f160 con 0x559e6bc61b90 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6ba368d0 con 0x559e6bc632a0 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53eba07b80 10 client.0 _ll_register_callbacks cb 0x559e6bb585d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc6bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc6bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39478/0 (socket says 172.21.15.31:39478) 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 -- 172.21.15.31:0/4051497241 learned_addr learned my addr 172.21.15.31:0/4051497241 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc61b90 0x559e6bc6b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d659c700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc632a0 0x559e6bc72cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 -- 172.21.15.31:0/4051497241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc632a0 msgr2=0x559e6bc72cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc632a0 0x559e6bc72cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 -- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc61b90 msgr2=0x559e6bc6b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc61b90 0x559e6bc6b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.563+0000 7f53d559a700 1 -- 172.21.15.31:0/4051497241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53cc009960 con 0x559e6bc624e0 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53d559a700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc6bc80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f53c0020000 tx=0x7f53c000cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c3a09147b89c86 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53b97fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53b97fa700 1 -- 172.21.15.31:0/4051497241 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00b5070 con 0x559e6bc624e0 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53b97fa700 1 -- 172.21.15.31:0/4051497241 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f53c009d3f0 con 0x559e6bc624e0 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e6bbddcc0 con 0x559e6bc624e0 2023-07-29T16:13:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53b97fa700 1 -- 172.21.15.31:0/4051497241 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53c00a1810 con 0x559e6bc624e0 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53eba07b80 10 client.15021 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559e6b9c9c10 con 0x559e6bc624e0 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53b97fa700 1 -- 172.21.15.31:0/4051497241 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e6bbddcc0 con 0x559e6bc624e0 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.564+0000 7f53b97fa700 1 -- 172.21.15.31:0/4051497241 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f53c00988b0 con 0x559e6bc624e0 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53eba07b80 10 client.15021 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53eba07b80 10 client.15021 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53eba07b80 20 client.15021 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:24.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53eba07b80 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bc76940 0x559e6bc75dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53eba07b80 4 client.15021 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:13:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e6bc30d20 con 0x559e6bc76940 2023-07-29T16:13:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bc76940 0x559e6bc75dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bc76940 0x559e6bc75dd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42df6b03a88b898 server_cookie=8749d7e564d52f28 in_seq=0 out_seq=0 2023-07-29T16:13:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.565+0000 7f53b97fa700 10 client.15021 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.566+0000 7f53b97fa700 1 -- 172.21.15.31:0/4051497241 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559e6bc30d20 con 0x559e6bc76940 2023-07-29T16:13:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.566+0000 7f53b97fa700 10 client.15021 handle_command_reply: tid=0 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.566+0000 7f53eba07b80 1 client.15021 shutdown 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 msgr2=0x559e6bc6bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc6bc80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f53c0020000 tx=0x7f53c000cdf0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 shutdown_connections 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bc76940 0x559e6bc75dd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc632a0 0x559e6bc72cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc61b90 0x559e6bc6b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 --2- 172.21.15.31:0/4051497241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bc6bc80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53b97fa700 0 client.15021 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:24.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 >> 172.21.15.31:0/4051497241 conn(0x559e6bc57390 msgr2=0x559e6bc5c010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 shutdown_connections 2023-07-29T16:13:24.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 1 -- 172.21.15.31:0/4051497241 wait complete. 2023-07-29T16:13:24.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.567+0000 7f53eba07b80 20 client.15021 trim_cache size 0 max 16384 2023-07-29T16:13:24.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.585+0000 7f53eba07b80 1 Processor -- start 2023-07-29T16:13:24.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.585+0000 7f53eba07b80 1 -- start start 2023-07-29T16:13:24.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.585+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 0x559e6bc66160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.585+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc624e0 0x559e6bc7f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.585+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc666a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53eba07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bc6c930 con 0x559e6bc61b90 2023-07-29T16:13:24.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53eba07b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bc30d20 con 0x559e6bc632a0 2023-07-29T16:13:24.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53eba07b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bb5c830 con 0x559e6bc624e0 2023-07-29T16:13:24.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 0x559e6bc66160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 0x559e6bc66160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39490/0 (socket says 172.21.15.31:39490) 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1865853909 learned_addr learned my addr 172.21.15.31:0/1865853909 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d659c700 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc666a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d559a700 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc624e0 0x559e6bc7f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1865853909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc624e0 msgr2=0x559e6bc7f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc624e0 0x559e6bc7f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1865853909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 msgr2=0x559e6bc666a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc666a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1865853909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e6ba07200 con 0x559e6bc61b90 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d659c700 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.586+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 0x559e6bc66160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009d30 tx=0x7f53cc005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865e0fdff9528c1f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53c4ff9700 1 -- 172.21.15.31:0/1865853909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc0044f0 con 0x559e6bc61b90 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53c4ff9700 1 -- 172.21.15.31:0/1865853909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53cc02a2f0 con 0x559e6bc61b90 2023-07-29T16:13:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53c4ff9700 1 -- 172.21.15.31:0/1865853909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc0052a0 con 0x559e6bc61b90 2023-07-29T16:13:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 -- 172.21.15.31:0/1865853909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 msgr2=0x559e6bc66160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 0x559e6bc66160 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009d30 tx=0x7f53cc005fa0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 -- 172.21.15.31:0/1865853909 shutdown_connections 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bc666a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc61b90 0x559e6bc66160 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1865853909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc624e0 0x559e6bc7f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 -- 172.21.15.31:0/1865853909 >> 172.21.15.31:0/1865853909 conn(0x559e6bc57390 msgr2=0x559e6bc69610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.587+0000 7f53eba07b80 1 -- 172.21.15.31:0/1865853909 shutdown_connections 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.588+0000 7f53eba07b80 1 -- 172.21.15.31:0/1865853909 wait complete. 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.588+0000 7f53eba07b80 1 -- start start 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.588+0000 7f53eba07b80 1 Processor -- start 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.588+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc7d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bcc64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53eba07b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bcc8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53eba07b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bcc9060 con 0x559e6bc624e0 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53eba07b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bcc91d0 con 0x559e6bc632a0 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53eba07b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e6bcc9390 con 0x559e6bc61b90 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53eba07b80 10 client.0 _ll_register_callbacks cb 0x559e6bb585d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc7d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bcc64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc7d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41576/0 (socket says 172.21.15.31:41576) 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1400561733 learned_addr learned my addr 172.21.15.31:0/1400561733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:24.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d659c700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bcc8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 msgr2=0x559e6bcc8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bcc8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1400561733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 msgr2=0x559e6bcc64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bcc64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d5d9b700 1 -- 172.21.15.31:0/1400561733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d01090a0 con 0x559e6bc61b90 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.589+0000 7f53d559a700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bcc64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:24.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53d5d9b700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc7d6c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f53cc02e040 tx=0x7f53cc011b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5ff53bc23aa618 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53b97fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53b97fa700 1 -- 172.21.15.31:0/1400561733 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc003680 con 0x559e6bc61b90 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53cc004380 con 0x559e6bc61b90 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53b97fa700 1 -- 172.21.15.31:0/1400561733 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f53cc0037e0 con 0x559e6bc61b90 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53b97fa700 1 -- 172.21.15.31:0/1400561733 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53cc0039c0 con 0x559e6bc61b90 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53eba07b80 10 client.25015 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:24.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559e6b9e0d90 con 0x559e6bc61b90 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.590+0000 7f53b97fa700 1 -- 172.21.15.31:0/1400561733 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53cc044020 con 0x559e6bc61b90 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53b97fa700 1 -- 172.21.15.31:0/1400561733 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f53cc011440 con 0x559e6bc61b90 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53eba07b80 10 client.25015 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53eba07b80 10 client.25015 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53eba07b80 20 client.25015 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bccde50 0x559e6bcca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:24.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53eba07b80 4 client.25015 mds_command: new command op to 24581 tid=0[{"prefix": "status"}] 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559e6bbabcf0 con 0x559e6bccde50 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53d559a700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bccde50 0x559e6bcca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53d559a700 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bccde50 0x559e6bcca7a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61fac5bba5e6322 server_cookie=7c83fa3c873512f4 in_seq=0 out_seq=0 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.591+0000 7f53b97fa700 10 client.25015 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53b97fa700 1 -- 172.21.15.31:0/1400561733 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x559e6bbabcf0 con 0x559e6bccde50 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53b97fa700 10 client.25015 handle_command_reply: tid=0 2023-07-29T16:13:24.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 client.25015 shutdown 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 msgr2=0x559e6bc7d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc7d6c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f53cc02e040 tx=0x7f53cc011b20 comp rx=0 tx=0).stop 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 shutdown_connections 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e6bccde50 0x559e6bcca7a0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e6bc632a0 0x559e6bcc8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e6bc61b90 0x559e6bc7d6c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 --2- 172.21.15.31:0/1400561733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e6bc624e0 0x559e6bcc64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53b97fa700 0 client.25015 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 >> 172.21.15.31:0/1400561733 conn(0x559e6bc57390 msgr2=0x559e6bc66160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 shutdown_connections 2023-07-29T16:13:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 1 -- 172.21.15.31:0/1400561733 wait complete. 2023-07-29T16:13:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.592+0000 7f53eba07b80 20 client.25015 trim_cache size 0 max 16384 2023-07-29T16:13:24.599 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:13:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:13:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2023-07-29T16:13:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24581, 2023-07-29T16:13:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-07-29T16:13:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-07-29T16:13:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-07-29T16:13:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 62.673540144, 2023-07-29T16:13:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2023-07-29T16:13:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 88, 2023-07-29T16:13:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 86, 2023-07-29T16:13:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 67.03807784 2023-07-29T16:13:24.601 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:13:24.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53bc0a0820 msgr2=0x7f53bc0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53bc0a0820 0x7f53bc0a2cd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f53d0075160 tx=0x7f53c8007420 comp rx=0 tx=0).stop 2023-07-29T16:13:24.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 msgr2=0x7f53d0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:24.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0073140 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f53c000e9c0 tx=0x7f53c000ecd0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 shutdown_connections 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53bc0a0820 0x7f53bc0a2cd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53d0101eb0 0x7f53d0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d0100330 0x7f53d0074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 --2- 172.21.15.31:0/3266283378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53d0101520 0x7f53d0073140 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 >> 172.21.15.31:0/3266283378 conn(0x7f53d00fb970 msgr2=0x7f53d0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 shutdown_connections 2023-07-29T16:13:24.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:24.593+0000 7f53d7fff700 1 -- 172.21.15.31:0/3266283378 wait complete. 2023-07-29T16:13:24.609 INFO:tasks.cephadm:Setup complete, yielding 2023-07-29T16:13:24.609 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-29T16:13:24.622 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T16:13:24.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph orch status' 2023-07-29T16:13:24.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:24 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3548819090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:25.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:24 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3548819090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:25.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:24 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3548819090' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:25.353 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:25.913 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:25 smithi031 ceph-mon[95504]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:26.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:25 smithi156 ceph-mon[129245]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:26.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:25 smithi039 ceph-mon[128858]: pgmap v306: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:26.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/478897791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8100e50 msgr2=0x7fd6e8103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:26.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/478897791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8100e50 0x7fd6e8103230 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009990 tx=0x7fd6d8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:26.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/478897791 shutdown_connections 2023-07-29T16:13:26.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/478897791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8106090 0x7fd6e81084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:26.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/478897791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8100e50 0x7fd6e8103230 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:26.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/478897791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e8103770 0x7fd6e8105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:26.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/478897791 >> 172.21.15.31:0/478897791 conn(0x7fd6e80fa8f0 msgr2=0x7fd6e80fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:26.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/478897791 shutdown_connections 2023-07-29T16:13:26.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.924+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/478897791 wait complete. 2023-07-29T16:13:26.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.925+0000 7fd6ee5a1700 1 Processor -- start 2023-07-29T16:13:26.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 -- start start 2023-07-29T16:13:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 0x7fd6e81007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e8103770 0x7fd6e80fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8106090 0x7fd6e80ff340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80ffac0 con 0x7fd6e8106090 2023-07-29T16:13:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80ffc30 con 0x7fd6e8103770 2023-07-29T16:13:26.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ee5a1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e81a0190 con 0x7fd6e8100e50 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6e7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 0x7fd6e81007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6e77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e8103770 0x7fd6e80fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6e7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 0x7fd6e81007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41602/0 (socket says 172.21.15.31:41602) 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6e7fff700 1 -- 172.21.15.31:0/4056635436 learned_addr learned my addr 172.21.15.31:0/4056635436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.926+0000 7fd6ecb3e700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8106090 0x7fd6e80ff340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e7fff700 1 -- 172.21.15.31:0/4056635436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e8103770 msgr2=0x7fd6e80fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:26.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e7fff700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e8103770 0x7fd6e80fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e7fff700 1 -- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8106090 msgr2=0x7fd6e80ff340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e7fff700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8106090 0x7fd6e80ff340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:26.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e7fff700 1 -- 172.21.15.31:0/4056635436 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc009520 con 0x7fd6e8100e50 2023-07-29T16:13:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e7fff700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 0x7fd6e81007b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b680 tx=0x7fd6d8005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d801d070 con 0x7fd6e8100e50 2023-07-29T16:13:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6d8022470 con 0x7fd6e8100e50 2023-07-29T16:13:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d80095f0 con 0x7fd6e8100e50 2023-07-29T16:13:26.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.927+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8017650 con 0x7fd6e8100e50 2023-07-29T16:13:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.928+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e81a06f0 con 0x7fd6e8100e50 2023-07-29T16:13:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.929+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e810bc10 con 0x7fd6e8100e50 2023-07-29T16:13:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.931+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fd6d80225e0 con 0x7fd6e8100e50 2023-07-29T16:13:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.932+0000 7fd6e57fa700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6d00a08a0 0x7fd6d00a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.932+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fd6d80cd150 con 0x7fd6e8100e50 2023-07-29T16:13:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.933+0000 7fd6e77fe700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6d00a08a0 0x7fd6d00a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:26.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.933+0000 7fd6e77fe700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6d00a08a0 0x7fd6d00a2d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009df0 tx=0x7fd6dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:26.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:13:26.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:26.934+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd6d8096b20 con 0x7fd6e8100e50 2023-07-29T16:13:27.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:13:27.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.188+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6e8102a40 con 0x7fd6d00a08a0 2023-07-29T16:13:27.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.189+0000 7fd6e57fa700 1 -- 172.21.15.31:0/4056635436 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd6e8102a40 con 0x7fd6d00a08a0 2023-07-29T16:13:27.187 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2023-07-29T16:13:27.187 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2023-07-29T16:13:27.187 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2023-07-29T16:13:27.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6d00a08a0 msgr2=0x7fd6d00a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6d00a08a0 0x7fd6d00a2d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009df0 tx=0x7fd6dc009260 comp rx=0 tx=0).stop 2023-07-29T16:13:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 msgr2=0x7fd6e81007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 0x7fd6e81007b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b680 tx=0x7fd6d8005740 comp rx=0 tx=0).stop 2023-07-29T16:13:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 shutdown_connections 2023-07-29T16:13:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6d00a08a0 0x7fd6d00a2d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:27.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e8106090 0x7fd6e80ff340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e8100e50 0x7fd6e81007b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 --2- 172.21.15.31:0/4056635436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e8103770 0x7fd6e80fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 >> 172.21.15.31:0/4056635436 conn(0x7fd6e80fa8f0 msgr2=0x7fd6e8102330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 shutdown_connections 2023-07-29T16:13:27.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:27.192+0000 7fd6ee5a1700 1 -- 172.21.15.31:0/4056635436 wait complete. 2023-07-29T16:13:27.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:13:27.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-29T16:13:28.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:27 smithi156 ceph-mon[129245]: from='client.25021 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:28.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:27 smithi156 ceph-mon[129245]: pgmap v307: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:28.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:27 smithi031 ceph-mon[95504]: from='client.25021 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:28.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:27 smithi031 ceph-mon[95504]: pgmap v307: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:28.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:27 smithi039 ceph-mon[128858]: from='client.25021 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:28.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:27 smithi039 ceph-mon[128858]: pgmap v307: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:28.397 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:29.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.927+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1981983111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a981003d0 msgr2=0x7f3a981007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.927+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1981983111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a981003d0 0x7f3a981007e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3a880098f0 tx=0x7f3a88009c00 comp rx=0 tx=0).stop 2023-07-29T16:13:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1981983111 shutdown_connections 2023-07-29T16:13:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1981983111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a98101f60 0x7f3a98108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1981983111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a981003d0 0x7f3a981007e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:29.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1981983111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981015d0 0x7f3a98101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:29.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1981983111 >> 172.21.15.31:0/1981983111 conn(0x7f3a980fb9e0 msgr2=0x7f3a980fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:29.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1981983111 shutdown_connections 2023-07-29T16:13:29.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.928+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1981983111 wait complete. 2023-07-29T16:13:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 Processor -- start 2023-07-29T16:13:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 -- start start 2023-07-29T16:13:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981003d0 0x7f3a98198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a981015d0 0x7f3a981988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:29.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 0x7f3a9819f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9819ff20 con 0x7f3a98101f60 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a981a0060 con 0x7f3a981003d0 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9d17c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a981a01f0 con 0x7f3a981015d0 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9759e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 0x7f3a9819f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a96d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981003d0 0x7f3a98198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9759e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 0x7f3a9819f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50764/0 (socket says 172.21.15.31:50764) 2023-07-29T16:13:29.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.929+0000 7f3a9759e700 1 -- 172.21.15.31:0/1012057381 learned_addr learned my addr 172.21.15.31:0/1012057381 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a9659c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a981015d0 0x7f3a981988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a96d9d700 1 -- 172.21.15.31:0/1012057381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a981015d0 msgr2=0x7f3a981988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a96d9d700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a981015d0 0x7f3a981988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a96d9d700 1 -- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 msgr2=0x7f3a9819f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a96d9d700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 0x7f3a9819f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a96d9d700 1 -- 172.21.15.31:0/1012057381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0095f0 con 0x7f3a981003d0 2023-07-29T16:13:29.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a9759e700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 0x7f3a9819f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:29.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a96d9d700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981003d0 0x7f3a98198390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3a88000c00 tx=0x7f3a8800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:29.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.930+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8801c070 con 0x7f3a981003d0 2023-07-29T16:13:29.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.931+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a8800f7d0 con 0x7f3a981003d0 2023-07-29T16:13:29.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.931+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88003970 con 0x7f3a981003d0 2023-07-29T16:13:29.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.931+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a88009520 con 0x7f3a981003d0 2023-07-29T16:13:29.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.931+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a981a06a0 con 0x7f3a981003d0 2023-07-29T16:13:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.932+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9810bbf0 con 0x7f3a981003d0 2023-07-29T16:13:29.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.934+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f3a88020030 con 0x7f3a981003d0 2023-07-29T16:13:29.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:29.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.935+0000 7f3a7ffff700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a800a0820 0x7f3a800a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:29.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.935+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f3a8804ee40 con 0x7f3a981003d0 2023-07-29T16:13:29.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.935+0000 7f3a9659c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a800a0820 0x7f3a800a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:29.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.936+0000 7f3a9659c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a800a0820 0x7f3a800a2cd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c005fd0 tx=0x7f3a8c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:29.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:29.937+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3a88096830 con 0x7f3a981003d0 2023-07-29T16:13:30.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.169+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a98107f70 con 0x7f3a800a0820 2023-07-29T16:13:30.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.184+0000 7f3a7ffff700 1 -- 172.21.15.31:0/1012057381 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7f3a98107f70 con 0x7f3a800a0820 2023-07-29T16:13:30.181 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-29T16:13:30.181 INFO:teuthology.orchestra.run.smithi031.stdout:mds.a smithi031 running (75s) 0s ago 74s 13.7M - 18.1.3-11-g0484097b b662b4d80c86 9fd0935710a6 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.b smithi039 running (73s) 1s ago 72s 29.1M - 18.1.3-11-g0484097b b662b4d80c86 4f70964c1645 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.c smithi156 running (85s) 1s ago 85s 21.7M - 18.1.3-11-g0484097b b662b4d80c86 a21ec13297e6 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.d smithi031 running (91s) 0s ago 91s 13.7M - 18.1.3-11-g0484097b b662b4d80c86 dd850115b779 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.e smithi039 running (89s) 1s ago 89s 16.5M - 18.1.3-11-g0484097b b662b4d80c86 50f94357d46f 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.f smithi156 running (87s) 1s ago 87s 20.6M - 18.1.3-11-g0484097b b662b4d80c86 c41de773f07e 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.g smithi031 running (78s) 0s ago 77s 18.3M - 18.1.3-11-g0484097b b662b4d80c86 e54d61e97c20 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.h smithi039 running (83s) 1s ago 82s 18.0M - 18.1.3-11-g0484097b b662b4d80c86 835ee9ef48d7 2023-07-29T16:13:30.182 INFO:teuthology.orchestra.run.smithi031.stdout:mds.i smithi156 running (81s) 1s ago 80s 21.4M - 18.1.3-11-g0484097b b662b4d80c86 e6a72ed9a4c6 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mds.j smithi031 running (94s) 0s ago 94s 13.7M - 18.1.3-11-g0484097b b662b4d80c86 ab1451c0064e 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mds.k smithi039 running (97s) 1s ago 97s 15.2M - 18.1.3-11-g0484097b b662b4d80c86 b47fa9568420 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mds.l smithi156 running (100s) 1s ago 99s 17.2M - 18.1.3-11-g0484097b b662b4d80c86 a6000ae264c4 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (8m) 0s ago 8m 520M - 18.1.3-11-g0484097b b662b4d80c86 9a4241da149d 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.y smithi039 *:8443,8765 running (6m) 1s ago 6m 448M - 18.1.3-11-g0484097b b662b4d80c86 639ccd0eec01 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.z smithi156 *:8443,8765 running (6m) 1s ago 6m 449M - 18.1.3-11-g0484097b b662b4d80c86 705bc654b2df 2023-07-29T16:13:30.183 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (8m) 0s ago 8m 92.8M 2048M 18.1.3-11-g0484097b b662b4d80c86 717b6f4839e4 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi039 running (6m) 1s ago 6m 65.2M 2048M 18.1.3-11-g0484097b b662b4d80c86 4c01eaf5f917 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi156 running (7m) 1s ago 6m 62.9M 2048M 18.1.3-11-g0484097b b662b4d80c86 46964b3f8e61 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (6m) 0s ago 6m 64.5M 4318M 18.1.3-11-g0484097b b662b4d80c86 c4093b61fda1 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi039 running (5m) 1s ago 5m 70.2M 4292M 18.1.3-11-g0484097b b662b4d80c86 ce825f307513 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi156 running (5m) 1s ago 5m 66.5M 4292M 18.1.3-11-g0484097b b662b4d80c86 ab5e3033ae2f 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (5m) 0s ago 5m 60.6M 4318M 18.1.3-11-g0484097b b662b4d80c86 c3b85c88d66f 2023-07-29T16:13:30.184 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi039 running (4m) 1s ago 4m 57.9M 4292M 18.1.3-11-g0484097b b662b4d80c86 d1945ccbba33 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi156 running (4m) 1s ago 4m 62.6M 4292M 18.1.3-11-g0484097b b662b4d80c86 4d9eb76feaff 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi031 running (3m) 0s ago 3m 57.5M 4318M 18.1.3-11-g0484097b b662b4d80c86 b917fb2502e6 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi039 running (3m) 1s ago 3m 62.6M 4292M 18.1.3-11-g0484097b b662b4d80c86 4c1077c8010d 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 smithi156 running (3m) 1s ago 3m 59.0M 4292M 18.1.3-11-g0484097b b662b4d80c86 020995144b11 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 smithi031 running (2m) 0s ago 2m 54.6M 4318M 18.1.3-11-g0484097b b662b4d80c86 f0976b2f4d6a 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 smithi039 running (2m) 1s ago 2m 60.8M 4292M 18.1.3-11-g0484097b b662b4d80c86 04f8fddb8970 2023-07-29T16:13:30.185 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 smithi156 running (119s) 1s ago 119s 52.0M 4292M 18.1.3-11-g0484097b b662b4d80c86 45e6a32f4da1 2023-07-29T16:13:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a800a0820 msgr2=0x7f3a800a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a800a0820 0x7f3a800a2cd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c005fd0 tx=0x7f3a8c009310 comp rx=0 tx=0).stop 2023-07-29T16:13:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981003d0 msgr2=0x7f3a98198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981003d0 0x7f3a98198390 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3a88000c00 tx=0x7f3a8800b790 comp rx=0 tx=0).stop 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 shutdown_connections 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a800a0820 0x7f3a800a2cd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a98101f60 0x7f3a9819f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a981003d0 0x7f3a98198390 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 --2- 172.21.15.31:0/1012057381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a981015d0 0x7f3a981988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 >> 172.21.15.31:0/1012057381 conn(0x7f3a980fb9e0 msgr2=0x7f3a980fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 shutdown_connections 2023-07-29T16:13:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:30.186+0000 7f3a9d17c700 1 -- 172.21.15.31:0/1012057381 wait complete. 2023-07-29T16:13:30.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-29T16:13:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:30 smithi156 ceph-mon[129245]: pgmap v308: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:30 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:30 smithi031 ceph-mon[95504]: pgmap v308: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:30.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:30.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:30 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:31.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:30 smithi039 ceph-mon[128858]: pgmap v308: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:31.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:31.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:30 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:31.340 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:31.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:31 smithi156 ceph-mon[129245]: from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:31.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:13:31.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:13:31.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:31.870 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:31 smithi031 ceph-mon[95504]: from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:31.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:13:31.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:13:31.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:32.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:31 smithi039 ceph-mon[128858]: from='client.25019 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:32.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:13:32.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:13:32.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:13:32.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 -- 172.21.15.31:0/2254353164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f4101f60 msgr2=0x7f83f4108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:32.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/2254353164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f4101f60 0x7f83f4108820 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00b0a0 tx=0x7f83ec00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 -- 172.21.15.31:0/2254353164 shutdown_connections 2023-07-29T16:13:32.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/2254353164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f4101f60 0x7f83f4108820 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/2254353164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41003d0 0x7f83f41007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/2254353164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83f41015d0 0x7f83f4101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 -- 172.21.15.31:0/2254353164 >> 172.21.15.31:0/2254353164 conn(0x7f83f40fb9e0 msgr2=0x7f83f40fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:32.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 -- 172.21.15.31:0/2254353164 shutdown_connections 2023-07-29T16:13:32.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.756+0000 7f83fa5b3700 1 -- 172.21.15.31:0/2254353164 wait complete. 2023-07-29T16:13:32.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.757+0000 7f83fa5b3700 1 Processor -- start 2023-07-29T16:13:32.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 -- start start 2023-07-29T16:13:32.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 0x7f83f4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:32.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41015d0 0x7f83f4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:32.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83f4101f60 0x7f83f419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f419ffd0 con 0x7f83f41003d0 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f41a0110 con 0x7f83f41015d0 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83fa5b3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f41a02a0 con 0x7f83f4101f60 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 0x7f83f4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 0x7f83f4198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50794/0 (socket says 172.21.15.31:50794) 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41015d0 0x7f83f4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:32.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83f3fff700 1 -- 172.21.15.31:0/1200006594 learned_addr learned my addr 172.21.15.31:0/1200006594 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.758+0000 7f83f8b50700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83f4101f60 0x7f83f419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f37fe700 1 -- 172.21.15.31:0/1200006594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83f4101f60 msgr2=0x7f83f419fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f37fe700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83f4101f60 0x7f83f419fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f37fe700 1 -- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 msgr2=0x7f83f4198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f37fe700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 0x7f83f4198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f37fe700 1 -- 172.21.15.31:0/1200006594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ec009d00 con 0x7f83f41015d0 2023-07-29T16:13:32.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f3fff700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 0x7f83f4198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f37fe700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41015d0 0x7f83f4198980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f83e800e9f0 tx=0x7f83e800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e800c9c0 con 0x7f83f41015d0 2023-07-29T16:13:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f41a0580 con 0x7f83f41015d0 2023-07-29T16:13:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83e8010430 con 0x7f83f41015d0 2023-07-29T16:13:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8018680 con 0x7f83f41015d0 2023-07-29T16:13:32.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.759+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f41a0a80 con 0x7f83f41015d0 2023-07-29T16:13:32.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.760+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f4192aa0 con 0x7f83f41015d0 2023-07-29T16:13:32.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.762+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f83e8010860 con 0x7f83f41015d0 2023-07-29T16:13:32.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.764+0000 7f83f17fa700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83dc0a0820 0x7f83dc0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:32.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.764+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f83e8014070 con 0x7f83f41015d0 2023-07-29T16:13:32.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.764+0000 7f83f3fff700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83dc0a0820 0x7f83dc0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:32.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.765+0000 7f83f3fff700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83dc0a0820 0x7f83dc0a2cd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f83e4005950 tx=0x7f83e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:32.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.766+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f83e8095730 con 0x7f83f41015d0 2023-07-29T16:13:32.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:32 smithi156 ceph-mon[129245]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:32.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:32 smithi031 ceph-mon[95504]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:32.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.978+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f83f4061190 con 0x7f83dc0a0820 2023-07-29T16:13:32.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.984+0000 7f83f17fa700 1 -- 172.21.15.31:0/1200006594 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7f83f4061190 con 0x7f83dc0a0820 2023-07-29T16:13:32.981 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-29T16:13:32.981 INFO:teuthology.orchestra.run.smithi031.stdout:mds.all 12/12 4s ago 105s smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi039=b;smithi039=e;smithi039=h;smithi039=k;smithi156=c;smithi156=f;smithi156=i;smithi156=l;count:12 2023-07-29T16:13:32.982 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 3/3 4s ago 6m smithi031=x;smithi039=y;smithi156=z;count:3 2023-07-29T16:13:32.982 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 4s ago 7m smithi031:172.21.15.31=a;smithi039:172.21.15.39=b;smithi156:172.21.15.156=c;count:3 2023-07-29T16:13:32.982 INFO:teuthology.orchestra.run.smithi031.stdout:osd 12 4s ago - 2023-07-29T16:13:32.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83dc0a0820 msgr2=0x7f83dc0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83dc0a0820 0x7f83dc0a2cd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f83e4005950 tx=0x7f83e4009260 comp rx=0 tx=0).stop 2023-07-29T16:13:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41015d0 msgr2=0x7f83f4198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41015d0 0x7f83f4198980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f83e800e9f0 tx=0x7f83e800ed00 comp rx=0 tx=0).stop 2023-07-29T16:13:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 shutdown_connections 2023-07-29T16:13:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83dc0a0820 0x7f83dc0a2cd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83f4101f60 0x7f83f419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83f41003d0 0x7f83f4198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 --2- 172.21.15.31:0/1200006594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83f41015d0 0x7f83f4198980 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.986+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 >> 172.21.15.31:0/1200006594 conn(0x7f83f40fb9e0 msgr2=0x7f83f4106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:32.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.987+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 shutdown_connections 2023-07-29T16:13:32.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:32.987+0000 7f83fa5b3700 1 -- 172.21.15.31:0/1200006594 wait complete. 2023-07-29T16:13:33.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:32 smithi039 ceph-mon[128858]: pgmap v309: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:33.421 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-29T16:13:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:34 smithi031 ceph-mon[95504]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:34 smithi031 ceph-mon[95504]: pgmap v310: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:34 smithi156 ceph-mon[129245]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:34 smithi156 ceph-mon[129245]: pgmap v310: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:35.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:34 smithi039 ceph-mon[128858]: from='client.25025 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:35.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:34 smithi039 ceph-mon[128858]: pgmap v310: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:35.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 -- 172.21.15.31:0/1664496163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 msgr2=0x7efc600731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 --2- 172.21.15.31:0/1664496163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc600731e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efc5c009990 tx=0x7efc5c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 -- 172.21.15.31:0/1664496163 shutdown_connections 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 --2- 172.21.15.31:0/1664496163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc601082a0 0x7efc6010a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 --2- 172.21.15.31:0/1664496163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc600731e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 --2- 172.21.15.31:0/1664496163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 -- 172.21.15.31:0/1664496163 >> 172.21.15.31:0/1664496163 conn(0x7efc600fb9e0 msgr2=0x7efc600fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 -- 172.21.15.31:0/1664496163 shutdown_connections 2023-07-29T16:13:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.659+0000 7efc67276700 1 -- 172.21.15.31:0/1664496163 wait complete. 2023-07-29T16:13:35.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.660+0000 7efc67276700 1 Processor -- start 2023-07-29T16:13:35.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.660+0000 7efc67276700 1 -- start start 2023-07-29T16:13:35.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.660+0000 7efc67276700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:35.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.660+0000 7efc67276700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc60198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:35.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc67276700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc601082a0 0x7efc6019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:35.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc67276700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6019fe90 con 0x7efc60074d80 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc67276700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6019ffd0 con 0x7efc600737b0 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc67276700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc601a0160 con 0x7efc601082a0 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc60198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc65012700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc60198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50814/0 (socket says 172.21.15.31:50814) 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc65012700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41274/0 (socket says 172.21.15.31:41274) 2023-07-29T16:13:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 -- 172.21.15.31:0/5110186 learned_addr learned my addr 172.21.15.31:0/5110186 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc65813700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc601082a0 0x7efc6019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 -- 172.21.15.31:0/5110186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc601082a0 msgr2=0x7efc6019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc601082a0 0x7efc6019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 -- 172.21.15.31:0/5110186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 msgr2=0x7efc60198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc64811700 1 -- 172.21.15.31:0/5110186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50009520 con 0x7efc60074d80 2023-07-29T16:13:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.661+0000 7efc65012700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:35.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.662+0000 7efc64811700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc60198840 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efc5000eb60 tx=0x7efc5000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:35.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.662+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5000cbd0 con 0x7efc60074d80 2023-07-29T16:13:35.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.662+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc500044e0 con 0x7efc60074d80 2023-07-29T16:13:35.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.662+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5c0095f0 con 0x7efc60074d80 2023-07-29T16:13:35.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.662+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50003df0 con 0x7efc60074d80 2023-07-29T16:13:35.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.662+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc601a0800 con 0x7efc60074d80 2023-07-29T16:13:35.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.663+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc60106950 con 0x7efc60074d80 2023-07-29T16:13:35.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.665+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7efc50004000 con 0x7efc60074d80 2023-07-29T16:13:35.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.667+0000 7efc567fc700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efc4c0a08a0 0x7efc4c0a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:35.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.667+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7efc50014070 con 0x7efc60074d80 2023-07-29T16:13:35.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.667+0000 7efc65012700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efc4c0a08a0 0x7efc4c0a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:35.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.668+0000 7efc65012700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efc4c0a08a0 0x7efc4c0a2d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00b2e0 tx=0x7efc5c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:35.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.671+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7efc50095950 con 0x7efc60074d80 2023-07-29T16:13:35.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.886+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7efc60061190 con 0x7efc4c0a08a0 2023-07-29T16:13:35.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.888+0000 7efc567fc700 1 -- 172.21.15.31:0/5110186 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7efc60061190 con 0x7efc4c0a08a0 2023-07-29T16:13:35.885 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2023-07-29T16:13:35.885 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2023-07-29T16:13:35.885 INFO:teuthology.orchestra.run.smithi031.stdout:smithi039 172.21.15.39 2023-07-29T16:13:35.885 INFO:teuthology.orchestra.run.smithi031.stdout:smithi156 172.21.15.156 2023-07-29T16:13:35.885 INFO:teuthology.orchestra.run.smithi031.stdout:3 hosts in cluster 2023-07-29T16:13:35.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efc4c0a08a0 msgr2=0x7efc4c0a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:35.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efc4c0a08a0 0x7efc4c0a2d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc5c00b2e0 tx=0x7efc5c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 msgr2=0x7efc60198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc60198840 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7efc5000eb60 tx=0x7efc5000ee70 comp rx=0 tx=0).stop 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 shutdown_connections 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efc4c0a08a0 0x7efc4c0a2d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc601082a0 0x7efc6019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc600737b0 0x7efc60198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 --2- 172.21.15.31:0/5110186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efc60074d80 0x7efc60198840 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:35.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 >> 172.21.15.31:0/5110186 conn(0x7efc600fb9e0 msgr2=0x7efc600fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 shutdown_connections 2023-07-29T16:13:35.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:35.890+0000 7efc67276700 1 -- 172.21.15.31:0/5110186 wait complete. 2023-07-29T16:13:36.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-29T16:13:36.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:36 smithi031 ceph-mon[95504]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:36.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:36 smithi156 ceph-mon[129245]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:37.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:36 smithi039 ceph-mon[128858]: pgmap v311: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:37.066 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:37.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:37 smithi031 ceph-mon[95504]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:37.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:37 smithi156 ceph-mon[129245]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:38.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:37 smithi039 ceph-mon[128858]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:38.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.464+0000 7f525e652700 1 -- 172.21.15.31:0/47298337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 msgr2=0x7f5258103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:38.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.464+0000 7f525e652700 1 --2- 172.21.15.31:0/47298337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f5258103230 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5248009990 tx=0x7f5248009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 -- 172.21.15.31:0/47298337 shutdown_connections 2023-07-29T16:13:38.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 --2- 172.21.15.31:0/47298337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258106090 0x7f52581084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 --2- 172.21.15.31:0/47298337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f5258103230 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 --2- 172.21.15.31:0/47298337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258103770 0x7f5258105b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 -- 172.21.15.31:0/47298337 >> 172.21.15.31:0/47298337 conn(0x7f52580fa8f0 msgr2=0x7f52580fcd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:38.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 -- 172.21.15.31:0/47298337 shutdown_connections 2023-07-29T16:13:38.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.465+0000 7f525e652700 1 -- 172.21.15.31:0/47298337 wait complete. 2023-07-29T16:13:38.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 Processor -- start 2023-07-29T16:13:38.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 -- start start 2023-07-29T16:13:38.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f52581007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:38.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258103770 0x7f52580fee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:38.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258106090 0x7f52580ff340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:38.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52580ffac0 con 0x7f5258100e50 2023-07-29T16:13:38.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52580ffc30 con 0x7f5258106090 2023-07-29T16:13:38.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f525e652700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52581a0110 con 0x7f5258103770 2023-07-29T16:13:38.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.466+0000 7f5257fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f52581007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:38.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f52581007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50818/0 (socket says 172.21.15.31:50818) 2023-07-29T16:13:38.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 -- 172.21.15.31:0/21490767 learned_addr learned my addr 172.21.15.31:0/21490767 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:38.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f52577fe700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258103770 0x7f52580fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:38.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f525cbef700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258106090 0x7f52580ff340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:38.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 -- 172.21.15.31:0/21490767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258103770 msgr2=0x7f52580fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:38.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258103770 0x7f52580fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 -- 172.21.15.31:0/21490767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258106090 msgr2=0x7f52580ff340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:38.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258106090 0x7f52580ff340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 -- 172.21.15.31:0/21490767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52480095f0 con 0x7f5258100e50 2023-07-29T16:13:38.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f52577fe700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258103770 0x7f52580fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:38.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f525cbef700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258106090 0x7f52580ff340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:38.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.467+0000 7f5257fff700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f52581007b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5248000c00 tx=0x7f52480057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:38.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.468+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f524801d070 con 0x7f5258100e50 2023-07-29T16:13:38.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.468+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52581a0390 con 0x7f5258100e50 2023-07-29T16:13:38.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.468+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f524800fb90 con 0x7f5258100e50 2023-07-29T16:13:38.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.468+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480173f0 con 0x7f5258100e50 2023-07-29T16:13:38.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.468+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52581a0880 con 0x7f5258100e50 2023-07-29T16:13:38.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.469+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f525810bbd0 con 0x7f5258100e50 2023-07-29T16:13:38.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.471+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f5248021410 con 0x7f5258100e50 2023-07-29T16:13:38.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.472+0000 7f52557fa700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f52400a0820 0x7f52400a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:38.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.472+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f52480ccff0 con 0x7f5258100e50 2023-07-29T16:13:38.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.472+0000 7f52577fe700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f52400a0820 0x7f52400a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:38.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.473+0000 7f52577fe700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f52400a0820 0x7f52400a2cd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f524c011490 tx=0x7f524c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:38.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.476+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5248096940 con 0x7f5258100e50 2023-07-29T16:13:38.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.692+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f52581029c0 con 0x7f52400a0820 2023-07-29T16:13:38.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.693+0000 7f52557fa700 1 -- 172.21.15.31:0/21490767 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f52581029c0 con 0x7f52400a0820 2023-07-29T16:13:38.691 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-29T16:13:38.691 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 2m ago LVM detected, locked 2023-07-29T16:13:38.691 INFO:teuthology.orchestra.run.smithi031.stdout:smithi039 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102QX4P0KGN 3726G No 2m ago LVM detected, locked 2023-07-29T16:13:38.691 INFO:teuthology.orchestra.run.smithi031.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 2m ago LVM detected, locked 2023-07-29T16:13:38.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f52400a0820 msgr2=0x7f52400a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:38.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f52400a0820 0x7f52400a2cd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f524c011490 tx=0x7f524c009260 comp rx=0 tx=0).stop 2023-07-29T16:13:38.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 msgr2=0x7f52581007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f52581007b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5248000c00 tx=0x7f52480057a0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 shutdown_connections 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f52400a0820 0x7f52400a2cd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5258106090 0x7f52580ff340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5258100e50 0x7f52581007b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 --2- 172.21.15.31:0/21490767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5258103770 0x7f52580fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:38.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 >> 172.21.15.31:0/21490767 conn(0x7f52580fa8f0 msgr2=0x7f52581022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:38.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 shutdown_connections 2023-07-29T16:13:38.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:38.696+0000 7f525e652700 1 -- 172.21.15.31:0/21490767 wait complete. 2023-07-29T16:13:38.803 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:38 smithi031 ceph-mon[95504]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:38.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:38 smithi156 ceph-mon[129245]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:39.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:38 smithi039 ceph-mon[128858]: pgmap v312: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:39.145 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-29T16:13:39.155 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T16:13:39.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-29T16:13:39.883 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:40.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:40 smithi031 ceph-mon[95504]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:40.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:40 smithi031 ceph-mon[95504]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:40.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:40 smithi156 ceph-mon[129245]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:40.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:40 smithi156 ceph-mon[129245]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:41.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:40 smithi039 ceph-mon[128858]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-29T16:13:41.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:40 smithi039 ceph-mon[128858]: pgmap v313: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:41.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 -- 172.21.15.31:0/2489280758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1760101f60 msgr2=0x7f1760108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:41.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 --2- 172.21.15.31:0/2489280758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1760101f60 0x7f1760108820 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f175800b0a0 tx=0x7f175800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 -- 172.21.15.31:0/2489280758 shutdown_connections 2023-07-29T16:13:41.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 --2- 172.21.15.31:0/2489280758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1760101f60 0x7f1760108820 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 --2- 172.21.15.31:0/2489280758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 0x7f17601007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 --2- 172.21.15.31:0/2489280758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17601015d0 0x7f1760101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 -- 172.21.15.31:0/2489280758 >> 172.21.15.31:0/2489280758 conn(0x7f17600fb9e0 msgr2=0x7f17600fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:41.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 -- 172.21.15.31:0/2489280758 shutdown_connections 2023-07-29T16:13:41.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.287+0000 7f17668a7700 1 -- 172.21.15.31:0/2489280758 wait complete. 2023-07-29T16:13:41.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.288+0000 7f17668a7700 1 Processor -- start 2023-07-29T16:13:41.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.288+0000 7f17668a7700 1 -- start start 2023-07-29T16:13:41.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.288+0000 7f17668a7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 0x7f1760198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f17668a7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 0x7f1760198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f17668a7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1760101f60 0x7f176019fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f17668a7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176019ffd0 con 0x7f17601015d0 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f17668a7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17601a0110 con 0x7f17601003d0 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f17668a7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17601a02a0 con 0x7f1760101f60 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 0x7f1760198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:41.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 0x7f1760198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60608/0 (socket says 172.21.15.31:60608) 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 -- 172.21.15.31:0/3969423175 learned_addr learned my addr 172.21.15.31:0/3969423175 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175ffff700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 0x7f1760198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f1764e44700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1760101f60 0x7f176019fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 -- 172.21.15.31:0/3969423175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1760101f60 msgr2=0x7f176019fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1760101f60 0x7f176019fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 -- 172.21.15.31:0/3969423175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 msgr2=0x7f1760198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 0x7f1760198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175f7fe700 1 -- 172.21.15.31:0/3969423175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758009d00 con 0x7f17601015d0 2023-07-29T16:13:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.289+0000 7f175ffff700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 0x7f1760198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.290+0000 7f175f7fe700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 0x7f1760198980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f175400e9d0 tx=0x7f175400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.290+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754009790 con 0x7f17601015d0 2023-07-29T16:13:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.290+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f175400ee70 con 0x7f17601015d0 2023-07-29T16:13:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.290+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17601a0580 con 0x7f17601015d0 2023-07-29T16:13:41.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.290+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754010ea0 con 0x7f17601015d0 2023-07-29T16:13:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.290+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17601a0a80 con 0x7f17601015d0 2023-07-29T16:13:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.291+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1760192aa0 con 0x7f17601015d0 2023-07-29T16:13:41.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.293+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f175400cbd0 con 0x7f17601015d0 2023-07-29T16:13:41.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.294+0000 7f175d7fa700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f17480a0820 0x7f17480a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:41.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.295+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f1754014070 con 0x7f17601015d0 2023-07-29T16:13:41.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.295+0000 7f175ffff700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f17480a0820 0x7f17480a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:41.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.295+0000 7f175ffff700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f17480a0820 0x7f17480a2cd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1760101430 tx=0x7f1750009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:41.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.299+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1754051720 con 0x7f17601015d0 2023-07-29T16:13:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.553+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f176002cc30 con 0x7f17601015d0 2023-07-29T16:13:41.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.554+0000 7f175d7fa700 1 -- 172.21.15.31:0/3969423175 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2874 (secure 0 0 0) 0x7f17540c9e60 con 0x7f17601015d0 2023-07-29T16:13:41.551 INFO:teuthology.orchestra.run.smithi031.stdout:e23 2023-07-29T16:13:41.552 INFO:teuthology.orchestra.run.smithi031.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-29T16:13:41.552 INFO:teuthology.orchestra.run.smithi031.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:13:41.552 INFO:teuthology.orchestra.run.smithi031.stdout:legacy client fscid: 1 2023-07-29T16:13:41.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:41.552 INFO:teuthology.orchestra.run.smithi031.stdout:Filesystem 'cephfs' (1) 2023-07-29T16:13:41.552 INFO:teuthology.orchestra.run.smithi031.stdout:fs_name cephfs 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 23 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-07-29T16:12:21.912364+0000 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2023-07-29T16:12:27.740070+0000 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:tableserver 0 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:root 0 2023-07-29T16:13:41.553 INFO:teuthology.orchestra.run.smithi031.stdout:session_timeout 300 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:session_autoclose 300 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:max_file_size 1099511627776 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:required_client_features {} 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure 0 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure_osd_epoch 0 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:max_mds 3 2023-07-29T16:13:41.554 INFO:teuthology.orchestra.run.smithi031.stdout:in 0,1,2 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:up {0=24581,1=24551,2=24550} 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:failed 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:damaged 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:stopped 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:data_pools [3] 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:metadata_pool 2 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:inline_data disabled 2023-07-29T16:13:41.555 INFO:teuthology.orchestra.run.smithi031.stdout:balancer 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:bal_rank_mask -1 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:standby_count_wanted 0 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b{0:24581} state up:active seq 3 addr [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c{0:24532} state up:standby-replay seq 1 addr [v2:172.21.15.156:6836/809205544,v1:172.21.15.156:6837/809205544] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.g{1:24551} state up:active seq 5 addr [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.h{1:24530} state up:standby-replay seq 1 addr [v2:172.21.15.39:6836/1297587845,v1:172.21.15.39:6837/1297587845] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.i{2:24550} state up:active seq 6 addr [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.556 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f{2:24520} state up:standby-replay seq 1 addr [v2:172.21.15.156:6834/1642153321,v1:172.21.15.156:6835/1642153321] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout:Standby daemons: 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.j{-1:14499} state up:standby seq 1 addr [v2:172.21.15.31:6834/2459121100,v1:172.21.15.31:6835/2459121100] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a{-1:14586} state up:standby seq 1 addr [v2:172.21.15.31:6840/3775768915,v1:172.21.15.31:6841/3775768915] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.557 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.l{-1:24463} state up:standby seq 1 addr [v2:172.21.15.156:6832/2085573271,v1:172.21.15.156:6833/2085573271] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.558 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.k{-1:24464} state up:standby seq 1 addr [v2:172.21.15.39:6832/3785261631,v1:172.21.15.39:6833/3785261631] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.558 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d{-1:24488} state up:standby seq 1 addr [v2:172.21.15.31:6836/3256727678,v1:172.21.15.31:6837/3256727678] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.558 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e{-1:24500} state up:standby seq 1 addr [v2:172.21.15.39:6834/2848213481,v1:172.21.15.39:6835/2848213481] compat {c=[1],r=[1],i=[7ff]}] 2023-07-29T16:13:41.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.556+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f17480a0820 msgr2=0x7f17480a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f17480a0820 0x7f17480a2cd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1760101430 tx=0x7f1750009260 comp rx=0 tx=0).stop 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 msgr2=0x7f1760198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 0x7f1760198980 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f175400e9d0 tx=0x7f175400ece0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 shutdown_connections 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f17480a0820 0x7f17480a2cd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1760101f60 0x7f176019fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17601003d0 0x7f1760198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 --2- 172.21.15.31:0/3969423175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17601015d0 0x7f1760198980 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 >> 172.21.15.31:0/3969423175 conn(0x7f17600fb9e0 msgr2=0x7f1760106ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 shutdown_connections 2023-07-29T16:13:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:41.557+0000 7f17668a7700 1 -- 172.21.15.31:0/3969423175 wait complete. 2023-07-29T16:13:41.560 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:13:41.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:41 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3969423175' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-29T16:13:41.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:41 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3969423175' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-29T16:13:41.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-29T16:13:42.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:41 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3969423175' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-29T16:13:42.692 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:42.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:42 smithi156 ceph-mon[129245]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:42.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:42 smithi031 ceph-mon[95504]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:43.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:42 smithi039 ceph-mon[128858]: pgmap v314: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:44.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 -- 172.21.15.31:0/1011390590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488101f60 msgr2=0x7f8488108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 --2- 172.21.15.31:0/1011390590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488101f60 0x7f8488108820 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f848000b0a0 tx=0x7f848000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 -- 172.21.15.31:0/1011390590 shutdown_connections 2023-07-29T16:13:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 --2- 172.21.15.31:0/1011390590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488101f60 0x7f8488108820 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 --2- 172.21.15.31:0/1011390590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84881003d0 0x7f84881007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 --2- 172.21.15.31:0/1011390590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84881015d0 0x7f8488101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 -- 172.21.15.31:0/1011390590 >> 172.21.15.31:0/1011390590 conn(0x7f84880fb9e0 msgr2=0x7f84880fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 -- 172.21.15.31:0/1011390590 shutdown_connections 2023-07-29T16:13:44.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.197+0000 7f848faaa700 1 -- 172.21.15.31:0/1011390590 wait complete. 2023-07-29T16:13:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.198+0000 7f848faaa700 1 Processor -- start 2023-07-29T16:13:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 -- start start 2023-07-29T16:13:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84881003d0 0x7f8488074af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 0x7f8488073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8488101f60 0x7f8488073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488073e00 con 0x7f84881015d0 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488073f70 con 0x7f84881003d0 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.199+0000 7f848faaa700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881a0210 con 0x7f8488101f60 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 0x7f8488073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d846700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84881003d0 0x7f8488074af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 0x7f8488073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60626/0 (socket says 172.21.15.31:60626) 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 -- 172.21.15.31:0/714548344 learned_addr learned my addr 172.21.15.31:0/714548344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848e047700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8488101f60 0x7f8488073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 -- 172.21.15.31:0/714548344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8488101f60 msgr2=0x7f8488073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8488101f60 0x7f8488073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 -- 172.21.15.31:0/714548344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84881003d0 msgr2=0x7f8488074af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84881003d0 0x7f8488074af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 -- 172.21.15.31:0/714548344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480009d00 con 0x7f84881015d0 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f848d045700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 0x7f8488073140 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f847800d950 tx=0x7f847800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.200+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847800f890 con 0x7f84881015d0 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.201+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8478010460 con 0x7f84881015d0 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.201+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847800e610 con 0x7f84881015d0 2023-07-29T16:13:44.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.201+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84881a04f0 con 0x7f84881015d0 2023-07-29T16:13:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.201+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84881a09f0 con 0x7f84881015d0 2023-07-29T16:13:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.202+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8488107950 con 0x7f84881015d0 2023-07-29T16:13:44.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.204+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f84780108d0 con 0x7f84881015d0 2023-07-29T16:13:44.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.205+0000 7f847effd700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84740a08a0 0x7f84740a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:44.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.205+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f84780cb990 con 0x7f84881015d0 2023-07-29T16:13:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.205+0000 7f848d846700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84740a08a0 0x7f84740a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:44.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.206+0000 7f848d846700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84740a08a0 0x7f84740a2d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8488101430 tx=0x7f8484009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:44.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.210+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f8478095250 con 0x7f84881015d0 2023-07-29T16:13:44.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.415+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f848802ce80 con 0x7f84881015d0 2023-07-29T16:13:44.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.416+0000 7f847effd700 1 -- 172.21.15.31:0/714548344 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5322 (secure 0 0 0) 0x7f8478020070 con 0x7f84881015d0 2023-07-29T16:13:44.414 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 88 2023-07-29T16:13:44.414 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 858a7894-2e29-11ee-9b36-001a4aab830c 2023-07-29T16:13:44.414 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-07-29T16:04:46.417698+0000 2023-07-29T16:13:44.415 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2023-07-29T16:12:50.274238+0000 2023-07-29T16:13:44.415 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-29T16:13:44.415 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 26 2023-07-29T16:13:44.415 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2023-07-29T16:13:44.415 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2023-07-29T16:13:44.416 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2023-07-29T16:13:44.416 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2023-07-29T16:13:44.416 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client luminous 2023-07-29T16:13:44.416 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release reef 2023-07-29T16:13:44.417 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2023-07-29T16:13:44.417 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-29T16:13:44.417 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 12 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/2308009899,v1:172.21.15.31:6803/2308009899] [v2:172.21.15.31:6804/2308009899,v1:172.21.15.31:6805/2308009899] exists,up 9f598978-c50a-4143-a36d-091f24731a98 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6800/701098917,v1:172.21.15.39:6801/701098917] [v2:172.21.15.39:6802/701098917,v1:172.21.15.39:6803/701098917] exists,up 2ceb36a3-18a7-4af6-ae87-03d42276da3c 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6800/3425187698,v1:172.21.15.156:6801/3425187698] [v2:172.21.15.156:6802/3425187698,v1:172.21.15.156:6803/3425187698] exists,up b70de0c9-39ca-4d04-bbe3-532e576fd586 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 26 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/1855156941,v1:172.21.15.31:6811/1855156941] [v2:172.21.15.31:6812/1855156941,v1:172.21.15.31:6813/1855156941] exists,up de564104-d127-432d-b638-ab1547a64d6f 2023-07-29T16:13:44.418 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 34 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6808/1826445248,v1:172.21.15.39:6809/1826445248] [v2:172.21.15.39:6810/1826445248,v1:172.21.15.39:6811/1826445248] exists,up 8c0cb642-c0c5-4349-8c45-584df086b71b 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6808/1995637959,v1:172.21.15.156:6809/1995637959] [v2:172.21.15.156:6810/1995637959,v1:172.21.15.156:6811/1995637959] exists,up 37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 47 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/2073594680,v1:172.21.15.31:6819/2073594680] [v2:172.21.15.31:6820/2073594680,v1:172.21.15.31:6821/2073594680] exists,up e93cbf91-6144-4ac7-8bb6-1e964c5945d9 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6816/4224022326,v1:172.21.15.39:6817/4224022326] [v2:172.21.15.39:6818/4224022326,v1:172.21.15.39:6819/4224022326] exists,up 78eb79f8-f2ac-4b63-a6d7-431e91959528 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6816/2938271220,v1:172.21.15.156:6817/2938271220] [v2:172.21.15.156:6818/2938271220,v1:172.21.15.156:6819/2938271220] exists,up fead396d-f980-4122-8ec0-cbb58dd236fe 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/3232146044,v1:172.21.15.31:6827/3232146044] [v2:172.21.15.31:6828/3232146044,v1:172.21.15.31:6829/3232146044] exists,up 4646b9f8-76b7-4985-b448-0323ecbe272a 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6824/2960119201,v1:172.21.15.39:6825/2960119201] [v2:172.21.15.39:6826/2960119201,v1:172.21.15.39:6827/2960119201] exists,up 7ced3e9f-39cb-4112-bd93-5a36a3949bd7 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6824/2124313764,v1:172.21.15.156:6825/2124313764] [v2:172.21.15.156:6826/2124313764,v1:172.21.15.156:6827/2124313764] exists,up 8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19 2023-07-29T16:13:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.c [2,4] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.1c [8,11] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.11 [4,6] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.17 [4,7] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.1e [5,8] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.28 [5,6] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.2a [1,9] 2023-07-29T16:13:44.420 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.31 [4,8] 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.38 [3,9] 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.3c [4,9] 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3069668148 expires 2023-07-30T16:05:49.191872+0000 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/1090542383 expires 2023-07-30T16:05:49.191872+0000 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/1090542383 expires 2023-07-30T16:05:49.191872+0000 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3115186416 expires 2023-07-30T16:05:49.191872+0000 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/480480667 expires 2023-07-30T16:05:22.573088+0000 2023-07-29T16:13:44.421 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3981163231 expires 2023-07-30T16:05:22.573088+0000 2023-07-29T16:13:44.422 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/890525828 expires 2023-07-30T16:05:22.573088+0000 2023-07-29T16:13:44.422 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/76880357 expires 2023-07-30T16:05:22.573088+0000 2023-07-29T16:13:44.422 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/76880357 expires 2023-07-30T16:05:22.573088+0000 2023-07-29T16:13:44.422 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/538594089 expires 2023-07-30T16:05:49.191872+0000 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.418+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84740a08a0 msgr2=0x7f84740a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.418+0000 7f848faaa700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84740a08a0 0x7f84740a2d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f8488101430 tx=0x7f8484009260 comp rx=0 tx=0).stop 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 msgr2=0x7f8488073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 0x7f8488073140 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f847800d950 tx=0x7f847800dc60 comp rx=0 tx=0).stop 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 shutdown_connections 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84740a08a0 0x7f84740a2d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8488101f60 0x7f8488073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84881003d0 0x7f8488074af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 --2- 172.21.15.31:0/714548344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f84881015d0 0x7f8488073140 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 >> 172.21.15.31:0/714548344 conn(0x7f84880fb9e0 msgr2=0x7f8488104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 shutdown_connections 2023-07-29T16:13:44.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:44.419+0000 7f848faaa700 1 -- 172.21.15.31:0/714548344 wait complete. 2023-07-29T16:13:44.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:44 smithi156 ceph-mon[129245]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:44.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:44 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/714548344' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-29T16:13:44.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:44 smithi031 ceph-mon[95504]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:44.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:44 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/714548344' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-29T16:13:44.975 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-29T16:13:44.987 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:45.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:44 smithi039 ceph-mon[128858]: pgmap v315: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-07-29T16:13:45.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:44 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/714548344' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-29T16:13:45.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 -- 172.21.15.31:0/19598781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288101530 msgr2=0x7fb288101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/19598781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288101530 0x7fb288101980 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009990 tx=0x7fb27c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 -- 172.21.15.31:0/19598781 shutdown_connections 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/19598781 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb288108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/19598781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288100340 0x7fb288100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/19598781 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288101530 0x7fb288101980 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 -- 172.21.15.31:0/19598781 >> 172.21.15.31:0/19598781 conn(0x7fb2880fb920 msgr2=0x7fb2880fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 -- 172.21.15.31:0/19598781 shutdown_connections 2023-07-29T16:13:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.199+0000 7fb28e7ff700 1 -- 172.21.15.31:0/19598781 wait complete. 2023-07-29T16:13:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.200+0000 7fb28e7ff700 1 Processor -- start 2023-07-29T16:13:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.200+0000 7fb28e7ff700 1 -- start start 2023-07-29T16:13:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28e7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288100340 0x7fb288198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28e7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288101530 0x7fb288198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28e7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb28819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28e7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28819fd80 con 0x7fb288101ec0 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28e7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb28819fec0 con 0x7fb288100340 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28e7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2881a0050 con 0x7fb288101530 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb28819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb28819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60640/0 (socket says 172.21.15.31:60640) 2023-07-29T16:13:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 -- 172.21.15.31:0/1451874344 learned_addr learned my addr 172.21.15.31:0/1451874344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb287fff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288100340 0x7fb288198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb2877fe700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288101530 0x7fb288198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 -- 172.21.15.31:0/1451874344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288101530 msgr2=0x7fb288198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288101530 0x7fb288198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 -- 172.21.15.31:0/1451874344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288100340 msgr2=0x7fb288198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288100340 0x7fb288198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.201+0000 7fb28cd9c700 1 -- 172.21.15.31:0/1451874344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb280009c40 con 0x7fb288101ec0 2023-07-29T16:13:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.202+0000 7fb28cd9c700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb28819f7b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb28000c370 tx=0x7fb28000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.202+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb280017070 con 0x7fb288101ec0 2023-07-29T16:13:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.202+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c0095f0 con 0x7fb288101ec0 2023-07-29T16:13:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.202+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb28000f040 con 0x7fb288101ec0 2023-07-29T16:13:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.202+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28000e050 con 0x7fb288101ec0 2023-07-29T16:13:45.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.202+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2881a06a0 con 0x7fb288101ec0 2023-07-29T16:13:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.203+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb28810bb10 con 0x7fb288101ec0 2023-07-29T16:13:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.205+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fb28000cb60 con 0x7fb288101ec0 2023-07-29T16:13:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.206+0000 7fb2857fa700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2700a0860 0x7fb2700a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.207+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fb2800cbbe0 con 0x7fb288101ec0 2023-07-29T16:13:45.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.207+0000 7fb287fff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2700a0860 0x7fb2700a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.207+0000 7fb287fff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2700a0860 0x7fb2700a2d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb288101390 tx=0x7fb278009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.209+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb2800954a0 con 0x7fb288101ec0 2023-07-29T16:13:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.417+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb28810bcd0 con 0x7fb288101ec0 2023-07-29T16:13:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.419+0000 7fb2857fa700 1 -- 172.21.15.31:0/1451874344 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22618 (secure 0 0 0) 0x7fb280098ac0 con 0x7fb288101ec0 2023-07-29T16:13:45.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:45.417 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:50.274238+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.421+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2700a0860 msgr2=0x7fb2700a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.421+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2700a0860 0x7fb2700a2d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb288101390 tx=0x7fb278009260 comp rx=0 tx=0).stop 2023-07-29T16:13:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.421+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 msgr2=0x7fb28819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.421+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb28819f7b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb28000c370 tx=0x7fb28000c680 comp rx=0 tx=0).stop 2023-07-29T16:13:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 shutdown_connections 2023-07-29T16:13:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2700a0860 0x7fb2700a2d10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb288101ec0 0x7fb28819f7b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb288100340 0x7fb288198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 --2- 172.21.15.31:0/1451874344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb288101530 0x7fb288198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 >> 172.21.15.31:0/1451874344 conn(0x7fb2880fb920 msgr2=0x7fb288106d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 shutdown_connections 2023-07-29T16:13:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.422+0000 7fb28e7ff700 1 -- 172.21.15.31:0/1451874344 wait complete. 2023-07-29T16:13:45.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:13:45.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:13:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 -- 172.21.15.31:0/3907018134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8300737b0 msgr2=0x7fe830073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 --2- 172.21.15.31:0/3907018134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8300737b0 0x7fe830073c20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe820009990 tx=0x7fe820009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 -- 172.21.15.31:0/3907018134 shutdown_connections 2023-07-29T16:13:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 --2- 172.21.15.31:0/3907018134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8301081e0 0x7fe83010a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 --2- 172.21.15.31:0/3907018134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe8300731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 --2- 172.21.15.31:0/3907018134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8300737b0 0x7fe830073c20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 -- 172.21.15.31:0/3907018134 >> 172.21.15.31:0/3907018134 conn(0x7fe8300fb920 msgr2=0x7fe8300fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 -- 172.21.15.31:0/3907018134 shutdown_connections 2023-07-29T16:13:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.652+0000 7fe837fff700 1 -- 172.21.15.31:0/3907018134 wait complete. 2023-07-29T16:13:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.654+0000 7fe837fff700 1 Processor -- start 2023-07-29T16:13:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.654+0000 7fe837fff700 1 -- start start 2023-07-29T16:13:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.654+0000 7fe837fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8300737b0 0x7fe830198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.654+0000 7fe837fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe830198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe837fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8301081e0 0x7fe83019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe837fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83019fe80 con 0x7fe8301081e0 2023-07-29T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe837fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83019ffc0 con 0x7fe8300737b0 2023-07-29T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe837fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8301a0150 con 0x7fe830074d90 2023-07-29T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe83559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe830198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8300737b0 0x7fe830198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe83559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe830198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38882/0 (socket says 172.21.15.31:38882) 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe83559a700 1 -- 172.21.15.31:0/528608651 learned_addr learned my addr 172.21.15.31:0/528608651 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe83659c700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8301081e0 0x7fe83019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 -- 172.21.15.31:0/528608651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 msgr2=0x7fe830198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe830198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 -- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8301081e0 msgr2=0x7fe83019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8301081e0 0x7fe83019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 -- 172.21.15.31:0/528608651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8200095f0 con 0x7fe8300737b0 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe83559a700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe830198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.655+0000 7fe835d9b700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8300737b0 0x7fe830198300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00e9c0 tx=0x7fe82c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.656+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c00c9c0 con 0x7fe8300737b0 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.656+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8301a0430 con 0x7fe8300737b0 2023-07-29T16:13:45.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.656+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe82c010430 con 0x7fe8300737b0 2023-07-29T16:13:45.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.656+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82c018680 con 0x7fe8300737b0 2023-07-29T16:13:45.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.656+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8301a0990 con 0x7fe8300737b0 2023-07-29T16:13:45.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.657+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe830106890 con 0x7fe8300737b0 2023-07-29T16:13:45.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.659+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fe82c00cb20 con 0x7fe8300737b0 2023-07-29T16:13:45.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.660+0000 7fe826ffd700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe81c0a0820 0x7fe81c0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:45.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.660+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fe82c014070 con 0x7fe8300737b0 2023-07-29T16:13:45.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.660+0000 7fe83559a700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe81c0a0820 0x7fe81c0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:45.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.661+0000 7fe83559a700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe81c0a0820 0x7fe81c0a2cd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe82000b2e0 tx=0x7fe820005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.663+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe82c095770 con 0x7fe8300737b0 2023-07-29T16:13:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.872+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe83002cea0 con 0x7fe8300737b0 2023-07-29T16:13:45.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.873+0000 7fe826ffd700 1 -- 172.21.15.31:0/528608651 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe82c098d90 con 0x7fe8300737b0 2023-07-29T16:13:45.870 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:13:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe81c0a0820 msgr2=0x7fe81c0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe81c0a0820 0x7fe81c0a2cd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe82000b2e0 tx=0x7fe820005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8300737b0 msgr2=0x7fe830198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8300737b0 0x7fe830198300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe82c00e9c0 tx=0x7fe82c00ecd0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 shutdown_connections 2023-07-29T16:13:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe81c0a0820 0x7fe81c0a2cd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8301081e0 0x7fe83019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8300737b0 0x7fe830198300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 --2- 172.21.15.31:0/528608651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe830074d90 0x7fe830198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 >> 172.21.15.31:0/528608651 conn(0x7fe8300fb920 msgr2=0x7fe8300fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:45.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 shutdown_connections 2023-07-29T16:13:45.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:45.876+0000 7fe837fff700 1 -- 172.21.15.31:0/528608651 wait complete. 2023-07-29T16:13:45.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:45 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1451874344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:45.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:13:46.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:45 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1451874344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:46.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:45 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1451874344' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 -- 172.21.15.31:0/438286151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 msgr2=0x7fd3b0108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/438286151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 0x7fd3b0108770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8009f90 tx=0x7fd3a800b330 comp rx=0 tx=0).stop 2023-07-29T16:13:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 -- 172.21.15.31:0/438286151 shutdown_connections 2023-07-29T16:13:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/438286151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 0x7fd3b0108770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/438286151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/438286151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3b0101520 0x7fd3b0101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 -- 172.21.15.31:0/438286151 >> 172.21.15.31:0/438286151 conn(0x7fd3b00fb970 msgr2=0x7fd3b00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:46.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 -- 172.21.15.31:0/438286151 shutdown_connections 2023-07-29T16:13:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.100+0000 7fd3b59ff700 1 -- 172.21.15.31:0/438286151 wait complete. 2023-07-29T16:13:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.101+0000 7fd3b59ff700 1 Processor -- start 2023-07-29T16:13:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.101+0000 7fd3b59ff700 1 -- start start 2023-07-29T16:13:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.101+0000 7fd3b59ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.101+0000 7fd3b59ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3b0101520 0x7fd3b0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.101+0000 7fd3b59ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 0x7fd3b019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.101+0000 7fd3b59ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b019feb0 con 0x7fd3b0100330 2023-07-29T16:13:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3b59ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b019fff0 con 0x7fd3b0101eb0 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3b59ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b01a0180 con 0x7fd3b0101520 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60658/0 (socket says 172.21.15.31:60658) 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 -- 172.21.15.31:0/2577939588 learned_addr learned my addr 172.21.15.31:0/2577939588 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3ae7fc700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3b0101520 0x7fd3b0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3af7fe700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 0x7fd3b019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 -- 172.21.15.31:0/2577939588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3b0101520 msgr2=0x7fd3b0198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3b0101520 0x7fd3b0198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 -- 172.21.15.31:0/2577939588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 msgr2=0x7fd3b019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 0x7fd3b019f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 -- 172.21.15.31:0/2577939588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a00095f0 con 0x7fd3b0100330 2023-07-29T16:13:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.102+0000 7fd3aeffd700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0198330 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000ebf0 tx=0x7fd3a000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.103+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0009a10 con 0x7fd3b0100330 2023-07-29T16:13:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.103+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3a000cc90 con 0x7fd3b0100330 2023-07-29T16:13:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.103+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0018750 con 0x7fd3b0100330 2023-07-29T16:13:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.103+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8009c40 con 0x7fd3b0100330 2023-07-29T16:13:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.103+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b01a0790 con 0x7fd3b0100330 2023-07-29T16:13:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.104+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b010bb20 con 0x7fd3b0100330 2023-07-29T16:13:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.106+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fd3a00188b0 con 0x7fd3b0100330 2023-07-29T16:13:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.107+0000 7fd3b49fd700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3980a0820 0x7fd3980a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.107+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fd3a0014070 con 0x7fd3b0100330 2023-07-29T16:13:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.107+0000 7fd3ae7fc700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3980a0820 0x7fd3980a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.108+0000 7fd3ae7fc700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3980a0820 0x7fd3980a2cd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009c60 tx=0x7fd3a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.111+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd3a0095c40 con 0x7fd3b0100330 2023-07-29T16:13:46.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.318+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3b01a0dd0 con 0x7fd3b0100330 2023-07-29T16:13:46.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.319+0000 7fd3b49fd700 1 -- 172.21.15.31:0/2577939588 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3a0099260 con 0x7fd3b0100330 2023-07-29T16:13:46.316 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:13:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3980a0820 msgr2=0x7fd3980a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3980a0820 0x7fd3980a2cd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009c60 tx=0x7fd3a4009260 comp rx=0 tx=0).stop 2023-07-29T16:13:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 msgr2=0x7fd3b0198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0198330 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000ebf0 tx=0x7fd3a000ef00 comp rx=0 tx=0).stop 2023-07-29T16:13:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 shutdown_connections 2023-07-29T16:13:46.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3980a0820 0x7fd3980a2cd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3b0101eb0 0x7fd3b019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3b0100330 0x7fd3b0198330 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 --2- 172.21.15.31:0/2577939588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3b0101520 0x7fd3b0198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.321+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 >> 172.21.15.31:0/2577939588 conn(0x7fd3b00fb970 msgr2=0x7fd3b0106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.322+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 shutdown_connections 2023-07-29T16:13:46.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.322+0000 7fd3b59ff700 1 -- 172.21.15.31:0/2577939588 wait complete. 2023-07-29T16:13:46.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:13:46.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 -- 172.21.15.31:0/811206607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478100330 msgr2=0x7f2478100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 --2- 172.21.15.31:0/811206607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478100330 0x7f2478100740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2474009990 tx=0x7f2474009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 -- 172.21.15.31:0/811206607 shutdown_connections 2023-07-29T16:13:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 --2- 172.21.15.31:0/811206607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f2478108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 --2- 172.21.15.31:0/811206607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478100330 0x7f2478100740 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 --2- 172.21.15.31:0/811206607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478101520 0x7f2478101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 -- 172.21.15.31:0/811206607 >> 172.21.15.31:0/811206607 conn(0x7f24780fb970 msgr2=0x7f24780fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 -- 172.21.15.31:0/811206607 shutdown_connections 2023-07-29T16:13:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.547+0000 7f24805ff700 1 -- 172.21.15.31:0/811206607 wait complete. 2023-07-29T16:13:46.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 Processor -- start 2023-07-29T16:13:46.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 -- start start 2023-07-29T16:13:46.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478100330 0x7f24781960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478101520 0x7f24781965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f247819d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247819dc30 con 0x7f2478101eb0 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247819dd70 con 0x7f2478101520 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.548+0000 7f24805ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247819df00 con 0x7f2478100330 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247eb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f247819d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478100330 0x7f24781960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247eb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f247819d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60676/0 (socket says 172.21.15.31:60676) 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247eb9c700 1 -- 172.21.15.31:0/2347031009 learned_addr learned my addr 172.21.15.31:0/2347031009 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247db9a700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478101520 0x7f24781965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 -- 172.21.15.31:0/2347031009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478101520 msgr2=0x7f24781965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478101520 0x7f24781965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 -- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 msgr2=0x7f247819d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f247819d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 -- 172.21.15.31:0/2347031009 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24740095f0 con 0x7f2478100330 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247eb9c700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f247819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.549+0000 7f247e39b700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478100330 0x7f24781960b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f24740059d0 tx=0x7f2474005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.550+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247401d070 con 0x7f2478100330 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.550+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247819e180 con 0x7f2478100330 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.550+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2474022470 con 0x7f2478100330 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.550+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474017650 con 0x7f2478100330 2023-07-29T16:13:46.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.550+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247819e5a0 con 0x7f2478100330 2023-07-29T16:13:46.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.551+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24781078f0 con 0x7f2478100330 2023-07-29T16:13:46.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.553+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f2474022890 con 0x7f2478100330 2023-07-29T16:13:46.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.554+0000 7f246f7fe700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24640a0820 0x7f24640a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.554+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f24740cd870 con 0x7f2478100330 2023-07-29T16:13:46.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.554+0000 7f247db9a700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24640a0820 0x7f24640a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.555+0000 7f247db9a700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24640a0820 0x7f24640a2cd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2468004630 tx=0x7f246800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:46.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.556+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f24740971c0 con 0x7f2478100330 2023-07-29T16:13:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:46 smithi031 ceph-mon[95504]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:46 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/528608651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:46 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2577939588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.768+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f247802d0b0 con 0x7f2478100330 2023-07-29T16:13:46.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.769+0000 7f246f7fe700 1 -- 172.21.15.31:0/2347031009 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f247802d0b0 con 0x7f2478100330 2023-07-29T16:13:46.766 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:13:46.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24640a0820 msgr2=0x7f24640a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24640a0820 0x7f24640a2cd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2468004630 tx=0x7f246800b3c0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478100330 msgr2=0x7f24781960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478100330 0x7f24781960b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f24740059d0 tx=0x7f2474005740 comp rx=0 tx=0).stop 2023-07-29T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 shutdown_connections 2023-07-29T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f24640a0820 0x7f24640a2cd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2478101eb0 0x7f247819d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478100330 0x7f24781960b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 --2- 172.21.15.31:0/2347031009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2478101520 0x7f24781965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 >> 172.21.15.31:0/2347031009 conn(0x7f24780fb970 msgr2=0x7f2478104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 shutdown_connections 2023-07-29T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.771+0000 7f24805ff700 1 -- 172.21.15.31:0/2347031009 wait complete. 2023-07-29T16:13:46.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:13:46.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 -- 172.21.15.31:0/368231484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 msgr2=0x7f09d0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 --2- 172.21.15.31:0/368231484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d0108780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f09c800b0a0 tx=0x7f09c800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 -- 172.21.15.31:0/368231484 shutdown_connections 2023-07-29T16:13:46.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 --2- 172.21.15.31:0/368231484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d0108780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 --2- 172.21.15.31:0/368231484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0100340 0x7f09d0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 --2- 172.21.15.31:0/368231484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0101530 0x7f09d0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 -- 172.21.15.31:0/368231484 >> 172.21.15.31:0/368231484 conn(0x7f09d00fb920 msgr2=0x7f09d00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:46.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.993+0000 7f09d6bff700 1 -- 172.21.15.31:0/368231484 shutdown_connections 2023-07-29T16:13:46.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.994+0000 7f09d6bff700 1 -- 172.21.15.31:0/368231484 wait complete. 2023-07-29T16:13:46.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.994+0000 7f09d6bff700 1 Processor -- start 2023-07-29T16:13:46.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 -- start start 2023-07-29T16:13:46.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0100340 0x7f09d0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 0x7f09d0198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d019fed0 con 0x7f09d0101530 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01a0010 con 0x7f09d0101ec0 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d6bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d01a01a0 con 0x7f09d0100340 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09cffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 0x7f09d0198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09cffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 0x7f09d0198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60708/0 (socket says 172.21.15.31:60708) 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09cffff700 1 -- 172.21.15.31:0/1286266651 learned_addr learned my addr 172.21.15.31:0/1286266651 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:46.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d499b700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0100340 0x7f09d0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.995+0000 7f09d519c700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cffff700 1 -- 172.21.15.31:0/1286266651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0100340 msgr2=0x7f09d0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cffff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0100340 0x7f09d0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cffff700 1 -- 172.21.15.31:0/1286266651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 msgr2=0x7f09d019f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cffff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d019f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cffff700 1 -- 172.21.15.31:0/1286266651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8009d00 con 0x7f09d0101530 2023-07-29T16:13:46.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09d519c700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:46.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cffff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 0x7f09d0198890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00e9a0 tx=0x7f09bc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:46.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc009790 con 0x7f09d0101530 2023-07-29T16:13:46.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09bc00cbd0 con 0x7f09d0101530 2023-07-29T16:13:46.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d01a0480 con 0x7f09d0101530 2023-07-29T16:13:46.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc018680 con 0x7f09d0101530 2023-07-29T16:13:46.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.996+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d01a09e0 con 0x7f09d0101530 2023-07-29T16:13:46.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.997+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d010bb60 con 0x7f09d0101530 2023-07-29T16:13:46.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:46.999+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f09bc00cd40 con 0x7f09d0101530 2023-07-29T16:13:46.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.001+0000 7f09cdffb700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09c00a08e0 0x7f09c00a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:46.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.001+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f09bc014070 con 0x7f09d0101530 2023-07-29T16:13:46.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.001+0000 7f09d499b700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09c00a08e0 0x7f09c00a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:46.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.001+0000 7f09d499b700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09c00a08e0 0x7f09c00a2d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f09d0101390 tx=0x7f09c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:47.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.003+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f09bc095b20 con 0x7f09d0101530 2023-07-29T16:13:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:46 smithi039 ceph-mon[128858]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:47.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:46 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/528608651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:47.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:46 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2577939588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:47.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:46 smithi156 ceph-mon[129245]: pgmap v316: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-29T16:13:47.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:46 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/528608651' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:47.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:46 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2577939588' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:47.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.259+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d01a0610 con 0x7f09d0101530 2023-07-29T16:13:47.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.260+0000 7f09cdffb700 1 -- 172.21.15.31:0/1286266651 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f09d01a0610 con 0x7f09d0101530 2023-07-29T16:13:47.257 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:47.257 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:13:47.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09c00a08e0 msgr2=0x7f09c00a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09c00a08e0 0x7f09c00a2d90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f09d0101390 tx=0x7f09c4009260 comp rx=0 tx=0).stop 2023-07-29T16:13:47.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 msgr2=0x7f09d0198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 0x7f09d0198890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00e9a0 tx=0x7f09bc00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 shutdown_connections 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09c00a08e0 0x7f09c00a2d90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09d0101ec0 0x7f09d019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09d0100340 0x7f09d0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 --2- 172.21.15.31:0/1286266651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d0101530 0x7f09d0198890 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 >> 172.21.15.31:0/1286266651 conn(0x7f09d00fb920 msgr2=0x7f09d0106d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:47.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 shutdown_connections 2023-07-29T16:13:47.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.262+0000 7f09d6bff700 1 -- 172.21.15.31:0/1286266651 wait complete. 2023-07-29T16:13:47.261 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:13:47.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:47.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 -- 172.21.15.31:0/3598543194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac103750 msgr2=0x7fc9ac105b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/3598543194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac103750 0x7fc9ac105b30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009990 tx=0x7fc99c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 -- 172.21.15.31:0/3598543194 shutdown_connections 2023-07-29T16:13:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/3598543194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac106070 0x7fc9ac108450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/3598543194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9ac100e30 0x7fc9ac103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/3598543194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac103750 0x7fc9ac105b30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 -- 172.21.15.31:0/3598543194 >> 172.21.15.31:0/3598543194 conn(0x7fc9ac0fa810 msgr2=0x7fc9ac0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:47.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 -- 172.21.15.31:0/3598543194 shutdown_connections 2023-07-29T16:13:47.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.485+0000 7fc9b3dff700 1 -- 172.21.15.31:0/3598543194 wait complete. 2023-07-29T16:13:47.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.486+0000 7fc9b3dff700 1 Processor -- start 2023-07-29T16:13:47.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 -- start start 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 0x7fc9ac196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9ac103750 0x7fc9ac196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac106070 0x7fc9ac19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac19db90 con 0x7fc9ac100e30 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac19dcd0 con 0x7fc9ac106070 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b3dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ac19de60 con 0x7fc9ac103750 2023-07-29T16:13:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 0x7fc9ac196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 0x7fc9ac196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60736/0 (socket says 172.21.15.31:60736) 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b1b9b700 1 -- 172.21.15.31:0/2579201155 learned_addr learned my addr 172.21.15.31:0/2579201155 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b239c700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac106070 0x7fc9ac19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.487+0000 7fc9b139a700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9ac103750 0x7fc9ac196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b1b9b700 1 -- 172.21.15.31:0/2579201155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9ac103750 msgr2=0x7fc9ac196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b1b9b700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9ac103750 0x7fc9ac196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b1b9b700 1 -- 172.21.15.31:0/2579201155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac106070 msgr2=0x7fc9ac19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b1b9b700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac106070 0x7fc9ac19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b1b9b700 1 -- 172.21.15.31:0/2579201155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a4009c40 con 0x7fc9ac100e30 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b1b9b700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 0x7fc9ac196010 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800e9a0 tx=0x7fc9a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8009790 con 0x7fc9ac100e30 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c0095f0 con 0x7fc9ac100e30 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9a800ce80 con 0x7fc9ac100e30 2023-07-29T16:13:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8018680 con 0x7fc9ac100e30 2023-07-29T16:13:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.488+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ac19e4c0 con 0x7fc9ac100e30 2023-07-29T16:13:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.489+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ac0fc380 con 0x7fc9ac100e30 2023-07-29T16:13:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.491+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fc9a8010430 con 0x7fc9ac100e30 2023-07-29T16:13:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.492+0000 7fc9a2ffd700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9980a0860 0x7fc9980a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.493+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fc9a8014070 con 0x7fc9ac100e30 2023-07-29T16:13:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.493+0000 7fc9b139a700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9980a0860 0x7fc9980a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.493+0000 7fc9b139a700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9980a0860 0x7fc9980a2d10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00b2e0 tx=0x7fc99c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:47.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.496+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc9a8051720 con 0x7fc9ac100e30 2023-07-29T16:13:47.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.708+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9ac02cc30 con 0x7fc9ac100e30 2023-07-29T16:13:47.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.710+0000 7fc9a2ffd700 1 -- 172.21.15.31:0/2579201155 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22618 (secure 0 0 0) 0x7fc9a8095810 con 0x7fc9ac100e30 2023-07-29T16:13:47.707 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:47.707 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:12:50.274238+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9980a0860 msgr2=0x7fc9980a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9980a0860 0x7fc9980a2d10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00b2e0 tx=0x7fc99c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 msgr2=0x7fc9ac196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 0x7fc9ac196010 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800e9a0 tx=0x7fc9a800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 shutdown_connections 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9980a0860 0x7fc9980a2d10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9ac106070 0x7fc9ac19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc9ac100e30 0x7fc9ac196010 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 --2- 172.21.15.31:0/2579201155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc9ac103750 0x7fc9ac196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 >> 172.21.15.31:0/2579201155 conn(0x7fc9ac0fa810 msgr2=0x7fc9ac101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:47.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 shutdown_connections 2023-07-29T16:13:47.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.712+0000 7fc9b3dff700 1 -- 172.21.15.31:0/2579201155 wait complete. 2023-07-29T16:13:47.728 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:13:47.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:13:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.938+0000 7f4acefff700 1 -- 172.21.15.31:0/3304722167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101ea0 msgr2=0x7f4ac8108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.938+0000 7f4acefff700 1 --2- 172.21.15.31:0/3304722167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101ea0 0x7f4ac8108760 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4ac000b0a0 tx=0x7f4ac000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 -- 172.21.15.31:0/3304722167 shutdown_connections 2023-07-29T16:13:47.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 --2- 172.21.15.31:0/3304722167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101ea0 0x7f4ac8108760 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 --2- 172.21.15.31:0/3304722167 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 0x7f4ac8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 --2- 172.21.15.31:0/3304722167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ac8101510 0x7f4ac8101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 -- 172.21.15.31:0/3304722167 >> 172.21.15.31:0/3304722167 conn(0x7f4ac80fb920 msgr2=0x7f4ac80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:47.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 -- 172.21.15.31:0/3304722167 shutdown_connections 2023-07-29T16:13:47.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.939+0000 7f4acefff700 1 -- 172.21.15.31:0/3304722167 wait complete. 2023-07-29T16:13:47.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 Processor -- start 2023-07-29T16:13:47.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 -- start start 2023-07-29T16:13:47.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 0x7f4ac81961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 0x7f4ac8196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ac8101ea0 0x7f4ac819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac819dd40 con 0x7f4ac8101510 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac819de80 con 0x7f4ac8101ea0 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4acefff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac819e010 con 0x7f4ac8100320 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4ac7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 0x7f4ac8196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4accd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 0x7f4ac81961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4ac7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 0x7f4ac8196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60746/0 (socket says 172.21.15.31:60746) 2023-07-29T16:13:47.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.940+0000 7f4ac7fff700 1 -- 172.21.15.31:0/917026143 learned_addr learned my addr 172.21.15.31:0/917026143 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4acd59c700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ac8101ea0 0x7f4ac819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac7fff700 1 -- 172.21.15.31:0/917026143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 msgr2=0x7f4ac81961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac7fff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 0x7f4ac81961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac7fff700 1 -- 172.21.15.31:0/917026143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ac8101ea0 msgr2=0x7f4ac819d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac7fff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ac8101ea0 0x7f4ac819d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac7fff700 1 -- 172.21.15.31:0/917026143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac0009d00 con 0x7f4ac8101510 2023-07-29T16:13:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4accd9b700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 0x7f4ac81961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac7fff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 0x7f4ac8196700 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400d980 tx=0x7f4ab400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4004220 con 0x7f4ac8101510 2023-07-29T16:13:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac819e2f0 con 0x7f4ac8101510 2023-07-29T16:13:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ab4007640 con 0x7f4ac8101510 2023-07-29T16:13:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4005690 con 0x7f4ac8101510 2023-07-29T16:13:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.941+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac819e800 con 0x7f4ac8101510 2023-07-29T16:13:47.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.942+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac81078b0 con 0x7f4ac8101510 2023-07-29T16:13:47.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.944+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f4ab4004750 con 0x7f4ac8101510 2023-07-29T16:13:47.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.946+0000 7f4ac5ffb700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ab80a0820 0x7f4ab80a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:47.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.946+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f4ab40cba20 con 0x7f4ac8101510 2023-07-29T16:13:47.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.946+0000 7f4accd9b700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ab80a0820 0x7f4ab80a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:47.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.947+0000 7f4accd9b700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ab80a0820 0x7f4ab80a2cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8101370 tx=0x7f4abc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:47.950+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4ab40952e0 con 0x7f4ac8101510 2023-07-29T16:13:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:47 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2347031009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:48.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:47 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1286266651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:48.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:47 smithi039 ceph-mon[128858]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:48.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:47 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2579201155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:47 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2347031009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:47 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1286266651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:48.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:47 smithi156 ceph-mon[129245]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:48.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:47 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2579201155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.156+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ac802ce70 con 0x7f4ac8101510 2023-07-29T16:13:48.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.156+0000 7f4ac5ffb700 1 -- 172.21.15.31:0/917026143 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ab4016030 con 0x7f4ac8101510 2023-07-29T16:13:48.154 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:13:48.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.158+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ab80a0820 msgr2=0x7f4ab80a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.158+0000 7f4acefff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ab80a0820 0x7f4ab80a2cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8101370 tx=0x7f4abc009260 comp rx=0 tx=0).stop 2023-07-29T16:13:48.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.158+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 msgr2=0x7f4ac8196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.158+0000 7f4acefff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 0x7f4ac8196700 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400d980 tx=0x7f4ab400dc90 comp rx=0 tx=0).stop 2023-07-29T16:13:48.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.158+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 shutdown_connections 2023-07-29T16:13:48.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4ab80a0820 0x7f4ab80a2cd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4ac8101ea0 0x7f4ac819d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4ac8100320 0x7f4ac81961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 --2- 172.21.15.31:0/917026143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4ac8101510 0x7f4ac8196700 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 >> 172.21.15.31:0/917026143 conn(0x7f4ac80fb920 msgr2=0x7f4ac81049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 shutdown_connections 2023-07-29T16:13:48.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.159+0000 7f4acefff700 1 -- 172.21.15.31:0/917026143 wait complete. 2023-07-29T16:13:48.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:47 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2347031009' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:48.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:47 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1286266651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:48.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:47 smithi031 ceph-mon[95504]: pgmap v317: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:48.172 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:47 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2579201155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:48.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:13:48.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 -- 172.21.15.31:0/1349204551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 msgr2=0x7fe1f0101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/1349204551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0101970 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0009990 tx=0x7fe1e0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 -- 172.21.15.31:0/1349204551 shutdown_connections 2023-07-29T16:13:48.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/1349204551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1f0101eb0 0x7fe1f0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/1349204551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f0100330 0x7fe1f0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/1349204551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0101970 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 -- 172.21.15.31:0/1349204551 >> 172.21.15.31:0/1349204551 conn(0x7fe1f00fb970 msgr2=0x7fe1f00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:48.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 -- 172.21.15.31:0/1349204551 shutdown_connections 2023-07-29T16:13:48.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.382+0000 7fe1f87ff700 1 -- 172.21.15.31:0/1349204551 wait complete. 2023-07-29T16:13:48.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.383+0000 7fe1f87ff700 1 Processor -- start 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.383+0000 7fe1f87ff700 1 -- start start 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f87ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1f0100330 0x7fe1f0074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f87ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f87ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f0101eb0 0x7fe1f0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f87ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0073e20 con 0x7fe1f0101520 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f87ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0073f90 con 0x7fe1f0101eb0 2023-07-29T16:13:48.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f87ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f01a00f0 con 0x7fe1f0100330 2023-07-29T16:13:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60764/0 (socket says 172.21.15.31:60764) 2023-07-29T16:13:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 -- 172.21.15.31:0/2429405970 learned_addr learned my addr 172.21.15.31:0/2429405970 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f6d9c700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f0101eb0 0x7fe1f0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f659b700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1f0100330 0x7fe1f0074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 -- 172.21.15.31:0/2429405970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1f0100330 msgr2=0x7fe1f0074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1f0100330 0x7fe1f0074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 -- 172.21.15.31:0/2429405970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f0101eb0 msgr2=0x7fe1f0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f0101eb0 0x7fe1f0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.384+0000 7fe1f5d9a700 1 -- 172.21.15.31:0/2429405970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e00095f0 con 0x7fe1f0101520 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.385+0000 7fe1f5d9a700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0073140 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0004760 tx=0x7fe1e0004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.385+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e001d070 con 0x7fe1f0101520 2023-07-29T16:13:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.385+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f01a0370 con 0x7fe1f0101520 2023-07-29T16:13:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.385+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1e0004830 con 0x7fe1f0101520 2023-07-29T16:13:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.385+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e0005350 con 0x7fe1f0101520 2023-07-29T16:13:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.385+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f01a0740 con 0x7fe1f0101520 2023-07-29T16:13:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.386+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f010bb60 con 0x7fe1f0101520 2023-07-29T16:13:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.388+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fe1e0005530 con 0x7fe1f0101520 2023-07-29T16:13:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.389+0000 7fe1e77fe700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe1dc0a0820 0x7fe1dc0a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.389+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fe1e00cce00 con 0x7fe1f0101520 2023-07-29T16:13:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.390+0000 7fe1f659b700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe1dc0a0820 0x7fe1dc0a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.390+0000 7fe1f659b700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe1dc0a0820 0x7fe1dc0a2cd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0101380 tx=0x7fe1ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:48.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.393+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe1e005a720 con 0x7fe1f0101520 2023-07-29T16:13:48.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.601+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1f01a0e40 con 0x7fe1f0101520 2023-07-29T16:13:48.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.602+0000 7fe1e77fe700 1 -- 172.21.15.31:0/2429405970 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe1e0096750 con 0x7fe1f0101520 2023-07-29T16:13:48.600 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:13:48.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.604+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe1dc0a0820 msgr2=0x7fe1dc0a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.604+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe1dc0a0820 0x7fe1dc0a2cd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0101380 tx=0x7fe1ec009260 comp rx=0 tx=0).stop 2023-07-29T16:13:48.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.604+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 msgr2=0x7fe1f0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.604+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0073140 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe1e0004760 tx=0x7fe1e0004790 comp rx=0 tx=0).stop 2023-07-29T16:13:48.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 shutdown_connections 2023-07-29T16:13:48.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe1dc0a0820 0x7fe1dc0a2cd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1f0101eb0 0x7fe1f0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1f0100330 0x7fe1f0074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 --2- 172.21.15.31:0/2429405970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1f0101520 0x7fe1f0073140 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 >> 172.21.15.31:0/2429405970 conn(0x7fe1f00fb970 msgr2=0x7fe1f00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:48.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 shutdown_connections 2023-07-29T16:13:48.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.605+0000 7fe1f87ff700 1 -- 172.21.15.31:0/2429405970 wait complete. 2023-07-29T16:13:48.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:13:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 -- 172.21.15.31:0/673481619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101520 msgr2=0x7f40fc101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 --2- 172.21.15.31:0/673481619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101520 0x7f40fc101970 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009990 tx=0x7f40f0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 -- 172.21.15.31:0/673481619 shutdown_connections 2023-07-29T16:13:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 --2- 172.21.15.31:0/673481619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40fc101eb0 0x7f40fc108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 --2- 172.21.15.31:0/673481619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 --2- 172.21.15.31:0/673481619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101520 0x7f40fc101970 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 -- 172.21.15.31:0/673481619 >> 172.21.15.31:0/673481619 conn(0x7f40fc0fb970 msgr2=0x7f40fc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 -- 172.21.15.31:0/673481619 shutdown_connections 2023-07-29T16:13:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.829+0000 7f41023ff700 1 -- 172.21.15.31:0/673481619 wait complete. 2023-07-29T16:13:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.830+0000 7f41023ff700 1 Processor -- start 2023-07-29T16:13:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 -- start start 2023-07-29T16:13:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40fc101520 0x7f40fc1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101eb0 0x7f40fc19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc19fe20 con 0x7f40fc101eb0 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc19ff60 con 0x7f40fc100330 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f41023ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1a00f0 con 0x7f40fc101520 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f410099c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101eb0 0x7f40fc19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f40fbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f410099c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101eb0 0x7f40fc19f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60786/0 (socket says 172.21.15.31:60786) 2023-07-29T16:13:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f410099c700 1 -- 172.21.15.31:0/2541108130 learned_addr learned my addr 172.21.15.31:0/2541108130 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f40fbfff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc1982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56276/0 (socket says 172.21.15.31:56276) 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.831+0000 7f40fb7fe700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40fc101520 0x7f40fc1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40fbfff700 1 -- 172.21.15.31:0/2541108130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40fc101520 msgr2=0x7f40fc1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40fbfff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40fc101520 0x7f40fc1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40fbfff700 1 -- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101eb0 msgr2=0x7f40fc19f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40fbfff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101eb0 0x7f40fc19f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40fbfff700 1 -- 172.21.15.31:0/2541108130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f00095f0 con 0x7f40fc100330 2023-07-29T16:13:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40fbfff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc1982a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00e9c0 tx=0x7f40ec00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec004220 con 0x7f40fc100330 2023-07-29T16:13:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc1a03d0 con 0x7f40fc100330 2023-07-29T16:13:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40ec00cc80 con 0x7f40fc100330 2023-07-29T16:13:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec003be0 con 0x7f40fc100330 2023-07-29T16:13:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.832+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc1a0930 con 0x7f40fc100330 2023-07-29T16:13:48.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.833+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40fc10bb60 con 0x7f40fc100330 2023-07-29T16:13:48.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.835+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7f40ec003d40 con 0x7f40fc100330 2023-07-29T16:13:48.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.837+0000 7f40f97fa700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40e40a0820 0x7f40e40a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.837+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7f40ec014070 con 0x7f40fc100330 2023-07-29T16:13:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.837+0000 7f40fb7fe700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40e40a0820 0x7f40e40a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:48.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.837+0000 7f40fb7fe700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40e40a0820 0x7f40e40a2cd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f40f000b2e0 tx=0x7f40f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:48.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:48.839+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f40ec095aa0 con 0x7f40fc100330 2023-07-29T16:13:48.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:48 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/917026143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:48.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:48 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2429405970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:49.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:48 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/917026143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:49.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:48 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2429405970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.049+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40fc10bd20 con 0x7f40fc100330 2023-07-29T16:13:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.050+0000 7f40f97fa700 1 -- 172.21.15.31:0/2541108130 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40ec0990c0 con 0x7f40fc100330 2023-07-29T16:13:49.047 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:13:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40e40a0820 msgr2=0x7f40e40a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40e40a0820 0x7f40e40a2cd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f40f000b2e0 tx=0x7f40f0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 msgr2=0x7f40fc1982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc1982a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00e9c0 tx=0x7f40ec00ecd0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 shutdown_connections 2023-07-29T16:13:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40e40a0820 0x7f40e40a2cd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40fc101eb0 0x7f40fc19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f40fc100330 0x7f40fc1982a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 --2- 172.21.15.31:0/2541108130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40fc101520 0x7f40fc1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 >> 172.21.15.31:0/2541108130 conn(0x7f40fc0fb970 msgr2=0x7f40fc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 shutdown_connections 2023-07-29T16:13:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.053+0000 7f41023ff700 1 -- 172.21.15.31:0/2541108130 wait complete. 2023-07-29T16:13:49.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:13:49.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:48 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/917026143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:49.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:48 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2429405970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:13:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 -- 172.21.15.31:0/670858952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e00737b0 msgr2=0x7fd6e0073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/670858952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e00737b0 0x7fd6e0073c20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009990 tx=0x7fd6d0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 -- 172.21.15.31:0/670858952 shutdown_connections 2023-07-29T16:13:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/670858952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0108200 0x7fd6e010a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/670858952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e0074d90 0x7fd6e00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/670858952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e00737b0 0x7fd6e0073c20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.275+0000 7fd6e85ff700 1 -- 172.21.15.31:0/670858952 >> 172.21.15.31:0/670858952 conn(0x7fd6e00fb920 msgr2=0x7fd6e00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:49.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.276+0000 7fd6e85ff700 1 -- 172.21.15.31:0/670858952 shutdown_connections 2023-07-29T16:13:49.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.276+0000 7fd6e85ff700 1 -- 172.21.15.31:0/670858952 wait complete. 2023-07-29T16:13:49.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.277+0000 7fd6e85ff700 1 Processor -- start 2023-07-29T16:13:49.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.277+0000 7fd6e85ff700 1 -- start start 2023-07-29T16:13:49.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e85ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 0x7fd6e01982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e85ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0074d90 0x7fd6e01987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e85ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e0108200 0x7fd6e019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e85ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e019fe20 con 0x7fd6e0074d90 2023-07-29T16:13:49.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e85ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e019ff60 con 0x7fd6e0108200 2023-07-29T16:13:49.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e85ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e01a00f0 con 0x7fd6e00737b0 2023-07-29T16:13:49.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 0x7fd6e01982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e5b9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0074d90 0x7fd6e01987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e6b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e0108200 0x7fd6e019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 0x7fd6e01982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38990/0 (socket says 172.21.15.31:38990) 2023-07-29T16:13:49.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e639b700 1 -- 172.21.15.31:0/1611714705 learned_addr learned my addr 172.21.15.31:0/1611714705 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:49.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.278+0000 7fd6e639b700 1 -- 172.21.15.31:0/1611714705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e0108200 msgr2=0x7fd6e019f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e639b700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e0108200 0x7fd6e019f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e639b700 1 -- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0074d90 msgr2=0x7fd6e01987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e639b700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0074d90 0x7fd6e01987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e639b700 1 -- 172.21.15.31:0/1611714705 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d00095f0 con 0x7fd6e00737b0 2023-07-29T16:13:49.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e5b9a700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0074d90 0x7fd6e01987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:49.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e639b700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 0x7fd6e01982a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00e970 tx=0x7fd6dc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:49.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc009790 con 0x7fd6e00737b0 2023-07-29T16:13:49.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6dc00c7e0 con 0x7fd6e00737b0 2023-07-29T16:13:49.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0054e0 con 0x7fd6e00737b0 2023-07-29T16:13:49.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.279+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e01a03d0 con 0x7fd6e00737b0 2023-07-29T16:13:49.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.280+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e01a0890 con 0x7fd6e00737b0 2023-07-29T16:13:49.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.281+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e01068b0 con 0x7fd6e00737b0 2023-07-29T16:13:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.284+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) v1 ==== 134524+0+0 (secure 0 0 0) 0x7fd6dc00c950 con 0x7fd6e00737b0 2023-07-29T16:13:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.286+0000 7fd6d77fe700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6cc0a08e0 0x7fd6cc0a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.287+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8269+0+0 (secure 0 0 0) 0x7fd6dc014070 con 0x7fd6e00737b0 2023-07-29T16:13:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.290+0000 7fd6e5b9a700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6cc0a08e0 0x7fd6cc0a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.291+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd6dc095a30 con 0x7fd6e00737b0 2023-07-29T16:13:49.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.291+0000 7fd6e5b9a700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6cc0a08e0 0x7fd6cc0a2d90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000b2e0 tx=0x7fd6d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:49.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.573+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e002cea0 con 0x7fd6e00737b0 2023-07-29T16:13:49.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.574+0000 7fd6d77fe700 1 -- 172.21.15.31:0/1611714705 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7fd6dc099050 con 0x7fd6e00737b0 2023-07-29T16:13:49.571 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:49.572 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:13:49.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6cc0a08e0 msgr2=0x7fd6cc0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6cc0a08e0 0x7fd6cc0a2d90 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd6d000b2e0 tx=0x7fd6d0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 msgr2=0x7fd6e01982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 0x7fd6e01982a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00e970 tx=0x7fd6dc00ec80 comp rx=0 tx=0).stop 2023-07-29T16:13:49.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 shutdown_connections 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6cc0a08e0 0x7fd6cc0a2d90 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6e0108200 0x7fd6e019f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6e00737b0 0x7fd6e01982a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 --2- 172.21.15.31:0/1611714705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6e0074d90 0x7fd6e01987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 >> 172.21.15.31:0/1611714705 conn(0x7fd6e00fb920 msgr2=0x7fd6e010a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 shutdown_connections 2023-07-29T16:13:49.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.577+0000 7fd6e85ff700 1 -- 172.21.15.31:0/1611714705 wait complete. 2023-07-29T16:13:49.576 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:13:49.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 -- 172.21.15.31:0/2831824989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c101ec0 msgr2=0x7f3b0c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 --2- 172.21.15.31:0/2831824989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c108780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400b0a0 tx=0x7f3b0400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 -- 172.21.15.31:0/2831824989 shutdown_connections 2023-07-29T16:13:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 --2- 172.21.15.31:0/2831824989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c108780 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 --2- 172.21.15.31:0/2831824989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c100340 0x7f3b0c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 --2- 172.21.15.31:0/2831824989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 0x7f3b0c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 -- 172.21.15.31:0/2831824989 >> 172.21.15.31:0/2831824989 conn(0x7f3b0c0fb920 msgr2=0x7f3b0c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 -- 172.21.15.31:0/2831824989 shutdown_connections 2023-07-29T16:13:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.806+0000 7f3b13bff700 1 -- 172.21.15.31:0/2831824989 wait complete. 2023-07-29T16:13:49.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.807+0000 7f3b13bff700 1 Processor -- start 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 -- start start 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 0x7f3b0c074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 0x7f3b0c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c073e20 con 0x7f3b0c100340 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c073f90 con 0x7f3b0c101ec0 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b13bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0c1a01a0 con 0x7f3b0c101530 2023-07-29T16:13:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b1199b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 0x7f3b0c074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b1199b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 0x7f3b0c074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60838/0 (socket says 172.21.15.31:60838) 2023-07-29T16:13:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b1199b700 1 -- 172.21.15.31:0/781333760 learned_addr learned my addr 172.21.15.31:0/781333760 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b1219c700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.808+0000 7f3b1119a700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 0x7f3b0c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1219c700 1 -- 172.21.15.31:0/781333760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 msgr2=0x7f3b0c073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1219c700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 0x7f3b0c073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1219c700 1 -- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 msgr2=0x7f3b0c074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1219c700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 0x7f3b0c074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1219c700 1 -- 172.21.15.31:0/781333760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b04009d00 con 0x7f3b0c101ec0 2023-07-29T16:13:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1119a700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 0x7f3b0c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1199b700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 0x7f3b0c074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b1219c700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c073680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400b070 tx=0x7f3b0400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0400e040 con 0x7f3b0c101ec0 2023-07-29T16:13:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b04009480 con 0x7f3b0c101ec0 2023-07-29T16:13:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0c1a0420 con 0x7f3b0c101ec0 2023-07-29T16:13:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b040052a0 con 0x7f3b0c101ec0 2023-07-29T16:13:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.809+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c1a0840 con 0x7f3b0c101ec0 2023-07-29T16:13:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.810+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0c107790 con 0x7f3b0c101ec0 2023-07-29T16:13:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.812+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3b0401d480 con 0x7f3b0c101ec0 2023-07-29T16:13:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.813+0000 7f3b02ffd700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3af80a0860 0x7f3af80a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.814+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b040ccf10 con 0x7f3b0c101ec0 2023-07-29T16:13:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.814+0000 7f3b1199b700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3af80a0860 0x7f3af80a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.814+0000 7f3b1199b700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3af80a0860 0x7f3af80a2d10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3b08005950 tx=0x7f3b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:49.815+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3b04096980 con 0x7f3b0c101ec0 2023-07-29T16:13:49.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:49 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2541108130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:49.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2023-07-29T16:13:49.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:49 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1, 5, 11]}]: dispatch 2023-07-29T16:13:49.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:49 smithi031 ceph-mon[95504]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:49.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:49 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1611714705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:50.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.030+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b0c02cc30 con 0x7f3b0c101ec0 2023-07-29T16:13:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.033+0000 7f3b02ffd700 1 -- 172.21.15.31:0/781333760 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f3b04004030 con 0x7f3b0c101ec0 2023-07-29T16:13:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:50.031 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":89,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:13:49.769754+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3af80a0860 msgr2=0x7f3af80a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3af80a0860 0x7f3af80a2d10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3b08005950 tx=0x7f3b08009260 comp rx=0 tx=0).stop 2023-07-29T16:13:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c101ec0 msgr2=0x7f3b0c073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c073680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400b070 tx=0x7f3b0400bc80 comp rx=0 tx=0).stop 2023-07-29T16:13:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 shutdown_connections 2023-07-29T16:13:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3af80a0860 0x7f3af80a2d10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b0c101ec0 0x7f3b0c073680 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b0c100340 0x7f3b0c074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 --2- 172.21.15.31:0/781333760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b0c101530 0x7f3b0c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 >> 172.21.15.31:0/781333760 conn(0x7f3b0c0fb920 msgr2=0x7f3b0c1048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 shutdown_connections 2023-07-29T16:13:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.035+0000 7f3b13bff700 1 -- 172.21.15.31:0/781333760 wait complete. 2023-07-29T16:13:50.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-29T16:12:21.912364+0000', 'modified': '2023-07-29T16:12:27.740070+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24581, 'mds_1': 24551, 'mds_2': 24550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/1642153321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 1642153321}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 1642153321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.39:6837/1297587845', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6836', 'nonce': 1297587845}, {'type': 'v1', 'addr': '172.21.15.39:6837', 'nonce': 1297587845}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6837/809205544', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 809205544}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 809205544}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24550': {'gid': 24550, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.156:6839/2888902152', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 2888902152}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 2888902152}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24551': {'gid': 24551, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.31:6839/3329463675', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 3329463675}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 3329463675}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24581': {'gid': 24581, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.39:6839/2751828361', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6838', 'nonce': 2751828361}, {'type': 'v1', 'addr': '172.21.15.39:6839', 'nonce': 2751828361}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-29T16:13:50.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-29T16:13:50.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-29T16:13:50.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:49 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2541108130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:50.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2023-07-29T16:13:50.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:49 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1, 5, 11]}]: dispatch 2023-07-29T16:13:50.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:49 smithi156 ceph-mon[129245]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:50.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:49 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1611714705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:50.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 -- 172.21.15.31:0/3057604494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 msgr2=0x7fca6c073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 --2- 172.21.15.31:0/3057604494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c073160 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009990 tx=0x7fca5c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 -- 172.21.15.31:0/3057604494 shutdown_connections 2023-07-29T16:13:50.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 --2- 172.21.15.31:0/3057604494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c1081e0 0x7fca6c10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 --2- 172.21.15.31:0/3057604494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c073160 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 --2- 172.21.15.31:0/3057604494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca6c073730 0x7fca6c073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 -- 172.21.15.31:0/3057604494 >> 172.21.15.31:0/3057604494 conn(0x7fca6c0fb970 msgr2=0x7fca6c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 -- 172.21.15.31:0/3057604494 shutdown_connections 2023-07-29T16:13:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.260+0000 7fca711ff700 1 -- 172.21.15.31:0/3057604494 wait complete. 2023-07-29T16:13:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.261+0000 7fca711ff700 1 Processor -- start 2023-07-29T16:13:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 -- start start 2023-07-29T16:13:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 0x7fca6c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c1987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca6c1081e0 0x7fca6c19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c19fe20 con 0x7fca6c074d10 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c19ff60 con 0x7fca6c073730 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca711ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6c1a00f0 con 0x7fca6c1081e0 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca6ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 0x7fca6c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c1987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca6ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 0x7fca6c1982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40932/0 (socket says 172.21.15.31:40932) 2023-07-29T16:13:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca6ad9d700 1 -- 172.21.15.31:0/3220735218 learned_addr learned my addr 172.21.15.31:0/3220735218 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c1987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49656/0 (socket says 172.21.15.31:49656) 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.262+0000 7fca6b59e700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca6c1081e0 0x7fca6c19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6ad9d700 1 -- 172.21.15.31:0/3220735218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca6c1081e0 msgr2=0x7fca6c19f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca6c1081e0 0x7fca6c19f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6ad9d700 1 -- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 msgr2=0x7fca6c1987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c1987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6ad9d700 1 -- 172.21.15.31:0/3220735218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca64009c40 con 0x7fca6c073730 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6a59c700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 0x7fca6c1982a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fca5c004740 tx=0x7fca5c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca53fff700 1 -- 172.21.15.31:0/3220735218 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c01d070 con 0x7fca6c073730 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca53fff700 1 -- 172.21.15.31:0/3220735218 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca5c004830 con 0x7fca6c073730 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c0095f0 con 0x7fca6c073730 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca53fff700 1 -- 172.21.15.31:0/3220735218 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0053c0 con 0x7fca6c073730 2023-07-29T16:13:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.263+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c1a05a0 con 0x7fca6c073730 2023-07-29T16:13:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.267+0000 7fca53fff700 1 -- 172.21.15.31:0/3220735218 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fca5c005580 con 0x7fca6c073730 2023-07-29T16:13:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.267+0000 7fca84b5bb80 1 Processor -- start 2023-07-29T16:13:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 -- start start 2023-07-29T16:13:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e563b90 0x55e62e563fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e564d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 0x55e62e5656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e4bb9c0 con 0x55e62e5644e0 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e4b9380 con 0x55e62e5652a0 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e4b4c70 con 0x55e62e563b90 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca53fff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca540a0790 0x7fca540a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 0x55e62e5656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca53fff700 1 -- 172.21.15.31:0/3220735218 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca5c0ccc90 con 0x7fca6c073730 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 0x55e62e5656f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40940/0 (socket says 172.21.15.31:40940) 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca6a59c700 1 -- 172.21.15.31:0/1458299826 learned_addr learned my addr 172.21.15.31:0/1458299826 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.268+0000 7fca6b59e700 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e563b90 0x55e62e563fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6b59e700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca540a0790 0x7fca540a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e564d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6a59c700 1 -- 172.21.15.31:0/1458299826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e563b90 msgr2=0x55e62e563fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6a59c700 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e563b90 0x55e62e563fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6a59c700 1 -- 172.21.15.31:0/1458299826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 msgr2=0x55e62e564d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6a59c700 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e564d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6a59c700 1 -- 172.21.15.31:0/1458299826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e62e2e2d90 con 0x55e62e5652a0 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6b59e700 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e563b90 0x55e62e563fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6a59c700 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 0x55e62e5656f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fca600115f0 tx=0x7fca60011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15fa697d7e5e0ef9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca6b59e700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca540a0790 0x7fca540a2c40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55e62e563930 tx=0x7fca6400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca517fa700 1 -- 172.21.15.31:0/1458299826 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6000f680 con 0x55e62e5652a0 2023-07-29T16:13:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca517fa700 1 -- 172.21.15.31:0/1458299826 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca6000fca0 con 0x55e62e5652a0 2023-07-29T16:13:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.269+0000 7fca517fa700 1 -- 172.21.15.31:0/1458299826 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60019cb0 con 0x55e62e5652a0 2023-07-29T16:13:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 -- 172.21.15.31:0/1458299826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 msgr2=0x55e62e5656f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 0x55e62e5656f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fca600115f0 tx=0x7fca60011900 comp rx=0 tx=0).stop 2023-07-29T16:13:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 -- 172.21.15.31:0/1458299826 shutdown_connections 2023-07-29T16:13:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5652a0 0x55e62e5656f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e563b90 0x55e62e563fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/1458299826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e564d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 -- 172.21.15.31:0/1458299826 >> 172.21.15.31:0/1458299826 conn(0x55e62e559390 msgr2=0x55e62e55e940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 -- 172.21.15.31:0/1458299826 shutdown_connections 2023-07-29T16:13:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.270+0000 7fca84b5bb80 1 -- 172.21.15.31:0/1458299826 wait complete. 2023-07-29T16:13:50.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.271+0000 7fca84b5bb80 1 -- start start 2023-07-29T16:13:50.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:49 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2541108130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:13:50.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]: dispatch 2023-07-29T16:13:50.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:49 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1, 5, 11]}]: dispatch 2023-07-29T16:13:50.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:49 smithi039 ceph-mon[128858]: pgmap v318: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-07-29T16:13:50.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:49 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1611714705' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:13:50.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.271+0000 7fca84b5bb80 1 Processor -- start 2023-07-29T16:13:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e563b90 0x55e62e56d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e56dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e574d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e4514c0 con 0x55e62e5644e0 2023-07-29T16:13:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e451160 con 0x55e62e563b90 2023-07-29T16:13:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e3388d0 con 0x55e62e5652a0 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca84b5bb80 10 client.0 _ll_register_callbacks cb 0x55e62e45a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e56dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e56dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49678/0 (socket says 172.21.15.31:49678) 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca6a59c700 1 -- 172.21.15.31:0/222617364 learned_addr learned my addr 172.21.15.31:0/222617364 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e563b90 0x55e62e56d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.272+0000 7fca6b59e700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e574d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6a59c700 1 -- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 msgr2=0x55e62e574d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6a59c700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e574d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6a59c700 1 -- 172.21.15.31:0/222617364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e563b90 msgr2=0x55e62e56d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6a59c700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e563b90 0x55e62e56d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6a59c700 1 -- 172.21.15.31:0/222617364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca60009dd0 con 0x55e62e5644e0 2023-07-29T16:13:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e563b90 0x55e62e56d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca6a59c700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e56dcf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fca6001c040 tx=0x7fca6001af20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5102266d87c622 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca4eff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca4eff5700 1 -- 172.21.15.31:0/222617364 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60004440 con 0x55e62e5644e0 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e62e4dfcc0 con 0x55e62e5644e0 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca4eff5700 1 -- 172.21.15.31:0/222617364 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca60005050 con 0x55e62e5644e0 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.273+0000 7fca4eff5700 1 -- 172.21.15.31:0/222617364 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca600045a0 con 0x55e62e5644e0 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 10 client.15147 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e62e2cbc10 con 0x55e62e5644e0 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca4eff5700 1 -- 172.21.15.31:0/222617364 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e62e4dfcc0 con 0x55e62e5644e0 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca4eff5700 1 -- 172.21.15.31:0/222617364 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca60004750 con 0x55e62e5644e0 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 10 client.15147 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 10 client.15147 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 20 client.15147 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e578970 0x55e62e577dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 4 client.15147 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e62e532d20 con 0x55e62e578970 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.274+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e578970 0x55e62e577dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.275+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e578970 0x55e62e577dd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c40f02895aa63032 server_cookie=917742f513bf1a93 in_seq=0 out_seq=0 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.275+0000 7fca4eff5700 10 client.15147 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca4eff5700 1 -- 172.21.15.31:0/222617364 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e62e532d20 con 0x55e62e578970 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca4eff5700 10 client.15147 handle_command_reply: tid=0 2023-07-29T16:13:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 client.15147 shutdown 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 msgr2=0x55e62e56dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e56dcf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fca6001c040 tx=0x7fca6001af20 comp rx=0 tx=0).stop 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 shutdown_connections 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e574d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e578970 0x55e62e577dd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e563b90 0x55e62e56d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/222617364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e5644e0 0x55e62e56dcf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca4eff5700 0 client.15147 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 >> 172.21.15.31:0/222617364 conn(0x55e62e559390 msgr2=0x55e62e55e010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 shutdown_connections 2023-07-29T16:13:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 1 -- 172.21.15.31:0/222617364 wait complete. 2023-07-29T16:13:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.276+0000 7fca84b5bb80 20 client.15147 trim_cache size 0 max 16384 2023-07-29T16:13:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.294+0000 7fca84b5bb80 1 Processor -- start 2023-07-29T16:13:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.294+0000 7fca84b5bb80 1 -- start start 2023-07-29T16:13:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e55e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e56c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e4b9cf0 con 0x55e62e563b90 2023-07-29T16:13:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e454440 con 0x55e62e5644e0 2023-07-29T16:13:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e45e830 con 0x55e62e5652a0 2023-07-29T16:13:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e55e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e55e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49688/0 (socket says 172.21.15.31:49688) 2023-07-29T16:13:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 -- 172.21.15.31:0/490828439 learned_addr learned my addr 172.21.15.31:0/490828439 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6a59c700 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6b59e700 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e56c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 -- 172.21.15.31:0/490828439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 msgr2=0x55e62e56c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e56c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 -- 172.21.15.31:0/490828439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 msgr2=0x55e62e574cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.295+0000 7fca6ad9d700 1 -- 172.21.15.31:0/490828439 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e62e309200 con 0x55e62e563b90 2023-07-29T16:13:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.296+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e55e010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca5c096de0 tx=0x7fca5c09d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f628b347fb857972 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.296+0000 7fca51ffb700 1 -- 172.21.15.31:0/490828439 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0b1070 con 0x55e62e563b90 2023-07-29T16:13:50.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.296+0000 7fca51ffb700 1 -- 172.21.15.31:0/490828439 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca5c09e3f0 con 0x55e62e563b90 2023-07-29T16:13:50.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.296+0000 7fca51ffb700 1 -- 172.21.15.31:0/490828439 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0ca7e0 con 0x55e62e563b90 2023-07-29T16:13:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.296+0000 7fca84b5bb80 1 -- 172.21.15.31:0/490828439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 msgr2=0x55e62e55e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.296+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e55e010 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fca5c096de0 tx=0x7fca5c09d040 comp rx=0 tx=0).stop 2023-07-29T16:13:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 -- 172.21.15.31:0/490828439 shutdown_connections 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e56c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e55e010 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/490828439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 -- 172.21.15.31:0/490828439 >> 172.21.15.31:0/490828439 conn(0x55e62e559390 msgr2=0x55e62e55e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 -- 172.21.15.31:0/490828439 shutdown_connections 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 -- 172.21.15.31:0/490828439 wait complete. 2023-07-29T16:13:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 -- start start 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.297+0000 7fca84b5bb80 1 Processor -- start 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e571d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e5747b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e574ed0 con 0x55e62e563b90 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e575040 con 0x55e62e5644e0 2023-07-29T16:13:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62e5ca4e0 con 0x55e62e5652a0 2023-07-29T16:13:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca84b5bb80 10 client.0 _ll_register_callbacks cb 0x55e62e45a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca6ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e571d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca6ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e571d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49700/0 (socket says 172.21.15.31:49700) 2023-07-29T16:13:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca6ad9d700 1 -- 172.21.15.31:0/80476715 learned_addr learned my addr 172.21.15.31:0/80476715 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.298+0000 7fca6b59e700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e5747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6a59c700 1 -- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 msgr2=0x55e62e5747b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6a59c700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e5747b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6a59c700 1 -- 172.21.15.31:0/80476715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 msgr2=0x55e62e571d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6a59c700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e571d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6a59c700 1 -- 172.21.15.31:0/80476715 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6c1a05a0 con 0x55e62e5644e0 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6b59e700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e5747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca6a59c700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fca60005850 tx=0x7fca6000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1980326bf8dc3307 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca4eff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca4eff5700 1 -- 172.21.15.31:0/80476715 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60004440 con 0x55e62e5644e0 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca4eff5700 1 -- 172.21.15.31:0/80476715 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca60000e40 con 0x55e62e5644e0 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca6000fca0 con 0x55e62e5644e0 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca4eff5700 1 -- 172.21.15.31:0/80476715 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca60004d70 con 0x55e62e5644e0 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.299+0000 7fca84b5bb80 10 client.25133 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e62e2e1260 con 0x55e62e5644e0 2023-07-29T16:13:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca4eff5700 1 -- 172.21.15.31:0/80476715 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca60044020 con 0x55e62e5644e0 2023-07-29T16:13:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca4eff5700 1 -- 172.21.15.31:0/80476715 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca600045a0 con 0x55e62e5644e0 2023-07-29T16:13:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 10 client.25133 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 10 client.25133 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-29T16:13:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 20 client.25133 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:50.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e5ceee0 0x55e62e5cb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 4 client.25133 mds_command: new command op to 24550 tid=0[{"prefix": "status"}] 2023-07-29T16:13:50.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e62e4adcf0 con 0x55e62e5ceee0 2023-07-29T16:13:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.300+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e5ceee0 0x55e62e5cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca6ad9d700 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e5ceee0 0x55e62e5cb850 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c33296ad4fc1882 server_cookie=6ca469cd09ac33f1 in_seq=0 out_seq=0 2023-07-29T16:13:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca4eff5700 10 client.25133 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca4eff5700 1 -- 172.21.15.31:0/80476715 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55e62e4adcf0 con 0x55e62e5ceee0 2023-07-29T16:13:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca4eff5700 10 client.25133 handle_command_reply: tid=0 2023-07-29T16:13:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 client.25133 shutdown 2023-07-29T16:13:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 msgr2=0x55e62e574270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fca60005850 tx=0x7fca6000f730 comp rx=0 tx=0).stop 2023-07-29T16:13:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 shutdown_connections 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e62e5652a0 0x55e62e5747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e62e563b90 0x55e62e571d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.301+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e62e5ceee0 0x55e62e5cb850 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca84b5bb80 1 --2- 172.21.15.31:0/80476715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e62e5644e0 0x55e62e574270 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca4eff5700 0 client.25133 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 >> 172.21.15.31:0/80476715 conn(0x55e62e559390 msgr2=0x55e62e577dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 shutdown_connections 2023-07-29T16:13:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca84b5bb80 1 -- 172.21.15.31:0/80476715 wait complete. 2023-07-29T16:13:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca84b5bb80 20 client.25133 trim_cache size 0 max 16384 2023-07-29T16:13:50.312 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:13:50.312 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:13:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2023-07-29T16:13:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24550, 2023-07-29T16:13:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-07-29T16:13:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-07-29T16:13:50.313 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-07-29T16:13:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 83.819438433000002, 2023-07-29T16:13:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2023-07-29T16:13:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2023-07-29T16:13:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 86, 2023-07-29T16:13:50.314 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 100.878933821 2023-07-29T16:13:50.314 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:13:50.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca540a0790 msgr2=0x7fca540a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca540a0790 0x7fca540a2c40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55e62e563930 tx=0x7fca6400b040 comp rx=0 tx=0).stop 2023-07-29T16:13:50.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 msgr2=0x7fca6c1982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 0x7fca6c1982a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fca5c004740 tx=0x7fca5c009580 comp rx=0 tx=0).stop 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 shutdown_connections 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca540a0790 0x7fca540a2c40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca6c1081e0 0x7fca6c19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca6c073730 0x7fca6c1982a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 --2- 172.21.15.31:0/3220735218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca6c074d10 0x7fca6c1987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.302+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 >> 172.21.15.31:0/3220735218 conn(0x7fca6c0fb970 msgr2=0x7fca6c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.303+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 shutdown_connections 2023-07-29T16:13:50.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.303+0000 7fca711ff700 1 -- 172.21.15.31:0/3220735218 wait complete. 2023-07-29T16:13:50.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2023-07-29T16:13:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 -- 172.21.15.31:0/4289031208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c100330 msgr2=0x7fde5c100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 --2- 172.21.15.31:0/4289031208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c100330 0x7fde5c100740 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fde4c009990 tx=0x7fde4c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 -- 172.21.15.31:0/4289031208 shutdown_connections 2023-07-29T16:13:50.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 --2- 172.21.15.31:0/4289031208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c101eb0 0x7fde5c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 --2- 172.21.15.31:0/4289031208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c100330 0x7fde5c100740 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 --2- 172.21.15.31:0/4289031208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 0x7fde5c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 -- 172.21.15.31:0/4289031208 >> 172.21.15.31:0/4289031208 conn(0x7fde5c0fb970 msgr2=0x7fde5c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 -- 172.21.15.31:0/4289031208 shutdown_connections 2023-07-29T16:13:50.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.525+0000 7fde60bff700 1 -- 172.21.15.31:0/4289031208 wait complete. 2023-07-29T16:13:50.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 Processor -- start 2023-07-29T16:13:50.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 -- start start 2023-07-29T16:13:50.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 0x7fde5c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 0x7fde5c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c101eb0 0x7fde5c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c19feb0 con 0x7fde5c100330 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c19fff0 con 0x7fde5c101eb0 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde60bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5c1a0180 con 0x7fde5c101520 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 0x7fde5c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.526+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 0x7fde5c198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49720/0 (socket says 172.21.15.31:49720) 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 -- 172.21.15.31:0/3202569740 learned_addr learned my addr 172.21.15.31:0/3202569740 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde53fff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 0x7fde5c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5ad9d700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c101eb0 0x7fde5c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 -- 172.21.15.31:0/3202569740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 msgr2=0x7fde5c198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 0x7fde5c198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 -- 172.21.15.31:0/3202569740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c101eb0 msgr2=0x7fde5c19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c101eb0 0x7fde5c19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 -- 172.21.15.31:0/3202569740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde4c0095f0 con 0x7fde5c100330 2023-07-29T16:13:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde53fff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 0x7fde5c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde5a59c700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 0x7fde5c198330 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fde4c005740 tx=0x7fde4c005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde537fe700 1 -- 172.21.15.31:0/3202569740 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c01d070 con 0x7fde5c100330 2023-07-29T16:13:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c1a0400 con 0x7fde5c100330 2023-07-29T16:13:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.527+0000 7fde537fe700 1 -- 172.21.15.31:0/3202569740 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde4c021410 con 0x7fde5c100330 2023-07-29T16:13:50.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.528+0000 7fde537fe700 1 -- 172.21.15.31:0/3202569740 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c017650 con 0x7fde5c100330 2023-07-29T16:13:50.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.528+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c1a0880 con 0x7fde5c100330 2023-07-29T16:13:50.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.531+0000 7fde537fe700 1 -- 172.21.15.31:0/3202569740 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fde4c021580 con 0x7fde5c100330 2023-07-29T16:13:50.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 Processor -- start 2023-07-29T16:13:50.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 -- start start 2023-07-29T16:13:50.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde537fe700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde480a0810 0x7fde480a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8dacfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8dadd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8dae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8d049c0 con 0x5563e8dacb90 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8d02380 con 0x5563e8dad4e0 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.532+0000 7fde7454fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8cfdc70 con 0x5563e8dae2a0 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8dacfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5ad9d700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde480a0810 0x7fde480a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde537fe700 1 -- 172.21.15.31:0/3202569740 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fde4c0ccb70 con 0x7fde5c100330 2023-07-29T16:13:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8dacfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49732/0 (socket says 172.21.15.31:49732) 2023-07-29T16:13:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 -- 172.21.15.31:0/2606483597 learned_addr learned my addr 172.21.15.31:0/2606483597 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde53fff700 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8dadd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5ad9d700 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8dae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 -- 172.21.15.31:0/2606483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 msgr2=0x5563e8dae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8dae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 -- 172.21.15.31:0/2606483597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 msgr2=0x5563e8dadd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8dadd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 -- 172.21.15.31:0/2606483597 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563e8b2bd90 con 0x5563e8dacb90 2023-07-29T16:13:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5a59c700 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8dacfa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0ca5b0 tx=0x7fde4c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d06724d10827fd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.533+0000 7fde5ad9d700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde480a0810 0x7fde480a2cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fde54009ec0 tx=0x7fde54009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde50ff9700 1 -- 172.21.15.31:0/2606483597 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c09a070 con 0x5563e8dacb90 2023-07-29T16:13:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde50ff9700 1 -- 172.21.15.31:0/2606483597 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde4c099de0 con 0x5563e8dacb90 2023-07-29T16:13:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde50ff9700 1 -- 172.21.15.31:0/2606483597 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c098640 con 0x5563e8dacb90 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 -- 172.21.15.31:0/2606483597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 msgr2=0x5563e8dacfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8dacfa0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0ca5b0 tx=0x7fde4c02e000 comp rx=0 tx=0).stop 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 -- 172.21.15.31:0/2606483597 shutdown_connections 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8dae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8dacfa0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 --2- 172.21.15.31:0/2606483597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8dadd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.534+0000 7fde7454fb80 1 -- 172.21.15.31:0/2606483597 >> 172.21.15.31:0/2606483597 conn(0x5563e8da2390 msgr2=0x5563e8da7940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.535+0000 7fde7454fb80 1 -- 172.21.15.31:0/2606483597 shutdown_connections 2023-07-29T16:13:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.535+0000 7fde7454fb80 1 -- 172.21.15.31:0/2606483597 wait complete. 2023-07-29T16:13:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.535+0000 7fde7454fb80 1 -- start start 2023-07-29T16:13:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.535+0000 7fde7454fb80 1 Processor -- start 2023-07-29T16:13:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 0x5563e8db6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8db69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dae2a0 0x5563e8dbda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8c9a4c0 con 0x5563e8dae2a0 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8c9a160 con 0x5563e8dad4e0 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8b818d0 con 0x5563e8dacb90 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde7454fb80 10 client.0 _ll_register_callbacks cb 0x5563e8ca35d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 0x5563e8db6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dae2a0 0x5563e8dbda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 0x5563e8db6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57892/0 (socket says 172.21.15.31:57892) 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 -- 172.21.15.31:0/3952310898 learned_addr learned my addr 172.21.15.31:0/3952310898 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde53fff700 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8db69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 -- 172.21.15.31:0/3952310898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 msgr2=0x5563e8db69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8db69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 -- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dae2a0 msgr2=0x5563e8dbda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dae2a0 0x5563e8dbda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.536+0000 7fde5a59c700 1 -- 172.21.15.31:0/3952310898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde5c1a0880 con 0x5563e8dacb90 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde5a59c700 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 0x5563e8db6480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0ca120 tx=0x7fde4c0a26e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9511da26b9cd68ba server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde41ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde41ffb700 1 -- 172.21.15.31:0/3952310898 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c09e2d0 con 0x5563e8dacb90 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde41ffb700 1 -- 172.21.15.31:0/3952310898 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fde4c09a2a0 con 0x5563e8dacb90 2023-07-29T16:13:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563e8d28cc0 con 0x5563e8dacb90 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde41ffb700 1 -- 172.21.15.31:0/3952310898 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde4c098640 con 0x5563e8dacb90 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde7454fb80 10 client.25153 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5563e8b14c10 con 0x5563e8dacb90 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.537+0000 7fde41ffb700 1 -- 172.21.15.31:0/3952310898 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5563e8d28cc0 con 0x5563e8dacb90 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde41ffb700 1 -- 172.21.15.31:0/3952310898 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fde4c09a590 con 0x5563e8dacb90 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde7454fb80 10 client.25153 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde7454fb80 10 client.25153 resolve_mds: resolved name 'g' to daemon mds.g 2023-07-29T16:13:50.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde7454fb80 20 client.25153 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:50.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde7454fb80 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8dc1850 0x5563e8dc0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde7454fb80 4 client.25153 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:13:50.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563e8d7bd20 con 0x5563e8dc1850 2023-07-29T16:13:50.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.538+0000 7fde53fff700 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8dc1850 0x5563e8dc0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde53fff700 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8dc1850 0x5563e8dc0930 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d9eb56226b2ea8b server_cookie=341fce64d7d010f6 in_seq=0 out_seq=0 2023-07-29T16:13:50.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde41ffb700 10 client.25153 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde41ffb700 1 -- 172.21.15.31:0/3952310898 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5563e8d7bd20 con 0x5563e8dc1850 2023-07-29T16:13:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde41ffb700 10 client.25153 handle_command_reply: tid=0 2023-07-29T16:13:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde7454fb80 1 client.25153 shutdown 2023-07-29T16:13:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 msgr2=0x5563e8db6480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.539+0000 7fde7454fb80 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 0x5563e8db6480 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0ca120 tx=0x7fde4c0a26e0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 shutdown_connections 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8dc1850 0x5563e8dc0930 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dae2a0 0x5563e8dbda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dacb90 0x5563e8db6480 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 --2- 172.21.15.31:0/3952310898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dad4e0 0x5563e8db69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde41ffb700 0 client.25153 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 >> 172.21.15.31:0/3952310898 conn(0x5563e8da2390 msgr2=0x5563e8da7010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 shutdown_connections 2023-07-29T16:13:50.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 1 -- 172.21.15.31:0/3952310898 wait complete. 2023-07-29T16:13:50.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.540+0000 7fde7454fb80 20 client.25153 trim_cache size 0 max 16384 2023-07-29T16:13:50.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 Processor -- start 2023-07-29T16:13:50.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 -- start start 2023-07-29T16:13:50.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dacb90 0x5563e8dc0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8da7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8db6050 con 0x5563e8dad4e0 2023-07-29T16:13:50.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8d7bd20 con 0x5563e8dacb90 2023-07-29T16:13:50.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.558+0000 7fde7454fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8d28cc0 con 0x5563e8dae2a0 2023-07-29T16:13:50.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8da7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49762/0 (socket says 172.21.15.31:49762) 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dacb90 0x5563e8dc0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 -- 172.21.15.31:0/1975162249 learned_addr learned my addr 172.21.15.31:0/1975162249 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 -- 172.21.15.31:0/1975162249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 msgr2=0x5563e8da7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8da7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 -- 172.21.15.31:0/1975162249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dacb90 msgr2=0x5563e8dc0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dacb90 0x5563e8dc0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 -- 172.21.15.31:0/1975162249 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563e8b52200 con 0x5563e8dad4e0 2023-07-29T16:13:50.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde5a59c700 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dacb90 0x5563e8dc0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.559+0000 7fde53fff700 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x5563e8db7010 tx=0x7fde44009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2f21a523da8aa0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde517fa700 1 -- 172.21.15.31:0/1975162249 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44017070 con 0x5563e8dad4e0 2023-07-29T16:13:50.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde517fa700 1 -- 172.21.15.31:0/1975162249 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde44018040 con 0x5563e8dad4e0 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde517fa700 1 -- 172.21.15.31:0/1975162249 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde44004ca0 con 0x5563e8dad4e0 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 -- 172.21.15.31:0/1975162249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 msgr2=0x5563e8dc9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x5563e8db7010 tx=0x7fde44009260 comp rx=0 tx=0).stop 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 -- 172.21.15.31:0/1975162249 shutdown_connections 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dae2a0 0x5563e8da7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dacb90 0x5563e8dc0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 --2- 172.21.15.31:0/1975162249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.560+0000 7fde7454fb80 1 -- 172.21.15.31:0/1975162249 >> 172.21.15.31:0/1975162249 conn(0x5563e8da2390 msgr2=0x5563e8db51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.561+0000 7fde7454fb80 1 -- 172.21.15.31:0/1975162249 shutdown_connections 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.561+0000 7fde7454fb80 1 -- 172.21.15.31:0/1975162249 wait complete. 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.561+0000 7fde7454fb80 1 -- start start 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.561+0000 7fde7454fb80 1 Processor -- start 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8db3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dae2a0 0x5563e8e12e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8e13540 con 0x5563e8dacb90 2023-07-29T16:13:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8e136b0 con 0x5563e8dae2a0 2023-07-29T16:13:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563e8e13870 con 0x5563e8dad4e0 2023-07-29T16:13:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde7454fb80 10 client.0 _ll_register_callbacks cb 0x5563e8ca35d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:50.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8db3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde5a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8db3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49764/0 (socket says 172.21.15.31:49764) 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde5a59c700 1 -- 172.21.15.31:0/593340034 learned_addr learned my addr 172.21.15.31:0/593340034 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.562+0000 7fde53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5ad9d700 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dae2a0 0x5563e8e12e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5a59c700 1 -- 172.21.15.31:0/593340034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dad4e0 msgr2=0x5563e8dc9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5a59c700 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5a59c700 1 -- 172.21.15.31:0/593340034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dae2a0 msgr2=0x5563e8e12e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5a59c700 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dae2a0 0x5563e8e12e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5a59c700 1 -- 172.21.15.31:0/593340034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563e8b14c10 con 0x5563e8dacb90 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde5a59c700 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8db3b40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00b620 tx=0x7fde4c0a27d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d69f9773b4cb00e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde41ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde41ffb700 1 -- 172.21.15.31:0/593340034 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4c099a20 con 0x5563e8dacb90 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde41ffb700 1 -- 172.21.15.31:0/593340034 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fde4c09a2a0 con 0x5563e8dacb90 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde4c099de0 con 0x5563e8dacb90 2023-07-29T16:13:50.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde41ffb700 1 -- 172.21.15.31:0/593340034 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde4c09a070 con 0x5563e8dacb90 2023-07-29T16:13:50.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde7454fb80 10 client.15171 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:50.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.563+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fde44012170 con 0x5563e8dacb90 2023-07-29T16:13:50.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde41ffb700 1 -- 172.21.15.31:0/593340034 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fde4c099de0 con 0x5563e8dacb90 2023-07-29T16:13:50.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde41ffb700 1 -- 172.21.15.31:0/593340034 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fde4c096090 con 0x5563e8dacb90 2023-07-29T16:13:50.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde7454fb80 10 client.15171 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:50.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde7454fb80 10 client.15171 resolve_mds: resolved name 'g' to daemon mds.g 2023-07-29T16:13:50.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde7454fb80 20 client.15171 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:50.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde7454fb80 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8e18240 0x5563e8e14b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde7454fb80 4 client.15171 mds_command: new command op to 24551 tid=0[{"prefix": "status"}] 2023-07-29T16:13:50.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5563e8cf6cf0 con 0x5563e8e18240 2023-07-29T16:13:50.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.564+0000 7fde53fff700 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8e18240 0x5563e8e14b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde53fff700 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8e18240 0x5563e8e14b90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f62e4f5adc024ff6 server_cookie=e9fcde5288e03d7f in_seq=0 out_seq=0 2023-07-29T16:13:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde41ffb700 10 client.15171 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde41ffb700 1 -- 172.21.15.31:0/593340034 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5563e8cf6cf0 con 0x5563e8e18240 2023-07-29T16:13:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde41ffb700 10 client.15171 handle_command_reply: tid=0 2023-07-29T16:13:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 client.15171 shutdown 2023-07-29T16:13:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 msgr2=0x5563e8db3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8db3b40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fde4c00b620 tx=0x7fde4c0a27d0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 shutdown_connections 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563e8e18240 0x5563e8e14b90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563e8dae2a0 0x5563e8e12e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563e8dacb90 0x5563e8db3b40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde41ffb700 0 client.15171 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:13:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 --2- 172.21.15.31:0/593340034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563e8dad4e0 0x5563e8dc9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 >> 172.21.15.31:0/593340034 conn(0x5563e8da2390 msgr2=0x5563e8dc0930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 shutdown_connections 2023-07-29T16:13:50.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.565+0000 7fde7454fb80 1 -- 172.21.15.31:0/593340034 wait complete. 2023-07-29T16:13:50.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde7454fb80 20 client.15171 trim_cache size 0 max 16384 2023-07-29T16:13:50.574 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24551, 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-07-29T16:13:50.575 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 85.100212080999995, 2023-07-29T16:13:50.576 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2023-07-29T16:13:50.576 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 89, 2023-07-29T16:13:50.576 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 86, 2023-07-29T16:13:50.576 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 98.17693251 2023-07-29T16:13:50.576 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:13:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde480a0810 msgr2=0x7fde480a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde480a0810 0x7fde480a2cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fde54009ec0 tx=0x7fde54009920 comp rx=0 tx=0).stop 2023-07-29T16:13:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 msgr2=0x7fde5c198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 0x7fde5c198330 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fde4c005740 tx=0x7fde4c005770 comp rx=0 tx=0).stop 2023-07-29T16:13:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 shutdown_connections 2023-07-29T16:13:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fde480a0810 0x7fde480a2cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde5c101eb0 0x7fde5c19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fde5c100330 0x7fde5c198330 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 --2- 172.21.15.31:0/3202569740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde5c101520 0x7fde5c198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 >> 172.21.15.31:0/3202569740 conn(0x7fde5c0fb970 msgr2=0x7fde5c106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 shutdown_connections 2023-07-29T16:13:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.566+0000 7fde60bff700 1 -- 172.21.15.31:0/3202569740 wait complete. 2023-07-29T16:13:50.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-29T16:13:50.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 -- 172.21.15.31:0/214995395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444100320 msgr2=0x7ff444100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 --2- 172.21.15.31:0/214995395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444100320 0x7ff444100730 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff440009990 tx=0x7ff440009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 -- 172.21.15.31:0/214995395 shutdown_connections 2023-07-29T16:13:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 --2- 172.21.15.31:0/214995395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101ea0 0x7ff444108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 --2- 172.21.15.31:0/214995395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444100320 0x7ff444100730 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 --2- 172.21.15.31:0/214995395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff444101510 0x7ff444101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 -- 172.21.15.31:0/214995395 >> 172.21.15.31:0/214995395 conn(0x7ff4440fb920 msgr2=0x7ff4440fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.790+0000 7ff44bfff700 1 -- 172.21.15.31:0/214995395 shutdown_connections 2023-07-29T16:13:50.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.791+0000 7ff44bfff700 1 -- 172.21.15.31:0/214995395 wait complete. 2023-07-29T16:13:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.791+0000 7ff44bfff700 1 Processor -- start 2023-07-29T16:13:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 -- start start 2023-07-29T16:13:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff444100320 0x7ff444196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 0x7ff4441965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444101ea0 0x7ff44419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419dbf0 con 0x7ff444100320 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419dd30 con 0x7ff444101510 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44bfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419dec0 con 0x7ff444101ea0 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 0x7ff4441965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff449d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff444100320 0x7ff444196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444101ea0 0x7ff44419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 0x7ff4441965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41036/0 (socket says 172.21.15.31:41036) 2023-07-29T16:13:50.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44959a700 1 -- 172.21.15.31:0/1903860762 learned_addr learned my addr 172.21.15.31:0/1903860762 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.792+0000 7ff44959a700 1 -- 172.21.15.31:0/1903860762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444101ea0 msgr2=0x7ff44419d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44959a700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444101ea0 0x7ff44419d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44959a700 1 -- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff444100320 msgr2=0x7ff444196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44959a700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff444100320 0x7ff444196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44959a700 1 -- 172.21.15.31:0/1903860762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400095f0 con 0x7ff444101510 2023-07-29T16:13:50.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44959a700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 0x7ff4441965b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff43400e9a0 tx=0x7ff43400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff43affd700 1 -- 172.21.15.31:0/1903860762 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434009790 con 0x7ff444101510 2023-07-29T16:13:50.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff444072fe0 con 0x7ff444101510 2023-07-29T16:13:50.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff43affd700 1 -- 172.21.15.31:0/1903860762 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff43400ce50 con 0x7ff444101510 2023-07-29T16:13:50.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff43affd700 1 -- 172.21.15.31:0/1903860762 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434018680 con 0x7ff444101510 2023-07-29T16:13:50.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.793+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4440734f0 con 0x7ff444101510 2023-07-29T16:13:50.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.798+0000 7ff43affd700 1 -- 172.21.15.31:0/1903860762 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff434010430 con 0x7ff444101510 2023-07-29T16:13:50.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 Processor -- start 2023-07-29T16:13:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 -- start start 2023-07-29T16:13:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdec8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec94e0 0x5651fdec9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdeca2a0 0x5651fdeca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fde209c0 con 0x5651fdec94e0 2023-07-29T16:13:50.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fde1e380 con 0x5651fdec8b90 2023-07-29T16:13:50.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.799+0000 7ff45f868b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fde19c70 con 0x5651fdeca2a0 2023-07-29T16:13:50.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff44a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdec8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff449d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec94e0 0x5651fdec9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff44a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdec8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41040/0 (socket says 172.21.15.31:41040) 2023-07-29T16:13:50.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff44a59c700 1 -- 172.21.15.31:0/3098027570 learned_addr learned my addr 172.21.15.31:0/3098027570 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff44959a700 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdeca2a0 0x5651fdeca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff43affd700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4300a0790 0x7ff4300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff449d9b700 1 -- 172.21.15.31:0/3098027570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdeca2a0 msgr2=0x5651fdeca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff449d9b700 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdeca2a0 0x5651fdeca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff449d9b700 1 -- 172.21.15.31:0/3098027570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 msgr2=0x5651fdec8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff449d9b700 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdec8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff43affd700 1 -- 172.21.15.31:0/1903860762 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff434014070 con 0x7ff444101510 2023-07-29T16:13:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff449d9b700 1 -- 172.21.15.31:0/3098027570 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651fdc47d90 con 0x5651fdec94e0 2023-07-29T16:13:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff44a59c700 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdec8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.800+0000 7ff44a59c700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4300a0790 0x7ff4300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff449d9b700 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec94e0 0x5651fdec9d10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b680 tx=0x7ff440005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7176090cb2c1bfbd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff42ffff700 1 -- 172.21.15.31:0/3098027570 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44001d070 con 0x5651fdec94e0 2023-07-29T16:13:50.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff42ffff700 1 -- 172.21.15.31:0/3098027570 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff44001e040 con 0x5651fdec94e0 2023-07-29T16:13:50.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff42ffff700 1 -- 172.21.15.31:0/3098027570 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400048e0 con 0x5651fdec94e0 2023-07-29T16:13:50.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff44a59c700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4300a0790 0x7ff4300a2c40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x5651fdec8930 tx=0x7ff43c004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff45f868b80 1 -- 172.21.15.31:0/3098027570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec94e0 msgr2=0x5651fdec9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.801+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec94e0 0x5651fdec9d10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b680 tx=0x7ff440005ea0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 -- 172.21.15.31:0/3098027570 shutdown_connections 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdeca2a0 0x5651fdeca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdec8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3098027570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec94e0 0x5651fdec9d10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 -- 172.21.15.31:0/3098027570 >> 172.21.15.31:0/3098027570 conn(0x5651fdebe390 msgr2=0x5651fdec3940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 -- 172.21.15.31:0/3098027570 shutdown_connections 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 -- 172.21.15.31:0/3098027570 wait complete. 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 -- start start 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.802+0000 7ff45f868b80 1 Processor -- start 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fded2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fded2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fded9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fddb64c0 con 0x5651fdeca2a0 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fddb6160 con 0x5651fdec8b90 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fdc9d8d0 con 0x5651fdec94e0 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff45f868b80 10 client.0 _ll_register_callbacks cb 0x5651fddbf5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff44a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fded9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff449d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fded2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff44a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fded9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49800/0 (socket says 172.21.15.31:49800) 2023-07-29T16:13:50.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff44a59c700 1 -- 172.21.15.31:0/631977265 learned_addr learned my addr 172.21.15.31:0/631977265 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.803+0000 7ff44959a700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fded2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff449d9b700 1 -- 172.21.15.31:0/631977265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 msgr2=0x5651fded2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff449d9b700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fded2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff449d9b700 1 -- 172.21.15.31:0/631977265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 msgr2=0x5651fded9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff449d9b700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fded9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff449d9b700 1 -- 172.21.15.31:0/631977265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c010620 con 0x5651fdec8b90 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff44a59c700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fded9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff449d9b700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fded2750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b2e0 tx=0x7ff44000f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1fbf2315ffdc05 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff42d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff42d7fa700 1 -- 172.21.15.31:0/631977265 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44001d040 con 0x5651fdec8b90 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5651fde44cc0 con 0x5651fdec8b90 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.804+0000 7ff42d7fa700 1 -- 172.21.15.31:0/631977265 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff44000fd10 con 0x5651fdec8b90 2023-07-29T16:13:50.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff42d7fa700 1 -- 172.21.15.31:0/631977265 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff440005050 con 0x5651fdec8b90 2023-07-29T16:13:50.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 10 client.25163 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:50.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7ff444109090 con 0x5651fdec8b90 2023-07-29T16:13:50.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff42d7fa700 1 -- 172.21.15.31:0/631977265 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5651fde44cc0 con 0x5651fdec8b90 2023-07-29T16:13:50.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff42d7fa700 1 -- 172.21.15.31:0/631977265 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff440004bf0 con 0x5651fdec8b90 2023-07-29T16:13:50.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 10 client.25163 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:50.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 10 client.25163 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-29T16:13:50.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 20 client.25163 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:50.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdedd4c0 0x5651fdedcc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 4 client.25163 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:13:50.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.805+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651fde97d20 con 0x5651fdedd4c0 2023-07-29T16:13:50.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.806+0000 7ff44959a700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdedd4c0 0x5651fdedcc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.806+0000 7ff44959a700 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdedd4c0 0x5651fdedcc00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e1658a861d49d31 server_cookie=59201be831916cba in_seq=0 out_seq=0 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.806+0000 7ff42d7fa700 10 client.25163 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff42d7fa700 1 -- 172.21.15.31:0/631977265 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5651fde97d20 con 0x5651fdedd4c0 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff42d7fa700 10 client.25163 handle_command_reply: tid=0 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 client.25163 shutdown 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 msgr2=0x5651fded2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fded2750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b2e0 tx=0x7ff44000f710 comp rx=0 tx=0).stop 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 shutdown_connections 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdedd4c0 0x5651fdedcc00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fded9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fded2750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 --2- 172.21.15.31:0/631977265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fded2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff42d7fa700 0 client.25163 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 >> 172.21.15.31:0/631977265 conn(0x5651fdebe390 msgr2=0x5651fdec3010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 shutdown_connections 2023-07-29T16:13:50.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 1 -- 172.21.15.31:0/631977265 wait complete. 2023-07-29T16:13:50.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.807+0000 7ff45f868b80 20 client.25163 trim_cache size 0 max 16384 2023-07-29T16:13:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 Processor -- start 2023-07-29T16:13:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 -- start start 2023-07-29T16:13:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 0x5651fdec38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdeca2a0 0x5651fdec3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fde44cc0 con 0x5651fdec8b90 2023-07-29T16:13:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fddc3830 con 0x5651fdeca2a0 2023-07-29T16:13:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.825+0000 7ff45f868b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fde1ecf0 con 0x5651fdec94e0 2023-07-29T16:13:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 0x5651fdec38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 0x5651fdec38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49814/0 (socket says 172.21.15.31:49814) 2023-07-29T16:13:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 -- 172.21.15.31:0/2544028477 learned_addr learned my addr 172.21.15.31:0/2544028477 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff44959a700 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff44a59c700 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdeca2a0 0x5651fdec3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 -- 172.21.15.31:0/2544028477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 msgr2=0x5651fdee6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 -- 172.21.15.31:0/2544028477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdeca2a0 msgr2=0x5651fdec3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdeca2a0 0x5651fdec3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 -- 172.21.15.31:0/2544028477 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651fdc30c10 con 0x5651fdec8b90 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.826+0000 7ff449d9b700 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 0x5651fdec38b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff440000c00 tx=0x7ff440009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1283827a25c879e8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff438ff9700 1 -- 172.21.15.31:0/2544028477 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000fd10 con 0x5651fdec8b90 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff438ff9700 1 -- 172.21.15.31:0/2544028477 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff44000f780 con 0x5651fdec8b90 2023-07-29T16:13:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff438ff9700 1 -- 172.21.15.31:0/2544028477 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440036ae0 con 0x5651fdec8b90 2023-07-29T16:13:50.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 -- 172.21.15.31:0/2544028477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 msgr2=0x5651fdec38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 0x5651fdec38b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff440000c00 tx=0x7ff440009290 comp rx=0 tx=0).stop 2023-07-29T16:13:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 -- 172.21.15.31:0/2544028477 shutdown_connections 2023-07-29T16:13:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdeca2a0 0x5651fdec3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdec8b90 0x5651fdec38b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 --2- 172.21.15.31:0/2544028477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 -- 172.21.15.31:0/2544028477 >> 172.21.15.31:0/2544028477 conn(0x5651fdebe390 msgr2=0x5651fded1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 -- 172.21.15.31:0/2544028477 shutdown_connections 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.827+0000 7ff45f868b80 1 -- 172.21.15.31:0/2544028477 wait complete. 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 -- start start 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 Processor -- start 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdee14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fdee4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fdee4de0 con 0x5651fdeca2a0 2023-07-29T16:13:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fdee4f50 con 0x5651fdec8b90 2023-07-29T16:13:50.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.828+0000 7ff45f868b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651fdee5110 con 0x5651fdec94e0 2023-07-29T16:13:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff45f868b80 10 client.0 _ll_register_callbacks cb 0x5651fddbf5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:13:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fdee4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57966/0 (socket says 172.21.15.31:57966) 2023-07-29T16:13:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 -- 172.21.15.31:0/3228296596 learned_addr learned my addr 172.21.15.31:0/3228296596 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff449d9b700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdee14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 -- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 msgr2=0x5651fdee14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdee14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 -- 172.21.15.31:0/3228296596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 msgr2=0x5651fdee4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:13:50.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fdee4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 -- 172.21.15.31:0/3228296596 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff444109090 con 0x5651fdec94e0 2023-07-29T16:13:50.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44a59c700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fdee4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff44959a700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee19e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff43401f040 tx=0x7ff4340cb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e31cbc4b5ac1d2a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.829+0000 7ff42d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:13:50.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff42d7fa700 1 -- 172.21.15.31:0/3228296596 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434099440 con 0x5651fdec94e0 2023-07-29T16:13:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff42d7fa700 1 -- 172.21.15.31:0/3228296596 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff43409d470 con 0x5651fdec94e0 2023-07-29T16:13:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5651fde12cf0 con 0x5651fdec94e0 2023-07-29T16:13:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff42d7fa700 1 -- 172.21.15.31:0/3228296596 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4340a1890 con 0x5651fdec94e0 2023-07-29T16:13:50.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 10 client.25177 fetch_fsmap learned FSMap version 23 2023-07-29T16:13:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5651fdc6e200 con 0x5651fdec94e0 2023-07-29T16:13:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff42d7fa700 1 -- 172.21.15.31:0/3228296596 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5651fde12cf0 con 0x5651fdec94e0 2023-07-29T16:13:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff42d7fa700 1 -- 172.21.15.31:0/3228296596 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff434098930 con 0x5651fdec94e0 2023-07-29T16:13:50.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 10 client.25177 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:13:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 10 client.25177 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-29T16:13:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 20 client.25177 populate_metadata read hostname 'smithi031' 2023-07-29T16:13:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdf34770 0x5651fdf310c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 4 client.25177 mds_command: new command op to 24581 tid=0[{"prefix": "status"}] 2023-07-29T16:13:50.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.830+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5651fde12a10 con 0x5651fdf34770 2023-07-29T16:13:50.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.831+0000 7ff449d9b700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdf34770 0x5651fdf310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:50.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.831+0000 7ff449d9b700 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdf34770 0x5651fdf310c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38e0254323dd0da7 server_cookie=c7a0fb749538aa6f in_seq=0 out_seq=0 2023-07-29T16:13:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.831+0000 7ff42d7fa700 10 client.25177 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff42d7fa700 1 -- 172.21.15.31:0/3228296596 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5651fde12a10 con 0x5651fdf34770 2023-07-29T16:13:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff42d7fa700 10 client.25177 handle_command_reply: tid=0 2023-07-29T16:13:50.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 client.25177 shutdown 2023-07-29T16:13:50.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 msgr2=0x5651fdee19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee19e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff43401f040 tx=0x7ff4340cb040 comp rx=0 tx=0).stop 2023-07-29T16:13:50.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 shutdown_connections 2023-07-29T16:13:50.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651fdf34770 0x5651fdf310c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651fdeca2a0 0x5651fdee4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651fdec8b90 0x5651fdee14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff42d7fa700 0 client.25177 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:13:50.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 --2- 172.21.15.31:0/3228296596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651fdec94e0 0x5651fdee19e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 >> 172.21.15.31:0/3228296596 conn(0x5651fdebe390 msgr2=0x5651fdecd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 shutdown_connections 2023-07-29T16:13:50.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 1 -- 172.21.15.31:0/3228296596 wait complete. 2023-07-29T16:13:50.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.832+0000 7ff45f868b80 20 client.25177 trim_cache size 0 max 16384 2023-07-29T16:13:50.844 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:13:50.844 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "858a7894-2e29-11ee-9b36-001a4aab830c", 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24581, 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 88.912421413999994, 2023-07-29T16:13:50.845 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 23, 2023-07-29T16:13:50.846 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 90, 2023-07-29T16:13:50.846 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 86, 2023-07-29T16:13:50.846 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 93.276970817000006 2023-07-29T16:13:50.846 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:13:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4300a0790 msgr2=0x7ff4300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4300a0790 0x7ff4300a2c40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x5651fdec8930 tx=0x7ff43c004ad0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 msgr2=0x7ff4441965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 0x7ff4441965b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff43400e9a0 tx=0x7ff43400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 shutdown_connections 2023-07-29T16:13:50.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff4300a0790 0x7ff4300a2c40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff444101ea0 0x7ff44419d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff444100320 0x7ff444196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 --2- 172.21.15.31:0/1903860762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff444101510 0x7ff4441965b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:50.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 >> 172.21.15.31:0/1903860762 conn(0x7ff4440fb920 msgr2=0x7ff444104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:50.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 shutdown_connections 2023-07-29T16:13:50.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:50.833+0000 7ff44bfff700 1 -- 172.21.15.31:0/1903860762 wait complete. 2023-07-29T16:13:50.851 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2023-07-29T16:13:50.851 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1, 5, 11]}]': finished 2023-07-29T16:13:50.851 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:50 smithi031 ceph-mon[95504]: osdmap e89: 12 total, 12 up, 12 in 2023-07-29T16:13:50.851 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:50 smithi031 ceph-mon[95504]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-07-29T16:13:50.852 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:50 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/781333760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:50.852 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:50 smithi031 ceph-mon[95504]: osdmap e90: 12 total, 12 up, 12 in 2023-07-29T16:13:50.853 INFO:teuthology.run_tasks:Running task kclient... 2023-07-29T16:13:50.877 INFO:tasks.kclient:Mounting kernel clients... 2023-07-29T16:13:50.878 INFO:tasks.kclient:config is {'client.0': {}} 2023-07-29T16:13:50.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:50.879 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-07-29T16:13:50.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:50.923 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2023-07-29T16:13:51.036 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2023-07-29T16:13:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:13:51.040 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2023-07-29T16:13:51.040 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-29T16:13:51.040 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-29T16:13:51.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph fs ls 2023-07-29T16:13:51.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2023-07-29T16:13:51.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1, 5, 11]}]': finished 2023-07-29T16:13:51.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:50 smithi156 ceph-mon[129245]: osdmap e89: 12 total, 12 up, 12 in 2023-07-29T16:13:51.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:50 smithi156 ceph-mon[129245]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-07-29T16:13:51.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:50 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/781333760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:51.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:50 smithi156 ceph-mon[129245]: osdmap e90: 12 total, 12 up, 12 in 2023-07-29T16:13:51.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 4]}]': finished 2023-07-29T16:13:51.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1, 5, 11]}]': finished 2023-07-29T16:13:51.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:50 smithi039 ceph-mon[128858]: osdmap e89: 12 total, 12 up, 12 in 2023-07-29T16:13:51.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:50 smithi039 ceph-mon[128858]: mgrmap e18: x(active, since 8m), standbys: z, y 2023-07-29T16:13:51.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:50 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/781333760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:51.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:50 smithi039 ceph-mon[128858]: osdmap e90: 12 total, 12 up, 12 in 2023-07-29T16:13:51.769 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:13:52.052 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:51 smithi031 ceph-mon[95504]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:52.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:51 smithi156 ceph-mon[129245]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:52.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:51 smithi039 ceph-mon[128858]: pgmap v321: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:13:53.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 -- 172.21.15.31:0/4270602086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4274100fe0 msgr2=0x7f4274104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:53.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 --2- 172.21.15.31:0/4270602086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4274100fe0 0x7f4274104fb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f42640099c0 tx=0x7f4264009cd0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 -- 172.21.15.31:0/4270602086 shutdown_connections 2023-07-29T16:13:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 --2- 172.21.15.31:0/4270602086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42741056e0 0x7f427410d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 --2- 172.21.15.31:0/4270602086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4274100640 0x7f4274100a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 --2- 172.21.15.31:0/4270602086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4274100fe0 0x7f4274104fb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 -- 172.21.15.31:0/4270602086 >> 172.21.15.31:0/4270602086 conn(0x7f42740fbea0 msgr2=0x7f42740fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 -- 172.21.15.31:0/4270602086 shutdown_connections 2023-07-29T16:13:53.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.167+0000 7f427b2c9700 1 -- 172.21.15.31:0/4270602086 wait complete. 2023-07-29T16:13:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.168+0000 7f427b2c9700 1 Processor -- start 2023-07-29T16:13:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 -- start start 2023-07-29T16:13:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4274100640 0x7f4274198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4274100fe0 0x7f4274198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 0x7f427419c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:53.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427419c610 con 0x7f42741056e0 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427419c780 con 0x7f4274100640 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f427b2c9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427419c940 con 0x7f4274100fe0 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f4279866700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 0x7f427419c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f4279866700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 0x7f427419c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49846/0 (socket says 172.21.15.31:49846) 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f4279866700 1 -- 172.21.15.31:0/2344079624 learned_addr learned my addr 172.21.15.31:0/2344079624 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f4279065700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4274100640 0x7f4274198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.169+0000 7f4278864700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4274100fe0 0x7f4274198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:53.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f4279866700 1 -- 172.21.15.31:0/2344079624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4274100fe0 msgr2=0x7f4274198c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f4279866700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4274100fe0 0x7f4274198c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f4279866700 1 -- 172.21.15.31:0/2344079624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4274100640 msgr2=0x7f4274198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f4279866700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4274100640 0x7f4274198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f4279866700 1 -- 172.21.15.31:0/2344079624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640095f0 con 0x7f42741056e0 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f4279866700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 0x7f427419c010 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f426c00ef30 tx=0x7f426c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c018070 con 0x7f42741056e0 2023-07-29T16:13:53.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f426c00f040 con 0x7f42741056e0 2023-07-29T16:13:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c00cd00 con 0x7f42741056e0 2023-07-29T16:13:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427419cbc0 con 0x7f42741056e0 2023-07-29T16:13:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.170+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427419ce70 con 0x7f42741056e0 2023-07-29T16:13:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.171+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427404ea50 con 0x7f42741056e0 2023-07-29T16:13:53.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.173+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f426c003a00 con 0x7f42741056e0 2023-07-29T16:13:53.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.175+0000 7f426a7fc700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42600a0790 0x7f42600a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:53.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.175+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f426c0cb570 con 0x7f42741056e0 2023-07-29T16:13:53.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.175+0000 7f4279065700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42600a0790 0x7f42600a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:53.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.176+0000 7f4279065700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42600a0790 0x7f42600a2c40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4270009c60 tx=0x7f4270009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:53.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.179+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f426c094fe0 con 0x7f42741056e0 2023-07-29T16:13:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.434+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f42741a55d0 con 0x7f42741056e0 2023-07-29T16:13:53.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.434+0000 7f426a7fc700 1 -- 172.21.15.31:0/2344079624 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+73 (secure 0 0 0) 0x7f426c098600 con 0x7f42741056e0 2023-07-29T16:13:53.432 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-29T16:13:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42600a0790 msgr2=0x7f42600a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42600a0790 0x7f42600a2c40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4270009c60 tx=0x7f4270009260 comp rx=0 tx=0).stop 2023-07-29T16:13:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 msgr2=0x7f427419c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:53.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 0x7f427419c010 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f426c00ef30 tx=0x7f426c00c3a0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 shutdown_connections 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42600a0790 0x7f42600a2c40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42741056e0 0x7f427419c010 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4274100640 0x7f4274198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 --2- 172.21.15.31:0/2344079624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4274100fe0 0x7f4274198c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 >> 172.21.15.31:0/2344079624 conn(0x7f42740fbea0 msgr2=0x7f42740fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:53.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 shutdown_connections 2023-07-29T16:13:53.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:53.437+0000 7f427b2c9700 1 -- 172.21.15.31:0/2344079624 wait complete. 2023-07-29T16:13:53.949 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-29T16:13:53.949 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-29T16:13:53.949 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2023-07-29T16:13:53.949 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-29T16:13:53.950 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-29T16:13:53.950 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-29T16:13:53.950 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-29T16:13:53.950 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-29T16:13:53.950 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-29T16:13:53.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:53.951 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2023-07-29T16:13:53.995 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-29T16:13:53.995 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2023-07-29T16:13:53.996 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2023-07-29T16:13:53.997 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-07-29T16:13:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 41424sec preferred_lft 41424sec 2023-07-29T16:13:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2023-07-29T16:13:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2023-07-29T16:13:53.998 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-29T16:13:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2023-07-29T16:13:53.998 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2023-07-29T16:13:53.999 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-29T16:13:53.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:54.000 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-07-29T16:13:54.000 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-07-29T16:13:54.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2023-07-29T16:13:54.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2023-07-29T16:13:54.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2023-07-29T16:13:54.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-29T16:13:54.001 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-07-29T16:13:54.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:54 smithi031 ceph-mon[95504]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:54.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:54 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2344079624' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-29T16:13:54.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:54.478 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-29T16:13:54.550 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-07-29T16:13:54.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:54.554 DEBUG:teuthology.orchestra.run.smithi031:> route 2023-07-29T16:13:54.620 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2023-07-29T16:13:54.620 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-29T16:13:54.620 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-07-29T16:13:54.621 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-07-29T16:13:54.621 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-29T16:13:54.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-07-29T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-07-29T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-07-29T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-07-29T16:13:54.622 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-07-29T16:13:54.623 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-07-29T16:13:54.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:54 smithi039 ceph-mon[128858]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:54 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2344079624' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-29T16:13:54.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:54 smithi156 ceph-mon[129245]: pgmap v322: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:54.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:54 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2344079624' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-29T16:13:55.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:55.350 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-07-29T16:13:55.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:55.395 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2023-07-29T16:13:55.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:55.479 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-07-29T16:13:55.480 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-07-29T16:13:55.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-29T16:13:55.480 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-29T16:13:55.480 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-07-29T16:13:55.666 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-29T16:13:55.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:55.667 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-07-29T16:13:55.667 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-07-29T16:13:55.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-29T16:13:55.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-29T16:13:55.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-29T16:13:55.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-29T16:13:55.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-29T16:13:55.668 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-07-29T16:13:55.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:55.946 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec bash -c ' 2023-07-29T16:13:55.946 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2023-07-29T16:13:55.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2023-07-29T16:13:55.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2023-07-29T16:13:55.947 DEBUG:teuthology.orchestra.run.smithi031:> ') 2023-07-29T16:13:56.122 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-29T16:13:56.122 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-29T16:13:56.166 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-29T16:13:56.166 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-29T16:13:56.167 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-29T16:13:56.248 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-07-29T16:13:56.249 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-07-29T16:13:56.249 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902 2023-07-29T16:13:56.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:56.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2023-07-29T16:13:56.408 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2023-07-29T16:13:56.409 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2023-07-29T16:13:56.409 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2023-07-29T16:13:56.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:56 smithi031 ceph-mon[95504]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 7 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:56.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:56 smithi039 ceph-mon[128858]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 7 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:56.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:56 smithi156 ceph-mon[129245]: pgmap v323: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 895 B/s wr, 7 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:56.857 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2023-07-29T16:13:56.857 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2023-07-29T16:13:56.857 INFO:teuthology.orchestra.run.smithi031.stdout:invalid new device string format 2023-07-29T16:13:56.857 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.39:3300,172.21.15.156:3300" 2023-07-29T16:13:56.858 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.31:3300,172.21.15.39:3300,172.21.15.156:3300:/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902 2023-07-29T16:13:56.858 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=858a7894-2e29-11ee-9b36-001a4aab830c" will pass to kernel 2023-07-29T16:13:56.864 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-29T16:13:56.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:56.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-29T16:13:56.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:56.942 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-29T16:13:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:client25186 2023-07-29T16:13:57.035 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:13:57.036 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-29T16:13:57.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:57.080 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-29T16:13:57.206 INFO:teuthology.orchestra.run.smithi031.stdout:client25186 2023-07-29T16:13:57.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:57.207 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-29T16:13:57.294 INFO:teuthology.orchestra.run.smithi031.stdout:client25186 2023-07-29T16:13:57.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:13:57.294 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/858a7894-2e29-11ee-9b36-001a4aab830c.client25186/status') 2023-07-29T16:13:57.406 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.25186 (3)172.21.15.31:0/2005059736 2023-07-29T16:13:57.407 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2023-07-29T16:13:57.407 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2023-07-29T16:13:57.407 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2023-07-29T16:13:57.407 INFO:teuthology.orchestra.run.smithi031.stderr:71 bytes copied, 5.1653e-05 s, 1.4 MB/s 2023-07-29T16:13:57.409 INFO:teuthology.run_tasks:Running task check-counter... 2023-07-29T16:13:57.421 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-07-29T16:13:57.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:57.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.652+0000 7fe4c99ff700 1 -- 172.21.15.31:0/2041580294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 msgr2=0x7fe4c4100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:57.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.652+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/2041580294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4100750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4009990 tx=0x7fe4b4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 -- 172.21.15.31:0/2041580294 shutdown_connections 2023-07-29T16:13:57.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/2041580294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4c4101ec0 0x7fe4c4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/2041580294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4100750 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/2041580294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 0x7fe4c4101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 -- 172.21.15.31:0/2041580294 >> 172.21.15.31:0/2041580294 conn(0x7fe4c40fb920 msgr2=0x7fe4c40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:57.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 -- 172.21.15.31:0/2041580294 shutdown_connections 2023-07-29T16:13:57.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.653+0000 7fe4c99ff700 1 -- 172.21.15.31:0/2041580294 wait complete. 2023-07-29T16:13:57.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 Processor -- start 2023-07-29T16:13:57.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 -- start start 2023-07-29T16:13:57.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:57.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 0x7fe4c41965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:57.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4c4101ec0 0x7fe4c419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:57.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c419dbe0 con 0x7fe4c4100340 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c419dd20 con 0x7fe4c4101530 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c99ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c419deb0 con 0x7fe4c4101ec0 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49852/0 (socket says 172.21.15.31:49852) 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c2ffd700 1 -- 172.21.15.31:0/988129333 learned_addr learned my addr 172.21.15.31:0/988129333 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.654+0000 7fe4c27fc700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 0x7fe4c41965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:57.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c37fe700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4c4101ec0 0x7fe4c419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c2ffd700 1 -- 172.21.15.31:0/988129333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4c4101ec0 msgr2=0x7fe4c419d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c2ffd700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4c4101ec0 0x7fe4c419d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c2ffd700 1 -- 172.21.15.31:0/988129333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 msgr2=0x7fe4c41965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c2ffd700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 0x7fe4c41965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c2ffd700 1 -- 172.21.15.31:0/988129333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40095f0 con 0x7fe4c4100340 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c27fc700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 0x7fe4c41965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c2ffd700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4196060 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4005740 tx=0x7fe4b4005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:57.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b401d070 con 0x7fe4c4100340 2023-07-29T16:13:57.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c419e130 con 0x7fe4c4100340 2023-07-29T16:13:57.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe4b4021410 con 0x7fe4c4100340 2023-07-29T16:13:57.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.655+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4017650 con 0x7fe4c4100340 2023-07-29T16:13:57.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.656+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c419e560 con 0x7fe4c4100340 2023-07-29T16:13:57.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.656+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c4107830 con 0x7fe4c4100340 2023-07-29T16:13:57.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.658+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe4b4021580 con 0x7fe4c4100340 2023-07-29T16:13:57.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.660+0000 7fe4c89fd700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4ac0a07e0 0x7fe4ac0a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:57.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.660+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe4b40ccc10 con 0x7fe4c4100340 2023-07-29T16:13:57.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.660+0000 7fe4c27fc700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4ac0a07e0 0x7fe4ac0a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:57.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.661+0000 7fe4c27fc700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4ac0a07e0 0x7fe4ac0a2c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8011490 tx=0x7fe4b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:57.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.664+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fe4b405b720 con 0x7fe4c4100340 2023-07-29T16:13:57.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.878+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4c402cc30 con 0x7fe4c4100340 2023-07-29T16:13:57.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.880+0000 7fe4c89fd700 1 -- 172.21.15.31:0/988129333 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22731 (secure 0 0 0) 0x7fe4b4096680 con 0x7fe4c4100340 2023-07-29T16:13:57.878 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:57.879 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:13:50.771629+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:57.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4ac0a07e0 msgr2=0x7fe4ac0a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:57.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4ac0a07e0 0x7fe4ac0a2c90 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8011490 tx=0x7fe4b8009260 comp rx=0 tx=0).stop 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 msgr2=0x7fe4c4196060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4196060 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4005740 tx=0x7fe4b4005770 comp rx=0 tx=0).stop 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 shutdown_connections 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4ac0a07e0 0x7fe4ac0a2c90 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4c4101ec0 0x7fe4c419d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4c4100340 0x7fe4c4196060 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 --2- 172.21.15.31:0/988129333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4c4101530 0x7fe4c41965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:57.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 >> 172.21.15.31:0/988129333 conn(0x7fe4c40fb920 msgr2=0x7fe4c4104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:57.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 shutdown_connections 2023-07-29T16:13:57.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:57.882+0000 7fe4c99ff700 1 -- 172.21.15.31:0/988129333 wait complete. 2023-07-29T16:13:57.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:13:57.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:13:58.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:57 smithi039 ceph-mon[128858]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 895 B/s wr, 3 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:57 smithi156 ceph-mon[129245]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 895 B/s wr, 3 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:58.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 -- 172.21.15.31:0/259737495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30101520 msgr2=0x7fbc30101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 --2- 172.21.15.31:0/259737495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30101520 0x7fbc30101970 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009990 tx=0x7fbc24009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 -- 172.21.15.31:0/259737495 shutdown_connections 2023-07-29T16:13:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 --2- 172.21.15.31:0/259737495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 0x7fbc30108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 --2- 172.21.15.31:0/259737495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc30100330 0x7fbc30100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 --2- 172.21.15.31:0/259737495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30101520 0x7fbc30101970 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 -- 172.21.15.31:0/259737495 >> 172.21.15.31:0/259737495 conn(0x7fbc300fb970 msgr2=0x7fbc300fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:58.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 -- 172.21.15.31:0/259737495 shutdown_connections 2023-07-29T16:13:58.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.113+0000 7fbc365ff700 1 -- 172.21.15.31:0/259737495 wait complete. 2023-07-29T16:13:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.114+0000 7fbc365ff700 1 Processor -- start 2023-07-29T16:13:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.114+0000 7fbc365ff700 1 -- start start 2023-07-29T16:13:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.114+0000 7fbc365ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 0x7fbc301982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc365ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc30101520 0x7fbc30198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc365ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 0x7fbc3019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc365ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3019fe40 con 0x7fbc30100330 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc365ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3019ff80 con 0x7fbc30101520 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc365ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc301a0110 con 0x7fbc30101eb0 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 0x7fbc301982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 0x7fbc301982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49870/0 (socket says 172.21.15.31:49870) 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 -- 172.21.15.31:0/1002668048 learned_addr learned my addr 172.21.15.31:0/1002668048 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:58.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2f7fe700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc30101520 0x7fbc30198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc34b9c700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 0x7fbc3019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 -- 172.21.15.31:0/1002668048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 msgr2=0x7fbc3019f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 0x7fbc3019f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 -- 172.21.15.31:0/1002668048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc30101520 msgr2=0x7fbc30198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc30101520 0x7fbc30198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc2ffff700 1 -- 172.21.15.31:0/1002668048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc240095f0 con 0x7fbc30100330 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.115+0000 7fbc34b9c700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 0x7fbc3019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:58.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.116+0000 7fbc2ffff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 0x7fbc301982c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000e970 tx=0x7fbc2000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.116+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20009790 con 0x7fbc30100330 2023-07-29T16:13:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.116+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc2000eec0 con 0x7fbc30100330 2023-07-29T16:13:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.116+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc301a03f0 con 0x7fbc30100330 2023-07-29T16:13:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.116+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc20003cf0 con 0x7fbc30100330 2023-07-29T16:13:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.116+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc301a0900 con 0x7fbc30100330 2023-07-29T16:13:58.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.117+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc30192a20 con 0x7fbc30100330 2023-07-29T16:13:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.119+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbc2000ccd0 con 0x7fbc30100330 2023-07-29T16:13:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.120+0000 7fbc2d7fa700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbc180a0790 0x7fbc180a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.120+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbc20014070 con 0x7fbc30100330 2023-07-29T16:13:58.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.120+0000 7fbc2f7fe700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbc180a0790 0x7fbc180a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.121+0000 7fbc2f7fe700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbc180a0790 0x7fbc180a2c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400b2e0 tx=0x7fbc24005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:58.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.124+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbc20095850 con 0x7fbc30100330 2023-07-29T16:13:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:57 smithi031 ceph-mon[95504]: pgmap v324: 97 pgs: 97 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 895 B/s wr, 3 op/s; 137 B/s, 0 objects/s recovering 2023-07-29T16:13:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.333+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc30061960 con 0x7fbc30100330 2023-07-29T16:13:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.334+0000 7fbc2d7fa700 1 -- 172.21.15.31:0/1002668048 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc20098e70 con 0x7fbc30100330 2023-07-29T16:13:58.332 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:13:58.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbc180a0790 msgr2=0x7fbc180a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbc180a0790 0x7fbc180a2c40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400b2e0 tx=0x7fbc24005fb0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 msgr2=0x7fbc301982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 0x7fbc301982c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000e970 tx=0x7fbc2000ec80 comp rx=0 tx=0).stop 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 shutdown_connections 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbc180a0790 0x7fbc180a2c40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc30101eb0 0x7fbc3019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbc30100330 0x7fbc301982c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 --2- 172.21.15.31:0/1002668048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbc30101520 0x7fbc30198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.336+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 >> 172.21.15.31:0/1002668048 conn(0x7fbc300fb970 msgr2=0x7fbc30106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.337+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 shutdown_connections 2023-07-29T16:13:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.337+0000 7fbc365ff700 1 -- 172.21.15.31:0/1002668048 wait complete. 2023-07-29T16:13:58.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:13:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 -- 172.21.15.31:0/930364881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac100320 msgr2=0x7f83ac100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 --2- 172.21.15.31:0/930364881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac100320 0x7f83ac100730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f83a8009990 tx=0x7f83a8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 -- 172.21.15.31:0/930364881 shutdown_connections 2023-07-29T16:13:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 --2- 172.21.15.31:0/930364881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 --2- 172.21.15.31:0/930364881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac100320 0x7f83ac100730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 --2- 172.21.15.31:0/930364881 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83ac101510 0x7f83ac101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 -- 172.21.15.31:0/930364881 >> 172.21.15.31:0/930364881 conn(0x7f83ac0fb920 msgr2=0x7f83ac0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.562+0000 7f83b33ff700 1 -- 172.21.15.31:0/930364881 shutdown_connections 2023-07-29T16:13:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.563+0000 7f83b33ff700 1 -- 172.21.15.31:0/930364881 wait complete. 2023-07-29T16:13:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.563+0000 7f83b33ff700 1 Processor -- start 2023-07-29T16:13:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 -- start start 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83ac100320 0x7f83ac074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac101510 0x7f83ac073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac073e20 con 0x7f83ac101ea0 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac073f90 con 0x7f83ac101510 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b33ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83ac1a0100 con 0x7f83ac100320 2023-07-29T16:13:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b199c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b199c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49884/0 (socket says 172.21.15.31:49884) 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b199c700 1 -- 172.21.15.31:0/2385629989 learned_addr learned my addr 172.21.15.31:0/2385629989 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b099a700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac101510 0x7f83ac073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.564+0000 7f83b119b700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83ac100320 0x7f83ac074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b199c700 1 -- 172.21.15.31:0/2385629989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83ac100320 msgr2=0x7f83ac074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b199c700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83ac100320 0x7f83ac074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b199c700 1 -- 172.21.15.31:0/2385629989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac101510 msgr2=0x7f83ac073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b199c700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac101510 0x7f83ac073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b199c700 1 -- 172.21.15.31:0/2385629989 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a80095f0 con 0x7f83ac101ea0 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b199c700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac073680 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f83a400ef30 tx=0x7f83a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a4018070 con 0x7f83ac101ea0 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83ac1a0380 con 0x7f83ac101ea0 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83a400f040 con 0x7f83ac101ea0 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a400cd00 con 0x7f83ac101ea0 2023-07-29T16:13:58.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.565+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83ac1a08e0 con 0x7f83ac101ea0 2023-07-29T16:13:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.566+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83ac10bb10 con 0x7f83ac101ea0 2023-07-29T16:13:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.568+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f83a4003a00 con 0x7f83ac101ea0 2023-07-29T16:13:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.570+0000 7f83a27fc700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83980a0790 0x7f83980a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.570+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f83a40cb620 con 0x7f83ac101ea0 2023-07-29T16:13:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.570+0000 7f83b119b700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83980a0790 0x7f83980a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.571+0000 7f83b119b700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83980a0790 0x7f83980a2c40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f83a8006010 tx=0x7f83a8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.573+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f83a4095090 con 0x7f83ac101ea0 2023-07-29T16:13:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.784+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83ac107270 con 0x7f83ac101ea0 2023-07-29T16:13:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.785+0000 7f83a27fc700 1 -- 172.21.15.31:0/2385629989 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f83a40986b0 con 0x7f83ac101ea0 2023-07-29T16:13:58.783 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:13:58.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83980a0790 msgr2=0x7f83980a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83980a0790 0x7f83980a2c40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f83a8006010 tx=0x7f83a8005960 comp rx=0 tx=0).stop 2023-07-29T16:13:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 msgr2=0x7f83ac073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac073680 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f83a400ef30 tx=0x7f83a400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 shutdown_connections 2023-07-29T16:13:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83980a0790 0x7f83980a2c40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f83ac101ea0 0x7f83ac073680 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83ac100320 0x7f83ac074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 --2- 172.21.15.31:0/2385629989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac101510 0x7f83ac073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.787+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 >> 172.21.15.31:0/2385629989 conn(0x7f83ac0fb920 msgr2=0x7f83ac0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.788+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 shutdown_connections 2023-07-29T16:13:58.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:58.788+0000 7f83b33ff700 1 -- 172.21.15.31:0/2385629989 wait complete. 2023-07-29T16:13:58.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:13:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2111115373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138074d30 msgr2=0x7fa138073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2111115373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138074d30 0x7fa138073180 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa128009990 tx=0x7fa128009ca0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2111115373 shutdown_connections 2023-07-29T16:13:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2111115373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1381081c0 0x7fa13810a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2111115373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138074d30 0x7fa138073180 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2111115373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa138073750 0x7fa138073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2111115373 >> 172.21.15.31:0/2111115373 conn(0x7fa1380fb970 msgr2=0x7fa1380fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:59.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2111115373 shutdown_connections 2023-07-29T16:13:59.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.013+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2111115373 wait complete. 2023-07-29T16:13:59.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.014+0000 7fa13e5ff700 1 Processor -- start 2023-07-29T16:13:59.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 -- start start 2023-07-29T16:13:59.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 0x7fa138198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa138074d30 0x7fa138198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1381081c0 0x7fa13819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13819fdc0 con 0x7fa138073750 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13819ff00 con 0x7fa138074d30 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1381a0090 con 0x7fa1381081c0 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa137fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 0x7fa138198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa137fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 0x7fa138198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49904/0 (socket says 172.21.15.31:49904) 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa137fff700 1 -- 172.21.15.31:0/2352229263 learned_addr learned my addr 172.21.15.31:0/2352229263 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:59.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa1377fe700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa138074d30 0x7fa138198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.015+0000 7fa13cb9c700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1381081c0 0x7fa13819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa137fff700 1 -- 172.21.15.31:0/2352229263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1381081c0 msgr2=0x7fa13819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa137fff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1381081c0 0x7fa13819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa137fff700 1 -- 172.21.15.31:0/2352229263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa138074d30 msgr2=0x7fa138198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa137fff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa138074d30 0x7fa138198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa137fff700 1 -- 172.21.15.31:0/2352229263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1280095f0 con 0x7fa138073750 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa137fff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 0x7fa138198240 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa128000c00 tx=0x7fa12800f460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12801d070 con 0x7fa138073750 2023-07-29T16:13:59.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1381a0310 con 0x7fa138073750 2023-07-29T16:13:59.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa12800fc00 con 0x7fa138073750 2023-07-29T16:13:59.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128017650 con 0x7fa138073750 2023-07-29T16:13:59.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.016+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1381a07c0 con 0x7fa138073750 2023-07-29T16:13:59.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.017+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa138106900 con 0x7fa138073750 2023-07-29T16:13:59.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.019+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa128021410 con 0x7fa138073750 2023-07-29T16:13:59.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.021+0000 7fa1357fa700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1200a0790 0x7fa1200a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.021+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa1280cd080 con 0x7fa138073750 2023-07-29T16:13:59.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.021+0000 7fa1377fe700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1200a0790 0x7fa1200a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.022+0000 7fa1377fe700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1200a0790 0x7fa1200a2c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa12c007c60 tx=0x7fa12c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:59.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.024+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fa128096af0 con 0x7fa138073750 2023-07-29T16:13:59.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:58 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/988129333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:59.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:58 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1002668048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:59.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:58 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/988129333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:59.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:58 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1002668048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:59.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:58 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/988129333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:13:59.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:58 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1002668048' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:13:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.233+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa138061960 con 0x7fa138073750 2023-07-29T16:13:59.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.233+0000 7fa1357fa700 1 -- 172.21.15.31:0/2352229263 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1280260a0 con 0x7fa138073750 2023-07-29T16:13:59.232 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:13:59.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1200a0790 msgr2=0x7fa1200a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1200a0790 0x7fa1200a2c40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa12c007c60 tx=0x7fa12c0073d0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 msgr2=0x7fa138198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 0x7fa138198240 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa128000c00 tx=0x7fa12800f460 comp rx=0 tx=0).stop 2023-07-29T16:13:59.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 shutdown_connections 2023-07-29T16:13:59.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa1200a0790 0x7fa1200a2c40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1381081c0 0x7fa13819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa138073750 0x7fa138198240 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 --2- 172.21.15.31:0/2352229263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa138074d30 0x7fa138198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.235+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 >> 172.21.15.31:0/2352229263 conn(0x7fa1380fb970 msgr2=0x7fa1380fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:59.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.236+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 shutdown_connections 2023-07-29T16:13:59.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.236+0000 7fa13e5ff700 1 -- 172.21.15.31:0/2352229263 wait complete. 2023-07-29T16:13:59.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:13:59.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 -- 172.21.15.31:0/2648092061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0100340 msgr2=0x7f86c0100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 --2- 172.21.15.31:0/2648092061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0100340 0x7f86c0100750 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0098f0 tx=0x7f86bc009c00 comp rx=0 tx=0).stop 2023-07-29T16:13:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 -- 172.21.15.31:0/2648092061 shutdown_connections 2023-07-29T16:13:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 --2- 172.21.15.31:0/2648092061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 0x7f86c0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 --2- 172.21.15.31:0/2648092061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0100340 0x7f86c0100750 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 --2- 172.21.15.31:0/2648092061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c0101530 0x7f86c0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 -- 172.21.15.31:0/2648092061 >> 172.21.15.31:0/2648092061 conn(0x7f86c00fb920 msgr2=0x7f86c00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 -- 172.21.15.31:0/2648092061 shutdown_connections 2023-07-29T16:13:59.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.458+0000 7f86c7fff700 1 -- 172.21.15.31:0/2648092061 wait complete. 2023-07-29T16:13:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.459+0000 7f86c7fff700 1 Processor -- start 2023-07-29T16:13:59.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.459+0000 7f86c7fff700 1 -- start start 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c0100340 0x7f86c0198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 0x7f86c0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 0x7f86c019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c7fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c019fe80 con 0x7f86c0101530 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c7fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c019ffc0 con 0x7f86c0100340 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c7fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c01a0150 con 0x7f86c0101ec0 2023-07-29T16:13:59.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 0x7f86c0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 0x7f86c0198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49934/0 (socket says 172.21.15.31:49934) 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 -- 172.21.15.31:0/1379449102 learned_addr learned my addr 172.21.15.31:0/1379449102 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c5d9b700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c0100340 0x7f86c0198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c659c700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 0x7f86c019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 -- 172.21.15.31:0/1379449102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 msgr2=0x7f86c019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 0x7f86c019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 -- 172.21.15.31:0/1379449102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c0100340 msgr2=0x7f86c0198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c0100340 0x7f86c0198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 -- 172.21.15.31:0/1379449102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc009520 con 0x7f86c0101530 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c659c700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 0x7f86c019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.460+0000 7f86c559a700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 0x7f86c0198840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f86b000e9a0 tx=0x7f86b000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.461+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0009790 con 0x7f86c0101530 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.461+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b000cbd0 con 0x7f86c0101530 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.461+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b0018680 con 0x7f86c0101530 2023-07-29T16:13:59.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.461+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c01a0430 con 0x7f86c0101530 2023-07-29T16:13:59.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.461+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c01a0940 con 0x7f86c0101530 2023-07-29T16:13:59.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.462+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c0192a20 con 0x7f86c0101530 2023-07-29T16:13:59.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.464+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f86b000cd40 con 0x7f86c0101530 2023-07-29T16:13:59.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.465+0000 7f86b6ffd700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0860 0x7f86ac0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.465+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f86b0014070 con 0x7f86c0101530 2023-07-29T16:13:59.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.466+0000 7f86c5d9b700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0860 0x7f86ac0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.466+0000 7f86c5d9b700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0860 0x7f86ac0a2d10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0098c0 tx=0x7f86bc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:59.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.469+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f86b0051720 con 0x7f86c0101530 2023-07-29T16:13:59.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.678+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c002cc30 con 0x7f86c0101530 2023-07-29T16:13:59.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.680+0000 7f86b6ffd700 1 -- 172.21.15.31:0/1379449102 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f86b0095900 con 0x7f86c0101530 2023-07-29T16:13:59.679 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:13:59.680 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:13:50.771629+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:13:59.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0860 msgr2=0x7f86ac0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0860 0x7f86ac0a2d10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0098c0 tx=0x7f86bc019040 comp rx=0 tx=0).stop 2023-07-29T16:13:59.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 msgr2=0x7f86c0198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 0x7f86c0198840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f86b000e9a0 tx=0x7f86b000ed60 comp rx=0 tx=0).stop 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 shutdown_connections 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0860 0x7f86ac0a2d10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c0101ec0 0x7f86c019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c0100340 0x7f86c0198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 --2- 172.21.15.31:0/1379449102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c0101530 0x7f86c0198840 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.682+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 >> 172.21.15.31:0/1379449102 conn(0x7f86c00fb920 msgr2=0x7f86c0106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.683+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 shutdown_connections 2023-07-29T16:13:59.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.683+0000 7f86c7fff700 1 -- 172.21.15.31:0/1379449102 wait complete. 2023-07-29T16:13:59.699 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:13:59.700 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:13:59.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 -- 172.21.15.31:0/1240533933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 msgr2=0x7f02c4100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 --2- 172.21.15.31:0/1240533933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4100740 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02b40098f0 tx=0x7f02b4009c00 comp rx=0 tx=0).stop 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 -- 172.21.15.31:0/1240533933 shutdown_connections 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 --2- 172.21.15.31:0/1240533933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 0x7f02c4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 --2- 172.21.15.31:0/1240533933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4100740 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 --2- 172.21.15.31:0/1240533933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 0x7f02c4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 -- 172.21.15.31:0/1240533933 >> 172.21.15.31:0/1240533933 conn(0x7f02c40fb970 msgr2=0x7f02c40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 -- 172.21.15.31:0/1240533933 shutdown_connections 2023-07-29T16:13:59.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.909+0000 7f02c93ff700 1 -- 172.21.15.31:0/1240533933 wait complete. 2023-07-29T16:13:59.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.910+0000 7f02c93ff700 1 Processor -- start 2023-07-29T16:13:59.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.910+0000 7f02c93ff700 1 -- start start 2023-07-29T16:13:59.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c93ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c93ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 0x7f02c41987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c93ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 0x7f02c419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c93ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c419fe00 con 0x7f02c4101eb0 2023-07-29T16:13:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c93ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c419ff40 con 0x7f02c4101520 2023-07-29T16:13:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c93ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c41a00d0 con 0x7f02c4100330 2023-07-29T16:13:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c27fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 0x7f02c41987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 0x7f02c419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c27fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 0x7f02c41987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34974/0 (socket says 172.21.15.31:34974) 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54184/0 (socket says 172.21.15.31:54184) 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c27fc700 1 -- 172.21.15.31:0/3970910956 learned_addr learned my addr 172.21.15.31:0/3970910956 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 -- 172.21.15.31:0/3970910956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 msgr2=0x7f02c41987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 0x7f02c41987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 -- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 msgr2=0x7f02c419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 0x7f02c419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c2ffd700 1 -- 172.21.15.31:0/3970910956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80095f0 con 0x7f02c4100330 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.911+0000 7f02c37fe700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 0x7f02c419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.912+0000 7f02c2ffd700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4198280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000c00 tx=0x7f02b400b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.912+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b401c070 con 0x7f02c4100330 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.912+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b400f7d0 con 0x7f02c4100330 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.912+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b40039b0 con 0x7f02c4100330 2023-07-29T16:13:59.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.912+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b4009520 con 0x7f02c4100330 2023-07-29T16:13:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.912+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c41a05f0 con 0x7f02c4100330 2023-07-29T16:13:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.913+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c41929d0 con 0x7f02c4100330 2023-07-29T16:13:59.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.916+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f02b4020030 con 0x7f02c4100330 2023-07-29T16:13:59.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.917+0000 7f02abfff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0790 0x7f02ac0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:13:59.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.917+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02b40cce30 con 0x7f02c4100330 2023-07-29T16:13:59.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.917+0000 7f02c27fc700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0790 0x7f02ac0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:13:59.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.918+0000 7f02c27fc700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0790 0x7f02ac0a2c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005d10 tx=0x7f02b8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:13:59.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:13:59.919+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f02b40968a0 con 0x7f02c4100330 2023-07-29T16:14:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:59 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2385629989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:59 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2352229263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:59 smithi156 ceph-mon[129245]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 115 B/s, 0 objects/s recovering 2023-07-29T16:14:00.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:13:59 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1379449102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:00.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.127+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02c402cdc0 con 0x7f02c4100330 2023-07-29T16:14:00.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.127+0000 7f02abfff700 1 -- 172.21.15.31:0/3970910956 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02b4096020 con 0x7f02c4100330 2023-07-29T16:14:00.126 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:14:00.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0790 msgr2=0x7f02ac0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0790 0x7f02ac0a2c40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005d10 tx=0x7f02b8009310 comp rx=0 tx=0).stop 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 msgr2=0x7f02c4198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4198280 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f02b4000c00 tx=0x7f02b400b790 comp rx=0 tx=0).stop 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 shutdown_connections 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0790 0x7f02ac0a2c40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c4101eb0 0x7f02c419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4100330 0x7f02c4198280 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 --2- 172.21.15.31:0/3970910956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c4101520 0x7f02c41987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 >> 172.21.15.31:0/3970910956 conn(0x7f02c40fb970 msgr2=0x7f02c4106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 shutdown_connections 2023-07-29T16:14:00.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.130+0000 7f02c93ff700 1 -- 172.21.15.31:0/3970910956 wait complete. 2023-07-29T16:14:00.144 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:59 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2385629989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:59 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2352229263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:59 smithi031 ceph-mon[95504]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 115 B/s, 0 objects/s recovering 2023-07-29T16:14:00.145 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:13:59 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1379449102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:00.147 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:14:00.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:59 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2385629989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:59 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2352229263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:59 smithi039 ceph-mon[128858]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s; 115 B/s, 0 objects/s recovering 2023-07-29T16:14:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:13:59 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1379449102' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:00.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 -- 172.21.15.31:0/3110477448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f4105fe0 msgr2=0x7f26f4108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3110477448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f4105fe0 0x7f26f4108440 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00b0a0 tx=0x7f26ec00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 -- 172.21.15.31:0/3110477448 shutdown_connections 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3110477448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f4105fe0 0x7f26f4108440 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3110477448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26f4100da0 0x7f26f4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3110477448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f41036c0 0x7f26f4105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 -- 172.21.15.31:0/3110477448 >> 172.21.15.31:0/3110477448 conn(0x7f26f40fa840 msgr2=0x7f26f40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:00.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 -- 172.21.15.31:0/3110477448 shutdown_connections 2023-07-29T16:14:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.355+0000 7f26f9bff700 1 -- 172.21.15.31:0/3110477448 wait complete. 2023-07-29T16:14:00.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.356+0000 7f26f9bff700 1 Processor -- start 2023-07-29T16:14:00.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.356+0000 7f26f9bff700 1 -- start start 2023-07-29T16:14:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f9bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f4100da0 0x7f26f4100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f9bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 0x7f26f40fed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f9bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26f4105fe0 0x7f26f40ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f9bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40ffa30 con 0x7f26f41036c0 2023-07-29T16:14:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f9bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40ffba0 con 0x7f26f4100da0 2023-07-29T16:14:00.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f9bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f41a01d0 con 0x7f26f4105fe0 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 0x7f26f40fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 0x7f26f40fed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43294/0 (socket says 172.21.15.31:43294) 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 -- 172.21.15.31:0/3880955679 learned_addr learned my addr 172.21.15.31:0/3880955679 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f37fe700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f4100da0 0x7f26f4100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f3fff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26f4105fe0 0x7f26f40ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 -- 172.21.15.31:0/3880955679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26f4105fe0 msgr2=0x7f26f40ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26f4105fe0 0x7f26f40ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 -- 172.21.15.31:0/3880955679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f4100da0 msgr2=0x7f26f4100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f4100da0 0x7f26f4100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.357+0000 7f26f2ffd700 1 -- 172.21.15.31:0/3880955679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec009d00 con 0x7f26f41036c0 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.358+0000 7f26f2ffd700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 0x7f26f40fed50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f26e800e9a0 tx=0x7f26e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.358+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8009790 con 0x7f26f41036c0 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.358+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26e8010430 con 0x7f26f41036c0 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.358+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8018680 con 0x7f26f41036c0 2023-07-29T16:14:00.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.358+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41a04b0 con 0x7f26f41036c0 2023-07-29T16:14:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.358+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41a0940 con 0x7f26f41036c0 2023-07-29T16:14:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.359+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f40fc3b0 con 0x7f26f41036c0 2023-07-29T16:14:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.361+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f26e80105a0 con 0x7f26f41036c0 2023-07-29T16:14:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.362+0000 7f26f0ff9700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f26dc0a0810 0x7f26dc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.362+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f26e8014070 con 0x7f26f41036c0 2023-07-29T16:14:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.363+0000 7f26f37fe700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f26dc0a0810 0x7f26dc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.363+0000 7f26f37fe700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f26dc0a0810 0x7f26dc0a2cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009c60 tx=0x7f26e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.366+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f26e8051720 con 0x7f26f41036c0 2023-07-29T16:14:00.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.573+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f26f4061a60 con 0x7f26f41036c0 2023-07-29T16:14:00.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.574+0000 7f26f0ff9700 1 -- 172.21.15.31:0/3880955679 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f26e8095770 con 0x7f26f41036c0 2023-07-29T16:14:00.572 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:14:00.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f26dc0a0810 msgr2=0x7f26dc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f26dc0a0810 0x7f26dc0a2cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009c60 tx=0x7f26e4009260 comp rx=0 tx=0).stop 2023-07-29T16:14:00.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 msgr2=0x7f26f40fed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 0x7f26f40fed50 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f26e800e9a0 tx=0x7f26e800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 shutdown_connections 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f26dc0a0810 0x7f26dc0a2cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26f4105fe0 0x7f26f40ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f26f4100da0 0x7f26f4100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 --2- 172.21.15.31:0/3880955679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26f41036c0 0x7f26f40fed50 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 >> 172.21.15.31:0/3880955679 conn(0x7f26f40fa840 msgr2=0x7f26f4101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:00.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 shutdown_connections 2023-07-29T16:14:00.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.576+0000 7f26f9bff700 1 -- 172.21.15.31:0/3880955679 wait complete. 2023-07-29T16:14:00.593 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:14:00.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 -- 172.21.15.31:0/590052016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00100330 msgr2=0x7fcc00100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 --2- 172.21.15.31:0/590052016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00100330 0x7fcc00100740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009990 tx=0x7fcbfc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 -- 172.21.15.31:0/590052016 shutdown_connections 2023-07-29T16:14:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 --2- 172.21.15.31:0/590052016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00101eb0 0x7fcc00108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 --2- 172.21.15.31:0/590052016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00100330 0x7fcc00100740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 --2- 172.21.15.31:0/590052016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101520 0x7fcc00101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 -- 172.21.15.31:0/590052016 >> 172.21.15.31:0/590052016 conn(0x7fcc000fb970 msgr2=0x7fcc000fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:00.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 -- 172.21.15.31:0/590052016 shutdown_connections 2023-07-29T16:14:00.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.801+0000 7fcc079ff700 1 -- 172.21.15.31:0/590052016 wait complete. 2023-07-29T16:14:00.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.802+0000 7fcc079ff700 1 Processor -- start 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.802+0000 7fcc079ff700 1 -- start start 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc079ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00100330 0x7fcc001982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc079ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 0x7fcc001987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc079ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101eb0 0x7fcc0019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc079ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0019fe20 con 0x7fcc00101eb0 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc079ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0019ff60 con 0x7fcc00100330 2023-07-29T16:14:00.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc079ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc001a00f0 con 0x7fcc00101520 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc04f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 0x7fcc001987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc05f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101eb0 0x7fcc0019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc0579b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00100330 0x7fcc001982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc04f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 0x7fcc001987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54218/0 (socket says 172.21.15.31:54218) 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.803+0000 7fcc04f9a700 1 -- 172.21.15.31:0/226464359 learned_addr learned my addr 172.21.15.31:0/226464359 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc05f9c700 1 -- 172.21.15.31:0/226464359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 msgr2=0x7fcc001987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc05f9c700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 0x7fcc001987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc05f9c700 1 -- 172.21.15.31:0/226464359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00100330 msgr2=0x7fcc001982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc05f9c700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00100330 0x7fcc001982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc05f9c700 1 -- 172.21.15.31:0/226464359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc0095f0 con 0x7fcc00101eb0 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc04f9a700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 0x7fcc001987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc0579b700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00100330 0x7fcc001982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc05f9c700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101eb0 0x7fcc0019f850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800efc0 tx=0x7fcbf800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8018070 con 0x7fcc00101eb0 2023-07-29T16:14:00.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc001a0370 con 0x7fcc00101eb0 2023-07-29T16:14:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcbf800f040 con 0x7fcc00101eb0 2023-07-29T16:14:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf800cd40 con 0x7fcc00101eb0 2023-07-29T16:14:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.804+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc001a08d0 con 0x7fcc00101eb0 2023-07-29T16:14:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.805+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc001929d0 con 0x7fcc00101eb0 2023-07-29T16:14:00.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.807+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fcbf8003a00 con 0x7fcc00101eb0 2023-07-29T16:14:00.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.808+0000 7fcbf67fc700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcbec0a0790 0x7fcbec0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:00.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.809+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcbf80cb6c0 con 0x7fcc00101eb0 2023-07-29T16:14:00.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.809+0000 7fcc0579b700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcbec0a0790 0x7fcbec0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:00.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.809+0000 7fcc0579b700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcbec0a0790 0x7fcbec0a2c40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00b2e0 tx=0x7fcbfc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:00.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:00.812+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcbf8095130 con 0x7fcc00101eb0 2023-07-29T16:14:00.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:00 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3970910956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:14:00.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:00 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3880955679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.022+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc00061960 con 0x7fcc00101eb0 2023-07-29T16:14:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.022+0000 7fcbf67fc700 1 -- 172.21.15.31:0/226464359 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbf8098750 con 0x7fcc00101eb0 2023-07-29T16:14:01.021 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:14:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcbec0a0790 msgr2=0x7fcbec0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcbec0a0790 0x7fcbec0a2c40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc00b2e0 tx=0x7fcbfc005fb0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101eb0 msgr2=0x7fcc0019f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101eb0 0x7fcc0019f850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800efc0 tx=0x7fcbf800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 shutdown_connections 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcbec0a0790 0x7fcbec0a2c40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc00101eb0 0x7fcc0019f850 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc00100330 0x7fcc001982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 --2- 172.21.15.31:0/226464359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc00101520 0x7fcc001987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.024+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 >> 172.21.15.31:0/226464359 conn(0x7fcc000fb970 msgr2=0x7fcc00106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.025+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 shutdown_connections 2023-07-29T16:14:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.025+0000 7fcc079ff700 1 -- 172.21.15.31:0/226464359 wait complete. 2023-07-29T16:14:01.041 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-07-29T16:14:01.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:14:01.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:00 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3970910956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:14:01.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:00 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3880955679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 -- 172.21.15.31:0/2876105403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 msgr2=0x7f0b280731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 --2- 172.21.15.31:0/2876105403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 0x7f0b280731e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c0098f0 tx=0x7f0b1c009c00 comp rx=0 tx=0).stop 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 -- 172.21.15.31:0/2876105403 shutdown_connections 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 --2- 172.21.15.31:0/2876105403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b2810c910 0x7f0b2810ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 --2- 172.21.15.31:0/2876105403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 0x7f0b280731e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 --2- 172.21.15.31:0/2876105403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b280737b0 0x7f0b28073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 -- 172.21.15.31:0/2876105403 >> 172.21.15.31:0/2876105403 conn(0x7f0b280fb920 msgr2=0x7f0b280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:01.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 -- 172.21.15.31:0/2876105403 shutdown_connections 2023-07-29T16:14:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.247+0000 7f0b2efff700 1 -- 172.21.15.31:0/2876105403 wait complete. 2023-07-29T16:14:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.248+0000 7f0b2efff700 1 Processor -- start 2023-07-29T16:14:01.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.248+0000 7f0b2efff700 1 -- start start 2023-07-29T16:14:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 0x7f0b2819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 0x7f0b2819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b2810c910 0x7f0b281a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281a4220 con 0x7f0b280737b0 2023-07-29T16:14:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281a4360 con 0x7f0b28074d90 2023-07-29T16:14:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281a44f0 con 0x7f0b2810c910 2023-07-29T16:14:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 0x7f0b2819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 0x7f0b2819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43332/0 (socket says 172.21.15.31:43332) 2023-07-29T16:14:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 -- 172.21.15.31:0/262457666 learned_addr learned my addr 172.21.15.31:0/262457666 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b27fff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 0x7f0b2819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2d59c700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b2810c910 0x7f0b281a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 -- 172.21.15.31:0/262457666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b2810c910 msgr2=0x7f0b281a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b2810c910 0x7f0b281a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 -- 172.21.15.31:0/262457666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 msgr2=0x7f0b2819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 0x7f0b2819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.249+0000 7f0b2cd9b700 1 -- 172.21.15.31:0/262457666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c009520 con 0x7f0b280737b0 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.250+0000 7f0b2cd9b700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 0x7f0b2819c6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009fd0 tx=0x7f0b1c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.250+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c01c070 con 0x7f0b280737b0 2023-07-29T16:14:01.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.250+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b281a4770 con 0x7f0b280737b0 2023-07-29T16:14:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.250+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b1c00f7d0 con 0x7f0b280737b0 2023-07-29T16:14:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.250+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c003b00 con 0x7f0b280737b0 2023-07-29T16:14:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.250+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b281a4bf0 con 0x7f0b280737b0 2023-07-29T16:14:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.251+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b28102460 con 0x7f0b280737b0 2023-07-29T16:14:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.253+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0b1c020030 con 0x7f0b280737b0 2023-07-29T16:14:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.254+0000 7f0b25ffb700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b180a07e0 0x7f0b180a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.254+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0b1c0ccae0 con 0x7f0b280737b0 2023-07-29T16:14:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.255+0000 7f0b27fff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b180a07e0 0x7f0b180a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.255+0000 7f0b27fff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b180a07e0 0x7f0b180a2c90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0b28074ab0 tx=0x7f0b14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.258+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0b1c0965b0 con 0x7f0b280737b0 2023-07-29T16:14:01.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:00 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3970910956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:14:01.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:00 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3880955679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:01.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.515+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b2802d080 con 0x7f0b280737b0 2023-07-29T16:14:01.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.516+0000 7f0b25ffb700 1 -- 172.21.15.31:0/262457666 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f0b1c099bd0 con 0x7f0b280737b0 2023-07-29T16:14:01.514 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:01.515 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:14:01.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.518+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b180a07e0 msgr2=0x7f0b180a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.518+0000 7f0b2efff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b180a07e0 0x7f0b180a2c90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0b28074ab0 tx=0x7f0b14009260 comp rx=0 tx=0).stop 2023-07-29T16:14:01.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.518+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 msgr2=0x7f0b2819c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.518+0000 7f0b2efff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 0x7f0b2819c6a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c009fd0 tx=0x7f0b1c00b790 comp rx=0 tx=0).stop 2023-07-29T16:14:01.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 shutdown_connections 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b180a07e0 0x7f0b180a2c90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b2810c910 0x7f0b281a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b280737b0 0x7f0b2819c6a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 --2- 172.21.15.31:0/262457666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b28074d90 0x7f0b2819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 >> 172.21.15.31:0/262457666 conn(0x7f0b280fb920 msgr2=0x7f0b280fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 shutdown_connections 2023-07-29T16:14:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.519+0000 7f0b2efff700 1 -- 172.21.15.31:0/262457666 wait complete. 2023-07-29T16:14:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:14:01.536 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-07-29T16:14:01.536 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-07-29T16:14:01.536 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-29T16:14:01.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 -- 172.21.15.31:0/1318849454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 msgr2=0x7f97c8101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/1318849454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8101980 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009990 tx=0x7f97b8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 -- 172.21.15.31:0/1318849454 shutdown_connections 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/1318849454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8101ec0 0x7f97c8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/1318849454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8100340 0x7f97c8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/1318849454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8101980 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 -- 172.21.15.31:0/1318849454 >> 172.21.15.31:0/1318849454 conn(0x7f97c80fb920 msgr2=0x7f97c80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:01.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 -- 172.21.15.31:0/1318849454 shutdown_connections 2023-07-29T16:14:01.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.742+0000 7f97cf3ff700 1 -- 172.21.15.31:0/1318849454 wait complete. 2023-07-29T16:14:01.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.743+0000 7f97cf3ff700 1 Processor -- start 2023-07-29T16:14:01.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.743+0000 7f97cf3ff700 1 -- start start 2023-07-29T16:14:01.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.743+0000 7f97cf3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8100340 0x7f97c8074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.743+0000 7f97cf3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cf3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8101ec0 0x7f97c8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cf3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c8073e20 con 0x7f97c8100340 2023-07-29T16:14:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cf3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c8073f90 con 0x7f97c8101530 2023-07-29T16:14:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cf3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c81a0110 con 0x7f97c8101ec0 2023-07-29T16:14:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cd19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8100340 0x7f97c8074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35042/0 (socket says 172.21.15.31:35042) 2023-07-29T16:14:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cd19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8100340 0x7f97c8074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43340/0 (socket says 172.21.15.31:43340) 2023-07-29T16:14:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 -- 172.21.15.31:0/225921754 learned_addr learned my addr 172.21.15.31:0/225921754 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cd99c700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8101ec0 0x7f97c8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 -- 172.21.15.31:0/225921754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8101ec0 msgr2=0x7f97c8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8101ec0 0x7f97c8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 -- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8100340 msgr2=0x7f97c8074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8100340 0x7f97c8074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cc99a700 1 -- 172.21.15.31:0/225921754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80095f0 con 0x7f97c8101530 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.744+0000 7f97cd99c700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8101ec0 0x7f97c8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.745+0000 7f97cc99a700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8073140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f97b8004760 tx=0x7f97b8004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.745+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b801d070 con 0x7f97c8101530 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.745+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b8004830 con 0x7f97c8101530 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.745+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b8005480 con 0x7f97c8101530 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.745+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c81a0390 con 0x7f97c8101530 2023-07-29T16:14:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.745+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c81a0750 con 0x7f97c8101530 2023-07-29T16:14:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.746+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c8107870 con 0x7f97c8101530 2023-07-29T16:14:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.748+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f97b800fad0 con 0x7f97c8101530 2023-07-29T16:14:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.749+0000 7f97be7fc700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97b40a0860 0x7f97b40a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.750+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f97b80cce20 con 0x7f97c8101530 2023-07-29T16:14:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.750+0000 7f97cd19b700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97b40a0860 0x7f97b40a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.750+0000 7f97cd19b700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97b40a0860 0x7f97b40a2d10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f97c8101390 tx=0x7f97c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.752+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f97b8096890 con 0x7f97c8101530 2023-07-29T16:14:01.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.957+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f97c8061190 con 0x7f97b40a0860 2023-07-29T16:14:01.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.970+0000 7f97be7fc700 1 -- 172.21.15.31:0/225921754 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343664 (secure 0 0 0) 0x7f97c8061190 con 0x7f97b40a0860 2023-07-29T16:14:01.971 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:01.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97b40a0860 msgr2=0x7f97b40a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97b40a0860 0x7f97b40a2d10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f97c8101390 tx=0x7f97c4009260 comp rx=0 tx=0).stop 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 msgr2=0x7f97c8073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8073140 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f97b8004760 tx=0x7f97b8004790 comp rx=0 tx=0).stop 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 shutdown_connections 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97b40a0860 0x7f97b40a2d10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c8101ec0 0x7f97c8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c8100340 0x7f97c8074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 --2- 172.21.15.31:0/225921754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c8101530 0x7f97c8073140 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:01.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 >> 172.21.15.31:0/225921754 conn(0x7f97c80fb920 msgr2=0x7f97c81049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:01.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 shutdown_connections 2023-07-29T16:14:01.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:01.974+0000 7f97cf3ff700 1 -- 172.21.15.31:0/225921754 wait complete. 2023-07-29T16:14:01.975 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:14:01.991 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2023-07-29T16:14:01.295437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657289,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":694,"num_read_kb":463,"num_write":347,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1182420,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337148,"kb_used_data":13716,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124604676,"statfs":{"total":1151940427776,"available":1151595188224,"internally_reserved":0,"allocated":14045184,"data_stored":7682421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5483,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":32,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.519441"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778124+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:13:49.778124+0000","last_peered":"2023-07-29T16:13:49.778124+0000","last_clean":"2023-07-29T16:13:49.778124+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:13:49.778124+0000","last_undegraded":"2023-07-29T16:13:49.778124+0000","last_fullsized":"2023-07-29T16:13:49.778124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776048+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:13:49.776048+0000","last_peered":"2023-07-29T16:13:49.776048+0000","last_clean":"2023-07-29T16:13:49.776048+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:13:49.776048+0000","last_undegraded":"2023-07-29T16:13:49.776048+0000","last_fullsized":"2023-07-29T16:13:49.776048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776110+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:13:49.776110+0000","last_peered":"2023-07-29T16:13:49.776110+0000","last_clean":"2023-07-29T16:13:49.776110+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:13:49.776110+0000","last_undegraded":"2023-07-29T16:13:49.776110+0000","last_fullsized":"2023-07-29T16:13:49.776110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830930+0000","last_change":"2023-07-29T16:12:50.286750+0000","last_active":"2023-07-29T16:13:49.830930+0000","last_peered":"2023-07-29T16:13:49.830930+0000","last_clean":"2023-07-29T16:13:49.830930+0000","last_became_active":"2023-07-29T16:12:50.286341+0000","last_became_peered":"2023-07-29T16:12:50.286341+0000","last_unstale":"2023-07-29T16:13:49.830930+0000","last_undegraded":"2023-07-29T16:13:49.830930+0000","last_fullsized":"2023-07-29T16:13:49.830930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.547598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829633+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:13:49.829633+0000","last_peered":"2023-07-29T16:13:49.829633+0000","last_clean":"2023-07-29T16:13:49.829633+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:13:49.829633+0000","last_undegraded":"2023-07-29T16:13:49.829633+0000","last_fullsized":"2023-07-29T16:13:49.829633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830834+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:13:49.830834+0000","last_peered":"2023-07-29T16:13:49.830834+0000","last_clean":"2023-07-29T16:13:49.830834+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:13:49.830834+0000","last_undegraded":"2023-07-29T16:13:49.830834+0000","last_fullsized":"2023-07-29T16:13:49.830834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774908+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:13:49.774908+0000","last_peered":"2023-07-29T16:13:49.774908+0000","last_clean":"2023-07-29T16:13:49.774908+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:13:49.774908+0000","last_undegraded":"2023-07-29T16:13:49.774908+0000","last_fullsized":"2023-07-29T16:13:49.774908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829985+0000","last_change":"2023-07-29T16:12:50.335631+0000","last_active":"2023-07-29T16:13:49.829985+0000","last_peered":"2023-07-29T16:13:49.829985+0000","last_clean":"2023-07-29T16:13:49.829985+0000","last_became_active":"2023-07-29T16:12:50.335213+0000","last_became_peered":"2023-07-29T16:12:50.335213+0000","last_unstale":"2023-07-29T16:13:49.829985+0000","last_undegraded":"2023-07-29T16:13:49.829985+0000","last_fullsized":"2023-07-29T16:13:49.829985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:04:33.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779120+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:13:49.779120+0000","last_peered":"2023-07-29T16:13:49.779120+0000","last_clean":"2023-07-29T16:13:49.779120+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:13:49.779120+0000","last_undegraded":"2023-07-29T16:13:49.779120+0000","last_fullsized":"2023-07-29T16:13:49.779120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830766+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:13:49.830766+0000","last_peered":"2023-07-29T16:13:49.830766+0000","last_clean":"2023-07-29T16:13:49.830766+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:13:49.830766+0000","last_undegraded":"2023-07-29T16:13:49.830766+0000","last_fullsized":"2023-07-29T16:13:49.830766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775219+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:13:49.775219+0000","last_peered":"2023-07-29T16:13:49.775219+0000","last_clean":"2023-07-29T16:13:49.775219+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:13:49.775219+0000","last_undegraded":"2023-07-29T16:13:49.775219+0000","last_fullsized":"2023-07-29T16:13:49.775219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830343+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:13:49.830343+0000","last_peered":"2023-07-29T16:13:49.830343+0000","last_clean":"2023-07-29T16:13:49.830343+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:13:49.830343+0000","last_undegraded":"2023-07-29T16:13:49.830343+0000","last_fullsized":"2023-07-29T16:13:49.830343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830414+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:13:49.830414+0000","last_peered":"2023-07-29T16:13:49.830414+0000","last_clean":"2023-07-29T16:13:49.830414+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:13:49.830414+0000","last_undegraded":"2023-07-29T16:13:49.830414+0000","last_fullsized":"2023-07-29T16:13:49.830414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775983+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:13:49.775983+0000","last_peered":"2023-07-29T16:13:49.775983+0000","last_clean":"2023-07-29T16:13:49.775983+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:13:49.775983+0000","last_undegraded":"2023-07-29T16:13:49.775983+0000","last_fullsized":"2023-07-29T16:13:49.775983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775874+0000","last_change":"2023-07-29T16:12:50.291287+0000","last_active":"2023-07-29T16:13:49.775874+0000","last_peered":"2023-07-29T16:13:49.775874+0000","last_clean":"2023-07-29T16:13:49.775874+0000","last_became_active":"2023-07-29T16:12:50.290878+0000","last_became_peered":"2023-07-29T16:12:50.290878+0000","last_unstale":"2023-07-29T16:13:49.775874+0000","last_undegraded":"2023-07-29T16:13:49.775874+0000","last_fullsized":"2023-07-29T16:13:49.775874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:59:20.470299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778600+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:13:49.778600+0000","last_peered":"2023-07-29T16:13:49.778600+0000","last_clean":"2023-07-29T16:13:49.778600+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:13:49.778600+0000","last_undegraded":"2023-07-29T16:13:49.778600+0000","last_fullsized":"2023-07-29T16:13:49.778600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778403+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:13:49.778403+0000","last_peered":"2023-07-29T16:13:49.778403+0000","last_clean":"2023-07-29T16:13:49.778403+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:13:49.778403+0000","last_undegraded":"2023-07-29T16:13:49.778403+0000","last_fullsized":"2023-07-29T16:13:49.778403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831266+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:13:49.831266+0000","last_peered":"2023-07-29T16:13:49.831266+0000","last_clean":"2023-07-29T16:13:49.831266+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:13:49.831266+0000","last_undegraded":"2023-07-29T16:13:49.831266+0000","last_fullsized":"2023-07-29T16:13:49.831266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774488+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:13:49.774488+0000","last_peered":"2023-07-29T16:13:49.774488+0000","last_clean":"2023-07-29T16:13:49.774488+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:13:49.774488+0000","last_undegraded":"2023-07-29T16:13:49.774488+0000","last_fullsized":"2023-07-29T16:13:49.774488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830125+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:13:49.830125+0000","last_peered":"2023-07-29T16:13:49.830125+0000","last_clean":"2023-07-29T16:13:49.830125+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:13:49.830125+0000","last_undegraded":"2023-07-29T16:13:49.830125+0000","last_fullsized":"2023-07-29T16:13:49.830125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830525+0000","last_change":"2023-07-29T16:12:50.286738+0000","last_active":"2023-07-29T16:13:49.830525+0000","last_peered":"2023-07-29T16:13:49.830525+0000","last_clean":"2023-07-29T16:13:49.830525+0000","last_became_active":"2023-07-29T16:12:50.286379+0000","last_became_peered":"2023-07-29T16:12:50.286379+0000","last_unstale":"2023-07-29T16:13:49.830525+0000","last_undegraded":"2023-07-29T16:13:49.830525+0000","last_fullsized":"2023-07-29T16:13:49.830525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:50:03.338896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830606+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:13:49.830606+0000","last_peered":"2023-07-29T16:13:49.830606+0000","last_clean":"2023-07-29T16:13:49.830606+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:13:49.830606+0000","last_undegraded":"2023-07-29T16:13:49.830606+0000","last_fullsized":"2023-07-29T16:13:49.830606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776300+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:13:49.776300+0000","last_peered":"2023-07-29T16:13:49.776300+0000","last_clean":"2023-07-29T16:13:49.776300+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:13:49.776300+0000","last_undegraded":"2023-07-29T16:13:49.776300+0000","last_fullsized":"2023-07-29T16:13:49.776300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830130+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:13:49.830130+0000","last_peered":"2023-07-29T16:13:49.830130+0000","last_clean":"2023-07-29T16:13:49.830130+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:13:49.830130+0000","last_undegraded":"2023-07-29T16:13:49.830130+0000","last_fullsized":"2023-07-29T16:13:49.830130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778941+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:13:49.778941+0000","last_peered":"2023-07-29T16:13:49.778941+0000","last_clean":"2023-07-29T16:13:49.778941+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:13:49.778941+0000","last_undegraded":"2023-07-29T16:13:49.778941+0000","last_fullsized":"2023-07-29T16:13:49.778941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777906+0000","last_change":"2023-07-29T16:12:50.288144+0000","last_active":"2023-07-29T16:13:49.777906+0000","last_peered":"2023-07-29T16:13:49.777906+0000","last_clean":"2023-07-29T16:13:49.777906+0000","last_became_active":"2023-07-29T16:12:50.283877+0000","last_became_peered":"2023-07-29T16:12:50.283877+0000","last_unstale":"2023-07-29T16:13:49.777906+0000","last_undegraded":"2023-07-29T16:13:49.777906+0000","last_fullsized":"2023-07-29T16:13:49.777906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T04:00:21.950054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778880+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:13:49.778880+0000","last_peered":"2023-07-29T16:13:49.778880+0000","last_clean":"2023-07-29T16:13:49.778880+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:13:49.778880+0000","last_undegraded":"2023-07-29T16:13:49.778880+0000","last_fullsized":"2023-07-29T16:13:49.778880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:57.798989+0000","last_change":"2023-07-29T16:13:50.790518+0000","last_active":"2023-07-29T16:13:57.798989+0000","last_peered":"2023-07-29T16:13:57.798989+0000","last_clean":"2023-07-29T16:13:57.798989+0000","last_became_active":"2023-07-29T16:13:50.783618+0000","last_became_peered":"2023-07-29T16:13:50.783618+0000","last_unstale":"2023-07-29T16:13:57.798989+0000","last_undegraded":"2023-07-29T16:13:57.798989+0000","last_fullsized":"2023-07-29T16:13:57.798989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:39:28.814617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1418,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829658+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:13:49.829658+0000","last_peered":"2023-07-29T16:13:49.829658+0000","last_clean":"2023-07-29T16:13:49.829658+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:13:49.829658+0000","last_undegraded":"2023-07-29T16:13:49.829658+0000","last_fullsized":"2023-07-29T16:13:49.829658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:50.877482+0000","last_change":"2023-07-29T16:13:50.877482+0000","last_active":"2023-07-29T16:13:50.877482+0000","last_peered":"2023-07-29T16:13:50.877482+0000","last_clean":"2023-07-29T16:13:50.877482+0000","last_became_active":"2023-07-29T16:13:50.874496+0000","last_became_peered":"2023-07-29T16:13:50.874496+0000","last_unstale":"2023-07-29T16:13:50.877482+0000","last_undegraded":"2023-07-29T16:13:50.877482+0000","last_fullsized":"2023-07-29T16:13:50.877482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:47:18.956678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779443+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:13:49.779443+0000","last_peered":"2023-07-29T16:13:49.779443+0000","last_clean":"2023-07-29T16:13:49.779443+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:13:49.779443+0000","last_undegraded":"2023-07-29T16:13:49.779443+0000","last_fullsized":"2023-07-29T16:13:49.779443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831234+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:13:49.831234+0000","last_peered":"2023-07-29T16:13:49.831234+0000","last_clean":"2023-07-29T16:13:49.831234+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:13:49.831234+0000","last_undegraded":"2023-07-29T16:13:49.831234+0000","last_fullsized":"2023-07-29T16:13:49.831234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830062+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:13:49.830062+0000","last_peered":"2023-07-29T16:13:49.830062+0000","last_clean":"2023-07-29T16:13:49.830062+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:13:49.830062+0000","last_undegraded":"2023-07-29T16:13:49.830062+0000","last_fullsized":"2023-07-29T16:13:49.830062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778416+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:13:49.778416+0000","last_peered":"2023-07-29T16:13:49.778416+0000","last_clean":"2023-07-29T16:13:49.778416+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:13:49.778416+0000","last_undegraded":"2023-07-29T16:13:49.778416+0000","last_fullsized":"2023-07-29T16:13:49.778416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830370+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:13:49.830370+0000","last_peered":"2023-07-29T16:13:49.830370+0000","last_clean":"2023-07-29T16:13:49.830370+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:13:49.830370+0000","last_undegraded":"2023-07-29T16:13:49.830370+0000","last_fullsized":"2023-07-29T16:13:49.830370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":110,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:55.795897+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:13:55.795897+0000","last_peered":"2023-07-29T16:13:55.795897+0000","last_clean":"2023-07-29T16:13:55.795897+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:13:55.795897+0000","last_undegraded":"2023-07-29T16:13:55.795897+0000","last_fullsized":"2023-07-29T16:13:55.795897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779064+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:13:49.779064+0000","last_peered":"2023-07-29T16:13:49.779064+0000","last_clean":"2023-07-29T16:13:49.779064+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:13:49.779064+0000","last_undegraded":"2023-07-29T16:13:49.779064+0000","last_fullsized":"2023-07-29T16:13:49.779064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775359+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:13:49.775359+0000","last_peered":"2023-07-29T16:13:49.775359+0000","last_clean":"2023-07-29T16:13:49.775359+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:13:49.775359+0000","last_undegraded":"2023-07-29T16:13:49.775359+0000","last_fullsized":"2023-07-29T16:13:49.775359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776218+0000","last_change":"2023-07-29T16:12:50.336088+0000","last_active":"2023-07-29T16:13:49.776218+0000","last_peered":"2023-07-29T16:13:49.776218+0000","last_clean":"2023-07-29T16:13:49.776218+0000","last_became_active":"2023-07-29T16:12:50.335715+0000","last_became_peered":"2023-07-29T16:12:50.335715+0000","last_unstale":"2023-07-29T16:13:49.776218+0000","last_undegraded":"2023-07-29T16:13:49.776218+0000","last_fullsized":"2023-07-29T16:13:49.776218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:53:33.332020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778964+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:13:49.778964+0000","last_peered":"2023-07-29T16:13:49.778964+0000","last_clean":"2023-07-29T16:13:49.778964+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:13:49.778964+0000","last_undegraded":"2023-07-29T16:13:49.778964+0000","last_fullsized":"2023-07-29T16:13:49.778964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779024+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:13:49.779024+0000","last_peered":"2023-07-29T16:13:49.779024+0000","last_clean":"2023-07-29T16:13:49.779024+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:13:49.779024+0000","last_undegraded":"2023-07-29T16:13:49.779024+0000","last_fullsized":"2023-07-29T16:13:49.779024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":234,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:56.939908+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:13:56.939908+0000","last_peered":"2023-07-29T16:13:56.939908+0000","last_clean":"2023-07-29T16:13:56.939908+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:13:56.939908+0000","last_undegraded":"2023-07-29T16:13:56.939908+0000","last_fullsized":"2023-07-29T16:13:56.939908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61789,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":188,"num_read_kb":149,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829885+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:13:49.829885+0000","last_peered":"2023-07-29T16:13:49.829885+0000","last_clean":"2023-07-29T16:13:49.829885+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:13:49.829885+0000","last_undegraded":"2023-07-29T16:13:49.829885+0000","last_fullsized":"2023-07-29T16:13:49.829885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774452+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:13:49.774452+0000","last_peered":"2023-07-29T16:13:49.774452+0000","last_clean":"2023-07-29T16:13:49.774452+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:13:49.774452+0000","last_undegraded":"2023-07-29T16:13:49.774452+0000","last_fullsized":"2023-07-29T16:13:49.774452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830632+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:13:49.830632+0000","last_peered":"2023-07-29T16:13:49.830632+0000","last_clean":"2023-07-29T16:13:49.830632+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:13:49.830632+0000","last_undegraded":"2023-07-29T16:13:49.830632+0000","last_fullsized":"2023-07-29T16:13:49.830632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779295+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:13:49.779295+0000","last_peered":"2023-07-29T16:13:49.779295+0000","last_clean":"2023-07-29T16:13:49.779295+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:13:49.779295+0000","last_undegraded":"2023-07-29T16:13:49.779295+0000","last_fullsized":"2023-07-29T16:13:49.779295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775110+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:13:49.775110+0000","last_peered":"2023-07-29T16:13:49.775110+0000","last_clean":"2023-07-29T16:13:49.775110+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:13:49.775110+0000","last_undegraded":"2023-07-29T16:13:49.775110+0000","last_fullsized":"2023-07-29T16:13:49.775110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":117,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:56.883507+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:13:56.883507+0000","last_peered":"2023-07-29T16:13:56.883507+0000","last_clean":"2023-07-29T16:13:56.883507+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:13:56.883507+0000","last_undegraded":"2023-07-29T16:13:56.883507+0000","last_fullsized":"2023-07-29T16:13:56.883507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1068,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775014+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:13:49.775014+0000","last_peered":"2023-07-29T16:13:49.775014+0000","last_clean":"2023-07-29T16:13:49.775014+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:13:49.775014+0000","last_undegraded":"2023-07-29T16:13:49.775014+0000","last_fullsized":"2023-07-29T16:13:49.775014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829533+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:13:49.829533+0000","last_peered":"2023-07-29T16:13:49.829533+0000","last_clean":"2023-07-29T16:13:49.829533+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:13:49.829533+0000","last_undegraded":"2023-07-29T16:13:49.829533+0000","last_fullsized":"2023-07-29T16:13:49.829533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778567+0000","last_change":"2023-07-29T16:12:50.286421+0000","last_active":"2023-07-29T16:13:49.778567+0000","last_peered":"2023-07-29T16:13:49.778567+0000","last_clean":"2023-07-29T16:13:49.778567+0000","last_became_active":"2023-07-29T16:12:50.286075+0000","last_became_peered":"2023-07-29T16:12:50.286075+0000","last_unstale":"2023-07-29T16:13:49.778567+0000","last_undegraded":"2023-07-29T16:13:49.778567+0000","last_fullsized":"2023-07-29T16:13:49.778567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:24:16.006080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829886+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:13:49.829886+0000","last_peered":"2023-07-29T16:13:49.829886+0000","last_clean":"2023-07-29T16:13:49.829886+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:13:49.829886+0000","last_undegraded":"2023-07-29T16:13:49.829886+0000","last_fullsized":"2023-07-29T16:13:49.829886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777964+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:13:49.777964+0000","last_peered":"2023-07-29T16:13:49.777964+0000","last_clean":"2023-07-29T16:13:49.777964+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:13:49.777964+0000","last_undegraded":"2023-07-29T16:13:49.777964+0000","last_fullsized":"2023-07-29T16:13:49.777964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830840+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:13:49.830840+0000","last_peered":"2023-07-29T16:13:49.830840+0000","last_clean":"2023-07-29T16:13:49.830840+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:13:49.830840+0000","last_undegraded":"2023-07-29T16:13:49.830840+0000","last_fullsized":"2023-07-29T16:13:49.830840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830261+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:13:49.830261+0000","last_peered":"2023-07-29T16:13:49.830261+0000","last_clean":"2023-07-29T16:13:49.830261+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:13:49.830261+0000","last_undegraded":"2023-07-29T16:13:49.830261+0000","last_fullsized":"2023-07-29T16:13:49.830261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775445+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:13:49.775445+0000","last_peered":"2023-07-29T16:13:49.775445+0000","last_clean":"2023-07-29T16:13:49.775445+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:13:49.775445+0000","last_undegraded":"2023-07-29T16:13:49.775445+0000","last_fullsized":"2023-07-29T16:13:49.775445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830695+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:13:49.830695+0000","last_peered":"2023-07-29T16:13:49.830695+0000","last_clean":"2023-07-29T16:13:49.830695+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:13:49.830695+0000","last_undegraded":"2023-07-29T16:13:49.830695+0000","last_fullsized":"2023-07-29T16:13:49.830695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778758+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:13:49.778758+0000","last_peered":"2023-07-29T16:13:49.778758+0000","last_clean":"2023-07-29T16:13:49.778758+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:13:49.778758+0000","last_undegraded":"2023-07-29T16:13:49.778758+0000","last_fullsized":"2023-07-29T16:13:49.778758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":114,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:57.884345+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:13:57.884345+0000","last_peered":"2023-07-29T16:13:57.884345+0000","last_clean":"2023-07-29T16:13:57.884345+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:13:57.884345+0000","last_undegraded":"2023-07-29T16:13:57.884345+0000","last_fullsized":"2023-07-29T16:13:57.884345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830891+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:13:49.830891+0000","last_peered":"2023-07-29T16:13:49.830891+0000","last_clean":"2023-07-29T16:13:49.830891+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:13:49.830891+0000","last_undegraded":"2023-07-29T16:13:49.830891+0000","last_fullsized":"2023-07-29T16:13:49.830891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830292+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:13:49.830292+0000","last_peered":"2023-07-29T16:13:49.830292+0000","last_clean":"2023-07-29T16:13:49.830292+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:13:49.830292+0000","last_undegraded":"2023-07-29T16:13:49.830292+0000","last_fullsized":"2023-07-29T16:13:49.830292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":332,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774940+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:13:49.774940+0000","last_peered":"2023-07-29T16:13:49.774940+0000","last_clean":"2023-07-29T16:13:49.774940+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:13:49.774940+0000","last_undegraded":"2023-07-29T16:13:49.774940+0000","last_fullsized":"2023-07-29T16:13:49.774940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831699+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:13:49.831699+0000","last_peered":"2023-07-29T16:13:49.831699+0000","last_clean":"2023-07-29T16:13:49.831699+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:13:49.831699+0000","last_undegraded":"2023-07-29T16:13:49.831699+0000","last_fullsized":"2023-07-29T16:13:49.831699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774897+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:13:49.774897+0000","last_peered":"2023-07-29T16:13:49.774897+0000","last_clean":"2023-07-29T16:13:49.774897+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:13:49.774897+0000","last_undegraded":"2023-07-29T16:13:49.774897+0000","last_fullsized":"2023-07-29T16:13:49.774897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829981+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:13:49.829981+0000","last_peered":"2023-07-29T16:13:49.829981+0000","last_clean":"2023-07-29T16:13:49.829981+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:13:49.829981+0000","last_undegraded":"2023-07-29T16:13:49.829981+0000","last_fullsized":"2023-07-29T16:13:49.829981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831508+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:13:49.831508+0000","last_peered":"2023-07-29T16:13:49.831508+0000","last_clean":"2023-07-29T16:13:49.831508+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:13:49.831508+0000","last_undegraded":"2023-07-29T16:13:49.831508+0000","last_fullsized":"2023-07-29T16:13:49.831508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:52.562037+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:13:52.562037+0000","last_peered":"2023-07-29T16:13:52.562037+0000","last_clean":"2023-07-29T16:13:52.562037+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:13:52.562037+0000","last_undegraded":"2023-07-29T16:13:52.562037+0000","last_fullsized":"2023-07-29T16:13:52.562037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830712+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:13:49.830712+0000","last_peered":"2023-07-29T16:13:49.830712+0000","last_clean":"2023-07-29T16:13:49.830712+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:13:49.830712+0000","last_undegraded":"2023-07-29T16:13:49.830712+0000","last_fullsized":"2023-07-29T16:13:49.830712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830267+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:13:49.830267+0000","last_peered":"2023-07-29T16:13:49.830267+0000","last_clean":"2023-07-29T16:13:49.830267+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:13:49.830267+0000","last_undegraded":"2023-07-29T16:13:49.830267+0000","last_fullsized":"2023-07-29T16:13:49.830267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776129+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:13:49.776129+0000","last_peered":"2023-07-29T16:13:49.776129+0000","last_clean":"2023-07-29T16:13:49.776129+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:13:49.776129+0000","last_undegraded":"2023-07-29T16:13:49.776129+0000","last_fullsized":"2023-07-29T16:13:49.776129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778725+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:13:49.778725+0000","last_peered":"2023-07-29T16:13:49.778725+0000","last_clean":"2023-07-29T16:13:49.778725+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:13:49.778725+0000","last_undegraded":"2023-07-29T16:13:49.778725+0000","last_fullsized":"2023-07-29T16:13:49.778725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829527+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:13:49.829527+0000","last_peered":"2023-07-29T16:13:49.829527+0000","last_clean":"2023-07-29T16:13:49.829527+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:13:49.829527+0000","last_undegraded":"2023-07-29T16:13:49.829527+0000","last_fullsized":"2023-07-29T16:13:49.829527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830174+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:13:49.830174+0000","last_peered":"2023-07-29T16:13:49.830174+0000","last_clean":"2023-07-29T16:13:49.830174+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:13:49.830174+0000","last_undegraded":"2023-07-29T16:13:49.830174+0000","last_fullsized":"2023-07-29T16:13:49.830174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776966+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:13:49.776966+0000","last_peered":"2023-07-29T16:13:49.776966+0000","last_clean":"2023-07-29T16:13:49.776966+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:13:49.776966+0000","last_undegraded":"2023-07-29T16:13:49.776966+0000","last_fullsized":"2023-07-29T16:13:49.776966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:50.781469+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:13:50.781469+0000","last_peered":"2023-07-29T16:13:50.781469+0000","last_clean":"2023-07-29T16:13:50.781469+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:13:50.781469+0000","last_undegraded":"2023-07-29T16:13:50.781469+0000","last_fullsized":"2023-07-29T16:13:50.781469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830383+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:13:49.830383+0000","last_peered":"2023-07-29T16:13:49.830383+0000","last_clean":"2023-07-29T16:13:49.830383+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:13:49.830383+0000","last_undegraded":"2023-07-29T16:13:49.830383+0000","last_fullsized":"2023-07-29T16:13:49.830383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829662+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:13:49.829662+0000","last_peered":"2023-07-29T16:13:49.829662+0000","last_clean":"2023-07-29T16:13:49.829662+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:13:49.829662+0000","last_undegraded":"2023-07-29T16:13:49.829662+0000","last_fullsized":"2023-07-29T16:13:49.829662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775210+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:13:49.775210+0000","last_peered":"2023-07-29T16:13:49.775210+0000","last_clean":"2023-07-29T16:13:49.775210+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:13:49.775210+0000","last_undegraded":"2023-07-29T16:13:49.775210+0000","last_fullsized":"2023-07-29T16:13:49.775210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830449+0000","last_change":"2023-07-29T16:12:50.296260+0000","last_active":"2023-07-29T16:13:49.830449+0000","last_peered":"2023-07-29T16:13:49.830449+0000","last_clean":"2023-07-29T16:13:49.830449+0000","last_became_active":"2023-07-29T16:12:50.290108+0000","last_became_peered":"2023-07-29T16:12:50.290108+0000","last_unstale":"2023-07-29T16:13:49.830449+0000","last_undegraded":"2023-07-29T16:13:49.830449+0000","last_fullsized":"2023-07-29T16:13:49.830449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:52:17.142674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775483+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:13:49.775483+0000","last_peered":"2023-07-29T16:13:49.775483+0000","last_clean":"2023-07-29T16:13:49.775483+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:13:49.775483+0000","last_undegraded":"2023-07-29T16:13:49.775483+0000","last_fullsized":"2023-07-29T16:13:49.775483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778871+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:13:49.778871+0000","last_peered":"2023-07-29T16:13:49.778871+0000","last_clean":"2023-07-29T16:13:49.778871+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:13:49.778871+0000","last_undegraded":"2023-07-29T16:13:49.778871+0000","last_fullsized":"2023-07-29T16:13:49.778871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775173+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:13:49.775173+0000","last_peered":"2023-07-29T16:13:49.775173+0000","last_clean":"2023-07-29T16:13:49.775173+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:13:49.775173+0000","last_undegraded":"2023-07-29T16:13:49.775173+0000","last_fullsized":"2023-07-29T16:13:49.775173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830173+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:13:49.830173+0000","last_peered":"2023-07-29T16:13:49.830173+0000","last_clean":"2023-07-29T16:13:49.830173+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:13:49.830173+0000","last_undegraded":"2023-07-29T16:13:49.830173+0000","last_fullsized":"2023-07-29T16:13:49.830173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775487+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:13:49.775487+0000","last_peered":"2023-07-29T16:13:49.775487+0000","last_clean":"2023-07-29T16:13:49.775487+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:13:49.775487+0000","last_undegraded":"2023-07-29T16:13:49.775487+0000","last_fullsized":"2023-07-29T16:13:49.775487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777770+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:13:49.777770+0000","last_peered":"2023-07-29T16:13:49.777770+0000","last_clean":"2023-07-29T16:13:49.777770+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:13:49.777770+0000","last_undegraded":"2023-07-29T16:13:49.777770+0000","last_fullsized":"2023-07-29T16:13:49.777770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774427+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:13:49.774427+0000","last_peered":"2023-07-29T16:13:49.774427+0000","last_clean":"2023-07-29T16:13:49.774427+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:13:49.774427+0000","last_undegraded":"2023-07-29T16:13:49.774427+0000","last_fullsized":"2023-07-29T16:13:49.774427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774317+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:13:49.774317+0000","last_peered":"2023-07-29T16:13:49.774317+0000","last_clean":"2023-07-29T16:13:49.774317+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:13:49.774317+0000","last_undegraded":"2023-07-29T16:13:49.774317+0000","last_fullsized":"2023-07-29T16:13:49.774317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776020+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:13:49.776020+0000","last_peered":"2023-07-29T16:13:49.776020+0000","last_clean":"2023-07-29T16:13:49.776020+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:13:49.776020+0000","last_undegraded":"2023-07-29T16:13:49.776020+0000","last_fullsized":"2023-07-29T16:13:49.776020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831492+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:13:49.831492+0000","last_peered":"2023-07-29T16:13:49.831492+0000","last_clean":"2023-07-29T16:13:49.831492+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:13:49.831492+0000","last_undegraded":"2023-07-29T16:13:49.831492+0000","last_fullsized":"2023-07-29T16:13:49.831492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778112+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:13:49.778112+0000","last_peered":"2023-07-29T16:13:49.778112+0000","last_clean":"2023-07-29T16:13:49.778112+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:13:49.778112+0000","last_undegraded":"2023-07-29T16:13:49.778112+0000","last_fullsized":"2023-07-29T16:13:49.778112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774750+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:13:49.774750+0000","last_peered":"2023-07-29T16:13:49.774750+0000","last_clean":"2023-07-29T16:13:49.774750+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:13:49.774750+0000","last_undegraded":"2023-07-29T16:13:49.774750+0000","last_fullsized":"2023-07-29T16:13:49.774750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830666+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:13:49.830666+0000","last_peered":"2023-07-29T16:13:49.830666+0000","last_clean":"2023-07-29T16:13:49.830666+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:13:49.830666+0000","last_undegraded":"2023-07-29T16:13:49.830666+0000","last_fullsized":"2023-07-29T16:13:49.830666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830961+0000","last_change":"2023-07-29T16:12:50.285320+0000","last_active":"2023-07-29T16:13:49.830961+0000","last_peered":"2023-07-29T16:13:49.830961+0000","last_clean":"2023-07-29T16:13:49.830961+0000","last_became_active":"2023-07-29T16:12:50.284666+0000","last_became_peered":"2023-07-29T16:12:50.284666+0000","last_unstale":"2023-07-29T16:13:49.830961+0000","last_undegraded":"2023-07-29T16:13:49.830961+0000","last_fullsized":"2023-07-29T16:13:49.830961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:19:09.408935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830254+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:13:49.830254+0000","last_peered":"2023-07-29T16:13:49.830254+0000","last_clean":"2023-07-29T16:13:49.830254+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:13:49.830254+0000","last_undegraded":"2023-07-29T16:13:49.830254+0000","last_fullsized":"2023-07-29T16:13:49.830254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830507+0000","last_change":"2023-07-29T16:12:50.335842+0000","last_active":"2023-07-29T16:13:49.830507+0000","last_peered":"2023-07-29T16:13:49.830507+0000","last_clean":"2023-07-29T16:13:49.830507+0000","last_became_active":"2023-07-29T16:12:50.335527+0000","last_became_peered":"2023-07-29T16:12:50.335527+0000","last_unstale":"2023-07-29T16:13:49.830507+0000","last_undegraded":"2023-07-29T16:13:49.830507+0000","last_fullsized":"2023-07-29T16:13:49.830507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:35:18.062454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830582+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:13:49.830582+0000","last_peered":"2023-07-29T16:13:49.830582+0000","last_clean":"2023-07-29T16:13:49.830582+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:13:49.830582+0000","last_undegraded":"2023-07-29T16:13:49.830582+0000","last_fullsized":"2023-07-29T16:13:49.830582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777983+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:13:49.777983+0000","last_peered":"2023-07-29T16:13:49.777983+0000","last_clean":"2023-07-29T16:13:49.777983+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:13:49.777983+0000","last_undegraded":"2023-07-29T16:13:49.777983+0000","last_fullsized":"2023-07-29T16:13:49.777983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66675,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":568,"num_read_kb":354,"num_write":164,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1684,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":999424,"data_stored":478732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.2050000000000001,"5min":4.2050000000000001,"15min":4.2050000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82099999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.87}]},{"osd":2,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.47299999999999998,"15min":0.46800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73399999999999999,"5min":0.755,"15min":0.755},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.61399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94099999999999995,"5min":0.996,"15min":0.996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91100000000000003,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67300000000000004,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.77000000000000002,"15min":0.73199999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.752,"15min":0.73699999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.9409999999999998,"5min":3.9409999999999998,"15min":3.9409999999999998},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.745,"15min":0.73999999999999999},"min":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.002,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.59299999999999997,"5min":0.495,"15min":0.495},"max":{"1min":0.88600000000000001,"5min":1.323,"15min":1.323},"last":0.92900000000000005}]},{"osd":8,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77300000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.82299999999999995,"15min":0.80400000000000005},"min":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":3.7570000000000001,"5min":3.7570000000000001,"15min":3.7570000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.83299999999999996,"15min":0.80600000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.8319999999999999,"5min":3.8319999999999999,"15min":3.8319999999999999},"last":0.85199999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.016,"5min":1.373,"15min":1.373},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95399999999999996,"5min":1.867,"15min":1.867},"last":0.88300000000000001}]}]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1679360,"data_stored":1126469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.433}]},{"osd":1,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72599999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.379}]},{"osd":2,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.624,"15min":0.624},"min":{"1min":0.50600000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80900000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.754,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82999999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82999999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.86099999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.95999999999999996,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.754},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.433,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98399999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.47599999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.87,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76600000000000001,"15min":0.77500000000000002},"min":{"1min":0.54800000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89600000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.58899999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.96899999999999997,"5min":1.319,"15min":1.319},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.82399999999999995,"15min":0.83999999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0349999999999999,"5min":1.494,"15min":1.494},"last":0.86499999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81899999999999995,"15min":0.82499999999999996},"min":{"1min":0.61499999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98299999999999998,"5min":1.419,"15min":1.419},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51200000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.995,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.80100000000000005}]},{"osd":11,"last update":"Sat Jul 29 16:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.79600000000000004},"min":{"1min":0.68799999999999994,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.89700000000000002}]}]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":1007616,"data_stored":477550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.53400000000000003,"15min":0.54900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.71699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71999999999999997,"5min":1.728,"15min":1.728},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0029999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81399999999999995,"5min":1.125,"15min":1.125},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.435,"15min":0.435},"max":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.80600000000000005,"15min":0.83399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92100000000000004,"5min":1.974,"15min":1.974},"last":0.63},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.80200000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.79200000000000004}]},{"osd":4,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.91500000000000004,"15min":0.94599999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95899999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.85499999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94699999999999995,"15min":0.97599999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0600000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.5}]},{"osd":6,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.92800000000000005,"15min":0.96299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95699999999999996,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89000000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.109,"5min":1.839,"15min":1.839},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.94299999999999995,"15min":0.97099999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0569999999999999,"5min":1.47,"15min":1.47},"last":0.82399999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.91700000000000004,"15min":0.94199999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0169999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.004,"15min":1.032},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1220000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.78400000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":0.96599999999999997},"min":{"1min":0.73199999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.038,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.63500000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.873}]}]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":1003520,"data_stored":477139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85499999999999998,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.83599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.63600000000000001,"5min":1.052,"15min":1.052},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.496,"15min":0.5},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72899999999999998,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.69099999999999995,"15min":0.65100000000000002},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":6.9240000000000004,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.624,"15min":0.63},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82299999999999995,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63},"min":{"1min":0.51100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79100000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58299999999999996,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.5}]},{"osd":6,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95499999999999996,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.622},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73499999999999999,"15min":0.753},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93300000000000005,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.73699999999999999,"15min":0.74099999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0369999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.65400000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.751,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.748}]},{"osd":10,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.73199999999999998,"15min":0.71999999999999997},"min":{"1min":0.66100000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72199999999999998,"15min":0.70899999999999996},"min":{"1min":0.61499999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.70199999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78200000000000003}]}]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28512,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716640,"statfs":{"total":95995035648,"available":95965839360,"internally_reserved":0,"allocated":1597440,"data_stored":1069644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.72199999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.44600000000000001,"15min":0.434},"min":{"1min":0.317,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.755,"15min":0.755},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.61899999999999999,"15min":0.60499999999999998},"min":{"1min":0.49399999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.51500000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.434,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52800000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81499999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.495}]},{"osd":5,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70799999999999996,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90300000000000002,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66000000000000003,"15min":0.64400000000000002},"min":{"1min":0.52900000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.495,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94099999999999995,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.72199999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72699999999999998,"15min":0.72099999999999997},"min":{"1min":0.622,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.755},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.72599999999999998,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93899999999999995,"5min":1.079,"15min":1.079},"last":0.76400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77400000000000002,"15min":0.77200000000000002},"min":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.74199999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:13:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79000000000000004,"15min":0.80100000000000005},"min":{"1min":0.53100000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98599999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.754,"15min":0.75},"min":{"1min":0.55900000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77700000000000002}]},{"osd":11,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78900000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":995328,"data_stored":476945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51800000000000002,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.317,"15min":0.317},"max":{"1min":1.6779999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.51300000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80900000000000005,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.62,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.872,"5min":1.379,"15min":1.379},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2010000000000001,"5min":1.486,"15min":1.486},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71699999999999997,"15min":0.69499999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97499999999999998,"5min":1.117,"15min":1.117},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69199999999999995,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88200000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64000000000000001,"15min":0.61099999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.6100000000000001,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.5589999999999999,"5min":2.25,"15min":2.25},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.54700000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0349999999999999,"5min":1.087,"15min":1.087},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.747,"15min":0.71899999999999997},"min":{"1min":0.46500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.77700000000000002}]},{"osd":5,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52400000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0589999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80200000000000005,"15min":0.78300000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.92800000000000005}]},{"osd":7,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.77900000000000003,"15min":0.75900000000000001},"min":{"1min":0.56799999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.79800000000000004,"15min":0.77300000000000002},"min":{"1min":0.68999999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.96499999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.89500000000000002,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":7.2009999999999996,"5min":7.2009999999999996,"15min":7.2009999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.87,"15min":0.86599999999999999},"min":{"1min":0.60499999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.0069999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.53800000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0780000000000001,"5min":2.165,"15min":2.165},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.91200000000000003,"15min":0.94199999999999995},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.244,"5min":2.331,"15min":2.331},"last":0.70799999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0089999999999999,"15min":0.95899999999999996},"min":{"1min":0.72599999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":8.1600000000000001,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.83399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":1060864,"data_stored":535619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56599999999999995,"15min":0.505},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80700000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.33000000000000002},"max":{"1min":0.73999999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.56899999999999995,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.57499999999999996,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.47899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58399999999999996,"15min":0.55600000000000005},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81200000000000006,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.501,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.436}]},{"osd":5,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.629,"15min":0.57999999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.66000000000000003,"15min":0.61099999999999999},"min":{"1min":0.51900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":1.02,"15min":1.02},"last":0.86799999999999999}]},{"osd":6,"last update":"Sat Jul 29 16:13:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67600000000000005},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90500000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.67000000000000004,"15min":0.66400000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.621}]},{"osd":7,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.62,"15min":0.58999999999999997},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":1.073,"15min":1.073},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.64200000000000002},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69399999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.754,"15min":0.75},"min":{"1min":0.57499999999999996,"5min":0.376,"15min":0.376},"max":{"1min":1.0920000000000001,"5min":1.101,"15min":1.101},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77000000000000002,"15min":0.76100000000000001},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.02,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.90600000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:13:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.85399999999999998},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.046,"5min":1.161,"15min":1.161},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84299999999999997,"15min":0.85699999999999998},"min":{"1min":0.622,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98099999999999998,"5min":1.175,"15min":1.175},"last":0.84199999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80700000000000005,"15min":0.81899999999999995},"min":{"1min":0.42799999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90300000000000002,"5min":1.129,"15min":1.129},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.81899999999999995},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.995,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.71599999999999997}]},{"osd":11,"last update":"Sat Jul 29 16:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":1015808,"data_stored":478210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56599999999999995,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59199999999999997,"15min":0.54500000000000004},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60999999999999999,"15min":0.58399999999999996},"min":{"1min":0.496,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90700000000000003,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.45100000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85999999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57099999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68200000000000005,"15min":0.66100000000000003},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94399999999999995,"5min":1.111,"15min":1.111},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Jul 29 16:13:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.43099999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.264,"5min":1.266,"15min":1.266},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.74099999999999999},"min":{"1min":0.435,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:13:52 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.67200000000000004}]},{"osd":8,"last update":"Sat Jul 29 16:13:30 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.84299999999999997,"15min":0.88300000000000001},"min":{"1min":0.55200000000000005,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.335,"15min":1.335},"last":0.621},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.89400000000000002},"min":{"1min":0.64100000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91100000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.68799999999999994}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.86499999999999999,"15min":0.88300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.079,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.751},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.94499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.014,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.71899999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:13:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.66100000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.91700000000000004,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.85699999999999998},"min":{"1min":0.64400000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.1160000000000001,"5min":1.212,"15min":1.212},"last":0.70599999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.82199999999999995},"min":{"1min":0.63500000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.745},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.749,"5min":0.626,"15min":0.626},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.78500000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1069056,"data_stored":538667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.375,"5min":0.308,"15min":0.308},"max":{"1min":3.4169999999999998,"5min":3.4169999999999998,"15min":3.4169999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60199999999999998,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78000000000000003,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58299999999999996,"15min":0.54600000000000004},"min":{"1min":0.40799999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78200000000000003,"5min":3.613,"15min":3.613},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.56899999999999995,"15min":0.54300000000000004},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.628}]},{"osd":3,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.60399999999999998,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.4550000000000001,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60599999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:13:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.45600000000000002,"15min":0.442},"min":{"1min":0.39100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66900000000000004,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.439},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.497,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73799999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:13:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.51800000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77900000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.753,"15min":0.66900000000000004},"min":{"1min":0.5,"5min":0.379,"15min":0.379},"max":{"1min":5.1139999999999999,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:13:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.624,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.85499999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.64800000000000002},"min":{"1min":0.51000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.79300000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.749}]},{"osd":9,"last update":"Sat Jul 29 16:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76400000000000001},"min":{"1min":0.56399999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90400000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.67500000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":3.5950000000000002,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.89600000000000002}]},{"osd":10,"last update":"Sat Jul 29 16:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.56899999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85899999999999999,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.69699999999999995,"15min":0.68200000000000005},"min":{"1min":0.59699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.82799999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.85199999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":1007616,"data_stored":476931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.46800000000000003,"15min":0.41099999999999998},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.56699999999999995,"15min":0.46800000000000003},"min":{"1min":0.39500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.766,"5min":8.766,"15min":8.766},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63200000000000001,"15min":0.58199999999999996},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.88500000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.55100000000000005,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70199999999999996,"15min":0.63200000000000001},"min":{"1min":0.61399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.69699999999999995,"15min":0.61699999999999999},"min":{"1min":0.56000000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1559999999999999,"5min":1.296,"15min":1.296},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64400000000000002,"15min":0.60699999999999998},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88400000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.53400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98099999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.57099999999999995,"5min":0.443,"15min":0.443},"max":{"1min":0.97799999999999998,"5min":1.343,"15min":1.343},"last":0.745},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72699999999999998,"15min":0.70099999999999996},"min":{"1min":0.53900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.84099999999999997,"5min":1.032,"15min":1.032},"last":0.88300000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:13:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.70099999999999996,"15min":0.68200000000000005},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83299999999999996,"5min":1.276,"15min":1.276},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70199999999999996,"15min":0.66400000000000003},"min":{"1min":0.48499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59699999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0800000000000001,"5min":1.323,"15min":1.323},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.83599999999999997,"15min":0.82799999999999996},"min":{"1min":0.60799999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96799999999999997,"5min":1.357,"15min":1.357},"last":0.999}]},{"osd":8,"last update":"Sat Jul 29 16:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":1.034},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.77900000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.90300000000000002,"15min":0.92300000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.091,"5min":1.476,"15min":1.476},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.90100000000000002,"15min":0.91800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0509999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.78700000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86799999999999999,"15min":0.87},"min":{"1min":0.74199999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.157,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.90200000000000002},"min":{"1min":0.71599999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.244,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.872}]},{"osd":11,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.93999999999999995}]}]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":1003520,"data_stored":478438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.56599999999999995},"min":{"1min":0.50700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60099999999999998,"15min":0.55600000000000005},"min":{"1min":0.45100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93300000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Jul 29 16:13:27 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.65900000000000003,"15min":0.60299999999999998},"min":{"1min":0.50600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.65000000000000002,"15min":0.60899999999999999},"min":{"1min":0.54600000000000004,"5min":0.374,"15min":0.374},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65200000000000002,"15min":0.628},"min":{"1min":0.46300000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.79100000000000004,"5min":1.018,"15min":1.018},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.58699999999999997},"min":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91700000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.749},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.628},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0129999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.60999999999999999}]},{"osd":8,"last update":"Sat Jul 29 16:13:32 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.76700000000000002,"15min":0.753},"min":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.78700000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79500000000000004,"15min":0.80000000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.04,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73999999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.115,"15min":1.115},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.76600000000000001,"15min":0.77000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.72899999999999998}]},{"osd":11,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.80200000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28528,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716624,"statfs":{"total":95995035648,"available":95965822976,"internally_reserved":0,"allocated":1605632,"data_stored":1068077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.56499999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.55100000000000005,"15min":0.51700000000000002},"min":{"1min":0.48999999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.746},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56599999999999995,"15min":0.53500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77200000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.42499999999999999,"15min":0.40699999999999997},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.435,"15min":0.41499999999999998},"min":{"1min":0.373,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60399999999999998,"15min":0.58399999999999996},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52300000000000002}]},{"osd":4,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.61399999999999999,"15min":0.58199999999999996},"min":{"1min":0.59999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.64900000000000002,"15min":0.63200000000000001},"min":{"1min":0.54700000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68200000000000005,"15min":0.65600000000000003},"min":{"1min":0.55900000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Jul 29 16:13:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.63400000000000001,"15min":0.63100000000000001},"min":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.46500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.81599999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.621,"15min":0.61099999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82699999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.51400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.622}]},{"osd":10,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.63200000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.73199999999999998,"15min":0.71899999999999997},"min":{"1min":0.65200000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.92000000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:14:01.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-29T16:14:02.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/226464359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:02.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:01 smithi156 ceph-mon[129245]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 104 B/s, 0 objects/s recovering 2023-07-29T16:14:02.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/262457666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:02.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/226464359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:02.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:01 smithi031 ceph-mon[95504]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 104 B/s, 0 objects/s recovering 2023-07-29T16:14:02.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/262457666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 -- 172.21.15.31:0/1618286860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70101530 msgr2=0x7f3b70101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 --2- 172.21.15.31:0/1618286860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70101530 0x7f3b70101980 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3b64009990 tx=0x7f3b64009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 -- 172.21.15.31:0/1618286860 shutdown_connections 2023-07-29T16:14:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 --2- 172.21.15.31:0/1618286860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 0x7f3b70108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 --2- 172.21.15.31:0/1618286860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b70100340 0x7f3b70100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 --2- 172.21.15.31:0/1618286860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70101530 0x7f3b70101980 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 -- 172.21.15.31:0/1618286860 >> 172.21.15.31:0/1618286860 conn(0x7f3b700fb920 msgr2=0x7f3b700fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 -- 172.21.15.31:0/1618286860 shutdown_connections 2023-07-29T16:14:02.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.236+0000 7f3b75bff700 1 -- 172.21.15.31:0/1618286860 wait complete. 2023-07-29T16:14:02.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.237+0000 7f3b75bff700 1 Processor -- start 2023-07-29T16:14:02.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 -- start start 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 0x7f3b70074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b70101530 0x7f3b70073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 0x7f3b70073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b70073e20 con 0x7f3b70100340 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b70073f90 con 0x7f3b70101530 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b75bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b701a00f0 con 0x7f3b70101ec0 2023-07-29T16:14:02.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b6f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 0x7f3b70074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b6f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 0x7f3b70074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43362/0 (socket says 172.21.15.31:43362) 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b6f7fe700 1 -- 172.21.15.31:0/3598783422 learned_addr learned my addr 172.21.15.31:0/3598783422 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b6effd700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b70101530 0x7f3b70073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.238+0000 7f3b6ffff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 0x7f3b70073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6f7fe700 1 -- 172.21.15.31:0/3598783422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 msgr2=0x7f3b70073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6f7fe700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 0x7f3b70073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6f7fe700 1 -- 172.21.15.31:0/3598783422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b70101530 msgr2=0x7f3b70073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6f7fe700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b70101530 0x7f3b70073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6f7fe700 1 -- 172.21.15.31:0/3598783422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b640095f0 con 0x7f3b70100340 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6ffff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 0x7f3b70073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6f7fe700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 0x7f3b70074b00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000e970 tx=0x7f3b6000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60009790 con 0x7f3b70100340 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b6000eec0 con 0x7f3b70100340 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b701a03d0 con 0x7f3b70100340 2023-07-29T16:14:02.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b60003cf0 con 0x7f3b70100340 2023-07-29T16:14:02.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.239+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b701a08e0 con 0x7f3b70100340 2023-07-29T16:14:02.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.240+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b701078b0 con 0x7f3b70100340 2023-07-29T16:14:02.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.242+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3b6000ccd0 con 0x7f3b70100340 2023-07-29T16:14:02.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.244+0000 7f3b6cff9700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b580a07e0 0x7f3b580a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.244+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b60014070 con 0x7f3b70100340 2023-07-29T16:14:02.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.244+0000 7f3b6effd700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b580a07e0 0x7f3b580a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.245+0000 7f3b6effd700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b580a07e0 0x7f3b580a2c90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b2e0 tx=0x7f3b64005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:02.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.248+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3b60095850 con 0x7f3b70100340 2023-07-29T16:14:02.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/226464359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:01 smithi039 ceph-mon[128858]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 6 op/s; 104 B/s, 0 objects/s recovering 2023-07-29T16:14:02.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/262457666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:02.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.456+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b70061190 con 0x7f3b580a07e0 2023-07-29T16:14:02.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.467+0000 7f3b6cff9700 1 -- 172.21.15.31:0/3598783422 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343664 (secure 0 0 0) 0x7f3b70061190 con 0x7f3b580a07e0 2023-07-29T16:14:02.469 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:02.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b580a07e0 msgr2=0x7f3b580a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b580a07e0 0x7f3b580a2c90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400b2e0 tx=0x7f3b64005fb0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 msgr2=0x7f3b70074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 0x7f3b70074b00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000e970 tx=0x7f3b6000ec80 comp rx=0 tx=0).stop 2023-07-29T16:14:02.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 shutdown_connections 2023-07-29T16:14:02.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b580a07e0 0x7f3b580a2c90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b70101ec0 0x7f3b70073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b70100340 0x7f3b70074b00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 --2- 172.21.15.31:0/3598783422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b70101530 0x7f3b70073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 >> 172.21.15.31:0/3598783422 conn(0x7f3b700fb920 msgr2=0x7f3b701049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:02.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 shutdown_connections 2023-07-29T16:14:02.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.472+0000 7f3b75bff700 1 -- 172.21.15.31:0/3598783422 wait complete. 2023-07-29T16:14:02.473 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:14:02.489 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2023-07-29T16:14:01.295437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657289,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":694,"num_read_kb":463,"num_write":347,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1182420,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337148,"kb_used_data":13716,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124604676,"statfs":{"total":1151940427776,"available":1151595188224,"internally_reserved":0,"allocated":14045184,"data_stored":7682421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5483,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":32,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.519441"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778124+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:13:49.778124+0000","last_peered":"2023-07-29T16:13:49.778124+0000","last_clean":"2023-07-29T16:13:49.778124+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:13:49.778124+0000","last_undegraded":"2023-07-29T16:13:49.778124+0000","last_fullsized":"2023-07-29T16:13:49.778124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776048+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:13:49.776048+0000","last_peered":"2023-07-29T16:13:49.776048+0000","last_clean":"2023-07-29T16:13:49.776048+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:13:49.776048+0000","last_undegraded":"2023-07-29T16:13:49.776048+0000","last_fullsized":"2023-07-29T16:13:49.776048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776110+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:13:49.776110+0000","last_peered":"2023-07-29T16:13:49.776110+0000","last_clean":"2023-07-29T16:13:49.776110+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:13:49.776110+0000","last_undegraded":"2023-07-29T16:13:49.776110+0000","last_fullsized":"2023-07-29T16:13:49.776110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830930+0000","last_change":"2023-07-29T16:12:50.286750+0000","last_active":"2023-07-29T16:13:49.830930+0000","last_peered":"2023-07-29T16:13:49.830930+0000","last_clean":"2023-07-29T16:13:49.830930+0000","last_became_active":"2023-07-29T16:12:50.286341+0000","last_became_peered":"2023-07-29T16:12:50.286341+0000","last_unstale":"2023-07-29T16:13:49.830930+0000","last_undegraded":"2023-07-29T16:13:49.830930+0000","last_fullsized":"2023-07-29T16:13:49.830930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.547598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829633+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:13:49.829633+0000","last_peered":"2023-07-29T16:13:49.829633+0000","last_clean":"2023-07-29T16:13:49.829633+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:13:49.829633+0000","last_undegraded":"2023-07-29T16:13:49.829633+0000","last_fullsized":"2023-07-29T16:13:49.829633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830834+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:13:49.830834+0000","last_peered":"2023-07-29T16:13:49.830834+0000","last_clean":"2023-07-29T16:13:49.830834+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:13:49.830834+0000","last_undegraded":"2023-07-29T16:13:49.830834+0000","last_fullsized":"2023-07-29T16:13:49.830834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774908+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:13:49.774908+0000","last_peered":"2023-07-29T16:13:49.774908+0000","last_clean":"2023-07-29T16:13:49.774908+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:13:49.774908+0000","last_undegraded":"2023-07-29T16:13:49.774908+0000","last_fullsized":"2023-07-29T16:13:49.774908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829985+0000","last_change":"2023-07-29T16:12:50.335631+0000","last_active":"2023-07-29T16:13:49.829985+0000","last_peered":"2023-07-29T16:13:49.829985+0000","last_clean":"2023-07-29T16:13:49.829985+0000","last_became_active":"2023-07-29T16:12:50.335213+0000","last_became_peered":"2023-07-29T16:12:50.335213+0000","last_unstale":"2023-07-29T16:13:49.829985+0000","last_undegraded":"2023-07-29T16:13:49.829985+0000","last_fullsized":"2023-07-29T16:13:49.829985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:04:33.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779120+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:13:49.779120+0000","last_peered":"2023-07-29T16:13:49.779120+0000","last_clean":"2023-07-29T16:13:49.779120+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:13:49.779120+0000","last_undegraded":"2023-07-29T16:13:49.779120+0000","last_fullsized":"2023-07-29T16:13:49.779120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830766+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:13:49.830766+0000","last_peered":"2023-07-29T16:13:49.830766+0000","last_clean":"2023-07-29T16:13:49.830766+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:13:49.830766+0000","last_undegraded":"2023-07-29T16:13:49.830766+0000","last_fullsized":"2023-07-29T16:13:49.830766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775219+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:13:49.775219+0000","last_peered":"2023-07-29T16:13:49.775219+0000","last_clean":"2023-07-29T16:13:49.775219+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:13:49.775219+0000","last_undegraded":"2023-07-29T16:13:49.775219+0000","last_fullsized":"2023-07-29T16:13:49.775219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830343+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:13:49.830343+0000","last_peered":"2023-07-29T16:13:49.830343+0000","last_clean":"2023-07-29T16:13:49.830343+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:13:49.830343+0000","last_undegraded":"2023-07-29T16:13:49.830343+0000","last_fullsized":"2023-07-29T16:13:49.830343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830414+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:13:49.830414+0000","last_peered":"2023-07-29T16:13:49.830414+0000","last_clean":"2023-07-29T16:13:49.830414+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:13:49.830414+0000","last_undegraded":"2023-07-29T16:13:49.830414+0000","last_fullsized":"2023-07-29T16:13:49.830414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775983+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:13:49.775983+0000","last_peered":"2023-07-29T16:13:49.775983+0000","last_clean":"2023-07-29T16:13:49.775983+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:13:49.775983+0000","last_undegraded":"2023-07-29T16:13:49.775983+0000","last_fullsized":"2023-07-29T16:13:49.775983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775874+0000","last_change":"2023-07-29T16:12:50.291287+0000","last_active":"2023-07-29T16:13:49.775874+0000","last_peered":"2023-07-29T16:13:49.775874+0000","last_clean":"2023-07-29T16:13:49.775874+0000","last_became_active":"2023-07-29T16:12:50.290878+0000","last_became_peered":"2023-07-29T16:12:50.290878+0000","last_unstale":"2023-07-29T16:13:49.775874+0000","last_undegraded":"2023-07-29T16:13:49.775874+0000","last_fullsized":"2023-07-29T16:13:49.775874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:59:20.470299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778600+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:13:49.778600+0000","last_peered":"2023-07-29T16:13:49.778600+0000","last_clean":"2023-07-29T16:13:49.778600+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:13:49.778600+0000","last_undegraded":"2023-07-29T16:13:49.778600+0000","last_fullsized":"2023-07-29T16:13:49.778600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778403+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:13:49.778403+0000","last_peered":"2023-07-29T16:13:49.778403+0000","last_clean":"2023-07-29T16:13:49.778403+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:13:49.778403+0000","last_undegraded":"2023-07-29T16:13:49.778403+0000","last_fullsized":"2023-07-29T16:13:49.778403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831266+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:13:49.831266+0000","last_peered":"2023-07-29T16:13:49.831266+0000","last_clean":"2023-07-29T16:13:49.831266+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:13:49.831266+0000","last_undegraded":"2023-07-29T16:13:49.831266+0000","last_fullsized":"2023-07-29T16:13:49.831266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774488+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:13:49.774488+0000","last_peered":"2023-07-29T16:13:49.774488+0000","last_clean":"2023-07-29T16:13:49.774488+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:13:49.774488+0000","last_undegraded":"2023-07-29T16:13:49.774488+0000","last_fullsized":"2023-07-29T16:13:49.774488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830125+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:13:49.830125+0000","last_peered":"2023-07-29T16:13:49.830125+0000","last_clean":"2023-07-29T16:13:49.830125+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:13:49.830125+0000","last_undegraded":"2023-07-29T16:13:49.830125+0000","last_fullsized":"2023-07-29T16:13:49.830125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830525+0000","last_change":"2023-07-29T16:12:50.286738+0000","last_active":"2023-07-29T16:13:49.830525+0000","last_peered":"2023-07-29T16:13:49.830525+0000","last_clean":"2023-07-29T16:13:49.830525+0000","last_became_active":"2023-07-29T16:12:50.286379+0000","last_became_peered":"2023-07-29T16:12:50.286379+0000","last_unstale":"2023-07-29T16:13:49.830525+0000","last_undegraded":"2023-07-29T16:13:49.830525+0000","last_fullsized":"2023-07-29T16:13:49.830525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:50:03.338896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830606+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:13:49.830606+0000","last_peered":"2023-07-29T16:13:49.830606+0000","last_clean":"2023-07-29T16:13:49.830606+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:13:49.830606+0000","last_undegraded":"2023-07-29T16:13:49.830606+0000","last_fullsized":"2023-07-29T16:13:49.830606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776300+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:13:49.776300+0000","last_peered":"2023-07-29T16:13:49.776300+0000","last_clean":"2023-07-29T16:13:49.776300+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:13:49.776300+0000","last_undegraded":"2023-07-29T16:13:49.776300+0000","last_fullsized":"2023-07-29T16:13:49.776300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830130+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:13:49.830130+0000","last_peered":"2023-07-29T16:13:49.830130+0000","last_clean":"2023-07-29T16:13:49.830130+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:13:49.830130+0000","last_undegraded":"2023-07-29T16:13:49.830130+0000","last_fullsized":"2023-07-29T16:13:49.830130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778941+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:13:49.778941+0000","last_peered":"2023-07-29T16:13:49.778941+0000","last_clean":"2023-07-29T16:13:49.778941+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:13:49.778941+0000","last_undegraded":"2023-07-29T16:13:49.778941+0000","last_fullsized":"2023-07-29T16:13:49.778941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777906+0000","last_change":"2023-07-29T16:12:50.288144+0000","last_active":"2023-07-29T16:13:49.777906+0000","last_peered":"2023-07-29T16:13:49.777906+0000","last_clean":"2023-07-29T16:13:49.777906+0000","last_became_active":"2023-07-29T16:12:50.283877+0000","last_became_peered":"2023-07-29T16:12:50.283877+0000","last_unstale":"2023-07-29T16:13:49.777906+0000","last_undegraded":"2023-07-29T16:13:49.777906+0000","last_fullsized":"2023-07-29T16:13:49.777906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T04:00:21.950054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778880+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:13:49.778880+0000","last_peered":"2023-07-29T16:13:49.778880+0000","last_clean":"2023-07-29T16:13:49.778880+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:13:49.778880+0000","last_undegraded":"2023-07-29T16:13:49.778880+0000","last_fullsized":"2023-07-29T16:13:49.778880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:57.798989+0000","last_change":"2023-07-29T16:13:50.790518+0000","last_active":"2023-07-29T16:13:57.798989+0000","last_peered":"2023-07-29T16:13:57.798989+0000","last_clean":"2023-07-29T16:13:57.798989+0000","last_became_active":"2023-07-29T16:13:50.783618+0000","last_became_peered":"2023-07-29T16:13:50.783618+0000","last_unstale":"2023-07-29T16:13:57.798989+0000","last_undegraded":"2023-07-29T16:13:57.798989+0000","last_fullsized":"2023-07-29T16:13:57.798989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:39:28.814617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1418,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829658+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:13:49.829658+0000","last_peered":"2023-07-29T16:13:49.829658+0000","last_clean":"2023-07-29T16:13:49.829658+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:13:49.829658+0000","last_undegraded":"2023-07-29T16:13:49.829658+0000","last_fullsized":"2023-07-29T16:13:49.829658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:50.877482+0000","last_change":"2023-07-29T16:13:50.877482+0000","last_active":"2023-07-29T16:13:50.877482+0000","last_peered":"2023-07-29T16:13:50.877482+0000","last_clean":"2023-07-29T16:13:50.877482+0000","last_became_active":"2023-07-29T16:13:50.874496+0000","last_became_peered":"2023-07-29T16:13:50.874496+0000","last_unstale":"2023-07-29T16:13:50.877482+0000","last_undegraded":"2023-07-29T16:13:50.877482+0000","last_fullsized":"2023-07-29T16:13:50.877482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:47:18.956678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779443+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:13:49.779443+0000","last_peered":"2023-07-29T16:13:49.779443+0000","last_clean":"2023-07-29T16:13:49.779443+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:13:49.779443+0000","last_undegraded":"2023-07-29T16:13:49.779443+0000","last_fullsized":"2023-07-29T16:13:49.779443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831234+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:13:49.831234+0000","last_peered":"2023-07-29T16:13:49.831234+0000","last_clean":"2023-07-29T16:13:49.831234+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:13:49.831234+0000","last_undegraded":"2023-07-29T16:13:49.831234+0000","last_fullsized":"2023-07-29T16:13:49.831234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830062+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:13:49.830062+0000","last_peered":"2023-07-29T16:13:49.830062+0000","last_clean":"2023-07-29T16:13:49.830062+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:13:49.830062+0000","last_undegraded":"2023-07-29T16:13:49.830062+0000","last_fullsized":"2023-07-29T16:13:49.830062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778416+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:13:49.778416+0000","last_peered":"2023-07-29T16:13:49.778416+0000","last_clean":"2023-07-29T16:13:49.778416+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:13:49.778416+0000","last_undegraded":"2023-07-29T16:13:49.778416+0000","last_fullsized":"2023-07-29T16:13:49.778416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830370+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:13:49.830370+0000","last_peered":"2023-07-29T16:13:49.830370+0000","last_clean":"2023-07-29T16:13:49.830370+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:13:49.830370+0000","last_undegraded":"2023-07-29T16:13:49.830370+0000","last_fullsized":"2023-07-29T16:13:49.830370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":110,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:55.795897+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:13:55.795897+0000","last_peered":"2023-07-29T16:13:55.795897+0000","last_clean":"2023-07-29T16:13:55.795897+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:13:55.795897+0000","last_undegraded":"2023-07-29T16:13:55.795897+0000","last_fullsized":"2023-07-29T16:13:55.795897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779064+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:13:49.779064+0000","last_peered":"2023-07-29T16:13:49.779064+0000","last_clean":"2023-07-29T16:13:49.779064+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:13:49.779064+0000","last_undegraded":"2023-07-29T16:13:49.779064+0000","last_fullsized":"2023-07-29T16:13:49.779064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775359+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:13:49.775359+0000","last_peered":"2023-07-29T16:13:49.775359+0000","last_clean":"2023-07-29T16:13:49.775359+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:13:49.775359+0000","last_undegraded":"2023-07-29T16:13:49.775359+0000","last_fullsized":"2023-07-29T16:13:49.775359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776218+0000","last_change":"2023-07-29T16:12:50.336088+0000","last_active":"2023-07-29T16:13:49.776218+0000","last_peered":"2023-07-29T16:13:49.776218+0000","last_clean":"2023-07-29T16:13:49.776218+0000","last_became_active":"2023-07-29T16:12:50.335715+0000","last_became_peered":"2023-07-29T16:12:50.335715+0000","last_unstale":"2023-07-29T16:13:49.776218+0000","last_undegraded":"2023-07-29T16:13:49.776218+0000","last_fullsized":"2023-07-29T16:13:49.776218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:53:33.332020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778964+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:13:49.778964+0000","last_peered":"2023-07-29T16:13:49.778964+0000","last_clean":"2023-07-29T16:13:49.778964+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:13:49.778964+0000","last_undegraded":"2023-07-29T16:13:49.778964+0000","last_fullsized":"2023-07-29T16:13:49.778964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779024+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:13:49.779024+0000","last_peered":"2023-07-29T16:13:49.779024+0000","last_clean":"2023-07-29T16:13:49.779024+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:13:49.779024+0000","last_undegraded":"2023-07-29T16:13:49.779024+0000","last_fullsized":"2023-07-29T16:13:49.779024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":234,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:56.939908+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:13:56.939908+0000","last_peered":"2023-07-29T16:13:56.939908+0000","last_clean":"2023-07-29T16:13:56.939908+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:13:56.939908+0000","last_undegraded":"2023-07-29T16:13:56.939908+0000","last_fullsized":"2023-07-29T16:13:56.939908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61789,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":188,"num_read_kb":149,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829885+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:13:49.829885+0000","last_peered":"2023-07-29T16:13:49.829885+0000","last_clean":"2023-07-29T16:13:49.829885+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:13:49.829885+0000","last_undegraded":"2023-07-29T16:13:49.829885+0000","last_fullsized":"2023-07-29T16:13:49.829885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774452+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:13:49.774452+0000","last_peered":"2023-07-29T16:13:49.774452+0000","last_clean":"2023-07-29T16:13:49.774452+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:13:49.774452+0000","last_undegraded":"2023-07-29T16:13:49.774452+0000","last_fullsized":"2023-07-29T16:13:49.774452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830632+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:13:49.830632+0000","last_peered":"2023-07-29T16:13:49.830632+0000","last_clean":"2023-07-29T16:13:49.830632+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:13:49.830632+0000","last_undegraded":"2023-07-29T16:13:49.830632+0000","last_fullsized":"2023-07-29T16:13:49.830632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779295+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:13:49.779295+0000","last_peered":"2023-07-29T16:13:49.779295+0000","last_clean":"2023-07-29T16:13:49.779295+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:13:49.779295+0000","last_undegraded":"2023-07-29T16:13:49.779295+0000","last_fullsized":"2023-07-29T16:13:49.779295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775110+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:13:49.775110+0000","last_peered":"2023-07-29T16:13:49.775110+0000","last_clean":"2023-07-29T16:13:49.775110+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:13:49.775110+0000","last_undegraded":"2023-07-29T16:13:49.775110+0000","last_fullsized":"2023-07-29T16:13:49.775110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":117,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:56.883507+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:13:56.883507+0000","last_peered":"2023-07-29T16:13:56.883507+0000","last_clean":"2023-07-29T16:13:56.883507+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:13:56.883507+0000","last_undegraded":"2023-07-29T16:13:56.883507+0000","last_fullsized":"2023-07-29T16:13:56.883507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1068,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775014+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:13:49.775014+0000","last_peered":"2023-07-29T16:13:49.775014+0000","last_clean":"2023-07-29T16:13:49.775014+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:13:49.775014+0000","last_undegraded":"2023-07-29T16:13:49.775014+0000","last_fullsized":"2023-07-29T16:13:49.775014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829533+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:13:49.829533+0000","last_peered":"2023-07-29T16:13:49.829533+0000","last_clean":"2023-07-29T16:13:49.829533+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:13:49.829533+0000","last_undegraded":"2023-07-29T16:13:49.829533+0000","last_fullsized":"2023-07-29T16:13:49.829533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778567+0000","last_change":"2023-07-29T16:12:50.286421+0000","last_active":"2023-07-29T16:13:49.778567+0000","last_peered":"2023-07-29T16:13:49.778567+0000","last_clean":"2023-07-29T16:13:49.778567+0000","last_became_active":"2023-07-29T16:12:50.286075+0000","last_became_peered":"2023-07-29T16:12:50.286075+0000","last_unstale":"2023-07-29T16:13:49.778567+0000","last_undegraded":"2023-07-29T16:13:49.778567+0000","last_fullsized":"2023-07-29T16:13:49.778567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:24:16.006080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829886+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:13:49.829886+0000","last_peered":"2023-07-29T16:13:49.829886+0000","last_clean":"2023-07-29T16:13:49.829886+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:13:49.829886+0000","last_undegraded":"2023-07-29T16:13:49.829886+0000","last_fullsized":"2023-07-29T16:13:49.829886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777964+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:13:49.777964+0000","last_peered":"2023-07-29T16:13:49.777964+0000","last_clean":"2023-07-29T16:13:49.777964+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:13:49.777964+0000","last_undegraded":"2023-07-29T16:13:49.777964+0000","last_fullsized":"2023-07-29T16:13:49.777964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830840+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:13:49.830840+0000","last_peered":"2023-07-29T16:13:49.830840+0000","last_clean":"2023-07-29T16:13:49.830840+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:13:49.830840+0000","last_undegraded":"2023-07-29T16:13:49.830840+0000","last_fullsized":"2023-07-29T16:13:49.830840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830261+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:13:49.830261+0000","last_peered":"2023-07-29T16:13:49.830261+0000","last_clean":"2023-07-29T16:13:49.830261+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:13:49.830261+0000","last_undegraded":"2023-07-29T16:13:49.830261+0000","last_fullsized":"2023-07-29T16:13:49.830261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775445+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:13:49.775445+0000","last_peered":"2023-07-29T16:13:49.775445+0000","last_clean":"2023-07-29T16:13:49.775445+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:13:49.775445+0000","last_undegraded":"2023-07-29T16:13:49.775445+0000","last_fullsized":"2023-07-29T16:13:49.775445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830695+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:13:49.830695+0000","last_peered":"2023-07-29T16:13:49.830695+0000","last_clean":"2023-07-29T16:13:49.830695+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:13:49.830695+0000","last_undegraded":"2023-07-29T16:13:49.830695+0000","last_fullsized":"2023-07-29T16:13:49.830695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778758+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:13:49.778758+0000","last_peered":"2023-07-29T16:13:49.778758+0000","last_clean":"2023-07-29T16:13:49.778758+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:13:49.778758+0000","last_undegraded":"2023-07-29T16:13:49.778758+0000","last_fullsized":"2023-07-29T16:13:49.778758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":114,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:57.884345+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:13:57.884345+0000","last_peered":"2023-07-29T16:13:57.884345+0000","last_clean":"2023-07-29T16:13:57.884345+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:13:57.884345+0000","last_undegraded":"2023-07-29T16:13:57.884345+0000","last_fullsized":"2023-07-29T16:13:57.884345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830891+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:13:49.830891+0000","last_peered":"2023-07-29T16:13:49.830891+0000","last_clean":"2023-07-29T16:13:49.830891+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:13:49.830891+0000","last_undegraded":"2023-07-29T16:13:49.830891+0000","last_fullsized":"2023-07-29T16:13:49.830891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830292+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:13:49.830292+0000","last_peered":"2023-07-29T16:13:49.830292+0000","last_clean":"2023-07-29T16:13:49.830292+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:13:49.830292+0000","last_undegraded":"2023-07-29T16:13:49.830292+0000","last_fullsized":"2023-07-29T16:13:49.830292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":332,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774940+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:13:49.774940+0000","last_peered":"2023-07-29T16:13:49.774940+0000","last_clean":"2023-07-29T16:13:49.774940+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:13:49.774940+0000","last_undegraded":"2023-07-29T16:13:49.774940+0000","last_fullsized":"2023-07-29T16:13:49.774940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831699+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:13:49.831699+0000","last_peered":"2023-07-29T16:13:49.831699+0000","last_clean":"2023-07-29T16:13:49.831699+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:13:49.831699+0000","last_undegraded":"2023-07-29T16:13:49.831699+0000","last_fullsized":"2023-07-29T16:13:49.831699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774897+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:13:49.774897+0000","last_peered":"2023-07-29T16:13:49.774897+0000","last_clean":"2023-07-29T16:13:49.774897+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:13:49.774897+0000","last_undegraded":"2023-07-29T16:13:49.774897+0000","last_fullsized":"2023-07-29T16:13:49.774897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829981+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:13:49.829981+0000","last_peered":"2023-07-29T16:13:49.829981+0000","last_clean":"2023-07-29T16:13:49.829981+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:13:49.829981+0000","last_undegraded":"2023-07-29T16:13:49.829981+0000","last_fullsized":"2023-07-29T16:13:49.829981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831508+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:13:49.831508+0000","last_peered":"2023-07-29T16:13:49.831508+0000","last_clean":"2023-07-29T16:13:49.831508+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:13:49.831508+0000","last_undegraded":"2023-07-29T16:13:49.831508+0000","last_fullsized":"2023-07-29T16:13:49.831508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:52.562037+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:13:52.562037+0000","last_peered":"2023-07-29T16:13:52.562037+0000","last_clean":"2023-07-29T16:13:52.562037+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:13:52.562037+0000","last_undegraded":"2023-07-29T16:13:52.562037+0000","last_fullsized":"2023-07-29T16:13:52.562037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830712+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:13:49.830712+0000","last_peered":"2023-07-29T16:13:49.830712+0000","last_clean":"2023-07-29T16:13:49.830712+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:13:49.830712+0000","last_undegraded":"2023-07-29T16:13:49.830712+0000","last_fullsized":"2023-07-29T16:13:49.830712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830267+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:13:49.830267+0000","last_peered":"2023-07-29T16:13:49.830267+0000","last_clean":"2023-07-29T16:13:49.830267+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:13:49.830267+0000","last_undegraded":"2023-07-29T16:13:49.830267+0000","last_fullsized":"2023-07-29T16:13:49.830267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776129+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:13:49.776129+0000","last_peered":"2023-07-29T16:13:49.776129+0000","last_clean":"2023-07-29T16:13:49.776129+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:13:49.776129+0000","last_undegraded":"2023-07-29T16:13:49.776129+0000","last_fullsized":"2023-07-29T16:13:49.776129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778725+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:13:49.778725+0000","last_peered":"2023-07-29T16:13:49.778725+0000","last_clean":"2023-07-29T16:13:49.778725+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:13:49.778725+0000","last_undegraded":"2023-07-29T16:13:49.778725+0000","last_fullsized":"2023-07-29T16:13:49.778725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829527+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:13:49.829527+0000","last_peered":"2023-07-29T16:13:49.829527+0000","last_clean":"2023-07-29T16:13:49.829527+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:13:49.829527+0000","last_undegraded":"2023-07-29T16:13:49.829527+0000","last_fullsized":"2023-07-29T16:13:49.829527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830174+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:13:49.830174+0000","last_peered":"2023-07-29T16:13:49.830174+0000","last_clean":"2023-07-29T16:13:49.830174+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:13:49.830174+0000","last_undegraded":"2023-07-29T16:13:49.830174+0000","last_fullsized":"2023-07-29T16:13:49.830174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776966+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:13:49.776966+0000","last_peered":"2023-07-29T16:13:49.776966+0000","last_clean":"2023-07-29T16:13:49.776966+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:13:49.776966+0000","last_undegraded":"2023-07-29T16:13:49.776966+0000","last_fullsized":"2023-07-29T16:13:49.776966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:50.781469+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:13:50.781469+0000","last_peered":"2023-07-29T16:13:50.781469+0000","last_clean":"2023-07-29T16:13:50.781469+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:13:50.781469+0000","last_undegraded":"2023-07-29T16:13:50.781469+0000","last_fullsized":"2023-07-29T16:13:50.781469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830383+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:13:49.830383+0000","last_peered":"2023-07-29T16:13:49.830383+0000","last_clean":"2023-07-29T16:13:49.830383+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:13:49.830383+0000","last_undegraded":"2023-07-29T16:13:49.830383+0000","last_fullsized":"2023-07-29T16:13:49.830383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829662+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:13:49.829662+0000","last_peered":"2023-07-29T16:13:49.829662+0000","last_clean":"2023-07-29T16:13:49.829662+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:13:49.829662+0000","last_undegraded":"2023-07-29T16:13:49.829662+0000","last_fullsized":"2023-07-29T16:13:49.829662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775210+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:13:49.775210+0000","last_peered":"2023-07-29T16:13:49.775210+0000","last_clean":"2023-07-29T16:13:49.775210+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:13:49.775210+0000","last_undegraded":"2023-07-29T16:13:49.775210+0000","last_fullsized":"2023-07-29T16:13:49.775210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830449+0000","last_change":"2023-07-29T16:12:50.296260+0000","last_active":"2023-07-29T16:13:49.830449+0000","last_peered":"2023-07-29T16:13:49.830449+0000","last_clean":"2023-07-29T16:13:49.830449+0000","last_became_active":"2023-07-29T16:12:50.290108+0000","last_became_peered":"2023-07-29T16:12:50.290108+0000","last_unstale":"2023-07-29T16:13:49.830449+0000","last_undegraded":"2023-07-29T16:13:49.830449+0000","last_fullsized":"2023-07-29T16:13:49.830449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:52:17.142674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775483+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:13:49.775483+0000","last_peered":"2023-07-29T16:13:49.775483+0000","last_clean":"2023-07-29T16:13:49.775483+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:13:49.775483+0000","last_undegraded":"2023-07-29T16:13:49.775483+0000","last_fullsized":"2023-07-29T16:13:49.775483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778871+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:13:49.778871+0000","last_peered":"2023-07-29T16:13:49.778871+0000","last_clean":"2023-07-29T16:13:49.778871+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:13:49.778871+0000","last_undegraded":"2023-07-29T16:13:49.778871+0000","last_fullsized":"2023-07-29T16:13:49.778871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775173+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:13:49.775173+0000","last_peered":"2023-07-29T16:13:49.775173+0000","last_clean":"2023-07-29T16:13:49.775173+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:13:49.775173+0000","last_undegraded":"2023-07-29T16:13:49.775173+0000","last_fullsized":"2023-07-29T16:13:49.775173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830173+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:13:49.830173+0000","last_peered":"2023-07-29T16:13:49.830173+0000","last_clean":"2023-07-29T16:13:49.830173+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:13:49.830173+0000","last_undegraded":"2023-07-29T16:13:49.830173+0000","last_fullsized":"2023-07-29T16:13:49.830173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775487+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:13:49.775487+0000","last_peered":"2023-07-29T16:13:49.775487+0000","last_clean":"2023-07-29T16:13:49.775487+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:13:49.775487+0000","last_undegraded":"2023-07-29T16:13:49.775487+0000","last_fullsized":"2023-07-29T16:13:49.775487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777770+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:13:49.777770+0000","last_peered":"2023-07-29T16:13:49.777770+0000","last_clean":"2023-07-29T16:13:49.777770+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:13:49.777770+0000","last_undegraded":"2023-07-29T16:13:49.777770+0000","last_fullsized":"2023-07-29T16:13:49.777770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774427+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:13:49.774427+0000","last_peered":"2023-07-29T16:13:49.774427+0000","last_clean":"2023-07-29T16:13:49.774427+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:13:49.774427+0000","last_undegraded":"2023-07-29T16:13:49.774427+0000","last_fullsized":"2023-07-29T16:13:49.774427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774317+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:13:49.774317+0000","last_peered":"2023-07-29T16:13:49.774317+0000","last_clean":"2023-07-29T16:13:49.774317+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:13:49.774317+0000","last_undegraded":"2023-07-29T16:13:49.774317+0000","last_fullsized":"2023-07-29T16:13:49.774317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776020+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:13:49.776020+0000","last_peered":"2023-07-29T16:13:49.776020+0000","last_clean":"2023-07-29T16:13:49.776020+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:13:49.776020+0000","last_undegraded":"2023-07-29T16:13:49.776020+0000","last_fullsized":"2023-07-29T16:13:49.776020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831492+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:13:49.831492+0000","last_peered":"2023-07-29T16:13:49.831492+0000","last_clean":"2023-07-29T16:13:49.831492+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:13:49.831492+0000","last_undegraded":"2023-07-29T16:13:49.831492+0000","last_fullsized":"2023-07-29T16:13:49.831492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778112+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:13:49.778112+0000","last_peered":"2023-07-29T16:13:49.778112+0000","last_clean":"2023-07-29T16:13:49.778112+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:13:49.778112+0000","last_undegraded":"2023-07-29T16:13:49.778112+0000","last_fullsized":"2023-07-29T16:13:49.778112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774750+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:13:49.774750+0000","last_peered":"2023-07-29T16:13:49.774750+0000","last_clean":"2023-07-29T16:13:49.774750+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:13:49.774750+0000","last_undegraded":"2023-07-29T16:13:49.774750+0000","last_fullsized":"2023-07-29T16:13:49.774750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830666+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:13:49.830666+0000","last_peered":"2023-07-29T16:13:49.830666+0000","last_clean":"2023-07-29T16:13:49.830666+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:13:49.830666+0000","last_undegraded":"2023-07-29T16:13:49.830666+0000","last_fullsized":"2023-07-29T16:13:49.830666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830961+0000","last_change":"2023-07-29T16:12:50.285320+0000","last_active":"2023-07-29T16:13:49.830961+0000","last_peered":"2023-07-29T16:13:49.830961+0000","last_clean":"2023-07-29T16:13:49.830961+0000","last_became_active":"2023-07-29T16:12:50.284666+0000","last_became_peered":"2023-07-29T16:12:50.284666+0000","last_unstale":"2023-07-29T16:13:49.830961+0000","last_undegraded":"2023-07-29T16:13:49.830961+0000","last_fullsized":"2023-07-29T16:13:49.830961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:19:09.408935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830254+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:13:49.830254+0000","last_peered":"2023-07-29T16:13:49.830254+0000","last_clean":"2023-07-29T16:13:49.830254+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:13:49.830254+0000","last_undegraded":"2023-07-29T16:13:49.830254+0000","last_fullsized":"2023-07-29T16:13:49.830254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830507+0000","last_change":"2023-07-29T16:12:50.335842+0000","last_active":"2023-07-29T16:13:49.830507+0000","last_peered":"2023-07-29T16:13:49.830507+0000","last_clean":"2023-07-29T16:13:49.830507+0000","last_became_active":"2023-07-29T16:12:50.335527+0000","last_became_peered":"2023-07-29T16:12:50.335527+0000","last_unstale":"2023-07-29T16:13:49.830507+0000","last_undegraded":"2023-07-29T16:13:49.830507+0000","last_fullsized":"2023-07-29T16:13:49.830507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:35:18.062454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830582+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:13:49.830582+0000","last_peered":"2023-07-29T16:13:49.830582+0000","last_clean":"2023-07-29T16:13:49.830582+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:13:49.830582+0000","last_undegraded":"2023-07-29T16:13:49.830582+0000","last_fullsized":"2023-07-29T16:13:49.830582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777983+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:13:49.777983+0000","last_peered":"2023-07-29T16:13:49.777983+0000","last_clean":"2023-07-29T16:13:49.777983+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:13:49.777983+0000","last_undegraded":"2023-07-29T16:13:49.777983+0000","last_fullsized":"2023-07-29T16:13:49.777983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66675,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":568,"num_read_kb":354,"num_write":164,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1684,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":999424,"data_stored":478732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.2050000000000001,"5min":4.2050000000000001,"15min":4.2050000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82099999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.87}]},{"osd":2,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.47299999999999998,"15min":0.46800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73399999999999999,"5min":0.755,"15min":0.755},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.61399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94099999999999995,"5min":0.996,"15min":0.996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91100000000000003,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67300000000000004,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.77000000000000002,"15min":0.73199999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.752,"15min":0.73699999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.9409999999999998,"5min":3.9409999999999998,"15min":3.9409999999999998},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.745,"15min":0.73999999999999999},"min":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.002,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.59299999999999997,"5min":0.495,"15min":0.495},"max":{"1min":0.88600000000000001,"5min":1.323,"15min":1.323},"last":0.92900000000000005}]},{"osd":8,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77300000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.82299999999999995,"15min":0.80400000000000005},"min":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":3.7570000000000001,"5min":3.7570000000000001,"15min":3.7570000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.83299999999999996,"15min":0.80600000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.8319999999999999,"5min":3.8319999999999999,"15min":3.8319999999999999},"last":0.85199999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.016,"5min":1.373,"15min":1.373},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95399999999999996,"5min":1.867,"15min":1.867},"last":0.88300000000000001}]}]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1679360,"data_stored":1126469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.433}]},{"osd":1,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72599999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.379}]},{"osd":2,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.624,"15min":0.624},"min":{"1min":0.50600000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80900000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.754,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82999999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82999999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.86099999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.95999999999999996,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.754},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.433,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98399999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.47599999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.87,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76600000000000001,"15min":0.77500000000000002},"min":{"1min":0.54800000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89600000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.58899999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.96899999999999997,"5min":1.319,"15min":1.319},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.82399999999999995,"15min":0.83999999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0349999999999999,"5min":1.494,"15min":1.494},"last":0.86499999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81899999999999995,"15min":0.82499999999999996},"min":{"1min":0.61499999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98299999999999998,"5min":1.419,"15min":1.419},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51200000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.995,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.80100000000000005}]},{"osd":11,"last update":"Sat Jul 29 16:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.79600000000000004},"min":{"1min":0.68799999999999994,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.89700000000000002}]}]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":1007616,"data_stored":477550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.53400000000000003,"15min":0.54900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.71699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71999999999999997,"5min":1.728,"15min":1.728},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0029999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81399999999999995,"5min":1.125,"15min":1.125},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.435,"15min":0.435},"max":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.80600000000000005,"15min":0.83399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92100000000000004,"5min":1.974,"15min":1.974},"last":0.63},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.80200000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.79200000000000004}]},{"osd":4,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.91500000000000004,"15min":0.94599999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95899999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.85499999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94699999999999995,"15min":0.97599999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0600000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.5}]},{"osd":6,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.92800000000000005,"15min":0.96299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95699999999999996,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89000000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.109,"5min":1.839,"15min":1.839},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.94299999999999995,"15min":0.97099999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0569999999999999,"5min":1.47,"15min":1.47},"last":0.82399999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.91700000000000004,"15min":0.94199999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0169999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.004,"15min":1.032},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1220000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.78400000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":0.96599999999999997},"min":{"1min":0.73199999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.038,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.63500000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.873}]}]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":1003520,"data_stored":477139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85499999999999998,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.83599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.63600000000000001,"5min":1.052,"15min":1.052},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.496,"15min":0.5},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72899999999999998,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.69099999999999995,"15min":0.65100000000000002},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":6.9240000000000004,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.624,"15min":0.63},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82299999999999995,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63},"min":{"1min":0.51100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79100000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58299999999999996,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.5}]},{"osd":6,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95499999999999996,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.622},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73499999999999999,"15min":0.753},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93300000000000005,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.73699999999999999,"15min":0.74099999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0369999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.65400000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.751,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.748}]},{"osd":10,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.73199999999999998,"15min":0.71999999999999997},"min":{"1min":0.66100000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72199999999999998,"15min":0.70899999999999996},"min":{"1min":0.61499999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.70199999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78200000000000003}]}]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28512,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716640,"statfs":{"total":95995035648,"available":95965839360,"internally_reserved":0,"allocated":1597440,"data_stored":1069644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.72199999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.44600000000000001,"15min":0.434},"min":{"1min":0.317,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.755,"15min":0.755},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.61899999999999999,"15min":0.60499999999999998},"min":{"1min":0.49399999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.51500000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.434,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52800000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81499999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.495}]},{"osd":5,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70799999999999996,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90300000000000002,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66000000000000003,"15min":0.64400000000000002},"min":{"1min":0.52900000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.495,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94099999999999995,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.72199999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72699999999999998,"15min":0.72099999999999997},"min":{"1min":0.622,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.755},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.72599999999999998,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93899999999999995,"5min":1.079,"15min":1.079},"last":0.76400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77400000000000002,"15min":0.77200000000000002},"min":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.74199999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:13:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79000000000000004,"15min":0.80100000000000005},"min":{"1min":0.53100000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98599999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.754,"15min":0.75},"min":{"1min":0.55900000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77700000000000002}]},{"osd":11,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78900000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":995328,"data_stored":476945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51800000000000002,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.317,"15min":0.317},"max":{"1min":1.6779999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.51300000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80900000000000005,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.62,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.872,"5min":1.379,"15min":1.379},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2010000000000001,"5min":1.486,"15min":1.486},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71699999999999997,"15min":0.69499999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97499999999999998,"5min":1.117,"15min":1.117},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69199999999999995,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88200000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64000000000000001,"15min":0.61099999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.6100000000000001,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.5589999999999999,"5min":2.25,"15min":2.25},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.54700000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0349999999999999,"5min":1.087,"15min":1.087},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.747,"15min":0.71899999999999997},"min":{"1min":0.46500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.77700000000000002}]},{"osd":5,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52400000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0589999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80200000000000005,"15min":0.78300000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.92800000000000005}]},{"osd":7,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.77900000000000003,"15min":0.75900000000000001},"min":{"1min":0.56799999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.79800000000000004,"15min":0.77300000000000002},"min":{"1min":0.68999999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.96499999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.89500000000000002,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":7.2009999999999996,"5min":7.2009999999999996,"15min":7.2009999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.87,"15min":0.86599999999999999},"min":{"1min":0.60499999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.0069999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.53800000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0780000000000001,"5min":2.165,"15min":2.165},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.91200000000000003,"15min":0.94199999999999995},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.244,"5min":2.331,"15min":2.331},"last":0.70799999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0089999999999999,"15min":0.95899999999999996},"min":{"1min":0.72599999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":8.1600000000000001,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.83399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":1060864,"data_stored":535619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56599999999999995,"15min":0.505},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80700000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.33000000000000002},"max":{"1min":0.73999999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.56899999999999995,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.57499999999999996,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.47899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58399999999999996,"15min":0.55600000000000005},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81200000000000006,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.501,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.436}]},{"osd":5,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.629,"15min":0.57999999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.66000000000000003,"15min":0.61099999999999999},"min":{"1min":0.51900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":1.02,"15min":1.02},"last":0.86799999999999999}]},{"osd":6,"last update":"Sat Jul 29 16:13:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67600000000000005},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90500000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.67000000000000004,"15min":0.66400000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.621}]},{"osd":7,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.62,"15min":0.58999999999999997},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":1.073,"15min":1.073},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.64200000000000002},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69399999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.754,"15min":0.75},"min":{"1min":0.57499999999999996,"5min":0.376,"15min":0.376},"max":{"1min":1.0920000000000001,"5min":1.101,"15min":1.101},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77000000000000002,"15min":0.76100000000000001},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.02,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.90600000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:13:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.85399999999999998},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.046,"5min":1.161,"15min":1.161},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84299999999999997,"15min":0.85699999999999998},"min":{"1min":0.622,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98099999999999998,"5min":1.175,"15min":1.175},"last":0.84199999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80700000000000005,"15min":0.81899999999999995},"min":{"1min":0.42799999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90300000000000002,"5min":1.129,"15min":1.129},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.81899999999999995},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.995,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.71599999999999997}]},{"osd":11,"last update":"Sat Jul 29 16:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":1015808,"data_stored":478210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56599999999999995,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59199999999999997,"15min":0.54500000000000004},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60999999999999999,"15min":0.58399999999999996},"min":{"1min":0.496,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90700000000000003,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.45100000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85999999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57099999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68200000000000005,"15min":0.66100000000000003},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94399999999999995,"5min":1.111,"15min":1.111},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Jul 29 16:13:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.43099999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.264,"5min":1.266,"15min":1.266},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.74099999999999999},"min":{"1min":0.435,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:13:52 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.67200000000000004}]},{"osd":8,"last update":"Sat Jul 29 16:13:30 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.84299999999999997,"15min":0.88300000000000001},"min":{"1min":0.55200000000000005,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.335,"15min":1.335},"last":0.621},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.89400000000000002},"min":{"1min":0.64100000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91100000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.68799999999999994}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.86499999999999999,"15min":0.88300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.079,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.751},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.94499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.014,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.71899999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:13:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.66100000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.91700000000000004,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.85699999999999998},"min":{"1min":0.64400000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.1160000000000001,"5min":1.212,"15min":1.212},"last":0.70599999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.82199999999999995},"min":{"1min":0.63500000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.745},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.749,"5min":0.626,"15min":0.626},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.78500000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1069056,"data_stored":538667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.375,"5min":0.308,"15min":0.308},"max":{"1min":3.4169999999999998,"5min":3.4169999999999998,"15min":3.4169999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60199999999999998,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78000000000000003,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58299999999999996,"15min":0.54600000000000004},"min":{"1min":0.40799999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78200000000000003,"5min":3.613,"15min":3.613},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.56899999999999995,"15min":0.54300000000000004},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.628}]},{"osd":3,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.60399999999999998,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.4550000000000001,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60599999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:13:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.45600000000000002,"15min":0.442},"min":{"1min":0.39100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66900000000000004,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.439},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.497,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73799999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:13:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.51800000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77900000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.753,"15min":0.66900000000000004},"min":{"1min":0.5,"5min":0.379,"15min":0.379},"max":{"1min":5.1139999999999999,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:13:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.624,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.85499999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.64800000000000002},"min":{"1min":0.51000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.79300000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.749}]},{"osd":9,"last update":"Sat Jul 29 16:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76400000000000001},"min":{"1min":0.56399999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90400000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.67500000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":3.5950000000000002,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.89600000000000002}]},{"osd":10,"last update":"Sat Jul 29 16:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.56899999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85899999999999999,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.69699999999999995,"15min":0.68200000000000005},"min":{"1min":0.59699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.82799999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.85199999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":1007616,"data_stored":476931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.46800000000000003,"15min":0.41099999999999998},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.56699999999999995,"15min":0.46800000000000003},"min":{"1min":0.39500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.766,"5min":8.766,"15min":8.766},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63200000000000001,"15min":0.58199999999999996},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.88500000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.55100000000000005,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70199999999999996,"15min":0.63200000000000001},"min":{"1min":0.61399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.69699999999999995,"15min":0.61699999999999999},"min":{"1min":0.56000000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1559999999999999,"5min":1.296,"15min":1.296},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64400000000000002,"15min":0.60699999999999998},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88400000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.53400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98099999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.57099999999999995,"5min":0.443,"15min":0.443},"max":{"1min":0.97799999999999998,"5min":1.343,"15min":1.343},"last":0.745},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72699999999999998,"15min":0.70099999999999996},"min":{"1min":0.53900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.84099999999999997,"5min":1.032,"15min":1.032},"last":0.88300000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:13:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.70099999999999996,"15min":0.68200000000000005},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83299999999999996,"5min":1.276,"15min":1.276},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70199999999999996,"15min":0.66400000000000003},"min":{"1min":0.48499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59699999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0800000000000001,"5min":1.323,"15min":1.323},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.83599999999999997,"15min":0.82799999999999996},"min":{"1min":0.60799999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96799999999999997,"5min":1.357,"15min":1.357},"last":0.999}]},{"osd":8,"last update":"Sat Jul 29 16:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":1.034},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.77900000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.90300000000000002,"15min":0.92300000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.091,"5min":1.476,"15min":1.476},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.90100000000000002,"15min":0.91800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0509999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.78700000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86799999999999999,"15min":0.87},"min":{"1min":0.74199999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.157,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.90200000000000002},"min":{"1min":0.71599999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.244,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.872}]},{"osd":11,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.93999999999999995}]}]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":1003520,"data_stored":478438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.56599999999999995},"min":{"1min":0.50700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60099999999999998,"15min":0.55600000000000005},"min":{"1min":0.45100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93300000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Jul 29 16:13:27 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.65900000000000003,"15min":0.60299999999999998},"min":{"1min":0.50600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.65000000000000002,"15min":0.60899999999999999},"min":{"1min":0.54600000000000004,"5min":0.374,"15min":0.374},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65200000000000002,"15min":0.628},"min":{"1min":0.46300000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.79100000000000004,"5min":1.018,"15min":1.018},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.58699999999999997},"min":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91700000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.749},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.628},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0129999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.60999999999999999}]},{"osd":8,"last update":"Sat Jul 29 16:13:32 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.76700000000000002,"15min":0.753},"min":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.78700000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79500000000000004,"15min":0.80000000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.04,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73999999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.115,"15min":1.115},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.76600000000000001,"15min":0.77000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.72899999999999998}]},{"osd":11,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.80200000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28528,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716624,"statfs":{"total":95995035648,"available":95965822976,"internally_reserved":0,"allocated":1605632,"data_stored":1068077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.56499999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.55100000000000005,"15min":0.51700000000000002},"min":{"1min":0.48999999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.746},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56599999999999995,"15min":0.53500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77200000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.42499999999999999,"15min":0.40699999999999997},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.435,"15min":0.41499999999999998},"min":{"1min":0.373,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60399999999999998,"15min":0.58399999999999996},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52300000000000002}]},{"osd":4,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.61399999999999999,"15min":0.58199999999999996},"min":{"1min":0.59999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.64900000000000002,"15min":0.63200000000000001},"min":{"1min":0.54700000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68200000000000005,"15min":0.65600000000000003},"min":{"1min":0.55900000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Jul 29 16:13:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.63400000000000001,"15min":0.63100000000000001},"min":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.46500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.81599999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.621,"15min":0.61099999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82699999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.51400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.622}]},{"osd":10,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.63200000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.73199999999999998,"15min":0.71899999999999997},"min":{"1min":0.65200000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.92000000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:14:02.496 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-07-29T16:14:02.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-29T16:14:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 -- 172.21.15.31:0/3401209367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710101530 msgr2=0x7f1710101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 --2- 172.21.15.31:0/3401209367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710101530 0x7f1710101980 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1700009990 tx=0x7f1700009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 -- 172.21.15.31:0/3401209367 shutdown_connections 2023-07-29T16:14:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 --2- 172.21.15.31:0/3401209367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f1710108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 --2- 172.21.15.31:0/3401209367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1710100340 0x7f1710100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 --2- 172.21.15.31:0/3401209367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710101530 0x7f1710101980 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.734+0000 7f17185ff700 1 -- 172.21.15.31:0/3401209367 >> 172.21.15.31:0/3401209367 conn(0x7f17100fb920 msgr2=0x7f17100fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:02.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.735+0000 7f17185ff700 1 -- 172.21.15.31:0/3401209367 shutdown_connections 2023-07-29T16:14:02.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.735+0000 7f17185ff700 1 -- 172.21.15.31:0/3401209367 wait complete. 2023-07-29T16:14:02.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 Processor -- start 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 -- start start 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710100340 0x7f17101982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1710101530 0x7f17101987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f171019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171019fe30 con 0x7f1710101ec0 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171019ff70 con 0x7f1710100340 2023-07-29T16:14:02.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.736+0000 7f17185ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17101a0100 con 0x7f1710101530 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f171019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f171019f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43390/0 (socket says 172.21.15.31:43390) 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 -- 172.21.15.31:0/2873298874 learned_addr learned my addr 172.21.15.31:0/2873298874 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1715b9a700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1710101530 0x7f17101987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f171639b700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710100340 0x7f17101982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 -- 172.21.15.31:0/2873298874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1710101530 msgr2=0x7f17101987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1710101530 0x7f17101987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 -- 172.21.15.31:0/2873298874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710100340 msgr2=0x7f17101982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710100340 0x7f17101982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 -- 172.21.15.31:0/2873298874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17000095f0 con 0x7f1710101ec0 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f1716b9c700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f171019f860 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f170800ef30 tx=0x7f170800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.737+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1708018070 con 0x7f1710101ec0 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.738+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f170800f040 con 0x7f1710101ec0 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.738+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170800cd00 con 0x7f1710101ec0 2023-07-29T16:14:02.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.738+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17101a0380 con 0x7f1710101ec0 2023-07-29T16:14:02.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.738+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17101a0840 con 0x7f1710101ec0 2023-07-29T16:14:02.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.739+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1710192a30 con 0x7f1710101ec0 2023-07-29T16:14:02.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.741+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1708003a00 con 0x7f1710101ec0 2023-07-29T16:14:02.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.742+0000 7f17077fe700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16fc0a0860 0x7f16fc0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:02.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.742+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f17080cb5a0 con 0x7f1710101ec0 2023-07-29T16:14:02.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.743+0000 7f171639b700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16fc0a0860 0x7f16fc0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:02.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.743+0000 7f171639b700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16fc0a0860 0x7f16fc0a2d10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1710101390 tx=0x7f170c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:02.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.746+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1708095010 con 0x7f1710101ec0 2023-07-29T16:14:02.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.950+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 --> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1710061190 con 0x7f16fc0a0860 2023-07-29T16:14:02.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.961+0000 7f17077fe700 1 -- 172.21.15.31:0/2873298874 <== mgr.14152 v2:172.21.15.31:6800/4163929785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343664 (secure 0 0 0) 0x7f1710061190 con 0x7f16fc0a0860 2023-07-29T16:14:02.963 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:02.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16fc0a0860 msgr2=0x7f16fc0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16fc0a0860 0x7f16fc0a2d10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1710101390 tx=0x7f170c009260 comp rx=0 tx=0).stop 2023-07-29T16:14:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 msgr2=0x7f171019f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f171019f860 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f170800ef30 tx=0x7f170800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 shutdown_connections 2023-07-29T16:14:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f16fc0a0860 0x7f16fc0a2d10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1710101ec0 0x7f171019f860 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1710100340 0x7f17101982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 --2- 172.21.15.31:0/2873298874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1710101530 0x7f17101987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:02.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 >> 172.21.15.31:0/2873298874 conn(0x7f17100fb920 msgr2=0x7f1710106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:02.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.965+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 shutdown_connections 2023-07-29T16:14:02.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:02.966+0000 7f17185ff700 1 -- 172.21.15.31:0/2873298874 wait complete. 2023-07-29T16:14:02.966 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-07-29T16:14:02.982 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2023-07-29T16:14:01.295437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657289,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":694,"num_read_kb":463,"num_write":347,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1182420,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337148,"kb_used_data":13716,"kb_used_omap":0,"kb_used_meta":323328,"kb_avail":1124604676,"statfs":{"total":1151940427776,"available":1151595188224,"internally_reserved":0,"allocated":14045184,"data_stored":7682421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":331087872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5483,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":32,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.519441"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778124+0000","last_change":"2023-07-29T16:12:21.931757+0000","last_active":"2023-07-29T16:13:49.778124+0000","last_peered":"2023-07-29T16:13:49.778124+0000","last_clean":"2023-07-29T16:13:49.778124+0000","last_became_active":"2023-07-29T16:12:21.930808+0000","last_became_peered":"2023-07-29T16:12:21.930808+0000","last_unstale":"2023-07-29T16:13:49.778124+0000","last_undegraded":"2023-07-29T16:13:49.778124+0000","last_fullsized":"2023-07-29T16:13:49.778124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:22:38.307780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776048+0000","last_change":"2023-07-29T16:12:21.943826+0000","last_active":"2023-07-29T16:13:49.776048+0000","last_peered":"2023-07-29T16:13:49.776048+0000","last_clean":"2023-07-29T16:13:49.776048+0000","last_became_active":"2023-07-29T16:12:21.943414+0000","last_became_peered":"2023-07-29T16:12:21.943414+0000","last_unstale":"2023-07-29T16:13:49.776048+0000","last_undegraded":"2023-07-29T16:13:49.776048+0000","last_fullsized":"2023-07-29T16:13:49.776048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:22:35.929025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776110+0000","last_change":"2023-07-29T16:12:21.928715+0000","last_active":"2023-07-29T16:13:49.776110+0000","last_peered":"2023-07-29T16:13:49.776110+0000","last_clean":"2023-07-29T16:13:49.776110+0000","last_became_active":"2023-07-29T16:12:21.928390+0000","last_became_peered":"2023-07-29T16:12:21.928390+0000","last_unstale":"2023-07-29T16:13:49.776110+0000","last_undegraded":"2023-07-29T16:13:49.776110+0000","last_fullsized":"2023-07-29T16:13:49.776110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:33:21.001968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830930+0000","last_change":"2023-07-29T16:12:50.286750+0000","last_active":"2023-07-29T16:13:49.830930+0000","last_peered":"2023-07-29T16:13:49.830930+0000","last_clean":"2023-07-29T16:13:49.830930+0000","last_became_active":"2023-07-29T16:12:50.286341+0000","last_became_peered":"2023-07-29T16:12:50.286341+0000","last_unstale":"2023-07-29T16:13:49.830930+0000","last_undegraded":"2023-07-29T16:13:49.830930+0000","last_fullsized":"2023-07-29T16:13:49.830930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.547598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829633+0000","last_change":"2023-07-29T16:12:21.939782+0000","last_active":"2023-07-29T16:13:49.829633+0000","last_peered":"2023-07-29T16:13:49.829633+0000","last_clean":"2023-07-29T16:13:49.829633+0000","last_became_active":"2023-07-29T16:12:21.939497+0000","last_became_peered":"2023-07-29T16:12:21.939497+0000","last_unstale":"2023-07-29T16:13:49.829633+0000","last_undegraded":"2023-07-29T16:13:49.829633+0000","last_fullsized":"2023-07-29T16:13:49.829633+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:01.900820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830834+0000","last_change":"2023-07-29T16:12:21.947040+0000","last_active":"2023-07-29T16:13:49.830834+0000","last_peered":"2023-07-29T16:13:49.830834+0000","last_clean":"2023-07-29T16:13:49.830834+0000","last_became_active":"2023-07-29T16:12:21.946735+0000","last_became_peered":"2023-07-29T16:12:21.946735+0000","last_unstale":"2023-07-29T16:13:49.830834+0000","last_undegraded":"2023-07-29T16:13:49.830834+0000","last_fullsized":"2023-07-29T16:13:49.830834+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:15:27.167694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774908+0000","last_change":"2023-07-29T16:12:21.940276+0000","last_active":"2023-07-29T16:13:49.774908+0000","last_peered":"2023-07-29T16:13:49.774908+0000","last_clean":"2023-07-29T16:13:49.774908+0000","last_became_active":"2023-07-29T16:12:21.939959+0000","last_became_peered":"2023-07-29T16:12:21.939959+0000","last_unstale":"2023-07-29T16:13:49.774908+0000","last_undegraded":"2023-07-29T16:13:49.774908+0000","last_fullsized":"2023-07-29T16:13:49.774908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:47:20.318594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829985+0000","last_change":"2023-07-29T16:12:50.335631+0000","last_active":"2023-07-29T16:13:49.829985+0000","last_peered":"2023-07-29T16:13:49.829985+0000","last_clean":"2023-07-29T16:13:49.829985+0000","last_became_active":"2023-07-29T16:12:50.335213+0000","last_became_peered":"2023-07-29T16:12:50.335213+0000","last_unstale":"2023-07-29T16:13:49.829985+0000","last_undegraded":"2023-07-29T16:13:49.829985+0000","last_fullsized":"2023-07-29T16:13:49.829985+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:04:33.083392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779120+0000","last_change":"2023-07-29T16:12:21.930159+0000","last_active":"2023-07-29T16:13:49.779120+0000","last_peered":"2023-07-29T16:13:49.779120+0000","last_clean":"2023-07-29T16:13:49.779120+0000","last_became_active":"2023-07-29T16:12:21.929824+0000","last_became_peered":"2023-07-29T16:12:21.929824+0000","last_unstale":"2023-07-29T16:13:49.779120+0000","last_undegraded":"2023-07-29T16:13:49.779120+0000","last_fullsized":"2023-07-29T16:13:49.779120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:23:56.252640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830766+0000","last_change":"2023-07-29T16:12:21.951833+0000","last_active":"2023-07-29T16:13:49.830766+0000","last_peered":"2023-07-29T16:13:49.830766+0000","last_clean":"2023-07-29T16:13:49.830766+0000","last_became_active":"2023-07-29T16:12:21.951457+0000","last_became_peered":"2023-07-29T16:12:21.951457+0000","last_unstale":"2023-07-29T16:13:49.830766+0000","last_undegraded":"2023-07-29T16:13:49.830766+0000","last_fullsized":"2023-07-29T16:13:49.830766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:33:16.824020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775219+0000","last_change":"2023-07-29T16:12:21.946431+0000","last_active":"2023-07-29T16:13:49.775219+0000","last_peered":"2023-07-29T16:13:49.775219+0000","last_clean":"2023-07-29T16:13:49.775219+0000","last_became_active":"2023-07-29T16:12:21.946125+0000","last_became_peered":"2023-07-29T16:12:21.946125+0000","last_unstale":"2023-07-29T16:13:49.775219+0000","last_undegraded":"2023-07-29T16:13:49.775219+0000","last_fullsized":"2023-07-29T16:13:49.775219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:31:52.116315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830343+0000","last_change":"2023-07-29T16:12:21.944779+0000","last_active":"2023-07-29T16:13:49.830343+0000","last_peered":"2023-07-29T16:13:49.830343+0000","last_clean":"2023-07-29T16:13:49.830343+0000","last_became_active":"2023-07-29T16:12:21.944413+0000","last_became_peered":"2023-07-29T16:12:21.944413+0000","last_unstale":"2023-07-29T16:13:49.830343+0000","last_undegraded":"2023-07-29T16:13:49.830343+0000","last_fullsized":"2023-07-29T16:13:49.830343+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:09:43.096071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830414+0000","last_change":"2023-07-29T16:12:21.946076+0000","last_active":"2023-07-29T16:13:49.830414+0000","last_peered":"2023-07-29T16:13:49.830414+0000","last_clean":"2023-07-29T16:13:49.830414+0000","last_became_active":"2023-07-29T16:12:21.945678+0000","last_became_peered":"2023-07-29T16:12:21.945678+0000","last_unstale":"2023-07-29T16:13:49.830414+0000","last_undegraded":"2023-07-29T16:13:49.830414+0000","last_fullsized":"2023-07-29T16:13:49.830414+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:13:18.222836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775983+0000","last_change":"2023-07-29T16:12:21.942920+0000","last_active":"2023-07-29T16:13:49.775983+0000","last_peered":"2023-07-29T16:13:49.775983+0000","last_clean":"2023-07-29T16:13:49.775983+0000","last_became_active":"2023-07-29T16:12:21.942619+0000","last_became_peered":"2023-07-29T16:12:21.942619+0000","last_unstale":"2023-07-29T16:13:49.775983+0000","last_undegraded":"2023-07-29T16:13:49.775983+0000","last_fullsized":"2023-07-29T16:13:49.775983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:20:42.228247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775874+0000","last_change":"2023-07-29T16:12:50.291287+0000","last_active":"2023-07-29T16:13:49.775874+0000","last_peered":"2023-07-29T16:13:49.775874+0000","last_clean":"2023-07-29T16:13:49.775874+0000","last_became_active":"2023-07-29T16:12:50.290878+0000","last_became_peered":"2023-07-29T16:12:50.290878+0000","last_unstale":"2023-07-29T16:13:49.775874+0000","last_undegraded":"2023-07-29T16:13:49.775874+0000","last_fullsized":"2023-07-29T16:13:49.775874+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:59:20.470299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778600+0000","last_change":"2023-07-29T16:12:21.949468+0000","last_active":"2023-07-29T16:13:49.778600+0000","last_peered":"2023-07-29T16:13:49.778600+0000","last_clean":"2023-07-29T16:13:49.778600+0000","last_became_active":"2023-07-29T16:12:21.949176+0000","last_became_peered":"2023-07-29T16:12:21.949176+0000","last_unstale":"2023-07-29T16:13:49.778600+0000","last_undegraded":"2023-07-29T16:13:49.778600+0000","last_fullsized":"2023-07-29T16:13:49.778600+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:43:26.095467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778403+0000","last_change":"2023-07-29T16:12:21.932169+0000","last_active":"2023-07-29T16:13:49.778403+0000","last_peered":"2023-07-29T16:13:49.778403+0000","last_clean":"2023-07-29T16:13:49.778403+0000","last_became_active":"2023-07-29T16:12:21.931890+0000","last_became_peered":"2023-07-29T16:12:21.931890+0000","last_unstale":"2023-07-29T16:13:49.778403+0000","last_undegraded":"2023-07-29T16:13:49.778403+0000","last_fullsized":"2023-07-29T16:13:49.778403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:44:06.426074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831266+0000","last_change":"2023-07-29T16:12:21.943662+0000","last_active":"2023-07-29T16:13:49.831266+0000","last_peered":"2023-07-29T16:13:49.831266+0000","last_clean":"2023-07-29T16:13:49.831266+0000","last_became_active":"2023-07-29T16:12:21.943332+0000","last_became_peered":"2023-07-29T16:12:21.943332+0000","last_unstale":"2023-07-29T16:13:49.831266+0000","last_undegraded":"2023-07-29T16:13:49.831266+0000","last_fullsized":"2023-07-29T16:13:49.831266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:03:39.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774488+0000","last_change":"2023-07-29T16:12:21.940345+0000","last_active":"2023-07-29T16:13:49.774488+0000","last_peered":"2023-07-29T16:13:49.774488+0000","last_clean":"2023-07-29T16:13:49.774488+0000","last_became_active":"2023-07-29T16:12:21.940027+0000","last_became_peered":"2023-07-29T16:12:21.940027+0000","last_unstale":"2023-07-29T16:13:49.774488+0000","last_undegraded":"2023-07-29T16:13:49.774488+0000","last_fullsized":"2023-07-29T16:13:49.774488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:13:34.700706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830125+0000","last_change":"2023-07-29T16:12:21.350007+0000","last_active":"2023-07-29T16:13:49.830125+0000","last_peered":"2023-07-29T16:13:49.830125+0000","last_clean":"2023-07-29T16:13:49.830125+0000","last_became_active":"2023-07-29T16:12:21.349714+0000","last_became_peered":"2023-07-29T16:12:21.349714+0000","last_unstale":"2023-07-29T16:13:49.830125+0000","last_undegraded":"2023-07-29T16:13:49.830125+0000","last_fullsized":"2023-07-29T16:13:49.830125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:23:02.491979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830525+0000","last_change":"2023-07-29T16:12:50.286738+0000","last_active":"2023-07-29T16:13:49.830525+0000","last_peered":"2023-07-29T16:13:49.830525+0000","last_clean":"2023-07-29T16:13:49.830525+0000","last_became_active":"2023-07-29T16:12:50.286379+0000","last_became_peered":"2023-07-29T16:12:50.286379+0000","last_unstale":"2023-07-29T16:13:49.830525+0000","last_undegraded":"2023-07-29T16:13:49.830525+0000","last_fullsized":"2023-07-29T16:13:49.830525+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:50:03.338896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830606+0000","last_change":"2023-07-29T16:12:21.352195+0000","last_active":"2023-07-29T16:13:49.830606+0000","last_peered":"2023-07-29T16:13:49.830606+0000","last_clean":"2023-07-29T16:13:49.830606+0000","last_became_active":"2023-07-29T16:12:21.351881+0000","last_became_peered":"2023-07-29T16:12:21.351881+0000","last_unstale":"2023-07-29T16:13:49.830606+0000","last_undegraded":"2023-07-29T16:13:49.830606+0000","last_fullsized":"2023-07-29T16:13:49.830606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:00:08.917217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776300+0000","last_change":"2023-07-29T16:12:21.934344+0000","last_active":"2023-07-29T16:13:49.776300+0000","last_peered":"2023-07-29T16:13:49.776300+0000","last_clean":"2023-07-29T16:13:49.776300+0000","last_became_active":"2023-07-29T16:12:21.934037+0000","last_became_peered":"2023-07-29T16:12:21.934037+0000","last_unstale":"2023-07-29T16:13:49.776300+0000","last_undegraded":"2023-07-29T16:13:49.776300+0000","last_fullsized":"2023-07-29T16:13:49.776300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:32:48.568366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830130+0000","last_change":"2023-07-29T16:12:21.345797+0000","last_active":"2023-07-29T16:13:49.830130+0000","last_peered":"2023-07-29T16:13:49.830130+0000","last_clean":"2023-07-29T16:13:49.830130+0000","last_became_active":"2023-07-29T16:12:21.345317+0000","last_became_peered":"2023-07-29T16:12:21.345317+0000","last_unstale":"2023-07-29T16:13:49.830130+0000","last_undegraded":"2023-07-29T16:13:49.830130+0000","last_fullsized":"2023-07-29T16:13:49.830130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:12:51.561265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778941+0000","last_change":"2023-07-29T16:12:21.945574+0000","last_active":"2023-07-29T16:13:49.778941+0000","last_peered":"2023-07-29T16:13:49.778941+0000","last_clean":"2023-07-29T16:13:49.778941+0000","last_became_active":"2023-07-29T16:12:21.945226+0000","last_became_peered":"2023-07-29T16:12:21.945226+0000","last_unstale":"2023-07-29T16:13:49.778941+0000","last_undegraded":"2023-07-29T16:13:49.778941+0000","last_fullsized":"2023-07-29T16:13:49.778941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:03:49.642883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777906+0000","last_change":"2023-07-29T16:12:50.288144+0000","last_active":"2023-07-29T16:13:49.777906+0000","last_peered":"2023-07-29T16:13:49.777906+0000","last_clean":"2023-07-29T16:13:49.777906+0000","last_became_active":"2023-07-29T16:12:50.283877+0000","last_became_peered":"2023-07-29T16:12:50.283877+0000","last_unstale":"2023-07-29T16:13:49.777906+0000","last_undegraded":"2023-07-29T16:13:49.777906+0000","last_fullsized":"2023-07-29T16:13:49.777906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T04:00:21.950054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778880+0000","last_change":"2023-07-29T16:12:21.937549+0000","last_active":"2023-07-29T16:13:49.778880+0000","last_peered":"2023-07-29T16:13:49.778880+0000","last_clean":"2023-07-29T16:13:49.778880+0000","last_became_active":"2023-07-29T16:12:21.937206+0000","last_became_peered":"2023-07-29T16:12:21.937206+0000","last_unstale":"2023-07-29T16:13:49.778880+0000","last_undegraded":"2023-07-29T16:13:49.778880+0000","last_fullsized":"2023-07-29T16:13:49.778880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:09:32.958711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:57.798989+0000","last_change":"2023-07-29T16:13:50.790518+0000","last_active":"2023-07-29T16:13:57.798989+0000","last_peered":"2023-07-29T16:13:57.798989+0000","last_clean":"2023-07-29T16:13:57.798989+0000","last_became_active":"2023-07-29T16:13:50.783618+0000","last_became_peered":"2023-07-29T16:13:50.783618+0000","last_unstale":"2023-07-29T16:13:57.798989+0000","last_undegraded":"2023-07-29T16:13:57.798989+0000","last_fullsized":"2023-07-29T16:13:57.798989+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:39:28.814617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1418,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829658+0000","last_change":"2023-07-29T16:12:21.933221+0000","last_active":"2023-07-29T16:13:49.829658+0000","last_peered":"2023-07-29T16:13:49.829658+0000","last_clean":"2023-07-29T16:13:49.829658+0000","last_became_active":"2023-07-29T16:12:21.932849+0000","last_became_peered":"2023-07-29T16:12:21.932849+0000","last_unstale":"2023-07-29T16:13:49.829658+0000","last_undegraded":"2023-07-29T16:13:49.829658+0000","last_fullsized":"2023-07-29T16:13:49.829658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:19:05.951083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:50.877482+0000","last_change":"2023-07-29T16:13:50.877482+0000","last_active":"2023-07-29T16:13:50.877482+0000","last_peered":"2023-07-29T16:13:50.877482+0000","last_clean":"2023-07-29T16:13:50.877482+0000","last_became_active":"2023-07-29T16:13:50.874496+0000","last_became_peered":"2023-07-29T16:13:50.874496+0000","last_unstale":"2023-07-29T16:13:50.877482+0000","last_undegraded":"2023-07-29T16:13:50.877482+0000","last_fullsized":"2023-07-29T16:13:50.877482+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:47:18.956678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,6],"acting":[10,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779443+0000","last_change":"2023-07-29T16:12:21.938197+0000","last_active":"2023-07-29T16:13:49.779443+0000","last_peered":"2023-07-29T16:13:49.779443+0000","last_clean":"2023-07-29T16:13:49.779443+0000","last_became_active":"2023-07-29T16:12:21.937337+0000","last_became_peered":"2023-07-29T16:12:21.937337+0000","last_unstale":"2023-07-29T16:13:49.779443+0000","last_undegraded":"2023-07-29T16:13:49.779443+0000","last_fullsized":"2023-07-29T16:13:49.779443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:07:37.748444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831234+0000","last_change":"2023-07-29T16:12:21.356111+0000","last_active":"2023-07-29T16:13:49.831234+0000","last_peered":"2023-07-29T16:13:49.831234+0000","last_clean":"2023-07-29T16:13:49.831234+0000","last_became_active":"2023-07-29T16:12:21.355805+0000","last_became_peered":"2023-07-29T16:12:21.355805+0000","last_unstale":"2023-07-29T16:13:49.831234+0000","last_undegraded":"2023-07-29T16:13:49.831234+0000","last_fullsized":"2023-07-29T16:13:49.831234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:42:02.059964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830062+0000","last_change":"2023-07-29T16:12:21.941012+0000","last_active":"2023-07-29T16:13:49.830062+0000","last_peered":"2023-07-29T16:13:49.830062+0000","last_clean":"2023-07-29T16:13:49.830062+0000","last_became_active":"2023-07-29T16:12:21.940715+0000","last_became_peered":"2023-07-29T16:12:21.940715+0000","last_unstale":"2023-07-29T16:13:49.830062+0000","last_undegraded":"2023-07-29T16:13:49.830062+0000","last_fullsized":"2023-07-29T16:13:49.830062+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:20.327338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778416+0000","last_change":"2023-07-29T16:12:21.345164+0000","last_active":"2023-07-29T16:13:49.778416+0000","last_peered":"2023-07-29T16:13:49.778416+0000","last_clean":"2023-07-29T16:13:49.778416+0000","last_became_active":"2023-07-29T16:12:21.344679+0000","last_became_peered":"2023-07-29T16:12:21.344679+0000","last_unstale":"2023-07-29T16:13:49.778416+0000","last_undegraded":"2023-07-29T16:13:49.778416+0000","last_fullsized":"2023-07-29T16:13:49.778416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:27:02.071926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830370+0000","last_change":"2023-07-29T16:12:21.943399+0000","last_active":"2023-07-29T16:13:49.830370+0000","last_peered":"2023-07-29T16:13:49.830370+0000","last_clean":"2023-07-29T16:13:49.830370+0000","last_became_active":"2023-07-29T16:12:21.942833+0000","last_became_peered":"2023-07-29T16:12:21.942833+0000","last_unstale":"2023-07-29T16:13:49.830370+0000","last_undegraded":"2023-07-29T16:13:49.830370+0000","last_fullsized":"2023-07-29T16:13:49.830370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:14:25.866240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":110,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:55.795897+0000","last_change":"2023-07-29T16:12:21.345837+0000","last_active":"2023-07-29T16:13:55.795897+0000","last_peered":"2023-07-29T16:13:55.795897+0000","last_clean":"2023-07-29T16:13:55.795897+0000","last_became_active":"2023-07-29T16:12:21.345495+0000","last_became_peered":"2023-07-29T16:12:21.345495+0000","last_unstale":"2023-07-29T16:13:55.795897+0000","last_undegraded":"2023-07-29T16:13:55.795897+0000","last_fullsized":"2023-07-29T16:13:55.795897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:00:26.515595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":90,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779064+0000","last_change":"2023-07-29T16:12:21.942645+0000","last_active":"2023-07-29T16:13:49.779064+0000","last_peered":"2023-07-29T16:13:49.779064+0000","last_clean":"2023-07-29T16:13:49.779064+0000","last_became_active":"2023-07-29T16:12:21.942265+0000","last_became_peered":"2023-07-29T16:12:21.942265+0000","last_unstale":"2023-07-29T16:13:49.779064+0000","last_undegraded":"2023-07-29T16:13:49.779064+0000","last_fullsized":"2023-07-29T16:13:49.779064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:15:42.361807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775359+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:13:49.775359+0000","last_peered":"2023-07-29T16:13:49.775359+0000","last_clean":"2023-07-29T16:13:49.775359+0000","last_became_active":"2023-07-29T16:12:21.347399+0000","last_became_peered":"2023-07-29T16:12:21.347399+0000","last_unstale":"2023-07-29T16:13:49.775359+0000","last_undegraded":"2023-07-29T16:13:49.775359+0000","last_fullsized":"2023-07-29T16:13:49.775359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:39:41.813657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776218+0000","last_change":"2023-07-29T16:12:50.336088+0000","last_active":"2023-07-29T16:13:49.776218+0000","last_peered":"2023-07-29T16:13:49.776218+0000","last_clean":"2023-07-29T16:13:49.776218+0000","last_became_active":"2023-07-29T16:12:50.335715+0000","last_became_peered":"2023-07-29T16:12:50.335715+0000","last_unstale":"2023-07-29T16:13:49.776218+0000","last_undegraded":"2023-07-29T16:13:49.776218+0000","last_fullsized":"2023-07-29T16:13:49.776218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:53:33.332020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778964+0000","last_change":"2023-07-29T16:12:21.346524+0000","last_active":"2023-07-29T16:13:49.778964+0000","last_peered":"2023-07-29T16:13:49.778964+0000","last_clean":"2023-07-29T16:13:49.778964+0000","last_became_active":"2023-07-29T16:12:21.346163+0000","last_became_peered":"2023-07-29T16:12:21.346163+0000","last_unstale":"2023-07-29T16:13:49.778964+0000","last_undegraded":"2023-07-29T16:13:49.778964+0000","last_fullsized":"2023-07-29T16:13:49.778964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:14.949965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779024+0000","last_change":"2023-07-29T16:12:21.945368+0000","last_active":"2023-07-29T16:13:49.779024+0000","last_peered":"2023-07-29T16:13:49.779024+0000","last_clean":"2023-07-29T16:13:49.779024+0000","last_became_active":"2023-07-29T16:12:21.945039+0000","last_became_peered":"2023-07-29T16:12:21.945039+0000","last_unstale":"2023-07-29T16:13:49.779024+0000","last_undegraded":"2023-07-29T16:13:49.779024+0000","last_fullsized":"2023-07-29T16:13:49.779024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:06:15.811567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'30","reported_seq":234,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:56.939908+0000","last_change":"2023-07-29T16:12:21.350347+0000","last_active":"2023-07-29T16:13:56.939908+0000","last_peered":"2023-07-29T16:13:56.939908+0000","last_clean":"2023-07-29T16:13:56.939908+0000","last_became_active":"2023-07-29T16:12:21.349992+0000","last_became_peered":"2023-07-29T16:12:21.349992+0000","last_unstale":"2023-07-29T16:13:56.939908+0000","last_undegraded":"2023-07-29T16:13:56.939908+0000","last_fullsized":"2023-07-29T16:13:56.939908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:03:11.143532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61789,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":188,"num_read_kb":149,"num_write":31,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829885+0000","last_change":"2023-07-29T16:12:21.949618+0000","last_active":"2023-07-29T16:13:49.829885+0000","last_peered":"2023-07-29T16:13:49.829885+0000","last_clean":"2023-07-29T16:13:49.829885+0000","last_became_active":"2023-07-29T16:12:21.949074+0000","last_became_peered":"2023-07-29T16:12:21.949074+0000","last_unstale":"2023-07-29T16:13:49.829885+0000","last_undegraded":"2023-07-29T16:13:49.829885+0000","last_fullsized":"2023-07-29T16:13:49.829885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:37:22.380188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774452+0000","last_change":"2023-07-29T16:12:21.347610+0000","last_active":"2023-07-29T16:13:49.774452+0000","last_peered":"2023-07-29T16:13:49.774452+0000","last_clean":"2023-07-29T16:13:49.774452+0000","last_became_active":"2023-07-29T16:12:21.347333+0000","last_became_peered":"2023-07-29T16:12:21.347333+0000","last_unstale":"2023-07-29T16:13:49.774452+0000","last_undegraded":"2023-07-29T16:13:49.774452+0000","last_fullsized":"2023-07-29T16:13:49.774452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:09:11.553810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830632+0000","last_change":"2023-07-29T16:12:21.949397+0000","last_active":"2023-07-29T16:13:49.830632+0000","last_peered":"2023-07-29T16:13:49.830632+0000","last_clean":"2023-07-29T16:13:49.830632+0000","last_became_active":"2023-07-29T16:12:21.948916+0000","last_became_peered":"2023-07-29T16:12:21.948916+0000","last_unstale":"2023-07-29T16:13:49.830632+0000","last_undegraded":"2023-07-29T16:13:49.830632+0000","last_fullsized":"2023-07-29T16:13:49.830632+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:19:37.778102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.779295+0000","last_change":"2023-07-29T16:12:21.345048+0000","last_active":"2023-07-29T16:13:49.779295+0000","last_peered":"2023-07-29T16:13:49.779295+0000","last_clean":"2023-07-29T16:13:49.779295+0000","last_became_active":"2023-07-29T16:12:21.344689+0000","last_became_peered":"2023-07-29T16:12:21.344689+0000","last_unstale":"2023-07-29T16:13:49.779295+0000","last_undegraded":"2023-07-29T16:13:49.779295+0000","last_fullsized":"2023-07-29T16:13:49.779295+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:54:48.809331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775110+0000","last_change":"2023-07-29T16:12:21.948470+0000","last_active":"2023-07-29T16:13:49.775110+0000","last_peered":"2023-07-29T16:13:49.775110+0000","last_clean":"2023-07-29T16:13:49.775110+0000","last_became_active":"2023-07-29T16:12:21.948154+0000","last_became_peered":"2023-07-29T16:12:21.948154+0000","last_unstale":"2023-07-29T16:13:49.775110+0000","last_undegraded":"2023-07-29T16:13:49.775110+0000","last_fullsized":"2023-07-29T16:13:49.775110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:21:43.008079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":117,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:56.883507+0000","last_change":"2023-07-29T16:12:21.346106+0000","last_active":"2023-07-29T16:13:56.883507+0000","last_peered":"2023-07-29T16:13:56.883507+0000","last_clean":"2023-07-29T16:13:56.883507+0000","last_became_active":"2023-07-29T16:12:21.345763+0000","last_became_peered":"2023-07-29T16:12:21.345763+0000","last_unstale":"2023-07-29T16:13:56.883507+0000","last_undegraded":"2023-07-29T16:13:56.883507+0000","last_fullsized":"2023-07-29T16:13:56.883507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:38.135408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1068,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":97,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775014+0000","last_change":"2023-07-29T16:12:21.947178+0000","last_active":"2023-07-29T16:13:49.775014+0000","last_peered":"2023-07-29T16:13:49.775014+0000","last_clean":"2023-07-29T16:13:49.775014+0000","last_became_active":"2023-07-29T16:12:21.946829+0000","last_became_peered":"2023-07-29T16:12:21.946829+0000","last_unstale":"2023-07-29T16:13:49.775014+0000","last_undegraded":"2023-07-29T16:13:49.775014+0000","last_fullsized":"2023-07-29T16:13:49.775014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:08:30.392539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829533+0000","last_change":"2023-07-29T16:12:21.346937+0000","last_active":"2023-07-29T16:13:49.829533+0000","last_peered":"2023-07-29T16:13:49.829533+0000","last_clean":"2023-07-29T16:13:49.829533+0000","last_became_active":"2023-07-29T16:12:21.346571+0000","last_became_peered":"2023-07-29T16:12:21.346571+0000","last_unstale":"2023-07-29T16:13:49.829533+0000","last_undegraded":"2023-07-29T16:13:49.829533+0000","last_fullsized":"2023-07-29T16:13:49.829533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:39:39.696163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778567+0000","last_change":"2023-07-29T16:12:50.286421+0000","last_active":"2023-07-29T16:13:49.778567+0000","last_peered":"2023-07-29T16:13:49.778567+0000","last_clean":"2023-07-29T16:13:49.778567+0000","last_became_active":"2023-07-29T16:12:50.286075+0000","last_became_peered":"2023-07-29T16:12:50.286075+0000","last_unstale":"2023-07-29T16:13:49.778567+0000","last_undegraded":"2023-07-29T16:13:49.778567+0000","last_fullsized":"2023-07-29T16:13:49.778567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:24:16.006080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829886+0000","last_change":"2023-07-29T16:12:21.338473+0000","last_active":"2023-07-29T16:13:49.829886+0000","last_peered":"2023-07-29T16:13:49.829886+0000","last_clean":"2023-07-29T16:13:49.829886+0000","last_became_active":"2023-07-29T16:12:21.338153+0000","last_became_peered":"2023-07-29T16:12:21.338153+0000","last_unstale":"2023-07-29T16:13:49.829886+0000","last_undegraded":"2023-07-29T16:13:49.829886+0000","last_fullsized":"2023-07-29T16:13:49.829886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:10:00.746617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777964+0000","last_change":"2023-07-29T16:12:21.937155+0000","last_active":"2023-07-29T16:13:49.777964+0000","last_peered":"2023-07-29T16:13:49.777964+0000","last_clean":"2023-07-29T16:13:49.777964+0000","last_became_active":"2023-07-29T16:12:21.936839+0000","last_became_peered":"2023-07-29T16:12:21.936839+0000","last_unstale":"2023-07-29T16:13:49.777964+0000","last_undegraded":"2023-07-29T16:13:49.777964+0000","last_fullsized":"2023-07-29T16:13:49.777964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:39:04.834085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830840+0000","last_change":"2023-07-29T16:12:21.351087+0000","last_active":"2023-07-29T16:13:49.830840+0000","last_peered":"2023-07-29T16:13:49.830840+0000","last_clean":"2023-07-29T16:13:49.830840+0000","last_became_active":"2023-07-29T16:12:21.350704+0000","last_became_peered":"2023-07-29T16:12:21.350704+0000","last_unstale":"2023-07-29T16:13:49.830840+0000","last_undegraded":"2023-07-29T16:13:49.830840+0000","last_fullsized":"2023-07-29T16:13:49.830840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:44:25.490602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830261+0000","last_change":"2023-07-29T16:12:21.950557+0000","last_active":"2023-07-29T16:13:49.830261+0000","last_peered":"2023-07-29T16:13:49.830261+0000","last_clean":"2023-07-29T16:13:49.830261+0000","last_became_active":"2023-07-29T16:12:21.950250+0000","last_became_peered":"2023-07-29T16:12:21.950250+0000","last_unstale":"2023-07-29T16:13:49.830261+0000","last_undegraded":"2023-07-29T16:13:49.830261+0000","last_fullsized":"2023-07-29T16:13:49.830261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:45:08.547618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775445+0000","last_change":"2023-07-29T16:12:21.351460+0000","last_active":"2023-07-29T16:13:49.775445+0000","last_peered":"2023-07-29T16:13:49.775445+0000","last_clean":"2023-07-29T16:13:49.775445+0000","last_became_active":"2023-07-29T16:12:21.351183+0000","last_became_peered":"2023-07-29T16:12:21.351183+0000","last_unstale":"2023-07-29T16:13:49.775445+0000","last_undegraded":"2023-07-29T16:13:49.775445+0000","last_fullsized":"2023-07-29T16:13:49.775445+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:57:56.931210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830695+0000","last_change":"2023-07-29T16:12:21.944188+0000","last_active":"2023-07-29T16:13:49.830695+0000","last_peered":"2023-07-29T16:13:49.830695+0000","last_clean":"2023-07-29T16:13:49.830695+0000","last_became_active":"2023-07-29T16:12:21.943827+0000","last_became_peered":"2023-07-29T16:12:21.943827+0000","last_unstale":"2023-07-29T16:13:49.830695+0000","last_undegraded":"2023-07-29T16:13:49.830695+0000","last_fullsized":"2023-07-29T16:13:49.830695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:54:55.570797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778758+0000","last_change":"2023-07-29T16:12:21.937394+0000","last_active":"2023-07-29T16:13:49.778758+0000","last_peered":"2023-07-29T16:13:49.778758+0000","last_clean":"2023-07-29T16:13:49.778758+0000","last_became_active":"2023-07-29T16:12:21.937061+0000","last_became_peered":"2023-07-29T16:12:21.937061+0000","last_unstale":"2023-07-29T16:13:49.778758+0000","last_undegraded":"2023-07-29T16:13:49.778758+0000","last_fullsized":"2023-07-29T16:13:49.778758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:07:38.326297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":114,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:57.884345+0000","last_change":"2023-07-29T16:12:21.354905+0000","last_active":"2023-07-29T16:13:57.884345+0000","last_peered":"2023-07-29T16:13:57.884345+0000","last_clean":"2023-07-29T16:13:57.884345+0000","last_became_active":"2023-07-29T16:12:21.354538+0000","last_became_peered":"2023-07-29T16:12:21.354538+0000","last_unstale":"2023-07-29T16:13:57.884345+0000","last_undegraded":"2023-07-29T16:13:57.884345+0000","last_fullsized":"2023-07-29T16:13:57.884345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:10:45.814928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830891+0000","last_change":"2023-07-29T16:12:21.928401+0000","last_active":"2023-07-29T16:13:49.830891+0000","last_peered":"2023-07-29T16:13:49.830891+0000","last_clean":"2023-07-29T16:13:49.830891+0000","last_became_active":"2023-07-29T16:12:21.928081+0000","last_became_peered":"2023-07-29T16:12:21.928081+0000","last_unstale":"2023-07-29T16:13:49.830891+0000","last_undegraded":"2023-07-29T16:13:49.830891+0000","last_fullsized":"2023-07-29T16:13:49.830891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:51:51.147937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830292+0000","last_change":"2023-07-29T16:12:21.347758+0000","last_active":"2023-07-29T16:13:49.830292+0000","last_peered":"2023-07-29T16:13:49.830292+0000","last_clean":"2023-07-29T16:13:49.830292+0000","last_became_active":"2023-07-29T16:12:21.347293+0000","last_became_peered":"2023-07-29T16:12:21.347293+0000","last_unstale":"2023-07-29T16:13:49.830292+0000","last_undegraded":"2023-07-29T16:13:49.830292+0000","last_fullsized":"2023-07-29T16:13:49.830292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:22:39.355256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":332,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774940+0000","last_change":"2023-07-29T16:11:19.759650+0000","last_active":"2023-07-29T16:13:49.774940+0000","last_peered":"2023-07-29T16:13:49.774940+0000","last_clean":"2023-07-29T16:13:49.774940+0000","last_became_active":"2023-07-29T16:11:19.759077+0000","last_became_peered":"2023-07-29T16:11:19.759077+0000","last_unstale":"2023-07-29T16:13:49.774940+0000","last_undegraded":"2023-07-29T16:13:49.774940+0000","last_fullsized":"2023-07-29T16:13:49.774940+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:08:09.854117+0000","last_clean_scrub_stamp":"2023-07-29T16:08:09.854117+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:48:18.557659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831699+0000","last_change":"2023-07-29T16:12:21.947016+0000","last_active":"2023-07-29T16:13:49.831699+0000","last_peered":"2023-07-29T16:13:49.831699+0000","last_clean":"2023-07-29T16:13:49.831699+0000","last_became_active":"2023-07-29T16:12:21.946650+0000","last_became_peered":"2023-07-29T16:12:21.946650+0000","last_unstale":"2023-07-29T16:13:49.831699+0000","last_undegraded":"2023-07-29T16:13:49.831699+0000","last_fullsized":"2023-07-29T16:13:49.831699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:57:26.562018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774897+0000","last_change":"2023-07-29T16:12:21.350190+0000","last_active":"2023-07-29T16:13:49.774897+0000","last_peered":"2023-07-29T16:13:49.774897+0000","last_clean":"2023-07-29T16:13:49.774897+0000","last_became_active":"2023-07-29T16:12:21.349873+0000","last_became_peered":"2023-07-29T16:12:21.349873+0000","last_unstale":"2023-07-29T16:13:49.774897+0000","last_undegraded":"2023-07-29T16:13:49.774897+0000","last_fullsized":"2023-07-29T16:13:49.774897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:11:03.497958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829981+0000","last_change":"2023-07-29T16:12:21.350896+0000","last_active":"2023-07-29T16:13:49.829981+0000","last_peered":"2023-07-29T16:13:49.829981+0000","last_clean":"2023-07-29T16:13:49.829981+0000","last_became_active":"2023-07-29T16:12:21.350581+0000","last_became_peered":"2023-07-29T16:12:21.350581+0000","last_unstale":"2023-07-29T16:13:49.829981+0000","last_undegraded":"2023-07-29T16:13:49.829981+0000","last_fullsized":"2023-07-29T16:13:49.829981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:49:20.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831508+0000","last_change":"2023-07-29T16:12:21.941698+0000","last_active":"2023-07-29T16:13:49.831508+0000","last_peered":"2023-07-29T16:13:49.831508+0000","last_clean":"2023-07-29T16:13:49.831508+0000","last_became_active":"2023-07-29T16:12:21.941364+0000","last_became_peered":"2023-07-29T16:12:21.941364+0000","last_unstale":"2023-07-29T16:13:49.831508+0000","last_undegraded":"2023-07-29T16:13:49.831508+0000","last_fullsized":"2023-07-29T16:13:49.831508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:25:29.640753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:52.562037+0000","last_change":"2023-07-29T16:12:21.337554+0000","last_active":"2023-07-29T16:13:52.562037+0000","last_peered":"2023-07-29T16:13:52.562037+0000","last_clean":"2023-07-29T16:13:52.562037+0000","last_became_active":"2023-07-29T16:12:21.337214+0000","last_became_peered":"2023-07-29T16:12:21.337214+0000","last_unstale":"2023-07-29T16:13:52.562037+0000","last_undegraded":"2023-07-29T16:13:52.562037+0000","last_fullsized":"2023-07-29T16:13:52.562037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:09:38.562994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830712+0000","last_change":"2023-07-29T16:12:21.937377+0000","last_active":"2023-07-29T16:13:49.830712+0000","last_peered":"2023-07-29T16:13:49.830712+0000","last_clean":"2023-07-29T16:13:49.830712+0000","last_became_active":"2023-07-29T16:12:21.936879+0000","last_became_peered":"2023-07-29T16:12:21.936879+0000","last_unstale":"2023-07-29T16:13:49.830712+0000","last_undegraded":"2023-07-29T16:13:49.830712+0000","last_fullsized":"2023-07-29T16:13:49.830712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:23:04.958441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830267+0000","last_change":"2023-07-29T16:12:21.345520+0000","last_active":"2023-07-29T16:13:49.830267+0000","last_peered":"2023-07-29T16:13:49.830267+0000","last_clean":"2023-07-29T16:13:49.830267+0000","last_became_active":"2023-07-29T16:12:21.345090+0000","last_became_peered":"2023-07-29T16:12:21.345090+0000","last_unstale":"2023-07-29T16:13:49.830267+0000","last_undegraded":"2023-07-29T16:13:49.830267+0000","last_fullsized":"2023-07-29T16:13:49.830267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:13:19.262103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776129+0000","last_change":"2023-07-29T16:12:21.940090+0000","last_active":"2023-07-29T16:13:49.776129+0000","last_peered":"2023-07-29T16:13:49.776129+0000","last_clean":"2023-07-29T16:13:49.776129+0000","last_became_active":"2023-07-29T16:12:21.939742+0000","last_became_peered":"2023-07-29T16:12:21.939742+0000","last_unstale":"2023-07-29T16:13:49.776129+0000","last_undegraded":"2023-07-29T16:13:49.776129+0000","last_fullsized":"2023-07-29T16:13:49.776129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:29:59.332484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778725+0000","last_change":"2023-07-29T16:12:21.346578+0000","last_active":"2023-07-29T16:13:49.778725+0000","last_peered":"2023-07-29T16:13:49.778725+0000","last_clean":"2023-07-29T16:13:49.778725+0000","last_became_active":"2023-07-29T16:12:21.346253+0000","last_became_peered":"2023-07-29T16:12:21.346253+0000","last_unstale":"2023-07-29T16:13:49.778725+0000","last_undegraded":"2023-07-29T16:13:49.778725+0000","last_fullsized":"2023-07-29T16:13:49.778725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:45:41.774046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829527+0000","last_change":"2023-07-29T16:12:21.941886+0000","last_active":"2023-07-29T16:13:49.829527+0000","last_peered":"2023-07-29T16:13:49.829527+0000","last_clean":"2023-07-29T16:13:49.829527+0000","last_became_active":"2023-07-29T16:12:21.941539+0000","last_became_peered":"2023-07-29T16:12:21.941539+0000","last_unstale":"2023-07-29T16:13:49.829527+0000","last_undegraded":"2023-07-29T16:13:49.829527+0000","last_fullsized":"2023-07-29T16:13:49.829527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:15:12.851028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830174+0000","last_change":"2023-07-29T16:12:21.347150+0000","last_active":"2023-07-29T16:13:49.830174+0000","last_peered":"2023-07-29T16:13:49.830174+0000","last_clean":"2023-07-29T16:13:49.830174+0000","last_became_active":"2023-07-29T16:12:21.346805+0000","last_became_peered":"2023-07-29T16:12:21.346805+0000","last_unstale":"2023-07-29T16:13:49.830174+0000","last_undegraded":"2023-07-29T16:13:49.830174+0000","last_fullsized":"2023-07-29T16:13:49.830174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:46:28.285661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776966+0000","last_change":"2023-07-29T16:12:21.930143+0000","last_active":"2023-07-29T16:13:49.776966+0000","last_peered":"2023-07-29T16:13:49.776966+0000","last_clean":"2023-07-29T16:13:49.776966+0000","last_became_active":"2023-07-29T16:12:21.929396+0000","last_became_peered":"2023-07-29T16:12:21.929396+0000","last_unstale":"2023-07-29T16:13:49.776966+0000","last_undegraded":"2023-07-29T16:13:49.776966+0000","last_fullsized":"2023-07-29T16:13:49.776966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:04:47.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"90'2","reported_seq":18,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-07-29T16:13:50.781469+0000","last_change":"2023-07-29T16:12:21.341006+0000","last_active":"2023-07-29T16:13:50.781469+0000","last_peered":"2023-07-29T16:13:50.781469+0000","last_clean":"2023-07-29T16:13:50.781469+0000","last_became_active":"2023-07-29T16:12:21.340768+0000","last_became_peered":"2023-07-29T16:12:21.340768+0000","last_unstale":"2023-07-29T16:13:50.781469+0000","last_undegraded":"2023-07-29T16:13:50.781469+0000","last_fullsized":"2023-07-29T16:13:50.781469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:19:26.716396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830383+0000","last_change":"2023-07-29T16:12:21.942168+0000","last_active":"2023-07-29T16:13:49.830383+0000","last_peered":"2023-07-29T16:13:49.830383+0000","last_clean":"2023-07-29T16:13:49.830383+0000","last_became_active":"2023-07-29T16:12:21.941626+0000","last_became_peered":"2023-07-29T16:12:21.941626+0000","last_unstale":"2023-07-29T16:13:49.830383+0000","last_undegraded":"2023-07-29T16:13:49.830383+0000","last_fullsized":"2023-07-29T16:13:49.830383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:42:21.866896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.829662+0000","last_change":"2023-07-29T16:12:21.344010+0000","last_active":"2023-07-29T16:13:49.829662+0000","last_peered":"2023-07-29T16:13:49.829662+0000","last_clean":"2023-07-29T16:13:49.829662+0000","last_became_active":"2023-07-29T16:12:21.343676+0000","last_became_peered":"2023-07-29T16:12:21.343676+0000","last_unstale":"2023-07-29T16:13:49.829662+0000","last_undegraded":"2023-07-29T16:13:49.829662+0000","last_fullsized":"2023-07-29T16:13:49.829662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:27.397385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775210+0000","last_change":"2023-07-29T16:12:21.947189+0000","last_active":"2023-07-29T16:13:49.775210+0000","last_peered":"2023-07-29T16:13:49.775210+0000","last_clean":"2023-07-29T16:13:49.775210+0000","last_became_active":"2023-07-29T16:12:21.946700+0000","last_became_peered":"2023-07-29T16:12:21.946700+0000","last_unstale":"2023-07-29T16:13:49.775210+0000","last_undegraded":"2023-07-29T16:13:49.775210+0000","last_fullsized":"2023-07-29T16:13:49.775210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:42:26.302977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830449+0000","last_change":"2023-07-29T16:12:50.296260+0000","last_active":"2023-07-29T16:13:49.830449+0000","last_peered":"2023-07-29T16:13:49.830449+0000","last_clean":"2023-07-29T16:13:49.830449+0000","last_became_active":"2023-07-29T16:12:50.290108+0000","last_became_peered":"2023-07-29T16:12:50.290108+0000","last_unstale":"2023-07-29T16:13:49.830449+0000","last_undegraded":"2023-07-29T16:13:49.830449+0000","last_fullsized":"2023-07-29T16:13:49.830449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:52:17.142674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775483+0000","last_change":"2023-07-29T16:12:21.946008+0000","last_active":"2023-07-29T16:13:49.775483+0000","last_peered":"2023-07-29T16:13:49.775483+0000","last_clean":"2023-07-29T16:13:49.775483+0000","last_became_active":"2023-07-29T16:12:21.945442+0000","last_became_peered":"2023-07-29T16:12:21.945442+0000","last_unstale":"2023-07-29T16:13:49.775483+0000","last_undegraded":"2023-07-29T16:13:49.775483+0000","last_fullsized":"2023-07-29T16:13:49.775483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T02:16:02.866279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778871+0000","last_change":"2023-07-29T16:12:21.346374+0000","last_active":"2023-07-29T16:13:49.778871+0000","last_peered":"2023-07-29T16:13:49.778871+0000","last_clean":"2023-07-29T16:13:49.778871+0000","last_became_active":"2023-07-29T16:12:21.345732+0000","last_became_peered":"2023-07-29T16:12:21.345732+0000","last_unstale":"2023-07-29T16:13:49.778871+0000","last_undegraded":"2023-07-29T16:13:49.778871+0000","last_fullsized":"2023-07-29T16:13:49.778871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:38:43.300010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775173+0000","last_change":"2023-07-29T16:12:21.945518+0000","last_active":"2023-07-29T16:13:49.775173+0000","last_peered":"2023-07-29T16:13:49.775173+0000","last_clean":"2023-07-29T16:13:49.775173+0000","last_became_active":"2023-07-29T16:12:21.945209+0000","last_became_peered":"2023-07-29T16:12:21.945209+0000","last_unstale":"2023-07-29T16:13:49.775173+0000","last_undegraded":"2023-07-29T16:13:49.775173+0000","last_fullsized":"2023-07-29T16:13:49.775173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T19:03:18.238019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":18,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830173+0000","last_change":"2023-07-29T16:12:21.354123+0000","last_active":"2023-07-29T16:13:49.830173+0000","last_peered":"2023-07-29T16:13:49.830173+0000","last_clean":"2023-07-29T16:13:49.830173+0000","last_became_active":"2023-07-29T16:12:21.353770+0000","last_became_peered":"2023-07-29T16:12:21.353770+0000","last_unstale":"2023-07-29T16:13:49.830173+0000","last_undegraded":"2023-07-29T16:13:49.830173+0000","last_fullsized":"2023-07-29T16:13:49.830173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:20.274704+0000","last_clean_scrub_stamp":"2023-07-29T16:12:20.274704+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T17:17:01.003485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.775487+0000","last_change":"2023-07-29T16:12:21.934883+0000","last_active":"2023-07-29T16:13:49.775487+0000","last_peered":"2023-07-29T16:13:49.775487+0000","last_clean":"2023-07-29T16:13:49.775487+0000","last_became_active":"2023-07-29T16:12:21.934551+0000","last_became_peered":"2023-07-29T16:12:21.934551+0000","last_unstale":"2023-07-29T16:13:49.775487+0000","last_undegraded":"2023-07-29T16:13:49.775487+0000","last_fullsized":"2023-07-29T16:13:49.775487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T20:12:21.472700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777770+0000","last_change":"2023-07-29T16:12:21.930331+0000","last_active":"2023-07-29T16:13:49.777770+0000","last_peered":"2023-07-29T16:13:49.777770+0000","last_clean":"2023-07-29T16:13:49.777770+0000","last_became_active":"2023-07-29T16:12:21.929828+0000","last_became_peered":"2023-07-29T16:12:21.929828+0000","last_unstale":"2023-07-29T16:13:49.777770+0000","last_undegraded":"2023-07-29T16:13:49.777770+0000","last_fullsized":"2023-07-29T16:13:49.777770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T00:55:35.819576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774427+0000","last_change":"2023-07-29T16:12:21.939306+0000","last_active":"2023-07-29T16:13:49.774427+0000","last_peered":"2023-07-29T16:13:49.774427+0000","last_clean":"2023-07-29T16:13:49.774427+0000","last_became_active":"2023-07-29T16:12:21.939008+0000","last_became_peered":"2023-07-29T16:12:21.939008+0000","last_unstale":"2023-07-29T16:13:49.774427+0000","last_undegraded":"2023-07-29T16:13:49.774427+0000","last_fullsized":"2023-07-29T16:13:49.774427+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T18:44:17.098765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774317+0000","last_change":"2023-07-29T16:12:21.925266+0000","last_active":"2023-07-29T16:13:49.774317+0000","last_peered":"2023-07-29T16:13:49.774317+0000","last_clean":"2023-07-29T16:13:49.774317+0000","last_became_active":"2023-07-29T16:12:21.925070+0000","last_became_peered":"2023-07-29T16:12:21.925070+0000","last_unstale":"2023-07-29T16:13:49.774317+0000","last_undegraded":"2023-07-29T16:13:49.774317+0000","last_fullsized":"2023-07-29T16:13:49.774317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:01:47.848182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.776020+0000","last_change":"2023-07-29T16:12:21.949011+0000","last_active":"2023-07-29T16:13:49.776020+0000","last_peered":"2023-07-29T16:13:49.776020+0000","last_clean":"2023-07-29T16:13:49.776020+0000","last_became_active":"2023-07-29T16:12:21.948664+0000","last_became_peered":"2023-07-29T16:12:21.948664+0000","last_unstale":"2023-07-29T16:13:49.776020+0000","last_undegraded":"2023-07-29T16:13:49.776020+0000","last_fullsized":"2023-07-29T16:13:49.776020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:38:03.744386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.831492+0000","last_change":"2023-07-29T16:12:21.946982+0000","last_active":"2023-07-29T16:13:49.831492+0000","last_peered":"2023-07-29T16:13:49.831492+0000","last_clean":"2023-07-29T16:13:49.831492+0000","last_became_active":"2023-07-29T16:12:21.946660+0000","last_became_peered":"2023-07-29T16:12:21.946660+0000","last_unstale":"2023-07-29T16:13:49.831492+0000","last_undegraded":"2023-07-29T16:13:49.831492+0000","last_fullsized":"2023-07-29T16:13:49.831492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:24:41.478585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.778112+0000","last_change":"2023-07-29T16:12:21.928346+0000","last_active":"2023-07-29T16:13:49.778112+0000","last_peered":"2023-07-29T16:13:49.778112+0000","last_clean":"2023-07-29T16:13:49.778112+0000","last_became_active":"2023-07-29T16:12:21.928016+0000","last_became_peered":"2023-07-29T16:12:21.928016+0000","last_unstale":"2023-07-29T16:13:49.778112+0000","last_undegraded":"2023-07-29T16:13:49.778112+0000","last_fullsized":"2023-07-29T16:13:49.778112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T16:22:49.655459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.774750+0000","last_change":"2023-07-29T16:12:21.929612+0000","last_active":"2023-07-29T16:13:49.774750+0000","last_peered":"2023-07-29T16:13:49.774750+0000","last_clean":"2023-07-29T16:13:49.774750+0000","last_became_active":"2023-07-29T16:12:21.929180+0000","last_became_peered":"2023-07-29T16:12:21.929180+0000","last_unstale":"2023-07-29T16:13:49.774750+0000","last_undegraded":"2023-07-29T16:13:49.774750+0000","last_fullsized":"2023-07-29T16:13:49.774750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:41:08.027613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830666+0000","last_change":"2023-07-29T16:12:21.947360+0000","last_active":"2023-07-29T16:13:49.830666+0000","last_peered":"2023-07-29T16:13:49.830666+0000","last_clean":"2023-07-29T16:13:49.830666+0000","last_became_active":"2023-07-29T16:12:21.947036+0000","last_became_peered":"2023-07-29T16:12:21.947036+0000","last_unstale":"2023-07-29T16:13:49.830666+0000","last_undegraded":"2023-07-29T16:13:49.830666+0000","last_fullsized":"2023-07-29T16:13:49.830666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:05:42.212264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830961+0000","last_change":"2023-07-29T16:12:50.285320+0000","last_active":"2023-07-29T16:13:49.830961+0000","last_peered":"2023-07-29T16:13:49.830961+0000","last_clean":"2023-07-29T16:13:49.830961+0000","last_became_active":"2023-07-29T16:12:50.284666+0000","last_became_peered":"2023-07-29T16:12:50.284666+0000","last_unstale":"2023-07-29T16:13:49.830961+0000","last_undegraded":"2023-07-29T16:13:49.830961+0000","last_fullsized":"2023-07-29T16:13:49.830961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T23:19:09.408935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830254+0000","last_change":"2023-07-29T16:12:21.929910+0000","last_active":"2023-07-29T16:13:49.830254+0000","last_peered":"2023-07-29T16:13:49.830254+0000","last_clean":"2023-07-29T16:13:49.830254+0000","last_became_active":"2023-07-29T16:12:21.929470+0000","last_became_peered":"2023-07-29T16:12:21.929470+0000","last_unstale":"2023-07-29T16:13:49.830254+0000","last_undegraded":"2023-07-29T16:13:49.830254+0000","last_fullsized":"2023-07-29T16:13:49.830254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T03:02:16.120801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830507+0000","last_change":"2023-07-29T16:12:50.335842+0000","last_active":"2023-07-29T16:13:49.830507+0000","last_peered":"2023-07-29T16:13:49.830507+0000","last_clean":"2023-07-29T16:13:49.830507+0000","last_became_active":"2023-07-29T16:12:50.335527+0000","last_became_peered":"2023-07-29T16:12:50.335527+0000","last_unstale":"2023-07-29T16:13:49.830507+0000","last_undegraded":"2023-07-29T16:13:49.830507+0000","last_fullsized":"2023-07-29T16:13:49.830507+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T22:35:18.062454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.830582+0000","last_change":"2023-07-29T16:12:21.953412+0000","last_active":"2023-07-29T16:13:49.830582+0000","last_peered":"2023-07-29T16:13:49.830582+0000","last_clean":"2023-07-29T16:13:49.830582+0000","last_became_active":"2023-07-29T16:12:21.952959+0000","last_became_peered":"2023-07-29T16:12:21.952959+0000","last_unstale":"2023-07-29T16:13:49.830582+0000","last_undegraded":"2023-07-29T16:13:49.830582+0000","last_fullsized":"2023-07-29T16:13:49.830582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-30T21:38:37.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-07-29T16:13:49.777983+0000","last_change":"2023-07-29T16:12:21.943567+0000","last_active":"2023-07-29T16:13:49.777983+0000","last_peered":"2023-07-29T16:13:49.777983+0000","last_clean":"2023-07-29T16:13:49.777983+0000","last_became_active":"2023-07-29T16:12:21.943223+0000","last_became_peered":"2023-07-29T16:12:21.943223+0000","last_unstale":"2023-07-29T16:13:49.777983+0000","last_undegraded":"2023-07-29T16:13:49.777983+0000","last_fullsized":"2023-07-29T16:13:49.777983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-29T16:12:21.311693+0000","last_clean_scrub_stamp":"2023-07-29T16:12:21.311693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-31T01:32:23.052063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66675,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":568,"num_read_kb":354,"num_write":164,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1684,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":199709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":999424,"data_stored":478732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":4.2050000000000001,"5min":4.2050000000000001,"15min":4.2050000000000001},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.48099999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.85799999999999998,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.42299999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.82099999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.87}]},{"osd":2,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.47499999999999998,"15min":0.46700000000000003},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.501,"5min":0.47299999999999998,"15min":0.46800000000000003},"min":{"1min":0.38300000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73399999999999999,"5min":0.755,"15min":0.755},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.64900000000000002,"15min":0.64000000000000001},"min":{"1min":0.61399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94099999999999995,"5min":0.996,"15min":0.996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.66200000000000003,"15min":0.66400000000000003},"min":{"1min":0.46999999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.91100000000000003,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.80500000000000005}]},{"osd":5,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67300000000000004,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.45200000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.80800000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":0.77000000000000002,"15min":0.73199999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.752,"15min":0.73699999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.9409999999999998,"5min":3.9409999999999998,"15min":3.9409999999999998},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.745,"15min":0.73999999999999999},"min":{"1min":0.65600000000000003,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.002,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.74199999999999999,"15min":0.74399999999999999},"min":{"1min":0.59299999999999997,"5min":0.495,"15min":0.495},"max":{"1min":0.88600000000000001,"5min":1.323,"15min":1.323},"last":0.92900000000000005}]},{"osd":8,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77300000000000002}]},{"osd":9,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.82299999999999995,"15min":0.80400000000000005},"min":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":3.7570000000000001,"5min":3.7570000000000001,"15min":3.7570000000000001},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":0.83299999999999996,"15min":0.80600000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.8319999999999999,"5min":3.8319999999999999,"15min":3.8319999999999999},"last":0.85199999999999998}]},{"osd":10,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.016,"5min":1.373,"15min":1.373},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.81399999999999995,"15min":0.81799999999999995},"min":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":0.95399999999999996,"5min":1.867,"15min":1.867},"last":0.88300000000000001}]}]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28592,"kb_used_data":1640,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716560,"statfs":{"total":95995035648,"available":95965757440,"internally_reserved":0,"allocated":1679360,"data_stored":1126469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.433}]},{"osd":1,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.30099999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72599999999999998,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.379}]},{"osd":2,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.624,"15min":0.624},"min":{"1min":0.50600000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80900000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.47299999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.754,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71899999999999997}]},{"osd":3,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.44,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82999999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.65000000000000002,"15min":0.65600000000000003},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60399999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39900000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82999999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.86099999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.53600000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.73099999999999998},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.95999999999999996,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.754},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74199999999999999,"15min":0.748},"min":{"1min":0.433,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.98399999999999999,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.64000000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.47599999999999998,"5min":0.443,"15min":0.443},"max":{"1min":0.87,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76600000000000001,"15min":0.77500000000000002},"min":{"1min":0.54800000000000004,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89600000000000002,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.58899999999999997,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.96899999999999997,"5min":1.319,"15min":1.319},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.82399999999999995,"15min":0.83999999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.0349999999999999,"5min":1.494,"15min":1.494},"last":0.86499999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.81899999999999995,"15min":0.82499999999999996},"min":{"1min":0.61499999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.98299999999999998,"5min":1.419,"15min":1.419},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.85099999999999998,"15min":0.86499999999999999},"min":{"1min":0.51200000000000001,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.995,"5min":1.5369999999999999,"15min":1.5369999999999999},"last":0.80100000000000005}]},{"osd":11,"last update":"Sat Jul 29 16:13:44 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.01,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.80000000000000004,"15min":0.79600000000000004},"min":{"1min":0.68799999999999994,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":0.997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.89700000000000002}]}]},{"osd":9,"up_from":68,"seq":292057776168,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":1007616,"data_stored":477550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.53400000000000003,"15min":0.54900000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.69899999999999995,"15min":0.71699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.71999999999999997,"5min":1.728,"15min":1.728},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.65500000000000003,"15min":0.66800000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0029999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81399999999999995,"5min":1.125,"15min":1.125},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.68700000000000006,"15min":0.69899999999999995},"min":{"1min":0.502,"5min":0.435,"15min":0.435},"max":{"1min":0.91700000000000004,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.48599999999999999}]},{"osd":3,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.80600000000000005,"15min":0.83399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.92100000000000004,"5min":1.974,"15min":1.974},"last":0.63},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.76100000000000001,"15min":0.80200000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.79200000000000004}]},{"osd":4,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.752,"15min":0.76600000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.91500000000000004,"15min":0.94599999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.95899999999999996,"5min":1.6579999999999999,"15min":1.6579999999999999},"last":0.85499999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.94699999999999995,"15min":0.97599999999999998},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.0600000000000001,"5min":1.8740000000000001,"15min":1.8740000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.88700000000000001,"15min":0.91800000000000004},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88100000000000001,"5min":1.4390000000000001,"15min":1.4390000000000001},"last":0.5}]},{"osd":6,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.92800000000000005,"15min":0.96299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.95699999999999996,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.84099999999999997,"15min":0.86599999999999999},"min":{"1min":0.53900000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.89000000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.56499999999999995}]},{"osd":7,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.109,"5min":1.839,"15min":1.839},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.94299999999999995,"15min":0.97099999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0569999999999999,"5min":1.47,"15min":1.47},"last":0.82399999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.91700000000000004,"15min":0.94199999999999995},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0169999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.83499999999999996,"5min":1.004,"15min":1.032},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.1220000000000001,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.78400000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.94599999999999995,"15min":0.96599999999999997},"min":{"1min":0.73199999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.038,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.91700000000000004,"15min":0.93100000000000005},"min":{"1min":0.63500000000000001,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":0.873}]}]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":1003520,"data_stored":477139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85499999999999998,"5min":2.0680000000000001,"15min":2.0680000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.83599999999999997,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Jul 29 16:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.61599999999999999,"15min":0.621},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46400000000000002,"15min":0.47399999999999998},"min":{"1min":0.32800000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.63600000000000001,"5min":1.052,"15min":1.052},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.496,"15min":0.5},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72899999999999998,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.69099999999999995,"15min":0.65100000000000002},"min":{"1min":0.44600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":6.9240000000000004,"5min":6.9240000000000004,"15min":6.9240000000000004},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.624,"15min":0.63},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82299999999999995,"5min":1.7250000000000001,"15min":1.7250000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.628,"15min":0.63},"min":{"1min":0.51100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85899999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79100000000000004,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58299999999999996,"15min":0.60299999999999998},"min":{"1min":0.32500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.5}]},{"osd":6,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.72399999999999998,"15min":0.74099999999999999},"min":{"1min":0.45900000000000002,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.95499999999999996,"5min":2.1760000000000002,"15min":2.1760000000000002},"last":0.622},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.73499999999999999,"15min":0.753},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.93300000000000005,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58999999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:28 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.73699999999999999,"15min":0.74099999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0369999999999999,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73699999999999999,"15min":0.749},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.65400000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:57 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.58299999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.751,"15min":0.747},"min":{"1min":0.56899999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.748}]},{"osd":10,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.73199999999999998,"15min":0.71999999999999997},"min":{"1min":0.66100000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72199999999999998,"15min":0.70899999999999996},"min":{"1min":0.61499999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.70199999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.78200000000000003}]}]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28512,"kb_used_data":1560,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716640,"statfs":{"total":95995035648,"available":95965839360,"internally_reserved":0,"allocated":1597440,"data_stored":1069644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:12:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57199999999999995,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1399999999999997,"5min":4.1399999999999997,"15min":4.1399999999999997},"last":0.55800000000000005}]},{"osd":1,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.42299999999999999,"15min":0.40100000000000002},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.72199999999999998,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.44600000000000001,"15min":0.434},"min":{"1min":0.317,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68000000000000005,"5min":0.755,"15min":0.755},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.61899999999999999,"15min":0.60499999999999998},"min":{"1min":0.49399999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.626,"15min":0.61499999999999999},"min":{"1min":0.51500000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58299999999999996}]},{"osd":3,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.59899999999999998,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.39600000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94799999999999995,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.61299999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.53600000000000003},"min":{"1min":0.434,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.74299999999999999,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52800000000000002,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81499999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.495}]},{"osd":5,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70799999999999996,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90300000000000002,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.66000000000000003,"15min":0.64400000000000002},"min":{"1min":0.52900000000000003,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.495,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.167,"5min":1.167,"15min":1.167},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.73199999999999998,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.94099999999999995,"5min":4.2060000000000004,"15min":4.2060000000000004},"last":0.72199999999999998}]},{"osd":8,"last update":"Sat Jul 29 16:13:34 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.72699999999999998,"15min":0.72099999999999997},"min":{"1min":0.622,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89100000000000001,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.755},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.72599999999999998,"15min":0.70699999999999996},"min":{"1min":0.60099999999999998,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93899999999999995,"5min":1.079,"15min":1.079},"last":0.76400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.55100000000000005,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77400000000000002,"15min":0.77200000000000002},"min":{"1min":0.58299999999999996,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.74199999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:13:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79000000000000004,"15min":0.80100000000000005},"min":{"1min":0.53100000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98599999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.754,"15min":0.75},"min":{"1min":0.55900000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77700000000000002}]},{"osd":11,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78900000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":995328,"data_stored":476945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.51800000000000002,"15min":0.48999999999999999},"min":{"1min":0.35399999999999998,"5min":0.317,"15min":0.317},"max":{"1min":1.6779999999999999,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55100000000000005,"15min":0.51300000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.80900000000000005,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.62,"15min":0.60299999999999998},"min":{"1min":0.44800000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.872,"5min":1.379,"15min":1.379},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67000000000000004,"15min":0.67300000000000004},"min":{"1min":0.47799999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.2010000000000001,"5min":1.486,"15min":1.486},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71699999999999997,"15min":0.69499999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.97499999999999998,"5min":1.117,"15min":1.117},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69199999999999995,"15min":0.67100000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.88200000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.64000000000000001,"15min":0.61099999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.6100000000000001,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.67000000000000004,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.5589999999999999,"5min":2.25,"15min":2.25},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.72699999999999998,"15min":0.71099999999999997},"min":{"1min":0.54700000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0349999999999999,"5min":1.087,"15min":1.087},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.747,"15min":0.71899999999999997},"min":{"1min":0.46500000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.0269999999999999,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":0.77700000000000002}]},{"osd":5,"last update":"Sat Jul 29 16:13:50 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.52400000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.0589999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80200000000000005,"15min":0.78300000000000003},"min":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.92800000000000005}]},{"osd":7,"last update":"Sat Jul 29 16:13:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.77900000000000003,"15min":0.75900000000000001},"min":{"1min":0.56799999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.79800000000000004,"15min":0.77300000000000002},"min":{"1min":0.68999999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.96499999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:13:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.89500000000000002,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":7.2009999999999996,"5min":7.2009999999999996,"15min":7.2009999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.87,"15min":0.86599999999999999},"min":{"1min":0.60499999999999998,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.0069999999999999}]},{"osd":9,"last update":"Sat Jul 29 16:13:57 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.92400000000000004,"15min":0.94099999999999995},"min":{"1min":0.53800000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0780000000000001,"5min":2.165,"15min":2.165},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.91200000000000003,"15min":0.94199999999999995},"min":{"1min":0.54300000000000004,"5min":0.503,"15min":0.503},"max":{"1min":1.244,"5min":2.331,"15min":2.331},"last":0.70799999999999996}]},{"osd":10,"last update":"Sat Jul 29 16:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.76300000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.256,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.0089999999999999,"15min":0.95899999999999996},"min":{"1min":0.72599999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":8.1600000000000001,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.83399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":1060864,"data_stored":535619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:54 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56599999999999995,"15min":0.505},"min":{"1min":0.39500000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80700000000000005,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56799999999999995,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.34300000000000003,"15min":0.33000000000000002},"max":{"1min":0.73999999999999999,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.56899999999999995,"15min":0.53300000000000003},"min":{"1min":0.42399999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.57499999999999996,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.65900000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:12:37 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.47899999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58399999999999996,"15min":0.55600000000000005},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.61499999999999999}]},{"osd":4,"last update":"Sat Jul 29 16:13:05 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.42699999999999999},"min":{"1min":0.27900000000000003,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81200000000000006,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.501,"15min":0.46899999999999997},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.436}]},{"osd":5,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.629,"15min":0.57999999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.66000000000000003,"15min":0.61099999999999999},"min":{"1min":0.51900000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95099999999999996,"5min":1.02,"15min":1.02},"last":0.86799999999999999}]},{"osd":6,"last update":"Sat Jul 29 16:13:45 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69399999999999995,"15min":0.67600000000000005},"min":{"1min":0.46999999999999997,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.90500000000000003,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.67000000000000004,"15min":0.66400000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.90600000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.621}]},{"osd":7,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.62,"15min":0.58999999999999997},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96599999999999997,"5min":1.073,"15min":1.073},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.67600000000000005,"15min":0.64200000000000002},"min":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69399999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.754,"15min":0.75},"min":{"1min":0.57499999999999996,"5min":0.376,"15min":0.376},"max":{"1min":1.0920000000000001,"5min":1.101,"15min":1.101},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.77000000000000002,"15min":0.76100000000000001},"min":{"1min":0.48899999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.02,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.90600000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:13:54 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.85399999999999998},"min":{"1min":0.63400000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.046,"5min":1.161,"15min":1.161},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.84299999999999997,"15min":0.85699999999999998},"min":{"1min":0.622,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.98099999999999998,"5min":1.175,"15min":1.175},"last":0.84199999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:13:17 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.80700000000000005,"15min":0.81899999999999995},"min":{"1min":0.42799999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90300000000000002,"5min":1.129,"15min":1.129},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.81299999999999994,"15min":0.81899999999999995},"min":{"1min":0.52300000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.995,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.71599999999999997}]},{"osd":11,"last update":"Sat Jul 29 16:13:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.78900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":1015808,"data_stored":478210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.56599999999999995,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.59199999999999997,"15min":0.54500000000000004},"min":{"1min":0.39400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.93799999999999994,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.43099999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60999999999999999,"15min":0.58399999999999996},"min":{"1min":0.496,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63300000000000001,"15min":0.60099999999999998},"min":{"1min":0.42999999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.90700000000000003,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.45100000000000001}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63,"15min":0.58999999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.85999999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60599999999999998,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84799999999999998,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57099999999999995}]},{"osd":5,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.68200000000000005,"15min":0.66100000000000003},"min":{"1min":0.50800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94399999999999995,"5min":1.111,"15min":1.111},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67000000000000004,"15min":0.64600000000000002},"min":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.91800000000000004,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Jul 29 16:13:52 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.43099999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.264,"5min":1.266,"15min":1.266},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.74099999999999999},"min":{"1min":0.435,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.288,"5min":1.288,"15min":1.288},"last":0.64000000000000001}]},{"osd":7,"last update":"Sat Jul 29 16:13:52 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.67200000000000004}]},{"osd":8,"last update":"Sat Jul 29 16:13:30 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.84299999999999997,"15min":0.88300000000000001},"min":{"1min":0.55200000000000005,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.056,"5min":1.335,"15min":1.335},"last":0.621},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.85599999999999998,"15min":0.89400000000000002},"min":{"1min":0.64100000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.91100000000000003,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.68799999999999994}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.86499999999999999,"15min":0.88300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.079,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.751},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.92100000000000004,"15min":0.94499999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.014,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.71899999999999997}]},{"osd":10,"last update":"Sat Jul 29 16:13:19 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84199999999999997,"15min":0.84399999999999997},"min":{"1min":0.66100000000000003,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.91700000000000004,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.85299999999999998,"15min":0.85699999999999998},"min":{"1min":0.64400000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.1160000000000001,"5min":1.212,"15min":1.212},"last":0.70599999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.83199999999999996,"15min":0.82199999999999995},"min":{"1min":0.63500000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.745},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.749,"5min":0.626,"15min":0.626},"max":{"1min":0.96799999999999997,"5min":0.999,"15min":0.999},"last":0.78500000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1069056,"data_stored":538667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.375,"5min":0.308,"15min":0.308},"max":{"1min":3.4169999999999998,"5min":3.4169999999999998,"15min":3.4169999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.60199999999999998,"15min":0.55200000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78000000000000003,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Jul 29 16:13:16 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58299999999999996,"15min":0.54600000000000004},"min":{"1min":0.40799999999999997,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78200000000000003,"5min":3.613,"15min":3.613},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.56899999999999995,"15min":0.54300000000000004},"min":{"1min":0.41499999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.628}]},{"osd":3,"last update":"Sat Jul 29 16:13:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.63200000000000001,"15min":0.55700000000000005},"min":{"1min":0.39500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":5.1509999999999998,"5min":5.1509999999999998,"15min":5.1509999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.60399999999999998,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":3.4550000000000001,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Sat Jul 29 16:13:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60599999999999998}]},{"osd":5,"last update":"Sat Jul 29 16:13:30 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.45600000000000002,"15min":0.442},"min":{"1min":0.39100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66900000000000004,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.439},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.497,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73799999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.67700000000000005}]},{"osd":6,"last update":"Sat Jul 29 16:13:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.66700000000000004,"15min":0.65500000000000003},"min":{"1min":0.51800000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.77900000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.753,"15min":0.66900000000000004},"min":{"1min":0.5,"5min":0.379,"15min":0.379},"max":{"1min":5.1139999999999999,"5min":7.1989999999999998,"15min":7.1989999999999998},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.86199999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.70799999999999996,"15min":0.71399999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":8,"last update":"Sat Jul 29 16:13:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.624,"15min":0.63600000000000001},"min":{"1min":0.44800000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.85499999999999998,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63700000000000001,"15min":0.64800000000000002},"min":{"1min":0.51000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.79300000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.749}]},{"osd":9,"last update":"Sat Jul 29 16:13:58 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76000000000000001,"15min":0.76400000000000001},"min":{"1min":0.56399999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.90400000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.81499999999999995,"15min":0.80400000000000005},"min":{"1min":0.67500000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":3.5950000000000002,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.89600000000000002}]},{"osd":10,"last update":"Sat Jul 29 16:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.56899999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85899999999999999,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.69699999999999995,"15min":0.68200000000000005},"min":{"1min":0.59699999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.82799999999999996}]},{"osd":11,"last update":"Sat Jul 29 16:13:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.85199999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":1007616,"data_stored":476931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.46800000000000003,"15min":0.41099999999999998},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.56699999999999995,"15min":0.46800000000000003},"min":{"1min":0.39500000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":8.766,"5min":8.766,"15min":8.766},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63200000000000001,"15min":0.58199999999999996},"min":{"1min":0.45500000000000002,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.88500000000000001,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.63500000000000001,"15min":0.57599999999999996},"min":{"1min":0.55100000000000005,"5min":0.372,"15min":0.372},"max":{"1min":0.81000000000000005,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:40 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.70199999999999996,"15min":0.63200000000000001},"min":{"1min":0.61399999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.69699999999999995,"15min":0.61699999999999999},"min":{"1min":0.56000000000000005,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1559999999999999,"5min":1.296,"15min":1.296},"last":0.60099999999999998}]},{"osd":4,"last update":"Sat Jul 29 16:13:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.64400000000000002,"15min":0.60699999999999998},"min":{"1min":0.51800000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88400000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.53400000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.98099999999999998,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.54000000000000004}]},{"osd":5,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71299999999999997,"15min":0.70099999999999996},"min":{"1min":0.57099999999999995,"5min":0.443,"15min":0.443},"max":{"1min":0.97799999999999998,"5min":1.343,"15min":1.343},"last":0.745},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72699999999999998,"15min":0.70099999999999996},"min":{"1min":0.53900000000000003,"5min":0.501,"15min":0.501},"max":{"1min":0.84099999999999997,"5min":1.032,"15min":1.032},"last":0.88300000000000001}]},{"osd":6,"last update":"Sat Jul 29 16:13:46 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.70099999999999996,"15min":0.68200000000000005},"min":{"1min":0.434,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.83299999999999996,"5min":1.276,"15min":1.276},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70199999999999996,"15min":0.66400000000000003},"min":{"1min":0.48499999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.58699999999999997}]},{"osd":7,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59699999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0800000000000001,"5min":1.323,"15min":1.323},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.83599999999999997,"15min":0.82799999999999996},"min":{"1min":0.60799999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96799999999999997,"5min":1.357,"15min":1.357},"last":0.999}]},{"osd":8,"last update":"Sat Jul 29 16:13:53 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":1.034},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.77900000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:56 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.90300000000000002,"15min":0.92300000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.091,"5min":1.476,"15min":1.476},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.90100000000000002,"15min":0.91800000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0509999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.78700000000000003}]},{"osd":10,"last update":"Sat Jul 29 16:13:18 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86799999999999999,"15min":0.87},"min":{"1min":0.74199999999999999,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.157,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90400000000000003,"15min":0.90200000000000002},"min":{"1min":0.71599999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.244,"5min":1.5349999999999999,"15min":1.5349999999999999},"last":0.872}]},{"osd":11,"last update":"Sat Jul 29 16:13:23 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.4850000000000001,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.93999999999999995}]}]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":1003520,"data_stored":478438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58299999999999996}]},{"osd":1,"last update":"Sat Jul 29 16:12:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.373,"15min":0.35899999999999999},"min":{"1min":0.29299999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.58599999999999997,"15min":0.58199999999999996},"min":{"1min":0.40500000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.59799999999999998,"15min":0.56599999999999995},"min":{"1min":0.50700000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Jul 29 16:13:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.60099999999999998,"15min":0.55600000000000005},"min":{"1min":0.45100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.93300000000000005,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.63900000000000001}]},{"osd":5,"last update":"Sat Jul 29 16:13:27 2023","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.65900000000000003,"15min":0.60299999999999998},"min":{"1min":0.50600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.65000000000000002,"15min":0.60899999999999999},"min":{"1min":0.54600000000000004,"5min":0.374,"15min":0.374},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.67000000000000004}]},{"osd":6,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.65200000000000002,"15min":0.628},"min":{"1min":0.46300000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94599999999999995,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.79100000000000004,"5min":1.018,"15min":1.018},"last":0.66000000000000003}]},{"osd":7,"last update":"Sat Jul 29 16:13:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.58699999999999997},"min":{"1min":0.39500000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.91700000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.749},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.627,"15min":0.628},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0129999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"last":0.60999999999999999}]},{"osd":8,"last update":"Sat Jul 29 16:13:32 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.55700000000000005,"5min":0.432,"15min":0.432},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.76700000000000002,"15min":0.753},"min":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.78700000000000003}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.79500000000000004,"15min":0.80000000000000004},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.04,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.754,"15min":0.748},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73999999999999999}]},{"osd":10,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.78300000000000003,"15min":0.79100000000000004},"min":{"1min":0.53200000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.0009999999999999,"5min":1.115,"15min":1.115},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.76600000000000001,"15min":0.77000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.72899999999999998}]},{"osd":11,"last update":"Sat Jul 29 16:13:21 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.80200000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28528,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":93716624,"statfs":{"total":95995035648,"available":95965822976,"internally_reserved":0,"allocated":1605632,"data_stored":1068077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.56599999999999995,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.38,"15min":0.38},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.497},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54200000000000004,"15min":0.53000000000000003},"min":{"1min":0.36899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.56499999999999995}]},{"osd":1,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.55100000000000005,"15min":0.51700000000000002},"min":{"1min":0.48999999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.746},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56599999999999995,"15min":0.53500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77200000000000002,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.42499999999999999,"15min":0.40699999999999997},"min":{"1min":0.34000000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.435,"15min":0.41499999999999998},"min":{"1min":0.373,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68899999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58299999999999996},"min":{"1min":0.45500000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60399999999999998,"15min":0.58399999999999996},"min":{"1min":0.48699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.52300000000000002}]},{"osd":4,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.61399999999999999,"15min":0.58199999999999996},"min":{"1min":0.59999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.77300000000000002}]},{"osd":6,"last update":"Sat Jul 29 16:13:51 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.64900000000000002,"15min":0.63200000000000001},"min":{"1min":0.54700000000000004,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68200000000000005,"15min":0.65600000000000003},"min":{"1min":0.55900000000000005,"5min":0.378,"15min":0.378},"max":{"1min":1.373,"5min":1.373,"15min":1.373},"last":0.58299999999999996}]},{"osd":7,"last update":"Sat Jul 29 16:13:14 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.63400000000000001,"15min":0.63100000000000001},"min":{"1min":0.53700000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.65700000000000003,"15min":0.64600000000000002},"min":{"1min":0.46500000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.81599999999999995}]},{"osd":8,"last update":"Sat Jul 29 16:13:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.621,"15min":0.61099999999999999},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82699999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.51400000000000001}]},{"osd":9,"last update":"Sat Jul 29 16:12:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.73299999999999998,"15min":0.73399999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74299999999999999,"15min":0.73799999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.622}]},{"osd":10,"last update":"Sat Jul 29 16:13:24 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.72799999999999998,"15min":0.71799999999999997},"min":{"1min":0.63200000000000001,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.73199999999999998,"15min":0.71899999999999997},"min":{"1min":0.65200000000000002,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.92000000000000004,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.78600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-29T16:14:02.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:14:03.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:02 smithi156 ceph-mon[129245]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:03.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:02 smithi156 ceph-mon[129245]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:03.082 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:02 smithi031 ceph-mon[95504]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:03.082 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:02 smithi031 ceph-mon[95504]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:03.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 -- 172.21.15.31:0/480223139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8100340 msgr2=0x7f13f8100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/480223139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8100340 0x7f13f8100750 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009990 tx=0x7f13e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 -- 172.21.15.31:0/480223139 shutdown_connections 2023-07-29T16:14:03.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/480223139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8101ec0 0x7f13f8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/480223139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8100340 0x7f13f8100750 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/480223139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101530 0x7f13f8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 -- 172.21.15.31:0/480223139 >> 172.21.15.31:0/480223139 conn(0x7f13f80fb920 msgr2=0x7f13f80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:03.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 -- 172.21.15.31:0/480223139 shutdown_connections 2023-07-29T16:14:03.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.227+0000 7f13fe7ff700 1 -- 172.21.15.31:0/480223139 wait complete. 2023-07-29T16:14:03.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 Processor -- start 2023-07-29T16:14:03.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 -- start start 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 0x7f13f8074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8101530 0x7f13f8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101ec0 0x7f13f8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f8073e20 con 0x7f13f8101530 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f8073f90 con 0x7f13f8100340 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.229+0000 7f13fe7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f81a00c0 con 0x7f13f8101ec0 2023-07-29T16:14:03.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 0x7f13f8074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8101530 0x7f13f8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13fcd9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101ec0 0x7f13f8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 0x7f13f8074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35090/0 (socket says 172.21.15.31:35090) 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 -- 172.21.15.31:0/4013984797 learned_addr learned my addr 172.21.15.31:0/4013984797 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 -- 172.21.15.31:0/4013984797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101ec0 msgr2=0x7f13f8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101ec0 0x7f13f8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 -- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8101530 msgr2=0x7f13f8073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8101530 0x7f13f8073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 -- 172.21.15.31:0/4013984797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec009520 con 0x7f13f8100340 2023-07-29T16:14:03.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13fcd9c700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101ec0 0x7f13f8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:03.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f77fe700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8101530 0x7f13f8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:03.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.230+0000 7f13f7fff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 0x7f13f8074b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13e8005740 tx=0x7f13e8005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:03.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.231+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e801d070 con 0x7f13f8100340 2023-07-29T16:14:03.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.231+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13e8021410 con 0x7f13f8100340 2023-07-29T16:14:03.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.231+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e80095f0 con 0x7f13f8100340 2023-07-29T16:14:03.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.231+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e8017650 con 0x7f13f8100340 2023-07-29T16:14:03.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.231+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f81a05d0 con 0x7f13f8100340 2023-07-29T16:14:03.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.232+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f8107830 con 0x7f13f8100340 2023-07-29T16:14:03.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.234+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f13e8021580 con 0x7f13f8100340 2023-07-29T16:14:03.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.235+0000 7f13f57fa700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e00a07e0 0x7f13e00a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.235+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13e80ccd10 con 0x7f13f8100340 2023-07-29T16:14:03.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.236+0000 7f13f77fe700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e00a07e0 0x7f13e00a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.236+0000 7f13f77fe700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e00a07e0 0x7f13e00a2c90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009fd0 tx=0x7f13ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:03.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.237+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f13e8096780 con 0x7f13f8100340 2023-07-29T16:14:03.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:02 smithi039 ceph-mon[128858]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:03.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:02 smithi039 ceph-mon[128858]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:03.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.448+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13f802ce00 con 0x7f13f8100340 2023-07-29T16:14:03.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.450+0000 7f13f57fa700 1 -- 172.21.15.31:0/4013984797 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f13e8099da0 con 0x7f13f8100340 2023-07-29T16:14:03.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:03.449 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:13:50.771629+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:14:03.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e00a07e0 msgr2=0x7f13e00a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e00a07e0 0x7f13e00a2c90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009fd0 tx=0x7f13ec009260 comp rx=0 tx=0).stop 2023-07-29T16:14:03.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 msgr2=0x7f13f8074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 0x7f13f8074b00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13e8005740 tx=0x7f13e8005770 comp rx=0 tx=0).stop 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 shutdown_connections 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e00a07e0 0x7f13e00a2c90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13f8101ec0 0x7f13f8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13f8100340 0x7f13f8074b00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 --2- 172.21.15.31:0/4013984797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13f8101530 0x7f13f8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 >> 172.21.15.31:0/4013984797 conn(0x7f13f80fb920 msgr2=0x7f13f8104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 shutdown_connections 2023-07-29T16:14:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.453+0000 7f13fe7ff700 1 -- 172.21.15.31:0/4013984797 wait complete. 2023-07-29T16:14:03.470 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:14:03.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:14:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2672535227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101510 msgr2=0x7f59f8101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2672535227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101510 0x7f59f8101960 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f59e8009990 tx=0x7f59e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2672535227 shutdown_connections 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2672535227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59f8101ea0 0x7f59f8108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2672535227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2672535227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101510 0x7f59f8101960 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2672535227 >> 172.21.15.31:0/2672535227 conn(0x7f59f80fb920 msgr2=0x7f59f80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2672535227 shutdown_connections 2023-07-29T16:14:03.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.679+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2672535227 wait complete. 2023-07-29T16:14:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.680+0000 7f59ff9ff700 1 Processor -- start 2023-07-29T16:14:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.680+0000 7f59ff9ff700 1 -- start start 2023-07-29T16:14:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59ff9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59ff9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59f8101510 0x7f59f81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59ff9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101ea0 0x7f59f819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59ff9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f819fe30 con 0x7f59f8101ea0 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59ff9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f819ff70 con 0x7f59f8100320 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59ff9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59f81a0100 con 0x7f59f8101510 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fd79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fd79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f81982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35106/0 (socket says 172.21.15.31:35106) 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fcf9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59f8101510 0x7f59f81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fd79b700 1 -- 172.21.15.31:0/2545253697 learned_addr learned my addr 172.21.15.31:0/2545253697 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fdf9c700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101ea0 0x7f59f819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fd79b700 1 -- 172.21.15.31:0/2545253697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59f8101510 msgr2=0x7f59f81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fd79b700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59f8101510 0x7f59f81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.681+0000 7f59fd79b700 1 -- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101ea0 msgr2=0x7f59f819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59fd79b700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101ea0 0x7f59f819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59fd79b700 1 -- 172.21.15.31:0/2545253697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e80095f0 con 0x7f59f8100320 2023-07-29T16:14:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59fd79b700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f81982b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f59f400e970 tx=0x7f59f400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4009790 con 0x7f59f8100320 2023-07-29T16:14:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59f400c7e0 con 0x7f59f8100320 2023-07-29T16:14:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f81a03e0 con 0x7f59f8100320 2023-07-29T16:14:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4003c70 con 0x7f59f8100320 2023-07-29T16:14:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.682+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f81a08f0 con 0x7f59f8100320 2023-07-29T16:14:03.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.683+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59f810bb50 con 0x7f59f8100320 2023-07-29T16:14:03.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.685+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f59f400cc50 con 0x7f59f8100320 2023-07-29T16:14:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.687+0000 7f59ee7fc700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59e40a0790 0x7f59e40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.687+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f59f4014070 con 0x7f59f8100320 2023-07-29T16:14:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.687+0000 7f59fcf9a700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59e40a0790 0x7f59e40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.688+0000 7f59fcf9a700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59e40a0790 0x7f59e40a2c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f59e800b2e0 tx=0x7f59e8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.689+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f59f40958f0 con 0x7f59f8100320 2023-07-29T16:14:03.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.897+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59f81a0f90 con 0x7f59f8100320 2023-07-29T16:14:03.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.898+0000 7f59ee7fc700 1 -- 172.21.15.31:0/2545253697 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f59f4003dd0 con 0x7f59f8100320 2023-07-29T16:14:03.897 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:14:03.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59e40a0790 msgr2=0x7f59e40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59e40a0790 0x7f59e40a2c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f59e800b2e0 tx=0x7f59e8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 msgr2=0x7f59f81982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:03.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f81982b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f59f400e970 tx=0x7f59f400ec80 comp rx=0 tx=0).stop 2023-07-29T16:14:03.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 shutdown_connections 2023-07-29T16:14:03.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59e40a0790 0x7f59e40a2c40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59f8101ea0 0x7f59f819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f59f8100320 0x7f59f81982b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 --2- 172.21.15.31:0/2545253697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f59f8101510 0x7f59f81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:03.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 >> 172.21.15.31:0/2545253697 conn(0x7f59f80fb920 msgr2=0x7f59f80fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:03.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 shutdown_connections 2023-07-29T16:14:03.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:03.901+0000 7f59ff9ff700 1 -- 172.21.15.31:0/2545253697 wait complete. 2023-07-29T16:14:03.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:14:04.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:03 smithi156 ceph-mon[129245]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:04.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:03 smithi156 ceph-mon[129245]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.3 KiB/s wr, 7 op/s; 91 B/s, 0 objects/s recovering 2023-07-29T16:14:04.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/4013984797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:04.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 -- 172.21.15.31:0/38874113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee70074d30 msgr2=0x7fee70073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 --2- 172.21.15.31:0/38874113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee70074d30 0x7fee70073180 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fee60009990 tx=0x7fee60009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 -- 172.21.15.31:0/38874113 shutdown_connections 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 --2- 172.21.15.31:0/38874113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee701081d0 0x7fee7010a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 --2- 172.21.15.31:0/38874113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee70074d30 0x7fee70073180 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 --2- 172.21.15.31:0/38874113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee70073750 0x7fee70073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 -- 172.21.15.31:0/38874113 >> 172.21.15.31:0/38874113 conn(0x7fee700fb960 msgr2=0x7fee700fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 -- 172.21.15.31:0/38874113 shutdown_connections 2023-07-29T16:14:04.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.127+0000 7fee753ff700 1 -- 172.21.15.31:0/38874113 wait complete. 2023-07-29T16:14:04.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.128+0000 7fee753ff700 1 Processor -- start 2023-07-29T16:14:04.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.128+0000 7fee753ff700 1 -- start start 2023-07-29T16:14:04.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee753ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee70073750 0x7fee701982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee753ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee70074d30 0x7fee701987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee753ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 0x7fee7019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee753ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7019fe20 con 0x7fee701081d0 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee753ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7019ff60 con 0x7fee70074d30 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee753ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee701a00f0 con 0x7fee70073750 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 0x7fee7019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 0x7fee7019f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43444/0 (socket says 172.21.15.31:43444) 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 -- 172.21.15.31:0/2506543961 learned_addr learned my addr 172.21.15.31:0/2506543961 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:04.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6effd700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee70073750 0x7fee701982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6e7fc700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee70074d30 0x7fee701987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 -- 172.21.15.31:0/2506543961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee70073750 msgr2=0x7fee701982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee70073750 0x7fee701982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 -- 172.21.15.31:0/2506543961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee70074d30 msgr2=0x7fee701987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee70074d30 0x7fee701987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.129+0000 7fee6f7fe700 1 -- 172.21.15.31:0/2506543961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600095f0 con 0x7fee701081d0 2023-07-29T16:14:04.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.130+0000 7fee6f7fe700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 0x7fee7019f850 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fee68007ee0 tx=0x7fee6800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:04.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.130+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68017070 con 0x7fee701081d0 2023-07-29T16:14:04.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.130+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee6800f040 con 0x7fee701081d0 2023-07-29T16:14:04.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.130+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee701a03d0 con 0x7fee701081d0 2023-07-29T16:14:04.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.130+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68005580 con 0x7fee701081d0 2023-07-29T16:14:04.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.130+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee701a08e0 con 0x7fee701081d0 2023-07-29T16:14:04.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.131+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee70106910 con 0x7fee701081d0 2023-07-29T16:14:04.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.133+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fee6800db20 con 0x7fee701081d0 2023-07-29T16:14:04.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.134+0000 7fee57fff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee580a0790 0x7fee580a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.135+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fee680cb210 con 0x7fee701081d0 2023-07-29T16:14:04.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.135+0000 7fee6effd700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee580a0790 0x7fee580a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.135+0000 7fee6effd700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee580a0790 0x7fee580a2c40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee6000b2e0 tx=0x7fee60005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:04.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.138+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fee68094bf0 con 0x7fee701081d0 2023-07-29T16:14:04.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:03 smithi031 ceph-mon[95504]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:04.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:03 smithi031 ceph-mon[95504]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.3 KiB/s wr, 7 op/s; 91 B/s, 0 objects/s recovering 2023-07-29T16:14:04.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/4013984797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:04.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:03 smithi039 ceph-mon[128858]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-29T16:14:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:03 smithi039 ceph-mon[128858]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.3 KiB/s wr, 7 op/s; 91 B/s, 0 objects/s recovering 2023-07-29T16:14:04.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/4013984797' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:04.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.347+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee70061960 con 0x7fee701081d0 2023-07-29T16:14:04.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.348+0000 7fee57fff700 1 -- 172.21.15.31:0/2506543961 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee68098210 con 0x7fee701081d0 2023-07-29T16:14:04.346 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:14:04.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee580a0790 msgr2=0x7fee580a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee580a0790 0x7fee580a2c40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fee6000b2e0 tx=0x7fee60005fb0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 msgr2=0x7fee7019f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 0x7fee7019f850 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fee68007ee0 tx=0x7fee6800d3b0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 shutdown_connections 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee580a0790 0x7fee580a2c40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee701081d0 0x7fee7019f850 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee70073750 0x7fee701982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 --2- 172.21.15.31:0/2506543961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee70074d30 0x7fee701987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 >> 172.21.15.31:0/2506543961 conn(0x7fee700fb960 msgr2=0x7fee700fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 shutdown_connections 2023-07-29T16:14:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.350+0000 7fee753ff700 1 -- 172.21.15.31:0/2506543961 wait complete. 2023-07-29T16:14:04.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:14:04.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 -- 172.21.15.31:0/3397185302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8101eb0 msgr2=0x7f46f8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 --2- 172.21.15.31:0/3397185302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8101eb0 0x7f46f8108770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f46f0009f90 tx=0x7f46f000b330 comp rx=0 tx=0).stop 2023-07-29T16:14:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 -- 172.21.15.31:0/3397185302 shutdown_connections 2023-07-29T16:14:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 --2- 172.21.15.31:0/3397185302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8101eb0 0x7f46f8108770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 --2- 172.21.15.31:0/3397185302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8100330 0x7f46f8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 --2- 172.21.15.31:0/3397185302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f8101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 -- 172.21.15.31:0/3397185302 >> 172.21.15.31:0/3397185302 conn(0x7f46f80fb970 msgr2=0x7f46f80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:04.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 -- 172.21.15.31:0/3397185302 shutdown_connections 2023-07-29T16:14:04.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.576+0000 7f46fcfff700 1 -- 172.21.15.31:0/3397185302 wait complete. 2023-07-29T16:14:04.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.578+0000 7f46fcfff700 1 Processor -- start 2023-07-29T16:14:04.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.578+0000 7f46fcfff700 1 -- start start 2023-07-29T16:14:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46fcfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8100330 0x7f46f8195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46fcfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f81964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46fcfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8101eb0 0x7f46f819d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46fcfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f819dae0 con 0x7f46f8101520 2023-07-29T16:14:04.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46fcfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f819dc20 con 0x7f46f8101eb0 2023-07-29T16:14:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46fcfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f819ddb0 con 0x7f46f8100330 2023-07-29T16:14:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f5d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f81964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f5d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f81964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43458/0 (socket says 172.21.15.31:43458) 2023-07-29T16:14:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f5d9b700 1 -- 172.21.15.31:0/1254680439 learned_addr learned my addr 172.21.15.31:0/1254680439 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f659c700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8100330 0x7f46f8195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f6d9d700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8101eb0 0x7f46f819d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f659c700 1 -- 172.21.15.31:0/1254680439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8101eb0 msgr2=0x7f46f819d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f659c700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8101eb0 0x7f46f819d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f659c700 1 -- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 msgr2=0x7f46f81964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f659c700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f81964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f659c700 1 -- 172.21.15.31:0/1254680439 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f0009c40 con 0x7f46f8100330 2023-07-29T16:14:04.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f5d9b700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f81964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:04.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.579+0000 7f46f6d9d700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8101eb0 0x7f46f819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:04.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.580+0000 7f46f659c700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8100330 0x7f46f8195f60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f46e8005fd0 tx=0x7f46e800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:04.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.580+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8004220 con 0x7f46f8100330 2023-07-29T16:14:04.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.580+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f819e090 con 0x7f46f8100330 2023-07-29T16:14:04.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.580+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46e8004380 con 0x7f46f8100330 2023-07-29T16:14:04.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.580+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8003be0 con 0x7f46f8100330 2023-07-29T16:14:04.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.580+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f819e5f0 con 0x7f46f8100330 2023-07-29T16:14:04.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.581+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f810bac0 con 0x7f46f8100330 2023-07-29T16:14:04.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.583+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f46e8003d40 con 0x7f46f8100330 2023-07-29T16:14:04.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.585+0000 7f46e77fe700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46e00a0790 0x7f46e00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:04.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.585+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f46e8014070 con 0x7f46f8100330 2023-07-29T16:14:04.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.585+0000 7f46f5d9b700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46e00a0790 0x7f46e00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:04.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.586+0000 7f46f5d9b700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46e00a0790 0x7f46e00a2c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f46ec011550 tx=0x7f46ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:04.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.587+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f46e8095900 con 0x7f46f8100330 2023-07-29T16:14:04.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.797+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46f8107170 con 0x7f46f8100330 2023-07-29T16:14:04.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.801+0000 7f46e77fe700 1 -- 172.21.15.31:0/1254680439 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f46e8098f20 con 0x7f46f8100330 2023-07-29T16:14:04.799 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:14:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.803+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46e00a0790 msgr2=0x7f46e00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.803+0000 7f46fcfff700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46e00a0790 0x7f46e00a2c40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f46ec011550 tx=0x7f46ec009260 comp rx=0 tx=0).stop 2023-07-29T16:14:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.803+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8100330 msgr2=0x7f46f8195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.803+0000 7f46fcfff700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8100330 0x7f46f8195f60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f46e8005fd0 tx=0x7f46e800eca0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 shutdown_connections 2023-07-29T16:14:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46e00a0790 0x7f46e00a2c40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46f8101eb0 0x7f46f819d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46f8100330 0x7f46f8195f60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 --2- 172.21.15.31:0/1254680439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46f8101520 0x7f46f81964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 >> 172.21.15.31:0/1254680439 conn(0x7f46f80fb970 msgr2=0x7f46f80fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 shutdown_connections 2023-07-29T16:14:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:04.804+0000 7f46fcfff700 1 -- 172.21.15.31:0/1254680439 wait complete. 2023-07-29T16:14:04.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:14:05.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 -- 172.21.15.31:0/3113802254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101ec0 msgr2=0x7fc338108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 --2- 172.21.15.31:0/3113802254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101ec0 0x7fc338108780 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc33000b0a0 tx=0x7fc33000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 -- 172.21.15.31:0/3113802254 shutdown_connections 2023-07-29T16:14:05.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 --2- 172.21.15.31:0/3113802254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101ec0 0x7fc338108780 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 --2- 172.21.15.31:0/3113802254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc338100340 0x7fc338100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 --2- 172.21.15.31:0/3113802254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338101530 0x7fc338101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 -- 172.21.15.31:0/3113802254 >> 172.21.15.31:0/3113802254 conn(0x7fc3380fb920 msgr2=0x7fc3380fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:05.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 -- 172.21.15.31:0/3113802254 shutdown_connections 2023-07-29T16:14:05.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.027+0000 7fc3407ff700 1 -- 172.21.15.31:0/3113802254 wait complete. 2023-07-29T16:14:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.028+0000 7fc3407ff700 1 Processor -- start 2023-07-29T16:14:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 -- start start 2023-07-29T16:14:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 0x7fc338198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101530 0x7fc338198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc338101ec0 0x7fc33819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33819fed0 con 0x7fc338101530 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381a0010 con 0x7fc338101ec0 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc3407ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381a01a0 con 0x7fc338100340 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc33dd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101530 0x7fc338198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc33e59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 0x7fc338198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc33e59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 0x7fc338198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54372/0 (socket says 172.21.15.31:54372) 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc33e59b700 1 -- 172.21.15.31:0/634808060 learned_addr learned my addr 172.21.15.31:0/634808060 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:05.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.029+0000 7fc33ed9c700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc338101ec0 0x7fc33819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc33e59b700 1 -- 172.21.15.31:0/634808060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc338101ec0 msgr2=0x7fc33819f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc33e59b700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc338101ec0 0x7fc33819f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc33e59b700 1 -- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101530 msgr2=0x7fc338198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc33e59b700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101530 0x7fc338198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc33e59b700 1 -- 172.21.15.31:0/634808060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc330009d00 con 0x7fc338100340 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc33e59b700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 0x7fc338198350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc33400e970 tx=0x7fc33400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334009790 con 0x7fc338100340 2023-07-29T16:14:05.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3381a0480 con 0x7fc338100340 2023-07-29T16:14:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc33400c7e0 con 0x7fc338100340 2023-07-29T16:14:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334003b60 con 0x7fc338100340 2023-07-29T16:14:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.030+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3381a09e0 con 0x7fc338100340 2023-07-29T16:14:05.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.031+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33810bb60 con 0x7fc338100340 2023-07-29T16:14:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.034+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc334003d00 con 0x7fc338100340 2023-07-29T16:14:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.035+0000 7fc32f7fe700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3240a0860 0x7fc3240a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.035+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc334014070 con 0x7fc338100340 2023-07-29T16:14:05.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.035+0000 7fc33dd9a700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3240a0860 0x7fc3240a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.036+0000 7fc33dd9a700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3240a0860 0x7fc3240a2d10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc328009c60 tx=0x7fc328009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:05.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.037+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc334095880 con 0x7fc338100340 2023-07-29T16:14:05.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:04 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2545253697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:14:05.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:04 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2506543961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:05.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:04 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2545253697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:14:05.226 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:04 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2506543961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:05.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:04 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2545253697' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:14:05.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:04 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2506543961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:14:05.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.287+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3381a1080 con 0x7fc338100340 2023-07-29T16:14:05.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.288+0000 7fc32f7fe700 1 -- 172.21.15.31:0/634808060 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7fc334098ea0 con 0x7fc338100340 2023-07-29T16:14:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:05.287 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3240a0860 msgr2=0x7fc3240a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3240a0860 0x7fc3240a2d10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc328009c60 tx=0x7fc328009260 comp rx=0 tx=0).stop 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 msgr2=0x7fc338198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 0x7fc338198350 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc33400e970 tx=0x7fc33400ec80 comp rx=0 tx=0).stop 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 shutdown_connections 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3240a0860 0x7fc3240a2d10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc338101ec0 0x7fc33819f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc338100340 0x7fc338198350 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 --2- 172.21.15.31:0/634808060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc338101530 0x7fc338198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 >> 172.21.15.31:0/634808060 conn(0x7fc3380fb920 msgr2=0x7fc3380fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 shutdown_connections 2023-07-29T16:14:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.290+0000 7fc3407ff700 1 -- 172.21.15.31:0/634808060 wait complete. 2023-07-29T16:14:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:14:05.307 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:14:05.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/1996535283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101530 msgr2=0x7f0f44101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/1996535283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101530 0x7f0f44101980 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0f38009990 tx=0x7f0f38009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/1996535283 shutdown_connections 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/1996535283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44101ec0 0x7f0f44108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/1996535283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44100340 0x7f0f44100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/1996535283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101530 0x7f0f44101980 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/1996535283 >> 172.21.15.31:0/1996535283 conn(0x7f0f440fb920 msgr2=0x7f0f440fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/1996535283 shutdown_connections 2023-07-29T16:14:05.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.515+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/1996535283 wait complete. 2023-07-29T16:14:05.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 Processor -- start 2023-07-29T16:14:05.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 -- start start 2023-07-29T16:14:05.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 0x7f0f441981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44101530 0x7f0f44198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101ec0 0x7f0f4419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4419fd70 con 0x7f0f44101ec0 2023-07-29T16:14:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4419feb0 con 0x7f0f44101530 2023-07-29T16:14:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.516+0000 7f0f4a9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f441a0040 con 0x7f0f44100340 2023-07-29T16:14:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 0x7f0f441981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 0x7f0f441981f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54388/0 (socket says 172.21.15.31:54388) 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 -- 172.21.15.31:0/3115756140 learned_addr learned my addr 172.21.15.31:0/3115756140 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f48f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101ec0 0x7f0f4419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f437fe700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44101530 0x7f0f44198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 -- 172.21.15.31:0/3115756140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44101530 msgr2=0x7f0f44198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44101530 0x7f0f44198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 -- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101ec0 msgr2=0x7f0f4419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101ec0 0x7f0f4419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 -- 172.21.15.31:0/3115756140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380095f0 con 0x7f0f44100340 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f437fe700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44101530 0x7f0f44198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f48f9c700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101ec0 0x7f0f4419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.517+0000 7f0f43fff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 0x7f0f441981f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400e970 tx=0x7f0f3400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.518+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f34009790 con 0x7f0f44100340 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.518+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f441a0320 con 0x7f0f44100340 2023-07-29T16:14:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.518+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f3400eec0 con 0x7f0f44100340 2023-07-29T16:14:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.518+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f34003cf0 con 0x7f0f44100340 2023-07-29T16:14:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.518+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f441a0830 con 0x7f0f44100340 2023-07-29T16:14:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.519+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4410bb10 con 0x7f0f44100340 2023-07-29T16:14:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.521+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0f3400ccd0 con 0x7f0f44100340 2023-07-29T16:14:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.522+0000 7f0f417fa700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0f2c0a07e0 0x7f0f2c0a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.522+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0f34014070 con 0x7f0f44100340 2023-07-29T16:14:05.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.523+0000 7f0f437fe700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0f2c0a07e0 0x7f0f2c0a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.523+0000 7f0f437fe700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0f2c0a07e0 0x7f0f2c0a2c90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0f38006010 tx=0x7f0f3800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:05.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.524+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0f34095990 con 0x7f0f44100340 2023-07-29T16:14:05.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.733+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f441a0ed0 con 0x7f0f44100340 2023-07-29T16:14:05.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.735+0000 7f0f417fa700 1 -- 172.21.15.31:0/3115756140 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f0f34098fb0 con 0x7f0f44100340 2023-07-29T16:14:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:05.734 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":90,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:13:50.771629+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:14:05.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0f2c0a07e0 msgr2=0x7f0f2c0a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0f2c0a07e0 0x7f0f2c0a2c90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0f38006010 tx=0x7f0f3800b260 comp rx=0 tx=0).stop 2023-07-29T16:14:05.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 msgr2=0x7f0f441981f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 0x7f0f441981f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400e970 tx=0x7f0f3400ec80 comp rx=0 tx=0).stop 2023-07-29T16:14:05.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 shutdown_connections 2023-07-29T16:14:05.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0f2c0a07e0 0x7f0f2c0a2c90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f44101ec0 0x7f0f4419f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f44100340 0x7f0f441981f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 --2- 172.21.15.31:0/3115756140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f44101530 0x7f0f44198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 >> 172.21.15.31:0/3115756140 conn(0x7f0f440fb920 msgr2=0x7f0f44106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:05.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 shutdown_connections 2023-07-29T16:14:05.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.737+0000 7f0f4a9ff700 1 -- 172.21.15.31:0/3115756140 wait complete. 2023-07-29T16:14:05.754 DEBUG:tasks.fwd_scrub:Yielding 2023-07-29T16:14:05.755 INFO:teuthology.run_tasks:Running task background_exec... 2023-07-29T16:14:05.768 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T16:14:05.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-07-29T16:14:05.769 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-07-29T16:14:05.769 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:14:05.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:14:05.774 INFO:teuthology.run_tasks:Running task workunit... 2023-07-29T16:14:05.830 INFO:tasks.workunit:Pulling workunits from ref 0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:14:05.831 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-07-29T16:14:05.831 INFO:tasks.workunit:timeout=3h 2023-07-29T16:14:05.831 INFO:tasks.workunit:cleanup=True 2023-07-29T16:14:05.832 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2023-07-29T16:14:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-07-29T16:14:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-07-29T16:14:05.878 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 79h/121d Inode: 1099511628283 Links: 2 2023-07-29T16:14:05.879 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-29T16:14:05.879 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:cephfs_t:s0 2023-07-29T16:14:05.879 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-07-29 16:12:29.269762019 +0000 2023-07-29T16:14:05.879 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-07-29 16:12:29.269762019 +0000 2023-07-29T16:14:05.879 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-07-29 16:13:56.934989810 +0000 2023-07-29T16:14:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-07-29 16:12:29.269762019 +0000 2023-07-29T16:14:05.879 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-07-29T16:14:05.880 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-07-29T16:14:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 -- 172.21.15.31:0/3196993870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 msgr2=0x7f561c073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 --2- 172.21.15.31:0/3196993870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c073c20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5610009990 tx=0x7f5610009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 -- 172.21.15.31:0/3196993870 shutdown_connections 2023-07-29T16:14:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 --2- 172.21.15.31:0/3196993870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f561c10c8f0 0x7f561c10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 --2- 172.21.15.31:0/3196993870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f561c074d90 0x7f561c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 --2- 172.21.15.31:0/3196993870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c073c20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 -- 172.21.15.31:0/3196993870 >> 172.21.15.31:0/3196993870 conn(0x7f561c0fb920 msgr2=0x7f561c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:05.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 -- 172.21.15.31:0/3196993870 shutdown_connections 2023-07-29T16:14:05.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.992+0000 7f5621bff700 1 -- 172.21.15.31:0/3196993870 wait complete. 2023-07-29T16:14:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 Processor -- start 2023-07-29T16:14:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 -- start start 2023-07-29T16:14:05.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c19c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f561c074d90 0x7f561c19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f561c10c8f0 0x7f561c1a3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c1a42c0 con 0x7f561c0737b0 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c1a4400 con 0x7f561c074d90 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.993+0000 7f5621bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561c1a4590 con 0x7f561c10c8f0 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c19c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43510/0 (socket says 172.21.15.31:43510) 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f561c10c8f0 0x7f561c1a3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 -- 172.21.15.31:0/1424349112 learned_addr learned my addr 172.21.15.31:0/1424349112 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561affd700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f561c074d90 0x7f561c19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 -- 172.21.15.31:0/1424349112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f561c10c8f0 msgr2=0x7f561c1a3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f561c10c8f0 0x7f561c1a3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 -- 172.21.15.31:0/1424349112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f561c074d90 msgr2=0x7f561c19cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:05.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f561c074d90 0x7f561c19cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 -- 172.21.15.31:0/1424349112 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56100095f0 con 0x7f561c0737b0 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.994+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c19c740 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f560c00e970 tx=0x7f560c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.995+0000 7f5618ff9700 1 -- 172.21.15.31:0/1424349112 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c009790 con 0x7f561c0737b0 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.995+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561c1a4870 con 0x7f561c0737b0 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.995+0000 7f5618ff9700 1 -- 172.21.15.31:0/1424349112 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f560c00c7e0 con 0x7f561c0737b0 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.995+0000 7f5618ff9700 1 -- 172.21.15.31:0/1424349112 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c0054e0 con 0x7f561c0737b0 2023-07-29T16:14:05.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.995+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c1a4dd0 con 0x7f561c0737b0 2023-07-29T16:14:05.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:05.998+0000 7f5618ff9700 1 -- 172.21.15.31:0/1424349112 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f560c00c950 con 0x7f561c0737b0 2023-07-29T16:14:05.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 Processor -- start 2023-07-29T16:14:05.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5618ff9700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56040a0790 0x7f56040a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 -- start start 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5618ff9700 1 -- 172.21.15.31:0/1424349112 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f560c014070 con 0x7f561c0737b0 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f561bfff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56040a0790 0x7f56040a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84637d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b84637dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84637e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:05.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8462d4a90 con 0x55b84637d5b0 2023-07-29T16:14:06.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8462d2450 con 0x55b84637cc60 2023-07-29T16:14:06.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.000+0000 7f5635436b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8462cdd40 con 0x55b84637e370 2023-07-29T16:14:06.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84637d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b84637dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b84637dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43524/0 (socket says 172.21.15.31:43524) 2023-07-29T16:14:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561affd700 1 -- 172.21.15.31:0/4279522266 learned_addr learned my addr 172.21.15.31:0/4279522266 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561bfff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56040a0790 0x7f56040a2c40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f561400f540 tx=0x7f561400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561bfff700 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84637e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 -- 172.21.15.31:0/4279522266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 msgr2=0x55b84637e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84637e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 -- 172.21.15.31:0/4279522266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 msgr2=0x55b84637dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b84637dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 -- 172.21.15.31:0/4279522266 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8460fbe60 con 0x55b84637cc60 2023-07-29T16:14:06.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.001+0000 7f561b7fe700 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84637d070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f560c00cc00 tx=0x7f560c098e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11d8b3758c79546c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.002+0000 7f5601ffb700 1 -- 172.21.15.31:0/4279522266 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c09d500 con 0x55b84637cc60 2023-07-29T16:14:06.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.002+0000 7f5601ffb700 1 -- 172.21.15.31:0/4279522266 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f560c0a1920 con 0x55b84637cc60 2023-07-29T16:14:06.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.002+0000 7f5601ffb700 1 -- 172.21.15.31:0/4279522266 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c099730 con 0x55b84637cc60 2023-07-29T16:14:06.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.002+0000 7f5635436b80 1 -- 172.21.15.31:0/4279522266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 msgr2=0x55b84637d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.002+0000 7f5635436b80 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84637d070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f560c00cc00 tx=0x7f560c098e30 comp rx=0 tx=0).stop 2023-07-29T16:14:06.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 -- 172.21.15.31:0/4279522266 shutdown_connections 2023-07-29T16:14:06.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84637e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84637d070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 --2- 172.21.15.31:0/4279522266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b84637dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 -- 172.21.15.31:0/4279522266 >> 172.21.15.31:0/4279522266 conn(0x55b846372460 msgr2=0x55b846377a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 -- 172.21.15.31:0/4279522266 shutdown_connections 2023-07-29T16:14:06.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 -- 172.21.15.31:0/4279522266 wait complete. 2023-07-29T16:14:06.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 -- start start 2023-07-29T16:14:06.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.003+0000 7f5635436b80 1 Processor -- start 2023-07-29T16:14:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b846386990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b846386ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84638df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b84626a590 con 0x55b84637d5b0 2023-07-29T16:14:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b84626a230 con 0x55b84637cc60 2023-07-29T16:14:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8461519a0 con 0x55b84637e370 2023-07-29T16:14:06.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f5635436b80 10 client.0 _ll_register_callbacks cb 0x55b8462736a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b846386990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b846386990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35224/0 (socket says 172.21.15.31:35224) 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.004+0000 7f561b7fe700 1 -- 172.21.15.31:0/1233696775 learned_addr learned my addr 172.21.15.31:0/1233696775 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561bfff700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84638df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561affd700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b846386ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561b7fe700 1 -- 172.21.15.31:0/1233696775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 msgr2=0x55b84638df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84638df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561b7fe700 1 -- 172.21.15.31:0/1233696775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 msgr2=0x55b846386ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b846386ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.012 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:05 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1254680439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:06.012 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:05 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/634808060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:06.012 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:05 smithi031 ceph-mon[95504]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 6 op/s 2023-07-29T16:14:06.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:05 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3115756140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561b7fe700 1 -- 172.21.15.31:0/1233696775 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c1a4dd0 con 0x55b84637cc60 2023-07-29T16:14:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561bfff700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84638df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b846386990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f560c097ac0 tx=0x7f560c098d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfcbcbccba351ef server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f55faffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:14:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f55faffd700 1 -- 172.21.15.31:0/1233696775 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c097870 con 0x55b84637cc60 2023-07-29T16:14:06.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f55faffd700 1 -- 172.21.15.31:0/1233696775 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f560c09d500 con 0x55b84637cc60 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.005+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b8462f8d90 con 0x55b84637cc60 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f55faffd700 1 -- 172.21.15.31:0/1233696775 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560c099940 con 0x55b84637cc60 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 10 client.25295 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b8460e4ce0 con 0x55b84637cc60 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f55faffd700 1 -- 172.21.15.31:0/1233696775 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f560c07c080 con 0x55b84637cc60 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f55faffd700 1 -- 172.21.15.31:0/1233696775 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f560c004a30 con 0x55b84637cc60 2023-07-29T16:14:06.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 10 client.25295 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 10 client.25295 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 20 client.25295 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b846392490 0x55b8463948a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 4 client.25295 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b84634bdf0 con 0x55b846392490 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.006+0000 7f561affd700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b846392490 0x55b8463948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.007+0000 7f55faffd700 10 client.25295 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.007+0000 7f561affd700 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b846392490 0x55b8463948a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be697134ff6ba8d3 server_cookie=96fca252ec19cb69 in_seq=0 out_seq=0 2023-07-29T16:14:06.018 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0484097b0572e96f42cf6402cbe0ac8dcb046577 2023-07-29T16:14:06.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f55faffd700 1 -- 172.21.15.31:0/1233696775 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b84634bdf0 con 0x55b846392490 2023-07-29T16:14:06.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f55faffd700 10 client.25295 handle_command_reply: tid=0 2023-07-29T16:14:06.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 client.25295 shutdown 2023-07-29T16:14:06.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 msgr2=0x55b846386990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b846386990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f560c097ac0 tx=0x7f560c098d20 comp rx=0 tx=0).stop 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 shutdown_connections 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b846392490 0x55b8463948a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637e370 0x55b84638df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b846386990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f55faffd700 0 client.25295 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 --2- 172.21.15.31:0/1233696775 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637d5b0 0x55b846386ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 >> 172.21.15.31:0/1233696775 conn(0x55b846372460 msgr2=0x55b8463770e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 shutdown_connections 2023-07-29T16:14:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 1 -- 172.21.15.31:0/1233696775 wait complete. 2023-07-29T16:14:06.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.008+0000 7f5635436b80 20 client.25295 trim_cache size 0 max 16384 2023-07-29T16:14:06.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 Processor -- start 2023-07-29T16:14:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 -- start start 2023-07-29T16:14:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84638bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637d5b0 0x55b846384810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b84638e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b84637dc40 con 0x55b84637e370 2023-07-29T16:14:06.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b846377810 con 0x55b84637cc60 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f5635436b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8463744e0 con 0x55b84637d5b0 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84638bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84638bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35236/0 (socket says 172.21.15.31:35236) 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.028+0000 7f561b7fe700 1 -- 172.21.15.31:0/2913610383 learned_addr learned my addr 172.21.15.31:0/2913610383 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561affd700 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637d5b0 0x55b846384810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561bfff700 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b84638e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561b7fe700 1 -- 172.21.15.31:0/2913610383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637d5b0 msgr2=0x55b846384810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561b7fe700 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637d5b0 0x55b846384810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561b7fe700 1 -- 172.21.15.31:0/2913610383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 msgr2=0x55b84638e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561b7fe700 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b84638e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561b7fe700 1 -- 172.21.15.31:0/2913610383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8463773b0 con 0x55b84637cc60 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561bfff700 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b84638e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f561b7fe700 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84638bec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f560c00efd0 tx=0x7f560c098bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b13a534213799a0f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f56027fc700 1 -- 172.21.15.31:0/2913610383 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c0976d0 con 0x55b84637cc60 2023-07-29T16:14:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f56027fc700 1 -- 172.21.15.31:0/2913610383 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b8462f8d90 con 0x55b84637cc60 2023-07-29T16:14:06.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.029+0000 7f56027fc700 1 -- 172.21.15.31:0/2913610383 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c004a30 con 0x55b84637cc60 2023-07-29T16:14:06.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 -- 172.21.15.31:0/2913610383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 msgr2=0x55b84638bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84638bec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f560c00efd0 tx=0x7f560c098bd0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 -- 172.21.15.31:0/2913610383 shutdown_connections 2023-07-29T16:14:06.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b84638e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637cc60 0x55b84638bec0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 --2- 172.21.15.31:0/2913610383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637d5b0 0x55b846384810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 -- 172.21.15.31:0/2913610383 >> 172.21.15.31:0/2913610383 conn(0x55b846372460 msgr2=0x55b846378710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 -- 172.21.15.31:0/2913610383 shutdown_connections 2023-07-29T16:14:06.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.030+0000 7f5635436b80 1 -- 172.21.15.31:0/2913610383 wait complete. 2023-07-29T16:14:06.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 -- start start 2023-07-29T16:14:06.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 Processor -- start 2023-07-29T16:14:06.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 0x55b8463dc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637d5b0 0x55b8463dce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b8463e3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8463e43a0 con 0x55b84637e370 2023-07-29T16:14:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8463e44e0 con 0x55b84637d5b0 2023-07-29T16:14:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8463e4670 con 0x55b84637cc60 2023-07-29T16:14:06.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.031+0000 7f5635436b80 10 client.0 _ll_register_callbacks cb 0x55b8462736a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:06.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 0x55b8463dc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 0x55b8463dc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54458/0 (socket says 172.21.15.31:54458) 2023-07-29T16:14:06.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 -- 172.21.15.31:0/1950499081 learned_addr learned my addr 172.21.15.31:0/1950499081 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561bfff700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b8463e3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561affd700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637d5b0 0x55b8463dce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 -- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637d5b0 msgr2=0x55b8463dce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637d5b0 0x55b8463dce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 -- 172.21.15.31:0/1950499081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 msgr2=0x55b8463e3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b8463e3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 -- 172.21.15.31:0/1950499081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8460e4ce0 con 0x55b84637cc60 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561affd700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637d5b0 0x55b8463dce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f561b7fe700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 0x55b8463dc8e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f560c00cc00 tx=0x7f560c097a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62ecea8cd4835fd4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.032+0000 7f55faffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f55faffd700 1 -- 172.21.15.31:0/1950499081 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560c0976d0 con 0x55b84637cc60 2023-07-29T16:14:06.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f55faffd700 1 -- 172.21.15.31:0/1950499081 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f560c09d500 con 0x55b84637cc60 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f560c0a1920 con 0x55b84637cc60 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f55faffd700 1 -- 172.21.15.31:0/1950499081 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f560c098e30 con 0x55b84637cc60 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 10 client.25312 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b84637ea10 con 0x55b84637cc60 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f55faffd700 1 -- 172.21.15.31:0/1950499081 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f560c0a8080 con 0x55b84637cc60 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f55faffd700 1 -- 172.21.15.31:0/1950499081 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f560c081060 con 0x55b84637cc60 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 10 client.25312 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:06.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 10 client.25312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 20 client.25312 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b8463e5900 0x55b8463e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 4 client.25312 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:14:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.033+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b8462c6dc0 con 0x55b8463e5900 2023-07-29T16:14:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.034+0000 7f561affd700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b8463e5900 0x55b8463e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.034+0000 7f561affd700 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b8463e5900 0x55b8463e7d10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64f9f6e5199a3d04 server_cookie=67235c2468610217 in_seq=0 out_seq=0 2023-07-29T16:14:06.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.034+0000 7f55faffd700 10 client.25312 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f55faffd700 1 -- 172.21.15.31:0/1950499081 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b8462c6dc0 con 0x55b8463e5900 2023-07-29T16:14:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f55faffd700 10 client.25312 handle_command_reply: tid=0 2023-07-29T16:14:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 client.25312 shutdown 2023-07-29T16:14:06.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 msgr2=0x55b8463dc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 0x55b8463dc8e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f560c00cc00 tx=0x7f560c097a70 comp rx=0 tx=0).stop 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 shutdown_connections 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b8463e5900 0x55b8463e7d10 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b84637e370 0x55b8463e3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b84637cc60 0x55b8463dc8e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 --2- 172.21.15.31:0/1950499081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b84637d5b0 0x55b8463dce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 >> 172.21.15.31:0/1950499081 conn(0x55b846372460 msgr2=0x55b846384810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f55faffd700 0 client.25312 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 shutdown_connections 2023-07-29T16:14:06.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 1 -- 172.21.15.31:0/1950499081 wait complete. 2023-07-29T16:14:06.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.036+0000 7f5635436b80 20 client.25312 trim_cache size 0 max 16384 2023-07-29T16:14:06.048 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:14:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:14:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "6ad0281a-2dcd-4fa3-931c-549c5cd7db01", 2023-07-29T16:14:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:14:06.049 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:14:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56040a0790 msgr2=0x7f56040a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56040a0790 0x7f56040a2c40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f561400f540 tx=0x7f561400b040 comp rx=0 tx=0).stop 2023-07-29T16:14:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 msgr2=0x7f561c19c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c19c740 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f560c00e970 tx=0x7f560c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:14:06.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 shutdown_connections 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56040a0790 0x7f56040a2c40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f561c10c8f0 0x7f561c1a3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f561c0737b0 0x7f561c19c740 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 --2- 172.21.15.31:0/1424349112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f561c074d90 0x7f561c19cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 >> 172.21.15.31:0/1424349112 conn(0x7f561c0fb920 msgr2=0x7f561c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 shutdown_connections 2023-07-29T16:14:06.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.037+0000 7f5621bff700 1 -- 172.21.15.31:0/1424349112 wait complete. 2023-07-29T16:14:06.054 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:14:06.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:05 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1254680439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:06.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:05 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/634808060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:06.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:05 smithi156 ceph-mon[129245]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 6 op/s 2023-07-29T16:14:06.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:05 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3115756140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:06.072 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-07-29T16:14:06.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:05 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1254680439' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:14:06.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:05 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/634808060' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:06.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:05 smithi039 ceph-mon[128858]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 853 B/s wr, 6 op/s 2023-07-29T16:14:06.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:05 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3115756140' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:14:06.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.281+0000 7f3a53fff700 1 -- 172.21.15.31:0/1047245643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101520 msgr2=0x7f3a4c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.281+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1047245643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101520 0x7f3a4c101970 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009990 tx=0x7f3a3c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 -- 172.21.15.31:0/1047245643 shutdown_connections 2023-07-29T16:14:06.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1047245643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101eb0 0x7f3a4c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1047245643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1047245643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101520 0x7f3a4c101970 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 -- 172.21.15.31:0/1047245643 >> 172.21.15.31:0/1047245643 conn(0x7f3a4c0fb970 msgr2=0x7f3a4c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 -- 172.21.15.31:0/1047245643 shutdown_connections 2023-07-29T16:14:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.282+0000 7f3a53fff700 1 -- 172.21.15.31:0/1047245643 wait complete. 2023-07-29T16:14:06.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 Processor -- start 2023-07-29T16:14:06.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 -- start start 2023-07-29T16:14:06.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 0x7f3a4c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101eb0 0x7f3a4c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c073e20 con 0x7f3a4c101520 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c073f90 con 0x7f3a4c100330 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.283+0000 7f3a53fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c1a0080 con 0x7f3a4c101eb0 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a5159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 0x7f3a4c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a5159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 0x7f3a4c073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43576/0 (socket says 172.21.15.31:43576) 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a5159a700 1 -- 172.21.15.31:0/1105393799 learned_addr learned my addr 172.21.15.31:0/1105393799 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35274/0 (socket says 172.21.15.31:35274) 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a5259c700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101eb0 0x7f3a4c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1105393799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101eb0 msgr2=0x7f3a4c073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101eb0 0x7f3a4c073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 msgr2=0x7f3a4c073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 0x7f3a4c073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1105393799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0095f0 con 0x7f3a4c100330 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a5159a700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 0x7f3a4c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.284+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c074b00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e9c0 tx=0x7f3a4800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.285+0000 7f3a42ffd700 1 -- 172.21.15.31:0/1105393799 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48004220 con 0x7f3a4c100330 2023-07-29T16:14:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.285+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c1a0360 con 0x7f3a4c100330 2023-07-29T16:14:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.285+0000 7f3a42ffd700 1 -- 172.21.15.31:0/1105393799 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a48004380 con 0x7f3a4c100330 2023-07-29T16:14:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.285+0000 7f3a42ffd700 1 -- 172.21.15.31:0/1105393799 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48003be0 con 0x7f3a4c100330 2023-07-29T16:14:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.285+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c1a0870 con 0x7f3a4c100330 2023-07-29T16:14:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.288+0000 7f3a42ffd700 1 -- 172.21.15.31:0/1105393799 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3a48003d40 con 0x7f3a4c100330 2023-07-29T16:14:06.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.289+0000 7f3a679d7b80 1 Processor -- start 2023-07-29T16:14:06.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 -- start start 2023-07-29T16:14:06.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e67ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e68530 0x555649e68d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e692f0 0x555649e69740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649dbfa10 con 0x555649e67be0 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649dbd3d0 con 0x555649e692f0 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649db8cc0 con 0x555649e68530 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e67ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e68530 0x555649e68d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e67ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43592/0 (socket says 172.21.15.31:43592) 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a5259c700 1 -- 172.21.15.31:0/2538701719 learned_addr learned my addr 172.21.15.31:0/2538701719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.290+0000 7f3a5159a700 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e692f0 0x555649e69740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a42ffd700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a380a0790 0x7f3a380a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5159a700 1 -- 172.21.15.31:0/2538701719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e68530 msgr2=0x555649e68d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5159a700 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e68530 0x555649e68d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5159a700 1 -- 172.21.15.31:0/2538701719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 msgr2=0x555649e67ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5159a700 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e67ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5159a700 1 -- 172.21.15.31:0/2538701719 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555649be6de0 con 0x555649e692f0 2023-07-29T16:14:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a42ffd700 1 -- 172.21.15.31:0/1105393799 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3a48014070 con 0x7f3a4c100330 2023-07-29T16:14:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5259c700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a380a0790 0x7f3a380a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a5159a700 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e692f0 0x555649e69740 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c000c00 tx=0x7f3a3c005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=107eda7d2e7971b4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a37fff700 1 -- 172.21.15.31:0/2538701719 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c01d050 con 0x555649e692f0 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a37fff700 1 -- 172.21.15.31:0/2538701719 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a3c01e040 con 0x555649e692f0 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.291+0000 7f3a37fff700 1 -- 172.21.15.31:0/2538701719 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c005660 con 0x555649e692f0 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 -- 172.21.15.31:0/2538701719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e692f0 msgr2=0x555649e69740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a5259c700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a380a0790 0x7f3a380a2c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x555649e67980 tx=0x7f3a44009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e692f0 0x555649e69740 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c000c00 tx=0x7f3a3c005ea0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 -- 172.21.15.31:0/2538701719 shutdown_connections 2023-07-29T16:14:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e692f0 0x555649e69740 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e67ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/2538701719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e68530 0x555649e68d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 -- 172.21.15.31:0/2538701719 >> 172.21.15.31:0/2538701719 conn(0x555649e5d3e0 msgr2=0x555649e62990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 -- 172.21.15.31:0/2538701719 shutdown_connections 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.292+0000 7f3a679d7b80 1 -- 172.21.15.31:0/2538701719 wait complete. 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 -- start start 2023-07-29T16:14:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 Processor -- start 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e71d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e78d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649d55510 con 0x555649e67be0 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649d551b0 con 0x555649e68530 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649c3c920 con 0x555649e692f0 2023-07-29T16:14:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.293+0000 7f3a679d7b80 10 client.0 _ll_register_callbacks cb 0x555649d5e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a5159a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e71d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43608/0 (socket says 172.21.15.31:43608) 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1639829674 learned_addr learned my addr 172.21.15.31:0/1639829674 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a5259c700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e78d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1639829674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 msgr2=0x555649e78d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e78d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 msgr2=0x555649e71d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e71d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1639829674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c1090a0 con 0x555649e67be0 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a5159a700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e71d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e717c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800c5b0 tx=0x7f3a480993a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc16238cf2de94 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.294+0000 7f3a357fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a357fa700 1 -- 172.21.15.31:0/1639829674 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4809d370 con 0x555649e67be0 2023-07-29T16:14:06.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555649de3d10 con 0x555649e67be0 2023-07-29T16:14:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a357fa700 1 -- 172.21.15.31:0/1639829674 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3a480044f0 con 0x555649e67be0 2023-07-29T16:14:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a357fa700 1 -- 172.21.15.31:0/1639829674 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a48016330 con 0x555649e67be0 2023-07-29T16:14:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 10 client.15333 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555649bcfc60 con 0x555649e67be0 2023-07-29T16:14:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a357fa700 1 -- 172.21.15.31:0/1639829674 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555649de3d10 con 0x555649e67be0 2023-07-29T16:14:06.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a357fa700 1 -- 172.21.15.31:0/1639829674 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3a480997b0 con 0x555649e67be0 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 10 client.15333 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 10 client.15333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 20 client.15333 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649e7d4d0 0x555649e7f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 4 client.15333 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.295+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555649e36d70 con 0x555649e7d4d0 2023-07-29T16:14:06.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.296+0000 7f3a5159a700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649e7d4d0 0x555649e7f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.296+0000 7f3a5159a700 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649e7d4d0 0x555649e7f8e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edaf60f6f3543257 server_cookie=bcb2d5fb0f345bf3 in_seq=0 out_seq=0 2023-07-29T16:14:06.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.296+0000 7f3a357fa700 10 client.15333 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.297+0000 7f3a357fa700 1 -- 172.21.15.31:0/1639829674 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555649e36d70 con 0x555649e7d4d0 2023-07-29T16:14:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.297+0000 7f3a357fa700 10 client.15333 handle_command_reply: tid=0 2023-07-29T16:14:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.297+0000 7f3a679d7b80 1 client.15333 shutdown 2023-07-29T16:14:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.297+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 msgr2=0x555649e717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.297+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e717c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800c5b0 tx=0x7f3a480993a0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 shutdown_connections 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649e7d4d0 0x555649e7f8e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e78d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e717c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a357fa700 0 client.15333 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1639829674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e71d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 >> 172.21.15.31:0/1639829674 conn(0x555649e5d3e0 msgr2=0x555649e62060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 shutdown_connections 2023-07-29T16:14:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1639829674 wait complete. 2023-07-29T16:14:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.298+0000 7f3a679d7b80 20 client.15333 trim_cache size 0 max 16384 2023-07-29T16:14:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 Processor -- start 2023-07-29T16:14:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 -- start start 2023-07-29T16:14:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e6f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e78d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e6c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649e68b50 con 0x555649e67be0 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649e643e0 con 0x555649e68530 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649e701b0 con 0x555649e692f0 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e6f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e6f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43614/0 (socket says 172.21.15.31:43614) 2023-07-29T16:14:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.315+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1409899589 learned_addr learned my addr 172.21.15.31:0/1409899589 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a5159a700 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e78d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a5259c700 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e6c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1409899589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 msgr2=0x555649e6c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e6c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1409899589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 msgr2=0x555649e78d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e78d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a51d9b700 1 -- 172.21.15.31:0/1409899589 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555649c0d250 con 0x555649e67be0 2023-07-29T16:14:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a5259c700 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e6c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:14:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e6f6f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800f010 tx=0x7f3a4800cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63cbb8721880853 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a40ff9700 1 -- 172.21.15.31:0/1409899589 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48099090 con 0x555649e67be0 2023-07-29T16:14:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a40ff9700 1 -- 172.21.15.31:0/1409899589 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a480044f0 con 0x555649e67be0 2023-07-29T16:14:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.316+0000 7f3a40ff9700 1 -- 172.21.15.31:0/1409899589 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4809d170 con 0x555649e67be0 2023-07-29T16:14:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1409899589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 msgr2=0x555649e6f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e6f6f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800f010 tx=0x7f3a4800cf10 comp rx=0 tx=0).stop 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1409899589 shutdown_connections 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e692f0 0x555649e6c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e67be0 0x555649e6f6f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/1409899589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e78d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1409899589 >> 172.21.15.31:0/1409899589 conn(0x555649e5d3e0 msgr2=0x555649e63690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1409899589 shutdown_connections 2023-07-29T16:14:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 -- 172.21.15.31:0/1409899589 wait complete. 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.317+0000 7f3a679d7b80 1 -- start start 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 Processor -- start 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 0x555649e74e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e75370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e692f0 0x555649e78390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649e788d0 con 0x555649e692f0 2023-07-29T16:14:06.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649e78a40 con 0x555649e68530 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555649e78c00 con 0x555649e67be0 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.318+0000 7f3a679d7b80 10 client.0 _ll_register_callbacks cb 0x555649d5e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e692f0 0x555649e78390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 0x555649e74e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 0x555649e74e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54512/0 (socket says 172.21.15.31:54512) 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 -- 172.21.15.31:0/3645953256 learned_addr learned my addr 172.21.15.31:0/3645953256 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a5159a700 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e75370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 -- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 msgr2=0x555649e75370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e75370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 -- 172.21.15.31:0/3645953256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e692f0 msgr2=0x555649e78390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e692f0 0x555649e78390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 -- 172.21.15.31:0/3645953256 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555649bcfc60 con 0x555649e67be0 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a51d9b700 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 0x555649e74e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cf490 tx=0x7f3a48005bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdcc82ab23feaa1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.319+0000 7f3a357fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:14:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a357fa700 1 -- 172.21.15.31:0/3645953256 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48098c40 con 0x555649e67be0 2023-07-29T16:14:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a357fa700 1 -- 172.21.15.31:0/3645953256 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3a48063060 con 0x555649e67be0 2023-07-29T16:14:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a3c01e040 con 0x555649e67be0 2023-07-29T16:14:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a357fa700 1 -- 172.21.15.31:0/3645953256 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a48099090 con 0x555649e67be0 2023-07-29T16:14:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 10 client.25327 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555649be52b0 con 0x555649e67be0 2023-07-29T16:14:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a357fa700 1 -- 172.21.15.31:0/3645953256 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a3c01e040 con 0x555649e67be0 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a357fa700 1 -- 172.21.15.31:0/3645953256 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3a480986b0 con 0x555649e67be0 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 10 client.25327 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 10 client.25327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 20 client.25327 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649ecfbe0 0x555649ed1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 4 client.25327 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:14:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.320+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555649db1d40 con 0x555649ecfbe0 2023-07-29T16:14:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.321+0000 7f3a5159a700 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649ecfbe0 0x555649ed1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.321+0000 7f3a5159a700 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649ecfbe0 0x555649ed1fc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a4be7c051c50ec server_cookie=b398768ceebf312b in_seq=0 out_seq=0 2023-07-29T16:14:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.321+0000 7f3a357fa700 10 client.25327 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a357fa700 1 -- 172.21.15.31:0/3645953256 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555649db1d40 con 0x555649ecfbe0 2023-07-29T16:14:06.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a357fa700 10 client.25327 handle_command_reply: tid=0 2023-07-29T16:14:06.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 client.25327 shutdown 2023-07-29T16:14:06.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 msgr2=0x555649e74e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 0x555649e74e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cf490 tx=0x7f3a48005bb0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 shutdown_connections 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555649ecfbe0 0x555649ed1fc0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555649e692f0 0x555649e78390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a357fa700 0 client.25327 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555649e67be0 0x555649e74e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 --2- 172.21.15.31:0/3645953256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555649e68530 0x555649e75370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 >> 172.21.15.31:0/3645953256 conn(0x555649e5d3e0 msgr2=0x555649e6f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 shutdown_connections 2023-07-29T16:14:06.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 1 -- 172.21.15.31:0/3645953256 wait complete. 2023-07-29T16:14:06.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.322+0000 7f3a679d7b80 20 client.25327 trim_cache size 0 max 16384 2023-07-29T16:14:06.329 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:14:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "6ad0281a-2dcd-4fa3-931c-549c5cd7db01": { 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "6ad0281a-2dcd-4fa3-931c-549c5cd7db01", 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:14:06.330 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:14:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a380a0790 msgr2=0x7f3a380a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a380a0790 0x7f3a380a2c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x555649e67980 tx=0x7f3a44009040 comp rx=0 tx=0).stop 2023-07-29T16:14:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 msgr2=0x7f3a4c074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c074b00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e9c0 tx=0x7f3a4800ecd0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 shutdown_connections 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a380a0790 0x7f3a380a2c40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a4c101eb0 0x7f3a4c073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a4c100330 0x7f3a4c074b00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 --2- 172.21.15.31:0/1105393799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a4c101520 0x7f3a4c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 >> 172.21.15.31:0/1105393799 conn(0x7f3a4c0fb970 msgr2=0x7f3a4c0fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 shutdown_connections 2023-07-29T16:14:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:06.323+0000 7f3a53fff700 1 -- 172.21.15.31:0/1105393799 wait complete. 2023-07-29T16:14:06.339 INFO:tasks.cephfs.filesystem:scrub status for tag:6ad0281a-2dcd-4fa3-931c-549c5cd7db01 - {'path': '/', 'tag': '6ad0281a-2dcd-4fa3-931c-549c5cd7db01', 'options': 'recursive,force'} 2023-07-29T16:14:07.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:06 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:14:07.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:07.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:14:07.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:07.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:06 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:14:07.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:07.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:14:07.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:07.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:06 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:14:07.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:07.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:14:07.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:08.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:07 smithi156 ceph-mon[129245]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2023-07-29T16:14:08.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:07 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:14:08.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:07 smithi031 ceph-mon[95504]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2023-07-29T16:14:08.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:07 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:14:08.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:07 smithi039 ceph-mon[128858]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2023-07-29T16:14:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:07 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:14:10.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:10 smithi031 ceph-mon[95504]: pgmap v330: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-07-29T16:14:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:10 smithi039 ceph-mon[128858]: pgmap v330: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-07-29T16:14:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:10 smithi156 ceph-mon[129245]: pgmap v330: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-07-29T16:14:12.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:12 smithi031 ceph-mon[95504]: pgmap v331: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 5 op/s 2023-07-29T16:14:12.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:12 smithi039 ceph-mon[128858]: pgmap v331: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 5 op/s 2023-07-29T16:14:12.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:12 smithi156 ceph-mon[129245]: pgmap v331: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 5 op/s 2023-07-29T16:14:14.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:14 smithi031 ceph-mon[95504]: pgmap v332: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-29T16:14:14.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:14 smithi039 ceph-mon[128858]: pgmap v332: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-29T16:14:14.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:14 smithi156 ceph-mon[129245]: pgmap v332: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 7 op/s 2023-07-29T16:14:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:16 smithi031 ceph-mon[95504]: pgmap v333: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:16 smithi039 ceph-mon[128858]: pgmap v333: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:16 smithi156 ceph-mon[129245]: pgmap v333: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:17 smithi156 ceph-mon[129245]: pgmap v334: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2023-07-29T16:14:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:17 smithi031 ceph-mon[95504]: pgmap v334: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2023-07-29T16:14:18.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:17 smithi039 ceph-mon[128858]: pgmap v334: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2023-07-29T16:14:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 -- 172.21.15.31:0/80505085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 msgr2=0x7fd00c100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 --2- 172.21.15.31:0/80505085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c100940 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009990 tx=0x7fcff8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 -- 172.21.15.31:0/80505085 shutdown_connections 2023-07-29T16:14:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 --2- 172.21.15.31:0/80505085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 0x7fd00c107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 --2- 172.21.15.31:0/80505085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd00c0ff240 0x7fd00c0ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 --2- 172.21.15.31:0/80505085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c100940 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 -- 172.21.15.31:0/80505085 >> 172.21.15.31:0/80505085 conn(0x7fd00c0fa840 msgr2=0x7fd00c0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:19.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 -- 172.21.15.31:0/80505085 shutdown_connections 2023-07-29T16:14:19.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.033+0000 7fd0131ff700 1 -- 172.21.15.31:0/80505085 wait complete. 2023-07-29T16:14:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.034+0000 7fd0131ff700 1 Processor -- start 2023-07-29T16:14:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.034+0000 7fd0131ff700 1 -- start start 2023-07-29T16:14:19.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd0131ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd00c0ff240 0x7fd00c196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd0131ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd0131ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 0x7fd00c1974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd0131ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c197b50 con 0x7fd00c1004d0 2023-07-29T16:14:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd0131ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c197cc0 con 0x7fd00c100e80 2023-07-29T16:14:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd0131ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c197e80 con 0x7fd00c0ff240 2023-07-29T16:14:19.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd01179c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 0x7fd00c1974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52502/0 (socket says 172.21.15.31:52502) 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd00bfff700 1 -- 172.21.15.31:0/238991369 learned_addr learned my addr 172.21.15.31:0/238991369 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd010f9b700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd00c0ff240 0x7fd00c196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd01179c700 1 -- 172.21.15.31:0/238991369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd00c0ff240 msgr2=0x7fd00c196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd01179c700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd00c0ff240 0x7fd00c196130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd01179c700 1 -- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 msgr2=0x7fd00c196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd01179c700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.035+0000 7fd01179c700 1 -- 172.21.15.31:0/238991369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd000009520 con 0x7fd00c100e80 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.036+0000 7fd01179c700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 0x7fd00c1974f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd00400efc0 tx=0x7fd00400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.036+0000 7fd009ffb700 1 -- 172.21.15.31:0/238991369 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd004018070 con 0x7fd00c100e80 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.036+0000 7fd009ffb700 1 -- 172.21.15.31:0/238991369 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd00400f040 con 0x7fd00c100e80 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.036+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff80095f0 con 0x7fd00c100e80 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.036+0000 7fd009ffb700 1 -- 172.21.15.31:0/238991369 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd004007bb0 con 0x7fd00c100e80 2023-07-29T16:14:19.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.036+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c19e740 con 0x7fd00c100e80 2023-07-29T16:14:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.040+0000 7fd009ffb700 1 -- 172.21.15.31:0/238991369 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd00400c940 con 0x7fd00c100e80 2023-07-29T16:14:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 Processor -- start 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 -- start start 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f247788cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247789bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2476dc9b0 con 0x55f247789200 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2476da370 con 0x55f2475e06b0 2023-07-29T16:14:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2476d5c60 con 0x55f24778a100 2023-07-29T16:14:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd009ffb700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcffc0a0790 0x7fcffc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd009ffb700 1 -- 172.21.15.31:0/238991369 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd0040cbbd0 con 0x7fd00c100e80 2023-07-29T16:14:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd010f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247789bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd010f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247789bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52516/0 (socket says 172.21.15.31:52516) 2023-07-29T16:14:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.041+0000 7fd010f9b700 1 -- 172.21.15.31:0/1894205039 learned_addr learned my addr 172.21.15.31:0/1894205039 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:19.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd01179c700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcffc0a0790 0x7fcffc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd00bfff700 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f247788cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd00bfff700 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd010f9b700 1 -- 172.21.15.31:0/1894205039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 msgr2=0x55f24778a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd010f9b700 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd010f9b700 1 -- 172.21.15.31:0/1894205039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 msgr2=0x55f247788cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd010f9b700 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f247788cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd010f9b700 1 -- 172.21.15.31:0/1894205039 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f247503d80 con 0x55f247789200 2023-07-29T16:14:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd01179c700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcffc0a0790 0x7fcffc0a2c40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd004009fd0 tx=0x7fd00400cf70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.042+0000 7fd010f9b700 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247789bc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd000009f10 tx=0x7fd000011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2223a2bd1a9467f3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fcff6ffd700 1 -- 172.21.15.31:0/1894205039 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000019630 con 0x55f247789200 2023-07-29T16:14:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fcff6ffd700 1 -- 172.21.15.31:0/1894205039 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd000019c50 con 0x55f247789200 2023-07-29T16:14:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fcff6ffd700 1 -- 172.21.15.31:0/1894205039 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00000ebd0 con 0x55f247789200 2023-07-29T16:14:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1894205039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 msgr2=0x55f247789bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247789bc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd000009f10 tx=0x7fd000011720 comp rx=0 tx=0).stop 2023-07-29T16:14:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1894205039 shutdown_connections 2023-07-29T16:14:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f247788cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1894205039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247789bc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1894205039 >> 172.21.15.31:0/1894205039 conn(0x55f247779fd0 msgr2=0x55f24777f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.043+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1894205039 shutdown_connections 2023-07-29T16:14:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.044+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1894205039 wait complete. 2023-07-29T16:14:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.044+0000 7fd026ba4b80 1 -- start start 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.044+0000 7fd026ba4b80 1 Processor -- start 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 0x55f247792300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247792840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f247799880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2476724b0 con 0x55f2475e06b0 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247672150 con 0x55f247789200 2023-07-29T16:14:19.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2475598c0 con 0x55f24778a100 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd026ba4b80 10 client.0 _ll_register_callbacks cb 0x55f24767b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 0x55f247792300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.045+0000 7fd010f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247792840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247792840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51322/0 (socket says 172.21.15.31:51322) 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 -- 172.21.15.31:0/3286153607 learned_addr learned my addr 172.21.15.31:0/3286153607 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 0x55f247792300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52520/0 (socket says 172.21.15.31:52520) 2023-07-29T16:14:19.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f247799880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 -- 172.21.15.31:0/3286153607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 msgr2=0x55f247799880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f247799880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 -- 172.21.15.31:0/3286153607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 msgr2=0x55f247792300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 0x55f247792300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 -- 172.21.15.31:0/3286153607 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd000009b70 con 0x55f247789200 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 0x55f247792300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fd010f9b700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247792840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd000009840 tx=0x7fd00000ebe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e3cb3c8edb0ae5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fcfebfff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.046+0000 7fcfebfff700 1 -- 172.21.15.31:0/3286153607 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000019dc0 con 0x55f247789200 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f247700cb0 con 0x55f247789200 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fcfebfff700 1 -- 172.21.15.31:0/3286153607 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd000019640 con 0x55f247789200 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fcfebfff700 1 -- 172.21.15.31:0/3286153607 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd000019800 con 0x55f247789200 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fd026ba4b80 10 client.25331 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f2474ecc00 con 0x55f247789200 2023-07-29T16:14:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fcfebfff700 1 -- 172.21.15.31:0/3286153607 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f247700cb0 con 0x55f247789200 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.047+0000 7fcfebfff700 1 -- 172.21.15.31:0/3286153607 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd000029020 con 0x55f247789200 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd026ba4b80 10 client.25331 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd026ba4b80 10 client.25331 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd026ba4b80 20 client.25331 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f24779df00 0x55f2477a0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd026ba4b80 4 client.25331 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f247753d10 con 0x55f24779df00 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f24779df00 0x55f2477a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f24779df00 0x55f2477a0310 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1da5f8dc8053254d server_cookie=9540fc50e5e8c70e in_seq=0 out_seq=0 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.048+0000 7fcfebfff700 10 client.25331 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.049+0000 7fcfebfff700 1 -- 172.21.15.31:0/3286153607 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f247753d10 con 0x55f24779df00 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.049+0000 7fcfebfff700 10 client.25331 handle_command_reply: tid=0 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 client.25331 shutdown 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 msgr2=0x55f247792840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247792840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd000009840 tx=0x7fd00000ebe0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 shutdown_connections 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f24779df00 0x55f2477a0310 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f247799880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2475e06b0 0x55f247792300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3286153607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247792840 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fcfebfff700 0 client.25331 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 >> 172.21.15.31:0/3286153607 conn(0x55f247779fd0 msgr2=0x55f24777ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 shutdown_connections 2023-07-29T16:14:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3286153607 wait complete. 2023-07-29T16:14:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.050+0000 7fd026ba4b80 20 client.25331 trim_cache size 0 max 16384 2023-07-29T16:14:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.065+0000 7fd026ba4b80 1 Processor -- start 2023-07-29T16:14:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.065+0000 7fd026ba4b80 1 -- start start 2023-07-29T16:14:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f2477a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247799840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24778aa10 con 0x55f247789200 2023-07-29T16:14:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247790230 con 0x55f2475e06b0 2023-07-29T16:14:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24778db20 con 0x55f24778a100 2023-07-29T16:14:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f2477a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f2477a02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51328/0 (socket says 172.21.15.31:51328) 2023-07-29T16:14:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 -- 172.21.15.31:0/3051855783 learned_addr learned my addr 172.21.15.31:0/3051855783 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd010f9b700 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247799840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 -- 172.21.15.31:0/3051855783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 msgr2=0x55f24778a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 -- 172.21.15.31:0/3051855783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 msgr2=0x55f247799840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247799840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.066+0000 7fd00bfff700 1 -- 172.21.15.31:0/3051855783 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f24752a1f0 con 0x55f2475e06b0 2023-07-29T16:14:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.067+0000 7fd00bfff700 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f2477a02e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcff80059d0 tx=0x7fcff801a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c70b8c0df20502b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.067+0000 7fcff77fe700 1 -- 172.21.15.31:0/3051855783 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8029890 con 0x55f2475e06b0 2023-07-29T16:14:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.067+0000 7fcff77fe700 1 -- 172.21.15.31:0/3051855783 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcff801e040 con 0x55f2475e06b0 2023-07-29T16:14:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.067+0000 7fcff77fe700 1 -- 172.21.15.31:0/3051855783 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff800fa90 con 0x55f2475e06b0 2023-07-29T16:14:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3051855783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 msgr2=0x55f2477a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f2477a02e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcff80059d0 tx=0x7fcff801a040 comp rx=0 tx=0).stop 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3051855783 shutdown_connections 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f24778a100 0x55f24778a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2475e06b0 0x55f2477a02e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/3051855783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f247789200 0x55f247799840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3051855783 >> 172.21.15.31:0/3051855783 conn(0x55f247779fd0 msgr2=0x55f247791050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3051855783 shutdown_connections 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 -- 172.21.15.31:0/3051855783 wait complete. 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.068+0000 7fd026ba4b80 1 -- start start 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 Processor -- start 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2475e06b0 0x55f247787de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247788320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f24778a100 0x55f2477848a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247784fb0 con 0x55f24778a100 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247785120 con 0x55f247789200 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2477852e0 con 0x55f2475e06b0 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.069+0000 7fd026ba4b80 10 client.0 _ll_register_callbacks cb 0x55f24767b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2475e06b0 0x55f247787de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f24778a100 0x55f2477848a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247788320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd00bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2475e06b0 0x55f247787de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36748/0 (socket says 172.21.15.31:36748) 2023-07-29T16:14:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd00bfff700 1 -- 172.21.15.31:0/1873487454 learned_addr learned my addr 172.21.15.31:0/1873487454 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247788320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51330/0 (socket says 172.21.15.31:51330) 2023-07-29T16:14:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 -- 172.21.15.31:0/1873487454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2475e06b0 msgr2=0x55f247787de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2475e06b0 0x55f247787de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 -- 172.21.15.31:0/1873487454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f24778a100 msgr2=0x55f2477848a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f24778a100 0x55f2477848a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 -- 172.21.15.31:0/1873487454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff8034040 con 0x55f247789200 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fd010f9b700 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247788320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd00000a010 tx=0x7fd000004920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29504490844a70f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.070+0000 7fcfebfff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fcfebfff700 1 -- 172.21.15.31:0/1873487454 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd000004d70 con 0x55f247789200 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fcfebfff700 1 -- 172.21.15.31:0/1873487454 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd000019640 con 0x55f247789200 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd000019c50 con 0x55f247789200 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fcfebfff700 1 -- 172.21.15.31:0/1873487454 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd000019800 con 0x55f247789200 2023-07-29T16:14:19.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 10 client.25337 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f247502250 con 0x55f247789200 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fcfebfff700 1 -- 172.21.15.31:0/1873487454 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd000019c50 con 0x55f247789200 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fcfebfff700 1 -- 172.21.15.31:0/1873487454 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd000011790 con 0x55f247789200 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 10 client.25337 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 10 client.25337 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 20 client.25337 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2477f4510 0x55f2477f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 4 client.25337 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:14:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.071+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f2476cece0 con 0x55f2477f4510 2023-07-29T16:14:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.072+0000 7fd00bfff700 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2477f4510 0x55f2477f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.072+0000 7fd00bfff700 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2477f4510 0x55f2477f68f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95fc117c57d228e1 server_cookie=2ecd1354782c873b in_seq=0 out_seq=0 2023-07-29T16:14:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.072+0000 7fcfebfff700 10 client.25337 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fcfebfff700 1 -- 172.21.15.31:0/1873487454 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f2476cece0 con 0x55f2477f4510 2023-07-29T16:14:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fcfebfff700 10 client.25337 handle_command_reply: tid=0 2023-07-29T16:14:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 client.25337 shutdown 2023-07-29T16:14:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 msgr2=0x55f247788320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247788320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd00000a010 tx=0x7fd000004920 comp rx=0 tx=0).stop 2023-07-29T16:14:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 shutdown_connections 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2477f4510 0x55f2477f68f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f24778a100 0x55f2477848a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2475e06b0 0x55f247787de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 --2- 172.21.15.31:0/1873487454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f247789200 0x55f247788320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fcfebfff700 0 client.25337 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 >> 172.21.15.31:0/1873487454 conn(0x55f247779fd0 msgr2=0x55f24778cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 shutdown_connections 2023-07-29T16:14:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.084+0000 7fd026ba4b80 1 -- 172.21.15.31:0/1873487454 wait complete. 2023-07-29T16:14:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd026ba4b80 20 client.25337 trim_cache size 0 max 16384 2023-07-29T16:14:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:14:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:14:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:14:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:14:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcffc0a0790 msgr2=0x7fcffc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcffc0a0790 0x7fcffc0a2c40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd004009fd0 tx=0x7fd00400cf70 comp rx=0 tx=0).stop 2023-07-29T16:14:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 msgr2=0x7fd00c1974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 0x7fd00c1974f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd00400efc0 tx=0x7fd00400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 shutdown_connections 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcffc0a0790 0x7fcffc0a2c40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd00c100e80 0x7fd00c1974f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd00c0ff240 0x7fd00c196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 --2- 172.21.15.31:0/238991369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd00c1004d0 0x7fd00c196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.085+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 >> 172.21.15.31:0/238991369 conn(0x7fd00c0fa840 msgr2=0x7fd00c068180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:19.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.086+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 shutdown_connections 2023-07-29T16:14:19.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:19.086+0000 7fd0131ff700 1 -- 172.21.15.31:0/238991369 wait complete. 2023-07-29T16:14:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:20 smithi031 ceph-mon[95504]: pgmap v335: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 8 op/s 2023-07-29T16:14:20.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:20 smithi039 ceph-mon[128858]: pgmap v335: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 8 op/s 2023-07-29T16:14:20.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:20 smithi156 ceph-mon[129245]: pgmap v335: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 8 op/s 2023-07-29T16:14:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:22 smithi031 ceph-mon[95504]: pgmap v336: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:14:22.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:22 smithi039 ceph-mon[128858]: pgmap v336: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:14:22.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:22 smithi156 ceph-mon[129245]: pgmap v336: 97 pgs: 97 active+clean; 648 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:14:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:24 smithi031 ceph-mon[95504]: pgmap v337: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 9 op/s 2023-07-29T16:14:24.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:24 smithi039 ceph-mon[128858]: pgmap v337: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 9 op/s 2023-07-29T16:14:24.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:24 smithi156 ceph-mon[129245]: pgmap v337: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 9 op/s 2023-07-29T16:14:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:26 smithi031 ceph-mon[95504]: pgmap v338: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-07-29T16:14:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:26 smithi039 ceph-mon[128858]: pgmap v338: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-07-29T16:14:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:26 smithi156 ceph-mon[129245]: pgmap v338: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-07-29T16:14:28.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:27 smithi156 ceph-mon[129245]: pgmap v339: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-07-29T16:14:28.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:27 smithi031 ceph-mon[95504]: pgmap v339: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-07-29T16:14:28.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:27 smithi039 ceph-mon[128858]: pgmap v339: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-07-29T16:14:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:30 smithi031 ceph-mon[95504]: pgmap v340: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2023-07-29T16:14:30.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:30 smithi039 ceph-mon[128858]: pgmap v340: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2023-07-29T16:14:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:30 smithi156 ceph-mon[129245]: pgmap v340: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2023-07-29T16:14:31.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:14:31.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:14:31.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:14:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 -- 172.21.15.31:0/434603812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 msgr2=0x7faee8108810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 --2- 172.21.15.31:0/434603812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8108810 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faee0009f90 tx=0x7faee000b330 comp rx=0 tx=0).stop 2023-07-29T16:14:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 -- 172.21.15.31:0/434603812 shutdown_connections 2023-07-29T16:14:32.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 --2- 172.21.15.31:0/434603812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8108810 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 --2- 172.21.15.31:0/434603812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faee81003d0 0x7faee81007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 --2- 172.21.15.31:0/434603812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81015c0 0x7faee8101a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 -- 172.21.15.31:0/434603812 >> 172.21.15.31:0/434603812 conn(0x7faee80fb9d0 msgr2=0x7faee80fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 -- 172.21.15.31:0/434603812 shutdown_connections 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.244+0000 7faeed9ff700 1 -- 172.21.15.31:0/434603812 wait complete. 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.245+0000 7faeed9ff700 1 Processor -- start 2023-07-29T16:14:32.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.245+0000 7faeed9ff700 1 -- start start 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faeed9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 0x7faee8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faeed9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faee81015c0 0x7faee8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faeed9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faeed9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81a01c0 con 0x7faee8101f50 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faeed9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81a0300 con 0x7faee81003d0 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faeed9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee81a0440 con 0x7faee81015c0 2023-07-29T16:14:32.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 0x7faee8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faedffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faee81015c0 0x7faee8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 0x7faee8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50744/0 (socket says 172.21.15.31:50744) 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 -- 172.21.15.31:0/3897629438 learned_addr learned my addr 172.21.15.31:0/3897629438 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee77fe700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 -- 172.21.15.31:0/3897629438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faee81015c0 msgr2=0x7faee8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faee81015c0 0x7faee8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 -- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 msgr2=0x7faee8073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee6ffd700 1 -- 172.21.15.31:0/3897629438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed80095f0 con 0x7faee81003d0 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.246+0000 7faee77fe700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.247+0000 7faee6ffd700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 0x7faee8073140 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faed800ebf0 tx=0x7faed800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.247+0000 7faeec9fd700 1 -- 172.21.15.31:0/3897629438 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8009a10 con 0x7faee81003d0 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.247+0000 7faeec9fd700 1 -- 172.21.15.31:0/3897629438 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faed800cd40 con 0x7faee81003d0 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.247+0000 7faeec9fd700 1 -- 172.21.15.31:0/3897629438 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8018820 con 0x7faee81003d0 2023-07-29T16:14:32.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.247+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed0009520 con 0x7faee81003d0 2023-07-29T16:14:32.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.247+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee0009c40 con 0x7faee81003d0 2023-07-29T16:14:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.250+0000 7faeec9fd700 1 -- 172.21.15.31:0/3897629438 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7faed8018980 con 0x7faee81003d0 2023-07-29T16:14:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.251+0000 7faf01377b80 1 Processor -- start 2023-07-29T16:14:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 -- start start 2023-07-29T16:14:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84da7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84dacf0 0x5606c84db160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84319b0 con 0x5606c84d9df0 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c842f370 con 0x5606c83356b0 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faf01377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c842ac60 con 0x5606c84dacf0 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faeec9fd700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faed40a0790 0x7faed40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84da7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faee77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faedffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84dacf0 0x5606c84db160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faee77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84d98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50746/0 (socket says 172.21.15.31:50746) 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faee77fe700 1 -- 172.21.15.31:0/3389653621 learned_addr learned my addr 172.21.15.31:0/3389653621 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faeec9fd700 1 -- 172.21.15.31:0/3897629438 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7faed8014070 con 0x7faee81003d0 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.252+0000 7faee77fe700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faed40a0790 0x7faed40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 -- 172.21.15.31:0/3389653621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84dacf0 msgr2=0x5606c84db160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84dacf0 0x5606c84db160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 -- 172.21.15.31:0/3389653621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 msgr2=0x5606c84da7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84da7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 -- 172.21.15.31:0/3389653621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606c8258d80 con 0x5606c83356b0 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84d98b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faee0015000 tx=0x7faee0008b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=239f76cae5e29ddf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faee77fe700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faed40a0790 0x7faed40a2c40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faee0024000 tx=0x7faee0026000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faeddffb700 1 -- 172.21.15.31:0/3389653621 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee000e040 con 0x5606c83356b0 2023-07-29T16:14:32.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faeddffb700 1 -- 172.21.15.31:0/3389653621 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faee000be50 con 0x5606c83356b0 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.253+0000 7faeddffb700 1 -- 172.21.15.31:0/3389653621 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee001fac0 con 0x5606c83356b0 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 -- 172.21.15.31:0/3389653621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 msgr2=0x5606c84d98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84d98b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faee0015000 tx=0x7faee0008b90 comp rx=0 tx=0).stop 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 -- 172.21.15.31:0/3389653621 shutdown_connections 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84dacf0 0x5606c84db160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84d98b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 --2- 172.21.15.31:0/3389653621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84da7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 -- 172.21.15.31:0/3389653621 >> 172.21.15.31:0/3389653621 conn(0x5606c84cefd0 msgr2=0x5606c84d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 -- 172.21.15.31:0/3389653621 shutdown_connections 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.254+0000 7faf01377b80 1 -- 172.21.15.31:0/3389653621 wait complete. 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.255+0000 7faf01377b80 1 -- start start 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.255+0000 7faf01377b80 1 Processor -- start 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84e33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84ea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c83c74b0 con 0x5606c84dacf0 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c83c7150 con 0x5606c83356b0 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c82ae8c0 con 0x5606c84d9df0 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faf01377b80 10 client.0 _ll_register_callbacks cb 0x5606c83d05c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50758/0 (socket says 172.21.15.31:50758) 2023-07-29T16:14:32.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faee6ffd700 1 -- 172.21.15.31:0/4014103990 learned_addr learned my addr 172.21.15.31:0/4014103990 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faedffff700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84e33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.256+0000 7faee77fe700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faee6ffd700 1 -- 172.21.15.31:0/4014103990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 msgr2=0x5606c84e33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faee6ffd700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84e33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faee6ffd700 1 -- 172.21.15.31:0/4014103990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 msgr2=0x5606c84ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faee6ffd700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faee6ffd700 1 -- 172.21.15.31:0/4014103990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee8109140 con 0x5606c83356b0 2023-07-29T16:14:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faee6ffd700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e2e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faed8000c00 tx=0x7faed8099ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e1430a4c6d31b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:32.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faec6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faec6ffd700 1 -- 172.21.15.31:0/4014103990 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed80a1b20 con 0x5606c83356b0 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faec6ffd700 1 -- 172.21.15.31:0/4014103990 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7faed8094160 con 0x5606c83356b0 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5606c8455cb0 con 0x5606c83356b0 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faec6ffd700 1 -- 172.21.15.31:0/4014103990 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faed8099960 con 0x5606c83356b0 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faf01377b80 10 client.25349 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.257+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5606c8241c00 con 0x5606c83356b0 2023-07-29T16:14:32.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faec6ffd700 1 -- 172.21.15.31:0/4014103990 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5606c8455cb0 con 0x5606c83356b0 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faec6ffd700 1 -- 172.21.15.31:0/4014103990 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7faed809a740 con 0x5606c83356b0 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faf01377b80 10 client.25349 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faf01377b80 10 client.25349 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faf01377b80 20 client.25349 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faf01377b80 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c84eebd0 0x5606c84f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faf01377b80 4 client.25349 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:32.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606c84a8d10 con 0x5606c84eebd0 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faedffff700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c84eebd0 0x5606c84f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faedffff700 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c84eebd0 0x5606c84f0fe0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21205de2a366b040 server_cookie=7a4e4c5aebb66901 in_seq=0 out_seq=0 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.258+0000 7faec6ffd700 10 client.25349 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.259+0000 7faec6ffd700 1 -- 172.21.15.31:0/4014103990 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5606c84a8d10 con 0x5606c84eebd0 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.259+0000 7faec6ffd700 10 client.25349 handle_command_reply: tid=0 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.259+0000 7faf01377b80 1 client.25349 shutdown 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.259+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 msgr2=0x5606c84e2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.259+0000 7faf01377b80 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e2e80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7faed8000c00 tx=0x7faed8099ea0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 shutdown_connections 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c84eebd0 0x5606c84f0fe0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faec6ffd700 0 client.25349 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e2e80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 --2- 172.21.15.31:0/4014103990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84e33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 >> 172.21.15.31:0/4014103990 conn(0x5606c84cefd0 msgr2=0x5606c84d3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 shutdown_connections 2023-07-29T16:14:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 1 -- 172.21.15.31:0/4014103990 wait complete. 2023-07-29T16:14:32.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.260+0000 7faf01377b80 20 client.25349 trim_cache size 0 max 16384 2023-07-29T16:14:32.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.275+0000 7faf01377b80 1 Processor -- start 2023-07-29T16:14:32.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.275+0000 7faf01377b80 1 -- start start 2023-07-29T16:14:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faf01377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faf01377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84db0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faf01377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84e1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faf01377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84db600 con 0x5606c84dacf0 2023-07-29T16:14:32.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faf01377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84de730 con 0x5606c83356b0 2023-07-29T16:14:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faf01377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84ee960 con 0x5606c84d9df0 2023-07-29T16:14:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faedffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84db0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faedffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84db0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56760/0 (socket says 172.21.15.31:56760) 2023-07-29T16:14:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50764/0 (socket says 172.21.15.31:50764) 2023-07-29T16:14:32.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faedffff700 1 -- 172.21.15.31:0/1285555182 learned_addr learned my addr 172.21.15.31:0/1285555182 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.276+0000 7faee77fe700 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84e1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faedffff700 1 -- 172.21.15.31:0/1285555182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 msgr2=0x5606c84e83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faedffff700 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faedffff700 1 -- 172.21.15.31:0/1285555182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 msgr2=0x5606c84e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faedffff700 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84e1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faedffff700 1 -- 172.21.15.31:0/1285555182 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606c827f1f0 con 0x5606c84d9df0 2023-07-29T16:14:32.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faedffff700 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84db0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x5606c84f18c0 tx=0x7faed0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4168acc9f8ae586f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:32.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faede7fc700 1 -- 172.21.15.31:0/1285555182 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0017040 con 0x5606c84d9df0 2023-07-29T16:14:32.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faede7fc700 1 -- 172.21.15.31:0/1285555182 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faed0018040 con 0x5606c84d9df0 2023-07-29T16:14:32.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.277+0000 7faede7fc700 1 -- 172.21.15.31:0/1285555182 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0021ca0 con 0x5606c84d9df0 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 -- 172.21.15.31:0/1285555182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 msgr2=0x5606c84db0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84db0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x5606c84f18c0 tx=0x7faed0009260 comp rx=0 tx=0).stop 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 -- 172.21.15.31:0/1285555182 shutdown_connections 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84dacf0 0x5606c84e1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c83356b0 0x5606c84e83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 --2- 172.21.15.31:0/1285555182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c84d9df0 0x5606c84db0c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 -- 172.21.15.31:0/1285555182 >> 172.21.15.31:0/1285555182 conn(0x5606c84cefd0 msgr2=0x5606c84d42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 -- 172.21.15.31:0/1285555182 shutdown_connections 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 -- 172.21.15.31:0/1285555182 wait complete. 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.278+0000 7faf01377b80 1 -- start start 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 Processor -- start 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c83356b0 0x5606c84e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84e8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c84dacf0 0x5606c84e6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84e6cb0 con 0x5606c84d9df0 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84e6e20 con 0x5606c84dacf0 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606c84e6fe0 con 0x5606c83356b0 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faf01377b80 10 client.0 _ll_register_callbacks cb 0x5606c83d05c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faee77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c84dacf0 0x5606c84e6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faee6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c83356b0 0x5606c84e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faee77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c84dacf0 0x5606c84e6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50774/0 (socket says 172.21.15.31:50774) 2023-07-29T16:14:32.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.279+0000 7faee77fe700 1 -- 172.21.15.31:0/4259306049 learned_addr learned my addr 172.21.15.31:0/4259306049 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faedffff700 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faee6ffd700 1 -- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c84dacf0 msgr2=0x5606c84e6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faee6ffd700 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c84dacf0 0x5606c84e6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faee6ffd700 1 -- 172.21.15.31:0/4259306049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 msgr2=0x5606c84e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faee6ffd700 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faee6ffd700 1 -- 172.21.15.31:0/4259306049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed0009da0 con 0x5606c83356b0 2023-07-29T16:14:32.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faee6ffd700 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c83356b0 0x5606c84e7e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faed809aaa0 tx=0x7faed8094390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4c4200ebd1c0a03 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faec6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faec6ffd700 1 -- 172.21.15.31:0/4259306049 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed809d990 con 0x5606c83356b0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faec6ffd700 1 -- 172.21.15.31:0/4259306049 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7faed80c3e40 con 0x5606c83356b0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faee000be50 con 0x5606c83356b0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.280+0000 7faec6ffd700 1 -- 172.21.15.31:0/4259306049 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7faed809a260 con 0x5606c83356b0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 10 client.25363 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5606c8257250 con 0x5606c83356b0 2023-07-29T16:14:32.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faec6ffd700 1 -- 172.21.15.31:0/4259306049 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faee000be50 con 0x5606c83356b0 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faec6ffd700 1 -- 172.21.15.31:0/4259306049 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7faed805fd00 con 0x5606c83356b0 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 10 client.25363 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 10 client.25363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 20 client.25363 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c85415e0 0x5606c85439c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 4 client.25363 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:14:32.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5606c8423ce0 con 0x5606c85415e0 2023-07-29T16:14:32.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.281+0000 7faedffff700 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c85415e0 0x5606c85439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:32.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.282+0000 7faedffff700 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c85415e0 0x5606c85439c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e79bb7afa20730e server_cookie=8aee479d9788a614 in_seq=0 out_seq=0 2023-07-29T16:14:32.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.282+0000 7faec6ffd700 10 client.25363 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:32.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faec6ffd700 1 -- 172.21.15.31:0/4259306049 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5606c8423ce0 con 0x5606c85415e0 2023-07-29T16:14:32.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faec6ffd700 10 client.25363 handle_command_reply: tid=0 2023-07-29T16:14:32.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 client.25363 shutdown 2023-07-29T16:14:32.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c83356b0 msgr2=0x5606c84e7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c83356b0 0x5606c84e7e20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7faed809aaa0 tx=0x7faed8094390 comp rx=0 tx=0).stop 2023-07-29T16:14:32.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 shutdown_connections 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606c85415e0 0x5606c85439c0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606c84dacf0 0x5606c84e6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faec6ffd700 0 client.25363 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606c83356b0 0x5606c84e7e20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 --2- 172.21.15.31:0/4259306049 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606c84d9df0 0x5606c84e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 >> 172.21.15.31:0/4259306049 conn(0x5606c84cefd0 msgr2=0x5606c84e1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:32.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 shutdown_connections 2023-07-29T16:14:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.285+0000 7faf01377b80 1 -- 172.21.15.31:0/4259306049 wait complete. 2023-07-29T16:14:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faf01377b80 20 client.25363 trim_cache size 0 max 16384 2023-07-29T16:14:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:14:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:14:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:14:32.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:14:32.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faed40a0790 msgr2=0x7faed40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faed40a0790 0x7faed40a2c40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7faee0024000 tx=0x7faee0026000 comp rx=0 tx=0).stop 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 msgr2=0x7faee8073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 0x7faee8073140 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faed800ebf0 tx=0x7faed800ef00 comp rx=0 tx=0).stop 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 shutdown_connections 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faed40a0790 0x7faed40a2c40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faee8101f50 0x7faee8073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faee81003d0 0x7faee8073140 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 --2- 172.21.15.31:0/3897629438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faee81015c0 0x7faee8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:32.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 >> 172.21.15.31:0/3897629438 conn(0x7faee80fb9d0 msgr2=0x7faee8104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 shutdown_connections 2023-07-29T16:14:32.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:32.286+0000 7faeed9ff700 1 -- 172.21.15.31:0/3897629438 wait complete. 2023-07-29T16:14:32.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:32 smithi031 ceph-mon[95504]: pgmap v341: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2023-07-29T16:14:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:32 smithi039 ceph-mon[128858]: pgmap v341: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2023-07-29T16:14:32.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:32 smithi156 ceph-mon[129245]: pgmap v341: 97 pgs: 97 active+clean; 649 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2023-07-29T16:14:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:34 smithi031 ceph-mon[95504]: pgmap v342: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2023-07-29T16:14:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:34 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:34 smithi039 ceph-mon[128858]: pgmap v342: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2023-07-29T16:14:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:34 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:34 smithi156 ceph-mon[129245]: pgmap v342: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2023-07-29T16:14:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:34 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:35 smithi156 ceph-mon[129245]: pgmap v343: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-07-29T16:14:36.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:14:36.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:14:36.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:35 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:35 smithi031 ceph-mon[95504]: pgmap v343: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-07-29T16:14:36.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:14:36.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:14:36.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:35 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:35 smithi039 ceph-mon[128858]: pgmap v343: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-07-29T16:14:36.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:14:36.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:14:36.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:35 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:14:36.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:14:36.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 -- 172.21.15.31:0/467603890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 msgr2=0x7facec073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 --2- 172.21.15.31:0/467603890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec073bc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7facd4009990 tx=0x7facd4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 -- 172.21.15.31:0/467603890 shutdown_connections 2023-07-29T16:14:36.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 --2- 172.21.15.31:0/467603890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7facec1081c0 0x7facec10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 --2- 172.21.15.31:0/467603890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facec074d30 0x7facec073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 --2- 172.21.15.31:0/467603890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec073bc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 -- 172.21.15.31:0/467603890 >> 172.21.15.31:0/467603890 conn(0x7facec0fb970 msgr2=0x7facec0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 -- 172.21.15.31:0/467603890 shutdown_connections 2023-07-29T16:14:36.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.574+0000 7facf09ff700 1 -- 172.21.15.31:0/467603890 wait complete. 2023-07-29T16:14:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.575+0000 7facf09ff700 1 Processor -- start 2023-07-29T16:14:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.575+0000 7facf09ff700 1 -- start start 2023-07-29T16:14:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facf09ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facf09ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facec074d30 0x7facec198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facf09ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7facec1081c0 0x7facec19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facf09ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec19fdb0 con 0x7facec073750 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facf09ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec19fef0 con 0x7facec074d30 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facf09ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec1a0080 con 0x7facec1081c0 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52042/0 (socket says 172.21.15.31:52042) 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 -- 172.21.15.31:0/922277542 learned_addr learned my addr 172.21.15.31:0/922277542 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7face3fff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facec074d30 0x7facec198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facead9d700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7facec1081c0 0x7facec19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 -- 172.21.15.31:0/922277542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7facec1081c0 msgr2=0x7facec19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7facec1081c0 0x7facec19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 -- 172.21.15.31:0/922277542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facec074d30 msgr2=0x7facec198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facec074d30 0x7facec198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.576+0000 7facea59c700 1 -- 172.21.15.31:0/922277542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40095f0 con 0x7facec073750 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.577+0000 7facea59c700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec198230 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7facdc00e9a0 tx=0x7facdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.577+0000 7face37fe700 1 -- 172.21.15.31:0/922277542 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc009790 con 0x7facec073750 2023-07-29T16:14:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.577+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec1a0360 con 0x7facec073750 2023-07-29T16:14:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.577+0000 7face37fe700 1 -- 172.21.15.31:0/922277542 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facdc00cbd0 con 0x7facec073750 2023-07-29T16:14:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.577+0000 7face37fe700 1 -- 172.21.15.31:0/922277542 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc018680 con 0x7facec073750 2023-07-29T16:14:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.577+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec1a08c0 con 0x7facec073750 2023-07-29T16:14:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.580+0000 7face37fe700 1 -- 172.21.15.31:0/922277542 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7facdc00cd40 con 0x7facec073750 2023-07-29T16:14:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.582+0000 7fad043dcb80 1 Processor -- start 2023-07-29T16:14:36.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.582+0000 7fad043dcb80 1 -- start start 2023-07-29T16:14:36.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd058ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd059d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd05a2f0 0x5624dd05a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7fad043dcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dcfb0a10 con 0x5624dd05a2f0 2023-07-29T16:14:36.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7fad043dcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dcfae3d0 con 0x5624dd059530 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7fad043dcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dcfa9cc0 con 0x5624dd058be0 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7face37fe700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7facd80a0810 0x7facd80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facea59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd059d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7face3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd05a2f0 0x5624dd05a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7face37fe700 1 -- 172.21.15.31:0/922277542 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7facdc014070 con 0x7facec073750 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facea59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd059d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50812/0 (socket says 172.21.15.31:50812) 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facea59c700 1 -- 172.21.15.31:0/2654334355 learned_addr learned my addr 172.21.15.31:0/2654334355 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facead9d700 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd058ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facead9d700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7facd80a0810 0x7facd80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facea59c700 1 -- 172.21.15.31:0/2654334355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 msgr2=0x5624dd058ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facea59c700 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd058ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.583+0000 7facea59c700 1 -- 172.21.15.31:0/2654334355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd05a2f0 msgr2=0x5624dd05a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7facea59c700 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd05a2f0 0x5624dd05a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7facea59c700 1 -- 172.21.15.31:0/2654334355 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624dcdd7de0 con 0x5624dd059530 2023-07-29T16:14:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7face3fff700 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd05a2f0 0x5624dd05a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:36.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7facea59c700 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd059d60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c5b0 tx=0x7facdc064c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c349914918a5da8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7facead9d700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7facd80a0810 0x7facd80a2cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x5624dd058980 tx=0x7face400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7face0ff9700 1 -- 172.21.15.31:0/2654334355 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc098fd0 con 0x5624dd059530 2023-07-29T16:14:36.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7face0ff9700 1 -- 172.21.15.31:0/2654334355 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facdc098df0 con 0x5624dd059530 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.584+0000 7face0ff9700 1 -- 172.21.15.31:0/2654334355 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0996f0 con 0x5624dd059530 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 -- 172.21.15.31:0/2654334355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 msgr2=0x5624dd059d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd059d60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c5b0 tx=0x7facdc064c80 comp rx=0 tx=0).stop 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 -- 172.21.15.31:0/2654334355 shutdown_connections 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd05a2f0 0x5624dd05a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd058ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2654334355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd059d60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 -- 172.21.15.31:0/2654334355 >> 172.21.15.31:0/2654334355 conn(0x5624dd04e3e0 msgr2=0x5624dd053990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 -- 172.21.15.31:0/2654334355 shutdown_connections 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 -- 172.21.15.31:0/2654334355 wait complete. 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.585+0000 7fad043dcb80 1 -- start start 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 Processor -- start 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd062840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 0x5624dd062d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd069dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dcf46510 con 0x5624dd059530 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dcf461b0 con 0x5624dd05a2f0 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dce2d920 con 0x5624dd058be0 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7fad043dcb80 10 client.0 _ll_register_callbacks cb 0x5624dcf4f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7face3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 0x5624dd062d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7face3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 0x5624dd062d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52058/0 (socket says 172.21.15.31:52058) 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7face3fff700 1 -- 172.21.15.31:0/2553445841 learned_addr learned my addr 172.21.15.31:0/2553445841 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.586+0000 7facea59c700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd062840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7facead9d700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd069dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7face3fff700 1 -- 172.21.15.31:0/2553445841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 msgr2=0x5624dd062840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7face3fff700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd062840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7face3fff700 1 -- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 msgr2=0x5624dd069dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7face3fff700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd069dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7face3fff700 1 -- 172.21.15.31:0/2553445841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec10b260 con 0x5624dd059530 2023-07-29T16:14:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7face3fff700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 0x5624dd062d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7facd4009990 tx=0x7facd4005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15158d7c9fa1f2c4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7facd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7facd1ffb700 1 -- 172.21.15.31:0/2553445841 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facd401d070 con 0x5624dd059530 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624dcfd4d10 con 0x5624dd059530 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.587+0000 7facd1ffb700 1 -- 172.21.15.31:0/2553445841 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7facd4004500 con 0x5624dd059530 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7facd1ffb700 1 -- 172.21.15.31:0/2553445841 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7facd40046c0 con 0x5624dd059530 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 10 client.15387 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7facec1a08c0 con 0x5624dd059530 2023-07-29T16:14:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7facd1ffb700 1 -- 172.21.15.31:0/2553445841 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5624dcfd4d10 con 0x5624dd059530 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7facd1ffb700 1 -- 172.21.15.31:0/2553445841 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7facd400f920 con 0x5624dd059530 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 10 client.15387 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 10 client.15387 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 20 client.15387 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd06e630 0x5624dd070a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 4 client.15387 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:36.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.588+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624dd027d70 con 0x5624dd06e630 2023-07-29T16:14:36.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.589+0000 7facea59c700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd06e630 0x5624dd070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.589+0000 7facea59c700 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd06e630 0x5624dd070a40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8d96c1ab12ddb9b server_cookie=192bf577c551b829 in_seq=0 out_seq=0 2023-07-29T16:14:36.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.589+0000 7facd1ffb700 10 client.15387 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7facd1ffb700 1 -- 172.21.15.31:0/2553445841 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5624dd027d70 con 0x5624dd06e630 2023-07-29T16:14:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7facd1ffb700 10 client.15387 handle_command_reply: tid=0 2023-07-29T16:14:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 client.15387 shutdown 2023-07-29T16:14:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 msgr2=0x5624dd062d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 0x5624dd062d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7facd4009990 tx=0x7facd4005f00 comp rx=0 tx=0).stop 2023-07-29T16:14:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 shutdown_connections 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd06e630 0x5624dd070a40 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd069dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd058be0 0x5624dd062840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7facd1ffb700 0 client.15387 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2553445841 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd059530 0x5624dd062d80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 >> 172.21.15.31:0/2553445841 conn(0x5624dd04e3e0 msgr2=0x5624dd053060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.590+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 shutdown_connections 2023-07-29T16:14:36.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.591+0000 7fad043dcb80 1 -- 172.21.15.31:0/2553445841 wait complete. 2023-07-29T16:14:36.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.591+0000 7fad043dcb80 20 client.15387 trim_cache size 0 max 16384 2023-07-29T16:14:36.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 Processor -- start 2023-07-29T16:14:36.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 -- start start 2023-07-29T16:14:36.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd053790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd059530 0x5624dd069d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd060750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dd050460 con 0x5624dd058be0 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dd0634c0 con 0x5624dd05a2f0 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7fad043dcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dd04b6e0 con 0x5624dd059530 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7facea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd053790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7facea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd053790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52074/0 (socket says 172.21.15.31:52074) 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7facea59c700 1 -- 172.21.15.31:0/706755601 learned_addr learned my addr 172.21.15.31:0/706755601 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.608+0000 7face3fff700 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd059530 0x5624dd069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facead9d700 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd060750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facea59c700 1 -- 172.21.15.31:0/706755601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd059530 msgr2=0x5624dd069d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facea59c700 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd059530 0x5624dd069d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facea59c700 1 -- 172.21.15.31:0/706755601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 msgr2=0x5624dd060750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facea59c700 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd060750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facea59c700 1 -- 172.21.15.31:0/706755601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624dcdc0c60 con 0x5624dd058be0 2023-07-29T16:14:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7facea59c700 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd053790 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7facdc095990 tx=0x7facdc01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52253fe4ce9f7ff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7face17fa700 1 -- 172.21.15.31:0/706755601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0b5070 con 0x5624dd058be0 2023-07-29T16:14:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7face17fa700 1 -- 172.21.15.31:0/706755601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facdc0c9940 con 0x5624dd058be0 2023-07-29T16:14:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.609+0000 7face17fa700 1 -- 172.21.15.31:0/706755601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc098980 con 0x5624dd058be0 2023-07-29T16:14:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 -- 172.21.15.31:0/706755601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 msgr2=0x5624dd053790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd053790 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7facdc095990 tx=0x7facdc01f000 comp rx=0 tx=0).stop 2023-07-29T16:14:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 -- 172.21.15.31:0/706755601 shutdown_connections 2023-07-29T16:14:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd05a2f0 0x5624dd060750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd053790 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 --2- 172.21.15.31:0/706755601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd059530 0x5624dd069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 -- 172.21.15.31:0/706755601 >> 172.21.15.31:0/706755601 conn(0x5624dd04e3e0 msgr2=0x5624dd061480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 -- 172.21.15.31:0/706755601 shutdown_connections 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 -- 172.21.15.31:0/706755601 wait complete. 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.610+0000 7fad043dcb80 1 -- start start 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 Processor -- start 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd069b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd068160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd05a2f0 0x5624dd0686a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dd068c70 con 0x5624dd058be0 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dd068db0 con 0x5624dd059530 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624dd068f70 con 0x5624dd05a2f0 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7fad043dcb80 10 client.0 _ll_register_callbacks cb 0x5624dcf4f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7facea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7facea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd069b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52088/0 (socket says 172.21.15.31:52088) 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7facea59c700 1 -- 172.21.15.31:0/2256352750 learned_addr learned my addr 172.21.15.31:0/2256352750 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7face3fff700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd068160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.611+0000 7facead9d700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd05a2f0 0x5624dd0686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facea59c700 1 -- 172.21.15.31:0/2256352750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd05a2f0 msgr2=0x5624dd0686a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facea59c700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd05a2f0 0x5624dd0686a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facea59c700 1 -- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 msgr2=0x5624dd068160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facea59c700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd068160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facea59c700 1 -- 172.21.15.31:0/2256352750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facdc0b1e70 con 0x5624dd058be0 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7face3fff700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd068160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facea59c700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd069b50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7facdc093fd0 tx=0x7facdc099490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928fafc8647ecb56 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facd1ffb700 1 -- 172.21.15.31:0/2256352750 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0b5050 con 0x5624dd058be0 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7facd1ffb700 1 -- 172.21.15.31:0/2256352750 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7facdc0ae070 con 0x5624dd058be0 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.612+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facdc098df0 con 0x5624dd058be0 2023-07-29T16:14:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7facd1ffb700 1 -- 172.21.15.31:0/2256352750 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7facdc0a1960 con 0x5624dd058be0 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 10 client.15393 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5624dcdfe250 con 0x5624dd058be0 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7facd1ffb700 1 -- 172.21.15.31:0/2256352750 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facdc098df0 con 0x5624dd058be0 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7facd1ffb700 1 -- 172.21.15.31:0/2256352750 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7facdc0983f0 con 0x5624dd058be0 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 10 client.15393 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 10 client.15393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 20 client.15393 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd0c1550 0x5624dd0c3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 4 client.15393 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5624dcfa2d40 con 0x5624dd0c1550 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.613+0000 7face3fff700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd0c1550 0x5624dd0c3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7facd1ffb700 10 client.15393 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7face3fff700 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd0c1550 0x5624dd0c3930 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4ded3a10eb7fa61 server_cookie=9b58a010c18962ce in_seq=0 out_seq=0 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7facd1ffb700 1 -- 172.21.15.31:0/2256352750 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5624dcfa2d40 con 0x5624dd0c1550 2023-07-29T16:14:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7facd1ffb700 10 client.15393 handle_command_reply: tid=0 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 client.15393 shutdown 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 msgr2=0x5624dd069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd069b50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7facdc093fd0 tx=0x7facdc099490 comp rx=0 tx=0).stop 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 shutdown_connections 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624dd0c1550 0x5624dd0c3930 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624dd05a2f0 0x5624dd0686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624dd058be0 0x5624dd069b50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 --2- 172.21.15.31:0/2256352750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624dd059530 0x5624dd068160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7facd1ffb700 0 client.15393 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.614+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 >> 172.21.15.31:0/2256352750 conn(0x5624dd04e3e0 msgr2=0x5624dd061480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 shutdown_connections 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7fad043dcb80 1 -- 172.21.15.31:0/2256352750 wait complete. 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7fad043dcb80 20 client.15393 trim_cache size 0 max 16384 2023-07-29T16:14:36.620 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:14:36.621 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:14:36.621 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:14:36.621 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:14:36.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7facd80a0810 msgr2=0x7facd80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7facd80a0810 0x7facd80a2cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x5624dd058980 tx=0x7face400b040 comp rx=0 tx=0).stop 2023-07-29T16:14:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 msgr2=0x7facec198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec198230 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7facdc00e9a0 tx=0x7facdc00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 shutdown_connections 2023-07-29T16:14:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7facd80a0810 0x7facd80a2cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7facec1081c0 0x7facec19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7facec073750 0x7facec198230 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 --2- 172.21.15.31:0/922277542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7facec074d30 0x7facec198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 >> 172.21.15.31:0/922277542 conn(0x7facec0fb970 msgr2=0x7facec0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.615+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 shutdown_connections 2023-07-29T16:14:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.616+0000 7facf09ff700 1 -- 172.21.15.31:0/922277542 wait complete. 2023-07-29T16:14:36.630 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:14:36.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:14:36.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 -- 172.21.15.31:0/1427913548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 msgr2=0x7f68a0101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 --2- 172.21.15.31:0/1427913548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0101980 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6894009990 tx=0x7f6894009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 -- 172.21.15.31:0/1427913548 shutdown_connections 2023-07-29T16:14:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 --2- 172.21.15.31:0/1427913548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68a0101ec0 0x7f68a0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 --2- 172.21.15.31:0/1427913548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0100340 0x7f68a0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 --2- 172.21.15.31:0/1427913548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0101980 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.838+0000 7f68a59ff700 1 -- 172.21.15.31:0/1427913548 >> 172.21.15.31:0/1427913548 conn(0x7f68a00fb920 msgr2=0x7f68a00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.839+0000 7f68a59ff700 1 -- 172.21.15.31:0/1427913548 shutdown_connections 2023-07-29T16:14:36.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.839+0000 7f68a59ff700 1 -- 172.21.15.31:0/1427913548 wait complete. 2023-07-29T16:14:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 Processor -- start 2023-07-29T16:14:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 -- start start 2023-07-29T16:14:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68a0100340 0x7f68a0196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0101ec0 0x7f68a019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a019dbd0 con 0x7f68a0101530 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a019dd10 con 0x7f68a0101ec0 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f68a59ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a019dea0 con 0x7f68a0100340 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f689e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f689e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52112/0 (socket says 172.21.15.31:52112) 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.840+0000 7f689e7fc700 1 -- 172.21.15.31:0/2082116643 learned_addr learned my addr 172.21.15.31:0/2082116643 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:36.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689effd700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68a0100340 0x7f68a0196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689f7fe700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0101ec0 0x7f68a019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689e7fc700 1 -- 172.21.15.31:0/2082116643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68a0100340 msgr2=0x7f68a0196050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689e7fc700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68a0100340 0x7f68a0196050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689e7fc700 1 -- 172.21.15.31:0/2082116643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0101ec0 msgr2=0x7f68a019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689e7fc700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0101ec0 0x7f68a019d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:36.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689e7fc700 1 -- 172.21.15.31:0/2082116643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68940095f0 con 0x7f68a0101530 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689f7fe700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0101ec0 0x7f68a019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f689e7fc700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0196590 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f689400b620 tx=0x7f6894005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689401d070 con 0x7f68a0101530 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a019e120 con 0x7f68a0101530 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.841+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6894022470 con 0x7f68a0101530 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.842+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894017650 con 0x7f68a0101530 2023-07-29T16:14:36.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.842+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a019e540 con 0x7f68a0101530 2023-07-29T16:14:36.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.843+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a0107820 con 0x7f68a0101530 2023-07-29T16:14:36.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.845+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f68940225e0 con 0x7f68a0101530 2023-07-29T16:14:36.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.846+0000 7f68a49fd700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68880a07e0 0x7f68880a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:36.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.846+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f68940ccff0 con 0x7f68a0101530 2023-07-29T16:14:36.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.846+0000 7f689effd700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68880a07e0 0x7f68880a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:36.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.847+0000 7f689effd700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68880a07e0 0x7f68880a2c90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f68a0101390 tx=0x7f6890009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:36.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:36.850+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6894096ac0 con 0x7f68a0101530 2023-07-29T16:14:37.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.106+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68a002d0c0 con 0x7f68a0101530 2023-07-29T16:14:37.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.107+0000 7f68a49fd700 1 -- 172.21.15.31:0/2082116643 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f689409a0e0 con 0x7f68a0101530 2023-07-29T16:14:37.105 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:14:37.105 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:14:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68880a07e0 msgr2=0x7f68880a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68880a07e0 0x7f68880a2c90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f68a0101390 tx=0x7f6890009260 comp rx=0 tx=0).stop 2023-07-29T16:14:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 msgr2=0x7f68a0196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0196590 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f689400b620 tx=0x7f6894005740 comp rx=0 tx=0).stop 2023-07-29T16:14:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 shutdown_connections 2023-07-29T16:14:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68880a07e0 0x7f68880a2c90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68a0101ec0 0x7f68a019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.109+0000 7f68a59ff700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68a0100340 0x7f68a0196050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.110+0000 7f68a59ff700 1 --2- 172.21.15.31:0/2082116643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68a0101530 0x7f68a0196590 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.110+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 >> 172.21.15.31:0/2082116643 conn(0x7f68a00fb920 msgr2=0x7f68a0104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.110+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 shutdown_connections 2023-07-29T16:14:37.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.110+0000 7f68a59ff700 1 -- 172.21.15.31:0/2082116643 wait complete. 2023-07-29T16:14:37.109 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:14:37.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:14:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 -- 172.21.15.31:0/2860925584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294100320 msgr2=0x7f8294100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 --2- 172.21.15.31:0/2860925584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294100320 0x7f8294100730 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f82840098f0 tx=0x7f8284009c00 comp rx=0 tx=0).stop 2023-07-29T16:14:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 -- 172.21.15.31:0/2860925584 shutdown_connections 2023-07-29T16:14:37.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 --2- 172.21.15.31:0/2860925584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294101ea0 0x7f8294108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 --2- 172.21.15.31:0/2860925584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294100320 0x7f8294100730 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 --2- 172.21.15.31:0/2860925584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8294101510 0x7f8294101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 -- 172.21.15.31:0/2860925584 >> 172.21.15.31:0/2860925584 conn(0x7f82940fb920 msgr2=0x7f82940fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 -- 172.21.15.31:0/2860925584 shutdown_connections 2023-07-29T16:14:37.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.336+0000 7f8299bff700 1 -- 172.21.15.31:0/2860925584 wait complete. 2023-07-29T16:14:37.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.337+0000 7f8299bff700 1 Processor -- start 2023-07-29T16:14:37.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 -- start start 2023-07-29T16:14:37.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 0x7f8294198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294101510 0x7f8294198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8294101ea0 0x7f829419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829419fe80 con 0x7f8294101ea0 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829419ffc0 con 0x7f8294100320 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8299bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82941a0150 con 0x7f8294101510 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f82937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 0x7f8294198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8292ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294101510 0x7f8294198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f82937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 0x7f8294198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50878/0 (socket says 172.21.15.31:50878) 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f82937fe700 1 -- 172.21.15.31:0/254005181 learned_addr learned my addr 172.21.15.31:0/254005181 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.338+0000 7f8293fff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8294101ea0 0x7f829419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f82937fe700 1 -- 172.21.15.31:0/254005181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294101510 msgr2=0x7f8294198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f82937fe700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294101510 0x7f8294198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f82937fe700 1 -- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8294101ea0 msgr2=0x7f829419f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f82937fe700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8294101ea0 0x7f829419f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f82937fe700 1 -- 172.21.15.31:0/254005181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82880095f0 con 0x7f8294100320 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f82937fe700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 0x7f8294198300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8284009fd0 tx=0x7f828400b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f8290ff9700 1 -- 172.21.15.31:0/254005181 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828401c070 con 0x7f8294100320 2023-07-29T16:14:37.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8284009520 con 0x7f8294100320 2023-07-29T16:14:37.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f8290ff9700 1 -- 172.21.15.31:0/254005181 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f828400f7d0 con 0x7f8294100320 2023-07-29T16:14:37.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f8290ff9700 1 -- 172.21.15.31:0/254005181 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284003a40 con 0x7f8294100320 2023-07-29T16:14:37.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.339+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82941a06c0 con 0x7f8294100320 2023-07-29T16:14:37.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.344+0000 7f8290ff9700 1 -- 172.21.15.31:0/254005181 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8284020030 con 0x7f8294100320 2023-07-29T16:14:37.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 Processor -- start 2023-07-29T16:14:37.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 -- start start 2023-07-29T16:14:37.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ed2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ed3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed42f0 0x563486ed4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486e2aa10 con 0x563486ed3530 2023-07-29T16:14:37.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486e283d0 con 0x563486ed42f0 2023-07-29T16:14:37.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.345+0000 7f82ad491b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486e23cc0 con 0x563486ed2be0 2023-07-29T16:14:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ed3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8293fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ed2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8293fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ed2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56872/0 (socket says 172.21.15.31:56872) 2023-07-29T16:14:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8293fff700 1 -- 172.21.15.31:0/1844406618 learned_addr learned my addr 172.21.15.31:0/1844406618 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8292ffd700 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed42f0 0x563486ed4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8290ff9700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f827c0a0790 0x7f827c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 -- 172.21.15.31:0/1844406618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 msgr2=0x563486ed2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ed2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 -- 172.21.15.31:0/1844406618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed42f0 msgr2=0x563486ed4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed42f0 0x563486ed4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 -- 172.21.15.31:0/1844406618 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563486c51de0 con 0x563486ed3530 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8292ffd700 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed42f0 0x563486ed4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8290ff9700 1 -- 172.21.15.31:0/254005181 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f82840cca70 con 0x7f8294100320 2023-07-29T16:14:37.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f8293fff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f827c0a0790 0x7f827c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.346+0000 7f82937fe700 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ed3d60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f82840ca450 tx=0x7f8284009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ce638df89dd7b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.347+0000 7f8279ffb700 1 -- 172.21.15.31:0/1844406618 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284099f70 con 0x563486ed3530 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.347+0000 7f8279ffb700 1 -- 172.21.15.31:0/1844406618 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8284099ce0 con 0x563486ed3530 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.347+0000 7f8279ffb700 1 -- 172.21.15.31:0/1844406618 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840a2560 con 0x563486ed3530 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 -- 172.21.15.31:0/1844406618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 msgr2=0x563486ed3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f8293fff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f827c0a0790 0x7f827c0a2c40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x563486ed2980 tx=0x7f828c009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ed3d60 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f82840ca450 tx=0x7f8284009c00 comp rx=0 tx=0).stop 2023-07-29T16:14:37.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 -- 172.21.15.31:0/1844406618 shutdown_connections 2023-07-29T16:14:37.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed42f0 0x563486ed4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ed2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1844406618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ed3d60 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 -- 172.21.15.31:0/1844406618 >> 172.21.15.31:0/1844406618 conn(0x563486ec83e0 msgr2=0x563486ecd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 -- 172.21.15.31:0/1844406618 shutdown_connections 2023-07-29T16:14:37.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 -- 172.21.15.31:0/1844406618 wait complete. 2023-07-29T16:14:37.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.348+0000 7f82ad491b80 1 -- start start 2023-07-29T16:14:37.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 Processor -- start 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486edc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486edc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ee3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486dc0510 con 0x563486ed3530 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486dc01b0 con 0x563486ed2be0 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486ca7920 con 0x563486ed42f0 2023-07-29T16:14:37.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.349+0000 7f82ad491b80 10 client.0 _ll_register_callbacks cb 0x563486dc9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ee3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ee3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56874/0 (socket says 172.21.15.31:56874) 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 -- 172.21.15.31:0/3357170463 learned_addr learned my addr 172.21.15.31:0/3357170463 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8292ffd700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486edc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f82937fe700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486edc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 -- 172.21.15.31:0/3357170463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 msgr2=0x563486edc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486edc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 -- 172.21.15.31:0/3357170463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 msgr2=0x563486edc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486edc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 -- 172.21.15.31:0/3357170463 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82941a06c0 con 0x563486ed42f0 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8293fff700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ee3970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f828c017e40 tx=0x7f828c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54c0418061190e2d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.350+0000 7f8272ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f8272ffd700 1 -- 172.21.15.31:0/3357170463 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c00e040 con 0x563486ed42f0 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563486e4ed10 con 0x563486ed42f0 2023-07-29T16:14:37.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f8272ffd700 1 -- 172.21.15.31:0/3357170463 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f828c01c030 con 0x563486ed42f0 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f8272ffd700 1 -- 172.21.15.31:0/3357170463 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f828c009310 con 0x563486ed42f0 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f82ad491b80 10 client.25399 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563486c3ac60 con 0x563486ed42f0 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.351+0000 7f8272ffd700 1 -- 172.21.15.31:0/3357170463 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563486e4ed10 con 0x563486ed42f0 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f8272ffd700 1 -- 172.21.15.31:0/3357170463 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f828c02c400 con 0x563486ed42f0 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82ad491b80 10 client.25399 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82ad491b80 10 client.25399 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:14:37.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82ad491b80 20 client.25399 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:37.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486ef0950 0x563486ef2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82ad491b80 4 client.25399 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:37.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563486ea1d70 con 0x563486ef0950 2023-07-29T16:14:37.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.352+0000 7f82937fe700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486ef0950 0x563486ef2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.353+0000 7f8272ffd700 10 client.25399 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:14:37.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.353+0000 7f82937fe700 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486ef0950 0x563486ef2d60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9334d2d495562d0 server_cookie=13e0d003496e418e in_seq=0 out_seq=0 2023-07-29T16:14:37.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.353+0000 7f8272ffd700 1 -- 172.21.15.31:0/3357170463 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563486ea1d70 con 0x563486ef0950 2023-07-29T16:14:37.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.353+0000 7f8272ffd700 10 client.25399 handle_command_reply: tid=0 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.353+0000 7f82ad491b80 1 client.25399 shutdown 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 msgr2=0x563486ee3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ee3970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f828c017e40 tx=0x7f828c004ad0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 shutdown_connections 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ee3970 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486ef0950 0x563486ef2d60 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486edc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3357170463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486edc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 >> 172.21.15.31:0/3357170463 conn(0x563486ec83e0 msgr2=0x563486ecd060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f8272ffd700 0 client.25399 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:14:37.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 shutdown_connections 2023-07-29T16:14:37.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 1 -- 172.21.15.31:0/3357170463 wait complete. 2023-07-29T16:14:37.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.354+0000 7f82ad491b80 20 client.25399 trim_cache size 0 max 16384 2023-07-29T16:14:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 Processor -- start 2023-07-29T16:14:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 -- start start 2023-07-29T16:14:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486ee1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ece7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ecd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486edd5a0 con 0x563486ed3530 2023-07-29T16:14:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486eda330 con 0x563486ed2be0 2023-07-29T16:14:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.369+0000 7f82ad491b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486ed3b50 con 0x563486ed42f0 2023-07-29T16:14:37.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ece7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ece7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52162/0 (socket says 172.21.15.31:52162) 2023-07-29T16:14:37.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 -- 172.21.15.31:0/1723808013 learned_addr learned my addr 172.21.15.31:0/1723808013 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8293fff700 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ecd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f82937fe700 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486ee1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 -- 172.21.15.31:0/1723808013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 msgr2=0x563486ecd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ecd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 -- 172.21.15.31:0/1723808013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 msgr2=0x563486ee1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486ee1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 -- 172.21.15.31:0/1723808013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563486c78250 con 0x563486ed3530 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8293fff700 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ecd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f8292ffd700 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ece7c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8288009f80 tx=0x7f8288009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a02968937c8f11d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.370+0000 7f827a7fc700 1 -- 172.21.15.31:0/1723808013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288017050 con 0x563486ed3530 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f827a7fc700 1 -- 172.21.15.31:0/1723808013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8288018040 con 0x563486ed3530 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f827a7fc700 1 -- 172.21.15.31:0/1723808013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288003c20 con 0x563486ed3530 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 -- 172.21.15.31:0/1723808013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 msgr2=0x563486ece7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ece7c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8288009f80 tx=0x7f8288009310 comp rx=0 tx=0).stop 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 -- 172.21.15.31:0/1723808013 shutdown_connections 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed42f0 0x563486ecd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed2be0 0x563486ee1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 --2- 172.21.15.31:0/1723808013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed3530 0x563486ece7c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 -- 172.21.15.31:0/1723808013 >> 172.21.15.31:0/1723808013 conn(0x563486ec83e0 msgr2=0x563486ecd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 -- 172.21.15.31:0/1723808013 shutdown_connections 2023-07-29T16:14:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.371+0000 7f82ad491b80 1 -- 172.21.15.31:0/1723808013 wait complete. 2023-07-29T16:14:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.372+0000 7f82ad491b80 1 -- start start 2023-07-29T16:14:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.372+0000 7f82ad491b80 1 Processor -- start 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.372+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ee0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.372+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed3530 0x563486ee05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82ad491b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 0x563486ee0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82ad491b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486ee1d10 con 0x563486ed42f0 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82ad491b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486f424d0 con 0x563486ed3530 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82ad491b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563486f42660 con 0x563486ed2be0 2023-07-29T16:14:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82ad491b80 10 client.0 _ll_register_callbacks cb 0x563486dc9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f8293fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 0x563486ee0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ee0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ee0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56882/0 (socket says 172.21.15.31:56882) 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 -- 172.21.15.31:0/3765798308 learned_addr learned my addr 172.21.15.31:0/3765798308 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f8292ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed3530 0x563486ee05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f8293fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 0x563486ee0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52178/0 (socket says 172.21.15.31:52178) 2023-07-29T16:14:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 -- 172.21.15.31:0/3765798308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed3530 msgr2=0x563486ee05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed3530 0x563486ee05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 -- 172.21.15.31:0/3765798308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 msgr2=0x563486ee0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 0x563486ee0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 -- 172.21.15.31:0/3765798308 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828c017e70 con 0x563486ed2be0 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f8293fff700 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 0x563486ee0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f82937fe700 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ee0090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82840966a0 tx=0x7f82840ce040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f128e86da4ea653 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.373+0000 7f8272ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:14:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f8272ffd700 1 -- 172.21.15.31:0/3765798308 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8284099f70 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f8272ffd700 1 -- 172.21.15.31:0/3765798308 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f828409e1d0 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8284099ce0 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f8272ffd700 1 -- 172.21.15.31:0/3765798308 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8284098540 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 10 client.25405 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f8288009c30 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f8272ffd700 1 -- 172.21.15.31:0/3765798308 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8284099ce0 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f8272ffd700 1 -- 172.21.15.31:0/3765798308 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f82840ca1a0 con 0x563486ed2be0 2023-07-29T16:14:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 10 client.25405 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 10 client.25405 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 20 client.25405 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486f43c00 0x563486f46010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 4 client.25405 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.374+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563486e1cd40 con 0x563486f43c00 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.375+0000 7f8292ffd700 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486f43c00 0x563486f46010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.375+0000 7f8292ffd700 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486f43c00 0x563486f46010 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=51ae8222e90ae180 server_cookie=273425a7045802c9 in_seq=0 out_seq=0 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.375+0000 7f8272ffd700 10 client.25405 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f8272ffd700 1 -- 172.21.15.31:0/3765798308 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563486e1cd40 con 0x563486f43c00 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f8272ffd700 10 client.25405 handle_command_reply: tid=0 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 client.25405 shutdown 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 msgr2=0x563486ee0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ee0090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f82840966a0 tx=0x7f82840ce040 comp rx=0 tx=0).stop 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 shutdown_connections 2023-07-29T16:14:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563486ed42f0 0x563486ee0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563486ed2be0 0x563486ee0090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x563486f43c00 0x563486f46010 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 --2- 172.21.15.31:0/3765798308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563486ed3530 0x563486ee05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 >> 172.21.15.31:0/3765798308 conn(0x563486ec83e0 msgr2=0x563486ed70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f8272ffd700 0 client.25405 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 shutdown_connections 2023-07-29T16:14:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 1 -- 172.21.15.31:0/3765798308 wait complete. 2023-07-29T16:14:37.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.376+0000 7f82ad491b80 20 client.25405 trim_cache size 0 max 16384 2023-07-29T16:14:37.382 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:14:37.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f827c0a0790 msgr2=0x7f827c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f827c0a0790 0x7f827c0a2c40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x563486ed2980 tx=0x7f828c009040 comp rx=0 tx=0).stop 2023-07-29T16:14:37.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 msgr2=0x7f8294198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 0x7f8294198300 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8284009fd0 tx=0x7f828400b790 comp rx=0 tx=0).stop 2023-07-29T16:14:37.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 shutdown_connections 2023-07-29T16:14:37.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f827c0a0790 0x7f827c0a2c40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8294101ea0 0x7f829419f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8294100320 0x7f8294198300 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 --2- 172.21.15.31:0/254005181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8294101510 0x7f8294198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 >> 172.21.15.31:0/254005181 conn(0x7f82940fb920 msgr2=0x7f8294106ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 shutdown_connections 2023-07-29T16:14:37.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.377+0000 7f8299bff700 1 -- 172.21.15.31:0/254005181 wait complete. 2023-07-29T16:14:37.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:14:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 -- 172.21.15.31:0/2215974794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf00ff220 msgr2=0x7f0cf00ff630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/2215974794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf00ff220 0x7f0cf00ff630 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009990 tx=0x7f0cec009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 -- 172.21.15.31:0/2215974794 shutdown_connections 2023-07-29T16:14:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/2215974794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 0x7f0cf0107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/2215974794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf00ff220 0x7f0cf00ff630 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/2215974794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf0100410 0x7f0cf0100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 -- 172.21.15.31:0/2215974794 >> 172.21.15.31:0/2215974794 conn(0x7f0cf00fa840 msgr2=0x7f0cf00fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 -- 172.21.15.31:0/2215974794 shutdown_connections 2023-07-29T16:14:37.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.605+0000 7f0cf7dff700 1 -- 172.21.15.31:0/2215974794 wait complete. 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.606+0000 7f0cf7dff700 1 Processor -- start 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 -- start start 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 0x7f0cf0196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf0100410 0x7f0cf01965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 0x7f0cf019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019dc10 con 0x7f0cf0100da0 2023-07-29T16:14:37.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019dd50 con 0x7f0cf00ff220 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf7dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf019dee0 con 0x7f0cf0100410 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 0x7f0cf0196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf639c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 0x7f0cf019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 0x7f0cf0196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50934/0 (socket says 172.21.15.31:50934) 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/1423374612 learned_addr learned my addr 172.21.15.31:0/1423374612 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.607+0000 7f0cf539a700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf0100410 0x7f0cf01965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/1423374612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf0100410 msgr2=0x7f0cf01965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf0100410 0x7f0cf01965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 msgr2=0x7f0cf019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 0x7f0cf019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/1423374612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec0095f0 con 0x7f0cf00ff220 2023-07-29T16:14:37.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf639c700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 0x7f0cf019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 0x7f0cf0196090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0cec005720 tx=0x7f0cec009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0ce6ffd700 1 -- 172.21.15.31:0/1423374612 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec01d070 con 0x7f0cf00ff220 2023-07-29T16:14:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf019e160 con 0x7f0cf00ff220 2023-07-29T16:14:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0ce6ffd700 1 -- 172.21.15.31:0/1423374612 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0cec021410 con 0x7f0cf00ff220 2023-07-29T16:14:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0ce6ffd700 1 -- 172.21.15.31:0/1423374612 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec017650 con 0x7f0cf00ff220 2023-07-29T16:14:37.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.608+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf019e630 con 0x7f0cf00ff220 2023-07-29T16:14:37.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.613+0000 7f0ce6ffd700 1 -- 172.21.15.31:0/1423374612 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0cec021580 con 0x7f0cf00ff220 2023-07-29T16:14:37.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 Processor -- start 2023-07-29T16:14:37.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 -- start start 2023-07-29T16:14:37.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa7b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa7c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928f9cea10 con 0x55928fa7b040 2023-07-29T16:14:37.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928f9cc3d0 con 0x55928fa7b990 2023-07-29T16:14:37.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.614+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928f9c7cc0 con 0x55928fa7c6d0 2023-07-29T16:14:37.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf639c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa7b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf639c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa7b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52214/0 (socket says 172.21.15.31:52214) 2023-07-29T16:14:37.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf639c700 1 -- 172.21.15.31:0/2468665904 learned_addr learned my addr 172.21.15.31:0/2468665904 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa7c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0ce6ffd700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0cdc0a0790 0x7f0cdc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 -- 172.21.15.31:0/2468665904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 msgr2=0x55928fa7c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa7c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 -- 172.21.15.31:0/2468665904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 msgr2=0x55928fa7b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa7b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 -- 172.21.15.31:0/2468665904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55928f7f5de0 con 0x55928fa7c6d0 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf639c700 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa7b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0ce6ffd700 1 -- 172.21.15.31:0/1423374612 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0cec0ccda0 con 0x7f0cf00ff220 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf639c700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0cdc0a0790 0x7f0cdc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.615+0000 7f0cf539a700 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7cb40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009cb0 tx=0x7f0ce0011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8545f0395026efb9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.616+0000 7f0cdbfff700 1 -- 172.21.15.31:0/2468665904 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0011d20 con 0x55928fa7c6d0 2023-07-29T16:14:37.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.616+0000 7f0cdbfff700 1 -- 172.21.15.31:0/2468665904 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ce0011e80 con 0x55928fa7c6d0 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.616+0000 7f0cdbfff700 1 -- 172.21.15.31:0/2468665904 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0005280 con 0x55928fa7c6d0 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.616+0000 7f0d0b711b80 1 -- 172.21.15.31:0/2468665904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 msgr2=0x55928fa7cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.616+0000 7f0cf639c700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0cdc0a0790 0x7f0cdc0a2c40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8003fd0 tx=0x7f0ce8004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.616+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7cb40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009cb0 tx=0x7f0ce0011440 comp rx=0 tx=0).stop 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 -- 172.21.15.31:0/2468665904 shutdown_connections 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7cb40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa7b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/2468665904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa7c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 -- 172.21.15.31:0/2468665904 >> 172.21.15.31:0/2468665904 conn(0x55928fa6c3e0 msgr2=0x55928fa71890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 -- 172.21.15.31:0/2468665904 shutdown_connections 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 -- 172.21.15.31:0/2468665904 wait complete. 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 -- start start 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.617+0000 7f0d0b711b80 1 Processor -- start 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa84c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa85150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928f964510 con 0x55928fa7b040 2023-07-29T16:14:37.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928f9641b0 con 0x55928fa7b990 2023-07-29T16:14:37.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928f84b920 con 0x55928fa7c6d0 2023-07-29T16:14:37.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0d0b711b80 10 client.0 _ll_register_callbacks cb 0x55928f96d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:37.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0cf5b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa84c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0cf539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa85150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0cf5b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa84c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52220/0 (socket says 172.21.15.31:52220) 2023-07-29T16:14:37.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0cf539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa85150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50948/0 (socket says 172.21.15.31:50948) 2023-07-29T16:14:37.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/3257726252 learned_addr learned my addr 172.21.15.31:0/3257726252 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.618+0000 7f0cf639c700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf539a700 1 -- 172.21.15.31:0/3257726252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 msgr2=0x55928fa8c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf539a700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf539a700 1 -- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 msgr2=0x55928fa84c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf539a700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa84c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf539a700 1 -- 172.21.15.31:0/3257726252 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce8003b90 con 0x55928fa7b990 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa84c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cf539a700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa85150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009c80 tx=0x7f0ce0005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846e73734d038ba0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cd97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3257726252 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0004260 con 0x55928fa7b990 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55928fa45d70 con 0x55928fa7b990 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3257726252 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ce00043c0 con 0x55928fa7b990 2023-07-29T16:14:37.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3257726252 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ce0004580 con 0x55928fa7b990 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0d0b711b80 10 client.25409 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.619+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f0ce00098e0 con 0x55928fa7b990 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3257726252 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55928fa45d70 con 0x55928fa7b990 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3257726252 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0ce0003b70 con 0x55928fa7b990 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0d0b711b80 10 client.25409 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0d0b711b80 10 client.25409 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0d0b711b80 20 client.25409 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:37.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fa907a0 0x55928fa92bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0d0b711b80 4 client.25409 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:37.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55928f9c0d40 con 0x55928fa907a0 2023-07-29T16:14:37.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.620+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fa907a0 0x55928fa92bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.621+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fa907a0 0x55928fa92bb0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75675d1154829c3b server_cookie=2c307d2d1612e6a2 in_seq=0 out_seq=0 2023-07-29T16:14:37.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.621+0000 7f0cd97fa700 10 client.25409 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:14:37.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.621+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3257726252 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55928f9c0d40 con 0x55928fa907a0 2023-07-29T16:14:37.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.621+0000 7f0cd97fa700 10 client.25409 handle_command_reply: tid=0 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.621+0000 7f0d0b711b80 1 client.25409 shutdown 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 msgr2=0x55928fa85150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa85150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009c80 tx=0x7f0ce0005d90 comp rx=0 tx=0).stop 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 shutdown_connections 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fa907a0 0x55928fa92bb0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa84c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3257726252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa85150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0cd97fa700 0 client.25409 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:14:37.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 >> 172.21.15.31:0/3257726252 conn(0x55928fa6c3e0 msgr2=0x55928fa71060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 shutdown_connections 2023-07-29T16:14:37.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3257726252 wait complete. 2023-07-29T16:14:37.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.622+0000 7f0d0b711b80 20 client.25409 trim_cache size 0 max 16384 2023-07-29T16:14:37.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.635+0000 7f0d0b711b80 1 Processor -- start 2023-07-29T16:14:37.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.635+0000 7f0d0b711b80 1 -- start start 2023-07-29T16:14:37.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.635+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b040 0x55928fa92b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 0x55928fa8c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928fa72ca0 con 0x55928fa7b990 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928fa82870 con 0x55928fa7b040 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928fa974d0 con 0x55928fa7c6d0 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 0x55928fa8c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 0x55928fa8c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52234/0 (socket says 172.21.15.31:52234) 2023-07-29T16:14:37.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 -- 172.21.15.31:0/312004233 learned_addr learned my addr 172.21.15.31:0/312004233 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b040 0x55928fa92b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf639c700 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 -- 172.21.15.31:0/312004233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 msgr2=0x55928fa7caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 -- 172.21.15.31:0/312004233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b040 msgr2=0x55928fa92b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b040 0x55928fa92b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.636+0000 7f0cf539a700 1 -- 172.21.15.31:0/312004233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55928f7dec60 con 0x55928fa7b990 2023-07-29T16:14:37.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0cf539a700 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 0x55928fa8c120 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0000c00 tx=0x7f0ce00117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c635e5bee03af8c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0ce4bff700 1 -- 172.21.15.31:0/312004233 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce0011b20 con 0x55928fa7b990 2023-07-29T16:14:37.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0ce4bff700 1 -- 172.21.15.31:0/312004233 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ce002b520 con 0x55928fa7b990 2023-07-29T16:14:37.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0ce4bff700 1 -- 172.21.15.31:0/312004233 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce002be50 con 0x55928fa7b990 2023-07-29T16:14:37.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 -- 172.21.15.31:0/312004233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 msgr2=0x55928fa8c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 0x55928fa8c120 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0000c00 tx=0x7f0ce00117d0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 -- 172.21.15.31:0/312004233 shutdown_connections 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa7caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b040 0x55928fa92b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/312004233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b990 0x55928fa8c120 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.637+0000 7f0d0b711b80 1 -- 172.21.15.31:0/312004233 >> 172.21.15.31:0/312004233 conn(0x55928fa6c3e0 msgr2=0x55928fa838b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.638+0000 7f0d0b711b80 1 -- 172.21.15.31:0/312004233 shutdown_connections 2023-07-29T16:14:37.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.638+0000 7f0d0b711b80 1 -- 172.21.15.31:0/312004233 wait complete. 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.638+0000 7f0d0b711b80 1 -- start start 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.638+0000 7f0d0b711b80 1 Processor -- start 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.638+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa87dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa88310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0d0b711b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928fa8b870 con 0x55928fa7b040 2023-07-29T16:14:37.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928fa8b9e0 con 0x55928fa7b990 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0d0b711b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928fa8bba0 con 0x55928fa7c6d0 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0d0b711b80 10 client.0 _ll_register_callbacks cb 0x55928f96d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa87dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa87dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52240/0 (socket says 172.21.15.31:52240) 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/3105531940 learned_addr learned my addr 172.21.15.31:0/3105531940 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf539a700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa88310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf639c700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/3105531940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 msgr2=0x55928fa8b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/3105531940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 msgr2=0x55928fa88310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa88310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 -- 172.21.15.31:0/3105531940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce00098e0 con 0x55928fa7b040 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cf5b9b700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa87dd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b680 tx=0x7f0cec0d07a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e0885338ad8e83 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.639+0000 7f0cd97fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:37.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3105531940 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec09a010 con 0x55928fa7b040 2023-07-29T16:14:37.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3105531940 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0cec0d0490 con 0x55928fa7b040 2023-07-29T16:14:37.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55928f9f2d10 con 0x55928fa7b040 2023-07-29T16:14:37.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3105531940 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cec099e30 con 0x55928fa7b040 2023-07-29T16:14:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 10 client.15435 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55928f81c250 con 0x55928fa7b040 2023-07-29T16:14:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3105531940 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55928f9f2d10 con 0x55928fa7b040 2023-07-29T16:14:37.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3105531940 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0cec0ca2a0 con 0x55928fa7b040 2023-07-29T16:14:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 10 client.15435 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 10 client.15435 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:14:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 20 client.15435 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:37.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fae2c70 0x55928fae5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 4 client.15435 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:14:37.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.640+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55928f9c0a60 con 0x55928fae2c70 2023-07-29T16:14:37.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0cf539a700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fae2c70 0x55928fae5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0cd97fa700 10 client.15435 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:14:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0cf539a700 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fae2c70 0x55928fae5080 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bada6263437c281e server_cookie=a67acada754f2a9e in_seq=0 out_seq=0 2023-07-29T16:14:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0cd97fa700 1 -- 172.21.15.31:0/3105531940 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55928f9c0a60 con 0x55928fae2c70 2023-07-29T16:14:37.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0cd97fa700 10 client.15435 handle_command_reply: tid=0 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0d0b711b80 1 client.15435 shutdown 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 msgr2=0x55928fa87dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.641+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa87dd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b680 tx=0x7f0cec0d07a0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 shutdown_connections 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55928fae2c70 0x55928fae5080 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928fa7c6d0 0x55928fa8b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928fa7b040 0x55928fa87dd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0cd97fa700 0 client.15435 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:14:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 --2- 172.21.15.31:0/3105531940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928fa7b990 0x55928fa88310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 >> 172.21.15.31:0/3105531940 conn(0x55928fa6c3e0 msgr2=0x55928fa7f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 shutdown_connections 2023-07-29T16:14:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 1 -- 172.21.15.31:0/3105531940 wait complete. 2023-07-29T16:14:37.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.642+0000 7f0d0b711b80 20 client.15435 trim_cache size 0 max 16384 2023-07-29T16:14:37.650 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:14:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0cdc0a0790 msgr2=0x7f0cdc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0cdc0a0790 0x7f0cdc0a2c40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8003fd0 tx=0x7f0ce8004ad0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 msgr2=0x7f0cf0196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 0x7f0cf0196090 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0cec005720 tx=0x7f0cec009580 comp rx=0 tx=0).stop 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 shutdown_connections 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0cdc0a0790 0x7f0cdc0a2c40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0cf0100da0 0x7f0cf019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0cf00ff220 0x7f0cf0196090 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 --2- 172.21.15.31:0/1423374612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0cf0100410 0x7f0cf01965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 >> 172.21.15.31:0/1423374612 conn(0x7f0cf00fa840 msgr2=0x7f0cf00fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 shutdown_connections 2023-07-29T16:14:37.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.643+0000 7f0cf7dff700 1 -- 172.21.15.31:0/1423374612 wait complete. 2023-07-29T16:14:37.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:14:37.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 -- 172.21.15.31:0/1385187299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b00737b0 msgr2=0x7f38b0073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 --2- 172.21.15.31:0/1385187299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b00737b0 0x7f38b0073c20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f38a40098f0 tx=0x7f38a4009c00 comp rx=0 tx=0).stop 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 -- 172.21.15.31:0/1385187299 shutdown_connections 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 --2- 172.21.15.31:0/1385187299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b01081f0 0x7f38b010a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 --2- 172.21.15.31:0/1385187299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38b0074d90 0x7f38b00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 --2- 172.21.15.31:0/1385187299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b00737b0 0x7f38b0073c20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 -- 172.21.15.31:0/1385187299 >> 172.21.15.31:0/1385187299 conn(0x7f38b00fb970 msgr2=0x7f38b00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 -- 172.21.15.31:0/1385187299 shutdown_connections 2023-07-29T16:14:37.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.870+0000 7f38b63ff700 1 -- 172.21.15.31:0/1385187299 wait complete. 2023-07-29T16:14:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.871+0000 7f38b63ff700 1 Processor -- start 2023-07-29T16:14:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.871+0000 7f38b63ff700 1 -- start start 2023-07-29T16:14:37.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.871+0000 7f38b63ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 0x7f38b0198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.871+0000 7f38b63ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b0074d90 0x7f38b0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.871+0000 7f38b63ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38b01081f0 0x7f38b019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38b63ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b019fe80 con 0x7f38b0074d90 2023-07-29T16:14:37.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38b63ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b019ffc0 con 0x7f38b01081f0 2023-07-29T16:14:37.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38b63ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b01a0150 con 0x7f38b00737b0 2023-07-29T16:14:37.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38affff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 0x7f38b0198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b0074d90 0x7f38b0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38affff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 0x7f38b0198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56954/0 (socket says 172.21.15.31:56954) 2023-07-29T16:14:37.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38affff700 1 -- 172.21.15.31:0/3828723511 learned_addr learned my addr 172.21.15.31:0/3828723511 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38b499c700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38b01081f0 0x7f38b019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 -- 172.21.15.31:0/3828723511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 msgr2=0x7f38b0198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 0x7f38b0198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 -- 172.21.15.31:0/3828723511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38b01081f0 msgr2=0x7f38b019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38b01081f0 0x7f38b019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 -- 172.21.15.31:0/3828723511 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8009d00 con 0x7f38b0074d90 2023-07-29T16:14:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38affff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 0x7f38b0198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.872+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b0074d90 0x7f38b0198840 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000c00 tx=0x7f38a400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.873+0000 7f38ad7fa700 1 -- 172.21.15.31:0/3828723511 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a401c070 con 0x7f38b0074d90 2023-07-29T16:14:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.873+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a4009520 con 0x7f38b0074d90 2023-07-29T16:14:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.873+0000 7f38ad7fa700 1 -- 172.21.15.31:0/3828723511 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38a400f7d0 con 0x7f38b0074d90 2023-07-29T16:14:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.873+0000 7f38ad7fa700 1 -- 172.21.15.31:0/3828723511 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a4003990 con 0x7f38b0074d90 2023-07-29T16:14:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.873+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01a0660 con 0x7f38b0074d90 2023-07-29T16:14:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.876+0000 7f38ad7fa700 1 -- 172.21.15.31:0/3828723511 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f38a4020070 con 0x7f38b0074d90 2023-07-29T16:14:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.877+0000 7f38c9d86b80 1 Processor -- start 2023-07-29T16:14:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.877+0000 7f38c9d86b80 1 -- start start 2023-07-29T16:14:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38ad7fa700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38980a0790 0x7f38980a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fb4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38ad7fa700 1 -- 172.21.15.31:0/3828723511 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38a40cc550 con 0x7f38b0074d90 2023-07-29T16:14:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fb5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841f0ca10 con 0x562841fb4be0 2023-07-29T16:14:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841f0a3d0 con 0x562841fb5530 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841f05cc0 con 0x562841fb62f0 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38b499c700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38980a0790 0x7f38980a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fb4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fb4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52258/0 (socket says 172.21.15.31:52258) 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38affff700 1 -- 172.21.15.31:0/3509056581 learned_addr learned my addr 172.21.15.31:0/3509056581 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38b499c700 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.878+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fb5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38affff700 1 -- 172.21.15.31:0/3509056581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 msgr2=0x562841fb6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38affff700 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38affff700 1 -- 172.21.15.31:0/3509056581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 msgr2=0x562841fb5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38affff700 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fb5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38affff700 1 -- 172.21.15.31:0/3509056581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562841d33de0 con 0x562841fb4be0 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38b499c700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38980a0790 0x7f38980a2c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009610 tx=0x7f38a80093a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38affff700 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fb4ff0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f38a0011680 tx=0x7f38a0011990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3377b713a34a9e3a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38967fc700 1 -- 172.21.15.31:0/3509056581 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0004230 con 0x562841fb4be0 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38967fc700 1 -- 172.21.15.31:0/3509056581 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38a0004390 con 0x562841fb4be0 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.879+0000 7f38967fc700 1 -- 172.21.15.31:0/3509056581 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0019960 con 0x562841fb4be0 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3509056581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 msgr2=0x562841fb4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fb4ff0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f38a0011680 tx=0x7f38a0011990 comp rx=0 tx=0).stop 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3509056581 shutdown_connections 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fb4ff0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3509056581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fb5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3509056581 >> 172.21.15.31:0/3509056581 conn(0x562841faa3e0 msgr2=0x562841faf990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3509056581 shutdown_connections 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.880+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3509056581 wait complete. 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.881+0000 7f38c9d86b80 1 -- start start 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.881+0000 7f38c9d86b80 1 Processor -- start 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fbe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fbeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841ea2510 con 0x562841fb4be0 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841ea21b0 con 0x562841fb5530 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841d89920 con 0x562841fb62f0 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38c9d86b80 10 client.0 _ll_register_callbacks cb 0x562841eab620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fbe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fbe860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52260/0 (socket says 172.21.15.31:52260) 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38affff700 1 -- 172.21.15.31:0/3000939952 learned_addr learned my addr 172.21.15.31:0/3000939952 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fbeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.882+0000 7f38b499c700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38affff700 1 -- 172.21.15.31:0/3000939952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 msgr2=0x562841fc5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38affff700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38affff700 1 -- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 msgr2=0x562841fbeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38affff700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fbeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38affff700 1 -- 172.21.15.31:0/3000939952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a0009e10 con 0x562841fb4be0 2023-07-29T16:14:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fbeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38affff700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fbe860 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f38a0011650 tx=0x7f38a0005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2e6ea1de0aa76d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f388f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f388f7fe700 1 -- 172.21.15.31:0/3000939952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0003b90 con 0x562841fb4be0 2023-07-29T16:14:37.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562841f30d10 con 0x562841fb4be0 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.883+0000 7f388f7fe700 1 -- 172.21.15.31:0/3000939952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38a0005040 con 0x562841fb4be0 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f388f7fe700 1 -- 172.21.15.31:0/3000939952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38a0005240 con 0x562841fb4be0 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 10 client.15447 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562841d1cc60 con 0x562841fb4be0 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f388f7fe700 1 -- 172.21.15.31:0/3000939952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562841f30d10 con 0x562841fb4be0 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f388f7fe700 1 -- 172.21.15.31:0/3000939952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f38a0021040 con 0x562841fb4be0 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 10 client.15447 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 10 client.15447 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 20 client.15447 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562841fca510 0x562841fcc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 4 client.15447 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.884+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562841f83d70 con 0x562841fca510 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.885+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562841fca510 0x562841fcc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.885+0000 7f38af7fe700 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562841fca510 0x562841fcc920 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71fa9a5b2f6eec76 server_cookie=db659ebc6b9d04a in_seq=0 out_seq=0 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.885+0000 7f388f7fe700 10 client.15447 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f388f7fe700 1 -- 172.21.15.31:0/3000939952 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562841f83d70 con 0x562841fca510 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f388f7fe700 10 client.15447 handle_command_reply: tid=0 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 client.15447 shutdown 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 msgr2=0x562841fbe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fbe860 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f38a0011650 tx=0x7f38a0005d00 comp rx=0 tx=0).stop 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 shutdown_connections 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562841fca510 0x562841fcc920 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fbe860 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f388f7fe700 0 client.15447 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/3000939952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fbeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 >> 172.21.15.31:0/3000939952 conn(0x562841faa3e0 msgr2=0x562841faf060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 shutdown_connections 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.886+0000 7f38c9d86b80 1 -- 172.21.15.31:0/3000939952 wait complete. 2023-07-29T16:14:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.887+0000 7f38c9d86b80 20 client.15447 trim_cache size 0 max 16384 2023-07-29T16:14:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 Processor -- start 2023-07-29T16:14:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 -- start start 2023-07-29T16:14:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb4be0 0x562841fbd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 0x562841fc5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841fbe430 con 0x562841fb5530 2023-07-29T16:14:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841fca000 con 0x562841fb4be0 2023-07-29T16:14:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.900+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841fb5970 con 0x562841fb62f0 2023-07-29T16:14:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 0x562841fc5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb4be0 0x562841fbd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 0x562841fc5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52268/0 (socket says 172.21.15.31:52268) 2023-07-29T16:14:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38af7fe700 1 -- 172.21.15.31:0/1017136000 learned_addr learned my addr 172.21.15.31:0/1017136000 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38b499c700 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 -- 172.21.15.31:0/1017136000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 msgr2=0x562841fb92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 -- 172.21.15.31:0/1017136000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 msgr2=0x562841fc5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 0x562841fc5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 -- 172.21.15.31:0/1017136000 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562841d5a250 con 0x562841fb4be0 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38af7fe700 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 0x562841fc5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f38affff700 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb4be0 0x562841fbd4e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38a0000c00 tx=0x7f38a0011990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b752466b97d426ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.901+0000 7f3896ffd700 1 -- 172.21.15.31:0/1017136000 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0019420 con 0x562841fb4be0 2023-07-29T16:14:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f3896ffd700 1 -- 172.21.15.31:0/1017136000 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38a0019800 con 0x562841fb4be0 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f3896ffd700 1 -- 172.21.15.31:0/1017136000 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0004970 con 0x562841fb4be0 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1017136000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb4be0 msgr2=0x562841fbd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb4be0 0x562841fbd4e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38a0000c00 tx=0x7f38a0011990 comp rx=0 tx=0).stop 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1017136000 shutdown_connections 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fb92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb4be0 0x562841fbd4e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1017136000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb5530 0x562841fc5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1017136000 >> 172.21.15.31:0/1017136000 conn(0x562841faa3e0 msgr2=0x562841fbc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1017136000 shutdown_connections 2023-07-29T16:14:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.902+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1017136000 wait complete. 2023-07-29T16:14:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 -- start start 2023-07-29T16:14:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 Processor -- start 2023-07-29T16:14:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fc3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fc3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841fc53a0 con 0x562841fb4be0 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841fc5510 con 0x562841fb5530 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562841fc56d0 con 0x562841fb62f0 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.903+0000 7f38c9d86b80 10 client.0 _ll_register_callbacks cb 0x562841eab620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38affff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fc3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fc3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38b499c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fc3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51042/0 (socket says 172.21.15.31:51042) 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 -- 172.21.15.31:0/1518120752 learned_addr learned my addr 172.21.15.31:0/1518120752 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 -- 172.21.15.31:0/1518120752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 msgr2=0x562841fc4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 -- 172.21.15.31:0/1518120752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 msgr2=0x562841fc3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fc3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 -- 172.21.15.31:0/1518120752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b01a0660 con 0x562841fb5530 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38affff700 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fc3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38af7fe700 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fc3e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f38a400fa40 tx=0x7f38a400bb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d41afc801f05e70e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f388f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f388f7fe700 1 -- 172.21.15.31:0/1518120752 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a40a2040 con 0x562841fb5530 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38a0004390 con 0x562841fb5530 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.904+0000 7f388f7fe700 1 -- 172.21.15.31:0/1518120752 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38a400b880 con 0x562841fb5530 2023-07-29T16:14:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f388f7fe700 1 -- 172.21.15.31:0/1518120752 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38a4099170 con 0x562841fb5530 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 10 client.25433 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562841d1cc60 con 0x562841fb5530 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f388f7fe700 1 -- 172.21.15.31:0/1518120752 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38a407c070 con 0x562841fb5530 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f388f7fe700 1 -- 172.21.15.31:0/1518120752 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f38a40b2130 con 0x562841fb5530 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 10 client.25433 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 10 client.25433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 20 client.25433 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56284201c960 0x56284201ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 4 client.25433 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562841efed40 con 0x56284201c960 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.905+0000 7f38affff700 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56284201c960 0x56284201ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38affff700 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56284201c960 0x56284201ed70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8902cab2088a0589 server_cookie=b60d39b0645ac673 in_seq=0 out_seq=0 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f388f7fe700 10 client.25433 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f388f7fe700 1 -- 172.21.15.31:0/1518120752 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562841efed40 con 0x56284201c960 2023-07-29T16:14:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f388f7fe700 10 client.25433 handle_command_reply: tid=0 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 client.25433 shutdown 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 msgr2=0x562841fc3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fc3e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f38a400fa40 tx=0x7f38a400bb50 comp rx=0 tx=0).stop 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 shutdown_connections 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56284201c960 0x56284201ed70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562841fb62f0 0x562841fc4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f388f7fe700 0 client.25433 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562841fb4be0 0x562841fc3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 --2- 172.21.15.31:0/1518120752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562841fb5530 0x562841fc3e50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.906+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 >> 172.21.15.31:0/1518120752 conn(0x562841faa3e0 msgr2=0x562841fbd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 shutdown_connections 2023-07-29T16:14:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38c9d86b80 1 -- 172.21.15.31:0/1518120752 wait complete. 2023-07-29T16:14:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38c9d86b80 20 client.25433 trim_cache size 0 max 16384 2023-07-29T16:14:37.913 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:14:37.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38980a0790 msgr2=0x7f38980a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38980a0790 0x7f38980a2c40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009610 tx=0x7f38a80093a0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b0074d90 msgr2=0x7f38b0198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b0074d90 0x7f38b0198840 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f38a4000c00 tx=0x7f38a400b790 comp rx=0 tx=0).stop 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 shutdown_connections 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38980a0790 0x7f38980a2c40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38b01081f0 0x7f38b019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b00737b0 0x7f38b0198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 --2- 172.21.15.31:0/3828723511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b0074d90 0x7f38b0198840 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 >> 172.21.15.31:0/3828723511 conn(0x7f38b00fb970 msgr2=0x7f38b00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.907+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 shutdown_connections 2023-07-29T16:14:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:37.908+0000 7f38b63ff700 1 -- 172.21.15.31:0/3828723511 wait complete. 2023-07-29T16:14:37.920 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:37 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2082116643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:37.921 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:37 smithi031 ceph-mon[95504]: pgmap v344: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:14:38.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:37 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2082116643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:38.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:37 smithi156 ceph-mon[129245]: pgmap v344: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:14:38.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:37 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2082116643' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:14:38.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:37 smithi039 ceph-mon[128858]: pgmap v344: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:14:38.923 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:14:38.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:14:39.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.137+0000 7f092e9ff700 1 -- 172.21.15.31:0/1322938130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928100e10 msgr2=0x7f09281031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.137+0000 7f092e9ff700 1 --2- 172.21.15.31:0/1322938130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928100e10 0x7f09281031f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f09180098f0 tx=0x7f0918009c00 comp rx=0 tx=0).stop 2023-07-29T16:14:39.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 -- 172.21.15.31:0/1322938130 shutdown_connections 2023-07-29T16:14:39.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 --2- 172.21.15.31:0/1322938130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928106050 0x7f09281084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 --2- 172.21.15.31:0/1322938130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928100e10 0x7f09281031f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 --2- 172.21.15.31:0/1322938130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928103730 0x7f0928105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 -- 172.21.15.31:0/1322938130 >> 172.21.15.31:0/1322938130 conn(0x7f09280fa8b0 msgr2=0x7f09280fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 -- 172.21.15.31:0/1322938130 shutdown_connections 2023-07-29T16:14:39.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.138+0000 7f092e9ff700 1 -- 172.21.15.31:0/1322938130 wait complete. 2023-07-29T16:14:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 Processor -- start 2023-07-29T16:14:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 -- start start 2023-07-29T16:14:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 0x7f0928100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928103730 0x7f09280fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928106050 0x7f09280ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09280ffaa0 con 0x7f0928100e10 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09280ffc10 con 0x7f0928106050 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f092e9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09281a0110 con 0x7f0928103730 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f0927fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 0x7f0928100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f0927fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 0x7f0928100780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52298/0 (socket says 172.21.15.31:52298) 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.139+0000 7f0927fff700 1 -- 172.21.15.31:0/444828542 learned_addr learned my addr 172.21.15.31:0/444828542 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f092cf9c700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928106050 0x7f09280ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f09277fe700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928103730 0x7f09280fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f0927fff700 1 -- 172.21.15.31:0/444828542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928103730 msgr2=0x7f09280fedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f0927fff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928103730 0x7f09280fedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f0927fff700 1 -- 172.21.15.31:0/444828542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928106050 msgr2=0x7f09280ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f0927fff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928106050 0x7f09280ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f0927fff700 1 -- 172.21.15.31:0/444828542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918009520 con 0x7f0928100e10 2023-07-29T16:14:39.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f0927fff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 0x7f0928100780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0918011040 tx=0x7f091800b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f09257fa700 1 -- 172.21.15.31:0/444828542 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091801c070 con 0x7f0928100e10 2023-07-29T16:14:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09281a0390 con 0x7f0928100e10 2023-07-29T16:14:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f09257fa700 1 -- 172.21.15.31:0/444828542 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091800f7d0 con 0x7f0928100e10 2023-07-29T16:14:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.140+0000 7f09257fa700 1 -- 172.21.15.31:0/444828542 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0918004a00 con 0x7f0928100e10 2023-07-29T16:14:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.141+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09281a0810 con 0x7f0928100e10 2023-07-29T16:14:39.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.144+0000 7f09257fa700 1 -- 172.21.15.31:0/444828542 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0918020070 con 0x7f0928100e10 2023-07-29T16:14:39.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.145+0000 7f0942288b80 1 Processor -- start 2023-07-29T16:14:39.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.145+0000 7f09257fa700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09100a0810 0x7f09100a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.145+0000 7f0942288b80 1 -- start start 2023-07-29T16:14:39.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09257fa700 1 -- 172.21.15.31:0/444828542 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f09180cc7a0 con 0x7f0928100e10 2023-07-29T16:14:39.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0942288b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e045070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f092cf9c700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09100a0810 0x7f09100a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0942288b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e045de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0942288b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e0467c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0942288b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6df9ca90 con 0x564e6e044c60 2023-07-29T16:14:39.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0942288b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6df9a450 con 0x564e6e0455b0 2023-07-29T16:14:39.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0942288b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6df95d40 con 0x564e6e046370 2023-07-29T16:14:39.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e045de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e045de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51058/0 (socket says 172.21.15.31:51058) 2023-07-29T16:14:39.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 -- 172.21.15.31:0/2511995562 learned_addr learned my addr 172.21.15.31:0/2511995562 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f0927fff700 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e045070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f092cf9c700 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e0467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 -- 172.21.15.31:0/2511995562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 msgr2=0x564e6e0467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e0467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 -- 172.21.15.31:0/2511995562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 msgr2=0x564e6e045070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.146+0000 7f09277fe700 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e045070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f09277fe700 1 -- 172.21.15.31:0/2511995562 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e6ddc3e60 con 0x564e6e0455b0 2023-07-29T16:14:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f0927fff700 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e045070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f092cf9c700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09100a0810 0x7f09100a2cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0928100de0 tx=0x7f0920009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f09277fe700 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e045de0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f091c009d60 tx=0x7f091c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491ea38d508dbf5b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f090e7fc700 1 -- 172.21.15.31:0/2511995562 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c019630 con 0x564e6e0455b0 2023-07-29T16:14:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f090e7fc700 1 -- 172.21.15.31:0/2511995562 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091c019c50 con 0x564e6e0455b0 2023-07-29T16:14:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.147+0000 7f090e7fc700 1 -- 172.21.15.31:0/2511995562 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c00ebd0 con 0x564e6e0455b0 2023-07-29T16:14:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 -- 172.21.15.31:0/2511995562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 msgr2=0x564e6e045de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e045de0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f091c009d60 tx=0x7f091c011720 comp rx=0 tx=0).stop 2023-07-29T16:14:39.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 -- 172.21.15.31:0/2511995562 shutdown_connections 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e0467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e045070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 --2- 172.21.15.31:0/2511995562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e045de0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 -- 172.21.15.31:0/2511995562 >> 172.21.15.31:0/2511995562 conn(0x564e6e03a460 msgr2=0x564e6e03fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 -- 172.21.15.31:0/2511995562 shutdown_connections 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.148+0000 7f0942288b80 1 -- 172.21.15.31:0/2511995562 wait complete. 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 -- start start 2023-07-29T16:14:39.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 Processor -- start 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e04e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e04edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e055e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6df32590 con 0x564e6e044c60 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6df32230 con 0x564e6e0455b0 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6de199a0 con 0x564e6e046370 2023-07-29T16:14:39.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.149+0000 7f0942288b80 10 client.0 _ll_register_callbacks cb 0x564e6df3b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e04e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e04e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52314/0 (socket says 172.21.15.31:52314) 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f09277fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e04edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f092cf9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e055e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 -- 172.21.15.31:0/2002874220 learned_addr learned my addr 172.21.15.31:0/2002874220 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 -- 172.21.15.31:0/2002874220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 msgr2=0x564e6e055e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e055e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 -- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 msgr2=0x564e6e04edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e04edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 -- 172.21.15.31:0/2002874220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c0099c0 con 0x564e6e044c60 2023-07-29T16:14:39.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f092cf9c700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e055e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f09277fe700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e04edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f0927fff700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e04e890 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0918099bd0 tx=0x7f0918099ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1fcf23b232b287 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.150+0000 7f09077fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f09077fe700 1 -- 172.21.15.31:0/2002874220 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09180995c0 con 0x564e6e044c60 2023-07-29T16:14:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f09077fe700 1 -- 172.21.15.31:0/2002874220 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f09180a2290 con 0x564e6e044c60 2023-07-29T16:14:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e6dfc0d90 con 0x564e6e044c60 2023-07-29T16:14:39.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f09077fe700 1 -- 172.21.15.31:0/2002874220 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09180b8170 con 0x564e6e044c60 2023-07-29T16:14:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 10 client.15465 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564e6ddacce0 con 0x564e6e044c60 2023-07-29T16:14:39.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f09077fe700 1 -- 172.21.15.31:0/2002874220 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564e6dfc0d90 con 0x564e6e044c60 2023-07-29T16:14:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f09077fe700 1 -- 172.21.15.31:0/2002874220 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0918094ae0 con 0x564e6e044c60 2023-07-29T16:14:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 10 client.15465 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 10 client.15465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 20 client.15465 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:39.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e05a1a0 0x564e6e05c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 4 client.15465 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.151+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e6e013df0 con 0x564e6e05a1a0 2023-07-29T16:14:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.152+0000 7f09277fe700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e05a1a0 0x564e6e05c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.152+0000 7f09277fe700 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e05a1a0 0x564e6e05c5b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c096f0b8ebb936 server_cookie=58191e1651a9556 in_seq=0 out_seq=0 2023-07-29T16:14:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.152+0000 7f09077fe700 10 client.15465 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.153+0000 7f09077fe700 1 -- 172.21.15.31:0/2002874220 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564e6e013df0 con 0x564e6e05a1a0 2023-07-29T16:14:39.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.153+0000 7f09077fe700 10 client.15465 handle_command_reply: tid=0 2023-07-29T16:14:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.153+0000 7f0942288b80 1 client.15465 shutdown 2023-07-29T16:14:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.153+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 msgr2=0x564e6e04e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.153+0000 7f0942288b80 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e04e890 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0918099bd0 tx=0x7f0918099ed0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 shutdown_connections 2023-07-29T16:14:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e05a1a0 0x564e6e05c5b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e046370 0x564e6e055e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e044c60 0x564e6e04e890 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f09077fe700 0 client.15465 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 --2- 172.21.15.31:0/2002874220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e04edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 >> 172.21.15.31:0/2002874220 conn(0x564e6e03a460 msgr2=0x564e6e03f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 shutdown_connections 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 1 -- 172.21.15.31:0/2002874220 wait complete. 2023-07-29T16:14:39.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.154+0000 7f0942288b80 20 client.15465 trim_cache size 0 max 16384 2023-07-29T16:14:39.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 Processor -- start 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 -- start start 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e044c60 0x564e6e03f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e0455b0 0x564e6e055dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e04c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6e04d290 con 0x564e6e046370 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6e041400 con 0x564e6e044c60 2023-07-29T16:14:39.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.171+0000 7f0942288b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6e03fbd0 con 0x564e6e0455b0 2023-07-29T16:14:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e04c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e04c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52322/0 (socket says 172.21.15.31:52322) 2023-07-29T16:14:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 -- 172.21.15.31:0/1210365897 learned_addr learned my addr 172.21.15.31:0/1210365897 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f09277fe700 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e0455b0 0x564e6e055dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f0927fff700 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e044c60 0x564e6e03f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 -- 172.21.15.31:0/1210365897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e0455b0 msgr2=0x564e6e055dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e0455b0 0x564e6e055dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 -- 172.21.15.31:0/1210365897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e044c60 msgr2=0x564e6e03f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e044c60 0x564e6e03f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.172+0000 7f092cf9c700 1 -- 172.21.15.31:0/1210365897 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e6ddea2d0 con 0x564e6e046370 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f092cf9c700 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e04c7c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0920019c80 tx=0x7f092001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8977f9f821ce0da server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f090effd700 1 -- 172.21.15.31:0/1210365897 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000e040 con 0x564e6e046370 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f090effd700 1 -- 172.21.15.31:0/1210365897 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f092000f040 con 0x564e6e046370 2023-07-29T16:14:39.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f090effd700 1 -- 172.21.15.31:0/1210365897 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920005500 con 0x564e6e046370 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f0942288b80 1 -- 172.21.15.31:0/1210365897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 msgr2=0x564e6e04c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f0942288b80 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e04c7c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0920019c80 tx=0x7f092001c000 comp rx=0 tx=0).stop 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f0942288b80 1 -- 172.21.15.31:0/1210365897 shutdown_connections 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f0942288b80 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e04c7c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f0942288b80 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e044c60 0x564e6e03f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.173+0000 7f0942288b80 1 --2- 172.21.15.31:0/1210365897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e0455b0 0x564e6e055dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.174+0000 7f0942288b80 1 -- 172.21.15.31:0/1210365897 >> 172.21.15.31:0/1210365897 conn(0x564e6e03a460 msgr2=0x564e6e040710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.174+0000 7f0942288b80 1 -- 172.21.15.31:0/1210365897 shutdown_connections 2023-07-29T16:14:39.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.174+0000 7f0942288b80 1 -- 172.21.15.31:0/1210365897 wait complete. 2023-07-29T16:14:39.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.174+0000 7f0942288b80 1 -- start start 2023-07-29T16:14:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.174+0000 7f0942288b80 1 Processor -- start 2023-07-29T16:14:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 0x564e6e054f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e055490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e0abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6e0ac2f0 con 0x564e6e046370 2023-07-29T16:14:39.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6e0ac460 con 0x564e6e0455b0 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6e0ac620 con 0x564e6e044c60 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0942288b80 10 client.0 _ll_register_callbacks cb 0x564e6df3b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 0x564e6e054f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f092cf9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e0abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 0x564e6e054f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57044/0 (socket says 172.21.15.31:57044) 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f09277fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e055490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 -- 172.21.15.31:0/2812665503 learned_addr learned my addr 172.21.15.31:0/2812665503 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 -- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 msgr2=0x564e6e055490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e055490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 -- 172.21.15.31:0/2812665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 msgr2=0x564e6e0abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e0abdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f0927fff700 1 -- 172.21.15.31:0/2812665503 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0920019930 con 0x564e6e044c60 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.175+0000 7f09277fe700 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e055490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:39.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f0927fff700 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 0x564e6e054f50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f09180968f0 tx=0x7f091809a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f42602e7867d19fd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f09077fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f09077fe700 1 -- 172.21.15.31:0/2812665503 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09180a2290 con 0x564e6e044c60 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f091c019c50 con 0x564e6e044c60 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f09077fe700 1 -- 172.21.15.31:0/2812665503 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f09180a1e00 con 0x564e6e044c60 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f09077fe700 1 -- 172.21.15.31:0/2812665503 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09180a1fc0 con 0x564e6e044c60 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f0942288b80 10 client.25459 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564e6ddc2330 con 0x564e6e044c60 2023-07-29T16:14:39.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.176+0000 7f09077fe700 1 -- 172.21.15.31:0/2812665503 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091807a020 con 0x564e6e044c60 2023-07-29T16:14:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f09077fe700 1 -- 172.21.15.31:0/2812665503 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f09180cc450 con 0x564e6e044c60 2023-07-29T16:14:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f0942288b80 10 client.25459 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f0942288b80 10 client.25459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f0942288b80 20 client.25459 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:39.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f0942288b80 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e0ada60 0x564e6e0afe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f0942288b80 4 client.25459 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:14:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564e6df8edc0 con 0x564e6e0ada60 2023-07-29T16:14:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.177+0000 7f09277fe700 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e0ada60 0x564e6e0afe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.178+0000 7f09277fe700 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e0ada60 0x564e6e0afe70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f61a7e23b238b0fc server_cookie=5533be3a1b4e08c2 in_seq=0 out_seq=0 2023-07-29T16:14:39.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.178+0000 7f09077fe700 10 client.25459 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.179+0000 7f09077fe700 1 -- 172.21.15.31:0/2812665503 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564e6df8edc0 con 0x564e6e0ada60 2023-07-29T16:14:39.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.179+0000 7f09077fe700 10 client.25459 handle_command_reply: tid=0 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.179+0000 7f0942288b80 1 client.25459 shutdown 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.179+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 msgr2=0x564e6e054f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.179+0000 7f0942288b80 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 0x564e6e054f50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f09180968f0 tx=0x7f091809a410 comp rx=0 tx=0).stop 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 shutdown_connections 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564e6e0ada60 0x564e6e0afe70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564e6e046370 0x564e6e0abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f09077fe700 0 client.25459 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e6e044c60 0x564e6e054f50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 --2- 172.21.15.31:0/2812665503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e6e0455b0 0x564e6e055490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 >> 172.21.15.31:0/2812665503 conn(0x564e6e03a460 msgr2=0x564e6e03f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 shutdown_connections 2023-07-29T16:14:39.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 1 -- 172.21.15.31:0/2812665503 wait complete. 2023-07-29T16:14:39.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f0942288b80 20 client.25459 trim_cache size 0 max 16384 2023-07-29T16:14:39.185 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:14:39.185 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:14:39.185 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "1e023985-7ba5-47c3-aa52-6757ddf647e8", 2023-07-29T16:14:39.185 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:14:39.186 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:14:39.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09100a0810 msgr2=0x7f09100a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f092e9ff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09100a0810 0x7f09100a2cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0928100de0 tx=0x7f0920009920 comp rx=0 tx=0).stop 2023-07-29T16:14:39.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 msgr2=0x7f0928100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.180+0000 7f092e9ff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 0x7f0928100780 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0918011040 tx=0x7f091800b790 comp rx=0 tx=0).stop 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 shutdown_connections 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09100a0810 0x7f09100a2cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928106050 0x7f09280ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0928100e10 0x7f0928100780 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 --2- 172.21.15.31:0/444828542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0928103730 0x7f09280fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 >> 172.21.15.31:0/444828542 conn(0x7f09280fa8b0 msgr2=0x7f0928101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 shutdown_connections 2023-07-29T16:14:39.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.181+0000 7f092e9ff700 1 -- 172.21.15.31:0/444828542 wait complete. 2023-07-29T16:14:39.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:14:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 -- 172.21.15.31:0/3066169040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 msgr2=0x7f5b8c105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 --2- 172.21.15.31:0/3066169040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 0x7f5b8c105b00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5b74009990 tx=0x7f5b74009ca0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 -- 172.21.15.31:0/3066169040 shutdown_connections 2023-07-29T16:14:39.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 --2- 172.21.15.31:0/3066169040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 0x7f5b8c108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 --2- 172.21.15.31:0/3066169040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c1031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 --2- 172.21.15.31:0/3066169040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 0x7f5b8c105b00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 -- 172.21.15.31:0/3066169040 >> 172.21.15.31:0/3066169040 conn(0x7f5b8c0fa840 msgr2=0x7f5b8c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 -- 172.21.15.31:0/3066169040 shutdown_connections 2023-07-29T16:14:39.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.406+0000 7f5b90dff700 1 -- 172.21.15.31:0/3066169040 wait complete. 2023-07-29T16:14:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 Processor -- start 2023-07-29T16:14:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 -- start start 2023-07-29T16:14:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 0x7f5b8c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 0x7f5b8c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c19dbc0 con 0x7f5b8c103720 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c19dd00 con 0x7f5b8c106040 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.407+0000 7f5b90dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c19de90 con 0x7f5b8c100e00 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 0x7f5b8c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57062/0 (socket says 172.21.15.31:57062) 2023-07-29T16:14:39.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 -- 172.21.15.31:0/1906996456 learned_addr learned my addr 172.21.15.31:0/1906996456 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 0x7f5b8c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 -- 172.21.15.31:0/1906996456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 msgr2=0x7f5b8c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 0x7f5b8c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 -- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 msgr2=0x7f5b8c196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 0x7f5b8c196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 -- 172.21.15.31:0/1906996456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b740095f0 con 0x7f5b8c100e00 2023-07-29T16:14:39.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 0x7f5b8c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.408+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c196040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00e9a0 tx=0x7f5b7c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.409+0000 7f5b837fe700 1 -- 172.21.15.31:0/1906996456 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c009790 con 0x7f5b8c100e00 2023-07-29T16:14:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.409+0000 7f5b837fe700 1 -- 172.21.15.31:0/1906996456 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7c010430 con 0x7f5b8c100e00 2023-07-29T16:14:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.409+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c19e170 con 0x7f5b8c100e00 2023-07-29T16:14:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.409+0000 7f5b837fe700 1 -- 172.21.15.31:0/1906996456 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c018680 con 0x7f5b8c100e00 2023-07-29T16:14:39.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.409+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c19e680 con 0x7f5b8c100e00 2023-07-29T16:14:39.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.412+0000 7f5b837fe700 1 -- 172.21.15.31:0/1906996456 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5b7c0105a0 con 0x7f5b8c100e00 2023-07-29T16:14:39.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.413+0000 7f5ba46e0b80 1 Processor -- start 2023-07-29T16:14:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 -- start start 2023-07-29T16:14:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b837fe700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b780a0790 0x7f5b780a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b837fe700 1 -- 172.21.15.31:0/1906996456 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5b7c014070 con 0x7f5b8c100e00 2023-07-29T16:14:39.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b780a0790 0x7f5b780a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6b5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba60ba10 con 0x563cba6b3be0 2023-07-29T16:14:39.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6093d0 con 0x563cba6b4530 2023-07-29T16:14:39.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba604cc0 con 0x563cba6b52f0 2023-07-29T16:14:39.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6b3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52348/0 (socket says 172.21.15.31:52348) 2023-07-29T16:14:39.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2620074384 learned_addr learned my addr 172.21.15.31:0/2620074384 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.414+0000 7f5b83fff700 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6b5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2620074384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 msgr2=0x563cba6b5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6b5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2620074384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 msgr2=0x563cba6b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2620074384 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cba432de0 con 0x563cba6b3be0 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b780a0790 0x7f5b780a2c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b84003e40 tx=0x7f5b84009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6b3ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c099410 tx=0x7f5b7c099090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d38b5ead7bb7d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b80ff9700 1 -- 172.21.15.31:0/2620074384 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c099480 con 0x563cba6b3be0 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b80ff9700 1 -- 172.21.15.31:0/2620074384 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7c09d4b0 con 0x563cba6b3be0 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.415+0000 7f5b80ff9700 1 -- 172.21.15.31:0/2620074384 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c098970 con 0x563cba6b3be0 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2620074384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 msgr2=0x563cba6b3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6b3ff0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c099410 tx=0x7f5b7c099090 comp rx=0 tx=0).stop 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2620074384 shutdown_connections 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6b3ff0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2620074384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2620074384 >> 172.21.15.31:0/2620074384 conn(0x563cba6a93e0 msgr2=0x563cba6ae990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2620074384 shutdown_connections 2023-07-29T16:14:39.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.416+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2620074384 wait complete. 2023-07-29T16:14:39.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 -- start start 2023-07-29T16:14:39.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 Processor -- start 2023-07-29T16:14:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b4530 0x563cba6c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b52f0 0x563cba6c9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba5a1510 con 0x563cba6b3be0 2023-07-29T16:14:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba5a11b0 con 0x563cba6b52f0 2023-07-29T16:14:39.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba488920 con 0x563cba6b4530 2023-07-29T16:14:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.417+0000 7f5ba46e0b80 10 client.0 _ll_register_callbacks cb 0x563cba5aa620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b4530 0x563cba6c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b4530 0x563cba6c21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57080/0 (socket says 172.21.15.31:57080) 2023-07-29T16:14:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b83fff700 1 -- 172.21.15.31:0/3357894077 learned_addr learned my addr 172.21.15.31:0/3357894077 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b52f0 0x563cba6c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 -- 172.21.15.31:0/3357894077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b4530 msgr2=0x563cba6c21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b4530 0x563cba6c21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 -- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b52f0 msgr2=0x563cba6c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b52f0 0x563cba6c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 -- 172.21.15.31:0/3357894077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c19e680 con 0x563cba6b3be0 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b52f0 0x563cba6c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:39.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c1c80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c099ae0 tx=0x7f5b7c0958d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9cb5bcaa4fc4e1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b71ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.418+0000 7f5b71ffb700 1 -- 172.21.15.31:0/3357894077 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c097820 con 0x563cba6b3be0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cba62fd10 con 0x563cba6b3be0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5b71ffb700 1 -- 172.21.15.31:0/3357894077 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5b7c0a1440 con 0x563cba6b3be0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5b71ffb700 1 -- 172.21.15.31:0/3357894077 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b7c0a1620 con 0x563cba6b3be0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 10 client.15483 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563cba41bc60 con 0x563cba6b3be0 2023-07-29T16:14:39.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5b71ffb700 1 -- 172.21.15.31:0/3357894077 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563cba62fd10 con 0x563cba6b3be0 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5b71ffb700 1 -- 172.21.15.31:0/3357894077 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5b7c0c9440 con 0x563cba6b3be0 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 10 client.15483 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 10 client.15483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 20 client.15483 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba6cd570 0x563cba6cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 4 client.15483 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:39.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.419+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cba682d70 con 0x563cba6cd570 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.420+0000 7f5b83fff700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba6cd570 0x563cba6cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.420+0000 7f5b83fff700 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba6cd570 0x563cba6cf980 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23ee2a9d42a8573 server_cookie=39f1a28fe6096c5c in_seq=0 out_seq=0 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.420+0000 7f5b71ffb700 10 client.15483 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.421+0000 7f5b71ffb700 1 -- 172.21.15.31:0/3357894077 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563cba682d70 con 0x563cba6cd570 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.421+0000 7f5b71ffb700 10 client.15483 handle_command_reply: tid=0 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.421+0000 7f5ba46e0b80 1 client.15483 shutdown 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 msgr2=0x563cba6c1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c1c80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c099ae0 tx=0x7f5b7c0958d0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 shutdown_connections 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba6cd570 0x563cba6cf980 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b52f0 0x563cba6c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c1c80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5b71ffb700 0 client.15483 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/3357894077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b4530 0x563cba6c21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 >> 172.21.15.31:0/3357894077 conn(0x563cba6a93e0 msgr2=0x563cba6ae060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 shutdown_connections 2023-07-29T16:14:39.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/3357894077 wait complete. 2023-07-29T16:14:39.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.422+0000 7f5ba46e0b80 20 client.15483 trim_cache size 0 max 16384 2023-07-29T16:14:39.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 Processor -- start 2023-07-29T16:14:39.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 -- start start 2023-07-29T16:14:39.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b3be0 0x563cba6c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b4530 0x563cba6c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6ae060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6c1170 con 0x563cba6b4530 2023-07-29T16:14:39.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6ab490 con 0x563cba6b3be0 2023-07-29T16:14:39.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6a67f0 con 0x563cba6b52f0 2023-07-29T16:14:39.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.439+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b4530 0x563cba6c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b4530 0x563cba6c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52368/0 (socket says 172.21.15.31:52368) 2023-07-29T16:14:39.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b83fff700 1 -- 172.21.15.31:0/4135962080 learned_addr learned my addr 172.21.15.31:0/4135962080 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b3be0 0x563cba6c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 -- 172.21.15.31:0/4135962080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 msgr2=0x563cba6ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6ae060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 -- 172.21.15.31:0/4135962080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b4530 msgr2=0x563cba6c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b4530 0x563cba6c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 -- 172.21.15.31:0/4135962080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cba459250 con 0x563cba6b3be0 2023-07-29T16:14:39.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b3be0 0x563cba6c08f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c095ef0 tx=0x7f5b7c099090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0042829a1c4f2f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b817fa700 1 -- 172.21.15.31:0/4135962080 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c00cae0 con 0x563cba6b3be0 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.440+0000 7f5b817fa700 1 -- 172.21.15.31:0/4135962080 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7c097820 con 0x563cba6b3be0 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5b817fa700 1 -- 172.21.15.31:0/4135962080 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c0a16a0 con 0x563cba6b3be0 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/4135962080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b3be0 msgr2=0x563cba6c08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b3be0 0x563cba6c08f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c095ef0 tx=0x7f5b7c099090 comp rx=0 tx=0).stop 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/4135962080 shutdown_connections 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b3be0 0x563cba6c08f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/4135962080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b4530 0x563cba6c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/4135962080 >> 172.21.15.31:0/4135962080 conn(0x563cba6a93e0 msgr2=0x563cba6af690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/4135962080 shutdown_connections 2023-07-29T16:14:39.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/4135962080 wait complete. 2023-07-29T16:14:39.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.441+0000 7f5ba46e0b80 1 -- start start 2023-07-29T16:14:39.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 Processor -- start 2023-07-29T16:14:39.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6c7ec0 con 0x563cba6b3be0 2023-07-29T16:14:39.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6c8030 con 0x563cba6b4530 2023-07-29T16:14:39.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cba6c81f0 con 0x563cba6b52f0 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5ba46e0b80 10 client.0 _ll_register_callbacks cb 0x563cba5aa620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5b83fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5b8a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52374/0 (socket says 172.21.15.31:52374) 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.442+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2147477679 learned_addr learned my addr 172.21.15.31:0/2147477679 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8ad9d700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2147477679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 msgr2=0x563cba6c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 msgr2=0x563cba6c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8a59c700 1 -- 172.21.15.31:0/2147477679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cba41bc60 con 0x563cba6b3be0 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b83fff700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b8a59c700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c8e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c0c94e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8a37674e4fc476 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b71ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:39.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b71ffb700 1 -- 172.21.15.31:0/2147477679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c09d2e0 con 0x563cba6b3be0 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b7c09d4b0 con 0x563cba6b3be0 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.443+0000 7f5b71ffb700 1 -- 172.21.15.31:0/2147477679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5b7c00cae0 con 0x563cba6b3be0 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5b71ffb700 1 -- 172.21.15.31:0/2147477679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b7c095480 con 0x563cba6b3be0 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 10 client.15489 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563cba4312b0 con 0x563cba6b3be0 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5b71ffb700 1 -- 172.21.15.31:0/2147477679 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b7c09d4b0 con 0x563cba6b3be0 2023-07-29T16:14:39.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5b71ffb700 1 -- 172.21.15.31:0/2147477679 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5b7c09a260 con 0x563cba6b3be0 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 10 client.15489 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 10 client.15489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 20 client.15489 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba7202e0 0x563cba7226c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 4 client.15489 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563cba5fdd40 con 0x563cba7202e0 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.444+0000 7f5b83fff700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba7202e0 0x563cba7226c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:39.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5b83fff700 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba7202e0 0x563cba7226c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b163fd2bf17775de server_cookie=387b988a12c97320 in_seq=0 out_seq=0 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5b71ffb700 10 client.15489 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5b71ffb700 1 -- 172.21.15.31:0/2147477679 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563cba5fdd40 con 0x563cba7202e0 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5b71ffb700 10 client.15489 handle_command_reply: tid=0 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 client.15489 shutdown 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 msgr2=0x563cba6c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c8e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c000c00 tx=0x7f5b7c0c94e0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 shutdown_connections 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563cba7202e0 0x563cba7226c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563cba6b52f0 0x563cba6c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563cba6b3be0 0x563cba6c8e30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 --2- 172.21.15.31:0/2147477679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563cba6b4530 0x563cba6c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 >> 172.21.15.31:0/2147477679 conn(0x563cba6a93e0 msgr2=0x563cba6c08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5b71ffb700 0 client.15489 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 shutdown_connections 2023-07-29T16:14:39.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.445+0000 7f5ba46e0b80 1 -- 172.21.15.31:0/2147477679 wait complete. 2023-07-29T16:14:39.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5ba46e0b80 20 client.15489 trim_cache size 0 max 16384 2023-07-29T16:14:39.452 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:14:39.452 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:14:39.452 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:14:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: "1e023985-7ba5-47c3-aa52-6757ddf647e8": { 2023-07-29T16:14:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:14:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "1e023985-7ba5-47c3-aa52-6757ddf647e8", 2023-07-29T16:14:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:14:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:14:39.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:14:39.454 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:14:39.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b780a0790 msgr2=0x7f5b780a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b780a0790 0x7f5b780a2c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5b84003e40 tx=0x7f5b84009920 comp rx=0 tx=0).stop 2023-07-29T16:14:39.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 msgr2=0x7f5b8c196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:39.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c196040 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c00e9a0 tx=0x7f5b7c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 shutdown_connections 2023-07-29T16:14:39.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5b780a0790 0x7f5b780a2c40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b8c106040 0x7f5b8c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5b8c100e00 0x7f5b8c196040 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 --2- 172.21.15.31:0/1906996456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5b8c103720 0x7f5b8c196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:39.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 >> 172.21.15.31:0/1906996456 conn(0x7f5b8c0fa840 msgr2=0x7f5b8c101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:39.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 shutdown_connections 2023-07-29T16:14:39.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:14:39.446+0000 7f5b90dff700 1 -- 172.21.15.31:0/1906996456 wait complete. 2023-07-29T16:14:39.461 INFO:tasks.cephfs.filesystem:scrub status for tag:1e023985-7ba5-47c3-aa52-6757ddf647e8 - {'path': '/', 'tag': '1e023985-7ba5-47c3-aa52-6757ddf647e8', 'options': 'recursive,force'} 2023-07-29T16:14:40.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:40 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:14:40.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:40 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:40.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:40 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:14:40.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:40 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:40.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:40 smithi031 ceph-mon[95504]: pgmap v345: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:14:40.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:40 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:14:40.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:40 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:40.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:40 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:14:40.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:40 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:40.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:40 smithi039 ceph-mon[128858]: pgmap v345: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:14:40.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:40 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:14:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:40 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:40 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:14:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:40 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:14:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:40 smithi156 ceph-mon[129245]: pgmap v345: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:14:42.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:42 smithi031 ceph-mon[95504]: pgmap v346: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:14:42.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:42 smithi039 ceph-mon[128858]: pgmap v346: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:14:42.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:42 smithi156 ceph-mon[129245]: pgmap v346: 97 pgs: 97 active+clean; 650 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:14:43.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:43 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:14:43.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:43 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:14:43.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:43 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:14:44.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:44 smithi031 ceph-mon[95504]: pgmap v347: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-07-29T16:14:44.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:44 smithi039 ceph-mon[128858]: pgmap v347: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-07-29T16:14:44.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:44 smithi156 ceph-mon[129245]: pgmap v347: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-07-29T16:14:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 -- 172.21.15.31:0/2467952890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 msgr2=0x7f0ad0108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/2467952890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0108480 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009e00 tx=0x7f0ac800b330 comp rx=0 tx=0).stop 2023-07-29T16:14:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 -- 172.21.15.31:0/2467952890 shutdown_connections 2023-07-29T16:14:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/2467952890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0108480 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/2467952890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad0100e60 0x7f0ad0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/2467952890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0103780 0x7f0ad0105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 -- 172.21.15.31:0/2467952890 >> 172.21.15.31:0/2467952890 conn(0x7f0ad00fa840 msgr2=0x7f0ad00fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 -- 172.21.15.31:0/2467952890 shutdown_connections 2023-07-29T16:14:45.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.451+0000 7f0ad63ff700 1 -- 172.21.15.31:0/2467952890 wait complete. 2023-07-29T16:14:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.452+0000 7f0ad63ff700 1 Processor -- start 2023-07-29T16:14:45.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 -- start start 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 0x7f0ad0198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad0103780 0x7f0ad0198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad0199f80 con 0x7f0ad01060a0 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad019a0f0 con 0x7f0ad0100e60 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad63ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad019a2b0 con 0x7f0ad0103780 2023-07-29T16:14:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0acffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 0x7f0ad0198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0acffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 0x7f0ad0198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42668/0 (socket says 172.21.15.31:42668) 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0ad499c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0acffff700 1 -- 172.21.15.31:0/69017734 learned_addr learned my addr 172.21.15.31:0/69017734 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.453+0000 7f0acf7fe700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad0103780 0x7f0ad0198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acffff700 1 -- 172.21.15.31:0/69017734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad0103780 msgr2=0x7f0ad0198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acffff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad0103780 0x7f0ad0198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acffff700 1 -- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 msgr2=0x7f0ad0199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acffff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acffff700 1 -- 172.21.15.31:0/69017734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac8005e70 con 0x7f0ad0100e60 2023-07-29T16:14:45.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0ad499c700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:45.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acffff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 0x7f0ad0198560 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000e9f0 tx=0x7f0ac000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:45.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.454+0000 7f0acd7fa700 1 -- 172.21.15.31:0/69017734 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0009690 con 0x7f0ad0100e60 2023-07-29T16:14:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.455+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad019a590 con 0x7f0ad0100e60 2023-07-29T16:14:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.455+0000 7f0acd7fa700 1 -- 172.21.15.31:0/69017734 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac000ce80 con 0x7f0ad0100e60 2023-07-29T16:14:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.455+0000 7f0acd7fa700 1 -- 172.21.15.31:0/69017734 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0018420 con 0x7f0ad0100e60 2023-07-29T16:14:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.455+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad01a0d50 con 0x7f0ad0100e60 2023-07-29T16:14:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.458+0000 7f0acd7fa700 1 -- 172.21.15.31:0/69017734 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0ac000c9c0 con 0x7f0ad0100e60 2023-07-29T16:14:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.459+0000 7f0ae9e09b80 1 Processor -- start 2023-07-29T16:14:45.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.459+0000 7f0acd7fa700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ab80a0790 0x7f0ab80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.459+0000 7f0ae9e09b80 1 -- start start 2023-07-29T16:14:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.459+0000 7f0acd7fa700 1 -- 172.21.15.31:0/69017734 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ac0014070 con 0x7f0ad0100e60 2023-07-29T16:14:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ad499c700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ab80a0790 0x7f0ab80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886e18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886e3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886399b0 con 0x557a886e1df0 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a88637370 con 0x557a886e2cf0 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a88632c60 con 0x557a8853d6b0 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886e18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886e27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38518/0 (socket says 172.21.15.31:38518) 2023-07-29T16:14:45.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acf7fe700 1 -- 172.21.15.31:0/3837646101 learned_addr learned my addr 172.21.15.31:0/3837646101 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886e18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51598/0 (socket says 172.21.15.31:51598) 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ad499c700 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 -- 172.21.15.31:0/3837646101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 msgr2=0x557a886e3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886e3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 -- 172.21.15.31:0/3837646101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 msgr2=0x557a886e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886e27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0acffff700 1 -- 172.21.15.31:0/3837646101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a88460d80 con 0x557a8853d6b0 2023-07-29T16:14:45.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.460+0000 7f0ad499c700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ab80a0790 0x7f0ab80a2c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8007df0 tx=0x7f0ac8009b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:45.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.461+0000 7f0acffff700 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886e18b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0010fd0 tx=0x7f0ac0098d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fad5a137d35beef server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.461+0000 7f0ab67fc700 1 -- 172.21.15.31:0/3837646101 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac009d170 con 0x557a8853d6b0 2023-07-29T16:14:45.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.461+0000 7f0ab67fc700 1 -- 172.21.15.31:0/3837646101 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac00a17b0 con 0x557a8853d6b0 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.461+0000 7f0ab67fc700 1 -- 172.21.15.31:0/3837646101 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0098ab0 con 0x557a8853d6b0 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/3837646101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 msgr2=0x557a886e18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886e18b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0010fd0 tx=0x7f0ac0098d40 comp rx=0 tx=0).stop 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/3837646101 shutdown_connections 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886e18b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/3837646101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/3837646101 >> 172.21.15.31:0/3837646101 conn(0x557a886d6fd0 msgr2=0x557a886dc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/3837646101 shutdown_connections 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/3837646101 wait complete. 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.462+0000 7f0ae9e09b80 1 -- start start 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 Processor -- start 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e1df0 0x557a886eb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 0x557a886f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a885cf4b0 con 0x557a886e2cf0 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a885cf150 con 0x557a886e1df0 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a884b68c0 con 0x557a8853d6b0 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ae9e09b80 10 client.0 _ll_register_callbacks cb 0x557a885d85c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ad499c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 0x557a886f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ad499c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 0x557a886f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38530/0 (socket says 172.21.15.31:38530) 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.463+0000 7f0ad499c700 1 -- 172.21.15.31:0/2898677852 learned_addr learned my addr 172.21.15.31:0/2898677852 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:45.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0acf7fe700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e1df0 0x557a886eb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0acffff700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ad499c700 1 -- 172.21.15.31:0/2898677852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 msgr2=0x557a886eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ad499c700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ad499c700 1 -- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e1df0 msgr2=0x557a886eb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ad499c700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e1df0 0x557a886eb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ad499c700 1 -- 172.21.15.31:0/2898677852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad01a0d50 con 0x557a886e2cf0 2023-07-29T16:14:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0acf7fe700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e1df0 0x557a886eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ad499c700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 0x557a886f24d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8015000 tx=0x7f0ac800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39d98b6f0b63d45 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0aaf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/2898677852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac802f070 con 0x557a886e2cf0 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a8865dcb0 con 0x557a886e2cf0 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.464+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/2898677852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ac801fe20 con 0x557a886e2cf0 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/2898677852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ac800e040 con 0x557a886e2cf0 2023-07-29T16:14:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 10 client.15501 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557a88449c00 con 0x557a886e2cf0 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/2898677852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac8039020 con 0x557a886e2cf0 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/2898677852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0ac8032030 con 0x557a886e2cf0 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 10 client.15501 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 10 client.15501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 20 client.15501 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a886f6d30 0x557a886f9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 4 client.15501 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.465+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a886b0d10 con 0x557a886f6d30 2023-07-29T16:14:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.466+0000 7f0acffff700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a886f6d30 0x557a886f9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.466+0000 7f0acffff700 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a886f6d30 0x557a886f9140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c9468d7348f00f3 server_cookie=c14c712efe071b5a in_seq=0 out_seq=0 2023-07-29T16:14:45.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.466+0000 7f0aaf7fe700 10 client.15501 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/2898677852 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557a886b0d10 con 0x557a886f6d30 2023-07-29T16:14:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0aaf7fe700 10 client.15501 handle_command_reply: tid=0 2023-07-29T16:14:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 client.15501 shutdown 2023-07-29T16:14:45.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 msgr2=0x557a886f24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 0x557a886f24d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8015000 tx=0x7f0ac800b330 comp rx=0 tx=0).stop 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 shutdown_connections 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a886f6d30 0x557a886f9140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e2cf0 0x557a886f24d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a8853d6b0 0x557a886eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0aaf7fe700 0 client.15501 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:45.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/2898677852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e1df0 0x557a886eb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.467+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 >> 172.21.15.31:0/2898677852 conn(0x557a886d6fd0 msgr2=0x557a886dba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.468+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 shutdown_connections 2023-07-29T16:14:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.468+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/2898677852 wait complete. 2023-07-29T16:14:45.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.468+0000 7f0ae9e09b80 20 client.15501 trim_cache size 0 max 16384 2023-07-29T16:14:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 Processor -- start 2023-07-29T16:14:45.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 -- start start 2023-07-29T16:14:45.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a8853d6b0 0x557a886e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886f2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e2cf0 0x557a886e6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886d42c0 con 0x557a886e1df0 2023-07-29T16:14:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886eaac0 con 0x557a8853d6b0 2023-07-29T16:14:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886d40e0 con 0x557a886e2cf0 2023-07-29T16:14:45.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0acf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0ad499c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e2cf0 0x557a886e6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0acf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886f2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38546/0 (socket says 172.21.15.31:38546) 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.483+0000 7f0acf7fe700 1 -- 172.21.15.31:0/71405717 learned_addr learned my addr 172.21.15.31:0/71405717 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0acffff700 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a8853d6b0 0x557a886e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ad499c700 1 -- 172.21.15.31:0/71405717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a8853d6b0 msgr2=0x557a886e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ad499c700 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a8853d6b0 0x557a886e5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ad499c700 1 -- 172.21.15.31:0/71405717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 msgr2=0x557a886f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ad499c700 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ad499c700 1 -- 172.21.15.31:0/71405717 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a884871f0 con 0x557a886e2cf0 2023-07-29T16:14:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ad499c700 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e2cf0 0x557a886e6140 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009c10 tx=0x7f0ac8009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5605f231e8d1ff1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ab6ffd700 1 -- 172.21.15.31:0/71405717 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac800e070 con 0x557a886e2cf0 2023-07-29T16:14:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ab6ffd700 1 -- 172.21.15.31:0/71405717 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac8039040 con 0x557a886e2cf0 2023-07-29T16:14:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.484+0000 7f0ab6ffd700 1 -- 172.21.15.31:0/71405717 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac802f070 con 0x557a886e2cf0 2023-07-29T16:14:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/71405717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e2cf0 msgr2=0x557a886e6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e2cf0 0x557a886e6140 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009c10 tx=0x7f0ac8009920 comp rx=0 tx=0).stop 2023-07-29T16:14:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/71405717 shutdown_connections 2023-07-29T16:14:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e2cf0 0x557a886e6140 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a8853d6b0 0x557a886e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/71405717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a886e1df0 0x557a886f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/71405717 >> 172.21.15.31:0/71405717 conn(0x557a886d6fd0 msgr2=0x557a886e90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/71405717 shutdown_connections 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/71405717 wait complete. 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.485+0000 7f0ae9e09b80 1 -- start start 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 Processor -- start 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 0x557a886ee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e1df0 0x557a886eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886f1b30 con 0x557a8853d6b0 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886f1ca0 con 0x557a886e2cf0 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a886f1e60 con 0x557a886e1df0 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ae9e09b80 10 client.0 _ll_register_callbacks cb 0x557a885d85c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0acffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 0x557a886ee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0acffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 0x557a886ee620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38554/0 (socket says 172.21.15.31:38554) 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0acffff700 1 -- 172.21.15.31:0/524628420 learned_addr learned my addr 172.21.15.31:0/524628420 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.486+0000 7f0ad499c700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acf7fe700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e1df0 0x557a886eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acffff700 1 -- 172.21.15.31:0/524628420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e1df0 msgr2=0x557a886eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acffff700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e1df0 0x557a886eeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acffff700 1 -- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 msgr2=0x557a886f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acffff700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acffff700 1 -- 172.21.15.31:0/524628420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a88449c00 con 0x557a8853d6b0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0acffff700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 0x557a886ee620 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000c5b0 tx=0x7f0ac0010870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a7429847705e0a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0aaf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/524628420 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac00b7040 con 0x557a8853d6b0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/524628420 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ac009d170 con 0x557a8853d6b0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ac00a17b0 con 0x557a8853d6b0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/524628420 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ac00956a0 con 0x557a8853d6b0 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.487+0000 7f0ae9e09b80 10 client.15507 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:45.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557a8845f250 con 0x557a8853d6b0 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/524628420 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ac00a17b0 con 0x557a8853d6b0 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/524628420 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0ac00a1320 con 0x557a8853d6b0 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 10 client.15507 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 10 client.15507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 20 client.15507 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a88749340 0x557a8874b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 4 client.15507 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:14:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557a8862bce0 con 0x557a88749340 2023-07-29T16:14:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.488+0000 7f0acf7fe700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a88749340 0x557a8874b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.489+0000 7f0acf7fe700 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a88749340 0x557a8874b750 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82b2812b9fb4eaae server_cookie=34159a4b1c33b367 in_seq=0 out_seq=0 2023-07-29T16:14:45.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.489+0000 7f0aaf7fe700 10 client.15507 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0aaf7fe700 1 -- 172.21.15.31:0/524628420 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557a8862bce0 con 0x557a88749340 2023-07-29T16:14:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0aaf7fe700 10 client.15507 handle_command_reply: tid=0 2023-07-29T16:14:45.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 client.15507 shutdown 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 msgr2=0x557a886ee620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 0x557a886ee620 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000c5b0 tx=0x7f0ac0010870 comp rx=0 tx=0).stop 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 shutdown_connections 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a88749340 0x557a8874b750 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a886e2cf0 0x557a886f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8853d6b0 0x557a886ee620 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0aaf7fe700 0 client.15507 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 --2- 172.21.15.31:0/524628420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a886e1df0 0x557a886eeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 >> 172.21.15.31:0/524628420 conn(0x557a886d6fd0 msgr2=0x557a886e5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 shutdown_connections 2023-07-29T16:14:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 1 -- 172.21.15.31:0/524628420 wait complete. 2023-07-29T16:14:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.497+0000 7f0ae9e09b80 20 client.15507 trim_cache size 0 max 16384 2023-07-29T16:14:45.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:14:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:14:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:14:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:14:45.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ab80a0790 msgr2=0x7f0ab80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ab80a0790 0x7f0ab80a2c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8007df0 tx=0x7f0ac8009b60 comp rx=0 tx=0).stop 2023-07-29T16:14:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 msgr2=0x7f0ad0198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 0x7f0ad0198560 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000e9f0 tx=0x7f0ac000ed00 comp rx=0 tx=0).stop 2023-07-29T16:14:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 shutdown_connections 2023-07-29T16:14:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ab80a0790 0x7f0ab80a2c40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0ad01060a0 0x7f0ad0199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ad0100e60 0x7f0ad0198560 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 --2- 172.21.15.31:0/69017734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ad0103780 0x7f0ad0198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 >> 172.21.15.31:0/69017734 conn(0x7f0ad00fa840 msgr2=0x7f0ad0101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 shutdown_connections 2023-07-29T16:14:45.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:45.498+0000 7f0ad63ff700 1 -- 172.21.15.31:0/69017734 wait complete. 2023-07-29T16:14:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:46 smithi031 ceph-mon[95504]: pgmap v348: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:46.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:46 smithi039 ceph-mon[128858]: pgmap v348: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:46.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:46 smithi156 ceph-mon[129245]: pgmap v348: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:47 smithi156 ceph-mon[129245]: pgmap v349: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:47 smithi031 ceph-mon[95504]: pgmap v349: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:48.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:47 smithi039 ceph-mon[128858]: pgmap v349: 97 pgs: 97 active+clean; 660 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 6 op/s 2023-07-29T16:14:50.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:50 smithi031 ceph-mon[95504]: pgmap v350: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-07-29T16:14:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:50 smithi039 ceph-mon[128858]: pgmap v350: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-07-29T16:14:50.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:50 smithi156 ceph-mon[129245]: pgmap v350: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-07-29T16:14:52.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:52 smithi031 ceph-mon[95504]: pgmap v351: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-07-29T16:14:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:52 smithi039 ceph-mon[128858]: pgmap v351: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-07-29T16:14:52.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:52 smithi156 ceph-mon[129245]: pgmap v351: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2023-07-29T16:14:54.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:54 smithi031 ceph-mon[95504]: pgmap v352: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-07-29T16:14:54.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:54 smithi039 ceph-mon[128858]: pgmap v352: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-07-29T16:14:54.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:54 smithi156 ceph-mon[129245]: pgmap v352: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2023-07-29T16:14:56.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:56 smithi031 ceph-mon[95504]: pgmap v353: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 7 op/s 2023-07-29T16:14:56.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:56 smithi039 ceph-mon[128858]: pgmap v353: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 7 op/s 2023-07-29T16:14:56.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:56 smithi156 ceph-mon[129245]: pgmap v353: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 853 B/s wr, 7 op/s 2023-07-29T16:14:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:14:57 smithi156 ceph-mon[129245]: pgmap v354: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2023-07-29T16:14:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:14:57 smithi031 ceph-mon[95504]: pgmap v354: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2023-07-29T16:14:58.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:14:57 smithi039 ceph-mon[128858]: pgmap v354: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2023-07-29T16:14:58.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 -- 172.21.15.31:0/3186365320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101ec0 msgr2=0x7fec28108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 --2- 172.21.15.31:0/3186365320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101ec0 0x7fec28108780 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fec2000b0a0 tx=0x7fec2000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 -- 172.21.15.31:0/3186365320 shutdown_connections 2023-07-29T16:14:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 --2- 172.21.15.31:0/3186365320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101ec0 0x7fec28108780 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 --2- 172.21.15.31:0/3186365320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28100340 0x7fec28100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 --2- 172.21.15.31:0/3186365320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec28101530 0x7fec28101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 -- 172.21.15.31:0/3186365320 >> 172.21.15.31:0/3186365320 conn(0x7fec280fb920 msgr2=0x7fec280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 -- 172.21.15.31:0/3186365320 shutdown_connections 2023-07-29T16:14:58.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.658+0000 7fec2efff700 1 -- 172.21.15.31:0/3186365320 wait complete. 2023-07-29T16:14:58.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.659+0000 7fec2efff700 1 Processor -- start 2023-07-29T16:14:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.659+0000 7fec2efff700 1 -- start start 2023-07-29T16:14:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec28100340 0x7fec28196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 0x7fec281967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28101ec0 0x7fec28197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec28197cb0 con 0x7fec28101530 2023-07-29T16:14:58.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec28197e20 con 0x7fec28101ec0 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec28197fe0 con 0x7fec28100340 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 0x7fec281967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 0x7fec281967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44896/0 (socket says 172.21.15.31:44896) 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 -- 172.21.15.31:0/2014386068 learned_addr learned my addr 172.21.15.31:0/2014386068 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2d59c700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28101ec0 0x7fec28197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec28100340 0x7fec28196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 -- 172.21.15.31:0/2014386068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec28100340 msgr2=0x7fec28196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec28100340 0x7fec28196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 -- 172.21.15.31:0/2014386068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28101ec0 msgr2=0x7fec28197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28101ec0 0x7fec28197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec27fff700 1 -- 172.21.15.31:0/2014386068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20009d00 con 0x7fec28101530 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.660+0000 7fec2d59c700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28101ec0 0x7fec28197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.661+0000 7fec27fff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 0x7fec281967d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e9a0 tx=0x7fec1400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.661+0000 7fec25ffb700 1 -- 172.21.15.31:0/2014386068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14009790 con 0x7fec28101530 2023-07-29T16:14:58.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.661+0000 7fec25ffb700 1 -- 172.21.15.31:0/2014386068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec14010430 con 0x7fec28101530 2023-07-29T16:14:58.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.661+0000 7fec25ffb700 1 -- 172.21.15.31:0/2014386068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14018680 con 0x7fec28101530 2023-07-29T16:14:58.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.661+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec281982c0 con 0x7fec28101530 2023-07-29T16:14:58.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.661+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2819ea80 con 0x7fec28101530 2023-07-29T16:14:58.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.664+0000 7fec25ffb700 1 -- 172.21.15.31:0/2014386068 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fec140105a0 con 0x7fec28101530 2023-07-29T16:14:58.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.666+0000 7fec25ffb700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fec180a0810 0x7fec180a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.666+0000 7fec25ffb700 1 -- 172.21.15.31:0/2014386068 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fec14014070 con 0x7fec28101530 2023-07-29T16:14:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.666+0000 7fec4290db80 1 Processor -- start 2023-07-29T16:14:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.666+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fec180a0810 0x7fec180a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.666+0000 7fec4290db80 1 -- start start 2023-07-29T16:14:58.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.666+0000 7fec4290db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52fa0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec4290db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52fa1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec4290db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fa2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec4290db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52ef49b0 con 0x55ee52fa2100 2023-07-29T16:14:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec4290db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52ef2370 con 0x55ee52df86b0 2023-07-29T16:14:58.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec4290db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52eedc60 con 0x55ee52fa1200 2023-07-29T16:14:58.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fec180a0810 0x7fec180a2cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec28101390 tx=0x7fec1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec27fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52fa1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52fa0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec27fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52fa1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34118/0 (socket says 172.21.15.31:34118) 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec27fff700 1 -- 172.21.15.31:0/1331374030 learned_addr learned my addr 172.21.15.31:0/1331374030 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fa2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2d59c700 1 -- 172.21.15.31:0/1331374030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 msgr2=0x55ee52fa1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52fa1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2d59c700 1 -- 172.21.15.31:0/1331374030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 msgr2=0x55ee52fa0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52fa0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2d59c700 1 -- 172.21.15.31:0/1331374030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee52d1bd80 con 0x55ee52fa2100 2023-07-29T16:14:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52fa0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.667+0000 7fec27fff700 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52fa1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:58.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.668+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fa2570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec20009fd0 tx=0x7fec20009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b6e33311237ef7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.668+0000 7fec12ffd700 1 -- 172.21.15.31:0/1331374030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000e040 con 0x55ee52fa2100 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.668+0000 7fec12ffd700 1 -- 172.21.15.31:0/1331374030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec20006000 con 0x55ee52fa2100 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.668+0000 7fec12ffd700 1 -- 172.21.15.31:0/1331374030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec20003e00 con 0x55ee52fa2100 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 -- 172.21.15.31:0/1331374030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 msgr2=0x55ee52fa2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fa2570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fec20009fd0 tx=0x7fec20009c00 comp rx=0 tx=0).stop 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 -- 172.21.15.31:0/1331374030 shutdown_connections 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fa2570 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52fa0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 --2- 172.21.15.31:0/1331374030 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52fa1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 -- 172.21.15.31:0/1331374030 >> 172.21.15.31:0/1331374030 conn(0x55ee52f91fd0 msgr2=0x55ee52f97300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 -- 172.21.15.31:0/1331374030 shutdown_connections 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.669+0000 7fec4290db80 1 -- 172.21.15.31:0/1331374030 wait complete. 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.670+0000 7fec4290db80 1 -- start start 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.670+0000 7fec4290db80 1 Processor -- start 2023-07-29T16:14:58.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52faa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52faa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fb17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52e8a4b0 con 0x55ee52fa2100 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52e8a150 con 0x55ee52df86b0 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52d718c0 con 0x55ee52fa1200 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec4290db80 10 client.0 _ll_register_callbacks cb 0x55ee52e935c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:58.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fb17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52faa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fb17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44918/0 (socket says 172.21.15.31:44918) 2023-07-29T16:14:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec27fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52faa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 -- 172.21.15.31:0/3459623177 learned_addr learned my addr 172.21.15.31:0/3459623177 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 -- 172.21.15.31:0/3459623177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 msgr2=0x55ee52faa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52faa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 -- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 msgr2=0x55ee52faa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52faa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2d59c700 1 -- 172.21.15.31:0/3459623177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c017070 con 0x55ee52fa2100 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.671+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52faa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec2d59c700 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fb17a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fec2000d010 tx=0x7fec20003f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806155e46b18cf22 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec07fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec07fff700 1 -- 172.21.15.31:0/3459623177 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000e040 con 0x55ee52fa2100 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec07fff700 1 -- 172.21.15.31:0/3459623177 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fec2000bce0 con 0x55ee52fa2100 2023-07-29T16:14:58.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee52f18cb0 con 0x55ee52fa2100 2023-07-29T16:14:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec07fff700 1 -- 172.21.15.31:0/3459623177 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec200050c0 con 0x55ee52fa2100 2023-07-29T16:14:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec4290db80 10 client.15519 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fec2819ea80 con 0x55ee52fa2100 2023-07-29T16:14:58.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.672+0000 7fec07fff700 1 -- 172.21.15.31:0/3459623177 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ee52f18cb0 con 0x55ee52fa2100 2023-07-29T16:14:58.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec07fff700 1 -- 172.21.15.31:0/3459623177 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fec20019070 con 0x55ee52fa2100 2023-07-29T16:14:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec4290db80 10 client.15519 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec4290db80 10 client.15519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec4290db80 20 client.15519 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec4290db80 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee52fb5ce0 0x55ee52fb80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec4290db80 4 client.15519 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:14:58.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee52f6bd10 con 0x55ee52fb5ce0 2023-07-29T16:14:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.673+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee52fb5ce0 0x55ee52fb80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.674+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee52fb5ce0 0x55ee52fb80f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebc97ec84422c1ce server_cookie=dda9cb5676257e5d in_seq=0 out_seq=0 2023-07-29T16:14:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.674+0000 7fec07fff700 10 client.15519 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.674+0000 7fec07fff700 1 -- 172.21.15.31:0/3459623177 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ee52f6bd10 con 0x55ee52fb5ce0 2023-07-29T16:14:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.674+0000 7fec07fff700 10 client.15519 handle_command_reply: tid=0 2023-07-29T16:14:58.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 client.15519 shutdown 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 msgr2=0x55ee52fb17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fb17a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fec2000d010 tx=0x7fec20003f30 comp rx=0 tx=0).stop 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 shutdown_connections 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee52fb5ce0 0x55ee52fb80f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fb17a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52df86b0 0x55ee52faa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec07fff700 0 client.15519 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:58.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 --2- 172.21.15.31:0/3459623177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa1200 0x55ee52faa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 >> 172.21.15.31:0/3459623177 conn(0x55ee52f91fd0 msgr2=0x55ee52f96a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 shutdown_connections 2023-07-29T16:14:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 1 -- 172.21.15.31:0/3459623177 wait complete. 2023-07-29T16:14:58.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.675+0000 7fec4290db80 20 client.15519 trim_cache size 0 max 16384 2023-07-29T16:14:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 Processor -- start 2023-07-29T16:14:58.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 -- start start 2023-07-29T16:14:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 0x55ee52f96a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa2100 0x55ee52fa8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52f983a0 con 0x55ee52df86b0 2023-07-29T16:14:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fb8750 con 0x55ee52fa1200 2023-07-29T16:14:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.690+0000 7fec4290db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fa81f0 con 0x55ee52fa2100 2023-07-29T16:14:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 0x55ee52f96a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 0x55ee52f96a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44934/0 (socket says 172.21.15.31:44934) 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 -- 172.21.15.31:0/1663089174 learned_addr learned my addr 172.21.15.31:0/1663089174 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec27fff700 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa2100 0x55ee52fa8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 -- 172.21.15.31:0/1663089174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa2100 msgr2=0x55ee52fa8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa2100 0x55ee52fa8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 -- 172.21.15.31:0/1663089174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 msgr2=0x55ee52fb1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 -- 172.21.15.31:0/1663089174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec281090b0 con 0x55ee52df86b0 2023-07-29T16:14:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.691+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 0x55ee52f96a40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55ee52fb8990 tx=0x7fec1c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d7c3b9518943bc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec137fe700 1 -- 172.21.15.31:0/1663089174 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c022030 con 0x55ee52df86b0 2023-07-29T16:14:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec137fe700 1 -- 172.21.15.31:0/1663089174 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec1c023040 con 0x55ee52df86b0 2023-07-29T16:14:58.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec137fe700 1 -- 172.21.15.31:0/1663089174 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c033700 con 0x55ee52df86b0 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 -- 172.21.15.31:0/1663089174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 msgr2=0x55ee52f96a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 0x55ee52f96a40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55ee52fb8990 tx=0x7fec1c014000 comp rx=0 tx=0).stop 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 -- 172.21.15.31:0/1663089174 shutdown_connections 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52fa2100 0x55ee52fa8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52df86b0 0x55ee52f96a40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 --2- 172.21.15.31:0/1663089174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 -- 172.21.15.31:0/1663089174 >> 172.21.15.31:0/1663089174 conn(0x55ee52f91fd0 msgr2=0x55ee52f972d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.692+0000 7fec4290db80 1 -- 172.21.15.31:0/1663089174 shutdown_connections 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 -- 172.21.15.31:0/1663089174 wait complete. 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 -- start start 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 Processor -- start 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52df86b0 0x55ee52fb0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fae060 con 0x55ee52fa2100 2023-07-29T16:14:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fae1d0 con 0x55ee52fa1200 2023-07-29T16:14:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fae390 con 0x55ee52df86b0 2023-07-29T16:14:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.693+0000 7fec4290db80 10 client.0 _ll_register_callbacks cb 0x55ee52e935c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:14:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fada80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44942/0 (socket says 172.21.15.31:44942) 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 -- 172.21.15.31:0/1391850807 learned_addr learned my addr 172.21.15.31:0/1391850807 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec27fff700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52df86b0 0x55ee52fb0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 -- 172.21.15.31:0/1391850807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52df86b0 msgr2=0x55ee52fb0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52df86b0 0x55ee52fb0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 -- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 msgr2=0x55ee52fb1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 -- 172.21.15.31:0/1391850807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c02b920 con 0x55ee52fa2100 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52df86b0 0x55ee52fb0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec2d59c700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fada80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fec20009a20 tx=0x7fec2000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c254d00092f7b978 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.694+0000 7fec07fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec07fff700 1 -- 172.21.15.31:0/1391850807 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000e040 con 0x55ee52fa2100 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec07fff700 1 -- 172.21.15.31:0/1391850807 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fec200050c0 con 0x55ee52fa2100 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec20006000 con 0x55ee52fa2100 2023-07-29T16:14:58.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec07fff700 1 -- 172.21.15.31:0/1391850807 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec20005280 con 0x55ee52fa2100 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 10 client.15525 fetch_fsmap learned FSMap version 23 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ee52d04c00 con 0x55ee52fa2100 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec07fff700 1 -- 172.21.15.31:0/1391850807 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fec20045080 con 0x55ee52fa2100 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec07fff700 1 -- 172.21.15.31:0/1391850807 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fec2000b0d0 con 0x55ee52fa2100 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 10 client.15525 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 10 client.15525 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:14:58.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 20 client.15525 populate_metadata read hostname 'smithi031' 2023-07-29T16:14:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee53008940 0x55ee5300ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:14:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 4 client.15525 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:14:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.695+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ee52ee6ce0 con 0x55ee53008940 2023-07-29T16:14:58.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.696+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee53008940 0x55ee5300ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:14:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.696+0000 7fec2cd9b700 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee53008940 0x55ee5300ad20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3fc30c77409359 server_cookie=8b33bc41364c081e in_seq=0 out_seq=0 2023-07-29T16:14:58.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.696+0000 7fec07fff700 10 client.15525 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.700+0000 7fec07fff700 1 -- 172.21.15.31:0/1391850807 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ee52ee6ce0 con 0x55ee53008940 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.700+0000 7fec07fff700 10 client.15525 handle_command_reply: tid=0 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.700+0000 7fec4290db80 1 client.15525 shutdown 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 msgr2=0x55ee52fada80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fada80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fec20009a20 tx=0x7fec2000b8b0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 shutdown_connections 2023-07-29T16:14:58.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee53008940 0x55ee5300ad20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee52fa2100 0x55ee52fada80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee52df86b0 0x55ee52fb0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 --2- 172.21.15.31:0/1391850807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee52fa1200 0x55ee52fb1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec07fff700 0 client.15525 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 >> 172.21.15.31:0/1391850807 conn(0x55ee52f91fd0 msgr2=0x55ee52fa4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 shutdown_connections 2023-07-29T16:14:58.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 1 -- 172.21.15.31:0/1391850807 wait complete. 2023-07-29T16:14:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.701+0000 7fec4290db80 20 client.15525 trim_cache size 0 max 16384 2023-07-29T16:14:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:14:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:14:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:14:58.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:14:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fec180a0810 msgr2=0x7fec180a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fec180a0810 0x7fec180a2cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec28101390 tx=0x7fec1c009260 comp rx=0 tx=0).stop 2023-07-29T16:14:58.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 msgr2=0x7fec281967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 0x7fec281967d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e9a0 tx=0x7fec1400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 shutdown_connections 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fec180a0810 0x7fec180a2cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fec28101ec0 0x7fec28197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fec28100340 0x7fec28196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 --2- 172.21.15.31:0/2014386068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec28101530 0x7fec281967d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:14:58.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 >> 172.21.15.31:0/2014386068 conn(0x7fec280fb920 msgr2=0x7fec281049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:14:58.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 shutdown_connections 2023-07-29T16:14:58.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:14:58.702+0000 7fec2efff700 1 -- 172.21.15.31:0/2014386068 wait complete. 2023-07-29T16:15:00.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:00 smithi031 ceph-mon[95504]: pgmap v355: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 9 op/s 2023-07-29T16:15:00.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:00 smithi039 ceph-mon[128858]: pgmap v355: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 9 op/s 2023-07-29T16:15:00.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:00 smithi156 ceph-mon[129245]: pgmap v355: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 9 op/s 2023-07-29T16:15:02.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:02 smithi031 ceph-mon[95504]: pgmap v356: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:02 smithi039 ceph-mon[128858]: pgmap v356: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:02.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:02 smithi156 ceph-mon[129245]: pgmap v356: 97 pgs: 97 active+clean; 661 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:04.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:04 smithi031 ceph-mon[95504]: pgmap v357: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:04 smithi039 ceph-mon[128858]: pgmap v357: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:04 smithi156 ceph-mon[129245]: pgmap v357: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:06 smithi031 ceph-mon[95504]: pgmap v358: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:06 smithi039 ceph-mon[128858]: pgmap v358: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:06.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:06 smithi156 ceph-mon[129245]: pgmap v358: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:08.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:07 smithi156 ceph-mon[129245]: pgmap v359: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:08.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:07 smithi031 ceph-mon[95504]: pgmap v359: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:07 smithi039 ceph-mon[128858]: pgmap v359: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:09.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:15:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 -- 172.21.15.31:0/3421472950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 msgr2=0x7f46c8108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 --2- 172.21.15.31:0/3421472950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 0x7f46c8108760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46c000b0a0 tx=0x7f46c000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 -- 172.21.15.31:0/3421472950 shutdown_connections 2023-07-29T16:15:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 --2- 172.21.15.31:0/3421472950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 0x7f46c8108760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 --2- 172.21.15.31:0/3421472950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8100320 0x7f46c8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 --2- 172.21.15.31:0/3421472950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8101510 0x7f46c8101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 -- 172.21.15.31:0/3421472950 >> 172.21.15.31:0/3421472950 conn(0x7f46c80fb920 msgr2=0x7f46c80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 -- 172.21.15.31:0/3421472950 shutdown_connections 2023-07-29T16:15:09.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.680+0000 7f46cdbff700 1 -- 172.21.15.31:0/3421472950 wait complete. 2023-07-29T16:15:09.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 Processor -- start 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 -- start start 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 0x7f46c8198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8101510 0x7f46c81986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 0x7f46c819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c819fd00 con 0x7f46c8100320 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c819fe40 con 0x7f46c8101510 2023-07-29T16:15:09.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.681+0000 7f46cdbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c819ffd0 con 0x7f46c8101ea0 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 0x7f46c8198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 0x7f46c8198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48832/0 (socket says 172.21.15.31:48832) 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 -- 172.21.15.31:0/928273396 learned_addr learned my addr 172.21.15.31:0/928273396 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8101510 0x7f46c81986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c7fff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 0x7f46c819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 -- 172.21.15.31:0/928273396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 msgr2=0x7f46c819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 0x7f46c819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 -- 172.21.15.31:0/928273396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8101510 msgr2=0x7f46c81986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8101510 0x7f46c81986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 -- 172.21.15.31:0/928273396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0009d00 con 0x7f46c8100320 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8101510 0x7f46c81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.682+0000 7f46c77fe700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 0x7f46c8198180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f46b800e970 tx=0x7f46b800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.683+0000 7f46c4ff9700 1 -- 172.21.15.31:0/928273396 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8009790 con 0x7f46c8100320 2023-07-29T16:15:09.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.683+0000 7f46c4ff9700 1 -- 172.21.15.31:0/928273396 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46b800c7e0 con 0x7f46c8100320 2023-07-29T16:15:09.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.683+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c81a02b0 con 0x7f46c8100320 2023-07-29T16:15:09.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.683+0000 7f46c4ff9700 1 -- 172.21.15.31:0/928273396 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8003c70 con 0x7f46c8100320 2023-07-29T16:15:09.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.683+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c81a07c0 con 0x7f46c8100320 2023-07-29T16:15:09.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.686+0000 7f46c4ff9700 1 -- 172.21.15.31:0/928273396 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f46b800cc50 con 0x7f46c8100320 2023-07-29T16:15:09.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.687+0000 7f46e15e2b80 1 Processor -- start 2023-07-29T16:15:09.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.687+0000 7f46e15e2b80 1 -- start start 2023-07-29T16:15:09.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.687+0000 7f46c4ff9700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46b00a0790 0x7f46b00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.687+0000 7f46c4ff9700 1 -- 172.21.15.31:0/928273396 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f46b8014070 con 0x7f46c8100320 2023-07-29T16:15:09.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.687+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954727ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.687+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954728d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5599547292f0 0x559954729740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55995467fa10 con 0x559954728530 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55995467d3d0 con 0x5599547292f0 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954678cc0 con 0x559954727be0 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c7fff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46b00a0790 0x7f46b00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954728d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954727ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954727ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39614/0 (socket says 172.21.15.31:39614) 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 -- 172.21.15.31:0/565899300 learned_addr learned my addr 172.21.15.31:0/565899300 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 -- 172.21.15.31:0/565899300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5599547292f0 msgr2=0x559954729740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5599547292f0 0x559954729740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 -- 172.21.15.31:0/565899300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 msgr2=0x559954728d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954728d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c77fe700 1 -- 172.21.15.31:0/565899300 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599544a6de0 con 0x559954727be0 2023-07-29T16:15:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954728d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.688+0000 7f46c7fff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46b00a0790 0x7f46b00a2c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f46c000b780 tx=0x7f46c000bbf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.689+0000 7f46c77fe700 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954727ff0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f46b8095ee0 tx=0x7f46b8003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c65d284f04b73f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.689+0000 7f46adffb700 1 -- 172.21.15.31:0/565899300 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8099470 con 0x559954727be0 2023-07-29T16:15:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.689+0000 7f46adffb700 1 -- 172.21.15.31:0/565899300 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46b809d4a0 con 0x559954727be0 2023-07-29T16:15:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.689+0000 7f46adffb700 1 -- 172.21.15.31:0/565899300 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b80996d0 con 0x559954727be0 2023-07-29T16:15:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.689+0000 7f46e15e2b80 1 -- 172.21.15.31:0/565899300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 msgr2=0x559954727ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.689+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954727ff0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f46b8095ee0 tx=0x7f46b8003710 comp rx=0 tx=0).stop 2023-07-29T16:15:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 -- 172.21.15.31:0/565899300 shutdown_connections 2023-07-29T16:15:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5599547292f0 0x559954729740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954727ff0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/565899300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954728d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 -- 172.21.15.31:0/565899300 >> 172.21.15.31:0/565899300 conn(0x55995471d3e0 msgr2=0x559954722990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 -- 172.21.15.31:0/565899300 shutdown_connections 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 -- 172.21.15.31:0/565899300 wait complete. 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 -- start start 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.690+0000 7f46e15e2b80 1 Processor -- start 2023-07-29T16:15:09.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 0x559954731880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954731dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x559954738e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954615510 con 0x559954728530 2023-07-29T16:15:09.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599546151b0 con 0x559954727be0 2023-07-29T16:15:09.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599544fc920 con 0x5599547292f0 2023-07-29T16:15:09.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46e15e2b80 10 client.0 _ll_register_callbacks cb 0x55995461e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 0x559954731880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954731dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 0x559954731880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34998/0 (socket says 172.21.15.31:34998) 2023-07-29T16:15:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 -- 172.21.15.31:0/2899941616 learned_addr learned my addr 172.21.15.31:0/2899941616 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c7fff700 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x559954738e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 -- 172.21.15.31:0/2899941616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 msgr2=0x559954738e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x559954738e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 -- 172.21.15.31:0/2899941616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 msgr2=0x559954731dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954731dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.691+0000 7f46c77fe700 1 -- 172.21.15.31:0/2899941616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc009c90 con 0x559954727be0 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46c77fe700 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 0x559954731880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f46b80047d0 tx=0x7f46b80a18c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b8b140a21791f9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46a6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:09.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46a6ffd700 1 -- 172.21.15.31:0/2899941616 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8099470 con 0x559954727be0 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46a6ffd700 1 -- 172.21.15.31:0/2899941616 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f46b80996d0 con 0x559954727be0 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599546a3d10 con 0x559954727be0 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46a6ffd700 1 -- 172.21.15.31:0/2899941616 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46b8099890 con 0x559954727be0 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46e15e2b80 10 client.25514 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f46c81a07c0 con 0x559954727be0 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.692+0000 7f46a6ffd700 1 -- 172.21.15.31:0/2899941616 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5599546a3d10 con 0x559954727be0 2023-07-29T16:15:09.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46a6ffd700 1 -- 172.21.15.31:0/2899941616 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f46b80b0030 con 0x559954727be0 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46e15e2b80 10 client.25514 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46e15e2b80 10 client.25514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46e15e2b80 20 client.25514 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995473d220 0x55995473f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46e15e2b80 4 client.25514 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599546f6d70 con 0x55995473d220 2023-07-29T16:15:09.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995473d220 0x55995473f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995473d220 0x55995473f630 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1beb2f53eb340892 server_cookie=b1209fa7e523e171 in_seq=0 out_seq=0 2023-07-29T16:15:09.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.693+0000 7f46a6ffd700 10 client.25514 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:09.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.694+0000 7f46a6ffd700 1 -- 172.21.15.31:0/2899941616 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5599546f6d70 con 0x55995473d220 2023-07-29T16:15:09.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.694+0000 7f46a6ffd700 10 client.25514 handle_command_reply: tid=0 2023-07-29T16:15:09.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 client.25514 shutdown 2023-07-29T16:15:09.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 msgr2=0x559954731880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 0x559954731880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f46b80047d0 tx=0x7f46b80a18c0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 shutdown_connections 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995473d220 0x55995473f630 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x559954738e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954727be0 0x559954731880 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/2899941616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954728530 0x559954731dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46a6ffd700 0 client.25514 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 >> 172.21.15.31:0/2899941616 conn(0x55995471d3e0 msgr2=0x559954722060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 shutdown_connections 2023-07-29T16:15:09.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 1 -- 172.21.15.31:0/2899941616 wait complete. 2023-07-29T16:15:09.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.695+0000 7f46e15e2b80 20 client.25514 trim_cache size 0 max 16384 2023-07-29T16:15:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.712+0000 7f46e15e2b80 1 Processor -- start 2023-07-29T16:15:09.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 -- start start 2023-07-29T16:15:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954730f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x559954738dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599547292f0 0x5599547296c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954729c00 con 0x5599547292f0 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954723cc0 con 0x559954728530 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55995472f4f0 con 0x559954727be0 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46c6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x559954738dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46c6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x559954738dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35014/0 (socket says 172.21.15.31:35014) 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46c6ffd700 1 -- 172.21.15.31:0/176188238 learned_addr learned my addr 172.21.15.31:0/176188238 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46c77fe700 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954730f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.713+0000 7f46c7fff700 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599547292f0 0x5599547296c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46c6ffd700 1 -- 172.21.15.31:0/176188238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 msgr2=0x559954730f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954730f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46c6ffd700 1 -- 172.21.15.31:0/176188238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599547292f0 msgr2=0x5599547296c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599547292f0 0x5599547296c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46c6ffd700 1 -- 172.21.15.31:0/176188238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55995448fc60 con 0x559954728530 2023-07-29T16:15:09.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x559954738dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f46bc000c00 tx=0x7f46bc009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b622e0177ec3954f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46ae7fc700 1 -- 172.21.15.31:0/176188238 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc023030 con 0x559954728530 2023-07-29T16:15:09.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46ae7fc700 1 -- 172.21.15.31:0/176188238 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46bc018040 con 0x559954728530 2023-07-29T16:15:09.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.714+0000 7f46ae7fc700 1 -- 172.21.15.31:0/176188238 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc017040 con 0x559954728530 2023-07-29T16:15:09.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 -- 172.21.15.31:0/176188238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 msgr2=0x559954738dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x559954738dc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f46bc000c00 tx=0x7f46bc009260 comp rx=0 tx=0).stop 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 -- 172.21.15.31:0/176188238 shutdown_connections 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5599547292f0 0x5599547296c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559954727be0 0x559954730f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/176188238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x559954738dc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 -- 172.21.15.31:0/176188238 >> 172.21.15.31:0/176188238 conn(0x55995471d3e0 msgr2=0x5599547213b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 -- 172.21.15.31:0/176188238 shutdown_connections 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 -- 172.21.15.31:0/176188238 wait complete. 2023-07-29T16:15:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.715+0000 7f46e15e2b80 1 -- start start 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 Processor -- start 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954727be0 0x559954736d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x5599547372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x5599547382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954738800 con 0x559954727be0 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954738970 con 0x559954728530 2023-07-29T16:15:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559954738b30 con 0x5599547292f0 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46e15e2b80 10 client.0 _ll_register_callbacks cb 0x55995461e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.716+0000 7f46c6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x5599547372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x5599547372b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35026/0 (socket says 172.21.15.31:35026) 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 -- 172.21.15.31:0/3784919942 learned_addr learned my addr 172.21.15.31:0/3784919942 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c7fff700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x5599547382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c77fe700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954727be0 0x559954736d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 -- 172.21.15.31:0/3784919942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 msgr2=0x5599547382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x5599547382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 -- 172.21.15.31:0/3784919942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954727be0 msgr2=0x559954736d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954727be0 0x559954736d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 -- 172.21.15.31:0/3784919942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc01f8b0 con 0x559954728530 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c77fe700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954727be0 0x559954736d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46c6ffd700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x5599547372b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006010 tx=0x7f46bc012250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=289ce273853b3cd8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.717+0000 7f46a6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46a6ffd700 1 -- 172.21.15.31:0/3784919942 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc012c30 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46a6ffd700 1 -- 172.21.15.31:0/3784919942 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f46bc000e40 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46b809d4a0 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46a6ffd700 1 -- 172.21.15.31:0/3784919942 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46bc01f040 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 10 client.25520 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5599544cd250 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46a6ffd700 1 -- 172.21.15.31:0/3784919942 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46b809d4a0 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46a6ffd700 1 -- 172.21.15.31:0/3784919942 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f46bc0048b0 con 0x559954728530 2023-07-29T16:15:09.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 10 client.25520 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 10 client.25520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 20 client.25520 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995478fed0 0x5599547922e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 4 client.25520 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:15:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.718+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559954671d40 con 0x55995478fed0 2023-07-29T16:15:09.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46c77fe700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995478fed0 0x5599547922e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46c77fe700 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995478fed0 0x5599547922e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a32e7fabf75a49 server_cookie=22ff2a98b08f00c7 in_seq=0 out_seq=0 2023-07-29T16:15:09.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46a6ffd700 10 client.25520 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46a6ffd700 1 -- 172.21.15.31:0/3784919942 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559954671d40 con 0x55995478fed0 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46a6ffd700 10 client.25520 handle_command_reply: tid=0 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46e15e2b80 1 client.25520 shutdown 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 msgr2=0x5599547372b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.719+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x5599547372b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f46bc006010 tx=0x7f46bc012250 comp rx=0 tx=0).stop 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 shutdown_connections 2023-07-29T16:15:09.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55995478fed0 0x5599547922e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5599547292f0 0x5599547382c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559954727be0 0x559954736d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 --2- 172.21.15.31:0/3784919942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559954728530 0x5599547372b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46a6ffd700 0 client.25520 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 >> 172.21.15.31:0/3784919942 conn(0x55995471d3e0 msgr2=0x55995472f740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 shutdown_connections 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 1 -- 172.21.15.31:0/3784919942 wait complete. 2023-07-29T16:15:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46e15e2b80 20 client.25520 trim_cache size 0 max 16384 2023-07-29T16:15:09.727 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:15:09.727 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:15:09.727 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:15:09.727 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:15:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46b00a0790 msgr2=0x7f46b00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46cdbff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46b00a0790 0x7f46b00a2c40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f46c000b780 tx=0x7f46c000bbf0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 msgr2=0x7f46c8198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.720+0000 7f46cdbff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 0x7f46c8198180 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f46b800e970 tx=0x7f46b800ec80 comp rx=0 tx=0).stop 2023-07-29T16:15:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 shutdown_connections 2023-07-29T16:15:09.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f46b00a0790 0x7f46b00a2c40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f46c8101ea0 0x7f46c819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46c8100320 0x7f46c8198180 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 --2- 172.21.15.31:0/928273396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f46c8101510 0x7f46c81986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 >> 172.21.15.31:0/928273396 conn(0x7f46c80fb920 msgr2=0x7f46c8106db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 shutdown_connections 2023-07-29T16:15:09.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.721+0000 7f46cdbff700 1 -- 172.21.15.31:0/928273396 wait complete. 2023-07-29T16:15:09.736 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:15:09.737 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:15:09.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/4270744530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 msgr2=0x7f3f2410a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/4270744530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 0x7f3f2410a630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00b0a0 tx=0x7f3f1c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/4270744530 shutdown_connections 2023-07-29T16:15:09.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/4270744530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 0x7f3f2410a630 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/4270744530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f24074d30 0x7f3f24073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/4270744530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f24073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.943+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/4270744530 >> 172.21.15.31:0/4270744530 conn(0x7f3f240fb940 msgr2=0x7f3f240fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:09.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.944+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/4270744530 shutdown_connections 2023-07-29T16:15:09.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.944+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/4270744530 wait complete. 2023-07-29T16:15:09.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.945+0000 7f3f2a5ff700 1 Processor -- start 2023-07-29T16:15:09.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.945+0000 7f3f2a5ff700 1 -- start start 2023-07-29T16:15:09.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f2a5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f2419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f2a5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f24074d30 0x7f3f2419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f2a5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 0x7f3f241a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f2a5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f241a4210 con 0x7f3f24074d30 2023-07-29T16:15:09.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f2a5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f241a4350 con 0x7f3f241081b0 2023-07-29T16:15:09.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f2a5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f241a44e0 con 0x7f3f24073750 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f2419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f2419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49440/0 (socket says 172.21.15.31:49440) 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 -- 172.21.15.31:0/2773834209 learned_addr learned my addr 172.21.15.31:0/2773834209 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f28b9c700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 0x7f3f241a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f237fe700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f24074d30 0x7f3f2419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 -- 172.21.15.31:0/2773834209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 msgr2=0x7f3f241a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 0x7f3f241a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 -- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f24074d30 msgr2=0x7f3f2419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:09.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f24074d30 0x7f3f2419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:09.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.946+0000 7f3f23fff700 1 -- 172.21.15.31:0/2773834209 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1c009d00 con 0x7f3f24073750 2023-07-29T16:15:09.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.947+0000 7f3f23fff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f2419c690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400e9a0 tx=0x7f3f1400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.947+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14009790 con 0x7f3f24073750 2023-07-29T16:15:09.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.947+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f14010430 con 0x7f3f24073750 2023-07-29T16:15:09.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.947+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f241a47c0 con 0x7f3f24073750 2023-07-29T16:15:09.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.947+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14018680 con 0x7f3f24073750 2023-07-29T16:15:09.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.947+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f241a4cd0 con 0x7f3f24073750 2023-07-29T16:15:09.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.948+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f241024e0 con 0x7f3f24073750 2023-07-29T16:15:09.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.951+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3f140105a0 con 0x7f3f24073750 2023-07-29T16:15:09.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.952+0000 7f3f217fa700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3f0c0a07e0 0x7f3f0c0a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:09.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.952+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3f14014070 con 0x7f3f24073750 2023-07-29T16:15:09.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.952+0000 7f3f237fe700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3f0c0a07e0 0x7f3f0c0a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:09.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.953+0000 7f3f237fe700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3f0c0a07e0 0x7f3f0c0a2c90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009c60 tx=0x7f3f18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:09.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:09.954+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f3f14095850 con 0x7f3f24073750 2023-07-29T16:15:10.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.208+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f2402cc30 con 0x7f3f24073750 2023-07-29T16:15:10.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.209+0000 7f3f217fa700 1 -- 172.21.15.31:0/2773834209 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f3f14098e70 con 0x7f3f24073750 2023-07-29T16:15:10.208 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:15:10.208 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:15:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3f0c0a07e0 msgr2=0x7f3f0c0a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3f0c0a07e0 0x7f3f0c0a2c90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f18009c60 tx=0x7f3f18009260 comp rx=0 tx=0).stop 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 msgr2=0x7f3f2419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f2419c690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400e9a0 tx=0x7f3f1400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 shutdown_connections 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3f0c0a07e0 0x7f3f0c0a2c90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f241081b0 0x7f3f241a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f24073750 0x7f3f2419c690 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 --2- 172.21.15.31:0/2773834209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3f24074d30 0x7f3f2419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 >> 172.21.15.31:0/2773834209 conn(0x7f3f240fb940 msgr2=0x7f3f2410a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 shutdown_connections 2023-07-29T16:15:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.212+0000 7f3f2a5ff700 1 -- 172.21.15.31:0/2773834209 wait complete. 2023-07-29T16:15:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:15:10.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:15:10.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 -- 172.21.15.31:0/1069783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8100330 msgr2=0x7f62b8100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 --2- 172.21.15.31:0/1069783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8100330 0x7f62b8100740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f62b4009990 tx=0x7f62b4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 -- 172.21.15.31:0/1069783011 shutdown_connections 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 --2- 172.21.15.31:0/1069783011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8101eb0 0x7f62b8108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 --2- 172.21.15.31:0/1069783011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8100330 0x7f62b8100740 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 --2- 172.21.15.31:0/1069783011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62b8101520 0x7f62b8101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 -- 172.21.15.31:0/1069783011 >> 172.21.15.31:0/1069783011 conn(0x7f62b80fb970 msgr2=0x7f62b80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 -- 172.21.15.31:0/1069783011 shutdown_connections 2023-07-29T16:15:10.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.440+0000 7f62c07ff700 1 -- 172.21.15.31:0/1069783011 wait complete. 2023-07-29T16:15:10.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 Processor -- start 2023-07-29T16:15:10.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 -- start start 2023-07-29T16:15:10.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 0x7f62b81960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62b8101520 0x7f62b8196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 0x7f62b819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819dc40 con 0x7f62b8101eb0 2023-07-29T16:15:10.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819dd80 con 0x7f62b8100330 2023-07-29T16:15:10.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.441+0000 7f62c07ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819df10 con 0x7f62b8101520 2023-07-29T16:15:10.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 0x7f62b819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62bdd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62b8101520 0x7f62b8196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 0x7f62b81960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 0x7f62b819d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42846/0 (socket says 172.21.15.31:42846) 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62bed9c700 1 -- 172.21.15.31:0/720282690 learned_addr learned my addr 172.21.15.31:0/720282690 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 0x7f62b81960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40412/0 (socket says 172.21.15.31:40412) 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 -- 172.21.15.31:0/720282690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62b8101520 msgr2=0x7f62b8196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62b8101520 0x7f62b8196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 -- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 msgr2=0x7f62b819d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 0x7f62b819d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 -- 172.21.15.31:0/720282690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b40095f0 con 0x7f62b8100330 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62bed9c700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 0x7f62b819d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.442+0000 7f62be59b700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 0x7f62b81960c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f62b400b620 tx=0x7f62b4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.443+0000 7f62af7fe700 1 -- 172.21.15.31:0/720282690 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b401d070 con 0x7f62b8100330 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.443+0000 7f62af7fe700 1 -- 172.21.15.31:0/720282690 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62b4021410 con 0x7f62b8100330 2023-07-29T16:15:10.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.443+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b819e190 con 0x7f62b8100330 2023-07-29T16:15:10.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.443+0000 7f62af7fe700 1 -- 172.21.15.31:0/720282690 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b4017650 con 0x7f62b8100330 2023-07-29T16:15:10.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.443+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b819e5b0 con 0x7f62b8100330 2023-07-29T16:15:10.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.446+0000 7f62af7fe700 1 -- 172.21.15.31:0/720282690 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f62b4021580 con 0x7f62b8100330 2023-07-29T16:15:10.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.448+0000 7f62d416cb80 1 Processor -- start 2023-07-29T16:15:10.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.448+0000 7f62d416cb80 1 -- start start 2023-07-29T16:15:10.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 0x56268ff0dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff0ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62af7fe700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f62a40a0790 0x7f62a40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0f2f0 0x56268ff0f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62d416cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fe65a10 con 0x56268ff0f2f0 2023-07-29T16:15:10.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62d416cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fe633d0 con 0x56268ff0e530 2023-07-29T16:15:10.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62d416cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fe5ecc0 con 0x56268ff0dbe0 2023-07-29T16:15:10.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62af7fe700 1 -- 172.21.15.31:0/720282690 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f62b40ccf00 con 0x7f62b8100330 2023-07-29T16:15:10.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 0x56268ff0dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bdd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff0ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62be59b700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f62a40a0790 0x7f62a40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 0x56268ff0dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49466/0 (socket says 172.21.15.31:49466) 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bed9c700 1 -- 172.21.15.31:0/1023349546 learned_addr learned my addr 172.21.15.31:0/1023349546 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bed9c700 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0f2f0 0x56268ff0f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bdd9a700 1 -- 172.21.15.31:0/1023349546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 msgr2=0x56268ff0dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 0x56268ff0dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bdd9a700 1 -- 172.21.15.31:0/1023349546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0f2f0 msgr2=0x56268ff0f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0f2f0 0x56268ff0f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bdd9a700 1 -- 172.21.15.31:0/1023349546 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268fc8cde0 con 0x56268ff0e530 2023-07-29T16:15:10.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.449+0000 7f62bed9c700 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 0x56268ff0dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:10.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.450+0000 7f62be59b700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f62a40a0790 0x7f62a40a2c40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f62b409a330 tx=0x7f62b4009ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.450+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff0ed60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f62a8007c70 tx=0x7f62a8007f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3530ae2d49345b83 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.450+0000 7f62acff9700 1 -- 172.21.15.31:0/1023349546 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a8010ca0 con 0x56268ff0e530 2023-07-29T16:15:10.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.450+0000 7f62acff9700 1 -- 172.21.15.31:0/1023349546 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62a8004510 con 0x56268ff0e530 2023-07-29T16:15:10.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.450+0000 7f62acff9700 1 -- 172.21.15.31:0/1023349546 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a8005020 con 0x56268ff0e530 2023-07-29T16:15:10.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 -- 172.21.15.31:0/1023349546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 msgr2=0x56268ff0ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff0ed60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f62a8007c70 tx=0x7f62a8007f80 comp rx=0 tx=0).stop 2023-07-29T16:15:10.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 -- 172.21.15.31:0/1023349546 shutdown_connections 2023-07-29T16:15:10.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0f2f0 0x56268ff0f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0dbe0 0x56268ff0dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 --2- 172.21.15.31:0/1023349546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff0ed60 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 -- 172.21.15.31:0/1023349546 >> 172.21.15.31:0/1023349546 conn(0x56268ff033e0 msgr2=0x56268ff08990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 -- 172.21.15.31:0/1023349546 shutdown_connections 2023-07-29T16:15:10.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 -- 172.21.15.31:0/1023349546 wait complete. 2023-07-29T16:15:10.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.451+0000 7f62d416cb80 1 -- start start 2023-07-29T16:15:10.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 Processor -- start 2023-07-29T16:15:10.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff1bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0e530 0x56268ff1c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0f2f0 0x56268ff231d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fdfb510 con 0x56268ff0dbe0 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fdfb1b0 con 0x56268ff0f2f0 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268fce2920 con 0x56268ff0e530 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62d416cb80 10 client.0 _ll_register_callbacks cb 0x56268fe04620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff1bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff1bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42856/0 (socket says 172.21.15.31:42856) 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62bed9c700 1 -- 172.21.15.31:0/4106547804 learned_addr learned my addr 172.21.15.31:0/4106547804 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.452+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0e530 0x56268ff1c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0f2f0 0x56268ff231d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 -- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0e530 msgr2=0x56268ff1c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0e530 0x56268ff1c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 -- 172.21.15.31:0/4106547804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0f2f0 msgr2=0x56268ff231d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0f2f0 0x56268ff231d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 -- 172.21.15.31:0/4106547804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b0008a60 con 0x56268ff0dbe0 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62bed9c700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff1bc50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f62b0008e70 tx=0x7f62b0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f962f355b935e33 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:10.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62a1ffb700 1 -- 172.21.15.31:0/4106547804 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0022070 con 0x56268ff0dbe0 2023-07-29T16:15:10.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56268fe89d10 con 0x56268ff0dbe0 2023-07-29T16:15:10.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62a1ffb700 1 -- 172.21.15.31:0/4106547804 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f62b0004bb0 con 0x56268ff0dbe0 2023-07-29T16:15:10.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.453+0000 7f62a1ffb700 1 -- 172.21.15.31:0/4106547804 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62b0004d70 con 0x56268ff0dbe0 2023-07-29T16:15:10.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 10 client.15561 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:10.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f62a80078a0 con 0x56268ff0dbe0 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62a1ffb700 1 -- 172.21.15.31:0/4106547804 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62b002c020 con 0x56268ff0dbe0 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62a1ffb700 1 -- 172.21.15.31:0/4106547804 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f62b0025030 con 0x56268ff0dbe0 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 10 client.15561 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 10 client.15561 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 20 client.15561 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff279f0 0x56268ff29e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 4 client.15561 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56268fedcd70 con 0x56268ff279f0 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.454+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff279f0 0x56268ff29e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.455+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff279f0 0x56268ff29e00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9182c9adfd300db server_cookie=9971257740e363fb in_seq=0 out_seq=0 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.455+0000 7f62a1ffb700 10 client.15561 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62a1ffb700 1 -- 172.21.15.31:0/4106547804 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56268fedcd70 con 0x56268ff279f0 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62a1ffb700 10 client.15561 handle_command_reply: tid=0 2023-07-29T16:15:10.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 client.15561 shutdown 2023-07-29T16:15:10.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 msgr2=0x56268ff1bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff1bc50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f62b0008e70 tx=0x7f62b0004b40 comp rx=0 tx=0).stop 2023-07-29T16:15:10.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 shutdown_connections 2023-07-29T16:15:10.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff279f0 0x56268ff29e00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0f2f0 0x56268ff231d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff1bc50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 --2- 172.21.15.31:0/4106547804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0e530 0x56268ff1c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62a1ffb700 0 client.15561 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:10.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 >> 172.21.15.31:0/4106547804 conn(0x56268ff033e0 msgr2=0x56268ff08060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 shutdown_connections 2023-07-29T16:15:10.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 1 -- 172.21.15.31:0/4106547804 wait complete. 2023-07-29T16:15:10.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.456+0000 7f62d416cb80 20 client.15561 trim_cache size 0 max 16384 2023-07-29T16:15:10.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 Processor -- start 2023-07-29T16:15:10.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 -- start start 2023-07-29T16:15:10.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0dbe0 0x56268ff21180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 0x56268ff15740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff12250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ff0ec00 con 0x56268ff0e530 2023-07-29T16:15:10.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ff05490 con 0x56268ff0dbe0 2023-07-29T16:15:10.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62d416cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ff0a360 con 0x56268ff0f2f0 2023-07-29T16:15:10.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62bdd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 0x56268ff15740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62bdd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 0x56268ff15740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42872/0 (socket says 172.21.15.31:42872) 2023-07-29T16:15:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62bdd9a700 1 -- 172.21.15.31:0/3185711017 learned_addr learned my addr 172.21.15.31:0/3185711017 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.470+0000 7f62bed9c700 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff12250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bed9c700 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0dbe0 0x56268ff21180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bdd9a700 1 -- 172.21.15.31:0/3185711017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 msgr2=0x56268ff12250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff12250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bdd9a700 1 -- 172.21.15.31:0/3185711017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0dbe0 msgr2=0x56268ff21180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0dbe0 0x56268ff21180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bdd9a700 1 -- 172.21.15.31:0/3185711017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56268fc75c60 con 0x56268ff0e530 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bed9c700 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0dbe0 0x56268ff21180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 0x56268ff15740 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f62a80106b0 tx=0x7f62a8007630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9237575495e31a0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62ad7fa700 1 -- 172.21.15.31:0/3185711017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a801f030 con 0x56268ff0e530 2023-07-29T16:15:10.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62ad7fa700 1 -- 172.21.15.31:0/3185711017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62a8020040 con 0x56268ff0e530 2023-07-29T16:15:10.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.471+0000 7f62ad7fa700 1 -- 172.21.15.31:0/3185711017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a80278b0 con 0x56268ff0e530 2023-07-29T16:15:10.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 -- 172.21.15.31:0/3185711017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 msgr2=0x56268ff15740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 0x56268ff15740 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f62a80106b0 tx=0x7f62a8007630 comp rx=0 tx=0).stop 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 -- 172.21.15.31:0/3185711017 shutdown_connections 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff12250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0dbe0 0x56268ff21180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 --2- 172.21.15.31:0/3185711017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0e530 0x56268ff15740 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 -- 172.21.15.31:0/3185711017 >> 172.21.15.31:0/3185711017 conn(0x56268ff033e0 msgr2=0x56268ff09690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 -- 172.21.15.31:0/3185711017 shutdown_connections 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 -- 172.21.15.31:0/3185711017 wait complete. 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.472+0000 7f62d416cb80 1 -- start start 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 Processor -- start 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff20f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff1f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff1fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ff20030 con 0x56268ff0dbe0 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ff201a0 con 0x56268ff0e530 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268ff20360 con 0x56268ff0f2f0 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62d416cb80 10 client.0 _ll_register_callbacks cb 0x56268fe04620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff20f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62bed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff20f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42886/0 (socket says 172.21.15.31:42886) 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.473+0000 7f62bed9c700 1 -- 172.21.15.31:0/330145568 learned_addr learned my addr 172.21.15.31:0/330145568 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff1f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff1fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 -- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 msgr2=0x56268ff1fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff1fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 -- 172.21.15.31:0/330145568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 msgr2=0x56268ff1f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff1f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 -- 172.21.15.31:0/330145568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56268fc8cde0 con 0x56268ff0dbe0 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62bed9c700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff20f10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f62b0000f80 tx=0x7f62b000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ff87b306509eb8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62a1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62a1ffb700 1 -- 172.21.15.31:0/330145568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0022070 con 0x56268ff0dbe0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62a1ffb700 1 -- 172.21.15.31:0/330145568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f62b002d080 con 0x56268ff0dbe0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62a8004510 con 0x56268ff0dbe0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.474+0000 7f62a1ffb700 1 -- 172.21.15.31:0/330145568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62b001b8b0 con 0x56268ff0dbe0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 10 client.15567 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56268fcb3250 con 0x56268ff0dbe0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62a1ffb700 1 -- 172.21.15.31:0/330145568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62a8004510 con 0x56268ff0dbe0 2023-07-29T16:15:10.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62a1ffb700 1 -- 172.21.15.31:0/330145568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f62b0025070 con 0x56268ff0dbe0 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 10 client.15567 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 10 client.15567 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 20 client.15567 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff7acf0 0x56268ff7d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 4 client.15567 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56268fe57d40 con 0x56268ff7acf0 2023-07-29T16:15:10.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.475+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff7acf0 0x56268ff7d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62bdd9a700 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff7acf0 0x56268ff7d100 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=642dc4d260f409b4 server_cookie=f4f57450697c9e2c in_seq=0 out_seq=0 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62a1ffb700 10 client.15567 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62a1ffb700 1 -- 172.21.15.31:0/330145568 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56268fe57d40 con 0x56268ff7acf0 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62a1ffb700 10 client.15567 handle_command_reply: tid=0 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 client.15567 shutdown 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 msgr2=0x56268ff20f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff20f10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f62b0000f80 tx=0x7f62b000b040 comp rx=0 tx=0).stop 2023-07-29T16:15:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 shutdown_connections 2023-07-29T16:15:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56268ff0f2f0 0x56268ff1fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56268ff0dbe0 0x56268ff20f10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x56268ff7acf0 0x56268ff7d100 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 --2- 172.21.15.31:0/330145568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56268ff0e530 0x56268ff1f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 >> 172.21.15.31:0/330145568 conn(0x56268ff033e0 msgr2=0x56268ff15740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.476+0000 7f62a1ffb700 0 client.15567 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 shutdown_connections 2023-07-29T16:15:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62d416cb80 1 -- 172.21.15.31:0/330145568 wait complete. 2023-07-29T16:15:10.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62d416cb80 20 client.15567 trim_cache size 0 max 16384 2023-07-29T16:15:10.485 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:15:10.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f62a40a0790 msgr2=0x7f62a40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f62a40a0790 0x7f62a40a2c40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f62b409a330 tx=0x7f62b4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 msgr2=0x7f62b81960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 0x7f62b81960c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f62b400b620 tx=0x7f62b4005740 comp rx=0 tx=0).stop 2023-07-29T16:15:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 shutdown_connections 2023-07-29T16:15:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f62a40a0790 0x7f62a40a2c40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62b8101eb0 0x7f62b819d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b8100330 0x7f62b81960c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 --2- 172.21.15.31:0/720282690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f62b8101520 0x7f62b8196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.477+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 >> 172.21.15.31:0/720282690 conn(0x7f62b80fb970 msgr2=0x7f62b80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.478+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 shutdown_connections 2023-07-29T16:15:10.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.478+0000 7f62c07ff700 1 -- 172.21.15.31:0/720282690 wait complete. 2023-07-29T16:15:10.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:10 smithi031 ceph-mon[95504]: pgmap v360: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:10.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2773834209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:15:10.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:15:10.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2838588393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed881002e0 msgr2=0x7fed881006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2838588393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed881002e0 0x7fed881006f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fed78009990 tx=0x7fed78009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2838588393 shutdown_connections 2023-07-29T16:15:10.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2838588393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed88101f20 0x7fed881087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2838588393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed881002e0 0x7fed881006f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2838588393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed88101570 0x7fed881019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2838588393 >> 172.21.15.31:0/2838588393 conn(0x7fed880fb900 msgr2=0x7fed880fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2838588393 shutdown_connections 2023-07-29T16:15:10.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.704+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2838588393 wait complete. 2023-07-29T16:15:10.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.705+0000 7fed8e5ff700 1 Processor -- start 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 -- start start 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 0x7fed88196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed88101570 0x7fed88196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed88101f20 0x7fed8819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8819dba0 con 0x7fed88101f20 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8819dce0 con 0x7fed881002e0 2023-07-29T16:15:10.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed8e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8819de70 con 0x7fed88101570 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed87fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 0x7fed88196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed87fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 0x7fed88196020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40450/0 (socket says 172.21.15.31:40450) 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed87fff700 1 -- 172.21.15.31:0/2061988160 learned_addr learned my addr 172.21.15.31:0/2061988160 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.706+0000 7fed877fe700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed88101570 0x7fed88196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed88101f20 0x7fed8819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed87fff700 1 -- 172.21.15.31:0/2061988160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed88101570 msgr2=0x7fed88196560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed87fff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed88101570 0x7fed88196560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed87fff700 1 -- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed88101f20 msgr2=0x7fed8819d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed87fff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed88101f20 0x7fed8819d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed87fff700 1 -- 172.21.15.31:0/2061988160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c009520 con 0x7fed881002e0 2023-07-29T16:15:10.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed88101f20 0x7fed8819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:10.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed87fff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 0x7fed88196020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fed78004740 tx=0x7fed78009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed857fa700 1 -- 172.21.15.31:0/2061988160 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7801d070 con 0x7fed881002e0 2023-07-29T16:15:10.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.707+0000 7fed857fa700 1 -- 172.21.15.31:0/2061988160 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed78004830 con 0x7fed881002e0 2023-07-29T16:15:10.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.708+0000 7fed857fa700 1 -- 172.21.15.31:0/2061988160 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed780053c0 con 0x7fed881002e0 2023-07-29T16:15:10.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.708+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed780095f0 con 0x7fed881002e0 2023-07-29T16:15:10.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.708+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8819e2d0 con 0x7fed881002e0 2023-07-29T16:15:10.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.712+0000 7fed857fa700 1 -- 172.21.15.31:0/2061988160 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fed78005580 con 0x7fed881002e0 2023-07-29T16:15:10.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.713+0000 7feda1e7eb80 1 Processor -- start 2023-07-29T16:15:10.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.713+0000 7feda1e7eb80 1 -- start start 2023-07-29T16:15:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576babf1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babf2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 0x5576babf2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bab44a10 con 0x5576babf2700 2023-07-29T16:15:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bab423d0 con 0x5576babf1940 2023-07-29T16:15:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bab3dcc0 con 0x5576babf0ff0 2023-07-29T16:15:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 0x5576babf2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed8cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576babf1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 0x5576babf2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42920/0 (socket says 172.21.15.31:42920) 2023-07-29T16:15:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed877fe700 1 -- 172.21.15.31:0/3528920529 learned_addr learned my addr 172.21.15.31:0/3528920529 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed87fff700 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babf2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed857fa700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fed700a0810 0x7fed700a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed8cb9c700 1 -- 172.21.15.31:0/3528920529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 msgr2=0x5576babf2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babf2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed8cb9c700 1 -- 172.21.15.31:0/3528920529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 msgr2=0x5576babf2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 0x5576babf2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed8cb9c700 1 -- 172.21.15.31:0/3528920529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576ba96bde0 con 0x5576babf0ff0 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.714+0000 7fed877fe700 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 0x5576babf2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.715+0000 7fed857fa700 1 -- 172.21.15.31:0/2061988160 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fed780ccbe0 con 0x7fed881002e0 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.715+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fed700a0810 0x7fed700a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.715+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576babf1400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fed80003db0 tx=0x7fed80009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f30cf09de0e4820 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.715+0000 7fed6e7fc700 1 -- 172.21.15.31:0/3528920529 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80007c70 con 0x5576babf0ff0 2023-07-29T16:15:10.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.715+0000 7fed6e7fc700 1 -- 172.21.15.31:0/3528920529 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8000f040 con 0x5576babf0ff0 2023-07-29T16:15:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.715+0000 7fed6e7fc700 1 -- 172.21.15.31:0/3528920529 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800156b0 con 0x5576babf0ff0 2023-07-29T16:15:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 -- 172.21.15.31:0/3528920529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 msgr2=0x5576babf1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fed700a0810 0x7fed700a2cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed80020000 tx=0x7fed80023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576babf1400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fed80003db0 tx=0x7fed80009920 comp rx=0 tx=0).stop 2023-07-29T16:15:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 -- 172.21.15.31:0/3528920529 shutdown_connections 2023-07-29T16:15:10.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf2700 0x5576babf2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576babf1400 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/3528920529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babf2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 -- 172.21.15.31:0/3528920529 >> 172.21.15.31:0/3528920529 conn(0x5576babe23e0 msgr2=0x5576babe7990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 -- 172.21.15.31:0/3528920529 shutdown_connections 2023-07-29T16:15:10.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.716+0000 7feda1e7eb80 1 -- 172.21.15.31:0/3528920529 wait complete. 2023-07-29T16:15:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 -- start start 2023-07-29T16:15:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 Processor -- start 2023-07-29T16:15:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 0x5576babfac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babfb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac021d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576baada510 con 0x5576babf0ff0 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576baada1b0 con 0x5576babf1940 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.717+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576ba9c1920 con 0x5576babf2700 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7feda1e7eb80 10 client.0 _ll_register_callbacks cb 0x5576baae3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 0x5576babfac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 0x5576babfac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42926/0 (socket says 172.21.15.31:42926) 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 -- 172.21.15.31:0/2681022175 learned_addr learned my addr 172.21.15.31:0/2681022175 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed877fe700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babfb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac021d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 -- 172.21.15.31:0/2681022175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 msgr2=0x5576bac021d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac021d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 -- 172.21.15.31:0/2681022175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 msgr2=0x5576babfb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babfb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 -- 172.21.15.31:0/2681022175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c009b70 con 0x5576babf0ff0 2023-07-29T16:15:10.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed87fff700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 0x5576babfac50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fed78096810 tx=0x7fed780a2520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2bbdac2ab4177d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.718+0000 7fed677fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7fed677fe700 1 -- 172.21.15.31:0/2681022175 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed78099e50 con 0x5576babf0ff0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7fed677fe700 1 -- 172.21.15.31:0/2681022175 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fed780a26d0 con 0x5576babf0ff0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5576bab68d10 con 0x5576babf0ff0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7fed677fe700 1 -- 172.21.15.31:0/2681022175 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed78099c70 con 0x5576babf0ff0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 10 client.15579 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fed80003a60 con 0x5576babf0ff0 2023-07-29T16:15:10.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7fed677fe700 1 -- 172.21.15.31:0/2681022175 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5576bab68d10 con 0x5576babf0ff0 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7fed677fe700 1 -- 172.21.15.31:0/2681022175 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fed78099800 con 0x5576babf0ff0 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 10 client.15579 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 10 client.15579 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 20 client.15579 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac06600 0x5576bac08a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.719+0000 7feda1e7eb80 4 client.15579 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.720+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5576babbbd70 con 0x5576bac06600 2023-07-29T16:15:10.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.720+0000 7fed877fe700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac06600 0x5576bac08a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.720+0000 7fed877fe700 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac06600 0x5576bac08a10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72168e39054e04bf server_cookie=913bf11b27b6e352 in_seq=0 out_seq=0 2023-07-29T16:15:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.720+0000 7fed677fe700 10 client.15579 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7fed677fe700 1 -- 172.21.15.31:0/2681022175 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5576babbbd70 con 0x5576bac06600 2023-07-29T16:15:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7fed677fe700 10 client.15579 handle_command_reply: tid=0 2023-07-29T16:15:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 client.15579 shutdown 2023-07-29T16:15:10.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 msgr2=0x5576babfac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 0x5576babfac50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fed78096810 tx=0x7fed780a2520 comp rx=0 tx=0).stop 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 shutdown_connections 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac06600 0x5576bac08a10 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf0ff0 0x5576babfac50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7fed677fe700 0 client.15579 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:10.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/2681022175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf1940 0x5576babfb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.721+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 >> 172.21.15.31:0/2681022175 conn(0x5576babe23e0 msgr2=0x5576babe7060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.722+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 shutdown_connections 2023-07-29T16:15:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.722+0000 7feda1e7eb80 1 -- 172.21.15.31:0/2681022175 wait complete. 2023-07-29T16:15:10.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.722+0000 7feda1e7eb80 20 client.15579 trim_cache size 0 max 16384 2023-07-29T16:15:10.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.735+0000 7feda1e7eb80 1 Processor -- start 2023-07-29T16:15:10.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.735+0000 7feda1e7eb80 1 -- start start 2023-07-29T16:15:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576bac089e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac02190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf2700 0x5576babf2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed88075050 con 0x5576babf1940 2023-07-29T16:15:10.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576babf3010 con 0x5576babf2700 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576babfa7c0 con 0x5576babf0ff0 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac02190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac02190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42940/0 (socket says 172.21.15.31:42940) 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 -- 172.21.15.31:0/40424074 learned_addr learned my addr 172.21.15.31:0/40424074 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed87fff700 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576bac089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf2700 0x5576babf2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 -- 172.21.15.31:0/40424074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 msgr2=0x5576bac089e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576bac089e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 -- 172.21.15.31:0/40424074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf2700 msgr2=0x5576babf2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf2700 0x5576babf2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 -- 172.21.15.31:0/40424074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576ba954c60 con 0x5576babf1940 2023-07-29T16:15:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.736+0000 7fed877fe700 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac02190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fed7c005c20 tx=0x7fed7c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471dd4a65271b987 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7fed6effd700 1 -- 172.21.15.31:0/40424074 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c016070 con 0x5576babf1940 2023-07-29T16:15:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7fed6effd700 1 -- 172.21.15.31:0/40424074 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed7c017040 con 0x5576babf1940 2023-07-29T16:15:10.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7fed6effd700 1 -- 172.21.15.31:0/40424074 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c010770 con 0x5576babf1940 2023-07-29T16:15:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 -- 172.21.15.31:0/40424074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 msgr2=0x5576bac02190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac02190 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fed7c005c20 tx=0x7fed7c004750 comp rx=0 tx=0).stop 2023-07-29T16:15:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 -- 172.21.15.31:0/40424074 shutdown_connections 2023-07-29T16:15:10.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf2700 0x5576babf2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf0ff0 0x5576bac089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/40424074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac02190 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.737+0000 7feda1e7eb80 1 -- 172.21.15.31:0/40424074 >> 172.21.15.31:0/40424074 conn(0x5576babe23e0 msgr2=0x5576babf9920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.738+0000 7feda1e7eb80 1 -- 172.21.15.31:0/40424074 shutdown_connections 2023-07-29T16:15:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.738+0000 7feda1e7eb80 1 -- 172.21.15.31:0/40424074 wait complete. 2023-07-29T16:15:10.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.738+0000 7feda1e7eb80 1 -- start start 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.738+0000 7feda1e7eb80 1 Processor -- start 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf0ff0 0x5576babfff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac00470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac01480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bac019c0 con 0x5576babf1940 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bac01b30 con 0x5576babf0ff0 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576bac01cf0 con 0x5576babf2700 2023-07-29T16:15:10.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7feda1e7eb80 10 client.0 _ll_register_callbacks cb 0x5576baae3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7fed877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac00470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7fed877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac00470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42956/0 (socket says 172.21.15.31:42956) 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7fed877fe700 1 -- 172.21.15.31:0/1743612645 learned_addr learned my addr 172.21.15.31:0/1743612645 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7fed87fff700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf0ff0 0x5576babfff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.739+0000 7fed8cb9c700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac01480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed877fe700 1 -- 172.21.15.31:0/1743612645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 msgr2=0x5576bac01480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed877fe700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac01480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed877fe700 1 -- 172.21.15.31:0/1743612645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf0ff0 msgr2=0x5576babfff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed877fe700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf0ff0 0x5576babfff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed877fe700 1 -- 172.21.15.31:0/1743612645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c0040f0 con 0x5576babf1940 2023-07-29T16:15:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed877fe700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac00470 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0092a0 tx=0x7fed7c004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e788b1c7e411aa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed677fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed677fe700 1 -- 172.21.15.31:0/1743612645 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7c016030 con 0x5576babf1940 2023-07-29T16:15:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed8000f040 con 0x5576babf1940 2023-07-29T16:15:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.740+0000 7fed677fe700 1 -- 172.21.15.31:0/1743612645 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fed7c01dba0 con 0x5576babf1940 2023-07-29T16:15:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7fed677fe700 1 -- 172.21.15.31:0/1743612645 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed7c01dd60 con 0x5576babf1940 2023-07-29T16:15:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 10 client.15585 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:10.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5576ba992250 con 0x5576babf1940 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7fed677fe700 1 -- 172.21.15.31:0/1743612645 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8000f040 con 0x5576babf1940 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7fed677fe700 1 -- 172.21.15.31:0/1743612645 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fed7c026040 con 0x5576babf1940 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 10 client.15585 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 10 client.15585 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 20 client.15585 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac59020 0x5576bac5b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 4 client.15585 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:15:10.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5576bab36d40 con 0x5576bac59020 2023-07-29T16:15:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.741+0000 7fed87fff700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac59020 0x5576bac5b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7fed677fe700 10 client.15585 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:10.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7fed87fff700 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac59020 0x5576bac5b430 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c4b97245c33c9ea server_cookie=5c5abd80a1352f9f in_seq=0 out_seq=0 2023-07-29T16:15:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7fed677fe700 1 -- 172.21.15.31:0/1743612645 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5576bab36d40 con 0x5576bac59020 2023-07-29T16:15:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7fed677fe700 10 client.15585 handle_command_reply: tid=0 2023-07-29T16:15:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 client.15585 shutdown 2023-07-29T16:15:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 msgr2=0x5576bac00470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac00470 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fed7c0092a0 tx=0x7fed7c004dd0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 shutdown_connections 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5576bac59020 0x5576bac5b430 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5576babf2700 0x5576bac01480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7fed677fe700 0 client.15585 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576babf0ff0 0x5576babfff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 --2- 172.21.15.31:0/1743612645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5576babf1940 0x5576bac00470 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 >> 172.21.15.31:0/1743612645 conn(0x5576babe23e0 msgr2=0x5576babf8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.742+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 shutdown_connections 2023-07-29T16:15:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7feda1e7eb80 1 -- 172.21.15.31:0/1743612645 wait complete. 2023-07-29T16:15:10.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7feda1e7eb80 20 client.15585 trim_cache size 0 max 16384 2023-07-29T16:15:10.751 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:15:10.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fed700a0810 msgr2=0x7fed700a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fed700a0810 0x7fed700a2cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed80020000 tx=0x7fed80023000 comp rx=0 tx=0).stop 2023-07-29T16:15:10.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 msgr2=0x7fed88196020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 0x7fed88196020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fed78004740 tx=0x7fed78009580 comp rx=0 tx=0).stop 2023-07-29T16:15:10.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 shutdown_connections 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fed700a0810 0x7fed700a2cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed88101f20 0x7fed8819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fed881002e0 0x7fed88196020 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.743+0000 7fed8e5ff700 1 --2- 172.21.15.31:0/2061988160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fed88101570 0x7fed88196560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.744+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 >> 172.21.15.31:0/2061988160 conn(0x7fed880fb900 msgr2=0x7fed880fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.744+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 shutdown_connections 2023-07-29T16:15:10.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.744+0000 7fed8e5ff700 1 -- 172.21.15.31:0/2061988160 wait complete. 2023-07-29T16:15:10.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:15:10.773 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:10 smithi039 ceph-mon[128858]: pgmap v360: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:10.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:10 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2773834209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:15:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:10 smithi156 ceph-mon[129245]: pgmap v360: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:10 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2773834209' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:15:10.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 -- 172.21.15.31:0/2654501065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210101ea0 msgr2=0x7f7210108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 --2- 172.21.15.31:0/2654501065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210101ea0 0x7f7210108760 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f720800b0a0 tx=0x7f720800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 -- 172.21.15.31:0/2654501065 shutdown_connections 2023-07-29T16:15:10.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 --2- 172.21.15.31:0/2654501065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210101ea0 0x7f7210108760 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 --2- 172.21.15.31:0/2654501065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7210100320 0x7f7210100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 --2- 172.21.15.31:0/2654501065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7210101510 0x7f7210101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 -- 172.21.15.31:0/2654501065 >> 172.21.15.31:0/2654501065 conn(0x7f72100fb920 msgr2=0x7f72100fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 -- 172.21.15.31:0/2654501065 shutdown_connections 2023-07-29T16:15:10.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.968+0000 7f72155ff700 1 -- 172.21.15.31:0/2654501065 wait complete. 2023-07-29T16:15:10.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.969+0000 7f72155ff700 1 Processor -- start 2023-07-29T16:15:10.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.969+0000 7f72155ff700 1 -- start start 2023-07-29T16:15:10.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f72155ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 0x7f7210074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f72155ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7210101510 0x7f7210073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f72155ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7210101ea0 0x7f7210073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f72155ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210073e20 con 0x7f7210100320 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f72155ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210073f90 con 0x7f7210101510 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f72155ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a0220 con 0x7f7210101ea0 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f720effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 0x7f7210074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f720effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 0x7f7210074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42972/0 (socket says 172.21.15.31:42972) 2023-07-29T16:15:10.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f720effd700 1 -- 172.21.15.31:0/249542934 learned_addr learned my addr 172.21.15.31:0/249542934 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f7207fff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7210101510 0x7f7210073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f720f7fe700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7210101ea0 0x7f7210073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f720effd700 1 -- 172.21.15.31:0/249542934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7210101ea0 msgr2=0x7f7210073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.970+0000 7f720effd700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7210101ea0 0x7f7210073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720effd700 1 -- 172.21.15.31:0/249542934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7210101510 msgr2=0x7f7210073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720effd700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7210101510 0x7f7210073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720effd700 1 -- 172.21.15.31:0/249542934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7208009d00 con 0x7f7210100320 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720effd700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 0x7f7210074b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f720000e970 tx=0x7f720000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720cff9700 1 -- 172.21.15.31:0/249542934 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200009790 con 0x7f7210100320 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720cff9700 1 -- 172.21.15.31:0/249542934 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720000eec0 con 0x7f7210100320 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f720cff9700 1 -- 172.21.15.31:0/249542934 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200003cf0 con 0x7f7210100320 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72101a0500 con 0x7f7210100320 2023-07-29T16:15:10.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.971+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72101a09c0 con 0x7f7210100320 2023-07-29T16:15:10.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.974+0000 7f720cff9700 1 -- 172.21.15.31:0/249542934 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f720000ccd0 con 0x7f7210100320 2023-07-29T16:15:10.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.975+0000 7f7228e33b80 1 Processor -- start 2023-07-29T16:15:10.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 -- start start 2023-07-29T16:15:10.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f720cff9700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71fc0a0810 0x7f71fc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b50ebe0 0x55602b50eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b50fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f720cff9700 1 -- 172.21.15.31:0/249542934 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7200014070 con 0x7f7210100320 2023-07-29T16:15:10.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b5102f0 0x55602b510740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b466a10 con 0x55602b50f530 2023-07-29T16:15:10.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b4643d0 con 0x55602b5102f0 2023-07-29T16:15:10.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7228e33b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b45fcc0 con 0x55602b50ebe0 2023-07-29T16:15:10.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f720f7fe700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71fc0a0810 0x7f71fc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7207fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b50fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7207fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b50fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42982/0 (socket says 172.21.15.31:42982) 2023-07-29T16:15:10.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f7207fff700 1 -- 172.21.15.31:0/869722412 learned_addr learned my addr 172.21.15.31:0/869722412 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f720effd700 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b50ebe0 0x55602b50eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.976+0000 7f720f7fe700 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b5102f0 0x55602b510740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f7207fff700 1 -- 172.21.15.31:0/869722412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b50ebe0 msgr2=0x55602b50eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f7207fff700 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b50ebe0 0x55602b50eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f7207fff700 1 -- 172.21.15.31:0/869722412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b5102f0 msgr2=0x55602b510740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f7207fff700 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b5102f0 0x55602b510740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f7207fff700 1 -- 172.21.15.31:0/869722412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55602b28dde0 con 0x55602b50f530 2023-07-29T16:15:10.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f720f7fe700 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b5102f0 0x55602b510740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:10.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f7207fff700 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b50fd60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009d80 tx=0x7f71f8011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b013b68afc2a72d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f720f7fe700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71fc0a0810 0x7f71fc0a2cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7210075160 tx=0x7f7208009c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f72057fa700 1 -- 172.21.15.31:0/869722412 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8004230 con 0x55602b50f530 2023-07-29T16:15:10.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f72057fa700 1 -- 172.21.15.31:0/869722412 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71f8004390 con 0x55602b50f530 2023-07-29T16:15:10.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.977+0000 7f72057fa700 1 -- 172.21.15.31:0/869722412 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8019880 con 0x55602b50f530 2023-07-29T16:15:10.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 -- 172.21.15.31:0/869722412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 msgr2=0x55602b50fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b50fd60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009d80 tx=0x7f71f8011720 comp rx=0 tx=0).stop 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 -- 172.21.15.31:0/869722412 shutdown_connections 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b5102f0 0x55602b510740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b50ebe0 0x55602b50eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 --2- 172.21.15.31:0/869722412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b50fd60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 -- 172.21.15.31:0/869722412 >> 172.21.15.31:0/869722412 conn(0x55602b5043e0 msgr2=0x55602b509990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 -- 172.21.15.31:0/869722412 shutdown_connections 2023-07-29T16:15:10.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.978+0000 7f7228e33b80 1 -- 172.21.15.31:0/869722412 wait complete. 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 -- start start 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 Processor -- start 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50ebe0 0x55602b518780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b518cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b3fc510 con 0x55602b50f530 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b3fc1b0 con 0x55602b50ebe0 2023-07-29T16:15:10.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.979+0000 7f7228e33b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b2e3920 con 0x55602b5102f0 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f7228e33b80 10 client.0 _ll_register_callbacks cb 0x55602b405620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f7207fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b518cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f7207fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b518cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42984/0 (socket says 172.21.15.31:42984) 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f7207fff700 1 -- 172.21.15.31:0/3829925878 learned_addr learned my addr 172.21.15.31:0/3829925878 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50ebe0 0x55602b518780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 -- 172.21.15.31:0/3829925878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 msgr2=0x55602b51fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 -- 172.21.15.31:0/3829925878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 msgr2=0x55602b518cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b518cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 -- 172.21.15.31:0/3829925878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f80099e0 con 0x55602b50ebe0 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720f7fe700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f7207fff700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b518cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f720effd700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50ebe0 0x55602b518780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7200099170 tx=0x7f7200016330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a331256f542433 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:10.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.980+0000 7f71f67fc700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f71f67fc700 1 -- 172.21.15.31:0/3829925878 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000a18c0 con 0x55602b50ebe0 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f71f67fc700 1 -- 172.21.15.31:0/3829925878 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f72000996d0 con 0x55602b50ebe0 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55602b48ad10 con 0x55602b50ebe0 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f71f67fc700 1 -- 172.21.15.31:0/3829925878 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7200099890 con 0x55602b50ebe0 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f7228e33b80 10 client.25574 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55602b276c60 con 0x55602b50ebe0 2023-07-29T16:15:10.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f71f67fc700 1 -- 172.21.15.31:0/3829925878 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720007c020 con 0x55602b50ebe0 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f71f67fc700 1 -- 172.21.15.31:0/3829925878 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f72000a14c0 con 0x55602b50ebe0 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f7228e33b80 10 client.25574 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f7228e33b80 10 client.25574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.981+0000 7f7228e33b80 20 client.25574 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.982+0000 7f7228e33b80 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5244d0 0x55602b5268b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.982+0000 7f7228e33b80 4 client.25574 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.982+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55602b4ddd70 con 0x55602b5244d0 2023-07-29T16:15:10.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.982+0000 7f7207fff700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5244d0 0x55602b5268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.982+0000 7f7207fff700 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5244d0 0x55602b5268b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=803015e2a91b8a01 server_cookie=ed0e02e8318d545b in_seq=0 out_seq=0 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.982+0000 7f71f67fc700 10 client.25574 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.983+0000 7f71f67fc700 1 -- 172.21.15.31:0/3829925878 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55602b4ddd70 con 0x55602b5244d0 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.983+0000 7f71f67fc700 10 client.25574 handle_command_reply: tid=0 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.983+0000 7f7228e33b80 1 client.25574 shutdown 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.983+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50ebe0 msgr2=0x55602b518780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:10.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.983+0000 7f7228e33b80 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50ebe0 0x55602b518780 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7200099170 tx=0x7f7200016330 comp rx=0 tx=0).stop 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 shutdown_connections 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5244d0 0x55602b5268b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50ebe0 0x55602b518780 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 --2- 172.21.15.31:0/3829925878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50f530 0x55602b518cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 >> 172.21.15.31:0/3829925878 conn(0x55602b5043e0 msgr2=0x55602b509060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f71f67fc700 0 client.25574 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 shutdown_connections 2023-07-29T16:15:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 1 -- 172.21.15.31:0/3829925878 wait complete. 2023-07-29T16:15:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.984+0000 7f7228e33b80 20 client.25574 trim_cache size 0 max 16384 2023-07-29T16:15:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 Processor -- start 2023-07-29T16:15:10.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 -- start start 2023-07-29T16:15:10.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b5106c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b516650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b5064d0 con 0x55602b50ebe0 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b5197c0 con 0x55602b50f530 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7228e33b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b516430 con 0x55602b5102f0 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f720effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b5106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f720effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b5106c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42994/0 (socket says 172.21.15.31:42994) 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f720effd700 1 -- 172.21.15.31:0/1016258628 learned_addr learned my addr 172.21.15.31:0/1016258628 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:10.999+0000 7f7207fff700 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720f7fe700 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b516650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720effd700 1 -- 172.21.15.31:0/1016258628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 msgr2=0x55602b516650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720effd700 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b516650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720effd700 1 -- 172.21.15.31:0/1016258628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 msgr2=0x55602b51fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720effd700 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720effd700 1 -- 172.21.15.31:0/1016258628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55602b2b4250 con 0x55602b50ebe0 2023-07-29T16:15:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f7207fff700 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f720effd700 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b5106c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7200005950 tx=0x7f72000958e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fedbdb302cc2334 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f7205ffb700 1 -- 172.21.15.31:0/1016258628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72000163a0 con 0x55602b50ebe0 2023-07-29T16:15:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f7205ffb700 1 -- 172.21.15.31:0/1016258628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720007c030 con 0x55602b50ebe0 2023-07-29T16:15:11.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.000+0000 7f7205ffb700 1 -- 172.21.15.31:0/1016258628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200098960 con 0x55602b50ebe0 2023-07-29T16:15:11.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 -- 172.21.15.31:0/1016258628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 msgr2=0x55602b5106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b5106c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7200005950 tx=0x7f72000958e0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 -- 172.21.15.31:0/1016258628 shutdown_connections 2023-07-29T16:15:11.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b516650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b5106c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1016258628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 -- 172.21.15.31:0/1016258628 >> 172.21.15.31:0/1016258628 conn(0x55602b5043e0 msgr2=0x55602b5174e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 -- 172.21.15.31:0/1016258628 shutdown_connections 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 -- 172.21.15.31:0/1016258628 wait complete. 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.001+0000 7f7228e33b80 1 -- start start 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 Processor -- start 2023-07-29T16:15:11.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b51bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b51f660 con 0x55602b50ebe0 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b51f7d0 con 0x55602b50f530 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602b51f990 con 0x55602b5102f0 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7228e33b80 10 client.0 _ll_register_callbacks cb 0x55602b405620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f720effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b51bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f720effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b51bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43006/0 (socket says 172.21.15.31:43006) 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f720effd700 1 -- 172.21.15.31:0/1619600301 learned_addr learned my addr 172.21.15.31:0/1619600301 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f720f7fe700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.002+0000 7f7207fff700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f720effd700 1 -- 172.21.15.31:0/1619600301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 msgr2=0x55602b51f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f720effd700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f720effd700 1 -- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 msgr2=0x55602b51c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f720effd700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f720effd700 1 -- 172.21.15.31:0/1619600301 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55602b28dde0 con 0x55602b50ebe0 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f720effd700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b51bbc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7200095970 tx=0x7f72000c9890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fe0f2fb5360656 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f71f67fc700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f71f67fc700 1 -- 172.21.15.31:0/1619600301 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200016330 con 0x55602b50ebe0 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f71f67fc700 1 -- 172.21.15.31:0/1619600301 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f720007d080 con 0x55602b50ebe0 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71f8004390 con 0x55602b50ebe0 2023-07-29T16:15:11.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f71f67fc700 1 -- 172.21.15.31:0/1619600301 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7200098960 con 0x55602b50ebe0 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f7228e33b80 10 client.15603 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.003+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55602b276c60 con 0x55602b50ebe0 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f71f67fc700 1 -- 172.21.15.31:0/1619600301 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72000bb080 con 0x55602b50ebe0 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f71f67fc700 1 -- 172.21.15.31:0/1619600301 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f7200095260 con 0x55602b50ebe0 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7228e33b80 10 client.15603 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7228e33b80 10 client.15603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7228e33b80 20 client.15603 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:11.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5769c0 0x55602b578da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7228e33b80 4 client.15603 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55602b458d40 con 0x55602b5769c0 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.004+0000 7f7207fff700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5769c0 0x55602b578da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7207fff700 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5769c0 0x55602b578da0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4500ce855beb0ed2 server_cookie=35bd39a85d349ea1 in_seq=0 out_seq=0 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f71f67fc700 10 client.15603 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f71f67fc700 1 -- 172.21.15.31:0/1619600301 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55602b458d40 con 0x55602b5769c0 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f71f67fc700 10 client.15603 handle_command_reply: tid=0 2023-07-29T16:15:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 client.15603 shutdown 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 msgr2=0x55602b51bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b51bbc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7200095970 tx=0x7f72000c9890 comp rx=0 tx=0).stop 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 shutdown_connections 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55602b5769c0 0x55602b578da0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55602b5102f0 0x55602b51f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55602b50ebe0 0x55602b51bbc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 --2- 172.21.15.31:0/1619600301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602b50f530 0x55602b51c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 >> 172.21.15.31:0/1619600301 conn(0x55602b5043e0 msgr2=0x55602b516650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f71f67fc700 0 client.15603 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:11.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 shutdown_connections 2023-07-29T16:15:11.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 1 -- 172.21.15.31:0/1619600301 wait complete. 2023-07-29T16:15:11.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.005+0000 7f7228e33b80 20 client.15603 trim_cache size 0 max 16384 2023-07-29T16:15:11.012 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:15:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71fc0a0810 msgr2=0x7f71fc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71fc0a0810 0x7f71fc0a2cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7210075160 tx=0x7f7208009c60 comp rx=0 tx=0).stop 2023-07-29T16:15:11.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 msgr2=0x7f7210074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 0x7f7210074b00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f720000e970 tx=0x7f720000ec80 comp rx=0 tx=0).stop 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 shutdown_connections 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71fc0a0810 0x7f71fc0a2cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7210101ea0 0x7f7210073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7210100320 0x7f7210074b00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 --2- 172.21.15.31:0/249542934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7210101510 0x7f7210073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 >> 172.21.15.31:0/249542934 conn(0x7f72100fb920 msgr2=0x7f72101049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 shutdown_connections 2023-07-29T16:15:11.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:11.006+0000 7f72155ff700 1 -- 172.21.15.31:0/249542934 wait complete. 2023-07-29T16:15:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 -- 172.21.15.31:0/993243665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 msgr2=0x7f4db810a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 --2- 172.21.15.31:0/993243665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db810a660 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4db000b0a0 tx=0x7f4db000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 -- 172.21.15.31:0/993243665 shutdown_connections 2023-07-29T16:15:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 --2- 172.21.15.31:0/993243665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db810a660 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 --2- 172.21.15.31:0/993243665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4db8074d30 0x7f4db8073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 --2- 172.21.15.31:0/993243665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4db8073750 0x7f4db8073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 -- 172.21.15.31:0/993243665 >> 172.21.15.31:0/993243665 conn(0x7f4db80fb970 msgr2=0x7f4db80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 -- 172.21.15.31:0/993243665 shutdown_connections 2023-07-29T16:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.854+0000 7f4dbefff700 1 -- 172.21.15.31:0/993243665 wait complete. 2023-07-29T16:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.856+0000 7f4dbefff700 1 Processor -- start 2023-07-29T16:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.856+0000 7f4dbefff700 1 -- start start 2023-07-29T16:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.856+0000 7f4dbefff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4db8073750 0x7f4db8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.856+0000 7f4dbefff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4db8074d30 0x7f4db81989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbefff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db8199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbefff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8199eb0 con 0x7f4db81081e0 2023-07-29T16:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbefff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db819a020 con 0x7f4db8073750 2023-07-29T16:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbefff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db819a1e0 con 0x7f4db8074d30 2023-07-29T16:15:11.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db8199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db8199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43030/0 (socket says 172.21.15.31:43030) 2023-07-29T16:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2097809596 learned_addr learned my addr 172.21.15.31:0/2097809596 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4db8074d30 0x7f4db81989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4db8073750 0x7f4db8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2097809596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4db8074d30 msgr2=0x7f4db81989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4db8074d30 0x7f4db81989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2097809596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4db8073750 msgr2=0x7f4db8198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4db8073750 0x7f4db8198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2097809596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0009d00 con 0x7f4db81081e0 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.857+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db8199850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009fd0 tx=0x7f4db000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.858+0000 7f4db5ffb700 1 -- 172.21.15.31:0/2097809596 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db000e040 con 0x7f4db81081e0 2023-07-29T16:15:11.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.858+0000 7f4db5ffb700 1 -- 172.21.15.31:0/2097809596 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4db00094a0 con 0x7f4db81081e0 2023-07-29T16:15:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.858+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db819a460 con 0x7f4db81081e0 2023-07-29T16:15:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.858+0000 7f4db5ffb700 1 -- 172.21.15.31:0/2097809596 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db001b6c0 con 0x7f4db81081e0 2023-07-29T16:15:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.858+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db81a0c80 con 0x7f4db81081e0 2023-07-29T16:15:11.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.861+0000 7f4db5ffb700 1 -- 172.21.15.31:0/2097809596 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4db001d470 con 0x7f4db81081e0 2023-07-29T16:15:11.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.862+0000 7f4db5ffb700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4da80a0810 0x7f4da80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.862+0000 7f4db5ffb700 1 -- 172.21.15.31:0/2097809596 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4db00ccc70 con 0x7f4db81081e0 2023-07-29T16:15:11.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.862+0000 7f4dd2989b80 1 Processor -- start 2023-07-29T16:15:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4da80a0810 0x7f4da80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 -- start start 2023-07-29T16:15:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 0x559725f2a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4da80a0810 0x7f4da80a2cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009c60 tx=0x7f4dac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725f2adf0 0x559725f2b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f2c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725e829b0 con 0x559725d866b0 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725e80370 con 0x559725f2bcf0 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.863+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725e7bc60 con 0x559725f2adf0 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbcd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 0x559725f2a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbcd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 0x559725f2a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43042/0 (socket says 172.21.15.31:43042) 2023-07-29T16:15:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbcd9b700 1 -- 172.21.15.31:0/2356083026 learned_addr learned my addr 172.21.15.31:0/2356083026 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725f2adf0 0x559725f2b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f2c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2356083026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725f2adf0 msgr2=0x559725f2b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725f2adf0 0x559725f2b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2356083026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 msgr2=0x559725f2a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 0x559725f2a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2356083026 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559725ca9d80 con 0x559725f2bcf0 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 0x559725f2a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f2c160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4db00ca6b0 tx=0x7f4db0009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e75b59c89f5d4ce8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4da2ffd700 1 -- 172.21.15.31:0/2356083026 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db009e3d0 con 0x559725f2bcf0 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.864+0000 7f4da2ffd700 1 -- 172.21.15.31:0/2356083026 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4db00a2760 con 0x559725f2bcf0 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4da2ffd700 1 -- 172.21.15.31:0/2356083026 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db009a800 con 0x559725f2bcf0 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2356083026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 msgr2=0x559725f2c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f2c160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4db00ca6b0 tx=0x7f4db0009810 comp rx=0 tx=0).stop 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2356083026 shutdown_connections 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f2c160 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725d866b0 0x559725f2a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2356083026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725f2adf0 0x559725f2b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2356083026 >> 172.21.15.31:0/2356083026 conn(0x559725f1ffd0 msgr2=0x559725f25300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2356083026 shutdown_connections 2023-07-29T16:15:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.865+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2356083026 wait complete. 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 -- start start 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 Processor -- start 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725e184b0 con 0x559725f2bcf0 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725e18150 con 0x559725f2adf0 2023-07-29T16:15:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.866+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725cff8c0 con 0x559725d866b0 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dd2989b80 10 client.0 _ll_register_callbacks cb 0x559725e215c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbd59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbcd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbcd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49596/0 (socket says 172.21.15.31:49596) 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbcd9b700 1 -- 172.21.15.31:0/3103858874 learned_addr learned my addr 172.21.15.31:0/3103858874 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4db7fff700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbd59c700 1 -- 172.21.15.31:0/3103858874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 msgr2=0x559725f33c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbd59c700 1 -- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 msgr2=0x559725f341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4dbd59c700 1 -- 172.21.15.31:0/3103858874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac017070 con 0x559725f2bcf0 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.867+0000 7f4db7fff700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3b1f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x559725f2a610 tx=0x7f4db00092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df53f4478448ea62 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4d97fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4d97fff700 1 -- 172.21.15.31:0/3103858874 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db00984e0 con 0x559725f2bcf0 2023-07-29T16:15:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559725ea6cb0 con 0x559725f2bcf0 2023-07-29T16:15:11.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4d97fff700 1 -- 172.21.15.31:0/3103858874 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4db0098740 con 0x559725f2bcf0 2023-07-29T16:15:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4d97fff700 1 -- 172.21.15.31:0/3103858874 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4db009e3d0 con 0x559725f2bcf0 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4dd2989b80 10 client.15615 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f4db81a0c80 con 0x559725f2bcf0 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4d97fff700 1 -- 172.21.15.31:0/3103858874 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4db00c8080 con 0x559725f2bcf0 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.868+0000 7f4d97fff700 1 -- 172.21.15.31:0/3103858874 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4db0096190 con 0x559725f2bcf0 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dd2989b80 10 client.15615 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dd2989b80 10 client.15615 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dd2989b80 20 client.15615 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f3f5d0 0x559725f419e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dd2989b80 4 client.15615 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559725ef9d10 con 0x559725f3f5d0 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.869+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f3f5d0 0x559725f419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.870+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f3f5d0 0x559725f419e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a7216800692a01 server_cookie=20e94b05e4d42525 in_seq=0 out_seq=0 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.870+0000 7f4d97fff700 10 client.15615 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.870+0000 7f4d97fff700 1 -- 172.21.15.31:0/3103858874 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559725ef9d10 con 0x559725f3f5d0 2023-07-29T16:15:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.870+0000 7f4d97fff700 10 client.15615 handle_command_reply: tid=0 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.870+0000 7f4dd2989b80 1 client.15615 shutdown 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 msgr2=0x559725f3b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3b1f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x559725f2a610 tx=0x7f4db00092e0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 shutdown_connections 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f3f5d0 0x559725f419e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3b1f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/3103858874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4d97fff700 0 client.15615 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 >> 172.21.15.31:0/3103858874 conn(0x559725f1ffd0 msgr2=0x559725f24a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 shutdown_connections 2023-07-29T16:15:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 1 -- 172.21.15.31:0/3103858874 wait complete. 2023-07-29T16:15:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.871+0000 7f4dd2989b80 20 client.15615 trim_cache size 0 max 16384 2023-07-29T16:15:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 Processor -- start 2023-07-29T16:15:11.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 -- start start 2023-07-29T16:15:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 0x559725f3b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725f26600 con 0x559725f2adf0 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725f1f9d0 con 0x559725f2bcf0 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725f34df0 con 0x559725d866b0 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4db7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 0x559725f3b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4db7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 0x559725f3b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43062/0 (socket says 172.21.15.31:43062) 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4db7fff700 1 -- 172.21.15.31:0/2105886506 learned_addr learned my addr 172.21.15.31:0/2105886506 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.886+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4db7fff700 1 -- 172.21.15.31:0/2105886506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 msgr2=0x559725f33c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4db7fff700 1 -- 172.21.15.31:0/2105886506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 msgr2=0x559725f341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4db7fff700 1 -- 172.21.15.31:0/2105886506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559725c92c00 con 0x559725f2adf0 2023-07-29T16:15:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 0x559725f3b180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4da4012770 tx=0x7f4da4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee52c2433d6d7fd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4da37fe700 1 -- 172.21.15.31:0/2105886506 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4017050 con 0x559725f2adf0 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4da37fe700 1 -- 172.21.15.31:0/2105886506 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4da4018040 con 0x559725f2adf0 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.887+0000 7f4da37fe700 1 -- 172.21.15.31:0/2105886506 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4010690 con 0x559725f2adf0 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2105886506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 msgr2=0x559725f3b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 0x559725f3b180 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f4da4012770 tx=0x7f4da4009260 comp rx=0 tx=0).stop 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2105886506 shutdown_connections 2023-07-29T16:15:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2bcf0 0x559725f341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f33c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2105886506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2adf0 0x559725f3b180 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2105886506 >> 172.21.15.31:0/2105886506 conn(0x559725f1ffd0 msgr2=0x559725f2eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2105886506 shutdown_connections 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2105886506 wait complete. 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.888+0000 7f4dd2989b80 1 -- start start 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 Processor -- start 2023-07-29T16:15:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f37450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f37990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725f3aef0 con 0x559725f2bcf0 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725f3b060 con 0x559725f2adf0 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559725f3b220 con 0x559725d866b0 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dd2989b80 10 client.0 _ll_register_callbacks cb 0x559725e215c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dbd59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dbd59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43068/0 (socket says 172.21.15.31:43068) 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2874773753 learned_addr learned my addr 172.21.15.31:0/2874773753 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4dbcd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f37450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.889+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f37990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2874773753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 msgr2=0x559725f37450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f37450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 msgr2=0x559725f37990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f37990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dbd59c700 1 -- 172.21.15.31:0/2874773753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da4012450 con 0x559725f2bcf0 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4db7fff700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f37990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dbd59c700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3a9b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4db000bfd0 tx=0x7f4db0009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379ec721d3ccb22e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4d97fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4d97fff700 1 -- 172.21.15.31:0/2874773753 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db009e3d0 con 0x559725f2bcf0 2023-07-29T16:15:11.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4db00a2760 con 0x559725f2bcf0 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4d97fff700 1 -- 172.21.15.31:0/2874773753 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4db009e170 con 0x559725f2bcf0 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4d97fff700 1 -- 172.21.15.31:0/2874773753 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4db00984e0 con 0x559725f2bcf0 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dd2989b80 10 client.15621 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.890+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559725cd01f0 con 0x559725f2bcf0 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4d97fff700 1 -- 172.21.15.31:0/2874773753 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4db00a2760 con 0x559725f2bcf0 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4d97fff700 1 -- 172.21.15.31:0/2874773753 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4db00994c0 con 0x559725f2bcf0 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dd2989b80 10 client.15621 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:11.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dd2989b80 10 client.15621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:11.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dd2989b80 20 client.15621 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:11.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f91e60 0x559725f94240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:11.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dd2989b80 4 client.15621 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:15:11.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559725e74ce0 con 0x559725f91e60 2023-07-29T16:15:11.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.891+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f91e60 0x559725f94240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:11.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.892+0000 7f4d97fff700 10 client.15621 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:11.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.892+0000 7f4dbcd9b700 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f91e60 0x559725f94240 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de173ab195c381c1 server_cookie=e2e0f71f338006d1 in_seq=0 out_seq=0 2023-07-29T16:15:11.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4d97fff700 1 -- 172.21.15.31:0/2874773753 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559725e74ce0 con 0x559725f91e60 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4d97fff700 10 client.15621 handle_command_reply: tid=0 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 client.15621 shutdown 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 msgr2=0x559725f3a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3a9b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4db000bfd0 tx=0x7f4db0009c90 comp rx=0 tx=0).stop 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 shutdown_connections 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559725f91e60 0x559725f94240 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559725f2bcf0 0x559725f3a9b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559725d866b0 0x559725f37450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 --2- 172.21.15.31:0/2874773753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559725f2adf0 0x559725f37990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4d97fff700 0 client.15621 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 >> 172.21.15.31:0/2874773753 conn(0x559725f1ffd0 msgr2=0x559725f33c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 shutdown_connections 2023-07-29T16:15:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.896+0000 7f4dd2989b80 1 -- 172.21.15.31:0/2874773753 wait complete. 2023-07-29T16:15:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dd2989b80 20 client.15621 trim_cache size 0 max 16384 2023-07-29T16:15:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:15:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:15:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:15:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:15:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4da80a0810 msgr2=0x7f4da80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4da80a0810 0x7f4da80a2cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4dac009c60 tx=0x7f4dac009260 comp rx=0 tx=0).stop 2023-07-29T16:15:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 msgr2=0x7f4db8199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db8199850 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009fd0 tx=0x7f4db000bce0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 shutdown_connections 2023-07-29T16:15:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4da80a0810 0x7f4da80a2cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4db81081e0 0x7f4db8199850 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4db8073750 0x7f4db8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 --2- 172.21.15.31:0/2097809596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4db8074d30 0x7f4db81989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 >> 172.21.15.31:0/2097809596 conn(0x7f4db80fb970 msgr2=0x7f4db80fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 shutdown_connections 2023-07-29T16:15:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:11.897+0000 7f4dbefff700 1 -- 172.21.15.31:0/2097809596 wait complete. 2023-07-29T16:15:12.022 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:15:12.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:15:12.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 -- 172.21.15.31:0/2259550471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898100320 msgr2=0x7f1898100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 --2- 172.21.15.31:0/2259550471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898100320 0x7f1898100730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1888009990 tx=0x7f1888009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 -- 172.21.15.31:0/2259550471 shutdown_connections 2023-07-29T16:15:12.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 --2- 172.21.15.31:0/2259550471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1898101ea0 0x7f1898108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 --2- 172.21.15.31:0/2259550471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898100320 0x7f1898100730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 --2- 172.21.15.31:0/2259550471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1898101510 0x7f1898101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 -- 172.21.15.31:0/2259550471 >> 172.21.15.31:0/2259550471 conn(0x7f18980fb920 msgr2=0x7f18980fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 -- 172.21.15.31:0/2259550471 shutdown_connections 2023-07-29T16:15:12.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.235+0000 7f189cdff700 1 -- 172.21.15.31:0/2259550471 wait complete. 2023-07-29T16:15:12.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.236+0000 7f189cdff700 1 Processor -- start 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.236+0000 7f189cdff700 1 -- start start 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189cdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1898100320 0x7f18981982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189cdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 0x7f1898198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189cdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1898101ea0 0x7f189819f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189cdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189819fe40 con 0x7f1898101510 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189cdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189819ff80 con 0x7f1898101ea0 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189cdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18981a0110 con 0x7f1898100320 2023-07-29T16:15:12.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 0x7f1898198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 0x7f1898198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43082/0 (socket says 172.21.15.31:43082) 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 -- 172.21.15.31:0/3411557414 learned_addr learned my addr 172.21.15.31:0/3411557414 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f1896d9d700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1898101ea0 0x7f189819f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f189659c700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1898100320 0x7f18981982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 -- 172.21.15.31:0/3411557414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1898100320 msgr2=0x7f18981982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1898100320 0x7f18981982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 -- 172.21.15.31:0/3411557414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1898101ea0 msgr2=0x7f189819f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1898101ea0 0x7f189819f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.237+0000 7f188ffff700 1 -- 172.21.15.31:0/3411557414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18880095f0 con 0x7f1898101510 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.238+0000 7f188ffff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 0x7f1898198800 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f188000e9a0 tx=0x7f188000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.238+0000 7f188f7fe700 1 -- 172.21.15.31:0/3411557414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880009790 con 0x7f1898101510 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.238+0000 7f188f7fe700 1 -- 172.21.15.31:0/3411557414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1880010430 con 0x7f1898101510 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.238+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18981a03f0 con 0x7f1898101510 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.238+0000 7f188f7fe700 1 -- 172.21.15.31:0/3411557414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1880018680 con 0x7f1898101510 2023-07-29T16:15:12.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.238+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18981a0900 con 0x7f1898101510 2023-07-29T16:15:12.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.241+0000 7f188f7fe700 1 -- 172.21.15.31:0/3411557414 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f18800105a0 con 0x7f1898101510 2023-07-29T16:15:12.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.242+0000 7f18b069ab80 1 Processor -- start 2023-07-29T16:15:12.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 -- start start 2023-07-29T16:15:12.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f188f7fe700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f188f7fe700 1 -- 172.21.15.31:0/3411557414 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1880014070 con 0x7f1898101510 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f1896d9d700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d6e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d6ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d6f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102cc5a90 con 0x562102d6dc60 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102cc3450 con 0x562102d6f370 2023-07-29T16:15:12.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.243+0000 7f18b069ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102cbed40 con 0x562102d6e5b0 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d6e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d6e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43084/0 (socket says 172.21.15.31:43084) 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 -- 172.21.15.31:0/3526535616 learned_addr learned my addr 172.21.15.31:0/3526535616 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f188ffff700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d6ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f1896d9d700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d6f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 -- 172.21.15.31:0/3526535616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 msgr2=0x562102d6ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d6ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 -- 172.21.15.31:0/3526535616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 msgr2=0x562102d6f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d6f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 -- 172.21.15.31:0/3526535616 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562102aece60 con 0x562102d6dc60 2023-07-29T16:15:12.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f188ffff700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d6ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:12.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f1896d9d700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d6f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:12.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f1896d9d700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f189000f600 tx=0x7f189000f9c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.244+0000 7f189659c700 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d6e070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1898101370 tx=0x7f1888005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20564a938159fe3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.245+0000 7f188cff9700 1 -- 172.21.15.31:0/3526535616 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188801d070 con 0x562102d6dc60 2023-07-29T16:15:12.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.245+0000 7f188cff9700 1 -- 172.21.15.31:0/3526535616 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f188801e040 con 0x562102d6dc60 2023-07-29T16:15:12.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.245+0000 7f188cff9700 1 -- 172.21.15.31:0/3526535616 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888004920 con 0x562102d6dc60 2023-07-29T16:15:12.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.245+0000 7f18b069ab80 1 -- 172.21.15.31:0/3526535616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 msgr2=0x562102d6e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.245+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d6e070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1898101370 tx=0x7f1888005ea0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 -- 172.21.15.31:0/3526535616 shutdown_connections 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d6f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d6e070 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3526535616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d6ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 -- 172.21.15.31:0/3526535616 >> 172.21.15.31:0/3526535616 conn(0x562102d63460 msgr2=0x562102d68a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 -- 172.21.15.31:0/3526535616 shutdown_connections 2023-07-29T16:15:12.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 -- 172.21.15.31:0/3526535616 wait complete. 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 -- start start 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.246+0000 7f18b069ab80 1 Processor -- start 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d77810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d77d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d7ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102c5b590 con 0x562102d6dc60 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102c5b230 con 0x562102d6f370 2023-07-29T16:15:12.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102b429a0 con 0x562102d6e5b0 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f18b069ab80 10 client.0 _ll_register_callbacks cb 0x562102c646a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f189659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d77810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f189659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d77810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43088/0 (socket says 172.21.15.31:43088) 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f189659c700 1 -- 172.21.15.31:0/2520374257 learned_addr learned my addr 172.21.15.31:0/2520374257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f1896d9d700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d7ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.247+0000 7f188ffff700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d77d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f189659c700 1 -- 172.21.15.31:0/2520374257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 msgr2=0x562102d77d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f189659c700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d77d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f189659c700 1 -- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 msgr2=0x562102d7ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f189659c700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d7ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f189659c700 1 -- 172.21.15.31:0/2520374257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18981a0900 con 0x562102d6dc60 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f1896d9d700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d7ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f189659c700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d77810 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1888009ce0 tx=0x7f188800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210bd2a99a3846b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f187dffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:12.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f187dffb700 1 -- 172.21.15.31:0/2520374257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188801d030 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f187dffb700 1 -- 172.21.15.31:0/2520374257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18880058e0 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562102ce9d90 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f187dffb700 1 -- 172.21.15.31:0/2520374257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1888005bc0 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.248+0000 7f18b069ab80 10 client.15633 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f1898109090 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f187dffb700 1 -- 172.21.15.31:0/2520374257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562102ce9d90 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f187dffb700 1 -- 172.21.15.31:0/2520374257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f188800fbd0 con 0x562102d6dc60 2023-07-29T16:15:12.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 10 client.15633 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:12.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 10 client.15633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:12.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 20 client.15633 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:12.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102d830a0 0x562102d85480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 4 client.15633 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:12.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.249+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562102d3cdf0 con 0x562102d830a0 2023-07-29T16:15:12.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.250+0000 7f188ffff700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102d830a0 0x562102d85480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.250+0000 7f188ffff700 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102d830a0 0x562102d85480 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a90544f60dd28bac server_cookie=3d43cd4be7a7fe1c in_seq=0 out_seq=0 2023-07-29T16:15:12.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.250+0000 7f187dffb700 10 client.15633 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f187dffb700 1 -- 172.21.15.31:0/2520374257 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562102d3cdf0 con 0x562102d830a0 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f187dffb700 10 client.15633 handle_command_reply: tid=0 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 client.15633 shutdown 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 msgr2=0x562102d77810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d77810 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1888009ce0 tx=0x7f188800ba20 comp rx=0 tx=0).stop 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 shutdown_connections 2023-07-29T16:15:12.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102d830a0 0x562102d85480 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6f370 0x562102d7ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6dc60 0x562102d77810 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f187dffb700 0 client.15633 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 --2- 172.21.15.31:0/2520374257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d77d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 >> 172.21.15.31:0/2520374257 conn(0x562102d63460 msgr2=0x562102d680e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 shutdown_connections 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 1 -- 172.21.15.31:0/2520374257 wait complete. 2023-07-29T16:15:12.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.251+0000 7f18b069ab80 20 client.15633 trim_cache size 0 max 16384 2023-07-29T16:15:12.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 Processor -- start 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 -- start start 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 0x562102d775c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d7ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d77b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102d6ec40 con 0x562102d6f370 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102d607f0 con 0x562102d6dc60 2023-07-29T16:15:12.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f18b069ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102d754f0 con 0x562102d6e5b0 2023-07-29T16:15:12.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f189659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 0x562102d775c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f189659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 0x562102d775c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40628/0 (socket says 172.21.15.31:40628) 2023-07-29T16:15:12.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f189659c700 1 -- 172.21.15.31:0/3103733676 learned_addr learned my addr 172.21.15.31:0/3103733676 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.269+0000 7f188ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d7ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f1896d9d700 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d77b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f189659c700 1 -- 172.21.15.31:0/3103733676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 msgr2=0x562102d7ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f189659c700 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d7ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f189659c700 1 -- 172.21.15.31:0/3103733676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 msgr2=0x562102d77b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f189659c700 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d77b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f189659c700 1 -- 172.21.15.31:0/3103733676 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562102ad5ce0 con 0x562102d6dc60 2023-07-29T16:15:12.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f189659c700 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 0x562102d775c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f18880098e0 tx=0x7f1888005320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffdd214b1df91aea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f188d7fa700 1 -- 172.21.15.31:0/3103733676 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188801d040 con 0x562102d6dc60 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f188d7fa700 1 -- 172.21.15.31:0/3103733676 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f188800ba90 con 0x562102d6dc60 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.270+0000 7f188d7fa700 1 -- 172.21.15.31:0/3103733676 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888004ba0 con 0x562102d6dc60 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 -- 172.21.15.31:0/3103733676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 msgr2=0x562102d775c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 0x562102d775c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f18880098e0 tx=0x7f1888005320 comp rx=0 tx=0).stop 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 -- 172.21.15.31:0/3103733676 shutdown_connections 2023-07-29T16:15:12.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d77b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6dc60 0x562102d775c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 --2- 172.21.15.31:0/3103733676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6e5b0 0x562102d7ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 -- 172.21.15.31:0/3103733676 >> 172.21.15.31:0/3103733676 conn(0x562102d63460 msgr2=0x562102d72270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 -- 172.21.15.31:0/3103733676 shutdown_connections 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 -- 172.21.15.31:0/3103733676 wait complete. 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.271+0000 7f18b069ab80 1 -- start start 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 Processor -- start 2023-07-29T16:15:12.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6dc60 0x562102d7bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 0x562102d7c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d7d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102d7d870 con 0x562102d6f370 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102d7d9e0 con 0x562102d6e5b0 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562102d7dba0 con 0x562102d6dc60 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f18b069ab80 10 client.0 _ll_register_callbacks cb 0x562102c646a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f188ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 0x562102d7c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f1896d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d7d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f188ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 0x562102d7c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40636/0 (socket says 172.21.15.31:40636) 2023-07-29T16:15:12.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f188ffff700 1 -- 172.21.15.31:0/4139221131 learned_addr learned my addr 172.21.15.31:0/4139221131 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.272+0000 7f189659c700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6dc60 0x562102d7bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f188ffff700 1 -- 172.21.15.31:0/4139221131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6dc60 msgr2=0x562102d7bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f188ffff700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6dc60 0x562102d7bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f188ffff700 1 -- 172.21.15.31:0/4139221131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 msgr2=0x562102d7d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f188ffff700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d7d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f188ffff700 1 -- 172.21.15.31:0/4139221131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1898109090 con 0x562102d6e5b0 2023-07-29T16:15:12.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f189659c700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6dc60 0x562102d7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f188ffff700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 0x562102d7c320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1880095ef0 tx=0x7f188001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c299c2f6ef3d2c5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f187dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f187dffb700 1 -- 172.21.15.31:0/4139221131 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188009d4b0 con 0x562102d6e5b0 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f188801e040 con 0x562102d6e5b0 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f187dffb700 1 -- 172.21.15.31:0/4139221131 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18800a18d0 con 0x562102d6e5b0 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f187dffb700 1 -- 172.21.15.31:0/4139221131 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1880010880 con 0x562102d6e5b0 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f18b069ab80 10 client.25616 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:12.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.273+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562102b132d0 con 0x562102d6e5b0 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f187dffb700 1 -- 172.21.15.31:0/4139221131 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f188007c080 con 0x562102d6e5b0 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f187dffb700 1 -- 172.21.15.31:0/4139221131 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f18800c92c0 con 0x562102d6e5b0 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f18b069ab80 10 client.25616 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f18b069ab80 10 client.25616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f18b069ab80 20 client.25616 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f18b069ab80 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102dd6f30 0x562102dd9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f18b069ab80 4 client.25616 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:15:12.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562102cb7dc0 con 0x562102dd6f30 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.274+0000 7f189659c700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102dd6f30 0x562102dd9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.275+0000 7f187dffb700 10 client.25616 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.275+0000 7f189659c700 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102dd6f30 0x562102dd9340 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=270faa4eceae118e server_cookie=f4911ced73c3b227 in_seq=0 out_seq=0 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f187dffb700 1 -- 172.21.15.31:0/4139221131 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562102cb7dc0 con 0x562102dd6f30 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f187dffb700 10 client.25616 handle_command_reply: tid=0 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 client.25616 shutdown 2023-07-29T16:15:12.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 msgr2=0x562102d7c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 0x562102d7c320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1880095ef0 tx=0x7f188001a040 comp rx=0 tx=0).stop 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 shutdown_connections 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562102dd6f30 0x562102dd9340 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562102d6f370 0x562102d7d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562102d6dc60 0x562102d7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f187dffb700 0 client.25616 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 --2- 172.21.15.31:0/4139221131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562102d6e5b0 0x562102d7c320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.276+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 >> 172.21.15.31:0/4139221131 conn(0x562102d63460 msgr2=0x562102d775c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 shutdown_connections 2023-07-29T16:15:12.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f18b069ab80 1 -- 172.21.15.31:0/4139221131 wait complete. 2023-07-29T16:15:12.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f18b069ab80 20 client.25616 trim_cache size 0 max 16384 2023-07-29T16:15:12.282 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:15:12.282 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:15:12.282 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "a35e2cc4-2b35-40f3-be04-6f751555e8b7", 2023-07-29T16:15:12.283 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:15:12.283 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:15:12.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 msgr2=0x7f18840a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f189000f600 tx=0x7f189000f9c0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 msgr2=0x7f1898198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 0x7f1898198800 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f188000e9a0 tx=0x7f188000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 shutdown_connections 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1898101ea0 0x7f189819f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1898100320 0x7f18981982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 --2- 172.21.15.31:0/3411557414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1898101510 0x7f1898198800 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.277+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 >> 172.21.15.31:0/3411557414 conn(0x7f18980fb920 msgr2=0x7f18980fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.278+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 shutdown_connections 2023-07-29T16:15:12.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.278+0000 7f189cdff700 1 -- 172.21.15.31:0/3411557414 wait complete. 2023-07-29T16:15:12.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:15:12.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 -- 172.21.15.31:0/3590336645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec074d30 msgr2=0x7f00ec073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 --2- 172.21.15.31:0/3590336645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec074d30 0x7f00ec073180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009990 tx=0x7f00e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 -- 172.21.15.31:0/3590336645 shutdown_connections 2023-07-29T16:15:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 --2- 172.21.15.31:0/3590336645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ec1081c0 0x7f00ec10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 --2- 172.21.15.31:0/3590336645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec074d30 0x7f00ec073180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 --2- 172.21.15.31:0/3590336645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec073750 0x7f00ec073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 -- 172.21.15.31:0/3590336645 >> 172.21.15.31:0/3590336645 conn(0x7f00ec0fb970 msgr2=0x7f00ec0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 -- 172.21.15.31:0/3590336645 shutdown_connections 2023-07-29T16:15:12.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.503+0000 7f00f37ff700 1 -- 172.21.15.31:0/3590336645 wait complete. 2023-07-29T16:15:12.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.504+0000 7f00f37ff700 1 Processor -- start 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.504+0000 7f00f37ff700 1 -- start start 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f37ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ec073750 0x7f00ec198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f37ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 0x7f00ec1987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f37ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec1081c0 0x7f00ec19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f37ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec19fdf0 con 0x7f00ec074d30 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f37ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec19ff30 con 0x7f00ec1081c0 2023-07-29T16:15:12.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f37ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ec1a00c0 con 0x7f00ec073750 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f0d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 0x7f00ec1987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f0d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 0x7f00ec1987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43116/0 (socket says 172.21.15.31:43116) 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f0d9a700 1 -- 172.21.15.31:0/2458057852 learned_addr learned my addr 172.21.15.31:0/2458057852 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec1081c0 0x7f00ec19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.505+0000 7f00f159b700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ec073750 0x7f00ec198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f0d9a700 1 -- 172.21.15.31:0/2458057852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ec073750 msgr2=0x7f00ec198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ec073750 0x7f00ec198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f0d9a700 1 -- 172.21.15.31:0/2458057852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec1081c0 msgr2=0x7f00ec19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec1081c0 0x7f00ec19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f0d9a700 1 -- 172.21.15.31:0/2458057852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00dc009520 con 0x7f00ec074d30 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec1081c0 0x7f00ec19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 0x7f00ec1987b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00eb30 tx=0x7f00dc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00e27fc700 1 -- 172.21.15.31:0/2458057852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc009880 con 0x7f00ec074d30 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e80095f0 con 0x7f00ec074d30 2023-07-29T16:15:12.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00e27fc700 1 -- 172.21.15.31:0/2458057852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00dc0044e0 con 0x7f00ec074d30 2023-07-29T16:15:12.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00e27fc700 1 -- 172.21.15.31:0/2458057852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00dc003d80 con 0x7f00ec074d30 2023-07-29T16:15:12.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.506+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec1a0770 con 0x7f00ec074d30 2023-07-29T16:15:12.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.510+0000 7f00e27fc700 1 -- 172.21.15.31:0/2458057852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f00dc003f20 con 0x7f00ec074d30 2023-07-29T16:15:12.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.511+0000 7f0107104b80 1 Processor -- start 2023-07-29T16:15:12.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.511+0000 7f0107104b80 1 -- start start 2023-07-29T16:15:12.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00e27fc700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00d80a0790 0x7f00d80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f0107104b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 0x55a862d9cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f0107104b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9d530 0x55a862d9dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f0107104b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862d9e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f0107104b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862cf4a10 con 0x55a862d9d530 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f0107104b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862cf23d0 con 0x55a862d9cbe0 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f0107104b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862cedcc0 con 0x55a862d9e2f0 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00e27fc700 1 -- 172.21.15.31:0/2458057852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f00dc014070 con 0x7f00ec074d30 2023-07-29T16:15:12.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f0d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9d530 0x55a862d9dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 0x55a862d9cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00d80a0790 0x7f00d80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f0d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9d530 0x55a862d9dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43118/0 (socket says 172.21.15.31:43118) 2023-07-29T16:15:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 0x55a862d9cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40658/0 (socket says 172.21.15.31:40658) 2023-07-29T16:15:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f0d9a700 1 -- 172.21.15.31:0/2915905996 learned_addr learned my addr 172.21.15.31:0/2915905996 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862d9e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 -- 172.21.15.31:0/2915905996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 msgr2=0x55a862d9e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862d9e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 -- 172.21.15.31:0/2915905996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9d530 msgr2=0x55a862d9dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9d530 0x55a862d9dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.512+0000 7f00f159b700 1 -- 172.21.15.31:0/2915905996 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a862b1bde0 con 0x55a862d9cbe0 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.513+0000 7f00f159b700 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 0x55a862d9cff0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f00e8000c00 tx=0x7f00e8005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d57a74955f4eea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.513+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00d80a0790 0x7f00d80a2c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009ec0 tx=0x7f00e4009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.513+0000 7f00d77fe700 1 -- 172.21.15.31:0/2915905996 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e801d070 con 0x55a862d9cbe0 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.513+0000 7f00d77fe700 1 -- 172.21.15.31:0/2915905996 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00e801e040 con 0x55a862d9cbe0 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.513+0000 7f00d77fe700 1 -- 172.21.15.31:0/2915905996 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e80048e0 con 0x55a862d9cbe0 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 -- 172.21.15.31:0/2915905996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 msgr2=0x55a862d9cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 0x55a862d9cff0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f00e8000c00 tx=0x7f00e8005ea0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 -- 172.21.15.31:0/2915905996 shutdown_connections 2023-07-29T16:15:12.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862d9e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9cbe0 0x55a862d9cff0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 --2- 172.21.15.31:0/2915905996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9d530 0x55a862d9dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 -- 172.21.15.31:0/2915905996 >> 172.21.15.31:0/2915905996 conn(0x55a862d923e0 msgr2=0x55a862d97990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 -- 172.21.15.31:0/2915905996 shutdown_connections 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 -- 172.21.15.31:0/2915905996 wait complete. 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.514+0000 7f0107104b80 1 -- start start 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 Processor -- start 2023-07-29T16:15:12.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862c8a510 con 0x55a862d9cbe0 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862c8a1b0 con 0x55a862d9d530 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862b71920 con 0x55a862d9e2f0 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f0107104b80 10 client.0 _ll_register_callbacks cb 0x55a862c93620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f00f159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f00f159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43134/0 (socket says 172.21.15.31:43134) 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.515+0000 7f00f159b700 1 -- 172.21.15.31:0/663082863 learned_addr learned my addr 172.21.15.31:0/663082863 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f159b700 1 -- 172.21.15.31:0/663082863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 msgr2=0x55a862dadd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f159b700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f159b700 1 -- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 msgr2=0x55a862da6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f159b700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f159b700 1 -- 172.21.15.31:0/663082863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ec10b260 con 0x55a862d9cbe0 2023-07-29T16:15:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00f159b700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da67b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f00e80098e0 tx=0x7f00e800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f486fd19c5338d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:12.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00d4ff9700 1 -- 172.21.15.31:0/663082863 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e801d040 con 0x55a862d9cbe0 2023-07-29T16:15:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f00d4ff9700 1 -- 172.21.15.31:0/663082863 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f00e800fc90 con 0x55a862d9cbe0 2023-07-29T16:15:12.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.516+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a862d18d10 con 0x55a862d9cbe0 2023-07-29T16:15:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f00d4ff9700 1 -- 172.21.15.31:0/663082863 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00e800fe50 con 0x55a862d9cbe0 2023-07-29T16:15:12.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f0107104b80 10 client.15651 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a862b04c60 con 0x55a862d9cbe0 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f00d4ff9700 1 -- 172.21.15.31:0/663082863 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a862d18d10 con 0x55a862d9cbe0 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f00d4ff9700 1 -- 172.21.15.31:0/663082863 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f00e80335d0 con 0x55a862d9cbe0 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f0107104b80 10 client.15651 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f0107104b80 10 client.15651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f0107104b80 20 client.15651 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:12.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.517+0000 7f0107104b80 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862db2040 0x55a862db4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.518+0000 7f0107104b80 4 client.15651 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.518+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a862d6bd70 con 0x55a862db2040 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.518+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862db2040 0x55a862db4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.518+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862db2040 0x55a862db4420 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a4df6435f8f0cd4 server_cookie=98f59566e2734d37 in_seq=0 out_seq=0 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.518+0000 7f00d4ff9700 10 client.15651 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f00d4ff9700 1 -- 172.21.15.31:0/663082863 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a862d6bd70 con 0x55a862db2040 2023-07-29T16:15:12.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f00d4ff9700 10 client.15651 handle_command_reply: tid=0 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 client.15651 shutdown 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 msgr2=0x55a862da67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da67b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f00e80098e0 tx=0x7f00e800f690 comp rx=0 tx=0).stop 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 shutdown_connections 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862db2040 0x55a862db4420 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f00d4ff9700 0 client.15651 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da67b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 --2- 172.21.15.31:0/663082863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.519+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 >> 172.21.15.31:0/663082863 conn(0x55a862d923e0 msgr2=0x55a862d97060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.520+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 shutdown_connections 2023-07-29T16:15:12.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.520+0000 7f0107104b80 1 -- 172.21.15.31:0/663082863 wait complete. 2023-07-29T16:15:12.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.520+0000 7f0107104b80 20 client.15651 trim_cache size 0 max 16384 2023-07-29T16:15:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 Processor -- start 2023-07-29T16:15:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 -- start start 2023-07-29T16:15:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862da6380 con 0x55a862d9cbe0 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862d94460 con 0x55a862d9d530 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f0107104b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862d8f6e0 con 0x55a862d9e2f0 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f00f159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f00f159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43148/0 (socket says 172.21.15.31:43148) 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f00f159b700 1 -- 172.21.15.31:0/3780957569 learned_addr learned my addr 172.21.15.31:0/3780957569 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.537+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00f159b700 1 -- 172.21.15.31:0/3780957569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 msgr2=0x55a862dadcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00f159b700 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00f159b700 1 -- 172.21.15.31:0/3780957569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 msgr2=0x55a862da4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00f159b700 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00f159b700 1 -- 172.21.15.31:0/3780957569 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a862b42250 con 0x55a862d9cbe0 2023-07-29T16:15:12.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00f159b700 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da4660 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f00e8005ee0 tx=0x7f00e800b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cb36e029d7621 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00d7fff700 1 -- 172.21.15.31:0/3780957569 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8005720 con 0x55a862d9cbe0 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00d7fff700 1 -- 172.21.15.31:0/3780957569 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00e800f700 con 0x55a862d9cbe0 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.538+0000 7f00d7fff700 1 -- 172.21.15.31:0/3780957569 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8017950 con 0x55a862d9cbe0 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 -- 172.21.15.31:0/3780957569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 msgr2=0x55a862da4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da4660 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f00e8005ee0 tx=0x7f00e800b320 comp rx=0 tx=0).stop 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 -- 172.21.15.31:0/3780957569 shutdown_connections 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9e2f0 0x55a862dadcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9cbe0 0x55a862da4660 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 --2- 172.21.15.31:0/3780957569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 -- 172.21.15.31:0/3780957569 >> 172.21.15.31:0/3780957569 conn(0x55a862d923e0 msgr2=0x55a862da5480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 -- 172.21.15.31:0/3780957569 shutdown_connections 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 -- 172.21.15.31:0/3780957569 wait complete. 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 -- start start 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.539+0000 7f0107104b80 1 Processor -- start 2023-07-29T16:15:12.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9cbe0 0x55a862da9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 0x55a862dacef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862dad430 con 0x55a862d9e2f0 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862dad5a0 con 0x55a862d9d530 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a862dad760 con 0x55a862d9cbe0 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f0107104b80 10 client.0 _ll_register_callbacks cb 0x55a862c93620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f00f1d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 0x55a862dacef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f00f1d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 0x55a862dacef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43160/0 (socket says 172.21.15.31:43160) 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f00f1d9c700 1 -- 172.21.15.31:0/182862534 learned_addr learned my addr 172.21.15.31:0/182862534 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f00f0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.540+0000 7f00f159b700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9cbe0 0x55a862da9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f1d9c700 1 -- 172.21.15.31:0/182862534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9cbe0 msgr2=0x55a862da9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9cbe0 0x55a862da9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f1d9c700 1 -- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 msgr2=0x55a862da9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:12.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f1d9c700 1 -- 172.21.15.31:0/182862534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a862b04c60 con 0x55a862d9e2f0 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f0d9a700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00f1d9c700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 0x55a862dacef0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f00e400ee30 tx=0x7f00e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5db894a08b9e8d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00d4ff9700 1 -- 172.21.15.31:0/182862534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4004070 con 0x55a862d9e2f0 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a862ce6d40 con 0x55a862d9e2f0 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00d4ff9700 1 -- 172.21.15.31:0/182862534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f00e40048b0 con 0x55a862d9e2f0 2023-07-29T16:15:12.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f00d4ff9700 1 -- 172.21.15.31:0/182862534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00e401aa90 con 0x55a862d9e2f0 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f0107104b80 10 client.15657 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.541+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a862b1a2b0 con 0x55a862d9e2f0 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f00d4ff9700 1 -- 172.21.15.31:0/182862534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a862ce6d40 con 0x55a862d9e2f0 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f00d4ff9700 1 -- 172.21.15.31:0/182862534 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f00e403a410 con 0x55a862d9e2f0 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f0107104b80 10 client.15657 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f0107104b80 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:12.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f0107104b80 20 client.15657 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f0107104b80 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862e04a40 0x55a862e06e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f0107104b80 4 client.15657 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:15:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a862ce6a60 con 0x55a862e04a40 2023-07-29T16:15:12.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.542+0000 7f00f159b700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862e04a40 0x55a862e06e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.543+0000 7f00f159b700 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862e04a40 0x55a862e06e50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2684380ccae743eb server_cookie=1b4462ff606db9fa in_seq=0 out_seq=0 2023-07-29T16:15:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.543+0000 7f00d4ff9700 10 client.15657 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.543+0000 7f00d4ff9700 1 -- 172.21.15.31:0/182862534 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a862ce6a60 con 0x55a862e04a40 2023-07-29T16:15:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.543+0000 7f00d4ff9700 10 client.15657 handle_command_reply: tid=0 2023-07-29T16:15:12.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.543+0000 7f0107104b80 1 client.15657 shutdown 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 msgr2=0x55a862dacef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 0x55a862dacef0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f00e400ee30 tx=0x7f00e400b040 comp rx=0 tx=0).stop 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 shutdown_connections 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a862e04a40 0x55a862e06e50 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a862d9e2f0 0x55a862dacef0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a862d9cbe0 0x55a862da9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 --2- 172.21.15.31:0/182862534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a862d9d530 0x55a862da9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 >> 172.21.15.31:0/182862534 conn(0x55a862d923e0 msgr2=0x55a862da4660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f00d4ff9700 0 client.15657 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 shutdown_connections 2023-07-29T16:15:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 1 -- 172.21.15.31:0/182862534 wait complete. 2023-07-29T16:15:12.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f0107104b80 20 client.15657 trim_cache size 0 max 16384 2023-07-29T16:15:12.550 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: "a35e2cc4-2b35-40f3-be04-6f751555e8b7": { 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "a35e2cc4-2b35-40f3-be04-6f751555e8b7", 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:15:12.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:15:12.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:15:12.552 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:15:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00d80a0790 msgr2=0x7f00d80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.544+0000 7f00f37ff700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00d80a0790 0x7f00d80a2c40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f00e4009ec0 tx=0x7f00e4009920 comp rx=0 tx=0).stop 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 msgr2=0x7f00ec1987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 0x7f00ec1987b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f00dc00eb30 tx=0x7f00dc00ee40 comp rx=0 tx=0).stop 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 shutdown_connections 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00d80a0790 0x7f00d80a2c40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ec1081c0 0x7f00ec19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ec073750 0x7f00ec198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 --2- 172.21.15.31:0/2458057852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ec074d30 0x7f00ec1987b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 >> 172.21.15.31:0/2458057852 conn(0x7f00ec0fb970 msgr2=0x7f00ec0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 shutdown_connections 2023-07-29T16:15:12.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:12.545+0000 7f00f37ff700 1 -- 172.21.15.31:0/2458057852 wait complete. 2023-07-29T16:15:12.561 INFO:tasks.cephfs.filesystem:scrub status for tag:a35e2cc4-2b35-40f3-be04-6f751555e8b7 - {'path': '/', 'tag': 'a35e2cc4-2b35-40f3-be04-6f751555e8b7', 'options': 'recursive,force'} 2023-07-29T16:15:12.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:12 smithi031 ceph-mon[95504]: pgmap v361: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:12 smithi039 ceph-mon[128858]: pgmap v361: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:12.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:12 smithi156 ceph-mon[129245]: pgmap v361: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:13.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:13 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:15:13.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:13.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:13 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:15:13.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:13.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:15:13.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:13 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:15:13.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:13.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:13 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:15:13.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:13.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:15:13.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:13 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:15:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:13 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:15:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:15:14.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:14 smithi031 ceph-mon[95504]: pgmap v362: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:15:14.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:14 smithi039 ceph-mon[128858]: pgmap v362: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:15:14.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:14 smithi156 ceph-mon[129245]: pgmap v362: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:15:16.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:16 smithi031 ceph-mon[95504]: pgmap v363: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:16 smithi039 ceph-mon[128858]: pgmap v363: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:16.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:16 smithi156 ceph-mon[129245]: pgmap v363: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:18.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:17 smithi156 ceph-mon[129245]: pgmap v364: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:17 smithi031 ceph-mon[95504]: pgmap v364: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:18.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:17 smithi039 ceph-mon[128858]: pgmap v364: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:20 smithi031 ceph-mon[95504]: pgmap v365: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:20.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:20 smithi039 ceph-mon[128858]: pgmap v365: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:20.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:20 smithi156 ceph-mon[129245]: pgmap v365: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:22 smithi031 ceph-mon[95504]: pgmap v366: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:22.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:22 smithi039 ceph-mon[128858]: pgmap v366: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:22.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:22 smithi156 ceph-mon[129245]: pgmap v366: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:24 smithi031 ceph-mon[95504]: pgmap v367: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:24.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:24 smithi039 ceph-mon[128858]: pgmap v367: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:24.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:24 smithi156 ceph-mon[129245]: pgmap v367: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 -- 172.21.15.31:0/3169805041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0103780 msgr2=0x7f97c0105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 --2- 172.21.15.31:0/3169805041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0103780 0x7f97c0105b60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009990 tx=0x7f97b4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 -- 172.21.15.31:0/3169805041 shutdown_connections 2023-07-29T16:15:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 --2- 172.21.15.31:0/3169805041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c01060a0 0x7f97c0108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 --2- 172.21.15.31:0/3169805041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0100e60 0x7f97c0103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 --2- 172.21.15.31:0/3169805041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0103780 0x7f97c0105b60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.059+0000 7f97c67ff700 1 -- 172.21.15.31:0/3169805041 >> 172.21.15.31:0/3169805041 conn(0x7f97c00fa840 msgr2=0x7f97c00fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.060+0000 7f97c67ff700 1 -- 172.21.15.31:0/3169805041 shutdown_connections 2023-07-29T16:15:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.060+0000 7f97c67ff700 1 -- 172.21.15.31:0/3169805041 wait complete. 2023-07-29T16:15:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 Processor -- start 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 -- start start 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c0100e60 0x7f97c01961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 0x7f97c0196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c01060a0 0x7f97c01975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0197c00 con 0x7f97c0103780 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0197d70 con 0x7f97c01060a0 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97c67ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c0197f30 con 0x7f97c0100e60 2023-07-29T16:15:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.061+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 0x7f97c0196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 0x7f97c0196720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51184/0 (socket says 172.21.15.31:51184) 2023-07-29T16:15:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 -- 172.21.15.31:0/2171123698 learned_addr learned my addr 172.21.15.31:0/2171123698 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97c4d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c01060a0 0x7f97c01975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bffff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c0100e60 0x7f97c01961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 -- 172.21.15.31:0/2171123698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c0100e60 msgr2=0x7f97c01961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c0100e60 0x7f97c01961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 -- 172.21.15.31:0/2171123698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c01060a0 msgr2=0x7f97c01975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c01060a0 0x7f97c01975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 -- 172.21.15.31:0/2171123698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b40095f0 con 0x7f97c0103780 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 0x7f97c0196720 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f97b4004740 tx=0x7f97b4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bd7fa700 1 -- 172.21.15.31:0/2171123698 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b401d070 con 0x7f97c0103780 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c01981b0 con 0x7f97c0103780 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bd7fa700 1 -- 172.21.15.31:0/2171123698 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b4004830 con 0x7f97c0103780 2023-07-29T16:15:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.062+0000 7f97bd7fa700 1 -- 172.21.15.31:0/2171123698 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b40053c0 con 0x7f97c0103780 2023-07-29T16:15:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.063+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c019e9d0 con 0x7f97c0103780 2023-07-29T16:15:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.066+0000 7f97bd7fa700 1 -- 172.21.15.31:0/2171123698 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f97b4005560 con 0x7f97c0103780 2023-07-29T16:15:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.067+0000 7f97bd7fa700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a80a0790 0x7f97a80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.067+0000 7f97da1f9b80 1 Processor -- start 2023-07-29T16:15:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.067+0000 7f97bd7fa700 1 -- 172.21.15.31:0/2171123698 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f97b40ccc90 con 0x7f97c0103780 2023-07-29T16:15:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.067+0000 7f97bffff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a80a0790 0x7f97a80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.067+0000 7f97da1f9b80 1 -- start start 2023-07-29T16:15:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3bca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3bd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba3109b0 con 0x55c7ba2146b0 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba30e370 con 0x55c7ba3be370 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba309c60 con 0x55c7ba3bcf80 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3bca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3bca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43226/0 (socket says 172.21.15.31:43226) 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97bf7fe700 1 -- 172.21.15.31:0/430459010 learned_addr learned my addr 172.21.15.31:0/430459010 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:25.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51190/0 (socket says 172.21.15.31:51190) 2023-07-29T16:15:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97bffff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a80a0790 0x7f97a80a2c40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f97c0068c00 tx=0x7f97b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.068+0000 7f97c4d9c700 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3bd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bffff700 1 -- 172.21.15.31:0/430459010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 msgr2=0x55c7ba3bd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bffff700 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3bd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bffff700 1 -- 172.21.15.31:0/430459010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 msgr2=0x55c7ba3bca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bffff700 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3bca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bffff700 1 -- 172.21.15.31:0/430459010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7ba137d80 con 0x55c7ba2146b0 2023-07-29T16:15:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3bca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97bffff700 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bde30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f97b0011c80 tx=0x7f97b0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff0747d6720a670 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97a67fc700 1 -- 172.21.15.31:0/430459010 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000eb00 con 0x55c7ba2146b0 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97a67fc700 1 -- 172.21.15.31:0/430459010 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b0021070 con 0x55c7ba2146b0 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.069+0000 7f97a67fc700 1 -- 172.21.15.31:0/430459010 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00253d0 con 0x55c7ba2146b0 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 -- 172.21.15.31:0/430459010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 msgr2=0x55c7ba3bde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bde30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f97b0011c80 tx=0x7f97b0014000 comp rx=0 tx=0).stop 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 -- 172.21.15.31:0/430459010 shutdown_connections 2023-07-29T16:15:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3bd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bde30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/430459010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3bca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 -- 172.21.15.31:0/430459010 >> 172.21.15.31:0/430459010 conn(0x55c7ba3adfd0 msgr2=0x55c7ba3b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 -- 172.21.15.31:0/430459010 shutdown_connections 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.070+0000 7f97da1f9b80 1 -- 172.21.15.31:0/430459010 wait complete. 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.071+0000 7f97da1f9b80 1 -- start start 2023-07-29T16:15:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.071+0000 7f97da1f9b80 1 Processor -- start 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba2146b0 0x55c7ba3c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3c6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba2a64b0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba2a6150 con 0x55c7ba3be370 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba18d8c0 con 0x55c7ba2146b0 2023-07-29T16:15:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97da1f9b80 10 client.0 _ll_register_callbacks cb 0x55c7ba2af5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3c6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97c4d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3c6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51196/0 (socket says 172.21.15.31:51196) 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 -- 172.21.15.31:0/3074325534 learned_addr learned my addr 172.21.15.31:0/3074325534 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bffff700 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba2146b0 0x55c7ba3c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 -- 172.21.15.31:0/3074325534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba2146b0 msgr2=0x55c7ba3c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba2146b0 0x55c7ba3c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 -- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 msgr2=0x55c7ba3cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.072+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f97bf7fe700 1 -- 172.21.15.31:0/3074325534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c019e9d0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3c6a10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f97b4094fd0 tx=0x7f97b40a0390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9483d99c667a609f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f979f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f979f7fe700 1 -- 172.21.15.31:0/3074325534 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b40a2780 con 0x55c7ba3bcf80 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f979f7fe700 1 -- 172.21.15.31:0/3074325534 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f97b40998b0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c7ba334cb0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f979f7fe700 1 -- 172.21.15.31:0/3074325534 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97b4099a70 con 0x55c7ba3bcf80 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f97da1f9b80 10 client.15669 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f97b0017070 con 0x55c7ba3bcf80 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.073+0000 7f979f7fe700 1 -- 172.21.15.31:0/3074325534 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c7ba334cb0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f979f7fe700 1 -- 172.21.15.31:0/3074325534 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f97b40cab90 con 0x55c7ba3bcf80 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97da1f9b80 10 client.15669 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:25.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97da1f9b80 10 client.15669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:25.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97da1f9b80 20 client.15669 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:25.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba3d1fa0 0x55c7ba3d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97da1f9b80 4 client.15669 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:25.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7ba387d10 con 0x55c7ba3d1fa0 2023-07-29T16:15:25.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.074+0000 7f97bffff700 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba3d1fa0 0x55c7ba3d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.075+0000 7f97bffff700 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba3d1fa0 0x55c7ba3d43b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a051a64c1fd054 server_cookie=e91655d0ce6240fe in_seq=0 out_seq=0 2023-07-29T16:15:25.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.075+0000 7f979f7fe700 10 client.15669 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:25.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f979f7fe700 1 -- 172.21.15.31:0/3074325534 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c7ba387d10 con 0x55c7ba3d1fa0 2023-07-29T16:15:25.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f979f7fe700 10 client.15669 handle_command_reply: tid=0 2023-07-29T16:15:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 client.15669 shutdown 2023-07-29T16:15:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 msgr2=0x55c7ba3c6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3c6a10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f97b4094fd0 tx=0x7f97b40a0390 comp rx=0 tx=0).stop 2023-07-29T16:15:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 shutdown_connections 2023-07-29T16:15:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba3d1fa0 0x55c7ba3d43b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba2146b0 0x55c7ba3c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3074325534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3c6a10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f979f7fe700 0 client.15669 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 >> 172.21.15.31:0/3074325534 conn(0x55c7ba3adfd0 msgr2=0x55c7ba3b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 shutdown_connections 2023-07-29T16:15:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3074325534 wait complete. 2023-07-29T16:15:25.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.076+0000 7f97da1f9b80 20 client.15669 trim_cache size 0 max 16384 2023-07-29T16:15:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 Processor -- start 2023-07-29T16:15:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 -- start start 2023-07-29T16:15:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3c1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba3c5cd0 con 0x55c7ba2146b0 2023-07-29T16:15:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba3c5e30 con 0x55c7ba3be370 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba3d1b20 con 0x55c7ba3bcf80 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51206/0 (socket says 172.21.15.31:51206) 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97bffff700 1 -- 172.21.15.31:0/3413076171 learned_addr learned my addr 172.21.15.31:0/3413076171 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97c4d9c700 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.092+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97bffff700 1 -- 172.21.15.31:0/3413076171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 msgr2=0x55c7ba3cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97bffff700 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97bffff700 1 -- 172.21.15.31:0/3413076171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 msgr2=0x55c7ba3c1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97bffff700 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3c1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97bffff700 1 -- 172.21.15.31:0/3413076171 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7ba120c00 con 0x55c7ba2146b0 2023-07-29T16:15:25.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97bffff700 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bc960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97b0011c80 tx=0x7f97b00092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a8c78fbeeb2a3b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97a6ffd700 1 -- 172.21.15.31:0/3413076171 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0005ce0 con 0x55c7ba2146b0 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97a6ffd700 1 -- 172.21.15.31:0/3413076171 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b0035170 con 0x55c7ba2146b0 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.093+0000 7f97a6ffd700 1 -- 172.21.15.31:0/3413076171 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0005040 con 0x55c7ba2146b0 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3413076171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 msgr2=0x55c7ba3bc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bc960 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97b0011c80 tx=0x7f97b00092d0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3413076171 shutdown_connections 2023-07-29T16:15:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3c1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba2146b0 0x55c7ba3bc960 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/3413076171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3413076171 >> 172.21.15.31:0/3413076171 conn(0x55c7ba3adfd0 msgr2=0x55c7ba3c4500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3413076171 shutdown_connections 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 -- 172.21.15.31:0/3413076171 wait complete. 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.094+0000 7f97da1f9b80 1 -- start start 2023-07-29T16:15:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 Processor -- start 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3c9c60 0x55c7ba3ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba3ca7b0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba423ae0 con 0x55c7ba3be370 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7ba423c20 con 0x55c7ba3c9c60 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97da1f9b80 10 client.0 _ll_register_callbacks cb 0x55c7ba3bdb50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51222/0 (socket says 172.21.15.31:51222) 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97bffff700 1 -- 172.21.15.31:0/1808662353 learned_addr learned my addr 172.21.15.31:0/1808662353 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.095+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97c4d9c700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3c9c60 0x55c7ba3ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97bffff700 1 -- 172.21.15.31:0/1808662353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3c9c60 msgr2=0x55c7ba3ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97bffff700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3c9c60 0x55c7ba3ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97bffff700 1 -- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 msgr2=0x55c7ba3cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97bffff700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97bffff700 1 -- 172.21.15.31:0/1808662353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b002a040 con 0x55c7ba3bcf80 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97bffff700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd1c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f97b001f010 tx=0x7f97b00117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b6c2cf9885ca89 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f979f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f979f7fe700 1 -- 172.21.15.31:0/1808662353 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000ec10 con 0x55c7ba3bcf80 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f979f7fe700 1 -- 172.21.15.31:0/1808662353 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f97b000ed70 con 0x55c7ba3bcf80 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97b0021070 con 0x55c7ba3bcf80 2023-07-29T16:15:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f979f7fe700 1 -- 172.21.15.31:0/1808662353 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97b001d630 con 0x55c7ba3bcf80 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97da1f9b80 10 client.15675 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.096+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c7ba15e1f0 con 0x55c7ba3bcf80 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f979f7fe700 1 -- 172.21.15.31:0/1808662353 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97b0021070 con 0x55c7ba3bcf80 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f979f7fe700 1 -- 172.21.15.31:0/1808662353 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f97b003d070 con 0x55c7ba3bcf80 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f97da1f9b80 10 client.15675 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f97da1f9b80 10 client.15675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f97da1f9b80 20 client.15675 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba425040 0x55c7ba427450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f97da1f9b80 4 client.15675 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:15:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.097+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c7ba302ce0 con 0x55c7ba425040 2023-07-29T16:15:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.098+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba425040 0x55c7ba427450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.098+0000 7f97bf7fe700 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba425040 0x55c7ba427450 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8899109dcfc1bdaf server_cookie=96d7f6f99dd176e4 in_seq=0 out_seq=0 2023-07-29T16:15:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.098+0000 7f979f7fe700 10 client.15675 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f979f7fe700 1 -- 172.21.15.31:0/1808662353 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c7ba302ce0 con 0x55c7ba425040 2023-07-29T16:15:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f979f7fe700 10 client.15675 handle_command_reply: tid=0 2023-07-29T16:15:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 client.15675 shutdown 2023-07-29T16:15:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 msgr2=0x55c7ba3cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd1c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f97b001f010 tx=0x7f97b00117d0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 shutdown_connections 2023-07-29T16:15:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7ba425040 0x55c7ba427450 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c7ba3c9c60 0x55c7ba3ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7ba3bcf80 0x55c7ba3cd1c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 --2- 172.21.15.31:0/1808662353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c7ba3be370 0x55c7ba3cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f979f7fe700 0 client.15675 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 >> 172.21.15.31:0/1808662353 conn(0x55c7ba3adfd0 msgr2=0x55c7ba3c1010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 shutdown_connections 2023-07-29T16:15:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.102+0000 7f97da1f9b80 1 -- 172.21.15.31:0/1808662353 wait complete. 2023-07-29T16:15:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97da1f9b80 20 client.15675 trim_cache size 0 max 16384 2023-07-29T16:15:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:15:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:15:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:15:25.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:15:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a80a0790 msgr2=0x7f97a80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a80a0790 0x7f97a80a2c40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f97c0068c00 tx=0x7f97b0009260 comp rx=0 tx=0).stop 2023-07-29T16:15:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 msgr2=0x7f97c0196720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 0x7f97c0196720 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f97b4004740 tx=0x7f97b4009580 comp rx=0 tx=0).stop 2023-07-29T16:15:25.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 shutdown_connections 2023-07-29T16:15:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97a80a0790 0x7f97a80a2c40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c01060a0 0x7f97c01975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97c0100e60 0x7f97c01961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 --2- 172.21.15.31:0/2171123698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f97c0103780 0x7f97c0196720 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 >> 172.21.15.31:0/2171123698 conn(0x7f97c00fa840 msgr2=0x7f97c0101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 shutdown_connections 2023-07-29T16:15:25.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:25.103+0000 7f97c67ff700 1 -- 172.21.15.31:0/2171123698 wait complete. 2023-07-29T16:15:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:26 smithi031 ceph-mon[95504]: pgmap v368: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:26 smithi039 ceph-mon[128858]: pgmap v368: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:26 smithi156 ceph-mon[129245]: pgmap v368: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:28.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:27 smithi156 ceph-mon[129245]: pgmap v369: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-07-29T16:15:28.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:27 smithi031 ceph-mon[95504]: pgmap v369: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-07-29T16:15:28.273 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:27 smithi039 ceph-mon[128858]: pgmap v369: 97 pgs: 97 active+clean; 662 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-07-29T16:15:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:30 smithi031 ceph-mon[95504]: pgmap v370: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:30 smithi039 ceph-mon[128858]: pgmap v370: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:30.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:30 smithi156 ceph-mon[129245]: pgmap v370: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:32.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:32 smithi031 ceph-mon[95504]: pgmap v371: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:32.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:32 smithi039 ceph-mon[128858]: pgmap v371: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:32.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:32 smithi156 ceph-mon[129245]: pgmap v371: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:34 smithi031 ceph-mon[95504]: pgmap v372: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:34 smithi039 ceph-mon[128858]: pgmap v372: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:34.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:34 smithi156 ceph-mon[129245]: pgmap v372: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:36.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:36 smithi031 ceph-mon[95504]: pgmap v373: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:36.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:36 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:15:36.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:36 smithi039 ceph-mon[128858]: pgmap v373: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:36.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:36 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:15:36.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:36 smithi156 ceph-mon[129245]: pgmap v373: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:36 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:15:37.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:37.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:37.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:15:37.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:15:37.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:37.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:37.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:37.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:15:37.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:15:37.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:38.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:38.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:38.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:15:38.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:15:38.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:15:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/2517495157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 msgr2=0x7fa4a4100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/2517495157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 0x7fa4a4100760 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009990 tx=0x7fa4a0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/2517495157 shutdown_connections 2023-07-29T16:15:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/2517495157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/2517495157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 0x7fa4a4100760 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/2517495157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4a4101540 0x7fa4a4101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/2517495157 >> 172.21.15.31:0/2517495157 conn(0x7fa4a40fb970 msgr2=0x7fa4a40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/2517495157 shutdown_connections 2023-07-29T16:15:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.256+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/2517495157 wait complete. 2023-07-29T16:15:38.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.257+0000 7fa4ab3ff700 1 Processor -- start 2023-07-29T16:15:38.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.257+0000 7fa4ab3ff700 1 -- start start 2023-07-29T16:15:38.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4ab3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 0x7fa4a4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4ab3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4a4101540 0x7fa4a4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4ab3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4ab3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41a0150 con 0x7fa4a4101ed0 2023-07-29T16:15:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4ab3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41a0290 con 0x7fa4a4100350 2023-07-29T16:15:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4ab3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a41a03d0 con 0x7fa4a4101540 2023-07-29T16:15:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4073bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54820/0 (socket says 172.21.15.31:54820) 2023-07-29T16:15:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 0x7fa4a4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 -- 172.21.15.31:0/3545380085 learned_addr learned my addr 172.21.15.31:0/3545380085 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a899a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4a4101540 0x7fa4a4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 -- 172.21.15.31:0/3545380085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4a4101540 msgr2=0x7fa4a4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4a4101540 0x7fa4a4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 -- 172.21.15.31:0/3545380085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 msgr2=0x7fa4a4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.258+0000 7fa4a999c700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 0x7fa4a4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa4a999c700 1 -- 172.21.15.31:0/3545380085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c009c40 con 0x7fa4a4101ed0 2023-07-29T16:15:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa4a999c700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4073bc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c370 tx=0x7fa49c00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa49a7fc700 1 -- 172.21.15.31:0/3545380085 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c017070 con 0x7fa4a4101ed0 2023-07-29T16:15:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a00095f0 con 0x7fa4a4101ed0 2023-07-29T16:15:38.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa49a7fc700 1 -- 172.21.15.31:0/3545380085 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa49c00f040 con 0x7fa4a4101ed0 2023-07-29T16:15:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa49a7fc700 1 -- 172.21.15.31:0/3545380085 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00e050 con 0x7fa4a4101ed0 2023-07-29T16:15:38.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.259+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41a09a0 con 0x7fa4a4101ed0 2023-07-29T16:15:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.262+0000 7fa49a7fc700 1 -- 172.21.15.31:0/3545380085 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa49c00cb60 con 0x7fa4a4101ed0 2023-07-29T16:15:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa4bed00b80 1 Processor -- start 2023-07-29T16:15:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa49a7fc700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4900a0790 0x7fa4900a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa4bed00b80 1 -- start start 2023-07-29T16:15:38.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa49a7fc700 1 -- 172.21.15.31:0/3545380085 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa49c0cb420 con 0x7fa4a4101ed0 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa4a899a700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4900a0790 0x7fa4900a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 0x5641d9eba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ebb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.264+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9e129b0 con 0x5641d9d166b0 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9e10370 con 0x5641d9ebadf0 2023-07-29T16:15:38.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9e0bc60 con 0x5641d9ebbcf0 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 0x5641d9eba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 0x5641d9eba8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54834/0 (socket says 172.21.15.31:54834) 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 -- 172.21.15.31:0/862547713 learned_addr learned my addr 172.21.15.31:0/862547713 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a999c700 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a899a700 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ebb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 -- 172.21.15.31:0/862547713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebbcf0 msgr2=0x5641d9ebc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 -- 172.21.15.31:0/862547713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 msgr2=0x5641d9ebb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ebb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a899a700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4900a0790 0x7fa4900a2c40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4074440 tx=0x7fa494011720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.265+0000 7fa4a919b700 1 -- 172.21.15.31:0/862547713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641d9c39d80 con 0x5641d9d166b0 2023-07-29T16:15:38.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.266+0000 7fa4a919b700 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 0x5641d9eba8b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009990 tx=0x7fa4a0005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797c68cdc7b1d8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.266+0000 7fa48f7fe700 1 -- 172.21.15.31:0/862547713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a001d070 con 0x5641d9d166b0 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.266+0000 7fa48f7fe700 1 -- 172.21.15.31:0/862547713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4a001e040 con 0x5641d9d166b0 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.266+0000 7fa48f7fe700 1 -- 172.21.15.31:0/862547713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0004920 con 0x5641d9d166b0 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 -- 172.21.15.31:0/862547713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 msgr2=0x5641d9eba8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 0x5641d9eba8b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009990 tx=0x7fa4a0005bc0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 -- 172.21.15.31:0/862547713 shutdown_connections 2023-07-29T16:15:38.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9d166b0 0x5641d9eba8b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/862547713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ebb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 -- 172.21.15.31:0/862547713 >> 172.21.15.31:0/862547713 conn(0x5641d9eaffd0 msgr2=0x5641d9eb5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 -- 172.21.15.31:0/862547713 shutdown_connections 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 -- 172.21.15.31:0/862547713 wait complete. 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.267+0000 7fa4bed00b80 1 -- start start 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 Processor -- start 2023-07-29T16:15:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ecb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9da84b0 con 0x5641d9ebbcf0 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9da8150 con 0x5641d9d166b0 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.268+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9c8f8c0 con 0x5641d9ebadf0 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4bed00b80 10 client.0 _ll_register_callbacks cb 0x5641d9db15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55666/0 (socket says 172.21.15.31:55666) 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 -- 172.21.15.31:0/2172135773 learned_addr learned my addr 172.21.15.31:0/2172135773 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a899a700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a999c700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ecb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 -- 172.21.15.31:0/2172135773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 msgr2=0x5641d9ec4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 -- 172.21.15.31:0/2172135773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 msgr2=0x5641d9ecb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ecb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 -- 172.21.15.31:0/2172135773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41090c0 con 0x5641d9d166b0 2023-07-29T16:15:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa4a919b700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec3e30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0006010 tx=0x7fa4a000baa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=396b35bef57f0468 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.269+0000 7fa48cff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa48cff9700 1 -- 172.21.15.31:0/2172135773 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a001d030 con 0x5641d9d166b0 2023-07-29T16:15:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa48cff9700 1 -- 172.21.15.31:0/2172135773 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa4a0004a90 con 0x5641d9d166b0 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641d9e36cb0 con 0x5641d9d166b0 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa48cff9700 1 -- 172.21.15.31:0/2172135773 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4a0004c50 con 0x5641d9d166b0 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 10 client.25664 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5641d9c22c00 con 0x5641d9d166b0 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa48cff9700 1 -- 172.21.15.31:0/2172135773 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5641d9e36cb0 con 0x5641d9d166b0 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa48cff9700 1 -- 172.21.15.31:0/2172135773 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa4a0022400 con 0x5641d9d166b0 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 10 client.25664 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:38.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 10 client.25664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 20 client.25664 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9ecf6e0 0x5641d9ed1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 4 client.25664 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.270+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641d9e89d10 con 0x5641d9ecf6e0 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.271+0000 7fa4a899a700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9ecf6e0 0x5641d9ed1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.271+0000 7fa4a899a700 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9ecf6e0 0x5641d9ed1af0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2c2ca7959ed782d server_cookie=17d7c627cec619ac in_seq=0 out_seq=0 2023-07-29T16:15:38.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.271+0000 7fa48cff9700 10 client.25664 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:38.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.272+0000 7fa48cff9700 1 -- 172.21.15.31:0/2172135773 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5641d9e89d10 con 0x5641d9ecf6e0 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.272+0000 7fa48cff9700 10 client.25664 handle_command_reply: tid=0 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.272+0000 7fa4bed00b80 1 client.25664 shutdown 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 msgr2=0x5641d9ec3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec3e30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0006010 tx=0x7fa4a000baa0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 shutdown_connections 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9ecf6e0 0x5641d9ed1af0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ecb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa48cff9700 0 client.25664 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec3e30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/2172135773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 >> 172.21.15.31:0/2172135773 conn(0x5641d9eaffd0 msgr2=0x5641d9eb4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 shutdown_connections 2023-07-29T16:15:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 1 -- 172.21.15.31:0/2172135773 wait complete. 2023-07-29T16:15:38.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.273+0000 7fa4bed00b80 20 client.25664 trim_cache size 0 max 16384 2023-07-29T16:15:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 Processor -- start 2023-07-29T16:15:38.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 -- start start 2023-07-29T16:15:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ebc600 con 0x5641d9ebbcf0 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9eb6190 con 0x5641d9d166b0 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ec4fb0 con 0x5641d9ebadf0 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4a999c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4a999c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54854/0 (socket says 172.21.15.31:54854) 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4a999c700 1 -- 172.21.15.31:0/1002992983 learned_addr learned my addr 172.21.15.31:0/1002992983 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4a919b700 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.288+0000 7fa4a899a700 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa4a999c700 1 -- 172.21.15.31:0/1002992983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 msgr2=0x5641d9ec2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa4a999c700 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa4a999c700 1 -- 172.21.15.31:0/1002992983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 msgr2=0x5641d9ec9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa4a999c700 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa4a999c700 1 -- 172.21.15.31:0/1002992983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641d9c601f0 con 0x5641d9ebbcf0 2023-07-29T16:15:38.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa4a999c700 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa49c094fc0 tx=0x7fa49c003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e439de87159d2443 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa48ffff700 1 -- 172.21.15.31:0/1002992983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0b5070 con 0x5641d9ebbcf0 2023-07-29T16:15:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa48ffff700 1 -- 172.21.15.31:0/1002992983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa49c0b1290 con 0x5641d9ebbcf0 2023-07-29T16:15:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.289+0000 7fa48ffff700 1 -- 172.21.15.31:0/1002992983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c096cd0 con 0x5641d9ebbcf0 2023-07-29T16:15:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1002992983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 msgr2=0x5641d9ebc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa49c094fc0 tx=0x7fa49c003dd0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1002992983 shutdown_connections 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ebc0c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9d166b0 0x5641d9ec9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1002992983 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9ebadf0 0x5641d9ec2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1002992983 >> 172.21.15.31:0/1002992983 conn(0x5641d9eaffd0 msgr2=0x5641d9ec2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1002992983 shutdown_connections 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.290+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1002992983 wait complete. 2023-07-29T16:15:38.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.291+0000 7fa4bed00b80 1 -- start start 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.291+0000 7fa4bed00b80 1 Processor -- start 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 0x5641d9ec8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ec7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ec7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ec9740 con 0x5641d9ebbcf0 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ec8180 con 0x5641d9ebadf0 2023-07-29T16:15:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ec8340 con 0x5641d9d166b0 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4bed00b80 10 client.0 _ll_register_callbacks cb 0x5641d9db15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a899a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ec7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 0x5641d9ec8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a999c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ec7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a899a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ec7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55684/0 (socket says 172.21.15.31:55684) 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 0x5641d9ec8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52670/0 (socket says 172.21.15.31:52670) 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a899a700 1 -- 172.21.15.31:0/1075025972 learned_addr learned my addr 172.21.15.31:0/1075025972 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 -- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 msgr2=0x5641d9ec7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ec7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 -- 172.21.15.31:0/1075025972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 msgr2=0x5641d9ec7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ec7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a919b700 1 -- 172.21.15.31:0/1075025972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a41a09a0 con 0x5641d9d166b0 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a899a700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ec7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.292+0000 7fa4a999c700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ec7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:38.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa4a919b700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 0x5641d9ec8f20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009350 tx=0x7fa4a0001e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66df6db711bc0daa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:38.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa48cff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:15:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa48cff9700 1 -- 172.21.15.31:0/1075025972 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a001d040 con 0x5641d9d166b0 2023-07-29T16:15:38.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4a001e040 con 0x5641d9d166b0 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa48cff9700 1 -- 172.21.15.31:0/1075025972 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa4a0004890 con 0x5641d9d166b0 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa48cff9700 1 -- 172.21.15.31:0/1075025972 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4a000bb10 con 0x5641d9d166b0 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa4bed00b80 10 client.25681 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.293+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5641d9c38250 con 0x5641d9d166b0 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa48cff9700 1 -- 172.21.15.31:0/1075025972 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4a001e040 con 0x5641d9d166b0 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa48cff9700 1 -- 172.21.15.31:0/1075025972 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa4a004b990 con 0x5641d9d166b0 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4bed00b80 10 client.25681 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:38.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4bed00b80 10 client.25681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4bed00b80 20 client.25681 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9f2abb0 0x5641d9f2cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4bed00b80 4 client.25681 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5641d9e04ce0 con 0x5641d9f2abb0 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.294+0000 7fa4a899a700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9f2abb0 0x5641d9f2cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.295+0000 7fa4a899a700 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9f2abb0 0x5641d9f2cfc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cb9de9d2aa5b39e server_cookie=1801b9a1d27de91b in_seq=0 out_seq=0 2023-07-29T16:15:38.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.295+0000 7fa48cff9700 10 client.25681 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.298+0000 7fa48cff9700 1 -- 172.21.15.31:0/1075025972 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5641d9e04ce0 con 0x5641d9f2abb0 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.298+0000 7fa48cff9700 10 client.25681 handle_command_reply: tid=0 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.298+0000 7fa4bed00b80 1 client.25681 shutdown 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 msgr2=0x5641d9ec8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 0x5641d9ec8f20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009350 tx=0x7fa4a0001e50 comp rx=0 tx=0).stop 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 shutdown_connections 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641d9f2abb0 0x5641d9f2cfc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641d9ebbcf0 0x5641d9ec7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641d9d166b0 0x5641d9ec8f20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa48cff9700 0 client.25681 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 --2- 172.21.15.31:0/1075025972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641d9ebadf0 0x5641d9ec7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 >> 172.21.15.31:0/1075025972 conn(0x5641d9eaffd0 msgr2=0x5641d9ebeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 shutdown_connections 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 1 -- 172.21.15.31:0/1075025972 wait complete. 2023-07-29T16:15:38.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4bed00b80 20 client.25681 trim_cache size 0 max 16384 2023-07-29T16:15:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:15:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:15:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:15:38.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:15:38.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4900a0790 msgr2=0x7fa4900a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4900a0790 0x7fa4900a2c40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4074440 tx=0x7fa494011720 comp rx=0 tx=0).stop 2023-07-29T16:15:38.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 msgr2=0x7fa4a4073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:38.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.299+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4073bc0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00c370 tx=0x7fa49c00c680 comp rx=0 tx=0).stop 2023-07-29T16:15:38.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 shutdown_connections 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4900a0790 0x7fa4900a2c40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4a4101ed0 0x7fa4a4073bc0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4a4100350 0x7fa4a4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 --2- 172.21.15.31:0/3545380085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4a4101540 0x7fa4a4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 >> 172.21.15.31:0/3545380085 conn(0x7fa4a40fb970 msgr2=0x7fa4a40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 shutdown_connections 2023-07-29T16:15:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:38.300+0000 7fa4ab3ff700 1 -- 172.21.15.31:0/3545380085 wait complete. 2023-07-29T16:15:38.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:38 smithi156 ceph-mon[129245]: pgmap v374: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:38.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:38 smithi031 ceph-mon[95504]: pgmap v374: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:39.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:38 smithi039 ceph-mon[128858]: pgmap v374: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:40.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:40 smithi156 ceph-mon[129245]: pgmap v375: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:40.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:40 smithi031 ceph-mon[95504]: pgmap v375: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:41.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:40 smithi039 ceph-mon[128858]: pgmap v375: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:42.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:15:42.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 -- 172.21.15.31:0/3547282145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 msgr2=0x7fdce4073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 --2- 172.21.15.31:0/3547282145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce4073c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8009990 tx=0x7fdcd8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 -- 172.21.15.31:0/3547282145 shutdown_connections 2023-07-29T16:15:42.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 --2- 172.21.15.31:0/3547282145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdce410c8f0 0x7fdce410ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 --2- 172.21.15.31:0/3547282145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 --2- 172.21.15.31:0/3547282145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce4073c20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 -- 172.21.15.31:0/3547282145 >> 172.21.15.31:0/3547282145 conn(0x7fdce40fb920 msgr2=0x7fdce40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 -- 172.21.15.31:0/3547282145 shutdown_connections 2023-07-29T16:15:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.787+0000 7fdce89ff700 1 -- 172.21.15.31:0/3547282145 wait complete. 2023-07-29T16:15:42.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.788+0000 7fdce89ff700 1 Processor -- start 2023-07-29T16:15:42.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 -- start start 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce419c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdce410c8f0 0x7fdce41a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce41a42a0 con 0x7fdce410c8f0 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce41a43e0 con 0x7fdce40737b0 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce89ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdce41a4570 con 0x7fdce4074d90 2023-07-29T16:15:42.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdce410c8f0 0x7fdce41a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce419c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce419cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34496/0 (socket says 172.21.15.31:34496) 2023-07-29T16:15:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.789+0000 7fdce1d9b700 1 -- 172.21.15.31:0/1766514925 learned_addr learned my addr 172.21.15.31:0/1766514925 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce259c700 1 -- 172.21.15.31:0/1766514925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 msgr2=0x7fdce419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce259c700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce259c700 1 -- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdce410c8f0 msgr2=0x7fdce41a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce259c700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdce410c8f0 0x7fdce41a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce259c700 1 -- 172.21.15.31:0/1766514925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80095f0 con 0x7fdce40737b0 2023-07-29T16:15:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdce259c700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce419c720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400e9c0 tx=0x7fdcd400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.790+0000 7fdcd37fe700 1 -- 172.21.15.31:0/1766514925 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd400c9c0 con 0x7fdce40737b0 2023-07-29T16:15:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.791+0000 7fdcd37fe700 1 -- 172.21.15.31:0/1766514925 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcd4010430 con 0x7fdce40737b0 2023-07-29T16:15:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.791+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce41a4850 con 0x7fdce40737b0 2023-07-29T16:15:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.791+0000 7fdcd37fe700 1 -- 172.21.15.31:0/1766514925 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4018680 con 0x7fdce40737b0 2023-07-29T16:15:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.791+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce41a4db0 con 0x7fdce40737b0 2023-07-29T16:15:42.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.794+0000 7fdcd37fe700 1 -- 172.21.15.31:0/1766514925 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdcd400cb20 con 0x7fdce40737b0 2023-07-29T16:15:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.796+0000 7fdcd37fe700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdccc0a0810 0x7fdccc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.796+0000 7fdcd37fe700 1 -- 172.21.15.31:0/1766514925 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdcd4014070 con 0x7fdce40737b0 2023-07-29T16:15:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.796+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdccc0a0810 0x7fdccc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.797+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdccc0a0810 0x7fdccc0a2cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800b2e0 tx=0x7fdcd8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:42.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.797+0000 7fdcfc3cab80 1 Processor -- start 2023-07-29T16:15:42.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.797+0000 7fdcfc3cab80 1 -- start start 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 0x5625e1d1bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1c530 0x5625e1d1cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d1d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1c73a10 con 0x5625e1d1bbe0 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1c713d0 con 0x5625e1d1c530 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1c6ccc0 con 0x5625e1d1d2f0 2023-07-29T16:15:42.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 0x5625e1d1bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1c530 0x5625e1d1cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 0x5625e1d1bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45334/0 (socket says 172.21.15.31:45334) 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 -- 172.21.15.31:0/904025833 learned_addr learned my addr 172.21.15.31:0/904025833 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce2d9d700 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d1d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 -- 172.21.15.31:0/904025833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 msgr2=0x5625e1d1d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d1d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 -- 172.21.15.31:0/904025833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1c530 msgr2=0x5625e1d1cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1c530 0x5625e1d1cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.798+0000 7fdce259c700 1 -- 172.21.15.31:0/904025833 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625e1a9ade0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.799+0000 7fdce259c700 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 0x5625e1d1bff0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4095cd0 tx=0x7fdcd4098c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24f6b3ecedba24 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.799+0000 7fdcd0ff9700 1 -- 172.21.15.31:0/904025833 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40a1620 con 0x5625e1d1bbe0 2023-07-29T16:15:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.799+0000 7fdcd0ff9700 1 -- 172.21.15.31:0/904025833 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcd40973a0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.799+0000 7fdcd0ff9700 1 -- 172.21.15.31:0/904025833 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40989b0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/904025833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 msgr2=0x5625e1d1bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 0x5625e1d1bff0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4095cd0 tx=0x7fdcd4098c40 comp rx=0 tx=0).stop 2023-07-29T16:15:42.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/904025833 shutdown_connections 2023-07-29T16:15:42.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d1d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1bbe0 0x5625e1d1bff0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/904025833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1c530 0x5625e1d1cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/904025833 >> 172.21.15.31:0/904025833 conn(0x5625e1d113e0 msgr2=0x5625e1d16990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/904025833 shutdown_connections 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/904025833 wait complete. 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.800+0000 7fdcfc3cab80 1 -- start start 2023-07-29T16:15:42.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 Processor -- start 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 0x5625e1d25840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d25d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1c09510 con 0x5625e1d1c530 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1c091b0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1af0920 con 0x5625e1d1d2f0 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdcfc3cab80 10 client.0 _ll_register_callbacks cb 0x5625e1c12620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:42.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.801+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d25d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 0x5625e1d25840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 0x5625e1d25840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55174/0 (socket says 172.21.15.31:55174) 2023-07-29T16:15:42.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 -- 172.21.15.31:0/4281500670 learned_addr learned my addr 172.21.15.31:0/4281500670 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:42.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce2d9d700 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 -- 172.21.15.31:0/4281500670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 msgr2=0x5625e1d2cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 -- 172.21.15.31:0/4281500670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 msgr2=0x5625e1d25d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d25d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 -- 172.21.15.31:0/4281500670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd4093e70 con 0x5625e1d1bbe0 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdce259c700 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 0x5625e1d25840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40a10f0 tx=0x7fdcd4097530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fda2e771ad959d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdcc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:42.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/4281500670 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40105a0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.802+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/4281500670 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdcd40a1620 con 0x5625e1d1bbe0 2023-07-29T16:15:42.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625e1c97d10 con 0x5625e1d1bbe0 2023-07-29T16:15:42.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/4281500670 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcd4010960 con 0x5625e1d1bbe0 2023-07-29T16:15:42.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 10 client.25682 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:42.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5625e1a83c60 con 0x5625e1d1bbe0 2023-07-29T16:15:42.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/4281500670 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcd407c020 con 0x5625e1d1bbe0 2023-07-29T16:15:42.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/4281500670 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdcd40b1bb0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 10 client.25682 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 10 client.25682 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 20 client.25682 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d311a0 0x5625e1d335b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 4 client.25682 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.803+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625e1cead70 con 0x5625e1d311a0 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.804+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d311a0 0x5625e1d335b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.804+0000 7fdcc9ffb700 10 client.25682 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.804+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d311a0 0x5625e1d335b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6845c94fda9a34 server_cookie=777cc6d763d3839a in_seq=0 out_seq=0 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/4281500670 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5625e1cead70 con 0x5625e1d311a0 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcc9ffb700 10 client.25682 handle_command_reply: tid=0 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 client.25682 shutdown 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 msgr2=0x5625e1d25840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 0x5625e1d25840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdcd40a10f0 tx=0x7fdcd4097530 comp rx=0 tx=0).stop 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 shutdown_connections 2023-07-29T16:15:42.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d311a0 0x5625e1d335b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1bbe0 0x5625e1d25840 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcc9ffb700 0 client.25682 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/4281500670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d25d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.805+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 >> 172.21.15.31:0/4281500670 conn(0x5625e1d113e0 msgr2=0x5625e1d16060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.806+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 shutdown_connections 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.806+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/4281500670 wait complete. 2023-07-29T16:15:42.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.806+0000 7fdcfc3cab80 20 client.25682 trim_cache size 0 max 16384 2023-07-29T16:15:42.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.823+0000 7fdcfc3cab80 1 Processor -- start 2023-07-29T16:15:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.823+0000 7fdcfc3cab80 1 -- start start 2023-07-29T16:15:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d24480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d16060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1d1cae0 con 0x5625e1d1c530 2023-07-29T16:15:42.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1d13460 con 0x5625e1d1d2f0 2023-07-29T16:15:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1d1c900 con 0x5625e1d1bbe0 2023-07-29T16:15:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d24480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45344/0 (socket says 172.21.15.31:45344) 2023-07-29T16:15:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce1d9b700 1 -- 172.21.15.31:0/721446869 learned_addr learned my addr 172.21.15.31:0/721446869 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:42.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce2d9d700 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d16060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce259c700 1 -- 172.21.15.31:0/721446869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 msgr2=0x5625e1d16060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce259c700 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d16060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce259c700 1 -- 172.21.15.31:0/721446869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 msgr2=0x5625e1d2cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce259c700 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce259c700 1 -- 172.21.15.31:0/721446869 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625e1ac1250 con 0x5625e1d1bbe0 2023-07-29T16:15:42.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.824+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.825+0000 7fdce259c700 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d24480 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4095cd0 tx=0x7fdcd4097810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21ba8522b20bc95e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.825+0000 7fdcd17fa700 1 -- 172.21.15.31:0/721446869 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4095050 con 0x5625e1d1bbe0 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.825+0000 7fdcd17fa700 1 -- 172.21.15.31:0/721446869 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5625e1c97d10 con 0x5625e1d1bbe0 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.825+0000 7fdcd17fa700 1 -- 172.21.15.31:0/721446869 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd40105a0 con 0x5625e1d1bbe0 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.825+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/721446869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 msgr2=0x5625e1d24480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d24480 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdcd4095cd0 tx=0x7fdcd4097810 comp rx=0 tx=0).stop 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/721446869 shutdown_connections 2023-07-29T16:15:42.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d16060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d24480 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/721446869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/721446869 >> 172.21.15.31:0/721446869 conn(0x5625e1d113e0 msgr2=0x5625e1d2d650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/721446869 shutdown_connections 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/721446869 wait complete. 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 -- start start 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.826+0000 7fdcfc3cab80 1 Processor -- start 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d2cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1d2c030 con 0x5625e1d1c530 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1d2c1a0 con 0x5625e1d1d2f0 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625e1d82d50 con 0x5625e1d1bbe0 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdcfc3cab80 10 client.0 _ll_register_callbacks cb 0x5625e1c12620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45358/0 (socket says 172.21.15.31:45358) 2023-07-29T16:15:42.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 -- 172.21.15.31:0/2307414790 learned_addr learned my addr 172.21.15.31:0/2307414790 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce2d9d700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce259c700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d2cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 -- 172.21.15.31:0/2307414790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 msgr2=0x5625e1d2cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d2cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 -- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 msgr2=0x5625e1d2b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce1d9b700 1 -- 172.21.15.31:0/2307414790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce410f680 con 0x5625e1d1c530 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.827+0000 7fdce259c700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d2cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdce1d9b700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2b330 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124fae661146efeb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcc9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/2307414790 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd80362c0 con 0x5625e1d1c530 2023-07-29T16:15:42.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/2307414790 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdcd8036420 con 0x5625e1d1c530 2023-07-29T16:15:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcd40973a0 con 0x5625e1d1c530 2023-07-29T16:15:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/2307414790 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcd8004500 con 0x5625e1d1c530 2023-07-29T16:15:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcfc3cab80 10 client.15711 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:42.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.828+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5625e1a83c60 con 0x5625e1d1c530 2023-07-29T16:15:42.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/2307414790 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcd40973a0 con 0x5625e1d1c530 2023-07-29T16:15:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/2307414790 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdcd800f620 con 0x5625e1d1c530 2023-07-29T16:15:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcfc3cab80 10 client.15711 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcfc3cab80 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcfc3cab80 20 client.15711 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d847d0 0x5625e1d86be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:42.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcfc3cab80 4 client.15711 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5625e1c65d40 con 0x5625e1d847d0 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.829+0000 7fdce259c700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d847d0 0x5625e1d86be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcc9ffb700 10 client.15711 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdce259c700 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d847d0 0x5625e1d86be0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e6b7f268d24799a server_cookie=de7e11a5d37c80b2 in_seq=0 out_seq=0 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcc9ffb700 1 -- 172.21.15.31:0/2307414790 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5625e1c65d40 con 0x5625e1d847d0 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcc9ffb700 10 client.15711 handle_command_reply: tid=0 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 client.15711 shutdown 2023-07-29T16:15:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 msgr2=0x5625e1d2b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2b330 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdcd8000c00 tx=0x7fdcd8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 shutdown_connections 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625e1d847d0 0x5625e1d86be0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625e1d1d2f0 0x5625e1d2b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625e1d1bbe0 0x5625e1d2cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 --2- 172.21.15.31:0/2307414790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625e1d1c530 0x5625e1d2b330 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 >> 172.21.15.31:0/2307414790 conn(0x5625e1d113e0 msgr2=0x5625e1d24480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:42.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcc9ffb700 0 client.15711 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:42.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 shutdown_connections 2023-07-29T16:15:42.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 1 -- 172.21.15.31:0/2307414790 wait complete. 2023-07-29T16:15:42.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.830+0000 7fdcfc3cab80 20 client.15711 trim_cache size 0 max 16384 2023-07-29T16:15:42.839 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:15:42.839 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:15:42.839 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:15:42.839 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:15:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdccc0a0810 msgr2=0x7fdccc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdccc0a0810 0x7fdccc0a2cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdcd800b2e0 tx=0x7fdcd8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 msgr2=0x7fdce419c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce419c720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400e9c0 tx=0x7fdcd400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 shutdown_connections 2023-07-29T16:15:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdccc0a0810 0x7fdccc0a2cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdce410c8f0 0x7fdce41a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdce40737b0 0x7fdce419c720 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 --2- 172.21.15.31:0/1766514925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdce4074d90 0x7fdce419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 >> 172.21.15.31:0/1766514925 conn(0x7fdce40fb920 msgr2=0x7fdce40fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 shutdown_connections 2023-07-29T16:15:42.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:42.831+0000 7fdce89ff700 1 -- 172.21.15.31:0/1766514925 wait complete. 2023-07-29T16:15:42.844 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:42 smithi031 ceph-mon[95504]: pgmap v376: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:42.846 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:15:42.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:15:43.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:42 smithi039 ceph-mon[128858]: pgmap v376: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:43.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 -- 172.21.15.31:0/4121123057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c100340 msgr2=0x7f650c100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 --2- 172.21.15.31:0/4121123057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c100340 0x7f650c100750 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6500009990 tx=0x7f6500009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 -- 172.21.15.31:0/4121123057 shutdown_connections 2023-07-29T16:15:43.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 --2- 172.21.15.31:0/4121123057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c101ec0 0x7f650c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 --2- 172.21.15.31:0/4121123057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c100340 0x7f650c100750 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 --2- 172.21.15.31:0/4121123057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f650c101530 0x7f650c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 -- 172.21.15.31:0/4121123057 >> 172.21.15.31:0/4121123057 conn(0x7f650c0fb920 msgr2=0x7f650c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 -- 172.21.15.31:0/4121123057 shutdown_connections 2023-07-29T16:15:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.059+0000 7f65131ff700 1 -- 172.21.15.31:0/4121123057 wait complete. 2023-07-29T16:15:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.060+0000 7f65131ff700 1 Processor -- start 2023-07-29T16:15:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 -- start start 2023-07-29T16:15:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 0x7f650c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f650c101530 0x7f650c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c101ec0 0x7f650c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c19fe00 con 0x7f650c101ec0 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c19ff40 con 0x7f650c100340 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f65131ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c1a00d0 con 0x7f650c101530 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f6510f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 0x7f650c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f6510f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 0x7f650c198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55212/0 (socket says 172.21.15.31:55212) 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f651179c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c101ec0 0x7f650c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f6510f9b700 1 -- 172.21.15.31:0/2100682978 learned_addr learned my addr 172.21.15.31:0/2100682978 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.061+0000 7f650bfff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f650c101530 0x7f650c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6510f9b700 1 -- 172.21.15.31:0/2100682978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f650c101530 msgr2=0x7f650c1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6510f9b700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f650c101530 0x7f650c1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6510f9b700 1 -- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c101ec0 msgr2=0x7f650c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6510f9b700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c101ec0 0x7f650c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6510f9b700 1 -- 172.21.15.31:0/2100682978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65000095f0 con 0x7f650c100340 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6510f9b700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 0x7f650c198280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6500005740 tx=0x7f6500005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650001d070 con 0x7f650c100340 2023-07-29T16:15:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c1a0350 con 0x7f650c100340 2023-07-29T16:15:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6500021410 con 0x7f650c100340 2023-07-29T16:15:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.062+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500017650 con 0x7f650c100340 2023-07-29T16:15:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.063+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c1a07d0 con 0x7f650c100340 2023-07-29T16:15:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.063+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c10bb60 con 0x7f650c100340 2023-07-29T16:15:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.066+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6500021580 con 0x7f650c100340 2023-07-29T16:15:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.068+0000 7f6509ffb700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64fc0a07e0 0x7f64fc0a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.068+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f65000cd0a0 con 0x7f650c100340 2023-07-29T16:15:43.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.068+0000 7f650bfff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64fc0a07e0 0x7f64fc0a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.068+0000 7f650bfff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64fc0a07e0 0x7f64fc0a2c90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009c60 tx=0x7f64f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.069+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6500096b10 con 0x7f650c100340 2023-07-29T16:15:43.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:42 smithi156 ceph-mon[129245]: pgmap v376: 97 pgs: 97 active+clean; 663 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:15:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.325+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f650c10bd50 con 0x7f650c100340 2023-07-29T16:15:43.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.326+0000 7f6509ffb700 1 -- 172.21.15.31:0/2100682978 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f650009a130 con 0x7f650c100340 2023-07-29T16:15:43.324 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:15:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:15:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64fc0a07e0 msgr2=0x7f64fc0a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64fc0a07e0 0x7f64fc0a2c90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f64f8009c60 tx=0x7f64f8009260 comp rx=0 tx=0).stop 2023-07-29T16:15:43.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 msgr2=0x7f650c198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 0x7f650c198280 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6500005740 tx=0x7f6500005770 comp rx=0 tx=0).stop 2023-07-29T16:15:43.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 shutdown_connections 2023-07-29T16:15:43.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f64fc0a07e0 0x7f64fc0a2c90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f650c101ec0 0x7f650c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f650c100340 0x7f650c198280 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 --2- 172.21.15.31:0/2100682978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f650c101530 0x7f650c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.328+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 >> 172.21.15.31:0/2100682978 conn(0x7f650c0fb920 msgr2=0x7f650c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.329+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 shutdown_connections 2023-07-29T16:15:43.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.329+0000 7f65131ff700 1 -- 172.21.15.31:0/2100682978 wait complete. 2023-07-29T16:15:43.329 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:15:43.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:15:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 -- 172.21.15.31:0/1542381599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c101520 msgr2=0x7f135c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 --2- 172.21.15.31:0/1542381599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c101520 0x7f135c101970 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1350009990 tx=0x7f1350009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 -- 172.21.15.31:0/1542381599 shutdown_connections 2023-07-29T16:15:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 --2- 172.21.15.31:0/1542381599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f135c101eb0 0x7f135c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 --2- 172.21.15.31:0/1542381599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c100330 0x7f135c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 --2- 172.21.15.31:0/1542381599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c101520 0x7f135c101970 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 -- 172.21.15.31:0/1542381599 >> 172.21.15.31:0/1542381599 conn(0x7f135c0fb970 msgr2=0x7f135c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 -- 172.21.15.31:0/1542381599 shutdown_connections 2023-07-29T16:15:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.558+0000 7f1361fff700 1 -- 172.21.15.31:0/1542381599 wait complete. 2023-07-29T16:15:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 Processor -- start 2023-07-29T16:15:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 -- start start 2023-07-29T16:15:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c100330 0x7f135c196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 0x7f135c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f135c101eb0 0x7f135c19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c19dd10 con 0x7f135c101520 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c19de50 con 0x7f135c100330 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.559+0000 7f1361fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c19dfe0 con 0x7f135c101eb0 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 0x7f135c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 0x7f135c1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45400/0 (socket says 172.21.15.31:45400) 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 -- 172.21.15.31:0/3326183511 learned_addr learned my addr 172.21.15.31:0/3326183511 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135bfff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f135c101eb0 0x7f135c19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135b7fe700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c100330 0x7f135c196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 -- 172.21.15.31:0/3326183511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f135c101eb0 msgr2=0x7f135c19d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f135c101eb0 0x7f135c19d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 -- 172.21.15.31:0/3326183511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c100330 msgr2=0x7f135c196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c100330 0x7f135c196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 -- 172.21.15.31:0/3326183511 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13500095f0 con 0x7f135c101520 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.560+0000 7f135affd700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 0x7f135c1966d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f135000c010 tx=0x7f1350004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.561+0000 7f1358ff9700 1 -- 172.21.15.31:0/3326183511 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135001d070 con 0x7f135c101520 2023-07-29T16:15:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.561+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c19e260 con 0x7f135c101520 2023-07-29T16:15:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.561+0000 7f1358ff9700 1 -- 172.21.15.31:0/3326183511 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1350005030 con 0x7f135c101520 2023-07-29T16:15:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.561+0000 7f1358ff9700 1 -- 172.21.15.31:0/3326183511 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350004b90 con 0x7f135c101520 2023-07-29T16:15:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.561+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c19e680 con 0x7f135c101520 2023-07-29T16:15:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.564+0000 7f1358ff9700 1 -- 172.21.15.31:0/3326183511 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f13500051a0 con 0x7f135c101520 2023-07-29T16:15:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.565+0000 7f1375969b80 1 Processor -- start 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.565+0000 7f1375969b80 1 -- start start 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1358ff9700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13440a0810 0x7f13440a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1375969b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d3a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1375969b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d3b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1358ff9700 1 -- 172.21.15.31:0/3326183511 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13500cccb0 con 0x7f135c101520 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1375969b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1375969b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3c8da10 con 0x55acd3d3a990 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1375969b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3c8b3d0 con 0x55acd3d3a040 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f1375969b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3c86cc0 con 0x55acd3d3b6d0 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135bfff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13440a0810 0x7f13440a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d3b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d3b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45408/0 (socket says 172.21.15.31:45408) 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 -- 172.21.15.31:0/1794678553 learned_addr learned my addr 172.21.15.31:0/1794678553 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135b7fe700 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d3a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135bfff700 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 -- 172.21.15.31:0/1794678553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3b6d0 msgr2=0x55acd3d3bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 -- 172.21.15.31:0/1794678553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 msgr2=0x55acd3d3a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.566+0000 7f135affd700 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d3a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.567+0000 7f135affd700 1 -- 172.21.15.31:0/1794678553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acd3ab4de0 con 0x55acd3d3a990 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.567+0000 7f135affd700 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d3b140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f135009a370 tx=0x7f1350099e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6287fbd1112c6aba server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.567+0000 7f135bfff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13440a0810 0x7f13440a2cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1354009e10 tx=0x7f1354009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.567+0000 7f1341ffb700 1 -- 172.21.15.31:0/1794678553 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500a27a0 con 0x55acd3d3a990 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.567+0000 7f1341ffb700 1 -- 172.21.15.31:0/1794678553 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1350098780 con 0x55acd3d3a990 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.567+0000 7f1341ffb700 1 -- 172.21.15.31:0/1794678553 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500a0300 con 0x55acd3d3a990 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 -- 172.21.15.31:0/1794678553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 msgr2=0x55acd3d3b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d3b140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f135009a370 tx=0x7f1350099e70 comp rx=0 tx=0).stop 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 -- 172.21.15.31:0/1794678553 shutdown_connections 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d3a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 --2- 172.21.15.31:0/1794678553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d3b140 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 -- 172.21.15.31:0/1794678553 >> 172.21.15.31:0/1794678553 conn(0x55acd3d2b3e0 msgr2=0x55acd3d308a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 -- 172.21.15.31:0/1794678553 shutdown_connections 2023-07-29T16:15:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 -- 172.21.15.31:0/1794678553 wait complete. 2023-07-29T16:15:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.568+0000 7f1375969b80 1 -- start start 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 Processor -- start 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d43820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d43d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3c23510 con 0x55acd3d3b6d0 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3c231b0 con 0x55acd3d3a040 2023-07-29T16:15:43.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3b0a920 con 0x55acd3d3a990 2023-07-29T16:15:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f1375969b80 10 client.0 _ll_register_callbacks cb 0x55acd3c2c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f135b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d43820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f135b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d43820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55236/0 (socket says 172.21.15.31:55236) 2023-07-29T16:15:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f135b7fe700 1 -- 172.21.15.31:0/3741880927 learned_addr learned my addr 172.21.15.31:0/3741880927 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f135bfff700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.569+0000 7f135affd700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d43d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f135affd700 1 -- 172.21.15.31:0/3741880927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 msgr2=0x55acd3d43820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f135affd700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d43820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f135affd700 1 -- 172.21.15.31:0/3741880927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3b6d0 msgr2=0x55acd3d4ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f135affd700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f135affd700 1 -- 172.21.15.31:0/3741880927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c19e680 con 0x55acd3d3a990 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f135affd700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d43d60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1350096e50 tx=0x7f13500a0330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6536de18d5846ca5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.570+0000 7f133affd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f133affd700 1 -- 172.21.15.31:0/3741880927 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135009e250 con 0x55acd3d3a990 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f133affd700 1 -- 172.21.15.31:0/3741880927 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13500b1030 con 0x55acd3d3a990 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55acd3cb1d10 con 0x55acd3d3a990 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f133affd700 1 -- 172.21.15.31:0/3741880927 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13500a27a0 con 0x55acd3d3a990 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f1375969b80 10 client.25717 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55acd3a9dc60 con 0x55acd3d3a990 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.571+0000 7f133affd700 1 -- 172.21.15.31:0/3741880927 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55acd3cb1d10 con 0x55acd3d3a990 2023-07-29T16:15:43.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f133affd700 1 -- 172.21.15.31:0/3741880927 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f13500a2410 con 0x55acd3d3a990 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f1375969b80 10 client.25717 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f1375969b80 10 client.25717 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f1375969b80 20 client.25717 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f1375969b80 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3d4f100 0x55acd3d51510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f1375969b80 4 client.25717 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55acd3d04d70 con 0x55acd3d4f100 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.572+0000 7f135b7fe700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3d4f100 0x55acd3d51510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.573+0000 7f133affd700 10 client.25717 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.573+0000 7f135b7fe700 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3d4f100 0x55acd3d51510 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=106adb36ed10d564 server_cookie=b9c6189b91dd7b32 in_seq=0 out_seq=0 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f133affd700 1 -- 172.21.15.31:0/3741880927 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55acd3d04d70 con 0x55acd3d4f100 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f133affd700 10 client.25717 handle_command_reply: tid=0 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 client.25717 shutdown 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 msgr2=0x55acd3d43d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d43d60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1350096e50 tx=0x7f13500a0330 comp rx=0 tx=0).stop 2023-07-29T16:15:43.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 shutdown_connections 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3d4f100 0x55acd3d51510 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3a040 0x55acd3d43820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 --2- 172.21.15.31:0/3741880927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d43d60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f133affd700 0 client.25717 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 >> 172.21.15.31:0/3741880927 conn(0x55acd3d2b3e0 msgr2=0x55acd3d30060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 shutdown_connections 2023-07-29T16:15:43.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.574+0000 7f1375969b80 1 -- 172.21.15.31:0/3741880927 wait complete. 2023-07-29T16:15:43.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.575+0000 7f1375969b80 20 client.25717 trim_cache size 0 max 16384 2023-07-29T16:15:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 Processor -- start 2023-07-29T16:15:43.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 -- start start 2023-07-29T16:15:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a040 0x55acd3d51510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d4ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3d42cd0 con 0x55acd3d3a040 2023-07-29T16:15:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3d2d460 con 0x55acd3d3b6d0 2023-07-29T16:15:43.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f1375969b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3d433c0 con 0x55acd3d3a990 2023-07-29T16:15:43.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f135bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f135b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a040 0x55acd3d51510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f135bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55242/0 (socket says 172.21.15.31:55242) 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.589+0000 7f135bfff700 1 -- 172.21.15.31:0/798003756 learned_addr learned my addr 172.21.15.31:0/798003756 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135affd700 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d4ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135bfff700 1 -- 172.21.15.31:0/798003756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 msgr2=0x55acd3d4ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135bfff700 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d4ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135bfff700 1 -- 172.21.15.31:0/798003756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a040 msgr2=0x55acd3d51510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135bfff700 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a040 0x55acd3d51510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135bfff700 1 -- 172.21.15.31:0/798003756 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55acd3adb250 con 0x55acd3d3b6d0 2023-07-29T16:15:43.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135affd700 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d4ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:43.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135b7fe700 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a040 0x55acd3d51510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:43.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f135bfff700 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3baa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f13540048b0 tx=0x7f1354004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f128f657acb148 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.590+0000 7f13427fc700 1 -- 172.21.15.31:0/798003756 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354029070 con 0x55acd3d3b6d0 2023-07-29T16:15:43.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f13427fc700 1 -- 172.21.15.31:0/798003756 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13540048e0 con 0x55acd3d3b6d0 2023-07-29T16:15:43.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f13427fc700 1 -- 172.21.15.31:0/798003756 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354005220 con 0x55acd3d3b6d0 2023-07-29T16:15:43.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 -- 172.21.15.31:0/798003756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 msgr2=0x55acd3d3baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3baa0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f13540048b0 tx=0x7f1354004b40 comp rx=0 tx=0).stop 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 -- 172.21.15.31:0/798003756 shutdown_connections 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d3baa0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a040 0x55acd3d51510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 --2- 172.21.15.31:0/798003756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a990 0x55acd3d4ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 -- 172.21.15.31:0/798003756 >> 172.21.15.31:0/798003756 conn(0x55acd3d2b3e0 msgr2=0x55acd3d30870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 -- 172.21.15.31:0/798003756 shutdown_connections 2023-07-29T16:15:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.591+0000 7f1375969b80 1 -- 172.21.15.31:0/798003756 wait complete. 2023-07-29T16:15:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 -- start start 2023-07-29T16:15:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 Processor -- start 2023-07-29T16:15:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a040 0x55acd3d46d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d47260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3d4a7c0 con 0x55acd3d3a990 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3d4a930 con 0x55acd3d3b6d0 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.592+0000 7f1375969b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55acd3d4aaf0 con 0x55acd3d3a040 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f1375969b80 10 client.0 _ll_register_callbacks cb 0x55acd3c2c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d47260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d47260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45428/0 (socket says 172.21.15.31:45428) 2023-07-29T16:15:43.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 -- 172.21.15.31:0/431606393 learned_addr learned my addr 172.21.15.31:0/431606393 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135b7fe700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a040 0x55acd3d46d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135bfff700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 -- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a040 msgr2=0x55acd3d46d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a040 0x55acd3d46d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 -- 172.21.15.31:0/431606393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 msgr2=0x55acd3d4a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 -- 172.21.15.31:0/431606393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c009c00 con 0x55acd3d3a990 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f135affd700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d47260 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f135009a090 tx=0x7f1350099e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba853ece99aae15a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f133affd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.593+0000 7f133affd700 1 -- 172.21.15.31:0/431606393 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500a27a0 con 0x55acd3d3a990 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f133affd700 1 -- 172.21.15.31:0/431606393 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13500b1040 con 0x55acd3d3a990 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1350098780 con 0x55acd3d3a990 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f133affd700 1 -- 172.21.15.31:0/431606393 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f135009e040 con 0x55acd3d3a990 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 10 client.15735 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:43.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55acd3a9dc60 con 0x55acd3d3a990 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f133affd700 1 -- 172.21.15.31:0/431606393 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13500a6020 con 0x55acd3d3a990 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f133affd700 1 -- 172.21.15.31:0/431606393 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f13500961d0 con 0x55acd3d3a990 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 10 client.15735 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 10 client.15735 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 20 client.15735 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3da19f0 0x55acd3da3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 4 client.15735 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:15:43.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.594+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55acd3c7fd40 con 0x55acd3da19f0 2023-07-29T16:15:43.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.595+0000 7f135b7fe700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3da19f0 0x55acd3da3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.595+0000 7f135b7fe700 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3da19f0 0x55acd3da3e00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29a7b6e6dd13c1cd server_cookie=c7d9afba29dc5740 in_seq=0 out_seq=0 2023-07-29T16:15:43.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.595+0000 7f133affd700 10 client.15735 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:43.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.595+0000 7f133affd700 1 -- 172.21.15.31:0/431606393 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55acd3c7fd40 con 0x55acd3da19f0 2023-07-29T16:15:43.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.595+0000 7f133affd700 10 client.15735 handle_command_reply: tid=0 2023-07-29T16:15:43.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.595+0000 7f1375969b80 1 client.15735 shutdown 2023-07-29T16:15:43.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 msgr2=0x55acd3d47260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d47260 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f135009a090 tx=0x7f1350099e70 comp rx=0 tx=0).stop 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 shutdown_connections 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55acd3da19f0 0x55acd3da3e00 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55acd3d3b6d0 0x55acd3d4a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55acd3d3a040 0x55acd3d46d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 --2- 172.21.15.31:0/431606393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55acd3d3a990 0x55acd3d47260 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f133affd700 0 client.15735 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 >> 172.21.15.31:0/431606393 conn(0x55acd3d2b3e0 msgr2=0x55acd3d41a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 shutdown_connections 2023-07-29T16:15:43.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 1 -- 172.21.15.31:0/431606393 wait complete. 2023-07-29T16:15:43.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.596+0000 7f1375969b80 20 client.15735 trim_cache size 0 max 16384 2023-07-29T16:15:43.602 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:15:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13440a0810 msgr2=0x7f13440a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13440a0810 0x7f13440a2cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1354009e10 tx=0x7f1354009490 comp rx=0 tx=0).stop 2023-07-29T16:15:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 msgr2=0x7f135c1966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 0x7f135c1966d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f135000c010 tx=0x7f1350004730 comp rx=0 tx=0).stop 2023-07-29T16:15:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 shutdown_connections 2023-07-29T16:15:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13440a0810 0x7f13440a2cc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f135c101eb0 0x7f135c19d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135c100330 0x7f135c196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 --2- 172.21.15.31:0/3326183511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f135c101520 0x7f135c1966d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 >> 172.21.15.31:0/3326183511 conn(0x7f135c0fb970 msgr2=0x7f135c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 shutdown_connections 2023-07-29T16:15:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.597+0000 7f1361fff700 1 -- 172.21.15.31:0/3326183511 wait complete. 2023-07-29T16:15:43.611 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:43 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2100682978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:15:43.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:15:43.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2235184376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101520 msgr2=0x7fc3d0101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2235184376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101520 0x7fc3d0101970 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40098f0 tx=0x7fc3c4009c00 comp rx=0 tx=0).stop 2023-07-29T16:15:43.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2235184376 shutdown_connections 2023-07-29T16:15:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2235184376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101eb0 0x7fc3d0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2235184376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3d0100330 0x7fc3d0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2235184376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101520 0x7fc3d0101970 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2235184376 >> 172.21.15.31:0/2235184376 conn(0x7fc3d00fb970 msgr2=0x7fc3d00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2235184376 shutdown_connections 2023-07-29T16:15:43.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.827+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2235184376 wait complete. 2023-07-29T16:15:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.828+0000 7fc3d59ff700 1 Processor -- start 2023-07-29T16:15:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.828+0000 7fc3d59ff700 1 -- start start 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3d59ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0100330 0x7fc3d0196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3d59ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 0x7fc3d01965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3d59ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3d0101eb0 0x7fc3d019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3d59ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d019dc10 con 0x7fc3d0101520 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3d59ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d019dd50 con 0x7fc3d0101eb0 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3d59ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d019dee0 con 0x7fc3d0100330 2023-07-29T16:15:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 0x7fc3d01965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ceffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0100330 0x7fc3d0196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 0x7fc3d01965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45450/0 (socket says 172.21.15.31:45450) 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2799345956 learned_addr learned my addr 172.21.15.31:0/2799345956 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3d0101eb0 0x7fc3d019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2799345956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0100330 msgr2=0x7fc3d0196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0100330 0x7fc3d0196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2799345956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3d0101eb0 msgr2=0x7fc3d019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3d0101eb0 0x7fc3d019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.829+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2799345956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8009d00 con 0x7fc3d0101520 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.830+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 0x7fc3d01965d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009fd0 tx=0x7fc3c400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.830+0000 7fc3d49fd700 1 -- 172.21.15.31:0/2799345956 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c401c070 con 0x7fc3d0101520 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.830+0000 7fc3d49fd700 1 -- 172.21.15.31:0/2799345956 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c400f7d0 con 0x7fc3d0101520 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.830+0000 7fc3d49fd700 1 -- 172.21.15.31:0/2799345956 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4004ad0 con 0x7fc3d0101520 2023-07-29T16:15:43.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.830+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c4009520 con 0x7fc3d0101520 2023-07-29T16:15:43.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.830+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d019e400 con 0x7fc3d0101520 2023-07-29T16:15:43.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.833+0000 7fc3d49fd700 1 -- 172.21.15.31:0/2799345956 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc3c4020070 con 0x7fc3d0101520 2023-07-29T16:15:43.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 Processor -- start 2023-07-29T16:15:43.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3d49fd700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3b80a0810 0x7fc3b80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 -- start start 2023-07-29T16:15:43.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3d49fd700 1 -- 172.21.15.31:0/2799345956 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc3c40cc690 con 0x7fc3d0101520 2023-07-29T16:15:43.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3b80a0810 0x7fc3b80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dbbe0 0x55762f9dbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 0x55762f9dcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dd2f0 0x55762f9dd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f933a10 con 0x55762f9dc530 2023-07-29T16:15:43.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9313d0 con 0x55762f9dd2f0 2023-07-29T16:15:43.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f92ccc0 con 0x55762f9dbbe0 2023-07-29T16:15:43.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.835+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 0x55762f9dcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 0x55762f9dcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45454/0 (socket says 172.21.15.31:45454) 2023-07-29T16:15:43.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/4112283630 learned_addr learned my addr 172.21.15.31:0/4112283630 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ceffd700 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dbbe0 0x55762f9dbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dd2f0 0x55762f9dd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/4112283630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dbbe0 msgr2=0x55762f9dbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dbbe0 0x55762f9dbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/4112283630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dd2f0 msgr2=0x55762f9dd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dd2f0 0x55762f9dd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/4112283630 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55762f75ade0 con 0x55762f9dc530 2023-07-29T16:15:43.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3b80a0810 0x7fc3b80a2cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009fd0 tx=0x7fc3c80078a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 0x55762f9dcd60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4099d50 tx=0x7fc3c4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbb477899b75b24 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.836+0000 7fc3b5ffb700 1 -- 172.21.15.31:0/4112283630 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40b1070 con 0x55762f9dc530 2023-07-29T16:15:43.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.837+0000 7fc3b5ffb700 1 -- 172.21.15.31:0/4112283630 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c409ddf0 con 0x55762f9dc530 2023-07-29T16:15:43.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.837+0000 7fc3b5ffb700 1 -- 172.21.15.31:0/4112283630 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4004c30 con 0x55762f9dc530 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.837+0000 7fc3e9355b80 1 -- 172.21.15.31:0/4112283630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 msgr2=0x55762f9dcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 0x55762f9dcd60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4099d50 tx=0x7fc3c4009c00 comp rx=0 tx=0).stop 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 -- 172.21.15.31:0/4112283630 shutdown_connections 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dd2f0 0x55762f9dd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dbbe0 0x55762f9dbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/4112283630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dc530 0x55762f9dcd60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 -- 172.21.15.31:0/4112283630 >> 172.21.15.31:0/4112283630 conn(0x55762f9d13e0 msgr2=0x55762f9d6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 -- 172.21.15.31:0/4112283630 shutdown_connections 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 -- 172.21.15.31:0/4112283630 wait complete. 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.838+0000 7fc3e9355b80 1 -- start start 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 Processor -- start 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dbbe0 0x55762f9e98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dc530 0x55762f9e9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 0x55762f9f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f8c9510 con 0x55762f9dd2f0 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f8c91b0 con 0x55762f9dbbe0 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f7b0920 con 0x55762f9dc530 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3e9355b80 10 client.0 _ll_register_callbacks cb 0x55762f8d2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3cf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 0x55762f9f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dc530 0x55762f9e9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3ceffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dbbe0 0x55762f9e98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3cf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 0x55762f9f0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45470/0 (socket says 172.21.15.31:45470) 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.839+0000 7fc3cf7fe700 1 -- 172.21.15.31:0/1915238166 learned_addr learned my addr 172.21.15.31:0/1915238166 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3cf7fe700 1 -- 172.21.15.31:0/1915238166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dc530 msgr2=0x55762f9e9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dc530 0x55762f9e9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3cf7fe700 1 -- 172.21.15.31:0/1915238166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dbbe0 msgr2=0x55762f9e98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dbbe0 0x55762f9e98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3cf7fe700 1 -- 172.21.15.31:0/1915238166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d019e400 con 0x55762f9dd2f0 2023-07-29T16:15:43.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dc530 0x55762f9e9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 0x55762f9f0e50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8018e20 tx=0x7fc3c8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7070265fe6f030 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3aeffd700 1 -- 172.21.15.31:0/1915238166 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c800e020 con 0x55762f9dd2f0 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55762f957d10 con 0x55762f9dd2f0 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.840+0000 7fc3aeffd700 1 -- 172.21.15.31:0/1915238166 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc3c8017a50 con 0x55762f9dd2f0 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3aeffd700 1 -- 172.21.15.31:0/1915238166 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3c8026070 con 0x55762f9dd2f0 2023-07-29T16:15:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 10 client.15747 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55762f743c60 con 0x55762f9dd2f0 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3aeffd700 1 -- 172.21.15.31:0/1915238166 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c8030020 con 0x55762f9dd2f0 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3aeffd700 1 -- 172.21.15.31:0/1915238166 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc3c8029030 con 0x55762f9dd2f0 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 10 client.15747 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 10 client.15747 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 20 client.15747 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762f9f56b0 0x55762f9f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 4 client.15747 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55762f9aad70 con 0x55762f9f56b0 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.841+0000 7fc3ceffd700 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762f9f56b0 0x55762f9f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.842+0000 7fc3ceffd700 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762f9f56b0 0x55762f9f7ac0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4225bfa5effc60c9 server_cookie=deac3c9880cf027a in_seq=0 out_seq=0 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.842+0000 7fc3aeffd700 10 client.15747 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.842+0000 7fc3aeffd700 1 -- 172.21.15.31:0/1915238166 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55762f9aad70 con 0x55762f9f56b0 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.842+0000 7fc3aeffd700 10 client.15747 handle_command_reply: tid=0 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 client.15747 shutdown 2023-07-29T16:15:43.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 msgr2=0x55762f9f0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 0x55762f9f0e50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8018e20 tx=0x7fc3c8004b40 comp rx=0 tx=0).stop 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 shutdown_connections 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762f9f56b0 0x55762f9f7ac0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dd2f0 0x55762f9f0e50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dbbe0 0x55762f9e98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3aeffd700 0 client.15747 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:43.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/1915238166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dc530 0x55762f9e9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 >> 172.21.15.31:0/1915238166 conn(0x55762f9d13e0 msgr2=0x55762f9d6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 shutdown_connections 2023-07-29T16:15:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 1 -- 172.21.15.31:0/1915238166 wait complete. 2023-07-29T16:15:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.843+0000 7fc3e9355b80 20 client.15747 trim_cache size 0 max 16384 2023-07-29T16:15:43.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 Processor -- start 2023-07-29T16:15:43.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 -- start start 2023-07-29T16:15:43.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9d6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9e3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9e4970 con 0x55762f9dbbe0 2023-07-29T16:15:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9d7ce0 con 0x55762f9dc530 2023-07-29T16:15:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9d3460 con 0x55762f9dd2f0 2023-07-29T16:15:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ceffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9d6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55300/0 (socket says 172.21.15.31:55300) 2023-07-29T16:15:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2505200100 learned_addr learned my addr 172.21.15.31:0/2505200100 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2505200100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 msgr2=0x55762f9e3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9e3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2505200100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 msgr2=0x55762f9d6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.858+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9d6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.859+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/2505200100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55762f781250 con 0x55762f9dc530 2023-07-29T16:15:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.859+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9f0e10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4005e90 tx=0x7fc3c4096230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21344e625d3b73af server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.859+0000 7fc3b67fc700 1 -- 172.21.15.31:0/2505200100 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c40b1040 con 0x55762f9dc530 2023-07-29T16:15:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.859+0000 7fc3b67fc700 1 -- 172.21.15.31:0/2505200100 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c40a2180 con 0x55762f9dc530 2023-07-29T16:15:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.859+0000 7fc3b67fc700 1 -- 172.21.15.31:0/2505200100 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c409a280 con 0x55762f9dc530 2023-07-29T16:15:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 -- 172.21.15.31:0/2505200100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 msgr2=0x55762f9f0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9f0e10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4005e90 tx=0x7fc3c4096230 comp rx=0 tx=0).stop 2023-07-29T16:15:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 -- 172.21.15.31:0/2505200100 shutdown_connections 2023-07-29T16:15:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9e3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9d6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/2505200100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9f0e10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 -- 172.21.15.31:0/2505200100 >> 172.21.15.31:0/2505200100 conn(0x55762f9d13e0 msgr2=0x55762f9d6900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 -- 172.21.15.31:0/2505200100 shutdown_connections 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 -- 172.21.15.31:0/2505200100 wait complete. 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 -- start start 2023-07-29T16:15:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.860+0000 7fc3e9355b80 1 Processor -- start 2023-07-29T16:15:43.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9ed010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9ed550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9f0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9f0ab0 con 0x55762f9dbbe0 2023-07-29T16:15:43.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9f0c20 con 0x55762f9dc530 2023-07-29T16:15:43.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55762f9f0de0 con 0x55762f9dd2f0 2023-07-29T16:15:43.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3e9355b80 10 client.0 _ll_register_callbacks cb 0x55762f8d2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:43.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3ceffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9ed550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3ceffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9ed010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45488/0 (socket says 172.21.15.31:45488) 2023-07-29T16:15:43.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3ceffd700 1 -- 172.21.15.31:0/359652484 learned_addr learned my addr 172.21.15.31:0/359652484 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3ce7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9ed550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55306/0 (socket says 172.21.15.31:55306) 2023-07-29T16:15:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.861+0000 7fc3cf7fe700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9f0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/359652484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 msgr2=0x55762f9f0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9f0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 msgr2=0x55762f9ed010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9ed010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ce7fc700 1 -- 172.21.15.31:0/359652484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55762f75ade0 con 0x55762f9dc530 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ceffd700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3ce7fc700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9ed550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40098c0 tx=0x7fc3c4009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad34c3ad1164d5dd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3aeffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3aeffd700 1 -- 172.21.15.31:0/359652484 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c407c060 con 0x55762f9dc530 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3aeffd700 1 -- 172.21.15.31:0/359652484 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc3c409a220 con 0x55762f9dc530 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3c409ddf0 con 0x55762f9dc530 2023-07-29T16:15:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3aeffd700 1 -- 172.21.15.31:0/359652484 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3c40b1070 con 0x55762f9dc530 2023-07-29T16:15:43.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3e9355b80 10 client.25730 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:43.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.862+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55762f7592b0 con 0x55762f9dc530 2023-07-29T16:15:43.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3aeffd700 1 -- 172.21.15.31:0/359652484 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c409ddf0 con 0x55762f9dc530 2023-07-29T16:15:43.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3aeffd700 1 -- 172.21.15.31:0/359652484 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc3c4080030 con 0x55762f9dc530 2023-07-29T16:15:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3e9355b80 10 client.25730 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3e9355b80 10 client.25730 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:15:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3e9355b80 20 client.25730 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762fa48130 0x55762fa4a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3e9355b80 4 client.25730 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:15:43.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55762f925d40 con 0x55762fa48130 2023-07-29T16:15:43.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.863+0000 7fc3ceffd700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762fa48130 0x55762fa4a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:43.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3ceffd700 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762fa48130 0x55762fa4a510 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5bfe722ecbfc738c server_cookie=59067e800ea086c in_seq=0 out_seq=0 2023-07-29T16:15:43.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3aeffd700 10 client.25730 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:43.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3aeffd700 1 -- 172.21.15.31:0/359652484 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55762f925d40 con 0x55762fa48130 2023-07-29T16:15:43.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3aeffd700 10 client.25730 handle_command_reply: tid=0 2023-07-29T16:15:43.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 client.25730 shutdown 2023-07-29T16:15:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 msgr2=0x55762f9ed550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9ed550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc3c40098c0 tx=0x7fc3c4009c00 comp rx=0 tx=0).stop 2023-07-29T16:15:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 shutdown_connections 2023-07-29T16:15:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55762fa48130 0x55762fa4a510 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55762f9dd2f0 0x55762f9f0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55762f9dbbe0 0x55762f9ed010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3aeffd700 0 client.25730 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 --2- 172.21.15.31:0/359652484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55762f9dc530 0x55762f9ed550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 >> 172.21.15.31:0/359652484 conn(0x55762f9d13e0 msgr2=0x55762f9e0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 shutdown_connections 2023-07-29T16:15:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 1 -- 172.21.15.31:0/359652484 wait complete. 2023-07-29T16:15:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.864+0000 7fc3e9355b80 20 client.25730 trim_cache size 0 max 16384 2023-07-29T16:15:43.876 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:15:43.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3b80a0810 msgr2=0x7fc3b80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3b80a0810 0x7fc3b80a2cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009fd0 tx=0x7fc3c80078a0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 msgr2=0x7fc3d01965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 0x7fc3d01965d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009fd0 tx=0x7fc3c400b790 comp rx=0 tx=0).stop 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 shutdown_connections 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3b80a0810 0x7fc3b80a2cc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc3d0101eb0 0x7fc3d019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0100330 0x7fc3d0196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 --2- 172.21.15.31:0/2799345956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3d0101520 0x7fc3d01965d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 >> 172.21.15.31:0/2799345956 conn(0x7fc3d00fb970 msgr2=0x7fc3d00fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 shutdown_connections 2023-07-29T16:15:43.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:43.865+0000 7fc3d59ff700 1 -- 172.21.15.31:0/2799345956 wait complete. 2023-07-29T16:15:43.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:15:44.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:43 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2100682978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:15:44.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:43 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2100682978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:15:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 -- 172.21.15.31:0/54041368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 msgr2=0x7fa538108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/54041368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa538108760 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa530009f90 tx=0x7fa53000b330 comp rx=0 tx=0).stop 2023-07-29T16:15:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 -- 172.21.15.31:0/54041368 shutdown_connections 2023-07-29T16:15:44.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/54041368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa538108760 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/54041368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538100320 0x7fa538100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/54041368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa538101510 0x7fa538101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 -- 172.21.15.31:0/54041368 >> 172.21.15.31:0/54041368 conn(0x7fa5380fb920 msgr2=0x7fa5380fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 -- 172.21.15.31:0/54041368 shutdown_connections 2023-07-29T16:15:44.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.091+0000 7fa53c9ff700 1 -- 172.21.15.31:0/54041368 wait complete. 2023-07-29T16:15:44.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 Processor -- start 2023-07-29T16:15:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 -- start start 2023-07-29T16:15:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa538100320 0x7fa5381961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 0x7fa538196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa53819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53819dd40 con 0x7fa538101ea0 2023-07-29T16:15:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53819de80 con 0x7fa538101510 2023-07-29T16:15:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53c9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53819e010 con 0x7fa538100320 2023-07-29T16:15:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa536d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa53819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 0x7fa538196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 0x7fa538196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55330/0 (socket says 172.21.15.31:55330) 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa53659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa538100320 0x7fa5381961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.093+0000 7fa535d9b700 1 -- 172.21.15.31:0/1076906677 learned_addr learned my addr 172.21.15.31:0/1076906677 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa536d9d700 1 -- 172.21.15.31:0/1076906677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa538100320 msgr2=0x7fa5381961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa536d9d700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa538100320 0x7fa5381961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa536d9d700 1 -- 172.21.15.31:0/1076906677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 msgr2=0x7fa538196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa536d9d700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 0x7fa538196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa536d9d700 1 -- 172.21.15.31:0/1076906677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530009c40 con 0x7fa538101ea0 2023-07-29T16:15:44.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 0x7fa538196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:44.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa536d9d700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa53819d770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa530007930 tx=0x7fa530004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:44.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa5277fe700 1 -- 172.21.15.31:0/1076906677 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53000e040 con 0x7fa538101ea0 2023-07-29T16:15:44.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa5277fe700 1 -- 172.21.15.31:0/1076906677 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa530008ef0 con 0x7fa538101ea0 2023-07-29T16:15:44.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53819e290 con 0x7fa538101ea0 2023-07-29T16:15:44.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa5277fe700 1 -- 172.21.15.31:0/1076906677 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300052a0 con 0x7fa538101ea0 2023-07-29T16:15:44.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.094+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53819e6c0 con 0x7fa538101ea0 2023-07-29T16:15:44.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.098+0000 7fa5277fe700 1 -- 172.21.15.31:0/1076906677 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa53001d470 con 0x7fa538101ea0 2023-07-29T16:15:44.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.099+0000 7fa5503a9b80 1 Processor -- start 2023-07-29T16:15:44.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 -- start start 2023-07-29T16:15:44.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d594400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d595170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5277fe700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5200a0790 0x7fa5200a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d595b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d4e7a10 con 0x561e3d595700 2023-07-29T16:15:44.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d4e53d0 con 0x561e3d593ff0 2023-07-29T16:15:44.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d4e0cc0 con 0x561e3d594940 2023-07-29T16:15:44.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa5277fe700 1 -- 172.21.15.31:0/1076906677 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa5300ccfe0 con 0x7fa538101ea0 2023-07-29T16:15:44.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa53659c700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5200a0790 0x7fa5200a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d594400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa536d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d595b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d594400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55340/0 (socket says 172.21.15.31:55340) 2023-07-29T16:15:44.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa535d9b700 1 -- 172.21.15.31:0/3973069952 learned_addr learned my addr 172.21.15.31:0/3973069952 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:44.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.100+0000 7fa535d9b700 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d595170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa536d9d700 1 -- 172.21.15.31:0/3973069952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 msgr2=0x561e3d595170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa536d9d700 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d595170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa536d9d700 1 -- 172.21.15.31:0/3973069952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 msgr2=0x561e3d594400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa536d9d700 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d594400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa536d9d700 1 -- 172.21.15.31:0/3973069952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e3d30ede0 con 0x561e3d595700 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa535d9b700 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d594400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:44.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa536d9d700 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d595b50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa5300caa40 tx=0x7fa53001db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286338827299c9f3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa53659c700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5200a0790 0x7fa5200a2c40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa538101370 tx=0x7fa528009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa524ff9700 1 -- 172.21.15.31:0/3973069952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53009a270 con 0x561e3d595700 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa524ff9700 1 -- 172.21.15.31:0/3973069952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa53009a090 con 0x561e3d595700 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.101+0000 7fa524ff9700 1 -- 172.21.15.31:0/3973069952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53009e760 con 0x561e3d595700 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 -- 172.21.15.31:0/3973069952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 msgr2=0x561e3d595b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d595b50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa5300caa40 tx=0x7fa53001db10 comp rx=0 tx=0).stop 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 -- 172.21.15.31:0/3973069952 shutdown_connections 2023-07-29T16:15:44.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d595b50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d594400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/3973069952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d595170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 -- 172.21.15.31:0/3973069952 >> 172.21.15.31:0/3973069952 conn(0x561e3d5853e0 msgr2=0x561e3d58a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 -- 172.21.15.31:0/3973069952 shutdown_connections 2023-07-29T16:15:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.102+0000 7fa5503a9b80 1 -- 172.21.15.31:0/3973069952 wait complete. 2023-07-29T16:15:44.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.103+0000 7fa5503a9b80 1 -- start start 2023-07-29T16:15:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.103+0000 7fa5503a9b80 1 Processor -- start 2023-07-29T16:15:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d593ff0 0x561e3d59da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 0x561e3d59df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d595700 0x561e3d5a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d47d510 con 0x561e3d594940 2023-07-29T16:15:44.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d47d1b0 con 0x561e3d595700 2023-07-29T16:15:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d364920 con 0x561e3d593ff0 2023-07-29T16:15:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa5503a9b80 10 client.0 _ll_register_callbacks cb 0x561e3d486620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 0x561e3d59df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa53659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d593ff0 0x561e3d59da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 0x561e3d59df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45530/0 (socket says 172.21.15.31:45530) 2023-07-29T16:15:44.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 -- 172.21.15.31:0/1821203514 learned_addr learned my addr 172.21.15.31:0/1821203514 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa536d9d700 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d595700 0x561e3d5a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 -- 172.21.15.31:0/1821203514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d593ff0 msgr2=0x561e3d59da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d593ff0 0x561e3d59da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 -- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d595700 msgr2=0x561e3d5a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d595700 0x561e3d5a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.104+0000 7fa535d9b700 1 -- 172.21.15.31:0/1821203514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c019040 con 0x561e3d594940 2023-07-29T16:15:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 0x561e3d59df70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009320 tx=0x7fa52c004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180acae578f27127 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa51dffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:44.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa51dffb700 1 -- 172.21.15.31:0/1821203514 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c011020 con 0x561e3d594940 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa51dffb700 1 -- 172.21.15.31:0/1821203514 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa52c011180 con 0x561e3d594940 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e3d50bd10 con 0x561e3d594940 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa51dffb700 1 -- 172.21.15.31:0/1821203514 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa52c003a90 con 0x561e3d594940 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa5503a9b80 10 client.15765 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fa53819e6c0 con 0x561e3d594940 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.105+0000 7fa51dffb700 1 -- 172.21.15.31:0/1821203514 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561e3d50bd10 con 0x561e3d594940 2023-07-29T16:15:44.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa51dffb700 1 -- 172.21.15.31:0/1821203514 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa52c026070 con 0x561e3d594940 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa5503a9b80 10 client.15765 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa5503a9b80 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa5503a9b80 20 client.15765 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5a9890 0x561e3d5abca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa5503a9b80 4 client.15765 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e3d55ed70 con 0x561e3d5a9890 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.106+0000 7fa53659c700 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5a9890 0x561e3d5abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.107+0000 7fa53659c700 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5a9890 0x561e3d5abca0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d923c958146b5c2 server_cookie=7616afe0be90bb6a in_seq=0 out_seq=0 2023-07-29T16:15:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.107+0000 7fa51dffb700 10 client.15765 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.107+0000 7fa51dffb700 1 -- 172.21.15.31:0/1821203514 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561e3d55ed70 con 0x561e3d5a9890 2023-07-29T16:15:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.107+0000 7fa51dffb700 10 client.15765 handle_command_reply: tid=0 2023-07-29T16:15:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 client.15765 shutdown 2023-07-29T16:15:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 msgr2=0x561e3d59df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 0x561e3d59df70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa52c009320 tx=0x7fa52c004320 comp rx=0 tx=0).stop 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 shutdown_connections 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5a9890 0x561e3d5abca0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d595700 0x561e3d5a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d593ff0 0x561e3d59da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1821203514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d594940 0x561e3d59df70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa51dffb700 0 client.15765 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 >> 172.21.15.31:0/1821203514 conn(0x561e3d5853e0 msgr2=0x561e3d58a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:44.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 shutdown_connections 2023-07-29T16:15:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1821203514 wait complete. 2023-07-29T16:15:44.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.108+0000 7fa5503a9b80 20 client.15765 trim_cache size 0 max 16384 2023-07-29T16:15:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.121+0000 7fa5503a9b80 1 Processor -- start 2023-07-29T16:15:44.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.121+0000 7fa5503a9b80 1 -- start start 2023-07-29T16:15:44.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 0x561e3d59c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d594940 0x561e3d5a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d595700 0x561e3d58a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d59ec20 con 0x561e3d593ff0 2023-07-29T16:15:44.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d59cee0 con 0x561e3d594940 2023-07-29T16:15:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d59d5d0 con 0x561e3d595700 2023-07-29T16:15:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa53659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 0x561e3d59c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa535d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d594940 0x561e3d5a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa53659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 0x561e3d59c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45546/0 (socket says 172.21.15.31:45546) 2023-07-29T16:15:44.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa53659c700 1 -- 172.21.15.31:0/2578419404 learned_addr learned my addr 172.21.15.31:0/2578419404 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:44.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa536d9d700 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d595700 0x561e3d58a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa535d9b700 1 -- 172.21.15.31:0/2578419404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d595700 msgr2=0x561e3d58a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa535d9b700 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d595700 0x561e3d58a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa535d9b700 1 -- 172.21.15.31:0/2578419404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 msgr2=0x561e3d59c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa535d9b700 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 0x561e3d59c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa535d9b700 1 -- 172.21.15.31:0/2578419404 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e3d2f7c60 con 0x561e3d594940 2023-07-29T16:15:44.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.122+0000 7fa53659c700 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 0x561e3d59c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.123+0000 7fa535d9b700 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d594940 0x561e3d5a4f70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa52c005a80 tx=0x7fa52c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7de05a4a3148f06 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.123+0000 7fa5257fa700 1 -- 172.21.15.31:0/2578419404 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c005190 con 0x561e3d594940 2023-07-29T16:15:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.123+0000 7fa5257fa700 1 -- 172.21.15.31:0/2578419404 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa52c0052f0 con 0x561e3d594940 2023-07-29T16:15:44.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.123+0000 7fa5257fa700 1 -- 172.21.15.31:0/2578419404 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52c004cd0 con 0x561e3d594940 2023-07-29T16:15:44.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 -- 172.21.15.31:0/2578419404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d594940 msgr2=0x561e3d5a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d594940 0x561e3d5a4f70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa52c005a80 tx=0x7fa52c011900 comp rx=0 tx=0).stop 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 -- 172.21.15.31:0/2578419404 shutdown_connections 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d595700 0x561e3d58a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d593ff0 0x561e3d59c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/2578419404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d594940 0x561e3d5a4f70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 -- 172.21.15.31:0/2578419404 >> 172.21.15.31:0/2578419404 conn(0x561e3d5853e0 msgr2=0x561e3d58a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 -- 172.21.15.31:0/2578419404 shutdown_connections 2023-07-29T16:15:44.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 -- 172.21.15.31:0/2578419404 wait complete. 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 -- start start 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.124+0000 7fa5503a9b80 1 Processor -- start 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d5a4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d5a1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d5a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d5a21a0 con 0x561e3d595700 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d5a2310 con 0x561e3d593ff0 2023-07-29T16:15:44.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3d5fb410 con 0x561e3d594940 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa5503a9b80 10 client.0 _ll_register_callbacks cb 0x561e3d486620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d5a4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d5a4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55360/0 (socket says 172.21.15.31:55360) 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 -- 172.21.15.31:0/1006102464 learned_addr learned my addr 172.21.15.31:0/1006102464 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa536d9d700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d5a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d5a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 -- 172.21.15.31:0/1006102464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 msgr2=0x561e3d5a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d5a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 -- 172.21.15.31:0/1006102464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 msgr2=0x561e3d5a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d5a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.125+0000 7fa53659c700 1 -- 172.21.15.31:0/1006102464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa528017070 con 0x561e3d593ff0 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa53659c700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d5a4c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa528005fd0 tx=0x7fa5280047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b65bfc6bcc9cb7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa51dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:44.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa51dffb700 1 -- 172.21.15.31:0/1006102464 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528022040 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa53009a090 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa51dffb700 1 -- 172.21.15.31:0/1006102464 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa52801da60 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa51dffb700 1 -- 172.21.15.31:0/1006102464 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa528028470 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa5503a9b80 10 client.25748 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fa53819e6c0 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.126+0000 7fa51dffb700 1 -- 172.21.15.31:0/1006102464 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa528040080 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa51dffb700 1 -- 172.21.15.31:0/1006102464 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa52803cd20 con 0x561e3d593ff0 2023-07-29T16:15:44.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa5503a9b80 10 client.25748 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa5503a9b80 10 client.25748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa5503a9b80 20 client.25748 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5fcdc0 0x561e3d5ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa5503a9b80 4 client.25748 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:15:44.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561e3d4d9d40 con 0x561e3d5fcdc0 2023-07-29T16:15:44.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5fcdc0 0x561e3d5ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:44.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa535d9b700 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5fcdc0 0x561e3d5ff1d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3082cbf4f466739 server_cookie=bc42e9a328e90210 in_seq=0 out_seq=0 2023-07-29T16:15:44.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.127+0000 7fa51dffb700 10 client.25748 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa51dffb700 1 -- 172.21.15.31:0/1006102464 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561e3d4d9d40 con 0x561e3d5fcdc0 2023-07-29T16:15:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa51dffb700 10 client.25748 handle_command_reply: tid=0 2023-07-29T16:15:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 client.25748 shutdown 2023-07-29T16:15:44.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 msgr2=0x561e3d5a4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d5a4c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa528005fd0 tx=0x7fa5280047c0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 shutdown_connections 2023-07-29T16:15:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561e3d5fcdc0 0x561e3d5ff1d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e3d595700 0x561e3d5a1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e3d593ff0 0x561e3d5a4c80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa51dffb700 0 client.25748 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 --2- 172.21.15.31:0/1006102464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e3d594940 0x561e3d5a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 >> 172.21.15.31:0/1006102464 conn(0x561e3d5853e0 msgr2=0x561e3d59bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 shutdown_connections 2023-07-29T16:15:44.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 1 -- 172.21.15.31:0/1006102464 wait complete. 2023-07-29T16:15:44.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.128+0000 7fa5503a9b80 20 client.25748 trim_cache size 0 max 16384 2023-07-29T16:15:44.139 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:15:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5200a0790 msgr2=0x7fa5200a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5200a0790 0x7fa5200a2c40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa538101370 tx=0x7fa528009310 comp rx=0 tx=0).stop 2023-07-29T16:15:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 msgr2=0x7fa53819d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa53819d770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa530007930 tx=0x7fa530004c80 comp rx=0 tx=0).stop 2023-07-29T16:15:44.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 shutdown_connections 2023-07-29T16:15:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5200a0790 0x7fa5200a2c40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa538101ea0 0x7fa53819d770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa538100320 0x7fa5381961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 --2- 172.21.15.31:0/1076906677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa538101510 0x7fa538196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 >> 172.21.15.31:0/1076906677 conn(0x7fa5380fb920 msgr2=0x7fa5381049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:44.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 shutdown_connections 2023-07-29T16:15:44.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:44.129+0000 7fa53c9ff700 1 -- 172.21.15.31:0/1076906677 wait complete. 2023-07-29T16:15:44.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:44 smithi031 ceph-mon[95504]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:45.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:44 smithi039 ceph-mon[128858]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:45.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:44 smithi156 ceph-mon[129245]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:15:45.145 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:15:45.146 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:15:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 -- 172.21.15.31:0/2773895688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818100320 msgr2=0x7f4818100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2773895688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818100320 0x7f4818100730 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f480c009990 tx=0x7f480c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 -- 172.21.15.31:0/2773895688 shutdown_connections 2023-07-29T16:15:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2773895688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101ea0 0x7f4818108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2773895688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818100320 0x7f4818100730 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2773895688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818101510 0x7f4818101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.359+0000 7f481f1ff700 1 -- 172.21.15.31:0/2773895688 >> 172.21.15.31:0/2773895688 conn(0x7f48180fb920 msgr2=0x7f48180fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.360+0000 7f481f1ff700 1 -- 172.21.15.31:0/2773895688 shutdown_connections 2023-07-29T16:15:45.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.360+0000 7f481f1ff700 1 -- 172.21.15.31:0/2773895688 wait complete. 2023-07-29T16:15:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 Processor -- start 2023-07-29T16:15:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 -- start start 2023-07-29T16:15:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818100320 0x7f48181982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101510 0x7f48181987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 0x7f481819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481819fe30 con 0x7f4818101ea0 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481819ff70 con 0x7f4818100320 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481f1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48181a0100 con 0x7f4818101510 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 0x7f481819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 0x7f481819f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45576/0 (socket says 172.21.15.31:45576) 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481d79c700 1 -- 172.21.15.31:0/2798494564 learned_addr learned my addr 172.21.15.31:0/2798494564 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f4817fff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101510 0x7f48181987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.361+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818100320 0x7f48181982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481d79c700 1 -- 172.21.15.31:0/2798494564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101510 msgr2=0x7f48181987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481d79c700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101510 0x7f48181987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481d79c700 1 -- 172.21.15.31:0/2798494564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818100320 msgr2=0x7f48181982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481d79c700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818100320 0x7f48181982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481d79c700 1 -- 172.21.15.31:0/2798494564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c0095f0 con 0x7f4818101ea0 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818100320 0x7f48181982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f4817fff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101510 0x7f48181987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481d79c700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 0x7f481819f860 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4810007ee0 tx=0x7f481000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f4815ffb700 1 -- 172.21.15.31:0/2798494564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810017070 con 0x7f4818101ea0 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48181a03e0 con 0x7f4818101ea0 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f4815ffb700 1 -- 172.21.15.31:0/2798494564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f481000f040 con 0x7f4818101ea0 2023-07-29T16:15:45.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f4815ffb700 1 -- 172.21.15.31:0/2798494564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810005740 con 0x7f4818101ea0 2023-07-29T16:15:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.362+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48181a0940 con 0x7f4818101ea0 2023-07-29T16:15:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.366+0000 7f4815ffb700 1 -- 172.21.15.31:0/2798494564 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f48100058a0 con 0x7f4818101ea0 2023-07-29T16:15:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 Processor -- start 2023-07-29T16:15:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4815ffb700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48080a0790 0x7f48080a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 -- start start 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4815ffb700 1 -- 172.21.15.31:0/2798494564 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f48100cb2d0 con 0x7f4818101ea0 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4817fff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48080a0790 0x7f48080a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b65b0 0x5637af1b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af10da90 con 0x5637af1b7370 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af10b450 con 0x5637af1b5c60 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.367+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af106d40 con 0x5637af1b65b0 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45582/0 (socket says 172.21.15.31:45582) 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 -- 172.21.15.31:0/1111734574 learned_addr learned my addr 172.21.15.31:0/1111734574 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481cf9b700 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f4817fff700 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b65b0 0x5637af1b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f4817fff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48080a0790 0x7f48080a2c40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4804005fd0 tx=0x7f480400f640 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 -- 172.21.15.31:0/1111734574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b65b0 msgr2=0x5637af1b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b65b0 0x5637af1b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 -- 172.21.15.31:0/1111734574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 msgr2=0x5637af1b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 -- 172.21.15.31:0/1111734574 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637aef34e60 con 0x5637af1b7370 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481cf9b700 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.368+0000 7f481d79c700 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b77c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4810098900 tx=0x7f4810004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8db8d3708f0af9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.369+0000 7f4802ffd700 1 -- 172.21.15.31:0/1111734574 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100a0dc0 con 0x5637af1b7370 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.369+0000 7f4802ffd700 1 -- 172.21.15.31:0/1111734574 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4810096ab0 con 0x5637af1b7370 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.369+0000 7f4802ffd700 1 -- 172.21.15.31:0/1111734574 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4810098dd0 con 0x5637af1b7370 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.369+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1111734574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 msgr2=0x5637af1b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.369+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b77c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4810098900 tx=0x7f4810004b50 comp rx=0 tx=0).stop 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1111734574 shutdown_connections 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b77c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1111734574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b65b0 0x5637af1b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1111734574 >> 172.21.15.31:0/1111734574 conn(0x5637af1ab460 msgr2=0x5637af1b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1111734574 shutdown_connections 2023-07-29T16:15:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1111734574 wait complete. 2023-07-29T16:15:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 -- start start 2023-07-29T16:15:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.370+0000 7f4832ae3b80 1 Processor -- start 2023-07-29T16:15:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1bf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b65b0 0x5637af1bfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af1c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af0a3590 con 0x5637af1b65b0 2023-07-29T16:15:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af0a3230 con 0x5637af1b5c60 2023-07-29T16:15:45.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637aef8a9a0 con 0x5637af1b7370 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4832ae3b80 10 client.0 _ll_register_callbacks cb 0x5637af0ac6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4817fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b65b0 0x5637af1bfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f481cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1bf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f481cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1bf8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55388/0 (socket says 172.21.15.31:55388) 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f4817fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b65b0 0x5637af1bfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45596/0 (socket says 172.21.15.31:45596) 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f481cf9b700 1 -- 172.21.15.31:0/2538712468 learned_addr learned my addr 172.21.15.31:0/2538712468 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.371+0000 7f481d79c700 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af1c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f481cf9b700 1 -- 172.21.15.31:0/2538712468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 msgr2=0x5637af1c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af1c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f481cf9b700 1 -- 172.21.15.31:0/2538712468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b65b0 msgr2=0x5637af1bfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b65b0 0x5637af1bfe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f481cf9b700 1 -- 172.21.15.31:0/2538712468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4818109090 con 0x5637af1b5c60 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1bf8c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4818101370 tx=0x7f480c005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e60d80f068e61522 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f47f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f47f7fff700 1 -- 172.21.15.31:0/2538712468 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c00bc00 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637af131d90 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.372+0000 7f47f7fff700 1 -- 172.21.15.31:0/2538712468 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f480c00bd60 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f47f7fff700 1 -- 172.21.15.31:0/2538712468 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f480c004500 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 10 client.25760 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f48181a0940 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f47f7fff700 1 -- 172.21.15.31:0/2538712468 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5637af131d90 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f47f7fff700 1 -- 172.21.15.31:0/2538712468 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f480c003b60 con 0x5637af1b5c60 2023-07-29T16:15:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 10 client.25760 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 10 client.25760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 20 client.25760 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af1cb600 0x5637af1cda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 4 client.25760 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.373+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637af184df0 con 0x5637af1cb600 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.374+0000 7f4817fff700 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af1cb600 0x5637af1cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.374+0000 7f4817fff700 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af1cb600 0x5637af1cda10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a761432128b04f server_cookie=2eb16e1bcaa2db4a in_seq=0 out_seq=0 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.374+0000 7f47f7fff700 10 client.25760 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f47f7fff700 1 -- 172.21.15.31:0/2538712468 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5637af184df0 con 0x5637af1cb600 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f47f7fff700 10 client.25760 handle_command_reply: tid=0 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 client.25760 shutdown 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 msgr2=0x5637af1bf8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1bf8c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4818101370 tx=0x7f480c005800 comp rx=0 tx=0).stop 2023-07-29T16:15:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 shutdown_connections 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af1cb600 0x5637af1cda10 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af1c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b5c60 0x5637af1bf8c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f47f7fff700 0 client.25760 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2538712468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b65b0 0x5637af1bfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 >> 172.21.15.31:0/2538712468 conn(0x5637af1ab460 msgr2=0x5637af1b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 shutdown_connections 2023-07-29T16:15:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.375+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2538712468 wait complete. 2023-07-29T16:15:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.376+0000 7f4832ae3b80 20 client.25760 trim_cache size 0 max 16384 2023-07-29T16:15:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 Processor -- start 2023-07-29T16:15:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 -- start start 2023-07-29T16:15:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b5c60 0x5637af1bfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af1c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af1ad4e0 con 0x5637af1b7370 2023-07-29T16:15:45.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af1b69f0 con 0x5637af1b65b0 2023-07-29T16:15:45.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.393+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af1afab0 con 0x5637af1b5c60 2023-07-29T16:15:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481cf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b5c60 0x5637af1bfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45598/0 (socket says 172.21.15.31:45598) 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481cf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b5c60 0x5637af1bfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34736/0 (socket says 172.21.15.31:34736) 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 -- 172.21.15.31:0/1643283422 learned_addr learned my addr 172.21.15.31:0/1643283422 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f4817fff700 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af1c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 -- 172.21.15.31:0/1643283422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b5c60 msgr2=0x5637af1bfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b5c60 0x5637af1bfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 -- 172.21.15.31:0/1643283422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 msgr2=0x5637af1c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af1c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 -- 172.21.15.31:0/1643283422 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637aef1dce0 con 0x5637af1b7370 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.394+0000 7f481d79c700 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b00e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f48100a0890 tx=0x7f48100985e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2a5f4bfd04c5b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f48037fe700 1 -- 172.21.15.31:0/1643283422 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100cb0b0 con 0x5637af1b7370 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f48037fe700 1 -- 172.21.15.31:0/1643283422 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48100a0dc0 con 0x5637af1b7370 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f48037fe700 1 -- 172.21.15.31:0/1643283422 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48100980c0 con 0x5637af1b7370 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1643283422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 msgr2=0x5637af1b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b00e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f48100a0890 tx=0x7f48100985e0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1643283422 shutdown_connections 2023-07-29T16:15:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b7370 0x5637af1b00e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b5c60 0x5637af1bfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/1643283422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af1c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.395+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1643283422 >> 172.21.15.31:0/1643283422 conn(0x5637af1ab460 msgr2=0x5637af1bf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.396+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1643283422 shutdown_connections 2023-07-29T16:15:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.396+0000 7f4832ae3b80 1 -- 172.21.15.31:0/1643283422 wait complete. 2023-07-29T16:15:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.396+0000 7f4832ae3b80 1 -- start start 2023-07-29T16:15:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.396+0000 7f4832ae3b80 1 Processor -- start 2023-07-29T16:15:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 0x5637af219160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af21b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af21dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af21e3e0 con 0x5637af1b5c60 2023-07-29T16:15:45.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af21e550 con 0x5637af1b65b0 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637af21e710 con 0x5637af1b7370 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4832ae3b80 10 client.0 _ll_register_callbacks cb 0x5637af0ac6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 0x5637af219160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 0x5637af219160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45614/0 (socket says 172.21.15.31:45614) 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 -- 172.21.15.31:0/2681840928 learned_addr learned my addr 172.21.15.31:0/2681840928 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f4817fff700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af21b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481d79c700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af21dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 -- 172.21.15.31:0/2681840928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 msgr2=0x5637af21dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af21dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 -- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 msgr2=0x5637af21b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af21b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481cf9b700 1 -- 172.21.15.31:0/2681840928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637aef34e60 con 0x5637af1b5c60 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.397+0000 7f481d79c700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af21dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f481cf9b700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 0x5637af219160 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f480c005a80 tx=0x7f480c00bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6fc4872fe0399b9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f47f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f47f7fff700 1 -- 172.21.15.31:0/2681840928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c009ee0 con 0x5637af1b5c60 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f47f7fff700 1 -- 172.21.15.31:0/2681840928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f480c000e40 con 0x5637af1b5c60 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4810096ab0 con 0x5637af1b5c60 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f47f7fff700 1 -- 172.21.15.31:0/2681840928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f480c004060 con 0x5637af1b5c60 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f4832ae3b80 10 client.15789 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5637aef5b2d0 con 0x5637af1b5c60 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.398+0000 7f47f7fff700 1 -- 172.21.15.31:0/2681840928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f480c044020 con 0x5637af1b5c60 2023-07-29T16:15:45.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f47f7fff700 1 -- 172.21.15.31:0/2681840928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f480c024620 con 0x5637af1b5c60 2023-07-29T16:15:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4832ae3b80 10 client.15789 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4832ae3b80 10 client.15789 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4832ae3b80 20 client.15789 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af21fc80 0x5637af222060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4832ae3b80 4 client.15789 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:15:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5637af0ffdc0 con 0x5637af21fc80 2023-07-29T16:15:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4817fff700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af21fc80 0x5637af222060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f4817fff700 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af21fc80 0x5637af222060 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c2516597d4e013 server_cookie=464e5e3e6ce246b in_seq=0 out_seq=0 2023-07-29T16:15:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.399+0000 7f47f7fff700 10 client.15789 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f47f7fff700 1 -- 172.21.15.31:0/2681840928 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5637af0ffdc0 con 0x5637af21fc80 2023-07-29T16:15:45.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f47f7fff700 10 client.15789 handle_command_reply: tid=0 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 client.15789 shutdown 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 msgr2=0x5637af219160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 0x5637af219160 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f480c005a80 tx=0x7f480c00bc30 comp rx=0 tx=0).stop 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 shutdown_connections 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637af21fc80 0x5637af222060 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637af1b7370 0x5637af21dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637af1b5c60 0x5637af219160 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f47f7fff700 0 client.15789 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 --2- 172.21.15.31:0/2681840928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637af1b65b0 0x5637af21b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.401+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 >> 172.21.15.31:0/2681840928 conn(0x5637af1ab460 msgr2=0x5637af1b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 shutdown_connections 2023-07-29T16:15:45.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f4832ae3b80 1 -- 172.21.15.31:0/2681840928 wait complete. 2023-07-29T16:15:45.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f4832ae3b80 20 client.15789 trim_cache size 0 max 16384 2023-07-29T16:15:45.408 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:15:45.408 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:15:45.409 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "c3115232-6eca-4d63-aa8d-4824d9c92f1e", 2023-07-29T16:15:45.409 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:15:45.409 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:15:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48080a0790 msgr2=0x7f48080a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48080a0790 0x7f48080a2c40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4804005fd0 tx=0x7f480400f640 comp rx=0 tx=0).stop 2023-07-29T16:15:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 msgr2=0x7f481819f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 0x7f481819f860 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4810007ee0 tx=0x7f481000d3b0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 shutdown_connections 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48080a0790 0x7f48080a2c40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4818101ea0 0x7f481819f860 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4818100320 0x7f48181982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 --2- 172.21.15.31:0/2798494564 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4818101510 0x7f48181987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.402+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 >> 172.21.15.31:0/2798494564 conn(0x7f48180fb920 msgr2=0x7f48180fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.403+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 shutdown_connections 2023-07-29T16:15:45.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.403+0000 7f481f1ff700 1 -- 172.21.15.31:0/2798494564 wait complete. 2023-07-29T16:15:45.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:15:45.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 -- 172.21.15.31:0/957611379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614100320 msgr2=0x7fe614100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/957611379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614100320 0x7fe614100730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe610009990 tx=0x7fe610009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 -- 172.21.15.31:0/957611379 shutdown_connections 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/957611379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe614101ea0 0x7fe614108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/957611379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614100320 0x7fe614100730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/957611379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614101510 0x7fe614101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 -- 172.21.15.31:0/957611379 >> 172.21.15.31:0/957611379 conn(0x7fe6140fb920 msgr2=0x7fe6140fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 -- 172.21.15.31:0/957611379 shutdown_connections 2023-07-29T16:15:45.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.633+0000 7fe61c7ff700 1 -- 172.21.15.31:0/957611379 wait complete. 2023-07-29T16:15:45.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.634+0000 7fe61c7ff700 1 Processor -- start 2023-07-29T16:15:45.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 -- start start 2023-07-29T16:15:45.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 0x7fe6141982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614101510 0x7fe6141987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe614101ea0 0x7fe61419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61419fe30 con 0x7fe614101ea0 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61419ff70 con 0x7fe614100320 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61c7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6141a0100 con 0x7fe614101510 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 0x7fe6141982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614101510 0x7fe6141987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 0x7fe6141982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55430/0 (socket says 172.21.15.31:55430) 2023-07-29T16:15:45.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61a59b700 1 -- 172.21.15.31:0/3745331504 learned_addr learned my addr 172.21.15.31:0/3745331504 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.635+0000 7fe61ad9c700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe614101ea0 0x7fe61419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61a59b700 1 -- 172.21.15.31:0/3745331504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614101510 msgr2=0x7fe6141987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61a59b700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614101510 0x7fe6141987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61a59b700 1 -- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe614101ea0 msgr2=0x7fe61419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61a59b700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe614101ea0 0x7fe61419f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61a59b700 1 -- 172.21.15.31:0/3745331504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100095f0 con 0x7fe614100320 2023-07-29T16:15:45.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61a59b700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 0x7fe6141982b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe610005720 tx=0x7fe610009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe60b7fe700 1 -- 172.21.15.31:0/3745331504 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61001d070 con 0x7fe614100320 2023-07-29T16:15:45.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6141a0380 con 0x7fe614100320 2023-07-29T16:15:45.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe60b7fe700 1 -- 172.21.15.31:0/3745331504 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe610021410 con 0x7fe614100320 2023-07-29T16:15:45.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe60b7fe700 1 -- 172.21.15.31:0/3745331504 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610017650 con 0x7fe614100320 2023-07-29T16:15:45.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.636+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6141a0800 con 0x7fe614100320 2023-07-29T16:15:45.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.639+0000 7fe60b7fe700 1 -- 172.21.15.31:0/3745331504 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe610021580 con 0x7fe614100320 2023-07-29T16:15:45.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.641+0000 7fe6301c3b80 1 Processor -- start 2023-07-29T16:15:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.641+0000 7fe60b7fe700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6000a0790 0x7fe6000a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.641+0000 7fe6301c3b80 1 -- start start 2023-07-29T16:15:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.641+0000 7fe60b7fe700 1 -- 172.21.15.31:0/3745331504 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe6100ccda0 con 0x7fe614100320 2023-07-29T16:15:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.641+0000 7fe61ad9c700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6000a0790 0x7fe6000a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f5bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f5cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f5d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7eb3a10 con 0x5594c7f5c530 2023-07-29T16:15:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7eb13d0 con 0x5594c7f5d2f0 2023-07-29T16:15:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7eaccc0 con 0x5594c7f5bbe0 2023-07-29T16:15:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe61ad9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f5d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f5cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe61a59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f5bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f5cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45636/0 (socket says 172.21.15.31:45636) 2023-07-29T16:15:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 -- 172.21.15.31:0/348625699 learned_addr learned my addr 172.21.15.31:0/348625699 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe61ad9c700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6000a0790 0x7fe6000a2c40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe60c00f590 tx=0x7fe60c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 -- 172.21.15.31:0/348625699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 msgr2=0x5594c7f5bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f5bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 -- 172.21.15.31:0/348625699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 msgr2=0x5594c7f5d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f5d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.642+0000 7fe619d9a700 1 -- 172.21.15.31:0/348625699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594c7cdade0 con 0x5594c7f5c530 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.643+0000 7fe61ad9c700 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f5d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.643+0000 7fe619d9a700 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f5cd60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe604009d80 tx=0x7fe604011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff3fc5096a6d208 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.643+0000 7fe608ff9700 1 -- 172.21.15.31:0/348625699 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604004230 con 0x5594c7f5c530 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.643+0000 7fe608ff9700 1 -- 172.21.15.31:0/348625699 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe604004390 con 0x5594c7f5c530 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.643+0000 7fe608ff9700 1 -- 172.21.15.31:0/348625699 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604019840 con 0x5594c7f5c530 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 -- 172.21.15.31:0/348625699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 msgr2=0x5594c7f5cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f5cd60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe604009d80 tx=0x7fe604011720 comp rx=0 tx=0).stop 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 -- 172.21.15.31:0/348625699 shutdown_connections 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f5d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f5bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/348625699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f5cd60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 -- 172.21.15.31:0/348625699 >> 172.21.15.31:0/348625699 conn(0x5594c7f513e0 msgr2=0x5594c7f56990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 -- 172.21.15.31:0/348625699 shutdown_connections 2023-07-29T16:15:45.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 -- 172.21.15.31:0/348625699 wait complete. 2023-07-29T16:15:45.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.644+0000 7fe6301c3b80 1 -- start start 2023-07-29T16:15:45.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.645+0000 7fe6301c3b80 1 Processor -- start 2023-07-29T16:15:45.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.645+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.645+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f65cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f6cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7e49510 con 0x5594c7f5c530 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7e491b0 con 0x5594c7f5d2f0 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7d30920 con 0x5594c7f5bbe0 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe6301c3b80 10 client.0 _ll_register_callbacks cb 0x5594c7e52620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f65cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f65cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45642/0 (socket says 172.21.15.31:45642) 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 -- 172.21.15.31:0/2329443676 learned_addr learned my addr 172.21.15.31:0/2329443676 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe61a59b700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe61ad9c700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f6cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 -- 172.21.15.31:0/2329443676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 msgr2=0x5594c7f657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 -- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 msgr2=0x5594c7f6cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f6cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.646+0000 7fe619d9a700 1 -- 172.21.15.31:0/2329443676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c017440 con 0x5594c7f5c530 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe619d9a700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f65cf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe604009770 tx=0x7fe604003930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e029a3e49aeb059f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe5fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe5fdffb700 1 -- 172.21.15.31:0/2329443676 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604005450 con 0x5594c7f5c530 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594c7ed7d10 con 0x5594c7f5c530 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe5fdffb700 1 -- 172.21.15.31:0/2329443676 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe604005ce0 con 0x5594c7f5c530 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe5fdffb700 1 -- 172.21.15.31:0/2329443676 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe604009ed0 con 0x5594c7f5c530 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe6301c3b80 10 client.15801 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fe6040099e0 con 0x5594c7f5c530 2023-07-29T16:15:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.647+0000 7fe5fdffb700 1 -- 172.21.15.31:0/2329443676 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5594c7ed7d10 con 0x5594c7f5c530 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe5fdffb700 1 -- 172.21.15.31:0/2329443676 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fe604021040 con 0x5594c7f5c530 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe6301c3b80 10 client.15801 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe6301c3b80 10 client.15801 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe6301c3b80 20 client.15801 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7f714b0 0x5594c7f738c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe6301c3b80 4 client.15801 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594c7f2ad70 con 0x5594c7f714b0 2023-07-29T16:15:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.648+0000 7fe61a59b700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7f714b0 0x5594c7f738c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.649+0000 7fe5fdffb700 10 client.15801 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.649+0000 7fe61a59b700 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7f714b0 0x5594c7f738c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c174781c87dc5d server_cookie=dd4a08d7d9f089ea in_seq=0 out_seq=0 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.649+0000 7fe5fdffb700 1 -- 172.21.15.31:0/2329443676 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5594c7f2ad70 con 0x5594c7f714b0 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.649+0000 7fe5fdffb700 10 client.15801 handle_command_reply: tid=0 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.649+0000 7fe6301c3b80 1 client.15801 shutdown 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 msgr2=0x5594c7f65cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f65cf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe604009770 tx=0x7fe604003930 comp rx=0 tx=0).stop 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 shutdown_connections 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7f714b0 0x5594c7f738c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe5fdffb700 0 client.15801 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5d2f0 0x5594c7f6cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2329443676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f65cf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 >> 172.21.15.31:0/2329443676 conn(0x5594c7f513e0 msgr2=0x5594c7f56060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 shutdown_connections 2023-07-29T16:15:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2329443676 wait complete. 2023-07-29T16:15:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.650+0000 7fe6301c3b80 20 client.15801 trim_cache size 0 max 16384 2023-07-29T16:15:45.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.670+0000 7fe6301c3b80 1 Processor -- start 2023-07-29T16:15:45.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.670+0000 7fe6301c3b80 1 -- start start 2023-07-29T16:15:45.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.670+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.670+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5c530 0x5594c7f6ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.670+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5d2f0 0x5594c7f60730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7f64c30 con 0x5594c7f5d2f0 2023-07-29T16:15:45.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7f5cb90 con 0x5594c7f5c530 2023-07-29T16:15:45.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7f58080 con 0x5594c7f5bbe0 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34792/0 (socket says 172.21.15.31:34792) 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 -- 172.21.15.31:0/2302846847 learned_addr learned my addr 172.21.15.31:0/2302846847 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61ad9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5d2f0 0x5594c7f60730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe619d9a700 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5c530 0x5594c7f6ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 -- 172.21.15.31:0/2302846847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5c530 msgr2=0x5594c7f6ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5c530 0x5594c7f6ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 -- 172.21.15.31:0/2302846847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5d2f0 msgr2=0x5594c7f60730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5d2f0 0x5594c7f60730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 -- 172.21.15.31:0/2302846847 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594c7cc3c60 con 0x5594c7f5bbe0 2023-07-29T16:15:45.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.671+0000 7fe61a59b700 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f601f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe610096f40 tx=0x7fe61009d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76b4b7bb4a43d0b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6097fa700 1 -- 172.21.15.31:0/2302846847 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6100a2890 con 0x5594c7f5bbe0 2023-07-29T16:15:45.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6097fa700 1 -- 172.21.15.31:0/2302846847 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6100af070 con 0x5594c7f5bbe0 2023-07-29T16:15:45.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6097fa700 1 -- 172.21.15.31:0/2302846847 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61009a730 con 0x5594c7f5bbe0 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2302846847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 msgr2=0x5594c7f601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f601f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe610096f40 tx=0x7fe61009d040 comp rx=0 tx=0).stop 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2302846847 shutdown_connections 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5d2f0 0x5594c7f60730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5bbe0 0x5594c7f601f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/2302846847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5c530 0x5594c7f6ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2302846847 >> 172.21.15.31:0/2302846847 conn(0x5594c7f513e0 msgr2=0x5594c7f63670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2302846847 shutdown_connections 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.672+0000 7fe6301c3b80 1 -- 172.21.15.31:0/2302846847 wait complete. 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 -- start start 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 Processor -- start 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5bbe0 0x5594c7f6c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f6cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5d2f0 0x5594c7f69220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7f69980 con 0x5594c7f5c530 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7f69af0 con 0x5594c7f5bbe0 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.673+0000 7fe6301c3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594c7f69cb0 con 0x5594c7f5d2f0 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe6301c3b80 10 client.0 _ll_register_callbacks cb 0x5594c7e52620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f6cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe61a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5bbe0 0x5594c7f6c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe61ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5d2f0 0x5594c7f69220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f6cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45654/0 (socket says 172.21.15.31:45654) 2023-07-29T16:15:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 -- 172.21.15.31:0/509379184 learned_addr learned my addr 172.21.15.31:0/509379184 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 -- 172.21.15.31:0/509379184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5d2f0 msgr2=0x5594c7f69220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5d2f0 0x5594c7f69220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 -- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5bbe0 msgr2=0x5594c7f6c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5bbe0 0x5594c7f6c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 -- 172.21.15.31:0/509379184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6141a0800 con 0x5594c7f5c530 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe61ad9c700 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5d2f0 0x5594c7f69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe619d9a700 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f6cc40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe604009de0 tx=0x7fe604005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae93926dfa995e0f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.674+0000 7fe5fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe5fdffb700 1 -- 172.21.15.31:0/509379184 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604019ca0 con 0x5594c7f5c530 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe5fdffb700 1 -- 172.21.15.31:0/509379184 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe604000e40 con 0x5594c7f5c530 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe604004390 con 0x5594c7f5c530 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe5fdffb700 1 -- 172.21.15.31:0/509379184 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe604005d50 con 0x5594c7f5c530 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe6301c3b80 10 client.15807 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5594c7d01250 con 0x5594c7f5c530 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe5fdffb700 1 -- 172.21.15.31:0/509379184 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe604004390 con 0x5594c7f5c530 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.675+0000 7fe5fdffb700 1 -- 172.21.15.31:0/509379184 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fe604021030 con 0x5594c7f5c530 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe6301c3b80 10 client.15807 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe6301c3b80 10 client.15807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe6301c3b80 20 client.15807 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7fc4220 0x5594c7fc6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe6301c3b80 4 client.15807 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:15:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5594c7ea5d40 con 0x5594c7fc4220 2023-07-29T16:15:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.676+0000 7fe61a59b700 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7fc4220 0x5594c7fc6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:45.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe61a59b700 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7fc4220 0x5594c7fc6600 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7de65a7e44e6ad74 server_cookie=eee6109fc9067570 in_seq=0 out_seq=0 2023-07-29T16:15:45.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe5fdffb700 10 client.15807 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe5fdffb700 1 -- 172.21.15.31:0/509379184 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5594c7ea5d40 con 0x5594c7fc4220 2023-07-29T16:15:45.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe5fdffb700 10 client.15807 handle_command_reply: tid=0 2023-07-29T16:15:45.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 client.15807 shutdown 2023-07-29T16:15:45.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 msgr2=0x5594c7f6cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f6cc40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe604009de0 tx=0x7fe604005a00 comp rx=0 tx=0).stop 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 shutdown_connections 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5594c7fc4220 0x5594c7fc6600 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5594c7f5d2f0 0x5594c7f69220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5594c7f5bbe0 0x5594c7f6c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe5fdffb700 0 client.15807 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:45.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 --2- 172.21.15.31:0/509379184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5594c7f5c530 0x5594c7f6cc40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 >> 172.21.15.31:0/509379184 conn(0x5594c7f513e0 msgr2=0x5594c7f601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 shutdown_connections 2023-07-29T16:15:45.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 1 -- 172.21.15.31:0/509379184 wait complete. 2023-07-29T16:15:45.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.677+0000 7fe6301c3b80 20 client.15807 trim_cache size 0 max 16384 2023-07-29T16:15:45.683 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:15:45.683 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: "c3115232-6eca-4d63-aa8d-4824d9c92f1e": { 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "c3115232-6eca-4d63-aa8d-4824d9c92f1e", 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:15:45.684 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:15:45.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6000a0790 msgr2=0x7fe6000a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6000a0790 0x7fe6000a2c40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe60c00f590 tx=0x7fe60c00b040 comp rx=0 tx=0).stop 2023-07-29T16:15:45.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 msgr2=0x7fe6141982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 0x7fe6141982b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe610005720 tx=0x7fe610009580 comp rx=0 tx=0).stop 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 shutdown_connections 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6000a0790 0x7fe6000a2c40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe614101ea0 0x7fe61419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe614100320 0x7fe6141982b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 --2- 172.21.15.31:0/3745331504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe614101510 0x7fe6141987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 >> 172.21.15.31:0/3745331504 conn(0x7fe6140fb920 msgr2=0x7fe614106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 shutdown_connections 2023-07-29T16:15:45.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:15:45.678+0000 7fe61c7ff700 1 -- 172.21.15.31:0/3745331504 wait complete. 2023-07-29T16:15:45.693 INFO:tasks.cephfs.filesystem:scrub status for tag:c3115232-6eca-4d63-aa8d-4824d9c92f1e - {'path': '/', 'tag': 'c3115232-6eca-4d63-aa8d-4824d9c92f1e', 'options': 'recursive,force'} 2023-07-29T16:15:46.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:46 smithi031 ceph-mon[95504]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:46.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:46 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:15:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:46 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:15:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:47.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:46 smithi039 ceph-mon[128858]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:47.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:46 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:15:47.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:47.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:46 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:15:47.024 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:47.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:46 smithi156 ceph-mon[129245]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:47.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:46 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:15:47.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:47.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:46 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:15:47.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:15:48.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:47 smithi156 ceph-mon[129245]: pgmap v379: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:48.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:47 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:15:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:47 smithi031 ceph-mon[95504]: pgmap v379: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:47 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:15:48.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:47 smithi039 ceph-mon[128858]: pgmap v379: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:15:48.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:47 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:15:50.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:50 smithi031 ceph-mon[95504]: pgmap v380: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:50 smithi039 ceph-mon[128858]: pgmap v380: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:50.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:50 smithi156 ceph-mon[129245]: pgmap v380: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:15:51.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 -- 172.21.15.31:0/1134990626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214100e40 msgr2=0x7f0214103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 --2- 172.21.15.31:0/1134990626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214100e40 0x7f0214103220 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0204009990 tx=0x7f0204009ca0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 -- 172.21.15.31:0/1134990626 shutdown_connections 2023-07-29T16:15:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 --2- 172.21.15.31:0/1134990626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0214106080 0x7f02141084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 --2- 172.21.15.31:0/1134990626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214100e40 0x7f0214103220 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 --2- 172.21.15.31:0/1134990626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214103760 0x7f0214105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 -- 172.21.15.31:0/1134990626 >> 172.21.15.31:0/1134990626 conn(0x7f02140fa900 msgr2=0x7f02140fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 -- 172.21.15.31:0/1134990626 shutdown_connections 2023-07-29T16:15:51.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.462+0000 7f0218fff700 1 -- 172.21.15.31:0/1134990626 wait complete. 2023-07-29T16:15:51.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.464+0000 7f0218fff700 1 Processor -- start 2023-07-29T16:15:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.464+0000 7f0218fff700 1 -- start start 2023-07-29T16:15:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.464+0000 7f0218fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0214100e40 0x7f0214198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0218fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 0x7f02141988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0218fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214106080 0x7f0214199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0218fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214199db0 con 0x7f0214103760 2023-07-29T16:15:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0218fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214199f20 con 0x7f0214100e40 2023-07-29T16:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0218fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021419a0e0 con 0x7f0214106080 2023-07-29T16:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 0x7f02141988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 0x7f02141988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49706/0 (socket says 172.21.15.31:49706) 2023-07-29T16:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 -- 172.21.15.31:0/147312968 learned_addr learned my addr 172.21.15.31:0/147312968 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f021259c700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0214100e40 0x7f0214198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0212d9d700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214106080 0x7f0214199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 -- 172.21.15.31:0/147312968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214106080 msgr2=0x7f0214199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214106080 0x7f0214199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 -- 172.21.15.31:0/147312968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0214100e40 msgr2=0x7f0214198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0214100e40 0x7f0214198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f020bfff700 1 -- 172.21.15.31:0/147312968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040095f0 con 0x7f0214103760 2023-07-29T16:15:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.465+0000 7f0212d9d700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214106080 0x7f0214199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.466+0000 7f020bfff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 0x7f02141988d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00d950 tx=0x7f01fc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.466+0000 7f020b7fe700 1 -- 172.21.15.31:0/147312968 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc009940 con 0x7f0214103760 2023-07-29T16:15:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.466+0000 7f020b7fe700 1 -- 172.21.15.31:0/147312968 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01fc010460 con 0x7f0214103760 2023-07-29T16:15:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.466+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021419a3c0 con 0x7f0214103760 2023-07-29T16:15:51.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.466+0000 7f020b7fe700 1 -- 172.21.15.31:0/147312968 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc00f620 con 0x7f0214103760 2023-07-29T16:15:51.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.466+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141a0b80 con 0x7f0214103760 2023-07-29T16:15:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.469+0000 7f020b7fe700 1 -- 172.21.15.31:0/147312968 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f01fc0108d0 con 0x7f0214103760 2023-07-29T16:15:51.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.470+0000 7f022c85eb80 1 Processor -- start 2023-07-29T16:15:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 -- start start 2023-07-29T16:15:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf5bc6b0 0x556abf7608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf7617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf761cf0 0x556abf762160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf6b89b0 con 0x556abf760df0 2023-07-29T16:15:51.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf6b6370 con 0x556abf761cf0 2023-07-29T16:15:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f022c85eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf6b1c60 con 0x556abf5bc6b0 2023-07-29T16:15:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f020b7fe700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02000a0810 0x7f02000a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f021259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf7617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f021259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf7617b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49714/0 (socket says 172.21.15.31:49714) 2023-07-29T16:15:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f021259c700 1 -- 172.21.15.31:0/3407009803 learned_addr learned my addr 172.21.15.31:0/3407009803 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:51.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f020b7fe700 1 -- 172.21.15.31:0/147312968 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f01fc0cb900 con 0x7f0214103760 2023-07-29T16:15:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f020bfff700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf761cf0 0x556abf762160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.471+0000 7f0212d9d700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf5bc6b0 0x556abf7608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f0212d9d700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02000a0810 0x7f02000a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f021259c700 1 -- 172.21.15.31:0/3407009803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf5bc6b0 msgr2=0x556abf7608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f021259c700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf5bc6b0 0x556abf7608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f021259c700 1 -- 172.21.15.31:0/3407009803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf761cf0 msgr2=0x556abf762160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f021259c700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf761cf0 0x556abf762160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f021259c700 1 -- 172.21.15.31:0/3407009803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556abf4dfd80 con 0x556abf760df0 2023-07-29T16:15:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f020bfff700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf761cf0 0x556abf762160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:15:51.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f0212d9d700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf5bc6b0 0x556abf7608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:15:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f021259c700 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf7617b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0204005f50 tx=0x7f0204009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95cb4fc7569a8b5d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f0212d9d700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02000a0810 0x7f02000a2cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f020c0076a0 tx=0x7f020c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f0208ff9700 1 -- 172.21.15.31:0/3407009803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020401d050 con 0x556abf760df0 2023-07-29T16:15:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f0208ff9700 1 -- 172.21.15.31:0/3407009803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f020401e040 con 0x556abf760df0 2023-07-29T16:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.472+0000 7f0208ff9700 1 -- 172.21.15.31:0/3407009803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040052e0 con 0x556abf760df0 2023-07-29T16:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 -- 172.21.15.31:0/3407009803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 msgr2=0x556abf7617b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf7617b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0204005f50 tx=0x7f0204009ee0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 -- 172.21.15.31:0/3407009803 shutdown_connections 2023-07-29T16:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf761cf0 0x556abf762160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf5bc6b0 0x556abf7608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 --2- 172.21.15.31:0/3407009803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf7617b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 -- 172.21.15.31:0/3407009803 >> 172.21.15.31:0/3407009803 conn(0x556abf755fd0 msgr2=0x556abf75b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:51.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 -- 172.21.15.31:0/3407009803 shutdown_connections 2023-07-29T16:15:51.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.473+0000 7f022c85eb80 1 -- 172.21.15.31:0/3407009803 wait complete. 2023-07-29T16:15:51.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.474+0000 7f022c85eb80 1 -- start start 2023-07-29T16:15:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.474+0000 7f022c85eb80 1 Processor -- start 2023-07-29T16:15:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf769e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf7713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf64e4b0 con 0x556abf760df0 2023-07-29T16:15:51.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf64e150 con 0x556abf5bc6b0 2023-07-29T16:15:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf5358c0 con 0x556abf761cf0 2023-07-29T16:15:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f022c85eb80 10 client.0 _ll_register_callbacks cb 0x556abf6575c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f0212d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf7713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f021259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf769e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f0212d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf7713b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59590/0 (socket says 172.21.15.31:59590) 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f0212d9d700 1 -- 172.21.15.31:0/1397229568 learned_addr learned my addr 172.21.15.31:0/1397229568 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49724/0 (socket says 172.21.15.31:49724) 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 -- 172.21.15.31:0/1397229568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 msgr2=0x556abf7713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf7713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 -- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 msgr2=0x556abf769e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf769e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.475+0000 7f020bfff700 1 -- 172.21.15.31:0/1397229568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141a0b80 con 0x556abf760df0 2023-07-29T16:15:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f020bfff700 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76a370 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f01fc095a10 tx=0x7f01fc098ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee62af69685a4b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f01f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:15:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1397229568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc099400 con 0x556abf760df0 2023-07-29T16:15:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1397229568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f01fc098fa0 con 0x556abf760df0 2023-07-29T16:15:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556abf6dccb0 con 0x556abf760df0 2023-07-29T16:15:51.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1397229568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01fc09cfd0 con 0x556abf760df0 2023-07-29T16:15:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f022c85eb80 10 client.15819 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f02141090d0 con 0x556abf760df0 2023-07-29T16:15:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1397229568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556abf6dccb0 con 0x556abf760df0 2023-07-29T16:15:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.476+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1397229568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f01fc0c8f20 con 0x556abf760df0 2023-07-29T16:15:51.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f022c85eb80 10 client.15819 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f022c85eb80 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f022c85eb80 20 client.15819 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf775bd0 0x556abf777fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f022c85eb80 4 client.15819 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556abf72fd10 con 0x556abf775bd0 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f021259c700 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf775bd0 0x556abf777fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f021259c700 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf775bd0 0x556abf777fe0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc57b39592431dde server_cookie=ff052a455c23a050 in_seq=0 out_seq=0 2023-07-29T16:15:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.477+0000 7f01f9ffb700 10 client.15819 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.478+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1397229568 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556abf72fd10 con 0x556abf775bd0 2023-07-29T16:15:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.478+0000 7f01f9ffb700 10 client.15819 handle_command_reply: tid=0 2023-07-29T16:15:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.478+0000 7f022c85eb80 1 client.15819 shutdown 2023-07-29T16:15:51.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 msgr2=0x556abf76a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76a370 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f01fc095a10 tx=0x7f01fc098ae0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 shutdown_connections 2023-07-29T16:15:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf775bd0 0x556abf777fe0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf7713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf769e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f01f9ffb700 0 client.15819 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1397229568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76a370 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 >> 172.21.15.31:0/1397229568 conn(0x556abf755fd0 msgr2=0x556abf75aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 shutdown_connections 2023-07-29T16:15:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 1 -- 172.21.15.31:0/1397229568 wait complete. 2023-07-29T16:15:51.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.479+0000 7f022c85eb80 20 client.15819 trim_cache size 0 max 16384 2023-07-29T16:15:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.494+0000 7f022c85eb80 1 Processor -- start 2023-07-29T16:15:51.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.494+0000 7f022c85eb80 1 -- start start 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.494+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.494+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 0x556abf764ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf761cf0 0x556abf7650e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f022c85eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf75cb30 con 0x556abf761cf0 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f022c85eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf761490 con 0x556abf5bc6b0 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f022c85eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf76ad00 con 0x556abf760df0 2023-07-29T16:15:51.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 0x556abf764ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 0x556abf764ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59604/0 (socket says 172.21.15.31:59604) 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 -- 172.21.15.31:0/1331074443 learned_addr learned my addr 172.21.15.31:0/1331074443 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f0212d9d700 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf761cf0 0x556abf7650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f021259c700 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 -- 172.21.15.31:0/1331074443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 msgr2=0x556abf76f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 -- 172.21.15.31:0/1331074443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf761cf0 msgr2=0x556abf7650e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf761cf0 0x556abf7650e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.495+0000 7f020bfff700 1 -- 172.21.15.31:0/1331074443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556abf4c8c00 con 0x556abf760df0 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f020bfff700 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 0x556abf764ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01fc098ca0 tx=0x7f01fc0c93c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd77f545e7332be5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f02097fa700 1 -- 172.21.15.31:0/1331074443 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc014070 con 0x556abf760df0 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f02097fa700 1 -- 172.21.15.31:0/1331074443 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01fc09cfd0 con 0x556abf760df0 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f02097fa700 1 -- 172.21.15.31:0/1331074443 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01fc099200 con 0x556abf760df0 2023-07-29T16:15:51.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f022c85eb80 1 -- 172.21.15.31:0/1331074443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 msgr2=0x556abf764ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 0x556abf764ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f01fc098ca0 tx=0x7f01fc0c93c0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.496+0000 7f022c85eb80 1 -- 172.21.15.31:0/1331074443 shutdown_connections 2023-07-29T16:15:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf761cf0 0x556abf7650e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1331074443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf760df0 0x556abf764ba0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 -- 172.21.15.31:0/1331074443 >> 172.21.15.31:0/1331074443 conn(0x556abf755fd0 msgr2=0x556abf768050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 -- 172.21.15.31:0/1331074443 shutdown_connections 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 -- 172.21.15.31:0/1331074443 wait complete. 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 -- start start 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.497+0000 7f022c85eb80 1 Processor -- start 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf76dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf76e290 con 0x556abf760df0 2023-07-29T16:15:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf76e400 con 0x556abf5bc6b0 2023-07-29T16:15:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556abf76e5c0 con 0x556abf761cf0 2023-07-29T16:15:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f022c85eb80 10 client.0 _ll_register_callbacks cb 0x556abf6575c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:15:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f020bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54552/0 (socket says 172.21.15.31:54552) 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 -- 172.21.15.31:0/1696758790 learned_addr learned my addr 172.21.15.31:0/1696758790 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f0212d9d700 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf76dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 -- 172.21.15.31:0/1696758790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 msgr2=0x556abf76dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf76dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 -- 172.21.15.31:0/1696758790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 msgr2=0x556abf76d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:15:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.498+0000 7f021259c700 1 -- 172.21.15.31:0/1696758790 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141090d0 con 0x556abf5bc6b0 2023-07-29T16:15:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f021259c700 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f0f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0204009f50 tx=0x7f0204005cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8efef722ce2cb757 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:15:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f01f9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:15:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1696758790 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020401d040 con 0x556abf5bc6b0 2023-07-29T16:15:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1696758790 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0204031160 con 0x556abf5bc6b0 2023-07-29T16:15:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f020401e040 con 0x556abf5bc6b0 2023-07-29T16:15:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1696758790 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0204031320 con 0x556abf5bc6b0 2023-07-29T16:15:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f022c85eb80 10 client.25802 fetch_fsmap learned FSMap version 23 2023-07-29T16:15:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556abf5061f0 con 0x556abf5bc6b0 2023-07-29T16:15:51.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.499+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1696758790 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f020401e040 con 0x556abf5bc6b0 2023-07-29T16:15:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1696758790 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0204005070 con 0x556abf5bc6b0 2023-07-29T16:15:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f022c85eb80 10 client.25802 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:15:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f022c85eb80 10 client.25802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:15:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f022c85eb80 20 client.25802 populate_metadata read hostname 'smithi031' 2023-07-29T16:15:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf7c8ad0 0x556abf7caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:15:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f022c85eb80 4 client.25802 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:15:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556abf6aace0 con 0x556abf7c8ad0 2023-07-29T16:15:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f020bfff700 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf7c8ad0 0x556abf7caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:15:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f020bfff700 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf7c8ad0 0x556abf7caee0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d99a6a38eb3393 server_cookie=c01a0f085d3cc5d9 in_seq=0 out_seq=0 2023-07-29T16:15:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.500+0000 7f01f9ffb700 10 client.25802 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f01f9ffb700 1 -- 172.21.15.31:0/1696758790 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556abf6aace0 con 0x556abf7c8ad0 2023-07-29T16:15:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f01f9ffb700 10 client.25802 handle_command_reply: tid=0 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f022c85eb80 1 client.25802 shutdown 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 msgr2=0x556abf76f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f0f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0204009f50 tx=0x7f0204005cb0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 shutdown_connections 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.504+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556abf7c8ad0 0x556abf7caee0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556abf761cf0 0x556abf76dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556abf5bc6b0 0x556abf76f0f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 1 --2- 172.21.15.31:0/1696758790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556abf760df0 0x556abf76d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f01f9ffb700 0 client.25802 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:15:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 >> 172.21.15.31:0/1696758790 conn(0x556abf755fd0 msgr2=0x556abf764ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 shutdown_connections 2023-07-29T16:15:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 1 -- 172.21.15.31:0/1696758790 wait complete. 2023-07-29T16:15:51.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f022c85eb80 20 client.25802 trim_cache size 0 max 16384 2023-07-29T16:15:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:15:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:15:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:15:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:15:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02000a0810 msgr2=0x7f02000a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02000a0810 0x7f02000a2cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f020c0076a0 tx=0x7f020c00b040 comp rx=0 tx=0).stop 2023-07-29T16:15:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 msgr2=0x7f02141988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:15:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 0x7f02141988d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f01fc00d950 tx=0x7f01fc00dc60 comp rx=0 tx=0).stop 2023-07-29T16:15:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 shutdown_connections 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02000a0810 0x7f02000a2cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0214106080 0x7f0214199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0214100e40 0x7f0214198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 --2- 172.21.15.31:0/147312968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0214103760 0x7f02141988d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.505+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 >> 172.21.15.31:0/147312968 conn(0x7f02140fa900 msgr2=0x7f02140ff4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.506+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 shutdown_connections 2023-07-29T16:15:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:15:51.506+0000 7f0218fff700 1 -- 172.21.15.31:0/147312968 wait complete. 2023-07-29T16:15:52.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:52 smithi031 ceph-mon[95504]: pgmap v381: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:52 smithi039 ceph-mon[128858]: pgmap v381: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:52.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:52 smithi156 ceph-mon[129245]: pgmap v381: 97 pgs: 97 active+clean; 664 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:54.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:54 smithi031 ceph-mon[95504]: pgmap v382: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:15:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:54 smithi039 ceph-mon[128858]: pgmap v382: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:15:54.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:54 smithi156 ceph-mon[129245]: pgmap v382: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:15:56.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:56 smithi031 ceph-mon[95504]: pgmap v383: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:56.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:56 smithi039 ceph-mon[128858]: pgmap v383: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:56.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:56 smithi156 ceph-mon[129245]: pgmap v383: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:58.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:15:57 smithi156 ceph-mon[129245]: pgmap v384: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:15:57 smithi031 ceph-mon[95504]: pgmap v384: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:15:58.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:15:57 smithi039 ceph-mon[128858]: pgmap v384: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:00.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:00 smithi031 ceph-mon[95504]: pgmap v385: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:00.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:00 smithi039 ceph-mon[128858]: pgmap v385: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:00.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:00 smithi156 ceph-mon[129245]: pgmap v385: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:02.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:02 smithi031 ceph-mon[95504]: pgmap v386: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:02 smithi039 ceph-mon[128858]: pgmap v386: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:02.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:02 smithi156 ceph-mon[129245]: pgmap v386: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 -- 172.21.15.31:0/3320756494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101540 msgr2=0x7fa548101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3320756494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101540 0x7fa548101990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa538009990 tx=0x7fa538009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 -- 172.21.15.31:0/3320756494 shutdown_connections 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3320756494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa548101ed0 0x7fa548108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3320756494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3320756494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101540 0x7fa548101990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 -- 172.21.15.31:0/3320756494 >> 172.21.15.31:0/3320756494 conn(0x7fa5480fb970 msgr2=0x7fa5480fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 -- 172.21.15.31:0/3320756494 shutdown_connections 2023-07-29T16:16:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.665+0000 7fa54ffff700 1 -- 172.21.15.31:0/3320756494 wait complete. 2023-07-29T16:16:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.667+0000 7fa54ffff700 1 Processor -- start 2023-07-29T16:16:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.667+0000 7fa54ffff700 1 -- start start 2023-07-29T16:16:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa548101540 0x7fa5481988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101ed0 0x7fa548199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54ffff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548199db0 con 0x7fa548101ed0 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54ffff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548199f20 con 0x7fa548100350 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54ffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54819a0e0 con 0x7fa548101540 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101ed0 0x7fa548199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47470/0 (socket says 172.21.15.31:47470) 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54dd9b700 1 -- 172.21.15.31:0/3200682157 learned_addr learned my addr 172.21.15.31:0/3200682157 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.668+0000 7fa54d59a700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa548101540 0x7fa5481988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54e59c700 1 -- 172.21.15.31:0/3200682157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa548101540 msgr2=0x7fa5481988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54e59c700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa548101540 0x7fa5481988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54e59c700 1 -- 172.21.15.31:0/3200682157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 msgr2=0x7fa548198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54e59c700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54e59c700 1 -- 172.21.15.31:0/3200682157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5380095f0 con 0x7fa548101ed0 2023-07-29T16:16:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54e59c700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101ed0 0x7fa548199750 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa54000efc0 tx=0x7fa54000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa53effd700 1 -- 172.21.15.31:0/3200682157 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540018070 con 0x7fa548101ed0 2023-07-29T16:16:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa53effd700 1 -- 172.21.15.31:0/3200682157 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54000f040 con 0x7fa548101ed0 2023-07-29T16:16:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa53effd700 1 -- 172.21.15.31:0/3200682157 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54000cd40 con 0x7fa548101ed0 2023-07-29T16:16:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54819a360 con 0x7fa548101ed0 2023-07-29T16:16:04.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.669+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5481a0b80 con 0x7fa548101ed0 2023-07-29T16:16:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.672+0000 7fa53effd700 1 -- 172.21.15.31:0/3200682157 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa540003a00 con 0x7fa548101ed0 2023-07-29T16:16:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.674+0000 7fa53effd700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5340a0810 0x7fa5340a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.674+0000 7fa563a16b80 1 Processor -- start 2023-07-29T16:16:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.674+0000 7fa53effd700 1 -- 172.21.15.31:0/3200682157 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa5400cb5d0 con 0x7fa548101ed0 2023-07-29T16:16:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.674+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5340a0810 0x7fa5340a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.674+0000 7fa563a16b80 1 -- start start 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.674+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b7818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b7827b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b783160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa563a16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b6d99b0 con 0x56259b5dd6b0 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa563a16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b6d7370 con 0x56259b781df0 2023-07-29T16:16:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa563a16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b6d2c60 con 0x56259b782cf0 2023-07-29T16:16:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54d59a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b7827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b7818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54d59a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b7827b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47476/0 (socket says 172.21.15.31:47476) 2023-07-29T16:16:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54d59a700 1 -- 172.21.15.31:0/3595280905 learned_addr learned my addr 172.21.15.31:0/3595280905 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54e59c700 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b783160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5340a0810 0x7fa5340a2cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa544005950 tx=0x7fa544009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 -- 172.21.15.31:0/3595280905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 msgr2=0x56259b783160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b783160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 -- 172.21.15.31:0/3595280905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 msgr2=0x56259b7827b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b7827b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54dd9b700 1 -- 172.21.15.31:0/3595280905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259b500d80 con 0x56259b5dd6b0 2023-07-29T16:16:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.675+0000 7fa54d59a700 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b7827b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.676+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b7818b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa54400e5c0 tx=0x7fa544014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399ba8e9ce6f5008 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.676+0000 7fa533fff700 1 -- 172.21.15.31:0/3595280905 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa544004220 con 0x56259b5dd6b0 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.676+0000 7fa533fff700 1 -- 172.21.15.31:0/3595280905 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa544021070 con 0x56259b5dd6b0 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.676+0000 7fa533fff700 1 -- 172.21.15.31:0/3595280905 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54401d670 con 0x56259b5dd6b0 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 -- 172.21.15.31:0/3595280905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 msgr2=0x56259b7818b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b7818b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa54400e5c0 tx=0x7fa544014000 comp rx=0 tx=0).stop 2023-07-29T16:16:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 -- 172.21.15.31:0/3595280905 shutdown_connections 2023-07-29T16:16:04.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b783160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b7818b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 --2- 172.21.15.31:0/3595280905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b7827b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 -- 172.21.15.31:0/3595280905 >> 172.21.15.31:0/3595280905 conn(0x56259b776fd0 msgr2=0x56259b77c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:04.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 -- 172.21.15.31:0/3595280905 shutdown_connections 2023-07-29T16:16:04.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 -- 172.21.15.31:0/3595280905 wait complete. 2023-07-29T16:16:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.677+0000 7fa563a16b80 1 -- start start 2023-07-29T16:16:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 Processor -- start 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b78af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b781df0 0x56259b78b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b782cf0 0x56259b792480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b66f4b0 con 0x56259b5dd6b0 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b66f150 con 0x56259b782cf0 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b5568c0 con 0x56259b781df0 2023-07-29T16:16:04.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa563a16b80 10 client.0 _ll_register_callbacks cb 0x56259b6785c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa54dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b78af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.678+0000 7fa54d59a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b781df0 0x56259b78b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54d59a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b781df0 0x56259b78b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37994/0 (socket says 172.21.15.31:37994) 2023-07-29T16:16:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54d59a700 1 -- 172.21.15.31:0/1606753605 learned_addr learned my addr 172.21.15.31:0/1606753605 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b782cf0 0x56259b792480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 -- 172.21.15.31:0/1606753605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b781df0 msgr2=0x56259b78b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b781df0 0x56259b78b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 -- 172.21.15.31:0/1606753605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 msgr2=0x56259b78af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b78af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 -- 172.21.15.31:0/1606753605 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5481090c0 con 0x56259b782cf0 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b782cf0 0x56259b792480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c6f0 tx=0x7fa540000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e1c2c93fef5f580 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa5317fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:04.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa5317fa700 1 -- 172.21.15.31:0/1606753605 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400b1460 con 0x56259b782cf0 2023-07-29T16:16:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56259b6fdcb0 con 0x56259b782cf0 2023-07-29T16:16:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.679+0000 7fa5317fa700 1 -- 172.21.15.31:0/1606753605 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa540004ad0 con 0x56259b782cf0 2023-07-29T16:16:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa5317fa700 1 -- 172.21.15.31:0/1606753605 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa540098d00 con 0x56259b782cf0 2023-07-29T16:16:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 10 client.25814 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fa544017070 con 0x56259b782cf0 2023-07-29T16:16:04.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa5317fa700 1 -- 172.21.15.31:0/1606753605 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56259b6fdcb0 con 0x56259b782cf0 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa5317fa700 1 -- 172.21.15.31:0/1606753605 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa5400a0c30 con 0x56259b782cf0 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 10 client.25814 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 10 client.25814 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 20 client.25814 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b796cf0 0x56259b799100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 4 client.25814 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:04.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.680+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56259b750d10 con 0x56259b796cf0 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.681+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b796cf0 0x56259b799100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.681+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b796cf0 0x56259b799100 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5933847f25e77f7 server_cookie=e9d364f4c3c80e28 in_seq=0 out_seq=0 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.681+0000 7fa5317fa700 10 client.25814 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa5317fa700 1 -- 172.21.15.31:0/1606753605 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56259b750d10 con 0x56259b796cf0 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa5317fa700 10 client.25814 handle_command_reply: tid=0 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 client.25814 shutdown 2023-07-29T16:16:04.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b782cf0 msgr2=0x56259b792480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b782cf0 0x56259b792480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa54000c6f0 tx=0x7fa540000f40 comp rx=0 tx=0).stop 2023-07-29T16:16:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 shutdown_connections 2023-07-29T16:16:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b796cf0 0x56259b799100 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b782cf0 0x56259b792480 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b5dd6b0 0x56259b78af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1606753605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b781df0 0x56259b78b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa5317fa700 0 client.25814 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 >> 172.21.15.31:0/1606753605 conn(0x56259b776fd0 msgr2=0x56259b77ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 shutdown_connections 2023-07-29T16:16:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 1 -- 172.21.15.31:0/1606753605 wait complete. 2023-07-29T16:16:04.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.682+0000 7fa563a16b80 20 client.25814 trim_cache size 0 max 16384 2023-07-29T16:16:04.692 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:04 smithi031 ceph-mon[95504]: pgmap v387: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 Processor -- start 2023-07-29T16:16:04.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 -- start start 2023-07-29T16:16:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b5dd6b0 0x56259b77ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b792440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 0x56259b7890f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b77d1c0 con 0x56259b782cf0 2023-07-29T16:16:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b7968a0 con 0x56259b781df0 2023-07-29T16:16:04.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa563a16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b7769d0 con 0x56259b5dd6b0 2023-07-29T16:16:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa54e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 0x56259b7890f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa54d59a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b792440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa54e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 0x56259b7890f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37498/0 (socket says 172.21.15.31:37498) 2023-07-29T16:16:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa54e59c700 1 -- 172.21.15.31:0/1895896297 learned_addr learned my addr 172.21.15.31:0/1895896297 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.699+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b5dd6b0 0x56259b77ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa54e59c700 1 -- 172.21.15.31:0/1895896297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b5dd6b0 msgr2=0x56259b77ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b5dd6b0 0x56259b77ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa54e59c700 1 -- 172.21.15.31:0/1895896297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 msgr2=0x56259b792440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b792440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa54e59c700 1 -- 172.21.15.31:0/1895896297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259b4e9c00 con 0x56259b782cf0 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa54e59c700 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 0x56259b7890f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa540098a00 tx=0x7fa540009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4898e9c2bbd36ac9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa53cff9700 1 -- 172.21.15.31:0/1895896297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540004ad0 con 0x56259b782cf0 2023-07-29T16:16:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa53cff9700 1 -- 172.21.15.31:0/1895896297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa540094af0 con 0x56259b782cf0 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.700+0000 7fa53cff9700 1 -- 172.21.15.31:0/1895896297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400b1260 con 0x56259b782cf0 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 -- 172.21.15.31:0/1895896297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 msgr2=0x56259b7890f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 0x56259b7890f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa540098a00 tx=0x7fa540009f90 comp rx=0 tx=0).stop 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 -- 172.21.15.31:0/1895896297 shutdown_connections 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b782cf0 0x56259b7890f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b5dd6b0 0x56259b77ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 --2- 172.21.15.31:0/1895896297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b781df0 0x56259b792440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 -- 172.21.15.31:0/1895896297 >> 172.21.15.31:0/1895896297 conn(0x56259b776fd0 msgr2=0x56259b789c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 -- 172.21.15.31:0/1895896297 shutdown_connections 2023-07-29T16:16:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 -- 172.21.15.31:0/1895896297 wait complete. 2023-07-29T16:16:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.701+0000 7fa563a16b80 1 -- start start 2023-07-29T16:16:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 Processor -- start 2023-07-29T16:16:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b5dd6b0 0x56259b78e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b781df0 0x56259b78ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b791b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b792050 con 0x56259b781df0 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b7921c0 con 0x56259b5dd6b0 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259b792380 con 0x56259b782cf0 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa563a16b80 10 client.0 _ll_register_callbacks cb 0x56259b6785c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa54e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b791b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa54d59a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b781df0 0x56259b78ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa54e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b791b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38020/0 (socket says 172.21.15.31:38020) 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa54e59c700 1 -- 172.21.15.31:0/539939366 learned_addr learned my addr 172.21.15.31:0/539939366 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.702+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b5dd6b0 0x56259b78e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54e59c700 1 -- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b5dd6b0 msgr2=0x56259b78e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54e59c700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b5dd6b0 0x56259b78e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54e59c700 1 -- 172.21.15.31:0/539939366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b781df0 msgr2=0x56259b78ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54e59c700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b781df0 0x56259b78ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54e59c700 1 -- 172.21.15.31:0/539939366 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa544017070 con 0x56259b782cf0 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54d59a700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b781df0 0x56259b78ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa54e59c700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b791b10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa54000ef90 tx=0x7fa540094a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38fefdc2f590e121 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa5317fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa5317fa700 1 -- 172.21.15.31:0/539939366 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400cb2b0 con 0x56259b782cf0 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa544021070 con 0x56259b782cf0 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa5317fa700 1 -- 172.21.15.31:0/539939366 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa540004ad0 con 0x56259b782cf0 2023-07-29T16:16:04.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa5317fa700 1 -- 172.21.15.31:0/539939366 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5400cb410 con 0x56259b782cf0 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.703+0000 7fa563a16b80 10 client.25831 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56259b5271f0 con 0x56259b782cf0 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa5317fa700 1 -- 172.21.15.31:0/539939366 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa544021070 con 0x56259b782cf0 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa5317fa700 1 -- 172.21.15.31:0/539939366 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa540099c00 con 0x56259b782cf0 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 10 client.25831 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 10 client.25831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 20 client.25831 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:04.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b7e94e0 0x56259b7eb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 4 client.25831 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:16:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56259b6cbce0 con 0x56259b7e94e0 2023-07-29T16:16:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.704+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b7e94e0 0x56259b7eb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.705+0000 7fa5317fa700 10 client.25831 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:04.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.705+0000 7fa54dd9b700 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b7e94e0 0x56259b7eb8c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760e04064c1e3eb8 server_cookie=db6ca39b05230d5e in_seq=0 out_seq=0 2023-07-29T16:16:04.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa5317fa700 1 -- 172.21.15.31:0/539939366 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56259b6cbce0 con 0x56259b7e94e0 2023-07-29T16:16:04.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa5317fa700 10 client.25831 handle_command_reply: tid=0 2023-07-29T16:16:04.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 client.25831 shutdown 2023-07-29T16:16:04.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 msgr2=0x56259b791b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b791b10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa54000ef90 tx=0x7fa540094a40 comp rx=0 tx=0).stop 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 shutdown_connections 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56259b7e94e0 0x56259b7eb8c0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56259b782cf0 0x56259b791b10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56259b5dd6b0 0x56259b78e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 --2- 172.21.15.31:0/539939366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56259b781df0 0x56259b78ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa5317fa700 0 client.25831 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 >> 172.21.15.31:0/539939366 conn(0x56259b776fd0 msgr2=0x56259b77afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 shutdown_connections 2023-07-29T16:16:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 1 -- 172.21.15.31:0/539939366 wait complete. 2023-07-29T16:16:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.709+0000 7fa563a16b80 20 client.25831 trim_cache size 0 max 16384 2023-07-29T16:16:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:16:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:16:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:16:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:16:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5340a0810 msgr2=0x7fa5340a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5340a0810 0x7fa5340a2cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa544005950 tx=0x7fa544009260 comp rx=0 tx=0).stop 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101ed0 msgr2=0x7fa548199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101ed0 0x7fa548199750 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa54000efc0 tx=0x7fa54000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 shutdown_connections 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa5340a0810 0x7fa5340a2cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa548101ed0 0x7fa548199750 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa548100350 0x7fa548198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 --2- 172.21.15.31:0/3200682157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa548101540 0x7fa5481988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 >> 172.21.15.31:0/3200682157 conn(0x7fa5480fb970 msgr2=0x7fa548106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 shutdown_connections 2023-07-29T16:16:04.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:04.710+0000 7fa54ffff700 1 -- 172.21.15.31:0/3200682157 wait complete. 2023-07-29T16:16:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:04 smithi039 ceph-mon[128858]: pgmap v387: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:04.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:04 smithi156 ceph-mon[129245]: pgmap v387: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:06 smithi031 ceph-mon[95504]: pgmap v388: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:06 smithi039 ceph-mon[128858]: pgmap v388: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:06.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:06 smithi156 ceph-mon[129245]: pgmap v388: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:08.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:07 smithi156 ceph-mon[129245]: pgmap v389: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:16:08.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:07 smithi031 ceph-mon[95504]: pgmap v389: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:16:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:07 smithi039 ceph-mon[128858]: pgmap v389: 97 pgs: 97 active+clean; 665 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:16:10.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:10 smithi031 ceph-mon[95504]: pgmap v390: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:10.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:10 smithi039 ceph-mon[128858]: pgmap v390: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:10.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:10 smithi156 ceph-mon[129245]: pgmap v390: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:12.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:12 smithi031 ceph-mon[95504]: pgmap v391: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:12 smithi039 ceph-mon[128858]: pgmap v391: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:12 smithi156 ceph-mon[129245]: pgmap v391: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:14.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:14 smithi031 ceph-mon[95504]: pgmap v392: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:14.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:14 smithi039 ceph-mon[128858]: pgmap v392: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:14.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:14 smithi156 ceph-mon[129245]: pgmap v392: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:15.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:16:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 -- 172.21.15.31:0/1124917733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101ea0 msgr2=0x7f189c108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 --2- 172.21.15.31:0/1124917733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101ea0 0x7f189c108760 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1894009990 tx=0x7f1894009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 -- 172.21.15.31:0/1124917733 shutdown_connections 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 --2- 172.21.15.31:0/1124917733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101ea0 0x7f189c108760 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 --2- 172.21.15.31:0/1124917733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 0x7f189c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 --2- 172.21.15.31:0/1124917733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f189c101510 0x7f189c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.918+0000 7f18a1dff700 1 -- 172.21.15.31:0/1124917733 >> 172.21.15.31:0/1124917733 conn(0x7f189c0fb920 msgr2=0x7f189c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.919+0000 7f18a1dff700 1 -- 172.21.15.31:0/1124917733 shutdown_connections 2023-07-29T16:16:15.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.919+0000 7f18a1dff700 1 -- 172.21.15.31:0/1124917733 wait complete. 2023-07-29T16:16:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 Processor -- start 2023-07-29T16:16:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 -- start start 2023-07-29T16:16:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 0x7f189c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 0x7f189c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f189c101ea0 0x7f189c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c19fde0 con 0x7f189c101510 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c19ff20 con 0x7f189c101ea0 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f18a1dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c1a00b0 con 0x7f189c100320 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f189affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 0x7f189c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 0x7f189c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f189affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 0x7f189c1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36048/0 (socket says 172.21.15.31:36048) 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.920+0000 7f189affd700 1 -- 172.21.15.31:0/2809648959 learned_addr learned my addr 172.21.15.31:0/2809648959 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189bfff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f189c101ea0 0x7f189c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189affd700 1 -- 172.21.15.31:0/2809648959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 msgr2=0x7f189c198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189affd700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 0x7f189c198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189affd700 1 -- 172.21.15.31:0/2809648959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f189c101ea0 msgr2=0x7f189c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189affd700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f189c101ea0 0x7f189c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189affd700 1 -- 172.21.15.31:0/2809648959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1890009520 con 0x7f189c101510 2023-07-29T16:16:15.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189b7fe700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 0x7f189c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f189affd700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 0x7f189c1987a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f189000eb30 tx=0x7f189000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f1898ff9700 1 -- 172.21.15.31:0/2809648959 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890009880 con 0x7f189c101510 2023-07-29T16:16:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f1898ff9700 1 -- 172.21.15.31:0/2809648959 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18900044e0 con 0x7f189c101510 2023-07-29T16:16:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18940095f0 con 0x7f189c101510 2023-07-29T16:16:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.921+0000 7f1898ff9700 1 -- 172.21.15.31:0/2809648959 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890003d80 con 0x7f189c101510 2023-07-29T16:16:15.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.922+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189c1a0710 con 0x7f189c101510 2023-07-29T16:16:15.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.925+0000 7f1898ff9700 1 -- 172.21.15.31:0/2809648959 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1890003f20 con 0x7f189c101510 2023-07-29T16:16:15.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 Processor -- start 2023-07-29T16:16:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 -- start start 2023-07-29T16:16:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3705ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e3706d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3707740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e365da10 con 0x5608e3706530 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e365b3d0 con 0x5608e37072f0 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.926+0000 7f18b5627b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e3656cc0 con 0x5608e3705be0 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f1898ff9700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e3706d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3705ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f1898ff9700 1 -- 172.21.15.31:0/2809648959 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1890014070 con 0x7f189c101510 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e3706d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36058/0 (socket says 172.21.15.31:36058) 2023-07-29T16:16:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189b7fe700 1 -- 172.21.15.31:0/3217215969 learned_addr learned my addr 172.21.15.31:0/3217215969 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189affd700 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3707740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189bfff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189affd700 1 -- 172.21.15.31:0/3217215969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 msgr2=0x5608e3705ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189affd700 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3705ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189affd700 1 -- 172.21.15.31:0/3217215969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 msgr2=0x5608e3706d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189affd700 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e3706d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189affd700 1 -- 172.21.15.31:0/3217215969 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608e3484de0 con 0x5608e37072f0 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.927+0000 7f189b7fe700 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e3706d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f189affd700 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3707740 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18900c9be0 tx=0x7f1890095c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e7706f596f8fa8c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f189bfff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x5608e3705980 tx=0x7f1894005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f1881ffb700 1 -- 172.21.15.31:0/3217215969 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189000cbd0 con 0x5608e37072f0 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f1881ffb700 1 -- 172.21.15.31:0/3217215969 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18900997f0 con 0x5608e37072f0 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f1881ffb700 1 -- 172.21.15.31:0/3217215969 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1890097cb0 con 0x5608e37072f0 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f18b5627b80 1 -- 172.21.15.31:0/3217215969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 msgr2=0x5608e3707740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.928+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3707740 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18900c9be0 tx=0x7f1890095c40 comp rx=0 tx=0).stop 2023-07-29T16:16:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 -- 172.21.15.31:0/3217215969 shutdown_connections 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3707740 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3705ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3217215969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e3706d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 -- 172.21.15.31:0/3217215969 >> 172.21.15.31:0/3217215969 conn(0x5608e36fb3e0 msgr2=0x5608e3700990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 -- 172.21.15.31:0/3217215969 shutdown_connections 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 -- 172.21.15.31:0/3217215969 wait complete. 2023-07-29T16:16:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 -- start start 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.929+0000 7f18b5627b80 1 Processor -- start 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e370f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3716e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e35f3510 con 0x5608e3706530 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e35f31b0 con 0x5608e37072f0 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e34da920 con 0x5608e3705be0 2023-07-29T16:16:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f18b5627b80 10 client.0 _ll_register_callbacks cb 0x5608e35fc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f189affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e370f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e370f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58936/0 (socket says 172.21.15.31:58936) 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f189affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36066/0 (socket says 172.21.15.31:36066) 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f189b7fe700 1 -- 172.21.15.31:0/3875842360 learned_addr learned my addr 172.21.15.31:0/3875842360 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.930+0000 7f189bfff700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3716e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189b7fe700 1 -- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 msgr2=0x5608e3716e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189b7fe700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3716e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189b7fe700 1 -- 172.21.15.31:0/3875842360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 msgr2=0x5608e370fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189b7fe700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189b7fe700 1 -- 172.21.15.31:0/3875842360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f188c003b20 con 0x5608e3705be0 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189affd700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f189b7fe700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e370f8c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f188c003e30 tx=0x7f188c00e7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e49e72412a6ebf6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f187affd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:16:15.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f187affd700 1 -- 172.21.15.31:0/3875842360 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c004db0 con 0x5608e3705be0 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608e3681d10 con 0x5608e3705be0 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.931+0000 7f187affd700 1 -- 172.21.15.31:0/3875842360 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f188c0090d0 con 0x5608e3705be0 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f187affd700 1 -- 172.21.15.31:0/3875842360 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f188c0092d0 con 0x5608e3705be0 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 10 client.25843 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f189c1a0710 con 0x5608e3705be0 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f187affd700 1 -- 172.21.15.31:0/3875842360 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5608e3681d10 con 0x5608e3705be0 2023-07-29T16:16:15.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f187affd700 1 -- 172.21.15.31:0/3875842360 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f188c01c030 con 0x5608e3705be0 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 10 client.25843 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 10 client.25843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 20 client.25843 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e371b6f0 0x5608e371db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 4 client.25843 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.932+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608e36d4d70 con 0x5608e371b6f0 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.933+0000 7f189affd700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e371b6f0 0x5608e371db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.933+0000 7f189affd700 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e371b6f0 0x5608e371db00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5e7718c8405867c server_cookie=5a328af66fb9b1df in_seq=0 out_seq=0 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.933+0000 7f187affd700 10 client.25843 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f187affd700 1 -- 172.21.15.31:0/3875842360 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5608e36d4d70 con 0x5608e371b6f0 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f187affd700 10 client.25843 handle_command_reply: tid=0 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 client.25843 shutdown 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 msgr2=0x5608e370f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e370f8c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f188c003e30 tx=0x7f188c00e7d0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 shutdown_connections 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e371b6f0 0x5608e371db00 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e3716e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e370f8c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f187affd700 0 client.25843 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 --2- 172.21.15.31:0/3875842360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 >> 172.21.15.31:0/3875842360 conn(0x5608e36fb3e0 msgr2=0x5608e3700060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.934+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 shutdown_connections 2023-07-29T16:16:15.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.935+0000 7f18b5627b80 1 -- 172.21.15.31:0/3875842360 wait complete. 2023-07-29T16:16:15.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.935+0000 7f18b5627b80 20 client.25843 trim_cache size 0 max 16384 2023-07-29T16:16:15.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 Processor -- start 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 -- start start 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3714df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e370a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e37018e0 con 0x5608e3706530 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e36fd460 con 0x5608e37072f0 2023-07-29T16:16:15.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f18b5627b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e36faea0 con 0x5608e3705be0 2023-07-29T16:16:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f189affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3714df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f189affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36074/0 (socket says 172.21.15.31:36074) 2023-07-29T16:16:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f189affd700 1 -- 172.21.15.31:0/4162128259 learned_addr learned my addr 172.21.15.31:0/4162128259 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.952+0000 7f189bfff700 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e370a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189affd700 1 -- 172.21.15.31:0/4162128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 msgr2=0x5608e3714df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189affd700 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3714df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189affd700 1 -- 172.21.15.31:0/4162128259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 msgr2=0x5608e370a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189affd700 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e370a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189affd700 1 -- 172.21.15.31:0/4162128259 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608e346dc60 con 0x5608e3706530 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189b7fe700 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3714df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f189affd700 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370a210 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1890000c00 tx=0x7f189009d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8acada0b6c3943bc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:15.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f18827fc700 1 -- 172.21.15.31:0/4162128259 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189000cbd0 con 0x5608e3706530 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f18827fc700 1 -- 172.21.15.31:0/4162128259 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f189009d820 con 0x5608e3706530 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.953+0000 7f18827fc700 1 -- 172.21.15.31:0/4162128259 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18900cc1f0 con 0x5608e3706530 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 -- 172.21.15.31:0/4162128259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 msgr2=0x5608e370a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370a210 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1890000c00 tx=0x7f189009d040 comp rx=0 tx=0).stop 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 -- 172.21.15.31:0/4162128259 shutdown_connections 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e37072f0 0x5608e370a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3705be0 0x5608e3714df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 --2- 172.21.15.31:0/4162128259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e3706530 0x5608e370a210 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 -- 172.21.15.31:0/4162128259 >> 172.21.15.31:0/4162128259 conn(0x5608e36fb3e0 msgr2=0x5608e370d700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:15.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 -- 172.21.15.31:0/4162128259 shutdown_connections 2023-07-29T16:16:15.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 -- 172.21.15.31:0/4162128259 wait complete. 2023-07-29T16:16:15.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 -- start start 2023-07-29T16:16:15.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.954+0000 7f18b5627b80 1 Processor -- start 2023-07-29T16:16:15.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 0x5608e3714ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3706530 0x5608e37130d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e37072f0 0x5608e3713610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e3713b50 con 0x5608e37072f0 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e3713cc0 con 0x5608e3705be0 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608e3713e30 con 0x5608e3706530 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f18b5627b80 10 client.0 _ll_register_callbacks cb 0x5608e35fc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 0x5608e3714ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f189bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e37072f0 0x5608e3713610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f189b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 0x5608e3714ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40872/0 (socket says 172.21.15.31:40872) 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f189b7fe700 1 -- 172.21.15.31:0/1770960778 learned_addr learned my addr 172.21.15.31:0/1770960778 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.955+0000 7f189affd700 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3706530 0x5608e37130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f189b7fe700 1 -- 172.21.15.31:0/1770960778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3706530 msgr2=0x5608e37130d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f189b7fe700 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3706530 0x5608e37130d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f189b7fe700 1 -- 172.21.15.31:0/1770960778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e37072f0 msgr2=0x5608e3713610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:15.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f189b7fe700 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e37072f0 0x5608e3713610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f189b7fe700 1 -- 172.21.15.31:0/1770960778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f189c1a0710 con 0x5608e3705be0 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f189b7fe700 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 0x5608e3714ac0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f188c0047d0 tx=0x7f188c028650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5463985f5bc79cc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f187affd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f187affd700 1 -- 172.21.15.31:0/1770960778 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c004e20 con 0x5608e3705be0 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18900997f0 con 0x5608e3705be0 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f187affd700 1 -- 172.21.15.31:0/1770960778 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f188c0287a0 con 0x5608e3705be0 2023-07-29T16:16:15.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f187affd700 1 -- 172.21.15.31:0/1770960778 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f188c0099a0 con 0x5608e3705be0 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f18b5627b80 10 client.25838 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.956+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5608e3484de0 con 0x5608e3705be0 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f187affd700 1 -- 172.21.15.31:0/1770960778 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18900997f0 con 0x5608e3705be0 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f187affd700 1 -- 172.21.15.31:0/1770960778 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f188c038070 con 0x5608e3705be0 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f18b5627b80 10 client.25838 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f18b5627b80 10 client.25838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f18b5627b80 20 client.25838 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:15.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f18b5627b80 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e376e950 0x5608e3770d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f18b5627b80 4 client.25838 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5608e364fd40 con 0x5608e376e950 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.957+0000 7f189affd700 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e376e950 0x5608e3770d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f189affd700 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e376e950 0x5608e3770d60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e92862430ff8301 server_cookie=b60198e96db877e1 in_seq=0 out_seq=0 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f187affd700 10 client.25838 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f187affd700 1 -- 172.21.15.31:0/1770960778 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5608e364fd40 con 0x5608e376e950 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f187affd700 10 client.25838 handle_command_reply: tid=0 2023-07-29T16:16:15.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 client.25838 shutdown 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 msgr2=0x5608e3714ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 0x5608e3714ac0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f188c0047d0 tx=0x7f188c028650 comp rx=0 tx=0).stop 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 shutdown_connections 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5608e376e950 0x5608e3770d60 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608e37072f0 0x5608e3713610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608e3705be0 0x5608e3714ac0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f187affd700 0 client.25838 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:15.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 --2- 172.21.15.31:0/1770960778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608e3706530 0x5608e37130d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 >> 172.21.15.31:0/1770960778 conn(0x5608e36fb3e0 msgr2=0x5608e370a210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:15.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 shutdown_connections 2023-07-29T16:16:15.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.958+0000 7f18b5627b80 1 -- 172.21.15.31:0/1770960778 wait complete. 2023-07-29T16:16:15.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18b5627b80 20 client.25838 trim_cache size 0 max 16384 2023-07-29T16:16:15.965 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:16:15.966 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:16:15.966 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:16:15.966 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:16:15.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 msgr2=0x7f18840a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x5608e3705980 tx=0x7f1894005fd0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 msgr2=0x7f189c1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 0x7f189c1987a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f189000eb30 tx=0x7f189000ee40 comp rx=0 tx=0).stop 2023-07-29T16:16:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 shutdown_connections 2023-07-29T16:16:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18840a0790 0x7f18840a2c40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f189c101ea0 0x7f189c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f189c100320 0x7f189c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 --2- 172.21.15.31:0/2809648959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f189c101510 0x7f189c1987a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:15.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 >> 172.21.15.31:0/2809648959 conn(0x7f189c0fb920 msgr2=0x7f189c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:15.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 shutdown_connections 2023-07-29T16:16:15.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:15.959+0000 7f18a1dff700 1 -- 172.21.15.31:0/2809648959 wait complete. 2023-07-29T16:16:15.975 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:16:15.976 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:16:16.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.181+0000 7faea91ff700 1 -- 172.21.15.31:0/3686594561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 msgr2=0x7faea4101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.181+0000 7faea91ff700 1 --2- 172.21.15.31:0/3686594561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4101980 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009990 tx=0x7fae8c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 -- 172.21.15.31:0/3686594561 shutdown_connections 2023-07-29T16:16:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 --2- 172.21.15.31:0/3686594561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea4101ec0 0x7faea4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 --2- 172.21.15.31:0/3686594561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faea4100340 0x7faea4100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 --2- 172.21.15.31:0/3686594561 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4101980 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 -- 172.21.15.31:0/3686594561 >> 172.21.15.31:0/3686594561 conn(0x7faea40fb920 msgr2=0x7faea40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 -- 172.21.15.31:0/3686594561 shutdown_connections 2023-07-29T16:16:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.182+0000 7faea91ff700 1 -- 172.21.15.31:0/3686594561 wait complete. 2023-07-29T16:16:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 Processor -- start 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 -- start start 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea4100340 0x7faea4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faea4101ec0 0x7faea4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea4073e20 con 0x7faea4101530 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea4073f90 con 0x7faea4100340 2023-07-29T16:16:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.183+0000 7faea91ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea41a00b0 con 0x7faea4101ec0 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36118/0 (socket says 172.21.15.31:36118) 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7faea359e700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faea4101ec0 0x7faea4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 -- 172.21.15.31:0/155646391 learned_addr learned my addr 172.21.15.31:0/155646391 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7faea2d9d700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea4100340 0x7faea4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 -- 172.21.15.31:0/155646391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faea4101ec0 msgr2=0x7faea4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faea4101ec0 0x7faea4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 -- 172.21.15.31:0/155646391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea4100340 msgr2=0x7faea4074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea4100340 0x7faea4074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 -- 172.21.15.31:0/155646391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0095f0 con 0x7faea4101530 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.184+0000 7fae9bfff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4073140 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fae8c004760 tx=0x7fae8c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.185+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c01d070 con 0x7faea4101530 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.185+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea41a0330 con 0x7faea4101530 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.185+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae8c004830 con 0x7faea4101530 2023-07-29T16:16:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.185+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c005350 con 0x7faea4101530 2023-07-29T16:16:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.185+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea41a06f0 con 0x7faea4101530 2023-07-29T16:16:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.186+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea4107820 con 0x7faea4101530 2023-07-29T16:16:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.188+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fae8c005510 con 0x7faea4101530 2023-07-29T16:16:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.189+0000 7faea0d99700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fae900a0860 0x7fae900a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.189+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fae8c0ccc80 con 0x7faea4101530 2023-07-29T16:16:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.189+0000 7faea2d9d700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fae900a0860 0x7fae900a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.190+0000 7faea2d9d700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fae900a0860 0x7fae900a2d10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faea4101390 tx=0x7fae94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.193+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fae8c05a720 con 0x7faea4101530 2023-07-29T16:16:16.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.449+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea402cc30 con 0x7faea4101530 2023-07-29T16:16:16.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.450+0000 7faea0d99700 1 -- 172.21.15.31:0/155646391 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7fae8c0966f0 con 0x7faea4101530 2023-07-29T16:16:16.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:16:16.449 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:16:16.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.452+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fae900a0860 msgr2=0x7fae900a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.452+0000 7faea91ff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fae900a0860 0x7fae900a2d10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faea4101390 tx=0x7fae94009260 comp rx=0 tx=0).stop 2023-07-29T16:16:16.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.452+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 msgr2=0x7faea4073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.452+0000 7faea91ff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4073140 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fae8c004760 tx=0x7fae8c004790 comp rx=0 tx=0).stop 2023-07-29T16:16:16.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 shutdown_connections 2023-07-29T16:16:16.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fae900a0860 0x7fae900a2d10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faea4101ec0 0x7faea4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faea4100340 0x7faea4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 --2- 172.21.15.31:0/155646391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faea4101530 0x7faea4073140 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 >> 172.21.15.31:0/155646391 conn(0x7faea40fb920 msgr2=0x7faea4104960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 shutdown_connections 2023-07-29T16:16:16.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.453+0000 7faea91ff700 1 -- 172.21.15.31:0/155646391 wait complete. 2023-07-29T16:16:16.453 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:16:16.470 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:16:16.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/1242186581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf40737b0 msgr2=0x7fdcf4073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/1242186581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf40737b0 0x7fdcf4073c20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009990 tx=0x7fdcdc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/1242186581 shutdown_connections 2023-07-29T16:16:16.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/1242186581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcf41081f0 0x7fdcf410a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/1242186581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf4074d90 0x7fdcf40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/1242186581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf40737b0 0x7fdcf4073c20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/1242186581 >> 172.21.15.31:0/1242186581 conn(0x7fdcf40fb970 msgr2=0x7fdcf40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/1242186581 shutdown_connections 2023-07-29T16:16:16.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.679+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/1242186581 wait complete. 2023-07-29T16:16:16.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 Processor -- start 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 -- start start 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 0x7fdcf41982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf4074d90 0x7fdcf4198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcf41081f0 0x7fdcf419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf419fe70 con 0x7fdcf40737b0 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf419ffb0 con 0x7fdcf41081f0 2023-07-29T16:16:16.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcfa1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf41a0140 con 0x7fdcf4074d90 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcf37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 0x7fdcf41982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcf37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 0x7fdcf41982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36140/0 (socket says 172.21.15.31:36140) 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.680+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf4074d90 0x7fdcf4198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 -- 172.21.15.31:0/15561577 learned_addr learned my addr 172.21.15.31:0/15561577 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcf41081f0 0x7fdcf419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 -- 172.21.15.31:0/15561577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf4074d90 msgr2=0x7fdcf4198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf4074d90 0x7fdcf4198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 -- 172.21.15.31:0/15561577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcf41081f0 msgr2=0x7fdcf419f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcf41081f0 0x7fdcf419f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 -- 172.21.15.31:0/15561577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4009520 con 0x7fdcf40737b0 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf37fe700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 0x7fdcf41982f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdce400ea30 tx=0x7fdce400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.681+0000 7fdcf17fa700 1 -- 172.21.15.31:0/15561577 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce400cbd0 con 0x7fdcf40737b0 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.682+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc0095f0 con 0x7fdcf40737b0 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.682+0000 7fdcf17fa700 1 -- 172.21.15.31:0/15561577 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdce400eed0 con 0x7fdcf40737b0 2023-07-29T16:16:16.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.682+0000 7fdcf17fa700 1 -- 172.21.15.31:0/15561577 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce4003d20 con 0x7fdcf40737b0 2023-07-29T16:16:16.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.682+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf41a07f0 con 0x7fdcf40737b0 2023-07-29T16:16:16.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:16 smithi031 ceph-mon[95504]: pgmap v393: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:16.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.685+0000 7fdcf17fa700 1 -- 172.21.15.31:0/15561577 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdce4003ee0 con 0x7fdcf40737b0 2023-07-29T16:16:16.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 Processor -- start 2023-07-29T16:16:16.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 -- start start 2023-07-29T16:16:16.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918df9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918dfad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdcf17fa700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdce00a0790 0x7fdce00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918dfb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918d51a10 con 0x562918df9be0 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918d4f3d0 con 0x562918dfb2f0 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.687+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918d4acc0 con 0x562918dfa530 2023-07-29T16:16:16.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcf17fa700 1 -- 172.21.15.31:0/15561577 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdce4014070 con 0x7fdcf40737b0 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918dfad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcf3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918df9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918dfad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58972/0 (socket says 172.21.15.31:58972) 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 -- 172.21.15.31:0/3878578907 learned_addr learned my addr 172.21.15.31:0/3878578907 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcf37fe700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdce00a0790 0x7fdce00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 -- 172.21.15.31:0/3878578907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 msgr2=0x562918dfb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918dfb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 -- 172.21.15.31:0/3878578907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 msgr2=0x562918df9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918df9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 -- 172.21.15.31:0/3878578907 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562918b78de0 con 0x562918dfa530 2023-07-29T16:16:16.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcebfff700 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918dfad60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4074ab0 tx=0x7fdcdc005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=329a186afb0a8722 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.688+0000 7fdcf37fe700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdce00a0790 0x7fdce00a2c40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdce4095af0 tx=0x7fdce4003780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdce9ffb700 1 -- 172.21.15.31:0/3878578907 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc01d070 con 0x562918dfa530 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdce9ffb700 1 -- 172.21.15.31:0/3878578907 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcdc01e040 con 0x562918dfa530 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdce9ffb700 1 -- 172.21.15.31:0/3878578907 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc0048f0 con 0x562918dfa530 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 -- 172.21.15.31:0/3878578907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 msgr2=0x562918dfad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918dfad60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4074ab0 tx=0x7fdcdc005ea0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 -- 172.21.15.31:0/3878578907 shutdown_connections 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918dfb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918df9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/3878578907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918dfad60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.689+0000 7fdd0da74b80 1 -- 172.21.15.31:0/3878578907 >> 172.21.15.31:0/3878578907 conn(0x562918def3e0 msgr2=0x562918df4990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.690+0000 7fdd0da74b80 1 -- 172.21.15.31:0/3878578907 shutdown_connections 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.690+0000 7fdd0da74b80 1 -- 172.21.15.31:0/3878578907 wait complete. 2023-07-29T16:16:16.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.690+0000 7fdd0da74b80 1 -- start start 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.690+0000 7fdd0da74b80 1 Processor -- start 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918e037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918e03d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918e0ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918ce7510 con 0x562918df9be0 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918ce71b0 con 0x562918dfb2f0 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918bce920 con 0x562918dfa530 2023-07-29T16:16:16.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdd0da74b80 10 client.0 _ll_register_callbacks cb 0x562918cf0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918e037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918e037d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36158/0 (socket says 172.21.15.31:36158) 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 -- 172.21.15.31:0/796579590 learned_addr learned my addr 172.21.15.31:0/796579590 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcebfff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918e03d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918e0ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 -- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 msgr2=0x562918e03d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918e03d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 -- 172.21.15.31:0/796579590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 msgr2=0x562918e0ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918e0ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.691+0000 7fdcf3fff700 1 -- 172.21.15.31:0/796579590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf410b290 con 0x562918df9be0 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918e037d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdcec01b670 tx=0x7fdcec0074c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78352583f1be083 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdcd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/796579590 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcec00e040 con 0x562918df9be0 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562918d75d10 con 0x562918df9be0 2023-07-29T16:16:16.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/796579590 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdcec004db0 con 0x562918df9be0 2023-07-29T16:16:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/796579590 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcec003a00 con 0x562918df9be0 2023-07-29T16:16:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdd0da74b80 10 client.15879 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562918b61c60 con 0x562918df9be0 2023-07-29T16:16:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.692+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/796579590 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcec02a080 con 0x562918df9be0 2023-07-29T16:16:16.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/796579590 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdcec004030 con 0x562918df9be0 2023-07-29T16:16:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdd0da74b80 10 client.15879 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdd0da74b80 10 client.15879 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:16:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdd0da74b80 20 client.15879 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e0f600 0x562918e11a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdd0da74b80 4 client.15879 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:16.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562918dc8d70 con 0x562918e0f600 2023-07-29T16:16:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.693+0000 7fdcebfff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e0f600 0x562918e11a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.694+0000 7fdcebfff700 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e0f600 0x562918e11a10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2b61b2cb88f5b0c server_cookie=c22557c9babad461 in_seq=0 out_seq=0 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.694+0000 7fdcd2ffd700 10 client.15879 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/796579590 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562918dc8d70 con 0x562918e0f600 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdcd2ffd700 10 client.15879 handle_command_reply: tid=0 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 client.15879 shutdown 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 msgr2=0x562918e037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918e037d0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdcec01b670 tx=0x7fdcec0074c0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 shutdown_connections 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e0f600 0x562918e11a10 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918dfb2f0 0x562918e0ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918df9be0 0x562918e037d0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdcd2ffd700 0 client.15879 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/796579590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfa530 0x562918e03d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 >> 172.21.15.31:0/796579590 conn(0x562918def3e0 msgr2=0x562918df4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 shutdown_connections 2023-07-29T16:16:16.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 1 -- 172.21.15.31:0/796579590 wait complete. 2023-07-29T16:16:16.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.695+0000 7fdd0da74b80 20 client.15879 trim_cache size 0 max 16384 2023-07-29T16:16:16.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 Processor -- start 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 -- start start 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e02480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e0ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918e0cad0 con 0x562918dfa530 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918e01470 con 0x562918df9be0 2023-07-29T16:16:16.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918e021b0 con 0x562918dfb2f0 2023-07-29T16:16:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e0ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e0ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36164/0 (socket says 172.21.15.31:36164) 2023-07-29T16:16:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdcebfff700 1 -- 172.21.15.31:0/2528582452 learned_addr learned my addr 172.21.15.31:0/2528582452 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e02480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.710+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcebfff700 1 -- 172.21.15.31:0/2528582452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 msgr2=0x562918e016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcebfff700 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcebfff700 1 -- 172.21.15.31:0/2528582452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 msgr2=0x562918e02480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcebfff700 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e02480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcebfff700 1 -- 172.21.15.31:0/2528582452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562918b9f250 con 0x562918dfa530 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcebfff700 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e0ad10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009ce0 tx=0x7fdcdc01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e3dcd0b1dcaeaf8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcea7fc700 1 -- 172.21.15.31:0/2528582452 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc01d040 con 0x562918dfa530 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcea7fc700 1 -- 172.21.15.31:0/2528582452 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcdc034340 con 0x562918dfa530 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.711+0000 7fdcea7fc700 1 -- 172.21.15.31:0/2528582452 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc034d00 con 0x562918dfa530 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2528582452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 msgr2=0x562918e0ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e0ad10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009ce0 tx=0x7fdcdc01a040 comp rx=0 tx=0).stop 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2528582452 shutdown_connections 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e02480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2528582452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e0ad10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2528582452 >> 172.21.15.31:0/2528582452 conn(0x562918def3e0 msgr2=0x562918df4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2528582452 shutdown_connections 2023-07-29T16:16:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2528582452 wait complete. 2023-07-29T16:16:16.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.712+0000 7fdd0da74b80 1 -- start start 2023-07-29T16:16:16.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 Processor -- start 2023-07-29T16:16:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e06f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e0a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918e0aa00 con 0x562918dfa530 2023-07-29T16:16:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918e0ab70 con 0x562918df9be0 2023-07-29T16:16:16.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562918e0ad30 con 0x562918dfb2f0 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdd0da74b80 10 client.0 _ll_register_callbacks cb 0x562918cf0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdcebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e074a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36172/0 (socket says 172.21.15.31:36172) 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.713+0000 7fdcebfff700 1 -- 172.21.15.31:0/2843967490 learned_addr learned my addr 172.21.15.31:0/2843967490 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e06f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e0a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcebfff700 1 -- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 msgr2=0x562918e0a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcebfff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e0a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcebfff700 1 -- 172.21.15.31:0/2843967490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 msgr2=0x562918e06f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcebfff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e06f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcebfff700 1 -- 172.21.15.31:0/2843967490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562918b78de0 con 0x562918dfa530 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcebfff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e074a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00b680 tx=0x7fdcdc034610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6f64812ea15575b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/2843967490 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc01d040 con 0x562918dfa530 2023-07-29T16:16:16.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/2843967490 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdcdc034c90 con 0x562918dfa530 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.714+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcdc01e040 con 0x562918dfa530 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/2843967490 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcdc034e50 con 0x562918dfa530 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 10 client.15885 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562918b772b0 con 0x562918dfa530 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/2843967490 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcdc01e040 con 0x562918dfa530 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/2843967490 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdcdc004ae0 con 0x562918dfa530 2023-07-29T16:16:16.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 10 client.15885 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 10 client.15885 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 20 client.15885 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e62040 0x562918e64450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 4 client.15885 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562918d43d40 con 0x562918e62040 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.715+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e62040 0x562918e64450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdcf3fff700 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e62040 0x562918e64450 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68b43bc97eb31864 server_cookie=aff6456fe2bb240f in_seq=0 out_seq=0 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdcd2ffd700 10 client.15885 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdcd2ffd700 1 -- 172.21.15.31:0/2843967490 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562918d43d40 con 0x562918e62040 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdcd2ffd700 10 client.15885 handle_command_reply: tid=0 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 client.15885 shutdown 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 msgr2=0x562918e074a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e074a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc00b680 tx=0x7fdcdc034610 comp rx=0 tx=0).stop 2023-07-29T16:16:16.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 shutdown_connections 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562918dfb2f0 0x562918e0a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562918e62040 0x562918e64450 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562918df9be0 0x562918e06f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 --2- 172.21.15.31:0/2843967490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562918dfa530 0x562918e074a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdcd2ffd700 0 client.15885 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.716+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 >> 172.21.15.31:0/2843967490 conn(0x562918def3e0 msgr2=0x562918e02480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 shutdown_connections 2023-07-29T16:16:16.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdd0da74b80 1 -- 172.21.15.31:0/2843967490 wait complete. 2023-07-29T16:16:16.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdd0da74b80 20 client.15885 trim_cache size 0 max 16384 2023-07-29T16:16:16.724 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdce00a0790 msgr2=0x7fdce00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdce00a0790 0x7fdce00a2c40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fdce4095af0 tx=0x7fdce4003780 comp rx=0 tx=0).stop 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 msgr2=0x7fdcf41982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 0x7fdcf41982f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdce400ea30 tx=0x7fdce400ed40 comp rx=0 tx=0).stop 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 shutdown_connections 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdce00a0790 0x7fdce00a2c40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdcf41081f0 0x7fdcf419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdcf40737b0 0x7fdcf41982f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 --2- 172.21.15.31:0/15561577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdcf4074d90 0x7fdcf4198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 >> 172.21.15.31:0/15561577 conn(0x7fdcf40fb970 msgr2=0x7fdcf40fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.717+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 shutdown_connections 2023-07-29T16:16:16.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.718+0000 7fdcfa1ff700 1 -- 172.21.15.31:0/15561577 wait complete. 2023-07-29T16:16:16.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:16:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:16 smithi039 ceph-mon[128858]: pgmap v393: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:16 smithi156 ceph-mon[129245]: pgmap v393: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 -- 172.21.15.31:0/3906490402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0100320 msgr2=0x7f67c0100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 --2- 172.21.15.31:0/3906490402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0100320 0x7f67c0100730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098f0 tx=0x7f67b0009c00 comp rx=0 tx=0).stop 2023-07-29T16:16:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 -- 172.21.15.31:0/3906490402 shutdown_connections 2023-07-29T16:16:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 --2- 172.21.15.31:0/3906490402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67c0101ea0 0x7f67c0108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 --2- 172.21.15.31:0/3906490402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0100320 0x7f67c0100730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 --2- 172.21.15.31:0/3906490402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0101510 0x7f67c0101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 -- 172.21.15.31:0/3906490402 >> 172.21.15.31:0/3906490402 conn(0x7f67c00fb920 msgr2=0x7f67c00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 -- 172.21.15.31:0/3906490402 shutdown_connections 2023-07-29T16:16:16.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.941+0000 7f67c55ff700 1 -- 172.21.15.31:0/3906490402 wait complete. 2023-07-29T16:16:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.942+0000 7f67c55ff700 1 Processor -- start 2023-07-29T16:16:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 -- start start 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0100320 0x7f67c01983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 0x7f67c0198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67c0101ea0 0x7f67c019f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c019ff60 con 0x7f67c0101510 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01a00a0 con 0x7f67c0101ea0 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67c55ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c01a0230 con 0x7f67c0100320 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67be7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 0x7f67c0198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67be7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 0x7f67c0198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36182/0 (socket says 172.21.15.31:36182) 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67be7fc700 1 -- 172.21.15.31:0/1263147545 learned_addr learned my addr 172.21.15.31:0/1263147545 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.943+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67c0101ea0 0x7f67c019f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67beffd700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0100320 0x7f67c01983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67be7fc700 1 -- 172.21.15.31:0/1263147545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0100320 msgr2=0x7f67c01983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67be7fc700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0100320 0x7f67c01983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67be7fc700 1 -- 172.21.15.31:0/1263147545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67c0101ea0 msgr2=0x7f67c019f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67be7fc700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67c0101ea0 0x7f67c019f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67be7fc700 1 -- 172.21.15.31:0/1263147545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b0009520 con 0x7f67c0101510 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67beffd700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0100320 0x7f67c01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67be7fc700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 0x7f67c0198920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f67b400b550 tx=0x7f67b400b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67a7fff700 1 -- 172.21.15.31:0/1263147545 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4011830 con 0x7f67c0101510 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01a0510 con 0x7f67c0101510 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67a7fff700 1 -- 172.21.15.31:0/1263147545 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67b4011e50 con 0x7f67c0101510 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67a7fff700 1 -- 172.21.15.31:0/1263147545 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b400f2e0 con 0x7f67c0101510 2023-07-29T16:16:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.944+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c01a0af0 con 0x7f67c0101510 2023-07-29T16:16:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.948+0000 7f67a7fff700 1 -- 172.21.15.31:0/1263147545 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f67b4011990 con 0x7f67c0101510 2023-07-29T16:16:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 Processor -- start 2023-07-29T16:16:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 -- start start 2023-07-29T16:16:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01a8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01a9530 0x557ae01a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67a7fff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67a80a0810 0x7f67a80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01aa2f0 0x557ae01aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae0100a10 con 0x557ae01a8be0 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae00fe3d0 con 0x557ae01aa2f0 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.949+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae00f9cc0 con 0x557ae01a9530 2023-07-29T16:16:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67a7fff700 1 -- 172.21.15.31:0/1263147545 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f67b40cb4d0 con 0x7f67c0101510 2023-07-29T16:16:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01a8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01a8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36188/0 (socket says 172.21.15.31:36188) 2023-07-29T16:16:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 -- 172.21.15.31:0/945585976 learned_addr learned my addr 172.21.15.31:0/945585976 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67beffd700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67a80a0810 0x7f67a80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67be7fc700 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01a9530 0x557ae01a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01aa2f0 0x557ae01aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 -- 172.21.15.31:0/945585976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01a9530 msgr2=0x557ae01a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01a9530 0x557ae01a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 -- 172.21.15.31:0/945585976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01aa2f0 msgr2=0x557ae01aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01aa2f0 0x557ae01aa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 -- 172.21.15.31:0/945585976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557adff27de0 con 0x557ae01a8be0 2023-07-29T16:16:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.950+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01a8ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009ec0 tx=0x7f67b80099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e78f216b2da2dbf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.951+0000 7f67beffd700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67a80a0810 0x7f67a80a2cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098c0 tx=0x7f67b0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.951+0000 7f67a57fa700 1 -- 172.21.15.31:0/945585976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b800e040 con 0x557ae01a8be0 2023-07-29T16:16:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.951+0000 7f67a57fa700 1 -- 172.21.15.31:0/945585976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67b800f040 con 0x557ae01a8be0 2023-07-29T16:16:16.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.951+0000 7f67a57fa700 1 -- 172.21.15.31:0/945585976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b8013cd0 con 0x557ae01a8be0 2023-07-29T16:16:16.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.951+0000 7f67d8f50b80 1 -- 172.21.15.31:0/945585976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 msgr2=0x557ae01a8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.951+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01a8ff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009ec0 tx=0x7f67b80099e0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 -- 172.21.15.31:0/945585976 shutdown_connections 2023-07-29T16:16:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01aa2f0 0x557ae01aa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01a8ff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/945585976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01a9530 0x557ae01a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 -- 172.21.15.31:0/945585976 >> 172.21.15.31:0/945585976 conn(0x557ae019e3e0 msgr2=0x557ae01a3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 -- 172.21.15.31:0/945585976 shutdown_connections 2023-07-29T16:16:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 -- 172.21.15.31:0/945585976 wait complete. 2023-07-29T16:16:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 -- start start 2023-07-29T16:16:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.952+0000 7f67d8f50b80 1 Processor -- start 2023-07-29T16:16:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae0096510 con 0x557ae01a9530 2023-07-29T16:16:16.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae00961b0 con 0x557ae01a8be0 2023-07-29T16:16:16.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557adff7d920 con 0x557ae01aa2f0 2023-07-29T16:16:16.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67d8f50b80 10 client.0 _ll_register_callbacks cb 0x557ae009f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67beffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67be7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67beffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41000/0 (socket says 172.21.15.31:41000) 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67beffd700 1 -- 172.21.15.31:0/2653679414 learned_addr learned my addr 172.21.15.31:0/2653679414 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.953+0000 7f67beffd700 1 -- 172.21.15.31:0/2653679414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 msgr2=0x557ae01b9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67beffd700 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67beffd700 1 -- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 msgr2=0x557ae01b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67beffd700 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67beffd700 1 -- 172.21.15.31:0/2653679414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b800f710 con 0x557ae01a8be0 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67beffd700 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b2810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098f0 tx=0x7f67b0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27178e135a8375e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67a2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67a2ff5700 1 -- 172.21.15.31:0/2653679414 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0028070 con 0x557ae01a8be0 2023-07-29T16:16:16.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ae0124d10 con 0x557ae01a8be0 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67a2ff5700 1 -- 172.21.15.31:0/2653679414 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f67b0005040 con 0x557ae01a8be0 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67a2ff5700 1 -- 172.21.15.31:0/2653679414 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67b001c040 con 0x557ae01a8be0 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67d8f50b80 10 client.25871 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.954+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557adff10c60 con 0x557ae01a8be0 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67a2ff5700 1 -- 172.21.15.31:0/2653679414 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557ae0124d10 con 0x557ae01a8be0 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67a2ff5700 1 -- 172.21.15.31:0/2653679414 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f67b002c410 con 0x557ae01a8be0 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67d8f50b80 10 client.25871 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:16.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67d8f50b80 10 client.25871 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:16:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67d8f50b80 20 client.25871 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae01be560 0x557ae01c0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67d8f50b80 4 client.25871 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ae0177d70 con 0x557ae01be560 2023-07-29T16:16:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.955+0000 7f67be7fc700 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae01be560 0x557ae01c0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.956+0000 7f67a2ff5700 10 client.25871 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.956+0000 7f67be7fc700 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae01be560 0x557ae01c0970 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be6336364d6e5fbc server_cookie=78a013f7487f3005 in_seq=0 out_seq=0 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.956+0000 7f67a2ff5700 1 -- 172.21.15.31:0/2653679414 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557ae0177d70 con 0x557ae01be560 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.956+0000 7f67a2ff5700 10 client.25871 handle_command_reply: tid=0 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 client.25871 shutdown 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 msgr2=0x557ae01b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b2810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098f0 tx=0x7f67b0009260 comp rx=0 tx=0).stop 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 shutdown_connections 2023-07-29T16:16:16.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae01be560 0x557ae01c0970 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67a2ff5700 0 client.25871 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:16.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b2810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2653679414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 >> 172.21.15.31:0/2653679414 conn(0x557ae019e3e0 msgr2=0x557ae01a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 shutdown_connections 2023-07-29T16:16:16.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2653679414 wait complete. 2023-07-29T16:16:16.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.957+0000 7f67d8f50b80 20 client.25871 trim_cache size 0 max 16384 2023-07-29T16:16:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 Processor -- start 2023-07-29T16:16:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 -- start start 2023-07-29T16:16:16.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01a3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae019b6e0 con 0x557ae01a9530 2023-07-29T16:16:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae01b04e0 con 0x557ae01a8be0 2023-07-29T16:16:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae01b11b0 con 0x557ae01aa2f0 2023-07-29T16:16:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67be7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67be7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36202/0 (socket says 172.21.15.31:36202) 2023-07-29T16:16:16.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67be7fc700 1 -- 172.21.15.31:0/2280077339 learned_addr learned my addr 172.21.15.31:0/2280077339 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67beffd700 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67be7fc700 1 -- 172.21.15.31:0/2280077339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 msgr2=0x557ae01a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.971+0000 7f67be7fc700 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67be7fc700 1 -- 172.21.15.31:0/2280077339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 msgr2=0x557ae01b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67be7fc700 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67be7fc700 1 -- 172.21.15.31:0/2280077339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557adff4e250 con 0x557ae01a9530 2023-07-29T16:16:16.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67be7fc700 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b9d50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f67b40c8f70 tx=0x7f67b40c8e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cebf149b8acaec02 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67a5ffb700 1 -- 172.21.15.31:0/2280077339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b40b5070 con 0x557ae01a9530 2023-07-29T16:16:16.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67a5ffb700 1 -- 172.21.15.31:0/2280077339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67b4098bd0 con 0x557ae01a9530 2023-07-29T16:16:16.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.972+0000 7f67a5ffb700 1 -- 172.21.15.31:0/2280077339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4096d10 con 0x557ae01a9530 2023-07-29T16:16:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2280077339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 msgr2=0x557ae01b9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b9d50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f67b40c8f70 tx=0x7f67b40c8e40 comp rx=0 tx=0).stop 2023-07-29T16:16:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2280077339 shutdown_connections 2023-07-29T16:16:16.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a8be0 0x557ae01b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/2280077339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a9530 0x557ae01b9d50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2280077339 >> 172.21.15.31:0/2280077339 conn(0x557ae019e3e0 msgr2=0x557ae01a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2280077339 shutdown_connections 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 -- 172.21.15.31:0/2280077339 wait complete. 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 -- start start 2023-07-29T16:16:16.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.973+0000 7f67d8f50b80 1 Processor -- start 2023-07-29T16:16:16.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01b9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a9530 0x557ae01b8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae01b8bc0 con 0x557ae01a8be0 2023-07-29T16:16:16.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae01b8d30 con 0x557ae01a9530 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ae01b8ea0 con 0x557ae01aa2f0 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67d8f50b80 10 client.0 _ll_register_callbacks cb 0x557ae009f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01b9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67bf7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67beffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01b9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36208/0 (socket says 172.21.15.31:36208) 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67beffd700 1 -- 172.21.15.31:0/549217627 learned_addr learned my addr 172.21.15.31:0/549217627 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.974+0000 7f67be7fc700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a9530 0x557ae01b8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67beffd700 1 -- 172.21.15.31:0/549217627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 msgr2=0x557ae01b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67beffd700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67beffd700 1 -- 172.21.15.31:0/549217627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a9530 msgr2=0x557ae01b8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67beffd700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a9530 0x557ae01b8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67beffd700 1 -- 172.21.15.31:0/549217627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c01a0af0 con 0x557ae01a8be0 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67bf7fe700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67beffd700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01b9b30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009c40 tx=0x7f67b000fea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266b469f002671ab server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:16.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67a2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67a2ff5700 1 -- 172.21.15.31:0/549217627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0028070 con 0x557ae01a8be0 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67b800f040 con 0x557ae01a8be0 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67a2ff5700 1 -- 172.21.15.31:0/549217627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f67b00050d0 con 0x557ae01a8be0 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.975+0000 7f67a2ff5700 1 -- 172.21.15.31:0/549217627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67b001c050 con 0x557ae01a8be0 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 10 client.15903 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557adff262b0 con 0x557ae01a8be0 2023-07-29T16:16:16.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67a2ff5700 1 -- 172.21.15.31:0/549217627 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67b800f040 con 0x557ae01a8be0 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67a2ff5700 1 -- 172.21.15.31:0/549217627 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f67b0003680 con 0x557ae01a8be0 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 10 client.15903 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 10 client.15903 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 20 client.15903 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae0211430 0x557ae0213840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 4 client.15903 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557ae00f2d40 con 0x557ae0211430 2023-07-29T16:16:16.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.976+0000 7f67be7fc700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae0211430 0x557ae0213840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67a2ff5700 10 client.15903 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67be7fc700 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae0211430 0x557ae0213840 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73e0a939758b163a server_cookie=2aafbfc745f0f39b in_seq=0 out_seq=0 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67a2ff5700 1 -- 172.21.15.31:0/549217627 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557ae00f2d40 con 0x557ae0211430 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67a2ff5700 10 client.15903 handle_command_reply: tid=0 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 client.15903 shutdown 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 msgr2=0x557ae01b9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01b9b30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009c40 tx=0x7f67b000fea0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 shutdown_connections 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x557ae0211430 0x557ae0213840 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557ae01aa2f0 0x557ae01b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ae01a8be0 0x557ae01b9b30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67a2ff5700 0 client.15903 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 --2- 172.21.15.31:0/549217627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ae01a9530 0x557ae01b8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 >> 172.21.15.31:0/549217627 conn(0x557ae019e3e0 msgr2=0x557ae01b1480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 shutdown_connections 2023-07-29T16:16:16.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.977+0000 7f67d8f50b80 1 -- 172.21.15.31:0/549217627 wait complete. 2023-07-29T16:16:16.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67d8f50b80 20 client.15903 trim_cache size 0 max 16384 2023-07-29T16:16:16.989 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:16:16.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67a80a0810 msgr2=0x7f67a80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67a80a0810 0x7f67a80a2cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f67b00098c0 tx=0x7f67b0019040 comp rx=0 tx=0).stop 2023-07-29T16:16:16.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 msgr2=0x7f67c0198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:16.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 0x7f67c0198920 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f67b400b550 tx=0x7f67b400b860 comp rx=0 tx=0).stop 2023-07-29T16:16:16.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 shutdown_connections 2023-07-29T16:16:16.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f67a80a0810 0x7f67a80a2cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f67c0101ea0 0x7f67c019f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f67c0100320 0x7f67c01983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 --2- 172.21.15.31:0/1263147545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f67c0101510 0x7f67c0198920 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:16.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 >> 172.21.15.31:0/1263147545 conn(0x7f67c00fb920 msgr2=0x7f67c0106ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:16.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 shutdown_connections 2023-07-29T16:16:16.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:16.978+0000 7f67c55ff700 1 -- 172.21.15.31:0/1263147545 wait complete. 2023-07-29T16:16:16.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:16:17.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.205+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/29469534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 msgr2=0x7f0b08100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.205+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/29469534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b08100730 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0afc009990 tx=0x7f0afc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/29469534 shutdown_connections 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/29469534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b08101ea0 0x7f0b08108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/29469534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b08100730 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/29469534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 0x7f0b08101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/29469534 >> 172.21.15.31:0/29469534 conn(0x7f0b080fb920 msgr2=0x7f0b080fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/29469534 shutdown_connections 2023-07-29T16:16:17.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.206+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/29469534 wait complete. 2023-07-29T16:16:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 Processor -- start 2023-07-29T16:16:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 -- start start 2023-07-29T16:16:17.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b081960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 0x7f0b08196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b08101ea0 0x7f0b0819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0819dc50 con 0x7f0b08100320 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0819dd90 con 0x7f0b08101ea0 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0f1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0819df20 con 0x7f0b08101510 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0cf9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b081960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b07fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 0x7f0b08196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0cf9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b081960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36226/0 (socket says 172.21.15.31:36226) 2023-07-29T16:16:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.207+0000 7f0b0cf9b700 1 -- 172.21.15.31:0/3616870897 learned_addr learned my addr 172.21.15.31:0/3616870897 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b08101ea0 0x7f0b0819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b07fff700 1 -- 172.21.15.31:0/3616870897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b08101ea0 msgr2=0x7f0b0819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b07fff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b08101ea0 0x7f0b0819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b07fff700 1 -- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 msgr2=0x7f0b081960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b07fff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b081960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b07fff700 1 -- 172.21.15.31:0/3616870897 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0afc0095f0 con 0x7f0b08101510 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b07fff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 0x7f0b08196610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0af400e9f0 tx=0x7f0af400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b05ffb700 1 -- 172.21.15.31:0/3616870897 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af400c9c0 con 0x7f0b08101510 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0819e200 con 0x7f0b08101510 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b05ffb700 1 -- 172.21.15.31:0/3616870897 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0af4010430 con 0x7f0b08101510 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.208+0000 7f0b05ffb700 1 -- 172.21.15.31:0/3616870897 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af4018680 con 0x7f0b08101510 2023-07-29T16:16:17.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.209+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0819e760 con 0x7f0b08101510 2023-07-29T16:16:17.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.212+0000 7f0b05ffb700 1 -- 172.21.15.31:0/3616870897 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0af4010860 con 0x7f0b08101510 2023-07-29T16:16:17.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.213+0000 7f0b22b89b80 1 Processor -- start 2023-07-29T16:16:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.213+0000 7f0b22b89b80 1 -- start start 2023-07-29T16:16:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88efd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88f0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc8846a10 con 0x55bfc88f02f0 2023-07-29T16:16:17.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc88443d0 con 0x55bfc88ef530 2023-07-29T16:16:17.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc883fcc0 con 0x55bfc88eebe0 2023-07-29T16:16:17.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b07fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88f0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41042/0 (socket says 172.21.15.31:41042) 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 -- 172.21.15.31:0/3454310807 learned_addr learned my addr 172.21.15.31:0/3454310807 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 -- 172.21.15.31:0/3454310807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 msgr2=0x55bfc88eeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88eeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 -- 172.21.15.31:0/3454310807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 msgr2=0x55bfc88f0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88f0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.214+0000 7f0b0cf9b700 1 -- 172.21.15.31:0/3454310807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfc866dde0 con 0x55bfc88ef530 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0b05ffb700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0af80a0790 0x7f0af80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88efd60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0afc005ca0 tx=0x7f0afc005ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f18633377d2761f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0b05ffb700 1 -- 172.21.15.31:0/3616870897 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0af4014070 con 0x7f0b08101510 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0af80a0790 0x7f0af80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0af2ffd700 1 -- 172.21.15.31:0/3454310807 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc01d070 con 0x55bfc88ef530 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0af2ffd700 1 -- 172.21.15.31:0/3454310807 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0afc01e040 con 0x55bfc88ef530 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.215+0000 7f0af2ffd700 1 -- 172.21.15.31:0/3454310807 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc0049f0 con 0x55bfc88ef530 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 -- 172.21.15.31:0/3454310807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 msgr2=0x55bfc88efd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0af80a0790 0x7f0af80a2c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55bfc88ee980 tx=0x7f0b00009100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88efd60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0afc005ca0 tx=0x7f0afc005ee0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 -- 172.21.15.31:0/3454310807 shutdown_connections 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88f0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88eeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/3454310807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88efd60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 -- 172.21.15.31:0/3454310807 >> 172.21.15.31:0/3454310807 conn(0x55bfc88e43e0 msgr2=0x55bfc88e9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 -- 172.21.15.31:0/3454310807 shutdown_connections 2023-07-29T16:16:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.216+0000 7f0b22b89b80 1 -- 172.21.15.31:0/3454310807 wait complete. 2023-07-29T16:16:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 -- start start 2023-07-29T16:16:17.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 Processor -- start 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88f8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88f8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc87dc510 con 0x55bfc88f02f0 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc87dc1b0 con 0x55bfc88ef530 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc86c3920 con 0x55bfc88eebe0 2023-07-29T16:16:17.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.217+0000 7f0b22b89b80 10 client.0 _ll_register_callbacks cb 0x55bfc87e5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88ffce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36244/0 (socket says 172.21.15.31:36244) 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b07fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88f8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1613594101 learned_addr learned my addr 172.21.15.31:0/1613594101 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1613594101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 msgr2=0x55bfc88f8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88f8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 msgr2=0x55bfc88f8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88f8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1613594101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b08109090 con 0x55bfc88f02f0 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b07fff700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88f8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88ffce0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0b000077b0 tx=0x7f0b00004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919ad2d6d0a7580c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.218+0000 7f0ae7fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1613594101 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00023640 con 0x55bfc88f02f0 2023-07-29T16:16:17.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bfc886ad10 con 0x55bfc88f02f0 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1613594101 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0b000048b0 con 0x55bfc88f02f0 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1613594101 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b000237a0 con 0x55bfc88f02f0 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0b22b89b80 10 client.15915 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bfc8656c60 con 0x55bfc88f02f0 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1613594101 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bfc886ad10 con 0x55bfc88f02f0 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1613594101 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0b00016b10 con 0x55bfc88f02f0 2023-07-29T16:16:17.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0b22b89b80 10 client.15915 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.219+0000 7f0b22b89b80 10 client.15915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.220+0000 7f0b22b89b80 20 client.15915 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.220+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc89045f0 0x55bfc8906a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.220+0000 7f0b22b89b80 4 client.15915 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.220+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bfc88bdd70 con 0x55bfc89045f0 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.220+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc89045f0 0x55bfc8906a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.221+0000 7f0ae7fff700 10 client.15915 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.221+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc89045f0 0x55bfc8906a00 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9aa33b863adbc2e server_cookie=cd8e79ded38fbacd in_seq=0 out_seq=0 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.221+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1613594101 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bfc88bdd70 con 0x55bfc89045f0 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0ae7fff700 10 client.15915 handle_command_reply: tid=0 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 client.15915 shutdown 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 msgr2=0x55bfc88ffce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88ffce0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0b000077b0 tx=0x7f0b00004ad0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 shutdown_connections 2023-07-29T16:16:17.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc89045f0 0x55bfc8906a00 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88ffce0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88f8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0ae7fff700 0 client.15915 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1613594101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88f8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 >> 172.21.15.31:0/1613594101 conn(0x55bfc88e43e0 msgr2=0x55bfc88e9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 shutdown_connections 2023-07-29T16:16:17.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1613594101 wait complete. 2023-07-29T16:16:17.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.222+0000 7f0b22b89b80 20 client.15915 trim_cache size 0 max 16384 2023-07-29T16:16:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 Processor -- start 2023-07-29T16:16:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 -- start start 2023-07-29T16:16:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ea690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88ef530 0x55bfc88ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 0x55bfc88e9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc89042e0 con 0x55bfc88ef530 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc88e1770 con 0x55bfc88f02f0 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc88f64f0 con 0x55bfc88eebe0 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b07fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88ef530 0x55bfc88ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 0x55bfc88e9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 0x55bfc88e9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41054/0 (socket says 172.21.15.31:41054) 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b0d79c700 1 -- 172.21.15.31:0/2815982479 learned_addr learned my addr 172.21.15.31:0/2815982479 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.236+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ea690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0d79c700 1 -- 172.21.15.31:0/2815982479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 msgr2=0x55bfc88ea690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ea690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0d79c700 1 -- 172.21.15.31:0/2815982479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88ef530 msgr2=0x55bfc88ffc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88ef530 0x55bfc88ffc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0d79c700 1 -- 172.21.15.31:0/2815982479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfc8694250 con 0x55bfc88f02f0 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 0x55bfc88e9060 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0b00018040 tx=0x7f0b00022590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f8375f2c15c87c1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0af37fe700 1 -- 172.21.15.31:0/2815982479 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00007da0 con 0x55bfc88f02f0 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0af37fe700 1 -- 172.21.15.31:0/2815982479 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b0000f040 con 0x55bfc88f02f0 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.237+0000 7f0af37fe700 1 -- 172.21.15.31:0/2815982479 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00016690 con 0x55bfc88f02f0 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 -- 172.21.15.31:0/2815982479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 msgr2=0x55bfc88e9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 0x55bfc88e9060 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0b00018040 tx=0x7f0b00022590 comp rx=0 tx=0).stop 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 -- 172.21.15.31:0/2815982479 shutdown_connections 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88f02f0 0x55bfc88e9060 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ea690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/2815982479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88ef530 0x55bfc88ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 -- 172.21.15.31:0/2815982479 >> 172.21.15.31:0/2815982479 conn(0x55bfc88e43e0 msgr2=0x55bfc88e9900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 -- 172.21.15.31:0/2815982479 shutdown_connections 2023-07-29T16:16:17.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 -- 172.21.15.31:0/2815982479 wait complete. 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.238+0000 7f0b22b89b80 1 -- start start 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 Processor -- start 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ff410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88ff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc88fc3f0 con 0x55bfc88f02f0 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc88fc560 con 0x55bfc88ef530 2023-07-29T16:16:17.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfc88fc720 con 0x55bfc88eebe0 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b22b89b80 10 client.0 _ll_register_callbacks cb 0x55bfc87e5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b07fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b0d79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88fbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36272/0 (socket says 172.21.15.31:36272) 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1636143908 learned_addr learned my addr 172.21.15.31:0/1636143908 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.239+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ff410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1636143908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 msgr2=0x55bfc88ff410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ff410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 msgr2=0x55bfc88ff950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88ff950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b0d79c700 1 -- 172.21.15.31:0/1636143908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfc8656c60 con 0x55bfc88f02f0 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b07fff700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b0d79c700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88fbeb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0b00004010 tx=0x7f0b00009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244a5fe20c9105da server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0ae7fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1636143908 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00004890 con 0x55bfc88f02f0 2023-07-29T16:16:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0afc01e040 con 0x55bfc88f02f0 2023-07-29T16:16:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1636143908 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0b000049f0 con 0x55bfc88f02f0 2023-07-29T16:16:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1636143908 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b00004bf0 con 0x55bfc88f02f0 2023-07-29T16:16:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b22b89b80 10 client.15921 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:17.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.240+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bfc866c2b0 con 0x55bfc88f02f0 2023-07-29T16:16:17.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1636143908 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0afc01e040 con 0x55bfc88f02f0 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1636143908 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0b0003f070 con 0x55bfc88f02f0 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b22b89b80 10 client.15921 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b22b89b80 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b22b89b80 20 client.15921 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc8957100 0x55bfc8959510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b22b89b80 4 client.15921 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:16:17.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bfc8838d40 con 0x55bfc8957100 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.241+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc8957100 0x55bfc8959510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b0cf9b700 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc8957100 0x55bfc8959510 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d1bb4857763d4c8 server_cookie=34f854f4e342a774 in_seq=0 out_seq=0 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0ae7fff700 10 client.15921 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0ae7fff700 1 -- 172.21.15.31:0/1636143908 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bfc8838d40 con 0x55bfc8957100 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0ae7fff700 10 client.15921 handle_command_reply: tid=0 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 client.15921 shutdown 2023-07-29T16:16:17.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 msgr2=0x55bfc88fbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88fbeb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0b00004010 tx=0x7f0b00009f70 comp rx=0 tx=0).stop 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 shutdown_connections 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bfc8957100 0x55bfc8959510 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bfc88f02f0 0x55bfc88fbeb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bfc88eebe0 0x55bfc88ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 --2- 172.21.15.31:0/1636143908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bfc88ef530 0x55bfc88ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0ae7fff700 0 client.15921 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 >> 172.21.15.31:0/1636143908 conn(0x55bfc88e43e0 msgr2=0x55bfc88f6740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 shutdown_connections 2023-07-29T16:16:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.242+0000 7f0b22b89b80 1 -- 172.21.15.31:0/1636143908 wait complete. 2023-07-29T16:16:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b22b89b80 20 client.15921 trim_cache size 0 max 16384 2023-07-29T16:16:17.250 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:16:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0af80a0790 msgr2=0x7f0af80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0af80a0790 0x7f0af80a2c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55bfc88ee980 tx=0x7f0b00009100 comp rx=0 tx=0).stop 2023-07-29T16:16:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 msgr2=0x7f0b08196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 0x7f0b08196610 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0af400e9f0 tx=0x7f0af400ed00 comp rx=0 tx=0).stop 2023-07-29T16:16:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 shutdown_connections 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0af80a0790 0x7f0af80a2c40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b08101ea0 0x7f0b0819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b08100320 0x7f0b081960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 --2- 172.21.15.31:0/3616870897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b08101510 0x7f0b08196610 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 >> 172.21.15.31:0/3616870897 conn(0x7f0b080fb920 msgr2=0x7f0b080fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 shutdown_connections 2023-07-29T16:16:17.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:17.243+0000 7f0b0f1ff700 1 -- 172.21.15.31:0/3616870897 wait complete. 2023-07-29T16:16:17.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:17 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/155646391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:16:17.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:17 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/155646391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:16:17.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:17 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/155646391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:16:17.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 -- 172.21.15.31:0/912408441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08100340 msgr2=0x7fac08100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/912408441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08100340 0x7fac08100750 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fabf8009990 tx=0x7fabf8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 -- 172.21.15.31:0/912408441 shutdown_connections 2023-07-29T16:16:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/912408441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac08101ec0 0x7fac08108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/912408441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08100340 0x7fac08100750 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/912408441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac08101530 0x7fac08101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 -- 172.21.15.31:0/912408441 >> 172.21.15.31:0/912408441 conn(0x7fac080fb920 msgr2=0x7fac080fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.863+0000 7fac0d1ff700 1 -- 172.21.15.31:0/912408441 shutdown_connections 2023-07-29T16:16:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.864+0000 7fac0d1ff700 1 -- 172.21.15.31:0/912408441 wait complete. 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.864+0000 7fac0d1ff700 1 Processor -- start 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 -- start start 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac08100340 0x7fac081983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 0x7fac081988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac08101ec0 0x7fac08199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08199dc0 con 0x7fac08101530 2023-07-29T16:16:17.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac08199f30 con 0x7fac08100340 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0d1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0819a0f0 con 0x7fac08101ec0 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 0x7fac081988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 0x7fac081988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36282/0 (socket says 172.21.15.31:36282) 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac0659c700 1 -- 172.21.15.31:0/4231287869 learned_addr learned my addr 172.21.15.31:0/4231287869 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.865+0000 7fac06d9d700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac08100340 0x7fac081983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0659c700 1 -- 172.21.15.31:0/4231287869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac08101ec0 msgr2=0x7fac08199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0659c700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac08101ec0 0x7fac08199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0659c700 1 -- 172.21.15.31:0/4231287869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac08100340 msgr2=0x7fac081983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0659c700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac08100340 0x7fac081983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0659c700 1 -- 172.21.15.31:0/4231287869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80095f0 con 0x7fac08101530 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0659c700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 0x7fac081988e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00e9a0 tx=0x7fabfc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fabeffff700 1 -- 172.21.15.31:0/4231287869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc009790 con 0x7fac08101530 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fabeffff700 1 -- 172.21.15.31:0/4231287869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabfc00cbd0 con 0x7fac08101530 2023-07-29T16:16:17.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0819a3d0 con 0x7fac08101530 2023-07-29T16:16:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fabeffff700 1 -- 172.21.15.31:0/4231287869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc018680 con 0x7fac08101530 2023-07-29T16:16:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.866+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac081a0b90 con 0x7fac08101530 2023-07-29T16:16:17.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.870+0000 7fabeffff700 1 -- 172.21.15.31:0/4231287869 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fabfc00cd40 con 0x7fac08101530 2023-07-29T16:16:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.871+0000 7fabeffff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fabf00a0790 0x7fabf00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.871+0000 7fac20a50b80 1 Processor -- start 2023-07-29T16:16:17.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.871+0000 7fabeffff700 1 -- 172.21.15.31:0/4231287869 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fabfc014070 con 0x7fac08101530 2023-07-29T16:16:17.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.871+0000 7fac20a50b80 1 -- start start 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.871+0000 7fac06d9d700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fabf00a0790 0x7fabf00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc0016b0 0x55c4dc1a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1a7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac20a50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc0fd9b0 con 0x55c4dc0016b0 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac20a50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc0fb370 con 0x55c4dc1a5df0 2023-07-29T16:16:17.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac20a50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc0f6c60 con 0x55c4dc1a6cf0 2023-07-29T16:16:17.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac06d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc0016b0 0x55c4dc1a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41094/0 (socket says 172.21.15.31:41094) 2023-07-29T16:16:17.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 -- 172.21.15.31:0/3503601618 learned_addr learned my addr 172.21.15.31:0/3503601618 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0759e700 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1a7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac06d9d700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fabf00a0790 0x7fabf00a2c40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006010 tx=0x7fabf80051f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 -- 172.21.15.31:0/3503601618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 msgr2=0x55c4dc1a7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1a7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 -- 172.21.15.31:0/3503601618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc0016b0 msgr2=0x55c4dc1a58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc0016b0 0x55c4dc1a58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac0659c700 1 -- 172.21.15.31:0/3503601618 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4dbf24d80 con 0x55c4dc1a5df0 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.872+0000 7fac06d9d700 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc0016b0 0x55c4dc1a58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:17.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.873+0000 7fac0659c700 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a67b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0994a0 tx=0x7fabfc0990d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b2df0f89d97d2fb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.873+0000 7fabed7fa700 1 -- 172.21.15.31:0/3503601618 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc099510 con 0x55c4dc1a5df0 2023-07-29T16:16:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.873+0000 7fabed7fa700 1 -- 172.21.15.31:0/3503601618 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabfc09d540 con 0x55c4dc1a5df0 2023-07-29T16:16:17.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.873+0000 7fabed7fa700 1 -- 172.21.15.31:0/3503601618 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc098c60 con 0x55c4dc1a5df0 2023-07-29T16:16:17.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 -- 172.21.15.31:0/3503601618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 msgr2=0x55c4dc1a67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a67b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0994a0 tx=0x7fabfc0990d0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 -- 172.21.15.31:0/3503601618 shutdown_connections 2023-07-29T16:16:17.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1a7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc0016b0 0x55c4dc1a58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 --2- 172.21.15.31:0/3503601618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a67b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 -- 172.21.15.31:0/3503601618 >> 172.21.15.31:0/3503601618 conn(0x55c4dc19afd0 msgr2=0x55c4dc1a0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 -- 172.21.15.31:0/3503601618 shutdown_connections 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 -- 172.21.15.31:0/3503601618 wait complete. 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.874+0000 7fac20a50b80 1 -- start start 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 Processor -- start 2023-07-29T16:16:17.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1af480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc0934b0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc093150 con 0x55c4dc0016b0 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dbf7a8c0 con 0x55c4dc1a6cf0 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac20a50b80 10 client.0 _ll_register_callbacks cb 0x55c4dc09c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:17.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.875+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1af480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36294/0 (socket says 172.21.15.31:36294) 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 -- 172.21.15.31:0/692014954 learned_addr learned my addr 172.21.15.31:0/692014954 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac06d9d700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0759e700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 -- 172.21.15.31:0/692014954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 msgr2=0x55c4dc1b64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 -- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 msgr2=0x55c4dc1aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 -- 172.21.15.31:0/692014954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac081090b0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac0659c700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1af480 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00e970 tx=0x7fabfc0a1960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520dee11d8db61f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fabeaff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fabeaff5700 1 -- 172.21.15.31:0/692014954 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0a16d0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4dc121cb0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.876+0000 7fabeaff5700 1 -- 172.21.15.31:0/692014954 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fabfc095300 con 0x55c4dc1a5df0 2023-07-29T16:16:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fabeaff5700 1 -- 172.21.15.31:0/692014954 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabfc0954c0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 10 client.15933 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fac081a0b90 con 0x55c4dc1a5df0 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fabeaff5700 1 -- 172.21.15.31:0/692014954 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabfc065020 con 0x55c4dc1a5df0 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fabeaff5700 1 -- 172.21.15.31:0/692014954 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fabfc09a2f0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 10 client.15933 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 10 client.15933 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 20 client.15933 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:17.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc1baa10 0x55c4dc1bce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 4 client.15933 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:17.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4dc174d10 con 0x55c4dc1baa10 2023-07-29T16:16:17.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.877+0000 7fac06d9d700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc1baa10 0x55c4dc1bce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.878+0000 7fac06d9d700 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc1baa10 0x55c4dc1bce20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=214eabf362bbb2b4 server_cookie=63758ecce80f9ae0 in_seq=0 out_seq=0 2023-07-29T16:16:17.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.878+0000 7fabeaff5700 10 client.15933 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fabeaff5700 1 -- 172.21.15.31:0/692014954 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c4dc174d10 con 0x55c4dc1baa10 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fabeaff5700 10 client.15933 handle_command_reply: tid=0 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 client.15933 shutdown 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 msgr2=0x55c4dc1af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1af480 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00e970 tx=0x7fabfc0a1960 comp rx=0 tx=0).stop 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 shutdown_connections 2023-07-29T16:16:17.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc1baa10 0x55c4dc1bce20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fabeaff5700 0 client.15933 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 --2- 172.21.15.31:0/692014954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1af480 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 >> 172.21.15.31:0/692014954 conn(0x55c4dc19afd0 msgr2=0x55c4dc19fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 shutdown_connections 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 1 -- 172.21.15.31:0/692014954 wait complete. 2023-07-29T16:16:17.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.879+0000 7fac20a50b80 20 client.15933 trim_cache size 0 max 16384 2023-07-29T16:16:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.894+0000 7fac20a50b80 1 Processor -- start 2023-07-29T16:16:17.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 -- start start 2023-07-29T16:16:17.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc1afe40 con 0x55c4dc1a5df0 2023-07-29T16:16:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc1ae3f0 con 0x55c4dc1a6cf0 2023-07-29T16:16:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac20a50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc1aeab0 con 0x55c4dc0016b0 2023-07-29T16:16:17.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36306/0 (socket says 172.21.15.31:36306) 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac06d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac0659c700 1 -- 172.21.15.31:0/4277172914 learned_addr learned my addr 172.21.15.31:0/4277172914 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.895+0000 7fac0759e700 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fac0659c700 1 -- 172.21.15.31:0/4277172914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc0016b0 msgr2=0x55c4dc1b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fac0659c700 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fac0659c700 1 -- 172.21.15.31:0/4277172914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a6cf0 msgr2=0x55c4dc1aa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fac0659c700 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1aa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fac0659c700 1 -- 172.21.15.31:0/4277172914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4dbf0dc00 con 0x55c4dc1a5df0 2023-07-29T16:16:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fac0659c700 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a9ca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0994a0 tx=0x7fabfc097ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec91b1f2f040b5f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fabedffb700 1 -- 172.21.15.31:0/4277172914 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc09a4f0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fabedffb700 1 -- 172.21.15.31:0/4277172914 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabfc065040 con 0x55c4dc1a5df0 2023-07-29T16:16:17.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.896+0000 7fabedffb700 1 -- 172.21.15.31:0/4277172914 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0955f0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 -- 172.21.15.31:0/4277172914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 msgr2=0x55c4dc1a9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a9ca0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0994a0 tx=0x7fabfc097ac0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 -- 172.21.15.31:0/4277172914 shutdown_connections 2023-07-29T16:16:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 --2- 172.21.15.31:0/4277172914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1a9ca0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 -- 172.21.15.31:0/4277172914 >> 172.21.15.31:0/4277172914 conn(0x55c4dc19afd0 msgr2=0x55c4dc1ad150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 -- 172.21.15.31:0/4277172914 shutdown_connections 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 -- 172.21.15.31:0/4277172914 wait complete. 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.897+0000 7fac20a50b80 1 -- start start 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 Processor -- start 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1b41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc1b2a60 con 0x55c4dc1a5df0 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc1b2bd0 con 0x55c4dc0016b0 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4dc1b2d90 con 0x55c4dc1a6cf0 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac20a50b80 10 client.0 _ll_register_callbacks cb 0x55c4dc09c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.898+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1b41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1b41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36312/0 (socket says 172.21.15.31:36312) 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 -- 172.21.15.31:0/1550475831 learned_addr learned my addr 172.21.15.31:0/1550475831 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac06d9d700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0759e700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 -- 172.21.15.31:0/1550475831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 msgr2=0x55c4dc1b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 -- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 msgr2=0x55c4dc1b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 -- 172.21.15.31:0/1550475831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac081a0b90 con 0x55c4dc1a5df0 2023-07-29T16:16:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac0659c700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1b41a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fabfc09a6b0 tx=0x7fabfc095980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3d81c937340e9b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fabeaff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fabeaff5700 1 -- 172.21.15.31:0/1550475831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc095570 con 0x55c4dc1a5df0 2023-07-29T16:16:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fabeaff5700 1 -- 172.21.15.31:0/1550475831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fabfc0c39f0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.899+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabfc09d540 con 0x55c4dc1a5df0 2023-07-29T16:16:17.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fabeaff5700 1 -- 172.21.15.31:0/1550475831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabfc0c3bb0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 10 client.15939 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55c4dbf4b1f0 con 0x55c4dc1a5df0 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fabeaff5700 1 -- 172.21.15.31:0/1550475831 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabfc09d540 con 0x55c4dc1a5df0 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fabeaff5700 1 -- 172.21.15.31:0/1550475831 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fabfc010430 con 0x55c4dc1a5df0 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 10 client.15939 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 10 client.15939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:17.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 20 client.15939 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc20d430 0x55c4dc20f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 4 client.15939 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:16:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c4dc0efce0 con 0x55c4dc20d430 2023-07-29T16:16:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.900+0000 7fac06d9d700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc20d430 0x55c4dc20f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.901+0000 7fac06d9d700 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc20d430 0x55c4dc20f810 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f064f003a402f43b server_cookie=8bb904ef110414f3 in_seq=0 out_seq=0 2023-07-29T16:16:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.901+0000 7fabeaff5700 10 client.15939 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.905+0000 7fabeaff5700 1 -- 172.21.15.31:0/1550475831 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c4dc0efce0 con 0x55c4dc20d430 2023-07-29T16:16:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.905+0000 7fabeaff5700 10 client.15939 handle_command_reply: tid=0 2023-07-29T16:16:17.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.905+0000 7fac20a50b80 1 client.15939 shutdown 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 msgr2=0x55c4dc1b41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1b41a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fabfc09a6b0 tx=0x7fabfc095980 comp rx=0 tx=0).stop 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 shutdown_connections 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c4dc20d430 0x55c4dc20f810 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4dc1a6cf0 0x55c4dc1b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c4dc0016b0 0x55c4dc1b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fabeaff5700 0 client.15939 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 --2- 172.21.15.31:0/1550475831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4dc1a5df0 0x55c4dc1b41a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 >> 172.21.15.31:0/1550475831 conn(0x55c4dc19afd0 msgr2=0x55c4dc1a9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 shutdown_connections 2023-07-29T16:16:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 1 -- 172.21.15.31:0/1550475831 wait complete. 2023-07-29T16:16:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac20a50b80 20 client.15939 trim_cache size 0 max 16384 2023-07-29T16:16:17.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:16:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:16:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:16:17.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fabf00a0790 msgr2=0x7fabf00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fabf00a0790 0x7fabf00a2c40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fabf8006010 tx=0x7fabf80051f0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 msgr2=0x7fac081988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.906+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 0x7fac081988e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fabfc00e9a0 tx=0x7fabfc00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 shutdown_connections 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fabf00a0790 0x7fabf00a2c40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac08101ec0 0x7fac08199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac08100340 0x7fac081983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 --2- 172.21.15.31:0/4231287869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac08101530 0x7fac081988e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 >> 172.21.15.31:0/4231287869 conn(0x7fac080fb920 msgr2=0x7fac080fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 shutdown_connections 2023-07-29T16:16:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:17.907+0000 7fac0d1ff700 1 -- 172.21.15.31:0/4231287869 wait complete. 2023-07-29T16:16:18.261 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:16:18.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:16:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 -- 172.21.15.31:0/3121271734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 msgr2=0x7f832c073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 --2- 172.21.15.31:0/3121271734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c073ba0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8314009990 tx=0x7f8314009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 -- 172.21.15.31:0/3121271734 shutdown_connections 2023-07-29T16:16:18.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 --2- 172.21.15.31:0/3121271734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c1081e0 0x7f832c10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 --2- 172.21.15.31:0/3121271734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f832c074d10 0x7f832c073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 --2- 172.21.15.31:0/3121271734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c073ba0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 -- 172.21.15.31:0/3121271734 >> 172.21.15.31:0/3121271734 conn(0x7f832c0fb970 msgr2=0x7f832c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 -- 172.21.15.31:0/3121271734 shutdown_connections 2023-07-29T16:16:18.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.473+0000 7f83327ff700 1 -- 172.21.15.31:0/3121271734 wait complete. 2023-07-29T16:16:18.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.474+0000 7f83327ff700 1 Processor -- start 2023-07-29T16:16:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.474+0000 7f83327ff700 1 -- start start 2023-07-29T16:16:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f83327ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f83327ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c074d10 0x7f832c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f83327ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f832c1081e0 0x7f832c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f83327ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c19fde0 con 0x7f832c073730 2023-07-29T16:16:18.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f83327ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c19ff20 con 0x7f832c074d10 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f83327ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c1a00b0 con 0x7f832c1081e0 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f832bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8323fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c074d10 0x7f832c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f832bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36328/0 (socket says 172.21.15.31:36328) 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f832bfff700 1 -- 172.21.15.31:0/4162001113 learned_addr learned my addr 172.21.15.31:0/4162001113 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8330d9c700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f832c1081e0 0x7f832c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8323fff700 1 -- 172.21.15.31:0/4162001113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f832c1081e0 msgr2=0x7f832c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8323fff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f832c1081e0 0x7f832c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8323fff700 1 -- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 msgr2=0x7f832c198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8323fff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f8323fff700 1 -- 172.21.15.31:0/4162001113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83140095f0 con 0x7f832c074d10 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.475+0000 7f832bfff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.476+0000 7f8323fff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c074d10 0x7f832c1987a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f831400b680 tx=0x7f8314005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.476+0000 7f8329ffb700 1 -- 172.21.15.31:0/4162001113 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831401d070 con 0x7f832c074d10 2023-07-29T16:16:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.476+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c1a0330 con 0x7f832c074d10 2023-07-29T16:16:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.476+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c1a07b0 con 0x7f832c074d10 2023-07-29T16:16:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.476+0000 7f8329ffb700 1 -- 172.21.15.31:0/4162001113 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8314021410 con 0x7f832c074d10 2023-07-29T16:16:18.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.476+0000 7f8329ffb700 1 -- 172.21.15.31:0/4162001113 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314017650 con 0x7f832c074d10 2023-07-29T16:16:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.479+0000 7f8329ffb700 1 -- 172.21.15.31:0/4162001113 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f83140217f0 con 0x7f832c074d10 2023-07-29T16:16:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.480+0000 7f83460d8b80 1 Processor -- start 2023-07-29T16:16:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f8329ffb700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83180a0790 0x7f83180a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 -- start start 2023-07-29T16:16:18.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f8329ffb700 1 -- 172.21.15.31:0/4162001113 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f83140ccc90 con 0x7f832c074d10 2023-07-29T16:16:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f8330d9c700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83180a0790 0x7f83180a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408bc60 0x55fef408c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef408cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef408d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef3fe3a90 con 0x55fef408c5b0 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef3fe1450 con 0x55fef408d370 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f83460d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef3fdcd40 con 0x55fef408bc60 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f8323fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef408cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.481+0000 7f8323fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef408cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36342/0 (socket says 172.21.15.31:36342) 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 -- 172.21.15.31:0/3928443466 learned_addr learned my addr 172.21.15.31:0/3928443466 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f832bfff700 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408bc60 0x55fef408c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8330d9c700 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef408d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 -- 172.21.15.31:0/3928443466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408bc60 msgr2=0x55fef408c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408bc60 0x55fef408c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 -- 172.21.15.31:0/3928443466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 msgr2=0x55fef408d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef408d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 -- 172.21.15.31:0/3928443466 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fef3e0ae60 con 0x55fef408c5b0 2023-07-29T16:16:18.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8330d9c700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83180a0790 0x7f83180a2c40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f832400f540 tx=0x7f8324009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f8323fff700 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef408cde0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f83140a2250 tx=0x7f8314099f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a78ded100f7c568 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f83227fc700 1 -- 172.21.15.31:0/3928443466 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831409a3c0 con 0x55fef408c5b0 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f83227fc700 1 -- 172.21.15.31:0/3928443466 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f831409e3f0 con 0x55fef408c5b0 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.482+0000 7f83227fc700 1 -- 172.21.15.31:0/3928443466 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831409a620 con 0x55fef408c5b0 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 -- 172.21.15.31:0/3928443466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 msgr2=0x55fef408cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef408cde0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f83140a2250 tx=0x7f8314099f00 comp rx=0 tx=0).stop 2023-07-29T16:16:18.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 -- 172.21.15.31:0/3928443466 shutdown_connections 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef408d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408bc60 0x55fef408c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3928443466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef408cde0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 -- 172.21.15.31:0/3928443466 >> 172.21.15.31:0/3928443466 conn(0x55fef4081460 msgr2=0x55fef4086a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 -- 172.21.15.31:0/3928443466 shutdown_connections 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.483+0000 7f83460d8b80 1 -- 172.21.15.31:0/3928443466 wait complete. 2023-07-29T16:16:18.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.484+0000 7f83460d8b80 1 -- start start 2023-07-29T16:16:18.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.484+0000 7f83460d8b80 1 Processor -- start 2023-07-29T16:16:18.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 0x55fef4095420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408c5b0 0x55fef4095960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef409c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef3f79590 con 0x55fef408bc60 2023-07-29T16:16:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef3f79230 con 0x55fef408d370 2023-07-29T16:16:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef3e609a0 con 0x55fef408c5b0 2023-07-29T16:16:18.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f83460d8b80 10 client.0 _ll_register_callbacks cb 0x55fef3f826a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f832bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 0x55fef4095420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f832bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 0x55fef4095420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36352/0 (socket says 172.21.15.31:36352) 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f832bfff700 1 -- 172.21.15.31:0/1112016920 learned_addr learned my addr 172.21.15.31:0/1112016920 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.485+0000 7f8330d9c700 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef409c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f832bfff700 1 -- 172.21.15.31:0/1112016920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408c5b0 msgr2=0x55fef4095960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f832bfff700 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408c5b0 0x55fef4095960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f832bfff700 1 -- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 msgr2=0x55fef409c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f832bfff700 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef409c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f832bfff700 1 -- 172.21.15.31:0/1112016920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c1a07b0 con 0x55fef408bc60 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f832bfff700 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 0x55fef4095420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f831c011890 tx=0x7f831c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f768484ce1281d2c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f830b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f830b7fe700 1 -- 172.21.15.31:0/1112016920 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c016070 con 0x55fef408bc60 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f830b7fe700 1 -- 172.21.15.31:0/1112016920 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f831c004320 con 0x55fef408bc60 2023-07-29T16:16:18.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fef4007d90 con 0x55fef408bc60 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f830b7fe700 1 -- 172.21.15.31:0/1112016920 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f831c0044e0 con 0x55fef408bc60 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.486+0000 7f83460d8b80 10 client.15951 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fef3df3ce0 con 0x55fef408bc60 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f830b7fe700 1 -- 172.21.15.31:0/1112016920 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fef4007d90 con 0x55fef408bc60 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f830b7fe700 1 -- 172.21.15.31:0/1112016920 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f831c0109e0 con 0x55fef408bc60 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 10 client.15951 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:18.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 20 client.15951 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40a1140 0x55fef40a3550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 4 client.15951 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.487+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fef405adf0 con 0x55fef40a1140 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.488+0000 7f8323fff700 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40a1140 0x55fef40a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.488+0000 7f8323fff700 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40a1140 0x55fef40a3550 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4cb78a1218eac4d server_cookie=6214e27489ecc007 in_seq=0 out_seq=0 2023-07-29T16:16:18.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.488+0000 7f830b7fe700 10 client.15951 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f830b7fe700 1 -- 172.21.15.31:0/1112016920 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55fef405adf0 con 0x55fef40a1140 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f830b7fe700 10 client.15951 handle_command_reply: tid=0 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 client.15951 shutdown 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 msgr2=0x55fef4095420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 0x55fef4095420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f831c011890 tx=0x7f831c011ba0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 shutdown_connections 2023-07-29T16:16:18.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40a1140 0x55fef40a3550 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408d370 0x55fef409c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408bc60 0x55fef4095420 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f830b7fe700 0 client.15951 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1112016920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408c5b0 0x55fef4095960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 >> 172.21.15.31:0/1112016920 conn(0x55fef4081460 msgr2=0x55fef40860e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 shutdown_connections 2023-07-29T16:16:18.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 1 -- 172.21.15.31:0/1112016920 wait complete. 2023-07-29T16:16:18.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.489+0000 7f83460d8b80 20 client.15951 trim_cache size 0 max 16384 2023-07-29T16:16:18.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.506+0000 7f83460d8b80 1 Processor -- start 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 -- start start 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef409a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef40900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef4090630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef40834e0 con 0x55fef408c5b0 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef4093370 con 0x55fef408bc60 2023-07-29T16:16:18.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f83460d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef40940d0 con 0x55fef408d370 2023-07-29T16:16:18.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f8323fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef40900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef409a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef409a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41166/0 (socket says 172.21.15.31:41166) 2023-07-29T16:16:18.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 -- 172.21.15.31:0/3364839023 learned_addr learned my addr 172.21.15.31:0/3364839023 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f8330d9c700 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef4090630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 -- 172.21.15.31:0/3364839023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 msgr2=0x55fef4090630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef4090630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 -- 172.21.15.31:0/3364839023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 msgr2=0x55fef40900f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef40900f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.507+0000 7f832bfff700 1 -- 172.21.15.31:0/3364839023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fef40943a0 con 0x55fef408bc60 2023-07-29T16:16:18.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.508+0000 7f832bfff700 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef409a950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f831c010d70 tx=0x7f831c0104b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e11903f4de44fb34 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.508+0000 7f8322ffd700 1 -- 172.21.15.31:0/3364839023 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c016040 con 0x55fef408bc60 2023-07-29T16:16:18.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.508+0000 7f8322ffd700 1 -- 172.21.15.31:0/3364839023 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f831c01b470 con 0x55fef408bc60 2023-07-29T16:16:18.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.508+0000 7f8322ffd700 1 -- 172.21.15.31:0/3364839023 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c01b9b0 con 0x55fef408bc60 2023-07-29T16:16:18.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.508+0000 7f83460d8b80 1 -- 172.21.15.31:0/3364839023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 msgr2=0x55fef409a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.508+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef409a950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f831c010d70 tx=0x7f831c0104b0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 -- 172.21.15.31:0/3364839023 shutdown_connections 2023-07-29T16:16:18.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef4090630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef409a950 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 --2- 172.21.15.31:0/3364839023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef40900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 -- 172.21.15.31:0/3364839023 >> 172.21.15.31:0/3364839023 conn(0x55fef4081460 msgr2=0x55fef40935c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 -- 172.21.15.31:0/3364839023 shutdown_connections 2023-07-29T16:16:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 -- 172.21.15.31:0/3364839023 wait complete. 2023-07-29T16:16:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 -- start start 2023-07-29T16:16:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.509+0000 7f83460d8b80 1 Processor -- start 2023-07-29T16:16:18.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef4099fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef409a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef40f3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef40f37a0 con 0x55fef408c5b0 2023-07-29T16:16:18.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef40f38e0 con 0x55fef408bc60 2023-07-29T16:16:18.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fef40f3a70 con 0x55fef408d370 2023-07-29T16:16:18.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f83460d8b80 10 client.0 _ll_register_callbacks cb 0x55fef3f826a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:18.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef409a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef409a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36374/0 (socket says 172.21.15.31:36374) 2023-07-29T16:16:18.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 -- 172.21.15.31:0/1710851435 learned_addr learned my addr 172.21.15.31:0/1710851435 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f832bfff700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef4099fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8330d9c700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef40f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 -- 172.21.15.31:0/1710851435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 msgr2=0x55fef40f3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef40f3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 -- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 msgr2=0x55fef4099fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef4099fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.510+0000 7f8323fff700 1 -- 172.21.15.31:0/1710851435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fef3df3ce0 con 0x55fef408c5b0 2023-07-29T16:16:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f8323fff700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef409a4e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8314096e30 tx=0x7f8314009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26860b5aba8f4279 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f830b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f830b7fe700 1 -- 172.21.15.31:0/1710851435 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314021580 con 0x55fef408c5b0 2023-07-29T16:16:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f830b7fe700 1 -- 172.21.15.31:0/1710851435 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f83140998b0 con 0x55fef408c5b0 2023-07-29T16:16:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f831409e3f0 con 0x55fef408c5b0 2023-07-29T16:16:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f830b7fe700 1 -- 172.21.15.31:0/1710851435 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8314099a70 con 0x55fef408c5b0 2023-07-29T16:16:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 10 client.15957 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55fef3e312d0 con 0x55fef408c5b0 2023-07-29T16:16:18.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f830b7fe700 1 -- 172.21.15.31:0/1710851435 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f831409e3f0 con 0x55fef408c5b0 2023-07-29T16:16:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f830b7fe700 1 -- 172.21.15.31:0/1710851435 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f83140a22f0 con 0x55fef408c5b0 2023-07-29T16:16:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 10 client.15957 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 10 client.15957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 20 client.15957 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.511+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40f50f0 0x55fef40f7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.512+0000 7f83460d8b80 4 client.15957 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:16:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.512+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55fef3fd5dc0 con 0x55fef40f50f0 2023-07-29T16:16:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.512+0000 7f832bfff700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40f50f0 0x55fef40f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.512+0000 7f832bfff700 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40f50f0 0x55fef40f7500 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a138616e743da3b server_cookie=b9095301422e8ded in_seq=0 out_seq=0 2023-07-29T16:16:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.512+0000 7f830b7fe700 10 client.15957 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f830b7fe700 1 -- 172.21.15.31:0/1710851435 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55fef3fd5dc0 con 0x55fef40f50f0 2023-07-29T16:16:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f830b7fe700 10 client.15957 handle_command_reply: tid=0 2023-07-29T16:16:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 client.15957 shutdown 2023-07-29T16:16:18.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 msgr2=0x55fef409a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef409a4e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8314096e30 tx=0x7f8314009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 shutdown_connections 2023-07-29T16:16:18.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fef40f50f0 0x55fef40f7500 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fef408d370 0x55fef40f3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fef408bc60 0x55fef4099fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 --2- 172.21.15.31:0/1710851435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fef408c5b0 0x55fef409a4e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 >> 172.21.15.31:0/1710851435 conn(0x55fef4081460 msgr2=0x55fef40935c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f830b7fe700 0 client.15957 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.514+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 shutdown_connections 2023-07-29T16:16:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.515+0000 7f83460d8b80 1 -- 172.21.15.31:0/1710851435 wait complete. 2023-07-29T16:16:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.515+0000 7f83460d8b80 20 client.15957 trim_cache size 0 max 16384 2023-07-29T16:16:18.528 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:16:18.529 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:16:18.529 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "9c3a9011-4c3e-446a-ae2f-18322ea3ed07", 2023-07-29T16:16:18.529 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:16:18.529 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:16:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.515+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83180a0790 msgr2=0x7f83180a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.515+0000 7f83327ff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83180a0790 0x7f83180a2c40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f832400f540 tx=0x7f8324009920 comp rx=0 tx=0).stop 2023-07-29T16:16:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.515+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c074d10 msgr2=0x7f832c1987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.515+0000 7f83327ff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c074d10 0x7f832c1987a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f831400b680 tx=0x7f8314005740 comp rx=0 tx=0).stop 2023-07-29T16:16:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 shutdown_connections 2023-07-29T16:16:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83180a0790 0x7f83180a2c40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f832c1081e0 0x7f832c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c073730 0x7f832c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 --2- 172.21.15.31:0/4162001113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f832c074d10 0x7f832c1987a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 >> 172.21.15.31:0/4162001113 conn(0x7f832c0fb970 msgr2=0x7f832c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 shutdown_connections 2023-07-29T16:16:18.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.516+0000 7f83327ff700 1 -- 172.21.15.31:0/4162001113 wait complete. 2023-07-29T16:16:18.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:16:18.594 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:18 smithi031 ceph-mon[95504]: pgmap v394: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:18.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 -- 172.21.15.31:0/1490067558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604100330 msgr2=0x7f1604100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 --2- 172.21.15.31:0/1490067558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604100330 0x7f1604100740 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f15f8009990 tx=0x7f15f8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 -- 172.21.15.31:0/1490067558 shutdown_connections 2023-07-29T16:16:18.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 --2- 172.21.15.31:0/1490067558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1604101eb0 0x7f1604108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 --2- 172.21.15.31:0/1490067558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604100330 0x7f1604100740 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 --2- 172.21.15.31:0/1490067558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604101520 0x7f1604101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 -- 172.21.15.31:0/1490067558 >> 172.21.15.31:0/1490067558 conn(0x7f16040fb970 msgr2=0x7f16040fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 -- 172.21.15.31:0/1490067558 shutdown_connections 2023-07-29T16:16:18.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.741+0000 7f160b1ff700 1 -- 172.21.15.31:0/1490067558 wait complete. 2023-07-29T16:16:18.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.742+0000 7f160b1ff700 1 Processor -- start 2023-07-29T16:16:18.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.742+0000 7f160b1ff700 1 -- start start 2023-07-29T16:16:18.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.742+0000 7f160b1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 0x7f16041982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f160b1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604101520 0x7f1604198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f160b1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1604101eb0 0x7f160419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f160b1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160419fe50 con 0x7f1604100330 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f160b1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160419ff90 con 0x7f1604101520 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f160b1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16041a0120 con 0x7f1604101eb0 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 0x7f16041982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 0x7f16041982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36398/0 (socket says 172.21.15.31:36398) 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 -- 172.21.15.31:0/3696516061 learned_addr learned my addr 172.21.15.31:0/3696516061 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1603fff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604101520 0x7f1604198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f160979c700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1604101eb0 0x7f160419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 -- 172.21.15.31:0/3696516061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1604101eb0 msgr2=0x7f160419f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1604101eb0 0x7f160419f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 -- 172.21.15.31:0/3696516061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604101520 msgr2=0x7f1604198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604101520 0x7f1604198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.743+0000 7f1608f9b700 1 -- 172.21.15.31:0/3696516061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f80095f0 con 0x7f1604100330 2023-07-29T16:16:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.744+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 0x7f16041982d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f15f8005740 tx=0x7f15f8005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.744+0000 7f1601ffb700 1 -- 172.21.15.31:0/3696516061 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f801d070 con 0x7f1604100330 2023-07-29T16:16:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.744+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16041a03a0 con 0x7f1604100330 2023-07-29T16:16:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.744+0000 7f1601ffb700 1 -- 172.21.15.31:0/3696516061 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15f8021410 con 0x7f1604100330 2023-07-29T16:16:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.744+0000 7f1601ffb700 1 -- 172.21.15.31:0/3696516061 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f8017650 con 0x7f1604100330 2023-07-29T16:16:18.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.744+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16041a0820 con 0x7f1604100330 2023-07-29T16:16:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.748+0000 7f1601ffb700 1 -- 172.21.15.31:0/3696516061 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f15f8021580 con 0x7f1604100330 2023-07-29T16:16:18.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.749+0000 7f1601ffb700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15f40a0790 0x7f15f40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.749+0000 7f161eb0fb80 1 Processor -- start 2023-07-29T16:16:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.749+0000 7f1601ffb700 1 -- 172.21.15.31:0/3696516061 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f15f80ccc10 con 0x7f1604100330 2023-07-29T16:16:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.749+0000 7f1603fff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15f40a0790 0x7f15f40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 -- start start 2023-07-29T16:16:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c77aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c77bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77c2f0 0x55831c77c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c6d2a10 con 0x55831c77c2f0 2023-07-29T16:16:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c6d03d0 con 0x55831c77b530 2023-07-29T16:16:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c6cbcc0 con 0x55831c77abe0 2023-07-29T16:16:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f1603fff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15f40a0790 0x7f15f40a2c40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009c60 tx=0x7f15f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f1603fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c77bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f160979c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77c2f0 0x55831c77c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f1603fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c77bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41204/0 (socket says 172.21.15.31:41204) 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f1603fff700 1 -- 172.21.15.31:0/4233307339 learned_addr learned my addr 172.21.15.31:0/4233307339 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.750+0000 7f1608f9b700 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c77aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1603fff700 1 -- 172.21.15.31:0/4233307339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 msgr2=0x55831c77aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1603fff700 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c77aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1603fff700 1 -- 172.21.15.31:0/4233307339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77c2f0 msgr2=0x55831c77c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1603fff700 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77c2f0 0x55831c77c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1603fff700 1 -- 172.21.15.31:0/4233307339 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55831c4f9de0 con 0x55831c77b530 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1608f9b700 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c77aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f160979c700 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77c2f0 0x55831c77c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f1603fff700 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c77bd60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15f000e430 tx=0x7f15f0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ee7790d01dc5ec3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f15eeffd700 1 -- 172.21.15.31:0/4233307339 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f000ee80 con 0x55831c77b530 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f15eeffd700 1 -- 172.21.15.31:0/4233307339 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15f0021070 con 0x55831c77b530 2023-07-29T16:16:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.751+0000 7f15eeffd700 1 -- 172.21.15.31:0/4233307339 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0025770 con 0x55831c77b530 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 -- 172.21.15.31:0/4233307339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 msgr2=0x55831c77bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c77bd60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15f000e430 tx=0x7f15f0014000 comp rx=0 tx=0).stop 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 -- 172.21.15.31:0/4233307339 shutdown_connections 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77c2f0 0x55831c77c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c77aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/4233307339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c77bd60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 -- 172.21.15.31:0/4233307339 >> 172.21.15.31:0/4233307339 conn(0x55831c7703e0 msgr2=0x55831c775990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 -- 172.21.15.31:0/4233307339 shutdown_connections 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.752+0000 7f161eb0fb80 1 -- 172.21.15.31:0/4233307339 wait complete. 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.753+0000 7f161eb0fb80 1 -- start start 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.753+0000 7f161eb0fb80 1 Processor -- start 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c784800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77b530 0x55831c784d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c78bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c668510 con 0x55831c77abe0 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c6681b0 con 0x55831c77c2f0 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c54f920 con 0x55831c77b530 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f161eb0fb80 10 client.0 _ll_register_callbacks cb 0x55831c671620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f1608f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c784800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f1608f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c784800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36418/0 (socket says 172.21.15.31:36418) 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f1608f9b700 1 -- 172.21.15.31:0/3845823622 learned_addr learned my addr 172.21.15.31:0/3845823622 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f160979c700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c78bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.754+0000 7f1603fff700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77b530 0x55831c784d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1608f9b700 1 -- 172.21.15.31:0/3845823622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77b530 msgr2=0x55831c784d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77b530 0x55831c784d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1608f9b700 1 -- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 msgr2=0x55831c78bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c78bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1608f9b700 1 -- 172.21.15.31:0/3845823622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc011230 con 0x55831c77abe0 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1603fff700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77b530 0x55831c784d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c784800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f15f80a21d0 tx=0x7f15f80a0330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a0fb326b15da26 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f15e3fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f15e3fff700 1 -- 172.21.15.31:0/3845823622 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f80a2700 con 0x55831c77abe0 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f15e3fff700 1 -- 172.21.15.31:0/3845823622 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f15f809a5a0 con 0x55831c77abe0 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55831c6f6d10 con 0x55831c77abe0 2023-07-29T16:16:18.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f15e3fff700 1 -- 172.21.15.31:0/3845823622 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15f809a760 con 0x55831c77abe0 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f161eb0fb80 10 client.15969 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.755+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f15f0017070 con 0x55831c77abe0 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f15e3fff700 1 -- 172.21.15.31:0/3845823622 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55831c6f6d10 con 0x55831c77abe0 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f15e3fff700 1 -- 172.21.15.31:0/3845823622 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f15f80a2300 con 0x55831c77abe0 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f161eb0fb80 10 client.15969 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f161eb0fb80 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:18.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f161eb0fb80 20 client.15969 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c790120 0x55831c792530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f161eb0fb80 4 client.15969 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55831c749d70 con 0x55831c790120 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.756+0000 7f1603fff700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c790120 0x55831c792530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.757+0000 7f1603fff700 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c790120 0x55831c792530 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=756d479e5d3e9003 server_cookie=916e151cdea2521a in_seq=0 out_seq=0 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.757+0000 7f15e3fff700 10 client.15969 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f15e3fff700 1 -- 172.21.15.31:0/3845823622 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55831c749d70 con 0x55831c790120 2023-07-29T16:16:18.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f15e3fff700 10 client.15969 handle_command_reply: tid=0 2023-07-29T16:16:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 client.15969 shutdown 2023-07-29T16:16:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 msgr2=0x55831c784800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c784800 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f15f80a21d0 tx=0x7f15f80a0330 comp rx=0 tx=0).stop 2023-07-29T16:16:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 shutdown_connections 2023-07-29T16:16:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c790120 0x55831c792530 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c78bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f15e3fff700 0 client.15969 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c784800 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3845823622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77b530 0x55831c784d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 >> 172.21.15.31:0/3845823622 conn(0x55831c7703e0 msgr2=0x55831c775060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 shutdown_connections 2023-07-29T16:16:18.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3845823622 wait complete. 2023-07-29T16:16:18.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.758+0000 7f161eb0fb80 20 client.15969 trim_cache size 0 max 16384 2023-07-29T16:16:18.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:18 smithi039 ceph-mon[128858]: pgmap v394: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:18.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.775+0000 7f161eb0fb80 1 Processor -- start 2023-07-29T16:16:18.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 -- start start 2023-07-29T16:16:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c775900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c78bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77c2f0 0x55831c776690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c7843d0 con 0x55831c77abe0 2023-07-29T16:16:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c7773b0 con 0x55831c77b530 2023-07-29T16:16:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c783cb0 con 0x55831c77c2f0 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f1608f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c775900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f1603fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c78bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f1608f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c775900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36428/0 (socket says 172.21.15.31:36428) 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f1608f9b700 1 -- 172.21.15.31:0/1444182429 learned_addr learned my addr 172.21.15.31:0/1444182429 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.776+0000 7f160979c700 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77c2f0 0x55831c776690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1603fff700 1 -- 172.21.15.31:0/1444182429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77c2f0 msgr2=0x55831c776690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1603fff700 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77c2f0 0x55831c776690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1603fff700 1 -- 172.21.15.31:0/1444182429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 msgr2=0x55831c775900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1603fff700 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c775900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1603fff700 1 -- 172.21.15.31:0/1444182429 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55831c4e2c60 con 0x55831c77b530 2023-07-29T16:16:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f160979c700 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77c2f0 0x55831c776690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1608f9b700 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c775900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:18.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f1603fff700 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c78bd40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f15f000ead0 tx=0x7f15f0009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ae458713f69006 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f15ef7fe700 1 -- 172.21.15.31:0/1444182429 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0036070 con 0x55831c77b530 2023-07-29T16:16:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f15ef7fe700 1 -- 172.21.15.31:0/1444182429 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15f001dd70 con 0x55831c77b530 2023-07-29T16:16:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.777+0000 7f15ef7fe700 1 -- 172.21.15.31:0/1444182429 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0005040 con 0x55831c77b530 2023-07-29T16:16:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 -- 172.21.15.31:0/1444182429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 msgr2=0x55831c78bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c78bd40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f15f000ead0 tx=0x7f15f0009730 comp rx=0 tx=0).stop 2023-07-29T16:16:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 -- 172.21.15.31:0/1444182429 shutdown_connections 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77c2f0 0x55831c776690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77abe0 0x55831c775900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/1444182429 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77b530 0x55831c78bd40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 -- 172.21.15.31:0/1444182429 >> 172.21.15.31:0/1444182429 conn(0x55831c7703e0 msgr2=0x55831c7834e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 -- 172.21.15.31:0/1444182429 shutdown_connections 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 -- 172.21.15.31:0/1444182429 wait complete. 2023-07-29T16:16:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.778+0000 7f161eb0fb80 1 -- start start 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 Processor -- start 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c78b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 0x55831c78bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c788300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c7889e0 con 0x55831c77b530 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c788b50 con 0x55831c77c2f0 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55831c788d10 con 0x55831c77abe0 2023-07-29T16:16:18.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.779+0000 7f161eb0fb80 10 client.0 _ll_register_callbacks cb 0x55831c671620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 0x55831c78bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 0x55831c78bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36432/0 (socket says 172.21.15.31:36432) 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 -- 172.21.15.31:0/3758271433 learned_addr learned my addr 172.21.15.31:0/3758271433 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c78b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f160979c700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c788300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 -- 172.21.15.31:0/3758271433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 msgr2=0x55831c78b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c78b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 -- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 msgr2=0x55831c788300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c788300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 -- 172.21.15.31:0/3758271433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f0017070 con 0x55831c77b530 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f1603fff700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 0x55831c78bcd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f15f0000c00 tx=0x7f15f0009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d2ebddfbac1978 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.780+0000 7f15e3fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f15e3fff700 1 -- 172.21.15.31:0/3758271433 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0036040 con 0x55831c77b530 2023-07-29T16:16:18.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15f0021070 con 0x55831c77b530 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f15e3fff700 1 -- 172.21.15.31:0/3758271433 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f15f0032050 con 0x55831c77b530 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f15e3fff700 1 -- 172.21.15.31:0/3758271433 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15f0032230 con 0x55831c77b530 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 10 client.15975 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f15f002a040 con 0x55831c77b530 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f15e3fff700 1 -- 172.21.15.31:0/3758271433 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15f0048080 con 0x55831c77b530 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f15e3fff700 1 -- 172.21.15.31:0/3758271433 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f15f0039070 con 0x55831c77b530 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 10 client.15975 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:18.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 20 client.15975 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c7e30b0 0x55831c7e54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 4 client.15975 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.781+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55831c6c4d40 con 0x55831c7e30b0 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.782+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c7e30b0 0x55831c7e54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.782+0000 7f1608f9b700 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c7e30b0 0x55831c7e54c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb256f7120dccd00 server_cookie=b4a07e24d6f42c2a in_seq=0 out_seq=0 2023-07-29T16:16:18.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.782+0000 7f15e3fff700 10 client.15975 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.782+0000 7f15e3fff700 1 -- 172.21.15.31:0/3758271433 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55831c6c4d40 con 0x55831c7e30b0 2023-07-29T16:16:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.782+0000 7f15e3fff700 10 client.15975 handle_command_reply: tid=0 2023-07-29T16:16:18.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.782+0000 7f161eb0fb80 1 client.15975 shutdown 2023-07-29T16:16:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 msgr2=0x55831c78bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 0x55831c78bcd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f15f0000c00 tx=0x7f15f0009360 comp rx=0 tx=0).stop 2023-07-29T16:16:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 shutdown_connections 2023-07-29T16:16:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55831c7e30b0 0x55831c7e54c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55831c77c2f0 0x55831c788300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55831c77abe0 0x55831c78b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 --2- 172.21.15.31:0/3758271433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55831c77b530 0x55831c78bcd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f15e3fff700 0 client.15975 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 >> 172.21.15.31:0/3758271433 conn(0x55831c7703e0 msgr2=0x55831c776690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 shutdown_connections 2023-07-29T16:16:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 1 -- 172.21.15.31:0/3758271433 wait complete. 2023-07-29T16:16:18.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.783+0000 7f161eb0fb80 20 client.15975 trim_cache size 0 max 16384 2023-07-29T16:16:18.791 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:16:18.791 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:16:18.791 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:16:18.791 INFO:teuthology.orchestra.run.smithi031.stdout: "9c3a9011-4c3e-446a-ae2f-18322ea3ed07": { 2023-07-29T16:16:18.791 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:16:18.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "9c3a9011-4c3e-446a-ae2f-18322ea3ed07", 2023-07-29T16:16:18.792 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:16:18.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:16:18.792 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:16:18.792 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:16:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15f40a0790 msgr2=0x7f15f40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15f40a0790 0x7f15f40a2c40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009c60 tx=0x7f15f0009260 comp rx=0 tx=0).stop 2023-07-29T16:16:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 msgr2=0x7f16041982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:18.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 0x7f16041982d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f15f8005740 tx=0x7f15f8005770 comp rx=0 tx=0).stop 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 shutdown_connections 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15f40a0790 0x7f15f40a2c40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1604101eb0 0x7f160419f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1604100330 0x7f16041982d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 --2- 172.21.15.31:0/3696516061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1604101520 0x7f1604198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 >> 172.21.15.31:0/3696516061 conn(0x7f16040fb970 msgr2=0x7f16040fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:18.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 shutdown_connections 2023-07-29T16:16:18.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:18.784+0000 7f160b1ff700 1 -- 172.21.15.31:0/3696516061 wait complete. 2023-07-29T16:16:18.800 INFO:tasks.cephfs.filesystem:scrub status for tag:9c3a9011-4c3e-446a-ae2f-18322ea3ed07 - {'path': '/', 'tag': '9c3a9011-4c3e-446a-ae2f-18322ea3ed07', 'options': 'recursive,force'} 2023-07-29T16:16:18.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:18 smithi156 ceph-mon[129245]: pgmap v394: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:19.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:19 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:16:19.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:19 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:19.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:19 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:16:19.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:19 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:19.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:19 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:16:19.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:19 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:19.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:19 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:16:19.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:19 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:19.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:19 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:16:19.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:19 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:19.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:19 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:16:19.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:19 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:20 smithi031 ceph-mon[95504]: pgmap v395: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:20.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:20 smithi039 ceph-mon[128858]: pgmap v395: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:20.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:20 smithi156 ceph-mon[129245]: pgmap v395: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:22 smithi031 ceph-mon[95504]: pgmap v396: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:22.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:22 smithi039 ceph-mon[128858]: pgmap v396: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:22.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:22 smithi156 ceph-mon[129245]: pgmap v396: 97 pgs: 97 active+clean; 666 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:23.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:23 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:16:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:23 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:16:23.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:23 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:16:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:24 smithi031 ceph-mon[95504]: pgmap v397: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:24.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:24 smithi039 ceph-mon[128858]: pgmap v397: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:24 smithi156 ceph-mon[129245]: pgmap v397: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:26 smithi031 ceph-mon[95504]: pgmap v398: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:26 smithi039 ceph-mon[128858]: pgmap v398: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:26.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:26 smithi156 ceph-mon[129245]: pgmap v398: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:28.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:27 smithi156 ceph-mon[129245]: pgmap v399: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:28.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:27 smithi031 ceph-mon[95504]: pgmap v399: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:28.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:27 smithi039 ceph-mon[128858]: pgmap v399: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:30.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:30 smithi031 ceph-mon[95504]: pgmap v400: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:30 smithi039 ceph-mon[128858]: pgmap v400: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:30 smithi156 ceph-mon[129245]: pgmap v400: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.068+0000 7fd13139d700 1 -- 172.21.15.31:0/1467211183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c101d00 msgr2=0x7fd12c1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.068+0000 7fd13139d700 1 --2- 172.21.15.31:0/1467211183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c101d00 0x7fd12c1085c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd12400b0a0 tx=0x7fd12400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 -- 172.21.15.31:0/1467211183 shutdown_connections 2023-07-29T16:16:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 --2- 172.21.15.31:0/1467211183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c101d00 0x7fd12c1085c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 --2- 172.21.15.31:0/1467211183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd12c100180 0x7fd12c100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 --2- 172.21.15.31:0/1467211183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c101370 0x7fd12c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 -- 172.21.15.31:0/1467211183 >> 172.21.15.31:0/1467211183 conn(0x7fd12c0fb7c0 msgr2=0x7fd12c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 -- 172.21.15.31:0/1467211183 shutdown_connections 2023-07-29T16:16:31.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.069+0000 7fd13139d700 1 -- 172.21.15.31:0/1467211183 wait complete. 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 Processor -- start 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 -- start start 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 0x7fd12c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd12c101370 0x7fd12c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c101d00 0x7fd12c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c19fd30 con 0x7fd12c100180 2023-07-29T16:16:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c19fe70 con 0x7fd12c101d00 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.070+0000 7fd13139d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1a0000 con 0x7fd12c101370 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 0x7fd12c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 0x7fd12c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40160/0 (socket says 172.21.15.31:40160) 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 -- 172.21.15.31:0/172649842 learned_addr learned my addr 172.21.15.31:0/172649842 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd12c101370 0x7fd12c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd130b9c700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c101d00 0x7fd12c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 -- 172.21.15.31:0/172649842 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd12c101370 msgr2=0x7fd12c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd12c101370 0x7fd12c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 -- 172.21.15.31:0/172649842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c101d00 msgr2=0x7fd12c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c101d00 0x7fd12c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 -- 172.21.15.31:0/172649842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd124009d00 con 0x7fd12c100180 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd12bfff700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 0x7fd12c1981b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00e970 tx=0x7fd11c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd1297fa700 1 -- 172.21.15.31:0/172649842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c009790 con 0x7fd12c100180 2023-07-29T16:16:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c1a02e0 con 0x7fd12c100180 2023-07-29T16:16:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd1297fa700 1 -- 172.21.15.31:0/172649842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11c00c7e0 con 0x7fd12c100180 2023-07-29T16:16:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.071+0000 7fd1297fa700 1 -- 172.21.15.31:0/172649842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c003c70 con 0x7fd12c100180 2023-07-29T16:16:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.072+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c1a07f0 con 0x7fd12c100180 2023-07-29T16:16:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.075+0000 7fd1297fa700 1 -- 172.21.15.31:0/172649842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd11c00cc50 con 0x7fd12c100180 2023-07-29T16:16:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.076+0000 7fd1297fa700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1140a0790 0x7fd1140a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.076+0000 7fd145fa0b80 1 Processor -- start 2023-07-29T16:16:31.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd1297fa700 1 -- 172.21.15.31:0/172649842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd11c014070 con 0x7fd12c100180 2023-07-29T16:16:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1140a0790 0x7fd1140a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 -- start start 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186bd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186be1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186beba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31860c9b0 con 0x55e3185106b0 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31860a370 con 0x55e3186be730 2023-07-29T16:16:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e318605c60 con 0x55e3186bd9b0 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1140a0790 0x7fd1140a2c40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd120009c60 tx=0x7fd120009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186bd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186bd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40168/0 (socket says 172.21.15.31:40168) 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.077+0000 7fd12bfff700 1 -- 172.21.15.31:0/2639468530 learned_addr learned my addr 172.21.15.31:0/2639468530 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd130b9c700 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186beba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186be1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12bfff700 1 -- 172.21.15.31:0/2639468530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 msgr2=0x55e3186be1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186be1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12bfff700 1 -- 172.21.15.31:0/2639468530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 msgr2=0x55e3186beba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186beba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12bfff700 1 -- 172.21.15.31:0/2639468530 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e318433d80 con 0x55e3185106b0 2023-07-29T16:16:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd130b9c700 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186beba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186bd470 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd11c095ee0 tx=0x7fd11c003710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573dc54dd3d352b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd1127fc700 1 -- 172.21.15.31:0/2639468530 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c09d4a0 con 0x55e3185106b0 2023-07-29T16:16:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd1127fc700 1 -- 172.21.15.31:0/2639468530 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11c0a18c0 con 0x55e3185106b0 2023-07-29T16:16:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.078+0000 7fd1127fc700 1 -- 172.21.15.31:0/2639468530 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c0998f0 con 0x55e3185106b0 2023-07-29T16:16:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2639468530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 msgr2=0x55e3186bd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186bd470 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd11c095ee0 tx=0x7fd11c003710 comp rx=0 tx=0).stop 2023-07-29T16:16:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2639468530 shutdown_connections 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186beba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186bd470 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2639468530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186be1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2639468530 >> 172.21.15.31:0/2639468530 conn(0x55e3186aa380 msgr2=0x55e3186af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2639468530 shutdown_connections 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.079+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2639468530 wait complete. 2023-07-29T16:16:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.080+0000 7fd145fa0b80 1 -- start start 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.080+0000 7fd145fa0b80 1 Processor -- start 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3185a24b0 con 0x55e3185106b0 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3185a2150 con 0x55e3186be730 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3184898c0 con 0x55e3186bd9b0 2023-07-29T16:16:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd145fa0b80 10 client.0 _ll_register_callbacks cb 0x55e3185ab5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186c68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd130b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186c68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40176/0 (socket says 172.21.15.31:40176) 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 -- 172.21.15.31:0/3747495276 learned_addr learned my addr 172.21.15.31:0/3747495276 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 -- 172.21.15.31:0/3747495276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 msgr2=0x55e3186c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 -- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 msgr2=0x55e3186cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd12bfff700 1 -- 172.21.15.31:0/3747495276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c108ef0 con 0x55e3185106b0 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.081+0000 7fd130b9c700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd12bfff700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186c68e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd11c0977c0 tx=0x7fd11c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3542755c676d3f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd10b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd10b7fe700 1 -- 172.21.15.31:0/3747495276 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c0996d0 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd10b7fe700 1 -- 172.21.15.31:0/3747495276 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd11c099830 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e318630cb0 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd10b7fe700 1 -- 172.21.15.31:0/3747495276 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd11c0a1430 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd145fa0b80 10 client.15987 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fd12c1a07f0 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.082+0000 7fd10b7fe700 1 -- 172.21.15.31:0/3747495276 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11c07e080 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd10b7fe700 1 -- 172.21.15.31:0/3747495276 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd11c0b0030 con 0x55e3185106b0 2023-07-29T16:16:31.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd145fa0b80 10 client.15987 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd145fa0b80 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd145fa0b80 20 client.15987 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3186d2260 0x55e3186d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd145fa0b80 4 client.15987 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e318683d10 con 0x55e3186d2260 2023-07-29T16:16:31.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.083+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3186d2260 0x55e3186d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.084+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3186d2260 0x55e3186d4640 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66d65b7f6affba0 server_cookie=8990cbecdc37509 in_seq=0 out_seq=0 2023-07-29T16:16:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.084+0000 7fd10b7fe700 10 client.15987 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd10b7fe700 1 -- 172.21.15.31:0/3747495276 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e318683d10 con 0x55e3186d2260 2023-07-29T16:16:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd10b7fe700 10 client.15987 handle_command_reply: tid=0 2023-07-29T16:16:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 client.15987 shutdown 2023-07-29T16:16:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 msgr2=0x55e3186c68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186c68e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd11c0977c0 tx=0x7fd11c004750 comp rx=0 tx=0).stop 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 shutdown_connections 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3186d2260 0x55e3186d4640 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186be730 0x55e3186cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3185106b0 0x55e3186c68e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd10b7fe700 0 client.15987 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/3747495276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3186bd9b0 0x55e3186c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 >> 172.21.15.31:0/3747495276 conn(0x55e3186aa380 msgr2=0x55e3186aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 shutdown_connections 2023-07-29T16:16:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.085+0000 7fd145fa0b80 1 -- 172.21.15.31:0/3747495276 wait complete. 2023-07-29T16:16:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.086+0000 7fd145fa0b80 20 client.15987 trim_cache size 0 max 16384 2023-07-29T16:16:31.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 Processor -- start 2023-07-29T16:16:31.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 -- start start 2023-07-29T16:16:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3186c2100 con 0x55e3186be730 2023-07-29T16:16:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3186b02c0 con 0x55e3186bd9b0 2023-07-29T16:16:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3186d8fa0 con 0x55e3185106b0 2023-07-29T16:16:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd130b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd12b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd130b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40182/0 (socket says 172.21.15.31:40182) 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186d4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50974/0 (socket says 172.21.15.31:50974) 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd12b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46332/0 (socket says 172.21.15.31:46332) 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.101+0000 7fd130b9c700 1 -- 172.21.15.31:0/2436720550 learned_addr learned my addr 172.21.15.31:0/2436720550 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12b7fe700 1 -- 172.21.15.31:0/2436720550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 msgr2=0x55e3186d4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186d4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12b7fe700 1 -- 172.21.15.31:0/2436720550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 msgr2=0x55e3186c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12b7fe700 1 -- 172.21.15.31:0/2436720550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e31841cc00 con 0x55e3186bd9b0 2023-07-29T16:16:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186d4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186cde20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55e3186d4ef0 tx=0x7fd120014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ac0c75401bb5007 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd112ffd700 1 -- 172.21.15.31:0/2436720550 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd120022070 con 0x55e3186bd9b0 2023-07-29T16:16:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd112ffd700 1 -- 172.21.15.31:0/2436720550 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd120023040 con 0x55e3186bd9b0 2023-07-29T16:16:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.102+0000 7fd112ffd700 1 -- 172.21.15.31:0/2436720550 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd120003ec0 con 0x55e3186bd9b0 2023-07-29T16:16:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2436720550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 msgr2=0x55e3186cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186cde20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x55e3186d4ef0 tx=0x7fd120014000 comp rx=0 tx=0).stop 2023-07-29T16:16:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2436720550 shutdown_connections 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186d4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2436720550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186cde20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2436720550 >> 172.21.15.31:0/2436720550 conn(0x55e3186aa380 msgr2=0x55e3186aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2436720550 shutdown_connections 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2436720550 wait complete. 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.103+0000 7fd145fa0b80 1 -- start start 2023-07-29T16:16:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 Processor -- start 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186ca100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186cd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3186cd660 con 0x55e3186be730 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3186cd7d0 con 0x55e3186bd9b0 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3186cd990 con 0x55e3185106b0 2023-07-29T16:16:31.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd145fa0b80 10 client.0 _ll_register_callbacks cb 0x55e3185ab5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd130b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.104+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50980/0 (socket says 172.21.15.31:50980) 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 -- 172.21.15.31:0/2131960172 learned_addr learned my addr 172.21.15.31:0/2131960172 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 -- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 msgr2=0x55e3186ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 -- 172.21.15.31:0/2131960172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 msgr2=0x55e3186cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 -- 172.21.15.31:0/2131960172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c1a07f0 con 0x55e3185106b0 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd12bfff700 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186c9bc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd11c095970 tx=0x7fd11c003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73288d5e0a11b58a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd10b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd10b7fe700 1 -- 172.21.15.31:0/2131960172 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c0998d0 con 0x55e3185106b0 2023-07-29T16:16:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd11c0a18c0 con 0x55e3185106b0 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.105+0000 7fd10b7fe700 1 -- 172.21.15.31:0/2131960172 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd11c065250 con 0x55e3185106b0 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd10b7fe700 1 -- 172.21.15.31:0/2131960172 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd11c09d030 con 0x55e3185106b0 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 10 client.25972 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fd120017070 con 0x55e3185106b0 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd10b7fe700 1 -- 172.21.15.31:0/2131960172 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11c0a18c0 con 0x55e3185106b0 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd10b7fe700 1 -- 172.21.15.31:0/2131960172 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd11c00c950 con 0x55e3185106b0 2023-07-29T16:16:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 10 client.25972 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 10 client.25972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 20 client.25972 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e318724830 0x55e318726c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 4 client.25972 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.106+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e3185fece0 con 0x55e318724830 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.107+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e318724830 0x55e318726c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.107+0000 7fd12b7fe700 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e318724830 0x55e318726c10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c247ab3cd79147 server_cookie=a15441ddc0121533 in_seq=0 out_seq=0 2023-07-29T16:16:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.107+0000 7fd10b7fe700 10 client.25972 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd10b7fe700 1 -- 172.21.15.31:0/2131960172 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e3185fece0 con 0x55e318724830 2023-07-29T16:16:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd10b7fe700 10 client.25972 handle_command_reply: tid=0 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 client.25972 shutdown 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 msgr2=0x55e3186c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186c9bc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd11c095970 tx=0x7fd11c003680 comp rx=0 tx=0).stop 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 shutdown_connections 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e318724830 0x55e318726c10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3186be730 0x55e3186cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3185106b0 0x55e3186c9bc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 --2- 172.21.15.31:0/2131960172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3186bd9b0 0x55e3186ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd10b7fe700 0 client.25972 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.111+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 >> 172.21.15.31:0/2131960172 conn(0x55e3186aa380 msgr2=0x55e3186c5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.112+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 shutdown_connections 2023-07-29T16:16:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.112+0000 7fd145fa0b80 1 -- 172.21.15.31:0/2131960172 wait complete. 2023-07-29T16:16:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.112+0000 7fd145fa0b80 20 client.25972 trim_cache size 0 max 16384 2023-07-29T16:16:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:16:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:16:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:16:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:16:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.112+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1140a0790 msgr2=0x7fd1140a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1140a0790 0x7fd1140a2c40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd120009c60 tx=0x7fd120009260 comp rx=0 tx=0).stop 2023-07-29T16:16:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 msgr2=0x7fd12c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 0x7fd12c1981b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00e970 tx=0x7fd11c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:16:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 shutdown_connections 2023-07-29T16:16:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1140a0790 0x7fd1140a2c40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd12c101d00 0x7fd12c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd12c100180 0x7fd12c1981b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 --2- 172.21.15.31:0/172649842 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd12c101370 0x7fd12c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 >> 172.21.15.31:0/172649842 conn(0x7fd12c0fb7c0 msgr2=0x7fd12c0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 shutdown_connections 2023-07-29T16:16:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:31.113+0000 7fd13139d700 1 -- 172.21.15.31:0/172649842 wait complete. 2023-07-29T16:16:32.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:32 smithi031 ceph-mon[95504]: pgmap v401: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:32.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:32 smithi039 ceph-mon[128858]: pgmap v401: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:32.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:32 smithi156 ceph-mon[129245]: pgmap v401: 97 pgs: 97 active+clean; 667 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:34.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:34 smithi031 ceph-mon[95504]: pgmap v402: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:16:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:34 smithi039 ceph-mon[128858]: pgmap v402: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:16:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:34 smithi156 ceph-mon[129245]: pgmap v402: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-07-29T16:16:36.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:36 smithi031 ceph-mon[95504]: pgmap v403: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:36.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:36 smithi039 ceph-mon[128858]: pgmap v403: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:36 smithi156 ceph-mon[129245]: pgmap v403: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:38.070 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:37 smithi156 ceph-mon[129245]: pgmap v404: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:38.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:37 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:16:38.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:37 smithi031 ceph-mon[95504]: pgmap v404: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:37 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:16:38.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:37 smithi039 ceph-mon[128858]: pgmap v404: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:38.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:37 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:16:39.320 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:16:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:16:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:16:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:16:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:39.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:16:39.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:16:39.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:16:40.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:40 smithi156 ceph-mon[129245]: pgmap v405: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:40.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:40 smithi031 ceph-mon[95504]: pgmap v405: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:40.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:40 smithi039 ceph-mon[128858]: pgmap v405: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:42.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:42 smithi031 ceph-mon[95504]: pgmap v406: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:42.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:42 smithi039 ceph-mon[128858]: pgmap v406: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:42.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:42 smithi156 ceph-mon[129245]: pgmap v406: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.274+0000 7fd1a3dff700 1 -- 172.21.15.31:0/3585511516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c101540 msgr2=0x7fd19c101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.274+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/3585511516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c101540 0x7fd19c101990 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009990 tx=0x7fd18c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 -- 172.21.15.31:0/3585511516 shutdown_connections 2023-07-29T16:16:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/3585511516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101ed0 0x7fd19c108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/3585511516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd19c100350 0x7fd19c100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/3585511516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c101540 0x7fd19c101990 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 -- 172.21.15.31:0/3585511516 >> 172.21.15.31:0/3585511516 conn(0x7fd19c0fb970 msgr2=0x7fd19c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 -- 172.21.15.31:0/3585511516 shutdown_connections 2023-07-29T16:16:44.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.275+0000 7fd1a3dff700 1 -- 172.21.15.31:0/3585511516 wait complete. 2023-07-29T16:16:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 Processor -- start 2023-07-29T16:16:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 -- start start 2023-07-29T16:16:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c100350 0x7fd19c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 0x7fd19c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd19c101ed0 0x7fd19c073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1a0140 con 0x7fd19c101540 2023-07-29T16:16:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1a0280 con 0x7fd19c100350 2023-07-29T16:16:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.276+0000 7fd1a3dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1a03c0 con 0x7fd19c101ed0 2023-07-29T16:16:44.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 0x7fd19c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 0x7fd19c073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48856/0 (socket says 172.21.15.31:48856) 2023-07-29T16:16:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 -- 172.21.15.31:0/2558720065 learned_addr learned my addr 172.21.15.31:0/2558720065 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c100350 0x7fd19c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a239c700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd19c101ed0 0x7fd19c073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 -- 172.21.15.31:0/2558720065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd19c101ed0 msgr2=0x7fd19c073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd19c101ed0 0x7fd19c073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 -- 172.21.15.31:0/2558720065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c100350 msgr2=0x7fd19c073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c100350 0x7fd19c073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 -- 172.21.15.31:0/2558720065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c0095f0 con 0x7fd19c101540 2023-07-29T16:16:44.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.277+0000 7fd1a139a700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 0x7fd19c073680 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd18c004760 tx=0x7fd18c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.278+0000 7fd192ffd700 1 -- 172.21.15.31:0/2558720065 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c01d070 con 0x7fd19c101540 2023-07-29T16:16:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.278+0000 7fd192ffd700 1 -- 172.21.15.31:0/2558720065 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd18c004830 con 0x7fd19c101540 2023-07-29T16:16:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.278+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19c1a05f0 con 0x7fd19c101540 2023-07-29T16:16:44.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.278+0000 7fd192ffd700 1 -- 172.21.15.31:0/2558720065 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c005350 con 0x7fd19c101540 2023-07-29T16:16:44.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.278+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c1a0a10 con 0x7fd19c101540 2023-07-29T16:16:44.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.281+0000 7fd192ffd700 1 -- 172.21.15.31:0/2558720065 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd18c005530 con 0x7fd19c101540 2023-07-29T16:16:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.282+0000 7fd192ffd700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1880a0810 0x7fd1880a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 Processor -- start 2023-07-29T16:16:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd192ffd700 1 -- 172.21.15.31:0/2558720065 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd18c0ccc80 con 0x7fd19c101540 2023-07-29T16:16:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1880a0810 0x7fd1880a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 -- start start 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04374f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f0437507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f043751160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0436a79b0 con 0x55f0435ab6b0 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0436a5370 con 0x55f043750cf0 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.283+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0436a0c60 con 0x55f04374fdf0 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04374f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04374f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48862/0 (socket says 172.21.15.31:48862) 2023-07-29T16:16:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3004570624 learned_addr learned my addr 172.21.15.31:0/3004570624 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a239c700 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f043751160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a139a700 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f0437507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1880a0810 0x7fd1880a2cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd19c074a50 tx=0x7fd198009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3004570624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 msgr2=0x55f0437507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f0437507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3004570624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 msgr2=0x55f043751160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f043751160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3004570624 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0434ced80 con 0x55f0435ab6b0 2023-07-29T16:16:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.284+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04374f8b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd198011d00 tx=0x7fd198014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace89c5a83424036 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd187fff700 1 -- 172.21.15.31:0/3004570624 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19800eb20 con 0x55f0435ab6b0 2023-07-29T16:16:44.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd187fff700 1 -- 172.21.15.31:0/3004570624 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd198021070 con 0x55f0435ab6b0 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd187fff700 1 -- 172.21.15.31:0/3004570624 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198003b80 con 0x55f0435ab6b0 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3004570624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 msgr2=0x55f04374f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04374f8b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd198011d00 tx=0x7fd198014000 comp rx=0 tx=0).stop 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3004570624 shutdown_connections 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f043751160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04374f8b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3004570624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f0437507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.285+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3004570624 >> 172.21.15.31:0/3004570624 conn(0x55f043744fd0 msgr2=0x55f04374a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.286+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3004570624 shutdown_connections 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.286+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3004570624 wait complete. 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.286+0000 7fd1b77c9b80 1 -- start start 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.286+0000 7fd1b77c9b80 1 Processor -- start 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043758f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f04374fdf0 0x55f043759470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f043750cf0 0x55f0437604b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04363d4b0 con 0x55f04374fdf0 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04363d150 con 0x55f0435ab6b0 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0435248c0 con 0x55f043750cf0 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1b77c9b80 10 client.0 _ll_register_callbacks cb 0x55f0436465c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043758f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043758f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43476/0 (socket says 172.21.15.31:43476) 2023-07-29T16:16:44.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/200990500 learned_addr learned my addr 172.21.15.31:0/200990500 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1a139a700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f04374fdf0 0x55f043759470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.287+0000 7fd1a239c700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f043750cf0 0x55f0437604b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/200990500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f043750cf0 msgr2=0x55f0437604b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f043750cf0 0x55f0437604b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/200990500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f04374fdf0 msgr2=0x55f043759470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f04374fdf0 0x55f043759470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/200990500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd198017070 con 0x55f0435ab6b0 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a239c700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f043750cf0 0x55f0437604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a139a700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f04374fdf0 0x55f043759470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043758f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd19800eb50 tx=0x7fd19800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=601053a21f85bd9c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1857fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1857fa700 1 -- 172.21.15.31:0/200990500 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19802a070 con 0x55f0435ab6b0 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1857fa700 1 -- 172.21.15.31:0/200990500 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd198005300 con 0x55f0435ab6b0 2023-07-29T16:16:44.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f0436cbcb0 con 0x55f0435ab6b0 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.288+0000 7fd1857fa700 1 -- 172.21.15.31:0/200990500 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1980054c0 con 0x55f0435ab6b0 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 10 client.25979 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f0434b7c00 con 0x55f0435ab6b0 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1857fa700 1 -- 172.21.15.31:0/200990500 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f0436cbcb0 con 0x55f0435ab6b0 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1857fa700 1 -- 172.21.15.31:0/200990500 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd198026860 con 0x55f0435ab6b0 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 10 client.25979 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 10 client.25979 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:44.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 20 client.25979 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f043764b40 0x55f043766f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 4 client.25979 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f04371ed10 con 0x55f043764b40 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.289+0000 7fd1a139a700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f043764b40 0x55f043766f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.290+0000 7fd1a139a700 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f043764b40 0x55f043766f50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56a2574e1d9b861 server_cookie=bffe93894518d460 in_seq=0 out_seq=0 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.290+0000 7fd1857fa700 10 client.25979 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:44.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1857fa700 1 -- 172.21.15.31:0/200990500 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f04371ed10 con 0x55f043764b40 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1857fa700 10 client.25979 handle_command_reply: tid=0 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 client.25979 shutdown 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 msgr2=0x55f043758f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043758f30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd19800eb50 tx=0x7fd19800eb80 comp rx=0 tx=0).stop 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 shutdown_connections 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f043764b40 0x55f043766f50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f043750cf0 0x55f0437604b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043758f30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1857fa700 0 client.25979 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/200990500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f04374fdf0 0x55f043759470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 >> 172.21.15.31:0/200990500 conn(0x55f043744fd0 msgr2=0x55f043749a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 shutdown_connections 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.291+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/200990500 wait complete. 2023-07-29T16:16:44.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.292+0000 7fd1b77c9b80 20 client.25979 trim_cache size 0 max 16384 2023-07-29T16:16:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 Processor -- start 2023-07-29T16:16:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 -- start start 2023-07-29T16:16:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043751140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f043760470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f043750cf0 0x55f04374a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f043759f60 con 0x55f043750cf0 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0437449d0 con 0x55f0435ab6b0 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.307+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f043757a00 con 0x55f04374fdf0 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043751140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043751140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43478/0 (socket says 172.21.15.31:43478) 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3624462001 learned_addr learned my addr 172.21.15.31:0/3624462001 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:44.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a239c700 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f043750cf0 0x55f04374a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a139a700 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f043760470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3624462001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 msgr2=0x55f043760470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f043760470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3624462001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f043750cf0 msgr2=0x55f04374a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f043750cf0 0x55f04374a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3624462001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0434f51f0 con 0x55f0435ab6b0 2023-07-29T16:16:44.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.308+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043751140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd198009860 tx=0x7fd198026dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcdf24c04851f012 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd190bff700 1 -- 172.21.15.31:0/3624462001 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19802a070 con 0x55f0435ab6b0 2023-07-29T16:16:44.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd190bff700 1 -- 172.21.15.31:0/3624462001 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd19801bc50 con 0x55f0435ab6b0 2023-07-29T16:16:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd190bff700 1 -- 172.21.15.31:0/3624462001 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198027ea0 con 0x55f0435ab6b0 2023-07-29T16:16:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3624462001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 msgr2=0x55f043751140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043751140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd198009860 tx=0x7fd198026dc0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3624462001 shutdown_connections 2023-07-29T16:16:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f043750cf0 0x55f04374a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0435ab6b0 0x55f043751140 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3624462001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f043760470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3624462001 >> 172.21.15.31:0/3624462001 conn(0x55f043744fd0 msgr2=0x55f043757c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3624462001 shutdown_connections 2023-07-29T16:16:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.309+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3624462001 wait complete. 2023-07-29T16:16:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 -- start start 2023-07-29T16:16:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 Processor -- start 2023-07-29T16:16:44.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04375c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f04375c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f04375f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04375fd40 con 0x55f0435ab6b0 2023-07-29T16:16:44.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f04375feb0 con 0x55f043750cf0 2023-07-29T16:16:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f043760070 con 0x55f04374fdf0 2023-07-29T16:16:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.310+0000 7fd1b77c9b80 10 client.0 _ll_register_callbacks cb 0x55f0436465c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04375c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04375c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48892/0 (socket says 172.21.15.31:48892) 2023-07-29T16:16:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3206636814 learned_addr learned my addr 172.21.15.31:0/3206636814 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a139a700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f04375c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a239c700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f04375f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3206636814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 msgr2=0x55f04375c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f04375c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 msgr2=0x55f04375f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f04375f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 -- 172.21.15.31:0/3206636814 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0434b7c00 con 0x55f0435ab6b0 2023-07-29T16:16:44.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1a1b9b700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04375c2a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd19800e970 tx=0x7fd19800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619c998759d62264 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1857fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.311+0000 7fd1857fa700 1 -- 172.21.15.31:0/3206636814 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19802a040 con 0x55f0435ab6b0 2023-07-29T16:16:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1857fa700 1 -- 172.21.15.31:0/3206636814 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd198005300 con 0x55f0435ab6b0 2023-07-29T16:16:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd198021070 con 0x55f0435ab6b0 2023-07-29T16:16:44.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1857fa700 1 -- 172.21.15.31:0/3206636814 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1980054c0 con 0x55f0435ab6b0 2023-07-29T16:16:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 10 client.16011 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55f0434cd250 con 0x55f0435ab6b0 2023-07-29T16:16:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1857fa700 1 -- 172.21.15.31:0/3206636814 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd19804f020 con 0x55f0435ab6b0 2023-07-29T16:16:44.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1857fa700 1 -- 172.21.15.31:0/3206636814 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd19801b790 con 0x55f0435ab6b0 2023-07-29T16:16:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 10 client.16011 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 10 client.16011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 20 client.16011 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:44.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0437b7260 0x55f0437b9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 4 client.16011 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:16:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.312+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f043699ce0 con 0x55f0437b7260 2023-07-29T16:16:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.313+0000 7fd1a139a700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0437b7260 0x55f0437b9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:44.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.313+0000 7fd1a139a700 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0437b7260 0x55f0437b9670 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c764e3464379e140 server_cookie=f520656cef47e615 in_seq=0 out_seq=0 2023-07-29T16:16:44.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.313+0000 7fd1857fa700 10 client.16011 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1857fa700 1 -- 172.21.15.31:0/3206636814 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f043699ce0 con 0x55f0437b7260 2023-07-29T16:16:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1857fa700 10 client.16011 handle_command_reply: tid=0 2023-07-29T16:16:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 client.16011 shutdown 2023-07-29T16:16:44.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 msgr2=0x55f04375c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04375c2a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd19800e970 tx=0x7fd19800eee0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 shutdown_connections 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0437b7260 0x55f0437b9670 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f043750cf0 0x55f04375f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0435ab6b0 0x55f04375c2a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 --2- 172.21.15.31:0/3206636814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f04374fdf0 0x55f04375c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1857fa700 0 client.16011 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.317+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 >> 172.21.15.31:0/3206636814 conn(0x55f043744fd0 msgr2=0x55f043756ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.318+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 shutdown_connections 2023-07-29T16:16:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.318+0000 7fd1b77c9b80 1 -- 172.21.15.31:0/3206636814 wait complete. 2023-07-29T16:16:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.318+0000 7fd1b77c9b80 20 client.16011 trim_cache size 0 max 16384 2023-07-29T16:16:44.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:16:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:16:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:16:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1880a0810 msgr2=0x7fd1880a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1880a0810 0x7fd1880a2cc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd19c074a50 tx=0x7fd198009260 comp rx=0 tx=0).stop 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 msgr2=0x7fd19c073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 0x7fd19c073680 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd18c004760 tx=0x7fd18c004790 comp rx=0 tx=0).stop 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 shutdown_connections 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1880a0810 0x7fd1880a2cc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd19c101ed0 0x7fd19c073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd19c100350 0x7fd19c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 --2- 172.21.15.31:0/2558720065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd19c101540 0x7fd19c073680 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 >> 172.21.15.31:0/2558720065 conn(0x7fd19c0fb970 msgr2=0x7fd19c104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 shutdown_connections 2023-07-29T16:16:44.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:44.319+0000 7fd1a3dff700 1 -- 172.21.15.31:0/2558720065 wait complete. 2023-07-29T16:16:44.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:44 smithi031 ceph-mon[95504]: pgmap v407: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:44.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:44 smithi039 ceph-mon[128858]: pgmap v407: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:44.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:44 smithi156 ceph-mon[129245]: pgmap v407: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:46.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:46 smithi031 ceph-mon[95504]: pgmap v408: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:46.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:46 smithi039 ceph-mon[128858]: pgmap v408: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:46 smithi156 ceph-mon[129245]: pgmap v408: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:16:48.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:47 smithi156 ceph-mon[129245]: pgmap v409: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:16:48.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:47 smithi031 ceph-mon[95504]: pgmap v409: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:16:48.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:47 smithi039 ceph-mon[128858]: pgmap v409: 97 pgs: 97 active+clean; 668 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:16:48.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:16:49.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 -- 172.21.15.31:0/2862651431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101510 msgr2=0x7f38d8101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 --2- 172.21.15.31:0/2862651431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101510 0x7f38d8101960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009990 tx=0x7f38c8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 -- 172.21.15.31:0/2862651431 shutdown_connections 2023-07-29T16:16:49.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 --2- 172.21.15.31:0/2862651431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8101ea0 0x7f38d8108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 --2- 172.21.15.31:0/2862651431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8100320 0x7f38d8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 --2- 172.21.15.31:0/2862651431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101510 0x7f38d8101960 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 -- 172.21.15.31:0/2862651431 >> 172.21.15.31:0/2862651431 conn(0x7f38d80fb920 msgr2=0x7f38d80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 -- 172.21.15.31:0/2862651431 shutdown_connections 2023-07-29T16:16:49.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.018+0000 7f38dffff700 1 -- 172.21.15.31:0/2862651431 wait complete. 2023-07-29T16:16:49.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.019+0000 7f38dffff700 1 Processor -- start 2023-07-29T16:16:49.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.019+0000 7f38dffff700 1 -- start start 2023-07-29T16:16:49.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8100320 0x7f38d8074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8101510 0x7f38d8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 0x7f38d8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dffff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d8073e20 con 0x7f38d8101ea0 2023-07-29T16:16:49.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dffff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d8073f90 con 0x7f38d8101510 2023-07-29T16:16:49.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d81a0120 con 0x7f38d8100320 2023-07-29T16:16:49.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 0x7f38d8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8100320 0x7f38d8074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 0x7f38d8073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48920/0 (socket says 172.21.15.31:48920) 2023-07-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 -- 172.21.15.31:0/2736653130 learned_addr learned my addr 172.21.15.31:0/2736653130 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dd59a700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8101510 0x7f38d8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 -- 172.21.15.31:0/2736653130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8100320 msgr2=0x7f38d8074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8100320 0x7f38d8074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 -- 172.21.15.31:0/2736653130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8101510 msgr2=0x7f38d8073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8101510 0x7f38d8073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38de59c700 1 -- 172.21.15.31:0/2736653130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c80095f0 con 0x7f38d8101ea0 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8100320 0x7f38d8074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.020+0000 7f38dd59a700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8101510 0x7f38d8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.021+0000 7f38de59c700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 0x7f38d8073680 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f38d0007ee0 tx=0x7f38d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.021+0000 7f38ceffd700 1 -- 172.21.15.31:0/2736653130 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0017070 con 0x7f38d8101ea0 2023-07-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.021+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d81a0400 con 0x7f38d8101ea0 2023-07-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.021+0000 7f38ceffd700 1 -- 172.21.15.31:0/2736653130 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38d000f040 con 0x7f38d8101ea0 2023-07-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.021+0000 7f38ceffd700 1 -- 172.21.15.31:0/2736653130 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0005550 con 0x7f38d8101ea0 2023-07-29T16:16:49.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.021+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d81a0910 con 0x7f38d8101ea0 2023-07-29T16:16:49.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.024+0000 7f38ceffd700 1 -- 172.21.15.31:0/2736653130 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f38d000db20 con 0x7f38d8101ea0 2023-07-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.025+0000 7f38f39a6b80 1 Processor -- start 2023-07-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 -- start start 2023-07-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38ceffd700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38c40a0790 0x7f38c40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 0x56116c9ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ce2f0 0x56116c9ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c924a10 con 0x56116c9cd530 2023-07-29T16:16:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c9223d0 con 0x56116c9ce2f0 2023-07-29T16:16:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c91dcc0 con 0x56116c9ccbe0 2023-07-29T16:16:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38ceffd700 1 -- 172.21.15.31:0/2736653130 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38d00cb1e0 con 0x7f38d8101ea0 2023-07-29T16:16:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 0x56116c9ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 0x56116c9ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42232/0 (socket says 172.21.15.31:42232) 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38ddd9b700 1 -- 172.21.15.31:0/3017934939 learned_addr learned my addr 172.21.15.31:0/3017934939 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38dd59a700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38c40a0790 0x7f38c40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38de59c700 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ce2f0 0x56116c9ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.026+0000 7f38dd59a700 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38ddd9b700 1 -- 172.21.15.31:0/3017934939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ce2f0 msgr2=0x56116c9ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ce2f0 0x56116c9ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38ddd9b700 1 -- 172.21.15.31:0/3017934939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 msgr2=0x56116c9cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38ddd9b700 1 -- 172.21.15.31:0/3017934939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56116c74bde0 con 0x56116c9ccbe0 2023-07-29T16:16:49.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38dd59a700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38c40a0790 0x7f38c40a2c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f38c800b2e0 tx=0x7f38c800bcc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 0x56116c9ccff0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38d40115c0 tx=0x7f38d40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=453bed7d2ddaeabf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38c3fff700 1 -- 172.21.15.31:0/3017934939 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4004220 con 0x56116c9ccbe0 2023-07-29T16:16:49.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38c3fff700 1 -- 172.21.15.31:0/3017934939 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38d4004cf0 con 0x56116c9ccbe0 2023-07-29T16:16:49.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.027+0000 7f38c3fff700 1 -- 172.21.15.31:0/3017934939 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4005650 con 0x56116c9ccbe0 2023-07-29T16:16:49.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 -- 172.21.15.31:0/3017934939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 msgr2=0x56116c9ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 0x56116c9ccff0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38d40115c0 tx=0x7f38d40118d0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 -- 172.21.15.31:0/3017934939 shutdown_connections 2023-07-29T16:16:49.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ce2f0 0x56116c9ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ccbe0 0x56116c9ccff0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/3017934939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 -- 172.21.15.31:0/3017934939 >> 172.21.15.31:0/3017934939 conn(0x56116c9c23e0 msgr2=0x56116c9c7990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 -- 172.21.15.31:0/3017934939 shutdown_connections 2023-07-29T16:16:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.028+0000 7f38f39a6b80 1 -- 172.21.15.31:0/3017934939 wait complete. 2023-07-29T16:16:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.029+0000 7f38f39a6b80 1 -- start start 2023-07-29T16:16:49.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.029+0000 7f38f39a6b80 1 Processor -- start 2023-07-29T16:16:49.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 0x56116c9d68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9d6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c8ba510 con 0x56116c9cd530 2023-07-29T16:16:49.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c8ba1b0 con 0x56116c9ccbe0 2023-07-29T16:16:49.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c7a1920 con 0x56116c9ce2f0 2023-07-29T16:16:49.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38f39a6b80 10 client.0 _ll_register_callbacks cb 0x56116c8c3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:49.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 0x56116c9d68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 0x56116c9d68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43514/0 (socket says 172.21.15.31:43514) 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38ddd9b700 1 -- 172.21.15.31:0/1411982737 learned_addr learned my addr 172.21.15.31:0/1411982737 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38dd59a700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9d6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.030+0000 7f38de59c700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38ddd9b700 1 -- 172.21.15.31:0/1411982737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 msgr2=0x56116c9dde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38ddd9b700 1 -- 172.21.15.31:0/1411982737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 msgr2=0x56116c9d6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9d6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38ddd9b700 1 -- 172.21.15.31:0/1411982737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d4009d50 con 0x56116c9ccbe0 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 0x56116c9d68d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f38d4011c20 tx=0x7f38d4005090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6332824e0549428b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38c17fa700 1 -- 172.21.15.31:0/1411982737 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4004030 con 0x56116c9ccbe0 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56116c948d10 con 0x56116c9ccbe0 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38c17fa700 1 -- 172.21.15.31:0/1411982737 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38d4004190 con 0x56116c9ccbe0 2023-07-29T16:16:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.031+0000 7f38c17fa700 1 -- 172.21.15.31:0/1411982737 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38d4004350 con 0x56116c9ccbe0 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 10 client.25997 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56116c734c60 con 0x56116c9ccbe0 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38c17fa700 1 -- 172.21.15.31:0/1411982737 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56116c948d10 con 0x56116c9ccbe0 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38c17fa700 1 -- 172.21.15.31:0/1411982737 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f38d402b060 con 0x56116c9ccbe0 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 10 client.25997 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 20 client.25997 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116c9e2520 0x56116c9e4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 4 client.25997 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56116c99bd70 con 0x56116c9e2520 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.032+0000 7f38dd59a700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116c9e2520 0x56116c9e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.033+0000 7f38dd59a700 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116c9e2520 0x56116c9e4930 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ced2719b791016f server_cookie=fab077724425f1e1 in_seq=0 out_seq=0 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.033+0000 7f38c17fa700 10 client.25997 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38c17fa700 1 -- 172.21.15.31:0/1411982737 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56116c99bd70 con 0x56116c9e2520 2023-07-29T16:16:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38c17fa700 10 client.25997 handle_command_reply: tid=0 2023-07-29T16:16:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 client.25997 shutdown 2023-07-29T16:16:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 msgr2=0x56116c9d68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 0x56116c9d68d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f38d4011c20 tx=0x7f38d4005090 comp rx=0 tx=0).stop 2023-07-29T16:16:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 shutdown_connections 2023-07-29T16:16:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116c9e2520 0x56116c9e4930 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38c17fa700 0 client.25997 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9ccbe0 0x56116c9d68d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/1411982737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9cd530 0x56116c9d6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 >> 172.21.15.31:0/1411982737 conn(0x56116c9c23e0 msgr2=0x56116c9c7060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 shutdown_connections 2023-07-29T16:16:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.034+0000 7f38f39a6b80 1 -- 172.21.15.31:0/1411982737 wait complete. 2023-07-29T16:16:49.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.035+0000 7f38f39a6b80 20 client.25997 trim_cache size 0 max 16384 2023-07-29T16:16:49.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 Processor -- start 2023-07-29T16:16:49.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 -- start start 2023-07-29T16:16:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9ddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c9d4550 con 0x56116c9ccbe0 2023-07-29T16:16:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c9c6a30 con 0x56116c9cd530 2023-07-29T16:16:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.052+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116c9d4eb0 con 0x56116c9ce2f0 2023-07-29T16:16:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9d12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48948/0 (socket says 172.21.15.31:48948) 2023-07-29T16:16:49.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 -- 172.21.15.31:0/218701194 learned_addr learned my addr 172.21.15.31:0/218701194 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38dd59a700 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38de59c700 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9ddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 -- 172.21.15.31:0/218701194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 msgr2=0x56116c9ddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9ddde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 -- 172.21.15.31:0/218701194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 msgr2=0x56116c9d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9d1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 -- 172.21.15.31:0/218701194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56116c772250 con 0x56116c9ccbe0 2023-07-29T16:16:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38dd59a700 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9d12d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f38d4005850 tx=0x7f38d40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3409b84f8420ae28 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.053+0000 7f38ccff9700 1 -- 172.21.15.31:0/218701194 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d40248b0 con 0x56116c9ccbe0 2023-07-29T16:16:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38ccff9700 1 -- 172.21.15.31:0/218701194 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38d40049e0 con 0x56116c9ccbe0 2023-07-29T16:16:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38ccff9700 1 -- 172.21.15.31:0/218701194 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4023640 con 0x56116c9ccbe0 2023-07-29T16:16:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 -- 172.21.15.31:0/218701194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 msgr2=0x56116c9d12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9d12d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f38d4005850 tx=0x7f38d40118d0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 -- 172.21.15.31:0/218701194 shutdown_connections 2023-07-29T16:16:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9ddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9d12d0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/218701194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 -- 172.21.15.31:0/218701194 >> 172.21.15.31:0/218701194 conn(0x56116c9c23e0 msgr2=0x56116c9d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 -- 172.21.15.31:0/218701194 shutdown_connections 2023-07-29T16:16:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.054+0000 7f38f39a6b80 1 -- 172.21.15.31:0/218701194 wait complete. 2023-07-29T16:16:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 -- start start 2023-07-29T16:16:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 Processor -- start 2023-07-29T16:16:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9dc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9dcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116ca34080 con 0x56116c9ccbe0 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116ca341c0 con 0x56116c9cd530 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56116ca34300 con 0x56116c9ce2f0 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.055+0000 7f38f39a6b80 10 client.0 _ll_register_callbacks cb 0x56116c8c3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42256/0 (socket says 172.21.15.31:42256) 2023-07-29T16:16:49.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 -- 172.21.15.31:0/2472221285 learned_addr learned my addr 172.21.15.31:0/2472221285 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9dc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38dd59a700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9dcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 -- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 msgr2=0x56116c9dcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9dcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 -- 172.21.15.31:0/2472221285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 msgr2=0x56116c9dc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9dc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 -- 172.21.15.31:0/2472221285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c8020040 con 0x56116c9ce2f0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38de59c700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dd040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f38d00952f0 tx=0x7f38d0098900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e5b865abc4df38 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.056+0000 7f38c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38c17fa700 1 -- 172.21.15.31:0/2472221285 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d00a0cd0 con 0x56116c9ce2f0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38c17fa700 1 -- 172.21.15.31:0/2472221285 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38d0004c40 con 0x56116c9ce2f0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38d4004cf0 con 0x56116c9ce2f0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38c17fa700 1 -- 172.21.15.31:0/2472221285 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38d0097d70 con 0x56116c9ce2f0 2023-07-29T16:16:49.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 10 client.26008 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56116c734c60 con 0x56116c9ce2f0 2023-07-29T16:16:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38c17fa700 1 -- 172.21.15.31:0/2472221285 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38d4004cf0 con 0x56116c9ce2f0 2023-07-29T16:16:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38c17fa700 1 -- 172.21.15.31:0/2472221285 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f38d00977e0 con 0x56116c9ce2f0 2023-07-29T16:16:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 10 client.26008 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 10 client.26008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 20 client.26008 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116ca35cc0 0x56116ca380d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 4 client.26008 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:16:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.057+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56116c916d40 con 0x56116ca35cc0 2023-07-29T16:16:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116ca35cc0 0x56116ca380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38ddd9b700 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116ca35cc0 0x56116ca380d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d475972f98eda1 server_cookie=368a915bac70d3d7 in_seq=0 out_seq=0 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38c17fa700 10 client.26008 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38c17fa700 1 -- 172.21.15.31:0/2472221285 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56116c916d40 con 0x56116ca35cc0 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38c17fa700 10 client.26008 handle_command_reply: tid=0 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38f39a6b80 1 client.26008 shutdown 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 msgr2=0x56116c9dd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dd040 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f38d00952f0 tx=0x7f38d0098900 comp rx=0 tx=0).stop 2023-07-29T16:16:49.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 shutdown_connections 2023-07-29T16:16:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56116ca35cc0 0x56116ca380d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.058+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56116c9ce2f0 0x56116c9dd040 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56116c9ccbe0 0x56116c9dc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38c17fa700 0 client.26008 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38f39a6b80 1 --2- 172.21.15.31:0/2472221285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56116c9cd530 0x56116c9dcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 >> 172.21.15.31:0/2472221285 conn(0x56116c9c23e0 msgr2=0x56116c9d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 shutdown_connections 2023-07-29T16:16:49.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38f39a6b80 1 -- 172.21.15.31:0/2472221285 wait complete. 2023-07-29T16:16:49.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38f39a6b80 20 client.26008 trim_cache size 0 max 16384 2023-07-29T16:16:49.068 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:16:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:16:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:16:49.069 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:16:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38c40a0790 msgr2=0x7f38c40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38dffff700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38c40a0790 0x7f38c40a2c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f38c800b2e0 tx=0x7f38c800bcc0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 msgr2=0x7f38d8073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.059+0000 7f38dffff700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 0x7f38d8073680 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f38d0007ee0 tx=0x7f38d000d3b0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 shutdown_connections 2023-07-29T16:16:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38c40a0790 0x7f38c40a2c40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38d8101ea0 0x7f38d8073680 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38d8100320 0x7f38d8074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 --2- 172.21.15.31:0/2736653130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38d8101510 0x7f38d8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 >> 172.21.15.31:0/2736653130 conn(0x7f38d80fb920 msgr2=0x7f38d80fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 shutdown_connections 2023-07-29T16:16:49.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.060+0000 7f38dffff700 1 -- 172.21.15.31:0/2736653130 wait complete. 2023-07-29T16:16:49.076 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:16:49.076 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:16:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.288+0000 7f4a891ff700 1 -- 172.21.15.31:0/720979943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 msgr2=0x7f4a84108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.288+0000 7f4a891ff700 1 --2- 172.21.15.31:0/720979943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 0x7f4a84108780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00b0a0 tx=0x7f4a7c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 -- 172.21.15.31:0/720979943 shutdown_connections 2023-07-29T16:16:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 --2- 172.21.15.31:0/720979943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 0x7f4a84108780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 --2- 172.21.15.31:0/720979943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 0x7f4a84100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 --2- 172.21.15.31:0/720979943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a84101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 -- 172.21.15.31:0/720979943 >> 172.21.15.31:0/720979943 conn(0x7f4a840fb920 msgr2=0x7f4a840fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 -- 172.21.15.31:0/720979943 shutdown_connections 2023-07-29T16:16:49.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.289+0000 7f4a891ff700 1 -- 172.21.15.31:0/720979943 wait complete. 2023-07-29T16:16:49.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 Processor -- start 2023-07-29T16:16:49.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 -- start start 2023-07-29T16:16:49.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 0x7f4a84195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a841964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 0x7f4a8419d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419db30 con 0x7f4a84101530 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419dc70 con 0x7f4a84100340 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.290+0000 7f4a891ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419de00 con 0x7f4a84101ec0 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a8259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a841964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 0x7f4a84195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a8359e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 0x7f4a8419d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a8259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a841964f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48974/0 (socket says 172.21.15.31:48974) 2023-07-29T16:16:49.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a8259c700 1 -- 172.21.15.31:0/1798376758 learned_addr learned my addr 172.21.15.31:0/1798376758 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 -- 172.21.15.31:0/1798376758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 msgr2=0x7f4a8419d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 0x7f4a8419d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 -- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 msgr2=0x7f4a841964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a841964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 -- 172.21.15.31:0/1798376758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c009d00 con 0x7f4a84100340 2023-07-29T16:16:49.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a8259c700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a841964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.291+0000 7f4a82d9d700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 0x7f4a84195fb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400e9c0 tx=0x7f4a7400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.292+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74004220 con 0x7f4a84100340 2023-07-29T16:16:49.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.292+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a74004380 con 0x7f4a84100340 2023-07-29T16:16:49.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.292+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a84072fe0 con 0x7f4a84100340 2023-07-29T16:16:49.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.292+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74003be0 con 0x7f4a84100340 2023-07-29T16:16:49.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.292+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a84073540 con 0x7f4a84100340 2023-07-29T16:16:49.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.293+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84107830 con 0x7f4a84100340 2023-07-29T16:16:49.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.295+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4a74003d40 con 0x7f4a84100340 2023-07-29T16:16:49.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.296+0000 7f4a6bfff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a6c0a0860 0x7f4a6c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.297+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a74014070 con 0x7f4a84100340 2023-07-29T16:16:49.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.297+0000 7f4a8259c700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a6c0a0860 0x7f4a6c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.297+0000 7f4a8259c700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a6c0a0860 0x7f4a6c0a2d10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009770 tx=0x7f4a78012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.299+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4a74095960 con 0x7f4a84100340 2023-07-29T16:16:49.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.562+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a8402cee0 con 0x7f4a84100340 2023-07-29T16:16:49.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.563+0000 7f4a6bfff700 1 -- 172.21.15.31:0/1798376758 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f4a7401f340 con 0x7f4a84100340 2023-07-29T16:16:49.561 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:16:49.561 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:16:49.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a6c0a0860 msgr2=0x7f4a6c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a6c0a0860 0x7f4a6c0a2d10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4a78009770 tx=0x7f4a78012040 comp rx=0 tx=0).stop 2023-07-29T16:16:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 msgr2=0x7f4a84195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 0x7f4a84195fb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400e9c0 tx=0x7f4a7400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 shutdown_connections 2023-07-29T16:16:49.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a6c0a0860 0x7f4a6c0a2d10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a84101ec0 0x7f4a8419d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a84100340 0x7f4a84195fb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 --2- 172.21.15.31:0/1798376758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a84101530 0x7f4a841964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 >> 172.21.15.31:0/1798376758 conn(0x7f4a840fb920 msgr2=0x7f4a84104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.565+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 shutdown_connections 2023-07-29T16:16:49.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.566+0000 7f4a891ff700 1 -- 172.21.15.31:0/1798376758 wait complete. 2023-07-29T16:16:49.566 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:16:49.582 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:16:49.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 -- 172.21.15.31:0/4080841171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740101510 msgr2=0x7f0740101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 --2- 172.21.15.31:0/4080841171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740101510 0x7f0740101960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f072c009990 tx=0x7f072c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 -- 172.21.15.31:0/4080841171 shutdown_connections 2023-07-29T16:16:49.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 --2- 172.21.15.31:0/4080841171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 0x7f0740108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 --2- 172.21.15.31:0/4080841171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0740100320 0x7f0740100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 --2- 172.21.15.31:0/4080841171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740101510 0x7f0740101960 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 -- 172.21.15.31:0/4080841171 >> 172.21.15.31:0/4080841171 conn(0x7f07400fb920 msgr2=0x7f07400fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 -- 172.21.15.31:0/4080841171 shutdown_connections 2023-07-29T16:16:49.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.793+0000 7f0746bff700 1 -- 172.21.15.31:0/4080841171 wait complete. 2023-07-29T16:16:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 Processor -- start 2023-07-29T16:16:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 -- start start 2023-07-29T16:16:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 0x7f0740196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0740101510 0x7f07401965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 0x7f074019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074019dbe0 con 0x7f0740101ea0 2023-07-29T16:16:49.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074019dd20 con 0x7f0740100320 2023-07-29T16:16:49.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.794+0000 7f0746bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074019deb0 con 0x7f0740101510 2023-07-29T16:16:49.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 0x7f0740196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074519c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 0x7f074019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 0x7f0740196060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43574/0 (socket says 172.21.15.31:43574) 2023-07-29T16:16:49.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 -- 172.21.15.31:0/1086909886 learned_addr learned my addr 172.21.15.31:0/1086909886 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f073ffff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0740101510 0x7f07401965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 -- 172.21.15.31:0/1086909886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0740101510 msgr2=0x7f07401965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0740101510 0x7f07401965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 -- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 msgr2=0x7f074019d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 0x7f074019d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 -- 172.21.15.31:0/1086909886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0095f0 con 0x7f0740100320 2023-07-29T16:16:49.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074519c700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 0x7f074019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f074499b700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 0x7f0740196060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f073400e970 tx=0x7f073400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.795+0000 7f073dffb700 1 -- 172.21.15.31:0/1086909886 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734009790 con 0x7f0740100320 2023-07-29T16:16:49.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.796+0000 7f073dffb700 1 -- 172.21.15.31:0/1086909886 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f073400c7e0 con 0x7f0740100320 2023-07-29T16:16:49.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.796+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074019e190 con 0x7f0740100320 2023-07-29T16:16:49.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.796+0000 7f073dffb700 1 -- 172.21.15.31:0/1086909886 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734003c70 con 0x7f0740100320 2023-07-29T16:16:49.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.796+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074019e6a0 con 0x7f0740100320 2023-07-29T16:16:49.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.799+0000 7f073dffb700 1 -- 172.21.15.31:0/1086909886 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f073400cc50 con 0x7f0740100320 2023-07-29T16:16:49.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 Processor -- start 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 -- start start 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe07ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe09740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fd5fa10 con 0x55a76fe07be0 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fd5d3d0 con 0x55a76fe092f0 2023-07-29T16:16:49.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.800+0000 7f075a59bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fd58cc0 con 0x55a76fe08530 2023-07-29T16:16:49.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f074519c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe07ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f074519c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe07ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49000/0 (socket says 172.21.15.31:49000) 2023-07-29T16:16:49.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe09740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f074499b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f074519c700 1 -- 172.21.15.31:0/4277418426 learned_addr learned my addr 172.21.15.31:0/4277418426 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073dffb700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f07300a0790 0x7f07300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 -- 172.21.15.31:0/4277418426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 msgr2=0x55a76fe08d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe08d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 -- 172.21.15.31:0/4277418426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 msgr2=0x55a76fe07ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe07ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073dffb700 1 -- 172.21.15.31:0/1086909886 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0734014070 con 0x7f0740100320 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 -- 172.21.15.31:0/4277418426 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a76fb86de0 con 0x55a76fe092f0 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f074519c700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f07300a0790 0x7f07300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.801+0000 7f073ffff700 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe09740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f072c0059d0 tx=0x7f072c005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2c0cb7153d9ae73 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.802+0000 7f072affd700 1 -- 172.21.15.31:0/4277418426 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c01d070 con 0x55a76fe092f0 2023-07-29T16:16:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.802+0000 7f072affd700 1 -- 172.21.15.31:0/4277418426 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f072c01e040 con 0x55a76fe092f0 2023-07-29T16:16:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.802+0000 7f072affd700 1 -- 172.21.15.31:0/4277418426 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c0048a0 con 0x55a76fe092f0 2023-07-29T16:16:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.802+0000 7f075a59bb80 1 -- 172.21.15.31:0/4277418426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 msgr2=0x55a76fe09740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.802+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe09740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f072c0059d0 tx=0x7f072c005bc0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.802+0000 7f074519c700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f07300a0790 0x7f07300a2c40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55a76fe07980 tx=0x7f0738004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 -- 172.21.15.31:0/4277418426 shutdown_connections 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe09740 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe07ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4277418426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe08d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 -- 172.21.15.31:0/4277418426 >> 172.21.15.31:0/4277418426 conn(0x55a76fdfd3e0 msgr2=0x55a76fe02990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 -- 172.21.15.31:0/4277418426 shutdown_connections 2023-07-29T16:16:49.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 -- 172.21.15.31:0/4277418426 wait complete. 2023-07-29T16:16:49.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 -- start start 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.803+0000 7f075a59bb80 1 Processor -- start 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe15ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe160e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe1d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fcf5510 con 0x55a76fe07be0 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fcf51b0 con 0x55a76fe08530 2023-07-29T16:16:49.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fbdc920 con 0x55a76fe092f0 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f075a59bb80 10 client.0 _ll_register_callbacks cb 0x55a76fcfe620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f074499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe15ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f074499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe15ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49012/0 (socket says 172.21.15.31:49012) 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f073ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe160e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f074499b700 1 -- 172.21.15.31:0/2029071901 learned_addr learned my addr 172.21.15.31:0/2029071901 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.804+0000 7f074519c700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe1d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f074499b700 1 -- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 msgr2=0x55a76fe1d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f074499b700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe1d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f074499b700 1 -- 172.21.15.31:0/2029071901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 msgr2=0x55a76fe160e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f074499b700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe160e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f074499b700 1 -- 172.21.15.31:0/2029071901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0740109090 con 0x55a76fe07be0 2023-07-29T16:16:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f073ffff700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f074499b700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe15ba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f073400efd0 tx=0x7f07340950f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35061b4273b7a269 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f071ffff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f071ffff700 1 -- 172.21.15.31:0/2029071901 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734098dd0 con 0x55a76fe07be0 2023-07-29T16:16:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f071ffff700 1 -- 172.21.15.31:0/2029071901 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0734099470 con 0x55a76fe07be0 2023-07-29T16:16:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a76fd83d10 con 0x55a76fe07be0 2023-07-29T16:16:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.805+0000 7f071ffff700 1 -- 172.21.15.31:0/2029071901 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0734003dd0 con 0x55a76fe07be0 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 10 client.16047 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a76fb6fc60 con 0x55a76fe07be0 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f071ffff700 1 -- 172.21.15.31:0/2029071901 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a76fd83d10 con 0x55a76fe07be0 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f071ffff700 1 -- 172.21.15.31:0/2029071901 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0734095260 con 0x55a76fe07be0 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 10 client.16047 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 10 client.16047 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 20 client.16047 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:49.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe21870 0x55a76fe23c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 4 client.16047 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.806+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a76fdd6d70 con 0x55a76fe21870 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.807+0000 7f073ffff700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe21870 0x55a76fe23c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.807+0000 7f073ffff700 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe21870 0x55a76fe23c50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1d37768dd3ea712 server_cookie=494c031e15929274 in_seq=0 out_seq=0 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.807+0000 7f071ffff700 10 client.16047 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f071ffff700 1 -- 172.21.15.31:0/2029071901 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a76fdd6d70 con 0x55a76fe21870 2023-07-29T16:16:49.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f071ffff700 10 client.16047 handle_command_reply: tid=0 2023-07-29T16:16:49.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 client.16047 shutdown 2023-07-29T16:16:49.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 msgr2=0x55a76fe15ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe15ba0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f073400efd0 tx=0x7f07340950f0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 shutdown_connections 2023-07-29T16:16:49.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe1d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe15ba0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe21870 0x55a76fe23c50 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 --2- 172.21.15.31:0/2029071901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe160e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 >> 172.21.15.31:0/2029071901 conn(0x55a76fdfd3e0 msgr2=0x55a76fe02060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f071ffff700 0 client.16047 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 shutdown_connections 2023-07-29T16:16:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.808+0000 7f075a59bb80 1 -- 172.21.15.31:0/2029071901 wait complete. 2023-07-29T16:16:49.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.809+0000 7f075a59bb80 20 client.16047 trim_cache size 0 max 16384 2023-07-29T16:16:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 Processor -- start 2023-07-29T16:16:49.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 -- start start 2023-07-29T16:16:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe16060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe1d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe02060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fe15770 con 0x55a76fe07be0 2023-07-29T16:16:49.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fdff460 con 0x55a76fe08530 2023-07-29T16:16:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.822+0000 7f075a59bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fe08930 con 0x55a76fe092f0 2023-07-29T16:16:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f074499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe16060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f074499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe16060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49022/0 (socket says 172.21.15.31:49022) 2023-07-29T16:16:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe1d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f074499b700 1 -- 172.21.15.31:0/4248499259 learned_addr learned my addr 172.21.15.31:0/4248499259 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f074519c700 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe02060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 -- 172.21.15.31:0/4248499259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 msgr2=0x55a76fe02060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe02060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 -- 172.21.15.31:0/4248499259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 msgr2=0x55a76fe16060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe16060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 -- 172.21.15.31:0/4248499259 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a76fbad250 con 0x55a76fe08530 2023-07-29T16:16:49.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f074499b700 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe16060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.823+0000 7f073ffff700 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe1d0e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f072c009990 tx=0x7f072c01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36dae6165f04c192 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f072b7fe700 1 -- 172.21.15.31:0/4248499259 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c01d040 con 0x55a76fe08530 2023-07-29T16:16:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f072b7fe700 1 -- 172.21.15.31:0/4248499259 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f072c00b2e0 con 0x55a76fe08530 2023-07-29T16:16:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f072b7fe700 1 -- 172.21.15.31:0/4248499259 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c004780 con 0x55a76fe08530 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 -- 172.21.15.31:0/4248499259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 msgr2=0x55a76fe1d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe1d0e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f072c009990 tx=0x7f072c01a040 comp rx=0 tx=0).stop 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 -- 172.21.15.31:0/4248499259 shutdown_connections 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe092f0 0x55a76fe02060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe16060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 --2- 172.21.15.31:0/4248499259 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe08530 0x55a76fe1d0e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 -- 172.21.15.31:0/4248499259 >> 172.21.15.31:0/4248499259 conn(0x55a76fdfd3e0 msgr2=0x55a76fe15950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.824+0000 7f075a59bb80 1 -- 172.21.15.31:0/4248499259 shutdown_connections 2023-07-29T16:16:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.825+0000 7f075a59bb80 1 -- 172.21.15.31:0/4248499259 wait complete. 2023-07-29T16:16:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.825+0000 7f075a59bb80 1 -- start start 2023-07-29T16:16:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.825+0000 7f075a59bb80 1 Processor -- start 2023-07-29T16:16:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.825+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe18f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.825+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe19450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f075a59bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe1c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f075a59bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fe1c9b0 con 0x55a76fe07be0 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f075a59bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fe1cb20 con 0x55a76fe092f0 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f075a59bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a76fe1cce0 con 0x55a76fe08530 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f075a59bb80 10 client.0 _ll_register_callbacks cb 0x55a76fcfe620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe18f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe18f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49032/0 (socket says 172.21.15.31:49032) 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 -- 172.21.15.31:0/280024096 learned_addr learned my addr 172.21.15.31:0/280024096 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f073ffff700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe19450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074519c700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe1c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 -- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 msgr2=0x55a76fe19450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe19450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 -- 172.21.15.31:0/280024096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 msgr2=0x55a76fe1c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe1c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 -- 172.21.15.31:0/280024096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a76fb6fc60 con 0x55a76fe07be0 2023-07-29T16:16:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f074499b700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe18f10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0734095e90 tx=0x7f0734003710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b1dea6625a45e7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.826+0000 7f071ffff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f071ffff700 1 -- 172.21.15.31:0/280024096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0734098dd0 con 0x55a76fe07be0 2023-07-29T16:16:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f071ffff700 1 -- 172.21.15.31:0/280024096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0734099470 con 0x55a76fe07be0 2023-07-29T16:16:49.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f072c01e040 con 0x55a76fe07be0 2023-07-29T16:16:49.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f071ffff700 1 -- 172.21.15.31:0/280024096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0734099240 con 0x55a76fe07be0 2023-07-29T16:16:49.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 10 client.16053 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:49.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a76fb86de0 con 0x55a76fe07be0 2023-07-29T16:16:49.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f071ffff700 1 -- 172.21.15.31:0/280024096 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0734065020 con 0x55a76fe07be0 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f071ffff700 1 -- 172.21.15.31:0/280024096 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f07340989f0 con 0x55a76fe07be0 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 10 client.16053 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 10 client.16053 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 20 client.16053 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe73d90 0x55a76fe76170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 4 client.16053 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.827+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a76fd51d40 con 0x55a76fe73d90 2023-07-29T16:16:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.828+0000 7f073ffff700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe73d90 0x55a76fe76170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.828+0000 7f073ffff700 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe73d90 0x55a76fe76170 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c89d94f8a0d40e0 server_cookie=a6ca561a408ee9ca in_seq=0 out_seq=0 2023-07-29T16:16:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.828+0000 7f071ffff700 10 client.16053 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f071ffff700 1 -- 172.21.15.31:0/280024096 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a76fd51d40 con 0x55a76fe73d90 2023-07-29T16:16:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f071ffff700 10 client.16053 handle_command_reply: tid=0 2023-07-29T16:16:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 client.16053 shutdown 2023-07-29T16:16:49.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 msgr2=0x55a76fe18f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe18f10 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0734095e90 tx=0x7f0734003710 comp rx=0 tx=0).stop 2023-07-29T16:16:49.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 shutdown_connections 2023-07-29T16:16:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a76fe73d90 0x55a76fe76170 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a76fe092f0 0x55a76fe1c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a76fe07be0 0x55a76fe18f10 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f071ffff700 0 client.16053 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:16:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 --2- 172.21.15.31:0/280024096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a76fe08530 0x55a76fe19450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 >> 172.21.15.31:0/280024096 conn(0x55a76fdfd3e0 msgr2=0x55a76fe06950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 shutdown_connections 2023-07-29T16:16:49.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 1 -- 172.21.15.31:0/280024096 wait complete. 2023-07-29T16:16:49.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.829+0000 7f075a59bb80 20 client.16053 trim_cache size 0 max 16384 2023-07-29T16:16:49.837 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:16:49.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f07300a0790 msgr2=0x7f07300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f07300a0790 0x7f07300a2c40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x55a76fe07980 tx=0x7f0738004ad0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 msgr2=0x7f0740196060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:49.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 0x7f0740196060 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f073400e970 tx=0x7f073400ec80 comp rx=0 tx=0).stop 2023-07-29T16:16:49.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 shutdown_connections 2023-07-29T16:16:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f07300a0790 0x7f07300a2c40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0740101ea0 0x7f074019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0740100320 0x7f0740196060 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 --2- 172.21.15.31:0/1086909886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0740101510 0x7f07401965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 >> 172.21.15.31:0/1086909886 conn(0x7f07400fb920 msgr2=0x7f07400fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:49.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 shutdown_connections 2023-07-29T16:16:49.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:49.830+0000 7f0746bff700 1 -- 172.21.15.31:0/1086909886 wait complete. 2023-07-29T16:16:49.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:16:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 -- 172.21.15.31:0/1768509159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14000737b0 msgr2=0x7f1400073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 --2- 172.21.15.31:0/1768509159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14000737b0 0x7f1400073c20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009990 tx=0x7f13f4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 -- 172.21.15.31:0/1768509159 shutdown_connections 2023-07-29T16:16:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 --2- 172.21.15.31:0/1768509159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f140010c900 0x7f140010ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 --2- 172.21.15.31:0/1768509159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 0x7f14000731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 --2- 172.21.15.31:0/1768509159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14000737b0 0x7f1400073c20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.055+0000 7f14051ff700 1 -- 172.21.15.31:0/1768509159 >> 172.21.15.31:0/1768509159 conn(0x7f14000fb970 msgr2=0x7f14000fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.056+0000 7f14051ff700 1 -- 172.21.15.31:0/1768509159 shutdown_connections 2023-07-29T16:16:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.056+0000 7f14051ff700 1 -- 172.21.15.31:0/1768509159 wait complete. 2023-07-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 Processor -- start 2023-07-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 -- start start 2023-07-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 0x7f140019c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 0x7f140019ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f140010c900 0x7f14001a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001a42f0 con 0x7f14000737b0 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001a4430 con 0x7f140010c900 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f14051ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001a45c0 con 0x7f1400074d90 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 0x7f140019c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 0x7f140019c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54636/0 (socket says 172.21.15.31:54636) 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f13fed9d700 1 -- 172.21.15.31:0/3391743626 learned_addr learned my addr 172.21.15.31:0/3391743626 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.057+0000 7f13fe59c700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 0x7f140019ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13ff59e700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f140010c900 0x7f14001a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fed9d700 1 -- 172.21.15.31:0/3391743626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 msgr2=0x7f140019ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fed9d700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 0x7f140019ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fed9d700 1 -- 172.21.15.31:0/3391743626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f140010c900 msgr2=0x7f14001a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fed9d700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f140010c900 0x7f14001a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fed9d700 1 -- 172.21.15.31:0/3391743626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f40095f0 con 0x7f14000737b0 2023-07-29T16:16:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fe59c700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 0x7f140019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13ff59e700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f140010c900 0x7f14001a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13fed9d700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 0x7f140019c770 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e970 tx=0x7f13f000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.058+0000 7f13e7fff700 1 -- 172.21.15.31:0/3391743626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0009790 con 0x7f14000737b0 2023-07-29T16:16:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.059+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14001a48a0 con 0x7f14000737b0 2023-07-29T16:16:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.059+0000 7f13e7fff700 1 -- 172.21.15.31:0/3391743626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13f000c7e0 con 0x7f14000737b0 2023-07-29T16:16:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.059+0000 7f13e7fff700 1 -- 172.21.15.31:0/3391743626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f00054e0 con 0x7f14000737b0 2023-07-29T16:16:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.059+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14001a4e00 con 0x7f14000737b0 2023-07-29T16:16:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.062+0000 7f13e7fff700 1 -- 172.21.15.31:0/3391743626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f13f000c950 con 0x7f14000737b0 2023-07-29T16:16:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 Processor -- start 2023-07-29T16:16:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 -- start start 2023-07-29T16:16:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e903ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e904d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e905740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e85ba10 con 0x56462e903be0 2023-07-29T16:16:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e8593d0 con 0x56462e9052f0 2023-07-29T16:16:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.064+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e854cc0 con 0x56462e904530 2023-07-29T16:16:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13e7fff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e80a0790 0x7f13e80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e904d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e905740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13ff59e700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e80a0790 0x7f13e80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e905740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36608/0 (socket says 172.21.15.31:36608) 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 -- 172.21.15.31:0/1483415684 learned_addr learned my addr 172.21.15.31:0/1483415684 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13e7fff700 1 -- 172.21.15.31:0/3391743626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13f0014070 con 0x7f14000737b0 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13ff59e700 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e903ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 -- 172.21.15.31:0/1483415684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 msgr2=0x56462e904d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e904d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 -- 172.21.15.31:0/1483415684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 msgr2=0x56462e903ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e903ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fe59c700 1 -- 172.21.15.31:0/1483415684 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56462e682de0 con 0x56462e9052f0 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.065+0000 7f13fed9d700 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e904d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.066+0000 7f13fe59c700 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e905740 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1400074ab0 tx=0x7f13f4005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab0b8fdbdbf7e63c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.066+0000 7f13ff59e700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e80a0790 0x7f13e80a2c40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009ec0 tx=0x7f13f8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.066+0000 7f13e57fa700 1 -- 172.21.15.31:0/1483415684 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f401d050 con 0x56462e9052f0 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.066+0000 7f13e57fa700 1 -- 172.21.15.31:0/1483415684 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13f401e040 con 0x56462e9052f0 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.066+0000 7f13e57fa700 1 -- 172.21.15.31:0/1483415684 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f4005690 con 0x56462e9052f0 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 -- 172.21.15.31:0/1483415684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 msgr2=0x56462e905740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e905740 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1400074ab0 tx=0x7f13f4005ea0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 -- 172.21.15.31:0/1483415684 shutdown_connections 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e905740 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e903ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/1483415684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e904d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 -- 172.21.15.31:0/1483415684 >> 172.21.15.31:0/1483415684 conn(0x56462e8f93e0 msgr2=0x56462e8fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 -- 172.21.15.31:0/1483415684 shutdown_connections 2023-07-29T16:16:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 -- 172.21.15.31:0/1483415684 wait complete. 2023-07-29T16:16:50.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.067+0000 7f1418bf5b80 1 -- start start 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 Processor -- start 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e90d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e90dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e914d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e7f1510 con 0x56462e9052f0 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e7f11b0 con 0x56462e904530 2023-07-29T16:16:50.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e6d8920 con 0x56462e903be0 2023-07-29T16:16:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f1418bf5b80 10 client.0 _ll_register_callbacks cb 0x56462e7fa620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e90d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e90d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35040/0 (socket says 172.21.15.31:35040) 2023-07-29T16:16:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f13fed9d700 1 -- 172.21.15.31:0/696544201 learned_addr learned my addr 172.21.15.31:0/696544201 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f13ff59e700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e914d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.068+0000 7f13fe59c700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e90dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13fed9d700 1 -- 172.21.15.31:0/696544201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 msgr2=0x56462e90dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13fed9d700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e90dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13fed9d700 1 -- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 msgr2=0x56462e914d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13fed9d700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e914d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13fed9d700 1 -- 172.21.15.31:0/696544201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14001a4e00 con 0x56462e903be0 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13fed9d700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e90d7c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f13f0099460 tx=0x7f13f001dcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c8b991c0fc69d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13e2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13e2ff5700 1 -- 172.21.15.31:0/696544201 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f009d500 con 0x56462e903be0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13e2ff5700 1 -- 172.21.15.31:0/696544201 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13f00a1920 con 0x56462e903be0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56462e87fd10 con 0x56462e903be0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f13e2ff5700 1 -- 172.21.15.31:0/696544201 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13f00c9980 con 0x56462e903be0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f1418bf5b80 10 client.26044 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.069+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f140010f690 con 0x56462e903be0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f13e2ff5700 1 -- 172.21.15.31:0/696544201 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56462e87fd10 con 0x56462e903be0 2023-07-29T16:16:50.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f13e2ff5700 1 -- 172.21.15.31:0/696544201 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f13f0064d30 con 0x56462e903be0 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f1418bf5b80 10 client.26044 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f1418bf5b80 10 client.26044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f1418bf5b80 20 client.26044 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e919510 0x56462e91b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f1418bf5b80 4 client.26044 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56462e8d2d70 con 0x56462e919510 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.070+0000 7f13fe59c700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e919510 0x56462e91b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f13fe59c700 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e919510 0x56462e91b920 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e40d75d2cf3dcaf server_cookie=7c0d01438369af80 in_seq=0 out_seq=0 2023-07-29T16:16:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f13e2ff5700 10 client.26044 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f13e2ff5700 1 -- 172.21.15.31:0/696544201 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56462e8d2d70 con 0x56462e919510 2023-07-29T16:16:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f13e2ff5700 10 client.26044 handle_command_reply: tid=0 2023-07-29T16:16:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f1418bf5b80 1 client.26044 shutdown 2023-07-29T16:16:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 msgr2=0x56462e90d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.071+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e90d7c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f13f0099460 tx=0x7f13f001dcb0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 shutdown_connections 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e919510 0x56462e91b920 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e914d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e90d7c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/696544201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e90dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 >> 172.21.15.31:0/696544201 conn(0x56462e8f93e0 msgr2=0x56462e8fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f13e2ff5700 0 client.26044 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:50.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 shutdown_connections 2023-07-29T16:16:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 1 -- 172.21.15.31:0/696544201 wait complete. 2023-07-29T16:16:50.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.072+0000 7f1418bf5b80 20 client.26044 trim_cache size 0 max 16384 2023-07-29T16:16:50.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.085+0000 7f1418bf5b80 1 Processor -- start 2023-07-29T16:16:50.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.085+0000 7f1418bf5b80 1 -- start start 2023-07-29T16:16:50.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e912cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e8ff840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e90c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e90e940 con 0x56462e9052f0 2023-07-29T16:16:50.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e904b50 con 0x56462e904530 2023-07-29T16:16:50.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e9003e0 con 0x56462e903be0 2023-07-29T16:16:50.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e912cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e912cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35042/0 (socket says 172.21.15.31:35042) 2023-07-29T16:16:50.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13ff59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e90c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 -- 172.21.15.31:0/3953973684 learned_addr learned my addr 172.21.15.31:0/3953973684 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fe59c700 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e8ff840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 -- 172.21.15.31:0/3953973684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 msgr2=0x56462e8ff840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e8ff840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 -- 172.21.15.31:0/3953973684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 msgr2=0x56462e90c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e90c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.086+0000 7f13fed9d700 1 -- 172.21.15.31:0/3953973684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56462e66bc60 con 0x56462e903be0 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.087+0000 7f13fed9d700 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e912cf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f13f000cc00 tx=0x7f13f0097a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=163d64a3d83f5bdf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.087+0000 7f13e5ffb700 1 -- 172.21.15.31:0/3953973684 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f009d500 con 0x56462e903be0 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.087+0000 7f13e5ffb700 1 -- 172.21.15.31:0/3953973684 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13f0065040 con 0x56462e903be0 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.087+0000 7f13e5ffb700 1 -- 172.21.15.31:0/3953973684 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0099730 con 0x56462e903be0 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 -- 172.21.15.31:0/3953973684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 msgr2=0x56462e912cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e912cf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f13f000cc00 tx=0x7f13f0097a80 comp rx=0 tx=0).stop 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 -- 172.21.15.31:0/3953973684 shutdown_connections 2023-07-29T16:16:50.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e9052f0 0x56462e90c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e903be0 0x56462e912cf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/3953973684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e904530 0x56462e8ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 -- 172.21.15.31:0/3953973684 >> 172.21.15.31:0/3953973684 conn(0x56462e8f93e0 msgr2=0x56462e8fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 -- 172.21.15.31:0/3953973684 shutdown_connections 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 -- 172.21.15.31:0/3953973684 wait complete. 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.088+0000 7f1418bf5b80 1 -- start start 2023-07-29T16:16:50.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 Processor -- start 2023-07-29T16:16:50.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e912650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e912b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e910c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e9112b0 con 0x56462e903be0 2023-07-29T16:16:50.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e911420 con 0x56462e9052f0 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56462e9115e0 con 0x56462e904530 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f1418bf5b80 10 client.0 _ll_register_callbacks cb 0x56462e7fa620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f13fed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e912650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f13ff59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e910c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f13ff59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e910c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36636/0 (socket says 172.21.15.31:36636) 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f13ff59e700 1 -- 172.21.15.31:0/631421208 learned_addr learned my addr 172.21.15.31:0/631421208 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.089+0000 7f13fe59c700 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e912b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13ff59e700 1 -- 172.21.15.31:0/631421208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 msgr2=0x56462e912b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13ff59e700 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e912b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13ff59e700 1 -- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 msgr2=0x56462e912650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13ff59e700 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e912650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13ff59e700 1 -- 172.21.15.31:0/631421208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140010f690 con 0x56462e9052f0 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13ff59e700 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e910c60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f13f800ede0 tx=0x7f13f8004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48c3c3123569ea70 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13e2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:50.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.090+0000 7f13e2ff5700 1 -- 172.21.15.31:0/631421208 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f801bdc0 con 0x56462e9052f0 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f13f401e040 con 0x56462e9052f0 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f13e2ff5700 1 -- 172.21.15.31:0/631421208 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f13f8020040 con 0x56462e9052f0 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f13e2ff5700 1 -- 172.21.15.31:0/631421208 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13f8008340 con 0x56462e9052f0 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 10 client.26045 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56462e6a9250 con 0x56462e9052f0 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f13e2ff5700 1 -- 172.21.15.31:0/631421208 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13f401e040 con 0x56462e9052f0 2023-07-29T16:16:50.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f13e2ff5700 1 -- 172.21.15.31:0/631421208 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f13f8027b70 con 0x56462e9052f0 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 10 client.26045 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 10 client.26045 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 20 client.26045 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e96c420 0x56462e96e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 4 client.26045 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.091+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56462e84dd40 con 0x56462e96c420 2023-07-29T16:16:50.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f13fed9d700 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e96c420 0x56462e96e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f13fed9d700 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e96c420 0x56462e96e830 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c6645465b033b1d4 server_cookie=e300c70584151313 in_seq=0 out_seq=0 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f13e2ff5700 10 client.26045 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f13e2ff5700 1 -- 172.21.15.31:0/631421208 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56462e84dd40 con 0x56462e96c420 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f13e2ff5700 10 client.26045 handle_command_reply: tid=0 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f1418bf5b80 1 client.26045 shutdown 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 msgr2=0x56462e910c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.092+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e910c60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f13f800ede0 tx=0x7f13f8004ad0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 shutdown_connections 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56462e96c420 0x56462e96e830 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56462e9052f0 0x56462e910c60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56462e903be0 0x56462e912650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 --2- 172.21.15.31:0/631421208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56462e904530 0x56462e912b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f13e2ff5700 0 client.26045 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 >> 172.21.15.31:0/631421208 conn(0x56462e8f93e0 msgr2=0x56462e908250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 shutdown_connections 2023-07-29T16:16:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 1 -- 172.21.15.31:0/631421208 wait complete. 2023-07-29T16:16:50.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f1418bf5b80 20 client.26045 trim_cache size 0 max 16384 2023-07-29T16:16:50.099 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:16:50.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e80a0790 msgr2=0x7f13e80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f14051ff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e80a0790 0x7f13e80a2c40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009ec0 tx=0x7f13f8009920 comp rx=0 tx=0).stop 2023-07-29T16:16:50.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 msgr2=0x7f140019c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.093+0000 7f14051ff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 0x7f140019c770 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e970 tx=0x7f13f000ec80 comp rx=0 tx=0).stop 2023-07-29T16:16:50.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 shutdown_connections 2023-07-29T16:16:50.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13e80a0790 0x7f13e80a2c40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f140010c900 0x7f14001a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14000737b0 0x7f140019c770 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 --2- 172.21.15.31:0/3391743626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1400074d90 0x7f140019ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 >> 172.21.15.31:0/3391743626 conn(0x7f14000fb970 msgr2=0x7f14000fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 shutdown_connections 2023-07-29T16:16:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.094+0000 7f14051ff700 1 -- 172.21.15.31:0/3391743626 wait complete. 2023-07-29T16:16:50.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:16:50.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 -- 172.21.15.31:0/2225415211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 msgr2=0x7f4798101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 --2- 172.21.15.31:0/2225415211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 0x7f4798101970 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4784009990 tx=0x7f4784009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 -- 172.21.15.31:0/2225415211 shutdown_connections 2023-07-29T16:16:50.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 --2- 172.21.15.31:0/2225415211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798101eb0 0x7f4798108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 --2- 172.21.15.31:0/2225415211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798100330 0x7f4798100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 --2- 172.21.15.31:0/2225415211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 0x7f4798101970 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 -- 172.21.15.31:0/2225415211 >> 172.21.15.31:0/2225415211 conn(0x7f47980fb970 msgr2=0x7f47980fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 -- 172.21.15.31:0/2225415211 shutdown_connections 2023-07-29T16:16:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.322+0000 7f479ebff700 1 -- 172.21.15.31:0/2225415211 wait complete. 2023-07-29T16:16:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.323+0000 7f479ebff700 1 Processor -- start 2023-07-29T16:16:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 -- start start 2023-07-29T16:16:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 0x7f4798196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 0x7f4798196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798101eb0 0x7f479819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479819dbb0 con 0x7f4798101eb0 2023-07-29T16:16:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479819dcf0 con 0x7f4798100330 2023-07-29T16:16:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479ebff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479819de80 con 0x7f4798101520 2023-07-29T16:16:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479d19c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798101eb0 0x7f479819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 0x7f4798196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 0x7f4798196030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36640/0 (socket says 172.21.15.31:36640) 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f479c99b700 1 -- 172.21.15.31:0/2081619545 learned_addr learned my addr 172.21.15.31:0/2081619545 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.324+0000 7f4797fff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 0x7f4798196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479c99b700 1 -- 172.21.15.31:0/2081619545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 msgr2=0x7f4798196570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479c99b700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 0x7f4798196570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479c99b700 1 -- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798101eb0 msgr2=0x7f479819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479c99b700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798101eb0 0x7f479819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479c99b700 1 -- 172.21.15.31:0/2081619545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47840095f0 con 0x7f4798100330 2023-07-29T16:16:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479d19c700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798101eb0 0x7f479819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479c99b700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 0x7f4798196030 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f478c00e970 tx=0x7f478c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f4795ffb700 1 -- 172.21.15.31:0/2081619545 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c009790 con 0x7f4798100330 2023-07-29T16:16:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f4795ffb700 1 -- 172.21.15.31:0/2081619545 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f478c00c7e0 con 0x7f4798100330 2023-07-29T16:16:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479819e160 con 0x7f4798100330 2023-07-29T16:16:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.325+0000 7f4795ffb700 1 -- 172.21.15.31:0/2081619545 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c003c70 con 0x7f4798100330 2023-07-29T16:16:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.326+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479819e6c0 con 0x7f4798100330 2023-07-29T16:16:50.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.330+0000 7f4795ffb700 1 -- 172.21.15.31:0/2081619545 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f478c00cc50 con 0x7f4798100330 2023-07-29T16:16:50.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.331+0000 7f47b255ab80 1 Processor -- start 2023-07-29T16:16:50.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.331+0000 7f47b255ab80 1 -- start start 2023-07-29T16:16:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded0940 0x555a1ded1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded1700 0x555a1ded1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f47b255ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1de23a10 con 0x555a1ded0940 2023-07-29T16:16:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f47b255ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1de213d0 con 0x555a1ded1700 2023-07-29T16:16:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f47b255ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1de1ccc0 con 0x555a1decfff0 2023-07-29T16:16:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f479c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded0940 0x555a1ded1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4797fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded1700 0x555a1ded1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f479d19c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f479c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded0940 0x555a1ded1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54714/0 (socket says 172.21.15.31:54714) 2023-07-29T16:16:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f479c99b700 1 -- 172.21.15.31:0/2171553287 learned_addr learned my addr 172.21.15.31:0/2171553287 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4795ffb700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47880a0810 0x7f47880a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4797fff700 1 -- 172.21.15.31:0/2171553287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 msgr2=0x555a1ded0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4797fff700 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4797fff700 1 -- 172.21.15.31:0/2171553287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded0940 msgr2=0x555a1ded1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4797fff700 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded0940 0x555a1ded1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4797fff700 1 -- 172.21.15.31:0/2171553287 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a1dc4ade0 con 0x555a1ded1700 2023-07-29T16:16:50.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.332+0000 7f4795ffb700 1 -- 172.21.15.31:0/2081619545 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f478c014070 con 0x7f4798100330 2023-07-29T16:16:50.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.333+0000 7f479d19c700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47880a0810 0x7f47880a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.333+0000 7f4797fff700 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded1700 0x555a1ded1b50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4784000c00 tx=0x7f4784005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdbc52aad6c0c9f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.333+0000 7f4782ffd700 1 -- 172.21.15.31:0/2171553287 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478401d070 con 0x555a1ded1700 2023-07-29T16:16:50.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.333+0000 7f4782ffd700 1 -- 172.21.15.31:0/2171553287 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f478401e040 con 0x555a1ded1700 2023-07-29T16:16:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.333+0000 7f4782ffd700 1 -- 172.21.15.31:0/2171553287 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47840048d0 con 0x555a1ded1700 2023-07-29T16:16:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 -- 172.21.15.31:0/2171553287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded1700 msgr2=0x555a1ded1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f479d19c700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47880a0810 0x7f47880a2cc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x555a1decfd90 tx=0x7f4790004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded1700 0x555a1ded1b50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4784000c00 tx=0x7f4784005ea0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 -- 172.21.15.31:0/2171553287 shutdown_connections 2023-07-29T16:16:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded1700 0x555a1ded1b50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 --2- 172.21.15.31:0/2171553287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded0940 0x555a1ded1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 -- 172.21.15.31:0/2171553287 >> 172.21.15.31:0/2171553287 conn(0x555a1dec13e0 msgr2=0x555a1dec6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 -- 172.21.15.31:0/2171553287 shutdown_connections 2023-07-29T16:16:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.334+0000 7f47b255ab80 1 -- 172.21.15.31:0/2171553287 wait complete. 2023-07-29T16:16:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.335+0000 7f47b255ab80 1 -- start start 2023-07-29T16:16:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.335+0000 7f47b255ab80 1 Processor -- start 2023-07-29T16:16:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1deda0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1dee1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1ddb9510 con 0x555a1ded1700 2023-07-29T16:16:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1ddb91b0 con 0x555a1ded0940 2023-07-29T16:16:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1dca0920 con 0x555a1decfff0 2023-07-29T16:16:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f47b255ab80 10 client.0 _ll_register_callbacks cb 0x555a1ddc2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1dee1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1dee1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54728/0 (socket says 172.21.15.31:54728) 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 -- 172.21.15.31:0/884123361 learned_addr learned my addr 172.21.15.31:0/884123361 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479c99b700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f4797fff700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1deda0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 -- 172.21.15.31:0/884123361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 msgr2=0x555a1ded9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 -- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 msgr2=0x555a1deda0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1deda0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.336+0000 7f479d19c700 1 -- 172.21.15.31:0/884123361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47981090a0 con 0x555a1ded1700 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f479d19c700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1dee1110 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f479000ea70 tx=0x7f479000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d078837bee474ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f4777fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f4777fff700 1 -- 172.21.15.31:0/884123361 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790017ad0 con 0x555a1ded1700 2023-07-29T16:16:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a1de47d10 con 0x555a1ded1700 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f4777fff700 1 -- 172.21.15.31:0/884123361 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f479001d030 con 0x555a1ded1700 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f4777fff700 1 -- 172.21.15.31:0/884123361 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4790024450 con 0x555a1ded1700 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f47b255ab80 10 client.16083 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.337+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555a1dc33c60 con 0x555a1ded1700 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f4777fff700 1 -- 172.21.15.31:0/884123361 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4790032020 con 0x555a1ded1700 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f4777fff700 1 -- 172.21.15.31:0/884123361 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f479002d660 con 0x555a1ded1700 2023-07-29T16:16:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f47b255ab80 10 client.16083 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f47b255ab80 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f47b255ab80 20 client.16083 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f47b255ab80 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1dee5a60 0x555a1dee7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f47b255ab80 4 client.16083 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a1de9ad70 con 0x555a1dee5a60 2023-07-29T16:16:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.338+0000 7f479c99b700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1dee5a60 0x555a1dee7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.339+0000 7f479c99b700 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1dee5a60 0x555a1dee7e70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4f791467ee9d8b2 server_cookie=47d0e49b275eb611 in_seq=0 out_seq=0 2023-07-29T16:16:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.339+0000 7f4777fff700 10 client.16083 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f4777fff700 1 -- 172.21.15.31:0/884123361 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555a1de9ad70 con 0x555a1dee5a60 2023-07-29T16:16:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f4777fff700 10 client.16083 handle_command_reply: tid=0 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 client.16083 shutdown 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 msgr2=0x555a1dee1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1dee1110 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f479000ea70 tx=0x7f479000b040 comp rx=0 tx=0).stop 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 shutdown_connections 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1dee5a60 0x555a1dee7e70 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1dee1110 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1decfff0 0x555a1ded9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f4777fff700 0 client.16083 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 --2- 172.21.15.31:0/884123361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1deda0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 >> 172.21.15.31:0/884123361 conn(0x555a1dec13e0 msgr2=0x555a1dec6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 shutdown_connections 2023-07-29T16:16:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 1 -- 172.21.15.31:0/884123361 wait complete. 2023-07-29T16:16:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.340+0000 7f47b255ab80 20 client.16083 trim_cache size 0 max 16384 2023-07-29T16:16:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 Processor -- start 2023-07-29T16:16:50.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 -- start start 2023-07-29T16:16:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 0x555a1dedf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1ded7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded1700 0x555a1ded4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1dec3460 con 0x555a1decfff0 2023-07-29T16:16:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1dec82e0 con 0x555a1ded0940 2023-07-29T16:16:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.354+0000 7f47b255ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1deda810 con 0x555a1ded1700 2023-07-29T16:16:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 0x555a1dedf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 0x555a1dedf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54740/0 (socket says 172.21.15.31:54740) 2023-07-29T16:16:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 -- 172.21.15.31:0/580967478 learned_addr learned my addr 172.21.15.31:0/580967478 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f4797fff700 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1ded7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479d19c700 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded1700 0x555a1ded4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 -- 172.21.15.31:0/580967478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded1700 msgr2=0x555a1ded4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded1700 0x555a1ded4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 -- 172.21.15.31:0/580967478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 msgr2=0x555a1ded7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1ded7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 -- 172.21.15.31:0/580967478 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a1dc71250 con 0x555a1decfff0 2023-07-29T16:16:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479d19c700 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded1700 0x555a1ded4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.355+0000 7f479c99b700 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 0x555a1dedf020 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f478c005950 tx=0x7f478c0cf5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6881cbe4a0dc90fe server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47837fe700 1 -- 172.21.15.31:0/580967478 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c0991d0 con 0x555a1decfff0 2023-07-29T16:16:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47837fe700 1 -- 172.21.15.31:0/580967478 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f478c098f40 con 0x555a1decfff0 2023-07-29T16:16:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47837fe700 1 -- 172.21.15.31:0/580967478 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c09d430 con 0x555a1decfff0 2023-07-29T16:16:50.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 -- 172.21.15.31:0/580967478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 msgr2=0x555a1dedf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 0x555a1dedf020 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f478c005950 tx=0x7f478c0cf5d0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 -- 172.21.15.31:0/580967478 shutdown_connections 2023-07-29T16:16:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded1700 0x555a1ded4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1decfff0 0x555a1dedf020 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 --2- 172.21.15.31:0/580967478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1ded0940 0x555a1ded7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 -- 172.21.15.31:0/580967478 >> 172.21.15.31:0/580967478 conn(0x555a1dec13e0 msgr2=0x555a1dec7700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 -- 172.21.15.31:0/580967478 shutdown_connections 2023-07-29T16:16:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.356+0000 7f47b255ab80 1 -- 172.21.15.31:0/580967478 wait complete. 2023-07-29T16:16:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.357+0000 7f47b255ab80 1 -- start start 2023-07-29T16:16:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.357+0000 7f47b255ab80 1 Processor -- start 2023-07-29T16:16:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.357+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 0x555a1deddbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.357+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded0940 0x555a1dede120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f47b255ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1df375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f47b255ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1dede9a0 con 0x555a1ded1700 2023-07-29T16:16:50.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f47b255ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1df37ae0 con 0x555a1decfff0 2023-07-29T16:16:50.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f47b255ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a1df37c20 con 0x555a1ded0940 2023-07-29T16:16:50.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f47b255ab80 10 client.0 _ll_register_callbacks cb 0x555a1ddc2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:50.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 0x555a1deddbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 0x555a1deddbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36664/0 (socket says 172.21.15.31:36664) 2023-07-29T16:16:50.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479d19c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1df375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 -- 172.21.15.31:0/3529218784 learned_addr learned my addr 172.21.15.31:0/3529218784 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:50.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f4797fff700 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded0940 0x555a1dede120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 -- 172.21.15.31:0/3529218784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded0940 msgr2=0x555a1dede120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded0940 0x555a1dede120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 -- 172.21.15.31:0/3529218784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 msgr2=0x555a1df375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:50.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1df375a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 -- 172.21.15.31:0/3529218784 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479819e6c0 con 0x555a1decfff0 2023-07-29T16:16:50.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f479c99b700 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 0x555a1deddbe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f478c095e70 tx=0x7f478c0a18d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4788a38e993282aa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:50.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.358+0000 7f4777fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:50.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f4777fff700 1 -- 172.21.15.31:0/3529218784 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f478c098d60 con 0x555a1decfff0 2023-07-29T16:16:50.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f4777fff700 1 -- 172.21.15.31:0/3529218784 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f478c09d430 con 0x555a1decfff0 2023-07-29T16:16:50.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f478401e040 con 0x555a1decfff0 2023-07-29T16:16:50.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f4777fff700 1 -- 172.21.15.31:0/3529218784 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f478c003dd0 con 0x555a1decfff0 2023-07-29T16:16:50.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f47b255ab80 10 client.26063 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:50.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x555a1dc492b0 con 0x555a1decfff0 2023-07-29T16:16:50.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f4777fff700 1 -- 172.21.15.31:0/3529218784 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f478401e040 con 0x555a1decfff0 2023-07-29T16:16:50.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.359+0000 7f4777fff700 1 -- 172.21.15.31:0/3529218784 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f478c0988f0 con 0x555a1decfff0 2023-07-29T16:16:50.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f47b255ab80 10 client.26063 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:50.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f47b255ab80 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:50.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f47b255ab80 20 client.26063 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:50.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f47b255ab80 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1df39220 0x555a1df3b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:50.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f47b255ab80 4 client.26063 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:16:50.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555a1de15d40 con 0x555a1df39220 2023-07-29T16:16:50.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f4797fff700 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1df39220 0x555a1df3b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:50.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f4797fff700 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1df39220 0x555a1df3b630 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b057e79ff5050d1 server_cookie=bc29fe26e1343a79 in_seq=0 out_seq=0 2023-07-29T16:16:50.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.360+0000 7f4777fff700 10 client.26063 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:50.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f4777fff700 1 -- 172.21.15.31:0/3529218784 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555a1de15d40 con 0x555a1df39220 2023-07-29T16:16:50.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f4777fff700 10 client.26063 handle_command_reply: tid=0 2023-07-29T16:16:50.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 client.26063 shutdown 2023-07-29T16:16:50.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 msgr2=0x555a1deddbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 0x555a1deddbe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f478c095e70 tx=0x7f478c0a18d0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 shutdown_connections 2023-07-29T16:16:50.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555a1df39220 0x555a1df3b630 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555a1ded1700 0x555a1df375a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a1decfff0 0x555a1deddbe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f4777fff700 0 client.26063 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:50.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 --2- 172.21.15.31:0/3529218784 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555a1ded0940 0x555a1dede120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 >> 172.21.15.31:0/3529218784 conn(0x555a1dec13e0 msgr2=0x555a1ded7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 shutdown_connections 2023-07-29T16:16:50.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 1 -- 172.21.15.31:0/3529218784 wait complete. 2023-07-29T16:16:50.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.361+0000 7f47b255ab80 20 client.26063 trim_cache size 0 max 16384 2023-07-29T16:16:50.380 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:16:50.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47880a0810 msgr2=0x7f47880a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47880a0810 0x7f47880a2cc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x555a1decfd90 tx=0x7f4790004ad0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 msgr2=0x7f4798196030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:50.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 0x7f4798196030 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f478c00e970 tx=0x7f478c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:16:50.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 shutdown_connections 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47880a0810 0x7f47880a2cc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4798101eb0 0x7f479819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4798100330 0x7f4798196030 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 --2- 172.21.15.31:0/2081619545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4798101520 0x7f4798196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 >> 172.21.15.31:0/2081619545 conn(0x7f47980fb970 msgr2=0x7f4798104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 shutdown_connections 2023-07-29T16:16:50.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:50.362+0000 7f479ebff700 1 -- 172.21.15.31:0/2081619545 wait complete. 2023-07-29T16:16:50.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:50 smithi031 ceph-mon[95504]: pgmap v410: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:50.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:50 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1798376758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:16:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:50 smithi039 ceph-mon[128858]: pgmap v410: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:50 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1798376758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:16:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:50 smithi156 ceph-mon[129245]: pgmap v410: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:16:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:50 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1798376758' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:16:51.384 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:16:51.384 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:16:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 -- 172.21.15.31:0/2641400434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fec10 msgr2=0x7fca940ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/2641400434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fec10 0x7fca940ff020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fca840098f0 tx=0x7fca84009c00 comp rx=0 tx=0).stop 2023-07-29T16:16:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 -- 172.21.15.31:0/2641400434 shutdown_connections 2023-07-29T16:16:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/2641400434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca940fff10 0x7fca94100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/2641400434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fec10 0x7fca940ff020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/2641400434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca940ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 -- 172.21.15.31:0/2641400434 >> 172.21.15.31:0/2641400434 conn(0x7fca940fa840 msgr2=0x7fca940fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 -- 172.21.15.31:0/2641400434 shutdown_connections 2023-07-29T16:16:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.601+0000 7fca9a5ff700 1 -- 172.21.15.31:0/2641400434 wait complete. 2023-07-29T16:16:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.602+0000 7fca9a5ff700 1 Processor -- start 2023-07-29T16:16:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.602+0000 7fca9a5ff700 1 -- start start 2023-07-29T16:16:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca9a5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca940fec10 0x7fca941982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca9a5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca94198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca9a5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fff10 0x7fca9419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca9a5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9419fe60 con 0x7fca940ff560 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca9a5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9419ffa0 con 0x7fca940fff10 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca9a5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca941a0130 con 0x7fca940fec10 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca94198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca94198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54764/0 (socket says 172.21.15.31:54764) 2023-07-29T16:16:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca8bfff700 1 -- 172.21.15.31:0/907563913 learned_addr learned my addr 172.21.15.31:0/907563913 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca93fff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca940fec10 0x7fca941982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca98b9c700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fff10 0x7fca9419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.603+0000 7fca8bfff700 1 -- 172.21.15.31:0/907563913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca940fec10 msgr2=0x7fca941982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca8bfff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca940fec10 0x7fca941982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca8bfff700 1 -- 172.21.15.31:0/907563913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fff10 msgr2=0x7fca9419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca8bfff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fff10 0x7fca9419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca8bfff700 1 -- 172.21.15.31:0/907563913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0095f0 con 0x7fca940ff560 2023-07-29T16:16:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca8bfff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca94198820 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00ec10 tx=0x7fca7c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca91ffb700 1 -- 172.21.15.31:0/907563913 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00cc90 con 0x7fca940ff560 2023-07-29T16:16:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca91ffb700 1 -- 172.21.15.31:0/907563913 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca7c0044e0 con 0x7fca940ff560 2023-07-29T16:16:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca91ffb700 1 -- 172.21.15.31:0/907563913 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c010430 con 0x7fca940ff560 2023-07-29T16:16:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca84009520 con 0x7fca940ff560 2023-07-29T16:16:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.604+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941a0740 con 0x7fca940ff560 2023-07-29T16:16:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.608+0000 7fca91ffb700 1 -- 172.21.15.31:0/907563913 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fca7c003fb0 con 0x7fca940ff560 2023-07-29T16:16:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.609+0000 7fcaadf29b80 1 Processor -- start 2023-07-29T16:16:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.609+0000 7fca91ffb700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca800a0810 0x7fca800a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.609+0000 7fca91ffb700 1 -- 172.21.15.31:0/907563913 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca7c014070 con 0x7fca940ff560 2023-07-29T16:16:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.609+0000 7fcaadf29b80 1 -- start start 2023-07-29T16:16:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.609+0000 7fca98b9c700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca800a0810 0x7fca800a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 0x562d2a804070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a804de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a805370 0x562d2a8057c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a75ba90 con 0x562d2a803c60 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a759450 con 0x562d2a805370 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a754d40 con 0x562d2a8045b0 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca98b9c700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca800a0810 0x7fca800a2cc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00f540 tx=0x7fca8c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca93fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 0x562d2a804070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a804de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca93fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 0x562d2a804070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54776/0 (socket says 172.21.15.31:54776) 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a804de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35142/0 (socket says 172.21.15.31:35142) 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca93fff700 1 -- 172.21.15.31:0/1194881200 learned_addr learned my addr 172.21.15.31:0/1194881200 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.610+0000 7fca98b9c700 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a805370 0x562d2a8057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca93fff700 1 -- 172.21.15.31:0/1194881200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 msgr2=0x562d2a804de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca93fff700 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a804de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca93fff700 1 -- 172.21.15.31:0/1194881200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a805370 msgr2=0x562d2a8057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca93fff700 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a805370 0x562d2a8057c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca93fff700 1 -- 172.21.15.31:0/1194881200 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d2a582e60 con 0x562d2a803c60 2023-07-29T16:16:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a804de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca93fff700 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 0x562d2a804070 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca8400f9e0 tx=0x7fca84005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3d173d2c8df15e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca8a7fc700 1 -- 172.21.15.31:0/1194881200 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8400fdb0 con 0x562d2a803c60 2023-07-29T16:16:51.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca8a7fc700 1 -- 172.21.15.31:0/1194881200 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca8401d040 con 0x562d2a803c60 2023-07-29T16:16:51.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.611+0000 7fca8a7fc700 1 -- 172.21.15.31:0/1194881200 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca84005040 con 0x562d2a803c60 2023-07-29T16:16:51.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1194881200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 msgr2=0x562d2a804070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 0x562d2a804070 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca8400f9e0 tx=0x7fca84005dc0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1194881200 shutdown_connections 2023-07-29T16:16:51.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a805370 0x562d2a8057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a803c60 0x562d2a804070 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1194881200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a804de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1194881200 >> 172.21.15.31:0/1194881200 conn(0x562d2a7f9460 msgr2=0x562d2a7fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1194881200 shutdown_connections 2023-07-29T16:16:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.612+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1194881200 wait complete. 2023-07-29T16:16:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.613+0000 7fcaadf29b80 1 -- start start 2023-07-29T16:16:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.613+0000 7fcaadf29b80 1 Processor -- start 2023-07-29T16:16:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a80dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a814db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a6f1590 con 0x562d2a8045b0 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a6f1230 con 0x562d2a803c60 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a5d89a0 con 0x562d2a805370 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fcaadf29b80 10 client.0 _ll_register_callbacks cb 0x562d2a6fa6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a80dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fca93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a80dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54790/0 (socket says 172.21.15.31:54790) 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fca8bfff700 1 -- 172.21.15.31:0/1716117308 learned_addr learned my addr 172.21.15.31:0/1716117308 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.614+0000 7fca98b9c700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a814db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca93fff700 1 -- 172.21.15.31:0/1716117308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 msgr2=0x562d2a814db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca93fff700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a814db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca93fff700 1 -- 172.21.15.31:0/1716117308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 msgr2=0x562d2a80dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca93fff700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a80dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca93fff700 1 -- 172.21.15.31:0/1716117308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941a0740 con 0x562d2a803c60 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a80dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca93fff700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x562d2a7f93e0 tx=0x7fca8400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42cf8b6b848a1f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca737fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca737fe700 1 -- 172.21.15.31:0/1716117308 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8400f660 con 0x562d2a803c60 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d2a77fd90 con 0x562d2a803c60 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.615+0000 7fca737fe700 1 -- 172.21.15.31:0/1716117308 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca840050d0 con 0x562d2a803c60 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fca737fe700 1 -- 172.21.15.31:0/1716117308 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca8400f7c0 con 0x562d2a803c60 2023-07-29T16:16:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 10 client.26075 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fca9410b260 con 0x562d2a803c60 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fca737fe700 1 -- 172.21.15.31:0/1716117308 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562d2a77fd90 con 0x562d2a803c60 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fca737fe700 1 -- 172.21.15.31:0/1716117308 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca84028070 con 0x562d2a803c60 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 10 client.26075 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 10 client.26075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 20 client.26075 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a819070 0x562d2a81b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 4 client.26075 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.616+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d2a7d2df0 con 0x562d2a819070 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.617+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a819070 0x562d2a81b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.617+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a819070 0x562d2a81b450 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86d5a716ce4525da server_cookie=d547d72e69e46a34 in_seq=0 out_seq=0 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.617+0000 7fca737fe700 10 client.26075 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fca737fe700 1 -- 172.21.15.31:0/1716117308 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562d2a7d2df0 con 0x562d2a819070 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fca737fe700 10 client.26075 handle_command_reply: tid=0 2023-07-29T16:16:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 client.26075 shutdown 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 msgr2=0x562d2a80d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x562d2a7f93e0 tx=0x7fca8400bfd0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 shutdown_connections 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a819070 0x562d2a81b450 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a814db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fca737fe700 0 client.26075 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1716117308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a80dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 >> 172.21.15.31:0/1716117308 conn(0x562d2a7f9460 msgr2=0x562d2a7fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.618+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 shutdown_connections 2023-07-29T16:16:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.619+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1716117308 wait complete. 2023-07-29T16:16:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.619+0000 7fcaadf29b80 20 client.26075 trim_cache size 0 max 16384 2023-07-29T16:16:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 Processor -- start 2023-07-29T16:16:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 -- start start 2023-07-29T16:16:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a814d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a80db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a80cce0 con 0x562d2a8045b0 2023-07-29T16:16:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a7fb4e0 con 0x562d2a803c60 2023-07-29T16:16:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a804980 con 0x562d2a805370 2023-07-29T16:16:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.636+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a814d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a814d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54804/0 (socket says 172.21.15.31:54804) 2023-07-29T16:16:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 -- 172.21.15.31:0/1937419246 learned_addr learned my addr 172.21.15.31:0/1937419246 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca98b9c700 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a80db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 -- 172.21.15.31:0/1937419246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 msgr2=0x562d2a80db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a80db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 -- 172.21.15.31:0/1937419246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 msgr2=0x562d2a80d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 -- 172.21.15.31:0/1937419246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d2a56bce0 con 0x562d2a8045b0 2023-07-29T16:16:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca98b9c700 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a80db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8bfff700 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a814d70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c5b0 tx=0x7fca7c09c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca686ceff2257a8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8affd700 1 -- 172.21.15.31:0/1937419246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c0c7070 con 0x562d2a8045b0 2023-07-29T16:16:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8affd700 1 -- 172.21.15.31:0/1937419246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca7c099510 con 0x562d2a8045b0 2023-07-29T16:16:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.637+0000 7fca8affd700 1 -- 172.21.15.31:0/1937419246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c0c9c80 con 0x562d2a8045b0 2023-07-29T16:16:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1937419246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 msgr2=0x562d2a814d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a814d70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c5b0 tx=0x7fca7c09c040 comp rx=0 tx=0).stop 2023-07-29T16:16:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1937419246 shutdown_connections 2023-07-29T16:16:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a805370 0x562d2a80db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a80d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/1937419246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a8045b0 0x562d2a814d70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1937419246 >> 172.21.15.31:0/1937419246 conn(0x562d2a7f9460 msgr2=0x562d2a808270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1937419246 shutdown_connections 2023-07-29T16:16:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 -- 172.21.15.31:0/1937419246 wait complete. 2023-07-29T16:16:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.638+0000 7fcaadf29b80 1 -- start start 2023-07-29T16:16:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 Processor -- start 2023-07-29T16:16:51.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a811dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a812310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a805370 0x562d2a813320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a813860 con 0x562d2a805370 2023-07-29T16:16:51.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a8139d0 con 0x562d2a803c60 2023-07-29T16:16:51.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d2a813b90 con 0x562d2a8045b0 2023-07-29T16:16:51.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fcaadf29b80 10 client.0 _ll_register_callbacks cb 0x562d2a6fa6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:51.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fca98b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a805370 0x562d2a813320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fca93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a811dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fca93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a811dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36708/0 (socket says 172.21.15.31:36708) 2023-07-29T16:16:51.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fca93fff700 1 -- 172.21.15.31:0/2339635550 learned_addr learned my addr 172.21.15.31:0/2339635550 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.639+0000 7fca8bfff700 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a812310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca93fff700 1 -- 172.21.15.31:0/2339635550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 msgr2=0x562d2a812310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca93fff700 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a812310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca93fff700 1 -- 172.21.15.31:0/2339635550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a805370 msgr2=0x562d2a813320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca93fff700 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a805370 0x562d2a813320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca93fff700 1 -- 172.21.15.31:0/2339635550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca84009520 con 0x562d2a803c60 2023-07-29T16:16:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca93fff700 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a811dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fca8400a010 tx=0x7fca8400ba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd3bdc1dd4f4d090 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca737fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca737fe700 1 -- 172.21.15.31:0/2339635550 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8402ae90 con 0x562d2a803c60 2023-07-29T16:16:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca8401d040 con 0x562d2a803c60 2023-07-29T16:16:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.640+0000 7fca737fe700 1 -- 172.21.15.31:0/2339635550 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca84000e40 con 0x562d2a803c60 2023-07-29T16:16:51.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fca737fe700 1 -- 172.21.15.31:0/2339635550 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca84005920 con 0x562d2a803c60 2023-07-29T16:16:51.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 10 client.26081 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:51.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562d2a5a92d0 con 0x562d2a803c60 2023-07-29T16:16:51.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fca737fe700 1 -- 172.21.15.31:0/2339635550 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca8404c020 con 0x562d2a803c60 2023-07-29T16:16:51.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fca737fe700 1 -- 172.21.15.31:0/2339635550 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca84009920 con 0x562d2a803c60 2023-07-29T16:16:51.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 10 client.26081 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:51.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 10 client.26081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 20 client.26081 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a86cb90 0x562d2a86ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 4 client.26081 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:16:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562d2a74ddc0 con 0x562d2a86cb90 2023-07-29T16:16:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.641+0000 7fca8bfff700 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a86cb90 0x562d2a86ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.642+0000 7fca8bfff700 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a86cb90 0x562d2a86ef70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34043edd35663d2f server_cookie=157472c64a65188a in_seq=0 out_seq=0 2023-07-29T16:16:51.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.642+0000 7fca737fe700 10 client.26081 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fca737fe700 1 -- 172.21.15.31:0/2339635550 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562d2a74ddc0 con 0x562d2a86cb90 2023-07-29T16:16:51.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fca737fe700 10 client.26081 handle_command_reply: tid=0 2023-07-29T16:16:51.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 client.26081 shutdown 2023-07-29T16:16:51.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 msgr2=0x562d2a811dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a811dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fca8400a010 tx=0x7fca8400ba80 comp rx=0 tx=0).stop 2023-07-29T16:16:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 shutdown_connections 2023-07-29T16:16:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d2a86cb90 0x562d2a86ef70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d2a805370 0x562d2a813320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d2a803c60 0x562d2a811dd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fca737fe700 0 client.26081 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 --2- 172.21.15.31:0/2339635550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d2a8045b0 0x562d2a812310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.643+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 >> 172.21.15.31:0/2339635550 conn(0x562d2a7f9460 msgr2=0x562d2a80d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 shutdown_connections 2023-07-29T16:16:51.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fcaadf29b80 1 -- 172.21.15.31:0/2339635550 wait complete. 2023-07-29T16:16:51.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fcaadf29b80 20 client.26081 trim_cache size 0 max 16384 2023-07-29T16:16:51.661 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:16:51.661 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:16:51.661 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "8a1e3538-9138-4e91-9be4-c7b485efb4c4", 2023-07-29T16:16:51.662 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:16:51.662 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:16:51.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca800a0810 msgr2=0x7fca800a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca800a0810 0x7fca800a2cc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00f540 tx=0x7fca8c00b040 comp rx=0 tx=0).stop 2023-07-29T16:16:51.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 msgr2=0x7fca94198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca94198820 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00ec10 tx=0x7fca7c00ef20 comp rx=0 tx=0).stop 2023-07-29T16:16:51.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 shutdown_connections 2023-07-29T16:16:51.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca800a0810 0x7fca800a2cc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca940fff10 0x7fca9419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca940fec10 0x7fca941982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 --2- 172.21.15.31:0/907563913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca940ff560 0x7fca94198820 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.644+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 >> 172.21.15.31:0/907563913 conn(0x7fca940fa840 msgr2=0x7fca940fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.645+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 shutdown_connections 2023-07-29T16:16:51.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.645+0000 7fca9a5ff700 1 -- 172.21.15.31:0/907563913 wait complete. 2023-07-29T16:16:51.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:16:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 -- 172.21.15.31:0/3152226216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af8074d90 msgr2=0x7f5af80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 --2- 172.21.15.31:0/3152226216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af8074d90 0x7f5af80731e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009990 tx=0x7f5ae8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 -- 172.21.15.31:0/3152226216 shutdown_connections 2023-07-29T16:16:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 --2- 172.21.15.31:0/3152226216 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 0x7f5af810a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 --2- 172.21.15.31:0/3152226216 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af8074d90 0x7f5af80731e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 --2- 172.21.15.31:0/3152226216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af80737b0 0x7f5af8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 -- 172.21.15.31:0/3152226216 >> 172.21.15.31:0/3152226216 conn(0x7f5af80fb920 msgr2=0x7f5af80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 -- 172.21.15.31:0/3152226216 shutdown_connections 2023-07-29T16:16:51.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.880+0000 7f5afcdff700 1 -- 172.21.15.31:0/3152226216 wait complete. 2023-07-29T16:16:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 Processor -- start 2023-07-29T16:16:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 -- start start 2023-07-29T16:16:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 0x7f5af8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 0x7f5af8198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 0x7f5af819f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af819fe90 con 0x7f5af80737b0 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af819ffd0 con 0x7f5af8074d90 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.881+0000 7f5afcdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af81a0160 con 0x7f5af81081e0 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 0x7f5af8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 0x7f5af819f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 0x7f5af8198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 0x7f5af8198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54820/0 (socket says 172.21.15.31:54820) 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 0x7f5af8198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36720/0 (socket says 172.21.15.31:36720) 2023-07-29T16:16:51.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af659c700 1 -- 172.21.15.31:0/2278694383 learned_addr learned my addr 172.21.15.31:0/2278694383 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 0x7f5af819f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35194/0 (socket says 172.21.15.31:35194) 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 -- 172.21.15.31:0/2278694383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 msgr2=0x7f5af819f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 0x7f5af819f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 -- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 msgr2=0x7f5af8198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 0x7f5af8198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 -- 172.21.15.31:0/2278694383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec009520 con 0x7f5af8074d90 2023-07-29T16:16:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af659c700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 0x7f5af8198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.882+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 0x7f5af8198850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e9c0 tx=0x7f5aec00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.883+0000 7f5ae77fe700 1 -- 172.21.15.31:0/2278694383 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00cb70 con 0x7f5af8074d90 2023-07-29T16:16:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.883+0000 7f5ae77fe700 1 -- 172.21.15.31:0/2278694383 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5aec00ee60 con 0x7f5af8074d90 2023-07-29T16:16:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.883+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae80095f0 con 0x7f5af8074d90 2023-07-29T16:16:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.883+0000 7f5ae77fe700 1 -- 172.21.15.31:0/2278694383 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec018680 con 0x7f5af8074d90 2023-07-29T16:16:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.883+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af81a07c0 con 0x7f5af8074d90 2023-07-29T16:16:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.886+0000 7f5ae77fe700 1 -- 172.21.15.31:0/2278694383 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5aec010430 con 0x7f5af8074d90 2023-07-29T16:16:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.887+0000 7f5b107f6b80 1 Processor -- start 2023-07-29T16:16:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 -- start start 2023-07-29T16:16:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6da99ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6da9ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6d9f1a10 con 0x558f6da9b2f0 2023-07-29T16:16:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6d9ef3d0 con 0x558f6da9a530 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6d9eacc0 con 0x558f6da99be0 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5ae77fe700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5ae00a0790 0x7f5ae00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5af659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6da9ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.888+0000 7f5af5d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6da9ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36734/0 (socket says 172.21.15.31:36734) 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 -- 172.21.15.31:0/1091888313 learned_addr learned my addr 172.21.15.31:0/1091888313 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6da99ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5ae77fe700 1 -- 172.21.15.31:0/2278694383 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5aec014070 con 0x7f5af8074d90 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5ae00a0790 0x7f5ae00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 -- 172.21.15.31:0/1091888313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 msgr2=0x558f6da99ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6da99ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 -- 172.21.15.31:0/1091888313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 msgr2=0x558f6da9b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 -- 172.21.15.31:0/1091888313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f6d818de0 con 0x558f6da9a530 2023-07-29T16:16:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af659c700 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6da9ad60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8005f50 tx=0x7f5ae8005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92dda110dd7ea41b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5ae00a0790 0x7f5ae00a2c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009ec0 tx=0x7f5af000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.889+0000 7f5ae4ff9700 1 -- 172.21.15.31:0/1091888313 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae801d070 con 0x558f6da9a530 2023-07-29T16:16:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5ae4ff9700 1 -- 172.21.15.31:0/1091888313 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ae801e040 con 0x558f6da9a530 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5ae4ff9700 1 -- 172.21.15.31:0/1091888313 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8004dd0 con 0x558f6da9a530 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 -- 172.21.15.31:0/1091888313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 msgr2=0x558f6da9ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6da9ad60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8005f50 tx=0x7f5ae8005bc0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 -- 172.21.15.31:0/1091888313 shutdown_connections 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6da99ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/1091888313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6da9ad60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 -- 172.21.15.31:0/1091888313 >> 172.21.15.31:0/1091888313 conn(0x558f6da8f3e0 msgr2=0x558f6da94990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 -- 172.21.15.31:0/1091888313 shutdown_connections 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.890+0000 7f5b107f6b80 1 -- 172.21.15.31:0/1091888313 wait complete. 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.891+0000 7f5b107f6b80 1 -- start start 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.891+0000 7f5b107f6b80 1 Processor -- start 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.891+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daa3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daa3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6daaad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6d987510 con 0x558f6da9b2f0 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6d9871b0 con 0x558f6da99be0 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6d86e920 con 0x558f6da9a530 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5b107f6b80 10 client.0 _ll_register_callbacks cb 0x558f6d990620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daa3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6daaad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daa3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36750/0 (socket says 172.21.15.31:36750) 2023-07-29T16:16:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 -- 172.21.15.31:0/2601196381 learned_addr learned my addr 172.21.15.31:0/2601196381 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daa3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 -- 172.21.15.31:0/2601196381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 msgr2=0x558f6daa3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daa3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 -- 172.21.15.31:0/2601196381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 msgr2=0x558f6daaad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6daaad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.892+0000 7f5af659c700 1 -- 172.21.15.31:0/2601196381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af810b280 con 0x558f6da99be0 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5af659c700 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daa3780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8000c00 tx=0x7f5ae80226b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2099bf099933b9fc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5addffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5addffb700 1 -- 172.21.15.31:0/2601196381 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae801d030 con 0x558f6da99be0 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f6da15d10 con 0x558f6da99be0 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5addffb700 1 -- 172.21.15.31:0/2601196381 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5ae8022de0 con 0x558f6da99be0 2023-07-29T16:16:51.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5addffb700 1 -- 172.21.15.31:0/2601196381 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ae800f460 con 0x558f6da99be0 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5b107f6b80 10 client.26093 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558f6d801c60 con 0x558f6da99be0 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.893+0000 7f5addffb700 1 -- 172.21.15.31:0/2601196381 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558f6da15d10 con 0x558f6da99be0 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5addffb700 1 -- 172.21.15.31:0/2601196381 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5ae802c5a0 con 0x558f6da99be0 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5b107f6b80 10 client.26093 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5b107f6b80 10 client.26093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:51.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5b107f6b80 20 client.26093 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6daaf0a0 0x558f6dab1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5b107f6b80 4 client.26093 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f6da68d70 con 0x558f6daaf0a0 2023-07-29T16:16:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6daaf0a0 0x558f6dab1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6daaf0a0 0x558f6dab1480 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54f27e9f24e554b1 server_cookie=2b3bc5dc1588ff3 in_seq=0 out_seq=0 2023-07-29T16:16:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.894+0000 7f5addffb700 10 client.26093 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.895+0000 7f5addffb700 1 -- 172.21.15.31:0/2601196381 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558f6da68d70 con 0x558f6daaf0a0 2023-07-29T16:16:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.895+0000 7f5addffb700 10 client.26093 handle_command_reply: tid=0 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.895+0000 7f5b107f6b80 1 client.26093 shutdown 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 msgr2=0x558f6daa3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daa3780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8000c00 tx=0x7f5ae80226b0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 shutdown_connections 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6daaf0a0 0x558f6dab1480 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6daaad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daa3780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5addffb700 0 client.26093 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2601196381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daa3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 >> 172.21.15.31:0/2601196381 conn(0x558f6da8f3e0 msgr2=0x558f6da94060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 shutdown_connections 2023-07-29T16:16:51.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2601196381 wait complete. 2023-07-29T16:16:51.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.896+0000 7f5b107f6b80 20 client.26093 trim_cache size 0 max 16384 2023-07-29T16:16:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 Processor -- start 2023-07-29T16:16:51.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 -- start start 2023-07-29T16:16:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6daa8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6daa1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6da91460 con 0x558f6da9b2f0 2023-07-29T16:16:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6da8c770 con 0x558f6da9a530 2023-07-29T16:16:51.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.913+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6daa1530 con 0x558f6da99be0 2023-07-29T16:16:51.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54838/0 (socket says 172.21.15.31:54838) 2023-07-29T16:16:51.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 -- 172.21.15.31:0/2735036613 learned_addr learned my addr 172.21.15.31:0/2735036613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af659c700 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6daa8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6daa1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 -- 172.21.15.31:0/2735036613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 msgr2=0x558f6daa8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6daa8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 -- 172.21.15.31:0/2735036613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 msgr2=0x558f6daa1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6daa1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 -- 172.21.15.31:0/2735036613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f6d83f250 con 0x558f6da9b2f0 2023-07-29T16:16:51.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6daa1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:51.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9e2d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5af0003cf0 tx=0x7f5af0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f0004953866a52 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.914+0000 7f5ae57fa700 1 -- 172.21.15.31:0/2735036613 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000e070 con 0x558f6da9b2f0 2023-07-29T16:16:51.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5ae57fa700 1 -- 172.21.15.31:0/2735036613 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5af000f040 con 0x558f6da9b2f0 2023-07-29T16:16:51.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5ae57fa700 1 -- 172.21.15.31:0/2735036613 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af001b450 con 0x558f6da9b2f0 2023-07-29T16:16:51.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2735036613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 msgr2=0x558f6da9e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9e2d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5af0003cf0 tx=0x7f5af0009920 comp rx=0 tx=0).stop 2023-07-29T16:16:51.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2735036613 shutdown_connections 2023-07-29T16:16:51.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6da9e2d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da99be0 0x558f6daa8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/2735036613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da9a530 0x558f6daa1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2735036613 >> 172.21.15.31:0/2735036613 conn(0x558f6da8f3e0 msgr2=0x558f6da95690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2735036613 shutdown_connections 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.915+0000 7f5b107f6b80 1 -- 172.21.15.31:0/2735036613 wait complete. 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 -- start start 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 Processor -- start 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daf8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daf9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6db00220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6db007d0 con 0x558f6da9b2f0 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6db00940 con 0x558f6da99be0 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f6db00b00 con 0x558f6da9a530 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.916+0000 7f5b107f6b80 10 client.0 _ll_register_callbacks cb 0x558f6d990620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6db00220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daf9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daf9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35230/0 (socket says 172.21.15.31:35230) 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6db00220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54846/0 (socket says 172.21.15.31:54846) 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af5d9b700 1 -- 172.21.15.31:0/3586011573 learned_addr learned my addr 172.21.15.31:0/3586011573 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af659c700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daf8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 -- 172.21.15.31:0/3586011573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 msgr2=0x558f6daf9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daf9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 -- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 msgr2=0x558f6daf8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:51.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daf8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 -- 172.21.15.31:0/3586011573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af0003a00 con 0x558f6da9b2f0 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af5d9b700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daf9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5af6d9d700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6db00220 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5af0003cf0 tx=0x7f5af0008460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ddf3454fa7594c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.917+0000 7f5addffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5addffb700 1 -- 172.21.15.31:0/3586011573 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000e040 con 0x558f6da9b2f0 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ae801e040 con 0x558f6da9b2f0 2023-07-29T16:16:51.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5addffb700 1 -- 172.21.15.31:0/3586011573 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5af0004cd0 con 0x558f6da9b2f0 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5addffb700 1 -- 172.21.15.31:0/3586011573 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5af001aed0 con 0x558f6da9b2f0 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5b107f6b80 10 client.16125 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558f6d8172b0 con 0x558f6da9b2f0 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5addffb700 1 -- 172.21.15.31:0/3586011573 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5af003d020 con 0x558f6da9b2f0 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5addffb700 1 -- 172.21.15.31:0/3586011573 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5af002e070 con 0x558f6da9b2f0 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5b107f6b80 10 client.16125 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:51.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5b107f6b80 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.918+0000 7f5b107f6b80 20 client.16125 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6db02370 0x558f6db04780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5b107f6b80 4 client.16125 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558f6d9e3d40 con 0x558f6db02370 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5af659c700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6db02370 0x558f6db04780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5af659c700 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6db02370 0x558f6db04780 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=366ca983562805a3 server_cookie=b626818c5b6f2025 in_seq=0 out_seq=0 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5addffb700 10 client.16125 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5addffb700 1 -- 172.21.15.31:0/3586011573 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558f6d9e3d40 con 0x558f6db02370 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5addffb700 10 client.16125 handle_command_reply: tid=0 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.919+0000 7f5b107f6b80 1 client.16125 shutdown 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 msgr2=0x558f6db00220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6db00220 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f5af0003cf0 tx=0x7f5af0008460 comp rx=0 tx=0).stop 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 shutdown_connections 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558f6db02370 0x558f6db04780 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558f6da9b2f0 0x558f6db00220 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558f6da99be0 0x558f6daf8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 --2- 172.21.15.31:0/3586011573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558f6da9a530 0x558f6daf9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5addffb700 0 client.16125 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:51.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 >> 172.21.15.31:0/3586011573 conn(0x558f6da8f3e0 msgr2=0x558f6daa1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 shutdown_connections 2023-07-29T16:16:51.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 1 -- 172.21.15.31:0/3586011573 wait complete. 2023-07-29T16:16:51.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.920+0000 7f5b107f6b80 20 client.16125 trim_cache size 0 max 16384 2023-07-29T16:16:51.934 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:16:51.934 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:16:51.934 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:16:51.934 INFO:teuthology.orchestra.run.smithi031.stdout: "8a1e3538-9138-4e91-9be4-c7b485efb4c4": { 2023-07-29T16:16:51.935 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:16:51.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "8a1e3538-9138-4e91-9be4-c7b485efb4c4", 2023-07-29T16:16:51.935 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:16:51.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:16:51.935 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:16:51.935 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:16:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5ae00a0790 msgr2=0x7f5ae00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5ae00a0790 0x7f5ae00a2c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009ec0 tx=0x7f5af000b040 comp rx=0 tx=0).stop 2023-07-29T16:16:51.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 msgr2=0x7f5af8198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 0x7f5af8198850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e9c0 tx=0x7f5aec00ecd0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 shutdown_connections 2023-07-29T16:16:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5ae00a0790 0x7f5ae00a2c40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af81081e0 0x7f5af819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af80737b0 0x7f5af8198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 --2- 172.21.15.31:0/2278694383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af8074d90 0x7f5af8198850 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 >> 172.21.15.31:0/2278694383 conn(0x7f5af80fb920 msgr2=0x7f5af80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 shutdown_connections 2023-07-29T16:16:51.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:16:51.921+0000 7f5afcdff700 1 -- 172.21.15.31:0/2278694383 wait complete. 2023-07-29T16:16:51.942 INFO:tasks.cephfs.filesystem:scrub status for tag:8a1e3538-9138-4e91-9be4-c7b485efb4c4 - {'path': '/', 'tag': '8a1e3538-9138-4e91-9be4-c7b485efb4c4', 'options': 'recursive,force'} 2023-07-29T16:16:52.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:52 smithi031 ceph-mon[95504]: pgmap v411: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:52 smithi039 ceph-mon[128858]: pgmap v411: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:52 smithi156 ceph-mon[129245]: pgmap v411: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:16:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:53 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:16:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:53 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:53 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:16:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:53 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:53.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:16:53.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:53 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:16:53.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:53 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:53 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:16:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:53 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:16:53.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:53 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:16:53.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:53 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:53.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:53 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:16:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:53 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:16:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:16:54.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:54 smithi031 ceph-mon[95504]: pgmap v412: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:54 smithi039 ceph-mon[128858]: pgmap v412: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:54 smithi156 ceph-mon[129245]: pgmap v412: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:16:56.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:56 smithi031 ceph-mon[95504]: pgmap v413: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:56.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:56 smithi039 ceph-mon[128858]: pgmap v413: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:56 smithi156 ceph-mon[129245]: pgmap v413: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 -- 172.21.15.31:0/3766183449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7158074d90 msgr2=0x7f71580731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 --2- 172.21.15.31:0/3766183449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7158074d90 0x7f71580731e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7148009990 tx=0x7f7148009ca0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 -- 172.21.15.31:0/3766183449 shutdown_connections 2023-07-29T16:16:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 --2- 172.21.15.31:0/3766183449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 0x7f715810ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 --2- 172.21.15.31:0/3766183449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7158074d90 0x7f71580731e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 --2- 172.21.15.31:0/3766183449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71580737b0 0x7f7158073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 -- 172.21.15.31:0/3766183449 >> 172.21.15.31:0/3766183449 conn(0x7f71580fb920 msgr2=0x7f71580fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:57.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 -- 172.21.15.31:0/3766183449 shutdown_connections 2023-07-29T16:16:57.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.479+0000 7f715e3ff700 1 -- 172.21.15.31:0/3766183449 wait complete. 2023-07-29T16:16:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.480+0000 7f715e3ff700 1 Processor -- start 2023-07-29T16:16:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 -- start start 2023-07-29T16:16:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 0x7f715819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7158074d90 0x7f715819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 0x7f715819dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715819e200 con 0x7f715810c910 2023-07-29T16:16:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715819e370 con 0x7f71580737b0 2023-07-29T16:16:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715819e530 con 0x7f7158074d90 2023-07-29T16:16:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f7157fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 0x7f715819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f715c99c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 0x7f715819dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f7157fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 0x7f715819c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36784/0 (socket says 172.21.15.31:36784) 2023-07-29T16:16:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f7157fff700 1 -- 172.21.15.31:0/2347557535 learned_addr learned my addr 172.21.15.31:0/2347557535 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.481+0000 7f71577fe700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7158074d90 0x7f715819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f7157fff700 1 -- 172.21.15.31:0/2347557535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7158074d90 msgr2=0x7f715819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f7157fff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7158074d90 0x7f715819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f7157fff700 1 -- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 msgr2=0x7f715819dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f7157fff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 0x7f715819dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f7157fff700 1 -- 172.21.15.31:0/2347557535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71480095f0 con 0x7f71580737b0 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f715c99c700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 0x7f715819dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:57.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f7157fff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 0x7f715819c7e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7148005720 tx=0x7f7148009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f71557fa700 1 -- 172.21.15.31:0/2347557535 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714801d070 con 0x7f71580737b0 2023-07-29T16:16:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715819e7b0 con 0x7f71580737b0 2023-07-29T16:16:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f71557fa700 1 -- 172.21.15.31:0/2347557535 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7148021410 con 0x7f71580737b0 2023-07-29T16:16:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.482+0000 7f71557fa700 1 -- 172.21.15.31:0/2347557535 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148017650 con 0x7f71580737b0 2023-07-29T16:16:57.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.483+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71581a4fd0 con 0x7f71580737b0 2023-07-29T16:16:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.486+0000 7f71557fa700 1 -- 172.21.15.31:0/2347557535 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f7148021580 con 0x7f71580737b0 2023-07-29T16:16:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.487+0000 7f7171d51b80 1 Processor -- start 2023-07-29T16:16:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 -- start start 2023-07-29T16:16:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced785b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced785c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced785d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced77b39b0 con 0x55ced785bdf0 2023-07-29T16:16:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced77b1370 con 0x55ced76b76b0 2023-07-29T16:16:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7171d51b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced77acc60 con 0x55ced785ccf0 2023-07-29T16:16:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f71557fa700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71400a0790 0x7f71400a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f71577fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced785d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f7157fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced785c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f71577fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced785d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35266/0 (socket says 172.21.15.31:35266) 2023-07-29T16:16:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f71577fe700 1 -- 172.21.15.31:0/413514116 learned_addr learned my addr 172.21.15.31:0/413514116 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f71557fa700 1 -- 172.21.15.31:0/2347557535 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f71480ccc60 con 0x7f71580737b0 2023-07-29T16:16:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.488+0000 7f715c99c700 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced785b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f715c99c700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71400a0790 0x7f71400a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f71577fe700 1 -- 172.21.15.31:0/413514116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 msgr2=0x55ced785b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f71577fe700 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced785b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f71577fe700 1 -- 172.21.15.31:0/413514116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 msgr2=0x55ced785c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f71577fe700 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced785c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f71577fe700 1 -- 172.21.15.31:0/413514116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ced75dad80 con 0x55ced785ccf0 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f71577fe700 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced785d160 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f714c009d30 tx=0x7f714c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e911ccad7aef728 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f715c99c700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71400a0790 0x7f71400a2c40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7150009fd0 tx=0x7f715000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.489+0000 7f713e7fc700 1 -- 172.21.15.31:0/413514116 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c004220 con 0x55ced785ccf0 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f713e7fc700 1 -- 172.21.15.31:0/413514116 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f714c004380 con 0x55ced785ccf0 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f713e7fc700 1 -- 172.21.15.31:0/413514116 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0198d0 con 0x55ced785ccf0 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 -- 172.21.15.31:0/413514116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 msgr2=0x55ced785d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced785d160 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f714c009d30 tx=0x7f714c011720 comp rx=0 tx=0).stop 2023-07-29T16:16:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 -- 172.21.15.31:0/413514116 shutdown_connections 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced785d160 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced785b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 --2- 172.21.15.31:0/413514116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced785c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 -- 172.21.15.31:0/413514116 >> 172.21.15.31:0/413514116 conn(0x55ced7850fd0 msgr2=0x55ced7856300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 -- 172.21.15.31:0/413514116 shutdown_connections 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.490+0000 7f7171d51b80 1 -- 172.21.15.31:0/413514116 wait complete. 2023-07-29T16:16:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.491+0000 7f7171d51b80 1 -- start start 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.491+0000 7f7171d51b80 1 Processor -- start 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7864ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced7865420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced786c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced77494b0 con 0x55ced76b76b0 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced7749150 con 0x55ced785ccf0 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced76308c0 con 0x55ced785bdf0 2023-07-29T16:16:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7171d51b80 10 client.0 _ll_register_callbacks cb 0x55ced77525c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced7865420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced7865420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35278/0 (socket says 172.21.15.31:35278) 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7157fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7864ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 -- 172.21.15.31:0/2307144203 learned_addr learned my addr 172.21.15.31:0/2307144203 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f715c99c700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced786c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 -- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 msgr2=0x55ced786c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced786c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 -- 172.21.15.31:0/2307144203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 msgr2=0x55ced7864ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7864ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f71577fe700 1 -- 172.21.15.31:0/2307144203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c009960 con 0x55ced785bdf0 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f715c99c700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced786c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.492+0000 7f7157fff700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7864ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f71577fe700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced7865420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f714c003d10 tx=0x7f714c005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5f4f674d30b44a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f71377fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:16:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f71377fe700 1 -- 172.21.15.31:0/2307144203 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c005040 con 0x55ced785bdf0 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ced77d7cb0 con 0x55ced785bdf0 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f71377fe700 1 -- 172.21.15.31:0/2307144203 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f714c0051a0 con 0x55ced785bdf0 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f71377fe700 1 -- 172.21.15.31:0/2307144203 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f714c005380 con 0x55ced785bdf0 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f7171d51b80 10 client.26116 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.493+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ced75c3c00 con 0x55ced785bdf0 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f71377fe700 1 -- 172.21.15.31:0/2307144203 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ced77d7cb0 con 0x55ced785bdf0 2023-07-29T16:16:57.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f71377fe700 1 -- 172.21.15.31:0/2307144203 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f714c021040 con 0x55ced785bdf0 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7171d51b80 10 client.26116 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7171d51b80 10 client.26116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7171d51b80 20 client.26116 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7171d51b80 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced7870b90 0x55ced7872fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7171d51b80 4 client.26116 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ced782ad10 con 0x55ced7870b90 2023-07-29T16:16:57.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.494+0000 7f7157fff700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced7870b90 0x55ced7872fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.495+0000 7f7157fff700 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced7870b90 0x55ced7872fa0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff633e381b8abdef server_cookie=9d9eabaac1b88da7 in_seq=0 out_seq=0 2023-07-29T16:16:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.495+0000 7f71377fe700 10 client.26116 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f71377fe700 1 -- 172.21.15.31:0/2307144203 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ced782ad10 con 0x55ced7870b90 2023-07-29T16:16:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f71377fe700 10 client.26116 handle_command_reply: tid=0 2023-07-29T16:16:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 client.26116 shutdown 2023-07-29T16:16:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 msgr2=0x55ced7865420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced7865420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f714c003d10 tx=0x7f714c005d90 comp rx=0 tx=0).stop 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 shutdown_connections 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced7870b90 0x55ced7872fa0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced786c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7864ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 --2- 172.21.15.31:0/2307144203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced7865420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f71377fe700 0 client.26116 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 >> 172.21.15.31:0/2307144203 conn(0x55ced7850fd0 msgr2=0x55ced7855a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 shutdown_connections 2023-07-29T16:16:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 1 -- 172.21.15.31:0/2307144203 wait complete. 2023-07-29T16:16:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.496+0000 7f7171d51b80 20 client.26116 trim_cache size 0 max 16384 2023-07-29T16:16:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.512+0000 7f7171d51b80 1 Processor -- start 2023-07-29T16:16:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.512+0000 7f7171d51b80 1 -- start start 2023-07-29T16:16:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7872f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced786a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced785d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7171d51b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced785d600 con 0x55ced76b76b0 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7171d51b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced78509d0 con 0x55ced785ccf0 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7171d51b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced7857df0 con 0x55ced785bdf0 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7157fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7872f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7157fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7872f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54900/0 (socket says 172.21.15.31:54900) 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f7157fff700 1 -- 172.21.15.31:0/600744883 learned_addr learned my addr 172.21.15.31:0/600744883 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f71577fe700 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced786a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.513+0000 7f715c99c700 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced785d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f7157fff700 1 -- 172.21.15.31:0/600744883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 msgr2=0x55ced786a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f7157fff700 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced786a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f7157fff700 1 -- 172.21.15.31:0/600744883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 msgr2=0x55ced785d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f7157fff700 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced785d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f7157fff700 1 -- 172.21.15.31:0/600744883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ced76011f0 con 0x55ced76b76b0 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f7157fff700 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7872f70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f71480ca6a0 tx=0x7f71480660d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de8cf3787560f58 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f713effd700 1 -- 172.21.15.31:0/600744883 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714809a390 con 0x55ced76b76b0 2023-07-29T16:16:57.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f713effd700 1 -- 172.21.15.31:0/600744883 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f714809e3c0 con 0x55ced76b76b0 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.514+0000 7f713effd700 1 -- 172.21.15.31:0/600744883 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714809e0b0 con 0x55ced76b76b0 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 -- 172.21.15.31:0/600744883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 msgr2=0x55ced7872f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7872f70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f71480ca6a0 tx=0x7f71480660d0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 -- 172.21.15.31:0/600744883 shutdown_connections 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced785ccf0 0x55ced785d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced76b76b0 0x55ced7872f70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 --2- 172.21.15.31:0/600744883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785bdf0 0x55ced786a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 -- 172.21.15.31:0/600744883 >> 172.21.15.31:0/600744883 conn(0x55ced7850fd0 msgr2=0x55ced7863bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 -- 172.21.15.31:0/600744883 shutdown_connections 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 -- 172.21.15.31:0/600744883 wait complete. 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 -- start start 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.515+0000 7f7171d51b80 1 Processor -- start 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced786a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced7856410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced78c06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced78c0da0 con 0x55ced785bdf0 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced78c0f10 con 0x55ced76b76b0 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ced78c10d0 con 0x55ced785ccf0 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7171d51b80 10 client.0 _ll_register_callbacks cb 0x55ced77525c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f71577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced7856410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f71577fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced7856410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54908/0 (socket says 172.21.15.31:54908) 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f7157fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced786a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f71577fe700 1 -- 172.21.15.31:0/783878553 learned_addr learned my addr 172.21.15.31:0/783878553 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.516+0000 7f715c99c700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced78c06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71577fe700 1 -- 172.21.15.31:0/783878553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 msgr2=0x55ced78c06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71577fe700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced78c06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71577fe700 1 -- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 msgr2=0x55ced786a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71577fe700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced786a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71577fe700 1 -- 172.21.15.31:0/783878553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71581a4fd0 con 0x55ced785bdf0 2023-07-29T16:16:57.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f7157fff700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced786a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:16:57.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71577fe700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced7856410 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f714c011af0 tx=0x7f714c003c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad50e25a7d387ca1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:16:57.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71377fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:16:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71377fe700 1 -- 172.21.15.31:0/783878553 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c019d80 con 0x55ced785bdf0 2023-07-29T16:16:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f714c004380 con 0x55ced785bdf0 2023-07-29T16:16:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71377fe700 1 -- 172.21.15.31:0/783878553 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f714c019730 con 0x55ced785bdf0 2023-07-29T16:16:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f71377fe700 1 -- 172.21.15.31:0/783878553 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f714c003680 con 0x55ced785bdf0 2023-07-29T16:16:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f7171d51b80 10 client.16143 fetch_fsmap learned FSMap version 23 2023-07-29T16:16:57.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.517+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ced75d9250 con 0x55ced785bdf0 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f71377fe700 1 -- 172.21.15.31:0/783878553 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f714c004380 con 0x55ced785bdf0 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f71377fe700 1 -- 172.21.15.31:0/783878553 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f714c011790 con 0x55ced785bdf0 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7171d51b80 10 client.16143 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7171d51b80 10 client.16143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7171d51b80 20 client.16143 populate_metadata read hostname 'smithi031' 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7171d51b80 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced78c2510 0x55ced78c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:16:57.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7171d51b80 4 client.16143 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ced77a5ce0 con 0x55ced78c2510 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.518+0000 7f7157fff700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced78c2510 0x55ced78c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.519+0000 7f7157fff700 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced78c2510 0x55ced78c4920 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5393f2bade39465 server_cookie=57af735da0c19211 in_seq=0 out_seq=0 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.519+0000 7f71377fe700 10 client.16143 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f71377fe700 1 -- 172.21.15.31:0/783878553 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ced77a5ce0 con 0x55ced78c2510 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f71377fe700 10 client.16143 handle_command_reply: tid=0 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 client.16143 shutdown 2023-07-29T16:16:57.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 msgr2=0x55ced7856410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced7856410 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f714c011af0 tx=0x7f714c003c40 comp rx=0 tx=0).stop 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 shutdown_connections 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ced78c2510 0x55ced78c4920 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ced785ccf0 0x55ced78c06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ced76b76b0 0x55ced786a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 --2- 172.21.15.31:0/783878553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ced785bdf0 0x55ced7856410 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f71377fe700 0 client.16143 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:16:57.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 >> 172.21.15.31:0/783878553 conn(0x55ced7850fd0 msgr2=0x55ced78630f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 shutdown_connections 2023-07-29T16:16:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 1 -- 172.21.15.31:0/783878553 wait complete. 2023-07-29T16:16:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.523+0000 7f7171d51b80 20 client.16143 trim_cache size 0 max 16384 2023-07-29T16:16:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:16:57.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:16:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:16:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:16:57.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71400a0790 msgr2=0x7f71400a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71400a0790 0x7f71400a2c40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7150009fd0 tx=0x7f715000b040 comp rx=0 tx=0).stop 2023-07-29T16:16:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 msgr2=0x7f715819c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:16:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 0x7f715819c7e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7148005720 tx=0x7f7148009580 comp rx=0 tx=0).stop 2023-07-29T16:16:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 shutdown_connections 2023-07-29T16:16:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f71400a0790 0x7f71400a2c40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f715810c910 0x7f715819dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71580737b0 0x7f715819c7e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 --2- 172.21.15.31:0/2347557535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7158074d90 0x7f715819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:16:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 >> 172.21.15.31:0/2347557535 conn(0x7f71580fb920 msgr2=0x7f71580fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:16:57.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 shutdown_connections 2023-07-29T16:16:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:16:57.524+0000 7f715e3ff700 1 -- 172.21.15.31:0/2347557535 wait complete. 2023-07-29T16:16:58.071 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:16:57 smithi156 ceph-mon[129245]: pgmap v414: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:58.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:16:57 smithi031 ceph-mon[95504]: pgmap v414: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:16:58.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:16:57 smithi039 ceph-mon[128858]: pgmap v414: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:00.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:00 smithi031 ceph-mon[95504]: pgmap v415: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-29T16:17:00.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:00 smithi039 ceph-mon[128858]: pgmap v415: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-29T16:17:00.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:00 smithi156 ceph-mon[129245]: pgmap v415: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-29T16:17:02.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:02 smithi031 ceph-mon[95504]: pgmap v416: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:02 smithi039 ceph-mon[128858]: pgmap v416: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:02 smithi156 ceph-mon[129245]: pgmap v416: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:04.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:04 smithi031 ceph-mon[95504]: pgmap v417: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:04 smithi039 ceph-mon[128858]: pgmap v417: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:04 smithi156 ceph-mon[129245]: pgmap v417: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:06.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:06 smithi031 ceph-mon[95504]: pgmap v418: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:06 smithi039 ceph-mon[128858]: pgmap v418: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:06 smithi156 ceph-mon[129245]: pgmap v418: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:07 smithi031 ceph-mon[95504]: pgmap v419: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:07 smithi039 ceph-mon[128858]: pgmap v419: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:07 smithi156 ceph-mon[129245]: pgmap v419: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1446955236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0101530 msgr2=0x7fbbc0101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1446955236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0101530 0x7fbbc0101980 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009990 tx=0x7fbbb4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1446955236 shutdown_connections 2023-07-29T16:17:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1446955236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc0101ec0 0x7fbbc0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1446955236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0100340 0x7fbbc0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1446955236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0101530 0x7fbbc0101980 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1446955236 >> 172.21.15.31:0/1446955236 conn(0x7fbbc00fb920 msgr2=0x7fbbc00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1446955236 shutdown_connections 2023-07-29T16:17:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.680+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1446955236 wait complete. 2023-07-29T16:17:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 Processor -- start 2023-07-29T16:17:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 -- start start 2023-07-29T16:17:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 0x7fbbc01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0101530 0x7fbbc0198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc0101ec0 0x7fbbc0199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0199df0 con 0x7fbbc0100340 2023-07-29T16:17:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0199f60 con 0x7fbbc0101ec0 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.681+0000 7fbbc5dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc019a120 con 0x7fbbc0101530 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 0x7fbbc01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 0x7fbbc01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42698/0 (socket says 172.21.15.31:42698) 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1692989568 learned_addr learned my addr 172.21.15.31:0/1692989568 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbffff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc0101ec0 0x7fbbc0199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0101530 0x7fbbc0198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1692989568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0101530 msgr2=0x7fbbc0198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0101530 0x7fbbc0198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1692989568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc0101ec0 msgr2=0x7fbbc0199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc0101ec0 0x7fbbc0199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1692989568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009c40 con 0x7fbbc0100340 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0101530 0x7fbbc0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.682+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 0x7fbbc01983d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000e950 tx=0x7fbbb000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.683+0000 7fbbbcff9700 1 -- 172.21.15.31:0/1692989568 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0009770 con 0x7fbbc0100340 2023-07-29T16:17:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.683+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb40095f0 con 0x7fbbc0100340 2023-07-29T16:17:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.683+0000 7fbbbcff9700 1 -- 172.21.15.31:0/1692989568 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbb000c7c0 con 0x7fbbc0100340 2023-07-29T16:17:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.683+0000 7fbbbcff9700 1 -- 172.21.15.31:0/1692989568 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0003c30 con 0x7fbbc0100340 2023-07-29T16:17:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.683+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc01a09e0 con 0x7fbbc0100340 2023-07-29T16:17:10.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:10 smithi031 ceph-mon[95504]: pgmap v420: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.686+0000 7fbbbcff9700 1 -- 172.21.15.31:0/1692989568 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbbb000c930 con 0x7fbbc0100340 2023-07-29T16:17:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.688+0000 7fbbd967fb80 1 Processor -- start 2023-07-29T16:17:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.688+0000 7fbbd967fb80 1 -- start start 2023-07-29T16:17:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245a9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624244ff9b0 con 0x5624244036b0 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624244fd370 con 0x5624245a7df0 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624244f8c60 con 0x5624245a8cf0 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbcff9700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbba80a0810 0x7fbba80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbeffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbeffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245a9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50020/0 (socket says 172.21.15.31:50020) 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3990650726 learned_addr learned my addr 172.21.15.31:0/3990650726 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:10.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbffff700 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbcff9700 1 -- 172.21.15.31:0/1692989568 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbbb0014070 con 0x7fbbc0100340 2023-07-29T16:17:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.689+0000 7fbbbffff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbba80a0810 0x7fbba80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3990650726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 msgr2=0x5624245a87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245a87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3990650726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 msgr2=0x5624245a78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3990650726 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562424326d80 con 0x5624245a8cf0 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245a9160 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4015000 tx=0x7fbbb4005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b73f86ca7a595f31 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbbbffff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbba80a0810 0x7fbba80a2cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009fd0 tx=0x7fbbb800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbba5ffb700 1 -- 172.21.15.31:0/3990650726 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb401d050 con 0x5624245a8cf0 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbba5ffb700 1 -- 172.21.15.31:0/3990650726 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbb401e040 con 0x5624245a8cf0 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.690+0000 7fbba5ffb700 1 -- 172.21.15.31:0/3990650726 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40055b0 con 0x5624245a8cf0 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3990650726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 msgr2=0x5624245a9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245a9160 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4015000 tx=0x7fbbb4005ea0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3990650726 shutdown_connections 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245a9160 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3990650726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245a87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3990650726 >> 172.21.15.31:0/3990650726 conn(0x56242459cfd0 msgr2=0x5624245a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3990650726 shutdown_connections 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.691+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3990650726 wait complete. 2023-07-29T16:17:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 -- start start 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 Processor -- start 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624244954b0 con 0x5624244036b0 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562424495150 con 0x5624245a8cf0 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56242437c8c0 con 0x5624245a7df0 2023-07-29T16:17:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.692+0000 7fbbd967fb80 10 client.0 _ll_register_callbacks cb 0x56242449e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42716/0 (socket says 172.21.15.31:42716) 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50028/0 (socket says 172.21.15.31:50028) 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/3311915185 learned_addr learned my addr 172.21.15.31:0/3311915185 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbffff700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 msgr2=0x5624245b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3311915185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 msgr2=0x5624245b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 -- 172.21.15.31:0/3311915185 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc01090b0 con 0x5624245a7df0 2023-07-29T16:17:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:10.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b13b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009990 tx=0x7fbbb4004ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2963e001127ff10b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:10.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.693+0000 7fbb9effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:10.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbb9effd700 1 -- 172.21.15.31:0/3311915185 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4004bf0 con 0x5624245a7df0 2023-07-29T16:17:10.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562424523cb0 con 0x5624245a7df0 2023-07-29T16:17:10.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbb9effd700 1 -- 172.21.15.31:0/3311915185 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbbb40058e0 con 0x5624245a7df0 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbb9effd700 1 -- 172.21.15.31:0/3311915185 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbb400f460 con 0x5624245a7df0 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbbd967fb80 10 client.26134 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56242430fc00 con 0x5624245a7df0 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.694+0000 7fbb9effd700 1 -- 172.21.15.31:0/3311915185 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562424523cb0 con 0x5624245a7df0 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbb9effd700 1 -- 172.21.15.31:0/3311915185 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbbb4023580 con 0x5624245a7df0 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbd967fb80 10 client.26134 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:10.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbd967fb80 10 client.26134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbd967fb80 20 client.26134 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624245bc710 0x5624245beaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbd967fb80 4 client.26134 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562424576d10 con 0x5624245bc710 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624245bc710 0x5624245beaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624245bc710 0x5624245beaf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd190d4efb13edc server_cookie=c2dfbaf0d15dd7 in_seq=0 out_seq=0 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.695+0000 7fbb9effd700 10 client.26134 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:10.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.696+0000 7fbb9effd700 1 -- 172.21.15.31:0/3311915185 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562424576d10 con 0x5624245bc710 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.696+0000 7fbb9effd700 10 client.26134 handle_command_reply: tid=0 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.696+0000 7fbbd967fb80 1 client.26134 shutdown 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 msgr2=0x5624245b13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b13b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009990 tx=0x7fbbb4004ec0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 shutdown_connections 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624245bc710 0x5624245beaf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/3311915185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b13b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbb9effd700 0 client.26134 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 >> 172.21.15.31:0/3311915185 conn(0x56242459cfd0 msgr2=0x5624245a1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 shutdown_connections 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 1 -- 172.21.15.31:0/3311915185 wait complete. 2023-07-29T16:17:10.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.697+0000 7fbbd967fb80 20 client.26134 trim_cache size 0 max 16384 2023-07-29T16:17:10.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 Processor -- start 2023-07-29T16:17:10.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 -- start start 2023-07-29T16:17:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624245b0a10 con 0x5624244036b0 2023-07-29T16:17:10.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624245a31b0 con 0x5624245a8cf0 2023-07-29T16:17:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624245a9600 con 0x5624245a7df0 2023-07-29T16:17:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42724/0 (socket says 172.21.15.31:42724) 2023-07-29T16:17:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/285378103 learned_addr learned my addr 172.21.15.31:0/285378103 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.712+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbffff700 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/285378103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 msgr2=0x5624245b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/285378103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 msgr2=0x5624245a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/285378103 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56242434d1f0 con 0x5624244036b0 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a33a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0005bf0 tx=0x7fbbb001cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9a7206f701ee69 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbba67fc700 1 -- 172.21.15.31:0/285378103 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00994f0 con 0x5624244036b0 2023-07-29T16:17:10.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbba67fc700 1 -- 172.21.15.31:0/285378103 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbb009d520 con 0x5624244036b0 2023-07-29T16:17:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.713+0000 7fbba67fc700 1 -- 172.21.15.31:0/285378103 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0097990 con 0x5624244036b0 2023-07-29T16:17:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 -- 172.21.15.31:0/285378103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 msgr2=0x5624245a33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a33a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0005bf0 tx=0x7fbbb001cf90 comp rx=0 tx=0).stop 2023-07-29T16:17:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 -- 172.21.15.31:0/285378103 shutdown_connections 2023-07-29T16:17:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a8cf0 0x5624245a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245a33a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/285378103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a7df0 0x5624245b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 -- 172.21.15.31:0/285378103 >> 172.21.15.31:0/285378103 conn(0x56242459cfd0 msgr2=0x5624245afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 -- 172.21.15.31:0/285378103 shutdown_connections 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 -- 172.21.15.31:0/285378103 wait complete. 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.714+0000 7fbbd967fb80 1 -- start start 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 Processor -- start 2023-07-29T16:17:10.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624245b5c40 con 0x5624244036b0 2023-07-29T16:17:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624245b5db0 con 0x5624245a7df0 2023-07-29T16:17:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624245b5f70 con 0x5624245a8cf0 2023-07-29T16:17:10.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbd967fb80 10 client.0 _ll_register_callbacks cb 0x56242449e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbbf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42740/0 (socket says 172.21.15.31:42740) 2023-07-29T16:17:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.715+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1909970455 learned_addr learned my addr 172.21.15.31:0/1909970455 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbffff700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1909970455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 msgr2=0x5624245b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 msgr2=0x5624245b46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245b46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbf7fe700 1 -- 172.21.15.31:0/1909970455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc01a09e0 con 0x5624244036b0 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbbf7fe700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b41b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0005950 tx=0x7fbbb000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58f3edfe897e63d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbb9effd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbb9effd700 1 -- 172.21.15.31:0/1909970455 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0097970 con 0x5624244036b0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbb401e040 con 0x5624244036b0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.716+0000 7fbb9effd700 1 -- 172.21.15.31:0/1909970455 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbbb00a1940 con 0x5624244036b0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbb9effd700 1 -- 172.21.15.31:0/1909970455 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbb00994f0 con 0x5624244036b0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 10 client.16161 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562424325250 con 0x5624244036b0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbb9effd700 1 -- 172.21.15.31:0/1909970455 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbb401e040 con 0x5624244036b0 2023-07-29T16:17:10.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbb9effd700 1 -- 172.21.15.31:0/1909970455 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbbb00989e0 con 0x5624244036b0 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 10 client.16161 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 10 client.16161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 20 client.16161 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56242460eed0 0x5624246112e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 4 client.16161 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5624244f1ce0 con 0x56242460eed0 2023-07-29T16:17:10.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.717+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56242460eed0 0x5624246112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.718+0000 7fbb9effd700 10 client.16161 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:10.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.718+0000 7fbbbeffd700 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56242460eed0 0x5624246112e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5259bb7038dc1b17 server_cookie=62c188ef15058fa1 in_seq=0 out_seq=0 2023-07-29T16:17:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbb9effd700 1 -- 172.21.15.31:0/1909970455 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5624244f1ce0 con 0x56242460eed0 2023-07-29T16:17:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbb9effd700 10 client.16161 handle_command_reply: tid=0 2023-07-29T16:17:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 client.16161 shutdown 2023-07-29T16:17:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 msgr2=0x5624245b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b41b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0005950 tx=0x7fbbb000cf10 comp rx=0 tx=0).stop 2023-07-29T16:17:10.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 shutdown_connections 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56242460eed0 0x5624246112e0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624245a8cf0 0x5624245b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624244036b0 0x5624245b41b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 --2- 172.21.15.31:0/1909970455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624245a7df0 0x5624245b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbb9effd700 0 client.16161 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 >> 172.21.15.31:0/1909970455 conn(0x56242459cfd0 msgr2=0x5624245abb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:10.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 shutdown_connections 2023-07-29T16:17:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.722+0000 7fbbd967fb80 1 -- 172.21.15.31:0/1909970455 wait complete. 2023-07-29T16:17:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbd967fb80 20 client.16161 trim_cache size 0 max 16384 2023-07-29T16:17:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:17:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:17:10.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:17:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:17:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbba80a0810 msgr2=0x7fbba80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbba80a0810 0x7fbba80a2cc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009fd0 tx=0x7fbbb800b040 comp rx=0 tx=0).stop 2023-07-29T16:17:10.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 msgr2=0x7fbbc01983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 0x7fbbc01983d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000e950 tx=0x7fbbb000ec60 comp rx=0 tx=0).stop 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 shutdown_connections 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbba80a0810 0x7fbba80a2cc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbc0101ec0 0x7fbbc0199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbbc0100340 0x7fbbc01983d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 --2- 172.21.15.31:0/1692989568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbbc0101530 0x7fbbc0198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:10.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 >> 172.21.15.31:0/1692989568 conn(0x7fbbc00fb920 msgr2=0x7fbbc00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 shutdown_connections 2023-07-29T16:17:10.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:10.723+0000 7fbbc5dff700 1 -- 172.21.15.31:0/1692989568 wait complete. 2023-07-29T16:17:10.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:10 smithi039 ceph-mon[128858]: pgmap v420: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:10 smithi156 ceph-mon[129245]: pgmap v420: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:12.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:12 smithi031 ceph-mon[95504]: pgmap v421: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-07-29T16:17:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:12 smithi039 ceph-mon[128858]: pgmap v421: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-07-29T16:17:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:12 smithi156 ceph-mon[129245]: pgmap v421: 97 pgs: 97 active+clean; 669 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-07-29T16:17:14.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:14 smithi031 ceph-mon[95504]: pgmap v422: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:14.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:14 smithi039 ceph-mon[128858]: pgmap v422: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:14 smithi156 ceph-mon[129245]: pgmap v422: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:16.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:16 smithi031 ceph-mon[95504]: pgmap v423: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:16 smithi039 ceph-mon[128858]: pgmap v423: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:16 smithi156 ceph-mon[129245]: pgmap v423: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:18.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:17 smithi031 ceph-mon[95504]: pgmap v424: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:18.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:17 smithi039 ceph-mon[128858]: pgmap v424: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:18.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:17 smithi156 ceph-mon[129245]: pgmap v424: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:20.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:20 smithi031 ceph-mon[95504]: pgmap v425: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:20.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:20 smithi039 ceph-mon[128858]: pgmap v425: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:20.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:20 smithi156 ceph-mon[129245]: pgmap v425: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:21.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:17:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 -- 172.21.15.31:0/3951112228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd90074d90 msgr2=0x7fcd900731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 --2- 172.21.15.31:0/3951112228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd90074d90 0x7fcd900731e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fcd80009990 tx=0x7fcd80009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 -- 172.21.15.31:0/3951112228 shutdown_connections 2023-07-29T16:17:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 --2- 172.21.15.31:0/3951112228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd901081f0 0x7fcd9010a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 --2- 172.21.15.31:0/3951112228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd90074d90 0x7fcd900731e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 --2- 172.21.15.31:0/3951112228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd900737b0 0x7fcd90073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 -- 172.21.15.31:0/3951112228 >> 172.21.15.31:0/3951112228 conn(0x7fcd900fb970 msgr2=0x7fcd900fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 -- 172.21.15.31:0/3951112228 shutdown_connections 2023-07-29T16:17:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.161+0000 7fcd965ff700 1 -- 172.21.15.31:0/3951112228 wait complete. 2023-07-29T16:17:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.162+0000 7fcd965ff700 1 Processor -- start 2023-07-29T16:17:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.162+0000 7fcd965ff700 1 -- start start 2023-07-29T16:17:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.162+0000 7fcd965ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 0x7fcd901982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.162+0000 7fcd965ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd90074d90 0x7fcd90198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd965ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd901081f0 0x7fcd9019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd965ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9019fe70 con 0x7fcd900737b0 2023-07-29T16:17:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd965ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9019ffb0 con 0x7fcd901081f0 2023-07-29T16:17:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd965ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd901a0140 con 0x7fcd90074d90 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 0x7fcd901982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 0x7fcd901982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42058/0 (socket says 172.21.15.31:42058) 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 -- 172.21.15.31:0/1192851890 learned_addr learned my addr 172.21.15.31:0/1192851890 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd90074d90 0x7fcd90198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd94b9c700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd901081f0 0x7fcd9019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 -- 172.21.15.31:0/1192851890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd90074d90 msgr2=0x7fcd90198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd90074d90 0x7fcd90198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 -- 172.21.15.31:0/1192851890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd901081f0 msgr2=0x7fcd9019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd901081f0 0x7fcd9019f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.163+0000 7fcd8ffff700 1 -- 172.21.15.31:0/1192851890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd800095f0 con 0x7fcd900737b0 2023-07-29T16:17:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.164+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 0x7fcd901982f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcd80004710 tx=0x7fcd80009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.164+0000 7fcd8d7fa700 1 -- 172.21.15.31:0/1192851890 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8001d070 con 0x7fcd900737b0 2023-07-29T16:17:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.164+0000 7fcd8d7fa700 1 -- 172.21.15.31:0/1192851890 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd80021410 con 0x7fcd900737b0 2023-07-29T16:17:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.164+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd901a03c0 con 0x7fcd900737b0 2023-07-29T16:17:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.164+0000 7fcd8d7fa700 1 -- 172.21.15.31:0/1192851890 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80017650 con 0x7fcd900737b0 2023-07-29T16:17:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.164+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd901a0840 con 0x7fcd900737b0 2023-07-29T16:17:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.167+0000 7fcd8d7fa700 1 -- 172.21.15.31:0/1192851890 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fcd80021580 con 0x7fcd900737b0 2023-07-29T16:17:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 Processor -- start 2023-07-29T16:17:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcd8d7fa700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd780a0810 0x7fcd780a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 -- start start 2023-07-29T16:17:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcd8d7fa700 1 -- 172.21.15.31:0/1192851890 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcd800cce50 con 0x7fcd900737b0 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcd94b9c700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd780a0810 0x7fcd780a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dca5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dca6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca72f0 0x56098dca7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dbfda10 con 0x56098dca5be0 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dbfb3d0 con 0x56098dca6530 2023-07-29T16:17:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.169+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dbf6cc0 con 0x56098dca72f0 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dca5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dca5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42072/0 (socket says 172.21.15.31:42072) 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2404547622 learned_addr learned my addr 172.21.15.31:0/2404547622 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dca6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd94b9c700 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca72f0 0x56098dca7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd94b9c700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd780a0810 0x7fcd780a2cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800f590 tx=0x7fcd8800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2404547622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca72f0 msgr2=0x56098dca7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca72f0 0x56098dca7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2404547622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 msgr2=0x56098dca6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dca6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2404547622 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56098da24de0 con 0x56098dca5be0 2023-07-29T16:17:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dca6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.170+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dca5ff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcd8009a510 tx=0x7fcd8009a0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33480cd73166730c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcd767fc700 1 -- 172.21.15.31:0/2404547622 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8009e5b0 con 0x56098dca5be0 2023-07-29T16:17:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcd767fc700 1 -- 172.21.15.31:0/2404547622 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd800a2940 con 0x56098dca5be0 2023-07-29T16:17:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcd767fc700 1 -- 172.21.15.31:0/2404547622 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8009aa40 con 0x56098dca5be0 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2404547622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 msgr2=0x56098dca5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dca5ff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcd8009a510 tx=0x7fcd8009a0c0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2404547622 shutdown_connections 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca72f0 0x56098dca7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dca5ff0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2404547622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dca6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2404547622 >> 172.21.15.31:0/2404547622 conn(0x56098dc9b3e0 msgr2=0x56098dca0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2404547622 shutdown_connections 2023-07-29T16:17:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.171+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2404547622 wait complete. 2023-07-29T16:17:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.172+0000 7fcda9ff1b80 1 -- start start 2023-07-29T16:17:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.172+0000 7fcda9ff1b80 1 Processor -- start 2023-07-29T16:17:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca5be0 0x56098dcaf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dcaf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca72f0 0x56098dcb6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098db93510 con 0x56098dca72f0 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098db931b0 con 0x56098dca6530 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098da7a920 con 0x56098dca5be0 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcda9ff1b80 10 client.0 _ll_register_callbacks cb 0x56098db9c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd94b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca72f0 0x56098dcb6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dcaf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dcaf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58070/0 (socket says 172.21.15.31:58070) 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd94b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca72f0 0x56098dcb6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42082/0 (socket says 172.21.15.31:42082) 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 -- 172.21.15.31:0/430869451 learned_addr learned my addr 172.21.15.31:0/430869451 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca5be0 0x56098dcaf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 -- 172.21.15.31:0/430869451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca5be0 msgr2=0x56098dcaf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca5be0 0x56098dcaf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 -- 172.21.15.31:0/430869451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca72f0 msgr2=0x56098dcb6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca72f0 0x56098dcb6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.173+0000 7fcd8f7fe700 1 -- 172.21.15.31:0/430869451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd840098e0 con 0x56098dca6530 2023-07-29T16:17:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dcaf9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcd84012960 tx=0x7fcd84012c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=716f90fce1731cbc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcd6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/430869451 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84017070 con 0x56098dca6530 2023-07-29T16:17:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56098dc21d10 con 0x56098dca6530 2023-07-29T16:17:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/430869451 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcd8400f590 con 0x56098dca6530 2023-07-29T16:17:22.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/430869451 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd8400f750 con 0x56098dca6530 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcda9ff1b80 10 client.26147 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.174+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fcd901a0840 con 0x56098dca6530 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/430869451 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56098dc21d10 con 0x56098dca6530 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/430869451 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fcd840269b0 con 0x56098dca6530 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcda9ff1b80 10 client.26147 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcda9ff1b80 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcda9ff1b80 20 client.26147 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:22.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dcbb2a0 0x56098dcbd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcda9ff1b80 4 client.26147 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56098dc74d70 con 0x56098dcbb2a0 2023-07-29T16:17:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.175+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dcbb2a0 0x56098dcbd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.176+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dcbb2a0 0x56098dcbd6b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59f881d62c47ae64 server_cookie=fd50b371052a87a3 in_seq=0 out_seq=0 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.176+0000 7fcd6f7fe700 10 client.26147 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.176+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/430869451 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56098dc74d70 con 0x56098dcbb2a0 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.176+0000 7fcd6f7fe700 10 client.26147 handle_command_reply: tid=0 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 client.26147 shutdown 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 msgr2=0x56098dcaf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dcaf9c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fcd84012960 tx=0x7fcd84012c70 comp rx=0 tx=0).stop 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 shutdown_connections 2023-07-29T16:17:22.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dcbb2a0 0x56098dcbd6b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca72f0 0x56098dcb6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca5be0 0x56098dcaf480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/430869451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca6530 0x56098dcaf9c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 >> 172.21.15.31:0/430869451 conn(0x56098dc9b3e0 msgr2=0x56098dca0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcd6f7fe700 0 client.26147 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 shutdown_connections 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/430869451 wait complete. 2023-07-29T16:17:22.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.177+0000 7fcda9ff1b80 20 client.26147 trim_cache size 0 max 16384 2023-07-29T16:17:22.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 Processor -- start 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 -- start start 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcaa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dcaa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dcae930 con 0x56098dca5be0 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dcaeff0 con 0x56098dca72f0 2023-07-29T16:17:22.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dc9d460 con 0x56098dca6530 2023-07-29T16:17:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42094/0 (socket says 172.21.15.31:42094) 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcd8ffff700 1 -- 172.21.15.31:0/3369027045 learned_addr learned my addr 172.21.15.31:0/3369027045 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcaa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.195+0000 7fcd94b9c700 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dcaa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8ffff700 1 -- 172.21.15.31:0/3369027045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 msgr2=0x56098dcaa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcaa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8ffff700 1 -- 172.21.15.31:0/3369027045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 msgr2=0x56098dcaa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dcaa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8ffff700 1 -- 172.21.15.31:0/3369027045 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56098da0dc60 con 0x56098dca5be0 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcaa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb4910 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcd8000b2e0 tx=0x7fcd80009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57dfbf3ee5b77e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd76ffd700 1 -- 172.21.15.31:0/3369027045 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd80098920 con 0x56098dca5be0 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd76ffd700 1 -- 172.21.15.31:0/3369027045 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd8009e170 con 0x56098dca5be0 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.196+0000 7fcd76ffd700 1 -- 172.21.15.31:0/3369027045 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8009aa40 con 0x56098dca5be0 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/3369027045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 msgr2=0x56098dcb4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb4910 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcd8000b2e0 tx=0x7fcd80009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/3369027045 shutdown_connections 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dcaa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb4910 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/3369027045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcaa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/3369027045 >> 172.21.15.31:0/3369027045 conn(0x56098dc9b3e0 msgr2=0x56098dcaf480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/3369027045 shutdown_connections 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/3369027045 wait complete. 2023-07-29T16:17:22.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.197+0000 7fcda9ff1b80 1 -- start start 2023-07-29T16:17:22.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 Processor -- start 2023-07-29T16:17:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcb3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dd0ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dd0d360 con 0x56098dca5be0 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dd0d4a0 con 0x56098dca72f0 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56098dd0d5e0 con 0x56098dca6530 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcda9ff1b80 10 client.0 _ll_register_callbacks cb 0x56098db9c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcd8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42100/0 (socket says 172.21.15.31:42100) 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2693288206 learned_addr learned my addr 172.21.15.31:0/2693288206 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.198+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcb3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd94b9c700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dd0ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2693288206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 msgr2=0x56098dcb3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcb3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 msgr2=0x56098dd0ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dd0ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd8ffff700 1 -- 172.21.15.31:0/2693288206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd901a0840 con 0x56098dca5be0 2023-07-29T16:17:22.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd8ffff700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb3820 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcd8009a510 tx=0x7fcd800ca910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e91ee9ee35631c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/2693288206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8009a7e0 con 0x56098dca5be0 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/2693288206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcd80096040 con 0x56098dca5be0 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd800a2940 con 0x56098dca5be0 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/2693288206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd8009a940 con 0x56098dca5be0 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcda9ff1b80 10 client.16179 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:22.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.199+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56098da24de0 con 0x56098dca5be0 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/2693288206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd800a6080 con 0x56098dca5be0 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/2693288206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fcd800ca070 con 0x56098dca5be0 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcda9ff1b80 10 client.16179 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcda9ff1b80 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcda9ff1b80 20 client.16179 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dd0ebc0 0x56098dd10fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcda9ff1b80 4 client.16179 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:17:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56098dbefd40 con 0x56098dd0ebc0 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.200+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dd0ebc0 0x56098dd10fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcd6f7fe700 10 client.16179 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcd8f7fe700 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dd0ebc0 0x56098dd10fd0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e14b8797463b2fe1 server_cookie=6b19437be2d7abf2 in_seq=0 out_seq=0 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcd6f7fe700 1 -- 172.21.15.31:0/2693288206 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56098dbefd40 con 0x56098dd0ebc0 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcd6f7fe700 10 client.16179 handle_command_reply: tid=0 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 client.16179 shutdown 2023-07-29T16:17:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 msgr2=0x56098dcb3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb3820 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcd8009a510 tx=0x7fcd800ca910 comp rx=0 tx=0).stop 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 shutdown_connections 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56098dd0ebc0 0x56098dd10fd0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56098dca72f0 0x56098dd0ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56098dca5be0 0x56098dcb3820 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcd6f7fe700 0 client.16179 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 --2- 172.21.15.31:0/2693288206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56098dca6530 0x56098dcb3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 >> 172.21.15.31:0/2693288206 conn(0x56098dc9b3e0 msgr2=0x56098dcaf420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 shutdown_connections 2023-07-29T16:17:22.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 1 -- 172.21.15.31:0/2693288206 wait complete. 2023-07-29T16:17:22.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.201+0000 7fcda9ff1b80 20 client.16179 trim_cache size 0 max 16384 2023-07-29T16:17:22.209 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:17:22.209 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:17:22.210 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:17:22.210 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:17:22.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd780a0810 msgr2=0x7fcd780a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd780a0810 0x7fcd780a2cc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800f590 tx=0x7fcd8800b040 comp rx=0 tx=0).stop 2023-07-29T16:17:22.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 msgr2=0x7fcd901982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 0x7fcd901982f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcd80004710 tx=0x7fcd80009580 comp rx=0 tx=0).stop 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 shutdown_connections 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcd780a0810 0x7fcd780a2cc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd901081f0 0x7fcd9019f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd900737b0 0x7fcd901982f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 --2- 172.21.15.31:0/1192851890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd90074d90 0x7fcd90198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 >> 172.21.15.31:0/1192851890 conn(0x7fcd900fb970 msgr2=0x7fcd900fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 shutdown_connections 2023-07-29T16:17:22.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.202+0000 7fcd965ff700 1 -- 172.21.15.31:0/1192851890 wait complete. 2023-07-29T16:17:22.218 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:17:22.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:17:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 -- 172.21.15.31:0/3209997714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 msgr2=0x7f94e4108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3209997714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4108780 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00b0a0 tx=0x7f94dc00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 -- 172.21.15.31:0/3209997714 shutdown_connections 2023-07-29T16:17:22.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3209997714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4108780 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3209997714 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4100340 0x7f94e4100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3209997714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94e4101530 0x7f94e4101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.426+0000 7f94ebfff700 1 -- 172.21.15.31:0/3209997714 >> 172.21.15.31:0/3209997714 conn(0x7f94e40fb920 msgr2=0x7f94e40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.427+0000 7f94ebfff700 1 -- 172.21.15.31:0/3209997714 shutdown_connections 2023-07-29T16:17:22.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.427+0000 7f94ebfff700 1 -- 172.21.15.31:0/3209997714 wait complete. 2023-07-29T16:17:22.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 Processor -- start 2023-07-29T16:17:22.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 -- start start 2023-07-29T16:17:22.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94e4100340 0x7f94e4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4101530 0x7f94e4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e4073e20 con 0x7f94e4101ec0 2023-07-29T16:17:22.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e4073f90 con 0x7f94e4101530 2023-07-29T16:17:22.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ebfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e41a01d0 con 0x7f94e4100340 2023-07-29T16:17:22.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42130/0 (socket says 172.21.15.31:42130) 2023-07-29T16:17:22.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94ea59c700 1 -- 172.21.15.31:0/3432130467 learned_addr learned my addr 172.21.15.31:0/3432130467 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94e959a700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4101530 0x7f94e4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.428+0000 7f94e9d9b700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94e4100340 0x7f94e4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ea59c700 1 -- 172.21.15.31:0/3432130467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94e4100340 msgr2=0x7f94e4074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ea59c700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94e4100340 0x7f94e4074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ea59c700 1 -- 172.21.15.31:0/3432130467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4101530 msgr2=0x7f94e4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ea59c700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4101530 0x7f94e4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ea59c700 1 -- 172.21.15.31:0/3432130467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc009d00 con 0x7f94e4101ec0 2023-07-29T16:17:22.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ea59c700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4073680 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00d010 tx=0x7f94dc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc00e040 con 0x7f94e4101ec0 2023-07-29T16:17:22.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94dc003e10 con 0x7f94e4101ec0 2023-07-29T16:17:22.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e41a0450 con 0x7f94e4101ec0 2023-07-29T16:17:22.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.429+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0052a0 con 0x7f94e4101ec0 2023-07-29T16:17:22.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.430+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e41a0870 con 0x7f94e4101ec0 2023-07-29T16:17:22.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.431+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e41077d0 con 0x7f94e4101ec0 2023-07-29T16:17:22.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.433+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f94dc01d480 con 0x7f94e4101ec0 2023-07-29T16:17:22.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.434+0000 7f94daffd700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94d00a0860 0x7f94d00a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.434+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f94dc0cd3a0 con 0x7f94e4101ec0 2023-07-29T16:17:22.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.434+0000 7f94e9d9b700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94d00a0860 0x7f94d00a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.435+0000 7f94e9d9b700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94d00a0860 0x7f94d00a2d10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f94e4101390 tx=0x7f94e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.437+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f94dc096e10 con 0x7f94e4101ec0 2023-07-29T16:17:22.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:22 smithi031 ceph-mon[95504]: pgmap v426: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:22.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.691+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94e402cc30 con 0x7f94e4101ec0 2023-07-29T16:17:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.691+0000 7f94daffd700 1 -- 172.21.15.31:0/3432130467 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f94dc09a430 con 0x7f94e4101ec0 2023-07-29T16:17:22.690 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:17:22.691 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:17:22.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94d00a0860 msgr2=0x7f94d00a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94d00a0860 0x7f94d00a2d10 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f94e4101390 tx=0x7f94e0009260 comp rx=0 tx=0).stop 2023-07-29T16:17:22.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 msgr2=0x7f94e4073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4073680 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00d010 tx=0x7f94dc00bc80 comp rx=0 tx=0).stop 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 shutdown_connections 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94d00a0860 0x7f94d00a2d10 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f94e4101ec0 0x7f94e4073680 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94e4100340 0x7f94e4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 --2- 172.21.15.31:0/3432130467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94e4101530 0x7f94e4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 >> 172.21.15.31:0/3432130467 conn(0x7f94e40fb920 msgr2=0x7f94e4104910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 shutdown_connections 2023-07-29T16:17:22.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.694+0000 7f94ebfff700 1 -- 172.21.15.31:0/3432130467 wait complete. 2023-07-29T16:17:22.695 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:17:22.712 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:17:22.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:22 smithi039 ceph-mon[128858]: pgmap v426: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:22.820 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:22 smithi156 ceph-mon[129245]: pgmap v426: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:22.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 -- 172.21.15.31:0/1999984673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084101eb0 msgr2=0x7ff084108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 --2- 172.21.15.31:0/1999984673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084101eb0 0x7ff084108770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff07c00b0a0 tx=0x7ff07c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 -- 172.21.15.31:0/1999984673 shutdown_connections 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 --2- 172.21.15.31:0/1999984673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084101eb0 0x7ff084108770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 --2- 172.21.15.31:0/1999984673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084100330 0x7ff084100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 --2- 172.21.15.31:0/1999984673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101520 0x7ff084101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 -- 172.21.15.31:0/1999984673 >> 172.21.15.31:0/1999984673 conn(0x7ff0840fb970 msgr2=0x7ff0840fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 -- 172.21.15.31:0/1999984673 shutdown_connections 2023-07-29T16:17:22.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.920+0000 7ff08adff700 1 -- 172.21.15.31:0/1999984673 wait complete. 2023-07-29T16:17:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.921+0000 7ff08adff700 1 Processor -- start 2023-07-29T16:17:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 -- start start 2023-07-29T16:17:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084100330 0x7ff084198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084101520 0x7ff084198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 0x7ff08419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08419fc80 con 0x7ff084101520 2023-07-29T16:17:22.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08419fdc0 con 0x7ff084101eb0 2023-07-29T16:17:22.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08adff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08419ff50 con 0x7ff084100330 2023-07-29T16:17:22.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff083fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084101520 0x7ff084198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08939c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 0x7ff08419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff088b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084100330 0x7ff084198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08939c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 0x7ff08419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58112/0 (socket says 172.21.15.31:58112) 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.922+0000 7ff08939c700 1 -- 172.21.15.31:0/1941941978 learned_addr learned my addr 172.21.15.31:0/1941941978 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08939c700 1 -- 172.21.15.31:0/1941941978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084100330 msgr2=0x7ff084198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08939c700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084100330 0x7ff084198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08939c700 1 -- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084101520 msgr2=0x7ff084198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08939c700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084101520 0x7ff084198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08939c700 1 -- 172.21.15.31:0/1941941978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c009d00 con 0x7ff084101eb0 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff083fff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084101520 0x7ff084198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08939c700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 0x7ff08419f6b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0062a0 tx=0x7ff07c00bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff081ffb700 1 -- 172.21.15.31:0/1941941978 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c00e040 con 0x7ff084101eb0 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff081ffb700 1 -- 172.21.15.31:0/1941941978 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff07c0094c0 con 0x7ff084101eb0 2023-07-29T16:17:22.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff081ffb700 1 -- 172.21.15.31:0/1941941978 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c0052a0 con 0x7ff084101eb0 2023-07-29T16:17:22.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0841a01d0 con 0x7ff084101eb0 2023-07-29T16:17:22.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.923+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0841a0600 con 0x7ff084101eb0 2023-07-29T16:17:22.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.927+0000 7ff081ffb700 1 -- 172.21.15.31:0/1941941978 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff07c01d470 con 0x7ff084101eb0 2023-07-29T16:17:22.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.928+0000 7ff09e67eb80 1 Processor -- start 2023-07-29T16:17:22.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.928+0000 7ff09e67eb80 1 -- start start 2023-07-29T16:17:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.928+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383cdff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.928+0000 7ff081ffb700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff0740a0790 0x7ff0740a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383ced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383cf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e38325a10 con 0x561e383ce530 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383233d0 con 0x561e383cf2f0 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e3831ecc0 con 0x561e383cdbe0 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff081ffb700 1 -- 172.21.15.31:0/1941941978 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff07c0cd0a0 con 0x7ff084101eb0 2023-07-29T16:17:22.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff088b9b700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff0740a0790 0x7ff0740a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383cdff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383cdff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56728/0 (socket says 172.21.15.31:56728) 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 -- 172.21.15.31:0/809311915 learned_addr learned my addr 172.21.15.31:0/809311915 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff08939c700 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 -- 172.21.15.31:0/809311915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 msgr2=0x561e383cdff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383cdff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 -- 172.21.15.31:0/809311915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 msgr2=0x561e383cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.929+0000 7ff083fff700 1 -- 172.21.15.31:0/809311915 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e3814cde0 con 0x561e383ce530 2023-07-29T16:17:22.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.930+0000 7ff08939c700 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:22.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.930+0000 7ff088b9b700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff0740a0790 0x7ff0740a2c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff084101380 tx=0x7ff078009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.930+0000 7ff083fff700 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383ced60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff070011ca0 tx=0x7ff070009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd081fc50c9eb2ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.930+0000 7ff06effd700 1 -- 172.21.15.31:0/809311915 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000f790 con 0x561e383ce530 2023-07-29T16:17:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.930+0000 7ff06effd700 1 -- 172.21.15.31:0/809311915 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff070011960 con 0x561e383ce530 2023-07-29T16:17:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.930+0000 7ff06effd700 1 -- 172.21.15.31:0/809311915 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07001ec60 con 0x561e383ce530 2023-07-29T16:17:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 -- 172.21.15.31:0/809311915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 msgr2=0x561e383ced60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383ced60 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff070011ca0 tx=0x7ff070009260 comp rx=0 tx=0).stop 2023-07-29T16:17:22.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 -- 172.21.15.31:0/809311915 shutdown_connections 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383cdff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/809311915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383ced60 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 -- 172.21.15.31:0/809311915 >> 172.21.15.31:0/809311915 conn(0x561e383c33e0 msgr2=0x561e383c8990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 -- 172.21.15.31:0/809311915 shutdown_connections 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 -- 172.21.15.31:0/809311915 wait complete. 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.931+0000 7ff09e67eb80 1 -- start start 2023-07-29T16:17:22.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 Processor -- start 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383d7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383d7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383dedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e382bb510 con 0x561e383ce530 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e382bb1b0 con 0x561e383cf2f0 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e381a2920 con 0x561e383cdbe0 2023-07-29T16:17:22.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff09e67eb80 10 client.0 _ll_register_callbacks cb 0x561e382c4620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff083fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383d7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff083fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383d7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42166/0 (socket says 172.21.15.31:42166) 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff083fff700 1 -- 172.21.15.31:0/3771458925 learned_addr learned my addr 172.21.15.31:0/3771458925 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.932+0000 7ff088b9b700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383d7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff08939c700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383dedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff083fff700 1 -- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 msgr2=0x561e383d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff083fff700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383d7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff083fff700 1 -- 172.21.15.31:0/3771458925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 msgr2=0x561e383dedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff083fff700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383dedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff083fff700 1 -- 172.21.15.31:0/3771458925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0841a0600 con 0x561e383ce530 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff08939c700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383dedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff083fff700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383d7db0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff070009790 tx=0x7ff07001ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8fe450176faf5f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff063fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff063fff700 1 -- 172.21.15.31:0/3771458925 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07001e600 con 0x561e383ce530 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e38349d10 con 0x561e383ce530 2023-07-29T16:17:22.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.933+0000 7ff063fff700 1 -- 172.21.15.31:0/3771458925 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff070000e40 con 0x561e383ce530 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff063fff700 1 -- 172.21.15.31:0/3771458925 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff07001e3a0 con 0x561e383ce530 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 10 client.16197 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7ff070019040 con 0x561e383ce530 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff063fff700 1 -- 172.21.15.31:0/3771458925 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff070036020 con 0x561e383ce530 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff063fff700 1 -- 172.21.15.31:0/3771458925 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff07001d070 con 0x561e383ce530 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 10 client.16197 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 10 client.16197 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:17:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 20 client.16197 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e383e34d0 0x561e383e58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 4 client.16197 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.934+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e3839cd70 con 0x561e383e34d0 2023-07-29T16:17:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.935+0000 7ff088b9b700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e383e34d0 0x561e383e58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff088b9b700 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e383e34d0 0x561e383e58e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=493c0629e28297dc server_cookie=60d83e84268149bd in_seq=0 out_seq=0 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff063fff700 10 client.16197 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff063fff700 1 -- 172.21.15.31:0/3771458925 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561e3839cd70 con 0x561e383e34d0 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff063fff700 10 client.16197 handle_command_reply: tid=0 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff09e67eb80 1 client.16197 shutdown 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 msgr2=0x561e383d7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.936+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383d7db0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff070009790 tx=0x7ff07001ee50 comp rx=0 tx=0).stop 2023-07-29T16:17:22.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 shutdown_connections 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e383e34d0 0x561e383e58e0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383dedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383d7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff063fff700 0 client.16197 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3771458925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383ce530 0x561e383d7db0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 >> 172.21.15.31:0/3771458925 conn(0x561e383c33e0 msgr2=0x561e383c8060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 shutdown_connections 2023-07-29T16:17:22.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3771458925 wait complete. 2023-07-29T16:17:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.937+0000 7ff09e67eb80 20 client.16197 trim_cache size 0 max 16384 2023-07-29T16:17:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 Processor -- start 2023-07-29T16:17:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 -- start start 2023-07-29T16:17:22.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 0x561e383e58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383ce530 0x561e383dcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383d6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383d84f0 con 0x561e383cdbe0 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383e5f40 con 0x561e383cf2f0 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.950+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383d5440 con 0x561e383ce530 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 0x561e383e58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 0x561e383e58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42172/0 (socket says 172.21.15.31:42172) 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 -- 172.21.15.31:0/2319869678 learned_addr learned my addr 172.21.15.31:0/2319869678 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff083fff700 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383ce530 0x561e383dcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff08939c700 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383d6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 -- 172.21.15.31:0/2319869678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383ce530 msgr2=0x561e383dcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383ce530 0x561e383dcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 -- 172.21.15.31:0/2319869678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 msgr2=0x561e383d6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383d6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 -- 172.21.15.31:0/2319869678 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e38135c60 con 0x561e383cdbe0 2023-07-29T16:17:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.951+0000 7ff088b9b700 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 0x561e383e58e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x561e383e61b0 tx=0x7ff078014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d90ae7f75225b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff06f7fe700 1 -- 172.21.15.31:0/2319869678 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078022040 con 0x561e383cdbe0 2023-07-29T16:17:22.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff06f7fe700 1 -- 172.21.15.31:0/2319869678 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff078028420 con 0x561e383cdbe0 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff06f7fe700 1 -- 172.21.15.31:0/2319869678 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07801d410 con 0x561e383cdbe0 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 -- 172.21.15.31:0/2319869678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 msgr2=0x561e383e58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 0x561e383e58e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x561e383e61b0 tx=0x7ff078014000 comp rx=0 tx=0).stop 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 -- 172.21.15.31:0/2319869678 shutdown_connections 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383cf2f0 0x561e383d6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cdbe0 0x561e383e58e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/2319869678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383ce530 0x561e383dcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 -- 172.21.15.31:0/2319869678 >> 172.21.15.31:0/2319869678 conn(0x561e383c33e0 msgr2=0x561e383d6510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.952+0000 7ff09e67eb80 1 -- 172.21.15.31:0/2319869678 shutdown_connections 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.953+0000 7ff09e67eb80 1 -- 172.21.15.31:0/2319869678 wait complete. 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.953+0000 7ff09e67eb80 1 -- start start 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.953+0000 7ff09e67eb80 1 Processor -- start 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383da9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 0x561e383daf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 0x561e383dbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383dc470 con 0x561e383cf2f0 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383dc5e0 con 0x561e383ce530 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e383dc7a0 con 0x561e383cdbe0 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff09e67eb80 10 client.0 _ll_register_callbacks cb 0x561e382c4620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff08939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 0x561e383dbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff083fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 0x561e383daf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff088b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383da9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff083fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 0x561e383daf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58166/0 (socket says 172.21.15.31:58166) 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff08939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 0x561e383dbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42178/0 (socket says 172.21.15.31:42178) 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.954+0000 7ff083fff700 1 -- 172.21.15.31:0/3382381839 learned_addr learned my addr 172.21.15.31:0/3382381839 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff083fff700 1 -- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 msgr2=0x561e383da9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff083fff700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383da9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff083fff700 1 -- 172.21.15.31:0/3382381839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 msgr2=0x561e383dbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff083fff700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 0x561e383dbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff083fff700 1 -- 172.21.15.31:0/3382381839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff078017070 con 0x561e383ce530 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff08939c700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 0x561e383dbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff088b9b700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383da9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff083fff700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 0x561e383daf20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff070035040 tx=0x7ff070027690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e480fed812df84 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff063fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff063fff700 1 -- 172.21.15.31:0/3382381839 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070011620 con 0x561e383ce530 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff063fff700 1 -- 172.21.15.31:0/3382381839 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff070000e40 con 0x561e383ce530 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.955+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff070011960 con 0x561e383ce530 2023-07-29T16:17:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff063fff700 1 -- 172.21.15.31:0/3382381839 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff07001e5b0 con 0x561e383ce530 2023-07-29T16:17:22.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 10 client.26177 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:22.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561e38173250 con 0x561e383ce530 2023-07-29T16:17:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff063fff700 1 -- 172.21.15.31:0/3382381839 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff070011960 con 0x561e383ce530 2023-07-29T16:17:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff063fff700 1 -- 172.21.15.31:0/3382381839 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff07000f3c0 con 0x561e383ce530 2023-07-29T16:17:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 10 client.26177 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 10 client.26177 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 20 client.26177 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e38435a00 0x561e38437e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 4 client.26177 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.956+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561e38317d40 con 0x561e38435a00 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff088b9b700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e38435a00 0x561e38437e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff088b9b700 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e38435a00 0x561e38437e10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=90ffff2ce9962a49 server_cookie=8ca204d38f4ffb6f in_seq=0 out_seq=0 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff063fff700 10 client.26177 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff063fff700 1 -- 172.21.15.31:0/3382381839 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561e38317d40 con 0x561e38435a00 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff063fff700 10 client.26177 handle_command_reply: tid=0 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff09e67eb80 1 client.26177 shutdown 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 msgr2=0x561e383daf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.957+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 0x561e383daf20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff070035040 tx=0x7ff070027690 comp rx=0 tx=0).stop 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 shutdown_connections 2023-07-29T16:17:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561e383cf2f0 0x561e383dbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561e383cdbe0 0x561e383da9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561e38435a00 0x561e38437e10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 --2- 172.21.15.31:0/3382381839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e383ce530 0x561e383daf20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff063fff700 0 client.26177 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 >> 172.21.15.31:0/3382381839 conn(0x561e383c33e0 msgr2=0x561e383d21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 shutdown_connections 2023-07-29T16:17:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 1 -- 172.21.15.31:0/3382381839 wait complete. 2023-07-29T16:17:22.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff09e67eb80 20 client.26177 trim_cache size 0 max 16384 2023-07-29T16:17:22.965 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:17:22.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff0740a0790 msgr2=0x7ff0740a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff08adff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff0740a0790 0x7ff0740a2c40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff084101380 tx=0x7ff078009260 comp rx=0 tx=0).stop 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 msgr2=0x7ff08419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.958+0000 7ff08adff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 0x7ff08419f6b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0062a0 tx=0x7ff07c00bd60 comp rx=0 tx=0).stop 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 shutdown_connections 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff0740a0790 0x7ff0740a2c40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff084101eb0 0x7ff08419f6b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff084100330 0x7ff084198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 --2- 172.21.15.31:0/1941941978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff084101520 0x7ff084198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 >> 172.21.15.31:0/1941941978 conn(0x7ff0840fb970 msgr2=0x7ff084106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 shutdown_connections 2023-07-29T16:17:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:22.959+0000 7ff08adff700 1 -- 172.21.15.31:0/1941941978 wait complete. 2023-07-29T16:17:22.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:17:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.183+0000 7f710c5ff700 1 -- 172.21.15.31:0/3855084451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 msgr2=0x7f7104073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.183+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3855084451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104073c20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009990 tx=0x7f70f4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 -- 172.21.15.31:0/3855084451 shutdown_connections 2023-07-29T16:17:23.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3855084451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71041081e0 0x7f710410a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3855084451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7104074d90 0x7f71040731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3855084451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104073c20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 -- 172.21.15.31:0/3855084451 >> 172.21.15.31:0/3855084451 conn(0x7f71040fb920 msgr2=0x7f71040fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 -- 172.21.15.31:0/3855084451 shutdown_connections 2023-07-29T16:17:23.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.184+0000 7f710c5ff700 1 -- 172.21.15.31:0/3855084451 wait complete. 2023-07-29T16:17:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 Processor -- start 2023-07-29T16:17:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 -- start start 2023-07-29T16:17:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7104074d90 0x7f7104198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71041081e0 0x7f710419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710419fe80 con 0x7f71040737b0 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710419ffc0 con 0x7f71041081e0 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710c5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71041a0150 con 0x7f7104074d90 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710a39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710a39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42202/0 (socket says 172.21.15.31:42202) 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f710a39b700 1 -- 172.21.15.31:0/3526329664 learned_addr learned my addr 172.21.15.31:0/3526329664 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.185+0000 7f7109b9a700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7104074d90 0x7f7104198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710ab9c700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71041081e0 0x7f710419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710a39b700 1 -- 172.21.15.31:0/3526329664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7104074d90 msgr2=0x7f7104198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710a39b700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7104074d90 0x7f7104198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710a39b700 1 -- 172.21.15.31:0/3526329664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71041081e0 msgr2=0x7f710419f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710a39b700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71041081e0 0x7f710419f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710a39b700 1 -- 172.21.15.31:0/3526329664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f40095f0 con 0x7f71040737b0 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710a39b700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104198300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f710000e970 tx=0x7f710000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f70fb7fe700 1 -- 172.21.15.31:0/3526329664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100009790 con 0x7f71040737b0 2023-07-29T16:17:23.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f70fb7fe700 1 -- 172.21.15.31:0/3526329664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f710000c7e0 con 0x7f71040737b0 2023-07-29T16:17:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71041a0430 con 0x7f71040737b0 2023-07-29T16:17:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f70fb7fe700 1 -- 172.21.15.31:0/3526329664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000054e0 con 0x7f71040737b0 2023-07-29T16:17:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.186+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71041a0990 con 0x7f71040737b0 2023-07-29T16:17:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.190+0000 7f70fb7fe700 1 -- 172.21.15.31:0/3526329664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f710000c950 con 0x7f71040737b0 2023-07-29T16:17:23.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 Processor -- start 2023-07-29T16:17:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 -- start start 2023-07-29T16:17:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f70fb7fe700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f70f00a0810 0x7f70f00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f70fb7fe700 1 -- 172.21.15.31:0/3526329664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7100014070 con 0x7f71040737b0 2023-07-29T16:17:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 0x55ac01bd6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd7530 0x55ac01bd7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01bd8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01b2ea10 con 0x55ac01bd82f0 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01b2c3d0 con 0x55ac01bd6be0 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.191+0000 7f711fe45b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01b27cc0 con 0x55ac01bd7530 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 0x55ac01bd6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 0x55ac01bd6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58190/0 (socket says 172.21.15.31:58190) 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 -- 172.21.15.31:0/341455446 learned_addr learned my addr 172.21.15.31:0/341455446 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710ab9c700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f70f00a0810 0x7f70f00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f7109b9a700 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd7530 0x55ac01bd7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 -- 172.21.15.31:0/341455446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd7530 msgr2=0x55ac01bd7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd7530 0x55ac01bd7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 -- 172.21.15.31:0/341455446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 msgr2=0x55ac01bd8740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:17:23.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01bd8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.192+0000 7f710a39b700 1 -- 172.21.15.31:0/341455446 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac01955de0 con 0x55ac01bd6be0 2023-07-29T16:17:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f710a39b700 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 0x55ac01bd6ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7100099450 tx=0x7f7100098e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=147017f444a960cb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f710ab9c700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f70f00a0810 0x7f70f00a2cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00f630 tx=0x7f70fc009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f70f8bfe700 1 -- 172.21.15.31:0/341455446 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000b1ea0 con 0x55ac01bd6be0 2023-07-29T16:17:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f70f8bfe700 1 -- 172.21.15.31:0/341455446 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71000994c0 con 0x55ac01bd6be0 2023-07-29T16:17:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f70f8bfe700 1 -- 172.21.15.31:0/341455446 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100005780 con 0x55ac01bd6be0 2023-07-29T16:17:23.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f711fe45b80 1 -- 172.21.15.31:0/341455446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 msgr2=0x55ac01bd6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f711fe45b80 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 0x55ac01bd6ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7100099450 tx=0x7f7100098e20 comp rx=0 tx=0).stop 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f711fe45b80 1 -- 172.21.15.31:0/341455446 shutdown_connections 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.193+0000 7f711fe45b80 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01bd8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd6be0 0x55ac01bd6ff0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 --2- 172.21.15.31:0/341455446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd7530 0x55ac01bd7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 -- 172.21.15.31:0/341455446 >> 172.21.15.31:0/341455446 conn(0x55ac01bcc3e0 msgr2=0x55ac01bd1990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 -- 172.21.15.31:0/341455446 shutdown_connections 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 -- 172.21.15.31:0/341455446 wait complete. 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 -- start start 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.194+0000 7f711fe45b80 1 Processor -- start 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01ac4510 con 0x55ac01bd82f0 2023-07-29T16:17:23.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01ac41b0 con 0x55ac01bd7530 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac019ab920 con 0x55ac01bd6be0 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f711fe45b80 10 client.0 _ll_register_callbacks cb 0x55ac01acd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58194/0 (socket says 172.21.15.31:58194) 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 -- 172.21.15.31:0/2871374098 learned_addr learned my addr 172.21.15.31:0/2871374098 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f710ab9c700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f710a39b700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 -- 172.21.15.31:0/2871374098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 msgr2=0x55ac01be0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 -- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 msgr2=0x55ac01be7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f7109b9a700 1 -- 172.21.15.31:0/2871374098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71041a0990 con 0x55ac01bd7530 2023-07-29T16:17:23.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.195+0000 7f710ab9c700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:23.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f7109b9a700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7104074ab0 tx=0x7f70f4005f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1e8ea80fda1e5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f70edffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f70edffb700 1 -- 172.21.15.31:0/2871374098 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f401d030 con 0x55ac01bd7530 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ac01b52d10 con 0x55ac01bd7530 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f70edffb700 1 -- 172.21.15.31:0/2871374098 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f70f4004700 con 0x55ac01bd7530 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f70edffb700 1 -- 172.21.15.31:0/2871374098 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70f40056f0 con 0x55ac01bd7530 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f711fe45b80 10 client.26189 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:23.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.196+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ac0193ec60 con 0x55ac01bd7530 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f70edffb700 1 -- 172.21.15.31:0/2871374098 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70f4032020 con 0x55ac01bd7530 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f70edffb700 1 -- 172.21.15.31:0/2871374098 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f70f400f990 con 0x55ac01bd7530 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f711fe45b80 10 client.26189 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f711fe45b80 10 client.26189 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f711fe45b80 20 client.26189 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f711fe45b80 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01bec5d0 0x55ac01bee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f711fe45b80 4 client.26189 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:23.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac01ba5d70 con 0x55ac01bec5d0 2023-07-29T16:17:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.197+0000 7f710a39b700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01bec5d0 0x55ac01bee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.198+0000 7f710a39b700 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01bec5d0 0x55ac01bee9e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76d4ca56bd0ec449 server_cookie=45c76f40b9957851 in_seq=0 out_seq=0 2023-07-29T16:17:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.198+0000 7f70edffb700 10 client.26189 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.198+0000 7f70edffb700 1 -- 172.21.15.31:0/2871374098 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ac01ba5d70 con 0x55ac01bec5d0 2023-07-29T16:17:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.198+0000 7f70edffb700 10 client.26189 handle_command_reply: tid=0 2023-07-29T16:17:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.198+0000 7f711fe45b80 1 client.26189 shutdown 2023-07-29T16:17:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 msgr2=0x55ac01be0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7104074ab0 tx=0x7f70f4005f00 comp rx=0 tx=0).stop 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 shutdown_connections 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01bec5d0 0x55ac01bee9e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f70edffb700 0 client.26189 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 --2- 172.21.15.31:0/2871374098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 >> 172.21.15.31:0/2871374098 conn(0x55ac01bcc3e0 msgr2=0x55ac01bd1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 shutdown_connections 2023-07-29T16:17:23.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 1 -- 172.21.15.31:0/2871374098 wait complete. 2023-07-29T16:17:23.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.199+0000 7f711fe45b80 20 client.26189 trim_cache size 0 max 16384 2023-07-29T16:17:23.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.212+0000 7f711fe45b80 1 Processor -- start 2023-07-29T16:17:23.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.212+0000 7f711fe45b80 1 -- start start 2023-07-29T16:17:23.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.212+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.212+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.212+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f711fe45b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01be1550 con 0x55ac01bd82f0 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f711fe45b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01bc9710 con 0x55ac01bd7530 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f711fe45b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01bd2690 con 0x55ac01bd6be0 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710ab9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56798/0 (socket says 172.21.15.31:56798) 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 -- 172.21.15.31:0/1896847401 learned_addr learned my addr 172.21.15.31:0/1896847401 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f7109b9a700 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 -- 172.21.15.31:0/1896847401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 msgr2=0x55ac01be0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 -- 172.21.15.31:0/1896847401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 msgr2=0x55ac01be0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 -- 172.21.15.31:0/1896847401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac0197c250 con 0x55ac01bd6be0 2023-07-29T16:17:23.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.213+0000 7f710a39b700 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7100095f30 tx=0x7f7100095800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f9043627252e93 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f70f93ff700 1 -- 172.21.15.31:0/1896847401 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000c9070 con 0x55ac01bd6be0 2023-07-29T16:17:23.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f70f93ff700 1 -- 172.21.15.31:0/1896847401 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71000a1910 con 0x55ac01bd6be0 2023-07-29T16:17:23.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f70f93ff700 1 -- 172.21.15.31:0/1896847401 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000b1ea0 con 0x55ac01bd6be0 2023-07-29T16:17:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 -- 172.21.15.31:0/1896847401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 msgr2=0x55ac01be5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7100095f30 tx=0x7f7100095800 comp rx=0 tx=0).stop 2023-07-29T16:17:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 -- 172.21.15.31:0/1896847401 shutdown_connections 2023-07-29T16:17:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5ca0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 --2- 172.21.15.31:0/1896847401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 -- 172.21.15.31:0/1896847401 >> 172.21.15.31:0/1896847401 conn(0x55ac01bcc3e0 msgr2=0x55ac01bde6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 -- 172.21.15.31:0/1896847401 shutdown_connections 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.214+0000 7f711fe45b80 1 -- 172.21.15.31:0/1896847401 wait complete. 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 -- start start 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 Processor -- start 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01be49e0 con 0x55ac01bd82f0 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01be4b50 con 0x55ac01bd7530 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.215+0000 7f711fe45b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac01be4d10 con 0x55ac01bd6be0 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f711fe45b80 10 client.0 _ll_register_callbacks cb 0x55ac01acd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710ab9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56802/0 (socket says 172.21.15.31:56802) 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 -- 172.21.15.31:0/163650606 learned_addr learned my addr 172.21.15.31:0/163650606 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f7109b9a700 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 -- 172.21.15.31:0/163650606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 msgr2=0x55ac01be3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 -- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 msgr2=0x55ac01be44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 -- 172.21.15.31:0/163650606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac01955de0 con 0x55ac01bd6be0 2023-07-29T16:17:23.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f710a39b700 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f710000cc00 tx=0x7f710000cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a19abf5666a703d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.216+0000 7f70edffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f70edffb700 1 -- 172.21.15.31:0/163650606 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000b1ea0 con 0x55ac01bd6be0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f70edffb700 1 -- 172.21.15.31:0/163650606 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f71000056d0 con 0x55ac01bd6be0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71000994c0 con 0x55ac01bd6be0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f70edffb700 1 -- 172.21.15.31:0/163650606 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7100097960 con 0x55ac01bd6be0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 10 client.26200 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ac019542b0 con 0x55ac01bd6be0 2023-07-29T16:17:23.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f70edffb700 1 -- 172.21.15.31:0/163650606 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71000994c0 con 0x55ac01bd6be0 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f70edffb700 1 -- 172.21.15.31:0/163650606 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f71000c9d20 con 0x55ac01bd6be0 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 10 client.26200 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 10 client.26200 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 20 client.26200 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01c3f3b0 0x55ac01c417c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 4 client.26200 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:17:23.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.217+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ac01b20d40 con 0x55ac01c3f3b0 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.218+0000 7f7109b9a700 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01c3f3b0 0x55ac01c417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.218+0000 7f7109b9a700 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01c3f3b0 0x55ac01c417c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=667809920b9a2b8e server_cookie=5211322985a51746 in_seq=0 out_seq=0 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.218+0000 7f70edffb700 10 client.26200 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.218+0000 7f70edffb700 1 -- 172.21.15.31:0/163650606 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ac01b20d40 con 0x55ac01c3f3b0 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.218+0000 7f70edffb700 10 client.26200 handle_command_reply: tid=0 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.218+0000 7f711fe45b80 1 client.26200 shutdown 2023-07-29T16:17:23.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 msgr2=0x55ac01be5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f710000cc00 tx=0x7f710000cf50 comp rx=0 tx=0).stop 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 shutdown_connections 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ac01c3f3b0 0x55ac01c417c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac01bd82f0 0x55ac01be44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac01bd6be0 0x55ac01be5950 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f70edffb700 0 client.26200 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 --2- 172.21.15.31:0/163650606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac01bd7530 0x55ac01be3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 >> 172.21.15.31:0/163650606 conn(0x55ac01bcc3e0 msgr2=0x55ac01be07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 shutdown_connections 2023-07-29T16:17:23.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 1 -- 172.21.15.31:0/163650606 wait complete. 2023-07-29T16:17:23.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f711fe45b80 20 client.26200 trim_cache size 0 max 16384 2023-07-29T16:17:23.226 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:17:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f70f00a0810 msgr2=0x7f70f00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.219+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f70f00a0810 0x7f70f00a2cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00f630 tx=0x7f70fc009920 comp rx=0 tx=0).stop 2023-07-29T16:17:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 msgr2=0x7f7104198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104198300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f710000e970 tx=0x7f710000ec80 comp rx=0 tx=0).stop 2023-07-29T16:17:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 shutdown_connections 2023-07-29T16:17:23.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f70f00a0810 0x7f70f00a2cc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f71041081e0 0x7f710419f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71040737b0 0x7f7104198300 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 --2- 172.21.15.31:0/3526329664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7104074d90 0x7f7104198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 >> 172.21.15.31:0/3526329664 conn(0x7f71040fb920 msgr2=0x7f71040fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 shutdown_connections 2023-07-29T16:17:23.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.220+0000 7f710c5ff700 1 -- 172.21.15.31:0/3526329664 wait complete. 2023-07-29T16:17:23.237 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:17:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 -- 172.21.15.31:0/2700203641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4100e40 msgr2=0x7f27a4103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/2700203641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4100e40 0x7f27a4103220 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009990 tx=0x7f27a0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 -- 172.21.15.31:0/2700203641 shutdown_connections 2023-07-29T16:17:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/2700203641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4106080 0x7f27a4108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/2700203641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4100e40 0x7f27a4103220 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/2700203641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4103760 0x7f27a4105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 -- 172.21.15.31:0/2700203641 >> 172.21.15.31:0/2700203641 conn(0x7f27a40fa840 msgr2=0x7f27a40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 -- 172.21.15.31:0/2700203641 shutdown_connections 2023-07-29T16:17:23.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.450+0000 7f27ac7ff700 1 -- 172.21.15.31:0/2700203641 wait complete. 2023-07-29T16:17:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.451+0000 7f27ac7ff700 1 Processor -- start 2023-07-29T16:17:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.451+0000 7f27ac7ff700 1 -- start start 2023-07-29T16:17:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27ac7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 0x7f27a41007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27ac7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4103760 0x7f27a40fedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27ac7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4106080 0x7f27a40ff330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27ac7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a40ffad0 con 0x7f27a4100e40 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27ac7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a40ffc40 con 0x7f27a4103760 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27ac7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a41a00a0 con 0x7f27a4106080 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 0x7f27a41007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 0x7f27a41007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42254/0 (socket says 172.21.15.31:42254) 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27aa59b700 1 -- 172.21.15.31:0/589145365 learned_addr learned my addr 172.21.15.31:0/589145365 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4103760 0x7f27a40fedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27aad9c700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4106080 0x7f27a40ff330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27aa59b700 1 -- 172.21.15.31:0/589145365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4106080 msgr2=0x7f27a40ff330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.452+0000 7f27aa59b700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4106080 0x7f27a40ff330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27aa59b700 1 -- 172.21.15.31:0/589145365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4103760 msgr2=0x7f27a40fedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27aa59b700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4103760 0x7f27a40fedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27aa59b700 1 -- 172.21.15.31:0/589145365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2794009520 con 0x7f27a4100e40 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27aad9c700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4106080 0x7f27a40ff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:23.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4103760 0x7f27a40fedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27aa59b700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 0x7f27a41007b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004740 tx=0x7f27a0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f279b7fe700 1 -- 172.21.15.31:0/589145365 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a001d070 con 0x7f27a4100e40 2023-07-29T16:17:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a00095f0 con 0x7f27a4100e40 2023-07-29T16:17:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f279b7fe700 1 -- 172.21.15.31:0/589145365 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27a0004830 con 0x7f27a4100e40 2023-07-29T16:17:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f279b7fe700 1 -- 172.21.15.31:0/589145365 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0005450 con 0x7f27a4100e40 2023-07-29T16:17:23.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.453+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a41a05b0 con 0x7f27a4100e40 2023-07-29T16:17:23.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.456+0000 7f279b7fe700 1 -- 172.21.15.31:0/589145365 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f27a000fb00 con 0x7f27a4100e40 2023-07-29T16:17:23.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.458+0000 7f27c0138b80 1 Processor -- start 2023-07-29T16:17:23.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.458+0000 7f27c0138b80 1 -- start start 2023-07-29T16:17:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b356ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b357530 0x55c30b357d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 0x55c30b358740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27c0138b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b2aea10 con 0x55c30b356be0 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27c0138b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b2ac3d0 con 0x55c30b3582f0 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27c0138b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b2a7cc0 con 0x55c30b357530 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f279b7fe700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f27900a0790 0x7f27900a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f279b7fe700 1 -- 172.21.15.31:0/589145365 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f27a00ccce0 con 0x7f27a4100e40 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27a9d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 0x55c30b358740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27aad9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b356ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27a9d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 0x55c30b358740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58236/0 (socket says 172.21.15.31:58236) 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27aad9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b356ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42256/0 (socket says 172.21.15.31:42256) 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27a9d9a700 1 -- 172.21.15.31:0/2885355846 learned_addr learned my addr 172.21.15.31:0/2885355846 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27aa59b700 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b357530 0x55c30b357d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.459+0000 7f27aad9c700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f27900a0790 0x7f27900a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27a9d9a700 1 -- 172.21.15.31:0/2885355846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b357530 msgr2=0x55c30b357d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b357530 0x55c30b357d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27a9d9a700 1 -- 172.21.15.31:0/2885355846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 msgr2=0x55c30b356ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b356ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27a9d9a700 1 -- 172.21.15.31:0/2885355846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c30b0d5de0 con 0x55c30b3582f0 2023-07-29T16:17:23.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27aa59b700 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b357530 0x55c30b357d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:23.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 0x55c30b358740 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2794011760 tx=0x7f2794011a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=283d9c7b28e2d921 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f27aad9c700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f27900a0790 0x7f27900a2c40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55c30b356980 tx=0x7f279c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f2798ff9700 1 -- 172.21.15.31:0/2885355846 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794019410 con 0x55c30b3582f0 2023-07-29T16:17:23.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f2798ff9700 1 -- 172.21.15.31:0/2885355846 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2794019a30 con 0x55c30b3582f0 2023-07-29T16:17:23.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.460+0000 7f2798ff9700 1 -- 172.21.15.31:0/2885355846 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279400fb50 con 0x55c30b3582f0 2023-07-29T16:17:23.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 -- 172.21.15.31:0/2885355846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 msgr2=0x55c30b358740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 0x55c30b358740 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2794011760 tx=0x7f2794011a70 comp rx=0 tx=0).stop 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 -- 172.21.15.31:0/2885355846 shutdown_connections 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b3582f0 0x55c30b358740 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b356ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 --2- 172.21.15.31:0/2885355846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b357530 0x55c30b357d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 -- 172.21.15.31:0/2885355846 >> 172.21.15.31:0/2885355846 conn(0x55c30b34c3e0 msgr2=0x55c30b351990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 -- 172.21.15.31:0/2885355846 shutdown_connections 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.461+0000 7f27c0138b80 1 -- 172.21.15.31:0/2885355846 wait complete. 2023-07-29T16:17:23.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 -- start start 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 Processor -- start 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 0x55c30b360800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b360d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b3582f0 0x55c30b367d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b244510 con 0x55c30b3582f0 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b2441b0 con 0x55c30b357530 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.462+0000 7f27c0138b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b12b920 con 0x55c30b356be0 2023-07-29T16:17:23.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27c0138b80 10 client.0 _ll_register_callbacks cb 0x55c30b24d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 0x55c30b360800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aad9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b3582f0 0x55c30b367d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 0x55c30b360800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56846/0 (socket says 172.21.15.31:56846) 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 -- 172.21.15.31:0/853274087 learned_addr learned my addr 172.21.15.31:0/853274087 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b360d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 -- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 msgr2=0x55c30b360d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b360d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 -- 172.21.15.31:0/853274087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b3582f0 msgr2=0x55c30b367d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b3582f0 0x55c30b367d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 -- 172.21.15.31:0/853274087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a41a05b0 con 0x55c30b356be0 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f27aa59b700 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 0x55c30b360800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004710 tx=0x7f27a00a0390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1de70414dbde4d65 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.463+0000 7f278dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f278dffb700 1 -- 172.21.15.31:0/853274087 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a009e440 con 0x55c30b356be0 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f278dffb700 1 -- 172.21.15.31:0/853274087 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f27a00a27d0 con 0x55c30b356be0 2023-07-29T16:17:23.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c30b2d2d10 con 0x55c30b356be0 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f278dffb700 1 -- 172.21.15.31:0/853274087 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27a00055f0 con 0x55c30b356be0 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 10 client.26212 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f2794011440 con 0x55c30b356be0 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f278dffb700 1 -- 172.21.15.31:0/853274087 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c30b2d2d10 con 0x55c30b356be0 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f278dffb700 1 -- 172.21.15.31:0/853274087 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f27a0099900 con 0x55c30b356be0 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 10 client.26212 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 10 client.26212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:23.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 20 client.26212 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b36e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 4 client.26212 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.464+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c30b325d70 con 0x55c30b36c1b0 2023-07-29T16:17:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.465+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b36e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.465+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b36e5c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55df0586f24d0cb4 server_cookie=c21d55919f478829 in_seq=0 out_seq=0 2023-07-29T16:17:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.465+0000 7f278dffb700 10 client.26212 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f278dffb700 1 -- 172.21.15.31:0/853274087 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c30b325d70 con 0x55c30b36c1b0 2023-07-29T16:17:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f278dffb700 10 client.26212 handle_command_reply: tid=0 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f27c0138b80 1 client.26212 shutdown 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 msgr2=0x55c30b360800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f27c0138b80 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 0x55c30b360800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004710 tx=0x7f27a00a0390 comp rx=0 tx=0).stop 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 shutdown_connections 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.466+0000 7f27c0138b80 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b36e5c0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b3582f0 0x55c30b367d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f278dffb700 0 client.26212 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b356be0 0x55c30b360800 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 1 --2- 172.21.15.31:0/853274087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b360d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 >> 172.21.15.31:0/853274087 conn(0x55c30b34c3e0 msgr2=0x55c30b351060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 shutdown_connections 2023-07-29T16:17:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 1 -- 172.21.15.31:0/853274087 wait complete. 2023-07-29T16:17:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.467+0000 7f27c0138b80 20 client.26212 trim_cache size 0 max 16384 2023-07-29T16:17:23.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 Processor -- start 2023-07-29T16:17:23.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 -- start start 2023-07-29T16:17:23.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b365d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b3586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b351900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b35fcb0 con 0x55c30b356be0 2023-07-29T16:17:23.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b352ce0 con 0x55c30b357530 2023-07-29T16:17:23.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27c0138b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b353340 con 0x55c30b3582f0 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b365d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b365d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42270/0 (socket says 172.21.15.31:42270) 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27aa59b700 1 -- 172.21.15.31:0/3543951900 learned_addr learned my addr 172.21.15.31:0/3543951900 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b3586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.481+0000 7f27aad9c700 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b351900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27aa59b700 1 -- 172.21.15.31:0/3543951900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 msgr2=0x55c30b351900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27aa59b700 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b351900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27aa59b700 1 -- 172.21.15.31:0/3543951900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 msgr2=0x55c30b3586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27aa59b700 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b3586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27aa59b700 1 -- 172.21.15.31:0/3543951900 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c30b0bec60 con 0x55c30b356be0 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b3586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27aa59b700 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b365d30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f27a0096e80 tx=0x7f27a00989c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f02a29caffb74f9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27997fa700 1 -- 172.21.15.31:0/3543951900 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a00a2850 con 0x55c30b356be0 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27997fa700 1 -- 172.21.15.31:0/3543951900 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27a009e440 con 0x55c30b356be0 2023-07-29T16:17:23.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.482+0000 7f27997fa700 1 -- 172.21.15.31:0/3543951900 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a009a8d0 con 0x55c30b356be0 2023-07-29T16:17:23.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 -- 172.21.15.31:0/3543951900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 msgr2=0x55c30b365d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b365d30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f27a0096e80 tx=0x7f27a00989c0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 -- 172.21.15.31:0/3543951900 shutdown_connections 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b351900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b356be0 0x55c30b365d30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3543951900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b357530 0x55c30b3586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 -- 172.21.15.31:0/3543951900 >> 172.21.15.31:0/3543951900 conn(0x55c30b34c3e0 msgr2=0x55c30b35f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 -- 172.21.15.31:0/3543951900 shutdown_connections 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 -- 172.21.15.31:0/3543951900 wait complete. 2023-07-29T16:17:23.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.483+0000 7f27c0138b80 1 -- start start 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 Processor -- start 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 0x55c30b363ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b357530 0x55c30b364010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b365020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b365560 con 0x55c30b357530 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b3656d0 con 0x55c30b356be0 2023-07-29T16:17:23.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c30b365890 con 0x55c30b3582f0 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27c0138b80 10 client.0 _ll_register_callbacks cb 0x55c30b373440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 0x55c30b363ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27a9d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b357530 0x55c30b364010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27aa59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 0x55c30b363ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58256/0 (socket says 172.21.15.31:58256) 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27aa59b700 1 -- 172.21.15.31:0/3671269861 learned_addr learned my addr 172.21.15.31:0/3671269861 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.484+0000 7f27aad9c700 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b365020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27aa59b700 1 -- 172.21.15.31:0/3671269861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 msgr2=0x55c30b365020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27aa59b700 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b365020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27aa59b700 1 -- 172.21.15.31:0/3671269861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b357530 msgr2=0x55c30b364010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27aa59b700 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b357530 0x55c30b364010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27aa59b700 1 -- 172.21.15.31:0/3671269861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c30b0d5de0 con 0x55c30b356be0 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27aa59b700 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 0x55c30b363ad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f27a00a22a0 tx=0x7f27a00c6200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=568aa5e276f83a57 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f278dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f278dffb700 1 -- 172.21.15.31:0/3671269861 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a009e230 con 0x55c30b356be0 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f278dffb700 1 -- 172.21.15.31:0/3671269861 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f27a00ca7a0 con 0x55c30b356be0 2023-07-29T16:17:23.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2794019a30 con 0x55c30b356be0 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f278dffb700 1 -- 172.21.15.31:0/3671269861 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27a009a670 con 0x55c30b356be0 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27c0138b80 10 client.26213 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.485+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f2794011440 con 0x55c30b356be0 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f278dffb700 1 -- 172.21.15.31:0/3671269861 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2794019a30 con 0x55c30b356be0 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f278dffb700 1 -- 172.21.15.31:0/3671269861 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f27a0066230 con 0x55c30b356be0 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27c0138b80 10 client.26213 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:23.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27c0138b80 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27c0138b80 20 client.26213 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b3c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27c0138b80 4 client.26213 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c30b2a0d40 con 0x55c30b36c1b0 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b3c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f27a9d9a700 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b3c07d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb692a24fcf9eb1f server_cookie=f1a0b6ce16040b88 in_seq=0 out_seq=0 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.486+0000 7f278dffb700 10 client.26213 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f278dffb700 1 -- 172.21.15.31:0/3671269861 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c30b2a0d40 con 0x55c30b36c1b0 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f278dffb700 10 client.26213 handle_command_reply: tid=0 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 client.26213 shutdown 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 msgr2=0x55c30b363ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 0x55c30b363ad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f27a00a22a0 tx=0x7f27a00c6200 comp rx=0 tx=0).stop 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 shutdown_connections 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c30b36c1b0 0x55c30b3c07d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c30b3582f0 0x55c30b365020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c30b356be0 0x55c30b363ad0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f278dffb700 0 client.26213 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 --2- 172.21.15.31:0/3671269861 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c30b357530 0x55c30b364010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 >> 172.21.15.31:0/3671269861 conn(0x55c30b34c3e0 msgr2=0x55c30b35e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 shutdown_connections 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 1 -- 172.21.15.31:0/3671269861 wait complete. 2023-07-29T16:17:23.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.487+0000 7f27c0138b80 20 client.26213 trim_cache size 0 max 16384 2023-07-29T16:17:23.495 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:17:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f27900a0790 msgr2=0x7f27900a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f27900a0790 0x7f27900a2c40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55c30b356980 tx=0x7f279c00b040 comp rx=0 tx=0).stop 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 msgr2=0x7f27a41007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 0x7f27a41007b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f27a0004740 tx=0x7f27a0009580 comp rx=0 tx=0).stop 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 shutdown_connections 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f27900a0790 0x7f27900a2c40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f27a4106080 0x7f27a40ff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27a4100e40 0x7f27a41007b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 --2- 172.21.15.31:0/589145365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f27a4103760 0x7f27a40fedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 >> 172.21.15.31:0/589145365 conn(0x7f27a40fa840 msgr2=0x7f27a4102280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 shutdown_connections 2023-07-29T16:17:23.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:23.488+0000 7f27ac7ff700 1 -- 172.21.15.31:0/589145365 wait complete. 2023-07-29T16:17:23.502 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3432130467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:17:23.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3432130467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:17:23.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3432130467' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:17:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/2536881455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 msgr2=0x7f2d940731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/2536881455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d940731e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009990 tx=0x7f2d90009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/2536881455 shutdown_connections 2023-07-29T16:17:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/2536881455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d94108210 0x7f2d9410a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/2536881455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d940731e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/2536881455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d940737b0 0x7f2d94073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/2536881455 >> 172.21.15.31:0/2536881455 conn(0x7f2d940fb970 msgr2=0x7f2d940fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/2536881455 shutdown_connections 2023-07-29T16:17:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.873+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/2536881455 wait complete. 2023-07-29T16:17:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.874+0000 7f2d9c1ff700 1 Processor -- start 2023-07-29T16:17:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.874+0000 7f2d9c1ff700 1 -- start start 2023-07-29T16:17:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9c1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 0x7f2d94198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9c1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d94198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9c1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d94108210 0x7f2d941997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9c1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d94199e20 con 0x7f2d940737b0 2023-07-29T16:17:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9c1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d94199f90 con 0x7f2d94074d90 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9c1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9419a150 con 0x7f2d94108210 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 0x7f2d94198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 0x7f2d94198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42288/0 (socket says 172.21.15.31:42288) 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 -- 172.21.15.31:0/1427365909 learned_addr learned my addr 172.21.15.31:0/1427365909 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9979a700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d94198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d94108210 0x7f2d941997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 -- 172.21.15.31:0/1427365909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d94108210 msgr2=0x7f2d941997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d94108210 0x7f2d941997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 -- 172.21.15.31:0/1427365909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 msgr2=0x7f2d94198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d94198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d99f9b700 1 -- 172.21.15.31:0/1427365909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d900095f0 con 0x7f2d940737b0 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.875+0000 7f2d9979a700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d94198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.876+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 0x7f2d94198400 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2d90004710 tx=0x7f2d90009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.876+0000 7f2d8affd700 1 -- 172.21.15.31:0/1427365909 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9001d070 con 0x7f2d940737b0 2023-07-29T16:17:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.876+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9419a3d0 con 0x7f2d940737b0 2023-07-29T16:17:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.876+0000 7f2d8affd700 1 -- 172.21.15.31:0/1427365909 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d90021410 con 0x7f2d940737b0 2023-07-29T16:17:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.876+0000 7f2d8affd700 1 -- 172.21.15.31:0/1427365909 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90017650 con 0x7f2d940737b0 2023-07-29T16:17:23.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.876+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d941a0bf0 con 0x7f2d940737b0 2023-07-29T16:17:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.879+0000 7f2d8affd700 1 -- 172.21.15.31:0/1427365909 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2d90021580 con 0x7f2d940737b0 2023-07-29T16:17:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 Processor -- start 2023-07-29T16:17:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2d8affd700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2d800a0790 0x7f2d800a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2d8affd700 1 -- 172.21.15.31:0/1427365909 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2d900cced0 con 0x7f2d940737b0 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 -- start start 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2d800a0790 0x7f2d800a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe72160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbdc89b0 con 0x5616fbe71cf0 2023-07-29T16:17:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbdc6370 con 0x5616fbccc6b0 2023-07-29T16:17:23.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.881+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbdc1c60 con 0x5616fbe70df0 2023-07-29T16:17:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9a79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe72160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56884/0 (socket says 172.21.15.31:56884) 2023-07-29T16:17:23.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 -- 172.21.15.31:0/1170203181 learned_addr learned my addr 172.21.15.31:0/1170203181 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2d800a0790 0x7f2d800a2c40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00f590 tx=0x7f2d8c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 -- 172.21.15.31:0/1170203181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 msgr2=0x5616fbe708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 -- 172.21.15.31:0/1170203181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 msgr2=0x5616fbe72160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe72160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 -- 172.21.15.31:0/1170203181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616fbbefd80 con 0x5616fbe70df0 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe72160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d9979a700 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe717b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d840115c0 tx=0x7f2d840118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7167a020cee6e91c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.882+0000 7f2d7ffff700 1 -- 172.21.15.31:0/1170203181 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d84004220 con 0x5616fbe70df0 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2d7ffff700 1 -- 172.21.15.31:0/1170203181 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d84004cf0 con 0x5616fbe70df0 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2d7ffff700 1 -- 172.21.15.31:0/1170203181 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d840056b0 con 0x5616fbe70df0 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1170203181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 msgr2=0x5616fbe717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe717b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d840115c0 tx=0x7f2d840118d0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1170203181 shutdown_connections 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe72160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1170203181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe717b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1170203181 >> 172.21.15.31:0/1170203181 conn(0x5616fbe65fd0 msgr2=0x5616fbe6b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1170203181 shutdown_connections 2023-07-29T16:17:23.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.883+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1170203181 wait complete. 2023-07-29T16:17:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.884+0000 7f2dafbceb80 1 -- start start 2023-07-29T16:17:23.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.884+0000 7f2dafbceb80 1 Processor -- start 2023-07-29T16:17:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe79f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbd5e4b0 con 0x5616fbe71cf0 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbd5e150 con 0x5616fbccc6b0 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbc458c0 con 0x5616fbe70df0 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2dafbceb80 10 client.0 _ll_register_callbacks cb 0x5616fbd675c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe79f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d9a79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d9979a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe79f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58296/0 (socket says 172.21.15.31:58296) 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 -- 172.21.15.31:0/170780475 learned_addr learned my addr 172.21.15.31:0/170780475 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 -- 172.21.15.31:0/170780475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 msgr2=0x5616fbe7a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 -- 172.21.15.31:0/170780475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 msgr2=0x5616fbe814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d99f9b700 1 -- 172.21.15.31:0/170780475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c019460 con 0x5616fbccc6b0 2023-07-29T16:17:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.885+0000 7f2d9979a700 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe79f40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2d90000c00 tx=0x7f2d90021b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b9fd4c386144b78 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d7d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/170780475 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d90099cf0 con 0x5616fbccc6b0 2023-07-29T16:17:23.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/170780475 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2d9009e630 con 0x5616fbccc6b0 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616fbdeccb0 con 0x5616fbccc6b0 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/170780475 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d900a29c0 con 0x5616fbccc6b0 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2dafbceb80 10 client.26225 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f2d84009d50 con 0x5616fbccc6b0 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/170780475 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d900c8080 con 0x5616fbccc6b0 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.886+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/170780475 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2d9009e280 con 0x5616fbccc6b0 2023-07-29T16:17:23.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2dafbceb80 10 client.26225 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2dafbceb80 10 client.26225 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2dafbceb80 20 client.26225 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbe85990 0x5616fbe87d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2dafbceb80 4 client.26225 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616fbe3fd10 con 0x5616fbe85990 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.887+0000 7f2d9979a700 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbe85990 0x5616fbe87d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.888+0000 7f2d9979a700 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbe85990 0x5616fbe87d70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3302e8b53c149c2 server_cookie=43d4d6740544a5b5 in_seq=0 out_seq=0 2023-07-29T16:17:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.888+0000 7f2d7d7fa700 10 client.26225 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.888+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/170780475 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5616fbe3fd10 con 0x5616fbe85990 2023-07-29T16:17:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.888+0000 7f2d7d7fa700 10 client.26225 handle_command_reply: tid=0 2023-07-29T16:17:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.888+0000 7f2dafbceb80 1 client.26225 shutdown 2023-07-29T16:17:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 msgr2=0x5616fbe79f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe79f40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2d90000c00 tx=0x7f2d90021b60 comp rx=0 tx=0).stop 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 shutdown_connections 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbe85990 0x5616fbe87d70 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe79f40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2d7d7fa700 0 client.26225 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/170780475 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 >> 172.21.15.31:0/170780475 conn(0x5616fbe65fd0 msgr2=0x5616fbe6aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 shutdown_connections 2023-07-29T16:17:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 1 -- 172.21.15.31:0/170780475 wait complete. 2023-07-29T16:17:23.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.889+0000 7f2dafbceb80 20 client.26225 trim_cache size 0 max 16384 2023-07-29T16:17:23.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 Processor -- start 2023-07-29T16:17:23.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 -- start start 2023-07-29T16:17:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 0x5616fbe720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe81480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbe71cf0 0x5616fbe780b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbe72600 con 0x5616fbccc6b0 2023-07-29T16:17:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbe6cdf0 con 0x5616fbe71cf0 2023-07-29T16:17:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbe77e60 con 0x5616fbe70df0 2023-07-29T16:17:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.904+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 0x5616fbe720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 0x5616fbe720c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42316/0 (socket says 172.21.15.31:42316) 2023-07-29T16:17:23.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 -- 172.21.15.31:0/78486060 learned_addr learned my addr 172.21.15.31:0/78486060 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d9979a700 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe81480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbe71cf0 0x5616fbe780b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 -- 172.21.15.31:0/78486060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 msgr2=0x5616fbe81480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe81480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 -- 172.21.15.31:0/78486060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbe71cf0 msgr2=0x5616fbe780b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbe71cf0 0x5616fbe780b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 -- 172.21.15.31:0/78486060 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616fbbd8c00 con 0x5616fbccc6b0 2023-07-29T16:17:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbe71cf0 0x5616fbe780b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 0x5616fbe720c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2d9009a590 tx=0x7f2d90098bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b6b6731a2f5948 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d88ff9700 1 -- 172.21.15.31:0/78486060 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9009e360 con 0x5616fbccc6b0 2023-07-29T16:17:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d88ff9700 1 -- 172.21.15.31:0/78486060 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616fbdeccb0 con 0x5616fbccc6b0 2023-07-29T16:17:23.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.905+0000 7f2d88ff9700 1 -- 172.21.15.31:0/78486060 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d900a2790 con 0x5616fbccc6b0 2023-07-29T16:17:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 -- 172.21.15.31:0/78486060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 msgr2=0x5616fbe720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 0x5616fbe720c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2d9009a590 tx=0x7f2d90098bb0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 -- 172.21.15.31:0/78486060 shutdown_connections 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbe71cf0 0x5616fbe780b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbccc6b0 0x5616fbe720c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/78486060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe81480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 -- 172.21.15.31:0/78486060 >> 172.21.15.31:0/78486060 conn(0x5616fbe65fd0 msgr2=0x5616fbe78c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 -- 172.21.15.31:0/78486060 shutdown_connections 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.906+0000 7f2dafbceb80 1 -- 172.21.15.31:0/78486060 wait complete. 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 -- start start 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 Processor -- start 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe7d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe80850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbe80d90 con 0x5616fbe71cf0 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbe80f00 con 0x5616fbccc6b0 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616fbe810c0 con 0x5616fbe70df0 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2dafbceb80 10 client.0 _ll_register_callbacks cb 0x5616fbd675c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2d9a79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe80850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2d9a79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe80850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42332/0 (socket says 172.21.15.31:42332) 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.907+0000 7f2d9a79c700 1 -- 172.21.15.31:0/1881393462 learned_addr learned my addr 172.21.15.31:0/1881393462 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:23.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9979a700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe7d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9a79c700 1 -- 172.21.15.31:0/1881393462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 msgr2=0x5616fbe7db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9a79c700 1 -- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 msgr2=0x5616fbe7d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe7d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9a79c700 1 -- 172.21.15.31:0/1881393462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d84009d50 con 0x5616fbe71cf0 2023-07-29T16:17:23.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d9a79c700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe80850 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c017740 tx=0x7f2d8c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178732fbfc2c0aa1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d7d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/1881393462 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c00e040 con 0x5616fbe71cf0 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d84004cf0 con 0x5616fbe71cf0 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.908+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/1881393462 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2d8c023800 con 0x5616fbe71cf0 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/1881393462 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d8c0239c0 con 0x5616fbe71cf0 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 10 client.16254 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:23.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5616fbc161f0 con 0x5616fbe71cf0 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/1881393462 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d84004cf0 con 0x5616fbe71cf0 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/1881393462 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2d8c037b10 con 0x5616fbe71cf0 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 10 client.16254 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 20 client.16254 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbed8680 0x5616fbedaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 4 client.16254 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:17:23.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.909+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5616fbdbace0 con 0x5616fbed8680 2023-07-29T16:17:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.910+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbed8680 0x5616fbedaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.910+0000 7f2d99f9b700 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbed8680 0x5616fbedaa90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1915fbe40040d74a server_cookie=11c08c22a9610de in_seq=0 out_seq=0 2023-07-29T16:17:23.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.910+0000 7f2d7d7fa700 10 client.16254 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2d7d7fa700 1 -- 172.21.15.31:0/1881393462 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5616fbdbace0 con 0x5616fbed8680 2023-07-29T16:17:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2d7d7fa700 10 client.16254 handle_command_reply: tid=0 2023-07-29T16:17:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 client.16254 shutdown 2023-07-29T16:17:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 msgr2=0x5616fbe80850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe80850 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c017740 tx=0x7f2d8c004b40 comp rx=0 tx=0).stop 2023-07-29T16:17:23.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 shutdown_connections 2023-07-29T16:17:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616fbed8680 0x5616fbedaa90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616fbe71cf0 0x5616fbe80850 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616fbccc6b0 0x5616fbe7d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 --2- 172.21.15.31:0/1881393462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616fbe70df0 0x5616fbe7db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2d7d7fa700 0 client.16254 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:23.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 >> 172.21.15.31:0/1881393462 conn(0x5616fbe65fd0 msgr2=0x5616fbe780b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.914+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 shutdown_connections 2023-07-29T16:17:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2dafbceb80 1 -- 172.21.15.31:0/1881393462 wait complete. 2023-07-29T16:17:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2dafbceb80 20 client.16254 trim_cache size 0 max 16384 2023-07-29T16:17:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:17:23.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:17:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:17:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:17:23.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2d800a0790 msgr2=0x7f2d800a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2d800a0790 0x7f2d800a2c40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c00f590 tx=0x7f2d8c009920 comp rx=0 tx=0).stop 2023-07-29T16:17:23.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 msgr2=0x7f2d94198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:23.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 0x7f2d94198400 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2d90004710 tx=0x7f2d90009580 comp rx=0 tx=0).stop 2023-07-29T16:17:23.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 shutdown_connections 2023-07-29T16:17:23.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2d800a0790 0x7f2d800a2c40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d94108210 0x7f2d941997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d940737b0 0x7f2d94198400 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 --2- 172.21.15.31:0/1427365909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2d94074d90 0x7f2d94198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.915+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 >> 172.21.15.31:0/1427365909 conn(0x7f2d940fb970 msgr2=0x7f2d940fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.916+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 shutdown_connections 2023-07-29T16:17:23.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:23.916+0000 7f2d9c1ff700 1 -- 172.21.15.31:0/1427365909 wait complete. 2023-07-29T16:17:24.505 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:17:24.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:17:24.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:24 smithi031 ceph-mon[95504]: pgmap v427: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 -- 172.21.15.31:0/2405767462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101520 msgr2=0x7f562c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 --2- 172.21.15.31:0/2405767462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101520 0x7f562c101970 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f5614009990 tx=0x7f5614009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 -- 172.21.15.31:0/2405767462 shutdown_connections 2023-07-29T16:17:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 --2- 172.21.15.31:0/2405767462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c101eb0 0x7f562c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 --2- 172.21.15.31:0/2405767462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f562c100330 0x7f562c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 --2- 172.21.15.31:0/2405767462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101520 0x7f562c101970 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 -- 172.21.15.31:0/2405767462 >> 172.21.15.31:0/2405767462 conn(0x7f562c0fb970 msgr2=0x7f562c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 -- 172.21.15.31:0/2405767462 shutdown_connections 2023-07-29T16:17:24.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.719+0000 7f56325ff700 1 -- 172.21.15.31:0/2405767462 wait complete. 2023-07-29T16:17:24.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 Processor -- start 2023-07-29T16:17:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 -- start start 2023-07-29T16:17:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c100330 0x7f562c074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f562c101520 0x7f562c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 0x7f562c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c073e20 con 0x7f562c101eb0 2023-07-29T16:17:24.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c073f90 con 0x7f562c100330 2023-07-29T16:17:24.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f56325ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c1a0110 con 0x7f562c101520 2023-07-29T16:17:24.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f5630b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 0x7f562c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f5630b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 0x7f562c073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42342/0 (socket says 172.21.15.31:42342) 2023-07-29T16:17:24.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.720+0000 7f5630b9c700 1 -- 172.21.15.31:0/2983332330 learned_addr learned my addr 172.21.15.31:0/2983332330 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:24.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f562bfff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c100330 0x7f562c074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5623fff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f562c101520 0x7f562c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5630b9c700 1 -- 172.21.15.31:0/2983332330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f562c101520 msgr2=0x7f562c073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5630b9c700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f562c101520 0x7f562c073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5630b9c700 1 -- 172.21.15.31:0/2983332330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c100330 msgr2=0x7f562c074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5630b9c700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c100330 0x7f562c074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5630b9c700 1 -- 172.21.15.31:0/2983332330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140095f0 con 0x7f562c101eb0 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f562bfff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c100330 0x7f562c074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:24.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5630b9c700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 0x7f562c073680 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5624007ee0 tx=0x7f562400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5629ffb700 1 -- 172.21.15.31:0/2983332330 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562400dd20 con 0x7f562c101eb0 2023-07-29T16:17:24.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c1a03f0 con 0x7f562c101eb0 2023-07-29T16:17:24.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.721+0000 7f5629ffb700 1 -- 172.21.15.31:0/2983332330 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f562400f040 con 0x7f562c101eb0 2023-07-29T16:17:24.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.722+0000 7f5629ffb700 1 -- 172.21.15.31:0/2983332330 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240048a0 con 0x7f562c101eb0 2023-07-29T16:17:24.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.722+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1a0900 con 0x7f562c101eb0 2023-07-29T16:17:24.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.725+0000 7f5629ffb700 1 -- 172.21.15.31:0/2983332330 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5624003a00 con 0x7f562c101eb0 2023-07-29T16:17:24.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.726+0000 7f5645f10b80 1 Processor -- start 2023-07-29T16:17:24.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.726+0000 7f5645f10b80 1 -- start start 2023-07-29T16:17:24.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.726+0000 7f5629ffb700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56180a0790 0x7f56180a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.726+0000 7f5629ffb700 1 -- 172.21.15.31:0/2983332330 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f56240cb310 con 0x7f562c101eb0 2023-07-29T16:17:24.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.726+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 0x56398f60a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.726+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f60ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60b370 0x56398f60b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5645f10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f561a90 con 0x56398f609c60 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5623fff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56180a0790 0x7f56180a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5645f10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f55f450 con 0x56398f60a5b0 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5645f10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f55ad40 con 0x56398f60b370 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 0x56398f60a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 0x56398f60a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42350/0 (socket says 172.21.15.31:42350) 2023-07-29T16:17:24.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 -- 172.21.15.31:0/3942699627 learned_addr learned my addr 172.21.15.31:0/3942699627 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:24.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5630b9c700 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60b370 0x56398f60b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5623fff700 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f60ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 -- 172.21.15.31:0/3942699627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60b370 msgr2=0x56398f60b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60b370 0x56398f60b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 -- 172.21.15.31:0/3942699627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 msgr2=0x56398f60ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f60ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f562bfff700 1 -- 172.21.15.31:0/3942699627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56398f388e60 con 0x56398f609c60 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5623fff700 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f60ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.727+0000 7f5623fff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56180a0790 0x7f56180a2c40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f561400b2e0 tx=0x7f5614005140 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.728+0000 7f562bfff700 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 0x56398f60a070 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f561c0115c0 tx=0x7f561c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8670752a6ef7c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.728+0000 7f56227fc700 1 -- 172.21.15.31:0/3942699627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c004220 con 0x56398f609c60 2023-07-29T16:17:24.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.728+0000 7f56227fc700 1 -- 172.21.15.31:0/3942699627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f561c004cf0 con 0x56398f609c60 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.728+0000 7f56227fc700 1 -- 172.21.15.31:0/3942699627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561c005650 con 0x56398f609c60 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 -- 172.21.15.31:0/3942699627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 msgr2=0x56398f60a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 0x56398f60a070 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f561c0115c0 tx=0x7f561c0118d0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 -- 172.21.15.31:0/3942699627 shutdown_connections 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60b370 0x56398f60b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f609c60 0x56398f60a070 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 --2- 172.21.15.31:0/3942699627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f60ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 -- 172.21.15.31:0/3942699627 >> 172.21.15.31:0/3942699627 conn(0x56398f5ff460 msgr2=0x56398f604a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 -- 172.21.15.31:0/3942699627 shutdown_connections 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 -- 172.21.15.31:0/3942699627 wait complete. 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.729+0000 7f5645f10b80 1 -- start start 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 Processor -- start 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f613890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f613dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f61ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f4f7590 con 0x56398f60b370 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f4f7230 con 0x56398f60a5b0 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f3de9a0 con 0x56398f609c60 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5645f10b80 10 client.0 _ll_register_callbacks cb 0x56398f5006a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5623fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f613dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5623fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f613dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58374/0 (socket says 172.21.15.31:58374) 2023-07-29T16:17:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5630b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f61ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.730+0000 7f5623fff700 1 -- 172.21.15.31:0/4198654883 learned_addr learned my addr 172.21.15.31:0/4198654883 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f562bfff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f613890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5623fff700 1 -- 172.21.15.31:0/4198654883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 msgr2=0x56398f613890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5623fff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f613890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5623fff700 1 -- 172.21.15.31:0/4198654883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 msgr2=0x56398f61ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5623fff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f61ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5623fff700 1 -- 172.21.15.31:0/4198654883 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561c009d50 con 0x56398f60a5b0 2023-07-29T16:17:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f562bfff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f613890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5630b9c700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f61ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f5623fff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f613dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5614015000 tx=0x7f5614009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56956c567da2f3ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.731+0000 7f560b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:24.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f560b7fe700 1 -- 172.21.15.31:0/4198654883 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5614004b20 con 0x56398f60a5b0 2023-07-29T16:17:24.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56398f585d90 con 0x56398f60a5b0 2023-07-29T16:17:24.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f560b7fe700 1 -- 172.21.15.31:0/4198654883 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5614004c80 con 0x56398f60a5b0 2023-07-29T16:17:24.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f560b7fe700 1 -- 172.21.15.31:0/4198654883 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5614004e40 con 0x56398f60a5b0 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f5645f10b80 10 client.26240 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56398f371ce0 con 0x56398f60a5b0 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f560b7fe700 1 -- 172.21.15.31:0/4198654883 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56398f585d90 con 0x56398f60a5b0 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.732+0000 7f560b7fe700 1 -- 172.21.15.31:0/4198654883 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f56140264c0 con 0x56398f60a5b0 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f5645f10b80 10 client.26240 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f5645f10b80 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:24.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f5645f10b80 20 client.26240 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f5645f10b80 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f61f620 0x56398f621a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f5645f10b80 4 client.26240 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56398f5d8df0 con 0x56398f61f620 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.733+0000 7f562bfff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f61f620 0x56398f621a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.734+0000 7f562bfff700 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f61f620 0x56398f621a30 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d33eb3f426e414 server_cookie=ba8638fe99f9c33b in_seq=0 out_seq=0 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.734+0000 7f560b7fe700 10 client.26240 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:24.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.734+0000 7f560b7fe700 1 -- 172.21.15.31:0/4198654883 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56398f5d8df0 con 0x56398f61f620 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.734+0000 7f560b7fe700 10 client.26240 handle_command_reply: tid=0 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 client.26240 shutdown 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 msgr2=0x56398f613dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f613dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5614015000 tx=0x7f5614009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 shutdown_connections 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f61f620 0x56398f621a30 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f61ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f613890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 --2- 172.21.15.31:0/4198654883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f613dd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f560b7fe700 0 client.26240 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 >> 172.21.15.31:0/4198654883 conn(0x56398f5ff460 msgr2=0x56398f6040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 shutdown_connections 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 1 -- 172.21.15.31:0/4198654883 wait complete. 2023-07-29T16:17:24.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.735+0000 7f5645f10b80 20 client.26240 trim_cache size 0 max 16384 2023-07-29T16:17:24.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 Processor -- start 2023-07-29T16:17:24.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 -- start start 2023-07-29T16:17:24.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f618d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f612520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f6117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f613460 con 0x56398f60b370 2023-07-29T16:17:24.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f6014e0 con 0x56398f60a5b0 2023-07-29T16:17:24.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5645f10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f6063a0 con 0x56398f609c60 2023-07-29T16:17:24.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.754+0000 7f5630b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f6117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f6117b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42358/0 (socket says 172.21.15.31:42358) 2023-07-29T16:17:24.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 -- 172.21.15.31:0/833542718 learned_addr learned my addr 172.21.15.31:0/833542718 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:24.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5623fff700 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f612520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f562bfff700 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f618d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 -- 172.21.15.31:0/833542718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 msgr2=0x56398f618d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f618d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 -- 172.21.15.31:0/833542718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 msgr2=0x56398f612520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f612520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 -- 172.21.15.31:0/833542718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56398f6043b0 con 0x56398f60b370 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5630b9c700 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f6117b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5624098940 tx=0x7f562409c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d239310d88a57e87 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5622ffd700 1 -- 172.21.15.31:0/833542718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240b1170 con 0x56398f60b370 2023-07-29T16:17:24.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5622ffd700 1 -- 172.21.15.31:0/833542718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56240b0070 con 0x56398f60b370 2023-07-29T16:17:24.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.755+0000 7f5622ffd700 1 -- 172.21.15.31:0/833542718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5624096e70 con 0x56398f60b370 2023-07-29T16:17:24.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 -- 172.21.15.31:0/833542718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 msgr2=0x56398f6117b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f6117b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5624098940 tx=0x7f562409c040 comp rx=0 tx=0).stop 2023-07-29T16:17:24.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 -- 172.21.15.31:0/833542718 shutdown_connections 2023-07-29T16:17:24.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f6117b0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f609c60 0x56398f618d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 --2- 172.21.15.31:0/833542718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f60a5b0 0x56398f612520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 -- 172.21.15.31:0/833542718 >> 172.21.15.31:0/833542718 conn(0x56398f5ff460 msgr2=0x56398f605710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 -- 172.21.15.31:0/833542718 shutdown_connections 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 -- 172.21.15.31:0/833542718 wait complete. 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.756+0000 7f5645f10b80 1 -- start start 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 Processor -- start 2023-07-29T16:17:24.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 0x56398f669a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60a5b0 0x56398f669f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f670fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f671500 con 0x56398f60b370 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f671640 con 0x56398f609c60 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56398f6717d0 con 0x56398f60a5b0 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5645f10b80 10 client.0 _ll_register_callbacks cb 0x56398f5006a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:24.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5630b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f670fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f562bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 0x56398f669a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f562bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 0x56398f669a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58396/0 (socket says 172.21.15.31:58396) 2023-07-29T16:17:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f562bfff700 1 -- 172.21.15.31:0/2571108782 learned_addr learned my addr 172.21.15.31:0/2571108782 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.757+0000 7f5623fff700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60a5b0 0x56398f669f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5630b9c700 1 -- 172.21.15.31:0/2571108782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60a5b0 msgr2=0x56398f669f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5630b9c700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60a5b0 0x56398f669f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5630b9c700 1 -- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 msgr2=0x56398f669a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5630b9c700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 0x56398f669a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5630b9c700 1 -- 172.21.15.31:0/2571108782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c1a0900 con 0x56398f60b370 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f562bfff700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 0x56398f669a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5623fff700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60a5b0 0x56398f669f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5630b9c700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f670fc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5624095420 tx=0x7f5624000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aeedc2be3b882c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f560b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f560b7fe700 1 -- 172.21.15.31:0/2571108782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240b1170 con 0x56398f60b370 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f560b7fe700 1 -- 172.21.15.31:0/2571108782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5624009410 con 0x56398f60b370 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f561c004cf0 con 0x56398f60b370 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f560b7fe700 1 -- 172.21.15.31:0/2571108782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56240a0e00 con 0x56398f60b370 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5645f10b80 10 client.16272 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.758+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56398f60ba10 con 0x56398f60b370 2023-07-29T16:17:24.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f560b7fe700 1 -- 172.21.15.31:0/2571108782 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f562407b020 con 0x56398f60b370 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f560b7fe700 1 -- 172.21.15.31:0/2571108782 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f562405ebc0 con 0x56398f60b370 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f5645f10b80 10 client.16272 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f5645f10b80 10 client.16272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f5645f10b80 20 client.16272 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f5645f10b80 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f672e30 0x56398f675210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f5645f10b80 4 client.16272 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:17:24.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56398f553dc0 con 0x56398f672e30 2023-07-29T16:17:24.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.759+0000 7f562bfff700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f672e30 0x56398f675210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.760+0000 7f562bfff700 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f672e30 0x56398f675210 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac85403d0e6686c9 server_cookie=f829aa0d0e139f72 in_seq=0 out_seq=0 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.760+0000 7f560b7fe700 10 client.16272 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f560b7fe700 1 -- 172.21.15.31:0/2571108782 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56398f553dc0 con 0x56398f672e30 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f560b7fe700 10 client.16272 handle_command_reply: tid=0 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 client.16272 shutdown 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 msgr2=0x56398f670fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f670fc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5624095420 tx=0x7f5624000f40 comp rx=0 tx=0).stop 2023-07-29T16:17:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 shutdown_connections 2023-07-29T16:17:24.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56398f672e30 0x56398f675210 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56398f60b370 0x56398f670fc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56398f609c60 0x56398f669a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 --2- 172.21.15.31:0/2571108782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56398f60a5b0 0x56398f669f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 >> 172.21.15.31:0/2571108782 conn(0x56398f5ff460 msgr2=0x56398f612520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.761+0000 7f560b7fe700 0 client.16272 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 shutdown_connections 2023-07-29T16:17:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f5645f10b80 1 -- 172.21.15.31:0/2571108782 wait complete. 2023-07-29T16:17:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f5645f10b80 20 client.16272 trim_cache size 0 max 16384 2023-07-29T16:17:24.769 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:17:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:17:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "60407637-59d1-4994-8559-f40d5441616f", 2023-07-29T16:17:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:17:24.770 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:17:24.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56180a0790 msgr2=0x7f56180a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56180a0790 0x7f56180a2c40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f561400b2e0 tx=0x7f5614005140 comp rx=0 tx=0).stop 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 msgr2=0x7f562c073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 0x7f562c073680 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5624007ee0 tx=0x7f562400d3b0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 shutdown_connections 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56180a0790 0x7f56180a2c40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f562c101eb0 0x7f562c073680 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f562c100330 0x7f562c074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 --2- 172.21.15.31:0/2983332330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f562c101520 0x7f562c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 >> 172.21.15.31:0/2983332330 conn(0x7f562c0fb970 msgr2=0x7f562c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.762+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 shutdown_connections 2023-07-29T16:17:24.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.763+0000 7f56325ff700 1 -- 172.21.15.31:0/2983332330 wait complete. 2023-07-29T16:17:24.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:24 smithi039 ceph-mon[128858]: pgmap v427: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:24.779 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:17:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:24 smithi156 ceph-mon[129245]: pgmap v427: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:24.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 -- 172.21.15.31:0/2793668244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 msgr2=0x7f97841019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2793668244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f97841019e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9774009990 tx=0x7f9774009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 -- 172.21.15.31:0/2793668244 shutdown_connections 2023-07-29T16:17:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2793668244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9784101f20 0x7f97841087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2793668244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97841002e0 0x7f97841006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2793668244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f97841019e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 -- 172.21.15.31:0/2793668244 >> 172.21.15.31:0/2793668244 conn(0x7f97840fb900 msgr2=0x7f97840fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 -- 172.21.15.31:0/2793668244 shutdown_connections 2023-07-29T16:17:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.990+0000 7f978b7ff700 1 -- 172.21.15.31:0/2793668244 wait complete. 2023-07-29T16:17:24.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 Processor -- start 2023-07-29T16:17:24.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 -- start start 2023-07-29T16:17:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 0x7f9784196000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f9784196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9784101f20 0x7f978419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978419db80 con 0x7f9784101570 2023-07-29T16:17:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978419dcc0 con 0x7f9784101f20 2023-07-29T16:17:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.992+0000 7f978b7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978419de50 con 0x7f97841002e0 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 0x7f9784196000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 0x7f9784196000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56976/0 (socket says 172.21.15.31:56976) 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 -- 172.21.15.31:0/2844160514 learned_addr learned my addr 172.21.15.31:0/2844160514 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f9788d9a700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f9784196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f9789d9c700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9784101f20 0x7f978419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 -- 172.21.15.31:0/2844160514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9784101f20 msgr2=0x7f978419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9784101f20 0x7f978419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 -- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 msgr2=0x7f9784196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f9784196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 -- 172.21.15.31:0/2844160514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97740095f0 con 0x7f97841002e0 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f9788d9a700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f9784196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.993+0000 7f978959b700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 0x7f9784196000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f978000e9a0 tx=0x7f978000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.994+0000 7f977a7fc700 1 -- 172.21.15.31:0/2844160514 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780009790 con 0x7f97841002e0 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.994+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978419e130 con 0x7f97841002e0 2023-07-29T16:17:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.994+0000 7f977a7fc700 1 -- 172.21.15.31:0/2844160514 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9780010430 con 0x7f97841002e0 2023-07-29T16:17:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.994+0000 7f977a7fc700 1 -- 172.21.15.31:0/2844160514 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780018680 con 0x7f97841002e0 2023-07-29T16:17:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.994+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978419e640 con 0x7f97841002e0 2023-07-29T16:17:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.997+0000 7f977a7fc700 1 -- 172.21.15.31:0/2844160514 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f97800105a0 con 0x7f97841002e0 2023-07-29T16:17:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.998+0000 7f979f13db80 1 Processor -- start 2023-07-29T16:17:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 -- start start 2023-07-29T16:17:25.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f977a7fc700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97700a0790 0x7f97700a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x559374464400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374464940 0x559374465170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374465700 0x559374465b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593743b7a10 con 0x559374465700 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593743b53d0 con 0x559374464940 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f979f13db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593743b0cc0 con 0x559374463ff0 2023-07-29T16:17:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f977a7fc700 1 -- 172.21.15.31:0/2844160514 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9780014070 con 0x7f97841002e0 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f978959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x559374464400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f978959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x559374464400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56978/0 (socket says 172.21.15.31:56978) 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f978959b700 1 -- 172.21.15.31:0/947485897 learned_addr learned my addr 172.21.15.31:0/947485897 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f9789d9c700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97700a0790 0x7f97700a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:24.999+0000 7f9788d9a700 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374464940 0x559374465170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f9789d9c700 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374465700 0x559374465b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f978959b700 1 -- 172.21.15.31:0/947485897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374464940 msgr2=0x559374465170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f978959b700 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374464940 0x559374465170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f978959b700 1 -- 172.21.15.31:0/947485897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374465700 msgr2=0x559374465b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f978959b700 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374465700 0x559374465b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f978959b700 1 -- 172.21.15.31:0/947485897 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593741dede0 con 0x559374463ff0 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f978959b700 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x559374464400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9780099410 tx=0x7f9780099040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de126cfb9f1be920 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f9789d9c700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97700a0790 0x7f97700a2c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f977c009550 tx=0x7f977c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f976f7fe700 1 -- 172.21.15.31:0/947485897 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800b2040 con 0x559374463ff0 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f976f7fe700 1 -- 172.21.15.31:0/947485897 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97800b1e60 con 0x559374463ff0 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.000+0000 7f976f7fe700 1 -- 172.21.15.31:0/947485897 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800a1960 con 0x559374463ff0 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 -- 172.21.15.31:0/947485897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 msgr2=0x559374464400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x559374464400 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9780099410 tx=0x7f9780099040 comp rx=0 tx=0).stop 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 -- 172.21.15.31:0/947485897 shutdown_connections 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374465700 0x559374465b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x559374464400 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 --2- 172.21.15.31:0/947485897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374464940 0x559374465170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 -- 172.21.15.31:0/947485897 >> 172.21.15.31:0/947485897 conn(0x5593744553e0 msgr2=0x55937445a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 -- 172.21.15.31:0/947485897 shutdown_connections 2023-07-29T16:17:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.001+0000 7f979f13db80 1 -- 172.21.15.31:0/947485897 wait complete. 2023-07-29T16:17:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 -- start start 2023-07-29T16:17:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 Processor -- start 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x55937446dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x55937446e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x5593744751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55937434d510 con 0x559374464940 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55937434d1b0 con 0x559374465700 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.002+0000 7f979f13db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559374234920 con 0x559374463ff0 2023-07-29T16:17:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f979f13db80 10 client.0 _ll_register_callbacks cb 0x559374356620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f9788d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x55937446e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x55937446dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f9788d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x55937446e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42400/0 (socket says 172.21.15.31:42400) 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f9788d9a700 1 -- 172.21.15.31:0/501178455 learned_addr learned my addr 172.21.15.31:0/501178455 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f9789d9c700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x5593744751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 -- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 msgr2=0x5593744751a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x5593744751a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 -- 172.21.15.31:0/501178455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 msgr2=0x55937446e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x55937446e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 -- 172.21.15.31:0/501178455 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978419e640 con 0x559374463ff0 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f9788d9a700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x55937446e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f978959b700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x55937446dc20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55937446c530 tx=0x7f97800b0590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddef0f6b7c54c1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.003+0000 7f976cff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f976cff9700 1 -- 172.21.15.31:0/501178455 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800b6070 con 0x559374463ff0 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f976cff9700 1 -- 172.21.15.31:0/501178455 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9780099480 con 0x559374463ff0 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593743dbd10 con 0x559374463ff0 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f976cff9700 1 -- 172.21.15.31:0/501178455 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97800a1940 con 0x559374463ff0 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 10 client.26266 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5593741c7c60 con 0x559374463ff0 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f976cff9700 1 -- 172.21.15.31:0/501178455 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9780063080 con 0x559374463ff0 2023-07-29T16:17:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f976cff9700 1 -- 172.21.15.31:0/501178455 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f97800989e0 con 0x559374463ff0 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 10 client.26266 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 10 client.26266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 20 client.26266 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744794b0 0x55937447b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 4 client.26266 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.004+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55937442ed70 con 0x5593744794b0 2023-07-29T16:17:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.005+0000 7f9788d9a700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744794b0 0x55937447b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.005+0000 7f9788d9a700 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744794b0 0x55937447b890 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2117ff475891c4c4 server_cookie=185271eab8208988 in_seq=0 out_seq=0 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.005+0000 7f976cff9700 10 client.26266 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f976cff9700 1 -- 172.21.15.31:0/501178455 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55937442ed70 con 0x5593744794b0 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f976cff9700 10 client.26266 handle_command_reply: tid=0 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f979f13db80 1 client.26266 shutdown 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 msgr2=0x55937446dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f979f13db80 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x55937446dc20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55937446c530 tx=0x7f97800b0590 comp rx=0 tx=0).stop 2023-07-29T16:17:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 shutdown_connections 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.006+0000 7f979f13db80 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744794b0 0x55937447b890 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x5593744751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x55937446dc20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 1 --2- 172.21.15.31:0/501178455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x55937446e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f976cff9700 0 client.26266 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 >> 172.21.15.31:0/501178455 conn(0x5593744553e0 msgr2=0x55937445a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 shutdown_connections 2023-07-29T16:17:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 1 -- 172.21.15.31:0/501178455 wait complete. 2023-07-29T16:17:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.007+0000 7f979f13db80 20 client.26266 trim_cache size 0 max 16384 2023-07-29T16:17:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 Processor -- start 2023-07-29T16:17:25.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 -- start start 2023-07-29T16:17:25.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 0x55937446d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374464940 0x559374475160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374465ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9784075050 con 0x559374463ff0 2023-07-29T16:17:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559374466040 con 0x559374465700 2023-07-29T16:17:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f979f13db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55937447bf40 con 0x559374464940 2023-07-29T16:17:25.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f978959b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 0x55937446d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f978959b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 0x55937446d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42406/0 (socket says 172.21.15.31:42406) 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.024+0000 7f978959b700 1 -- 172.21.15.31:0/2506190873 learned_addr learned my addr 172.21.15.31:0/2506190873 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f9789d9c700 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374465ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f9788d9a700 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374464940 0x559374475160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f978959b700 1 -- 172.21.15.31:0/2506190873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374464940 msgr2=0x559374475160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f978959b700 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374464940 0x559374475160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f978959b700 1 -- 172.21.15.31:0/2506190873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 msgr2=0x559374465ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f978959b700 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374465ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f978959b700 1 -- 172.21.15.31:0/2506190873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559374205250 con 0x559374463ff0 2023-07-29T16:17:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f9788d9a700 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374464940 0x559374475160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:25.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f978959b700 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 0x55937446d2d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f978000c2d0 tx=0x7f97800b0600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7f7809b548f493 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:25.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f976ffff700 1 -- 172.21.15.31:0/2506190873 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97800b6070 con 0x559374463ff0 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f976ffff700 1 -- 172.21.15.31:0/2506190873 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5593743dbd10 con 0x559374463ff0 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.025+0000 7f976ffff700 1 -- 172.21.15.31:0/2506190873 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780010880 con 0x559374463ff0 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 -- 172.21.15.31:0/2506190873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 msgr2=0x55937446d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 0x55937446d2d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f978000c2d0 tx=0x7f97800b0600 comp rx=0 tx=0).stop 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 -- 172.21.15.31:0/2506190873 shutdown_connections 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374465ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374463ff0 0x55937446d2d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 --2- 172.21.15.31:0/2506190873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374464940 0x559374475160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 -- 172.21.15.31:0/2506190873 >> 172.21.15.31:0/2506190873 conn(0x5593744553e0 msgr2=0x55937446c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 -- 172.21.15.31:0/2506190873 shutdown_connections 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 -- 172.21.15.31:0/2506190873 wait complete. 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.026+0000 7f979f13db80 1 -- start start 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 Processor -- start 2023-07-29T16:17:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x5593744712f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x559374471830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374474240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559374474780 con 0x559374464940 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593744748f0 con 0x559374465700 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.027+0000 7f979f13db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559374474ab0 con 0x559374463ff0 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f979f13db80 10 client.0 _ll_register_callbacks cb 0x559374356620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x559374471830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x559374471830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42418/0 (socket says 172.21.15.31:42418) 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 -- 172.21.15.31:0/3739492246 learned_addr learned my addr 172.21.15.31:0/3739492246 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f978959b700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x5593744712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9789d9c700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374474240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 -- 172.21.15.31:0/3739492246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 msgr2=0x5593744712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x5593744712f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 -- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 msgr2=0x559374474240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374474240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 -- 172.21.15.31:0/3739492246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593741c7c60 con 0x559374464940 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f9788d9a700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x559374471830 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f977400b680 tx=0x7f97740052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d11c5c71f58b20d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.028+0000 7f976cff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f976cff9700 1 -- 172.21.15.31:0/3739492246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977401d040 con 0x559374464940 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f97800b1e60 con 0x559374464940 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f976cff9700 1 -- 172.21.15.31:0/3739492246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f97740056a0 con 0x559374464940 2023-07-29T16:17:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f976cff9700 1 -- 172.21.15.31:0/3739492246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f977400f660 con 0x559374464940 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f979f13db80 10 client.16287 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5593741dd2b0 con 0x559374464940 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.029+0000 7f976cff9700 1 -- 172.21.15.31:0/3739492246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97800b1e60 con 0x559374464940 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f976cff9700 1 -- 172.21.15.31:0/3739492246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9774017970 con 0x559374464940 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f979f13db80 10 client.16287 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f979f13db80 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f979f13db80 20 client.16287 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:25.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f979f13db80 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744cbba0 0x5593744cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:25.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f979f13db80 4 client.16287 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:17:25.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5593743a9d40 con 0x5593744cbba0 2023-07-29T16:17:25.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f978959b700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744cbba0 0x5593744cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f978959b700 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744cbba0 0x5593744cdfb0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b982338940119e8 server_cookie=142606c64875df33 in_seq=0 out_seq=0 2023-07-29T16:17:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.030+0000 7f976cff9700 10 client.16287 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f976cff9700 1 -- 172.21.15.31:0/3739492246 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5593743a9d40 con 0x5593744cbba0 2023-07-29T16:17:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f976cff9700 10 client.16287 handle_command_reply: tid=0 2023-07-29T16:17:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 client.16287 shutdown 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 msgr2=0x559374471830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x559374471830 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f977400b680 tx=0x7f97740052e0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 shutdown_connections 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5593744cbba0 0x5593744cdfb0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559374465700 0x559374474240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559374463ff0 0x5593744712f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 --2- 172.21.15.31:0/3739492246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559374464940 0x559374471830 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 >> 172.21.15.31:0/3739492246 conn(0x5593744553e0 msgr2=0x55937446d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f976cff9700 0 client.16287 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 shutdown_connections 2023-07-29T16:17:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 1 -- 172.21.15.31:0/3739492246 wait complete. 2023-07-29T16:17:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.031+0000 7f979f13db80 20 client.16287 trim_cache size 0 max 16384 2023-07-29T16:17:25.038 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "60407637-59d1-4994-8559-f40d5441616f": { 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "60407637-59d1-4994-8559-f40d5441616f", 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:17:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:17:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:17:25.040 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:17:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97700a0790 msgr2=0x7f97700a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97700a0790 0x7f97700a2c40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f977c009550 tx=0x7f977c009920 comp rx=0 tx=0).stop 2023-07-29T16:17:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 msgr2=0x7f9784196000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 0x7f9784196000 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f978000e9a0 tx=0x7f978000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 shutdown_connections 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f97700a0790 0x7f97700a2c40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9784101f20 0x7f978419d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97841002e0 0x7f9784196000 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 --2- 172.21.15.31:0/2844160514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9784101570 0x7f9784196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 >> 172.21.15.31:0/2844160514 conn(0x7f97840fb900 msgr2=0x7f97840fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 shutdown_connections 2023-07-29T16:17:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:25.032+0000 7f978b7ff700 1 -- 172.21.15.31:0/2844160514 wait complete. 2023-07-29T16:17:25.049 INFO:tasks.cephfs.filesystem:scrub status for tag:60407637-59d1-4994-8559-f40d5441616f - {'path': '/', 'tag': '60407637-59d1-4994-8559-f40d5441616f', 'options': 'recursive,force'} 2023-07-29T16:17:26.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:26 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:17:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:17:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:26 smithi031 ceph-mon[95504]: pgmap v428: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:17:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:26 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:17:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:17:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:26 smithi039 ceph-mon[128858]: pgmap v428: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:17:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:26 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:17:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:17:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:26 smithi156 ceph-mon[129245]: pgmap v428: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-29T16:17:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:27 smithi031 ceph-mon[95504]: pgmap v429: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-07-29T16:17:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:27 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:17:28.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:27 smithi039 ceph-mon[128858]: pgmap v429: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-07-29T16:17:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:27 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:17:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:27 smithi156 ceph-mon[129245]: pgmap v429: 97 pgs: 97 active+clean; 670 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-07-29T16:17:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:27 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:17:30.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:30 smithi031 ceph-mon[95504]: pgmap v430: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:30 smithi039 ceph-mon[128858]: pgmap v430: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:30 smithi156 ceph-mon[129245]: pgmap v430: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:32.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:32 smithi031 ceph-mon[95504]: pgmap v431: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:32.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:32 smithi039 ceph-mon[128858]: pgmap v431: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:32 smithi156 ceph-mon[129245]: pgmap v431: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:34.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:34 smithi031 ceph-mon[95504]: pgmap v432: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:34 smithi039 ceph-mon[128858]: pgmap v432: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:34 smithi156 ceph-mon[129245]: pgmap v432: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:17:36.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:36 smithi031 ceph-mon[95504]: pgmap v433: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:36.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:36 smithi039 ceph-mon[128858]: pgmap v433: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:36 smithi156 ceph-mon[129245]: pgmap v433: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.077+0000 7f6f44fff700 1 -- 172.21.15.31:0/2878597563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 msgr2=0x7f6f40108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.077+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2878597563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40108790 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b0a0 tx=0x7f6f3800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 -- 172.21.15.31:0/2878597563 shutdown_connections 2023-07-29T16:17:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2878597563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40108790 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2878597563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f40100350 0x7f6f40100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2878597563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 -- 172.21.15.31:0/2878597563 >> 172.21.15.31:0/2878597563 conn(0x7f6f400fb970 msgr2=0x7f6f400fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 -- 172.21.15.31:0/2878597563 shutdown_connections 2023-07-29T16:17:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.078+0000 7f6f44fff700 1 -- 172.21.15.31:0/2878597563 wait complete. 2023-07-29T16:17:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.079+0000 7f6f44fff700 1 Processor -- start 2023-07-29T16:17:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.079+0000 7f6f44fff700 1 -- start start 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f44fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f40100350 0x7f6f40073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f44fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f44fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f44fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401a0210 con 0x7f6f40101ed0 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f44fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401a0350 con 0x7f6f40101540 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f44fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401a0490 con 0x7f6f40100350 2023-07-29T16:17:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f3dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f3ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f3dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47798/0 (socket says 172.21.15.31:47798) 2023-07-29T16:17:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/2084706915 learned_addr learned my addr 172.21.15.31:0/2084706915 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.080+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f40100350 0x7f6f40073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/2084706915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f40100350 msgr2=0x7f6f40073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f40100350 0x7f6f40073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 msgr2=0x7f6f40073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/2084706915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009d00 con 0x7f6f40101540 2023-07-29T16:17:37.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:37.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40073680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e9a0 tx=0x7f6f3400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:37.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f2f7fe700 1 -- 172.21.15.31:0/2084706915 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34009790 con 0x7f6f40101540 2023-07-29T16:17:37.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f401a0720 con 0x7f6f40101540 2023-07-29T16:17:37.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f2f7fe700 1 -- 172.21.15.31:0/2084706915 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f34010430 con 0x7f6f40101540 2023-07-29T16:17:37.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f2f7fe700 1 -- 172.21.15.31:0/2084706915 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34018680 con 0x7f6f40101540 2023-07-29T16:17:37.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.081+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f401a0c30 con 0x7f6f40101540 2023-07-29T16:17:37.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.085+0000 7f6f2f7fe700 1 -- 172.21.15.31:0/2084706915 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6f340105a0 con 0x7f6f40101540 2023-07-29T16:17:37.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.086+0000 7f6f58881b80 1 Processor -- start 2023-07-29T16:17:37.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.086+0000 7f6f58881b80 1 -- start start 2023-07-29T16:17:37.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f2f7fe700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0790 0x7f6f280a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbd17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f58881b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cb289b0 con 0x559e8cbd1cf0 2023-07-29T16:17:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f58881b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cb26370 con 0x559e8ca2c6b0 2023-07-29T16:17:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f58881b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cb21c60 con 0x559e8cbd0df0 2023-07-29T16:17:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f2f7fe700 1 -- 172.21.15.31:0/2084706915 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6f34014070 con 0x7f6f40101540 2023-07-29T16:17:37.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f3e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f3e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47810/0 (socket says 172.21.15.31:47810) 2023-07-29T16:17:37.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f3e59c700 1 -- 172.21.15.31:0/3607168948 learned_addr learned my addr 172.21.15.31:0/3607168948 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:37.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0790 0x7f6f280a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.087+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbd17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3e59c700 1 -- 172.21.15.31:0/3607168948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 msgr2=0x559e8cbd17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbd17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3e59c700 1 -- 172.21.15.31:0/3607168948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 msgr2=0x559e8cbd2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3e59c700 1 -- 172.21.15.31:0/3607168948 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e8c94fd80 con 0x559e8ca2c6b0 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd08b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009cf0 tx=0x7f6f30011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97b8ae675ca3c797 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:37.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0790 0x7f6f280a2c40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b780 tx=0x7f6f380074e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f2cff9700 1 -- 172.21.15.31:0/3607168948 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30011d70 con 0x559e8ca2c6b0 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f2cff9700 1 -- 172.21.15.31:0/3607168948 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f300044e0 con 0x559e8ca2c6b0 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.088+0000 7f6f2cff9700 1 -- 172.21.15.31:0/3607168948 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30005280 con 0x559e8ca2c6b0 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 -- 172.21.15.31:0/3607168948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 msgr2=0x559e8cbd08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd08b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009cf0 tx=0x7f6f30011440 comp rx=0 tx=0).stop 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 -- 172.21.15.31:0/3607168948 shutdown_connections 2023-07-29T16:17:37.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd08b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3607168948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbd17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 -- 172.21.15.31:0/3607168948 >> 172.21.15.31:0/3607168948 conn(0x559e8cbc5fd0 msgr2=0x559e8cbcb300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 -- 172.21.15.31:0/3607168948 shutdown_connections 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.089+0000 7f6f58881b80 1 -- 172.21.15.31:0/3607168948 wait complete. 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.090+0000 7f6f58881b80 1 -- start start 2023-07-29T16:17:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.090+0000 7f6f58881b80 1 Processor -- start 2023-07-29T16:17:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 0x559e8cbda3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbe1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cabe4b0 con 0x559e8cbd1cf0 2023-07-29T16:17:37.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cabe150 con 0x559e8cbd0df0 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8c9a58c0 con 0x559e8ca2c6b0 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f58881b80 10 client.0 _ll_register_callbacks cb 0x559e8cac75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f3dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 0x559e8cbda3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f3ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbe1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f3dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 0x559e8cbda3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47818/0 (socket says 172.21.15.31:47818) 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/3981006297 learned_addr learned my addr 172.21.15.31:0/3981006297 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:37.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.091+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/3981006297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8ca2c6b0 msgr2=0x559e8cbd9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/3981006297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 msgr2=0x559e8cbe1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbe1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/3981006297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f30009920 con 0x559e8cbd0df0 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 0x559e8cbda3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c2d0 tx=0x7f6f34014510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eb998c5dad5e997 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:37.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f25ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f25ffb700 1 -- 172.21.15.31:0/3981006297 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3401de00 con 0x559e8cbd0df0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f25ffb700 1 -- 172.21.15.31:0/3981006297 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6f340071c0 con 0x559e8cbd0df0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e8cb4ccb0 con 0x559e8cbd0df0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f25ffb700 1 -- 172.21.15.31:0/3981006297 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f34099480 con 0x559e8cbd0df0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f58881b80 10 client.26276 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.092+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559e8c938c00 con 0x559e8cbd0df0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f25ffb700 1 -- 172.21.15.31:0/3981006297 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e8cb4ccb0 con 0x559e8cbd0df0 2023-07-29T16:17:37.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f25ffb700 1 -- 172.21.15.31:0/3981006297 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f6f34014670 con 0x559e8cbd0df0 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f58881b80 10 client.26276 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f58881b80 10 client.26276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f58881b80 20 client.26276 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cbe5c10 0x559e8cbe8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f58881b80 4 client.26276 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e8cb9fd10 con 0x559e8cbe5c10 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.093+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cbe5c10 0x559e8cbe8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.094+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cbe5c10 0x559e8cbe8020 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a615629573a3b744 server_cookie=e843e14fdfeed182 in_seq=0 out_seq=0 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.094+0000 7f6f25ffb700 10 client.26276 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.094+0000 7f6f25ffb700 1 -- 172.21.15.31:0/3981006297 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559e8cb9fd10 con 0x559e8cbe5c10 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.094+0000 7f6f25ffb700 10 client.26276 handle_command_reply: tid=0 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 client.26276 shutdown 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 msgr2=0x559e8cbda3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 0x559e8cbda3f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c2d0 tx=0x7f6f34014510 comp rx=0 tx=0).stop 2023-07-29T16:17:37.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 shutdown_connections 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cbe5c10 0x559e8cbe8020 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd1cf0 0x559e8cbe1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8ca2c6b0 0x559e8cbd9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f25ffb700 0 client.26276 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 --2- 172.21.15.31:0/3981006297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd0df0 0x559e8cbda3f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 >> 172.21.15.31:0/3981006297 conn(0x559e8cbc5fd0 msgr2=0x559e8cbcaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 shutdown_connections 2023-07-29T16:17:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 1 -- 172.21.15.31:0/3981006297 wait complete. 2023-07-29T16:17:37.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.095+0000 7f6f58881b80 20 client.26276 trim_cache size 0 max 16384 2023-07-29T16:17:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.110+0000 7f6f58881b80 1 Processor -- start 2023-07-29T16:17:37.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 -- start start 2023-07-29T16:17:37.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbe7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 0x559e8cbdf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cbd2640 con 0x559e8cbd0df0 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cbe3120 con 0x559e8ca2c6b0 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f58881b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cbcc190 con 0x559e8cbd1cf0 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f3dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 0x559e8cbdf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f3dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 0x559e8cbdf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42844/0 (socket says 172.21.15.31:42844) 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/844997449 learned_addr learned my addr 172.21.15.31:0/844997449 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbe7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.111+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/844997449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd1cf0 msgr2=0x559e8cbd20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/844997449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 msgr2=0x559e8cbe7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbe7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3dd9b700 1 -- 172.21.15.31:0/844997449 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e8c9761f0 con 0x559e8cbd0df0 2023-07-29T16:17:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 0x559e8cbdf3e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6f340c9b90 tx=0x7f6f3401dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e950dd2fd2b55ca server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f2d7fa700 1 -- 172.21.15.31:0/844997449 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f340997d0 con 0x559e8cbd0df0 2023-07-29T16:17:37.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f2d7fa700 1 -- 172.21.15.31:0/844997449 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f34099480 con 0x559e8cbd0df0 2023-07-29T16:17:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.112+0000 7f6f2d7fa700 1 -- 172.21.15.31:0/844997449 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34014580 con 0x559e8cbd0df0 2023-07-29T16:17:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 -- 172.21.15.31:0/844997449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 msgr2=0x559e8cbdf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 0x559e8cbdf3e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f6f340c9b90 tx=0x7f6f3401dcb0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 -- 172.21.15.31:0/844997449 shutdown_connections 2023-07-29T16:17:37.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd1cf0 0x559e8cbd20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8ca2c6b0 0x559e8cbe7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 --2- 172.21.15.31:0/844997449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8cbd0df0 0x559e8cbdf3e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 -- 172.21.15.31:0/844997449 >> 172.21.15.31:0/844997449 conn(0x559e8cbc5fd0 msgr2=0x559e8cbd8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 -- 172.21.15.31:0/844997449 shutdown_connections 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 -- 172.21.15.31:0/844997449 wait complete. 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.113+0000 7f6f58881b80 1 -- start start 2023-07-29T16:17:37.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 Processor -- start 2023-07-29T16:17:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 0x559e8cbdd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbdd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd1cf0 0x559e8cbde6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cbdebe0 con 0x559e8ca2c6b0 2023-07-29T16:17:37.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cbded50 con 0x559e8cbd1cf0 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e8cbdef10 con 0x559e8cbd0df0 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f58881b80 10 client.0 _ll_register_callbacks cb 0x559e8cac75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f3e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 0x559e8cbdd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f3e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 0x559e8cbdd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42860/0 (socket says 172.21.15.31:42860) 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f3e59c700 1 -- 172.21.15.31:0/2468689090 learned_addr learned my addr 172.21.15.31:0/2468689090 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbdd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.114+0000 7f6f3ed9d700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd1cf0 0x559e8cbde6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3e59c700 1 -- 172.21.15.31:0/2468689090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 msgr2=0x559e8cbdd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbdd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3e59c700 1 -- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd1cf0 msgr2=0x559e8cbde6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd1cf0 0x559e8cbde6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3e59c700 1 -- 172.21.15.31:0/2468689090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e8c938c00 con 0x559e8ca2c6b0 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbdd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f3e59c700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 0x559e8cbdd150 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009cc0 tx=0x7f6f30009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a0308781271f72 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:37.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f25ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f25ffb700 1 -- 172.21.15.31:0/2468689090 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3002c130 con 0x559e8ca2c6b0 2023-07-29T16:17:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f300044e0 con 0x559e8ca2c6b0 2023-07-29T16:17:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.115+0000 7f6f25ffb700 1 -- 172.21.15.31:0/2468689090 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6f3001f040 con 0x559e8ca2c6b0 2023-07-29T16:17:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f25ffb700 1 -- 172.21.15.31:0/2468689090 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f3002c290 con 0x559e8ca2c6b0 2023-07-29T16:17:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 10 client.16305 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:37.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559e8c94e250 con 0x559e8ca2c6b0 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f25ffb700 1 -- 172.21.15.31:0/2468689090 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f300044e0 con 0x559e8ca2c6b0 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f25ffb700 1 -- 172.21.15.31:0/2468689090 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f6f3003c070 con 0x559e8ca2c6b0 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 10 client.16305 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 10 client.16305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 20 client.16305 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cc38600 0x559e8cc3aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:37.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 4 client.16305 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:17:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.116+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559e8cb1ace0 con 0x559e8cc38600 2023-07-29T16:17:37.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.117+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cc38600 0x559e8cc3aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.117+0000 7f6f25ffb700 10 client.16305 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.117+0000 7f6f3dd9b700 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cc38600 0x559e8cc3aa10 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba7ab646b2c1d1ff server_cookie=908e27278722e7bd in_seq=0 out_seq=0 2023-07-29T16:17:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f25ffb700 1 -- 172.21.15.31:0/2468689090 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559e8cb1ace0 con 0x559e8cc38600 2023-07-29T16:17:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f25ffb700 10 client.16305 handle_command_reply: tid=0 2023-07-29T16:17:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 client.16305 shutdown 2023-07-29T16:17:37.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 msgr2=0x559e8cbdd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 0x559e8cbdd150 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009cc0 tx=0x7f6f30009710 comp rx=0 tx=0).stop 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 shutdown_connections 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e8cc38600 0x559e8cc3aa10 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e8cbd1cf0 0x559e8cbde6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e8ca2c6b0 0x559e8cbdd150 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f25ffb700 0 client.16305 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 --2- 172.21.15.31:0/2468689090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8cbd0df0 0x559e8cbdd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 >> 172.21.15.31:0/2468689090 conn(0x559e8cbc5fd0 msgr2=0x559e8cbd8050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 shutdown_connections 2023-07-29T16:17:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.121+0000 7f6f58881b80 1 -- 172.21.15.31:0/2468689090 wait complete. 2023-07-29T16:17:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f58881b80 20 client.16305 trim_cache size 0 max 16384 2023-07-29T16:17:37.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:17:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:17:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:17:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:17:37.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0790 msgr2=0x7f6f280a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0790 0x7f6f280a2c40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b780 tx=0x7f6f380074e0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 msgr2=0x7f6f40073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40073680 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400e9a0 tx=0x7f6f3400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 shutdown_connections 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f280a0790 0x7f6f280a2c40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f40101ed0 0x7f6f40073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f40100350 0x7f6f40073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 --2- 172.21.15.31:0/2084706915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f40101540 0x7f6f40073680 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 >> 172.21.15.31:0/2084706915 conn(0x7f6f400fb970 msgr2=0x7f6f400fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.122+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 shutdown_connections 2023-07-29T16:17:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:37.123+0000 7f6f44fff700 1 -- 172.21.15.31:0/2084706915 wait complete. 2023-07-29T16:17:38.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:37 smithi031 ceph-mon[95504]: pgmap v434: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:38.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:37 smithi039 ceph-mon[128858]: pgmap v434: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:37 smithi156 ceph-mon[129245]: pgmap v434: 97 pgs: 97 active+clean; 671 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:17:39.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:38 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:17:39.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:38 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:17:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:38 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:17:40.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:39 smithi031 ceph-mon[95504]: pgmap v435: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-29T16:17:40.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:17:40.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:17:40.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:39 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:17:40.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:39 smithi039 ceph-mon[128858]: pgmap v435: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-29T16:17:40.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:17:40.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:17:40.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:39 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:17:40.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:39 smithi156 ceph-mon[129245]: pgmap v435: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-29T16:17:40.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:17:40.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:17:40.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:39 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:17:42.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:42 smithi031 ceph-mon[95504]: pgmap v436: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:42.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:42 smithi039 ceph-mon[128858]: pgmap v436: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:42 smithi156 ceph-mon[129245]: pgmap v436: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:44.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:44 smithi031 ceph-mon[95504]: pgmap v437: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:44.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:44 smithi039 ceph-mon[128858]: pgmap v437: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:44 smithi156 ceph-mon[129245]: pgmap v437: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:46.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:46 smithi031 ceph-mon[95504]: pgmap v438: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:46.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:46 smithi039 ceph-mon[128858]: pgmap v438: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:46 smithi156 ceph-mon[129245]: pgmap v438: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:47 smithi031 ceph-mon[95504]: pgmap v439: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:48.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:47 smithi039 ceph-mon[128858]: pgmap v439: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:48.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:47 smithi156 ceph-mon[129245]: pgmap v439: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:50.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 -- 172.21.15.31:0/544105177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4101530 msgr2=0x7feaa4101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/544105177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4101530 0x7feaa4101980 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fea98009990 tx=0x7fea98009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 -- 172.21.15.31:0/544105177 shutdown_connections 2023-07-29T16:17:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/544105177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101ec0 0x7feaa4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/544105177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4100340 0x7feaa4100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/544105177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4101530 0x7feaa4101980 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 -- 172.21.15.31:0/544105177 >> 172.21.15.31:0/544105177 conn(0x7feaa40fb920 msgr2=0x7feaa40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.361+0000 7feaaa3ff700 1 -- 172.21.15.31:0/544105177 shutdown_connections 2023-07-29T16:17:50.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.362+0000 7feaaa3ff700 1 -- 172.21.15.31:0/544105177 wait complete. 2023-07-29T16:17:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.363+0000 7feaaa3ff700 1 Processor -- start 2023-07-29T16:17:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.363+0000 7feaaa3ff700 1 -- start start 2023-07-29T16:17:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaaa3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4100340 0x7feaa4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaaa3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101530 0x7feaa4198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaaa3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4101ec0 0x7feaa4199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaaa3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4199e70 con 0x7feaa4100340 2023-07-29T16:17:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaaa3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4199fe0 con 0x7feaa4101ec0 2023-07-29T16:17:50.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaaa3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa419a1a0 con 0x7feaa4101530 2023-07-29T16:17:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.364+0000 7feaa899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4101ec0 0x7feaa4199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101530 0x7feaa4198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4101ec0 0x7feaa4199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42680/0 (socket says 172.21.15.31:42680) 2023-07-29T16:17:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa899c700 1 -- 172.21.15.31:0/2040135988 learned_addr learned my addr 172.21.15.31:0/2040135988 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa3fff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4100340 0x7feaa4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 -- 172.21.15.31:0/2040135988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4101ec0 msgr2=0x7feaa4199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4101ec0 0x7feaa4199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 -- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4100340 msgr2=0x7feaa4198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4100340 0x7feaa4198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 -- 172.21.15.31:0/2040135988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea980095f0 con 0x7feaa4101530 2023-07-29T16:17:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.365+0000 7feaa37fe700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101530 0x7feaa4198990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fea9800b5b0 tx=0x7fea98005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.366+0000 7feaa17fa700 1 -- 172.21.15.31:0/2040135988 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9801d070 con 0x7feaa4101530 2023-07-29T16:17:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.366+0000 7feaa17fa700 1 -- 172.21.15.31:0/2040135988 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea9800f460 con 0x7feaa4101530 2023-07-29T16:17:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.366+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa419a420 con 0x7feaa4101530 2023-07-29T16:17:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.366+0000 7feaa17fa700 1 -- 172.21.15.31:0/2040135988 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea98021670 con 0x7feaa4101530 2023-07-29T16:17:50.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.366+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa41a0c40 con 0x7feaa4101530 2023-07-29T16:17:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.372+0000 7feaa17fa700 1 -- 172.21.15.31:0/2040135988 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fea9800f840 con 0x7feaa4101530 2023-07-29T16:17:50.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.373+0000 7feabdc80b80 1 Processor -- start 2023-07-29T16:17:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.373+0000 7feabdc80b80 1 -- start start 2023-07-29T16:17:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 0x558740b188b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b197b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b1a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feabdc80b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740a709b0 con 0x558740b19cf0 2023-07-29T16:17:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feabdc80b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740a6e370 con 0x5587409746b0 2023-07-29T16:17:50.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feabdc80b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740a69c60 con 0x558740b18df0 2023-07-29T16:17:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feaa899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 0x558740b188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feaa899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 0x558740b188b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42690/0 (socket says 172.21.15.31:42690) 2023-07-29T16:17:50.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feaa899c700 1 -- 172.21.15.31:0/3398349401 learned_addr learned my addr 172.21.15.31:0/3398349401 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feaa3fff700 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b197b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.374+0000 7feaa37fe700 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b1a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa17fa700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fea8c0a0810 0x7fea8c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 -- 172.21.15.31:0/3398349401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 msgr2=0x558740b197b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b197b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 -- 172.21.15.31:0/3398349401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 msgr2=0x558740b1a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b1a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 -- 172.21.15.31:0/3398349401 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558740897d80 con 0x5587409746b0 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa37fe700 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b1a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa17fa700 1 -- 172.21.15.31:0/2040135988 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fea980cce90 con 0x7feaa4101530 2023-07-29T16:17:50.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fea8c0a0810 0x7fea8c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.375+0000 7feaa899c700 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 0x558740b188b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e690 tx=0x7fea9c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef5b42b8e38a812 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.376+0000 7fea8a7fc700 1 -- 172.21.15.31:0/3398349401 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c00e040 con 0x5587409746b0 2023-07-29T16:17:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.376+0000 7fea8a7fc700 1 -- 172.21.15.31:0/3398349401 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea9c00f040 con 0x5587409746b0 2023-07-29T16:17:50.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.376+0000 7fea8a7fc700 1 -- 172.21.15.31:0/3398349401 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0159f0 con 0x5587409746b0 2023-07-29T16:17:50.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 -- 172.21.15.31:0/3398349401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 msgr2=0x558740b188b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 0x558740b188b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e690 tx=0x7fea9c009920 comp rx=0 tx=0).stop 2023-07-29T16:17:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feaa899c700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fea8c0a0810 0x7fea8c0a2cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fea9c020000 tx=0x7fea9c023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 -- 172.21.15.31:0/3398349401 shutdown_connections 2023-07-29T16:17:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b1a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587409746b0 0x558740b188b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3398349401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b197b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 -- 172.21.15.31:0/3398349401 >> 172.21.15.31:0/3398349401 conn(0x558740b0dfd0 msgr2=0x558740b13300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 -- 172.21.15.31:0/3398349401 shutdown_connections 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.377+0000 7feabdc80b80 1 -- 172.21.15.31:0/3398349401 wait complete. 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.378+0000 7feabdc80b80 1 -- start start 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.378+0000 7feabdc80b80 1 Processor -- start 2023-07-29T16:17:50.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b2a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b2ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b31c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740a064b0 con 0x5587409746b0 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740a06150 con 0x558740b19cf0 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587408ed8c0 con 0x558740b18df0 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.379+0000 7feabdc80b80 10 client.0 _ll_register_callbacks cb 0x558740a0f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:50.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b2ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b31c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b2ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40066/0 (socket says 172.21.15.31:40066) 2023-07-29T16:17:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa37fe700 1 -- 172.21.15.31:0/2209890806 learned_addr learned my addr 172.21.15.31:0/2209890806 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:50.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa3fff700 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b2a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa899c700 1 -- 172.21.15.31:0/2209890806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 msgr2=0x558740b2ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa899c700 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b2ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa899c700 1 -- 172.21.15.31:0/2209890806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 msgr2=0x558740b2a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa899c700 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b2a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.380+0000 7feaa899c700 1 -- 172.21.15.31:0/2209890806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9c00e370 con 0x558740b19cf0 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.381+0000 7feaa899c700 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b31c40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e690 tx=0x7fea9c014da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b0c6d88ede1449 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.381+0000 7fea837fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:50.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.381+0000 7fea837fe700 1 -- 172.21.15.31:0/2209890806 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c00e100 con 0x558740b19cf0 2023-07-29T16:17:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.381+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558740a94cb0 con 0x558740b19cf0 2023-07-29T16:17:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.381+0000 7fea837fe700 1 -- 172.21.15.31:0/2209890806 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fea9c0157f0 con 0x558740b19cf0 2023-07-29T16:17:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.381+0000 7fea837fe700 1 -- 172.21.15.31:0/2209890806 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea9c0159b0 con 0x558740b19cf0 2023-07-29T16:17:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.382+0000 7feabdc80b80 10 client.26291 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.382+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558740880c00 con 0x558740b19cf0 2023-07-29T16:17:50.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.382+0000 7fea837fe700 1 -- 172.21.15.31:0/2209890806 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558740a94cb0 con 0x558740b19cf0 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7fea837fe700 1 -- 172.21.15.31:0/2209890806 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fea9c015c60 con 0x558740b19cf0 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7feabdc80b80 10 client.26291 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7feabdc80b80 10 client.26291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7feabdc80b80 20 client.26291 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7feabdc80b80 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b361f0 0x558740b38600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7feabdc80b80 4 client.26291 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:50.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.383+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558740ae7d10 con 0x558740b361f0 2023-07-29T16:17:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.384+0000 7feaa3fff700 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b361f0 0x558740b38600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.384+0000 7feaa3fff700 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b361f0 0x558740b38600 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=913fdba1a9db81b7 server_cookie=17c72343469c160e in_seq=0 out_seq=0 2023-07-29T16:17:50.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.384+0000 7fea837fe700 10 client.26291 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.385+0000 7fea837fe700 1 -- 172.21.15.31:0/2209890806 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558740ae7d10 con 0x558740b361f0 2023-07-29T16:17:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.385+0000 7fea837fe700 10 client.26291 handle_command_reply: tid=0 2023-07-29T16:17:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.385+0000 7feabdc80b80 1 client.26291 shutdown 2023-07-29T16:17:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 msgr2=0x558740b31c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b31c40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e690 tx=0x7fea9c014da0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 shutdown_connections 2023-07-29T16:17:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b361f0 0x558740b38600 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b31c40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b2a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7fea837fe700 0 client.26291 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 --2- 172.21.15.31:0/2209890806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b2ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 >> 172.21.15.31:0/2209890806 conn(0x558740b0dfd0 msgr2=0x558740b12a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 shutdown_connections 2023-07-29T16:17:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 1 -- 172.21.15.31:0/2209890806 wait complete. 2023-07-29T16:17:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.386+0000 7feabdc80b80 20 client.26291 trim_cache size 0 max 16384 2023-07-29T16:17:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 Processor -- start 2023-07-29T16:17:50.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 -- start start 2023-07-29T16:17:50.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b1a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b31c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b13170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740b35df0 con 0x5587409746b0 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740b29b40 con 0x558740b19cf0 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.416+0000 7feabdc80b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740b1a600 con 0x558740b18df0 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b13170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b1a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b1a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37524/0 (socket says 172.21.15.31:37524) 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa3fff700 1 -- 172.21.15.31:0/3218151502 learned_addr learned my addr 172.21.15.31:0/3218151502 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa899c700 1 -- 172.21.15.31:0/3218151502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 msgr2=0x558740b31c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa899c700 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b31c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa899c700 1 -- 172.21.15.31:0/3218151502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 msgr2=0x558740b1a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa899c700 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b1a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa899c700 1 -- 172.21.15.31:0/3218151502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587408be1f0 con 0x558740b19cf0 2023-07-29T16:17:50.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.417+0000 7feaa3fff700 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b1a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:50.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.418+0000 7feaa899c700 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b13170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fea9c03a040 tx=0x7fea9c0154d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d91f19752d07ebab server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:50.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.420+0000 7fea8affd700 1 -- 172.21.15.31:0/3218151502 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c014e60 con 0x558740b19cf0 2023-07-29T16:17:50.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.420+0000 7fea8affd700 1 -- 172.21.15.31:0/3218151502 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea9c00e100 con 0x558740b19cf0 2023-07-29T16:17:50.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.420+0000 7fea8affd700 1 -- 172.21.15.31:0/3218151502 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c015940 con 0x558740b19cf0 2023-07-29T16:17:50.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 -- 172.21.15.31:0/3218151502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 msgr2=0x558740b13170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b13170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fea9c03a040 tx=0x7fea9c0154d0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 -- 172.21.15.31:0/3218151502 shutdown_connections 2023-07-29T16:17:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b19cf0 0x558740b13170 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5587409746b0 0x558740b1a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 --2- 172.21.15.31:0/3218151502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558740b18df0 0x558740b31c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 -- 172.21.15.31:0/3218151502 >> 172.21.15.31:0/3218151502 conn(0x558740b0dfd0 msgr2=0x558740b12a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 -- 172.21.15.31:0/3218151502 shutdown_connections 2023-07-29T16:17:50.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.421+0000 7feabdc80b80 1 -- 172.21.15.31:0/3218151502 wait complete. 2023-07-29T16:17:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.422+0000 7feabdc80b80 1 -- start start 2023-07-29T16:17:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.422+0000 7feabdc80b80 1 Processor -- start 2023-07-29T16:17:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 0x558740b316d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b18df0 0x558740b2e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b2e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740b2ef00 con 0x558740b19cf0 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740b31d80 con 0x558740b18df0 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558740b31f40 con 0x5587409746b0 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feabdc80b80 10 client.0 _ll_register_callbacks cb 0x558740a0f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 0x558740b316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 0x558740b316d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40070/0 (socket says 172.21.15.31:40070) 2023-07-29T16:17:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feaa3fff700 1 -- 172.21.15.31:0/540957278 learned_addr learned my addr 172.21.15.31:0/540957278 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:50.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.423+0000 7feaa37fe700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b18df0 0x558740b2e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa3fff700 1 -- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b18df0 msgr2=0x558740b2e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa899c700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b2e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa3fff700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b18df0 0x558740b2e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa3fff700 1 -- 172.21.15.31:0/540957278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 msgr2=0x558740b2e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa3fff700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b2e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa3fff700 1 -- 172.21.15.31:0/540957278 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9400e040 con 0x5587409746b0 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa37fe700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b18df0 0x558740b2e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa899c700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b2e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.424+0000 7feaa3fff700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 0x558740b316d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x558740b38ea0 tx=0x7fea94009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9536782b013f2207 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7fea837fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:50.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7fea837fe700 1 -- 172.21.15.31:0/540957278 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9401f070 con 0x5587409746b0 2023-07-29T16:17:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7fea837fe700 1 -- 172.21.15.31:0/540957278 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fea94015ab0 con 0x5587409746b0 2023-07-29T16:17:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558740a62ce0 con 0x5587409746b0 2023-07-29T16:17:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7fea837fe700 1 -- 172.21.15.31:0/540957278 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea94015c70 con 0x5587409746b0 2023-07-29T16:17:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7feabdc80b80 10 client.26302 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.425+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x558740880c00 con 0x5587409746b0 2023-07-29T16:17:50.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7fea837fe700 1 -- 172.21.15.31:0/540957278 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558740a62ce0 con 0x5587409746b0 2023-07-29T16:17:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7fea837fe700 1 -- 172.21.15.31:0/540957278 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fea9402b070 con 0x5587409746b0 2023-07-29T16:17:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7feabdc80b80 10 client.26302 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7feabdc80b80 10 client.26302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7feabdc80b80 20 client.26302 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:50.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7feabdc80b80 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b264c0 0x558740b288d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7feabdc80b80 4 client.26302 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:17:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.426+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558740a62a00 con 0x558740b264c0 2023-07-29T16:17:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.427+0000 7feaa37fe700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b264c0 0x558740b288d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:50.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.427+0000 7feaa37fe700 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b264c0 0x558740b288d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89324f1c6708cc1 server_cookie=8a048b9238be6751 in_seq=0 out_seq=0 2023-07-29T16:17:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.427+0000 7fea837fe700 10 client.26302 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.431+0000 7fea837fe700 1 -- 172.21.15.31:0/540957278 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558740a62a00 con 0x558740b264c0 2023-07-29T16:17:50.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.431+0000 7fea837fe700 10 client.26302 handle_command_reply: tid=0 2023-07-29T16:17:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.431+0000 7feabdc80b80 1 client.26302 shutdown 2023-07-29T16:17:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 msgr2=0x558740b316d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 0x558740b316d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x558740b38ea0 tx=0x7fea94009e40 comp rx=0 tx=0).stop 2023-07-29T16:17:50.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 shutdown_connections 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558740b264c0 0x558740b288d0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558740b19cf0 0x558740b2e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5587409746b0 0x558740b316d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7fea837fe700 0 client.26302 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 --2- 172.21.15.31:0/540957278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558740b18df0 0x558740b2e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 >> 172.21.15.31:0/540957278 conn(0x558740b0dfd0 msgr2=0x558740b1ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:50.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 shutdown_connections 2023-07-29T16:17:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.432+0000 7feabdc80b80 1 -- 172.21.15.31:0/540957278 wait complete. 2023-07-29T16:17:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.433+0000 7feabdc80b80 20 client.26302 trim_cache size 0 max 16384 2023-07-29T16:17:50.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:17:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:17:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:17:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:17:50.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fea8c0a0810 msgr2=0x7fea8c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fea8c0a0810 0x7fea8c0a2cc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fea9c020000 tx=0x7fea9c023000 comp rx=0 tx=0).stop 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101530 msgr2=0x7feaa4198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101530 0x7feaa4198990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fea9800b5b0 tx=0x7fea98005740 comp rx=0 tx=0).stop 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 shutdown_connections 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fea8c0a0810 0x7fea8c0a2cc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feaa4101ec0 0x7feaa4199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feaa4100340 0x7feaa4198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 --2- 172.21.15.31:0/2040135988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feaa4101530 0x7feaa4198990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 >> 172.21.15.31:0/2040135988 conn(0x7feaa40fb920 msgr2=0x7feaa4106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 shutdown_connections 2023-07-29T16:17:50.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:17:50.434+0000 7feaaa3ff700 1 -- 172.21.15.31:0/2040135988 wait complete. 2023-07-29T16:17:50.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:50 smithi031 ceph-mon[95504]: pgmap v440: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:50 smithi039 ceph-mon[128858]: pgmap v440: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:50 smithi156 ceph-mon[129245]: pgmap v440: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:52.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:52 smithi031 ceph-mon[95504]: pgmap v441: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:17:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:52 smithi039 ceph-mon[128858]: pgmap v441: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:17:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:52 smithi156 ceph-mon[129245]: pgmap v441: 97 pgs: 97 active+clean; 672 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-07-29T16:17:54.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:54 smithi031 ceph-mon[95504]: pgmap v442: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:54 smithi039 ceph-mon[128858]: pgmap v442: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:54 smithi156 ceph-mon[129245]: pgmap v442: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2023-07-29T16:17:55.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:17:55.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 -- 172.21.15.31:0/32381608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208101510 msgr2=0x7f6208101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 --2- 172.21.15.31:0/32381608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208101510 0x7f6208101960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f61f8009990 tx=0x7f61f8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 -- 172.21.15.31:0/32381608 shutdown_connections 2023-07-29T16:17:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 --2- 172.21.15.31:0/32381608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 0x7f6208108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 --2- 172.21.15.31:0/32381608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6208100320 0x7f6208100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 --2- 172.21.15.31:0/32381608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208101510 0x7f6208101960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 -- 172.21.15.31:0/32381608 >> 172.21.15.31:0/32381608 conn(0x7f62080fb920 msgr2=0x7f62080fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 -- 172.21.15.31:0/32381608 shutdown_connections 2023-07-29T16:17:55.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.401+0000 7f620cfff700 1 -- 172.21.15.31:0/32381608 wait complete. 2023-07-29T16:17:55.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.403+0000 7f620cfff700 1 Processor -- start 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.403+0000 7f620cfff700 1 -- start start 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.403+0000 7f620cfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 0x7f6208193e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.403+0000 7f620cfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6208101510 0x7f62081993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.403+0000 7f620cfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 0x7f62081943f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620cfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620806a770 con 0x7f6208101510 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620cfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620806a8e0 con 0x7f6208101ea0 2023-07-29T16:17:55.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620cfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620806aaa0 con 0x7f6208100320 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 0x7f6208193e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 0x7f6208193e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40090/0 (socket says 172.21.15.31:40090) 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620659c700 1 -- 172.21.15.31:0/3270969958 learned_addr learned my addr 172.21.15.31:0/3270969958 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6206d9d700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 0x7f62081943f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6205d9b700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6208101510 0x7f62081993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6206d9d700 1 -- 172.21.15.31:0/3270969958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 msgr2=0x7f6208193e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6206d9d700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 0x7f6208193e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6206d9d700 1 -- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6208101510 msgr2=0x7f62081993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6206d9d700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6208101510 0x7f62081993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f6206d9d700 1 -- 172.21.15.31:0/3270969958 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f80095f0 con 0x7f6208101ea0 2023-07-29T16:17:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.404+0000 7f620659c700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 0x7f6208193e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:55.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.405+0000 7f6206d9d700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 0x7f62081943f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f620000efc0 tx=0x7f620000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.405+0000 7f61ff7fe700 1 -- 172.21.15.31:0/3270969958 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6200018070 con 0x7f6208101ea0 2023-07-29T16:17:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.405+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620806ad20 con 0x7f6208101ea0 2023-07-29T16:17:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.405+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620806b280 con 0x7f6208101ea0 2023-07-29T16:17:55.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.406+0000 7f61ff7fe700 1 -- 172.21.15.31:0/3270969958 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f620000f040 con 0x7f6208101ea0 2023-07-29T16:17:55.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.406+0000 7f61ff7fe700 1 -- 172.21.15.31:0/3270969958 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000cdf0 con 0x7f6208101ea0 2023-07-29T16:17:55.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.410+0000 7f61ff7fe700 1 -- 172.21.15.31:0/3270969958 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6200003a00 con 0x7f6208101ea0 2023-07-29T16:17:55.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.412+0000 7f622096cb80 1 Processor -- start 2023-07-29T16:17:55.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.412+0000 7f622096cb80 1 -- start start 2023-07-29T16:17:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f622096cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bcaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f622096cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bcbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f622096cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bcc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f622096cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0b22a10 con 0x55ccd0bcb530 2023-07-29T16:17:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f622096cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0b203d0 con 0x55ccd0bcc2f0 2023-07-29T16:17:55.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f622096cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0b1bcc0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f61ff7fe700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61f40a0810 0x7f61f40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bcaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bcaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40106/0 (socket says 172.21.15.31:40106) 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f6205d9b700 1 -- 172.21.15.31:0/964995350 learned_addr learned my addr 172.21.15.31:0/964995350 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.413+0000 7f6205d9b700 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bcc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f620659c700 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bcbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6205d9b700 1 -- 172.21.15.31:0/964995350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcc2f0 msgr2=0x55ccd0bcc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6205d9b700 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bcc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6205d9b700 1 -- 172.21.15.31:0/964995350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 msgr2=0x55ccd0bcbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6205d9b700 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bcbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6205d9b700 1 -- 172.21.15.31:0/964995350 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccd0949de0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6205d9b700 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bcaff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f61f800b2e0 tx=0x7f61f801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e02d5eff84bdec71 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f61ff7fe700 1 -- 172.21.15.31:0/3270969958 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f62000cb5e0 con 0x7f6208101ea0 2023-07-29T16:17:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f6206d9d700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61f40a0810 0x7f61f40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.414+0000 7f61fcff9700 1 -- 172.21.15.31:0/964995350 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f800be40 con 0x55ccd0bcabe0 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.415+0000 7f61fcff9700 1 -- 172.21.15.31:0/964995350 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f801e070 con 0x55ccd0bcabe0 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.415+0000 7f61fcff9700 1 -- 172.21.15.31:0/964995350 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f8004d90 con 0x55ccd0bcabe0 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.415+0000 7f622096cb80 1 -- 172.21.15.31:0/964995350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 msgr2=0x55ccd0bcaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.415+0000 7f6206d9d700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61f40a0810 0x7f61f40a2cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6200098a10 tx=0x7f6200009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.415+0000 7f622096cb80 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bcaff0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f61f800b2e0 tx=0x7f61f801a040 comp rx=0 tx=0).stop 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 -- 172.21.15.31:0/964995350 shutdown_connections 2023-07-29T16:17:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bcc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bcaff0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 --2- 172.21.15.31:0/964995350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bcbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 -- 172.21.15.31:0/964995350 >> 172.21.15.31:0/964995350 conn(0x55ccd0bc03e0 msgr2=0x55ccd0bc5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 -- 172.21.15.31:0/964995350 shutdown_connections 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.416+0000 7f622096cb80 1 -- 172.21.15.31:0/964995350 wait complete. 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.417+0000 7f622096cb80 1 -- start start 2023-07-29T16:17:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.417+0000 7f622096cb80 1 Processor -- start 2023-07-29T16:17:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bdbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0ab8510 con 0x55ccd0bcc2f0 2023-07-29T16:17:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0ab81b0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd099f920 con 0x55ccd0bcb530 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f622096cb80 10 client.0 _ll_register_callbacks cb 0x55ccd0ac1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42750/0 (socket says 172.21.15.31:42750) 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f6205d9b700 1 -- 172.21.15.31:0/41616139 learned_addr learned my addr 172.21.15.31:0/41616139 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f6205d9b700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bdbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.418+0000 7f620659c700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f6205d9b700 1 -- 172.21.15.31:0/41616139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcb530 msgr2=0x55ccd0bd4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f6205d9b700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f6205d9b700 1 -- 172.21.15.31:0/41616139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcc2f0 msgr2=0x55ccd0bdbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f6205d9b700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bdbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f6205d9b700 1 -- 172.21.15.31:0/41616139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6208109090 con 0x55ccd0bcabe0 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f6205d9b700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd4850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f61f800b2b0 tx=0x7f61f8005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38592e0e6c5978ca server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f61edffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.419+0000 7f61edffb700 1 -- 172.21.15.31:0/41616139 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f802b070 con 0x55ccd0bcabe0 2023-07-29T16:17:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.420+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ccd0b46d10 con 0x55ccd0bcabe0 2023-07-29T16:17:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.420+0000 7f61edffb700 1 -- 172.21.15.31:0/41616139 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f61f8004d20 con 0x55ccd0bcabe0 2023-07-29T16:17:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.420+0000 7f61edffb700 1 -- 172.21.15.31:0/41616139 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61f8004ee0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.420+0000 7f622096cb80 10 client.26306 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.420+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ccd0932c60 con 0x55ccd0bcabe0 2023-07-29T16:17:55.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.420+0000 7f61edffb700 1 -- 172.21.15.31:0/41616139 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ccd0b46d10 con 0x55ccd0bcabe0 2023-07-29T16:17:55.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.421+0000 7f61edffb700 1 -- 172.21.15.31:0/41616139 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f61f8005040 con 0x55ccd0bcabe0 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.421+0000 7f622096cb80 10 client.26306 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.421+0000 7f622096cb80 10 client.26306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.422+0000 7f622096cb80 20 client.26306 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.422+0000 7f622096cb80 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0be00e0 0x55ccd0be24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.422+0000 7f622096cb80 4 client.26306 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.422+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ccd0b99d70 con 0x55ccd0be00e0 2023-07-29T16:17:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.422+0000 7f620659c700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0be00e0 0x55ccd0be24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.423+0000 7f620659c700 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0be00e0 0x55ccd0be24f0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5846ad03aef8704e server_cookie=4bcab947a56a7982 in_seq=0 out_seq=0 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.423+0000 7f61edffb700 10 client.26306 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f61edffb700 1 -- 172.21.15.31:0/41616139 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ccd0b99d70 con 0x55ccd0be00e0 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f61edffb700 10 client.26306 handle_command_reply: tid=0 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 client.26306 shutdown 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 msgr2=0x55ccd0bd4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd4850 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f61f800b2b0 tx=0x7f61f8005c00 comp rx=0 tx=0).stop 2023-07-29T16:17:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 shutdown_connections 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0be00e0 0x55ccd0be24f0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcc2f0 0x55ccd0bdbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd4850 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 --2- 172.21.15.31:0/41616139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 >> 172.21.15.31:0/41616139 conn(0x55ccd0bc03e0 msgr2=0x55ccd0bc5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.424+0000 7f61edffb700 0 client.26306 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.425+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 shutdown_connections 2023-07-29T16:17:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.425+0000 7f622096cb80 1 -- 172.21.15.31:0/41616139 wait complete. 2023-07-29T16:17:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.425+0000 7f622096cb80 20 client.26306 trim_cache size 0 max 16384 2023-07-29T16:17:55.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.458+0000 7f622096cb80 1 Processor -- start 2023-07-29T16:17:55.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 -- start start 2023-07-29T16:17:55.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcabe0 0x55ccd0be20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0bbd710 con 0x55ccd0bcabe0 2023-07-29T16:17:55.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0bd58b0 con 0x55ccd0bcb530 2023-07-29T16:17:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.459+0000 7f622096cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0bc24b0 con 0x55ccd0bc03e0 2023-07-29T16:17:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f620659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcabe0 0x55ccd0be20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f620659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcabe0 0x55ccd0be20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37582/0 (socket says 172.21.15.31:37582) 2023-07-29T16:17:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f620659c700 1 -- 172.21.15.31:0/2924879107 learned_addr learned my addr 172.21.15.31:0/2924879107 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 -- 172.21.15.31:0/2924879107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcb530 msgr2=0x55ccd0bd2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 -- 172.21.15.31:0/2924879107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcabe0 msgr2=0x55ccd0be20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcabe0 0x55ccd0be20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 -- 172.21.15.31:0/2924879107 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccd0970250 con 0x55ccd0bc03e0 2023-07-29T16:17:55.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.460+0000 7f6205d9b700 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f61f800b930 tx=0x7f61f8004aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2c95f973a22ece0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.461+0000 7f61fd7fa700 1 -- 172.21.15.31:0/2924879107 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f802b040 con 0x55ccd0bc03e0 2023-07-29T16:17:55.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.461+0000 7f61fd7fa700 1 -- 172.21.15.31:0/2924879107 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f80057e0 con 0x55ccd0bc03e0 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.461+0000 7f61fd7fa700 1 -- 172.21.15.31:0/2924879107 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f800f900 con 0x55ccd0bc03e0 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 -- 172.21.15.31:0/2924879107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 msgr2=0x55ccd0bd9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f61f800b930 tx=0x7f61f8004aa0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 -- 172.21.15.31:0/2924879107 shutdown_connections 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9d80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 --2- 172.21.15.31:0/2924879107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcabe0 0x55ccd0be20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 -- 172.21.15.31:0/2924879107 >> 172.21.15.31:0/2924879107 conn(0x55ccd0bc6aa0 msgr2=0x55ccd0bc5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 -- 172.21.15.31:0/2924879107 shutdown_connections 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 -- 172.21.15.31:0/2924879107 wait complete. 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.462+0000 7f622096cb80 1 -- start start 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 Processor -- start 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0bd8090 con 0x55ccd0bcb530 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0bd8200 con 0x55ccd0bcabe0 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.463+0000 7f622096cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccd0bd83c0 con 0x55ccd0bc03e0 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f622096cb80 10 client.0 _ll_register_callbacks cb 0x55ccd0ac1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f620659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f6205d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37588/0 (socket says 172.21.15.31:37588) 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f6205d9b700 1 -- 172.21.15.31:0/3737744458 learned_addr learned my addr 172.21.15.31:0/3737744458 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f620659c700 1 -- 172.21.15.31:0/3737744458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 msgr2=0x55ccd0bd9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f620659c700 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f620659c700 1 -- 172.21.15.31:0/3737744458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 msgr2=0x55ccd0bd7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f620659c700 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.464+0000 7f620659c700 1 -- 172.21.15.31:0/3737744458 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ccd0932c60 con 0x55ccd0bcabe0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f620659c700 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd9a80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55ccd0bcbcc0 tx=0x7f61f00214f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df59c2aefe95492d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f61edffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f61edffb700 1 -- 172.21.15.31:0/3737744458 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0017070 con 0x55ccd0bcabe0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61f801e070 con 0x55ccd0bcabe0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f61edffb700 1 -- 172.21.15.31:0/3737744458 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f61f00218c0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f61edffb700 1 -- 172.21.15.31:0/3737744458 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61f0021a80 con 0x55ccd0bcabe0 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f622096cb80 10 client.26312 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.465+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ccd09482b0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f61edffb700 1 -- 172.21.15.31:0/3737744458 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61f801e070 con 0x55ccd0bcabe0 2023-07-29T16:17:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f61edffb700 1 -- 172.21.15.31:0/3737744458 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f61f00037a0 con 0x55ccd0bcabe0 2023-07-29T16:17:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f622096cb80 10 client.26312 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f622096cb80 10 client.26312 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f622096cb80 20 client.26312 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f622096cb80 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0bcc2f0 0x55ccd0c34ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f622096cb80 4 client.26312 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.466+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ccd0b14d40 con 0x55ccd0bcc2f0 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.467+0000 7f6205d9b700 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0bcc2f0 0x55ccd0c34ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.467+0000 7f6205d9b700 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0bcc2f0 0x55ccd0c34ec0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74a20a3d2215ec83 server_cookie=7023be6fb4c9d600 in_seq=0 out_seq=0 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.467+0000 7f61edffb700 10 client.26312 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.467+0000 7f61edffb700 1 -- 172.21.15.31:0/3737744458 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ccd0b14d40 con 0x55ccd0bcc2f0 2023-07-29T16:17:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.467+0000 7f61edffb700 10 client.26312 handle_command_reply: tid=0 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.467+0000 7f622096cb80 1 client.26312 shutdown 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 msgr2=0x55ccd0bd9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd9a80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x55ccd0bcbcc0 tx=0x7f61f00214f0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 shutdown_connections 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ccd0bcc2f0 0x55ccd0c34ec0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ccd0bcb530 0x55ccd0bd7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ccd0bc03e0 0x55ccd0bd9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 --2- 172.21.15.31:0/3737744458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ccd0bcabe0 0x55ccd0bd9a80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 >> 172.21.15.31:0/3737744458 conn(0x55ccd0bc6aa0 msgr2=0x55ccd0be20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f61edffb700 0 client.26312 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 shutdown_connections 2023-07-29T16:17:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 1 -- 172.21.15.31:0/3737744458 wait complete. 2023-07-29T16:17:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.468+0000 7f622096cb80 20 client.26312 trim_cache size 0 max 16384 2023-07-29T16:17:55.474 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:17:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:17:55.474 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:17:55.474 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:17:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61f40a0810 msgr2=0x7f61f40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61f40a0810 0x7f61f40a2cc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6200098a10 tx=0x7f6200009040 comp rx=0 tx=0).stop 2023-07-29T16:17:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 msgr2=0x7f62081943f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 0x7f62081943f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f620000efc0 tx=0x7f620000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 shutdown_connections 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61f40a0810 0x7f61f40a2cc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208101ea0 0x7f62081943f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6208100320 0x7f6208193e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 --2- 172.21.15.31:0/3270969958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6208101510 0x7f62081993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 >> 172.21.15.31:0/3270969958 conn(0x7f62080fb920 msgr2=0x7f62080fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 shutdown_connections 2023-07-29T16:17:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.469+0000 7f620cfff700 1 -- 172.21.15.31:0/3270969958 wait complete. 2023-07-29T16:17:55.486 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:17:55.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:17:55.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 -- 172.21.15.31:0/928327958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8141036b0 msgr2=0x7fb814105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 --2- 172.21.15.31:0/928327958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8141036b0 0x7fb814105a90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb808009990 tx=0x7fb808009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 -- 172.21.15.31:0/928327958 shutdown_connections 2023-07-29T16:17:55.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 --2- 172.21.15.31:0/928327958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 0x7fb814108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 --2- 172.21.15.31:0/928327958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb814100d90 0x7fb814103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 --2- 172.21.15.31:0/928327958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8141036b0 0x7fb814105a90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 -- 172.21.15.31:0/928327958 >> 172.21.15.31:0/928327958 conn(0x7fb8140fa810 msgr2=0x7fb8140fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.701+0000 7fb8189ff700 1 -- 172.21.15.31:0/928327958 shutdown_connections 2023-07-29T16:17:55.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.702+0000 7fb8189ff700 1 -- 172.21.15.31:0/928327958 wait complete. 2023-07-29T16:17:55.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.702+0000 7fb8189ff700 1 Processor -- start 2023-07-29T16:17:55.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 -- start start 2023-07-29T16:17:55.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb814100d90 0x7fb8141981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8141036b0 0x7fb814198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 0x7fb81419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81419fd50 con 0x7fb8141036b0 2023-07-29T16:17:55.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81419fe90 con 0x7fb814100d90 2023-07-29T16:17:55.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb8189ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8141a0020 con 0x7fb814105fd0 2023-07-29T16:17:55.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb811d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8141036b0 0x7fb814198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb811d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8141036b0 0x7fb814198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37614/0 (socket says 172.21.15.31:37614) 2023-07-29T16:17:55.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.703+0000 7fb811d9b700 1 -- 172.21.15.31:0/3030525419 learned_addr learned my addr 172.21.15.31:0/3030525419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:55.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 0x7fb81419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb81259c700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb814100d90 0x7fb8141981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 -- 172.21.15.31:0/3030525419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb814100d90 msgr2=0x7fb8141981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb814100d90 0x7fb8141981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 -- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8141036b0 msgr2=0x7fb814198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8141036b0 0x7fb814198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 -- 172.21.15.31:0/3030525419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080095f0 con 0x7fb814105fd0 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.704+0000 7fb812d9d700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 0x7fb81419f780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb80c007f10 tx=0x7fb80c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.705+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c00ddd0 con 0x7fb814105fd0 2023-07-29T16:17:55.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.705+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb80c00f040 con 0x7fb814105fd0 2023-07-29T16:17:55.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.705+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8141a0300 con 0x7fb814105fd0 2023-07-29T16:17:55.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.705+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80c005a90 con 0x7fb814105fd0 2023-07-29T16:17:55.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.705+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8141a0810 con 0x7fb814105fd0 2023-07-29T16:17:55.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.707+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb814107b40 con 0x7fb814105fd0 2023-07-29T16:17:55.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.710+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb80c004ad0 con 0x7fb814105fd0 2023-07-29T16:17:55.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.711+0000 7fb8037fe700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb7fc0a0860 0x7fb7fc0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.711+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb80c0cb290 con 0x7fb814105fd0 2023-07-29T16:17:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.712+0000 7fb81259c700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb7fc0a0860 0x7fb7fc0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:55.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.712+0000 7fb81259c700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb7fc0a0860 0x7fb7fc0a2d10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb80400bbd0 tx=0x7fb80400b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:55.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.715+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb80c04e720 con 0x7fb814105fd0 2023-07-29T16:17:55.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.973+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb81402cc30 con 0x7fb814105fd0 2023-07-29T16:17:55.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.974+0000 7fb8037fe700 1 -- 172.21.15.31:0/3030525419 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7fb80c094cd0 con 0x7fb814105fd0 2023-07-29T16:17:55.973 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:17:55.973 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:17:55.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb7fc0a0860 msgr2=0x7fb7fc0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb7fc0a0860 0x7fb7fc0a2d10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb80400bbd0 tx=0x7fb80400b5b0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 msgr2=0x7fb81419f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:55.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 0x7fb81419f780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb80c007f10 tx=0x7fb80c00d3b0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 shutdown_connections 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb7fc0a0860 0x7fb7fc0a2d10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb814105fd0 0x7fb81419f780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb814100d90 0x7fb8141981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 --2- 172.21.15.31:0/3030525419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8141036b0 0x7fb814198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 >> 172.21.15.31:0/3030525419 conn(0x7fb8140fa810 msgr2=0x7fb8140ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 shutdown_connections 2023-07-29T16:17:55.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:55.977+0000 7fb8189ff700 1 -- 172.21.15.31:0/3030525419 wait complete. 2023-07-29T16:17:55.978 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:17:55.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:17:56.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 -- 172.21.15.31:0/2869173685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 msgr2=0x7f2a20105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 --2- 172.21.15.31:0/2869173685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 0x7f2a20105aa0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009990 tx=0x7f2a14009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 -- 172.21.15.31:0/2869173685 shutdown_connections 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 --2- 172.21.15.31:0/2869173685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20105fe0 0x7f2a20108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 --2- 172.21.15.31:0/2869173685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a20100da0 0x7f2a20103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 --2- 172.21.15.31:0/2869173685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 0x7f2a20105aa0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 -- 172.21.15.31:0/2869173685 >> 172.21.15.31:0/2869173685 conn(0x7f2a200fa840 msgr2=0x7f2a200fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 -- 172.21.15.31:0/2869173685 shutdown_connections 2023-07-29T16:17:56.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.226+0000 7f2a24dff700 1 -- 172.21.15.31:0/2869173685 wait complete. 2023-07-29T16:17:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.227+0000 7f2a24dff700 1 Processor -- start 2023-07-29T16:17:56.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 -- start start 2023-07-29T16:17:56.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 0x7f2a20193f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 0x7f2a20194480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a20105fe0 0x7f2a2019b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2019bac0 con 0x7f2a20105fe0 2023-07-29T16:17:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2019bc30 con 0x7f2a201036c0 2023-07-29T16:17:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a24dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2019bdf0 con 0x7f2a20100da0 2023-07-29T16:17:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a1e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 0x7f2a20193f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a1e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 0x7f2a20193f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40156/0 (socket says 172.21.15.31:40156) 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.228+0000 7f2a1e59c700 1 -- 172.21.15.31:0/1324009587 learned_addr learned my addr 172.21.15.31:0/1324009587 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1e59c700 1 -- 172.21.15.31:0/1324009587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 msgr2=0x7f2a20194480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a20105fe0 0x7f2a2019b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1e59c700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 0x7f2a20194480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1e59c700 1 -- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a20105fe0 msgr2=0x7f2a2019b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1e59c700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a20105fe0 0x7f2a2019b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1e59c700 1 -- 172.21.15.31:0/1324009587 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a140095f0 con 0x7f2a20100da0 2023-07-29T16:17:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.229+0000 7f2a1e59c700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 0x7f2a20193f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000b530 tx=0x7f2a1000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.230+0000 7f2a0f7fe700 1 -- 172.21.15.31:0/1324009587 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10011830 con 0x7f2a20100da0 2023-07-29T16:17:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.230+0000 7f2a0f7fe700 1 -- 172.21.15.31:0/1324009587 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a10011e50 con 0x7f2a20100da0 2023-07-29T16:17:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.230+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2019c070 con 0x7f2a20100da0 2023-07-29T16:17:56.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.230+0000 7f2a0f7fe700 1 -- 172.21.15.31:0/1324009587 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000f2e0 con 0x7f2a20100da0 2023-07-29T16:17:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.230+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2019c580 con 0x7f2a20100da0 2023-07-29T16:17:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.234+0000 7f2a38705b80 1 Processor -- start 2023-07-29T16:17:56.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 -- start start 2023-07-29T16:17:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec18ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a0f7fe700 1 -- 172.21.15.31:0/1324009587 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2a10010580 con 0x7f2a20100da0 2023-07-29T16:17:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec19d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec1a2f0 0x55942ec1a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942eb70a10 con 0x55942ec19530 2023-07-29T16:17:56.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942eb6e3d0 con 0x55942ec1a2f0 2023-07-29T16:17:56.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.235+0000 7f2a38705b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942eb69cc0 con 0x55942ec18be0 2023-07-29T16:17:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.236+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec18ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.236+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec18ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40168/0 (socket says 172.21.15.31:40168) 2023-07-29T16:17:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.236+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1443336681 learned_addr learned my addr 172.21.15.31:0/1443336681 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.236+0000 7f2a1e59c700 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec19d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.236+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec1a2f0 0x55942ec1a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1443336681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec1a2f0 msgr2=0x55942ec1a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec1a2f0 0x55942ec1a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1443336681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 msgr2=0x55942ec19d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec19d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1443336681 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55942e997de0 con 0x55942ec18be0 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec1a2f0 0x55942ec1a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec18ff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009cf0 tx=0x7f2a18007500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a0de41923f2f0d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a0cff9700 1 -- 172.21.15.31:0/1443336681 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18004bb0 con 0x55942ec18be0 2023-07-29T16:17:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.237+0000 7f2a0f7fe700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a080a0790 0x7f2a080a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.238+0000 7f2a0cff9700 1 -- 172.21.15.31:0/1443336681 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a18004d10 con 0x55942ec18be0 2023-07-29T16:17:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.238+0000 7f2a0cff9700 1 -- 172.21.15.31:0/1443336681 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18014660 con 0x55942ec18be0 2023-07-29T16:17:56.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.238+0000 7f2a0f7fe700 1 -- 172.21.15.31:0/1324009587 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2a100cb630 con 0x7f2a20100da0 2023-07-29T16:17:56.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.238+0000 7f2a1e59c700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a080a0790 0x7f2a080a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 -- 172.21.15.31:0/1443336681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 msgr2=0x55942ec18ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec18ff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009cf0 tx=0x7f2a18007500 comp rx=0 tx=0).stop 2023-07-29T16:17:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 -- 172.21.15.31:0/1443336681 shutdown_connections 2023-07-29T16:17:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec1a2f0 0x55942ec1a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec18ff0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1443336681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec19d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 -- 172.21.15.31:0/1443336681 >> 172.21.15.31:0/1443336681 conn(0x55942ec0e3e0 msgr2=0x55942ec13990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 -- 172.21.15.31:0/1443336681 shutdown_connections 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a1e59c700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a080a0790 0x7f2a080a2c40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2a10000c00 tx=0x7f2a100d1190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.239+0000 7f2a38705b80 1 -- 172.21.15.31:0/1443336681 wait complete. 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.240+0000 7f2a38705b80 1 -- start start 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.240+0000 7f2a38705b80 1 Processor -- start 2023-07-29T16:17:56.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec22880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec22dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec29e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942eb06510 con 0x55942ec19530 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942eb061b0 con 0x55942ec18be0 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942e9ed920 con 0x55942ec1a2f0 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a38705b80 10 client.0 _ll_register_callbacks cb 0x55942eb0f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:56.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec22880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec22880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42794/0 (socket says 172.21.15.31:42794) 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/3722186225 learned_addr learned my addr 172.21.15.31:0/3722186225 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec29e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 msgr2=0x55942ec29e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec29e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/3722186225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 msgr2=0x55942ec22dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:17:56.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec22dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.241+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/3722186225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a20109030 con 0x55942ec18be0 2023-07-29T16:17:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.242+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec22880 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2a18015000 tx=0x7f2a18005b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd63515a115aa9b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.242+0000 7f2a05ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:56.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.242+0000 7f2a05ffb700 1 -- 172.21.15.31:0/3722186225 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18014ea0 con 0x55942ec18be0 2023-07-29T16:17:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.242+0000 7f2a05ffb700 1 -- 172.21.15.31:0/3722186225 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2a18004e80 con 0x55942ec18be0 2023-07-29T16:17:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.242+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55942eb94d10 con 0x55942ec18be0 2023-07-29T16:17:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a05ffb700 1 -- 172.21.15.31:0/3722186225 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a18005040 con 0x55942ec18be0 2023-07-29T16:17:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a38705b80 10 client.26327 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f2a18009920 con 0x55942ec18be0 2023-07-29T16:17:56.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a05ffb700 1 -- 172.21.15.31:0/3722186225 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55942eb94d10 con 0x55942ec18be0 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a05ffb700 1 -- 172.21.15.31:0/3722186225 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2a18021920 con 0x55942ec18be0 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a38705b80 10 client.26327 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.243+0000 7f2a38705b80 10 client.26327 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.244+0000 7f2a38705b80 20 client.26327 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.244+0000 7f2a38705b80 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec2e4c0 0x55942ec308d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.244+0000 7f2a38705b80 4 client.26327 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:56.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.244+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55942ebe7d70 con 0x55942ec2e4c0 2023-07-29T16:17:56.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.244+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec2e4c0 0x55942ec308d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.245+0000 7f2a05ffb700 10 client.26327 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:56.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.245+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec2e4c0 0x55942ec308d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9197e9edd2a0c9af server_cookie=d08adf655878ea3e in_seq=0 out_seq=0 2023-07-29T16:17:56.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a05ffb700 1 -- 172.21.15.31:0/3722186225 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55942ebe7d70 con 0x55942ec2e4c0 2023-07-29T16:17:56.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a05ffb700 10 client.26327 handle_command_reply: tid=0 2023-07-29T16:17:56.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 client.26327 shutdown 2023-07-29T16:17:56.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 msgr2=0x55942ec22880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec22880 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2a18015000 tx=0x7f2a18005b30 comp rx=0 tx=0).stop 2023-07-29T16:17:56.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 shutdown_connections 2023-07-29T16:17:56.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec29e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec2e4c0 0x55942ec308d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec22880 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 --2- 172.21.15.31:0/3722186225 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec22dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 >> 172.21.15.31:0/3722186225 conn(0x55942ec0e3e0 msgr2=0x55942ec13060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a05ffb700 0 client.26327 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:56.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 shutdown_connections 2023-07-29T16:17:56.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.246+0000 7f2a38705b80 1 -- 172.21.15.31:0/3722186225 wait complete. 2023-07-29T16:17:56.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.247+0000 7f2a38705b80 20 client.26327 trim_cache size 0 max 16384 2023-07-29T16:17:56.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.272+0000 7f2a38705b80 1 Processor -- start 2023-07-29T16:17:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.272+0000 7f2a38705b80 1 -- start start 2023-07-29T16:17:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec1a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec29dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a38705b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942ec2e2e0 con 0x55942ec19530 2023-07-29T16:17:56.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a38705b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942ec23a00 con 0x55942ec18be0 2023-07-29T16:17:56.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a38705b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942ec204d0 con 0x55942ec1a2f0 2023-07-29T16:17:56.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec1a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec1a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42800/0 (socket says 172.21.15.31:42800) 2023-07-29T16:17:56.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1693544590 learned_addr learned my addr 172.21.15.31:0/1693544590 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.273+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec29dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1693544590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 msgr2=0x55942ec21510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec21510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1693544590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 msgr2=0x55942ec29dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec29dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a1ed9d700 1 -- 172.21.15.31:0/1693544590 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55942e980c60 con 0x55942ec18be0 2023-07-29T16:17:56.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec1a6c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2a180078d0 tx=0x7f2a1800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b72f0ee4083bc2ff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.274+0000 7f2a0d7fa700 1 -- 172.21.15.31:0/1693544590 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18009490 con 0x55942ec18be0 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.275+0000 7f2a0d7fa700 1 -- 172.21.15.31:0/1693544590 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a18014ca0 con 0x55942ec18be0 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.275+0000 7f2a0d7fa700 1 -- 172.21.15.31:0/1693544590 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18023940 con 0x55942ec18be0 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.275+0000 7f2a38705b80 1 -- 172.21.15.31:0/1693544590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 msgr2=0x55942ec1a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.275+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec1a6c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2a180078d0 tx=0x7f2a1800b040 comp rx=0 tx=0).stop 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 -- 172.21.15.31:0/1693544590 shutdown_connections 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec1a2f0 0x55942ec21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec18be0 0x55942ec1a6c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 --2- 172.21.15.31:0/1693544590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec19530 0x55942ec29dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 -- 172.21.15.31:0/1693544590 >> 172.21.15.31:0/1693544590 conn(0x55942ec0e3e0 msgr2=0x55942ec14690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 -- 172.21.15.31:0/1693544590 shutdown_connections 2023-07-29T16:17:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 -- 172.21.15.31:0/1693544590 wait complete. 2023-07-29T16:17:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.276+0000 7f2a38705b80 1 -- start start 2023-07-29T16:17:56.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 Processor -- start 2023-07-29T16:17:56.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec25a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 0x55942ec25fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec1a2f0 0x55942ec28ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942ec29530 con 0x55942ec1a2f0 2023-07-29T16:17:56.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942ec296a0 con 0x55942ec19530 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55942ec29860 con 0x55942ec18be0 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.277+0000 7f2a38705b80 10 client.0 _ll_register_callbacks cb 0x55942eb0f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec25a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 0x55942ec25fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 0x55942ec25fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42812/0 (socket says 172.21.15.31:42812) 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 -- 172.21.15.31:0/2817046472 learned_addr learned my addr 172.21.15.31:0/2817046472 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec1a2f0 0x55942ec28ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 -- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 msgr2=0x55942ec25a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec25a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 -- 172.21.15.31:0/2817046472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec1a2f0 msgr2=0x55942ec28ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec1a2f0 0x55942ec28ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.278+0000 7f2a1dd9b700 1 -- 172.21.15.31:0/2817046472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a18009920 con 0x55942ec19530 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a1dd9b700 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 0x55942ec25fd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a140098e0 tx=0x7f2a1400b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7131269cdef93cbc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a05ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a05ffb700 1 -- 172.21.15.31:0/2817046472 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1401d040 con 0x55942ec19530 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a18004d10 con 0x55942ec19530 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a05ffb700 1 -- 172.21.15.31:0/2817046472 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2a1400baa0 con 0x55942ec19530 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a05ffb700 1 -- 172.21.15.31:0/2817046472 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a14003ca0 con 0x55942ec19530 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a38705b80 10 client.26333 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.279+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55942e9be250 con 0x55942ec19530 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a05ffb700 1 -- 172.21.15.31:0/2817046472 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a18004d10 con 0x55942ec19530 2023-07-29T16:17:56.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a05ffb700 1 -- 172.21.15.31:0/2817046472 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2a140051d0 con 0x55942ec19530 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a38705b80 10 client.26333 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a38705b80 10 client.26333 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a38705b80 20 client.26333 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a38705b80 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec80f80 0x55942ec83390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a38705b80 4 client.26333 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.280+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55942eb62d40 con 0x55942ec80f80 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec80f80 0x55942ec83390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a1ed9d700 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec80f80 0x55942ec83390 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d729aa758448d3a8 server_cookie=5687d6fc8648b9d8 in_seq=0 out_seq=0 2023-07-29T16:17:56.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a05ffb700 10 client.26333 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:56.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a05ffb700 1 -- 172.21.15.31:0/2817046472 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55942eb62d40 con 0x55942ec80f80 2023-07-29T16:17:56.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a05ffb700 10 client.26333 handle_command_reply: tid=0 2023-07-29T16:17:56.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a38705b80 1 client.26333 shutdown 2023-07-29T16:17:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 msgr2=0x55942ec25fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.281+0000 7f2a38705b80 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 0x55942ec25fd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a140098e0 tx=0x7f2a1400b880 comp rx=0 tx=0).stop 2023-07-29T16:17:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 shutdown_connections 2023-07-29T16:17:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55942ec1a2f0 0x55942ec28ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55942ec18be0 0x55942ec25a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55942ec80f80 0x55942ec83390 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 --2- 172.21.15.31:0/2817046472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55942ec19530 0x55942ec25fd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 >> 172.21.15.31:0/2817046472 conn(0x55942ec0e3e0 msgr2=0x55942ec20720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a05ffb700 0 client.26333 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 shutdown_connections 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 1 -- 172.21.15.31:0/2817046472 wait complete. 2023-07-29T16:17:56.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.282+0000 7f2a38705b80 20 client.26333 trim_cache size 0 max 16384 2023-07-29T16:17:56.288 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:17:56.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a080a0790 msgr2=0x7f2a080a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a080a0790 0x7f2a080a2c40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2a10000c00 tx=0x7f2a100d1190 comp rx=0 tx=0).stop 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 msgr2=0x7f2a20193f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 0x7f2a20193f40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000b530 tx=0x7f2a1000b8f0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 shutdown_connections 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a080a0790 0x7f2a080a2c40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a20105fe0 0x7f2a2019b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a20100da0 0x7f2a20193f40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 --2- 172.21.15.31:0/1324009587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a201036c0 0x7f2a20194480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 >> 172.21.15.31:0/1324009587 conn(0x7f2a200fa840 msgr2=0x7f2a200ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 shutdown_connections 2023-07-29T16:17:56.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.283+0000 7f2a24dff700 1 -- 172.21.15.31:0/1324009587 wait complete. 2023-07-29T16:17:56.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:17:56.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 -- 172.21.15.31:0/1595236150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 msgr2=0x7fc740103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 --2- 172.21.15.31:0/1595236150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc740103180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc730009990 tx=0x7fc730009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 -- 172.21.15.31:0/1595236150 shutdown_connections 2023-07-29T16:17:56.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 --2- 172.21.15.31:0/1595236150 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740105fe0 0x7fc740108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 --2- 172.21.15.31:0/1595236150 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc740103180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 --2- 172.21.15.31:0/1595236150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7401036c0 0x7fc740105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 -- 172.21.15.31:0/1595236150 >> 172.21.15.31:0/1595236150 conn(0x7fc7400fa840 msgr2=0x7fc7400fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 -- 172.21.15.31:0/1595236150 shutdown_connections 2023-07-29T16:17:56.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.538+0000 7fc745bff700 1 -- 172.21.15.31:0/1595236150 wait complete. 2023-07-29T16:17:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 Processor -- start 2023-07-29T16:17:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 -- start start 2023-07-29T16:17:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc7401960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7401036c0 0x7fc740196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 0x7fc74019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74019dc40 con 0x7fc7401036c0 2023-07-29T16:17:56.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74019dd80 con 0x7fc740105fe0 2023-07-29T16:17:56.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.540+0000 7fc745bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74019df10 con 0x7fc740100da0 2023-07-29T16:17:56.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc7401960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 0x7fc74019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 0x7fc74019d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42834/0 (socket says 172.21.15.31:42834) 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc7401960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40202/0 (socket says 172.21.15.31:40202) 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 -- 172.21.15.31:0/1736620860 learned_addr learned my addr 172.21.15.31:0/1736620860 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73effd700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7401036c0 0x7fc740196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 -- 172.21.15.31:0/1736620860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 msgr2=0x7fc7401960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc7401960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 -- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7401036c0 msgr2=0x7fc740196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7401036c0 0x7fc740196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 -- 172.21.15.31:0/1736620860 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300095f0 con 0x7fc740105fe0 2023-07-29T16:17:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73effd700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7401036c0 0x7fc740196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.541+0000 7fc73ffff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 0x7fc74019d670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc73800ef30 tx=0x7fc73800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.542+0000 7fc73cff9700 1 -- 172.21.15.31:0/1736620860 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738018070 con 0x7fc740105fe0 2023-07-29T16:17:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.542+0000 7fc73cff9700 1 -- 172.21.15.31:0/1736620860 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc73800f040 con 0x7fc740105fe0 2023-07-29T16:17:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.542+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7400feb90 con 0x7fc740105fe0 2023-07-29T16:17:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.542+0000 7fc73cff9700 1 -- 172.21.15.31:0/1736620860 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73800cdb0 con 0x7fc740105fe0 2023-07-29T16:17:56.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.542+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7400ff0f0 con 0x7fc740105fe0 2023-07-29T16:17:56.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.547+0000 7fc73cff9700 1 -- 172.21.15.31:0/1736620860 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc738003a00 con 0x7fc740105fe0 2023-07-29T16:17:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.548+0000 7fc7595afb80 1 Processor -- start 2023-07-29T16:17:56.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.548+0000 7fc7595afb80 1 -- start start 2023-07-29T16:17:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56441548fbe0 0x56441548fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x564415490d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x564415491740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc7595afb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644153e7a10 con 0x564415490530 2023-07-29T16:17:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc7595afb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644153e53d0 con 0x5644154912f0 2023-07-29T16:17:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc7595afb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644153e0cc0 con 0x56441548fbe0 2023-07-29T16:17:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc73effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x564415491740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x564415490d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc73effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x564415491740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42842/0 (socket says 172.21.15.31:42842) 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc73effd700 1 -- 172.21.15.31:0/2631140482 learned_addr learned my addr 172.21.15.31:0/2631140482 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x564415490d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37682/0 (socket says 172.21.15.31:37682) 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.549+0000 7fc73effd700 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56441548fbe0 0x56441548fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73effd700 1 -- 172.21.15.31:0/2631140482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56441548fbe0 msgr2=0x56441548fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73effd700 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56441548fbe0 0x56441548fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73effd700 1 -- 172.21.15.31:0/2631140482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 msgr2=0x564415490d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73effd700 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x564415490d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73effd700 1 -- 172.21.15.31:0/2631140482 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56441520ede0 con 0x5644154912f0 2023-07-29T16:17:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73cff9700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7280a0810 0x7fc7280a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73cff9700 1 -- 172.21.15.31:0/1736620860 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc7380cb630 con 0x7fc740105fe0 2023-07-29T16:17:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.550+0000 7fc73effd700 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x564415491740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x56441548f980 tx=0x7fc734009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eac20217232a2296 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.551+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7280a0810 0x7fc7280a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.551+0000 7fc725ffb700 1 -- 172.21.15.31:0/2631140482 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73400f450 con 0x5644154912f0 2023-07-29T16:17:56.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.551+0000 7fc725ffb700 1 -- 172.21.15.31:0/2631140482 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc73400fa70 con 0x5644154912f0 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.551+0000 7fc725ffb700 1 -- 172.21.15.31:0/2631140482 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73401e8c0 con 0x5644154912f0 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 -- 172.21.15.31:0/2631140482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 msgr2=0x564415491740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7280a0810 0x7fc7280a2cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x564415490940 tx=0x7fc730005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x564415491740 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x56441548f980 tx=0x7fc734009820 comp rx=0 tx=0).stop 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 -- 172.21.15.31:0/2631140482 shutdown_connections 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x564415491740 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56441548fbe0 0x56441548fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2631140482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x564415490d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 -- 172.21.15.31:0/2631140482 >> 172.21.15.31:0/2631140482 conn(0x5644154853e0 msgr2=0x56441548a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 -- 172.21.15.31:0/2631140482 shutdown_connections 2023-07-29T16:17:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.552+0000 7fc7595afb80 1 -- 172.21.15.31:0/2631140482 wait complete. 2023-07-29T16:17:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.553+0000 7fc7595afb80 1 -- start start 2023-07-29T16:17:56.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.553+0000 7fc7595afb80 1 Processor -- start 2023-07-29T16:17:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 0x564415499890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x564415499dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x5644154a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441537d510 con 0x56441548fbe0 2023-07-29T16:17:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441537d1b0 con 0x5644154912f0 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564415264920 con 0x564415490530 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc7595afb80 10 client.0 _ll_register_callbacks cb 0x564415386620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 0x564415499890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 0x564415499890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37688/0 (socket says 172.21.15.31:37688) 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc73f7fe700 1 -- 172.21.15.31:0/2283914387 learned_addr learned my addr 172.21.15.31:0/2283914387 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.554+0000 7fc73effd700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x564415499dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73ffff700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x5644154a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73effd700 1 -- 172.21.15.31:0/2283914387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 msgr2=0x5644154a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73effd700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x5644154a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73effd700 1 -- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 msgr2=0x564415499890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73effd700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 0x564415499890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73effd700 1 -- 172.21.15.31:0/2283914387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73401a040 con 0x564415490530 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 0x564415499890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc73effd700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x564415499dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc734011010 tx=0x7fc734005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb6fee0e752f0925 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc71effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.555+0000 7fc71effd700 1 -- 172.21.15.31:0/2283914387 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734009e70 con 0x564415490530 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.556+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56441540bd10 con 0x564415490530 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.556+0000 7fc71effd700 1 -- 172.21.15.31:0/2283914387 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc734000e40 con 0x564415490530 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.556+0000 7fc71effd700 1 -- 172.21.15.31:0/2283914387 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc73400f880 con 0x564415490530 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.556+0000 7fc7595afb80 10 client.26356 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.556+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5644151f7c60 con 0x564415490530 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.556+0000 7fc71effd700 1 -- 172.21.15.31:0/2283914387 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56441540bd10 con 0x564415490530 2023-07-29T16:17:56.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc71effd700 1 -- 172.21.15.31:0/2283914387 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc734004750 con 0x564415490530 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc7595afb80 10 client.26356 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc7595afb80 10 client.26356 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc7595afb80 20 client.26356 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154a54d0 0x5644154a78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc7595afb80 4 client.26356 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56441545ed70 con 0x5644154a54d0 2023-07-29T16:17:56.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.557+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154a54d0 0x5644154a78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.558+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154a54d0 0x5644154a78e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ed959b8a3bf025e server_cookie=8963e9e45793a060 in_seq=0 out_seq=0 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.558+0000 7fc71effd700 10 client.26356 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.559+0000 7fc71effd700 1 -- 172.21.15.31:0/2283914387 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56441545ed70 con 0x5644154a54d0 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.559+0000 7fc71effd700 10 client.26356 handle_command_reply: tid=0 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.559+0000 7fc7595afb80 1 client.26356 shutdown 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.559+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 msgr2=0x564415499dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.559+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x564415499dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc734011010 tx=0x7fc734005d50 comp rx=0 tx=0).stop 2023-07-29T16:17:56.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 shutdown_connections 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154a54d0 0x5644154a78e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5644154912f0 0x5644154a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56441548fbe0 0x564415499890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2283914387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x564415499dd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 >> 172.21.15.31:0/2283914387 conn(0x5644154853e0 msgr2=0x56441548a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc71effd700 0 client.26356 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 shutdown_connections 2023-07-29T16:17:56.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 1 -- 172.21.15.31:0/2283914387 wait complete. 2023-07-29T16:17:56.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.560+0000 7fc7595afb80 20 client.26356 trim_cache size 0 max 16384 2023-07-29T16:17:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 Processor -- start 2023-07-29T16:17:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 -- start start 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x5644154ac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x56441549edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644154912f0 0x5644154916c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564415487460 con 0x5644154912f0 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441548c2e0 con 0x56441548fbe0 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.576+0000 7fc7595afb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441549a510 con 0x564415490530 2023-07-29T16:17:56.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x5644154ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x5644154ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42846/0 (socket says 172.21.15.31:42846) 2023-07-29T16:17:56.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 -- 172.21.15.31:0/2122123673 learned_addr learned my addr 172.21.15.31:0/2122123673 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73ffff700 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644154912f0 0x5644154916c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73effd700 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x56441549edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 -- 172.21.15.31:0/2122123673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 msgr2=0x56441549edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x56441549edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 -- 172.21.15.31:0/2122123673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644154912f0 msgr2=0x5644154916c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644154912f0 0x5644154916c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.577+0000 7fc73f7fe700 1 -- 172.21.15.31:0/2122123673 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564415235250 con 0x56441548fbe0 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.578+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x5644154ac240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc73000b680 tx=0x7fc730027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=535df8052f159c8e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.578+0000 7fc7267fc700 1 -- 172.21.15.31:0/2122123673 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73003a160 con 0x56441548fbe0 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.578+0000 7fc7267fc700 1 -- 172.21.15.31:0/2122123673 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc730022040 con 0x56441548fbe0 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.578+0000 7fc7267fc700 1 -- 172.21.15.31:0/2122123673 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730004740 con 0x56441548fbe0 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 -- 172.21.15.31:0/2122123673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 msgr2=0x5644154ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x5644154ac240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc73000b680 tx=0x7fc730027000 comp rx=0 tx=0).stop 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 -- 172.21.15.31:0/2122123673 shutdown_connections 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5644154912f0 0x5644154916c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x5644154ac240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2122123673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564415490530 0x56441549edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 -- 172.21.15.31:0/2122123673 >> 172.21.15.31:0/2122123673 conn(0x5644154853e0 msgr2=0x564415498510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 -- 172.21.15.31:0/2122123673 shutdown_connections 2023-07-29T16:17:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 -- 172.21.15.31:0/2122123673 wait complete. 2023-07-29T16:17:56.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.579+0000 7fc7595afb80 1 -- start start 2023-07-29T16:17:56.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 Processor -- start 2023-07-29T16:17:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x56441549c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x56441549cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 0x56441549de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441549e340 con 0x564415490530 2023-07-29T16:17:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441549e4b0 con 0x56441548fbe0 2023-07-29T16:17:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56441549e670 con 0x5644154912f0 2023-07-29T16:17:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.580+0000 7fc7595afb80 10 client.0 _ll_register_callbacks cb 0x564415386620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 0x56441549de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x56441549cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 0x56441549de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40246/0 (socket says 172.21.15.31:40246) 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 -- 172.21.15.31:0/2498944457 learned_addr learned my addr 172.21.15.31:0/2498944457 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73effd700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x56441549cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37710/0 (socket says 172.21.15.31:37710) 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x56441549c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 -- 172.21.15.31:0/2498944457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 msgr2=0x56441549c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x56441549c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 -- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 msgr2=0x56441549cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x56441549cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 -- 172.21.15.31:0/2498944457 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc740109030 con 0x5644154912f0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc73ffff700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 0x56441549de00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc738098cf0 tx=0x7fc7380092e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c93418cdd1c3b2c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc71effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.581+0000 7fc71effd700 1 -- 172.21.15.31:0/2498944457 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738098450 con 0x5644154912f0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc71effd700 1 -- 172.21.15.31:0/2498944457 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc738098d60 con 0x5644154912f0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc73400fa70 con 0x5644154912f0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc71effd700 1 -- 172.21.15.31:0/2498944457 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc73809cd90 con 0x5644154912f0 2023-07-29T16:17:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 10 client.26362 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56441520d2b0 con 0x5644154912f0 2023-07-29T16:17:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc71effd700 1 -- 172.21.15.31:0/2498944457 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc73400fa70 con 0x5644154912f0 2023-07-29T16:17:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc71effd700 1 -- 172.21.15.31:0/2498944457 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc7380b11d0 con 0x5644154912f0 2023-07-29T16:17:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 10 client.26362 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 10 client.26362 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:17:56.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 20 client.26362 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154f7d80 0x5644154fa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 4 client.26362 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.582+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5644153d9d40 con 0x5644154f7d80 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.583+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154f7d80 0x5644154fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.583+0000 7fc73f7fe700 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154f7d80 0x5644154fa190 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0ac09f45ad59fd server_cookie=32565090b83147b in_seq=0 out_seq=0 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.583+0000 7fc71effd700 10 client.26362 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.583+0000 7fc71effd700 1 -- 172.21.15.31:0/2498944457 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5644153d9d40 con 0x5644154f7d80 2023-07-29T16:17:56.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc71effd700 10 client.26362 handle_command_reply: tid=0 2023-07-29T16:17:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 client.26362 shutdown 2023-07-29T16:17:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 msgr2=0x56441549de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 0x56441549de00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc738098cf0 tx=0x7fc7380092e0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 shutdown_connections 2023-07-29T16:17:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5644154f7d80 0x5644154fa190 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644154912f0 0x56441549de00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441548fbe0 0x56441549c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc71effd700 0 client.26362 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 --2- 172.21.15.31:0/2498944457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564415490530 0x56441549cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 >> 172.21.15.31:0/2498944457 conn(0x5644154853e0 msgr2=0x5644154976b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 shutdown_connections 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 1 -- 172.21.15.31:0/2498944457 wait complete. 2023-07-29T16:17:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.584+0000 7fc7595afb80 20 client.26362 trim_cache size 0 max 16384 2023-07-29T16:17:56.592 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:17:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7280a0810 msgr2=0x7fc7280a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7280a0810 0x7fc7280a2cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x564415490940 tx=0x7fc730005fb0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 msgr2=0x7fc74019d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 0x7fc74019d670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc73800ef30 tx=0x7fc73800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 shutdown_connections 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7280a0810 0x7fc7280a2cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc740105fe0 0x7fc74019d670 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc740100da0 0x7fc7401960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 --2- 172.21.15.31:0/1736620860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc7401036c0 0x7fc740196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 >> 172.21.15.31:0/1736620860 conn(0x7fc7400fa840 msgr2=0x7fc740101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 shutdown_connections 2023-07-29T16:17:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.585+0000 7fc745bff700 1 -- 172.21.15.31:0/1736620860 wait complete. 2023-07-29T16:17:56.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:56 smithi031 ceph-mon[95504]: pgmap v443: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:56.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:56 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3030525419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:17:56.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:17:56.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:56 smithi039 ceph-mon[128858]: pgmap v443: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:56 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3030525419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:17:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:56 smithi156 ceph-mon[129245]: pgmap v443: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:56 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3030525419' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:17:56.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 -- 172.21.15.31:0/541356891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984100210 msgr2=0x7f8984100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 --2- 172.21.15.31:0/541356891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984100210 0x7f8984100620 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8974009990 tx=0x7f8974009ca0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 -- 172.21.15.31:0/541356891 shutdown_connections 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 --2- 172.21.15.31:0/541356891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8984101e50 0x7f89841086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 --2- 172.21.15.31:0/541356891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984100210 0x7f8984100620 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 --2- 172.21.15.31:0/541356891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89841014a0 0x7f8984101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 -- 172.21.15.31:0/541356891 >> 172.21.15.31:0/541356891 conn(0x7f89840fb850 msgr2=0x7f89840fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 -- 172.21.15.31:0/541356891 shutdown_connections 2023-07-29T16:17:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.825+0000 7f8988d9d700 1 -- 172.21.15.31:0/541356891 wait complete. 2023-07-29T16:17:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 Processor -- start 2023-07-29T16:17:56.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 -- start start 2023-07-29T16:17:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 0x7f8984197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89841014a0 0x7f8984198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984101e50 0x7f898419f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898419fb50 con 0x7f89841014a0 2023-07-29T16:17:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898419fc90 con 0x7f8984100210 2023-07-29T16:17:56.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.827+0000 7f8988d9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898419fe20 con 0x7f8984101e50 2023-07-29T16:17:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 0x7f8984197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 0x7f8984197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42886/0 (socket says 172.21.15.31:42886) 2023-07-29T16:17:56.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 -- 172.21.15.31:0/1053347739 learned_addr learned my addr 172.21.15.31:0/1053347739 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f8982ffd700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89841014a0 0x7f8984198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f8983fff700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984101e50 0x7f898419f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 -- 172.21.15.31:0/1053347739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984101e50 msgr2=0x7f898419f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984101e50 0x7f898419f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 -- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89841014a0 msgr2=0x7f8984198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89841014a0 0x7f8984198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.828+0000 7f89837fe700 1 -- 172.21.15.31:0/1053347739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89740095f0 con 0x7f8984100210 2023-07-29T16:17:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.829+0000 7f89837fe700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 0x7f8984197fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8974006010 tx=0x7f8974005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.829+0000 7f8980ff9700 1 -- 172.21.15.31:0/1053347739 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897400f660 con 0x7f8984100210 2023-07-29T16:17:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.829+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89841026c0 con 0x7f8984100210 2023-07-29T16:17:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.829+0000 7f8980ff9700 1 -- 172.21.15.31:0/1053347739 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f897400bec0 con 0x7f8984100210 2023-07-29T16:17:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.829+0000 7f8980ff9700 1 -- 172.21.15.31:0/1053347739 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974017c30 con 0x7f8984100210 2023-07-29T16:17:56.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.829+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8984102bc0 con 0x7f8984100210 2023-07-29T16:17:56.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.834+0000 7f8980ff9700 1 -- 172.21.15.31:0/1053347739 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f897400f7c0 con 0x7f8984100210 2023-07-29T16:17:56.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.835+0000 7f8980ff9700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f896c0a08b0 0x7f896c0a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f8980ff9700 1 -- 172.21.15.31:0/1053347739 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f89740d5ba0 con 0x7f8984100210 2023-07-29T16:17:56.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 Processor -- start 2023-07-29T16:17:56.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 -- start start 2023-07-29T16:17:56.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f8982ffd700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f896c0a08b0 0x7f896c0a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981c1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c1690 0x5562981c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c24d0 0x5562981c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556298118a10 con 0x5562981c1690 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562981163d0 con 0x5562981c24d0 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.836+0000 7f899da00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556298111cc0 con 0x5562981c0d40 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981c1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981c1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40268/0 (socket says 172.21.15.31:40268) 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 -- 172.21.15.31:0/1019043891 learned_addr learned my addr 172.21.15.31:0/1019043891 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f8983fff700 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c24d0 0x5562981c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f8982ffd700 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c1690 0x5562981c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 -- 172.21.15.31:0/1019043891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c24d0 msgr2=0x5562981c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c24d0 0x5562981c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 -- 172.21.15.31:0/1019043891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c1690 msgr2=0x5562981c1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c1690 0x5562981c1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f89837fe700 1 -- 172.21.15.31:0/1019043891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556297f3fde0 con 0x5562981c0d40 2023-07-29T16:17:56.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.837+0000 7f8982ffd700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f896c0a08b0 0x7f896c0a2d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8978013570 tx=0x7f8978013930 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.838+0000 7f89837fe700 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981c1150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89740a3260 tx=0x7f8974005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9385f86c33131bd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.838+0000 7f8969ffb700 1 -- 172.21.15.31:0/1019043891 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740ba070 con 0x5562981c0d40 2023-07-29T16:17:56.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.838+0000 7f8969ffb700 1 -- 172.21.15.31:0/1019043891 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f89740a70d0 con 0x5562981c0d40 2023-07-29T16:17:56.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.838+0000 7f8969ffb700 1 -- 172.21.15.31:0/1019043891 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740a9170 con 0x5562981c0d40 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 -- 172.21.15.31:0/1019043891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 msgr2=0x5562981c1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981c1150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f89740a3260 tx=0x7f8974005ad0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 -- 172.21.15.31:0/1019043891 shutdown_connections 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c24d0 0x5562981c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981c1150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 --2- 172.21.15.31:0/1019043891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c1690 0x5562981c1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 -- 172.21.15.31:0/1019043891 >> 172.21.15.31:0/1019043891 conn(0x5562981b6540 msgr2=0x5562981bbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 -- 172.21.15.31:0/1019043891 shutdown_connections 2023-07-29T16:17:56.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.839+0000 7f899da00b80 1 -- 172.21.15.31:0/1019043891 wait complete. 2023-07-29T16:17:56.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.840+0000 7f899da00b80 1 -- start start 2023-07-29T16:17:56.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.840+0000 7f899da00b80 1 Processor -- start 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.840+0000 7f899da00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.840+0000 7f899da00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f899da00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f899da00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562980ae510 con 0x5562981c24d0 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f899da00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562980ae1b0 con 0x5562981c1690 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f899da00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556297f95920 con 0x5562981c0d40 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f899da00b80 10 client.0 _ll_register_callbacks cb 0x5562980b7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40272/0 (socket says 172.21.15.31:40272) 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 -- 172.21.15.31:0/3891044331 learned_addr learned my addr 172.21.15.31:0/3891044331 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f8982ffd700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f8983fff700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 -- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 msgr2=0x5562981cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981cabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 -- 172.21.15.31:0/3891044331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 msgr2=0x5562981d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981d1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.841+0000 7f89837fe700 1 -- 172.21.15.31:0/3891044331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8984102bc0 con 0x5562981c0d40 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f89837fe700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981ca680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f897409f7d0 tx=0x7f89740a31a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5c0af84a3875804 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f8962ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f8962ffd700 1 -- 172.21.15.31:0/3891044331 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89740ba030 con 0x5562981c0d40 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f8962ffd700 1 -- 172.21.15.31:0/3891044331 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f89740bb070 con 0x5562981c0d40 2023-07-29T16:17:56.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55629813cd10 con 0x5562981c0d40 2023-07-29T16:17:56.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f8962ffd700 1 -- 172.21.15.31:0/3891044331 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89740ab690 con 0x5562981c0d40 2023-07-29T16:17:56.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f899da00b80 10 client.26374 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:56.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556297f28c60 con 0x5562981c0d40 2023-07-29T16:17:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.842+0000 7f8962ffd700 1 -- 172.21.15.31:0/3891044331 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f89740d3020 con 0x5562981c0d40 2023-07-29T16:17:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f8962ffd700 1 -- 172.21.15.31:0/3891044331 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f89740ab200 con 0x5562981c0d40 2023-07-29T16:17:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f899da00b80 10 client.26374 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f899da00b80 10 client.26374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f899da00b80 20 client.26374 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:56.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f899da00b80 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5562981d5d80 0x5562981d8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f899da00b80 4 client.26374 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55629818fd70 con 0x5562981d5d80 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.843+0000 7f8982ffd700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5562981d5d80 0x5562981d8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.844+0000 7f8982ffd700 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5562981d5d80 0x5562981d8190 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f5e162cd6924b8 server_cookie=4a7abffdd7bbcc6b in_seq=0 out_seq=0 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.844+0000 7f8962ffd700 10 client.26374 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.844+0000 7f8962ffd700 1 -- 172.21.15.31:0/3891044331 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55629818fd70 con 0x5562981d5d80 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.844+0000 7f8962ffd700 10 client.26374 handle_command_reply: tid=0 2023-07-29T16:17:56.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.844+0000 7f899da00b80 1 client.26374 shutdown 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 msgr2=0x5562981ca680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981ca680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f897409f7d0 tx=0x7f89740a31a0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 shutdown_connections 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5562981d5d80 0x5562981d8190 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981d1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c0d40 0x5562981ca680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 --2- 172.21.15.31:0/3891044331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 >> 172.21.15.31:0/3891044331 conn(0x5562981b6540 msgr2=0x5562981bb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f8962ffd700 0 client.26374 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:56.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 shutdown_connections 2023-07-29T16:17:56.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 1 -- 172.21.15.31:0/3891044331 wait complete. 2023-07-29T16:17:56.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.845+0000 7f899da00b80 20 client.26374 trim_cache size 0 max 16384 2023-07-29T16:17:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.870+0000 7f899da00b80 1 Processor -- start 2023-07-29T16:17:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 -- start start 2023-07-29T16:17:56.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c0d40 0x5562981c5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981d1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 0x5562981c58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562981bd510 con 0x5562981c0d40 2023-07-29T16:17:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55629818fda0 con 0x5562981c1690 2023-07-29T16:17:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.871+0000 7f899da00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562981cb870 con 0x5562981c24d0 2023-07-29T16:17:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 0x5562981c58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f89837fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c0d40 0x5562981c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 0x5562981c58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40288/0 (socket says 172.21.15.31:40288) 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 -- 172.21.15.31:0/2386230711 learned_addr learned my addr 172.21.15.31:0/2386230711 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 -- 172.21.15.31:0/2386230711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 msgr2=0x5562981d1bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8982ffd700 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981d1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981d1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 -- 172.21.15.31:0/2386230711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c0d40 msgr2=0x5562981c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c0d40 0x5562981c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8983fff700 1 -- 172.21.15.31:0/2386230711 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556297f66250 con 0x5562981c24d0 2023-07-29T16:17:56.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f8982ffd700 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:17:56.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.872+0000 7f89837fe700 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c0d40 0x5562981c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:56.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.873+0000 7f8983fff700 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 0x5562981c58d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f897c007600 tx=0x7f897c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5dde8ff60601e5fe server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.873+0000 7f896a7fc700 1 -- 172.21.15.31:0/2386230711 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c007ea0 con 0x5562981c24d0 2023-07-29T16:17:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.873+0000 7f896a7fc700 1 -- 172.21.15.31:0/2386230711 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f897c00f040 con 0x5562981c24d0 2023-07-29T16:17:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.873+0000 7f896a7fc700 1 -- 172.21.15.31:0/2386230711 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897c00e040 con 0x5562981c24d0 2023-07-29T16:17:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 -- 172.21.15.31:0/2386230711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 msgr2=0x5562981c58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 0x5562981c58d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f897c007600 tx=0x7f897c009920 comp rx=0 tx=0).stop 2023-07-29T16:17:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 -- 172.21.15.31:0/2386230711 shutdown_connections 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c24d0 0x5562981c58d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c0d40 0x5562981c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 --2- 172.21.15.31:0/2386230711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c1690 0x5562981d1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 -- 172.21.15.31:0/2386230711 >> 172.21.15.31:0/2386230711 conn(0x5562981b6540 msgr2=0x5562981c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 -- 172.21.15.31:0/2386230711 shutdown_connections 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.874+0000 7f899da00b80 1 -- 172.21.15.31:0/2386230711 wait complete. 2023-07-29T16:17:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.875+0000 7f899da00b80 1 -- start start 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.875+0000 7f899da00b80 1 Processor -- start 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c0d40 0x5562981d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c1690 0x5562981ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562981ceea0 con 0x5562981c24d0 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562981cf010 con 0x5562981c0d40 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556298227900 con 0x5562981c1690 2023-07-29T16:17:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f899da00b80 10 client.0 _ll_register_callbacks cb 0x5562980b7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f89837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c0d40 0x5562981d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f89837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c0d40 0x5562981d17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42910/0 (socket says 172.21.15.31:42910) 2023-07-29T16:17:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f89837fe700 1 -- 172.21.15.31:0/321208489 learned_addr learned my addr 172.21.15.31:0/321208489 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.876+0000 7f8982ffd700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c1690 0x5562981ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8983fff700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8982ffd700 1 -- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c0d40 msgr2=0x5562981d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8982ffd700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c0d40 0x5562981d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8982ffd700 1 -- 172.21.15.31:0/321208489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 msgr2=0x5562981ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8982ffd700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8982ffd700 1 -- 172.21.15.31:0/321208489 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8984102bc0 con 0x5562981c1690 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f8962ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.877+0000 7f8982ffd700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c1690 0x5562981ce420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5562981d8a70 tx=0x7f89780047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20829df7bc97f4b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f8962ffd700 1 -- 172.21.15.31:0/321208489 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978018040 con 0x5562981c1690 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f8962ffd700 1 -- 172.21.15.31:0/321208489 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f89780288e0 con 0x5562981c1690 2023-07-29T16:17:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89740a70d0 con 0x5562981c1690 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f8962ffd700 1 -- 172.21.15.31:0/321208489 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8978028aa0 con 0x5562981c1690 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f899da00b80 10 client.26380 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.878+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f897c0092e0 con 0x5562981c1690 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f8962ffd700 1 -- 172.21.15.31:0/321208489 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8978037020 con 0x5562981c1690 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f8962ffd700 1 -- 172.21.15.31:0/321208489 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8978020430 con 0x5562981c1690 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f899da00b80 10 client.26380 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f899da00b80 10 client.26380 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:56.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f899da00b80 20 client.26380 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f899da00b80 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556298229110 0x55629822b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f899da00b80 4 client.26380 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.879+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55629810ad40 con 0x556298229110 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.880+0000 7f89837fe700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556298229110 0x55629822b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.880+0000 7f89837fe700 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556298229110 0x55629822b520 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0bd4f32f643c6d1 server_cookie=b943a3bf2a5b1f15 in_seq=0 out_seq=0 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.880+0000 7f8962ffd700 10 client.26380 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:56.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.880+0000 7f8962ffd700 1 -- 172.21.15.31:0/321208489 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55629810ad40 con 0x556298229110 2023-07-29T16:17:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.880+0000 7f8962ffd700 10 client.26380 handle_command_reply: tid=0 2023-07-29T16:17:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.880+0000 7f899da00b80 1 client.26380 shutdown 2023-07-29T16:17:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c1690 msgr2=0x5562981ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c1690 0x5562981ce420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5562981d8a70 tx=0x7f89780047c0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 shutdown_connections 2023-07-29T16:17:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556298229110 0x55629822b520 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5562981c24d0 0x5562981ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5562981c0d40 0x5562981d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 --2- 172.21.15.31:0/321208489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5562981c1690 0x5562981ce420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f8962ffd700 0 client.26380 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 >> 172.21.15.31:0/321208489 conn(0x5562981b6540 msgr2=0x5562981c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 shutdown_connections 2023-07-29T16:17:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 1 -- 172.21.15.31:0/321208489 wait complete. 2023-07-29T16:17:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.881+0000 7f899da00b80 20 client.26380 trim_cache size 0 max 16384 2023-07-29T16:17:56.888 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:17:56.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f896c0a08b0 msgr2=0x7f896c0a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f896c0a08b0 0x7f896c0a2d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8978013570 tx=0x7f8978013930 comp rx=0 tx=0).stop 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 msgr2=0x7f8984197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 0x7f8984197fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8974006010 tx=0x7f8974005f10 comp rx=0 tx=0).stop 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 shutdown_connections 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f896c0a08b0 0x7f896c0a2d60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8984101e50 0x7f898419f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8984100210 0x7f8984197fd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 --2- 172.21.15.31:0/1053347739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89841014a0 0x7f8984198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 >> 172.21.15.31:0/1053347739 conn(0x7f89840fb850 msgr2=0x7f89840fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.882+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 shutdown_connections 2023-07-29T16:17:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:56.883+0000 7f8988d9d700 1 -- 172.21.15.31:0/1053347739 wait complete. 2023-07-29T16:17:57.911 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:17:57.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:17:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:57 smithi031 ceph-mon[95504]: pgmap v444: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 -- 172.21.15.31:0/3541536671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca601036c0 msgr2=0x7fca60105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 --2- 172.21.15.31:0/3541536671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca601036c0 0x7fca60105aa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00faf0 tx=0x7fca5c00fe00 comp rx=0 tx=0).stop 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 -- 172.21.15.31:0/3541536671 shutdown_connections 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 --2- 172.21.15.31:0/3541536671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca60105fe0 0x7fca60108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 --2- 172.21.15.31:0/3541536671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca60100da0 0x7fca60103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 --2- 172.21.15.31:0/3541536671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca601036c0 0x7fca60105aa0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.265+0000 7fca67fff700 1 -- 172.21.15.31:0/3541536671 >> 172.21.15.31:0/3541536671 conn(0x7fca600fa840 msgr2=0x7fca600fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.266+0000 7fca67fff700 1 -- 172.21.15.31:0/3541536671 shutdown_connections 2023-07-29T16:17:58.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.266+0000 7fca67fff700 1 -- 172.21.15.31:0/3541536671 wait complete. 2023-07-29T16:17:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.267+0000 7fca67fff700 1 Processor -- start 2023-07-29T16:17:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.267+0000 7fca67fff700 1 -- start start 2023-07-29T16:17:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca67fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 0x7fca60193e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca67fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca601036c0 0x7fca601943d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca67fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca60105fe0 0x7fca6019b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca67fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6019ba10 con 0x7fca60105fe0 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca67fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6019bb50 con 0x7fca601036c0 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca67fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6019bce0 con 0x7fca60100da0 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 0x7fca60193e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 0x7fca60193e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40308/0 (socket says 172.21.15.31:40308) 2023-07-29T16:17:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca65d9b700 1 -- 172.21.15.31:0/2340036538 learned_addr learned my addr 172.21.15.31:0/2340036538 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca6559a700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca601036c0 0x7fca601943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.268+0000 7fca6659c700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca60105fe0 0x7fca6019b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca6559a700 1 -- 172.21.15.31:0/2340036538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 msgr2=0x7fca60193e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca6559a700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 0x7fca60193e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca6559a700 1 -- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca60105fe0 msgr2=0x7fca6019b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca6559a700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca60105fe0 0x7fca6019b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca6559a700 1 -- 172.21.15.31:0/2340036538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c00f750 con 0x7fca601036c0 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca65d9b700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 0x7fca60193e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca6559a700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca601036c0 0x7fca601943d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00fa40 tx=0x7fca5c00f300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.269+0000 7fca52ffd700 1 -- 172.21.15.31:0/2340036538 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c025070 con 0x7fca601036c0 2023-07-29T16:17:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.270+0000 7fca52ffd700 1 -- 172.21.15.31:0/2340036538 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca5c003db0 con 0x7fca601036c0 2023-07-29T16:17:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.270+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6019bf60 con 0x7fca601036c0 2023-07-29T16:17:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.270+0000 7fca52ffd700 1 -- 172.21.15.31:0/2340036538 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c00ed30 con 0x7fca601036c0 2023-07-29T16:17:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.270+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6019c2d0 con 0x7fca601036c0 2023-07-29T16:17:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.275+0000 7fca52ffd700 1 -- 172.21.15.31:0/2340036538 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fca5c017420 con 0x7fca601036c0 2023-07-29T16:17:58.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:57 smithi039 ceph-mon[128858]: pgmap v444: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.276+0000 7fca7b893b80 1 Processor -- start 2023-07-29T16:17:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.276+0000 7fca7b893b80 1 -- start start 2023-07-29T16:17:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 0x556be78bc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bc590 0x556be78bcd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bd320 0x556be78bd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca7b893b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be7816a90 con 0x556be78bc590 2023-07-29T16:17:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca7b893b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be7814450 con 0x556be78bbc40 2023-07-29T16:17:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca7b893b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be780fd40 con 0x556be78bd320 2023-07-29T16:17:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca6659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 0x556be78bc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.277+0000 7fca6659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 0x556be78bc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42932/0 (socket says 172.21.15.31:42932) 2023-07-29T16:17:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 -- 172.21.15.31:0/732371160 learned_addr learned my addr 172.21.15.31:0/732371160 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6559a700 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bd320 0x556be78bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca65d9b700 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bc590 0x556be78bcd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 -- 172.21.15.31:0/732371160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bd320 msgr2=0x556be78bd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bd320 0x556be78bd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 -- 172.21.15.31:0/732371160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bc590 msgr2=0x556be78bcd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bc590 0x556be78bcd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 -- 172.21.15.31:0/732371160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556be763de60 con 0x556be78bbc40 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca52ffd700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca4c0a0890 0x7fca4c0a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca65d9b700 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bc590 0x556be78bcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca52ffd700 1 -- 172.21.15.31:0/2340036538 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca5c0d0570 con 0x7fca601036c0 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca6659c700 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 0x556be78bc050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca58011680 tx=0x7fca58009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934ab87bd0032a79 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.278+0000 7fca65d9b700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca4c0a0890 0x7fca4c0a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.279+0000 7fca4bfff700 1 -- 172.21.15.31:0/732371160 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5800e040 con 0x556be78bbc40 2023-07-29T16:17:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.279+0000 7fca4bfff700 1 -- 172.21.15.31:0/732371160 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca5800f040 con 0x556be78bbc40 2023-07-29T16:17:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.279+0000 7fca4bfff700 1 -- 172.21.15.31:0/732371160 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580048c0 con 0x556be78bbc40 2023-07-29T16:17:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 -- 172.21.15.31:0/732371160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 msgr2=0x556be78bc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca65d9b700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca4c0a0890 0x7fca4c0a2d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x556be78bc9a0 tx=0x7fca54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 0x556be78bc050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fca58011680 tx=0x7fca58009920 comp rx=0 tx=0).stop 2023-07-29T16:17:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 -- 172.21.15.31:0/732371160 shutdown_connections 2023-07-29T16:17:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bd320 0x556be78bd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bbc40 0x556be78bc050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 --2- 172.21.15.31:0/732371160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bc590 0x556be78bcd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 -- 172.21.15.31:0/732371160 >> 172.21.15.31:0/732371160 conn(0x556be78b4460 msgr2=0x556be78b9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 -- 172.21.15.31:0/732371160 shutdown_connections 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.280+0000 7fca7b893b80 1 -- 172.21.15.31:0/732371160 wait complete. 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.281+0000 7fca7b893b80 1 -- start start 2023-07-29T16:17:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.281+0000 7fca7b893b80 1 Processor -- start 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78d9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78e0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be77ac590 con 0x556be78bd320 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be77ac230 con 0x556be78bc590 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be76939a0 con 0x556be78bbc40 2023-07-29T16:17:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.282+0000 7fca7b893b80 10 client.0 _ll_register_callbacks cb 0x556be77b56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:58.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40326/0 (socket says 172.21.15.31:40326) 2023-07-29T16:17:58.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca65d9b700 1 -- 172.21.15.31:0/1063607949 learned_addr learned my addr 172.21.15.31:0/1063607949 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78d9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6659c700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78e0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 -- 172.21.15.31:0/1063607949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 msgr2=0x556be78d9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 -- 172.21.15.31:0/1063607949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 msgr2=0x556be78e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78e0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 -- 172.21.15.31:0/1063607949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca58011330 con 0x556be78bc590 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca65d9b700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca6559a700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78d9e80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00f560 tx=0x7fca5c00e100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31e7ae937b4ea7a0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.283+0000 7fca497fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca497fa700 1 -- 172.21.15.31:0/1063607949 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c09d420 con 0x556be78bc590 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca497fa700 1 -- 172.21.15.31:0/1063607949 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca5c00ee90 con 0x556be78bc590 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556be783ad90 con 0x556be78bc590 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca497fa700 1 -- 172.21.15.31:0/1063607949 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca5c09db00 con 0x556be78bc590 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca7b893b80 10 client.26378 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556be7626ce0 con 0x556be78bc590 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.284+0000 7fca497fa700 1 -- 172.21.15.31:0/1063607949 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556be783ad90 con 0x556be78bc590 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca497fa700 1 -- 172.21.15.31:0/1063607949 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca5c0a5c60 con 0x556be78bc590 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca7b893b80 10 client.26378 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca7b893b80 10 client.26378 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca7b893b80 20 client.26378 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca7b893b80 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be78e54d0 0x556be78e78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca7b893b80 4 client.26378 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:58.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556be788ddf0 con 0x556be78e54d0 2023-07-29T16:17:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.285+0000 7fca65d9b700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be78e54d0 0x556be78e78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.286+0000 7fca65d9b700 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be78e54d0 0x556be78e78b0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad9ebbaee9ef8cde server_cookie=e91a3e654f87796f in_seq=0 out_seq=0 2023-07-29T16:17:58.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.286+0000 7fca497fa700 10 client.26378 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca497fa700 1 -- 172.21.15.31:0/1063607949 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556be788ddf0 con 0x556be78e54d0 2023-07-29T16:17:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca497fa700 10 client.26378 handle_command_reply: tid=0 2023-07-29T16:17:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca7b893b80 1 client.26378 shutdown 2023-07-29T16:17:58.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 msgr2=0x556be78d9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca7b893b80 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78d9e80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00f560 tx=0x7fca5c00e100 comp rx=0 tx=0).stop 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 shutdown_connections 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca7b893b80 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be78e54d0 0x556be78e78b0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.287+0000 7fca7b893b80 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78e0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca7b893b80 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca7b893b80 1 --2- 172.21.15.31:0/1063607949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78d9e80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca497fa700 0 client.26378 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 >> 172.21.15.31:0/1063607949 conn(0x556be78b4460 msgr2=0x556be78b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 shutdown_connections 2023-07-29T16:17:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca7b893b80 1 -- 172.21.15.31:0/1063607949 wait complete. 2023-07-29T16:17:58.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.288+0000 7fca7b893b80 20 client.26378 trim_cache size 0 max 16384 2023-07-29T16:17:58.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:57 smithi156 ceph-mon[129245]: pgmap v444: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:17:58.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 Processor -- start 2023-07-29T16:17:58.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 -- start start 2023-07-29T16:17:58.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78e0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78c3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be783ad90 con 0x556be78bd320 2023-07-29T16:17:58.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be78dab30 con 0x556be78bc590 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.322+0000 7fca7b893b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be78d8df0 con 0x556be78bbc40 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40336/0 (socket says 172.21.15.31:40336) 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 -- 172.21.15.31:0/3086557315 learned_addr learned my addr 172.21.15.31:0/3086557315 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca6559a700 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78e0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca6659c700 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78c3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 -- 172.21.15.31:0/3086557315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 msgr2=0x556be78e0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78e0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 -- 172.21.15.31:0/3086557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 msgr2=0x556be78c3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78c3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 -- 172.21.15.31:0/3086557315 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556be78bd9c0 con 0x556be78bbc40 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.323+0000 7fca65d9b700 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d8590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca54012e40 tx=0x7fca5401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d713d839d3659bdf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.324+0000 7fca50ff9700 1 -- 172.21.15.31:0/3086557315 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54024040 con 0x556be78bbc40 2023-07-29T16:17:58.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.324+0000 7fca50ff9700 1 -- 172.21.15.31:0/3086557315 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca54011ea0 con 0x556be78bbc40 2023-07-29T16:17:58.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.324+0000 7fca50ff9700 1 -- 172.21.15.31:0/3086557315 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54017070 con 0x556be78bbc40 2023-07-29T16:17:58.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 -- 172.21.15.31:0/3086557315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 msgr2=0x556be78d8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d8590 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fca54012e40 tx=0x7fca5401a000 comp rx=0 tx=0).stop 2023-07-29T16:17:58.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 -- 172.21.15.31:0/3086557315 shutdown_connections 2023-07-29T16:17:58.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bd320 0x556be78c3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bbc40 0x556be78d8590 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 --2- 172.21.15.31:0/3086557315 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bc590 0x556be78e0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 -- 172.21.15.31:0/3086557315 >> 172.21.15.31:0/3086557315 conn(0x556be78b4460 msgr2=0x556be78b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 -- 172.21.15.31:0/3086557315 shutdown_connections 2023-07-29T16:17:58.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.325+0000 7fca7b893b80 1 -- 172.21.15.31:0/3086557315 wait complete. 2023-07-29T16:17:58.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.326+0000 7fca7b893b80 1 -- start start 2023-07-29T16:17:58.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.326+0000 7fca7b893b80 1 Processor -- start 2023-07-29T16:17:58.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 0x556be78df610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bc590 0x556be78dfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bd320 0x556be78e0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be78e1290 con 0x556be78bbc40 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be7937020 con 0x556be78bd320 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556be7937160 con 0x556be78bc590 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca7b893b80 10 client.0 _ll_register_callbacks cb 0x556be77b56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 0x556be78df610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca6559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bc590 0x556be78dfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca65d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 0x556be78df610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37808/0 (socket says 172.21.15.31:37808) 2023-07-29T16:17:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca65d9b700 1 -- 172.21.15.31:0/2938039502 learned_addr learned my addr 172.21.15.31:0/2938039502 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.327+0000 7fca6659c700 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bd320 0x556be78e0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca65d9b700 1 -- 172.21.15.31:0/2938039502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bc590 msgr2=0x556be78dfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca65d9b700 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bc590 0x556be78dfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca65d9b700 1 -- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bd320 msgr2=0x556be78e0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca65d9b700 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bd320 0x556be78e0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca65d9b700 1 -- 172.21.15.31:0/2938039502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca540118f0 con 0x556be78bbc40 2023-07-29T16:17:58.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca65d9b700 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 0x556be78df610 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fca5400f7f0 tx=0x7fca54012470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabd129f7b327177 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.328+0000 7fca497fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:17:58.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca497fa700 1 -- 172.21.15.31:0/2938039502 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54017070 con 0x556be78bbc40 2023-07-29T16:17:58.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca497fa700 1 -- 172.21.15.31:0/2938039502 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca5400fa40 con 0x556be78bbc40 2023-07-29T16:17:58.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca5800f040 con 0x556be78bbc40 2023-07-29T16:17:58.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca497fa700 1 -- 172.21.15.31:0/2938039502 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca54024040 con 0x556be78bbc40 2023-07-29T16:17:58.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca7b893b80 10 client.16404 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:58.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556be76642d0 con 0x556be78bbc40 2023-07-29T16:17:58.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.329+0000 7fca497fa700 1 -- 172.21.15.31:0/2938039502 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca5800f040 con 0x556be78bbc40 2023-07-29T16:17:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca497fa700 1 -- 172.21.15.31:0/2938039502 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca54029960 con 0x556be78bbc40 2023-07-29T16:17:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca7b893b80 10 client.16404 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca7b893b80 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca7b893b80 20 client.16404 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca7b893b80 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be7938ae0 0x556be793aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca7b893b80 4 client.16404 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556be7808dc0 con 0x556be7938ae0 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.330+0000 7fca6559a700 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be7938ae0 0x556be793aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.331+0000 7fca6559a700 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be7938ae0 0x556be793aef0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb12b7aa877fab2 server_cookie=7772235fef1a1c84 in_seq=0 out_seq=0 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.331+0000 7fca497fa700 10 client.16404 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.332+0000 7fca497fa700 1 -- 172.21.15.31:0/2938039502 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556be7808dc0 con 0x556be7938ae0 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.332+0000 7fca497fa700 10 client.16404 handle_command_reply: tid=0 2023-07-29T16:17:58.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.332+0000 7fca7b893b80 1 client.16404 shutdown 2023-07-29T16:17:58.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 msgr2=0x556be78df610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 0x556be78df610 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fca5400f7f0 tx=0x7fca54012470 comp rx=0 tx=0).stop 2023-07-29T16:17:58.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 shutdown_connections 2023-07-29T16:17:58.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556be7938ae0 0x556be793aef0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556be78bd320 0x556be78e0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556be78bbc40 0x556be78df610 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 --2- 172.21.15.31:0/2938039502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556be78bc590 0x556be78dfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 >> 172.21.15.31:0/2938039502 conn(0x556be78b4460 msgr2=0x556be78d8590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca497fa700 0 client.16404 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 shutdown_connections 2023-07-29T16:17:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 1 -- 172.21.15.31:0/2938039502 wait complete. 2023-07-29T16:17:58.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.333+0000 7fca7b893b80 20 client.16404 trim_cache size 0 max 16384 2023-07-29T16:17:58.340 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:17:58.340 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:17:58.340 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "9575601d-fc52-49da-8e99-a1472b33d61c", 2023-07-29T16:17:58.340 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:17:58.341 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:17:58.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.334+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca4c0a0890 msgr2=0x7fca4c0a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.334+0000 7fca67fff700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca4c0a0890 0x7fca4c0a2d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x556be78bc9a0 tx=0x7fca54009260 comp rx=0 tx=0).stop 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.334+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca601036c0 msgr2=0x7fca601943d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.334+0000 7fca67fff700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca601036c0 0x7fca601943d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00fa40 tx=0x7fca5c00f300 comp rx=0 tx=0).stop 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 shutdown_connections 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca4c0a0890 0x7fca4c0a2d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca60105fe0 0x7fca6019b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca60100da0 0x7fca60193e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 --2- 172.21.15.31:0/2340036538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca601036c0 0x7fca601943d0 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00fa40 tx=0x7fca5c00f300 comp rx=0 tx=0).stop 2023-07-29T16:17:58.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 >> 172.21.15.31:0/2340036538 conn(0x7fca600fa840 msgr2=0x7fca60102250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 shutdown_connections 2023-07-29T16:17:58.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.335+0000 7fca67fff700 1 -- 172.21.15.31:0/2340036538 wait complete. 2023-07-29T16:17:58.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:17:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 -- 172.21.15.31:0/2970054181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4438103eb0 msgr2=0x7f4438104320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 --2- 172.21.15.31:0/2970054181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4438103eb0 0x7f4438104320 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4430009f90 tx=0x7f443000b330 comp rx=0 tx=0).stop 2023-07-29T16:17:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 -- 172.21.15.31:0/2970054181 shutdown_connections 2023-07-29T16:17:58.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 --2- 172.21.15.31:0/2970054181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4438103eb0 0x7f4438104320 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 --2- 172.21.15.31:0/2970054181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44380feb90 0x7f4438100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 --2- 172.21.15.31:0/2970054181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44381014f0 0x7f4438103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 -- 172.21.15.31:0/2970054181 >> 172.21.15.31:0/2970054181 conn(0x7f44380fa840 msgr2=0x7f44380fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 -- 172.21.15.31:0/2970054181 shutdown_connections 2023-07-29T16:17:58.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.706+0000 7f443e5ff700 1 -- 172.21.15.31:0/2970054181 wait complete. 2023-07-29T16:17:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.707+0000 7f443e5ff700 1 Processor -- start 2023-07-29T16:17:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.707+0000 7f443e5ff700 1 -- start start 2023-07-29T16:17:58.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.707+0000 7f443e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 0x7f443819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.707+0000 7f443e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44381014f0 0x7f443819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.707+0000 7f443e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4438103eb0 0x7f44381a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f443e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44381a4330 con 0x7f4438103eb0 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f443e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44381a4470 con 0x7f44380feb90 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f443e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44381a4600 con 0x7f44381014f0 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 0x7f443819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 0x7f443819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42980/0 (socket says 172.21.15.31:42980) 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 -- 172.21.15.31:0/3849871300 learned_addr learned my addr 172.21.15.31:0/3849871300 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f443cb9c700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4438103eb0 0x7f44381a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f44377fe700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44381014f0 0x7f443819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 -- 172.21.15.31:0/3849871300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44381014f0 msgr2=0x7f443819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44381014f0 0x7f443819ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 -- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4438103eb0 msgr2=0x7f44381a3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4438103eb0 0x7f44381a3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f4437fff700 1 -- 172.21.15.31:0/3849871300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430009c40 con 0x7f44380feb90 2023-07-29T16:17:58.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.708+0000 7f44377fe700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44381014f0 0x7f443819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:17:58.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.709+0000 7f4437fff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 0x7f443819c7b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f442800e9a0 tx=0x7f442800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.709+0000 7f44357fa700 1 -- 172.21.15.31:0/3849871300 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428009790 con 0x7f44380feb90 2023-07-29T16:17:58.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.709+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44381a48e0 con 0x7f44380feb90 2023-07-29T16:17:58.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.709+0000 7f44357fa700 1 -- 172.21.15.31:0/3849871300 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f442800cbd0 con 0x7f44380feb90 2023-07-29T16:17:58.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.709+0000 7f44357fa700 1 -- 172.21.15.31:0/3849871300 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428018680 con 0x7f44380feb90 2023-07-29T16:17:58.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.709+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44381a4e40 con 0x7f44380feb90 2023-07-29T16:17:58.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.715+0000 7f44357fa700 1 -- 172.21.15.31:0/3849871300 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f442800cd40 con 0x7f44380feb90 2023-07-29T16:17:58.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.715+0000 7f4451fd8b80 1 Processor -- start 2023-07-29T16:17:58.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.715+0000 7f4451fd8b80 1 -- start start 2023-07-29T16:17:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b1be0 0x55bec45b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b32f0 0x55bec45b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec4509a10 con 0x55bec45b1be0 2023-07-29T16:17:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45073d0 con 0x55bec45b32f0 2023-07-29T16:17:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec4502cc0 con 0x55bec45b2530 2023-07-29T16:17:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f443cb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b1be0 0x55bec45b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.716+0000 7f4437fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45b2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40368/0 (socket says 172.21.15.31:40368) 2023-07-29T16:17:58.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 -- 172.21.15.31:0/771174444 learned_addr learned my addr 172.21.15.31:0/771174444 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f44357fa700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44200a0810 0x7f44200a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 -- 172.21.15.31:0/771174444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b32f0 msgr2=0x55bec45b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f44377fe700 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b32f0 0x55bec45b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b32f0 0x55bec45b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 -- 172.21.15.31:0/771174444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b1be0 msgr2=0x55bec45b1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b1be0 0x55bec45b1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f44357fa700 1 -- 172.21.15.31:0/3849871300 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4428014070 con 0x7f44380feb90 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f4437fff700 1 -- 172.21.15.31:0/771174444 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bec4330de0 con 0x55bec45b2530 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.717+0000 7f44377fe700 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b32f0 0x55bec45b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:17:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.718+0000 7f4437fff700 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45b2d60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f44280c9830 tx=0x7f442801f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c75ebf331d544bd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.718+0000 7f443cb9c700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44200a0810 0x7f44200a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.718+0000 7f441e7fc700 1 -- 172.21.15.31:0/771174444 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428099260 con 0x55bec45b2530 2023-07-29T16:17:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.718+0000 7f441e7fc700 1 -- 172.21.15.31:0/771174444 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4428098fd0 con 0x55bec45b2530 2023-07-29T16:17:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.718+0000 7f441e7fc700 1 -- 172.21.15.31:0/771174444 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428099490 con 0x55bec45b2530 2023-07-29T16:17:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 -- 172.21.15.31:0/771174444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 msgr2=0x55bec45b2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f443cb9c700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44200a0810 0x7f44200a2cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55bec45b1980 tx=0x7f4430009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45b2d60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f44280c9830 tx=0x7f442801f000 comp rx=0 tx=0).stop 2023-07-29T16:17:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 -- 172.21.15.31:0/771174444 shutdown_connections 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b32f0 0x55bec45b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b1be0 0x55bec45b1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/771174444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45b2d60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 -- 172.21.15.31:0/771174444 >> 172.21.15.31:0/771174444 conn(0x55bec45a73e0 msgr2=0x55bec45ac990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 -- 172.21.15.31:0/771174444 shutdown_connections 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.719+0000 7f4451fd8b80 1 -- 172.21.15.31:0/771174444 wait complete. 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.720+0000 7f4451fd8b80 1 -- start start 2023-07-29T16:17:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.720+0000 7f4451fd8b80 1 Processor -- start 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45bb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b2530 0x55bec45bbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b32f0 0x55bec45c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec449f510 con 0x55bec45b2530 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec449f1b0 con 0x55bec45b1be0 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec4386920 con 0x55bec45b32f0 2023-07-29T16:17:58.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.721+0000 7f4451fd8b80 10 client.0 _ll_register_callbacks cb 0x55bec44a8620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:58.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f443cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b32f0 0x55bec45c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45bb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43000/0 (socket says 172.21.15.31:43000) 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 -- 172.21.15.31:0/3112066734 learned_addr learned my addr 172.21.15.31:0/3112066734 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f44377fe700 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b2530 0x55bec45bbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 -- 172.21.15.31:0/3112066734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b32f0 msgr2=0x55bec45c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b32f0 0x55bec45c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 -- 172.21.15.31:0/3112066734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b2530 msgr2=0x55bec45bbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b2530 0x55bec45bbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 -- 172.21.15.31:0/3112066734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44381a4e40 con 0x55bec45b1be0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f4437fff700 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45bb770 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4428099190 tx=0x7f4428098ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=244e6af09f64a38f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.722+0000 7f44177fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f44177fe700 1 -- 172.21.15.31:0/3112066734 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44280a18e0 con 0x55bec45b1be0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f44177fe700 1 -- 172.21.15.31:0/3112066734 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4428099490 con 0x55bec45b1be0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bec452dd10 con 0x55bec45b1be0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f44177fe700 1 -- 172.21.15.31:0/3112066734 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f442809d120 con 0x55bec45b1be0 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 10 client.26393 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:58.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bec4319c60 con 0x55bec45b1be0 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f44177fe700 1 -- 172.21.15.31:0/3112066734 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bec452dd10 con 0x55bec45b1be0 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f44177fe700 1 -- 172.21.15.31:0/3112066734 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f44280a14e0 con 0x55bec45b1be0 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 10 client.26393 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 10 client.26393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 20 client.26393 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec45c70f0 0x55bec45c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 4 client.26393 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:17:58.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.723+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bec4580d70 con 0x55bec45c70f0 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.724+0000 7f44377fe700 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec45c70f0 0x55bec45c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.724+0000 7f44177fe700 10 client.26393 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.724+0000 7f44377fe700 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec45c70f0 0x55bec45c94d0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b348e23e1ef389e server_cookie=8530ad7f68dc77df in_seq=0 out_seq=0 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.725+0000 7f44177fe700 1 -- 172.21.15.31:0/3112066734 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bec4580d70 con 0x55bec45c70f0 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.725+0000 7f44177fe700 10 client.26393 handle_command_reply: tid=0 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.725+0000 7f4451fd8b80 1 client.26393 shutdown 2023-07-29T16:17:58.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 msgr2=0x55bec45bb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45bb770 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4428099190 tx=0x7f4428098ee0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 shutdown_connections 2023-07-29T16:17:58.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec45c70f0 0x55bec45c94d0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b32f0 0x55bec45c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45bb770 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3112066734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b2530 0x55bec45bbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 >> 172.21.15.31:0/3112066734 conn(0x55bec45a73e0 msgr2=0x55bec45ac060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f44177fe700 0 client.26393 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 shutdown_connections 2023-07-29T16:17:58.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3112066734 wait complete. 2023-07-29T16:17:58.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.726+0000 7f4451fd8b80 20 client.26393 trim_cache size 0 max 16384 2023-07-29T16:17:58.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.759+0000 7f4451fd8b80 1 Processor -- start 2023-07-29T16:17:58.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.759+0000 7f4451fd8b80 1 -- start start 2023-07-29T16:17:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45b9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45b2b90 con 0x55bec45b32f0 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45a46e0 con 0x55bec45b1be0 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45b94b0 con 0x55bec45b2530 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45b9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43012/0 (socket says 172.21.15.31:43012) 2023-07-29T16:17:58.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f4437fff700 1 -- 172.21.15.31:0/3608763692 learned_addr learned my addr 172.21.15.31:0/3608763692 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f44377fe700 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.760+0000 7f443cb9c700 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f4437fff700 1 -- 172.21.15.31:0/3608763692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 msgr2=0x55bec45c2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f4437fff700 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f4437fff700 1 -- 172.21.15.31:0/3608763692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 msgr2=0x55bec45b9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f4437fff700 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45b9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f4437fff700 1 -- 172.21.15.31:0/3608763692 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bec4357250 con 0x55bec45b1be0 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f4437fff700 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45b9700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f44280c9ba0 tx=0x7f44280cf5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab3dc0131925fb55 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f441effd700 1 -- 172.21.15.31:0/3608763692 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442806bd40 con 0x55bec45b1be0 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f441effd700 1 -- 172.21.15.31:0/3608763692 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44280cbe90 con 0x55bec45b1be0 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.761+0000 7f441effd700 1 -- 172.21.15.31:0/3608763692 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44280983f0 con 0x55bec45b1be0 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3608763692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 msgr2=0x55bec45b9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45b9700 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f44280c9ba0 tx=0x7f44280cf5f0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3608763692 shutdown_connections 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45b9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45b9700 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3608763692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3608763692 >> 172.21.15.31:0/3608763692 conn(0x55bec45a73e0 msgr2=0x55bec45b6230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3608763692 shutdown_connections 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.762+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3608763692 wait complete. 2023-07-29T16:17:58.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 -- start start 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 Processor -- start 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45c2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45befa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45bf520 con 0x55bec45b32f0 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45bf690 con 0x55bec45b1be0 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.763+0000 7f4451fd8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bec45bf850 con 0x55bec45b2530 2023-07-29T16:17:58.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f4451fd8b80 10 client.0 _ll_register_callbacks cb 0x55bec44a8620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:17:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45c2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f443cb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45befa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f44377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f4437fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45c2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43018/0 (socket says 172.21.15.31:43018) 2023-07-29T16:17:58.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f4437fff700 1 -- 172.21.15.31:0/3620154319 learned_addr learned my addr 172.21.15.31:0/3620154319 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:17:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f44377fe700 1 -- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 msgr2=0x55bec45c2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f44377fe700 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45c2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f44377fe700 1 -- 172.21.15.31:0/3620154319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 msgr2=0x55bec45befa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:17:58.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f44377fe700 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45befa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f44377fe700 1 -- 172.21.15.31:0/3620154319 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bec4319c60 con 0x55bec45b2530 2023-07-29T16:17:58.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.764+0000 7f4437fff700 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:17:58.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f44377fe700 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2a40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55bec45ad8b0 tx=0x7f442c01caa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b38c26e69e8239c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:17:58.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f44177fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:17:58.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f44177fe700 1 -- 172.21.15.31:0/3620154319 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c017040 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4428098fd0 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f44177fe700 1 -- 172.21.15.31:0/3620154319 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f442c004250 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f44177fe700 1 -- 172.21.15.31:0/3620154319 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f442c004410 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f4451fd8b80 10 client.26416 fetch_fsmap learned FSMap version 23 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.765+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bec432f2b0 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f44177fe700 1 -- 172.21.15.31:0/3620154319 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4428098fd0 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f44177fe700 1 -- 172.21.15.31:0/3620154319 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f442c025650 con 0x55bec45b2530 2023-07-29T16:17:58.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4451fd8b80 10 client.26416 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4451fd8b80 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4451fd8b80 20 client.26416 populate_metadata read hostname 'smithi031' 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec461a170 0x55bec461c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4451fd8b80 4 client.26416 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bec44fbd40 con 0x55bec461a170 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.766+0000 7f4437fff700 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec461a170 0x55bec461c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:17:58.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f4437fff700 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec461a170 0x55bec461c580 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47da55907ee6fd92 server_cookie=f7c3e32cdbe3824a in_seq=0 out_seq=0 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f44177fe700 10 client.26416 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f44177fe700 1 -- 172.21.15.31:0/3620154319 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55bec44fbd40 con 0x55bec461a170 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f44177fe700 10 client.26416 handle_command_reply: tid=0 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f4451fd8b80 1 client.26416 shutdown 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 msgr2=0x55bec45c2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.767+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2a40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x55bec45ad8b0 tx=0x7f442c01caa0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 shutdown_connections 2023-07-29T16:17:58.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bec461a170 0x55bec461c580 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bec45b32f0 0x55bec45befa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bec45b1be0 0x55bec45c2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 --2- 172.21.15.31:0/3620154319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bec45b2530 0x55bec45c2a40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 >> 172.21.15.31:0/3620154319 conn(0x55bec45a73e0 msgr2=0x55bec45b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f44177fe700 0 client.26416 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 shutdown_connections 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 1 -- 172.21.15.31:0/3620154319 wait complete. 2023-07-29T16:17:58.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.768+0000 7f4451fd8b80 20 client.26416 trim_cache size 0 max 16384 2023-07-29T16:17:58.774 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:17:58.774 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:17:58.774 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:17:58.774 INFO:teuthology.orchestra.run.smithi031.stdout: "9575601d-fc52-49da-8e99-a1472b33d61c": { 2023-07-29T16:17:58.774 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:17:58.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "9575601d-fc52-49da-8e99-a1472b33d61c", 2023-07-29T16:17:58.775 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:17:58.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:17:58.775 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:17:58.775 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:17:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44200a0810 msgr2=0x7f44200a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44200a0810 0x7f44200a2cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55bec45b1980 tx=0x7f4430009920 comp rx=0 tx=0).stop 2023-07-29T16:17:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 msgr2=0x7f443819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:17:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 0x7f443819c7b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f442800e9a0 tx=0x7f442800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 shutdown_connections 2023-07-29T16:17:58.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f44200a0810 0x7f44200a2cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4438103eb0 0x7f44381a3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44380feb90 0x7f443819c7b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 --2- 172.21.15.31:0/3849871300 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44381014f0 0x7f443819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:17:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 >> 172.21.15.31:0/3849871300 conn(0x7f44380fa840 msgr2=0x7f44380ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:17:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 shutdown_connections 2023-07-29T16:17:58.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:17:58.769+0000 7f443e5ff700 1 -- 172.21.15.31:0/3849871300 wait complete. 2023-07-29T16:17:58.794 INFO:tasks.cephfs.filesystem:scrub status for tag:9575601d-fc52-49da-8e99-a1472b33d61c - {'path': '/', 'tag': '9575601d-fc52-49da-8e99-a1472b33d61c', 'options': 'recursive,force'} 2023-07-29T16:17:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:58 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:17:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:17:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:59.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:58 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:17:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:17:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:59.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:58 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:17:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:17:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:17:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:00.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:17:59 smithi031 ceph-mon[95504]: pgmap v445: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:17:59 smithi039 ceph-mon[128858]: pgmap v445: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:17:59 smithi156 ceph-mon[129245]: pgmap v445: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:02.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:02 smithi031 ceph-mon[95504]: pgmap v446: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:18:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:02 smithi039 ceph-mon[128858]: pgmap v446: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:18:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:02 smithi156 ceph-mon[129245]: pgmap v446: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:18:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 -- 172.21.15.31:0/2133036919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818105640 msgr2=0x7ff818107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 --2- 172.21.15.31:0/2133036919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818105640 0x7ff818107a20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009990 tx=0x7ff80c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 -- 172.21.15.31:0/2133036919 shutdown_connections 2023-07-29T16:18:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 --2- 172.21.15.31:0/2133036919 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818107f60 0x7ff81810a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 --2- 172.21.15.31:0/2133036919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff818068e80 0x7ff818105100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 --2- 172.21.15.31:0/2133036919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818105640 0x7ff818107a20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 -- 172.21.15.31:0/2133036919 >> 172.21.15.31:0/2133036919 conn(0x7ff8180fa690 msgr2=0x7ff8180fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 -- 172.21.15.31:0/2133036919 shutdown_connections 2023-07-29T16:18:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.600+0000 7ff81d39d700 1 -- 172.21.15.31:0/2133036919 wait complete. 2023-07-29T16:18:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.601+0000 7ff81d39d700 1 Processor -- start 2023-07-29T16:18:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 -- start start 2023-07-29T16:18:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818068e80 0x7ff818198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818105640 0x7ff818198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff818107f60 0x7ff81819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81819fc90 con 0x7ff818107f60 2023-07-29T16:18:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81819fdd0 con 0x7ff818068e80 2023-07-29T16:18:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.602+0000 7ff81d39d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81819ff60 con 0x7ff818105640 2023-07-29T16:18:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff8177fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818105640 0x7ff818198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818068e80 0x7ff818198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff8177fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818105640 0x7ff818198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:60854/0 (socket says 172.21.15.31:60854) 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff8177fe700 1 -- 172.21.15.31:0/264650406 learned_addr learned my addr 172.21.15.31:0/264650406 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff818107f60 0x7ff81819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 -- 172.21.15.31:0/264650406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818105640 msgr2=0x7ff818198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818105640 0x7ff818198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 -- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff818107f60 msgr2=0x7ff81819f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff818107f60 0x7ff81819f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 -- 172.21.15.31:0/264650406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0095f0 con 0x7ff818068e80 2023-07-29T16:18:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.603+0000 7ff817fff700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818068e80 0x7ff818198110 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff80800e9c0 tx=0x7ff80800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.604+0000 7ff8157fa700 1 -- 172.21.15.31:0/264650406 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80800c9c0 con 0x7ff818068e80 2023-07-29T16:18:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.604+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8181a0240 con 0x7ff818068e80 2023-07-29T16:18:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.604+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8181a07a0 con 0x7ff818068e80 2023-07-29T16:18:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.604+0000 7ff8157fa700 1 -- 172.21.15.31:0/264650406 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff808010430 con 0x7ff818068e80 2023-07-29T16:18:03.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.604+0000 7ff8157fa700 1 -- 172.21.15.31:0/264650406 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808018680 con 0x7ff818068e80 2023-07-29T16:18:03.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.609+0000 7ff831e83b80 1 Processor -- start 2023-07-29T16:18:03.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.609+0000 7ff8157fa700 1 -- 172.21.15.31:0/264650406 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff80800cb20 con 0x7ff818068e80 2023-07-29T16:18:03.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.609+0000 7ff831e83b80 1 -- start start 2023-07-29T16:18:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.609+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da81bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.609+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da81c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.609+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da81d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff831e83b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da7739b0 con 0x5631da81c150 2023-07-29T16:18:03.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff831e83b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da771370 con 0x5631da81cf00 2023-07-29T16:18:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff831e83b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da76cc60 con 0x5631da6776b0 2023-07-29T16:18:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff81cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da81bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff81cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da81bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:60860/0 (socket says 172.21.15.31:60860) 2023-07-29T16:18:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff81cb9c700 1 -- 172.21.15.31:0/3577906400 learned_addr learned my addr 172.21.15.31:0/3577906400 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff8177fe700 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da81d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff817fff700 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da81c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff8177fe700 1 -- 172.21.15.31:0/3577906400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 msgr2=0x5631da81bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff8177fe700 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da81bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff8177fe700 1 -- 172.21.15.31:0/3577906400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 msgr2=0x5631da81c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff8177fe700 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da81c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff8177fe700 1 -- 172.21.15.31:0/3577906400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631da59ad80 con 0x5631da81cf00 2023-07-29T16:18:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.610+0000 7ff817fff700 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da81c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:18:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.611+0000 7ff8177fe700 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da81d3a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00b680 tx=0x7ff80c005f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ef6479a3a09117 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.611+0000 7ff7fe7fc700 1 -- 172.21.15.31:0/3577906400 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c01d070 con 0x5631da81cf00 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.611+0000 7ff7fe7fc700 1 -- 172.21.15.31:0/3577906400 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff80c01e040 con 0x5631da81cf00 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.611+0000 7ff7fe7fc700 1 -- 172.21.15.31:0/3577906400 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0049c0 con 0x5631da81cf00 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff8157fa700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8000a0810 0x7ff8000a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff8157fa700 1 -- 172.21.15.31:0/264650406 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff808014070 con 0x7ff818068e80 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 -- 172.21.15.31:0/3577906400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 msgr2=0x5631da81d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da81d3a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff80c00b680 tx=0x7ff80c005f00 comp rx=0 tx=0).stop 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8000a0810 0x7ff8000a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 -- 172.21.15.31:0/3577906400 shutdown_connections 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da81d3a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da81bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3577906400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da81c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 -- 172.21.15.31:0/3577906400 >> 172.21.15.31:0/3577906400 conn(0x5631da811380 msgr2=0x5631da8166a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.612+0000 7ff831e83b80 1 -- 172.21.15.31:0/3577906400 shutdown_connections 2023-07-29T16:18:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.613+0000 7ff831e83b80 1 -- 172.21.15.31:0/3577906400 wait complete. 2023-07-29T16:18:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.613+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8000a0810 0x7ff8000a2cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5631da81b970 tx=0x7ff810009100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.613+0000 7ff831e83b80 1 -- start start 2023-07-29T16:18:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 Processor -- start 2023-07-29T16:18:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da825030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81c150 0x5631da825570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 0x5631da82c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da7094b0 con 0x5631da81cf00 2023-07-29T16:18:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da709150 con 0x5631da6776b0 2023-07-29T16:18:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da5f08c0 con 0x5631da81c150 2023-07-29T16:18:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.614+0000 7ff831e83b80 10 client.0 _ll_register_callbacks cb 0x5631da7125c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff81cb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 0x5631da82c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff81cb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 0x5631da82c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43118/0 (socket says 172.21.15.31:43118) 2023-07-29T16:18:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff81cb9c700 1 -- 172.21.15.31:0/1855498078 learned_addr learned my addr 172.21.15.31:0/1855498078 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da825030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff8177fe700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81c150 0x5631da825570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 -- 172.21.15.31:0/1855498078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81c150 msgr2=0x5631da825570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81c150 0x5631da825570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 -- 172.21.15.31:0/1855498078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 msgr2=0x5631da82c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 0x5631da82c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 -- 172.21.15.31:0/1855498078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81810b120 con 0x5631da6776b0 2023-07-29T16:18:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 0x5631da82c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff817fff700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da825030 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff8080c9690 tx=0x7ff808099830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133f6d42853ef21e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.615+0000 7ff7f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff7f77fe700 1 -- 172.21.15.31:0/1855498078 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8080992f0 con 0x5631da6776b0 2023-07-29T16:18:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff7f77fe700 1 -- 172.21.15.31:0/1855498078 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff8080cbe40 con 0x5631da6776b0 2023-07-29T16:18:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631da797cb0 con 0x5631da6776b0 2023-07-29T16:18:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff7f77fe700 1 -- 172.21.15.31:0/1855498078 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff80809d320 con 0x5631da6776b0 2023-07-29T16:18:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff831e83b80 10 client.26411 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5631da583c00 con 0x5631da6776b0 2023-07-29T16:18:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff7f77fe700 1 -- 172.21.15.31:0/1855498078 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631da797cb0 con 0x5631da6776b0 2023-07-29T16:18:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.616+0000 7ff7f77fe700 1 -- 172.21.15.31:0/1855498078 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff8080c9b50 con 0x5631da6776b0 2023-07-29T16:18:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff831e83b80 10 client.26411 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff831e83b80 10 client.26411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff831e83b80 20 client.26411 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da830d40 0x5631da833150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff831e83b80 4 client.26411 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631da7ead10 con 0x5631da830d40 2023-07-29T16:18:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.617+0000 7ff8177fe700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da830d40 0x5631da833150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.618+0000 7ff8177fe700 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da830d40 0x5631da833150 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b3a191ad9a5d973 server_cookie=760176d45c3cd837 in_seq=0 out_seq=0 2023-07-29T16:18:03.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.618+0000 7ff7f77fe700 10 client.26411 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.618+0000 7ff7f77fe700 1 -- 172.21.15.31:0/1855498078 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5631da7ead10 con 0x5631da830d40 2023-07-29T16:18:03.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff7f77fe700 10 client.26411 handle_command_reply: tid=0 2023-07-29T16:18:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 client.26411 shutdown 2023-07-29T16:18:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 msgr2=0x5631da825030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da825030 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff8080c9690 tx=0x7ff808099830 comp rx=0 tx=0).stop 2023-07-29T16:18:03.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 shutdown_connections 2023-07-29T16:18:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da830d40 0x5631da833150 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81cf00 0x5631da82c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da825030 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1855498078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81c150 0x5631da825570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 >> 172.21.15.31:0/1855498078 conn(0x5631da811380 msgr2=0x5631da815df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:03.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff7f77fe700 0 client.26411 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.619+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 shutdown_connections 2023-07-29T16:18:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.620+0000 7ff831e83b80 1 -- 172.21.15.31:0/1855498078 wait complete. 2023-07-29T16:18:03.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.620+0000 7ff831e83b80 20 client.26411 trim_cache size 0 max 16384 2023-07-29T16:18:03.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.637+0000 7ff831e83b80 1 Processor -- start 2023-07-29T16:18:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 -- start start 2023-07-29T16:18:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da823280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da8237c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da82c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da80e4c0 con 0x5631da81c150 2023-07-29T16:18:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da830b60 con 0x5631da81cf00 2023-07-29T16:18:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff831e83b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da82e240 con 0x5631da6776b0 2023-07-29T16:18:03.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff817fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da823280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff817fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da823280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:60886/0 (socket says 172.21.15.31:60886) 2023-07-29T16:18:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff817fff700 1 -- 172.21.15.31:0/1654029791 learned_addr learned my addr 172.21.15.31:0/1654029791 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.638+0000 7ff8177fe700 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da8237c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da82c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff817fff700 1 -- 172.21.15.31:0/1654029791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 msgr2=0x5631da82c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff817fff700 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da82c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff817fff700 1 -- 172.21.15.31:0/1654029791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 msgr2=0x5631da8237c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff817fff700 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da8237c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff817fff700 1 -- 172.21.15.31:0/1654029791 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631da5c11f0 con 0x5631da6776b0 2023-07-29T16:18:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff8177fe700 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da8237c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff817fff700 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da823280 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff8080957f0 tx=0x7ff8080978a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b461521871270acc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.639+0000 7ff7feffd700 1 -- 172.21.15.31:0/1654029791 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808099370 con 0x5631da6776b0 2023-07-29T16:18:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff7feffd700 1 -- 172.21.15.31:0/1654029791 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff80809d320 con 0x5631da6776b0 2023-07-29T16:18:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff7feffd700 1 -- 172.21.15.31:0/1654029791 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808098550 con 0x5631da6776b0 2023-07-29T16:18:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 -- 172.21.15.31:0/1654029791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 msgr2=0x5631da823280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da823280 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff8080957f0 tx=0x7ff8080978a0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 -- 172.21.15.31:0/1654029791 shutdown_connections 2023-07-29T16:18:03.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da81cf00 0x5631da82c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da6776b0 0x5631da823280 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 --2- 172.21.15.31:0/1654029791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da8237c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.640+0000 7ff831e83b80 1 -- 172.21.15.31:0/1654029791 >> 172.21.15.31:0/1654029791 conn(0x5631da811380 msgr2=0x5631da832d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.641+0000 7ff831e83b80 1 -- 172.21.15.31:0/1654029791 shutdown_connections 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.641+0000 7ff831e83b80 1 -- 172.21.15.31:0/1654029791 wait complete. 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.641+0000 7ff831e83b80 1 -- start start 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.641+0000 7ff831e83b80 1 Processor -- start 2023-07-29T16:18:03.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da828020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da828560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 0x5631da82b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da82bac0 con 0x5631da81c150 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da82bc30 con 0x5631da6776b0 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631da82bdf0 con 0x5631da81cf00 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff831e83b80 10 client.0 _ll_register_callbacks cb 0x5631da7125c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:03.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff81cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 0x5631da82b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff81cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 0x5631da82b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:60902/0 (socket says 172.21.15.31:60902) 2023-07-29T16:18:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.642+0000 7ff81cb9c700 1 -- 172.21.15.31:0/3993248466 learned_addr learned my addr 172.21.15.31:0/3993248466 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff81cb9c700 1 -- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 msgr2=0x5631da828020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:18:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da828020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff81cb9c700 1 -- 172.21.15.31:0/3993248466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 msgr2=0x5631da828560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:03.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff8177fe700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da828560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da828560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff81cb9c700 1 -- 172.21.15.31:0/3993248466 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5631da583c00 con 0x5631da81cf00 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff8177fe700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da828560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff81cb9c700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 0x5631da82b580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff810016a60 tx=0x7ff810004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8264e5f7530e3e6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.643+0000 7ff7f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.644+0000 7ff7f77fe700 1 -- 172.21.15.31:0/3993248466 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810007c80 con 0x5631da81cf00 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.644+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff80c01e040 con 0x5631da81cf00 2023-07-29T16:18:03.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.644+0000 7ff7f77fe700 1 -- 172.21.15.31:0/3993248466 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff8100048d0 con 0x5631da81cf00 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.644+0000 7ff7f77fe700 1 -- 172.21.15.31:0/3993248466 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff810007de0 con 0x5631da81cf00 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.644+0000 7ff831e83b80 10 client.26434 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.644+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5631da599250 con 0x5631da81cf00 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff7f77fe700 1 -- 172.21.15.31:0/3993248466 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff80c01e040 con 0x5631da81cf00 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff7f77fe700 1 -- 172.21.15.31:0/3993248466 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff810038540 con 0x5631da81cf00 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff831e83b80 10 client.26434 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff831e83b80 10 client.26434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff831e83b80 20 client.26434 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da88be90 0x5631da88e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff831e83b80 4 client.26434 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:18:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.645+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5631da765ce0 con 0x5631da88be90 2023-07-29T16:18:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.646+0000 7ff817fff700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da88be90 0x5631da88e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.646+0000 7ff7f77fe700 10 client.26434 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.646+0000 7ff817fff700 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da88be90 0x5631da88e2a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4551471aa10c379 server_cookie=2ff19ed1801dd189 in_seq=0 out_seq=0 2023-07-29T16:18:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.650+0000 7ff7f77fe700 1 -- 172.21.15.31:0/3993248466 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5631da765ce0 con 0x5631da88be90 2023-07-29T16:18:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff7f77fe700 10 client.26434 handle_command_reply: tid=0 2023-07-29T16:18:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 client.26434 shutdown 2023-07-29T16:18:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 msgr2=0x5631da82b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 0x5631da82b580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff810016a60 tx=0x7ff810004ad0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 shutdown_connections 2023-07-29T16:18:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631da88be90 0x5631da88e2a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631da81cf00 0x5631da82b580 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631da6776b0 0x5631da828020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff7f77fe700 0 client.26434 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 --2- 172.21.15.31:0/3993248466 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631da81c150 0x5631da828560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 >> 172.21.15.31:0/3993248466 conn(0x5631da811380 msgr2=0x5631da823280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 shutdown_connections 2023-07-29T16:18:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.651+0000 7ff831e83b80 1 -- 172.21.15.31:0/3993248466 wait complete. 2023-07-29T16:18:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff831e83b80 20 client.26434 trim_cache size 0 max 16384 2023-07-29T16:18:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:18:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:18:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:18:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:18:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8000a0810 msgr2=0x7ff8000a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff81d39d700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8000a0810 0x7ff8000a2cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5631da81b970 tx=0x7ff810009100 comp rx=0 tx=0).stop 2023-07-29T16:18:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818068e80 msgr2=0x7ff818198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff81d39d700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818068e80 0x7ff818198110 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff80800e9c0 tx=0x7ff80800ecd0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 shutdown_connections 2023-07-29T16:18:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.652+0000 7ff81d39d700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8000a0810 0x7ff8000a2cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.653+0000 7ff81d39d700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff818107f60 0x7ff81819f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.653+0000 7ff81d39d700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff818068e80 0x7ff818198110 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.653+0000 7ff81d39d700 1 --2- 172.21.15.31:0/264650406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff818105640 0x7ff818198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.653+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 >> 172.21.15.31:0/264650406 conn(0x7ff8180fa690 msgr2=0x7ff8181062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.653+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 shutdown_connections 2023-07-29T16:18:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:03.653+0000 7ff81d39d700 1 -- 172.21.15.31:0/264650406 wait complete. 2023-07-29T16:18:03.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:03 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:18:03.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:03 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:18:03.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:03 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:18:04.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:04 smithi031 ceph-mon[95504]: pgmap v447: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:04 smithi039 ceph-mon[128858]: pgmap v447: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:04 smithi156 ceph-mon[129245]: pgmap v447: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:06.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:06 smithi031 ceph-mon[95504]: pgmap v448: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-07-29T16:18:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:06 smithi039 ceph-mon[128858]: pgmap v448: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-07-29T16:18:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:06 smithi156 ceph-mon[129245]: pgmap v448: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-07-29T16:18:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:07 smithi031 ceph-mon[95504]: pgmap v449: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2023-07-29T16:18:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:07 smithi039 ceph-mon[128858]: pgmap v449: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2023-07-29T16:18:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:07 smithi156 ceph-mon[129245]: pgmap v449: 97 pgs: 97 active+clean; 673 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2023-07-29T16:18:10.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:10 smithi031 ceph-mon[95504]: pgmap v450: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:10.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:10 smithi039 ceph-mon[128858]: pgmap v450: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:10 smithi156 ceph-mon[129245]: pgmap v450: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:12.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:12 smithi031 ceph-mon[95504]: pgmap v451: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:18:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:12 smithi039 ceph-mon[128858]: pgmap v451: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:18:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:12 smithi156 ceph-mon[129245]: pgmap v451: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-29T16:18:13.796 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 64% (7457/11620) Updating files: 65% (7553/11620) Updating files: 66% (7670/11620) Updating files: 67% (7786/11620) Updating files: 68% (7902/11620) Updating files: 69% (8018/11620) Updating files: 70% (8134/11620) Updating files: 71% (8251/11620) Updating files: 72% (8367/11620) Updating files: 73% (8483/11620) Updating files: 74% (8599/11620) Updating files: 75% (8715/11620) Updating files: 76% (8832/11620) Updating files: 77% (8948/11620) Updating files: 78% (9064/11620) Updating files: 79% (9180/11620) Updating files: 80% (9296/11620) Updating files: 81% (9413/11620) Updating files: 82% (9529/11620) Updating files: 83% (9645/11620) Updating files: 84% (9761/11620) Updating files: 85% (9877/11620) Updating files: 86% (9994/11620) Updating files: 87% (10110/11620) Updating files: 88% (10226/11620) Updating files: 89% (10342/11620) Updating files: 90% (10458/11620) Updating files: 91% (10575/11620) Updating files: 92% (10691/11620) Updating files: 93% (10807/11620) Updating files: 94% (10923/11620) Updating files: 95% (11039/11620) Updating files: 96% (11156/11620) Updating files: 97% (11272/11620) Updating files: 98% (11388/11620) Updating files: 99% (11504/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-07-29T16:18:14.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:14 smithi031 ceph-mon[95504]: pgmap v452: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to '0484097b0572e96f42cf6402cbe0ac8dcb046577'. 2023-07-29T16:18:14.702 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-07-29T16:18:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2023-07-29T16:18:14.702 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.703 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2023-07-29T16:18:14.704 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.704 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2023-07-29T16:18:14.704 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.704 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-07-29T16:18:14.704 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-07-29T16:18:14.704 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at 0484097b057 Merge pull request #52618 from vshankar/tr-62147 2023-07-29T16:18:14.714 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-07-29T16:18:14.763 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-07-29T16:18:14.767 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-07-29T16:18:14.767 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-07-29T16:18:14.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:14 smithi039 ceph-mon[128858]: pgmap v452: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:14 smithi156 ceph-mon[129245]: pgmap v452: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2023-07-29T16:18:15.339 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-07-29T16:18:15.449 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-07-29T16:18:15.514 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-07-29T16:18:15.518 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-07-29T16:18:15.518 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-07-29T16:18:15.581 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-07-29T16:18:15.588 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:18:15.589 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-07-29T16:18:15.631 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2023-07-29T16:18:15.633 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2023-07-29T16:18:15.633 DEBUG:teuthology.orchestra.run.smithi031:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0484097b0572e96f42cf6402cbe0ac8dcb046577 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2023-07-29T16:18:15.747 INFO:tasks.workunit.client.0.smithi031.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2023-07-29T16:18:15.747 INFO:tasks.workunit.client.0.smithi031.stdout: 2023-07-29T16:18:15.747 INFO:tasks.workunit.client.0.smithi031.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2023-07-29T16:18:15.754 INFO:tasks.workunit.client.0.smithi031.stdout:0 of 1 processes prepared for launch 0 sec 2023-07-29T16:18:15.854 INFO:tasks.workunit.client.0.smithi031.stdout:1 of 1 processes prepared for launch 0 sec 2023-07-29T16:18:15.854 INFO:tasks.workunit.client.0.smithi031.stdout:releasing clients 2023-07-29T16:18:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:16 smithi039 ceph-mon[128858]: pgmap v453: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:18:16.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 -- 172.21.15.31:0/3690901226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec108290 msgr2=0x7f1dec10a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 --2- 172.21.15.31:0/3690901226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec108290 0x7f1dec10a6f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009f90 tx=0x7f1de400b330 comp rx=0 tx=0).stop 2023-07-29T16:18:16.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 -- 172.21.15.31:0/3690901226 shutdown_connections 2023-07-29T16:18:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 --2- 172.21.15.31:0/3690901226 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec108290 0x7f1dec10a6f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 --2- 172.21.15.31:0/3690901226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec074d10 0x7f1dec073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 --2- 172.21.15.31:0/3690901226 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 0x7f1dec073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 -- 172.21.15.31:0/3690901226 >> 172.21.15.31:0/3690901226 conn(0x7f1dec0fba00 msgr2=0x7f1dec0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 -- 172.21.15.31:0/3690901226 shutdown_connections 2023-07-29T16:18:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.817+0000 7f1df31ff700 1 -- 172.21.15.31:0/3690901226 wait complete. 2023-07-29T16:18:16.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.818+0000 7f1df31ff700 1 Processor -- start 2023-07-29T16:18:16.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.818+0000 7f1df31ff700 1 -- start start 2023-07-29T16:18:16.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df31ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 0x7f1dec198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df31ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec074d10 0x7f1dec1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df31ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 0x7f1dec199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df31ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec199ea0 con 0x7f1dec073730 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df31ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19a010 con 0x7f1dec108290 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df31ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec19a1d0 con 0x7f1dec074d10 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df0f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 0x7f1dec198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 0x7f1dec199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 0x7f1dec199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52752/0 (socket says 172.21.15.31:52752) 2023-07-29T16:18:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df0f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 0x7f1dec198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38542/0 (socket says 172.21.15.31:38542) 2023-07-29T16:18:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 -- 172.21.15.31:0/2445783592 learned_addr learned my addr 172.21.15.31:0/2445783592 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1debfff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec074d10 0x7f1dec1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 -- 172.21.15.31:0/2445783592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec074d10 msgr2=0x7f1dec1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec074d10 0x7f1dec1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 -- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 msgr2=0x7f1dec198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 0x7f1dec198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.819+0000 7f1df179c700 1 -- 172.21.15.31:0/2445783592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00095f0 con 0x7f1dec108290 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.820+0000 7f1df179c700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 0x7f1dec199840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009f60 tx=0x7f1de4004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.820+0000 7f1de9ffb700 1 -- 172.21.15.31:0/2445783592 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400e040 con 0x7f1dec108290 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.820+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de4009c40 con 0x7f1dec108290 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.820+0000 7f1de9ffb700 1 -- 172.21.15.31:0/2445783592 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1de4004e80 con 0x7f1dec108290 2023-07-29T16:18:16.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.820+0000 7f1de9ffb700 1 -- 172.21.15.31:0/2445783592 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de40052a0 con 0x7f1dec108290 2023-07-29T16:18:16.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.820+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a0a90 con 0x7f1dec108290 2023-07-29T16:18:16.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:16 smithi156 ceph-mon[129245]: pgmap v453: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:18:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.823+0000 7f1de9ffb700 1 -- 172.21.15.31:0/2445783592 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1de401d470 con 0x7f1dec108290 2023-07-29T16:18:16.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.825+0000 7f1e06a4eb80 1 Processor -- start 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 -- start start 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061374e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061374f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1de9ffb700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1ddc0a0790 0x7f1ddc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613750160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606136a69b0 con 0x56061374fcf0 2023-07-29T16:18:16.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606136a4370 con 0x5606135aa6b0 2023-07-29T16:18:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061369fc60 con 0x56061374edf0 2023-07-29T16:18:16.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1de9ffb700 1 -- 172.21.15.31:0/2445783592 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1de40ccfa0 con 0x7f1dec108290 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1debfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061374e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1debfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061374e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52756/0 (socket says 172.21.15.31:52756) 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1debfff700 1 -- 172.21.15.31:0/3510930964 learned_addr learned my addr 172.21.15.31:0/3510930964 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.826+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1ddc0a0790 0x7f1ddc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061374f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1df179c700 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613750160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 -- 172.21.15.31:0/3510930964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 msgr2=0x56061374f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061374f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 -- 172.21.15.31:0/3510930964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 msgr2=0x560613750160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613750160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 -- 172.21.15.31:0/3510930964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606134cdd80 con 0x5606135aa6b0 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1ddc0a0790 0x7f1ddc0a2c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1de0005fd0 tx=0x7f1de0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1debfff700 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061374e8b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80118e0 tx=0x7f1dd8009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ad1c3b597cb1ef6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:16.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.827+0000 7f1dd6ffd700 1 -- 172.21.15.31:0/3510930964 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8004050 con 0x5606135aa6b0 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1dd6ffd700 1 -- 172.21.15.31:0/3510930964 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1dd8004830 con 0x5606135aa6b0 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1dd6ffd700 1 -- 172.21.15.31:0/3510930964 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8005310 con 0x5606135aa6b0 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3510930964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 msgr2=0x56061374e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061374e8b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80118e0 tx=0x7f1dd8009880 comp rx=0 tx=0).stop 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3510930964 shutdown_connections 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613750160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061374e8b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3510930964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061374f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3510930964 >> 172.21.15.31:0/3510930964 conn(0x560613743fd0 msgr2=0x560613749300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3510930964 shutdown_connections 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.828+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3510930964 wait complete. 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.829+0000 7f1e06a4eb80 1 -- start start 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.829+0000 7f1e06a4eb80 1 Processor -- start 2023-07-29T16:18:16.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606135aa6b0 0x560613757f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 0x560613758440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x56061375f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061363c4b0 con 0x56061374fcf0 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061363c150 con 0x56061374edf0 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606135238c0 con 0x5606135aa6b0 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1e06a4eb80 10 client.0 _ll_register_callbacks cb 0x5606136455c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1debfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 0x560613758440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1debfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 0x560613758440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52770/0 (socket says 172.21.15.31:52770) 2023-07-29T16:18:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1debfff700 1 -- 172.21.15.31:0/3663077719 learned_addr learned my addr 172.21.15.31:0/3663077719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df179c700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x56061375f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606135aa6b0 0x560613757f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df0f9b700 1 -- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 msgr2=0x560613758440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 0x560613758440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df0f9b700 1 -- 172.21.15.31:0/3663077719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 msgr2=0x56061375f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x56061375f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df0f9b700 1 -- 172.21.15.31:0/3663077719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd8019040 con 0x5606135aa6b0 2023-07-29T16:18:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1debfff700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 0x560613758440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.830+0000 7f1df179c700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x56061375f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606135aa6b0 0x560613757f00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1de000ec50 tx=0x7f1de00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29a40d4c7c3aaba0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1dcbfff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1dcbfff700 1 -- 172.21.15.31:0/3663077719 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de001c8b0 con 0x5606135aa6b0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1dcbfff700 1 -- 172.21.15.31:0/3663077719 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1de0023070 con 0x5606135aa6b0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5606136cacb0 con 0x5606135aa6b0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1dcbfff700 1 -- 172.21.15.31:0/3663077719 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1de001ca10 con 0x5606135aa6b0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1e06a4eb80 10 client.26446 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.831+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5606134b6c00 con 0x5606135aa6b0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1dcbfff700 1 -- 172.21.15.31:0/3663077719 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5606136cacb0 con 0x5606135aa6b0 2023-07-29T16:18:16.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1dcbfff700 1 -- 172.21.15.31:0/3663077719 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f1de0032410 con 0x5606135aa6b0 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1e06a4eb80 10 client.26446 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1e06a4eb80 10 client.26446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1e06a4eb80 20 client.26446 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560613763c50 0x560613766060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1e06a4eb80 4 client.26446 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56061371dd10 con 0x560613763c50 2023-07-29T16:18:16.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1debfff700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560613763c50 0x560613766060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1debfff700 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560613763c50 0x560613766060 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96cc0d2c81bbfa96 server_cookie=573891b8a8ed2a8b in_seq=0 out_seq=0 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.832+0000 7f1dcbfff700 10 client.26446 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.833+0000 7f1dcbfff700 1 -- 172.21.15.31:0/3663077719 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56061371dd10 con 0x560613763c50 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.833+0000 7f1dcbfff700 10 client.26446 handle_command_reply: tid=0 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.833+0000 7f1e06a4eb80 1 client.26446 shutdown 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.833+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606135aa6b0 msgr2=0x560613757f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.833+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606135aa6b0 0x560613757f00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1de000ec50 tx=0x7f1de00047c0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 shutdown_connections 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560613763c50 0x560613766060 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x56061375f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606135aa6b0 0x560613757f00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/3663077719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56061374edf0 0x560613758440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1dcbfff700 0 client.26446 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 >> 172.21.15.31:0/3663077719 conn(0x560613743fd0 msgr2=0x560613748a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:16.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 shutdown_connections 2023-07-29T16:18:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/3663077719 wait complete. 2023-07-29T16:18:16.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.834+0000 7f1e06a4eb80 20 client.26446 trim_cache size 0 max 16384 2023-07-29T16:18:16.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.849+0000 7f1e06a4eb80 1 Processor -- start 2023-07-29T16:18:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 -- start start 2023-07-29T16:18:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x5606137500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061375f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613756110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560613759080 con 0x56061374fcf0 2023-07-29T16:18:16.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560613763a40 con 0x5606135aa6b0 2023-07-29T16:18:16.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560613750600 con 0x56061374edf0 2023-07-29T16:18:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x5606137500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df179c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613756110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x5606137500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52784/0 (socket says 172.21.15.31:52784) 2023-07-29T16:18:16.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 -- 172.21.15.31:0/1388630239 learned_addr learned my addr 172.21.15.31:0/1388630239 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1debfff700 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061375f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 -- 172.21.15.31:0/1388630239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 msgr2=0x56061375f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061375f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 -- 172.21.15.31:0/1388630239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 msgr2=0x560613756110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613756110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df0f9b700 1 -- 172.21.15.31:0/1388630239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606134f41f0 con 0x5606135aa6b0 2023-07-29T16:18:16.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.850+0000 7f1df179c700 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613756110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.851+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x5606137500c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1de0000c00 tx=0x7f1de0032910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6d814db4d4668a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.851+0000 7f1dd77fe700 1 -- 172.21.15.31:0/1388630239 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0038070 con 0x5606135aa6b0 2023-07-29T16:18:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.851+0000 7f1dd77fe700 1 -- 172.21.15.31:0/1388630239 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1de0021070 con 0x5606135aa6b0 2023-07-29T16:18:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.851+0000 7f1dd77fe700 1 -- 172.21.15.31:0/1388630239 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de001cb70 con 0x5606135aa6b0 2023-07-29T16:18:16.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1388630239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 msgr2=0x5606137500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x5606137500c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1de0000c00 tx=0x7f1de0032910 comp rx=0 tx=0).stop 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1388630239 shutdown_connections 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374fcf0 0x560613756110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x5606137500c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1388630239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374edf0 0x56061375f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1388630239 >> 172.21.15.31:0/1388630239 conn(0x560613743fd0 msgr2=0x560613756bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:16.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1388630239 shutdown_connections 2023-07-29T16:18:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1388630239 wait complete. 2023-07-29T16:18:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 -- start start 2023-07-29T16:18:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.852+0000 7f1e06a4eb80 1 Processor -- start 2023-07-29T16:18:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061375eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374edf0 0x56061375f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374fcf0 0x56061375b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061375be10 con 0x56061374edf0 2023-07-29T16:18:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061375bf80 con 0x5606135aa6b0 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061375c140 con 0x56061374fcf0 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1e06a4eb80 10 client.0 _ll_register_callbacks cb 0x5606136455c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1df0f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061375eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1df0f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061375eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52792/0 (socket says 172.21.15.31:52792) 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1debfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374edf0 0x56061375f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1df0f9b700 1 -- 172.21.15.31:0/1847101675 learned_addr learned my addr 172.21.15.31:0/1847101675 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1df179c700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374fcf0 0x56061375b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.853+0000 7f1debfff700 1 -- 172.21.15.31:0/1847101675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374fcf0 msgr2=0x56061375b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1debfff700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374fcf0 0x56061375b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1debfff700 1 -- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 msgr2=0x56061375eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1debfff700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061375eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1debfff700 1 -- 172.21.15.31:0/1847101675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a0a90 con 0x56061374edf0 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061375eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1df179c700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374fcf0 0x56061375b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1debfff700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374edf0 0x56061375f2e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x560613766910 tx=0x7f1dd8009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d2db765343de87 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1dcbfff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1dcbfff700 1 -- 172.21.15.31:0/1847101675 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8024040 con 0x56061374edf0 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1dd8004830 con 0x56061374edf0 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1dcbfff700 1 -- 172.21.15.31:0/1847101675 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1dd8009cb0 con 0x56061374edf0 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1dcbfff700 1 -- 172.21.15.31:0/1847101675 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dd802d040 con 0x56061374edf0 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1e06a4eb80 10 client.16452 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.854+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5606134b6c00 con 0x56061374edf0 2023-07-29T16:18:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1dcbfff700 1 -- 172.21.15.31:0/1847101675 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1dd8004830 con 0x56061374edf0 2023-07-29T16:18:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1dcbfff700 1 -- 172.21.15.31:0/1847101675 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f1dd801d070 con 0x56061374edf0 2023-07-29T16:18:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1e06a4eb80 10 client.16452 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1e06a4eb80 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1e06a4eb80 20 client.16452 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606137b6ac0 0x5606137b8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1e06a4eb80 4 client.16452 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:18:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560613698ce0 con 0x5606137b6ac0 2023-07-29T16:18:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.855+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606137b6ac0 0x5606137b8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:16.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 2436 28.10 MB/sec warmup 1 sec latency 9.112 ms 2023-07-29T16:18:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.856+0000 7f1df0f9b700 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606137b6ac0 0x5606137b8ea0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8c1961e97ca5ebb server_cookie=3a299c131ae5d719 in_seq=0 out_seq=0 2023-07-29T16:18:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.856+0000 7f1dcbfff700 10 client.16452 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:16.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1dcbfff700 1 -- 172.21.15.31:0/1847101675 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560613698ce0 con 0x5606137b6ac0 2023-07-29T16:18:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1dcbfff700 10 client.16452 handle_command_reply: tid=0 2023-07-29T16:18:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 client.16452 shutdown 2023-07-29T16:18:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374edf0 msgr2=0x56061375f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374edf0 0x56061375f2e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x560613766910 tx=0x7f1dd8009f60 comp rx=0 tx=0).stop 2023-07-29T16:18:16.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 shutdown_connections 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606137b6ac0 0x5606137b8ea0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56061374fcf0 0x56061375b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606135aa6b0 0x56061375eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1dcbfff700 0 client.16452 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 --2- 172.21.15.31:0/1847101675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56061374edf0 0x56061375f2e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.911+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 >> 172.21.15.31:0/1847101675 conn(0x560613743fd0 msgr2=0x560613756110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:16.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 shutdown_connections 2023-07-29T16:18:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1e06a4eb80 1 -- 172.21.15.31:0/1847101675 wait complete. 2023-07-29T16:18:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1e06a4eb80 20 client.16452 trim_cache size 0 max 16384 2023-07-29T16:18:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:18:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:18:16.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:18:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:18:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1ddc0a0790 msgr2=0x7f1ddc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1df31ff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1ddc0a0790 0x7f1ddc0a2c40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1de0005fd0 tx=0x7f1de0009310 comp rx=0 tx=0).stop 2023-07-29T16:18:16.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 msgr2=0x7f1dec199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1df31ff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 0x7f1dec199840 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009f60 tx=0x7f1de4004c60 comp rx=0 tx=0).stop 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.912+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 shutdown_connections 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1ddc0a0790 0x7f1ddc0a2c40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dec108290 0x7f1dec199840 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dec073730 0x7f1dec198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 --2- 172.21.15.31:0/2445783592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dec074d10 0x7f1dec1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 >> 172.21.15.31:0/2445783592 conn(0x7f1dec0fba00 msgr2=0x7f1dec104220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:16.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 shutdown_connections 2023-07-29T16:18:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:16.913+0000 7f1df31ff700 1 -- 172.21.15.31:0/2445783592 wait complete. 2023-07-29T16:18:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:16 smithi031 ceph-mon[95504]: pgmap v453: 97 pgs: 97 active+clean; 674 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-29T16:18:17.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 4349 20.24 MB/sec warmup 2 sec latency 19.074 ms 2023-07-29T16:18:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:17 smithi031 ceph-mon[95504]: pgmap v454: 97 pgs: 97 active+clean; 802 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-07-29T16:18:18.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:17 smithi039 ceph-mon[128858]: pgmap v454: 97 pgs: 97 active+clean; 802 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-07-29T16:18:18.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:17 smithi156 ceph-mon[129245]: pgmap v454: 97 pgs: 97 active+clean; 802 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-07-29T16:18:18.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 7328 19.63 MB/sec warmup 3 sec latency 7.334 ms 2023-07-29T16:18:19.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 9492 16.55 MB/sec warmup 4 sec latency 9.303 ms 2023-07-29T16:18:20.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:20 smithi031 ceph-mon[95504]: pgmap v455: 97 pgs: 97 active+clean; 2.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 220 KiB/s wr, 55 op/s 2023-07-29T16:18:20.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:20 smithi039 ceph-mon[128858]: pgmap v455: 97 pgs: 97 active+clean; 2.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 220 KiB/s wr, 55 op/s 2023-07-29T16:18:20.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:20 smithi156 ceph-mon[129245]: pgmap v455: 97 pgs: 97 active+clean; 2.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 220 KiB/s wr, 55 op/s 2023-07-29T16:18:20.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 11372 15.71 MB/sec warmup 5 sec latency 16.927 ms 2023-07-29T16:18:21.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 13854 15.27 MB/sec warmup 6 sec latency 94.684 ms 2023-07-29T16:18:22.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:22 smithi031 ceph-mon[95504]: pgmap v456: 97 pgs: 97 active+clean; 2.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 220 KiB/s wr, 53 op/s 2023-07-29T16:18:22.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:22 smithi039 ceph-mon[128858]: pgmap v456: 97 pgs: 97 active+clean; 2.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 220 KiB/s wr, 53 op/s 2023-07-29T16:18:22.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:22 smithi156 ceph-mon[129245]: pgmap v456: 97 pgs: 97 active+clean; 2.8 MiB data, 339 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 220 KiB/s wr, 53 op/s 2023-07-29T16:18:22.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 15643 14.64 MB/sec warmup 7 sec latency 9.583 ms 2023-07-29T16:18:23.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 18127 14.57 MB/sec warmup 8 sec latency 7.886 ms 2023-07-29T16:18:24.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:24 smithi031 ceph-mon[95504]: pgmap v457: 97 pgs: 97 active+clean; 7.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 697 KiB/s wr, 124 op/s 2023-07-29T16:18:24.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:24 smithi039 ceph-mon[128858]: pgmap v457: 97 pgs: 97 active+clean; 7.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 697 KiB/s wr, 124 op/s 2023-07-29T16:18:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:24 smithi156 ceph-mon[129245]: pgmap v457: 97 pgs: 97 active+clean; 7.6 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 697 KiB/s wr, 124 op/s 2023-07-29T16:18:24.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 20385 13.85 MB/sec warmup 9 sec latency 8.575 ms 2023-07-29T16:18:25.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 22208 13.64 MB/sec warmup 10 sec latency 19.919 ms 2023-07-29T16:18:26.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:26 smithi031 ceph-mon[95504]: pgmap v458: 97 pgs: 97 active+clean; 8.9 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 829 KiB/s wr, 130 op/s 2023-07-29T16:18:26.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:26 smithi039 ceph-mon[128858]: pgmap v458: 97 pgs: 97 active+clean; 8.9 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 829 KiB/s wr, 130 op/s 2023-07-29T16:18:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:26 smithi156 ceph-mon[129245]: pgmap v458: 97 pgs: 97 active+clean; 8.9 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 829 KiB/s wr, 130 op/s 2023-07-29T16:18:26.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 24681 13.66 MB/sec warmup 11 sec latency 116.356 ms 2023-07-29T16:18:27.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 26381 13.33 MB/sec warmup 12 sec latency 8.802 ms 2023-07-29T16:18:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:27 smithi031 ceph-mon[95504]: pgmap v459: 97 pgs: 97 active+clean; 9.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 858 KiB/s wr, 132 op/s 2023-07-29T16:18:28.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:27 smithi039 ceph-mon[128858]: pgmap v459: 97 pgs: 97 active+clean; 9.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 858 KiB/s wr, 132 op/s 2023-07-29T16:18:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:27 smithi156 ceph-mon[129245]: pgmap v459: 97 pgs: 97 active+clean; 9.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 858 KiB/s wr, 132 op/s 2023-07-29T16:18:28.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:18:28.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 28785 13.38 MB/sec warmup 13 sec latency 8.843 ms 2023-07-29T16:18:29.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 -- 172.21.15.31:0/9161287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938100320 msgr2=0x7f0938100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 --2- 172.21.15.31:0/9161287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938100320 0x7f0938100730 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0928009990 tx=0x7f0928009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 -- 172.21.15.31:0/9161287 shutdown_connections 2023-07-29T16:18:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 --2- 172.21.15.31:0/9161287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101ea0 0x7f0938108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 --2- 172.21.15.31:0/9161287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938100320 0x7f0938100730 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 --2- 172.21.15.31:0/9161287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938101510 0x7f0938101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 -- 172.21.15.31:0/9161287 >> 172.21.15.31:0/9161287 conn(0x7f09380fb920 msgr2=0x7f09380fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 -- 172.21.15.31:0/9161287 shutdown_connections 2023-07-29T16:18:29.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.012+0000 7f093e9ff700 1 -- 172.21.15.31:0/9161287 wait complete. 2023-07-29T16:18:29.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.013+0000 7f093e9ff700 1 Processor -- start 2023-07-29T16:18:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.013+0000 7f093e9ff700 1 -- start start 2023-07-29T16:18:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093e9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938100320 0x7f0938074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093e9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 0x7f0938073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093e9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938101ea0 0x7f0938073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093e9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938073e20 con 0x7f0938100320 2023-07-29T16:18:29.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093e9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938073f90 con 0x7f0938101510 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093e9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09381a0080 con 0x7f0938101ea0 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 0x7f0938073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f0937fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938100320 0x7f0938074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 0x7f0938073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36418/0 (socket says 172.21.15.31:36418) 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 -- 172.21.15.31:0/891008730 learned_addr learned my addr 172.21.15.31:0/891008730 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093cf9c700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938101ea0 0x7f0938073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 -- 172.21.15.31:0/891008730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938101ea0 msgr2=0x7f0938073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938101ea0 0x7f0938073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 -- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938100320 msgr2=0x7f0938074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938100320 0x7f0938074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f09377fe700 1 -- 172.21.15.31:0/891008730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09280095f0 con 0x7f0938101510 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f093cf9c700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938101ea0 0x7f0938073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.014+0000 7f0937fff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938100320 0x7f0938074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.015+0000 7f09377fe700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 0x7f0938073140 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f092c00e9a0 tx=0x7f092c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.015+0000 7f09357fa700 1 -- 172.21.15.31:0/891008730 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c009790 con 0x7f0938101510 2023-07-29T16:18:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.015+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09381a0360 con 0x7f0938101510 2023-07-29T16:18:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.015+0000 7f09357fa700 1 -- 172.21.15.31:0/891008730 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f092c00cbd0 con 0x7f0938101510 2023-07-29T16:18:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.015+0000 7f09357fa700 1 -- 172.21.15.31:0/891008730 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c018680 con 0x7f0938101510 2023-07-29T16:18:29.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.015+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09381a08c0 con 0x7f0938101510 2023-07-29T16:18:29.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.018+0000 7f09357fa700 1 -- 172.21.15.31:0/891008730 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f092c00cd40 con 0x7f0938101510 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.019+0000 7f0952344b80 1 Processor -- start 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.019+0000 7f0952344b80 1 -- start start 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0952344b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 0x5613f11cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0952344b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cc530 0x5613f11ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0952344b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cd2f0 0x5613f11cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0952344b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f1123a10 con 0x5613f11cc530 2023-07-29T16:18:29.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0952344b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f11213d0 con 0x5613f11cbbe0 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0952344b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f111ccc0 con 0x5613f11cd2f0 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f09357fa700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09200a0790 0x7f09200a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f09357fa700 1 -- 172.21.15.31:0/891008730 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f092c014070 con 0x7f0938101510 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f0937fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cc530 0x5613f11ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f093cf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 0x5613f11cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f093cf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 0x5613f11cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36420/0 (socket says 172.21.15.31:36420) 2023-07-29T16:18:29.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f093cf9c700 1 -- 172.21.15.31:0/3823679120 learned_addr learned my addr 172.21.15.31:0/3823679120 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f09377fe700 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cd2f0 0x5613f11cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.020+0000 7f093cf9c700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09200a0790 0x7f09200a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f093cf9c700 1 -- 172.21.15.31:0/3823679120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cd2f0 msgr2=0x5613f11cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f093cf9c700 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cd2f0 0x5613f11cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f093cf9c700 1 -- 172.21.15.31:0/3823679120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cc530 msgr2=0x5613f11ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f093cf9c700 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cc530 0x5613f11ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f093cf9c700 1 -- 172.21.15.31:0/3823679120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613f0f4ade0 con 0x5613f11cbbe0 2023-07-29T16:18:29.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f09377fe700 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cd2f0 0x5613f11cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:29.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f093cf9c700 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 0x5613f11cbff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0930012c90 tx=0x7f0930017350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63806e97fbb96e3f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f091e7fc700 1 -- 172.21.15.31:0/3823679120 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093000e040 con 0x5613f11cbbe0 2023-07-29T16:18:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f091e7fc700 1 -- 172.21.15.31:0/3823679120 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09300125e0 con 0x5613f11cbbe0 2023-07-29T16:18:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.021+0000 7f091e7fc700 1 -- 172.21.15.31:0/3823679120 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930021660 con 0x5613f11cbbe0 2023-07-29T16:18:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 -- 172.21.15.31:0/3823679120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 msgr2=0x5613f11cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f093cf9c700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09200a0790 0x7f09200a2c40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0930021000 tx=0x7f093001f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 0x5613f11cbff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0930012c90 tx=0x7f0930017350 comp rx=0 tx=0).stop 2023-07-29T16:18:29.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 -- 172.21.15.31:0/3823679120 shutdown_connections 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cd2f0 0x5613f11cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cbbe0 0x5613f11cbff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 --2- 172.21.15.31:0/3823679120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cc530 0x5613f11ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 -- 172.21.15.31:0/3823679120 >> 172.21.15.31:0/3823679120 conn(0x5613f11c13e0 msgr2=0x5613f11c6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 -- 172.21.15.31:0/3823679120 shutdown_connections 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.022+0000 7f0952344b80 1 -- 172.21.15.31:0/3823679120 wait complete. 2023-07-29T16:18:29.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 -- start start 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 Processor -- start 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11d57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11dcd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f10b9510 con 0x5613f11cd2f0 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f10b91b0 con 0x5613f11cc530 2023-07-29T16:18:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f0fa0920 con 0x5613f11cbbe0 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.023+0000 7f0952344b80 10 client.0 _ll_register_callbacks cb 0x5613f10c2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f0937fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11d57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36422/0 (socket says 172.21.15.31:36422) 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f0937fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11d57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36776/0 (socket says 172.21.15.31:36776) 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 -- 172.21.15.31:0/2991828145 learned_addr learned my addr 172.21.15.31:0/2991828145 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f093cf9c700 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11dcd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 -- 172.21.15.31:0/2991828145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 msgr2=0x5613f11d57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11d57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 -- 172.21.15.31:0/2991828145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 msgr2=0x5613f11dcd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11dcd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 -- 172.21.15.31:0/2991828145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09381a08c0 con 0x5613f11cc530 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09377fe700 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d5cf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f092c00e970 tx=0x7f092c099a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39e078bd98acdf97 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09177fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09177fe700 1 -- 172.21.15.31:0/2991828145 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c09d350 con 0x5613f11cc530 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f09177fe700 1 -- 172.21.15.31:0/2991828145 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f092c09d540 con 0x5613f11cc530 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.024+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5613f1147d10 con 0x5613f11cc530 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f09177fe700 1 -- 172.21.15.31:0/2991828145 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f092c0a1960 con 0x5613f11cc530 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 10 client.26444 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f0930012840 con 0x5613f11cc530 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f09177fe700 1 -- 172.21.15.31:0/2991828145 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5613f1147d10 con 0x5613f11cc530 2023-07-29T16:18:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f09177fe700 1 -- 172.21.15.31:0/2991828145 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f092c098470 con 0x5613f11cc530 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 10 client.26444 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 10 client.26444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 20 client.26444 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f11e1510 0x5613f11e3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 4 client.26444 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5613f119ad70 con 0x5613f11e1510 2023-07-29T16:18:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.025+0000 7f0937fff700 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f11e1510 0x5613f11e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.026+0000 7f0937fff700 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f11e1510 0x5613f11e3920 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=276ad01a0d5e213f server_cookie=789d1c20c2d0c881 in_seq=0 out_seq=0 2023-07-29T16:18:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.026+0000 7f09177fe700 10 client.26444 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f09177fe700 1 -- 172.21.15.31:0/2991828145 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5613f119ad70 con 0x5613f11e1510 2023-07-29T16:18:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f09177fe700 10 client.26444 handle_command_reply: tid=0 2023-07-29T16:18:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 client.26444 shutdown 2023-07-29T16:18:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 msgr2=0x5613f11d5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d5cf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f092c00e970 tx=0x7f092c099a50 comp rx=0 tx=0).stop 2023-07-29T16:18:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 shutdown_connections 2023-07-29T16:18:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f11e1510 0x5613f11e3920 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11dcd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11d57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 --2- 172.21.15.31:0/2991828145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d5cf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f09177fe700 0 client.26444 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 >> 172.21.15.31:0/2991828145 conn(0x5613f11c13e0 msgr2=0x5613f11c6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 shutdown_connections 2023-07-29T16:18:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 1 -- 172.21.15.31:0/2991828145 wait complete. 2023-07-29T16:18:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.027+0000 7f0952344b80 20 client.26444 trim_cache size 0 max 16384 2023-07-29T16:18:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 Processor -- start 2023-07-29T16:18:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 -- start start 2023-07-29T16:18:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11c6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11dccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f11c34a0 con 0x5613f11cd2f0 2023-07-29T16:18:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f11dea20 con 0x5613f11cc530 2023-07-29T16:18:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.045+0000 7f0952344b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f11c7690 con 0x5613f11cbbe0 2023-07-29T16:18:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f09377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11dccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36784/0 (socket says 172.21.15.31:36784) 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 -- 172.21.15.31:0/2455078565 learned_addr learned my addr 172.21.15.31:0/2455078565 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f093cf9c700 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 -- 172.21.15.31:0/2455078565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 msgr2=0x5613f11dccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11dccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 -- 172.21.15.31:0/2455078565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 msgr2=0x5613f11d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 -- 172.21.15.31:0/2455078565 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613f0f33c60 con 0x5613f11cbbe0 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f09377fe700 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11dccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.046+0000 7f0937fff700 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11c6060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0938101370 tx=0x7f0928005960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8984d3c1b8c39309 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f091effd700 1 -- 172.21.15.31:0/2455078565 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092801d040 con 0x5613f11cbbe0 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f091effd700 1 -- 172.21.15.31:0/2455078565 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f092801e040 con 0x5613f11cbbe0 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f091effd700 1 -- 172.21.15.31:0/2455078565 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928004500 con 0x5613f11cbbe0 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 -- 172.21.15.31:0/2455078565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 msgr2=0x5613f11c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11c6060 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0938101370 tx=0x7f0928005960 comp rx=0 tx=0).stop 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 -- 172.21.15.31:0/2455078565 shutdown_connections 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11c6060 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 --2- 172.21.15.31:0/2455078565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11dccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 -- 172.21.15.31:0/2455078565 >> 172.21.15.31:0/2455078565 conn(0x5613f11c13e0 msgr2=0x5613f11c6900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 -- 172.21.15.31:0/2455078565 shutdown_connections 2023-07-29T16:18:29.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.047+0000 7f0952344b80 1 -- 172.21.15.31:0/2455078565 wait complete. 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 -- start start 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 Processor -- start 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11dca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f11d9f30 con 0x5613f11cd2f0 2023-07-29T16:18:29.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f11da0a0 con 0x5613f11cc530 2023-07-29T16:18:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613f1233090 con 0x5613f11cbbe0 2023-07-29T16:18:29.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.048+0000 7f0952344b80 10 client.0 _ll_register_callbacks cb 0x5613f10c2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f0937fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11dca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f093cf9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f0937fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11dca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36790/0 (socket says 172.21.15.31:36790) 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f0937fff700 1 -- 172.21.15.31:0/3981711419 learned_addr learned my addr 172.21.15.31:0/3981711419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36438/0 (socket says 172.21.15.31:36438) 2023-07-29T16:18:29.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 -- 172.21.15.31:0/3981711419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 msgr2=0x5613f11dca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11dca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 -- 172.21.15.31:0/3981711419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 msgr2=0x5613f11d99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 -- 172.21.15.31:0/3981711419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0930012840 con 0x5613f11cc530 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09377fe700 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d94b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f092c0987f0 tx=0x7f092c0993f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a953f342e7c9002 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.049+0000 7f09177fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:29.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f09177fe700 1 -- 172.21.15.31:0/3981711419 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092c0a16d0 con 0x5613f11cc530 2023-07-29T16:18:29.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09300125e0 con 0x5613f11cc530 2023-07-29T16:18:29.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f09177fe700 1 -- 172.21.15.31:0/3981711419 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f092c09d540 con 0x5613f11cc530 2023-07-29T16:18:29.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f09177fe700 1 -- 172.21.15.31:0/3981711419 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f092c0a1960 con 0x5613f11cc530 2023-07-29T16:18:29.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 10 client.26450 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:29.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f0938109090 con 0x5613f11cc530 2023-07-29T16:18:29.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f09177fe700 1 -- 172.21.15.31:0/3981711419 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09300125e0 con 0x5613f11cc530 2023-07-29T16:18:29.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f09177fe700 1 -- 172.21.15.31:0/3981711419 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f092c09a2f0 con 0x5613f11cc530 2023-07-29T16:18:29.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 10 client.26450 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:29.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 10 client.26450 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:29.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 20 client.26450 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:29.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f1234690 0x5613f1236aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 4 client.26450 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.050+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5613f1115d40 con 0x5613f1234690 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.051+0000 7f0937fff700 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f1234690 0x5613f1236aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.051+0000 7f0937fff700 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f1234690 0x5613f1236aa0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee0079ea0a89c1e8 server_cookie=d4981ca9caa6b1b3 in_seq=0 out_seq=0 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.051+0000 7f09177fe700 10 client.26450 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.052+0000 7f09177fe700 1 -- 172.21.15.31:0/3981711419 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5613f1115d40 con 0x5613f1234690 2023-07-29T16:18:29.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.052+0000 7f09177fe700 10 client.26450 handle_command_reply: tid=0 2023-07-29T16:18:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.052+0000 7f0952344b80 1 client.26450 shutdown 2023-07-29T16:18:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 msgr2=0x5613f11d94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d94b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f092c0987f0 tx=0x7f092c0993f0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 shutdown_connections 2023-07-29T16:18:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5613f1234690 0x5613f1236aa0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5613f11cd2f0 0x5613f11d99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5613f11cbbe0 0x5613f11dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f09177fe700 0 client.26450 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 --2- 172.21.15.31:0/3981711419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5613f11cc530 0x5613f11d94b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 >> 172.21.15.31:0/3981711419 conn(0x5613f11c13e0 msgr2=0x5613f11d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 shutdown_connections 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 1 -- 172.21.15.31:0/3981711419 wait complete. 2023-07-29T16:18:29.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f0952344b80 20 client.26450 trim_cache size 0 max 16384 2023-07-29T16:18:29.060 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:18:29.060 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:18:29.060 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:18:29.060 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:18:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09200a0790 msgr2=0x7f09200a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f093e9ff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09200a0790 0x7f09200a2c40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0930021000 tx=0x7f093001f000 comp rx=0 tx=0).stop 2023-07-29T16:18:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 msgr2=0x7f0938073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.053+0000 7f093e9ff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 0x7f0938073140 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f092c00e9a0 tx=0x7f092c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 shutdown_connections 2023-07-29T16:18:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09200a0790 0x7f09200a2c40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0938101ea0 0x7f0938073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0938100320 0x7f0938074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 --2- 172.21.15.31:0/891008730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0938101510 0x7f0938073140 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 >> 172.21.15.31:0/891008730 conn(0x7f09380fb920 msgr2=0x7f09380fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 shutdown_connections 2023-07-29T16:18:29.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.054+0000 7f093e9ff700 1 -- 172.21.15.31:0/891008730 wait complete. 2023-07-29T16:18:29.070 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:18:29.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:18:29.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2932112088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 msgr2=0x7f4c24100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2932112088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24100750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009990 tx=0x7f4c14009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2932112088 shutdown_connections 2023-07-29T16:18:29.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2932112088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c24101ec0 0x7f4c24108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2932112088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24100750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2932112088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c24101530 0x7f4c24101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2932112088 >> 172.21.15.31:0/2932112088 conn(0x7f4c240fb920 msgr2=0x7f4c240fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2932112088 shutdown_connections 2023-07-29T16:18:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.277+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2932112088 wait complete. 2023-07-29T16:18:29.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 Processor -- start 2023-07-29T16:18:29.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 -- start start 2023-07-29T16:18:29.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c24101530 0x7f4c241987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c24101ec0 0x7f4c2419f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2419fe00 con 0x7f4c24101530 2023-07-29T16:18:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2419ff40 con 0x7f4c24100340 2023-07-29T16:18:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.278+0000 7f4c2a1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c241a00d0 con 0x7f4c24101ec0 2023-07-29T16:18:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36466/0 (socket says 172.21.15.31:36466) 2023-07-29T16:18:29.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 -- 172.21.15.31:0/2416137532 learned_addr learned my addr 172.21.15.31:0/2416137532 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c22ffd700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c24101530 0x7f4c241987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c23fff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c24101ec0 0x7f4c2419f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 -- 172.21.15.31:0/2416137532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c24101ec0 msgr2=0x7f4c2419f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c24101ec0 0x7f4c2419f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 -- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c24101530 msgr2=0x7f4c241987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c24101530 0x7f4c241987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 -- 172.21.15.31:0/2416137532 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c140095f0 con 0x7f4c24100340 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.279+0000 7f4c237fe700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24198280 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4c14005740 tx=0x7f4c14005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.280+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1401d070 con 0x7f4c24100340 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.280+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c14021410 con 0x7f4c24100340 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.280+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14017670 con 0x7f4c24100340 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.280+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c241a0350 con 0x7f4c24100340 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.280+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c241a0770 con 0x7f4c24100340 2023-07-29T16:18:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.281+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c24192a20 con 0x7f4c24100340 2023-07-29T16:18:29.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.283+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4c14021580 con 0x7f4c24100340 2023-07-29T16:18:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.284+0000 7f4c20ff9700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4c0c0a0860 0x7f4c0c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.284+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4c140ccbf0 con 0x7f4c24100340 2023-07-29T16:18:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.284+0000 7f4c22ffd700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4c0c0a0860 0x7f4c0c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.285+0000 7f4c22ffd700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4c0c0a0860 0x7f4c0c0a2d10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009c60 tx=0x7f4c18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.287+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f4c140966e0 con 0x7f4c24100340 2023-07-29T16:18:29.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.558+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c2402cc30 con 0x7f4c24100340 2023-07-29T16:18:29.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.559+0000 7f4c20ff9700 1 -- 172.21.15.31:0/2416137532 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f4c14026090 con 0x7f4c24100340 2023-07-29T16:18:29.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:18:29.558 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:12:27.740070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/809205544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":809205544},{"type":"v1","addr":"172.21.15.156:6837","nonce":809205544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:18:29.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.561+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4c0c0a0860 msgr2=0x7f4c0c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.561+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4c0c0a0860 0x7f4c0c0a2d10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4c18009c60 tx=0x7f4c18009260 comp rx=0 tx=0).stop 2023-07-29T16:18:29.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.561+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 msgr2=0x7f4c24198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.561+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24198280 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4c14005740 tx=0x7f4c14005770 comp rx=0 tx=0).stop 2023-07-29T16:18:29.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.561+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 shutdown_connections 2023-07-29T16:18:29.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4c0c0a0860 0x7f4c0c0a2d10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4c24101ec0 0x7f4c2419f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c24100340 0x7f4c24198280 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 --2- 172.21.15.31:0/2416137532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c24101530 0x7f4c241987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 >> 172.21.15.31:0/2416137532 conn(0x7f4c240fb920 msgr2=0x7f4c24106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 shutdown_connections 2023-07-29T16:18:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.562+0000 7f4c2a1ff700 1 -- 172.21.15.31:0/2416137532 wait complete. 2023-07-29T16:18:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 23 2023-07-29T16:18:29.578 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:18:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 -- 172.21.15.31:0/338743268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c074d10 msgr2=0x7fdc8c073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 --2- 172.21.15.31:0/338743268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c074d10 0x7fdc8c073160 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c009990 tx=0x7fdc7c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 -- 172.21.15.31:0/338743268 shutdown_connections 2023-07-29T16:18:29.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 --2- 172.21.15.31:0/338743268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c1081f0 0x7fdc8c10a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 --2- 172.21.15.31:0/338743268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c074d10 0x7fdc8c073160 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 --2- 172.21.15.31:0/338743268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc8c073730 0x7fdc8c073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 -- 172.21.15.31:0/338743268 >> 172.21.15.31:0/338743268 conn(0x7fdc8c0fb960 msgr2=0x7fdc8c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 -- 172.21.15.31:0/338743268 shutdown_connections 2023-07-29T16:18:29.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.793+0000 7fdc927ff700 1 -- 172.21.15.31:0/338743268 wait complete. 2023-07-29T16:18:29.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 Processor -- start 2023-07-29T16:18:29.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 -- start start 2023-07-29T16:18:29.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 0x7fdc8c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc8c074d10 0x7fdc8c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c1081f0 0x7fdc8c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c19fe00 con 0x7fdc8c074d10 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c19ff40 con 0x7fdc8c1081f0 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.794+0000 7fdc927ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8c1a00d0 con 0x7fdc8c073730 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 0x7fdc8c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 0x7fdc8c198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36840/0 (socket says 172.21.15.31:36840) 2023-07-29T16:18:29.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 -- 172.21.15.31:0/903003735 learned_addr learned my addr 172.21.15.31:0/903003735 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc8c074d10 0x7fdc8c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc90d9c700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c1081f0 0x7fdc8c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 -- 172.21.15.31:0/903003735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c1081f0 msgr2=0x7fdc8c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c1081f0 0x7fdc8c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 -- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc8c074d10 msgr2=0x7fdc8c1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc8c074d10 0x7fdc8c1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 -- 172.21.15.31:0/903003735 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0095f0 con 0x7fdc8c073730 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 0x7fdc8c198280 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b680 tx=0x7fdc7c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc897fa700 1 -- 172.21.15.31:0/903003735 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c01d070 con 0x7fdc8c073730 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc897fa700 1 -- 172.21.15.31:0/903003735 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc7c022470 con 0x7fdc8c073730 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.795+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8c1a0350 con 0x7fdc8c073730 2023-07-29T16:18:29.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.796+0000 7fdc897fa700 1 -- 172.21.15.31:0/903003735 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c017650 con 0x7fdc8c073730 2023-07-29T16:18:29.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.796+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1a07d0 con 0x7fdc8c073730 2023-07-29T16:18:29.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.799+0000 7fdc897fa700 1 -- 172.21.15.31:0/903003735 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdc7c0225e0 con 0x7fdc8c073730 2023-07-29T16:18:29.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.800+0000 7fdc897fa700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdc740a0810 0x7fdc740a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.800+0000 7fdc897fa700 1 -- 172.21.15.31:0/903003735 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdc7c0ccec0 con 0x7fdc8c073730 2023-07-29T16:18:29.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdc740a0810 0x7fdc740a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 Processor -- start 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 -- start start 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdc740a0810 0x7fdc740a2cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009c60 tx=0x7fdc80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f890faff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f890fbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f890fc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f89052a10 con 0x564f890fc2f0 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f890503d0 con 0x564f890fb530 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.801+0000 7fdca6178b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8904bcc0 con 0x564f890fabe0 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f890fbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc90d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f890fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f890fbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36492/0 (socket says 172.21.15.31:36492) 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8b7fe700 1 -- 172.21.15.31:0/2246038275 learned_addr learned my addr 172.21.15.31:0/2246038275 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f890faff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2246038275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 msgr2=0x564f890fbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f890fbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2246038275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 msgr2=0x564f890fc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f890fc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2246038275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f88e79de0 con 0x564f890fabe0 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.802+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f890faff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c006010 tx=0x7fdc7c09a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3558331d8de6f1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.803+0000 7fdc727fc700 1 -- 172.21.15.31:0/2246038275 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c09e620 con 0x564f890fabe0 2023-07-29T16:18:29.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.803+0000 7fdc727fc700 1 -- 172.21.15.31:0/2246038275 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc7c0a29b0 con 0x564f890fabe0 2023-07-29T16:18:29.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.803+0000 7fdc727fc700 1 -- 172.21.15.31:0/2246038275 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c099d20 con 0x564f890fabe0 2023-07-29T16:18:29.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.803+0000 7fdca6178b80 1 -- 172.21.15.31:0/2246038275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 msgr2=0x564f890faff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.803+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f890faff0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c006010 tx=0x7fdc7c09a130 comp rx=0 tx=0).stop 2023-07-29T16:18:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 -- 172.21.15.31:0/2246038275 shutdown_connections 2023-07-29T16:18:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f890fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f890faff0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2246038275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f890fbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 -- 172.21.15.31:0/2246038275 >> 172.21.15.31:0/2246038275 conn(0x564f890f03e0 msgr2=0x564f890f5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 -- 172.21.15.31:0/2246038275 shutdown_connections 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 -- 172.21.15.31:0/2246038275 wait complete. 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 -- start start 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.804+0000 7fdca6178b80 1 Processor -- start 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 0x564f89104450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fb530 0x564f89104990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f8910b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f88fe8510 con 0x564f890fc2f0 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f88fe81b0 con 0x564f890fabe0 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f88ecf920 con 0x564f890fb530 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdca6178b80 10 client.0 _ll_register_callbacks cb 0x564f88ff1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 0x564f89104450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 0x564f89104450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36494/0 (socket says 172.21.15.31:36494) 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2383002080 learned_addr learned my addr 172.21.15.31:0/2383002080 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fb530 0x564f89104990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.805+0000 7fdc90d9c700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f8910b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fb530 msgr2=0x564f89104990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fb530 0x564f89104990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2383002080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 msgr2=0x564f8910b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f8910b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc8bfff700 1 -- 172.21.15.31:0/2383002080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8c1a07d0 con 0x564f890fabe0 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 0x564f89104450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c097060 tx=0x7fdc7c096270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fbad26725ade5bf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/2383002080 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c09e620 con 0x564f890fabe0 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/2383002080 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdc7c0a2520 con 0x564f890fabe0 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f89076d10 con 0x564f890fabe0 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/2383002080 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc7c0a26e0 con 0x564f890fabe0 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdca6178b80 10 client.26468 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.806+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564f88e62c60 con 0x564f890fabe0 2023-07-29T16:18:29.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/2383002080 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc7c0a6070 con 0x564f890fabe0 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/2383002080 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdc7c0b0070 con 0x564f890fabe0 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdca6178b80 10 client.26468 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdca6178b80 10 client.26468 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdca6178b80 20 client.26468 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f8910fd40 0x564f89112120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdca6178b80 4 client.26468 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f890c9d70 con 0x564f8910fd40 2023-07-29T16:18:29.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.807+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f8910fd40 0x564f89112120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.808+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f8910fd40 0x564f89112120 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=405a49144d7fa463 server_cookie=17d8eca965cf7f9c in_seq=0 out_seq=0 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.808+0000 7fdc6b7fe700 10 client.26468 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.808+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/2383002080 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564f890c9d70 con 0x564f8910fd40 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.808+0000 7fdc6b7fe700 10 client.26468 handle_command_reply: tid=0 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.808+0000 7fdca6178b80 1 client.26468 shutdown 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 msgr2=0x564f89104450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 0x564f89104450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c097060 tx=0x7fdc7c096270 comp rx=0 tx=0).stop 2023-07-29T16:18:29.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 shutdown_connections 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f8910b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f8910fd40 0x564f89112120 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fabe0 0x564f89104450 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 --2- 172.21.15.31:0/2383002080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fb530 0x564f89104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdc6b7fe700 0 client.26468 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 >> 172.21.15.31:0/2383002080 conn(0x564f890f03e0 msgr2=0x564f890f5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 shutdown_connections 2023-07-29T16:18:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 1 -- 172.21.15.31:0/2383002080 wait complete. 2023-07-29T16:18:29.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.809+0000 7fdca6178b80 20 client.26468 trim_cache size 0 max 16384 2023-07-29T16:18:29.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.823+0000 7fdca6178b80 1 Processor -- start 2023-07-29T16:18:29.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.823+0000 7fdca6178b80 1 -- start start 2023-07-29T16:18:29.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f89112120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f8910b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f89103b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdca6178b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f890fbc20 con 0x564f890fc2f0 2023-07-29T16:18:29.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdca6178b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f890f2490 con 0x564f890fb530 2023-07-29T16:18:29.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdca6178b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f89116b00 con 0x564f890fabe0 2023-07-29T16:18:29.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f89112120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc90d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f89103b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f8910b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f89112120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36868/0 (socket says 172.21.15.31:36868) 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 -- 172.21.15.31:0/3114520770 learned_addr learned my addr 172.21.15.31:0/3114520770 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 -- 172.21.15.31:0/3114520770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 msgr2=0x564f8910b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f8910b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 -- 172.21.15.31:0/3114520770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 msgr2=0x564f89103b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f89103b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.824+0000 7fdc8bfff700 1 -- 172.21.15.31:0/3114520770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f88ea0250 con 0x564f890fabe0 2023-07-29T16:18:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f89112120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c006010 tx=0x7fdc7c09a470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=951aa4a33bf5a640 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdc72ffd700 1 -- 172.21.15.31:0/3114520770 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c099550 con 0x564f890fabe0 2023-07-29T16:18:29.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdc72ffd700 1 -- 172.21.15.31:0/3114520770 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc7c0a6070 con 0x564f890fabe0 2023-07-29T16:18:29.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdc72ffd700 1 -- 172.21.15.31:0/3114520770 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c0a2520 con 0x564f890fabe0 2023-07-29T16:18:29.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 -- 172.21.15.31:0/3114520770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 msgr2=0x564f89112120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f89112120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c006010 tx=0x7fdc7c09a470 comp rx=0 tx=0).stop 2023-07-29T16:18:29.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 -- 172.21.15.31:0/3114520770 shutdown_connections 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fc2f0 0x564f89103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f89112120 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 --2- 172.21.15.31:0/3114520770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fb530 0x564f8910b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.825+0000 7fdca6178b80 1 -- 172.21.15.31:0/3114520770 >> 172.21.15.31:0/3114520770 conn(0x564f890f03e0 msgr2=0x564f890f5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.826+0000 7fdca6178b80 1 -- 172.21.15.31:0/3114520770 shutdown_connections 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.826+0000 7fdca6178b80 1 -- 172.21.15.31:0/3114520770 wait complete. 2023-07-29T16:18:29.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.826+0000 7fdca6178b80 1 -- start start 2023-07-29T16:18:29.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.826+0000 7fdca6178b80 1 Processor -- start 2023-07-29T16:18:29.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f891074d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fb530 0x564f89107a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fc2f0 0x564f8910a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8910af00 con 0x564f890fb530 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8910b070 con 0x564f890fc2f0 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8910b230 con 0x564f890fabe0 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdca6178b80 10 client.0 _ll_register_callbacks cb 0x564f88ff1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f891074d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc90d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fc2f0 0x564f8910a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f891074d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36882/0 (socket says 172.21.15.31:36882) 2023-07-29T16:18:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 -- 172.21.15.31:0/1786081724 learned_addr learned my addr 172.21.15.31:0/1786081724 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fb530 0x564f89107a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 -- 172.21.15.31:0/1786081724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fc2f0 msgr2=0x564f8910a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fc2f0 0x564f8910a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 -- 172.21.15.31:0/1786081724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fb530 msgr2=0x564f89107a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fb530 0x564f89107a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc8bfff700 1 -- 172.21.15.31:0/1786081724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f88e62c60 con 0x564f890fabe0 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.827+0000 7fdc90d9c700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fc2f0 0x564f8910a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdc8bfff700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f891074d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c096af0 tx=0x7fdc7c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed96a88fc5d1edd2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdc6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/1786081724 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c09e370 con 0x564f890fabe0 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc7c0a29b0 con 0x564f890fabe0 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/1786081724 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdc7c09e620 con 0x564f890fabe0 2023-07-29T16:18:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/1786081724 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc7c096750 con 0x564f890fabe0 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdca6178b80 10 client.26494 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.828+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564f88e782b0 con 0x564f890fabe0 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/1786081724 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc7c0a29b0 con 0x564f890fabe0 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/1786081724 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdc7c0b0070 con 0x564f890fabe0 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdca6178b80 10 client.26494 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdca6178b80 10 client.26494 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:18:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdca6178b80 20 client.26494 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdca6178b80 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f89161ec0 0x564f891642a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdca6178b80 4 client.26494 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564f89044d40 con 0x564f89161ec0 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.829+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f89161ec0 0x564f891642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdc8b7fe700 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f89161ec0 0x564f891642a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27926145d8175a7a server_cookie=756dbdec99eaac8f in_seq=0 out_seq=0 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdc6b7fe700 10 client.26494 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:18:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdc6b7fe700 1 -- 172.21.15.31:0/1786081724 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564f89044d40 con 0x564f89161ec0 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdc6b7fe700 10 client.26494 handle_command_reply: tid=0 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 client.26494 shutdown 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 msgr2=0x564f891074d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f891074d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c096af0 tx=0x7fdc7c009310 comp rx=0 tx=0).stop 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 shutdown_connections 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564f89161ec0 0x564f891642a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f890fc2f0 0x564f8910a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f890fabe0 0x564f891074d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 --2- 172.21.15.31:0/1786081724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f890fb530 0x564f89107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 >> 172.21.15.31:0/1786081724 conn(0x564f890f03e0 msgr2=0x564f89104200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdc6b7fe700 0 client.26494 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:18:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 shutdown_connections 2023-07-29T16:18:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.830+0000 7fdca6178b80 1 -- 172.21.15.31:0/1786081724 wait complete. 2023-07-29T16:18:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdca6178b80 20 client.26494 trim_cache size 0 max 16384 2023-07-29T16:18:29.838 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:18:29.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdc740a0810 msgr2=0x7fdc740a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdc740a0810 0x7fdc740a2cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009c60 tx=0x7fdc80009260 comp rx=0 tx=0).stop 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 msgr2=0x7fdc8c198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 0x7fdc8c198280 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b680 tx=0x7fdc7c005740 comp rx=0 tx=0).stop 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 shutdown_connections 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdc740a0810 0x7fdc740a2cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc8c1081f0 0x7fdc8c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdc8c073730 0x7fdc8c198280 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 --2- 172.21.15.31:0/903003735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc8c074d10 0x7fdc8c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 >> 172.21.15.31:0/903003735 conn(0x7fdc8c0fb960 msgr2=0x7fdc8c104180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 shutdown_connections 2023-07-29T16:18:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:29.831+0000 7fdc927ff700 1 -- 172.21.15.31:0/903003735 wait complete. 2023-07-29T16:18:29.848 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:18:29.855 INFO:tasks.workunit.client.0.smithi031.stdout: 1 30109 12.80 MB/sec warmup 14 sec latency 83.549 ms 2023-07-29T16:18:30.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 -- 172.21.15.31:0/4055729459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250100e40 msgr2=0x7f2250103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 --2- 172.21.15.31:0/4055729459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250100e40 0x7f2250103220 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2240009990 tx=0x7f2240009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 -- 172.21.15.31:0/4055729459 shutdown_connections 2023-07-29T16:18:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 --2- 172.21.15.31:0/4055729459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250106080 0x7f2250108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 --2- 172.21.15.31:0/4055729459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250100e40 0x7f2250103220 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 --2- 172.21.15.31:0/4055729459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250103760 0x7f2250105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.068+0000 7f22549ff700 1 -- 172.21.15.31:0/4055729459 >> 172.21.15.31:0/4055729459 conn(0x7f22500fa840 msgr2=0x7f22500fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.069+0000 7f22549ff700 1 -- 172.21.15.31:0/4055729459 shutdown_connections 2023-07-29T16:18:30.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.069+0000 7f22549ff700 1 -- 172.21.15.31:0/4055729459 wait complete. 2023-07-29T16:18:30.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.070+0000 7f22549ff700 1 Processor -- start 2023-07-29T16:18:30.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.070+0000 7f22549ff700 1 -- start start 2023-07-29T16:18:30.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.070+0000 7f22549ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250100e40 0x7f2250198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.070+0000 7f22549ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 0x7f22501987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.070+0000 7f22549ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250106080 0x7f225019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.070+0000 7f22549ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225019fde0 con 0x7f2250100e40 2023-07-29T16:18:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f22549ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225019ff20 con 0x7f2250106080 2023-07-29T16:18:30.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f22549ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22501a00b0 con 0x7f2250103760 2023-07-29T16:18:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 -- 172.21.15.31:0/1776261514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da81002b0 msgr2=0x7f3da81006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 --2- 172.21.15.31:0/1776261514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da81002b0 0x7f3da81006c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009990 tx=0x7f3d98009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f224e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250100e40 0x7f2250198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 0x7f22501987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 0x7f22501987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41580/0 (socket says 172.21.15.31:41580) 2023-07-29T16:18:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 -- 172.21.15.31:0/1199885499 learned_addr learned my addr 172.21.15.31:0/1199885499 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f224ed9d700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250106080 0x7f225019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 -- 172.21.15.31:0/1776261514 shutdown_connections 2023-07-29T16:18:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 --2- 172.21.15.31:0/1776261514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da8101ef0 0x7f3da8108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 --2- 172.21.15.31:0/1776261514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da81002b0 0x7f3da81006c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 --2- 172.21.15.31:0/1776261514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101540 0x7f3da81019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 -- 172.21.15.31:0/1776261514 >> 172.21.15.31:0/1776261514 conn(0x7f3da80fb8d0 msgr2=0x7f3da80fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 -- 172.21.15.31:0/1776261514 shutdown_connections 2023-07-29T16:18:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f3dadbff700 1 -- 172.21.15.31:0/1776261514 wait complete. 2023-07-29T16:18:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 -- 172.21.15.31:0/1199885499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250106080 msgr2=0x7f225019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250106080 0x7f225019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 -- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250100e40 msgr2=0x7f2250198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250100e40 0x7f2250198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 -- 172.21.15.31:0/1199885499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400095f0 con 0x7f2250103760 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f224e59c700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250100e40 0x7f2250198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.071+0000 7f2247fff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 0x7f22501987a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f223800d980 tx=0x7f223800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f22477fe700 1 -- 172.21.15.31:0/1199885499 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238004220 con 0x7f2250103760 2023-07-29T16:18:30.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22501a0390 con 0x7f2250103760 2023-07-29T16:18:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22501a08f0 con 0x7f2250103760 2023-07-29T16:18:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f22477fe700 1 -- 172.21.15.31:0/1199885499 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2238007640 con 0x7f2250103760 2023-07-29T16:18:30.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f22477fe700 1 -- 172.21.15.31:0/1199885499 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223800bd10 con 0x7f2250103760 2023-07-29T16:18:30.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f3dadbff700 1 Processor -- start 2023-07-29T16:18:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.072+0000 7f3dadbff700 1 -- start start 2023-07-29T16:18:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3dadbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da81002b0 0x7f3da8196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3dadbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 0x7f3da8196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3dadbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101ef0 0x7f3da819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3dadbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819dbb0 con 0x7f3da8101ef0 2023-07-29T16:18:30.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3dadbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819dcf0 con 0x7f3da8101540 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3dadbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819de80 con 0x7f3da81002b0 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 0x7f3da8196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da81002b0 0x7f3da8196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 0x7f3da8196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56674/0 (socket says 172.21.15.31:56674) 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3da6ffd700 1 -- 172.21.15.31:0/2265542830 learned_addr learned my addr 172.21.15.31:0/2265542830 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.073+0000 7f3da7fff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101ef0 0x7f3da819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da6ffd700 1 -- 172.21.15.31:0/2265542830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da81002b0 msgr2=0x7f3da8196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da81002b0 0x7f3da8196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da6ffd700 1 -- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101ef0 msgr2=0x7f3da819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101ef0 0x7f3da819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da6ffd700 1 -- 172.21.15.31:0/2265542830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d980095f0 con 0x7f3da8101540 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da7fff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101ef0 0x7f3da819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 0x7f3da8196570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00d980 tx=0x7f3d9c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da4ff9700 1 -- 172.21.15.31:0/2265542830 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c004220 con 0x7f3da8101540 2023-07-29T16:18:30.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da4ff9700 1 -- 172.21.15.31:0/2265542830 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d9c007640 con 0x7f3da8101540 2023-07-29T16:18:30.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da819e160 con 0x7f3da8101540 2023-07-29T16:18:30.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3da4ff9700 1 -- 172.21.15.31:0/2265542830 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c005610 con 0x7f3da8101540 2023-07-29T16:18:30.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.074+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da819e6c0 con 0x7f3da8101540 2023-07-29T16:18:30.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.076+0000 7f22477fe700 1 -- 172.21.15.31:0/1199885499 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2238004380 con 0x7f2250103760 2023-07-29T16:18:30.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.077+0000 7f22477fe700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f223c0a0810 0x7f223c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.077+0000 7f22477fe700 1 -- 172.21.15.31:0/1199885499 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f22380cb8c0 con 0x7f2250103760 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.078+0000 7f224e59c700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f223c0a0810 0x7f223c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.078+0000 7f2268254b80 1 Processor -- start 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.078+0000 7f2268254b80 1 -- start start 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f224e59c700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f223c0a0810 0x7f223c0a2cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f224000b5c0 tx=0x7f2240005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2268254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb684eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2268254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb684ebd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2268254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb684ec740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2268254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb68442a10 con 0x55cb684eabe0 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2268254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb684403d0 con 0x55cb684ec2f0 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2268254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb6843bcc0 con 0x55cb684eb530 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f224ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb684ec740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f224e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb684eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2247fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb684ebd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2247fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb684ebd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41610/0 (socket says 172.21.15.31:41610) 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f2247fff700 1 -- 172.21.15.31:0/1592065231 learned_addr learned my addr 172.21.15.31:0/1592065231 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.079+0000 7f224ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb684ec740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56684/0 (socket says 172.21.15.31:56684) 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2247fff700 1 -- 172.21.15.31:0/1592065231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 msgr2=0x55cb684ec740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2247fff700 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb684ec740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2247fff700 1 -- 172.21.15.31:0/1592065231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 msgr2=0x55cb684eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2247fff700 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb684eaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2247fff700 1 -- 172.21.15.31:0/1592065231 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb68269de0 con 0x55cb684eb530 2023-07-29T16:18:30.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f224e59c700 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb684eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:30.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f3da4ff9700 1 -- 172.21.15.31:0/2265542830 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3d9c004750 con 0x7f3da8101540 2023-07-29T16:18:30.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2247fff700 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb684ebd60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f22380a0e80 tx=0x7f22380989c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3722fbb6d71d605 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f3dc1431b80 1 Processor -- start 2023-07-29T16:18:30.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2244ff9700 1 -- 172.21.15.31:0/1592065231 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380b0070 con 0x55cb684eb530 2023-07-29T16:18:30.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2244ff9700 1 -- 172.21.15.31:0/1592065231 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2238098f60 con 0x55cb684eb530 2023-07-29T16:18:30.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.080+0000 7f2244ff9700 1 -- 172.21.15.31:0/1592065231 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380986b0 con 0x55cb684eb530 2023-07-29T16:18:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 -- start start 2023-07-29T16:18:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 0x56411a5e5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e4f80 0x56411a5e5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5389b0 con 0x56411a43c6b0 2023-07-29T16:18:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a536370 con 0x56411a5e6370 2023-07-29T16:18:30.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a531c60 con 0x56411a5e4f80 2023-07-29T16:18:30.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 -- 172.21.15.31:0/1592065231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 msgr2=0x55cb684ebd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb684ebd60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f22380a0e80 tx=0x7f22380989c0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 -- 172.21.15.31:0/1592065231 shutdown_connections 2023-07-29T16:18:30.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb684ec740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb684eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 --2- 172.21.15.31:0/1592065231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb684ebd60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 -- 172.21.15.31:0/1592065231 >> 172.21.15.31:0/1592065231 conn(0x55cb684e03e0 msgr2=0x55cb684e5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 0x56411a5e5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 0x56411a5e5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52800/0 (socket says 172.21.15.31:52800) 2023-07-29T16:18:30.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3da7fff700 1 -- 172.21.15.31:0/647585048 learned_addr learned my addr 172.21.15.31:0/647585048 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 -- 172.21.15.31:0/1592065231 shutdown_connections 2023-07-29T16:18:30.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f2268254b80 1 -- 172.21.15.31:0/1592065231 wait complete. 2023-07-29T16:18:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e4f80 0x56411a5e5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.081+0000 7f3da77fe700 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da7fff700 1 -- 172.21.15.31:0/647585048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e4f80 msgr2=0x56411a5e5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da7fff700 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e4f80 0x56411a5e5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da7fff700 1 -- 172.21.15.31:0/647585048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 msgr2=0x56411a5e4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da7fff700 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5e4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da7fff700 1 -- 172.21.15.31:0/647585048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56411a35fd80 con 0x56411a43c6b0 2023-07-29T16:18:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da77fe700 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:30.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f2268254b80 1 -- start start 2023-07-29T16:18:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da4ff9700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d900a0810 0x7f3d900a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da7fff700 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 0x56411a5e5e30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009440 tx=0x7f3da0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53595e305309679d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f2268254b80 1 Processor -- start 2023-07-29T16:18:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.082+0000 7f3da4ff9700 1 -- 172.21.15.31:0/2265542830 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3d9c0cb990 con 0x7f3da8101540 2023-07-29T16:18:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f3d8dffb700 1 -- 172.21.15.31:0/647585048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da000e070 con 0x56411a43c6b0 2023-07-29T16:18:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f3da77fe700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d900a0810 0x7f3d900a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f3d8dffb700 1 -- 172.21.15.31:0/647585048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3da000f040 con 0x56411a43c6b0 2023-07-29T16:18:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f3d8dffb700 1 -- 172.21.15.31:0/647585048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0015bf0 con 0x56411a43c6b0 2023-07-29T16:18:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 0x55cb684fd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eb530 0x55cb684fd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684ec2f0 0x55cb685045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb683d8510 con 0x55cb684eb530 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb683d81b0 con 0x55cb684eabe0 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb682bf920 con 0x55cb684ec2f0 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f2268254b80 10 client.0 _ll_register_callbacks cb 0x55cb683e1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f224e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 0x55cb684fd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f224e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 0x55cb684fd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56698/0 (socket says 172.21.15.31:56698) 2023-07-29T16:18:30.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f224e59c700 1 -- 172.21.15.31:0/5144984 learned_addr learned my addr 172.21.15.31:0/5144984 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.083+0000 7f224ed9d700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684ec2f0 0x55cb685045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f2247fff700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eb530 0x55cb684fd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f224e59c700 1 -- 172.21.15.31:0/5144984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684ec2f0 msgr2=0x55cb685045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f224e59c700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684ec2f0 0x55cb685045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f224e59c700 1 -- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eb530 msgr2=0x55cb684fd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f224e59c700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eb530 0x55cb684fd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f224e59c700 1 -- 172.21.15.31:0/5144984 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22501a08f0 con 0x55cb684eabe0 2023-07-29T16:18:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f224e59c700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 0x55cb684fd060 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2240015000 tx=0x7f2240009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec3c068f00ebdee server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f2235ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3da77fe700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d900a0810 0x7f3d900a2cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800b2e0 tx=0x7f3d98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 -- 172.21.15.31:0/647585048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 msgr2=0x56411a5e5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 0x56411a5e5e30 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009440 tx=0x7f3da0009920 comp rx=0 tx=0).stop 2023-07-29T16:18:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 -- 172.21.15.31:0/647585048 shutdown_connections 2023-07-29T16:18:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e4f80 0x56411a5e5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a43c6b0 0x56411a5e5e30 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f2235ffb700 1 -- 172.21.15.31:0/5144984 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240030630 con 0x55cb684eabe0 2023-07-29T16:18:30.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/647585048 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f3dc1431b80 1 -- 172.21.15.31:0/647585048 >> 172.21.15.31:0/647585048 conn(0x56411a5d5fd0 msgr2=0x56411a5db310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f2235ffb700 1 -- 172.21.15.31:0/5144984 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2240030790 con 0x55cb684eabe0 2023-07-29T16:18:30.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.084+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb68466d10 con 0x55cb684eabe0 2023-07-29T16:18:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f3dc1431b80 1 -- 172.21.15.31:0/647585048 shutdown_connections 2023-07-29T16:18:30.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f3dc1431b80 1 -- 172.21.15.31:0/647585048 wait complete. 2023-07-29T16:18:30.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2235ffb700 1 -- 172.21.15.31:0/5144984 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2240005a00 con 0x55cb684eabe0 2023-07-29T16:18:30.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 10 client.26495 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:30.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cb68252c60 con 0x55cb684eabe0 2023-07-29T16:18:30.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2235ffb700 1 -- 172.21.15.31:0/5144984 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cb68466d10 con 0x55cb684eabe0 2023-07-29T16:18:30.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f3dc1431b80 1 -- start start 2023-07-29T16:18:30.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2235ffb700 1 -- 172.21.15.31:0/5144984 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2240024b40 con 0x55cb684eabe0 2023-07-29T16:18:30.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 10 client.26495 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:30.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 10 client.26495 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:18:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 20 client.26495 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb68508e70 0x55cb6850b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 4 client.26495 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:30.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb684b9d70 con 0x55cb68508e70 2023-07-29T16:18:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.085+0000 7f3dc1431b80 1 Processor -- start 2023-07-29T16:18:30.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f2247fff700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb68508e70 0x55cb6850b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5ee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5eea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a4ce4b0 con 0x56411a5e4f80 2023-07-29T16:18:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f2247fff700 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb68508e70 0x55cb6850b280 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=99cd92b92783bf6e server_cookie=b2b5afce217e000c in_seq=0 out_seq=0 2023-07-29T16:18:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f2235ffb700 10 client.26495 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:18:30.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a4ce150 con 0x56411a43c6b0 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a3b58c0 con 0x56411a5e6370 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3dc1431b80 10 client.0 _ll_register_callbacks cb 0x56411a4d75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5ee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5ee510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56700/0 (socket says 172.21.15.31:56700) 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.086+0000 7f3da77fe700 1 -- 172.21.15.31:0/90791655 learned_addr learned my addr 172.21.15.31:0/90791655 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da7fff700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da77fe700 1 -- 172.21.15.31:0/90791655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 msgr2=0x56411a5f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da77fe700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da77fe700 1 -- 172.21.15.31:0/90791655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 msgr2=0x56411a5eea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da77fe700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5eea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da77fe700 1 -- 172.21.15.31:0/90791655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da8109000 con 0x56411a43c6b0 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:18:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3da77fe700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5ee510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3d98005770 tx=0x7f3d98009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3675c0227f5beca2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.087+0000 7f3d86ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3d86ffd700 1 -- 172.21.15.31:0/90791655 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98004ea0 con 0x56411a43c6b0 2023-07-29T16:18:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3d86ffd700 1 -- 172.21.15.31:0/90791655 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3d98026460 con 0x56411a43c6b0 2023-07-29T16:18:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56411a55ccb0 con 0x56411a43c6b0 2023-07-29T16:18:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2235ffb700 1 -- 172.21.15.31:0/5144984 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55cb684b9d70 con 0x55cb68508e70 2023-07-29T16:18:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2235ffb700 10 client.26495 handle_command_reply: tid=0 2023-07-29T16:18:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 client.26495 shutdown 2023-07-29T16:18:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3d86ffd700 1 -- 172.21.15.31:0/90791655 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d98026620 con 0x56411a43c6b0 2023-07-29T16:18:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3dc1431b80 10 client.26498 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f3da00090f0 con 0x56411a43c6b0 2023-07-29T16:18:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 msgr2=0x55cb684fd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 0x55cb684fd060 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2240015000 tx=0x7f2240009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 shutdown_connections 2023-07-29T16:18:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb68508e70 0x55cb6850b280 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684ec2f0 0x55cb685045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eabe0 0x55cb684fd060 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2235ffb700 0 client.26495 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:18:30.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 --2- 172.21.15.31:0/5144984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eb530 0x55cb684fd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 >> 172.21.15.31:0/5144984 conn(0x55cb684e03e0 msgr2=0x55cb684e5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.088+0000 7f3d86ffd700 1 -- 172.21.15.31:0/90791655 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56411a55ccb0 con 0x56411a43c6b0 2023-07-29T16:18:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3d86ffd700 1 -- 172.21.15.31:0/90791655 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3d9800f8d0 con 0x56411a43c6b0 2023-07-29T16:18:30.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 shutdown_connections 2023-07-29T16:18:30.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f2268254b80 1 -- 172.21.15.31:0/5144984 wait complete. 2023-07-29T16:18:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3dc1431b80 10 client.26498 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3dc1431b80 10 client.26498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3dc1431b80 20 client.26498 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a5fa2e0 0x56411a5fc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3dc1431b80 4 client.26498 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:30.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56411a5afd10 con 0x56411a5fa2e0 2023-07-29T16:18:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f2268254b80 20 client.26495 trim_cache size 0 max 16384 2023-07-29T16:18:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.089+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a5fa2e0 0x56411a5fc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.090+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a5fa2e0 0x56411a5fc6f0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e041b874746ac1c server_cookie=a9f20a63fb4ae86b in_seq=0 out_seq=0 2023-07-29T16:18:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.090+0000 7f3d86ffd700 10 client.26498 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.090+0000 7f3d86ffd700 1 -- 172.21.15.31:0/90791655 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56411a5afd10 con 0x56411a5fa2e0 2023-07-29T16:18:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3d86ffd700 10 client.26498 handle_command_reply: tid=0 2023-07-29T16:18:30.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 client.26498 shutdown 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 msgr2=0x56411a5ee510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5ee510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3d98005770 tx=0x7f3d98009310 comp rx=0 tx=0).stop 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 shutdown_connections 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a5fa2e0 0x56411a5fc6f0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5ee510 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/90791655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5eea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3d86ffd700 0 client.26498 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 >> 172.21.15.31:0/90791655 conn(0x56411a5d5fd0 msgr2=0x56411a5daa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 shutdown_connections 2023-07-29T16:18:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 1 -- 172.21.15.31:0/90791655 wait complete. 2023-07-29T16:18:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.091+0000 7f3dc1431b80 20 client.26498 trim_cache size 0 max 16384 2023-07-29T16:18:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.103+0000 7f2268254b80 1 Processor -- start 2023-07-29T16:18:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.103+0000 7f2268254b80 1 -- start start 2023-07-29T16:18:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2268254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 0x55cb6850fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2268254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eb530 0x55cb68504570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2268254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684ec2f0 0x55cb684ec6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2268254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb684e2460 con 0x55cb684ec2f0 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2268254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb684e6690 con 0x55cb684eb530 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2268254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb684eafb0 con 0x55cb684eabe0 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 0x55cb6850fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 0x55cb6850fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41644/0 (socket says 172.21.15.31:41644) 2023-07-29T16:18:30.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 -- 172.21.15.31:0/3681933257 learned_addr learned my addr 172.21.15.31:0/3681933257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f2247fff700 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eb530 0x55cb68504570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224ed9d700 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684ec2f0 0x55cb684ec6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 -- 172.21.15.31:0/3681933257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eb530 msgr2=0x55cb68504570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eb530 0x55cb68504570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 -- 172.21.15.31:0/3681933257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684ec2f0 msgr2=0x55cb684ec6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684ec2f0 0x55cb684ec6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 -- 172.21.15.31:0/3681933257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb68290250 con 0x55cb684eabe0 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224ed9d700 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684ec2f0 0x55cb684ec6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.104+0000 7f224e59c700 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 0x55cb6850fbe0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2240024fd0 tx=0x7f224000be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2a5e69688f77609 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f22457fa700 1 -- 172.21.15.31:0/3681933257 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224000fbf0 con 0x55cb684eabe0 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f22457fa700 1 -- 172.21.15.31:0/3681933257 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f224002c040 con 0x55cb684eabe0 2023-07-29T16:18:30.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f22457fa700 1 -- 172.21.15.31:0/3681933257 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2240016ae0 con 0x55cb684eabe0 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f2268254b80 1 -- 172.21.15.31:0/3681933257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 msgr2=0x55cb6850fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f2268254b80 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 0x55cb6850fbe0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2240024fd0 tx=0x7f224000be00 comp rx=0 tx=0).stop 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f2268254b80 1 -- 172.21.15.31:0/3681933257 shutdown_connections 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f2268254b80 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684ec2f0 0x55cb684ec6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.105+0000 7f2268254b80 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eabe0 0x55cb6850fbe0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.106+0000 7f2268254b80 1 --2- 172.21.15.31:0/3681933257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684eb530 0x55cb68504570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.106+0000 7f2268254b80 1 -- 172.21.15.31:0/3681933257 >> 172.21.15.31:0/3681933257 conn(0x55cb684e03e0 msgr2=0x55cb684e5960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.106+0000 7f2268254b80 1 -- 172.21.15.31:0/3681933257 shutdown_connections 2023-07-29T16:18:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.106+0000 7f2268254b80 1 -- 172.21.15.31:0/3681933257 wait complete. 2023-07-29T16:18:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.106+0000 7f2268254b80 1 -- start start 2023-07-29T16:18:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.106+0000 7f2268254b80 1 Processor -- start 2023-07-29T16:18:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb68500530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb68500a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb68503a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb68503fd0 con 0x55cb684eabe0 2023-07-29T16:18:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb68504140 con 0x55cb684ec2f0 2023-07-29T16:18:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb68504300 con 0x55cb684eb530 2023-07-29T16:18:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2268254b80 10 client.0 _ll_register_callbacks cb 0x55cb683e1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:30.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f3dc1431b80 1 Processor -- start 2023-07-29T16:18:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f224ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb68503a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f224ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb68503a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56728/0 (socket says 172.21.15.31:56728) 2023-07-29T16:18:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f224ed9d700 1 -- 172.21.15.31:0/4099178390 learned_addr learned my addr 172.21.15.31:0/4099178390 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f3dc1431b80 1 -- start start 2023-07-29T16:18:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.107+0000 7f2247fff700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb68500a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f224e59c700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb68500530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 0x56411a5e4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5ee800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5eded0 con 0x56411a5e4f80 2023-07-29T16:18:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5dbee0 con 0x56411a5e6370 2023-07-29T16:18:30.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5f5e80 con 0x56411a43c6b0 2023-07-29T16:18:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2247fff700 1 -- 172.21.15.31:0/4099178390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 msgr2=0x55cb68503a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2247fff700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb68503a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2247fff700 1 -- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 msgr2=0x55cb68500530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2247fff700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb68500530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2247fff700 1 -- 172.21.15.31:0/4099178390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb68252c60 con 0x55cb684eb530 2023-07-29T16:18:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 0x56411a5e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 0x56411a5e4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41664/0 (socket says 172.21.15.31:41664) 2023-07-29T16:18:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 -- 172.21.15.31:0/467474246 learned_addr learned my addr 172.21.15.31:0/467474246 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da7fff700 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2247fff700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb68500a70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f22380995c0 tx=0x7f2238097300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a569310fa485d3d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2235ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:30.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2235ffb700 1 -- 172.21.15.31:0/4099178390 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22380b0040 con 0x55cb684eb530 2023-07-29T16:18:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2235ffb700 1 -- 172.21.15.31:0/4099178390 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f22380970a0 con 0x55cb684eb530 2023-07-29T16:18:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2238098f60 con 0x55cb684eb530 2023-07-29T16:18:30.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 -- 172.21.15.31:0/467474246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 msgr2=0x56411a5ee800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5ee800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 -- 172.21.15.31:0/467474246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 msgr2=0x56411a5ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5ee2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.108+0000 7f3da77fe700 1 -- 172.21.15.31:0/467474246 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56411a348c00 con 0x56411a43c6b0 2023-07-29T16:18:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2235ffb700 1 -- 172.21.15.31:0/4099178390 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f223800b670 con 0x55cb684eb530 2023-07-29T16:18:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 10 client.26524 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55cb682682b0 con 0x55cb684eb530 2023-07-29T16:18:30.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f3da77fe700 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 0x56411a5e4960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009ce0 tx=0x7f3d98005ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=805e035a36e6553b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2235ffb700 1 -- 172.21.15.31:0/4099178390 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2238098f60 con 0x55cb684eb530 2023-07-29T16:18:30.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f3d8e7fc700 1 -- 172.21.15.31:0/467474246 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98004d70 con 0x56411a43c6b0 2023-07-29T16:18:30.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f3d8e7fc700 1 -- 172.21.15.31:0/467474246 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d9801e040 con 0x56411a43c6b0 2023-07-29T16:18:30.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f3d8e7fc700 1 -- 172.21.15.31:0/467474246 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98024bb0 con 0x56411a43c6b0 2023-07-29T16:18:30.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2235ffb700 1 -- 172.21.15.31:0/4099178390 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2238093b70 con 0x55cb684eb530 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 10 client.26524 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 10 client.26524 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 20 client.26524 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb6855b6d0 0x55cb6855dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 4 client.26524 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.109+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cb68434d40 con 0x55cb6855b6d0 2023-07-29T16:18:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f224e59c700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb6855b6d0 0x55cb6855dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 -- 172.21.15.31:0/467474246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 msgr2=0x56411a5e4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 0x56411a5e4960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3d98009ce0 tx=0x7f3d98005ac0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 -- 172.21.15.31:0/467474246 shutdown_connections 2023-07-29T16:18:30.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a5e6370 0x56411a5ee800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a43c6b0 0x56411a5e4960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/467474246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 -- 172.21.15.31:0/467474246 >> 172.21.15.31:0/467474246 conn(0x56411a5d5fd0 msgr2=0x56411a5e8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 -- 172.21.15.31:0/467474246 shutdown_connections 2023-07-29T16:18:30.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f3dc1431b80 1 -- 172.21.15.31:0/467474246 wait complete. 2023-07-29T16:18:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f2235ffb700 10 client.26524 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:18:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.110+0000 7f224e59c700 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb6855b6d0 0x55cb6855dae0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84c64ad2d99dac15 server_cookie=4449d5f285d43c3e in_seq=0 out_seq=0 2023-07-29T16:18:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2235ffb700 1 -- 172.21.15.31:0/4099178390 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cb68434d40 con 0x55cb6855b6d0 2023-07-29T16:18:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2235ffb700 10 client.26524 handle_command_reply: tid=0 2023-07-29T16:18:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 client.26524 shutdown 2023-07-29T16:18:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 -- start start 2023-07-29T16:18:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 msgr2=0x55cb68500a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb68500a70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f22380995c0 tx=0x7f2238097300 comp rx=0 tx=0).stop 2023-07-29T16:18:30.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 Processor -- start 2023-07-29T16:18:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 shutdown_connections 2023-07-29T16:18:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55cb6855b6d0 0x55cb6855dae0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cb684ec2f0 0x55cb68503a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cb684eabe0 0x55cb68500530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2235ffb700 0 client.26524 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 --2- 172.21.15.31:0/4099178390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cb684eb530 0x55cb68500a70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 >> 172.21.15.31:0/4099178390 conn(0x55cb684e03e0 msgr2=0x55cb684f3510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 shutdown_connections 2023-07-29T16:18:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f2268254b80 1 -- 172.21.15.31:0/4099178390 wait complete. 2023-07-29T16:18:30.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5e0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5e1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5df680 con 0x56411a5e4f80 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5df7f0 con 0x56411a43c6b0 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56411a5df9b0 con 0x56411a5e6370 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.111+0000 7f3dc1431b80 10 client.0 _ll_register_callbacks cb 0x56411a4d75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5e0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5e0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56744/0 (socket says 172.21.15.31:56744) 2023-07-29T16:18:30.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 -- 172.21.15.31:0/3148774369 learned_addr learned my addr 172.21.15.31:0/3148774369 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f2268254b80 20 client.26524 trim_cache size 0 max 16384 2023-07-29T16:18:30.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5e1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:18:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da7fff700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 -- 172.21.15.31:0/3148774369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 msgr2=0x56411a5f5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 -- 172.21.15.31:0/3148774369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 msgr2=0x56411a5e1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5e1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 -- 172.21.15.31:0/3148774369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da00090f0 con 0x56411a43c6b0 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da7fff700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3da77fe700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5e0e10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800fc60 tx=0x7f3d98024ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=829cb0956fa99b96 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f3d86ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f223c0a0810 msgr2=0x7f223c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.112+0000 7f22549ff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f223c0a0810 0x7f223c0a2cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f224000b5c0 tx=0x7f2240005fd0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 msgr2=0x7f22501987a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 0x7f22501987a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f223800d980 tx=0x7f223800dc90 comp rx=0 tx=0).stop 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 shutdown_connections 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f223c0a0810 0x7f223c0a2cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2250106080 0x7f225019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2250100e40 0x7f2250198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 --2- 172.21.15.31:0/1199885499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2250103760 0x7f22501987a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 >> 172.21.15.31:0/1199885499 conn(0x7f22500fa840 msgr2=0x7f2250102280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f3d86ffd700 1 -- 172.21.15.31:0/3148774369 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800f870 con 0x56411a43c6b0 2023-07-29T16:18:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f3d86ffd700 1 -- 172.21.15.31:0/3148774369 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3d9800f9d0 con 0x56411a43c6b0 2023-07-29T16:18:30.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3da000f040 con 0x56411a43c6b0 2023-07-29T16:18:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 shutdown_connections 2023-07-29T16:18:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f22549ff700 1 -- 172.21.15.31:0/1199885499 wait complete. 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f3d86ffd700 1 -- 172.21.15.31:0/3148774369 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d9800b8f0 con 0x56411a43c6b0 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f3dc1431b80 10 client.26510 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.113+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56411a3861f0 con 0x56411a43c6b0 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3d86ffd700 1 -- 172.21.15.31:0/3148774369 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3da000f040 con 0x56411a43c6b0 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3d86ffd700 1 -- 172.21.15.31:0/3148774369 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3d98026460 con 0x56411a43c6b0 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3dc1431b80 10 client.26510 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:30.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3dc1431b80 10 client.26510 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3dc1431b80 20 client.26510 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a650f70 0x56411a653350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3dc1431b80 4 client.26510 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56411a52ace0 con 0x56411a650f70 2023-07-29T16:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a650f70 0x56411a653350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3da6ffd700 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a650f70 0x56411a653350 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73fbf7c3f6a0a9df server_cookie=9dea177a6fb0e66f in_seq=0 out_seq=0 2023-07-29T16:18:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.114+0000 7f3d86ffd700 10 client.26510 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:18:30.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3d86ffd700 1 -- 172.21.15.31:0/3148774369 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56411a52ace0 con 0x56411a650f70 2023-07-29T16:18:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3d86ffd700 10 client.26510 handle_command_reply: tid=0 2023-07-29T16:18:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 client.26510 shutdown 2023-07-29T16:18:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 msgr2=0x56411a5e0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5e0e10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800fc60 tx=0x7f3d98024ba0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 shutdown_connections 2023-07-29T16:18:30.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56411a650f70 0x56411a653350 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56411a5e6370 0x56411a5f5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56411a43c6b0 0x56411a5e0e10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 --2- 172.21.15.31:0/3148774369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411a5e4f80 0x56411a5e1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 >> 172.21.15.31:0/3148774369 conn(0x56411a5d5fd0 msgr2=0x56411a5ee2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.285+0000 7f3d86ffd700 0 client.26510 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 shutdown_connections 2023-07-29T16:18:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dc1431b80 1 -- 172.21.15.31:0/3148774369 wait complete. 2023-07-29T16:18:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dc1431b80 20 client.26510 trim_cache size 0 max 16384 2023-07-29T16:18:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:18:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:18:30.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:18:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:18:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d900a0810 msgr2=0x7f3d900a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dadbff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d900a0810 0x7f3d900a2cc0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800b2e0 tx=0x7f3d98005fb0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 msgr2=0x7f3da8196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.286+0000 7f3dadbff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 0x7f3da8196570 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00d980 tx=0x7f3d9c00dc90 comp rx=0 tx=0).stop 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 shutdown_connections 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d900a0810 0x7f3d900a2cc0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3da8101ef0 0x7f3da819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da81002b0 0x7f3da8196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 --2- 172.21.15.31:0/2265542830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3da8101540 0x7f3da8196570 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 >> 172.21.15.31:0/2265542830 conn(0x7f3da80fb8d0 msgr2=0x7f3da8104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 shutdown_connections 2023-07-29T16:18:30.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:30.287+0000 7f3dadbff700 1 -- 172.21.15.31:0/2265542830 wait complete. 2023-07-29T16:18:30.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 -- 172.21.15.31:0/21071414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 msgr2=0x7f58d80731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 --2- 172.21.15.31:0/21071414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d80731e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009990 tx=0x7f58d4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 -- 172.21.15.31:0/21071414 shutdown_connections 2023-07-29T16:18:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 --2- 172.21.15.31:0/21071414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d810c900 0x7f58d810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 --2- 172.21.15.31:0/21071414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d80731e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 --2- 172.21.15.31:0/21071414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58d80737b0 0x7f58d8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 -- 172.21.15.31:0/21071414 >> 172.21.15.31:0/21071414 conn(0x7f58d80fb970 msgr2=0x7f58d80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 -- 172.21.15.31:0/21071414 shutdown_connections 2023-07-29T16:18:30.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.435+0000 7f58df5ff700 1 -- 172.21.15.31:0/21071414 wait complete. 2023-07-29T16:18:30.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 Processor -- start 2023-07-29T16:18:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 -- start start 2023-07-29T16:18:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d80737b0 0x7f58d819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58d810c900 0x7f58d81a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d81a4250 con 0x7f58d80737b0 2023-07-29T16:18:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d81a4390 con 0x7f58d8074d90 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.436+0000 7f58df5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d81a4520 con 0x7f58d810c900 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d819cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56770/0 (socket says 172.21.15.31:56770) 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 -- 172.21.15.31:0/3740284613 learned_addr learned my addr 172.21.15.31:0/3740284613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58d810c900 0x7f58d81a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dd39b700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d80737b0 0x7f58d819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 -- 172.21.15.31:0/3740284613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58d810c900 msgr2=0x7f58d81a3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58d810c900 0x7f58d81a3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 -- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d80737b0 msgr2=0x7f58d819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d80737b0 0x7f58d819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 -- 172.21.15.31:0/3740284613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40095f0 con 0x7f58d8074d90 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d819cc10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f58c800e970 tx=0x7f58c800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58ce7fc700 1 -- 172.21.15.31:0/3740284613 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8009790 con 0x7f58d8074d90 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58ce7fc700 1 -- 172.21.15.31:0/3740284613 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58c800eec0 con 0x7f58d8074d90 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d81a4800 con 0x7f58d8074d90 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.437+0000 7f58ce7fc700 1 -- 172.21.15.31:0/3740284613 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c8003cf0 con 0x7f58d8074d90 2023-07-29T16:18:30.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.438+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d81a4d10 con 0x7f58d8074d90 2023-07-29T16:18:30.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.441+0000 7f58ce7fc700 1 -- 172.21.15.31:0/3740284613 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f58c800ccd0 con 0x7f58d8074d90 2023-07-29T16:18:30.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.442+0000 7f58f3003b80 1 Processor -- start 2023-07-29T16:18:30.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.442+0000 7f58f3003b80 1 -- start start 2023-07-29T16:18:30.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.442+0000 7f58ce7fc700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58c40a0790 0x7f58c40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.442+0000 7f58ce7fc700 1 -- 172.21.15.31:0/3740284613 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58c8014070 con 0x7f58d8074d90 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eeeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58c40a0790 0x7f58c40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627eef530 0x559627eefd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627ef02f0 0x559627ef0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58f3003b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627e46a10 con 0x559627ef02f0 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58f3003b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627e443d0 con 0x559627eef530 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58f3003b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627e3fcc0 con 0x559627eeebe0 2023-07-29T16:18:30.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eeeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eeeff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41694/0 (socket says 172.21.15.31:41694) 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dcb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627eef530 0x559627eefd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 -- 172.21.15.31:0/689996729 learned_addr learned my addr 172.21.15.31:0/689996729 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627ef02f0 0x559627ef0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 -- 172.21.15.31:0/689996729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627eef530 msgr2=0x559627eefd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627eef530 0x559627eefd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 -- 172.21.15.31:0/689996729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627ef02f0 msgr2=0x559627ef0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627ef02f0 0x559627ef0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dd39b700 1 -- 172.21.15.31:0/689996729 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559627c6dde0 con 0x559627eeebe0 2023-07-29T16:18:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.443+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627eef530 0x559627eefd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.444+0000 7f58dd39b700 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eeeff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009990 tx=0x7f58d4005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd39477482231b0e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.444+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58c40a0790 0x7f58c40a2c40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f58d000f870 tx=0x7f58d000fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.444+0000 7f58c37fe700 1 -- 172.21.15.31:0/689996729 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d400beb0 con 0x559627eeebe0 2023-07-29T16:18:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.444+0000 7f58c37fe700 1 -- 172.21.15.31:0/689996729 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58d401e040 con 0x559627eeebe0 2023-07-29T16:18:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.444+0000 7f58c37fe700 1 -- 172.21.15.31:0/689996729 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4004af0 con 0x559627eeebe0 2023-07-29T16:18:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 -- 172.21.15.31:0/689996729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 msgr2=0x559627eeeff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eeeff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009990 tx=0x7f58d4005e80 comp rx=0 tx=0).stop 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 -- 172.21.15.31:0/689996729 shutdown_connections 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627ef02f0 0x559627ef0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eeeff0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 --2- 172.21.15.31:0/689996729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627eef530 0x559627eefd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 -- 172.21.15.31:0/689996729 >> 172.21.15.31:0/689996729 conn(0x559627ee43e0 msgr2=0x559627ee9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 -- 172.21.15.31:0/689996729 shutdown_connections 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 -- 172.21.15.31:0/689996729 wait complete. 2023-07-29T16:18:30.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.445+0000 7f58f3003b80 1 -- start start 2023-07-29T16:18:30.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 Processor -- start 2023-07-29T16:18:30.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eeebe0 0x559627ef8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eef530 0x559627ef8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627effd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627ddc510 con 0x559627eeebe0 2023-07-29T16:18:30.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627ddc1b0 con 0x559627ef02f0 2023-07-29T16:18:30.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627cc3920 con 0x559627eef530 2023-07-29T16:18:30.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58f3003b80 10 client.0 _ll_register_callbacks cb 0x559627de5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:30.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58ddb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627effd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58dcb9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eef530 0x559627ef8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eeebe0 0x559627ef8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58ddb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627effd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56788/0 (socket says 172.21.15.31:56788) 2023-07-29T16:18:30.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.446+0000 7f58ddb9c700 1 -- 172.21.15.31:0/1995261704 learned_addr learned my addr 172.21.15.31:0/1995261704 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58ddb9c700 1 -- 172.21.15.31:0/1995261704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eef530 msgr2=0x559627ef8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eef530 0x559627ef8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58ddb9c700 1 -- 172.21.15.31:0/1995261704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eeebe0 msgr2=0x559627ef8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eeebe0 0x559627ef8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58ddb9c700 1 -- 172.21.15.31:0/1995261704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d81a4d10 con 0x559627ef02f0 2023-07-29T16:18:30.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627effd10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f58d0018b00 tx=0x7f58d0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b767d7e3278704d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:30.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58c0ff9700 1 -- 172.21.15.31:0/1995261704 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d000e040 con 0x559627ef02f0 2023-07-29T16:18:30.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58c0ff9700 1 -- 172.21.15.31:0/1995261704 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58d0008db0 con 0x559627ef02f0 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.447+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559627e6ad10 con 0x559627ef02f0 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58c0ff9700 1 -- 172.21.15.31:0/1995261704 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58d0018ed0 con 0x559627ef02f0 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 10 client.26522 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f58d810f690 con 0x559627ef02f0 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58c0ff9700 1 -- 172.21.15.31:0/1995261704 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559627e6ad10 con 0x559627ef02f0 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58c0ff9700 1 -- 172.21.15.31:0/1995261704 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f58d001a440 con 0x559627ef02f0 2023-07-29T16:18:30.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 10 client.26522 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 10 client.26522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 20 client.26522 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f04550 0x559627f06960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 4 client.26522 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.448+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559627ebdd70 con 0x559627f04550 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.449+0000 7f58dd39b700 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f04550 0x559627f06960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.449+0000 7f58dd39b700 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f04550 0x559627f06960 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7ebd26d16dc087a server_cookie=259abd580d9f6d4f in_seq=0 out_seq=0 2023-07-29T16:18:30.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.449+0000 7f58c0ff9700 10 client.26522 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58c0ff9700 1 -- 172.21.15.31:0/1995261704 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559627ebdd70 con 0x559627f04550 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58c0ff9700 10 client.26522 handle_command_reply: tid=0 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 client.26522 shutdown 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 msgr2=0x559627effd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627effd10 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f58d0018b00 tx=0x7f58d0004b40 comp rx=0 tx=0).stop 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 shutdown_connections 2023-07-29T16:18:30.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f04550 0x559627f06960 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627effd10 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58c0ff9700 0 client.26522 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eeebe0 0x559627ef8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 --2- 172.21.15.31:0/1995261704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eef530 0x559627ef8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 >> 172.21.15.31:0/1995261704 conn(0x559627ee43e0 msgr2=0x559627ee9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 shutdown_connections 2023-07-29T16:18:30.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 1 -- 172.21.15.31:0/1995261704 wait complete. 2023-07-29T16:18:30.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.450+0000 7f58f3003b80 20 client.26522 trim_cache size 0 max 16384 2023-07-29T16:18:30.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.464+0000 7f58f3003b80 1 Processor -- start 2023-07-29T16:18:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.464+0000 7f58f3003b80 1 -- start start 2023-07-29T16:18:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.464+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627ef06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.464+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627ef66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58f3003b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627ee6460 con 0x559627eef530 2023-07-29T16:18:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58f3003b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627ef6490 con 0x559627ef02f0 2023-07-29T16:18:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58f3003b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627eeb080 con 0x559627eeebe0 2023-07-29T16:18:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627ef06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627ef66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58dcb9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627ef06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41710/0 (socket says 172.21.15.31:41710) 2023-07-29T16:18:30.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58dd39b700 1 -- 172.21.15.31:0/816583209 learned_addr learned my addr 172.21.15.31:0/816583209 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 -- 172.21.15.31:0/816583209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 msgr2=0x559627ef06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627ef06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 -- 172.21.15.31:0/816583209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 msgr2=0x559627effcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 -- 172.21.15.31:0/816583209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559627c56c60 con 0x559627ef02f0 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:30.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.465+0000 7f58ddb9c700 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627ef66e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f58d0018ad0 tx=0x7f58d00098a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ee707dc27fca55 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58c3fff700 1 -- 172.21.15.31:0/816583209 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d000e030 con 0x559627ef02f0 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58c3fff700 1 -- 172.21.15.31:0/816583209 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58d000f040 con 0x559627ef02f0 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58c3fff700 1 -- 172.21.15.31:0/816583209 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d001a5b0 con 0x559627ef02f0 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 -- 172.21.15.31:0/816583209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 msgr2=0x559627ef66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627ef66e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f58d0018ad0 tx=0x7f58d00098a0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 -- 172.21.15.31:0/816583209 shutdown_connections 2023-07-29T16:18:30.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627ef66e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627ef06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 --2- 172.21.15.31:0/816583209 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 -- 172.21.15.31:0/816583209 >> 172.21.15.31:0/816583209 conn(0x559627ee43e0 msgr2=0x559627ef7480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.466+0000 7f58f3003b80 1 -- 172.21.15.31:0/816583209 shutdown_connections 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.467+0000 7f58f3003b80 1 -- 172.21.15.31:0/816583209 wait complete. 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.467+0000 7f58f3003b80 1 -- start start 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.467+0000 7f58f3003b80 1 Processor -- start 2023-07-29T16:18:30.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eff610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627efc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627efc790 con 0x559627eef530 2023-07-29T16:18:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627efc900 con 0x559627ef02f0 2023-07-29T16:18:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559627efcac0 con 0x559627eeebe0 2023-07-29T16:18:30.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58f3003b80 10 client.0 _ll_register_callbacks cb 0x559627de5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:30.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58ddb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627efc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58dcb9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58ddb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627efc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56794/0 (socket says 172.21.15.31:56794) 2023-07-29T16:18:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58ddb9c700 1 -- 172.21.15.31:0/4155490941 learned_addr learned my addr 172.21.15.31:0/4155490941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.468+0000 7f58dd39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eff610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41726/0 (socket says 172.21.15.31:41726) 2023-07-29T16:18:30.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dd39b700 1 -- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 msgr2=0x559627efc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dd39b700 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627efc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dd39b700 1 -- 172.21.15.31:0/4155490941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 msgr2=0x559627effb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dd39b700 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dd39b700 1 -- 172.21.15.31:0/4155490941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d81a4800 con 0x559627eeebe0 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58dd39b700 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eff610 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f58d4025040 tx=0x7f58d400bfc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f973b073cec02f8b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:30.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58c0ff9700 1 -- 172.21.15.31:0/4155490941 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4005450 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58c0ff9700 1 -- 172.21.15.31:0/4155490941 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58d40059a0 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58d401e040 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.469+0000 7f58c0ff9700 1 -- 172.21.15.31:0/4155490941 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58d40055b0 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 10 client.26545 fetch_fsmap learned FSMap version 23 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f58d810f690 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58c0ff9700 1 -- 172.21.15.31:0/4155490941 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58d401e040 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58c0ff9700 1 -- 172.21.15.31:0/4155490941 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 23) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f58d4040580 con 0x559627eeebe0 2023-07-29T16:18:30.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 10 client.26545 fetch_fsmap finished waiting for FSMap version 23 2023-07-29T16:18:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 10 client.26545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 20 client.26545 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f571b0 0x559627f59590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 4 client.26545 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:18:30.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559627e38d40 con 0x559627f571b0 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.470+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f571b0 0x559627f59590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.471+0000 7f58dcb9a700 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f571b0 0x559627f59590 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d5be40c2242c14b server_cookie=6e6b5072d531d3c0 in_seq=0 out_seq=0 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.471+0000 7f58c0ff9700 10 client.26545 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.471+0000 7f58c0ff9700 1 -- 172.21.15.31:0/4155490941 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559627e38d40 con 0x559627f571b0 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.471+0000 7f58c0ff9700 10 client.26545 handle_command_reply: tid=0 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.471+0000 7f58f3003b80 1 client.26545 shutdown 2023-07-29T16:18:30.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 msgr2=0x559627eff610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eff610 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f58d4025040 tx=0x7f58d400bfc0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 shutdown_connections 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559627f571b0 0x559627f59590 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559627ef02f0 0x559627efc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559627eeebe0 0x559627eff610 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 --2- 172.21.15.31:0/4155490941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559627eef530 0x559627effb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58c0ff9700 0 client.26545 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 >> 172.21.15.31:0/4155490941 conn(0x559627ee43e0 msgr2=0x559627ef66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 shutdown_connections 2023-07-29T16:18:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 1 -- 172.21.15.31:0/4155490941 wait complete. 2023-07-29T16:18:30.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.472+0000 7f58f3003b80 20 client.26545 trim_cache size 0 max 16384 2023-07-29T16:18:30.488 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:18:30.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58c40a0790 msgr2=0x7f58c40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58c40a0790 0x7f58c40a2c40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f58d000f870 tx=0x7f58d000fc30 comp rx=0 tx=0).stop 2023-07-29T16:18:30.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 msgr2=0x7f58d819cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:30.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d819cc10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f58c800e970 tx=0x7f58c800ec80 comp rx=0 tx=0).stop 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 shutdown_connections 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58c40a0790 0x7f58c40a2c40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58d810c900 0x7f58d81a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58d80737b0 0x7f58d819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 --2- 172.21.15.31:0/3740284613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58d8074d90 0x7f58d819cc10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 >> 172.21.15.31:0/3740284613 conn(0x7f58d80fb970 msgr2=0x7f58d80fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 shutdown_connections 2023-07-29T16:18:30.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:30.473+0000 7f58df5ff700 1 -- 172.21.15.31:0/3740284613 wait complete. 2023-07-29T16:18:30.681 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:30 smithi156 ceph-mon[129245]: pgmap v460: 97 pgs: 97 active+clean; 13 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.3 MiB/s wr, 194 op/s 2023-07-29T16:18:30.681 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:30 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2416137532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:18:30.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:30 smithi031 ceph-mon[95504]: pgmap v460: 97 pgs: 97 active+clean; 13 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.3 MiB/s wr, 194 op/s 2023-07-29T16:18:30.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:30 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2416137532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:18:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:30 smithi039 ceph-mon[128858]: pgmap v460: 97 pgs: 97 active+clean; 13 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.3 MiB/s wr, 194 op/s 2023-07-29T16:18:30.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:30 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2416137532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:18:30.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 32241 12.85 MB/sec warmup 15 sec latency 52.113 ms 2023-07-29T16:18:31.071 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:18:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: ignoring --setuser ceph since I am not root 2023-07-29T16:18:31.071 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:18:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: ignoring --setgroup ceph since I am not root 2023-07-29T16:18:31.072 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:18:30 smithi156 ceph-mds[155210]: -- 172.21.15.156:0/1824526183 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558b2c5171e0 con 0x558b2c4ed000 2023-07-29T16:18:31.072 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:18:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: starting mds.c at 2023-07-29T16:18:31.491 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:18:31.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:18:31.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:31 smithi031 ceph-mon[95504]: daemon mds.c restarted 2023-07-29T16:18:31.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:31 smithi031 ceph-mon[95504]: osdmap e91: 12 total, 12 up, 12 in 2023-07-29T16:18:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 -- 172.21.15.31:0/2199920624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154101510 msgr2=0x7fc154101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2199920624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154101510 0x7fc154101960 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc148009990 tx=0x7fc148009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 -- 172.21.15.31:0/2199920624 shutdown_connections 2023-07-29T16:18:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2199920624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154101ea0 0x7fc154108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2199920624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc154100320 0x7fc154100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2199920624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154101510 0x7fc154101960 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 -- 172.21.15.31:0/2199920624 >> 172.21.15.31:0/2199920624 conn(0x7fc1540fb920 msgr2=0x7fc1540fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 -- 172.21.15.31:0/2199920624 shutdown_connections 2023-07-29T16:18:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.711+0000 7fc1593ff700 1 -- 172.21.15.31:0/2199920624 wait complete. 2023-07-29T16:18:31.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.712+0000 7fc1593ff700 1 Processor -- start 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.712+0000 7fc1593ff700 1 -- start start 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1593ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 0x7fc154198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1593ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc154101510 0x7fc1541987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1593ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154101ea0 0x7fc15419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1593ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15419fdf0 con 0x7fc154101510 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1593ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15419ff30 con 0x7fc154101ea0 2023-07-29T16:18:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1593ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1541a00c0 con 0x7fc154100320 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 0x7fc154198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 0x7fc154198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41740/0 (socket says 172.21.15.31:41740) 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 -- 172.21.15.31:0/2840141697 learned_addr learned my addr 172.21.15.31:0/2840141697 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1537fe700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154101ea0 0x7fc15419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc154101510 0x7fc1541987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 -- 172.21.15.31:0/2840141697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154101ea0 msgr2=0x7fc15419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154101ea0 0x7fc15419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 -- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc154101510 msgr2=0x7fc1541987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc154101510 0x7fc1541987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.713+0000 7fc152ffd700 1 -- 172.21.15.31:0/2840141697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1480095f0 con 0x7fc154100320 2023-07-29T16:18:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.714+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 0x7fc154198270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e970 tx=0x7fc14400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.714+0000 7fc13bfff700 1 -- 172.21.15.31:0/2840141697 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144009790 con 0x7fc154100320 2023-07-29T16:18:31.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.714+0000 7fc13bfff700 1 -- 172.21.15.31:0/2840141697 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc14400c7e0 con 0x7fc154100320 2023-07-29T16:18:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.714+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1541a03a0 con 0x7fc154100320 2023-07-29T16:18:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.714+0000 7fc13bfff700 1 -- 172.21.15.31:0/2840141697 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144003c70 con 0x7fc154100320 2023-07-29T16:18:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.714+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1541a0900 con 0x7fc154100320 2023-07-29T16:18:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.717+0000 7fc13bfff700 1 -- 172.21.15.31:0/2840141697 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc14400cc50 con 0x7fc154100320 2023-07-29T16:18:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.718+0000 7fc16cd4cb80 1 Processor -- start 2023-07-29T16:18:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 -- start start 2023-07-29T16:18:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc13bfff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc13c0a0810 0x7fc13c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d70070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d70de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc13bfff700 1 -- 172.21.15.31:0/2840141697 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc144014070 con 0x7fc154100320 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395cc7a90 con 0x564395d705b0 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395cc5450 con 0x564395d71370 2023-07-29T16:18:31.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395cc0d40 con 0x564395d6fc60 2023-07-29T16:18:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc1527fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d70de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc1527fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d70de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52922/0 (socket says 172.21.15.31:52922) 2023-07-29T16:18:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc1527fc700 1 -- 172.21.15.31:0/3655602505 learned_addr learned my addr 172.21.15.31:0/3655602505 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc1537fe700 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc152ffd700 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d70070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.719+0000 7fc1537fe700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc13c0a0810 0x7fc13c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc152ffd700 1 -- 172.21.15.31:0/3655602505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 msgr2=0x564395d717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc152ffd700 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d717c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc152ffd700 1 -- 172.21.15.31:0/3655602505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 msgr2=0x564395d70de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc152ffd700 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d70de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc152ffd700 1 -- 172.21.15.31:0/3655602505 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564395aeee60 con 0x564395d6fc60 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc152ffd700 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d70070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc144099100 tx=0x7fc144003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b44e33ea237e89 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc1397fa700 1 -- 172.21.15.31:0/3655602505 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14409d430 con 0x564395d6fc60 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc1537fe700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc13c0a0810 0x7fc13c0a2cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009ec0 tx=0x7fc14c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc1397fa700 1 -- 172.21.15.31:0/3655602505 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1440a1850 con 0x564395d6fc60 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.720+0000 7fc1397fa700 1 -- 172.21.15.31:0/3655602505 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1440998c0 con 0x564395d6fc60 2023-07-29T16:18:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/3655602505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 msgr2=0x564395d70070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d70070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc144099100 tx=0x7fc144003710 comp rx=0 tx=0).stop 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/3655602505 shutdown_connections 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d717c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d70070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/3655602505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d70de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/3655602505 >> 172.21.15.31:0/3655602505 conn(0x564395d65460 msgr2=0x564395d6aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/3655602505 shutdown_connections 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.721+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/3655602505 wait complete. 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 -- start start 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 Processor -- start 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d798c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d79e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d80e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395c5d590 con 0x564395d705b0 2023-07-29T16:18:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395c5d230 con 0x564395d71370 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.722+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395b449a0 con 0x564395d6fc60 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc16cd4cb80 10 client.0 _ll_register_callbacks cb 0x564395c666a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d798c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc1537fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d80e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d798c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41756/0 (socket says 172.21.15.31:41756) 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 -- 172.21.15.31:0/145070708 learned_addr learned my addr 172.21.15.31:0/145070708 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc1537fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d80e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56816/0 (socket says 172.21.15.31:56816) 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc1527fc700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d79e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 -- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 msgr2=0x564395d80e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d80e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 -- 172.21.15.31:0/145070708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 msgr2=0x564395d79e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d79e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 -- 172.21.15.31:0/145070708 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1541a0900 con 0x564395d6fc60 2023-07-29T16:18:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc1527fc700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d79e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc152ffd700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d798c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc144009230 tx=0x7fc144098c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b7b4a25889603c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.723+0000 7fc136ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc136ff5700 1 -- 172.21.15.31:0/145070708 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14409d430 con 0x564395d6fc60 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc136ff5700 1 -- 172.21.15.31:0/145070708 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc144099660 con 0x564395d6fc60 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564395cebd90 con 0x564395d6fc60 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc136ff5700 1 -- 172.21.15.31:0/145070708 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc144099820 con 0x564395d6fc60 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 10 client.26563 fetch_fsmap learned FSMap version 25 2023-07-29T16:18:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564395ad7ce0 con 0x564395d6fc60 2023-07-29T16:18:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc136ff5700 1 -- 172.21.15.31:0/145070708 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564395cebd90 con 0x564395d6fc60 2023-07-29T16:18:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc136ff5700 1 -- 172.21.15.31:0/145070708 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 25) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc1440953b0 con 0x564395d6fc60 2023-07-29T16:18:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 10 client.26563 fetch_fsmap finished waiting for FSMap version 25 2023-07-29T16:18:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 10 client.26563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 20 client.26563 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395d851c0 0x564395d875d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 4 client.26563 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.724+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564395d3edf0 con 0x564395d851c0 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.725+0000 7fc1527fc700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395d851c0 0x564395d875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.725+0000 7fc1527fc700 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395d851c0 0x564395d875d0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=828cc1a0c3c76b59 server_cookie=da739473ca63b619 in_seq=0 out_seq=0 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.725+0000 7fc136ff5700 10 client.26563 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc136ff5700 1 -- 172.21.15.31:0/145070708 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564395d3edf0 con 0x564395d851c0 2023-07-29T16:18:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc136ff5700 10 client.26563 handle_command_reply: tid=0 2023-07-29T16:18:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 client.26563 shutdown 2023-07-29T16:18:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 msgr2=0x564395d798c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d798c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc144009230 tx=0x7fc144098c60 comp rx=0 tx=0).stop 2023-07-29T16:18:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 shutdown_connections 2023-07-29T16:18:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395d851c0 0x564395d875d0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d71370 0x564395d80e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d6fc60 0x564395d798c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/145070708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d79e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc136ff5700 0 client.26563 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:31.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.727+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 >> 172.21.15.31:0/145070708 conn(0x564395d65460 msgr2=0x564395d6a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.728+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 shutdown_connections 2023-07-29T16:18:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.728+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/145070708 wait complete. 2023-07-29T16:18:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.728+0000 7fc16cd4cb80 20 client.26563 trim_cache size 0 max 16384 2023-07-29T16:18:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.746+0000 7fc16cd4cb80 1 Processor -- start 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.746+0000 7fc16cd4cb80 1 -- start start 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d6fc60 0x564395d6a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 0x564395d80e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d77710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395d64f20 con 0x564395d6fc60 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395d78d70 con 0x564395d705b0 2023-07-29T16:18:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395d674e0 con 0x564395d71370 2023-07-29T16:18:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 0x564395d80e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 0x564395d80e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56824/0 (socket says 172.21.15.31:56824) 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 -- 172.21.15.31:0/2153743796 learned_addr learned my addr 172.21.15.31:0/2153743796 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d6fc60 0x564395d6a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1537fe700 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d77710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 -- 172.21.15.31:0/2153743796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 msgr2=0x564395d77710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d77710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 -- 172.21.15.31:0/2153743796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d6fc60 msgr2=0x564395d6a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d6fc60 0x564395d6a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc1527fc700 1 -- 172.21.15.31:0/2153743796 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564395b152d0 con 0x564395d705b0 2023-07-29T16:18:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.747+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d6fc60 0x564395d6a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.748+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 0x564395d80e00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x564395d709c0 tx=0x7fc148009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2cb10d0f8508397 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.748+0000 7fc139ffb700 1 -- 172.21.15.31:0/2153743796 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14801d070 con 0x564395d705b0 2023-07-29T16:18:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.748+0000 7fc139ffb700 1 -- 172.21.15.31:0/2153743796 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc14801e040 con 0x564395d705b0 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.748+0000 7fc139ffb700 1 -- 172.21.15.31:0/2153743796 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14800f6c0 con 0x564395d705b0 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.748+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2153743796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 msgr2=0x564395d80e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.748+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 0x564395d80e00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x564395d709c0 tx=0x7fc148009310 comp rx=0 tx=0).stop 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2153743796 shutdown_connections 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d77710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d6fc60 0x564395d6a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2153743796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d705b0 0x564395d80e00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2153743796 >> 172.21.15.31:0/2153743796 conn(0x564395d65460 msgr2=0x564395d78560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2153743796 shutdown_connections 2023-07-29T16:18:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2153743796 wait complete. 2023-07-29T16:18:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 -- start start 2023-07-29T16:18:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.749+0000 7fc16cd4cb80 1 Processor -- start 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 0x564395d7e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d7e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d7f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395d7fce0 con 0x564395d705b0 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395d7fe50 con 0x564395d6fc60 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564395d80010 con 0x564395d71370 2023-07-29T16:18:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc16cd4cb80 10 client.0 _ll_register_callbacks cb 0x564395c666a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc152ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 0x564395d7e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc152ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 0x564395d7e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56832/0 (socket says 172.21.15.31:56832) 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc152ffd700 1 -- 172.21.15.31:0/2842536935 learned_addr learned my addr 172.21.15.31:0/2842536935 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d7e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.750+0000 7fc1537fe700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d7f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc152ffd700 1 -- 172.21.15.31:0/2842536935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 msgr2=0x564395d7f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d7f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc152ffd700 1 -- 172.21.15.31:0/2842536935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 msgr2=0x564395d7e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d7e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc152ffd700 1 -- 172.21.15.31:0/2842536935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc154109090 con 0x564395d6fc60 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc152ffd700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 0x564395d7e250 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc144099390 tx=0x7fc1440979b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef73de5d825ee15 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc136ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc136ff5700 1 -- 172.21.15.31:0/2842536935 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1440a15c0 con 0x564395d6fc60 2023-07-29T16:18:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc136ff5700 1 -- 172.21.15.31:0/2842536935 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc144095b10 con 0x564395d6fc60 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1440a1850 con 0x564395d6fc60 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc136ff5700 1 -- 172.21.15.31:0/2842536935 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc144099870 con 0x564395d6fc60 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc16cd4cb80 10 client.26552 fetch_fsmap learned FSMap version 25 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.751+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564395aed330 con 0x564395d6fc60 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc136ff5700 1 -- 172.21.15.31:0/2842536935 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1440a1850 con 0x564395d6fc60 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc136ff5700 1 -- 172.21.15.31:0/2842536935 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 25) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc1440952d0 con 0x564395d6fc60 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc16cd4cb80 10 client.26552 fetch_fsmap finished waiting for FSMap version 25 2023-07-29T16:18:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc16cd4cb80 10 client.26552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc16cd4cb80 20 client.26552 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395dd8d50 0x564395ddb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc16cd4cb80 4 client.26552 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564395cb9dc0 con 0x564395dd8d50 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395dd8d50 0x564395ddb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc1527fc700 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395dd8d50 0x564395ddb130 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7a6eb34809faad7 server_cookie=bb494e23262f8e10 in_seq=0 out_seq=0 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.752+0000 7fc136ff5700 10 client.26552 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc136ff5700 1 -- 172.21.15.31:0/2842536935 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564395cb9dc0 con 0x564395dd8d50 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc136ff5700 10 client.26552 handle_command_reply: tid=0 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 client.26552 shutdown 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 msgr2=0x564395d7e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 0x564395d7e250 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc144099390 tx=0x7fc1440979b0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 shutdown_connections 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564395dd8d50 0x564395ddb130 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564395d71370 0x564395d7f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564395d6fc60 0x564395d7e250 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 --2- 172.21.15.31:0/2842536935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564395d705b0 0x564395d7e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc136ff5700 0 client.26552 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 >> 172.21.15.31:0/2842536935 conn(0x564395d65460 msgr2=0x564395d6a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 shutdown_connections 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.755+0000 7fc16cd4cb80 1 -- 172.21.15.31:0/2842536935 wait complete. 2023-07-29T16:18:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc16cd4cb80 20 client.26552 trim_cache size 0 max 16384 2023-07-29T16:18:31.761 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:18:31.761 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:18:31.761 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "47d15a20-2e4b-4fed-bf84-243809e4ab15", 2023-07-29T16:18:31.761 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:18:31.761 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:18:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc13c0a0810 msgr2=0x7fc13c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc13c0a0810 0x7fc13c0a2cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009ec0 tx=0x7fc14c00b040 comp rx=0 tx=0).stop 2023-07-29T16:18:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 msgr2=0x7fc154198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 0x7fc154198270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc14400e970 tx=0x7fc14400ec80 comp rx=0 tx=0).stop 2023-07-29T16:18:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 shutdown_connections 2023-07-29T16:18:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc13c0a0810 0x7fc13c0a2cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc154101ea0 0x7fc15419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc154100320 0x7fc154198270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 --2- 172.21.15.31:0/2840141697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc154101510 0x7fc1541987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 >> 172.21.15.31:0/2840141697 conn(0x7fc1540fb920 msgr2=0x7fc154106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 shutdown_connections 2023-07-29T16:18:31.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.756+0000 7fc1593ff700 1 -- 172.21.15.31:0/2840141697 wait complete. 2023-07-29T16:18:31.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:18:31.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:31 smithi039 ceph-mon[128858]: daemon mds.c restarted 2023-07-29T16:18:31.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:31 smithi039 ceph-mon[128858]: osdmap e91: 12 total, 12 up, 12 in 2023-07-29T16:18:31.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:31 smithi156 ceph-mon[129245]: daemon mds.c restarted 2023-07-29T16:18:31.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:31 smithi156 ceph-mon[129245]: osdmap e91: 12 total, 12 up, 12 in 2023-07-29T16:18:31.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 34213 12.43 MB/sec warmup 16 sec latency 35.706 ms 2023-07-29T16:18:31.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3408834024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 msgr2=0x7f2fe8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3408834024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe8108770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000b0a0 tx=0x7f2fe000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3408834024 shutdown_connections 2023-07-29T16:18:31.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3408834024 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe8108770 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3408834024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3408834024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fe8101520 0x7f2fe8101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3408834024 >> 172.21.15.31:0/3408834024 conn(0x7f2fe80fb970 msgr2=0x7f2fe80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:31.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3408834024 shutdown_connections 2023-07-29T16:18:31.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.990+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3408834024 wait complete. 2023-07-29T16:18:31.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.991+0000 7f2fed5ff700 1 Processor -- start 2023-07-29T16:18:31.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.991+0000 7f2fed5ff700 1 -- start start 2023-07-29T16:18:31.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.991+0000 7f2fed5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe81961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.991+0000 7f2fed5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fe8101520 0x7f2fe81966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.991+0000 7f2fed5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe819d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fed5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe819dd20 con 0x7f2fe8101eb0 2023-07-29T16:18:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fed5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe819de60 con 0x7f2fe8100330 2023-07-29T16:18:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fed5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe819dff0 con 0x7f2fe8101520 2023-07-29T16:18:31.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe81961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe81961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56842/0 (socket says 172.21.15.31:56842) 2023-07-29T16:18:31.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/3178876443 learned_addr learned my addr 172.21.15.31:0/3178876443 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe819d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fe8101520 0x7f2fe81966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:31.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/3178876443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fe8101520 msgr2=0x7f2fe81966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fe8101520 0x7f2fe81966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 msgr2=0x7f2fe819d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:31.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe819d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/3178876443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009d00 con 0x7f2fe8100330 2023-07-29T16:18:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe819d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:31.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe81961a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000e970 tx=0x7f2fd000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.992+0000 7f2fdffff700 1 -- 172.21.15.31:0/3178876443 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0009790 con 0x7f2fe8100330 2023-07-29T16:18:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.993+0000 7f2fdffff700 1 -- 172.21.15.31:0/3178876443 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd00044e0 con 0x7f2fe8100330 2023-07-29T16:18:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.993+0000 7f2fdffff700 1 -- 172.21.15.31:0/3178876443 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0003e30 con 0x7f2fe8100330 2023-07-29T16:18:31.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.993+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe819e2d0 con 0x7f2fe8100330 2023-07-29T16:18:32.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.993+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe819e790 con 0x7f2fe8100330 2023-07-29T16:18:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.996+0000 7f2fdffff700 1 -- 172.21.15.31:0/3178876443 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2fd0003f90 con 0x7f2fe8100330 2023-07-29T16:18:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 Processor -- start 2023-07-29T16:18:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 -- start start 2023-07-29T16:18:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f2fdffff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2fd40a0790 0x7f2fd40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c755096400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f2fdffff700 1 -- 172.21.15.31:0/3178876443 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2fd0014070 con 0x7f2fe8100330 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c755097b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c754fe9a10 con 0x55c755095ff0 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c754fe73d0 con 0x55c755096940 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f3000e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c754fe2cc0 con 0x55c755097700 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.998+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2fd40a0790 0x7f2fd40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c755097b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c755096400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c755096400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52984/0 (socket says 172.21.15.31:52984) 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c755097b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41792/0 (socket says 172.21.15.31:41792) 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/594967782 learned_addr learned my addr 172.21.15.31:0/594967782 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/594967782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 msgr2=0x55c755097b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c755097b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/594967782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 msgr2=0x55c755097170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/594967782 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c754e10de0 con 0x55c755095ff0 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c755096400 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0099540 tx=0x7f2fd0099480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b03fb4298baaa34 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:31.999+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2fd40a0790 0x7f2fd40a2c40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00062a0 tx=0x7f2fe00094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f2fdd7fa700 1 -- 172.21.15.31:0/594967782 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00995b0 con 0x55c755095ff0 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f2fdd7fa700 1 -- 172.21.15.31:0/594967782 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd009d5e0 con 0x55c755095ff0 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f2fdd7fa700 1 -- 172.21.15.31:0/594967782 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0099810 con 0x55c755095ff0 2023-07-29T16:18:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 -- 172.21.15.31:0/594967782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 msgr2=0x55c755096400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c755096400 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0099540 tx=0x7f2fd0099480 comp rx=0 tx=0).stop 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 -- 172.21.15.31:0/594967782 shutdown_connections 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c755097b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c755096400 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 --2- 172.21.15.31:0/594967782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.000+0000 7f3000e84b80 1 -- 172.21.15.31:0/594967782 >> 172.21.15.31:0/594967782 conn(0x55c7550873e0 msgr2=0x55c75508c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:32.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.001+0000 7f3000e84b80 1 -- 172.21.15.31:0/594967782 shutdown_connections 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.001+0000 7f3000e84b80 1 -- 172.21.15.31:0/594967782 wait complete. 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.001+0000 7f3000e84b80 1 -- start start 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.001+0000 7f3000e84b80 1 Processor -- start 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 0x55c75509fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c7550a0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755097700 0x55c7550a7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c754f7f510 con 0x55c755097700 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c754f7f1b0 con 0x55c755096940 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c754e66920 con 0x55c755095ff0 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f3000e84b80 10 client.0 _ll_register_callbacks cb 0x55c754f88620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f2fe67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c7550a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 0x55c75509fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f2fe67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c7550a0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56854/0 (socket says 172.21.15.31:56854) 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 0x55c75509fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41800/0 (socket says 172.21.15.31:41800) 2023-07-29T16:18:32.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f2fe67fc700 1 -- 172.21.15.31:0/1513860290 learned_addr learned my addr 172.21.15.31:0/1513860290 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.002+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755097700 0x55c7550a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe67fc700 1 -- 172.21.15.31:0/1513860290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 msgr2=0x55c75509fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 0x55c75509fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe67fc700 1 -- 172.21.15.31:0/1513860290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755097700 msgr2=0x55c7550a7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755097700 0x55c7550a7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe67fc700 1 -- 172.21.15.31:0/1513860290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe819e790 con 0x55c755096940 2023-07-29T16:18:32.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 0x55c75509fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755097700 0x55c7550a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c7550a0140 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8012700 tx=0x7f2fd8012a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd67f1d4f0ceb60b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fce7fc700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1513860290 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8017070 con 0x55c755096940 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1513860290 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2fd8004260 con 0x55c755096940 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c75500dd10 con 0x55c755096940 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1513860290 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fd8004420 con 0x55c755096940 2023-07-29T16:18:32.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f3000e84b80 10 client.26564 fetch_fsmap learned FSMap version 25 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c754df9c60 con 0x55c755096940 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.003+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1513860290 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c75500dd10 con 0x55c755096940 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1513860290 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 25) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2fd8011820 con 0x55c755096940 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f3000e84b80 10 client.26564 fetch_fsmap finished waiting for FSMap version 25 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f3000e84b80 10 client.26564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f3000e84b80 20 client.26564 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:32.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550ab930 0x55c7550add40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f3000e84b80 4 client.26564 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c755060d70 con 0x55c7550ab930 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550ab930 0x55c7550add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550ab930 0x55c7550add40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f983d22894bfdc6 server_cookie=7d6603cc7370877c in_seq=0 out_seq=0 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.004+0000 7f2fce7fc700 10 client.26564 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.006+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1513860290 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c755060d70 con 0x55c7550ab930 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.006+0000 7f2fce7fc700 10 client.26564 handle_command_reply: tid=0 2023-07-29T16:18:32.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 client.26564 shutdown 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 msgr2=0x55c7550a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c7550a0140 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8012700 tx=0x7f2fd8012a10 comp rx=0 tx=0).stop 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 shutdown_connections 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550ab930 0x55c7550add40 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755097700 0x55c7550a7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755095ff0 0x55c75509fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1513860290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c7550a0140 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f2fce7fc700 0 client.26564 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:32.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 >> 172.21.15.31:0/1513860290 conn(0x55c7550873e0 msgr2=0x55c75508c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:32.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 shutdown_connections 2023-07-29T16:18:32.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 1 -- 172.21.15.31:0/1513860290 wait complete. 2023-07-29T16:18:32.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.007+0000 7f3000e84b80 20 client.26564 trim_cache size 0 max 16384 2023-07-29T16:18:32.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 Processor -- start 2023-07-29T16:18:32.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 -- start start 2023-07-29T16:18:32.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c7550a5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c75509dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c755089460 con 0x55c755095ff0 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7550a0880 con 0x55c755096940 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.024+0000 7f3000e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c75509d970 con 0x55c755097700 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c7550a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c7550a5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52990/0 (socket says 172.21.15.31:52990) 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/236604570 learned_addr learned my addr 172.21.15.31:0/236604570 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56862/0 (socket says 172.21.15.31:56862) 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c75509dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 -- 172.21.15.31:0/236604570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 msgr2=0x55c75509dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c75509dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 -- 172.21.15.31:0/236604570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 msgr2=0x55c7550a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c7550a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 -- 172.21.15.31:0/236604570 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c754e37250 con 0x55c755096940 2023-07-29T16:18:32.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.025+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097ad0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80126d0 tx=0x7f2fd80096f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4db69d096926978f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:32.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f2fddffb700 1 -- 172.21.15.31:0/236604570 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8017070 con 0x55c755096940 2023-07-29T16:18:32.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f2fddffb700 1 -- 172.21.15.31:0/236604570 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd8018040 con 0x55c755096940 2023-07-29T16:18:32.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f2fddffb700 1 -- 172.21.15.31:0/236604570 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8003b90 con 0x55c755096940 2023-07-29T16:18:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 -- 172.21.15.31:0/236604570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 msgr2=0x55c755097ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097ad0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80126d0 tx=0x7f2fd80096f0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 -- 172.21.15.31:0/236604570 shutdown_connections 2023-07-29T16:18:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c75509dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755095ff0 0x55c7550a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 --2- 172.21.15.31:0/236604570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755096940 0x55c755097ad0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 -- 172.21.15.31:0/236604570 >> 172.21.15.31:0/236604570 conn(0x55c7550873e0 msgr2=0x55c75509e830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 -- 172.21.15.31:0/236604570 shutdown_connections 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.026+0000 7f3000e84b80 1 -- 172.21.15.31:0/236604570 wait complete. 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 -- start start 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 Processor -- start 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 0x55c7550f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755096940 0x55c7550f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c7550fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7550fca30 con 0x55c755096940 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7550fcb70 con 0x55c755095ff0 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7550fccb0 con 0x55c755097700 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.027+0000 7f3000e84b80 10 client.0 _ll_register_callbacks cb 0x55c754f88620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 0x55c7550f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 0x55c7550f56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56878/0 (socket says 172.21.15.31:56878) 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/1796276980 learned_addr learned my addr 172.21.15.31:0/1796276980 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:32.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755096940 0x55c7550f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe77fe700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c7550fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/1796276980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 msgr2=0x55c7550fc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c7550fc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/1796276980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755096940 msgr2=0x55c7550f5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755096940 0x55c7550f5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 -- 172.21.15.31:0/1796276980 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c754df9c60 con 0x55c755095ff0 2023-07-29T16:18:32.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fe6ffd700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 0x55c7550f56c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00a14d0 tx=0x7f2fd00c9c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1896d7c04c2ccbb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fce7fc700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1796276980 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00995b0 con 0x55c755095ff0 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1796276980 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2fd00a1a00 con 0x55c755095ff0 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.028+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fd009d5e0 con 0x55c755095ff0 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1796276980 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fd000cbd0 con 0x55c755095ff0 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 10 client.26570 fetch_fsmap learned FSMap version 25 2023-07-29T16:18:32.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c754e0f2b0 con 0x55c755095ff0 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1796276980 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd007c080 con 0x55c755095ff0 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1796276980 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 25) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2fd00ae070 con 0x55c755095ff0 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 10 client.26570 fetch_fsmap finished waiting for FSMap version 25 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 10 client.26570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 20 client.26570 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550fe400 0x55c755100810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 4 client.26570 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:18:32.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c754fdbd40 con 0x55c7550fe400 2023-07-29T16:18:32.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.029+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550fe400 0x55c755100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:32.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.030+0000 7f2fe67fc700 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550fe400 0x55c755100810 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39af97b9076644f2 server_cookie=4dc1c27fa08acdb6 in_seq=0 out_seq=0 2023-07-29T16:18:32.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.030+0000 7f2fce7fc700 10 client.26570 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:32.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.032+0000 7f2fce7fc700 1 -- 172.21.15.31:0/1796276980 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c754fdbd40 con 0x55c7550fe400 2023-07-29T16:18:32.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.032+0000 7f2fce7fc700 10 client.26570 handle_command_reply: tid=0 2023-07-29T16:18:32.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.032+0000 7f3000e84b80 1 client.26570 shutdown 2023-07-29T16:18:32.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 msgr2=0x55c7550f56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:32.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 0x55c7550f56c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00a14d0 tx=0x7f2fd00c9c70 comp rx=0 tx=0).stop 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 shutdown_connections 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c7550fe400 0x55c755100810 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c755097700 0x55c7550fc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c755095ff0 0x55c7550f56c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f2fce7fc700 0 client.26570 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 --2- 172.21.15.31:0/1796276980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c755096940 0x55c7550f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 >> 172.21.15.31:0/1796276980 conn(0x55c7550873e0 msgr2=0x55c75509e830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:32.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 shutdown_connections 2023-07-29T16:18:32.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 1 -- 172.21.15.31:0/1796276980 wait complete. 2023-07-29T16:18:32.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f3000e84b80 20 client.26570 trim_cache size 0 max 16384 2023-07-29T16:18:32.038 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:18:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:18:32.038 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "47d15a20-2e4b-4fed-bf84-243809e4ab15": { 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "47d15a20-2e4b-4fed-bf84-243809e4ab15", 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:18:32.039 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:18:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2fd40a0790 msgr2=0x7f2fd40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2fd40a0790 0x7f2fd40a2c40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00062a0 tx=0x7f2fe00094f0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 msgr2=0x7f2fe81961a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.033+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe81961a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000e970 tx=0x7f2fd000ed30 comp rx=0 tx=0).stop 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 shutdown_connections 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2fd40a0790 0x7f2fd40a2c40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe8101eb0 0x7f2fe819d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fe8100330 0x7f2fe81961a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 --2- 172.21.15.31:0/3178876443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2fe8101520 0x7f2fe81966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 >> 172.21.15.31:0/3178876443 conn(0x7f2fe80fb970 msgr2=0x7f2fe80fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:32.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 shutdown_connections 2023-07-29T16:18:32.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:18:32.034+0000 7f2fed5ff700 1 -- 172.21.15.31:0/3178876443 wait complete. 2023-07-29T16:18:32.050 INFO:tasks.cephfs.filesystem:scrub status for tag:47d15a20-2e4b-4fed-bf84-243809e4ab15 - {'path': '/', 'tag': '47d15a20-2e4b-4fed-bf84-243809e4ab15', 'options': 'recursive,force'} 2023-07-29T16:18:32.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:32 smithi031 ceph-mon[95504]: pgmap v462: 97 pgs: 97 active+clean; 13 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 1.3 MiB/s wr, 175 op/s 2023-07-29T16:18:32.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:32 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6836/80628213,v1:172.21.15.156:6837/80628213] up:boot 2023-07-29T16:18:32.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:32 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2023-07-29T16:18:32.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:32 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:18:32.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:32 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:18:32.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:32 smithi039 ceph-mon[128858]: pgmap v462: 97 pgs: 97 active+clean; 13 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 1.3 MiB/s wr, 175 op/s 2023-07-29T16:18:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:32 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6836/80628213,v1:172.21.15.156:6837/80628213] up:boot 2023-07-29T16:18:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:32 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2023-07-29T16:18:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:32 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:18:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:32 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:18:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:32 smithi156 ceph-mon[129245]: pgmap v462: 97 pgs: 97 active+clean; 13 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 1.3 MiB/s wr, 175 op/s 2023-07-29T16:18:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:32 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6836/80628213,v1:172.21.15.156:6837/80628213] up:boot 2023-07-29T16:18:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:32 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2023-07-29T16:18:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:32 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:18:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:32 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:18:32.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 35899 12.48 MB/sec warmup 17 sec latency 109.738 ms 2023-07-29T16:18:33.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:33 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:18:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:33 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:33 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:18:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:33 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:33 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:18:33.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:33 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:18:33.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:33 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:33.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:33 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:18:33.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:33 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:33.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:33 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:18:33.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 38098 12.24 MB/sec warmup 18 sec latency 15.483 ms 2023-07-29T16:18:33.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:33 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:18:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:33 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:33 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:18:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:33 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:18:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:33 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:18:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:34 smithi039 ceph-mon[128858]: pgmap v463: 97 pgs: 97 active+clean; 8.0 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.2 MiB/s wr, 220 op/s 2023-07-29T16:18:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:34 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:18:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:34 smithi156 ceph-mon[129245]: pgmap v463: 97 pgs: 97 active+clean; 8.0 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.2 MiB/s wr, 220 op/s 2023-07-29T16:18:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:34 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:18:34.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39296 12.16 MB/sec warmup 19 sec latency 352.242 ms 2023-07-29T16:18:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:34 smithi031 ceph-mon[95504]: pgmap v463: 97 pgs: 97 active+clean; 8.0 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.2 MiB/s wr, 220 op/s 2023-07-29T16:18:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:34 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:18:35.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39296 11.55 MB/sec warmup 20 sec latency 1352.365 ms 2023-07-29T16:18:36.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:36 smithi039 ceph-mon[128858]: pgmap v464: 97 pgs: 97 active+clean; 8.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 1.1 MiB/s wr, 287 op/s 2023-07-29T16:18:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:36 smithi156 ceph-mon[129245]: pgmap v464: 97 pgs: 97 active+clean; 8.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 1.1 MiB/s wr, 287 op/s 2023-07-29T16:18:36.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39296 11.00 MB/sec warmup 21 sec latency 2352.517 ms 2023-07-29T16:18:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:36 smithi031 ceph-mon[95504]: pgmap v464: 97 pgs: 97 active+clean; 8.9 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 1.1 MiB/s wr, 287 op/s 2023-07-29T16:18:37.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39379 10.50 MB/sec warmup 22 sec latency 3100.760 ms 2023-07-29T16:18:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:37 smithi031 ceph-mon[95504]: pgmap v465: 97 pgs: 97 active+clean; 8.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.1 MiB/s wr, 289 op/s 2023-07-29T16:18:38.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:37 smithi039 ceph-mon[128858]: pgmap v465: 97 pgs: 97 active+clean; 8.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.1 MiB/s wr, 289 op/s 2023-07-29T16:18:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:37 smithi156 ceph-mon[129245]: pgmap v465: 97 pgs: 97 active+clean; 8.9 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 1.1 MiB/s wr, 289 op/s 2023-07-29T16:18:38.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39379 10.04 MB/sec warmup 23 sec latency 1109.758 ms 2023-07-29T16:18:39.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 39777 9.65 MB/sec warmup 24 sec latency 1715.321 ms 2023-07-29T16:18:40.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:40 smithi031 ceph-mon[95504]: pgmap v466: 97 pgs: 97 active+clean; 10 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 779 KiB/s wr, 243 op/s 2023-07-29T16:18:40.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:18:40.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:40 smithi039 ceph-mon[128858]: pgmap v466: 97 pgs: 97 active+clean; 10 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 779 KiB/s wr, 243 op/s 2023-07-29T16:18:40.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:18:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:40 smithi156 ceph-mon[129245]: pgmap v466: 97 pgs: 97 active+clean; 10 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 779 KiB/s wr, 243 op/s 2023-07-29T16:18:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:18:40.856 INFO:tasks.workunit.client.0.smithi031.stdout: 1 42029 9.75 MB/sec warmup 25 sec latency 17.961 ms 2023-07-29T16:18:41.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:41 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:18:41.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:41 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:18:41.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:41 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:18:41.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:41 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:18:41.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:41 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:18:41.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:41 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:18:41.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:41 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:18:41.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:41 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:18:41.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:41 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:18:41.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 43388 9.65 MB/sec warmup 26 sec latency 12.251 ms 2023-07-29T16:18:42.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:42 smithi031 ceph-mon[95504]: pgmap v467: 97 pgs: 97 active+clean; 10 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 731 KiB/s wr, 227 op/s 2023-07-29T16:18:42.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:42 smithi039 ceph-mon[128858]: pgmap v467: 97 pgs: 97 active+clean; 10 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 731 KiB/s wr, 227 op/s 2023-07-29T16:18:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:42 smithi156 ceph-mon[129245]: pgmap v467: 97 pgs: 97 active+clean; 10 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 731 KiB/s wr, 227 op/s 2023-07-29T16:18:42.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 45937 9.81 MB/sec warmup 27 sec latency 12.962 ms 2023-07-29T16:18:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 -- 172.21.15.31:0/71943351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90100340 msgr2=0x7f0e90100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 --2- 172.21.15.31:0/71943351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90100340 0x7f0e90100750 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009990 tx=0x7f0e8c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 -- 172.21.15.31:0/71943351 shutdown_connections 2023-07-29T16:18:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 --2- 172.21.15.31:0/71943351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e90101ec0 0x7f0e90108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 --2- 172.21.15.31:0/71943351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90100340 0x7f0e90100750 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 --2- 172.21.15.31:0/71943351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90101530 0x7f0e90101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 -- 172.21.15.31:0/71943351 >> 172.21.15.31:0/71943351 conn(0x7f0e900fb920 msgr2=0x7f0e900fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 -- 172.21.15.31:0/71943351 shutdown_connections 2023-07-29T16:18:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.439+0000 7f0e97bff700 1 -- 172.21.15.31:0/71943351 wait complete. 2023-07-29T16:18:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.440+0000 7f0e97bff700 1 Processor -- start 2023-07-29T16:18:43.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.440+0000 7f0e97bff700 1 -- start start 2023-07-29T16:18:43.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e97bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 0x7f0e90073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e97bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e90101530 0x7f0e90073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e97bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90101ec0 0x7f0e90073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e97bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e901a0150 con 0x7f0e90101ec0 2023-07-29T16:18:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e97bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e901a0290 con 0x7f0e90100340 2023-07-29T16:18:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e97bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e901a03d0 con 0x7f0e90101530 2023-07-29T16:18:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 0x7f0e90073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 0x7f0e90073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59942/0 (socket says 172.21.15.31:59942) 2023-07-29T16:18:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 -- 172.21.15.31:0/288017838 learned_addr learned my addr 172.21.15.31:0/288017838 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9519a700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e90101530 0x7f0e90073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9619c700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90101ec0 0x7f0e90073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 -- 172.21.15.31:0/288017838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e90101530 msgr2=0x7f0e90073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e90101530 0x7f0e90073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 -- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90101ec0 msgr2=0x7f0e90073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90101ec0 0x7f0e90073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.441+0000 7f0e9599b700 1 -- 172.21.15.31:0/288017838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c0095f0 con 0x7f0e90100340 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.442+0000 7f0e9599b700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 0x7f0e90073140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c005740 tx=0x7f0e8c005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:43.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.442+0000 7f0e86ffd700 1 -- 172.21.15.31:0/288017838 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c01d070 con 0x7f0e90100340 2023-07-29T16:18:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.442+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e901a0600 con 0x7f0e90100340 2023-07-29T16:18:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.442+0000 7f0e86ffd700 1 -- 172.21.15.31:0/288017838 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e8c021410 con 0x7f0e90100340 2023-07-29T16:18:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.442+0000 7f0e86ffd700 1 -- 172.21.15.31:0/288017838 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c017650 con 0x7f0e90100340 2023-07-29T16:18:43.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.442+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e901a0a80 con 0x7f0e90100340 2023-07-29T16:18:43.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.445+0000 7f0e86ffd700 1 -- 172.21.15.31:0/288017838 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0e8c021580 con 0x7f0e90100340 2023-07-29T16:18:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0e86ffd700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0e7c0a0790 0x7f0e7c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 Processor -- start 2023-07-29T16:18:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0e86ffd700 1 -- 172.21.15.31:0/288017838 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0e8c0ccc10 con 0x7f0e90100340 2023-07-29T16:18:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 -- start start 2023-07-29T16:18:43.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0e9519a700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0e7c0a0790 0x7f0e7c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d5c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d5d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d5e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44cb49b0 con 0x563b44d5cdf0 2023-07-29T16:18:43.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44cb2370 con 0x563b44d5dcf0 2023-07-29T16:18:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0eab591b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44cadc60 con 0x563b44bb86b0 2023-07-29T16:18:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.447+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d5c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d5c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:43696/0 (socket says 172.21.15.31:43696) 2023-07-29T16:18:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 -- 172.21.15.31:0/941543652 learned_addr learned my addr 172.21.15.31:0/941543652 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9519a700 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d5d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9619c700 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d5e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 -- 172.21.15.31:0/941543652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 msgr2=0x563b44d5e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9519a700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0e7c0a0790 0x7f0e7c0a2c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0e90074440 tx=0x7f0e80011720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d5e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 -- 172.21.15.31:0/941543652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 msgr2=0x563b44d5d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d5d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 -- 172.21.15.31:0/941543652 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b44adbd80 con 0x563b44bb86b0 2023-07-29T16:18:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9619c700 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d5e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e9599b700 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d5c8b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0a21d0 tx=0x7f0e8c09a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a3277ce05e6c0b9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e7bfff700 1 -- 172.21.15.31:0/941543652 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c09e170 con 0x563b44bb86b0 2023-07-29T16:18:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e7bfff700 1 -- 172.21.15.31:0/941543652 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e8c09a340 con 0x563b44bb86b0 2023-07-29T16:18:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.448+0000 7f0e7bfff700 1 -- 172.21.15.31:0/941543652 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c09a5a0 con 0x563b44bb86b0 2023-07-29T16:18:43.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 -- 172.21.15.31:0/941543652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 msgr2=0x563b44d5c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d5c8b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c0a21d0 tx=0x7f0e8c09a1e0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 -- 172.21.15.31:0/941543652 shutdown_connections 2023-07-29T16:18:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d5e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d5c8b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 --2- 172.21.15.31:0/941543652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d5d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 -- 172.21.15.31:0/941543652 >> 172.21.15.31:0/941543652 conn(0x563b44d51fd0 msgr2=0x563b44d57300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.449+0000 7f0eab591b80 1 -- 172.21.15.31:0/941543652 shutdown_connections 2023-07-29T16:18:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.450+0000 7f0eab591b80 1 -- 172.21.15.31:0/941543652 wait complete. 2023-07-29T16:18:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.450+0000 7f0eab591b80 1 -- start start 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.450+0000 7f0eab591b80 1 Processor -- start 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d65b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5cdf0 0x563b44d66070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5dcf0 0x563b44d6d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44c4a4b0 con 0x563b44d5dcf0 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44c4a150 con 0x563b44bb86b0 2023-07-29T16:18:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44b318c0 con 0x563b44d5cdf0 2023-07-29T16:18:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0eab591b80 10 client.0 _ll_register_callbacks cb 0x563b44c535c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d65b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d65b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59968/0 (socket says 172.21.15.31:59968) 2023-07-29T16:18:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 -- 172.21.15.31:0/2140002490 learned_addr learned my addr 172.21.15.31:0/2140002490 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9519a700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5cdf0 0x563b44d66070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9619c700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5dcf0 0x563b44d6d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 -- 172.21.15.31:0/2140002490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5cdf0 msgr2=0x563b44d66070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5cdf0 0x563b44d66070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 -- 172.21.15.31:0/2140002490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5dcf0 msgr2=0x563b44d6d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5dcf0 0x563b44d6d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.451+0000 7f0e9599b700 1 -- 172.21.15.31:0/2140002490 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e901a0a80 con 0x563b44bb86b0 2023-07-29T16:18:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e9599b700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d65b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009350 tx=0x7f0e8c095ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2598e1fa19d4ea21 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e797fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e797fa700 1 -- 172.21.15.31:0/2140002490 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c09a7a0 con 0x563b44bb86b0 2023-07-29T16:18:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e797fa700 1 -- 172.21.15.31:0/2140002490 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0e8c09e370 con 0x563b44bb86b0 2023-07-29T16:18:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b44cd8cb0 con 0x563b44bb86b0 2023-07-29T16:18:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e797fa700 1 -- 172.21.15.31:0/2140002490 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e8c09e170 con 0x563b44bb86b0 2023-07-29T16:18:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0eab591b80 10 client.26582 fetch_fsmap learned FSMap version 26 2023-07-29T16:18:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x563b44ac4c00 con 0x563b44bb86b0 2023-07-29T16:18:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e797fa700 1 -- 172.21.15.31:0/2140002490 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563b44cd8cb0 con 0x563b44bb86b0 2023-07-29T16:18:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.452+0000 7f0e797fa700 1 -- 172.21.15.31:0/2140002490 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f0e8c0cab10 con 0x563b44bb86b0 2023-07-29T16:18:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0eab591b80 10 client.26582 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:18:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0eab591b80 10 client.26582 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0eab591b80 20 client.26582 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44d71400 0x563b44d73810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0eab591b80 4 client.26582 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b44d2bd10 con 0x563b44d71400 2023-07-29T16:18:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0e9519a700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44d71400 0x563b44d73810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0e9519a700 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44d71400 0x563b44d73810 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abcaf5da847d631 server_cookie=6d4047a7f94166df in_seq=0 out_seq=0 2023-07-29T16:18:43.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.453+0000 7f0e797fa700 10 client.26582 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.454+0000 7f0e797fa700 1 -- 172.21.15.31:0/2140002490 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563b44d2bd10 con 0x563b44d71400 2023-07-29T16:18:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.454+0000 7f0e797fa700 10 client.26582 handle_command_reply: tid=0 2023-07-29T16:18:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.454+0000 7f0eab591b80 1 client.26582 shutdown 2023-07-29T16:18:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.454+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 msgr2=0x563b44d65b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d65b30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009350 tx=0x7f0e8c095ff0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 shutdown_connections 2023-07-29T16:18:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44d71400 0x563b44d73810 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5dcf0 0x563b44d6d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d65b30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2140002490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5cdf0 0x563b44d66070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0e797fa700 0 client.26582 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 >> 172.21.15.31:0/2140002490 conn(0x563b44d51fd0 msgr2=0x563b44d56a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 shutdown_connections 2023-07-29T16:18:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 1 -- 172.21.15.31:0/2140002490 wait complete. 2023-07-29T16:18:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.455+0000 7f0eab591b80 20 client.26582 trim_cache size 0 max 16384 2023-07-29T16:18:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.470+0000 7f0eab591b80 1 Processor -- start 2023-07-29T16:18:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.470+0000 7f0eab591b80 1 -- start start 2023-07-29T16:18:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d6d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 0x563b44d65e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0eab591b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44d585f0 con 0x563b44d5cdf0 2023-07-29T16:18:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0eab591b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44d519d0 con 0x563b44bb86b0 2023-07-29T16:18:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0eab591b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44d5d5d0 con 0x563b44d5dcf0 2023-07-29T16:18:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 0x563b44d65e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9519a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d6d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 0x563b44d65e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:43722/0 (socket says 172.21.15.31:43722) 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 -- 172.21.15.31:0/3480090765 learned_addr learned my addr 172.21.15.31:0/3480090765 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9599b700 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 -- 172.21.15.31:0/3480090765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 msgr2=0x563b44d658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d658e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 -- 172.21.15.31:0/3480090765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 msgr2=0x563b44d6d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d6d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9619c700 1 -- 172.21.15.31:0/3480090765 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b44b021f0 con 0x563b44d5dcf0 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.471+0000 7f0e9519a700 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d6d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0e9619c700 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 0x563b44d65e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0e88007630 tx=0x7f0e8800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1108fc5285f275a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0e84bff700 1 -- 172.21.15.31:0/3480090765 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8800e040 con 0x563b44d5dcf0 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0e84bff700 1 -- 172.21.15.31:0/3480090765 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e8800f040 con 0x563b44d5dcf0 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0e84bff700 1 -- 172.21.15.31:0/3480090765 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e88017670 con 0x563b44d5dcf0 2023-07-29T16:18:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0eab591b80 1 -- 172.21.15.31:0/3480090765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 msgr2=0x563b44d65e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0eab591b80 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 0x563b44d65e20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0e88007630 tx=0x7f0e8800b040 comp rx=0 tx=0).stop 2023-07-29T16:18:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0eab591b80 1 -- 172.21.15.31:0/3480090765 shutdown_connections 2023-07-29T16:18:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0eab591b80 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44d5dcf0 0x563b44d65e20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0eab591b80 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44bb86b0 0x563b44d658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.472+0000 7f0eab591b80 1 --2- 172.21.15.31:0/3480090765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d6d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.473+0000 7f0eab591b80 1 -- 172.21.15.31:0/3480090765 >> 172.21.15.31:0/3480090765 conn(0x563b44d51fd0 msgr2=0x563b44d60a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.473+0000 7f0eab591b80 1 -- 172.21.15.31:0/3480090765 shutdown_connections 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.473+0000 7f0eab591b80 1 -- 172.21.15.31:0/3480090765 wait complete. 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.473+0000 7f0eab591b80 1 -- start start 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.473+0000 7f0eab591b80 1 Processor -- start 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d6cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d69820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d69d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44d6d420 con 0x563b44d5cdf0 2023-07-29T16:18:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44d6a2a0 con 0x563b44d5dcf0 2023-07-29T16:18:43.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b44d6a460 con 0x563b44bb86b0 2023-07-29T16:18:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0eab591b80 10 client.0 _ll_register_callbacks cb 0x563b44c535c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d6cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d6cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:43738/0 (socket says 172.21.15.31:43738) 2023-07-29T16:18:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 -- 172.21.15.31:0/2766768126 learned_addr learned my addr 172.21.15.31:0/2766768126 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9619c700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d69d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9519a700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d69820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 -- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 msgr2=0x563b44d69d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d69d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 -- 172.21.15.31:0/2766768126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 msgr2=0x563b44d69820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:43.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d69820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.474+0000 7f0e9599b700 1 -- 172.21.15.31:0/2766768126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e800287c0 con 0x563b44bb86b0 2023-07-29T16:18:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e9599b700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d6cc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c00b620 tx=0x7f0e8c0988a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1df05ea227e8e2d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:43.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e797fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e797fa700 1 -- 172.21.15.31:0/2766768126 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c0a2470 con 0x563b44bb86b0 2023-07-29T16:18:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e797fa700 1 -- 172.21.15.31:0/2766768126 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0e8c0a2700 con 0x563b44bb86b0 2023-07-29T16:18:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e8c09a340 con 0x563b44bb86b0 2023-07-29T16:18:43.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e797fa700 1 -- 172.21.15.31:0/2766768126 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e8c0ca450 con 0x563b44bb86b0 2023-07-29T16:18:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 10 client.26605 fetch_fsmap learned FSMap version 26 2023-07-29T16:18:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f0e880092c0 con 0x563b44bb86b0 2023-07-29T16:18:43.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e797fa700 1 -- 172.21.15.31:0/2766768126 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e8c09a340 con 0x563b44bb86b0 2023-07-29T16:18:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0e797fa700 1 -- 172.21.15.31:0/2766768126 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f0e8c09a5a0 con 0x563b44bb86b0 2023-07-29T16:18:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 10 client.26605 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:18:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 10 client.26605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:43.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 20 client.26605 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44dc4420 0x563b44dc6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 4 client.26605 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:18:43.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.475+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563b44ca6ce0 con 0x563b44dc4420 2023-07-29T16:18:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.476+0000 7f0e9519a700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44dc4420 0x563b44dc6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.476+0000 7f0e9519a700 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44dc4420 0x563b44dc6830 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51953986f197a77 server_cookie=911c8b4b8d8aca1 in_seq=0 out_seq=0 2023-07-29T16:18:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.476+0000 7f0e797fa700 10 client.26605 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.546+0000 7f0e797fa700 1 -- 172.21.15.31:0/2766768126 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563b44ca6ce0 con 0x563b44dc4420 2023-07-29T16:18:43.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.546+0000 7f0e797fa700 10 client.26605 handle_command_reply: tid=0 2023-07-29T16:18:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.546+0000 7f0eab591b80 1 client.26605 shutdown 2023-07-29T16:18:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 msgr2=0x563b44d6cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d6cc00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c00b620 tx=0x7f0e8c0988a0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 shutdown_connections 2023-07-29T16:18:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563b44dc4420 0x563b44dc6830 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563b44d5dcf0 0x563b44d69d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563b44bb86b0 0x563b44d6cc00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0e797fa700 0 client.26605 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 --2- 172.21.15.31:0/2766768126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563b44d5cdf0 0x563b44d69820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 >> 172.21.15.31:0/2766768126 conn(0x563b44d51fd0 msgr2=0x563b44d658e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 shutdown_connections 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 1 -- 172.21.15.31:0/2766768126 wait complete. 2023-07-29T16:18:43.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.547+0000 7f0eab591b80 20 client.26605 trim_cache size 0 max 16384 2023-07-29T16:18:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:18:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:18:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:18:43.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:18:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0e7c0a0790 msgr2=0x7f0e7c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0e7c0a0790 0x7f0e7c0a2c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0e90074440 tx=0x7f0e80011720 comp rx=0 tx=0).stop 2023-07-29T16:18:43.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 msgr2=0x7f0e90073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 0x7f0e90073140 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c005740 tx=0x7f0e8c005770 comp rx=0 tx=0).stop 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 shutdown_connections 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0e7c0a0790 0x7f0e7c0a2c40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e90101ec0 0x7f0e90073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e90100340 0x7f0e90073140 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 --2- 172.21.15.31:0/288017838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0e90101530 0x7f0e90073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:43.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 >> 172.21.15.31:0/288017838 conn(0x7f0e900fb920 msgr2=0x7f0e900fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 shutdown_connections 2023-07-29T16:18:43.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:43.548+0000 7f0e97bff700 1 -- 172.21.15.31:0/288017838 wait complete. 2023-07-29T16:18:43.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 46928 9.64 MB/sec warmup 28 sec latency 56.406 ms 2023-07-29T16:18:44.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:44 smithi031 ceph-mon[95504]: pgmap v468: 97 pgs: 97 active+clean; 13 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 905 KiB/s wr, 254 op/s 2023-07-29T16:18:44.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:44 smithi039 ceph-mon[128858]: pgmap v468: 97 pgs: 97 active+clean; 13 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 905 KiB/s wr, 254 op/s 2023-07-29T16:18:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:44 smithi156 ceph-mon[129245]: pgmap v468: 97 pgs: 97 active+clean; 13 MiB data, 400 MiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 905 KiB/s wr, 254 op/s 2023-07-29T16:18:44.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 49406 9.79 MB/sec warmup 29 sec latency 20.494 ms 2023-07-29T16:18:45.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 50796 9.69 MB/sec warmup 30 sec latency 12.062 ms 2023-07-29T16:18:46.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:46 smithi031 ceph-mon[95504]: pgmap v469: 97 pgs: 97 active+clean; 13 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 615 KiB/s wr, 166 op/s 2023-07-29T16:18:46.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:46 smithi039 ceph-mon[128858]: pgmap v469: 97 pgs: 97 active+clean; 13 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 615 KiB/s wr, 166 op/s 2023-07-29T16:18:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:46 smithi156 ceph-mon[129245]: pgmap v469: 97 pgs: 97 active+clean; 13 MiB data, 403 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 615 KiB/s wr, 166 op/s 2023-07-29T16:18:46.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 52699 9.72 MB/sec warmup 31 sec latency 183.608 ms 2023-07-29T16:18:47.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 53768 9.62 MB/sec warmup 32 sec latency 278.535 ms 2023-07-29T16:18:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:47 smithi031 ceph-mon[95504]: pgmap v470: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 662 KiB/s wr, 115 op/s 2023-07-29T16:18:48.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:47 smithi039 ceph-mon[128858]: pgmap v470: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 662 KiB/s wr, 115 op/s 2023-07-29T16:18:48.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:47 smithi156 ceph-mon[129245]: pgmap v470: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 662 KiB/s wr, 115 op/s 2023-07-29T16:18:48.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 56004 9.66 MB/sec warmup 33 sec latency 16.443 ms 2023-07-29T16:18:49.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 57354 9.62 MB/sec warmup 34 sec latency 13.260 ms 2023-07-29T16:18:50.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:50 smithi031 ceph-mon[95504]: pgmap v471: 97 pgs: 97 active+clean; 34 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 3.7 MiB/s wr, 250 op/s 2023-07-29T16:18:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:50 smithi039 ceph-mon[128858]: pgmap v471: 97 pgs: 97 active+clean; 34 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 3.7 MiB/s wr, 250 op/s 2023-07-29T16:18:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:50 smithi156 ceph-mon[129245]: pgmap v471: 97 pgs: 97 active+clean; 34 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 3.7 MiB/s wr, 250 op/s 2023-07-29T16:18:50.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 59672 9.68 MB/sec warmup 35 sec latency 16.706 ms 2023-07-29T16:18:51.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 60930 9.61 MB/sec warmup 36 sec latency 15.138 ms 2023-07-29T16:18:52.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:52 smithi031 ceph-mon[95504]: pgmap v472: 97 pgs: 97 active+clean; 34 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.6 MiB/s wr, 224 op/s 2023-07-29T16:18:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:52 smithi039 ceph-mon[128858]: pgmap v472: 97 pgs: 97 active+clean; 34 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.6 MiB/s wr, 224 op/s 2023-07-29T16:18:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:52 smithi156 ceph-mon[129245]: pgmap v472: 97 pgs: 97 active+clean; 34 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 3.6 MiB/s wr, 224 op/s 2023-07-29T16:18:52.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 62190 9.50 MB/sec warmup 37 sec latency 174.360 ms 2023-07-29T16:18:53.857 INFO:tasks.workunit.client.0.smithi031.stdout: 1 64158 9.60 MB/sec warmup 38 sec latency 45.024 ms 2023-07-29T16:18:54.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:54 smithi031 ceph-mon[95504]: pgmap v473: 97 pgs: 97 active+clean; 43 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 284 op/s 2023-07-29T16:18:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:54 smithi039 ceph-mon[128858]: pgmap v473: 97 pgs: 97 active+clean; 43 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 284 op/s 2023-07-29T16:18:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:54 smithi156 ceph-mon[129245]: pgmap v473: 97 pgs: 97 active+clean; 43 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 284 op/s 2023-07-29T16:18:54.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 65562 9.50 MB/sec warmup 39 sec latency 30.855 ms 2023-07-29T16:18:55.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 67729 9.60 MB/sec warmup 40 sec latency 13.969 ms 2023-07-29T16:18:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.703+0000 7fb1383ff700 1 -- 172.21.15.31:0/2663632670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 msgr2=0x7fb130105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.703+0000 7fb1383ff700 1 --2- 172.21.15.31:0/2663632670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 0x7fb130105b60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb120009990 tx=0x7fb120009ca0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 -- 172.21.15.31:0/2663632670 shutdown_connections 2023-07-29T16:18:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 --2- 172.21.15.31:0/2663632670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1301060a0 0x7fb130108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 --2- 172.21.15.31:0/2663632670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb130100e60 0x7fb130103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 --2- 172.21.15.31:0/2663632670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 0x7fb130105b60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 -- 172.21.15.31:0/2663632670 >> 172.21.15.31:0/2663632670 conn(0x7fb1300fa840 msgr2=0x7fb1300fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:56.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 -- 172.21.15.31:0/2663632670 shutdown_connections 2023-07-29T16:18:56.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.704+0000 7fb1383ff700 1 -- 172.21.15.31:0/2663632670 wait complete. 2023-07-29T16:18:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.705+0000 7fb1383ff700 1 Processor -- start 2023-07-29T16:18:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.705+0000 7fb1383ff700 1 -- start start 2023-07-29T16:18:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.705+0000 7fb1383ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb130100e60 0x7fb1301961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb1383ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 0x7fb130196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb1383ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 0x7fb130197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb1383ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130197be0 con 0x7fb130100e60 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb1383ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130197d50 con 0x7fb130103780 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb1383ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130197f10 con 0x7fb1301060a0 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 0x7fb130197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13619b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb130100e60 0x7fb1301961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13599a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 0x7fb130196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 0x7fb130197580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58736/0 (socket says 172.21.15.31:58736) 2023-07-29T16:18:56.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 -- 172.21.15.31:0/576628127 learned_addr learned my addr 172.21.15.31:0/576628127 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 -- 172.21.15.31:0/576628127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 msgr2=0x7fb130196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 0x7fb130196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 -- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb130100e60 msgr2=0x7fb1301961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb130100e60 0x7fb1301961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13699c700 1 -- 172.21.15.31:0/576628127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200095f0 con 0x7fb1301060a0 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.706+0000 7fb13619b700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb130100e60 0x7fb1301961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:56.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.707+0000 7fb13699c700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 0x7fb130197580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb12800efc0 tx=0x7fb12800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.707+0000 7fb1277fe700 1 -- 172.21.15.31:0/576628127 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128018070 con 0x7fb1301060a0 2023-07-29T16:18:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.707+0000 7fb1277fe700 1 -- 172.21.15.31:0/576628127 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb12800f040 con 0x7fb1301060a0 2023-07-29T16:18:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.707+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb130198190 con 0x7fb1301060a0 2023-07-29T16:18:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.707+0000 7fb1277fe700 1 -- 172.21.15.31:0/576628127 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb128007bd0 con 0x7fb1301060a0 2023-07-29T16:18:56.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.707+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13019e9b0 con 0x7fb1301060a0 2023-07-29T16:18:56.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.711+0000 7fb1277fe700 1 -- 172.21.15.31:0/576628127 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb128007d30 con 0x7fb1301060a0 2023-07-29T16:18:56.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.712+0000 7fb14bdf1b80 1 Processor -- start 2023-07-29T16:18:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.712+0000 7fb14bdf1b80 1 -- start start 2023-07-29T16:18:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb1277fe700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb11c0a0790 0x7fb11c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc74d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ddf0 0x556ebc74e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb1277fe700 1 -- 172.21.15.31:0/576628127 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb1280cba50 con 0x7fb1301060a0 2023-07-29T16:18:56.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc74f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc6a59b0 con 0x556ebc74ddf0 2023-07-29T16:18:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc6a3370 con 0x556ebc74ecf0 2023-07-29T16:18:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc69ec60 con 0x556ebc5a96b0 2023-07-29T16:18:56.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13599a700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb11c0a0790 0x7fb11c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc74f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13619b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc74d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc74f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36116/0 (socket says 172.21.15.31:36116) 2023-07-29T16:18:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 -- 172.21.15.31:0/1465397665 learned_addr learned my addr 172.21.15.31:0/1465397665 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13599a700 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ddf0 0x556ebc74e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 -- 172.21.15.31:0/1465397665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 msgr2=0x556ebc74d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc74d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 -- 172.21.15.31:0/1465397665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ddf0 msgr2=0x556ebc74e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.713+0000 7fb13699c700 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ddf0 0x556ebc74e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.714+0000 7fb13699c700 1 -- 172.21.15.31:0/1465397665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ebc4ccd80 con 0x556ebc74ecf0 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.714+0000 7fb13699c700 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc74f160 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb1280090a0 tx=0x7fb1280990b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de0d9be326bccde server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.714+0000 7fb124bfe700 1 -- 172.21.15.31:0/1465397665 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1280991e0 con 0x556ebc74ecf0 2023-07-29T16:18:56.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.714+0000 7fb13599a700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb11c0a0790 0x7fb11c0a2c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb12000b2e0 tx=0x7fb12000b850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.714+0000 7fb124bfe700 1 -- 172.21.15.31:0/1465397665 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb12809d210 con 0x556ebc74ecf0 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.714+0000 7fb124bfe700 1 -- 172.21.15.31:0/1465397665 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1280973b0 con 0x556ebc74ecf0 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/1465397665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 msgr2=0x556ebc74f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc74f160 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb1280090a0 tx=0x7fb1280990b0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/1465397665 shutdown_connections 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc74f160 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc74d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/1465397665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ddf0 0x556ebc74e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/1465397665 >> 172.21.15.31:0/1465397665 conn(0x556ebc742fd0 msgr2=0x556ebc748300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/1465397665 shutdown_connections 2023-07-29T16:18:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/1465397665 wait complete. 2023-07-29T16:18:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.715+0000 7fb14bdf1b80 1 -- start start 2023-07-29T16:18:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 Processor -- start 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc756ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc757420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc75e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc63b4b0 con 0x556ebc5a96b0 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc63b150 con 0x556ebc74ecf0 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc5228c0 con 0x556ebc74ddf0 2023-07-29T16:18:56.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.716+0000 7fb14bdf1b80 10 client.0 _ll_register_callbacks cb 0x556ebc6445c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc757420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc757420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58744/0 (socket says 172.21.15.31:58744) 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 -- 172.21.15.31:0/537131080 learned_addr learned my addr 172.21.15.31:0/537131080 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13699c700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc75e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13619b700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc756ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 -- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 msgr2=0x556ebc75e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc75e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 -- 172.21.15.31:0/537131080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 msgr2=0x556ebc756ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc756ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 -- 172.21.15.31:0/537131080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13019e9b0 con 0x556ebc74ddf0 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb13599a700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc757420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb120015000 tx=0x7fb120005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7491de6a56fc8a1a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.717+0000 7fb119ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb119ffb700 1 -- 172.21.15.31:0/537131080 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12002d070 con 0x556ebc74ddf0 2023-07-29T16:18:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb119ffb700 1 -- 172.21.15.31:0/537131080 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb120004b20 con 0x556ebc74ddf0 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ebc6c9cb0 con 0x556ebc74ddf0 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb119ffb700 1 -- 172.21.15.31:0/537131080 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb120004ce0 con 0x556ebc74ddf0 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb14bdf1b80 10 client.26617 fetch_fsmap learned FSMap version 26 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556ebc4b5c00 con 0x556ebc74ddf0 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.718+0000 7fb119ffb700 1 -- 172.21.15.31:0/537131080 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556ebc6c9cb0 con 0x556ebc74ddf0 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb119ffb700 1 -- 172.21.15.31:0/537131080 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fb120031410 con 0x556ebc74ddf0 2023-07-29T16:18:56.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb14bdf1b80 10 client.26617 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb14bdf1b80 10 client.26617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb14bdf1b80 20 client.26617 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc762cd0 0x556ebc7650e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb14bdf1b80 4 client.26617 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ebc71cd10 con 0x556ebc762cd0 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb13619b700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc762cd0 0x556ebc7650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb13619b700 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc762cd0 0x556ebc7650e0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c260e3e3c60965 server_cookie=c32a179a74423ad7 in_seq=0 out_seq=0 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.719+0000 7fb119ffb700 10 client.26617 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.720+0000 7fb119ffb700 1 -- 172.21.15.31:0/537131080 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556ebc71cd10 con 0x556ebc762cd0 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.720+0000 7fb119ffb700 10 client.26617 handle_command_reply: tid=0 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.720+0000 7fb14bdf1b80 1 client.26617 shutdown 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.720+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 msgr2=0x556ebc757420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.720+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc757420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb120015000 tx=0x7fb120005fb0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 shutdown_connections 2023-07-29T16:18:56.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc762cd0 0x556ebc7650e0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc75e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc756ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/537131080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc757420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb119ffb700 0 client.26617 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 >> 172.21.15.31:0/537131080 conn(0x556ebc742fd0 msgr2=0x556ebc747a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 shutdown_connections 2023-07-29T16:18:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/537131080 wait complete. 2023-07-29T16:18:56.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.721+0000 7fb14bdf1b80 20 client.26617 trim_cache size 0 max 16384 2023-07-29T16:18:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.737+0000 7fb14bdf1b80 1 Processor -- start 2023-07-29T16:18:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.737+0000 7fb14bdf1b80 1 -- start start 2023-07-29T16:18:56.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc755140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc75e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc751c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc756390 con 0x556ebc5a96b0 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc756a50 con 0x556ebc74ecf0 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc748ff0 con 0x556ebc74ddf0 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb13599a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc75e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb13699c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc751c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb13619b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc755140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb13699c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc751c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36144/0 (socket says 172.21.15.31:36144) 2023-07-29T16:18:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb13699c700 1 -- 172.21.15.31:0/2493386444 learned_addr learned my addr 172.21.15.31:0/2493386444 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.738+0000 7fb13619b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc755140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47508/0 (socket says 172.21.15.31:47508) 2023-07-29T16:18:56.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13619b700 1 -- 172.21.15.31:0/2493386444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 msgr2=0x556ebc75e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13619b700 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc75e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13619b700 1 -- 172.21.15.31:0/2493386444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 msgr2=0x556ebc751c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13619b700 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc751c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13619b700 1 -- 172.21.15.31:0/2493386444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ebc4f31f0 con 0x556ebc5a96b0 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13699c700 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc751c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb13619b700 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc755140 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x556ebc7656c0 tx=0x7fb12c013ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de283722cbe0751 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb1253ff700 1 -- 172.21.15.31:0/2493386444 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c022240 con 0x556ebc5a96b0 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb1253ff700 1 -- 172.21.15.31:0/2493386444 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb12c019030 con 0x556ebc5a96b0 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.739+0000 7fb1253ff700 1 -- 172.21.15.31:0/2493386444 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c003b80 con 0x556ebc5a96b0 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2493386444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 msgr2=0x556ebc755140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc755140 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x556ebc7656c0 tx=0x7fb12c013ef0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2493386444 shutdown_connections 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ecf0 0x556ebc751c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc5a96b0 0x556ebc755140 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2493386444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc74ddf0 0x556ebc75e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2493386444 >> 172.21.15.31:0/2493386444 conn(0x556ebc742fd0 msgr2=0x556ebc755b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2493386444 shutdown_connections 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2493386444 wait complete. 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.740+0000 7fb14bdf1b80 1 -- start start 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 Processor -- start 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc75ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 0x556ebc75e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ecf0 0x556ebc75a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc75afe0 con 0x556ebc74ecf0 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc75b150 con 0x556ebc74ddf0 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ebc75b310 con 0x556ebc5a96b0 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb14bdf1b80 10 client.0 _ll_register_callbacks cb 0x556ebc6445c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb13599a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 0x556ebc75e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb13599a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 0x556ebc75e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36146/0 (socket says 172.21.15.31:36146) 2023-07-29T16:18:56.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.741+0000 7fb13599a700 1 -- 172.21.15.31:0/2236106746 learned_addr learned my addr 172.21.15.31:0/2236106746 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:18:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13619b700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc75ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13599a700 1 -- 172.21.15.31:0/2236106746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 msgr2=0x556ebc75ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.746 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13599a700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc75ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13699c700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ecf0 0x556ebc75a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13599a700 1 -- 172.21.15.31:0/2236106746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ecf0 msgr2=0x556ebc75a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13599a700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ecf0 0x556ebc75a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13599a700 1 -- 172.21.15.31:0/2236106746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ebc4ccd80 con 0x556ebc74ddf0 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13619b700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc75ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13699c700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ecf0 0x556ebc75a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:18:56.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb13599a700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 0x556ebc75e330 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb1200098e0 tx=0x7fb120005490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9bf33e088d4b2b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb119ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb119ffb700 1 -- 172.21.15.31:0/2236106746 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12002d040 con 0x556ebc74ddf0 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb119ffb700 1 -- 172.21.15.31:0/2236106746 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb120004d30 con 0x556ebc74ddf0 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.742+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb12809d210 con 0x556ebc74ddf0 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb119ffb700 1 -- 172.21.15.31:0/2236106746 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb12000f460 con 0x556ebc74ddf0 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 10 client.26606 fetch_fsmap learned FSMap version 26 2023-07-29T16:18:56.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fb12c0139d0 con 0x556ebc74ddf0 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb119ffb700 1 -- 172.21.15.31:0/2236106746 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb12809d210 con 0x556ebc74ddf0 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb119ffb700 1 -- 172.21.15.31:0/2236106746 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fb120017590 con 0x556ebc74ddf0 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 10 client.26606 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 10 client.26606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 20 client.26606 populate_metadata read hostname 'smithi031' 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc7b5ad0 0x556ebc7b7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:18:56.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 4 client.26606 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:18:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556ebc697ce0 con 0x556ebc7b5ad0 2023-07-29T16:18:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.743+0000 7fb13619b700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc7b5ad0 0x556ebc7b7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:18:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.744+0000 7fb13619b700 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc7b5ad0 0x556ebc7b7ee0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f336adb348d52bc server_cookie=7d094fa1e872b354 in_seq=0 out_seq=0 2023-07-29T16:18:56.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.744+0000 7fb119ffb700 10 client.26606 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:56.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:56 smithi039 ceph-mon[128858]: pgmap v474: 97 pgs: 97 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 308 op/s 2023-07-29T16:18:56.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb119ffb700 1 -- 172.21.15.31:0/2236106746 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556ebc697ce0 con 0x556ebc7b5ad0 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb119ffb700 10 client.26606 handle_command_reply: tid=0 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 client.26606 shutdown 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 msgr2=0x556ebc75e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 0x556ebc75e330 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb1200098e0 tx=0x7fb120005490 comp rx=0 tx=0).stop 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 shutdown_connections 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556ebc7b5ad0 0x556ebc7b7ee0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ebc74ecf0 0x556ebc75a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556ebc5a96b0 0x556ebc75ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 --2- 172.21.15.31:0/2236106746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556ebc74ddf0 0x556ebc75e330 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb119ffb700 0 client.26606 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:18:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 >> 172.21.15.31:0/2236106746 conn(0x556ebc742fd0 msgr2=0x556ebc755140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 shutdown_connections 2023-07-29T16:18:56.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.812+0000 7fb14bdf1b80 1 -- 172.21.15.31:0/2236106746 wait complete. 2023-07-29T16:18:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.813+0000 7fb14bdf1b80 20 client.26606 trim_cache size 0 max 16384 2023-07-29T16:18:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:18:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:18:56.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:18:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:18:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.813+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb11c0a0790 msgr2=0x7fb11c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.813+0000 7fb1383ff700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb11c0a0790 0x7fb11c0a2c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb12000b2e0 tx=0x7fb12000b850 comp rx=0 tx=0).stop 2023-07-29T16:18:56.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.813+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 msgr2=0x7fb130197580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.813+0000 7fb1383ff700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 0x7fb130197580 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb12800efc0 tx=0x7fb12800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 shutdown_connections 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb11c0a0790 0x7fb11c0a2c40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1301060a0 0x7fb130197580 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb130100e60 0x7fb1301961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 --2- 172.21.15.31:0/576628127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb130103780 0x7fb130196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:18:56.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 >> 172.21.15.31:0/576628127 conn(0x7fb1300fa840 msgr2=0x7fb130101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:18:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 shutdown_connections 2023-07-29T16:18:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:18:56.814+0000 7fb1383ff700 1 -- 172.21.15.31:0/576628127 wait complete. 2023-07-29T16:18:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:56 smithi156 ceph-mon[129245]: pgmap v474: 97 pgs: 97 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 308 op/s 2023-07-29T16:18:56.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 69069 9.50 MB/sec warmup 41 sec latency 41.434 ms 2023-07-29T16:18:56.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:56 smithi031 ceph-mon[95504]: pgmap v474: 97 pgs: 97 active+clean; 44 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.9 MiB/s wr, 308 op/s 2023-07-29T16:18:57.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 71210 9.60 MB/sec warmup 42 sec latency 16.575 ms 2023-07-29T16:18:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:18:57 smithi031 ceph-mon[95504]: pgmap v475: 97 pgs: 97 active+clean; 43 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 304 op/s 2023-07-29T16:18:58.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:18:57 smithi039 ceph-mon[128858]: pgmap v475: 97 pgs: 97 active+clean; 43 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 304 op/s 2023-07-29T16:18:58.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:18:57 smithi156 ceph-mon[129245]: pgmap v475: 97 pgs: 97 active+clean; 43 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 6.1 MiB/s wr, 304 op/s 2023-07-29T16:18:58.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 72482 9.50 MB/sec warmup 43 sec latency 65.556 ms 2023-07-29T16:18:59.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 74775 9.60 MB/sec warmup 44 sec latency 14.139 ms 2023-07-29T16:19:00.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:00 smithi031 ceph-mon[95504]: pgmap v476: 97 pgs: 97 active+clean; 31 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 413 op/s 2023-07-29T16:19:00.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:00 smithi039 ceph-mon[128858]: pgmap v476: 97 pgs: 97 active+clean; 31 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 413 op/s 2023-07-29T16:19:00.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:00 smithi156 ceph-mon[129245]: pgmap v476: 97 pgs: 97 active+clean; 31 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.4 MiB/s wr, 413 op/s 2023-07-29T16:19:00.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 76031 9.50 MB/sec warmup 45 sec latency 45.654 ms 2023-07-29T16:19:01.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 78315 9.60 MB/sec warmup 46 sec latency 13.979 ms 2023-07-29T16:19:02.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:19:02.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 -- 172.21.15.31:0/799053763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101f10 msgr2=0x7eff741087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 --2- 172.21.15.31:0/799053763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101f10 0x7eff741087d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009f90 tx=0x7eff6c00b330 comp rx=0 tx=0).stop 2023-07-29T16:19:02.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 -- 172.21.15.31:0/799053763 shutdown_connections 2023-07-29T16:19:02.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 --2- 172.21.15.31:0/799053763 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101f10 0x7eff741087d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 --2- 172.21.15.31:0/799053763 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff74100390 0x7eff741007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 --2- 172.21.15.31:0/799053763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101580 0x7eff741019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 -- 172.21.15.31:0/799053763 >> 172.21.15.31:0/799053763 conn(0x7eff740fb9b0 msgr2=0x7eff740fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 -- 172.21.15.31:0/799053763 shutdown_connections 2023-07-29T16:19:02.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.268+0000 7eff795ff700 1 -- 172.21.15.31:0/799053763 wait complete. 2023-07-29T16:19:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 Processor -- start 2023-07-29T16:19:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 -- start start 2023-07-29T16:19:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff74100390 0x7eff74198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 0x7eff741988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101f10 0x7eff7419f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff7419fef0 con 0x7eff74100390 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff741a0030 con 0x7eff74101580 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.270+0000 7eff795ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff741a01c0 con 0x7eff74101f10 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 0x7eff741988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff737fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101f10 0x7eff7419f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 0x7eff741988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34864/0 (socket says 172.21.15.31:34864) 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 -- 172.21.15.31:0/1570047154 learned_addr learned my addr 172.21.15.31:0/1570047154 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:02.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff72ffd700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff74100390 0x7eff74198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 -- 172.21.15.31:0/1570047154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101f10 msgr2=0x7eff7419f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101f10 0x7eff7419f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 -- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff74100390 msgr2=0x7eff74198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff74100390 0x7eff74198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 -- 172.21.15.31:0/1570047154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c009c40 con 0x7eff74101580 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff737fe700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101f10 0x7eff7419f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.271+0000 7eff6bfff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 0x7eff741988b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e9a0 tx=0x7eff5c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.272+0000 7eff70ff9700 1 -- 172.21.15.31:0/1570047154 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c009790 con 0x7eff74101580 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.272+0000 7eff70ff9700 1 -- 172.21.15.31:0/1570047154 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff5c00ce80 con 0x7eff74101580 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.272+0000 7eff70ff9700 1 -- 172.21.15.31:0/1570047154 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c018680 con 0x7eff74101580 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.272+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff741a04a0 con 0x7eff74101580 2023-07-29T16:19:02.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.272+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff741a09b0 con 0x7eff74101580 2023-07-29T16:19:02.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.275+0000 7eff70ff9700 1 -- 172.21.15.31:0/1570047154 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7eff5c010430 con 0x7eff74101580 2023-07-29T16:19:02.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.276+0000 7eff8cec7b80 1 Processor -- start 2023-07-29T16:19:02.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.276+0000 7eff8cec7b80 1 -- start start 2023-07-29T16:19:02.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff70ff9700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff600a0810 0x7eff600a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff70ff9700 1 -- 172.21.15.31:0/1570047154 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7eff5c014070 con 0x7eff74101580 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcef9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcefad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcefb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dce51a10 con 0x5597dcefa530 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dce4f3d0 con 0x5597dcef9be0 2023-07-29T16:19:02.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dce4acc0 con 0x5597dcefb2f0 2023-07-29T16:19:02.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff737fe700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff600a0810 0x7eff600a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff72ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcef9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcefad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff72ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcef9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34868/0 (socket says 172.21.15.31:34868) 2023-07-29T16:19:02.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.277+0000 7eff72ffd700 1 -- 172.21.15.31:0/359743436 learned_addr learned my addr 172.21.15.31:0/359743436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:02.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff737fe700 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcefb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff72ffd700 1 -- 172.21.15.31:0/359743436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 msgr2=0x5597dcefb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff72ffd700 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcefb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff72ffd700 1 -- 172.21.15.31:0/359743436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 msgr2=0x5597dcefad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff72ffd700 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcefad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff72ffd700 1 -- 172.21.15.31:0/359743436 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597dcc78de0 con 0x5597dcef9be0 2023-07-29T16:19:02.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff72ffd700 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcef9ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7eff64009d30 tx=0x7eff64011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=426284851a4ba823 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff737fe700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff600a0810 0x7eff600a2cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009790 tx=0x7eff6c007e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff697fa700 1 -- 172.21.15.31:0/359743436 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64004220 con 0x5597dcef9be0 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff697fa700 1 -- 172.21.15.31:0/359743436 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff64004cf0 con 0x5597dcef9be0 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.278+0000 7eff697fa700 1 -- 172.21.15.31:0/359743436 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff640054c0 con 0x5597dcef9be0 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 -- 172.21.15.31:0/359743436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 msgr2=0x5597dcef9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcef9ff0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7eff64009d30 tx=0x7eff64011720 comp rx=0 tx=0).stop 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 -- 172.21.15.31:0/359743436 shutdown_connections 2023-07-29T16:19:02.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcefb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcef9ff0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/359743436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcefad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 -- 172.21.15.31:0/359743436 >> 172.21.15.31:0/359743436 conn(0x5597dceef3e0 msgr2=0x5597dcef4990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 -- 172.21.15.31:0/359743436 shutdown_connections 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.279+0000 7eff8cec7b80 1 -- 172.21.15.31:0/359743436 wait complete. 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.280+0000 7eff8cec7b80 1 -- start start 2023-07-29T16:19:02.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.280+0000 7eff8cec7b80 1 Processor -- start 2023-07-29T16:19:02.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 0x5597dcf03d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefb2f0 0x5597dcf0ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcde7510 con 0x5597dcefb2f0 2023-07-29T16:19:02.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcde71b0 con 0x5597dcef9be0 2023-07-29T16:19:02.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dccce920 con 0x5597dcefa530 2023-07-29T16:19:02.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff8cec7b80 10 client.0 _ll_register_callbacks cb 0x5597dcdf0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 0x5597dcf03d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefb2f0 0x5597dcf0ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 0x5597dcf03d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41274/0 (socket says 172.21.15.31:41274) 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff6bfff700 1 -- 172.21.15.31:0/2765644987 learned_addr learned my addr 172.21.15.31:0/2765644987 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.281+0000 7eff72ffd700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff6bfff700 1 -- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 msgr2=0x5597dcf037d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff6bfff700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf037d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff6bfff700 1 -- 172.21.15.31:0/2765644987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefb2f0 msgr2=0x5597dcf0ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff6bfff700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefb2f0 0x5597dcf0ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff6bfff700 1 -- 172.21.15.31:0/2765644987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff64009960 con 0x5597dcefa530 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff737fe700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefb2f0 0x5597dcf0ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff6bfff700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 0x5597dcf03d10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e970 tx=0x7eff5c099940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90cb33e8d64f9c6b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff5a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff5a7fc700 1 -- 172.21.15.31:0/2765644987 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c09d430 con 0x5597dcefa530 2023-07-29T16:19:02.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597dce75d10 con 0x5597dcefa530 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.282+0000 7eff5a7fc700 1 -- 172.21.15.31:0/2765644987 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7eff5c0a1850 con 0x5597dcefa530 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff5a7fc700 1 -- 172.21.15.31:0/2765644987 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff5c010710 con 0x5597dcefa530 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 10 client.26635 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5597dcc61c60 con 0x5597dcefa530 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff5a7fc700 1 -- 172.21.15.31:0/2765644987 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5597dce75d10 con 0x5597dcefa530 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff5a7fc700 1 -- 172.21.15.31:0/2765644987 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7eff5c0952f0 con 0x5597dcefa530 2023-07-29T16:19:02.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 10 client.26635 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 10 client.26635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 20 client.26635 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf0f590 0x5597dcf119a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 4 client.26635 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.283+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597dcec8d70 con 0x5597dcf0f590 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.284+0000 7eff72ffd700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf0f590 0x5597dcf119a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.284+0000 7eff72ffd700 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf0f590 0x5597dcf119a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e83c376571c4b431 server_cookie=19c30c2ac19be182 in_seq=0 out_seq=0 2023-07-29T16:19:02.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.284+0000 7eff5a7fc700 10 client.26635 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff5a7fc700 1 -- 172.21.15.31:0/2765644987 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5597dcec8d70 con 0x5597dcf0f590 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff5a7fc700 10 client.26635 handle_command_reply: tid=0 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 client.26635 shutdown 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 msgr2=0x5597dcf03d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 0x5597dcf03d10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e970 tx=0x7eff5c099940 comp rx=0 tx=0).stop 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 shutdown_connections 2023-07-29T16:19:02.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf0f590 0x5597dcf119a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefb2f0 0x5597dcf0ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff5a7fc700 0 client.26635 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2765644987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefa530 0x5597dcf03d10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 >> 172.21.15.31:0/2765644987 conn(0x5597dceef3e0 msgr2=0x5597dcef4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.285+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 shutdown_connections 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.286+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2765644987 wait complete. 2023-07-29T16:19:02.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.286+0000 7eff8cec7b80 20 client.26635 trim_cache size 0 max 16384 2023-07-29T16:19:02.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.303+0000 7eff8cec7b80 1 Processor -- start 2023-07-29T16:19:02.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 -- start start 2023-07-29T16:19:02.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf08d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcef4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcf02cb0 con 0x5597dcefa530 2023-07-29T16:19:02.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcf03370 con 0x5597dcef9be0 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcef14b0 con 0x5597dcefb2f0 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcef4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff737fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcef4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46168/0 (socket says 172.21.15.31:46168) 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff6bfff700 1 -- 172.21.15.31:0/2837406121 learned_addr learned my addr 172.21.15.31:0/2837406121 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff737fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf016c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41290/0 (socket says 172.21.15.31:41290) 2023-07-29T16:19:02.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.304+0000 7eff72ffd700 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf08d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff737fe700 1 -- 172.21.15.31:0/2837406121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 msgr2=0x5597dcf08d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff737fe700 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf08d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff737fe700 1 -- 172.21.15.31:0/2837406121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 msgr2=0x5597dcef4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff737fe700 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcef4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff737fe700 1 -- 172.21.15.31:0/2837406121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597dcc9f250 con 0x5597dcefb2f0 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff6bfff700 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcef4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff72ffd700 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf08d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:02.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff737fe700 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf016c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009c10 tx=0x7eff6c023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23cce22530f2f5b7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff69ffb700 1 -- 172.21.15.31:0/2837406121 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00e040 con 0x5597dcefb2f0 2023-07-29T16:19:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff69ffb700 1 -- 172.21.15.31:0/2837406121 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff6c00bb90 con 0x5597dcefb2f0 2023-07-29T16:19:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.305+0000 7eff69ffb700 1 -- 172.21.15.31:0/2837406121 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c02b480 con 0x5597dcefb2f0 2023-07-29T16:19:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2837406121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 msgr2=0x5597dcf016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf016c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009c10 tx=0x7eff6c023000 comp rx=0 tx=0).stop 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2837406121 shutdown_connections 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf016c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcef9be0 0x5597dcf08d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/2837406121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcefa530 0x5597dcef4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2837406121 >> 172.21.15.31:0/2837406121 conn(0x5597dceef3e0 msgr2=0x5597dcef5690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2837406121 shutdown_connections 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 -- 172.21.15.31:0/2837406121 wait complete. 2023-07-29T16:19:02.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.306+0000 7eff8cec7b80 1 -- start start 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 Processor -- start 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcef9be0 0x5597dcf075e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 0x5597dcf07b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf08060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcf610d0 con 0x5597dcef9be0 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcf61240 con 0x5597dcefa530 2023-07-29T16:19:02.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597dcf61400 con 0x5597dcefb2f0 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff8cec7b80 10 client.0 _ll_register_callbacks cb 0x5597dcdf0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 0x5597dcf07b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff6bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 0x5597dcf07b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34894/0 (socket says 172.21.15.31:34894) 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff6bfff700 1 -- 172.21.15.31:0/3651390531 learned_addr learned my addr 172.21.15.31:0/3651390531 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.307+0000 7eff72ffd700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcef9be0 0x5597dcf075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff737fe700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf08060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff6bfff700 1 -- 172.21.15.31:0/3651390531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 msgr2=0x5597dcf08060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff6bfff700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf08060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff6bfff700 1 -- 172.21.15.31:0/3651390531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcef9be0 msgr2=0x5597dcf075e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff6bfff700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcef9be0 0x5597dcf075e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff6bfff700 1 -- 172.21.15.31:0/3651390531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597dcc78de0 con 0x5597dcefa530 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff6bfff700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 0x5597dcf07b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e970 tx=0x7eff5c09d360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=529020819e4d0538 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff5a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff5a7fc700 1 -- 172.21.15.31:0/3651390531 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0953f0 con 0x5597dcefa530 2023-07-29T16:19:02.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff5a7fc700 1 -- 172.21.15.31:0/3651390531 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7eff5c0986f0 con 0x5597dcefa530 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7eff64004cf0 con 0x5597dcefa530 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.308+0000 7eff5a7fc700 1 -- 172.21.15.31:0/3651390531 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff5c0a1850 con 0x5597dcefa530 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 10 client.26624 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5597dcc61c60 con 0x5597dcefa530 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff5a7fc700 1 -- 172.21.15.31:0/3651390531 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff64004cf0 con 0x5597dcefa530 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff5a7fc700 1 -- 172.21.15.31:0/3651390531 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7eff5c080070 con 0x5597dcefa530 2023-07-29T16:19:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 10 client.26624 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 10 client.26624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 20 client.26624 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf62b30 0x5597dcf64f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 4 client.26624 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5597dce43d40 con 0x5597dcf62b30 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.309+0000 7eff72ffd700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf62b30 0x5597dcf64f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff72ffd700 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf62b30 0x5597dcf64f40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adad8fcb84c372aa server_cookie=79124ba1dbff8743 in_seq=0 out_seq=0 2023-07-29T16:19:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff5a7fc700 10 client.26624 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:02.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff5a7fc700 1 -- 172.21.15.31:0/3651390531 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5597dce43d40 con 0x5597dcf62b30 2023-07-29T16:19:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff5a7fc700 10 client.26624 handle_command_reply: tid=0 2023-07-29T16:19:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 client.26624 shutdown 2023-07-29T16:19:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 msgr2=0x5597dcf07b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 0x5597dcf07b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e970 tx=0x7eff5c09d360 comp rx=0 tx=0).stop 2023-07-29T16:19:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 shutdown_connections 2023-07-29T16:19:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597dcf62b30 0x5597dcf64f40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597dcefb2f0 0x5597dcf08060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597dcef9be0 0x5597dcf075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff5a7fc700 0 client.26624 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 --2- 172.21.15.31:0/3651390531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597dcefa530 0x5597dcf07b20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.310+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 >> 172.21.15.31:0/3651390531 conn(0x5597dceef3e0 msgr2=0x5597dcef4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 shutdown_connections 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff8cec7b80 1 -- 172.21.15.31:0/3651390531 wait complete. 2023-07-29T16:19:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff8cec7b80 20 client.26624 trim_cache size 0 max 16384 2023-07-29T16:19:02.319 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:19:02.319 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:19:02.319 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:19:02.319 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:19:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff600a0810 msgr2=0x7eff600a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff795ff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff600a0810 0x7eff600a2cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009790 tx=0x7eff6c007e10 comp rx=0 tx=0).stop 2023-07-29T16:19:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 msgr2=0x7eff741988b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff795ff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 0x7eff741988b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7eff5c00e9a0 tx=0x7eff5c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.311+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 shutdown_connections 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7eff600a0810 0x7eff600a2cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff74101f10 0x7eff7419f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7eff74100390 0x7eff74198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 --2- 172.21.15.31:0/1570047154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff74101580 0x7eff741988b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 >> 172.21.15.31:0/1570047154 conn(0x7eff740fb9b0 msgr2=0x7eff74106dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 shutdown_connections 2023-07-29T16:19:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.312+0000 7eff795ff700 1 -- 172.21.15.31:0/1570047154 wait complete. 2023-07-29T16:19:02.329 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:19:02.330 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:19:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 -- 172.21.15.31:0/2430316071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101530 msgr2=0x7fccf4101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 --2- 172.21.15.31:0/2430316071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101530 0x7fccf4101980 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009990 tx=0x7fcce8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 -- 172.21.15.31:0/2430316071 shutdown_connections 2023-07-29T16:19:02.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 --2- 172.21.15.31:0/2430316071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4101ec0 0x7fccf4108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 --2- 172.21.15.31:0/2430316071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccf4100340 0x7fccf4100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 --2- 172.21.15.31:0/2430316071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101530 0x7fccf4101980 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 -- 172.21.15.31:0/2430316071 >> 172.21.15.31:0/2430316071 conn(0x7fccf40fb920 msgr2=0x7fccf40fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 -- 172.21.15.31:0/2430316071 shutdown_connections 2023-07-29T16:19:02.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.544+0000 7fccf9dff700 1 -- 172.21.15.31:0/2430316071 wait complete. 2023-07-29T16:19:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.545+0000 7fccf9dff700 1 Processor -- start 2023-07-29T16:19:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 -- start start 2023-07-29T16:19:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 0x7fccf4198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccf4101530 0x7fccf4198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101ec0 0x7fccf419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf419fdd0 con 0x7fccf4100340 2023-07-29T16:19:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf419ff10 con 0x7fccf4101ec0 2023-07-29T16:19:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf9dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf41a00a0 con 0x7fccf4101530 2023-07-29T16:19:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.546+0000 7fccf37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 0x7fccf4198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101ec0 0x7fccf419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 0x7fccf4198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46206/0 (socket says 172.21.15.31:46206) 2023-07-29T16:19:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 -- 172.21.15.31:0/3429330134 learned_addr learned my addr 172.21.15.31:0/3429330134 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf2ffd700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccf4101530 0x7fccf4198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 -- 172.21.15.31:0/3429330134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccf4101530 msgr2=0x7fccf4198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccf4101530 0x7fccf4198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 -- 172.21.15.31:0/3429330134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101ec0 msgr2=0x7fccf419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101ec0 0x7fccf419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 -- 172.21.15.31:0/3429330134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce80095f0 con 0x7fccf4100340 2023-07-29T16:19:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf37fe700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 0x7fccf4198250 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e970 tx=0x7fcce400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.547+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4009790 con 0x7fccf4100340 2023-07-29T16:19:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.548+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcce400c7e0 con 0x7fccf4100340 2023-07-29T16:19:02.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.548+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf41a0380 con 0x7fccf4100340 2023-07-29T16:19:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.548+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4003c70 con 0x7fccf4100340 2023-07-29T16:19:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.548+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf41a08e0 con 0x7fccf4100340 2023-07-29T16:19:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.549+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf4192a30 con 0x7fccf4100340 2023-07-29T16:19:02.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.551+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fcce400cc50 con 0x7fccf4100340 2023-07-29T16:19:02.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.553+0000 7fccf0ff9700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fccdc0a0860 0x7fccdc0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:02.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.553+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcce4014070 con 0x7fccf4100340 2023-07-29T16:19:02.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.553+0000 7fccf2ffd700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fccdc0a0860 0x7fccdc0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:02.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.554+0000 7fccf2ffd700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fccdc0a0860 0x7fccdc0a2d10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcce800b2e0 tx=0x7fcce8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.558+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fcce4095710 con 0x7fccf4100340 2023-07-29T16:19:02.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:02 smithi031 ceph-mon[95504]: pgmap v477: 97 pgs: 97 active+clean; 31 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 275 op/s 2023-07-29T16:19:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:02 smithi039 ceph-mon[128858]: pgmap v477: 97 pgs: 97 active+clean; 31 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 275 op/s 2023-07-29T16:19:02.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:02 smithi156 ceph-mon[129245]: pgmap v477: 97 pgs: 97 active+clean; 31 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.3 MiB/s wr, 275 op/s 2023-07-29T16:19:02.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.842+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccf402cc30 con 0x7fccf4100340 2023-07-29T16:19:02.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.843+0000 7fccf0ff9700 1 -- 172.21.15.31:0/3429330134 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10777 (secure 0 0 0) 0x7fcce4003dd0 con 0x7fccf4100340 2023-07-29T16:19:02.849 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:19:02.849 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:18:33.432992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:19:02.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fccdc0a0860 msgr2=0x7fccdc0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fccdc0a0860 0x7fccdc0a2d10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcce800b2e0 tx=0x7fcce8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 msgr2=0x7fccf4198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 0x7fccf4198250 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fcce400e970 tx=0x7fcce400ec80 comp rx=0 tx=0).stop 2023-07-29T16:19:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 shutdown_connections 2023-07-29T16:19:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fccdc0a0860 0x7fccdc0a2d10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf4101ec0 0x7fccf419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fccf4100340 0x7fccf4198250 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 --2- 172.21.15.31:0/3429330134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccf4101530 0x7fccf4198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:02.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 >> 172.21.15.31:0/3429330134 conn(0x7fccf40fb920 msgr2=0x7fccf4106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:02.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.845+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 shutdown_connections 2023-07-29T16:19:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:02.846+0000 7fccf9dff700 1 -- 172.21.15.31:0/3429330134 wait complete. 2023-07-29T16:19:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2023-07-29T16:19:02.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 79159 9.44 MB/sec warmup 47 sec latency 20.961 ms 2023-07-29T16:19:02.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:19:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 -- 172.21.15.31:0/622161414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 msgr2=0x7f6fdc0ff630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/622161414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc0ff630 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc009990 tx=0x7f6fcc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 -- 172.21.15.31:0/622161414 shutdown_connections 2023-07-29T16:19:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/622161414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc100da0 0x7f6fdc107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/622161414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc0ff630 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/622161414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fdc100410 0x7f6fdc100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.091+0000 7f6fe33ff700 1 -- 172.21.15.31:0/622161414 >> 172.21.15.31:0/622161414 conn(0x7f6fdc0fa840 msgr2=0x7f6fdc0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.092+0000 7f6fe33ff700 1 -- 172.21.15.31:0/622161414 shutdown_connections 2023-07-29T16:19:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.092+0000 7f6fe33ff700 1 -- 172.21.15.31:0/622161414 wait complete. 2023-07-29T16:19:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.093+0000 7f6fe33ff700 1 Processor -- start 2023-07-29T16:19:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.093+0000 7f6fe33ff700 1 -- start start 2023-07-29T16:19:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.093+0000 7f6fe33ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe33ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fdc100410 0x7f6fdc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe33ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc100da0 0x7f6fdc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe33ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc19fe10 con 0x7f6fdc100410 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe33ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc19ff50 con 0x7f6fdc100da0 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe33ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc1a00e0 con 0x7f6fdc0ff220 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41340/0 (socket says 172.21.15.31:41340) 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 -- 172.21.15.31:0/224010351 learned_addr learned my addr 172.21.15.31:0/224010351 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe099a700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fdc100410 0x7f6fdc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe199c700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc100da0 0x7f6fdc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 -- 172.21.15.31:0/224010351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc100da0 msgr2=0x7f6fdc19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc100da0 0x7f6fdc19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 -- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fdc100410 msgr2=0x7f6fdc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.094+0000 7f6fe119b700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fdc100410 0x7f6fdc1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fe119b700 1 -- 172.21.15.31:0/224010351 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc0095f0 con 0x7f6fdc0ff220 2023-07-29T16:19:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fe119b700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc198290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc000c00 tx=0x7f6fcc00f670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fd27fc700 1 -- 172.21.15.31:0/224010351 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc01d070 con 0x7f6fdc0ff220 2023-07-29T16:19:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fd27fc700 1 -- 172.21.15.31:0/224010351 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fcc005510 con 0x7f6fdc0ff220 2023-07-29T16:19:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc1a0360 con 0x7f6fdc0ff220 2023-07-29T16:19:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fd27fc700 1 -- 172.21.15.31:0/224010351 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc004bd0 con 0x7f6fdc0ff220 2023-07-29T16:19:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.095+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc1a07e0 con 0x7f6fdc0ff220 2023-07-29T16:19:03.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.100+0000 7f6fd27fc700 1 -- 172.21.15.31:0/224010351 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6fcc005050 con 0x7f6fdc0ff220 2023-07-29T16:19:03.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.101+0000 7f6ff6c43b80 1 Processor -- start 2023-07-29T16:19:03.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.101+0000 7f6fd27fc700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6fc80a0810 0x7f6fc80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 -- start start 2023-07-29T16:19:03.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6fd27fc700 1 -- 172.21.15.31:0/224010351 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6fcc0cca60 con 0x7f6fdc0ff220 2023-07-29T16:19:03.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83ac0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ac1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ac2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83a18a10 con 0x564d83ac1530 2023-07-29T16:19:03.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83a163d0 con 0x564d83ac22f0 2023-07-29T16:19:03.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83a11cc0 con 0x564d83ac0be0 2023-07-29T16:19:03.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83ac0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83ac0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41354/0 (socket says 172.21.15.31:41354) 2023-07-29T16:19:03.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6fe119b700 1 -- 172.21.15.31:0/1316398870 learned_addr learned my addr 172.21.15.31:0/1316398870 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.102+0000 7f6fe099a700 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ac1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe199c700 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ac2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe199c700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6fc80a0810 0x7f6fc80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe119b700 1 -- 172.21.15.31:0/1316398870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 msgr2=0x564d83ac2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe119b700 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ac2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe119b700 1 -- 172.21.15.31:0/1316398870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 msgr2=0x564d83ac1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe119b700 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ac1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe119b700 1 -- 172.21.15.31:0/1316398870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d8383fde0 con 0x564d83ac0be0 2023-07-29T16:19:03.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fe119b700 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83ac0ff0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc09a180 tx=0x7f6fcc009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e3bfc9f4fa150f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.103+0000 7f6fc77fe700 1 -- 172.21.15.31:0/1316398870 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0a00d0 con 0x564d83ac0be0 2023-07-29T16:19:03.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.104+0000 7f6fe199c700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6fc80a0810 0x7f6fc80a2cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4007fd0 tx=0x7f6fd4004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.104+0000 7f6fc77fe700 1 -- 172.21.15.31:0/1316398870 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fcc09e020 con 0x564d83ac0be0 2023-07-29T16:19:03.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.104+0000 7f6fc77fe700 1 -- 172.21.15.31:0/1316398870 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc098590 con 0x564d83ac0be0 2023-07-29T16:19:03.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.104+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/1316398870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 msgr2=0x564d83ac0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.104+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83ac0ff0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc09a180 tx=0x7f6fcc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/1316398870 shutdown_connections 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ac2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83ac0ff0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/1316398870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ac1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/1316398870 >> 172.21.15.31:0/1316398870 conn(0x564d83ab63e0 msgr2=0x564d83abb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/1316398870 shutdown_connections 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/1316398870 wait complete. 2023-07-29T16:19:03.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.105+0000 7f6ff6c43b80 1 -- start start 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 Processor -- start 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83aca490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83aca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ad1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d839ae510 con 0x564d83ac1530 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d839ae1b0 con 0x564d83ac22f0 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83895920 con 0x564d83ac0be0 2023-07-29T16:19:03.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6ff6c43b80 10 client.0 _ll_register_callbacks cb 0x564d839b7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.106+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83aca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83aca490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41368/0 (socket says 172.21.15.31:41368) 2023-07-29T16:19:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ad1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 -- 172.21.15.31:0/4277506369 learned_addr learned my addr 172.21.15.31:0/4277506369 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe099a700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83aca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 -- 172.21.15.31:0/4277506369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 msgr2=0x564d83ad1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ad1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 -- 172.21.15.31:0/4277506369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 msgr2=0x564d83aca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83aca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 -- 172.21.15.31:0/4277506369 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc1a07e0 con 0x564d83ac0be0 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe199c700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ad1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fe119b700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83aca490 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0ca550 tx=0x7f6fcc099a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29115c2fc131a1a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fc4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.107+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/4277506369 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc00fb50 con 0x564d83ac0be0 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/4277506369 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6fcc09a1f0 con 0x564d83ac0be0 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d83a3cd10 con 0x564d83ac0be0 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/4277506369 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fcc098590 con 0x564d83ac0be0 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 10 client.26659 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564d83828c60 con 0x564d83ac0be0 2023-07-29T16:19:03.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/4277506369 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fcc07d020 con 0x564d83ac0be0 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/4277506369 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6fcc0a2120 con 0x564d83ac0be0 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 10 client.26659 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 10 client.26659 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 20 client.26659 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83ad5d40 0x564d83ad8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 4 client.26659 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:03.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.108+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d83a8fd70 con 0x564d83ad5d40 2023-07-29T16:19:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.109+0000 7f6fe099a700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83ad5d40 0x564d83ad8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.109+0000 7f6fe099a700 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83ad5d40 0x564d83ad8150 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6749922881133bf server_cookie=c9f7e453bb821a4b in_seq=0 out_seq=0 2023-07-29T16:19:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.109+0000 7f6fc4ff9700 10 client.26659 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/4277506369 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564d83a8fd70 con 0x564d83ad5d40 2023-07-29T16:19:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6fc4ff9700 10 client.26659 handle_command_reply: tid=0 2023-07-29T16:19:03.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6ff6c43b80 1 client.26659 shutdown 2023-07-29T16:19:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 msgr2=0x564d83aca490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83aca490 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc0ca550 tx=0x7f6fcc099a90 comp rx=0 tx=0).stop 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 shutdown_connections 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83ad5d40 0x564d83ad8150 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.110+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83ad1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83aca490 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/4277506369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83aca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6fc4ff9700 0 client.26659 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 >> 172.21.15.31:0/4277506369 conn(0x564d83ab63e0 msgr2=0x564d83abb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 shutdown_connections 2023-07-29T16:19:03.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/4277506369 wait complete. 2023-07-29T16:19:03.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.111+0000 7f6ff6c43b80 20 client.26659 trim_cache size 0 max 16384 2023-07-29T16:19:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.126+0000 7f6ff6c43b80 1 Processor -- start 2023-07-29T16:19:03.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.126+0000 7f6ff6c43b80 1 -- start start 2023-07-29T16:19:03.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 0x564d83ac26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac22f0 0x564d83ac9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83aca060 con 0x564d83ac22f0 2023-07-29T16:19:03.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83ac2c30 con 0x564d83ac1530 2023-07-29T16:19:03.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83ac1bc0 con 0x564d83ac0be0 2023-07-29T16:19:03.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 0x564d83ac26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 0x564d83ac26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34962/0 (socket says 172.21.15.31:34962) 2023-07-29T16:19:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe099a700 1 -- 172.21.15.31:0/2254571217 learned_addr learned my addr 172.21.15.31:0/2254571217 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe199c700 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac22f0 0x564d83ac9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe119b700 1 -- 172.21.15.31:0/2254571217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 msgr2=0x564d83ac26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe119b700 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 0x564d83ac26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe119b700 1 -- 172.21.15.31:0/2254571217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac22f0 msgr2=0x564d83ac9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe119b700 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac22f0 0x564d83ac9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe119b700 1 -- 172.21.15.31:0/2254571217 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d83866250 con 0x564d83ac0be0 2023-07-29T16:19:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.127+0000 7f6fe099a700 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 0x564d83ac26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.128+0000 7f6fe119b700 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acf9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc09a180 tx=0x7f6fcc00ff50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=609e3fe86873dc69 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.128+0000 7f6fc7fff700 1 -- 172.21.15.31:0/2254571217 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0a2640 con 0x564d83ac0be0 2023-07-29T16:19:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.128+0000 7f6fc7fff700 1 -- 172.21.15.31:0/2254571217 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fcc07d070 con 0x564d83ac0be0 2023-07-29T16:19:03.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.128+0000 7f6fc7fff700 1 -- 172.21.15.31:0/2254571217 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fcc0b8460 con 0x564d83ac0be0 2023-07-29T16:19:03.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/2254571217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 msgr2=0x564d83acf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acf9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc09a180 tx=0x7f6fcc00ff50 comp rx=0 tx=0).stop 2023-07-29T16:19:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/2254571217 shutdown_connections 2023-07-29T16:19:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac22f0 0x564d83ac9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acf9c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/2254571217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac1530 0x564d83ac26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/2254571217 >> 172.21.15.31:0/2254571217 conn(0x564d83ab63e0 msgr2=0x564d83abb900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/2254571217 shutdown_connections 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/2254571217 wait complete. 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 -- start start 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.129+0000 7f6ff6c43b80 1 Processor -- start 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ace0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83aceb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83acf060 con 0x564d83ac1530 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83acf1d0 con 0x564d83ac22f0 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d83acf390 con 0x564d83ac0be0 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6ff6c43b80 10 client.0 _ll_register_callbacks cb 0x564d839b7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6fe199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83aceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6fe199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83aceb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34966/0 (socket says 172.21.15.31:34966) 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6fe199c700 1 -- 172.21.15.31:0/3363211104 learned_addr learned my addr 172.21.15.31:0/3363211104 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6fe119b700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.130+0000 7f6fe099a700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ace0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fe199c700 1 -- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 msgr2=0x564d83acdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fe199c700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fe199c700 1 -- 172.21.15.31:0/3363211104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 msgr2=0x564d83ace0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fe199c700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ace0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fe199c700 1 -- 172.21.15.31:0/3363211104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd8007e30 con 0x564d83ac22f0 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fe199c700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83aceb20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40184b0 tx=0x7f6fd4004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93791fd59c4fd707 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fc4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/3363211104 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd4004bb0 con 0x564d83ac22f0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fcc09e020 con 0x564d83ac22f0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.131+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/3363211104 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6fd4004d10 con 0x564d83ac22f0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/3363211104 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fd4018bb0 con 0x564d83ac22f0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 10 client.26645 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564d83828c60 con 0x564d83ac22f0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/3363211104 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fcc09e020 con 0x564d83ac22f0 2023-07-29T16:19:03.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/3363211104 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6fd4005020 con 0x564d83ac22f0 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 10 client.26645 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 10 client.26645 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 20 client.26645 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83b286e0 0x564d83b2aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 4 client.26645 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.132+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564d83a0ad40 con 0x564d83b286e0 2023-07-29T16:19:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6fe119b700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83b286e0 0x564d83b2aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6fe119b700 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83b286e0 0x564d83b2aaf0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8b66ab73d5a90cc server_cookie=446b4169a97b31c6 in_seq=0 out_seq=0 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6fc4ff9700 10 client.26645 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6fc4ff9700 1 -- 172.21.15.31:0/3363211104 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564d83a0ad40 con 0x564d83b286e0 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6fc4ff9700 10 client.26645 handle_command_reply: tid=0 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 client.26645 shutdown 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 msgr2=0x564d83aceb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83aceb20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6fd40184b0 tx=0x7f6fd4004ad0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 shutdown_connections 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x564d83b286e0 0x564d83b2aaf0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d83ac22f0 0x564d83aceb20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d83ac0be0 0x564d83acdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 --2- 172.21.15.31:0/3363211104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d83ac1530 0x564d83ace0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6fc4ff9700 0 client.26645 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.133+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 >> 172.21.15.31:0/3363211104 conn(0x564d83ab63e0 msgr2=0x564d83ac8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 shutdown_connections 2023-07-29T16:19:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6ff6c43b80 1 -- 172.21.15.31:0/3363211104 wait complete. 2023-07-29T16:19:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6ff6c43b80 20 client.26645 trim_cache size 0 max 16384 2023-07-29T16:19:03.147 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:19:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6fc80a0810 msgr2=0x7f6fc80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6fc80a0810 0x7f6fc80a2cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4007fd0 tx=0x7f6fd4004b40 comp rx=0 tx=0).stop 2023-07-29T16:19:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 msgr2=0x7f6fdc198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.134+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc198290 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6fcc000c00 tx=0x7f6fcc00f670 comp rx=0 tx=0).stop 2023-07-29T16:19:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 shutdown_connections 2023-07-29T16:19:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6fc80a0810 0x7f6fc80a2cc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fdc100da0 0x7f6fdc19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fdc0ff220 0x7f6fdc198290 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 --2- 172.21.15.31:0/224010351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fdc100410 0x7f6fdc1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 >> 172.21.15.31:0/224010351 conn(0x7f6fdc0fa840 msgr2=0x7f6fdc105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 shutdown_connections 2023-07-29T16:19:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.135+0000 7f6fe33ff700 1 -- 172.21.15.31:0/224010351 wait complete. 2023-07-29T16:19:03.152 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:19:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.370+0000 7f1b575ff700 1 -- 172.21.15.31:0/1124433627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50100330 msgr2=0x7f1b50100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.370+0000 7f1b575ff700 1 --2- 172.21.15.31:0/1124433627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50100330 0x7f1b50100740 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0098f0 tx=0x7f1b4c009c00 comp rx=0 tx=0).stop 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 -- 172.21.15.31:0/1124433627 shutdown_connections 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 --2- 172.21.15.31:0/1124433627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b50101eb0 0x7f1b50108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 --2- 172.21.15.31:0/1124433627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50100330 0x7f1b50100740 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 --2- 172.21.15.31:0/1124433627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50101520 0x7f1b50101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 -- 172.21.15.31:0/1124433627 >> 172.21.15.31:0/1124433627 conn(0x7f1b500fb970 msgr2=0x7f1b500fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 -- 172.21.15.31:0/1124433627 shutdown_connections 2023-07-29T16:19:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.371+0000 7f1b575ff700 1 -- 172.21.15.31:0/1124433627 wait complete. 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 Processor -- start 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 -- start start 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 0x7f1b50074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50101520 0x7f1b50073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b50101eb0 0x7f1b50073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50073e20 con 0x7f1b50101520 2023-07-29T16:19:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50073f90 con 0x7f1b50100330 2023-07-29T16:19:03.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b575ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b501a0150 con 0x7f1b50101eb0 2023-07-29T16:19:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b5539b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 0x7f1b50074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.372+0000 7f1b54b9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50101520 0x7f1b50073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 0x7f1b50074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34986/0 (socket says 172.21.15.31:34986) 2023-07-29T16:19:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 -- 172.21.15.31:0/3708763454 learned_addr learned my addr 172.21.15.31:0/3708763454 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b50101eb0 0x7f1b50073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 -- 172.21.15.31:0/3708763454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b50101eb0 msgr2=0x7f1b50073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b50101eb0 0x7f1b50073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 -- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50101520 msgr2=0x7f1b50073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50101520 0x7f1b50073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 -- 172.21.15.31:0/3708763454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c009520 con 0x7f1b50100330 2023-07-29T16:19:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50101520 0x7f1b50073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 0x7f1b50074b00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005e90 tx=0x7f1b4c00b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b467fc700 1 -- 172.21.15.31:0/3708763454 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c01c070 con 0x7f1b50100330 2023-07-29T16:19:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b467fc700 1 -- 172.21.15.31:0/3708763454 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b4c00f7f0 con 0x7f1b50100330 2023-07-29T16:19:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.373+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b501a03d0 con 0x7f1b50100330 2023-07-29T16:19:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.374+0000 7f1b467fc700 1 -- 172.21.15.31:0/3708763454 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c003990 con 0x7f1b50100330 2023-07-29T16:19:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.374+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b501a07f0 con 0x7f1b50100330 2023-07-29T16:19:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.377+0000 7f1b467fc700 1 -- 172.21.15.31:0/3708763454 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1b4c020030 con 0x7f1b50100330 2023-07-29T16:19:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.378+0000 7f1b6afbcb80 1 Processor -- start 2023-07-29T16:19:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.378+0000 7f1b6afbcb80 1 -- start start 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119ca1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119ca2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119bf9a10 con 0x564119ca2530 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119bf73d0 con 0x564119ca32f0 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119bf2cc0 con 0x564119ca1be0 2023-07-29T16:19:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119ca1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b5539b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119ca2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b54b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b54b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34998/0 (socket says 172.21.15.31:34998) 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b54b9a700 1 -- 172.21.15.31:0/1850050435 learned_addr learned my addr 172.21.15.31:0/1850050435 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119ca1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41400/0 (socket says 172.21.15.31:41400) 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b467fc700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b3c0a0790 0x7f1b3c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 -- 172.21.15.31:0/1850050435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 msgr2=0x564119ca3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 -- 172.21.15.31:0/1850050435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 msgr2=0x564119ca2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119ca2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b55b9c700 1 -- 172.21.15.31:0/1850050435 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564119a20de0 con 0x564119ca1be0 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.379+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.380+0000 7f1b467fc700 1 -- 172.21.15.31:0/3708763454 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1b4c0cca60 con 0x7f1b50100330 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.380+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b3c0a0790 0x7f1b3c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.380+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119ca1ff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b48011550 tx=0x7f1b48009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f22581bcb03ec081 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.380+0000 7f1b3b7fe700 1 -- 172.21.15.31:0/1850050435 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800e040 con 0x564119ca1be0 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.380+0000 7f1b3b7fe700 1 -- 172.21.15.31:0/1850050435 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b4800f040 con 0x564119ca1be0 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.380+0000 7f1b3b7fe700 1 -- 172.21.15.31:0/1850050435 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48007900 con 0x564119ca1be0 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1850050435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 msgr2=0x564119ca1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b3c0a0790 0x7f1b3c0a2c40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1b4801d000 tx=0x7f1b48007ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119ca1ff0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1b48011550 tx=0x7f1b48009920 comp rx=0 tx=0).stop 2023-07-29T16:19:03.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1850050435 shutdown_connections 2023-07-29T16:19:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119ca1ff0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1850050435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119ca2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1850050435 >> 172.21.15.31:0/1850050435 conn(0x564119c973e0 msgr2=0x564119c9c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1850050435 shutdown_connections 2023-07-29T16:19:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.381+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1850050435 wait complete. 2023-07-29T16:19:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.382+0000 7f1b6afbcb80 1 -- start start 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.382+0000 7f1b6afbcb80 1 Processor -- start 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca1be0 0x564119cab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 0x564119cabcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca32f0 0x564119cb2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119b8f510 con 0x564119ca1be0 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119b8f1b0 con 0x564119ca2530 2023-07-29T16:19:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119a76920 con 0x564119ca32f0 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b6afbcb80 10 client.0 _ll_register_callbacks cb 0x564119b98620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b54b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 0x564119cabcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b54b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 0x564119cabcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35014/0 (socket says 172.21.15.31:35014) 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b54b9a700 1 -- 172.21.15.31:0/3658405544 learned_addr learned my addr 172.21.15.31:0/3658405544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.383+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca32f0 0x564119cb2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca1be0 0x564119cab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b54b9a700 1 -- 172.21.15.31:0/3658405544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca32f0 msgr2=0x564119cb2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca32f0 0x564119cb2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b54b9a700 1 -- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca1be0 msgr2=0x564119cab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca1be0 0x564119cab770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b54b9a700 1 -- 172.21.15.31:0/3658405544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b40009d90 con 0x564119ca2530 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca1be0 0x564119cab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 0x564119cabcb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b40012da0 tx=0x7f1b4000f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16af3d3a3aff8be server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b38ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b38ff9700 1 -- 172.21.15.31:0/3658405544 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40017070 con 0x564119ca2530 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b38ff9700 1 -- 172.21.15.31:0/3658405544 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1b400120d0 con 0x564119ca2530 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.384+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564119c1dd10 con 0x564119ca2530 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b38ff9700 1 -- 172.21.15.31:0/3658405544 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b40012290 con 0x564119ca2530 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 10 client.26657 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f1b48011230 con 0x564119ca2530 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b38ff9700 1 -- 172.21.15.31:0/3658405544 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564119c1dd10 con 0x564119ca2530 2023-07-29T16:19:03.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b38ff9700 1 -- 172.21.15.31:0/3658405544 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f1b400104c0 con 0x564119ca2530 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 10 client.26657 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 10 client.26657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 20 client.26657 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119cb7520 0x564119cb9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 4 client.26657 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.385+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564119c70d70 con 0x564119cb7520 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.386+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119cb7520 0x564119cb9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.386+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119cb7520 0x564119cb9930 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e92db305c9d55168 server_cookie=b54524b4c7c80273 in_seq=0 out_seq=0 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.386+0000 7f1b38ff9700 10 client.26657 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b38ff9700 1 -- 172.21.15.31:0/3658405544 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564119c70d70 con 0x564119cb7520 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b38ff9700 10 client.26657 handle_command_reply: tid=0 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 client.26657 shutdown 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 msgr2=0x564119cabcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 0x564119cabcb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1b40012da0 tx=0x7f1b4000f6a0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 shutdown_connections 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119cb7520 0x564119cb9930 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca32f0 0x564119cb2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca1be0 0x564119cab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b38ff9700 0 client.26657 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3658405544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca2530 0x564119cabcb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 >> 172.21.15.31:0/3658405544 conn(0x564119c973e0 msgr2=0x564119c9c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.387+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 shutdown_connections 2023-07-29T16:19:03.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.388+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3658405544 wait complete. 2023-07-29T16:19:03.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.388+0000 7f1b6afbcb80 20 client.26657 trim_cache size 0 max 16384 2023-07-29T16:19:03.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 Processor -- start 2023-07-29T16:19:03.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 -- start start 2023-07-29T16:19:03.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119cb9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119cb0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119c99480 con 0x564119ca2530 2023-07-29T16:19:03.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119c9dcc0 con 0x564119ca32f0 2023-07-29T16:19:03.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119c9e0b0 con 0x564119ca1be0 2023-07-29T16:19:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b55b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b55b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35022/0 (socket says 172.21.15.31:35022) 2023-07-29T16:19:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b55b9c700 1 -- 172.21.15.31:0/3231916846 learned_addr learned my addr 172.21.15.31:0/3231916846 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119cb0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.403+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119cb9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b55b9c700 1 -- 172.21.15.31:0/3231916846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 msgr2=0x564119cb9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119cb9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b55b9c700 1 -- 172.21.15.31:0/3231916846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 msgr2=0x564119cb0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119cb0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b55b9c700 1 -- 172.21.15.31:0/3231916846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564119a09c60 con 0x564119ca32f0 2023-07-29T16:19:03.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b5539b700 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119cb9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca36c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009550 tx=0x7f1b48028000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d955f5d5d27db4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b3bfff700 1 -- 172.21.15.31:0/3231916846 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48031070 con 0x564119ca32f0 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b3bfff700 1 -- 172.21.15.31:0/3231916846 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b48011b40 con 0x564119ca32f0 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.404+0000 7f1b3bfff700 1 -- 172.21.15.31:0/3231916846 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48025070 con 0x564119ca32f0 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3231916846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 msgr2=0x564119ca36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca36c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009550 tx=0x7f1b48028000 comp rx=0 tx=0).stop 2023-07-29T16:19:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3231916846 shutdown_connections 2023-07-29T16:19:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca32f0 0x564119ca36c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca1be0 0x564119cb9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/3231916846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca2530 0x564119cb0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3231916846 >> 172.21.15.31:0/3231916846 conn(0x564119c973e0 msgr2=0x564119caa4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3231916846 shutdown_connections 2023-07-29T16:19:03.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.405+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/3231916846 wait complete. 2023-07-29T16:19:03.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 -- start start 2023-07-29T16:19:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 Processor -- start 2023-07-29T16:19:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 0x564119caee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca2530 0x564119caf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca32f0 0x564119cb00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119cb05f0 con 0x564119ca32f0 2023-07-29T16:19:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119cb0760 con 0x564119ca1be0 2023-07-29T16:19:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564119cb0920 con 0x564119ca2530 2023-07-29T16:19:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.406+0000 7f1b6afbcb80 10 client.0 _ll_register_callbacks cb 0x564119b98620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:03.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 0x564119caee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b55b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca32f0 0x564119cb00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 0x564119caee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35026/0 (socket says 172.21.15.31:35026) 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 -- 172.21.15.31:0/1373746218 learned_addr learned my addr 172.21.15.31:0/1373746218 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca2530 0x564119caf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 -- 172.21.15.31:0/1373746218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca2530 msgr2=0x564119caf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca2530 0x564119caf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 -- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca32f0 msgr2=0x564119cb00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca32f0 0x564119cb00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 -- 172.21.15.31:0/1373746218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564119a20de0 con 0x564119ca1be0 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b55b9c700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca32f0 0x564119cb00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b5539b700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 0x564119caee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0ca4a0 tx=0x7f1b4c00b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f7bba740a2df589 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b38ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b38ff9700 1 -- 172.21.15.31:0/1373746218 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c099af0 con 0x564119ca1be0 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b38ff9700 1 -- 172.21.15.31:0/1373746218 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1b4c0a2550 con 0x564119ca1be0 2023-07-29T16:19:03.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b4800f040 con 0x564119ca1be0 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.407+0000 7f1b38ff9700 1 -- 172.21.15.31:0/1373746218 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b4c09a190 con 0x564119ca1be0 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 10 client.26663 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564119a47250 con 0x564119ca1be0 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b38ff9700 1 -- 172.21.15.31:0/1373746218 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b4800f040 con 0x564119ca1be0 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b38ff9700 1 -- 172.21.15.31:0/1373746218 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f1b4c0968a0 con 0x564119ca1be0 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 10 client.26663 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 10 client.26663 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:19:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 20 client.26663 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119d09bf0 0x564119d0bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 4 client.26663 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564119bebd40 con 0x564119d09bf0 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.408+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119d09bf0 0x564119d0bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b54b9a700 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119d09bf0 0x564119d0bfd0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2cf449de006aa595 server_cookie=c57f65f94805400c in_seq=0 out_seq=0 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b38ff9700 10 client.26663 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:03.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b38ff9700 1 -- 172.21.15.31:0/1373746218 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564119bebd40 con 0x564119d09bf0 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b38ff9700 10 client.26663 handle_command_reply: tid=0 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 client.26663 shutdown 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 msgr2=0x564119caee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 0x564119caee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0ca4a0 tx=0x7f1b4c00b9d0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 shutdown_connections 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564119d09bf0 0x564119d0bfd0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564119ca32f0 0x564119cb00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564119ca1be0 0x564119caee40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b38ff9700 0 client.26663 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:03.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 --2- 172.21.15.31:0/1373746218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564119ca2530 0x564119caf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 >> 172.21.15.31:0/1373746218 conn(0x564119c973e0 msgr2=0x564119ca6150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 shutdown_connections 2023-07-29T16:19:03.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.409+0000 7f1b6afbcb80 1 -- 172.21.15.31:0/1373746218 wait complete. 2023-07-29T16:19:03.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b6afbcb80 20 client.26663 trim_cache size 0 max 16384 2023-07-29T16:19:03.420 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:19:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b3c0a0790 msgr2=0x7f1b3c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b3c0a0790 0x7f1b3c0a2c40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1b4801d000 tx=0x7f1b48007ef0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 msgr2=0x7f1b50074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 0x7f1b50074b00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c005e90 tx=0x7f1b4c00b730 comp rx=0 tx=0).stop 2023-07-29T16:19:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 shutdown_connections 2023-07-29T16:19:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b3c0a0790 0x7f1b3c0a2c40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b50101eb0 0x7f1b50073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b50100330 0x7f1b50074b00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 --2- 172.21.15.31:0/3708763454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b50101520 0x7f1b50073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 >> 172.21.15.31:0/3708763454 conn(0x7f1b500fb970 msgr2=0x7f1b500fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.410+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 shutdown_connections 2023-07-29T16:19:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.411+0000 7f1b575ff700 1 -- 172.21.15.31:0/3708763454 wait complete. 2023-07-29T16:19:03.429 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:19:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 -- 172.21.15.31:0/4094318352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440100330 msgr2=0x7f9440100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 --2- 172.21.15.31:0/4094318352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440100330 0x7f9440100740 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f94300098f0 tx=0x7f9430009c00 comp rx=0 tx=0).stop 2023-07-29T16:19:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 -- 172.21.15.31:0/4094318352 shutdown_connections 2023-07-29T16:19:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 --2- 172.21.15.31:0/4094318352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9440101eb0 0x7f9440108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 --2- 172.21.15.31:0/4094318352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440100330 0x7f9440100740 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 --2- 172.21.15.31:0/4094318352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101520 0x7f9440101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 -- 172.21.15.31:0/4094318352 >> 172.21.15.31:0/4094318352 conn(0x7f94400fb970 msgr2=0x7f94400fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 -- 172.21.15.31:0/4094318352 shutdown_connections 2023-07-29T16:19:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.654+0000 7f94449ff700 1 -- 172.21.15.31:0/4094318352 wait complete. 2023-07-29T16:19:03.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.655+0000 7f94449ff700 1 Processor -- start 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.655+0000 7f94449ff700 1 -- start start 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f94449ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9440100330 0x7f94401982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f94449ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440101520 0x7f9440198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f94449ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 0x7f944019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f94449ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944019fe50 con 0x7f9440100330 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f94449ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944019ff90 con 0x7f9440101eb0 2023-07-29T16:19:03.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f94449ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94401a0120 con 0x7f9440101520 2023-07-29T16:19:03.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f943ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 0x7f944019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f943e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9440100330 0x7f94401982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f943ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 0x7f944019f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35056/0 (socket says 172.21.15.31:35056) 2023-07-29T16:19:03.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f943ed9d700 1 -- 172.21.15.31:0/569472937 learned_addr learned my addr 172.21.15.31:0/569472937 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.656+0000 7f943dd9b700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440101520 0x7f9440198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943ed9d700 1 -- 172.21.15.31:0/569472937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440101520 msgr2=0x7f9440198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943ed9d700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440101520 0x7f9440198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943ed9d700 1 -- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9440100330 msgr2=0x7f94401982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943ed9d700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9440100330 0x7f94401982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943ed9d700 1 -- 172.21.15.31:0/569472937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340095f0 con 0x7f9440101eb0 2023-07-29T16:19:03.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943dd9b700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440101520 0x7f9440198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:03.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f943ed9d700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 0x7f944019f880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f943800ef30 tx=0x7f943800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f942f7fe700 1 -- 172.21.15.31:0/569472937 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438018070 con 0x7f9440101eb0 2023-07-29T16:19:03.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f942f7fe700 1 -- 172.21.15.31:0/569472937 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f943800f040 con 0x7f9440101eb0 2023-07-29T16:19:03.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f942f7fe700 1 -- 172.21.15.31:0/569472937 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943800cd00 con 0x7f9440101eb0 2023-07-29T16:19:03.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9430009520 con 0x7f9440101eb0 2023-07-29T16:19:03.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.657+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94401a06d0 con 0x7f9440101eb0 2023-07-29T16:19:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.661+0000 7f942f7fe700 1 -- 172.21.15.31:0/569472937 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9438003a00 con 0x7f9440101eb0 2023-07-29T16:19:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 Processor -- start 2023-07-29T16:19:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 -- start start 2023-07-29T16:19:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 0x5649abfceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcf530 0x5649abfcfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abf26a10 con 0x5649abfcf530 2023-07-29T16:19:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abf243d0 con 0x5649abfcebe0 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f9458291b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abf1fcc0 con 0x5649abfd02f0 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f942f7fe700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94280a0810 0x7f94280a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.662+0000 7f943dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 0x5649abfceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 0x5649abfceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35058/0 (socket says 172.21.15.31:35058) 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 -- 172.21.15.31:0/3201047347 learned_addr learned my addr 172.21.15.31:0/3201047347 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943e59c700 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcf530 0x5649abfcfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f942f7fe700 1 -- 172.21.15.31:0/569472937 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f94380cb5a0 con 0x7f9440101eb0 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 -- 172.21.15.31:0/3201047347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 msgr2=0x5649abfd0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 -- 172.21.15.31:0/3201047347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcf530 msgr2=0x5649abfcfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcf530 0x5649abfcfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943dd9b700 1 -- 172.21.15.31:0/3201047347 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649abd4dde0 con 0x5649abfcebe0 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.663+0000 7f943ed9d700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94280a0810 0x7f94280a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.664+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 0x5649abfceff0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9434011e20 tx=0x7f9434009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bac5b8792a9194b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.664+0000 7f942cff9700 1 -- 172.21.15.31:0/3201047347 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943400f9b0 con 0x5649abfcebe0 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.664+0000 7f942cff9700 1 -- 172.21.15.31:0/3201047347 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9434004030 con 0x5649abfcebe0 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.664+0000 7f942cff9700 1 -- 172.21.15.31:0/3201047347 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434004ba0 con 0x5649abfcebe0 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 -- 172.21.15.31:0/3201047347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 msgr2=0x5649abfceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f943ed9d700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94280a0810 0x7f94280a2cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f943801c040 tx=0x7f94380092e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 0x5649abfceff0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9434011e20 tx=0x7f9434009ed0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 -- 172.21.15.31:0/3201047347 shutdown_connections 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcebe0 0x5649abfceff0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 --2- 172.21.15.31:0/3201047347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcf530 0x5649abfcfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 -- 172.21.15.31:0/3201047347 >> 172.21.15.31:0/3201047347 conn(0x5649abfc43e0 msgr2=0x5649abfc9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 -- 172.21.15.31:0/3201047347 shutdown_connections 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 -- 172.21.15.31:0/3201047347 wait complete. 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.665+0000 7f9458291b80 1 -- start start 2023-07-29T16:19:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.666+0000 7f9458291b80 1 Processor -- start 2023-07-29T16:19:03.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.666+0000 7f9458291b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcebe0 0x5649abfd8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f9458291b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfd8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f9458291b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfd02f0 0x5649abfdfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f9458291b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abebc510 con 0x5649abfd02f0 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f9458291b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abebc1b0 con 0x5649abfcf530 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f9458291b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abda3920 con 0x5649abfcebe0 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f9458291b80 10 client.0 _ll_register_callbacks cb 0x5649abec5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcebe0 0x5649abfd8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfd8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcebe0 0x5649abfd8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41458/0 (socket says 172.21.15.31:41458) 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfd8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35066/0 (socket says 172.21.15.31:35066) 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943dd9b700 1 -- 172.21.15.31:0/3559264095 learned_addr learned my addr 172.21.15.31:0/3559264095 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfd02f0 0x5649abfdfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 -- 172.21.15.31:0/3559264095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcebe0 msgr2=0x5649abfd8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcebe0 0x5649abfd8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 -- 172.21.15.31:0/3559264095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfd02f0 msgr2=0x5649abfdfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfd02f0 0x5649abfdfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.667+0000 7f943e59c700 1 -- 172.21.15.31:0/3559264095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9434019040 con 0x5649abfcf530 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f943e59c700 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfd8d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9430000c00 tx=0x7f943000b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0a5944caac3bc2e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9425ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9425ffb700 1 -- 172.21.15.31:0/3559264095 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430027070 con 0x5649abfcf530 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9425ffb700 1 -- 172.21.15.31:0/3559264095 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f943000bb10 con 0x5649abfcf530 2023-07-29T16:19:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649abf4ad10 con 0x5649abfcf530 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9425ffb700 1 -- 172.21.15.31:0/3559264095 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9430005040 con 0x5649abfcf530 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9458291b80 10 client.26675 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.668+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5649abd36c60 con 0x5649abfcf530 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9425ffb700 1 -- 172.21.15.31:0/3559264095 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9430031020 con 0x5649abfcf530 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9425ffb700 1 -- 172.21.15.31:0/3559264095 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f943002a030 con 0x5649abfcf530 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9458291b80 10 client.26675 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9458291b80 10 client.26675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9458291b80 20 client.26675 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9458291b80 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649abfe4680 0x5649abfe6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9458291b80 4 client.26675 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649abf9dd70 con 0x5649abfe4680 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.669+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649abfe4680 0x5649abfe6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.670+0000 7f943dd9b700 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649abfe4680 0x5649abfe6a90 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed9234f27d86260 server_cookie=c96798cc3b25ee0c in_seq=0 out_seq=0 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.670+0000 7f9425ffb700 10 client.26675 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9425ffb700 1 -- 172.21.15.31:0/3559264095 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5649abf9dd70 con 0x5649abfe4680 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9425ffb700 10 client.26675 handle_command_reply: tid=0 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 client.26675 shutdown 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 msgr2=0x5649abfd8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfd8d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9430000c00 tx=0x7f943000b790 comp rx=0 tx=0).stop 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 shutdown_connections 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649abfe4680 0x5649abfe6a90 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfd02f0 0x5649abfdfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcebe0 0x5649abfd8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 --2- 172.21.15.31:0/3559264095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfd8d80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9425ffb700 0 client.26675 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 >> 172.21.15.31:0/3559264095 conn(0x5649abfc43e0 msgr2=0x5649abfc9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 shutdown_connections 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 1 -- 172.21.15.31:0/3559264095 wait complete. 2023-07-29T16:19:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.671+0000 7f9458291b80 20 client.26675 trim_cache size 0 max 16384 2023-07-29T16:19:03.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3429330134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:19:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 Processor -- start 2023-07-29T16:19:03.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 -- start start 2023-07-29T16:19:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfd06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfdfd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abfd64f0 con 0x5649abfcebe0 2023-07-29T16:19:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abfcb320 con 0x5649abfcf530 2023-07-29T16:19:03.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.686+0000 7f9458291b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abfeb640 con 0x5649abfd02f0 2023-07-29T16:19:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfdfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfdfd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35076/0 (socket says 172.21.15.31:35076) 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 -- 172.21.15.31:0/456178124 learned_addr learned my addr 172.21.15.31:0/456178124 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943dd9b700 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfd06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 -- 172.21.15.31:0/456178124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 msgr2=0x5649abfd6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 -- 172.21.15.31:0/456178124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 msgr2=0x5649abfd06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfd06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 -- 172.21.15.31:0/456178124 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649abd74250 con 0x5649abfcf530 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943dd9b700 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfd06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f943e59c700 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfdfd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9430005e90 tx=0x7f9430009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ca0b03d80a4d67b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f942d7fa700 1 -- 172.21.15.31:0/456178124 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430027070 con 0x5649abfcf530 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.687+0000 7f942d7fa700 1 -- 172.21.15.31:0/456178124 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9430031020 con 0x5649abfcf530 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f942d7fa700 1 -- 172.21.15.31:0/456178124 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94300038e0 con 0x5649abfcf530 2023-07-29T16:19:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 -- 172.21.15.31:0/456178124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 msgr2=0x5649abfdfd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfdfd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9430005e90 tx=0x7f9430009fd0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 -- 172.21.15.31:0/456178124 shutdown_connections 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfd02f0 0x5649abfd6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfd06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 --2- 172.21.15.31:0/456178124 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfcf530 0x5649abfdfd80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 -- 172.21.15.31:0/456178124 >> 172.21.15.31:0/456178124 conn(0x5649abfc43e0 msgr2=0x5649abfd7510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 -- 172.21.15.31:0/456178124 shutdown_connections 2023-07-29T16:19:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.688+0000 7f9458291b80 1 -- 172.21.15.31:0/456178124 wait complete. 2023-07-29T16:19:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.689+0000 7f9458291b80 1 -- start start 2023-07-29T16:19:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.689+0000 7f9458291b80 1 Processor -- start 2023-07-29T16:19:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfdfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 0x5649abfdc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfd02f0 0x5649abfdca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abfe0250 con 0x5649abfcebe0 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abfdcf50 con 0x5649abfd02f0 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649abfdd110 con 0x5649abfcf530 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f9458291b80 10 client.0 _ll_register_callbacks cb 0x5649abec5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:03.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f943e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 0x5649abfdc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f943e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 0x5649abfdc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41470/0 (socket says 172.21.15.31:41470) 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f943e59c700 1 -- 172.21.15.31:0/4220603994 learned_addr learned my addr 172.21.15.31:0/4220603994 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f943dd9b700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfdfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.690+0000 7f943dd9b700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfd02f0 0x5649abfdca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f943e59c700 1 -- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfd02f0 msgr2=0x5649abfdca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f943e59c700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfd02f0 0x5649abfdca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f943e59c700 1 -- 172.21.15.31:0/4220603994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 msgr2=0x5649abfdfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f943e59c700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfdfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f943e59c700 1 -- 172.21.15.31:0/4220603994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649abd36c60 con 0x5649abfcf530 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f943e59c700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 0x5649abfdc4d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f943000fa40 tx=0x7f9430009480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=381a639e389ba42a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9425ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9425ffb700 1 -- 172.21.15.31:0/4220603994 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430027070 con 0x5649abfcf530 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9425ffb700 1 -- 172.21.15.31:0/4220603994 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9430000e40 con 0x5649abfcf530 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9434004030 con 0x5649abfcf530 2023-07-29T16:19:03.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9425ffb700 1 -- 172.21.15.31:0/4220603994 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f943000bb10 con 0x5649abfcf530 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9458291b80 10 client.26698 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.691+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5649abd4c2b0 con 0x5649abfcf530 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9425ffb700 1 -- 172.21.15.31:0/4220603994 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9434004030 con 0x5649abfcf530 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9425ffb700 1 -- 172.21.15.31:0/4220603994 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9430009c70 con 0x5649abfcf530 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9458291b80 10 client.26698 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9458291b80 10 client.26698 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9458291b80 20 client.26698 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:03.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9458291b80 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649ac037700 0x5649ac039b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9458291b80 4 client.26698 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5649abf18d40 con 0x5649ac037700 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.692+0000 7f943dd9b700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649ac037700 0x5649ac039b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f943dd9b700 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649ac037700 0x5649ac039b10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40df903ba76f6cd7 server_cookie=c42744b2cb848cca in_seq=0 out_seq=0 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9425ffb700 10 client.26698 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9425ffb700 1 -- 172.21.15.31:0/4220603994 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5649abf18d40 con 0x5649ac037700 2023-07-29T16:19:03.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9425ffb700 10 client.26698 handle_command_reply: tid=0 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 client.26698 shutdown 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 msgr2=0x5649abfdc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 0x5649abfdc4d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f943000fa40 tx=0x7f9430009480 comp rx=0 tx=0).stop 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 shutdown_connections 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5649ac037700 0x5649ac039b10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649abfd02f0 0x5649abfdca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5649abfcebe0 0x5649abfdfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9425ffb700 0 client.26698 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 --2- 172.21.15.31:0/4220603994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649abfcf530 0x5649abfdc4d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 >> 172.21.15.31:0/4220603994 conn(0x5649abfc43e0 msgr2=0x5649abfd6740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 shutdown_connections 2023-07-29T16:19:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 1 -- 172.21.15.31:0/4220603994 wait complete. 2023-07-29T16:19:03.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.693+0000 7f9458291b80 20 client.26698 trim_cache size 0 max 16384 2023-07-29T16:19:03.700 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:19:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94280a0810 msgr2=0x7f94280a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94280a0810 0x7f94280a2cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f943801c040 tx=0x7f94380092e0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 msgr2=0x7f944019f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:03.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 0x7f944019f880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f943800ef30 tx=0x7f943800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 shutdown_connections 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94280a0810 0x7f94280a2cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9440101eb0 0x7f944019f880 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9440100330 0x7f94401982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 --2- 172.21.15.31:0/569472937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9440101520 0x7f9440198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 >> 172.21.15.31:0/569472937 conn(0x7f94400fb970 msgr2=0x7f9440106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:03.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 shutdown_connections 2023-07-29T16:19:03.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:03.694+0000 7f94449ff700 1 -- 172.21.15.31:0/569472937 wait complete. 2023-07-29T16:19:03.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3429330134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:19:03.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3429330134' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:19:03.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 81460 9.50 MB/sec warmup 48 sec latency 13.016 ms 2023-07-29T16:19:04.713 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:19:04.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:19:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:04 smithi039 ceph-mon[128858]: pgmap v478: 97 pgs: 97 active+clean; 34 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 353 op/s 2023-07-29T16:19:04.779 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:04 smithi031 ceph-mon[95504]: pgmap v478: 97 pgs: 97 active+clean; 34 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 353 op/s 2023-07-29T16:19:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:04 smithi156 ceph-mon[129245]: pgmap v478: 97 pgs: 97 active+clean; 34 MiB data, 563 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.7 MiB/s wr, 353 op/s 2023-07-29T16:19:04.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 82587 9.42 MB/sec warmup 49 sec latency 17.970 ms 2023-07-29T16:19:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 -- 172.21.15.31:0/4211791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090100e00 msgr2=0x7fe0901031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 --2- 172.21.15.31:0/4211791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090100e00 0x7fe0901031e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe084009990 tx=0x7fe084009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 -- 172.21.15.31:0/4211791923 shutdown_connections 2023-07-29T16:19:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 --2- 172.21.15.31:0/4211791923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe090106040 0x7fe090108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 --2- 172.21.15.31:0/4211791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090100e00 0x7fe0901031e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 --2- 172.21.15.31:0/4211791923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090103720 0x7fe090105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 -- 172.21.15.31:0/4211791923 >> 172.21.15.31:0/4211791923 conn(0x7fe0900fa840 msgr2=0x7fe0900fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 -- 172.21.15.31:0/4211791923 shutdown_connections 2023-07-29T16:19:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.936+0000 7fe096bff700 1 -- 172.21.15.31:0/4211791923 wait complete. 2023-07-29T16:19:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.937+0000 7fe096bff700 1 Processor -- start 2023-07-29T16:19:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.937+0000 7fe096bff700 1 -- start start 2023-07-29T16:19:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe096bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe090100e00 0x7fe0901982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe096bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 0x7fe0901987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe096bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090106040 0x7fe09019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe096bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09019fe20 con 0x7fe090106040 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe096bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09019ff60 con 0x7fe090103720 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe096bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0901a00f0 con 0x7fe090100e00 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 0x7fe0901987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe09499b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe090100e00 0x7fe0901982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 0x7fe0901987e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35096/0 (socket says 172.21.15.31:35096) 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 -- 172.21.15.31:0/3667078133 learned_addr learned my addr 172.21.15.31:0/3667078133 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe09519c700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090106040 0x7fe09019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 -- 172.21.15.31:0/3667078133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe090100e00 msgr2=0x7fe0901982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe090100e00 0x7fe0901982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 -- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090106040 msgr2=0x7fe09019f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090106040 0x7fe09019f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe08ffff700 1 -- 172.21.15.31:0/3667078133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840095f0 con 0x7fe090103720 2023-07-29T16:19:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.938+0000 7fe09519c700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090106040 0x7fe09019f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.939+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 0x7fe0901987e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00e9a0 tx=0x7fe07c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.939+0000 7fe08dffb700 1 -- 172.21.15.31:0/3667078133 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c009790 con 0x7fe090103720 2023-07-29T16:19:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.939+0000 7fe08dffb700 1 -- 172.21.15.31:0/3667078133 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe07c00cbd0 con 0x7fe090103720 2023-07-29T16:19:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.939+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0901a03d0 con 0x7fe090103720 2023-07-29T16:19:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.939+0000 7fe08dffb700 1 -- 172.21.15.31:0/3667078133 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c018680 con 0x7fe090103720 2023-07-29T16:19:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.939+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0901a08e0 con 0x7fe090103720 2023-07-29T16:19:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.943+0000 7fe08dffb700 1 -- 172.21.15.31:0/3667078133 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe07c00cd40 con 0x7fe090103720 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.944+0000 7fe0aa4d8b80 1 Processor -- start 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.944+0000 7fe0aa4d8b80 1 -- start start 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.944+0000 7fe08dffb700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe0800a0790 0x7fe0800a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08dffb700 1 -- 172.21.15.31:0/3667078133 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe07c014070 con 0x7fe090103720 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a09c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a09cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a09d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558109ff3a90 con 0x55810a09bc60 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558109ff1450 con 0x55810a09c5b0 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558109fecd40 con 0x55810a09d370 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe09519c700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe0800a0790 0x7fe0800a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a09cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a09cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35098/0 (socket says 172.21.15.31:35098) 2023-07-29T16:19:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 -- 172.21.15.31:0/3887704804 learned_addr learned my addr 172.21.15.31:0/3887704804 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe09499b700 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a09c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 -- 172.21.15.31:0/3887704804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 msgr2=0x55810a09d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a09d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 -- 172.21.15.31:0/3887704804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 msgr2=0x55810a09c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a09c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe08ffff700 1 -- 172.21.15.31:0/3887704804 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558109e1ae60 con 0x55810a09c5b0 2023-07-29T16:19:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.945+0000 7fe09499b700 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a09c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.946+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a09cde0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe07c095a10 tx=0x7fe07c0990d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbd00081ba66801 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.946+0000 7fe09519c700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe0800a0790 0x7fe0800a2c40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe0880093c0 tx=0x7fe088009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.946+0000 7fe07affd700 1 -- 172.21.15.31:0/3887704804 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c09d540 con 0x55810a09c5b0 2023-07-29T16:19:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.946+0000 7fe07affd700 1 -- 172.21.15.31:0/3887704804 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe07c0a1960 con 0x55810a09c5b0 2023-07-29T16:19:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.946+0000 7fe07affd700 1 -- 172.21.15.31:0/3887704804 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c098c60 con 0x55810a09c5b0 2023-07-29T16:19:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3887704804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 msgr2=0x55810a09cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a09cde0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe07c095a10 tx=0x7fe07c0990d0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3887704804 shutdown_connections 2023-07-29T16:19:04.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a09d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a09c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3887704804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a09cde0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3887704804 >> 172.21.15.31:0/3887704804 conn(0x55810a091460 msgr2=0x55810a096a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:04.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3887704804 shutdown_connections 2023-07-29T16:19:04.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3887704804 wait complete. 2023-07-29T16:19:04.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.947+0000 7fe0aa4d8b80 1 -- start start 2023-07-29T16:19:04.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.948+0000 7fe0aa4d8b80 1 Processor -- start 2023-07-29T16:19:04.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 0x55810a0a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09c5b0 0x55810a0a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a0acd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558109f89590 con 0x55810a09c5b0 2023-07-29T16:19:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558109f89230 con 0x55810a09bc60 2023-07-29T16:19:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558109e709a0 con 0x55810a09d370 2023-07-29T16:19:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe0aa4d8b80 10 client.0 _ll_register_callbacks cb 0x558109f926a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 0x55810a0a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 0x55810a0a57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35106/0 (socket says 172.21.15.31:35106) 2023-07-29T16:19:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 -- 172.21.15.31:0/3776956750 learned_addr learned my addr 172.21.15.31:0/3776956750 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09c5b0 0x55810a0a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09519c700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a0acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 -- 172.21.15.31:0/3776956750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 msgr2=0x55810a0acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a0acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 -- 172.21.15.31:0/3776956750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09c5b0 msgr2=0x55810a0a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09c5b0 0x55810a0a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09499b700 1 -- 172.21.15.31:0/3776956750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0901a08e0 con 0x55810a09bc60 2023-07-29T16:19:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.949+0000 7fe09519c700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe09499b700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 0x55810a0a57e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe084009990 tx=0x7fe08400bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d741a837d212b57a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe06ffff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe06ffff700 1 -- 172.21.15.31:0/3776956750 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084028070 con 0x55810a09bc60 2023-07-29T16:19:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55810a017d90 con 0x55810a09bc60 2023-07-29T16:19:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe06ffff700 1 -- 172.21.15.31:0/3776956750 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe084004500 con 0x55810a09bc60 2023-07-29T16:19:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe06ffff700 1 -- 172.21.15.31:0/3776956750 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe08401d070 con 0x55810a09bc60 2023-07-29T16:19:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe0aa4d8b80 10 client.26693 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.950+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558109e03ce0 con 0x55810a09bc60 2023-07-29T16:19:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe06ffff700 1 -- 172.21.15.31:0/3776956750 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe084032020 con 0x55810a09bc60 2023-07-29T16:19:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe06ffff700 1 -- 172.21.15.31:0/3776956750 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fe08402b030 con 0x55810a09bc60 2023-07-29T16:19:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe0aa4d8b80 10 client.26693 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe0aa4d8b80 10 client.26693 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe0aa4d8b80 20 client.26693 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:04.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a0b1610 0x55810a0b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe0aa4d8b80 4 client.26693 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55810a06adf0 con 0x55810a0b1610 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a0b1610 0x55810a0b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe06ffff700 10 client.26693 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.951+0000 7fe08ffff700 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a0b1610 0x55810a0b3a20 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e743c8da8984711 server_cookie=6ad7f618655d626c in_seq=0 out_seq=0 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe06ffff700 1 -- 172.21.15.31:0/3776956750 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55810a06adf0 con 0x55810a0b1610 2023-07-29T16:19:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe06ffff700 10 client.26693 handle_command_reply: tid=0 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 client.26693 shutdown 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 msgr2=0x55810a0a57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 0x55810a0a57e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe084009990 tx=0x7fe08400bf60 comp rx=0 tx=0).stop 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 shutdown_connections 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a0b1610 0x55810a0b3a20 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09bc60 0x55810a0a57e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/3776956750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09c5b0 0x55810a0a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe06ffff700 0 client.26693 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 >> 172.21.15.31:0/3776956750 conn(0x55810a091460 msgr2=0x55810a0960e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 shutdown_connections 2023-07-29T16:19:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/3776956750 wait complete. 2023-07-29T16:19:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.954+0000 7fe0aa4d8b80 20 client.26693 trim_cache size 0 max 16384 2023-07-29T16:19:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.972+0000 7fe0aa4d8b80 1 Processor -- start 2023-07-29T16:19:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.972+0000 7fe0aa4d8b80 1 -- start start 2023-07-29T16:19:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.972+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0aad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.972+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09c5b0 0x55810a0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.972+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09d370 0x55810a0a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.972+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55810a0a4c90 con 0x55810a09bc60 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55810a0ae9d0 con 0x55810a09d370 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55810a0983c0 con 0x55810a09c5b0 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0aad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46410/0 (socket says 172.21.15.31:46410) 2023-07-29T16:19:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 -- 172.21.15.31:0/1955864573 learned_addr learned my addr 172.21.15.31:0/1955864573 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09519c700 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09d370 0x55810a0a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe08ffff700 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09c5b0 0x55810a0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 -- 172.21.15.31:0/1955864573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09c5b0 msgr2=0x55810a0a3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09c5b0 0x55810a0a3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 -- 172.21.15.31:0/1955864573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09d370 msgr2=0x55810a0a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09d370 0x55810a0a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 -- 172.21.15.31:0/1955864573 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55810a0963b0 con 0x55810a09bc60 2023-07-29T16:19:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09519c700 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09d370 0x55810a0a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.973+0000 7fe09499b700 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0aad10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098e0 tx=0x7fe08400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611f457293955677 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.974+0000 7fe07b7fe700 1 -- 172.21.15.31:0/1955864573 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe084028040 con 0x55810a09bc60 2023-07-29T16:19:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.974+0000 7fe07b7fe700 1 -- 172.21.15.31:0/1955864573 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe084032030 con 0x55810a09bc60 2023-07-29T16:19:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.974+0000 7fe07b7fe700 1 -- 172.21.15.31:0/1955864573 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08401d040 con 0x55810a09bc60 2023-07-29T16:19:04.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.974+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/1955864573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 msgr2=0x55810a0aad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.974+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0aad10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098e0 tx=0x7fe08400bfd0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/1955864573 shutdown_connections 2023-07-29T16:19:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09d370 0x55810a0a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0aad10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/1955864573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09c5b0 0x55810a0a3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/1955864573 >> 172.21.15.31:0/1955864573 conn(0x55810a091460 msgr2=0x55810a096810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:04.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/1955864573 shutdown_connections 2023-07-29T16:19:04.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/1955864573 wait complete. 2023-07-29T16:19:04.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 -- start start 2023-07-29T16:19:04.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.975+0000 7fe0aa4d8b80 1 Processor -- start 2023-07-29T16:19:04.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a0aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a1032c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55810a103800 con 0x55810a09bc60 2023-07-29T16:19:04.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55810a103970 con 0x55810a09c5b0 2023-07-29T16:19:04.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55810a103b30 con 0x55810a09d370 2023-07-29T16:19:04.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe0aa4d8b80 10 client.0 _ll_register_callbacks cb 0x558109f926a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a0aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a0aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35130/0 (socket says 172.21.15.31:35130) 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 -- 172.21.15.31:0/586352331 learned_addr learned my addr 172.21.15.31:0/586352331 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe09499b700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe09519c700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a1032c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 -- 172.21.15.31:0/586352331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 msgr2=0x55810a1032c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a1032c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 -- 172.21.15.31:0/586352331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 msgr2=0x55810a0a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.976+0000 7fe08ffff700 1 -- 172.21.15.31:0/586352331 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe088007e40 con 0x55810a09c5b0 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe08ffff700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a0aa3b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0c9c20 tx=0x7fe07c010aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28d3142e50580c78 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe06ffff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe06ffff700 1 -- 172.21.15.31:0/586352331 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0b7040 con 0x55810a09c5b0 2023-07-29T16:19:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe07c0a1960 con 0x55810a09c5b0 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe06ffff700 1 -- 172.21.15.31:0/586352331 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe07c0c8040 con 0x55810a09c5b0 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe06ffff700 1 -- 172.21.15.31:0/586352331 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe07c09d170 con 0x55810a09c5b0 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe0aa4d8b80 10 client.26699 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558109e03ce0 con 0x55810a09c5b0 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe06ffff700 1 -- 172.21.15.31:0/586352331 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe07c0a1960 con 0x55810a09c5b0 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe06ffff700 1 -- 172.21.15.31:0/586352331 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fe07c0a1560 con 0x55810a09c5b0 2023-07-29T16:19:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe0aa4d8b80 10 client.26699 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:04.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe0aa4d8b80 10 client.26699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:04.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.977+0000 7fe0aa4d8b80 20 client.26699 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:04.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.978+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a105160 0x55810a107570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:04.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.978+0000 7fe0aa4d8b80 4 client.26699 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:19:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.978+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558109fe5dc0 con 0x55810a105160 2023-07-29T16:19:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.978+0000 7fe09499b700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a105160 0x55810a107570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:04.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.978+0000 7fe09499b700 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a105160 0x55810a107570 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e80fd3412db113f6 server_cookie=e64863f59c1743d0 in_seq=0 out_seq=0 2023-07-29T16:19:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.978+0000 7fe06ffff700 10 client.26699 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe06ffff700 1 -- 172.21.15.31:0/586352331 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558109fe5dc0 con 0x55810a105160 2023-07-29T16:19:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe06ffff700 10 client.26699 handle_command_reply: tid=0 2023-07-29T16:19:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 client.26699 shutdown 2023-07-29T16:19:04.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 msgr2=0x55810a0aa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a0aa3b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0c9c20 tx=0x7fe07c010aa0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 shutdown_connections 2023-07-29T16:19:04.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55810a105160 0x55810a107570 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55810a09d370 0x55810a1032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55810a09bc60 0x55810a0a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 --2- 172.21.15.31:0/586352331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55810a09c5b0 0x55810a0aa3b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 >> 172.21.15.31:0/586352331 conn(0x55810a091460 msgr2=0x55810a0a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:04.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe06ffff700 0 client.26699 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:04.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 shutdown_connections 2023-07-29T16:19:04.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.979+0000 7fe0aa4d8b80 1 -- 172.21.15.31:0/586352331 wait complete. 2023-07-29T16:19:04.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe0aa4d8b80 20 client.26699 trim_cache size 0 max 16384 2023-07-29T16:19:04.993 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:19:04.994 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:19:04.994 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "b8b6a2f4-e3d4-4054-b522-b3af00e74b57", 2023-07-29T16:19:04.994 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:19:04.994 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:19:04.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe0800a0790 msgr2=0x7fe0800a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe0800a0790 0x7fe0800a2c40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fe0880093c0 tx=0x7fe088009920 comp rx=0 tx=0).stop 2023-07-29T16:19:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 msgr2=0x7fe0901987e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 0x7fe0901987e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00e9a0 tx=0x7fe07c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 shutdown_connections 2023-07-29T16:19:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe0800a0790 0x7fe0800a2c40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe090106040 0x7fe09019f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe090100e00 0x7fe0901982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 --2- 172.21.15.31:0/3667078133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe090103720 0x7fe0901987e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:04.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 >> 172.21.15.31:0/3667078133 conn(0x7fe0900fa840 msgr2=0x7fe0900ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:04.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.980+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 shutdown_connections 2023-07-29T16:19:04.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:04.981+0000 7fe096bff700 1 -- 172.21.15.31:0/3667078133 wait complete. 2023-07-29T16:19:05.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:19:05.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 -- 172.21.15.31:0/790634699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 msgr2=0x7f18f0108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 --2- 172.21.15.31:0/790634699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f0108480 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f90 tx=0x7f18e800b330 comp rx=0 tx=0).stop 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 -- 172.21.15.31:0/790634699 shutdown_connections 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 --2- 172.21.15.31:0/790634699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f0108480 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 --2- 172.21.15.31:0/790634699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f01031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 --2- 172.21.15.31:0/790634699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 0x7f18f0105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 -- 172.21.15.31:0/790634699 >> 172.21.15.31:0/790634699 conn(0x7f18f00fa860 msgr2=0x7f18f00fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:05.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 -- 172.21.15.31:0/790634699 shutdown_connections 2023-07-29T16:19:05.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.238+0000 7f18f69ff700 1 -- 172.21.15.31:0/790634699 wait complete. 2023-07-29T16:19:05.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.239+0000 7f18f69ff700 1 Processor -- start 2023-07-29T16:19:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.239+0000 7f18f69ff700 1 -- start start 2023-07-29T16:19:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f69ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f0194c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f69ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 0x7f18f0195150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f69ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f01917a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f69ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0191f40 con 0x7f18f0106020 2023-07-29T16:19:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f69ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f01920b0 con 0x7f18f0103700 2023-07-29T16:19:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f69ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0192270 con 0x7f18f0100de0 2023-07-29T16:19:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f0194c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18ef7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 0x7f18f0195150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f0194c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41564/0 (socket says 172.21.15.31:41564) 2023-07-29T16:19:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 -- 172.21.15.31:0/1451742372 learned_addr learned my addr 172.21.15.31:0/1451742372 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18f4f9c700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f01917a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 -- 172.21.15.31:0/1451742372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 msgr2=0x7f18f0195150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 0x7f18f0195150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 -- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 msgr2=0x7f18f01917a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f01917a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.240+0000 7f18effff700 1 -- 172.21.15.31:0/1451742372 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e8009c40 con 0x7f18f0100de0 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 0x7f18f0195150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18f4f9c700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f01917a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18effff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f0194c10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f18e000e9a0 tx=0x7f18e000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18ed7fa700 1 -- 172.21.15.31:0/1451742372 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e0009790 con 0x7f18f0100de0 2023-07-29T16:19:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18ed7fa700 1 -- 172.21.15.31:0/1451742372 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e000cbd0 con 0x7f18f0100de0 2023-07-29T16:19:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f01a48d0 con 0x7f18f0100de0 2023-07-29T16:19:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18ed7fa700 1 -- 172.21.15.31:0/1451742372 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e000eee0 con 0x7f18f0100de0 2023-07-29T16:19:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.241+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f01a4db0 con 0x7f18f0100de0 2023-07-29T16:19:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.246+0000 7f18ed7fa700 1 -- 172.21.15.31:0/1451742372 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f18e000cd40 con 0x7f18f0100de0 2023-07-29T16:19:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.247+0000 7f190a41cb80 1 Processor -- start 2023-07-29T16:19:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 -- start start 2023-07-29T16:19:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18ed7fa700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18d80a0790 0x7f18d80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18ed7fa700 1 -- 172.21.15.31:0/1451742372 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18e0014070 con 0x7f18f0100de0 2023-07-29T16:19:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd70bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd70cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70d2f0 0x55e3dd70d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd663a10 con 0x55e3dd70c530 2023-07-29T16:19:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd6613d0 con 0x55e3dd70d2f0 2023-07-29T16:19:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f190a41cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd65ccc0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18f4f9c700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18d80a0790 0x7f18d80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18effff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd70bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18effff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd70bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41568/0 (socket says 172.21.15.31:41568) 2023-07-29T16:19:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18effff700 1 -- 172.21.15.31:0/1369759021 learned_addr learned my addr 172.21.15.31:0/1369759021 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd70cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.248+0000 7f18f4f9c700 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70d2f0 0x55e3dd70d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18effff700 1 -- 172.21.15.31:0/1369759021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70d2f0 msgr2=0x55e3dd70d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18effff700 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70d2f0 0x55e3dd70d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18effff700 1 -- 172.21.15.31:0/1369759021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 msgr2=0x55e3dd70cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18effff700 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd70cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18effff700 1 -- 172.21.15.31:0/1369759021 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3dd48ade0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18effff700 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd70bff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f18e00a1470 tx=0x7f18e0098fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=582e89424db215c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18f4f9c700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18d80a0790 0x7f18d80a2c40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009c10 tx=0x7f18e8009ac0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18d67fc700 1 -- 172.21.15.31:0/1369759021 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00a19a0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18d67fc700 1 -- 172.21.15.31:0/1369759021 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e00978f0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.249+0000 7f18d67fc700 1 -- 172.21.15.31:0/1369759021 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00a1770 con 0x55e3dd70bbe0 2023-07-29T16:19:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 -- 172.21.15.31:0/1369759021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 msgr2=0x55e3dd70bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd70bff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f18e00a1470 tx=0x7f18e0098fb0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 -- 172.21.15.31:0/1369759021 shutdown_connections 2023-07-29T16:19:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70d2f0 0x55e3dd70d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd70bff0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1369759021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd70cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 -- 172.21.15.31:0/1369759021 >> 172.21.15.31:0/1369759021 conn(0x55e3dd7013e0 msgr2=0x55e3dd706990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 -- 172.21.15.31:0/1369759021 shutdown_connections 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.250+0000 7f190a41cb80 1 -- 172.21.15.31:0/1369759021 wait complete. 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.251+0000 7f190a41cb80 1 -- start start 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.251+0000 7f190a41cb80 1 Processor -- start 2023-07-29T16:19:05.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.251+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd726870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.251+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 0x55e3dd726db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f190a41cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd5f9510 con 0x55e3dd70d2f0 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f190a41cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd5f91b0 con 0x55e3dd70c530 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f190a41cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd4e0920 con 0x55e3dd70bbe0 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 0x55e3dd726db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f190a41cb80 10 client.0 _ll_register_callbacks cb 0x55e3dd602620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:05.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 0x55e3dd726db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35174/0 (socket says 172.21.15.31:35174) 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 -- 172.21.15.31:0/1319052488 learned_addr learned my addr 172.21.15.31:0/1319052488 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18effff700 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd726870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 -- 172.21.15.31:0/1319052488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 msgr2=0x55e3dd726870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd726870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 -- 172.21.15.31:0/1319052488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 msgr2=0x55e3dd72ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 -- 172.21.15.31:0/1319052488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e0095040 con 0x55e3dd70c530 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 0x55e3dd726db0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f18e4012c90 tx=0x7f18e4009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d134a0ba67794c7e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.252+0000 7f18cf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f18cf7fe700 1 -- 172.21.15.31:0/1319052488 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4017070 con 0x55e3dd70c530 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3dd687d10 con 0x55e3dd70c530 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f18cf7fe700 1 -- 172.21.15.31:0/1319052488 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18e4004050 con 0x55e3dd70c530 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f18cf7fe700 1 -- 172.21.15.31:0/1319052488 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e4004250 con 0x55e3dd70c530 2023-07-29T16:19:05.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f190a41cb80 10 client.26711 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e3dd473c60 con 0x55e3dd70c530 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.253+0000 7f18cf7fe700 1 -- 172.21.15.31:0/1319052488 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e3dd687d10 con 0x55e3dd70c530 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f18cf7fe700 1 -- 172.21.15.31:0/1319052488 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f18e4003810 con 0x55e3dd70c530 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f190a41cb80 10 client.26711 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f190a41cb80 10 client.26711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f190a41cb80 20 client.26711 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:05.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd7326a0 0x55e3dd734ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f190a41cb80 4 client.26711 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3dd6dad70 con 0x55e3dd7326a0 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f18effff700 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd7326a0 0x55e3dd734ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f18effff700 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd7326a0 0x55e3dd734ab0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce29796bd122c12e server_cookie=5cf72134556edca2 in_seq=0 out_seq=0 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.254+0000 7f18cf7fe700 10 client.26711 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.256+0000 7f18cf7fe700 1 -- 172.21.15.31:0/1319052488 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e3dd6dad70 con 0x55e3dd7326a0 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.256+0000 7f18cf7fe700 10 client.26711 handle_command_reply: tid=0 2023-07-29T16:19:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.256+0000 7f190a41cb80 1 client.26711 shutdown 2023-07-29T16:19:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 msgr2=0x55e3dd726db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 0x55e3dd726db0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f18e4012c90 tx=0x7f18e4009da0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 shutdown_connections 2023-07-29T16:19:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd7326a0 0x55e3dd734ab0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70bbe0 0x55e3dd726870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f18cf7fe700 0 client.26711 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1319052488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70c530 0x55e3dd726db0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 >> 172.21.15.31:0/1319052488 conn(0x55e3dd7013e0 msgr2=0x55e3dd706060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.257+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 shutdown_connections 2023-07-29T16:19:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.258+0000 7f190a41cb80 1 -- 172.21.15.31:0/1319052488 wait complete. 2023-07-29T16:19:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.258+0000 7f190a41cb80 20 client.26711 trim_cache size 0 max 16384 2023-07-29T16:19:05.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 Processor -- start 2023-07-29T16:19:05.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 -- start start 2023-07-29T16:19:05.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70c530 0x55e3dd7101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd710730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd707cc0 con 0x55e3dd70d2f0 2023-07-29T16:19:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd7279f0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f190a41cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd7274f0 con 0x55e3dd70c530 2023-07-29T16:19:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f18effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f18f4f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd710730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f18effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35180/0 (socket says 172.21.15.31:35180) 2023-07-29T16:19:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.282+0000 7f18effff700 1 -- 172.21.15.31:0/1307657970 learned_addr learned my addr 172.21.15.31:0/1307657970 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70c530 0x55e3dd7101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18effff700 1 -- 172.21.15.31:0/1307657970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70c530 msgr2=0x55e3dd7101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18effff700 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70c530 0x55e3dd7101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18effff700 1 -- 172.21.15.31:0/1307657970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 msgr2=0x55e3dd710730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18effff700 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd710730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18effff700 1 -- 172.21.15.31:0/1307657970 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3dd4b1250 con 0x55e3dd70bbe0 2023-07-29T16:19:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18effff700 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bd00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f18e0099020 tx=0x7f18e0098a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b372af81938c8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18d6ffd700 1 -- 172.21.15.31:0/1307657970 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00b5030 con 0x55e3dd70bbe0 2023-07-29T16:19:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18d6ffd700 1 -- 172.21.15.31:0/1307657970 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e00b7050 con 0x55e3dd70bbe0 2023-07-29T16:19:05.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.283+0000 7f18d6ffd700 1 -- 172.21.15.31:0/1307657970 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00a1510 con 0x55e3dd70bbe0 2023-07-29T16:19:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 -- 172.21.15.31:0/1307657970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 msgr2=0x55e3dd72bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bd00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f18e0099020 tx=0x7f18e0098a40 comp rx=0 tx=0).stop 2023-07-29T16:19:05.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 -- 172.21.15.31:0/1307657970 shutdown_connections 2023-07-29T16:19:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70d2f0 0x55e3dd710730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bd00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 --2- 172.21.15.31:0/1307657970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70c530 0x55e3dd7101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.284+0000 7f190a41cb80 1 -- 172.21.15.31:0/1307657970 >> 172.21.15.31:0/1307657970 conn(0x55e3dd7013e0 msgr2=0x55e3dd713670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:05.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.285+0000 7f190a41cb80 1 -- 172.21.15.31:0/1307657970 shutdown_connections 2023-07-29T16:19:05.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.285+0000 7f190a41cb80 1 -- 172.21.15.31:0/1307657970 wait complete. 2023-07-29T16:19:05.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.285+0000 7f190a41cb80 1 -- start start 2023-07-29T16:19:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.285+0000 7f190a41cb80 1 Processor -- start 2023-07-29T16:19:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd72a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd72ac30 con 0x55e3dd70c530 2023-07-29T16:19:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd72ad70 con 0x55e3dd70bbe0 2023-07-29T16:19:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3dd72af30 con 0x55e3dd70d2f0 2023-07-29T16:19:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f190a41cb80 10 client.0 _ll_register_callbacks cb 0x55e3dd602620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:05.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f18effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f18effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35194/0 (socket says 172.21.15.31:35194) 2023-07-29T16:19:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f18effff700 1 -- 172.21.15.31:0/3073252154 learned_addr learned my addr 172.21.15.31:0/3073252154 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd72a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.286+0000 7f18f4f9c700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18effff700 1 -- 172.21.15.31:0/3073252154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70d2f0 msgr2=0x55e3dd72a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18effff700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18effff700 1 -- 172.21.15.31:0/3073252154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 msgr2=0x55e3dd72a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18effff700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd72a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18effff700 1 -- 172.21.15.31:0/3073252154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3dd48ade0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18effff700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bb10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18e0095fc0 tx=0x7f18e0095940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc0d4a77c8f60ed6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:05.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18cf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18cf7fe700 1 -- 172.21.15.31:0/3073252154 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00b5050 con 0x55e3dd70bbe0 2023-07-29T16:19:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18cf7fe700 1 -- 172.21.15.31:0/3073252154 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18e00b6070 con 0x55e3dd70bbe0 2023-07-29T16:19:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18e00978f0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f18cf7fe700 1 -- 172.21.15.31:0/3073252154 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18e009d390 con 0x55e3dd70bbe0 2023-07-29T16:19:05.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f190a41cb80 10 client.26717 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:05.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.287+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e3dd4892b0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f18cf7fe700 1 -- 172.21.15.31:0/3073252154 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18e00978f0 con 0x55e3dd70bbe0 2023-07-29T16:19:05.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f18cf7fe700 1 -- 172.21.15.31:0/3073252154 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f18e00c3a30 con 0x55e3dd70bbe0 2023-07-29T16:19:05.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f190a41cb80 10 client.26717 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:05.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f190a41cb80 10 client.26717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:05.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f190a41cb80 20 client.26717 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:05.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f190a41cb80 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd785570 0x55e3dd787950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:05.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f190a41cb80 4 client.26717 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:19:05.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e3dd655d40 con 0x55e3dd785570 2023-07-29T16:19:05.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.288+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd785570 0x55e3dd787950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:05.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f18ef7fe700 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd785570 0x55e3dd787950 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a32c8dc2f76e77ef server_cookie=6532b8b1b52de7db in_seq=0 out_seq=0 2023-07-29T16:19:05.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f18cf7fe700 10 client.26717 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:05.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f18cf7fe700 1 -- 172.21.15.31:0/3073252154 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e3dd655d40 con 0x55e3dd785570 2023-07-29T16:19:05.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f18cf7fe700 10 client.26717 handle_command_reply: tid=0 2023-07-29T16:19:05.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 client.26717 shutdown 2023-07-29T16:19:05.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 msgr2=0x55e3dd72bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bb10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18e0095fc0 tx=0x7f18e0095940 comp rx=0 tx=0).stop 2023-07-29T16:19:05.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 shutdown_connections 2023-07-29T16:19:05.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e3dd785570 0x55e3dd787950 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e3dd70d2f0 0x55e3dd72a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e3dd70bbe0 0x55e3dd72bb10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f18cf7fe700 0 client.26717 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:05.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 --2- 172.21.15.31:0/3073252154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e3dd70c530 0x55e3dd72a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.289+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 >> 172.21.15.31:0/3073252154 conn(0x55e3dd7013e0 msgr2=0x55e3dd7101f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:05.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 shutdown_connections 2023-07-29T16:19:05.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f190a41cb80 1 -- 172.21.15.31:0/3073252154 wait complete. 2023-07-29T16:19:05.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f190a41cb80 20 client.26717 trim_cache size 0 max 16384 2023-07-29T16:19:05.309 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:19:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:19:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:19:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "b8b6a2f4-e3d4-4054-b522-b3af00e74b57": { 2023-07-29T16:19:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:19:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "b8b6a2f4-e3d4-4054-b522-b3af00e74b57", 2023-07-29T16:19:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:19:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:19:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:19:05.311 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:19:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18d80a0790 msgr2=0x7f18d80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18d80a0790 0x7f18d80a2c40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009c10 tx=0x7f18e8009ac0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 msgr2=0x7f18f0194c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f0194c10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f18e000e9a0 tx=0x7f18e000ed60 comp rx=0 tx=0).stop 2023-07-29T16:19:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 shutdown_connections 2023-07-29T16:19:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18d80a0790 0x7f18d80a2c40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18f0106020 0x7f18f01917a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18f0100de0 0x7f18f0194c10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 --2- 172.21.15.31:0/1451742372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18f0103700 0x7f18f0195150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:05.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.290+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 >> 172.21.15.31:0/1451742372 conn(0x7f18f00fa860 msgr2=0x7f18f00ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.291+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 shutdown_connections 2023-07-29T16:19:05.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:05.291+0000 7f18f69ff700 1 -- 172.21.15.31:0/1451742372 wait complete. 2023-07-29T16:19:05.315 INFO:tasks.cephfs.filesystem:scrub status for tag:b8b6a2f4-e3d4-4054-b522-b3af00e74b57 - {'path': '/', 'tag': 'b8b6a2f4-e3d4-4054-b522-b3af00e74b57', 'options': 'recursive,force'} 2023-07-29T16:19:05.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 84616 9.47 MB/sec warmup 50 sec latency 64.004 ms 2023-07-29T16:19:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:06 smithi039 ceph-mon[128858]: pgmap v479: 97 pgs: 97 active+clean; 35 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 304 op/s 2023-07-29T16:19:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:19:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:06 smithi156 ceph-mon[129245]: pgmap v479: 97 pgs: 97 active+clean; 35 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 304 op/s 2023-07-29T16:19:06.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 85909 9.42 MB/sec warmup 51 sec latency 13.447 ms 2023-07-29T16:19:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:19:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:06 smithi031 ceph-mon[95504]: pgmap v479: 97 pgs: 97 active+clean; 35 MiB data, 567 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 304 op/s 2023-07-29T16:19:07.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 88109 9.47 MB/sec warmup 52 sec latency 50.287 ms 2023-07-29T16:19:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:07 smithi031 ceph-mon[95504]: pgmap v480: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 234 op/s 2023-07-29T16:19:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:07 smithi039 ceph-mon[128858]: pgmap v480: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 234 op/s 2023-07-29T16:19:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:07 smithi156 ceph-mon[129245]: pgmap v480: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 234 op/s 2023-07-29T16:19:08.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 89436 9.42 MB/sec warmup 53 sec latency 10.645 ms 2023-07-29T16:19:09.858 INFO:tasks.workunit.client.0.smithi031.stdout: 1 91698 9.47 MB/sec warmup 54 sec latency 13.183 ms 2023-07-29T16:19:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.982+0000 7f02da9ff700 1 -- 172.21.15.31:0/260249211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 msgr2=0x7f02d4103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.982+0000 7f02da9ff700 1 --2- 172.21.15.31:0/260249211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d4103240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009990 tx=0x7f02c4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 -- 172.21.15.31:0/260249211 shutdown_connections 2023-07-29T16:19:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 --2- 172.21.15.31:0/260249211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 0x7f02d4108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 --2- 172.21.15.31:0/260249211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d4103240 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 --2- 172.21.15.31:0/260249211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 0x7f02d4105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 -- 172.21.15.31:0/260249211 >> 172.21.15.31:0/260249211 conn(0x7f02d40fa840 msgr2=0x7f02d40fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 -- 172.21.15.31:0/260249211 shutdown_connections 2023-07-29T16:19:09.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.983+0000 7f02da9ff700 1 -- 172.21.15.31:0/260249211 wait complete. 2023-07-29T16:19:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 Processor -- start 2023-07-29T16:19:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 -- start start 2023-07-29T16:19:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d40fee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 0x7f02d40ff350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 0x7f02d40ff890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41a0140 con 0x7f02d4103780 2023-07-29T16:19:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41a0280 con 0x7f02d4100e60 2023-07-29T16:19:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.984+0000 7f02da9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d41a03c0 con 0x7f02d41060a0 2023-07-29T16:19:09.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d8f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 0x7f02d40ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d40fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 0x7f02d40ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d8f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 0x7f02d40ff890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34460/0 (socket says 172.21.15.31:34460) 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d8f9c700 1 -- 172.21.15.31:0/323478844 learned_addr learned my addr 172.21.15.31:0/323478844 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 -- 172.21.15.31:0/323478844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 msgr2=0x7f02d40ff890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 0x7f02d40ff890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 -- 172.21.15.31:0/323478844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 msgr2=0x7f02d40fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d40fee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 -- 172.21.15.31:0/323478844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c40095f0 con 0x7f02d4103780 2023-07-29T16:19:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d3fff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d40fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.985+0000 7f02d37fe700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 0x7f02d40ff350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f02c800e9f0 tx=0x7f02c800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.986+0000 7f02d17fa700 1 -- 172.21.15.31:0/323478844 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c800c9c0 con 0x7f02d4103780 2023-07-29T16:19:09.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.986+0000 7f02d17fa700 1 -- 172.21.15.31:0/323478844 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02c8010430 con 0x7f02d4103780 2023-07-29T16:19:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.986+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d41a0650 con 0x7f02d4103780 2023-07-29T16:19:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.986+0000 7f02d17fa700 1 -- 172.21.15.31:0/323478844 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8018680 con 0x7f02d4103780 2023-07-29T16:19:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.986+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d41a0bb0 con 0x7f02d4103780 2023-07-29T16:19:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.990+0000 7f02d17fa700 1 -- 172.21.15.31:0/323478844 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f02c8010860 con 0x7f02d4103780 2023-07-29T16:19:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 Processor -- start 2023-07-29T16:19:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02d17fa700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02bc0a0810 0x7f02bc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 -- start start 2023-07-29T16:19:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02d17fa700 1 -- 172.21.15.31:0/323478844 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02c8014070 con 0x7f02d4103780 2023-07-29T16:19:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd2fd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd2fe160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:09.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd2549b0 con 0x5605dd1586b0 2023-07-29T16:19:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd252370 con 0x5605dd2fcdf0 2023-07-29T16:19:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02ee260b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd24dc60 con 0x5605dd2fdcf0 2023-07-29T16:19:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02bc0a0810 0x7f02bc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02d37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd2fd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02d37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd2fd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42840/0 (socket says 172.21.15.31:42840) 2023-07-29T16:19:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.992+0000 7f02d37fe700 1 -- 172.21.15.31:0/3345265392 learned_addr learned my addr 172.21.15.31:0/3345265392 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd2fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d3fff700 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:09.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d37fe700 1 -- 172.21.15.31:0/3345265392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 msgr2=0x5605dd2fe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d37fe700 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd2fe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d37fe700 1 -- 172.21.15.31:0/3345265392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 msgr2=0x5605dd2fc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d37fe700 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d37fe700 1 -- 172.21.15.31:0/3345265392 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605dd07bd80 con 0x5605dd2fcdf0 2023-07-29T16:19:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d37fe700 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd2fd7b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f02c8099310 tx=0x7f02c8098ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d18eb4a099b1bf56 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:09.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02bc0a0810 0x7f02bc0a2cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009c10 tx=0x7f02cc0093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02ba7fc700 1 -- 172.21.15.31:0/3345265392 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c809d1b0 con 0x5605dd2fcdf0 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02ba7fc700 1 -- 172.21.15.31:0/3345265392 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02c8099380 con 0x5605dd2fcdf0 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.993+0000 7f02ba7fc700 1 -- 172.21.15.31:0/3345265392 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8097840 con 0x5605dd2fcdf0 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 -- 172.21.15.31:0/3345265392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 msgr2=0x5605dd2fd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd2fd7b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f02c8099310 tx=0x7f02c8098ec0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 -- 172.21.15.31:0/3345265392 shutdown_connections 2023-07-29T16:19:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd2fe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3345265392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd2fd7b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 -- 172.21.15.31:0/3345265392 >> 172.21.15.31:0/3345265392 conn(0x5605dd2f1fd0 msgr2=0x5605dd2f7300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 -- 172.21.15.31:0/3345265392 shutdown_connections 2023-07-29T16:19:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.994+0000 7f02ee260b80 1 -- 172.21.15.31:0/3345265392 wait complete. 2023-07-29T16:19:09.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.995+0000 7f02ee260b80 1 -- start start 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.995+0000 7f02ee260b80 1 Processor -- start 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 0x5605dd305f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd306440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fdcf0 0x5605dd30d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd1ea4b0 con 0x5605dd2fcdf0 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd1ea150 con 0x5605dd2fdcf0 2023-07-29T16:19:10.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd0d18c0 con 0x5605dd1586b0 2023-07-29T16:19:10.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02ee260b80 10 client.0 _ll_register_callbacks cb 0x5605dd1f35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 0x5605dd305f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d8f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fdcf0 0x5605dd30d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 0x5605dd305f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34482/0 (socket says 172.21.15.31:34482) 2023-07-29T16:19:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d8f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fdcf0 0x5605dd30d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42856/0 (socket says 172.21.15.31:42856) 2023-07-29T16:19:10.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 -- 172.21.15.31:0/178752594 learned_addr learned my addr 172.21.15.31:0/178752594 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d37fe700 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd306440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 -- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fdcf0 msgr2=0x5605dd30d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fdcf0 0x5605dd30d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 -- 172.21.15.31:0/178752594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 msgr2=0x5605dd306440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd306440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.996+0000 7f02d3fff700 1 -- 172.21.15.31:0/178752594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d41a0bb0 con 0x5605dd1586b0 2023-07-29T16:19:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02d3fff700 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 0x5605dd305f00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f02d4068c00 tx=0x7f02c4005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa51f3ed4b86de0e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02b37fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02b37fe700 1 -- 172.21.15.31:0/178752594 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c401d070 con 0x5605dd1586b0 2023-07-29T16:19:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02b37fe700 1 -- 172.21.15.31:0/178752594 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02c4004910 con 0x5605dd1586b0 2023-07-29T16:19:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605dd278cb0 con 0x5605dd1586b0 2023-07-29T16:19:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02b37fe700 1 -- 172.21.15.31:0/178752594 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02c4004ad0 con 0x5605dd1586b0 2023-07-29T16:19:10.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02ee260b80 10 client.26743 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.997+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5605dd064c00 con 0x5605dd1586b0 2023-07-29T16:19:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02b37fe700 1 -- 172.21.15.31:0/178752594 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5605dd278cb0 con 0x5605dd1586b0 2023-07-29T16:19:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02b37fe700 1 -- 172.21.15.31:0/178752594 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f02c4024450 con 0x5605dd1586b0 2023-07-29T16:19:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02ee260b80 10 client.26743 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02ee260b80 10 client.26743 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02ee260b80 20 client.26743 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02ee260b80 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd311d40 0x5605dd314150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02ee260b80 4 client.26743 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605dd2cbd10 con 0x5605dd311d40 2023-07-29T16:19:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.998+0000 7f02d37fe700 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd311d40 0x5605dd314150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.999+0000 7f02d37fe700 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd311d40 0x5605dd314150 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53fc8ec58411b256 server_cookie=628b8079df150a3 in_seq=0 out_seq=0 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:09.999+0000 7f02b37fe700 10 client.26743 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02b37fe700 1 -- 172.21.15.31:0/178752594 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5605dd2cbd10 con 0x5605dd311d40 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02b37fe700 10 client.26743 handle_command_reply: tid=0 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 client.26743 shutdown 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 msgr2=0x5605dd305f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:10.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 0x5605dd305f00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f02d4068c00 tx=0x7f02c4005250 comp rx=0 tx=0).stop 2023-07-29T16:19:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 shutdown_connections 2023-07-29T16:19:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd311d40 0x5605dd314150 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fdcf0 0x5605dd30d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd1586b0 0x5605dd305f00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 --2- 172.21.15.31:0/178752594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd306440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02b37fe700 0 client.26743 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.000+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 >> 172.21.15.31:0/178752594 conn(0x5605dd2f1fd0 msgr2=0x5605dd2f6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.001+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 shutdown_connections 2023-07-29T16:19:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.001+0000 7f02ee260b80 1 -- 172.21.15.31:0/178752594 wait complete. 2023-07-29T16:19:10.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.001+0000 7f02ee260b80 20 client.26743 trim_cache size 0 max 16384 2023-07-29T16:19:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.017+0000 7f02ee260b80 1 Processor -- start 2023-07-29T16:19:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.017+0000 7f02ee260b80 1 -- start start 2023-07-29T16:19:10.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fe0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd30d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd304070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02ee260b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd30f110 con 0x5605dd1586b0 2023-07-29T16:19:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02ee260b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd307080 con 0x5605dd2fcdf0 2023-07-29T16:19:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02ee260b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd303e20 con 0x5605dd2fdcf0 2023-07-29T16:19:10.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd30d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fe0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd30d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42864/0 (socket says 172.21.15.31:42864) 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d37fe700 1 -- 172.21.15.31:0/3493875682 learned_addr learned my addr 172.21.15.31:0/3493875682 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd304070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d37fe700 1 -- 172.21.15.31:0/3493875682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 msgr2=0x5605dd304070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.018+0000 7f02d37fe700 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd304070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02d37fe700 1 -- 172.21.15.31:0/3493875682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 msgr2=0x5605dd2fe0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:10.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02d37fe700 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fe0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02d37fe700 1 -- 172.21.15.31:0/3493875682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605dd0a21f0 con 0x5605dd2fcdf0 2023-07-29T16:19:10.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02d3fff700 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:10.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02d37fe700 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd30d440 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f02c800e9c0 tx=0x7f02c8005b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebf62d74ae6f8559 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02baffd700 1 -- 172.21.15.31:0/3493875682 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c809d1b0 con 0x5605dd2fcdf0 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02baffd700 1 -- 172.21.15.31:0/3493875682 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02c80974c0 con 0x5605dd2fcdf0 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.019+0000 7f02baffd700 1 -- 172.21.15.31:0/3493875682 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c8010690 con 0x5605dd2fcdf0 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 -- 172.21.15.31:0/3493875682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 msgr2=0x5605dd30d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd30d440 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f02c800e9c0 tx=0x7f02c8005b40 comp rx=0 tx=0).stop 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 -- 172.21.15.31:0/3493875682 shutdown_connections 2023-07-29T16:19:10.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd304070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd1586b0 0x5605dd2fe0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 --2- 172.21.15.31:0/3493875682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd2fcdf0 0x5605dd30d440 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 -- 172.21.15.31:0/3493875682 >> 172.21.15.31:0/3493875682 conn(0x5605dd2f1fd0 msgr2=0x5605dd304c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 -- 172.21.15.31:0/3493875682 shutdown_connections 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 -- 172.21.15.31:0/3493875682 wait complete. 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.020+0000 7f02ee260b80 1 -- start start 2023-07-29T16:19:10.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 Processor -- start 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd1586b0 0x5605dd308eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd3093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd30c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd30c950 con 0x5605dd2fcdf0 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd30cac0 con 0x5605dd1586b0 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605dd30cc80 con 0x5605dd2fdcf0 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.021+0000 7f02ee260b80 10 client.0 _ll_register_callbacks cb 0x5605dd1f35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:10.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd30c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd1586b0 0x5605dd308eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd30c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34492/0 (socket says 172.21.15.31:34492) 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 -- 172.21.15.31:0/681431569 learned_addr learned my addr 172.21.15.31:0/681431569 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d37fe700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd3093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 -- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd1586b0 msgr2=0x5605dd308eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd1586b0 0x5605dd308eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 -- 172.21.15.31:0/681431569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 msgr2=0x5605dd3093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd3093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d8f9c700 1 -- 172.21.15.31:0/681431569 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605dd064c00 con 0x5605dd2fdcf0 2023-07-29T16:19:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.022+0000 7f02d3fff700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd1586b0 0x5605dd308eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02d8f9c700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd30c410 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f02cc01b6d0 tx=0x7f02cc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c3d8ee329ea16e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:10.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02b37fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02b37fe700 1 -- 172.21.15.31:0/681431569 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc00e070 con 0x5605dd2fdcf0 2023-07-29T16:19:10.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02c8099380 con 0x5605dd2fdcf0 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02b37fe700 1 -- 172.21.15.31:0/681431569 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02cc021040 con 0x5605dd2fdcf0 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02b37fe700 1 -- 172.21.15.31:0/681431569 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02cc01aee0 con 0x5605dd2fdcf0 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02ee260b80 10 client.26749 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.023+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f02c80b5040 con 0x5605dd2fdcf0 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02b37fe700 1 -- 172.21.15.31:0/681431569 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02c8099380 con 0x5605dd2fdcf0 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02b37fe700 1 -- 172.21.15.31:0/681431569 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f02cc031560 con 0x5605dd2fdcf0 2023-07-29T16:19:10.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02ee260b80 10 client.26749 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02ee260b80 10 client.26749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02ee260b80 20 client.26749 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02ee260b80 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd364540 0x5605dd366950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02ee260b80 4 client.26749 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5605dd246ce0 con 0x5605dd364540 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.024+0000 7f02d3fff700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd364540 0x5605dd366950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.025+0000 7f02d3fff700 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd364540 0x5605dd366950 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b54160b5d965ddb9 server_cookie=a688bf511d70786d in_seq=0 out_seq=0 2023-07-29T16:19:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.025+0000 7f02b37fe700 10 client.26749 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:10.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02b37fe700 1 -- 172.21.15.31:0/681431569 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5605dd246ce0 con 0x5605dd364540 2023-07-29T16:19:10.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02b37fe700 10 client.26749 handle_command_reply: tid=0 2023-07-29T16:19:10.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 client.26749 shutdown 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 msgr2=0x5605dd30c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd30c410 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f02cc01b6d0 tx=0x7f02cc00b040 comp rx=0 tx=0).stop 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 shutdown_connections 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5605dd364540 0x5605dd366950 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5605dd2fdcf0 0x5605dd30c410 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5605dd1586b0 0x5605dd308eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 --2- 172.21.15.31:0/681431569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5605dd2fcdf0 0x5605dd3093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 >> 172.21.15.31:0/681431569 conn(0x5605dd2f1fd0 msgr2=0x5605dd304070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02b37fe700 0 client.26749 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 shutdown_connections 2023-07-29T16:19:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 1 -- 172.21.15.31:0/681431569 wait complete. 2023-07-29T16:19:10.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.109+0000 7f02ee260b80 20 client.26749 trim_cache size 0 max 16384 2023-07-29T16:19:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:19:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:19:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:19:10.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02bc0a0810 msgr2=0x7f02bc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02bc0a0810 0x7f02bc0a2cc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009c10 tx=0x7f02cc0093f0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 msgr2=0x7f02d40ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 0x7f02d40ff350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f02c800e9f0 tx=0x7f02c800ed00 comp rx=0 tx=0).stop 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 shutdown_connections 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02bc0a0810 0x7f02bc0a2cc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41060a0 0x7f02d40ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02d4100e60 0x7f02d40fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 --2- 172.21.15.31:0/323478844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02d4103780 0x7f02d40ff350 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 >> 172.21.15.31:0/323478844 conn(0x7f02d40fa840 msgr2=0x7f02d4101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 shutdown_connections 2023-07-29T16:19:10.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:10.110+0000 7f02da9ff700 1 -- 172.21.15.31:0/323478844 wait complete. 2023-07-29T16:19:10.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:10 smithi031 ceph-mon[95504]: pgmap v481: 97 pgs: 97 active+clean; 39 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 313 op/s 2023-07-29T16:19:10.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:10 smithi039 ceph-mon[128858]: pgmap v481: 97 pgs: 97 active+clean; 39 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 313 op/s 2023-07-29T16:19:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:10 smithi156 ceph-mon[129245]: pgmap v481: 97 pgs: 97 active+clean; 39 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.4 MiB/s wr, 313 op/s 2023-07-29T16:19:10.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 92826 9.42 MB/sec warmup 55 sec latency 33.892 ms 2023-07-29T16:19:11.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 95165 9.47 MB/sec warmup 56 sec latency 15.813 ms 2023-07-29T16:19:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:12 smithi039 ceph-mon[128858]: pgmap v482: 97 pgs: 97 active+clean; 39 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 952 KiB/s wr, 191 op/s 2023-07-29T16:19:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:12 smithi156 ceph-mon[129245]: pgmap v482: 97 pgs: 97 active+clean; 39 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 952 KiB/s wr, 191 op/s 2023-07-29T16:19:12.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 96399 9.43 MB/sec warmup 57 sec latency 17.307 ms 2023-07-29T16:19:12.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:12 smithi031 ceph-mon[95504]: pgmap v482: 97 pgs: 97 active+clean; 39 MiB data, 584 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 952 KiB/s wr, 191 op/s 2023-07-29T16:19:13.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:19:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:19:13.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 98584 9.45 MB/sec warmup 58 sec latency 16.826 ms 2023-07-29T16:19:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:19:14.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:14 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:19:14.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:14 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:19:14.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:14 smithi039 ceph-mon[128858]: pgmap v483: 97 pgs: 97 active+clean; 36 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 235 op/s 2023-07-29T16:19:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:14 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:19:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:14 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:19:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:14 smithi156 ceph-mon[129245]: pgmap v483: 97 pgs: 97 active+clean; 36 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 235 op/s 2023-07-29T16:19:14.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 99888 9.43 MB/sec warmup 59 sec latency 14.505 ms 2023-07-29T16:19:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:14 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:19:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:14 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:19:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:14 smithi031 ceph-mon[95504]: pgmap v483: 97 pgs: 97 active+clean; 36 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.1 MiB/s wr, 235 op/s 2023-07-29T16:19:15.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 101956 9.41 MB/sec warmup 60 sec latency 13.592 ms 2023-07-29T16:19:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:16 smithi039 ceph-mon[128858]: pgmap v484: 97 pgs: 97 active+clean; 40 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1020 KiB/s wr, 241 op/s 2023-07-29T16:19:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:16 smithi156 ceph-mon[129245]: pgmap v484: 97 pgs: 97 active+clean; 40 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1020 KiB/s wr, 241 op/s 2023-07-29T16:19:16.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 103376 9.44 MB/sec warmup 61 sec latency 17.297 ms 2023-07-29T16:19:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:16 smithi031 ceph-mon[95504]: pgmap v484: 97 pgs: 97 active+clean; 40 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1020 KiB/s wr, 241 op/s 2023-07-29T16:19:17.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 105351 9.40 MB/sec warmup 62 sec latency 12.284 ms 2023-07-29T16:19:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:17 smithi031 ceph-mon[95504]: pgmap v485: 97 pgs: 97 active+clean; 40 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 968 KiB/s wr, 232 op/s 2023-07-29T16:19:18.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:17 smithi039 ceph-mon[128858]: pgmap v485: 97 pgs: 97 active+clean; 40 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 968 KiB/s wr, 232 op/s 2023-07-29T16:19:18.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:17 smithi156 ceph-mon[129245]: pgmap v485: 97 pgs: 97 active+clean; 40 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 968 KiB/s wr, 232 op/s 2023-07-29T16:19:18.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 106890 9.44 MB/sec warmup 63 sec latency 15.029 ms 2023-07-29T16:19:19.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 109009 9.42 MB/sec warmup 64 sec latency 13.700 ms 2023-07-29T16:19:20.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:20 smithi039 ceph-mon[128858]: pgmap v486: 97 pgs: 97 active+clean; 44 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 312 op/s 2023-07-29T16:19:20.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:20 smithi156 ceph-mon[129245]: pgmap v486: 97 pgs: 97 active+clean; 44 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 312 op/s 2023-07-29T16:19:20.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 110526 9.44 MB/sec warmup 65 sec latency 11.240 ms 2023-07-29T16:19:20.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:20 smithi031 ceph-mon[95504]: pgmap v486: 97 pgs: 97 active+clean; 44 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.4 MiB/s wr, 312 op/s 2023-07-29T16:19:21.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 112779 9.47 MB/sec warmup 66 sec latency 17.228 ms 2023-07-29T16:19:22.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:22 smithi039 ceph-mon[128858]: pgmap v487: 97 pgs: 97 active+clean; 44 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1000 KiB/s wr, 215 op/s 2023-07-29T16:19:22.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:22 smithi156 ceph-mon[129245]: pgmap v487: 97 pgs: 97 active+clean; 44 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1000 KiB/s wr, 215 op/s 2023-07-29T16:19:22.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 113923 9.45 MB/sec warmup 67 sec latency 164.802 ms 2023-07-29T16:19:22.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:22 smithi031 ceph-mon[95504]: pgmap v487: 97 pgs: 97 active+clean; 44 MiB data, 616 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1000 KiB/s wr, 215 op/s 2023-07-29T16:19:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 -- 172.21.15.31:0/103583383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164074d10 msgr2=0x7f2164073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 --2- 172.21.15.31:0/103583383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164074d10 0x7f2164073160 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2154009990 tx=0x7f2154009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 -- 172.21.15.31:0/103583383 shutdown_connections 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 --2- 172.21.15.31:0/103583383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2164108200 0x7f216410a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 --2- 172.21.15.31:0/103583383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164074d10 0x7f2164073160 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 --2- 172.21.15.31:0/103583383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 0x7f2164073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 -- 172.21.15.31:0/103583383 >> 172.21.15.31:0/103583383 conn(0x7f21640fb970 msgr2=0x7f21640fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 -- 172.21.15.31:0/103583383 shutdown_connections 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.264+0000 7f21699ff700 1 -- 172.21.15.31:0/103583383 wait complete. 2023-07-29T16:19:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.265+0000 7f21699ff700 1 Processor -- start 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.265+0000 7f21699ff700 1 -- start start 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21699ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 0x7f2164198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21699ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2164074d10 0x7f21641988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21699ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 0x7f2164199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21699ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2164199d90 con 0x7f2164074d10 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21699ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2164199f00 con 0x7f2164108200 2023-07-29T16:19:23.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21699ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216419a0c0 con 0x7f2164073730 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21627fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2164074d10 0x7f21641988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 0x7f2164199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 0x7f2164198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 0x7f2164199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48966/0 (socket says 172.21.15.31:48966) 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 0x7f2164198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41276/0 (socket says 172.21.15.31:41276) 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 -- 172.21.15.31:0/1423561205 learned_addr learned my addr 172.21.15.31:0/1423561205 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:23.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 -- 172.21.15.31:0/1423561205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 msgr2=0x7f2164198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 0x7f2164198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 -- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2164074d10 msgr2=0x7f21641988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2164074d10 0x7f21641988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.266+0000 7f21637fe700 1 -- 172.21.15.31:0/1423561205 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21540095f0 con 0x7f2164108200 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.267+0000 7f21637fe700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 0x7f2164199730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f215c00ef30 tx=0x7f215c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.267+0000 7f21689fd700 1 -- 172.21.15.31:0/1423561205 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c018070 con 0x7f2164108200 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.267+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216419a340 con 0x7f2164108200 2023-07-29T16:19:23.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.267+0000 7f21689fd700 1 -- 172.21.15.31:0/1423561205 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f215c00f040 con 0x7f2164108200 2023-07-29T16:19:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.267+0000 7f21689fd700 1 -- 172.21.15.31:0/1423561205 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c00cd00 con 0x7f2164108200 2023-07-29T16:19:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.267+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21641a0b60 con 0x7f2164108200 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.271+0000 7f21689fd700 1 -- 172.21.15.31:0/1423561205 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f215c003a00 con 0x7f2164108200 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 Processor -- start 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 -- start start 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dc48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dc57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dc6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68d1c9b0 con 0x560d68dc5cf0 2023-07-29T16:19:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68d1a370 con 0x560d68c206b0 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.272+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68d15c60 con 0x560d68dc4df0 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dc57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dc57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41286/0 (socket says 172.21.15.31:41286) 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 -- 172.21.15.31:0/1349980141 learned_addr learned my addr 172.21.15.31:0/1349980141 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f21627fc700 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dc48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f21627fc700 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dc6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f21689fd700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f214c0a0790 0x7f214c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 -- 172.21.15.31:0/1349980141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 msgr2=0x560d68dc48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dc48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 -- 172.21.15.31:0/1349980141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 msgr2=0x560d68dc6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dc6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 -- 172.21.15.31:0/1349980141 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d68b43d80 con 0x560d68dc4df0 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f21627fc700 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dc6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:23.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f21689fd700 1 -- 172.21.15.31:0/1423561205 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f215c0cb5b0 con 0x7f2164108200 2023-07-29T16:19:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f2162ffd700 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dc57b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2154015000 tx=0x7f2154005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d10fc60880e8fee4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:23.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.273+0000 7f21637fe700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f214c0a0790 0x7f214c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.274+0000 7f2149ffb700 1 -- 172.21.15.31:0/1349980141 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215401d070 con 0x560d68dc4df0 2023-07-29T16:19:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.274+0000 7f2149ffb700 1 -- 172.21.15.31:0/1349980141 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f215401e040 con 0x560d68dc4df0 2023-07-29T16:19:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.274+0000 7f2149ffb700 1 -- 172.21.15.31:0/1349980141 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21540049c0 con 0x560d68dc4df0 2023-07-29T16:19:23.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.274+0000 7f217d3cfb80 1 -- 172.21.15.31:0/1349980141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 msgr2=0x560d68dc57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.274+0000 7f21637fe700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f214c0a0790 0x7f214c0a2c40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f215c0a0b70 tx=0x7f215c0092e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.274+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dc57b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2154015000 tx=0x7f2154005bc0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 -- 172.21.15.31:0/1349980141 shutdown_connections 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dc6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dc48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/1349980141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dc57b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 -- 172.21.15.31:0/1349980141 >> 172.21.15.31:0/1349980141 conn(0x560d68db9fd0 msgr2=0x560d68dbf300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:23.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 -- 172.21.15.31:0/1349980141 shutdown_connections 2023-07-29T16:19:23.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.275+0000 7f217d3cfb80 1 -- 172.21.15.31:0/1349980141 wait complete. 2023-07-29T16:19:23.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.276+0000 7f217d3cfb80 1 -- start start 2023-07-29T16:19:23.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.276+0000 7f217d3cfb80 1 Processor -- start 2023-07-29T16:19:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dcde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc5cf0 0x560d68dd53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68cb24b0 con 0x560d68dc4df0 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68cb2150 con 0x560d68c206b0 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68b998c0 con 0x560d68dc5cf0 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f217d3cfb80 10 client.0 _ll_register_callbacks cb 0x560d68cbb5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dcde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dcde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48984/0 (socket says 172.21.15.31:48984) 2023-07-29T16:19:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 -- 172.21.15.31:0/3429002006 learned_addr learned my addr 172.21.15.31:0/3429002006 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc5cf0 0x560d68dd53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 -- 172.21.15.31:0/3429002006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc5cf0 msgr2=0x560d68dd53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc5cf0 0x560d68dd53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f21627fc700 1 -- 172.21.15.31:0/3429002006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 msgr2=0x560d68dce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.277+0000 7f2162ffd700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f21627fc700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f21627fc700 1 -- 172.21.15.31:0/3429002006 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216410b280 con 0x560d68c206b0 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f2162ffd700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f21627fc700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dcde50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2158005910 tx=0x7f21580047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778d70af6f8e241b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f2142ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f2142ffd700 1 -- 172.21.15.31:0/3429002006 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2158020d70 con 0x560d68c206b0 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f2142ffd700 1 -- 172.21.15.31:0/3429002006 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2158028030 con 0x560d68c206b0 2023-07-29T16:19:23.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.278+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560d68d40cb0 con 0x560d68c206b0 2023-07-29T16:19:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.279+0000 7f2142ffd700 1 -- 172.21.15.31:0/3429002006 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2158004030 con 0x560d68c206b0 2023-07-29T16:19:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.279+0000 7f217d3cfb80 10 client.26747 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:23.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.279+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560d68b2cc00 con 0x560d68c206b0 2023-07-29T16:19:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.279+0000 7f2142ffd700 1 -- 172.21.15.31:0/3429002006 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560d68d40cb0 con 0x560d68c206b0 2023-07-29T16:19:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f2142ffd700 1 -- 172.21.15.31:0/3429002006 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2158003b10 con 0x560d68c206b0 2023-07-29T16:19:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f217d3cfb80 10 client.26747 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f217d3cfb80 10 client.26747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f217d3cfb80 20 client.26747 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68dd9cd0 0x560d68ddc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f217d3cfb80 4 client.26747 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d68d93d10 con 0x560d68dd9cd0 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.280+0000 7f2162ffd700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68dd9cd0 0x560d68ddc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.281+0000 7f2162ffd700 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68dd9cd0 0x560d68ddc0e0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=910e46d4bc61d8e8 server_cookie=6e0fd2ede9847bf0 in_seq=0 out_seq=0 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.281+0000 7f2142ffd700 10 client.26747 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f2142ffd700 1 -- 172.21.15.31:0/3429002006 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560d68d93d10 con 0x560d68dd9cd0 2023-07-29T16:19:23.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f2142ffd700 10 client.26747 handle_command_reply: tid=0 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 client.26747 shutdown 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 msgr2=0x560d68dcde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dcde50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2158005910 tx=0x7f21580047c0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 shutdown_connections 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68dd9cd0 0x560d68ddc0e0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc5cf0 0x560d68dd53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dcde50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/3429002006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f2142ffd700 0 client.26747 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.282+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 >> 172.21.15.31:0/3429002006 conn(0x560d68db9fd0 msgr2=0x560d68dbea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.283+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 shutdown_connections 2023-07-29T16:19:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.283+0000 7f217d3cfb80 1 -- 172.21.15.31:0/3429002006 wait complete. 2023-07-29T16:19:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.283+0000 7f217d3cfb80 20 client.26747 trim_cache size 0 max 16384 2023-07-29T16:19:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 Processor -- start 2023-07-29T16:19:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 -- start start 2023-07-29T16:19:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68c206b0 0x560d68dd3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dc60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68dc5cf0 0x560d68dbf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68dcd300 con 0x560d68dc4df0 2023-07-29T16:19:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68dc6600 con 0x560d68dc5cf0 2023-07-29T16:19:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.298+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68dc0b30 con 0x560d68c206b0 2023-07-29T16:19:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f21627fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68c206b0 0x560d68dd3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dc60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f21627fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68c206b0 0x560d68dd3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41300/0 (socket says 172.21.15.31:41300) 2023-07-29T16:19:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dc60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42620/0 (socket says 172.21.15.31:42620) 2023-07-29T16:19:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f21627fc700 1 -- 172.21.15.31:0/2336539186 learned_addr learned my addr 172.21.15.31:0/2336539186 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f21627fc700 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68dc5cf0 0x560d68dbf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 -- 172.21.15.31:0/2336539186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68c206b0 msgr2=0x560d68dd3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68c206b0 0x560d68dd3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 -- 172.21.15.31:0/2336539186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68dc5cf0 msgr2=0x560d68dbf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68dc5cf0 0x560d68dbf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 -- 172.21.15.31:0/2336539186 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d68b6a1f0 con 0x560d68dc4df0 2023-07-29T16:19:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.299+0000 7f2162ffd700 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dc60c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2154005810 tx=0x7f2154000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cb8dc5be63b9c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f214a7fc700 1 -- 172.21.15.31:0/2336539186 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215401d040 con 0x560d68dc4df0 2023-07-29T16:19:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f214a7fc700 1 -- 172.21.15.31:0/2336539186 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f215400b2c0 con 0x560d68dc4df0 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f214a7fc700 1 -- 172.21.15.31:0/2336539186 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215402c840 con 0x560d68dc4df0 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 -- 172.21.15.31:0/2336539186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 msgr2=0x560d68dc60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dc60c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2154005810 tx=0x7f2154000bc0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 -- 172.21.15.31:0/2336539186 shutdown_connections 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68dc5cf0 0x560d68dbf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68c206b0 0x560d68dd3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/2336539186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc4df0 0x560d68dc60c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 -- 172.21.15.31:0/2336539186 >> 172.21.15.31:0/2336539186 conn(0x560d68db9fd0 msgr2=0x560d68dccbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.300+0000 7f217d3cfb80 1 -- 172.21.15.31:0/2336539186 shutdown_connections 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.301+0000 7f217d3cfb80 1 -- 172.21.15.31:0/2336539186 wait complete. 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.301+0000 7f217d3cfb80 1 -- start start 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.301+0000 7f217d3cfb80 1 Processor -- start 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.301+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dd1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.301+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dd1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f217d3cfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dd2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68dd2ab0 con 0x560d68dc5cf0 2023-07-29T16:19:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68dd2c20 con 0x560d68c206b0 2023-07-29T16:19:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f217d3cfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d68dd2de0 con 0x560d68dc4df0 2023-07-29T16:19:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f217d3cfb80 10 client.0 _ll_register_callbacks cb 0x560d68cbb5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f21627fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dd1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dd1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dd1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41314/0 (socket says 172.21.15.31:41314) 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 -- 172.21.15.31:0/412391399 learned_addr learned my addr 172.21.15.31:0/412391399 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f21627fc700 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dd2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 -- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 msgr2=0x560d68dd1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dd1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 -- 172.21.15.31:0/412391399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 msgr2=0x560d68dd2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dd2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 -- 172.21.15.31:0/412391399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d68b43d80 con 0x560d68dc4df0 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2162ffd700 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dd1560 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21540098e0 tx=0x7f215402c5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf8f900b7638eb38 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.302+0000 7f2142ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f2142ffd700 1 -- 172.21.15.31:0/412391399 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215401d040 con 0x560d68dc4df0 2023-07-29T16:19:23.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f215401e040 con 0x560d68dc4df0 2023-07-29T16:19:23.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f2142ffd700 1 -- 172.21.15.31:0/412391399 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f215402cba0 con 0x560d68dc4df0 2023-07-29T16:19:23.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f2142ffd700 1 -- 172.21.15.31:0/412391399 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f215402cd60 con 0x560d68dc4df0 2023-07-29T16:19:23.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f217d3cfb80 10 client.26767 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:23.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560d68b42250 con 0x560d68dc4df0 2023-07-29T16:19:23.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.303+0000 7f2142ffd700 1 -- 172.21.15.31:0/412391399 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f215401e040 con 0x560d68dc4df0 2023-07-29T16:19:23.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f2142ffd700 1 -- 172.21.15.31:0/412391399 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2154039030 con 0x560d68dc4df0 2023-07-29T16:19:23.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f217d3cfb80 10 client.26767 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:23.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f217d3cfb80 10 client.26767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:23.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f217d3cfb80 20 client.26767 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68e2c1f0 0x560d68e2e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f217d3cfb80 4 client.26767 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:19:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.304+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560d68d0ece0 con 0x560d68e2c1f0 2023-07-29T16:19:23.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.306+0000 7f21627fc700 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68e2c1f0 0x560d68e2e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.307+0000 7f21627fc700 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68e2c1f0 0x560d68e2e5d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15df1a81fd0e87aa server_cookie=ba38b14b571fae88 in_seq=0 out_seq=0 2023-07-29T16:19:23.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.307+0000 7f2142ffd700 10 client.26767 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f2142ffd700 1 -- 172.21.15.31:0/412391399 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560d68d0ece0 con 0x560d68e2c1f0 2023-07-29T16:19:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f2142ffd700 10 client.26767 handle_command_reply: tid=0 2023-07-29T16:19:23.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 client.26767 shutdown 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 msgr2=0x560d68dd1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dd1560 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f21540098e0 tx=0x7f215402c5a0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 shutdown_connections 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560d68e2c1f0 0x560d68e2e5d0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d68dc5cf0 0x560d68dd2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d68c206b0 0x560d68dd1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f2142ffd700 0 client.26767 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:23.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 --2- 172.21.15.31:0/412391399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560d68dc4df0 0x560d68dd1560 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 >> 172.21.15.31:0/412391399 conn(0x560d68db9fd0 msgr2=0x560d68dcc090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 shutdown_connections 2023-07-29T16:19:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 1 -- 172.21.15.31:0/412391399 wait complete. 2023-07-29T16:19:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.385+0000 7f217d3cfb80 20 client.26767 trim_cache size 0 max 16384 2023-07-29T16:19:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:19:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:19:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:19:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:19:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f214c0a0790 msgr2=0x7f214c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f214c0a0790 0x7f214c0a2c40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f215c0a0b70 tx=0x7f215c0092e0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 msgr2=0x7f2164199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 0x7f2164199730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f215c00ef30 tx=0x7f215c00c3a0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 shutdown_connections 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f214c0a0790 0x7f214c0a2c40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2164108200 0x7f2164199730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2164073730 0x7f2164198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 --2- 172.21.15.31:0/1423561205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2164074d10 0x7f21641988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 >> 172.21.15.31:0/1423561205 conn(0x7f21640fb970 msgr2=0x7f21640fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 shutdown_connections 2023-07-29T16:19:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:23.386+0000 7f21699ff700 1 -- 172.21.15.31:0/1423561205 wait complete. 2023-07-29T16:19:23.859 INFO:tasks.workunit.client.0.smithi031.stdout: 1 115238 9.39 MB/sec warmup 68 sec latency 41.667 ms 2023-07-29T16:19:24.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:24 smithi039 ceph-mon[128858]: pgmap v488: 97 pgs: 97 active+clean; 47 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2023-07-29T16:19:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:24 smithi156 ceph-mon[129245]: pgmap v488: 97 pgs: 97 active+clean; 47 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2023-07-29T16:19:24.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 117391 9.45 MB/sec warmup 69 sec latency 17.905 ms 2023-07-29T16:19:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:24 smithi031 ceph-mon[95504]: pgmap v488: 97 pgs: 97 active+clean; 47 MiB data, 646 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 260 op/s 2023-07-29T16:19:25.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 118794 9.39 MB/sec warmup 70 sec latency 12.621 ms 2023-07-29T16:19:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:26 smithi039 ceph-mon[128858]: pgmap v489: 97 pgs: 97 active+clean; 43 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 266 op/s 2023-07-29T16:19:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:26 smithi156 ceph-mon[129245]: pgmap v489: 97 pgs: 97 active+clean; 43 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 266 op/s 2023-07-29T16:19:26.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 120929 9.45 MB/sec warmup 71 sec latency 15.083 ms 2023-07-29T16:19:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:26 smithi031 ceph-mon[95504]: pgmap v489: 97 pgs: 97 active+clean; 43 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.6 MiB/s wr, 266 op/s 2023-07-29T16:19:27.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 122409 9.40 MB/sec warmup 72 sec latency 38.219 ms 2023-07-29T16:19:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:27 smithi031 ceph-mon[95504]: pgmap v490: 97 pgs: 97 active+clean; 43 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-07-29T16:19:28.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:27 smithi039 ceph-mon[128858]: pgmap v490: 97 pgs: 97 active+clean; 43 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-07-29T16:19:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:27 smithi156 ceph-mon[129245]: pgmap v490: 97 pgs: 97 active+clean; 43 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2023-07-29T16:19:28.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 124418 9.45 MB/sec warmup 73 sec latency 47.008 ms 2023-07-29T16:19:29.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 125835 9.40 MB/sec warmup 74 sec latency 18.563 ms 2023-07-29T16:19:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:30 smithi039 ceph-mon[128858]: pgmap v491: 97 pgs: 97 active+clean; 46 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 298 op/s 2023-07-29T16:19:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:30 smithi156 ceph-mon[129245]: pgmap v491: 97 pgs: 97 active+clean; 46 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 298 op/s 2023-07-29T16:19:30.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 127999 9.45 MB/sec warmup 75 sec latency 30.579 ms 2023-07-29T16:19:30.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:30 smithi031 ceph-mon[95504]: pgmap v491: 97 pgs: 97 active+clean; 46 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 298 op/s 2023-07-29T16:19:31.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 129419 9.40 MB/sec warmup 76 sec latency 13.230 ms 2023-07-29T16:19:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:32 smithi039 ceph-mon[128858]: pgmap v492: 97 pgs: 97 active+clean; 46 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 213 op/s 2023-07-29T16:19:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:32 smithi156 ceph-mon[129245]: pgmap v492: 97 pgs: 97 active+clean; 46 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 213 op/s 2023-07-29T16:19:32.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 131565 9.46 MB/sec warmup 77 sec latency 16.481 ms 2023-07-29T16:19:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:32 smithi031 ceph-mon[95504]: pgmap v492: 97 pgs: 97 active+clean; 46 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.8 MiB/s wr, 213 op/s 2023-07-29T16:19:33.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 132809 9.41 MB/sec warmup 78 sec latency 57.989 ms 2023-07-29T16:19:34.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:34 smithi039 ceph-mon[128858]: pgmap v493: 97 pgs: 97 active+clean; 49 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 299 op/s 2023-07-29T16:19:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:34 smithi156 ceph-mon[129245]: pgmap v493: 97 pgs: 97 active+clean; 49 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 299 op/s 2023-07-29T16:19:34.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 135063 9.46 MB/sec warmup 79 sec latency 18.317 ms 2023-07-29T16:19:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:34 smithi031 ceph-mon[95504]: pgmap v493: 97 pgs: 97 active+clean; 49 MiB data, 676 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.5 MiB/s wr, 299 op/s 2023-07-29T16:19:35.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:19:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 -- 172.21.15.31:0/1483936178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4101510 msgr2=0x7f2ad4101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/1483936178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4101510 0x7f2ad4101960 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009990 tx=0x7f2ac8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 -- 172.21.15.31:0/1483936178 shutdown_connections 2023-07-29T16:19:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/1483936178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/1483936178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4100320 0x7f2ad4100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/1483936178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4101510 0x7f2ad4101960 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 -- 172.21.15.31:0/1483936178 >> 172.21.15.31:0/1483936178 conn(0x7f2ad40fb920 msgr2=0x7f2ad40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 -- 172.21.15.31:0/1483936178 shutdown_connections 2023-07-29T16:19:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.537+0000 7f2ada1ff700 1 -- 172.21.15.31:0/1483936178 wait complete. 2023-07-29T16:19:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.539+0000 7f2ada1ff700 1 Processor -- start 2023-07-29T16:19:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.539+0000 7f2ada1ff700 1 -- start start 2023-07-29T16:19:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.539+0000 7f2ada1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 0x7f2ad4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ada1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 0x7f2ad4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ada1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ada1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad4073e20 con 0x7f2ad4101510 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ada1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad4073f90 con 0x7f2ad4101ea0 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ada1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad41a00b0 con 0x7f2ad4100320 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 0x7f2ad4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 0x7f2ad4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 0x7f2ad4074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44840/0 (socket says 172.21.15.31:44840) 2023-07-29T16:19:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 0x7f2ad4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33190/0 (socket says 172.21.15.31:33190) 2023-07-29T16:19:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad37fe700 1 -- 172.21.15.31:0/153946650 learned_addr learned my addr 172.21.15.31:0/153946650 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49646/0 (socket says 172.21.15.31:49646) 2023-07-29T16:19:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/153946650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 msgr2=0x7f2ad4074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 0x7f2ad4074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/153946650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 msgr2=0x7f2ad4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/153946650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac80095f0 con 0x7f2ad4101510 2023-07-29T16:19:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 0x7f2ad4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.540+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.541+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 0x7f2ad4073140 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800b680 tx=0x7f2ac8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.541+0000 7f2ad0ff9700 1 -- 172.21.15.31:0/153946650 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac801d070 con 0x7f2ad4101510 2023-07-29T16:19:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.541+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad41a0330 con 0x7f2ad4101510 2023-07-29T16:19:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.541+0000 7f2ad0ff9700 1 -- 172.21.15.31:0/153946650 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ac8021410 con 0x7f2ad4101510 2023-07-29T16:19:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.541+0000 7f2ad0ff9700 1 -- 172.21.15.31:0/153946650 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8017650 con 0x7f2ad4101510 2023-07-29T16:19:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.541+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad41a0800 con 0x7f2ad4101510 2023-07-29T16:19:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.546+0000 7f2ad0ff9700 1 -- 172.21.15.31:0/153946650 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2ac8021810 con 0x7f2ad4101510 2023-07-29T16:19:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.547+0000 7f2aeda36b80 1 Processor -- start 2023-07-29T16:19:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 -- start start 2023-07-29T16:19:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c045ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c046530 0x560a0c046d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c047740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0bf9da10 con 0x560a0c046530 2023-07-29T16:19:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0bf9b3d0 con 0x560a0c0472f0 2023-07-29T16:19:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0bf96cc0 con 0x560a0c045be0 2023-07-29T16:19:35.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c045ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c045ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44848/0 (socket says 172.21.15.31:44848) 2023-07-29T16:19:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2ad3fff700 1 -- 172.21.15.31:0/622814368 learned_addr learned my addr 172.21.15.31:0/622814368 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c046530 0x560a0c046d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.548+0000 7f2ad0ff9700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2abc0a0790 0x7f2abc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c047740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 -- 172.21.15.31:0/622814368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 msgr2=0x560a0c047740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c047740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad0ff9700 1 -- 172.21.15.31:0/153946650 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2ac80ccde0 con 0x7f2ad4101510 2023-07-29T16:19:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 -- 172.21.15.31:0/622814368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c046530 msgr2=0x560a0c046d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c046530 0x560a0c046d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 -- 172.21.15.31:0/622814368 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a0bdc4de0 con 0x560a0c045be0 2023-07-29T16:19:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2abc0a0790 0x7f2abc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.549+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c045ff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2acc003f70 tx=0x7f2acc004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c292f92df401a7b9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2ab9ffb700 1 -- 172.21.15.31:0/622814368 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0090b0 con 0x560a0c045be0 2023-07-29T16:19:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2ab9ffb700 1 -- 172.21.15.31:0/622814368 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2acc00f040 con 0x560a0c045be0 2023-07-29T16:19:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2abc0a0790 0x7f2abc0a2c40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2acc022000 tx=0x7f2acc01f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2ab9ffb700 1 -- 172.21.15.31:0/622814368 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc018820 con 0x560a0c045be0 2023-07-29T16:19:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 -- 172.21.15.31:0/622814368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 msgr2=0x560a0c045ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c045ff0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2acc003f70 tx=0x7f2acc004320 comp rx=0 tx=0).stop 2023-07-29T16:19:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 -- 172.21.15.31:0/622814368 shutdown_connections 2023-07-29T16:19:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c047740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c045ff0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/622814368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c046530 0x560a0c046d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.550+0000 7f2aeda36b80 1 -- 172.21.15.31:0/622814368 >> 172.21.15.31:0/622814368 conn(0x560a0c03b3e0 msgr2=0x560a0c040990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.551+0000 7f2aeda36b80 1 -- 172.21.15.31:0/622814368 shutdown_connections 2023-07-29T16:19:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.551+0000 7f2aeda36b80 1 -- 172.21.15.31:0/622814368 wait complete. 2023-07-29T16:19:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.551+0000 7f2aeda36b80 1 -- start start 2023-07-29T16:19:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.551+0000 7f2aeda36b80 1 Processor -- start 2023-07-29T16:19:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c045be0 0x560a0c04f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c04fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c056e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0bf33510 con 0x560a0c045be0 2023-07-29T16:19:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0bf331b0 con 0x560a0c0472f0 2023-07-29T16:19:35.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0be1a920 con 0x560a0c046530 2023-07-29T16:19:35.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2aeda36b80 10 client.0 _ll_register_callbacks cb 0x560a0bf3c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:35.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c045be0 0x560a0c04f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c04fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c056e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c04fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44858/0 (socket says 172.21.15.31:44858) 2023-07-29T16:19:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c056e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49658/0 (socket says 172.21.15.31:49658) 2023-07-29T16:19:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/810603794 learned_addr learned my addr 172.21.15.31:0/810603794 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad37fe700 1 -- 172.21.15.31:0/810603794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 msgr2=0x560a0c04fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c04fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad37fe700 1 -- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 msgr2=0x560a0c056e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c056e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad37fe700 1 -- 172.21.15.31:0/810603794 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2acc003b70 con 0x560a0c045be0 2023-07-29T16:19:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.552+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c04fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c045be0 0x560a0c04f890 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x560a0c046940 tx=0x7f2ac4009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4eca48de4dcd0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2ab2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:19:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/810603794 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4016070 con 0x560a0c045be0 2023-07-29T16:19:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a0bfc1d10 con 0x560a0c045be0 2023-07-29T16:19:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/810603794 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2ac4004320 con 0x560a0c045be0 2023-07-29T16:19:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/810603794 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ac40044e0 con 0x560a0c045be0 2023-07-29T16:19:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2aeda36b80 10 client.16767 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.553+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560a0bdadc60 con 0x560a0c045be0 2023-07-29T16:19:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/810603794 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ac4028080 con 0x560a0c045be0 2023-07-29T16:19:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/810603794 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2ac4003cc0 con 0x560a0c045be0 2023-07-29T16:19:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2aeda36b80 10 client.16767 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2aeda36b80 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2aeda36b80 20 client.16767 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c05b6b0 0x560a0c05dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2aeda36b80 4 client.16767 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a0c014d70 con 0x560a0c05b6b0 2023-07-29T16:19:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.554+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c05b6b0 0x560a0c05dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.555+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c05b6b0 0x560a0c05dac0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=985de6e4984e1362 server_cookie=cdb9f3df1f8efef9 in_seq=0 out_seq=0 2023-07-29T16:19:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.555+0000 7f2ab2ffd700 10 client.16767 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/810603794 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560a0c014d70 con 0x560a0c05b6b0 2023-07-29T16:19:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2ab2ffd700 10 client.16767 handle_command_reply: tid=0 2023-07-29T16:19:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 client.16767 shutdown 2023-07-29T16:19:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c045be0 msgr2=0x560a0c04f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c045be0 0x560a0c04f890 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x560a0c046940 tx=0x7f2ac4009ed0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 shutdown_connections 2023-07-29T16:19:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c05b6b0 0x560a0c05dac0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c0472f0 0x560a0c056e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c045be0 0x560a0c04f890 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2ab2ffd700 0 client.16767 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/810603794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c04fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 >> 172.21.15.31:0/810603794 conn(0x560a0c03b3e0 msgr2=0x560a0c040060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 shutdown_connections 2023-07-29T16:19:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.556+0000 7f2aeda36b80 1 -- 172.21.15.31:0/810603794 wait complete. 2023-07-29T16:19:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.557+0000 7f2aeda36b80 20 client.16767 trim_cache size 0 max 16384 2023-07-29T16:19:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 Processor -- start 2023-07-29T16:19:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 -- start start 2023-07-29T16:19:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c04a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 0x560a0c056dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c04a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0c04eb80 con 0x560a0c0472f0 2023-07-29T16:19:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0c058a80 con 0x560a0c046530 2023-07-29T16:19:35.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0c03d460 con 0x560a0c045be0 2023-07-29T16:19:35.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2ad2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 0x560a0c056dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2ad2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 0x560a0c056dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49668/0 (socket says 172.21.15.31:49668) 2023-07-29T16:19:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/868129739 learned_addr learned my addr 172.21.15.31:0/868129739 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c04a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.575+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c04a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/868129739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 msgr2=0x560a0c04a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c04a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/868129739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 msgr2=0x560a0c04a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c04a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad2ffd700 1 -- 172.21.15.31:0/868129739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a0bdeb250 con 0x560a0c046530 2023-07-29T16:19:35.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c04a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:35.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 0x560a0c056dd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8000c00 tx=0x7f2ac80ca6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aacaa89292da6e3a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2aba7fc700 1 -- 172.21.15.31:0/868129739 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8099e70 con 0x560a0c046530 2023-07-29T16:19:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2aba7fc700 1 -- 172.21.15.31:0/868129739 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ac809a510 con 0x560a0c046530 2023-07-29T16:19:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.576+0000 7f2aba7fc700 1 -- 172.21.15.31:0/868129739 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac80ca930 con 0x560a0c046530 2023-07-29T16:19:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 -- 172.21.15.31:0/868129739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 msgr2=0x560a0c056dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 0x560a0c056dd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8000c00 tx=0x7f2ac80ca6f0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 -- 172.21.15.31:0/868129739 shutdown_connections 2023-07-29T16:19:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c04a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c045be0 0x560a0c04a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/868129739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c046530 0x560a0c056dd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 -- 172.21.15.31:0/868129739 >> 172.21.15.31:0/868129739 conn(0x560a0c03b3e0 msgr2=0x560a0c04e470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 -- 172.21.15.31:0/868129739 shutdown_connections 2023-07-29T16:19:35.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 -- 172.21.15.31:0/868129739 wait complete. 2023-07-29T16:19:35.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 -- start start 2023-07-29T16:19:35.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.577+0000 7f2aeda36b80 1 Processor -- start 2023-07-29T16:19:35.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 0x560a0c056d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c055350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c055930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0c056090 con 0x560a0c0472f0 2023-07-29T16:19:35.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0c056200 con 0x560a0c045be0 2023-07-29T16:19:35.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a0c0ad2b0 con 0x560a0c046530 2023-07-29T16:19:35.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2aeda36b80 10 client.0 _ll_register_callbacks cb 0x560a0bf3c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:35.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 0x560a0c056d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 0x560a0c056d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49676/0 (socket says 172.21.15.31:49676) 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2ad37fe700 1 -- 172.21.15.31:0/2218365296 learned_addr learned my addr 172.21.15.31:0/2218365296 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c055350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.578+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c055930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad37fe700 1 -- 172.21.15.31:0/2218365296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 msgr2=0x560a0c055350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c055350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad37fe700 1 -- 172.21.15.31:0/2218365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 msgr2=0x560a0c055930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:35.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c055930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad37fe700 1 -- 172.21.15.31:0/2218365296 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad41a0800 con 0x560a0c045be0 2023-07-29T16:19:35.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad3fff700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c055930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:35.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ad37fe700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 0x560a0c056d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4006010 tx=0x7f2ac4005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f8d056168af352 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ab2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:35.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/2218365296 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4016050 con 0x560a0c045be0 2023-07-29T16:19:35.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2acc00f040 con 0x560a0c045be0 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/2218365296 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2ac4000e40 con 0x560a0c045be0 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/2218365296 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ac401a720 con 0x560a0c045be0 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.579+0000 7f2aeda36b80 10 client.26771 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560a0bdc32b0 con 0x560a0c045be0 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/2218365296 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ac403a020 con 0x560a0c045be0 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/2218365296 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2ac4003ad0 con 0x560a0c045be0 2023-07-29T16:19:35.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 10 client.26771 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:35.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 10 client.26771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:35.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 20 client.26771 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:35.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c0aeca0 0x560a0c0b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 4 client.26771 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:19:35.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560a0bf8fd40 con 0x560a0c0aeca0 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.580+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c0aeca0 0x560a0c0b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.581+0000 7f2ad2ffd700 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c0aeca0 0x560a0c0b10b0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a692d4fac97077f6 server_cookie=5eb98d945518173 in_seq=0 out_seq=0 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.581+0000 7f2ab2ffd700 10 client.26771 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.582+0000 7f2ab2ffd700 1 -- 172.21.15.31:0/2218365296 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560a0bf8fd40 con 0x560a0c0aeca0 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2ab2ffd700 10 client.26771 handle_command_reply: tid=0 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 client.26771 shutdown 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 msgr2=0x560a0c056d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 0x560a0c056d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4006010 tx=0x7f2ac4005bb0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 shutdown_connections 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560a0c0aeca0 0x560a0c0b10b0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a0c0472f0 0x560a0c055930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2ab2ffd700 0 client.26771 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560a0c045be0 0x560a0c056d40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 --2- 172.21.15.31:0/2218365296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560a0c046530 0x560a0c055350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 >> 172.21.15.31:0/2218365296 conn(0x560a0c03b3e0 msgr2=0x560a0c04a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 shutdown_connections 2023-07-29T16:19:35.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 1 -- 172.21.15.31:0/2218365296 wait complete. 2023-07-29T16:19:35.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.583+0000 7f2aeda36b80 20 client.26771 trim_cache size 0 max 16384 2023-07-29T16:19:35.606 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:19:35.606 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:19:35.606 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:19:35.606 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:19:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2abc0a0790 msgr2=0x7f2abc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2abc0a0790 0x7f2abc0a2c40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2acc022000 tx=0x7f2acc01f000 comp rx=0 tx=0).stop 2023-07-29T16:19:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 msgr2=0x7f2ad4073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 0x7f2ad4073140 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800b680 tx=0x7f2ac8009580 comp rx=0 tx=0).stop 2023-07-29T16:19:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 shutdown_connections 2023-07-29T16:19:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2abc0a0790 0x7f2abc0a2c40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2ad4101ea0 0x7f2ad4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ad4100320 0x7f2ad4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 --2- 172.21.15.31:0/153946650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2ad4101510 0x7f2ad4073140 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 >> 172.21.15.31:0/153946650 conn(0x7f2ad40fb920 msgr2=0x7f2ad40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 shutdown_connections 2023-07-29T16:19:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.584+0000 7f2ada1ff700 1 -- 172.21.15.31:0/153946650 wait complete. 2023-07-29T16:19:35.610 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:19:35.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:19:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 -- 172.21.15.31:0/3966715636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74100dd0 msgr2=0x7f2b741031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 --2- 172.21.15.31:0/3966715636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74100dd0 0x7f2b741031b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009990 tx=0x7f2b64009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 -- 172.21.15.31:0/3966715636 shutdown_connections 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 --2- 172.21.15.31:0/3966715636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b74106010 0x7f2b74108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 --2- 172.21.15.31:0/3966715636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74100dd0 0x7f2b741031b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 --2- 172.21.15.31:0/3966715636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74105ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 -- 172.21.15.31:0/3966715636 >> 172.21.15.31:0/3966715636 conn(0x7f2b740fa810 msgr2=0x7f2b740fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 -- 172.21.15.31:0/3966715636 shutdown_connections 2023-07-29T16:19:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.829+0000 7f2b791ff700 1 -- 172.21.15.31:0/3966715636 wait complete. 2023-07-29T16:19:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.830+0000 7f2b791ff700 1 Processor -- start 2023-07-29T16:19:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.830+0000 7f2b791ff700 1 -- start start 2023-07-29T16:19:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b791ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b74100dd0 0x7f2b74196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b791ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b791ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74106010 0x7f2b7419d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b791ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7419dba0 con 0x7f2b74100dd0 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b791ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7419dce0 con 0x7f2b74106010 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b791ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7419de70 con 0x7f2b741036f0 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b7259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b72d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b74100dd0 0x7f2b74196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b7259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44894/0 (socket says 172.21.15.31:44894) 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b7259c700 1 -- 172.21.15.31:0/350224208 learned_addr learned my addr 172.21.15.31:0/350224208 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.831+0000 7f2b7359e700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74106010 0x7f2b7419d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b7259c700 1 -- 172.21.15.31:0/350224208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74106010 msgr2=0x7f2b7419d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b7259c700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74106010 0x7f2b7419d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b7259c700 1 -- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b74100dd0 msgr2=0x7f2b74196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b7259c700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b74100dd0 0x7f2b74196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b7259c700 1 -- 172.21.15.31:0/350224208 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b640095f0 con 0x7f2b741036f0 2023-07-29T16:19:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b7259c700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74196560 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800e9a0 tx=0x7f2b6800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b68009790 con 0x7f2b741036f0 2023-07-29T16:19:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b68010430 con 0x7f2b741036f0 2023-07-29T16:19:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7419e150 con 0x7f2b741036f0 2023-07-29T16:19:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b68018680 con 0x7f2b741036f0 2023-07-29T16:19:35.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.832+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7419e660 con 0x7f2b741036f0 2023-07-29T16:19:35.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.834+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b740fc380 con 0x7f2b741036f0 2023-07-29T16:19:35.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.836+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2b680105a0 con 0x7f2b741036f0 2023-07-29T16:19:35.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.838+0000 7f2b5bfff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2b5c0a07e0 0x7f2b5c0a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:35.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.838+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2b68014070 con 0x7f2b741036f0 2023-07-29T16:19:35.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.838+0000 7f2b72d9d700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2b5c0a07e0 0x7f2b5c0a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:35.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.839+0000 7f2b72d9d700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2b5c0a07e0 0x7f2b5c0a2c90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b2e0 tx=0x7f2b64005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:35.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:35.842+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f2b68051720 con 0x7f2b741036f0 2023-07-29T16:19:35.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 136395 9.41 MB/sec warmup 80 sec latency 17.336 ms 2023-07-29T16:19:36.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.143+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b7402cc30 con 0x7f2b741036f0 2023-07-29T16:19:36.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.144+0000 7f2b5bfff700 1 -- 172.21.15.31:0/350224208 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f2b680957c0 con 0x7f2b741036f0 2023-07-29T16:19:36.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:19:36.143 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:18:33.432992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:19:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.146+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2b5c0a07e0 msgr2=0x7f2b5c0a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.146+0000 7f2b791ff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2b5c0a07e0 0x7f2b5c0a2c90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400b2e0 tx=0x7f2b64005fb0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.146+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 msgr2=0x7f2b74196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.146+0000 7f2b791ff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74196560 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800e9a0 tx=0x7f2b6800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.146+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 shutdown_connections 2023-07-29T16:19:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.146+0000 7f2b791ff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2b5c0a07e0 0x7f2b5c0a2c90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.147+0000 7f2b791ff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2b74106010 0x7f2b7419d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.147+0000 7f2b791ff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b74100dd0 0x7f2b74196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.147+0000 7f2b791ff700 1 --2- 172.21.15.31:0/350224208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b741036f0 0x7f2b74196560 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.147+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 >> 172.21.15.31:0/350224208 conn(0x7f2b740fa810 msgr2=0x7f2b74101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.147+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 shutdown_connections 2023-07-29T16:19:36.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.147+0000 7f2b791ff700 1 -- 172.21.15.31:0/350224208 wait complete. 2023-07-29T16:19:36.147 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2023-07-29T16:19:36.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:19:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.377+0000 7f54aefff700 1 -- 172.21.15.31:0/1116405736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 msgr2=0x7f54a8103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.377+0000 7f54aefff700 1 --2- 172.21.15.31:0/1116405736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a8103180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f549c009990 tx=0x7f549c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 -- 172.21.15.31:0/1116405736 shutdown_connections 2023-07-29T16:19:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 --2- 172.21.15.31:0/1116405736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a8105fe0 0x7f54a8108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 --2- 172.21.15.31:0/1116405736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a8103180 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 --2- 172.21.15.31:0/1116405736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54a81036c0 0x7f54a8105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 -- 172.21.15.31:0/1116405736 >> 172.21.15.31:0/1116405736 conn(0x7f54a80fa840 msgr2=0x7f54a80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 -- 172.21.15.31:0/1116405736 shutdown_connections 2023-07-29T16:19:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.378+0000 7f54aefff700 1 -- 172.21.15.31:0/1116405736 wait complete. 2023-07-29T16:19:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 Processor -- start 2023-07-29T16:19:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 -- start start 2023-07-29T16:19:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a81036c0 0x7f54a81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54a8105fe0 0x7f54a819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a819fe30 con 0x7f54a81036c0 2023-07-29T16:19:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a819ff70 con 0x7f54a8105fe0 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54aefff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a81a0100 con 0x7f54a8100da0 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a81036c0 0x7f54a81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a81982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44908/0 (socket says 172.21.15.31:44908) 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.379+0000 7f54acd9b700 1 -- 172.21.15.31:0/1013928272 learned_addr learned my addr 172.21.15.31:0/1013928272 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54ad59c700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54a8105fe0 0x7f54a819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54acd9b700 1 -- 172.21.15.31:0/1013928272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54a8105fe0 msgr2=0x7f54a819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54acd9b700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54a8105fe0 0x7f54a819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54acd9b700 1 -- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a81036c0 msgr2=0x7f54a81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54acd9b700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a81036c0 0x7f54a81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54acd9b700 1 -- 172.21.15.31:0/1013928272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549c0095f0 con 0x7f54a8100da0 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54a7fff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a81036c0 0x7f54a81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54acd9b700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a81982b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f549c004740 tx=0x7f549c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54a5ffb700 1 -- 172.21.15.31:0/1013928272 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c01d070 con 0x7f54a8100da0 2023-07-29T16:19:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a81a0380 con 0x7f54a8100da0 2023-07-29T16:19:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54a5ffb700 1 -- 172.21.15.31:0/1013928272 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549c004830 con 0x7f54a8100da0 2023-07-29T16:19:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.380+0000 7f54a5ffb700 1 -- 172.21.15.31:0/1013928272 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c0053c0 con 0x7f54a8100da0 2023-07-29T16:19:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.381+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81a07a0 con 0x7f54a8100da0 2023-07-29T16:19:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.384+0000 7f54a5ffb700 1 -- 172.21.15.31:0/1013928272 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f549c005560 con 0x7f54a8100da0 2023-07-29T16:19:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.385+0000 7f54c29dab80 1 Processor -- start 2023-07-29T16:19:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.385+0000 7f54c29dab80 1 -- start start 2023-07-29T16:19:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a23ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a24530 0x55c699a24d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a252f0 0x55c699a25740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54c29dab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c69997ba10 con 0x55c699a23be0 2023-07-29T16:19:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54c29dab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6999793d0 con 0x55c699a252f0 2023-07-29T16:19:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54c29dab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699974cc0 con 0x55c699a24530 2023-07-29T16:19:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a5ffb700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54980a0790 0x7f54980a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a24530 0x55c699a24d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a252f0 0x55c699a25740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a24530 0x55c699a24d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44914/0 (socket says 172.21.15.31:44914) 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54acd9b700 1 -- 172.21.15.31:0/4145851998 learned_addr learned my addr 172.21.15.31:0/4145851998 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54ad59c700 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a23ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a7fff700 1 -- 172.21.15.31:0/4145851998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a24530 msgr2=0x55c699a24d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a7fff700 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a24530 0x55c699a24d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54ad59c700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54980a0790 0x7f54980a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a7fff700 1 -- 172.21.15.31:0/4145851998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 msgr2=0x55c699a23ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a7fff700 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a23ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a7fff700 1 -- 172.21.15.31:0/4145851998 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6997a2de0 con 0x55c699a252f0 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.386+0000 7f54a5ffb700 1 -- 172.21.15.31:0/1013928272 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f549c0ccc90 con 0x7f54a8100da0 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.387+0000 7f54a7fff700 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a252f0 0x55c699a25740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54940114f0 tx=0x7f5494011800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=353f5bb2503d9304 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.387+0000 7f5492ffd700 1 -- 172.21.15.31:0/4145851998 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494004230 con 0x55c699a252f0 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.387+0000 7f5492ffd700 1 -- 172.21.15.31:0/4145851998 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5494004cf0 con 0x55c699a252f0 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.387+0000 7f5492ffd700 1 -- 172.21.15.31:0/4145851998 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940056c0 con 0x55c699a252f0 2023-07-29T16:19:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 -- 172.21.15.31:0/4145851998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a252f0 msgr2=0x55c699a25740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54ad59c700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54980a0790 0x7f54980a2c40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009ec0 tx=0x7f54a000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a252f0 0x55c699a25740 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f54940114f0 tx=0x7f5494011800 comp rx=0 tx=0).stop 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 -- 172.21.15.31:0/4145851998 shutdown_connections 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a252f0 0x55c699a25740 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a23ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 --2- 172.21.15.31:0/4145851998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a24530 0x55c699a24d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 -- 172.21.15.31:0/4145851998 >> 172.21.15.31:0/4145851998 conn(0x55c699a193e0 msgr2=0x55c699a1e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 -- 172.21.15.31:0/4145851998 shutdown_connections 2023-07-29T16:19:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.388+0000 7f54c29dab80 1 -- 172.21.15.31:0/4145851998 wait complete. 2023-07-29T16:19:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.389+0000 7f54c29dab80 1 -- start start 2023-07-29T16:19:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.389+0000 7f54c29dab80 1 Processor -- start 2023-07-29T16:19:36.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a2d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a2dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a34dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699911510 con 0x55c699a24530 2023-07-29T16:19:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6999111b0 con 0x55c699a23be0 2023-07-29T16:19:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6997f8920 con 0x55c699a252f0 2023-07-29T16:19:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a2d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54c29dab80 10 client.0 _ll_register_callbacks cb 0x55c69991a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a2dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a2dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33252/0 (socket says 172.21.15.31:33252) 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a2d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49742/0 (socket says 172.21.15.31:49742) 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54a7fff700 1 -- 172.21.15.31:0/928732986 learned_addr learned my addr 172.21.15.31:0/928732986 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54ad59c700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a34dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 -- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 msgr2=0x55c699a34dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a34dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 -- 172.21.15.31:0/928732986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 msgr2=0x55c699a2dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a2dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.390+0000 7f54acd9b700 1 -- 172.21.15.31:0/928732986 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5494009c90 con 0x55c699a23be0 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f54a7fff700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a2dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f54acd9b700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a2d850 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f549c0968c0 tx=0x7f549c096490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5abd1f5001b260cc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f5487fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f5487fff700 1 -- 172.21.15.31:0/928732986 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c09e230 con 0x55c699a23be0 2023-07-29T16:19:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f5487fff700 1 -- 172.21.15.31:0/928732986 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f549c09a3c0 con 0x55c699a23be0 2023-07-29T16:19:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c69999fd10 con 0x55c699a23be0 2023-07-29T16:19:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f5487fff700 1 -- 172.21.15.31:0/928732986 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f549c099f00 con 0x55c699a23be0 2023-07-29T16:19:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f54c29dab80 10 client.26789 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c69978bc60 con 0x55c699a23be0 2023-07-29T16:19:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.391+0000 7f5487fff700 1 -- 172.21.15.31:0/928732986 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549c0b6080 con 0x55c699a23be0 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f5487fff700 1 -- 172.21.15.31:0/928732986 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f549c0998b0 con 0x55c699a23be0 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54c29dab80 10 client.26789 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54c29dab80 10 client.26789 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54c29dab80 20 client.26789 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54c29dab80 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a39240 0x55c699a3b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54c29dab80 4 client.26789 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6999f2d70 con 0x55c699a39240 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.392+0000 7f54a7fff700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a39240 0x55c699a3b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.393+0000 7f54a7fff700 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a39240 0x55c699a3b620 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e174353fa2da9c40 server_cookie=d669cce59e721743 in_seq=0 out_seq=0 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.393+0000 7f5487fff700 10 client.26789 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.393+0000 7f5487fff700 1 -- 172.21.15.31:0/928732986 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c6999f2d70 con 0x55c699a39240 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.393+0000 7f5487fff700 10 client.26789 handle_command_reply: tid=0 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.393+0000 7f54c29dab80 1 client.26789 shutdown 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 msgr2=0x55c699a2d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a2d850 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f549c0968c0 tx=0x7f549c096490 comp rx=0 tx=0).stop 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 shutdown_connections 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a34dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a39240 0x55c699a3b620 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a2d850 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f5487fff700 0 client.26789 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 --2- 172.21.15.31:0/928732986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a2dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 >> 172.21.15.31:0/928732986 conn(0x55c699a193e0 msgr2=0x55c699a1e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 shutdown_connections 2023-07-29T16:19:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 1 -- 172.21.15.31:0/928732986 wait complete. 2023-07-29T16:19:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.394+0000 7f54c29dab80 20 client.26789 trim_cache size 0 max 16384 2023-07-29T16:19:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.408+0000 7f54c29dab80 1 Processor -- start 2023-07-29T16:19:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.408+0000 7f54c29dab80 1 -- start start 2023-07-29T16:19:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.408+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a1f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.408+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 0x55c699a34d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a1e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54c29dab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699a2cd00 con 0x55c699a23be0 2023-07-29T16:19:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54c29dab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699a1b460 con 0x55c699a24530 2023-07-29T16:19:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54c29dab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699a24a10 con 0x55c699a252f0 2023-07-29T16:19:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 0x55c699a34d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a1f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 0x55c699a34d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49752/0 (socket says 172.21.15.31:49752) 2023-07-29T16:19:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54ad59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a1e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 -- 172.21.15.31:0/3406023648 learned_addr learned my addr 172.21.15.31:0/3406023648 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 -- 172.21.15.31:0/3406023648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 msgr2=0x55c699a1e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a1e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 -- 172.21.15.31:0/3406023648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 msgr2=0x55c699a1f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a1f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 -- 172.21.15.31:0/3406023648 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6997c9250 con 0x55c699a24530 2023-07-29T16:19:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.409+0000 7f54a7fff700 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 0x55c699a34d90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f54940114f0 tx=0x7f5494009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5521de597093704b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.410+0000 7f54937fe700 1 -- 172.21.15.31:0/3406023648 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54940055c0 con 0x55c699a24530 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.410+0000 7f54937fe700 1 -- 172.21.15.31:0/3406023648 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5494004b80 con 0x55c699a24530 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.410+0000 7f54937fe700 1 -- 172.21.15.31:0/3406023648 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549400eab0 con 0x55c699a24530 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.410+0000 7f54c29dab80 1 -- 172.21.15.31:0/3406023648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 msgr2=0x55c699a34d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.410+0000 7f54c29dab80 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 0x55c699a34d90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f54940114f0 tx=0x7f5494009750 comp rx=0 tx=0).stop 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 -- 172.21.15.31:0/3406023648 shutdown_connections 2023-07-29T16:19:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a1e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a23be0 0x55c699a1f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 --2- 172.21.15.31:0/3406023648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a24530 0x55c699a34d90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 -- 172.21.15.31:0/3406023648 >> 172.21.15.31:0/3406023648 conn(0x55c699a193e0 msgr2=0x55c699a2c510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 -- 172.21.15.31:0/3406023648 shutdown_connections 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 -- 172.21.15.31:0/3406023648 wait complete. 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 -- start start 2023-07-29T16:19:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.411+0000 7f54c29dab80 1 Processor -- start 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a349e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a31480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a319c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699a31f00 con 0x55c699a24530 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699a32070 con 0x55c699a23be0 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c699a8ad90 con 0x55c699a252f0 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54c29dab80 10 client.0 _ll_register_callbacks cb 0x55c69991a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a31480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54acd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54a7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a31480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33266/0 (socket says 172.21.15.31:33266) 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54a7fff700 1 -- 172.21.15.31:0/1357112484 learned_addr learned my addr 172.21.15.31:0/1357112484 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54ad59c700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54acd9b700 1 -- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 msgr2=0x55c699a319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54acd9b700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a319c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54acd9b700 1 -- 172.21.15.31:0/1357112484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 msgr2=0x55c699a31480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54acd9b700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a31480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.412+0000 7f54acd9b700 1 -- 172.21.15.31:0/1357112484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549401d040 con 0x55c699a23be0 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f54a7fff700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a31480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f5487fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f54acd9b700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a349e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f549c005f50 tx=0x7f549c0a22f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30bbb35a23099b4d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f5487fff700 1 -- 172.21.15.31:0/1357112484 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c09a3c0 con 0x55c699a23be0 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5494004cf0 con 0x55c699a23be0 2023-07-29T16:19:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f5487fff700 1 -- 172.21.15.31:0/1357112484 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f549c09e3f0 con 0x55c699a23be0 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.413+0000 7f5487fff700 1 -- 172.21.15.31:0/1357112484 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f549c0a2780 con 0x55c699a23be0 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 10 client.26795 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55c6997a12b0 con 0x55c699a23be0 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f5487fff700 1 -- 172.21.15.31:0/1357112484 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5494004cf0 con 0x55c699a23be0 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f5487fff700 1 -- 172.21.15.31:0/1357112484 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f549c0ca1a0 con 0x55c699a23be0 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 10 client.26795 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 10 client.26795 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 20 client.26795 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a8c110 0x55c699a8e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 4 client.26795 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c69996dd40 con 0x55c699a8c110 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.414+0000 7f54a7fff700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a8c110 0x55c699a8e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f5487fff700 10 client.26795 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54a7fff700 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a8c110 0x55c699a8e4f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87106ce7727c4781 server_cookie=a300c73f0431899a in_seq=0 out_seq=0 2023-07-29T16:19:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f5487fff700 1 -- 172.21.15.31:0/1357112484 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c69996dd40 con 0x55c699a8c110 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f5487fff700 10 client.26795 handle_command_reply: tid=0 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 client.26795 shutdown 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 msgr2=0x55c699a349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a349e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f549c005f50 tx=0x7f549c0a22f0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 shutdown_connections 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c699a252f0 0x55c699a319c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c699a8c110 0x55c699a8e4f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c699a23be0 0x55c699a349e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 --2- 172.21.15.31:0/1357112484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c699a24530 0x55c699a31480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f5487fff700 0 client.26795 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 >> 172.21.15.31:0/1357112484 conn(0x55c699a193e0 msgr2=0x55c699a1e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 shutdown_connections 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.415+0000 7f54c29dab80 1 -- 172.21.15.31:0/1357112484 wait complete. 2023-07-29T16:19:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54c29dab80 20 client.26795 trim_cache size 0 max 16384 2023-07-29T16:19:36.423 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:19:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54980a0790 msgr2=0x7f54980a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54980a0790 0x7f54980a2c40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f54a0009ec0 tx=0x7f54a000b040 comp rx=0 tx=0).stop 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 msgr2=0x7f54a81982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a81982b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f549c004740 tx=0x7f549c009580 comp rx=0 tx=0).stop 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 shutdown_connections 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54980a0790 0x7f54980a2c40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f54a8105fe0 0x7f54a819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f54a8100da0 0x7f54a81982b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 --2- 172.21.15.31:0/1013928272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f54a81036c0 0x7f54a81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.416+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 >> 172.21.15.31:0/1013928272 conn(0x7f54a80fa840 msgr2=0x7f54a80ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.417+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 shutdown_connections 2023-07-29T16:19:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.417+0000 7f54aefff700 1 -- 172.21.15.31:0/1013928272 wait complete. 2023-07-29T16:19:36.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:19:36.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 -- 172.21.15.31:0/196227845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 msgr2=0x7f42900731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 --2- 172.21.15.31:0/196227845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42900731e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4280009990 tx=0x7f4280009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 -- 172.21.15.31:0/196227845 shutdown_connections 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 --2- 172.21.15.31:0/196227845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4290108210 0x7f429010a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 --2- 172.21.15.31:0/196227845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42900731e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 --2- 172.21.15.31:0/196227845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42900737b0 0x7f4290073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 -- 172.21.15.31:0/196227845 >> 172.21.15.31:0/196227845 conn(0x7f42900fb970 msgr2=0x7f42900fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 -- 172.21.15.31:0/196227845 shutdown_connections 2023-07-29T16:19:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.552+0000 7f42961ff700 1 -- 172.21.15.31:0/196227845 wait complete. 2023-07-29T16:19:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.553+0000 7f42961ff700 1 Processor -- start 2023-07-29T16:19:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.553+0000 7f42961ff700 1 -- start start 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f42961ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42900737b0 0x7f4290198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f42961ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42901988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f42961ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4290108210 0x7f4290199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f42961ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4290199d90 con 0x7f4290108210 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f42961ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4290199f00 con 0x7f42900737b0 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f42961ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429019a0c0 con 0x7f4290074d90 2023-07-29T16:19:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42900737b0 0x7f4290198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4290108210 0x7f4290199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42900737b0 0x7f4290198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49792/0 (socket says 172.21.15.31:49792) 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428f7fe700 1 -- 172.21.15.31:0/913221759 learned_addr learned my addr 172.21.15.31:0/913221759 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428effd700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42901988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428ffff700 1 -- 172.21.15.31:0/913221759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 msgr2=0x7f42901988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428ffff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42901988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428ffff700 1 -- 172.21.15.31:0/913221759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42900737b0 msgr2=0x7f4290198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428ffff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42900737b0 0x7f4290198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428ffff700 1 -- 172.21.15.31:0/913221759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800095f0 con 0x7f4290108210 2023-07-29T16:19:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.554+0000 7f428effd700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42901988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.555+0000 7f428ffff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4290108210 0x7f4290199730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f428800efc0 tx=0x7f428800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.555+0000 7f428cff9700 1 -- 172.21.15.31:0/913221759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4288018070 con 0x7f4290108210 2023-07-29T16:19:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.555+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429019a340 con 0x7f4290108210 2023-07-29T16:19:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.555+0000 7f428cff9700 1 -- 172.21.15.31:0/913221759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f428800f040 con 0x7f4290108210 2023-07-29T16:19:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.555+0000 7f428cff9700 1 -- 172.21.15.31:0/913221759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428800cd40 con 0x7f4290108210 2023-07-29T16:19:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.555+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42901a0b60 con 0x7f4290108210 2023-07-29T16:19:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.560+0000 7f428cff9700 1 -- 172.21.15.31:0/913221759 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4288003a00 con 0x7f4290108210 2023-07-29T16:19:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.561+0000 7f42a9a74b80 1 Processor -- start 2023-07-29T16:19:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.561+0000 7f42a9a74b80 1 -- start start 2023-07-29T16:19:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b61bd6b0 0x5629b63618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6361df0 0x5629b63627b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6362cf0 0x5629b6363160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b62b99b0 con 0x5629b6362cf0 2023-07-29T16:19:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b62b7370 con 0x5629b61bd6b0 2023-07-29T16:19:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b62b2c60 con 0x5629b6361df0 2023-07-29T16:19:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428cff9700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42780a0810 0x7f42780a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b61bd6b0 0x5629b63618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b61bd6b0 0x5629b63618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49802/0 (socket says 172.21.15.31:49802) 2023-07-29T16:19:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428effd700 1 -- 172.21.15.31:0/2736152076 learned_addr learned my addr 172.21.15.31:0/2736152076 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6361df0 0x5629b63627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428effd700 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6362cf0 0x5629b6363160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.562+0000 7f428cff9700 1 -- 172.21.15.31:0/913221759 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f42880cb5d0 con 0x7f4290108210 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428ffff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42780a0810 0x7f42780a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428f7fe700 1 -- 172.21.15.31:0/2736152076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b61bd6b0 msgr2=0x5629b63618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b61bd6b0 0x5629b63618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428f7fe700 1 -- 172.21.15.31:0/2736152076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6362cf0 msgr2=0x5629b6363160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6362cf0 0x5629b6363160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428f7fe700 1 -- 172.21.15.31:0/2736152076 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629b60e0d80 con 0x5629b6361df0 2023-07-29T16:19:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6361df0 0x5629b63627b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4280000c00 tx=0x7f4280005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1415e36681b18562 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f428ffff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42780a0810 0x7f42780a2cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4288009fd0 tx=0x7f4288004dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.563+0000 7f4275ffb700 1 -- 172.21.15.31:0/2736152076 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428001d070 con 0x5629b6361df0 2023-07-29T16:19:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f4275ffb700 1 -- 172.21.15.31:0/2736152076 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f428001e040 con 0x5629b6361df0 2023-07-29T16:19:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f4275ffb700 1 -- 172.21.15.31:0/2736152076 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4280004880 con 0x5629b6361df0 2023-07-29T16:19:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2736152076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6361df0 msgr2=0x5629b63627b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6361df0 0x5629b63627b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4280000c00 tx=0x7f4280005e80 comp rx=0 tx=0).stop 2023-07-29T16:19:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2736152076 shutdown_connections 2023-07-29T16:19:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6362cf0 0x5629b6363160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b61bd6b0 0x5629b63618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2736152076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6361df0 0x5629b63627b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.564+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2736152076 >> 172.21.15.31:0/2736152076 conn(0x5629b6356fd0 msgr2=0x5629b635c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.565+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2736152076 shutdown_connections 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.565+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2736152076 wait complete. 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.565+0000 7f42a9a74b80 1 -- start start 2023-07-29T16:19:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.565+0000 7f42a9a74b80 1 Processor -- start 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b636ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b636b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b63723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b624f4b0 con 0x5629b6361df0 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b624f150 con 0x5629b6362cf0 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b61368c0 con 0x5629b61bd6b0 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f42a9a74b80 10 client.0 _ll_register_callbacks cb 0x5629b62585c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f428effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b636ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b636b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f428effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b636ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44976/0 (socket says 172.21.15.31:44976) 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f428effd700 1 -- 172.21.15.31:0/4017735513 learned_addr learned my addr 172.21.15.31:0/4017735513 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.566+0000 7f428effd700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b63723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428effd700 1 -- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 msgr2=0x5629b63723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428effd700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b63723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428effd700 1 -- 172.21.15.31:0/4017735513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 msgr2=0x5629b636b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428effd700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b636b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428effd700 1 -- 172.21.15.31:0/4017735513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429010b2b0 con 0x5629b61bd6b0 2023-07-29T16:19:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428f7fe700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b636b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f428effd700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b636ae50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4284011ef0 tx=0x7f42840047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a64f3c291eb316a2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f426effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f426effd700 1 -- 172.21.15.31:0/4017735513 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4284016070 con 0x5629b61bd6b0 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629b62ddcb0 con 0x5629b61bd6b0 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f426effd700 1 -- 172.21.15.31:0/4017735513 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f428400fe30 con 0x5629b61bd6b0 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f426effd700 1 -- 172.21.15.31:0/4017735513 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4284004030 con 0x5629b61bd6b0 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f42a9a74b80 10 client.26821 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.567+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5629b60c9c00 con 0x5629b61bd6b0 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f426effd700 1 -- 172.21.15.31:0/4017735513 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f428402f080 con 0x5629b61bd6b0 2023-07-29T16:19:36.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f426effd700 1 -- 172.21.15.31:0/4017735513 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f4284021450 con 0x5629b61bd6b0 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f42a9a74b80 10 client.26821 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f42a9a74b80 10 client.26821 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f42a9a74b80 20 client.26821 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b6376cc0 0x5629b63790d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f42a9a74b80 4 client.26821 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629b6330d10 con 0x5629b6376cc0 2023-07-29T16:19:36.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.568+0000 7f428f7fe700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b6376cc0 0x5629b63790d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.569+0000 7f428f7fe700 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b6376cc0 0x5629b63790d0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=318f3b909ee2df67 server_cookie=675a63d0a1363e07 in_seq=0 out_seq=0 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.569+0000 7f426effd700 10 client.26821 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f426effd700 1 -- 172.21.15.31:0/4017735513 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5629b6330d10 con 0x5629b6376cc0 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f426effd700 10 client.26821 handle_command_reply: tid=0 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 client.26821 shutdown 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 msgr2=0x5629b636ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b636ae50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4284011ef0 tx=0x7f42840047c0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 shutdown_connections 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b6376cc0 0x5629b63790d0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b63723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b636ae50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/4017735513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b636b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f426effd700 0 client.26821 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 >> 172.21.15.31:0/4017735513 conn(0x5629b6356fd0 msgr2=0x5629b635ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 shutdown_connections 2023-07-29T16:19:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 1 -- 172.21.15.31:0/4017735513 wait complete. 2023-07-29T16:19:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.570+0000 7f42a9a74b80 20 client.26821 trim_cache size 0 max 16384 2023-07-29T16:19:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.585+0000 7f42a9a74b80 1 Processor -- start 2023-07-29T16:19:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.585+0000 7f42a9a74b80 1 -- start start 2023-07-29T16:19:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.585+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b6370380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b635c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b6369030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b63621c0 con 0x5629b6361df0 2023-07-29T16:19:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b636c040 con 0x5629b6362cf0 2023-07-29T16:19:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b636a350 con 0x5629b61bd6b0 2023-07-29T16:19:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b635c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b6370380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b635c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33312/0 (socket says 172.21.15.31:33312) 2023-07-29T16:19:36.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b6370380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44992/0 (socket says 172.21.15.31:44992) 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 -- 172.21.15.31:0/1562355010 learned_addr learned my addr 172.21.15.31:0/1562355010 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428effd700 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b6369030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 -- 172.21.15.31:0/1562355010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 msgr2=0x5629b6370380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b6370380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 -- 172.21.15.31:0/1562355010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 msgr2=0x5629b6369030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b6369030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.586+0000 7f428f7fe700 1 -- 172.21.15.31:0/1562355010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629b61071f0 con 0x5629b6361df0 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.587+0000 7f428f7fe700 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b635c2d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f428000b680 tx=0x7f4280005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4101a1ffa05fdba2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.587+0000 7f42767fc700 1 -- 172.21.15.31:0/1562355010 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428001d040 con 0x5629b6361df0 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.587+0000 7f42767fc700 1 -- 172.21.15.31:0/1562355010 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4280005580 con 0x5629b6361df0 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.587+0000 7f42767fc700 1 -- 172.21.15.31:0/1562355010 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428000fd90 con 0x5629b6361df0 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 -- 172.21.15.31:0/1562355010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 msgr2=0x5629b635c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b635c2d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f428000b680 tx=0x7f4280005fb0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 -- 172.21.15.31:0/1562355010 shutdown_connections 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6362cf0 0x5629b6369030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b61bd6b0 0x5629b6370380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/1562355010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b6361df0 0x5629b635c2d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 -- 172.21.15.31:0/1562355010 >> 172.21.15.31:0/1562355010 conn(0x5629b6356fd0 msgr2=0x5629b635ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 -- 172.21.15.31:0/1562355010 shutdown_connections 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 -- 172.21.15.31:0/1562355010 wait complete. 2023-07-29T16:19:36.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 -- start start 2023-07-29T16:19:36.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.588+0000 7f42a9a74b80 1 Processor -- start 2023-07-29T16:19:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 0x5629b636e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 0x5629b636e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6362cf0 0x5629b636f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b636fcb0 con 0x5629b61bd6b0 2023-07-29T16:19:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b636fe20 con 0x5629b6361df0 2023-07-29T16:19:36.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b636ffe0 con 0x5629b6362cf0 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f42a9a74b80 10 client.0 _ll_register_callbacks cb 0x5629b62585c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 0x5629b636e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f428effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 0x5629b636e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f428effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 0x5629b636e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33320/0 (socket says 172.21.15.31:33320) 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f428f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 0x5629b636e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49818/0 (socket says 172.21.15.31:49818) 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f428effd700 1 -- 172.21.15.31:0/2534980817 learned_addr learned my addr 172.21.15.31:0/2534980817 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.589+0000 7f428effd700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6362cf0 0x5629b636f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428f7fe700 1 -- 172.21.15.31:0/2534980817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6362cf0 msgr2=0x5629b636f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6362cf0 0x5629b636f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428f7fe700 1 -- 172.21.15.31:0/2534980817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 msgr2=0x5629b636e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 0x5629b636e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428f7fe700 1 -- 172.21.15.31:0/2534980817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629b60e0d80 con 0x5629b6361df0 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428effd700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 0x5629b636e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f428f7fe700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 0x5629b636e760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4280009ce0 tx=0x7f42800056f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3e91578e761801 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f426effd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f426effd700 1 -- 172.21.15.31:0/2534980817 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428001d070 con 0x5629b6361df0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f428001e040 con 0x5629b6361df0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.590+0000 7f426effd700 1 -- 172.21.15.31:0/2534980817 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4280031070 con 0x5629b6361df0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f426effd700 1 -- 172.21.15.31:0/2534980817 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42800052c0 con 0x5629b6361df0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 10 client.26813 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5629b60df250 con 0x5629b6361df0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f426effd700 1 -- 172.21.15.31:0/2534980817 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f428001e040 con 0x5629b6361df0 2023-07-29T16:19:36.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f426effd700 1 -- 172.21.15.31:0/2534980817 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f428003a070 con 0x5629b6361df0 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 10 client.26813 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 10 client.26813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 20 client.26813 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b63c9450 0x5629b63cb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 4 client.26813 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.591+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5629b62abce0 con 0x5629b63c9450 2023-07-29T16:19:36.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.592+0000 7f428effd700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b63c9450 0x5629b63cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.592+0000 7f428effd700 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b63c9450 0x5629b63cb830 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77298923b330d2e2 server_cookie=37b5aff3938f3354 in_seq=0 out_seq=0 2023-07-29T16:19:36.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.592+0000 7f426effd700 10 client.26813 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f426effd700 1 -- 172.21.15.31:0/2534980817 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5629b62abce0 con 0x5629b63c9450 2023-07-29T16:19:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f426effd700 10 client.26813 handle_command_reply: tid=0 2023-07-29T16:19:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 client.26813 shutdown 2023-07-29T16:19:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 msgr2=0x5629b636e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 0x5629b636e760 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4280009ce0 tx=0x7f42800056f0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 shutdown_connections 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5629b63c9450 0x5629b63cb830 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629b6362cf0 0x5629b636f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f426effd700 0 client.26813 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5629b61bd6b0 0x5629b636e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 --2- 172.21.15.31:0/2534980817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629b6361df0 0x5629b636e760 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 >> 172.21.15.31:0/2534980817 conn(0x5629b6356fd0 msgr2=0x5629b6369030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 shutdown_connections 2023-07-29T16:19:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 1 -- 172.21.15.31:0/2534980817 wait complete. 2023-07-29T16:19:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.663+0000 7f42a9a74b80 20 client.26813 trim_cache size 0 max 16384 2023-07-29T16:19:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:19:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:19:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:19:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:19:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42780a0810 msgr2=0x7f42780a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42780a0810 0x7f42780a2cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4288009fd0 tx=0x7f4288004dc0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4290108210 msgr2=0x7f4290199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4290108210 0x7f4290199730 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f428800efc0 tx=0x7f428800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 shutdown_connections 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42780a0810 0x7f42780a2cc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4290108210 0x7f4290199730 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42900737b0 0x7f4290198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 --2- 172.21.15.31:0/913221759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4290074d90 0x7f42901988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 >> 172.21.15.31:0/913221759 conn(0x7f42900fb970 msgr2=0x7f42900fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 shutdown_connections 2023-07-29T16:19:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:36.664+0000 7f42961ff700 1 -- 172.21.15.31:0/913221759 wait complete. 2023-07-29T16:19:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 -- 172.21.15.31:0/1774135060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448101520 msgr2=0x7f9448101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 --2- 172.21.15.31:0/1774135060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448101520 0x7f9448101970 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9438009990 tx=0x7f9438009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 -- 172.21.15.31:0/1774135060 shutdown_connections 2023-07-29T16:19:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 --2- 172.21.15.31:0/1774135060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9448101eb0 0x7f9448108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 --2- 172.21.15.31:0/1774135060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9448100330 0x7f9448100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 --2- 172.21.15.31:0/1774135060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448101520 0x7f9448101970 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 -- 172.21.15.31:0/1774135060 >> 172.21.15.31:0/1774135060 conn(0x7f94480fb970 msgr2=0x7f94480fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 -- 172.21.15.31:0/1774135060 shutdown_connections 2023-07-29T16:19:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.672+0000 7f94507ff700 1 -- 172.21.15.31:0/1774135060 wait complete. 2023-07-29T16:19:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.673+0000 7f94507ff700 1 Processor -- start 2023-07-29T16:19:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.673+0000 7f94507ff700 1 -- start start 2023-07-29T16:19:36.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f94507ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 0x7f9448074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f94507ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9448101520 0x7f9448073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f94507ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9448101eb0 0x7f9448073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f94507ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9448073e20 con 0x7f9448101520 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f94507ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9448073f90 con 0x7f9448100330 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f94507ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94481a0130 con 0x7f9448101eb0 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f944e59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 0x7f9448074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f944e59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 0x7f9448074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49824/0 (socket says 172.21.15.31:49824) 2023-07-29T16:19:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f944ed9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9448101eb0 0x7f9448073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f944e59b700 1 -- 172.21.15.31:0/729215346 learned_addr learned my addr 172.21.15.31:0/729215346 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.674+0000 7f944dd9a700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9448101520 0x7f9448073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f944e59b700 1 -- 172.21.15.31:0/729215346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9448101eb0 msgr2=0x7f9448073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f944e59b700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9448101eb0 0x7f9448073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f944e59b700 1 -- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9448101520 msgr2=0x7f9448073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f944e59b700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9448101520 0x7f9448073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f944e59b700 1 -- 172.21.15.31:0/729215346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380095f0 con 0x7f9448100330 2023-07-29T16:19:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f944e59b700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 0x7f9448074b00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f944400e970 tx=0x7f944400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f943f7fe700 1 -- 172.21.15.31:0/729215346 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444009790 con 0x7f9448100330 2023-07-29T16:19:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f943f7fe700 1 -- 172.21.15.31:0/729215346 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f944400eec0 con 0x7f9448100330 2023-07-29T16:19:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94481a0410 con 0x7f9448100330 2023-07-29T16:19:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f943f7fe700 1 -- 172.21.15.31:0/729215346 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444003cf0 con 0x7f9448100330 2023-07-29T16:19:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.675+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94481a0920 con 0x7f9448100330 2023-07-29T16:19:36.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.678+0000 7f943f7fe700 1 -- 172.21.15.31:0/729215346 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f944400ccd0 con 0x7f9448100330 2023-07-29T16:19:36.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.680+0000 7f9464110b80 1 Processor -- start 2023-07-29T16:19:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.680+0000 7f9464110b80 1 -- start start 2023-07-29T16:19:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.680+0000 7f9464110b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbf3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.680+0000 7f943f7fe700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94340a0790 0x7f94340a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.680+0000 7f9464110b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 0x55eb6bbf4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f9464110b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f9464110b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bb4ba10 con 0x55eb6bbf3be0 2023-07-29T16:19:36.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f9464110b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bb493d0 con 0x55eb6bbf52f0 2023-07-29T16:19:36.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f9464110b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bb44cc0 con 0x55eb6bbf4530 2023-07-29T16:19:36.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f943f7fe700 1 -- 172.21.15.31:0/729215346 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9444014070 con 0x7f9448100330 2023-07-29T16:19:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 0x55eb6bbf4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 0x55eb6bbf4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45040/0 (socket says 172.21.15.31:45040) 2023-07-29T16:19:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 -- 172.21.15.31:0/596984354 learned_addr learned my addr 172.21.15.31:0/596984354 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944ed9c700 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944e59b700 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbf3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944ed9c700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94340a0790 0x7f94340a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 -- 172.21.15.31:0/596984354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 msgr2=0x55eb6bbf5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 -- 172.21.15.31:0/596984354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf3be0 msgr2=0x55eb6bbf3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.681+0000 7f944dd9a700 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbf3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.682+0000 7f944dd9a700 1 -- 172.21.15.31:0/596984354 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb6b972de0 con 0x55eb6bbf4530 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.682+0000 7f944dd9a700 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 0x55eb6bbf4d60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9438015000 tx=0x7f9438005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e717ef7b8c7b2b61 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.682+0000 7f944ed9c700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94340a0790 0x7f94340a2c40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9440009ec0 tx=0x7f944000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.682+0000 7f943cff9700 1 -- 172.21.15.31:0/596984354 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94380056d0 con 0x55eb6bbf4530 2023-07-29T16:19:36.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.682+0000 7f943cff9700 1 -- 172.21.15.31:0/596984354 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f943801e040 con 0x55eb6bbf4530 2023-07-29T16:19:36.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.682+0000 7f943cff9700 1 -- 172.21.15.31:0/596984354 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438004bd0 con 0x55eb6bbf4530 2023-07-29T16:19:36.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 -- 172.21.15.31:0/596984354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 msgr2=0x55eb6bbf4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 0x55eb6bbf4d60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9438015000 tx=0x7f9438005e80 comp rx=0 tx=0).stop 2023-07-29T16:19:36.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 -- 172.21.15.31:0/596984354 shutdown_connections 2023-07-29T16:19:36.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbf3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 --2- 172.21.15.31:0/596984354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf4530 0x55eb6bbf4d60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 -- 172.21.15.31:0/596984354 >> 172.21.15.31:0/596984354 conn(0x55eb6bbe93e0 msgr2=0x55eb6bbee990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 -- 172.21.15.31:0/596984354 shutdown_connections 2023-07-29T16:19:36.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.683+0000 7f9464110b80 1 -- 172.21.15.31:0/596984354 wait complete. 2023-07-29T16:19:36.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.684+0000 7f9464110b80 1 -- start start 2023-07-29T16:19:36.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.684+0000 7f9464110b80 1 Processor -- start 2023-07-29T16:19:36.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:36 smithi031 ceph-mon[95504]: pgmap v494: 97 pgs: 97 active+clean; 51 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 271 op/s 2023-07-29T16:19:36.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:36 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/350224208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:19:36.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbfd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bbfdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc04cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bae1510 con 0x55eb6bbf4530 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bae11b0 con 0x55eb6bbf3be0 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6b9c8920 con 0x55eb6bbf52f0 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f9464110b80 10 client.0 _ll_register_callbacks cb 0x55eb6baea620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f944e59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbfd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f944e59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbfd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49848/0 (socket says 172.21.15.31:49848) 2023-07-29T16:19:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f944ed9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc04cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.685+0000 7f944e59b700 1 -- 172.21.15.31:0/1226404855 learned_addr learned my addr 172.21.15.31:0/1226404855 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bbfdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944e59b700 1 -- 172.21.15.31:0/1226404855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf52f0 msgr2=0x55eb6bc04cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944e59b700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc04cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944e59b700 1 -- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 msgr2=0x55eb6bbfdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944e59b700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bbfdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944e59b700 1 -- 172.21.15.31:0/1226404855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94481090a0 con 0x55eb6bbf3be0 2023-07-29T16:19:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bbfdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f944e59b700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbfd770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f944400c5b0 tx=0x7f9444098f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b3e5ecaca89a547 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f9431ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f9431ffb700 1 -- 172.21.15.31:0/1226404855 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444099970 con 0x55eb6bbf3be0 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f9431ffb700 1 -- 172.21.15.31:0/1226404855 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9444016330 con 0x55eb6bbf3be0 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb6bb6fd10 con 0x55eb6bbf3be0 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.686+0000 7f9431ffb700 1 -- 172.21.15.31:0/1226404855 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94440976f0 con 0x55eb6bbf3be0 2023-07-29T16:19:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 10 client.26825 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55eb6b95bc60 con 0x55eb6bbf3be0 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9431ffb700 1 -- 172.21.15.31:0/1226404855 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9444063020 con 0x55eb6bbf3be0 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9431ffb700 1 -- 172.21.15.31:0/1226404855 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9444098a00 con 0x55eb6bbf3be0 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 10 client.26825 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 10 client.26825 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 20 client.26825 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc09530 0x55eb6bc0b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 4 client.26825 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb6bbc2d70 con 0x55eb6bc09530 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.687+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc09530 0x55eb6bc0b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.688+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc09530 0x55eb6bc0b940 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc09403d968f19fe server_cookie=3f510fa1c86f2f13 in_seq=0 out_seq=0 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.688+0000 7f9431ffb700 10 client.26825 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9431ffb700 1 -- 172.21.15.31:0/1226404855 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55eb6bbc2d70 con 0x55eb6bc09530 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9431ffb700 10 client.26825 handle_command_reply: tid=0 2023-07-29T16:19:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 client.26825 shutdown 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 msgr2=0x55eb6bbfd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbfd770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f944400c5b0 tx=0x7f9444098f40 comp rx=0 tx=0).stop 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 shutdown_connections 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc09530 0x55eb6bc0b940 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc04cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf3be0 0x55eb6bbfd770 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 --2- 172.21.15.31:0/1226404855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bbfdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9431ffb700 0 client.26825 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 >> 172.21.15.31:0/1226404855 conn(0x55eb6bbe93e0 msgr2=0x55eb6bbee060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 shutdown_connections 2023-07-29T16:19:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 1 -- 172.21.15.31:0/1226404855 wait complete. 2023-07-29T16:19:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.689+0000 7f9464110b80 20 client.26825 trim_cache size 0 max 16384 2023-07-29T16:19:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.703+0000 7f9464110b80 1 Processor -- start 2023-07-29T16:19:36.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.703+0000 7f9464110b80 1 -- start start 2023-07-29T16:19:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f9464110b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc0b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f9464110b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc02ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f9464110b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f9464110b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bbf02e0 con 0x55eb6bbf4530 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f9464110b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bbfb440 con 0x55eb6bbf52f0 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f9464110b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bc102e0 con 0x55eb6bbf3be0 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f944e59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc0b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f944e59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc0b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45068/0 (socket says 172.21.15.31:45068) 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f944dd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc02ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f944e59b700 1 -- 172.21.15.31:0/723727848 learned_addr learned my addr 172.21.15.31:0/723727848 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.704+0000 7f944ed9c700 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f944e59b700 1 -- 172.21.15.31:0/723727848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 msgr2=0x55eb6bbf56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f944e59b700 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f944e59b700 1 -- 172.21.15.31:0/723727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 msgr2=0x55eb6bc02ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f944e59b700 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc02ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f944e59b700 1 -- 172.21.15.31:0/723727848 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb6b999250 con 0x55eb6bbf3be0 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f944e59b700 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc0b910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9444005950 tx=0x7f9444095980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e62f742ed22f1f30 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f943d7fa700 1 -- 172.21.15.31:0/723727848 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9444097b70 con 0x55eb6bbf3be0 2023-07-29T16:19:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f943d7fa700 1 -- 172.21.15.31:0/723727848 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9444063070 con 0x55eb6bbf3be0 2023-07-29T16:19:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.705+0000 7f943d7fa700 1 -- 172.21.15.31:0/723727848 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94440995a0 con 0x55eb6bbf3be0 2023-07-29T16:19:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 -- 172.21.15.31:0/723727848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 msgr2=0x55eb6bc0b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc0b910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9444005950 tx=0x7f9444095980 comp rx=0 tx=0).stop 2023-07-29T16:19:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 -- 172.21.15.31:0/723727848 shutdown_connections 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bbf56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc0b910 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 --2- 172.21.15.31:0/723727848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc02ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 -- 172.21.15.31:0/723727848 >> 172.21.15.31:0/723727848 conn(0x55eb6bbe93e0 msgr2=0x55eb6bbee060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 -- 172.21.15.31:0/723727848 shutdown_connections 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 -- 172.21.15.31:0/723727848 wait complete. 2023-07-29T16:19:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 -- start start 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.706+0000 7f9464110b80 1 Processor -- start 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc005b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc00af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc01b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bc02040 con 0x55eb6bbf4530 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bc021b0 con 0x55eb6bbf52f0 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6bc02370 con 0x55eb6bbf3be0 2023-07-29T16:19:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f9464110b80 10 client.0 _ll_register_callbacks cb 0x55eb6baea620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f944ed9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc01b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f944dd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc00af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f944e59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f944dd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc00af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33360/0 (socket says 172.21.15.31:33360) 2023-07-29T16:19:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.707+0000 7f944dd9a700 1 -- 172.21.15.31:0/1079630382 learned_addr learned my addr 172.21.15.31:0/1079630382 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944dd9a700 1 -- 172.21.15.31:0/1079630382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 msgr2=0x55eb6bc005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944dd9a700 1 -- 172.21.15.31:0/1079630382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 msgr2=0x55eb6bc01b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc01b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944dd9a700 1 -- 172.21.15.31:0/1079630382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb6b95bc60 con 0x55eb6bbf4530 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944e59b700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944ed9c700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc01b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f944dd9a700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc00af0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9438005320 tx=0x7f94380094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13996570a94039c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f9431ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f9431ffb700 1 -- 172.21.15.31:0/1079630382 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9438032070 con 0x55eb6bbf4530 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f943801e040 con 0x55eb6bbf4530 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.708+0000 7f9431ffb700 1 -- 172.21.15.31:0/1079630382 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9438004870 con 0x55eb6bbf4530 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9431ffb700 1 -- 172.21.15.31:0/1079630382 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9438004a30 con 0x55eb6bbf4530 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 10 client.16833 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55eb6b9712b0 con 0x55eb6bbf4530 2023-07-29T16:19:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9431ffb700 1 -- 172.21.15.31:0/1079630382 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f943801e040 con 0x55eb6bbf4530 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9431ffb700 1 -- 172.21.15.31:0/1079630382 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9438041460 con 0x55eb6bbf4530 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 10 client.16833 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 10 client.16833 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 20 client.16833 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc5b700 0x55eb6bc5dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 4 client.16833 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:19:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.709+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55eb6bb3dd40 con 0x55eb6bc5b700 2023-07-29T16:19:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f944e59b700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc5b700 0x55eb6bc5dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f944e59b700 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc5b700 0x55eb6bc5dae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54f2e4875928c367 server_cookie=1f946c4914514bcb in_seq=0 out_seq=0 2023-07-29T16:19:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9431ffb700 10 client.16833 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9431ffb700 1 -- 172.21.15.31:0/1079630382 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55eb6bb3dd40 con 0x55eb6bc5b700 2023-07-29T16:19:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9431ffb700 10 client.16833 handle_command_reply: tid=0 2023-07-29T16:19:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9464110b80 1 client.16833 shutdown 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 msgr2=0x55eb6bc00af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9464110b80 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc00af0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f9438005320 tx=0x7f94380094b0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.710+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 shutdown_connections 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55eb6bc5b700 0x55eb6bc5dae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eb6bbf52f0 0x55eb6bc01b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eb6bbf3be0 0x55eb6bc005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 --2- 172.21.15.31:0/1079630382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eb6bbf4530 0x55eb6bc00af0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9431ffb700 0 client.16833 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:19:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 >> 172.21.15.31:0/1079630382 conn(0x55eb6bbe93e0 msgr2=0x55eb6bbfc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 shutdown_connections 2023-07-29T16:19:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 1 -- 172.21.15.31:0/1079630382 wait complete. 2023-07-29T16:19:36.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.711+0000 7f9464110b80 20 client.16833 trim_cache size 0 max 16384 2023-07-29T16:19:36.718 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:19:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94340a0790 msgr2=0x7f94340a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94340a0790 0x7f94340a2c40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9440009ec0 tx=0x7f944000b040 comp rx=0 tx=0).stop 2023-07-29T16:19:36.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 msgr2=0x7f9448074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 0x7f9448074b00 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f944400e970 tx=0x7f944400ec80 comp rx=0 tx=0).stop 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 shutdown_connections 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94340a0790 0x7f94340a2c40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9448101eb0 0x7f9448073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9448100330 0x7f9448074b00 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 --2- 172.21.15.31:0/729215346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9448101520 0x7f9448073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 >> 172.21.15.31:0/729215346 conn(0x7f94480fb970 msgr2=0x7f94480fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 shutdown_connections 2023-07-29T16:19:36.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.712+0000 7f94507ff700 1 -- 172.21.15.31:0/729215346 wait complete. 2023-07-29T16:19:36.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:19:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:36 smithi039 ceph-mon[128858]: pgmap v494: 97 pgs: 97 active+clean; 51 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 271 op/s 2023-07-29T16:19:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:36 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/350224208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:19:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:36 smithi156 ceph-mon[129245]: pgmap v494: 97 pgs: 97 active+clean; 51 MiB data, 666 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 271 op/s 2023-07-29T16:19:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:36 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/350224208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:19:36.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 138450 9.45 MB/sec warmup 81 sec latency 42.737 ms 2023-07-29T16:19:36.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 -- 172.21.15.31:0/954754734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 msgr2=0x7f99ec105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 --2- 172.21.15.31:0/954754734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec105aa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009990 tx=0x7f99e0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 -- 172.21.15.31:0/954754734 shutdown_connections 2023-07-29T16:19:36.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 --2- 172.21.15.31:0/954754734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 --2- 172.21.15.31:0/954754734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 --2- 172.21.15.31:0/954754734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec105aa0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 -- 172.21.15.31:0/954754734 >> 172.21.15.31:0/954754734 conn(0x7f99ec0fa840 msgr2=0x7f99ec0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 -- 172.21.15.31:0/954754734 shutdown_connections 2023-07-29T16:19:36.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.946+0000 7f99f1bff700 1 -- 172.21.15.31:0/954754734 wait complete. 2023-07-29T16:19:36.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.947+0000 7f99f1bff700 1 Processor -- start 2023-07-29T16:19:36.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 -- start start 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec19fde0 con 0x7f99ec1036c0 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec19ff20 con 0x7f99ec105fe0 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99f1bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec1a00b0 con 0x7f99ec100da0 2023-07-29T16:19:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45104/0 (socket says 172.21.15.31:45104) 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec1987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33378/0 (socket says 172.21.15.31:33378) 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99eb7fe700 1 -- 172.21.15.31:0/4058942963 learned_addr learned my addr 172.21.15.31:0/4058942963 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.948+0000 7f99ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec19f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49896/0 (socket says 172.21.15.31:49896) 2023-07-29T16:19:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99eb7fe700 1 -- 172.21.15.31:0/4058942963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 msgr2=0x7f99ec19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99eb7fe700 1 -- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 msgr2=0x7f99ec1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99eb7fe700 1 -- 172.21.15.31:0/4058942963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e4009c40 con 0x7f99ec100da0 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99ebfff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec198260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e9a0 tx=0x7f99dc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99e8ff9700 1 -- 172.21.15.31:0/4058942963 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc009790 con 0x7f99ec100da0 2023-07-29T16:19:36.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e00095f0 con 0x7f99ec100da0 2023-07-29T16:19:36.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99e8ff9700 1 -- 172.21.15.31:0/4058942963 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99dc00ce80 con 0x7f99ec100da0 2023-07-29T16:19:36.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.949+0000 7f99e8ff9700 1 -- 172.21.15.31:0/4058942963 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99dc018680 con 0x7f99ec100da0 2023-07-29T16:19:36.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.950+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec1a0760 con 0x7f99ec100da0 2023-07-29T16:19:36.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.953+0000 7f99e8ff9700 1 -- 172.21.15.31:0/4058942963 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f99dc010430 con 0x7f99ec100da0 2023-07-29T16:19:36.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.954+0000 7f9a054e5b80 1 Processor -- start 2023-07-29T16:19:36.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.954+0000 7f99e8ff9700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99d40a0790 0x7f99d40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.954+0000 7f9a054e5b80 1 -- start start 2023-07-29T16:19:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99e8ff9700 1 -- 172.21.15.31:0/4058942963 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f99dc014070 con 0x7f99ec100da0 2023-07-29T16:19:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99ebfff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99d40a0790 0x7f99d40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf06ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 0x561afcf07d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf082f0 0x561afcf08740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afce5ea10 con 0x561afcf07530 2023-07-29T16:19:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afce5c3d0 con 0x561afcf082f0 2023-07-29T16:19:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afce57cc0 con 0x561afcf06be0 2023-07-29T16:19:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 0x561afcf07d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 0x561afcf07d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33394/0 (socket says 172.21.15.31:33394) 2023-07-29T16:19:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99eaffd700 1 -- 172.21.15.31:0/1079528859 learned_addr learned my addr 172.21.15.31:0/1079528859 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf06ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.955+0000 7f99ebfff700 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf082f0 0x561afcf08740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eaffd700 1 -- 172.21.15.31:0/1079528859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 msgr2=0x561afcf06ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eaffd700 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf06ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eaffd700 1 -- 172.21.15.31:0/1079528859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf082f0 msgr2=0x561afcf08740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eaffd700 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf082f0 0x561afcf08740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eaffd700 1 -- 172.21.15.31:0/1079528859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561afcc85de0 con 0x561afcf07530 2023-07-29T16:19:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99ebfff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99d40a0790 0x7f99d40a2c40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009fd0 tx=0x7f99e400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf06ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99eaffd700 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 0x561afcf07d60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f99e0015000 tx=0x7f99e0005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75a1ac21bc0d1ef server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99d1ffb700 1 -- 172.21.15.31:0/1079528859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e001d070 con 0x561afcf07530 2023-07-29T16:19:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99d1ffb700 1 -- 172.21.15.31:0/1079528859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99e001e040 con 0x561afcf07530 2023-07-29T16:19:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.956+0000 7f99d1ffb700 1 -- 172.21.15.31:0/1079528859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0004950 con 0x561afcf07530 2023-07-29T16:19:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1079528859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 msgr2=0x561afcf07d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 0x561afcf07d60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f99e0015000 tx=0x7f99e0005ea0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1079528859 shutdown_connections 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf082f0 0x561afcf08740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf06ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1079528859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf07530 0x561afcf07d60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1079528859 >> 172.21.15.31:0/1079528859 conn(0x561afcefc3e0 msgr2=0x561afcf01990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1079528859 shutdown_connections 2023-07-29T16:19:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.957+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1079528859 wait complete. 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 -- start start 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 Processor -- start 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf10760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf10ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf17ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcdf4510 con 0x561afcf082f0 2023-07-29T16:19:36.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcdf41b0 con 0x561afcf07530 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afccdb920 con 0x561afcf06be0 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.958+0000 7f9a054e5b80 10 client.0 _ll_register_callbacks cb 0x561afcdfd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf10ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf10760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf10ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49916/0 (socket says 172.21.15.31:49916) 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 -- 172.21.15.31:0/1744445972 learned_addr learned my addr 172.21.15.31:0/1744445972 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf10760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45120/0 (socket says 172.21.15.31:45120) 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99ebfff700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf17ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 -- 172.21.15.31:0/1744445972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 msgr2=0x561afcf10760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf10760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 -- 172.21.15.31:0/1744445972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 msgr2=0x561afcf17ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf17ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 -- 172.21.15.31:0/1744445972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec1a0760 con 0x561afcf07530 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf10760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99eaffd700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf10ca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f99e00098e0 tx=0x7f99e0022b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e208c3b252b06171 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.959+0000 7f99caffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f99caffd700 1 -- 172.21.15.31:0/1744445972 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e001d030 con 0x561afcf07530 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561afce82d10 con 0x561afcf07530 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f99caffd700 1 -- 172.21.15.31:0/1744445972 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f99e000f460 con 0x561afcf07530 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f99caffd700 1 -- 172.21.15.31:0/1744445972 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99e0005bc0 con 0x561afcf07530 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f9a054e5b80 10 client.26843 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f99ec109030 con 0x561afcf07530 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f99caffd700 1 -- 172.21.15.31:0/1744445972 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99e003e020 con 0x561afcf07530 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f99caffd700 1 -- 172.21.15.31:0/1744445972 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f99e002c5c0 con 0x561afcf07530 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f9a054e5b80 10 client.26843 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f9a054e5b80 10 client.26843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.960+0000 7f9a054e5b80 20 client.26843 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.961+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf1c060 0x561afcf1e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.961+0000 7f9a054e5b80 4 client.26843 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.961+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561afced5d70 con 0x561afcf1c060 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.961+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf1c060 0x561afcf1e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.961+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf1c060 0x561afcf1e470 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4824b376a272d45a server_cookie=2d1a2132e88b634e in_seq=0 out_seq=0 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.961+0000 7f99caffd700 10 client.26843 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f99caffd700 1 -- 172.21.15.31:0/1744445972 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561afced5d70 con 0x561afcf1c060 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f99caffd700 10 client.26843 handle_command_reply: tid=0 2023-07-29T16:19:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 client.26843 shutdown 2023-07-29T16:19:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 msgr2=0x561afcf10ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf10ca0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f99e00098e0 tx=0x7f99e0022b40 comp rx=0 tx=0).stop 2023-07-29T16:19:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 shutdown_connections 2023-07-29T16:19:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf1c060 0x561afcf1e470 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf17ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf06be0 0x561afcf10760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f99caffd700 0 client.26843 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/1744445972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf10ca0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 >> 172.21.15.31:0/1744445972 conn(0x561afcefc3e0 msgr2=0x561afcf01060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 shutdown_connections 2023-07-29T16:19:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.962+0000 7f9a054e5b80 1 -- 172.21.15.31:0/1744445972 wait complete. 2023-07-29T16:19:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.963+0000 7f9a054e5b80 20 client.26843 trim_cache size 0 max 16384 2023-07-29T16:19:36.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.976+0000 7f9a054e5b80 1 Processor -- start 2023-07-29T16:19:36.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 -- start start 2023-07-29T16:19:36.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 0x561afcf15c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 0x561afcf0e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf0ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcefe460 con 0x561afcf082f0 2023-07-29T16:19:36.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcf118e0 con 0x561afcf06be0 2023-07-29T16:19:36.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcf03080 con 0x561afcf07530 2023-07-29T16:19:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 0x561afcf15c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 0x561afcf0e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 0x561afcf0e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45130/0 (socket says 172.21.15.31:45130) 2023-07-29T16:19:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 0x561afcf15c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49924/0 (socket says 172.21.15.31:49924) 2023-07-29T16:19:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 -- 172.21.15.31:0/2461904042 learned_addr learned my addr 172.21.15.31:0/2461904042 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99ebfff700 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf0ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 -- 172.21.15.31:0/2461904042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 msgr2=0x561afcf15c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 0x561afcf15c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 -- 172.21.15.31:0/2461904042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 msgr2=0x561afcf0ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf0ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eaffd700 1 -- 172.21.15.31:0/2461904042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561afcc6ec60 con 0x561afcf07530 2023-07-29T16:19:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.977+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 0x561afcf15c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.978+0000 7f99eaffd700 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 0x561afcf0e690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005ac0 tx=0x7f99e0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=210584f9dea4c1ca server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.978+0000 7f99d27fc700 1 -- 172.21.15.31:0/2461904042 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e001d040 con 0x561afcf07530 2023-07-29T16:19:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.978+0000 7f99d27fc700 1 -- 172.21.15.31:0/2461904042 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99e003e030 con 0x561afcf07530 2023-07-29T16:19:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.978+0000 7f99d27fc700 1 -- 172.21.15.31:0/2461904042 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e002ada0 con 0x561afcf07530 2023-07-29T16:19:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 -- 172.21.15.31:0/2461904042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 msgr2=0x561afcf0e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 0x561afcf0e690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005ac0 tx=0x7f99e0009310 comp rx=0 tx=0).stop 2023-07-29T16:19:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 -- 172.21.15.31:0/2461904042 shutdown_connections 2023-07-29T16:19:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf082f0 0x561afcf0ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf06be0 0x561afcf15c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/2461904042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf07530 0x561afcf0e690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 -- 172.21.15.31:0/2461904042 >> 172.21.15.31:0/2461904042 conn(0x561afcefc3e0 msgr2=0x561afcf02690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 -- 172.21.15.31:0/2461904042 shutdown_connections 2023-07-29T16:19:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 -- 172.21.15.31:0/2461904042 wait complete. 2023-07-29T16:19:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 -- start start 2023-07-29T16:19:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.979+0000 7f9a054e5b80 1 Processor -- start 2023-07-29T16:19:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf06be0 0x561afcf136c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf13c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf082f0 0x561afcf14c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcf15150 con 0x561afcf06be0 2023-07-29T16:19:36.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcf152c0 con 0x561afcf07530 2023-07-29T16:19:36.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561afcf15480 con 0x561afcf082f0 2023-07-29T16:19:36.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f9a054e5b80 10 client.0 _ll_register_callbacks cb 0x561afcdfd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:36.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf13c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf06be0 0x561afcf136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf13c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49934/0 (socket says 172.21.15.31:49934) 2023-07-29T16:19:36.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 -- 172.21.15.31:0/3343645089 learned_addr learned my addr 172.21.15.31:0/3343645089 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:36.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99ebfff700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf082f0 0x561afcf14c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 -- 172.21.15.31:0/3343645089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf082f0 msgr2=0x561afcf14c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf082f0 0x561afcf14c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 -- 172.21.15.31:0/3343645089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf06be0 msgr2=0x561afcf136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:36.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf06be0 0x561afcf136c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99eaffd700 1 -- 172.21.15.31:0/3343645089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e00095f0 con 0x561afcf07530 2023-07-29T16:19:36.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.980+0000 7f99ebfff700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf082f0 0x561afcf14c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f99eaffd700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf13c00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005f50 tx=0x7f99e002cac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa72e7da11df55b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f99caffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f99caffd700 1 -- 172.21.15.31:0/3343645089 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e001d040 con 0x561afcf07530 2023-07-29T16:19:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99e001e040 con 0x561afcf07530 2023-07-29T16:19:36.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f99caffd700 1 -- 172.21.15.31:0/3343645089 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f99e0000e40 con 0x561afcf07530 2023-07-29T16:19:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f99caffd700 1 -- 172.21.15.31:0/3343645089 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99e002c610 con 0x561afcf07530 2023-07-29T16:19:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f9a054e5b80 10 client.26849 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f99ec109030 con 0x561afcf07530 2023-07-29T16:19:36.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.981+0000 7f99caffd700 1 -- 172.21.15.31:0/3343645089 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99e001e040 con 0x561afcf07530 2023-07-29T16:19:36.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f99caffd700 1 -- 172.21.15.31:0/3343645089 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f99e000faa0 con 0x561afcf07530 2023-07-29T16:19:36.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 10 client.26849 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:36.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 10 client.26849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:36.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 20 client.26849 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:36.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf6e460 0x561afcf70870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 4 client.26849 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561afce50d40 con 0x561afcf6e460 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf6e460 0x561afcf70870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f99eb7fe700 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf6e460 0x561afcf70870 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65f150eb742b9763 server_cookie=e7b566374dc170b5 in_seq=0 out_seq=0 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f99caffd700 10 client.26849 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f99caffd700 1 -- 172.21.15.31:0/3343645089 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561afce50d40 con 0x561afcf6e460 2023-07-29T16:19:36.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f99caffd700 10 client.26849 handle_command_reply: tid=0 2023-07-29T16:19:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.982+0000 7f9a054e5b80 1 client.26849 shutdown 2023-07-29T16:19:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 msgr2=0x561afcf13c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf13c00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f99e0005f50 tx=0x7f99e002cac0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 shutdown_connections 2023-07-29T16:19:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561afcf6e460 0x561afcf70870 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561afcf082f0 0x561afcf14c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561afcf06be0 0x561afcf136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 --2- 172.21.15.31:0/3343645089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561afcf07530 0x561afcf13c00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f99caffd700 0 client.26849 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 >> 172.21.15.31:0/3343645089 conn(0x561afcefc3e0 msgr2=0x561afcf0e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 shutdown_connections 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 1 -- 172.21.15.31:0/3343645089 wait complete. 2023-07-29T16:19:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.983+0000 7f9a054e5b80 20 client.26849 trim_cache size 0 max 16384 2023-07-29T16:19:36.998 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:19:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99d40a0790 msgr2=0x7f99d40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99d40a0790 0x7f99d40a2c40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009fd0 tx=0x7f99e400b040 comp rx=0 tx=0).stop 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 msgr2=0x7f99ec198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec198260 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f99dc00e9a0 tx=0x7f99dc00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 shutdown_connections 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99d40a0790 0x7f99d40a2c40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99ec105fe0 0x7f99ec19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99ec100da0 0x7f99ec198260 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 --2- 172.21.15.31:0/4058942963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f99ec1036c0 0x7f99ec1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 >> 172.21.15.31:0/4058942963 conn(0x7f99ec0fa840 msgr2=0x7f99ec0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 shutdown_connections 2023-07-29T16:19:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:36.984+0000 7f99f1bff700 1 -- 172.21.15.31:0/4058942963 wait complete. 2023-07-29T16:19:37.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 139551 9.38 MB/sec warmup 82 sec latency 18.584 ms 2023-07-29T16:19:38.003 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:19:38.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:19:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:37 smithi031 ceph-mon[95504]: pgmap v495: 97 pgs: 97 active+clean; 51 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 248 op/s 2023-07-29T16:19:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 -- 172.21.15.31:0/388983807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 msgr2=0x7f53e0101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 --2- 172.21.15.31:0/388983807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0101970 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009990 tx=0x7f53d4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 -- 172.21.15.31:0/388983807 shutdown_connections 2023-07-29T16:19:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 --2- 172.21.15.31:0/388983807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0101eb0 0x7f53e0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 --2- 172.21.15.31:0/388983807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53e0100330 0x7f53e0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 --2- 172.21.15.31:0/388983807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0101970 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 -- 172.21.15.31:0/388983807 >> 172.21.15.31:0/388983807 conn(0x7f53e00fb970 msgr2=0x7f53e00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 -- 172.21.15.31:0/388983807 shutdown_connections 2023-07-29T16:19:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.256+0000 7f53e59ff700 1 -- 172.21.15.31:0/388983807 wait complete. 2023-07-29T16:19:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.257+0000 7f53e59ff700 1 Processor -- start 2023-07-29T16:19:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 -- start start 2023-07-29T16:19:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0100330 0x7f53e0074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53e0101eb0 0x7f53e0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0073e20 con 0x7f53e0100330 2023-07-29T16:19:38.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0073f90 con 0x7f53e0101eb0 2023-07-29T16:19:38.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53e59ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e01a0080 con 0x7f53e0101520 2023-07-29T16:19:38.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53deffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0100330 0x7f53e0074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53deffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0100330 0x7f53e0074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33432/0 (socket says 172.21.15.31:33432) 2023-07-29T16:19:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53deffd700 1 -- 172.21.15.31:0/2803915672 learned_addr learned my addr 172.21.15.31:0/2803915672 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53de7fc700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.258+0000 7f53df7fe700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53e0101eb0 0x7f53e0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53de7fc700 1 -- 172.21.15.31:0/2803915672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53e0101eb0 msgr2=0x7f53e0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53de7fc700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53e0101eb0 0x7f53e0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53de7fc700 1 -- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0100330 msgr2=0x7f53e0074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53de7fc700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0100330 0x7f53e0074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53de7fc700 1 -- 172.21.15.31:0/2803915672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40095f0 con 0x7f53e0101520 2023-07-29T16:19:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53de7fc700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0073140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006010 tx=0x7f53d40057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53e49fd700 1 -- 172.21.15.31:0/2803915672 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d401d070 con 0x7f53e0101520 2023-07-29T16:19:38.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53e49fd700 1 -- 172.21.15.31:0/2803915672 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53d400fb90 con 0x7f53e0101520 2023-07-29T16:19:38.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53e49fd700 1 -- 172.21.15.31:0/2803915672 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4017650 con 0x7f53e0101520 2023-07-29T16:19:38.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.259+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e01a0300 con 0x7f53e0101520 2023-07-29T16:19:38.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.260+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e01a0730 con 0x7f53e0101520 2023-07-29T16:19:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:37 smithi039 ceph-mon[128858]: pgmap v495: 97 pgs: 97 active+clean; 51 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 248 op/s 2023-07-29T16:19:38.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.263+0000 7f53e49fd700 1 -- 172.21.15.31:0/2803915672 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f53d4022470 con 0x7f53e0101520 2023-07-29T16:19:38.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.265+0000 7f53f92d7b80 1 Processor -- start 2023-07-29T16:19:38.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.265+0000 7f53e49fd700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a0810 0x7f53c80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.265+0000 7f53f92d7b80 1 -- start start 2023-07-29T16:19:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.265+0000 7f53e49fd700 1 -- 172.21.15.31:0/2803915672 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f53d40ccff0 con 0x7f53e0101520 2023-07-29T16:19:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.265+0000 7f53df7fe700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a0810 0x7f53c80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca43c60 0x55b58ca44070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca44de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca457c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58c99ba90 con 0x55b58ca45370 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58c999450 con 0x55b58ca445b0 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58c994d40 con 0x55b58ca43c60 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca44de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53deffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca43c60 0x55b58ca44070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca44de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49964/0 (socket says 172.21.15.31:49964) 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 -- 172.21.15.31:0/3604536041 learned_addr learned my addr 172.21.15.31:0/3604536041 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 -- 172.21.15.31:0/3604536041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca43c60 msgr2=0x55b58ca44070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca43c60 0x55b58ca44070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 -- 172.21.15.31:0/3604536041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 msgr2=0x55b58ca457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.266+0000 7f53de7fc700 1 -- 172.21.15.31:0/3604536041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b58c7c2e60 con 0x55b58ca445b0 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca44de0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f53d4096c60 tx=0x7f53d4009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baf54fca12c03b5a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53df7fe700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a0810 0x7f53c80a2cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f53e0075160 tx=0x7f53d8003f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53c5ffb700 1 -- 172.21.15.31:0/3604536041 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d409e790 con 0x55b58ca445b0 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53c5ffb700 1 -- 172.21.15.31:0/3604536041 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53d40a2b20 con 0x55b58ca445b0 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53c5ffb700 1 -- 172.21.15.31:0/3604536041 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d4099c50 con 0x55b58ca445b0 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3604536041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 msgr2=0x55b58ca44de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.267+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca44de0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f53d4096c60 tx=0x7f53d4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3604536041 shutdown_connections 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca43c60 0x55b58ca44070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3604536041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca44de0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3604536041 >> 172.21.15.31:0/3604536041 conn(0x55b58ca39460 msgr2=0x55b58ca3ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3604536041 shutdown_connections 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3604536041 wait complete. 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.268+0000 7f53f92d7b80 1 -- start start 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 Processor -- start 2023-07-29T16:19:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca4d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca4d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca54a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58c931590 con 0x55b58ca43c60 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58c931230 con 0x55b58ca445b0 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58c8189a0 con 0x55b58ca45370 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53f92d7b80 10 client.0 _ll_register_callbacks cb 0x55b58c93a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53de7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca4d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53de7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca4d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49974/0 (socket says 172.21.15.31:49974) 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53de7fc700 1 -- 172.21.15.31:0/3263239077 learned_addr learned my addr 172.21.15.31:0/3263239077 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.269+0000 7f53deffd700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca4d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53df7fe700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca54a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53de7fc700 1 -- 172.21.15.31:0/3263239077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 msgr2=0x55b58ca54a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca54a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53de7fc700 1 -- 172.21.15.31:0/3263239077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 msgr2=0x55b58ca4d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca4d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53de7fc700 1 -- 172.21.15.31:0/3263239077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e01a0730 con 0x55b58ca445b0 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53df7fe700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca54a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca4d9f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f53d40059d0 tx=0x7f53d4099e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c55f716ce4a4a9ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53beffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53beffd700 1 -- 172.21.15.31:0/3263239077 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d40988a0 con 0x55b58ca445b0 2023-07-29T16:19:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b58c9bfd90 con 0x55b58ca445b0 2023-07-29T16:19:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53beffd700 1 -- 172.21.15.31:0/3263239077 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f53d409e790 con 0x55b58ca445b0 2023-07-29T16:19:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53beffd700 1 -- 172.21.15.31:0/3263239077 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53d40a28a0 con 0x55b58ca445b0 2023-07-29T16:19:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53f92d7b80 10 client.26861 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.270+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b58c7abce0 con 0x55b58ca445b0 2023-07-29T16:19:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53beffd700 1 -- 172.21.15.31:0/3263239077 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b58c9bfd90 con 0x55b58ca445b0 2023-07-29T16:19:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53beffd700 1 -- 172.21.15.31:0/3263239077 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f53d40996c0 con 0x55b58ca445b0 2023-07-29T16:19:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53f92d7b80 10 client.26861 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53f92d7b80 10 client.26861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53f92d7b80 20 client.26861 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58ca58d90 0x55b58ca5b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53f92d7b80 4 client.26861 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b58ca12df0 con 0x55b58ca58d90 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.271+0000 7f53deffd700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58ca58d90 0x55b58ca5b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.272+0000 7f53beffd700 10 client.26861 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.272+0000 7f53deffd700 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58ca58d90 0x55b58ca5b1a0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1f20689e95197ac server_cookie=7dc85a306212acc9 in_seq=0 out_seq=0 2023-07-29T16:19:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.272+0000 7f53beffd700 1 -- 172.21.15.31:0/3263239077 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b58ca12df0 con 0x55b58ca58d90 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.272+0000 7f53beffd700 10 client.26861 handle_command_reply: tid=0 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 client.26861 shutdown 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 msgr2=0x55b58ca4d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca4d9f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f53d40059d0 tx=0x7f53d4099e60 comp rx=0 tx=0).stop 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 shutdown_connections 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58ca58d90 0x55b58ca5b1a0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca54a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca4d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53beffd700 0 client.26861 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3263239077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca4d9f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 >> 172.21.15.31:0/3263239077 conn(0x55b58ca39460 msgr2=0x55b58ca3e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 shutdown_connections 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3263239077 wait complete. 2023-07-29T16:19:38.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.273+0000 7f53f92d7b80 20 client.26861 trim_cache size 0 max 16384 2023-07-29T16:19:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 Processor -- start 2023-07-29T16:19:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 -- start start 2023-07-29T16:19:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca3e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca4b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58ca4c960 con 0x55b58ca43c60 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58ca3b4e0 con 0x55b58ca445b0 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.291+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58ca44c40 con 0x55b58ca45370 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca3e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca3e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33456/0 (socket says 172.21.15.31:33456) 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 -- 172.21.15.31:0/3955510290 learned_addr learned my addr 172.21.15.31:0/3955510290 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53df7fe700 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca4b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 -- 172.21.15.31:0/3955510290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 msgr2=0x55b58ca4b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca4b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 -- 172.21.15.31:0/3955510290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 msgr2=0x55b58ca549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.292+0000 7f53deffd700 1 -- 172.21.15.31:0/3955510290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b58ca4c500 con 0x55b58ca43c60 2023-07-29T16:19:38.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.293+0000 7f53deffd700 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca3e0e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f53d0005d90 tx=0x7f53d0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678cc9aaf9e7c73 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.294+0000 7f53c67fc700 1 -- 172.21.15.31:0/3955510290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0017040 con 0x55b58ca43c60 2023-07-29T16:19:38.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.294+0000 7f53c67fc700 1 -- 172.21.15.31:0/3955510290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53d0018040 con 0x55b58ca43c60 2023-07-29T16:19:38.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.294+0000 7f53c67fc700 1 -- 172.21.15.31:0/3955510290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0005520 con 0x55b58ca43c60 2023-07-29T16:19:38.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3955510290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 msgr2=0x55b58ca3e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca3e0e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f53d0005d90 tx=0x7f53d0009260 comp rx=0 tx=0).stop 2023-07-29T16:19:38.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3955510290 shutdown_connections 2023-07-29T16:19:38.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca45370 0x55b58ca4b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca43c60 0x55b58ca3e0e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3955510290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca445b0 0x55b58ca549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3955510290 >> 172.21.15.31:0/3955510290 conn(0x55b58ca39460 msgr2=0x55b58ca3f710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.295+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3955510290 shutdown_connections 2023-07-29T16:19:38.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.296+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3955510290 wait complete. 2023-07-29T16:19:38.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.296+0000 7f53f92d7b80 1 -- start start 2023-07-29T16:19:38.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.296+0000 7f53f92d7b80 1 Processor -- start 2023-07-29T16:19:38.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca43c60 0x55b58ca51c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca445b0 0x55b58ca521b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58ca53700 con 0x55b58ca45370 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58ca53870 con 0x55b58ca43c60 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b58ca53a30 con 0x55b58ca445b0 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.297+0000 7f53f92d7b80 10 client.0 _ll_register_callbacks cb 0x55b58c93a6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.298+0000 7f53df7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.298+0000 7f53df7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33468/0 (socket says 172.21.15.31:33468) 2023-07-29T16:19:38.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53df7fe700 1 -- 172.21.15.31:0/3886002183 learned_addr learned my addr 172.21.15.31:0/3886002183 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53deffd700 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca43c60 0x55b58ca51c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53deffd700 1 -- 172.21.15.31:0/3886002183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca445b0 msgr2=0x55b58ca521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53deffd700 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca445b0 0x55b58ca521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53deffd700 1 -- 172.21.15.31:0/3886002183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 msgr2=0x55b58ca531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53deffd700 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.299+0000 7f53deffd700 1 -- 172.21.15.31:0/3886002183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d0009c20 con 0x55b58ca43c60 2023-07-29T16:19:38.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.300+0000 7f53deffd700 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca43c60 0x55b58ca51c70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f53d000a010 tx=0x7f53d001f430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481832df39082202 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.300+0000 7f53beffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:38.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53beffd700 1 -- 172.21.15.31:0/3886002183 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0004750 con 0x55b58ca43c60 2023-07-29T16:19:38.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53beffd700 1 -- 172.21.15.31:0/3886002183 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f53d00048b0 con 0x55b58ca43c60 2023-07-29T16:19:38.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53d40a2b20 con 0x55b58ca43c60 2023-07-29T16:19:38.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53beffd700 1 -- 172.21.15.31:0/3886002183 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53d0004a70 con 0x55b58ca43c60 2023-07-29T16:19:38.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53f92d7b80 10 client.26867 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:38.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55b58ca45a10 con 0x55b58ca43c60 2023-07-29T16:19:38.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.301+0000 7f53beffd700 1 -- 172.21.15.31:0/3886002183 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53d40a2b20 con 0x55b58ca43c60 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53beffd700 1 -- 172.21.15.31:0/3886002183 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f53d0037070 con 0x55b58ca43c60 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53f92d7b80 10 client.26867 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53f92d7b80 10 client.26867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53f92d7b80 20 client.26867 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58cab54d0 0x55b58cab78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53f92d7b80 4 client.26867 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:19:38.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.302+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b58c98ddc0 con 0x55b58cab54d0 2023-07-29T16:19:38.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.303+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58cab54d0 0x55b58cab78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.306+0000 7f53de7fc700 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58cab54d0 0x55b58cab78e0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9134587ac26ae50b server_cookie=6afe8dd5302ab490 in_seq=0 out_seq=0 2023-07-29T16:19:38.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.306+0000 7f53beffd700 10 client.26867 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.312+0000 7f53beffd700 1 -- 172.21.15.31:0/3886002183 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b58c98ddc0 con 0x55b58cab54d0 2023-07-29T16:19:38.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.312+0000 7f53beffd700 10 client.26867 handle_command_reply: tid=0 2023-07-29T16:19:38.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.312+0000 7f53f92d7b80 1 client.26867 shutdown 2023-07-29T16:19:38.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca43c60 msgr2=0x55b58ca51c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca43c60 0x55b58ca51c70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f53d000a010 tx=0x7f53d001f430 comp rx=0 tx=0).stop 2023-07-29T16:19:38.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 shutdown_connections 2023-07-29T16:19:38.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b58cab54d0 0x55b58cab78e0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b58ca45370 0x55b58ca531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b58ca43c60 0x55b58ca51c70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 --2- 172.21.15.31:0/3886002183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b58ca445b0 0x55b58ca521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 >> 172.21.15.31:0/3886002183 conn(0x55b58ca39460 msgr2=0x55b58ca3e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.316+0000 7f53beffd700 0 client.26867 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.320+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 shutdown_connections 2023-07-29T16:19:38.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.322+0000 7f53f92d7b80 1 -- 172.21.15.31:0/3886002183 wait complete. 2023-07-29T16:19:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:37 smithi156 ceph-mon[129245]: pgmap v495: 97 pgs: 97 active+clean; 51 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 248 op/s 2023-07-29T16:19:38.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.325+0000 7f53f92d7b80 20 client.26867 trim_cache size 0 max 16384 2023-07-29T16:19:38.332 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:19:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:19:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "1b29ffc1-cf48-4e62-a23f-6bb0da9865f1", 2023-07-29T16:19:38.332 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:19:38.333 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:19:38.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.337+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a0810 msgr2=0x7f53c80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.337+0000 7f53e59ff700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a0810 0x7f53c80a2cc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f53e0075160 tx=0x7f53d8003f90 comp rx=0 tx=0).stop 2023-07-29T16:19:38.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.337+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 msgr2=0x7f53e0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.337+0000 7f53e59ff700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0073140 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006010 tx=0x7f53d40057b0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.338+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 shutdown_connections 2023-07-29T16:19:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.338+0000 7f53e59ff700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53c80a0810 0x7f53c80a2cc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.338+0000 7f53e59ff700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53e0101eb0 0x7f53e0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.338+0000 7f53e59ff700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53e0100330 0x7f53e0074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.338+0000 7f53e59ff700 1 --2- 172.21.15.31:0/2803915672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53e0101520 0x7f53e0073140 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f53d4006010 tx=0x7f53d40057b0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.338+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 >> 172.21.15.31:0/2803915672 conn(0x7f53e00fb970 msgr2=0x7f53e00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.341+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 shutdown_connections 2023-07-29T16:19:38.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.341+0000 7f53e59ff700 1 -- 172.21.15.31:0/2803915672 wait complete. 2023-07-29T16:19:38.363 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:19:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 -- 172.21.15.31:0/3249412571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101510 msgr2=0x7f194c101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 --2- 172.21.15.31:0/3249412571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101510 0x7f194c101960 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1940009990 tx=0x7f1940009ca0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 -- 172.21.15.31:0/3249412571 shutdown_connections 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 --2- 172.21.15.31:0/3249412571 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101ea0 0x7f194c108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 --2- 172.21.15.31:0/3249412571 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 0x7f194c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 --2- 172.21.15.31:0/3249412571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101510 0x7f194c101960 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 -- 172.21.15.31:0/3249412571 >> 172.21.15.31:0/3249412571 conn(0x7f194c0fb920 msgr2=0x7f194c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 -- 172.21.15.31:0/3249412571 shutdown_connections 2023-07-29T16:19:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.582+0000 7f19511ff700 1 -- 172.21.15.31:0/3249412571 wait complete. 2023-07-29T16:19:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.583+0000 7f19511ff700 1 Processor -- start 2023-07-29T16:19:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 -- start start 2023-07-29T16:19:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 0x7f194c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 0x7f194c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101ea0 0x7f194c19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c19dbc0 con 0x7f194c100320 2023-07-29T16:19:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c19dd00 con 0x7f194c101510 2023-07-29T16:19:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f19511ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c19de90 con 0x7f194c101ea0 2023-07-29T16:19:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f194a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 0x7f194c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f194ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 0x7f194c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f194a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 0x7f194c196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50000/0 (socket says 172.21.15.31:50000) 2023-07-29T16:19:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f194a59c700 1 -- 172.21.15.31:0/4154679745 learned_addr learned my addr 172.21.15.31:0/4154679745 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.584+0000 7f194b59e700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101ea0 0x7f194c19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194a59c700 1 -- 172.21.15.31:0/4154679745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101ea0 msgr2=0x7f194c19d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194a59c700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101ea0 0x7f194c19d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194a59c700 1 -- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 msgr2=0x7f194c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194a59c700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 0x7f194c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194a59c700 1 -- 172.21.15.31:0/4154679745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400095f0 con 0x7f194c101510 2023-07-29T16:19:38.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194ad9d700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 0x7f194c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:38.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f194a59c700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 0x7f194c196580 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1940004740 tx=0x7f1940009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f1933fff700 1 -- 172.21.15.31:0/4154679745 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194001d070 con 0x7f194c101510 2023-07-29T16:19:38.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c19e110 con 0x7f194c101510 2023-07-29T16:19:38.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f1933fff700 1 -- 172.21.15.31:0/4154679745 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1940004830 con 0x7f194c101510 2023-07-29T16:19:38.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f1933fff700 1 -- 172.21.15.31:0/4154679745 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400053f0 con 0x7f194c101510 2023-07-29T16:19:38.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.585+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c19e590 con 0x7f194c101510 2023-07-29T16:19:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.589+0000 7f1933fff700 1 -- 172.21.15.31:0/4154679745 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f19400055d0 con 0x7f194c101510 2023-07-29T16:19:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 Processor -- start 2023-07-29T16:19:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 -- start start 2023-07-29T16:19:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df08be0 0x555d7df08ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1933fff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19340a0790 0x7f19340a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df09d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7de60a10 con 0x555d7df0a2f0 2023-07-29T16:19:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7de5e3d0 con 0x555d7df09530 2023-07-29T16:19:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.590+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7de59cc0 con 0x555d7df08be0 2023-07-29T16:19:38.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f1933fff700 1 -- 172.21.15.31:0/4154679745 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f19400cccb0 con 0x7f194c101510 2023-07-29T16:19:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df09d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194b59e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df08be0 0x555d7df08ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df09d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50016/0 (socket says 172.21.15.31:50016) 2023-07-29T16:19:38.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 -- 172.21.15.31:0/534851182 learned_addr learned my addr 172.21.15.31:0/534851182 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194b59e700 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194ad9d700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19340a0790 0x7f19340a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 -- 172.21.15.31:0/534851182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df08be0 msgr2=0x555d7df08ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df08be0 0x555d7df08ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 -- 172.21.15.31:0/534851182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 msgr2=0x555d7df0a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 -- 172.21.15.31:0/534851182 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d7dc87de0 con 0x555d7df09530 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194b59e700 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.591+0000 7f194a59c700 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df09d60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f19400968e0 tx=0x7f194009a2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d53ae20cc4da576 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f19317fa700 1 -- 172.21.15.31:0/534851182 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400a3660 con 0x555d7df09530 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f19317fa700 1 -- 172.21.15.31:0/534851182 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f194009a3e0 con 0x555d7df09530 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f19317fa700 1 -- 172.21.15.31:0/534851182 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400a00b0 con 0x555d7df09530 2023-07-29T16:19:38.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 -- 172.21.15.31:0/534851182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 msgr2=0x555d7df09d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f194ad9d700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19340a0790 0x7f19340a2c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f194c101370 tx=0x7f193c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df09d60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f19400968e0 tx=0x7f194009a2e0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 -- 172.21.15.31:0/534851182 shutdown_connections 2023-07-29T16:19:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df08be0 0x555d7df08ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/534851182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df09d60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 -- 172.21.15.31:0/534851182 >> 172.21.15.31:0/534851182 conn(0x555d7defe3e0 msgr2=0x555d7df03990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.592+0000 7f1964a8ab80 1 -- 172.21.15.31:0/534851182 shutdown_connections 2023-07-29T16:19:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.593+0000 7f1964a8ab80 1 -- 172.21.15.31:0/534851182 wait complete. 2023-07-29T16:19:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.593+0000 7f1964a8ab80 1 -- start start 2023-07-29T16:19:38.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.593+0000 7f1964a8ab80 1 Processor -- start 2023-07-29T16:19:38.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df0a2f0 0x555d7df19a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7ddf6510 con 0x555d7df08be0 2023-07-29T16:19:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7ddf61b0 con 0x555d7df09530 2023-07-29T16:19:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7dcdd920 con 0x555d7df0a2f0 2023-07-29T16:19:38.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f1964a8ab80 10 client.0 _ll_register_callbacks cb 0x555d7ddff620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50028/0 (socket says 172.21.15.31:50028) 2023-07-29T16:19:38.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 -- 172.21.15.31:0/360503288 learned_addr learned my addr 172.21.15.31:0/360503288 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194b59e700 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df0a2f0 0x555d7df19a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 -- 172.21.15.31:0/360503288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df0a2f0 msgr2=0x555d7df19a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df0a2f0 0x555d7df19a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 -- 172.21.15.31:0/360503288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 msgr2=0x555d7df124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.594+0000 7f194a59c700 1 -- 172.21.15.31:0/360503288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c19e590 con 0x555d7df09530 2023-07-29T16:19:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f194a59c700 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df129f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f19400a2270 tx=0x7f194009aa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60f8dc6de0bd6bdd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f192eff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f192eff5700 1 -- 172.21.15.31:0/360503288 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400a02c0 con 0x555d7df09530 2023-07-29T16:19:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f192eff5700 1 -- 172.21.15.31:0/360503288 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f194009e410 con 0x555d7df09530 2023-07-29T16:19:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d7de84d10 con 0x555d7df09530 2023-07-29T16:19:38.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f192eff5700 1 -- 172.21.15.31:0/360503288 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19400998d0 con 0x555d7df09530 2023-07-29T16:19:38.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f1964a8ab80 10 client.26879 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:38.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x555d7dc70c60 con 0x555d7df09530 2023-07-29T16:19:38.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f192eff5700 1 -- 172.21.15.31:0/360503288 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555d7de84d10 con 0x555d7df09530 2023-07-29T16:19:38.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.595+0000 7f192eff5700 1 -- 172.21.15.31:0/360503288 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f19400abd90 con 0x555d7df09530 2023-07-29T16:19:38.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f1964a8ab80 10 client.26879 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:38.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f1964a8ab80 10 client.26879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:38.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f1964a8ab80 20 client.26879 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:38.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df1ddd0 0x555d7df201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f1964a8ab80 4 client.26879 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:38.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d7ded7d70 con 0x555d7df1ddd0 2023-07-29T16:19:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f194ad9d700 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df1ddd0 0x555d7df201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f194ad9d700 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df1ddd0 0x555d7df201e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c034274c40ac94 server_cookie=29e95e1f88a001e5 in_seq=0 out_seq=0 2023-07-29T16:19:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.596+0000 7f192eff5700 10 client.26879 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f192eff5700 1 -- 172.21.15.31:0/360503288 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555d7ded7d70 con 0x555d7df1ddd0 2023-07-29T16:19:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f192eff5700 10 client.26879 handle_command_reply: tid=0 2023-07-29T16:19:38.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 client.26879 shutdown 2023-07-29T16:19:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 msgr2=0x555d7df129f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df129f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f19400a2270 tx=0x7f194009aa00 comp rx=0 tx=0).stop 2023-07-29T16:19:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 shutdown_connections 2023-07-29T16:19:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df1ddd0 0x555d7df201e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df0a2f0 0x555d7df19a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/360503288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 >> 172.21.15.31:0/360503288 conn(0x555d7defe3e0 msgr2=0x555d7df03060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f192eff5700 0 client.26879 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 shutdown_connections 2023-07-29T16:19:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 1 -- 172.21.15.31:0/360503288 wait complete. 2023-07-29T16:19:38.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.598+0000 7f1964a8ab80 20 client.26879 trim_cache size 0 max 16384 2023-07-29T16:19:38.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.615+0000 7f1964a8ab80 1 Processor -- start 2023-07-29T16:19:38.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 -- start start 2023-07-29T16:19:38.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7df03770 con 0x555d7df0a2f0 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7df11960 con 0x555d7df08be0 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7df19de0 con 0x555d7df09530 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f194b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f194a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f194b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33516/0 (socket says 172.21.15.31:33516) 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f194b59e700 1 -- 172.21.15.31:0/651937920 learned_addr learned my addr 172.21.15.31:0/651937920 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.616+0000 7f194ad9d700 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194a59c700 1 -- 172.21.15.31:0/651937920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df08be0 msgr2=0x555d7df124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194a59c700 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194a59c700 1 -- 172.21.15.31:0/651937920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 msgr2=0x555d7df0d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194a59c700 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194a59c700 1 -- 172.21.15.31:0/651937920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d7dcae250 con 0x555d7df09530 2023-07-29T16:19:38.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194b59e700 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:38.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f194a59c700 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df129f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f19400a9970 tx=0x7f194009a2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0afacb5f82cd7a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f1931ffb700 1 -- 172.21.15.31:0/651937920 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400ca770 con 0x555d7df09530 2023-07-29T16:19:38.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f1931ffb700 1 -- 172.21.15.31:0/651937920 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f19400a02c0 con 0x555d7df09530 2023-07-29T16:19:38.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.617+0000 7f1931ffb700 1 -- 172.21.15.31:0/651937920 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19400a2310 con 0x555d7df09530 2023-07-29T16:19:38.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 -- 172.21.15.31:0/651937920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 msgr2=0x555d7df129f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df129f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f19400a9970 tx=0x7f194009a2e0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 -- 172.21.15.31:0/651937920 shutdown_connections 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df0a2f0 0x555d7df0d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df08be0 0x555d7df124b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/651937920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df129f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 -- 172.21.15.31:0/651937920 >> 172.21.15.31:0/651937920 conn(0x555d7defe3e0 msgr2=0x555d7df03060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 -- 172.21.15.31:0/651937920 shutdown_connections 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 -- 172.21.15.31:0/651937920 wait complete. 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 -- start start 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.618+0000 7f1964a8ab80 1 Processor -- start 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df6be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df6c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df0a2f0 0x555d7df73410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7df73950 con 0x555d7df08be0 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7df73a90 con 0x555d7df0a2f0 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d7df73c20 con 0x555d7df09530 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f1964a8ab80 10 client.0 _ll_register_callbacks cb 0x555d7ddff620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df6c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194b59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df0a2f0 0x555d7df73410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df6c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45236/0 (socket says 172.21.15.31:45236) 2023-07-29T16:19:38.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194a59c700 1 -- 172.21.15.31:0/309011003 learned_addr learned my addr 172.21.15.31:0/309011003 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:38.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194ad9d700 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df6be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194b59e700 1 -- 172.21.15.31:0/309011003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 msgr2=0x555d7df6c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194b59e700 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df6c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194b59e700 1 -- 172.21.15.31:0/309011003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 msgr2=0x555d7df6be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:38.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194b59e700 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df6be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.619+0000 7f194b59e700 1 -- 172.21.15.31:0/309011003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1944007660 con 0x555d7df0a2f0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f194b59e700 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df0a2f0 0x555d7df73410 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x555d7df206f0 tx=0x7f194401b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4426118cadfd449d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f192eff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f192eff5700 1 -- 172.21.15.31:0/309011003 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194400e040 con 0x555d7df0a2f0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f194009a3e0 con 0x555d7df0a2f0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f192eff5700 1 -- 172.21.15.31:0/309011003 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f194401bdf0 con 0x555d7df0a2f0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f192eff5700 1 -- 172.21.15.31:0/309011003 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1944004bb0 con 0x555d7df0a2f0 2023-07-29T16:19:38.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f1964a8ab80 10 client.26885 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.620+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x555d7dc70c60 con 0x555d7df0a2f0 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f192eff5700 1 -- 172.21.15.31:0/309011003 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f194009a3e0 con 0x555d7df0a2f0 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f192eff5700 1 -- 172.21.15.31:0/309011003 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f194402dd00 con 0x555d7df0a2f0 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f1964a8ab80 10 client.26885 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f1964a8ab80 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f1964a8ab80 20 client.26885 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:38.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df75700 0x555d7df77b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f1964a8ab80 4 client.26885 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:19:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555d7de52d40 con 0x555d7df75700 2023-07-29T16:19:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.621+0000 7f194ad9d700 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df75700 0x555d7df77b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.622+0000 7f192eff5700 10 client.26885 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.622+0000 7f194ad9d700 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df75700 0x555d7df77b10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b83d6c69b03e5ed1 server_cookie=6af54a5eeff7d26b in_seq=0 out_seq=0 2023-07-29T16:19:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f192eff5700 1 -- 172.21.15.31:0/309011003 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555d7de52d40 con 0x555d7df75700 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f192eff5700 10 client.26885 handle_command_reply: tid=0 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 client.26885 shutdown 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df0a2f0 msgr2=0x555d7df73410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df0a2f0 0x555d7df73410 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x555d7df206f0 tx=0x7f194401b730 comp rx=0 tx=0).stop 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 shutdown_connections 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555d7df75700 0x555d7df77b10 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d7df0a2f0 0x555d7df73410 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d7df08be0 0x555d7df6be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 --2- 172.21.15.31:0/309011003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555d7df09530 0x555d7df6c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f192eff5700 0 client.26885 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 >> 172.21.15.31:0/309011003 conn(0x555d7defe3e0 msgr2=0x555d7df12450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 shutdown_connections 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 1 -- 172.21.15.31:0/309011003 wait complete. 2023-07-29T16:19:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.623+0000 7f1964a8ab80 20 client.26885 trim_cache size 0 max 16384 2023-07-29T16:19:38.638 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:19:38.638 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:19:38.638 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:19:38.638 INFO:teuthology.orchestra.run.smithi031.stdout: "1b29ffc1-cf48-4e62-a23f-6bb0da9865f1": { 2023-07-29T16:19:38.638 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:19:38.639 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "1b29ffc1-cf48-4e62-a23f-6bb0da9865f1", 2023-07-29T16:19:38.639 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:19:38.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:19:38.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:19:38.639 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:19:38.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19340a0790 msgr2=0x7f19340a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19340a0790 0x7f19340a2c40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f194c101370 tx=0x7f193c009260 comp rx=0 tx=0).stop 2023-07-29T16:19:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 msgr2=0x7f194c196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 0x7f194c196580 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f1940004740 tx=0x7f1940009580 comp rx=0 tx=0).stop 2023-07-29T16:19:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 shutdown_connections 2023-07-29T16:19:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19340a0790 0x7f19340a2c40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f194c101ea0 0x7f194c19d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f194c100320 0x7f194c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 --2- 172.21.15.31:0/4154679745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f194c101510 0x7f194c196580 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 >> 172.21.15.31:0/4154679745 conn(0x7f194c0fb920 msgr2=0x7f194c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 shutdown_connections 2023-07-29T16:19:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:19:38.624+0000 7f19511ff700 1 -- 172.21.15.31:0/4154679745 wait complete. 2023-07-29T16:19:38.642 INFO:tasks.cephfs.filesystem:scrub status for tag:1b29ffc1-cf48-4e62-a23f-6bb0da9865f1 - {'path': '/', 'tag': '1b29ffc1-cf48-4e62-a23f-6bb0da9865f1', 'options': 'recursive,force'} 2023-07-29T16:19:38.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 141556 9.40 MB/sec warmup 83 sec latency 44.756 ms 2023-07-29T16:19:38.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:19:38.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:38.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:38.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:38.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:19:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:19:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:19:39.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 142737 9.36 MB/sec warmup 84 sec latency 16.880 ms 2023-07-29T16:19:40.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:39 smithi031 ceph-mon[95504]: pgmap v496: 97 pgs: 97 active+clean; 50 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 356 op/s 2023-07-29T16:19:40.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:39 smithi039 ceph-mon[128858]: pgmap v496: 97 pgs: 97 active+clean; 50 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 356 op/s 2023-07-29T16:19:40.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:39 smithi156 ceph-mon[129245]: pgmap v496: 97 pgs: 97 active+clean; 50 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 356 op/s 2023-07-29T16:19:40.860 INFO:tasks.workunit.client.0.smithi031.stdout: 1 145116 9.40 MB/sec warmup 85 sec latency 15.614 ms 2023-07-29T16:19:41.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:41 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:19:41.524 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:41 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:19:41.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:41 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:19:41.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 146013 9.36 MB/sec warmup 86 sec latency 130.604 ms 2023-07-29T16:19:42.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:42 smithi031 ceph-mon[95504]: pgmap v497: 97 pgs: 97 active+clean; 50 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 247 op/s 2023-07-29T16:19:42.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:19:42.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:19:42.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:19:42.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:42 smithi039 ceph-mon[128858]: pgmap v497: 97 pgs: 97 active+clean; 50 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 247 op/s 2023-07-29T16:19:42.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:19:42.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:19:42.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:19:42.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:42 smithi156 ceph-mon[129245]: pgmap v497: 97 pgs: 97 active+clean; 50 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 247 op/s 2023-07-29T16:19:42.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:19:42.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:19:42.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:19:42.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 147902 9.34 MB/sec warmup 87 sec latency 76.415 ms 2023-07-29T16:19:43.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 149393 9.36 MB/sec warmup 88 sec latency 58.033 ms 2023-07-29T16:19:44.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:44 smithi039 ceph-mon[128858]: pgmap v498: 97 pgs: 97 active+clean; 53 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 341 op/s 2023-07-29T16:19:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:44 smithi156 ceph-mon[129245]: pgmap v498: 97 pgs: 97 active+clean; 53 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 341 op/s 2023-07-29T16:19:44.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 150609 9.32 MB/sec warmup 89 sec latency 97.660 ms 2023-07-29T16:19:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:44 smithi031 ceph-mon[95504]: pgmap v498: 97 pgs: 97 active+clean; 53 MiB data, 723 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 341 op/s 2023-07-29T16:19:45.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 152819 9.37 MB/sec warmup 90 sec latency 11.690 ms 2023-07-29T16:19:46.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:46 smithi039 ceph-mon[128858]: pgmap v499: 97 pgs: 97 active+clean; 53 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 270 op/s 2023-07-29T16:19:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:46 smithi156 ceph-mon[129245]: pgmap v499: 97 pgs: 97 active+clean; 53 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 270 op/s 2023-07-29T16:19:46.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 154245 9.32 MB/sec warmup 91 sec latency 15.332 ms 2023-07-29T16:19:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:46 smithi031 ceph-mon[95504]: pgmap v499: 97 pgs: 97 active+clean; 53 MiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 270 op/s 2023-07-29T16:19:47.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 156393 9.37 MB/sec warmup 92 sec latency 17.146 ms 2023-07-29T16:19:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:47 smithi031 ceph-mon[95504]: pgmap v500: 97 pgs: 97 active+clean; 55 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2023-07-29T16:19:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:47 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:19:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:47 smithi039 ceph-mon[128858]: pgmap v500: 97 pgs: 97 active+clean; 55 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2023-07-29T16:19:48.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:47 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:19:48.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:47 smithi156 ceph-mon[129245]: pgmap v500: 97 pgs: 97 active+clean; 55 MiB data, 773 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.3 MiB/s wr, 296 op/s 2023-07-29T16:19:48.384 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:47 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:19:48.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 157731 9.33 MB/sec warmup 93 sec latency 20.793 ms 2023-07-29T16:19:49.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:48 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:19:49.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:48 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:19:49.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:48 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:19:49.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:48 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:19:49.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:48 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:19:49.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:48 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:19:49.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 -- 172.21.15.31:0/2618285767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 msgr2=0x7f1458108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2618285767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 0x7f1458108480 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1450009ec0 tx=0x7f145000b330 comp rx=0 tx=0).stop 2023-07-29T16:19:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 -- 172.21.15.31:0/2618285767 shutdown_connections 2023-07-29T16:19:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2618285767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 0x7f1458108480 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2618285767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2618285767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458103780 0x7f1458105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 -- 172.21.15.31:0/2618285767 >> 172.21.15.31:0/2618285767 conn(0x7f14580fa840 msgr2=0x7f14580fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 -- 172.21.15.31:0/2618285767 shutdown_connections 2023-07-29T16:19:49.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.829+0000 7f145e9ff700 1 -- 172.21.15.31:0/2618285767 wait complete. 2023-07-29T16:19:49.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.830+0000 7f145e9ff700 1 Processor -- start 2023-07-29T16:19:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 -- start start 2023-07-29T16:19:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458103780 0x7f14581966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 0x7f1458197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1458197b80 con 0x7f1458103780 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1458197cf0 con 0x7f14581060a0 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145e9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1458197eb0 con 0x7f1458100e60 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f1457fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f1457fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458196160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56272/0 (socket says 172.21.15.31:56272) 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f1457fff700 1 -- 172.21.15.31:0/2104003491 learned_addr learned my addr 172.21.15.31:0/2104003491 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:49.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f145cf9c700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 0x7f1458197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.831+0000 7f14577fe700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458103780 0x7f14581966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f1457fff700 1 -- 172.21.15.31:0/2104003491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 msgr2=0x7f1458197520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f1457fff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 0x7f1458197520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f1457fff700 1 -- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458103780 msgr2=0x7f14581966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f1457fff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458103780 0x7f14581966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f1457fff700 1 -- 172.21.15.31:0/2104003491 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1450005e70 con 0x7f1458100e60 2023-07-29T16:19:49.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f1457fff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458196160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f144800e970 tx=0x7f144800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f14557fa700 1 -- 172.21.15.31:0/2104003491 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448009790 con 0x7f1458100e60 2023-07-29T16:19:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f14557fa700 1 -- 172.21.15.31:0/2104003491 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144800c7e0 con 0x7f1458100e60 2023-07-29T16:19:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1458198190 con 0x7f1458100e60 2023-07-29T16:19:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.832+0000 7f14557fa700 1 -- 172.21.15.31:0/2104003491 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480054e0 con 0x7f1458100e60 2023-07-29T16:19:49.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.833+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145819e950 con 0x7f1458100e60 2023-07-29T16:19:49.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.837+0000 7f14557fa700 1 -- 172.21.15.31:0/2104003491 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f144800c950 con 0x7f1458100e60 2023-07-29T16:19:49.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.838+0000 7f14723b2b80 1 Processor -- start 2023-07-29T16:19:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 -- start start 2023-07-29T16:19:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dafe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61daea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61daf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61d029b0 con 0x556d61db0370 2023-07-29T16:19:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61d00370 con 0x556d61daef80 2023-07-29T16:19:49.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14723b2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61cfbc60 con 0x556d61c066b0 2023-07-29T16:19:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14557fa700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14400a0810 0x7f14400a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14577fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61daf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f145cf9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dafe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14577fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61daf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38428/0 (socket says 172.21.15.31:38428) 2023-07-29T16:19:49.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.839+0000 7f14577fe700 1 -- 172.21.15.31:0/3733136554 learned_addr learned my addr 172.21.15.31:0/3733136554 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14557fa700 1 -- 172.21.15.31:0/2104003491 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1448014070 con 0x7f1458100e60 2023-07-29T16:19:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f1457fff700 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61daea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14577fe700 1 -- 172.21.15.31:0/3733136554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 msgr2=0x556d61dafe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14577fe700 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dafe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14577fe700 1 -- 172.21.15.31:0/3733136554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 msgr2=0x556d61daea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14577fe700 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61daea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14577fe700 1 -- 172.21.15.31:0/3733136554 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d61b29d80 con 0x556d61daef80 2023-07-29T16:19:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f145cf9c700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14400a0810 0x7f14400a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f14577fe700 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61daf420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f144c009d80 tx=0x7f144c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f458afaa838ad38f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f143e7fc700 1 -- 172.21.15.31:0/3733136554 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c004230 con 0x556d61daef80 2023-07-29T16:19:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f143e7fc700 1 -- 172.21.15.31:0/3733136554 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144c004390 con 0x556d61daef80 2023-07-29T16:19:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.840+0000 7f143e7fc700 1 -- 172.21.15.31:0/3733136554 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c019880 con 0x556d61daef80 2023-07-29T16:19:49.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f145cf9c700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14400a0810 0x7f14400a2cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1450009fd0 tx=0x7f14500099e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 -- 172.21.15.31:0/3733136554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 msgr2=0x556d61daf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61daf420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f144c009d80 tx=0x7f144c011720 comp rx=0 tx=0).stop 2023-07-29T16:19:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 -- 172.21.15.31:0/3733136554 shutdown_connections 2023-07-29T16:19:49.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61daf420 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dafe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 --2- 172.21.15.31:0/3733136554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61daea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 -- 172.21.15.31:0/3733136554 >> 172.21.15.31:0/3733136554 conn(0x556d61d9ffd0 msgr2=0x556d61da5310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:49.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 -- 172.21.15.31:0/3733136554 shutdown_connections 2023-07-29T16:19:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.841+0000 7f14723b2b80 1 -- 172.21.15.31:0/3733136554 wait complete. 2023-07-29T16:19:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.842+0000 7f14723b2b80 1 -- start start 2023-07-29T16:19:49.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.842+0000 7f14723b2b80 1 Processor -- start 2023-07-29T16:19:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61db84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61db8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61c984b0 con 0x556d61db0370 2023-07-29T16:19:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61c98150 con 0x556d61c066b0 2023-07-29T16:19:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61b7f8c0 con 0x556d61daef80 2023-07-29T16:19:49.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14723b2b80 10 client.0 _ll_register_callbacks cb 0x556d61ca15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f1457fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61db84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f1457fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61db84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38444/0 (socket says 172.21.15.31:38444) 2023-07-29T16:19:49.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f1457fff700 1 -- 172.21.15.31:0/2468734598 learned_addr learned my addr 172.21.15.31:0/2468734598 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f14577fe700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61db8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.843+0000 7f145cf9c700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f1457fff700 1 -- 172.21.15.31:0/2468734598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 msgr2=0x556d61db8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f1457fff700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61db8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f1457fff700 1 -- 172.21.15.31:0/2468734598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 msgr2=0x556d61dbf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f1457fff700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f1457fff700 1 -- 172.21.15.31:0/2468734598 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144c0099e0 con 0x556d61c066b0 2023-07-29T16:19:49.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f145cf9c700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:19:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f1457fff700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61db84e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f144800efd0 tx=0x7f144800cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8b9e006839e2b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14377fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:19:49.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14377fe700 1 -- 172.21.15.31:0/2468734598 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480b1ea0 con 0x556d61c066b0 2023-07-29T16:19:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14377fe700 1 -- 172.21.15.31:0/2468734598 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f144809d4f0 con 0x556d61c066b0 2023-07-29T16:19:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d61d26cb0 con 0x556d61c066b0 2023-07-29T16:19:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14377fe700 1 -- 172.21.15.31:0/2468734598 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14480a1910 con 0x556d61c066b0 2023-07-29T16:19:49.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14723b2b80 10 client.26897 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556d61b12c00 con 0x556d61c066b0 2023-07-29T16:19:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.844+0000 7f14377fe700 1 -- 172.21.15.31:0/2468734598 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d61d26cb0 con 0x556d61c066b0 2023-07-29T16:19:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14377fe700 1 -- 172.21.15.31:0/2468734598 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f1448099870 con 0x556d61c066b0 2023-07-29T16:19:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14723b2b80 10 client.26897 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:49.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14723b2b80 10 client.26897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14723b2b80 20 client.26897 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61dc4270 0x556d61dc6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14723b2b80 4 client.26897 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:19:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d61d79d10 con 0x556d61dc4270 2023-07-29T16:19:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.845+0000 7f14577fe700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61dc4270 0x556d61dc6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.846+0000 7f14577fe700 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61dc4270 0x556d61dc6680 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be859db67256cadf server_cookie=4469818c43cb633d in_seq=0 out_seq=0 2023-07-29T16:19:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.846+0000 7f14377fe700 10 client.26897 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.846+0000 7f14377fe700 1 -- 172.21.15.31:0/2468734598 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556d61d79d10 con 0x556d61dc4270 2023-07-29T16:19:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.846+0000 7f14377fe700 10 client.26897 handle_command_reply: tid=0 2023-07-29T16:19:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.846+0000 7f14723b2b80 1 client.26897 shutdown 2023-07-29T16:19:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 msgr2=0x556d61db84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61db84e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f144800efd0 tx=0x7f144800cc50 comp rx=0 tx=0).stop 2023-07-29T16:19:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 shutdown_connections 2023-07-29T16:19:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61dc4270 0x556d61dc6680 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61db84e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2468734598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61db8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14377fe700 0 client.26897 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 >> 172.21.15.31:0/2468734598 conn(0x556d61d9ffd0 msgr2=0x556d61da4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 shutdown_connections 2023-07-29T16:19:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 1 -- 172.21.15.31:0/2468734598 wait complete. 2023-07-29T16:19:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.847+0000 7f14723b2b80 20 client.26897 trim_cache size 0 max 16384 2023-07-29T16:19:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.862+0000 7f14723b2b80 1 Processor -- start 2023-07-29T16:19:49.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.862+0000 7f14723b2b80 1 -- start start 2023-07-29T16:19:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.862+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61da63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.862+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61dbf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.862+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f14723b2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61dc40d0 con 0x556d61db0370 2023-07-29T16:19:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f14723b2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61db3ad0 con 0x556d61c066b0 2023-07-29T16:19:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f14723b2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61da6160 con 0x556d61daef80 2023-07-29T16:19:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61da63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f14577fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61dbf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61da63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38450/0 (socket says 172.21.15.31:38450) 2023-07-29T16:19:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 -- 172.21.15.31:0/2151211682 learned_addr learned my addr 172.21.15.31:0/2151211682 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:49.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f145cf9c700 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 159854 9.37 MB/sec warmup 94 sec latency 23.381 ms 2023-07-29T16:19:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 -- 172.21.15.31:0/2151211682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 msgr2=0x556d61dbf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61dbf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 -- 172.21.15.31:0/2151211682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 msgr2=0x556d61dae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 -- 172.21.15.31:0/2151211682 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d61b501f0 con 0x556d61c066b0 2023-07-29T16:19:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.863+0000 7f1457fff700 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61da63b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f14480959c0 tx=0x7f1448098ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903de4a25de59df9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f143effd700 1 -- 172.21.15.31:0/2151211682 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144800b2c0 con 0x556d61c066b0 2023-07-29T16:19:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f143effd700 1 -- 172.21.15.31:0/2151211682 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144809d4f0 con 0x556d61c066b0 2023-07-29T16:19:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f143effd700 1 -- 172.21.15.31:0/2151211682 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144809d330 con 0x556d61c066b0 2023-07-29T16:19:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 -- 172.21.15.31:0/2151211682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 msgr2=0x556d61da63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61da63b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f14480959c0 tx=0x7f1448098ad0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 -- 172.21.15.31:0/2151211682 shutdown_connections 2023-07-29T16:19:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61c066b0 0x556d61da63b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 --2- 172.21.15.31:0/2151211682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61daef80 0x556d61dbf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 -- 172.21.15.31:0/2151211682 >> 172.21.15.31:0/2151211682 conn(0x556d61d9ffd0 msgr2=0x556d61da52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 -- 172.21.15.31:0/2151211682 shutdown_connections 2023-07-29T16:19:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.864+0000 7f14723b2b80 1 -- 172.21.15.31:0/2151211682 wait complete. 2023-07-29T16:19:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.865+0000 7f14723b2b80 1 -- start start 2023-07-29T16:19:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.865+0000 7f14723b2b80 1 Processor -- start 2023-07-29T16:19:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dbb020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61dbb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbe420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61dbe960 con 0x556d61db0370 2023-07-29T16:19:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61dbead0 con 0x556d61daef80 2023-07-29T16:19:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d61dbec90 con 0x556d61c066b0 2023-07-29T16:19:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14723b2b80 10 client.0 _ll_register_callbacks cb 0x556d61ca15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:19:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dbb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14577fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61dbb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dbb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58652/0 (socket says 172.21.15.31:58652) 2023-07-29T16:19:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 -- 172.21.15.31:0/1189056181 learned_addr learned my addr 172.21.15.31:0/1189056181 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:19:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14577fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61dbb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38464/0 (socket says 172.21.15.31:38464) 2023-07-29T16:19:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f145cf9c700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbe420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 -- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 msgr2=0x556d61dbb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61dbb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 -- 172.21.15.31:0/1189056181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 msgr2=0x556d61dbe420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:19:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbe420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f1457fff700 1 -- 172.21.15.31:0/1189056181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d61b12c00 con 0x556d61c066b0 2023-07-29T16:19:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.866+0000 7f14577fe700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61dbb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:19:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.867+0000 7f1457fff700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dbb020 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f144800cc00 tx=0x7f144809d660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c889411fb4034fcb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:19:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.867+0000 7f14377fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:19:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.867+0000 7f14377fe700 1 -- 172.21.15.31:0/1189056181 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480a1910 con 0x556d61c066b0 2023-07-29T16:19:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.867+0000 7f14377fe700 1 -- 172.21.15.31:0/1189056181 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f144809d2b0 con 0x556d61c066b0 2023-07-29T16:19:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.867+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f144c004390 con 0x556d61c066b0 2023-07-29T16:19:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.874+0000 7f14377fe700 1 -- 172.21.15.31:0/1189056181 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f144800b2c0 con 0x556d61c066b0 2023-07-29T16:19:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.874+0000 7f14723b2b80 10 client.26911 fetch_fsmap learned FSMap version 26 2023-07-29T16:19:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.874+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556d61b28250 con 0x556d61c066b0 2023-07-29T16:19:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14377fe700 1 -- 172.21.15.31:0/1189056181 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144c004390 con 0x556d61c066b0 2023-07-29T16:19:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14377fe700 1 -- 172.21.15.31:0/1189056181 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f144809a2a0 con 0x556d61c066b0 2023-07-29T16:19:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14723b2b80 10 client.26911 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:19:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14723b2b80 10 client.26911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:19:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14723b2b80 20 client.26911 populate_metadata read hostname 'smithi031' 2023-07-29T16:19:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14723b2b80 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61e15f90 0x556d61e18370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:19:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14723b2b80 4 client.26911 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:19:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556d61cf4ce0 con 0x556d61e15f90 2023-07-29T16:19:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14577fe700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61e15f90 0x556d61e18370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:19:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.875+0000 7f14577fe700 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61e15f90 0x556d61e18370 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e942b3268f11e2e server_cookie=30ad85c05d9ae060 in_seq=0 out_seq=0 2023-07-29T16:19:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.876+0000 7f14377fe700 10 client.26911 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14377fe700 1 -- 172.21.15.31:0/1189056181 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556d61cf4ce0 con 0x556d61e15f90 2023-07-29T16:19:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14377fe700 10 client.26911 handle_command_reply: tid=0 2023-07-29T16:19:49.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 client.26911 shutdown 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 msgr2=0x556d61dbb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dbb020 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f144800cc00 tx=0x7f144809d660 comp rx=0 tx=0).stop 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 shutdown_connections 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d61e15f90 0x556d61e18370 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d61db0370 0x556d61dbe420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d61c066b0 0x556d61dbb020 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 --2- 172.21.15.31:0/1189056181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d61daef80 0x556d61dbb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 >> 172.21.15.31:0/1189056181 conn(0x556d61d9ffd0 msgr2=0x556d61db2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14377fe700 0 client.26911 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:19:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 shutdown_connections 2023-07-29T16:19:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.946+0000 7f14723b2b80 1 -- 172.21.15.31:0/1189056181 wait complete. 2023-07-29T16:19:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.947+0000 7f14723b2b80 20 client.26911 trim_cache size 0 max 16384 2023-07-29T16:19:49.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:19:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:19:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:19:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.947+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14400a0810 msgr2=0x7f14400a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.947+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14400a0810 0x7f14400a2cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1450009fd0 tx=0x7f14500099e0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.947+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 msgr2=0x7f1458196160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.947+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458196160 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f144800e970 tx=0x7f144800ec80 comp rx=0 tx=0).stop 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 shutdown_connections 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14400a0810 0x7f14400a2cc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f14581060a0 0x7f1458197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1458100e60 0x7f1458196160 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 --2- 172.21.15.31:0/2104003491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1458103780 0x7f14581966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:19:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 >> 172.21.15.31:0/2104003491 conn(0x7f14580fa840 msgr2=0x7f1458101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:19:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 shutdown_connections 2023-07-29T16:19:49.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:19:49.948+0000 7f145e9ff700 1 -- 172.21.15.31:0/2104003491 wait complete. 2023-07-29T16:19:50.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:49 smithi031 ceph-mon[95504]: pgmap v501: 97 pgs: 97 active+clean; 57 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2023-07-29T16:19:50.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:49 smithi039 ceph-mon[128858]: pgmap v501: 97 pgs: 97 active+clean; 57 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2023-07-29T16:19:50.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:49 smithi156 ceph-mon[129245]: pgmap v501: 97 pgs: 97 active+clean; 57 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2023-07-29T16:19:50.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 161065 9.33 MB/sec warmup 95 sec latency 43.181 ms 2023-07-29T16:19:51.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 163364 9.37 MB/sec warmup 96 sec latency 15.961 ms 2023-07-29T16:19:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:52 smithi039 ceph-mon[128858]: pgmap v502: 97 pgs: 97 active+clean; 57 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2023-07-29T16:19:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:52 smithi156 ceph-mon[129245]: pgmap v502: 97 pgs: 97 active+clean; 57 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2023-07-29T16:19:52.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 164689 9.34 MB/sec warmup 97 sec latency 13.042 ms 2023-07-29T16:19:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:52 smithi031 ceph-mon[95504]: pgmap v502: 97 pgs: 97 active+clean; 57 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.9 MiB/s wr, 201 op/s 2023-07-29T16:19:53.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 166891 9.37 MB/sec warmup 98 sec latency 45.142 ms 2023-07-29T16:19:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:54 smithi039 ceph-mon[128858]: pgmap v503: 97 pgs: 97 active+clean; 56 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 315 op/s 2023-07-29T16:19:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:54 smithi156 ceph-mon[129245]: pgmap v503: 97 pgs: 97 active+clean; 56 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 315 op/s 2023-07-29T16:19:54.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 168225 9.34 MB/sec warmup 99 sec latency 19.358 ms 2023-07-29T16:19:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:54 smithi031 ceph-mon[95504]: pgmap v503: 97 pgs: 97 active+clean; 56 MiB data, 772 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 315 op/s 2023-07-29T16:19:55.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 170509 9.39 MB/sec warmup 100 sec latency 15.259 ms 2023-07-29T16:19:56.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:56 smithi039 ceph-mon[128858]: pgmap v504: 97 pgs: 97 active+clean; 54 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 254 op/s 2023-07-29T16:19:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:56 smithi156 ceph-mon[129245]: pgmap v504: 97 pgs: 97 active+clean; 54 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 254 op/s 2023-07-29T16:19:56.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 171805 9.35 MB/sec warmup 101 sec latency 12.922 ms 2023-07-29T16:19:56.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:56 smithi031 ceph-mon[95504]: pgmap v504: 97 pgs: 97 active+clean; 54 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 254 op/s 2023-07-29T16:19:57.861 INFO:tasks.workunit.client.0.smithi031.stdout: 1 174058 9.39 MB/sec warmup 102 sec latency 19.164 ms 2023-07-29T16:19:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:19:57 smithi031 ceph-mon[95504]: pgmap v505: 97 pgs: 97 active+clean; 55 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-07-29T16:19:58.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:19:57 smithi039 ceph-mon[128858]: pgmap v505: 97 pgs: 97 active+clean; 55 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-07-29T16:19:58.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:19:57 smithi156 ceph-mon[129245]: pgmap v505: 97 pgs: 97 active+clean; 55 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-07-29T16:19:58.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 175315 9.35 MB/sec warmup 103 sec latency 30.893 ms 2023-07-29T16:19:59.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 177523 9.38 MB/sec warmup 104 sec latency 38.085 ms 2023-07-29T16:20:00.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:00 smithi039 ceph-mon[128858]: pgmap v506: 97 pgs: 97 active+clean; 59 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 314 op/s 2023-07-29T16:20:00.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:00 smithi039 ceph-mon[128858]: overall HEALTH_OK 2023-07-29T16:20:00.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:00 smithi156 ceph-mon[129245]: pgmap v506: 97 pgs: 97 active+clean; 59 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 314 op/s 2023-07-29T16:20:00.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:00 smithi156 ceph-mon[129245]: overall HEALTH_OK 2023-07-29T16:20:00.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 178368 9.31 MB/sec warmup 105 sec latency 99.533 ms 2023-07-29T16:20:00.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:00 smithi031 ceph-mon[95504]: pgmap v506: 97 pgs: 97 active+clean; 59 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 314 op/s 2023-07-29T16:20:00.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:00 smithi031 ceph-mon[95504]: overall HEALTH_OK 2023-07-29T16:20:01.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 180874 9.38 MB/sec warmup 106 sec latency 15.716 ms 2023-07-29T16:20:02.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:02 smithi039 ceph-mon[128858]: pgmap v507: 97 pgs: 97 active+clean; 59 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 265 op/s 2023-07-29T16:20:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:02 smithi156 ceph-mon[129245]: pgmap v507: 97 pgs: 97 active+clean; 59 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 265 op/s 2023-07-29T16:20:02.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 182067 9.33 MB/sec warmup 107 sec latency 10.690 ms 2023-07-29T16:20:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:02 smithi031 ceph-mon[95504]: pgmap v507: 97 pgs: 97 active+clean; 59 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.1 MiB/s wr, 265 op/s 2023-07-29T16:20:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 -- 172.21.15.31:0/357858644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 msgr2=0x7f364c101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 --2- 172.21.15.31:0/357858644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c101980 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3640009990 tx=0x7f3640009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 -- 172.21.15.31:0/357858644 shutdown_connections 2023-07-29T16:20:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 --2- 172.21.15.31:0/357858644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c101ec0 0x7f364c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 --2- 172.21.15.31:0/357858644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f364c100340 0x7f364c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 --2- 172.21.15.31:0/357858644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c101980 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 -- 172.21.15.31:0/357858644 >> 172.21.15.31:0/357858644 conn(0x7f364c0fb920 msgr2=0x7f364c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 -- 172.21.15.31:0/357858644 shutdown_connections 2023-07-29T16:20:03.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.116+0000 7f36525ff700 1 -- 172.21.15.31:0/357858644 wait complete. 2023-07-29T16:20:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.117+0000 7f36525ff700 1 Processor -- start 2023-07-29T16:20:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 -- start start 2023-07-29T16:20:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c100340 0x7f364c1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f364c101ec0 0x7f364c199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c199de0 con 0x7f364c100340 2023-07-29T16:20:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c199f50 con 0x7f364c101530 2023-07-29T16:20:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f36525ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c19a110 con 0x7f364c101ec0 2023-07-29T16:20:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f364bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c100340 0x7f364c1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:32924/0 (socket says 172.21.15.31:32924) 2023-07-29T16:20:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.118+0000 7f364b7fe700 1 -- 172.21.15.31:0/524613888 learned_addr learned my addr 172.21.15.31:0/524613888 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f3650b9c700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f364c101ec0 0x7f364c199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f364b7fe700 1 -- 172.21.15.31:0/524613888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f364c101ec0 msgr2=0x7f364c199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f364b7fe700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f364c101ec0 0x7f364c199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f364b7fe700 1 -- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c100340 msgr2=0x7f364c1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f364b7fe700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c100340 0x7f364c1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f364b7fe700 1 -- 172.21.15.31:0/524613888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400095f0 con 0x7f364c101530 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f364b7fe700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c198900 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f364000b620 tx=0x7f3640005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f36497fa700 1 -- 172.21.15.31:0/524613888 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364001d070 con 0x7f364c101530 2023-07-29T16:20:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f36497fa700 1 -- 172.21.15.31:0/524613888 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3640022470 con 0x7f364c101530 2023-07-29T16:20:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.119+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c19a390 con 0x7f364c101530 2023-07-29T16:20:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.120+0000 7f36497fa700 1 -- 172.21.15.31:0/524613888 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640017650 con 0x7f364c101530 2023-07-29T16:20:03.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.120+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c1a0bb0 con 0x7f364c101530 2023-07-29T16:20:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.123+0000 7f36497fa700 1 -- 172.21.15.31:0/524613888 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f36400225e0 con 0x7f364c101530 2023-07-29T16:20:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.124+0000 7f3665eb0b80 1 Processor -- start 2023-07-29T16:20:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.124+0000 7f3665eb0b80 1 -- start start 2023-07-29T16:20:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcba8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcbb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efc129b0 con 0x55e5efcbadf0 2023-07-29T16:20:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efc10370 con 0x55e5efcbbcf0 2023-07-29T16:20:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efc0bc60 con 0x55e5efb166b0 2023-07-29T16:20:03.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f36497fa700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36340a0810 0x7f36340a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:32940/0 (socket says 172.21.15.31:32940) 2023-07-29T16:20:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f364b7fe700 1 -- 172.21.15.31:0/4256899174 learned_addr learned my addr 172.21.15.31:0/4256899174 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:03.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f36497fa700 1 -- 172.21.15.31:0/524613888 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f36400ccff0 con 0x7f364c101530 2023-07-29T16:20:03.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f364bfff700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcbb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.125+0000 7f3650b9c700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcba8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364b7fe700 1 -- 172.21.15.31:0/4256899174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 msgr2=0x55e5efcba8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364b7fe700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcba8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364b7fe700 1 -- 172.21.15.31:0/4256899174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 msgr2=0x55e5efcbb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364b7fe700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcbb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364b7fe700 1 -- 172.21.15.31:0/4256899174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5efa39d80 con 0x55e5efcbbcf0 2023-07-29T16:20:03.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364bfff700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcbb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:03.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f3650b9c700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcba8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f3650b9c700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36340a0810 0x7f36340a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f364b7fe700 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbc160 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3640022890 tx=0x7f364006dc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb1ff9f11427f382 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f36327fc700 1 -- 172.21.15.31:0/4256899174 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364009a0c0 con 0x55e5efcbbcf0 2023-07-29T16:20:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f36327fc700 1 -- 172.21.15.31:0/4256899174 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3640011110 con 0x55e5efcbbcf0 2023-07-29T16:20:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.126+0000 7f36327fc700 1 -- 172.21.15.31:0/4256899174 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640098b90 con 0x55e5efcbbcf0 2023-07-29T16:20:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4256899174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 msgr2=0x55e5efcbc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbc160 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3640022890 tx=0x7f364006dc20 comp rx=0 tx=0).stop 2023-07-29T16:20:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3650b9c700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36340a0810 0x7f36340a2cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3644009550 tx=0x7f3644009100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4256899174 shutdown_connections 2023-07-29T16:20:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbc160 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcba8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4256899174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcbb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4256899174 >> 172.21.15.31:0/4256899174 conn(0x55e5efcaffd0 msgr2=0x55e5efcb5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4256899174 shutdown_connections 2023-07-29T16:20:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.127+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4256899174 wait complete. 2023-07-29T16:20:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.128+0000 7f3665eb0b80 1 -- start start 2023-07-29T16:20:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.128+0000 7f3665eb0b80 1 Processor -- start 2023-07-29T16:20:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efb166b0 0x55e5efcc3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efcc40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efccb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efba84b0 con 0x55e5efb166b0 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efba8150 con 0x55e5efcbbcf0 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efa8f8c0 con 0x55e5efcbadf0 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3665eb0b80 10 client.0 _ll_register_callbacks cb 0x55e5efbb15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efccb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efcc40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f364bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efb166b0 0x55e5efcc3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efccb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:32950/0 (socket says 172.21.15.31:32950) 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 -- 172.21.15.31:0/4079401877 learned_addr learned my addr 172.21.15.31:0/4079401877 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 -- 172.21.15.31:0/4079401877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 msgr2=0x55e5efcc40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efcc40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 -- 172.21.15.31:0/4079401877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efb166b0 msgr2=0x55e5efcc3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efb166b0 0x55e5efcc3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f3650b9c700 1 -- 172.21.15.31:0/4079401877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c1a0bb0 con 0x55e5efcbbcf0 2023-07-29T16:20:03.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.129+0000 7f364b7fe700 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efcc40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f3650b9c700 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efccb0f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f36440198a0 tx=0x7f3644004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b870f7da9f7f80b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f362b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f362b7fe700 1 -- 172.21.15.31:0/4079401877 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644018730 con 0x55e5efcbbcf0 2023-07-29T16:20:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f362b7fe700 1 -- 172.21.15.31:0/4079401877 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3644018890 con 0x55e5efcbbcf0 2023-07-29T16:20:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e5efc36cb0 con 0x55e5efcbbcf0 2023-07-29T16:20:03.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f362b7fe700 1 -- 172.21.15.31:0/4079401877 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3644018a50 con 0x55e5efcbbcf0 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f3665eb0b80 10 client.26915 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.130+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e5efa22c00 con 0x55e5efcbbcf0 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f362b7fe700 1 -- 172.21.15.31:0/4079401877 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3644032080 con 0x55e5efcbbcf0 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f362b7fe700 1 -- 172.21.15.31:0/4079401877 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f3644024c50 con 0x55e5efcbbcf0 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f3665eb0b80 10 client.26915 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f3665eb0b80 10 client.26915 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:03.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f3665eb0b80 20 client.26915 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efccfa40 0x55e5efcd1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f3665eb0b80 4 client.26915 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5efc89d10 con 0x55e5efccfa40 2023-07-29T16:20:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.131+0000 7f364bfff700 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efccfa40 0x55e5efcd1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.132+0000 7f364bfff700 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efccfa40 0x55e5efcd1e50 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c97ee3c82d431f8 server_cookie=6b0ee1248f84781c in_seq=0 out_seq=0 2023-07-29T16:20:03.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.132+0000 7f362b7fe700 10 client.26915 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.132+0000 7f362b7fe700 1 -- 172.21.15.31:0/4079401877 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e5efc89d10 con 0x55e5efccfa40 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.132+0000 7f362b7fe700 10 client.26915 handle_command_reply: tid=0 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.132+0000 7f3665eb0b80 1 client.26915 shutdown 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 msgr2=0x55e5efccb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efccb0f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f36440198a0 tx=0x7f3644004b40 comp rx=0 tx=0).stop 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 shutdown_connections 2023-07-29T16:20:03.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efccfa40 0x55e5efcd1e50 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efccb0f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efb166b0 0x55e5efcc3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/4079401877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efcc40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f362b7fe700 0 client.26915 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 >> 172.21.15.31:0/4079401877 conn(0x55e5efcaffd0 msgr2=0x55e5efcb4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 shutdown_connections 2023-07-29T16:20:03.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 1 -- 172.21.15.31:0/4079401877 wait complete. 2023-07-29T16:20:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.133+0000 7f3665eb0b80 20 client.26915 trim_cache size 0 max 16384 2023-07-29T16:20:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.150+0000 7f3665eb0b80 1 Processor -- start 2023-07-29T16:20:03.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.150+0000 7f3665eb0b80 1 -- start start 2023-07-29T16:20:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.150+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efb166b0 0x55e5efcbea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.150+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efccb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.150+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efcb2050 con 0x55e5efcbbcf0 2023-07-29T16:20:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efcc3020 con 0x55e5efb166b0 2023-07-29T16:20:03.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efcc36e0 con 0x55e5efcbadf0 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efccb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efb166b0 0x55e5efcbea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efccb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32798/0 (socket says 172.21.15.31:32798) 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 -- 172.21.15.31:0/1168066085 learned_addr learned my addr 172.21.15.31:0/1168066085 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f3650b9c700 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 -- 172.21.15.31:0/1168066085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efb166b0 msgr2=0x55e5efcbea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efb166b0 0x55e5efcbea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 -- 172.21.15.31:0/1168066085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbbcf0 msgr2=0x55e5efcbefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 -- 172.21.15.31:0/1168066085 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5efa601f0 con 0x55e5efcbadf0 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f3650b9c700 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.151+0000 7f364b7fe700 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efccb0b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3640011280 tx=0x7f3640005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fabffb0138f42647 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.152+0000 7f3632ffd700 1 -- 172.21.15.31:0/1168066085 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364009a760 con 0x55e5efcbadf0 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.152+0000 7f3632ffd700 1 -- 172.21.15.31:0/1168066085 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f364009e790 con 0x55e5efcbadf0 2023-07-29T16:20:03.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.152+0000 7f3632ffd700 1 -- 172.21.15.31:0/1168066085 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640099f20 con 0x55e5efcbadf0 2023-07-29T16:20:03.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.152+0000 7f3665eb0b80 1 -- 172.21.15.31:0/1168066085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 msgr2=0x55e5efccb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.152+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efccb0b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3640011280 tx=0x7f3640005c90 comp rx=0 tx=0).stop 2023-07-29T16:20:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 -- 172.21.15.31:0/1168066085 shutdown_connections 2023-07-29T16:20:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbbcf0 0x55e5efcbefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efb166b0 0x55e5efcbea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/1168066085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efcbadf0 0x55e5efccb0b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 -- 172.21.15.31:0/1168066085 >> 172.21.15.31:0/1168066085 conn(0x55e5efcaffd0 msgr2=0x55e5efcb4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 -- 172.21.15.31:0/1168066085 shutdown_connections 2023-07-29T16:20:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 -- 172.21.15.31:0/1168066085 wait complete. 2023-07-29T16:20:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 -- start start 2023-07-29T16:20:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.153+0000 7f3665eb0b80 1 Processor -- start 2023-07-29T16:20:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcc6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcc7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efcca7e0 con 0x55e5efcbadf0 2023-07-29T16:20:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efcca950 con 0x55e5efcbbcf0 2023-07-29T16:20:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5efccab10 con 0x55e5efb166b0 2023-07-29T16:20:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3665eb0b80 10 client.0 _ll_register_callbacks cb 0x55e5efbb15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcc6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f3650b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcc6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32810/0 (socket says 172.21.15.31:32810) 2023-07-29T16:20:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 -- 172.21.15.31:0/407715941 learned_addr learned my addr 172.21.15.31:0/407715941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364b7fe700 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcc7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 -- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 msgr2=0x55e5efcca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcca2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 -- 172.21.15.31:0/407715941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 msgr2=0x55e5efcc7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcc7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.154+0000 7f364bfff700 1 -- 172.21.15.31:0/407715941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5efa39d80 con 0x55e5efb166b0 2023-07-29T16:20:03.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f364bfff700 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcc6d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x55e5efcd26c0 tx=0x7f363c004320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c22f2cfc466680 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f362b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f362b7fe700 1 -- 172.21.15.31:0/407715941 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c017040 con 0x55e5efb166b0 2023-07-29T16:20:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3640011110 con 0x55e5efb166b0 2023-07-29T16:20:03.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f362b7fe700 1 -- 172.21.15.31:0/407715941 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f363c003c40 con 0x55e5efb166b0 2023-07-29T16:20:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f362b7fe700 1 -- 172.21.15.31:0/407715941 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f363c003e00 con 0x55e5efb166b0 2023-07-29T16:20:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f3665eb0b80 10 client.26929 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.155+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e5efa38250 con 0x55e5efb166b0 2023-07-29T16:20:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f362b7fe700 1 -- 172.21.15.31:0/407715941 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3640011110 con 0x55e5efb166b0 2023-07-29T16:20:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f362b7fe700 1 -- 172.21.15.31:0/407715941 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f363c0108e0 con 0x55e5efb166b0 2023-07-29T16:20:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f3665eb0b80 10 client.26929 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f3665eb0b80 10 client.26929 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f3665eb0b80 20 client.26929 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efd223c0 0x55e5efd247d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f3665eb0b80 4 client.26929 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:20:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e5efc04ce0 con 0x55e5efd223c0 2023-07-29T16:20:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.156+0000 7f364b7fe700 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efd223c0 0x55e5efd247d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.157+0000 7f364b7fe700 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efd223c0 0x55e5efd247d0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffdddcde7e2b8c68 server_cookie=a2a8d6bf6ab4843f in_seq=0 out_seq=0 2023-07-29T16:20:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.157+0000 7f362b7fe700 10 client.26929 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f362b7fe700 1 -- 172.21.15.31:0/407715941 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e5efc04ce0 con 0x55e5efd223c0 2023-07-29T16:20:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f362b7fe700 10 client.26929 handle_command_reply: tid=0 2023-07-29T16:20:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 client.26929 shutdown 2023-07-29T16:20:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 msgr2=0x55e5efcc6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcc6d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x55e5efcd26c0 tx=0x7f363c004320 comp rx=0 tx=0).stop 2023-07-29T16:20:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 shutdown_connections 2023-07-29T16:20:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e5efd223c0 0x55e5efd247d0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5efcbbcf0 0x55e5efcca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f362b7fe700 0 client.26929 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e5efb166b0 0x55e5efcc6d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 --2- 172.21.15.31:0/407715941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e5efcbadf0 0x55e5efcc7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 >> 172.21.15.31:0/407715941 conn(0x55e5efcaffd0 msgr2=0x55e5efcbea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 shutdown_connections 2023-07-29T16:20:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.227+0000 7f3665eb0b80 1 -- 172.21.15.31:0/407715941 wait complete. 2023-07-29T16:20:03.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f3665eb0b80 20 client.26929 trim_cache size 0 max 16384 2023-07-29T16:20:03.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:20:03.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:20:03.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:20:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:20:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36340a0810 msgr2=0x7f36340a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36340a0810 0x7f36340a2cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3644009550 tx=0x7f3644009100 comp rx=0 tx=0).stop 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 msgr2=0x7f364c198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c198900 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f364000b620 tx=0x7f3640005740 comp rx=0 tx=0).stop 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 shutdown_connections 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36340a0810 0x7f36340a2cc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f364c101ec0 0x7f364c199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f364c100340 0x7f364c1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.228+0000 7f36525ff700 1 --2- 172.21.15.31:0/524613888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f364c101530 0x7f364c198900 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.229+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 >> 172.21.15.31:0/524613888 conn(0x7f364c0fb920 msgr2=0x7f364c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.229+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 shutdown_connections 2023-07-29T16:20:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:03.229+0000 7f36525ff700 1 -- 172.21.15.31:0/524613888 wait complete. 2023-07-29T16:20:03.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 184275 9.36 MB/sec warmup 108 sec latency 49.405 ms 2023-07-29T16:20:04.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:04 smithi039 ceph-mon[128858]: pgmap v508: 97 pgs: 97 active+clean; 60 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2023-07-29T16:20:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:04 smithi156 ceph-mon[129245]: pgmap v508: 97 pgs: 97 active+clean; 60 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2023-07-29T16:20:04.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 185247 9.32 MB/sec warmup 109 sec latency 33.399 ms 2023-07-29T16:20:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:04 smithi031 ceph-mon[95504]: pgmap v508: 97 pgs: 97 active+clean; 60 MiB data, 836 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2023-07-29T16:20:05.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 187428 9.34 MB/sec warmup 110 sec latency 15.967 ms 2023-07-29T16:20:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:06 smithi039 ceph-mon[128858]: pgmap v509: 97 pgs: 97 active+clean; 62 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 281 op/s 2023-07-29T16:20:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:06 smithi156 ceph-mon[129245]: pgmap v509: 97 pgs: 97 active+clean; 62 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 281 op/s 2023-07-29T16:20:06.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 188599 9.32 MB/sec warmup 111 sec latency 36.034 ms 2023-07-29T16:20:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:06 smithi031 ceph-mon[95504]: pgmap v509: 97 pgs: 97 active+clean; 62 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 281 op/s 2023-07-29T16:20:07.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 190921 9.35 MB/sec warmup 112 sec latency 16.883 ms 2023-07-29T16:20:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:07 smithi031 ceph-mon[95504]: pgmap v510: 97 pgs: 97 active+clean; 58 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 262 op/s 2023-07-29T16:20:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:07 smithi039 ceph-mon[128858]: pgmap v510: 97 pgs: 97 active+clean; 58 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 262 op/s 2023-07-29T16:20:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:07 smithi156 ceph-mon[129245]: pgmap v510: 97 pgs: 97 active+clean; 58 MiB data, 821 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 262 op/s 2023-07-29T16:20:08.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:20:08.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 192140 9.32 MB/sec warmup 113 sec latency 12.496 ms 2023-07-29T16:20:08.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.870+0000 7f4aa91ff700 1 -- 172.21.15.31:0/2348215098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa41036c0 msgr2=0x7f4aa4105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.870+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/2348215098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa41036c0 0x7f4aa4105aa0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009990 tx=0x7f4a98009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 -- 172.21.15.31:0/2348215098 shutdown_connections 2023-07-29T16:20:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/2348215098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4105fe0 0x7f4aa4108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/2348215098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 0x7f4aa4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/2348215098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa41036c0 0x7f4aa4105aa0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 -- 172.21.15.31:0/2348215098 >> 172.21.15.31:0/2348215098 conn(0x7f4aa40fa840 msgr2=0x7f4aa40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 -- 172.21.15.31:0/2348215098 shutdown_connections 2023-07-29T16:20:08.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.871+0000 7f4aa91ff700 1 -- 172.21.15.31:0/2348215098 wait complete. 2023-07-29T16:20:08.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.872+0000 7f4aa91ff700 1 Processor -- start 2023-07-29T16:20:08.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.872+0000 7f4aa91ff700 1 -- start start 2023-07-29T16:20:08.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.872+0000 7f4aa91ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 0x7f4aa41982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.872+0000 7f4aa91ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa41036c0 0x7f4aa41987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa91ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 0x7f4aa419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa91ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa419fe30 con 0x7f4aa4100da0 2023-07-29T16:20:08.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa91ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa419ff70 con 0x7f4aa4105fe0 2023-07-29T16:20:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa91ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa41a0100 con 0x7f4aa41036c0 2023-07-29T16:20:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 0x7f4aa419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 0x7f4aa41982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 0x7f4aa419f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33004/0 (socket says 172.21.15.31:33004) 2023-07-29T16:20:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 -- 172.21.15.31:0/421595890 learned_addr learned my addr 172.21.15.31:0/421595890 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa259c700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa41036c0 0x7f4aa41987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 -- 172.21.15.31:0/421595890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa41036c0 msgr2=0x7f4aa41987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa41036c0 0x7f4aa41987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 -- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 msgr2=0x7f4aa41982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 0x7f4aa41982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa359e700 1 -- 172.21.15.31:0/421595890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980095f0 con 0x7f4aa4105fe0 2023-07-29T16:20:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.873+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 0x7f4aa41982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.874+0000 7f4aa359e700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 0x7f4aa419f860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c007ee0 tx=0x7f4a9c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:08.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.874+0000 7f4a8bfff700 1 -- 172.21.15.31:0/421595890 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c00dd20 con 0x7f4aa4105fe0 2023-07-29T16:20:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.874+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa41a03e0 con 0x7f4aa4105fe0 2023-07-29T16:20:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.874+0000 7f4a8bfff700 1 -- 172.21.15.31:0/421595890 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a9c00f040 con 0x7f4aa4105fe0 2023-07-29T16:20:08.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.874+0000 7f4a8bfff700 1 -- 172.21.15.31:0/421595890 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c0048a0 con 0x7f4aa4105fe0 2023-07-29T16:20:08.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.874+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa41a08a0 con 0x7f4aa4105fe0 2023-07-29T16:20:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.878+0000 7f4a8bfff700 1 -- 172.21.15.31:0/421595890 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4a9c003a00 con 0x7f4aa4105fe0 2023-07-29T16:20:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.880+0000 7f4a8bfff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a8c0a0790 0x7f4a8c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.880+0000 7f4abcb32b80 1 Processor -- start 2023-07-29T16:20:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.880+0000 7f4a8bfff700 1 -- 172.21.15.31:0/421595890 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a9c0cb360 con 0x7f4aa4105fe0 2023-07-29T16:20:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.880+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a8c0a0790 0x7f4a8c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.880+0000 7f4abcb32b80 1 -- start start 2023-07-29T16:20:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.880+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f58ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f59d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f5a2f0 0x559502f5a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502eb0a10 con 0x559502f58be0 2023-07-29T16:20:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502eae3d0 con 0x559502f5a2f0 2023-07-29T16:20:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502ea9cc0 con 0x559502f59530 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f59d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f59d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32834/0 (socket says 172.21.15.31:32834) 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 -- 172.21.15.31:0/1527745417 learned_addr learned my addr 172.21.15.31:0/1527745417 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f58ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa359e700 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f5a2f0 0x559502f5a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 -- 172.21.15.31:0/1527745417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f5a2f0 msgr2=0x559502f5a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f5a2f0 0x559502f5a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 -- 172.21.15.31:0/1527745417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 msgr2=0x559502f58ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f58ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a8c0a0790 0x7f4a8c0a2c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011550 tx=0x7f4a94009f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.881+0000 7f4aa259c700 1 -- 172.21.15.31:0/1527745417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559502cd7de0 con 0x559502f59530 2023-07-29T16:20:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.882+0000 7f4aa259c700 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f59d60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a98015000 tx=0x7f4a98005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f62a09eaab43d38 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.882+0000 7f4a897fa700 1 -- 172.21.15.31:0/1527745417 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9801d070 con 0x559502f59530 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.882+0000 7f4a897fa700 1 -- 172.21.15.31:0/1527745417 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a9801e040 con 0x559502f59530 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.882+0000 7f4a897fa700 1 -- 172.21.15.31:0/1527745417 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98004980 con 0x559502f59530 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1527745417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 msgr2=0x559502f59d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f59d60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4a98015000 tx=0x7f4a98005bc0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1527745417 shutdown_connections 2023-07-29T16:20:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f5a2f0 0x559502f5a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f58ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1527745417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f59d60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1527745417 >> 172.21.15.31:0/1527745417 conn(0x559502f4e3e0 msgr2=0x559502f53990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1527745417 shutdown_connections 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1527745417 wait complete. 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.883+0000 7f4abcb32b80 1 -- start start 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 Processor -- start 2023-07-29T16:20:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f627e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 0x559502f62d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f69d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502e46510 con 0x559502f58be0 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502e461b0 con 0x559502f59530 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502d2d920 con 0x559502f5a2f0 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4abcb32b80 10 client.0 _ll_register_callbacks cb 0x559502e4f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f627e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.884+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 0x559502f62d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 0x559502f62d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33016/0 (socket says 172.21.15.31:33016) 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 -- 172.21.15.31:0/3920126482 learned_addr learned my addr 172.21.15.31:0/3920126482 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f627e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32844/0 (socket says 172.21.15.31:32844) 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa359e700 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f69d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 -- 172.21.15.31:0/3920126482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 msgr2=0x559502f69d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f69d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 -- 172.21.15.31:0/3920126482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 msgr2=0x559502f627e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f627e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 -- 172.21.15.31:0/3920126482 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa4109030 con 0x559502f59530 2023-07-29T16:20:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4aa259c700 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 0x559502f62d20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4a98005920 tx=0x7f4a980226b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5987cfe0621a87 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4a86ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4a86ff5700 1 -- 172.21.15.31:0/3920126482 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a98032070 con 0x559502f59530 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4a86ff5700 1 -- 172.21.15.31:0/3920126482 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a98022b70 con 0x559502f59530 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.885+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559502ed4d10 con 0x559502f59530 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4a86ff5700 1 -- 172.21.15.31:0/3920126482 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a9801d070 con 0x559502f59530 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 10 client.26933 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x559502cc0c60 con 0x559502f59530 2023-07-29T16:20:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4a86ff5700 1 -- 172.21.15.31:0/3920126482 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a9803c020 con 0x559502f59530 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4a86ff5700 1 -- 172.21.15.31:0/3920126482 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f4a98035030 con 0x559502f59530 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 10 client.26933 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 10 client.26933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 20 client.26933 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502f6e100 0x559502f70510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 4 client.26933 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.886+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559502f27d70 con 0x559502f6e100 2023-07-29T16:20:08.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.887+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502f6e100 0x559502f70510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.887+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502f6e100 0x559502f70510 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93793332263dc544 server_cookie=f1b619608da38177 in_seq=0 out_seq=0 2023-07-29T16:20:08.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.887+0000 7f4a86ff5700 10 client.26933 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:08.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4a86ff5700 1 -- 172.21.15.31:0/3920126482 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559502f27d70 con 0x559502f6e100 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4a86ff5700 10 client.26933 handle_command_reply: tid=0 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 client.26933 shutdown 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 msgr2=0x559502f62d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 0x559502f62d20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4a98005920 tx=0x7f4a980226b0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 shutdown_connections 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502f6e100 0x559502f70510 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f69d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f58be0 0x559502f627e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3920126482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f59530 0x559502f62d20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 >> 172.21.15.31:0/3920126482 conn(0x559502f4e3e0 msgr2=0x559502f53060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4a86ff5700 0 client.26933 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 shutdown_connections 2023-07-29T16:20:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.889+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3920126482 wait complete. 2023-07-29T16:20:08.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.890+0000 7f4abcb32b80 20 client.26933 trim_cache size 0 max 16384 2023-07-29T16:20:08.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.907+0000 7f4abcb32b80 1 Processor -- start 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.907+0000 7f4abcb32b80 1 -- start start 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.907+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f5d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.907+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f59530 0x559502f69d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.907+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f5d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502f4b6e0 con 0x559502f59530 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502f604b0 con 0x559502f58be0 2023-07-29T16:20:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502f59b50 con 0x559502f5a2f0 2023-07-29T16:20:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f5d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f59530 0x559502f69d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f5d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33022/0 (socket says 172.21.15.31:33022) 2023-07-29T16:20:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/3123452346 learned_addr learned my addr 172.21.15.31:0/3123452346 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa359e700 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f5d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/3123452346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 msgr2=0x559502f5d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f5d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/3123452346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f59530 msgr2=0x559502f69d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f59530 0x559502f69d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/3123452346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559502cfe250 con 0x559502f58be0 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa259c700 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f59530 0x559502f69d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.908+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f5d230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4a94004380 tx=0x7f4a9401c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d6f9bc79f108bd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4a89ffb700 1 -- 172.21.15.31:0/3123452346 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94022040 con 0x559502f58be0 2023-07-29T16:20:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4a89ffb700 1 -- 172.21.15.31:0/3123452346 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a94003680 con 0x559502f58be0 2023-07-29T16:20:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4a89ffb700 1 -- 172.21.15.31:0/3123452346 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9401d6a0 con 0x559502f58be0 2023-07-29T16:20:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3123452346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 msgr2=0x559502f5d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f5d230 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f4a94004380 tx=0x7f4a9401c000 comp rx=0 tx=0).stop 2023-07-29T16:20:08.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3123452346 shutdown_connections 2023-07-29T16:20:08.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f5a2f0 0x559502f5d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f5d230 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/3123452346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f59530 0x559502f69d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3123452346 >> 172.21.15.31:0/3123452346 conn(0x559502f4e3e0 msgr2=0x559502f70110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:08.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3123452346 shutdown_connections 2023-07-29T16:20:08.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.909+0000 7f4abcb32b80 1 -- 172.21.15.31:0/3123452346 wait complete. 2023-07-29T16:20:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 -- start start 2023-07-29T16:20:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 Processor -- start 2023-07-29T16:20:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f69500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f69a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f5a2f0 0x559502f65fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502f664e0 con 0x559502f5a2f0 2023-07-29T16:20:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502f66650 con 0x559502f58be0 2023-07-29T16:20:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.910+0000 7f4abcb32b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559502f66810 con 0x559502f59530 2023-07-29T16:20:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4abcb32b80 10 client.0 _ll_register_callbacks cb 0x559502e4f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f69500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f69500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33028/0 (socket says 172.21.15.31:33028) 2023-07-29T16:20:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/1235813200 learned_addr learned my addr 172.21.15.31:0/1235813200 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa359e700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f5a2f0 0x559502f65fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa259c700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f69a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/1235813200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 msgr2=0x559502f69a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f69a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/1235813200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f5a2f0 msgr2=0x559502f65fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f5a2f0 0x559502f65fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 -- 172.21.15.31:0/1235813200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559502cc0c60 con 0x559502f58be0 2023-07-29T16:20:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa359e700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f5a2f0 0x559502f65fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4aa2d9d700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f69500 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4a94004010 tx=0x7f4a94004450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84c22c13bcb10bdb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.911+0000 7f4a86ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4a86ff5700 1 -- 172.21.15.31:0/1235813200 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a940039f0 con 0x559502f58be0 2023-07-29T16:20:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4a86ff5700 1 -- 172.21.15.31:0/1235813200 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a94022050 con 0x559502f58be0 2023-07-29T16:20:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a9801e040 con 0x559502f58be0 2023-07-29T16:20:08.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4a86ff5700 1 -- 172.21.15.31:0/1235813200 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a9402a0f0 con 0x559502f58be0 2023-07-29T16:20:08.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4abcb32b80 10 client.26939 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:08.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f4a940040f0 con 0x559502f58be0 2023-07-29T16:20:08.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.912+0000 7f4a86ff5700 1 -- 172.21.15.31:0/1235813200 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a9801e040 con 0x559502f58be0 2023-07-29T16:20:08.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4a86ff5700 1 -- 172.21.15.31:0/1235813200 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f4a9401d440 con 0x559502f58be0 2023-07-29T16:20:08.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4abcb32b80 10 client.26939 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:08.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4abcb32b80 10 client.26939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:08.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4abcb32b80 20 client.26939 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502fc1170 0x559502fc3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4abcb32b80 4 client.26939 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:20:08.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559502ea2d40 con 0x559502fc1170 2023-07-29T16:20:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4aa259c700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502fc1170 0x559502fc3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4aa259c700 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502fc1170 0x559502fc3580 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d79686325837726 server_cookie=1af97b41f630c466 in_seq=0 out_seq=0 2023-07-29T16:20:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.913+0000 7f4a86ff5700 10 client.26939 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:08.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4a86ff5700 1 -- 172.21.15.31:0/1235813200 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559502ea2d40 con 0x559502fc1170 2023-07-29T16:20:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4a86ff5700 10 client.26939 handle_command_reply: tid=0 2023-07-29T16:20:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 client.26939 shutdown 2023-07-29T16:20:08.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 msgr2=0x559502f69500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f69500 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4a94004010 tx=0x7f4a94004450 comp rx=0 tx=0).stop 2023-07-29T16:20:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 shutdown_connections 2023-07-29T16:20:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559502fc1170 0x559502fc3580 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559502f5a2f0 0x559502f65fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559502f58be0 0x559502f69500 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 --2- 172.21.15.31:0/1235813200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559502f59530 0x559502f69a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4a86ff5700 0 client.26939 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.915+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 >> 172.21.15.31:0/1235813200 conn(0x559502f4e3e0 msgr2=0x559502f5d230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 shutdown_connections 2023-07-29T16:20:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4abcb32b80 1 -- 172.21.15.31:0/1235813200 wait complete. 2023-07-29T16:20:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4abcb32b80 20 client.26939 trim_cache size 0 max 16384 2023-07-29T16:20:08.931 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:20:08.932 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:20:08.932 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:20:08.932 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:20:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a8c0a0790 msgr2=0x7f4a8c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a8c0a0790 0x7f4a8c0a2c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4a94011550 tx=0x7f4a94009f90 comp rx=0 tx=0).stop 2023-07-29T16:20:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 msgr2=0x7f4aa419f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 0x7f4aa419f860 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c007ee0 tx=0x7f4a9c00d3b0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 shutdown_connections 2023-07-29T16:20:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a8c0a0790 0x7f4a8c0a2c40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4aa4105fe0 0x7f4aa419f860 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4aa4100da0 0x7f4aa41982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 --2- 172.21.15.31:0/421595890 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa41036c0 0x7f4aa41987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 >> 172.21.15.31:0/421595890 conn(0x7f4aa40fa840 msgr2=0x7f4aa40ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.916+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 shutdown_connections 2023-07-29T16:20:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:08.917+0000 7f4aa91ff700 1 -- 172.21.15.31:0/421595890 wait complete. 2023-07-29T16:20:08.937 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:20:08.938 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:20:09.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/1771342472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101ec0 msgr2=0x7fdb58108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/1771342472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101ec0 0x7fdb58108780 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdb5000b0a0 tx=0x7fdb5000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/1771342472 shutdown_connections 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/1771342472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101ec0 0x7fdb58108780 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/1771342472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb58100340 0x7fdb58100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/1771342472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101530 0x7fdb58101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/1771342472 >> 172.21.15.31:0/1771342472 conn(0x7fdb580fb920 msgr2=0x7fdb580fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/1771342472 shutdown_connections 2023-07-29T16:20:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.183+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/1771342472 wait complete. 2023-07-29T16:20:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 Processor -- start 2023-07-29T16:20:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 -- start start 2023-07-29T16:20:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb58100340 0x7fdb58198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 0x7fdb58198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101ec0 0x7fdb5819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5819fec0 con 0x7fdb58100340 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb581a0000 con 0x7fdb58101530 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.184+0000 7fdb5e1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb581a0190 con 0x7fdb58101ec0 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 0x7fdb58198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 0x7fdb58198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33042/0 (socket says 172.21.15.31:33042) 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 -- 172.21.15.31:0/2016974181 learned_addr learned my addr 172.21.15.31:0/2016974181 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb57fff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101ec0 0x7fdb5819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb577fe700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb58100340 0x7fdb58198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 -- 172.21.15.31:0/2016974181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101ec0 msgr2=0x7fdb5819f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101ec0 0x7fdb5819f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 -- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb58100340 msgr2=0x7fdb58198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb58100340 0x7fdb58198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 -- 172.21.15.31:0/2016974181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb50009d00 con 0x7fdb58101530 2023-07-29T16:20:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb57fff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101ec0 0x7fdb5819f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.185+0000 7fdb56ffd700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 0x7fdb58198880 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00d980 tx=0x7fdb4c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.186+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c004220 con 0x7fdb58101530 2023-07-29T16:20:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.186+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb581a0470 con 0x7fdb58101530 2023-07-29T16:20:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.186+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb4c007640 con 0x7fdb58101530 2023-07-29T16:20:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.186+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c005690 con 0x7fdb58101530 2023-07-29T16:20:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.186+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb581a09d0 con 0x7fdb58101530 2023-07-29T16:20:09.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.187+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5810bb60 con 0x7fdb58101530 2023-07-29T16:20:09.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.189+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdb4c004750 con 0x7fdb58101530 2023-07-29T16:20:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.190+0000 7fdb54ff9700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb400a07e0 0x7fdb400a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.191+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdb4c0cbd40 con 0x7fdb58101530 2023-07-29T16:20:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.191+0000 7fdb577fe700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb400a07e0 0x7fdb400a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.191+0000 7fdb577fe700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb400a07e0 0x7fdb400a2c90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdb58101390 tx=0x7fdb48007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.194+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdb4c095720 con 0x7fdb58101530 2023-07-29T16:20:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.487+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb581a1010 con 0x7fdb58101530 2023-07-29T16:20:09.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.488+0000 7fdb54ff9700 1 -- 172.21.15.31:0/2016974181 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10777 (secure 0 0 0) 0x7fdb4c016030 con 0x7fdb58101530 2023-07-29T16:20:09.488 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:20:09.488 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:18:33.432992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:20:09.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.490+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb400a07e0 msgr2=0x7fdb400a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.490+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb400a07e0 0x7fdb400a2c90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdb58101390 tx=0x7fdb48007400 comp rx=0 tx=0).stop 2023-07-29T16:20:09.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 msgr2=0x7fdb58198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 0x7fdb58198880 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00d980 tx=0x7fdb4c00dc90 comp rx=0 tx=0).stop 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 shutdown_connections 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdb400a07e0 0x7fdb400a2c90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdb58101ec0 0x7fdb5819f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb58100340 0x7fdb58198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 --2- 172.21.15.31:0/2016974181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdb58101530 0x7fdb58198880 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 >> 172.21.15.31:0/2016974181 conn(0x7fdb580fb920 msgr2=0x7fdb58106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 shutdown_connections 2023-07-29T16:20:09.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.491+0000 7fdb5e1ff700 1 -- 172.21.15.31:0/2016974181 wait complete. 2023-07-29T16:20:09.492 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2023-07-29T16:20:09.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:20:09.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 -- 172.21.15.31:0/2283749372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20100330 msgr2=0x7f3b20100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 --2- 172.21.15.31:0/2283749372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20100330 0x7f3b20100740 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009990 tx=0x7f3b14009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 -- 172.21.15.31:0/2283749372 shutdown_connections 2023-07-29T16:20:09.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 --2- 172.21.15.31:0/2283749372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20101eb0 0x7f3b20108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 --2- 172.21.15.31:0/2283749372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20100330 0x7f3b20100740 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 --2- 172.21.15.31:0/2283749372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 0x7f3b20101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 -- 172.21.15.31:0/2283749372 >> 172.21.15.31:0/2283749372 conn(0x7f3b200fb970 msgr2=0x7f3b200fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.755+0000 7f3b26fff700 1 -- 172.21.15.31:0/2283749372 shutdown_connections 2023-07-29T16:20:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.756+0000 7f3b26fff700 1 -- 172.21.15.31:0/2283749372 wait complete. 2023-07-29T16:20:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.756+0000 7f3b26fff700 1 Processor -- start 2023-07-29T16:20:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 -- start start 2023-07-29T16:20:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20100330 0x7f3b20196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 0x7f3b201965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20101eb0 0x7f3b2019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2019dc10 con 0x7f3b20101eb0 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2019dd50 con 0x7f3b20101520 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b26fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2019dee0 con 0x7f3b20100330 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b2559c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20101eb0 0x7f3b2019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 0x7f3b201965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b24d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20100330 0x7f3b20196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 0x7f3b201965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33052/0 (socket says 172.21.15.31:33052) 2023-07-29T16:20:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.757+0000 7f3b1ffff700 1 -- 172.21.15.31:0/1869830774 learned_addr learned my addr 172.21.15.31:0/1869830774 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b2559c700 1 -- 172.21.15.31:0/1869830774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20100330 msgr2=0x7f3b20196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b2559c700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20100330 0x7f3b20196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b2559c700 1 -- 172.21.15.31:0/1869830774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 msgr2=0x7f3b201965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b2559c700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 0x7f3b201965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b2559c700 1 -- 172.21.15.31:0/1869830774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0c009520 con 0x7f3b20101eb0 2023-07-29T16:20:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20100330 0x7f3b20196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b2559c700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20101eb0 0x7f3b2019d640 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800efc0 tx=0x7f3b1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b1dffb700 1 -- 172.21.15.31:0/1869830774 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18018070 con 0x7f3b20101eb0 2023-07-29T16:20:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b140095f0 con 0x7f3b20101eb0 2023-07-29T16:20:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b1dffb700 1 -- 172.21.15.31:0/1869830774 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b1800f040 con 0x7f3b20101eb0 2023-07-29T16:20:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b1dffb700 1 -- 172.21.15.31:0/1869830774 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1800cd40 con 0x7f3b20101eb0 2023-07-29T16:20:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.758+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2019e4e0 con 0x7f3b20101eb0 2023-07-29T16:20:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.763+0000 7f3b1dffb700 1 -- 172.21.15.31:0/1869830774 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3b18003a00 con 0x7f3b20101eb0 2023-07-29T16:20:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.764+0000 7f3b3a864b80 1 Processor -- start 2023-07-29T16:20:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.764+0000 7f3b3a864b80 1 -- start start 2023-07-29T16:20:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dcdf5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdf6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dcdf7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcd4da10 con 0x5616dcdf6530 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcd4b3d0 con 0x5616dcdf5be0 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcd46cc0 con 0x5616dcdf72f0 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b1dffb700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b100a0790 0x7f3b100a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b1dffb700 1 -- 172.21.15.31:0/1869830774 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b180cb6c0 con 0x7f3b20101eb0 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b24d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdf6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dcdf5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b24d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdf6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32898/0 (socket says 172.21.15.31:32898) 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b24d9b700 1 -- 172.21.15.31:0/88698541 learned_addr learned my addr 172.21.15.31:0/88698541 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dcdf5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33062/0 (socket says 172.21.15.31:33062) 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dcdf7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.765+0000 7f3b2559c700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b100a0790 0x7f3b100a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b1ffff700 1 -- 172.21.15.31:0/88698541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 msgr2=0x5616dcdf7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dcdf7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b1ffff700 1 -- 172.21.15.31:0/88698541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 msgr2=0x5616dcdf6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdf6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b1ffff700 1 -- 172.21.15.31:0/88698541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616dcb74de0 con 0x5616dcdf5be0 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdf6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dcdf5ff0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c013860 tx=0x7f3b0c009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a94885e30ba30ec server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b2559c700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b100a0790 0x7f3b100a2c40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009320 tx=0x7f3b18004dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b0affd700 1 -- 172.21.15.31:0/88698541 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c018070 con 0x5616dcdf5be0 2023-07-29T16:20:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b0affd700 1 -- 172.21.15.31:0/88698541 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b0c0044f0 con 0x5616dcdf5be0 2023-07-29T16:20:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.766+0000 7f3b0affd700 1 -- 172.21.15.31:0/88698541 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c00f680 con 0x5616dcdf5be0 2023-07-29T16:20:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 -- 172.21.15.31:0/88698541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 msgr2=0x5616dcdf5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dcdf5ff0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c013860 tx=0x7f3b0c009900 comp rx=0 tx=0).stop 2023-07-29T16:20:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 -- 172.21.15.31:0/88698541 shutdown_connections 2023-07-29T16:20:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dcdf7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dcdf5ff0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/88698541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdf6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 -- 172.21.15.31:0/88698541 >> 172.21.15.31:0/88698541 conn(0x5616dcdeb3e0 msgr2=0x5616dcdf0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 -- 172.21.15.31:0/88698541 shutdown_connections 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.767+0000 7f3b3a864b80 1 -- 172.21.15.31:0/88698541 wait complete. 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 -- start start 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 Processor -- start 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dcdff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf72f0 0x5616dce06db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcce3510 con 0x5616dcdf6530 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcce31b0 con 0x5616dcdf72f0 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcbca920 con 0x5616dcdf5be0 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.768+0000 7f3b3a864b80 10 client.0 _ll_register_callbacks cb 0x5616dccec620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dcdff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dcdff830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32902/0 (socket says 172.21.15.31:32902) 2023-07-29T16:20:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b1ffff700 1 -- 172.21.15.31:0/2841953350 learned_addr learned my addr 172.21.15.31:0/2841953350 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:09.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf72f0 0x5616dce06db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 -- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 msgr2=0x5616dcdff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dcdff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 -- 172.21.15.31:0/2841953350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf72f0 msgr2=0x5616dce06db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf72f0 0x5616dce06db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 -- 172.21.15.31:0/2841953350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b201090a0 con 0x5616dcdf6530 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdffd70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x5616dcdf6940 tx=0x7f3b14005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d78d9b5fc57c28d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.769+0000 7f3afffff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3afffff700 1 -- 172.21.15.31:0/2841953350 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1401d070 con 0x5616dcdf6530 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3afffff700 1 -- 172.21.15.31:0/2841953350 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b14004910 con 0x5616dcdf6530 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616dcd71d10 con 0x5616dcdf6530 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3afffff700 1 -- 172.21.15.31:0/2841953350 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b14005700 con 0x5616dcdf6530 2023-07-29T16:20:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3b3a864b80 10 client.16950 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f3b0c013500 con 0x5616dcdf6530 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3afffff700 1 -- 172.21.15.31:0/2841953350 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616dcd71d10 con 0x5616dcdf6530 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3afffff700 1 -- 172.21.15.31:0/2841953350 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f3b1402e070 con 0x5616dcdf6530 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3b3a864b80 10 client.16950 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.770+0000 7f3b3a864b80 10 client.16950 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3b3a864b80 20 client.16950 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:09.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce0b5c0 0x5616dce0d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3b3a864b80 4 client.16950 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616dcdc4d70 con 0x5616dce0b5c0 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce0b5c0 0x5616dce0d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce0b5c0 0x5616dce0d9a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c3f12a508c60c63f server_cookie=9b80ceda5d1805b6 in_seq=0 out_seq=0 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.771+0000 7f3afffff700 10 client.16950 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.772+0000 7f3afffff700 1 -- 172.21.15.31:0/2841953350 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5616dcdc4d70 con 0x5616dce0b5c0 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.772+0000 7f3afffff700 10 client.16950 handle_command_reply: tid=0 2023-07-29T16:20:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.772+0000 7f3b3a864b80 1 client.16950 shutdown 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.772+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 msgr2=0x5616dcdffd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.772+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdffd70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x5616dcdf6940 tx=0x7f3b14005270 comp rx=0 tx=0).stop 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 shutdown_connections 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce0b5c0 0x5616dce0d9a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf72f0 0x5616dce06db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dcdff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3afffff700 0 client.16950 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/2841953350 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dcdffd70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 >> 172.21.15.31:0/2841953350 conn(0x5616dcdeb3e0 msgr2=0x5616dcdf0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 shutdown_connections 2023-07-29T16:20:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 1 -- 172.21.15.31:0/2841953350 wait complete. 2023-07-29T16:20:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.773+0000 7f3b3a864b80 20 client.16950 trim_cache size 0 max 16384 2023-07-29T16:20:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.794+0000 7f3b3a864b80 1 Processor -- start 2023-07-29T16:20:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.794+0000 7f3b3a864b80 1 -- start start 2023-07-29T16:20:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dce0d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf6530 0x5616dce06d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 0x5616dcdf76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcdfd4e0 con 0x5616dcdf72f0 2023-07-29T16:20:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dce0b360 con 0x5616dcdf6530 2023-07-29T16:20:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dcdf2380 con 0x5616dcdf5be0 2023-07-29T16:20:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dce0d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 0x5616dcdf76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 0x5616dcdf76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32904/0 (socket says 172.21.15.31:32904) 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.795+0000 7f3b1ffff700 1 -- 172.21.15.31:0/1731059252 learned_addr learned my addr 172.21.15.31:0/1731059252 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.796+0000 7f3b1ffff700 1 -- 172.21.15.31:0/1731059252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 msgr2=0x5616dce0d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.796+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dce0d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.796+0000 7f3b1ffff700 1 -- 172.21.15.31:0/1731059252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf6530 msgr2=0x5616dce06d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.796+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf6530 0x5616dce06d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.796+0000 7f3b1ffff700 1 -- 172.21.15.31:0/1731059252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616dcb5dc60 con 0x5616dcdf72f0 2023-07-29T16:20:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.797+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 0x5616dcdf76c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c013e50 tx=0x7f3b0c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c269380a0a2674a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.798+0000 7f3b0b7fe700 1 -- 172.21.15.31:0/1731059252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c018070 con 0x5616dcdf72f0 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.798+0000 7f3b0b7fe700 1 -- 172.21.15.31:0/1731059252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b0c030420 con 0x5616dcdf72f0 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.798+0000 7f3b0b7fe700 1 -- 172.21.15.31:0/1731059252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c026040 con 0x5616dcdf72f0 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 -- 172.21.15.31:0/1731059252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 msgr2=0x5616dcdf76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 0x5616dcdf76c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c013e50 tx=0x7f3b0c009260 comp rx=0 tx=0).stop 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 -- 172.21.15.31:0/1731059252 shutdown_connections 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf72f0 0x5616dcdf76c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf5be0 0x5616dce0d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/1731059252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf6530 0x5616dce06d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 -- 172.21.15.31:0/1731059252 >> 172.21.15.31:0/1731059252 conn(0x5616dcdeb3e0 msgr2=0x5616dcdf0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 -- 172.21.15.31:0/1731059252 shutdown_connections 2023-07-29T16:20:09.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.799+0000 7f3b3a864b80 1 -- 172.21.15.31:0/1731059252 wait complete. 2023-07-29T16:20:09.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.800+0000 7f3b3a864b80 1 -- start start 2023-07-29T16:20:09.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.800+0000 7f3b3a864b80 1 Processor -- start 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dce047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dce04d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dce05d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dce06260 con 0x5616dcdf6530 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dce063d0 con 0x5616dcdf5be0 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616dce06590 con 0x5616dcdf72f0 2023-07-29T16:20:09.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b3a864b80 10 client.0 _ll_register_callbacks cb 0x5616dccec620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b24d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dce04d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dce047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dce047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33090/0 (socket says 172.21.15.31:33090) 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b24d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dce04d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32920/0 (socket says 172.21.15.31:32920) 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 -- 172.21.15.31:0/3247152875 learned_addr learned my addr 172.21.15.31:0/3247152875 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dce05d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 -- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 msgr2=0x5616dce05d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dce05d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 -- 172.21.15.31:0/3247152875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 msgr2=0x5616dce04d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.801+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dce04d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3b1ffff700 1 -- 172.21.15.31:0/3247152875 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0c029040 con 0x5616dcdf5be0 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dce04d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3b1ffff700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dce047d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009c00 tx=0x7f3b0c0137f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c6fab8ffd13ff19 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3afffff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:09.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3afffff700 1 -- 172.21.15.31:0/3247152875 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c018070 con 0x5616dcdf5be0 2023-07-29T16:20:09.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3afffff700 1 -- 172.21.15.31:0/3247152875 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b0c003e10 con 0x5616dcdf5be0 2023-07-29T16:20:09.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.802+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b0c0044f0 con 0x5616dcdf5be0 2023-07-29T16:20:09.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3afffff700 1 -- 172.21.15.31:0/3247152875 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b0c026040 con 0x5616dcdf5be0 2023-07-29T16:20:09.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 10 client.26960 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:09.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5616dcb9b250 con 0x5616dcdf5be0 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3afffff700 1 -- 172.21.15.31:0/3247152875 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b0c0044f0 con 0x5616dcdf5be0 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3afffff700 1 -- 172.21.15.31:0/3247152875 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f3b0c00f5a0 con 0x5616dcdf5be0 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 10 client.26960 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 10 client.26960 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 20 client.26960 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce66030 0x5616dce68440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:09.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 4 client.26960 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.803+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5616dcd3fd40 con 0x5616dce66030 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce66030 0x5616dce68440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b24d9b700 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce66030 0x5616dce68440 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=980ecbf686a88045 server_cookie=aea092dca594098d in_seq=0 out_seq=0 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3afffff700 10 client.26960 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3afffff700 1 -- 172.21.15.31:0/3247152875 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5616dcd3fd40 con 0x5616dce66030 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3afffff700 10 client.26960 handle_command_reply: tid=0 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b3a864b80 1 client.26960 shutdown 2023-07-29T16:20:09.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 msgr2=0x5616dce047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dce047d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c009c00 tx=0x7f3b0c0137f0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 shutdown_connections 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.804+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616dcdf72f0 0x5616dce05d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616dce66030 0x5616dce68440 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616dcdf5be0 0x5616dce047d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3afffff700 0 client.26960 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:09.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 1 --2- 172.21.15.31:0/3247152875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616dcdf6530 0x5616dce04d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 >> 172.21.15.31:0/3247152875 conn(0x5616dcdeb3e0 msgr2=0x5616dcdfe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 shutdown_connections 2023-07-29T16:20:09.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 1 -- 172.21.15.31:0/3247152875 wait complete. 2023-07-29T16:20:09.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.805+0000 7f3b3a864b80 20 client.26960 trim_cache size 0 max 16384 2023-07-29T16:20:09.810 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:20:09.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b100a0790 msgr2=0x7f3b100a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b100a0790 0x7f3b100a2c40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009320 tx=0x7f3b18004dc0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20101eb0 msgr2=0x7f3b2019d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:09.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20101eb0 0x7f3b2019d640 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800efc0 tx=0x7f3b1800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 shutdown_connections 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b100a0790 0x7f3b100a2c40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b20101eb0 0x7f3b2019d640 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b20100330 0x7f3b20196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 --2- 172.21.15.31:0/1869830774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b20101520 0x7f3b201965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.806+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 >> 172.21.15.31:0/1869830774 conn(0x7f3b200fb970 msgr2=0x7f3b20104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.807+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 shutdown_connections 2023-07-29T16:20:09.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:09.807+0000 7f3b26fff700 1 -- 172.21.15.31:0/1869830774 wait complete. 2023-07-29T16:20:09.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:20:09.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 193983 9.30 MB/sec warmup 114 sec latency 32.478 ms 2023-07-29T16:20:10.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 -- 172.21.15.31:0/887440547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101520 msgr2=0x7f0530101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 --2- 172.21.15.31:0/887440547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101520 0x7f0530101970 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0524009990 tx=0x7f0524009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 -- 172.21.15.31:0/887440547 shutdown_connections 2023-07-29T16:20:10.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 --2- 172.21.15.31:0/887440547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0530101eb0 0x7f0530108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 --2- 172.21.15.31:0/887440547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 --2- 172.21.15.31:0/887440547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101520 0x7f0530101970 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 -- 172.21.15.31:0/887440547 >> 172.21.15.31:0/887440547 conn(0x7f05300fb970 msgr2=0x7f05300fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.048+0000 7f05369ff700 1 -- 172.21.15.31:0/887440547 shutdown_connections 2023-07-29T16:20:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.049+0000 7f05369ff700 1 -- 172.21.15.31:0/887440547 wait complete. 2023-07-29T16:20:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 Processor -- start 2023-07-29T16:20:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 -- start start 2023-07-29T16:20:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0530101520 0x7f0530073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101eb0 0x7f0530073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530073e20 con 0x7f0530101520 2023-07-29T16:20:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0530073f90 con 0x7f0530100330 2023-07-29T16:20:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f05369ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05301a0080 con 0x7f0530101eb0 2023-07-29T16:20:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f052ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f052ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56948/0 (socket says 172.21.15.31:56948) 2023-07-29T16:20:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f052ffff700 1 -- 172.21.15.31:0/1956872142 learned_addr learned my addr 172.21.15.31:0/1956872142 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f0534f9c700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101eb0 0x7f0530073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.050+0000 7f052f7fe700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0530101520 0x7f0530073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052ffff700 1 -- 172.21.15.31:0/1956872142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101eb0 msgr2=0x7f0530073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052ffff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101eb0 0x7f0530073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052ffff700 1 -- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0530101520 msgr2=0x7f0530073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052ffff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0530101520 0x7f0530073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052ffff700 1 -- 172.21.15.31:0/1956872142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05240095f0 con 0x7f0530100330 2023-07-29T16:20:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052ffff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530074b00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f052000e970 tx=0x7f052000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052d7fa700 1 -- 172.21.15.31:0/1956872142 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520009790 con 0x7f0530100330 2023-07-29T16:20:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052d7fa700 1 -- 172.21.15.31:0/1956872142 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f052000c7e0 con 0x7f0530100330 2023-07-29T16:20:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05301a0360 con 0x7f0530100330 2023-07-29T16:20:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.051+0000 7f052d7fa700 1 -- 172.21.15.31:0/1956872142 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520003c70 con 0x7f0530100330 2023-07-29T16:20:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.052+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05301a0870 con 0x7f0530100330 2023-07-29T16:20:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.055+0000 7f052d7fa700 1 -- 172.21.15.31:0/1956872142 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f052000cc50 con 0x7f0530100330 2023-07-29T16:20:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 Processor -- start 2023-07-29T16:20:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 -- start start 2023-07-29T16:20:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459cf9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459cfad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459cfb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459c51a10 con 0x560459cfb2f0 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459c4f3d0 con 0x560459cf9be0 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f054a317b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459c4acc0 con 0x560459cfa530 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.056+0000 7f052d7fa700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05180a0810 0x7f05180a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459cfad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f0534f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459cf9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459cfad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42148/0 (socket says 172.21.15.31:42148) 2023-07-29T16:20:10.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 -- 172.21.15.31:0/2910534264 learned_addr learned my addr 172.21.15.31:0/2910534264 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052f7fe700 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459cfb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052d7fa700 1 -- 172.21.15.31:0/1956872142 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0520014070 con 0x7f0530100330 2023-07-29T16:20:10.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f0534f9c700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05180a0810 0x7f05180a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 -- 172.21.15.31:0/2910534264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 msgr2=0x560459cf9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459cf9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 -- 172.21.15.31:0/2910534264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 msgr2=0x560459cfb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459cfb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 -- 172.21.15.31:0/2910534264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560459a78de0 con 0x560459cfa530 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.057+0000 7f052ffff700 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459cfad60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f052006c860 tx=0x7f052006b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e806ec3a9cec202 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.058+0000 7f05167fc700 1 -- 172.21.15.31:0/2910534264 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520003dd0 con 0x560459cfa530 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.058+0000 7f05167fc700 1 -- 172.21.15.31:0/2910534264 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05200993d0 con 0x560459cfa530 2023-07-29T16:20:10.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.058+0000 7f0534f9c700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05180a0810 0x7f05180a2cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x560459cf9980 tx=0x7f0528009100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.058+0000 7f05167fc700 1 -- 172.21.15.31:0/2910534264 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520097890 con 0x560459cfa530 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.058+0000 7f054a317b80 1 -- 172.21.15.31:0/2910534264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 msgr2=0x560459cfad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.058+0000 7f054a317b80 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459cfad60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f052006c860 tx=0x7f052006b7e0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 -- 172.21.15.31:0/2910534264 shutdown_connections 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459cfb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459cf9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 --2- 172.21.15.31:0/2910534264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459cfad60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 -- 172.21.15.31:0/2910534264 >> 172.21.15.31:0/2910534264 conn(0x560459cef3e0 msgr2=0x560459cf4990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 -- 172.21.15.31:0/2910534264 shutdown_connections 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 -- 172.21.15.31:0/2910534264 wait complete. 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 -- start start 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.059+0000 7f054a317b80 1 Processor -- start 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d03830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459d03d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d0adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459be7510 con 0x560459cfb2f0 2023-07-29T16:20:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459be71b0 con 0x560459cf9be0 2023-07-29T16:20:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459ace920 con 0x560459cfa530 2023-07-29T16:20:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f054a317b80 10 client.0 _ll_register_callbacks cb 0x560459bf0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f052ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d03830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f052ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d03830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56978/0 (socket says 172.21.15.31:56978) 2023-07-29T16:20:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f052ffff700 1 -- 172.21.15.31:0/91456896 learned_addr learned my addr 172.21.15.31:0/91456896 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f052f7fe700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459d03d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.060+0000 7f0534f9c700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d0adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f052ffff700 1 -- 172.21.15.31:0/91456896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 msgr2=0x560459d03d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f052ffff700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459d03d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f052ffff700 1 -- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 msgr2=0x560459d0adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f052ffff700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d0adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f052ffff700 1 -- 172.21.15.31:0/91456896 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05301a0870 con 0x560459cf9be0 2023-07-29T16:20:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f052ffff700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d03830 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f05200cbe20 tx=0x7f05200992d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3737438b3dc2cdd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f050f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f050f7fe700 1 -- 172.21.15.31:0/91456896 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520098ac0 con 0x560459cf9be0 2023-07-29T16:20:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f050f7fe700 1 -- 172.21.15.31:0/91456896 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f05200a1820 con 0x560459cf9be0 2023-07-29T16:20:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560459c75d10 con 0x560459cf9be0 2023-07-29T16:20:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f050f7fe700 1 -- 172.21.15.31:0/91456896 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05200a1390 con 0x560459cf9be0 2023-07-29T16:20:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f054a317b80 10 client.26972 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.061+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560459a61c60 con 0x560459cf9be0 2023-07-29T16:20:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f050f7fe700 1 -- 172.21.15.31:0/91456896 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560459c75d10 con 0x560459cf9be0 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f050f7fe700 1 -- 172.21.15.31:0/91456896 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f0520098330 con 0x560459cf9be0 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f054a317b80 10 client.26972 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f054a317b80 10 client.26972 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f054a317b80 20 client.26972 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f054a317b80 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d0f130 0x560459d11510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f054a317b80 4 client.26972 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560459cc8d70 con 0x560459d0f130 2023-07-29T16:20:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.062+0000 7f052f7fe700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d0f130 0x560459d11510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.063+0000 7f052f7fe700 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d0f130 0x560459d11510 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b9b664a6043cd5ab server_cookie=3e5424c79325bd0c in_seq=0 out_seq=0 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.063+0000 7f050f7fe700 10 client.26972 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.063+0000 7f050f7fe700 1 -- 172.21.15.31:0/91456896 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560459cc8d70 con 0x560459d0f130 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.063+0000 7f050f7fe700 10 client.26972 handle_command_reply: tid=0 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.063+0000 7f054a317b80 1 client.26972 shutdown 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 msgr2=0x560459d03830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d03830 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f05200cbe20 tx=0x7f05200992d0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 shutdown_connections 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d0f130 0x560459d11510 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d0adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d03830 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f050f7fe700 0 client.26972 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 --2- 172.21.15.31:0/91456896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfa530 0x560459d03d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 >> 172.21.15.31:0/91456896 conn(0x560459cef3e0 msgr2=0x560459cf4060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 shutdown_connections 2023-07-29T16:20:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 1 -- 172.21.15.31:0/91456896 wait complete. 2023-07-29T16:20:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.064+0000 7f054a317b80 20 client.26972 trim_cache size 0 max 16384 2023-07-29T16:20:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.078+0000 7f054a317b80 1 Processor -- start 2023-07-29T16:20:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.078+0000 7f054a317b80 1 -- start start 2023-07-29T16:20:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f054a317b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f054a317b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfa530 0x560459cfb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f054a317b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfb2f0 0x560459cf5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f054a317b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459cfab80 con 0x560459cfa530 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f054a317b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459cf5bd0 con 0x560459cf9be0 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f054a317b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459d160a0 con 0x560459cfb2f0 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d08d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56986/0 (socket says 172.21.15.31:56986) 2023-07-29T16:20:10.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 -- 172.21.15.31:0/1221037184 learned_addr learned my addr 172.21.15.31:0/1221037184 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f0534f9c700 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfb2f0 0x560459cf5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052f7fe700 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfa530 0x560459cfb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 -- 172.21.15.31:0/1221037184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfb2f0 msgr2=0x560459cf5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfb2f0 0x560459cf5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 -- 172.21.15.31:0/1221037184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfa530 msgr2=0x560459cfb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfa530 0x560459cfb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.079+0000 7f052ffff700 1 -- 172.21.15.31:0/1221037184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560459a9f250 con 0x560459cf9be0 2023-07-29T16:20:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.080+0000 7f052ffff700 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d08d60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f05200958d0 tx=0x7f05200999f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f030af22595dba server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.080+0000 7f0516ffd700 1 -- 172.21.15.31:0/1221037184 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200a1820 con 0x560459cf9be0 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.080+0000 7f0516ffd700 1 -- 172.21.15.31:0/1221037184 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0520098330 con 0x560459cf9be0 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.080+0000 7f0516ffd700 1 -- 172.21.15.31:0/1221037184 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520098b20 con 0x560459cf9be0 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 -- 172.21.15.31:0/1221037184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 msgr2=0x560459d08d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d08d60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f05200958d0 tx=0x7f05200999f0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 -- 172.21.15.31:0/1221037184 shutdown_connections 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cfb2f0 0x560459cf5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cf9be0 0x560459d08d60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 --2- 172.21.15.31:0/1221037184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfa530 0x560459cfb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 -- 172.21.15.31:0/1221037184 >> 172.21.15.31:0/1221037184 conn(0x560459cef3e0 msgr2=0x560459d02510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 -- 172.21.15.31:0/1221037184 shutdown_connections 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 -- 172.21.15.31:0/1221037184 wait complete. 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 -- start start 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.081+0000 7f054a317b80 1 Processor -- start 2023-07-29T16:20:10.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 0x560459d08700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cfa530 0x560459d08c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d06d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459d073f0 con 0x560459cfb2f0 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459d07560 con 0x560459cfa530 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560459d07720 con 0x560459cf9be0 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f054a317b80 10 client.0 _ll_register_callbacks cb 0x560459bf0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 0x560459d08700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cfa530 0x560459d08c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 0x560459d08700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42178/0 (socket says 172.21.15.31:42178) 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 -- 172.21.15.31:0/1206070000 learned_addr learned my addr 172.21.15.31:0/1206070000 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f0534f9c700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d06d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 -- 172.21.15.31:0/1206070000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cfa530 msgr2=0x560459d08c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cfa530 0x560459d08c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 -- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 msgr2=0x560459d06d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d06d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052ffff700 1 -- 172.21.15.31:0/1206070000 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560459a61c60 con 0x560459cf9be0 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f0534f9c700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d06d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.082+0000 7f052f7fe700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cfa530 0x560459d08c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f052ffff700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 0x560459d08700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f05200a12f0 tx=0x7f05200c97f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=468e16d11ac0da62 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f050f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f050f7fe700 1 -- 172.21.15.31:0/1206070000 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200a1650 con 0x560459cf9be0 2023-07-29T16:20:10.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f050f7fe700 1 -- 172.21.15.31:0/1206070000 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f05200a1820 con 0x560459cf9be0 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05200993d0 con 0x560459cf9be0 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f050f7fe700 1 -- 172.21.15.31:0/1206070000 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05200984a0 con 0x560459cf9be0 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f054a317b80 10 client.26989 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560459a772b0 con 0x560459cf9be0 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.083+0000 7f050f7fe700 1 -- 172.21.15.31:0/1206070000 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f05200993d0 con 0x560459cf9be0 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f050f7fe700 1 -- 172.21.15.31:0/1206070000 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f05200c9200 con 0x560459cf9be0 2023-07-29T16:20:10.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f054a317b80 10 client.26989 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f054a317b80 10 client.26989 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:20:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f054a317b80 20 client.26989 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f054a317b80 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d61910 0x560459d63cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f054a317b80 4 client.26989 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:20:10.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560459c43d40 con 0x560459d61910 2023-07-29T16:20:10.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.084+0000 7f052f7fe700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d61910 0x560459d63cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f052f7fe700 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d61910 0x560459d63cf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ebceef2d0e5357d server_cookie=78d1c66bc52e6365 in_seq=0 out_seq=0 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f050f7fe700 10 client.26989 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f050f7fe700 1 -- 172.21.15.31:0/1206070000 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560459c43d40 con 0x560459d61910 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f050f7fe700 10 client.26989 handle_command_reply: tid=0 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 client.26989 shutdown 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 msgr2=0x560459d08700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 0x560459d08700 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f05200a12f0 tx=0x7f05200c97f0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 shutdown_connections 2023-07-29T16:20:10.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560459d61910 0x560459d63cf0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560459cfb2f0 0x560459d06d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560459cf9be0 0x560459d08700 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 --2- 172.21.15.31:0/1206070000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560459cfa530 0x560459d08c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f050f7fe700 0 client.26989 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 >> 172.21.15.31:0/1206070000 conn(0x560459cef3e0 msgr2=0x560459d016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 shutdown_connections 2023-07-29T16:20:10.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 1 -- 172.21.15.31:0/1206070000 wait complete. 2023-07-29T16:20:10.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.085+0000 7f054a317b80 20 client.26989 trim_cache size 0 max 16384 2023-07-29T16:20:10.094 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05180a0810 msgr2=0x7f05180a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05180a0810 0x7f05180a2cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x560459cf9980 tx=0x7f0528009100 comp rx=0 tx=0).stop 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 msgr2=0x7f0530074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530074b00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f052000e970 tx=0x7f052000ec80 comp rx=0 tx=0).stop 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 shutdown_connections 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05180a0810 0x7f05180a2cc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0530101eb0 0x7f0530073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0530100330 0x7f0530074b00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 --2- 172.21.15.31:0/1956872142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0530101520 0x7f0530073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 >> 172.21.15.31:0/1956872142 conn(0x7f05300fb970 msgr2=0x7f05300fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 shutdown_connections 2023-07-29T16:20:10.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.086+0000 7f05369ff700 1 -- 172.21.15.31:0/1956872142 wait complete. 2023-07-29T16:20:10.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:20:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 -- 172.21.15.31:0/768497113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc105fe0 msgr2=0x7fddbc108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 --2- 172.21.15.31:0/768497113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc105fe0 0x7fddbc108440 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fddb400b0a0 tx=0x7fddb400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 -- 172.21.15.31:0/768497113 shutdown_connections 2023-07-29T16:20:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 --2- 172.21.15.31:0/768497113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc105fe0 0x7fddbc108440 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 --2- 172.21.15.31:0/768497113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc100da0 0x7fddbc103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 --2- 172.21.15.31:0/768497113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc1036c0 0x7fddbc105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.365+0000 7fddc09ff700 1 -- 172.21.15.31:0/768497113 >> 172.21.15.31:0/768497113 conn(0x7fddbc0fa840 msgr2=0x7fddbc0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.366+0000 7fddc09ff700 1 -- 172.21.15.31:0/768497113 shutdown_connections 2023-07-29T16:20:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.366+0000 7fddc09ff700 1 -- 172.21.15.31:0/768497113 wait complete. 2023-07-29T16:20:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 Processor -- start 2023-07-29T16:20:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 -- start start 2023-07-29T16:20:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 0x7fddbc100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc1036c0 0x7fddbc0fed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc105fe0 0x7fddbc0ff290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0ffa30 con 0x7fddbc1036c0 2023-07-29T16:20:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc0ffba0 con 0x7fddbc100da0 2023-07-29T16:20:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddc09ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc1a01c0 con 0x7fddbc105fe0 2023-07-29T16:20:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddba59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 0x7fddbc100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc105fe0 0x7fddbc0ff290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc1036c0 0x7fddbc0fed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddba59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 0x7fddbc100710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57004/0 (socket says 172.21.15.31:57004) 2023-07-29T16:20:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddba59c700 1 -- 172.21.15.31:0/350270075 learned_addr learned my addr 172.21.15.31:0/350270075 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.367+0000 7fddbad9d700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc105fe0 0x7fddbc0ff290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42198/0 (socket says 172.21.15.31:42198) 2023-07-29T16:20:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddba59c700 1 -- 172.21.15.31:0/350270075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc105fe0 msgr2=0x7fddbc0ff290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddba59c700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc105fe0 0x7fddbc0ff290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddba59c700 1 -- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc1036c0 msgr2=0x7fddbc0fed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddba59c700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc1036c0 0x7fddbc0fed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddba59c700 1 -- 172.21.15.31:0/350270075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb4009d00 con 0x7fddbc100da0 2023-07-29T16:20:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddba59c700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 0x7fddbc100710 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fddac00e9a0 tx=0x7fddac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddab7fe700 1 -- 172.21.15.31:0/350270075 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac009790 con 0x7fddbc100da0 2023-07-29T16:20:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddab7fe700 1 -- 172.21.15.31:0/350270075 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddac010430 con 0x7fddbc100da0 2023-07-29T16:20:10.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc1a04a0 con 0x7fddbc100da0 2023-07-29T16:20:10.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddab7fe700 1 -- 172.21.15.31:0/350270075 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac018680 con 0x7fddbc100da0 2023-07-29T16:20:10.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.368+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc1a0a00 con 0x7fddbc100da0 2023-07-29T16:20:10.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.372+0000 7fddab7fe700 1 -- 172.21.15.31:0/350270075 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fddac0105a0 con 0x7fddbc100da0 2023-07-29T16:20:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.374+0000 7fddd435db80 1 Processor -- start 2023-07-29T16:20:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.374+0000 7fddd435db80 1 -- start start 2023-07-29T16:20:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.374+0000 7fddab7fe700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdda40a0810 0x7fdda40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddd435db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d23ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddd435db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d24530 0x55cee6d24d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddd435db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d252f0 0x55cee6d25740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddab7fe700 1 -- 172.21.15.31:0/350270075 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fddac014070 con 0x7fddbc100da0 2023-07-29T16:20:10.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddd435db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6c7ba10 con 0x55cee6d252f0 2023-07-29T16:20:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddd435db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6c793d0 con 0x55cee6d24530 2023-07-29T16:20:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddd435db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6c74cc0 con 0x55cee6d23be0 2023-07-29T16:20:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddbad9d700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdda40a0810 0x7fdda40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d24530 0x55cee6d24d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d23ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d24530 0x55cee6d24d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57012/0 (socket says 172.21.15.31:57012) 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d252f0 0x55cee6d25740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddb9d9b700 1 -- 172.21.15.31:0/112502845 learned_addr learned my addr 172.21.15.31:0/112502845 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d23ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42214/0 (socket says 172.21.15.31:42214) 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 -- 172.21.15.31:0/112502845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d24530 msgr2=0x55cee6d24d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d24530 0x55cee6d24d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 -- 172.21.15.31:0/112502845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d252f0 msgr2=0x55cee6d25740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d252f0 0x55cee6d25740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.375+0000 7fddba59c700 1 -- 172.21.15.31:0/112502845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cee6aa2de0 con 0x55cee6d23be0 2023-07-29T16:20:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.376+0000 7fddbad9d700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdda40a0810 0x7fdda40a2cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006130 tx=0x7fddb4006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.376+0000 7fddba59c700 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d23ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c2d0 tx=0x7fddac01f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=306dfd17484ee445 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.376+0000 7fdda8ff9700 1 -- 172.21.15.31:0/112502845 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0ae070 con 0x55cee6d23be0 2023-07-29T16:20:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.376+0000 7fdda8ff9700 1 -- 172.21.15.31:0/112502845 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddac099430 con 0x55cee6d23be0 2023-07-29T16:20:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.376+0000 7fdda8ff9700 1 -- 172.21.15.31:0/112502845 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac010880 con 0x55cee6d23be0 2023-07-29T16:20:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 -- 172.21.15.31:0/112502845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 msgr2=0x55cee6d23ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d23ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c2d0 tx=0x7fddac01f000 comp rx=0 tx=0).stop 2023-07-29T16:20:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 -- 172.21.15.31:0/112502845 shutdown_connections 2023-07-29T16:20:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d252f0 0x55cee6d25740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d23ff0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 --2- 172.21.15.31:0/112502845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d24530 0x55cee6d24d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 -- 172.21.15.31:0/112502845 >> 172.21.15.31:0/112502845 conn(0x55cee6d193e0 msgr2=0x55cee6d1e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 -- 172.21.15.31:0/112502845 shutdown_connections 2023-07-29T16:20:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.377+0000 7fddd435db80 1 -- 172.21.15.31:0/112502845 wait complete. 2023-07-29T16:20:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.378+0000 7fddd435db80 1 -- start start 2023-07-29T16:20:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.378+0000 7fddd435db80 1 Processor -- start 2023-07-29T16:20:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.378+0000 7fddd435db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d2d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.378+0000 7fddd435db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d2ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddd435db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d34e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddd435db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6c11510 con 0x55cee6d23be0 2023-07-29T16:20:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddd435db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6c111b0 con 0x55cee6d252f0 2023-07-29T16:20:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddd435db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6af8920 con 0x55cee6d24530 2023-07-29T16:20:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddd435db80 10 client.0 _ll_register_callbacks cb 0x55cee6c1a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d2ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d34e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddba59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d2d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d2ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42216/0 (socket says 172.21.15.31:42216) 2023-07-29T16:20:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d34e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57018/0 (socket says 172.21.15.31:57018) 2023-07-29T16:20:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddb9d9b700 1 -- 172.21.15.31:0/1478494313 learned_addr learned my addr 172.21.15.31:0/1478494313 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 -- 172.21.15.31:0/1478494313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 msgr2=0x55cee6d2ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d2ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 -- 172.21.15.31:0/1478494313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 msgr2=0x55cee6d2d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d2d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddbad9d700 1 -- 172.21.15.31:0/1478494313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc1a0a00 con 0x55cee6d252f0 2023-07-29T16:20:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.379+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d2ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fddbad9d700 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d34e00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009fd0 tx=0x7fddb400b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f367a52c6582640 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fdda1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fdda1ffb700 1 -- 172.21.15.31:0/1478494313 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb400e040 con 0x55cee6d252f0 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fdda1ffb700 1 -- 172.21.15.31:0/1478494313 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fddb4021870 con 0x55cee6d252f0 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cee6c9fd10 con 0x55cee6d252f0 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fdda1ffb700 1 -- 172.21.15.31:0/1478494313 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddb4021a30 con 0x55cee6d252f0 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fddd435db80 10 client.26990 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55cee6a8bc60 con 0x55cee6d252f0 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.380+0000 7fdda1ffb700 1 -- 172.21.15.31:0/1478494313 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddb4038020 con 0x55cee6d252f0 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fdda1ffb700 1 -- 172.21.15.31:0/1478494313 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fddb4005830 con 0x55cee6d252f0 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddd435db80 10 client.26990 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddd435db80 10 client.26990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddd435db80 20 client.26990 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddd435db80 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d39640 0x55cee6d3ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddd435db80 4 client.26990 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:10.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cee6cf2d70 con 0x55cee6d39640 2023-07-29T16:20:10.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddba59c700 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d39640 0x55cee6d3ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fddba59c700 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d39640 0x55cee6d3ba50 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a23f6f315cce03 server_cookie=9d357ae6cf5c2893 in_seq=0 out_seq=0 2023-07-29T16:20:10.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.381+0000 7fdda1ffb700 10 client.26990 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:10.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.382+0000 7fdda1ffb700 1 -- 172.21.15.31:0/1478494313 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55cee6cf2d70 con 0x55cee6d39640 2023-07-29T16:20:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.382+0000 7fdda1ffb700 10 client.26990 handle_command_reply: tid=0 2023-07-29T16:20:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.382+0000 7fddd435db80 1 client.26990 shutdown 2023-07-29T16:20:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.382+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 msgr2=0x55cee6d34e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.382+0000 7fddd435db80 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d34e00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009fd0 tx=0x7fddb400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 shutdown_connections 2023-07-29T16:20:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d39640 0x55cee6d3ba50 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d34e00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d2d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 --2- 172.21.15.31:0/1478494313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d2ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 >> 172.21.15.31:0/1478494313 conn(0x55cee6d193e0 msgr2=0x55cee6d1e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fdda1ffb700 0 client.26990 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 shutdown_connections 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 1 -- 172.21.15.31:0/1478494313 wait complete. 2023-07-29T16:20:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.383+0000 7fddd435db80 20 client.26990 trim_cache size 0 max 16384 2023-07-29T16:20:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 Processor -- start 2023-07-29T16:20:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 -- start start 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d3b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d34dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d282d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6d1b460 con 0x55cee6d23be0 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6d2be80 con 0x55cee6d252f0 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.397+0000 7fddd435db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6d2e500 con 0x55cee6d24530 2023-07-29T16:20:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d34dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddba59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d3b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d34dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42218/0 (socket says 172.21.15.31:42218) 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 -- 172.21.15.31:0/2537973664 learned_addr learned my addr 172.21.15.31:0/2537973664 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d282d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 -- 172.21.15.31:0/2537973664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 msgr2=0x55cee6d282d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d282d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 -- 172.21.15.31:0/2537973664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 msgr2=0x55cee6d3b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d3b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 -- 172.21.15.31:0/2537973664 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cee6ac9250 con 0x55cee6d24530 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddbad9d700 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d282d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.398+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d34dc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fddb00127d0 tx=0x7fddb0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=534513aa6b094acc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.399+0000 7fdda97fa700 1 -- 172.21.15.31:0/2537973664 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0017050 con 0x55cee6d24530 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.399+0000 7fdda97fa700 1 -- 172.21.15.31:0/2537973664 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddb0018040 con 0x55cee6d24530 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.399+0000 7fdda97fa700 1 -- 172.21.15.31:0/2537973664 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb001c660 con 0x55cee6d24530 2023-07-29T16:20:10.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.399+0000 7fddd435db80 1 -- 172.21.15.31:0/2537973664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 msgr2=0x55cee6d34dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.399+0000 7fddd435db80 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d34dc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fddb00127d0 tx=0x7fddb0009260 comp rx=0 tx=0).stop 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 -- 172.21.15.31:0/2537973664 shutdown_connections 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d282d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d23be0 0x55cee6d3b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 --2- 172.21.15.31:0/2537973664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d24530 0x55cee6d34dc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 -- 172.21.15.31:0/2537973664 >> 172.21.15.31:0/2537973664 conn(0x55cee6d193e0 msgr2=0x55cee6d2b770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 -- 172.21.15.31:0/2537973664 shutdown_connections 2023-07-29T16:20:10.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 -- 172.21.15.31:0/2537973664 wait complete. 2023-07-29T16:20:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 -- start start 2023-07-29T16:20:10.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.400+0000 7fddd435db80 1 Processor -- start 2023-07-29T16:20:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d32d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d24530 0x55cee6d332a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d342b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6d347f0 con 0x55cee6d24530 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6d34960 con 0x55cee6d252f0 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cee6d34ad0 con 0x55cee6d23be0 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddd435db80 10 client.0 _ll_register_callbacks cb 0x55cee6c1a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d32d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d342b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57024/0 (socket says 172.21.15.31:57024) 2023-07-29T16:20:10.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddbad9d700 1 -- 172.21.15.31:0/3383568956 learned_addr learned my addr 172.21.15.31:0/3383568956 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.401+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d24530 0x55cee6d332a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddba59c700 1 -- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 msgr2=0x55cee6d342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddba59c700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d342b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddba59c700 1 -- 172.21.15.31:0/3383568956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d24530 msgr2=0x55cee6d332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddba59c700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d24530 0x55cee6d332a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddba59c700 1 -- 172.21.15.31:0/3383568956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cee6a8bc60 con 0x55cee6d23be0 2023-07-29T16:20:10.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddbad9d700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddba59c700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d32d60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c2d0 tx=0x7fddac0092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17aa933720a40cd3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fdda1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fdda1ffb700 1 -- 172.21.15.31:0/3383568956 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac0ae070 con 0x55cee6d23be0 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fdda1ffb700 1 -- 172.21.15.31:0/3383568956 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fddac09d460 con 0x55cee6d23be0 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.402+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddac099430 con 0x55cee6d23be0 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fdda1ffb700 1 -- 172.21.15.31:0/3383568956 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddac097570 con 0x55cee6d23be0 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 10 client.27007 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:10.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fddb0012450 con 0x55cee6d23be0 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fdda1ffb700 1 -- 172.21.15.31:0/3383568956 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddac099430 con 0x55cee6d23be0 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fdda1ffb700 1 -- 172.21.15.31:0/3383568956 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fddac0a13f0 con 0x55cee6d23be0 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 10 client.27007 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 10 client.27007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 20 client.27007 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d860d0 0x55cee6d884e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:10.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 4 client.27007 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cee6c6dd40 con 0x55cee6d860d0 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.403+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d860d0 0x55cee6d884e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fdda1ffb700 10 client.27007 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fddb9d9b700 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d860d0 0x55cee6d884e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6fed59becfbefb4 server_cookie=b555f1b1e588baa6 in_seq=0 out_seq=0 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fdda1ffb700 1 -- 172.21.15.31:0/3383568956 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cee6c6dd40 con 0x55cee6d860d0 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fdda1ffb700 10 client.27007 handle_command_reply: tid=0 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fddd435db80 1 client.27007 shutdown 2023-07-29T16:20:10.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 msgr2=0x55cee6d32d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.404+0000 7fddd435db80 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d32d60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fddac00c2d0 tx=0x7fddac0092d0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 shutdown_connections 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cee6d860d0 0x55cee6d884e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cee6d252f0 0x55cee6d342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cee6d23be0 0x55cee6d32d60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fdda1ffb700 0 client.27007 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:10.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 --2- 172.21.15.31:0/3383568956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cee6d24530 0x55cee6d332a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 >> 172.21.15.31:0/3383568956 conn(0x55cee6d193e0 msgr2=0x55cee6d3b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 shutdown_connections 2023-07-29T16:20:10.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 1 -- 172.21.15.31:0/3383568956 wait complete. 2023-07-29T16:20:10.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.405+0000 7fddd435db80 20 client.27007 trim_cache size 0 max 16384 2023-07-29T16:20:10.413 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:20:10.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdda40a0810 msgr2=0x7fdda40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdda40a0810 0x7fdda40a2cc0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006130 tx=0x7fddb4006000 comp rx=0 tx=0).stop 2023-07-29T16:20:10.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 msgr2=0x7fddbc100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:10.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 0x7fddbc100710 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fddac00e9a0 tx=0x7fddac00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 shutdown_connections 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdda40a0810 0x7fdda40a2cc0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fddbc105fe0 0x7fddbc0ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddbc100da0 0x7fddbc100710 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 --2- 172.21.15.31:0/350270075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddbc1036c0 0x7fddbc0fed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 >> 172.21.15.31:0/350270075 conn(0x7fddbc0fa840 msgr2=0x7fddbc101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 shutdown_connections 2023-07-29T16:20:10.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:10.406+0000 7fddc09ff700 1 -- 172.21.15.31:0/350270075 wait complete. 2023-07-29T16:20:10.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:10 smithi031 ceph-mon[95504]: pgmap v511: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:20:10.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2016974181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:20:10.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:10 smithi039 ceph-mon[128858]: pgmap v511: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:20:10.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:10 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2016974181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:20:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:10 smithi156 ceph-mon[129245]: pgmap v511: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:20:10.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:10 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2016974181' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:20:10.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 195500 9.33 MB/sec warmup 115 sec latency 12.435 ms 2023-07-29T16:20:11.427 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:20:11.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:20:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.650+0000 7fafc73ff700 1 -- 172.21.15.31:0/23768189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00feb90 msgr2=0x7fafc00fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.650+0000 7fafc73ff700 1 --2- 172.21.15.31:0/23768189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00feb90 0x7fafc00fefa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fafbc009990 tx=0x7fafbc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 -- 172.21.15.31:0/23768189 shutdown_connections 2023-07-29T16:20:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 --2- 172.21.15.31:0/23768189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc00ffe90 0x7fafc0100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 --2- 172.21.15.31:0/23768189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00feb90 0x7fafc00fefa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 --2- 172.21.15.31:0/23768189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ff4e0 0x7fafc00ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 -- 172.21.15.31:0/23768189 >> 172.21.15.31:0/23768189 conn(0x7fafc00fa840 msgr2=0x7fafc00fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 -- 172.21.15.31:0/23768189 shutdown_connections 2023-07-29T16:20:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.651+0000 7fafc73ff700 1 -- 172.21.15.31:0/23768189 wait complete. 2023-07-29T16:20:11.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 Processor -- start 2023-07-29T16:20:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 -- start start 2023-07-29T16:20:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc00feb90 0x7fafc0198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 0x7fafc01987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ffe90 0x7fafc019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc019fe10 con 0x7fafc00feb90 2023-07-29T16:20:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc019ff50 con 0x7fafc00ff4e0 2023-07-29T16:20:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.652+0000 7fafc73ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc01a00e0 con 0x7fafc00ffe90 2023-07-29T16:20:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 0x7fafc01987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc599c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ffe90 0x7fafc019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc519b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc00feb90 0x7fafc0198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 0x7fafc01987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57042/0 (socket says 172.21.15.31:57042) 2023-07-29T16:20:11.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc599c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ffe90 0x7fafc019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42248/0 (socket says 172.21.15.31:42248) 2023-07-29T16:20:11.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 -- 172.21.15.31:0/2806523932 learned_addr learned my addr 172.21.15.31:0/2806523932 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 -- 172.21.15.31:0/2806523932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ffe90 msgr2=0x7fafc019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ffe90 0x7fafc019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 -- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc00feb90 msgr2=0x7fafc0198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc00feb90 0x7fafc0198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 -- 172.21.15.31:0/2806523932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc0095f0 con 0x7fafc00ff4e0 2023-07-29T16:20:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.653+0000 7fafc499a700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 0x7fafc01987d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fafb000e9a0 tx=0x7fafb000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.654+0000 7fafb67fc700 1 -- 172.21.15.31:0/2806523932 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0009790 con 0x7fafc00ff4e0 2023-07-29T16:20:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.654+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc01a03c0 con 0x7fafc00ff4e0 2023-07-29T16:20:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.654+0000 7fafb67fc700 1 -- 172.21.15.31:0/2806523932 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafb0010430 con 0x7fafc00ff4e0 2023-07-29T16:20:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.654+0000 7fafb67fc700 1 -- 172.21.15.31:0/2806523932 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0018680 con 0x7fafc00ff4e0 2023-07-29T16:20:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.654+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc01a08d0 con 0x7fafc00ff4e0 2023-07-29T16:20:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.657+0000 7fafb67fc700 1 -- 172.21.15.31:0/2806523932 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fafb00105a0 con 0x7fafc00ff4e0 2023-07-29T16:20:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.659+0000 7fafdae03b80 1 Processor -- start 2023-07-29T16:20:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.659+0000 7fafb67fc700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafac0a0790 0x7fafac0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.659+0000 7fafdae03b80 1 -- start start 2023-07-29T16:20:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.659+0000 7fafb67fc700 1 -- 172.21.15.31:0/2806523932 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fafb0014070 con 0x7fafc00ff4e0 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.659+0000 7fafc599c700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafac0a0790 0x7fafac0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 0x564da4268070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da42685b0 0x564da4268de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42697c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafdae03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da41bfa90 con 0x564da4269370 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafdae03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da41bd450 con 0x564da4267c60 2023-07-29T16:20:11.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafdae03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da41b8d40 con 0x564da42685b0 2023-07-29T16:20:11.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 0x564da4268070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 0x564da4268070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57058/0 (socket says 172.21.15.31:57058) 2023-07-29T16:20:11.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 -- 172.21.15.31:0/1255769329 learned_addr learned my addr 172.21.15.31:0/1255769329 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc499a700 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da42685b0 0x564da4268de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 -- 172.21.15.31:0/1255769329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da42685b0 msgr2=0x564da4268de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da42685b0 0x564da4268de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 -- 172.21.15.31:0/1255769329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 msgr2=0x564da42697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.660+0000 7fafc519b700 1 -- 172.21.15.31:0/1255769329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564da3fe6e60 con 0x564da4267c60 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.661+0000 7fafc599c700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafac0a0790 0x7fafac0a2c40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fafb800f590 tx=0x7fafb8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.661+0000 7fafc519b700 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 0x564da4268070 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fafbc015000 tx=0x7fafbc005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46504f676bdd8ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.661+0000 7fafab7fe700 1 -- 172.21.15.31:0/1255769329 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc01d070 con 0x564da4267c60 2023-07-29T16:20:11.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.661+0000 7fafab7fe700 1 -- 172.21.15.31:0/1255769329 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafbc01e040 con 0x564da4267c60 2023-07-29T16:20:11.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.661+0000 7fafab7fe700 1 -- 172.21.15.31:0/1255769329 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc004870 con 0x564da4267c60 2023-07-29T16:20:11.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 -- 172.21.15.31:0/1255769329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 msgr2=0x564da4268070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 0x564da4268070 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fafbc015000 tx=0x7fafbc005ea0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 -- 172.21.15.31:0/1255769329 shutdown_connections 2023-07-29T16:20:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4267c60 0x564da4268070 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1255769329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da42685b0 0x564da4268de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 -- 172.21.15.31:0/1255769329 >> 172.21.15.31:0/1255769329 conn(0x564da425d460 msgr2=0x564da4262a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 -- 172.21.15.31:0/1255769329 shutdown_connections 2023-07-29T16:20:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.662+0000 7fafdae03b80 1 -- 172.21.15.31:0/1255769329 wait complete. 2023-07-29T16:20:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.663+0000 7fafdae03b80 1 -- start start 2023-07-29T16:20:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.663+0000 7fafdae03b80 1 Processor -- start 2023-07-29T16:20:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4267c60 0x564da4271760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4271ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4269370 0x564da4278ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da4155590 con 0x564da4267c60 2023-07-29T16:20:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da4155230 con 0x564da42685b0 2023-07-29T16:20:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da403c9a0 con 0x564da4269370 2023-07-29T16:20:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafdae03b80 10 client.0 _ll_register_callbacks cb 0x564da415e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafc499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4271ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafc599c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4269370 0x564da4278ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafc499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4271ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57062/0 (socket says 172.21.15.31:57062) 2023-07-29T16:20:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafc499a700 1 -- 172.21.15.31:0/1853731883 learned_addr learned my addr 172.21.15.31:0/1853731883 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.664+0000 7fafc519b700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4267c60 0x564da4271760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc499a700 1 -- 172.21.15.31:0/1853731883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4269370 msgr2=0x564da4278ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc499a700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4269370 0x564da4278ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc499a700 1 -- 172.21.15.31:0/1853731883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4267c60 msgr2=0x564da4271760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc499a700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4267c60 0x564da4271760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc499a700 1 -- 172.21.15.31:0/1853731883 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc010b320 con 0x564da42685b0 2023-07-29T16:20:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc599c700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4269370 0x564da4278ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafc499a700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4271ca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fafb0095980 tx=0x7fafb0099480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a3c82adcc5e12b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafa8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafa8ff9700 1 -- 172.21.15.31:0/1853731883 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00a18d0 con 0x564da42685b0 2023-07-29T16:20:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafa8ff9700 1 -- 172.21.15.31:0/1853731883 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fafb009d160 con 0x564da42685b0 2023-07-29T16:20:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564da41e3d90 con 0x564da42685b0 2023-07-29T16:20:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.665+0000 7fafa8ff9700 1 -- 172.21.15.31:0/1853731883 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb0098970 con 0x564da42685b0 2023-07-29T16:20:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 10 client.27008 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564da3fcfce0 con 0x564da42685b0 2023-07-29T16:20:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafa8ff9700 1 -- 172.21.15.31:0/1853731883 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564da41e3d90 con 0x564da42685b0 2023-07-29T16:20:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafa8ff9700 1 -- 172.21.15.31:0/1853731883 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fafb00a1560 con 0x564da42685b0 2023-07-29T16:20:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 10 client.27008 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 10 client.27008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 20 client.27008 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da427d570 0x564da427f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 4 client.27008 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564da4236df0 con 0x564da427d570 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.666+0000 7fafc519b700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da427d570 0x564da427f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.667+0000 7fafc519b700 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da427d570 0x564da427f980 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e68c4c62700dc8 server_cookie=71accc99c8a7af5d in_seq=0 out_seq=0 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.667+0000 7fafa8ff9700 10 client.27008 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafa8ff9700 1 -- 172.21.15.31:0/1853731883 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564da4236df0 con 0x564da427d570 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafa8ff9700 10 client.27008 handle_command_reply: tid=0 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 client.27008 shutdown 2023-07-29T16:20:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 msgr2=0x564da4271ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4271ca0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fafb0095980 tx=0x7fafb0099480 comp rx=0 tx=0).stop 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 shutdown_connections 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da427d570 0x564da427f980 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4269370 0x564da4278ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4267c60 0x564da4271760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 --2- 172.21.15.31:0/1853731883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4271ca0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 >> 172.21.15.31:0/1853731883 conn(0x564da425d460 msgr2=0x564da42620e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafa8ff9700 0 client.27008 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 shutdown_connections 2023-07-29T16:20:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.668+0000 7fafdae03b80 1 -- 172.21.15.31:0/1853731883 wait complete. 2023-07-29T16:20:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.669+0000 7fafdae03b80 20 client.27008 trim_cache size 0 max 16384 2023-07-29T16:20:11.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 Processor -- start 2023-07-29T16:20:11.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 -- start start 2023-07-29T16:20:11.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da42620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da42685b0 0x564da4278ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4269370 0x564da426f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da426f4a0 con 0x564da42685b0 2023-07-29T16:20:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da427d290 con 0x564da4269370 2023-07-29T16:20:11.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafdae03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da4268bd0 con 0x564da4267c60 2023-07-29T16:20:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafc499a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da42685b0 0x564da4278ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafc519b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da42620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafc599c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4269370 0x564da426f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafc519b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da42620e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42280/0 (socket says 172.21.15.31:42280) 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.694+0000 7fafc519b700 1 -- 172.21.15.31:0/919516816 learned_addr learned my addr 172.21.15.31:0/919516816 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafc519b700 1 -- 172.21.15.31:0/919516816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4269370 msgr2=0x564da426f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafc519b700 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4269370 0x564da426f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafc519b700 1 -- 172.21.15.31:0/919516816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da42685b0 msgr2=0x564da4278ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafc519b700 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da42685b0 0x564da4278ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafc519b700 1 -- 172.21.15.31:0/919516816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564da42707d0 con 0x564da4267c60 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafc519b700 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da42620e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fafbc000c00 tx=0x7fafbc01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aa0e47dc7704c74 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafabfff700 1 -- 172.21.15.31:0/919516816 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc01d040 con 0x564da4267c60 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafabfff700 1 -- 172.21.15.31:0/919516816 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafbc00bb60 con 0x564da4267c60 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.695+0000 7fafabfff700 1 -- 172.21.15.31:0/919516816 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc034c00 con 0x564da4267c60 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 -- 172.21.15.31:0/919516816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 msgr2=0x564da42620e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da42620e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fafbc000c00 tx=0x7fafbc01a040 comp rx=0 tx=0).stop 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 -- 172.21.15.31:0/919516816 shutdown_connections 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da4269370 0x564da426f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da42620e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 --2- 172.21.15.31:0/919516816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da42685b0 0x564da4278ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 -- 172.21.15.31:0/919516816 >> 172.21.15.31:0/919516816 conn(0x564da425d460 msgr2=0x564da4263710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 -- 172.21.15.31:0/919516816 shutdown_connections 2023-07-29T16:20:11.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.696+0000 7fafdae03b80 1 -- 172.21.15.31:0/919516816 wait complete. 2023-07-29T16:20:11.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.697+0000 7fafdae03b80 1 -- start start 2023-07-29T16:20:11.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.697+0000 7fafdae03b80 1 Processor -- start 2023-07-29T16:20:11.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.697+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da4275880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4275dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafdae03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafdae03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da4278de0 con 0x564da4269370 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafdae03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da4278f50 con 0x564da42685b0 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafdae03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564da42cf630 con 0x564da4267c60 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafdae03b80 10 client.0 _ll_register_callbacks cb 0x564da415e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:11.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4275dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc519b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da4275880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4275dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57082/0 (socket says 172.21.15.31:57082) 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 -- 172.21.15.31:0/514117200 learned_addr learned my addr 172.21.15.31:0/514117200 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc599c700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 -- 172.21.15.31:0/514117200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 msgr2=0x564da4275880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da4275880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 -- 172.21.15.31:0/514117200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 msgr2=0x564da42cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc499a700 1 -- 172.21.15.31:0/514117200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564da3fe6e60 con 0x564da42685b0 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc599c700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.698+0000 7fafc519b700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da4275880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafc499a700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4275dc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fafb00a13a0 tx=0x7fafb009d390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bae73cfe5b10a9f9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafa8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafa8ff9700 1 -- 172.21.15.31:0/514117200 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb0063070 con 0x564da42685b0 2023-07-29T16:20:11.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafa8ff9700 1 -- 172.21.15.31:0/514117200 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fafb00c9680 con 0x564da42685b0 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564da41b1dc0 con 0x564da42685b0 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafa8ff9700 1 -- 172.21.15.31:0/514117200 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb00c9490 con 0x564da42685b0 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 10 client.27014 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x564da400d2d0 con 0x564da42685b0 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafa8ff9700 1 -- 172.21.15.31:0/514117200 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564da41b1dc0 con 0x564da42685b0 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafa8ff9700 1 -- 172.21.15.31:0/514117200 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fafb0095270 con 0x564da42685b0 2023-07-29T16:20:11.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 10 client.27014 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 10 client.27014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 20 client.27014 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da42d0bc0 0x564da42d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 4 client.27014 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.699+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564da41b1ae0 con 0x564da42d0bc0 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.700+0000 7fafc519b700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da42d0bc0 0x564da42d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.700+0000 7fafa8ff9700 10 client.27014 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:11.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.700+0000 7fafc519b700 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da42d0bc0 0x564da42d2fd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d424b47e1afc1084 server_cookie=3eb6ceaa2c99c06a in_seq=0 out_seq=0 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.701+0000 7fafa8ff9700 1 -- 172.21.15.31:0/514117200 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564da41b1ae0 con 0x564da42d0bc0 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.701+0000 7fafa8ff9700 10 client.27014 handle_command_reply: tid=0 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.701+0000 7fafdae03b80 1 client.27014 shutdown 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 msgr2=0x564da4275dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4275dc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fafb00a13a0 tx=0x7fafb009d390 comp rx=0 tx=0).stop 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 shutdown_connections 2023-07-29T16:20:11.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564da42d0bc0 0x564da42d2fd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564da4269370 0x564da42cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564da4267c60 0x564da4275880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafa8ff9700 0 client.27014 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 --2- 172.21.15.31:0/514117200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564da42685b0 0x564da4275dc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 >> 172.21.15.31:0/514117200 conn(0x564da425d460 msgr2=0x564da42620e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 shutdown_connections 2023-07-29T16:20:11.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 1 -- 172.21.15.31:0/514117200 wait complete. 2023-07-29T16:20:11.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.702+0000 7fafdae03b80 20 client.27014 trim_cache size 0 max 16384 2023-07-29T16:20:11.709 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:20:11.709 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:20:11.709 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "b413990e-19a5-4462-be86-259f2712ce9e", 2023-07-29T16:20:11.709 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:20:11.709 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:20:11.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafac0a0790 msgr2=0x7fafac0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafac0a0790 0x7fafac0a2c40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fafb800f590 tx=0x7fafb8009920 comp rx=0 tx=0).stop 2023-07-29T16:20:11.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 msgr2=0x7fafc01987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 0x7fafc01987d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fafb000e9a0 tx=0x7fafb000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 shutdown_connections 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fafac0a0790 0x7fafac0a2c40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fafc00ffe90 0x7fafc019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafc00feb90 0x7fafc0198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 --2- 172.21.15.31:0/2806523932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fafc00ff4e0 0x7fafc01987d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 >> 172.21.15.31:0/2806523932 conn(0x7fafc00fa840 msgr2=0x7fafc00fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 shutdown_connections 2023-07-29T16:20:11.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.703+0000 7fafc73ff700 1 -- 172.21.15.31:0/2806523932 wait complete. 2023-07-29T16:20:11.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:20:11.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 197075 9.29 MB/sec warmup 116 sec latency 40.662 ms 2023-07-29T16:20:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/2393586099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae4105fe0 msgr2=0x7f7ae4108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/2393586099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae4105fe0 0x7f7ae4108440 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009f90 tx=0x7f7adc00b330 comp rx=0 tx=0).stop 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/2393586099 shutdown_connections 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/2393586099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae4105fe0 0x7f7ae4108440 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/2393586099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/2393586099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ae41036c0 0x7f7ae4105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/2393586099 >> 172.21.15.31:0/2393586099 conn(0x7f7ae40fa840 msgr2=0x7f7ae40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/2393586099 shutdown_connections 2023-07-29T16:20:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.958+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/2393586099 wait complete. 2023-07-29T16:20:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.959+0000 7f7aeb3ff700 1 Processor -- start 2023-07-29T16:20:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.959+0000 7f7aeb3ff700 1 -- start start 2023-07-29T16:20:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.959+0000 7f7aeb3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae41983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.959+0000 7f7aeb3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 0x7f7ae41988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.959+0000 7f7aeb3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ae4105fe0 0x7f7ae419f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7aeb3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae419ff30 con 0x7f7ae4105fe0 2023-07-29T16:20:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7aeb3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae41a0070 con 0x7f7ae4100da0 2023-07-29T16:20:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7aeb3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae41a0200 con 0x7f7ae41036c0 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae41983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae899a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 0x7f7ae41988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae41983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57094/0 (socket says 172.21.15.31:57094) 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 -- 172.21.15.31:0/4024597544 learned_addr learned my addr 172.21.15.31:0/4024597544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae899a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 0x7f7ae41988f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42300/0 (socket says 172.21.15.31:42300) 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae999c700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ae4105fe0 0x7f7ae419f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 -- 172.21.15.31:0/4024597544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 msgr2=0x7f7ae41988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 0x7f7ae41988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 -- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ae4105fe0 msgr2=0x7f7ae419f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ae4105fe0 0x7f7ae419f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 -- 172.21.15.31:0/4024597544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00095f0 con 0x7f7ae4100da0 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae899a700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 0x7f7ae41988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.960+0000 7f7ae919b700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae41983b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000ec10 tx=0x7f7ae000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.961+0000 7f7ada7fc700 1 -- 172.21.15.31:0/4024597544 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae000cc90 con 0x7f7ae4100da0 2023-07-29T16:20:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.961+0000 7f7ada7fc700 1 -- 172.21.15.31:0/4024597544 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ae00044e0 con 0x7f7ae4100da0 2023-07-29T16:20:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.961+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc009c40 con 0x7f7ae4100da0 2023-07-29T16:20:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.961+0000 7f7ada7fc700 1 -- 172.21.15.31:0/4024597544 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0010430 con 0x7f7ae4100da0 2023-07-29T16:20:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.961+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae41a08b0 con 0x7f7ae4100da0 2023-07-29T16:20:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.964+0000 7f7ada7fc700 1 -- 172.21.15.31:0/4024597544 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f7ae0003fb0 con 0x7f7ae4100da0 2023-07-29T16:20:11.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.965+0000 7f7afed4eb80 1 Processor -- start 2023-07-29T16:20:11.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.965+0000 7f7ada7fc700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7ad00a0810 0x7f7ad00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 -- start start 2023-07-29T16:20:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ada7fc700 1 -- 172.21.15.31:0/4024597544 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7ae0014070 con 0x7f7ae4100da0 2023-07-29T16:20:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb433ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb434d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ae999c700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7ad00a0810 0x7f7ad00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb435740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb38ba10 con 0x55d8cb4352f0 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb3893d0 con 0x55d8cb434530 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb384cc0 con 0x55d8cb433be0 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ae919b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb433ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ae919b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb433ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42306/0 (socket says 172.21.15.31:42306) 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ae919b700 1 -- 172.21.15.31:0/142603390 learned_addr learned my addr 172.21.15.31:0/142603390 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ae999c700 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb435740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.966+0000 7f7ae899a700 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb434d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae919b700 1 -- 172.21.15.31:0/142603390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 msgr2=0x55d8cb434d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae919b700 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb434d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae919b700 1 -- 172.21.15.31:0/142603390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 msgr2=0x55d8cb435740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae919b700 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb435740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae919b700 1 -- 172.21.15.31:0/142603390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8cb1b2de0 con 0x55d8cb433be0 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae999c700 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb435740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae919b700 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb433ff0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00a16c0 tx=0x7f7ae0099640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f6b1c3e449b31 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7ae999c700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7ad00a0810 0x7f7ad00a2cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009550 tx=0x7f7adc009440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7acf7fe700 1 -- 172.21.15.31:0/142603390 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00b1070 con 0x55d8cb433be0 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7acf7fe700 1 -- 172.21.15.31:0/142603390 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ae009d7d0 con 0x55d8cb433be0 2023-07-29T16:20:11.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.967+0000 7f7acf7fe700 1 -- 172.21.15.31:0/142603390 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0099a00 con 0x55d8cb433be0 2023-07-29T16:20:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 -- 172.21.15.31:0/142603390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 msgr2=0x55d8cb433ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb433ff0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00a16c0 tx=0x7f7ae0099640 comp rx=0 tx=0).stop 2023-07-29T16:20:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 -- 172.21.15.31:0/142603390 shutdown_connections 2023-07-29T16:20:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb435740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb433ff0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/142603390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb434d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 -- 172.21.15.31:0/142603390 >> 172.21.15.31:0/142603390 conn(0x55d8cb4293e0 msgr2=0x55d8cb42e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 -- 172.21.15.31:0/142603390 shutdown_connections 2023-07-29T16:20:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.968+0000 7f7afed4eb80 1 -- 172.21.15.31:0/142603390 wait complete. 2023-07-29T16:20:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 -- start start 2023-07-29T16:20:11.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 Processor -- start 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb43d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb43dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb444de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb321510 con 0x55d8cb4352f0 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb3211b0 con 0x55d8cb434530 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb208920 con 0x55d8cb433be0 2023-07-29T16:20:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7afed4eb80 10 client.0 _ll_register_callbacks cb 0x55d8cb32a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.969+0000 7f7ae899a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb43dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae999c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb444de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb43d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae899a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb43dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57114/0 (socket says 172.21.15.31:57114) 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae899a700 1 -- 172.21.15.31:0/1436725344 learned_addr learned my addr 172.21.15.31:0/1436725344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 -- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 msgr2=0x55d8cb43dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb43dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 -- 172.21.15.31:0/1436725344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 msgr2=0x55d8cb444de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb444de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 -- 172.21.15.31:0/1436725344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae41a08b0 con 0x55d8cb433be0 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae999c700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb444de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae899a700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb43dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7ae919b700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb43d860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0097d90 tx=0x7f7ae0099920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29c8157f76d72383 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.970+0000 7f7accff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7accff9700 1 -- 172.21.15.31:0/1436725344 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00b1070 con 0x55d8cb433be0 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7accff9700 1 -- 172.21.15.31:0/1436725344 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7ae0099a00 con 0x55d8cb433be0 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8cb3afd10 con 0x55d8cb433be0 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7accff9700 1 -- 172.21.15.31:0/1436725344 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ae0099bc0 con 0x55d8cb433be0 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 10 client.27037 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d8cb19bc60 con 0x55d8cb433be0 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7accff9700 1 -- 172.21.15.31:0/1436725344 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d8cb3afd10 con 0x55d8cb433be0 2023-07-29T16:20:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7accff9700 1 -- 172.21.15.31:0/1436725344 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f7ae0098970 con 0x55d8cb433be0 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 10 client.27037 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 10 client.27037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 20 client.27037 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb4491a0 0x55d8cb44b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 4 client.27037 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.971+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8cb402d70 con 0x55d8cb4491a0 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.972+0000 7f7ae899a700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb4491a0 0x55d8cb44b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.972+0000 7f7ae899a700 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb4491a0 0x55d8cb44b5b0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9e68e27f778f30 server_cookie=c4ce854011e59edd in_seq=0 out_seq=0 2023-07-29T16:20:11.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.972+0000 7f7accff9700 10 client.27037 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:11.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.973+0000 7f7accff9700 1 -- 172.21.15.31:0/1436725344 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d8cb402d70 con 0x55d8cb4491a0 2023-07-29T16:20:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.973+0000 7f7accff9700 10 client.27037 handle_command_reply: tid=0 2023-07-29T16:20:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.973+0000 7f7afed4eb80 1 client.27037 shutdown 2023-07-29T16:20:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.973+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 msgr2=0x55d8cb43d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.973+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb43d860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0097d90 tx=0x7f7ae0099920 comp rx=0 tx=0).stop 2023-07-29T16:20:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 shutdown_connections 2023-07-29T16:20:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb4491a0 0x55d8cb44b5b0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb4352f0 0x55d8cb444de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb43d860 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/1436725344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb434530 0x55d8cb43dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7accff9700 0 client.27037 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 >> 172.21.15.31:0/1436725344 conn(0x55d8cb4293e0 msgr2=0x55d8cb42e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 shutdown_connections 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 1 -- 172.21.15.31:0/1436725344 wait complete. 2023-07-29T16:20:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.974+0000 7f7afed4eb80 20 client.27037 trim_cache size 0 max 16384 2023-07-29T16:20:11.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.992+0000 7f7afed4eb80 1 Processor -- start 2023-07-29T16:20:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.992+0000 7f7afed4eb80 1 -- start start 2023-07-29T16:20:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb438250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb444da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb438790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:11.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb448fd0 con 0x55d8cb434530 2023-07-29T16:20:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb4266e0 con 0x55d8cb4352f0 2023-07-29T16:20:11.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb43cd10 con 0x55d8cb433be0 2023-07-29T16:20:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb438790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae899a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb444da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb438790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57122/0 (socket says 172.21.15.31:57122) 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 -- 172.21.15.31:0/316177047 learned_addr learned my addr 172.21.15.31:0/316177047 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae919b700 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb438250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 -- 172.21.15.31:0/316177047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 msgr2=0x55d8cb438250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb438250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 -- 172.21.15.31:0/316177047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 msgr2=0x55d8cb444da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb444da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae999c700 1 -- 172.21.15.31:0/316177047 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8cb1d9250 con 0x55d8cb4352f0 2023-07-29T16:20:11.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.993+0000 7f7ae899a700 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb444da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.994+0000 7f7ae999c700 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb438790 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7adc003e50 tx=0x7f7adc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=501931f10bd81b6c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.994+0000 7f7acffff700 1 -- 172.21.15.31:0/316177047 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc025070 con 0x55d8cb4352f0 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.994+0000 7f7acffff700 1 -- 172.21.15.31:0/316177047 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7adc026040 con 0x55d8cb4352f0 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.994+0000 7f7acffff700 1 -- 172.21.15.31:0/316177047 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc00e040 con 0x55d8cb4352f0 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 -- 172.21.15.31:0/316177047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 msgr2=0x55d8cb438790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb438790 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7adc003e50 tx=0x7f7adc00b330 comp rx=0 tx=0).stop 2023-07-29T16:20:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 -- 172.21.15.31:0/316177047 shutdown_connections 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb438790 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb438250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/316177047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb444da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 -- 172.21.15.31:0/316177047 >> 172.21.15.31:0/316177047 conn(0x55d8cb4293e0 msgr2=0x55d8cb43b720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 -- 172.21.15.31:0/316177047 shutdown_connections 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 -- 172.21.15.31:0/316177047 wait complete. 2023-07-29T16:20:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 -- start start 2023-07-29T16:20:11.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.995+0000 7f7afed4eb80 1 Processor -- start 2023-07-29T16:20:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb4418d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb441e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb49ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb444e30 con 0x55d8cb434530 2023-07-29T16:20:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb444fa0 con 0x55d8cb4352f0 2023-07-29T16:20:12.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8cb445160 con 0x55d8cb433be0 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7afed4eb80 10 client.0 _ll_register_callbacks cb 0x55d8cb32a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7ae999c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb49ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7ae999c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb49ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57128/0 (socket says 172.21.15.31:57128) 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7ae999c700 1 -- 172.21.15.31:0/3016326041 learned_addr learned my addr 172.21.15.31:0/3016326041 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7ae919b700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb4418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.996+0000 7f7ae899a700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb441e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae999c700 1 -- 172.21.15.31:0/3016326041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 msgr2=0x55d8cb4418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae999c700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb4418d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae999c700 1 -- 172.21.15.31:0/3016326041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 msgr2=0x55d8cb441e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae999c700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb441e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae999c700 1 -- 172.21.15.31:0/3016326041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad4012a00 con 0x55d8cb4352f0 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae919b700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb4418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7ae999c700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb49ad20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00bcd0 tx=0x7f7adc0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7be5e17cf2d7ec77 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7accff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7accff9700 1 -- 172.21.15.31:0/3016326041 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc025040 con 0x55d8cb4352f0 2023-07-29T16:20:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ae009d7d0 con 0x55d8cb4352f0 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7accff9700 1 -- 172.21.15.31:0/3016326041 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7adc01d7e0 con 0x55d8cb4352f0 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7accff9700 1 -- 172.21.15.31:0/3016326041 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7adc00e040 con 0x55d8cb4352f0 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7afed4eb80 10 client.27032 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.997+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f7ae4109030 con 0x55d8cb4352f0 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7accff9700 1 -- 172.21.15.31:0/3016326041 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ae009d7d0 con 0x55d8cb4352f0 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7accff9700 1 -- 172.21.15.31:0/3016326041 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f7adc020450 con 0x55d8cb4352f0 2023-07-29T16:20:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7afed4eb80 10 client.27032 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7afed4eb80 10 client.27032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7afed4eb80 20 client.27032 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb49c8f0 0x55d8cb49ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7afed4eb80 4 client.27032 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d8cb37dd40 con 0x55d8cb49c8f0 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.998+0000 7f7ae919b700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb49c8f0 0x55d8cb49ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.999+0000 7f7ae919b700 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb49c8f0 0x55d8cb49ed00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bab87d94619ef68 server_cookie=8da1a07325d59710 in_seq=0 out_seq=0 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:11.999+0000 7f7accff9700 10 client.27032 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7accff9700 1 -- 172.21.15.31:0/3016326041 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d8cb37dd40 con 0x55d8cb49c8f0 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7accff9700 10 client.27032 handle_command_reply: tid=0 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 client.27032 shutdown 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 msgr2=0x55d8cb49ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb49ad20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00bcd0 tx=0x7f7adc0096c0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 shutdown_connections 2023-07-29T16:20:12.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d8cb49c8f0 0x55d8cb49ed00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d8cb4352f0 0x55d8cb49ad20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d8cb433be0 0x55d8cb4418d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7accff9700 0 client.27032 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 --2- 172.21.15.31:0/3016326041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d8cb434530 0x55d8cb441e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.000+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 >> 172.21.15.31:0/3016326041 conn(0x55d8cb4293e0 msgr2=0x55d8cb438250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 shutdown_connections 2023-07-29T16:20:12.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7afed4eb80 1 -- 172.21.15.31:0/3016326041 wait complete. 2023-07-29T16:20:12.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7afed4eb80 20 client.27032 trim_cache size 0 max 16384 2023-07-29T16:20:12.007 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:20:12.007 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:20:12.007 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout: "b413990e-19a5-4462-be86-259f2712ce9e": { 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "b413990e-19a5-4462-be86-259f2712ce9e", 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:20:12.008 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:20:12.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7ad00a0810 msgr2=0x7f7ad00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:12.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7ad00a0810 0x7f7ad00a2cc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009550 tx=0x7f7adc009440 comp rx=0 tx=0).stop 2023-07-29T16:20:12.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 msgr2=0x7f7ae41983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:12.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.001+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae41983b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7ae000ec10 tx=0x7f7ae000ef20 comp rx=0 tx=0).stop 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 shutdown_connections 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7ad00a0810 0x7f7ad00a2cc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ae4105fe0 0x7f7ae419f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ae4100da0 0x7f7ae41983b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 --2- 172.21.15.31:0/4024597544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ae41036c0 0x7f7ae41988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 >> 172.21.15.31:0/4024597544 conn(0x7f7ae40fa840 msgr2=0x7f7ae40ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 shutdown_connections 2023-07-29T16:20:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:12.002+0000 7f7aeb3ff700 1 -- 172.21.15.31:0/4024597544 wait complete. 2023-07-29T16:20:12.020 INFO:tasks.cephfs.filesystem:scrub status for tag:b413990e-19a5-4462-be86-259f2712ce9e - {'path': '/', 'tag': 'b413990e-19a5-4462-be86-259f2712ce9e', 'options': 'recursive,force'} 2023-07-29T16:20:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:12 smithi039 ceph-mon[128858]: pgmap v512: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:20:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:12 smithi156 ceph-mon[129245]: pgmap v512: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:20:12.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 199007 9.33 MB/sec warmup 117 sec latency 16.399 ms 2023-07-29T16:20:12.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:12 smithi031 ceph-mon[95504]: pgmap v512: 97 pgs: 97 active+clean; 59 MiB data, 835 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:20:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:20:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:20:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:20:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:20:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:20:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:20:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:20:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:20:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 200781 9.30 MB/sec warmup 118 sec latency 12.512 ms 2023-07-29T16:20:13.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:20:13.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:20:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:20:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:20:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:14.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:14 smithi039 ceph-mon[128858]: pgmap v513: 97 pgs: 97 active+clean; 63 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 317 op/s 2023-07-29T16:20:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:14 smithi156 ceph-mon[129245]: pgmap v513: 97 pgs: 97 active+clean; 63 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 317 op/s 2023-07-29T16:20:14.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 202495 9.33 MB/sec warmup 119 sec latency 28.157 ms 2023-07-29T16:20:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:14 smithi031 ceph-mon[95504]: pgmap v513: 97 pgs: 97 active+clean; 63 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 317 op/s 2023-07-29T16:20:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 -- 172.21.15.31:0/3312865164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 msgr2=0x7fd1c4101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/3312865164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 0x7fd1c4101990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009990 tx=0x7fd1b4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 -- 172.21.15.31:0/3312865164 shutdown_connections 2023-07-29T16:20:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/3312865164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4101ed0 0x7fd1c4108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/3312865164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4100350 0x7fd1c4100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/3312865164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 0x7fd1c4101990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.390+0000 7fd1cbfff700 1 -- 172.21.15.31:0/3312865164 >> 172.21.15.31:0/3312865164 conn(0x7fd1c40fb970 msgr2=0x7fd1c40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.391+0000 7fd1cbfff700 1 -- 172.21.15.31:0/3312865164 shutdown_connections 2023-07-29T16:20:16.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.391+0000 7fd1cbfff700 1 -- 172.21.15.31:0/3312865164 wait complete. 2023-07-29T16:20:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 Processor -- start 2023-07-29T16:20:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 -- start start 2023-07-29T16:20:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 0x7fd1c4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 0x7fd1c41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4101ed0 0x7fd1c4199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c4199db0 con 0x7fd1c4101ed0 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c4199f20 con 0x7fd1c4100350 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1cbfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c419a0e0 con 0x7fd1c4101540 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 0x7fd1c4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 0x7fd1c4198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57144/0 (socket says 172.21.15.31:57144) 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.392+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2264407866 learned_addr learned my addr 172.21.15.31:0/2264407866 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c959a700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 0x7fd1c41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4101ed0 0x7fd1c4199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2264407866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 msgr2=0x7fd1c41988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 0x7fd1c41988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4101ed0 msgr2=0x7fd1c4199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4101ed0 0x7fd1c4199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2264407866 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b40095f0 con 0x7fd1c4100350 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4101ed0 0x7fd1c4199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 0x7fd1c4198390 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000e970 tx=0x7fd1c000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1baffd700 1 -- 172.21.15.31:0/2264407866 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0009790 con 0x7fd1c4100350 2023-07-29T16:20:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1baffd700 1 -- 172.21.15.31:0/2264407866 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1c000eec0 con 0x7fd1c4100350 2023-07-29T16:20:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1baffd700 1 -- 172.21.15.31:0/2264407866 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0003cf0 con 0x7fd1c4100350 2023-07-29T16:20:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.393+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c419a3c0 con 0x7fd1c4100350 2023-07-29T16:20:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.394+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c41a0b80 con 0x7fd1c4100350 2023-07-29T16:20:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.397+0000 7fd1baffd700 1 -- 172.21.15.31:0/2264407866 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd1c000ccd0 con 0x7fd1c4100350 2023-07-29T16:20:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.398+0000 7fd1df8bfb80 1 Processor -- start 2023-07-29T16:20:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.398+0000 7fd1df8bfb80 1 -- start start 2023-07-29T16:20:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170c9df0 0x5606170ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170cacf0 0x5606170cb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170219b0 con 0x5606170c9df0 2023-07-29T16:20:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061701f370 con 0x5606170cacf0 2023-07-29T16:20:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061701ac60 con 0x560616f256b0 2023-07-29T16:20:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1baffd700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1b00a0810 0x7fd1b00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170c98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42374/0 (socket says 172.21.15.31:42374) 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 -- 172.21.15.31:0/2870739931 learned_addr learned my addr 172.21.15.31:0/2870739931 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170c9df0 0x5606170ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1c959a700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170cacf0 0x5606170cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1baffd700 1 -- 172.21.15.31:0/2264407866 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd1c0014070 con 0x7fd1c4100350 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1b00a0810 0x7fd1b00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 -- 172.21.15.31:0/2870739931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170cacf0 msgr2=0x5606170cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170cacf0 0x5606170cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 -- 172.21.15.31:0/2870739931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170c9df0 msgr2=0x5606170ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170c9df0 0x5606170ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1ca59c700 1 -- 172.21.15.31:0/2870739931 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560616e48d80 con 0x560616f256b0 2023-07-29T16:20:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1c959a700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170cacf0 0x5606170cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.399+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170c9df0 0x5606170ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.400+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170c98b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0076a0 tx=0x7fd1bc0079b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a7f5ba794b8fad0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.400+0000 7fd1affff700 1 -- 172.21.15.31:0/2870739931 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00e070 con 0x560616f256b0 2023-07-29T16:20:16.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.400+0000 7fd1affff700 1 -- 172.21.15.31:0/2870739931 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1bc00f040 con 0x560616f256b0 2023-07-29T16:20:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.400+0000 7fd1affff700 1 -- 172.21.15.31:0/2870739931 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc016400 con 0x560616f256b0 2023-07-29T16:20:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2870739931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 msgr2=0x5606170c98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1b00a0810 0x7fd1b00a2cc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc021000 tx=0x7fd1bc01e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170c98b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc0076a0 tx=0x7fd1bc0079b0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2870739931 shutdown_connections 2023-07-29T16:20:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170cacf0 0x5606170cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170c98b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2870739931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170c9df0 0x5606170ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2870739931 >> 172.21.15.31:0/2870739931 conn(0x5606170befd0 msgr2=0x5606170c4300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:16.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2870739931 shutdown_connections 2023-07-29T16:20:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.401+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2870739931 wait complete. 2023-07-29T16:20:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 -- start start 2023-07-29T16:20:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 Processor -- start 2023-07-29T16:20:16.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 0x5606170d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170c9df0 0x5606170d3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170da480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560616fb74b0 con 0x5606170cacf0 2023-07-29T16:20:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560616fb7150 con 0x560616f256b0 2023-07-29T16:20:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560616e9e8c0 con 0x5606170c9df0 2023-07-29T16:20:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.402+0000 7fd1df8bfb80 10 client.0 _ll_register_callbacks cb 0x560616fc05c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 0x5606170d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1ca59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170da480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 0x5606170d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57164/0 (socket says 172.21.15.31:57164) 2023-07-29T16:20:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/4273904384 learned_addr learned my addr 172.21.15.31:0/4273904384 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c959a700 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170c9df0 0x5606170d3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/4273904384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170c9df0 msgr2=0x5606170d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170c9df0 0x5606170d3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/4273904384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 msgr2=0x5606170da480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170da480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/4273904384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c41a0b80 con 0x560616f256b0 2023-07-29T16:20:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 0x5606170d2f00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000c5b0 tx=0x7fd1c0098e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0e36905607c9caa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.403+0000 7fd1ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/4273904384 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0099470 con 0x560616f256b0 2023-07-29T16:20:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/4273904384 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd1c009d4a0 con 0x560616f256b0 2023-07-29T16:20:16.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560617045cb0 con 0x560616f256b0 2023-07-29T16:20:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/4273904384 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1c00a18c0 con 0x560616f256b0 2023-07-29T16:20:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 10 client.27044 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7fd1c41090c0 con 0x560616f256b0 2023-07-29T16:20:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/4273904384 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560617045cb0 con 0x560616f256b0 2023-07-29T16:20:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/4273904384 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fd1c00a1430 con 0x560616f256b0 2023-07-29T16:20:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 10 client.27044 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 10 client.27044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 20 client.27044 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606170decf0 0x5606170e1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 4 client.27044 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.404+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560617098d10 con 0x5606170decf0 2023-07-29T16:20:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.405+0000 7fd1c959a700 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606170decf0 0x5606170e1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.405+0000 7fd1ad7fa700 10 client.27044 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.405+0000 7fd1c959a700 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606170decf0 0x5606170e1100 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d0fe084c67cbdf9 server_cookie=758a971d656587d7 in_seq=0 out_seq=0 2023-07-29T16:20:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/4273904384 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560617098d10 con 0x5606170decf0 2023-07-29T16:20:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1ad7fa700 10 client.27044 handle_command_reply: tid=0 2023-07-29T16:20:16.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 client.27044 shutdown 2023-07-29T16:20:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 msgr2=0x5606170d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 0x5606170d2f00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000c5b0 tx=0x7fd1c0098e50 comp rx=0 tx=0).stop 2023-07-29T16:20:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 shutdown_connections 2023-07-29T16:20:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5606170decf0 0x5606170e1100 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170da480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560616f256b0 0x5606170d2f00 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/4273904384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170c9df0 0x5606170d3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1ad7fa700 0 client.27044 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 >> 172.21.15.31:0/4273904384 conn(0x5606170befd0 msgr2=0x5606170c3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 shutdown_connections 2023-07-29T16:20:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.406+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/4273904384 wait complete. 2023-07-29T16:20:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.407+0000 7fd1df8bfb80 20 client.27044 trim_cache size 0 max 16384 2023-07-29T16:20:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.422+0000 7fd1df8bfb80 1 Processor -- start 2023-07-29T16:20:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 -- start start 2023-07-29T16:20:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170d40f0 con 0x5606170cacf0 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170d23b0 con 0x5606170c9df0 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170dc0f0 con 0x560616f256b0 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170d1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42396/0 (socket says 172.21.15.31:42396) 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2176259628 learned_addr learned my addr 172.21.15.31:0/2176259628 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2176259628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 msgr2=0x5606170da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2176259628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 msgr2=0x5606170c3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170c3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.423+0000 7fd1c9d9b700 1 -- 172.21.15.31:0/2176259628 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc009230 con 0x560616f256b0 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.424+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170d1bd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0004800 tx=0x7fd1c0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e8c15a6a3942f44 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.424+0000 7fd1b8ff9700 1 -- 172.21.15.31:0/2176259628 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c00c9070 con 0x560616f256b0 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.424+0000 7fd1b8ff9700 1 -- 172.21.15.31:0/2176259628 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1c0098470 con 0x560616f256b0 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.424+0000 7fd1b8ff9700 1 -- 172.21.15.31:0/2176259628 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c009d120 con 0x560616f256b0 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2176259628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 msgr2=0x5606170d1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170d1bd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0004800 tx=0x7fd1c0000bc0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2176259628 shutdown_connections 2023-07-29T16:20:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606170cacf0 0x5606170c3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560616f256b0 0x5606170d1bd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/2176259628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2176259628 >> 172.21.15.31:0/2176259628 conn(0x5606170befd0 msgr2=0x5606170c04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2176259628 shutdown_connections 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/2176259628 wait complete. 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 -- start start 2023-07-29T16:20:16.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.425+0000 7fd1df8bfb80 1 Processor -- start 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560616f256b0 0x5606170d9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170cacf0 0x5606170d6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170d6ea0 con 0x560616f256b0 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170d7010 con 0x5606170c9df0 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606170d71d0 con 0x5606170cacf0 2023-07-29T16:20:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1df8bfb80 10 client.0 _ll_register_callbacks cb 0x560616fc05c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560616f256b0 0x5606170d9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1c959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1ca59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170cacf0 0x5606170d6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1c9d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560616f256b0 0x5606170d9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50634/0 (socket says 172.21.15.31:50634) 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1c959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57184/0 (socket says 172.21.15.31:57184) 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.426+0000 7fd1c959a700 1 -- 172.21.15.31:0/1849737669 learned_addr learned my addr 172.21.15.31:0/1849737669 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:16.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ca59c700 1 -- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 msgr2=0x5606170da2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ca59c700 1 -- 172.21.15.31:0/1849737669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560616f256b0 msgr2=0x5606170d9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560616f256b0 0x5606170d9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ca59c700 1 -- 172.21.15.31:0/1849737669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c41090c0 con 0x5606170cacf0 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1c959a700 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ca59c700 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170cacf0 0x5606170d6810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc000f80 tx=0x7fd1bc009ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33a1bdf2b0f4b7db server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:16.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/1849737669 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc030070 con 0x5606170cacf0 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/1849737669 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd1bc004a70 con 0x5606170cacf0 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.427+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1bc00f040 con 0x5606170cacf0 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/1849737669 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1bc00e040 con 0x5606170cacf0 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 10 client.27061 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560616e31c00 con 0x5606170cacf0 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/1849737669 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1bc00f040 con 0x5606170cacf0 2023-07-29T16:20:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/1849737669 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fd1bc012030 con 0x5606170cacf0 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 10 client.27061 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 10 client.27061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 20 client.27061 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560617131e00 0x560617134210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 4 client.27061 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560617013ce0 con 0x560617131e00 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.428+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560617131e00 0x560617134210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:16.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.429+0000 7fd1c9d9b700 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560617131e00 0x560617134210 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e0d6647cb372bc3 server_cookie=e0b4364f2be5d876 in_seq=0 out_seq=0 2023-07-29T16:20:16.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.429+0000 7fd1ad7fa700 10 client.27061 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.493+0000 7fd1ad7fa700 1 -- 172.21.15.31:0/1849737669 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560617013ce0 con 0x560617131e00 2023-07-29T16:20:16.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.493+0000 7fd1ad7fa700 10 client.27061 handle_command_reply: tid=0 2023-07-29T16:20:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.493+0000 7fd1df8bfb80 1 client.27061 shutdown 2023-07-29T16:20:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170cacf0 msgr2=0x5606170d6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170cacf0 0x5606170d6810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc000f80 tx=0x7fd1bc009ac0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 shutdown_connections 2023-07-29T16:20:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560617131e00 0x560617134210 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5606170cacf0 0x5606170d6810 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560616f256b0 0x5606170d9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1ad7fa700 0 client.27061 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:16.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 --2- 172.21.15.31:0/1849737669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5606170c9df0 0x5606170da2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 >> 172.21.15.31:0/1849737669 conn(0x5606170befd0 msgr2=0x5606170d1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:16.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 shutdown_connections 2023-07-29T16:20:16.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 1 -- 172.21.15.31:0/1849737669 wait complete. 2023-07-29T16:20:16.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.494+0000 7fd1df8bfb80 20 client.27061 trim_cache size 0 max 16384 2023-07-29T16:20:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:20:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:20:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:20:16.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:20:16.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1b00a0810 msgr2=0x7fd1b00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1b00a0810 0x7fd1b00a2cc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc021000 tx=0x7fd1bc01e000 comp rx=0 tx=0).stop 2023-07-29T16:20:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 msgr2=0x7fd1c4198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 0x7fd1c4198390 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000e970 tx=0x7fd1c000ec80 comp rx=0 tx=0).stop 2023-07-29T16:20:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 shutdown_connections 2023-07-29T16:20:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1b00a0810 0x7fd1b00a2cc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1c4101ed0 0x7fd1c4199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1c4100350 0x7fd1c4198390 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 --2- 172.21.15.31:0/2264407866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1c4101540 0x7fd1c41988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 >> 172.21.15.31:0/2264407866 conn(0x7fd1c40fb970 msgr2=0x7fd1c41047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 shutdown_connections 2023-07-29T16:20:16.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:16.495+0000 7fd1cbfff700 1 -- 172.21.15.31:0/2264407866 wait complete. 2023-07-29T16:20:16.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:16 smithi039 ceph-mon[128858]: pgmap v514: 97 pgs: 97 active+clean; 62 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2023-07-29T16:20:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:16 smithi156 ceph-mon[129245]: pgmap v514: 97 pgs: 97 active+clean; 62 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2023-07-29T16:20:16.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 205985 13.43 MB/sec execute 1 sec latency 59.715 ms 2023-07-29T16:20:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:16 smithi031 ceph-mon[95504]: pgmap v514: 97 pgs: 97 active+clean; 62 MiB data, 887 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 288 op/s 2023-07-29T16:20:17.862 INFO:tasks.workunit.client.0.smithi031.stdout: 1 207369 9.48 MB/sec execute 2 sec latency 11.127 ms 2023-07-29T16:20:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:17 smithi031 ceph-mon[95504]: pgmap v515: 97 pgs: 97 active+clean; 64 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2023-07-29T16:20:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:17 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:20:18.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:17 smithi039 ceph-mon[128858]: pgmap v515: 97 pgs: 97 active+clean; 64 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2023-07-29T16:20:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:17 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:20:18.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:17 smithi156 ceph-mon[129245]: pgmap v515: 97 pgs: 97 active+clean; 64 MiB data, 895 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.2 MiB/s wr, 273 op/s 2023-07-29T16:20:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:17 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:20:18.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 209519 10.85 MB/sec execute 3 sec latency 17.886 ms 2023-07-29T16:20:19.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:18 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:20:19.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:18 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:20:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:18 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:20:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:18 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:20:19.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:18 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:20:19.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:18 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:20:19.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 210713 9.48 MB/sec execute 4 sec latency 43.553 ms 2023-07-29T16:20:20.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:19 smithi031 ceph-mon[95504]: pgmap v516: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 335 op/s 2023-07-29T16:20:20.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:19 smithi039 ceph-mon[128858]: pgmap v516: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 335 op/s 2023-07-29T16:20:20.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:19 smithi156 ceph-mon[129245]: pgmap v516: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 335 op/s 2023-07-29T16:20:20.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 213055 10.33 MB/sec execute 5 sec latency 10.808 ms 2023-07-29T16:20:21.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 214350 9.52 MB/sec execute 6 sec latency 12.419 ms 2023-07-29T16:20:22.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:22 smithi039 ceph-mon[128858]: pgmap v517: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 246 op/s 2023-07-29T16:20:22.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:22 smithi156 ceph-mon[129245]: pgmap v517: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 246 op/s 2023-07-29T16:20:22.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 216631 10.12 MB/sec execute 7 sec latency 18.097 ms 2023-07-29T16:20:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:22 smithi031 ceph-mon[95504]: pgmap v517: 97 pgs: 97 active+clean; 61 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 246 op/s 2023-07-29T16:20:23.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 217685 9.16 MB/sec execute 8 sec latency 12.091 ms 2023-07-29T16:20:24.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:24 smithi039 ceph-mon[128858]: pgmap v518: 97 pgs: 97 active+clean; 63 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 323 op/s 2023-07-29T16:20:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:24 smithi156 ceph-mon[129245]: pgmap v518: 97 pgs: 97 active+clean; 63 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 323 op/s 2023-07-29T16:20:24.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 219933 9.84 MB/sec execute 9 sec latency 30.092 ms 2023-07-29T16:20:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:24 smithi031 ceph-mon[95504]: pgmap v518: 97 pgs: 97 active+clean; 63 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 323 op/s 2023-07-29T16:20:25.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 220879 9.10 MB/sec execute 10 sec latency 17.200 ms 2023-07-29T16:20:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:26 smithi039 ceph-mon[128858]: pgmap v519: 97 pgs: 97 active+clean; 64 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 274 op/s 2023-07-29T16:20:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:26 smithi156 ceph-mon[129245]: pgmap v519: 97 pgs: 97 active+clean; 64 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 274 op/s 2023-07-29T16:20:26.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 223314 9.52 MB/sec execute 11 sec latency 15.798 ms 2023-07-29T16:20:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:26 smithi031 ceph-mon[95504]: pgmap v519: 97 pgs: 97 active+clean; 64 MiB data, 940 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.3 MiB/s wr, 274 op/s 2023-07-29T16:20:27.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 224416 9.18 MB/sec execute 12 sec latency 11.598 ms 2023-07-29T16:20:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:27 smithi031 ceph-mon[95504]: pgmap v520: 97 pgs: 97 active+clean; 66 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 299 op/s 2023-07-29T16:20:28.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:27 smithi039 ceph-mon[128858]: pgmap v520: 97 pgs: 97 active+clean; 66 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 299 op/s 2023-07-29T16:20:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:27 smithi156 ceph-mon[129245]: pgmap v520: 97 pgs: 97 active+clean; 66 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.2 MiB/s wr, 299 op/s 2023-07-29T16:20:28.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 226308 9.26 MB/sec execute 13 sec latency 159.331 ms 2023-07-29T16:20:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.666+0000 7fd63219d700 1 -- 172.21.15.31:0/792595527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c1023a0 msgr2=0x7fd62c1027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.666+0000 7fd63219d700 1 --2- 172.21.15.31:0/792595527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c1023a0 0x7fd62c1027b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd628009990 tx=0x7fd628009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 -- 172.21.15.31:0/792595527 shutdown_connections 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 --2- 172.21.15.31:0/792595527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c103f20 0x7fd62c10a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 --2- 172.21.15.31:0/792595527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c1023a0 0x7fd62c1027b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 --2- 172.21.15.31:0/792595527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd62c103590 0x7fd62c1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 -- 172.21.15.31:0/792595527 >> 172.21.15.31:0/792595527 conn(0x7fd62c0fd9c0 msgr2=0x7fd62c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 -- 172.21.15.31:0/792595527 shutdown_connections 2023-07-29T16:20:29.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.667+0000 7fd63219d700 1 -- 172.21.15.31:0/792595527 wait complete. 2023-07-29T16:20:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 Processor -- start 2023-07-29T16:20:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 -- start start 2023-07-29T16:20:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 0x7fd62c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd62c103590 0x7fd62c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.668 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c103f20 0x7fd62c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c19fc60 con 0x7fd62c103590 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c19fda0 con 0x7fd62c103f20 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63219d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c19ff30 con 0x7fd62c1023a0 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 0x7fd62c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.668+0000 7fd63099a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd62c103590 0x7fd62c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 0x7fd62c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44074/0 (socket says 172.21.15.31:44074) 2023-07-29T16:20:29.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 -- 172.21.15.31:0/4280201069 learned_addr learned my addr 172.21.15.31:0/4280201069 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63199c700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c103f20 0x7fd62c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 -- 172.21.15.31:0/4280201069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c103f20 msgr2=0x7fd62c19f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c103f20 0x7fd62c19f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 -- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd62c103590 msgr2=0x7fd62c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd62c103590 0x7fd62c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 -- 172.21.15.31:0/4280201069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6280095f0 con 0x7fd62c1023a0 2023-07-29T16:20:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd63119b700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 0x7fd62c1980e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd628005740 tx=0x7fd628005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.669+0000 7fd6227fc700 1 -- 172.21.15.31:0/4280201069 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62801d070 con 0x7fd62c1023a0 2023-07-29T16:20:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.670+0000 7fd6227fc700 1 -- 172.21.15.31:0/4280201069 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd628021410 con 0x7fd62c1023a0 2023-07-29T16:20:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.670+0000 7fd6227fc700 1 -- 172.21.15.31:0/4280201069 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628017650 con 0x7fd62c1023a0 2023-07-29T16:20:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.670+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c1a01b0 con 0x7fd62c1023a0 2023-07-29T16:20:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.670+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c1a0580 con 0x7fd62c1023a0 2023-07-29T16:20:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.673+0000 7fd6227fc700 1 -- 172.21.15.31:0/4280201069 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd628021580 con 0x7fd62c1023a0 2023-07-29T16:20:29.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.674+0000 7fd646e06b80 1 Processor -- start 2023-07-29T16:20:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 -- start start 2023-07-29T16:20:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd6227fc700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6180a0790 0x7fd6180a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x557013ffec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013fff150 0x557013fff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013ffff00 0x5570140003a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557013f569b0 con 0x557013fff150 2023-07-29T16:20:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557013f54370 con 0x557013e5a6b0 2023-07-29T16:20:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd646e06b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557013f4fc60 con 0x557013ffff00 2023-07-29T16:20:29.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd6227fc700 1 -- 172.21.15.31:0/4280201069 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd6280ccc30 con 0x7fd62c1023a0 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd63199c700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6180a0790 0x7fd6180a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd63119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x557013ffec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd63119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x557013ffec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55106/0 (socket says 172.21.15.31:55106) 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd63119b700 1 -- 172.21.15.31:0/973741479 learned_addr learned my addr 172.21.15.31:0/973741479 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.675+0000 7fd63099a700 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013fff150 0x557013fff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63199c700 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013ffff00 0x5570140003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63119b700 1 -- 172.21.15.31:0/973741479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013ffff00 msgr2=0x5570140003a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63119b700 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013ffff00 0x5570140003a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63119b700 1 -- 172.21.15.31:0/973741479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013fff150 msgr2=0x557013fff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63119b700 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013fff150 0x557013fff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63119b700 1 -- 172.21.15.31:0/973741479 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557013d7dd80 con 0x557013e5a6b0 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63119b700 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x557013ffec10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd628096860 tx=0x7fd62809a1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c65adb4dd47f68da server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd63199c700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6180a0790 0x7fd6180a2c40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd624009ec0 tx=0x7fd624009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd6177fe700 1 -- 172.21.15.31:0/973741479 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62806abc0 con 0x557013e5a6b0 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd6177fe700 1 -- 172.21.15.31:0/973741479 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd62806d040 con 0x557013e5a6b0 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.676+0000 7fd6177fe700 1 -- 172.21.15.31:0/973741479 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280987d0 con 0x557013e5a6b0 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 -- 172.21.15.31:0/973741479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 msgr2=0x557013ffec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x557013ffec10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd628096860 tx=0x7fd62809a1b0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 -- 172.21.15.31:0/973741479 shutdown_connections 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013ffff00 0x5570140003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x557013ffec10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 --2- 172.21.15.31:0/973741479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013fff150 0x557013fff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 -- 172.21.15.31:0/973741479 >> 172.21.15.31:0/973741479 conn(0x557013ff4380 msgr2=0x557013ff96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 -- 172.21.15.31:0/973741479 shutdown_connections 2023-07-29T16:20:29.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.677+0000 7fd646e06b80 1 -- 172.21.15.31:0/973741479 wait complete. 2023-07-29T16:20:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.678+0000 7fd646e06b80 1 -- start start 2023-07-29T16:20:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.678+0000 7fd646e06b80 1 Processor -- start 2023-07-29T16:20:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013e5a6b0 0x557014007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 0x5570140081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x55701400f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557013eec4b0 con 0x557013ffff00 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557013eec150 con 0x557013fff150 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557013dd38c0 con 0x557013e5a6b0 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd646e06b80 10 client.0 _ll_register_callbacks cb 0x557013ef55c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 0x5570140081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 0x5570140081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55122/0 (socket says 172.21.15.31:55122) 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 -- 172.21.15.31:0/994827261 learned_addr learned my addr 172.21.15.31:0/994827261 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63119b700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013e5a6b0 0x557014007ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63199c700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x55701400f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 -- 172.21.15.31:0/994827261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013e5a6b0 msgr2=0x557014007ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013e5a6b0 0x557014007ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 -- 172.21.15.31:0/994827261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 msgr2=0x55701400f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x55701400f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 -- 172.21.15.31:0/994827261 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c1a0580 con 0x557013fff150 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd63099a700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 0x5570140081e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd61c012760 tx=0x7fd61c012a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dd4e10d8eb3984d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:29.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.679+0000 7fd614ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd614ff9700 1 -- 172.21.15.31:0/994827261 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c017070 con 0x557013fff150 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557013f7acb0 con 0x557013fff150 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd614ff9700 1 -- 172.21.15.31:0/994827261 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd61c00f480 con 0x557013fff150 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd614ff9700 1 -- 172.21.15.31:0/994827261 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd61c0122e0 con 0x557013fff150 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd646e06b80 10 client.27062 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557013d66c00 con 0x557013fff150 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd614ff9700 1 -- 172.21.15.31:0/994827261 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557013f7acb0 con 0x557013fff150 2023-07-29T16:20:29.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd614ff9700 1 -- 172.21.15.31:0/994827261 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fd61c026860 con 0x557013fff150 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd646e06b80 10 client.27062 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd646e06b80 10 client.27062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.680+0000 7fd646e06b80 20 client.27062 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.681+0000 7fd646e06b80 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014013b20 0x557014015f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.681+0000 7fd646e06b80 4 client.27062 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.681+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557013fcdd10 con 0x557014013b20 2023-07-29T16:20:29.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.681+0000 7fd63119b700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014013b20 0x557014015f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.681+0000 7fd63119b700 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014013b20 0x557014015f30 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e4ffbbe50f1f35f server_cookie=fd73ed6e1606038f in_seq=0 out_seq=0 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.681+0000 7fd614ff9700 10 client.27062 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd614ff9700 1 -- 172.21.15.31:0/994827261 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557013fcdd10 con 0x557014013b20 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd614ff9700 10 client.27062 handle_command_reply: tid=0 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 client.27062 shutdown 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 msgr2=0x5570140081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 0x5570140081e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd61c012760 tx=0x7fd61c012a70 comp rx=0 tx=0).stop 2023-07-29T16:20:29.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 shutdown_connections 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014013b20 0x557014015f30 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x55701400f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd614ff9700 0 client.27062 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013e5a6b0 0x557014007ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 --2- 172.21.15.31:0/994827261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013fff150 0x5570140081e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 >> 172.21.15.31:0/994827261 conn(0x557013ff4380 msgr2=0x557013ff8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:29.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 shutdown_connections 2023-07-29T16:20:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.682+0000 7fd646e06b80 1 -- 172.21.15.31:0/994827261 wait complete. 2023-07-29T16:20:29.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.683+0000 7fd646e06b80 20 client.27062 trim_cache size 0 max 16384 2023-07-29T16:20:29.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.698+0000 7fd646e06b80 1 Processor -- start 2023-07-29T16:20:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 -- start start 2023-07-29T16:20:29.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x55701400d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x557014002b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x5570140030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557014007150 con 0x557013ffff00 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557014007810 con 0x557013e5a6b0 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd646e06b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557014010ec0 con 0x557013fff150 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x55701400d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63099a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x557014002b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x55701400d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55124/0 (socket says 172.21.15.31:55124) 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 -- 172.21.15.31:0/1069088022 learned_addr learned my addr 172.21.15.31:0/1069088022 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63199c700 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x5570140030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 -- 172.21.15.31:0/1069088022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 msgr2=0x557014002b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x557014002b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 -- 172.21.15.31:0/1069088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 msgr2=0x5570140030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x5570140030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63119b700 1 -- 172.21.15.31:0/1069088022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557013da41f0 con 0x557013e5a6b0 2023-07-29T16:20:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.699+0000 7fd63199c700 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x5570140030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.700+0000 7fd63119b700 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x55701400d130 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd628096dd0 tx=0x7fd6280092b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d7eba93001ce00d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.700+0000 7fd617fff700 1 -- 172.21.15.31:0/1069088022 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628098700 con 0x557013e5a6b0 2023-07-29T16:20:29.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.700+0000 7fd617fff700 1 -- 172.21.15.31:0/1069088022 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6280ca780 con 0x557013e5a6b0 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.700+0000 7fd617fff700 1 -- 172.21.15.31:0/1069088022 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62809a890 con 0x557013e5a6b0 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 -- 172.21.15.31:0/1069088022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 msgr2=0x55701400d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x55701400d130 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd628096dd0 tx=0x7fd6280092b0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 -- 172.21.15.31:0/1069088022 shutdown_connections 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013ffff00 0x5570140030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013e5a6b0 0x55701400d130 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1069088022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x557014002b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 -- 172.21.15.31:0/1069088022 >> 172.21.15.31:0/1069088022 conn(0x557013ff4380 msgr2=0x557014007ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 -- 172.21.15.31:0/1069088022 shutdown_connections 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 -- 172.21.15.31:0/1069088022 wait complete. 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 -- start start 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.701+0000 7fd646e06b80 1 Processor -- start 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013e5a6b0 0x55701400c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x55701400cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013ffff00 0x55701400b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701400b5d0 con 0x557013e5a6b0 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701400b740 con 0x557013ffff00 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55701400b900 con 0x557013fff150 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd646e06b80 10 client.0 _ll_register_callbacks cb 0x557013ef55c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013ffff00 0x55701400b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x55701400cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x55701400cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44116/0 (socket says 172.21.15.31:44116) 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 -- 172.21.15.31:0/1159929285 learned_addr learned my addr 172.21.15.31:0/1159929285 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63119b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013e5a6b0 0x55701400c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013ffff00 0x55701400b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55140/0 (socket says 172.21.15.31:55140) 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 -- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013ffff00 msgr2=0x55701400b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013ffff00 0x55701400b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 -- 172.21.15.31:0/1159929285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013e5a6b0 msgr2=0x55701400c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013e5a6b0 0x55701400c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.702+0000 7fd63099a700 1 -- 172.21.15.31:0/1159929285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557013d7dd80 con 0x557013fff150 2023-07-29T16:20:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd63099a700 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x55701400cf30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd61c000c00 tx=0x7fd61c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b80fc4a2c6310a1c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd614ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd614ff9700 1 -- 172.21.15.31:0/1159929285 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c017070 con 0x557013fff150 2023-07-29T16:20:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd614ff9700 1 -- 172.21.15.31:0/1159929285 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd61c02b070 con 0x557013fff150 2023-07-29T16:20:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd62806d040 con 0x557013fff150 2023-07-29T16:20:29.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd614ff9700 1 -- 172.21.15.31:0/1159929285 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd61c0269f0 con 0x557013fff150 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd646e06b80 10 client.27079 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.703+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557013d7c250 con 0x557013fff150 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd614ff9700 1 -- 172.21.15.31:0/1159929285 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd62806d040 con 0x557013fff150 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd614ff9700 1 -- 172.21.15.31:0/1159929285 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fd61c0104d0 con 0x557013fff150 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd646e06b80 10 client.27079 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd646e06b80 10 client.27079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:29.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd646e06b80 20 client.27079 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014066950 0x557014068d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd646e06b80 4 client.27079 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:20:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557013f48ce0 con 0x557014066950 2023-07-29T16:20:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.704+0000 7fd63119b700 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014066950 0x557014068d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:29.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.705+0000 7fd63119b700 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014066950 0x557014068d60 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=475fa9b2157e7343 server_cookie=9dd888c7ee984f44 in_seq=0 out_seq=0 2023-07-29T16:20:29.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.705+0000 7fd614ff9700 10 client.27079 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:29.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd614ff9700 1 -- 172.21.15.31:0/1159929285 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557013f48ce0 con 0x557014066950 2023-07-29T16:20:29.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd614ff9700 10 client.27079 handle_command_reply: tid=0 2023-07-29T16:20:29.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 client.27079 shutdown 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 msgr2=0x55701400cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x55701400cf30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd61c000c00 tx=0x7fd61c005270 comp rx=0 tx=0).stop 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 shutdown_connections 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557014066950 0x557014068d60 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557013ffff00 0x55701400b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd614ff9700 0 client.27079 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:29.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557013e5a6b0 0x55701400c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 --2- 172.21.15.31:0/1159929285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557013fff150 0x55701400cf30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.776+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 >> 172.21.15.31:0/1159929285 conn(0x557013ff4380 msgr2=0x557014007c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.777+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 shutdown_connections 2023-07-29T16:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.777+0000 7fd646e06b80 1 -- 172.21.15.31:0/1159929285 wait complete. 2023-07-29T16:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.777+0000 7fd646e06b80 20 client.27079 trim_cache size 0 max 16384 2023-07-29T16:20:29.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:20:29.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:20:29.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:20:29.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6180a0790 msgr2=0x7fd6180a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6180a0790 0x7fd6180a2c40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd624009ec0 tx=0x7fd624009920 comp rx=0 tx=0).stop 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 msgr2=0x7fd62c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 0x7fd62c1980e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd628005740 tx=0x7fd628005770 comp rx=0 tx=0).stop 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 shutdown_connections 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd6180a0790 0x7fd6180a2c40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd62c103f20 0x7fd62c19f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd62c1023a0 0x7fd62c1980e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 --2- 172.21.15.31:0/4280201069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd62c103590 0x7fd62c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 >> 172.21.15.31:0/4280201069 conn(0x7fd62c0fd9c0 msgr2=0x7fd62c0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 shutdown_connections 2023-07-29T16:20:29.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:29.778+0000 7fd63219d700 1 -- 172.21.15.31:0/4280201069 wait complete. 2023-07-29T16:20:29.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 227494 9.18 MB/sec execute 14 sec latency 63.523 ms 2023-07-29T16:20:30.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:30 smithi039 ceph-mon[128858]: pgmap v521: 97 pgs: 97 active+clean; 66 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 282 op/s 2023-07-29T16:20:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:30 smithi156 ceph-mon[129245]: pgmap v521: 97 pgs: 97 active+clean; 66 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 282 op/s 2023-07-29T16:20:30.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 229510 9.05 MB/sec execute 15 sec latency 27.550 ms 2023-07-29T16:20:30.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:30 smithi031 ceph-mon[95504]: pgmap v521: 97 pgs: 97 active+clean; 66 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 282 op/s 2023-07-29T16:20:31.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 231023 9.23 MB/sec execute 16 sec latency 14.247 ms 2023-07-29T16:20:32.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:32 smithi039 ceph-mon[128858]: pgmap v522: 97 pgs: 97 active+clean; 66 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2023-07-29T16:20:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:32 smithi156 ceph-mon[129245]: pgmap v522: 97 pgs: 97 active+clean; 66 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2023-07-29T16:20:32.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 232964 9.09 MB/sec execute 17 sec latency 11.415 ms 2023-07-29T16:20:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:32 smithi031 ceph-mon[95504]: pgmap v522: 97 pgs: 97 active+clean; 66 MiB data, 993 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 206 op/s 2023-07-29T16:20:33.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 234488 9.26 MB/sec execute 18 sec latency 71.722 ms 2023-07-29T16:20:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:34 smithi039 ceph-mon[128858]: pgmap v523: 97 pgs: 97 active+clean; 64 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 288 op/s 2023-07-29T16:20:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:34 smithi156 ceph-mon[129245]: pgmap v523: 97 pgs: 97 active+clean; 64 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 288 op/s 2023-07-29T16:20:34.863 INFO:tasks.workunit.client.0.smithi031.stdout: 1 235839 9.05 MB/sec execute 19 sec latency 50.128 ms 2023-07-29T16:20:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:34 smithi031 ceph-mon[95504]: pgmap v523: 97 pgs: 97 active+clean; 64 MiB data, 1001 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 288 op/s 2023-07-29T16:20:35.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 237942 9.28 MB/sec execute 20 sec latency 17.331 ms 2023-07-29T16:20:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:36 smithi039 ceph-mon[128858]: pgmap v524: 97 pgs: 97 active+clean; 65 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 285 op/s 2023-07-29T16:20:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:36 smithi156 ceph-mon[129245]: pgmap v524: 97 pgs: 97 active+clean; 65 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 285 op/s 2023-07-29T16:20:36.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 239106 9.09 MB/sec execute 21 sec latency 85.138 ms 2023-07-29T16:20:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:36 smithi031 ceph-mon[95504]: pgmap v524: 97 pgs: 97 active+clean; 65 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 285 op/s 2023-07-29T16:20:37.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 241236 9.24 MB/sec execute 22 sec latency 56.495 ms 2023-07-29T16:20:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:37 smithi031 ceph-mon[95504]: pgmap v525: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 263 op/s 2023-07-29T16:20:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:37 smithi039 ceph-mon[128858]: pgmap v525: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 263 op/s 2023-07-29T16:20:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:37 smithi156 ceph-mon[129245]: pgmap v525: 97 pgs: 97 active+clean; 66 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 263 op/s 2023-07-29T16:20:38.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 242306 9.00 MB/sec execute 23 sec latency 11.698 ms 2023-07-29T16:20:39.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 244445 9.10 MB/sec execute 24 sec latency 47.666 ms 2023-07-29T16:20:40.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:40 smithi039 ceph-mon[128858]: pgmap v526: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 285 op/s 2023-07-29T16:20:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:40 smithi156 ceph-mon[129245]: pgmap v526: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 285 op/s 2023-07-29T16:20:40.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 245577 8.98 MB/sec execute 25 sec latency 25.772 ms 2023-07-29T16:20:40.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:40 smithi031 ceph-mon[95504]: pgmap v526: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 285 op/s 2023-07-29T16:20:41.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 247961 9.13 MB/sec execute 26 sec latency 14.804 ms 2023-07-29T16:20:42.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:20:42.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 -- 172.21.15.31:0/3973116090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 msgr2=0x7f9d0810a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 --2- 172.21.15.31:0/3973116090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0810a990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000b0a0 tx=0x7f9d0000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 -- 172.21.15.31:0/3973116090 shutdown_connections 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 --2- 172.21.15.31:0/3973116090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0810a990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 --2- 172.21.15.31:0/3973116090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08102550 0x7f9d08102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 --2- 172.21.15.31:0/3973116090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d08103740 0x7f9d08103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 -- 172.21.15.31:0/3973116090 >> 172.21.15.31:0/3973116090 conn(0x7f9d080fdb90 msgr2=0x7f9d080fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 -- 172.21.15.31:0/3973116090 shutdown_connections 2023-07-29T16:20:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.262+0000 7f9d0edff700 1 -- 172.21.15.31:0/3973116090 wait complete. 2023-07-29T16:20:42.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.263+0000 7f9d0edff700 1 Processor -- start 2023-07-29T16:20:42.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.263+0000 7f9d0edff700 1 -- start start 2023-07-29T16:20:42.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0edff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d08102550 0x7f9d081981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0edff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 0x7f9d081986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0edff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0edff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0819fd30 con 0x7f9d08102550 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0edff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0819fe70 con 0x7f9d08103740 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0edff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d081a0000 con 0x7f9d081040d0 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d07fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 0x7f9d081986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d07fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 0x7f9d081986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55444/0 (socket says 172.21.15.31:55444) 2023-07-29T16:20:42.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d08102550 0x7f9d081981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d07fff700 1 -- 172.21.15.31:0/457490211 learned_addr learned my addr 172.21.15.31:0/457490211 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.264+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d07fff700 1 -- 172.21.15.31:0/457490211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 msgr2=0x7f9d0819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d07fff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d07fff700 1 -- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d08102550 msgr2=0x7f9d081981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d07fff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d08102550 0x7f9d081981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d07fff700 1 -- 172.21.15.31:0/457490211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d00009d00 con 0x7f9d08103740 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d07fff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 0x7f9d081986f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e9a0 tx=0x7f9cf400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d05ffb700 1 -- 172.21.15.31:0/457490211 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4009790 con 0x7f9d08103740 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d05ffb700 1 -- 172.21.15.31:0/457490211 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cf4010430 con 0x7f9d08103740 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d081a02e0 con 0x7f9d08103740 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.265+0000 7f9d05ffb700 1 -- 172.21.15.31:0/457490211 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4018680 con 0x7f9d08103740 2023-07-29T16:20:42.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.266+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081a0840 con 0x7f9d08103740 2023-07-29T16:20:42.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.270+0000 7f9d05ffb700 1 -- 172.21.15.31:0/457490211 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9cf40105a0 con 0x7f9d08103740 2023-07-29T16:20:42.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.270+0000 7f9d22639b80 1 Processor -- start 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 -- start start 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d3be0 0x5631035d3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035d4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d52f0 0x5631035d5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310352ba10 con 0x5631035d3be0 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631035293d0 con 0x5631035d52f0 2023-07-29T16:20:42.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d22639b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563103524cc0 con 0x5631035d4530 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.271+0000 7f9d05ffb700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9cf80a0810 0x7f9cf80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035d4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035d4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48390/0 (socket says 172.21.15.31:48390) 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/2511935815 learned_addr learned my addr 172.21.15.31:0/2511935815 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d0d39c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d3be0 0x5631035d3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d52f0 0x5631035d5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d05ffb700 1 -- 172.21.15.31:0/457490211 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9cf4014070 con 0x7f9d08103740 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 -- 172.21.15.31:0/2511935815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 msgr2=0x5631035d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035d4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 -- 172.21.15.31:0/2511935815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d3be0 msgr2=0x5631035d3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d3be0 0x5631035d3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9cf80a0810 0x7f9cf80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 -- 172.21.15.31:0/2511935815 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563103352de0 con 0x5631035d52f0 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.272+0000 7f9d07fff700 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d52f0 0x5631035d5740 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e970 tx=0x7f9cf4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1598ce086f9a3fc3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.273+0000 7f9cf2ffd700 1 -- 172.21.15.31:0/2511935815 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf40ae070 con 0x5631035d52f0 2023-07-29T16:20:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.273+0000 7f9cf2ffd700 1 -- 172.21.15.31:0/2511935815 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cf4098d90 con 0x5631035d52f0 2023-07-29T16:20:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.273+0000 7f9cf2ffd700 1 -- 172.21.15.31:0/2511935815 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf409d080 con 0x5631035d52f0 2023-07-29T16:20:42.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.273+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9cf80a0810 0x7f9cf80a2cc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000b780 tx=0x7f9d0000b3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.273+0000 7f9d22639b80 1 -- 172.21.15.31:0/2511935815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d52f0 msgr2=0x5631035d5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.273+0000 7f9d22639b80 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d52f0 0x5631035d5740 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e970 tx=0x7f9cf4014000 comp rx=0 tx=0).stop 2023-07-29T16:20:42.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 -- 172.21.15.31:0/2511935815 shutdown_connections 2023-07-29T16:20:42.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d52f0 0x5631035d5740 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d3be0 0x5631035d3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 --2- 172.21.15.31:0/2511935815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035d4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 -- 172.21.15.31:0/2511935815 >> 172.21.15.31:0/2511935815 conn(0x5631035c93e0 msgr2=0x5631035ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 -- 172.21.15.31:0/2511935815 shutdown_connections 2023-07-29T16:20:42.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 -- 172.21.15.31:0/2511935815 wait complete. 2023-07-29T16:20:42.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.274+0000 7f9d22639b80 1 -- start start 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 Processor -- start 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 0x5631035dd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d4530 0x5631035ddd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d52f0 0x5631035e4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631034c1510 con 0x5631035d4530 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631034c11b0 con 0x5631035d3be0 2023-07-29T16:20:42.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631033a8920 con 0x5631035d52f0 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d22639b80 10 client.0 _ll_register_callbacks cb 0x5631034ca620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.275+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 0x5631035dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 0x5631035dd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55466/0 (socket says 172.21.15.31:55466) 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3349005931 learned_addr learned my addr 172.21.15.31:0/3349005931 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d07fff700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d4530 0x5631035ddd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d52f0 0x5631035e4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3349005931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d52f0 msgr2=0x5631035e4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d52f0 0x5631035e4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3349005931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d4530 msgr2=0x5631035ddd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d4530 0x5631035ddd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3349005931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d081a0840 con 0x5631035d3be0 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d07fff700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d4530 0x5631035ddd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 0x5631035dd840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0126e0 tx=0x7f9cfc0129f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e73a4032a5b4256a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9ce7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:42.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9ce7fff700 1 -- 172.21.15.31:0/3349005931 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc017070 con 0x5631035d3be0 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.276+0000 7f9ce7fff700 1 -- 172.21.15.31:0/3349005931 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9cfc004050 con 0x5631035d3be0 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56310354fd10 con 0x5631035d3be0 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9ce7fff700 1 -- 172.21.15.31:0/3349005931 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cfc004210 con 0x5631035d3be0 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9d22639b80 10 client.27080 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x56310333bc60 con 0x5631035d3be0 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9ce7fff700 1 -- 172.21.15.31:0/3349005931 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56310354fd10 con 0x5631035d3be0 2023-07-29T16:20:42.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9ce7fff700 1 -- 172.21.15.31:0/3349005931 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9cfc00f510 con 0x5631035d3be0 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9d22639b80 10 client.27080 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9d22639b80 10 client.27080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.277+0000 7f9d22639b80 20 client.27080 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.278+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631035e9660 0x5631035eba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.278+0000 7f9d22639b80 4 client.27080 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.278+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631035a2d70 con 0x5631035e9660 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.278+0000 7f9d07fff700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631035e9660 0x5631035eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.278+0000 7f9ce7fff700 10 client.27080 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.278+0000 7f9d07fff700 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631035e9660 0x5631035eba70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31b549c72328decf server_cookie=593215d069a561dc in_seq=0 out_seq=0 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9ce7fff700 1 -- 172.21.15.31:0/3349005931 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5631035a2d70 con 0x5631035e9660 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9ce7fff700 10 client.27080 handle_command_reply: tid=0 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 client.27080 shutdown 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 msgr2=0x5631035dd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 0x5631035dd840 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc0126e0 tx=0x7f9cfc0129f0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 shutdown_connections 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5631035e9660 0x5631035eba70 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d52f0 0x5631035e4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d3be0 0x5631035dd840 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3349005931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d4530 0x5631035ddd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9ce7fff700 0 client.27080 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 >> 172.21.15.31:0/3349005931 conn(0x5631035c93e0 msgr2=0x5631035ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 shutdown_connections 2023-07-29T16:20:42.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.279+0000 7f9d22639b80 1 -- 172.21.15.31:0/3349005931 wait complete. 2023-07-29T16:20:42.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.280+0000 7f9d22639b80 20 client.27080 trim_cache size 0 max 16384 2023-07-29T16:20:42.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 Processor -- start 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 -- start start 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035dc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d56c0 0x5631035db760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631035e9380 con 0x5631035d52f0 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631035dc210 con 0x5631035d56c0 2023-07-29T16:20:42.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.298+0000 7f9d22639b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631035d0380 con 0x5631035d4530 2023-07-29T16:20:42.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035dc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d07fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035dc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48408/0 (socket says 172.21.15.31:48408) 2023-07-29T16:20:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3229624127 learned_addr learned my addr 172.21.15.31:0/3229624127 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d56c0 0x5631035db760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3229624127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d56c0 msgr2=0x5631035db760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d56c0 0x5631035db760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3229624127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 msgr2=0x5631035e4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/3229624127 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563103379250 con 0x5631035d4530 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d07fff700 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.299+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035dc4e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00f8c0 tx=0x7f9cfc009480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=171e92f9d40cc3a2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.300+0000 7f9cf37fe700 1 -- 172.21.15.31:0/3229624127 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc017040 con 0x5631035d4530 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.300+0000 7f9cf37fe700 1 -- 172.21.15.31:0/3229624127 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cfc00fe90 con 0x5631035d4530 2023-07-29T16:20:42.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.300+0000 7f9cf37fe700 1 -- 172.21.15.31:0/3229624127 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0038e0 con 0x5631035d4530 2023-07-29T16:20:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.300+0000 7f9d22639b80 1 -- 172.21.15.31:0/3229624127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 msgr2=0x5631035dc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.300+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035dc4e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00f8c0 tx=0x7f9cfc009480 comp rx=0 tx=0).stop 2023-07-29T16:20:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 -- 172.21.15.31:0/3229624127 shutdown_connections 2023-07-29T16:20:42.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d56c0 0x5631035db760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d4530 0x5631035dc4e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 --2- 172.21.15.31:0/3229624127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 -- 172.21.15.31:0/3229624127 >> 172.21.15.31:0/3229624127 conn(0x5631035c93e0 msgr2=0x5631035cf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 -- 172.21.15.31:0/3229624127 shutdown_connections 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 -- 172.21.15.31:0/3229624127 wait complete. 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.301+0000 7f9d22639b80 1 -- start start 2023-07-29T16:20:42.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 Processor -- start 2023-07-29T16:20:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 0x5631035e3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d56c0 0x5631035e41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310363b310 con 0x5631035d52f0 2023-07-29T16:20:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310363b480 con 0x5631035d4530 2023-07-29T16:20:42.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56310363b640 con 0x5631035d56c0 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.302+0000 7f9d22639b80 10 client.0 _ll_register_callbacks cb 0x5631034ca620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 0x5631035e3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 0x5631035e3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55470/0 (socket says 172.21.15.31:55470) 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/1683352186 learned_addr learned my addr 172.21.15.31:0/1683352186 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d07fff700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d56c0 0x5631035e41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/1683352186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d56c0 msgr2=0x5631035e41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d56c0 0x5631035e41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/1683352186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 msgr2=0x5631035e3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 -- 172.21.15.31:0/1683352186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563103352de0 con 0x5631035d4530 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d07fff700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0d39c700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d56c0 0x5631035e41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9d0cb9b700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 0x5631035e3770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00f990 tx=0x7f9cfc00fc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e142bec0450e3e9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.303+0000 7f9ce7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9ce7fff700 1 -- 172.21.15.31:0/1683352186 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc017040 con 0x5631035d4530 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9ce7fff700 1 -- 172.21.15.31:0/1683352186 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9cfc003780 con 0x5631035d4530 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9cf4098d90 con 0x5631035d4530 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9ce7fff700 1 -- 172.21.15.31:0/1683352186 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cfc0054c0 con 0x5631035d4530 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 10 client.27086 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f9cfc021040 con 0x5631035d4530 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9ce7fff700 1 -- 172.21.15.31:0/1683352186 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cf4098d90 con 0x5631035d4530 2023-07-29T16:20:42.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9ce7fff700 1 -- 172.21.15.31:0/1683352186 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f9cfc02fb60 con 0x5631035d4530 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 10 client.27086 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 10 client.27086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 20 client.27086 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56310363cd20 0x56310363f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 4 client.27086 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.304+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56310351dd40 con 0x56310363cd20 2023-07-29T16:20:42.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.305+0000 7f9d07fff700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56310363cd20 0x56310363f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.305+0000 7f9ce7fff700 10 client.27086 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.305+0000 7f9d07fff700 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56310363cd20 0x56310363f130 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc0845d2d6229a2e server_cookie=efcfbd9ca01ac0b in_seq=0 out_seq=0 2023-07-29T16:20:42.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.305+0000 7f9ce7fff700 1 -- 172.21.15.31:0/1683352186 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56310351dd40 con 0x56310363cd20 2023-07-29T16:20:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.305+0000 7f9ce7fff700 10 client.27086 handle_command_reply: tid=0 2023-07-29T16:20:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.305+0000 7f9d22639b80 1 client.27086 shutdown 2023-07-29T16:20:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 msgr2=0x5631035e3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 0x5631035e3770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00f990 tx=0x7f9cfc00fc90 comp rx=0 tx=0).stop 2023-07-29T16:20:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 shutdown_connections 2023-07-29T16:20:42.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56310363cd20 0x56310363f130 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631035d56c0 0x5631035e41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5631035d4530 0x5631035e3770 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 --2- 172.21.15.31:0/1683352186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5631035d52f0 0x5631035e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9ce7fff700 0 client.27086 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 >> 172.21.15.31:0/1683352186 conn(0x5631035c93e0 msgr2=0x5631035ce060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 shutdown_connections 2023-07-29T16:20:42.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 1 -- 172.21.15.31:0/1683352186 wait complete. 2023-07-29T16:20:42.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.306+0000 7f9d22639b80 20 client.27086 trim_cache size 0 max 16384 2023-07-29T16:20:42.313 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:20:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:20:42.313 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:20:42.313 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:20:42.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9cf80a0810 msgr2=0x7f9cf80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9cf80a0810 0x7f9cf80a2cc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000b780 tx=0x7f9d0000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 msgr2=0x7f9d081986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 0x7f9d081986f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400e9a0 tx=0x7f9cf400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 shutdown_connections 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9cf80a0810 0x7f9cf80a2cc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9d081040d0 0x7f9d0819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d08102550 0x7f9d081981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 --2- 172.21.15.31:0/457490211 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9d08103740 0x7f9d081986f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 >> 172.21.15.31:0/457490211 conn(0x7f9d080fdb90 msgr2=0x7f9d08106bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 shutdown_connections 2023-07-29T16:20:42.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.307+0000 7f9d0edff700 1 -- 172.21.15.31:0/457490211 wait complete. 2023-07-29T16:20:42.324 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:20:42.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:20:42.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 -- 172.21.15.31:0/3727699620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100737b0 msgr2=0x7f1b10073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 --2- 172.21.15.31:0/3727699620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100737b0 0x7f1b10073c20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009990 tx=0x7f1b00009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 -- 172.21.15.31:0/3727699620 shutdown_connections 2023-07-29T16:20:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 --2- 172.21.15.31:0/3727699620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b10108200 0x7f1b1010a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 --2- 172.21.15.31:0/3727699620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b100731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 --2- 172.21.15.31:0/3727699620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100737b0 0x7f1b10073c20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 -- 172.21.15.31:0/3727699620 >> 172.21.15.31:0/3727699620 conn(0x7f1b100fb920 msgr2=0x7f1b100fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 -- 172.21.15.31:0/3727699620 shutdown_connections 2023-07-29T16:20:42.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.550+0000 7f1b17bff700 1 -- 172.21.15.31:0/3727699620 wait complete. 2023-07-29T16:20:42.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.551+0000 7f1b17bff700 1 Processor -- start 2023-07-29T16:20:42.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 -- start start 2023-07-29T16:20:42.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b100737b0 0x7f1b101982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b10198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b10108200 0x7f1b1019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1019fe40 con 0x7f1b100737b0 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1019ff80 con 0x7f1b10074d90 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b17bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b101a0110 con 0x7f1b10108200 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b1599b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b100737b0 0x7f1b101982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b1519a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b10198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b1599b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b100737b0 0x7f1b101982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43386/0 (socket says 172.21.15.31:43386) 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b1519a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b10198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55490/0 (socket says 172.21.15.31:55490) 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b1599b700 1 -- 172.21.15.31:0/797919787 learned_addr learned my addr 172.21.15.31:0/797919787 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.552+0000 7f1b1619c700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b10108200 0x7f1b1019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1519a700 1 -- 172.21.15.31:0/797919787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b10108200 msgr2=0x7f1b1019f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1519a700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b10108200 0x7f1b1019f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1519a700 1 -- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b100737b0 msgr2=0x7f1b101982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1519a700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b100737b0 0x7f1b101982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1519a700 1 -- 172.21.15.31:0/797919787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000095f0 con 0x7f1b10074d90 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1619c700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b10108200 0x7f1b1019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b1519a700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b10198800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1b000098e0 tx=0x7f1b00004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0001d070 con 0x7f1b10074d90 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b00005020 con 0x7f1b10074d90 2023-07-29T16:20:42.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b101a0390 con 0x7f1b10074d90 2023-07-29T16:20:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00004b00 con 0x7f1b10074d90 2023-07-29T16:20:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.553+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b101a07b0 con 0x7f1b10074d90 2023-07-29T16:20:42.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.554+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b101068b0 con 0x7f1b10074d90 2023-07-29T16:20:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.557+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1b00005190 con 0x7f1b10074d90 2023-07-29T16:20:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.558+0000 7f1b06ffd700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1afc0a0860 0x7f1afc0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.558+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1b000ccdb0 con 0x7f1b10074d90 2023-07-29T16:20:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.559+0000 7f1b1599b700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1afc0a0860 0x7f1afc0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.559+0000 7f1b1599b700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1afc0a0860 0x7f1afc0a2d10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1b08005950 tx=0x7f1b08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.561+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b00096820 con 0x7f1b10074d90 2023-07-29T16:20:42.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:42 smithi031 ceph-mon[95504]: pgmap v527: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 244 op/s 2023-07-29T16:20:42.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:20:42.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:42 smithi039 ceph-mon[128858]: pgmap v527: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 244 op/s 2023-07-29T16:20:42.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:20:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:42 smithi156 ceph-mon[129245]: pgmap v527: 97 pgs: 97 active+clean; 68 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 244 op/s 2023-07-29T16:20:42.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:20:42.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:20:42.836 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:18:33.432992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:20:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.834+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b1002cc30 con 0x7f1b10074d90 2023-07-29T16:20:42.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.835+0000 7f1b06ffd700 1 -- 172.21.15.31:0/797919787 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10777 (secure 0 0 0) 0x7f1b00099e40 con 0x7f1b10074d90 2023-07-29T16:20:42.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.840+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1afc0a0860 msgr2=0x7f1afc0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.840+0000 7f1b17bff700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1afc0a0860 0x7f1afc0a2d10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1b08005950 tx=0x7f1b08009260 comp rx=0 tx=0).stop 2023-07-29T16:20:42.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 msgr2=0x7f1b10198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b10198800 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1b000098e0 tx=0x7f1b00004730 comp rx=0 tx=0).stop 2023-07-29T16:20:42.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 shutdown_connections 2023-07-29T16:20:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1afc0a0860 0x7f1afc0a2d10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b10108200 0x7f1b1019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b100737b0 0x7f1b101982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 --2- 172.21.15.31:0/797919787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1b10074d90 0x7f1b10198800 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 >> 172.21.15.31:0/797919787 conn(0x7f1b100fb920 msgr2=0x7f1b100fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 shutdown_connections 2023-07-29T16:20:42.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:42.841+0000 7f1b17bff700 1 -- 172.21.15.31:0/797919787 wait complete. 2023-07-29T16:20:42.842 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2023-07-29T16:20:42.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:20:42.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 248779 9.01 MB/sec execute 27 sec latency 22.382 ms 2023-07-29T16:20:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 -- 172.21.15.31:0/801627133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c103750 msgr2=0x7ff15c103ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 --2- 172.21.15.31:0/801627133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c103750 0x7ff15c103ba0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff148013ef0 tx=0x7ff1480097f0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 -- 172.21.15.31:0/801627133 shutdown_connections 2023-07-29T16:20:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 --2- 172.21.15.31:0/801627133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff15c1040e0 0x7ff15c10a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 --2- 172.21.15.31:0/801627133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff15c102560 0x7ff15c102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 --2- 172.21.15.31:0/801627133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c103750 0x7ff15c103ba0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.954+0000 7ff162dff700 1 -- 172.21.15.31:0/801627133 >> 172.21.15.31:0/801627133 conn(0x7ff15c0fdb60 msgr2=0x7ff15c0fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.955+0000 7ff162dff700 1 -- 172.21.15.31:0/801627133 shutdown_connections 2023-07-29T16:20:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.955+0000 7ff162dff700 1 -- 172.21.15.31:0/801627133 wait complete. 2023-07-29T16:20:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 Processor -- start 2023-07-29T16:20:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 -- start start 2023-07-29T16:20:42.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 0x7ff15c193ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff15c103750 0x7ff15c194430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff15c1040e0 0x7ff15c1952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c195910 con 0x7ff15c103750 2023-07-29T16:20:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c195a80 con 0x7ff15c1040e0 2023-07-29T16:20:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.956+0000 7ff162dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15c195c40 con 0x7ff15c102560 2023-07-29T16:20:42.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 0x7ff15c193ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 0x7ff15c193ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48466/0 (socket says 172.21.15.31:48466) 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 -- 172.21.15.31:0/1191193149 learned_addr learned my addr 172.21.15.31:0/1191193149 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff16139c700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff15c1040e0 0x7ff15c1952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff15bfff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff15c103750 0x7ff15c194430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 -- 172.21.15.31:0/1191193149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff15c1040e0 msgr2=0x7ff15c1952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff15c1040e0 0x7ff15c1952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 -- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff15c103750 msgr2=0x7ff15c194430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff15c103750 0x7ff15c194430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 -- 172.21.15.31:0/1191193149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148013b50 con 0x7ff15c102560 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.957+0000 7ff160b9b700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 0x7ff15c193ef0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff15000e970 tx=0x7ff15000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.958+0000 7ff159ffb700 1 -- 172.21.15.31:0/1191193149 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150009790 con 0x7ff15c102560 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.958+0000 7ff159ffb700 1 -- 172.21.15.31:0/1191193149 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15000eec0 con 0x7ff15c102560 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.958+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff15c195f20 con 0x7ff15c102560 2023-07-29T16:20:42.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.958+0000 7ff159ffb700 1 -- 172.21.15.31:0/1191193149 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150003d60 con 0x7ff15c102560 2023-07-29T16:20:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.958+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff15c19c6e0 con 0x7ff15c102560 2023-07-29T16:20:42.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.962+0000 7ff159ffb700 1 -- 172.21.15.31:0/1191193149 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff150003ec0 con 0x7ff15c102560 2023-07-29T16:20:42.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.964+0000 7ff159ffb700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff14c0a0810 0x7ff14c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.964+0000 7ff159ffb700 1 -- 172.21.15.31:0/1191193149 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff150014070 con 0x7ff15c102560 2023-07-29T16:20:42.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.964+0000 7ff15bfff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff14c0a0810 0x7ff14c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.965+0000 7ff176802b80 1 Processor -- start 2023-07-29T16:20:42.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.965+0000 7ff176802b80 1 -- start start 2023-07-29T16:20:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.965+0000 7ff15bfff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff14c0a0810 0x7ff14c0a2cc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff148006010 tx=0x7ff148015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff176802b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a5332198b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff176802b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53321a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff176802b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff176802b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5331719b0 con 0x55a5330756b0 2023-07-29T16:20:42.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff176802b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a53316f370 con 0x55a53321acf0 2023-07-29T16:20:42.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff176802b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a53316ac60 con 0x55a533219df0 2023-07-29T16:20:42.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff15bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53321a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55522/0 (socket says 172.21.15.31:55522) 2023-07-29T16:20:42.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 -- 172.21.15.31:0/4030750290 learned_addr learned my addr 172.21.15.31:0/4030750290 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff160b9b700 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a5332198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 -- 172.21.15.31:0/4030750290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 msgr2=0x55a53321a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53321a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 -- 172.21.15.31:0/4030750290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 msgr2=0x55a5332198b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a5332198b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.966+0000 7ff16139c700 1 -- 172.21.15.31:0/4030750290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a532f98d80 con 0x55a53321acf0 2023-07-29T16:20:42.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.967+0000 7ff16139c700 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321b160 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff154011680 tx=0x7ff154009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a08858c22cf4589 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.967+0000 7ff146ffd700 1 -- 172.21.15.31:0/4030750290 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15400e040 con 0x55a53321acf0 2023-07-29T16:20:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.967+0000 7ff146ffd700 1 -- 172.21.15.31:0/4030750290 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15400f040 con 0x55a53321acf0 2023-07-29T16:20:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.967+0000 7ff146ffd700 1 -- 172.21.15.31:0/4030750290 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1540048c0 con 0x55a53321acf0 2023-07-29T16:20:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 -- 172.21.15.31:0/4030750290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 msgr2=0x55a53321b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321b160 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff154011680 tx=0x7ff154009920 comp rx=0 tx=0).stop 2023-07-29T16:20:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 -- 172.21.15.31:0/4030750290 shutdown_connections 2023-07-29T16:20:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321b160 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a5332198b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 --2- 172.21.15.31:0/4030750290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53321a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 -- 172.21.15.31:0/4030750290 >> 172.21.15.31:0/4030750290 conn(0x55a53320efd0 msgr2=0x55a533214300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 -- 172.21.15.31:0/4030750290 shutdown_connections 2023-07-29T16:20:42.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.968+0000 7ff176802b80 1 -- 172.21.15.31:0/4030750290 wait complete. 2023-07-29T16:20:42.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.969+0000 7ff176802b80 1 -- start start 2023-07-29T16:20:42.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 Processor -- start 2023-07-29T16:20:42.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a533222f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a533223460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53322a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5331074b0 con 0x55a5330756b0 2023-07-29T16:20:42.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a533107150 con 0x55a53321acf0 2023-07-29T16:20:42.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a532fee8c0 con 0x55a533219df0 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.970+0000 7ff176802b80 10 client.0 _ll_register_callbacks cb 0x55a5331105c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a533222f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a533222f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43418/0 (socket says 172.21.15.31:43418) 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 -- 172.21.15.31:0/2783906870 learned_addr learned my addr 172.21.15.31:0/2783906870 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff15bfff700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a533223460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff16139c700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53322a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 -- 172.21.15.31:0/2783906870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 msgr2=0x55a533223460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a533223460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 -- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 msgr2=0x55a53322a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53322a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.971+0000 7ff160b9b700 1 -- 172.21.15.31:0/2783906870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff154011330 con 0x55a5330756b0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff160b9b700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a533222f20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff15000c5b0 tx=0x7ff150016400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bf9d25f8cf3a76 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff13bfff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff13bfff700 1 -- 172.21.15.31:0/2783906870 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15009d4d0 con 0x55a5330756b0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff13bfff700 1 -- 172.21.15.31:0/2783906870 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff1500a18f0 con 0x55a5330756b0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a533195cb0 con 0x55a5330756b0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff13bfff700 1 -- 172.21.15.31:0/2783906870 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff15009d120 con 0x55a5330756b0 2023-07-29T16:20:42.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff176802b80 10 client.17097 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a532f81c00 con 0x55a5330756b0 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.972+0000 7ff13bfff700 1 -- 172.21.15.31:0/2783906870 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15007c020 con 0x55a5330756b0 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff13bfff700 1 -- 172.21.15.31:0/2783906870 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff1500a14f0 con 0x55a5330756b0 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff176802b80 10 client.17097 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff176802b80 10 client.17097 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff176802b80 20 client.17097 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:42.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff176802b80 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a53322ecc0 0x55a5332310d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff176802b80 4 client.17097 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5331e8d10 con 0x55a53322ecc0 2023-07-29T16:20:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff15bfff700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a53322ecc0 0x55a5332310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff13bfff700 10 client.17097 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.973+0000 7ff15bfff700 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a53322ecc0 0x55a5332310d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12b1029e7b823224 server_cookie=b22d09028ca3a62d in_seq=0 out_seq=0 2023-07-29T16:20:42.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.974+0000 7ff13bfff700 1 -- 172.21.15.31:0/2783906870 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a5331e8d10 con 0x55a53322ecc0 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.974+0000 7ff13bfff700 10 client.17097 handle_command_reply: tid=0 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.974+0000 7ff176802b80 1 client.17097 shutdown 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.974+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 msgr2=0x55a533222f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.974+0000 7ff176802b80 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a533222f20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7ff15000c5b0 tx=0x7ff150016400 comp rx=0 tx=0).stop 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 shutdown_connections 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a53322ecc0 0x55a5332310d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53322a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff13bfff700 0 client.17097 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:42.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5330756b0 0x55a533222f20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 --2- 172.21.15.31:0/2783906870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a533223460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 >> 172.21.15.31:0/2783906870 conn(0x55a53320efd0 msgr2=0x55a533213a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:42.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 shutdown_connections 2023-07-29T16:20:42.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 1 -- 172.21.15.31:0/2783906870 wait complete. 2023-07-29T16:20:42.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.975+0000 7ff176802b80 20 client.17097 trim_cache size 0 max 16384 2023-07-29T16:20:42.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 Processor -- start 2023-07-29T16:20:42.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 -- start start 2023-07-29T16:20:42.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 0x55a533213a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a533219df0 0x55a53322a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:42.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a53320e9d0 con 0x55a533219df0 2023-07-29T16:20:42.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5332153a0 con 0x55a53321acf0 2023-07-29T16:20:42.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.991+0000 7ff176802b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a533215d50 con 0x55a5330756b0 2023-07-29T16:20:42.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 0x55a533213a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff16139c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 0x55a533213a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48506/0 (socket says 172.21.15.31:48506) 2023-07-29T16:20:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 -- 172.21.15.31:0/2433607177 learned_addr learned my addr 172.21.15.31:0/2433607177 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff15bfff700 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a533219df0 0x55a53322a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 -- 172.21.15.31:0/2433607177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 msgr2=0x55a53321db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 -- 172.21.15.31:0/2433607177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a533219df0 msgr2=0x55a53322a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a533219df0 0x55a53322a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 -- 172.21.15.31:0/2433607177 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a532fbf1f0 con 0x55a5330756b0 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff15bfff700 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a533219df0 0x55a53322a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.992+0000 7ff160b9b700 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 0x55a533213a40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff150004800 tx=0x7ff150097a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9399248787f8489c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.993+0000 7ff1477fe700 1 -- 172.21.15.31:0/2433607177 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500954d0 con 0x55a5330756b0 2023-07-29T16:20:42.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.993+0000 7ff1477fe700 1 -- 172.21.15.31:0/2433607177 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15007c070 con 0x55a5330756b0 2023-07-29T16:20:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.993+0000 7ff1477fe700 1 -- 172.21.15.31:0/2433607177 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150098990 con 0x55a5330756b0 2023-07-29T16:20:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.993+0000 7ff176802b80 1 -- 172.21.15.31:0/2433607177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 msgr2=0x55a533213a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.993+0000 7ff176802b80 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 0x55a533213a40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff150004800 tx=0x7ff150097a50 comp rx=0 tx=0).stop 2023-07-29T16:20:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 -- 172.21.15.31:0/2433607177 shutdown_connections 2023-07-29T16:20:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a53321acf0 0x55a53321db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a5330756b0 0x55a533213a40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 --2- 172.21.15.31:0/2433607177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a533219df0 0x55a53322a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 -- 172.21.15.31:0/2433607177 >> 172.21.15.31:0/2433607177 conn(0x55a53320efd0 msgr2=0x55a533221c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 -- 172.21.15.31:0/2433607177 shutdown_connections 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 -- 172.21.15.31:0/2433607177 wait complete. 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 -- start start 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.994+0000 7ff176802b80 1 Processor -- start 2023-07-29T16:20:43.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 0x55a533229db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53322a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a53321acf0 0x55a533226850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a533226f30 con 0x55a53321acf0 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a533227070 con 0x55a5330756b0 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a533227230 con 0x55a533219df0 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff176802b80 10 client.0 _ll_register_callbacks cb 0x55a5331105c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 0x55a533229db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff160b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 0x55a533229db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55538/0 (socket says 172.21.15.31:55538) 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.995+0000 7ff160b9b700 1 -- 172.21.15.31:0/1683612149 learned_addr learned my addr 172.21.15.31:0/1683612149 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff15bfff700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53322a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff16139c700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a53321acf0 0x55a533226850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff160b9b700 1 -- 172.21.15.31:0/1683612149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 msgr2=0x55a53322a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff160b9b700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53322a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff160b9b700 1 -- 172.21.15.31:0/1683612149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a53321acf0 msgr2=0x55a533226850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff160b9b700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a53321acf0 0x55a533226850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff160b9b700 1 -- 172.21.15.31:0/1683612149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a532f81c00 con 0x55a5330756b0 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff16139c700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a53321acf0 0x55a533226850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff160b9b700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 0x55a533229db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff150095220 tx=0x7ff15009d580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19a3aaf4a9cb25c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff13bfff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff13bfff700 1 -- 172.21.15.31:0/1683612149 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150095500 con 0x55a5330756b0 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff13bfff700 1 -- 172.21.15.31:0/1683612149 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff15007d080 con 0x55a5330756b0 2023-07-29T16:20:43.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.996+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff15400f040 con 0x55a5330756b0 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff13bfff700 1 -- 172.21.15.31:0/1683612149 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff150098400 con 0x55a5330756b0 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 10 client.27110 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55a532f97250 con 0x55a5330756b0 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff13bfff700 1 -- 172.21.15.31:0/1683612149 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff15400f040 con 0x55a5330756b0 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff13bfff700 1 -- 172.21.15.31:0/1683612149 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff1500c9550 con 0x55a5330756b0 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 10 client.27110 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 10 client.27110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 20 client.27110 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a5332816c0 0x55a533283aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 4 client.27110 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.997+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a533163ce0 con 0x55a5332816c0 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.998+0000 7ff15bfff700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a5332816c0 0x55a533283aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.998+0000 7ff13bfff700 10 client.27110 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:43.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:42.998+0000 7ff15bfff700 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a5332816c0 0x55a533283aa0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5396a10abdf48a85 server_cookie=3f7ddc46421a844e in_seq=0 out_seq=0 2023-07-29T16:20:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff13bfff700 1 -- 172.21.15.31:0/1683612149 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a533163ce0 con 0x55a5332816c0 2023-07-29T16:20:43.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff13bfff700 10 client.27110 handle_command_reply: tid=0 2023-07-29T16:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 client.27110 shutdown 2023-07-29T16:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 msgr2=0x55a533229db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 0x55a533229db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff150095220 tx=0x7ff15009d580 comp rx=0 tx=0).stop 2023-07-29T16:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 shutdown_connections 2023-07-29T16:20:43.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a5332816c0 0x55a533283aa0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a53321acf0 0x55a533226850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a5330756b0 0x55a533229db0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 --2- 172.21.15.31:0/1683612149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a533219df0 0x55a53322a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff13bfff700 0 client.27110 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 >> 172.21.15.31:0/1683612149 conn(0x55a53320efd0 msgr2=0x55a533212fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 shutdown_connections 2023-07-29T16:20:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.092+0000 7ff176802b80 1 -- 172.21.15.31:0/1683612149 wait complete. 2023-07-29T16:20:43.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff176802b80 20 client.27110 trim_cache size 0 max 16384 2023-07-29T16:20:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:20:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:20:43.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:20:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:20:43.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff14c0a0810 msgr2=0x7ff14c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff14c0a0810 0x7ff14c0a2cc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff148006010 tx=0x7ff148015040 comp rx=0 tx=0).stop 2023-07-29T16:20:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 msgr2=0x7ff15c193ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 0x7ff15c193ef0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff15000e970 tx=0x7ff15000ec80 comp rx=0 tx=0).stop 2023-07-29T16:20:43.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 shutdown_connections 2023-07-29T16:20:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff14c0a0810 0x7ff14c0a2cc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff15c1040e0 0x7ff15c1952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff15c102560 0x7ff15c193ef0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 --2- 172.21.15.31:0/1191193149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff15c103750 0x7ff15c194430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 >> 172.21.15.31:0/1191193149 conn(0x7ff15c0fdb60 msgr2=0x7ff15c0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 shutdown_connections 2023-07-29T16:20:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:43.093+0000 7ff162dff700 1 -- 172.21.15.31:0/1191193149 wait complete. 2023-07-29T16:20:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 -- 172.21.15.31:0/2171070384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 msgr2=0x7fc250103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 --2- 172.21.15.31:0/2171070384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc250103b90 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc244009990 tx=0x7fc244009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 -- 172.21.15.31:0/2171070384 shutdown_connections 2023-07-29T16:20:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 --2- 172.21.15.31:0/2171070384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2501040d0 0x7fc25010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 --2- 172.21.15.31:0/2171070384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc250102550 0x7fc250102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 --2- 172.21.15.31:0/2171070384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc250103b90 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 -- 172.21.15.31:0/2171070384 >> 172.21.15.31:0/2171070384 conn(0x7fc2500fdb90 msgr2=0x7fc2500fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 -- 172.21.15.31:0/2171070384 shutdown_connections 2023-07-29T16:20:43.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.127+0000 7fc2555ff700 1 -- 172.21.15.31:0/2171070384 wait complete. 2023-07-29T16:20:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.128+0000 7fc2555ff700 1 Processor -- start 2023-07-29T16:20:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 -- start start 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc250102550 0x7fc250193e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc2501943c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2501040d0 0x7fc25019b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25019ba00 con 0x7fc250102550 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc25019bb70 con 0x7fc250103740 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc2555ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2501accf0 con 0x7fc2501040d0 2023-07-29T16:20:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc24e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc2501943c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc24e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc2501943c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55550/0 (socket says 172.21.15.31:55550) 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.129+0000 7fc24e7fc700 1 -- 172.21.15.31:0/927584097 learned_addr learned my addr 172.21.15.31:0/927584097 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2501040d0 0x7fc25019b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24e7fc700 1 -- 172.21.15.31:0/927584097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2501040d0 msgr2=0x7fc25019b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2501040d0 0x7fc25019b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24effd700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc250102550 0x7fc250193e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24e7fc700 1 -- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc250102550 msgr2=0x7fc250193e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc250102550 0x7fc250193e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24e7fc700 1 -- 172.21.15.31:0/927584097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2440095f0 con 0x7fc250103740 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24effd700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc250102550 0x7fc250193e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.130+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc2501943c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc24400c010 tx=0x7fc244004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.131+0000 7fc237fff700 1 -- 172.21.15.31:0/927584097 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24401d070 con 0x7fc250103740 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.131+0000 7fc237fff700 1 -- 172.21.15.31:0/927584097 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2440054f0 con 0x7fc250103740 2023-07-29T16:20:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.131+0000 7fc237fff700 1 -- 172.21.15.31:0/927584097 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244004ca0 con 0x7fc250103740 2023-07-29T16:20:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.131+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2501acf70 con 0x7fc250103740 2023-07-29T16:20:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.131+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2501ad2e0 con 0x7fc250103740 2023-07-29T16:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.134+0000 7fc237fff700 1 -- 172.21.15.31:0/927584097 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc24400fbc0 con 0x7fc250103740 2023-07-29T16:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.135+0000 7fc268f40b80 1 Processor -- start 2023-07-29T16:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 -- start start 2023-07-29T16:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34abd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34ac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3402a10 con 0x55f9f34aabe0 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f34003d0 con 0x55f9f34ac2f0 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc268f40b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f33fbcc0 con 0x55f9f34ab530 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc24e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc24f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc24e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55554/0 (socket says 172.21.15.31:55554) 2023-07-29T16:20:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.136+0000 7fc24e7fc700 1 -- 172.21.15.31:0/946044972 learned_addr learned my addr 172.21.15.31:0/946044972 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc237fff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc2380a0810 0x7fc2380a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24effd700 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24e7fc700 1 -- 172.21.15.31:0/946044972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 msgr2=0x55f9f34abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34abd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24e7fc700 1 -- 172.21.15.31:0/946044972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 msgr2=0x55f9f34aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34aaff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24e7fc700 1 -- 172.21.15.31:0/946044972 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9f3229de0 con 0x55f9f34ac2f0 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc237fff700 1 -- 172.21.15.31:0/927584097 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc2440ccd90 con 0x7fc250103740 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc2380a0810 0x7fc2380a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34ac740 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc24400afd0 tx=0x7fc24402d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b03a87fee48f4d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.137+0000 7fc2357fa700 1 -- 172.21.15.31:0/946044972 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24409a290 con 0x55f9f34ac2f0 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc2357fa700 1 -- 172.21.15.31:0/946044972 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2440ce070 con 0x55f9f34ac2f0 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc2357fa700 1 -- 172.21.15.31:0/946044972 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244005660 con 0x55f9f34ac2f0 2023-07-29T16:20:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 -- 172.21.15.31:0/946044972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 msgr2=0x55f9f34ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc2380a0810 0x7fc2380a2cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55f9f34aa980 tx=0x7fc248004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34ac740 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc24400afd0 tx=0x7fc24402d000 comp rx=0 tx=0).stop 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 -- 172.21.15.31:0/946044972 shutdown_connections 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34ac740 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34aaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 --2- 172.21.15.31:0/946044972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.138+0000 7fc268f40b80 1 -- 172.21.15.31:0/946044972 >> 172.21.15.31:0/946044972 conn(0x55f9f34a03e0 msgr2=0x55f9f34a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.139+0000 7fc268f40b80 1 -- 172.21.15.31:0/946044972 shutdown_connections 2023-07-29T16:20:43.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.139+0000 7fc268f40b80 1 -- 172.21.15.31:0/946044972 wait complete. 2023-07-29T16:20:43.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.139+0000 7fc268f40b80 1 -- start start 2023-07-29T16:20:43.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 Processor -- start 2023-07-29T16:20:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 0x55f9f34b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34b4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ac2f0 0x55f9f34bb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3398510 con 0x55f9f34ac2f0 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f33981b0 con 0x55f9f34aabe0 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f327f920 con 0x55f9f34ab530 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.140+0000 7fc268f40b80 10 client.0 _ll_register_callbacks cb 0x55f9f33a1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 0x55f9f34b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 0x55f9f34b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55556/0 (socket says 172.21.15.31:55556) 2023-07-29T16:20:43.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 -- 172.21.15.31:0/38234364 learned_addr learned my addr 172.21.15.31:0/38234364 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ac2f0 0x55f9f34bb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 -- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 msgr2=0x55f9f34b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34b4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 -- 172.21.15.31:0/38234364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ac2f0 msgr2=0x55f9f34bb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ac2f0 0x55f9f34bb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 -- 172.21.15.31:0/38234364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2501ad2e0 con 0x55f9f34aabe0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ac2f0 0x55f9f34bb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc24effd700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 0x55f9f34b4450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc240013710 tx=0x7fc240013a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35fa4bf6af3a152 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc232ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc232ff5700 1 -- 172.21.15.31:0/38234364 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240011770 con 0x55f9f34aabe0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.141+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9f3426d10 con 0x55f9f34aabe0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc232ff5700 1 -- 172.21.15.31:0/38234364 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc2400118d0 con 0x55f9f34aabe0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc232ff5700 1 -- 172.21.15.31:0/38234364 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc240011a90 con 0x55f9f34aabe0 2023-07-29T16:20:43.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 10 client.27122 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f9f3212c60 con 0x55f9f34aabe0 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc232ff5700 1 -- 172.21.15.31:0/38234364 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f9f3426d10 con 0x55f9f34aabe0 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc232ff5700 1 -- 172.21.15.31:0/38234364 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fc24000ed20 con 0x55f9f34aabe0 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 10 client.27122 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 10 client.27122 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 20 client.27122 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f34c01f0 0x55f9f34c2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 4 client.27122 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.142+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9f3479d70 con 0x55f9f34c01f0 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.143+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f34c01f0 0x55f9f34c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.143+0000 7fc232ff5700 10 client.27122 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.143+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f34c01f0 0x55f9f34c2600 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a802c7567cd19c08 server_cookie=db39d9d0f9325ac6 in_seq=0 out_seq=0 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc232ff5700 1 -- 172.21.15.31:0/38234364 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f9f3479d70 con 0x55f9f34c01f0 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc232ff5700 10 client.27122 handle_command_reply: tid=0 2023-07-29T16:20:43.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 client.27122 shutdown 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 msgr2=0x55f9f34b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 0x55f9f34b4450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc240013710 tx=0x7fc240013a20 comp rx=0 tx=0).stop 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 shutdown_connections 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ac2f0 0x55f9f34bb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f34c01f0 0x55f9f34c2600 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34aabe0 0x55f9f34b4450 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 --2- 172.21.15.31:0/38234364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34b4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 >> 172.21.15.31:0/38234364 conn(0x55f9f34a03e0 msgr2=0x55f9f34a5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.144+0000 7fc232ff5700 0 client.27122 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:43.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.145+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 shutdown_connections 2023-07-29T16:20:43.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.145+0000 7fc268f40b80 1 -- 172.21.15.31:0/38234364 wait complete. 2023-07-29T16:20:43.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.145+0000 7fc268f40b80 20 client.27122 trim_cache size 0 max 16384 2023-07-29T16:20:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 Processor -- start 2023-07-29T16:20:43.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 -- start start 2023-07-29T16:20:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34af630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f349ff40 con 0x55f9f34aabe0 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f34a7490 con 0x55f9f34ac2f0 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc268f40b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f34a2460 con 0x55f9f34ab530 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc24f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc24effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc24effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34b98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43480/0 (socket says 172.21.15.31:43480) 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc24effd700 1 -- 172.21.15.31:0/897501151 learned_addr learned my addr 172.21.15.31:0/897501151 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.161+0000 7fc24f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34af630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55566/0 (socket says 172.21.15.31:55566) 2023-07-29T16:20:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24effd700 1 -- 172.21.15.31:0/897501151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 msgr2=0x55f9f34af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24effd700 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24effd700 1 -- 172.21.15.31:0/897501151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 msgr2=0x55f9f34af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24effd700 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24effd700 1 -- 172.21.15.31:0/897501151 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9f3250250 con 0x55f9f34aabe0 2023-07-29T16:20:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc24effd700 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34b98e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc2400103d0 tx=0x7fc240010670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac47fd8ca390ee05 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc235ffb700 1 -- 172.21.15.31:0/897501151 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240010b80 con 0x55f9f34aabe0 2023-07-29T16:20:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.162+0000 7fc235ffb700 1 -- 172.21.15.31:0/897501151 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc240010740 con 0x55f9f34aabe0 2023-07-29T16:20:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc235ffb700 1 -- 172.21.15.31:0/897501151 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24001b810 con 0x55f9f34aabe0 2023-07-29T16:20:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 -- 172.21.15.31:0/897501151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 msgr2=0x55f9f34b98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34b98e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fc2400103d0 tx=0x7fc240010670 comp rx=0 tx=0).stop 2023-07-29T16:20:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 -- 172.21.15.31:0/897501151 shutdown_connections 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34aabe0 0x55f9f34b98e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 --2- 172.21.15.31:0/897501151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34ab530 0x55f9f34af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 -- 172.21.15.31:0/897501151 >> 172.21.15.31:0/897501151 conn(0x55f9f34a03e0 msgr2=0x55f9f34b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 -- 172.21.15.31:0/897501151 shutdown_connections 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.163+0000 7fc268f40b80 1 -- 172.21.15.31:0/897501151 wait complete. 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.164+0000 7fc268f40b80 1 -- start start 2023-07-29T16:20:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.164+0000 7fc268f40b80 1 Processor -- start 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 0x55f9f34b92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ab530 0x55f9f34b9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f34b7fc0 con 0x55f9f34ab530 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f34b8130 con 0x55f9f34ac2f0 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f34b82f0 con 0x55f9f34aabe0 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc268f40b80 10 client.0 _ll_register_callbacks cb 0x55f9f33a1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc24effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 0x55f9f34b92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc24f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc24effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 0x55f9f34b92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48568/0 (socket says 172.21.15.31:48568) 2023-07-29T16:20:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc24e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ab530 0x55f9f34b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.165+0000 7fc24effd700 1 -- 172.21.15.31:0/3851068806 learned_addr learned my addr 172.21.15.31:0/3851068806 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24effd700 1 -- 172.21.15.31:0/3851068806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 msgr2=0x55f9f34b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24effd700 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24effd700 1 -- 172.21.15.31:0/3851068806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ab530 msgr2=0x55f9f34b9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24effd700 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ab530 0x55f9f34b9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24effd700 1 -- 172.21.15.31:0/3851068806 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc248009e40 con 0x55f9f34aabe0 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24f7fe700 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc24effd700 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 0x55f9f34b92c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc240011cf0 tx=0x7fc240009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c98ece7556f0ec0e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.166+0000 7fc232ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc232ff5700 1 -- 172.21.15.31:0/3851068806 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc240010d90 con 0x55f9f34aabe0 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc232ff5700 1 -- 172.21.15.31:0/3851068806 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc240000e40 con 0x55f9f34aabe0 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2440ce070 con 0x55f9f34aabe0 2023-07-29T16:20:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc232ff5700 1 -- 172.21.15.31:0/3851068806 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc24001b510 con 0x55f9f34aabe0 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc268f40b80 10 client.27139 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f9f3212c60 con 0x55f9f34aabe0 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.167+0000 7fc232ff5700 1 -- 172.21.15.31:0/3851068806 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2440ce070 con 0x55f9f34aabe0 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc232ff5700 1 -- 172.21.15.31:0/3851068806 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fc24002e650 con 0x55f9f34aabe0 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc268f40b80 10 client.27139 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc268f40b80 10 client.27139 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc268f40b80 20 client.27139 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc268f40b80 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f3512e70 0x55f9f3515280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc268f40b80 4 client.27139 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f9f33f4d40 con 0x55f9f3512e70 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f3512e70 0x55f9f3515280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.168+0000 7fc24e7fc700 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f3512e70 0x55f9f3515280 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c40659ece9dd5d6c server_cookie=50056683e50686d6 in_seq=0 out_seq=0 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc232ff5700 10 client.27139 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc232ff5700 1 -- 172.21.15.31:0/3851068806 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f9f33f4d40 con 0x55f9f3512e70 2023-07-29T16:20:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc232ff5700 10 client.27139 handle_command_reply: tid=0 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 client.27139 shutdown 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 msgr2=0x55f9f34b92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 0x55f9f34b92c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc240011cf0 tx=0x7fc240009260 comp rx=0 tx=0).stop 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 shutdown_connections 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f9f3512e70 0x55f9f3515280 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f34ac2f0 0x55f9f34b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f34aabe0 0x55f9f34b92c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc232ff5700 0 client.27139 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:20:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 --2- 172.21.15.31:0/3851068806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f34ab530 0x55f9f34b9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 >> 172.21.15.31:0/3851068806 conn(0x55f9f34a03e0 msgr2=0x55f9f34b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 shutdown_connections 2023-07-29T16:20:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.169+0000 7fc268f40b80 1 -- 172.21.15.31:0/3851068806 wait complete. 2023-07-29T16:20:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc268f40b80 20 client.27139 trim_cache size 0 max 16384 2023-07-29T16:20:43.174 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:20:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc2380a0810 msgr2=0x7fc2380a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc2380a0810 0x7fc2380a2cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55f9f34aa980 tx=0x7fc248004ad0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 msgr2=0x7fc2501943c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc2501943c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc24400c010 tx=0x7fc244004730 comp rx=0 tx=0).stop 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 shutdown_connections 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc2380a0810 0x7fc2380a2cc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2501040d0 0x7fc25019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc250102550 0x7fc250193e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 --2- 172.21.15.31:0/927584097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc250103740 0x7fc2501943c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 >> 172.21.15.31:0/927584097 conn(0x7fc2500fdb90 msgr2=0x7fc2500ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.170+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 shutdown_connections 2023-07-29T16:20:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.171+0000 7fc2555ff700 1 -- 172.21.15.31:0/927584097 wait complete. 2023-07-29T16:20:43.192 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:20:43.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 -- 172.21.15.31:0/889963122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 msgr2=0x7f4188108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 --2- 172.21.15.31:0/889963122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f4188108440 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4180009f90 tx=0x7f418000b330 comp rx=0 tx=0).stop 2023-07-29T16:20:43.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 -- 172.21.15.31:0/889963122 shutdown_connections 2023-07-29T16:20:43.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 --2- 172.21.15.31:0/889963122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f4188108440 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 --2- 172.21.15.31:0/889963122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 --2- 172.21.15.31:0/889963122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41881036c0 0x7f4188105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.412+0000 7f418f9ff700 1 -- 172.21.15.31:0/889963122 >> 172.21.15.31:0/889963122 conn(0x7f41880fa840 msgr2=0x7f41880fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.413+0000 7f418f9ff700 1 -- 172.21.15.31:0/889963122 shutdown_connections 2023-07-29T16:20:43.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.413+0000 7f418f9ff700 1 -- 172.21.15.31:0/889963122 wait complete. 2023-07-29T16:20:43.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 Processor -- start 2023-07-29T16:20:43.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 -- start start 2023-07-29T16:20:43.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41881036c0 0x7f4188196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f418819d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418819dcb0 con 0x7f41881036c0 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418819ddf0 con 0x7f4188100da0 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418f9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418819df80 con 0x7f4188105fe0 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418d79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418d79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55592/0 (socket says 172.21.15.31:55592) 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418d79b700 1 -- 172.21.15.31:0/3833048606 learned_addr learned my addr 172.21.15.31:0/3833048606 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.414+0000 7f418cf9a700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41881036c0 0x7f4188196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418df9c700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f418819d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418d79b700 1 -- 172.21.15.31:0/3833048606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 msgr2=0x7f418819d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418d79b700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f418819d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418d79b700 1 -- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41881036c0 msgr2=0x7f4188196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418d79b700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41881036c0 0x7f4188196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418d79b700 1 -- 172.21.15.31:0/3833048606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4180009c40 con 0x7f4188100da0 2023-07-29T16:20:43.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418df9c700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f418819d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418d79b700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188196130 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f418400e9c0 tx=0x7f418400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f417e7fc700 1 -- 172.21.15.31:0/3833048606 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400cbd0 con 0x7f4188100da0 2023-07-29T16:20:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f417e7fc700 1 -- 172.21.15.31:0/3833048606 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f418400eeb0 con 0x7f4188100da0 2023-07-29T16:20:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f417e7fc700 1 -- 172.21.15.31:0/3833048606 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184005580 con 0x7f4188100da0 2023-07-29T16:20:43.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.415+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418819e260 con 0x7f4188100da0 2023-07-29T16:20:43.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.416+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418819e770 con 0x7f4188100da0 2023-07-29T16:20:43.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.420+0000 7f417e7fc700 1 -- 172.21.15.31:0/3833048606 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4184004750 con 0x7f4188100da0 2023-07-29T16:20:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.421+0000 7f41a3394b80 1 Processor -- start 2023-07-29T16:20:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 -- start start 2023-07-29T16:20:43.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322dea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322deb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322debb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322d3da10 con 0x558322dea990 2023-07-29T16:20:43.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322d3b3d0 con 0x558322deb6d0 2023-07-29T16:20:43.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f41a3394b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322d36cc0 con 0x558322dea040 2023-07-29T16:20:43.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f418cf9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322debb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f418df9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322dea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f418d79b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322deb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f418cf9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322debb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55596/0 (socket says 172.21.15.31:55596) 2023-07-29T16:20:43.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.422+0000 7f418cf9a700 1 -- 172.21.15.31:0/3679633303 learned_addr learned my addr 172.21.15.31:0/3679633303 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f417e7fc700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41740a0810 0x7f41740a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418cf9a700 1 -- 172.21.15.31:0/3679633303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 msgr2=0x558322dea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418cf9a700 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322dea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418cf9a700 1 -- 172.21.15.31:0/3679633303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 msgr2=0x558322deb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418cf9a700 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322deb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418cf9a700 1 -- 172.21.15.31:0/3679633303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558322b64de0 con 0x558322deb6d0 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f417e7fc700 1 -- 172.21.15.31:0/3833048606 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4184014070 con 0x7f4188100da0 2023-07-29T16:20:43.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418df9c700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41740a0810 0x7f41740a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f418cf9a700 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322debb40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4178009d80 tx=0x7f4178011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e1a889f47957d23 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f41737fe700 1 -- 172.21.15.31:0/3679633303 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178004230 con 0x558322deb6d0 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f41737fe700 1 -- 172.21.15.31:0/3679633303 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4178004390 con 0x558322deb6d0 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.423+0000 7f41737fe700 1 -- 172.21.15.31:0/3679633303 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178019880 con 0x558322deb6d0 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.424+0000 7f41a3394b80 1 -- 172.21.15.31:0/3679633303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 msgr2=0x558322debb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.424+0000 7f418df9c700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41740a0810 0x7f41740a2cc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4180006130 tx=0x7f4180009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.424+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322debb40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4178009d80 tx=0x7f4178011720 comp rx=0 tx=0).stop 2023-07-29T16:20:43.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 -- 172.21.15.31:0/3679633303 shutdown_connections 2023-07-29T16:20:43.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322debb40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322dea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3679633303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322deb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 -- 172.21.15.31:0/3679633303 >> 172.21.15.31:0/3679633303 conn(0x558322ddb3e0 msgr2=0x558322de08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 -- 172.21.15.31:0/3679633303 shutdown_connections 2023-07-29T16:20:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 -- 172.21.15.31:0/3679633303 wait complete. 2023-07-29T16:20:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.425+0000 7f41a3394b80 1 -- start start 2023-07-29T16:20:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 Processor -- start 2023-07-29T16:20:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322dfb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322cd3510 con 0x558322dea990 2023-07-29T16:20:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322cd31b0 con 0x558322deb6d0 2023-07-29T16:20:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322bba920 con 0x558322dea040 2023-07-29T16:20:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.426+0000 7f41a3394b80 10 client.0 _ll_register_callbacks cb 0x558322cdc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418cf9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48602/0 (socket says 172.21.15.31:48602) 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 -- 172.21.15.31:0/2445574761 learned_addr learned my addr 172.21.15.31:0/2445574761 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418df9c700 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322dfb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 -- 172.21.15.31:0/2445574761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 msgr2=0x558322dfb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322dfb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 -- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 msgr2=0x558322df40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 -- 172.21.15.31:0/2445574761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41780099e0 con 0x558322dea040 2023-07-29T16:20:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f418d79b700 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df3b90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4184000c00 tx=0x7f4184009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62667613600b87ab server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f4170ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:43.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f4170ff9700 1 -- 172.21.15.31:0/2445574761 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184098f80 con 0x558322dea040 2023-07-29T16:20:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f4170ff9700 1 -- 172.21.15.31:0/2445574761 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f418400cd30 con 0x558322dea040 2023-07-29T16:20:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.427+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558322d61d10 con 0x558322dea040 2023-07-29T16:20:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f4170ff9700 1 -- 172.21.15.31:0/2445574761 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41840a1a70 con 0x558322dea040 2023-07-29T16:20:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 10 client.27151 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558322b4dc60 con 0x558322dea040 2023-07-29T16:20:43.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f4170ff9700 1 -- 172.21.15.31:0/2445574761 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558322d61d10 con 0x558322dea040 2023-07-29T16:20:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f4170ff9700 1 -- 172.21.15.31:0/2445574761 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f41840999a0 con 0x558322dea040 2023-07-29T16:20:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 10 client.27151 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 10 client.27151 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:20:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 20 client.27151 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322dff8a0 0x558322e01cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 4 client.27151 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.428+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558322db4d70 con 0x558322dff8a0 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.429+0000 7f418cf9a700 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322dff8a0 0x558322e01cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.429+0000 7f4170ff9700 10 client.27151 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.429+0000 7f418cf9a700 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322dff8a0 0x558322e01cb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=18b0e05ca5f0ccdd server_cookie=bf6ba588ead5c624 in_seq=0 out_seq=0 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f4170ff9700 1 -- 172.21.15.31:0/2445574761 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558322db4d70 con 0x558322dff8a0 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f4170ff9700 10 client.27151 handle_command_reply: tid=0 2023-07-29T16:20:43.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 client.27151 shutdown 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 msgr2=0x558322df3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df3b90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4184000c00 tx=0x7f4184009f80 comp rx=0 tx=0).stop 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 shutdown_connections 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322dff8a0 0x558322e01cb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322deb6d0 0x558322dfb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f4170ff9700 0 client.27151 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:43.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df3b90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2445574761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 >> 172.21.15.31:0/2445574761 conn(0x558322ddb3e0 msgr2=0x558322de0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.430+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 shutdown_connections 2023-07-29T16:20:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.431+0000 7f41a3394b80 1 -- 172.21.15.31:0/2445574761 wait complete. 2023-07-29T16:20:43.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.431+0000 7f41a3394b80 20 client.27151 trim_cache size 0 max 16384 2023-07-29T16:20:43.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 Processor -- start 2023-07-29T16:20:43.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 -- start start 2023-07-29T16:20:43.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 0x558322de18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322deb6d0 0x558322df3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322dff680 con 0x558322dea990 2023-07-29T16:20:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322df1840 con 0x558322dea040 2023-07-29T16:20:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.447+0000 7f41a3394b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322e06610 con 0x558322deb6d0 2023-07-29T16:20:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418df9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322deb6d0 0x558322df3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 0x558322de18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418df9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322deb6d0 0x558322df3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48616/0 (socket says 172.21.15.31:48616) 2023-07-29T16:20:43.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 0x558322de18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55606/0 (socket says 172.21.15.31:55606) 2023-07-29T16:20:43.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418df9c700 1 -- 172.21.15.31:0/3707063016 learned_addr learned my addr 172.21.15.31:0/3707063016 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418cf9a700 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 -- 172.21.15.31:0/3707063016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322deb6d0 msgr2=0x558322df3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322deb6d0 0x558322df3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 -- 172.21.15.31:0/3707063016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 msgr2=0x558322df90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 -- 172.21.15.31:0/3707063016 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558322b8b250 con 0x558322dea040 2023-07-29T16:20:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.448+0000 7f418d79b700 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 0x558322de18e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4184096090 tx=0x7f41840cc0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e932d08c6c267a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f4173fff700 1 -- 172.21.15.31:0/3707063016 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184098f80 con 0x558322dea040 2023-07-29T16:20:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f4173fff700 1 -- 172.21.15.31:0/3707063016 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f418400cd30 con 0x558322dea040 2023-07-29T16:20:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f4173fff700 1 -- 172.21.15.31:0/3707063016 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184099ae0 con 0x558322dea040 2023-07-29T16:20:43.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 -- 172.21.15.31:0/3707063016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 msgr2=0x558322de18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 0x558322de18e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4184096090 tx=0x7f41840cc0b0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 -- 172.21.15.31:0/3707063016 shutdown_connections 2023-07-29T16:20:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322deb6d0 0x558322df3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea040 0x558322de18e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 --2- 172.21.15.31:0/3707063016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322dea990 0x558322df90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 -- 172.21.15.31:0/3707063016 >> 172.21.15.31:0/3707063016 conn(0x558322ddb3e0 msgr2=0x558322df28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.449+0000 7f41a3394b80 1 -- 172.21.15.31:0/3707063016 shutdown_connections 2023-07-29T16:20:43.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.450+0000 7f41a3394b80 1 -- 172.21.15.31:0/3707063016 wait complete. 2023-07-29T16:20:43.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.450+0000 7f41a3394b80 1 -- start start 2023-07-29T16:20:43.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.450+0000 7f41a3394b80 1 Processor -- start 2023-07-29T16:20:43.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea990 0x558322df72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322deb6d0 0x558322df82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322df8820 con 0x558322deb6d0 2023-07-29T16:20:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322df8990 con 0x558322dea990 2023-07-29T16:20:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558322df8b50 con 0x558322dea040 2023-07-29T16:20:43.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f41a3394b80 10 client.0 _ll_register_callbacks cb 0x558322cdc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418d79b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418cf9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea990 0x558322df72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418d79b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48618/0 (socket says 172.21.15.31:48618) 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418d79b700 1 -- 172.21.15.31:0/2867224937 learned_addr learned my addr 172.21.15.31:0/2867224937 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418df9c700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322deb6d0 0x558322df82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418cf9a700 1 -- 172.21.15.31:0/2867224937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 msgr2=0x558322df6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418cf9a700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418cf9a700 1 -- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322deb6d0 msgr2=0x558322df82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418cf9a700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322deb6d0 0x558322df82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418cf9a700 1 -- 172.21.15.31:0/2867224937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558322b4dc60 con 0x558322dea990 2023-07-29T16:20:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.451+0000 7f418d79b700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f418cf9a700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea990 0x558322df72d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4178005950 tx=0x7f4178005b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2f0624e1bf81dd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f4170ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:43.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f4170ff9700 1 -- 172.21.15.31:0/2867224937 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4178029040 con 0x558322dea990 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4178004390 con 0x558322dea990 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f4170ff9700 1 -- 172.21.15.31:0/2867224937 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4178005240 con 0x558322dea990 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f4170ff9700 1 -- 172.21.15.31:0/2867224937 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4178005420 con 0x558322dea990 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f41a3394b80 10 client.27146 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.452+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x558322b632b0 con 0x558322dea990 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f4170ff9700 1 -- 172.21.15.31:0/2867224937 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4178004390 con 0x558322dea990 2023-07-29T16:20:43.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f4170ff9700 1 -- 172.21.15.31:0/2867224937 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f4178011440 con 0x558322dea990 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f41a3394b80 10 client.27146 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f41a3394b80 10 client.27146 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f41a3394b80 20 client.27146 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322e521f0 0x558322e54600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f41a3394b80 4 client.27146 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558322d2fd40 con 0x558322e521f0 2023-07-29T16:20:43.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.453+0000 7f418d79b700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322e521f0 0x558322e54600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f418d79b700 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322e521f0 0x558322e54600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b1a042d85a8adf79 server_cookie=9de87b0e9c2eacdb in_seq=0 out_seq=0 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f4170ff9700 10 client.27146 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f4170ff9700 1 -- 172.21.15.31:0/2867224937 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558322d2fd40 con 0x558322e521f0 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f4170ff9700 10 client.27146 handle_command_reply: tid=0 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 client.27146 shutdown 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea990 msgr2=0x558322df72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea990 0x558322df72d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4178005950 tx=0x7f4178005b00 comp rx=0 tx=0).stop 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 shutdown_connections 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x558322e521f0 0x558322e54600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558322deb6d0 0x558322df82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558322dea040 0x558322df6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 --2- 172.21.15.31:0/2867224937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558322dea990 0x558322df72d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f4170ff9700 0 client.27146 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:20:43.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 >> 172.21.15.31:0/2867224937 conn(0x558322ddb3e0 msgr2=0x558322df1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 shutdown_connections 2023-07-29T16:20:43.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 1 -- 172.21.15.31:0/2867224937 wait complete. 2023-07-29T16:20:43.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.454+0000 7f41a3394b80 20 client.27146 trim_cache size 0 max 16384 2023-07-29T16:20:43.467 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:20:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41740a0810 msgr2=0x7f41740a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41740a0810 0x7f41740a2cc0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4180006130 tx=0x7f4180009920 comp rx=0 tx=0).stop 2023-07-29T16:20:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 msgr2=0x7f4188196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188196130 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f418400e9c0 tx=0x7f418400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 shutdown_connections 2023-07-29T16:20:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f41740a0810 0x7f41740a2cc0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4188105fe0 0x7f418819d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4188100da0 0x7f4188196130 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 --2- 172.21.15.31:0/3833048606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41881036c0 0x7f4188196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 >> 172.21.15.31:0/3833048606 conn(0x7f41880fa840 msgr2=0x7f4188101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 shutdown_connections 2023-07-29T16:20:43.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.455+0000 7f418f9ff700 1 -- 172.21.15.31:0/3833048606 wait complete. 2023-07-29T16:20:43.473 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:20:43.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 -- 172.21.15.31:0/1363823022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101540 msgr2=0x7fe380101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 --2- 172.21.15.31:0/1363823022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101540 0x7fe380101990 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe374009990 tx=0x7fe374009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 -- 172.21.15.31:0/1363823022 shutdown_connections 2023-07-29T16:20:43.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 --2- 172.21.15.31:0/1363823022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101ed0 0x7fe380108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 --2- 172.21.15.31:0/1363823022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 --2- 172.21.15.31:0/1363823022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101540 0x7fe380101990 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.708+0000 7fe3861ff700 1 -- 172.21.15.31:0/1363823022 >> 172.21.15.31:0/1363823022 conn(0x7fe3800fb970 msgr2=0x7fe3800fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.709+0000 7fe3861ff700 1 -- 172.21.15.31:0/1363823022 shutdown_connections 2023-07-29T16:20:43.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.709+0000 7fe3861ff700 1 -- 172.21.15.31:0/1363823022 wait complete. 2023-07-29T16:20:43.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 Processor -- start 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 -- start start 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101540 0x7fe380198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101ed0 0x7fe38019f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38019fdc0 con 0x7fe380101540 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38019ff00 con 0x7fe380100350 2023-07-29T16:20:43.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.710+0000 7fe3861ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3801a0090 con 0x7fe380101ed0 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55648/0 (socket says 172.21.15.31:55648) 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 -- 172.21.15.31:0/625593690 learned_addr learned my addr 172.21.15.31:0/625593690 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101ed0 0x7fe38019f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37effd700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101540 0x7fe380198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 -- 172.21.15.31:0/625593690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101ed0 msgr2=0x7fe38019f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101ed0 0x7fe38019f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 -- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101540 msgr2=0x7fe380198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101540 0x7fe380198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 -- 172.21.15.31:0/625593690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe378009c40 con 0x7fe380100350 2023-07-29T16:20:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37ffff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101ed0 0x7fe38019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:43.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.711+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380198240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe37000e970 tx=0x7fe37000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.712+0000 7fe37cff9700 1 -- 172.21.15.31:0/625593690 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370009790 con 0x7fe380100350 2023-07-29T16:20:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.712+0000 7fe37cff9700 1 -- 172.21.15.31:0/625593690 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe37000c7c0 con 0x7fe380100350 2023-07-29T16:20:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.712+0000 7fe37cff9700 1 -- 172.21.15.31:0/625593690 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370003c20 con 0x7fe380100350 2023-07-29T16:20:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.712+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3740095f0 con 0x7fe380100350 2023-07-29T16:20:43.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.712+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3801a06a0 con 0x7fe380100350 2023-07-29T16:20:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.715+0000 7fe37cff9700 1 -- 172.21.15.31:0/625593690 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe37000c930 con 0x7fe380100350 2023-07-29T16:20:43.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.716+0000 7fe399b09b80 1 Processor -- start 2023-07-29T16:20:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 -- start start 2023-07-29T16:20:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37cff9700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3680a0810 0x7fe3680a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37cff9700 1 -- 172.21.15.31:0/625593690 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe370014070 con 0x7fe380100350 2023-07-29T16:20:43.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aa0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37ffff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3680a0810 0x7fe3680a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a29f8a10 con 0x55f6a2aa22f0 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a29f63d0 con 0x55f6a2aa1530 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe399b09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a29f1cc0 con 0x55f6a2aa0be0 2023-07-29T16:20:43.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aa0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aa0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48644/0 (socket says 172.21.15.31:48644) 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37f7fe700 1 -- 172.21.15.31:0/279619046 learned_addr learned my addr 172.21.15.31:0/279619046 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.717+0000 7fe37effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37ffff700 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37f7fe700 1 -- 172.21.15.31:0/279619046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 msgr2=0x55f6a2aa1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37f7fe700 1 -- 172.21.15.31:0/279619046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa22f0 msgr2=0x55f6a2aa2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37f7fe700 1 -- 172.21.15.31:0/279619046 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6a281fde0 con 0x55f6a2aa0be0 2023-07-29T16:20:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37ffff700 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:20:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aa0ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe3700a1320 tx=0x7fe370098e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97f64906f4921871 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe365ffb700 1 -- 172.21.15.31:0/279619046 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3700a1850 con 0x55f6a2aa0be0 2023-07-29T16:20:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe365ffb700 1 -- 172.21.15.31:0/279619046 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3700977a0 con 0x55f6a2aa0be0 2023-07-29T16:20:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.718+0000 7fe365ffb700 1 -- 172.21.15.31:0/279619046 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3700998c0 con 0x55f6a2aa0be0 2023-07-29T16:20:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe37ffff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3680a0810 0x7fe3680a2cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe378009fd0 tx=0x7fe378009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 -- 172.21.15.31:0/279619046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 msgr2=0x55f6a2aa0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aa0ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe3700a1320 tx=0x7fe370098e50 comp rx=0 tx=0).stop 2023-07-29T16:20:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 -- 172.21.15.31:0/279619046 shutdown_connections 2023-07-29T16:20:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aa0ff0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 --2- 172.21.15.31:0/279619046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 -- 172.21.15.31:0/279619046 >> 172.21.15.31:0/279619046 conn(0x55f6a2a963e0 msgr2=0x55f6a2a9b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 -- 172.21.15.31:0/279619046 shutdown_connections 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.719+0000 7fe399b09b80 1 -- 172.21.15.31:0/279619046 wait complete. 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.720+0000 7fe399b09b80 1 -- start start 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.720+0000 7fe399b09b80 1 Processor -- start 2023-07-29T16:20:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.720+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aaa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.720+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa1530 0x55f6a2aaad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.720+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2ab1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe399b09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a298e510 con 0x55f6a2aa1530 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe399b09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a298e1b0 con 0x55f6a2aa22f0 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe399b09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2875920 con 0x55f6a2aa0be0 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe399b09b80 10 client.0 _ll_register_callbacks cb 0x55f6a2997620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2ab1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa1530 0x55f6a2aaad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2ab1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55660/0 (socket says 172.21.15.31:55660) 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa1530 0x55f6a2aaad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43602/0 (socket says 172.21.15.31:43602) 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 -- 172.21.15.31:0/3153886502 learned_addr learned my addr 172.21.15.31:0/3153886502 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aaa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 -- 172.21.15.31:0/3153886502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 msgr2=0x55f6a2aaa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aaa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 -- 172.21.15.31:0/3153886502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa1530 msgr2=0x55f6a2aaad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa1530 0x55f6a2aaad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37ffff700 1 -- 172.21.15.31:0/3153886502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3801a06a0 con 0x55f6a2aa22f0 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.721+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aaa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe37ffff700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2ab1d90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe3780045e0 tx=0x7fe378004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71079d5211b62be3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe35effd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe35effd700 1 -- 172.21.15.31:0/3153886502 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3780048f0 con 0x55f6a2aa22f0 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f6a2a1cd10 con 0x55f6a2aa22f0 2023-07-29T16:20:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe35effd700 1 -- 172.21.15.31:0/3153886502 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe37800eb90 con 0x55f6a2aa22f0 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe35effd700 1 -- 172.21.15.31:0/3153886502 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe37800ed50 con 0x55f6a2aa22f0 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe399b09b80 10 client.27158 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.722+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f6a2808c60 con 0x55f6a2aa22f0 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe35effd700 1 -- 172.21.15.31:0/3153886502 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe378030020 con 0x55f6a2aa22f0 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe35effd700 1 -- 172.21.15.31:0/3153886502 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fe37802c440 con 0x55f6a2aa22f0 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe399b09b80 10 client.27158 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe399b09b80 10 client.27158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:43.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe399b09b80 20 client.27158 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe399b09b80 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2ab6600 0x55f6a2ab8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe399b09b80 4 client.27158 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:43.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6a2a6fd70 con 0x55f6a2ab6600 2023-07-29T16:20:43.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.723+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2ab6600 0x55f6a2ab8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.724+0000 7fe35effd700 10 client.27158 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:43.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.724+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2ab6600 0x55f6a2ab8a10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=702104734ed41f26 server_cookie=d8c5c3785c34992d in_seq=0 out_seq=0 2023-07-29T16:20:43.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe35effd700 1 -- 172.21.15.31:0/3153886502 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f6a2a6fd70 con 0x55f6a2ab6600 2023-07-29T16:20:43.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe35effd700 10 client.27158 handle_command_reply: tid=0 2023-07-29T16:20:43.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 client.27158 shutdown 2023-07-29T16:20:43.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 msgr2=0x55f6a2ab1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2ab1d90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe3780045e0 tx=0x7fe378004ad0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 shutdown_connections 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2ab6600 0x55f6a2ab8a10 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2ab1d90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aaa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe35effd700 0 client.27158 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 --2- 172.21.15.31:0/3153886502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa1530 0x55f6a2aaad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 >> 172.21.15.31:0/3153886502 conn(0x55f6a2a963e0 msgr2=0x55f6a2a9b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 shutdown_connections 2023-07-29T16:20:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.725+0000 7fe399b09b80 1 -- 172.21.15.31:0/3153886502 wait complete. 2023-07-29T16:20:43.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.726+0000 7fe399b09b80 20 client.27158 trim_cache size 0 max 16384 2023-07-29T16:20:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.740+0000 7fe399b09b80 1 Processor -- start 2023-07-29T16:20:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.740+0000 7fe399b09b80 1 -- start start 2023-07-29T16:20:43.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.740+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.740+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe399b09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2aaa380 con 0x55f6a2aa0be0 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe399b09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2ab3970 con 0x55f6a2aa1530 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe399b09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2a98460 con 0x55f6a2aa22f0 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55670/0 (socket says 172.21.15.31:55670) 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 -- 172.21.15.31:0/903212625 learned_addr learned my addr 172.21.15.31:0/903212625 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37ffff700 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 -- 172.21.15.31:0/903212625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa22f0 msgr2=0x55f6a2aa94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 -- 172.21.15.31:0/903212625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 msgr2=0x55f6a2aafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aafd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.741+0000 7fe37effd700 1 -- 172.21.15.31:0/903212625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6a2846250 con 0x55f6a2aa1530 2023-07-29T16:20:43.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.742+0000 7fe37effd700 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa26c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe3740098e0 tx=0x7fe374005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=672ca5d8abc76c28 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.742+0000 7fe3667fc700 1 -- 172.21.15.31:0/903212625 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37401d030 con 0x55f6a2aa1530 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.742+0000 7fe3667fc700 1 -- 172.21.15.31:0/903212625 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe37401e040 con 0x55f6a2aa1530 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.742+0000 7fe3667fc700 1 -- 172.21.15.31:0/903212625 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374004870 con 0x55f6a2aa1530 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 -- 172.21.15.31:0/903212625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 msgr2=0x55f6a2aa26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa26c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe3740098e0 tx=0x7fe374005fb0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 -- 172.21.15.31:0/903212625 shutdown_connections 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aa94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 --2- 172.21.15.31:0/903212625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa1530 0x55f6a2aa26c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 -- 172.21.15.31:0/903212625 >> 172.21.15.31:0/903212625 conn(0x55f6a2a963e0 msgr2=0x55f6a2a9b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 -- 172.21.15.31:0/903212625 shutdown_connections 2023-07-29T16:20:43.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 -- 172.21.15.31:0/903212625 wait complete. 2023-07-29T16:20:43.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.743+0000 7fe399b09b80 1 -- start start 2023-07-29T16:20:43.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 Processor -- start 2023-07-29T16:20:43.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aadc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa1530 0x55f6a2aae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aaf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2aaf6f0 con 0x55f6a2aa0be0 2023-07-29T16:20:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2aaf860 con 0x55f6a2aa22f0 2023-07-29T16:20:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6a2aafa20 con 0x55f6a2aa1530 2023-07-29T16:20:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe399b09b80 10 client.0 _ll_register_callbacks cb 0x55f6a2997620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aadc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe37effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa1530 0x55f6a2aae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe37f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aadc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43622/0 (socket says 172.21.15.31:43622) 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.744+0000 7fe37f7fe700 1 -- 172.21.15.31:0/1601421748 learned_addr learned my addr 172.21.15.31:0/1601421748 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37ffff700 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aaf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37effd700 1 -- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 msgr2=0x55f6a2aaf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37effd700 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aaf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37effd700 1 -- 172.21.15.31:0/1601421748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 msgr2=0x55f6a2aadc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37effd700 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aadc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37effd700 1 -- 172.21.15.31:0/1601421748 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3801090c0 con 0x55f6a2aa1530 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe37effd700 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa1530 0x55f6a2aae1a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe37402a040 tx=0x7fe374009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdd75bcab43c8baa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.745+0000 7fe35effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe35effd700 1 -- 172.21.15.31:0/1601421748 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37401d040 con 0x55f6a2aa1530 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe35effd700 1 -- 172.21.15.31:0/1601421748 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe37400fa30 con 0x55f6a2aa1530 2023-07-29T16:20:43.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3700977a0 con 0x55f6a2aa1530 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe35effd700 1 -- 172.21.15.31:0/1601421748 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe37400fbf0 con 0x55f6a2aa1530 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe399b09b80 10 client.27175 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f6a281e2b0 con 0x55f6a2aa1530 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe35effd700 1 -- 172.21.15.31:0/1601421748 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe374040020 con 0x55f6a2aa1530 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe35effd700 1 -- 172.21.15.31:0/1601421748 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fe374025430 con 0x55f6a2aa1530 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.746+0000 7fe399b09b80 10 client.27175 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe399b09b80 10 client.27175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:43.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe399b09b80 20 client.27175 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe399b09b80 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2b11620 0x55f6a2b13a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe399b09b80 4 client.27175 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f6a29ead40 con 0x55f6a2b11620 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2b11620 0x55f6a2b13a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe37f7fe700 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2b11620 0x55f6a2b13a00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91652519c93b6833 server_cookie=a7aebe8ce407f4f3 in_seq=0 out_seq=0 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe35effd700 10 client.27175 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:43.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe35effd700 1 -- 172.21.15.31:0/1601421748 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f6a29ead40 con 0x55f6a2b11620 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe35effd700 10 client.27175 handle_command_reply: tid=0 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.747+0000 7fe399b09b80 1 client.27175 shutdown 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa1530 msgr2=0x55f6a2aae1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa1530 0x55f6a2aae1a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe37402a040 tx=0x7fe374009510 comp rx=0 tx=0).stop 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 shutdown_connections 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f6a2b11620 0x55f6a2b13a00 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f6a2aa22f0 0x55f6a2aaf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f6a2aa0be0 0x55f6a2aadc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 --2- 172.21.15.31:0/1601421748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f6a2aa1530 0x55f6a2aae1a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 >> 172.21.15.31:0/1601421748 conn(0x55f6a2a963e0 msgr2=0x55f6a2aa86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe35effd700 0 client.27175 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:43.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 shutdown_connections 2023-07-29T16:20:43.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 1 -- 172.21.15.31:0/1601421748 wait complete. 2023-07-29T16:20:43.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.748+0000 7fe399b09b80 20 client.27175 trim_cache size 0 max 16384 2023-07-29T16:20:43.754 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:20:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3680a0810 msgr2=0x7fe3680a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3680a0810 0x7fe3680a2cc0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe378009fd0 tx=0x7fe378009920 comp rx=0 tx=0).stop 2023-07-29T16:20:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 msgr2=0x7fe380198240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380198240 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe37000e970 tx=0x7fe37000ec80 comp rx=0 tx=0).stop 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 shutdown_connections 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe3680a0810 0x7fe3680a2cc0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe380101ed0 0x7fe38019f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe380100350 0x7fe380198240 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 --2- 172.21.15.31:0/625593690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe380101540 0x7fe380198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 >> 172.21.15.31:0/625593690 conn(0x7fe3800fb970 msgr2=0x7fe380106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:43.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 shutdown_connections 2023-07-29T16:20:43.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:43.749+0000 7fe3861ff700 1 -- 172.21.15.31:0/625593690 wait complete. 2023-07-29T16:20:43.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:43 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/797919787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:20:43.765 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:20:43.765 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:20:43.765 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:20:43.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:43 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/797919787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:20:43.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:20:43.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:20:43.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:20:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:43 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/797919787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:20:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:20:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:20:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:20:43.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 250155 8.88 MB/sec execute 28 sec latency 41.414 ms 2023-07-29T16:20:44.768 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:20:44.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:20:44.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:44 smithi039 ceph-mon[128858]: pgmap v528: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 292 op/s 2023-07-29T16:20:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:44 smithi156 ceph-mon[129245]: pgmap v528: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 292 op/s 2023-07-29T16:20:44.830 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:44 smithi031 ceph-mon[95504]: pgmap v528: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 292 op/s 2023-07-29T16:20:44.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 252154 9.04 MB/sec execute 29 sec latency 19.678 ms 2023-07-29T16:20:45.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 -- 172.21.15.31:0/747068142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c074d90 msgr2=0x7ff78c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 --2- 172.21.15.31:0/747068142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c074d90 0x7ff78c0731e0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff77c009990 tx=0x7ff77c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 -- 172.21.15.31:0/747068142 shutdown_connections 2023-07-29T16:20:45.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 --2- 172.21.15.31:0/747068142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff78c10c8f0 0x7ff78c10ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 --2- 172.21.15.31:0/747068142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c074d90 0x7ff78c0731e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 --2- 172.21.15.31:0/747068142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff78c0737b0 0x7ff78c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 -- 172.21.15.31:0/747068142 >> 172.21.15.31:0/747068142 conn(0x7ff78c0fb920 msgr2=0x7ff78c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 -- 172.21.15.31:0/747068142 shutdown_connections 2023-07-29T16:20:45.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.025+0000 7ff791bff700 1 -- 172.21.15.31:0/747068142 wait complete. 2023-07-29T16:20:45.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.026+0000 7ff791bff700 1 Processor -- start 2023-07-29T16:20:45.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 -- start start 2023-07-29T16:20:45.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 0x7ff78c19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff78c074d90 0x7ff78c19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff78c10c8f0 0x7ff78c1a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c1a42a0 con 0x7ff78c0737b0 2023-07-29T16:20:45.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c1a43e0 con 0x7ff78c10c8f0 2023-07-29T16:20:45.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff791bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c1a4570 con 0x7ff78c074d90 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 0x7ff78c19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff78c074d90 0x7ff78c19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff78c10c8f0 0x7ff78c1a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 0x7ff78c19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43648/0 (socket says 172.21.15.31:43648) 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 -- 172.21.15.31:0/2823265941 learned_addr learned my addr 172.21.15.31:0/2823265941 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 -- 172.21.15.31:0/2823265941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff78c074d90 msgr2=0x7ff78c19cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff78c074d90 0x7ff78c19cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 -- 172.21.15.31:0/2823265941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff78c10c8f0 msgr2=0x7ff78c1a3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff78c10c8f0 0x7ff78c1a3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.027+0000 7ff78b7fe700 1 -- 172.21.15.31:0/2823265941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff77c0095f0 con 0x7ff78c0737b0 2023-07-29T16:20:45.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.028+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 0x7ff78c19c720 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff77c004710 tx=0x7ff77c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.028+0000 7ff788ff9700 1 -- 172.21.15.31:0/2823265941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c01d070 con 0x7ff78c0737b0 2023-07-29T16:20:45.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.028+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c1a47f0 con 0x7ff78c0737b0 2023-07-29T16:20:45.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.028+0000 7ff788ff9700 1 -- 172.21.15.31:0/2823265941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77c021410 con 0x7ff78c0737b0 2023-07-29T16:20:45.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.028+0000 7ff788ff9700 1 -- 172.21.15.31:0/2823265941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c017650 con 0x7ff78c0737b0 2023-07-29T16:20:45.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.028+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1a4c70 con 0x7ff78c0737b0 2023-07-29T16:20:45.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.033+0000 7ff788ff9700 1 -- 172.21.15.31:0/2823265941 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff77c021580 con 0x7ff78c0737b0 2023-07-29T16:20:45.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 Processor -- start 2023-07-29T16:20:45.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 -- start start 2023-07-29T16:20:45.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cb4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cb4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cb57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38c0ba90 con 0x562a38cb45b0 2023-07-29T16:20:45.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38c09450 con 0x562a38cb5370 2023-07-29T16:20:45.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38c04d40 con 0x562a38cb3c60 2023-07-29T16:20:45.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff78affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cb57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cb4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cb4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cb4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48700/0 (socket says 172.21.15.31:48700) 2023-07-29T16:20:45.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff78bfff700 1 -- 172.21.15.31:0/1490785120 learned_addr learned my addr 172.21.15.31:0/1490785120 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.034+0000 7ff78affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cb57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55704/0 (socket says 172.21.15.31:55704) 2023-07-29T16:20:45.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff788ff9700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff7740a0790 0x7ff7740a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78affd700 1 -- 172.21.15.31:0/1490785120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 msgr2=0x562a38cb4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78affd700 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cb4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78affd700 1 -- 172.21.15.31:0/1490785120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 msgr2=0x562a38cb4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78affd700 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cb4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78affd700 1 -- 172.21.15.31:0/1490785120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a38a32e60 con 0x562a38cb5370 2023-07-29T16:20:45.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff788ff9700 1 -- 172.21.15.31:0/2823265941 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff77c0cceb0 con 0x7ff78c0737b0 2023-07-29T16:20:45.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78bfff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff7740a0790 0x7ff7740a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff78affd700 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cb57c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff780009c80 tx=0x7ff780009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b60307b3a9d41dec server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.035+0000 7ff771ffb700 1 -- 172.21.15.31:0/1490785120 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780011cb0 con 0x562a38cb5370 2023-07-29T16:20:45.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff771ffb700 1 -- 172.21.15.31:0/1490785120 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff780011e10 con 0x562a38cb5370 2023-07-29T16:20:45.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff771ffb700 1 -- 172.21.15.31:0/1490785120 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780005280 con 0x562a38cb5370 2023-07-29T16:20:45.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1490785120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 msgr2=0x562a38cb57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff78bfff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff7740a0790 0x7ff7740a2c40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x562a38cb3a00 tx=0x7ff784009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cb57c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff780009c80 tx=0x7ff780009f90 comp rx=0 tx=0).stop 2023-07-29T16:20:45.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1490785120 shutdown_connections 2023-07-29T16:20:45.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cb57c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cb4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1490785120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cb4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.036+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1490785120 >> 172.21.15.31:0/1490785120 conn(0x562a38ca9460 msgr2=0x562a38caea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.037+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1490785120 shutdown_connections 2023-07-29T16:20:45.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.037+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1490785120 wait complete. 2023-07-29T16:20:45.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.037+0000 7ff7a55c0b80 1 -- start start 2023-07-29T16:20:45.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.037+0000 7ff7a55c0b80 1 Processor -- start 2023-07-29T16:20:45.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb3c60 0x562a38cbd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cbddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cc4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38ba1590 con 0x562a38cb3c60 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38ba1230 con 0x562a38cb5370 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38a889a0 con 0x562a38cb45b0 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff7a55c0b80 10 client.0 _ll_register_callbacks cb 0x562a38baa6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cc4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cbddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cbddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48712/0 (socket says 172.21.15.31:48712) 2023-07-29T16:20:45.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 -- 172.21.15.31:0/735914228 learned_addr learned my addr 172.21.15.31:0/735914228 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cc4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55710/0 (socket says 172.21.15.31:55710) 2023-07-29T16:20:45.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb3c60 0x562a38cbd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 -- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 msgr2=0x562a38cc4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cc4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 -- 172.21.15.31:0/735914228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb3c60 msgr2=0x562a38cbd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb3c60 0x562a38cbd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.038+0000 7ff78affd700 1 -- 172.21.15.31:0/735914228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7800098e0 con 0x562a38cb45b0 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff78bfff700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cc4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff78affd700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cbddf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff780005e90 tx=0x7ff780011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d1b20ab0b90bb00 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff76affd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff76affd700 1 -- 172.21.15.31:0/735914228 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780004020 con 0x562a38cb45b0 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a38c2fd90 con 0x562a38cb45b0 2023-07-29T16:20:45.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff76affd700 1 -- 172.21.15.31:0/735914228 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff780004180 con 0x562a38cb45b0 2023-07-29T16:20:45.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff76affd700 1 -- 172.21.15.31:0/735914228 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff780004340 con 0x562a38cb45b0 2023-07-29T16:20:45.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff7a55c0b80 10 client.27187 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:45.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.039+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562a38a1bce0 con 0x562a38cb45b0 2023-07-29T16:20:45.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff76affd700 1 -- 172.21.15.31:0/735914228 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562a38c2fd90 con 0x562a38cb45b0 2023-07-29T16:20:45.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff76affd700 1 -- 172.21.15.31:0/735914228 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff780003990 con 0x562a38cb45b0 2023-07-29T16:20:45.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff7a55c0b80 10 client.27187 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff7a55c0b80 10 client.27187 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff7a55c0b80 20 client.27187 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38cc9560 0x562a38ccb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff7a55c0b80 4 client.27187 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a38c82df0 con 0x562a38cc9560 2023-07-29T16:20:45.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.040+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38cc9560 0x562a38ccb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.041+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38cc9560 0x562a38ccb970 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d54ec772f8f49d9 server_cookie=a383a21520004dfc in_seq=0 out_seq=0 2023-07-29T16:20:45.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.041+0000 7ff76affd700 10 client.27187 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff76affd700 1 -- 172.21.15.31:0/735914228 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562a38c82df0 con 0x562a38cc9560 2023-07-29T16:20:45.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff76affd700 10 client.27187 handle_command_reply: tid=0 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 client.27187 shutdown 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 msgr2=0x562a38cbddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cbddf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff780005e90 tx=0x7ff780011ab0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 shutdown_connections 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38cc9560 0x562a38ccb970 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38cc4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff76affd700 0 client.27187 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb3c60 0x562a38cbd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/735914228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cbddf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 >> 172.21.15.31:0/735914228 conn(0x562a38ca9460 msgr2=0x562a38cae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 shutdown_connections 2023-07-29T16:20:45.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.042+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/735914228 wait complete. 2023-07-29T16:20:45.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.043+0000 7ff7a55c0b80 20 client.27187 trim_cache size 0 max 16384 2023-07-29T16:20:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.062+0000 7ff7a55c0b80 1 Processor -- start 2023-07-29T16:20:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.062+0000 7ff7a55c0b80 1 -- start start 2023-07-29T16:20:45.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.062+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 0x562a38cc2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.062+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cb8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb5370 0x562a38cb8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38cafd40 con 0x562a38cb5370 2023-07-29T16:20:45.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38cbc070 con 0x562a38cb3c60 2023-07-29T16:20:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38ca6760 con 0x562a38cb45b0 2023-07-29T16:20:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 0x562a38cc2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb5370 0x562a38cb8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 0x562a38cc2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55726/0 (socket says 172.21.15.31:55726) 2023-07-29T16:20:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 -- 172.21.15.31:0/1717548666 learned_addr learned my addr 172.21.15.31:0/1717548666 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78affd700 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cb8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 -- 172.21.15.31:0/1717548666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 msgr2=0x562a38cb8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cb8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 -- 172.21.15.31:0/1717548666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb5370 msgr2=0x562a38cb8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb5370 0x562a38cb8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 -- 172.21.15.31:0/1717548666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a38cb5a10 con 0x562a38cb3c60 2023-07-29T16:20:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.063+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 0x562a38cc2de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff77c097050 tx=0x7ff77c0a0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a05ed3cce1d8973 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7727fc700 1 -- 172.21.15.31:0/1717548666 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c09a120 con 0x562a38cb3c60 2023-07-29T16:20:45.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7727fc700 1 -- 172.21.15.31:0/1717548666 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77c099f40 con 0x562a38cb3c60 2023-07-29T16:20:45.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7727fc700 1 -- 172.21.15.31:0/1717548666 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c09a840 con 0x562a38cb3c60 2023-07-29T16:20:45.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1717548666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 msgr2=0x562a38cc2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 0x562a38cc2de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff77c097050 tx=0x7ff77c0a0040 comp rx=0 tx=0).stop 2023-07-29T16:20:45.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1717548666 shutdown_connections 2023-07-29T16:20:45.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb5370 0x562a38cb8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb3c60 0x562a38cc2de0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/1717548666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb45b0 0x562a38cb8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1717548666 >> 172.21.15.31:0/1717548666 conn(0x562a38ca9460 msgr2=0x562a38cbd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1717548666 shutdown_connections 2023-07-29T16:20:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.064+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/1717548666 wait complete. 2023-07-29T16:20:45.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 -- start start 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 Processor -- start 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cc2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cc2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38d1b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38d1b9e0 con 0x562a38cb45b0 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38d1bb20 con 0x562a38cb5370 2023-07-29T16:20:45.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a38d1bce0 con 0x562a38cb3c60 2023-07-29T16:20:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.065+0000 7ff7a55c0b80 10 client.0 _ll_register_callbacks cb 0x562a38baa6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cc2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cc2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38d1b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cc2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48728/0 (socket says 172.21.15.31:48728) 2023-07-29T16:20:45.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 -- 172.21.15.31:0/519562958 learned_addr learned my addr 172.21.15.31:0/519562958 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 -- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 msgr2=0x562a38d1b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38d1b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 -- 172.21.15.31:0/519562958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 msgr2=0x562a38cc2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cc2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 -- 172.21.15.31:0/519562958 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1a4c70 con 0x562a38cb3c60 2023-07-29T16:20:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78affd700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cc2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78bfff700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38d1b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff78b7fe700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cc2210 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff77c096ae0 tx=0x7ff77c09a120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b1260087a75e271 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.066+0000 7ff76affd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff76affd700 1 -- 172.21.15.31:0/519562958 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff77c09a5e0 con 0x562a38cb3c60 2023-07-29T16:20:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff76affd700 1 -- 172.21.15.31:0/519562958 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff77c0a29a0 con 0x562a38cb3c60 2023-07-29T16:20:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff780011e10 con 0x562a38cb3c60 2023-07-29T16:20:45.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff76affd700 1 -- 172.21.15.31:0/519562958 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff77c099ad0 con 0x562a38cb3c60 2023-07-29T16:20:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 10 client.27193 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562a38cbc830 con 0x562a38cb3c60 2023-07-29T16:20:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff76affd700 1 -- 172.21.15.31:0/519562958 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff77c064080 con 0x562a38cb3c60 2023-07-29T16:20:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff76affd700 1 -- 172.21.15.31:0/519562958 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7ff77c0a2510 con 0x562a38cb3c60 2023-07-29T16:20:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 10 client.27193 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:45.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 10 client.27193 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 20 client.27193 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38d1d240 0x562a38d1f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 4 client.27193 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:20:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562a38bfddc0 con 0x562a38d1d240 2023-07-29T16:20:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.067+0000 7ff78affd700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38d1d240 0x562a38d1f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.068+0000 7ff78affd700 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38d1d240 0x562a38d1f650 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbc5d188416adce1 server_cookie=c6e98fbe04152372 in_seq=0 out_seq=0 2023-07-29T16:20:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.068+0000 7ff76affd700 10 client.27193 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff76affd700 1 -- 172.21.15.31:0/519562958 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562a38bfddc0 con 0x562a38d1d240 2023-07-29T16:20:45.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff76affd700 10 client.27193 handle_command_reply: tid=0 2023-07-29T16:20:45.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 client.27193 shutdown 2023-07-29T16:20:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 msgr2=0x562a38cc2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cc2210 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff77c096ae0 tx=0x7ff77c09a120 comp rx=0 tx=0).stop 2023-07-29T16:20:45.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 shutdown_connections 2023-07-29T16:20:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562a38d1d240 0x562a38d1f650 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a38cb5370 0x562a38d1b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a38cb3c60 0x562a38cc2210 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 --2- 172.21.15.31:0/519562958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a38cb45b0 0x562a38cc2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff76affd700 0 client.27193 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 >> 172.21.15.31:0/519562958 conn(0x562a38ca9460 msgr2=0x562a38cbd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 shutdown_connections 2023-07-29T16:20:45.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 1 -- 172.21.15.31:0/519562958 wait complete. 2023-07-29T16:20:45.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.087+0000 7ff7a55c0b80 20 client.27193 trim_cache size 0 max 16384 2023-07-29T16:20:45.091 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:20:45.091 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:20:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "57afcdb8-8ff8-4fa4-9abc-57ae2358e64b", 2023-07-29T16:20:45.092 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:20:45.092 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:20:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff7740a0790 msgr2=0x7ff7740a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff7740a0790 0x7ff7740a2c40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x562a38cb3a00 tx=0x7ff784009040 comp rx=0 tx=0).stop 2023-07-29T16:20:45.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 msgr2=0x7ff78c19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 0x7ff78c19c720 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff77c004710 tx=0x7ff77c009580 comp rx=0 tx=0).stop 2023-07-29T16:20:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 shutdown_connections 2023-07-29T16:20:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff7740a0790 0x7ff7740a2c40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff78c10c8f0 0x7ff78c1a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff78c0737b0 0x7ff78c19c720 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 --2- 172.21.15.31:0/2823265941 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff78c074d90 0x7ff78c19cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 >> 172.21.15.31:0/2823265941 conn(0x7ff78c0fb920 msgr2=0x7ff78c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 shutdown_connections 2023-07-29T16:20:45.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.088+0000 7ff791bff700 1 -- 172.21.15.31:0/2823265941 wait complete. 2023-07-29T16:20:45.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:20:45.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 -- 172.21.15.31:0/3562927203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94100330 msgr2=0x7f3b94100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 --2- 172.21.15.31:0/3562927203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94100330 0x7f3b94100740 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f3b840098f0 tx=0x7f3b84009c00 comp rx=0 tx=0).stop 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 -- 172.21.15.31:0/3562927203 shutdown_connections 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 --2- 172.21.15.31:0/3562927203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94101eb0 0x7f3b94108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 --2- 172.21.15.31:0/3562927203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94100330 0x7f3b94100740 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 --2- 172.21.15.31:0/3562927203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 0x7f3b94101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 -- 172.21.15.31:0/3562927203 >> 172.21.15.31:0/3562927203 conn(0x7f3b940fb970 msgr2=0x7f3b940fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 -- 172.21.15.31:0/3562927203 shutdown_connections 2023-07-29T16:20:45.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.326+0000 7f3b99dff700 1 -- 172.21.15.31:0/3562927203 wait complete. 2023-07-29T16:20:45.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 Processor -- start 2023-07-29T16:20:45.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 -- start start 2023-07-29T16:20:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 0x7f3b94074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 0x7f3b94073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94101eb0 0x7f3b94073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94073e20 con 0x7f3b94101eb0 2023-07-29T16:20:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94073f90 con 0x7f3b94100330 2023-07-29T16:20:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.327+0000 7f3b99dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941a0170 con 0x7f3b94101520 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 0x7f3b94074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 0x7f3b94073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 0x7f3b94074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55744/0 (socket says 172.21.15.31:55744) 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b93fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94101eb0 0x7f3b94073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b937fe700 1 -- 172.21.15.31:0/1961613772 learned_addr learned my addr 172.21.15.31:0/1961613772 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 -- 172.21.15.31:0/1961613772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 msgr2=0x7f3b94074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 0x7f3b94074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 -- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94101eb0 msgr2=0x7f3b94073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94101eb0 0x7f3b94073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 -- 172.21.15.31:0/1961613772 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c009d00 con 0x7f3b94101520 2023-07-29T16:20:45.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b937fe700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 0x7f3b94074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:45.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.328+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 0x7f3b94073140 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800e9f0 tx=0x7f3b8800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.329+0000 7f3b90ff9700 1 -- 172.21.15.31:0/1961613772 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8800c9c0 con 0x7f3b94101520 2023-07-29T16:20:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.329+0000 7f3b90ff9700 1 -- 172.21.15.31:0/1961613772 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b88010430 con 0x7f3b94101520 2023-07-29T16:20:45.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.329+0000 7f3b90ff9700 1 -- 172.21.15.31:0/1961613772 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b88018680 con 0x7f3b94101520 2023-07-29T16:20:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.329+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b84009520 con 0x7f3b94101520 2023-07-29T16:20:45.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.329+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b941a0780 con 0x7f3b94101520 2023-07-29T16:20:45.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.332+0000 7f3b90ff9700 1 -- 172.21.15.31:0/1961613772 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3b88010860 con 0x7f3b94101520 2023-07-29T16:20:45.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.333+0000 7f3bad74eb80 1 Processor -- start 2023-07-29T16:20:45.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.333+0000 7f3b90ff9700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b7c0a0810 0x7f3b7c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.333+0000 7f3bad74eb80 1 -- start start 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b90ff9700 1 -- 172.21.15.31:0/1961613772 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b88014070 con 0x7f3b94101520 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b93fff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b7c0a0810 0x7f3b7c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c05ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c06d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c07740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3b5da10 con 0x5559d3c06530 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3b5b3d0 con 0x5559d3c05be0 2023-07-29T16:20:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3b56cc0 con 0x5559d3c072f0 2023-07-29T16:20:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b92ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c06d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c05ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c05ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55756/0 (socket says 172.21.15.31:55756) 2023-07-29T16:20:45.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b937fe700 1 -- 172.21.15.31:0/2603130591 learned_addr learned my addr 172.21.15.31:0/2603130591 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.334+0000 7f3b93fff700 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c07740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b937fe700 1 -- 172.21.15.31:0/2603130591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 msgr2=0x5559d3c07740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b937fe700 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c07740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b937fe700 1 -- 172.21.15.31:0/2603130591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 msgr2=0x5559d3c06d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b937fe700 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c06d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b937fe700 1 -- 172.21.15.31:0/2603130591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d3984de0 con 0x5559d3c05be0 2023-07-29T16:20:45.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b93fff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b7c0a0810 0x7f3b7c0a2cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3b94075160 tx=0x7f3b8c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b937fe700 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c05ff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3b84000c00 tx=0x7f3b84005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c980c619a49baf0d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b79ffb700 1 -- 172.21.15.31:0/2603130591 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8401c050 con 0x5559d3c05be0 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b79ffb700 1 -- 172.21.15.31:0/2603130591 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b8401d040 con 0x5559d3c05be0 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.335+0000 7f3b79ffb700 1 -- 172.21.15.31:0/2603130591 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b840053a0 con 0x5559d3c05be0 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 -- 172.21.15.31:0/2603130591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 msgr2=0x5559d3c05ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c05ff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f3b84000c00 tx=0x7f3b84005dc0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 -- 172.21.15.31:0/2603130591 shutdown_connections 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c07740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c05ff0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/2603130591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c06d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 -- 172.21.15.31:0/2603130591 >> 172.21.15.31:0/2603130591 conn(0x5559d3bfb3e0 msgr2=0x5559d3c00990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 -- 172.21.15.31:0/2603130591 shutdown_connections 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.336+0000 7f3bad74eb80 1 -- 172.21.15.31:0/2603130591 wait complete. 2023-07-29T16:20:45.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.337+0000 7f3bad74eb80 1 -- start start 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.337+0000 7f3bad74eb80 1 Processor -- start 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c0f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c072f0 0x5559d3c16d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3af3510 con 0x5559d3c072f0 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3af31b0 con 0x5559d3c05be0 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d39da920 con 0x5559d3c06530 2023-07-29T16:20:45.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3bad74eb80 10 client.0 _ll_register_callbacks cb 0x5559d3afc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:45.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c0f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b92ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b93fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c072f0 0x5559d3c16d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b92ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48774/0 (socket says 172.21.15.31:48774) 2023-07-29T16:20:45.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b92ffd700 1 -- 172.21.15.31:0/799398729 learned_addr learned my addr 172.21.15.31:0/799398729 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c0f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55770/0 (socket says 172.21.15.31:55770) 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 -- 172.21.15.31:0/799398729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 msgr2=0x5559d3c0fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 -- 172.21.15.31:0/799398729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c072f0 msgr2=0x5559d3c16d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c072f0 0x5559d3c16d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.338+0000 7f3b937fe700 1 -- 172.21.15.31:0/799398729 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b941090a0 con 0x5559d3c05be0 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3b937fe700 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c0f7b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400f9e0 tx=0x7f3b8400b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75cf39fa9893b2ff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3b72ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:45.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3b72ffd700 1 -- 172.21.15.31:0/799398729 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84005530 con 0x5559d3c05be0 2023-07-29T16:20:45.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559d3b81d10 con 0x5559d3c05be0 2023-07-29T16:20:45.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3b72ffd700 1 -- 172.21.15.31:0/799398729 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b84005690 con 0x5559d3c05be0 2023-07-29T16:20:45.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3b72ffd700 1 -- 172.21.15.31:0/799398729 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b8401c030 con 0x5559d3c05be0 2023-07-29T16:20:45.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3bad74eb80 10 client.27194 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:45.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5559d396dc60 con 0x5559d3c05be0 2023-07-29T16:20:45.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.339+0000 7f3b72ffd700 1 -- 172.21.15.31:0/799398729 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5559d3b81d10 con 0x5559d3c05be0 2023-07-29T16:20:45.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3b72ffd700 1 -- 172.21.15.31:0/799398729 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f3b8402a070 con 0x5559d3c05be0 2023-07-29T16:20:45.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3bad74eb80 10 client.27194 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:45.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3bad74eb80 10 client.27194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:45.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3bad74eb80 20 client.27194 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:45.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c1b060 0x5559d3c1d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3bad74eb80 4 client.27194 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:45.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559d3bd4d70 con 0x5559d3c1b060 2023-07-29T16:20:45.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.340+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c1b060 0x5559d3c1d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c1b060 0x5559d3c1d470 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd92a660978b2ecf server_cookie=83437aa07943e615 in_seq=0 out_seq=0 2023-07-29T16:20:45.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3b72ffd700 10 client.27194 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3b72ffd700 1 -- 172.21.15.31:0/799398729 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5559d3bd4d70 con 0x5559d3c1b060 2023-07-29T16:20:45.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3b72ffd700 10 client.27194 handle_command_reply: tid=0 2023-07-29T16:20:45.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3bad74eb80 1 client.27194 shutdown 2023-07-29T16:20:45.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 msgr2=0x5559d3c0f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.341+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c0f7b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400f9e0 tx=0x7f3b8400b730 comp rx=0 tx=0).stop 2023-07-29T16:20:45.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 shutdown_connections 2023-07-29T16:20:45.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c1b060 0x5559d3c1d470 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c072f0 0x5559d3c16d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c0f7b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/799398729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 >> 172.21.15.31:0/799398729 conn(0x5559d3bfb3e0 msgr2=0x5559d3c00060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3b72ffd700 0 client.27194 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 shutdown_connections 2023-07-29T16:20:45.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 1 -- 172.21.15.31:0/799398729 wait complete. 2023-07-29T16:20:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.342+0000 7f3bad74eb80 20 client.27194 trim_cache size 0 max 16384 2023-07-29T16:20:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 Processor -- start 2023-07-29T16:20:45.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 -- start start 2023-07-29T16:20:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c05be0 0x5559d3c14ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c072f0 0x5559d3c0a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3b81d10 con 0x5559d3c05be0 2023-07-29T16:20:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3bfaf40 con 0x5559d3c072f0 2023-07-29T16:20:45.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.360+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3bfd460 con 0x5559d3c06530 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c072f0 0x5559d3c0a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b92ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b93fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c072f0 0x5559d3c0a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55778/0 (socket says 172.21.15.31:55778) 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b93fff700 1 -- 172.21.15.31:0/1978968643 learned_addr learned my addr 172.21.15.31:0/1978968643 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b937fe700 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c05be0 0x5559d3c14ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b92ffd700 1 -- 172.21.15.31:0/1978968643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c072f0 msgr2=0x5559d3c0a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c072f0 0x5559d3c0a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b92ffd700 1 -- 172.21.15.31:0/1978968643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c05be0 msgr2=0x5559d3c14ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c05be0 0x5559d3c14ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b92ffd700 1 -- 172.21.15.31:0/1978968643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559d39ab250 con 0x5559d3c06530 2023-07-29T16:20:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.361+0000 7f3b937fe700 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c05be0 0x5559d3c14ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:45.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.362+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0a1f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3b88095880 tx=0x7f3b8809d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b2ccdad0833bb01 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.362+0000 7f3b7a7fc700 1 -- 172.21.15.31:0/1978968643 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b880b1070 con 0x5559d3c06530 2023-07-29T16:20:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.362+0000 7f3b7a7fc700 1 -- 172.21.15.31:0/1978968643 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b88099380 con 0x5559d3c06530 2023-07-29T16:20:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.362+0000 7f3b7a7fc700 1 -- 172.21.15.31:0/1978968643 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b880105a0 con 0x5559d3c06530 2023-07-29T16:20:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1978968643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 msgr2=0x5559d3c0a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0a1f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3b88095880 tx=0x7f3b8809d040 comp rx=0 tx=0).stop 2023-07-29T16:20:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1978968643 shutdown_connections 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c072f0 0x5559d3c0a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c05be0 0x5559d3c14ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1978968643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c06530 0x5559d3c0a1f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1978968643 >> 172.21.15.31:0/1978968643 conn(0x5559d3bfb3e0 msgr2=0x5559d3c0d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1978968643 shutdown_connections 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1978968643 wait complete. 2023-07-29T16:20:45.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.363+0000 7f3bad74eb80 1 -- start start 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 Processor -- start 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c13490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c139d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c13f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3c6cba0 con 0x5559d3c06530 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3c6cce0 con 0x5559d3c05be0 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559d3c6ce20 con 0x5559d3c072f0 2023-07-29T16:20:45.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3bad74eb80 10 client.0 _ll_register_callbacks cb 0x5559d3afc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c13490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3b937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c13490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55790/0 (socket says 172.21.15.31:55790) 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3b937fe700 1 -- 172.21.15.31:0/1281343465 learned_addr learned my addr 172.21.15.31:0/1281343465 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3b93fff700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c13f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.364+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b937fe700 1 -- 172.21.15.31:0/1281343465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 msgr2=0x5559d3c13f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b937fe700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c13f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b937fe700 1 -- 172.21.15.31:0/1281343465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 msgr2=0x5559d3c139d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b937fe700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c139d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b937fe700 1 -- 172.21.15.31:0/1281343465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b941a0780 con 0x5559d3c05be0 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b937fe700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c13490 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3b8402dfb0 tx=0x7f3b84023500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9f076e60c5d8086 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b72ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b72ffd700 1 -- 172.21.15.31:0/1281343465 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8401c040 con 0x5559d3c05be0 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b72ffd700 1 -- 172.21.15.31:0/1281343465 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3b84000e40 con 0x5559d3c05be0 2023-07-29T16:20:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b8401d040 con 0x5559d3c05be0 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3b72ffd700 1 -- 172.21.15.31:0/1281343465 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b84005710 con 0x5559d3c05be0 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3bad74eb80 10 client.27200 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.365+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5559d39832b0 con 0x5559d3c05be0 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3b72ffd700 1 -- 172.21.15.31:0/1281343465 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b8401d040 con 0x5559d3c05be0 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3b72ffd700 1 -- 172.21.15.31:0/1281343465 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f3b84041070 con 0x5559d3c05be0 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3bad74eb80 10 client.27200 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3bad74eb80 10 client.27200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3bad74eb80 20 client.27200 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c6e380 0x5559d3c70790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3bad74eb80 4 client.27200 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:20:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5559d3b4fd40 con 0x5559d3c6e380 2023-07-29T16:20:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.366+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c6e380 0x5559d3c70790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3b92ffd700 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c6e380 0x5559d3c70790 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92fd42c216623cb server_cookie=e9041cb2f5e3a5e5 in_seq=0 out_seq=0 2023-07-29T16:20:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3b72ffd700 10 client.27200 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3b72ffd700 1 -- 172.21.15.31:0/1281343465 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5559d3b4fd40 con 0x5559d3c6e380 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3b72ffd700 10 client.27200 handle_command_reply: tid=0 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3bad74eb80 1 client.27200 shutdown 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 msgr2=0x5559d3c13490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.367+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c13490 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f3b8402dfb0 tx=0x7f3b84023500 comp rx=0 tx=0).stop 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 shutdown_connections 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5559d3c6e380 0x5559d3c70790 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559d3c072f0 0x5559d3c13f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5559d3c05be0 0x5559d3c13490 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3b72ffd700 0 client.27200 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 --2- 172.21.15.31:0/1281343465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559d3c06530 0x5559d3c139d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 >> 172.21.15.31:0/1281343465 conn(0x5559d3bfb3e0 msgr2=0x5559d3c0a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 shutdown_connections 2023-07-29T16:20:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 1 -- 172.21.15.31:0/1281343465 wait complete. 2023-07-29T16:20:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3bad74eb80 20 client.27200 trim_cache size 0 max 16384 2023-07-29T16:20:45.375 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:20:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:20:45.375 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "57afcdb8-8ff8-4fa4-9abc-57ae2358e64b": { 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "57afcdb8-8ff8-4fa4-9abc-57ae2358e64b", 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:20:45.376 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:20:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b7c0a0810 msgr2=0x7f3b7c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3b99dff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b7c0a0810 0x7f3b7c0a2cc0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3b94075160 tx=0x7f3b8c00b040 comp rx=0 tx=0).stop 2023-07-29T16:20:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 msgr2=0x7f3b94073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:45.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.368+0000 7f3b99dff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 0x7f3b94073140 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800e9f0 tx=0x7f3b8800ed00 comp rx=0 tx=0).stop 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 shutdown_connections 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3b7c0a0810 0x7f3b7c0a2cc0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b94101eb0 0x7f3b94073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b94100330 0x7f3b94074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 --2- 172.21.15.31:0/1961613772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b94101520 0x7f3b94073140 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 >> 172.21.15.31:0/1961613772 conn(0x7f3b940fb970 msgr2=0x7f3b940fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:45.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 shutdown_connections 2023-07-29T16:20:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:20:45.369+0000 7f3b99dff700 1 -- 172.21.15.31:0/1961613772 wait complete. 2023-07-29T16:20:45.386 INFO:tasks.cephfs.filesystem:scrub status for tag:57afcdb8-8ff8-4fa4-9abc-57ae2358e64b - {'path': '/', 'tag': '57afcdb8-8ff8-4fa4-9abc-57ae2358e64b', 'options': 'recursive,force'} 2023-07-29T16:20:45.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 253063 8.81 MB/sec execute 30 sec latency 110.244 ms 2023-07-29T16:20:46.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:20:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:20:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:20:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:20:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:46 smithi039 ceph-mon[128858]: pgmap v529: 97 pgs: 97 active+clean; 71 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 281 op/s 2023-07-29T16:20:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:20:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:46 smithi156 ceph-mon[129245]: pgmap v529: 97 pgs: 97 active+clean; 71 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 281 op/s 2023-07-29T16:20:46.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 255554 9.04 MB/sec execute 31 sec latency 16.358 ms 2023-07-29T16:20:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:20:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:20:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:46 smithi031 ceph-mon[95504]: pgmap v529: 97 pgs: 97 active+clean; 71 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 281 op/s 2023-07-29T16:20:47.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 256603 8.86 MB/sec execute 32 sec latency 17.296 ms 2023-07-29T16:20:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:47 smithi031 ceph-mon[95504]: pgmap v530: 97 pgs: 97 active+clean; 69 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2023-07-29T16:20:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:47 smithi039 ceph-mon[128858]: pgmap v530: 97 pgs: 97 active+clean; 69 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2023-07-29T16:20:48.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:47 smithi156 ceph-mon[129245]: pgmap v530: 97 pgs: 97 active+clean; 69 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2023-07-29T16:20:48.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 259038 9.06 MB/sec execute 33 sec latency 13.063 ms 2023-07-29T16:20:49.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 260038 8.90 MB/sec execute 34 sec latency 15.445 ms 2023-07-29T16:20:50.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:50 smithi039 ceph-mon[128858]: pgmap v531: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 302 op/s 2023-07-29T16:20:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:50 smithi156 ceph-mon[129245]: pgmap v531: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 302 op/s 2023-07-29T16:20:50.865 INFO:tasks.workunit.client.0.smithi031.stdout: 1 262156 8.97 MB/sec execute 35 sec latency 57.249 ms 2023-07-29T16:20:50.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:50 smithi031 ceph-mon[95504]: pgmap v531: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 302 op/s 2023-07-29T16:20:51.864 INFO:tasks.workunit.client.0.smithi031.stdout: 1 263320 8.90 MB/sec execute 36 sec latency 16.179 ms 2023-07-29T16:20:52.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:52 smithi039 ceph-mon[128858]: pgmap v532: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 2.0 MiB/s wr, 228 op/s 2023-07-29T16:20:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:52 smithi156 ceph-mon[129245]: pgmap v532: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 2.0 MiB/s wr, 228 op/s 2023-07-29T16:20:52.865 INFO:tasks.workunit.client.0.smithi031.stdout: 1 265516 8.98 MB/sec execute 37 sec latency 17.578 ms 2023-07-29T16:20:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:52 smithi031 ceph-mon[95504]: pgmap v532: 97 pgs: 97 active+clean; 69 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 2.0 MiB/s wr, 228 op/s 2023-07-29T16:20:53.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:20:53.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:20:53.865 INFO:tasks.workunit.client.0.smithi031.stdout: 1 266652 8.92 MB/sec execute 38 sec latency 12.674 ms 2023-07-29T16:20:53.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:20:54.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:54 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:20:54.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:54 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:20:54.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:54 smithi039 ceph-mon[128858]: pgmap v533: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 308 op/s 2023-07-29T16:20:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:54 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:20:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:54 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:20:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:54 smithi156 ceph-mon[129245]: pgmap v533: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 308 op/s 2023-07-29T16:20:54.865 INFO:tasks.workunit.client.0.smithi031.stdout: 1 268636 8.89 MB/sec execute 39 sec latency 87.913 ms 2023-07-29T16:20:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:54 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:20:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:54 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:20:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:54 smithi031 ceph-mon[95504]: pgmap v533: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 308 op/s 2023-07-29T16:20:55.865 INFO:tasks.workunit.client.0.smithi031.stdout: 1 269994 8.95 MB/sec execute 40 sec latency 73.276 ms 2023-07-29T16:20:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2377505990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8101540 msgr2=0x7fb8b8101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2377505990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8101540 0x7fb8b8101990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009990 tx=0x7fb8ac009ca0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2377505990 shutdown_connections 2023-07-29T16:20:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2377505990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8b8101ed0 0x7fb8b8108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2377505990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100350 0x7fb8b8100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2377505990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8101540 0x7fb8b8101990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2377505990 >> 172.21.15.31:0/2377505990 conn(0x7fb8b80fb970 msgr2=0x7fb8b80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:56.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2377505990 shutdown_connections 2023-07-29T16:20:56.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.256+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2377505990 wait complete. 2023-07-29T16:20:56.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.257+0000 7fb8bc9ff700 1 Processor -- start 2023-07-29T16:20:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 -- start start 2023-07-29T16:20:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 0x7fb8b8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8101540 0x7fb8b8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8b8101ed0 0x7fb8b8073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81a01a0 con 0x7fb8b8101ed0 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81a02e0 con 0x7fb8b8100350 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8bc9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b81a0420 con 0x7fb8b8101540 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 0x7fb8b8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 0x7fb8b8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54488/0 (socket says 172.21.15.31:54488) 2023-07-29T16:20:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8b659c700 1 -- 172.21.15.31:0/2326499612 learned_addr learned my addr 172.21.15.31:0/2326499612 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8101540 0x7fb8b8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.258+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8b8101ed0 0x7fb8b8073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8b659c700 1 -- 172.21.15.31:0/2326499612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8101540 msgr2=0x7fb8b8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8b659c700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8101540 0x7fb8b8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8b659c700 1 -- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8b8101ed0 msgr2=0x7fb8b8073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8b659c700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8b8101ed0 0x7fb8b8073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8b659c700 1 -- 172.21.15.31:0/2326499612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0095f0 con 0x7fb8b8100350 2023-07-29T16:20:56.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8b659c700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 0x7fb8b8073140 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800e970 tx=0x7fb8a800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:56.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8a77fe700 1 -- 172.21.15.31:0/2326499612 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8009790 con 0x7fb8b8100350 2023-07-29T16:20:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8a77fe700 1 -- 172.21.15.31:0/2326499612 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8a800eec0 con 0x7fb8b8100350 2023-07-29T16:20:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b81a06b0 con 0x7fb8b8100350 2023-07-29T16:20:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8a77fe700 1 -- 172.21.15.31:0/2326499612 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8003cf0 con 0x7fb8b8100350 2023-07-29T16:20:56.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.259+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b81a0c10 con 0x7fb8b8100350 2023-07-29T16:20:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.263+0000 7fb8a77fe700 1 -- 172.21.15.31:0/2326499612 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb8a800ccd0 con 0x7fb8b8100350 2023-07-29T16:20:56.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.264+0000 7fb8d036db80 1 Processor -- start 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.264+0000 7fb8d036db80 1 -- start start 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.264+0000 7fb8a77fe700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8a00a0810 0x7fb8a00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8a77fe700 1 -- 172.21.15.31:0/2326499612 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb8a8014070 con 0x7fb8b8100350 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8a00a0810 0x7fb8a00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f3df0 0x5600550f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8d036db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56005504b9b0 con 0x5600550f4cf0 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8d036db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560055049370 con 0x5600550f3df0 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8d036db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560055044c60 con 0x560054f4f6b0 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44378/0 (socket says 172.21.15.31:44378) 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 -- 172.21.15.31:0/1177538045 learned_addr learned my addr 172.21.15.31:0/1177538045 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:56.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f3df0 0x5600550f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 -- 172.21.15.31:0/1177538045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f3df0 msgr2=0x5600550f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f3df0 0x5600550f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 -- 172.21.15.31:0/1177538045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 msgr2=0x5600550f5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.265+0000 7fb8b659c700 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.266+0000 7fb8b659c700 1 -- 172.21.15.31:0/1177538045 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560054e72d80 con 0x560054f4f6b0 2023-07-29T16:20:56.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.266+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8a00a0810 0x7fb8a00a2cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00076f0 tx=0x7fb8b0007ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.266+0000 7fb8b659c700 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550f38b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8095ee0 tx=0x7fb8a8099080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b572d62236dfaad server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.266+0000 7fb8a4ff9700 1 -- 172.21.15.31:0/1177538045 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8099470 con 0x560054f4f6b0 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.266+0000 7fb8a4ff9700 1 -- 172.21.15.31:0/1177538045 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8a809d4a0 con 0x560054f4f6b0 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.266+0000 7fb8a4ff9700 1 -- 172.21.15.31:0/1177538045 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8099980 con 0x560054f4f6b0 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 -- 172.21.15.31:0/1177538045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 msgr2=0x5600550f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550f38b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8095ee0 tx=0x7fb8a8099080 comp rx=0 tx=0).stop 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 -- 172.21.15.31:0/1177538045 shutdown_connections 2023-07-29T16:20:56.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550f38b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 --2- 172.21.15.31:0/1177538045 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f3df0 0x5600550f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 -- 172.21.15.31:0/1177538045 >> 172.21.15.31:0/1177538045 conn(0x5600550e8fd0 msgr2=0x5600550ee300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 -- 172.21.15.31:0/1177538045 shutdown_connections 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 -- 172.21.15.31:0/1177538045 wait complete. 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.267+0000 7fb8d036db80 1 -- start start 2023-07-29T16:20:56.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 Processor -- start 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x5600550fd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055104470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560054fe14b0 con 0x5600550f3df0 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560054fe1150 con 0x5600550f4cf0 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560054ec88c0 con 0x560054f4f6b0 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.268+0000 7fb8d036db80 10 client.0 _ll_register_callbacks cb 0x560054fea5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:56.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550fcef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44388/0 (socket says 172.21.15.31:44388) 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 -- 172.21.15.31:0/2289787410 learned_addr learned my addr 172.21.15.31:0/2289787410 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055104470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x5600550fd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 -- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 msgr2=0x560055104470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055104470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 -- 172.21.15.31:0/2289787410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 msgr2=0x5600550fd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x5600550fd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 -- 172.21.15.31:0/2289787410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b81a0c10 con 0x560054f4f6b0 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055104470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8b659c700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550fcef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80047f0 tx=0x7fb8a8099a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9edcf8e503224a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb89dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb89dffb700 1 -- 172.21.15.31:0/2289787410 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8099470 con 0x560054f4f6b0 2023-07-29T16:20:56.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb89dffb700 1 -- 172.21.15.31:0/2289787410 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb8a8098960 con 0x560054f4f6b0 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.269+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56005506fcb0 con 0x560054f4f6b0 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb89dffb700 1 -- 172.21.15.31:0/2289787410 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8a8098b20 con 0x560054f4f6b0 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 10 client.27223 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560054e5bc00 con 0x560054f4f6b0 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb89dffb700 1 -- 172.21.15.31:0/2289787410 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56005506fcb0 con 0x560054f4f6b0 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb89dffb700 1 -- 172.21.15.31:0/2289787410 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fb8a80b0030 con 0x560054f4f6b0 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 10 client.27223 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:56.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 10 client.27223 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 20 client.27223 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560055108920 0x56005510ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 4 client.27223 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:20:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600550c2d10 con 0x560055108920 2023-07-29T16:20:56.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.270+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560055108920 0x56005510ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.271+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560055108920 0x56005510ad30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=711d68db3f2a5e39 server_cookie=c213aa1129b9337a in_seq=0 out_seq=0 2023-07-29T16:20:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.271+0000 7fb89dffb700 10 client.27223 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:56.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb89dffb700 1 -- 172.21.15.31:0/2289787410 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5600550c2d10 con 0x560055108920 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb89dffb700 10 client.27223 handle_command_reply: tid=0 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 client.27223 shutdown 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 msgr2=0x5600550fcef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550fcef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80047f0 tx=0x7fb8a8099a90 comp rx=0 tx=0).stop 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 shutdown_connections 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560055108920 0x56005510ad30 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055104470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x5600550fcef0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb89dffb700 0 client.27223 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 --2- 172.21.15.31:0/2289787410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x5600550fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.273+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 >> 172.21.15.31:0/2289787410 conn(0x5600550e8fd0 msgr2=0x5600550eda40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.274+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 shutdown_connections 2023-07-29T16:20:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.274+0000 7fb8d036db80 1 -- 172.21.15.31:0/2289787410 wait complete. 2023-07-29T16:20:56.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.274+0000 7fb8d036db80 20 client.27223 trim_cache size 0 max 16384 2023-07-29T16:20:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.289+0000 7fb8d036db80 1 Processor -- start 2023-07-29T16:20:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.289+0000 7fb8d036db80 1 -- start start 2023-07-29T16:20:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.289+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 0x560055102420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600550f3df0 0x5600550ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8d036db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600550fcac0 con 0x5600550f4cf0 2023-07-29T16:20:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8d036db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600550fe070 con 0x560054f4f6b0 2023-07-29T16:20:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8d036db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600550ef170 con 0x5600550f3df0 2023-07-29T16:20:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 0x560055102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600550f3df0 0x5600550ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 0x560055102420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54516/0 (socket says 172.21.15.31:54516) 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 -- 172.21.15.31:0/3640752523 learned_addr learned my addr 172.21.15.31:0/3640752523 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 -- 172.21.15.31:0/3640752523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600550f3df0 msgr2=0x5600550ea4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600550f3df0 0x5600550ea4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 -- 172.21.15.31:0/3640752523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 msgr2=0x5600550f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 -- 172.21.15.31:0/3640752523 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560054e991f0 con 0x560054f4f6b0 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600550f3df0 0x5600550ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.290+0000 7fb8b659c700 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 0x560055102420 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80a1390 tx=0x7fb8a8097a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12e70f96b3c4da0b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8a57fa700 1 -- 172.21.15.31:0/3640752523 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8099470 con 0x560054f4f6b0 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8a57fa700 1 -- 172.21.15.31:0/3640752523 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8a80a18c0 con 0x560054f4f6b0 2023-07-29T16:20:56.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8a57fa700 1 -- 172.21.15.31:0/3640752523 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80b7600 con 0x560054f4f6b0 2023-07-29T16:20:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 -- 172.21.15.31:0/3640752523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 msgr2=0x560055102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 0x560055102420 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80a1390 tx=0x7fb8a8097a20 comp rx=0 tx=0).stop 2023-07-29T16:20:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 -- 172.21.15.31:0/3640752523 shutdown_connections 2023-07-29T16:20:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f4cf0 0x5600550f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560054f4f6b0 0x560055102420 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3640752523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600550f3df0 0x5600550ea4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.291+0000 7fb8d036db80 1 -- 172.21.15.31:0/3640752523 >> 172.21.15.31:0/3640752523 conn(0x5600550e8fd0 msgr2=0x5600550ee2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:56.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.292+0000 7fb8d036db80 1 -- 172.21.15.31:0/3640752523 shutdown_connections 2023-07-29T16:20:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.292+0000 7fb8d036db80 1 -- 172.21.15.31:0/3640752523 wait complete. 2023-07-29T16:20:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.292+0000 7fb8d036db80 1 -- start start 2023-07-29T16:20:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.292+0000 7fb8d036db80 1 Processor -- start 2023-07-29T16:20:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x560055100310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x560055100850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055101860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560055101da0 con 0x5600550f3df0 2023-07-29T16:20:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560055101f10 con 0x5600550f4cf0 2023-07-29T16:20:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600551020d0 con 0x560054f4f6b0 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8d036db80 10 client.0 _ll_register_callbacks cb 0x560054fea5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x560055100310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x560055100310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44402/0 (socket says 172.21.15.31:44402) 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 -- 172.21.15.31:0/3425924936 learned_addr learned my addr 172.21.15.31:0/3425924936 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b6d9d700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x560055100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 -- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 msgr2=0x560055101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055101860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 -- 172.21.15.31:0/3425924936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 msgr2=0x560055100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x560055100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.293+0000 7fb8b659c700 1 -- 172.21.15.31:0/3425924936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b81090c0 con 0x560054f4f6b0 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb8b659c700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x560055100310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb8a801b040 tx=0x7fb8a80b74c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed78527db701d67f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb89dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:20:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb89dffb700 1 -- 172.21.15.31:0/3425924936 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8099470 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb89dffb700 1 -- 172.21.15.31:0/3425924936 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb8a80a1640 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8a809d4a0 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb89dffb700 1 -- 172.21.15.31:0/3425924936 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8a8098960 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb8d036db80 10 client.27229 fetch_fsmap learned FSMap version 26 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560054e5bc00 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.294+0000 7fb89dffb700 1 -- 172.21.15.31:0/3425924936 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8a809d4a0 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb89dffb700 1 -- 172.21.15.31:0/3425924936 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 26) v1 ==== 7462+0+0 (secure 0 0 0) 0x7fb8a80b0070 con 0x560054f4f6b0 2023-07-29T16:20:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8d036db80 10 client.27229 fetch_fsmap finished waiting for FSMap version 26 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8d036db80 10 client.27229 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8d036db80 20 client.27229 populate_metadata read hostname 'smithi031' 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56005515aea0 0x56005515d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8d036db80 4 client.27229 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56005503dce0 con 0x56005515aea0 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.295+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56005515aea0 0x56005515d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:20:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.296+0000 7fb8b5d9b700 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56005515aea0 0x56005515d280 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2f4ddc3a8463be server_cookie=615132652e43ec43 in_seq=0 out_seq=0 2023-07-29T16:20:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.296+0000 7fb89dffb700 10 client.27229 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:56.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb89dffb700 1 -- 172.21.15.31:0/3425924936 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56005503dce0 con 0x56005515aea0 2023-07-29T16:20:56.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb89dffb700 10 client.27229 handle_command_reply: tid=0 2023-07-29T16:20:56.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 client.27229 shutdown 2023-07-29T16:20:56.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 msgr2=0x560055100310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x560055100310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb8a801b040 tx=0x7fb8a80b74c0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 shutdown_connections 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56005515aea0 0x56005515d280 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600550f4cf0 0x560055101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560054f4f6b0 0x560055100310 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 --2- 172.21.15.31:0/3425924936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600550f3df0 0x560055100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb89dffb700 0 client.27229 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 >> 172.21.15.31:0/3425924936 conn(0x5600550e8fd0 msgr2=0x5600550ee210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:56.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.378+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 shutdown_connections 2023-07-29T16:20:56.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.379+0000 7fb8d036db80 1 -- 172.21.15.31:0/3425924936 wait complete. 2023-07-29T16:20:56.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.379+0000 7fb8d036db80 20 client.27229 trim_cache size 0 max 16384 2023-07-29T16:20:56.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:20:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:20:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:20:56.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.379+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8a00a0810 msgr2=0x7fb8a00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.379+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8a00a0810 0x7fb8a00a2cc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00076f0 tx=0x7fb8b0007ab0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.379+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 msgr2=0x7fb8b8073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.379+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 0x7fb8b8073140 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800e970 tx=0x7fb8a800ec80 comp rx=0 tx=0).stop 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 shutdown_connections 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8a00a0810 0x7fb8a00a2cc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8b8101ed0 0x7fb8b8073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8b8100350 0x7fb8b8073140 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 --2- 172.21.15.31:0/2326499612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8101540 0x7fb8b8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:20:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 >> 172.21.15.31:0/2326499612 conn(0x7fb8b80fb970 msgr2=0x7fb8b8104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:20:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 shutdown_connections 2023-07-29T16:20:56.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:20:56.380+0000 7fb8bc9ff700 1 -- 172.21.15.31:0/2326499612 wait complete. 2023-07-29T16:20:56.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:56 smithi039 ceph-mon[128858]: pgmap v534: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 275 op/s 2023-07-29T16:20:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:56 smithi156 ceph-mon[129245]: pgmap v534: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 275 op/s 2023-07-29T16:20:56.865 INFO:tasks.workunit.client.0.smithi031.stdout: 1 271744 8.87 MB/sec execute 41 sec latency 31.960 ms 2023-07-29T16:20:56.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:56 smithi031 ceph-mon[95504]: pgmap v534: 97 pgs: 97 active+clean; 72 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 275 op/s 2023-07-29T16:20:57.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 273518 8.98 MB/sec execute 42 sec latency 13.905 ms 2023-07-29T16:20:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:57 smithi031 ceph-mon[95504]: pgmap v535: 97 pgs: 97 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2023-07-29T16:20:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:57 smithi039 ceph-mon[128858]: pgmap v535: 97 pgs: 97 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2023-07-29T16:20:58.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:57 smithi156 ceph-mon[129245]: pgmap v535: 97 pgs: 97 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 248 op/s 2023-07-29T16:20:58.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 275131 8.90 MB/sec execute 43 sec latency 15.849 ms 2023-07-29T16:20:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:58 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:20:59.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:58 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:20:59.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:58 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:20:59.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 277057 9.00 MB/sec execute 44 sec latency 17.667 ms 2023-07-29T16:21:00.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:20:59 smithi031 ceph-mon[95504]: pgmap v536: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 297 op/s 2023-07-29T16:21:00.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:20:59 smithi039 ceph-mon[128858]: pgmap v536: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 297 op/s 2023-07-29T16:21:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:20:59 smithi156 ceph-mon[129245]: pgmap v536: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 297 op/s 2023-07-29T16:21:00.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 278608 8.93 MB/sec execute 45 sec latency 35.692 ms 2023-07-29T16:21:01.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 280567 9.02 MB/sec execute 46 sec latency 13.239 ms 2023-07-29T16:21:02.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:02 smithi039 ceph-mon[128858]: pgmap v537: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 2.1 MiB/s wr, 231 op/s 2023-07-29T16:21:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:02 smithi156 ceph-mon[129245]: pgmap v537: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 2.1 MiB/s wr, 231 op/s 2023-07-29T16:21:02.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 282221 8.95 MB/sec execute 47 sec latency 11.545 ms 2023-07-29T16:21:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:02 smithi031 ceph-mon[95504]: pgmap v537: 97 pgs: 97 active+clean; 67 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 2.1 MiB/s wr, 231 op/s 2023-07-29T16:21:03.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 284124 9.05 MB/sec execute 48 sec latency 15.894 ms 2023-07-29T16:21:04.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:04 smithi039 ceph-mon[128858]: pgmap v538: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 313 op/s 2023-07-29T16:21:04.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:04 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:21:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:04 smithi156 ceph-mon[129245]: pgmap v538: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 313 op/s 2023-07-29T16:21:04.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:04 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:21:04.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 285699 8.98 MB/sec execute 49 sec latency 15.662 ms 2023-07-29T16:21:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:04 smithi031 ceph-mon[95504]: pgmap v538: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.7 MiB/s wr, 313 op/s 2023-07-29T16:21:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:04 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:21:05.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 287585 9.06 MB/sec execute 50 sec latency 57.066 ms 2023-07-29T16:21:06.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:06 smithi039 ceph-mon[128858]: pgmap v539: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 253 op/s 2023-07-29T16:21:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:06 smithi156 ceph-mon[129245]: pgmap v539: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 253 op/s 2023-07-29T16:21:06.866 INFO:tasks.workunit.client.0.smithi031.stdout: 1 288972 9.00 MB/sec execute 51 sec latency 15.948 ms 2023-07-29T16:21:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:06 smithi031 ceph-mon[95504]: pgmap v539: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 253 op/s 2023-07-29T16:21:07.867 INFO:tasks.workunit.client.0.smithi031.stdout: 1 291124 9.08 MB/sec execute 52 sec latency 22.498 ms 2023-07-29T16:21:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:07 smithi031 ceph-mon[95504]: pgmap v540: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2023-07-29T16:21:08.274 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:07 smithi039 ceph-mon[128858]: pgmap v540: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2023-07-29T16:21:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:07 smithi156 ceph-mon[129245]: pgmap v540: 97 pgs: 97 active+clean; 71 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.1 MiB/s wr, 245 op/s 2023-07-29T16:21:08.867 INFO:tasks.workunit.client.0.smithi031.stdout: 1 292496 9.02 MB/sec execute 53 sec latency 14.411 ms 2023-07-29T16:21:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 -- 172.21.15.31:0/3633414715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 msgr2=0x7f00ac1083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 --2- 172.21.15.31:0/3633414715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac1083d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ec0 tx=0x7f00a400b330 comp rx=0 tx=0).stop 2023-07-29T16:21:09.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 -- 172.21.15.31:0/3633414715 shutdown_connections 2023-07-29T16:21:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 --2- 172.21.15.31:0/3633414715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac1083d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 --2- 172.21.15.31:0/3633414715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 0x7f00ac102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 --2- 172.21.15.31:0/3633414715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 0x7f00ac105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 -- 172.21.15.31:0/3633414715 >> 172.21.15.31:0/3633414715 conn(0x7f00ac0fa840 msgr2=0x7f00ac0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 -- 172.21.15.31:0/3633414715 shutdown_connections 2023-07-29T16:21:09.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.533+0000 7f00b23ff700 1 -- 172.21.15.31:0/3633414715 wait complete. 2023-07-29T16:21:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.534+0000 7f00b23ff700 1 Processor -- start 2023-07-29T16:21:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 -- start start 2023-07-29T16:21:09.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 0x7f00ac198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 0x7f00ac1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac199d90 con 0x7f00ac068e80 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac199f00 con 0x7f00ac105f20 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b23ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac19a0c0 con 0x7f00ac103410 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 0x7f00ac1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b099c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 0x7f00ac1988b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42212/0 (socket says 172.21.15.31:42212) 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00b099c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45908/0 (socket says 172.21.15.31:45908) 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.535+0000 7f00ab7fe700 1 -- 172.21.15.31:0/1341174079 learned_addr learned my addr 172.21.15.31:0/1341174079 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00abfff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 0x7f00ac198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b099c700 1 -- 172.21.15.31:0/1341174079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 msgr2=0x7f00ac1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b099c700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 0x7f00ac1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b099c700 1 -- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 msgr2=0x7f00ac198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b099c700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 0x7f00ac198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b099c700 1 -- 172.21.15.31:0/1341174079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a4005e70 con 0x7f00ac105f20 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00abfff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 0x7f00ac198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b099c700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac199730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ec0 tx=0x7f00a40094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00a97fa700 1 -- 172.21.15.31:0/1341174079 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a400e040 con 0x7f00ac105f20 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00a97fa700 1 -- 172.21.15.31:0/1341174079 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00a401d940 con 0x7f00ac105f20 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac19a340 con 0x7f00ac105f20 2023-07-29T16:21:09.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.536+0000 7f00a97fa700 1 -- 172.21.15.31:0/1341174079 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a401b980 con 0x7f00ac105f20 2023-07-29T16:21:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.537+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac1a0b60 con 0x7f00ac105f20 2023-07-29T16:21:09.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.540+0000 7f00a97fa700 1 -- 172.21.15.31:0/1341174079 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f00a401dab0 con 0x7f00ac105f20 2023-07-29T16:21:09.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.542+0000 7f00c5d17b80 1 Processor -- start 2023-07-29T16:21:09.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.542+0000 7f00a97fa700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00940a0810 0x7f00940a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.542+0000 7f00c5d17b80 1 -- start start 2023-07-29T16:21:09.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00a97fa700 1 -- 172.21.15.31:0/1341174079 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f00a40cd810 con 0x7f00ac105f20 2023-07-29T16:21:09.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00ab7fe700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00940a0810 0x7f00940a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d1166698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d11666a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d11666b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1165c19b0 con 0x55d11666acf0 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1165bf370 con 0x55d116669df0 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1165bac60 con 0x55d1164c56b0 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00abfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d1166698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00abfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d1166698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42220/0 (socket says 172.21.15.31:42220) 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00abfff700 1 -- 172.21.15.31:0/1808307803 learned_addr learned my addr 172.21.15.31:0/1808307803 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:09.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00ab7fe700 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d11666a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.543+0000 7f00b099c700 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d11666b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00abfff700 1 -- 172.21.15.31:0/1808307803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 msgr2=0x55d11666a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00abfff700 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d11666a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00abfff700 1 -- 172.21.15.31:0/1808307803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 msgr2=0x55d11666b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00abfff700 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d11666b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00abfff700 1 -- 172.21.15.31:0/1808307803 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1163e8d80 con 0x55d1164c56b0 2023-07-29T16:21:09.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00ab7fe700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00940a0810 0x7f00940a2cc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009d70 tx=0x7f00a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00b099c700 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d11666b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00abfff700 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d1166698b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f009c009c90 tx=0x7f009c010370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcab219e1bafa8a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00927fc700 1 -- 172.21.15.31:0/1808307803 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c010b90 con 0x55d1164c56b0 2023-07-29T16:21:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00927fc700 1 -- 172.21.15.31:0/1808307803 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f009c0044f0 con 0x55d1164c56b0 2023-07-29T16:21:09.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.544+0000 7f00927fc700 1 -- 172.21.15.31:0/1808307803 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c003e50 con 0x55d1164c56b0 2023-07-29T16:21:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1808307803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 msgr2=0x55d1166698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d1166698b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f009c009c90 tx=0x7f009c010370 comp rx=0 tx=0).stop 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1808307803 shutdown_connections 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d11666b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d1166698b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1808307803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d11666a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1808307803 >> 172.21.15.31:0/1808307803 conn(0x55d11665efd0 msgr2=0x55d116664300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1808307803 shutdown_connections 2023-07-29T16:21:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.545+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1808307803 wait complete. 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.546+0000 7f00c5d17b80 1 -- start start 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.546+0000 7f00c5d17b80 1 Processor -- start 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.546+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116672f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.546+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116673460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.546+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d11667a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1165574b0 con 0x55d1164c56b0 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d116557150 con 0x55d116669df0 2023-07-29T16:21:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11643e8c0 con 0x55d11666acf0 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00c5d17b80 10 client.0 _ll_register_callbacks cb 0x55d1165605c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116673460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00abfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116672f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116673460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45920/0 (socket says 172.21.15.31:45920) 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00ab7fe700 1 -- 172.21.15.31:0/3296673376 learned_addr learned my addr 172.21.15.31:0/3296673376 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d11667a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 -- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 msgr2=0x55d116673460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116673460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 -- 172.21.15.31:0/3296673376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 msgr2=0x55d116672f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116672f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 -- 172.21.15.31:0/3296673376 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c0098c0 con 0x55d11666acf0 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00ab7fe700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116673460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f00b099c700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d11667a4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f00a4097440 tx=0x7f00a409b080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfac3908c8fb8d86 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.547+0000 7f008b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f008b7fe700 1 -- 172.21.15.31:0/3296673376 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a40992e0 con 0x55d11666acf0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1165e5cb0 con 0x55d11666acf0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f008b7fe700 1 -- 172.21.15.31:0/3296673376 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f00a409a430 con 0x55d11666acf0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f008b7fe700 1 -- 172.21.15.31:0/3296673376 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00a40b9280 con 0x55d11666acf0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f00c5d17b80 10 client.27241 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d1163d1c00 con 0x55d11666acf0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f008b7fe700 1 -- 172.21.15.31:0/3296673376 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1165e5cb0 con 0x55d11666acf0 2023-07-29T16:21:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f008b7fe700 1 -- 172.21.15.31:0/3296673376 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f00a40aebb0 con 0x55d11666acf0 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f00c5d17b80 10 client.27241 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f00c5d17b80 10 client.27241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.548+0000 7f00c5d17b80 20 client.27241 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.549+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d11667ea20 0x55d116680e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.549+0000 7f00c5d17b80 4 client.27241 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.549+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d116638d10 con 0x55d11667ea20 2023-07-29T16:21:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.549+0000 7f00abfff700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d11667ea20 0x55d116680e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.549+0000 7f008b7fe700 10 client.27241 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.549+0000 7f00abfff700 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d11667ea20 0x55d116680e00 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d182e7598c3d13c4 server_cookie=ed364f025c24e3ae in_seq=0 out_seq=0 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.550+0000 7f008b7fe700 1 -- 172.21.15.31:0/3296673376 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d116638d10 con 0x55d11667ea20 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.550+0000 7f008b7fe700 10 client.27241 handle_command_reply: tid=0 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.550+0000 7f00c5d17b80 1 client.27241 shutdown 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.550+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 msgr2=0x55d11667a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.550+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d11667a4a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f00a4097440 tx=0x7f00a409b080 comp rx=0 tx=0).stop 2023-07-29T16:21:09.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 shutdown_connections 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d11667ea20 0x55d116680e00 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d11667a4a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116672f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f008b7fe700 0 client.27241 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/3296673376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116673460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 >> 172.21.15.31:0/3296673376 conn(0x55d11665efd0 msgr2=0x55d116663a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:09.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 shutdown_connections 2023-07-29T16:21:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 1 -- 172.21.15.31:0/3296673376 wait complete. 2023-07-29T16:21:09.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.551+0000 7f00c5d17b80 20 client.27241 trim_cache size 0 max 16384 2023-07-29T16:21:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 Processor -- start 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 -- start start 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116678450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116663a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d1166710f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1166723d0 con 0x55d1164c56b0 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d116672a90 con 0x55d116669df0 2023-07-29T16:21:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.566+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d116665e20 con 0x55d11666acf0 2023-07-29T16:21:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116663a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d1166710f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00abfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116678450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116663a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45922/0 (socket says 172.21.15.31:45922) 2023-07-29T16:21:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00ab7fe700 1 -- 172.21.15.31:0/2384110771 learned_addr learned my addr 172.21.15.31:0/2384110771 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 -- 172.21.15.31:0/2384110771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 msgr2=0x55d116663a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116663a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 -- 172.21.15.31:0/2384110771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 msgr2=0x55d116678450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116678450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 -- 172.21.15.31:0/2384110771 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d11640f1f0 con 0x55d11666acf0 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f00b099c700 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d1166710f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f00a40a2dd0 tx=0x7f00a4097370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=767025c99fc2ca7a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f0092ffd700 1 -- 172.21.15.31:0/2384110771 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a40b9280 con 0x55d11666acf0 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f0092ffd700 1 -- 172.21.15.31:0/2384110771 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00a40992e0 con 0x55d11666acf0 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.567+0000 7f0092ffd700 1 -- 172.21.15.31:0/2384110771 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a409a690 con 0x55d11666acf0 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 -- 172.21.15.31:0/2384110771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 msgr2=0x55d1166710f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d1166710f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f00a40a2dd0 tx=0x7f00a4097370 comp rx=0 tx=0).stop 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 -- 172.21.15.31:0/2384110771 shutdown_connections 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d11666acf0 0x55d1166710f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d1164c56b0 0x55d116678450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/2384110771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d116663a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 -- 172.21.15.31:0/2384110771 >> 172.21.15.31:0/2384110771 conn(0x55d11665efd0 msgr2=0x55d116671c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 -- 172.21.15.31:0/2384110771 shutdown_connections 2023-07-29T16:21:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.568+0000 7f00c5d17b80 1 -- 172.21.15.31:0/2384110771 wait complete. 2023-07-29T16:21:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 -- start start 2023-07-29T16:21:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 Processor -- start 2023-07-29T16:21:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d116677c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d1166781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d116676440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d116676a50 con 0x55d11666acf0 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d116676bc0 con 0x55d116669df0 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d116676d80 con 0x55d1164c56b0 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00c5d17b80 10 client.0 _ll_register_callbacks cb 0x55d1165605c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00abfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d116677c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00ab7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d1166781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.569+0000 7f00abfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d116677c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42250/0 (socket says 172.21.15.31:42250) 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 -- 172.21.15.31:0/1533412550 learned_addr learned my addr 172.21.15.31:0/1533412550 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00b099c700 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d116676440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 -- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 msgr2=0x55d1166781c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d1166781c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 -- 172.21.15.31:0/1533412550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 msgr2=0x55d116676440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d116676440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 -- 172.21.15.31:0/1533412550 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1163d1c00 con 0x55d1164c56b0 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00abfff700 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d116677c00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f009c0092a0 tx=0x7f009c0057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4486bbc0c130e0c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f008b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f008b7fe700 1 -- 172.21.15.31:0/1533412550 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c021070 con 0x55d1164c56b0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f009c0044f0 con 0x55d1164c56b0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.570+0000 7f008b7fe700 1 -- 172.21.15.31:0/1533412550 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f009c016070 con 0x55d1164c56b0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f008b7fe700 1 -- 172.21.15.31:0/1533412550 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f009c003900 con 0x55d1164c56b0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 10 client.27247 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d1163e7250 con 0x55d1164c56b0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f008b7fe700 1 -- 172.21.15.31:0/1533412550 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f009c0044f0 con 0x55d1164c56b0 2023-07-29T16:21:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f008b7fe700 1 -- 172.21.15.31:0/1533412550 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f009c003b80 con 0x55d1164c56b0 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 10 client.27247 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 10 client.27247 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 20 client.27247 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d1166d1690 0x55d1166d3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 4 client.27247 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.571+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d1165b3ce0 con 0x55d1166d1690 2023-07-29T16:21:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.572+0000 7f00ab7fe700 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d1166d1690 0x55d1166d3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.572+0000 7f008b7fe700 10 client.27247 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.572+0000 7f00ab7fe700 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d1166d1690 0x55d1166d3aa0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f5f4ede76e95bb5 server_cookie=42177d051fc4fcd6 in_seq=0 out_seq=0 2023-07-29T16:21:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f008b7fe700 1 -- 172.21.15.31:0/1533412550 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d1165b3ce0 con 0x55d1166d1690 2023-07-29T16:21:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f008b7fe700 10 client.27247 handle_command_reply: tid=0 2023-07-29T16:21:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 client.27247 shutdown 2023-07-29T16:21:09.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 msgr2=0x55d116677c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d116677c00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f009c0092a0 tx=0x7f009c0057b0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 shutdown_connections 2023-07-29T16:21:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d1166d1690 0x55d1166d3aa0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d11666acf0 0x55d116676440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d1164c56b0 0x55d116677c00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f008b7fe700 0 client.27247 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 --2- 172.21.15.31:0/1533412550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d116669df0 0x55d1166781c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 >> 172.21.15.31:0/1533412550 conn(0x55d11665efd0 msgr2=0x55d116662fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 shutdown_connections 2023-07-29T16:21:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.659+0000 7f00c5d17b80 1 -- 172.21.15.31:0/1533412550 wait complete. 2023-07-29T16:21:09.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.660+0000 7f00c5d17b80 20 client.27247 trim_cache size 0 max 16384 2023-07-29T16:21:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:21:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:21:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:21:09.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:21:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.660+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00940a0810 msgr2=0x7f00940a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.660+0000 7f00b23ff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00940a0810 0x7f00940a2cc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009d70 tx=0x7f00a0009260 comp rx=0 tx=0).stop 2023-07-29T16:21:09.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.660+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 msgr2=0x7f00ac199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.660+0000 7f00b23ff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac199730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ec0 tx=0x7f00a40094f0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 shutdown_connections 2023-07-29T16:21:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f00940a0810 0x7f00940a2cc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00ac105f20 0x7f00ac199730 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f00ac068e80 0x7f00ac198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 --2- 172.21.15.31:0/1341174079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00ac103410 0x7f00ac1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 >> 172.21.15.31:0/1341174079 conn(0x7f00ac0fa840 msgr2=0x7f00ac0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 shutdown_connections 2023-07-29T16:21:09.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:09.661+0000 7f00b23ff700 1 -- 172.21.15.31:0/1341174079 wait complete. 2023-07-29T16:21:09.867 INFO:tasks.workunit.client.0.smithi031.stdout: 1 294578 9.08 MB/sec execute 54 sec latency 20.945 ms 2023-07-29T16:21:10.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:10 smithi039 ceph-mon[128858]: pgmap v541: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2023-07-29T16:21:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:10 smithi156 ceph-mon[129245]: pgmap v541: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2023-07-29T16:21:10.867 INFO:tasks.workunit.client.0.smithi031.stdout: 1 295662 8.98 MB/sec execute 55 sec latency 46.880 ms 2023-07-29T16:21:10.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:10 smithi031 ceph-mon[95504]: pgmap v541: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 301 op/s 2023-07-29T16:21:11.867 INFO:tasks.workunit.client.0.smithi031.stdout: 1 298022 9.10 MB/sec execute 56 sec latency 42.165 ms 2023-07-29T16:21:12.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:12 smithi039 ceph-mon[128858]: pgmap v542: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 2.1 MiB/s wr, 209 op/s 2023-07-29T16:21:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:12 smithi156 ceph-mon[129245]: pgmap v542: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 2.1 MiB/s wr, 209 op/s 2023-07-29T16:21:12.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 298697 8.97 MB/sec execute 57 sec latency 109.480 ms 2023-07-29T16:21:12.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:12 smithi031 ceph-mon[95504]: pgmap v542: 97 pgs: 97 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 2.1 MiB/s wr, 209 op/s 2023-07-29T16:21:13.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 300984 9.03 MB/sec execute 58 sec latency 15.010 ms 2023-07-29T16:21:14.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:14 smithi039 ceph-mon[128858]: pgmap v543: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 301 op/s 2023-07-29T16:21:14.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:14 smithi156 ceph-mon[129245]: pgmap v543: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 301 op/s 2023-07-29T16:21:14.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 302267 8.99 MB/sec execute 59 sec latency 12.815 ms 2023-07-29T16:21:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:14 smithi031 ceph-mon[95504]: pgmap v543: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 301 op/s 2023-07-29T16:21:15.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:21:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 -- 172.21.15.31:0/941053641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c0100210 msgr2=0x7f95c0100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/941053641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c0100210 0x7f95c0100620 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00b0a0 tx=0x7f95bc00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 -- 172.21.15.31:0/941053641 shutdown_connections 2023-07-29T16:21:15.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/941053641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0101e50 0x7f95c01086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/941053641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c0100210 0x7f95c0100620 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/941053641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c01014a0 0x7f95c0101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 -- 172.21.15.31:0/941053641 >> 172.21.15.31:0/941053641 conn(0x7f95c00fb850 msgr2=0x7f95c00fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 -- 172.21.15.31:0/941053641 shutdown_connections 2023-07-29T16:21:15.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.639+0000 7f95c6b9d700 1 -- 172.21.15.31:0/941053641 wait complete. 2023-07-29T16:21:15.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.641+0000 7f95c6b9d700 1 Processor -- start 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.641+0000 7f95c6b9d700 1 -- start start 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c6b9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 0x7f95c0195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c6b9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c01014a0 0x7f95c01964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c6b9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c0101e50 0x7f95c019d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c6b9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c019db00 con 0x7f95c01014a0 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c6b9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c019dc40 con 0x7f95c0100210 2023-07-29T16:21:15.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c6b9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c019ddd0 con 0x7f95c0101e50 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 0x7f95c0195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 0x7f95c0195f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35756/0 (socket says 172.21.15.31:35756) 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 -- 172.21.15.31:0/2360951022 learned_addr learned my addr 172.21.15.31:0/2360951022 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c01014a0 0x7f95c01964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c639c700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c0101e50 0x7f95c019d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 -- 172.21.15.31:0/2360951022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c0101e50 msgr2=0x7f95c019d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c0101e50 0x7f95c019d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 -- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c01014a0 msgr2=0x7f95c01964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c01014a0 0x7f95c01964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.642+0000 7f95c5b9b700 1 -- 172.21.15.31:0/2360951022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b8009520 con 0x7f95c0100210 2023-07-29T16:21:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.643+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 0x7f95c0195f80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f95bc007a60 tx=0x7f95bc00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.643+0000 7f95b6ffd700 1 -- 172.21.15.31:0/2360951022 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00e040 con 0x7f95c0100210 2023-07-29T16:21:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.643+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc009d00 con 0x7f95c0100210 2023-07-29T16:21:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.643+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c019e390 con 0x7f95c0100210 2023-07-29T16:21:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.643+0000 7f95b6ffd700 1 -- 172.21.15.31:0/2360951022 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95bc004e60 con 0x7f95c0100210 2023-07-29T16:21:15.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.643+0000 7f95b6ffd700 1 -- 172.21.15.31:0/2360951022 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0052a0 con 0x7f95c0100210 2023-07-29T16:21:15.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.646+0000 7f95b6ffd700 1 -- 172.21.15.31:0/2360951022 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f95bc005400 con 0x7f95c0100210 2023-07-29T16:21:15.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.648+0000 7f95b6ffd700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95ac0a08b0 0x7f95ac0a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.648+0000 7f95db771b80 1 Processor -- start 2023-07-29T16:21:15.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.648+0000 7f95b6ffd700 1 -- 172.21.15.31:0/2360951022 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f95bc0ccfd0 con 0x7f95c0100210 2023-07-29T16:21:15.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.648+0000 7f95c539a700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95ac0a08b0 0x7f95ac0a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.648+0000 7f95db771b80 1 -- start start 2023-07-29T16:21:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95db771b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a079150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95db771b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a079f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95db771b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a07a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95db771b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59fd0a10 con 0x55db5a079690 2023-07-29T16:21:15.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95db771b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59fce3d0 con 0x55db5a07a4d0 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95db771b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59fc9cc0 con 0x55db5a078d40 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a079150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a079150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37658/0 (socket says 172.21.15.31:37658) 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 -- 172.21.15.31:0/1357249128 learned_addr learned my addr 172.21.15.31:0/1357249128 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c639c700 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a07a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c539a700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95ac0a08b0 0x7f95ac0a2d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009c60 tx=0x7f95b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c539a700 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a079f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 -- 172.21.15.31:0/1357249128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 msgr2=0x55db5a07a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a07a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 -- 172.21.15.31:0/1357249128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 msgr2=0x55db5a079f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a079f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.649+0000 7f95c5b9b700 1 -- 172.21.15.31:0/1357249128 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db59df7de0 con 0x55db5a078d40 2023-07-29T16:21:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.650+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a079150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0a2590 tx=0x7f95bc00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84cd6109fc087ea8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.650+0000 7f95abfff700 1 -- 172.21.15.31:0/1357249128 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc09a700 con 0x55db5a078d40 2023-07-29T16:21:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.650+0000 7f95abfff700 1 -- 172.21.15.31:0/1357249128 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95bc09e730 con 0x55db5a078d40 2023-07-29T16:21:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.650+0000 7f95abfff700 1 -- 172.21.15.31:0/1357249128 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc099e50 con 0x55db5a078d40 2023-07-29T16:21:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.650+0000 7f95db771b80 1 -- 172.21.15.31:0/1357249128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 msgr2=0x55db5a079150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a079150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0a2590 tx=0x7f95bc00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 -- 172.21.15.31:0/1357249128 shutdown_connections 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a07a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a079150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 --2- 172.21.15.31:0/1357249128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a079f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 -- 172.21.15.31:0/1357249128 >> 172.21.15.31:0/1357249128 conn(0x55db5a06e540 msgr2=0x55db5a073ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 -- 172.21.15.31:0/1357249128 shutdown_connections 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 -- 172.21.15.31:0/1357249128 wait complete. 2023-07-29T16:21:15.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.651+0000 7f95db771b80 1 -- start start 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 Processor -- start 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a086c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a079690 0x55db5a087160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a07a4d0 0x55db5a08e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59f66510 con 0x55db5a07a4d0 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59f661b0 con 0x55db5a078d40 2023-07-29T16:21:15.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59e4d920 con 0x55db5a079690 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.652+0000 7f95db771b80 10 client.0 _ll_register_callbacks cb 0x55db59f6f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a086c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c539a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a079690 0x55db5a087160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a086c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35772/0 (socket says 172.21.15.31:35772) 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3996295263 learned_addr learned my addr 172.21.15.31:0/3996295263 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c639c700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a07a4d0 0x55db5a08e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3996295263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a079690 msgr2=0x55db5a087160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a079690 0x55db5a087160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3996295263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a07a4d0 msgr2=0x55db5a08e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a07a4d0 0x55db5a08e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3996295263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c019e390 con 0x55db5a078d40 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c539a700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a079690 0x55db5a087160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a086c20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009fd0 tx=0x7f95bc09a9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c27e5b5ee14b1cce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95a97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:15.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.653+0000 7f95a97fa700 1 -- 172.21.15.31:0/3996295263 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0a2ac0 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db59ff4d10 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95a97fa700 1 -- 172.21.15.31:0/3996295263 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f95bc09a700 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95a97fa700 1 -- 172.21.15.31:0/3996295263 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95bc0a2630 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 10 client.27248 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55db59de0c60 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95a97fa700 1 -- 172.21.15.31:0/3996295263 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55db59ff4d10 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95a97fa700 1 -- 172.21.15.31:0/3996295263 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f95bc0c4be0 con 0x55db5a078d40 2023-07-29T16:21:15.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 10 client.27248 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 20 client.27248 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0924b0 0x55db5a0948c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 4 client.27248 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.654+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db5a047d70 con 0x55db5a0924b0 2023-07-29T16:21:15.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.655+0000 7f95c539a700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0924b0 0x55db5a0948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.655+0000 7f95a97fa700 10 client.27248 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.655+0000 7f95c539a700 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0924b0 0x55db5a0948c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=649dae11c5ab024 server_cookie=3fdb238718876078 in_seq=0 out_seq=0 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95a97fa700 1 -- 172.21.15.31:0/3996295263 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55db5a047d70 con 0x55db5a0924b0 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95a97fa700 10 client.27248 handle_command_reply: tid=0 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 client.27248 shutdown 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 msgr2=0x55db5a086c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a086c20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009fd0 tx=0x7f95bc09a9e0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 shutdown_connections 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0924b0 0x55db5a0948c0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a07a4d0 0x55db5a08e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a086c20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95a97fa700 0 client.27248 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 --2- 172.21.15.31:0/3996295263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a079690 0x55db5a087160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 >> 172.21.15.31:0/3996295263 conn(0x55db5a06e540 msgr2=0x55db5a0731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 shutdown_connections 2023-07-29T16:21:15.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.656+0000 7f95db771b80 1 -- 172.21.15.31:0/3996295263 wait complete. 2023-07-29T16:21:15.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.657+0000 7f95db771b80 20 client.27248 trim_cache size 0 max 16384 2023-07-29T16:21:15.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 Processor -- start 2023-07-29T16:21:15.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 -- start start 2023-07-29T16:21:15.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a0731c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a081b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5a079d20 con 0x55db5a079690 2023-07-29T16:21:15.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5a074e10 con 0x55db5a07a4d0 2023-07-29T16:21:15.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95db771b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5a0862d0 con 0x55db5a078d40 2023-07-29T16:21:15.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95c539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a0731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95c639c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a081b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95c539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55954/0 (socket says 172.21.15.31:55954) 2023-07-29T16:21:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.677+0000 7f95c539a700 1 -- 172.21.15.31:0/4058122055 learned_addr learned my addr 172.21.15.31:0/4058122055 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c639c700 1 -- 172.21.15.31:0/4058122055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 msgr2=0x55db5a0731c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c639c700 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a0731c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c639c700 1 -- 172.21.15.31:0/4058122055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 msgr2=0x55db5a08e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c639c700 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c639c700 1 -- 172.21.15.31:0/4058122055 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db59e1e250 con 0x55db5a07a4d0 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c539a700 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95c639c700 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a081b30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009fd0 tx=0x7f95b8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5faa6a2ddbd6291 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95b4bff700 1 -- 172.21.15.31:0/4058122055 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8020070 con 0x55db5a07a4d0 2023-07-29T16:21:15.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95b4bff700 1 -- 172.21.15.31:0/4058122055 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95b8018040 con 0x55db5a07a4d0 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.678+0000 7f95b4bff700 1 -- 172.21.15.31:0/4058122055 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8017050 con 0x55db5a07a4d0 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 -- 172.21.15.31:0/4058122055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 msgr2=0x55db5a081b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a081b30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009fd0 tx=0x7f95b8009260 comp rx=0 tx=0).stop 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 -- 172.21.15.31:0/4058122055 shutdown_connections 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a07a4d0 0x55db5a081b30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a078d40 0x55db5a0731c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 --2- 172.21.15.31:0/4058122055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 -- 172.21.15.31:0/4058122055 >> 172.21.15.31:0/4058122055 conn(0x55db5a06e540 msgr2=0x55db5a073ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 -- 172.21.15.31:0/4058122055 shutdown_connections 2023-07-29T16:21:15.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 -- 172.21.15.31:0/4058122055 wait complete. 2023-07-29T16:21:15.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.679+0000 7f95db771b80 1 -- start start 2023-07-29T16:21:15.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.680+0000 7f95db771b80 1 Processor -- start 2023-07-29T16:21:15.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a08df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a07a4d0 0x55db5a08b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5a08b6f0 con 0x55db5a079690 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5a0e4030 con 0x55db5a078d40 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db5a0e4170 con 0x55db5a07a4d0 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95db771b80 10 client.0 _ll_register_callbacks cb 0x55db59f6f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:15.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a08df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a08df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35794/0 (socket says 172.21.15.31:35794) 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55958/0 (socket says 172.21.15.31:55958) 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3882181189 learned_addr learned my addr 172.21.15.31:0/3882181189 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c639c700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a07a4d0 0x55db5a08b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3882181189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a07a4d0 msgr2=0x55db5a08b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a07a4d0 0x55db5a08b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3882181189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 msgr2=0x55db5a08aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c5b9b700 1 -- 172.21.15.31:0/3882181189 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db59de0c60 con 0x55db5a078d40 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c639c700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a07a4d0 0x55db5a08b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.681+0000 7f95c539a700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95c5b9b700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a08df90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f95bc097170 tx=0x7f95bc096430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6a189fcd471620 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95a97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:15.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95a97fa700 1 -- 172.21.15.31:0/3882181189 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95bc0a2830 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95a97fa700 1 -- 172.21.15.31:0/3882181189 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f95bc09a700 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95bc09e730 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95a97fa700 1 -- 172.21.15.31:0/3882181189 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95bc0a2ac0 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 10 client.27254 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f95b0017070 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95a97fa700 1 -- 172.21.15.31:0/3882181189 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95bc09e730 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95a97fa700 1 -- 172.21.15.31:0/3882181189 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f95bc09e130 con 0x55db5a078d40 2023-07-29T16:21:15.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 10 client.27254 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 10 client.27254 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 20 client.27254 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0e5670 0x55db5a0e7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 4 client.27254 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.682+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55db59fc2d40 con 0x55db5a0e5670 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.683+0000 7f95c539a700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0e5670 0x55db5a0e7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.683+0000 7f95c539a700 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0e5670 0x55db5a0e7a80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=911a3bda5e387d0b server_cookie=237450e1c5036d59 in_seq=0 out_seq=0 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.683+0000 7f95a97fa700 10 client.27254 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.684+0000 7f95a97fa700 1 -- 172.21.15.31:0/3882181189 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55db59fc2d40 con 0x55db5a0e5670 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.684+0000 7f95a97fa700 10 client.27254 handle_command_reply: tid=0 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.684+0000 7f95db771b80 1 client.27254 shutdown 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.684+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 msgr2=0x55db5a08df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.684+0000 7f95db771b80 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a08df90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f95bc097170 tx=0x7f95bc096430 comp rx=0 tx=0).stop 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 shutdown_connections 2023-07-29T16:21:15.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55db5a0e5670 0x55db5a0e7a80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55db5a07a4d0 0x55db5a08b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db5a078d40 0x55db5a08df90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95a97fa700 0 client.27254 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 --2- 172.21.15.31:0/3882181189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55db5a079690 0x55db5a08aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 >> 172.21.15.31:0/3882181189 conn(0x55db5a06e540 msgr2=0x55db5a081b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 shutdown_connections 2023-07-29T16:21:15.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 1 -- 172.21.15.31:0/3882181189 wait complete. 2023-07-29T16:21:15.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95db771b80 20 client.27254 trim_cache size 0 max 16384 2023-07-29T16:21:15.692 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:21:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:21:15.692 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:21:15.692 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:21:15.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95ac0a08b0 msgr2=0x7f95ac0a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95ac0a08b0 0x7f95ac0a2d60 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009c60 tx=0x7f95b0009260 comp rx=0 tx=0).stop 2023-07-29T16:21:15.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 msgr2=0x7f95c0195f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 0x7f95c0195f80 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f95bc007a60 tx=0x7f95bc00bab0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 shutdown_connections 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95ac0a08b0 0x7f95ac0a2d60 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95c0101e50 0x7f95c019d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f95c0100210 0x7f95c0195f80 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.685+0000 7f95c6b9d700 1 --2- 172.21.15.31:0/2360951022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c01014a0 0x7f95c01964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.686+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 >> 172.21.15.31:0/2360951022 conn(0x7f95c00fb850 msgr2=0x7f95c00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.686+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 shutdown_connections 2023-07-29T16:21:15.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.686+0000 7f95c6b9d700 1 -- 172.21.15.31:0/2360951022 wait complete. 2023-07-29T16:21:15.704 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:21:15.704 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:21:15.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 304288 9.02 MB/sec execute 60 sec latency 22.459 ms 2023-07-29T16:21:15.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.924+0000 7f1ba6bff700 1 -- 172.21.15.31:0/4253905923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 msgr2=0x7f1ba0100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.924+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/4253905923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0100750 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009990 tx=0x7f1b94009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 -- 172.21.15.31:0/4253905923 shutdown_connections 2023-07-29T16:21:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/4253905923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0101ec0 0x7f1ba0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/4253905923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0100750 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/4253905923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba0101530 0x7f1ba0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 -- 172.21.15.31:0/4253905923 >> 172.21.15.31:0/4253905923 conn(0x7f1ba00fb920 msgr2=0x7f1ba00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 -- 172.21.15.31:0/4253905923 shutdown_connections 2023-07-29T16:21:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.925+0000 7f1ba6bff700 1 -- 172.21.15.31:0/4253905923 wait complete. 2023-07-29T16:21:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.926+0000 7f1ba6bff700 1 Processor -- start 2023-07-29T16:21:15.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 -- start start 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba0101530 0x7f1ba01965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0101ec0 0x7f1ba019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba019dbf0 con 0x7f1ba0101530 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba019dd30 con 0x7f1ba0101ec0 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba6bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba019dec0 con 0x7f1ba0100340 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1b9ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba0101530 0x7f1ba01965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0196070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37706/0 (socket says 172.21.15.31:37706) 2023-07-29T16:21:15.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 -- 172.21.15.31:0/529436229 learned_addr learned my addr 172.21.15.31:0/529436229 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:15.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba519c700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0101ec0 0x7f1ba019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 -- 172.21.15.31:0/529436229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0101ec0 msgr2=0x7f1ba019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0101ec0 0x7f1ba019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 -- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba0101530 msgr2=0x7f1ba01965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.927+0000 7f1ba499b700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba0101530 0x7f1ba01965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1ba499b700 1 -- 172.21.15.31:0/529436229 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98009c40 con 0x7f1ba0100340 2023-07-29T16:21:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1ba499b700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0196070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1b940095c0 tx=0x7f1b94005720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9401d070 con 0x7f1ba0100340 2023-07-29T16:21:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b94022470 con 0x7f1ba0100340 2023-07-29T16:21:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94017650 con 0x7f1ba0100340 2023-07-29T16:21:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b940095f0 con 0x7f1ba0100340 2023-07-29T16:21:15.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.928+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba019e3e0 con 0x7f1ba0100340 2023-07-29T16:21:15.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.929+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba01078f0 con 0x7f1ba0100340 2023-07-29T16:21:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.931+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1b940225e0 con 0x7f1ba0100340 2023-07-29T16:21:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.932+0000 7f1b9dffb700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b900a07e0 0x7f1b900a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:15.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.933+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1b940cd0e0 con 0x7f1ba0100340 2023-07-29T16:21:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.933+0000 7f1b9ffff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b900a07e0 0x7f1b900a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.933+0000 7f1b9ffff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b900a07e0 0x7f1b900a2c90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009c60 tx=0x7f1b8c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:15.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:15.934+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1b94096b50 con 0x7f1ba0100340 2023-07-29T16:21:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.204+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba002cc30 con 0x7f1ba0100340 2023-07-29T16:21:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.205+0000 7f1b9dffb700 1 -- 172.21.15.31:0/529436229 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f1b94027090 con 0x7f1ba0100340 2023-07-29T16:21:16.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:21:16.205 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:21:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b900a07e0 msgr2=0x7f1b900a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b900a07e0 0x7f1b900a2c90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1b8c009c60 tx=0x7f1b8c009260 comp rx=0 tx=0).stop 2023-07-29T16:21:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 msgr2=0x7f1ba0196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0196070 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1b940095c0 tx=0x7f1b94005720 comp rx=0 tx=0).stop 2023-07-29T16:21:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 shutdown_connections 2023-07-29T16:21:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1b900a07e0 0x7f1b900a2c90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0101ec0 0x7f1ba019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ba0100340 0x7f1ba0196070 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 --2- 172.21.15.31:0/529436229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ba0101530 0x7f1ba01965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 >> 172.21.15.31:0/529436229 conn(0x7f1ba00fb920 msgr2=0x7f1ba0104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 shutdown_connections 2023-07-29T16:21:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.208+0000 7f1ba6bff700 1 -- 172.21.15.31:0/529436229 wait complete. 2023-07-29T16:21:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:21:16.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:21:16.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 -- 172.21.15.31:0/316452747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 msgr2=0x7fa61c102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 --2- 172.21.15.31:0/316452747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c102960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa618009990 tx=0x7fa618009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 -- 172.21.15.31:0/316452747 shutdown_connections 2023-07-29T16:21:16.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 --2- 172.21.15.31:0/316452747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa61c1040d0 0x7fa61c10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 --2- 172.21.15.31:0/316452747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c102960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 --2- 172.21.15.31:0/316452747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c103740 0x7fa61c103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 -- 172.21.15.31:0/316452747 >> 172.21.15.31:0/316452747 conn(0x7fa61c0fdb90 msgr2=0x7fa61c0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 -- 172.21.15.31:0/316452747 shutdown_connections 2023-07-29T16:21:16.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.447+0000 7fa623dff700 1 -- 172.21.15.31:0/316452747 wait complete. 2023-07-29T16:21:16.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.448+0000 7fa623dff700 1 Processor -- start 2023-07-29T16:21:16.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 -- start start 2023-07-29T16:21:16.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa61c103740 0x7fa61c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c1040d0 0x7fa61c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c19fe30 con 0x7fa61c1040d0 2023-07-29T16:21:16.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c19ff70 con 0x7fa61c102550 2023-07-29T16:21:16.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa623dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c1a0100 con 0x7fa61c103740 2023-07-29T16:21:16.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35836/0 (socket says 172.21.15.31:35836) 2023-07-29T16:21:16.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa621b9b700 1 -- 172.21.15.31:0/2744406416 learned_addr learned my addr 172.21.15.31:0/2744406416 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa62139a700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa61c103740 0x7fa61c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa62239c700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c1040d0 0x7fa61c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa621b9b700 1 -- 172.21.15.31:0/2744406416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa61c103740 msgr2=0x7fa61c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa621b9b700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa61c103740 0x7fa61c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.449+0000 7fa621b9b700 1 -- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c1040d0 msgr2=0x7fa61c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa621b9b700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c1040d0 0x7fa61c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa621b9b700 1 -- 172.21.15.31:0/2744406416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180095f0 con 0x7fa61c102550 2023-07-29T16:21:16.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa62239c700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c1040d0 0x7fa61c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa621b9b700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c1982b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa618000c00 tx=0x7fa6180057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa612ffd700 1 -- 172.21.15.31:0/2744406416 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61801d070 con 0x7fa61c102550 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa612ffd700 1 -- 172.21.15.31:0/2744406416 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa61800fc40 con 0x7fa61c102550 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa612ffd700 1 -- 172.21.15.31:0/2744406416 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618017650 con 0x7fa61c102550 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c1a0380 con 0x7fa61c102550 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.450+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c1a07e0 con 0x7fa61c102550 2023-07-29T16:21:16.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.454+0000 7fa612ffd700 1 -- 172.21.15.31:0/2744406416 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa618021410 con 0x7fa61c102550 2023-07-29T16:21:16.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.455+0000 7fa63778fb80 1 Processor -- start 2023-07-29T16:21:16.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 -- start start 2023-07-29T16:21:16.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f5558ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5559530 0x55a8f5559d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f555a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f54b0a10 con 0x55a8f555a2f0 2023-07-29T16:21:16.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f54ae3d0 con 0x55a8f5559530 2023-07-29T16:21:16.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa63778fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f54a9cc0 con 0x55a8f5558be0 2023-07-29T16:21:16.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5559530 0x55a8f5559d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa62239c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f5558ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa612ffd700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6080a0810 0x7fa6080a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa62239c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f5558ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37722/0 (socket says 172.21.15.31:37722) 2023-07-29T16:21:16.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa62239c700 1 -- 172.21.15.31:0/2265164078 learned_addr learned my addr 172.21.15.31:0/2265164078 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.456+0000 7fa62139a700 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f555a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa62239c700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6080a0810 0x7fa6080a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa612ffd700 1 -- 172.21.15.31:0/2744406416 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6180ccff0 con 0x7fa61c102550 2023-07-29T16:21:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa621b9b700 1 -- 172.21.15.31:0/2265164078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 msgr2=0x55a8f5558ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa621b9b700 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f5558ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa621b9b700 1 -- 172.21.15.31:0/2265164078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 msgr2=0x55a8f555a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa621b9b700 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f555a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa621b9b700 1 -- 172.21.15.31:0/2265164078 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8f52d7de0 con 0x55a8f5559530 2023-07-29T16:21:16.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa621b9b700 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5559530 0x55a8f5559d60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa61800ffd0 tx=0x7fa61809bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a51a2eae1f8bc2c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa607fff700 1 -- 172.21.15.31:0/2265164078 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61809a280 con 0x55a8f5559530 2023-07-29T16:21:16.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa607fff700 1 -- 172.21.15.31:0/2265164078 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa61809a0a0 con 0x55a8f5559530 2023-07-29T16:21:16.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.457+0000 7fa607fff700 1 -- 172.21.15.31:0/2265164078 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61809bba0 con 0x55a8f5559530 2023-07-29T16:21:16.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 -- 172.21.15.31:0/2265164078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5559530 msgr2=0x55a8f5559d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa62239c700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6080a0810 0x7fa6080a2cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x55a8f5558980 tx=0x7fa614004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5559530 0x55a8f5559d60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa61800ffd0 tx=0x7fa61809bf90 comp rx=0 tx=0).stop 2023-07-29T16:21:16.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 -- 172.21.15.31:0/2265164078 shutdown_connections 2023-07-29T16:21:16.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f555a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f5558ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2265164078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5559530 0x55a8f5559d60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 -- 172.21.15.31:0/2265164078 >> 172.21.15.31:0/2265164078 conn(0x55a8f554e3e0 msgr2=0x55a8f5553990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 -- 172.21.15.31:0/2265164078 shutdown_connections 2023-07-29T16:21:16.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.458+0000 7fa63778fb80 1 -- 172.21.15.31:0/2265164078 wait complete. 2023-07-29T16:21:16.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.459+0000 7fa63778fb80 1 -- start start 2023-07-29T16:21:16.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.459+0000 7fa63778fb80 1 Processor -- start 2023-07-29T16:21:16.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f55624b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5559530 0x55a8f55629f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f5569a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f5446510 con 0x55a8f5559530 2023-07-29T16:21:16.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f54461b0 con 0x55a8f555a2f0 2023-07-29T16:21:16.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f532d920 con 0x55a8f5558be0 2023-07-29T16:21:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa63778fb80 10 client.0 _ll_register_callbacks cb 0x55a8f544f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f55624b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f55624b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37738/0 (socket says 172.21.15.31:37738) 2023-07-29T16:21:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa621b9b700 1 -- 172.21.15.31:0/3158078146 learned_addr learned my addr 172.21.15.31:0/3158078146 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa62139a700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5559530 0x55a8f55629f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.460+0000 7fa62239c700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f5569a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa621b9b700 1 -- 172.21.15.31:0/3158078146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 msgr2=0x55a8f5569a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa621b9b700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f5569a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa621b9b700 1 -- 172.21.15.31:0/3158078146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5559530 msgr2=0x55a8f55629f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa621b9b700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5559530 0x55a8f55629f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa621b9b700 1 -- 172.21.15.31:0/3158078146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c1a07e0 con 0x55a8f5558be0 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa621b9b700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f55624b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa618009550 tx=0x7fa61809a640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d9ccfee4dc2730 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa6057fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa6057fa700 1 -- 172.21.15.31:0/3158078146 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180b5030 con 0x55a8f5558be0 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa6057fa700 1 -- 172.21.15.31:0/3158078146 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6180b6070 con 0x55a8f5558be0 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8f54d4d10 con 0x55a8f5558be0 2023-07-29T16:21:16.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa6057fa700 1 -- 172.21.15.31:0/3158078146 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa61809a280 con 0x55a8f5558be0 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa63778fb80 10 client.27283 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a8f52c0c60 con 0x55a8f5558be0 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.461+0000 7fa6057fa700 1 -- 172.21.15.31:0/3158078146 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa61807d020 con 0x55a8f5558be0 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa6057fa700 1 -- 172.21.15.31:0/3158078146 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa6180a2670 con 0x55a8f5558be0 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa63778fb80 10 client.27283 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa63778fb80 10 client.27283 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:21:16.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa63778fb80 20 client.27283 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:16.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa63778fb80 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f556ddb0 0x55a8f55701c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa63778fb80 4 client.27283 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:16.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8f5527d70 con 0x55a8f556ddb0 2023-07-29T16:21:16.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.462+0000 7fa62139a700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f556ddb0 0x55a8f55701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.463+0000 7fa6057fa700 10 client.27283 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:16.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.463+0000 7fa62139a700 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f556ddb0 0x55a8f55701c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b825ed10a34903ff server_cookie=fbb6d03acf6a3663 in_seq=0 out_seq=0 2023-07-29T16:21:16.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.463+0000 7fa6057fa700 1 -- 172.21.15.31:0/3158078146 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a8f5527d70 con 0x55a8f556ddb0 2023-07-29T16:21:16.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.463+0000 7fa6057fa700 10 client.27283 handle_command_reply: tid=0 2023-07-29T16:21:16.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.463+0000 7fa63778fb80 1 client.27283 shutdown 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 msgr2=0x55a8f55624b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f55624b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa618009550 tx=0x7fa61809a640 comp rx=0 tx=0).stop 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 shutdown_connections 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f556ddb0 0x55a8f55701c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f5569a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5558be0 0x55a8f55624b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 --2- 172.21.15.31:0/3158078146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5559530 0x55a8f55629f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa6057fa700 0 client.27283 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 >> 172.21.15.31:0/3158078146 conn(0x55a8f554e3e0 msgr2=0x55a8f5553060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 shutdown_connections 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 1 -- 172.21.15.31:0/3158078146 wait complete. 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.464+0000 7fa63778fb80 20 client.27283 trim_cache size 0 max 16384 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.478+0000 7fa63778fb80 1 Processor -- start 2023-07-29T16:21:16.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.478+0000 7fa63778fb80 1 -- start start 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5558be0 0x55a8f55679e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5562260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f55627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa63778fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f5561960 con 0x55a8f5558be0 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa63778fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f5550460 con 0x55a8f555a2f0 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa63778fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f5563630 con 0x55a8f5559530 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5558be0 0x55a8f55679e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62139a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5562260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5558be0 0x55a8f55679e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55996/0 (socket says 172.21.15.31:55996) 2023-07-29T16:21:16.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62139a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5562260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37742/0 (socket says 172.21.15.31:37742) 2023-07-29T16:21:16.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62239c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f55627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa621b9b700 1 -- 172.21.15.31:0/2105780832 learned_addr learned my addr 172.21.15.31:0/2105780832 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62139a700 1 -- 172.21.15.31:0/2105780832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 msgr2=0x55a8f55627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62139a700 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f55627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62139a700 1 -- 172.21.15.31:0/2105780832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5558be0 msgr2=0x55a8f55679e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.479+0000 7fa62139a700 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5558be0 0x55a8f55679e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.480+0000 7fa62139a700 1 -- 172.21.15.31:0/2105780832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8f52fe250 con 0x55a8f5559530 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.480+0000 7fa62139a700 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5562260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55a8f55707a0 tx=0x7fa60c0073d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c2f55855e8a82a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.480+0000 7fa610bff700 1 -- 172.21.15.31:0/2105780832 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c016040 con 0x55a8f5559530 2023-07-29T16:21:16.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.480+0000 7fa610bff700 1 -- 172.21.15.31:0/2105780832 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa60c017040 con 0x55a8f5559530 2023-07-29T16:21:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.480+0000 7fa610bff700 1 -- 172.21.15.31:0/2105780832 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c011b20 con 0x55a8f5559530 2023-07-29T16:21:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 -- 172.21.15.31:0/2105780832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 msgr2=0x55a8f5562260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5562260 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x55a8f55707a0 tx=0x7fa60c0073d0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 -- 172.21.15.31:0/2105780832 shutdown_connections 2023-07-29T16:21:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f555a2f0 0x55a8f55627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f5558be0 0x55a8f55679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 --2- 172.21.15.31:0/2105780832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5562260 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 -- 172.21.15.31:0/2105780832 >> 172.21.15.31:0/2105780832 conn(0x55a8f554e3e0 msgr2=0x55a8f555d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 -- 172.21.15.31:0/2105780832 shutdown_connections 2023-07-29T16:21:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 -- 172.21.15.31:0/2105780832 wait complete. 2023-07-29T16:21:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.481+0000 7fa63778fb80 1 -- start start 2023-07-29T16:21:16.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 Processor -- start 2023-07-29T16:21:16.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 0x55a8f5567980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5565f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f55665c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f5566c90 con 0x55a8f555a2f0 2023-07-29T16:21:16.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f5566e00 con 0x55a8f5558be0 2023-07-29T16:21:16.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8f55bf920 con 0x55a8f5559530 2023-07-29T16:21:16.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa63778fb80 10 client.0 _ll_register_callbacks cb 0x55a8f544f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 0x55a8f5567980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa621b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 0x55a8f5567980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35878/0 (socket says 172.21.15.31:35878) 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.482+0000 7fa621b9b700 1 -- 172.21.15.31:0/981422857 learned_addr learned my addr 172.21.15.31:0/981422857 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa62139a700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5565f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa62239c700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f55665c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa621b9b700 1 -- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 msgr2=0x55a8f5565f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa621b9b700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5565f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa621b9b700 1 -- 172.21.15.31:0/981422857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 msgr2=0x55a8f55665c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa621b9b700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f55665c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa621b9b700 1 -- 172.21.15.31:0/981422857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c004040 con 0x55a8f5558be0 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa62239c700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f55665c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa621b9b700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 0x55a8f5567980 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa61809a210 tx=0x7fa618099ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29e04e8de30a0d4d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.483+0000 7fa6057fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:16.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.484+0000 7fa6057fa700 1 -- 172.21.15.31:0/981422857 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180b5020 con 0x55a8f5558be0 2023-07-29T16:21:16.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa6057fa700 1 -- 172.21.15.31:0/981422857 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa61807e080 con 0x55a8f5558be0 2023-07-29T16:21:16.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa61809a0a0 con 0x55a8f5558be0 2023-07-29T16:21:16.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa6057fa700 1 -- 172.21.15.31:0/981422857 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa61809a740 con 0x55a8f5558be0 2023-07-29T16:21:16.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 10 client.27278 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:16.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a8f52d62b0 con 0x55a8f5558be0 2023-07-29T16:21:16.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa6057fa700 1 -- 172.21.15.31:0/981422857 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa61809a0a0 con 0x55a8f5558be0 2023-07-29T16:21:16.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa6057fa700 1 -- 172.21.15.31:0/981422857 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa61809e170 con 0x55a8f5558be0 2023-07-29T16:21:16.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 10 client.27278 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:16.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 10 client.27278 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:21:16.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 20 client.27278 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:16.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f55c0d00 0x55a8f55c30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 4 client.27278 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.485+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a8f54a2d40 con 0x55a8f55c0d00 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.486+0000 7fa62139a700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f55c0d00 0x55a8f55c30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.486+0000 7fa6057fa700 10 client.27278 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.486+0000 7fa62139a700 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f55c0d00 0x55a8f55c30e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=22860f479492a7e1 server_cookie=be2fac1edede6e3c in_seq=0 out_seq=0 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa6057fa700 1 -- 172.21.15.31:0/981422857 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a8f54a2d40 con 0x55a8f55c0d00 2023-07-29T16:21:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa6057fa700 10 client.27278 handle_command_reply: tid=0 2023-07-29T16:21:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 client.27278 shutdown 2023-07-29T16:21:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 msgr2=0x55a8f5567980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 0x55a8f5567980 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa61809a210 tx=0x7fa618099ee0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 shutdown_connections 2023-07-29T16:21:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8f555a2f0 0x55a8f55665c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a8f55c0d00 0x55a8f55c30e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8f5558be0 0x55a8f5567980 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 --2- 172.21.15.31:0/981422857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8f5559530 0x55a8f5565f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 >> 172.21.15.31:0/981422857 conn(0x55a8f554e3e0 msgr2=0x55a8f5562260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa6057fa700 0 client.27278 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 shutdown_connections 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.487+0000 7fa63778fb80 1 -- 172.21.15.31:0/981422857 wait complete. 2023-07-29T16:21:16.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.488+0000 7fa63778fb80 20 client.27278 trim_cache size 0 max 16384 2023-07-29T16:21:16.500 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:21:16.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.488+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6080a0810 msgr2=0x7fa6080a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.488+0000 7fa623dff700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6080a0810 0x7fa6080a2cc0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x55a8f5558980 tx=0x7fa614004b40 comp rx=0 tx=0).stop 2023-07-29T16:21:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.488+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 msgr2=0x7fa61c1982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.488+0000 7fa623dff700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c1982b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa618000c00 tx=0x7fa6180057c0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 shutdown_connections 2023-07-29T16:21:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6080a0810 0x7fa6080a2cc0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa61c1040d0 0x7fa61c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa61c102550 0x7fa61c1982b0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 --2- 172.21.15.31:0/2744406416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa61c103740 0x7fa61c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 >> 172.21.15.31:0/2744406416 conn(0x7fa61c0fdb90 msgr2=0x7fa61c0ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 shutdown_connections 2023-07-29T16:21:16.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.489+0000 7fa623dff700 1 -- 172.21.15.31:0/2744406416 wait complete. 2023-07-29T16:21:16.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:21:16.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 -- 172.21.15.31:0/1098538811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c0737b0 msgr2=0x7ffa6c073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 --2- 172.21.15.31:0/1098538811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c0737b0 0x7ffa6c073c20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009990 tx=0x7ffa60009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 -- 172.21.15.31:0/1098538811 shutdown_connections 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 --2- 172.21.15.31:0/1098538811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa6c10c900 0x7ffa6c10ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 --2- 172.21.15.31:0/1098538811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa6c074d90 0x7ffa6c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 --2- 172.21.15.31:0/1098538811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c0737b0 0x7ffa6c073c20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 -- 172.21.15.31:0/1098538811 >> 172.21.15.31:0/1098538811 conn(0x7ffa6c0fb970 msgr2=0x7ffa6c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 -- 172.21.15.31:0/1098538811 shutdown_connections 2023-07-29T16:21:16.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.733+0000 7ffa709ff700 1 -- 172.21.15.31:0/1098538811 wait complete. 2023-07-29T16:21:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 Processor -- start 2023-07-29T16:21:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 -- start start 2023-07-29T16:21:16.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa6c0737b0 0x7ffa6c19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 0x7ffa6c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa6c10c900 0x7ffa6c1a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c1a4240 con 0x7ffa6c10c900 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c1a4380 con 0x7ffa6c0737b0 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.734+0000 7ffa709ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c1a4510 con 0x7ffa6c074d90 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 0x7ffa6c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 0x7ffa6c19cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37760/0 (socket says 172.21.15.31:37760) 2023-07-29T16:21:16.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 -- 172.21.15.31:0/921175600 learned_addr learned my addr 172.21.15.31:0/921175600 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa6c0737b0 0x7ffa6c19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa6c10c900 0x7ffa6c1a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 -- 172.21.15.31:0/921175600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa6c0737b0 msgr2=0x7ffa6c19c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa6c0737b0 0x7ffa6c19c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 -- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa6c10c900 msgr2=0x7ffa6c1a3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa6c10c900 0x7ffa6c1a3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 -- 172.21.15.31:0/921175600 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa600095f0 con 0x7ffa6c074d90 2023-07-29T16:21:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.735+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 0x7ffa6c19cc00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa60004710 tx=0x7ffa60009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.736+0000 7ffa5b7fe700 1 -- 172.21.15.31:0/921175600 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6001d070 con 0x7ffa6c074d90 2023-07-29T16:21:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.736+0000 7ffa5b7fe700 1 -- 172.21.15.31:0/921175600 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa60021410 con 0x7ffa6c074d90 2023-07-29T16:21:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.736+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c1a4790 con 0x7ffa6c074d90 2023-07-29T16:21:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.736+0000 7ffa5b7fe700 1 -- 172.21.15.31:0/921175600 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60017650 con 0x7ffa6c074d90 2023-07-29T16:21:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.736+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c1a4bb0 con 0x7ffa6c074d90 2023-07-29T16:21:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.739+0000 7ffa5b7fe700 1 -- 172.21.15.31:0/921175600 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ffa60021580 con 0x7ffa6c074d90 2023-07-29T16:21:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.740+0000 7ffa8436cb80 1 Processor -- start 2023-07-29T16:21:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.740+0000 7ffa8436cb80 1 -- start start 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa5b7fe700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffa540a0790 0x7ffa540a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47638dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47638ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47638f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4762e5a10 con 0x55b47638e530 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4762e33d0 con 0x55b47638dbe0 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4762decc0 con 0x55b47638f2f0 2023-07-29T16:21:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa5b7fe700 1 -- 172.21.15.31:0/921175600 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ffa600ccf00 con 0x7ffa6c074d90 2023-07-29T16:21:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47638dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47638dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35890/0 (socket says 172.21.15.31:35890) 2023-07-29T16:21:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa6a59c700 1 -- 172.21.15.31:0/2623114137 learned_addr learned my addr 172.21.15.31:0/2623114137 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffa540a0790 0x7ffa540a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47638ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.741+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47638f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6a59c700 1 -- 172.21.15.31:0/2623114137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 msgr2=0x55b47638f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47638f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6a59c700 1 -- 172.21.15.31:0/2623114137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 msgr2=0x55b47638ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47638ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6a59c700 1 -- 172.21.15.31:0/2623114137 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b47610cde0 con 0x55b47638dbe0 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47638dff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009d20 tx=0x7ffa5c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccf4b0b07528c73e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffa540a0790 0x7ffa540a2c40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009ec0 tx=0x7ffa64009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa58ff9700 1 -- 172.21.15.31:0/2623114137 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c019630 con 0x55b47638dbe0 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa58ff9700 1 -- 172.21.15.31:0/2623114137 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa5c019c50 con 0x55b47638dbe0 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.742+0000 7ffa58ff9700 1 -- 172.21.15.31:0/2623114137 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00eb90 con 0x55b47638dbe0 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 -- 172.21.15.31:0/2623114137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 msgr2=0x55b47638dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47638dff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009d20 tx=0x7ffa5c011720 comp rx=0 tx=0).stop 2023-07-29T16:21:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 -- 172.21.15.31:0/2623114137 shutdown_connections 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47638f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47638dff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/2623114137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47638ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 -- 172.21.15.31:0/2623114137 >> 172.21.15.31:0/2623114137 conn(0x55b4763833e0 msgr2=0x55b476388990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 -- 172.21.15.31:0/2623114137 shutdown_connections 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.743+0000 7ffa8436cb80 1 -- 172.21.15.31:0/2623114137 wait complete. 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 -- start start 2023-07-29T16:21:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 Processor -- start 2023-07-29T16:21:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638dbe0 0x55b4763977d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b476397d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 0x55b47639ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47627b510 con 0x55b47638dbe0 2023-07-29T16:21:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47627b1b0 con 0x55b47638f2f0 2023-07-29T16:21:16.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b476162920 con 0x55b47638e530 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.744+0000 7ffa8436cb80 10 client.0 _ll_register_callbacks cb 0x55b476284620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa6ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 0x55b47639ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b476397d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b476397d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37786/0 (socket says 172.21.15.31:37786) 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638dbe0 0x55b4763977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa6ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 0x55b47639ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35892/0 (socket says 172.21.15.31:35892) 2023-07-29T16:21:16.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 -- 172.21.15.31:0/4272833092 learned_addr learned my addr 172.21.15.31:0/4272833092 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 -- 172.21.15.31:0/4272833092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 msgr2=0x55b47639ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 0x55b47639ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 -- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638dbe0 msgr2=0x55b4763977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638dbe0 0x55b4763977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 -- 172.21.15.31:0/4272833092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5c009980 con 0x55b47638e530 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 0x55b47639ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:16.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b476397d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffa600ca940 tx=0x7ffa60021b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e445c4c1affef71 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa51ffb700 1 -- 172.21.15.31:0/4272833092 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa60099d20 con 0x55b47638e530 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa51ffb700 1 -- 172.21.15.31:0/4272833092 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ffa6009a630 con 0x55b47638e530 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.745+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b476309d10 con 0x55b47638e530 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa51ffb700 1 -- 172.21.15.31:0/4272833092 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa6009e660 con 0x55b47638e530 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 10 client.27301 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b4760f5c60 con 0x55b47638e530 2023-07-29T16:21:16.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa51ffb700 1 -- 172.21.15.31:0/4272833092 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b476309d10 con 0x55b47638e530 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa51ffb700 1 -- 172.21.15.31:0/4272833092 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ffa600a25f0 con 0x55b47638e530 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 10 client.27301 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 10 client.27301 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 20 client.27301 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763a3210 0x55b4763a5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 4 client.27301 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b47635cd70 con 0x55b4763a3210 2023-07-29T16:21:16.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.746+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763a3210 0x55b4763a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.747+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763a3210 0x55b4763a5620 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb7c5a0d740a519e server_cookie=ee4b3491dda7aec5 in_seq=0 out_seq=0 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.747+0000 7ffa51ffb700 10 client.27301 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa51ffb700 1 -- 172.21.15.31:0/4272833092 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b47635cd70 con 0x55b4763a3210 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa51ffb700 10 client.27301 handle_command_reply: tid=0 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 client.27301 shutdown 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 msgr2=0x55b476397d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b476397d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffa600ca940 tx=0x7ffa60021b60 comp rx=0 tx=0).stop 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 shutdown_connections 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763a3210 0x55b4763a5620 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638f2f0 0x55b47639ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa51ffb700 0 client.27301 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638dbe0 0x55b4763977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/4272833092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b476397d10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 >> 172.21.15.31:0/4272833092 conn(0x55b4763833e0 msgr2=0x55b476388060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 shutdown_connections 2023-07-29T16:21:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 1 -- 172.21.15.31:0/4272833092 wait complete. 2023-07-29T16:21:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.748+0000 7ffa8436cb80 20 client.27301 trim_cache size 0 max 16384 2023-07-29T16:21:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 Processor -- start 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 -- start start 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b4763956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638f2f0 0x55b476395c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b476396cc0 con 0x55b47638f2f0 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b476397370 con 0x55b47638dbe0 2023-07-29T16:21:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.762+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4763854b0 con 0x55b47638e530 2023-07-29T16:21:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35908/0 (socket says 172.21.15.31:35908) 2023-07-29T16:21:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1835242768 learned_addr learned my addr 172.21.15.31:0/1835242768 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b4763956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638f2f0 0x55b476395c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1835242768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 msgr2=0x55b4763956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b4763956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1835242768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638f2f0 msgr2=0x55b476395c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638f2f0 0x55b476395c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1835242768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b476133250 con 0x55b47638dbe0 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b4763956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638f2f0 0x55b476395c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.763+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639ccd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005950 tx=0x7ffa5c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd738c71845f669 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa597fa700 1 -- 172.21.15.31:0/1835242768 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00e580 con 0x55b47638dbe0 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa597fa700 1 -- 172.21.15.31:0/1835242768 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa5c00edd0 con 0x55b47638dbe0 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa597fa700 1 -- 172.21.15.31:0/1835242768 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c005040 con 0x55b47638dbe0 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1835242768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 msgr2=0x55b47639ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639ccd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c005950 tx=0x7ffa5c000bc0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1835242768 shutdown_connections 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638f2f0 0x55b476395c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639ccd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1835242768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638e530 0x55b4763956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1835242768 >> 172.21.15.31:0/1835242768 conn(0x55b4763833e0 msgr2=0x55b476389690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1835242768 shutdown_connections 2023-07-29T16:21:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.764+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1835242768 wait complete. 2023-07-29T16:21:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.765+0000 7ffa8436cb80 1 -- start start 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.765+0000 7ffa8436cb80 1 Processor -- start 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.765+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47639cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa8436cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47639af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47639b630 con 0x55b47638e530 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47639b7a0 con 0x55b47638dbe0 2023-07-29T16:21:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa8436cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b47639b960 con 0x55b47638f2f0 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa8436cb80 10 client.0 _ll_register_callbacks cb 0x55b476284620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35920/0 (socket says 172.21.15.31:35920) 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1629192640 learned_addr learned my addr 172.21.15.31:0/1629192640 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6ad9d700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47639af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47639cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1629192640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 msgr2=0x55b47639af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47639af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 msgr2=0x55b47639cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47639cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa6a59c700 1 -- 172.21.15.31:0/1629192640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4760f5c60 con 0x55b47638dbe0 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.766+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47639cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa6a59c700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639c820 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c011b20 tx=0x7ffa5c00ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55ac491ced3857dc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa51ffb700 1 -- 172.21.15.31:0/1629192640 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c019430 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa51ffb700 1 -- 172.21.15.31:0/1629192640 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ffa5c019590 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa5c019c50 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa51ffb700 1 -- 172.21.15.31:0/1629192640 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa5c00e910 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa8436cb80 10 client.27296 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b47610cde0 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.767+0000 7ffa51ffb700 1 -- 172.21.15.31:0/1629192640 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffa5c019c50 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa51ffb700 1 -- 172.21.15.31:0/1629192640 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ffa5c02b030 con 0x55b47638dbe0 2023-07-29T16:21:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa8436cb80 10 client.27296 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa8436cb80 10 client.27296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa8436cb80 20 client.27296 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763f6040 0x55b4763f8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa8436cb80 4 client.27296 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b4762d7d40 con 0x55b4763f6040 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763f6040 0x55b4763f8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa69d9b700 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763f6040 0x55b4763f8450 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d70763a5f0242d0a server_cookie=e170372596abdab8 in_seq=0 out_seq=0 2023-07-29T16:21:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.768+0000 7ffa51ffb700 10 client.27296 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa51ffb700 1 -- 172.21.15.31:0/1629192640 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b4762d7d40 con 0x55b4763f6040 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa51ffb700 10 client.27296 handle_command_reply: tid=0 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 client.27296 shutdown 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 msgr2=0x55b47639c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639c820 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c011b20 tx=0x7ffa5c00ef40 comp rx=0 tx=0).stop 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 shutdown_connections 2023-07-29T16:21:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55b4763f6040 0x55b4763f8450 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b47638f2f0 0x55b47639af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b47638dbe0 0x55b47639c820 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa51ffb700 0 client.27296 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 --2- 172.21.15.31:0/1629192640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b47638e530 0x55b47639cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 >> 172.21.15.31:0/1629192640 conn(0x55b4763833e0 msgr2=0x55b4763956c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 shutdown_connections 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 1 -- 172.21.15.31:0/1629192640 wait complete. 2023-07-29T16:21:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.769+0000 7ffa8436cb80 20 client.27296 trim_cache size 0 max 16384 2023-07-29T16:21:16.777 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:21:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffa540a0790 msgr2=0x7ffa540a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffa540a0790 0x7ffa540a2c40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ffa64009ec0 tx=0x7ffa64009920 comp rx=0 tx=0).stop 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 msgr2=0x7ffa6c19cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 0x7ffa6c19cc00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ffa60004710 tx=0x7ffa60009580 comp rx=0 tx=0).stop 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 shutdown_connections 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffa540a0790 0x7ffa540a2c40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa6c10c900 0x7ffa6c1a3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa6c0737b0 0x7ffa6c19c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 --2- 172.21.15.31:0/921175600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa6c074d90 0x7ffa6c19cc00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 >> 172.21.15.31:0/921175600 conn(0x7ffa6c0fb970 msgr2=0x7ffa6c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 shutdown_connections 2023-07-29T16:21:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:16.770+0000 7ffa709ff700 1 -- 172.21.15.31:0/921175600 wait complete. 2023-07-29T16:21:16.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:16 smithi039 ceph-mon[128858]: pgmap v544: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2023-07-29T16:21:16.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:16 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/529436229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:21:16.785 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:16 smithi031 ceph-mon[95504]: pgmap v544: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2023-07-29T16:21:16.786 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:16 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/529436229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:21:16.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:21:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:16 smithi156 ceph-mon[129245]: pgmap v544: 97 pgs: 97 active+clean; 72 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 259 op/s 2023-07-29T16:21:16.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:16 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/529436229' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:21:16.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 305502 9.00 MB/sec execute 61 sec latency 34.406 ms 2023-07-29T16:21:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 -- 172.21.15.31:0/2883631372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 msgr2=0x7f147c100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 --2- 172.21.15.31:0/2883631372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c100730 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1478009990 tx=0x7f1478009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 -- 172.21.15.31:0/2883631372 shutdown_connections 2023-07-29T16:21:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 --2- 172.21.15.31:0/2883631372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f147c101ea0 0x7f147c108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 --2- 172.21.15.31:0/2883631372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c100730 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 --2- 172.21.15.31:0/2883631372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f147c101510 0x7f147c101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.003+0000 7f1483fff700 1 -- 172.21.15.31:0/2883631372 >> 172.21.15.31:0/2883631372 conn(0x7f147c0fb920 msgr2=0x7f147c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.004+0000 7f1483fff700 1 -- 172.21.15.31:0/2883631372 shutdown_connections 2023-07-29T16:21:17.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.004+0000 7f1483fff700 1 -- 172.21.15.31:0/2883631372 wait complete. 2023-07-29T16:21:17.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 Processor -- start 2023-07-29T16:21:17.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 -- start start 2023-07-29T16:21:17.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c1960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f147c101510 0x7f147c196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f147c101ea0 0x7f147c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c19dc50 con 0x7f147c101ea0 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c19dd90 con 0x7f147c100320 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1483fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c19df20 con 0x7f147c101510 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1481d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c1960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1481d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c1960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35932/0 (socket says 172.21.15.31:35932) 2023-07-29T16:21:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.005+0000 7f1481d9b700 1 -- 172.21.15.31:0/854985802 learned_addr learned my addr 172.21.15.31:0/854985802 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f148159a700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f147c101510 0x7f147c196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f148259c700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f147c101ea0 0x7f147c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1481d9b700 1 -- 172.21.15.31:0/854985802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f147c101510 msgr2=0x7f147c196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1481d9b700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f147c101510 0x7f147c196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1481d9b700 1 -- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f147c101ea0 msgr2=0x7f147c19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1481d9b700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f147c101ea0 0x7f147c19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1481d9b700 1 -- 172.21.15.31:0/854985802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14780095f0 con 0x7f147c100320 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1481d9b700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c1960d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f14780098e0 tx=0x7f1478004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.006+0000 7f1472ffd700 1 -- 172.21.15.31:0/854985802 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147801d070 con 0x7f147c100320 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.007+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c19e1a0 con 0x7f147c100320 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.007+0000 7f1472ffd700 1 -- 172.21.15.31:0/854985802 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14780054f0 con 0x7f147c100320 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.007+0000 7f1472ffd700 1 -- 172.21.15.31:0/854985802 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1478004ca0 con 0x7f147c100320 2023-07-29T16:21:17.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.007+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c19e570 con 0x7f147c100320 2023-07-29T16:21:17.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.011+0000 7f1472ffd700 1 -- 172.21.15.31:0/854985802 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f147800fb90 con 0x7f147c100320 2023-07-29T16:21:17.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.012+0000 7f14979d6b80 1 Processor -- start 2023-07-29T16:21:17.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.012+0000 7f14979d6b80 1 -- start start 2023-07-29T16:21:17.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42e0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e0940 0x5652b42e1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f14979d6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b4233a10 con 0x5652b42e0940 2023-07-29T16:21:17.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f14979d6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42313d0 con 0x5652b42dfff0 2023-07-29T16:21:17.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f14979d6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b422ccc0 con 0x5652b42e1700 2023-07-29T16:21:17.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42e0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42e0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35934/0 (socket says 172.21.15.31:35934) 2023-07-29T16:21:17.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 -- 172.21.15.31:0/2083659069 learned_addr learned my addr 172.21.15.31:0/2083659069 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:17.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f1481d9b700 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e0940 0x5652b42e1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148159a700 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 -- 172.21.15.31:0/2083659069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 msgr2=0x5652b42e1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 -- 172.21.15.31:0/2083659069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e0940 msgr2=0x5652b42e1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e0940 0x5652b42e1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.013+0000 7f148259c700 1 -- 172.21.15.31:0/2083659069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652b405ade0 con 0x5652b42dfff0 2023-07-29T16:21:17.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f1472ffd700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14680a0790 0x7f14680a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f148259c700 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42e0400 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1474009dc0 tx=0x7f14740077e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ec78843c068f35a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:17.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f1472ffd700 1 -- 172.21.15.31:0/854985802 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f14780ccde0 con 0x7f147c100320 2023-07-29T16:21:17.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f1481d9b700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14680a0790 0x7f14680a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f1467fff700 1 -- 172.21.15.31:0/2083659069 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474014610 con 0x5652b42dfff0 2023-07-29T16:21:17.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f1467fff700 1 -- 172.21.15.31:0/2083659069 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f147400f050 con 0x5652b42dfff0 2023-07-29T16:21:17.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.014+0000 7f1467fff700 1 -- 172.21.15.31:0/2083659069 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1474013960 con 0x5652b42dfff0 2023-07-29T16:21:17.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 -- 172.21.15.31:0/2083659069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 msgr2=0x5652b42e0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f1481d9b700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14680a0790 0x7f14680a2c40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1478096a10 tx=0x7f14780d07e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42e0400 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1474009dc0 tx=0x7f14740077e0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 -- 172.21.15.31:0/2083659069 shutdown_connections 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42e0400 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 --2- 172.21.15.31:0/2083659069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e0940 0x5652b42e1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 -- 172.21.15.31:0/2083659069 >> 172.21.15.31:0/2083659069 conn(0x5652b42d13e0 msgr2=0x5652b42d6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:17.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 -- 172.21.15.31:0/2083659069 shutdown_connections 2023-07-29T16:21:17.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 -- 172.21.15.31:0/2083659069 wait complete. 2023-07-29T16:21:17.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.015+0000 7f14979d6b80 1 -- start start 2023-07-29T16:21:17.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 Processor -- start 2023-07-29T16:21:17.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42dfff0 0x5652b42e9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ea1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42f1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b41c9510 con 0x5652b42e1700 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b41c91b0 con 0x5652b42e0940 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b40b0920 con 0x5652b42dfff0 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.016+0000 7f14979d6b80 10 client.0 _ll_register_callbacks cb 0x5652b41d2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42dfff0 0x5652b42e9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ea1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42dfff0 0x5652b42e9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37840/0 (socket says 172.21.15.31:37840) 2023-07-29T16:21:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148259c700 1 -- 172.21.15.31:0/1905055769 learned_addr learned my addr 172.21.15.31:0/1905055769 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148259c700 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42f1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 -- 172.21.15.31:0/1905055769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42dfff0 msgr2=0x5652b42e9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42dfff0 0x5652b42e9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 -- 172.21.15.31:0/1905055769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 msgr2=0x5652b42f1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42f1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 -- 172.21.15.31:0/1905055769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1474009a20 con 0x5652b42e0940 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f148159a700 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ea1c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f146c0116c0 tx=0x7f146c0119d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d936a72bc3faa2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f14657fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.017+0000 7f14657fa700 1 -- 172.21.15.31:0/1905055769 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c016070 con 0x5652b42e0940 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652b4257d10 con 0x5652b42e0940 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14657fa700 1 -- 172.21.15.31:0/1905055769 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f146c004320 con 0x5652b42e0940 2023-07-29T16:21:17.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14657fa700 1 -- 172.21.15.31:0/1905055769 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146c0044e0 con 0x5652b42e0940 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 10 client.27308 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5652b4043c60 con 0x5652b42e0940 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14657fa700 1 -- 172.21.15.31:0/1905055769 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5652b4257d10 con 0x5652b42e0940 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14657fa700 1 -- 172.21.15.31:0/1905055769 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f146c00f7d0 con 0x5652b42e0940 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 10 client.27308 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 10 client.27308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:17.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 20 client.27308 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:17.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b42f5aa0 0x5652b42f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 4 client.27308 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:17.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.018+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652b42aad70 con 0x5652b42f5aa0 2023-07-29T16:21:17.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.019+0000 7f148259c700 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b42f5aa0 0x5652b42f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.019+0000 7f148259c700 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b42f5aa0 0x5652b42f7eb0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=261aa982a875c08 server_cookie=10a45ad3e9fd1001 in_seq=0 out_seq=0 2023-07-29T16:21:17.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.019+0000 7f14657fa700 10 client.27308 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:17.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14657fa700 1 -- 172.21.15.31:0/1905055769 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5652b42aad70 con 0x5652b42f5aa0 2023-07-29T16:21:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14657fa700 10 client.27308 handle_command_reply: tid=0 2023-07-29T16:21:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 client.27308 shutdown 2023-07-29T16:21:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 msgr2=0x5652b42ea1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ea1c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f146c0116c0 tx=0x7f146c0119d0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 shutdown_connections 2023-07-29T16:21:17.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b42f5aa0 0x5652b42f7eb0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42f1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42dfff0 0x5652b42e9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14657fa700 0 client.27308 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 --2- 172.21.15.31:0/1905055769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ea1c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 >> 172.21.15.31:0/1905055769 conn(0x5652b42d13e0 msgr2=0x5652b42d6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 shutdown_connections 2023-07-29T16:21:17.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 1 -- 172.21.15.31:0/1905055769 wait complete. 2023-07-29T16:21:17.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.023+0000 7f14979d6b80 20 client.27308 trim_cache size 0 max 16384 2023-07-29T16:21:17.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 Processor -- start 2023-07-29T16:21:17.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 -- start start 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 0x5652b42f7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ef110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42ce6e0 con 0x5652b42dfff0 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42d3460 con 0x5652b42e0940 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.037+0000 7f14979d6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42f2ef0 con 0x5652b42e1700 2023-07-29T16:21:17.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 0x5652b42f7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 0x5652b42f7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56104/0 (socket says 172.21.15.31:56104) 2023-07-29T16:21:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 -- 172.21.15.31:0/3866913595 learned_addr learned my addr 172.21.15.31:0/3866913595 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148159a700 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ef110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 -- 172.21.15.31:0/3866913595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 msgr2=0x5652b42e1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 -- 172.21.15.31:0/3866913595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 msgr2=0x5652b42ef110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ef110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 -- 172.21.15.31:0/3866913595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652b4081250 con 0x5652b42dfff0 2023-07-29T16:21:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148159a700 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:17.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.038+0000 7f148259c700 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 0x5652b42f7e80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1474007b30 tx=0x7f1474007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5aba213d790e3ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:17.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f1470ff9700 1 -- 172.21.15.31:0/3866913595 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147402d490 con 0x5652b42dfff0 2023-07-29T16:21:17.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f1470ff9700 1 -- 172.21.15.31:0/3866913595 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14740138c0 con 0x5652b42dfff0 2023-07-29T16:21:17.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f1470ff9700 1 -- 172.21.15.31:0/3866913595 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147402d9d0 con 0x5652b42dfff0 2023-07-29T16:21:17.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 -- 172.21.15.31:0/3866913595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 msgr2=0x5652b42f7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 0x5652b42f7e80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f1474007b30 tx=0x7f1474007f50 comp rx=0 tx=0).stop 2023-07-29T16:21:17.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 -- 172.21.15.31:0/3866913595 shutdown_connections 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e1700 0x5652b42e1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42dfff0 0x5652b42f7e80 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3866913595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42e0940 0x5652b42ef110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.039+0000 7f14979d6b80 1 -- 172.21.15.31:0/3866913595 >> 172.21.15.31:0/3866913595 conn(0x5652b42d13e0 msgr2=0x5652b42e8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.040+0000 7f14979d6b80 1 -- 172.21.15.31:0/3866913595 shutdown_connections 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.040+0000 7f14979d6b80 1 -- 172.21.15.31:0/3866913595 wait complete. 2023-07-29T16:21:17.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.040+0000 7f14979d6b80 1 -- start start 2023-07-29T16:21:17.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.040+0000 7f14979d6b80 1 Processor -- start 2023-07-29T16:21:17.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42ed350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 0x5652b42ed890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42ee9e0 con 0x5652b42e1700 2023-07-29T16:21:17.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42eeb50 con 0x5652b42dfff0 2023-07-29T16:21:17.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652b42eed10 con 0x5652b42e0940 2023-07-29T16:21:17.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f14979d6b80 10 client.0 _ll_register_callbacks cb 0x5652b41d2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f148159a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 0x5652b42ed890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f148159a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 0x5652b42ed890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37856/0 (socket says 172.21.15.31:37856) 2023-07-29T16:21:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f148159a700 1 -- 172.21.15.31:0/3877253072 learned_addr learned my addr 172.21.15.31:0/3877253072 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:17.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f148259c700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42ed350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.041+0000 7f148259c700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148159a700 1 -- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 msgr2=0x5652b42ed350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148159a700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42ed350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148159a700 1 -- 172.21.15.31:0/3877253072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 msgr2=0x5652b42ee4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148159a700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42ee4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148159a700 1 -- 172.21.15.31:0/3877253072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652b4043c60 con 0x5652b42e0940 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148259c700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f148159a700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 0x5652b42ed890 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f146c00fb60 tx=0x7f146c005330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63ad8edea04c6d7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14657fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14657fa700 1 -- 172.21.15.31:0/3877253072 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c016040 con 0x5652b42e0940 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f147400f050 con 0x5652b42e0940 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14657fa700 1 -- 172.21.15.31:0/3877253072 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f146c000e40 con 0x5652b42e0940 2023-07-29T16:21:17.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14657fa700 1 -- 172.21.15.31:0/3877253072 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f146c005690 con 0x5652b42e0940 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14979d6b80 10 client.27325 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.042+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f1474028040 con 0x5652b42e0940 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14657fa700 1 -- 172.21.15.31:0/3877253072 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f147400f050 con 0x5652b42e0940 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14657fa700 1 -- 172.21.15.31:0/3877253072 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f146c035070 con 0x5652b42e0940 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14979d6b80 10 client.27325 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14979d6b80 10 client.27325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14979d6b80 20 client.27325 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:17.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b43485a0 0x5652b434a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14979d6b80 4 client.27325 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:21:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5652b4225d40 con 0x5652b43485a0 2023-07-29T16:21:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.043+0000 7f148259c700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b43485a0 0x5652b434a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14657fa700 10 client.27325 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:17.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f148259c700 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b43485a0 0x5652b434a9b0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9414a4d75e7ee1e server_cookie=b2d269ad5ce91945 in_seq=0 out_seq=0 2023-07-29T16:21:17.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14657fa700 1 -- 172.21.15.31:0/3877253072 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5652b4225d40 con 0x5652b43485a0 2023-07-29T16:21:17.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14657fa700 10 client.27325 handle_command_reply: tid=0 2023-07-29T16:21:17.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 client.27325 shutdown 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 msgr2=0x5652b42ed890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 0x5652b42ed890 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f146c00fb60 tx=0x7f146c005330 comp rx=0 tx=0).stop 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 shutdown_connections 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652b43485a0 0x5652b434a9b0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652b42e1700 0x5652b42ee4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652b42dfff0 0x5652b42ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 --2- 172.21.15.31:0/3877253072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652b42e0940 0x5652b42ed890 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14657fa700 0 client.27325 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 >> 172.21.15.31:0/3877253072 conn(0x5652b42d13e0 msgr2=0x5652b42e7b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 shutdown_connections 2023-07-29T16:21:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.044+0000 7f14979d6b80 1 -- 172.21.15.31:0/3877253072 wait complete. 2023-07-29T16:21:17.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f14979d6b80 20 client.27325 trim_cache size 0 max 16384 2023-07-29T16:21:17.054 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:21:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14680a0790 msgr2=0x7f14680a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14680a0790 0x7f14680a2c40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1478096a10 tx=0x7f14780d07e0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 msgr2=0x7f147c1960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:17.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c1960d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f14780098e0 tx=0x7f1478004730 comp rx=0 tx=0).stop 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 shutdown_connections 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f14680a0790 0x7f14680a2c40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f147c101ea0 0x7f147c19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f147c100320 0x7f147c1960d0 secure :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f14780098e0 tx=0x7f1478004730 comp rx=0 tx=0).stop 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 --2- 172.21.15.31:0/854985802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f147c101510 0x7f147c196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 >> 172.21.15.31:0/854985802 conn(0x7f147c0fb920 msgr2=0x7f147c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:17.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 shutdown_connections 2023-07-29T16:21:17.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:17.045+0000 7f1483fff700 1 -- 172.21.15.31:0/854985802 wait complete. 2023-07-29T16:21:17.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 307501 8.97 MB/sec execute 62 sec latency 43.513 ms 2023-07-29T16:21:18.064 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:21:18.065 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:21:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:17 smithi031 ceph-mon[95504]: pgmap v545: 97 pgs: 97 active+clean; 74 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 284 op/s 2023-07-29T16:21:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:17 smithi039 ceph-mon[128858]: pgmap v545: 97 pgs: 97 active+clean; 74 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 284 op/s 2023-07-29T16:21:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 -- 172.21.15.31:0/2743017643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708100e40 msgr2=0x7fa708103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/2743017643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708100e40 0x7fa708103220 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8009990 tx=0x7fa6f8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 -- 172.21.15.31:0/2743017643 shutdown_connections 2023-07-29T16:21:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/2743017643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708106080 0x7fa708108460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/2743017643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708100e40 0x7fa708103220 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/2743017643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa708103760 0x7fa708105b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 -- 172.21.15.31:0/2743017643 >> 172.21.15.31:0/2743017643 conn(0x7fa7080fa840 msgr2=0x7fa7080fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.283+0000 7fa70d1ff700 1 -- 172.21.15.31:0/2743017643 shutdown_connections 2023-07-29T16:21:18.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.284+0000 7fa70d1ff700 1 -- 172.21.15.31:0/2743017643 wait complete. 2023-07-29T16:21:18.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 Processor -- start 2023-07-29T16:21:18.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 -- start start 2023-07-29T16:21:18.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708100e40 0x7fa708198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 0x7fa7081987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa708106080 0x7fa70819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70819fde0 con 0x7fa708100e40 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70819ff20 con 0x7fa708103760 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa70d1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7081a00b0 con 0x7fa708106080 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.285+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708100e40 0x7fa708198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa70659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 0x7fa7081987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa70659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 0x7fa7081987a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35964/0 (socket says 172.21.15.31:35964) 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa70659c700 1 -- 172.21.15.31:0/1318591651 learned_addr learned my addr 172.21.15.31:0/1318591651 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa70759e700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa708106080 0x7fa70819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa706d9d700 1 -- 172.21.15.31:0/1318591651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa708106080 msgr2=0x7fa70819f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa706d9d700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa708106080 0x7fa70819f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa706d9d700 1 -- 172.21.15.31:0/1318591651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 msgr2=0x7fa7081987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa706d9d700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 0x7fa7081987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa706d9d700 1 -- 172.21.15.31:0/1318591651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f80095f0 con 0x7fa708100e40 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa70659c700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 0x7fa7081987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.286+0000 7fa706d9d700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708100e40 0x7fa708198260 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098e0 tx=0x7fa6f8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.287+0000 7fa6effff700 1 -- 172.21.15.31:0/1318591651 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f801d070 con 0x7fa708100e40 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.287+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7081a0330 con 0x7fa708100e40 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.287+0000 7fa6effff700 1 -- 172.21.15.31:0/1318591651 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6f8021410 con 0x7fa708100e40 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.287+0000 7fa6effff700 1 -- 172.21.15.31:0/1318591651 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8005720 con 0x7fa708100e40 2023-07-29T16:21:18.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.287+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7081a0750 con 0x7fa708100e40 2023-07-29T16:21:18.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.290+0000 7fa6effff700 1 -- 172.21.15.31:0/1318591651 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa6f8021580 con 0x7fa708100e40 2023-07-29T16:21:18.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.291+0000 7fa720aa0b80 1 Processor -- start 2023-07-29T16:21:18.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 -- start start 2023-07-29T16:21:18.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f1e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa6effff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6f00a0790 0x7fa6f00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f1ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f1f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164e75a90 con 0x557164f1dc60 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164e73450 con 0x557164f1f370 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164e6ed40 con 0x557164f1e5b0 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa6effff700 1 -- 172.21.15.31:0/1318591651 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6f80ccdd0 con 0x7fa708100e40 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f1e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa70659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f1ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa70659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f1ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37880/0 (socket says 172.21.15.31:37880) 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.292+0000 7fa70659c700 1 -- 172.21.15.31:0/2996683401 learned_addr learned my addr 172.21.15.31:0/2996683401 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70759e700 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f1f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70759e700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6f00a0790 0x7fa6f00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70659c700 1 -- 172.21.15.31:0/2996683401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 msgr2=0x557164f1f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70659c700 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f1f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70659c700 1 -- 172.21.15.31:0/2996683401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 msgr2=0x557164f1e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70659c700 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f1e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70659c700 1 -- 172.21.15.31:0/2996683401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557164c9ce60 con 0x557164f1e5b0 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70659c700 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f1ede0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0117e0 tx=0x7fa6fc011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f4c92325b31e37 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa6ed7fa700 1 -- 172.21.15.31:0/2996683401 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc00f480 con 0x557164f1e5b0 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa70759e700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6f00a0790 0x7fa6f00a2c40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa700009fd0 tx=0x7fa70000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa6ed7fa700 1 -- 172.21.15.31:0/2996683401 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6fc00faa0 con 0x557164f1e5b0 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.293+0000 7fa6ed7fa700 1 -- 172.21.15.31:0/2996683401 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc019b60 con 0x557164f1e5b0 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 -- 172.21.15.31:0/2996683401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 msgr2=0x557164f1ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f1ede0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0117e0 tx=0x7fa6fc011af0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 -- 172.21.15.31:0/2996683401 shutdown_connections 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f1f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f1e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/2996683401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f1ede0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.294+0000 7fa720aa0b80 1 -- 172.21.15.31:0/2996683401 >> 172.21.15.31:0/2996683401 conn(0x557164f13460 msgr2=0x557164f18a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.295+0000 7fa720aa0b80 1 -- 172.21.15.31:0/2996683401 shutdown_connections 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.295+0000 7fa720aa0b80 1 -- 172.21.15.31:0/2996683401 wait complete. 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.295+0000 7fa720aa0b80 1 -- start start 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.295+0000 7fa720aa0b80 1 Processor -- start 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f27870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f27db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f2edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164e0b590 con 0x557164f1f370 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164e0b230 con 0x557164f1dc60 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164cf29a0 con 0x557164f1e5b0 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa720aa0b80 10 client.0 _ll_register_callbacks cb 0x557164e146a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f27870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa70659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f27db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa70659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f27db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37888/0 (socket says 172.21.15.31:37888) 2023-07-29T16:21:18.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f27870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35986/0 (socket says 172.21.15.31:35986) 2023-07-29T16:21:18.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa70659c700 1 -- 172.21.15.31:0/3520644401 learned_addr learned my addr 172.21.15.31:0/3520644401 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.296+0000 7fa70759e700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f2edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa70659c700 1 -- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 msgr2=0x557164f27870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa70659c700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f27870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa70659c700 1 -- 172.21.15.31:0/3520644401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 msgr2=0x557164f2edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa70659c700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f2edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa70659c700 1 -- 172.21.15.31:0/3520644401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6fc011440 con 0x557164f1e5b0 2023-07-29T16:21:18.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa706d9d700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f27870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa70659c700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f27db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00a010 tx=0x7fa6fc005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b92d79cfcc7f7e8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa6eaff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3520644401 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc005ce0 con 0x557164f1e5b0 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3520644401 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6fc004250 con 0x557164f1e5b0 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.297+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557164e99d90 con 0x557164f1e5b0 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3520644401 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6fc004410 con 0x557164f1e5b0 2023-07-29T16:21:18.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 10 client.27337 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557164c85ce0 con 0x557164f1e5b0 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3520644401 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557164e99d90 con 0x557164f1e5b0 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3520644401 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa6fc0052f0 con 0x557164f1e5b0 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 10 client.27337 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 10 client.27337 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 20 client.27337 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f335a0 0x557164f359b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 4 client.27337 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.298+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557164eecdf0 con 0x557164f335a0 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.299+0000 7fa706d9d700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f335a0 0x557164f359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.299+0000 7fa706d9d700 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f335a0 0x557164f359b0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd7de23db12bef7c server_cookie=585fc3060c7bd2fb in_seq=0 out_seq=0 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.299+0000 7fa6eaff5700 10 client.27337 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3520644401 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557164eecdf0 con 0x557164f335a0 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa6eaff5700 10 client.27337 handle_command_reply: tid=0 2023-07-29T16:21:18.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 client.27337 shutdown 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 msgr2=0x557164f27db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f27db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00a010 tx=0x7fa6fc005810 comp rx=0 tx=0).stop 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 shutdown_connections 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f335a0 0x557164f359b0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f2edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f27870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa6eaff5700 0 client.27337 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3520644401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f27db0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 >> 172.21.15.31:0/3520644401 conn(0x557164f13460 msgr2=0x557164f180e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 shutdown_connections 2023-07-29T16:21:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.300+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3520644401 wait complete. 2023-07-29T16:21:18.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.301+0000 7fa720aa0b80 20 client.27337 trim_cache size 0 max 16384 2023-07-29T16:21:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.319+0000 7fa720aa0b80 1 Processor -- start 2023-07-29T16:21:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.319+0000 7fa720aa0b80 1 -- start start 2023-07-29T16:21:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.319+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f2cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f22810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164f26d50 con 0x557164f1f370 2023-07-29T16:21:18.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164f15510 con 0x557164f1dc60 2023-07-29T16:21:18.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164f33310 con 0x557164f1e5b0 2023-07-29T16:21:18.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f2cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa70659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f2cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36002/0 (socket says 172.21.15.31:36002) 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 -- 172.21.15.31:0/1422724050 learned_addr learned my addr 172.21.15.31:0/1422724050 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa70759e700 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f22810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 -- 172.21.15.31:0/1422724050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 msgr2=0x557164f222d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f222d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 -- 172.21.15.31:0/1422724050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 msgr2=0x557164f22810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f22810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa706d9d700 1 -- 172.21.15.31:0/1422724050 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557164cc32d0 con 0x557164f1dc60 2023-07-29T16:21:18.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.320+0000 7fa70659c700 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.321+0000 7fa706d9d700 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f2cda0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8096fd0 tx=0x7fa6f80ca740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec0ea6d0bb9cae0a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.321+0000 7fa6edffb700 1 -- 172.21.15.31:0/1422724050 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f80b6070 con 0x557164f1dc60 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.321+0000 7fa6edffb700 1 -- 172.21.15.31:0/1422724050 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6f809e590 con 0x557164f1dc60 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.321+0000 7fa6edffb700 1 -- 172.21.15.31:0/1422724050 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8099a50 con 0x557164f1dc60 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 -- 172.21.15.31:0/1422724050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 msgr2=0x557164f2cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f2cda0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8096fd0 tx=0x7fa6f80ca740 comp rx=0 tx=0).stop 2023-07-29T16:21:18.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 -- 172.21.15.31:0/1422724050 shutdown_connections 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1f370 0x557164f22810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1dc60 0x557164f2cda0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/1422724050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 -- 172.21.15.31:0/1422724050 >> 172.21.15.31:0/1422724050 conn(0x557164f13460 msgr2=0x557164f25720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 -- 172.21.15.31:0/1422724050 shutdown_connections 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 -- 172.21.15.31:0/1422724050 wait complete. 2023-07-29T16:21:18.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 -- start start 2023-07-29T16:21:18.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:17 smithi156 ceph-mon[129245]: pgmap v545: 97 pgs: 97 active+clean; 74 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 284 op/s 2023-07-29T16:21:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.322+0000 7fa720aa0b80 1 Processor -- start 2023-07-29T16:21:18.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f2b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f2bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f2c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164f85210 con 0x557164f1dc60 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164f85380 con 0x557164f1f370 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557164f85540 con 0x557164f1e5b0 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa720aa0b80 10 client.0 _ll_register_callbacks cb 0x557164e146a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:18.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa70759e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f2c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa706d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f2b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa70759e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f2c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36004/0 (socket says 172.21.15.31:36004) 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa70759e700 1 -- 172.21.15.31:0/3996021970 learned_addr learned my addr 172.21.15.31:0/3996021970 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa70659c700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f2bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa706d9d700 1 -- 172.21.15.31:0/3996021970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 msgr2=0x557164f2bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa706d9d700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f2bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa706d9d700 1 -- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 msgr2=0x557164f2c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa706d9d700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f2c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa706d9d700 1 -- 172.21.15.31:0/3996021970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557164c85ce0 con 0x557164f1dc60 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa70759e700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f2c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.323+0000 7fa70659c700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f2bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa706d9d700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f2b840 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8000c00 tx=0x7fa6f8009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa386bd3937f216e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa6eaff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3996021970 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f80b6070 con 0x557164f1dc60 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3996021970 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6f80986a0 con 0x557164f1dc60 2023-07-29T16:21:18.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6fc00faa0 con 0x557164f1dc60 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3996021970 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6f80caa80 con 0x557164f1dc60 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa720aa0b80 10 client.17307 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fa7081a0750 con 0x557164f1dc60 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.324+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3996021970 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6f8064020 con 0x557164f1dc60 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3996021970 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa6f8096350 con 0x557164f1dc60 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa720aa0b80 10 client.17307 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:18.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa720aa0b80 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa720aa0b80 20 client.17307 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f86b80 0x557164f88f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa720aa0b80 4 client.17307 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557164e67dc0 con 0x557164f86b80 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa70659c700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f86b80 0x557164f88f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa70659c700 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f86b80 0x557164f88f90 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0aecb112473bc0c server_cookie=b8a8f5025639e4d4 in_seq=0 out_seq=0 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.325+0000 7fa6eaff5700 10 client.17307 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa6eaff5700 1 -- 172.21.15.31:0/3996021970 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557164e67dc0 con 0x557164f86b80 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa6eaff5700 10 client.17307 handle_command_reply: tid=0 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 client.17307 shutdown 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 msgr2=0x557164f2b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f2b840 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa6f8000c00 tx=0x7fa6f8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 shutdown_connections 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557164f86b80 0x557164f88f90 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557164f1f370 0x557164f2c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557164f1dc60 0x557164f2b840 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 --2- 172.21.15.31:0/3996021970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557164f1e5b0 0x557164f2bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 >> 172.21.15.31:0/3996021970 conn(0x557164f13460 msgr2=0x557164f222d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa6eaff5700 0 client.17307 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 shutdown_connections 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 1 -- 172.21.15.31:0/3996021970 wait complete. 2023-07-29T16:21:18.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.327+0000 7fa720aa0b80 20 client.17307 trim_cache size 0 max 16384 2023-07-29T16:21:18.335 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:21:18.335 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:21:18.335 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "0e625b9d-854e-4a28-8f44-656a5e7e15d1", 2023-07-29T16:21:18.335 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:21:18.335 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:21:18.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6f00a0790 msgr2=0x7fa6f00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6f00a0790 0x7fa6f00a2c40 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa700009fd0 tx=0x7fa70000b040 comp rx=0 tx=0).stop 2023-07-29T16:21:18.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708100e40 msgr2=0x7fa708198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708100e40 0x7fa708198260 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80098e0 tx=0x7fa6f8009580 comp rx=0 tx=0).stop 2023-07-29T16:21:18.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 shutdown_connections 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6f00a0790 0x7fa6f00a2c40 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa708106080 0x7fa70819f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa708100e40 0x7fa708198260 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 --2- 172.21.15.31:0/1318591651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa708103760 0x7fa7081987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 >> 172.21.15.31:0/1318591651 conn(0x7fa7080fa840 msgr2=0x7fa7080ff4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 shutdown_connections 2023-07-29T16:21:18.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.328+0000 7fa70d1ff700 1 -- 172.21.15.31:0/1318591651 wait complete. 2023-07-29T16:21:18.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:21:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 -- 172.21.15.31:0/3740551622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 msgr2=0x7fa304108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 --2- 172.21.15.31:0/3740551622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa304108760 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00b0a0 tx=0x7fa2fc00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 -- 172.21.15.31:0/3740551622 shutdown_connections 2023-07-29T16:21:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 --2- 172.21.15.31:0/3740551622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa304108760 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 --2- 172.21.15.31:0/3740551622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304100320 0x7fa304100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 --2- 172.21.15.31:0/3740551622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304101510 0x7fa304101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 -- 172.21.15.31:0/3740551622 >> 172.21.15.31:0/3740551622 conn(0x7fa3040fb920 msgr2=0x7fa3040fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 -- 172.21.15.31:0/3740551622 shutdown_connections 2023-07-29T16:21:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.565+0000 7fa3093ff700 1 -- 172.21.15.31:0/3740551622 wait complete. 2023-07-29T16:21:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.566+0000 7fa3093ff700 1 Processor -- start 2023-07-29T16:21:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.566+0000 7fa3093ff700 1 -- start start 2023-07-29T16:21:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3093ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304100320 0x7fa304196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3093ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304101510 0x7fa3041966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3093ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa30419d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3093ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30419dd10 con 0x7fa304100320 2023-07-29T16:21:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3093ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30419de50 con 0x7fa304101ea0 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3093ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30419dfe0 con 0x7fa304101510 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304101510 0x7fa3041966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304100320 0x7fa304196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3037fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa30419d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304100320 0x7fa304196190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56164/0 (socket says 172.21.15.31:56164) 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304101510 0x7fa3041966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37908/0 (socket says 172.21.15.31:37908) 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.567+0000 7fa302ffd700 1 -- 172.21.15.31:0/417437123 learned_addr learned my addr 172.21.15.31:0/417437123 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3037fe700 1 -- 172.21.15.31:0/417437123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304101510 msgr2=0x7fa3041966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3037fe700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304101510 0x7fa3041966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3037fe700 1 -- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304100320 msgr2=0x7fa304196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3037fe700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304100320 0x7fa304196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3037fe700 1 -- 172.21.15.31:0/417437123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc009d00 con 0x7fa304101ea0 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3037fe700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa30419d740 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00b070 tx=0x7fa2fc00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa2ebfff700 1 -- 172.21.15.31:0/417437123 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00e040 con 0x7fa304101ea0 2023-07-29T16:21:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa2ebfff700 1 -- 172.21.15.31:0/417437123 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2fc009480 con 0x7fa304101ea0 2023-07-29T16:21:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30419e260 con 0x7fa304101ea0 2023-07-29T16:21:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa2ebfff700 1 -- 172.21.15.31:0/417437123 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0052a0 con 0x7fa304101ea0 2023-07-29T16:21:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.568+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30419e630 con 0x7fa304101ea0 2023-07-29T16:21:18.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.573+0000 7fa2ebfff700 1 -- 172.21.15.31:0/417437123 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa2fc01d480 con 0x7fa304101ea0 2023-07-29T16:21:18.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.574+0000 7fa31ccf7b80 1 Processor -- start 2023-07-29T16:21:18.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.574+0000 7fa31ccf7b80 1 -- start start 2023-07-29T16:21:18.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22cbe0 0x556f9b22cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b22dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b22e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b184a10 con 0x556f9b22cbe0 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b1823d0 con 0x556f9b22d530 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b17dcc0 con 0x556f9b22e2f0 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa2ebfff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2ec0a0790 0x7fa2ec0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b22dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.584 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22cbe0 0x556f9b22cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b22dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36018/0 (socket says 172.21.15.31:36018) 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa302ffd700 1 -- 172.21.15.31:0/196428293 learned_addr learned my addr 172.21.15.31:0/196428293 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa2ebfff700 1 -- 172.21.15.31:0/417437123 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa2fc0ccd20 con 0x7fa304101ea0 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa3037fe700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2ec0a0790 0x7fa2ec0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa3027fc700 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b22e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa302ffd700 1 -- 172.21.15.31:0/196428293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 msgr2=0x556f9b22e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.575+0000 7fa302ffd700 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b22e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa302ffd700 1 -- 172.21.15.31:0/196428293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22cbe0 msgr2=0x556f9b22cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa302ffd700 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22cbe0 0x556f9b22cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa302ffd700 1 -- 172.21.15.31:0/196428293 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f9afabde0 con 0x556f9b22d530 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa3027fc700 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b22e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa302ffd700 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b22dd60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009df0 tx=0x7fa2f4011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b63ad882c9ec9c5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa2e97fa700 1 -- 172.21.15.31:0/196428293 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4004230 con 0x556f9b22d530 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa2e97fa700 1 -- 172.21.15.31:0/196428293 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2f4004390 con 0x556f9b22d530 2023-07-29T16:21:18.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.576+0000 7fa2e97fa700 1 -- 172.21.15.31:0/196428293 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40198c0 con 0x556f9b22d530 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/196428293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 msgr2=0x556f9b22dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b22dd60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4009df0 tx=0x7fa2f4011720 comp rx=0 tx=0).stop 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa3037fe700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2ec0a0790 0x7fa2ec0a2c40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc096950 tx=0x7fa2fc009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/196428293 shutdown_connections 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b22e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22cbe0 0x556f9b22cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/196428293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b22dd60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/196428293 >> 172.21.15.31:0/196428293 conn(0x556f9b2223e0 msgr2=0x556f9b227990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/196428293 shutdown_connections 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.577+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/196428293 wait complete. 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 -- start start 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 Processor -- start 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b236890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 0x556f9b236dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b23de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b11a510 con 0x556f9b22d530 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b11a1b0 con 0x556f9b22cbe0 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.578+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b001920 con 0x556f9b22e2f0 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa31ccf7b80 10 client.0 _ll_register_callbacks cb 0x556f9b123620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 0x556f9b236dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b236890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 0x556f9b236dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56176/0 (socket says 172.21.15.31:56176) 2023-07-29T16:21:18.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 -- 172.21.15.31:0/3333875635 learned_addr learned my addr 172.21.15.31:0/3333875635 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa3027fc700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b23de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 -- 172.21.15.31:0/3333875635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 msgr2=0x556f9b23de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b23de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 -- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 msgr2=0x556f9b236890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b236890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa302ffd700 1 -- 172.21.15.31:0/3333875635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f4009a50 con 0x556f9b22d530 2023-07-29T16:21:18.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.579+0000 7fa3027fc700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b236890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa302ffd700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 0x556f9b236dd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000c00 tx=0x7fa2f40038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ee5c1e040bad86 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa2e6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/3333875635 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4005250 con 0x556f9b22d530 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556f9b1a8d10 con 0x556f9b22d530 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/3333875635 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa2f40053b0 con 0x556f9b22d530 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/3333875635 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2f4005570 con 0x556f9b22d530 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa31ccf7b80 10 client.17319 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556f9af94c60 con 0x556f9b22d530 2023-07-29T16:21:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.580+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/3333875635 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556f9b1a8d10 con 0x556f9b22d530 2023-07-29T16:21:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/3333875635 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa2f4021040 con 0x556f9b22d530 2023-07-29T16:21:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa31ccf7b80 10 client.17319 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa31ccf7b80 10 client.17319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa31ccf7b80 20 client.17319 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b2424f0 0x556f9b244900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa31ccf7b80 4 client.17319 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f9b1fbd70 con 0x556f9b2424f0 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.581+0000 7fa3027fc700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b2424f0 0x556f9b244900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa2e6ff5700 10 client.17319 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa3027fc700 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b2424f0 0x556f9b244900 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9ea140265e9cbb4 server_cookie=ef05432b48e87463 in_seq=0 out_seq=0 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/3333875635 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556f9b1fbd70 con 0x556f9b2424f0 2023-07-29T16:21:18.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa2e6ff5700 10 client.17319 handle_command_reply: tid=0 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa31ccf7b80 1 client.17319 shutdown 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 msgr2=0x556f9b236dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.582+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 0x556f9b236dd0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4000c00 tx=0x7fa2f40038b0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 shutdown_connections 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b2424f0 0x556f9b244900 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22e2f0 0x556f9b23de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b236890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/3333875635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22d530 0x556f9b236dd0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 >> 172.21.15.31:0/3333875635 conn(0x556f9b2223e0 msgr2=0x556f9b227060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa2e6ff5700 0 client.17319 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 shutdown_connections 2023-07-29T16:21:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/3333875635 wait complete. 2023-07-29T16:21:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.583+0000 7fa31ccf7b80 20 client.17319 trim_cache size 0 max 16384 2023-07-29T16:21:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.602+0000 7fa31ccf7b80 1 Processor -- start 2023-07-29T16:21:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.602+0000 7fa31ccf7b80 1 -- start start 2023-07-29T16:21:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b2346e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 0x556f9b2311f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b231730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b21f6e0 con 0x556f9b22e2f0 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b234490 con 0x556f9b22cbe0 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b2379e0 con 0x556f9b22d530 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 0x556f9b2311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b2346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 0x556f9b2311f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37936/0 (socket says 172.21.15.31:37936) 2023-07-29T16:21:18.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b2346e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36028/0 (socket says 172.21.15.31:36028) 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 -- 172.21.15.31:0/2165235103 learned_addr learned my addr 172.21.15.31:0/2165235103 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa3027fc700 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b231730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 -- 172.21.15.31:0/2165235103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 msgr2=0x556f9b2346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b2346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 -- 172.21.15.31:0/2165235103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 msgr2=0x556f9b231730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b231730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 -- 172.21.15.31:0/2165235103 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f9b235810 con 0x556f9b22d530 2023-07-29T16:21:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa3027fc700 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b231730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.603+0000 7fa302ffd700 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 0x556f9b2311f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4004770 tx=0x7fa2f4003920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa305b5490ba76b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa2e9ffb700 1 -- 172.21.15.31:0/2165235103 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4004840 con 0x556f9b22d530 2023-07-29T16:21:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa2e9ffb700 1 -- 172.21.15.31:0/2165235103 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2f4019950 con 0x556f9b22d530 2023-07-29T16:21:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa2e9ffb700 1 -- 172.21.15.31:0/2165235103 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40236a0 con 0x556f9b22d530 2023-07-29T16:21:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/2165235103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 msgr2=0x556f9b2311f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 0x556f9b2311f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4004770 tx=0x7fa2f4003920 comp rx=0 tx=0).stop 2023-07-29T16:21:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/2165235103 shutdown_connections 2023-07-29T16:21:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b231730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.604+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22cbe0 0x556f9b2346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.605+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/2165235103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22d530 0x556f9b2311f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.605+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/2165235103 >> 172.21.15.31:0/2165235103 conn(0x556f9b2223e0 msgr2=0x556f9b244500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.605+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/2165235103 shutdown_connections 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.605+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/2165235103 wait complete. 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.605+0000 7fa31ccf7b80 1 -- start start 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.605+0000 7fa31ccf7b80 1 Processor -- start 2023-07-29T16:21:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22cbe0 0x556f9b2902e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b290820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b297860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b297da0 con 0x556f9b22e2f0 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b297ee0 con 0x556f9b22d530 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f9b298070 con 0x556f9b22cbe0 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa31ccf7b80 10 client.0 _ll_register_callbacks cb 0x556f9b123620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:18.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa3027fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22cbe0 0x556f9b2902e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b290820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b290820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36038/0 (socket says 172.21.15.31:36038) 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 -- 172.21.15.31:0/159467279 learned_addr learned my addr 172.21.15.31:0/159467279 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa3027fc700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b297860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 -- 172.21.15.31:0/159467279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22cbe0 msgr2=0x556f9b2902e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22cbe0 0x556f9b2902e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 -- 172.21.15.31:0/159467279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 msgr2=0x556f9b297860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b297860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa302ffd700 1 -- 172.21.15.31:0/159467279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f9af94c60 con 0x556f9b22d530 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.606+0000 7fa3027fc700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22cbe0 0x556f9b2902e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa302ffd700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b290820 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005e90 tx=0x7fa2f4019d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1675fc30de2851db server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa2e6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/159467279 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f403d070 con 0x556f9b22d530 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/159467279 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa2f4019630 con 0x556f9b22d530 2023-07-29T16:21:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2f4004390 con 0x556f9b22d530 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/159467279 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2f4004840 con 0x556f9b22d530 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa31ccf7b80 10 client.27350 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.607+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556f9afd2250 con 0x556f9b22d530 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/159467279 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa2f4047020 con 0x556f9b22d530 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/159467279 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa2f4040030 con 0x556f9b22d530 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa31ccf7b80 10 client.27350 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa31ccf7b80 10 client.27350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa31ccf7b80 20 client.27350 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b299540 0x556f9b29b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa31ccf7b80 4 client.27350 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556f9b176d40 con 0x556f9b299540 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.608+0000 7fa3027fc700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b299540 0x556f9b29b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa2e6ff5700 10 client.27350 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa3027fc700 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b299540 0x556f9b29b950 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68cc5cff5eab5419 server_cookie=413ccf37799e5469 in_seq=0 out_seq=0 2023-07-29T16:21:18.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa2e6ff5700 1 -- 172.21.15.31:0/159467279 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556f9b176d40 con 0x556f9b299540 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa2e6ff5700 10 client.27350 handle_command_reply: tid=0 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 client.27350 shutdown 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 msgr2=0x556f9b290820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b290820 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4005e90 tx=0x7fa2f4019d70 comp rx=0 tx=0).stop 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 shutdown_connections 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556f9b299540 0x556f9b29b950 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f9b22e2f0 0x556f9b297860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556f9b22cbe0 0x556f9b2902e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 --2- 172.21.15.31:0/159467279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556f9b22d530 0x556f9b290820 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa2e6ff5700 0 client.27350 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 >> 172.21.15.31:0/159467279 conn(0x556f9b2223e0 msgr2=0x556f9b2346e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 shutdown_connections 2023-07-29T16:21:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 1 -- 172.21.15.31:0/159467279 wait complete. 2023-07-29T16:21:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.609+0000 7fa31ccf7b80 20 client.27350 trim_cache size 0 max 16384 2023-07-29T16:21:18.618 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:21:18.618 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:21:18.618 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:21:18.618 INFO:teuthology.orchestra.run.smithi031.stdout: "0e625b9d-854e-4a28-8f44-656a5e7e15d1": { 2023-07-29T16:21:18.618 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:21:18.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "0e625b9d-854e-4a28-8f44-656a5e7e15d1", 2023-07-29T16:21:18.619 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:21:18.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:21:18.619 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:21:18.619 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:21:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2ec0a0790 msgr2=0x7fa2ec0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2ec0a0790 0x7fa2ec0a2c40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc096950 tx=0x7fa2fc009040 comp rx=0 tx=0).stop 2023-07-29T16:21:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 msgr2=0x7fa30419d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa30419d740 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00b070 tx=0x7fa2fc00bc80 comp rx=0 tx=0).stop 2023-07-29T16:21:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 shutdown_connections 2023-07-29T16:21:18.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa2ec0a0790 0x7fa2ec0a2c40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa304101ea0 0x7fa30419d740 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa304100320 0x7fa304196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 --2- 172.21.15.31:0/417437123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa304101510 0x7fa3041966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 >> 172.21.15.31:0/417437123 conn(0x7fa3040fb920 msgr2=0x7fa3040fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 shutdown_connections 2023-07-29T16:21:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:18.610+0000 7fa3093ff700 1 -- 172.21.15.31:0/417437123 wait complete. 2023-07-29T16:21:18.627 INFO:tasks.cephfs.filesystem:scrub status for tag:0e625b9d-854e-4a28-8f44-656a5e7e15d1 - {'path': '/', 'tag': '0e625b9d-854e-4a28-8f44-656a5e7e15d1', 'options': 'recursive,force'} 2023-07-29T16:21:18.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 308997 9.02 MB/sec execute 63 sec latency 19.289 ms 2023-07-29T16:21:19.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:21:19.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:21:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:21:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:19.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 310653 8.97 MB/sec execute 64 sec latency 16.675 ms 2023-07-29T16:21:20.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:19 smithi031 ceph-mon[95504]: pgmap v546: 97 pgs: 97 active+clean; 75 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 326 op/s 2023-07-29T16:21:20.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:19 smithi039 ceph-mon[128858]: pgmap v546: 97 pgs: 97 active+clean; 75 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 326 op/s 2023-07-29T16:21:20.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:19 smithi156 ceph-mon[129245]: pgmap v546: 97 pgs: 97 active+clean; 75 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 326 op/s 2023-07-29T16:21:20.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 312435 9.03 MB/sec execute 65 sec latency 73.227 ms 2023-07-29T16:21:21.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 313714 8.98 MB/sec execute 66 sec latency 20.081 ms 2023-07-29T16:21:22.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:22 smithi039 ceph-mon[128858]: pgmap v547: 97 pgs: 97 active+clean; 75 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1.9 MiB/s wr, 226 op/s 2023-07-29T16:21:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 -- 172.21.15.31:0/839982379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 msgr2=0x7f7aa8100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/839982379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8100770 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4009990 tx=0x7f7aa4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 -- 172.21.15.31:0/839982379 shutdown_connections 2023-07-29T16:21:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/839982379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101ee0 0x7f7aa81087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/839982379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8100770 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/839982379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aa8101550 0x7f7aa81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 -- 172.21.15.31:0/839982379 >> 172.21.15.31:0/839982379 conn(0x7f7aa80fb960 msgr2=0x7f7aa80fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 -- 172.21.15.31:0/839982379 shutdown_connections 2023-07-29T16:21:22.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.817+0000 7f7ab07ff700 1 -- 172.21.15.31:0/839982379 wait complete. 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.818+0000 7f7ab07ff700 1 Processor -- start 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 -- start start 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101550 0x7f7aa81988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aa8101ee0 0x7f7aa8199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8199d90 con 0x7f7aa8101ee0 2023-07-29T16:21:22.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa8199f00 con 0x7f7aa8100360 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7ab07ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa819a0c0 con 0x7f7aa8101550 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44526/0 (socket says 172.21.15.31:44526) 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7aae59b700 1 -- 172.21.15.31:0/3315652483 learned_addr learned my addr 172.21.15.31:0/3315652483 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7aadd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101550 0x7f7aa81988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.819+0000 7f7aaed9c700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aa8101ee0 0x7f7aa8199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aae59b700 1 -- 172.21.15.31:0/3315652483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101550 msgr2=0x7f7aa81988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aae59b700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101550 0x7f7aa81988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aae59b700 1 -- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aa8101ee0 msgr2=0x7f7aa8199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aae59b700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aa8101ee0 0x7f7aa8199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aae59b700 1 -- 172.21.15.31:0/3315652483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa40095f0 con 0x7f7aa8100360 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101550 0x7f7aa81988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7aae59b700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8198370 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4000c00 tx=0x7f7aa40057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:22.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7a9f7fe700 1 -- 172.21.15.31:0/3315652483 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa401d070 con 0x7f7aa8100360 2023-07-29T16:21:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7a9f7fe700 1 -- 172.21.15.31:0/3315652483 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aa400fbc0 con 0x7f7aa8100360 2023-07-29T16:21:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa819a340 con 0x7f7aa8100360 2023-07-29T16:21:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7a9f7fe700 1 -- 172.21.15.31:0/3315652483 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4017650 con 0x7f7aa8100360 2023-07-29T16:21:22.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.820+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa81a0b60 con 0x7f7aa8100360 2023-07-29T16:21:22.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:22 smithi156 ceph-mon[129245]: pgmap v547: 97 pgs: 97 active+clean; 75 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1.9 MiB/s wr, 226 op/s 2023-07-29T16:21:22.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.824+0000 7f7a9f7fe700 1 -- 172.21.15.31:0/3315652483 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f7aa400fd30 con 0x7f7aa8100360 2023-07-29T16:21:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 Processor -- start 2023-07-29T16:21:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 -- start start 2023-07-29T16:21:22.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7a9f7fe700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7a940a0790 0x7f7a940a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 0x55565dd6f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd6fdf0 0x55565dd707b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd71160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dcc79b0 con 0x55565dd70cf0 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dcc5370 con 0x55565dd6fdf0 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.826+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dcc0c60 con 0x55565dbcb6b0 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7a9f7fe700 1 -- 172.21.15.31:0/3315652483 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7aa40cd060 con 0x7f7aa8100360 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 0x55565dd6f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aaed9c700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7a940a0790 0x7f7a940a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 0x55565dd6f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40906/0 (socket says 172.21.15.31:40906) 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aadd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd6fdf0 0x55565dd707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 -- 172.21.15.31:0/1126376151 learned_addr learned my addr 172.21.15.31:0/1126376151 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aaed9c700 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd71160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 -- 172.21.15.31:0/1126376151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd6fdf0 msgr2=0x55565dd707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd6fdf0 0x55565dd707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 -- 172.21.15.31:0/1126376151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 msgr2=0x55565dd71160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd71160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 -- 172.21.15.31:0/1126376151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55565daeed80 con 0x55565dbcb6b0 2023-07-29T16:21:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.827+0000 7f7aae59b700 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 0x55565dd6f8b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7aa409a720 tx=0x7f7aa4005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d21585c7bdb16bcc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7aaed9c700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7a940a0790 0x7f7a940a2c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55565dd6f610 tx=0x7f7aa0009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7a9cff9700 1 -- 172.21.15.31:0/1126376151 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa400c4b0 con 0x55565dbcb6b0 2023-07-29T16:21:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7a9cff9700 1 -- 172.21.15.31:0/1126376151 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aa4015300 con 0x55565dbcb6b0 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7a9cff9700 1 -- 172.21.15.31:0/1126376151 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4098be0 con 0x55565dbcb6b0 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 -- 172.21.15.31:0/1126376151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 msgr2=0x55565dd6f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 0x55565dd6f8b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7aa409a720 tx=0x7f7aa4005fd0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 -- 172.21.15.31:0/1126376151 shutdown_connections 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd71160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dbcb6b0 0x55565dd6f8b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/1126376151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd6fdf0 0x55565dd707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.828+0000 7f7ac4052b80 1 -- 172.21.15.31:0/1126376151 >> 172.21.15.31:0/1126376151 conn(0x55565dd64fd0 msgr2=0x55565dd6a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.829+0000 7f7ac4052b80 1 -- 172.21.15.31:0/1126376151 shutdown_connections 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.829+0000 7f7ac4052b80 1 -- 172.21.15.31:0/1126376151 wait complete. 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.829+0000 7f7ac4052b80 1 -- start start 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.829+0000 7f7ac4052b80 1 Processor -- start 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd78b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd79070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dc5d4b0 con 0x55565dd70cf0 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dc5d150 con 0x55565dbcb6b0 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565db448c0 con 0x55565dd6fdf0 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7ac4052b80 10 client.0 _ll_register_callbacks cb 0x55565dc665c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7aadd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd79070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd78b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7aadd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd79070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40916/0 (socket says 172.21.15.31:40916) 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.830+0000 7f7aadd9a700 1 -- 172.21.15.31:0/2051973168 learned_addr learned my addr 172.21.15.31:0/2051973168 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aaed9c700 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd800b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aadd9a700 1 -- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 msgr2=0x55565dd78b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd78b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aadd9a700 1 -- 172.21.15.31:0/2051973168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 msgr2=0x55565dd800b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd800b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aadd9a700 1 -- 172.21.15.31:0/2051973168 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa81a0b60 con 0x55565dd6fdf0 2023-07-29T16:21:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd79070 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7a98012d20 tx=0x7f7a9800f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f8d4a702c1b72c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7a91ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7a91ffb700 1 -- 172.21.15.31:0/2051973168 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98017070 con 0x55565dd6fdf0 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55565dcebcb0 con 0x55565dd6fdf0 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.831+0000 7f7a91ffb700 1 -- 172.21.15.31:0/2051973168 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7a9800fc30 con 0x55565dd6fdf0 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7a91ffb700 1 -- 172.21.15.31:0/2051973168 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a980122e0 con 0x55565dd6fdf0 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 10 client.27370 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:22.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55565dad7c00 con 0x55565dd6fdf0 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7a91ffb700 1 -- 172.21.15.31:0/2051973168 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55565dcebcb0 con 0x55565dd6fdf0 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7a91ffb700 1 -- 172.21.15.31:0/2051973168 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f7a98010430 con 0x55565dd6fdf0 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 10 client.27370 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 10 client.27370 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 20 client.27370 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565dd84950 0x55565dd86d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 4 client.27370 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:22.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55565dd3ed10 con 0x55565dd84950 2023-07-29T16:21:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.832+0000 7f7aae59b700 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565dd84950 0x55565dd86d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.833+0000 7f7a91ffb700 10 client.27370 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.833+0000 7f7aae59b700 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565dd84950 0x55565dd86d60 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=718f0f81084b3b04 server_cookie=1ad02bc3c6924fd0 in_seq=0 out_seq=0 2023-07-29T16:21:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7a91ffb700 1 -- 172.21.15.31:0/2051973168 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55565dd3ed10 con 0x55565dd84950 2023-07-29T16:21:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7a91ffb700 10 client.27370 handle_command_reply: tid=0 2023-07-29T16:21:22.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 client.27370 shutdown 2023-07-29T16:21:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 msgr2=0x55565dd79070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd79070 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7a98012d20 tx=0x7f7a9800f6a0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 shutdown_connections 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565dd84950 0x55565dd86d60 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd800b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd78b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7a91ffb700 0 client.27370 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/2051973168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd79070 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 >> 172.21.15.31:0/2051973168 conn(0x55565dd64fd0 msgr2=0x55565dd69a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:22.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 shutdown_connections 2023-07-29T16:21:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 1 -- 172.21.15.31:0/2051973168 wait complete. 2023-07-29T16:21:22.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.834+0000 7f7ac4052b80 20 client.27370 trim_cache size 0 max 16384 2023-07-29T16:21:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 Processor -- start 2023-07-29T16:21:22.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 -- start start 2023-07-29T16:21:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dbcb6b0 0x55565dd7dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd69a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd70cf0 0x55565dd69f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dd6b5f0 con 0x55565dbcb6b0 2023-07-29T16:21:22.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dd77fe0 con 0x55565dd70cf0 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dd786a0 con 0x55565dd6fdf0 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7aaed9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd70cf0 0x55565dd69f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7aadd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd69a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7aaed9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd70cf0 0x55565dd69f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44562/0 (socket says 172.21.15.31:44562) 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7aaed9c700 1 -- 172.21.15.31:0/447903053 learned_addr learned my addr 172.21.15.31:0/447903053 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.850+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dbcb6b0 0x55565dd7dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7aadd9a700 1 -- 172.21.15.31:0/447903053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd70cf0 msgr2=0x55565dd69f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd70cf0 0x55565dd69f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7aadd9a700 1 -- 172.21.15.31:0/447903053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dbcb6b0 msgr2=0x55565dd7dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dbcb6b0 0x55565dd7dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7aadd9a700 1 -- 172.21.15.31:0/447903053 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55565db151f0 con 0x55565dd6fdf0 2023-07-29T16:21:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd69a40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a98010790 tx=0x7f7a98026ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ca0c3784a9b1a9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7a9d7fa700 1 -- 172.21.15.31:0/447903053 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98017040 con 0x55565dd6fdf0 2023-07-29T16:21:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7a9d7fa700 1 -- 172.21.15.31:0/447903053 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a980122e0 con 0x55565dd6fdf0 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.851+0000 7f7a9d7fa700 1 -- 172.21.15.31:0/447903053 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a98005790 con 0x55565dd6fdf0 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 -- 172.21.15.31:0/447903053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 msgr2=0x55565dd69a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd69a40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a98010790 tx=0x7f7a98026ef0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 -- 172.21.15.31:0/447903053 shutdown_connections 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dd70cf0 0x55565dd69f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dbcb6b0 0x55565dd7dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/447903053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd69a40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 -- 172.21.15.31:0/447903053 >> 172.21.15.31:0/447903053 conn(0x55565dd64fd0 msgr2=0x55565dd73a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 -- 172.21.15.31:0/447903053 shutdown_connections 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 -- 172.21.15.31:0/447903053 wait complete. 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.852+0000 7f7ac4052b80 1 -- start start 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 Processor -- start 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd7d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd7df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd7c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dd7c780 con 0x55565dd70cf0 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dd7c8f0 con 0x55565dbcb6b0 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55565dd7cab0 con 0x55565dd6fdf0 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7ac4052b80 10 client.0 _ll_register_callbacks cb 0x55565dc665c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.853+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd7d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd7d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44576/0 (socket says 172.21.15.31:44576) 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 -- 172.21.15.31:0/4120159387 learned_addr learned my addr 172.21.15.31:0/4120159387 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:22.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aaed9c700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd7c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd7df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 -- 172.21.15.31:0/4120159387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 msgr2=0x55565dd7df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd7df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 -- 172.21.15.31:0/4120159387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 msgr2=0x55565dd7c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd7c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 -- 172.21.15.31:0/4120159387 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55565dad7c00 con 0x55565dbcb6b0 2023-07-29T16:21:22.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7aae59b700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd7d9e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b620 tx=0x7f7aa4009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=704091d97d4e30c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7a91ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.854+0000 7f7a91ffb700 1 -- 172.21.15.31:0/4120159387 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa409a790 con 0x55565dbcb6b0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7a91ffb700 1 -- 172.21.15.31:0/4120159387 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7aa40cab20 con 0x55565dbcb6b0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7aa4015300 con 0x55565dbcb6b0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7a91ffb700 1 -- 172.21.15.31:0/4120159387 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7aa40cace0 con 0x55565dbcb6b0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 10 client.27368 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55565daed250 con 0x55565dbcb6b0 2023-07-29T16:21:22.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7a91ffb700 1 -- 172.21.15.31:0/4120159387 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7aa407d020 con 0x55565dbcb6b0 2023-07-29T16:21:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7a91ffb700 1 -- 172.21.15.31:0/4120159387 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f7aa409a9f0 con 0x55565dbcb6b0 2023-07-29T16:21:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 10 client.27368 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:22.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 10 client.27368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 20 client.27368 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565ddd7590 0x55565ddd99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 4 client.27368 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.855+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55565dcb9ce0 con 0x55565ddd7590 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.856+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565ddd7590 0x55565ddd99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.856+0000 7f7aadd9a700 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565ddd7590 0x55565ddd99a0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=321a4495ecb14c0d server_cookie=25973a36c081611c in_seq=0 out_seq=0 2023-07-29T16:21:22.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.856+0000 7f7a91ffb700 10 client.27368 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:22.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 315846 9.03 MB/sec execute 67 sec latency 21.024 ms 2023-07-29T16:21:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.929+0000 7f7a91ffb700 1 -- 172.21.15.31:0/4120159387 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55565dcb9ce0 con 0x55565ddd7590 2023-07-29T16:21:22.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.929+0000 7f7a91ffb700 10 client.27368 handle_command_reply: tid=0 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.929+0000 7f7ac4052b80 1 client.27368 shutdown 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 msgr2=0x55565dd7d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd7d9e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f7aa400b620 tx=0x7f7aa4009310 comp rx=0 tx=0).stop 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 shutdown_connections 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55565ddd7590 0x55565ddd99a0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55565dd70cf0 0x55565dd7c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55565dbcb6b0 0x55565dd7d9e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 --2- 172.21.15.31:0/4120159387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55565dd6fdf0 0x55565dd7df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 >> 172.21.15.31:0/4120159387 conn(0x55565dd64fd0 msgr2=0x55565dd69a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7a91ffb700 0 client.27368 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 shutdown_connections 2023-07-29T16:21:22.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 1 -- 172.21.15.31:0/4120159387 wait complete. 2023-07-29T16:21:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.930+0000 7f7ac4052b80 20 client.27368 trim_cache size 0 max 16384 2023-07-29T16:21:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:21:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:21:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:21:22.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:21:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7a940a0790 msgr2=0x7f7a940a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7a940a0790 0x7f7a940a2c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x55565dd6f610 tx=0x7f7aa0009920 comp rx=0 tx=0).stop 2023-07-29T16:21:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 msgr2=0x7f7aa8198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:22.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8198370 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4000c00 tx=0x7f7aa40057c0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 shutdown_connections 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f7a940a0790 0x7f7a940a2c40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7aa8101ee0 0x7f7aa8199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7aa8100360 0x7f7aa8198370 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 --2- 172.21.15.31:0/3315652483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aa8101550 0x7f7aa81988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 >> 172.21.15.31:0/3315652483 conn(0x7f7aa80fb960 msgr2=0x7f7aa80fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 shutdown_connections 2023-07-29T16:21:22.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:22.931+0000 7f7ab07ff700 1 -- 172.21.15.31:0/3315652483 wait complete. 2023-07-29T16:21:22.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:22 smithi031 ceph-mon[95504]: pgmap v547: 97 pgs: 97 active+clean; 75 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1.9 MiB/s wr, 226 op/s 2023-07-29T16:21:23.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 316649 8.93 MB/sec execute 68 sec latency 69.942 ms 2023-07-29T16:21:24.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:24 smithi039 ceph-mon[128858]: pgmap v548: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 305 op/s 2023-07-29T16:21:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:24 smithi156 ceph-mon[129245]: pgmap v548: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 305 op/s 2023-07-29T16:21:24.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 318720 8.97 MB/sec execute 69 sec latency 22.645 ms 2023-07-29T16:21:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:24 smithi031 ceph-mon[95504]: pgmap v548: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 305 op/s 2023-07-29T16:21:25.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 319923 8.94 MB/sec execute 70 sec latency 23.395 ms 2023-07-29T16:21:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:26 smithi039 ceph-mon[128858]: pgmap v549: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2023-07-29T16:21:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:26 smithi156 ceph-mon[129245]: pgmap v549: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2023-07-29T16:21:26.868 INFO:tasks.workunit.client.0.smithi031.stdout: 1 322006 8.97 MB/sec execute 71 sec latency 79.819 ms 2023-07-29T16:21:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:26 smithi031 ceph-mon[95504]: pgmap v549: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2023-07-29T16:21:27.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 323266 8.95 MB/sec execute 72 sec latency 12.561 ms 2023-07-29T16:21:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:27 smithi031 ceph-mon[95504]: pgmap v550: 97 pgs: 97 active+clean; 76 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-07-29T16:21:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:27 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:21:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:27 smithi039 ceph-mon[128858]: pgmap v550: 97 pgs: 97 active+clean; 76 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-07-29T16:21:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:27 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:21:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:27 smithi156 ceph-mon[129245]: pgmap v550: 97 pgs: 97 active+clean; 76 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 266 op/s 2023-07-29T16:21:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:27 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:21:28.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 325263 8.94 MB/sec execute 73 sec latency 15.078 ms 2023-07-29T16:21:29.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:28 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:21:29.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:28 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:21:29.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:28 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:21:29.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:28 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:21:29.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:28 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:21:29.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:28 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:21:29.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 326719 8.97 MB/sec execute 74 sec latency 63.894 ms 2023-07-29T16:21:30.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:29 smithi031 ceph-mon[95504]: pgmap v551: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2023-07-29T16:21:30.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:29 smithi039 ceph-mon[128858]: pgmap v551: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2023-07-29T16:21:30.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:29 smithi156 ceph-mon[129245]: pgmap v551: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 285 op/s 2023-07-29T16:21:30.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 328479 8.93 MB/sec execute 75 sec latency 16.818 ms 2023-07-29T16:21:31.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 330161 8.98 MB/sec execute 76 sec latency 40.691 ms 2023-07-29T16:21:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:32 smithi039 ceph-mon[128858]: pgmap v552: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2023-07-29T16:21:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:32 smithi156 ceph-mon[129245]: pgmap v552: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2023-07-29T16:21:32.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 331525 8.94 MB/sec execute 77 sec latency 13.926 ms 2023-07-29T16:21:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:32 smithi031 ceph-mon[95504]: pgmap v552: 97 pgs: 97 active+clean; 77 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2023-07-29T16:21:33.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 333673 9.00 MB/sec execute 78 sec latency 20.052 ms 2023-07-29T16:21:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:34 smithi039 ceph-mon[128858]: pgmap v553: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:21:34.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:34 smithi156 ceph-mon[129245]: pgmap v553: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:21:34.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 334924 8.95 MB/sec execute 79 sec latency 25.086 ms 2023-07-29T16:21:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:34 smithi031 ceph-mon[95504]: pgmap v553: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:21:35.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 337197 9.01 MB/sec execute 80 sec latency 11.595 ms 2023-07-29T16:21:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 -- 172.21.15.31:0/1079973445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988100350 msgr2=0x7fc988100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 --2- 172.21.15.31:0/1079973445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988100350 0x7fc988100760 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009990 tx=0x7fc97c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 -- 172.21.15.31:0/1079973445 shutdown_connections 2023-07-29T16:21:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 --2- 172.21.15.31:0/1079973445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc988101ed0 0x7fc988108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 --2- 172.21.15.31:0/1079973445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988100350 0x7fc988100760 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 --2- 172.21.15.31:0/1079973445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988101540 0x7fc988101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 -- 172.21.15.31:0/1079973445 >> 172.21.15.31:0/1079973445 conn(0x7fc9880fb970 msgr2=0x7fc9880fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 -- 172.21.15.31:0/1079973445 shutdown_connections 2023-07-29T16:21:36.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.118+0000 7fc98efff700 1 -- 172.21.15.31:0/1079973445 wait complete. 2023-07-29T16:21:36.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 Processor -- start 2023-07-29T16:21:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 -- start start 2023-07-29T16:21:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 0x7fc9881983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 0x7fc988198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc988101ed0 0x7fc988199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc988199df0 con 0x7fc988101ed0 2023-07-29T16:21:36.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc988199f60 con 0x7fc988101540 2023-07-29T16:21:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.119+0000 7fc98efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98819a120 con 0x7fc988100350 2023-07-29T16:21:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 0x7fc9881983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 0x7fc988198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 0x7fc9881983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48756/0 (socket says 172.21.15.31:48756) 2023-07-29T16:21:36.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 0x7fc988198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48714/0 (socket says 172.21.15.31:48714) 2023-07-29T16:21:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc98cd9b700 1 -- 172.21.15.31:0/3821757938 learned_addr learned my addr 172.21.15.31:0/3821757938 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc988101ed0 0x7fc988199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 -- 172.21.15.31:0/3821757938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 msgr2=0x7fc9881983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 0x7fc9881983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 -- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc988101ed0 msgr2=0x7fc988199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc988101ed0 0x7fc988199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 -- 172.21.15.31:0/3821757938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc980009c40 con 0x7fc988101540 2023-07-29T16:21:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 0x7fc9881983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.120+0000 7fc987fff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 0x7fc988198910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9f0 tx=0x7fc97400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:36.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.121+0000 7fc985ffb700 1 -- 172.21.15.31:0/3821757938 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400c9c0 con 0x7fc988101540 2023-07-29T16:21:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.121+0000 7fc985ffb700 1 -- 172.21.15.31:0/3821757938 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc974010430 con 0x7fc988101540 2023-07-29T16:21:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.121+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c0095f0 con 0x7fc988101540 2023-07-29T16:21:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.121+0000 7fc985ffb700 1 -- 172.21.15.31:0/3821757938 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974018680 con 0x7fc988101540 2023-07-29T16:21:36.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.121+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98819a5f0 con 0x7fc988101540 2023-07-29T16:21:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.126+0000 7fc985ffb700 1 -- 172.21.15.31:0/3821757938 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc974010860 con 0x7fc988101540 2023-07-29T16:21:36.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.127+0000 7fc9a28feb80 1 Processor -- start 2023-07-29T16:21:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.127+0000 7fc9a28feb80 1 -- start start 2023-07-29T16:21:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.127+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56282801f6b0 0x5628281c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.127+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c3df0 0x5628281c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.127+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.127+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56282811b9b0 con 0x5628281c3df0 2023-07-29T16:21:36.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562828119370 con 0x56282801f6b0 2023-07-29T16:21:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562828114c60 con 0x5628281c4cf0 2023-07-29T16:21:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc985ffb700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9780a0810 0x7fc9780a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc987fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56282801f6b0 0x5628281c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c3df0 0x5628281c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56282801f6b0 0x5628281c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48730/0 (socket says 172.21.15.31:48730) 2023-07-29T16:21:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc987fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48770/0 (socket says 172.21.15.31:48770) 2023-07-29T16:21:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98d59c700 1 -- 172.21.15.31:0/188503371 learned_addr learned my addr 172.21.15.31:0/188503371 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc985ffb700 1 -- 172.21.15.31:0/3821757938 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc974014070 con 0x7fc988101540 2023-07-29T16:21:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98cd9b700 1 -- 172.21.15.31:0/188503371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 msgr2=0x5628281c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9780a0810 0x7fc9780a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98cd9b700 1 -- 172.21.15.31:0/188503371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56282801f6b0 msgr2=0x5628281c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56282801f6b0 0x5628281c38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc98cd9b700 1 -- 172.21.15.31:0/188503371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562827f42d80 con 0x5628281c3df0 2023-07-29T16:21:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.128+0000 7fc987fff700 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:36.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.129+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c3df0 0x5628281c47b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc97c006010 tx=0x7fc97c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72d3c0178cbdc33 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.129+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9780a0810 0x7fc9780a2cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc980009f80 tx=0x7fc980004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.129+0000 7fc972ffd700 1 -- 172.21.15.31:0/188503371 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c01d050 con 0x5628281c3df0 2023-07-29T16:21:36.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.129+0000 7fc972ffd700 1 -- 172.21.15.31:0/188503371 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc97c01e040 con 0x5628281c3df0 2023-07-29T16:21:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.129+0000 7fc972ffd700 1 -- 172.21.15.31:0/188503371 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c005660 con 0x5628281c3df0 2023-07-29T16:21:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 -- 172.21.15.31:0/188503371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c3df0 msgr2=0x5628281c47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c3df0 0x5628281c47b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc97c006010 tx=0x7fc97c005c90 comp rx=0 tx=0).stop 2023-07-29T16:21:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 -- 172.21.15.31:0/188503371 shutdown_connections 2023-07-29T16:21:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56282801f6b0 0x5628281c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/188503371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c3df0 0x5628281c47b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 -- 172.21.15.31:0/188503371 >> 172.21.15.31:0/188503371 conn(0x5628281b8fd0 msgr2=0x5628281be300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 -- 172.21.15.31:0/188503371 shutdown_connections 2023-07-29T16:21:36.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.130+0000 7fc9a28feb80 1 -- 172.21.15.31:0/188503371 wait complete. 2023-07-29T16:21:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 -- start start 2023-07-29T16:21:36.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 Processor -- start 2023-07-29T16:21:36.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c3df0 0x5628281cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c4cf0 0x5628281d43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628280b14b0 con 0x56282801f6b0 2023-07-29T16:21:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628280b1150 con 0x5628281c4cf0 2023-07-29T16:21:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562827f988c0 con 0x5628281c3df0 2023-07-29T16:21:36.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.131+0000 7fc9a28feb80 10 client.0 _ll_register_callbacks cb 0x5628280ba5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c4cf0 0x5628281d43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281cce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46058/0 (socket says 172.21.15.31:46058) 2023-07-29T16:21:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98cd9b700 1 -- 172.21.15.31:0/3867253777 learned_addr learned my addr 172.21.15.31:0/3867253777 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc987fff700 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c3df0 0x5628281cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 -- 172.21.15.31:0/3867253777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c3df0 msgr2=0x5628281cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c3df0 0x5628281cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 -- 172.21.15.31:0/3867253777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 msgr2=0x5628281cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 -- 172.21.15.31:0/3867253777 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98819a5f0 con 0x5628281c4cf0 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c4cf0 0x5628281d43d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc980004010 tx=0x7fc980004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79f2b27ff180de55 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.132+0000 7fc967fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:36.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc967fff700 1 -- 172.21.15.31:0/3867253777 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000e040 con 0x5628281c4cf0 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56282813fcb0 con 0x5628281c4cf0 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc967fff700 1 -- 172.21.15.31:0/3867253777 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc980007a50 con 0x5628281c4cf0 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc967fff700 1 -- 172.21.15.31:0/3867253777 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc980007c10 con 0x5628281c4cf0 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 10 client.27380 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fc9881090c0 con 0x5628281c4cf0 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc967fff700 1 -- 172.21.15.31:0/3867253777 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56282813fcb0 con 0x5628281c4cf0 2023-07-29T16:21:36.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc967fff700 1 -- 172.21.15.31:0/3867253777 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc980032410 con 0x5628281c4cf0 2023-07-29T16:21:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 10 client.27380 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 10 client.27380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 20 client.27380 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5628281e9c50 0x5628281ec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 4 client.27380 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.133+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562828192d10 con 0x5628281e9c50 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.134+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5628281e9c50 0x5628281ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.134+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5628281e9c50 0x5628281ec060 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f3d21d70ffed98e server_cookie=42617fab1484b31e in_seq=0 out_seq=0 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.134+0000 7fc967fff700 10 client.27380 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc967fff700 1 -- 172.21.15.31:0/3867253777 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562828192d10 con 0x5628281e9c50 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc967fff700 10 client.27380 handle_command_reply: tid=0 2023-07-29T16:21:36.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 client.27380 shutdown 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c4cf0 msgr2=0x5628281d43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c4cf0 0x5628281d43d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc980004010 tx=0x7fc980004ad0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 shutdown_connections 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5628281e9c50 0x5628281ec060 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c4cf0 0x5628281d43d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3867253777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c3df0 0x5628281cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc967fff700 0 client.27380 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 >> 172.21.15.31:0/3867253777 conn(0x5628281b8fd0 msgr2=0x5628281be2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 shutdown_connections 2023-07-29T16:21:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.135+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3867253777 wait complete. 2023-07-29T16:21:36.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.136+0000 7fc9a28feb80 20 client.27380 trim_cache size 0 max 16384 2023-07-29T16:21:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 Processor -- start 2023-07-29T16:21:36.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 -- start start 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628281cc9c0 con 0x56282801f6b0 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628281bfb30 con 0x5628281c3df0 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628281e9a40 con 0x5628281c4cf0 2023-07-29T16:21:36.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.151+0000 7fc987fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48748/0 (socket says 172.21.15.31:48748) 2023-07-29T16:21:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 -- 172.21.15.31:0/3583792731 learned_addr learned my addr 172.21.15.31:0/3583792731 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc98d59c700 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 -- 172.21.15.31:0/3583792731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 msgr2=0x5628281c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 -- 172.21.15.31:0/3583792731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 msgr2=0x5628281c7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281c7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 -- 172.21.15.31:0/3583792731 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562827f2bc00 con 0x5628281c3df0 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc987fff700 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4390 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9c0 tx=0x7fc9740af040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5698b6b8ada6dbe7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc9737fe700 1 -- 172.21.15.31:0/3583792731 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9740b1070 con 0x5628281c3df0 2023-07-29T16:21:36.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc9737fe700 1 -- 172.21.15.31:0/3583792731 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc974098ea0 con 0x5628281c3df0 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.152+0000 7fc9737fe700 1 -- 172.21.15.31:0/3583792731 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97409d150 con 0x5628281c3df0 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3583792731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 msgr2=0x5628281d4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4390 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9c0 tx=0x7fc9740af040 comp rx=0 tx=0).stop 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3583792731 shutdown_connections 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5628281c4cf0 0x5628281c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56282801f6b0 0x5628281c7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/3583792731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4390 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3583792731 >> 172.21.15.31:0/3583792731 conn(0x5628281b8fd0 msgr2=0x5628281cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3583792731 shutdown_connections 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 -- 172.21.15.31:0/3583792731 wait complete. 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.153+0000 7fc9a28feb80 1 -- start start 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 Processor -- start 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 0x5628281d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c4cf0 0x5628281d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628281d10c0 con 0x5628281c4cf0 2023-07-29T16:21:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628281d1230 con 0x5628281c3df0 2023-07-29T16:21:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628281d13f0 con 0x56282801f6b0 2023-07-29T16:21:36.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc9a28feb80 10 client.0 _ll_register_callbacks cb 0x5628281f0f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.154+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 0x5628281d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc987fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 0x5628281d3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48792/0 (socket says 172.21.15.31:48792) 2023-07-29T16:21:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc987fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48762/0 (socket says 172.21.15.31:48762) 2023-07-29T16:21:36.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 -- 172.21.15.31:0/50063527 learned_addr learned my addr 172.21.15.31:0/50063527 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98d59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c4cf0 0x5628281d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 -- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 msgr2=0x5628281d4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 -- 172.21.15.31:0/50063527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c4cf0 msgr2=0x5628281d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c4cf0 0x5628281d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 -- 172.21.15.31:0/50063527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c0095f0 con 0x56282801f6b0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc98cd9b700 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 0x5628281d3e20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00b620 tx=0x7fc97c005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f3b0054f83b6252 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc967fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.155+0000 7fc967fff700 1 -- 172.21.15.31:0/50063527 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c01d070 con 0x56282801f6b0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc967fff700 1 -- 172.21.15.31:0/50063527 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc97c023460 con 0x56282801f6b0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc97c01e040 con 0x56282801f6b0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc967fff700 1 -- 172.21.15.31:0/50063527 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc97c023620 con 0x56282801f6b0 2023-07-29T16:21:36.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 10 client.27394 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562827f691f0 con 0x56282801f6b0 2023-07-29T16:21:36.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc967fff700 1 -- 172.21.15.31:0/50063527 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc97c01e040 con 0x56282801f6b0 2023-07-29T16:21:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc967fff700 1 -- 172.21.15.31:0/50063527 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc97c035030 con 0x56282801f6b0 2023-07-29T16:21:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 10 client.27394 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 10 client.27394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 20 client.27394 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56282823caf0 0x56282823ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:36.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 4 client.27394 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:21:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.156+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56282810dce0 con 0x56282823caf0 2023-07-29T16:21:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.157+0000 7fc987fff700 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56282823caf0 0x56282823ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.157+0000 7fc987fff700 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56282823caf0 0x56282823ef00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd05b5614f7ac5e3 server_cookie=dd40d6682278e726 in_seq=0 out_seq=0 2023-07-29T16:21:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.157+0000 7fc967fff700 10 client.27394 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.231+0000 7fc967fff700 1 -- 172.21.15.31:0/50063527 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56282810dce0 con 0x56282823caf0 2023-07-29T16:21:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.231+0000 7fc967fff700 10 client.27394 handle_command_reply: tid=0 2023-07-29T16:21:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.231+0000 7fc9a28feb80 1 client.27394 shutdown 2023-07-29T16:21:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 msgr2=0x5628281d3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 0x5628281d3e20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00b620 tx=0x7fc97c005a00 comp rx=0 tx=0).stop 2023-07-29T16:21:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 shutdown_connections 2023-07-29T16:21:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56282823caf0 0x56282823ef00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5628281c4cf0 0x5628281d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282801f6b0 0x5628281d3e20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc967fff700 0 client.27394 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:36.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 --2- 172.21.15.31:0/50063527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628281c3df0 0x5628281d4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 >> 172.21.15.31:0/50063527 conn(0x5628281b8fd0 msgr2=0x5628281c7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 shutdown_connections 2023-07-29T16:21:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 1 -- 172.21.15.31:0/50063527 wait complete. 2023-07-29T16:21:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.232+0000 7fc9a28feb80 20 client.27394 trim_cache size 0 max 16384 2023-07-29T16:21:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:21:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:21:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:21:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9780a0810 msgr2=0x7fc9780a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9780a0810 0x7fc9780a2cc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc980009f80 tx=0x7fc980004b40 comp rx=0 tx=0).stop 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 msgr2=0x7fc988198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 0x7fc988198910 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc97400e9f0 tx=0x7fc97400ed00 comp rx=0 tx=0).stop 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 shutdown_connections 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc9780a0810 0x7fc9780a2cc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc988101ed0 0x7fc988199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc988100350 0x7fc9881983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 --2- 172.21.15.31:0/3821757938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc988101540 0x7fc988198910 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.233+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 >> 172.21.15.31:0/3821757938 conn(0x7fc9880fb970 msgr2=0x7fc9880fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.234+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 shutdown_connections 2023-07-29T16:21:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:36.234+0000 7fc98efff700 1 -- 172.21.15.31:0/3821757938 wait complete. 2023-07-29T16:21:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:36 smithi039 ceph-mon[128858]: pgmap v554: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2023-07-29T16:21:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:36 smithi156 ceph-mon[129245]: pgmap v554: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2023-07-29T16:21:36.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 338045 8.93 MB/sec execute 81 sec latency 69.895 ms 2023-07-29T16:21:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:36 smithi031 ceph-mon[95504]: pgmap v554: 97 pgs: 97 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.4 MiB/s wr, 252 op/s 2023-07-29T16:21:37.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 340502 9.01 MB/sec execute 82 sec latency 14.947 ms 2023-07-29T16:21:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:37 smithi031 ceph-mon[95504]: pgmap v555: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2023-07-29T16:21:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:37 smithi039 ceph-mon[128858]: pgmap v555: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2023-07-29T16:21:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:37 smithi156 ceph-mon[129245]: pgmap v555: 97 pgs: 97 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2023-07-29T16:21:38.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 341508 8.93 MB/sec execute 83 sec latency 11.696 ms 2023-07-29T16:21:39.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 343714 8.97 MB/sec execute 84 sec latency 49.066 ms 2023-07-29T16:21:40.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:40 smithi039 ceph-mon[128858]: pgmap v556: 97 pgs: 97 active+clean; 79 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2023-07-29T16:21:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:40 smithi156 ceph-mon[129245]: pgmap v556: 97 pgs: 97 active+clean; 79 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2023-07-29T16:21:40.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 344880 8.94 MB/sec execute 85 sec latency 12.227 ms 2023-07-29T16:21:40.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:40 smithi031 ceph-mon[95504]: pgmap v556: 97 pgs: 97 active+clean; 79 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 303 op/s 2023-07-29T16:21:41.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 347033 8.98 MB/sec execute 86 sec latency 72.469 ms 2023-07-29T16:21:42.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:42 smithi039 ceph-mon[128858]: pgmap v557: 97 pgs: 97 active+clean; 79 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 2.1 MiB/s wr, 239 op/s 2023-07-29T16:21:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:42 smithi156 ceph-mon[129245]: pgmap v557: 97 pgs: 97 active+clean; 79 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 2.1 MiB/s wr, 239 op/s 2023-07-29T16:21:42.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 348168 8.95 MB/sec execute 87 sec latency 19.509 ms 2023-07-29T16:21:42.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:42 smithi031 ceph-mon[95504]: pgmap v557: 97 pgs: 97 active+clean; 79 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 2.1 MiB/s wr, 239 op/s 2023-07-29T16:21:43.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:21:43.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:21:43.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 350095 8.94 MB/sec execute 88 sec latency 14.505 ms 2023-07-29T16:21:43.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:21:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: pgmap v558: 97 pgs: 97 active+clean; 80 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2023-07-29T16:21:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:21:44.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:21:44.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:21:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: pgmap v558: 97 pgs: 97 active+clean; 80 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:21:44.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:21:44.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:21:44.869 INFO:tasks.workunit.client.0.smithi031.stdout: 1 351557 8.96 MB/sec execute 89 sec latency 20.782 ms 2023-07-29T16:21:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: pgmap v558: 97 pgs: 97 active+clean; 80 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 298 op/s 2023-07-29T16:21:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:21:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:21:44.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:21:44.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:21:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:45 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:21:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:45 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:21:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:45 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:21:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:45 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:21:45.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:45 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:21:45.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:45 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:21:45.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:45 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:21:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:45 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:21:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:45 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:21:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:45 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:21:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:45 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:21:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:45 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:21:45.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 353518 8.94 MB/sec execute 90 sec latency 20.496 ms 2023-07-29T16:21:45.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:45 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:21:45.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:45 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:21:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:45 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:21:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:45 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:21:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:45 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:21:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:45 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:21:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:46 smithi039 ceph-mon[128858]: pgmap v559: 97 pgs: 97 active+clean; 81 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2023-07-29T16:21:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:46 smithi156 ceph-mon[129245]: pgmap v559: 97 pgs: 97 active+clean; 81 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2023-07-29T16:21:46.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 355022 8.97 MB/sec execute 91 sec latency 53.174 ms 2023-07-29T16:21:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:46 smithi031 ceph-mon[95504]: pgmap v559: 97 pgs: 97 active+clean; 81 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 281 op/s 2023-07-29T16:21:47.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 356791 8.94 MB/sec execute 92 sec latency 11.282 ms 2023-07-29T16:21:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:47 smithi031 ceph-mon[95504]: pgmap v560: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2023-07-29T16:21:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:47 smithi039 ceph-mon[128858]: pgmap v560: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2023-07-29T16:21:48.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:47 smithi156 ceph-mon[129245]: pgmap v560: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 271 op/s 2023-07-29T16:21:48.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:21:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 -- 172.21.15.31:0/2018786961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234100320 msgr2=0x7f8234100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 --2- 172.21.15.31:0/2018786961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234100320 0x7f8234100730 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8228009990 tx=0x7f8228009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 -- 172.21.15.31:0/2018786961 shutdown_connections 2023-07-29T16:21:48.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 --2- 172.21.15.31:0/2018786961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234101ea0 0x7f8234108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 --2- 172.21.15.31:0/2018786961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234100320 0x7f8234100730 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 --2- 172.21.15.31:0/2018786961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234101510 0x7f8234101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 -- 172.21.15.31:0/2018786961 >> 172.21.15.31:0/2018786961 conn(0x7f82340fb920 msgr2=0x7f82340fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:48.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 -- 172.21.15.31:0/2018786961 shutdown_connections 2023-07-29T16:21:48.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.850+0000 7f823abff700 1 -- 172.21.15.31:0/2018786961 wait complete. 2023-07-29T16:21:48.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.851+0000 7f823abff700 1 Processor -- start 2023-07-29T16:21:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.851+0000 7f823abff700 1 -- start start 2023-07-29T16:21:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823abff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 0x7f82341982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823abff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234101510 0x7f82341987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823abff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234101ea0 0x7f823419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823abff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823419fe30 con 0x7f8234101510 2023-07-29T16:21:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823abff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f823419ff70 con 0x7f8234100320 2023-07-29T16:21:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823abff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82341a0100 con 0x7f8234101ea0 2023-07-29T16:21:48.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 0x7f82341982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 0x7f82341982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45478/0 (socket says 172.21.15.31:45478) 2023-07-29T16:21:48.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 -- 172.21.15.31:0/3812765197 learned_addr learned my addr 172.21.15.31:0/3812765197 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:48.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f8233fff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234101510 0x7f82341987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823919c700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234101ea0 0x7f823419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 -- 172.21.15.31:0/3812765197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234101ea0 msgr2=0x7f823419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234101ea0 0x7f823419f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 -- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234101510 msgr2=0x7f82341987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234101510 0x7f82341987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 -- 172.21.15.31:0/3812765197 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82280095f0 con 0x7f8234100320 2023-07-29T16:21:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.852+0000 7f823899b700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 0x7f82341982b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8228005720 tx=0x7f8228009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:48.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.853+0000 7f8231ffb700 1 -- 172.21.15.31:0/3812765197 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822801d070 con 0x7f8234100320 2023-07-29T16:21:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.853+0000 7f8231ffb700 1 -- 172.21.15.31:0/3812765197 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8228021410 con 0x7f8234100320 2023-07-29T16:21:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.853+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82341a0380 con 0x7f8234100320 2023-07-29T16:21:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.853+0000 7f8231ffb700 1 -- 172.21.15.31:0/3812765197 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228017650 con 0x7f8234100320 2023-07-29T16:21:48.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.853+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82341a0850 con 0x7f8234100320 2023-07-29T16:21:48.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.856+0000 7f8231ffb700 1 -- 172.21.15.31:0/3812765197 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8228021580 con 0x7f8234100320 2023-07-29T16:21:48.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.858+0000 7f8231ffb700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82240a0810 0x7f82240a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.858+0000 7f8231ffb700 1 -- 172.21.15.31:0/3812765197 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f82280ccd20 con 0x7f8234100320 2023-07-29T16:21:48.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.858+0000 7f8233fff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82240a0810 0x7f82240a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.858+0000 7f824e563b80 1 Processor -- start 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.858+0000 7f824e563b80 1 -- start start 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f8233fff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82240a0810 0x7f82240a2cc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8220009c60 tx=0x7f8220009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f824e563b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f824e563b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d4bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f824e563b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f824e563b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572ca2a10 con 0x55c572d4b530 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f824e563b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572ca03d0 con 0x55c572d4c2f0 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f824e563b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572c9bcc0 con 0x55c572d4abe0 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32786/0 (socket says 172.21.15.31:32786) 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 -- 172.21.15.31:0/2583231314 learned_addr learned my addr 172.21.15.31:0/2583231314 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f8233fff700 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d4bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823919c700 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 -- 172.21.15.31:0/2583231314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 msgr2=0x55c572d4c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 -- 172.21.15.31:0/2583231314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 msgr2=0x55c572d4bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d4bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f823899b700 1 -- 172.21.15.31:0/2583231314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c572ac9de0 con 0x55c572d4abe0 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.859+0000 7f8233fff700 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d4bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.860+0000 7f823899b700 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4aff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8228096ec0 tx=0x7f822809a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=807f72e7dca5fcc4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.860+0000 7f821effd700 1 -- 172.21.15.31:0/2583231314 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822809a450 con 0x55c572d4abe0 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.860+0000 7f821effd700 1 -- 172.21.15.31:0/2583231314 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f822809e480 con 0x55c572d4abe0 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.860+0000 7f821effd700 1 -- 172.21.15.31:0/2583231314 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82280a0320 con 0x55c572d4abe0 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 -- 172.21.15.31:0/2583231314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 msgr2=0x55c572d4aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4aff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8228096ec0 tx=0x7f822809a060 comp rx=0 tx=0).stop 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 -- 172.21.15.31:0/2583231314 shutdown_connections 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4aff0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 --2- 172.21.15.31:0/2583231314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d4bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 -- 172.21.15.31:0/2583231314 >> 172.21.15.31:0/2583231314 conn(0x55c572d403e0 msgr2=0x55c572d45990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 -- 172.21.15.31:0/2583231314 shutdown_connections 2023-07-29T16:21:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 -- 172.21.15.31:0/2583231314 wait complete. 2023-07-29T16:21:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.861+0000 7f824e563b80 1 -- start start 2023-07-29T16:21:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 Processor -- start 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d54430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d54970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d5b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572c38510 con 0x55c572d4b530 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572c381b0 con 0x55c572d4c2f0 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572b1f920 con 0x55c572d4abe0 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f824e563b80 10 client.0 _ll_register_callbacks cb 0x55c572c41620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f823899b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d54430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f8233fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d54970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f823899b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d54430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32788/0 (socket says 172.21.15.31:32788) 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f823899b700 1 -- 172.21.15.31:0/16343434 learned_addr learned my addr 172.21.15.31:0/16343434 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.862+0000 7f823919c700 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d5b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f823899b700 1 -- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 msgr2=0x55c572d5b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f823899b700 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d5b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f823899b700 1 -- 172.21.15.31:0/16343434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 msgr2=0x55c572d54970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f823899b700 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d54970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f823899b700 1 -- 172.21.15.31:0/16343434 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82341a0850 con 0x55c572d4abe0 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f823899b700 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d54430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f82280a22e0 tx=0x7f8228096890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d12a85593891abf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f8213fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f8213fff700 1 -- 172.21.15.31:0/16343434 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8228099b40 con 0x55c572d4abe0 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f8213fff700 1 -- 172.21.15.31:0/16343434 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f822809a450 con 0x55c572d4abe0 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.863+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c572cc6d10 con 0x55c572d4abe0 2023-07-29T16:21:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f8213fff700 1 -- 172.21.15.31:0/16343434 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82280a2380 con 0x55c572d4abe0 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 10 client.27406 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c572ab2c60 con 0x55c572d4abe0 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f8213fff700 1 -- 172.21.15.31:0/16343434 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f822807d020 con 0x55c572d4abe0 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f8213fff700 1 -- 172.21.15.31:0/16343434 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8228096240 con 0x55c572d4abe0 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 10 client.27406 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 10 client.27406 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 20 client.27406 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572d5fd90 0x55c572d621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 4 client.27406 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c572d19d70 con 0x55c572d5fd90 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.864+0000 7f8233fff700 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572d5fd90 0x55c572d621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.865+0000 7f8213fff700 10 client.27406 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.865+0000 7f8233fff700 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572d5fd90 0x55c572d621a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a056259b6e27e175 server_cookie=db7bf316df3c7b46 in_seq=0 out_seq=0 2023-07-29T16:21:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f8213fff700 1 -- 172.21.15.31:0/16343434 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c572d19d70 con 0x55c572d5fd90 2023-07-29T16:21:48.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f8213fff700 10 client.27406 handle_command_reply: tid=0 2023-07-29T16:21:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 client.27406 shutdown 2023-07-29T16:21:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 msgr2=0x55c572d54430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d54430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f82280a22e0 tx=0x7f8228096890 comp rx=0 tx=0).stop 2023-07-29T16:21:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 shutdown_connections 2023-07-29T16:21:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572d5fd90 0x55c572d621a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d5b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d54430 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f8213fff700 0 client.27406 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 --2- 172.21.15.31:0/16343434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d54970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 >> 172.21.15.31:0/16343434 conn(0x55c572d403e0 msgr2=0x55c572d45060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 shutdown_connections 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.866+0000 7f824e563b80 1 -- 172.21.15.31:0/16343434 wait complete. 2023-07-29T16:21:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.867+0000 7f824e563b80 20 client.27406 trim_cache size 0 max 16384 2023-07-29T16:21:48.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 358609 8.99 MB/sec execute 93 sec latency 22.230 ms 2023-07-29T16:21:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.885+0000 7f824e563b80 1 Processor -- start 2023-07-29T16:21:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.885+0000 7f824e563b80 1 -- start start 2023-07-29T16:21:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f824e563b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f824e563b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d5b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f824e563b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f824e563b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572d3ff40 con 0x55c572d4b530 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f824e563b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572d538e0 con 0x55c572d4c2f0 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f824e563b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572d42490 con 0x55c572d4abe0 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823899b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823919c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823899b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32800/0 (socket says 172.21.15.31:32800) 2023-07-29T16:21:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823899b700 1 -- 172.21.15.31:0/2397608895 learned_addr learned my addr 172.21.15.31:0/2397608895 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f8233fff700 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d5b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823919c700 1 -- 172.21.15.31:0/2397608895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 msgr2=0x55c572d4f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823919c700 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823919c700 1 -- 172.21.15.31:0/2397608895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 msgr2=0x55c572d5b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823919c700 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d5b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823919c700 1 -- 172.21.15.31:0/2397608895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c572af0250 con 0x55c572d4c2f0 2023-07-29T16:21:48.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.886+0000 7f823899b700 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.887+0000 7f823919c700 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4f650 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f822c0118d0 tx=0x7f822c009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=657219b3b623f8b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.887+0000 7f821f7fe700 1 -- 172.21.15.31:0/2397608895 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c00e040 con 0x55c572d4c2f0 2023-07-29T16:21:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.887+0000 7f821f7fe700 1 -- 172.21.15.31:0/2397608895 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f822c00f040 con 0x55c572d4c2f0 2023-07-29T16:21:48.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.887+0000 7f821f7fe700 1 -- 172.21.15.31:0/2397608895 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f822c0046f0 con 0x55c572d4c2f0 2023-07-29T16:21:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 -- 172.21.15.31:0/2397608895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 msgr2=0x55c572d4f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4f650 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f822c0118d0 tx=0x7f822c009920 comp rx=0 tx=0).stop 2023-07-29T16:21:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 -- 172.21.15.31:0/2397608895 shutdown_connections 2023-07-29T16:21:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4c2f0 0x55c572d4f650 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4abe0 0x55c572d4f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 --2- 172.21.15.31:0/2397608895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4b530 0x55c572d5b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 -- 172.21.15.31:0/2397608895 >> 172.21.15.31:0/2397608895 conn(0x55c572d403e0 msgr2=0x55c572d45060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 -- 172.21.15.31:0/2397608895 shutdown_connections 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 -- 172.21.15.31:0/2397608895 wait complete. 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 -- start start 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.888+0000 7f824e563b80 1 Processor -- start 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 0x55c572d58590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4b530 0x55c572d58ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4c2f0 0x55c572db1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572d5baf0 con 0x55c572d4abe0 2023-07-29T16:21:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572d5bc60 con 0x55c572d4b530 2023-07-29T16:21:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c572db1e50 con 0x55c572d4c2f0 2023-07-29T16:21:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f824e563b80 10 client.0 _ll_register_callbacks cb 0x55c572c41620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f823899b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 0x55c572d58590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f8233fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4b530 0x55c572d58ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f823899b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 0x55c572d58590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33712/0 (socket says 172.21.15.31:33712) 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f823899b700 1 -- 172.21.15.31:0/724424888 learned_addr learned my addr 172.21.15.31:0/724424888 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.889+0000 7f823919c700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4c2f0 0x55c572db1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8233fff700 1 -- 172.21.15.31:0/724424888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4c2f0 msgr2=0x55c572db1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8233fff700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4c2f0 0x55c572db1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8233fff700 1 -- 172.21.15.31:0/724424888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 msgr2=0x55c572d58590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8233fff700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 0x55c572d58590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8233fff700 1 -- 172.21.15.31:0/724424888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c572ab2c60 con 0x55c572d4b530 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f823899b700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 0x55c572d58590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8233fff700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4b530 0x55c572d58ad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55c572d62a50 tx=0x7f82200047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=944b899f0544889e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8213fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8213fff700 1 -- 172.21.15.31:0/724424888 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8220029050 con 0x55c572d4b530 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f822809e480 con 0x55c572d4b530 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8213fff700 1 -- 172.21.15.31:0/724424888 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8220003b60 con 0x55c572d4b530 2023-07-29T16:21:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.890+0000 7f8213fff700 1 -- 172.21.15.31:0/724424888 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8220003d20 con 0x55c572d4b530 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 10 client.27404 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f822c0115b0 con 0x55c572d4b530 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f8213fff700 1 -- 172.21.15.31:0/724424888 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f822809e480 con 0x55c572d4b530 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f8213fff700 1 -- 172.21.15.31:0/724424888 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f82200396c0 con 0x55c572d4b530 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 10 client.27404 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 10 client.27404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 20 client.27404 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572db37c0 0x55c572db5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 4 client.27404 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c572c94d40 con 0x55c572db37c0 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.891+0000 7f823899b700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572db37c0 0x55c572db5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f823899b700 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572db37c0 0x55c572db5bd0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0839b485e83beb5 server_cookie=243ccaa59546b1d in_seq=0 out_seq=0 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f8213fff700 10 client.27404 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f8213fff700 1 -- 172.21.15.31:0/724424888 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c572c94d40 con 0x55c572db37c0 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f8213fff700 10 client.27404 handle_command_reply: tid=0 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 client.27404 shutdown 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4b530 msgr2=0x55c572d58ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4b530 0x55c572d58ad0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x55c572d62a50 tx=0x7f82200047c0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 shutdown_connections 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c572db37c0 0x55c572db5bd0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c572d4c2f0 0x55c572db1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f8213fff700 0 client.27404 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c572d4abe0 0x55c572d58590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 --2- 172.21.15.31:0/724424888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c572d4b530 0x55c572d58ad0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 >> 172.21.15.31:0/724424888 conn(0x55c572d403e0 msgr2=0x55c572d4f110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 shutdown_connections 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.892+0000 7f824e563b80 1 -- 172.21.15.31:0/724424888 wait complete. 2023-07-29T16:21:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f824e563b80 20 client.27404 trim_cache size 0 max 16384 2023-07-29T16:21:48.901 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:21:48.901 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:21:48.901 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:21:48.901 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:21:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82240a0810 msgr2=0x7f82240a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82240a0810 0x7f82240a2cc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8220009c60 tx=0x7f8220009260 comp rx=0 tx=0).stop 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 msgr2=0x7f82341982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 0x7f82341982b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8228005720 tx=0x7f8228009580 comp rx=0 tx=0).stop 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 shutdown_connections 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82240a0810 0x7f82240a2cc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8234101ea0 0x7f823419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8234100320 0x7f82341982b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 --2- 172.21.15.31:0/3812765197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234101510 0x7f82341987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 >> 172.21.15.31:0/3812765197 conn(0x7f82340fb920 msgr2=0x7f8234106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 shutdown_connections 2023-07-29T16:21:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:48.893+0000 7f823abff700 1 -- 172.21.15.31:0/3812765197 wait complete. 2023-07-29T16:21:48.910 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:21:48.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:21:49.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/1917259075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 msgr2=0x7f9a88108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/1917259075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a88108780 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000b0a0 tx=0x7f9a8000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/1917259075 shutdown_connections 2023-07-29T16:21:49.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/1917259075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a88108780 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/1917259075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88100340 0x7f9a88100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/1917259075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a88101530 0x7f9a88101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/1917259075 >> 172.21.15.31:0/1917259075 conn(0x7f9a880fb920 msgr2=0x7f9a880fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/1917259075 shutdown_connections 2023-07-29T16:21:49.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.137+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/1917259075 wait complete. 2023-07-29T16:21:49.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.138+0000 7f9a8f5ff700 1 Processor -- start 2023-07-29T16:21:49.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 -- start start 2023-07-29T16:21:49.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a88100340 0x7f9a881980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 0x7f9a88198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a8819f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8819fc50 con 0x7f9a88100340 2023-07-29T16:21:49.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8819fd90 con 0x7f9a88101530 2023-07-29T16:21:49.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8f5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8819ff20 con 0x7f9a88101ec0 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8cb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 0x7f9a88198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8db9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a8819f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8cb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 0x7f9a88198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45514/0 (socket says 172.21.15.31:45514) 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8cb9a700 1 -- 172.21.15.31:0/3404869110 learned_addr learned my addr 172.21.15.31:0/3404869110 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.139+0000 7f9a8d39b700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a88100340 0x7f9a881980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8cb9a700 1 -- 172.21.15.31:0/3404869110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 msgr2=0x7f9a8819f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8cb9a700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a8819f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8cb9a700 1 -- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a88100340 msgr2=0x7f9a881980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8cb9a700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a88100340 0x7f9a881980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8cb9a700 1 -- 172.21.15.31:0/3404869110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84009520 con 0x7f9a88101530 2023-07-29T16:21:49.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8db9c700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a8819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8cb9a700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 0x7f9a88198610 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800e9a0 tx=0x7f9a7800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78009790 con 0x7f9a88101530 2023-07-29T16:21:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a80009d00 con 0x7f9a88101530 2023-07-29T16:21:49.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a78010430 con 0x7f9a88101530 2023-07-29T16:21:49.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78018680 con 0x7f9a88101530 2023-07-29T16:21:49.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.140+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a881a0580 con 0x7f9a88101530 2023-07-29T16:21:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.141+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8810ba70 con 0x7f9a88101530 2023-07-29T16:21:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.144+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9a780105a0 con 0x7f9a88101530 2023-07-29T16:21:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.145+0000 7f9a7e7fc700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9a740a07e0 0x7f9a740a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.145+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9a78014070 con 0x7f9a88101530 2023-07-29T16:21:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.145+0000 7f9a8d39b700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9a740a07e0 0x7f9a740a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.146+0000 7f9a8d39b700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9a740a07e0 0x7f9a740a2c90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9a88101390 tx=0x7f9a84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.147+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9a78095850 con 0x7f9a88101530 2023-07-29T16:21:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 -- 172.21.15.31:0/2738084870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4108150 msgr2=0x7f86c410a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/2738084870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4108150 0x7f86c410a5d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f86bc009ec0 tx=0x7f86bc00b330 comp rx=0 tx=0).stop 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 -- 172.21.15.31:0/2738084870 shutdown_connections 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/2738084870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4108150 0x7f86c410a5d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/2738084870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41052f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/2738084870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c4105830 0x7f86c4107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 -- 172.21.15.31:0/2738084870 >> 172.21.15.31:0/2738084870 conn(0x7f86c40fa840 msgr2=0x7f86c40fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 -- 172.21.15.31:0/2738084870 shutdown_connections 2023-07-29T16:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.401+0000 7f86ca9ff700 1 -- 172.21.15.31:0/2738084870 wait complete. 2023-07-29T16:21:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.402+0000 7f86ca9ff700 1 Processor -- start 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 -- start start 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4105830 0x7f86c4198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c4108150 0x7f86c41997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4199e10 con 0x7f86c4108150 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c4199f80 con 0x7f86c4105830 2023-07-29T16:21:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86ca9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c419a140 con 0x7f86c4068e60 2023-07-29T16:21:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32852/0 (socket says 172.21.15.31:32852) 2023-07-29T16:21:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86c3fff700 1 -- 172.21.15.31:0/1798823461 learned_addr learned my addr 172.21.15.31:0/1798823461 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.403+0000 7f86c37fe700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4105830 0x7f86c4198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c4108150 0x7f86c41997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c3fff700 1 -- 172.21.15.31:0/1798823461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4105830 msgr2=0x7f86c4198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c3fff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4105830 0x7f86c4198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c3fff700 1 -- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c4108150 msgr2=0x7f86c41997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c3fff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c4108150 0x7f86c41997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c3fff700 1 -- 172.21.15.31:0/1798823461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc005e70 con 0x7f86c4068e60 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c3fff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41983f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f86b400e970 tx=0x7f86b400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.404+0000 7f86c17fa700 1 -- 172.21.15.31:0/1798823461 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4009790 con 0x7f86c4068e60 2023-07-29T16:21:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.405+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c419a420 con 0x7f86c4068e60 2023-07-29T16:21:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.405+0000 7f86c17fa700 1 -- 172.21.15.31:0/1798823461 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b400eec0 con 0x7f86c4068e60 2023-07-29T16:21:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.405+0000 7f86c17fa700 1 -- 172.21.15.31:0/1798823461 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4005590 con 0x7f86c4068e60 2023-07-29T16:21:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.405+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c406a950 con 0x7f86c4068e60 2023-07-29T16:21:49.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.408+0000 7f86c17fa700 1 -- 172.21.15.31:0/1798823461 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f86b400cbd0 con 0x7f86c4068e60 2023-07-29T16:21:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 Processor -- start 2023-07-29T16:21:49.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 -- start start 2023-07-29T16:21:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562e08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86c17fa700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0810 0x7f86ac0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562389b0 con 0x5566562e1cf0 2023-07-29T16:21:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556656236370 con 0x55665613c6b0 2023-07-29T16:21:49.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.410+0000 7f86de2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556656231c60 con 0x5566562e0df0 2023-07-29T16:21:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c17fa700 1 -- 172.21.15.31:0/1798823461 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f86b4014070 con 0x7f86c4068e60 2023-07-29T16:21:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562e08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562e08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45532/0 (socket says 172.21.15.31:45532) 2023-07-29T16:21:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 -- 172.21.15.31:0/1336980743 learned_addr learned my addr 172.21.15.31:0/1336980743 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c37fe700 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 -- 172.21.15.31:0/1336980743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 msgr2=0x5566562e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 -- 172.21.15.31:0/1336980743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 msgr2=0x5566562e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0810 0x7f86ac0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c3fff700 1 -- 172.21.15.31:0/1336980743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55665605fd80 con 0x55665613c6b0 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.411+0000 7f86c37fe700 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.412+0000 7f86c3fff700 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562e08b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f86b4099440 tx=0x7f86b40953d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a63b21d24f8fef0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.412+0000 7f86aa7fc700 1 -- 172.21.15.31:0/1336980743 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b409d4e0 con 0x55665613c6b0 2023-07-29T16:21:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.412+0000 7f86aa7fc700 1 -- 172.21.15.31:0/1336980743 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b40b1e90 con 0x55665613c6b0 2023-07-29T16:21:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.412+0000 7f86aa7fc700 1 -- 172.21.15.31:0/1336980743 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4099910 con 0x55665613c6b0 2023-07-29T16:21:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 -- 172.21.15.31:0/1336980743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 msgr2=0x5566562e08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562e08b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f86b4099440 tx=0x7f86b40953d0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0810 0x7f86ac0a2cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0060b0 tx=0x7f86bc006040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 -- 172.21.15.31:0/1336980743 shutdown_connections 2023-07-29T16:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562e08b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 --2- 172.21.15.31:0/1336980743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 -- 172.21.15.31:0/1336980743 >> 172.21.15.31:0/1336980743 conn(0x5566562d5fd0 msgr2=0x5566562db300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 -- 172.21.15.31:0/1336980743 shutdown_connections 2023-07-29T16:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.413+0000 7f86de2beb80 1 -- 172.21.15.31:0/1336980743 wait complete. 2023-07-29T16:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.414+0000 7f86de2beb80 1 -- start start 2023-07-29T16:21:49.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.414+0000 7f86de2beb80 1 Processor -- start 2023-07-29T16:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55665613c6b0 0x5566562e9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562ea410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566562e1cf0 0x5566562f1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566561ce4b0 con 0x55665613c6b0 2023-07-29T16:21:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566561ce150 con 0x5566562e1cf0 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566560b58c0 con 0x5566562e0df0 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86de2beb80 10 client.0 _ll_register_callbacks cb 0x5566561d75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86c8f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566562e1cf0 0x5566562f1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55665613c6b0 0x5566562e9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86c37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562ea410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55665613c6b0 0x5566562e9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33758/0 (socket says 172.21.15.31:33758) 2023-07-29T16:21:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.415+0000 7f86c3fff700 1 -- 172.21.15.31:0/412335630 learned_addr learned my addr 172.21.15.31:0/412335630 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c37fe700 1 -- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566562e1cf0 msgr2=0x5566562f1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c37fe700 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566562e1cf0 0x5566562f1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c37fe700 1 -- 172.21.15.31:0/412335630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55665613c6b0 msgr2=0x5566562e9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c37fe700 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55665613c6b0 0x5566562e9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c37fe700 1 -- 172.21.15.31:0/412335630 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c406a950 con 0x5566562e0df0 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566562e1cf0 0x5566562f1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86c37fe700 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562ea410 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f86b8009f50 tx=0x7f86b8011540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6acf8c8614e8e68a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86a37fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86a37fe700 1 -- 172.21.15.31:0/412335630 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8015460 con 0x5566562e0df0 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86a37fe700 1 -- 172.21.15.31:0/412335630 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f86b80155c0 con 0x5566562e0df0 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.416+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55665625ccb0 con 0x5566562e0df0 2023-07-29T16:21:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86a37fe700 1 -- 172.21.15.31:0/412335630 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86b80157c0 con 0x5566562e0df0 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 10 client.27430 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556656048c00 con 0x5566562e0df0 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86a37fe700 1 -- 172.21.15.31:0/412335630 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55665625ccb0 con 0x5566562e0df0 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86a37fe700 1 -- 172.21.15.31:0/412335630 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f86b8014a40 con 0x5566562e0df0 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 10 client.27430 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 10 client.27430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 20 client.27430 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5566562f5ca0 0x5566562f80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 4 client.27430 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.417+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566562afd10 con 0x5566562f5ca0 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.418+0000 7f86c3fff700 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5566562f5ca0 0x5566562f80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.418+0000 7f86c3fff700 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5566562f5ca0 0x5566562f80b0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=468223d479f567d9 server_cookie=959335fe91313c09 in_seq=0 out_seq=0 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.418+0000 7f86a37fe700 10 client.27430 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86a37fe700 1 -- 172.21.15.31:0/412335630 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5566562afd10 con 0x5566562f5ca0 2023-07-29T16:21:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86a37fe700 10 client.27430 handle_command_reply: tid=0 2023-07-29T16:21:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 client.27430 shutdown 2023-07-29T16:21:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 msgr2=0x5566562ea410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562ea410 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f86b8009f50 tx=0x7f86b8011540 comp rx=0 tx=0).stop 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 shutdown_connections 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5566562f5ca0 0x5566562f80b0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566562e1cf0 0x5566562f1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86a37fe700 0 client.27430 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55665613c6b0 0x5566562e9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 --2- 172.21.15.31:0/412335630 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562ea410 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 >> 172.21.15.31:0/412335630 conn(0x5566562d5fd0 msgr2=0x5566562daa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 shutdown_connections 2023-07-29T16:21:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.419+0000 7f86de2beb80 1 -- 172.21.15.31:0/412335630 wait complete. 2023-07-29T16:21:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.420+0000 7f86de2beb80 20 client.27430 trim_cache size 0 max 16384 2023-07-29T16:21:49.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.434+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a8810bc60 con 0x7f9a88101530 2023-07-29T16:21:49.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.435+0000 7f9a7e7fc700 1 -- 172.21.15.31:0/3404869110 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f9a78098e70 con 0x7f9a88101530 2023-07-29T16:21:49.434 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:21:49.434 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:21:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 Processor -- start 2023-07-29T16:21:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 -- start start 2023-07-29T16:21:49.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9a740a07e0 msgr2=0x7f9a740a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9a740a07e0 0x7f9a740a2c90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f9a88101390 tx=0x7f9a84009260 comp rx=0 tx=0).stop 2023-07-29T16:21:49.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 msgr2=0x7f9a88198610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 0x7f9a88198610 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9a7800e9a0 tx=0x7f9a7800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 shutdown_connections 2023-07-29T16:21:49.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9a740a07e0 0x7f9a740a2c90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a88101ec0 0x7f9a8819f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a88100340 0x7f9a881980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 --2- 172.21.15.31:0/3404869110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a88101530 0x7f9a88198610 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 >> 172.21.15.31:0/3404869110 conn(0x7f9a880fb920 msgr2=0x7f9a880fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562ef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 shutdown_connections 2023-07-29T16:21:49.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f9a8f5ff700 1 -- 172.21.15.31:0/3404869110 wait complete. 2023-07-29T16:21:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562d5a60 con 0x5566562e1cf0 2023-07-29T16:21:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562dc3a0 con 0x55665613c6b0 2023-07-29T16:21:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86de2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562e14f0 con 0x5566562e0df0 2023-07-29T16:21:49.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.439+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562ef400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45556/0 (socket says 172.21.15.31:45556) 2023-07-29T16:21:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 -- 172.21.15.31:0/3871991815 learned_addr learned my addr 172.21.15.31:0/3871991815 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c37fe700 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 -- 172.21.15.31:0/3871991815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 msgr2=0x5566562e4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 -- 172.21.15.31:0/3871991815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 msgr2=0x5566562e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 -- 172.21.15.31:0/3871991815 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566560861f0 con 0x55665613c6b0 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86c3fff700 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562ef400 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f86b40959b0 tx=0x7f86b4095620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a7e70643c226c1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.440+0000 7f86aaffd700 1 -- 172.21.15.31:0/3871991815 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40b7040 con 0x55665613c6b0 2023-07-29T16:21:49.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86aaffd700 1 -- 172.21.15.31:0/3871991815 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b409d4e0 con 0x55665613c6b0 2023-07-29T16:21:49.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86aaffd700 1 -- 172.21.15.31:0/3871991815 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4098c00 con 0x55665613c6b0 2023-07-29T16:21:49.445 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:21:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 -- 172.21.15.31:0/3871991815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 msgr2=0x5566562ef400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562ef400 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f86b40959b0 tx=0x7f86b4095620 comp rx=0 tx=0).stop 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 -- 172.21.15.31:0/3871991815 shutdown_connections 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e1cf0 0x5566562e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562ef400 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 --2- 172.21.15.31:0/3871991815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e0df0 0x5566562e4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.441+0000 7f86de2beb80 1 -- 172.21.15.31:0/3871991815 >> 172.21.15.31:0/3871991815 conn(0x5566562d5fd0 msgr2=0x5566562e80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.442+0000 7f86de2beb80 1 -- 172.21.15.31:0/3871991815 shutdown_connections 2023-07-29T16:21:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.442+0000 7f86de2beb80 1 -- 172.21.15.31:0/3871991815 wait complete. 2023-07-29T16:21:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.442+0000 7f86de2beb80 1 -- start start 2023-07-29T16:21:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.442+0000 7f86de2beb80 1 Processor -- start 2023-07-29T16:21:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562eed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e0df0 0x5566562ef2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 0x5566562ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562edad0 con 0x5566562e0df0 2023-07-29T16:21:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562edc40 con 0x55665613c6b0 2023-07-29T16:21:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566562ede00 con 0x5566562e1cf0 2023-07-29T16:21:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86de2beb80 10 client.0 _ll_register_callbacks cb 0x5566561d75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:49.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562eed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86c8f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 0x5566562ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86c37fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e0df0 0x5566562ef2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86c3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562eed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45560/0 (socket says 172.21.15.31:45560) 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86c8f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 0x5566562ed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32888/0 (socket says 172.21.15.31:32888) 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.443+0000 7f86c3fff700 1 -- 172.21.15.31:0/2224374749 learned_addr learned my addr 172.21.15.31:0/2224374749 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c37fe700 1 -- 172.21.15.31:0/2224374749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 msgr2=0x5566562ed3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c37fe700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 0x5566562ed3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c37fe700 1 -- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 msgr2=0x5566562eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c37fe700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562eed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c37fe700 1 -- 172.21.15.31:0/2224374749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55665605fd80 con 0x5566562e0df0 2023-07-29T16:21:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c3fff700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c8f9c700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 0x5566562ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86c37fe700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e0df0 0x5566562ef2d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f86b8014e50 tx=0x7f86b8011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cba7e4a5c8a58cf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86a37fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86a37fe700 1 -- 172.21.15.31:0/2224374749 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8005e60 con 0x5566562e0df0 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86b40b1e90 con 0x5566562e0df0 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.444+0000 7f86a37fe700 1 -- 172.21.15.31:0/2224374749 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f86b8000e40 con 0x5566562e0df0 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.445+0000 7f86a37fe700 1 -- 172.21.15.31:0/2224374749 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86b80143e0 con 0x5566562e0df0 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.445+0000 7f86de2beb80 10 client.17400 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:49.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.445+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55665605e250 con 0x5566562e0df0 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.445+0000 7f86a37fe700 1 -- 172.21.15.31:0/2224374749 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86b40b1e90 con 0x5566562e0df0 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.445+0000 7f86a37fe700 1 -- 172.21.15.31:0/2224374749 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f86b801daa0 con 0x5566562e0df0 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86de2beb80 10 client.17400 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86de2beb80 10 client.17400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86de2beb80 20 client.17400 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86de2beb80 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556656348b80 0x55665634af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86de2beb80 4 client.17400 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:21:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55665622ace0 con 0x556656348b80 2023-07-29T16:21:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86c3fff700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556656348b80 0x55665634af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86c3fff700 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556656348b80 0x55665634af90 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b721b5fc2fe19987 server_cookie=3e852aa53751c7c5 in_seq=0 out_seq=0 2023-07-29T16:21:49.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.446+0000 7f86a37fe700 10 client.17400 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:49.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86a37fe700 1 -- 172.21.15.31:0/2224374749 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55665622ace0 con 0x556656348b80 2023-07-29T16:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86a37fe700 10 client.17400 handle_command_reply: tid=0 2023-07-29T16:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 client.17400 shutdown 2023-07-29T16:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e0df0 msgr2=0x5566562ef2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e0df0 0x5566562ef2d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f86b8014e50 tx=0x7f86b8011f60 comp rx=0 tx=0).stop 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 shutdown_connections 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556656348b80 0x55665634af90 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5566562e1cf0 0x5566562ed3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55665613c6b0 0x5566562eed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86a37fe700 0 client.17400 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 --2- 172.21.15.31:0/2224374749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5566562e0df0 0x5566562ef2d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 >> 172.21.15.31:0/2224374749 conn(0x5566562d5fd0 msgr2=0x5566562e4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 shutdown_connections 2023-07-29T16:21:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 1 -- 172.21.15.31:0/2224374749 wait complete. 2023-07-29T16:21:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.524+0000 7f86de2beb80 20 client.17400 trim_cache size 0 max 16384 2023-07-29T16:21:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:21:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:21:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:21:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:21:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0810 msgr2=0x7f86ac0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0810 0x7f86ac0a2cc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0060b0 tx=0x7f86bc006040 comp rx=0 tx=0).stop 2023-07-29T16:21:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 msgr2=0x7f86c41983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41983f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f86b400e970 tx=0x7f86b400ec80 comp rx=0 tx=0).stop 2023-07-29T16:21:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 shutdown_connections 2023-07-29T16:21:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86ac0a0810 0x7f86ac0a2cc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86c4108150 0x7f86c41997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4068e60 0x7f86c41983f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 --2- 172.21.15.31:0/1798823461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f86c4105830 0x7f86c4198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 >> 172.21.15.31:0/1798823461 conn(0x7f86c40fa840 msgr2=0x7f86c4106490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.525+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 shutdown_connections 2023-07-29T16:21:49.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:21:49.526+0000 7f86ca9ff700 1 -- 172.21.15.31:0/1798823461 wait complete. 2023-07-29T16:21:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 -- 172.21.15.31:0/3318096911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 msgr2=0x7fc4ec102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/3318096911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec102960 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc009990 tx=0x7fc4dc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 -- 172.21.15.31:0/3318096911 shutdown_connections 2023-07-29T16:21:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/3318096911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec1040d0 0x7fc4ec10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/3318096911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec102960 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/3318096911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ec103740 0x7fc4ec103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 -- 172.21.15.31:0/3318096911 >> 172.21.15.31:0/3318096911 conn(0x7fc4ec0fdb90 msgr2=0x7fc4ec0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 -- 172.21.15.31:0/3318096911 shutdown_connections 2023-07-29T16:21:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.712+0000 7fc4f21ff700 1 -- 172.21.15.31:0/3318096911 wait complete. 2023-07-29T16:21:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.713+0000 7fc4f21ff700 1 Processor -- start 2023-07-29T16:21:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.713+0000 7fc4f21ff700 1 -- start start 2023-07-29T16:21:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4f21ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4f21ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec103740 0x7fc4ec1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4f21ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ec1040d0 0x7fc4ec19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4f21ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec19fe30 con 0x7fc4ec103740 2023-07-29T16:21:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4f21ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec19ff70 con 0x7fc4ec102550 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4f21ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec1a0100 con 0x7fc4ec1040d0 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45580/0 (socket says 172.21.15.31:45580) 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/884011551 learned_addr learned my addr 172.21.15.31:0/884011551 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ec1040d0 0x7fc4ec19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.714+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec103740 0x7fc4ec1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/884011551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ec1040d0 msgr2=0x7fc4ec19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ec1040d0 0x7fc4ec19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec103740 msgr2=0x7fc4ec1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec103740 0x7fc4ec1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/884011551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0095f0 con 0x7fc4ec102550 2023-07-29T16:21:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec103740 0x7fc4ec1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec1982b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc004730 tx=0x7fc4dc004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4e8ff9700 1 -- 172.21.15.31:0/884011551 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc01d070 con 0x7fc4ec102550 2023-07-29T16:21:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4e8ff9700 1 -- 172.21.15.31:0/884011551 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4dc021410 con 0x7fc4ec102550 2023-07-29T16:21:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4e8ff9700 1 -- 172.21.15.31:0/884011551 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc017650 con 0x7fc4ec102550 2023-07-29T16:21:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec1a0380 con 0x7fc4ec102550 2023-07-29T16:21:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.715+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec1a0750 con 0x7fc4ec102550 2023-07-29T16:21:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.719+0000 7fc4e8ff9700 1 -- 172.21.15.31:0/884011551 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc4dc021810 con 0x7fc4ec102550 2023-07-29T16:21:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.720+0000 7fc505ab7b80 1 Processor -- start 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.720+0000 7fc505ab7b80 1 -- start start 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b60ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 0x560168b61d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b622f0 0x560168b62740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168ab8a10 con 0x560168b60be0 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168ab63d0 con 0x560168b61530 2023-07-29T16:21:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168ab1cc0 con 0x560168b622f0 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 0x560168b61d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 0x560168b61d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45594/0 (socket says 172.21.15.31:45594) 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/1336128503 learned_addr learned my addr 172.21.15.31:0/1336128503 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b622f0 0x560168b62740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.721+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b60ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/1336128503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b622f0 msgr2=0x560168b62740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b622f0 0x560168b62740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/1336128503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 msgr2=0x560168b60ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b60ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/1336128503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601688dfde0 con 0x560168b61530 2023-07-29T16:21:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4e8ff9700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4d40a0790 0x7fc4d40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b60ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4e8ff9700 1 -- 172.21.15.31:0/884011551 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4dc0cced0 con 0x7fc4ec102550 2023-07-29T16:21:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 0x560168b61d60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0d0e10 tx=0x7fc4dc0d53c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2292e8b0ab8a3b48 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4d40a0790 0x7fc4d40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4d1ffb700 1 -- 172.21.15.31:0/1336128503 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0969a0 con 0x560168b61530 2023-07-29T16:21:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4d1ffb700 1 -- 172.21.15.31:0/1336128503 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4dc09a430 con 0x560168b61530 2023-07-29T16:21:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.722+0000 7fc4d1ffb700 1 -- 172.21.15.31:0/1336128503 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc099fc0 con 0x560168b61530 2023-07-29T16:21:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 -- 172.21.15.31:0/1336128503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 msgr2=0x560168b61d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 0x560168b61d60 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0d0e10 tx=0x7fc4dc0d53c0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4d40a0790 0x7fc4d40a2c40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009e40 tx=0x7fc4e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 -- 172.21.15.31:0/1336128503 shutdown_connections 2023-07-29T16:21:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b622f0 0x560168b62740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b60ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/1336128503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b61530 0x560168b61d60 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 -- 172.21.15.31:0/1336128503 >> 172.21.15.31:0/1336128503 conn(0x560168b563e0 msgr2=0x560168b5b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 -- 172.21.15.31:0/1336128503 shutdown_connections 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.723+0000 7fc505ab7b80 1 -- 172.21.15.31:0/1336128503 wait complete. 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 -- start start 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 Processor -- start 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b6a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b6a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b71a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168a4e510 con 0x560168b60be0 2023-07-29T16:21:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168a4e1b0 con 0x560168b622f0 2023-07-29T16:21:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168935920 con 0x560168b61530 2023-07-29T16:21:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.724+0000 7fc505ab7b80 10 client.0 _ll_register_callbacks cb 0x560168a57620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b71a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b71a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45600/0 (socket says 172.21.15.31:45600) 2023-07-29T16:21:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 -- 172.21.15.31:0/960417402 learned_addr learned my addr 172.21.15.31:0/960417402 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b6a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b6a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 -- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 msgr2=0x560168b6a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b6a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 -- 172.21.15.31:0/960417402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 msgr2=0x560168b6a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b6a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 -- 172.21.15.31:0/960417402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec1a0750 con 0x560168b622f0 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b71a30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007c00 tx=0x7fc4e4007c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc172a80b9a8228 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.725+0000 7fc4caffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc4caffd700 1 -- 172.21.15.31:0/960417402 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e400e040 con 0x560168b622f0 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560168adcd10 con 0x560168b622f0 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc4caffd700 1 -- 172.21.15.31:0/960417402 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4e4004bb0 con 0x560168b622f0 2023-07-29T16:21:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc4caffd700 1 -- 172.21.15.31:0/960417402 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4e4007e70 con 0x560168b622f0 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 10 client.27440 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601688c8c60 con 0x560168b622f0 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc4caffd700 1 -- 172.21.15.31:0/960417402 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560168adcd10 con 0x560168b622f0 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc4caffd700 1 -- 172.21.15.31:0/960417402 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc4e4016780 con 0x560168b622f0 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 10 client.27440 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 10 client.27440 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:21:49.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 20 client.27440 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168b76280 0x560168b78690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 4 client.27440 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.726+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560168b2fd70 con 0x560168b76280 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.727+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168b76280 0x560168b78690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.727+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168b76280 0x560168b78690 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=97312f76df840407 server_cookie=3aca17bf68929b75 in_seq=0 out_seq=0 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.727+0000 7fc4caffd700 10 client.27440 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc4caffd700 1 -- 172.21.15.31:0/960417402 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560168b2fd70 con 0x560168b76280 2023-07-29T16:21:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc4caffd700 10 client.27440 handle_command_reply: tid=0 2023-07-29T16:21:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 client.27440 shutdown 2023-07-29T16:21:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 msgr2=0x560168b71a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b71a30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4007c00 tx=0x7fc4e4007c30 comp rx=0 tx=0).stop 2023-07-29T16:21:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 shutdown_connections 2023-07-29T16:21:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168b76280 0x560168b78690 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b71a30 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b60be0 0x560168b6a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/960417402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b6a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc4caffd700 0 client.27440 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.728+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 >> 172.21.15.31:0/960417402 conn(0x560168b563e0 msgr2=0x560168b5b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.729+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 shutdown_connections 2023-07-29T16:21:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.729+0000 7fc505ab7b80 1 -- 172.21.15.31:0/960417402 wait complete. 2023-07-29T16:21:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.729+0000 7fc505ab7b80 20 client.27440 trim_cache size 0 max 16384 2023-07-29T16:21:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.743+0000 7fc505ab7b80 1 Processor -- start 2023-07-29T16:21:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.743+0000 7fc505ab7b80 1 -- start start 2023-07-29T16:21:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 0x560168b69210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b622f0 0x560168b78290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168b584a0 con 0x560168b622f0 2023-07-29T16:21:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168b73720 con 0x560168b60be0 2023-07-29T16:21:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168b68430 con 0x560168b61530 2023-07-29T16:21:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 0x560168b69210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 0x560168b69210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45614/0 (socket says 172.21.15.31:45614) 2023-07-29T16:21:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/3558504029 learned_addr learned my addr 172.21.15.31:0/3558504029 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4ebfff700 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b622f0 0x560168b78290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/3558504029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 msgr2=0x560168b719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/3558504029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b622f0 msgr2=0x560168b78290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b622f0 0x560168b78290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.744+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/3558504029 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560168906250 con 0x560168b60be0 2023-07-29T16:21:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.745+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 0x560168b69210 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc000c00 tx=0x7fc4dc01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4d543cc4759b33c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.745+0000 7fc4d27fc700 1 -- 172.21.15.31:0/3558504029 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0d0480 con 0x560168b60be0 2023-07-29T16:21:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.745+0000 7fc4d27fc700 1 -- 172.21.15.31:0/3558504029 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4dc0969a0 con 0x560168b60be0 2023-07-29T16:21:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.745+0000 7fc4d27fc700 1 -- 172.21.15.31:0/3558504029 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc09a660 con 0x560168b60be0 2023-07-29T16:21:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.745+0000 7fc505ab7b80 1 -- 172.21.15.31:0/3558504029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 msgr2=0x560168b69210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.745+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 0x560168b69210 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc000c00 tx=0x7fc4dc01a040 comp rx=0 tx=0).stop 2023-07-29T16:21:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 -- 172.21.15.31:0/3558504029 shutdown_connections 2023-07-29T16:21:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b622f0 0x560168b78290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b60be0 0x560168b69210 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/3558504029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b61530 0x560168b719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 -- 172.21.15.31:0/3558504029 >> 172.21.15.31:0/3558504029 conn(0x560168b563e0 msgr2=0x560168b5b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 -- 172.21.15.31:0/3558504029 shutdown_connections 2023-07-29T16:21:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 -- 172.21.15.31:0/3558504029 wait complete. 2023-07-29T16:21:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.746+0000 7fc505ab7b80 1 -- start start 2023-07-29T16:21:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 Processor -- start 2023-07-29T16:21:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 0x560168b6d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b61530 0x560168b6ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b70df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168b71330 con 0x560168b61530 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168b714a0 con 0x560168b622f0 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560168b71660 con 0x560168b60be0 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc505ab7b80 10 client.0 _ll_register_callbacks cb 0x560168a57620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:49.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 0x560168b6d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc4eb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 0x560168b6d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:32948/0 (socket says 172.21.15.31:32948) 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/2213602592 learned_addr learned my addr 172.21.15.31:0/2213602592 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b70df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.747+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b61530 0x560168b6ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/2213602592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 msgr2=0x560168b70df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b70df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/2213602592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b61530 msgr2=0x560168b6ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b61530 0x560168b6ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eb7fe700 1 -- 172.21.15.31:0/2213602592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601688dfde0 con 0x560168b60be0 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b61530 0x560168b6ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4eb7fe700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 0x560168b6d890 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00b2e0 tx=0x7fc4dc021bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f1d455e3917419 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4caffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4caffd700 1 -- 172.21.15.31:0/2213602592 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc099fc0 con 0x560168b60be0 2023-07-29T16:21:49.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4dc09a430 con 0x560168b60be0 2023-07-29T16:21:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4caffd700 1 -- 172.21.15.31:0/2213602592 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4dc0d0480 con 0x560168b60be0 2023-07-29T16:21:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc4caffd700 1 -- 172.21.15.31:0/2213602592 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4dc09a1a0 con 0x560168b60be0 2023-07-29T16:21:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc505ab7b80 10 client.27454 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.748+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601688de2b0 con 0x560168b60be0 2023-07-29T16:21:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc4caffd700 1 -- 172.21.15.31:0/2213602592 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4dc09a430 con 0x560168b60be0 2023-07-29T16:21:49.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc4caffd700 1 -- 172.21.15.31:0/2213602592 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc4dc09a8c0 con 0x560168b60be0 2023-07-29T16:21:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc505ab7b80 10 client.27454 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc505ab7b80 10 client.27454 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:21:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc505ab7b80 20 client.27454 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168bc8910 0x560168bcad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc505ab7b80 4 client.27454 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:21:49.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560168aaad40 con 0x560168bc8910 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.749+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168bc8910 0x560168bcad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc4eaffd700 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168bc8910 0x560168bcad20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db19898566c7548 server_cookie=344ae64cf180b232 in_seq=0 out_seq=0 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc4caffd700 10 client.27454 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc4caffd700 1 -- 172.21.15.31:0/2213602592 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560168aaad40 con 0x560168bc8910 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc4caffd700 10 client.27454 handle_command_reply: tid=0 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 client.27454 shutdown 2023-07-29T16:21:49.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 msgr2=0x560168b6d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 0x560168b6d890 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00b2e0 tx=0x7fc4dc021bb0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 shutdown_connections 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560168bc8910 0x560168bcad20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560168b622f0 0x560168b70df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560168b60be0 0x560168b6d890 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc4caffd700 0 client.27454 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:21:49.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 --2- 172.21.15.31:0/2213602592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560168b61530 0x560168b6ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 >> 172.21.15.31:0/2213602592 conn(0x560168b563e0 msgr2=0x560168b78290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 shutdown_connections 2023-07-29T16:21:49.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 1 -- 172.21.15.31:0/2213602592 wait complete. 2023-07-29T16:21:49.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.750+0000 7fc505ab7b80 20 client.27454 trim_cache size 0 max 16384 2023-07-29T16:21:49.764 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:21:49.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4d40a0790 msgr2=0x7fc4d40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4d40a0790 0x7fc4d40a2c40 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009e40 tx=0x7fc4e0009260 comp rx=0 tx=0).stop 2023-07-29T16:21:49.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 msgr2=0x7fc4ec1982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec1982b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc004730 tx=0x7fc4dc004760 comp rx=0 tx=0).stop 2023-07-29T16:21:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 shutdown_connections 2023-07-29T16:21:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4d40a0790 0x7fc4d40a2c40 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ec1040d0 0x7fc4ec19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4ec102550 0x7fc4ec1982b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 --2- 172.21.15.31:0/884011551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4ec103740 0x7fc4ec1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:49.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 >> 172.21.15.31:0/884011551 conn(0x7fc4ec0fdb90 msgr2=0x7fc4ec0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:49.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 shutdown_connections 2023-07-29T16:21:49.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:49.751+0000 7fc4f21ff700 1 -- 172.21.15.31:0/884011551 wait complete. 2023-07-29T16:21:49.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:49 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3404869110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:21:49.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:21:49.774 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:49 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3404869110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:21:49.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:49 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3404869110' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:21:49.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 359755 8.95 MB/sec execute 94 sec latency 26.545 ms 2023-07-29T16:21:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 -- 172.21.15.31:0/1922047925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 msgr2=0x7ff1bc0ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/1922047925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc0ff000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff1b800b0a0 tx=0x7ff1b800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 -- 172.21.15.31:0/1922047925 shutdown_connections 2023-07-29T16:21:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/1922047925 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/1922047925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc0ff000 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/1922047925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1bc0ff540 0x7ff1bc0ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 -- 172.21.15.31:0/1922047925 >> 172.21.15.31:0/1922047925 conn(0x7ff1bc0fa840 msgr2=0x7ff1bc0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 -- 172.21.15.31:0/1922047925 shutdown_connections 2023-07-29T16:21:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.016+0000 7ff1c37ff700 1 -- 172.21.15.31:0/1922047925 wait complete. 2023-07-29T16:21:50.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.017+0000 7ff1c37ff700 1 Processor -- start 2023-07-29T16:21:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.017+0000 7ff1c37ff700 1 -- start start 2023-07-29T16:21:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c37ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c37ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1bc0ff540 0x7ff1bc198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c37ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc19fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c37ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc19ffd0 con 0x7ff1bc0ff540 2023-07-29T16:21:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c37ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc1a0110 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c37ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1bc1a02a0 con 0x7ff1bc0febf0 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc19fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c0d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1bc0ff540 0x7ff1bc198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc19fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47126/0 (socket says 172.21.15.31:47126) 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.018+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/761675528 learned_addr learned my addr 172.21.15.31:0/761675528 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/761675528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 msgr2=0x7ff1bc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1bc0ff540 msgr2=0x7ff1bc198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1bc0ff540 0x7ff1bc198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/761675528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b4009520 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c159b700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc19fa00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff1b400ea80 tx=0x7ff1b400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1b27fc700 1 -- 172.21.15.31:0/761675528 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40098a0 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1b27fc700 1 -- 172.21.15.31:0/761675528 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1b400ce10 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1b27fc700 1 -- 172.21.15.31:0/761675528 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4003dc0 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b8009d00 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.019+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc1a08b0 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.024+0000 7ff1b27fc700 1 -- 172.21.15.31:0/761675528 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff1b400c950 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.025+0000 7ff1d7074b80 1 Processor -- start 2023-07-29T16:21:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 -- start start 2023-07-29T16:21:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1b27fc700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1a80a0790 0x7ff1a80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793cf530 0x5563793cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 0x5563793d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1b27fc700 1 -- 172.21.15.31:0/761675528 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff1b4014070 con 0x7ff1bc0ffef0 2023-07-29T16:21:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556379326a10 con 0x5563793cebe0 2023-07-29T16:21:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793243d0 con 0x5563793cf530 2023-07-29T16:21:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637931fcc0 con 0x5563793d02f0 2023-07-29T16:21:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 0x5563793d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1c0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793cf530 0x5563793cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 0x5563793d0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36944/0 (socket says 172.21.15.31:36944) 2023-07-29T16:21:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.026+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/520560220 learned_addr learned my addr 172.21.15.31:0/520560220 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c159b700 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1a80a0790 0x7ff1a80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/520560220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793cf530 msgr2=0x5563793cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793cf530 0x5563793cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/520560220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 msgr2=0x5563793ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/520560220 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55637914dde0 con 0x5563793d02f0 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 0x5563793d0740 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40959a0 tx=0x7ff1b4003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be92a8a7a2b5893c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1a77fe700 1 -- 172.21.15.31:0/520560220 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4004220 con 0x5563793d02f0 2023-07-29T16:21:50.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1a77fe700 1 -- 172.21.15.31:0/520560220 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1b40994a0 con 0x5563793d02f0 2023-07-29T16:21:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.027+0000 7ff1a77fe700 1 -- 172.21.15.31:0/520560220 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4099700 con 0x5563793d02f0 2023-07-29T16:21:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 -- 172.21.15.31:0/520560220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 msgr2=0x5563793d0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1a80a0790 0x7ff1a80a2c40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x5563793cf940 tx=0x7ff1ac0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 0x5563793d0740 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40959a0 tx=0x7ff1b4003680 comp rx=0 tx=0).stop 2023-07-29T16:21:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 -- 172.21.15.31:0/520560220 shutdown_connections 2023-07-29T16:21:50.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793d02f0 0x5563793d0740 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/520560220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793cf530 0x5563793cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 -- 172.21.15.31:0/520560220 >> 172.21.15.31:0/520560220 conn(0x5563793c43e0 msgr2=0x5563793c9990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 -- 172.21.15.31:0/520560220 shutdown_connections 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.028+0000 7ff1d7074b80 1 -- 172.21.15.31:0/520560220 wait complete. 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.029+0000 7ff1d7074b80 1 -- start start 2023-07-29T16:21:50.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.029+0000 7ff1d7074b80 1 Processor -- start 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793d8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793d8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563792bc510 con 0x5563793cebe0 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563792bc1b0 con 0x5563793d02f0 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563791a3920 con 0x5563793cf530 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1d7074b80 10 client.0 _ll_register_callbacks cb 0x5563792c5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:50.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793dfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47138/0 (socket says 172.21.15.31:47138) 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2526606586 learned_addr learned my addr 172.21.15.31:0/2526606586 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793d8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2526606586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 msgr2=0x5563793d8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793d8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793d8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 msgr2=0x5563793d8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793d8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.030+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2526606586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc1a08b0 con 0x5563793d02f0 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.031+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793d8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.031+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793dfd90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009230 tx=0x7ff1b4004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d486e7f577b517f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.031+0000 7ff1a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.031+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2526606586 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40a18f0 con 0x5563793d02f0 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.031+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2526606586 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff1b4097840 con 0x5563793d02f0 2023-07-29T16:21:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.031+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55637934ad10 con 0x5563793d02f0 2023-07-29T16:21:50.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.032+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2526606586 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1b4098990 con 0x5563793d02f0 2023-07-29T16:21:50.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.032+0000 7ff1d7074b80 10 client.27458 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:50.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.032+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556379136c60 con 0x5563793d02f0 2023-07-29T16:21:50.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.032+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2526606586 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55637934ad10 con 0x5563793d02f0 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2526606586 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff1b40a1560 con 0x5563793d02f0 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1d7074b80 10 client.27458 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1d7074b80 10 client.27458 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1d7074b80 20 client.27458 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563793e4110 0x5563793e6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1d7074b80 4 client.27458 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:50.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55637939dd70 con 0x5563793e4110 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.033+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563793e4110 0x5563793e6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.034+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563793e4110 0x5563793e6520 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5432353e07c37af server_cookie=77444facee865197 in_seq=0 out_seq=0 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.034+0000 7ff1a4ff9700 10 client.27458 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.035+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2526606586 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55637939dd70 con 0x5563793e4110 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.035+0000 7ff1a4ff9700 10 client.27458 handle_command_reply: tid=0 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.035+0000 7ff1d7074b80 1 client.27458 shutdown 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.035+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 msgr2=0x5563793dfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.035+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793dfd90 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4009230 tx=0x7ff1b4004590 comp rx=0 tx=0).stop 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 shutdown_connections 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5563793e4110 0x5563793e6520 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793dfd90 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1a4ff9700 0 client.27458 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793d8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2526606586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793d8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 >> 172.21.15.31:0/2526606586 conn(0x5563793c43e0 msgr2=0x5563793c9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 shutdown_connections 2023-07-29T16:21:50.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2526606586 wait complete. 2023-07-29T16:21:50.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.036+0000 7ff1d7074b80 20 client.27458 trim_cache size 0 max 16384 2023-07-29T16:21:50.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.051+0000 7ff1d7074b80 1 Processor -- start 2023-07-29T16:21:50.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.051+0000 7ff1d7074b80 1 -- start start 2023-07-29T16:21:50.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 0x5563793d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cf530 0x5563793dfd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793d66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793d8380 con 0x5563793cf530 2023-07-29T16:21:50.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793c6460 con 0x5563793d02f0 2023-07-29T16:21:50.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793cf970 con 0x5563793cebe0 2023-07-29T16:21:50.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 0x5563793d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793d66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 0x5563793d06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36962/0 (socket says 172.21.15.31:36962) 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793d66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47150/0 (socket says 172.21.15.31:47150) 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c0d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cf530 0x5563793dfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c159b700 1 -- 172.21.15.31:0/2449297657 learned_addr learned my addr 172.21.15.31:0/2449297657 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2449297657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 msgr2=0x5563793d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 0x5563793d06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2449297657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cf530 msgr2=0x5563793dfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cf530 0x5563793dfd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2449297657 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556379174250 con 0x5563793d02f0 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 0x5563793d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.052+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cf530 0x5563793dfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.053+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793d66c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40991a0 tx=0x7ff1b4003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565d66d7c2a1a1a0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.053+0000 7ff1a7fff700 1 -- 172.21.15.31:0/2449297657 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4004420 con 0x5563793d02f0 2023-07-29T16:21:50.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.053+0000 7ff1a7fff700 1 -- 172.21.15.31:0/2449297657 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1b4098760 con 0x5563793d02f0 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.053+0000 7ff1a7fff700 1 -- 172.21.15.31:0/2449297657 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b40c9540 con 0x5563793d02f0 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2449297657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 msgr2=0x5563793d66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793d66c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff1b40991a0 tx=0x7ff1b4003680 comp rx=0 tx=0).stop 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2449297657 shutdown_connections 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793d66c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cebe0 0x5563793d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2449297657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cf530 0x5563793dfd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2449297657 >> 172.21.15.31:0/2449297657 conn(0x5563793c43e0 msgr2=0x5563793d74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2449297657 shutdown_connections 2023-07-29T16:21:50.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2449297657 wait complete. 2023-07-29T16:21:50.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 -- start start 2023-07-29T16:21:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.054+0000 7ff1d7074b80 1 Processor -- start 2023-07-29T16:21:50.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793dbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793dbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793df000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793df540 con 0x5563793cebe0 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793df6b0 con 0x5563793d02f0 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563793df870 con 0x5563793cf530 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1d7074b80 10 client.0 _ll_register_callbacks cb 0x5563792c5620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:50.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793df000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1c0d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793dbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1c1d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793df000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47152/0 (socket says 172.21.15.31:47152) 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1c0d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793dbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36974/0 (socket says 172.21.15.31:36974) 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1c1d9c700 1 -- 172.21.15.31:0/2055928777 learned_addr learned my addr 172.21.15.31:0/2055928777 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.055+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793dbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c0d9a700 1 -- 172.21.15.31:0/2055928777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 msgr2=0x5563793df000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793df000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c0d9a700 1 -- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 msgr2=0x5563793dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793dbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c0d9a700 1 -- 172.21.15.31:0/2055928777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1bc10b290 con 0x5563793cf530 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c1d9c700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793df000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793dbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1c0d9a700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793dbfe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55637934ad30 tx=0x7ff1ac0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fda6e6bdba55c33e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.056+0000 7ff1a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:21:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.057+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2055928777 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac004a30 con 0x5563793cf530 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.057+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1b40994a0 con 0x5563793cf530 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.057+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2055928777 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff1ac004b90 con 0x5563793cf530 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.057+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2055928777 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1ac004d50 con 0x5563793cf530 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.057+0000 7ff1d7074b80 10 client.27472 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.057+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556379136c60 con 0x5563793cf530 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2055928777 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1b40994a0 con 0x5563793cf530 2023-07-29T16:21:50.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2055928777 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff1ac010a50 con 0x5563793cf530 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1d7074b80 10 client.27472 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1d7074b80 10 client.27472 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1d7074b80 20 client.27472 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556379436c60 0x556379439070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1d7074b80 4 client.27472 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556379318d40 con 0x556379436c60 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.058+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556379436c60 0x556379439070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1c159b700 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556379436c60 0x556379439070 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f85dc617477d1a31 server_cookie=9f04563aaf638cb6 in_seq=0 out_seq=0 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1a4ff9700 10 client.27472 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1a4ff9700 1 -- 172.21.15.31:0/2055928777 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556379318d40 con 0x556379436c60 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1a4ff9700 10 client.27472 handle_command_reply: tid=0 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1d7074b80 1 client.27472 shutdown 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 msgr2=0x5563793dbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.059+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793dbfe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55637934ad30 tx=0x7ff1ac0047c0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 shutdown_connections 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556379436c60 0x556379439070 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5563793d02f0 0x5563793df000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5563793cebe0 0x5563793dbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 --2- 172.21.15.31:0/2055928777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5563793cf530 0x5563793dbfe0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1a4ff9700 0 client.27472 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 >> 172.21.15.31:0/2055928777 conn(0x5563793c43e0 msgr2=0x5563793d66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 shutdown_connections 2023-07-29T16:21:50.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 1 -- 172.21.15.31:0/2055928777 wait complete. 2023-07-29T16:21:50.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.060+0000 7ff1d7074b80 20 client.27472 trim_cache size 0 max 16384 2023-07-29T16:21:50.066 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1a80a0790 msgr2=0x7ff1a80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1a80a0790 0x7ff1a80a2c40 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x5563793cf940 tx=0x7ff1ac0073d0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 msgr2=0x7ff1bc19fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc19fa00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff1b400ea80 tx=0x7ff1b400ed90 comp rx=0 tx=0).stop 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 shutdown_connections 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1a80a0790 0x7ff1a80a2c40 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1bc0ffef0 0x7ff1bc19fa00 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1bc0febf0 0x7ff1bc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 --2- 172.21.15.31:0/761675528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff1bc0ff540 0x7ff1bc198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 >> 172.21.15.31:0/761675528 conn(0x7ff1bc0fa840 msgr2=0x7ff1bc0fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 shutdown_connections 2023-07-29T16:21:50.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.061+0000 7ff1c37ff700 1 -- 172.21.15.31:0/761675528 wait complete. 2023-07-29T16:21:50.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:21:50.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 -- 172.21.15.31:0/1250660159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528106050 msgr2=0x7f15281084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 --2- 172.21.15.31:0/1250660159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528106050 0x7f15281084b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1520009f90 tx=0x7f152000b330 comp rx=0 tx=0).stop 2023-07-29T16:21:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 -- 172.21.15.31:0/1250660159 shutdown_connections 2023-07-29T16:21:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 --2- 172.21.15.31:0/1250660159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528106050 0x7f15281084b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 --2- 172.21.15.31:0/1250660159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528100e10 0x7f15281031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 --2- 172.21.15.31:0/1250660159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1528103730 0x7f1528105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 -- 172.21.15.31:0/1250660159 >> 172.21.15.31:0/1250660159 conn(0x7f15280fa8b0 msgr2=0x7f15280fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 -- 172.21.15.31:0/1250660159 shutdown_connections 2023-07-29T16:21:50.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.313+0000 7f152dfff700 1 -- 172.21.15.31:0/1250660159 wait complete. 2023-07-29T16:21:50.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 Processor -- start 2023-07-29T16:21:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 -- start start 2023-07-29T16:21:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 0x7f1528196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1528103730 0x7f1528196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528106050 0x7f152819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152819dcc0 con 0x7f1528103730 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152819de00 con 0x7f1528106050 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.315+0000 7f152dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152819df90 con 0x7f1528100e10 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 0x7f1528196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f1527fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528106050 0x7f152819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 0x7f1528196140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36994/0 (socket says 172.21.15.31:36994) 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 -- 172.21.15.31:0/1807393684 learned_addr learned my addr 172.21.15.31:0/1807393684 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f151ffff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1528103730 0x7f1528196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 -- 172.21.15.31:0/1807393684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528106050 msgr2=0x7f152819d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528106050 0x7f152819d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 -- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1528103730 msgr2=0x7f1528196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1528103730 0x7f1528196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 -- 172.21.15.31:0/1807393684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1520009c40 con 0x7f1528100e10 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f1527fff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528106050 0x7f152819d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.316+0000 7f15277fe700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 0x7f1528196140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f151800e9c0 tx=0x7f151800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.317+0000 7f15257fa700 1 -- 172.21.15.31:0/1807393684 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1518004a30 con 0x7f1528100e10 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.317+0000 7f15257fa700 1 -- 172.21.15.31:0/1807393684 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f151800cc80 con 0x7f1528100e10 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.317+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152819e270 con 0x7f1528100e10 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.317+0000 7f15257fa700 1 -- 172.21.15.31:0/1807393684 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15180054e0 con 0x7f1528100e10 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.317+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152819e780 con 0x7f1528100e10 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.320+0000 7f15257fa700 1 -- 172.21.15.31:0/1807393684 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1518004b90 con 0x7f1528100e10 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.321+0000 7f1541a1ab80 1 Processor -- start 2023-07-29T16:21:50.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 -- start start 2023-07-29T16:21:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea5aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f15257fa700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15140a0790 0x7f15140a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea5bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea5c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdee9b2a10 con 0x55bdeea5abe0 2023-07-29T16:21:50.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdee9b03d0 con 0x55bdeea5c2f0 2023-07-29T16:21:50.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdee9abcc0 con 0x55bdeea5b530 2023-07-29T16:21:50.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.322+0000 7f15257fa700 1 -- 172.21.15.31:0/1807393684 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1518014070 con 0x7f1528100e10 2023-07-29T16:21:50.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f1527fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea5aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f15277fe700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15140a0790 0x7f15140a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f1527fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea5aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43598/0 (socket says 172.21.15.31:43598) 2023-07-29T16:21:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f1527fff700 1 -- 172.21.15.31:0/186688455 learned_addr learned my addr 172.21.15.31:0/186688455 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f151ffff700 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea5bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f1527fff700 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea5c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f151ffff700 1 -- 172.21.15.31:0/186688455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 msgr2=0x55bdeea5c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f151ffff700 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea5c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f151ffff700 1 -- 172.21.15.31:0/186688455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 msgr2=0x55bdeea5aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f151ffff700 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea5aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f151ffff700 1 -- 172.21.15.31:0/186688455 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdee7d9de0 con 0x55bdeea5b530 2023-07-29T16:21:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f1527fff700 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea5c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:50.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.323+0000 7f15277fe700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15140a0790 0x7f15140a2c40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1518095e00 tx=0x7f1518095040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.324+0000 7f151ffff700 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea5bd60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1510009cb0 tx=0x7f1510011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc7b6738e18dc34 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.324+0000 7f151dffb700 1 -- 172.21.15.31:0/186688455 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510011d20 con 0x55bdeea5b530 2023-07-29T16:21:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.324+0000 7f151dffb700 1 -- 172.21.15.31:0/186688455 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1510011e80 con 0x55bdeea5b530 2023-07-29T16:21:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.324+0000 7f151dffb700 1 -- 172.21.15.31:0/186688455 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1510005280 con 0x55bdeea5b530 2023-07-29T16:21:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 -- 172.21.15.31:0/186688455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 msgr2=0x55bdeea5bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea5bd60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1510009cb0 tx=0x7f1510011440 comp rx=0 tx=0).stop 2023-07-29T16:21:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 -- 172.21.15.31:0/186688455 shutdown_connections 2023-07-29T16:21:50.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea5c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea5aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/186688455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea5bd60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 -- 172.21.15.31:0/186688455 >> 172.21.15.31:0/186688455 conn(0x55bdeea503e0 msgr2=0x55bdeea55990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 -- 172.21.15.31:0/186688455 shutdown_connections 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 -- 172.21.15.31:0/186688455 wait complete. 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.325+0000 7f1541a1ab80 1 -- start start 2023-07-29T16:21:50.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 Processor -- start 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea64850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea64d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea6bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdee948510 con 0x55bdeea5abe0 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdee9481b0 con 0x55bdeea5c2f0 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdee82f920 con 0x55bdeea5b530 2023-07-29T16:21:50.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.326+0000 7f1541a1ab80 10 client.0 _ll_register_callbacks cb 0x55bdee951620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea64850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea64850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43608/0 (socket says 172.21.15.31:43608) 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 -- 172.21.15.31:0/2984830607 learned_addr learned my addr 172.21.15.31:0/2984830607 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f151ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea64d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea6bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 -- 172.21.15.31:0/2984830607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 msgr2=0x55bdeea64d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea64d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 -- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 msgr2=0x55bdeea6bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea6bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 -- 172.21.15.31:0/2984830607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15100098e0 con 0x55bdeea5abe0 2023-07-29T16:21:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f151ffff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea64d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1527fff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea64850 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1520009a00 tx=0x7f1520008e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec716dcc642214b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1506ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.327+0000 7f1506ffd700 1 -- 172.21.15.31:0/2984830607 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520027070 con 0x55bdeea5abe0 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdeea29d70 con 0x55bdeea5abe0 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1506ffd700 1 -- 172.21.15.31:0/2984830607 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f152001b460 con 0x55bdeea5abe0 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1506ffd700 1 -- 172.21.15.31:0/2984830607 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f152000e040 con 0x55bdeea5abe0 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 10 client.17445 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:50.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bdee7c2c60 con 0x55bdeea5abe0 2023-07-29T16:21:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1506ffd700 1 -- 172.21.15.31:0/2984830607 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bdeea29d70 con 0x55bdeea5abe0 2023-07-29T16:21:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1506ffd700 1 -- 172.21.15.31:0/2984830607 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f152002a030 con 0x55bdeea5abe0 2023-07-29T16:21:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 10 client.17445 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 10 client.17445 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 20 client.17445 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:50.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeea705a0 0x55bdeea729b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 4 client.17445 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.328+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdee9a4d40 con 0x55bdeea705a0 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.329+0000 7f151ffff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeea705a0 0x55bdeea729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.329+0000 7f151ffff700 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeea705a0 0x55bdeea729b0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35ab2c459b2316f0 server_cookie=93cc1beb2b4bb613 in_seq=0 out_seq=0 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.329+0000 7f1506ffd700 10 client.17445 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1506ffd700 1 -- 172.21.15.31:0/2984830607 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bdee9a4d40 con 0x55bdeea705a0 2023-07-29T16:21:50.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1506ffd700 10 client.17445 handle_command_reply: tid=0 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 client.17445 shutdown 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 msgr2=0x55bdeea64850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea64850 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1520009a00 tx=0x7f1520008e90 comp rx=0 tx=0).stop 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 shutdown_connections 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeea705a0 0x55bdeea729b0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea6bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5abe0 0x55bdeea64850 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/2984830607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5b530 0x55bdeea64d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1506ffd700 0 client.17445 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 >> 172.21.15.31:0/2984830607 conn(0x55bdeea503e0 msgr2=0x55bdeea55060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.330+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 shutdown_connections 2023-07-29T16:21:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.331+0000 7f1541a1ab80 1 -- 172.21.15.31:0/2984830607 wait complete. 2023-07-29T16:21:50.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.331+0000 7f1541a1ab80 20 client.17445 trim_cache size 0 max 16384 2023-07-29T16:21:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 Processor -- start 2023-07-29T16:21:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 -- start start 2023-07-29T16:21:50.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeea634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeea55060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea62720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea4fed0 con 0x55bdeea5b530 2023-07-29T16:21:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea63d30 con 0x55bdeea5c2f0 2023-07-29T16:21:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeea6c180 con 0x55bdeea5abe0 2023-07-29T16:21:50.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f151ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeea55060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f1527fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea62720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f151ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeea55060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43618/0 (socket says 172.21.15.31:43618) 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.346+0000 7f151ffff700 1 -- 172.21.15.31:0/500397275 learned_addr learned my addr 172.21.15.31:0/500397275 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeea634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 -- 172.21.15.31:0/500397275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 msgr2=0x55bdeea634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeea634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 -- 172.21.15.31:0/500397275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 msgr2=0x55bdeea55060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeea55060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 -- 172.21.15.31:0/500397275 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdee800250 con 0x55bdeea5c2f0 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f1527fff700 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea62720 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f152000d010 tx=0x7f15200096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69432033c6e35d9a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f151e7fc700 1 -- 172.21.15.31:0/500397275 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000e070 con 0x55bdeea5c2f0 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f151e7fc700 1 -- 172.21.15.31:0/500397275 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f152000f040 con 0x55bdeea5c2f0 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.347+0000 7f151e7fc700 1 -- 172.21.15.31:0/500397275 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1520027070 con 0x55bdeea5c2f0 2023-07-29T16:21:50.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 -- 172.21.15.31:0/500397275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 msgr2=0x55bdeea62720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea62720 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f152000d010 tx=0x7f15200096c0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 -- 172.21.15.31:0/500397275 shutdown_connections 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeea62720 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeea634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/500397275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeea55060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 -- 172.21.15.31:0/500397275 >> 172.21.15.31:0/500397275 conn(0x55bdeea503e0 msgr2=0x55bdeea56690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 -- 172.21.15.31:0/500397275 shutdown_connections 2023-07-29T16:21:50.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.348+0000 7f1541a1ab80 1 -- 172.21.15.31:0/500397275 wait complete. 2023-07-29T16:21:50.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 -- start start 2023-07-29T16:21:50.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 Processor -- start 2023-07-29T16:21:50.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeeabe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeeabe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeeac52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeeac5820 con 0x55bdeea5b530 2023-07-29T16:21:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeeac5990 con 0x55bdeea5c2f0 2023-07-29T16:21:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeeac5b50 con 0x55bdeea5abe0 2023-07-29T16:21:50.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.349+0000 7f1541a1ab80 10 client.0 _ll_register_callbacks cb 0x55bdee951620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeeac52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f151ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeeabe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeeac52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47204/0 (socket says 172.21.15.31:47204) 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 -- 172.21.15.31:0/813944529 learned_addr learned my addr 172.21.15.31:0/813944529 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeeabe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 -- 172.21.15.31:0/813944529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 msgr2=0x55bdeeabe350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeeabe350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 -- 172.21.15.31:0/813944529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 msgr2=0x55bdeeabe890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeeabe890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 -- 172.21.15.31:0/813944529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdee7c2c60 con 0x55bdeea5c2f0 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f151ffff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeeabe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1527fff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeeac52e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f152000b840 tx=0x7f1520009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e3e6e12b05feb2b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.350+0000 7f1506ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:50.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1506ffd700 1 -- 172.21.15.31:0/813944529 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000e070 con 0x55bdeea5c2f0 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1506ffd700 1 -- 172.21.15.31:0/813944529 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f15200123f0 con 0x55bdeea5c2f0 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdee9d6d10 con 0x55bdeea5c2f0 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1506ffd700 1 -- 172.21.15.31:0/813944529 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1520027040 con 0x55bdeea5c2f0 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1541a1ab80 10 client.27482 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bdee7d82b0 con 0x55bdeea5c2f0 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1506ffd700 1 -- 172.21.15.31:0/813944529 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bdee9d6d10 con 0x55bdeea5c2f0 2023-07-29T16:21:50.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1506ffd700 1 -- 172.21.15.31:0/813944529 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f1520037030 con 0x55bdeea5c2f0 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1541a1ab80 10 client.27482 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1541a1ab80 10 client.27482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.351+0000 7f1541a1ab80 20 client.27482 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeeac7070 0x55bdeeac9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1541a1ab80 4 client.27482 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bdee9a4a60 con 0x55bdeeac7070 2023-07-29T16:21:50.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f151ffff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeeac7070 0x55bdeeac9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f151ffff700 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeeac7070 0x55bdeeac9450 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d34e00737ecaa51f server_cookie=b70203756e5ce960 in_seq=0 out_seq=0 2023-07-29T16:21:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1506ffd700 10 client.27482 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1506ffd700 1 -- 172.21.15.31:0/813944529 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bdee9a4a60 con 0x55bdeeac7070 2023-07-29T16:21:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1506ffd700 10 client.27482 handle_command_reply: tid=0 2023-07-29T16:21:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.352+0000 7f1541a1ab80 1 client.27482 shutdown 2023-07-29T16:21:50.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 msgr2=0x55bdeeac52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeeac52e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f152000b840 tx=0x7f1520009f10 comp rx=0 tx=0).stop 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 shutdown_connections 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bdeeac7070 0x55bdeeac9450 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bdeea5c2f0 0x55bdeeac52e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bdeea5abe0 0x55bdeeabe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1506ffd700 0 client.27482 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:50.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 --2- 172.21.15.31:0/813944529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bdeea5b530 0x55bdeeabe890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 >> 172.21.15.31:0/813944529 conn(0x55bdeea503e0 msgr2=0x55bdeea55060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 shutdown_connections 2023-07-29T16:21:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 1 -- 172.21.15.31:0/813944529 wait complete. 2023-07-29T16:21:50.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.353+0000 7f1541a1ab80 20 client.27482 trim_cache size 0 max 16384 2023-07-29T16:21:50.363 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:21:50.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15140a0790 msgr2=0x7f15140a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15140a0790 0x7f15140a2c40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1518095e00 tx=0x7f1518095040 comp rx=0 tx=0).stop 2023-07-29T16:21:50.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 msgr2=0x7f1528196140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:50.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 0x7f1528196140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f151800e9c0 tx=0x7f151800ecd0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 shutdown_connections 2023-07-29T16:21:50.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f15140a0790 0x7f15140a2c40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1528106050 0x7f152819d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1528100e10 0x7f1528196140 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 --2- 172.21.15.31:0/1807393684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1528103730 0x7f1528196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:50.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 >> 172.21.15.31:0/1807393684 conn(0x7f15280fa8b0 msgr2=0x7f1528101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:50.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 shutdown_connections 2023-07-29T16:21:50.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:50.354+0000 7f152dfff700 1 -- 172.21.15.31:0/1807393684 wait complete. 2023-07-29T16:21:50.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:50 smithi031 ceph-mon[95504]: pgmap v561: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2023-07-29T16:21:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:50 smithi039 ceph-mon[128858]: pgmap v561: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2023-07-29T16:21:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:50 smithi156 ceph-mon[129245]: pgmap v561: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 287 op/s 2023-07-29T16:21:50.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 361943 8.99 MB/sec execute 95 sec latency 11.007 ms 2023-07-29T16:21:51.372 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:21:51.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:21:51.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 -- 172.21.15.31:0/1178811374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 msgr2=0x7f11141007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 --2- 172.21.15.31:0/1178811374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 0x7f11141007a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f1108009990 tx=0x7f1108009ca0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 -- 172.21.15.31:0/1178811374 shutdown_connections 2023-07-29T16:21:51.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 --2- 172.21.15.31:0/1178811374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101f10 0x7f11141087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 --2- 172.21.15.31:0/1178811374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 0x7f11141007a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 --2- 172.21.15.31:0/1178811374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1114101580 0x7f11141019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 -- 172.21.15.31:0/1178811374 >> 172.21.15.31:0/1178811374 conn(0x7f11140fb9b0 msgr2=0x7f11140fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 -- 172.21.15.31:0/1178811374 shutdown_connections 2023-07-29T16:21:51.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.619+0000 7f111adff700 1 -- 172.21.15.31:0/1178811374 wait complete. 2023-07-29T16:21:51.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 Processor -- start 2023-07-29T16:21:51.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 -- start start 2023-07-29T16:21:51.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 0x7f11141960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 0x7f1114196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1114101f10 0x7f111419d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111419dc50 con 0x7f1114101f10 2023-07-29T16:21:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111419dd90 con 0x7f1114100390 2023-07-29T16:21:51.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f111adff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111419df20 con 0x7f1114101580 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.620+0000 7f1113fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 0x7f1114196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 0x7f1114196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37068/0 (socket says 172.21.15.31:37068) 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 -- 172.21.15.31:0/3178201295 learned_addr learned my addr 172.21.15.31:0/3178201295 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f111939c700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1114101f10 0x7f111419d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1118b9b700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 0x7f11141960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 -- 172.21.15.31:0/3178201295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 msgr2=0x7f11141960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 0x7f11141960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 -- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1114101f10 msgr2=0x7f111419d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1114101f10 0x7f111419d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 -- 172.21.15.31:0/3178201295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080095f0 con 0x7f1114101580 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1113fff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 0x7f1114196610 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f110000e9a0 tx=0x7f110000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.621+0000 7f1111ffb700 1 -- 172.21.15.31:0/3178201295 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100009790 con 0x7f1114101580 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.622+0000 7f1111ffb700 1 -- 172.21.15.31:0/3178201295 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1100010430 con 0x7f1114101580 2023-07-29T16:21:51.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.622+0000 7f1111ffb700 1 -- 172.21.15.31:0/3178201295 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100018680 con 0x7f1114101580 2023-07-29T16:21:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.622+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111419e200 con 0x7f1114101580 2023-07-29T16:21:51.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.622+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111419e6c0 con 0x7f1114101580 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.626+0000 7f1111ffb700 1 -- 172.21.15.31:0/3178201295 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f11000105a0 con 0x7f1114101580 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.627+0000 7f112e700b80 1 Processor -- start 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 -- start start 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f1111ffb700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f11040a0810 0x7f11040a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9b8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f1111ffb700 1 -- 172.21.15.31:0/3178201295 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1100014070 con 0x7f1114101580 2023-07-29T16:21:51.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b9320 0x55ac8f9b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f90fa90 con 0x55ac8f9b9320 2023-07-29T16:21:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f90d450 con 0x55ac8f9b85b0 2023-07-29T16:21:51.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.628+0000 7f112e700b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f908d40 con 0x55ac8f9b7c60 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f111939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b9320 0x55ac8f9b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f111939c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b9320 0x55ac8f9b97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43652/0 (socket says 172.21.15.31:43652) 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f111939c700 1 -- 172.21.15.31:0/940941599 learned_addr learned my addr 172.21.15.31:0/940941599 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f111939c700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f11040a0810 0x7f11040a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1118b9b700 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1113fff700 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1118b9b700 1 -- 172.21.15.31:0/940941599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 msgr2=0x55ac8f9b8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1118b9b700 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1118b9b700 1 -- 172.21.15.31:0/940941599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b9320 msgr2=0x55ac8f9b97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1118b9b700 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b9320 0x55ac8f9b97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1118b9b700 1 -- 172.21.15.31:0/940941599 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac8f736e60 con 0x55ac8f9b7c60 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.629+0000 7f1113fff700 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.630+0000 7f1118b9b700 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9b8070 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f11080098e0 tx=0x7f1108005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb86b5c583f85f51 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.630+0000 7f111939c700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f11040a0810 0x7f11040a2cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f110c009fd0 tx=0x7f110c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.630+0000 7f10feffd700 1 -- 172.21.15.31:0/940941599 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110801d070 con 0x55ac8f9b7c60 2023-07-29T16:21:51.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.630+0000 7f10feffd700 1 -- 172.21.15.31:0/940941599 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f110801e040 con 0x55ac8f9b7c60 2023-07-29T16:21:51.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.630+0000 7f10feffd700 1 -- 172.21.15.31:0/940941599 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108004860 con 0x55ac8f9b7c60 2023-07-29T16:21:51.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 -- 172.21.15.31:0/940941599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 msgr2=0x55ac8f9b8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9b8070 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f11080098e0 tx=0x7f1108005ea0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 -- 172.21.15.31:0/940941599 shutdown_connections 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b9320 0x55ac8f9b97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9b8070 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 --2- 172.21.15.31:0/940941599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 -- 172.21.15.31:0/940941599 >> 172.21.15.31:0/940941599 conn(0x55ac8f9ad460 msgr2=0x55ac8f9b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 -- 172.21.15.31:0/940941599 shutdown_connections 2023-07-29T16:21:51.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.631+0000 7f112e700b80 1 -- 172.21.15.31:0/940941599 wait complete. 2023-07-29T16:21:51.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.632+0000 7f112e700b80 1 -- start start 2023-07-29T16:21:51.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.632+0000 7f112e700b80 1 Processor -- start 2023-07-29T16:21:51.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.632+0000 7f112e700b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.632+0000 7f112e700b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9ca5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f112e700b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b9320 0x55ac8f9d1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f112e700b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f8a5590 con 0x55ac8f9b7c60 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f112e700b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f8a5230 con 0x55ac8f9b85b0 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f112e700b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f78c9a0 con 0x55ac8f9b9320 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f112e700b80 10 client.0 _ll_register_callbacks cb 0x55ac8f8ae6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f1118b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f1118b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43666/0 (socket says 172.21.15.31:43666) 2023-07-29T16:21:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f1118b9b700 1 -- 172.21.15.31:0/1123244628 learned_addr learned my addr 172.21.15.31:0/1123244628 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.633+0000 7f111939c700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b9320 0x55ac8f9d1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.634+0000 7f1113fff700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9ca5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.634+0000 7f1118b9b700 1 -- 172.21.15.31:0/1123244628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b9320 msgr2=0x55ac8f9d1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.634+0000 7f1118b9b700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b9320 0x55ac8f9d1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.634+0000 7f1118b9b700 1 -- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 msgr2=0x55ac8f9ca5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.634+0000 7f1118b9b700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9ca5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.634+0000 7f1118b9b700 1 -- 172.21.15.31:0/1123244628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1114109100 con 0x55ac8f9b7c60 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.635+0000 7f1118b9b700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9ca080 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1108009ce0 tx=0x7f110800bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a260f9398f932adf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.635+0000 7f10f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.635+0000 7f10f3fff700 1 -- 172.21.15.31:0/1123244628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110801d030 con 0x55ac8f9b7c60 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.635+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ac8f933d90 con 0x55ac8f9b7c60 2023-07-29T16:21:51.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.635+0000 7f10f3fff700 1 -- 172.21.15.31:0/1123244628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f11080049f0 con 0x55ac8f9b7c60 2023-07-29T16:21:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.636+0000 7f10f3fff700 1 -- 172.21.15.31:0/1123244628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f110800bd70 con 0x55ac8f9b7c60 2023-07-29T16:21:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.636+0000 7f112e700b80 10 client.17463 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:51.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.636+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ac8f71fce0 con 0x55ac8f9b7c60 2023-07-29T16:21:51.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.637+0000 7f10f3fff700 1 -- 172.21.15.31:0/1123244628 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f110803e080 con 0x55ac8f9b7c60 2023-07-29T16:21:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.637+0000 7f10f3fff700 1 -- 172.21.15.31:0/1123244628 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f110800fcc0 con 0x55ac8f9b7c60 2023-07-29T16:21:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.638+0000 7f112e700b80 10 client.17463 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:51.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.638+0000 7f112e700b80 10 client.17463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.638+0000 7f112e700b80 20 client.17463 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.638+0000 7f112e700b80 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9d58a0 0x55ac8f9d7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.638+0000 7f112e700b80 4 client.17463 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:51.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.638+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac8f986df0 con 0x55ac8f9d58a0 2023-07-29T16:21:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.639+0000 7f1113fff700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9d58a0 0x55ac8f9d7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.640+0000 7f1113fff700 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9d58a0 0x55ac8f9d7c80 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4071cf57de00fe7e server_cookie=512d329917f64517 in_seq=0 out_seq=0 2023-07-29T16:21:51.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.640+0000 7f10f3fff700 10 client.17463 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.641+0000 7f10f3fff700 1 -- 172.21.15.31:0/1123244628 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ac8f986df0 con 0x55ac8f9d58a0 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.641+0000 7f10f3fff700 10 client.17463 handle_command_reply: tid=0 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.641+0000 7f112e700b80 1 client.17463 shutdown 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 msgr2=0x55ac8f9ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9ca080 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1108009ce0 tx=0x7f110800bad0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 shutdown_connections 2023-07-29T16:21:51.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9d58a0 0x55ac8f9d7c80 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b9320 0x55ac8f9d1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9ca080 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 --2- 172.21.15.31:0/1123244628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9ca5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 >> 172.21.15.31:0/1123244628 conn(0x55ac8f9ad460 msgr2=0x55ac8f9b20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f10f3fff700 0 client.17463 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 shutdown_connections 2023-07-29T16:21:51.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.642+0000 7f112e700b80 1 -- 172.21.15.31:0/1123244628 wait complete. 2023-07-29T16:21:51.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.643+0000 7f112e700b80 20 client.17463 trim_cache size 0 max 16384 2023-07-29T16:21:51.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.672+0000 7f112e700b80 1 Processor -- start 2023-07-29T16:21:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.672+0000 7f112e700b80 1 -- start start 2023-07-29T16:21:51.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.672+0000 7f112e700b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.672+0000 7f112e700b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9d15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.672+0000 7f112e700b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f112e700b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f9b43a0 con 0x55ac8f9b7c60 2023-07-29T16:21:51.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f112e700b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f9ba270 con 0x55ac8f9ad460 2023-07-29T16:21:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f112e700b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f9b8bd0 con 0x55ac8f9b85b0 2023-07-29T16:21:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1113fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9d15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47242/0 (socket says 172.21.15.31:47242) 2023-07-29T16:21:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 -- 172.21.15.31:0/3019600686 learned_addr learned my addr 172.21.15.31:0/3019600686 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f111939c700 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 -- 172.21.15.31:0/3019600686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b85b0 msgr2=0x55ac8f9b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 -- 172.21.15.31:0/3019600686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 msgr2=0x55ac8f9d15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9d15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1118b9b700 1 -- 172.21.15.31:0/3019600686 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac8f75d2d0 con 0x55ac8f9ad460 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f111939c700 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.673+0000 7f1113fff700 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9d15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:51.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f1118b9b700 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9ca540 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f110802c450 tx=0x7f1108009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ffffa131d30492 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f10ff7fe700 1 -- 172.21.15.31:0/3019600686 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110801d070 con 0x55ac8f9ad460 2023-07-29T16:21:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f10ff7fe700 1 -- 172.21.15.31:0/3019600686 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ac8f933d90 con 0x55ac8f9ad460 2023-07-29T16:21:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f10ff7fe700 1 -- 172.21.15.31:0/3019600686 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108004a50 con 0x55ac8f9ad460 2023-07-29T16:21:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f112e700b80 1 -- 172.21.15.31:0/3019600686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 msgr2=0x55ac8f9ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f112e700b80 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9ca540 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f110802c450 tx=0x7f1108009290 comp rx=0 tx=0).stop 2023-07-29T16:21:51.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f112e700b80 1 -- 172.21.15.31:0/3019600686 shutdown_connections 2023-07-29T16:21:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.674+0000 7f112e700b80 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9ca540 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 --2- 172.21.15.31:0/3019600686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9d15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 -- 172.21.15.31:0/3019600686 >> 172.21.15.31:0/3019600686 conn(0x55ac8f9b1790 msgr2=0x55ac8f9c9e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 -- 172.21.15.31:0/3019600686 shutdown_connections 2023-07-29T16:21:51.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 -- 172.21.15.31:0/3019600686 wait complete. 2023-07-29T16:21:51.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 -- start start 2023-07-29T16:21:51.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.675+0000 7f112e700b80 1 Processor -- start 2023-07-29T16:21:51.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9cea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9cef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f9d04c0 con 0x55ac8f9b85b0 2023-07-29T16:21:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f9d0630 con 0x55ac8f9ad460 2023-07-29T16:21:51.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac8f9d07f0 con 0x55ac8f9b7c60 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f112e700b80 10 client.0 _ll_register_callbacks cb 0x55ac8f8ae6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9cea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47250/0 (socket says 172.21.15.31:47250) 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 -- 172.21.15.31:0/2523087550 learned_addr learned my addr 172.21.15.31:0/2523087550 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f111939c700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1113fff700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 -- 172.21.15.31:0/2523087550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 msgr2=0x55ac8f9cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 -- 172.21.15.31:0/2523087550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b85b0 msgr2=0x55ac8f9cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.676+0000 7f1118b9b700 1 -- 172.21.15.31:0/2523087550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111419e6c0 con 0x55ac8f9ad460 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.677+0000 7f1113fff700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.677+0000 7f1118b9b700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9cea30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f11080098e0 tx=0x7f110802cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdbec66418ade870 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.677+0000 7f10f3fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:51.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.677+0000 7f10f3fff700 1 -- 172.21.15.31:0/2523087550 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110801d030 con 0x55ac8f9ad460 2023-07-29T16:21:51.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.677+0000 7f10f3fff700 1 -- 172.21.15.31:0/2523087550 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1108000e40 con 0x55ac8f9ad460 2023-07-29T16:21:51.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.677+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f110801e040 con 0x55ac8f9ad460 2023-07-29T16:21:51.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f10f3fff700 1 -- 172.21.15.31:0/2523087550 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1108017560 con 0x55ac8f9ad460 2023-07-29T16:21:51.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 10 client.27497 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:51.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ac8f71fce0 con 0x55ac8f9ad460 2023-07-29T16:21:51.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f10f3fff700 1 -- 172.21.15.31:0/2523087550 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f110801e040 con 0x55ac8f9ad460 2023-07-29T16:21:51.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f10f3fff700 1 -- 172.21.15.31:0/2523087550 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f1108017c90 con 0x55ac8f9ad460 2023-07-29T16:21:51.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 10 client.27497 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:51.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 10 client.27497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:51.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 20 client.27497 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:51.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9b9320 0x55ac8fa2b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 4 client.27497 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:21:51.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ac8f901dc0 con 0x55ac8f9b9320 2023-07-29T16:21:51.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.678+0000 7f1113fff700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9b9320 0x55ac8fa2b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.679+0000 7f1113fff700 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9b9320 0x55ac8fa2b420 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7b18dbc74de3e32 server_cookie=f7736b8d176a0db7 in_seq=0 out_seq=0 2023-07-29T16:21:51.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.679+0000 7f10f3fff700 10 client.27497 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:51.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f10f3fff700 1 -- 172.21.15.31:0/2523087550 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ac8f901dc0 con 0x55ac8f9b9320 2023-07-29T16:21:51.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f10f3fff700 10 client.27497 handle_command_reply: tid=0 2023-07-29T16:21:51.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 client.27497 shutdown 2023-07-29T16:21:51.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 msgr2=0x55ac8f9cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9cea30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f11080098e0 tx=0x7f110802cc10 comp rx=0 tx=0).stop 2023-07-29T16:21:51.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 shutdown_connections 2023-07-29T16:21:51.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ac8f9b9320 0x55ac8fa2b420 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ac8f9b85b0 0x55ac8f9cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ac8f9ad460 0x55ac8f9cea30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 --2- 172.21.15.31:0/2523087550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ac8f9b7c60 0x55ac8f9cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 >> 172.21.15.31:0/2523087550 conn(0x55ac8f9b1790 msgr2=0x55ac8f9b69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f10f3fff700 0 client.27497 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:51.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 shutdown_connections 2023-07-29T16:21:51.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 1 -- 172.21.15.31:0/2523087550 wait complete. 2023-07-29T16:21:51.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.683+0000 7f112e700b80 20 client.27497 trim_cache size 0 max 16384 2023-07-29T16:21:51.698 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:21:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:21:51.698 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "86f8fa22-3704-4dfa-8231-d50b2aa0cb03", 2023-07-29T16:21:51.699 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:21:51.699 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:21:51.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f11040a0810 msgr2=0x7f11040a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f11040a0810 0x7f11040a2cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f110c009fd0 tx=0x7f110c00b040 comp rx=0 tx=0).stop 2023-07-29T16:21:51.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 msgr2=0x7f1114196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 0x7f1114196610 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f110000e9a0 tx=0x7f110000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 shutdown_connections 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f11040a0810 0x7f11040a2cc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1114101f10 0x7f111419d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1114100390 0x7f11141960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 --2- 172.21.15.31:0/3178201295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114101580 0x7f1114196610 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 >> 172.21.15.31:0/3178201295 conn(0x7f11140fb9b0 msgr2=0x7f11140fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 shutdown_connections 2023-07-29T16:21:51.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.684+0000 7f111adff700 1 -- 172.21.15.31:0/3178201295 wait complete. 2023-07-29T16:21:51.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:21:51.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 362704 8.91 MB/sec execute 96 sec latency 105.042 ms 2023-07-29T16:21:51.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 -- 172.21.15.31:0/2880730383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14105fe0 msgr2=0x7feb14108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 --2- 172.21.15.31:0/2880730383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14105fe0 0x7feb14108440 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009f90 tx=0x7feb0c00b330 comp rx=0 tx=0).stop 2023-07-29T16:21:51.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 -- 172.21.15.31:0/2880730383 shutdown_connections 2023-07-29T16:21:51.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 --2- 172.21.15.31:0/2880730383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14105fe0 0x7feb14108440 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 --2- 172.21.15.31:0/2880730383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb14100da0 0x7feb14103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 --2- 172.21.15.31:0/2880730383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb141036c0 0x7feb14105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 -- 172.21.15.31:0/2880730383 >> 172.21.15.31:0/2880730383 conn(0x7feb140fa840 msgr2=0x7feb140fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 -- 172.21.15.31:0/2880730383 shutdown_connections 2023-07-29T16:21:51.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.955+0000 7feb1adff700 1 -- 172.21.15.31:0/2880730383 wait complete. 2023-07-29T16:21:51.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.956+0000 7feb1adff700 1 Processor -- start 2023-07-29T16:21:51.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.956+0000 7feb1adff700 1 -- start start 2023-07-29T16:21:51.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1adff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 0x7feb14198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1adff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb141036c0 0x7feb141988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1adff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb14105fe0 0x7feb1419f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1adff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1419ff10 con 0x7feb14105fe0 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1adff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141a0050 con 0x7feb14100da0 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1adff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141a01e0 con 0x7feb141036c0 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 0x7feb14198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 0x7feb14198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47272/0 (socket says 172.21.15.31:47272) 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb18b9b700 1 -- 172.21.15.31:0/1487187894 learned_addr learned my addr 172.21.15.31:0/1487187894 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb13fff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb141036c0 0x7feb141988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb1939c700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb14105fe0 0x7feb1419f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb18b9b700 1 -- 172.21.15.31:0/1487187894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb141036c0 msgr2=0x7feb141988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.957+0000 7feb18b9b700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb141036c0 0x7feb141988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb18b9b700 1 -- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb14105fe0 msgr2=0x7feb1419f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb18b9b700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb14105fe0 0x7feb1419f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb18b9b700 1 -- 172.21.15.31:0/1487187894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c009c40 con 0x7feb14100da0 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb13fff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb141036c0 0x7feb141988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb18b9b700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 0x7feb14198390 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7feb0800e9c0 tx=0x7feb0800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb11ffb700 1 -- 172.21.15.31:0/1487187894 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08004a30 con 0x7feb14100da0 2023-07-29T16:21:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb11ffb700 1 -- 172.21.15.31:0/1487187894 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb0800cc80 con 0x7feb14100da0 2023-07-29T16:21:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb141a04c0 con 0x7feb14100da0 2023-07-29T16:21:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb11ffb700 1 -- 172.21.15.31:0/1487187894 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080054e0 con 0x7feb14100da0 2023-07-29T16:21:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.958+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb141a0a20 con 0x7feb14100da0 2023-07-29T16:21:51.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.961+0000 7feb11ffb700 1 -- 172.21.15.31:0/1487187894 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7feb08004b90 con 0x7feb14100da0 2023-07-29T16:21:51.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.963+0000 7feb11ffb700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb040a0810 0x7feb040a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.963+0000 7feb11ffb700 1 -- 172.21.15.31:0/1487187894 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feb08014070 con 0x7feb14100da0 2023-07-29T16:21:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.963+0000 7feb2e74fb80 1 Processor -- start 2023-07-29T16:21:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.963+0000 7feb13fff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb040a0810 0x7feb040a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 -- start start 2023-07-29T16:21:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 0x564f8c996ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c997530 0x564f8c997d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c998740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb13fff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb040a0810 0x7feb040a2cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feb00005950 tx=0x7feb0000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c8eea10 con 0x564f8c997530 2023-07-29T16:21:51.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c8ec3d0 con 0x564f8c9982f0 2023-07-29T16:21:51.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c8e7cc0 con 0x564f8c996be0 2023-07-29T16:21:51.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 0x564f8c996ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 0x564f8c996ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37130/0 (socket says 172.21.15.31:37130) 2023-07-29T16:21:51.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb18b9b700 1 -- 172.21.15.31:0/977860514 learned_addr learned my addr 172.21.15.31:0/977860514 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.964+0000 7feb13fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c997530 0x564f8c997d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb1939c700 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c998740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb18b9b700 1 -- 172.21.15.31:0/977860514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 msgr2=0x564f8c998740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb18b9b700 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c998740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb18b9b700 1 -- 172.21.15.31:0/977860514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c997530 msgr2=0x564f8c997d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb18b9b700 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c997530 0x564f8c997d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb18b9b700 1 -- 172.21.15.31:0/977860514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f8c715de0 con 0x564f8c996be0 2023-07-29T16:21:51.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feb18b9b700 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 0x564f8c996ff0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feb08099270 tx=0x7feb08098cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7083552fbab9b84 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feafeffd700 1 -- 172.21.15.31:0/977860514 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0809d310 con 0x564f8c996be0 2023-07-29T16:21:51.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feafeffd700 1 -- 172.21.15.31:0/977860514 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb080a1730 con 0x564f8c996be0 2023-07-29T16:21:51.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.965+0000 7feafeffd700 1 -- 172.21.15.31:0/977860514 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080987d0 con 0x564f8c996be0 2023-07-29T16:21:51.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 -- 172.21.15.31:0/977860514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 msgr2=0x564f8c996ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 0x564f8c996ff0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7feb08099270 tx=0x7feb08098cc0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 -- 172.21.15.31:0/977860514 shutdown_connections 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c998740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c996be0 0x564f8c996ff0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/977860514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c997530 0x564f8c997d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 -- 172.21.15.31:0/977860514 >> 172.21.15.31:0/977860514 conn(0x564f8c98c3e0 msgr2=0x564f8c991990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 -- 172.21.15.31:0/977860514 shutdown_connections 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.966+0000 7feb2e74fb80 1 -- 172.21.15.31:0/977860514 wait complete. 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.967+0000 7feb2e74fb80 1 -- start start 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.967+0000 7feb2e74fb80 1 Processor -- start 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c884510 con 0x564f8c996be0 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c8841b0 con 0x564f8c9982f0 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c76b920 con 0x564f8c997530 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb2e74fb80 10 client.0 _ll_register_callbacks cb 0x564f8c88d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb1939c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb1939c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47294/0 (socket says 172.21.15.31:47294) 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb1939c700 1 -- 172.21.15.31:0/2023623032 learned_addr learned my addr 172.21.15.31:0/2023623032 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.968+0000 7feb13fff700 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb1939c700 1 -- 172.21.15.31:0/2023623032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 msgr2=0x564f8c9a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb1939c700 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb1939c700 1 -- 172.21.15.31:0/2023623032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 msgr2=0x564f8c9a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb1939c700 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb1939c700 1 -- 172.21.15.31:0/2023623032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb141a0a20 con 0x564f8c9982f0 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb1939c700 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a7dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feb0c015000 tx=0x7feb0c007ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b21295c4fc05ddcb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feaf3fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feaf3fff700 1 -- 172.21.15.31:0/2023623032 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00e040 con 0x564f8c9982f0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feaf3fff700 1 -- 172.21.15.31:0/2023623032 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feb0c003c00 con 0x564f8c9982f0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f8c912d10 con 0x564f8c9982f0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.969+0000 7feaf3fff700 1 -- 172.21.15.31:0/2023623032 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb0c003e00 con 0x564f8c9982f0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 10 client.27509 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564f8c6fec60 con 0x564f8c9982f0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feaf3fff700 1 -- 172.21.15.31:0/2023623032 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564f8c912d10 con 0x564f8c9982f0 2023-07-29T16:21:51.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feaf3fff700 1 -- 172.21.15.31:0/2023623032 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7feb0c019070 con 0x564f8c9982f0 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 10 client.27509 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 10 client.27509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 20 client.27509 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ac510 0x564f8c9ae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 4 client.27509 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.970+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f8c965d70 con 0x564f8c9ac510 2023-07-29T16:21:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.971+0000 7feb18b9b700 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ac510 0x564f8c9ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.971+0000 7feb18b9b700 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ac510 0x564f8c9ae920 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=663c60df73081038 server_cookie=5dcfc3056edb96ea in_seq=0 out_seq=0 2023-07-29T16:21:51.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.971+0000 7feaf3fff700 10 client.27509 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feaf3fff700 1 -- 172.21.15.31:0/2023623032 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564f8c965d70 con 0x564f8c9ac510 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feaf3fff700 10 client.27509 handle_command_reply: tid=0 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 client.27509 shutdown 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 msgr2=0x564f8c9a7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a7dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7feb0c015000 tx=0x7feb0c007ce0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 shutdown_connections 2023-07-29T16:21:51.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ac510 0x564f8c9ae920 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a7dc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feaf3fff700 0 client.27509 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2023623032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 >> 172.21.15.31:0/2023623032 conn(0x564f8c98c3e0 msgr2=0x564f8c991060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.972+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 shutdown_connections 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.973+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2023623032 wait complete. 2023-07-29T16:21:51.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.973+0000 7feb2e74fb80 20 client.27509 trim_cache size 0 max 16384 2023-07-29T16:21:51.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 Processor -- start 2023-07-29T16:21:51.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 -- start start 2023-07-29T16:21:51.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 0x564f8c99f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c9982f0 0x564f8c99b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c98e460 con 0x564f8c9982f0 2023-07-29T16:21:51.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c99e440 con 0x564f8c996be0 2023-07-29T16:21:51.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.991+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c99f210 con 0x564f8c997530 2023-07-29T16:21:51.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 0x564f8c99f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb13fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 0x564f8c99f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47306/0 (socket says 172.21.15.31:47306) 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb13fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37138/0 (socket says 172.21.15.31:37138) 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 -- 172.21.15.31:0/3258109639 learned_addr learned my addr 172.21.15.31:0/3258109639 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb1939c700 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c9982f0 0x564f8c99b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 -- 172.21.15.31:0/3258109639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 msgr2=0x564f8c9a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 -- 172.21.15.31:0/3258109639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c9982f0 msgr2=0x564f8c99b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:51.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c9982f0 0x564f8c99b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 -- 172.21.15.31:0/3258109639 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f8c73c250 con 0x564f8c996be0 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feb18b9b700 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 0x564f8c99f4e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7feb08098db0 tx=0x7feb08009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d5ad34ff6657213 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feaff7fe700 1 -- 172.21.15.31:0/3258109639 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0809d310 con 0x564f8c996be0 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feaff7fe700 1 -- 172.21.15.31:0/3258109639 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb080c7070 con 0x564f8c996be0 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.992+0000 7feaff7fe700 1 -- 172.21.15.31:0/3258109639 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb08099540 con 0x564f8c996be0 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 -- 172.21.15.31:0/3258109639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 msgr2=0x564f8c99f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:51.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 0x564f8c99f4e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7feb08098db0 tx=0x7feb08009550 comp rx=0 tx=0).stop 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 -- 172.21.15.31:0/3258109639 shutdown_connections 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c9982f0 0x564f8c99b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c996be0 0x564f8c99f4e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/3258109639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 -- 172.21.15.31:0/3258109639 >> 172.21.15.31:0/3258109639 conn(0x564f8c98c3e0 msgr2=0x564f8c99e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 -- 172.21.15.31:0/3258109639 shutdown_connections 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.993+0000 7feb2e74fb80 1 -- 172.21.15.31:0/3258109639 wait complete. 2023-07-29T16:21:51.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 -- start start 2023-07-29T16:21:51.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 Processor -- start 2023-07-29T16:21:51.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:51.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c9a45e0 con 0x564f8c996be0 2023-07-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c9a4750 con 0x564f8c9982f0 2023-07-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.994+0000 7feb2e74fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f8c9a4910 con 0x564f8c997530 2023-07-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb2e74fb80 10 client.0 _ll_register_callbacks cb 0x564f8c88d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:21:52.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb13fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb18b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb13fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37146/0 (socket says 172.21.15.31:37146) 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb13fff700 1 -- 172.21.15.31:0/2497057717 learned_addr learned my addr 172.21.15.31:0/2497057717 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb1939c700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb1939c700 1 -- 172.21.15.31:0/2497057717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 msgr2=0x564f8c9a7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb1939c700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb1939c700 1 -- 172.21.15.31:0/2497057717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 msgr2=0x564f8c9a75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb1939c700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb1939c700 1 -- 172.21.15.31:0/2497057717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f8c715de0 con 0x564f8c9982f0 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb18b9b700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.995+0000 7feb13fff700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feb1939c700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a4050 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7feb0c026bb0 tx=0x7feb0c026cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e423a67bb0830e59 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feaf3fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feaf3fff700 1 -- 172.21.15.31:0/2497057717 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00e040 con 0x564f8c9982f0 2023-07-29T16:21:52.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb080a1730 con 0x564f8c9982f0 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feaf3fff700 1 -- 172.21.15.31:0/2497057717 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feb0c00be00 con 0x564f8c9982f0 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feaf3fff700 1 -- 172.21.15.31:0/2497057717 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb0c00b680 con 0x564f8c9982f0 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feb2e74fb80 10 client.27515 fetch_fsmap learned FSMap version 28 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564f8c7142b0 con 0x564f8c9982f0 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.996+0000 7feaf3fff700 1 -- 172.21.15.31:0/2497057717 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb080a1730 con 0x564f8c9982f0 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feaf3fff700 1 -- 172.21.15.31:0/2497057717 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7feb0c019050 con 0x564f8c9982f0 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb2e74fb80 10 client.27515 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:21:52.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb2e74fb80 10 client.27515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb2e74fb80 20 client.27515 populate_metadata read hostname 'smithi031' 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ff270 0x564f8ca01680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb2e74fb80 4 client.27515 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564f8c8e0d40 con 0x564f8c9ff270 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb18b9b700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ff270 0x564f8ca01680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feb18b9b700 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ff270 0x564f8ca01680 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f537fab08786e55 server_cookie=dd644b51d18c2f49 in_seq=0 out_seq=0 2023-07-29T16:21:52.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.997+0000 7feaf3fff700 10 client.27515 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feaf3fff700 1 -- 172.21.15.31:0/2497057717 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564f8c8e0d40 con 0x564f8c9ff270 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feaf3fff700 10 client.27515 handle_command_reply: tid=0 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 client.27515 shutdown 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 msgr2=0x564f8c9a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a4050 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7feb0c026bb0 tx=0x7feb0c026cd0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 shutdown_connections 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564f8c9ff270 0x564f8ca01680 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564f8c9982f0 0x564f8c9a4050 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f8c996be0 0x564f8c9a75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 --2- 172.21.15.31:0/2497057717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564f8c997530 0x564f8c9a7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feaf3fff700 0 client.27515 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 >> 172.21.15.31:0/2497057717 conn(0x564f8c98c3e0 msgr2=0x564f8c99f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 shutdown_connections 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 1 -- 172.21.15.31:0/2497057717 wait complete. 2023-07-29T16:21:52.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.998+0000 7feb2e74fb80 20 client.27515 trim_cache size 0 max 16384 2023-07-29T16:21:52.007 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:21:52.007 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:21:52.007 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:21:52.007 INFO:teuthology.orchestra.run.smithi031.stdout: "86f8fa22-3704-4dfa-8231-d50b2aa0cb03": { 2023-07-29T16:21:52.007 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:21:52.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "86f8fa22-3704-4dfa-8231-d50b2aa0cb03", 2023-07-29T16:21:52.008 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:21:52.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:21:52.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:21:52.008 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:21:52.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb040a0810 msgr2=0x7feb040a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:52.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb040a0810 0x7feb040a2cc0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feb00005950 tx=0x7feb0000b410 comp rx=0 tx=0).stop 2023-07-29T16:21:52.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 msgr2=0x7feb14198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:21:52.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 0x7feb14198390 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7feb0800e9c0 tx=0x7feb0800ecd0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 shutdown_connections 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb040a0810 0x7feb040a2cc0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb14105fe0 0x7feb1419f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb14100da0 0x7feb14198390 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 --2- 172.21.15.31:0/1487187894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb141036c0 0x7feb141988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 >> 172.21.15.31:0/1487187894 conn(0x7feb140fa840 msgr2=0x7feb141055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 shutdown_connections 2023-07-29T16:21:52.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:21:51.999+0000 7feb1adff700 1 -- 172.21.15.31:0/1487187894 wait complete. 2023-07-29T16:21:52.017 INFO:tasks.cephfs.filesystem:scrub status for tag:86f8fa22-3704-4dfa-8231-d50b2aa0cb03 - {'path': '/', 'tag': '86f8fa22-3704-4dfa-8231-d50b2aa0cb03', 'options': 'recursive,force'} 2023-07-29T16:21:52.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:52 smithi039 ceph-mon[128858]: pgmap v562: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2023-07-29T16:21:52.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:52 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:52.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:52 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:52 smithi156 ceph-mon[129245]: pgmap v562: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2023-07-29T16:21:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:52 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:52 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:52.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 365150 8.96 MB/sec execute 97 sec latency 17.659 ms 2023-07-29T16:21:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:52 smithi031 ceph-mon[95504]: pgmap v562: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 204 op/s 2023-07-29T16:21:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:52 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:52 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:53 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:21:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:53 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:53 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:53 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:21:53.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:53 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:53 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:53 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:21:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:53 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:53 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:53 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:21:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:53 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:53 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 366399 8.94 MB/sec execute 98 sec latency 11.895 ms 2023-07-29T16:21:53.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:53 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:21:53.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:53 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:53 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:53 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:21:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:53 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:53 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:21:54.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:54 smithi039 ceph-mon[128858]: pgmap v563: 97 pgs: 97 active+clean; 82 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-07-29T16:21:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:54 smithi156 ceph-mon[129245]: pgmap v563: 97 pgs: 97 active+clean; 82 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-07-29T16:21:54.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 368689 8.98 MB/sec execute 99 sec latency 13.295 ms 2023-07-29T16:21:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:54 smithi031 ceph-mon[95504]: pgmap v563: 97 pgs: 97 active+clean; 82 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-07-29T16:21:55.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 369862 8.94 MB/sec execute 100 sec latency 17.182 ms 2023-07-29T16:21:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:56 smithi039 ceph-mon[128858]: pgmap v564: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2023-07-29T16:21:56.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:56 smithi156 ceph-mon[129245]: pgmap v564: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2023-07-29T16:21:56.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 371815 8.97 MB/sec execute 101 sec latency 40.445 ms 2023-07-29T16:21:56.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:56 smithi031 ceph-mon[95504]: pgmap v564: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2023-07-29T16:21:57.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 373135 8.95 MB/sec execute 102 sec latency 12.738 ms 2023-07-29T16:21:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:57 smithi031 ceph-mon[95504]: pgmap v565: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2023-07-29T16:21:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:57 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:21:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:57 smithi039 ceph-mon[128858]: pgmap v565: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2023-07-29T16:21:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:57 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:21:58.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:57 smithi156 ceph-mon[129245]: pgmap v565: 97 pgs: 97 active+clean; 84 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 264 op/s 2023-07-29T16:21:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:57 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:21:58.870 INFO:tasks.workunit.client.0.smithi031.stdout: 1 375444 8.98 MB/sec execute 103 sec latency 15.378 ms 2023-07-29T16:21:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:58 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:21:59.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:58 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:21:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:58 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:21:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:58 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:21:59.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:58 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:21:59.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:58 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:21:59.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 376729 8.96 MB/sec execute 104 sec latency 12.629 ms 2023-07-29T16:22:00.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:21:59 smithi031 ceph-mon[95504]: pgmap v566: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 309 op/s 2023-07-29T16:22:00.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:21:59 smithi039 ceph-mon[128858]: pgmap v566: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 309 op/s 2023-07-29T16:22:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:21:59 smithi156 ceph-mon[129245]: pgmap v566: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 309 op/s 2023-07-29T16:22:00.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 378942 8.99 MB/sec execute 105 sec latency 16.151 ms 2023-07-29T16:22:01.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 380139 8.97 MB/sec execute 106 sec latency 51.014 ms 2023-07-29T16:22:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 -- 172.21.15.31:0/4131483312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638103450 msgr2=0x7f8638105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 --2- 172.21.15.31:0/4131483312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638103450 0x7f8638105890 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8624009990 tx=0x7f8624009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 -- 172.21.15.31:0/4131483312 shutdown_connections 2023-07-29T16:22:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 --2- 172.21.15.31:0/4131483312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638105f20 0x7f86381083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 --2- 172.21.15.31:0/4131483312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8638068e40 0x7f8638102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 --2- 172.21.15.31:0/4131483312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638103450 0x7f8638105890 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 -- 172.21.15.31:0/4131483312 >> 172.21.15.31:0/4131483312 conn(0x7f86380fa840 msgr2=0x7f86380fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 -- 172.21.15.31:0/4131483312 shutdown_connections 2023-07-29T16:22:02.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.691+0000 7f863ebff700 1 -- 172.21.15.31:0/4131483312 wait complete. 2023-07-29T16:22:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.692+0000 7f863ebff700 1 Processor -- start 2023-07-29T16:22:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.692+0000 7f863ebff700 1 -- start start 2023-07-29T16:22:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863ebff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8638068e40 0x7f86381983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863ebff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 0x7f8638198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863ebff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638105f20 0x7f86381997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863ebff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638199e00 con 0x7f8638068e40 2023-07-29T16:22:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863ebff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8638199f70 con 0x7f8638103450 2023-07-29T16:22:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863ebff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863819a130 con 0x7f8638105f20 2023-07-29T16:22:02.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 0x7f8638198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 0x7f8638198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37540/0 (socket says 172.21.15.31:37540) 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 -- 172.21.15.31:0/2233166364 learned_addr learned my addr 172.21.15.31:0/2233166364 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8638068e40 0x7f86381983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f863d19c700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638105f20 0x7f86381997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 -- 172.21.15.31:0/2233166364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638105f20 msgr2=0x7f86381997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638105f20 0x7f86381997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 -- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8638068e40 msgr2=0x7f86381983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8638068e40 0x7f86381983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.693+0000 7f8637fff700 1 -- 172.21.15.31:0/2233166364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86240095f0 con 0x7f8638103450 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.694+0000 7f8637fff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 0x7f8638198920 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f862400b680 tx=0x7f8624005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.694+0000 7f8635ffb700 1 -- 172.21.15.31:0/2233166364 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862401d070 con 0x7f8638103450 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.694+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863819a3b0 con 0x7f8638103450 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.694+0000 7f8635ffb700 1 -- 172.21.15.31:0/2233166364 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8624022470 con 0x7f8638103450 2023-07-29T16:22:02.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.694+0000 7f8635ffb700 1 -- 172.21.15.31:0/2233166364 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624017650 con 0x7f8638103450 2023-07-29T16:22:02.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.694+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86381a0bd0 con 0x7f8638103450 2023-07-29T16:22:02.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.698+0000 7f8635ffb700 1 -- 172.21.15.31:0/2233166364 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f86240225e0 con 0x7f8638103450 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.698+0000 7f8652613b80 1 Processor -- start 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 -- start start 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x55741756c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756cdf0 0x55741756d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741756e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574174c49b0 con 0x55741756cdf0 2023-07-29T16:22:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574174c2370 con 0x5574173c86b0 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8652613b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574174bdc60 con 0x55741756dcf0 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f863d19c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x55741756c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f863d19c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x55741756c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37552/0 (socket says 172.21.15.31:37552) 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f863d19c700 1 -- 172.21.15.31:0/3750882702 learned_addr learned my addr 172.21.15.31:0/3750882702 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f863c99b700 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756cdf0 0x55741756d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8637fff700 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741756e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.699+0000 7f8635ffb700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86280a0790 0x7f86280a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 -- 172.21.15.31:0/3750882702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 msgr2=0x55741756e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741756e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f8635ffb700 1 -- 172.21.15.31:0/2233166364 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f86240ccfb0 con 0x7f8638103450 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 -- 172.21.15.31:0/3750882702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756cdf0 msgr2=0x55741756d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756cdf0 0x55741756d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 -- 172.21.15.31:0/3750882702 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574172ebd80 con 0x5574173c86b0 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863c99b700 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756cdf0 0x55741756d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:02.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86280a0790 0x7f86280a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.700+0000 7f863d19c700 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x55741756c8b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f86300115c0 tx=0x7f86300118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d830349928dd2f1e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8622ffd700 1 -- 172.21.15.31:0/3750882702 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863000e040 con 0x5574173c86b0 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8622ffd700 1 -- 172.21.15.31:0/3750882702 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8630027070 con 0x5574173c86b0 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8622ffd700 1 -- 172.21.15.31:0/3750882702 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630023650 con 0x5574173c86b0 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 -- 172.21.15.31:0/3750882702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 msgr2=0x55741756c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x55741756c8b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f86300115c0 tx=0x7f86300118d0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f863d19c700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86280a0790 0x7f86280a2c40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8630022000 tx=0x7f863001f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:02.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 -- 172.21.15.31:0/3750882702 shutdown_connections 2023-07-29T16:22:02.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741756e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x55741756c8b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 --2- 172.21.15.31:0/3750882702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756cdf0 0x55741756d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 -- 172.21.15.31:0/3750882702 >> 172.21.15.31:0/3750882702 conn(0x557417561fd0 msgr2=0x557417567300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 -- 172.21.15.31:0/3750882702 shutdown_connections 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.701+0000 7f8652613b80 1 -- 172.21.15.31:0/3750882702 wait complete. 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.702+0000 7f8652613b80 1 -- start start 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.702+0000 7f8652613b80 1 Processor -- start 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417575ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 0x557417576430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741757d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741745a4b0 con 0x5574173c86b0 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741745a150 con 0x55741756cdf0 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574173418c0 con 0x55741756dcf0 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8652613b80 10 client.0 _ll_register_callbacks cb 0x5574174635c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 0x557417576430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 0x557417576430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37566/0 (socket says 172.21.15.31:37566) 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 -- 172.21.15.31:0/2450693664 learned_addr learned my addr 172.21.15.31:0/2450693664 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f863c99b700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417575ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f863d19c700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741757d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 -- 172.21.15.31:0/2450693664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 msgr2=0x55741757d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741757d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 -- 172.21.15.31:0/2450693664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 msgr2=0x557417575ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417575ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.703+0000 7f8637fff700 1 -- 172.21.15.31:0/2450693664 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c0098e0 con 0x55741756cdf0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8637fff700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 0x557417576430 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8624006010 tx=0x7f8624009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8ae05326efa2951 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8617fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8617fff700 1 -- 172.21.15.31:0/2450693664 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862409ab40 con 0x55741756cdf0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574174e8cb0 con 0x55741756cdf0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8617fff700 1 -- 172.21.15.31:0/2450693664 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f862409a6e0 con 0x55741756cdf0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8617fff700 1 -- 172.21.15.31:0/2450693664 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f862409a4b0 con 0x55741756cdf0 2023-07-29T16:22:02.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 10 client.27527 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8630005e70 con 0x55741756cdf0 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8617fff700 1 -- 172.21.15.31:0/2450693664 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5574174e8cb0 con 0x55741756cdf0 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8617fff700 1 -- 172.21.15.31:0/2450693664 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8624099bd0 con 0x55741756cdf0 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 10 client.27527 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 10 client.27527 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 20 client.27527 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:02.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175817f0 0x557417583bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 4 client.27527 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.704+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55741753bd10 con 0x5574175817f0 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.705+0000 7f863c99b700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175817f0 0x557417583bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.705+0000 7f863c99b700 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175817f0 0x557417583bd0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5560887b93c0045f server_cookie=ef1c39940fa3282f in_seq=0 out_seq=0 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.705+0000 7f8617fff700 10 client.27527 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8617fff700 1 -- 172.21.15.31:0/2450693664 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55741753bd10 con 0x5574175817f0 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8617fff700 10 client.27527 handle_command_reply: tid=0 2023-07-29T16:22:02.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 client.27527 shutdown 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 msgr2=0x557417576430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 0x557417576430 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8624006010 tx=0x7f8624009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 shutdown_connections 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175817f0 0x557417583bd0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756dcf0 0x55741757d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417575ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 --2- 172.21.15.31:0/2450693664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756cdf0 0x557417576430 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 >> 172.21.15.31:0/2450693664 conn(0x557417561fd0 msgr2=0x557417566a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8617fff700 0 client.27527 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 shutdown_connections 2023-07-29T16:22:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.707+0000 7f8652613b80 1 -- 172.21.15.31:0/2450693664 wait complete. 2023-07-29T16:22:02.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.708+0000 7f8652613b80 20 client.27527 trim_cache size 0 max 16384 2023-07-29T16:22:02.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 Processor -- start 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 -- start start 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x5574175740b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x5574175745f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756dcf0 0x55741757d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557417564050 con 0x55741756dcf0 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574175753a0 con 0x5574173c86b0 2023-07-29T16:22:02.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.724+0000 7f8652613b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557417575a60 con 0x55741756cdf0 2023-07-29T16:22:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f863c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x5574175740b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x5574175745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f863c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x5574175740b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37568/0 (socket says 172.21.15.31:37568) 2023-07-29T16:22:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f863c99b700 1 -- 172.21.15.31:0/68705051 learned_addr learned my addr 172.21.15.31:0/68705051 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f863d19c700 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756dcf0 0x55741757d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 -- 172.21.15.31:0/68705051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 msgr2=0x5574175740b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x5574175740b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 -- 172.21.15.31:0/68705051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756dcf0 msgr2=0x55741757d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756dcf0 0x55741757d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 -- 172.21.15.31:0/68705051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574172d4c00 con 0x55741756cdf0 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f863c99b700 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x5574175740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f8637fff700 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x5574175745f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8624097100 tx=0x7f86240967b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e29261666643f3f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:02.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f86237fe700 1 -- 172.21.15.31:0/68705051 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86240c8070 con 0x55741756cdf0 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.725+0000 7f86237fe700 1 -- 172.21.15.31:0/68705051 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86240a2aa0 con 0x55741756cdf0 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f86237fe700 1 -- 172.21.15.31:0/68705051 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86240d0350 con 0x55741756cdf0 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 -- 172.21.15.31:0/68705051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 msgr2=0x5574175745f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x5574175745f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8624097100 tx=0x7f86240967b0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 -- 172.21.15.31:0/68705051 shutdown_connections 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55741756dcf0 0x55741757d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5574173c86b0 0x5574175740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 --2- 172.21.15.31:0/68705051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x5574175745f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 -- 172.21.15.31:0/68705051 >> 172.21.15.31:0/68705051 conn(0x557417561fd0 msgr2=0x557417583800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:02.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 -- 172.21.15.31:0/68705051 shutdown_connections 2023-07-29T16:22:02.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.726+0000 7f8652613b80 1 -- 172.21.15.31:0/68705051 wait complete. 2023-07-29T16:22:02.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.727+0000 7f8652613b80 1 -- start start 2023-07-29T16:22:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.727+0000 7f8652613b80 1 Processor -- start 2023-07-29T16:22:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417579660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x557417579ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756dcf0 0x55741757ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741757cf70 con 0x5574173c86b0 2023-07-29T16:22:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741757d0e0 con 0x55741756dcf0 2023-07-29T16:22:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741757d2a0 con 0x55741756cdf0 2023-07-29T16:22:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8652613b80 10 client.0 _ll_register_callbacks cb 0x5574174635c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:02.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f863c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417579660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x557417579ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f863c99b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417579660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43252/0 (socket says 172.21.15.31:43252) 2023-07-29T16:22:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f863c99b700 1 -- 172.21.15.31:0/3047826001 learned_addr learned my addr 172.21.15.31:0/3047826001 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f863d19c700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756dcf0 0x55741757ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 -- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756dcf0 msgr2=0x55741757ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756dcf0 0x55741757ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 -- 172.21.15.31:0/3047826001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 msgr2=0x557417579660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417579660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 -- 172.21.15.31:0/3047826001 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c009d80 con 0x55741756cdf0 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f863c99b700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417579660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8637fff700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x557417579ba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8624097100 tx=0x7f86240d0320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8450719ae965396c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:02.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.728+0000 7f8617fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8617fff700 1 -- 172.21.15.31:0/3047826001 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86240c8030 con 0x55741756cdf0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8617fff700 1 -- 172.21.15.31:0/3047826001 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f862409a6e0 con 0x55741756cdf0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8630027070 con 0x55741756cdf0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8617fff700 1 -- 172.21.15.31:0/3047826001 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f862409a220 con 0x55741756cdf0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 10 client.27544 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8630005e70 con 0x55741756cdf0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8617fff700 1 -- 172.21.15.31:0/3047826001 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8630027070 con 0x55741756cdf0 2023-07-29T16:22:02.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8617fff700 1 -- 172.21.15.31:0/3047826001 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8624099640 con 0x55741756cdf0 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 10 client.27544 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 10 client.27544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 20 client.27544 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.729+0000 7f8652613b80 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175d40c0 0x5574175d64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.730+0000 7f8652613b80 4 client.27544 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.730+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5574174b6ce0 con 0x5574175d40c0 2023-07-29T16:22:02.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.730+0000 7f863c99b700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175d40c0 0x5574175d64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:02.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.730+0000 7f863c99b700 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175d40c0 0x5574175d64a0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8decfd27cc0f02b0 server_cookie=9706bd8b1af351a3 in_seq=0 out_seq=0 2023-07-29T16:22:02.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.730+0000 7f8617fff700 10 client.27544 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:02.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:02 smithi039 ceph-mon[128858]: pgmap v567: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:22:02.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.821+0000 7f8617fff700 1 -- 172.21.15.31:0/3047826001 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5574174b6ce0 con 0x5574175d40c0 2023-07-29T16:22:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.821+0000 7f8617fff700 10 client.27544 handle_command_reply: tid=0 2023-07-29T16:22:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.821+0000 7f8652613b80 1 client.27544 shutdown 2023-07-29T16:22:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 msgr2=0x557417579ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x557417579ba0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8624097100 tx=0x7f86240d0320 comp rx=0 tx=0).stop 2023-07-29T16:22:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 shutdown_connections 2023-07-29T16:22:02.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5574175d40c0 0x5574175d64a0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741756dcf0 0x55741757ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574173c86b0 0x557417579660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 --2- 172.21.15.31:0/3047826001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55741756cdf0 0x557417579ba0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 >> 172.21.15.31:0/3047826001 conn(0x557417561fd0 msgr2=0x5574175740b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8617fff700 0 client.27544 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 shutdown_connections 2023-07-29T16:22:02.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 1 -- 172.21.15.31:0/3047826001 wait complete. 2023-07-29T16:22:02.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:02 smithi156 ceph-mon[129245]: pgmap v567: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:22:02.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.822+0000 7f8652613b80 20 client.27544 trim_cache size 0 max 16384 2023-07-29T16:22:02.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:22:02.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:22:02.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:22:02.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:22:02.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86280a0790 msgr2=0x7f86280a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86280a0790 0x7f86280a2c40 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8630022000 tx=0x7f863001f000 comp rx=0 tx=0).stop 2023-07-29T16:22:02.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 msgr2=0x7f8638198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:02.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 0x7f8638198920 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f862400b680 tx=0x7f8624005740 comp rx=0 tx=0).stop 2023-07-29T16:22:02.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 shutdown_connections 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86280a0790 0x7f86280a2c40 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8638105f20 0x7f86381997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8638068e40 0x7f86381983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 --2- 172.21.15.31:0/2233166364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8638103450 0x7f8638198920 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 >> 172.21.15.31:0/2233166364 conn(0x7f86380fa840 msgr2=0x7f86380fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 shutdown_connections 2023-07-29T16:22:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:02.823+0000 7f863ebff700 1 -- 172.21.15.31:0/2233166364 wait complete. 2023-07-29T16:22:02.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 382322 8.99 MB/sec execute 107 sec latency 35.122 ms 2023-07-29T16:22:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:02 smithi031 ceph-mon[95504]: pgmap v567: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:22:03.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 383632 8.98 MB/sec execute 108 sec latency 12.389 ms 2023-07-29T16:22:04.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:04 smithi039 ceph-mon[128858]: pgmap v568: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 317 op/s 2023-07-29T16:22:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:04 smithi156 ceph-mon[129245]: pgmap v568: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 317 op/s 2023-07-29T16:22:04.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 385583 8.96 MB/sec execute 109 sec latency 49.354 ms 2023-07-29T16:22:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:04 smithi031 ceph-mon[95504]: pgmap v568: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 317 op/s 2023-07-29T16:22:05.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 387096 8.99 MB/sec execute 110 sec latency 89.661 ms 2023-07-29T16:22:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:06 smithi039 ceph-mon[128858]: pgmap v569: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 287 op/s 2023-07-29T16:22:06.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:06 smithi156 ceph-mon[129245]: pgmap v569: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 287 op/s 2023-07-29T16:22:06.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 388876 8.96 MB/sec execute 111 sec latency 25.209 ms 2023-07-29T16:22:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:06 smithi031 ceph-mon[95504]: pgmap v569: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 287 op/s 2023-07-29T16:22:07.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 390588 9.00 MB/sec execute 112 sec latency 31.071 ms 2023-07-29T16:22:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:07 smithi031 ceph-mon[95504]: pgmap v570: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 230 op/s 2023-07-29T16:22:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:07 smithi039 ceph-mon[128858]: pgmap v570: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 230 op/s 2023-07-29T16:22:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:07 smithi156 ceph-mon[129245]: pgmap v570: 97 pgs: 97 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 230 op/s 2023-07-29T16:22:08.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 392397 8.97 MB/sec execute 113 sec latency 13.542 ms 2023-07-29T16:22:09.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 394127 9.01 MB/sec execute 114 sec latency 24.750 ms 2023-07-29T16:22:10.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:10 smithi039 ceph-mon[128858]: pgmap v571: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2023-07-29T16:22:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:10 smithi156 ceph-mon[129245]: pgmap v571: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2023-07-29T16:22:10.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 395542 8.98 MB/sec execute 115 sec latency 17.246 ms 2023-07-29T16:22:10.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:10 smithi031 ceph-mon[95504]: pgmap v571: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2023-07-29T16:22:11.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 397577 9.02 MB/sec execute 116 sec latency 13.426 ms 2023-07-29T16:22:12.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:12 smithi039 ceph-mon[128858]: pgmap v572: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2023-07-29T16:22:12.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:12 smithi156 ceph-mon[129245]: pgmap v572: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2023-07-29T16:22:12.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 398762 8.99 MB/sec execute 117 sec latency 57.092 ms 2023-07-29T16:22:12.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:12 smithi031 ceph-mon[95504]: pgmap v572: 97 pgs: 97 active+clean; 90 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2023-07-29T16:22:13.871 INFO:tasks.workunit.client.0.smithi031.stdout: 1 401002 9.02 MB/sec execute 118 sec latency 17.418 ms 2023-07-29T16:22:14.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:14 smithi039 ceph-mon[128858]: pgmap v573: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2023-07-29T16:22:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:14 smithi156 ceph-mon[129245]: pgmap v573: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2023-07-29T16:22:14.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 402247 9.00 MB/sec execute 119 sec latency 16.719 ms 2023-07-29T16:22:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:14 smithi031 ceph-mon[95504]: pgmap v573: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2023-07-29T16:22:15.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 404366 9.02 MB/sec execute 120 sec latency 15.927 ms 2023-07-29T16:22:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 -- 172.21.15.31:0/2939548016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 msgr2=0x7f31240fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 --2- 172.21.15.31:0/2939548016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31240fe4c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3118009990 tx=0x7f3118009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 -- 172.21.15.31:0/2939548016 shutdown_connections 2023-07-29T16:22:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 --2- 172.21.15.31:0/2939548016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3124105fa0 0x7f3124108450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 --2- 172.21.15.31:0/2939548016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31240fe4c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 --2- 172.21.15.31:0/2939548016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 0x7f3124105a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 -- 172.21.15.31:0/2939548016 >> 172.21.15.31:0/2939548016 conn(0x7f31240f9b70 msgr2=0x7f31240fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:15.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 -- 172.21.15.31:0/2939548016 shutdown_connections 2023-07-29T16:22:15.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.987+0000 7f312afff700 1 -- 172.21.15.31:0/2939548016 wait complete. 2023-07-29T16:22:15.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.989+0000 7f312afff700 1 Processor -- start 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.989+0000 7f312afff700 1 -- start start 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312afff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31241983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312afff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 0x7f3124198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312afff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3124105fa0 0x7f3124199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312afff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124199de0 con 0x7f31240fea00 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312afff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124199f50 con 0x7f3124105fa0 2023-07-29T16:22:15.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312afff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312419a110 con 0x7f31240fe0b0 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31241983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31241983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55510/0 (socket says 172.21.15.31:55510) 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 -- 172.21.15.31:0/914381669 learned_addr learned my addr 172.21.15.31:0/914381669 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3123fff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 0x7f3124198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f312959c700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3124105fa0 0x7f3124199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 -- 172.21.15.31:0/914381669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3124105fa0 msgr2=0x7f3124199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:15.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3124105fa0 0x7f3124199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 -- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 msgr2=0x7f3124198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 0x7f3124198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 -- 172.21.15.31:0/914381669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31180095f0 con 0x7f31240fe0b0 2023-07-29T16:22:15.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3123fff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 0x7f3124198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.990+0000 7f3128d9b700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31241983c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f311800b680 tx=0x7f3118005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.991+0000 7f3121ffb700 1 -- 172.21.15.31:0/914381669 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311801d070 con 0x7f31240fe0b0 2023-07-29T16:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.991+0000 7f3121ffb700 1 -- 172.21.15.31:0/914381669 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3118022470 con 0x7f31240fe0b0 2023-07-29T16:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.991+0000 7f3121ffb700 1 -- 172.21.15.31:0/914381669 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118017650 con 0x7f31240fe0b0 2023-07-29T16:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.991+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312419a390 con 0x7f31240fe0b0 2023-07-29T16:22:15.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.991+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31241a0bb0 con 0x7f31240fe0b0 2023-07-29T16:22:15.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.994+0000 7f3121ffb700 1 -- 172.21.15.31:0/914381669 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f31180225e0 con 0x7f31240fe0b0 2023-07-29T16:22:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.996+0000 7f3121ffb700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31140a0810 0x7f31140a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.996+0000 7f313e84db80 1 Processor -- start 2023-07-29T16:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.996+0000 7f3121ffb700 1 -- 172.21.15.31:0/914381669 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f31180ccfb0 con 0x7f31240fe0b0 2023-07-29T16:22:15.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.996+0000 7f3123fff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31140a0810 0x7f31140a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.996+0000 7f313e84db80 1 -- start start 2023-07-29T16:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31140a0810 0x7f31140a2cc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f31240ff600 tx=0x7f3110009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f313e84db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fe88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f313e84db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 0x5653e1fe97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f313e84db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1fea160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f313e84db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1f409b0 con 0x5653e1fe9cf0 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f313e84db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1f3e370 con 0x5653e1fe8df0 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f313e84db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1f39c60 con 0x5653e1e446b0 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 0x5653e1fe97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 0x5653e1fe97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58564/0 (socket says 172.21.15.31:58564) 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 -- 172.21.15.31:0/1094429445 learned_addr learned my addr 172.21.15.31:0/1094429445 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3128d9b700 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fe88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f312959c700 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1fea160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 -- 172.21.15.31:0/1094429445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 msgr2=0x5653e1fe88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fe88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 -- 172.21.15.31:0/1094429445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 msgr2=0x5653e1fea160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1fea160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.997+0000 7f3123fff700 1 -- 172.21.15.31:0/1094429445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653e1d67d80 con 0x5653e1fe8df0 2023-07-29T16:22:15.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.998+0000 7f3123fff700 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 0x5653e1fe97b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3110011ed0 tx=0x7f3110014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e753c655bcb8d4a2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.998+0000 7f310effd700 1 -- 172.21.15.31:0/1094429445 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110012dc0 con 0x5653e1fe8df0 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.998+0000 7f310effd700 1 -- 172.21.15.31:0/1094429445 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3110016070 con 0x5653e1fe8df0 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.998+0000 7f310effd700 1 -- 172.21.15.31:0/1094429445 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311001d5f0 con 0x5653e1fe8df0 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 -- 172.21.15.31:0/1094429445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 msgr2=0x5653e1fe97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 0x5653e1fe97b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f3110011ed0 tx=0x7f3110014000 comp rx=0 tx=0).stop 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 -- 172.21.15.31:0/1094429445 shutdown_connections 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1fea160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fe88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 --2- 172.21.15.31:0/1094429445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe8df0 0x5653e1fe97b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 -- 172.21.15.31:0/1094429445 >> 172.21.15.31:0/1094429445 conn(0x5653e1fddfd0 msgr2=0x5653e1fe3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 -- 172.21.15.31:0/1094429445 shutdown_connections 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 -- 172.21.15.31:0/1094429445 wait complete. 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:15.999+0000 7f313e84db80 1 -- start start 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 Processor -- start 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 0x5653e1ff1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ed64b0 con 0x5653e1e446b0 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ed6150 con 0x5653e1fe9cf0 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1dbd8c0 con 0x5653e1fe8df0 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f313e84db80 10 client.0 _ll_register_callbacks cb 0x5653e1edf5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 0x5653e1ff1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.000+0000 7f312959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 0x5653e1ff1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54770/0 (socket says 172.21.15.31:54770) 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 -- 172.21.15.31:0/1879744982 learned_addr learned my addr 172.21.15.31:0/1879744982 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3123fff700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 -- 172.21.15.31:0/1879744982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 msgr2=0x5653e1ff2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 -- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 msgr2=0x5653e1ff94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 -- 172.21.15.31:0/1879744982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3110011c60 con 0x5653e1e446b0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f312959c700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3128d9b700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 0x5653e1ff1f20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f31180a2570 tx=0x7f3118022ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ab8295cf68faf1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3103fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3103fff700 1 -- 172.21.15.31:0/1879744982 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118099dd0 con 0x5653e1e446b0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.001+0000 7f3103fff700 1 -- 172.21.15.31:0/1879744982 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f311809a6e0 con 0x5653e1e446b0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5653e1f64cb0 con 0x5653e1e446b0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f3103fff700 1 -- 172.21.15.31:0/1879744982 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f311809e710 con 0x5653e1e446b0 2023-07-29T16:22:16.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 10 client.17517 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5653e1d50c00 con 0x5653e1e446b0 2023-07-29T16:22:16.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f3103fff700 1 -- 172.21.15.31:0/1879744982 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5653e1f64cb0 con 0x5653e1e446b0 2023-07-29T16:22:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f3103fff700 1 -- 172.21.15.31:0/1879744982 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f311800fb10 con 0x5653e1e446b0 2023-07-29T16:22:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 10 client.17517 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 10 client.17517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:16.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 20 client.17517 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.002+0000 7f313e84db80 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e1ffd880 0x5653e1fffc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.003+0000 7f313e84db80 4 client.17517 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.003+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5653e1fb7d10 con 0x5653e1ffd880 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.003+0000 7f3123fff700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e1ffd880 0x5653e1fffc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.003+0000 7f3103fff700 10 client.17517 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.003+0000 7f3123fff700 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e1ffd880 0x5653e1fffc90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ad062e062d7e0bd server_cookie=af0db1851bcf2032 in_seq=0 out_seq=0 2023-07-29T16:22:16.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f3103fff700 1 -- 172.21.15.31:0/1879744982 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5653e1fb7d10 con 0x5653e1ffd880 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f3103fff700 10 client.17517 handle_command_reply: tid=0 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 client.17517 shutdown 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 msgr2=0x5653e1ff1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 0x5653e1ff1f20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f31180a2570 tx=0x7f3118022ba0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 shutdown_connections 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e1ffd880 0x5653e1fffc90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1e446b0 0x5653e1ff1f20 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f3103fff700 0 client.17517 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 --2- 172.21.15.31:0/1879744982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 >> 172.21.15.31:0/1879744982 conn(0x5653e1fddfd0 msgr2=0x5653e1fe2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.004+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 shutdown_connections 2023-07-29T16:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.005+0000 7f313e84db80 1 -- 172.21.15.31:0/1879744982 wait complete. 2023-07-29T16:22:16.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.005+0000 7f313e84db80 20 client.17517 trim_cache size 0 max 16384 2023-07-29T16:22:16.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.020+0000 7f313e84db80 1 Processor -- start 2023-07-29T16:22:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.020+0000 7f313e84db80 1 -- start start 2023-07-29T16:22:16.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.020+0000 7f313e84db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.020+0000 7f313e84db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe8df0 0x5653e1ff9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.020+0000 7f313e84db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1fed140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f313e84db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ff2c80 con 0x5653e1fe8df0 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f313e84db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ff2f60 con 0x5653e1fe9cf0 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f313e84db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ff13d0 con 0x5653e1e446b0 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f312959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1fed140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3123fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe8df0 0x5653e1ff9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fecc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55542/0 (socket says 172.21.15.31:55542) 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 -- 172.21.15.31:0/3128772846 learned_addr learned my addr 172.21.15.31:0/3128772846 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f312959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1fed140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58578/0 (socket says 172.21.15.31:58578) 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 -- 172.21.15.31:0/3128772846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 msgr2=0x5653e1fed140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1fed140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 -- 172.21.15.31:0/3128772846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe8df0 msgr2=0x5653e1ff9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe8df0 0x5653e1ff9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.021+0000 7f3128d9b700 1 -- 172.21.15.31:0/3128772846 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653e1d8e1f0 con 0x5653e1e446b0 2023-07-29T16:22:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.022+0000 7f3128d9b700 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fecc00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3118098a30 tx=0x7f3118098c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f459848a731743 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.022+0000 7f310f7fe700 1 -- 172.21.15.31:0/3128772846 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311809e970 con 0x5653e1e446b0 2023-07-29T16:22:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.022+0000 7f310f7fe700 1 -- 172.21.15.31:0/3128772846 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31180c4e90 con 0x5653e1e446b0 2023-07-29T16:22:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.022+0000 7f310f7fe700 1 -- 172.21.15.31:0/3128772846 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180a2870 con 0x5653e1e446b0 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.022+0000 7f313e84db80 1 -- 172.21.15.31:0/3128772846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 msgr2=0x5653e1fecc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.022+0000 7f313e84db80 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fecc00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3118098a30 tx=0x7f3118098c90 comp rx=0 tx=0).stop 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 -- 172.21.15.31:0/3128772846 shutdown_connections 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1fe9cf0 0x5653e1fed140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1e446b0 0x5653e1fecc00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 --2- 172.21.15.31:0/3128772846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe8df0 0x5653e1ff9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 -- 172.21.15.31:0/3128772846 >> 172.21.15.31:0/3128772846 conn(0x5653e1fddfd0 msgr2=0x5653e1ff00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 -- 172.21.15.31:0/3128772846 shutdown_connections 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 -- 172.21.15.31:0/3128772846 wait complete. 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 -- start start 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.023+0000 7f313e84db80 1 Processor -- start 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 0x5653e1ff5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ff8d60 con 0x5653e1fe9cf0 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ff8ea0 con 0x5653e1e446b0 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653e1ff9060 con 0x5653e1fe8df0 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f313e84db80 10 client.0 _ll_register_callbacks cb 0x5653e1edf5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 0x5653e1ff5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 0x5653e1ff5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58582/0 (socket says 172.21.15.31:58582) 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 -- 172.21.15.31:0/2908723199 learned_addr learned my addr 172.21.15.31:0/2908723199 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f312959c700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3123fff700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 -- 172.21.15.31:0/2908723199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 msgr2=0x5653e1ff5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 -- 172.21.15.31:0/2908723199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 msgr2=0x5653e1ff8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.024+0000 7f3128d9b700 1 -- 172.21.15.31:0/2908723199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653e1d50c00 con 0x5653e1e446b0 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3128d9b700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 0x5653e1ff5690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3118096be0 tx=0x7f31180a28c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c2443edd1711ee server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3103fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3103fff700 1 -- 172.21.15.31:0/2908723199 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31180c4c50 con 0x5653e1e446b0 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3103fff700 1 -- 172.21.15.31:0/2908723199 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f311809a6e0 con 0x5653e1e446b0 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3110016070 con 0x5653e1e446b0 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3103fff700 1 -- 172.21.15.31:0/2908723199 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f311809bb40 con 0x5653e1e446b0 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f313e84db80 10 client.27551 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5653e1d66250 con 0x5653e1e446b0 2023-07-29T16:22:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3103fff700 1 -- 172.21.15.31:0/2908723199 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3110016070 con 0x5653e1e446b0 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.025+0000 7f3103fff700 1 -- 172.21.15.31:0/2908723199 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3118099840 con 0x5653e1e446b0 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f313e84db80 10 client.27551 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f313e84db80 10 client.27551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f313e84db80 20 client.27551 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f313e84db80 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e204fce0 0x5653e20520c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f313e84db80 4 client.27551 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5653e1f32ce0 con 0x5653e204fce0 2023-07-29T16:22:16.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f3123fff700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e204fce0 0x5653e20520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f3103fff700 10 client.27551 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:16.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.026+0000 7f3123fff700 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e204fce0 0x5653e20520c0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8944e6e37013809c server_cookie=cf72e652c3167246 in_seq=0 out_seq=0 2023-07-29T16:22:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.096+0000 7f3103fff700 1 -- 172.21.15.31:0/2908723199 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5653e1f32ce0 con 0x5653e204fce0 2023-07-29T16:22:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.096+0000 7f3103fff700 10 client.27551 handle_command_reply: tid=0 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.096+0000 7f313e84db80 1 client.27551 shutdown 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.096+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 msgr2=0x5653e1ff5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.096+0000 7f313e84db80 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 0x5653e1ff5690 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f3118096be0 tx=0x7f31180a28c0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 shutdown_connections 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5653e204fce0 0x5653e20520c0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5653e1fe9cf0 0x5653e1ff8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653e1e446b0 0x5653e1ff5690 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f3103fff700 0 client.27551 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 --2- 172.21.15.31:0/2908723199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5653e1fe8df0 0x5653e1ff5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 >> 172.21.15.31:0/2908723199 conn(0x5653e1fddfd0 msgr2=0x5653e1ff00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 shutdown_connections 2023-07-29T16:22:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 1 -- 172.21.15.31:0/2908723199 wait complete. 2023-07-29T16:22:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.097+0000 7f313e84db80 20 client.27551 trim_cache size 0 max 16384 2023-07-29T16:22:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:22:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:22:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:22:16.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:22:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31140a0810 msgr2=0x7f31140a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31140a0810 0x7f31140a2cc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f31240ff600 tx=0x7f3110009260 comp rx=0 tx=0).stop 2023-07-29T16:22:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 msgr2=0x7f31241983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31241983c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f311800b680 tx=0x7f3118005740 comp rx=0 tx=0).stop 2023-07-29T16:22:16.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 shutdown_connections 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31140a0810 0x7f31140a2cc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3124105fa0 0x7f3124199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31240fe0b0 0x7f31241983c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 --2- 172.21.15.31:0/914381669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31240fea00 0x7f3124198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 >> 172.21.15.31:0/914381669 conn(0x7f31240f9b70 msgr2=0x7f31240fbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 shutdown_connections 2023-07-29T16:22:16.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:16.098+0000 7f312afff700 1 -- 172.21.15.31:0/914381669 wait complete. 2023-07-29T16:22:16.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:16 smithi039 ceph-mon[128858]: pgmap v574: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 2.5 MiB/s wr, 237 op/s 2023-07-29T16:22:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:16 smithi156 ceph-mon[129245]: pgmap v574: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 2.5 MiB/s wr, 237 op/s 2023-07-29T16:22:16.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 405201 8.97 MB/sec execute 121 sec latency 41.664 ms 2023-07-29T16:22:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:16 smithi031 ceph-mon[95504]: pgmap v574: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 2.5 MiB/s wr, 237 op/s 2023-07-29T16:22:17.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 407367 8.99 MB/sec execute 122 sec latency 38.503 ms 2023-07-29T16:22:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:17 smithi031 ceph-mon[95504]: pgmap v575: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 2.2 MiB/s wr, 235 op/s 2023-07-29T16:22:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:17 smithi039 ceph-mon[128858]: pgmap v575: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 2.2 MiB/s wr, 235 op/s 2023-07-29T16:22:18.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:17 smithi156 ceph-mon[129245]: pgmap v575: 97 pgs: 97 active+clean; 93 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 2.2 MiB/s wr, 235 op/s 2023-07-29T16:22:18.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 408348 8.97 MB/sec execute 123 sec latency 126.997 ms 2023-07-29T16:22:19.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 410284 8.96 MB/sec execute 124 sec latency 12.983 ms 2023-07-29T16:22:20.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:20 smithi039 ceph-mon[128858]: pgmap v576: 97 pgs: 97 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2023-07-29T16:22:20.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:20 smithi156 ceph-mon[129245]: pgmap v576: 97 pgs: 97 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2023-07-29T16:22:20.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 411862 8.98 MB/sec execute 125 sec latency 13.605 ms 2023-07-29T16:22:20.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:20 smithi031 ceph-mon[95504]: pgmap v576: 97 pgs: 97 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 315 op/s 2023-07-29T16:22:21.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 413140 8.95 MB/sec execute 126 sec latency 40.670 ms 2023-07-29T16:22:22.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:22:22.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 -- 172.21.15.31:0/266814328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f50074d10 msgr2=0x7f8f50073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 --2- 172.21.15.31:0/266814328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f50074d10 0x7f8f50073160 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8f400098f0 tx=0x7f8f40009c00 comp rx=0 tx=0).stop 2023-07-29T16:22:22.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 -- 172.21.15.31:0/266814328 shutdown_connections 2023-07-29T16:22:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 --2- 172.21.15.31:0/266814328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f501081e0 0x7f8f5010a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 --2- 172.21.15.31:0/266814328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f50074d10 0x7f8f50073160 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 --2- 172.21.15.31:0/266814328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50073730 0x7f8f50073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 -- 172.21.15.31:0/266814328 >> 172.21.15.31:0/266814328 conn(0x7f8f500fb970 msgr2=0x7f8f500fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.266+0000 7f8f54fff700 1 -- 172.21.15.31:0/266814328 shutdown_connections 2023-07-29T16:22:22.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.267+0000 7f8f54fff700 1 -- 172.21.15.31:0/266814328 wait complete. 2023-07-29T16:22:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.267+0000 7f8f54fff700 1 Processor -- start 2023-07-29T16:22:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 -- start start 2023-07-29T16:22:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 0x7f8f50198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50074d10 0x7f8f50198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501081e0 0x7f8f5019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5019fe90 con 0x7f8f501081e0 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5019ffd0 con 0x7f8f50073730 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f54fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f501a0160 con 0x7f8f50074d10 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 0x7f8f50198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f4dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50074d10 0x7f8f50198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 0x7f8f50198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44424/0 (socket says 172.21.15.31:44424) 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f4e59c700 1 -- 172.21.15.31:0/2019947697 learned_addr learned my addr 172.21.15.31:0/2019947697 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.268+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501081e0 0x7f8f5019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2019947697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 msgr2=0x7f8f50198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 0x7f8f50198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501081e0 msgr2=0x7f8f5019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501081e0 0x7f8f5019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2019947697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f48009d00 con 0x7f8f50074d10 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 0x7f8f50198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501081e0 0x7f8f5019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50074d10 0x7f8f50198850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400e9f0 tx=0x7f8f4400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f3f7fe700 1 -- 172.21.15.31:0/2019947697 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4400eee0 con 0x7f8f50074d10 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f3f7fe700 1 -- 172.21.15.31:0/2019947697 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f4400cbd0 con 0x7f8f50074d10 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f40009520 con 0x7f8f50074d10 2023-07-29T16:22:22.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f3f7fe700 1 -- 172.21.15.31:0/2019947697 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44018680 con 0x7f8f50074d10 2023-07-29T16:22:22.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.269+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f501a07c0 con 0x7f8f50074d10 2023-07-29T16:22:22.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.273+0000 7f8f3f7fe700 1 -- 172.21.15.31:0/2019947697 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8f4400cd40 con 0x7f8f50074d10 2023-07-29T16:22:22.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.273+0000 7f8f689ceb80 1 Processor -- start 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 -- start start 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb32fbe0 0x564abb32fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb330d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 0x564abb331740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb287a10 con 0x564abb330530 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb2853d0 con 0x564abb3312f0 2023-07-29T16:22:22.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb280cc0 con 0x564abb32fbe0 2023-07-29T16:22:22.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f3f7fe700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f380a0790 0x7f8f380a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f3f7fe700 1 -- 172.21.15.31:0/2019947697 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f44014070 con 0x7f8f50074d10 2023-07-29T16:22:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f4dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 0x564abb331740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f4dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 0x564abb331740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44436/0 (socket says 172.21.15.31:44436) 2023-07-29T16:22:22.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2488381169 learned_addr learned my addr 172.21.15.31:0/2488381169 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb32fbe0 0x564abb32fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb330d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.274+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f380a0790 0x7f8f380a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2488381169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb32fbe0 msgr2=0x564abb32fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb32fbe0 0x564abb32fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2488381169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 msgr2=0x564abb330d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb330d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f4dd9b700 1 -- 172.21.15.31:0/2488381169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564abb0aede0 con 0x564abb3312f0 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 0x564abb331740 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8f44095f10 tx=0x7f8f440958e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d988b0bbd3f6b9f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f3cff9700 1 -- 172.21.15.31:0/2488381169 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4409e5b0 con 0x564abb3312f0 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f3cff9700 1 -- 172.21.15.31:0/2488381169 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f440994a0 con 0x564abb3312f0 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.275+0000 7f8f3cff9700 1 -- 172.21.15.31:0/2488381169 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440a16a0 con 0x564abb3312f0 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 -- 172.21.15.31:0/2488381169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 msgr2=0x564abb331740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f380a0790 0x7f8f380a2c40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x564abb32f980 tx=0x7f8f4800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 0x564abb331740 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8f44095f10 tx=0x7f8f440958e0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 -- 172.21.15.31:0/2488381169 shutdown_connections 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb3312f0 0x564abb331740 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb32fbe0 0x564abb32fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/2488381169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb330d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 -- 172.21.15.31:0/2488381169 >> 172.21.15.31:0/2488381169 conn(0x564abb3253e0 msgr2=0x564abb32a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 -- 172.21.15.31:0/2488381169 shutdown_connections 2023-07-29T16:22:22.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 -- 172.21.15.31:0/2488381169 wait complete. 2023-07-29T16:22:22.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.276+0000 7f8f689ceb80 1 -- start start 2023-07-29T16:22:22.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 Processor -- start 2023-07-29T16:22:22.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb3397f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb339d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb340d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb21d510 con 0x564abb330530 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb21d1b0 con 0x564abb32fbe0 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb104920 con 0x564abb3312f0 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f689ceb80 10 client.0 _ll_register_callbacks cb 0x564abb226620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb3397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb3397f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44438/0 (socket says 172.21.15.31:44438) 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f4e59c700 1 -- 172.21.15.31:0/934343906 learned_addr learned my addr 172.21.15.31:0/934343906 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb339d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.277+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb340d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4e59c700 1 -- 172.21.15.31:0/934343906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 msgr2=0x564abb340d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb340d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4e59c700 1 -- 172.21.15.31:0/934343906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 msgr2=0x564abb339d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb339d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4e59c700 1 -- 172.21.15.31:0/934343906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f501a07c0 con 0x564abb32fbe0 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb339d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb3397f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009fd0 tx=0x7f8f4000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1456643c1671eaf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f35ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f35ffb700 1 -- 172.21.15.31:0/934343906 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4001c040 con 0x564abb32fbe0 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564abb2abd10 con 0x564abb32fbe0 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.278+0000 7f8f35ffb700 1 -- 172.21.15.31:0/934343906 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f40005240 con 0x564abb32fbe0 2023-07-29T16:22:22.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f35ffb700 1 -- 172.21.15.31:0/934343906 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f40005400 con 0x564abb32fbe0 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 10 client.27563 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564abb097c60 con 0x564abb32fbe0 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f35ffb700 1 -- 172.21.15.31:0/934343906 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564abb2abd10 con 0x564abb32fbe0 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f35ffb700 1 -- 172.21.15.31:0/934343906 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8f40015020 con 0x564abb32fbe0 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 10 client.27563 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 10 client.27563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:22.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 20 client.27563 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb345600 0x564abb347a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 4 client.27563 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.279+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564abb2fed70 con 0x564abb345600 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.280+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb345600 0x564abb347a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.280+0000 7f8f35ffb700 10 client.27563 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.280+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb345600 0x564abb347a10 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9f52b51760caa1e server_cookie=605f7e71b9504c68 in_seq=0 out_seq=0 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.281+0000 7f8f35ffb700 1 -- 172.21.15.31:0/934343906 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564abb2fed70 con 0x564abb345600 2023-07-29T16:22:22.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.281+0000 7f8f35ffb700 10 client.27563 handle_command_reply: tid=0 2023-07-29T16:22:22.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.281+0000 7f8f689ceb80 1 client.27563 shutdown 2023-07-29T16:22:22.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.281+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 msgr2=0x564abb3397f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.281+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb3397f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009fd0 tx=0x7f8f4000b840 comp rx=0 tx=0).stop 2023-07-29T16:22:22.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 shutdown_connections 2023-07-29T16:22:22.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb345600 0x564abb347a10 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb340d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f35ffb700 0 client.27563 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb3397f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/934343906 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb339d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 >> 172.21.15.31:0/934343906 conn(0x564abb3253e0 msgr2=0x564abb32a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 shutdown_connections 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 1 -- 172.21.15.31:0/934343906 wait complete. 2023-07-29T16:22:22.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.282+0000 7f8f689ceb80 20 client.27563 trim_cache size 0 max 16384 2023-07-29T16:22:22.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.299+0000 7f8f689ceb80 1 Processor -- start 2023-07-29T16:22:22.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.299+0000 7f8f689ceb80 1 -- start start 2023-07-29T16:22:22.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.299+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb330530 0x564abb3341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb3312f0 0x564abb3346f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb337d80 con 0x564abb3312f0 2023-07-29T16:22:22.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb337420 con 0x564abb32fbe0 2023-07-29T16:22:22.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb345320 con 0x564abb330530 2023-07-29T16:22:22.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44454/0 (socket says 172.21.15.31:44454) 2023-07-29T16:22:22.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 -- 172.21.15.31:0/3436970068 learned_addr learned my addr 172.21.15.31:0/3436970068 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:22.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb3312f0 0x564abb3346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb330530 0x564abb3341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 -- 172.21.15.31:0/3436970068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb330530 msgr2=0x564abb3341b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb330530 0x564abb3341b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 -- 172.21.15.31:0/3436970068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb3312f0 msgr2=0x564abb3346f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb3312f0 0x564abb3346f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 -- 172.21.15.31:0/3436970068 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564abb0d5250 con 0x564abb32fbe0 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb330530 0x564abb3341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.300+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ed20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8f40003d90 tx=0x7f8f4000b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a33d8275a9970eb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f3d7fa700 1 -- 172.21.15.31:0/3436970068 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4001c040 con 0x564abb32fbe0 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f3d7fa700 1 -- 172.21.15.31:0/3436970068 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f40022570 con 0x564abb32fbe0 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f3d7fa700 1 -- 172.21.15.31:0/3436970068 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f400227b0 con 0x564abb32fbe0 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 -- 172.21.15.31:0/3436970068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 msgr2=0x564abb33ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ed20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8f40003d90 tx=0x7f8f4000b2c0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 -- 172.21.15.31:0/3436970068 shutdown_connections 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb3312f0 0x564abb3346f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ed20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/3436970068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb330530 0x564abb3341b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 -- 172.21.15.31:0/3436970068 >> 172.21.15.31:0/3436970068 conn(0x564abb3253e0 msgr2=0x564abb337670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.301+0000 7f8f689ceb80 1 -- 172.21.15.31:0/3436970068 shutdown_connections 2023-07-29T16:22:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.302+0000 7f8f689ceb80 1 -- 172.21.15.31:0/3436970068 wait complete. 2023-07-29T16:22:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.302+0000 7f8f689ceb80 1 -- start start 2023-07-29T16:22:22.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.302+0000 7f8f689ceb80 1 Processor -- start 2023-07-29T16:22:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb33d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb33d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb33da90 con 0x564abb330530 2023-07-29T16:22:22.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb33dc00 con 0x564abb32fbe0 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb33ddc0 con 0x564abb3312f0 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f689ceb80 10 client.0 _ll_register_callbacks cb 0x564abb226620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44456/0 (socket says 172.21.15.31:44456) 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 -- 172.21.15.31:0/203183346 learned_addr learned my addr 172.21.15.31:0/203183346 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4ed9d700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb33d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb33d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 -- 172.21.15.31:0/203183346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 msgr2=0x564abb33d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb33d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 -- 172.21.15.31:0/203183346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 msgr2=0x564abb33d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.303+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb33d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f4e59c700 1 -- 172.21.15.31:0/203183346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564abb097c60 con 0x564abb32fbe0 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f4e59c700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ea00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009c40 tx=0x7f8f4000bc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa44d1d0ef455542 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f35ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f35ffb700 1 -- 172.21.15.31:0/203183346 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4001c040 con 0x564abb32fbe0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f35ffb700 1 -- 172.21.15.31:0/203183346 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f400227b0 con 0x564abb32fbe0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f440994a0 con 0x564abb32fbe0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f35ffb700 1 -- 172.21.15.31:0/203183346 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f40003bd0 con 0x564abb32fbe0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f689ceb80 10 client.27569 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.304+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564abb0ad2b0 con 0x564abb32fbe0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f35ffb700 1 -- 172.21.15.31:0/203183346 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f440994a0 con 0x564abb32fbe0 2023-07-29T16:22:22.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f35ffb700 1 -- 172.21.15.31:0/203183346 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8f40009c70 con 0x564abb32fbe0 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f689ceb80 10 client.27569 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f689ceb80 10 client.27569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f689ceb80 20 client.27569 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb3983a0 0x564abb39a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f689ceb80 4 client.27569 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564abb279d40 con 0x564abb3983a0 2023-07-29T16:22:22.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb3983a0 0x564abb39a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f4dd9b700 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb3983a0 0x564abb39a780 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=471e45130c1a000a server_cookie=87a609d1ad0761be in_seq=0 out_seq=0 2023-07-29T16:22:22.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.305+0000 7f8f35ffb700 10 client.27569 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:22.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f35ffb700 1 -- 172.21.15.31:0/203183346 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564abb279d40 con 0x564abb3983a0 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f35ffb700 10 client.27569 handle_command_reply: tid=0 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 client.27569 shutdown 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 msgr2=0x564abb33ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ea00 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009c40 tx=0x7f8f4000bc70 comp rx=0 tx=0).stop 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 shutdown_connections 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564abb3983a0 0x564abb39a780 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564abb3312f0 0x564abb33d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564abb32fbe0 0x564abb33ea00 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 --2- 172.21.15.31:0/203183346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564abb330530 0x564abb33d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f35ffb700 0 client.27569 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 >> 172.21.15.31:0/203183346 conn(0x564abb3253e0 msgr2=0x564abb3341b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 shutdown_connections 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.308+0000 7f8f689ceb80 1 -- 172.21.15.31:0/203183346 wait complete. 2023-07-29T16:22:22.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f689ceb80 20 client.27569 trim_cache size 0 max 16384 2023-07-29T16:22:22.315 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:22:22.315 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:22:22.315 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:22:22.315 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:22:22.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f380a0790 msgr2=0x7f8f380a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f380a0790 0x7f8f380a2c40 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x564abb32f980 tx=0x7f8f4800b040 comp rx=0 tx=0).stop 2023-07-29T16:22:22.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50074d10 msgr2=0x7f8f50198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50074d10 0x7f8f50198850 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400e9f0 tx=0x7f8f4400ed00 comp rx=0 tx=0).stop 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 shutdown_connections 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f380a0790 0x7f8f380a2c40 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f501081e0 0x7f8f5019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f50073730 0x7f8f50198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 --2- 172.21.15.31:0/2019947697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f50074d10 0x7f8f50198850 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 >> 172.21.15.31:0/2019947697 conn(0x7f8f500fb970 msgr2=0x7f8f500fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 shutdown_connections 2023-07-29T16:22:22.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.309+0000 7f8f54fff700 1 -- 172.21.15.31:0/2019947697 wait complete. 2023-07-29T16:22:22.327 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:22:22.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:22:22.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 -- 172.21.15.31:0/3428542251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc100340 msgr2=0x7f38fc100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 --2- 172.21.15.31:0/3428542251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc100340 0x7f38fc100750 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009990 tx=0x7f38ec009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 -- 172.21.15.31:0/3428542251 shutdown_connections 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 --2- 172.21.15.31:0/3428542251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38fc101ec0 0x7f38fc108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 --2- 172.21.15.31:0/3428542251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc100340 0x7f38fc100750 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 --2- 172.21.15.31:0/3428542251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc101530 0x7f38fc101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 -- 172.21.15.31:0/3428542251 >> 172.21.15.31:0/3428542251 conn(0x7f38fc0fb920 msgr2=0x7f38fc0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 -- 172.21.15.31:0/3428542251 shutdown_connections 2023-07-29T16:22:22.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.541+0000 7f39023ff700 1 -- 172.21.15.31:0/3428542251 wait complete. 2023-07-29T16:22:22.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 Processor -- start 2023-07-29T16:22:22.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 -- start start 2023-07-29T16:22:22.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 0x7f38fc1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc101530 0x7f38fc198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38fc101ec0 0x7f38fc19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc19fe40 con 0x7f38fc101ec0 2023-07-29T16:22:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc19ff80 con 0x7f38fc101530 2023-07-29T16:22:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f39023ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc1a0110 con 0x7f38fc100340 2023-07-29T16:22:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f38fbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 0x7f38fc1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f38fb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc101530 0x7f38fc198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f38fbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 0x7f38fc1982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54508/0 (socket says 172.21.15.31:54508) 2023-07-29T16:22:22.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.542+0000 7f38fbfff700 1 -- 172.21.15.31:0/2584965139 learned_addr learned my addr 172.21.15.31:0/2584965139 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f390099c700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38fc101ec0 0x7f38fc19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fb7fe700 1 -- 172.21.15.31:0/2584965139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 msgr2=0x7f38fc1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fb7fe700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 0x7f38fc1982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fb7fe700 1 -- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38fc101ec0 msgr2=0x7f38fc19f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fb7fe700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38fc101ec0 0x7f38fc19f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fb7fe700 1 -- 172.21.15.31:0/2584965139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec0095f0 con 0x7f38fc101530 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fbfff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 0x7f38fc1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38fb7fe700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc101530 0x7f38fc198800 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f38f000e9f0 tx=0x7f38f000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.543+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f000c9c0 con 0x7f38fc101530 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.544+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38f0010430 con 0x7f38fc101530 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.544+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc1a03f0 con 0x7f38fc101530 2023-07-29T16:22:22.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.544+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0018680 con 0x7f38fc101530 2023-07-29T16:22:22.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.544+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc1a0900 con 0x7f38fc101530 2023-07-29T16:22:22.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.545+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc192a20 con 0x7f38fc101530 2023-07-29T16:22:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.547+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f38f0010860 con 0x7f38fc101530 2023-07-29T16:22:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.548+0000 7f38f97fa700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38e40a07e0 0x7f38e40a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.548+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f38f0014070 con 0x7f38fc101530 2023-07-29T16:22:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.548+0000 7f38fbfff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38e40a07e0 0x7f38e40a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:22.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.549+0000 7f38fbfff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38e40a07e0 0x7f38e40a2c90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00b2e0 tx=0x7f38ec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:22.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.550+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f38f00957a0 con 0x7f38fc101530 2023-07-29T16:22:22.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:22 smithi031 ceph-mon[95504]: pgmap v577: 97 pgs: 97 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.0 MiB/s wr, 224 op/s 2023-07-29T16:22:22.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:22 smithi039 ceph-mon[128858]: pgmap v577: 97 pgs: 97 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.0 MiB/s wr, 224 op/s 2023-07-29T16:22:22.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:22 smithi156 ceph-mon[129245]: pgmap v577: 97 pgs: 97 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.0 MiB/s wr, 224 op/s 2023-07-29T16:22:22.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.834+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38fc02cc30 con 0x7f38fc101530 2023-07-29T16:22:22.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.835+0000 7f38f97fa700 1 -- 172.21.15.31:0/2584965139 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f38f0098dc0 con 0x7f38fc101530 2023-07-29T16:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:22:22.835 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:22:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38e40a07e0 msgr2=0x7f38e40a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38e40a07e0 0x7f38e40a2c90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00b2e0 tx=0x7f38ec005fb0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc101530 msgr2=0x7f38fc198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:22.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc101530 0x7f38fc198800 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f38f000e9f0 tx=0x7f38f000ed00 comp rx=0 tx=0).stop 2023-07-29T16:22:22.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 shutdown_connections 2023-07-29T16:22:22.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f38e40a07e0 0x7f38e40a2c90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38fc101ec0 0x7f38fc19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38fc100340 0x7f38fc1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 --2- 172.21.15.31:0/2584965139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38fc101530 0x7f38fc198800 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:22.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 >> 172.21.15.31:0/2584965139 conn(0x7f38fc0fb920 msgr2=0x7f38fc106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:22.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 shutdown_connections 2023-07-29T16:22:22.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:22.838+0000 7f39023ff700 1 -- 172.21.15.31:0/2584965139 wait complete. 2023-07-29T16:22:22.839 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:22:22.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:22:22.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 415145 8.98 MB/sec execute 127 sec latency 57.334 ms 2023-07-29T16:22:23.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 -- 172.21.15.31:0/639517176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0073750 msgr2=0x7efef0073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 --2- 172.21.15.31:0/639517176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0073750 0x7efef0073bc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efee4009990 tx=0x7efee4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 -- 172.21.15.31:0/639517176 shutdown_connections 2023-07-29T16:22:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 --2- 172.21.15.31:0/639517176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef010a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 --2- 172.21.15.31:0/639517176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef0074d30 0x7efef0073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 --2- 172.21.15.31:0/639517176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0073750 0x7efef0073bc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 -- 172.21.15.31:0/639517176 >> 172.21.15.31:0/639517176 conn(0x7efef00fb970 msgr2=0x7efef00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 -- 172.21.15.31:0/639517176 shutdown_connections 2023-07-29T16:22:23.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.087+0000 7efef59ff700 1 -- 172.21.15.31:0/639517176 wait complete. 2023-07-29T16:22:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.088+0000 7efef59ff700 1 Processor -- start 2023-07-29T16:22:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.088+0000 7efef59ff700 1 -- start start 2023-07-29T16:22:23.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efef59ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef0073750 0x7efef01981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efef59ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0074d30 0x7efef0198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efef59ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efef59ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef019fd70 con 0x7efef0073750 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efef59ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef019feb0 con 0x7efef0074d30 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efef59ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef01a0040 con 0x7efef01081c0 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef0073750 0x7efef01981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef019f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54524/0 (socket says 172.21.15.31:54524) 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 -- 172.21.15.31:0/3192573622 learned_addr learned my addr 172.21.15.31:0/3192573622 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeee7fc700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0074d30 0x7efef0198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 -- 172.21.15.31:0/3192573622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0074d30 msgr2=0x7efef0198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0074d30 0x7efef0198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 -- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef0073750 msgr2=0x7efef01981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef0073750 0x7efef01981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeef7fe700 1 -- 172.21.15.31:0/3192573622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee40095f0 con 0x7efef01081c0 2023-07-29T16:22:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.089+0000 7efeee7fc700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0074d30 0x7efef0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.090+0000 7efeef7fe700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef019f7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efee800ef30 tx=0x7efee800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.090+0000 7efef49fd700 1 -- 172.21.15.31:0/3192573622 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8018070 con 0x7efef01081c0 2023-07-29T16:22:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.090+0000 7efef49fd700 1 -- 172.21.15.31:0/3192573622 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee800f040 con 0x7efef01081c0 2023-07-29T16:22:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.090+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef01a02c0 con 0x7efef01081c0 2023-07-29T16:22:23.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.090+0000 7efef49fd700 1 -- 172.21.15.31:0/3192573622 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee800cd00 con 0x7efef01081c0 2023-07-29T16:22:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.090+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef01a07d0 con 0x7efef01081c0 2023-07-29T16:22:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.093+0000 7efef49fd700 1 -- 172.21.15.31:0/3192573622 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7efee8003a00 con 0x7efef01081c0 2023-07-29T16:22:23.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.094+0000 7eff092e0b80 1 Processor -- start 2023-07-29T16:22:23.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.094+0000 7eff092e0b80 1 -- start start 2023-07-29T16:22:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 0x5616ce94eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94f530 0x5616ce94fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce950740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7eff092e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce8a6a10 con 0x5616ce94f530 2023-07-29T16:22:23.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7eff092e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce8a43d0 con 0x5616ce94ebe0 2023-07-29T16:22:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7eff092e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce89fcc0 con 0x5616ce9502f0 2023-07-29T16:22:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efeee7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 0x5616ce94eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efeee7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 0x5616ce94eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44484/0 (socket says 172.21.15.31:44484) 2023-07-29T16:22:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efeee7fc700 1 -- 172.21.15.31:0/2545087411 learned_addr learned my addr 172.21.15.31:0/2545087411 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94f530 0x5616ce94fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efef49fd700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efed80a0790 0x7efed80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efeee7fc700 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce950740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.095+0000 7efef49fd700 1 -- 172.21.15.31:0/3192573622 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efee80cb620 con 0x7efef01081c0 2023-07-29T16:22:23.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeef7fe700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efed80a0790 0x7efed80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeeeffd700 1 -- 172.21.15.31:0/2545087411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 msgr2=0x5616ce950740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce950740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeeeffd700 1 -- 172.21.15.31:0/2545087411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 msgr2=0x5616ce94eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 0x5616ce94eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeeeffd700 1 -- 172.21.15.31:0/2545087411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616ce6cdde0 con 0x5616ce94f530 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeee7fc700 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 0x5616ce94eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.099 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94f530 0x5616ce94fd60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7efee0009cb0 tx=0x7efee0011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f562a50068f7c82 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efeef7fe700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efed80a0790 0x7efed80a2c40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efee8009c10 tx=0x7efee8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efed5ffb700 1 -- 172.21.15.31:0/2545087411 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0011d20 con 0x5616ce94f530 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efed5ffb700 1 -- 172.21.15.31:0/2545087411 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee0011e80 con 0x5616ce94f530 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.096+0000 7efed5ffb700 1 -- 172.21.15.31:0/2545087411 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0005280 con 0x5616ce94f530 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 -- 172.21.15.31:0/2545087411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94f530 msgr2=0x5616ce94fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94f530 0x5616ce94fd60 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7efee0009cb0 tx=0x7efee0011440 comp rx=0 tx=0).stop 2023-07-29T16:22:23.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 -- 172.21.15.31:0/2545087411 shutdown_connections 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce950740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94ebe0 0x5616ce94eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2545087411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94f530 0x5616ce94fd60 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 -- 172.21.15.31:0/2545087411 >> 172.21.15.31:0/2545087411 conn(0x5616ce9443e0 msgr2=0x5616ce949990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 -- 172.21.15.31:0/2545087411 shutdown_connections 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.097+0000 7eff092e0b80 1 -- 172.21.15.31:0/2545087411 wait complete. 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.098+0000 7eff092e0b80 1 -- start start 2023-07-29T16:22:23.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.098+0000 7eff092e0b80 1 Processor -- start 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.098+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce958830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.098+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce958d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.098+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7eff092e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce83c510 con 0x5616ce94ebe0 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7eff092e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce83c1b0 con 0x5616ce94f530 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7eff092e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce723920 con 0x5616ce9502f0 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7eff092e0b80 10 client.0 _ll_register_callbacks cb 0x5616ce845620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:23.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce958d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeee7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce958830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce958d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44490/0 (socket says 172.21.15.31:44490) 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 -- 172.21.15.31:0/1962672706 learned_addr learned my addr 172.21.15.31:0/1962672706 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeee7fc700 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 -- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 msgr2=0x5616ce95fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 -- 172.21.15.31:0/1962672706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 msgr2=0x5616ce958830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce958830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 -- 172.21.15.31:0/1962672706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee4020040 con 0x5616ce94f530 2023-07-29T16:22:23.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeeeffd700 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce958d70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efee0000c00 tx=0x7efee0005ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9847613c8e0137 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.099+0000 7efeceffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7efeceffd700 1 -- 172.21.15.31:0/1962672706 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0004850 con 0x5616ce94f530 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616ce91dd70 con 0x5616ce94f530 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7efeceffd700 1 -- 172.21.15.31:0/1962672706 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efee0004260 con 0x5616ce94f530 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7efeceffd700 1 -- 172.21.15.31:0/1962672706 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efee0004420 con 0x5616ce94f530 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7eff092e0b80 10 client.27587 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:23.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7efee00098e0 con 0x5616ce94f530 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7efeceffd700 1 -- 172.21.15.31:0/1962672706 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616ce91dd70 con 0x5616ce94f530 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.100+0000 7efeceffd700 1 -- 172.21.15.31:0/1962672706 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7efee0003840 con 0x5616ce94f530 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7eff092e0b80 10 client.27587 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7eff092e0b80 10 client.27587 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7eff092e0b80 20 client.27587 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7eff092e0b80 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce964410 0x5616ce966820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7eff092e0b80 4 client.27587 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616ce898d40 con 0x5616ce964410 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7efeee7fc700 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce964410 0x5616ce966820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7efeceffd700 10 client.27587 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.101+0000 7efeee7fc700 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce964410 0x5616ce966820 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e8e08002327030d server_cookie=af7ed686227355f0 in_seq=0 out_seq=0 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7efeceffd700 1 -- 172.21.15.31:0/1962672706 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5616ce898d40 con 0x5616ce964410 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7efeceffd700 10 client.27587 handle_command_reply: tid=0 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 client.27587 shutdown 2023-07-29T16:22:23.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 msgr2=0x5616ce958d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce958d70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efee0000c00 tx=0x7efee0005ce0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 shutdown_connections 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce958830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce964410 0x5616ce966820 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 --2- 172.21.15.31:0/1962672706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce958d70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 >> 172.21.15.31:0/1962672706 conn(0x5616ce9443e0 msgr2=0x5616ce949060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.102+0000 7efeceffd700 0 client.27587 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.103+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 shutdown_connections 2023-07-29T16:22:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.103+0000 7eff092e0b80 1 -- 172.21.15.31:0/1962672706 wait complete. 2023-07-29T16:22:23.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.103+0000 7eff092e0b80 20 client.27587 trim_cache size 0 max 16384 2023-07-29T16:22:23.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 Processor -- start 2023-07-29T16:22:23.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 -- start start 2023-07-29T16:22:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce94a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce949060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce941770 con 0x5616ce94ebe0 2023-07-29T16:22:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce956430 con 0x5616ce94f530 2023-07-29T16:22:23.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7eff092e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce94b080 con 0x5616ce9502f0 2023-07-29T16:22:23.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7efeee7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce94a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44492/0 (socket says 172.21.15.31:44492) 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7efeeeffd700 1 -- 172.21.15.31:0/244194153 learned_addr learned my addr 172.21.15.31:0/244194153 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7efeee7fc700 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce94a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34086/0 (socket says 172.21.15.31:34086) 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.117+0000 7efeee7fc700 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce949060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efeeeffd700 1 -- 172.21.15.31:0/244194153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 msgr2=0x5616ce949060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efeeeffd700 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce949060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efeeeffd700 1 -- 172.21.15.31:0/244194153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 msgr2=0x5616ce94a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efeeeffd700 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce94a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efeeeffd700 1 -- 172.21.15.31:0/244194153 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616ce6b6c60 con 0x5616ce94f530 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efeeeffd700 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95fd70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efee0003c70 tx=0x7efee0011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=517c3b3d203ab0d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efed67fc700 1 -- 172.21.15.31:0/244194153 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0003ee0 con 0x5616ce94f530 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efed67fc700 1 -- 172.21.15.31:0/244194153 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee0004a90 con 0x5616ce94f530 2023-07-29T16:22:23.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.118+0000 7efed67fc700 1 -- 172.21.15.31:0/244194153 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee002b810 con 0x5616ce94f530 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 -- 172.21.15.31:0/244194153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 msgr2=0x5616ce95fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95fd70 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efee0003c70 tx=0x7efee0011440 comp rx=0 tx=0).stop 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 -- 172.21.15.31:0/244194153 shutdown_connections 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce949060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce94a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 --2- 172.21.15.31:0/244194153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95fd70 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 -- 172.21.15.31:0/244194153 >> 172.21.15.31:0/244194153 conn(0x5616ce9443e0 msgr2=0x5616ce9574e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 -- 172.21.15.31:0/244194153 shutdown_connections 2023-07-29T16:22:23.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.119+0000 7eff092e0b80 1 -- 172.21.15.31:0/244194153 wait complete. 2023-07-29T16:22:23.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 -- start start 2023-07-29T16:22:23.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 Processor -- start 2023-07-29T16:22:23.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce95f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce960170 con 0x5616ce94ebe0 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce95ce70 con 0x5616ce94f530 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.120+0000 7eff092e0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616ce95d030 con 0x5616ce9502f0 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7eff092e0b80 10 client.0 _ll_register_callbacks cb 0x5616ce845620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeee7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce95f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeee7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce95f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34098/0 (socket says 172.21.15.31:34098) 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeee7fc700 1 -- 172.21.15.31:0/2313341819 learned_addr learned my addr 172.21.15.31:0/2313341819 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeee7fc700 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 -- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 msgr2=0x5616ce95c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 -- 172.21.15.31:0/2313341819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 msgr2=0x5616ce95f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce95f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 -- 172.21.15.31:0/2313341819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee00098e0 con 0x5616ce94f530 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeeeffd700 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95c3f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efee0009bd0 tx=0x7efee0004540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccd37940ec30b261 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.121+0000 7efeceffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7efeceffd700 1 -- 172.21.15.31:0/2313341819 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0003e90 con 0x5616ce94f530 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616ce8cad10 con 0x5616ce94f530 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7efeceffd700 1 -- 172.21.15.31:0/2313341819 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efee0000e40 con 0x5616ce94f530 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7efeceffd700 1 -- 172.21.15.31:0/2313341819 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efee002b590 con 0x5616ce94f530 2023-07-29T16:22:23.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 10 client.27593 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5616ce6f4250 con 0x5616ce94f530 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7efeceffd700 1 -- 172.21.15.31:0/2313341819 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616ce8cad10 con 0x5616ce94f530 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7efeceffd700 1 -- 172.21.15.31:0/2313341819 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7efee0045070 con 0x5616ce94f530 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 10 client.27593 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 10 client.27593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 20 client.27593 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce9bf9b0 0x5616ce9c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 4 client.27593 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.122+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5616ce898a60 con 0x5616ce9bf9b0 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.123+0000 7efeee7fc700 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce9bf9b0 0x5616ce9c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.123+0000 7efeee7fc700 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce9bf9b0 0x5616ce9c1d90 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d4494ffae128216 server_cookie=1be9883412349af6 in_seq=0 out_seq=0 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.123+0000 7efeceffd700 10 client.27593 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7efeceffd700 1 -- 172.21.15.31:0/2313341819 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5616ce898a60 con 0x5616ce9bf9b0 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7efeceffd700 10 client.27593 handle_command_reply: tid=0 2023-07-29T16:22:23.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 client.27593 shutdown 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 msgr2=0x5616ce95c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95c3f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7efee0009bd0 tx=0x7efee0004540 comp rx=0 tx=0).stop 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 shutdown_connections 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616ce9502f0 0x5616ce95c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616ce94ebe0 0x5616ce95f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5616ce9bf9b0 0x5616ce9c1d90 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 --2- 172.21.15.31:0/2313341819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616ce94f530 0x5616ce95c3f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 >> 172.21.15.31:0/2313341819 conn(0x5616ce9443e0 msgr2=0x5616ce949060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7efeceffd700 0 client.27593 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 shutdown_connections 2023-07-29T16:22:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 1 -- 172.21.15.31:0/2313341819 wait complete. 2023-07-29T16:22:23.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.124+0000 7eff092e0b80 20 client.27593 trim_cache size 0 max 16384 2023-07-29T16:22:23.131 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:22:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efed80a0790 msgr2=0x7efed80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efed80a0790 0x7efed80a2c40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7efee8009c10 tx=0x7efee8000f40 comp rx=0 tx=0).stop 2023-07-29T16:22:23.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 msgr2=0x7efef019f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef019f7a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efee800ef30 tx=0x7efee800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 shutdown_connections 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efed80a0790 0x7efed80a2c40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef01081c0 0x7efef019f7a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efef0073750 0x7efef01981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 --2- 172.21.15.31:0/3192573622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efef0074d30 0x7efef0198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 >> 172.21.15.31:0/3192573622 conn(0x7efef00fb970 msgr2=0x7efef00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 shutdown_connections 2023-07-29T16:22:23.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.125+0000 7efef59ff700 1 -- 172.21.15.31:0/3192573622 wait complete. 2023-07-29T16:22:23.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:22:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.399+0000 7fa37fdff700 1 -- 172.21.15.31:0/2090923889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 msgr2=0x7fa3780731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.399+0000 7fa37fdff700 1 --2- 172.21.15.31:0/2090923889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa3780731e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa374009990 tx=0x7fa374009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 -- 172.21.15.31:0/2090923889 shutdown_connections 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 --2- 172.21.15.31:0/2090923889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 0x7fa37810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 --2- 172.21.15.31:0/2090923889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa3780731e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 --2- 172.21.15.31:0/2090923889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 0x7fa378073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 -- 172.21.15.31:0/2090923889 >> 172.21.15.31:0/2090923889 conn(0x7fa3780fb970 msgr2=0x7fa3780fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 -- 172.21.15.31:0/2090923889 shutdown_connections 2023-07-29T16:22:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.400+0000 7fa37fdff700 1 -- 172.21.15.31:0/2090923889 wait complete. 2023-07-29T16:22:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 Processor -- start 2023-07-29T16:22:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 -- start start 2023-07-29T16:22:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 0x7fa37819c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa37819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 0x7fa3781a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3781a42f0 con 0x7fa3780737b0 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3781a4430 con 0x7fa37810c900 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37fdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3781a45c0 con 0x7fa378074d90 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.401+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa37819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37e39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 0x7fa3781a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa37819ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54584/0 (socket says 172.21.15.31:54584) 2023-07-29T16:22:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 -- 172.21.15.31:0/1119866233 learned_addr learned my addr 172.21.15.31:0/1119866233 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37db9b700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 0x7fa37819c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 -- 172.21.15.31:0/1119866233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 msgr2=0x7fa3781a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 0x7fa3781a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 -- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 msgr2=0x7fa37819c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 0x7fa37819c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 -- 172.21.15.31:0/1119866233 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa370009c40 con 0x7fa378074d90 2023-07-29T16:22:23.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37db9b700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 0x7fa37819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37e39c700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 0x7fa3781a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa37d39a700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa37819ccb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa36800e970 tx=0x7fa36800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.402+0000 7fa36effd700 1 -- 172.21.15.31:0/1119866233 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368009790 con 0x7fa378074d90 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.403+0000 7fa36effd700 1 -- 172.21.15.31:0/1119866233 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa36800eeb0 con 0x7fa378074d90 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.403+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3740095f0 con 0x7fa378074d90 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.403+0000 7fa36effd700 1 -- 172.21.15.31:0/1119866233 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368003d20 con 0x7fa378074d90 2023-07-29T16:22:23.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.403+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3781a4c20 con 0x7fa378074d90 2023-07-29T16:22:23.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.406+0000 7fa36effd700 1 -- 172.21.15.31:0/1119866233 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa368003e80 con 0x7fa378074d90 2023-07-29T16:22:23.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 Processor -- start 2023-07-29T16:22:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 -- start start 2023-07-29T16:22:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa36effd700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3640a0790 0x7fa3640a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa36effd700 1 -- 172.21.15.31:0/1119866233 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa368014070 con 0x7fa378074d90 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8d0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 0x56529d8d1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d828a10 con 0x56529d8d0be0 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8263d0 con 0x56529d8d1530 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.408+0000 7fa39365db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d821cc0 con 0x56529d8d22f0 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37e39c700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3640a0790 0x7fa3640a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37db9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8d0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 0x56529d8d1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 0x56529d8d1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44522/0 (socket says 172.21.15.31:44522) 2023-07-29T16:22:23.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 -- 172.21.15.31:0/2601810489 learned_addr learned my addr 172.21.15.31:0/2601810489 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37e39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 -- 172.21.15.31:0/2601810489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 msgr2=0x56529d8d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8d2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 -- 172.21.15.31:0/2601810489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 msgr2=0x56529d8d0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8d0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 -- 172.21.15.31:0/2601810489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529d64fde0 con 0x56529d8d1530 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37d39a700 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 0x56529d8d1d60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa3680994d0 tx=0x7fa368099150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87bec787dd425a73 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.409+0000 7fa37e39c700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3640a0790 0x7fa3640a2c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa370009550 tx=0x7fa3700094e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.410+0000 7fa363fff700 1 -- 172.21.15.31:0/2601810489 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa368099540 con 0x56529d8d1530 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.410+0000 7fa363fff700 1 -- 172.21.15.31:0/2601810489 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa36809d570 con 0x56529d8d1530 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.410+0000 7fa363fff700 1 -- 172.21.15.31:0/2601810489 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36809d350 con 0x56529d8d1530 2023-07-29T16:22:23.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.410+0000 7fa39365db80 1 -- 172.21.15.31:0/2601810489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 msgr2=0x56529d8d1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.410+0000 7fa39365db80 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 0x56529d8d1d60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fa3680994d0 tx=0x7fa368099150 comp rx=0 tx=0).stop 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 -- 172.21.15.31:0/2601810489 shutdown_connections 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8d0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 --2- 172.21.15.31:0/2601810489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d1530 0x56529d8d1d60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 -- 172.21.15.31:0/2601810489 >> 172.21.15.31:0/2601810489 conn(0x56529d8c63e0 msgr2=0x56529d8cb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 -- 172.21.15.31:0/2601810489 shutdown_connections 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 -- 172.21.15.31:0/2601810489 wait complete. 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 -- start start 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.411+0000 7fa39365db80 1 Processor -- start 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8da850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d1530 0x56529d8dad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d7be510 con 0x56529d8d1530 2023-07-29T16:22:23.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d7be1b0 con 0x56529d8d0be0 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d6a5920 con 0x56529d8d22f0 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa39365db80 10 client.0 _ll_register_callbacks cb 0x56529d7c7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37db9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8da850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37e39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37db9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8da850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44532/0 (socket says 172.21.15.31:44532) 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d1530 0x56529d8dad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37db9b700 1 -- 172.21.15.31:0/2224350187 learned_addr learned my addr 172.21.15.31:0/2224350187 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37e39c700 1 -- 172.21.15.31:0/2224350187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 msgr2=0x56529d8da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37e39c700 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8da850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37e39c700 1 -- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d1530 msgr2=0x56529d8dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37e39c700 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d1530 0x56529d8dad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.412+0000 7fa37e39c700 1 -- 172.21.15.31:0/2224350187 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3781a4c20 con 0x56529d8d22f0 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa37d39a700 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d1530 0x56529d8dad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa37e39c700 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8e1dd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa370006130 tx=0x7fa370004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b80b9795a1ee5d56 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa3617fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:23.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa3617fa700 1 -- 172.21.15.31:0/2224350187 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa370007780 con 0x56529d8d22f0 2023-07-29T16:22:23.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56529d84cd10 con 0x56529d8d22f0 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa3617fa700 1 -- 172.21.15.31:0/2224350187 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa370021040 con 0x56529d8d22f0 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa3617fa700 1 -- 172.21.15.31:0/2224350187 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3700046b0 con 0x56529d8d22f0 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa39365db80 10 client.27616 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.413+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56529d638c60 con 0x56529d8d22f0 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa3617fa700 1 -- 172.21.15.31:0/2224350187 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56529d84cd10 con 0x56529d8d22f0 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa3617fa700 1 -- 172.21.15.31:0/2224350187 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa370025440 con 0x56529d8d22f0 2023-07-29T16:22:23.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa39365db80 10 client.27616 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa39365db80 10 client.27616 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa39365db80 20 client.27616 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa39365db80 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d8e65e0 0x56529d8e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa39365db80 4 client.27616 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56529d89fd70 con 0x56529d8e65e0 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.414+0000 7fa37db9b700 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d8e65e0 0x56529d8e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.415+0000 7fa37db9b700 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d8e65e0 0x56529d8e89f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6e574e66125c94e9 server_cookie=3740898dff0cac6 in_seq=0 out_seq=0 2023-07-29T16:22:23.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.415+0000 7fa3617fa700 10 client.27616 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.415+0000 7fa3617fa700 1 -- 172.21.15.31:0/2224350187 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56529d89fd70 con 0x56529d8e65e0 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.415+0000 7fa3617fa700 10 client.27616 handle_command_reply: tid=0 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.415+0000 7fa39365db80 1 client.27616 shutdown 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 msgr2=0x56529d8e1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8e1dd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa370006130 tx=0x7fa370004b40 comp rx=0 tx=0).stop 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 shutdown_connections 2023-07-29T16:22:23.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d8e65e0 0x56529d8e89f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d22f0 0x56529d8e1dd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8da850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 --2- 172.21.15.31:0/2224350187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d1530 0x56529d8dad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa3617fa700 0 client.27616 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 >> 172.21.15.31:0/2224350187 conn(0x56529d8c63e0 msgr2=0x56529d8cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 shutdown_connections 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 1 -- 172.21.15.31:0/2224350187 wait complete. 2023-07-29T16:22:23.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.416+0000 7fa39365db80 20 client.27616 trim_cache size 0 max 16384 2023-07-29T16:22:23.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.431+0000 7fa39365db80 1 Processor -- start 2023-07-29T16:22:23.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.431+0000 7fa39365db80 1 -- start start 2023-07-29T16:22:23.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.431+0000 7fa39365db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa39365db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa39365db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d22f0 0x56529d8d26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa39365db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8da3c0 con 0x56529d8d22f0 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa39365db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8c8460 con 0x56529d8d0be0 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa39365db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8db4d0 con 0x56529d8d1530 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8d0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44546/0 (socket says 172.21.15.31:44546) 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37e39c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d22f0 0x56529d8d26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 -- 172.21.15.31:0/979040903 learned_addr learned my addr 172.21.15.31:0/979040903 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37d39a700 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 -- 172.21.15.31:0/979040903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 msgr2=0x56529d8e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8e1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 -- 172.21.15.31:0/979040903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d22f0 msgr2=0x56529d8d26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d22f0 0x56529d8d26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 -- 172.21.15.31:0/979040903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529d676250 con 0x56529d8d0be0 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37d39a700 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.432+0000 7fa37db9b700 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8d0fb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa37400b2e0 tx=0x7fa374005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9964074686e00ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa36cbff700 1 -- 172.21.15.31:0/979040903 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400bbf0 con 0x56529d8d0be0 2023-07-29T16:22:23.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa36cbff700 1 -- 172.21.15.31:0/979040903 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa37401e040 con 0x56529d8d0be0 2023-07-29T16:22:23.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa36cbff700 1 -- 172.21.15.31:0/979040903 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374025400 con 0x56529d8d0be0 2023-07-29T16:22:23.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 -- 172.21.15.31:0/979040903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 msgr2=0x56529d8d0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8d0fb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa37400b2e0 tx=0x7fa374005ab0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 -- 172.21.15.31:0/979040903 shutdown_connections 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d22f0 0x56529d8d26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d0be0 0x56529d8d0fb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 --2- 172.21.15.31:0/979040903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.433+0000 7fa39365db80 1 -- 172.21.15.31:0/979040903 >> 172.21.15.31:0/979040903 conn(0x56529d8c63e0 msgr2=0x56529d8cb900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.434+0000 7fa39365db80 1 -- 172.21.15.31:0/979040903 shutdown_connections 2023-07-29T16:22:23.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.434+0000 7fa39365db80 1 -- 172.21.15.31:0/979040903 wait complete. 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.434+0000 7fa39365db80 1 -- start start 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.434+0000 7fa39365db80 1 Processor -- start 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8dda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8ddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d22f0 0x56529d8e0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8e1520 con 0x56529d8d0be0 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8e1690 con 0x56529d8d22f0 2023-07-29T16:22:23.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529d8e1850 con 0x56529d8d1530 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa39365db80 10 client.0 _ll_register_callbacks cb 0x56529d7c7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37db9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8ddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37d39a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8ddfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54616/0 (socket says 172.21.15.31:54616) 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37d39a700 1 -- 172.21.15.31:0/3000846524 learned_addr learned my addr 172.21.15.31:0/3000846524 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37e39c700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d22f0 0x56529d8e0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37db9b700 1 -- 172.21.15.31:0/3000846524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 msgr2=0x56529d8ddfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37db9b700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8ddfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37db9b700 1 -- 172.21.15.31:0/3000846524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d22f0 msgr2=0x56529d8e0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37db9b700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d22f0 0x56529d8e0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37db9b700 1 -- 172.21.15.31:0/3000846524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa37810f690 con 0x56529d8d0be0 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.435+0000 7fa37d39a700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8ddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa37db9b700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8dda80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x56529d8cd460 tx=0x7fa374005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4ba3b536a3f62a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa3617fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:22:23.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa3617fa700 1 -- 172.21.15.31:0/3000846524 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37400be00 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa3617fa700 1 -- 172.21.15.31:0/3000846524 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa37400f460 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa36809d570 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa3617fa700 1 -- 172.21.15.31:0/3000846524 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa37400f620 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa39365db80 10 client.17583 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56529d64e2b0 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.436+0000 7fa3617fa700 1 -- 172.21.15.31:0/3000846524 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa374041020 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa3617fa700 1 -- 172.21.15.31:0/3000846524 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa374034c20 con 0x56529d8d0be0 2023-07-29T16:22:23.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa39365db80 10 client.17583 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa39365db80 10 client.17583 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa39365db80 20 client.17583 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa39365db80 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d938bb0 0x56529d93af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa39365db80 4 client.17583 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56529d81ad40 con 0x56529d938bb0 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.437+0000 7fa37d39a700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d938bb0 0x56529d93af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa37d39a700 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d938bb0 0x56529d93af90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14c20188c6dde38c server_cookie=2b1d10b055f1d40b in_seq=0 out_seq=0 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa3617fa700 10 client.17583 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa3617fa700 1 -- 172.21.15.31:0/3000846524 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56529d81ad40 con 0x56529d938bb0 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa3617fa700 10 client.17583 handle_command_reply: tid=0 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 client.17583 shutdown 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 msgr2=0x56529d8dda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8dda80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x56529d8cd460 tx=0x7fa374005810 comp rx=0 tx=0).stop 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 shutdown_connections 2023-07-29T16:22:23.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56529d938bb0 0x56529d93af90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529d8d22f0 0x56529d8e0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529d8d0be0 0x56529d8dda80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 --2- 172.21.15.31:0/3000846524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529d8d1530 0x56529d8ddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 >> 172.21.15.31:0/3000846524 conn(0x56529d8c63e0 msgr2=0x56529d8d8740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa3617fa700 0 client.17583 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 shutdown_connections 2023-07-29T16:22:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 1 -- 172.21.15.31:0/3000846524 wait complete. 2023-07-29T16:22:23.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.438+0000 7fa39365db80 20 client.17583 trim_cache size 0 max 16384 2023-07-29T16:22:23.446 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:22:23.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3640a0790 msgr2=0x7fa3640a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3640a0790 0x7fa3640a2c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa370009550 tx=0x7fa3700094e0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 msgr2=0x7fa37819ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa37819ccb0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa36800e970 tx=0x7fa36800ed30 comp rx=0 tx=0).stop 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 shutdown_connections 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3640a0790 0x7fa3640a2c40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa37810c900 0x7fa3781a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3780737b0 0x7fa37819c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 --2- 172.21.15.31:0/1119866233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa378074d90 0x7fa37819ccb0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 >> 172.21.15.31:0/1119866233 conn(0x7fa3780fb970 msgr2=0x7fa3780fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 shutdown_connections 2023-07-29T16:22:23.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.439+0000 7fa37fdff700 1 -- 172.21.15.31:0/1119866233 wait complete. 2023-07-29T16:22:23.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:22:23.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 -- 172.21.15.31:0/3439009782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54101510 msgr2=0x7f6e54101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 --2- 172.21.15.31:0/3439009782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54101510 0x7f6e54101960 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c009990 tx=0x7f6e3c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 -- 172.21.15.31:0/3439009782 shutdown_connections 2023-07-29T16:22:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 --2- 172.21.15.31:0/3439009782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e54101ea0 0x7f6e54108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 --2- 172.21.15.31:0/3439009782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e54100320 0x7f6e54100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 --2- 172.21.15.31:0/3439009782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54101510 0x7f6e54101960 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 -- 172.21.15.31:0/3439009782 >> 172.21.15.31:0/3439009782 conn(0x7f6e540fb920 msgr2=0x7f6e540fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 -- 172.21.15.31:0/3439009782 shutdown_connections 2023-07-29T16:22:23.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.676+0000 7f6e58fff700 1 -- 172.21.15.31:0/3439009782 wait complete. 2023-07-29T16:22:23.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.677+0000 7f6e58fff700 1 Processor -- start 2023-07-29T16:22:23.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.677+0000 7f6e58fff700 1 -- start start 2023-07-29T16:22:23.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.677+0000 7f6e58fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 0x7f6e54074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.677+0000 7f6e58fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e54101510 0x7f6e54073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.677+0000 7f6e58fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e54101ea0 0x7f6e54073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e58fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54073e20 con 0x7f6e54101510 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e58fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e54073f90 con 0x7f6e54101ea0 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e58fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e541a00a0 con 0x7f6e54100320 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 0x7f6e54074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e54101510 0x7f6e54073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 0x7f6e54074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54622/0 (socket says 172.21.15.31:54622) 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 -- 172.21.15.31:0/151921001 learned_addr learned my addr 172.21.15.31:0/151921001 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e54101ea0 0x7f6e54073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 -- 172.21.15.31:0/151921001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e54101ea0 msgr2=0x7f6e54073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e54101ea0 0x7f6e54073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 -- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e54101510 msgr2=0x7f6e54073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e54101510 0x7f6e54073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 -- 172.21.15.31:0/151921001 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c0095f0 con 0x7f6e54100320 2023-07-29T16:22:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.678+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 0x7f6e54074b00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400e970 tx=0x7f6e4400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.679+0000 7f6e4bfff700 1 -- 172.21.15.31:0/151921001 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44009790 con 0x7f6e54100320 2023-07-29T16:22:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.679+0000 7f6e4bfff700 1 -- 172.21.15.31:0/151921001 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4400c7e0 con 0x7f6e54100320 2023-07-29T16:22:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.679+0000 7f6e4bfff700 1 -- 172.21.15.31:0/151921001 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44003c70 con 0x7f6e54100320 2023-07-29T16:22:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.679+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e541a0380 con 0x7f6e54100320 2023-07-29T16:22:23.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.679+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e541a0840 con 0x7f6e54100320 2023-07-29T16:22:23.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.682+0000 7f6e4bfff700 1 -- 172.21.15.31:0/151921001 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6e4400cc50 con 0x7f6e54100320 2023-07-29T16:22:23.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.684+0000 7f6e6c8b5b80 1 Processor -- start 2023-07-29T16:22:23.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.684+0000 7f6e6c8b5b80 1 -- start start 2023-07-29T16:22:23.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.684+0000 7f6e4bfff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e400a0810 0x7f6e400a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a588ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4bfff700 1 -- 172.21.15.31:0/151921001 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6e44014070 con 0x7f6e54100320 2023-07-29T16:22:23.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a589530 0x55bc6a589d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a58a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a4e0a10 con 0x55bc6a58a2f0 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a4de3d0 con 0x55bc6a588be0 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a4d9cc0 con 0x55bc6a589530 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a588ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a588ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44578/0 (socket says 172.21.15.31:44578) 2023-07-29T16:22:23.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 -- 172.21.15.31:0/742323192 learned_addr learned my addr 172.21.15.31:0/742323192 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e400a0810 0x7f6e400a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e5259c700 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a589530 0x55bc6a589d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 -- 172.21.15.31:0/742323192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a589530 msgr2=0x55bc6a589d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a589530 0x55bc6a589d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 -- 172.21.15.31:0/742323192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 msgr2=0x55bc6a58a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.685+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a58a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e4a59c700 1 -- 172.21.15.31:0/742323192 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc6a307de0 con 0x55bc6a588be0 2023-07-29T16:22:23.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e5259c700 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a589530 0x55bc6a589d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:23.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a588ff0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6e440958d0 tx=0x7f6e44098d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11863c94c1cf5e0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e48d99700 1 -- 172.21.15.31:0/742323192 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4409d400 con 0x55bc6a588be0 2023-07-29T16:22:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e400a0810 0x7f6e400a2cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009ec0 tx=0x7f6e4c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e48d99700 1 -- 172.21.15.31:0/742323192 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e440a1820 con 0x55bc6a588be0 2023-07-29T16:22:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.686+0000 7f6e48d99700 1 -- 172.21.15.31:0/742323192 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e44099890 con 0x55bc6a588be0 2023-07-29T16:22:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/742323192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 msgr2=0x55bc6a588ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a588ff0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f6e440958d0 tx=0x7f6e44098d30 comp rx=0 tx=0).stop 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/742323192 shutdown_connections 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a58a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a588ff0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/742323192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a589530 0x55bc6a589d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/742323192 >> 172.21.15.31:0/742323192 conn(0x55bc6a57e3e0 msgr2=0x55bc6a583990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/742323192 shutdown_connections 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.687+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/742323192 wait complete. 2023-07-29T16:22:23.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.688+0000 7f6e6c8b5b80 1 -- start start 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.688+0000 7f6e6c8b5b80 1 Processor -- start 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a592830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a589530 0x55bc6a592d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a599db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a476510 con 0x55bc6a58a2f0 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a4761b0 con 0x55bc6a589530 2023-07-29T16:22:23.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a35d920 con 0x55bc6a588be0 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e6c8b5b80 10 client.0 _ll_register_callbacks cb 0x55bc6a47f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a592830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e5259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a589530 0x55bc6a592d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a592830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54640/0 (socket says 172.21.15.31:54640) 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e5259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a589530 0x55bc6a592d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44594/0 (socket says 172.21.15.31:44594) 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 -- 172.21.15.31:0/2331160087 learned_addr learned my addr 172.21.15.31:0/2331160087 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a599db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 -- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a589530 msgr2=0x55bc6a592d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a589530 0x55bc6a592d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 -- 172.21.15.31:0/2331160087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 msgr2=0x55bc6a599db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a599db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e4a59c700 1 -- 172.21.15.31:0/2331160087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e541a0840 con 0x55bc6a588be0 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.689+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a599db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a592830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e440990d0 tx=0x7f6e440991c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80fdf13fbbdd8ab6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e39ffb700 1 -- 172.21.15.31:0/2331160087 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4409d400 con 0x55bc6a588be0 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e39ffb700 1 -- 172.21.15.31:0/2331160087 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6e44099630 con 0x55bc6a588be0 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc6a504d10 con 0x55bc6a588be0 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e39ffb700 1 -- 172.21.15.31:0/2331160087 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e44099830 con 0x55bc6a588be0 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e6c8b5b80 10 client.27634 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.690+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bc6a2f0c60 con 0x55bc6a588be0 2023-07-29T16:22:23.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e39ffb700 1 -- 172.21.15.31:0/2331160087 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bc6a504d10 con 0x55bc6a588be0 2023-07-29T16:22:23.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e39ffb700 1 -- 172.21.15.31:0/2331160087 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f6e44098330 con 0x55bc6a588be0 2023-07-29T16:22:23.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e6c8b5b80 10 client.27634 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:23.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e6c8b5b80 10 client.27634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:23.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e6c8b5b80 20 client.27634 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:23.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a59e1a0 0x55bc6a5a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e6c8b5b80 4 client.27634 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc6a557d70 con 0x55bc6a59e1a0 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.691+0000 7f6e5259c700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a59e1a0 0x55bc6a5a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.693+0000 7f6e5259c700 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a59e1a0 0x55bc6a5a05b0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264844eb0c28d77f server_cookie=d81b9f761a98e845 in_seq=0 out_seq=0 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.693+0000 7f6e39ffb700 10 client.27634 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.696+0000 7f6e39ffb700 1 -- 172.21.15.31:0/2331160087 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bc6a557d70 con 0x55bc6a59e1a0 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.696+0000 7f6e39ffb700 10 client.27634 handle_command_reply: tid=0 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.696+0000 7f6e6c8b5b80 1 client.27634 shutdown 2023-07-29T16:22:23.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.696+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 msgr2=0x55bc6a592830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.696+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a592830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6e440990d0 tx=0x7f6e440991c0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 shutdown_connections 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a59e1a0 0x55bc6a5a05b0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a58a2f0 0x55bc6a599db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a592830 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e39ffb700 0 client.27634 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:23.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2331160087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a589530 0x55bc6a592d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 >> 172.21.15.31:0/2331160087 conn(0x55bc6a57e3e0 msgr2=0x55bc6a583060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 shutdown_connections 2023-07-29T16:22:23.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2331160087 wait complete. 2023-07-29T16:22:23.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.697+0000 7f6e6c8b5b80 20 client.27634 trim_cache size 0 max 16384 2023-07-29T16:22:23.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.711+0000 7f6e6c8b5b80 1 Processor -- start 2023-07-29T16:22:23.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.711+0000 7f6e6c8b5b80 1 -- start start 2023-07-29T16:22:23.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a58a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a58a2f0 0x55bc6a5914e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a591ce0 con 0x55bc6a589530 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a580460 con 0x55bc6a58a2f0 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a589930 con 0x55bc6a588be0 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e52d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a58a2f0 0x55bc6a5914e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e5259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34208/0 (socket says 172.21.15.31:34208) 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e5259c700 1 -- 172.21.15.31:0/2249193157 learned_addr learned my addr 172.21.15.31:0/2249193157 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a58a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e52d9d700 1 -- 172.21.15.31:0/2249193157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 msgr2=0x55bc6a58a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a58a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e52d9d700 1 -- 172.21.15.31:0/2249193157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 msgr2=0x55bc6a599d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e52d9d700 1 -- 172.21.15.31:0/2249193157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc6a32e250 con 0x55bc6a58a2f0 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.712+0000 7f6e5259c700 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.713+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a58a2f0 0x55bc6a5914e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6e54075160 tx=0x7f6e4c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b8d513fe76ef311 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.713+0000 7f6e4959a700 1 -- 172.21.15.31:0/2249193157 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c00e040 con 0x55bc6a58a2f0 2023-07-29T16:22:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.713+0000 7f6e4959a700 1 -- 172.21.15.31:0/2249193157 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4c00f040 con 0x55bc6a58a2f0 2023-07-29T16:22:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.713+0000 7f6e4959a700 1 -- 172.21.15.31:0/2249193157 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c004590 con 0x55bc6a58a2f0 2023-07-29T16:22:23.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2249193157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a58a2f0 msgr2=0x55bc6a5914e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a58a2f0 0x55bc6a5914e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6e54075160 tx=0x7f6e4c015000 comp rx=0 tx=0).stop 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2249193157 shutdown_connections 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a58a2f0 0x55bc6a5914e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a588be0 0x55bc6a58a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/2249193157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2249193157 >> 172.21.15.31:0/2249193157 conn(0x55bc6a57e3e0 msgr2=0x55bc6a583960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2249193157 shutdown_connections 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/2249193157 wait complete. 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.714+0000 7f6e6c8b5b80 1 -- start start 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 Processor -- start 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a599630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a58a2f0 0x55bc6a597c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a598260 con 0x55bc6a589530 2023-07-29T16:22:23.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a5983d0 con 0x55bc6a588be0 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc6a598590 con 0x55bc6a58a2f0 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.715+0000 7f6e6c8b5b80 10 client.0 _ll_register_callbacks cb 0x55bc6a47f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a599630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a599630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44614/0 (socket says 172.21.15.31:44614) 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 -- 172.21.15.31:0/1684940794 learned_addr learned my addr 172.21.15.31:0/1684940794 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e5259c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e52d9d700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a58a2f0 0x55bc6a597c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 -- 172.21.15.31:0/1684940794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a58a2f0 msgr2=0x55bc6a597c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a58a2f0 0x55bc6a597c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 -- 172.21.15.31:0/1684940794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 msgr2=0x55bc6a599b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 -- 172.21.15.31:0/1684940794 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e54109090 con 0x55bc6a588be0 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e5259c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:23.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e4a59c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a599630 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6e440958d0 tx=0x7f6e44098d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dfa2157e95f74e7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.716+0000 7f6e39ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e39ffb700 1 -- 172.21.15.31:0/1684940794 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e440953c0 con 0x55bc6a588be0 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e440a1820 con 0x55bc6a588be0 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e39ffb700 1 -- 172.21.15.31:0/1684940794 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6e4409d400 con 0x55bc6a588be0 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e39ffb700 1 -- 172.21.15.31:0/1684940794 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e44099630 con 0x55bc6a588be0 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 10 client.27629 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f6e4c008da0 con 0x55bc6a588be0 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e39ffb700 1 -- 172.21.15.31:0/1684940794 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e440a1820 con 0x55bc6a588be0 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e39ffb700 1 -- 172.21.15.31:0/1684940794 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f6e440a1b90 con 0x55bc6a588be0 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 10 client.27629 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 10 client.27629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 20 client.27629 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a5f0b50 0x55bc6a5f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:23.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 4 client.27629 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:22:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bc6a4d2d40 con 0x55bc6a5f0b50 2023-07-29T16:22:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.717+0000 7f6e5259c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a5f0b50 0x55bc6a5f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.718+0000 7f6e5259c700 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a5f0b50 0x55bc6a5f2f30 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5796a7656936b448 server_cookie=42c8a62f7b0ba355 in_seq=0 out_seq=0 2023-07-29T16:22:23.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.718+0000 7f6e39ffb700 10 client.27629 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:23.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e39ffb700 1 -- 172.21.15.31:0/1684940794 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bc6a4d2d40 con 0x55bc6a5f0b50 2023-07-29T16:22:23.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e39ffb700 10 client.27629 handle_command_reply: tid=0 2023-07-29T16:22:23.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 client.27629 shutdown 2023-07-29T16:22:23.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 msgr2=0x55bc6a599630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a599630 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f6e440958d0 tx=0x7f6e44098d30 comp rx=0 tx=0).stop 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 shutdown_connections 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc6a5f0b50 0x55bc6a5f2f30 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc6a58a2f0 0x55bc6a597c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e39ffb700 0 client.27629 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6a588be0 0x55bc6a599630 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 --2- 172.21.15.31:0/1684940794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc6a589530 0x55bc6a599b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 >> 172.21.15.31:0/1684940794 conn(0x55bc6a57e3e0 msgr2=0x55bc6a590740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 shutdown_connections 2023-07-29T16:22:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.721+0000 7f6e6c8b5b80 1 -- 172.21.15.31:0/1684940794 wait complete. 2023-07-29T16:22:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e6c8b5b80 20 client.27629 trim_cache size 0 max 16384 2023-07-29T16:22:23.727 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:22:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e400a0810 msgr2=0x7f6e400a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e400a0810 0x7f6e400a2cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009ec0 tx=0x7f6e4c009920 comp rx=0 tx=0).stop 2023-07-29T16:22:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 msgr2=0x7f6e54074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 0x7f6e54074b00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400e970 tx=0x7f6e4400ec80 comp rx=0 tx=0).stop 2023-07-29T16:22:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 shutdown_connections 2023-07-29T16:22:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e400a0810 0x7f6e400a2cc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e54101ea0 0x7f6e54073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e54100320 0x7f6e54074b00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 --2- 172.21.15.31:0/151921001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e54101510 0x7f6e54073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:23.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 >> 172.21.15.31:0/151921001 conn(0x7f6e540fb920 msgr2=0x7f6e540fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:23.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 shutdown_connections 2023-07-29T16:22:23.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:23.722+0000 7f6e58fff700 1 -- 172.21.15.31:0/151921001 wait complete. 2023-07-29T16:22:23.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2584965139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:22:23.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2584965139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:22:23.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2584965139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:22:23.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 415841 8.93 MB/sec execute 128 sec latency 27.743 ms 2023-07-29T16:22:24.741 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:22:24.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:22:24.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:24 smithi039 ceph-mon[128858]: pgmap v578: 97 pgs: 97 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 297 op/s 2023-07-29T16:22:24.806 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:24 smithi031 ceph-mon[95504]: pgmap v578: 97 pgs: 97 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 297 op/s 2023-07-29T16:22:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:24 smithi156 ceph-mon[129245]: pgmap v578: 97 pgs: 97 active+clean; 95 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 297 op/s 2023-07-29T16:22:24.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 418031 8.95 MB/sec execute 129 sec latency 82.893 ms 2023-07-29T16:22:24.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/1959946018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101ea0 msgr2=0x7f8f48108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:24.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/1959946018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101ea0 0x7f8f48108760 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8f4000b0a0 tx=0x7f8f4000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/1959946018 shutdown_connections 2023-07-29T16:22:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/1959946018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101ea0 0x7f8f48108760 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/1959946018 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48100320 0x7f8f48100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/1959946018 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48101510 0x7f8f48101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/1959946018 >> 172.21.15.31:0/1959946018 conn(0x7f8f480fb920 msgr2=0x7f8f480fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:24.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.959+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/1959946018 shutdown_connections 2023-07-29T16:22:24.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.960+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/1959946018 wait complete. 2023-07-29T16:22:24.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 Processor -- start 2023-07-29T16:22:24.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 -- start start 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 0x7f8f48074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101510 0x7f8f48073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48101ea0 0x7f8f48073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48073e20 con 0x7f8f48101ea0 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48073f90 con 0x7f8f48101510 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f481a01c0 con 0x7f8f48100320 2023-07-29T16:22:24.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 0x7f8f48074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 0x7f8f48074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54670/0 (socket says 172.21.15.31:54670) 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f47fff700 1 -- 172.21.15.31:0/271710670 learned_addr learned my addr 172.21.15.31:0/271710670 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f477fe700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101510 0x7f8f48073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.961+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48101ea0 0x7f8f48073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f47fff700 1 -- 172.21.15.31:0/271710670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101510 msgr2=0x7f8f48073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f47fff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101510 0x7f8f48073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f47fff700 1 -- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48101ea0 msgr2=0x7f8f48073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f47fff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48101ea0 0x7f8f48073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f47fff700 1 -- 172.21.15.31:0/271710670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40009d00 con 0x7f8f48100320 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48101ea0 0x7f8f48073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f47fff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 0x7f8f48074b00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800e970 tx=0x7f8f3800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f457fa700 1 -- 172.21.15.31:0/271710670 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38009790 con 0x7f8f48100320 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f481a04a0 con 0x7f8f48100320 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f457fa700 1 -- 172.21.15.31:0/271710670 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f3800eec0 con 0x7f8f48100320 2023-07-29T16:22:24.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f457fa700 1 -- 172.21.15.31:0/271710670 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38003cf0 con 0x7f8f48100320 2023-07-29T16:22:24.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.962+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f481a09b0 con 0x7f8f48100320 2023-07-29T16:22:24.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.966+0000 7f8f457fa700 1 -- 172.21.15.31:0/271710670 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8f3800ccd0 con 0x7f8f48100320 2023-07-29T16:22:24.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 Processor -- start 2023-07-29T16:22:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f457fa700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f300a0790 0x7f8f300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 -- start start 2023-07-29T16:22:24.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f457fa700 1 -- 172.21.15.31:0/271710670 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f38014070 con 0x7f8f48100320 2023-07-29T16:22:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f300a0790 0x7f8f300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f12a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f12ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f12b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f081a90 con 0x55b34f12b370 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f07f450 con 0x55b34f12a5b0 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.967+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f07ad40 con 0x55b34f129c60 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f12a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f12a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54674/0 (socket says 172.21.15.31:54674) 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f47fff700 1 -- 172.21.15.31:0/2091368733 learned_addr learned my addr 172.21.15.31:0/2091368733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f12ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f12b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 -- 172.21.15.31:0/2091368733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 msgr2=0x55b34f12a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f12a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 -- 172.21.15.31:0/2091368733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 msgr2=0x55b34f12b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f12b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 -- 172.21.15.31:0/2091368733 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b34eea8e60 con 0x55b34f12a5b0 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f47fff700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f12a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f12b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f300a0790 0x7f8f300a2c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8f48075160 tx=0x7f8f40009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.968+0000 7f8f477fe700 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f12ade0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009cb0 tx=0x7f8f3c011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2abbce3f74d4cbfd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f2e7fc700 1 -- 172.21.15.31:0/2091368733 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c011d20 con 0x55b34f12a5b0 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f2e7fc700 1 -- 172.21.15.31:0/2091368733 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f3c011e80 con 0x55b34f12a5b0 2023-07-29T16:22:24.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f2e7fc700 1 -- 172.21.15.31:0/2091368733 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c005280 con 0x55b34f12a5b0 2023-07-29T16:22:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/2091368733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 msgr2=0x55b34f12ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:24.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f12ade0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009cb0 tx=0x7f8f3c011440 comp rx=0 tx=0).stop 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/2091368733 shutdown_connections 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f12b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f12a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/2091368733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f12ade0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.969+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/2091368733 >> 172.21.15.31:0/2091368733 conn(0x55b34f11f460 msgr2=0x55b34f124a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.970+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/2091368733 shutdown_connections 2023-07-29T16:22:24.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.970+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/2091368733 wait complete. 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.970+0000 7f8f61e3fb80 1 -- start start 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.970+0000 7f8f61e3fb80 1 Processor -- start 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f133810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f133d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f13ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f017590 con 0x55b34f12b370 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f017230 con 0x55b34f129c60 2023-07-29T16:22:24.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34eefe9a0 con 0x55b34f12a5b0 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f61e3fb80 10 client.0 _ll_register_callbacks cb 0x55b34f0206a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f133810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f133810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44656/0 (socket says 172.21.15.31:44656) 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 -- 172.21.15.31:0/1486696126 learned_addr learned my addr 172.21.15.31:0/1486696126 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f477fe700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f13ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 -- 172.21.15.31:0/1486696126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 msgr2=0x55b34f133d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f133d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 -- 172.21.15.31:0/1486696126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 msgr2=0x55b34f13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f13ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f47fff700 1 -- 172.21.15.31:0/1486696126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f481a09b0 con 0x55b34f129c60 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.971+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f13ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f477fe700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f47fff700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f133810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8f380a1430 tx=0x7f8f38099360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41c1cc33ad84ab6f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:24.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f277fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f277fe700 1 -- 172.21.15.31:0/1486696126 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38099970 con 0x55b34f129c60 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f277fe700 1 -- 172.21.15.31:0/1486696126 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f3809d540 con 0x55b34f129c60 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b34f0f8df0 con 0x55b34f129c60 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f277fe700 1 -- 172.21.15.31:0/1486696126 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f380a1960 con 0x55b34f129c60 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f61e3fb80 10 client.27641 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8f3c0098e0 con 0x55b34f129c60 2023-07-29T16:22:24.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.972+0000 7f8f277fe700 1 -- 172.21.15.31:0/1486696126 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f38063020 con 0x55b34f129c60 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f277fe700 1 -- 172.21.15.31:0/1486696126 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8f38098470 con 0x55b34f129c60 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f61e3fb80 10 client.27641 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f61e3fb80 10 client.27641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f61e3fb80 20 client.27641 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f13f570 0x55b34f141980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f61e3fb80 4 client.27641 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:24.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b34f073dc0 con 0x55b34f13f570 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f477fe700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f13f570 0x55b34f141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.974+0000 7f8f277fe700 10 client.27641 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.973+0000 7f8f477fe700 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f13f570 0x55b34f141980 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84c0e67515b28ef server_cookie=bb96f3a1f8512070 in_seq=0 out_seq=0 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.974+0000 7f8f277fe700 1 -- 172.21.15.31:0/1486696126 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b34f073dc0 con 0x55b34f13f570 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.974+0000 7f8f277fe700 10 client.27641 handle_command_reply: tid=0 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.974+0000 7f8f61e3fb80 1 client.27641 shutdown 2023-07-29T16:22:24.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 msgr2=0x55b34f133810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f133810 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8f380a1430 tx=0x7f8f38099360 comp rx=0 tx=0).stop 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 shutdown_connections 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f13f570 0x55b34f141980 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f12b370 0x55b34f13ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f133810 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/1486696126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f133d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f277fe700 0 client.27641 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:24.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 >> 172.21.15.31:0/1486696126 conn(0x55b34f11f460 msgr2=0x55b34f1240e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 shutdown_connections 2023-07-29T16:22:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/1486696126 wait complete. 2023-07-29T16:22:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.975+0000 7f8f61e3fb80 20 client.27641 trim_cache size 0 max 16384 2023-07-29T16:22:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 Processor -- start 2023-07-29T16:22:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 -- start start 2023-07-29T16:22:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f1240e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f13ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f1317c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f1333e0 con 0x55b34f11f460 2023-07-29T16:22:24.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f1214e0 con 0x55b34f129c60 2023-07-29T16:22:24.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f11c7f0 con 0x55b34f12a5b0 2023-07-29T16:22:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f13ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f47fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f1240e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.993+0000 7f8f4cb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f1317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f13ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44668/0 (socket says 172.21.15.31:44668) 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 -- 172.21.15.31:0/3776791923 learned_addr learned my addr 172.21.15.31:0/3776791923 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f1317c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54702/0 (socket says 172.21.15.31:54702) 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 -- 172.21.15.31:0/3776791923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 msgr2=0x55b34f1317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f1317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 -- 172.21.15.31:0/3776791923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 msgr2=0x55b34f1240e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f1240e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 -- 172.21.15.31:0/3776791923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b34f12ba10 con 0x55b34f129c60 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f1317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f477fe700 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f13ad50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x55b34f12a9c0 tx=0x7f8f3c004a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ed78bea3701777 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f2effd700 1 -- 172.21.15.31:0/3776791923 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c004030 con 0x55b34f129c60 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f2effd700 1 -- 172.21.15.31:0/3776791923 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f3c003680 con 0x55b34f129c60 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.994+0000 7f8f2effd700 1 -- 172.21.15.31:0/3776791923 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c025680 con 0x55b34f129c60 2023-07-29T16:22:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3776791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 msgr2=0x55b34f13ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f13ad50 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x55b34f12a9c0 tx=0x7f8f3c004a60 comp rx=0 tx=0).stop 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3776791923 shutdown_connections 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f12a5b0 0x55b34f1317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f1240e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3776791923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f129c60 0x55b34f13ad50 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3776791923 >> 172.21.15.31:0/3776791923 conn(0x55b34f1328c0 msgr2=0x55b34f125710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3776791923 shutdown_connections 2023-07-29T16:22:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.995+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3776791923 wait complete. 2023-07-29T16:22:24.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.996+0000 7f8f61e3fb80 1 -- start start 2023-07-29T16:22:24.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.996+0000 7f8f61e3fb80 1 Processor -- start 2023-07-29T16:22:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f137a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f137f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f1910f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f13af70 con 0x55b34f11f460 2023-07-29T16:22:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f13b0e0 con 0x55b34f12a5b0 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b34f191630 con 0x55b34f129c60 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f61e3fb80 10 client.0 _ll_register_callbacks cb 0x55b34f0206a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f137f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f137f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54710/0 (socket says 172.21.15.31:54710) 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 -- 172.21.15.31:0/3763097810 learned_addr learned my addr 172.21.15.31:0/3763097810 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f47fff700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f4cb9c700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f1910f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 -- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 msgr2=0x55b34f1910f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f1910f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 -- 172.21.15.31:0/3763097810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 msgr2=0x55b34f137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.997+0000 7f8f477fe700 1 -- 172.21.15.31:0/3763097810 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48109090 con 0x55b34f129c60 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f277fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f477fe700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f137f50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009340 tx=0x7f8f3c0110b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82124f5d77ea1a7e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f277fe700 1 -- 172.21.15.31:0/3763097810 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c0111a0 con 0x55b34f129c60 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b34f0a5d90 con 0x55b34f129c60 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f277fe700 1 -- 172.21.15.31:0/3763097810 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f3c0037f0 con 0x55b34f129c60 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f277fe700 1 -- 172.21.15.31:0/3763097810 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f3c011300 con 0x55b34f129c60 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f61e3fb80 10 client.27658 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.998+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8f3c01d040 con 0x55b34f129c60 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f277fe700 1 -- 172.21.15.31:0/3763097810 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f3c041020 con 0x55b34f129c60 2023-07-29T16:22:25.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f277fe700 1 -- 172.21.15.31:0/3763097810 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8f3c02b020 con 0x55b34f129c60 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f61e3fb80 10 client.27658 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f61e3fb80 10 client.27658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f61e3fb80 20 client.27658 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f12b370 0x55b34f194a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f61e3fb80 4 client.27658 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b34f073ae0 con 0x55b34f12b370 2023-07-29T16:22:25.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:24.999+0000 7f8f47fff700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f12b370 0x55b34f194a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.000+0000 7f8f47fff700 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f12b370 0x55b34f194a50 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86a9e7e1dec7c15b server_cookie=e810758d80ed2394 in_seq=0 out_seq=0 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.000+0000 7f8f277fe700 10 client.27658 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f277fe700 1 -- 172.21.15.31:0/3763097810 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b34f073ae0 con 0x55b34f12b370 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f277fe700 10 client.27658 handle_command_reply: tid=0 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 client.27658 shutdown 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 msgr2=0x55b34f137f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f137f50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c009340 tx=0x7f8f3c0110b0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 shutdown_connections 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b34f12b370 0x55b34f194a50 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b34f12a5b0 0x55b34f1910f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b34f11f460 0x55b34f137a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 --2- 172.21.15.31:0/3763097810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b34f129c60 0x55b34f137f50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 >> 172.21.15.31:0/3763097810 conn(0x55b34f1328c0 msgr2=0x55b34f1240e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f277fe700 0 client.27658 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 shutdown_connections 2023-07-29T16:22:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.001+0000 7f8f61e3fb80 1 -- 172.21.15.31:0/3763097810 wait complete. 2023-07-29T16:22:25.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f61e3fb80 20 client.27658 trim_cache size 0 max 16384 2023-07-29T16:22:25.008 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:22:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:22:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "eddf7cb5-414e-4b9d-9eaf-531eca2a1014", 2023-07-29T16:22:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:22:25.009 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:22:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f300a0790 msgr2=0x7f8f300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f300a0790 0x7f8f300a2c40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8f48075160 tx=0x7f8f40009590 comp rx=0 tx=0).stop 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 msgr2=0x7f8f48074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 0x7f8f48074b00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800e970 tx=0x7f8f3800ec80 comp rx=0 tx=0).stop 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 shutdown_connections 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f300a0790 0x7f8f300a2c40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f48101ea0 0x7f8f48073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f48100320 0x7f8f48074b00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 --2- 172.21.15.31:0/271710670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f48101510 0x7f8f48073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 >> 172.21.15.31:0/271710670 conn(0x7f8f480fb920 msgr2=0x7f8f480fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 shutdown_connections 2023-07-29T16:22:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.002+0000 7f8f4e5ff700 1 -- 172.21.15.31:0/271710670 wait complete. 2023-07-29T16:22:25.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:22:25.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 -- 172.21.15.31:0/3813549421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c074d90 msgr2=0x7fb85c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 --2- 172.21.15.31:0/3813549421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c074d90 0x7fb85c0731e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb858009990 tx=0x7fb858009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 -- 172.21.15.31:0/3813549421 shutdown_connections 2023-07-29T16:22:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 --2- 172.21.15.31:0/3813549421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c1081f0 0x7fb85c10a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 --2- 172.21.15.31:0/3813549421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c074d90 0x7fb85c0731e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 --2- 172.21.15.31:0/3813549421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 -- 172.21.15.31:0/3813549421 >> 172.21.15.31:0/3813549421 conn(0x7fb85c0fb970 msgr2=0x7fb85c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.237+0000 7fb8635ff700 1 -- 172.21.15.31:0/3813549421 shutdown_connections 2023-07-29T16:22:25.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.238+0000 7fb8635ff700 1 -- 172.21.15.31:0/3813549421 wait complete. 2023-07-29T16:22:25.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 Processor -- start 2023-07-29T16:22:25.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 -- start start 2023-07-29T16:22:25.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c1982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c074d90 0x7fb85c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c1081f0 0x7fb85c19f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c19fe70 con 0x7fb85c074d90 2023-07-29T16:22:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c19ffb0 con 0x7fb85c1081f0 2023-07-29T16:22:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.239+0000 7fb8635ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85c1a0140 con 0x7fb85c0737b0 2023-07-29T16:22:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c1982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb861b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c1081f0 0x7fb85c19f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb860b9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c074d90 0x7fb85c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c1982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54734/0 (socket says 172.21.15.31:54734) 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 -- 172.21.15.31:0/4163898051 learned_addr learned my addr 172.21.15.31:0/4163898051 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 -- 172.21.15.31:0/4163898051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c1081f0 msgr2=0x7fb85c19f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c1081f0 0x7fb85c19f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 -- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c074d90 msgr2=0x7fb85c198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c074d90 0x7fb85c198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 -- 172.21.15.31:0/4163898051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c009520 con 0x7fb85c0737b0 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb861b9c700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c1081f0 0x7fb85c19f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb86139b700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c1982f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb858004710 tx=0x7fb858009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.240+0000 7fb8527fc700 1 -- 172.21.15.31:0/4163898051 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85801d070 con 0x7fb85c0737b0 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.241+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8580095f0 con 0x7fb85c0737b0 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.241+0000 7fb8527fc700 1 -- 172.21.15.31:0/4163898051 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb858021410 con 0x7fb85c0737b0 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.241+0000 7fb8527fc700 1 -- 172.21.15.31:0/4163898051 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858017650 con 0x7fb85c0737b0 2023-07-29T16:22:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.241+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c1a06b0 con 0x7fb85c0737b0 2023-07-29T16:22:25.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.245+0000 7fb8527fc700 1 -- 172.21.15.31:0/4163898051 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb858021580 con 0x7fb85c0737b0 2023-07-29T16:22:25.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.245+0000 7fb876f7cb80 1 Processor -- start 2023-07-29T16:22:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 -- start start 2023-07-29T16:22:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa0eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa0fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa10740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a966a10 con 0x55a35aa0ebe0 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a9643d0 con 0x55a35aa0f530 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a95fcc0 con 0x55a35aa102f0 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb861b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa0eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb86139b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa0fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb860b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa10740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb8527fc700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8480a0790 0x7fb8480a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb861b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa0eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34288/0 (socket says 172.21.15.31:34288) 2023-07-29T16:22:25.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb86139b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa0fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44686/0 (socket says 172.21.15.31:44686) 2023-07-29T16:22:25.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.246+0000 7fb861b9c700 1 -- 172.21.15.31:0/4281330704 learned_addr learned my addr 172.21.15.31:0/4281330704 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb8527fc700 1 -- 172.21.15.31:0/4163898051 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb8580cceb0 con 0x7fb85c0737b0 2023-07-29T16:22:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb860b9a700 1 -- 172.21.15.31:0/4281330704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 msgr2=0x55a35aa0fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb860b9a700 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa0fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb860b9a700 1 -- 172.21.15.31:0/4281330704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 msgr2=0x55a35aa0eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb860b9a700 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa0eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb860b9a700 1 -- 172.21.15.31:0/4281330704 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a35a78dde0 con 0x55a35aa102f0 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb86139b700 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa0fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb861b9c700 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa0eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb861b9c700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8480a0790 0x7fb8480a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb860b9a700 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa10740 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009de0 tx=0x7fb84c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b31fbe210592b87 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb8477fe700 1 -- 172.21.15.31:0/4281330704 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c004250 con 0x55a35aa102f0 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb8477fe700 1 -- 172.21.15.31:0/4281330704 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb84c004cf0 con 0x55a35aa102f0 2023-07-29T16:22:25.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.247+0000 7fb8477fe700 1 -- 172.21.15.31:0/4281330704 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c005670 con 0x55a35aa102f0 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4281330704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 msgr2=0x55a35aa10740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb861b9c700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8480a0790 0x7fb8480a2c40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55a35aa0e980 tx=0x7fb854009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa10740 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009de0 tx=0x7fb84c011720 comp rx=0 tx=0).stop 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4281330704 shutdown_connections 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa10740 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa0eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4281330704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa0fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4281330704 >> 172.21.15.31:0/4281330704 conn(0x55a35aa043e0 msgr2=0x55a35aa09990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4281330704 shutdown_connections 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.248+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4281330704 wait complete. 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 -- start start 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 Processor -- start 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0ebe0 0x55a35aa187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 0x55a35aa18d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa102f0 0x55a35aa1fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a8fc510 con 0x55a35aa102f0 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a8fc1b0 con 0x55a35aa0ebe0 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.249+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35a7e3920 con 0x55a35aa0f530 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb876f7cb80 10 client.0 _ll_register_callbacks cb 0x55a35a905620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 0x55a35aa18d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 0x55a35aa18d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54752/0 (socket says 172.21.15.31:54752) 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 -- 172.21.15.31:0/4144049183 learned_addr learned my addr 172.21.15.31:0/4144049183 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:25.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb861b9c700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa102f0 0x55a35aa1fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb86139b700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0ebe0 0x55a35aa187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 -- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0ebe0 msgr2=0x55a35aa187d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0ebe0 0x55a35aa187d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 -- 172.21.15.31:0/4144049183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa102f0 msgr2=0x55a35aa1fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa102f0 0x55a35aa1fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.250+0000 7fb860b9a700 1 -- 172.21.15.31:0/4144049183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c1a06b0 con 0x55a35aa0f530 2023-07-29T16:22:25.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb860b9a700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 0x55a35aa18d10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009840 tx=0x7fb84c004fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c610c2e3d495da5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb844ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb844ff9700 1 -- 172.21.15.31:0/4144049183 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c004b20 con 0x55a35aa0f530 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb844ff9700 1 -- 172.21.15.31:0/4144049183 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb84c004050 con 0x55a35aa0f530 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a35a98ad10 con 0x55a35aa0f530 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb844ff9700 1 -- 172.21.15.31:0/4144049183 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb84c004210 con 0x55a35aa0f530 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb876f7cb80 10 client.27670 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fb84c009a40 con 0x55a35aa0f530 2023-07-29T16:22:25.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.251+0000 7fb844ff9700 1 -- 172.21.15.31:0/4144049183 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a35a98ad10 con 0x55a35aa0f530 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb844ff9700 1 -- 172.21.15.31:0/4144049183 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb84c0225c0 con 0x55a35aa0f530 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb876f7cb80 10 client.27670 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb876f7cb80 10 client.27670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb876f7cb80 20 client.27670 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa244c0 0x55a35aa268d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb876f7cb80 4 client.27670 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a35a9ddd70 con 0x55a35aa244c0 2023-07-29T16:22:25.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb86139b700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa244c0 0x55a35aa268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb844ff9700 10 client.27670 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:25.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.252+0000 7fb86139b700 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa244c0 0x55a35aa268d0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf7ef6e9d30fef42 server_cookie=2060c6b161a6a6ac in_seq=0 out_seq=0 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.253+0000 7fb844ff9700 1 -- 172.21.15.31:0/4144049183 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a35a9ddd70 con 0x55a35aa244c0 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.253+0000 7fb844ff9700 10 client.27670 handle_command_reply: tid=0 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.253+0000 7fb876f7cb80 1 client.27670 shutdown 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 msgr2=0x55a35aa18d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 0x55a35aa18d10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb84c009840 tx=0x7fb84c004fa0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 shutdown_connections 2023-07-29T16:22:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa244c0 0x55a35aa268d0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa102f0 0x55a35aa1fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0ebe0 0x55a35aa187d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/4144049183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0f530 0x55a35aa18d10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 >> 172.21.15.31:0/4144049183 conn(0x55a35aa043e0 msgr2=0x55a35aa09060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb844ff9700 0 client.27670 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 shutdown_connections 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 1 -- 172.21.15.31:0/4144049183 wait complete. 2023-07-29T16:22:25.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.254+0000 7fb876f7cb80 20 client.27670 trim_cache size 0 max 16384 2023-07-29T16:22:25.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.272+0000 7fb876f7cb80 1 Processor -- start 2023-07-29T16:22:25.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.272+0000 7fb876f7cb80 1 -- start start 2023-07-29T16:22:25.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0f530 0x55a35aa187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa102f0 0x55a35aa18d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35aa0ad20 con 0x55a35aa0f530 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35aa18370 con 0x55a35aa102f0 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35aa064b0 con 0x55a35aa0ebe0 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb861b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa102f0 0x55a35aa18d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54754/0 (socket says 172.21.15.31:54754) 2023-07-29T16:22:25.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb860b9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0f530 0x55a35aa187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 -- 172.21.15.31:0/2853501503 learned_addr learned my addr 172.21.15.31:0/2853501503 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:25.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb861b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa102f0 0x55a35aa18d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44716/0 (socket says 172.21.15.31:44716) 2023-07-29T16:22:25.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 -- 172.21.15.31:0/2853501503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa102f0 msgr2=0x55a35aa18d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa102f0 0x55a35aa18d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 -- 172.21.15.31:0/2853501503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0f530 msgr2=0x55a35aa187d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0f530 0x55a35aa187d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.273+0000 7fb86139b700 1 -- 172.21.15.31:0/2853501503 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a35a776c60 con 0x55a35aa0ebe0 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.274+0000 7fb86139b700 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1dd00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb858097050 tx=0x7fb8580be040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4f99d320b9acd47 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.274+0000 7fb847fff700 1 -- 172.21.15.31:0/2853501503 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858099f40 con 0x55a35aa0ebe0 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.274+0000 7fb847fff700 1 -- 172.21.15.31:0/2853501503 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb85809a5e0 con 0x55a35aa0ebe0 2023-07-29T16:22:25.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.274+0000 7fb847fff700 1 -- 172.21.15.31:0/2853501503 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85809a840 con 0x55a35aa0ebe0 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.274+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2853501503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 msgr2=0x55a35aa1dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.274+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1dd00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb858097050 tx=0x7fb8580be040 comp rx=0 tx=0).stop 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2853501503 shutdown_connections 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa102f0 0x55a35aa18d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1dd00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2853501503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0f530 0x55a35aa187d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2853501503 >> 172.21.15.31:0/2853501503 conn(0x55a35aa043e0 msgr2=0x55a35aa16640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2853501503 shutdown_connections 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2853501503 wait complete. 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 -- start start 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.275+0000 7fb876f7cb80 1 Processor -- start 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa1c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa1c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35aa1ccc0 con 0x55a35aa0ebe0 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35aa1ce30 con 0x55a35aa0f530 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35aa1cff0 con 0x55a35aa102f0 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb876f7cb80 10 client.0 _ll_register_callbacks cb 0x55a35aa2b750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb86139b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb860b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa1c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb86139b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34328/0 (socket says 172.21.15.31:34328) 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb86139b700 1 -- 172.21.15.31:0/2725348566 learned_addr learned my addr 172.21.15.31:0/2725348566 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:25.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb861b9c700 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa1c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb860b9a700 1 -- 172.21.15.31:0/2725348566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 msgr2=0x55a35aa1c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb860b9a700 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa1c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb860b9a700 1 -- 172.21.15.31:0/2725348566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 msgr2=0x55a35aa1db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb860b9a700 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.276+0000 7fb860b9a700 1 -- 172.21.15.31:0/2725348566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8580095f0 con 0x55a35aa0f530 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb860b9a700 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa1c160 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb84c022a20 tx=0x7fb84c005240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=101bccd1e1d8fce1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb844ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb844ff9700 1 -- 172.21.15.31:0/2725348566 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c004b20 con 0x55a35aa0f530 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb84c004cf0 con 0x55a35aa0f530 2023-07-29T16:22:25.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb844ff9700 1 -- 172.21.15.31:0/2725348566 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb84c022c90 con 0x55a35aa0f530 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb844ff9700 1 -- 172.21.15.31:0/2725348566 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb84c022610 con 0x55a35aa0f530 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb876f7cb80 10 client.27665 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a35a7b4250 con 0x55a35aa0f530 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.277+0000 7fb844ff9700 1 -- 172.21.15.31:0/2725348566 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb84c043020 con 0x55a35aa0f530 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb844ff9700 1 -- 172.21.15.31:0/2725348566 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb84c003710 con 0x55a35aa0f530 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb876f7cb80 10 client.27665 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb876f7cb80 10 client.27665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:25.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb876f7cb80 20 client.27665 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa775b0 0x55a35aa799c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb876f7cb80 4 client.27665 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a35a958d40 con 0x55a35aa775b0 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb86139b700 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa775b0 0x55a35aa799c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb86139b700 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa775b0 0x55a35aa799c0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1c71995c5a7eae8 server_cookie=993d0ff80ad4ee6a in_seq=0 out_seq=0 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.278+0000 7fb844ff9700 10 client.27665 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:25.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb844ff9700 1 -- 172.21.15.31:0/2725348566 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a35a958d40 con 0x55a35aa775b0 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb844ff9700 10 client.27665 handle_command_reply: tid=0 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 client.27665 shutdown 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 msgr2=0x55a35aa1c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa1c160 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb84c022a20 tx=0x7fb84c005240 comp rx=0 tx=0).stop 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 shutdown_connections 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a35aa775b0 0x55a35aa799c0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a35aa102f0 0x55a35aa1c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a35aa0ebe0 0x55a35aa1db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 --2- 172.21.15.31:0/2725348566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a35aa0f530 0x55a35aa1c160 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb844ff9700 0 client.27665 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 >> 172.21.15.31:0/2725348566 conn(0x55a35aa043e0 msgr2=0x55a35aa13150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 shutdown_connections 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 1 -- 172.21.15.31:0/2725348566 wait complete. 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.279+0000 7fb876f7cb80 20 client.27665 trim_cache size 0 max 16384 2023-07-29T16:22:25.286 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: "eddf7cb5-414e-4b9d-9eaf-531eca2a1014": { 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "eddf7cb5-414e-4b9d-9eaf-531eca2a1014", 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:22:25.287 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:22:25.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:22:25.288 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:22:25.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8480a0790 msgr2=0x7fb8480a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8480a0790 0x7fb8480a2c40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55a35aa0e980 tx=0x7fb854009040 comp rx=0 tx=0).stop 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 msgr2=0x7fb85c1982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c1982f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb858004710 tx=0x7fb858009580 comp rx=0 tx=0).stop 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 shutdown_connections 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8480a0790 0x7fb8480a2c40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb85c1081f0 0x7fb85c19f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb85c0737b0 0x7fb85c1982f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 --2- 172.21.15.31:0/4163898051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb85c074d90 0x7fb85c198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:25.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 >> 172.21.15.31:0/4163898051 conn(0x7fb85c0fb970 msgr2=0x7fb85c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:25.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 shutdown_connections 2023-07-29T16:22:25.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:25.280+0000 7fb8635ff700 1 -- 172.21.15.31:0/4163898051 wait complete. 2023-07-29T16:22:25.298 INFO:tasks.cephfs.filesystem:scrub status for tag:eddf7cb5-414e-4b9d-9eaf-531eca2a1014 - {'path': '/', 'tag': 'eddf7cb5-414e-4b9d-9eaf-531eca2a1014', 'options': 'recursive,force'} 2023-07-29T16:22:25.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:25 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:22:25.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:25 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:25.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:25 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:22:25.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:25 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:25.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 419153 8.93 MB/sec execute 130 sec latency 21.447 ms 2023-07-29T16:22:25.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:25 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:22:25.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:25 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:22:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:22:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:22:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:26 smithi039 ceph-mon[128858]: pgmap v579: 97 pgs: 97 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.3 MiB/s wr, 243 op/s 2023-07-29T16:22:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:22:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:22:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:22:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:26 smithi156 ceph-mon[129245]: pgmap v579: 97 pgs: 97 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.3 MiB/s wr, 243 op/s 2023-07-29T16:22:26.872 INFO:tasks.workunit.client.0.smithi031.stdout: 1 421342 8.95 MB/sec execute 131 sec latency 16.649 ms 2023-07-29T16:22:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:22:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:22:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:22:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:26.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:26 smithi031 ceph-mon[95504]: pgmap v579: 97 pgs: 97 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 2.3 MiB/s wr, 243 op/s 2023-07-29T16:22:27.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 422585 8.94 MB/sec execute 132 sec latency 72.032 ms 2023-07-29T16:22:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:27 smithi031 ceph-mon[95504]: pgmap v580: 97 pgs: 97 active+clean; 97 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 254 op/s 2023-07-29T16:22:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:27 smithi039 ceph-mon[128858]: pgmap v580: 97 pgs: 97 active+clean; 97 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 254 op/s 2023-07-29T16:22:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:27 smithi156 ceph-mon[129245]: pgmap v580: 97 pgs: 97 active+clean; 97 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 254 op/s 2023-07-29T16:22:28.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 424740 8.93 MB/sec execute 133 sec latency 22.817 ms 2023-07-29T16:22:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 -- 172.21.15.31:0/169958929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0100350 msgr2=0x7f9bd0100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/169958929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0100350 0x7f9bd0100760 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009990 tx=0x7f9bc0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 -- 172.21.15.31:0/169958929 shutdown_connections 2023-07-29T16:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/169958929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0101ed0 0x7f9bd0108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/169958929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0100350 0x7f9bd0100760 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/169958929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101540 0x7f9bd0101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 -- 172.21.15.31:0/169958929 >> 172.21.15.31:0/169958929 conn(0x7f9bd00fb970 msgr2=0x7f9bd00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 -- 172.21.15.31:0/169958929 shutdown_connections 2023-07-29T16:22:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.252+0000 7f9bd49ff700 1 -- 172.21.15.31:0/169958929 wait complete. 2023-07-29T16:22:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.253+0000 7f9bd49ff700 1 Processor -- start 2023-07-29T16:22:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.253+0000 7f9bd49ff700 1 -- start start 2023-07-29T16:22:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bd49ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 0x7f9bd0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bd49ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0101540 0x7f9bd0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bd49ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101ed0 0x7f9bd0073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bd49ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01a0120 con 0x7f9bd0101ed0 2023-07-29T16:22:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bd49ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01a0260 con 0x7f9bd0101540 2023-07-29T16:22:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bd49ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd01a03a0 con 0x7f9bd0100350 2023-07-29T16:22:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 0x7f9bd0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 0x7f9bd0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54780/0 (socket says 172.21.15.31:54780) 2023-07-29T16:22:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 -- 172.21.15.31:0/351389641 learned_addr learned my addr 172.21.15.31:0/351389641 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bcdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0101540 0x7f9bd0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bced9d700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101ed0 0x7f9bd0073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 -- 172.21.15.31:0/351389641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0101540 msgr2=0x7f9bd0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0101540 0x7f9bd0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 -- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101ed0 msgr2=0x7f9bd0073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101ed0 0x7f9bd0073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bce59c700 1 -- 172.21.15.31:0/351389641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00095f0 con 0x7f9bd0100350 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.254+0000 7f9bced9d700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101ed0 0x7f9bd0073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.255+0000 7f9bce59c700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 0x7f9bd0073140 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000b2e0 tx=0x7f9bc0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.255+0000 7f9bbf7fe700 1 -- 172.21.15.31:0/351389641 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc001d070 con 0x7f9bd0100350 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.255+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd01a05d0 con 0x7f9bd0100350 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.255+0000 7f9bbf7fe700 1 -- 172.21.15.31:0/351389641 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bc00047e0 con 0x7f9bd0100350 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.255+0000 7f9bbf7fe700 1 -- 172.21.15.31:0/351389641 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0005490 con 0x7f9bd0100350 2023-07-29T16:22:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.255+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd01a0990 con 0x7f9bd0100350 2023-07-29T16:22:29.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.258+0000 7f9bbf7fe700 1 -- 172.21.15.31:0/351389641 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9bc0004950 con 0x7f9bd0100350 2023-07-29T16:22:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.260+0000 7f9bbf7fe700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bb80a0790 0x7f9bb80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.260+0000 7f9bbf7fe700 1 -- 172.21.15.31:0/351389641 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9bc00ccdd0 con 0x7f9bd0100350 2023-07-29T16:22:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.260+0000 7f9be835ab80 1 Processor -- start 2023-07-29T16:22:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bb80a0790 0x7f9bb80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 -- start start 2023-07-29T16:22:29.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 0x55654acb68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb6df0 0x55654acb77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acb8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ac0e9b0 con 0x55654acb6df0 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ac0c370 con 0x55654acb7cf0 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9be835ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ac07c60 con 0x55654ab126b0 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 0x55654acb68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 0x55654acb68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54792/0 (socket says 172.21.15.31:54792) 2023-07-29T16:22:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9bce59c700 1 -- 172.21.15.31:0/326402515 learned_addr learned my addr 172.21.15.31:0/326402515 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9bced9d700 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acb8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.261+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb6df0 0x55654acb77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bce59c700 1 -- 172.21.15.31:0/326402515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 msgr2=0x55654acb8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bce59c700 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acb8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bce59c700 1 -- 172.21.15.31:0/326402515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb6df0 msgr2=0x55654acb77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bce59c700 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb6df0 0x55654acb77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bce59c700 1 -- 172.21.15.31:0/326402515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55654aa35d80 con 0x55654ab126b0 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bb80a0790 0x7f9bb80a2c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0074440 tx=0x7f9bc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bce59c700 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 0x55654acb68b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00a2390 tx=0x7f9bc009a580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f8981f5bb628df server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bbcff9700 1 -- 172.21.15.31:0/326402515 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0098640 con 0x55654ab126b0 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bbcff9700 1 -- 172.21.15.31:0/326402515 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bc0096040 con 0x55654ab126b0 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.262+0000 7f9bbcff9700 1 -- 172.21.15.31:0/326402515 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0099c50 con 0x55654ab126b0 2023-07-29T16:22:29.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 -- 172.21.15.31:0/326402515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 msgr2=0x55654acb68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 0x55654acb68b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00a2390 tx=0x7f9bc009a580 comp rx=0 tx=0).stop 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 -- 172.21.15.31:0/326402515 shutdown_connections 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acb8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654ab126b0 0x55654acb68b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 --2- 172.21.15.31:0/326402515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb6df0 0x55654acb77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 -- 172.21.15.31:0/326402515 >> 172.21.15.31:0/326402515 conn(0x55654acabfd0 msgr2=0x55654acb1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:29.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 -- 172.21.15.31:0/326402515 shutdown_connections 2023-07-29T16:22:29.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.263+0000 7f9be835ab80 1 -- 172.21.15.31:0/326402515 wait complete. 2023-07-29T16:22:29.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 -- start start 2023-07-29T16:22:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 Processor -- start 2023-07-29T16:22:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654acbfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acc7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654aba44b0 con 0x55654ab126b0 2023-07-29T16:22:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654aba4150 con 0x55654acb7cf0 2023-07-29T16:22:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654aa8b8c0 con 0x55654acb6df0 2023-07-29T16:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.264+0000 7f9be835ab80 10 client.0 _ll_register_callbacks cb 0x55654abad5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654acbfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acc7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acc7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44760/0 (socket says 172.21.15.31:44760) 2023-07-29T16:22:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 -- 172.21.15.31:0/1360487 learned_addr learned my addr 172.21.15.31:0/1360487 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 -- 172.21.15.31:0/1360487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 msgr2=0x55654acc00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 -- 172.21.15.31:0/1360487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 msgr2=0x55654acbfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654acbfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 -- 172.21.15.31:0/1360487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd01a0990 con 0x55654acb7cf0 2023-07-29T16:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bce59c700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654acbfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bced9d700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acc7110 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8007b30 tx=0x7f9bc8007b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca4ae5354cf3c04f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bb5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/1360487 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc800e070 con 0x55654acb7cf0 2023-07-29T16:22:29.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/1360487 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9bc8003a00 con 0x55654acb7cf0 2023-07-29T16:22:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.265+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55654ac32cb0 con 0x55654acb7cf0 2023-07-29T16:22:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/1360487 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bc8003bc0 con 0x55654acb7cf0 2023-07-29T16:22:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 10 client.27677 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:29.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55654aa1ec00 con 0x55654acb7cf0 2023-07-29T16:22:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/1360487 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55654ac32cb0 con 0x55654acb7cf0 2023-07-29T16:22:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/1360487 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9bc8016920 con 0x55654acb7cf0 2023-07-29T16:22:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 10 client.27677 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:29.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 10 client.27677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 20 client.27677 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654accb980 0x55654accdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 4 client.27677 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:29.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.266+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55654ac85d10 con 0x55654accb980 2023-07-29T16:22:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.267+0000 7f9bce59c700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654accb980 0x55654accdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.267+0000 7f9bce59c700 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654accb980 0x55654accdd90 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3573d3feb8f77ed7 server_cookie=1c11b666ad75cd38 in_seq=0 out_seq=0 2023-07-29T16:22:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.267+0000 7f9bb5ffb700 10 client.27677 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:29.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/1360487 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55654ac85d10 con 0x55654accb980 2023-07-29T16:22:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9bb5ffb700 10 client.27677 handle_command_reply: tid=0 2023-07-29T16:22:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 client.27677 shutdown 2023-07-29T16:22:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 msgr2=0x55654acc7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acc7110 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8007b30 tx=0x7f9bc8007b60 comp rx=0 tx=0).stop 2023-07-29T16:22:29.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 shutdown_connections 2023-07-29T16:22:29.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654accb980 0x55654accdd90 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654acc7110 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654acbfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 --2- 172.21.15.31:0/1360487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 >> 172.21.15.31:0/1360487 conn(0x55654acabfd0 msgr2=0x55654acb0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9bb5ffb700 0 client.27677 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:29.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 shutdown_connections 2023-07-29T16:22:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.268+0000 7f9be835ab80 1 -- 172.21.15.31:0/1360487 wait complete. 2023-07-29T16:22:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.269+0000 7f9be835ab80 20 client.27677 trim_cache size 0 max 16384 2023-07-29T16:22:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.285+0000 7f9be835ab80 1 Processor -- start 2023-07-29T16:22:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.285+0000 7f9be835ab80 1 -- start start 2023-07-29T16:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 0x55654acc50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb7cf0 0x55654acb0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9be835ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ac0c370 con 0x55654acb7cf0 2023-07-29T16:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9be835ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ac85d10 con 0x55654ab126b0 2023-07-29T16:22:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9be835ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ac32cb0 con 0x55654acb6df0 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 0x55654acc50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bcdd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 0x55654acc50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44774/0 (socket says 172.21.15.31:44774) 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 -- 172.21.15.31:0/3568954470 learned_addr learned my addr 172.21.15.31:0/3568954470 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bcdd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54820/0 (socket says 172.21.15.31:54820) 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bced9d700 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb7cf0 0x55654acb0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 -- 172.21.15.31:0/3568954470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 msgr2=0x55654acc0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 -- 172.21.15.31:0/3568954470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb7cf0 msgr2=0x55654acb0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb7cf0 0x55654acb0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.286+0000 7f9bce59c700 1 -- 172.21.15.31:0/3568954470 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55654aa5c1f0 con 0x55654ab126b0 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.287+0000 7f9bce59c700 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 0x55654acc50c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005000 tx=0x7f9bc0009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8b9d325dcc5f399 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.287+0000 7f9bbd7fa700 1 -- 172.21.15.31:0/3568954470 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0098640 con 0x55654ab126b0 2023-07-29T16:22:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.287+0000 7f9bbd7fa700 1 -- 172.21.15.31:0/3568954470 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bc00b6070 con 0x55654ab126b0 2023-07-29T16:22:29.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.287+0000 7f9bbd7fa700 1 -- 172.21.15.31:0/3568954470 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc009e290 con 0x55654ab126b0 2023-07-29T16:22:29.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.287+0000 7f9be835ab80 1 -- 172.21.15.31:0/3568954470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 msgr2=0x55654acc50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.287+0000 7f9be835ab80 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 0x55654acc50c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0005000 tx=0x7f9bc0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 -- 172.21.15.31:0/3568954470 shutdown_connections 2023-07-29T16:22:29.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654acb7cf0 0x55654acb0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654ab126b0 0x55654acc50c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 --2- 172.21.15.31:0/3568954470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654acc0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 -- 172.21.15.31:0/3568954470 >> 172.21.15.31:0/3568954470 conn(0x55654acabfd0 msgr2=0x55654acbf940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 -- 172.21.15.31:0/3568954470 shutdown_connections 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 -- 172.21.15.31:0/3568954470 wait complete. 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 -- start start 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.288+0000 7f9be835ab80 1 Processor -- start 2023-07-29T16:22:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654ad174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654ad19a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654ad1bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654acc54e0 con 0x55654ab126b0 2023-07-29T16:22:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ad1c4e0 con 0x55654acb7cf0 2023-07-29T16:22:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654ad1c6a0 con 0x55654acb6df0 2023-07-29T16:22:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9be835ab80 10 client.0 _ll_register_callbacks cb 0x55654abad5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654ad174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654ad1bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654ad19a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bce59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654ad174d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34382/0 (socket says 172.21.15.31:34382) 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bced9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654ad1bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44786/0 (socket says 172.21.15.31:44786) 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bce59c700 1 -- 172.21.15.31:0/4159542870 learned_addr learned my addr 172.21.15.31:0/4159542870 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654ad19a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54834/0 (socket says 172.21.15.31:54834) 2023-07-29T16:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 -- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 msgr2=0x55654ad1bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654ad1bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 -- 172.21.15.31:0/4159542870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 msgr2=0x55654ad174d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654ad174d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bcdd9b700 1 -- 172.21.15.31:0/4159542870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55654aa35d80 con 0x55654acb6df0 2023-07-29T16:22:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bced9d700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654ad1bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.289+0000 7f9bce59c700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654ad174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9bcdd9b700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654ad19a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4011fb0 tx=0x7f9bc40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=717ecd26b1a3bd4a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9bb5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/4159542870 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4004a30 con 0x55654acb6df0 2023-07-29T16:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bc0096040 con 0x55654acb6df0 2023-07-29T16:22:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/4159542870 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9bc4004b90 con 0x55654acb6df0 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/4159542870 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bc4004d50 con 0x55654acb6df0 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9be835ab80 10 client.27694 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.290+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55654aa34250 con 0x55654acb6df0 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/4159542870 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bc4036080 con 0x55654acb6df0 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/4159542870 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9bc401d670 con 0x55654acb6df0 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9be835ab80 10 client.27694 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9be835ab80 10 client.27694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9be835ab80 20 client.27694 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9be835ab80 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654ad1e1a0 0x55654ad205b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9be835ab80 4 client.27694 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55654ac00ce0 con 0x55654ad1e1a0 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.291+0000 7f9bce59c700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654ad1e1a0 0x55654ad205b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.292+0000 7f9bce59c700 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654ad1e1a0 0x55654ad205b0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e8feb3ddb17818 server_cookie=bac337bde1f723af in_seq=0 out_seq=0 2023-07-29T16:22:29.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.292+0000 7f9bb5ffb700 10 client.27694 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.365+0000 7f9bb5ffb700 1 -- 172.21.15.31:0/4159542870 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55654ac00ce0 con 0x55654ad1e1a0 2023-07-29T16:22:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.365+0000 7f9bb5ffb700 10 client.27694 handle_command_reply: tid=0 2023-07-29T16:22:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 client.27694 shutdown 2023-07-29T16:22:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 msgr2=0x55654ad19a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654ad19a20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4011fb0 tx=0x7f9bc40047c0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 shutdown_connections 2023-07-29T16:22:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55654ad1e1a0 0x55654ad205b0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55654acb7cf0 0x55654ad1bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55654ab126b0 0x55654ad174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 --2- 172.21.15.31:0/4159542870 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55654acb6df0 0x55654ad19a20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9bb5ffb700 0 client.27694 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 >> 172.21.15.31:0/4159542870 conn(0x55654acabfd0 msgr2=0x55654acb0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 shutdown_connections 2023-07-29T16:22:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 1 -- 172.21.15.31:0/4159542870 wait complete. 2023-07-29T16:22:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.366+0000 7f9be835ab80 20 client.27694 trim_cache size 0 max 16384 2023-07-29T16:22:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:22:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:22:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:22:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:22:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bb80a0790 msgr2=0x7f9bb80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bb80a0790 0x7f9bb80a2c40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0074440 tx=0x7f9bc4009260 comp rx=0 tx=0).stop 2023-07-29T16:22:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 msgr2=0x7f9bd0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 0x7f9bd0073140 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000b2e0 tx=0x7f9bc0009580 comp rx=0 tx=0).stop 2023-07-29T16:22:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 shutdown_connections 2023-07-29T16:22:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bb80a0790 0x7f9bb80a2c40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bd0101ed0 0x7f9bd0073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bd0100350 0x7f9bd0073140 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 --2- 172.21.15.31:0/351389641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bd0101540 0x7f9bd0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 >> 172.21.15.31:0/351389641 conn(0x7f9bd00fb970 msgr2=0x7f9bd00fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 shutdown_connections 2023-07-29T16:22:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:29.367+0000 7f9bd49ff700 1 -- 172.21.15.31:0/351389641 wait complete. 2023-07-29T16:22:29.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 426116 8.95 MB/sec execute 134 sec latency 72.645 ms 2023-07-29T16:22:30.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:30 smithi039 ceph-mon[128858]: pgmap v581: 97 pgs: 97 active+clean; 96 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2023-07-29T16:22:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:30 smithi156 ceph-mon[129245]: pgmap v581: 97 pgs: 97 active+clean; 96 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2023-07-29T16:22:30.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 428092 8.94 MB/sec execute 135 sec latency 16.649 ms 2023-07-29T16:22:30.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:30 smithi031 ceph-mon[95504]: pgmap v581: 97 pgs: 97 active+clean; 96 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2023-07-29T16:22:31.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 429649 8.96 MB/sec execute 136 sec latency 17.929 ms 2023-07-29T16:22:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:32 smithi039 ceph-mon[128858]: pgmap v582: 97 pgs: 97 active+clean; 96 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2023-07-29T16:22:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:32 smithi156 ceph-mon[129245]: pgmap v582: 97 pgs: 97 active+clean; 96 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2023-07-29T16:22:32.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 431556 8.94 MB/sec execute 137 sec latency 45.970 ms 2023-07-29T16:22:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:32 smithi031 ceph-mon[95504]: pgmap v582: 97 pgs: 97 active+clean; 96 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 1.9 MiB/s wr, 193 op/s 2023-07-29T16:22:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:33 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:22:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:33 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:22:33.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:33 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:22:33.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:33 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:22:33.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 433147 8.97 MB/sec execute 138 sec latency 14.923 ms 2023-07-29T16:22:33.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:33 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:22:33.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:33 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:22:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:34 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:22:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:34 smithi039 ceph-mon[128858]: pgmap v583: 97 pgs: 97 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2023-07-29T16:22:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:34 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:22:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:34 smithi156 ceph-mon[129245]: pgmap v583: 97 pgs: 97 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2023-07-29T16:22:34.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 434806 8.94 MB/sec execute 139 sec latency 12.393 ms 2023-07-29T16:22:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:34 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:22:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:34 smithi031 ceph-mon[95504]: pgmap v583: 97 pgs: 97 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2023-07-29T16:22:35.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 436557 8.97 MB/sec execute 140 sec latency 119.061 ms 2023-07-29T16:22:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:36 smithi039 ceph-mon[128858]: pgmap v584: 97 pgs: 97 active+clean; 96 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2023-07-29T16:22:36.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:36 smithi156 ceph-mon[129245]: pgmap v584: 97 pgs: 97 active+clean; 96 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2023-07-29T16:22:36.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 437962 8.95 MB/sec execute 141 sec latency 16.235 ms 2023-07-29T16:22:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:36 smithi031 ceph-mon[95504]: pgmap v584: 97 pgs: 97 active+clean; 96 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2023-07-29T16:22:37.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 440067 8.98 MB/sec execute 142 sec latency 54.848 ms 2023-07-29T16:22:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:37 smithi031 ceph-mon[95504]: pgmap v585: 97 pgs: 97 active+clean; 97 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 264 op/s 2023-07-29T16:22:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:37 smithi039 ceph-mon[128858]: pgmap v585: 97 pgs: 97 active+clean; 97 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 264 op/s 2023-07-29T16:22:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:37 smithi156 ceph-mon[129245]: pgmap v585: 97 pgs: 97 active+clean; 97 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 264 op/s 2023-07-29T16:22:38.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 441240 8.96 MB/sec execute 143 sec latency 49.116 ms 2023-07-29T16:22:39.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 443540 8.98 MB/sec execute 144 sec latency 22.250 ms 2023-07-29T16:22:40.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:40 smithi039 ceph-mon[128858]: pgmap v586: 97 pgs: 97 active+clean; 99 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 324 op/s 2023-07-29T16:22:40.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:40 smithi156 ceph-mon[129245]: pgmap v586: 97 pgs: 97 active+clean; 99 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 324 op/s 2023-07-29T16:22:40.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 444570 8.94 MB/sec execute 145 sec latency 62.787 ms 2023-07-29T16:22:40.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:40 smithi031 ceph-mon[95504]: pgmap v586: 97 pgs: 97 active+clean; 99 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 324 op/s 2023-07-29T16:22:41.873 INFO:tasks.workunit.client.0.smithi031.stdout: 1 447019 8.99 MB/sec execute 146 sec latency 16.569 ms 2023-07-29T16:22:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 -- 172.21.15.31:0/3156891621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 msgr2=0x7fbaf0101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/3156891621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 0x7fbaf0101980 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fbae0009990 tx=0x7fbae0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 -- 172.21.15.31:0/3156891621 shutdown_connections 2023-07-29T16:22:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/3156891621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf0101ec0 0x7fbaf0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/3156891621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/3156891621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 0x7fbaf0101980 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 -- 172.21.15.31:0/3156891621 >> 172.21.15.31:0/3156891621 conn(0x7fbaf00fb920 msgr2=0x7fbaf00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 -- 172.21.15.31:0/3156891621 shutdown_connections 2023-07-29T16:22:42.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.529+0000 7fbaf87ff700 1 -- 172.21.15.31:0/3156891621 wait complete. 2023-07-29T16:22:42.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.530+0000 7fbaf87ff700 1 Processor -- start 2023-07-29T16:22:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.530+0000 7fbaf87ff700 1 -- start start 2023-07-29T16:22:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf87ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf87ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 0x7fbaf0198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf87ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf0101ec0 0x7fbaf01996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf87ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0199d50 con 0x7fbaf0101ec0 2023-07-29T16:22:42.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf87ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf0199ec0 con 0x7fbaf0100340 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf87ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf019a080 con 0x7fbaf0101530 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf5d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 0x7fbaf0198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37392/0 (socket says 172.21.15.31:37392) 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 -- 172.21.15.31:0/2429380542 learned_addr learned my addr 172.21.15.31:0/2429380542 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf6d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf0101ec0 0x7fbaf01996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 -- 172.21.15.31:0/2429380542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 msgr2=0x7fbaf0198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 0x7fbaf0198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 -- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf0101ec0 msgr2=0x7fbaf01996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf0101ec0 0x7fbaf01996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.531+0000 7fbaf659b700 1 -- 172.21.15.31:0/2429380542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaec009520 con 0x7fbaf0100340 2023-07-29T16:22:42.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.532+0000 7fbaf659b700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0198330 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00ea50 tx=0x7fbaec00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.532+0000 7fbae77fe700 1 -- 172.21.15.31:0/2429380542 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec00cbd0 con 0x7fbaf0100340 2023-07-29T16:22:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.532+0000 7fbae77fe700 1 -- 172.21.15.31:0/2429380542 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbaec00cd30 con 0x7fbaf0100340 2023-07-29T16:22:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.532+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae00095f0 con 0x7fbaf0100340 2023-07-29T16:22:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.532+0000 7fbae77fe700 1 -- 172.21.15.31:0/2429380542 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec018680 con 0x7fbaf0100340 2023-07-29T16:22:42.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.532+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf019a550 con 0x7fbaf0100340 2023-07-29T16:22:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.535+0000 7fbae77fe700 1 -- 172.21.15.31:0/2429380542 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbaec010430 con 0x7fbaf0100340 2023-07-29T16:22:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.537+0000 7fbae77fe700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbadc0a0810 0x7fbadc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.537+0000 7fbae77fe700 1 -- 172.21.15.31:0/2429380542 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbaec014070 con 0x7fbaf0100340 2023-07-29T16:22:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.537+0000 7fbaf5d9a700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbadc0a0810 0x7fbadc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.537+0000 7fbb0c1f6b80 1 Processor -- start 2023-07-29T16:22:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 -- start start 2023-07-29T16:22:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbaf5d9a700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbadc0a0810 0x7fbadc0a2cc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbae000b2e0 tx=0x7fbae0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918a5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa917fb9b0 con 0x55fa918a3df0 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa917f9370 con 0x55fa918a4cf0 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa917f4c60 con 0x55fa916ff6b0 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbaf6d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbaf5d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49834/0 (socket says 172.21.15.31:49834) 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.538+0000 7fbaf659b700 1 -- 172.21.15.31:0/4063331608 learned_addr learned my addr 172.21.15.31:0/4063331608 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbaf659b700 1 -- 172.21.15.31:0/4063331608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 msgr2=0x55fa918a5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbaf659b700 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918a5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbaf659b700 1 -- 172.21.15.31:0/4063331608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 msgr2=0x55fa918a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbaf659b700 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbaf659b700 1 -- 172.21.15.31:0/4063331608 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa91622d80 con 0x55fa916ff6b0 2023-07-29T16:22:42.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbaf659b700 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918a38b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbaec095c10 tx=0x7fbaec099320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e1cdd3c799056b7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbae4ff9700 1 -- 172.21.15.31:0/4063331608 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0b1070 con 0x55fa916ff6b0 2023-07-29T16:22:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbae4ff9700 1 -- 172.21.15.31:0/4063331608 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbaec099710 con 0x55fa916ff6b0 2023-07-29T16:22:42.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.539+0000 7fbae4ff9700 1 -- 172.21.15.31:0/4063331608 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0a1930 con 0x55fa916ff6b0 2023-07-29T16:22:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/4063331608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 msgr2=0x55fa918a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918a38b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fbaec095c10 tx=0x7fbaec099320 comp rx=0 tx=0).stop 2023-07-29T16:22:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/4063331608 shutdown_connections 2023-07-29T16:22:42.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918a38b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/4063331608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/4063331608 >> 172.21.15.31:0/4063331608 conn(0x55fa91898fd0 msgr2=0x55fa9189e300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/4063331608 shutdown_connections 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.540+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/4063331608 wait complete. 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.541+0000 7fbb0c1f6b80 1 -- start start 2023-07-29T16:22:42.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.541+0000 7fbb0c1f6b80 1 Processor -- start 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa916ff6b0 0x55fa918acd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 0x55fa918b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa917914b0 con 0x55fa918a3df0 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa91791150 con 0x55fa916ff6b0 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa916788c0 con 0x55fa918a4cf0 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbb0c1f6b80 10 client.0 _ll_register_callbacks cb 0x55fa9179a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:42.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbaf6d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 0x55fa918b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbaf6d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 0x55fa918b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49838/0 (socket says 172.21.15.31:49838) 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbaf6d9c700 1 -- 172.21.15.31:0/3299488180 learned_addr learned my addr 172.21.15.31:0/3299488180 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbaf659b700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa916ff6b0 0x55fa918acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.542+0000 7fbaf5d9a700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf6d9c700 1 -- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa916ff6b0 msgr2=0x55fa918acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf6d9c700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa916ff6b0 0x55fa918acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf6d9c700 1 -- 172.21.15.31:0/3299488180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 msgr2=0x55fa918ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf6d9c700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918ad2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf6d9c700 1 -- 172.21.15.31:0/3299488180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf019a550 con 0x55fa918a4cf0 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf659b700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa916ff6b0 0x55fa918acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbaf6d9c700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 0x55fa918b42e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbae8012e20 tx=0x7fbae8014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4038194c452a7c70 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbad9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbad9ffb700 1 -- 172.21.15.31:0/3299488180 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae800e040 con 0x55fa918a4cf0 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa9181fcb0 con 0x55fa918a4cf0 2023-07-29T16:22:42.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.543+0000 7fbad9ffb700 1 -- 172.21.15.31:0/3299488180 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbae8014b90 con 0x55fa918a4cf0 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbad9ffb700 1 -- 172.21.15.31:0/3299488180 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae8014d50 con 0x55fa918a4cf0 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 10 client.27706 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fa9160bc00 con 0x55fa918a4cf0 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbad9ffb700 1 -- 172.21.15.31:0/3299488180 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fa9181fcb0 con 0x55fa918a4cf0 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbad9ffb700 1 -- 172.21.15.31:0/3299488180 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbae801c020 con 0x55fa918a4cf0 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 10 client.27706 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:42.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 10 client.27706 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 20 client.27706 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa918b8b60 0x55fa918baf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 4 client.27706 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.544+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa91872d10 con 0x55fa918b8b60 2023-07-29T16:22:42.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.545+0000 7fbaf659b700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa918b8b60 0x55fa918baf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.545+0000 7fbaf659b700 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa918b8b60 0x55fa918baf70 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad1421fb553843ed server_cookie=a1bd048bf9f9a790 in_seq=0 out_seq=0 2023-07-29T16:22:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.545+0000 7fbad9ffb700 10 client.27706 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbad9ffb700 1 -- 172.21.15.31:0/3299488180 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55fa91872d10 con 0x55fa918b8b60 2023-07-29T16:22:42.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbad9ffb700 10 client.27706 handle_command_reply: tid=0 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 client.27706 shutdown 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 msgr2=0x55fa918b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 0x55fa918b42e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbae8012e20 tx=0x7fbae8014600 comp rx=0 tx=0).stop 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 shutdown_connections 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa918b8b60 0x55fa918baf70 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa918a4cf0 0x55fa918b42e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa916ff6b0 0x55fa918acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbad9ffb700 0 client.27706 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/3299488180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 >> 172.21.15.31:0/3299488180 conn(0x55fa91898fd0 msgr2=0x55fa9189da40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 shutdown_connections 2023-07-29T16:22:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/3299488180 wait complete. 2023-07-29T16:22:42.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.546+0000 7fbb0c1f6b80 20 client.27706 trim_cache size 0 max 16384 2023-07-29T16:22:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.561+0000 7fbb0c1f6b80 1 Processor -- start 2023-07-29T16:22:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 -- start start 2023-07-29T16:22:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918acd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa918ad7d0 con 0x55fa918a3df0 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa918ac900 con 0x55fa918a4cf0 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa9189f790 con 0x55fa916ff6b0 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbaf5d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918acd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49854/0 (socket says 172.21.15.31:49854) 2023-07-29T16:22:42.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbaf659b700 1 -- 172.21.15.31:0/2706559374 learned_addr learned my addr 172.21.15.31:0/2706559374 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.562+0000 7fbaf6d9c700 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf659b700 1 -- 172.21.15.31:0/2706559374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 msgr2=0x55fa918ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf659b700 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf659b700 1 -- 172.21.15.31:0/2706559374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 msgr2=0x55fa918b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf659b700 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf659b700 1 -- 172.21.15.31:0/2706559374 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa916491f0 con 0x55fa916ff6b0 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf6d9c700 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbaf659b700 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918acd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005950 tx=0x7fbaec01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a338715c10558d35 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbae57fa700 1 -- 172.21.15.31:0/2706559374 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec0b1040 con 0x55fa916ff6b0 2023-07-29T16:22:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbae57fa700 1 -- 172.21.15.31:0/2706559374 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbaec09d740 con 0x55fa916ff6b0 2023-07-29T16:22:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.563+0000 7fbae57fa700 1 -- 172.21.15.31:0/2706559374 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec010940 con 0x55fa916ff6b0 2023-07-29T16:22:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/2706559374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 msgr2=0x55fa918acd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918acd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbaec005950 tx=0x7fbaec01a040 comp rx=0 tx=0).stop 2023-07-29T16:22:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/2706559374 shutdown_connections 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a4cf0 0x55fa918ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918acd20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/2706559374 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a3df0 0x55fa918b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/2706559374 >> 172.21.15.31:0/2706559374 conn(0x55fa91898fd0 msgr2=0x55fa918a7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/2706559374 shutdown_connections 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/2706559374 wait complete. 2023-07-29T16:22:42.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.564+0000 7fbb0c1f6b80 1 -- start start 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 Processor -- start 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 0x55fa918b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a4cf0 0x55fa918b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa918b3ac0 con 0x55fa918a4cf0 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa918b3c30 con 0x55fa918a3df0 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa918b3df0 con 0x55fa916ff6b0 2023-07-29T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbb0c1f6b80 10 client.0 _ll_register_callbacks cb 0x55fa9179a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.565+0000 7fbaf5d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 0x55fa918b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49862/0 (socket says 172.21.15.31:49862) 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf659b700 1 -- 172.21.15.31:0/1212128830 learned_addr learned my addr 172.21.15.31:0/1212128830 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 0x55fa918b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37432/0 (socket says 172.21.15.31:37432) 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf6d9c700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a4cf0 0x55fa918b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 -- 172.21.15.31:0/1212128830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 msgr2=0x55fa918b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 -- 172.21.15.31:0/1212128830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a4cf0 msgr2=0x55fa918b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a4cf0 0x55fa918b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 -- 172.21.15.31:0/1212128830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa9160bc00 con 0x55fa918a3df0 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf659b700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbaf5d9a700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 0x55fa918b0970 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000c00 tx=0x7fbae0009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cea26d25d8d1636e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:42.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.566+0000 7fbad9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbad9ffb700 1 -- 172.21.15.31:0/1212128830 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbae002d030 con 0x55fa918a3df0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbaec099710 con 0x55fa918a3df0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbad9ffb700 1 -- 172.21.15.31:0/1212128830 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbae000f870 con 0x55fa918a3df0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbad9ffb700 1 -- 172.21.15.31:0/1212128830 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbae000fa30 con 0x55fa918a3df0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 10 client.27701 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fa91622d80 con 0x55fa918a3df0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbad9ffb700 1 -- 172.21.15.31:0/1212128830 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbaec099710 con 0x55fa918a3df0 2023-07-29T16:22:42.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbad9ffb700 1 -- 172.21.15.31:0/1212128830 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbae0034bb0 con 0x55fa918a3df0 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 10 client.27701 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 10 client.27701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 20 client.27701 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa9190b590 0x55fa9190d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 4 client.27701 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.567+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fa917edce0 con 0x55fa9190b590 2023-07-29T16:22:42.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.568+0000 7fbaf659b700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa9190b590 0x55fa9190d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.568+0000 7fbaf659b700 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa9190b590 0x55fa9190d9a0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20a1696c879ba3a2 server_cookie=35fa7dc7c8c66e7b in_seq=0 out_seq=0 2023-07-29T16:22:42.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.568+0000 7fbad9ffb700 10 client.27701 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbad9ffb700 1 -- 172.21.15.31:0/1212128830 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fa917edce0 con 0x55fa9190b590 2023-07-29T16:22:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbad9ffb700 10 client.27701 handle_command_reply: tid=0 2023-07-29T16:22:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 client.27701 shutdown 2023-07-29T16:22:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 msgr2=0x55fa918b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 0x55fa918b0970 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbae0000c00 tx=0x7fbae0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 shutdown_connections 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa9190b590 0x55fa9190d9a0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa918a4cf0 0x55fa918b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa916ff6b0 0x55fa918b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 --2- 172.21.15.31:0/1212128830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa918a3df0 0x55fa918b0970 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbad9ffb700 0 client.27701 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:42.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 >> 172.21.15.31:0/1212128830 conn(0x55fa91898fd0 msgr2=0x55fa918a7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 shutdown_connections 2023-07-29T16:22:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 1 -- 172.21.15.31:0/1212128830 wait complete. 2023-07-29T16:22:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.699+0000 7fbb0c1f6b80 20 client.27701 trim_cache size 0 max 16384 2023-07-29T16:22:42.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:22:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:22:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:22:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:22:42.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbadc0a0810 msgr2=0x7fbadc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbadc0a0810 0x7fbadc0a2cc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbae000b2e0 tx=0x7fbae0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 msgr2=0x7fbaf0198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0198330 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00ea50 tx=0x7fbaec00ed60 comp rx=0 tx=0).stop 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 shutdown_connections 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbadc0a0810 0x7fbadc0a2cc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbaf0101ec0 0x7fbaf01996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbaf0100340 0x7fbaf0198330 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 --2- 172.21.15.31:0/2429380542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaf0101530 0x7fbaf0198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 >> 172.21.15.31:0/2429380542 conn(0x7fbaf00fb920 msgr2=0x7fbaf0106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 shutdown_connections 2023-07-29T16:22:42.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:42.700+0000 7fbaf87ff700 1 -- 172.21.15.31:0/2429380542 wait complete. 2023-07-29T16:22:42.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:42 smithi039 ceph-mon[128858]: pgmap v587: 97 pgs: 97 active+clean; 99 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2023-07-29T16:22:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:42 smithi156 ceph-mon[129245]: pgmap v587: 97 pgs: 97 active+clean; 99 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2023-07-29T16:22:42.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 447738 8.94 MB/sec execute 147 sec latency 85.755 ms 2023-07-29T16:22:42.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:42 smithi031 ceph-mon[95504]: pgmap v587: 97 pgs: 97 active+clean; 99 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 250 op/s 2023-07-29T16:22:43.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 449968 8.96 MB/sec execute 148 sec latency 21.020 ms 2023-07-29T16:22:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:44 smithi039 ceph-mon[128858]: pgmap v588: 97 pgs: 97 active+clean; 100 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 281 op/s 2023-07-29T16:22:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:44 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:22:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:44 smithi156 ceph-mon[129245]: pgmap v588: 97 pgs: 97 active+clean; 100 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 281 op/s 2023-07-29T16:22:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:44 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:22:44.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 451161 8.95 MB/sec execute 149 sec latency 11.976 ms 2023-07-29T16:22:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:44 smithi031 ceph-mon[95504]: pgmap v588: 97 pgs: 97 active+clean; 100 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 281 op/s 2023-07-29T16:22:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:44 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:22:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:22:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:22:45.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:45 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:22:45.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:22:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:22:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:45 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:22:45.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 453263 8.96 MB/sec execute 150 sec latency 30.183 ms 2023-07-29T16:22:45.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:22:45.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:22:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:45 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:22:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:46 smithi039 ceph-mon[128858]: pgmap v589: 97 pgs: 97 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2023-07-29T16:22:46.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:46 smithi156 ceph-mon[129245]: pgmap v589: 97 pgs: 97 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2023-07-29T16:22:46.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 454581 8.96 MB/sec execute 151 sec latency 13.785 ms 2023-07-29T16:22:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:46 smithi031 ceph-mon[95504]: pgmap v589: 97 pgs: 97 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 265 op/s 2023-07-29T16:22:47.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 456480 8.94 MB/sec execute 152 sec latency 32.263 ms 2023-07-29T16:22:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:47 smithi031 ceph-mon[95504]: pgmap v590: 97 pgs: 97 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 251 op/s 2023-07-29T16:22:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:47 smithi039 ceph-mon[128858]: pgmap v590: 97 pgs: 97 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 251 op/s 2023-07-29T16:22:48.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:47 smithi156 ceph-mon[129245]: pgmap v590: 97 pgs: 97 active+clean; 104 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 251 op/s 2023-07-29T16:22:48.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 457993 8.96 MB/sec execute 153 sec latency 26.270 ms 2023-07-29T16:22:49.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 459015 9.04 MB/sec execute 154 sec latency 64.998 ms 2023-07-29T16:22:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:50 smithi039 ceph-mon[128858]: pgmap v591: 97 pgs: 97 active+clean; 101 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 311 op/s 2023-07-29T16:22:50.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:50 smithi156 ceph-mon[129245]: pgmap v591: 97 pgs: 97 active+clean; 101 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 311 op/s 2023-07-29T16:22:50.874 INFO:tasks.workunit.client.0.smithi031.stdout: 1 461359 9.07 MB/sec execute 155 sec latency 12.511 ms 2023-07-29T16:22:50.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:50 smithi031 ceph-mon[95504]: pgmap v591: 97 pgs: 97 active+clean; 101 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 311 op/s 2023-07-29T16:22:51.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 462461 9.05 MB/sec execute 156 sec latency 93.079 ms 2023-07-29T16:22:52.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:52 smithi039 ceph-mon[128858]: pgmap v592: 97 pgs: 97 active+clean; 101 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 1.9 MiB/s wr, 221 op/s 2023-07-29T16:22:52.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:52 smithi156 ceph-mon[129245]: pgmap v592: 97 pgs: 97 active+clean; 101 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 1.9 MiB/s wr, 221 op/s 2023-07-29T16:22:52.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 464779 9.06 MB/sec execute 157 sec latency 14.353 ms 2023-07-29T16:22:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:52 smithi031 ceph-mon[95504]: pgmap v592: 97 pgs: 97 active+clean; 101 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 1.9 MiB/s wr, 221 op/s 2023-07-29T16:22:53.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 465952 9.06 MB/sec execute 158 sec latency 50.510 ms 2023-07-29T16:22:54.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:54 smithi039 ceph-mon[128858]: pgmap v593: 97 pgs: 97 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-07-29T16:22:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:54 smithi156 ceph-mon[129245]: pgmap v593: 97 pgs: 97 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-07-29T16:22:54.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 468205 9.07 MB/sec execute 159 sec latency 18.269 ms 2023-07-29T16:22:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:54 smithi031 ceph-mon[95504]: pgmap v593: 97 pgs: 97 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-07-29T16:22:55.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:22:55.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.522+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/1022398569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4101510 msgr2=0x7fc6b4101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.522+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/1022398569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4101510 0x7fc6b4101960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009990 tx=0x7fc6a4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/1022398569 shutdown_connections 2023-07-29T16:22:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/1022398569 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b4101ea0 0x7fc6b4108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/1022398569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4100320 0x7fc6b4100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/1022398569 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4101510 0x7fc6b4101960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/1022398569 >> 172.21.15.31:0/1022398569 conn(0x7fc6b40fb920 msgr2=0x7fc6b40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/1022398569 shutdown_connections 2023-07-29T16:22:55.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.523+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/1022398569 wait complete. 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.524+0000 7fc6bb5ff700 1 Processor -- start 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.524+0000 7fc6bb5ff700 1 -- start start 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6bb5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 0x7fc6b41982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6bb5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4101510 0x7fc6b4198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6bb5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b4101ea0 0x7fc6b419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6bb5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b419fe60 con 0x7fc6b4101510 2023-07-29T16:22:55.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6bb5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b419ffa0 con 0x7fc6b4100320 2023-07-29T16:22:55.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6bb5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41a0130 con 0x7fc6b4101ea0 2023-07-29T16:22:55.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 0x7fc6b41982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 0x7fc6b41982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35164/0 (socket says 172.21.15.31:35164) 2023-07-29T16:22:55.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 -- 172.21.15.31:0/3451766395 learned_addr learned my addr 172.21.15.31:0/3451766395 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b4101ea0 0x7fc6b419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4101510 0x7fc6b4198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 -- 172.21.15.31:0/3451766395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b4101ea0 msgr2=0x7fc6b419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b4101ea0 0x7fc6b419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 -- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4101510 msgr2=0x7fc6b4198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4101510 0x7fc6b4198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b939b700 1 -- 172.21.15.31:0/3451766395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a40095f0 con 0x7fc6b4100320 2023-07-29T16:22:55.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.525+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4101510 0x7fc6b4198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.526+0000 7fc6b939b700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 0x7fc6b41982e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000e970 tx=0x7fc6b000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.526+0000 7fc6aa7fc700 1 -- 172.21.15.31:0/3451766395 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0009790 con 0x7fc6b4100320 2023-07-29T16:22:55.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.526+0000 7fc6aa7fc700 1 -- 172.21.15.31:0/3451766395 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6b000eec0 con 0x7fc6b4100320 2023-07-29T16:22:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.526+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b41a0410 con 0x7fc6b4100320 2023-07-29T16:22:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.526+0000 7fc6aa7fc700 1 -- 172.21.15.31:0/3451766395 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0003cf0 con 0x7fc6b4100320 2023-07-29T16:22:55.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.526+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b41a0920 con 0x7fc6b4100320 2023-07-29T16:22:55.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.530+0000 7fc6aa7fc700 1 -- 172.21.15.31:0/3451766395 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc6b000ccd0 con 0x7fc6b4100320 2023-07-29T16:22:55.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 Processor -- start 2023-07-29T16:22:55.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 -- start start 2023-07-29T16:22:55.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190237ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190238d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6aa7fc700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc6a00a0790 0x7fc6a00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190239740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c19018fa10 con 0x55c190237be0 2023-07-29T16:22:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c19018d3d0 con 0x55c1902392f0 2023-07-29T16:22:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c190188cc0 con 0x55c190238530 2023-07-29T16:22:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.531+0000 7fc6aa7fc700 1 -- 172.21.15.31:0/3451766395 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc6b0014070 con 0x7fc6b4100320 2023-07-29T16:22:55.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190238d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190238d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58058/0 (socket says 172.21.15.31:58058) 2023-07-29T16:22:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/3998359752 learned_addr learned my addr 172.21.15.31:0/3998359752 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b939b700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc6a00a0790 0x7fc6a00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190237ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190239740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/3998359752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 msgr2=0x55c190239740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190239740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/3998359752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 msgr2=0x55c190237ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190237ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/3998359752 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c18ffb6de0 con 0x55c190238530 2023-07-29T16:22:55.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190237ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:55.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b939b700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc6a00a0790 0x7fc6a00a2c40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0095f80 tx=0x7fc6b000cbd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.532+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190238d60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400b2e0 tx=0x7fc6a4005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f18fe4df75ba5106 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.533+0000 7fc69f7fe700 1 -- 172.21.15.31:0/3998359752 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400baf0 con 0x55c190238530 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.533+0000 7fc69f7fe700 1 -- 172.21.15.31:0/3998359752 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6a401e040 con 0x55c190238530 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.533+0000 7fc69f7fe700 1 -- 172.21.15.31:0/3998359752 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40054f0 con 0x55c190238530 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/3998359752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 msgr2=0x55c190238d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190238d60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400b2e0 tx=0x7fc6a4005c00 comp rx=0 tx=0).stop 2023-07-29T16:22:55.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/3998359752 shutdown_connections 2023-07-29T16:22:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190239740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190237ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/3998359752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190238d60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/3998359752 >> 172.21.15.31:0/3998359752 conn(0x55c19022d3e0 msgr2=0x55c190232990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/3998359752 shutdown_connections 2023-07-29T16:22:55.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/3998359752 wait complete. 2023-07-29T16:22:55.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.534+0000 7fc6ceec0b80 1 -- start start 2023-07-29T16:22:55.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 Processor -- start 2023-07-29T16:22:55.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190241830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190241d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190248db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c190125510 con 0x55c190237be0 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1901251b0 con 0x55c1902392f0 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c19000c920 con 0x55c190238530 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6ceec0b80 10 client.0 _ll_register_callbacks cb 0x55c19012e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6b8b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190241d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6b8b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190241d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58072/0 (socket says 172.21.15.31:58072) 2023-07-29T16:22:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/1324504719 learned_addr learned my addr 172.21.15.31:0/1324504719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.535+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190241830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190248db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 msgr2=0x55c190248db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190248db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/1324504719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 msgr2=0x55c190241830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190241830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/1324504719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b4109090 con 0x55c190238530 2023-07-29T16:22:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190241830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:55.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190241d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4005b40 tx=0x7fc6a4005550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df983e948a0bcbea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc69cff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc69cff9700 1 -- 172.21.15.31:0/1324504719 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4031070 con 0x55c190238530 2023-07-29T16:22:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc69cff9700 1 -- 172.21.15.31:0/1324504719 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc6a40049f0 con 0x55c190238530 2023-07-29T16:22:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.536+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c1901b3d10 con 0x55c190238530 2023-07-29T16:22:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc69cff9700 1 -- 172.21.15.31:0/1324504719 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a400be40 con 0x55c190238530 2023-07-29T16:22:55.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 10 client.27724 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c18ff9fc60 con 0x55c190238530 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc69cff9700 1 -- 172.21.15.31:0/1324504719 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6a403b020 con 0x55c190238530 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc69cff9700 1 -- 172.21.15.31:0/1324504719 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc6a4034030 con 0x55c190238530 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 10 client.27724 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 10 client.27724 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 20 client.27724 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:55.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19024d0c0 0x55c19024f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 4 client.27724 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.537+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c190206d70 con 0x55c19024d0c0 2023-07-29T16:22:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.538+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19024d0c0 0x55c19024f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.538+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19024d0c0 0x55c19024f4a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb357822f9f276ee server_cookie=414ffeb9d079d7b6 in_seq=0 out_seq=0 2023-07-29T16:22:55.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.538+0000 7fc69cff9700 10 client.27724 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc69cff9700 1 -- 172.21.15.31:0/1324504719 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c190206d70 con 0x55c19024d0c0 2023-07-29T16:22:55.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc69cff9700 10 client.27724 handle_command_reply: tid=0 2023-07-29T16:22:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 client.27724 shutdown 2023-07-29T16:22:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 msgr2=0x55c190241d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190241d70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4005b40 tx=0x7fc6a4005550 comp rx=0 tx=0).stop 2023-07-29T16:22:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 shutdown_connections 2023-07-29T16:22:55.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19024d0c0 0x55c19024f4a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1902392f0 0x55c190248db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190241830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/1324504719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190238530 0x55c190241d70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc69cff9700 0 client.27724 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 >> 172.21.15.31:0/1324504719 conn(0x55c19022d3e0 msgr2=0x55c190232060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 shutdown_connections 2023-07-29T16:22:55.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/1324504719 wait complete. 2023-07-29T16:22:55.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.539+0000 7fc6ceec0b80 20 client.27724 trim_cache size 0 max 16384 2023-07-29T16:22:55.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.558+0000 7fc6ceec0b80 1 Processor -- start 2023-07-29T16:22:55.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.558+0000 7fc6ceec0b80 1 -- start start 2023-07-29T16:22:55.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.558+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190237be0 0x55c190246d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c19023f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1902392f0 0x55c19023c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c190240ce0 con 0x55c1902392f0 2023-07-29T16:22:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1902413a0 con 0x55c190238530 2023-07-29T16:22:55.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c19022f460 con 0x55c190237be0 2023-07-29T16:22:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b8b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c19023f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190237be0 0x55c190246d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b8b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c19023f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35190/0 (socket says 172.21.15.31:35190) 2023-07-29T16:22:55.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/414961767 learned_addr learned my addr 172.21.15.31:0/414961767 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1902392f0 0x55c19023c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 -- 172.21.15.31:0/414961767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 msgr2=0x55c19023f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c19023f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 -- 172.21.15.31:0/414961767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1902392f0 msgr2=0x55c19023c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1902392f0 0x55c19023c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b9b9c700 1 -- 172.21.15.31:0/414961767 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c18ffdd250 con 0x55c190237be0 2023-07-29T16:22:55.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.559+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c19023f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.560+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190237be0 0x55c190246d60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55c19024fd10 tx=0x7fc6ac00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d3560d3c1fce615 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.560+0000 7fc69ffff700 1 -- 172.21.15.31:0/414961767 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac00e040 con 0x55c190237be0 2023-07-29T16:22:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.560+0000 7fc69ffff700 1 -- 172.21.15.31:0/414961767 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6ac00f040 con 0x55c190237be0 2023-07-29T16:22:55.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.560+0000 7fc69ffff700 1 -- 172.21.15.31:0/414961767 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac007c10 con 0x55c190237be0 2023-07-29T16:22:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/414961767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190237be0 msgr2=0x55c190246d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190237be0 0x55c190246d60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55c19024fd10 tx=0x7fc6ac00b040 comp rx=0 tx=0).stop 2023-07-29T16:22:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/414961767 shutdown_connections 2023-07-29T16:22:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1902392f0 0x55c19023c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c190237be0 0x55c190246d60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/414961767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c19023f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/414961767 >> 172.21.15.31:0/414961767 conn(0x55c19022d3e0 msgr2=0x55c190233690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/414961767 shutdown_connections 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/414961767 wait complete. 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.561+0000 7fc6ceec0b80 1 -- start start 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 Processor -- start 2023-07-29T16:22:55.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190246b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c190245120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1902392f0 0x55c190245660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c190245be0 con 0x55c190237be0 2023-07-29T16:22:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c190245d50 con 0x55c190238530 2023-07-29T16:22:55.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c190245f10 con 0x55c1902392f0 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6ceec0b80 10 client.0 _ll_register_callbacks cb 0x55c19012e620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.562+0000 7fc6b9b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190246b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b9b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190246b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50724/0 (socket says 172.21.15.31:50724) 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b9b9c700 1 -- 172.21.15.31:0/2879768621 learned_addr learned my addr 172.21.15.31:0/2879768621 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c190245120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1902392f0 0x55c190245660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/2879768621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1902392f0 msgr2=0x55c190245660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1902392f0 0x55c190245660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/2879768621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 msgr2=0x55c190246b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190246b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 -- 172.21.15.31:0/2879768621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c18ff9fc60 con 0x55c190238530 2023-07-29T16:22:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc6b8b9a700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c190245120 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400b2e0 tx=0x7fc6a400bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b03eab0f83a922f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.563+0000 7fc69cff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc69cff9700 1 -- 172.21.15.31:0/2879768621 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4031040 con 0x55c190238530 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc69cff9700 1 -- 172.21.15.31:0/2879768621 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc6a4000e40 con 0x55c190238530 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6a401e040 con 0x55c190238530 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc69cff9700 1 -- 172.21.15.31:0/2879768621 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a4005a00 con 0x55c190238530 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc6ceec0b80 10 client.27719 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fc6ac023290 con 0x55c190238530 2023-07-29T16:22:55.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc69cff9700 1 -- 172.21.15.31:0/2879768621 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6a401e040 con 0x55c190238530 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.564+0000 7fc69cff9700 1 -- 172.21.15.31:0/2879768621 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc6a4034030 con 0x55c190238530 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6ceec0b80 10 client.27719 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6ceec0b80 10 client.27719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6ceec0b80 20 client.27719 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19029ff20 0x55c1902a2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6ceec0b80 4 client.27719 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:22:55.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c190181d40 con 0x55c19029ff20 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19029ff20 0x55c1902a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc69cff9700 10 client.27719 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.565+0000 7fc6b9b9c700 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19029ff20 0x55c1902a2330 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e18f35897bd53aa server_cookie=75571e897fd0f0da in_seq=0 out_seq=0 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc69cff9700 1 -- 172.21.15.31:0/2879768621 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c190181d40 con 0x55c19029ff20 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc69cff9700 10 client.27719 handle_command_reply: tid=0 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 client.27719 shutdown 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 msgr2=0x55c190245120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c190245120 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400b2e0 tx=0x7fc6a400bf90 comp rx=0 tx=0).stop 2023-07-29T16:22:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 shutdown_connections 2023-07-29T16:22:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c19029ff20 0x55c1902a2330 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1902392f0 0x55c190245660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c190237be0 0x55c190246b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 --2- 172.21.15.31:0/2879768621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c190238530 0x55c190245120 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc69cff9700 0 client.27719 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 >> 172.21.15.31:0/2879768621 conn(0x55c19022d3e0 msgr2=0x55c19023f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 shutdown_connections 2023-07-29T16:22:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 1 -- 172.21.15.31:0/2879768621 wait complete. 2023-07-29T16:22:55.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.566+0000 7fc6ceec0b80 20 client.27719 trim_cache size 0 max 16384 2023-07-29T16:22:55.576 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:22:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:22:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:22:55.576 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:22:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc6a00a0790 msgr2=0x7fc6a00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc6a00a0790 0x7fc6a00a2c40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0095f80 tx=0x7fc6b000cbd0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 msgr2=0x7fc6b41982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 0x7fc6b41982e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000e970 tx=0x7fc6b000ec80 comp rx=0 tx=0).stop 2023-07-29T16:22:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 shutdown_connections 2023-07-29T16:22:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc6a00a0790 0x7fc6a00a2c40 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6b4101ea0 0x7fc6b419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6b4100320 0x7fc6b41982e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 --2- 172.21.15.31:0/3451766395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6b4101510 0x7fc6b4198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 >> 172.21.15.31:0/3451766395 conn(0x7fc6b40fb920 msgr2=0x7fc6b40fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 shutdown_connections 2023-07-29T16:22:55.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.567+0000 7fc6bb5ff700 1 -- 172.21.15.31:0/3451766395 wait complete. 2023-07-29T16:22:55.584 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:22:55.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:22:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 -- 172.21.15.31:0/2242202325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 msgr2=0x7f29a8108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 --2- 172.21.15.31:0/2242202325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 0x7f29a8108780 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f29a000b0a0 tx=0x7f29a000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 -- 172.21.15.31:0/2242202325 shutdown_connections 2023-07-29T16:22:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 --2- 172.21.15.31:0/2242202325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 0x7f29a8108780 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 --2- 172.21.15.31:0/2242202325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a8100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 --2- 172.21.15.31:0/2242202325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 0x7f29a8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 -- 172.21.15.31:0/2242202325 >> 172.21.15.31:0/2242202325 conn(0x7f29a80fb920 msgr2=0x7f29a80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 -- 172.21.15.31:0/2242202325 shutdown_connections 2023-07-29T16:22:55.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.807+0000 7f29af3ff700 1 -- 172.21.15.31:0/2242202325 wait complete. 2023-07-29T16:22:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.808+0000 7f29af3ff700 1 Processor -- start 2023-07-29T16:22:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 -- start start 2023-07-29T16:22:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a81981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 0x7f29a8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 0x7f29a819f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a819fd50 con 0x7f29a8100340 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a819fe90 con 0x7f29a8101ec0 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29af3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a81a0020 con 0x7f29a8101530 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29ad19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a81981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29ad19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a81981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50732/0 (socket says 172.21.15.31:50732) 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29ad19b700 1 -- 172.21.15.31:0/4024379579 learned_addr learned my addr 172.21.15.31:0/4024379579 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29ad99c700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 0x7f29a819f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.809+0000 7f29ac99a700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 0x7f29a8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ad19b700 1 -- 172.21.15.31:0/4024379579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 msgr2=0x7f29a8198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ad19b700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 0x7f29a8198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ad19b700 1 -- 172.21.15.31:0/4024379579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 msgr2=0x7f29a819f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ad19b700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 0x7f29a819f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ad19b700 1 -- 172.21.15.31:0/4024379579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a0009d00 con 0x7f29a8100340 2023-07-29T16:22:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ac99a700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 0x7f29a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29ad19b700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a81981d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e970 tx=0x7f29a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4009790 con 0x7f29a8100340 2023-07-29T16:22:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29a400c7e0 con 0x7f29a8100340 2023-07-29T16:22:55.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a4003c70 con 0x7f29a8100340 2023-07-29T16:22:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.810+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a81a0300 con 0x7f29a8100340 2023-07-29T16:22:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.811+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a81a07c0 con 0x7f29a8100340 2023-07-29T16:22:55.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.812+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a810bb60 con 0x7f29a8100340 2023-07-29T16:22:55.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.815+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f29a400cc50 con 0x7f29a8100340 2023-07-29T16:22:55.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.816+0000 7f299e7fc700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f29940a0860 0x7f29940a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.817+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f29a4014070 con 0x7f29a8100340 2023-07-29T16:22:55.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.817+0000 7f29ac99a700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f29940a0860 0x7f29940a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.817+0000 7f29ac99a700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f29940a0860 0x7f29940a2d10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2998011490 tx=0x7f2998009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:55.818+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f29a4095880 con 0x7f29a8100340 2023-07-29T16:22:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 -- 172.21.15.31:0/856935163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c103410 msgr2=0x7f3a2c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 --2- 172.21.15.31:0/856935163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c103410 0x7f3a2c105890 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0098f0 tx=0x7f3a1c009c00 comp rx=0 tx=0).stop 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 -- 172.21.15.31:0/856935163 shutdown_connections 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 --2- 172.21.15.31:0/856935163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a2c105f20 0x7f3a2c1083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 --2- 172.21.15.31:0/856935163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c068e80 0x7f3a2c102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 --2- 172.21.15.31:0/856935163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c103410 0x7f3a2c105890 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 -- 172.21.15.31:0/856935163 >> 172.21.15.31:0/856935163 conn(0x7f3a2c0fa840 msgr2=0x7f3a2c0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 -- 172.21.15.31:0/856935163 shutdown_connections 2023-07-29T16:22:55.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.867+0000 7f3a33bff700 1 -- 172.21.15.31:0/856935163 wait complete. 2023-07-29T16:22:55.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.868+0000 7f3a33bff700 1 Processor -- start 2023-07-29T16:22:55.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 -- start start 2023-07-29T16:22:55.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 0x7f3a2c196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c103410 0x7f3a2c196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a2c105f20 0x7f3a2c1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c197c50 con 0x7f3a2c105f20 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c197dc0 con 0x7f3a2c103410 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a33bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a2c197f80 con 0x7f3a2c068e80 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.869+0000 7f3a3199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 0x7f3a2c196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a2c105f20 0x7f3a2c1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3119a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c103410 0x7f3a2c196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 0x7f3a2c196230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58146/0 (socket says 172.21.15.31:58146) 2023-07-29T16:22:55.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 -- 172.21.15.31:0/2541037936 learned_addr learned my addr 172.21.15.31:0/2541037936 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 -- 172.21.15.31:0/2541037936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c103410 msgr2=0x7f3a2c196770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c103410 0x7f3a2c196770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 -- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a2c105f20 msgr2=0x7f3a2c1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a2c105f20 0x7f3a2c1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 -- 172.21.15.31:0/2541037936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c009520 con 0x7f3a2c068e80 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3119a700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c103410 0x7f3a2c196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.870+0000 7f3a3199b700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 0x7f3a2c196230 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800e9f0 tx=0x7f3a2800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.871+0000 7f3a22ffd700 1 -- 172.21.15.31:0/2541037936 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28009690 con 0x7f3a2c068e80 2023-07-29T16:22:55.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.871+0000 7f3a22ffd700 1 -- 172.21.15.31:0/2541037936 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a2800ce80 con 0x7f3a2c068e80 2023-07-29T16:22:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.871+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c198260 con 0x7f3a2c068e80 2023-07-29T16:22:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.871+0000 7f3a22ffd700 1 -- 172.21.15.31:0/2541037936 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28018420 con 0x7f3a2c068e80 2023-07-29T16:22:55.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.871+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c19ea20 con 0x7f3a2c068e80 2023-07-29T16:22:55.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 469462 9.06 MB/sec execute 160 sec latency 22.100 ms 2023-07-29T16:22:55.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.876+0000 7f3a22ffd700 1 -- 172.21.15.31:0/2541037936 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3a2800c9c0 con 0x7f3a2c068e80 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 Processor -- start 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 -- start start 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2bdb6b0 0x5598c2d83cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d84bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2d85100 0x5598c2d85570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2cd79b0 con 0x5598c2bdb6b0 2023-07-29T16:22:55.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2cd5370 con 0x5598c2d84200 2023-07-29T16:22:55.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a47447b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2cd0c60 con 0x5598c2d85100 2023-07-29T16:22:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.877+0000 7f3a3219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2bdb6b0 0x5598c2d83cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2bdb6b0 0x5598c2d83cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50754/0 (socket says 172.21.15.31:50754) 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3219c700 1 -- 172.21.15.31:0/886333064 learned_addr learned my addr 172.21.15.31:0/886333064 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3199b700 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d84bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2d85100 0x5598c2d85570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 -- 172.21.15.31:0/886333064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 msgr2=0x5598c2d84bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d84bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 -- 172.21.15.31:0/886333064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2bdb6b0 msgr2=0x5598c2d83cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2bdb6b0 0x5598c2d83cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 -- 172.21.15.31:0/886333064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598c2afed80 con 0x5598c2d85100 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a3119a700 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2d85100 0x5598c2d85570 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009230 tx=0x7f3a1c005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8bf5fbe5552f343 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a22ffd700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a180a0890 0x7f3a180a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.878+0000 7f3a17fff700 1 -- 172.21.15.31:0/886333064 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c01c050 con 0x5598c2d85100 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.879+0000 7f3a17fff700 1 -- 172.21.15.31:0/886333064 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a1c01d040 con 0x5598c2d85100 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.879+0000 7f3a17fff700 1 -- 172.21.15.31:0/886333064 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c0053e0 con 0x5598c2d85100 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.879+0000 7f3a22ffd700 1 -- 172.21.15.31:0/2541037936 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3a28014070 con 0x7f3a2c068e80 2023-07-29T16:22:55.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.879+0000 7f3a3219c700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a180a0890 0x7f3a180a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.879+0000 7f3a47447b80 1 -- 172.21.15.31:0/886333064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2d85100 msgr2=0x5598c2d85570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.879+0000 7f3a47447b80 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2d85100 0x5598c2d85570 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009230 tx=0x7f3a1c005dc0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a3219c700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a180a0890 0x7f3a180a2d40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3a240075c0 tx=0x7f3a2400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 -- 172.21.15.31:0/886333064 shutdown_connections 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2d85100 0x5598c2d85570 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2bdb6b0 0x5598c2d83cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 --2- 172.21.15.31:0/886333064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d84bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 -- 172.21.15.31:0/886333064 >> 172.21.15.31:0/886333064 conn(0x5598c2d74fd0 msgr2=0x5598c2d7a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 -- 172.21.15.31:0/886333064 shutdown_connections 2023-07-29T16:22:55.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 -- 172.21.15.31:0/886333064 wait complete. 2023-07-29T16:22:55.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.880+0000 7f3a47447b80 1 -- start start 2023-07-29T16:22:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 Processor -- start 2023-07-29T16:22:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d8d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d947f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2c6d4b0 con 0x5598c2d85100 2023-07-29T16:22:55.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2c6d150 con 0x5598c2d84200 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.881+0000 7f3a47447b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2b548c0 con 0x5598c2bdb6b0 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a47447b80 10 client.0 _ll_register_callbacks cb 0x5598c2c765c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35242/0 (socket says 172.21.15.31:35242) 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 -- 172.21.15.31:0/1899707771 learned_addr learned my addr 172.21.15.31:0/1899707771 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3199b700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d8d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 -- 172.21.15.31:0/1899707771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 msgr2=0x5598c2d8d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d8d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 -- 172.21.15.31:0/1899707771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 msgr2=0x5598c2d947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3119a700 1 -- 172.21.15.31:0/1899707771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c109090 con 0x5598c2d84200 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.882+0000 7f3a3199b700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a3119a700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d7b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c005e90 tx=0x7f3a1c005420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59285f6bee813b45 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a157fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a157fa700 1 -- 172.21.15.31:0/1899707771 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c01c070 con 0x5598c2d84200 2023-07-29T16:22:55.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5598c2cfbcb0 con 0x5598c2d84200 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a157fa700 1 -- 172.21.15.31:0/1899707771 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3a1c0056c0 con 0x5598c2d84200 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a157fa700 1 -- 172.21.15.31:0/1899707771 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a1c00b970 con 0x5598c2d84200 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a47447b80 10 client.27737 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5598c2ae7c00 con 0x5598c2d84200 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.883+0000 7f3a157fa700 1 -- 172.21.15.31:0/1899707771 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5598c2cfbcb0 con 0x5598c2d84200 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a157fa700 1 -- 172.21.15.31:0/1899707771 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3a1c009c70 con 0x5598c2d84200 2023-07-29T16:22:55.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a47447b80 10 client.27737 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a47447b80 10 client.27737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a47447b80 20 client.27737 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a47447b80 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2d98b80 0x5598c2d9af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a47447b80 4 client.27737 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598c2d4ed10 con 0x5598c2d98b80 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a3199b700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2d98b80 0x5598c2d9af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.885+0000 7f3a157fa700 10 client.27737 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:55.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.884+0000 7f3a3199b700 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2d98b80 0x5598c2d9af90 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3413d47b33a53e0 server_cookie=2e1cac96009b57f5 in_seq=0 out_seq=0 2023-07-29T16:22:55.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a157fa700 1 -- 172.21.15.31:0/1899707771 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5598c2d4ed10 con 0x5598c2d98b80 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a157fa700 10 client.27737 handle_command_reply: tid=0 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 client.27737 shutdown 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 msgr2=0x5598c2d8d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d7b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c005e90 tx=0x7f3a1c005420 comp rx=0 tx=0).stop 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 shutdown_connections 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2d98b80 0x5598c2d9af90 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a157fa700 0 client.27737 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 --2- 172.21.15.31:0/1899707771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d7b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 >> 172.21.15.31:0/1899707771 conn(0x5598c2d74fd0 msgr2=0x5598c2d79a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 shutdown_connections 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.886+0000 7f3a47447b80 1 -- 172.21.15.31:0/1899707771 wait complete. 2023-07-29T16:22:55.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.887+0000 7f3a47447b80 20 client.27737 trim_cache size 0 max 16384 2023-07-29T16:22:55.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.904+0000 7f3a47447b80 1 Processor -- start 2023-07-29T16:22:55.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.904+0000 7f3a47447b80 1 -- start start 2023-07-29T16:22:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d8d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a47447b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2d8c720 con 0x5598c2d85100 2023-07-29T16:22:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a47447b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2d98940 con 0x5598c2d84200 2023-07-29T16:22:55.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a47447b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2d7bb30 con 0x5598c2bdb6b0 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58178/0 (socket says 172.21.15.31:58178) 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 -- 172.21.15.31:0/2300394913 learned_addr learned my addr 172.21.15.31:0/2300394913 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3119a700 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3219c700 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d8d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 -- 172.21.15.31:0/2300394913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 msgr2=0x5598c2d8d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 -- 172.21.15.31:0/2300394913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 msgr2=0x5598c2d8d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d8d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3199b700 1 -- 172.21.15.31:0/2300394913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598c2b251f0 con 0x5598c2bdb6b0 2023-07-29T16:22:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.905+0000 7f3a3219c700 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d8d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:22:55.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.906+0000 7f3a3199b700 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92770 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009fd0 tx=0x7f3a280d1530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc8f1346383c7b14 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.906+0000 7f3a20bff700 1 -- 172.21.15.31:0/2300394913 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28098e20 con 0x5598c2bdb6b0 2023-07-29T16:22:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.906+0000 7f3a20bff700 1 -- 172.21.15.31:0/2300394913 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a28098c40 con 0x5598c2bdb6b0 2023-07-29T16:22:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.906+0000 7f3a20bff700 1 -- 172.21.15.31:0/2300394913 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a28097420 con 0x5598c2bdb6b0 2023-07-29T16:22:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.906+0000 7f3a47447b80 1 -- 172.21.15.31:0/2300394913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 msgr2=0x5598c2d92770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.906+0000 7f3a47447b80 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92770 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009fd0 tx=0x7f3a280d1530 comp rx=0 tx=0).stop 2023-07-29T16:22:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 -- 172.21.15.31:0/2300394913 shutdown_connections 2023-07-29T16:22:55.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d85100 0x5598c2d8d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92770 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 --2- 172.21.15.31:0/2300394913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d84200 0x5598c2d8d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 -- 172.21.15.31:0/2300394913 >> 172.21.15.31:0/2300394913 conn(0x5598c2d74fd0 msgr2=0x5598c2d8b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 -- 172.21.15.31:0/2300394913 shutdown_connections 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 -- 172.21.15.31:0/2300394913 wait complete. 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 -- start start 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.907+0000 7f3a47447b80 1 Processor -- start 2023-07-29T16:22:55.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d84200 0x5598c2d927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d85100 0x5598c2d90970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2d91010 con 0x5598c2d84200 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2d91180 con 0x5598c2d85100 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598c2d91340 con 0x5598c2bdb6b0 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a47447b80 10 client.0 _ll_register_callbacks cb 0x5598c2c765c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3119a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d84200 0x5598c2d927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3119a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d84200 0x5598c2d927b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50778/0 (socket says 172.21.15.31:50778) 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3119a700 1 -- 172.21.15.31:0/3489550145 learned_addr learned my addr 172.21.15.31:0/3489550145 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3219c700 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d85100 0x5598c2d90970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3199b700 1 -- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d85100 msgr2=0x5598c2d90970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3199b700 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d85100 0x5598c2d90970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3199b700 1 -- 172.21.15.31:0/3489550145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d84200 msgr2=0x5598c2d927b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:55.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3199b700 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d84200 0x5598c2d927b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.908+0000 7f3a3199b700 1 -- 172.21.15.31:0/3489550145 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c19ea20 con 0x5598c2bdb6b0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a3199b700 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92270 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009fd0 tx=0x7f3a2800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e14a9df54e69c10c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a157fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a157fa700 1 -- 172.21.15.31:0/3489550145 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a280992e0 con 0x5598c2bdb6b0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a157fa700 1 -- 172.21.15.31:0/3489550145 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3a2809d310 con 0x5598c2bdb6b0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a1c01d040 con 0x5598c2bdb6b0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a157fa700 1 -- 172.21.15.31:0/3489550145 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a28010430 con 0x5598c2bdb6b0 2023-07-29T16:22:55.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a47447b80 10 client.27754 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5598c2afd250 con 0x5598c2bdb6b0 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.909+0000 7f3a157fa700 1 -- 172.21.15.31:0/3489550145 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a1c01d040 con 0x5598c2bdb6b0 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a157fa700 1 -- 172.21.15.31:0/3489550145 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3a280a12a0 con 0x5598c2bdb6b0 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a47447b80 10 client.27754 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a47447b80 10 client.27754 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a47447b80 20 client.27754 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:55.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a47447b80 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2deb7b0 0x5598c2dedb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a47447b80 4 client.27754 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:22:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5598c2cc9ce0 con 0x5598c2deb7b0 2023-07-29T16:22:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a3119a700 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2deb7b0 0x5598c2dedb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a3119a700 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2deb7b0 0x5598c2dedb90 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c510af72809da93a server_cookie=bd0c6cae8137850c in_seq=0 out_seq=0 2023-07-29T16:22:55.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:55.910+0000 7f3a157fa700 10 client.27754 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.026+0000 7f3a157fa700 1 -- 172.21.15.31:0/3489550145 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5598c2cc9ce0 con 0x5598c2deb7b0 2023-07-29T16:22:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.026+0000 7f3a157fa700 10 client.27754 handle_command_reply: tid=0 2023-07-29T16:22:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.026+0000 7f3a47447b80 1 client.27754 shutdown 2023-07-29T16:22:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.026+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 msgr2=0x5598c2d92270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.026+0000 7f3a47447b80 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92270 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009fd0 tx=0x7f3a2800c6e0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.026+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 shutdown_connections 2023-07-29T16:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5598c2deb7b0 0x5598c2dedb90 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5598c2d85100 0x5598c2d90970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5598c2bdb6b0 0x5598c2d92270 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 --2- 172.21.15.31:0/3489550145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5598c2d84200 0x5598c2d927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a157fa700 0 client.27754 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 >> 172.21.15.31:0/3489550145 conn(0x5598c2d74fd0 msgr2=0x5598c2d87fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 shutdown_connections 2023-07-29T16:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 1 -- 172.21.15.31:0/3489550145 wait complete. 2023-07-29T16:22:56.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.027+0000 7f3a47447b80 20 client.27754 trim_cache size 0 max 16384 2023-07-29T16:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:22:56.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a180a0890 msgr2=0x7f3a180a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a180a0890 0x7f3a180a2d40 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3a240075c0 tx=0x7f3a2400b040 comp rx=0 tx=0).stop 2023-07-29T16:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 msgr2=0x7f3a2c196230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 0x7f3a2c196230 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800e9f0 tx=0x7f3a2800ed00 comp rx=0 tx=0).stop 2023-07-29T16:22:56.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 shutdown_connections 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3a180a0890 0x7f3a180a2d40 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a2c105f20 0x7f3a2c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a2c068e80 0x7f3a2c196230 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 --2- 172.21.15.31:0/2541037936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a2c103410 0x7f3a2c196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 >> 172.21.15.31:0/2541037936 conn(0x7f3a2c0fa840 msgr2=0x7f3a2c101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 shutdown_connections 2023-07-29T16:22:56.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:22:56.028+0000 7f3a33bff700 1 -- 172.21.15.31:0/2541037936 wait complete. 2023-07-29T16:22:56.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.128+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a810bd20 con 0x7f29a8100340 2023-07-29T16:22:56.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.129+0000 7f299e7fc700 1 -- 172.21.15.31:0/4024379579 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f29a4098ea0 con 0x7f29a8100340 2023-07-29T16:22:56.128 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:22:56.128 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:22:56.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f29940a0860 msgr2=0x7f29940a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f29940a0860 0x7f29940a2d10 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2998011490 tx=0x7f2998009260 comp rx=0 tx=0).stop 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 msgr2=0x7f29a81981d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a81981d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e970 tx=0x7f29a400ec80 comp rx=0 tx=0).stop 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 shutdown_connections 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f29940a0860 0x7f29940a2d10 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29a8101ec0 0x7f29a819f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f29a8100340 0x7f29a81981d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 --2- 172.21.15.31:0/4024379579 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29a8101530 0x7f29a8198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 >> 172.21.15.31:0/4024379579 conn(0x7f29a80fb920 msgr2=0x7f29a80fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 shutdown_connections 2023-07-29T16:22:56.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.132+0000 7f29af3ff700 1 -- 172.21.15.31:0/4024379579 wait complete. 2023-07-29T16:22:56.132 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:22:56.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:22:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.376+0000 7fee94d9d700 1 -- 172.21.15.31:0/2338948072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 msgr2=0x7fee90073b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.376+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2338948072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90073b10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fee84009990 tx=0x7fee84009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 -- 172.21.15.31:0/2338948072 shutdown_connections 2023-07-29T16:22:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2338948072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 0x7fee9010a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2338948072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee90074c70 0x7fee90073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2338948072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90073b10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 -- 172.21.15.31:0/2338948072 >> 172.21.15.31:0/2338948072 conn(0x7fee900fb850 msgr2=0x7fee900fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 -- 172.21.15.31:0/2338948072 shutdown_connections 2023-07-29T16:22:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.377+0000 7fee94d9d700 1 -- 172.21.15.31:0/2338948072 wait complete. 2023-07-29T16:22:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 Processor -- start 2023-07-29T16:22:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 -- start start 2023-07-29T16:22:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee90074c70 0x7fee901985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 0x7fee9019f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9019fbe0 con 0x7fee90108130 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9019fd20 con 0x7fee900736a0 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee94d9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9019feb0 con 0x7fee90074c70 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35276/0 (socket says 172.21.15.31:35276) 2023-07-29T16:22:56.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee8f7fe700 1 -- 172.21.15.31:0/2211481090 learned_addr learned my addr 172.21.15.31:0/2211481090 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee8ffff700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 0x7fee9019f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.378+0000 7fee8effd700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee90074c70 0x7fee901985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8ffff700 1 -- 172.21.15.31:0/2211481090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee90074c70 msgr2=0x7fee901985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8ffff700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee90074c70 0x7fee901985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8ffff700 1 -- 172.21.15.31:0/2211481090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 msgr2=0x7fee90198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8ffff700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8ffff700 1 -- 172.21.15.31:0/2211481090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee840095f0 con 0x7fee90108130 2023-07-29T16:22:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8ffff700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 0x7fee9019f610 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fee8800efc0 tx=0x7fee8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8cff9700 1 -- 172.21.15.31:0/2211481090 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88018070 con 0x7fee90108130 2023-07-29T16:22:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee901a0130 con 0x7fee90108130 2023-07-29T16:22:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8cff9700 1 -- 172.21.15.31:0/2211481090 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee8800f040 con 0x7fee90108130 2023-07-29T16:22:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.379+0000 7fee8cff9700 1 -- 172.21.15.31:0/2211481090 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8800cd40 con 0x7fee90108130 2023-07-29T16:22:56.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.380+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee901a0640 con 0x7fee90108130 2023-07-29T16:22:56.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.383+0000 7fee8cff9700 1 -- 172.21.15.31:0/2211481090 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fee88003a00 con 0x7fee90108130 2023-07-29T16:22:56.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.384+0000 7fee8cff9700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee780a0830 0x7fee780a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.384+0000 7feea98f2b80 1 Processor -- start 2023-07-29T16:22:56.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7fee8cff9700 1 -- 172.21.15.31:0/2211481090 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fee880cb6c0 con 0x7fee90108130 2023-07-29T16:22:56.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee780a0830 0x7fee780a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 -- start start 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebab150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebabf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebac920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eceb02a10 con 0x555ecebac4d0 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eceb003d0 con 0x555ecebaad40 2023-07-29T16:22:56.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.385+0000 7feea98f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eceafbcc0 con 0x555ecebab690 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebab150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35280/0 (socket says 172.21.15.31:35280) 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1435067240 learned_addr learned my addr 172.21.15.31:0/1435067240 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8effd700 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebabf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8ffff700 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebac920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee780a0830 0x7fee780a2ce0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fee80005950 tx=0x7fee80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1435067240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 msgr2=0x555ecebabf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebabf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1435067240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 msgr2=0x555ecebac920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebac920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1435067240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ece929de0 con 0x555ecebaad40 2023-07-29T16:22:56.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8ffff700 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:56.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebab150 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e4b0 tx=0x7fee80014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d39c8300440c13 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.386+0000 7fee75ffb700 1 -- 172.21.15.31:0/1435067240 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8000ee50 con 0x555ecebaad40 2023-07-29T16:22:56.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7fee75ffb700 1 -- 172.21.15.31:0/1435067240 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee80021070 con 0x555ecebaad40 2023-07-29T16:22:56.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7fee75ffb700 1 -- 172.21.15.31:0/1435067240 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8001d410 con 0x555ecebaad40 2023-07-29T16:22:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 -- 172.21.15.31:0/1435067240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 msgr2=0x555ecebab150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebab150 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fee8000e4b0 tx=0x7fee80014000 comp rx=0 tx=0).stop 2023-07-29T16:22:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 -- 172.21.15.31:0/1435067240 shutdown_connections 2023-07-29T16:22:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebac920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebab150 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1435067240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebabf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.387+0000 7feea98f2b80 1 -- 172.21.15.31:0/1435067240 >> 172.21.15.31:0/1435067240 conn(0x555eceba0540 msgr2=0x555eceba5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.388+0000 7feea98f2b80 1 -- 172.21.15.31:0/1435067240 shutdown_connections 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.388+0000 7feea98f2b80 1 -- 172.21.15.31:0/1435067240 wait complete. 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.388+0000 7feea98f2b80 1 -- start start 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.388+0000 7feea98f2b80 1 Processor -- start 2023-07-29T16:22:56.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbbf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecea98510 con 0x555ecebac4d0 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecea981b0 con 0x555ecebaad40 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ece97f920 con 0x555ecebab690 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7feea98f2b80 10 client.0 _ll_register_callbacks cb 0x555eceaa1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35294/0 (socket says 172.21.15.31:35294) 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7fee8f7fe700 1 -- 172.21.15.31:0/4262585624 learned_addr learned my addr 172.21.15.31:0/4262585624 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7fee8ffff700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.389+0000 7fee8effd700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee8f7fe700 1 -- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 msgr2=0x555ecebb4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee8f7fe700 1 -- 172.21.15.31:0/4262585624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 msgr2=0x555ecebbbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee8f7fe700 1 -- 172.21.15.31:0/4262585624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee80017070 con 0x555ecebaad40 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb4a10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fee80009be0 tx=0x7fee80005680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4069c7ad3fc287ad server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee6effd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee6effd700 1 -- 172.21.15.31:0/4262585624 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80004220 con 0x555ecebaad40 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555eceb26d10 con 0x555ecebaad40 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee6effd700 1 -- 172.21.15.31:0/4262585624 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fee8000ee10 con 0x555ecebaad40 2023-07-29T16:22:56.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.390+0000 7fee6effd700 1 -- 172.21.15.31:0/4262585624 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee80004380 con 0x555ecebaad40 2023-07-29T16:22:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 10 client.27755 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555ece912c60 con 0x555ecebaad40 2023-07-29T16:22:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7fee6effd700 1 -- 172.21.15.31:0/4262585624 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555eceb26d10 con 0x555ecebaad40 2023-07-29T16:22:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7fee6effd700 1 -- 172.21.15.31:0/4262585624 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fee80003a90 con 0x555ecebaad40 2023-07-29T16:22:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 10 client.27755 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:56.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 10 client.27755 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 20 client.27755 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecebc06b0 0x555ecebc2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 4 client.27755 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555eceb79d70 con 0x555ecebc06b0 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.391+0000 7fee8effd700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecebc06b0 0x555ecebc2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.392+0000 7fee8effd700 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecebc06b0 0x555ecebc2ac0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2dea9539d02c832a server_cookie=a37244a1b8293007 in_seq=0 out_seq=0 2023-07-29T16:22:56.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.392+0000 7fee6effd700 10 client.27755 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.392+0000 7fee6effd700 1 -- 172.21.15.31:0/4262585624 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555eceb79d70 con 0x555ecebc06b0 2023-07-29T16:22:56.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.392+0000 7fee6effd700 10 client.27755 handle_command_reply: tid=0 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 client.27755 shutdown 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 msgr2=0x555ecebb4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb4a10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fee80009be0 tx=0x7fee80005680 comp rx=0 tx=0).stop 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 shutdown_connections 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecebc06b0 0x555ecebc2ac0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb4a10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 --2- 172.21.15.31:0/4262585624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7fee6effd700 0 client.27755 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:56.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 >> 172.21.15.31:0/4262585624 conn(0x555eceba0540 msgr2=0x555eceba51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 shutdown_connections 2023-07-29T16:22:56.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 1 -- 172.21.15.31:0/4262585624 wait complete. 2023-07-29T16:22:56.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.393+0000 7feea98f2b80 20 client.27755 trim_cache size 0 max 16384 2023-07-29T16:22:56.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 Processor -- start 2023-07-29T16:22:56.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 -- start start 2023-07-29T16:22:56.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 0x555ecebc7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebab690 0x555ecebbbf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebac4d0 0x555ecebb40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecebb5b90 con 0x555ecebab690 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecebb33f0 con 0x555ecebac4d0 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.409+0000 7feea98f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecebac8a0 con 0x555ecebaad40 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 0x555ecebc7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 0x555ecebc7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58230/0 (socket says 172.21.15.31:58230) 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1158826836 learned_addr learned my addr 172.21.15.31:0/1158826836 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8effd700 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebab690 0x555ecebbbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8ffff700 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebac4d0 0x555ecebb40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1158826836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebac4d0 msgr2=0x555ecebb40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebac4d0 0x555ecebb40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1158826836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebab690 msgr2=0x555ecebbbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebab690 0x555ecebbbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 -- 172.21.15.31:0/1158826836 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ece950250 con 0x555ecebaad40 2023-07-29T16:22:56.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.410+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 0x555ecebc7420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee8000eb20 tx=0x7fee80005360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=840acf97b35f6bd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7fee767fc700 1 -- 172.21.15.31:0/1158826836 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee800041c0 con 0x555ecebaad40 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7fee767fc700 1 -- 172.21.15.31:0/1158826836 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee80026550 con 0x555ecebaad40 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7fee767fc700 1 -- 172.21.15.31:0/1158826836 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee80004bc0 con 0x555ecebaad40 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 -- 172.21.15.31:0/1158826836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 msgr2=0x555ecebc7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 0x555ecebc7420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fee8000eb20 tx=0x7fee80005360 comp rx=0 tx=0).stop 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 -- 172.21.15.31:0/1158826836 shutdown_connections 2023-07-29T16:22:56.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebac4d0 0x555ecebb40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebaad40 0x555ecebc7420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 --2- 172.21.15.31:0/1158826836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebab690 0x555ecebbbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.411+0000 7feea98f2b80 1 -- 172.21.15.31:0/1158826836 >> 172.21.15.31:0/1158826836 conn(0x555eceba0540 msgr2=0x555eceba5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.412+0000 7feea98f2b80 1 -- 172.21.15.31:0/1158826836 shutdown_connections 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.412+0000 7feea98f2b80 1 -- 172.21.15.31:0/1158826836 wait complete. 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.412+0000 7feea98f2b80 1 -- start start 2023-07-29T16:22:56.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.412+0000 7feea98f2b80 1 Processor -- start 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecebbbb00 con 0x555ecebac4d0 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecebbbc70 con 0x555ecebaad40 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ecebbbe30 con 0x555ecebab690 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7feea98f2b80 10 client.0 _ll_register_callbacks cb 0x555eceaa1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:56.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58242/0 (socket says 172.21.15.31:58242) 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 -- 172.21.15.31:0/489578409 learned_addr learned my addr 172.21.15.31:0/489578409 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 -- 172.21.15.31:0/489578409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 msgr2=0x555ecebb8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 -- 172.21.15.31:0/489578409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 msgr2=0x555ecebbb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8effd700 1 -- 172.21.15.31:0/489578409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ece912c60 con 0x555ecebab690 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.413+0000 7fee8ffff700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7fee8effd700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb85a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fee84009990 tx=0x7fee84005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb918b2fb24fa00f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7fee6effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7fee6effd700 1 -- 172.21.15.31:0/489578409 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8401d070 con 0x555ecebab690 2023-07-29T16:22:56.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee80021070 con 0x555ecebab690 2023-07-29T16:22:56.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7fee6effd700 1 -- 172.21.15.31:0/489578409 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fee84005420 con 0x555ecebab690 2023-07-29T16:22:56.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7fee6effd700 1 -- 172.21.15.31:0/489578409 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee840055e0 con 0x555ecebab690 2023-07-29T16:22:56.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7feea98f2b80 10 client.27772 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:56.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555ece9282b0 con 0x555ecebab690 2023-07-29T16:22:56.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.414+0000 7fee6effd700 1 -- 172.21.15.31:0/489578409 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee80021070 con 0x555ecebab690 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7fee6effd700 1 -- 172.21.15.31:0/489578409 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fee84004620 con 0x555ecebab690 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7feea98f2b80 10 client.27772 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7feea98f2b80 10 client.27772 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7feea98f2b80 20 client.27772 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7feea98f2b80 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecec13650 0x555ecec15a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7feea98f2b80 4 client.27772 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:22:56.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555eceaf4d40 con 0x555ecec13650 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.415+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecec13650 0x555ecec15a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7fee8f7fe700 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecec13650 0x555ecec15a60 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=737ca29b293e1082 server_cookie=6fca1fd83eba7759 in_seq=0 out_seq=0 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7fee6effd700 10 client.27772 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7fee6effd700 1 -- 172.21.15.31:0/489578409 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555eceaf4d40 con 0x555ecec13650 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7fee6effd700 10 client.27772 handle_command_reply: tid=0 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 client.27772 shutdown 2023-07-29T16:22:56.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 msgr2=0x555ecebb85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb85a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fee84009990 tx=0x7fee84005fb0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 shutdown_connections 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ecebac4d0 0x555ecebbb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x555ecec13650 0x555ecec15a60 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ecebaad40 0x555ecebb8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 --2- 172.21.15.31:0/489578409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ecebab690 0x555ecebb85a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 >> 172.21.15.31:0/489578409 conn(0x555eceba0540 msgr2=0x555ecebb28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7fee6effd700 0 client.27772 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:22:56.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 shutdown_connections 2023-07-29T16:22:56.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 1 -- 172.21.15.31:0/489578409 wait complete. 2023-07-29T16:22:56.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.416+0000 7feea98f2b80 20 client.27772 trim_cache size 0 max 16384 2023-07-29T16:22:56.423 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:22:56.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee780a0830 msgr2=0x7fee780a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee780a0830 0x7fee780a2ce0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fee80005950 tx=0x7fee80009260 comp rx=0 tx=0).stop 2023-07-29T16:22:56.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 msgr2=0x7fee9019f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 0x7fee9019f610 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fee8800efc0 tx=0x7fee8800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 shutdown_connections 2023-07-29T16:22:56.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fee780a0830 0x7fee780a2ce0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee90108130 0x7fee9019f610 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee900736a0 0x7fee90198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 --2- 172.21.15.31:0/2211481090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee90074c70 0x7fee901985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 >> 172.21.15.31:0/2211481090 conn(0x7fee900fb850 msgr2=0x7fee900fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 shutdown_connections 2023-07-29T16:22:56.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.417+0000 7fee94d9d700 1 -- 172.21.15.31:0/2211481090 wait complete. 2023-07-29T16:22:56.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:22:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/345593772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 msgr2=0x7fbcb8101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/345593772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 0x7fbcb8101960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbca80098f0 tx=0x7fbca8009c00 comp rx=0 tx=0).stop 2023-07-29T16:22:56.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/345593772 shutdown_connections 2023-07-29T16:22:56.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/345593772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8101ea0 0x7fbcb8108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/345593772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb8100320 0x7fbcb8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/345593772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 0x7fbcb8101960 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/345593772 >> 172.21.15.31:0/345593772 conn(0x7fbcb80fb920 msgr2=0x7fbcb80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/345593772 shutdown_connections 2023-07-29T16:22:56.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.665+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/345593772 wait complete. 2023-07-29T16:22:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.666+0000 7fbcbf3ff700 1 Processor -- start 2023-07-29T16:22:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.666+0000 7fbcbf3ff700 1 -- start start 2023-07-29T16:22:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbf3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 0x7fbcb81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbf3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 0x7fbcb81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbf3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb8101ea0 0x7fbcb819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbf3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb819fe30 con 0x7fbcb8101510 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbf3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb819ff70 con 0x7fbcb8100320 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbf3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcb81a0100 con 0x7fbcb8101ea0 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 0x7fbcb81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 0x7fbcb81982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35346/0 (socket says 172.21.15.31:35346) 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 -- 172.21.15.31:0/1326688909 learned_addr learned my addr 172.21.15.31:0/1326688909 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbc99a700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 0x7fbcb81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb8101ea0 0x7fbcb819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 -- 172.21.15.31:0/1326688909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb8101ea0 msgr2=0x7fbcb819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb8101ea0 0x7fbcb819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 -- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 msgr2=0x7fbcb81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 0x7fbcb81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.667+0000 7fbcbd19b700 1 -- 172.21.15.31:0/1326688909 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8009520 con 0x7fbcb8100320 2023-07-29T16:22:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.668+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 0x7fbcb81982b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400e9f0 tx=0x7fbcb400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.668+0000 7fbcae7fc700 1 -- 172.21.15.31:0/1326688909 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4004220 con 0x7fbcb8100320 2023-07-29T16:22:56.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.668+0000 7fbcae7fc700 1 -- 172.21.15.31:0/1326688909 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcb4004380 con 0x7fbcb8100320 2023-07-29T16:22:56.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.668+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcb81a03e0 con 0x7fbcb8100320 2023-07-29T16:22:56.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.668+0000 7fbcae7fc700 1 -- 172.21.15.31:0/1326688909 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4003be0 con 0x7fbcb8100320 2023-07-29T16:22:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.668+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb81a08f0 con 0x7fbcb8100320 2023-07-29T16:22:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.671+0000 7fbcae7fc700 1 -- 172.21.15.31:0/1326688909 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbcb4003d40 con 0x7fbcb8100320 2023-07-29T16:22:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcd2c78b80 1 Processor -- start 2023-07-29T16:22:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcd2c78b80 1 -- start start 2023-07-29T16:22:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcae7fc700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbca40a0790 0x7fbca40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 0x559953b4cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b4dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcae7fc700 1 -- 172.21.15.31:0/1326688909 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbcb4014070 con 0x7fbcb8100320 2023-07-29T16:22:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.673+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4e2f0 0x559953b4e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953aa4a10 con 0x559953b4e2f0 2023-07-29T16:22:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953aa23d0 con 0x559953b4cbe0 2023-07-29T16:22:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953a9dcc0 con 0x559953b4d530 2023-07-29T16:22:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbca40a0790 0x7fbca40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 0x559953b4cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbc99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b4dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 0x559953b4cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35352/0 (socket says 172.21.15.31:35352) 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2222742158 learned_addr learned my addr 172.21.15.31:0/2222742158 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2222742158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 msgr2=0x559953b4dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b4dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4e2f0 0x559953b4e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2222742158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4e2f0 msgr2=0x559953b4e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4e2f0 0x559953b4e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2222742158 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599538cbde0 con 0x559953b4cbe0 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4e2f0 0x559953b4e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.674+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 0x559953b4cff0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4095dd0 tx=0x7fbcb4098f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b03bbd1f030e49ec server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbca40a0790 0x7fbca40a2c40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000f6e0 tx=0x7fbcb0009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbca37fe700 1 -- 172.21.15.31:0/2222742158 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb409d150 con 0x559953b4cbe0 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbca37fe700 1 -- 172.21.15.31:0/2222742158 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcb4099360 con 0x559953b4cbe0 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbca37fe700 1 -- 172.21.15.31:0/2222742158 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4016f10 con 0x559953b4cbe0 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2222742158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 msgr2=0x559953b4cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 0x559953b4cff0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4095dd0 tx=0x7fbcb4098f20 comp rx=0 tx=0).stop 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2222742158 shutdown_connections 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4e2f0 0x559953b4e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4cbe0 0x559953b4cff0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2222742158 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b4dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.675+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2222742158 >> 172.21.15.31:0/2222742158 conn(0x559953b423e0 msgr2=0x559953b47990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.676+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2222742158 shutdown_connections 2023-07-29T16:22:56.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.676+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2222742158 wait complete. 2023-07-29T16:22:56.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.676+0000 7fbcd2c78b80 1 -- start start 2023-07-29T16:22:56.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.676+0000 7fbcd2c78b80 1 Processor -- start 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b56ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953a3a510 con 0x559953b4d530 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953a3a1b0 con 0x559953b4e2f0 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953921920 con 0x559953b4cbe0 2023-07-29T16:22:56.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcd2c78b80 10 client.0 _ll_register_callbacks cb 0x559953a43620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcbc99a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b56ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58274/0 (socket says 172.21.15.31:58274) 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2020487508 learned_addr learned my addr 172.21.15.31:0/2020487508 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.677+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2020487508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 msgr2=0x559953b5dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 msgr2=0x559953b56ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b56ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcbd19b700 1 -- 172.21.15.31:0/2020487508 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcb81a08f0 con 0x559953b4cbe0 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b567a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4095860 tx=0x7fbcb409d390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=772412ba66049bdd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbca0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbca0ff9700 1 -- 172.21.15.31:0/2020487508 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4016ef0 con 0x559953b4cbe0 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbca0ff9700 1 -- 172.21.15.31:0/2020487508 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbcb40a17b0 con 0x559953b4cbe0 2023-07-29T16:22:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559953ac8d10 con 0x559953b4cbe0 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbca0ff9700 1 -- 172.21.15.31:0/2020487508 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcb40c92c0 con 0x559953b4cbe0 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcd2c78b80 10 client.27784 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.678+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5599538b4c60 con 0x559953b4cbe0 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbca0ff9700 1 -- 172.21.15.31:0/2020487508 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcb407c020 con 0x559953b4cbe0 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbca0ff9700 1 -- 172.21.15.31:0/2020487508 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbcb40c9bc0 con 0x559953b4cbe0 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcd2c78b80 10 client.27784 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcd2c78b80 10 client.27784 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcd2c78b80 20 client.27784 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953b620c0 0x559953b644d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcd2c78b80 4 client.27784 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559953b1bd70 con 0x559953b620c0 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.679+0000 7fbcbc99a700 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953b620c0 0x559953b644d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.680+0000 7fbcbc99a700 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953b620c0 0x559953b644d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=530d65c55cf4d761 server_cookie=1aa30c70227f86ff in_seq=0 out_seq=0 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.680+0000 7fbca0ff9700 10 client.27784 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:56.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.680+0000 7fbca0ff9700 1 -- 172.21.15.31:0/2020487508 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559953b1bd70 con 0x559953b620c0 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.680+0000 7fbca0ff9700 10 client.27784 handle_command_reply: tid=0 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 client.27784 shutdown 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 msgr2=0x559953b567a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b567a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4095860 tx=0x7fbcb409d390 comp rx=0 tx=0).stop 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 shutdown_connections 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953b620c0 0x559953b644d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b567a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/2020487508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b56ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 >> 172.21.15.31:0/2020487508 conn(0x559953b423e0 msgr2=0x559953b47060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbca0ff9700 0 client.27784 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 shutdown_connections 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/2020487508 wait complete. 2023-07-29T16:22:56.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.681+0000 7fbcd2c78b80 20 client.27784 trim_cache size 0 max 16384 2023-07-29T16:22:56.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:56 smithi031 ceph-mon[95504]: pgmap v594: 97 pgs: 97 active+clean; 89 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2023-07-29T16:22:56.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:56 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/4024379579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:22:56.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 Processor -- start 2023-07-29T16:22:56.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 -- start start 2023-07-29T16:22:56.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b4e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b5dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b54640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953b543f0 con 0x559953b4d530 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953b4db50 con 0x559953b4e2f0 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.697+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953b55210 con 0x559953b4cbe0 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b54640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbc99a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b5dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b54640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35382/0 (socket says 172.21.15.31:35382) 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 -- 172.21.15.31:0/4069278083 learned_addr learned my addr 172.21.15.31:0/4069278083 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b4e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 -- 172.21.15.31:0/4069278083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 msgr2=0x559953b4e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b4e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 -- 172.21.15.31:0/4069278083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 msgr2=0x559953b5dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b5dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 -- 172.21.15.31:0/4069278083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599538f2250 con 0x559953b4e2f0 2023-07-29T16:22:56.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b4e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.698+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b54640 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009330 tx=0x7fbcb0011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27551569e8b8abf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.699+0000 7fbca3fff700 1 -- 172.21.15.31:0/4069278083 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb002a070 con 0x559953b4e2f0 2023-07-29T16:22:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.699+0000 7fbca3fff700 1 -- 172.21.15.31:0/4069278083 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcb000f040 con 0x559953b4e2f0 2023-07-29T16:22:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.699+0000 7fbca3fff700 1 -- 172.21.15.31:0/4069278083 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb000e040 con 0x559953b4e2f0 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/4069278083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 msgr2=0x559953b54640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b54640 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009330 tx=0x7fbcb0011000 comp rx=0 tx=0).stop 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/4069278083 shutdown_connections 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b54640 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4cbe0 0x559953b4e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/4069278083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4d530 0x559953b5dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/4069278083 >> 172.21.15.31:0/4069278083 conn(0x559953b423e0 msgr2=0x559953b554e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/4069278083 shutdown_connections 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/4069278083 wait complete. 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 -- start start 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.700+0000 7fbcd2c78b80 1 Processor -- start 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4cbe0 0x559953b5d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b5a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953b5e0c0 con 0x559953b4cbe0 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953b5adf0 con 0x559953b4e2f0 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559953b5afb0 con 0x559953b4d530 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcd2c78b80 10 client.0 _ll_register_callbacks cb 0x559953a43620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcbc99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b5a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcbd99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcbd19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4cbe0 0x559953b5d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcbc99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b5a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58294/0 (socket says 172.21.15.31:58294) 2023-07-29T16:22:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcbc99a700 1 -- 172.21.15.31:0/1443801224 learned_addr learned my addr 172.21.15.31:0/1443801224 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.701+0000 7fbcbd99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35390/0 (socket says 172.21.15.31:35390) 2023-07-29T16:22:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcbd99c700 1 -- 172.21.15.31:0/1443801224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 msgr2=0x559953b5a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b5a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcbd99c700 1 -- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4cbe0 msgr2=0x559953b5d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4cbe0 0x559953b5d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcbd99c700 1 -- 172.21.15.31:0/1443801224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599538b4c60 con 0x559953b4e2f0 2023-07-29T16:22:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcbd99c700 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5a880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009ec0 tx=0x7fbcb0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=965d8cc1d6145bdc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbca0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbca0ff9700 1 -- 172.21.15.31:0/1443801224 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb002a040 con 0x559953b4e2f0 2023-07-29T16:22:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbca0ff9700 1 -- 172.21.15.31:0/1443801224 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbcb0004200 con 0x559953b4e2f0 2023-07-29T16:22:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcb4099360 con 0x559953b4e2f0 2023-07-29T16:22:56.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbca0ff9700 1 -- 172.21.15.31:0/1443801224 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcb000e040 con 0x559953b4e2f0 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcd2c78b80 10 client.27779 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.702+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fbcb0009040 con 0x559953b4e2f0 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbca0ff9700 1 -- 172.21.15.31:0/1443801224 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbcb003f080 con 0x559953b4e2f0 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbca0ff9700 1 -- 172.21.15.31:0/1443801224 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbcb0026aa0 con 0x559953b4e2f0 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcd2c78b80 10 client.27779 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcd2c78b80 10 client.27779 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:22:56.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcd2c78b80 20 client.27779 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953bb5440 0x559953bb7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcd2c78b80 4 client.27779 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559953a96d40 con 0x559953bb5440 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.703+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953bb5440 0x559953bb7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbca0ff9700 10 client.27779 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcbd19b700 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953bb5440 0x559953bb7850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26373186ca4ccee0 server_cookie=4feefa02e5052675 in_seq=0 out_seq=0 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbca0ff9700 1 -- 172.21.15.31:0/1443801224 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559953a96d40 con 0x559953bb5440 2023-07-29T16:22:56.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbca0ff9700 10 client.27779 handle_command_reply: tid=0 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 client.27779 shutdown 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 msgr2=0x559953b5a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5a880 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0009ec0 tx=0x7fbcb0004b40 comp rx=0 tx=0).stop 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 shutdown_connections 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x559953bb5440 0x559953bb7850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559953b4e2f0 0x559953b5a880 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559953b4cbe0 0x559953b5d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbca0ff9700 0 client.27779 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 --2- 172.21.15.31:0/1443801224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559953b4d530 0x559953b5a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 >> 172.21.15.31:0/1443801224 conn(0x559953b423e0 msgr2=0x559953b51170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 shutdown_connections 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.704+0000 7fbcd2c78b80 1 -- 172.21.15.31:0/1443801224 wait complete. 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcd2c78b80 20 client.27779 trim_cache size 0 max 16384 2023-07-29T16:22:56.710 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:22:56.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbca40a0790 msgr2=0x7fbca40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbca40a0790 0x7fbca40a2c40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbcb000f6e0 tx=0x7fbcb0009920 comp rx=0 tx=0).stop 2023-07-29T16:22:56.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 msgr2=0x7fbcb81982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 0x7fbcb81982b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fbcb400e9f0 tx=0x7fbcb400ed00 comp rx=0 tx=0).stop 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 shutdown_connections 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbca40a0790 0x7fbca40a2c40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbcb8101ea0 0x7fbcb819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcb8100320 0x7fbcb81982b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 --2- 172.21.15.31:0/1326688909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcb8101510 0x7fbcb81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 >> 172.21.15.31:0/1326688909 conn(0x7fbcb80fb920 msgr2=0x7fbcb80fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 shutdown_connections 2023-07-29T16:22:56.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.705+0000 7fbcbf3ff700 1 -- 172.21.15.31:0/1326688909 wait complete. 2023-07-29T16:22:56.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:22:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:56 smithi039 ceph-mon[128858]: pgmap v594: 97 pgs: 97 active+clean; 89 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2023-07-29T16:22:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:56 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/4024379579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:22:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:56 smithi156 ceph-mon[129245]: pgmap v594: 97 pgs: 97 active+clean; 89 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 2.5 MiB/s wr, 293 op/s 2023-07-29T16:22:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:56 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/4024379579' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:22:56.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 471366 9.05 MB/sec execute 161 sec latency 25.386 ms 2023-07-29T16:22:56.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 -- 172.21.15.31:0/2376677085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c1036c0 msgr2=0x7fac9c105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 --2- 172.21.15.31:0/2376677085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c1036c0 0x7fac9c105aa0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fac90009990 tx=0x7fac90009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 -- 172.21.15.31:0/2376677085 shutdown_connections 2023-07-29T16:22:56.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 --2- 172.21.15.31:0/2376677085 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c105fe0 0x7fac9c108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 --2- 172.21.15.31:0/2376677085 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c100da0 0x7fac9c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 --2- 172.21.15.31:0/2376677085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c1036c0 0x7fac9c105aa0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 -- 172.21.15.31:0/2376677085 >> 172.21.15.31:0/2376677085 conn(0x7fac9c0fa840 msgr2=0x7fac9c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 -- 172.21.15.31:0/2376677085 shutdown_connections 2023-07-29T16:22:56.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.949+0000 7faca0bff700 1 -- 172.21.15.31:0/2376677085 wait complete. 2023-07-29T16:22:56.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 Processor -- start 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 -- start start 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 0x7fac9c196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1036c0 0x7fac9c196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c105fe0 0x7fac9c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19dbd0 con 0x7fac9c1036c0 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19dd10 con 0x7fac9c100da0 2023-07-29T16:22:56.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.950+0000 7faca0bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19dea0 con 0x7fac9c105fe0 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 0x7fac9c196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 0x7fac9c196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35420/0 (socket says 172.21.15.31:35420) 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 -- 172.21.15.31:0/640296744 learned_addr learned my addr 172.21.15.31:0/640296744 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac99d9b700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1036c0 0x7fac9c196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c105fe0 0x7fac9c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 -- 172.21.15.31:0/640296744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c105fe0 msgr2=0x7fac9c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c105fe0 0x7fac9c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 -- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1036c0 msgr2=0x7fac9c196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1036c0 0x7fac9c196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 -- 172.21.15.31:0/640296744 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900095f0 con 0x7fac9c100da0 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c105fe0 0x7fac9c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.951+0000 7fac9a59c700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 0x7fac9c196050 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00e9a0 tx=0x7fac8c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.952+0000 7fac8b7fe700 1 -- 172.21.15.31:0/640296744 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c009790 con 0x7fac9c100da0 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.952+0000 7fac8b7fe700 1 -- 172.21.15.31:0/640296744 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac8c00cbd0 con 0x7fac9c100da0 2023-07-29T16:22:56.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.952+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c0febf0 con 0x7fac9c100da0 2023-07-29T16:22:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.952+0000 7fac8b7fe700 1 -- 172.21.15.31:0/640296744 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8c018680 con 0x7fac9c100da0 2023-07-29T16:22:56.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.952+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c0ff100 con 0x7fac9c100da0 2023-07-29T16:22:56.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.956+0000 7facb4557b80 1 Processor -- start 2023-07-29T16:22:56.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 -- start start 2023-07-29T16:22:56.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7fac8b7fe700 1 -- 172.21.15.31:0/640296744 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fac8c00cd40 con 0x7fac9c100da0 2023-07-29T16:22:56.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad43ff5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad43ff6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43f4da10 con 0x55ad43ff6530 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43f4b3d0 con 0x55ad43ff72f0 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7facb4557b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43f46cc0 con 0x55ad43ff5be0 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7fac9ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad43ff5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7fac9ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad43ff5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58326/0 (socket says 172.21.15.31:58326) 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7fac9ad9d700 1 -- 172.21.15.31:0/314518387 learned_addr learned my addr 172.21.15.31:0/314518387 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7fac99d9b700 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.957+0000 7fac9a59c700 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad43ff6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac9ad9d700 1 -- 172.21.15.31:0/314518387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 msgr2=0x55ad43ff7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac9ad9d700 1 -- 172.21.15.31:0/314518387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 msgr2=0x55ad43ff6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad43ff6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac9ad9d700 1 -- 172.21.15.31:0/314518387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad43d74de0 con 0x55ad43ff5be0 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad43ff5ff0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fac94009440 tx=0x7fac94009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbd94baafe8b2e4e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac88ff9700 1 -- 172.21.15.31:0/314518387 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9400e070 con 0x55ad43ff5be0 2023-07-29T16:22:56.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac88ff9700 1 -- 172.21.15.31:0/314518387 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac9400f040 con 0x55ad43ff5be0 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.958+0000 7fac88ff9700 1 -- 172.21.15.31:0/314518387 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94014b20 con 0x55ad43ff5be0 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 -- 172.21.15.31:0/314518387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 msgr2=0x55ad43ff5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad43ff5ff0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fac94009440 tx=0x7fac94009920 comp rx=0 tx=0).stop 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 -- 172.21.15.31:0/314518387 shutdown_connections 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad43ff5ff0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 --2- 172.21.15.31:0/314518387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad43ff6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 -- 172.21.15.31:0/314518387 >> 172.21.15.31:0/314518387 conn(0x55ad43feb3e0 msgr2=0x55ad43ff0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 -- 172.21.15.31:0/314518387 shutdown_connections 2023-07-29T16:22:56.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.959+0000 7facb4557b80 1 -- 172.21.15.31:0/314518387 wait complete. 2023-07-29T16:22:56.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.960+0000 7fac8b7fe700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fac840a07f0 0x7fac840a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.960+0000 7fac8b7fe700 1 -- 172.21.15.31:0/640296744 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fac8c014070 con 0x7fac9c100da0 2023-07-29T16:22:56.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.960+0000 7fac99d9b700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fac840a07f0 0x7fac840a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.960+0000 7facb4557b80 1 -- start start 2023-07-29T16:22:56.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 Processor -- start 2023-07-29T16:22:56.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7fac99d9b700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fac840a07f0 0x7fac840a2ca0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x55ad43feb360 tx=0x7fac90005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad43fff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 0x55ad43fffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff72f0 0x55ad44006d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43ee3510 con 0x55ad43ff5be0 2023-07-29T16:22:56.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43ee31b0 con 0x55ad43ff6530 2023-07-29T16:22:56.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43dca920 con 0x55ad43ff72f0 2023-07-29T16:22:56.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7facb4557b80 10 client.0 _ll_register_callbacks cb 0x55ad43eec620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:56.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.961+0000 7fac99d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 0x55ad43fffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 0x55ad43fffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35440/0 (socket says 172.21.15.31:35440) 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 -- 172.21.15.31:0/1265606463 learned_addr learned my addr 172.21.15.31:0/1265606463 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff72f0 0x55ad44006d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac9a59c700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad43fff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 -- 172.21.15.31:0/1265606463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff72f0 msgr2=0x55ad44006d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff72f0 0x55ad44006d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 -- 172.21.15.31:0/1265606463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 msgr2=0x55ad43fff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad43fff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 -- 172.21.15.31:0/1265606463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac940090f0 con 0x55ad43ff6530 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac99d9b700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 0x55ad43fffd50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fac90015000 tx=0x7fac90009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86060ac45543bdac server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.962+0000 7fac81ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7fac81ffb700 1 -- 172.21.15.31:0/1265606463 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90021050 con 0x55ad43ff6530 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad43f71d10 con 0x55ad43ff6530 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7fac81ffb700 1 -- 172.21.15.31:0/1265606463 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fac90004500 con 0x55ad43ff6530 2023-07-29T16:22:56.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7fac81ffb700 1 -- 172.21.15.31:0/1265606463 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac900056d0 con 0x55ad43ff6530 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7facb4557b80 10 client.27791 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ad43d5dc60 con 0x55ad43ff6530 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7fac81ffb700 1 -- 172.21.15.31:0/1265606463 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ad43f71d10 con 0x55ad43ff6530 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7fac81ffb700 1 -- 172.21.15.31:0/1265606463 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fac9000f8a0 con 0x55ad43ff6530 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7facb4557b80 10 client.27791 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.963+0000 7facb4557b80 10 client.27791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:56.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7facb4557b80 20 client.27791 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:56.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7facb4557b80 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4400b500 0x55ad4400d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7facb4557b80 4 client.27791 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:56.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad43fc4d70 con 0x55ad4400b500 2023-07-29T16:22:56.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7fac9a59c700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4400b500 0x55ad4400d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7fac9a59c700 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4400b500 0x55ad4400d910 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=494fdb530e5230ca server_cookie=cb0229dbca0eccac in_seq=0 out_seq=0 2023-07-29T16:22:56.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.964+0000 7fac81ffb700 10 client.27791 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7fac81ffb700 1 -- 172.21.15.31:0/1265606463 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x7fac8c0b1e90 con 0x55ad4400b500 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7fac81ffb700 10 client.27791 handle_command_reply: tid=0 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 client.27791 shutdown 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 msgr2=0x55ad43fffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 0x55ad43fffd50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fac90015000 tx=0x7fac90009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 shutdown_connections 2023-07-29T16:22:56.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4400b500 0x55ad4400d910 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff72f0 0x55ad44006d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad43fff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 --2- 172.21.15.31:0/1265606463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff6530 0x55ad43fffd50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 >> 172.21.15.31:0/1265606463 conn(0x55ad43feb3e0 msgr2=0x55ad43ff0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7fac81ffb700 0 client.27791 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:56.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 shutdown_connections 2023-07-29T16:22:56.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.966+0000 7facb4557b80 1 -- 172.21.15.31:0/1265606463 wait complete. 2023-07-29T16:22:56.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.967+0000 7facb4557b80 20 client.27791 trim_cache size 0 max 16384 2023-07-29T16:22:56.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 Processor -- start 2023-07-29T16:22:56.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 -- start start 2023-07-29T16:22:56.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad4400d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff6530 0x55ad44006d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad44012290 con 0x55ad43ff5be0 2023-07-29T16:22:56.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad44012400 con 0x55ad43ff72f0 2023-07-29T16:22:56.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7facb4557b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad43ff1690 con 0x55ad43ff6530 2023-07-29T16:22:56.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.982+0000 7fac9ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad4400d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35450/0 (socket says 172.21.15.31:35450) 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 -- 172.21.15.31:0/4108461709 learned_addr learned my addr 172.21.15.31:0/4108461709 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac99d9b700 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff6530 0x55ad44006d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 -- 172.21.15.31:0/4108461709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff6530 msgr2=0x55ad44006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff6530 0x55ad44006d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 -- 172.21.15.31:0/4108461709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 msgr2=0x55ad4400d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad4400d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 -- 172.21.15.31:0/4108461709 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad43d9b250 con 0x55ad43ff72f0 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff76c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55ad43ff5980 tx=0x7fac94009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f7bd28816872751 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac897fa700 1 -- 172.21.15.31:0/4108461709 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9400e040 con 0x55ad43ff72f0 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac897fa700 1 -- 172.21.15.31:0/4108461709 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac94004c70 con 0x55ad43ff72f0 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.983+0000 7fac897fa700 1 -- 172.21.15.31:0/4108461709 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac94014ea0 con 0x55ad43ff72f0 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 -- 172.21.15.31:0/4108461709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 msgr2=0x55ad43ff76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff76c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x55ad43ff5980 tx=0x7fac94009d40 comp rx=0 tx=0).stop 2023-07-29T16:22:56.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 -- 172.21.15.31:0/4108461709 shutdown_connections 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad43ff76c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff5be0 0x55ad4400d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 --2- 172.21.15.31:0/4108461709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff6530 0x55ad44006d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 -- 172.21.15.31:0/4108461709 >> 172.21.15.31:0/4108461709 conn(0x55ad43feb3e0 msgr2=0x55ad43ffe510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 -- 172.21.15.31:0/4108461709 shutdown_connections 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.984+0000 7facb4557b80 1 -- 172.21.15.31:0/4108461709 wait complete. 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 -- start start 2023-07-29T16:22:56.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 Processor -- start 2023-07-29T16:22:56.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad44002f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad440034a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad440064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad44006a00 con 0x55ad43ff6530 2023-07-29T16:22:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad44006b70 con 0x55ad43ff72f0 2023-07-29T16:22:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad44006d30 con 0x55ad43ff5be0 2023-07-29T16:22:56.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.985+0000 7facb4557b80 10 client.0 _ll_register_callbacks cb 0x55ad43eec620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad440064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad440064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35460/0 (socket says 172.21.15.31:35460) 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 -- 172.21.15.31:0/2288497288 learned_addr learned my addr 172.21.15.31:0/2288497288 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac99d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad440034a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9a59c700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad44002f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 -- 172.21.15.31:0/2288497288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 msgr2=0x55ad44002f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad44002f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 -- 172.21.15.31:0/2288497288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 msgr2=0x55ad440034a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad440034a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 -- 172.21.15.31:0/2288497288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad43d74de0 con 0x55ad43ff72f0 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9a59c700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad44002f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac9ad9d700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad440064c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fac94014420 tx=0x7fac940147a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffa0c60529d1e670 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.986+0000 7fac81ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:56.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7fac81ffb700 1 -- 172.21.15.31:0/2288497288 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9400e050 con 0x55ad43ff72f0 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7fac81ffb700 1 -- 172.21.15.31:0/2288497288 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fac94015440 con 0x55ad43ff72f0 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fac9400f040 con 0x55ad43ff72f0 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7fac81ffb700 1 -- 172.21.15.31:0/2288497288 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac94015600 con 0x55ad43ff72f0 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 10 client.27797 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ad43d732b0 con 0x55ad43ff72f0 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7fac81ffb700 1 -- 172.21.15.31:0/2288497288 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac9403e020 con 0x55ad43ff72f0 2023-07-29T16:22:56.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7fac81ffb700 1 -- 172.21.15.31:0/2288497288 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fac9401c070 con 0x55ad43ff72f0 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 10 client.27797 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 10 client.27797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 20 client.27797 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4405e020 0x55ad44060430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 4 client.27797 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.987+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ad43f3fd40 con 0x55ad4405e020 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.988+0000 7fac9a59c700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4405e020 0x55ad44060430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:56.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.988+0000 7fac9a59c700 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4405e020 0x55ad44060430 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=735216f4577c021 server_cookie=dafd46fe688e322f in_seq=0 out_seq=0 2023-07-29T16:22:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.988+0000 7fac81ffb700 10 client.27797 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7fac81ffb700 1 -- 172.21.15.31:0/2288497288 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ad43f3fd40 con 0x55ad4405e020 2023-07-29T16:22:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7fac81ffb700 10 client.27797 handle_command_reply: tid=0 2023-07-29T16:22:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 client.27797 shutdown 2023-07-29T16:22:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 msgr2=0x55ad440064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad440064c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fac94014420 tx=0x7fac940147a0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 shutdown_connections 2023-07-29T16:22:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ad4405e020 0x55ad44060430 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ad43ff72f0 0x55ad440064c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ad43ff5be0 0x55ad44002f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7fac81ffb700 0 client.27797 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 --2- 172.21.15.31:0/2288497288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ad43ff6530 0x55ad440034a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.990+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 >> 172.21.15.31:0/2288497288 conn(0x55ad43feb3e0 msgr2=0x55ad43ffd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 shutdown_connections 2023-07-29T16:22:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7facb4557b80 1 -- 172.21.15.31:0/2288497288 wait complete. 2023-07-29T16:22:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7facb4557b80 20 client.27797 trim_cache size 0 max 16384 2023-07-29T16:22:56.997 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:22:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fac840a07f0 msgr2=0x7fac840a2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fac840a07f0 0x7fac840a2ca0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x55ad43feb360 tx=0x7fac90005fb0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 msgr2=0x7fac9c196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 0x7fac9c196050 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00e9a0 tx=0x7fac8c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 shutdown_connections 2023-07-29T16:22:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fac840a07f0 0x7fac840a2ca0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fac9c105fe0 0x7fac9c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac9c100da0 0x7fac9c196050 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 --2- 172.21.15.31:0/640296744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9c1036c0 0x7fac9c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:56.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.991+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 >> 172.21.15.31:0/640296744 conn(0x7fac9c0fa840 msgr2=0x7fac9c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:56.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.992+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 shutdown_connections 2023-07-29T16:22:56.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:56.992+0000 7faca0bff700 1 -- 172.21.15.31:0/640296744 wait complete. 2023-07-29T16:22:57.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 472922 9.07 MB/sec execute 162 sec latency 47.402 ms 2023-07-29T16:22:58.010 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:22:58.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:22:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:57 smithi031 ceph-mon[95504]: pgmap v595: 97 pgs: 97 active+clean; 88 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.1 MiB/s wr, 225 op/s 2023-07-29T16:22:58.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 -- 172.21.15.31:0/2348024993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 msgr2=0x7fb240103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 --2- 172.21.15.31:0/2348024993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb240103240 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb234009990 tx=0x7fb234009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 -- 172.21.15.31:0/2348024993 shutdown_connections 2023-07-29T16:22:58.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 --2- 172.21.15.31:0/2348024993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2401060a0 0x7fb240108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 --2- 172.21.15.31:0/2348024993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb240103240 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 --2- 172.21.15.31:0/2348024993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb240103780 0x7fb240105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 -- 172.21.15.31:0/2348024993 >> 172.21.15.31:0/2348024993 conn(0x7fb2400fa840 msgr2=0x7fb2400fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 -- 172.21.15.31:0/2348024993 shutdown_connections 2023-07-29T16:22:58.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.234+0000 7fb246dff700 1 -- 172.21.15.31:0/2348024993 wait complete. 2023-07-29T16:22:58.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.235+0000 7fb246dff700 1 Processor -- start 2023-07-29T16:22:58.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.235+0000 7fb246dff700 1 -- start start 2023-07-29T16:22:58.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb246dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb2401982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb246dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 0x7fb240198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb246dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2401060a0 0x7fb24019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb246dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24019fe60 con 0x7fb240100e60 2023-07-29T16:22:58.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb246dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24019ffa0 con 0x7fb2401060a0 2023-07-29T16:22:58.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb246dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401a0130 con 0x7fb240103780 2023-07-29T16:22:58.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb244b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb2401982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 0x7fb240198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 0x7fb240198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58380/0 (socket says 172.21.15.31:58380) 2023-07-29T16:22:58.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 -- 172.21.15.31:0/1817941544 learned_addr learned my addr 172.21.15.31:0/1817941544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb244b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb2401982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50934/0 (socket says 172.21.15.31:50934) 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb24539c700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2401060a0 0x7fb24019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 -- 172.21.15.31:0/1817941544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2401060a0 msgr2=0x7fb24019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2401060a0 0x7fb24019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 -- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 msgr2=0x7fb2401982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb2401982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb23ffff700 1 -- 172.21.15.31:0/1817941544 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c009520 con 0x7fb240103780 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb24539c700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2401060a0 0x7fb24019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.236+0000 7fb244b9b700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb2401982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.237+0000 7fb23ffff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 0x7fb240198820 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00eb30 tx=0x7fb22c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.237+0000 7fb23dffb700 1 -- 172.21.15.31:0/1817941544 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c009880 con 0x7fb240103780 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.237+0000 7fb23dffb700 1 -- 172.21.15.31:0/1817941544 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb22c0044e0 con 0x7fb240103780 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.237+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2340095f0 con 0x7fb240103780 2023-07-29T16:22:58.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.237+0000 7fb23dffb700 1 -- 172.21.15.31:0/1817941544 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c003e90 con 0x7fb240103780 2023-07-29T16:22:58.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.237+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401a0790 con 0x7fb240103780 2023-07-29T16:22:58.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.240+0000 7fb23dffb700 1 -- 172.21.15.31:0/1817941544 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb22c004000 con 0x7fb240103780 2023-07-29T16:22:58.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 Processor -- start 2023-07-29T16:22:58.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 -- start start 2023-07-29T16:22:58.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb23dffb700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2300a0790 0x7fb2300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f392e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f392ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392f370 0x55f9f392f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3885a90 con 0x55f9f392dc60 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3883450 con 0x55f9f392e5b0 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f387ed40 con 0x55f9f392f370 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.242+0000 7fb23dffb700 1 -- 172.21.15.31:0/1817941544 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb22c014070 con 0x7fb240103780 2023-07-29T16:22:58.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f392ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb24539c700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2300a0790 0x7fb2300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f392ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35478/0 (socket says 172.21.15.31:35478) 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 -- 172.21.15.31:0/536818121 learned_addr learned my addr 172.21.15.31:0/536818121 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb244b9b700 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f392e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb24539c700 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392f370 0x55f9f392f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 -- 172.21.15.31:0/536818121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392f370 msgr2=0x55f9f392f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392f370 0x55f9f392f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 -- 172.21.15.31:0/536818121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 msgr2=0x55f9f392e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f392e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 -- 172.21.15.31:0/536818121 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9f36ace60 con 0x55f9f392e5b0 2023-07-29T16:22:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb23ffff700 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f392ede0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb22c095c40 tx=0x7fb22c099510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d538b6d1a817104 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb24539c700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2300a0790 0x7fb2300a2c40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb238012630 tx=0x7fb238009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb22affd700 1 -- 172.21.15.31:0/536818121 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00cc90 con 0x55f9f392e5b0 2023-07-29T16:22:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb22affd700 1 -- 172.21.15.31:0/536818121 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb22c04b070 con 0x55f9f392e5b0 2023-07-29T16:22:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.243+0000 7fb22affd700 1 -- 172.21.15.31:0/536818121 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c051c60 con 0x55f9f392e5b0 2023-07-29T16:22:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/536818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 msgr2=0x55f9f392ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f392ede0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb22c095c40 tx=0x7fb22c099510 comp rx=0 tx=0).stop 2023-07-29T16:22:58.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/536818121 shutdown_connections 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392f370 0x55f9f392f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f392e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/536818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f392ede0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/536818121 >> 172.21.15.31:0/536818121 conn(0x55f9f3923460 msgr2=0x55f9f3928a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/536818121 shutdown_connections 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.244+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/536818121 wait complete. 2023-07-29T16:22:58.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.245+0000 7fb25a6c7b80 1 -- start start 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.245+0000 7fb25a6c7b80 1 Processor -- start 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.245+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f39378a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f3937de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f393ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f381b590 con 0x55f9f392dc60 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f381b230 con 0x55f9f392f370 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f37029a0 con 0x55f9f392e5b0 2023-07-29T16:22:58.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb25a6c7b80 10 client.0 _ll_register_callbacks cb 0x55f9f38246a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f39378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f39378a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50958/0 (socket says 172.21.15.31:50958) 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 -- 172.21.15.31:0/3794875675 learned_addr learned my addr 172.21.15.31:0/3794875675 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f3937de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb24539c700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f393ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 -- 172.21.15.31:0/3794875675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 msgr2=0x55f9f3937de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f3937de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 -- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 msgr2=0x55f9f393ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f393ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb244b9b700 1 -- 172.21.15.31:0/3794875675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401a0790 con 0x55f9f392dc60 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb24539c700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f393ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.246+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f3937de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb244b9b700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f39378a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb234009990 tx=0x7fb234005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3432e1c721d1f1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb21ffff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:22:58.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb21ffff700 1 -- 172.21.15.31:0/3794875675 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb23401d070 con 0x55f9f392dc60 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9f38a9d90 con 0x55f9f392dc60 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb21ffff700 1 -- 172.21.15.31:0/3794875675 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb2340056c0 con 0x55f9f392dc60 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb21ffff700 1 -- 172.21.15.31:0/3794875675 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb234004910 con 0x55f9f392dc60 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb25a6c7b80 10 client.17766 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.247+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f9f3695ce0 con 0x55f9f392dc60 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb21ffff700 1 -- 172.21.15.31:0/3794875675 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f9f38a9d90 con 0x55f9f392dc60 2023-07-29T16:22:58.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb21ffff700 1 -- 172.21.15.31:0/3794875675 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb234024b20 con 0x55f9f392dc60 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb25a6c7b80 10 client.17766 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb25a6c7b80 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb25a6c7b80 20 client.17766 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f3943610 0x55f9f3945a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb25a6c7b80 4 client.17766 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9f38fcdf0 con 0x55f9f3943610 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.248+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f3943610 0x55f9f3945a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.249+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f3943610 0x55f9f3945a20 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9794d8f2fee371 server_cookie=fcb8ace38dde7b46 in_seq=0 out_seq=0 2023-07-29T16:22:58.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.249+0000 7fb21ffff700 10 client.17766 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.249+0000 7fb21ffff700 1 -- 172.21.15.31:0/3794875675 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f9f38fcdf0 con 0x55f9f3943610 2023-07-29T16:22:58.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.249+0000 7fb21ffff700 10 client.17766 handle_command_reply: tid=0 2023-07-29T16:22:58.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.249+0000 7fb25a6c7b80 1 client.17766 shutdown 2023-07-29T16:22:58.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 msgr2=0x55f9f39378a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f39378a0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb234009990 tx=0x7fb234005270 comp rx=0 tx=0).stop 2023-07-29T16:22:58.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 shutdown_connections 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f3943610 0x55f9f3945a20 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f393ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f39378a0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3794875675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f3937de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 >> 172.21.15.31:0/3794875675 conn(0x55f9f3923460 msgr2=0x55f9f39280e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb21ffff700 0 client.17766 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 shutdown_connections 2023-07-29T16:22:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3794875675 wait complete. 2023-07-29T16:22:58.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.250+0000 7fb25a6c7b80 20 client.17766 trim_cache size 0 max 16384 2023-07-29T16:22:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.268+0000 7fb25a6c7b80 1 Processor -- start 2023-07-29T16:22:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 -- start start 2023-07-29T16:22:58.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f3932270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f393ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f39327b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f392a430 con 0x55f9f392dc60 2023-07-29T16:22:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3935510 con 0x55f9f392f370 2023-07-29T16:22:58.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3943330 con 0x55f9f392e5b0 2023-07-29T16:22:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb24539c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f39327b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb24539c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f39327b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35504/0 (socket says 172.21.15.31:35504) 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb24539c700 1 -- 172.21.15.31:0/3312313104 learned_addr learned my addr 172.21.15.31:0/3312313104 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f393ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.269+0000 7fb244b9b700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f3932270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb23ffff700 1 -- 172.21.15.31:0/3312313104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 msgr2=0x55f9f39327b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f39327b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb23ffff700 1 -- 172.21.15.31:0/3312313104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 msgr2=0x55f9f3932270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f3932270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb23ffff700 1 -- 172.21.15.31:0/3312313104 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9f36d32d0 con 0x55f9f392e5b0 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb244b9b700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f3932270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb24539c700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f39327b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb23ffff700 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f393ede0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55f9f392e9c0 tx=0x7fb22c003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9663246bb44e3106 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb22b7fe700 1 -- 172.21.15.31:0/3312313104 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c00cc90 con 0x55f9f392e5b0 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb22b7fe700 1 -- 172.21.15.31:0/3312313104 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb22c09d770 con 0x55f9f392e5b0 2023-07-29T16:22:58.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.270+0000 7fb22b7fe700 1 -- 172.21.15.31:0/3312313104 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c098c30 con 0x55f9f392e5b0 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3312313104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 msgr2=0x55f9f393ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f393ede0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55f9f392e9c0 tx=0x7fb22c003680 comp rx=0 tx=0).stop 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3312313104 shutdown_connections 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392f370 0x55f9f39327b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392dc60 0x55f9f3932270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/3312313104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392e5b0 0x55f9f393ede0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3312313104 >> 172.21.15.31:0/3312313104 conn(0x55f9f3923460 msgr2=0x55f9f3935730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3312313104 shutdown_connections 2023-07-29T16:22:58.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/3312313104 wait complete. 2023-07-29T16:22:58.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.271+0000 7fb25a6c7b80 1 -- start start 2023-07-29T16:22:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 Processor -- start 2023-07-29T16:22:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392dc60 0x55f9f393ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f393b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392f370 0x55f9f393bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f393c110 con 0x55f9f392f370 2023-07-29T16:22:58.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f393c280 con 0x55f9f392e5b0 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f3995190 con 0x55f9f392dc60 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb25a6c7b80 10 client.0 _ll_register_callbacks cb 0x55f9f38246a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f393b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb24539c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392f370 0x55f9f393bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb244b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392dc60 0x55f9f393ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb23ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f393b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35514/0 (socket says 172.21.15.31:35514) 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.272+0000 7fb23ffff700 1 -- 172.21.15.31:0/1770966996 learned_addr learned my addr 172.21.15.31:0/1770966996 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb23ffff700 1 -- 172.21.15.31:0/1770966996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392dc60 msgr2=0x55f9f393ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb23ffff700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392dc60 0x55f9f393ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb23ffff700 1 -- 172.21.15.31:0/1770966996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392f370 msgr2=0x55f9f393bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb23ffff700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392f370 0x55f9f393bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb23ffff700 1 -- 172.21.15.31:0/1770966996 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9f3695ce0 con 0x55f9f392e5b0 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb244b9b700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392dc60 0x55f9f393ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb24539c700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392f370 0x55f9f393bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb23ffff700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f393b690 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb22c095c40 tx=0x7fb22c099380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=980627c19821e9ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb21ffff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb21ffff700 1 -- 172.21.15.31:0/1770966996 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c099740 con 0x55f9f392e5b0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb21ffff700 1 -- 172.21.15.31:0/1770966996 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb22c00cc90 con 0x55f9f392e5b0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb22c04b070 con 0x55f9f392e5b0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb21ffff700 1 -- 172.21.15.31:0/1770966996 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb22c097b50 con 0x55f9f392e5b0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb25a6c7b80 10 client.27815 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.273+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fb238022040 con 0x55f9f392e5b0 2023-07-29T16:22:58.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb21ffff700 1 -- 172.21.15.31:0/1770966996 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb22c04b070 con 0x55f9f392e5b0 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb21ffff700 1 -- 172.21.15.31:0/1770966996 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb22c0b6070 con 0x55f9f392e5b0 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb25a6c7b80 10 client.27815 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb25a6c7b80 10 client.27815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb25a6c7b80 20 client.27815 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f39966b0 0x55f9f3998ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb25a6c7b80 4 client.27815 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f9f3877dc0 con 0x55f9f39966b0 2023-07-29T16:22:58.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.274+0000 7fb244b9b700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f39966b0 0x55f9f3998ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.275+0000 7fb244b9b700 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f39966b0 0x55f9f3998ac0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3b27e90285efaba server_cookie=36971ae89a737c00 in_seq=0 out_seq=0 2023-07-29T16:22:58.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.275+0000 7fb21ffff700 10 client.27815 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:57 smithi039 ceph-mon[128858]: pgmap v595: 97 pgs: 97 active+clean; 88 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.1 MiB/s wr, 225 op/s 2023-07-29T16:22:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb21ffff700 1 -- 172.21.15.31:0/1770966996 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f9f3877dc0 con 0x55f9f39966b0 2023-07-29T16:22:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb21ffff700 10 client.27815 handle_command_reply: tid=0 2023-07-29T16:22:58.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 client.27815 shutdown 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 msgr2=0x55f9f393b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f393b690 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb22c095c40 tx=0x7fb22c099380 comp rx=0 tx=0).stop 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 shutdown_connections 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f9f39966b0 0x55f9f3998ac0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f9f392f370 0x55f9f393bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f9f392dc60 0x55f9f393ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 --2- 172.21.15.31:0/1770966996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f9f392e5b0 0x55f9f393b690 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb21ffff700 0 client.27815 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 >> 172.21.15.31:0/1770966996 conn(0x55f9f3923460 msgr2=0x55f9f3936560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 shutdown_connections 2023-07-29T16:22:58.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 1 -- 172.21.15.31:0/1770966996 wait complete. 2023-07-29T16:22:58.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.278+0000 7fb25a6c7b80 20 client.27815 trim_cache size 0 max 16384 2023-07-29T16:22:58.284 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:22:58.285 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:22:58.285 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "67757a35-6d27-47c3-99c3-f6c8f2d9930f", 2023-07-29T16:22:58.285 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:22:58.285 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2300a0790 msgr2=0x7fb2300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2300a0790 0x7fb2300a2c40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb238012630 tx=0x7fb238009920 comp rx=0 tx=0).stop 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 msgr2=0x7fb240198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 0x7fb240198820 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00eb30 tx=0x7fb22c00ee40 comp rx=0 tx=0).stop 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 shutdown_connections 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2300a0790 0x7fb2300a2c40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2401060a0 0x7fb24019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb240100e60 0x7fb2401982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 --2- 172.21.15.31:0/1817941544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb240103780 0x7fb240198820 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 >> 172.21.15.31:0/1817941544 conn(0x7fb2400fa840 msgr2=0x7fb2400ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 shutdown_connections 2023-07-29T16:22:58.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.279+0000 7fb246dff700 1 -- 172.21.15.31:0/1817941544 wait complete. 2023-07-29T16:22:58.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:22:58.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:57 smithi156 ceph-mon[129245]: pgmap v595: 97 pgs: 97 active+clean; 88 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.1 MiB/s wr, 225 op/s 2023-07-29T16:22:58.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.527+0000 7f48497ff700 1 -- 172.21.15.31:0/2238478651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 msgr2=0x7f4844102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.527+0000 7f48497ff700 1 --2- 172.21.15.31:0/2238478651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f4844102960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4834009990 tx=0x7f4834009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 -- 172.21.15.31:0/2238478651 shutdown_connections 2023-07-29T16:22:58.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 --2- 172.21.15.31:0/2238478651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48441040d0 0x7f484410a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 --2- 172.21.15.31:0/2238478651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f4844102960 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 --2- 172.21.15.31:0/2238478651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4844103740 0x7f4844103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 -- 172.21.15.31:0/2238478651 >> 172.21.15.31:0/2238478651 conn(0x7f48440fdb90 msgr2=0x7f48440fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 -- 172.21.15.31:0/2238478651 shutdown_connections 2023-07-29T16:22:58.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.528+0000 7f48497ff700 1 -- 172.21.15.31:0/2238478651 wait complete. 2023-07-29T16:22:58.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 Processor -- start 2023-07-29T16:22:58.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 -- start start 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f48441982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4844103740 0x7f4844198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48441040d0 0x7f484419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484419fe50 con 0x7f48441040d0 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484419ff90 con 0x7f4844102550 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.529+0000 7f48497ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48441a0120 con 0x7f4844103740 2023-07-29T16:22:58.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f48441982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f48441982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35534/0 (socket says 172.21.15.31:35534) 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 -- 172.21.15.31:0/2252157106 learned_addr learned my addr 172.21.15.31:0/2252157106 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f48427fc700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4844103740 0x7f4844198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f48437fe700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48441040d0 0x7f484419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 -- 172.21.15.31:0/2252157106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4844103740 msgr2=0x7f4844198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4844103740 0x7f4844198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 -- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48441040d0 msgr2=0x7f484419f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48441040d0 0x7f484419f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 -- 172.21.15.31:0/2252157106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48340095f0 con 0x7f4844102550 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f48441982d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4834004730 tx=0x7f4834004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.530+0000 7f482bfff700 1 -- 172.21.15.31:0/2252157106 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483401d070 con 0x7f4844102550 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.531+0000 7f482bfff700 1 -- 172.21.15.31:0/2252157106 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4834021410 con 0x7f4844102550 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.531+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48441a03a0 con 0x7f4844102550 2023-07-29T16:22:58.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.531+0000 7f482bfff700 1 -- 172.21.15.31:0/2252157106 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834017650 con 0x7f4844102550 2023-07-29T16:22:58.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.531+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48441a0820 con 0x7f4844102550 2023-07-29T16:22:58.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.534+0000 7f482bfff700 1 -- 172.21.15.31:0/2252157106 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4834021580 con 0x7f4844102550 2023-07-29T16:22:58.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.535+0000 7f485d0d4b80 1 Processor -- start 2023-07-29T16:22:58.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.535+0000 7f482bfff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f482c0a0810 0x7f482c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.535+0000 7f485d0d4b80 1 -- start start 2023-07-29T16:22:58.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f482bfff700 1 -- 172.21.15.31:0/2252157106 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f48340cce30 con 0x7f4844102550 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 0x55c86c0adff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0af2f0 0x55c86c0af740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c005a10 con 0x55c86c0ae530 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f48437fe700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f482c0a0810 0x7f482c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0033d0 con 0x55c86c0af2f0 2023-07-29T16:22:58.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86bffecc0 con 0x55c86c0adbe0 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 0x55c86c0adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 0x55c86c0adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58434/0 (socket says 172.21.15.31:58434) 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 -- 172.21.15.31:0/2284173438 learned_addr learned my addr 172.21.15.31:0/2284173438 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f48427fc700 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f48437fe700 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0af2f0 0x55c86c0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 -- 172.21.15.31:0/2284173438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0af2f0 msgr2=0x55c86c0af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0af2f0 0x55c86c0af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 -- 172.21.15.31:0/2284173438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 msgr2=0x55c86c0aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.536+0000 7f4842ffd700 1 -- 172.21.15.31:0/2284173438 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c86be2cde0 con 0x55c86c0adbe0 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.537+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 0x55c86c0adff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4834096a60 tx=0x7f483409a120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c8ca8acdafb0099 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.537+0000 7f48437fe700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f482c0a0810 0x7f482c0a2cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f483c009ec0 tx=0x7f483c013480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.537+0000 7f48297fa700 1 -- 172.21.15.31:0/2284173438 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483409e590 con 0x55c86c0adbe0 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.537+0000 7f48297fa700 1 -- 172.21.15.31:0/2284173438 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48340a2920 con 0x55c86c0adbe0 2023-07-29T16:22:58.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.537+0000 7f48297fa700 1 -- 172.21.15.31:0/2284173438 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483409a7c0 con 0x55c86c0adbe0 2023-07-29T16:22:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2284173438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 msgr2=0x55c86c0adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 0x55c86c0adff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4834096a60 tx=0x7f483409a120 comp rx=0 tx=0).stop 2023-07-29T16:22:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2284173438 shutdown_connections 2023-07-29T16:22:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0af2f0 0x55c86c0af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0adbe0 0x55c86c0adff0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2284173438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2284173438 >> 172.21.15.31:0/2284173438 conn(0x55c86c0a33e0 msgr2=0x55c86c0a8990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2284173438 shutdown_connections 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2284173438 wait complete. 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 -- start start 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.538+0000 7f485d0d4b80 1 Processor -- start 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0adbe0 0x55c86c0b7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 0x55c86c0b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0bea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86bf9b510 con 0x55c86c0adbe0 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86bf9b1b0 con 0x55c86c0ae530 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86be82920 con 0x55c86c0af2f0 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f485d0d4b80 10 client.0 _ll_register_callbacks cb 0x55c86bfa4620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f48427fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 0x55c86c0b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f48427fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 0x55c86c0b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35546/0 (socket says 172.21.15.31:35546) 2023-07-29T16:22:58.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f48427fc700 1 -- 172.21.15.31:0/2224137700 learned_addr learned my addr 172.21.15.31:0/2224137700 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0adbe0 0x55c86c0b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.539+0000 7f48437fe700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0bea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f48427fc700 1 -- 172.21.15.31:0/2224137700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 msgr2=0x55c86c0bea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f48427fc700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0bea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f48427fc700 1 -- 172.21.15.31:0/2224137700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0adbe0 msgr2=0x55c86c0b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f48427fc700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0adbe0 0x55c86c0b7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f48427fc700 1 -- 172.21.15.31:0/2224137700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48441a0820 con 0x55c86c0ae530 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f48427fc700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 0x55c86c0b79d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4838011780 tx=0x7f4838011a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f255d43bfb51ac server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f4826ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f4826ff5700 1 -- 172.21.15.31:0/2224137700 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838016070 con 0x55c86c0ae530 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c86c029d10 con 0x55c86c0ae530 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f4826ff5700 1 -- 172.21.15.31:0/2224137700 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4838004830 con 0x55c86c0ae530 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f4826ff5700 1 -- 172.21.15.31:0/2224137700 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4838004a30 con 0x55c86c0ae530 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f485d0d4b80 10 client.27827 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:58.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.540+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c86be15c60 con 0x55c86c0ae530 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f4826ff5700 1 -- 172.21.15.31:0/2224137700 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4838028020 con 0x55c86c0ae530 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f4826ff5700 1 -- 172.21.15.31:0/2224137700 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f48380050b0 con 0x55c86c0ae530 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f485d0d4b80 10 client.27827 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f485d0d4b80 10 client.27827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f485d0d4b80 20 client.27827 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c0c3290 0x55c86c0c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f485d0d4b80 4 client.27827 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:22:58.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c86c07cd70 con 0x55c86c0c3290 2023-07-29T16:22:58.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.541+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c0c3290 0x55c86c0c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.542+0000 7f4842ffd700 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c0c3290 0x55c86c0c56a0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=536ad120c5e5b2b4 server_cookie=46bf8f7879cb7180 in_seq=0 out_seq=0 2023-07-29T16:22:58.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.542+0000 7f4826ff5700 10 client.27827 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f4826ff5700 1 -- 172.21.15.31:0/2224137700 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c86c07cd70 con 0x55c86c0c3290 2023-07-29T16:22:58.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f4826ff5700 10 client.27827 handle_command_reply: tid=0 2023-07-29T16:22:58.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 client.27827 shutdown 2023-07-29T16:22:58.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 msgr2=0x55c86c0b79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 0x55c86c0b79d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4838011780 tx=0x7f4838011a90 comp rx=0 tx=0).stop 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 shutdown_connections 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c0c3290 0x55c86c0c56a0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0bea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0adbe0 0x55c86c0b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/2224137700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0ae530 0x55c86c0b79d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f4826ff5700 0 client.27827 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.543+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 >> 172.21.15.31:0/2224137700 conn(0x55c86c0a33e0 msgr2=0x55c86c0a8060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.544+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 shutdown_connections 2023-07-29T16:22:58.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.544+0000 7f485d0d4b80 1 -- 172.21.15.31:0/2224137700 wait complete. 2023-07-29T16:22:58.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.544+0000 7f485d0d4b80 20 client.27827 trim_cache size 0 max 16384 2023-07-29T16:22:58.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 Processor -- start 2023-07-29T16:22:58.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 -- start start 2023-07-29T16:22:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0be9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0b7000 con 0x55c86c0ae530 2023-07-29T16:22:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0a5460 con 0x55c86c0adbe0 2023-07-29T16:22:58.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.561+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0a9690 con 0x55c86c0af2f0 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f48427fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f48437fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0be9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35558/0 (socket says 172.21.15.31:35558) 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 -- 172.21.15.31:0/221108386 learned_addr learned my addr 172.21.15.31:0/221108386 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 -- 172.21.15.31:0/221108386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 msgr2=0x55c86c0be9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0be9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 -- 172.21.15.31:0/221108386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 msgr2=0x55c86c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 -- 172.21.15.31:0/221108386 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c86be53250 con 0x55c86c0adbe0 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f48427fc700 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4842ffd700 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0b21b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f483409a4f0 tx=0x7f4834009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5445c3d88f63c31f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.562+0000 7f4829ffb700 1 -- 172.21.15.31:0/221108386 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483409e590 con 0x55c86c0adbe0 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f4829ffb700 1 -- 172.21.15.31:0/221108386 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f483409e170 con 0x55c86c0adbe0 2023-07-29T16:22:58.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f4829ffb700 1 -- 172.21.15.31:0/221108386 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834099c70 con 0x55c86c0adbe0 2023-07-29T16:22:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 -- 172.21.15.31:0/221108386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 msgr2=0x55c86c0b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0b21b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f483409a4f0 tx=0x7f4834009ca0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 -- 172.21.15.31:0/221108386 shutdown_connections 2023-07-29T16:22:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0af2f0 0x55c86c0be9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0b21b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/221108386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0ae530 0x55c86c0b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 -- 172.21.15.31:0/221108386 >> 172.21.15.31:0/221108386 conn(0x55c86c0a33e0 msgr2=0x55c86c0b5630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 -- 172.21.15.31:0/221108386 shutdown_connections 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.563+0000 7f485d0d4b80 1 -- 172.21.15.31:0/221108386 wait complete. 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 -- start start 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 Processor -- start 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0be4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0ae530 0x55c86c0bafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0af2f0 0x55c86c0bb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0bbbf0 con 0x55c86c0af2f0 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0beb60 con 0x55c86c0adbe0 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c86c0becf0 con 0x55c86c0ae530 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.564+0000 7f485d0d4b80 10 client.0 _ll_register_callbacks cb 0x55c86bfa4620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0be4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0ae530 0x55c86c0bafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f4842ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0be4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35570/0 (socket says 172.21.15.31:35570) 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f4842ffd700 1 -- 172.21.15.31:0/1428246857 learned_addr learned my addr 172.21.15.31:0/1428246857 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48437fe700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0af2f0 0x55c86c0bb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 -- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 msgr2=0x55c86c0be4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0be4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 -- 172.21.15.31:0/1428246857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0af2f0 msgr2=0x55c86c0bb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:22:58.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0af2f0 0x55c86c0bb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 -- 172.21.15.31:0/1428246857 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c86be15c60 con 0x55c86c0ae530 2023-07-29T16:22:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f4842ffd700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:22:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f48427fc700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0ae530 0x55c86c0bafd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4838005510 tx=0x7f48380110c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=443f82f73a618d00 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:22:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.565+0000 7f4826ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:22:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f4826ff5700 1 -- 172.21.15.31:0/1428246857 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838016040 con 0x55c86c0ae530 2023-07-29T16:22:58.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f4826ff5700 1 -- 172.21.15.31:0/1428246857 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4838029080 con 0x55c86c0ae530 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48340a2920 con 0x55c86c0ae530 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f4826ff5700 1 -- 172.21.15.31:0/1428246857 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4838011360 con 0x55c86c0ae530 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 10 client.27844 fetch_fsmap learned FSMap version 28 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c86be2cde0 con 0x55c86c0ae530 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f4826ff5700 1 -- 172.21.15.31:0/1428246857 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f483803a020 con 0x55c86c0ae530 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f4826ff5700 1 -- 172.21.15.31:0/1428246857 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4838024440 con 0x55c86c0ae530 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 10 client.27844 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:22:58.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 10 client.27844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 20 client.27844 populate_metadata read hostname 'smithi031' 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c116570 0x55c86c118980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 4 client.27844 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.566+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c86bff7d40 con 0x55c86c116570 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.567+0000 7f4842ffd700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c116570 0x55c86c118980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.567+0000 7f4842ffd700 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c116570 0x55c86c118980 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8edbd9d6cc085b3b server_cookie=a2cd9dd64736d60b in_seq=0 out_seq=0 2023-07-29T16:22:58.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.567+0000 7f4826ff5700 10 client.27844 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f4826ff5700 1 -- 172.21.15.31:0/1428246857 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c86bff7d40 con 0x55c86c116570 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f4826ff5700 10 client.27844 handle_command_reply: tid=0 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 client.27844 shutdown 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0ae530 msgr2=0x55c86c0bafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0ae530 0x55c86c0bafd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4838005510 tx=0x7f48380110c0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 shutdown_connections 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c86c116570 0x55c86c118980 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c86c0af2f0 0x55c86c0bb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c86c0adbe0 0x55c86c0be4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 --2- 172.21.15.31:0/1428246857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c86c0ae530 0x55c86c0bafd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f4826ff5700 0 client.27844 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:22:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 >> 172.21.15.31:0/1428246857 conn(0x55c86c0a33e0 msgr2=0x55c86c0b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 shutdown_connections 2023-07-29T16:22:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.568+0000 7f485d0d4b80 1 -- 172.21.15.31:0/1428246857 wait complete. 2023-07-29T16:22:58.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f485d0d4b80 20 client.27844 trim_cache size 0 max 16384 2023-07-29T16:22:58.577 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:22:58.577 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:22:58.577 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:22:58.577 INFO:teuthology.orchestra.run.smithi031.stdout: "67757a35-6d27-47c3-99c3-f6c8f2d9930f": { 2023-07-29T16:22:58.577 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:22:58.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "67757a35-6d27-47c3-99c3-f6c8f2d9930f", 2023-07-29T16:22:58.578 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:22:58.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:22:58.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:22:58.578 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f482c0a0810 msgr2=0x7f482c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f482c0a0810 0x7f482c0a2cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f483c009ec0 tx=0x7f483c013480 comp rx=0 tx=0).stop 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 msgr2=0x7f48441982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f48441982d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4834004730 tx=0x7f4834004760 comp rx=0 tx=0).stop 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 shutdown_connections 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f482c0a0810 0x7f482c0a2cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48441040d0 0x7f484419f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844102550 0x7f48441982d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 --2- 172.21.15.31:0/2252157106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4844103740 0x7f4844198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:22:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 >> 172.21.15.31:0/2252157106 conn(0x7f48440fdb90 msgr2=0x7f48440ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:22:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 shutdown_connections 2023-07-29T16:22:58.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:22:58.569+0000 7f48497ff700 1 -- 172.21.15.31:0/2252157106 wait complete. 2023-07-29T16:22:58.586 INFO:tasks.cephfs.filesystem:scrub status for tag:67757a35-6d27-47c3-99c3-f6c8f2d9930f - {'path': '/', 'tag': '67757a35-6d27-47c3-99c3-f6c8f2d9930f', 'options': 'recursive,force'} 2023-07-29T16:22:58.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 474315 9.05 MB/sec execute 163 sec latency 53.885 ms 2023-07-29T16:22:59.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:22:59.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:22:59.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:22:59.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:22:59.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:22:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:22:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:22:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:22:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:22:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:22:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:22:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:22:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:22:59.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 476442 9.07 MB/sec execute 164 sec latency 18.874 ms 2023-07-29T16:23:00.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:22:59 smithi031 ceph-mon[95504]: pgmap v596: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 2.8 MiB/s wr, 314 op/s 2023-07-29T16:23:00.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:22:59 smithi039 ceph-mon[128858]: pgmap v596: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 2.8 MiB/s wr, 314 op/s 2023-07-29T16:23:00.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:22:59 smithi156 ceph-mon[129245]: pgmap v596: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 2.8 MiB/s wr, 314 op/s 2023-07-29T16:23:00.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 478181 9.05 MB/sec execute 165 sec latency 67.517 ms 2023-07-29T16:23:01.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 480061 9.08 MB/sec execute 166 sec latency 13.085 ms 2023-07-29T16:23:02.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:02 smithi039 ceph-mon[128858]: pgmap v597: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:23:02.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:02 smithi156 ceph-mon[129245]: pgmap v597: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:23:02.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 482046 9.07 MB/sec execute 167 sec latency 45.642 ms 2023-07-29T16:23:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:02 smithi031 ceph-mon[95504]: pgmap v597: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 2.0 MiB/s wr, 233 op/s 2023-07-29T16:23:03.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 483582 9.09 MB/sec execute 168 sec latency 51.611 ms 2023-07-29T16:23:04.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:04 smithi039 ceph-mon[128858]: pgmap v598: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 2.6 MiB/s wr, 319 op/s 2023-07-29T16:23:04.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:04 smithi156 ceph-mon[129245]: pgmap v598: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 2.6 MiB/s wr, 319 op/s 2023-07-29T16:23:04.875 INFO:tasks.workunit.client.0.smithi031.stdout: 1 485642 9.07 MB/sec execute 169 sec latency 12.688 ms 2023-07-29T16:23:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:04 smithi031 ceph-mon[95504]: pgmap v598: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 2.6 MiB/s wr, 319 op/s 2023-07-29T16:23:05.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 487209 9.09 MB/sec execute 170 sec latency 13.602 ms 2023-07-29T16:23:06.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:06 smithi156 ceph-mon[129245]: pgmap v599: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.2 MiB/s wr, 269 op/s 2023-07-29T16:23:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:06 smithi039 ceph-mon[128858]: pgmap v599: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.2 MiB/s wr, 269 op/s 2023-07-29T16:23:06.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 489520 9.10 MB/sec execute 171 sec latency 15.453 ms 2023-07-29T16:23:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:06 smithi031 ceph-mon[95504]: pgmap v599: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 2.2 MiB/s wr, 269 op/s 2023-07-29T16:23:07.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 490910 9.10 MB/sec execute 172 sec latency 17.943 ms 2023-07-29T16:23:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:07 smithi031 ceph-mon[95504]: pgmap v600: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 2.0 MiB/s wr, 248 op/s 2023-07-29T16:23:08.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:07 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:23:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:07 smithi039 ceph-mon[128858]: pgmap v600: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 2.0 MiB/s wr, 248 op/s 2023-07-29T16:23:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:07 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:23:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:07 smithi156 ceph-mon[129245]: pgmap v600: 97 pgs: 97 active+clean; 87 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 2.0 MiB/s wr, 248 op/s 2023-07-29T16:23:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:07 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:23:08.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 493182 9.11 MB/sec execute 173 sec latency 45.980 ms 2023-07-29T16:23:09.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:08 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:23:09.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:08 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:23:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 -- 172.21.15.31:0/1126585109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 msgr2=0x7fa660101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 --2- 172.21.15.31:0/1126585109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 0x7fa660101980 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098f0 tx=0x7fa650009c00 comp rx=0 tx=0).stop 2023-07-29T16:23:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 -- 172.21.15.31:0/1126585109 shutdown_connections 2023-07-29T16:23:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 --2- 172.21.15.31:0/1126585109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660101ec0 0x7fa660108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 --2- 172.21.15.31:0/1126585109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660100340 0x7fa660100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 --2- 172.21.15.31:0/1126585109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 0x7fa660101980 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 -- 172.21.15.31:0/1126585109 >> 172.21.15.31:0/1126585109 conn(0x7fa6600fb920 msgr2=0x7fa6600fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 -- 172.21.15.31:0/1126585109 shutdown_connections 2023-07-29T16:23:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.190+0000 7fa667fff700 1 -- 172.21.15.31:0/1126585109 wait complete. 2023-07-29T16:23:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.191+0000 7fa667fff700 1 Processor -- start 2023-07-29T16:23:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.191+0000 7fa667fff700 1 -- start start 2023-07-29T16:23:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa667fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660100340 0x7fa660073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa667fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 0x7fa660073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa667fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 0x7fa660073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa667fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601a01b0 con 0x7fa660101530 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa667fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601a02f0 con 0x7fa660101ec0 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa667fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6601a0430 con 0x7fa660100340 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 0x7fa660073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa665d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660100340 0x7fa660073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 0x7fa660073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa665d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660100340 0x7fa660073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49480/0 (socket says 172.21.15.31:49480) 2023-07-29T16:23:09.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 0x7fa660073bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41040/0 (socket says 172.21.15.31:41040) 2023-07-29T16:23:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa665d9b700 1 -- 172.21.15.31:0/3720385929 learned_addr learned my addr 172.21.15.31:0/3720385929 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 -- 172.21.15.31:0/3720385929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660100340 msgr2=0x7fa660073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660100340 0x7fa660073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 -- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 msgr2=0x7fa660073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 0x7fa660073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.192+0000 7fa66659c700 1 -- 172.21.15.31:0/3720385929 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa650009520 con 0x7fa660101ec0 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.193+0000 7fa66659c700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 0x7fa660073bc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa658007f10 tx=0x7fa65800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.193+0000 7fa656ffd700 1 -- 172.21.15.31:0/3720385929 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658004db0 con 0x7fa660101ec0 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.193+0000 7fa656ffd700 1 -- 172.21.15.31:0/3720385929 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa65800f040 con 0x7fa660101ec0 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.193+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6601a06c0 con 0x7fa660101ec0 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.193+0000 7fa656ffd700 1 -- 172.21.15.31:0/3720385929 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658005990 con 0x7fa660101ec0 2023-07-29T16:23:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.193+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6601a0ba0 con 0x7fa660101ec0 2023-07-29T16:23:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.196+0000 7fa656ffd700 1 -- 172.21.15.31:0/3720385929 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa65800db20 con 0x7fa660101ec0 2023-07-29T16:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa656ffd700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa64c0a0790 0x7fa64c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa67b90fb80 1 Processor -- start 2023-07-29T16:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa656ffd700 1 -- 172.21.15.31:0/3720385929 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6580cb370 con 0x7fa660101ec0 2023-07-29T16:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa665d9b700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa64c0a0790 0x7fa64c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa67b90fb80 1 -- start start 2023-07-29T16:23:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c4d26b0 0x55f14c6768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 0x55f14c6777b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.198+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c678160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c5ce9b0 con 0x55f14c4d26b0 2023-07-29T16:23:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c5cc370 con 0x55f14c677cf0 2023-07-29T16:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c5c7c60 con 0x55f14c676df0 2023-07-29T16:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 0x55f14c6777b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 0x55f14c6777b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49482/0 (socket says 172.21.15.31:49482) 2023-07-29T16:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 -- 172.21.15.31:0/831529832 learned_addr learned my addr 172.21.15.31:0/831529832 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66659c700 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c678160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 -- 172.21.15.31:0/831529832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 msgr2=0x55f14c678160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c678160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 -- 172.21.15.31:0/831529832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c4d26b0 msgr2=0x55f14c6768b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c4d26b0 0x55f14c6768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 -- 172.21.15.31:0/831529832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f14c3f5d80 con 0x55f14c676df0 2023-07-29T16:23:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa665d9b700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa64c0a0790 0x7fa64c0a2c40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa660074a50 tx=0x7fa65c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.199+0000 7fa66559a700 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 0x55f14c6777b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098c0 tx=0x7fa650005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c0085571c6312a4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa64bfff700 1 -- 172.21.15.31:0/831529832 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65000fe70 con 0x55f14c676df0 2023-07-29T16:23:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa64bfff700 1 -- 172.21.15.31:0/831529832 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa65001d040 con 0x55f14c676df0 2023-07-29T16:23:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa64bfff700 1 -- 172.21.15.31:0/831529832 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6500052a0 con 0x55f14c676df0 2023-07-29T16:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 -- 172.21.15.31:0/831529832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 msgr2=0x55f14c6777b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 0x55f14c6777b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa6500098c0 tx=0x7fa650005dc0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 -- 172.21.15.31:0/831529832 shutdown_connections 2023-07-29T16:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c678160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c4d26b0 0x55f14c6768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/831529832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c676df0 0x55f14c6777b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.200+0000 7fa67b90fb80 1 -- 172.21.15.31:0/831529832 >> 172.21.15.31:0/831529832 conn(0x55f14c66bfd0 msgr2=0x55f14c671300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.201+0000 7fa67b90fb80 1 -- 172.21.15.31:0/831529832 shutdown_connections 2023-07-29T16:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.201+0000 7fa67b90fb80 1 -- 172.21.15.31:0/831529832 wait complete. 2023-07-29T16:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.201+0000 7fa67b90fb80 1 -- start start 2023-07-29T16:23:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.201+0000 7fa67b90fb80 1 Processor -- start 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c67fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 0x55f14c680360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c677cf0 0x55f14c6873a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c5644b0 con 0x55f14c677cf0 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c564150 con 0x55f14c676df0 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c44b8c0 con 0x55f14c4d26b0 2023-07-29T16:23:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa67b90fb80 10 client.0 _ll_register_callbacks cb 0x55f14c56d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 0x55f14c680360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 0x55f14c680360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41070/0 (socket says 172.21.15.31:41070) 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 -- 172.21.15.31:0/3700644530 learned_addr learned my addr 172.21.15.31:0/3700644530 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66659c700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c677cf0 0x55f14c6873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa665d9b700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c67fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 -- 172.21.15.31:0/3700644530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 msgr2=0x55f14c67fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c67fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 -- 172.21.15.31:0/3700644530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c677cf0 msgr2=0x55f14c6873a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c677cf0 0x55f14c6873a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66559a700 1 -- 172.21.15.31:0/3700644530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6601090b0 con 0x55f14c676df0 2023-07-29T16:23:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.202+0000 7fa66659c700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c677cf0 0x55f14c6873a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.203+0000 7fa6497fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.203+0000 7fa66559a700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 0x55f14c680360 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa650005e90 tx=0x7fa65000f7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b91aa17ea25976c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.203+0000 7fa6497fa700 1 -- 172.21.15.31:0/3700644530 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa650005540 con 0x55f14c676df0 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.203+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f14c5f2cb0 con 0x55f14c676df0 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.203+0000 7fa6497fa700 1 -- 172.21.15.31:0/3700644530 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa6500056a0 con 0x55f14c676df0 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.203+0000 7fa6497fa700 1 -- 172.21.15.31:0/3700644530 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa650003680 con 0x55f14c676df0 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 10 client.27845 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f14c3dec00 con 0x55f14c676df0 2023-07-29T16:23:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa6497fa700 1 -- 172.21.15.31:0/3700644530 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa65003a020 con 0x55f14c676df0 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa6497fa700 1 -- 172.21.15.31:0/3700644530 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa650009c70 con 0x55f14c676df0 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 10 client.27845 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 10 client.27845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 20 client.27845 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c68dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 4 client.27845 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f14c645d10 con 0x55f14c68b6d0 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.204+0000 7fa665d9b700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c68dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.205+0000 7fa665d9b700 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c68dae0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd557c34f947d5b8 server_cookie=383319fce512b714 in_seq=0 out_seq=0 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.205+0000 7fa6497fa700 10 client.27845 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.205+0000 7fa6497fa700 1 -- 172.21.15.31:0/3700644530 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f14c645d10 con 0x55f14c68b6d0 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.205+0000 7fa6497fa700 10 client.27845 handle_command_reply: tid=0 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 client.27845 shutdown 2023-07-29T16:23:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 msgr2=0x55f14c680360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 0x55f14c680360 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fa650005e90 tx=0x7fa65000f7c0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 shutdown_connections 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c68dae0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c677cf0 0x55f14c6873a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa6497fa700 0 client.27845 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c67fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/3700644530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c676df0 0x55f14c680360 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 >> 172.21.15.31:0/3700644530 conn(0x55f14c66bfd0 msgr2=0x55f14c670a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:09.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 shutdown_connections 2023-07-29T16:23:09.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 1 -- 172.21.15.31:0/3700644530 wait complete. 2023-07-29T16:23:09.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.206+0000 7fa67b90fb80 20 client.27845 trim_cache size 0 max 16384 2023-07-29T16:23:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 Processor -- start 2023-07-29T16:23:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 -- start start 2023-07-29T16:23:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c692440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c687360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c6780c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c680ac0 con 0x55f14c676df0 2023-07-29T16:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c67de00 con 0x55f14c677cf0 2023-07-29T16:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.223+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c680f70 con 0x55f14c4d26b0 2023-07-29T16:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c692440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c692440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49498/0 (socket says 172.21.15.31:49498) 2023-07-29T16:23:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 -- 172.21.15.31:0/1855176953 learned_addr learned my addr 172.21.15.31:0/1855176953 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa66659c700 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c6780c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa66559a700 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c687360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 -- 172.21.15.31:0/1855176953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 msgr2=0x55f14c6780c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c6780c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 -- 172.21.15.31:0/1855176953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 msgr2=0x55f14c687360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c687360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 -- 172.21.15.31:0/1855176953 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f14c41c1f0 con 0x55f14c4d26b0 2023-07-29T16:23:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.224+0000 7fa665d9b700 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c692440 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009550 tx=0x7fa65c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=455c6d51d8a99bcd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.225+0000 7fa654ff9700 1 -- 172.21.15.31:0/1855176953 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c023070 con 0x55f14c4d26b0 2023-07-29T16:23:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.225+0000 7fa654ff9700 1 -- 172.21.15.31:0/1855176953 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa65c024040 con 0x55f14c4d26b0 2023-07-29T16:23:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.225+0000 7fa654ff9700 1 -- 172.21.15.31:0/1855176953 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c025df0 con 0x55f14c4d26b0 2023-07-29T16:23:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.225+0000 7fa67b90fb80 1 -- 172.21.15.31:0/1855176953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 msgr2=0x55f14c692440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.225+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c692440 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009550 tx=0x7fa65c014000 comp rx=0 tx=0).stop 2023-07-29T16:23:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 -- 172.21.15.31:0/1855176953 shutdown_connections 2023-07-29T16:23:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c6780c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c692440 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/1855176953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c687360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 -- 172.21.15.31:0/1855176953 >> 172.21.15.31:0/1855176953 conn(0x55f14c66bfd0 msgr2=0x55f14c670a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:09.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 -- 172.21.15.31:0/1855176953 shutdown_connections 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 -- 172.21.15.31:0/1855176953 wait complete. 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 -- start start 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.226+0000 7fa67b90fb80 1 Processor -- start 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c683580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c683ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c686ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c687020 con 0x55f14c676df0 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c687190 con 0x55f14c677cf0 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14c687350 con 0x55f14c4d26b0 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa67b90fb80 10 client.0 _ll_register_callbacks cb 0x55f14c56d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c683580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa66559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c683ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c683580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49510/0 (socket says 172.21.15.31:49510) 2023-07-29T16:23:09.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 -- 172.21.15.31:0/4270714635 learned_addr learned my addr 172.21.15.31:0/4270714635 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:09.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa66659c700 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c686ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 -- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 msgr2=0x55f14c686ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c686ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 -- 172.21.15.31:0/4270714635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 msgr2=0x55f14c683ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:09.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c683ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.227+0000 7fa665d9b700 1 -- 172.21.15.31:0/4270714635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c017070 con 0x55f14c4d26b0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa665d9b700 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c683580 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0095a0 tx=0x7fa65c0252e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26737308313d030d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa6497fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa6497fa700 1 -- 172.21.15.31:0/4270714635 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c023040 con 0x55f14c4d26b0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa65001d040 con 0x55f14c4d26b0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa6497fa700 1 -- 172.21.15.31:0/4270714635 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa65c025d70 con 0x55f14c4d26b0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa6497fa700 1 -- 172.21.15.31:0/4270714635 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa65c003b10 con 0x55f14c4d26b0 2023-07-29T16:23:09.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa67b90fb80 10 client.27862 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.228+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f14c3f4250 con 0x55f14c4d26b0 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa6497fa700 1 -- 172.21.15.31:0/4270714635 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa65c043080 con 0x55f14c4d26b0 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa6497fa700 1 -- 172.21.15.31:0/4270714635 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa65c02b020 con 0x55f14c4d26b0 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa67b90fb80 10 client.27862 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa67b90fb80 10 client.27862 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa67b90fb80 20 client.27862 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:09.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c6e0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:09.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa67b90fb80 4 client.27862 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:23:09.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f14c5c0ce0 con 0x55f14c68b6d0 2023-07-29T16:23:09.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa66559a700 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c6e0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:09.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa66559a700 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c6e0520 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64b5e3b230199700 server_cookie=b7d0a8ba68aaff26 in_seq=0 out_seq=0 2023-07-29T16:23:09.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.229+0000 7fa6497fa700 10 client.27862 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:09.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:08 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:23:09.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:08 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:23:09.292 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:08 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:23:09.292 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:08 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:23:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.304+0000 7fa6497fa700 1 -- 172.21.15.31:0/4270714635 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f14c5c0ce0 con 0x55f14c68b6d0 2023-07-29T16:23:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.304+0000 7fa6497fa700 10 client.27862 handle_command_reply: tid=0 2023-07-29T16:23:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.304+0000 7fa67b90fb80 1 client.27862 shutdown 2023-07-29T16:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 msgr2=0x55f14c683580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c683580 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa65c0095a0 tx=0x7fa65c0252e0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 shutdown_connections 2023-07-29T16:23:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f14c68b6d0 0x55f14c6e0520 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f14c677cf0 0x55f14c686ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f14c4d26b0 0x55f14c683580 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 --2- 172.21.15.31:0/4270714635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f14c676df0 0x55f14c683ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 >> 172.21.15.31:0/4270714635 conn(0x55f14c66bfd0 msgr2=0x55f14c67e050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa6497fa700 0 client.27862 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 shutdown_connections 2023-07-29T16:23:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 1 -- 172.21.15.31:0/4270714635 wait complete. 2023-07-29T16:23:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.305+0000 7fa67b90fb80 20 client.27862 trim_cache size 0 max 16384 2023-07-29T16:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:23:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:23:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:23:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:23:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa64c0a0790 msgr2=0x7fa64c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa64c0a0790 0x7fa64c0a2c40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa660074a50 tx=0x7fa65c009310 comp rx=0 tx=0).stop 2023-07-29T16:23:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 msgr2=0x7fa660073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 0x7fa660073bc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa658007f10 tx=0x7fa65800d3b0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 shutdown_connections 2023-07-29T16:23:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa64c0a0790 0x7fa64c0a2c40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa660101ec0 0x7fa660073bc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa660100340 0x7fa660073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 --2- 172.21.15.31:0/3720385929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa660101530 0x7fa660073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 >> 172.21.15.31:0/3720385929 conn(0x7fa6600fb920 msgr2=0x7fa6600fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:09.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 shutdown_connections 2023-07-29T16:23:09.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:09.306+0000 7fa667fff700 1 -- 172.21.15.31:0/3720385929 wait complete. 2023-07-29T16:23:09.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 494386 9.10 MB/sec execute 174 sec latency 57.357 ms 2023-07-29T16:23:10.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:09 smithi031 ceph-mon[95504]: pgmap v601: 97 pgs: 97 active+clean; 90 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.9 MiB/s wr, 318 op/s 2023-07-29T16:23:10.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:09 smithi039 ceph-mon[128858]: pgmap v601: 97 pgs: 97 active+clean; 90 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.9 MiB/s wr, 318 op/s 2023-07-29T16:23:10.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:09 smithi156 ceph-mon[129245]: pgmap v601: 97 pgs: 97 active+clean; 90 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 2.9 MiB/s wr, 318 op/s 2023-07-29T16:23:10.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 496728 9.11 MB/sec execute 175 sec latency 13.055 ms 2023-07-29T16:23:11.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 498132 9.11 MB/sec execute 176 sec latency 11.124 ms 2023-07-29T16:23:12.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:12 smithi156 ceph-mon[129245]: pgmap v602: 97 pgs: 97 active+clean; 90 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 2.0 MiB/s wr, 209 op/s 2023-07-29T16:23:12.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:12 smithi039 ceph-mon[128858]: pgmap v602: 97 pgs: 97 active+clean; 90 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 2.0 MiB/s wr, 209 op/s 2023-07-29T16:23:12.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 500588 9.13 MB/sec execute 177 sec latency 15.044 ms 2023-07-29T16:23:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:12 smithi031 ceph-mon[95504]: pgmap v602: 97 pgs: 97 active+clean; 90 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 2.0 MiB/s wr, 209 op/s 2023-07-29T16:23:13.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 501469 9.11 MB/sec execute 178 sec latency 172.101 ms 2023-07-29T16:23:14.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:14 smithi039 ceph-mon[128858]: pgmap v603: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2023-07-29T16:23:14.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:14 smithi156 ceph-mon[129245]: pgmap v603: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2023-07-29T16:23:14.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 503710 9.12 MB/sec execute 179 sec latency 17.100 ms 2023-07-29T16:23:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:14 smithi031 ceph-mon[95504]: pgmap v603: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2023-07-29T16:23:15.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 505094 9.12 MB/sec execute 180 sec latency 36.976 ms 2023-07-29T16:23:16.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:16 smithi039 ceph-mon[128858]: pgmap v604: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:23:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:16 smithi156 ceph-mon[129245]: pgmap v604: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:23:16.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 507487 9.14 MB/sec execute 181 sec latency 13.667 ms 2023-07-29T16:23:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:16 smithi031 ceph-mon[95504]: pgmap v604: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:23:17.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 508906 9.13 MB/sec execute 182 sec latency 11.603 ms 2023-07-29T16:23:18.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:17 smithi031 ceph-mon[95504]: pgmap v605: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 2.2 MiB/s wr, 261 op/s 2023-07-29T16:23:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:17 smithi039 ceph-mon[128858]: pgmap v605: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 2.2 MiB/s wr, 261 op/s 2023-07-29T16:23:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:17 smithi156 ceph-mon[129245]: pgmap v605: 97 pgs: 97 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 2.2 MiB/s wr, 261 op/s 2023-07-29T16:23:18.876 INFO:tasks.workunit.client.0.smithi031.stdout: 1 511015 9.14 MB/sec execute 183 sec latency 64.529 ms 2023-07-29T16:23:19.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 511841 9.13 MB/sec execute 184 sec latency 436.518 ms 2023-07-29T16:23:20.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:20 smithi039 ceph-mon[128858]: pgmap v606: 97 pgs: 97 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 3.0 MiB/s wr, 332 op/s 2023-07-29T16:23:20.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:20 smithi156 ceph-mon[129245]: pgmap v606: 97 pgs: 97 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 3.0 MiB/s wr, 332 op/s 2023-07-29T16:23:20.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 511841 9.08 MB/sec execute 185 sec latency 1436.638 ms 2023-07-29T16:23:20.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:20 smithi031 ceph-mon[95504]: pgmap v606: 97 pgs: 97 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 3.0 MiB/s wr, 332 op/s 2023-07-29T16:23:21.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 511841 9.03 MB/sec execute 186 sec latency 2436.760 ms 2023-07-29T16:23:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 -- 172.21.15.31:0/3623982971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 msgr2=0x7fb2f0100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/3623982971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f0100760 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0009990 tx=0x7fb2e0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 -- 172.21.15.31:0/3623982971 shutdown_connections 2023-07-29T16:23:22.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/3623982971 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 0x7fb2f0108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/3623982971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f0100760 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/3623982971 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 0x7fb2f0101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 -- 172.21.15.31:0/3623982971 >> 172.21.15.31:0/3623982971 conn(0x7fb2f00fb970 msgr2=0x7fb2f00fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 -- 172.21.15.31:0/3623982971 shutdown_connections 2023-07-29T16:23:22.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.474+0000 7fb2f63ff700 1 -- 172.21.15.31:0/3623982971 wait complete. 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.475+0000 7fb2f63ff700 1 Processor -- start 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.475+0000 7fb2f63ff700 1 -- start start 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f63ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f01983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f63ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 0x7fb2f0198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f63ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 0x7fb2f0199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f63ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0199de0 con 0x7fb2f0101ed0 2023-07-29T16:23:22.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f63ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0199f50 con 0x7fb2f0100350 2023-07-29T16:23:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f63ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f019a110 con 0x7fb2f0101540 2023-07-29T16:23:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f01983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f499c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 0x7fb2f0199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f01983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49196/0 (socket says 172.21.15.31:49196) 2023-07-29T16:23:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 -- 172.21.15.31:0/354233424 learned_addr learned my addr 172.21.15.31:0/354233424 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 0x7fb2f0198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 -- 172.21.15.31:0/354233424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 msgr2=0x7fb2f0198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 0x7fb2f0198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 -- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 msgr2=0x7fb2f0199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 0x7fb2f0199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2effff700 1 -- 172.21.15.31:0/354233424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00095f0 con 0x7fb2f0100350 2023-07-29T16:23:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 0x7fb2f0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.476+0000 7fb2f499c700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 0x7fb2f0199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.477+0000 7fb2effff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f01983c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0004730 tx=0x7fb2e0004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.477+0000 7fb2ed7fa700 1 -- 172.21.15.31:0/354233424 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e001d070 con 0x7fb2f0100350 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.477+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f019a390 con 0x7fb2f0100350 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.477+0000 7fb2ed7fa700 1 -- 172.21.15.31:0/354233424 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2e0021410 con 0x7fb2f0100350 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.477+0000 7fb2ed7fa700 1 -- 172.21.15.31:0/354233424 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0017650 con 0x7fb2f0100350 2023-07-29T16:23:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.477+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f01a0bb0 con 0x7fb2f0100350 2023-07-29T16:23:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.481+0000 7fb2ed7fa700 1 -- 172.21.15.31:0/354233424 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb2e0021580 con 0x7fb2f0100350 2023-07-29T16:23:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 Processor -- start 2023-07-29T16:23:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 -- start start 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb2ed7fa700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2d80a0790 0x7fb2d80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579ce988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 0x55579ce997b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cdf09b0 con 0x55579ce99cf0 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cdee370 con 0x55579ccf46b0 2023-07-29T16:23:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cde9c60 con 0x55579ce98df0 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.482+0000 7fb2ed7fa700 1 -- 172.21.15.31:0/354233424 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb2e00cced0 con 0x7fb2f0100350 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 0x55579ce997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2f499c700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2d80a0790 0x7fb2d80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 0x55579ce997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40392/0 (socket says 172.21.15.31:40392) 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/4259611452 learned_addr learned my addr 172.21.15.31:0/4259611452 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2effff700 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579ce988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2f499c700 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/4259611452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 msgr2=0x55579ce988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579ce988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/4259611452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 msgr2=0x55579ce9a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/4259611452 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55579cc17d80 con 0x55579ce98df0 2023-07-29T16:23:22.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.483+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 0x55579ce997b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40115f0 tx=0x7fb2e4011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79aaca0d90c5158 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.484+0000 7fb2f499c700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2d80a0790 0x7fb2d80a2c40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x55579ce98610 tx=0x7fb2e8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.484+0000 7fb2d67fc700 1 -- 172.21.15.31:0/4259611452 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4019630 con 0x55579ce98df0 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.484+0000 7fb2d67fc700 1 -- 172.21.15.31:0/4259611452 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2e4019c50 con 0x55579ce98df0 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.484+0000 7fb2d67fc700 1 -- 172.21.15.31:0/4259611452 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e400ec20 con 0x55579ce98df0 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.484+0000 7fb309d9fb80 1 -- 172.21.15.31:0/4259611452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 msgr2=0x55579ce997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.484+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 0x55579ce997b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb2e40115f0 tx=0x7fb2e4011900 comp rx=0 tx=0).stop 2023-07-29T16:23:22.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 -- 172.21.15.31:0/4259611452 shutdown_connections 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579ce988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/4259611452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce98df0 0x55579ce997b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 -- 172.21.15.31:0/4259611452 >> 172.21.15.31:0/4259611452 conn(0x55579ce8dfd0 msgr2=0x55579ce93300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 -- 172.21.15.31:0/4259611452 shutdown_connections 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 -- 172.21.15.31:0/4259611452 wait complete. 2023-07-29T16:23:22.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 -- start start 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.485+0000 7fb309d9fb80 1 Processor -- start 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579cea1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cea23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce99cf0 0x55579cea9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cd864b0 con 0x55579ce98df0 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cd86150 con 0x55579ce99cf0 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cc6d8c0 con 0x55579ccf46b0 2023-07-29T16:23:22.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb309d9fb80 10 client.0 _ll_register_callbacks cb 0x55579cd8f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cea23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cea23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43988/0 (socket says 172.21.15.31:43988) 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/2748850976 learned_addr learned my addr 172.21.15.31:0/2748850976 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb2f499c700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce99cf0 0x55579cea9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.486+0000 7fb2effff700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579cea1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/2748850976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 msgr2=0x55579cea1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579cea1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce99cf0 msgr2=0x55579cea9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce99cf0 0x55579cea9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/2748850976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e4009dd0 con 0x55579ce98df0 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cea23f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb2e400efc0 tx=0x7fb2e4019e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114835c9f2663dc5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2cf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/2748850976 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4019840 con 0x55579ce98df0 2023-07-29T16:23:22.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55579ce14cb0 con 0x55579ce98df0 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.487+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/2748850976 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb2e4005ce0 con 0x55579ce98df0 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/2748850976 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2e40199a0 con 0x55579ce98df0 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 10 client.17817 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55579cc00c00 con 0x55579ce98df0 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/2748850976 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55579ce14cb0 con 0x55579ce98df0 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/2748850976 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb2e4004b90 con 0x55579ce98df0 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 10 client.17817 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 10 client.17817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 20 client.17817 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579ceadbe0 0x55579ceafff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 4 client.17817 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55579ce67d10 con 0x55579ceadbe0 2023-07-29T16:23:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.488+0000 7fb2effff700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579ceadbe0 0x55579ceafff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.489+0000 7fb2effff700 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579ceadbe0 0x55579ceafff0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef7584b9dae75387 server_cookie=9c50ca8f07042638 in_seq=0 out_seq=0 2023-07-29T16:23:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.489+0000 7fb2cf7fe700 10 client.17817 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/2748850976 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55579ce67d10 con 0x55579ceadbe0 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb2cf7fe700 10 client.17817 handle_command_reply: tid=0 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 client.17817 shutdown 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 msgr2=0x55579cea23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cea23f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb2e400efc0 tx=0x7fb2e4019e60 comp rx=0 tx=0).stop 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 shutdown_connections 2023-07-29T16:23:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579ceadbe0 0x55579ceafff0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce99cf0 0x55579cea9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579cea1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/2748850976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cea23f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb2cf7fe700 0 client.17817 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 >> 172.21.15.31:0/2748850976 conn(0x55579ce8dfd0 msgr2=0x55579ce92a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 shutdown_connections 2023-07-29T16:23:22.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 1 -- 172.21.15.31:0/2748850976 wait complete. 2023-07-29T16:23:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.490+0000 7fb309d9fb80 20 client.17817 trim_cache size 0 max 16384 2023-07-29T16:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.505+0000 7fb309d9fb80 1 Processor -- start 2023-07-29T16:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 -- start start 2023-07-29T16:23:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579ceaffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 0x55579cea73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579ce9a600 con 0x55579ce99cf0 2023-07-29T16:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579ce943a0 con 0x55579ce98df0 2023-07-29T16:23:22.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cea2b30 con 0x55579ccf46b0 2023-07-29T16:23:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 0x55579cea73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb2f499c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 0x55579cea73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49234/0 (socket says 172.21.15.31:49234) 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb2effff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579ceaffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.506+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/685793939 learned_addr learned my addr 172.21.15.31:0/685793939 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/685793939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 msgr2=0x55579ceaffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579ceaffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/685793939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 msgr2=0x55579ce9a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2ef7fe700 1 -- 172.21.15.31:0/685793939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55579cc3e1f0 con 0x55579ce98df0 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 0x55579cea73e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4030040 tx=0x7fb2e4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac02fe3b40ff169 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2d6ffd700 1 -- 172.21.15.31:0/685793939 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4005ce0 con 0x55579ce98df0 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2d6ffd700 1 -- 172.21.15.31:0/685793939 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2e4023620 con 0x55579ce98df0 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.507+0000 7fb2d6ffd700 1 -- 172.21.15.31:0/685793939 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4023da0 con 0x55579ce98df0 2023-07-29T16:23:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 -- 172.21.15.31:0/685793939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 msgr2=0x55579cea73e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 0x55579cea73e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4030040 tx=0x7fb2e4000bc0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 -- 172.21.15.31:0/685793939 shutdown_connections 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce99cf0 0x55579ce9a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ccf46b0 0x55579ceaffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/685793939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ce98df0 0x55579cea73e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 -- 172.21.15.31:0/685793939 >> 172.21.15.31:0/685793939 conn(0x55579ce8dfd0 msgr2=0x55579ce92a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 -- 172.21.15.31:0/685793939 shutdown_connections 2023-07-29T16:23:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 -- 172.21.15.31:0/685793939 wait complete. 2023-07-29T16:23:22.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.508+0000 7fb309d9fb80 1 -- start start 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 Processor -- start 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579cef9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cefbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce99cf0 0x55579cefe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cea7820 con 0x55579ce98df0 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cefe740 con 0x55579ccf46b0 2023-07-29T16:23:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55579cefe900 con 0x55579ce99cf0 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb309d9fb80 10 client.0 _ll_register_callbacks cb 0x55579cd8f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb2effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579cef9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb2effff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579cef9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49240/0 (socket says 172.21.15.31:49240) 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb2effff700 1 -- 172.21.15.31:0/890404224 learned_addr learned my addr 172.21.15.31:0/890404224 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.509+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cefbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2f499c700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce99cf0 0x55579cefe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2effff700 1 -- 172.21.15.31:0/890404224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce99cf0 msgr2=0x55579cefe1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2effff700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce99cf0 0x55579cefe1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2effff700 1 -- 172.21.15.31:0/890404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 msgr2=0x55579cefbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2effff700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cefbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2effff700 1 -- 172.21.15.31:0/890404224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55579cc00c00 con 0x55579ccf46b0 2023-07-29T16:23:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2effff700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579cef9730 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0096b00 tx=0x7fb2e0009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8914d4d3492d202 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2cf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/890404224 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00bf2e0 con 0x55579ccf46b0 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/890404224 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb2e00a29c0 con 0x55579ccf46b0 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2e4019c50 con 0x55579ccf46b0 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/890404224 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2e00c4a50 con 0x55579ccf46b0 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb309d9fb80 10 client.27869 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.510+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55579cc16250 con 0x55579ccf46b0 2023-07-29T16:23:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/890404224 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2e00c8020 con 0x55579ccf46b0 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/890404224 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb2e0099c10 con 0x55579ccf46b0 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb309d9fb80 10 client.27869 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb309d9fb80 10 client.27869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb309d9fb80 20 client.27869 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579cf000c0 0x55579cf024a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb309d9fb80 4 client.27869 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55579cde2ce0 con 0x55579cf000c0 2023-07-29T16:23:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.511+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579cf000c0 0x55579cf024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.512+0000 7fb2ef7fe700 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579cf000c0 0x55579cf024a0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35a554111bcfba3d server_cookie=3d6c8ab2cc5a730c in_seq=0 out_seq=0 2023-07-29T16:23:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.512+0000 7fb2cf7fe700 10 client.27869 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.586+0000 7fb2cf7fe700 1 -- 172.21.15.31:0/890404224 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55579cde2ce0 con 0x55579cf000c0 2023-07-29T16:23:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.586+0000 7fb2cf7fe700 10 client.27869 handle_command_reply: tid=0 2023-07-29T16:23:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 client.27869 shutdown 2023-07-29T16:23:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 msgr2=0x55579cef9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579cef9730 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0096b00 tx=0x7fb2e0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 shutdown_connections 2023-07-29T16:23:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55579cf000c0 0x55579cf024a0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55579ce99cf0 0x55579cefe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55579ccf46b0 0x55579cef9730 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb2cf7fe700 0 client.27869 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 --2- 172.21.15.31:0/890404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55579ce98df0 0x55579cefbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 >> 172.21.15.31:0/890404224 conn(0x55579ce8dfd0 msgr2=0x55579cea1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 shutdown_connections 2023-07-29T16:23:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 1 -- 172.21.15.31:0/890404224 wait complete. 2023-07-29T16:23:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.587+0000 7fb309d9fb80 20 client.27869 trim_cache size 0 max 16384 2023-07-29T16:23:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:23:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:23:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:23:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2d80a0790 msgr2=0x7fb2d80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2d80a0790 0x7fb2d80a2c40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x55579ce98610 tx=0x7fb2e8009920 comp rx=0 tx=0).stop 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 msgr2=0x7fb2f01983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f01983c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0004730 tx=0x7fb2e0004760 comp rx=0 tx=0).stop 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 shutdown_connections 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb2d80a0790 0x7fb2d80a2c40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f0101ed0 0x7fb2f0199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2f0100350 0x7fb2f01983c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 --2- 172.21.15.31:0/354233424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2f0101540 0x7fb2f0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 >> 172.21.15.31:0/354233424 conn(0x7fb2f00fb970 msgr2=0x7fb2f0106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 shutdown_connections 2023-07-29T16:23:22.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:22.588+0000 7fb2f63ff700 1 -- 172.21.15.31:0/354233424 wait complete. 2023-07-29T16:23:22.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:22 smithi039 ceph-mon[128858]: pgmap v607: 97 pgs: 97 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 2.0 MiB/s wr, 240 op/s 2023-07-29T16:23:22.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:22 smithi156 ceph-mon[129245]: pgmap v607: 97 pgs: 97 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 2.0 MiB/s wr, 240 op/s 2023-07-29T16:23:22.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 511841 8.98 MB/sec execute 187 sec latency 3436.885 ms 2023-07-29T16:23:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:22 smithi031 ceph-mon[95504]: pgmap v607: 97 pgs: 97 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 2.0 MiB/s wr, 240 op/s 2023-07-29T16:23:23.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 511841 8.93 MB/sec execute 188 sec latency 4436.998 ms 2023-07-29T16:23:24.571 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:24 smithi156 ceph-mon[129245]: pgmap v608: 97 pgs: 97 active+clean; 93 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:23:24.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:24 smithi039 ceph-mon[128858]: pgmap v608: 97 pgs: 97 active+clean; 93 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:23:24.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 512136 8.89 MB/sec execute 189 sec latency 5030.543 ms 2023-07-29T16:23:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:24 smithi031 ceph-mon[95504]: pgmap v608: 97 pgs: 97 active+clean; 93 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:23:25.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 514455 8.90 MB/sec execute 190 sec latency 22.193 ms 2023-07-29T16:23:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:26 smithi039 ceph-mon[128858]: pgmap v609: 97 pgs: 97 active+clean; 98 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.6 MiB/s wr, 206 op/s 2023-07-29T16:23:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:26 smithi156 ceph-mon[129245]: pgmap v609: 97 pgs: 97 active+clean; 98 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.6 MiB/s wr, 206 op/s 2023-07-29T16:23:26.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 515744 8.89 MB/sec execute 191 sec latency 18.949 ms 2023-07-29T16:23:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:26 smithi031 ceph-mon[95504]: pgmap v609: 97 pgs: 97 active+clean; 98 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.6 MiB/s wr, 206 op/s 2023-07-29T16:23:27.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 518113 8.91 MB/sec execute 192 sec latency 14.500 ms 2023-07-29T16:23:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:27 smithi031 ceph-mon[95504]: pgmap v610: 97 pgs: 97 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.7 MiB/s wr, 187 op/s 2023-07-29T16:23:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:27 smithi039 ceph-mon[128858]: pgmap v610: 97 pgs: 97 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.7 MiB/s wr, 187 op/s 2023-07-29T16:23:28.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:27 smithi156 ceph-mon[129245]: pgmap v610: 97 pgs: 97 active+clean; 102 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.7 MiB/s wr, 187 op/s 2023-07-29T16:23:28.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:23:28.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.843+0000 7fc1997ff700 1 -- 172.21.15.31:0/1121973691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 msgr2=0x7fc194073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.843+0000 7fc1997ff700 1 --2- 172.21.15.31:0/1121973691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc194073c20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc188009990 tx=0x7fc188009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 -- 172.21.15.31:0/1121973691 shutdown_connections 2023-07-29T16:23:28.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 --2- 172.21.15.31:0/1121973691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 0x7fc19410a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 --2- 172.21.15.31:0/1121973691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc194074d90 0x7fc1940731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 --2- 172.21.15.31:0/1121973691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc194073c20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 -- 172.21.15.31:0/1121973691 >> 172.21.15.31:0/1121973691 conn(0x7fc1940fb920 msgr2=0x7fc1940fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 -- 172.21.15.31:0/1121973691 shutdown_connections 2023-07-29T16:23:28.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.844+0000 7fc1997ff700 1 -- 172.21.15.31:0/1121973691 wait complete. 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 Processor -- start 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 -- start start 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc1941982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc194074d90 0x7fc194198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 0x7fc19419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19419fe60 con 0x7fc1941081e0 2023-07-29T16:23:28.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19419ffa0 con 0x7fc1940737b0 2023-07-29T16:23:28.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc1997ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1941a0130 con 0x7fc194074d90 2023-07-29T16:23:28.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.845+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc1941982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc1941982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49260/0 (socket says 172.21.15.31:49260) 2023-07-29T16:23:28.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 -- 172.21.15.31:0/4183997847 learned_addr learned my addr 172.21.15.31:0/4183997847 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:28.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc1927fc700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc194074d90 0x7fc194198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc1937fe700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 0x7fc19419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 -- 172.21.15.31:0/4183997847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc194074d90 msgr2=0x7fc194198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc194074d90 0x7fc194198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 -- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 msgr2=0x7fc19419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 0x7fc19419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 -- 172.21.15.31:0/4183997847 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1880095f0 con 0x7fc1940737b0 2023-07-29T16:23:28.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc1937fe700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 0x7fc19419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc192ffd700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc1941982e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc18400e970 tx=0x7fc18400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc17bfff700 1 -- 172.21.15.31:0/4183997847 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184009790 con 0x7fc1940737b0 2023-07-29T16:23:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc17bfff700 1 -- 172.21.15.31:0/4183997847 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc18400eec0 con 0x7fc1940737b0 2023-07-29T16:23:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1941a0410 con 0x7fc1940737b0 2023-07-29T16:23:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.846+0000 7fc17bfff700 1 -- 172.21.15.31:0/4183997847 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc184005590 con 0x7fc1940737b0 2023-07-29T16:23:28.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.847+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1941a0920 con 0x7fc1940737b0 2023-07-29T16:23:28.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.850+0000 7fc17bfff700 1 -- 172.21.15.31:0/4183997847 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc18400cbd0 con 0x7fc1940737b0 2023-07-29T16:23:28.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc17bfff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc17c0a0790 0x7fc17c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc17bfff700 1 -- 172.21.15.31:0/4183997847 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc184014070 con 0x7fc1940737b0 2023-07-29T16:23:28.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc1ad0ddb80 1 Processor -- start 2023-07-29T16:23:28.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc1927fc700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc17c0a0790 0x7fc17c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc1ad0ddb80 1 -- start start 2023-07-29T16:23:28.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12321ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.852+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12322530 0x55bb12322d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12323740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb12279a10 con 0x55bb12321be0 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb122773d0 con 0x55bb12322530 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb12272cc0 con 0x55bb123232f0 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12323740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12321ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12323740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40450/0 (socket says 172.21.15.31:40450) 2023-07-29T16:23:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 -- 172.21.15.31:0/344491283 learned_addr learned my addr 172.21.15.31:0/344491283 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1927fc700 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12322530 0x55bb12322d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1927fc700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc17c0a0790 0x7fc17c0a2c40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b2e0 tx=0x7fc188005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:28.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 -- 172.21.15.31:0/344491283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12322530 msgr2=0x55bb12322d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12322530 0x55bb12322d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 -- 172.21.15.31:0/344491283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 msgr2=0x55bb12321ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12321ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc1937fe700 1 -- 172.21.15.31:0/344491283 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb120a0de0 con 0x55bb123232f0 2023-07-29T16:23:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.853+0000 7fc192ffd700 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12321ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:28.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.854+0000 7fc1937fe700 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12323740 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009440 tx=0x7fc18c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d0adc2b73ce4684 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.854+0000 7fc1797fa700 1 -- 172.21.15.31:0/344491283 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00e070 con 0x55bb123232f0 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.854+0000 7fc1797fa700 1 -- 172.21.15.31:0/344491283 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc18c00f040 con 0x55bb123232f0 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.854+0000 7fc1797fa700 1 -- 172.21.15.31:0/344491283 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c015c90 con 0x55bb123232f0 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/344491283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 msgr2=0x55bb12323740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12323740 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc18c009440 tx=0x7fc18c009920 comp rx=0 tx=0).stop 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/344491283 shutdown_connections 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12323740 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12321ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/344491283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12322530 0x55bb12322d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/344491283 >> 172.21.15.31:0/344491283 conn(0x55bb123173e0 msgr2=0x55bb1231c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/344491283 shutdown_connections 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/344491283 wait complete. 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.855+0000 7fc1ad0ddb80 1 -- start start 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 Processor -- start 2023-07-29T16:23:28.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb1233c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb1233cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb12343d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb1220f510 con 0x55bb12321be0 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb1220f1b0 con 0x55bb123232f0 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb120f6920 con 0x55bb12322530 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.856+0000 7fc1ad0ddb80 10 client.0 _ll_register_callbacks cb 0x55bb12218620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:28.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb1233c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb12343d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb12343d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49288/0 (socket says 172.21.15.31:49288) 2023-07-29T16:23:28.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1937fe700 1 -- 172.21.15.31:0/2949321260 learned_addr learned my addr 172.21.15.31:0/2949321260 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:28.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1927fc700 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb1233cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1927fc700 1 -- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 msgr2=0x55bb12343d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1927fc700 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb12343d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1927fc700 1 -- 172.21.15.31:0/2949321260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 msgr2=0x55bb1233c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1927fc700 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb1233c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.857+0000 7fc1927fc700 1 -- 172.21.15.31:0/2949321260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1941a0920 con 0x55bb12322530 2023-07-29T16:23:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc1927fc700 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb1233cd40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc1880098e0 tx=0x7fc18801a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e7c1ce932aced74 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc176ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:28.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc176ff5700 1 -- 172.21.15.31:0/2949321260 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188005220 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc176ff5700 1 -- 172.21.15.31:0/2949321260 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc188005380 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb1229dd10 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc176ff5700 1 -- 172.21.15.31:0/2949321260 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc18800f730 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc1ad0ddb80 10 client.27892 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.858+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fc18c0090f0 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc176ff5700 1 -- 172.21.15.31:0/2949321260 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bb1229dd10 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc176ff5700 1 -- 172.21.15.31:0/2949321260 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc188037070 con 0x55bb12322530 2023-07-29T16:23:28.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc1ad0ddb80 10 client.27892 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc1ad0ddb80 10 client.27892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc1ad0ddb80 20 client.27892 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb123485c0 0x55bb1234a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc1ad0ddb80 4 client.27892 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb122f0d70 con 0x55bb123485c0 2023-07-29T16:23:28.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.859+0000 7fc192ffd700 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb123485c0 0x55bb1234a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.860+0000 7fc192ffd700 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb123485c0 0x55bb1234a9d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8030336bb86dcd7 server_cookie=93480eb6f25e4884 in_seq=0 out_seq=0 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.860+0000 7fc176ff5700 10 client.27892 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc176ff5700 1 -- 172.21.15.31:0/2949321260 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bb122f0d70 con 0x55bb123485c0 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc176ff5700 10 client.27892 handle_command_reply: tid=0 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 client.27892 shutdown 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 msgr2=0x55bb1233cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb1233cd40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc1880098e0 tx=0x7fc18801a050 comp rx=0 tx=0).stop 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 shutdown_connections 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb123485c0 0x55bb1234a9d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb12343d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb1233c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/2949321260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb1233cd40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc176ff5700 0 client.27892 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.861+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 >> 172.21.15.31:0/2949321260 conn(0x55bb123173e0 msgr2=0x55bb1231c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:28.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.862+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 shutdown_connections 2023-07-29T16:23:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.862+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/2949321260 wait complete. 2023-07-29T16:23:28.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.862+0000 7fc1ad0ddb80 20 client.27892 trim_cache size 0 max 16384 2023-07-29T16:23:28.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 519389 8.90 MB/sec execute 193 sec latency 28.510 ms 2023-07-29T16:23:28.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.881+0000 7fc1ad0ddb80 1 Processor -- start 2023-07-29T16:23:28.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.881+0000 7fc1ad0ddb80 1 -- start start 2023-07-29T16:23:28.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12341d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb12329720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb1233c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb1233bcb0 con 0x55bb12321be0 2023-07-29T16:23:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb1233c370 con 0x55bb123232f0 2023-07-29T16:23:28.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb12319460 con 0x55bb12322530 2023-07-29T16:23:28.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12341d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb1233c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb1233c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49292/0 (socket says 172.21.15.31:49292) 2023-07-29T16:23:28.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12341d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44062/0 (socket says 172.21.15.31:44062) 2023-07-29T16:23:28.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1937fe700 1 -- 172.21.15.31:0/3403507184 learned_addr learned my addr 172.21.15.31:0/3403507184 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.882+0000 7fc1927fc700 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb12329720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1937fe700 1 -- 172.21.15.31:0/3403507184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 msgr2=0x55bb12329720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1937fe700 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb12329720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1937fe700 1 -- 172.21.15.31:0/3403507184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 msgr2=0x55bb12341d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1937fe700 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12341d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1937fe700 1 -- 172.21.15.31:0/3403507184 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb12089c60 con 0x55bb123232f0 2023-07-29T16:23:28.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1927fc700 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb12329720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc1937fe700 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb1233c800 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc18c006180 tx=0x7fc18c009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c22cc73f0bbe82 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc179ffb700 1 -- 172.21.15.31:0/3403507184 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c00e040 con 0x55bb123232f0 2023-07-29T16:23:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc179ffb700 1 -- 172.21.15.31:0/3403507184 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc18c003a00 con 0x55bb123232f0 2023-07-29T16:23:28.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.883+0000 7fc179ffb700 1 -- 172.21.15.31:0/3403507184 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18c015ca0 con 0x55bb123232f0 2023-07-29T16:23:28.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/3403507184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 msgr2=0x55bb1233c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb1233c800 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fc18c006180 tx=0x7fc18c009660 comp rx=0 tx=0).stop 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/3403507184 shutdown_connections 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb123232f0 0x55bb1233c800 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12321be0 0x55bb12341d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/3403507184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb12322530 0x55bb12329720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/3403507184 >> 172.21.15.31:0/3403507184 conn(0x55bb123173e0 msgr2=0x55bb1231d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/3403507184 shutdown_connections 2023-07-29T16:23:28.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.884+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/3403507184 wait complete. 2023-07-29T16:23:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.885+0000 7fc1ad0ddb80 1 -- start start 2023-07-29T16:23:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.885+0000 7fc1ad0ddb80 1 Processor -- start 2023-07-29T16:23:28.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 0x55bb12340330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12322530 0x55bb12340870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12340db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb12341fb0 con 0x55bb12322530 2023-07-29T16:23:28.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb1239a140 con 0x55bb12321be0 2023-07-29T16:23:28.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb1239a280 con 0x55bb123232f0 2023-07-29T16:23:28.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1ad0ddb80 10 client.0 _ll_register_callbacks cb 0x55bb12218620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:28.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1927fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12322530 0x55bb12340870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 0x55bb12340330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 0x55bb12340330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49308/0 (socket says 172.21.15.31:49308) 2023-07-29T16:23:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 -- 172.21.15.31:0/54439586 learned_addr learned my addr 172.21.15.31:0/54439586 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:28.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1937fe700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12340db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 -- 172.21.15.31:0/54439586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 msgr2=0x55bb12340db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12340db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 -- 172.21.15.31:0/54439586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12322530 msgr2=0x55bb12340870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12322530 0x55bb12340870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 -- 172.21.15.31:0/54439586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18c0090f0 con 0x55bb12321be0 2023-07-29T16:23:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc1937fe700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12340db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.886+0000 7fc192ffd700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 0x55bb12340330 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc18400c5b0 tx=0x7fc18409c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96624d4245f5c286 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc176ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:28.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc176ff5700 1 -- 172.21.15.31:0/54439586 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1840b72a0 con 0x55bb12321be0 2023-07-29T16:23:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc176ff5700 1 -- 172.21.15.31:0/54439586 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc1840b6040 con 0x55bb12321be0 2023-07-29T16:23:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc18c00f040 con 0x55bb12321be0 2023-07-29T16:23:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc176ff5700 1 -- 172.21.15.31:0/54439586 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1840b7400 con 0x55bb12321be0 2023-07-29T16:23:28.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 10 client.27887 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bb120a0de0 con 0x55bb12321be0 2023-07-29T16:23:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc176ff5700 1 -- 172.21.15.31:0/54439586 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc18c00f040 con 0x55bb12321be0 2023-07-29T16:23:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc176ff5700 1 -- 172.21.15.31:0/54439586 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc1840989f0 con 0x55bb12321be0 2023-07-29T16:23:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 10 client.27887 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:28.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 10 client.27887 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 20 client.27887 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb1239bac0 0x55bb1239ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 4 client.27887 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:23:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.887+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bb1226bd40 con 0x55bb1239bac0 2023-07-29T16:23:28.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1927fc700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb1239bac0 0x55bb1239ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1927fc700 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb1239bac0 0x55bb1239ded0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e2c368778afcc54 server_cookie=2894ad09cf868e2b in_seq=0 out_seq=0 2023-07-29T16:23:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc176ff5700 10 client.27887 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc176ff5700 1 -- 172.21.15.31:0/54439586 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55bb1226bd40 con 0x55bb1239bac0 2023-07-29T16:23:28.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc176ff5700 10 client.27887 handle_command_reply: tid=0 2023-07-29T16:23:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 client.27887 shutdown 2023-07-29T16:23:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 msgr2=0x55bb12340330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 0x55bb12340330 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc18400c5b0 tx=0x7fc18409c040 comp rx=0 tx=0).stop 2023-07-29T16:23:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 shutdown_connections 2023-07-29T16:23:28.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bb1239bac0 0x55bb1239ded0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bb123232f0 0x55bb12340db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb12321be0 0x55bb12340330 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.888+0000 7fc1ad0ddb80 1 --2- 172.21.15.31:0/54439586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bb12322530 0x55bb12340870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.889+0000 7fc176ff5700 0 client.27887 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:28.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.889+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 >> 172.21.15.31:0/54439586 conn(0x55bb123173e0 msgr2=0x55bb12329720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:28.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.889+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 shutdown_connections 2023-07-29T16:23:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.889+0000 7fc1ad0ddb80 1 -- 172.21.15.31:0/54439586 wait complete. 2023-07-29T16:23:28.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.889+0000 7fc1ad0ddb80 20 client.27887 trim_cache size 0 max 16384 2023-07-29T16:23:28.906 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:23:28.906 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:23:28.906 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:23:28.906 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:23:28.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc17c0a0790 msgr2=0x7fc17c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc17c0a0790 0x7fc17c0a2c40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b2e0 tx=0x7fc188005fb0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 msgr2=0x7fc1941982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:28.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc1941982e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc18400e970 tx=0x7fc18400ec80 comp rx=0 tx=0).stop 2023-07-29T16:23:28.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 shutdown_connections 2023-07-29T16:23:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc17c0a0790 0x7fc17c0a2c40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1941081e0 0x7fc19419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1940737b0 0x7fc1941982e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 --2- 172.21.15.31:0/4183997847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc194074d90 0x7fc194198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 >> 172.21.15.31:0/4183997847 conn(0x7fc1940fb920 msgr2=0x7fc1940fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:28.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 shutdown_connections 2023-07-29T16:23:28.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:28.890+0000 7fc1997ff700 1 -- 172.21.15.31:0/4183997847 wait complete. 2023-07-29T16:23:28.910 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:23:28.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:23:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.153+0000 7f1a0bfff700 1 -- 172.21.15.31:0/811060063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 msgr2=0x7f1a0c103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.153+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/811060063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c103080 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009990 tx=0x7f19fc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 -- 172.21.15.31:0/811060063 shutdown_connections 2023-07-29T16:23:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/811060063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/811060063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c103080 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/811060063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a0c1035c0 0x7f1a0c1059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 -- 172.21.15.31:0/811060063 >> 172.21.15.31:0/811060063 conn(0x7f1a0c0fa720 msgr2=0x7f1a0c0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 -- 172.21.15.31:0/811060063 shutdown_connections 2023-07-29T16:23:29.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.154+0000 7f1a0bfff700 1 -- 172.21.15.31:0/811060063 wait complete. 2023-07-29T16:23:29.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.155+0000 7f1a0bfff700 1 Processor -- start 2023-07-29T16:23:29.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.155+0000 7f1a0bfff700 1 -- start start 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a0c1035c0 0x7f1a0c1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c19d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c19d8e0 con 0x7f1a0c1035c0 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c19da50 con 0x7f1a0c105ee0 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0bfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c19dc10 con 0x7f1a0c100ca0 2023-07-29T16:23:29.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c19d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49338/0 (socket says 172.21.15.31:49338) 2023-07-29T16:23:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0b7fe700 1 -- 172.21.15.31:0/4124354416 learned_addr learned my addr 172.21.15.31:0/4124354416 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.156+0000 7f1a0a7fc700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a0c1035c0 0x7f1a0c1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0b7fe700 1 -- 172.21.15.31:0/4124354416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 msgr2=0x7f1a0c195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0b7fe700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0b7fe700 1 -- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a0c1035c0 msgr2=0x7f1a0c1962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0b7fe700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a0c1035c0 0x7f1a0c1962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0b7fe700 1 -- 172.21.15.31:0/4124354416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0095f0 con 0x7f1a0c105ee0 2023-07-29T16:23:29.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0affd700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:29.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0b7fe700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c19d310 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400ef30 tx=0x7f1a0400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04018070 con 0x7f1a0c105ee0 2023-07-29T16:23:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a0400f040 con 0x7f1a0c105ee0 2023-07-29T16:23:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c19de90 con 0x7f1a0c105ee0 2023-07-29T16:23:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0400cd00 con 0x7f1a0c105ee0 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.157+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c19e310 con 0x7f1a0c105ee0 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.158+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c0fc290 con 0x7f1a0c105ee0 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.161+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1a04003a00 con 0x7f1a0c105ee0 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.162+0000 7f1a109fd700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19f40a0900 0x7f19f40a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.162+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1a040cbf00 con 0x7f1a0c105ee0 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.162+0000 7f1a0affd700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19f40a0900 0x7f19f40a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.163+0000 7f1a0affd700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19f40a0900 0x7f19f40a2db0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f19fc000c00 tx=0x7f19fc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.165+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1a04095970 con 0x7f1a0c105ee0 2023-07-29T16:23:29.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.444+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a0c02cc30 con 0x7f1a0c105ee0 2023-07-29T16:23:29.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.444+0000 7f1a109fd700 1 -- 172.21.15.31:0/4124354416 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f1a04098f90 con 0x7f1a0c105ee0 2023-07-29T16:23:29.444 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:23:29.445 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:23:29.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19f40a0900 msgr2=0x7f19f40a2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19f40a0900 0x7f19f40a2db0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f19fc000c00 tx=0x7f19fc00b260 comp rx=0 tx=0).stop 2023-07-29T16:23:29.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 msgr2=0x7f1a0c19d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c19d310 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400ef30 tx=0x7f1a0400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 shutdown_connections 2023-07-29T16:23:29.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f19f40a0900 0x7f19f40a2db0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a0c105ee0 0x7f1a0c19d310 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1a0c100ca0 0x7f1a0c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 --2- 172.21.15.31:0/4124354416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a0c1035c0 0x7f1a0c1962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 >> 172.21.15.31:0/4124354416 conn(0x7f1a0c0fa720 msgr2=0x7f1a0c101900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 shutdown_connections 2023-07-29T16:23:29.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.448+0000 7f1a0bfff700 1 -- 172.21.15.31:0/4124354416 wait complete. 2023-07-29T16:23:29.450 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:23:29.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:23:29.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 -- 172.21.15.31:0/1606732446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198101510 msgr2=0x7f5198101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 --2- 172.21.15.31:0/1606732446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198101510 0x7f5198101960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f518c009990 tx=0x7f518c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 -- 172.21.15.31:0/1606732446 shutdown_connections 2023-07-29T16:23:29.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 --2- 172.21.15.31:0/1606732446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5198101ea0 0x7f5198108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 --2- 172.21.15.31:0/1606732446 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198100320 0x7f5198100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 --2- 172.21.15.31:0/1606732446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198101510 0x7f5198101960 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 -- 172.21.15.31:0/1606732446 >> 172.21.15.31:0/1606732446 conn(0x7f51980fb920 msgr2=0x7f51980fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 -- 172.21.15.31:0/1606732446 shutdown_connections 2023-07-29T16:23:29.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.704+0000 7f519dfff700 1 -- 172.21.15.31:0/1606732446 wait complete. 2023-07-29T16:23:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.705+0000 7f519dfff700 1 Processor -- start 2023-07-29T16:23:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.705+0000 7f519dfff700 1 -- start start 2023-07-29T16:23:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f519dfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 0x7f5198074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f519dfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198101510 0x7f5198073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f519dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5198101ea0 0x7f5198073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f519dfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198073e20 con 0x7f5198101510 2023-07-29T16:23:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f519dfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5198073f90 con 0x7f5198100320 2023-07-29T16:23:29.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f519dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51981a0110 con 0x7f5198101ea0 2023-07-29T16:23:29.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f51977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 0x7f5198074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f51977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 0x7f5198074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49346/0 (socket says 172.21.15.31:49346) 2023-07-29T16:23:29.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f51977fe700 1 -- 172.21.15.31:0/15916933 learned_addr learned my addr 172.21.15.31:0/15916933 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f5196ffd700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198101510 0x7f5198073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.706+0000 7f5197fff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5198101ea0 0x7f5198073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f51977fe700 1 -- 172.21.15.31:0/15916933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5198101ea0 msgr2=0x7f5198073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f51977fe700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5198101ea0 0x7f5198073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f51977fe700 1 -- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198101510 msgr2=0x7f5198073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f51977fe700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198101510 0x7f5198073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f51977fe700 1 -- 172.21.15.31:0/15916933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0095f0 con 0x7f5198100320 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f51977fe700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 0x7f5198074b00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f518800e970 tx=0x7f518800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f5194ff9700 1 -- 172.21.15.31:0/15916933 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188009790 con 0x7f5198100320 2023-07-29T16:23:29.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f5194ff9700 1 -- 172.21.15.31:0/15916933 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f518800c7e0 con 0x7f5198100320 2023-07-29T16:23:29.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f5194ff9700 1 -- 172.21.15.31:0/15916933 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188003c70 con 0x7f5198100320 2023-07-29T16:23:29.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51981a03f0 con 0x7f5198100320 2023-07-29T16:23:29.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.707+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51981a08b0 con 0x7f5198100320 2023-07-29T16:23:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.711+0000 7f5194ff9700 1 -- 172.21.15.31:0/15916933 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f518800cc50 con 0x7f5198100320 2023-07-29T16:23:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.713+0000 7f51b18a7b80 1 Processor -- start 2023-07-29T16:23:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.713+0000 7f51b18a7b80 1 -- start start 2023-07-29T16:23:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.713+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c4eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.713+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c4fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.713+0000 7f5194ff9700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51800a0810 0x7f51800a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.713+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c50740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9ba6a10 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9ba43d0 con 0x55c1c9c4f530 2023-07-29T16:23:29.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9b9fcc0 con 0x55c1c9c502f0 2023-07-29T16:23:29.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5194ff9700 1 -- 172.21.15.31:0/15916933 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5188014070 con 0x7f5198100320 2023-07-29T16:23:29.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f51977fe700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51800a0810 0x7f51800a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c4fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c4fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49360/0 (socket says 172.21.15.31:49360) 2023-07-29T16:23:29.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 -- 172.21.15.31:0/3299096668 learned_addr learned my addr 172.21.15.31:0/3299096668 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5197fff700 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c4eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5197fff700 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c50740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 -- 172.21.15.31:0/3299096668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 msgr2=0x55c1c9c50740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c50740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 -- 172.21.15.31:0/3299096668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 msgr2=0x55c1c9c4eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c4eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5196ffd700 1 -- 172.21.15.31:0/3299096668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1c99cdde0 con 0x55c1c9c4f530 2023-07-29T16:23:29.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f5197fff700 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c4eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:29.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.714+0000 7f51977fe700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51800a0810 0x7f51800a2cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f51880958d0 tx=0x7f5188003710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.715+0000 7f5196ffd700 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c4fd60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f518c015000 tx=0x7f518c005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e567c338690af58 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.715+0000 7f517dffb700 1 -- 172.21.15.31:0/3299096668 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c01d070 con 0x55c1c9c4f530 2023-07-29T16:23:29.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.715+0000 7f517dffb700 1 -- 172.21.15.31:0/3299096668 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f518c01e040 con 0x55c1c9c4f530 2023-07-29T16:23:29.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.715+0000 7f517dffb700 1 -- 172.21.15.31:0/3299096668 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c0049a0 con 0x55c1c9c4f530 2023-07-29T16:23:29.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 -- 172.21.15.31:0/3299096668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 msgr2=0x55c1c9c4fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c4fd60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f518c015000 tx=0x7f518c005ea0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 -- 172.21.15.31:0/3299096668 shutdown_connections 2023-07-29T16:23:29.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c50740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c4eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/3299096668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c4fd60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 -- 172.21.15.31:0/3299096668 >> 172.21.15.31:0/3299096668 conn(0x55c1c9c443e0 msgr2=0x55c1c9c49990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 -- 172.21.15.31:0/3299096668 shutdown_connections 2023-07-29T16:23:29.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.716+0000 7f51b18a7b80 1 -- 172.21.15.31:0/3299096668 wait complete. 2023-07-29T16:23:29.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 -- start start 2023-07-29T16:23:29.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 Processor -- start 2023-07-29T16:23:29.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c58830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c58d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9b3c510 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9b3c1b0 con 0x55c1c9c4f530 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9a23920 con 0x55c1c9c502f0 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.717+0000 7f51b18a7b80 10 client.0 _ll_register_callbacks cb 0x55c1c9b45620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5197fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c58830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c58d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c58d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49364/0 (socket says 172.21.15.31:49364) 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 -- 172.21.15.31:0/1094968853 learned_addr learned my addr 172.21.15.31:0/1094968853 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5197fff700 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 -- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 msgr2=0x55c1c9c5fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 -- 172.21.15.31:0/1094968853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 msgr2=0x55c1c9c58830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c58830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 -- 172.21.15.31:0/1094968853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5198109090 con 0x55c1c9c4f530 2023-07-29T16:23:29.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5196ffd700 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c58d70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f518c000c00 tx=0x7f518c00ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e94d46cd7b956159 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.718+0000 7f5176ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:29.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f5176ffd700 1 -- 172.21.15.31:0/1094968853 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c01d030 con 0x55c1c9c4f530 2023-07-29T16:23:29.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f5176ffd700 1 -- 172.21.15.31:0/1094968853 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f518c004920 con 0x55c1c9c4f530 2023-07-29T16:23:29.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c1c9bcad10 con 0x55c1c9c4f530 2023-07-29T16:23:29.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f5176ffd700 1 -- 172.21.15.31:0/1094968853 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f518c004ae0 con 0x55c1c9c4f530 2023-07-29T16:23:29.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 10 client.27905 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:29.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c1c99b6c60 con 0x55c1c9c4f530 2023-07-29T16:23:29.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f5176ffd700 1 -- 172.21.15.31:0/1094968853 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c1c9bcad10 con 0x55c1c9c4f530 2023-07-29T16:23:29.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f5176ffd700 1 -- 172.21.15.31:0/1094968853 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f518c02c400 con 0x55c1c9c4f530 2023-07-29T16:23:29.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 10 client.27905 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:29.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 10 client.27905 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:23:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 20 client.27905 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9c640c0 0x55c1c9c664a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 4 client.27905 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:29.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.719+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c1c9c1dd70 con 0x55c1c9c640c0 2023-07-29T16:23:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.720+0000 7f5197fff700 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9c640c0 0x55c1c9c664a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.720+0000 7f5197fff700 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9c640c0 0x55c1c9c664a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2b26745da6aa2b16 server_cookie=99655004af3e1956 in_seq=0 out_seq=0 2023-07-29T16:23:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.720+0000 7f5176ffd700 10 client.27905 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:23:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f5176ffd700 1 -- 172.21.15.31:0/1094968853 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x7f51900255f0 con 0x55c1c9c640c0 2023-07-29T16:23:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f5176ffd700 10 client.27905 handle_command_reply: tid=0 2023-07-29T16:23:29.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 client.27905 shutdown 2023-07-29T16:23:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 msgr2=0x55c1c9c58d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c58d70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f518c000c00 tx=0x7f518c00ba20 comp rx=0 tx=0).stop 2023-07-29T16:23:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 shutdown_connections 2023-07-29T16:23:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c58830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9c640c0 0x55c1c9c664a0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/1094968853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c4f530 0x55c1c9c58d70 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f5176ffd700 0 client.27905 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:23:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 >> 172.21.15.31:0/1094968853 conn(0x55c1c9c443e0 msgr2=0x55c1c9c49060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 shutdown_connections 2023-07-29T16:23:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.721+0000 7f51b18a7b80 1 -- 172.21.15.31:0/1094968853 wait complete. 2023-07-29T16:23:29.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.722+0000 7f51b18a7b80 20 client.27905 trim_cache size 0 max 16384 2023-07-29T16:23:29.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 Processor -- start 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 -- start start 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c506c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c566e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9c583a0 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9c46460 con 0x55c1c9c502f0 2023-07-29T16:23:29.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9c416e0 con 0x55c1c9c4f530 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5197fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40528/0 (socket says 172.21.15.31:40528) 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 -- 172.21.15.31:0/2976662455 learned_addr learned my addr 172.21.15.31:0/2976662455 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5197fff700 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 -- 172.21.15.31:0/2976662455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 msgr2=0x55c1c9c566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 -- 172.21.15.31:0/2976662455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 msgr2=0x55c1c9c506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.737+0000 7f5196ffd700 1 -- 172.21.15.31:0/2976662455 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1c99f4250 con 0x55c1c9c4f530 2023-07-29T16:23:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.738+0000 7f5196ffd700 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5fd70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f518c006010 tx=0x7f518c02c9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42b668b7a0517692 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.738+0000 7f517e7fc700 1 -- 172.21.15.31:0/2976662455 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c01d040 con 0x55c1c9c4f530 2023-07-29T16:23:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.738+0000 7f517e7fc700 1 -- 172.21.15.31:0/2976662455 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f518c00ba90 con 0x55c1c9c4f530 2023-07-29T16:23:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.738+0000 7f517e7fc700 1 -- 172.21.15.31:0/2976662455 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c02a6e0 con 0x55c1c9c4f530 2023-07-29T16:23:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 -- 172.21.15.31:0/2976662455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 msgr2=0x55c1c9c5fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5fd70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f518c006010 tx=0x7f518c02c9b0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 -- 172.21.15.31:0/2976662455 shutdown_connections 2023-07-29T16:23:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/2976662455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5fd70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 -- 172.21.15.31:0/2976662455 >> 172.21.15.31:0/2976662455 conn(0x55c1c9c443e0 msgr2=0x55c1c9c574e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 -- 172.21.15.31:0/2976662455 shutdown_connections 2023-07-29T16:23:29.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 -- 172.21.15.31:0/2976662455 wait complete. 2023-07-29T16:23:29.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.739+0000 7f51b18a7b80 1 -- start start 2023-07-29T16:23:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 Processor -- start 2023-07-29T16:23:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c5bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9c5f730 con 0x55c1c9c4f530 2023-07-29T16:23:29.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9c5f8a0 con 0x55c1c9c502f0 2023-07-29T16:23:29.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1c9c5fa60 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f51b18a7b80 10 client.0 _ll_register_callbacks cb 0x55c1c9b45620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:29.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f5197fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c5bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f5197fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c5bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40532/0 (socket says 172.21.15.31:40532) 2023-07-29T16:23:29.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.740+0000 7f5197fff700 1 -- 172.21.15.31:0/90309767 learned_addr learned my addr 172.21.15.31:0/90309767 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5196ffd700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 -- 172.21.15.31:0/90309767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 msgr2=0x55c1c9c5f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 -- 172.21.15.31:0/90309767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4f530 msgr2=0x55c1c9c5c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:29.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 -- 172.21.15.31:0/90309767 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1c99b6c60 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5197fff700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c5bce0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5190004b10 tx=0x7f5190004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a83a7ef38fe206a3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5176ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.741+0000 7f5176ffd700 1 -- 172.21.15.31:0/90309767 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519000e040 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f518c01e040 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f5176ffd700 1 -- 172.21.15.31:0/90309767 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5190004770 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f5176ffd700 1 -- 172.21.15.31:0/90309767 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5190004930 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 10 client.27922 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c1c99cc2b0 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f5176ffd700 1 -- 172.21.15.31:0/90309767 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f518c01e040 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f5176ffd700 1 -- 172.21.15.31:0/90309767 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f519001ac00 con 0x55c1c9c4ebe0 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 10 client.27922 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 10 client.27922 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 20 client.27922 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9cb6cf0 0x55c1c9cb9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 4 client.27922 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:23:29.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.742+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c1c9b98d40 con 0x55c1c9cb6cf0 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f5196ffd700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9cb6cf0 0x55c1c9cb9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f5196ffd700 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9cb6cf0 0x55c1c9cb9100 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86e3db72172b1adc server_cookie=b2a227331cf1322a in_seq=0 out_seq=0 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f5176ffd700 10 client.27922 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f5176ffd700 1 -- 172.21.15.31:0/90309767 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c1c9b98d40 con 0x55c1c9cb6cf0 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f5176ffd700 10 client.27922 handle_command_reply: tid=0 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f51b18a7b80 1 client.27922 shutdown 2023-07-29T16:23:29.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 msgr2=0x55c1c9c5bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.743+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c5bce0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5190004b10 tx=0x7f5190004b40 comp rx=0 tx=0).stop 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 shutdown_connections 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55c1c9cb6cf0 0x55c1c9cb9100 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c1c9c502f0 0x55c1c9c5f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f5176ffd700 0 client.27922 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c1c9c4ebe0 0x55c1c9c5bce0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 --2- 172.21.15.31:0/90309767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c1c9c4f530 0x55c1c9c5c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 >> 172.21.15.31:0/90309767 conn(0x55c1c9c443e0 msgr2=0x55c1c9c566e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 shutdown_connections 2023-07-29T16:23:29.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 1 -- 172.21.15.31:0/90309767 wait complete. 2023-07-29T16:23:29.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.744+0000 7f51b18a7b80 20 client.27922 trim_cache size 0 max 16384 2023-07-29T16:23:29.758 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:23:29.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51800a0810 msgr2=0x7f51800a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51800a0810 0x7f51800a2cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f51880958d0 tx=0x7f5188003710 comp rx=0 tx=0).stop 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 msgr2=0x7f5198074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 0x7f5198074b00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f518800e970 tx=0x7f518800ec80 comp rx=0 tx=0).stop 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 shutdown_connections 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51800a0810 0x7f51800a2cc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5198101ea0 0x7f5198073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5198100320 0x7f5198074b00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 --2- 172.21.15.31:0/15916933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5198101510 0x7f5198073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 >> 172.21.15.31:0/15916933 conn(0x7f51980fb920 msgr2=0x7f51980fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 shutdown_connections 2023-07-29T16:23:29.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.745+0000 7f519dfff700 1 -- 172.21.15.31:0/15916933 wait complete. 2023-07-29T16:23:29.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:29 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/4124354416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:23:29.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:23:29.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:29 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/4124354416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:23:29.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:29 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/4124354416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:23:29.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 521391 8.91 MB/sec execute 194 sec latency 22.439 ms 2023-07-29T16:23:29.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 -- 172.21.15.31:0/2816926203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 msgr2=0x7f8dcc100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/2816926203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc100740 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8009990 tx=0x7f8dc8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 -- 172.21.15.31:0/2816926203 shutdown_connections 2023-07-29T16:23:29.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/2816926203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dcc101eb0 0x7f8dcc108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/2816926203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc100740 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/2816926203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc101520 0x7f8dcc101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 -- 172.21.15.31:0/2816926203 >> 172.21.15.31:0/2816926203 conn(0x7f8dcc0fb970 msgr2=0x7f8dcc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:29.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 -- 172.21.15.31:0/2816926203 shutdown_connections 2023-07-29T16:23:29.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.989+0000 7f8dd3dff700 1 -- 172.21.15.31:0/2816926203 wait complete. 2023-07-29T16:23:29.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.990+0000 7f8dd3dff700 1 Processor -- start 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.990+0000 7f8dd3dff700 1 -- start start 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd3dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd3dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dcc101520 0x7f8dcc073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd3dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc101eb0 0x7f8dcc073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd3dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc073e20 con 0x7f8dcc101520 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd3dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc073f90 con 0x7f8dcc100330 2023-07-29T16:23:29.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd3dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dcc1a0170 con 0x7f8dcc101eb0 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46412/0 (socket says 172.21.15.31:46412) 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/418813626 learned_addr learned my addr 172.21.15.31:0/418813626 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd139a700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dcc101520 0x7f8dcc073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd239c700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc101eb0 0x7f8dcc073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/418813626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc101eb0 msgr2=0x7f8dcc073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc101eb0 0x7f8dcc073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dcc101520 msgr2=0x7f8dcc073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:29.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dcc101520 0x7f8dcc073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:29.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.991+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/418813626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc80095f0 con 0x7f8dcc100330 2023-07-29T16:23:29.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.992+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc074b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8005740 tx=0x7f8dc8005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:29.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.992+0000 7f8dc2ffd700 1 -- 172.21.15.31:0/418813626 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc801d070 con 0x7f8dcc100330 2023-07-29T16:23:29.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.992+0000 7f8dc2ffd700 1 -- 172.21.15.31:0/418813626 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dc8021410 con 0x7f8dcc100330 2023-07-29T16:23:29.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.992+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dcc1a03f0 con 0x7f8dcc100330 2023-07-29T16:23:29.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.992+0000 7f8dc2ffd700 1 -- 172.21.15.31:0/418813626 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8017650 con 0x7f8dcc100330 2023-07-29T16:23:29.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.992+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dcc1a0870 con 0x7f8dcc100330 2023-07-29T16:23:29.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.995+0000 7f8dc2ffd700 1 -- 172.21.15.31:0/418813626 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8dc8021580 con 0x7f8dcc100330 2023-07-29T16:23:29.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 Processor -- start 2023-07-29T16:23:29.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8dc2ffd700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8db80a0810 0x7f8db80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 -- start start 2023-07-29T16:23:29.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8dc2ffd700 1 -- 172.21.15.31:0/418813626 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8dc80ccd50 con 0x7f8dcc100330 2023-07-29T16:23:29.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f58fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8dd239c700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8db80a0810 0x7f8db80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 0x56439f590d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f591740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f4e7a10 con 0x56439f5912f0 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f4e53d0 con 0x56439f58fbe0 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.997+0000 7f8de76edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f4e0cc0 con 0x56439f590530 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd139a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 0x56439f590d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f58fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd139a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 0x56439f590d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57192/0 (socket says 172.21.15.31:57192) 2023-07-29T16:23:29.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd139a700 1 -- 172.21.15.31:0/125111071 learned_addr learned my addr 172.21.15.31:0/125111071 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd239c700 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f591740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/125111071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 msgr2=0x56439f590d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 0x56439f590d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/125111071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 msgr2=0x56439f591740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f591740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/125111071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56439f30ede0 con 0x56439f58fbe0 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd139a700 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 0x56439f590d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:30.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f58fff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8dc809a180 tx=0x7f8dc8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceb9b46cc905530c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8dd239c700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8db80a0810 0x7f8db80a2cc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc075160 tx=0x7f8dc400fc00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.998+0000 7f8db7fff700 1 -- 172.21.15.31:0/125111071 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc809a480 con 0x56439f58fbe0 2023-07-29T16:23:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8db7fff700 1 -- 172.21.15.31:0/125111071 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dc80b0070 con 0x56439f58fbe0 2023-07-29T16:23:30.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8db7fff700 1 -- 172.21.15.31:0/125111071 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc809a6e0 con 0x56439f58fbe0 2023-07-29T16:23:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 -- 172.21.15.31:0/125111071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 msgr2=0x56439f58fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f58fff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8dc809a180 tx=0x7f8dc8009310 comp rx=0 tx=0).stop 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 -- 172.21.15.31:0/125111071 shutdown_connections 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f591740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f58fff0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 --2- 172.21.15.31:0/125111071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f590530 0x56439f590d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:29.999+0000 7f8de76edb80 1 -- 172.21.15.31:0/125111071 >> 172.21.15.31:0/125111071 conn(0x56439f5853e0 msgr2=0x56439f58a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.000+0000 7f8de76edb80 1 -- 172.21.15.31:0/125111071 shutdown_connections 2023-07-29T16:23:30.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.000+0000 7f8de76edb80 1 -- 172.21.15.31:0/125111071 wait complete. 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.000+0000 7f8de76edb80 1 -- start start 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.000+0000 7f8de76edb80 1 Processor -- start 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 0x56439f5994f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f599a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f5a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f47d510 con 0x56439f5912f0 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f47d1b0 con 0x56439f590530 2023-07-29T16:23:30.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f364920 con 0x56439f58fbe0 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8de76edb80 10 client.0 _ll_register_callbacks cb 0x56439f486620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 0x56439f5994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 0x56439f5994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57204/0 (socket says 172.21.15.31:57204) 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/1148143916 learned_addr learned my addr 172.21.15.31:0/1148143916 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8dd139a700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f599a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.001+0000 7f8dd239c700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f5a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/1148143916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 msgr2=0x56439f599a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f599a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 msgr2=0x56439f5a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f5a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/1148143916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dbc0099e0 con 0x56439f58fbe0 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 0x56439f5994f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8098a70 tx=0x7f8dc80967c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5e9b8b28f167911 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8db57fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8db57fa700 1 -- 172.21.15.31:0/1148143916 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc809a480 con 0x56439f58fbe0 2023-07-29T16:23:30.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56439f50bd10 con 0x56439f58fbe0 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8db57fa700 1 -- 172.21.15.31:0/1148143916 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8dc809e4b0 con 0x56439f58fbe0 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8db57fa700 1 -- 172.21.15.31:0/1148143916 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dc809e170 con 0x56439f58fbe0 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.002+0000 7f8de76edb80 10 client.27934 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f8dcc1a0870 con 0x56439f58fbe0 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8db57fa700 1 -- 172.21.15.31:0/1148143916 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56439f50bd10 con 0x56439f58fbe0 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8db57fa700 1 -- 172.21.15.31:0/1148143916 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8dc80a00a0 con 0x56439f58fbe0 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 10 client.27934 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:30.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 10 client.27934 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 20 client.27934 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5a4d10 0x56439f5a7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 4 client.27934 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56439f55ed70 con 0x56439f5a4d10 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.003+0000 7f8dd139a700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5a4d10 0x56439f5a7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.004+0000 7f8dd139a700 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5a4d10 0x56439f5a7120 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1672f4e5d08cd52a server_cookie=859f8a6d43f61f1d in_seq=0 out_seq=0 2023-07-29T16:23:30.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.004+0000 7f8db57fa700 10 client.27934 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:23:30.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.004+0000 7f8db57fa700 1 -- 172.21.15.31:0/1148143916 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56439f55ed70 con 0x56439f5a4d10 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.004+0000 7f8db57fa700 10 client.27934 handle_command_reply: tid=0 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 client.27934 shutdown 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 msgr2=0x56439f5994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 0x56439f5994f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8098a70 tx=0x7f8dc80967c0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 shutdown_connections 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5a4d10 0x56439f5a7120 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f5912f0 0x56439f5a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f58fbe0 0x56439f5994f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 --2- 172.21.15.31:0/1148143916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f599a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 >> 172.21.15.31:0/1148143916 conn(0x56439f5853e0 msgr2=0x56439f58a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8db57fa700 0 client.27934 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 shutdown_connections 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 1 -- 172.21.15.31:0/1148143916 wait complete. 2023-07-29T16:23:30.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.005+0000 7f8de76edb80 20 client.27934 trim_cache size 0 max 16384 2023-07-29T16:23:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 Processor -- start 2023-07-29T16:23:30.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 -- start start 2023-07-29T16:23:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f59ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f590530 0x56439f58a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f5a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f5a4b50 con 0x56439f590530 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f590bc0 con 0x56439f58fbe0 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8de76edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f587490 con 0x56439f5912f0 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8dd139a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f590530 0x56439f58a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f59ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8dd239c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f5a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f59ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46438/0 (socket says 172.21.15.31:46438) 2023-07-29T16:23:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.020+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/518332408 learned_addr learned my addr 172.21.15.31:0/518332408 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/518332408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 msgr2=0x56439f5a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f5a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/518332408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f590530 msgr2=0x56439f58a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f590530 0x56439f58a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dd1b9b700 1 -- 172.21.15.31:0/518332408 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56439f2f7c60 con 0x56439f58fbe0 2023-07-29T16:23:30.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f59ea20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8dc809a180 tx=0x7f8dc8021c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c7015fdf3f6e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dc0bff700 1 -- 172.21.15.31:0/518332408 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8098890 con 0x56439f58fbe0 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dc0bff700 1 -- 172.21.15.31:0/518332408 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dc80a6070 con 0x56439f58fbe0 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.021+0000 7f8dc0bff700 1 -- 172.21.15.31:0/518332408 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc80a23b0 con 0x56439f58fbe0 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 -- 172.21.15.31:0/518332408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 msgr2=0x56439f59ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f59ea20 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8dc809a180 tx=0x7f8dc8021c00 comp rx=0 tx=0).stop 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 -- 172.21.15.31:0/518332408 shutdown_connections 2023-07-29T16:23:30.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f5a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f58fbe0 0x56439f59ea20 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 --2- 172.21.15.31:0/518332408 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f590530 0x56439f58a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 -- 172.21.15.31:0/518332408 >> 172.21.15.31:0/518332408 conn(0x56439f5853e0 msgr2=0x56439f5981b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 -- 172.21.15.31:0/518332408 shutdown_connections 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.022+0000 7f8de76edb80 1 -- 172.21.15.31:0/518332408 wait complete. 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 -- start start 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 Processor -- start 2023-07-29T16:23:30.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f58fbe0 0x56439f59cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f59d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f59dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f59e230 con 0x56439f58fbe0 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f59e370 con 0x56439f590530 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56439f59e530 con 0x56439f5912f0 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.023+0000 7f8de76edb80 10 client.0 _ll_register_callbacks cb 0x56439f486620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd1b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f58fbe0 0x56439f59cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f59dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd139a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f59d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f59dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57210/0 (socket says 172.21.15.31:57210) 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 -- 172.21.15.31:0/752734936 learned_addr learned my addr 172.21.15.31:0/752734936 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 -- 172.21.15.31:0/752734936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 msgr2=0x56439f59d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f59d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 -- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f58fbe0 msgr2=0x56439f59cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f58fbe0 0x56439f59cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 -- 172.21.15.31:0/752734936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dcc1a0870 con 0x56439f5912f0 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd139a700 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f59d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8dd239c700 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f59dcf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4018b50 tx=0x7f8dc4004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84c9822e7fdfc0b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.024+0000 7f8db57fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8db57fa700 1 -- 172.21.15.31:0/752734936 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc400e040 con 0x56439f5912f0 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8dc80b0070 con 0x56439f5912f0 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8db57fa700 1 -- 172.21.15.31:0/752734936 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8dc4018e60 con 0x56439f5912f0 2023-07-29T16:23:30.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8db57fa700 1 -- 172.21.15.31:0/752734936 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dc4008db0 con 0x56439f5912f0 2023-07-29T16:23:30.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8de76edb80 10 client.27940 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:30.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56439f335250 con 0x56439f5912f0 2023-07-29T16:23:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8db57fa700 1 -- 172.21.15.31:0/752734936 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dc80b0070 con 0x56439f5912f0 2023-07-29T16:23:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8db57fa700 1 -- 172.21.15.31:0/752734936 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8dc402c020 con 0x56439f5912f0 2023-07-29T16:23:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.025+0000 7f8de76edb80 10 client.27940 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:30.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8de76edb80 10 client.27940 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8de76edb80 20 client.27940 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8de76edb80 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5f7740 0x56439f5f9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8de76edb80 4 client.27940 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56439f4d9d40 con 0x56439f5f7740 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5f7740 0x56439f5f9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8dd1b9b700 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5f7740 0x56439f5f9b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=47a2b55277d2f16d server_cookie=a2d5ba02703812 in_seq=0 out_seq=0 2023-07-29T16:23:30.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.026+0000 7f8db57fa700 10 client.27940 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8db57fa700 1 -- 172.21.15.31:0/752734936 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56439f4d9d40 con 0x56439f5f7740 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8db57fa700 10 client.27940 handle_command_reply: tid=0 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 client.27940 shutdown 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 msgr2=0x56439f59dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f59dcf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8dc4018b50 tx=0x7f8dc4004b40 comp rx=0 tx=0).stop 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 shutdown_connections 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56439f5f7740 0x56439f5f9b50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56439f5912f0 0x56439f59dcf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8db57fa700 0 client.27940 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56439f58fbe0 0x56439f59cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 --2- 172.21.15.31:0/752734936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56439f590530 0x56439f59d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 >> 172.21.15.31:0/752734936 conn(0x56439f5853e0 msgr2=0x56439f5a6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 shutdown_connections 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 1 -- 172.21.15.31:0/752734936 wait complete. 2023-07-29T16:23:30.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.027+0000 7f8de76edb80 20 client.27940 trim_cache size 0 max 16384 2023-07-29T16:23:30.035 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:23:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8db80a0810 msgr2=0x7f8db80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8db80a0810 0x7f8db80a2cc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc075160 tx=0x7f8dc400fc00 comp rx=0 tx=0).stop 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 msgr2=0x7f8dcc074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc074b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8005740 tx=0x7f8dc8005770 comp rx=0 tx=0).stop 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 shutdown_connections 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8db80a0810 0x7f8db80a2cc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8dcc101eb0 0x7f8dcc073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8dcc100330 0x7f8dcc074b00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 --2- 172.21.15.31:0/418813626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8dcc101520 0x7f8dcc073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 >> 172.21.15.31:0/418813626 conn(0x7f8dcc0fb970 msgr2=0x7f8dcc104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 shutdown_connections 2023-07-29T16:23:30.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.028+0000 7f8dd3dff700 1 -- 172.21.15.31:0/418813626 wait complete. 2023-07-29T16:23:30.047 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:23:30.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 -- 172.21.15.31:0/324687945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4754100210 msgr2=0x7f4754100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 --2- 172.21.15.31:0/324687945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4754100210 0x7f4754100620 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f4748009990 tx=0x7f4748009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 -- 172.21.15.31:0/324687945 shutdown_connections 2023-07-29T16:23:30.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 --2- 172.21.15.31:0/324687945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4754101e50 0x7f47541086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 --2- 172.21.15.31:0/324687945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4754100210 0x7f4754100620 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 --2- 172.21.15.31:0/324687945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47541014a0 0x7f4754101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 -- 172.21.15.31:0/324687945 >> 172.21.15.31:0/324687945 conn(0x7f47540fb850 msgr2=0x7f47540fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 -- 172.21.15.31:0/324687945 shutdown_connections 2023-07-29T16:23:30.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.279+0000 7f4759d9d700 1 -- 172.21.15.31:0/324687945 wait complete. 2023-07-29T16:23:30.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.280+0000 7f4759d9d700 1 Processor -- start 2023-07-29T16:23:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 -- start start 2023-07-29T16:23:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4754100210 0x7f4754197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 0x7f4754198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4754101e50 0x7f475419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419fb70 con 0x7f4754100210 2023-07-29T16:23:30.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419fcb0 con 0x7f47541014a0 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4759d9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475419fe40 con 0x7f4754101e50 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4753fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 0x7f4754198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4758d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4754100210 0x7f4754197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4753fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 0x7f4754198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46460/0 (socket says 172.21.15.31:46460) 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f4753fff700 1 -- 172.21.15.31:0/3034716233 learned_addr learned my addr 172.21.15.31:0/3034716233 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.281+0000 7f475959c700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4754101e50 0x7f475419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4753fff700 1 -- 172.21.15.31:0/3034716233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4754101e50 msgr2=0x7f475419f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4753fff700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4754101e50 0x7f475419f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4753fff700 1 -- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4754100210 msgr2=0x7f4754197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4753fff700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4754100210 0x7f4754197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4753fff700 1 -- 172.21.15.31:0/3034716233 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47480095f0 con 0x7f47541014a0 2023-07-29T16:23:30.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f475959c700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4754101e50 0x7f475419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:30.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4753fff700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 0x7f4754198530 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f474000e9a0 tx=0x7f474000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4751ffb700 1 -- 172.21.15.31:0/3034716233 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740009790 con 0x7f47541014a0 2023-07-29T16:23:30.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4751ffb700 1 -- 172.21.15.31:0/3034716233 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f474000cbd0 con 0x7f47541014a0 2023-07-29T16:23:30.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4751ffb700 1 -- 172.21.15.31:0/3034716233 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4740018680 con 0x7f47541014a0 2023-07-29T16:23:30.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47541a0120 con 0x7f47541014a0 2023-07-29T16:23:30.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.282+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47541a05e0 con 0x7f47541014a0 2023-07-29T16:23:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.286+0000 7f4751ffb700 1 -- 172.21.15.31:0/3034716233 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f474000cd40 con 0x7f47541014a0 2023-07-29T16:23:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.287+0000 7f476e94bb80 1 Processor -- start 2023-07-29T16:23:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.287+0000 7f476e94bb80 1 -- start start 2023-07-29T16:23:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.287+0000 7f4751ffb700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47440a08b0 0x7f47440a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.287+0000 7f4751ffb700 1 -- 172.21.15.31:0/3034716233 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4740014070 con 0x7f47541014a0 2023-07-29T16:23:30.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.287+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 0x55e0ea667150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.287+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea667f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea668920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f476e94bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea5bea10 con 0x55e0ea6684d0 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f476e94bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea5bc3d0 con 0x55e0ea666d40 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f476e94bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea5b7cc0 con 0x55e0ea667690 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f475959c700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47440a08b0 0x7f47440a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 0x55e0ea667150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 0x55e0ea667150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46468/0 (socket says 172.21.15.31:46468) 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 -- 172.21.15.31:0/3326592790 learned_addr learned my addr 172.21.15.31:0/3326592790 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f475959c700 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea668920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4753fff700 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea667f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 -- 172.21.15.31:0/3326592790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 msgr2=0x55e0ea667f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea667f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 -- 172.21.15.31:0/3326592790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 msgr2=0x55e0ea668920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea668920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.288+0000 7f4758d9b700 1 -- 172.21.15.31:0/3326592790 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0ea3e5de0 con 0x55e0ea666d40 2023-07-29T16:23:30.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.289+0000 7f4758d9b700 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 0x55e0ea667150 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4748015000 tx=0x7f4748005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3388b455d9cd5e6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.289+0000 7f475959c700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47440a08b0 0x7f47440a2d60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f474c0129d0 tx=0x7f474c012240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.289+0000 7f473effd700 1 -- 172.21.15.31:0/3326592790 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474801d070 con 0x55e0ea666d40 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.289+0000 7f473effd700 1 -- 172.21.15.31:0/3326592790 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f474801e040 con 0x55e0ea666d40 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.289+0000 7f473effd700 1 -- 172.21.15.31:0/3326592790 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47480048e0 con 0x55e0ea666d40 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 -- 172.21.15.31:0/3326592790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 msgr2=0x55e0ea667150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 0x55e0ea667150 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4748015000 tx=0x7f4748005ea0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 -- 172.21.15.31:0/3326592790 shutdown_connections 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea668920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea666d40 0x55e0ea667150 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3326592790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea667f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 -- 172.21.15.31:0/3326592790 >> 172.21.15.31:0/3326592790 conn(0x55e0ea65c540 msgr2=0x55e0ea661ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 -- 172.21.15.31:0/3326592790 shutdown_connections 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 -- 172.21.15.31:0/3326592790 wait complete. 2023-07-29T16:23:30.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.290+0000 7f476e94bb80 1 -- start start 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 Processor -- start 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea6708c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea670e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea6684d0 0x55e0ea677e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea554510 con 0x55e0ea666d40 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea5541b0 con 0x55e0ea667690 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea43b920 con 0x55e0ea6684d0 2023-07-29T16:23:30.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f476e94bb80 10 client.0 _ll_register_callbacks cb 0x55e0ea55d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f4753fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea670e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f4758d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea6708c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f4753fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea670e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46480/0 (socket says 172.21.15.31:46480) 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.291+0000 7f4753fff700 1 -- 172.21.15.31:0/2575401104 learned_addr learned my addr 172.21.15.31:0/2575401104 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f475959c700 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea6684d0 0x55e0ea677e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4753fff700 1 -- 172.21.15.31:0/2575401104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea6684d0 msgr2=0x55e0ea677e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.296 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4753fff700 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea6684d0 0x55e0ea677e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4753fff700 1 -- 172.21.15.31:0/2575401104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 msgr2=0x55e0ea6708c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4753fff700 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea6708c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4753fff700 1 -- 172.21.15.31:0/2575401104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4754108f60 con 0x55e0ea667690 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4753fff700 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea670e00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4740095210 tx=0x7f4740099490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f570ab6e9a4ee756 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4733fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4733fff700 1 -- 172.21.15.31:0/2575401104 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47400a18e0 con 0x55e0ea667690 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f4733fff700 1 -- 172.21.15.31:0/2575401104 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f47400ae070 con 0x55e0ea667690 2023-07-29T16:23:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.292+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0ea5e2d10 con 0x55e0ea667690 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f4733fff700 1 -- 172.21.15.31:0/2575401104 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4740098980 con 0x55e0ea667690 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 10 client.27941 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e0ea3cec60 con 0x55e0ea667690 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f4733fff700 1 -- 172.21.15.31:0/2575401104 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47400c7020 con 0x55e0ea667690 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f4733fff700 1 -- 172.21.15.31:0/2575401104 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f47400983f0 con 0x55e0ea667690 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 10 client.27941 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:30.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 10 client.27941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 20 client.27941 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea67c640 0x55e0ea67ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 4 client.27941 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.293+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0ea635d70 con 0x55e0ea67c640 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.294+0000 7f4758d9b700 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea67c640 0x55e0ea67ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.294+0000 7f4758d9b700 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea67c640 0x55e0ea67ea50 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c91b87e4b8829453 server_cookie=e1d39fd391f9276a in_seq=0 out_seq=0 2023-07-29T16:23:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.294+0000 7f4733fff700 10 client.27941 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:30.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f4733fff700 1 -- 172.21.15.31:0/2575401104 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e0ea635d70 con 0x55e0ea67c640 2023-07-29T16:23:30.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f4733fff700 10 client.27941 handle_command_reply: tid=0 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 client.27941 shutdown 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 msgr2=0x55e0ea670e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea670e00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4740095210 tx=0x7f4740099490 comp rx=0 tx=0).stop 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 shutdown_connections 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea67c640 0x55e0ea67ea50 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea6684d0 0x55e0ea677e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea6708c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 --2- 172.21.15.31:0/2575401104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea670e00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f4733fff700 0 client.27941 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.295+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 >> 172.21.15.31:0/2575401104 conn(0x55e0ea65c540 msgr2=0x55e0ea6611c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.296+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 shutdown_connections 2023-07-29T16:23:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.296+0000 7f476e94bb80 1 -- 172.21.15.31:0/2575401104 wait complete. 2023-07-29T16:23:30.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.296+0000 7f476e94bb80 20 client.27941 trim_cache size 0 max 16384 2023-07-29T16:23:30.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 Processor -- start 2023-07-29T16:23:30.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 -- start start 2023-07-29T16:23:30.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea666d40 0x55e0ea67ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea677e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.311 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea66f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea66fba0 con 0x55e0ea6684d0 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea66f390 con 0x55e0ea667690 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f476e94bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea6833d0 con 0x55e0ea666d40 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f4753fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea677e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f4753fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea677e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46496/0 (socket says 172.21.15.31:46496) 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f4753fff700 1 -- 172.21.15.31:0/1412412307 learned_addr learned my addr 172.21.15.31:0/1412412307 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.312+0000 7f4758d9b700 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea666d40 0x55e0ea67ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f475959c700 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea66f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4758d9b700 1 -- 172.21.15.31:0/1412412307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 msgr2=0x55e0ea677e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4758d9b700 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea677e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4758d9b700 1 -- 172.21.15.31:0/1412412307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 msgr2=0x55e0ea66f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4758d9b700 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea66f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4758d9b700 1 -- 172.21.15.31:0/1412412307 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0ea40c250 con 0x55e0ea666d40 2023-07-29T16:23:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4753fff700 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea677e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f4758d9b700 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea666d40 0x55e0ea67ea20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f474800bb30 tx=0x7f474801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34c0d02d6a442f10 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.313+0000 7f473f7fe700 1 -- 172.21.15.31:0/1412412307 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474801d040 con 0x55e0ea666d40 2023-07-29T16:23:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f473f7fe700 1 -- 172.21.15.31:0/1412412307 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f474800be60 con 0x55e0ea666d40 2023-07-29T16:23:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f473f7fe700 1 -- 172.21.15.31:0/1412412307 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748034c90 con 0x55e0ea666d40 2023-07-29T16:23:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 -- 172.21.15.31:0/1412412307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea666d40 msgr2=0x55e0ea67ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea666d40 0x55e0ea67ea20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f474800bb30 tx=0x7f474801a040 comp rx=0 tx=0).stop 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 -- 172.21.15.31:0/1412412307 shutdown_connections 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea6684d0 0x55e0ea66f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea666d40 0x55e0ea67ea20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 --2- 172.21.15.31:0/1412412307 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea667690 0x55e0ea677e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 -- 172.21.15.31:0/1412412307 >> 172.21.15.31:0/1412412307 conn(0x55e0ea65c540 msgr2=0x55e0ea65da10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 -- 172.21.15.31:0/1412412307 shutdown_connections 2023-07-29T16:23:30.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.314+0000 7f476e94bb80 1 -- 172.21.15.31:0/1412412307 wait complete. 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 -- start start 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 Processor -- start 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea673f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea674460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea6684d0 0x55e0ea677480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea6779c0 con 0x55e0ea666d40 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea677b30 con 0x55e0ea6684d0 2023-07-29T16:23:30.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0ea677cf0 con 0x55e0ea667690 2023-07-29T16:23:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.315+0000 7f476e94bb80 10 client.0 _ll_register_callbacks cb 0x55e0ea55d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea674460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f475959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea6684d0 0x55e0ea677480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea674460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57262/0 (socket says 172.21.15.31:57262) 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 -- 172.21.15.31:0/3847763590 learned_addr learned my addr 172.21.15.31:0/3847763590 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4758d9b700 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea673f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 -- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea6684d0 msgr2=0x55e0ea677480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea6684d0 0x55e0ea677480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 -- 172.21.15.31:0/3847763590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 msgr2=0x55e0ea673f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea673f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 -- 172.21.15.31:0/3847763590 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e0ea3cec60 con 0x55e0ea667690 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4753fff700 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea674460 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4740095990 tx=0x7f474009a810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c3b0d67010f188 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.316+0000 7f4733fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f4733fff700 1 -- 172.21.15.31:0/3847763590 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474009a270 con 0x55e0ea667690 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f474801e040 con 0x55e0ea667690 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f4733fff700 1 -- 172.21.15.31:0/3847763590 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f47400ae040 con 0x55e0ea667690 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f4733fff700 1 -- 172.21.15.31:0/3847763590 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f474009a3d0 con 0x55e0ea667690 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 10 client.27958 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:30.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e0ea3e5de0 con 0x55e0ea667690 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f4733fff700 1 -- 172.21.15.31:0/3847763590 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f474801e040 con 0x55e0ea667690 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f4733fff700 1 -- 172.21.15.31:0/3847763590 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4740098a10 con 0x55e0ea667690 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 10 client.27958 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 10 client.27958 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 20 client.27958 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea6cf020 0x55e0ea6d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:30.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 4 client.27958 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:23:30.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.317+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e0ea5b0d40 con 0x55e0ea6cf020 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.318+0000 7f4758d9b700 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea6cf020 0x55e0ea6d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.318+0000 7f4758d9b700 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea6cf020 0x55e0ea6d1430 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea8e3e0b96e1948e server_cookie=6a8a10537245282a in_seq=0 out_seq=0 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.318+0000 7f4733fff700 10 client.27958 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f4733fff700 1 -- 172.21.15.31:0/3847763590 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e0ea5b0d40 con 0x55e0ea6cf020 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f4733fff700 10 client.27958 handle_command_reply: tid=0 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 client.27958 shutdown 2023-07-29T16:23:30.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 msgr2=0x55e0ea674460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea674460 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f4740095990 tx=0x7f474009a810 comp rx=0 tx=0).stop 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 shutdown_connections 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e0ea6cf020 0x55e0ea6d1430 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e0ea6684d0 0x55e0ea677480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e0ea666d40 0x55e0ea673f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 --2- 172.21.15.31:0/3847763590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e0ea667690 0x55e0ea674460 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f4733fff700 0 client.27958 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:30.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 >> 172.21.15.31:0/3847763590 conn(0x55e0ea65c540 msgr2=0x55e0ea66f660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 shutdown_connections 2023-07-29T16:23:30.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.319+0000 7f476e94bb80 1 -- 172.21.15.31:0/3847763590 wait complete. 2023-07-29T16:23:30.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.320+0000 7f476e94bb80 20 client.27958 trim_cache size 0 max 16384 2023-07-29T16:23:30.326 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:23:30.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.320+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47440a08b0 msgr2=0x7f47440a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.320+0000 7f4759d9d700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47440a08b0 0x7f47440a2d60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f474c0129d0 tx=0x7f474c012240 comp rx=0 tx=0).stop 2023-07-29T16:23:30.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.320+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 msgr2=0x7f4754198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:30.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.320+0000 7f4759d9d700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 0x7f4754198530 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f474000e9a0 tx=0x7f474000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 shutdown_connections 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47440a08b0 0x7f47440a2d60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4754101e50 0x7f475419f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4754100210 0x7f4754197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 --2- 172.21.15.31:0/3034716233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47541014a0 0x7f4754198530 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 >> 172.21.15.31:0/3034716233 conn(0x7f47540fb850 msgr2=0x7f47540fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 shutdown_connections 2023-07-29T16:23:30.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:30.321+0000 7f4759d9d700 1 -- 172.21.15.31:0/3034716233 wait complete. 2023-07-29T16:23:30.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:30 smithi031 ceph-mon[95504]: pgmap v611: 97 pgs: 97 active+clean; 109 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 4.0 MiB/s wr, 256 op/s 2023-07-29T16:23:30.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:30 smithi039 ceph-mon[128858]: pgmap v611: 97 pgs: 97 active+clean; 109 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 4.0 MiB/s wr, 256 op/s 2023-07-29T16:23:30.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:30 smithi156 ceph-mon[129245]: pgmap v611: 97 pgs: 97 active+clean; 109 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 4.0 MiB/s wr, 256 op/s 2023-07-29T16:23:30.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 522628 8.91 MB/sec execute 195 sec latency 59.897 ms 2023-07-29T16:23:31.339 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:23:31.339 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:23:31.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 -- 172.21.15.31:0/1915163676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101520 msgr2=0x7fb65c101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 --2- 172.21.15.31:0/1915163676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101520 0x7fb65c101970 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb650009990 tx=0x7fb650009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 -- 172.21.15.31:0/1915163676 shutdown_connections 2023-07-29T16:23:31.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 --2- 172.21.15.31:0/1915163676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c101eb0 0x7fb65c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 --2- 172.21.15.31:0/1915163676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb65c100330 0x7fb65c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 --2- 172.21.15.31:0/1915163676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101520 0x7fb65c101970 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 -- 172.21.15.31:0/1915163676 >> 172.21.15.31:0/1915163676 conn(0x7fb65c0fb970 msgr2=0x7fb65c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 -- 172.21.15.31:0/1915163676 shutdown_connections 2023-07-29T16:23:31.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.567+0000 7fb6629ff700 1 -- 172.21.15.31:0/1915163676 wait complete. 2023-07-29T16:23:31.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.569+0000 7fb6629ff700 1 Processor -- start 2023-07-29T16:23:31.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.569+0000 7fb6629ff700 1 -- start start 2023-07-29T16:23:31.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb6629ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c100330 0x7fb65c074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb6629ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb65c101520 0x7fb65c073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb6629ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101eb0 0x7fb65c073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb6629ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c073e20 con 0x7fb65c101520 2023-07-29T16:23:31.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb6629ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c073f90 con 0x7fb65c101eb0 2023-07-29T16:23:31.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb6629ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1a00f0 con 0x7fb65c100330 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb65bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c100330 0x7fb65c074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb660f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101eb0 0x7fb65c073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb65bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c100330 0x7fb65c074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57278/0 (socket says 172.21.15.31:57278) 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb65bfff700 1 -- 172.21.15.31:0/3564299581 learned_addr learned my addr 172.21.15.31:0/3564299581 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb65c101520 0x7fb65c073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb660f9c700 1 -- 172.21.15.31:0/3564299581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c100330 msgr2=0x7fb65c074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb660f9c700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c100330 0x7fb65c074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb660f9c700 1 -- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb65c101520 msgr2=0x7fb65c073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb660f9c700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb65c101520 0x7fb65c073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.570+0000 7fb660f9c700 1 -- 172.21.15.31:0/3564299581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6500095f0 con 0x7fb65c101eb0 2023-07-29T16:23:31.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.571+0000 7fb660f9c700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101eb0 0x7fb65c073680 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb65400efc0 tx=0x7fb65400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.571+0000 7fb6597fa700 1 -- 172.21.15.31:0/3564299581 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654018070 con 0x7fb65c101eb0 2023-07-29T16:23:31.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.571+0000 7fb6597fa700 1 -- 172.21.15.31:0/3564299581 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb65400f040 con 0x7fb65c101eb0 2023-07-29T16:23:31.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.571+0000 7fb6597fa700 1 -- 172.21.15.31:0/3564299581 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65400cd40 con 0x7fb65c101eb0 2023-07-29T16:23:31.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.571+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65c1a0370 con 0x7fb65c101eb0 2023-07-29T16:23:31.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.571+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c1a0880 con 0x7fb65c101eb0 2023-07-29T16:23:31.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.575+0000 7fb6597fa700 1 -- 172.21.15.31:0/3564299581 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb654003a00 con 0x7fb65c101eb0 2023-07-29T16:23:31.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.576+0000 7fb6597fa700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb6440a0810 0x7fb6440a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.576+0000 7fb6597fa700 1 -- 172.21.15.31:0/3564299581 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb6540cb5d0 con 0x7fb65c101eb0 2023-07-29T16:23:31.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.576+0000 7fb65bfff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb6440a0810 0x7fb6440a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 Processor -- start 2023-07-29T16:23:31.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 -- start start 2023-07-29T16:23:31.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb65bfff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb6440a0810 0x7fb6440a2cc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb64c005950 tx=0x7fb64c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba1f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 0x55facba1fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba20370 0x55facba207c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facb976a90 con 0x55facba1ec60 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facb974450 con 0x55facba20370 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb676404b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facb96fd40 con 0x55facba1f5b0 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.577+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 0x55facba1fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb660f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba20370 0x55facba207c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 0x55facba1fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57286/0 (socket says 172.21.15.31:57286) 2023-07-29T16:23:31.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 -- 172.21.15.31:0/2661798802 learned_addr learned my addr 172.21.15.31:0/2661798802 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65bfff700 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba1f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 -- 172.21.15.31:0/2661798802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba20370 msgr2=0x55facba207c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba20370 0x55facba207c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 -- 172.21.15.31:0/2661798802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 msgr2=0x55facba1f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba1f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 -- 172.21.15.31:0/2661798802 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55facb79de60 con 0x55facba1f5b0 2023-07-29T16:23:31.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65bfff700 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba1f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:31.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.578+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 0x55facba1fde0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb650015000 tx=0x7fb650005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99889d31c9f06f7c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.579+0000 7fb6427fc700 1 -- 172.21.15.31:0/2661798802 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65001d070 con 0x55facba1f5b0 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.579+0000 7fb6427fc700 1 -- 172.21.15.31:0/2661798802 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb65001e040 con 0x55facba1f5b0 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.579+0000 7fb6427fc700 1 -- 172.21.15.31:0/2661798802 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500048f0 con 0x55facba1f5b0 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.579+0000 7fb676404b80 1 -- 172.21.15.31:0/2661798802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 msgr2=0x55facba1fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.579+0000 7fb676404b80 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 0x55facba1fde0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb650015000 tx=0x7fb650005ea0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 -- 172.21.15.31:0/2661798802 shutdown_connections 2023-07-29T16:23:31.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba20370 0x55facba207c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba1f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 --2- 172.21.15.31:0/2661798802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1f5b0 0x55facba1fde0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 -- 172.21.15.31:0/2661798802 >> 172.21.15.31:0/2661798802 conn(0x55facba14460 msgr2=0x55facba19a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 -- 172.21.15.31:0/2661798802 shutdown_connections 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 -- 172.21.15.31:0/2661798802 wait complete. 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.580+0000 7fb676404b80 1 -- start start 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 Processor -- start 2023-07-29T16:23:31.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.583 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba28da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba2fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facb90c590 con 0x55facba20370 2023-07-29T16:23:31.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facb90c230 con 0x55facba1f5b0 2023-07-29T16:23:31.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facb7f39a0 con 0x55facba1ec60 2023-07-29T16:23:31.585 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb676404b80 10 client.0 _ll_register_callbacks cb 0x55facb9156a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.581+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba28da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba28da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46546/0 (socket says 172.21.15.31:46546) 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57296/0 (socket says 172.21.15.31:57296) 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 -- 172.21.15.31:0/197364294 learned_addr learned my addr 172.21.15.31:0/197364294 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb660f9c700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba2fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 -- 172.21.15.31:0/197364294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 msgr2=0x55facba28860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.586 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 -- 172.21.15.31:0/197364294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 msgr2=0x55facba2fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba2fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 -- 172.21.15.31:0/197364294 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c1090a0 con 0x55facba1f5b0 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65bfff700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb660f9c700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba2fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba28da0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb650009350 tx=0x7fb65000ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e528e09428c063fa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.582+0000 7fb63b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb63b7fe700 1 -- 172.21.15.31:0/197364294 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65001d030 con 0x55facba1f5b0 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb63b7fe700 1 -- 172.21.15.31:0/197364294 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb650004a80 con 0x55facba1f5b0 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55facb99ad90 con 0x55facba1f5b0 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb63b7fe700 1 -- 172.21.15.31:0/197364294 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb650005bc0 con 0x55facba1f5b0 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 10 client.27959 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55facb786ce0 con 0x55facba1f5b0 2023-07-29T16:23:31.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb63b7fe700 1 -- 172.21.15.31:0/197364294 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55facb99ad90 con 0x55facba1f5b0 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb63b7fe700 1 -- 172.21.15.31:0/197364294 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb65002c400 con 0x55facba1f5b0 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 10 client.27959 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 10 client.27959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 20 client.27959 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba34160 0x55facba36540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 4 client.27959 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.583+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55facb9eddf0 con 0x55facba34160 2023-07-29T16:23:31.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.584+0000 7fb65bfff700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba34160 0x55facba36540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.584+0000 7fb65bfff700 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba34160 0x55facba36540 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77d02eb62a51a29 server_cookie=cffc0e621a3bb428 in_seq=0 out_seq=0 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.584+0000 7fb63b7fe700 10 client.27959 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb63b7fe700 1 -- 172.21.15.31:0/197364294 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55facb9eddf0 con 0x55facba34160 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb63b7fe700 10 client.27959 handle_command_reply: tid=0 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 client.27959 shutdown 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 msgr2=0x55facba28da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba28da0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb650009350 tx=0x7fb65000ba20 comp rx=0 tx=0).stop 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 shutdown_connections 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba34160 0x55facba36540 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba2fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 --2- 172.21.15.31:0/197364294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba28da0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 >> 172.21.15.31:0/197364294 conn(0x55facba14460 msgr2=0x55facba190e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.585+0000 7fb63b7fe700 0 client.27959 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.586+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 shutdown_connections 2023-07-29T16:23:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.586+0000 7fb676404b80 1 -- 172.21.15.31:0/197364294 wait complete. 2023-07-29T16:23:31.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.586+0000 7fb676404b80 20 client.27959 trim_cache size 0 max 16384 2023-07-29T16:23:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.604+0000 7fb676404b80 1 Processor -- start 2023-07-29T16:23:31.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.604+0000 7fb676404b80 1 -- start start 2023-07-29T16:23:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.604+0000 7fb676404b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.604+0000 7fb676404b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb676404b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba20370 0x55facba28da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb676404b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facba1b360 con 0x55facba1ec60 2023-07-29T16:23:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb676404b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facba29850 con 0x55facba1f5b0 2023-07-29T16:23:31.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb676404b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facba11760 con 0x55facba20370 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba20370 0x55facba28da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb65bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46558/0 (socket says 172.21.15.31:46558) 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb65b7fe700 1 -- 172.21.15.31:0/958754008 learned_addr learned my addr 172.21.15.31:0/958754008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 -- 172.21.15.31:0/958754008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 msgr2=0x55facba2fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 -- 172.21.15.31:0/958754008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 msgr2=0x55facba28860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 -- 172.21.15.31:0/958754008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55facba20a10 con 0x55facba20370 2023-07-29T16:23:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.605+0000 7fb660f9c700 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba20370 0x55facba28da0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb65400c6f0 tx=0x7fb6540cb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e754f25d569b3e1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.606+0000 7fb642ffd700 1 -- 172.21.15.31:0/958754008 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654004ad0 con 0x55facba20370 2023-07-29T16:23:31.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.606+0000 7fb642ffd700 1 -- 172.21.15.31:0/958754008 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb654098d00 con 0x55facba20370 2023-07-29T16:23:31.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.606+0000 7fb642ffd700 1 -- 172.21.15.31:0/958754008 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654098f60 con 0x55facba20370 2023-07-29T16:23:31.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.606+0000 7fb676404b80 1 -- 172.21.15.31:0/958754008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba20370 msgr2=0x55facba28da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.606+0000 7fb676404b80 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba20370 0x55facba28da0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb65400c6f0 tx=0x7fb6540cb040 comp rx=0 tx=0).stop 2023-07-29T16:23:31.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 -- 172.21.15.31:0/958754008 shutdown_connections 2023-07-29T16:23:31.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba20370 0x55facba28da0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba1ec60 0x55facba28860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 --2- 172.21.15.31:0/958754008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 -- 172.21.15.31:0/958754008 >> 172.21.15.31:0/958754008 conn(0x55facba14460 msgr2=0x55facba26780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 -- 172.21.15.31:0/958754008 shutdown_connections 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 -- 172.21.15.31:0/958754008 wait complete. 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 -- start start 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.607+0000 7fb676404b80 1 Processor -- start 2023-07-29T16:23:31.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba2caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba85cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facba2d520 con 0x55facba20370 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facba2d690 con 0x55facba1f5b0 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55facba861f0 con 0x55facba1ec60 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb676404b80 10 client.0 _ll_register_callbacks cb 0x55facb9156a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46572/0 (socket says 172.21.15.31:46572) 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba2caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 -- 172.21.15.31:0/1689081011 learned_addr learned my addr 172.21.15.31:0/1689081011 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb660f9c700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba85cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 -- 172.21.15.31:0/1689081011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 msgr2=0x55facba2caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba2caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 -- 172.21.15.31:0/1689081011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 msgr2=0x55facba85cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba85cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.608+0000 7fb65b7fe700 1 -- 172.21.15.31:0/1689081011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c1a0880 con 0x55facba1f5b0 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb65bfff700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba2caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb65b7fe700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2cfe0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb65000be00 tx=0x7fb65002c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d1740bbaed9fe5f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb63b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb63b7fe700 1 -- 172.21.15.31:0/1689081011 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65001d070 con 0x55facba1f5b0 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb63b7fe700 1 -- 172.21.15.31:0/1689081011 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb650000e40 con 0x55facba1f5b0 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb65001e040 con 0x55facba1f5b0 2023-07-29T16:23:31.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb63b7fe700 1 -- 172.21.15.31:0/1689081011 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb65000bee0 con 0x55facba1f5b0 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb676404b80 10 client.27965 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.609+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55facb7c42d0 con 0x55facba1f5b0 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb63b7fe700 1 -- 172.21.15.31:0/1689081011 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb65001e040 con 0x55facba1f5b0 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb63b7fe700 1 -- 172.21.15.31:0/1689081011 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb65000f920 con 0x55facba1f5b0 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb676404b80 10 client.27965 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb676404b80 10 client.27965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:31.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb676404b80 20 client.27965 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:31.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb676404b80 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba877b0 0x55facba89b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb676404b80 4 client.27965 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:23:31.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55facb968dc0 con 0x55facba877b0 2023-07-29T16:23:31.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.610+0000 7fb65bfff700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba877b0 0x55facba89b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.611+0000 7fb65bfff700 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba877b0 0x55facba89b90 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66758fa9c680f4d6 server_cookie=58d0a19def03f35f in_seq=0 out_seq=0 2023-07-29T16:23:31.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.611+0000 7fb63b7fe700 10 client.27965 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb63b7fe700 1 -- 172.21.15.31:0/1689081011 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55facb968dc0 con 0x55facba877b0 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb63b7fe700 10 client.27965 handle_command_reply: tid=0 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 client.27965 shutdown 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 msgr2=0x55facba2cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2cfe0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb65000be00 tx=0x7fb65002c630 comp rx=0 tx=0).stop 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 shutdown_connections 2023-07-29T16:23:31.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55facba877b0 0x55facba89b90 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb63b7fe700 0 client.27965 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55facba20370 0x55facba85cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55facba1ec60 0x55facba2caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 --2- 172.21.15.31:0/1689081011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55facba1f5b0 0x55facba2cfe0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.612+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 >> 172.21.15.31:0/1689081011 conn(0x55facba14460 msgr2=0x55facba28800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 shutdown_connections 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb676404b80 1 -- 172.21.15.31:0/1689081011 wait complete. 2023-07-29T16:23:31.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb676404b80 20 client.27965 trim_cache size 0 max 16384 2023-07-29T16:23:31.619 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:23:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:23:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "643b8f1a-c166-4ad7-997f-c2db2caed4f1", 2023-07-29T16:23:31.619 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:23:31.619 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:23:31.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb6440a0810 msgr2=0x7fb6440a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb6629ff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb6440a0810 0x7fb6440a2cc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb64c005950 tx=0x7fb64c009260 comp rx=0 tx=0).stop 2023-07-29T16:23:31.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101eb0 msgr2=0x7fb65c073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.613+0000 7fb6629ff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101eb0 0x7fb65c073680 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fb65400efc0 tx=0x7fb65400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 shutdown_connections 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb6440a0810 0x7fb6440a2cc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb65c101eb0 0x7fb65c073680 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb65c100330 0x7fb65c074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 --2- 172.21.15.31:0/3564299581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb65c101520 0x7fb65c073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 >> 172.21.15.31:0/3564299581 conn(0x7fb65c0fb970 msgr2=0x7fb65c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 shutdown_connections 2023-07-29T16:23:31.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.614+0000 7fb6629ff700 1 -- 172.21.15.31:0/3564299581 wait complete. 2023-07-29T16:23:31.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:23:31.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2937302748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978101510 msgr2=0x7fa978101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2937302748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978101510 0x7fa978101960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009990 tx=0x7fa96c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2937302748 shutdown_connections 2023-07-29T16:23:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2937302748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa978101ea0 0x7fa978108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2937302748 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978100320 0x7fa978100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2937302748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978101510 0x7fa978101960 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2937302748 >> 172.21.15.31:0/2937302748 conn(0x7fa9780fb920 msgr2=0x7fa9780fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2937302748 shutdown_connections 2023-07-29T16:23:31.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.876+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2937302748 wait complete. 2023-07-29T16:23:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.877+0000 7fa97e3ff700 1 Processor -- start 2023-07-29T16:23:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.877+0000 7fa97e3ff700 1 -- start start 2023-07-29T16:23:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97e3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 0x7fa9781982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97e3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978101510 0x7fa978198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa978101ea0 0x7fa97819f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97e3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97819fe60 con 0x7fa978101510 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97e3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97819ffa0 con 0x7fa978100320 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9781a0130 con 0x7fa978101ea0 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 0x7fa9781982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 0x7fa9781982e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46586/0 (socket says 172.21.15.31:46586) 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978101510 0x7fa978198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 -- 172.21.15.31:0/2190606250 learned_addr learned my addr 172.21.15.31:0/2190606250 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa97c99c700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa978101ea0 0x7fa97819f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 -- 172.21.15.31:0/2190606250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa978101ea0 msgr2=0x7fa97819f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa978101ea0 0x7fa97819f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 -- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978101510 msgr2=0x7fa978198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978101510 0x7fa978198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.878+0000 7fa977fff700 1 -- 172.21.15.31:0/2190606250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c0095f0 con 0x7fa978100320 2023-07-29T16:23:31.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa9777fe700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978101510 0x7fa978198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:31.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa977fff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 0x7fa9781982e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa96800e970 tx=0x7fa96800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa9757fa700 1 -- 172.21.15.31:0/2190606250 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968009790 con 0x7fa978100320 2023-07-29T16:23:31.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 524574 8.90 MB/sec execute 196 sec latency 15.406 ms 2023-07-29T16:23:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa9757fa700 1 -- 172.21.15.31:0/2190606250 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa96800c7e0 con 0x7fa978100320 2023-07-29T16:23:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9781a0410 con 0x7fa978100320 2023-07-29T16:23:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa9757fa700 1 -- 172.21.15.31:0/2190606250 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968003c70 con 0x7fa978100320 2023-07-29T16:23:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.879+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9781a0970 con 0x7fa978100320 2023-07-29T16:23:31.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.883+0000 7fa9757fa700 1 -- 172.21.15.31:0/2190606250 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa96800cc50 con 0x7fa978100320 2023-07-29T16:23:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.884+0000 7fa991c5ab80 1 Processor -- start 2023-07-29T16:23:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.884+0000 7fa991c5ab80 1 -- start start 2023-07-29T16:23:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa9757fa700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9600a0810 0x7fa9600a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c68aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ab530 0x5561c68abd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68ac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c6802a10 con 0x5561c68ab530 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c68003d0 con 0x5561c68aabe0 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c67fbcc0 con 0x5561c68ac2f0 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa9757fa700 1 -- 172.21.15.31:0/2190606250 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa968014070 con 0x7fa978100320 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa97c99c700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9600a0810 0x7fa9600a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa977fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c68aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ab530 0x5561c68abd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa977fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c68aaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46598/0 (socket says 172.21.15.31:46598) 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa977fff700 1 -- 172.21.15.31:0/2460064544 learned_addr learned my addr 172.21.15.31:0/2460064544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.885+0000 7fa97c99c700 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa977fff700 1 -- 172.21.15.31:0/2460064544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 msgr2=0x5561c68ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa977fff700 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68ac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa977fff700 1 -- 172.21.15.31:0/2460064544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ab530 msgr2=0x5561c68abd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa977fff700 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ab530 0x5561c68abd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa977fff700 1 -- 172.21.15.31:0/2460064544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561c6629de0 con 0x5561c68aabe0 2023-07-29T16:23:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa977fff700 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c68aaff0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a1320 tx=0x7fa96800cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf1cf8b311c7a45 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa97c99c700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9600a0810 0x7fa9600a2cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa970009ec0 tx=0x7fa9700092f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa95e7fc700 1 -- 172.21.15.31:0/2460064544 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa968076e70 con 0x5561c68aabe0 2023-07-29T16:23:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa95e7fc700 1 -- 172.21.15.31:0/2460064544 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa968076c90 con 0x5561c68aabe0 2023-07-29T16:23:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.886+0000 7fa95e7fc700 1 -- 172.21.15.31:0/2460064544 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680a1850 con 0x5561c68aabe0 2023-07-29T16:23:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 -- 172.21.15.31:0/2460064544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 msgr2=0x5561c68aaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c68aaff0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa9680a1320 tx=0x7fa96800cf50 comp rx=0 tx=0).stop 2023-07-29T16:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 -- 172.21.15.31:0/2460064544 shutdown_connections 2023-07-29T16:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68ac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c68aaff0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/2460064544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ab530 0x5561c68abd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 -- 172.21.15.31:0/2460064544 >> 172.21.15.31:0/2460064544 conn(0x5561c68a03e0 msgr2=0x5561c68a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 -- 172.21.15.31:0/2460064544 shutdown_connections 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.887+0000 7fa991c5ab80 1 -- 172.21.15.31:0/2460064544 wait complete. 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 -- start start 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 Processor -- start 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 0x5561c68b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68bbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c6798510 con 0x5561c68aabe0 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c67981b0 con 0x5561c68ab530 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.888+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c667f920 con 0x5561c68ac2f0 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa991c5ab80 10 client.0 _ll_register_callbacks cb 0x5561c67a1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 0x5561c68b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa97c99c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68bbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa97c99c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68bbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57358/0 (socket says 172.21.15.31:57358) 2023-07-29T16:23:31.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 0x5561c68b4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46608/0 (socket says 172.21.15.31:46608) 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa97c99c700 1 -- 172.21.15.31:0/1081830313 learned_addr learned my addr 172.21.15.31:0/1081830313 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 -- 172.21.15.31:0/1081830313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 msgr2=0x5561c68bbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68bbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 -- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 msgr2=0x5561c68b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 0x5561c68b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 -- 172.21.15.31:0/1081830313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9781a0970 con 0x5561c68aabe0 2023-07-29T16:23:31.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa9777fe700 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 0x5561c68b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa977fff700 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b47b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa968076c20 tx=0x7fa9680950d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a4e442fa78981e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.889+0000 7fa9577fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa9577fe700 1 -- 172.21.15.31:0/1081830313 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680a1850 con 0x5561c68aabe0 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa9577fe700 1 -- 172.21.15.31:0/1081830313 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa968099400 con 0x5561c68aabe0 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561c6826d10 con 0x5561c68aabe0 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa9577fe700 1 -- 172.21.15.31:0/1081830313 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9680782b0 con 0x5561c68aabe0 2023-07-29T16:23:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa991c5ab80 10 client.17931 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5561c6612c60 con 0x5561c68aabe0 2023-07-29T16:23:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa9577fe700 1 -- 172.21.15.31:0/1081830313 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9680c7080 con 0x5561c68aabe0 2023-07-29T16:23:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa9577fe700 1 -- 172.21.15.31:0/1081830313 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa968098980 con 0x5561c68aabe0 2023-07-29T16:23:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa991c5ab80 10 client.17931 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa991c5ab80 10 client.17931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.890+0000 7fa991c5ab80 20 client.17931 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.891+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c68c01c0 0x5561c68c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.891+0000 7fa991c5ab80 4 client.17931 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.891+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5561c6879d70 con 0x5561c68c01c0 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.891+0000 7fa9777fe700 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c68c01c0 0x5561c68c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.891+0000 7fa9777fe700 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c68c01c0 0x5561c68c25d0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef1e27308cc8ecee server_cookie=1ec6c0e09a2a9ac in_seq=0 out_seq=0 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.891+0000 7fa9577fe700 10 client.17931 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.892+0000 7fa9577fe700 1 -- 172.21.15.31:0/1081830313 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5561c6879d70 con 0x5561c68c01c0 2023-07-29T16:23:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.892+0000 7fa9577fe700 10 client.17931 handle_command_reply: tid=0 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.892+0000 7fa991c5ab80 1 client.17931 shutdown 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.892+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 msgr2=0x5561c68b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.892+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b47b0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fa968076c20 tx=0x7fa9680950d0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 shutdown_connections 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c68c01c0 0x5561c68c25d0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ac2f0 0x5561c68bbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa9577fe700 0 client.17931 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b47b0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1081830313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ab530 0x5561c68b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 >> 172.21.15.31:0/1081830313 conn(0x5561c68a03e0 msgr2=0x5561c68a5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 shutdown_connections 2023-07-29T16:23:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1081830313 wait complete. 2023-07-29T16:23:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.893+0000 7fa991c5ab80 20 client.17931 trim_cache size 0 max 16384 2023-07-29T16:23:31.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.912+0000 7fa991c5ab80 1 Processor -- start 2023-07-29T16:23:31.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.912+0000 7fa991c5ab80 1 -- start start 2023-07-29T16:23:31.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c68af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ac2f0 0x5561c68af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c689ff10 con 0x5561c68aabe0 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c68b2510 con 0x5561c68ac2f0 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c68abb50 con 0x5561c68ab530 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa977fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c68af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa977fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51578/0 (socket says 172.21.15.31:51578) 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa977fff700 1 -- 172.21.15.31:0/1366323873 learned_addr learned my addr 172.21.15.31:0/1366323873 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa97c99c700 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ac2f0 0x5561c68af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa9777fe700 1 -- 172.21.15.31:0/1366323873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ac2f0 msgr2=0x5561c68af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa9777fe700 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ac2f0 0x5561c68af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa9777fe700 1 -- 172.21.15.31:0/1366323873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 msgr2=0x5561c68b9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa9777fe700 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa9777fe700 1 -- 172.21.15.31:0/1366323873 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561c6650250 con 0x5561c68ab530 2023-07-29T16:23:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa977fff700 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:31.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.913+0000 7fa97c99c700 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ac2f0 0x5561c68af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:31.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.914+0000 7fa9777fe700 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c68af290 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa96c000c00 tx=0x7fa96c005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68edee8715d7bbe6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.915+0000 7fa95effd700 1 -- 172.21.15.31:0/1366323873 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c02a070 con 0x5561c68ab530 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.915+0000 7fa95effd700 1 -- 172.21.15.31:0/1366323873 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa96c02e070 con 0x5561c68ab530 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.915+0000 7fa95effd700 1 -- 172.21.15.31:0/1366323873 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c01d040 con 0x5561c68ab530 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1366323873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 msgr2=0x5561c68af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c68af290 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa96c000c00 tx=0x7fa96c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1366323873 shutdown_connections 2023-07-29T16:23:31.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68ac2f0 0x5561c68af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68aabe0 0x5561c68b9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/1366323873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c68af290 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1366323873 >> 172.21.15.31:0/1366323873 conn(0x5561c68a03e0 msgr2=0x5561c68b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1366323873 shutdown_connections 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.916+0000 7fa991c5ab80 1 -- 172.21.15.31:0/1366323873 wait complete. 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 -- start start 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 Processor -- start 2023-07-29T16:23:31.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c6909ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c690a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ac2f0 0x5561c6911570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c6911ab0 con 0x5561c68ac2f0 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c6911bf0 con 0x5561c68aabe0 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561c6911d80 con 0x5561c68ab530 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.917+0000 7fa991c5ab80 10 client.0 _ll_register_callbacks cb 0x5561c67a1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:31.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa977fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c6909ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c690a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa977fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c6909ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46630/0 (socket says 172.21.15.31:46630) 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa977fff700 1 -- 172.21.15.31:0/30311087 learned_addr learned my addr 172.21.15.31:0/30311087 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c690a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57376/0 (socket says 172.21.15.31:57376) 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa97c99c700 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ac2f0 0x5561c6911570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 -- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 msgr2=0x5561c6909ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c6909ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 -- 172.21.15.31:0/30311087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ac2f0 msgr2=0x5561c6911570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ac2f0 0x5561c6911570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.918+0000 7fa9777fe700 1 -- 172.21.15.31:0/30311087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5561c6612c60 con 0x5561c68ab530 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9777fe700 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c690a530 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00b620 tx=0x7fa96c00fda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2b0a008c8f64635 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9577fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9577fe700 1 -- 172.21.15.31:0/30311087 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa96c02a050 con 0x5561c68ab530 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa968076c90 con 0x5561c68ab530 2023-07-29T16:23:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9577fe700 1 -- 172.21.15.31:0/30311087 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa96c0055a0 con 0x5561c68ab530 2023-07-29T16:23:31.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9577fe700 1 -- 172.21.15.31:0/30311087 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa96c01d040 con 0x5561c68ab530 2023-07-29T16:23:31.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa991c5ab80 10 client.27994 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:31.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fa97001a900 con 0x5561c68ab530 2023-07-29T16:23:31.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9577fe700 1 -- 172.21.15.31:0/30311087 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa96c042080 con 0x5561c68ab530 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa9577fe700 1 -- 172.21.15.31:0/30311087 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa96c033b30 con 0x5561c68ab530 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa991c5ab80 10 client.27994 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa991c5ab80 10 client.27994 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.919+0000 7fa991c5ab80 20 client.27994 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.920+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c69132c0 0x5561c69156a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.920+0000 7fa991c5ab80 4 client.27994 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:23:31.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.920+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5561c67f4d40 con 0x5561c69132c0 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.920+0000 7fa977fff700 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c69132c0 0x5561c69156a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.920+0000 7fa977fff700 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c69132c0 0x5561c69156a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7180d8d23e342105 server_cookie=a9d2e9eb84addbf6 in_seq=0 out_seq=0 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.920+0000 7fa9577fe700 10 client.27994 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.923+0000 7fa9577fe700 1 -- 172.21.15.31:0/30311087 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5561c67f4d40 con 0x5561c69132c0 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.923+0000 7fa9577fe700 10 client.27994 handle_command_reply: tid=0 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.923+0000 7fa991c5ab80 1 client.27994 shutdown 2023-07-29T16:23:31.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 msgr2=0x5561c690a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c690a530 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa96c00b620 tx=0x7fa96c00fda0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 shutdown_connections 2023-07-29T16:23:31.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5561c69132c0 0x5561c69156a0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5561c68ac2f0 0x5561c6911570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5561c68aabe0 0x5561c6909ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa9577fe700 0 client.27994 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 --2- 172.21.15.31:0/30311087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561c68ab530 0x5561c690a530 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 >> 172.21.15.31:0/30311087 conn(0x5561c68a03e0 msgr2=0x5561c68af290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 shutdown_connections 2023-07-29T16:23:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 1 -- 172.21.15.31:0/30311087 wait complete. 2023-07-29T16:23:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.924+0000 7fa991c5ab80 20 client.27994 trim_cache size 0 max 16384 2023-07-29T16:23:31.929 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:23:31.929 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:23:31.929 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:23:31.929 INFO:teuthology.orchestra.run.smithi031.stdout: "643b8f1a-c166-4ad7-997f-c2db2caed4f1": { 2023-07-29T16:23:31.929 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:23:31.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "643b8f1a-c166-4ad7-997f-c2db2caed4f1", 2023-07-29T16:23:31.930 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:23:31.930 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:23:31.930 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:23:31.930 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:23:31.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9600a0810 msgr2=0x7fa9600a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9600a0810 0x7fa9600a2cc0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa970009ec0 tx=0x7fa9700092f0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 msgr2=0x7fa9781982e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 0x7fa9781982e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fa96800e970 tx=0x7fa96800ec80 comp rx=0 tx=0).stop 2023-07-29T16:23:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 shutdown_connections 2023-07-29T16:23:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9600a0810 0x7fa9600a2cc0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa978101ea0 0x7fa97819f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa978100320 0x7fa9781982e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 --2- 172.21.15.31:0/2190606250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa978101510 0x7fa978198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:31.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 >> 172.21.15.31:0/2190606250 conn(0x7fa9780fb920 msgr2=0x7fa9780fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:31.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 shutdown_connections 2023-07-29T16:23:31.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:23:31.925+0000 7fa97e3ff700 1 -- 172.21.15.31:0/2190606250 wait complete. 2023-07-29T16:23:31.943 INFO:tasks.cephfs.filesystem:scrub status for tag:643b8f1a-c166-4ad7-997f-c2db2caed4f1 - {'path': '/', 'tag': '643b8f1a-c166-4ad7-997f-c2db2caed4f1', 'options': 'recursive,force'} 2023-07-29T16:23:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: pgmap v612: 97 pgs: 97 active+clean; 109 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 3.1 MiB/s wr, 171 op/s 2023-07-29T16:23:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:23:32.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:23:32.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:23:32.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:32 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: pgmap v612: 97 pgs: 97 active+clean; 109 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 3.1 MiB/s wr, 171 op/s 2023-07-29T16:23:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:23:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:23:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:23:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:32 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 526157 8.91 MB/sec execute 197 sec latency 15.006 ms 2023-07-29T16:23:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: pgmap v612: 97 pgs: 97 active+clean; 109 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 3.1 MiB/s wr, 171 op/s 2023-07-29T16:23:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:23:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:23:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:23:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:32 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:33.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:33 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:23:33.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:33 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:33.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:33 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:23:33.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:33 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:33.877 INFO:tasks.workunit.client.0.smithi031.stdout: 1 527824 8.90 MB/sec execute 198 sec latency 52.958 ms 2023-07-29T16:23:33.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:33 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:23:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:33 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:23:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:34 smithi039 ceph-mon[128858]: pgmap v613: 97 pgs: 97 active+clean; 110 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 3.6 MiB/s wr, 217 op/s 2023-07-29T16:23:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:34 smithi156 ceph-mon[129245]: pgmap v613: 97 pgs: 97 active+clean; 110 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 3.6 MiB/s wr, 217 op/s 2023-07-29T16:23:34.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 529424 8.91 MB/sec execute 199 sec latency 221.277 ms 2023-07-29T16:23:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:34 smithi031 ceph-mon[95504]: pgmap v613: 97 pgs: 97 active+clean; 110 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 3.6 MiB/s wr, 217 op/s 2023-07-29T16:23:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 -- 172.21.15.31:0/578867576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c100350 msgr2=0x7fd91c100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 --2- 172.21.15.31:0/578867576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c100350 0x7fd91c100760 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fd90c009990 tx=0x7fd90c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 -- 172.21.15.31:0/578867576 shutdown_connections 2023-07-29T16:23:35.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 --2- 172.21.15.31:0/578867576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c101ed0 0x7fd91c108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 --2- 172.21.15.31:0/578867576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c100350 0x7fd91c100760 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 --2- 172.21.15.31:0/578867576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 0x7fd91c101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 -- 172.21.15.31:0/578867576 >> 172.21.15.31:0/578867576 conn(0x7fd91c0fb970 msgr2=0x7fd91c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 -- 172.21.15.31:0/578867576 shutdown_connections 2023-07-29T16:23:35.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.752+0000 7fd921dff700 1 -- 172.21.15.31:0/578867576 wait complete. 2023-07-29T16:23:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.754+0000 7fd921dff700 1 Processor -- start 2023-07-29T16:23:35.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.754+0000 7fd921dff700 1 -- start start 2023-07-29T16:23:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.754+0000 7fd921dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 0x7fd91c1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.754+0000 7fd921dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 0x7fd91c1966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.754+0000 7fd921dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 0x7fd91c197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd921dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c197bd0 con 0x7fd91c101540 2023-07-29T16:23:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd921dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c197d40 con 0x7fd91c101ed0 2023-07-29T16:23:35.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd921dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c197f00 con 0x7fd91c100350 2023-07-29T16:23:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 0x7fd91c1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 0x7fd91c197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 0x7fd91c1961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57402/0 (socket says 172.21.15.31:57402) 2023-07-29T16:23:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 0x7fd91c1966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3684404760 learned_addr learned my addr 172.21.15.31:0/3684404760 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:35.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91bfff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 0x7fd91c197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46650/0 (socket says 172.21.15.31:46650) 2023-07-29T16:23:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3684404760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 msgr2=0x7fd91c197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 0x7fd91c197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 msgr2=0x7fd91c1966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 0x7fd91c1966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3684404760 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90c0095f0 con 0x7fd91c100350 2023-07-29T16:23:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91bfff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 0x7fd91c197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.755+0000 7fd91affd700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 0x7fd91c1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.756+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 0x7fd91c1961b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd90c000c00 tx=0x7fd90c0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.756+0000 7fd918ff9700 1 -- 172.21.15.31:0/3684404760 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c01d070 con 0x7fd91c100350 2023-07-29T16:23:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.756+0000 7fd918ff9700 1 -- 172.21.15.31:0/3684404760 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd90c00fb90 con 0x7fd91c100350 2023-07-29T16:23:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.756+0000 7fd918ff9700 1 -- 172.21.15.31:0/3684404760 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0173f0 con 0x7fd91c100350 2023-07-29T16:23:35.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.756+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c198180 con 0x7fd91c100350 2023-07-29T16:23:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.756+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c19e9a0 con 0x7fd91c100350 2023-07-29T16:23:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.760+0000 7fd918ff9700 1 -- 172.21.15.31:0/3684404760 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd90c00fd00 con 0x7fd91c100350 2023-07-29T16:23:35.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.761+0000 7fd93579db80 1 Processor -- start 2023-07-29T16:23:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.761+0000 7fd93579db80 1 -- start start 2023-07-29T16:23:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd918ff9700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9040a0810 0x7fd9040a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd918ff9700 1 -- 172.21.15.31:0/3684404760 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd90c0ccc80 con 0x7fd91c100350 2023-07-29T16:23:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd93579db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338398bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd91bfff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9040a0810 0x7fd9040a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd93579db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd93579db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338398d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338398bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd93579db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633838df9b0 con 0x56338398c200 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd93579db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633838dd370 con 0x5633837e36b0 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd93579db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633838d8c60 con 0x56338398d100 2023-07-29T16:23:35.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338398bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46654/0 (socket says 172.21.15.31:46654) 2023-07-29T16:23:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd91b7fe700 1 -- 172.21.15.31:0/2946417074 learned_addr learned my addr 172.21.15.31:0/2946417074 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd91affd700 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.762+0000 7fd91bfff700 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338398d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91b7fe700 1 -- 172.21.15.31:0/2946417074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 msgr2=0x56338398d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338398d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91b7fe700 1 -- 172.21.15.31:0/2946417074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 msgr2=0x56338398cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91b7fe700 1 -- 172.21.15.31:0/2946417074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563383706d80 con 0x5633837e36b0 2023-07-29T16:23:35.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91bfff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9040a0810 0x7fd9040a2cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd9140076f0 tx=0x7fd914007ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338398bcc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd90c09a340 tx=0x7fd90c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f599970d16b51ed1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd901ffb700 1 -- 172.21.15.31:0/2946417074 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c09e3e0 con 0x5633837e36b0 2023-07-29T16:23:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd901ffb700 1 -- 172.21.15.31:0/2946417074 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd90c0a2770 con 0x5633837e36b0 2023-07-29T16:23:35.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.763+0000 7fd901ffb700 1 -- 172.21.15.31:0/2946417074 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c099b00 con 0x5633837e36b0 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 -- 172.21.15.31:0/2946417074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 msgr2=0x56338398bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338398bcc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd90c09a340 tx=0x7fd90c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 -- 172.21.15.31:0/2946417074 shutdown_connections 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338398d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338398bcc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 --2- 172.21.15.31:0/2946417074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 -- 172.21.15.31:0/2946417074 >> 172.21.15.31:0/2946417074 conn(0x56338397cfd0 msgr2=0x563383982300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 -- 172.21.15.31:0/2946417074 shutdown_connections 2023-07-29T16:23:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.764+0000 7fd93579db80 1 -- 172.21.15.31:0/2946417074 wait complete. 2023-07-29T16:23:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.765+0000 7fd93579db80 1 -- start start 2023-07-29T16:23:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.765+0000 7fd93579db80 1 Processor -- start 2023-07-29T16:23:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x563383994f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383995490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338399c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633838754b0 con 0x56338398c200 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563383875150 con 0x5633837e36b0 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338375c8c0 con 0x56338398d100 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd93579db80 10 client.0 _ll_register_callbacks cb 0x56338387e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x563383994f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x563383994f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46666/0 (socket says 172.21.15.31:46666) 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3838666548 learned_addr learned my addr 172.21.15.31:0/3838666548 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91bfff700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338399c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91affd700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383995490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3838666548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 msgr2=0x56338399c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338399c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3838666548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 msgr2=0x563383995490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383995490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.766+0000 7fd91b7fe700 1 -- 172.21.15.31:0/3838666548 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c19e9a0 con 0x5633837e36b0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x563383994f50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0a2240 tx=0x7fd90c0094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d987776f3611a3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd8faffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd8faffd700 1 -- 172.21.15.31:0/3838666548 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0a22e0 con 0x5633837e36b0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd8faffd700 1 -- 172.21.15.31:0/3838666548 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd90c0a2440 con 0x5633837e36b0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563383903cb0 con 0x5633837e36b0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd8faffd700 1 -- 172.21.15.31:0/3838666548 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd90c0cab80 con 0x5633837e36b0 2023-07-29T16:23:35.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd93579db80 10 client.27995 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.767+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5633836efc00 con 0x5633837e36b0 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd8faffd700 1 -- 172.21.15.31:0/3838666548 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563383903cb0 con 0x5633837e36b0 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd8faffd700 1 -- 172.21.15.31:0/3838666548 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd90c099310 con 0x5633837e36b0 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd93579db80 10 client.27995 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd93579db80 10 client.27995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd93579db80 20 client.27995 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd93579db80 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839a0820 0x5633839a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd93579db80 4 client.27995 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563383956d10 con 0x5633839a0820 2023-07-29T16:23:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.768+0000 7fd91affd700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839a0820 0x5633839a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.769+0000 7fd91affd700 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839a0820 0x5633839a2c30 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1773d2f50c9c175 server_cookie=337b96549fa76a08 in_seq=0 out_seq=0 2023-07-29T16:23:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.769+0000 7fd8faffd700 10 client.27995 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.770+0000 7fd8faffd700 1 -- 172.21.15.31:0/3838666548 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563383956d10 con 0x5633839a0820 2023-07-29T16:23:35.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.770+0000 7fd8faffd700 10 client.27995 handle_command_reply: tid=0 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.770+0000 7fd93579db80 1 client.27995 shutdown 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 msgr2=0x563383994f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x563383994f50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0a2240 tx=0x7fd90c0094b0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 shutdown_connections 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839a0820 0x5633839a2c30 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x56338399c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x563383994f50 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd8faffd700 0 client.27995 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 --2- 172.21.15.31:0/3838666548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383995490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 >> 172.21.15.31:0/3838666548 conn(0x56338397cfd0 msgr2=0x563383981a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 shutdown_connections 2023-07-29T16:23:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 1 -- 172.21.15.31:0/3838666548 wait complete. 2023-07-29T16:23:35.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.771+0000 7fd93579db80 20 client.27995 trim_cache size 0 max 16384 2023-07-29T16:23:35.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 Processor -- start 2023-07-29T16:23:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 -- start start 2023-07-29T16:23:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338399a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383994d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563383996140 con 0x56338398c200 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633839830a0 con 0x5633837e36b0 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd93579db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563383995fa0 con 0x56338398d100 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338399a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338399a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46676/0 (socket says 172.21.15.31:46676) 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd91b7fe700 1 -- 172.21.15.31:0/1146799251 learned_addr learned my addr 172.21.15.31:0/1146799251 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd91affd700 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.787+0000 7fd91bfff700 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383994d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91b7fe700 1 -- 172.21.15.31:0/1146799251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 msgr2=0x563383994d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383994d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91b7fe700 1 -- 172.21.15.31:0/1146799251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 msgr2=0x56338398d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91b7fe700 1 -- 172.21.15.31:0/1146799251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56338372d1f0 con 0x5633837e36b0 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91bfff700 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383994d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338399a450 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0996a0 tx=0x7fd90c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcef1190f31c1c7f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:35.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd9027fc700 1 -- 172.21.15.31:0/1146799251 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0ca580 con 0x5633837e36b0 2023-07-29T16:23:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd9027fc700 1 -- 172.21.15.31:0/1146799251 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd90c0c4800 con 0x5633837e36b0 2023-07-29T16:23:35.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.788+0000 7fd9027fc700 1 -- 172.21.15.31:0/1146799251 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0a2540 con 0x5633837e36b0 2023-07-29T16:23:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 -- 172.21.15.31:0/1146799251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 msgr2=0x56338399a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338399a450 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0996a0 tx=0x7fd90c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 -- 172.21.15.31:0/1146799251 shutdown_connections 2023-07-29T16:23:35.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383994d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x56338399a450 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 --2- 172.21.15.31:0/1146799251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x56338398d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 -- 172.21.15.31:0/1146799251 >> 172.21.15.31:0/1146799251 conn(0x56338397cfd0 msgr2=0x56338398fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 -- 172.21.15.31:0/1146799251 shutdown_connections 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.789+0000 7fd93579db80 1 -- 172.21.15.31:0/1146799251 wait complete. 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 -- start start 2023-07-29T16:23:35.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 Processor -- start 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x5633839983d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383998910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383999920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563383999e60 con 0x56338398c200 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563383999fd0 con 0x5633837e36b0 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56338399a190 con 0x56338398d100 2023-07-29T16:23:35.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd93579db80 10 client.0 _ll_register_callbacks cb 0x56338387e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.790+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x5633839983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x5633839983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46692/0 (socket says 172.21.15.31:46692) 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 -- 172.21.15.31:0/597133204 learned_addr learned my addr 172.21.15.31:0/597133204 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91affd700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383998910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91bfff700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383999920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 -- 172.21.15.31:0/597133204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 msgr2=0x563383999920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383999920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 -- 172.21.15.31:0/597133204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 msgr2=0x563383998910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383998910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 -- 172.21.15.31:0/597133204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633836efc00 con 0x5633837e36b0 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd91b7fe700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x5633839983d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c4990 tx=0x7fd90c0997f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3277512ba53f993 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd8faffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd8faffd700 1 -- 172.21.15.31:0/597133204 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c0cad80 con 0x5633837e36b0 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd90c0a2770 con 0x5633837e36b0 2023-07-29T16:23:35.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.791+0000 7fd8faffd700 1 -- 172.21.15.31:0/597133204 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd90c0ca240 con 0x5633837e36b0 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd8faffd700 1 -- 172.21.15.31:0/597133204 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd90c0ca400 con 0x5633837e36b0 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 10 client.28001 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563383705250 con 0x5633837e36b0 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd8faffd700 1 -- 172.21.15.31:0/597133204 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd90c0a6080 con 0x5633837e36b0 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd8faffd700 1 -- 172.21.15.31:0/597133204 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd90c09b190 con 0x5633837e36b0 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 10 client.28001 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:35.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 10 client.28001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 20 client.28001 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839f2ed0 0x5633839f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 4 client.28001 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5633838d1ce0 con 0x5633839f2ed0 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.792+0000 7fd91affd700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839f2ed0 0x5633839f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.793+0000 7fd91affd700 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839f2ed0 0x5633839f52b0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33e13ffd42dbf04b server_cookie=1ac8f16f91a235db in_seq=0 out_seq=0 2023-07-29T16:23:35.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.793+0000 7fd8faffd700 10 client.28001 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:35.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 530513 8.89 MB/sec execute 200 sec latency 45.742 ms 2023-07-29T16:23:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd8faffd700 1 -- 172.21.15.31:0/597133204 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5633838d1ce0 con 0x5633839f2ed0 2023-07-29T16:23:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd8faffd700 10 client.28001 handle_command_reply: tid=0 2023-07-29T16:23:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 client.28001 shutdown 2023-07-29T16:23:35.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 msgr2=0x5633839983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x5633839983d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0c4990 tx=0x7fd90c0997f0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 shutdown_connections 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633839f2ed0 0x5633839f52b0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56338398d100 0x563383999920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633837e36b0 0x5633839983d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd8faffd700 0 client.28001 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:35.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 --2- 172.21.15.31:0/597133204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56338398c200 0x563383998910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 >> 172.21.15.31:0/597133204 conn(0x56338397cfd0 msgr2=0x56338398fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 shutdown_connections 2023-07-29T16:23:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.881+0000 7fd93579db80 1 -- 172.21.15.31:0/597133204 wait complete. 2023-07-29T16:23:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd93579db80 20 client.28001 trim_cache size 0 max 16384 2023-07-29T16:23:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:23:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:23:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:23:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:23:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9040a0810 msgr2=0x7fd9040a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9040a0810 0x7fd9040a2cc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd9140076f0 tx=0x7fd914007ab0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 msgr2=0x7fd91c1961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 0x7fd91c1961b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd90c000c00 tx=0x7fd90c0057c0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 shutdown_connections 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd9040a0810 0x7fd9040a2cc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd91c101ed0 0x7fd91c197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd91c100350 0x7fd91c1961b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 --2- 172.21.15.31:0/3684404760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd91c101540 0x7fd91c1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 >> 172.21.15.31:0/3684404760 conn(0x7fd91c0fb970 msgr2=0x7fd91c104a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.882+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 shutdown_connections 2023-07-29T16:23:35.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:35.883+0000 7fd921dff700 1 -- 172.21.15.31:0/3684404760 wait complete. 2023-07-29T16:23:36.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:36 smithi156 ceph-mon[129245]: pgmap v614: 97 pgs: 97 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 3.7 MiB/s wr, 223 op/s 2023-07-29T16:23:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:36 smithi039 ceph-mon[128858]: pgmap v614: 97 pgs: 97 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 3.7 MiB/s wr, 223 op/s 2023-07-29T16:23:36.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 532830 8.92 MB/sec execute 201 sec latency 42.638 ms 2023-07-29T16:23:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:36 smithi031 ceph-mon[95504]: pgmap v614: 97 pgs: 97 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 3.7 MiB/s wr, 223 op/s 2023-07-29T16:23:37.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 534042 8.89 MB/sec execute 202 sec latency 11.599 ms 2023-07-29T16:23:38.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:37 smithi031 ceph-mon[95504]: pgmap v615: 97 pgs: 97 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 3.2 MiB/s wr, 217 op/s 2023-07-29T16:23:38.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:37 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:23:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:37 smithi039 ceph-mon[128858]: pgmap v615: 97 pgs: 97 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 3.2 MiB/s wr, 217 op/s 2023-07-29T16:23:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:37 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:23:38.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:37 smithi156 ceph-mon[129245]: pgmap v615: 97 pgs: 97 active+clean; 114 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 3.2 MiB/s wr, 217 op/s 2023-07-29T16:23:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:37 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:23:38.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 536516 8.93 MB/sec execute 203 sec latency 13.473 ms 2023-07-29T16:23:39.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:38 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:23:39.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:38 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:23:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:38 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:23:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:38 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:23:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:38 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:23:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:38 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:23:39.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 537594 8.90 MB/sec execute 204 sec latency 57.851 ms 2023-07-29T16:23:40.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:39 smithi031 ceph-mon[95504]: pgmap v616: 97 pgs: 97 active+clean; 110 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.5 MiB/s wr, 313 op/s 2023-07-29T16:23:40.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:39 smithi039 ceph-mon[128858]: pgmap v616: 97 pgs: 97 active+clean; 110 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.5 MiB/s wr, 313 op/s 2023-07-29T16:23:40.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:39 smithi156 ceph-mon[129245]: pgmap v616: 97 pgs: 97 active+clean; 110 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 3.5 MiB/s wr, 313 op/s 2023-07-29T16:23:40.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 540107 8.93 MB/sec execute 205 sec latency 13.464 ms 2023-07-29T16:23:41.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 541404 8.92 MB/sec execute 206 sec latency 43.659 ms 2023-07-29T16:23:42.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:42 smithi039 ceph-mon[128858]: pgmap v617: 97 pgs: 97 active+clean; 110 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.0 MiB/s wr, 230 op/s 2023-07-29T16:23:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:42 smithi156 ceph-mon[129245]: pgmap v617: 97 pgs: 97 active+clean; 110 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.0 MiB/s wr, 230 op/s 2023-07-29T16:23:42.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 543733 8.94 MB/sec execute 207 sec latency 17.744 ms 2023-07-29T16:23:42.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:42 smithi031 ceph-mon[95504]: pgmap v617: 97 pgs: 97 active+clean; 110 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.0 MiB/s wr, 230 op/s 2023-07-29T16:23:43.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 545160 8.93 MB/sec execute 208 sec latency 12.331 ms 2023-07-29T16:23:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:44 smithi039 ceph-mon[128858]: pgmap v618: 97 pgs: 97 active+clean; 112 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2023-07-29T16:23:44.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:44 smithi156 ceph-mon[129245]: pgmap v618: 97 pgs: 97 active+clean; 112 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2023-07-29T16:23:44.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 547290 8.95 MB/sec execute 209 sec latency 48.277 ms 2023-07-29T16:23:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:44 smithi031 ceph-mon[95504]: pgmap v618: 97 pgs: 97 active+clean; 112 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2023-07-29T16:23:45.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 548480 8.93 MB/sec execute 210 sec latency 98.695 ms 2023-07-29T16:23:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:46 smithi039 ceph-mon[128858]: pgmap v619: 97 pgs: 97 active+clean; 113 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.6 MiB/s wr, 269 op/s 2023-07-29T16:23:46.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:23:46.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:23:46.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:23:46.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:23:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:46 smithi156 ceph-mon[129245]: pgmap v619: 97 pgs: 97 active+clean; 113 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.6 MiB/s wr, 269 op/s 2023-07-29T16:23:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:23:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:23:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:23:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:23:46.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 550618 8.95 MB/sec execute 211 sec latency 50.471 ms 2023-07-29T16:23:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:46 smithi031 ceph-mon[95504]: pgmap v619: 97 pgs: 97 active+clean; 113 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.6 MiB/s wr, 269 op/s 2023-07-29T16:23:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:23:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:23:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:23:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:23:47.878 INFO:tasks.workunit.client.0.smithi031.stdout: 1 551998 8.94 MB/sec execute 212 sec latency 16.924 ms 2023-07-29T16:23:48.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:47 smithi031 ceph-mon[95504]: pgmap v620: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.4 MiB/s wr, 265 op/s 2023-07-29T16:23:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:47 smithi039 ceph-mon[128858]: pgmap v620: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.4 MiB/s wr, 265 op/s 2023-07-29T16:23:48.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:47 smithi156 ceph-mon[129245]: pgmap v620: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.4 MiB/s wr, 265 op/s 2023-07-29T16:23:48.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 554357 8.96 MB/sec execute 213 sec latency 14.367 ms 2023-07-29T16:23:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 -- 172.21.15.31:0/1292077857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc100340 msgr2=0x7fd0dc100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/1292077857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc100340 0x7fd0dc100750 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009990 tx=0x7fd0d8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 -- 172.21.15.31:0/1292077857 shutdown_connections 2023-07-29T16:23:49.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/1292077857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101ec0 0x7fd0dc108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/1292077857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc100340 0x7fd0dc100750 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/1292077857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc101530 0x7fd0dc101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 -- 172.21.15.31:0/1292077857 >> 172.21.15.31:0/1292077857 conn(0x7fd0dc0fb920 msgr2=0x7fd0dc0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 -- 172.21.15.31:0/1292077857 shutdown_connections 2023-07-29T16:23:49.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.036+0000 7fd0e43ff700 1 -- 172.21.15.31:0/1292077857 wait complete. 2023-07-29T16:23:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.037+0000 7fd0e43ff700 1 Processor -- start 2023-07-29T16:23:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.037+0000 7fd0e43ff700 1 -- start start 2023-07-29T16:23:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e43ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc100340 0x7fd0dc073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e43ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 0x7fd0dc073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e43ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc101ec0 0x7fd0dc073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e43ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc1a01b0 con 0x7fd0dc100340 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e43ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc1a02f0 con 0x7fd0dc101ec0 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e43ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0dc1a0430 con 0x7fd0dc101530 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 0x7fd0dc073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc100340 0x7fd0dc073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 0x7fd0dc073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49110/0 (socket says 172.21.15.31:49110) 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 -- 172.21.15.31:0/2518433758 learned_addr learned my addr 172.21.15.31:0/2518433758 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:49.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e299c700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc101ec0 0x7fd0dc073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 -- 172.21.15.31:0/2518433758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc101ec0 msgr2=0x7fd0dc073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc101ec0 0x7fd0dc073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 -- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc100340 msgr2=0x7fd0dc073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc100340 0x7fd0dc073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.038+0000 7fd0e199a700 1 -- 172.21.15.31:0/2518433758 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80095f0 con 0x7fd0dc101530 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.039+0000 7fd0e199a700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 0x7fd0dc073680 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00e9a0 tx=0x7fd0cc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:49.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.039+0000 7fd0d37fe700 1 -- 172.21.15.31:0/2518433758 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc009790 con 0x7fd0dc101530 2023-07-29T16:23:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.039+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0dc1a06c0 con 0x7fd0dc101530 2023-07-29T16:23:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.039+0000 7fd0d37fe700 1 -- 172.21.15.31:0/2518433758 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0cc010430 con 0x7fd0dc101530 2023-07-29T16:23:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.039+0000 7fd0d37fe700 1 -- 172.21.15.31:0/2518433758 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc018680 con 0x7fd0dc101530 2023-07-29T16:23:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.039+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc1a0bd0 con 0x7fd0dc101530 2023-07-29T16:23:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.042+0000 7fd0d37fe700 1 -- 172.21.15.31:0/2518433758 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd0cc0105a0 con 0x7fd0dc101530 2023-07-29T16:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.044+0000 7fd0d37fe700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0c80a0790 0x7fd0c80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.044+0000 7fd0f7d46b80 1 Processor -- start 2023-07-29T16:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.044+0000 7fd0d37fe700 1 -- 172.21.15.31:0/2518433758 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd0cc014070 con 0x7fd0dc101530 2023-07-29T16:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.044+0000 7fd0f7d46b80 1 -- start start 2023-07-29T16:23:49.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.044+0000 7fd0e219b700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0c80a0790 0x7fd0c80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.044+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x56105894c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 0x56105894d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x56105894e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610588a49b0 con 0x56105894dcf0 2023-07-29T16:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610588a2370 con 0x56105894cdf0 2023-07-29T16:23:49.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105889dc60 con 0x5610587a86b0 2023-07-29T16:23:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 0x56105894d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 0x56105894d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51632/0 (socket says 172.21.15.31:51632) 2023-07-29T16:23:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 -- 172.21.15.31:0/2988209296 learned_addr learned my addr 172.21.15.31:0/2988209296 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e219b700 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x56105894c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e299c700 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x56105894e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 -- 172.21.15.31:0/2988209296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 msgr2=0x56105894c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x56105894c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 -- 172.21.15.31:0/2988209296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 msgr2=0x56105894e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x56105894e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 -- 172.21.15.31:0/2988209296 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610586cbd80 con 0x56105894cdf0 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e299c700 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x56105894e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e219b700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0c80a0790 0x7fd0c80a2c40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8000c00 tx=0x7fd0d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:49.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.045+0000 7fd0e199a700 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 0x56105894d7b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc099410 tx=0x7fd0cc099090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aafb3c387c730e68 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0d0bfe700 1 -- 172.21.15.31:0/2988209296 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc099500 con 0x56105894cdf0 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0d0bfe700 1 -- 172.21.15.31:0/2988209296 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0cc09d4b0 con 0x56105894cdf0 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0d0bfe700 1 -- 172.21.15.31:0/2988209296 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc098b70 con 0x56105894cdf0 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/2988209296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 msgr2=0x56105894d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 0x56105894d7b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc099410 tx=0x7fd0cc099090 comp rx=0 tx=0).stop 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/2988209296 shutdown_connections 2023-07-29T16:23:49.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x56105894e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x56105894c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/2988209296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894cdf0 0x56105894d7b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.046+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/2988209296 >> 172.21.15.31:0/2988209296 conn(0x561058941fd0 msgr2=0x561058947300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.047+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/2988209296 shutdown_connections 2023-07-29T16:23:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.047+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/2988209296 wait complete. 2023-07-29T16:23:49.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.047+0000 7fd0f7d46b80 1 -- start start 2023-07-29T16:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 Processor -- start 2023-07-29T16:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058955f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058956460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105883a4b0 con 0x56105894cdf0 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105883a150 con 0x56105894dcf0 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610587218c0 con 0x5610587a86b0 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.048+0000 7fd0f7d46b80 10 client.0 _ll_register_callbacks cb 0x5610588435c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058955f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058955f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49136/0 (socket says 172.21.15.31:49136) 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e299c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 -- 172.21.15.31:0/961588400 learned_addr learned my addr 172.21.15.31:0/961588400 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e199a700 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058956460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 -- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 msgr2=0x56105895d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 -- 172.21.15.31:0/961588400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 msgr2=0x561058956460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058956460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 -- 172.21.15.31:0/961588400 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0dc1a0bd0 con 0x5610587a86b0 2023-07-29T16:23:49.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0e219b700 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058955f20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8015000 tx=0x7fd0d8009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c409e191b95d30a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.049+0000 7fd0c5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/961588400 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d802e630 con 0x5610587a86b0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610588c8cb0 con 0x5610587a86b0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/961588400 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd0d802e790 con 0x5610587a86b0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/961588400 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0d802e990 con 0x5610587a86b0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0f7d46b80 10 client.28024 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5610586b4c00 con 0x5610587a86b0 2023-07-29T16:23:49.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/961588400 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5610588c8cb0 con 0x5610587a86b0 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/961588400 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd0d8025a10 con 0x5610587a86b0 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0f7d46b80 10 client.28024 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0f7d46b80 10 client.28024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.050+0000 7fd0f7d46b80 20 client.28024 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.051+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561058961ce0 0x5610589640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.051+0000 7fd0f7d46b80 4 client.28024 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:23:49.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.051+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56105891bd10 con 0x561058961ce0 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.051+0000 7fd0e199a700 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561058961ce0 0x5610589640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.051+0000 7fd0e199a700 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561058961ce0 0x5610589640f0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b26db1723585652a server_cookie=61d90c75da1f6713 in_seq=0 out_seq=0 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.051+0000 7fd0c5ffb700 10 client.28024 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.052+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/961588400 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56105891bd10 con 0x561058961ce0 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.052+0000 7fd0c5ffb700 10 client.28024 handle_command_reply: tid=0 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.052+0000 7fd0f7d46b80 1 client.28024 shutdown 2023-07-29T16:23:49.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 msgr2=0x561058955f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058955f20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8015000 tx=0x7fd0d8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 shutdown_connections 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561058961ce0 0x5610589640f0 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058955f20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/961588400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058956460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0c5ffb700 0 client.28024 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 >> 172.21.15.31:0/961588400 conn(0x561058941fd0 msgr2=0x561058946a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 shutdown_connections 2023-07-29T16:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/961588400 wait complete. 2023-07-29T16:23:49.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.053+0000 7fd0f7d46b80 20 client.28024 trim_cache size 0 max 16384 2023-07-29T16:23:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.068+0000 7fd0f7d46b80 1 Processor -- start 2023-07-29T16:23:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.068+0000 7fd0f7d46b80 1 -- start start 2023-07-29T16:23:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 0x56105895b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56105894cdf0 0x56105894e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x561058954c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105894d1e0 con 0x56105894dcf0 2023-07-29T16:23:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610589481b0 con 0x5610587a86b0 2023-07-29T16:23:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561058961ad0 con 0x56105894cdf0 2023-07-29T16:23:49.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 0x56105895b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56105894cdf0 0x56105894e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 0x56105895b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51642/0 (socket says 172.21.15.31:51642) 2023-07-29T16:23:49.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e219b700 1 -- 172.21.15.31:0/3198350363 learned_addr learned my addr 172.21.15.31:0/3198350363 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e299c700 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x561058954c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e199a700 1 -- 172.21.15.31:0/3198350363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 msgr2=0x56105895b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e199a700 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 0x56105895b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e199a700 1 -- 172.21.15.31:0/3198350363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 msgr2=0x561058954c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e199a700 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x561058954c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e199a700 1 -- 172.21.15.31:0/3198350363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610586f21f0 con 0x56105894cdf0 2023-07-29T16:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.069+0000 7fd0e219b700 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 0x56105895b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:49.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.070+0000 7fd0e199a700 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56105894cdf0 0x56105894e0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc099ae0 tx=0x7fd0cc098f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffc23275cdf8d097 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:49.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.070+0000 7fd0d13ff700 1 -- 172.21.15.31:0/3198350363 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc0a18d0 con 0x56105894cdf0 2023-07-29T16:23:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.070+0000 7fd0d13ff700 1 -- 172.21.15.31:0/3198350363 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0cc09d210 con 0x56105894cdf0 2023-07-29T16:23:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.070+0000 7fd0d13ff700 1 -- 172.21.15.31:0/3198350363 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc0a1440 con 0x56105894cdf0 2023-07-29T16:23:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/3198350363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56105894cdf0 msgr2=0x56105894e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56105894cdf0 0x56105894e0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc099ae0 tx=0x7fd0cc098f10 comp rx=0 tx=0).stop 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/3198350363 shutdown_connections 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894dcf0 0x561058954c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610587a86b0 0x56105895b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/3198350363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56105894cdf0 0x56105894e0c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/3198350363 >> 172.21.15.31:0/3198350363 conn(0x561058941fd0 msgr2=0x5610589472d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/3198350363 shutdown_connections 2023-07-29T16:23:49.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/3198350363 wait complete. 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 -- start start 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.071+0000 7fd0f7d46b80 1 Processor -- start 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058959310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058959850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105895ada0 con 0x56105894cdf0 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105895af10 con 0x56105894dcf0 2023-07-29T16:23:49.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56105895b0d0 con 0x5610587a86b0 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0f7d46b80 10 client.0 _ll_register_callbacks cb 0x5610588435c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058959310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058959850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e299c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058959850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:32774/0 (socket says 172.21.15.31:32774) 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 -- 172.21.15.31:0/511056505 learned_addr learned my addr 172.21.15.31:0/511056505 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:23:49.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058959310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49160/0 (socket says 172.21.15.31:49160) 2023-07-29T16:23:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 -- 172.21.15.31:0/511056505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 msgr2=0x561058959310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058959310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 -- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 msgr2=0x56105895a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:23:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e199a700 1 -- 172.21.15.31:0/511056505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610586b4c00 con 0x56105894cdf0 2023-07-29T16:23:49.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e219b700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058959310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.072+0000 7fd0e299c700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:23:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0e199a700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058959850 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc098fe0 tx=0x7fd0cc097820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d799c8c184327a13 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:23:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0c5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:23:49.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/511056505 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0cc0989e0 con 0x56105894cdf0 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd0cc09d4b0 con 0x56105894cdf0 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/511056505 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd0cc098b40 con 0x56105894cdf0 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/511056505 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0cc0a1440 con 0x56105894cdf0 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0f7d46b80 10 client.17973 fetch_fsmap learned FSMap version 28 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5610586cbd80 con 0x56105894cdf0 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.073+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/511056505 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0cc09d4b0 con 0x56105894cdf0 2023-07-29T16:23:49.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/511056505 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd0cc065250 con 0x56105894cdf0 2023-07-29T16:23:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0f7d46b80 10 client.17973 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:23:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0f7d46b80 10 client.17973 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:23:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0f7d46b80 20 client.17973 populate_metadata read hostname 'smithi031' 2023-07-29T16:23:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5610589b4460 0x5610589b6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:23:49.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0f7d46b80 4 client.17973 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:23:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561058896ce0 con 0x5610589b4460 2023-07-29T16:23:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0e219b700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5610589b4460 0x5610589b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:23:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0c5ffb700 10 client.17973 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:49.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.074+0000 7fd0e219b700 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5610589b4460 0x5610589b6870 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab5d9f19f17d087c server_cookie=391363826b874911 in_seq=0 out_seq=0 2023-07-29T16:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.148+0000 7fd0c5ffb700 1 -- 172.21.15.31:0/511056505 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561058896ce0 con 0x5610589b4460 2023-07-29T16:23:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.148+0000 7fd0c5ffb700 10 client.17973 handle_command_reply: tid=0 2023-07-29T16:23:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.148+0000 7fd0f7d46b80 1 client.17973 shutdown 2023-07-29T16:23:49.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 msgr2=0x561058959850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058959850 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc098fe0 tx=0x7fd0cc097820 comp rx=0 tx=0).stop 2023-07-29T16:23:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 shutdown_connections 2023-07-29T16:23:49.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5610589b4460 0x5610589b6870 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56105894dcf0 0x56105895a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5610587a86b0 0x561058959310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 --2- 172.21.15.31:0/511056505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56105894cdf0 0x561058959850 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 >> 172.21.15.31:0/511056505 conn(0x561058941fd0 msgr2=0x5610589540d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0c5ffb700 0 client.17973 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:23:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 shutdown_connections 2023-07-29T16:23:49.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.149+0000 7fd0f7d46b80 1 -- 172.21.15.31:0/511056505 wait complete. 2023-07-29T16:23:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0f7d46b80 20 client.17973 trim_cache size 0 max 16384 2023-07-29T16:23:49.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:23:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:23:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:23:49.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:23:49.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0c80a0790 msgr2=0x7fd0c80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0c80a0790 0x7fd0c80a2c40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8000c00 tx=0x7fd0d8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 msgr2=0x7fd0dc073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:23:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 0x7fd0dc073680 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc00e9a0 tx=0x7fd0cc00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 shutdown_connections 2023-07-29T16:23:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0c80a0790 0x7fd0c80a2c40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd0dc101ec0 0x7fd0dc073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd0dc100340 0x7fd0dc073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 --2- 172.21.15.31:0/2518433758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd0dc101530 0x7fd0dc073680 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:23:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.150+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 >> 172.21.15.31:0/2518433758 conn(0x7fd0dc0fb920 msgr2=0x7fd0dc0fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:23:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.151+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 shutdown_connections 2023-07-29T16:23:49.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:23:49.151+0000 7fd0e43ff700 1 -- 172.21.15.31:0/2518433758 wait complete. 2023-07-29T16:23:49.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 555206 8.93 MB/sec execute 214 sec latency 77.252 ms 2023-07-29T16:23:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:50 smithi039 ceph-mon[128858]: pgmap v621: 97 pgs: 97 active+clean; 111 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 310 op/s 2023-07-29T16:23:50.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:50 smithi156 ceph-mon[129245]: pgmap v621: 97 pgs: 97 active+clean; 111 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 310 op/s 2023-07-29T16:23:50.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 557510 8.95 MB/sec execute 215 sec latency 17.078 ms 2023-07-29T16:23:50.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:50 smithi031 ceph-mon[95504]: pgmap v621: 97 pgs: 97 active+clean; 111 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 310 op/s 2023-07-29T16:23:51.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 558619 8.93 MB/sec execute 216 sec latency 12.293 ms 2023-07-29T16:23:52.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:52 smithi039 ceph-mon[128858]: pgmap v622: 97 pgs: 97 active+clean; 111 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.0 MiB/s wr, 196 op/s 2023-07-29T16:23:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:52 smithi156 ceph-mon[129245]: pgmap v622: 97 pgs: 97 active+clean; 111 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.0 MiB/s wr, 196 op/s 2023-07-29T16:23:52.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 560910 8.95 MB/sec execute 217 sec latency 16.195 ms 2023-07-29T16:23:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:52 smithi031 ceph-mon[95504]: pgmap v622: 97 pgs: 97 active+clean; 111 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 2.0 MiB/s wr, 196 op/s 2023-07-29T16:23:53.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 562166 8.94 MB/sec execute 218 sec latency 17.106 ms 2023-07-29T16:23:54.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:54 smithi039 ceph-mon[128858]: pgmap v623: 97 pgs: 97 active+clean; 114 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.6 MiB/s wr, 294 op/s 2023-07-29T16:23:54.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:54 smithi156 ceph-mon[129245]: pgmap v623: 97 pgs: 97 active+clean; 114 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.6 MiB/s wr, 294 op/s 2023-07-29T16:23:54.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 564448 8.95 MB/sec execute 219 sec latency 16.040 ms 2023-07-29T16:23:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:54 smithi031 ceph-mon[95504]: pgmap v623: 97 pgs: 97 active+clean; 114 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 2.6 MiB/s wr, 294 op/s 2023-07-29T16:23:55.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 565585 8.94 MB/sec execute 220 sec latency 14.951 ms 2023-07-29T16:23:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:56 smithi039 ceph-mon[128858]: pgmap v624: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.5 MiB/s wr, 272 op/s 2023-07-29T16:23:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:56 smithi156 ceph-mon[129245]: pgmap v624: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.5 MiB/s wr, 272 op/s 2023-07-29T16:23:56.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 567809 8.95 MB/sec execute 221 sec latency 16.048 ms 2023-07-29T16:23:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:56 smithi031 ceph-mon[95504]: pgmap v624: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.5 MiB/s wr, 272 op/s 2023-07-29T16:23:57.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 569068 8.94 MB/sec execute 222 sec latency 12.151 ms 2023-07-29T16:23:58.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:23:57 smithi031 ceph-mon[95504]: pgmap v625: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.1 MiB/s wr, 257 op/s 2023-07-29T16:23:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:23:57 smithi039 ceph-mon[128858]: pgmap v625: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.1 MiB/s wr, 257 op/s 2023-07-29T16:23:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:23:57 smithi156 ceph-mon[129245]: pgmap v625: 97 pgs: 97 active+clean; 115 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.1 MiB/s wr, 257 op/s 2023-07-29T16:23:58.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 570934 8.94 MB/sec execute 223 sec latency 104.419 ms 2023-07-29T16:23:59.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 572287 8.95 MB/sec execute 224 sec latency 49.623 ms 2023-07-29T16:24:00.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:00 smithi039 ceph-mon[128858]: pgmap v626: 97 pgs: 97 active+clean; 118 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2023-07-29T16:24:00.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:00 smithi156 ceph-mon[129245]: pgmap v626: 97 pgs: 97 active+clean; 118 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2023-07-29T16:24:00.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 574247 8.94 MB/sec execute 225 sec latency 19.904 ms 2023-07-29T16:24:00.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:00 smithi031 ceph-mon[95504]: pgmap v626: 97 pgs: 97 active+clean; 118 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.1 MiB/s wr, 279 op/s 2023-07-29T16:24:01.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 575802 8.95 MB/sec execute 226 sec latency 15.446 ms 2023-07-29T16:24:01.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:24:02.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 -- 172.21.15.31:0/2221116394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4018074d90 msgr2=0x7f40180731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 --2- 172.21.15.31:0/2221116394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4018074d90 0x7f40180731e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f400c009990 tx=0x7f400c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 -- 172.21.15.31:0/2221116394 shutdown_connections 2023-07-29T16:24:02.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 --2- 172.21.15.31:0/2221116394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f401810c900 0x7f401810ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 --2- 172.21.15.31:0/2221116394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4018074d90 0x7f40180731e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 --2- 172.21.15.31:0/2221116394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 0x7f4018073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 -- 172.21.15.31:0/2221116394 >> 172.21.15.31:0/2221116394 conn(0x7f40180fb970 msgr2=0x7f40180fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 -- 172.21.15.31:0/2221116394 shutdown_connections 2023-07-29T16:24:02.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.166+0000 7f401efff700 1 -- 172.21.15.31:0/2221116394 wait complete. 2023-07-29T16:24:02.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 Processor -- start 2023-07-29T16:24:02.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 -- start start 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 0x7f401819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4018074d90 0x7f401819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 0x7f40181a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181a4250 con 0x7f4018074d90 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181a4390 con 0x7f401810c900 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.167+0000 7f401efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181a4520 con 0x7f40180737b0 2023-07-29T16:24:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 0x7f401819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 0x7f40181a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f4017fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4018074d90 0x7f401819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 0x7f40181a3c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46390/0 (socket says 172.21.15.31:46390) 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 0x7f401819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48486/0 (socket says 172.21.15.31:48486) 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 -- 172.21.15.31:0/4171628423 learned_addr learned my addr 172.21.15.31:0/4171628423 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 -- 172.21.15.31:0/4171628423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 msgr2=0x7f401819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 0x7f401819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 -- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4018074d90 msgr2=0x7f401819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4018074d90 0x7f401819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 -- 172.21.15.31:0/4171628423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c0095f0 con 0x7f401810c900 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.168+0000 7f401d59c700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 0x7f40181a3c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f401000ef30 tx=0x7f401000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.169+0000 7f4015ffb700 1 -- 172.21.15.31:0/4171628423 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010018070 con 0x7f401810c900 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.169+0000 7f4015ffb700 1 -- 172.21.15.31:0/4171628423 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f401000f040 con 0x7f401810c900 2023-07-29T16:24:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.169+0000 7f4015ffb700 1 -- 172.21.15.31:0/4171628423 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401000cd00 con 0x7f401810c900 2023-07-29T16:24:02.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.169+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40181a47a0 con 0x7f401810c900 2023-07-29T16:24:02.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.169+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40181a4cb0 con 0x7f401810c900 2023-07-29T16:24:02.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.172+0000 7f4015ffb700 1 -- 172.21.15.31:0/4171628423 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4010003a00 con 0x7f401810c900 2023-07-29T16:24:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.175+0000 7f4015ffb700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40080a0810 0x7f40080a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.175+0000 7f4015ffb700 1 -- 172.21.15.31:0/4171628423 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f40100cb580 con 0x7f401810c900 2023-07-29T16:24:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.175+0000 7f4032832b80 1 Processor -- start 2023-07-29T16:24:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.175+0000 7f401cd9b700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40080a0810 0x7f40080a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.175+0000 7f4032832b80 1 -- start start 2023-07-29T16:24:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4032832b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1d9dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4032832b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4032832b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1d9f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4032832b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1cf5a10 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4032832b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1cf33d0 con 0x55f8e1d9f2f0 2023-07-29T16:24:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4032832b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1ceecc0 con 0x55f8e1d9e530 2023-07-29T16:24:02.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f401d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1d9f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4017fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f401d59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1d9f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46396/0 (socket says 172.21.15.31:46396) 2023-07-29T16:24:02.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4017fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48492/0 (socket says 172.21.15.31:48492) 2023-07-29T16:24:02.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f401d59c700 1 -- 172.21.15.31:0/3392472656 learned_addr learned my addr 172.21.15.31:0/3392472656 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1d9dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f401cd9b700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40080a0810 0x7f40080a2cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f400c00b2e0 tx=0x7f400c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4017fff700 1 -- 172.21.15.31:0/3392472656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 msgr2=0x55f8e1d9f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4017fff700 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1d9f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4017fff700 1 -- 172.21.15.31:0/3392472656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9dbe0 msgr2=0x55f8e1d9dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.176+0000 7f4017fff700 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1d9dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.177+0000 7f4017fff700 1 -- 172.21.15.31:0/3392472656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8e1b1cde0 con 0x55f8e1d9e530 2023-07-29T16:24:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.177+0000 7f401d59c700 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1d9f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:02.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.177+0000 7f4017fff700 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9ed60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4004009c80 tx=0x7f4004009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1009fcabbf2f1e2d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.177+0000 7f4002ffd700 1 -- 172.21.15.31:0/3392472656 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004011cb0 con 0x55f8e1d9e530 2023-07-29T16:24:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.177+0000 7f4002ffd700 1 -- 172.21.15.31:0/3392472656 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4004011e10 con 0x55f8e1d9e530 2023-07-29T16:24:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.177+0000 7f4002ffd700 1 -- 172.21.15.31:0/3392472656 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004005280 con 0x55f8e1d9e530 2023-07-29T16:24:02.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 -- 172.21.15.31:0/3392472656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 msgr2=0x55f8e1d9ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9ed60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4004009c80 tx=0x7f4004009f90 comp rx=0 tx=0).stop 2023-07-29T16:24:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 -- 172.21.15.31:0/3392472656 shutdown_connections 2023-07-29T16:24:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1d9f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1d9dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 --2- 172.21.15.31:0/3392472656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9ed60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 -- 172.21.15.31:0/3392472656 >> 172.21.15.31:0/3392472656 conn(0x55f8e1d933e0 msgr2=0x55f8e1d98990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 -- 172.21.15.31:0/3392472656 shutdown_connections 2023-07-29T16:24:02.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 -- 172.21.15.31:0/3392472656 wait complete. 2023-07-29T16:24:02.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.178+0000 7f4032832b80 1 -- start start 2023-07-29T16:24:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 Processor -- start 2023-07-29T16:24:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1da7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1da7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1c8b510 con 0x55f8e1d9f2f0 2023-07-29T16:24:02.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1c8b1b0 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1b72920 con 0x55f8e1d9e530 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4032832b80 10 client.0 _ll_register_callbacks cb 0x55f8e1c94620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f4017fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1da7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.179+0000 7f401cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1da7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1da7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48496/0 (socket says 172.21.15.31:48496) 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 -- 172.21.15.31:0/541807633 learned_addr learned my addr 172.21.15.31:0/541807633 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f401d59c700 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 -- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 msgr2=0x55f8e1da7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1da7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 -- 172.21.15.31:0/541807633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 msgr2=0x55f8e1daedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 -- 172.21.15.31:0/541807633 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40181a4cb0 con 0x55f8e1d9e530 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f4017fff700 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1da7d90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4004011ab0 tx=0x7f4004011ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8882e982f357e070 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f3ff7fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:02.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.180+0000 7f3ff7fff700 1 -- 172.21.15.31:0/541807633 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4004005080 con 0x55f8e1d9e530 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8e1d19d10 con 0x55f8e1d9e530 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f3ff7fff700 1 -- 172.21.15.31:0/541807633 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f40040051e0 con 0x55f8e1d9e530 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f3ff7fff700 1 -- 172.21.15.31:0/541807633 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40040053a0 con 0x55f8e1d9e530 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f4032832b80 10 client.28042 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f40040098e0 con 0x55f8e1d9e530 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f3ff7fff700 1 -- 172.21.15.31:0/541807633 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8e1d19d10 con 0x55f8e1d9e530 2023-07-29T16:24:02.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.181+0000 7f3ff7fff700 1 -- 172.21.15.31:0/541807633 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4004022400 con 0x55f8e1d9e530 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f4032832b80 10 client.28042 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f4032832b80 10 client.28042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f4032832b80 20 client.28042 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f4032832b80 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1db34a0 0x55f8e1db58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f4032832b80 4 client.28042 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8e1d6cd70 con 0x55f8e1db34a0 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f401cd9b700 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1db34a0 0x55f8e1db58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f401cd9b700 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1db34a0 0x55f8e1db58b0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=352aa83432697ed server_cookie=466ca5e2f44c6b74 in_seq=0 out_seq=0 2023-07-29T16:24:02.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.182+0000 7f3ff7fff700 10 client.28042 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f3ff7fff700 1 -- 172.21.15.31:0/541807633 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f8e1d6cd70 con 0x55f8e1db34a0 2023-07-29T16:24:02.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f3ff7fff700 10 client.28042 handle_command_reply: tid=0 2023-07-29T16:24:02.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 client.28042 shutdown 2023-07-29T16:24:02.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 msgr2=0x55f8e1da7d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1da7d90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4004011ab0 tx=0x7f4004011ae0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 shutdown_connections 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1db34a0 0x55f8e1db58b0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1da7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 --2- 172.21.15.31:0/541807633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1da7d90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 >> 172.21.15.31:0/541807633 conn(0x55f8e1d933e0 msgr2=0x55f8e1d98060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f3ff7fff700 0 client.28042 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 shutdown_connections 2023-07-29T16:24:02.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.186+0000 7f4032832b80 1 -- 172.21.15.31:0/541807633 wait complete. 2023-07-29T16:24:02.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.187+0000 7f4032832b80 20 client.28042 trim_cache size 0 max 16384 2023-07-29T16:24:02.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 Processor -- start 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 -- start start 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dacd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1da6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1d996c0 con 0x55f8e1d9e530 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1d9eb70 con 0x55f8e1d9f2f0 2023-07-29T16:24:02.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4032832b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1d906e0 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f401cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dacd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f401cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dacd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48498/0 (socket says 172.21.15.31:48498) 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f401cd9b700 1 -- 172.21.15.31:0/3268939998 learned_addr learned my addr 172.21.15.31:0/3268939998 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f401d59c700 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1da6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.206+0000 7f4017fff700 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f401cd9b700 1 -- 172.21.15.31:0/3268939998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 msgr2=0x55f8e1da6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1da6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f401cd9b700 1 -- 172.21.15.31:0/3268939998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9e530 msgr2=0x55f8e1d9f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f401cd9b700 1 -- 172.21.15.31:0/3268939998 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8e1b05c60 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dacd80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4018075160 tx=0x7f400c020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f987491a5bad774 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f40037fe700 1 -- 172.21.15.31:0/3268939998 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c033110 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f40037fe700 1 -- 172.21.15.31:0/3268939998 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f400c01e040 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.207+0000 7f40037fe700 1 -- 172.21.15.31:0/3268939998 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c004500 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 -- 172.21.15.31:0/3268939998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 msgr2=0x55f8e1dacd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dacd80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4018075160 tx=0x7f400c020000 comp rx=0 tx=0).stop 2023-07-29T16:24:02.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 -- 172.21.15.31:0/3268939998 shutdown_connections 2023-07-29T16:24:02.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1da6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dacd80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 --2- 172.21.15.31:0/3268939998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9e530 0x55f8e1d9f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 -- 172.21.15.31:0/3268939998 >> 172.21.15.31:0/3268939998 conn(0x55f8e1d933e0 msgr2=0x55f8e1d98940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 -- 172.21.15.31:0/3268939998 shutdown_connections 2023-07-29T16:24:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.208+0000 7f4032832b80 1 -- 172.21.15.31:0/3268939998 wait complete. 2023-07-29T16:24:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.209+0000 7f4032832b80 1 -- start start 2023-07-29T16:24:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.209+0000 7f4032832b80 1 Processor -- start 2023-07-29T16:24:02.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1dacca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1dab4c0 con 0x55f8e1d9f2f0 2023-07-29T16:24:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1dab630 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8e1dab7f0 con 0x55f8e1d9e530 2023-07-29T16:24:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4032832b80 10 client.0 _ll_register_callbacks cb 0x55f8e1c94620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:02.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4017fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1dacca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f401cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4017fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1dacca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48508/0 (socket says 172.21.15.31:48508) 2023-07-29T16:24:02.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f4017fff700 1 -- 172.21.15.31:0/3012060541 learned_addr learned my addr 172.21.15.31:0/3012060541 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f401d59c700 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f401cd9b700 1 -- 172.21.15.31:0/3012060541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 msgr2=0x55f8e1dacca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1dacca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f401cd9b700 1 -- 172.21.15.31:0/3012060541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 msgr2=0x55f8e1daadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.210+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f401cd9b700 1 -- 172.21.15.31:0/3012060541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401810f690 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f401cd9b700 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dac760 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x55f8e1da8a90 tx=0x7f400c009340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad54935a6209872 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f3ff7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f3ff7fff700 1 -- 172.21.15.31:0/3012060541 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f400c033310 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f3ff7fff700 1 -- 172.21.15.31:0/3012060541 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f400c005170 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4004011e10 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f3ff7fff700 1 -- 172.21.15.31:0/3012060541 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f400c033470 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f4032832b80 10 client.28037 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.211+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f8e1b43250 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f3ff7fff700 1 -- 172.21.15.31:0/3012060541 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4004011e10 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f3ff7fff700 1 -- 172.21.15.31:0/3012060541 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f400c00f4e0 con 0x55f8e1d9dbe0 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4032832b80 10 client.28037 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:02.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4032832b80 10 client.28037 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4032832b80 20 client.28037 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4032832b80 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1e05f20 0x55f8e1e08300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4032832b80 4 client.28037 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:24:02.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f8e1ce7d40 con 0x55f8e1e05f20 2023-07-29T16:24:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.212+0000 7f4017fff700 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1e05f20 0x55f8e1e08300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4017fff700 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1e05f20 0x55f8e1e08300 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=544f56d4f85f75a6 server_cookie=dd7805f0efbf7e53 in_seq=0 out_seq=0 2023-07-29T16:24:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f3ff7fff700 10 client.28037 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f3ff7fff700 1 -- 172.21.15.31:0/3012060541 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f8e1ce7d40 con 0x55f8e1e05f20 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f3ff7fff700 10 client.28037 handle_command_reply: tid=0 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 client.28037 shutdown 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 msgr2=0x55f8e1dac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dac760 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x55f8e1da8a90 tx=0x7f400c009340 comp rx=0 tx=0).stop 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 shutdown_connections 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8e1e05f20 0x55f8e1e08300 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8e1d9f2f0 0x55f8e1daadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8e1d9dbe0 0x55f8e1dac760 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 --2- 172.21.15.31:0/3012060541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8e1d9e530 0x55f8e1dacca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f3ff7fff700 0 client.28037 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 >> 172.21.15.31:0/3012060541 conn(0x55f8e1d933e0 msgr2=0x55f8e1da6510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 shutdown_connections 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 1 -- 172.21.15.31:0/3012060541 wait complete. 2023-07-29T16:24:02.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.213+0000 7f4032832b80 20 client.28037 trim_cache size 0 max 16384 2023-07-29T16:24:02.224 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:24:02.224 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:24:02.224 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:24:02.224 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:24:02.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40080a0810 msgr2=0x7f40080a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40080a0810 0x7f40080a2cc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f400c00b2e0 tx=0x7f400c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 msgr2=0x7f40181a3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 0x7f40181a3c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f401000ef30 tx=0x7f401000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 shutdown_connections 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f40080a0810 0x7f40080a2cc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f401810c900 0x7f40181a3c80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f40180737b0 0x7f401819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 --2- 172.21.15.31:0/4171628423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4018074d90 0x7f401819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 >> 172.21.15.31:0/4171628423 conn(0x7f40180fb970 msgr2=0x7f40180fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 shutdown_connections 2023-07-29T16:24:02.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.214+0000 7f401efff700 1 -- 172.21.15.31:0/4171628423 wait complete. 2023-07-29T16:24:02.240 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:24:02.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:24:02.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.326+0000 7feb7cdff700 1 -- 172.21.15.31:0/1953555822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78068e80 msgr2=0x7feb78102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.326+0000 7feb7cdff700 1 --2- 172.21.15.31:0/1953555822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78068e80 0x7feb78102ed0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7feb68009990 tx=0x7feb68009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 -- 172.21.15.31:0/1953555822 shutdown_connections 2023-07-29T16:24:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 --2- 172.21.15.31:0/1953555822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb78105f20 0x7feb781083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 --2- 172.21.15.31:0/1953555822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78068e80 0x7feb78102ed0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 --2- 172.21.15.31:0/1953555822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb78103410 0x7feb78105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 -- 172.21.15.31:0/1953555822 >> 172.21.15.31:0/1953555822 conn(0x7feb780fa840 msgr2=0x7feb780fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 -- 172.21.15.31:0/1953555822 shutdown_connections 2023-07-29T16:24:02.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.327+0000 7feb7cdff700 1 -- 172.21.15.31:0/1953555822 wait complete. 2023-07-29T16:24:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.328+0000 7feb7cdff700 1 Processor -- start 2023-07-29T16:24:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.328+0000 7feb7cdff700 1 -- start start 2023-07-29T16:24:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7cdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb78068e80 0x7feb78198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7cdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 0x7feb78198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7cdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb78105f20 0x7feb781997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7cdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78199e20 con 0x7feb78068e80 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7cdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb78199f90 con 0x7feb78103410 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7cdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7819a150 con 0x7feb78105f20 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb6ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 0x7feb78198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb7659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb78068e80 0x7feb78198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb6ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 0x7feb78198940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46446/0 (socket says 172.21.15.31:46446) 2023-07-29T16:24:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.329+0000 7feb6ffff700 1 -- 172.21.15.31:0/40921867 learned_addr learned my addr 172.21.15.31:0/40921867 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6ffff700 1 -- 172.21.15.31:0/40921867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb78105f20 msgr2=0x7feb781997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6ffff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb78105f20 0x7feb781997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6ffff700 1 -- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb78068e80 msgr2=0x7feb78198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6ffff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb78068e80 0x7feb78198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6ffff700 1 -- 172.21.15.31:0/40921867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb680095f0 con 0x7feb78103410 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6ffff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 0x7feb78198940 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7feb6000ef10 tx=0x7feb6000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6f7fe700 1 -- 172.21.15.31:0/40921867 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6000ca70 con 0x7feb78103410 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6f7fe700 1 -- 172.21.15.31:0/40921867 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6000f040 con 0x7feb78103410 2023-07-29T16:24:02.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb6f7fe700 1 -- 172.21.15.31:0/40921867 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb60014480 con 0x7feb78103410 2023-07-29T16:24:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7819a3d0 con 0x7feb78103410 2023-07-29T16:24:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.330+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781a0bf0 con 0x7feb78103410 2023-07-29T16:24:02.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.334+0000 7feb6f7fe700 1 -- 172.21.15.31:0/40921867 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7feb60007500 con 0x7feb78103410 2023-07-29T16:24:02.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.335+0000 7feb90794b80 1 Processor -- start 2023-07-29T16:24:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.335+0000 7feb90794b80 1 -- start start 2023-07-29T16:24:02.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.335+0000 7feb6f7fe700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb640a08e0 0x7feb640a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb6f7fe700 1 -- 172.21.15.31:0/40921867 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feb600cbd50 con 0x7feb78103410 2023-07-29T16:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb90794b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efa88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb90794b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa8df0 0x55759efa97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb90794b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb90794b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759ef009b0 con 0x55759efa9cf0 2023-07-29T16:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb90794b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759eefe370 con 0x55759ee046b0 2023-07-29T16:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb90794b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759eef9c60 con 0x55759efa8df0 2023-07-29T16:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb76d9d700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb640a08e0 0x7feb640a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb7659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efa88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb7659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efa88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46456/0 (socket says 172.21.15.31:46456) 2023-07-29T16:24:02.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.336+0000 7feb7659c700 1 -- 172.21.15.31:0/1828958728 learned_addr learned my addr 172.21.15.31:0/1828958728 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa8df0 0x55759efa97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb7659c700 1 -- 172.21.15.31:0/1828958728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa8df0 msgr2=0x55759efa97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb7659c700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa8df0 0x55759efa97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb76d9d700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb7659c700 1 -- 172.21.15.31:0/1828958728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 msgr2=0x55759efaa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb7659c700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb7659c700 1 -- 172.21.15.31:0/1828958728 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55759ed27d80 con 0x55759ee046b0 2023-07-29T16:24:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa8df0 0x55759efa97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb76d9d700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb7659c700 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efa88b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feb68009990 tx=0x7feb68005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf07cb4d554cd1c1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb76d9d700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb640a08e0 0x7feb640a2d90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55759efa8610 tx=0x7feb70009ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.337+0000 7feb6cff9700 1 -- 172.21.15.31:0/1828958728 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6801d070 con 0x55759ee046b0 2023-07-29T16:24:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb6cff9700 1 -- 172.21.15.31:0/1828958728 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6801e040 con 0x55759ee046b0 2023-07-29T16:24:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb6cff9700 1 -- 172.21.15.31:0/1828958728 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb68004960 con 0x55759ee046b0 2023-07-29T16:24:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 -- 172.21.15.31:0/1828958728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 msgr2=0x55759efa88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efa88b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7feb68009990 tx=0x7feb68005ea0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 -- 172.21.15.31:0/1828958728 shutdown_connections 2023-07-29T16:24:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efa88b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 --2- 172.21.15.31:0/1828958728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa8df0 0x55759efa97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.338+0000 7feb90794b80 1 -- 172.21.15.31:0/1828958728 >> 172.21.15.31:0/1828958728 conn(0x55759ef9dfd0 msgr2=0x55759efa3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.339+0000 7feb90794b80 1 -- 172.21.15.31:0/1828958728 shutdown_connections 2023-07-29T16:24:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.339+0000 7feb90794b80 1 -- 172.21.15.31:0/1828958728 wait complete. 2023-07-29T16:24:02.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.339+0000 7feb90794b80 1 -- start start 2023-07-29T16:24:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.339+0000 7feb90794b80 1 Processor -- start 2023-07-29T16:24:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa9cf0 0x55759efb9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759ee964b0 con 0x55759efa8df0 2023-07-29T16:24:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759ee96150 con 0x55759efa9cf0 2023-07-29T16:24:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759ed7d8c0 con 0x55759ee046b0 2023-07-29T16:24:02.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb90794b80 10 client.0 _ll_register_callbacks cb 0x55759ee9f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb7659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb7659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48548/0 (socket says 172.21.15.31:48548) 2023-07-29T16:24:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb7659c700 1 -- 172.21.15.31:0/2997468242 learned_addr learned my addr 172.21.15.31:0/2997468242 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb76d9d700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa9cf0 0x55759efb9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.340+0000 7feb6ffff700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb7659c700 1 -- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa9cf0 msgr2=0x55759efb9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb7659c700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa9cf0 0x55759efb9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb7659c700 1 -- 172.21.15.31:0/2997468242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 msgr2=0x55759efb2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb7659c700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb7659c700 1 -- 172.21.15.31:0/2997468242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb78109090 con 0x55759ee046b0 2023-07-29T16:24:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb6ffff700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb7659c700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb1e00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feb68000c00 tx=0x7feb68022760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d439d30befb98fa6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb5dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:02.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb5dffb700 1 -- 172.21.15.31:0/2997468242 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6801d070 con 0x55759ee046b0 2023-07-29T16:24:02.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55759ef24cb0 con 0x55759ee046b0 2023-07-29T16:24:02.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.341+0000 7feb5dffb700 1 -- 172.21.15.31:0/2997468242 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feb68022e10 con 0x55759ee046b0 2023-07-29T16:24:02.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb5dffb700 1 -- 172.21.15.31:0/2997468242 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb6800f460 con 0x55759ee046b0 2023-07-29T16:24:02.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 10 client.28057 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:02.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55759ed10c00 con 0x55759ee046b0 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb5dffb700 1 -- 172.21.15.31:0/2997468242 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55759ef24cb0 con 0x55759ee046b0 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb5dffb700 1 -- 172.21.15.31:0/2997468242 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7feb680375c0 con 0x55759ee046b0 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 10 client.28057 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 10 client.28057 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 20 client.28057 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759efbd5b0 0x55759efbf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 4 client.28057 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:02.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.342+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55759ef77d10 con 0x55759efbd5b0 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.343+0000 7feb6ffff700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759efbd5b0 0x55759efbf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.343+0000 7feb6ffff700 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759efbd5b0 0x55759efbf990 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d22a90c28864c7ca server_cookie=c8f026a27933b113 in_seq=0 out_seq=0 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.343+0000 7feb5dffb700 10 client.28057 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.344+0000 7feb5dffb700 1 -- 172.21.15.31:0/2997468242 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55759ef77d10 con 0x55759efbd5b0 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.344+0000 7feb5dffb700 10 client.28057 handle_command_reply: tid=0 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.344+0000 7feb90794b80 1 client.28057 shutdown 2023-07-29T16:24:02.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 msgr2=0x55759efb1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb1e00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7feb68000c00 tx=0x7feb68022760 comp rx=0 tx=0).stop 2023-07-29T16:24:02.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 shutdown_connections 2023-07-29T16:24:02.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759efbd5b0 0x55759efbf990 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa9cf0 0x55759efb9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb5dffb700 0 client.28057 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb1e00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 --2- 172.21.15.31:0/2997468242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 >> 172.21.15.31:0/2997468242 conn(0x55759ef9dfd0 msgr2=0x55759efa2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 shutdown_connections 2023-07-29T16:24:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 1 -- 172.21.15.31:0/2997468242 wait complete. 2023-07-29T16:24:02.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.345+0000 7feb90794b80 20 client.28057 trim_cache size 0 max 16384 2023-07-29T16:24:02.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 Processor -- start 2023-07-29T16:24:02.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 -- start start 2023-07-29T16:24:02.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 0x55759efb0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759efbd410 con 0x55759efa9cf0 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759efa4b60 con 0x55759efa8df0 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.365+0000 7feb90794b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759efaa630 con 0x55759ee046b0 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 0x55759efb0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 0x55759efb0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46482/0 (socket says 172.21.15.31:46482) 2023-07-29T16:24:02.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb7659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 -- 172.21.15.31:0/1548222442 learned_addr learned my addr 172.21.15.31:0/1548222442 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb76d9d700 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 -- 172.21.15.31:0/1548222442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 msgr2=0x55759efb7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 -- 172.21.15.31:0/1548222442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 msgr2=0x55759efaa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 -- 172.21.15.31:0/1548222442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55759ed4e1f0 con 0x55759efa8df0 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.366+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 0x55759efb0bb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7feb60095e60 tx=0x7feb600c9660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153c8fbf2110a1a5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb6d7fa700 1 -- 172.21.15.31:0/1548222442 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb600b5070 con 0x55759efa8df0 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb6d7fa700 1 -- 172.21.15.31:0/1548222442 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb600a1840 con 0x55759efa8df0 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb6d7fa700 1 -- 172.21.15.31:0/1548222442 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6009d120 con 0x55759efa8df0 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 -- 172.21.15.31:0/1548222442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 msgr2=0x55759efb0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 0x55759efb0bb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7feb60095e60 tx=0x7feb600c9660 comp rx=0 tx=0).stop 2023-07-29T16:24:02.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 -- 172.21.15.31:0/1548222442 shutdown_connections 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa9cf0 0x55759efaa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759ee046b0 0x55759efb7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 --2- 172.21.15.31:0/1548222442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759efa8df0 0x55759efb0bb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.367+0000 7feb90794b80 1 -- 172.21.15.31:0/1548222442 >> 172.21.15.31:0/1548222442 conn(0x55759ef9dfd0 msgr2=0x55759efa2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.368+0000 7feb90794b80 1 -- 172.21.15.31:0/1548222442 shutdown_connections 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.368+0000 7feb90794b80 1 -- 172.21.15.31:0/1548222442 wait complete. 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.368+0000 7feb90794b80 1 -- start start 2023-07-29T16:24:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.368+0000 7feb90794b80 1 Processor -- start 2023-07-29T16:24:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efb6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa9cf0 0x55759efb5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759efb76e0 con 0x55759efa8df0 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759efb5fa0 con 0x55759ee046b0 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55759efb6160 con 0x55759efa9cf0 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb90794b80 10 client.0 _ll_register_callbacks cb 0x55759ee9f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb7659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efb6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb7659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efb6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46484/0 (socket says 172.21.15.31:46484) 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb7659c700 1 -- 172.21.15.31:0/1204605022 learned_addr learned my addr 172.21.15.31:0/1204605022 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb76d9d700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa9cf0 0x55759efb5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.369+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb7659c700 1 -- 172.21.15.31:0/1204605022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa9cf0 msgr2=0x55759efb5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb7659c700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa9cf0 0x55759efb5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb7659c700 1 -- 172.21.15.31:0/1204605022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 msgr2=0x55759efb5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb7659c700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb7659c700 1 -- 172.21.15.31:0/1204605022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb781a0bf0 con 0x55759ee046b0 2023-07-29T16:24:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb7659c700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efb6ec0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7feb68009ce0 tx=0x7feb6802a5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8103d16c41abc0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb5dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb5dffb700 1 -- 172.21.15.31:0/1204605022 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6801d040 con 0x55759ee046b0 2023-07-29T16:24:02.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feb6801e040 con 0x55759ee046b0 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb5dffb700 1 -- 172.21.15.31:0/1204605022 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feb68000e40 con 0x55759ee046b0 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb5dffb700 1 -- 172.21.15.31:0/1204605022 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb68022880 con 0x55759ee046b0 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb90794b80 10 client.28055 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.370+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55759ed26250 con 0x55759ee046b0 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb5dffb700 1 -- 172.21.15.31:0/1204605022 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb6801e040 con 0x55759ee046b0 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb5dffb700 1 -- 172.21.15.31:0/1204605022 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7feb68004500 con 0x55759ee046b0 2023-07-29T16:24:02.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb90794b80 10 client.28055 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb90794b80 10 client.28055 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb90794b80 20 client.28055 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb90794b80 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759f0102c0 0x55759f0126a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb90794b80 4 client.28055 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55759eef2ce0 con 0x55759f0102c0 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.371+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759f0102c0 0x55759f0126a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.372+0000 7feb6ffff700 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759f0102c0 0x55759f0126a0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92dae861783372dd server_cookie=84b27e16b22049a9 in_seq=0 out_seq=0 2023-07-29T16:24:02.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.372+0000 7feb5dffb700 10 client.28055 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb5dffb700 1 -- 172.21.15.31:0/1204605022 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55759eef2ce0 con 0x55759f0102c0 2023-07-29T16:24:02.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb5dffb700 10 client.28055 handle_command_reply: tid=0 2023-07-29T16:24:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 client.28055 shutdown 2023-07-29T16:24:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 msgr2=0x55759efb6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efb6ec0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7feb68009ce0 tx=0x7feb6802a5e0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 shutdown_connections 2023-07-29T16:24:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55759f0102c0 0x55759f0126a0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55759efa9cf0 0x55759efb5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55759ee046b0 0x55759efb6ec0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 --2- 172.21.15.31:0/1204605022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55759efa8df0 0x55759efb5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb5dffb700 0 client.28055 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.457+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 >> 172.21.15.31:0/1204605022 conn(0x55759ef9dfd0 msgr2=0x55759efacba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.458+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 shutdown_connections 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.458+0000 7feb90794b80 1 -- 172.21.15.31:0/1204605022 wait complete. 2023-07-29T16:24:02.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.458+0000 7feb90794b80 20 client.28055 trim_cache size 0 max 16384 2023-07-29T16:24:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:24:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:24:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:24:02.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:24:02.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb640a08e0 msgr2=0x7feb640a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb640a08e0 0x7feb640a2d90 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x55759efa8610 tx=0x7feb70009ce0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 msgr2=0x7feb78198940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 0x7feb78198940 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7feb6000ef10 tx=0x7feb6000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 shutdown_connections 2023-07-29T16:24:02.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feb640a08e0 0x7feb640a2d90 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb78105f20 0x7feb781997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb78068e80 0x7feb78198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 --2- 172.21.15.31:0/40921867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb78103410 0x7feb78198940 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 >> 172.21.15.31:0/40921867 conn(0x7feb780fa840 msgr2=0x7feb780fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 shutdown_connections 2023-07-29T16:24:02.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:02.459+0000 7feb7cdff700 1 -- 172.21.15.31:0/40921867 wait complete. 2023-07-29T16:24:02.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 -- 172.21.15.31:0/4069367996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 msgr2=0x7f22781031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 --2- 172.21.15.31:0/4069367996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f22781031f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f226c009990 tx=0x7f226c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 -- 172.21.15.31:0/4069367996 shutdown_connections 2023-07-29T16:24:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 --2- 172.21.15.31:0/4069367996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2278106050 0x7f2278108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 --2- 172.21.15.31:0/4069367996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f22781031f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 --2- 172.21.15.31:0/4069367996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2278103730 0x7f2278105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 -- 172.21.15.31:0/4069367996 >> 172.21.15.31:0/4069367996 conn(0x7f22780fa810 msgr2=0x7f22780fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 -- 172.21.15.31:0/4069367996 shutdown_connections 2023-07-29T16:24:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.493+0000 7f227f1ff700 1 -- 172.21.15.31:0/4069367996 wait complete. 2023-07-29T16:24:02.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.494+0000 7f227f1ff700 1 Processor -- start 2023-07-29T16:24:02.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.494+0000 7f227f1ff700 1 -- start start 2023-07-29T16:24:02.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227f1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f2278100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227f1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2278103730 0x7f22780fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227f1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2278106050 0x7f22780ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227f1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780ffaa0 con 0x7f2278106050 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227f1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22780ffc10 con 0x7f2278100e10 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227f1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22781a0060 con 0x7f2278103730 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f2278100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f2278100780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46502/0 (socket says 172.21.15.31:46502) 2023-07-29T16:24:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 -- 172.21.15.31:0/1823661248 learned_addr learned my addr 172.21.15.31:0/1823661248 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f2277fff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2278103730 0x7f22780fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227d79c700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2278106050 0x7f22780ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 -- 172.21.15.31:0/1823661248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2278103730 msgr2=0x7f22780fedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2278103730 0x7f22780fedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 -- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2278106050 msgr2=0x7f22780ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2278106050 0x7f22780ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.495+0000 7f227cf9b700 1 -- 172.21.15.31:0/1823661248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264009520 con 0x7f2278100e10 2023-07-29T16:24:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.496+0000 7f227cf9b700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f2278100780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f226c006010 tx=0x7f226c0057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.496+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c01d070 con 0x7f2278100e10 2023-07-29T16:24:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.496+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f226c022470 con 0x7f2278100e10 2023-07-29T16:24:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.496+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c0095f0 con 0x7f2278100e10 2023-07-29T16:24:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.496+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226c017650 con 0x7f2278100e10 2023-07-29T16:24:02.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.496+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22781a0570 con 0x7f2278100e10 2023-07-29T16:24:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.497+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22780fc380 con 0x7f2278100e10 2023-07-29T16:24:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.500+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f226c022890 con 0x7f2278100e10 2023-07-29T16:24:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.501+0000 7f2275ffb700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22680a0860 0x7f22680a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.501+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f226c0cd000 con 0x7f2278100e10 2023-07-29T16:24:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.501+0000 7f2277fff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22680a0860 0x7f22680a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:02.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.502+0000 7f2277fff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22680a0860 0x7f22680a2d10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2264009df0 tx=0x7f2264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:02.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.503+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f226c096a70 con 0x7f2278100e10 2023-07-29T16:24:02.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:02 smithi031 ceph-mon[95504]: pgmap v627: 97 pgs: 97 active+clean; 118 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.5 MiB/s wr, 217 op/s 2023-07-29T16:24:02.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:02 smithi039 ceph-mon[128858]: pgmap v627: 97 pgs: 97 active+clean; 118 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.5 MiB/s wr, 217 op/s 2023-07-29T16:24:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.778+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f227802cc30 con 0x7f2278100e10 2023-07-29T16:24:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.778+0000 7f2275ffb700 1 -- 172.21.15.31:0/1823661248 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7f226c09a090 con 0x7f2278100e10 2023-07-29T16:24:02.779 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:24:02.779 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:24:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22680a0860 msgr2=0x7f22680a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22680a0860 0x7f22680a2d10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2264009df0 tx=0x7f2264009260 comp rx=0 tx=0).stop 2023-07-29T16:24:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 msgr2=0x7f2278100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:02.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f2278100780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f226c006010 tx=0x7f226c0057b0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 shutdown_connections 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f22680a0860 0x7f22680a2d10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2278106050 0x7f22780ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2278100e10 0x7f2278100780 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 --2- 172.21.15.31:0/1823661248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2278103730 0x7f22780fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 >> 172.21.15.31:0/1823661248 conn(0x7f22780fa810 msgr2=0x7f2278101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:02.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 shutdown_connections 2023-07-29T16:24:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:02.781+0000 7f227f1ff700 1 -- 172.21.15.31:0/1823661248 wait complete. 2023-07-29T16:24:02.783 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:24:02.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:24:02.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:02 smithi156 ceph-mon[129245]: pgmap v627: 97 pgs: 97 active+clean; 118 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 1.5 MiB/s wr, 217 op/s 2023-07-29T16:24:02.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 576951 8.94 MB/sec execute 227 sec latency 39.144 ms 2023-07-29T16:24:03.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 -- 172.21.15.31:0/3820289198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8103700 msgr2=0x7f49f8105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/3820289198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8103700 0x7f49f8105ae0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009990 tx=0x7f49e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 -- 172.21.15.31:0/3820289198 shutdown_connections 2023-07-29T16:24:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/3820289198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f8106020 0x7f49f8108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/3820289198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8100de0 0x7f49f81031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/3820289198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8103700 0x7f49f8105ae0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 -- 172.21.15.31:0/3820289198 >> 172.21.15.31:0/3820289198 conn(0x7f49f80fa8a0 msgr2=0x7f49f80fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 -- 172.21.15.31:0/3820289198 shutdown_connections 2023-07-29T16:24:03.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.039+0000 7f49ff7ff700 1 -- 172.21.15.31:0/3820289198 wait complete. 2023-07-29T16:24:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.041+0000 7f49ff7ff700 1 Processor -- start 2023-07-29T16:24:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.041+0000 7f49ff7ff700 1 -- start start 2023-07-29T16:24:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49ff7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 0x7f49f81982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49ff7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8103700 0x7f49f81987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49ff7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f8106020 0x7f49f819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49ff7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819fe30 con 0x7f49f8106020 2023-07-29T16:24:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49ff7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819ff70 con 0x7f49f8100de0 2023-07-29T16:24:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49ff7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f81a0100 con 0x7f49f8103700 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 0x7f49f81982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 0x7f49f81982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46516/0 (socket says 172.21.15.31:46516) 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 -- 172.21.15.31:0/2766286451 learned_addr learned my addr 172.21.15.31:0/2766286451 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8103700 0x7f49f81987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f8106020 0x7f49f819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 -- 172.21.15.31:0/2766286451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8103700 msgr2=0x7f49f81987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8103700 0x7f49f81987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 -- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f8106020 msgr2=0x7f49f819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f8106020 0x7f49f819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fd59b700 1 -- 172.21.15.31:0/2766286451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e80095f0 con 0x7f49f8100de0 2023-07-29T16:24:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.042+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8103700 0x7f49f81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.043+0000 7f49fd59b700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 0x7f49f81982b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f49f400e9a0 tx=0x7f49f400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.043+0000 7f49ee7fc700 1 -- 172.21.15.31:0/2766286451 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4009790 con 0x7f49f8100de0 2023-07-29T16:24:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.043+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f81a03e0 con 0x7f49f8100de0 2023-07-29T16:24:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.043+0000 7f49ee7fc700 1 -- 172.21.15.31:0/2766286451 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49f4010430 con 0x7f49f8100de0 2023-07-29T16:24:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.043+0000 7f49ee7fc700 1 -- 172.21.15.31:0/2766286451 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4018680 con 0x7f49f8100de0 2023-07-29T16:24:03.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.043+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f81a08f0 con 0x7f49f8100de0 2023-07-29T16:24:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.046+0000 7f49ee7fc700 1 -- 172.21.15.31:0/2766286451 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f49f40105a0 con 0x7f49f8100de0 2023-07-29T16:24:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 Processor -- start 2023-07-29T16:24:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 -- start start 2023-07-29T16:24:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f49ee7fc700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49e40a0790 0x7f49e40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f49ee7fc700 1 -- 172.21.15.31:0/2766286451 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f49f4014070 con 0x7f49f8100de0 2023-07-29T16:24:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49e40a0790 0x7f49e40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x555568103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568104d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568105740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55556805ba10 con 0x555568104530 2023-07-29T16:24:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555680593d0 con 0x5555681052f0 2023-07-29T16:24:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.048+0000 7f4a13041b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555568054cc0 con 0x555568103be0 2023-07-29T16:24:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x555568103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568105740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x555568103ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48610/0 (socket says 172.21.15.31:48610) 2023-07-29T16:24:03.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 -- 172.21.15.31:0/528755979 learned_addr learned my addr 172.21.15.31:0/528755979 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 -- 172.21.15.31:0/528755979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 msgr2=0x555568105740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568105740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568104d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 -- 172.21.15.31:0/528755979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 msgr2=0x555568104d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568104d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 -- 172.21.15.31:0/528755979 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555567e82de0 con 0x555568103be0 2023-07-29T16:24:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:03.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49e40a0790 0x7f49e40a2c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f49f000f870 tx=0x7f49f000fc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49fd59b700 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x555568103ff0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f49f4099410 tx=0x7f49f4099010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6638d29ba034248c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.049+0000 7f49e37fe700 1 -- 172.21.15.31:0/528755979 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4099480 con 0x555568103be0 2023-07-29T16:24:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f49e37fe700 1 -- 172.21.15.31:0/528755979 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49f409d4b0 con 0x555568103be0 2023-07-29T16:24:03.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f49e37fe700 1 -- 172.21.15.31:0/528755979 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f4098970 con 0x555568103be0 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 -- 172.21.15.31:0/528755979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 msgr2=0x555568103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x555568103ff0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f49f4099410 tx=0x7f49f4099010 comp rx=0 tx=0).stop 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 -- 172.21.15.31:0/528755979 shutdown_connections 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x555568103ff0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 --2- 172.21.15.31:0/528755979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 -- 172.21.15.31:0/528755979 >> 172.21.15.31:0/528755979 conn(0x5555680f93e0 msgr2=0x5555680fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.050+0000 7f4a13041b80 1 -- 172.21.15.31:0/528755979 shutdown_connections 2023-07-29T16:24:03.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.051+0000 7f4a13041b80 1 -- 172.21.15.31:0/528755979 wait complete. 2023-07-29T16:24:03.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.051+0000 7f4a13041b80 1 -- start start 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.051+0000 7f4a13041b80 1 Processor -- start 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x55556810de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568114e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555567ff1510 con 0x555568103be0 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555567ff11b0 con 0x5555681052f0 2023-07-29T16:24:03.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555567ed8920 con 0x555568104530 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f4a13041b80 10 client.0 _ll_register_callbacks cb 0x555567ffa620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568114e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46524/0 (socket says 172.21.15.31:46524) 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f49fdd9c700 1 -- 172.21.15.31:0/2259101333 learned_addr learned my addr 172.21.15.31:0/2259101333 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x55556810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.052+0000 7f49fcd9a700 1 -- 172.21.15.31:0/2259101333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 msgr2=0x555568114e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568114e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49fcd9a700 1 -- 172.21.15.31:0/2259101333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 msgr2=0x55556810d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556810d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49fcd9a700 1 -- 172.21.15.31:0/2259101333 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f81a08f0 con 0x555568104530 2023-07-29T16:24:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x55556810de00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f49e8000c00 tx=0x7f49e8005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1ac805ea19140d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49e0ff9700 1 -- 172.21.15.31:0/2259101333 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8004910 con 0x555568104530 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49e0ff9700 1 -- 172.21.15.31:0/2259101333 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f49e80056c0 con 0x555568104530 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55556807fd10 con 0x555568104530 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f49e0ff9700 1 -- 172.21.15.31:0/2259101333 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49e8004a70 con 0x555568104530 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f4a13041b80 10 client.28081 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.053+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555567e6bc60 con 0x555568104530 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f49e0ff9700 1 -- 172.21.15.31:0/2259101333 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55556807fd10 con 0x555568104530 2023-07-29T16:24:03.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f49e0ff9700 1 -- 172.21.15.31:0/2259101333 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f49e800fc90 con 0x555568104530 2023-07-29T16:24:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f4a13041b80 10 client.28081 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f4a13041b80 10 client.28081 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:24:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f4a13041b80 20 client.28081 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f4a13041b80 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5555681194f0 0x55556811b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f4a13041b80 4 client.28081 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5555680d2d70 con 0x5555681194f0 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.054+0000 7f49fd59b700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5555681194f0 0x55556811b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.055+0000 7f49e0ff9700 10 client.28081 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.055+0000 7f49fd59b700 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5555681194f0 0x55556811b900 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd4fee47e00eb64 server_cookie=dd12c1e980a9fd40 in_seq=0 out_seq=0 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.055+0000 7f49e0ff9700 1 -- 172.21.15.31:0/2259101333 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5555680d2d70 con 0x5555681194f0 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.055+0000 7f49e0ff9700 10 client.28081 handle_command_reply: tid=0 2023-07-29T16:24:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.055+0000 7f4a13041b80 1 client.28081 shutdown 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 msgr2=0x55556810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x55556810de00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f49e8000c00 tx=0x7f49e8005270 comp rx=0 tx=0).stop 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 shutdown_connections 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5555681194f0 0x55556811b900 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 --2- 172.21.15.31:0/2259101333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x55556810de00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f49e0ff9700 0 client.28081 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 >> 172.21.15.31:0/2259101333 conn(0x5555680f93e0 msgr2=0x5555680fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 shutdown_connections 2023-07-29T16:24:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 1 -- 172.21.15.31:0/2259101333 wait complete. 2023-07-29T16:24:03.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.056+0000 7f4a13041b80 20 client.28081 trim_cache size 0 max 16384 2023-07-29T16:24:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.070+0000 7f4a13041b80 1 Processor -- start 2023-07-29T16:24:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.070+0000 7f4a13041b80 1 -- start start 2023-07-29T16:24:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556811b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x555568112df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x5555681056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f4a13041b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55556810d430 con 0x555568103be0 2023-07-29T16:24:03.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f4a13041b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555680fb460 con 0x5555681052f0 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f4a13041b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555568119020 con 0x555568104530 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556811b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x5555681056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fcd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x555568112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556811b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55564/0 (socket says 172.21.15.31:55564) 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x5555681056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46538/0 (socket says 172.21.15.31:46538) 2023-07-29T16:24:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fcd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x555568112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48630/0 (socket says 172.21.15.31:48630) 2023-07-29T16:24:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fd59b700 1 -- 172.21.15.31:0/174984281 learned_addr learned my addr 172.21.15.31:0/174984281 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fcd9a700 1 -- 172.21.15.31:0/174984281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 msgr2=0x5555681056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x5555681056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fcd9a700 1 -- 172.21.15.31:0/174984281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 msgr2=0x55556811b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.071+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556811b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.072+0000 7f49fcd9a700 1 -- 172.21.15.31:0/174984281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555567ea9250 con 0x555568104530 2023-07-29T16:24:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.072+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x555568112df0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49e80098e0 tx=0x7f49e8009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db5c23590ebc3439 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.072+0000 7f49e3fff700 1 -- 172.21.15.31:0/174984281 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8005bc0 con 0x555568104530 2023-07-29T16:24:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.072+0000 7f49e3fff700 1 -- 172.21.15.31:0/174984281 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49e801e050 con 0x555568104530 2023-07-29T16:24:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.072+0000 7f49e3fff700 1 -- 172.21.15.31:0/174984281 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e802c860 con 0x555568104530 2023-07-29T16:24:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 -- 172.21.15.31:0/174984281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 msgr2=0x555568112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x555568112df0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49e80098e0 tx=0x7f49e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 -- 172.21.15.31:0/174984281 shutdown_connections 2023-07-29T16:24:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x5555681056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568103be0 0x55556811b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 --2- 172.21.15.31:0/174984281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568104530 0x555568112df0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 -- 172.21.15.31:0/174984281 >> 172.21.15.31:0/174984281 conn(0x5555680f93e0 msgr2=0x55556810c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 -- 172.21.15.31:0/174984281 shutdown_connections 2023-07-29T16:24:03.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.073+0000 7f4a13041b80 1 -- 172.21.15.31:0/174984281 wait complete. 2023-07-29T16:24:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.074+0000 7f4a13041b80 1 -- start start 2023-07-29T16:24:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.074+0000 7f4a13041b80 1 Processor -- start 2023-07-29T16:24:03.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.074+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x5555681107f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.074+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568110d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.074+0000 7f4a13041b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568111d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f4a13041b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555568112280 con 0x555568104530 2023-07-29T16:24:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f4a13041b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555681123f0 con 0x5555681052f0 2023-07-29T16:24:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f4a13041b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555681125b0 con 0x555568103be0 2023-07-29T16:24:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f4a13041b80 10 client.0 _ll_register_callbacks cb 0x555567ffa620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568111d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568111d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46554/0 (socket says 172.21.15.31:46554) 2023-07-29T16:24:03.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 -- 172.21.15.31:0/576526160 learned_addr learned my addr 172.21.15.31:0/576526160 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fd59b700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x5555681107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568110d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 -- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 msgr2=0x5555681107f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x5555681107f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 -- 172.21.15.31:0/576526160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 msgr2=0x555568110d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568110d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fdd9c700 1 -- 172.21.15.31:0/576526160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555567e6bc60 con 0x5555681052f0 2023-07-29T16:24:03.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fd59b700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x5555681107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:03.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.075+0000 7f49fcd9a700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568110d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:03.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f49fdd9c700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568111d40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f49f00094a0 tx=0x7f49f0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62bbe51f3090c57b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f49e0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f49e0ff9700 1 -- 172.21.15.31:0/576526160 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49f000e040 con 0x5555681052f0 2023-07-29T16:24:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49f409d4b0 con 0x5555681052f0 2023-07-29T16:24:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f49e0ff9700 1 -- 172.21.15.31:0/576526160 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f49f0003c00 con 0x5555681052f0 2023-07-29T16:24:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f49e0ff9700 1 -- 172.21.15.31:0/576526160 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49f0008e90 con 0x5555681052f0 2023-07-29T16:24:03.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f4a13041b80 10 client.28079 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555567e812b0 con 0x5555681052f0 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.076+0000 7f49e0ff9700 1 -- 172.21.15.31:0/576526160 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49f409d4b0 con 0x5555681052f0 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f49e0ff9700 1 -- 172.21.15.31:0/576526160 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f49f001a440 con 0x5555681052f0 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f4a13041b80 10 client.28079 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f4a13041b80 10 client.28079 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f4a13041b80 20 client.28079 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f4a13041b80 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55556816bc80 0x55556816e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f4a13041b80 4 client.28079 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55556804dd40 con 0x55556816bc80 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f49fd59b700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55556816bc80 0x55556816e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f49fd59b700 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55556816bc80 0x55556816e090 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e44f23a0fe1f6bdd server_cookie=37bb6df68573a938 in_seq=0 out_seq=0 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.077+0000 7f49e0ff9700 10 client.28079 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f49e0ff9700 1 -- 172.21.15.31:0/576526160 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55556804dd40 con 0x55556816bc80 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f49e0ff9700 10 client.28079 handle_command_reply: tid=0 2023-07-29T16:24:03.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 client.28079 shutdown 2023-07-29T16:24:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 msgr2=0x555568111d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568111d40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f49f00094a0 tx=0x7f49f0004b40 comp rx=0 tx=0).stop 2023-07-29T16:24:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 shutdown_connections 2023-07-29T16:24:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55556816bc80 0x55556816e090 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555681052f0 0x555568111d40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555568103be0 0x5555681107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 --2- 172.21.15.31:0/576526160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555568104530 0x555568110d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f49e0ff9700 0 client.28079 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 >> 172.21.15.31:0/576526160 conn(0x5555680f93e0 msgr2=0x55556810b700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 shutdown_connections 2023-07-29T16:24:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.078+0000 7f4a13041b80 1 -- 172.21.15.31:0/576526160 wait complete. 2023-07-29T16:24:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f4a13041b80 20 client.28079 trim_cache size 0 max 16384 2023-07-29T16:24:03.089 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:24:03.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49e40a0790 msgr2=0x7f49e40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49e40a0790 0x7f49e40a2c40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f49f000f870 tx=0x7f49f000fc30 comp rx=0 tx=0).stop 2023-07-29T16:24:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 msgr2=0x7f49f81982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 0x7f49f81982b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f49f400e9a0 tx=0x7f49f400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 shutdown_connections 2023-07-29T16:24:03.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49e40a0790 0x7f49e40a2c40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49f8106020 0x7f49f819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49f8100de0 0x7f49f81982b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 --2- 172.21.15.31:0/2766286451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49f8103700 0x7f49f81987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 >> 172.21.15.31:0/2766286451 conn(0x7f49f80fa8a0 msgr2=0x7f49f80ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 shutdown_connections 2023-07-29T16:24:03.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.079+0000 7f49ff7ff700 1 -- 172.21.15.31:0/2766286451 wait complete. 2023-07-29T16:24:03.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:24:03.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 -- 172.21.15.31:0/1605954845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c100e00 msgr2=0x7f692c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 --2- 172.21.15.31:0/1605954845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c100e00 0x7f692c1031e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f691c009990 tx=0x7f691c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 -- 172.21.15.31:0/1605954845 shutdown_connections 2023-07-29T16:24:03.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 --2- 172.21.15.31:0/1605954845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 0x7f692c108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 --2- 172.21.15.31:0/1605954845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c100e00 0x7f692c1031e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 --2- 172.21.15.31:0/1605954845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c103720 0x7f692c105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.341+0000 7f6930bff700 1 -- 172.21.15.31:0/1605954845 >> 172.21.15.31:0/1605954845 conn(0x7f692c0fa840 msgr2=0x7f692c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.342+0000 7f6930bff700 1 -- 172.21.15.31:0/1605954845 shutdown_connections 2023-07-29T16:24:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.342+0000 7f6930bff700 1 -- 172.21.15.31:0/1605954845 wait complete. 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 Processor -- start 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 -- start start 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c100e00 0x7f692c100770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 0x7f692c0fedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 0x7f692c0ff2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c0ffa90 con 0x7f692c106040 2023-07-29T16:24:03.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c0ffc00 con 0x7f692c103720 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.343+0000 7f6930bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c1a0110 con 0x7f692c100e00 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f692a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c100e00 0x7f692c100770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 0x7f692c0fedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f692a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c100e00 0x7f692c100770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48654/0 (socket says 172.21.15.31:48654) 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 0x7f692c0fedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46570/0 (socket says 172.21.15.31:46570) 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f692a59c700 1 -- 172.21.15.31:0/1828938924 learned_addr learned my addr 172.21.15.31:0/1828938924 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f692ad9d700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 0x7f692c0ff2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 -- 172.21.15.31:0/1828938924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c100e00 msgr2=0x7f692c100770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c100e00 0x7f692c100770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 -- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 msgr2=0x7f692c0ff2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 0x7f692c0ff2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 -- 172.21.15.31:0/1828938924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924009c40 con 0x7f692c103720 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f692ad9d700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 0x7f692c0ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.344+0000 7f6929d9b700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 0x7f692c0fedb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f692000e9f0 tx=0x7f692000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.345+0000 7f691b7fe700 1 -- 172.21.15.31:0/1828938924 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692000eee0 con 0x7f692c103720 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.345+0000 7f691b7fe700 1 -- 172.21.15.31:0/1828938924 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f692000cbd0 con 0x7f692c103720 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.345+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c0095f0 con 0x7f692c103720 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.345+0000 7f691b7fe700 1 -- 172.21.15.31:0/1828938924 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920018680 con 0x7f692c103720 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.345+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c1a07c0 con 0x7f692c103720 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.348+0000 7f691b7fe700 1 -- 172.21.15.31:0/1828938924 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f692000cd40 con 0x7f692c103720 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.349+0000 7f6944427b80 1 Processor -- start 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.349+0000 7f6944427b80 1 -- start start 2023-07-29T16:24:03.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.349+0000 7f6944427b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7baff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.349+0000 7f6944427b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7bbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.349+0000 7f6944427b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f6944427b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee712a10 con 0x5601ee7bb530 2023-07-29T16:24:03.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f6944427b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7103d0 con 0x5601ee7babe0 2023-07-29T16:24:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f6944427b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee70bcc0 con 0x5601ee7bc2f0 2023-07-29T16:24:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7baff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7baff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46584/0 (socket says 172.21.15.31:46584) 2023-07-29T16:24:03.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 -- 172.21.15.31:0/134253257 learned_addr learned my addr 172.21.15.31:0/134253257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f6929d9b700 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692a59c700 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7bbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 -- 172.21.15.31:0/134253257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bc2f0 msgr2=0x5601ee7bc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 -- 172.21.15.31:0/134253257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 msgr2=0x5601ee7bbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7bbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f692ad9d700 1 -- 172.21.15.31:0/134253257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601ee539de0 con 0x5601ee7babe0 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.350+0000 7f691b7fe700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f69140a0810 0x7f69140a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.351+0000 7f692ad9d700 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7baff0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6924007550 tx=0x7f6924009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5694f6bac1526277 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.351+0000 7f691b7fe700 1 -- 172.21.15.31:0/1828938924 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6920014070 con 0x7f692c103720 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.351+0000 7f692a59c700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f69140a0810 0x7f69140a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.351+0000 7f6918ff9700 1 -- 172.21.15.31:0/134253257 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692400e070 con 0x5601ee7babe0 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.351+0000 7f6918ff9700 1 -- 172.21.15.31:0/134253257 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f692400f040 con 0x5601ee7babe0 2023-07-29T16:24:03.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.351+0000 7f6918ff9700 1 -- 172.21.15.31:0/134253257 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6924014ac0 con 0x5601ee7babe0 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 -- 172.21.15.31:0/134253257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 msgr2=0x5601ee7baff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f692a59c700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f69140a0810 0x7f69140a2cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x5601ee7bb940 tx=0x7f691c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7baff0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6924007550 tx=0x7f6924009920 comp rx=0 tx=0).stop 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 -- 172.21.15.31:0/134253257 shutdown_connections 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7baff0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 --2- 172.21.15.31:0/134253257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7bbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 -- 172.21.15.31:0/134253257 >> 172.21.15.31:0/134253257 conn(0x5601ee7b03e0 msgr2=0x5601ee7b5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 -- 172.21.15.31:0/134253257 shutdown_connections 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 -- 172.21.15.31:0/134253257 wait complete. 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.352+0000 7f6944427b80 1 -- start start 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 Processor -- start 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 0x5601ee7c4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7cbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee6a8510 con 0x5601ee7bc2f0 2023-07-29T16:24:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee6a81b0 con 0x5601ee7bb530 2023-07-29T16:24:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee58f920 con 0x5601ee7babe0 2023-07-29T16:24:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6944427b80 10 client.0 _ll_register_callbacks cb 0x5601ee6b1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6929d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 0x5601ee7c4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6929d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 0x5601ee7c4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46596/0 (socket says 172.21.15.31:46596) 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f6929d9b700 1 -- 172.21.15.31:0/1574515665 learned_addr learned my addr 172.21.15.31:0/1574515665 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f692a59c700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.353+0000 7f692ad9d700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7cbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6929d9b700 1 -- 172.21.15.31:0/1574515665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 msgr2=0x5601ee7c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6929d9b700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6929d9b700 1 -- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 msgr2=0x5601ee7cbe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6929d9b700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7cbe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6929d9b700 1 -- 172.21.15.31:0/1574515665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6924009280 con 0x5601ee7bb530 2023-07-29T16:24:03.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f692ad9d700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:03.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6929d9b700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 0x5601ee7c4de0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6920095eb0 tx=0x7f6920099980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f6b6adb8ce4e14 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6911ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6911ffb700 1 -- 172.21.15.31:0/1574515665 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6920099210 con 0x5601ee7bb530 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6911ffb700 1 -- 172.21.15.31:0/1574515665 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6920099440 con 0x5601ee7bb530 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601ee736d10 con 0x5601ee7bb530 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6911ffb700 1 -- 172.21.15.31:0/1574515665 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6920098f80 con 0x5601ee7bb530 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.354+0000 7f6944427b80 10 client.28091 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601ee522c60 con 0x5601ee7bb530 2023-07-29T16:24:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6911ffb700 1 -- 172.21.15.31:0/1574515665 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5601ee736d10 con 0x5601ee7bb530 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6911ffb700 1 -- 172.21.15.31:0/1574515665 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f69200b1030 con 0x5601ee7bb530 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 10 client.28091 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 10 client.28091 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 20 client.28091 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee7d0700 0x5601ee7d2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 4 client.28091 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601ee789d70 con 0x5601ee7d0700 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.355+0000 7f692a59c700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee7d0700 0x5601ee7d2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.356+0000 7f692a59c700 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee7d0700 0x5601ee7d2b10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1fe7f9feedfc2317 server_cookie=6a6d03088c11ef5e in_seq=0 out_seq=0 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.356+0000 7f6911ffb700 10 client.28091 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.356+0000 7f6911ffb700 1 -- 172.21.15.31:0/1574515665 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5601ee789d70 con 0x5601ee7d0700 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.356+0000 7f6911ffb700 10 client.28091 handle_command_reply: tid=0 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 client.28091 shutdown 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 msgr2=0x5601ee7c4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 0x5601ee7c4de0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f6920095eb0 tx=0x7f6920099980 comp rx=0 tx=0).stop 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 shutdown_connections 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee7d0700 0x5601ee7d2b10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7cbe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 --2- 172.21.15.31:0/1574515665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bb530 0x5601ee7c4de0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6911ffb700 0 client.28091 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:03.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 >> 172.21.15.31:0/1574515665 conn(0x5601ee7b03e0 msgr2=0x5601ee7b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 shutdown_connections 2023-07-29T16:24:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 1 -- 172.21.15.31:0/1574515665 wait complete. 2023-07-29T16:24:03.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.357+0000 7f6944427b80 20 client.28091 trim_cache size 0 max 16384 2023-07-29T16:24:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 Processor -- start 2023-07-29T16:24:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 -- start start 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7d7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7c5520 con 0x5601ee7bb530 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7ad6e0 con 0x5601ee7bc2f0 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6944427b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7c2540 con 0x5601ee7babe0 2023-07-29T16:24:03.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f692ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f692a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7d7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f692ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46610/0 (socket says 172.21.15.31:46610) 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f692ad9d700 1 -- 172.21.15.31:0/3908033845 learned_addr learned my addr 172.21.15.31:0/3908033845 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.371+0000 7f6929d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f692a59c700 1 -- 172.21.15.31:0/3908033845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bc2f0 msgr2=0x5601ee7bc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f692a59c700 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f692a59c700 1 -- 172.21.15.31:0/3908033845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 msgr2=0x5601ee7cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f692a59c700 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f692a59c700 1 -- 172.21.15.31:0/3908033845 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601ee560250 con 0x5601ee7babe0 2023-07-29T16:24:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f692a59c700 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7d7470 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x5601ee7d33f0 tx=0x7f691c01a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d1856b23d677b21 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.372+0000 7f69197fa700 1 -- 172.21.15.31:0/3908033845 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c034250 con 0x5601ee7babe0 2023-07-29T16:24:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f69197fa700 1 -- 172.21.15.31:0/3908033845 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f691c01e040 con 0x5601ee7babe0 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f69197fa700 1 -- 172.21.15.31:0/3908033845 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c004c60 con 0x5601ee7babe0 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 -- 172.21.15.31:0/3908033845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 msgr2=0x5601ee7d7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7d7470 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x5601ee7d33f0 tx=0x7f691c01a050 comp rx=0 tx=0).stop 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 -- 172.21.15.31:0/3908033845 shutdown_connections 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7bc2f0 0x5601ee7bc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7babe0 0x5601ee7d7470 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 --2- 172.21.15.31:0/3908033845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bb530 0x5601ee7cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 -- 172.21.15.31:0/3908033845 >> 172.21.15.31:0/3908033845 conn(0x5601ee7b03e0 msgr2=0x5601ee7b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 -- 172.21.15.31:0/3908033845 shutdown_connections 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.373+0000 7f6944427b80 1 -- 172.21.15.31:0/3908033845 wait complete. 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.374+0000 7f6944427b80 1 -- start start 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.374+0000 7f6944427b80 1 Processor -- start 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bb530 0x5601ee7c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7cc190 con 0x5601ee7bc2f0 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7caa00 con 0x5601ee7babe0 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601ee7cabc0 con 0x5601ee7bb530 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6944427b80 10 client.0 _ll_register_callbacks cb 0x5601ee6b1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46616/0 (socket says 172.21.15.31:46616) 2023-07-29T16:24:03.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 -- 172.21.15.31:0/3340361348 learned_addr learned my addr 172.21.15.31:0/3340361348 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692ad9d700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f6929d9b700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bb530 0x5601ee7c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 -- 172.21.15.31:0/3340361348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bb530 msgr2=0x5601ee7c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bb530 0x5601ee7c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 -- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 msgr2=0x5601ee7ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.375+0000 7f692a59c700 1 -- 172.21.15.31:0/3340361348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c109030 con 0x5601ee7babe0 2023-07-29T16:24:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f692a59c700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7cb970 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f691c00bef0 tx=0x7f691c004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf2787bfbc117df server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6911ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6911ffb700 1 -- 172.21.15.31:0/3340361348 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c004960 con 0x5601ee7babe0 2023-07-29T16:24:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f692400f040 con 0x5601ee7babe0 2023-07-29T16:24:03.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6911ffb700 1 -- 172.21.15.31:0/3340361348 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f691c004ac0 con 0x5601ee7babe0 2023-07-29T16:24:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6911ffb700 1 -- 172.21.15.31:0/3340361348 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f691c004c80 con 0x5601ee7babe0 2023-07-29T16:24:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6944427b80 10 client.28097 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.376+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601ee5382b0 con 0x5601ee7babe0 2023-07-29T16:24:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6911ffb700 1 -- 172.21.15.31:0/3340361348 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f692400f040 con 0x5601ee7babe0 2023-07-29T16:24:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6911ffb700 1 -- 172.21.15.31:0/3340361348 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f691c026830 con 0x5601ee7babe0 2023-07-29T16:24:03.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6944427b80 10 client.28097 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6944427b80 10 client.28097 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6944427b80 20 client.28097 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6944427b80 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee8232c0 0x5601ee8256d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6944427b80 4 client.28097 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5601ee704d40 con 0x5601ee8232c0 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.377+0000 7f6929d9b700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee8232c0 0x5601ee8256d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6929d9b700 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee8232c0 0x5601ee8256d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7755f9577d739c72 server_cookie=d1ba722688521a3f in_seq=0 out_seq=0 2023-07-29T16:24:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6911ffb700 10 client.28097 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6911ffb700 1 -- 172.21.15.31:0/3340361348 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5601ee704d40 con 0x5601ee8232c0 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6911ffb700 10 client.28097 handle_command_reply: tid=0 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 client.28097 shutdown 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 msgr2=0x5601ee7cb970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7cb970 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f691c00bef0 tx=0x7f691c004600 comp rx=0 tx=0).stop 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 shutdown_connections 2023-07-29T16:24:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5601ee8232c0 0x5601ee8256d0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5601ee7bc2f0 0x5601ee7ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6911ffb700 0 client.28097 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5601ee7babe0 0x5601ee7cb970 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 --2- 172.21.15.31:0/3340361348 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601ee7bb530 0x5601ee7c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 >> 172.21.15.31:0/3340361348 conn(0x5601ee7b03e0 msgr2=0x5601ee7c2790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 shutdown_connections 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.378+0000 7f6944427b80 1 -- 172.21.15.31:0/3340361348 wait complete. 2023-07-29T16:24:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6944427b80 20 client.28097 trim_cache size 0 max 16384 2023-07-29T16:24:03.389 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:24:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f69140a0810 msgr2=0x7f69140a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f69140a0810 0x7f69140a2cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x5601ee7bb940 tx=0x7f691c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 msgr2=0x7f692c0fedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 0x7f692c0fedb0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f692000e9f0 tx=0x7f692000ed00 comp rx=0 tx=0).stop 2023-07-29T16:24:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 shutdown_connections 2023-07-29T16:24:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f69140a0810 0x7f69140a2cc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f692c106040 0x7f692c0ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f692c100e00 0x7f692c100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 --2- 172.21.15.31:0/1828938924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f692c103720 0x7f692c0fedb0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 >> 172.21.15.31:0/1828938924 conn(0x7f692c0fa840 msgr2=0x7f692c101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.379+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 shutdown_connections 2023-07-29T16:24:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.380+0000 7f6930bff700 1 -- 172.21.15.31:0/1828938924 wait complete. 2023-07-29T16:24:03.397 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:24:03.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 -- 172.21.15.31:0/384910341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec100e10 msgr2=0x7fdaec1031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/384910341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec100e10 0x7fdaec1031f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009990 tx=0x7fdadc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 -- 172.21.15.31:0/384910341 shutdown_connections 2023-07-29T16:24:03.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/384910341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaec106050 0x7fdaec1084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/384910341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec100e10 0x7fdaec1031f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/384910341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec103730 0x7fdaec105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 -- 172.21.15.31:0/384910341 >> 172.21.15.31:0/384910341 conn(0x7fdaec0fa8b0 msgr2=0x7fdaec0fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 -- 172.21.15.31:0/384910341 shutdown_connections 2023-07-29T16:24:03.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.618+0000 7fdaf1bff700 1 -- 172.21.15.31:0/384910341 wait complete. 2023-07-29T16:24:03.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.619+0000 7fdaf1bff700 1 Processor -- start 2023-07-29T16:24:03.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 -- start start 2023-07-29T16:24:03.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 0x7fdaec100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaec103730 0x7fdaec0fedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec106050 0x7fdaec0ff300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0ffaa0 con 0x7fdaec103730 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0ffc10 con 0x7fdaec106050 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaf1bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec1a0060 con 0x7fdaec100e10 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 0x7fdaec100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 0x7fdaec100780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48712/0 (socket says 172.21.15.31:48712) 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.620+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/777624937 learned_addr learned my addr 172.21.15.31:0/777624937 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaec103730 0x7fdaec0fedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaebfff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec106050 0x7fdaec0ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/777624937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec106050 msgr2=0x7fdaec0ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec106050 0x7fdaec0ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaec103730 msgr2=0x7fdaec0fedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaec103730 0x7fdaec0fedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/777624937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae0009520 con 0x7fdaec100e10 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaebfff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec106050 0x7fdaec0ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 0x7fdaec100780 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00b680 tx=0x7fdadc005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.621+0000 7fdae8ff9700 1 -- 172.21.15.31:0/777624937 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc01d070 con 0x7fdaec100e10 2023-07-29T16:24:03.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.622+0000 7fdae8ff9700 1 -- 172.21.15.31:0/777624937 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdadc022470 con 0x7fdaec100e10 2023-07-29T16:24:03.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.622+0000 7fdae8ff9700 1 -- 172.21.15.31:0/777624937 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc017650 con 0x7fdaec100e10 2023-07-29T16:24:03.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.622+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc0095f0 con 0x7fdaec100e10 2023-07-29T16:24:03.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.622+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec1a0520 con 0x7fdaec100e10 2023-07-29T16:24:03.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.625+0000 7fdae8ff9700 1 -- 172.21.15.31:0/777624937 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdadc0225e0 con 0x7fdaec100e10 2023-07-29T16:24:03.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.627+0000 7fdae8ff9700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdad40a0790 0x7fdad40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.627+0000 7fdae8ff9700 1 -- 172.21.15.31:0/777624937 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdadc0ccfb0 con 0x7fdaec100e10 2023-07-29T16:24:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.627+0000 7fdb05582b80 1 Processor -- start 2023-07-29T16:24:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.627+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdad40a0790 0x7fdad40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.627+0000 7fdb05582b80 1 -- start start 2023-07-29T16:24:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f07be0 0x55df52f07ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f09740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdb05582b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52e5fa10 con 0x55df52f07be0 2023-07-29T16:24:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdb05582b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52e5d3d0 con 0x55df52f08530 2023-07-29T16:24:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdb05582b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52e58cc0 con 0x55df52f092f0 2023-07-29T16:24:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdad40a0790 0x7fdad40a2c40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009df0 tx=0x7fdae0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f08d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46644/0 (socket says 172.21.15.31:46644) 2023-07-29T16:24:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 -- 172.21.15.31:0/1459636008 learned_addr learned my addr 172.21.15.31:0/1459636008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaebfff700 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f09740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f07be0 0x55df52f07ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 -- 172.21.15.31:0/1459636008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 msgr2=0x55df52f09740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f09740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 -- 172.21.15.31:0/1459636008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f07be0 msgr2=0x55df52f07ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f07be0 0x55df52f07ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.628+0000 7fdaeaffd700 1 -- 172.21.15.31:0/1459636008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df52c86de0 con 0x55df52f08530 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.629+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f08d60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdae000e600 tx=0x7fdae0014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b21e083edae9631b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.629+0000 7fdad1ffb700 1 -- 172.21.15.31:0/1459636008 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae0004220 con 0x55df52f08530 2023-07-29T16:24:03.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.629+0000 7fdad1ffb700 1 -- 172.21.15.31:0/1459636008 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae0021070 con 0x55df52f08530 2023-07-29T16:24:03.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.629+0000 7fdad1ffb700 1 -- 172.21.15.31:0/1459636008 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae00266e0 con 0x55df52f08530 2023-07-29T16:24:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 -- 172.21.15.31:0/1459636008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 msgr2=0x55df52f08d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f08d60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdae000e600 tx=0x7fdae0014000 comp rx=0 tx=0).stop 2023-07-29T16:24:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 -- 172.21.15.31:0/1459636008 shutdown_connections 2023-07-29T16:24:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f09740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f07be0 0x55df52f07ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 --2- 172.21.15.31:0/1459636008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f08d60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 -- 172.21.15.31:0/1459636008 >> 172.21.15.31:0/1459636008 conn(0x55df52efd3e0 msgr2=0x55df52f02990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 -- 172.21.15.31:0/1459636008 shutdown_connections 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 -- 172.21.15.31:0/1459636008 wait complete. 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.630+0000 7fdb05582b80 1 -- start start 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 Processor -- start 2023-07-29T16:24:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f11850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f08530 0x55df52f11d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f18dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52df5510 con 0x55df52f092f0 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52df51b0 con 0x55df52f07be0 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52cdc920 con 0x55df52f08530 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdb05582b80 10 client.0 _ll_register_callbacks cb 0x55df52dfe620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f11850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f11850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46656/0 (socket says 172.21.15.31:46656) 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/2874961868 learned_addr learned my addr 172.21.15.31:0/2874961868 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f08530 0x55df52f11d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.631+0000 7fdaebfff700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f18dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/2874961868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f08530 msgr2=0x55df52f11d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f08530 0x55df52f11d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/2874961868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 msgr2=0x55df52f18dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f18dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/2874961868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae0017070 con 0x55df52f07be0 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaebfff700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f18dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f11850 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdadc005910 tx=0x7fdadc022bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=545625707613935 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdacaffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdacaffd700 1 -- 172.21.15.31:0/2874961868 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc099dd0 con 0x55df52f07be0 2023-07-29T16:24:03.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55df52e83d10 con 0x55df52f07be0 2023-07-29T16:24:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdacaffd700 1 -- 172.21.15.31:0/2874961868 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdadc09a6e0 con 0x55df52f07be0 2023-07-29T16:24:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.632+0000 7fdacaffd700 1 -- 172.21.15.31:0/2874961868 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdadc09e710 con 0x55df52f07be0 2023-07-29T16:24:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 10 client.28109 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55df52c6fc60 con 0x55df52f07be0 2023-07-29T16:24:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdacaffd700 1 -- 172.21.15.31:0/2874961868 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55df52e83d10 con 0x55df52f07be0 2023-07-29T16:24:03.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdacaffd700 1 -- 172.21.15.31:0/2874961868 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdadc00fba0 con 0x55df52f07be0 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 10 client.28109 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 10 client.28109 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 20 client.28109 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f1d1a0 0x55df52f1f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 4 client.28109 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df52ed6d70 con 0x55df52f1d1a0 2023-07-29T16:24:03.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.633+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f1d1a0 0x55df52f1f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.634+0000 7fdacaffd700 10 client.28109 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.634+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f1d1a0 0x55df52f1f580 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7119df9cef215c6 server_cookie=cc74fcbdf729ab22 in_seq=0 out_seq=0 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdacaffd700 1 -- 172.21.15.31:0/2874961868 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55df52ed6d70 con 0x55df52f1d1a0 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdacaffd700 10 client.28109 handle_command_reply: tid=0 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 client.28109 shutdown 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 msgr2=0x55df52f11850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f11850 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdadc005910 tx=0x7fdadc022bc0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 shutdown_connections 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f1d1a0 0x55df52f1f580 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f18dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f11850 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdacaffd700 0 client.28109 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2874961868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f08530 0x55df52f11d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 >> 172.21.15.31:0/2874961868 conn(0x55df52efd3e0 msgr2=0x55df52f02060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 shutdown_connections 2023-07-29T16:24:03.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.635+0000 7fdb05582b80 1 -- 172.21.15.31:0/2874961868 wait complete. 2023-07-29T16:24:03.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.636+0000 7fdb05582b80 20 client.28109 trim_cache size 0 max 16384 2023-07-29T16:24:03.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 Processor -- start 2023-07-29T16:24:03.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 -- start start 2023-07-29T16:24:03.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f07be0 0x55df52f1f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f18d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52f09c00 con 0x55df52f092f0 2023-07-29T16:24:03.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52eff460 con 0x55df52f08530 2023-07-29T16:24:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.650+0000 7fdb05582b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52f03cf0 con 0x55df52f07be0 2023-07-29T16:24:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f18d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f07be0 0x55df52f1f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f18d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46660/0 (socket says 172.21.15.31:46660) 2023-07-29T16:24:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 -- 172.21.15.31:0/2126223729 learned_addr learned my addr 172.21.15.31:0/2126223729 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaebfff700 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 -- 172.21.15.31:0/2126223729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f07be0 msgr2=0x55df52f1f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f07be0 0x55df52f1f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 -- 172.21.15.31:0/2126223729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 msgr2=0x55df52f096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 -- 172.21.15.31:0/2126223729 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df52cad250 con 0x55df52f08530 2023-07-29T16:24:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f18d90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55df52f08940 tx=0x7fdae0011ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90e0cf89eb31fc0b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.651+0000 7fdad27fc700 1 -- 172.21.15.31:0/2126223729 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae002a040 con 0x55df52f08530 2023-07-29T16:24:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdad27fc700 1 -- 172.21.15.31:0/2126223729 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae0003b20 con 0x55df52f08530 2023-07-29T16:24:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdad27fc700 1 -- 172.21.15.31:0/2126223729 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae00052b0 con 0x55df52f08530 2023-07-29T16:24:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 -- 172.21.15.31:0/2126223729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 msgr2=0x55df52f18d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f18d90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x55df52f08940 tx=0x7fdae0011ad0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 -- 172.21.15.31:0/2126223729 shutdown_connections 2023-07-29T16:24:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f092f0 0x55df52f096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f07be0 0x55df52f1f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 --2- 172.21.15.31:0/2126223729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f08530 0x55df52f18d90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 -- 172.21.15.31:0/2126223729 >> 172.21.15.31:0/2126223729 conn(0x55df52efd3e0 msgr2=0x55df52f104e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 -- 172.21.15.31:0/2126223729 shutdown_connections 2023-07-29T16:24:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.652+0000 7fdb05582b80 1 -- 172.21.15.31:0/2126223729 wait complete. 2023-07-29T16:24:03.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.653+0000 7fdb05582b80 1 -- start start 2023-07-29T16:24:03.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.653+0000 7fdb05582b80 1 Processor -- start 2023-07-29T16:24:03.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.653+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f14c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.653+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f08530 0x55df52f15160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdb05582b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f18180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdb05582b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52f186c0 con 0x55df52f08530 2023-07-29T16:24:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdb05582b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52f18830 con 0x55df52f07be0 2023-07-29T16:24:03.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdb05582b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df52f189f0 con 0x55df52f092f0 2023-07-29T16:24:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdb05582b80 10 client.0 _ll_register_callbacks cb 0x55df52dfe620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f14c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f08530 0x55df52f15160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f14c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46664/0 (socket says 172.21.15.31:46664) 2023-07-29T16:24:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/593002027 learned_addr learned my addr 172.21.15.31:0/593002027 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaebfff700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f18180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/593002027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 msgr2=0x55df52f18180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f18180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/593002027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f08530 msgr2=0x55df52f15160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:03.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f08530 0x55df52f15160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 -- 172.21.15.31:0/593002027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df52c86de0 con 0x55df52f07be0 2023-07-29T16:24:03.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaebfff700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f18180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:03.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdaeb7fe700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f14c20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0a2570 tx=0x7fdadc022c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd9ef233a5ead16 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:03.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.654+0000 7fdacaffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:03.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdacaffd700 1 -- 172.21.15.31:0/593002027 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc09a6e0 con 0x55df52f07be0 2023-07-29T16:24:03.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdacaffd700 1 -- 172.21.15.31:0/593002027 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdadc09e710 con 0x55df52f07be0 2023-07-29T16:24:03.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdae0021070 con 0x55df52f07be0 2023-07-29T16:24:03.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdacaffd700 1 -- 172.21.15.31:0/593002027 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdadc0a2aa0 con 0x55df52f07be0 2023-07-29T16:24:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdb05582b80 10 client.28115 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55df52c852b0 con 0x55df52f07be0 2023-07-29T16:24:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdacaffd700 1 -- 172.21.15.31:0/593002027 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdae0021070 con 0x55df52f07be0 2023-07-29T16:24:03.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.655+0000 7fdacaffd700 1 -- 172.21.15.31:0/593002027 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdadc0c4b30 con 0x55df52f07be0 2023-07-29T16:24:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdb05582b80 10 client.28115 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdb05582b80 10 client.28115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:03.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdb05582b80 20 client.28115 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdb05582b80 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f6f7c0 0x55df52f71bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdb05582b80 4 client.28115 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:24:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55df52e51d40 con 0x55df52f6f7c0 2023-07-29T16:24:03.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f6f7c0 0x55df52f71bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdaeaffd700 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f6f7c0 0x55df52f71bd0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b47b0d7d2396d23 server_cookie=52df6634c5e205ed in_seq=0 out_seq=0 2023-07-29T16:24:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.656+0000 7fdacaffd700 10 client.28115 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdacaffd700 1 -- 172.21.15.31:0/593002027 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55df52e51d40 con 0x55df52f6f7c0 2023-07-29T16:24:03.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdacaffd700 10 client.28115 handle_command_reply: tid=0 2023-07-29T16:24:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 client.28115 shutdown 2023-07-29T16:24:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 msgr2=0x55df52f14c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f14c20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0a2570 tx=0x7fdadc022c80 comp rx=0 tx=0).stop 2023-07-29T16:24:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 shutdown_connections 2023-07-29T16:24:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55df52f6f7c0 0x55df52f71bd0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55df52f092f0 0x55df52f18180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df52f07be0 0x55df52f14c20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdacaffd700 0 client.28115 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 --2- 172.21.15.31:0/593002027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55df52f08530 0x55df52f15160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 >> 172.21.15.31:0/593002027 conn(0x55df52efd3e0 msgr2=0x55df52f0f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 shutdown_connections 2023-07-29T16:24:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 1 -- 172.21.15.31:0/593002027 wait complete. 2023-07-29T16:24:03.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.657+0000 7fdb05582b80 20 client.28115 trim_cache size 0 max 16384 2023-07-29T16:24:03.679 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:24:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdad40a0790 msgr2=0x7fdad40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdad40a0790 0x7fdad40a2c40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009df0 tx=0x7fdae0009260 comp rx=0 tx=0).stop 2023-07-29T16:24:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 msgr2=0x7fdaec100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 0x7fdaec100780 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdadc00b680 tx=0x7fdadc005740 comp rx=0 tx=0).stop 2023-07-29T16:24:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 shutdown_connections 2023-07-29T16:24:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdad40a0790 0x7fdad40a2c40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaec106050 0x7fdaec0ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdaec100e10 0x7fdaec100780 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 --2- 172.21.15.31:0/777624937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdaec103730 0x7fdaec0fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 >> 172.21.15.31:0/777624937 conn(0x7fdaec0fa8b0 msgr2=0x7fdaec101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:03.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 shutdown_connections 2023-07-29T16:24:03.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:03.658+0000 7fdaf1bff700 1 -- 172.21.15.31:0/777624937 wait complete. 2023-07-29T16:24:03.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1823661248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:24:03.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1823661248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:24:03.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1823661248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:24:03.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 578943 8.95 MB/sec execute 228 sec latency 45.621 ms 2023-07-29T16:24:04.685 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:24:04.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:24:04.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:04 smithi039 ceph-mon[128858]: pgmap v628: 97 pgs: 97 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 2.2 MiB/s wr, 299 op/s 2023-07-29T16:24:04.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:04 smithi156 ceph-mon[129245]: pgmap v628: 97 pgs: 97 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 2.2 MiB/s wr, 299 op/s 2023-07-29T16:24:04.879 INFO:tasks.workunit.client.0.smithi031.stdout: 1 579852 8.92 MB/sec execute 229 sec latency 24.599 ms 2023-07-29T16:24:04.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 -- 172.21.15.31:0/1965079960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d0103740 msgr2=0x7f39d0103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1965079960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d0103740 0x7f39d0103b90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f39c4009990 tx=0x7f39c4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 -- 172.21.15.31:0/1965079960 shutdown_connections 2023-07-29T16:24:04.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1965079960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d01040d0 0x7f39d010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1965079960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 0x7f39d0102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1965079960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d0103740 0x7f39d0103b90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 -- 172.21.15.31:0/1965079960 >> 172.21.15.31:0/1965079960 conn(0x7f39d00fdb90 msgr2=0x7f39d00fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 -- 172.21.15.31:0/1965079960 shutdown_connections 2023-07-29T16:24:04.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.919+0000 7f39d67ff700 1 -- 172.21.15.31:0/1965079960 wait complete. 2023-07-29T16:24:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.920+0000 7f39d67ff700 1 Processor -- start 2023-07-29T16:24:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 -- start start 2023-07-29T16:24:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 0x7f39d0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 0x7f39d0198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d01040d0 0x7f39d019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d019fdd0 con 0x7f39d0102550 2023-07-29T16:24:04.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d019ff10 con 0x7f39d0103740 2023-07-29T16:24:04.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d67ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d01a00a0 con 0x7f39d01040d0 2023-07-29T16:24:04.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 0x7f39d0198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 0x7f39d0198790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46678/0 (socket says 172.21.15.31:46678) 2023-07-29T16:24:04.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1810991229 learned_addr learned my addr 172.21.15.31:0/1810991229 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:04.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39cffff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 0x7f39d0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.921+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d01040d0 0x7f39d019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1810991229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d01040d0 msgr2=0x7f39d019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d01040d0 0x7f39d019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 msgr2=0x7f39d0198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 0x7f39d0198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1810991229 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c40095f0 con 0x7f39d0103740 2023-07-29T16:24:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cffff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 0x7f39d0198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 0x7f39d0198790 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f39c4004740 tx=0x7f39c4009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cd7fa700 1 -- 172.21.15.31:0/1810991229 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c401d070 con 0x7f39d0103740 2023-07-29T16:24:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d01a0320 con 0x7f39d0103740 2023-07-29T16:24:04.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cd7fa700 1 -- 172.21.15.31:0/1810991229 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c4004830 con 0x7f39d0103740 2023-07-29T16:24:04.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39cd7fa700 1 -- 172.21.15.31:0/1810991229 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4005330 con 0x7f39d0103740 2023-07-29T16:24:04.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.922+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d01a07a0 con 0x7f39d0103740 2023-07-29T16:24:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.927+0000 7f39cd7fa700 1 -- 172.21.15.31:0/1810991229 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f39c4005510 con 0x7f39d0103740 2023-07-29T16:24:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 Processor -- start 2023-07-29T16:24:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 -- start start 2023-07-29T16:24:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 0x563a63a57070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a57de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a58370 0x563a63a587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a639aea90 con 0x563a63a575b0 2023-07-29T16:24:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a639ac450 con 0x563a63a56c60 2023-07-29T16:24:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.928+0000 7f39ea057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a639a7d40 con 0x563a63a58370 2023-07-29T16:24:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 0x563a63a57070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 0x563a63a57070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46680/0 (socket says 172.21.15.31:46680) 2023-07-29T16:24:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 -- 172.21.15.31:0/3066278669 learned_addr learned my addr 172.21.15.31:0/3066278669 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39cffff700 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a57de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a58370 0x563a63a587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 -- 172.21.15.31:0/3066278669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a58370 msgr2=0x563a63a587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a58370 0x563a63a587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 -- 172.21.15.31:0/3066278669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 msgr2=0x563a63a57de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a57de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 -- 172.21.15.31:0/3066278669 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a637d5e60 con 0x563a63a56c60 2023-07-29T16:24:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39cd7fa700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39b80a0810 0x7f39b80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39cd7fa700 1 -- 172.21.15.31:0/1810991229 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39c40ccba0 con 0x7f39d0103740 2023-07-29T16:24:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.929+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 0x563a63a57070 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f39c8011680 tx=0x7f39c8009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76797d116a1fdd54 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.930+0000 7f39cffff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39b80a0810 0x7f39b80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.930+0000 7f39b67fc700 1 -- 172.21.15.31:0/3066278669 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c800e040 con 0x563a63a56c60 2023-07-29T16:24:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.930+0000 7f39b67fc700 1 -- 172.21.15.31:0/3066278669 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c800f040 con 0x563a63a56c60 2023-07-29T16:24:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.930+0000 7f39b67fc700 1 -- 172.21.15.31:0/3066278669 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c80048c0 con 0x563a63a56c60 2023-07-29T16:24:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 -- 172.21.15.31:0/3066278669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 msgr2=0x563a63a57070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 0x563a63a57070 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f39c8011680 tx=0x7f39c8009920 comp rx=0 tx=0).stop 2023-07-29T16:24:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39cffff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39b80a0810 0x7f39b80a2cc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x563a63a579c0 tx=0x7f39c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 -- 172.21.15.31:0/3066278669 shutdown_connections 2023-07-29T16:24:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a58370 0x563a63a587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a56c60 0x563a63a57070 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 --2- 172.21.15.31:0/3066278669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a57de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 -- 172.21.15.31:0/3066278669 >> 172.21.15.31:0/3066278669 conn(0x563a63a4c460 msgr2=0x563a63a51a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 -- 172.21.15.31:0/3066278669 shutdown_connections 2023-07-29T16:24:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 -- 172.21.15.31:0/3066278669 wait complete. 2023-07-29T16:24:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.931+0000 7f39ea057b80 1 -- start start 2023-07-29T16:24:04.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 Processor -- start 2023-07-29T16:24:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a608f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a60e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a67e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63944590 con 0x563a63a575b0 2023-07-29T16:24:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63944230 con 0x563a63a58370 2023-07-29T16:24:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a6382b9a0 con 0x563a63a56c60 2023-07-29T16:24:04.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39ea057b80 10 client.0 _ll_register_callbacks cb 0x563a6394d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39d4d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a67e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a60e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39d4d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a67e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46694/0 (socket says 172.21.15.31:46694) 2023-07-29T16:24:04.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39d4d9c700 1 -- 172.21.15.31:0/1207398584 learned_addr learned my addr 172.21.15.31:0/1207398584 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:04.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.932+0000 7f39cffff700 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a608f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39d4d9c700 1 -- 172.21.15.31:0/1207398584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 msgr2=0x563a63a608f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a608f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39d4d9c700 1 -- 172.21.15.31:0/1207398584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 msgr2=0x563a63a60e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a60e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39d4d9c700 1 -- 172.21.15.31:0/1207398584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c8011330 con 0x563a63a58370 2023-07-29T16:24:04.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a67e70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009540 tx=0x7f39c80095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bd5ab2ca25c42af server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:04.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39af7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:04.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39af7fe700 1 -- 172.21.15.31:0/1207398584 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c800e070 con 0x563a63a58370 2023-07-29T16:24:04.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39af7fe700 1 -- 172.21.15.31:0/1207398584 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39c8003c10 con 0x563a63a58370 2023-07-29T16:24:04.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.933+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563a639d2d90 con 0x563a63a58370 2023-07-29T16:24:04.947 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:04 smithi031 ceph-mon[95504]: pgmap v628: 97 pgs: 97 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 2.2 MiB/s wr, 299 op/s 2023-07-29T16:24:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39af7fe700 1 -- 172.21.15.31:0/1207398584 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39c8003dd0 con 0x563a63a58370 2023-07-29T16:24:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 10 client.28127 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563a637bece0 con 0x563a63a58370 2023-07-29T16:24:04.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39af7fe700 1 -- 172.21.15.31:0/1207398584 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563a639d2d90 con 0x563a63a58370 2023-07-29T16:24:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39af7fe700 1 -- 172.21.15.31:0/1207398584 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f39c80274f0 con 0x563a63a58370 2023-07-29T16:24:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 10 client.28127 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:04.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 10 client.28127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 20 client.28127 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63a6c580 0x563a63a6e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 4 client.28127 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.934+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a63a25df0 con 0x563a63a6c580 2023-07-29T16:24:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.935+0000 7f39cffff700 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63a6c580 0x563a63a6e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.935+0000 7f39af7fe700 10 client.28127 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.935+0000 7f39cffff700 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63a6c580 0x563a63a6e990 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8caf5e64683d977 server_cookie=47fa109344c553eb in_seq=0 out_seq=0 2023-07-29T16:24:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39af7fe700 1 -- 172.21.15.31:0/1207398584 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563a63a25df0 con 0x563a63a6c580 2023-07-29T16:24:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39af7fe700 10 client.28127 handle_command_reply: tid=0 2023-07-29T16:24:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 client.28127 shutdown 2023-07-29T16:24:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 msgr2=0x563a63a67e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a67e70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009540 tx=0x7f39c80095e0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 shutdown_connections 2023-07-29T16:24:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63a6c580 0x563a63a6e990 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a67e70 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a608f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1207398584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a575b0 0x563a63a60e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39af7fe700 0 client.28127 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:04.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.936+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 >> 172.21.15.31:0/1207398584 conn(0x563a63a4c460 msgr2=0x563a63a510e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:04.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.937+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 shutdown_connections 2023-07-29T16:24:04.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.937+0000 7f39ea057b80 1 -- 172.21.15.31:0/1207398584 wait complete. 2023-07-29T16:24:04.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.937+0000 7f39ea057b80 20 client.28127 trim_cache size 0 max 16384 2023-07-29T16:24:04.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 Processor -- start 2023-07-29T16:24:04.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 -- start start 2023-07-29T16:24:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a56c60 0x563a63a510e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 0x563a63a67e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a5e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63a4e4e0 con 0x563a63a56c60 2023-07-29T16:24:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63a61a70 con 0x563a63a58370 2023-07-29T16:24:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.956+0000 7f39ea057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63a5e4f0 con 0x563a63a575b0 2023-07-29T16:24:04.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 0x563a63a67e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a56c60 0x563a63a510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 0x563a63a67e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48812/0 (socket says 172.21.15.31:48812) 2023-07-29T16:24:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1700612998 learned_addr learned my addr 172.21.15.31:0/1700612998 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:04.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39d4d9c700 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a5e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1700612998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 msgr2=0x563a63a5e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a5e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1700612998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a56c60 msgr2=0x563a63a510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a56c60 0x563a63a510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 -- 172.21.15.31:0/1700612998 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a63a58a10 con 0x563a63a575b0 2023-07-29T16:24:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cffff700 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a56c60 0x563a63a510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.957+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 0x563a63a67e30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f39c4005cb0 tx=0x7f39c4009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7ea0ed7ef6daaa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:04.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39b6ffd700 1 -- 172.21.15.31:0/1700612998 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4099c30 con 0x563a63a575b0 2023-07-29T16:24:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39b6ffd700 1 -- 172.21.15.31:0/1700612998 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c40d2070 con 0x563a63a575b0 2023-07-29T16:24:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39b6ffd700 1 -- 172.21.15.31:0/1700612998 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c40a00b0 con 0x563a63a575b0 2023-07-29T16:24:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 -- 172.21.15.31:0/1700612998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 msgr2=0x563a63a67e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 0x563a63a67e30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f39c4005cb0 tx=0x7f39c4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 -- 172.21.15.31:0/1700612998 shutdown_connections 2023-07-29T16:24:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a58370 0x563a63a5e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a56c60 0x563a63a510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 --2- 172.21.15.31:0/1700612998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a575b0 0x563a63a67e30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 -- 172.21.15.31:0/1700612998 >> 172.21.15.31:0/1700612998 conn(0x563a63a4c460 msgr2=0x563a63a5f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:04.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.958+0000 7f39ea057b80 1 -- 172.21.15.31:0/1700612998 shutdown_connections 2023-07-29T16:24:04.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.959+0000 7f39ea057b80 1 -- 172.21.15.31:0/1700612998 wait complete. 2023-07-29T16:24:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.959+0000 7f39ea057b80 1 -- start start 2023-07-29T16:24:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.959+0000 7f39ea057b80 1 Processor -- start 2023-07-29T16:24:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a64e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 0x563a63a653d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a58370 0x563a63a663e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63a66920 con 0x563a63a58370 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63a66a90 con 0x563a63a575b0 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a63a66c50 con 0x563a63a56c60 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39ea057b80 10 client.0 _ll_register_callbacks cb 0x563a6394d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 0x563a63a653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39d4d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a58370 0x563a63a663e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39cf7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 0x563a63a653d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46714/0 (socket says 172.21.15.31:46714) 2023-07-29T16:24:04.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39cf7fe700 1 -- 172.21.15.31:0/2133770136 learned_addr learned my addr 172.21.15.31:0/2133770136 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.960+0000 7f39cffff700 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a64e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39cf7fe700 1 -- 172.21.15.31:0/2133770136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 msgr2=0x563a63a64e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a64e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39cf7fe700 1 -- 172.21.15.31:0/2133770136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a58370 msgr2=0x563a63a663e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a58370 0x563a63a663e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39cf7fe700 1 -- 172.21.15.31:0/2133770136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c0004080 con 0x563a63a575b0 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39cf7fe700 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 0x563a63a653d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f39c4099fd0 tx=0x7f39c4099ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b22145ea8623df server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:04.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39af7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39af7fe700 1 -- 172.21.15.31:0/2133770136 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c4099c30 con 0x563a63a575b0 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39c800f040 con 0x563a63a575b0 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39af7fe700 1 -- 172.21.15.31:0/2133770136 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39c409e300 con 0x563a63a575b0 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39af7fe700 1 -- 172.21.15.31:0/2133770136 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39c409a2d0 con 0x563a63a575b0 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39ea057b80 10 client.28133 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.961+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f39c40d4040 con 0x563a63a575b0 2023-07-29T16:24:04.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39af7fe700 1 -- 172.21.15.31:0/2133770136 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c800f040 con 0x563a63a575b0 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39af7fe700 1 -- 172.21.15.31:0/2133770136 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f39c40960c0 con 0x563a63a575b0 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39ea057b80 10 client.28133 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39ea057b80 10 client.28133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39ea057b80 20 client.28133 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39ea057b80 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63ac0110 0x563a63ac24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39ea057b80 4 client.28133 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563a639a0dc0 con 0x563a63ac0110 2023-07-29T16:24:04.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.962+0000 7f39cffff700 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63ac0110 0x563a63ac24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.963+0000 7f39cffff700 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63ac0110 0x563a63ac24f0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e837c1b8a843292a server_cookie=b701ff6d6c4ba710 in_seq=0 out_seq=0 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.963+0000 7f39af7fe700 10 client.28133 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.965+0000 7f39af7fe700 1 -- 172.21.15.31:0/2133770136 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563a639a0dc0 con 0x563a63ac0110 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.965+0000 7f39af7fe700 10 client.28133 handle_command_reply: tid=0 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.965+0000 7f39ea057b80 1 client.28133 shutdown 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 msgr2=0x563a63a653d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 0x563a63a653d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f39c4099fd0 tx=0x7f39c4099ed0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 shutdown_connections 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563a63ac0110 0x563a63ac24f0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563a63a58370 0x563a63a663e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39af7fe700 0 client.28133 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563a63a56c60 0x563a63a64e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 --2- 172.21.15.31:0/2133770136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a63a575b0 0x563a63a653d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 >> 172.21.15.31:0/2133770136 conn(0x563a63a4c460 msgr2=0x563a63a510e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 shutdown_connections 2023-07-29T16:24:04.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 1 -- 172.21.15.31:0/2133770136 wait complete. 2023-07-29T16:24:04.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.966+0000 7f39ea057b80 20 client.28133 trim_cache size 0 max 16384 2023-07-29T16:24:04.976 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:24:04.976 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:24:04.976 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "bbc1ce1b-6593-4d90-a160-2d6a38aed94f", 2023-07-29T16:24:04.977 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:24:04.977 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:24:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39b80a0810 msgr2=0x7f39b80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39b80a0810 0x7f39b80a2cc0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x563a63a579c0 tx=0x7f39c0009260 comp rx=0 tx=0).stop 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 msgr2=0x7f39d0198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 0x7f39d0198790 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f39c4004740 tx=0x7f39c4009580 comp rx=0 tx=0).stop 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 shutdown_connections 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39b80a0810 0x7f39b80a2cc0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d01040d0 0x7f39d019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d0102550 0x7f39d0198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 --2- 172.21.15.31:0/1810991229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d0103740 0x7f39d0198790 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 >> 172.21.15.31:0/1810991229 conn(0x7f39d00fdb90 msgr2=0x7f39d00ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 shutdown_connections 2023-07-29T16:24:04.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:04.967+0000 7f39d67ff700 1 -- 172.21.15.31:0/1810991229 wait complete. 2023-07-29T16:24:04.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:24:05.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 -- 172.21.15.31:0/2865491453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a0100da0 msgr2=0x7fd7a0103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/2865491453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a0100da0 0x7fd7a0103180 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fd790009990 tx=0x7fd790009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 -- 172.21.15.31:0/2865491453 shutdown_connections 2023-07-29T16:24:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/2865491453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0105fe0 0x7fd7a0108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/2865491453 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a0100da0 0x7fd7a0103180 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/2865491453 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a01036c0 0x7fd7a0105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 -- 172.21.15.31:0/2865491453 >> 172.21.15.31:0/2865491453 conn(0x7fd7a00fa840 msgr2=0x7fd7a00fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:05.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 -- 172.21.15.31:0/2865491453 shutdown_connections 2023-07-29T16:24:05.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.226+0000 7fd7a69ff700 1 -- 172.21.15.31:0/2865491453 wait complete. 2023-07-29T16:24:05.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 Processor -- start 2023-07-29T16:24:05.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 -- start start 2023-07-29T16:24:05.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 0x7fd7a01960b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a01036c0 0x7fd7a01965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a0105fe0 0x7fd7a019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a019dc30 con 0x7fd7a01036c0 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a019dd70 con 0x7fd7a0100da0 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.227+0000 7fd7a69ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a019df00 con 0x7fd7a0105fe0 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 0x7fd7a01960b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 0x7fd7a01960b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46728/0 (socket says 172.21.15.31:46728) 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 -- 172.21.15.31:0/552116077 learned_addr learned my addr 172.21.15.31:0/552116077 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a01036c0 0x7fd7a01965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a0105fe0 0x7fd7a019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 -- 172.21.15.31:0/552116077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a0105fe0 msgr2=0x7fd7a019d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a0105fe0 0x7fd7a019d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 -- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a01036c0 msgr2=0x7fd7a01965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a01036c0 0x7fd7a01965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 -- 172.21.15.31:0/552116077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7900095f0 con 0x7fd7a0100da0 2023-07-29T16:24:05.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a0105fe0 0x7fd7a019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:05.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a01036c0 0x7fd7a01965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:05.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.228+0000 7fd79ffff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 0x7fd7a01960b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd79000b680 tx=0x7fd790005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:05.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.229+0000 7fd79d7fa700 1 -- 172.21.15.31:0/552116077 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79001d070 con 0x7fd7a0100da0 2023-07-29T16:24:05.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.229+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a019e180 con 0x7fd7a0100da0 2023-07-29T16:24:05.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.229+0000 7fd79d7fa700 1 -- 172.21.15.31:0/552116077 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd79000fbc0 con 0x7fd7a0100da0 2023-07-29T16:24:05.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.229+0000 7fd79d7fa700 1 -- 172.21.15.31:0/552116077 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790017650 con 0x7fd7a0100da0 2023-07-29T16:24:05.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.229+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a019e600 con 0x7fd7a0100da0 2023-07-29T16:24:05.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.233+0000 7fd79d7fa700 1 -- 172.21.15.31:0/552116077 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd790022470 con 0x7fd7a0100da0 2023-07-29T16:24:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.234+0000 7fd79d7fa700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd7880a0790 0x7fd7880a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.234+0000 7fd79d7fa700 1 -- 172.21.15.31:0/552116077 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd7900cd260 con 0x7fd7a0100da0 2023-07-29T16:24:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.234+0000 7fd7ba2beb80 1 Processor -- start 2023-07-29T16:24:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd7880a0790 0x7fd7880a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 -- start start 2023-07-29T16:24:05.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5701ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b5702d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b5703740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b5659a10 con 0x5641b5702530 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b56573d0 con 0x5641b57032f0 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b5652cc0 con 0x5641b5701be0 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7a4f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b5703740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7a4f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b5703740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46742/0 (socket says 172.21.15.31:46742) 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5701ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.235+0000 7fd7a4f9c700 1 -- 172.21.15.31:0/2347335523 learned_addr learned my addr 172.21.15.31:0/2347335523 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b5702d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd7880a0790 0x7fd7880a2c40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd794009cb0 tx=0x7fd794009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79ffff700 1 -- 172.21.15.31:0/2347335523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 msgr2=0x5641b5703740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79ffff700 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b5703740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79ffff700 1 -- 172.21.15.31:0/2347335523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 msgr2=0x5641b5702d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79ffff700 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b5702d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79ffff700 1 -- 172.21.15.31:0/2347335523 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641b5480de0 con 0x5641b5701be0 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b5703740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b5702d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd79ffff700 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5701ff0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd79009a920 tx=0x7fd790005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44f7aa3c2bb536af server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd7867fc700 1 -- 172.21.15.31:0/2347335523 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790022ea0 con 0x5641b5701be0 2023-07-29T16:24:05.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd7867fc700 1 -- 172.21.15.31:0/2347335523 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7900a2d50 con 0x5641b5701be0 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.236+0000 7fd7867fc700 1 -- 172.21.15.31:0/2347335523 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79009a0e0 con 0x5641b5701be0 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/2347335523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 msgr2=0x5641b5701ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5701ff0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd79009a920 tx=0x7fd790005fd0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/2347335523 shutdown_connections 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b5703740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5701ff0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/2347335523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b5702d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/2347335523 >> 172.21.15.31:0/2347335523 conn(0x5641b56f73e0 msgr2=0x5641b56fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/2347335523 shutdown_connections 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.237+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/2347335523 wait complete. 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.238+0000 7fd7ba2beb80 1 -- start start 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.238+0000 7fd7ba2beb80 1 Processor -- start 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.238+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 0x5641b570ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b57032f0 0x5641b5712a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b55ef510 con 0x5641b5701be0 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b55ef1b0 con 0x5641b5702530 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b54d6920 con 0x5641b57032f0 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7ba2beb80 10 client.0 _ll_register_callbacks cb 0x5641b55f8620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 0x5641b570ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 0x5641b570ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46756/0 (socket says 172.21.15.31:46756) 2023-07-29T16:24:05.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 -- 172.21.15.31:0/1910070458 learned_addr learned my addr 172.21.15.31:0/1910070458 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79ffff700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b57032f0 0x5641b5712a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 -- 172.21.15.31:0/1910070458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b57032f0 msgr2=0x5641b5712a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b57032f0 0x5641b5712a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 -- 172.21.15.31:0/1910070458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 msgr2=0x5641b570b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.239+0000 7fd79f7fe700 1 -- 172.21.15.31:0/1910070458 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7980090f0 con 0x5641b5702530 2023-07-29T16:24:05.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 0x5641b570ba20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd794012820 tx=0x7fd7940047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be47c817bc26ef4f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd77f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd77f7fe700 1 -- 172.21.15.31:0/1910070458 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794029630 con 0x5641b5702530 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd77f7fe700 1 -- 172.21.15.31:0/1910070458 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd794018070 con 0x5641b5702530 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641b567dd10 con 0x5641b5702530 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd77f7fe700 1 -- 172.21.15.31:0/1910070458 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd794029790 con 0x5641b5702530 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd7ba2beb80 10 client.28145 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:05.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fd7a019e600 con 0x5641b5702530 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.240+0000 7fd77f7fe700 1 -- 172.21.15.31:0/1910070458 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5641b567dd10 con 0x5641b5702530 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd77f7fe700 1 -- 172.21.15.31:0/1910070458 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd794033b50 con 0x5641b5702530 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd7ba2beb80 10 client.28145 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd7ba2beb80 10 client.28145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd7ba2beb80 20 client.28145 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b5717300 0x5641b5719710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd7ba2beb80 4 client.28145 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:05.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641b56d0d70 con 0x5641b5717300 2023-07-29T16:24:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.241+0000 7fd79ffff700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b5717300 0x5641b5719710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.242+0000 7fd79ffff700 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b5717300 0x5641b5719710 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d03ab2a43a69933 server_cookie=8d77efa5444ecfc in_seq=0 out_seq=0 2023-07-29T16:24:05.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.242+0000 7fd77f7fe700 10 client.28145 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.242+0000 7fd77f7fe700 1 -- 172.21.15.31:0/1910070458 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5641b56d0d70 con 0x5641b5717300 2023-07-29T16:24:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.242+0000 7fd77f7fe700 10 client.28145 handle_command_reply: tid=0 2023-07-29T16:24:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.242+0000 7fd7ba2beb80 1 client.28145 shutdown 2023-07-29T16:24:05.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 msgr2=0x5641b570ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 0x5641b570ba20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd794012820 tx=0x7fd7940047c0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 shutdown_connections 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b5717300 0x5641b5719710 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b57032f0 0x5641b5712a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/1910070458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b5702530 0x5641b570ba20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 >> 172.21.15.31:0/1910070458 conn(0x5641b56f73e0 msgr2=0x5641b56fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd77f7fe700 0 client.28145 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 shutdown_connections 2023-07-29T16:24:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/1910070458 wait complete. 2023-07-29T16:24:05.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.243+0000 7fd7ba2beb80 20 client.28145 trim_cache size 0 max 16384 2023-07-29T16:24:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.263+0000 7fd7ba2beb80 1 Processor -- start 2023-07-29T16:24:05.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.263+0000 7fd7ba2beb80 1 -- start start 2023-07-29T16:24:05.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.263+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5710a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.263+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b570a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.263+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b57096d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b570b050 con 0x5641b5702530 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b56f9460 con 0x5641b57032f0 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b56fc960 con 0x5641b5701be0 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5710a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5710a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48882/0 (socket says 172.21.15.31:48882) 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 -- 172.21.15.31:0/3217413373 learned_addr learned my addr 172.21.15.31:0/3217413373 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b570a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b57096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 -- 172.21.15.31:0/3217413373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 msgr2=0x5641b57096d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b57096d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 -- 172.21.15.31:0/3217413373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 msgr2=0x5641b570a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b570a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd79ffff700 1 -- 172.21.15.31:0/3217413373 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641b5469c60 con 0x5641b5701be0 2023-07-29T16:24:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.264+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b57096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.265+0000 7fd79ffff700 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5710a10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd790096e90 tx=0x7fd79009eb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87fcb2c89e8a9e7d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.265+0000 7fd786ffd700 1 -- 172.21.15.31:0/3217413373 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900b3120 con 0x5641b5701be0 2023-07-29T16:24:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.265+0000 7fd786ffd700 1 -- 172.21.15.31:0/3217413373 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd79000fdb0 con 0x5641b5701be0 2023-07-29T16:24:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.265+0000 7fd786ffd700 1 -- 172.21.15.31:0/3217413373 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900a28c0 con 0x5641b5701be0 2023-07-29T16:24:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/3217413373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 msgr2=0x5641b5710a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5710a10 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd790096e90 tx=0x7fd79009eb40 comp rx=0 tx=0).stop 2023-07-29T16:24:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/3217413373 shutdown_connections 2023-07-29T16:24:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b57096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5701be0 0x5641b5710a10 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/3217413373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5702530 0x5641b570a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/3217413373 >> 172.21.15.31:0/3217413373 conn(0x5641b56f73e0 msgr2=0x5641b56fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/3217413373 shutdown_connections 2023-07-29T16:24:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/3217413373 wait complete. 2023-07-29T16:24:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.266+0000 7fd7ba2beb80 1 -- start start 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 Processor -- start 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 0x5641b570f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b570fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b5768e80 con 0x5641b5701be0 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b5768fc0 con 0x5641b57032f0 2023-07-29T16:24:05.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641b5769100 con 0x5641b5702530 2023-07-29T16:24:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd7ba2beb80 10 client.0 _ll_register_callbacks cb 0x5641b55f8620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.267+0000 7fd79ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 0x5641b570f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 0x5641b570f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48896/0 (socket says 172.21.15.31:48896) 2023-07-29T16:24:05.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 -- 172.21.15.31:0/61450407 learned_addr learned my addr 172.21.15.31:0/61450407 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd7a4f9c700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b570fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 -- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 msgr2=0x5641b570fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b570fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 -- 172.21.15.31:0/61450407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 msgr2=0x5641b570f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 -- 172.21.15.31:0/61450407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641b5480de0 con 0x5641b5702530 2023-07-29T16:24:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79ffff700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:05.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd79f7fe700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 0x5641b570f780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd794012820 tx=0x7fd794020780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30d082674f015dca server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd77f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd77f7fe700 1 -- 172.21.15.31:0/61450407 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794020df0 con 0x5641b5702530 2023-07-29T16:24:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.268+0000 7fd77f7fe700 1 -- 172.21.15.31:0/61450407 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd794018050 con 0x5641b5702530 2023-07-29T16:24:05.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7900a2d50 con 0x5641b5702530 2023-07-29T16:24:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd77f7fe700 1 -- 172.21.15.31:0/61450407 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd794033460 con 0x5641b5702530 2023-07-29T16:24:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 10 client.28159 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:05.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5641b54a7250 con 0x5641b5702530 2023-07-29T16:24:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd77f7fe700 1 -- 172.21.15.31:0/61450407 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7900a2d50 con 0x5641b5702530 2023-07-29T16:24:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd77f7fe700 1 -- 172.21.15.31:0/61450407 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd794029760 con 0x5641b5702530 2023-07-29T16:24:05.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 10 client.28159 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:05.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 10 client.28159 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:05.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 20 client.28159 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b576ef30 0x5641b5771340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 4 client.28159 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:24:05.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.269+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5641b564bd40 con 0x5641b576ef30 2023-07-29T16:24:05.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd79ffff700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b576ef30 0x5641b5771340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:05.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd77f7fe700 10 client.28159 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:05.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd79ffff700 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b576ef30 0x5641b5771340 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c3bf6bfb89bad3e server_cookie=65dbf75281266613 in_seq=0 out_seq=0 2023-07-29T16:24:05.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd77f7fe700 1 -- 172.21.15.31:0/61450407 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5641b564bd40 con 0x5641b576ef30 2023-07-29T16:24:05.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd77f7fe700 10 client.28159 handle_command_reply: tid=0 2023-07-29T16:24:05.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 client.28159 shutdown 2023-07-29T16:24:05.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 msgr2=0x5641b570f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 0x5641b570f780 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd794012820 tx=0x7fd794020780 comp rx=0 tx=0).stop 2023-07-29T16:24:05.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 shutdown_connections 2023-07-29T16:24:05.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5641b576ef30 0x5641b5771340 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5641b57032f0 0x5641b570fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641b5701be0 0x5641b570f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd77f7fe700 0 client.28159 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:05.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 --2- 172.21.15.31:0/61450407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5641b5702530 0x5641b570f780 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.270+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 >> 172.21.15.31:0/61450407 conn(0x5641b56f73e0 msgr2=0x5641b5719310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:05.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 shutdown_connections 2023-07-29T16:24:05.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7ba2beb80 1 -- 172.21.15.31:0/61450407 wait complete. 2023-07-29T16:24:05.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7ba2beb80 20 client.28159 trim_cache size 0 max 16384 2023-07-29T16:24:05.287 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:24:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:24:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:24:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "bbc1ce1b-6593-4d90-a160-2d6a38aed94f": { 2023-07-29T16:24:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:24:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "bbc1ce1b-6593-4d90-a160-2d6a38aed94f", 2023-07-29T16:24:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:24:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:24:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:24:05.288 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:24:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd7880a0790 msgr2=0x7fd7880a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd7880a0790 0x7fd7880a2c40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd794009cb0 tx=0x7fd794009260 comp rx=0 tx=0).stop 2023-07-29T16:24:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 msgr2=0x7fd7a01960b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.271+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 0x7fd7a01960b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd79000b680 tx=0x7fd790005ef0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 shutdown_connections 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd7880a0790 0x7fd7880a2c40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd7a0105fe0 0x7fd7a019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7a0100da0 0x7fd7a01960b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 --2- 172.21.15.31:0/552116077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7a01036c0 0x7fd7a01965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 >> 172.21.15.31:0/552116077 conn(0x7fd7a00fa840 msgr2=0x7fd7a0101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 shutdown_connections 2023-07-29T16:24:05.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:05.272+0000 7fd7a69ff700 1 -- 172.21.15.31:0/552116077 wait complete. 2023-07-29T16:24:05.291 INFO:tasks.cephfs.filesystem:scrub status for tag:bbc1ce1b-6593-4d90-a160-2d6a38aed94f - {'path': '/', 'tag': 'bbc1ce1b-6593-4d90-a160-2d6a38aed94f', 'options': 'recursive,force'} 2023-07-29T16:24:05.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:05 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:05.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:05 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:05.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:05 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:05.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:05 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:05.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:05 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:05.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:05 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:05.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 581905 8.94 MB/sec execute 230 sec latency 22.163 ms 2023-07-29T16:24:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:24:06.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:06.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:06.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:06.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:06 smithi039 ceph-mon[128858]: pgmap v629: 97 pgs: 97 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 2.0 MiB/s wr, 248 op/s 2023-07-29T16:24:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:24:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:06.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:06.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:06.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:06 smithi156 ceph-mon[129245]: pgmap v629: 97 pgs: 97 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 2.0 MiB/s wr, 248 op/s 2023-07-29T16:24:06.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 583119 8.93 MB/sec execute 231 sec latency 12.533 ms 2023-07-29T16:24:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:24:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:06 smithi031 ceph-mon[95504]: pgmap v629: 97 pgs: 97 active+clean; 121 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 2.0 MiB/s wr, 248 op/s 2023-07-29T16:24:07.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 585303 8.93 MB/sec execute 232 sec latency 16.730 ms 2023-07-29T16:24:08.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:07 smithi031 ceph-mon[95504]: pgmap v630: 97 pgs: 97 active+clean; 117 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 1.6 MiB/s wr, 215 op/s 2023-07-29T16:24:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:07 smithi039 ceph-mon[128858]: pgmap v630: 97 pgs: 97 active+clean; 117 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 1.6 MiB/s wr, 215 op/s 2023-07-29T16:24:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:07 smithi156 ceph-mon[129245]: pgmap v630: 97 pgs: 97 active+clean; 117 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 1.6 MiB/s wr, 215 op/s 2023-07-29T16:24:08.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 586498 8.93 MB/sec execute 233 sec latency 46.859 ms 2023-07-29T16:24:09.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 588212 8.92 MB/sec execute 234 sec latency 49.823 ms 2023-07-29T16:24:10.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:10 smithi039 ceph-mon[128858]: pgmap v631: 97 pgs: 97 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 306 op/s 2023-07-29T16:24:10.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:10 smithi156 ceph-mon[129245]: pgmap v631: 97 pgs: 97 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 306 op/s 2023-07-29T16:24:10.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 589932 8.94 MB/sec execute 235 sec latency 24.589 ms 2023-07-29T16:24:10.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:10 smithi031 ceph-mon[95504]: pgmap v631: 97 pgs: 97 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 306 op/s 2023-07-29T16:24:11.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 591713 8.92 MB/sec execute 236 sec latency 13.285 ms 2023-07-29T16:24:12.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:12 smithi039 ceph-mon[128858]: pgmap v632: 97 pgs: 97 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 2.1 MiB/s wr, 236 op/s 2023-07-29T16:24:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:12 smithi156 ceph-mon[129245]: pgmap v632: 97 pgs: 97 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 2.1 MiB/s wr, 236 op/s 2023-07-29T16:24:12.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 593532 8.94 MB/sec execute 237 sec latency 15.750 ms 2023-07-29T16:24:12.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:12 smithi031 ceph-mon[95504]: pgmap v632: 97 pgs: 97 active+clean; 120 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 2.1 MiB/s wr, 236 op/s 2023-07-29T16:24:13.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:24:13.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:24:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:24:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:24:13.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 595369 8.93 MB/sec execute 238 sec latency 13.984 ms 2023-07-29T16:24:13.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:24:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:24:14.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:14 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:24:14.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:14 smithi039 ceph-mon[128858]: pgmap v633: 97 pgs: 97 active+clean; 122 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2023-07-29T16:24:14.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:14 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:24:14.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:14 smithi156 ceph-mon[129245]: pgmap v633: 97 pgs: 97 active+clean; 122 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2023-07-29T16:24:14.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 596965 8.95 MB/sec execute 239 sec latency 75.941 ms 2023-07-29T16:24:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:14 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:24:14.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:14 smithi031 ceph-mon[95504]: pgmap v633: 97 pgs: 97 active+clean; 122 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2023-07-29T16:24:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 -- 172.21.15.31:0/728160518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 msgr2=0x7fada0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 --2- 172.21.15.31:0/728160518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0108780 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fad9800b0a0 tx=0x7fad9800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 -- 172.21.15.31:0/728160518 shutdown_connections 2023-07-29T16:24:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 --2- 172.21.15.31:0/728160518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0108780 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 --2- 172.21.15.31:0/728160518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada0100340 0x7fada0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 --2- 172.21.15.31:0/728160518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0101530 0x7fada0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 -- 172.21.15.31:0/728160518 >> 172.21.15.31:0/728160518 conn(0x7fada00fb920 msgr2=0x7fada00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 -- 172.21.15.31:0/728160518 shutdown_connections 2023-07-29T16:24:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.619+0000 7fada85ff700 1 -- 172.21.15.31:0/728160518 wait complete. 2023-07-29T16:24:15.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.620+0000 7fada85ff700 1 Processor -- start 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.620+0000 7fada85ff700 1 -- start start 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada85ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0100340 0x7fada0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada85ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada0101530 0x7fada0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada85ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada85ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada01a01d0 con 0x7fada0101530 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada85ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada01a0310 con 0x7fada0101ec0 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada85ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada01a0450 con 0x7fada0100340 2023-07-29T16:24:15.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0100340 0x7fada0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0100340 0x7fada0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48478/0 (socket says 172.21.15.31:48478) 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada639b700 1 -- 172.21.15.31:0/75302978 learned_addr learned my addr 172.21.15.31:0/75302978 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0073bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36970/0 (socket says 172.21.15.31:36970) 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada5b9a700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada0101530 0x7fada0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 -- 172.21.15.31:0/75302978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0100340 msgr2=0x7fada0073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0100340 0x7fada0073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 -- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada0101530 msgr2=0x7fada0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada0101530 0x7fada0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.621+0000 7fada6b9c700 1 -- 172.21.15.31:0/75302978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c009520 con 0x7fada0101ec0 2023-07-29T16:24:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.622+0000 7fada6b9c700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0073bc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fad9800b0a0 tx=0x7fad98007f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:15.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.622+0000 7fad977fe700 1 -- 172.21.15.31:0/75302978 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9800e040 con 0x7fada0101ec0 2023-07-29T16:24:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.622+0000 7fad977fe700 1 -- 172.21.15.31:0/75302978 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad9800bde0 con 0x7fada0101ec0 2023-07-29T16:24:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.622+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad98009d00 con 0x7fada0101ec0 2023-07-29T16:24:15.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.622+0000 7fad977fe700 1 -- 172.21.15.31:0/75302978 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9801b6b0 con 0x7fada0101ec0 2023-07-29T16:24:15.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.622+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada01a09c0 con 0x7fada0101ec0 2023-07-29T16:24:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.625+0000 7fad977fe700 1 -- 172.21.15.31:0/75302978 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fad9801d480 con 0x7fada0101ec0 2023-07-29T16:24:15.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.626+0000 7fadbbf64b80 1 Processor -- start 2023-07-29T16:24:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 -- start start 2023-07-29T16:24:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864dda8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864ddb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864d329b0 con 0x563864ddadf0 2023-07-29T16:24:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864d30370 con 0x563864ddbcf0 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864d2bc60 con 0x563864c366b0 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fada639b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864ddb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fada639b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864ddb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33024/0 (socket says 172.21.15.31:33024) 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fad977fe700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fad8c0a0810 0x7fad8c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fada639b700 1 -- 172.21.15.31:0/3124098639 learned_addr learned my addr 172.21.15.31:0/3124098639 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.627+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864dda8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fad977fe700 1 -- 172.21.15.31:0/75302978 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fad980ccff0 con 0x7fada0101ec0 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada6b9c700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fad8c0a0810 0x7fad8c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada639b700 1 -- 172.21.15.31:0/3124098639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 msgr2=0x563864dda8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada639b700 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864dda8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada639b700 1 -- 172.21.15.31:0/3124098639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 msgr2=0x563864ddc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada639b700 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada639b700 1 -- 172.21.15.31:0/3124098639 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563864b59d80 con 0x563864ddadf0 2023-07-29T16:24:15.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada639b700 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864ddb7b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009f60 tx=0x7fad9c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c43d8b203aa36e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fada6b9c700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fad8c0a0810 0x7fad8c0a2cc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fad980971d0 tx=0x7fad980099e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fad94bfe700 1 -- 172.21.15.31:0/3124098639 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c004a30 con 0x563864ddadf0 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fad94bfe700 1 -- 172.21.15.31:0/3124098639 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad9c0054e0 con 0x563864ddadf0 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.628+0000 7fad94bfe700 1 -- 172.21.15.31:0/3124098639 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c019880 con 0x563864ddadf0 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3124098639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 msgr2=0x563864ddb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864ddb7b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009f60 tx=0x7fad9c009f90 comp rx=0 tx=0).stop 2023-07-29T16:24:15.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3124098639 shutdown_connections 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864dda8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3124098639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864ddb7b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3124098639 >> 172.21.15.31:0/3124098639 conn(0x563864dcffd0 msgr2=0x563864dd5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3124098639 shutdown_connections 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.629+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3124098639 wait complete. 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 -- start start 2023-07-29T16:24:15.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 Processor -- start 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 0x563864de4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864deb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864cc84b0 con 0x563864c366b0 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864cc8150 con 0x563864ddbcf0 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.630+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864baf8c0 con 0x563864ddadf0 2023-07-29T16:24:15.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fadbbf64b80 10 client.0 _ll_register_callbacks cb 0x563864cd15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 0x563864de4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 0x563864de4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48504/0 (socket says 172.21.15.31:48504) 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 -- 172.21.15.31:0/3582514920 learned_addr learned my addr 172.21.15.31:0/3582514920 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864deb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 -- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 msgr2=0x563864deb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864deb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 -- 172.21.15.31:0/3582514920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 msgr2=0x563864de3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 -- 172.21.15.31:0/3582514920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9c009b90 con 0x563864ddadf0 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fada639b700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 0x563864de4390 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fad9c000c00 tx=0x7fad9c005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f99b96a4c5280f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fad89ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.631+0000 7fad89ffb700 1 -- 172.21.15.31:0/3582514920 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c004960 con 0x563864ddadf0 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563864d56cb0 con 0x563864ddadf0 2023-07-29T16:24:15.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fad89ffb700 1 -- 172.21.15.31:0/3582514920 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fad9c004020 con 0x563864ddadf0 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fad89ffb700 1 -- 172.21.15.31:0/3582514920 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad9c004ac0 con 0x563864ddadf0 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 10 client.28171 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563864b42c00 con 0x563864ddadf0 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fad89ffb700 1 -- 172.21.15.31:0/3582514920 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563864d56cb0 con 0x563864ddadf0 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fad89ffb700 1 -- 172.21.15.31:0/3582514920 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fad9c019580 con 0x563864ddadf0 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 10 client.28171 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 10 client.28171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 20 client.28171 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864defb70 0x563864df1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 4 client.28171 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.632+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563864da9d10 con 0x563864defb70 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.633+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864defb70 0x563864df1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.633+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864defb70 0x563864df1f80 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaed5244d9b5d377 server_cookie=5c5be692398f032f in_seq=0 out_seq=0 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.633+0000 7fad89ffb700 10 client.28171 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fad89ffb700 1 -- 172.21.15.31:0/3582514920 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563864da9d10 con 0x563864defb70 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fad89ffb700 10 client.28171 handle_command_reply: tid=0 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 client.28171 shutdown 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 msgr2=0x563864de4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 0x563864de4390 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fad9c000c00 tx=0x7fad9c005400 comp rx=0 tx=0).stop 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 shutdown_connections 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864defb70 0x563864df1f80 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864deb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fad89ffb700 0 client.28171 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3582514920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddadf0 0x563864de4390 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.634+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 >> 172.21.15.31:0/3582514920 conn(0x563864dcffd0 msgr2=0x563864dd4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.635+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 shutdown_connections 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.635+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3582514920 wait complete. 2023-07-29T16:24:15.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.635+0000 7fadbbf64b80 20 client.28171 trim_cache size 0 max 16384 2023-07-29T16:24:15.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 Processor -- start 2023-07-29T16:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 -- start start 2023-07-29T16:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864df1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864de9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864def6f0 con 0x563864ddadf0 2023-07-29T16:24:15.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864ddc600 con 0x563864ddbcf0 2023-07-29T16:24:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864de1d20 con 0x563864c366b0 2023-07-29T16:24:15.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fada5b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864df1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fada639b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864de9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fada5b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864df1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48508/0 (socket says 172.21.15.31:48508) 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fada5b9a700 1 -- 172.21.15.31:0/682849792 learned_addr learned my addr 172.21.15.31:0/682849792 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.651+0000 7fada5b9a700 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada639b700 1 -- 172.21.15.31:0/682849792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 msgr2=0x563864df1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada639b700 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864df1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada639b700 1 -- 172.21.15.31:0/682849792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 msgr2=0x563864ddc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada639b700 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada639b700 1 -- 172.21.15.31:0/682849792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563864b801f0 con 0x563864ddadf0 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada5b9a700 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864df1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fada639b700 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864de9380 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fad9c019990 tx=0x7fad9c005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37900e8ac74f82c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fad953ff700 1 -- 172.21.15.31:0/682849792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c019c00 con 0x563864ddadf0 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fad953ff700 1 -- 172.21.15.31:0/682849792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad9c00f880 con 0x563864ddadf0 2023-07-29T16:24:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.652+0000 7fad953ff700 1 -- 172.21.15.31:0/682849792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9c00fc00 con 0x563864ddadf0 2023-07-29T16:24:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 -- 172.21.15.31:0/682849792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 msgr2=0x563864de9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864de9380 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fad9c019990 tx=0x7fad9c005470 comp rx=0 tx=0).stop 2023-07-29T16:24:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 -- 172.21.15.31:0/682849792 shutdown_connections 2023-07-29T16:24:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddbcf0 0x563864ddc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864c366b0 0x563864df1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/682849792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864ddadf0 0x563864de9380 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 -- 172.21.15.31:0/682849792 >> 172.21.15.31:0/682849792 conn(0x563864dcffd0 msgr2=0x563864de2c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 -- 172.21.15.31:0/682849792 shutdown_connections 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.653+0000 7fadbbf64b80 1 -- 172.21.15.31:0/682849792 wait complete. 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.654+0000 7fadbbf64b80 1 -- start start 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.654+0000 7fadbbf64b80 1 Processor -- start 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.654+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 0x563864e3d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fadbbf64b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddbcf0 0x563864e3fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864e404a0 con 0x563864c366b0 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864e40610 con 0x563864ddadf0 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fadbbf64b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563864e407d0 con 0x563864ddbcf0 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fadbbf64b80 10 client.0 _ll_register_callbacks cb 0x563864cd15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada639b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 0x563864e3d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada639b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 0x563864e3d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36992/0 (socket says 172.21.15.31:36992) 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada639b700 1 -- 172.21.15.31:0/3877784611 learned_addr learned my addr 172.21.15.31:0/3877784611 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:15.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 -- 172.21.15.31:0/3877784611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddbcf0 msgr2=0x563864e3fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddbcf0 0x563864e3fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 -- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 msgr2=0x563864e3d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 0x563864e3d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 -- 172.21.15.31:0/3877784611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9002b040 con 0x563864c366b0 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada639b700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 0x563864e3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fada5b9a700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de9380 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x563864dd68f0 tx=0x7fad90023e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e639b31c93591a07 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:15.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.655+0000 7fad89ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fad89ffb700 1 -- 172.21.15.31:0/3877784611 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90016040 con 0x563864c366b0 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fad89ffb700 1 -- 172.21.15.31:0/3877784611 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fad90003b90 con 0x563864c366b0 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad9c0054e0 con 0x563864c366b0 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fad89ffb700 1 -- 172.21.15.31:0/3877784611 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad90003d50 con 0x563864c366b0 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 10 client.18117 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563864b42c00 con 0x563864c366b0 2023-07-29T16:24:15.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fad89ffb700 1 -- 172.21.15.31:0/3877784611 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad9c0054e0 con 0x563864c366b0 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fad89ffb700 1 -- 172.21.15.31:0/3877784611 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fad90020480 con 0x563864c366b0 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 10 client.18117 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 10 client.18117 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 20 client.18117 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864e421b0 0x563864e445c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 4 client.18117 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.656+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563864d24ce0 con 0x563864e421b0 2023-07-29T16:24:15.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.657+0000 7fada639b700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864e421b0 0x563864e445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.657+0000 7fada639b700 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864e421b0 0x563864e445c0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80b60c99dd148070 server_cookie=52c32ee94d6e677d in_seq=0 out_seq=0 2023-07-29T16:24:15.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.657+0000 7fad89ffb700 10 client.18117 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.735+0000 7fad89ffb700 1 -- 172.21.15.31:0/3877784611 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563864d24ce0 con 0x563864e421b0 2023-07-29T16:24:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.735+0000 7fad89ffb700 10 client.18117 handle_command_reply: tid=0 2023-07-29T16:24:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.735+0000 7fadbbf64b80 1 client.18117 shutdown 2023-07-29T16:24:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 msgr2=0x563864de9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de9380 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x563864dd68f0 tx=0x7fad90023e50 comp rx=0 tx=0).stop 2023-07-29T16:24:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 shutdown_connections 2023-07-29T16:24:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563864e421b0 0x563864e445c0 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563864ddbcf0 0x563864e3fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563864c366b0 0x563864de9380 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 --2- 172.21.15.31:0/3877784611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563864ddadf0 0x563864e3d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fad89ffb700 0 client.18117 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 >> 172.21.15.31:0/3877784611 conn(0x563864dcffd0 msgr2=0x563864ddeac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 shutdown_connections 2023-07-29T16:24:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 1 -- 172.21.15.31:0/3877784611 wait complete. 2023-07-29T16:24:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.736+0000 7fadbbf64b80 20 client.18117 trim_cache size 0 max 16384 2023-07-29T16:24:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:24:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:24:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:24:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:24:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fad8c0a0810 msgr2=0x7fad8c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fad8c0a0810 0x7fad8c0a2cc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fad980971d0 tx=0x7fad980099e0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 msgr2=0x7fada0073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:15.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0073bc0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fad9800b0a0 tx=0x7fad98007f30 comp rx=0 tx=0).stop 2023-07-29T16:24:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 shutdown_connections 2023-07-29T16:24:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fad8c0a0810 0x7fad8c0a2cc0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fada0101ec0 0x7fada0073bc0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fada0100340 0x7fada0073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 --2- 172.21.15.31:0/75302978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fada0101530 0x7fada0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 >> 172.21.15.31:0/75302978 conn(0x7fada00fb920 msgr2=0x7fada00fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 shutdown_connections 2023-07-29T16:24:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:15.737+0000 7fada85ff700 1 -- 172.21.15.31:0/75302978 wait complete. 2023-07-29T16:24:15.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 598163 8.93 MB/sec execute 240 sec latency 31.664 ms 2023-07-29T16:24:16.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:16 smithi039 ceph-mon[128858]: pgmap v634: 97 pgs: 97 active+clean; 123 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.4 MiB/s wr, 247 op/s 2023-07-29T16:24:16.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:16 smithi156 ceph-mon[129245]: pgmap v634: 97 pgs: 97 active+clean; 123 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.4 MiB/s wr, 247 op/s 2023-07-29T16:24:16.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 600406 8.95 MB/sec execute 241 sec latency 16.680 ms 2023-07-29T16:24:16.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:16 smithi031 ceph-mon[95504]: pgmap v634: 97 pgs: 97 active+clean; 123 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.4 MiB/s wr, 247 op/s 2023-07-29T16:24:17.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 601751 8.94 MB/sec execute 242 sec latency 14.799 ms 2023-07-29T16:24:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:17 smithi039 ceph-mon[128858]: pgmap v635: 97 pgs: 97 active+clean; 123 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.2 MiB/s wr, 211 op/s 2023-07-29T16:24:18.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:17 smithi156 ceph-mon[129245]: pgmap v635: 97 pgs: 97 active+clean; 123 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.2 MiB/s wr, 211 op/s 2023-07-29T16:24:18.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:17 smithi031 ceph-mon[95504]: pgmap v635: 97 pgs: 97 active+clean; 123 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.2 MiB/s wr, 211 op/s 2023-07-29T16:24:18.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 604022 8.96 MB/sec execute 243 sec latency 13.761 ms 2023-07-29T16:24:19.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 605397 8.94 MB/sec execute 244 sec latency 13.410 ms 2023-07-29T16:24:20.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:20 smithi039 ceph-mon[128858]: pgmap v636: 97 pgs: 97 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 322 op/s 2023-07-29T16:24:20.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:20 smithi156 ceph-mon[129245]: pgmap v636: 97 pgs: 97 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 322 op/s 2023-07-29T16:24:20.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 607409 8.96 MB/sec execute 245 sec latency 46.572 ms 2023-07-29T16:24:20.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:20 smithi031 ceph-mon[95504]: pgmap v636: 97 pgs: 97 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.9 MiB/s wr, 322 op/s 2023-07-29T16:24:21.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 608531 8.93 MB/sec execute 246 sec latency 27.354 ms 2023-07-29T16:24:22.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:22 smithi039 ceph-mon[128858]: pgmap v637: 97 pgs: 97 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2023-07-29T16:24:22.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:22 smithi156 ceph-mon[129245]: pgmap v637: 97 pgs: 97 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2023-07-29T16:24:22.880 INFO:tasks.workunit.client.0.smithi031.stdout: 1 610964 8.96 MB/sec execute 247 sec latency 18.098 ms 2023-07-29T16:24:22.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:22 smithi031 ceph-mon[95504]: pgmap v637: 97 pgs: 97 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 2.1 MiB/s wr, 227 op/s 2023-07-29T16:24:23.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 612152 8.94 MB/sec execute 248 sec latency 11.235 ms 2023-07-29T16:24:24.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:24 smithi039 ceph-mon[128858]: pgmap v638: 97 pgs: 97 active+clean; 119 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:24:24.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:24 smithi156 ceph-mon[129245]: pgmap v638: 97 pgs: 97 active+clean; 119 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:24:24.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 614543 8.97 MB/sec execute 249 sec latency 13.787 ms 2023-07-29T16:24:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:24 smithi031 ceph-mon[95504]: pgmap v638: 97 pgs: 97 active+clean; 119 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:24:25.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 615350 8.94 MB/sec execute 250 sec latency 111.742 ms 2023-07-29T16:24:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:26 smithi039 ceph-mon[128858]: pgmap v639: 97 pgs: 97 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 2.5 MiB/s wr, 264 op/s 2023-07-29T16:24:26.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:26 smithi156 ceph-mon[129245]: pgmap v639: 97 pgs: 97 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 2.5 MiB/s wr, 264 op/s 2023-07-29T16:24:26.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 617610 8.95 MB/sec execute 251 sec latency 16.949 ms 2023-07-29T16:24:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:26 smithi031 ceph-mon[95504]: pgmap v639: 97 pgs: 97 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 2.5 MiB/s wr, 264 op/s 2023-07-29T16:24:27.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 618927 8.94 MB/sec execute 252 sec latency 10.951 ms 2023-07-29T16:24:28.183 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:27 smithi031 ceph-mon[95504]: pgmap v640: 97 pgs: 97 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.3 MiB/s wr, 241 op/s 2023-07-29T16:24:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:27 smithi039 ceph-mon[128858]: pgmap v640: 97 pgs: 97 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.3 MiB/s wr, 241 op/s 2023-07-29T16:24:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:27 smithi156 ceph-mon[129245]: pgmap v640: 97 pgs: 97 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 2.3 MiB/s wr, 241 op/s 2023-07-29T16:24:28.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 621168 8.96 MB/sec execute 253 sec latency 26.003 ms 2023-07-29T16:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/3434126727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101ed0 msgr2=0x7fe6b4108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/3434126727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101ed0 0x7fe6b4108790 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009f90 tx=0x7fe6ac00b330 comp rx=0 tx=0).stop 2023-07-29T16:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/3434126727 shutdown_connections 2023-07-29T16:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/3434126727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101ed0 0x7fe6b4108790 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/3434126727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b4100350 0x7fe6b4100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/3434126727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4101540 0x7fe6b4101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/3434126727 >> 172.21.15.31:0/3434126727 conn(0x7fe6b40fb970 msgr2=0x7fe6b40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/3434126727 shutdown_connections 2023-07-29T16:24:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.892+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/3434126727 wait complete. 2023-07-29T16:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 Processor -- start 2023-07-29T16:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 -- start start 2023-07-29T16:24:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 0x7fe6b4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101540 0x7fe6b4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b4101ed0 0x7fe6b4073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41a0010 con 0x7fe6b4101ed0 2023-07-29T16:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41a0150 con 0x7fe6b4100350 2023-07-29T16:24:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.893+0000 7fe6bc5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b41a0290 con 0x7fe6b4101540 2023-07-29T16:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 0x7fe6b4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 0x7fe6b4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50980/0 (socket says 172.21.15.31:50980) 2023-07-29T16:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 -- 172.21.15.31:0/1555148939 learned_addr learned my addr 172.21.15.31:0/1555148939 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101540 0x7fe6b4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6bab9c700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b4101ed0 0x7fe6b4073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 -- 172.21.15.31:0/1555148939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101540 msgr2=0x7fe6b4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101540 0x7fe6b4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 -- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b4101ed0 msgr2=0x7fe6b4073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b4101ed0 0x7fe6b4073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 -- 172.21.15.31:0/1555148939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0009520 con 0x7fe6b4100350 2023-07-29T16:24:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101540 0x7fe6b4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.894+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 0x7fe6b4073140 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000ea50 tx=0x7fe6b000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.895+0000 7fe6ab7fe700 1 -- 172.21.15.31:0/1555148939 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b000eee0 con 0x7fe6b4100350 2023-07-29T16:24:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.895+0000 7fe6ab7fe700 1 -- 172.21.15.31:0/1555148939 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6b000cbd0 con 0x7fe6b4100350 2023-07-29T16:24:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.895+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ac009c40 con 0x7fe6b4100350 2023-07-29T16:24:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.895+0000 7fe6ab7fe700 1 -- 172.21.15.31:0/1555148939 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0018680 con 0x7fe6b4100350 2023-07-29T16:24:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.895+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b41a08f0 con 0x7fe6b4100350 2023-07-29T16:24:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.898+0000 7fe6ab7fe700 1 -- 172.21.15.31:0/1555148939 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe6b0010430 con 0x7fe6b4100350 2023-07-29T16:24:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.900+0000 7fe6ab7fe700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6a00a0810 0x7fe6a00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.900+0000 7fe6cff0ab80 1 Processor -- start 2023-07-29T16:24:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.900+0000 7fe6ab7fe700 1 -- 172.21.15.31:0/1555148939 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe6b0014070 con 0x7fe6b4100350 2023-07-29T16:24:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.900+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6a00a0810 0x7fe6a00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.900+0000 7fe6cff0ab80 1 -- start start 2023-07-29T16:24:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 0x55baa4f208b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4f20df0 0x55baa4f217b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f22160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4e789b0 con 0x55baa4f20df0 2023-07-29T16:24:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4e76370 con 0x55baa4f21cf0 2023-07-29T16:24:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4e71c60 con 0x55baa4d7c6b0 2023-07-29T16:24:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 0x55baa4f208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 0x55baa4f208b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44756/0 (socket says 172.21.15.31:44756) 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 -- 172.21.15.31:0/614784237 learned_addr learned my addr 172.21.15.31:0/614784237 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6bab9c700 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f22160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 -- 172.21.15.31:0/614784237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 msgr2=0x55baa4f22160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f22160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 -- 172.21.15.31:0/614784237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4f20df0 msgr2=0x55baa4f217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4f20df0 0x55baa4f217b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6a00a0810 0x7fe6a00a2cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4074440 tx=0x7fe6a4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6ba39b700 1 -- 172.21.15.31:0/614784237 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baa4c9fd80 con 0x55baa4d7c6b0 2023-07-29T16:24:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.901+0000 7fe6bab9c700 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f22160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.902+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 0x55baa4f208b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0011fd0 tx=0x7fe6b0099790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d8c44f191d58cfb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.902+0000 7fe6a8bfe700 1 -- 172.21.15.31:0/614784237 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00b1070 con 0x55baa4d7c6b0 2023-07-29T16:24:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.902+0000 7fe6a8bfe700 1 -- 172.21.15.31:0/614784237 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6b00a1be0 con 0x55baa4d7c6b0 2023-07-29T16:24:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.902+0000 7fe6a8bfe700 1 -- 172.21.15.31:0/614784237 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00a1990 con 0x55baa4d7c6b0 2023-07-29T16:24:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.902+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/614784237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 msgr2=0x55baa4f208b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.902+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 0x55baa4f208b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe6b0011fd0 tx=0x7fe6b0099790 comp rx=0 tx=0).stop 2023-07-29T16:24:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/614784237 shutdown_connections 2023-07-29T16:24:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f22160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4d7c6b0 0x55baa4f208b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/614784237 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4f20df0 0x55baa4f217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/614784237 >> 172.21.15.31:0/614784237 conn(0x55baa4f15fd0 msgr2=0x55baa4f1b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/614784237 shutdown_connections 2023-07-29T16:24:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/614784237 wait complete. 2023-07-29T16:24:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 -- start start 2023-07-29T16:24:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.903+0000 7fe6cff0ab80 1 Processor -- start 2023-07-29T16:24:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f29ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f2a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f31460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4e0e4b0 con 0x55baa4d7c6b0 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4e0e150 con 0x55baa4f20df0 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4cf58c0 con 0x55baa4f21cf0 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6cff0ab80 10 client.0 _ll_register_callbacks cb 0x55baa4e175c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f2a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6bab9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f31460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f2a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51006/0 (socket says 172.21.15.31:51006) 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/1981610835 learned_addr learned my addr 172.21.15.31:0/1981610835 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f29ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/1981610835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 msgr2=0x55baa4f31460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f31460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/1981610835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 msgr2=0x55baa4f29ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f29ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.904+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/1981610835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b41a08f0 con 0x55baa4f20df0 2023-07-29T16:24:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f2a420 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40039d0 tx=0x7fe6a4004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9aa7a2667cbce44 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe69dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe69dffb700 1 -- 172.21.15.31:0/1981610835 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a400e8f0 con 0x55baa4f20df0 2023-07-29T16:24:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe69dffb700 1 -- 172.21.15.31:0/1981610835 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe6a400ea50 con 0x55baa4f20df0 2023-07-29T16:24:28.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55baa4e9ccb0 con 0x55baa4f20df0 2023-07-29T16:24:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe69dffb700 1 -- 172.21.15.31:0/1981610835 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6a400ec10 con 0x55baa4f20df0 2023-07-29T16:24:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe6cff0ab80 10 client.28178 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.905+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55baa4c88c00 con 0x55baa4f20df0 2023-07-29T16:24:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe69dffb700 1 -- 172.21.15.31:0/1981610835 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55baa4e9ccb0 con 0x55baa4f20df0 2023-07-29T16:24:28.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe69dffb700 1 -- 172.21.15.31:0/1981610835 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fe6a4024020 con 0x55baa4f20df0 2023-07-29T16:24:28.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6cff0ab80 10 client.28178 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:28.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6cff0ab80 10 client.28178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:28.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6cff0ab80 20 client.28178 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:28.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f35c90 0x55baa4f380a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6cff0ab80 4 client.28178 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55baa4eefd10 con 0x55baa4f35c90 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f35c90 0x55baa4f380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f35c90 0x55baa4f380a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=668a6828f750c847 server_cookie=a3daa59b3ff4e511 in_seq=0 out_seq=0 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.906+0000 7fe69dffb700 10 client.28178 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe69dffb700 1 -- 172.21.15.31:0/1981610835 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55baa4eefd10 con 0x55baa4f35c90 2023-07-29T16:24:28.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe69dffb700 10 client.28178 handle_command_reply: tid=0 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 client.28178 shutdown 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 msgr2=0x55baa4f2a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f2a420 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40039d0 tx=0x7fe6a4004750 comp rx=0 tx=0).stop 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 shutdown_connections 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f35c90 0x55baa4f380a0 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f31460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f29ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe69dffb700 0 client.28178 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/1981610835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f2a420 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.907+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 >> 172.21.15.31:0/1981610835 conn(0x55baa4f15fd0 msgr2=0x55baa4f1aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.908+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 shutdown_connections 2023-07-29T16:24:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.908+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/1981610835 wait complete. 2023-07-29T16:24:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.908+0000 7fe6cff0ab80 20 client.28178 trim_cache size 0 max 16384 2023-07-29T16:24:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.923+0000 7fe6cff0ab80 1 Processor -- start 2023-07-29T16:24:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 -- start start 2023-07-29T16:24:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f28be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f31420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f28130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4f1ce30 con 0x55baa4d7c6b0 2023-07-29T16:24:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4f13150 con 0x55baa4f20df0 2023-07-29T16:24:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4f21490 con 0x55baa4f21cf0 2023-07-29T16:24:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f31420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f31420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51014/0 (socket says 172.21.15.31:51014) 2023-07-29T16:24:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6ba39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f28be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/238937612 learned_addr learned my addr 172.21.15.31:0/238937612 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6bab9c700 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f28130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/238937612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 msgr2=0x55baa4f28130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f28130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/238937612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 msgr2=0x55baa4f28be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.924+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f28be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.925+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/238937612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baa4cc61f0 con 0x55baa4f20df0 2023-07-29T16:24:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.925+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f31420 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009bb0 tx=0x7fe6a400e470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=827ff83935053ce1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.925+0000 7fe6a93ff700 1 -- 172.21.15.31:0/238937612 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4035030 con 0x55baa4f20df0 2023-07-29T16:24:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.925+0000 7fe6a93ff700 1 -- 172.21.15.31:0/238937612 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6a4009400 con 0x55baa4f20df0 2023-07-29T16:24:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.925+0000 7fe6a93ff700 1 -- 172.21.15.31:0/238937612 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a401e680 con 0x55baa4f20df0 2023-07-29T16:24:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/238937612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 msgr2=0x55baa4f31420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f31420 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4009bb0 tx=0x7fe6a400e470 comp rx=0 tx=0).stop 2023-07-29T16:24:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/238937612 shutdown_connections 2023-07-29T16:24:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f21cf0 0x55baa4f28130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f28be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/238937612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f20df0 0x55baa4f31420 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/238937612 >> 172.21.15.31:0/238937612 conn(0x55baa4f15fd0 msgr2=0x55baa4f1aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/238937612 shutdown_connections 2023-07-29T16:24:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/238937612 wait complete. 2023-07-29T16:24:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.926+0000 7fe6cff0ab80 1 -- start start 2023-07-29T16:24:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 Processor -- start 2023-07-29T16:24:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f2d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 0x55baa4f2db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f30b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4f310c0 con 0x55baa4d7c6b0 2023-07-29T16:24:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4f31230 con 0x55baa4f21cf0 2023-07-29T16:24:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baa4f313f0 con 0x55baa4f20df0 2023-07-29T16:24:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.927+0000 7fe6cff0ab80 10 client.0 _ll_register_callbacks cb 0x55baa4e175c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 0x55baa4f2db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6ba39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f2d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6bab9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f30b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 0x55baa4f2db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44788/0 (socket says 172.21.15.31:44788) 2023-07-29T16:24:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/2090528298 learned_addr learned my addr 172.21.15.31:0/2090528298 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 msgr2=0x55baa4f30b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f30b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/2090528298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 msgr2=0x55baa4f2d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:28.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f2d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 -- 172.21.15.31:0/2090528298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55baa4c88c00 con 0x55baa4f20df0 2023-07-29T16:24:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe6b9b9a700 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 0x55baa4f2db60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4013010 tx=0x7fe6a400e9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4686a6d14a750c0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:28.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.928+0000 7fe69dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe69dffb700 1 -- 172.21.15.31:0/2090528298 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a4043070 con 0x55baa4f20df0 2023-07-29T16:24:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6b00a1be0 con 0x55baa4f20df0 2023-07-29T16:24:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe69dffb700 1 -- 172.21.15.31:0/2090528298 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe6a4036070 con 0x55baa4f20df0 2023-07-29T16:24:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe69dffb700 1 -- 172.21.15.31:0/2090528298 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6a4035040 con 0x55baa4f20df0 2023-07-29T16:24:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 10 client.28192 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:28.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55baa4c9e250 con 0x55baa4f20df0 2023-07-29T16:24:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe69dffb700 1 -- 172.21.15.31:0/2090528298 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6a404d020 con 0x55baa4f20df0 2023-07-29T16:24:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe69dffb700 1 -- 172.21.15.31:0/2090528298 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fe6a4046030 con 0x55baa4f20df0 2023-07-29T16:24:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 10 client.28192 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 10 client.28192 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 20 client.28192 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:28.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f88780 0x55baa4f8ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:28.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 4 client.28192 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:24:28.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.929+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55baa4e6ace0 con 0x55baa4f88780 2023-07-29T16:24:28.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.930+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f88780 0x55baa4f8ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:28.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.930+0000 7fe6ba39b700 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f88780 0x55baa4f8ab90 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e97c3b941084a38 server_cookie=c1553dae8139c5a in_seq=0 out_seq=0 2023-07-29T16:24:28.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:28.930+0000 7fe69dffb700 10 client.28192 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:29.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.017+0000 7fe69dffb700 1 -- 172.21.15.31:0/2090528298 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55baa4e6ace0 con 0x55baa4f88780 2023-07-29T16:24:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.017+0000 7fe69dffb700 10 client.28192 handle_command_reply: tid=0 2023-07-29T16:24:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.017+0000 7fe6cff0ab80 1 client.28192 shutdown 2023-07-29T16:24:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 msgr2=0x55baa4f2db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 0x55baa4f2db60 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4013010 tx=0x7fe6a400e9a0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 shutdown_connections 2023-07-29T16:24:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55baa4f88780 0x55baa4f8ab90 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55baa4f21cf0 0x55baa4f30b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55baa4d7c6b0 0x55baa4f2d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 --2- 172.21.15.31:0/2090528298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55baa4f20df0 0x55baa4f2db60 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe69dffb700 0 client.28192 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 >> 172.21.15.31:0/2090528298 conn(0x55baa4f15fd0 msgr2=0x55baa4f28be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 shutdown_connections 2023-07-29T16:24:29.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 1 -- 172.21.15.31:0/2090528298 wait complete. 2023-07-29T16:24:29.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.018+0000 7fe6cff0ab80 20 client.28192 trim_cache size 0 max 16384 2023-07-29T16:24:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:24:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:24:29.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:24:29.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:24:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6a00a0810 msgr2=0x7fe6a00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6a00a0810 0x7fe6a00a2cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe6b4074440 tx=0x7fe6a4009310 comp rx=0 tx=0).stop 2023-07-29T16:24:29.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 msgr2=0x7fe6b4073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:29.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 0x7fe6b4073140 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000ea50 tx=0x7fe6b000ed60 comp rx=0 tx=0).stop 2023-07-29T16:24:29.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 shutdown_connections 2023-07-29T16:24:29.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe6a00a0810 0x7fe6a00a2cc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6b4101ed0 0x7fe6b4073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6b4100350 0x7fe6b4073140 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 --2- 172.21.15.31:0/1555148939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6b4101540 0x7fe6b4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:29.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 >> 172.21.15.31:0/1555148939 conn(0x7fe6b40fb970 msgr2=0x7fe6b4104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 shutdown_connections 2023-07-29T16:24:29.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:29.019+0000 7fe6bc5ff700 1 -- 172.21.15.31:0/1555148939 wait complete. 2023-07-29T16:24:29.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 622350 8.95 MB/sec execute 254 sec latency 30.056 ms 2023-07-29T16:24:30.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:30 smithi039 ceph-mon[128858]: pgmap v641: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 323 op/s 2023-07-29T16:24:30.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:30 smithi156 ceph-mon[129245]: pgmap v641: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 323 op/s 2023-07-29T16:24:30.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 624501 8.96 MB/sec execute 255 sec latency 52.368 ms 2023-07-29T16:24:30.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:30 smithi031 ceph-mon[95504]: pgmap v641: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 323 op/s 2023-07-29T16:24:31.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 625809 8.95 MB/sec execute 256 sec latency 24.085 ms 2023-07-29T16:24:32.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:32 smithi039 ceph-mon[128858]: pgmap v642: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.1 MiB/s wr, 200 op/s 2023-07-29T16:24:32.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:32 smithi156 ceph-mon[129245]: pgmap v642: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.1 MiB/s wr, 200 op/s 2023-07-29T16:24:32.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 628080 8.96 MB/sec execute 257 sec latency 13.330 ms 2023-07-29T16:24:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:32 smithi031 ceph-mon[95504]: pgmap v642: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.1 MiB/s wr, 200 op/s 2023-07-29T16:24:33.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 629372 8.96 MB/sec execute 258 sec latency 12.771 ms 2023-07-29T16:24:34.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:34 smithi039 ceph-mon[128858]: pgmap v643: 97 pgs: 97 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 2.7 MiB/s wr, 293 op/s 2023-07-29T16:24:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:34 smithi156 ceph-mon[129245]: pgmap v643: 97 pgs: 97 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 2.7 MiB/s wr, 293 op/s 2023-07-29T16:24:34.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 631629 8.97 MB/sec execute 259 sec latency 13.673 ms 2023-07-29T16:24:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:34 smithi031 ceph-mon[95504]: pgmap v643: 97 pgs: 97 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 2.7 MiB/s wr, 293 op/s 2023-07-29T16:24:35.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:24:35.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 -- 172.21.15.31:0/3135909629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360100320 msgr2=0x7fc360100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 --2- 172.21.15.31:0/3135909629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360100320 0x7fc360100730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc350009990 tx=0x7fc350009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 -- 172.21.15.31:0/3135909629 shutdown_connections 2023-07-29T16:24:35.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 --2- 172.21.15.31:0/3135909629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101ea0 0x7fc360108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 --2- 172.21.15.31:0/3135909629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360100320 0x7fc360100730 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 --2- 172.21.15.31:0/3135909629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360101510 0x7fc360101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 -- 172.21.15.31:0/3135909629 >> 172.21.15.31:0/3135909629 conn(0x7fc3600fb920 msgr2=0x7fc3600fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 -- 172.21.15.31:0/3135909629 shutdown_connections 2023-07-29T16:24:35.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.519+0000 7fc365fff700 1 -- 172.21.15.31:0/3135909629 wait complete. 2023-07-29T16:24:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 Processor -- start 2023-07-29T16:24:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 -- start start 2023-07-29T16:24:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360100320 0x7fc360074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101510 0x7fc360073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360101ea0 0x7fc360073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc360073e20 con 0x7fc360100320 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc360073f90 con 0x7fc360101ea0 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.521+0000 7fc365fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3601a0140 con 0x7fc360101510 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360100320 0x7fc360074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360100320 0x7fc360074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48572/0 (socket says 172.21.15.31:48572) 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35f7fe700 1 -- 172.21.15.31:0/3329049635 learned_addr learned my addr 172.21.15.31:0/3329049635 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101510 0x7fc360073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35ffff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360101ea0 0x7fc360073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 -- 172.21.15.31:0/3329049635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360101ea0 msgr2=0x7fc360073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360101ea0 0x7fc360073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 -- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360100320 msgr2=0x7fc360074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360100320 0x7fc360074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 -- 172.21.15.31:0/3329049635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500095f0 con 0x7fc360101510 2023-07-29T16:24:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.522+0000 7fc35effd700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101510 0x7fc360073140 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc35400d950 tx=0x7fc35400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.523+0000 7fc35cff9700 1 -- 172.21.15.31:0/3329049635 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35400f890 con 0x7fc360101510 2023-07-29T16:24:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.523+0000 7fc35cff9700 1 -- 172.21.15.31:0/3329049635 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc354010460 con 0x7fc360101510 2023-07-29T16:24:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.523+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3601a0420 con 0x7fc360101510 2023-07-29T16:24:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.523+0000 7fc35cff9700 1 -- 172.21.15.31:0/3329049635 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35400e610 con 0x7fc360101510 2023-07-29T16:24:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.523+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3601a0980 con 0x7fc360101510 2023-07-29T16:24:35.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.526+0000 7fc35cff9700 1 -- 172.21.15.31:0/3329049635 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc3540105d0 con 0x7fc360101510 2023-07-29T16:24:35.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 Processor -- start 2023-07-29T16:24:35.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc35cff9700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3480a0810 0x7fc3480a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc35cff9700 1 -- 172.21.15.31:0/3329049635 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc3540cb6f0 con 0x7fc360101510 2023-07-29T16:24:35.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 -- start start 2023-07-29T16:24:35.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3480a0810 0x7fc3480a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde22ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde23d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde24740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befdd7aa10 con 0x55befde23530 2023-07-29T16:24:35.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befdd783d0 con 0x55befde242f0 2023-07-29T16:24:35.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.528+0000 7fc379873b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befdd73cc0 con 0x55befde22be0 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde22ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde22ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55766/0 (socket says 172.21.15.31:55766) 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35f7fe700 1 -- 172.21.15.31:0/2498279467 learned_addr learned my addr 172.21.15.31:0/2498279467 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35ffff700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde24740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35effd700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde23d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35ffff700 1 -- 172.21.15.31:0/2498279467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 msgr2=0x55befde22ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35ffff700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde22ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35ffff700 1 -- 172.21.15.31:0/2498279467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 msgr2=0x55befde23d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35ffff700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde23d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35ffff700 1 -- 172.21.15.31:0/2498279467 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55befdba1de0 con 0x55befde242f0 2023-07-29T16:24:35.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde22ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:35.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35effd700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde23d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:35.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.529+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3480a0810 0x7fc3480a2cc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc350005cd0 tx=0x7fc3500054d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.530+0000 7fc35ffff700 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde24740 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc358011550 tx=0x7fc358009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f5a9346e98e5686 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.530+0000 7fc345ffb700 1 -- 172.21.15.31:0/2498279467 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800e040 con 0x55befde242f0 2023-07-29T16:24:35.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.530+0000 7fc345ffb700 1 -- 172.21.15.31:0/2498279467 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc35800f040 con 0x55befde242f0 2023-07-29T16:24:35.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.530+0000 7fc345ffb700 1 -- 172.21.15.31:0/2498279467 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358004370 con 0x55befde242f0 2023-07-29T16:24:35.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 -- 172.21.15.31:0/2498279467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 msgr2=0x55befde24740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde24740 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc358011550 tx=0x7fc358009920 comp rx=0 tx=0).stop 2023-07-29T16:24:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 -- 172.21.15.31:0/2498279467 shutdown_connections 2023-07-29T16:24:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde24740 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde22ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 --2- 172.21.15.31:0/2498279467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde23d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 -- 172.21.15.31:0/2498279467 >> 172.21.15.31:0/2498279467 conn(0x55befde183e0 msgr2=0x55befde1d990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 -- 172.21.15.31:0/2498279467 shutdown_connections 2023-07-29T16:24:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.531+0000 7fc379873b80 1 -- 172.21.15.31:0/2498279467 wait complete. 2023-07-29T16:24:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.532+0000 7fc379873b80 1 -- start start 2023-07-29T16:24:35.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.532+0000 7fc379873b80 1 Processor -- start 2023-07-29T16:24:35.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde2c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde2cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde33d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befdd10510 con 0x55befde23530 2023-07-29T16:24:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befdd101b0 con 0x55befde242f0 2023-07-29T16:24:35.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befdbf7920 con 0x55befde22be0 2023-07-29T16:24:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc379873b80 10 client.0 _ll_register_callbacks cb 0x55befdd19620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde2c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde2c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55782/0 (socket says 172.21.15.31:55782) 2023-07-29T16:24:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4135254691 learned_addr learned my addr 172.21.15.31:0/4135254691 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35ffff700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde33d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35effd700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde2cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 msgr2=0x55befde33d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde33d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4135254691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 msgr2=0x55befde2cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde2cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4135254691 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc360109090 con 0x55befde22be0 2023-07-29T16:24:35.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35effd700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde2cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:35.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde2c800 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc350009570 tx=0x7fc350009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea59bdd9aa51d0b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.533+0000 7fc33effd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:35.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc33effd700 1 -- 172.21.15.31:0/4135254691 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350004d70 con 0x55befde22be0 2023-07-29T16:24:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55befdd9ed10 con 0x55befde22be0 2023-07-29T16:24:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc33effd700 1 -- 172.21.15.31:0/4135254691 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc35000f460 con 0x55befde22be0 2023-07-29T16:24:35.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc33effd700 1 -- 172.21.15.31:0/4135254691 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc35000f620 con 0x55befde22be0 2023-07-29T16:24:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc379873b80 10 client.28204 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fc3601a0980 con 0x55befde22be0 2023-07-29T16:24:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc33effd700 1 -- 172.21.15.31:0/4135254691 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55befdd9ed10 con 0x55befde22be0 2023-07-29T16:24:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.534+0000 7fc33effd700 1 -- 172.21.15.31:0/4135254691 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc350018810 con 0x55befde22be0 2023-07-29T16:24:35.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc379873b80 10 client.28204 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc379873b80 10 client.28204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc379873b80 20 client.28204 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc379873b80 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde38460 0x55befde3a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc379873b80 4 client.28204 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55befddf1d70 con 0x55befde38460 2023-07-29T16:24:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc35effd700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde38460 0x55befde3a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc35effd700 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde38460 0x55befde3a870 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9178343b8f7bec75 server_cookie=1d7564c2167f0798 in_seq=0 out_seq=0 2023-07-29T16:24:35.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.535+0000 7fc33effd700 10 client.28204 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:35.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.536+0000 7fc33effd700 1 -- 172.21.15.31:0/4135254691 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55befddf1d70 con 0x55befde38460 2023-07-29T16:24:35.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.536+0000 7fc33effd700 10 client.28204 handle_command_reply: tid=0 2023-07-29T16:24:35.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.536+0000 7fc379873b80 1 client.28204 shutdown 2023-07-29T16:24:35.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.536+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 msgr2=0x55befde2c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.536+0000 7fc379873b80 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde2c800 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc350009570 tx=0x7fc350009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 shutdown_connections 2023-07-29T16:24:35.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde38460 0x55befde3a870 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde33d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde22be0 0x55befde2c800 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 --2- 172.21.15.31:0/4135254691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde2cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 >> 172.21.15.31:0/4135254691 conn(0x55befde183e0 msgr2=0x55befde1d060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc33effd700 0 client.28204 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:35.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 shutdown_connections 2023-07-29T16:24:35.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 1 -- 172.21.15.31:0/4135254691 wait complete. 2023-07-29T16:24:35.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.537+0000 7fc379873b80 20 client.28204 trim_cache size 0 max 16384 2023-07-29T16:24:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.557+0000 7fc379873b80 1 Processor -- start 2023-07-29T16:24:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 -- start start 2023-07-29T16:24:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde22be0 0x55befde2b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde23530 0x55befde33d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde1d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befde1a460 con 0x55befde22be0 2023-07-29T16:24:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befde37f80 con 0x55befde242f0 2023-07-29T16:24:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc379873b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befde15770 con 0x55befde23530 2023-07-29T16:24:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde1d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde23530 0x55befde33d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde1d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48472/0 (socket says 172.21.15.31:48472) 2023-07-29T16:24:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35ffff700 1 -- 172.21.15.31:0/3951196031 learned_addr learned my addr 172.21.15.31:0/3951196031 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde22be0 0x55befde2b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35ffff700 1 -- 172.21.15.31:0/3951196031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde23530 msgr2=0x55befde33d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35ffff700 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde23530 0x55befde33d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.558+0000 7fc35ffff700 1 -- 172.21.15.31:0/3951196031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde22be0 msgr2=0x55befde2b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.559+0000 7fc35ffff700 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde22be0 0x55befde2b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.559+0000 7fc35ffff700 1 -- 172.21.15.31:0/3951196031 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc358011230 con 0x55befde242f0 2023-07-29T16:24:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.559+0000 7fc35ffff700 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde1d060 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55befde18360 tx=0x7fc358011e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd3f7e2251e9807f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.559+0000 7fc3467fc700 1 -- 172.21.15.31:0/3951196031 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800e040 con 0x55befde242f0 2023-07-29T16:24:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.559+0000 7fc3467fc700 1 -- 172.21.15.31:0/3951196031 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc358007500 con 0x55befde242f0 2023-07-29T16:24:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.559+0000 7fc3467fc700 1 -- 172.21.15.31:0/3951196031 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc358027660 con 0x55befde242f0 2023-07-29T16:24:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 -- 172.21.15.31:0/3951196031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 msgr2=0x55befde1d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde1d060 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x55befde18360 tx=0x7fc358011e50 comp rx=0 tx=0).stop 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 -- 172.21.15.31:0/3951196031 shutdown_connections 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde242f0 0x55befde1d060 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde22be0 0x55befde2b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 --2- 172.21.15.31:0/3951196031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde23530 0x55befde33d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 -- 172.21.15.31:0/3951196031 >> 172.21.15.31:0/3951196031 conn(0x55befde183e0 msgr2=0x55befde1e690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 -- 172.21.15.31:0/3951196031 shutdown_connections 2023-07-29T16:24:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.560+0000 7fc379873b80 1 -- 172.21.15.31:0/3951196031 wait complete. 2023-07-29T16:24:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 -- start start 2023-07-29T16:24:35.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 Processor -- start 2023-07-29T16:24:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde22be0 0x55befde33770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde33cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde242f0 0x55befde30300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befde309a0 con 0x55befde23530 2023-07-29T16:24:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befde30b10 con 0x55befde22be0 2023-07-29T16:24:35.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55befde30cd0 con 0x55befde242f0 2023-07-29T16:24:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc379873b80 10 client.0 _ll_register_callbacks cb 0x55befdd19620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc35effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde33cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc35f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde22be0 0x55befde33770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc35effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde33cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48606/0 (socket says 172.21.15.31:48606) 2023-07-29T16:24:35.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc35effd700 1 -- 172.21.15.31:0/4278276269 learned_addr learned my addr 172.21.15.31:0/4278276269 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.561+0000 7fc35ffff700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde242f0 0x55befde30300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4278276269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde242f0 msgr2=0x55befde30300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde242f0 0x55befde30300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4278276269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 msgr2=0x55befde33cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde33cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35f7fe700 1 -- 172.21.15.31:0/4278276269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55befdba1de0 con 0x55befde22be0 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35effd700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde33cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35ffff700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde242f0 0x55befde30300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc35f7fe700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde22be0 0x55befde33770 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc350005f50 tx=0x7fc3500057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34c80a581f1ebee0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc33effd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc33effd700 1 -- 172.21.15.31:0/4278276269 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc350018810 con 0x55befde22be0 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc35800f040 con 0x55befde22be0 2023-07-29T16:24:35.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.562+0000 7fc33effd700 1 -- 172.21.15.31:0/4278276269 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc350018970 con 0x55befde22be0 2023-07-29T16:24:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc33effd700 1 -- 172.21.15.31:0/4278276269 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3500047e0 con 0x55befde22be0 2023-07-29T16:24:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 10 client.28202 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55befdb8ac60 con 0x55befde22be0 2023-07-29T16:24:35.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc33effd700 1 -- 172.21.15.31:0/4278276269 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc350051020 con 0x55befde22be0 2023-07-29T16:24:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc33effd700 1 -- 172.21.15.31:0/4278276269 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc350043480 con 0x55befde22be0 2023-07-29T16:24:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 10 client.28202 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 10 client.28202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 20 client.28202 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde8b240 0x55befde8d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 4 client.28202 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.563+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55befdd6cd40 con 0x55befde8b240 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc35effd700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde8b240 0x55befde8d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc35effd700 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde8b240 0x55befde8d620 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ab607f33ec32d9 server_cookie=15dcbe7a474816fd in_seq=0 out_seq=0 2023-07-29T16:24:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc33effd700 10 client.28202 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc33effd700 1 -- 172.21.15.31:0/4278276269 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55befdd6cd40 con 0x55befde8b240 2023-07-29T16:24:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc33effd700 10 client.28202 handle_command_reply: tid=0 2023-07-29T16:24:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc379873b80 1 client.28202 shutdown 2023-07-29T16:24:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde22be0 msgr2=0x55befde33770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc379873b80 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde22be0 0x55befde33770 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc350005f50 tx=0x7fc3500057c0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 shutdown_connections 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc379873b80 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55befde8b240 0x55befde8d620 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.564+0000 7fc379873b80 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55befde242f0 0x55befde30300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc379873b80 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55befde22be0 0x55befde33770 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc379873b80 1 --2- 172.21.15.31:0/4278276269 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55befde23530 0x55befde33cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 >> 172.21.15.31:0/4278276269 conn(0x55befde183e0 msgr2=0x55befde1d060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc33effd700 0 client.28202 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:35.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 shutdown_connections 2023-07-29T16:24:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc379873b80 1 -- 172.21.15.31:0/4278276269 wait complete. 2023-07-29T16:24:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.565+0000 7fc379873b80 20 client.28202 trim_cache size 0 max 16384 2023-07-29T16:24:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:24:35.576 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:24:35.576 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:24:35.576 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:24:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3480a0810 msgr2=0x7fc3480a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3480a0810 0x7fc3480a2cc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc350005cd0 tx=0x7fc3500054d0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101510 msgr2=0x7fc360073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101510 0x7fc360073140 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc35400d950 tx=0x7fc35400dc60 comp rx=0 tx=0).stop 2023-07-29T16:24:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 shutdown_connections 2023-07-29T16:24:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc3480a0810 0x7fc3480a2cc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc360101ea0 0x7fc360073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc360100320 0x7fc360074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 --2- 172.21.15.31:0/3329049635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc360101510 0x7fc360073140 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 >> 172.21.15.31:0/3329049635 conn(0x7fc3600fb920 msgr2=0x7fc360104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 shutdown_connections 2023-07-29T16:24:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.566+0000 7fc365fff700 1 -- 172.21.15.31:0/3329049635 wait complete. 2023-07-29T16:24:35.584 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:24:35.585 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:24:35.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.812+0000 7fbf32bff700 1 -- 172.21.15.31:0/339308998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c1036f0 msgr2=0x7fbf2c105ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.812+0000 7fbf32bff700 1 --2- 172.21.15.31:0/339308998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c105ad0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009990 tx=0x7fbf18009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 -- 172.21.15.31:0/339308998 shutdown_connections 2023-07-29T16:24:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 --2- 172.21.15.31:0/339308998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c106010 0x7fbf2c108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 --2- 172.21.15.31:0/339308998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c1031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 --2- 172.21.15.31:0/339308998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c105ad0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 -- 172.21.15.31:0/339308998 >> 172.21.15.31:0/339308998 conn(0x7fbf2c0fa810 msgr2=0x7fbf2c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 -- 172.21.15.31:0/339308998 shutdown_connections 2023-07-29T16:24:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.813+0000 7fbf32bff700 1 -- 172.21.15.31:0/339308998 wait complete. 2023-07-29T16:24:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.814+0000 7fbf32bff700 1 Processor -- start 2023-07-29T16:24:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 -- start start 2023-07-29T16:24:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c106010 0x7fbf2c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c19dbd0 con 0x7fbf2c1036f0 2023-07-29T16:24:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c19dd10 con 0x7fbf2c106010 2023-07-29T16:24:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf32bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c19dea0 con 0x7fbf2c100dd0 2023-07-29T16:24:35.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf3099b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf3099b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55814/0 (socket says 172.21.15.31:55814) 2023-07-29T16:24:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf3099b700 1 -- 172.21.15.31:0/2503705825 learned_addr learned my addr 172.21.15.31:0/2503705825 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.815+0000 7fbf3119c700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c106010 0x7fbf2c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf3099b700 1 -- 172.21.15.31:0/2503705825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c106010 msgr2=0x7fbf2c19d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf3099b700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c106010 0x7fbf2c19d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf3099b700 1 -- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c1036f0 msgr2=0x7fbf2c196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf3099b700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf3099b700 1 -- 172.21.15.31:0/2503705825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf180095f0 con 0x7fbf2c100dd0 2023-07-29T16:24:35.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf3099b700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c196050 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000e9a0 tx=0x7fbf2000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.816+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20009790 con 0x7fbf2c100dd0 2023-07-29T16:24:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.817+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf2000cbd0 con 0x7fbf2c100dd0 2023-07-29T16:24:35.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.817+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c0fec20 con 0x7fbf2c100dd0 2023-07-29T16:24:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.817+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20018680 con 0x7fbf2c100dd0 2023-07-29T16:24:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.817+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c0ff180 con 0x7fbf2c100dd0 2023-07-29T16:24:35.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.818+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c10bb90 con 0x7fbf2c100dd0 2023-07-29T16:24:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.820+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbf2000cd40 con 0x7fbf2c100dd0 2023-07-29T16:24:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.822+0000 7fbf29ffb700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbf1c0a0860 0x7fbf1c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.822+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbf20014070 con 0x7fbf2c100dd0 2023-07-29T16:24:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.822+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbf1c0a0860 0x7fbf1c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:35.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.823+0000 7fbf2bfff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbf1c0a0860 0x7fbf1c0a2d10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800b2e0 tx=0x7fbf18005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:35.825+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbf20095970 con 0x7fbf2c100dd0 2023-07-29T16:24:35.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 632692 8.96 MB/sec execute 260 sec latency 62.705 ms 2023-07-29T16:24:36.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.124+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf2c0ffab0 con 0x7fbf2c100dd0 2023-07-29T16:24:36.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.125+0000 7fbf29ffb700 1 -- 172.21.15.31:0/2503705825 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10774 (secure 0 0 0) 0x7fbf20098f90 con 0x7fbf2c100dd0 2023-07-29T16:24:36.124 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:24:36.124 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:21:03.460328+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_26551":{"gid":26551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/80628213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":80628213},{"type":"v1","addr":"172.21.15.156:6837","nonce":80628213}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:24:36.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbf1c0a0860 msgr2=0x7fbf1c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbf1c0a0860 0x7fbf1c0a2d10 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800b2e0 tx=0x7fbf18005fd0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 msgr2=0x7fbf2c196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c196050 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000e9a0 tx=0x7fbf2000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 shutdown_connections 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbf1c0a0860 0x7fbf1c0a2d10 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbf2c106010 0x7fbf2c19d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf2c100dd0 0x7fbf2c196050 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 --2- 172.21.15.31:0/2503705825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf2c1036f0 0x7fbf2c196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 >> 172.21.15.31:0/2503705825 conn(0x7fbf2c0fa810 msgr2=0x7fbf2c102210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 shutdown_connections 2023-07-29T16:24:36.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.128+0000 7fbf32bff700 1 -- 172.21.15.31:0/2503705825 wait complete. 2023-07-29T16:24:36.128 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2023-07-29T16:24:36.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:24:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 -- 172.21.15.31:0/363303005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038100330 msgr2=0x7f9038100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 --2- 172.21.15.31:0/363303005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038100330 0x7f9038100740 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9028009990 tx=0x7f9028009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 -- 172.21.15.31:0/363303005 shutdown_connections 2023-07-29T16:24:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 --2- 172.21.15.31:0/363303005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9038101eb0 0x7f9038108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 --2- 172.21.15.31:0/363303005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038100330 0x7f9038100740 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 --2- 172.21.15.31:0/363303005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038101520 0x7f9038101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 -- 172.21.15.31:0/363303005 >> 172.21.15.31:0/363303005 conn(0x7f90380fb970 msgr2=0x7f90380fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 -- 172.21.15.31:0/363303005 shutdown_connections 2023-07-29T16:24:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.385+0000 7f903e9ff700 1 -- 172.21.15.31:0/363303005 wait complete. 2023-07-29T16:24:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.386+0000 7f903e9ff700 1 Processor -- start 2023-07-29T16:24:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 -- start start 2023-07-29T16:24:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 0x7f9038198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038101520 0x7f90381987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9038101eb0 0x7f903819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903819fdf0 con 0x7f9038101520 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903819ff30 con 0x7f9038100330 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903e9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90381a00c0 con 0x7f9038101eb0 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f9037fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 0x7f9038198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f9037fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 0x7f9038198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48512/0 (socket says 172.21.15.31:48512) 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f9037fff700 1 -- 172.21.15.31:0/3417495913 learned_addr learned my addr 172.21.15.31:0/3417495913 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f90377fe700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038101520 0x7f90381987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.387+0000 7f903cf9c700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9038101eb0 0x7f903819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f9037fff700 1 -- 172.21.15.31:0/3417495913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9038101eb0 msgr2=0x7f903819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f9037fff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9038101eb0 0x7f903819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f9037fff700 1 -- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038101520 msgr2=0x7f90381987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f9037fff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038101520 0x7f90381987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f9037fff700 1 -- 172.21.15.31:0/3417495913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90280095f0 con 0x7f9038100330 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f9037fff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 0x7f9038198270 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9028000c00 tx=0x7f90280057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f90357fa700 1 -- 172.21.15.31:0/3417495913 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902801d070 con 0x7f9038100330 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90381a0340 con 0x7f9038100330 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f90357fa700 1 -- 172.21.15.31:0/3417495913 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9028021410 con 0x7f9038100330 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f90357fa700 1 -- 172.21.15.31:0/3417495913 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028017650 con 0x7f9038100330 2023-07-29T16:24:36.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.388+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90381a0840 con 0x7f9038100330 2023-07-29T16:24:36.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.391+0000 7f90357fa700 1 -- 172.21.15.31:0/3417495913 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9028021580 con 0x7f9038100330 2023-07-29T16:24:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 Processor -- start 2023-07-29T16:24:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90357fa700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90200a0790 0x7f90200a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.392 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 -- start start 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90357fa700 1 -- 172.21.15.31:0/3417495913 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f90280cce60 con 0x7f9038100330 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9facff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fadd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fae740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9f04a10 con 0x55a4f9fae2f0 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9f023d0 con 0x55a4f9fad530 2023-07-29T16:24:36.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f90522fdb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9efdcc0 con 0x55a4f9facbe0 2023-07-29T16:24:36.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.393+0000 7f903cf9c700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90200a0790 0x7f90200a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f9037fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9facff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fadd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f9037fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9facff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55842/0 (socket says 172.21.15.31:55842) 2023-07-29T16:24:36.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f9037fff700 1 -- 172.21.15.31:0/1790738795 learned_addr learned my addr 172.21.15.31:0/1790738795 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f903cf9c700 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 -- 172.21.15.31:0/1790738795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 msgr2=0x55a4f9facff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9facff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 -- 172.21.15.31:0/1790738795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 msgr2=0x55a4f9fae740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fae740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 -- 172.21.15.31:0/1790738795 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4f9d2bde0 con 0x55a4f9fad530 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f903cf9c700 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f9037fff700 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9facff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f90377fe700 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fadd60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f902c009cb0 tx=0x7f902c011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beba01f550c8024b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.394+0000 7f903cf9c700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90200a0790 0x7f90200a2c40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f903000f6e0 tx=0x7f9030009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f901e7fc700 1 -- 172.21.15.31:0/1790738795 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c011d20 con 0x55a4f9fad530 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f901e7fc700 1 -- 172.21.15.31:0/1790738795 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f902c011e80 con 0x55a4f9fad530 2023-07-29T16:24:36.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f901e7fc700 1 -- 172.21.15.31:0/1790738795 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c005280 con 0x55a4f9fad530 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 -- 172.21.15.31:0/1790738795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 msgr2=0x55a4f9fadd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fadd60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f902c009cb0 tx=0x7f902c011440 comp rx=0 tx=0).stop 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 -- 172.21.15.31:0/1790738795 shutdown_connections 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fae740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9facff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1790738795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fadd60 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.395+0000 7f90522fdb80 1 -- 172.21.15.31:0/1790738795 >> 172.21.15.31:0/1790738795 conn(0x55a4f9fa23e0 msgr2=0x55a4f9fa7990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.399 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.396+0000 7f90522fdb80 1 -- 172.21.15.31:0/1790738795 shutdown_connections 2023-07-29T16:24:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.396+0000 7f90522fdb80 1 -- 172.21.15.31:0/1790738795 wait complete. 2023-07-29T16:24:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.396+0000 7f90522fdb80 1 -- start start 2023-07-29T16:24:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.396+0000 7f90522fdb80 1 Processor -- start 2023-07-29T16:24:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fbdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9e9a510 con 0x55a4f9fad530 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9e9a1b0 con 0x55a4f9fae2f0 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9d81920 con 0x55a4f9facbe0 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90522fdb80 10 client.0 _ll_register_callbacks cb 0x55a4f9ea3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f903cf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fbdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f9037fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f903cf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fbdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48532/0 (socket says 172.21.15.31:48532) 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f903cf9c700 1 -- 172.21.15.31:0/3547449529 learned_addr learned my addr 172.21.15.31:0/3547449529 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f90377fe700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f9037fff700 1 -- 172.21.15.31:0/3547449529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 msgr2=0x55a4f9fbdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f9037fff700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fbdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f9037fff700 1 -- 172.21.15.31:0/3547449529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 msgr2=0x55a4f9fb6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f9037fff700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f9037fff700 1 -- 172.21.15.31:0/3547449529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90381a0840 con 0x55a4f9facbe0 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.397+0000 7f903cf9c700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fbdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90377fe700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f9037fff700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb67f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f90280a2420 tx=0x7f9028021b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40465a9614b239e3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90177fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90177fe700 1 -- 172.21.15.31:0/3547449529 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902809e350 con 0x55a4f9facbe0 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90177fe700 1 -- 172.21.15.31:0/3547449529 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f902809e5c0 con 0x55a4f9facbe0 2023-07-29T16:24:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4f9f7bd70 con 0x55a4f9facbe0 2023-07-29T16:24:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90177fe700 1 -- 172.21.15.31:0/3547449529 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90280a2950 con 0x55a4f9facbe0 2023-07-29T16:24:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90522fdb80 10 client.28228 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f902c0098e0 con 0x55a4f9facbe0 2023-07-29T16:24:36.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.398+0000 7f90177fe700 1 -- 172.21.15.31:0/3547449529 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a4f9f7bd70 con 0x55a4f9facbe0 2023-07-29T16:24:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90177fe700 1 -- 172.21.15.31:0/3547449529 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f90280ca2a0 con 0x55a4f9facbe0 2023-07-29T16:24:36.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90522fdb80 10 client.28228 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90522fdb80 10 client.28228 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90522fdb80 20 client.28228 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90522fdb80 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4f9fc20c0 0x55a4f9fc44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90522fdb80 4 client.28228 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4f9ef6d40 con 0x55a4f9fc20c0 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.399+0000 7f90377fe700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4f9fc20c0 0x55a4f9fc44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90377fe700 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4f9fc20c0 0x55a4f9fc44a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e76ff74531f2823 server_cookie=81cb8c7aadbb9b97 in_seq=0 out_seq=0 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90177fe700 10 client.28228 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90177fe700 1 -- 172.21.15.31:0/3547449529 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a4f9ef6d40 con 0x55a4f9fc20c0 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90177fe700 10 client.28228 handle_command_reply: tid=0 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90522fdb80 1 client.28228 shutdown 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 msgr2=0x55a4f9fb67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.400+0000 7f90522fdb80 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb67f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f90280a2420 tx=0x7f9028021b60 comp rx=0 tx=0).stop 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 shutdown_connections 2023-07-29T16:24:36.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4f9fc20c0 0x55a4f9fc44a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fbdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb67f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90177fe700 0 client.28228 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 --2- 172.21.15.31:0/3547449529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 >> 172.21.15.31:0/3547449529 conn(0x55a4f9fa23e0 msgr2=0x55a4f9fa7060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 shutdown_connections 2023-07-29T16:24:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 1 -- 172.21.15.31:0/3547449529 wait complete. 2023-07-29T16:24:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.401+0000 7f90522fdb80 20 client.28228 trim_cache size 0 max 16384 2023-07-29T16:24:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 Processor -- start 2023-07-29T16:24:36.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 -- start start 2023-07-29T16:24:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fc44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fbbd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fb5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9fb5ca0 con 0x55a4f9fae2f0 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9fadbc0 con 0x55a4f9fad530 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90522fdb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9fa4460 con 0x55a4f9facbe0 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fbbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f903cf9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fb5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f9037fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fc44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fbbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48534/0 (socket says 172.21.15.31:48534) 2023-07-29T16:24:36.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.415+0000 7f90377fe700 1 -- 172.21.15.31:0/88962986 learned_addr learned my addr 172.21.15.31:0/88962986 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f90377fe700 1 -- 172.21.15.31:0/88962986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 msgr2=0x55a4f9fc44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f90377fe700 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fc44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f90377fe700 1 -- 172.21.15.31:0/88962986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 msgr2=0x55a4f9fb5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f90377fe700 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fb5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f90377fe700 1 -- 172.21.15.31:0/88962986 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4f9d14c60 con 0x55a4f9fad530 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f90377fe700 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fbbd20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f902c009cb0 tx=0x7f902c004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836f27a8df6f0291 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f901effd700 1 -- 172.21.15.31:0/88962986 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c004710 con 0x55a4f9fad530 2023-07-29T16:24:36.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f901effd700 1 -- 172.21.15.31:0/88962986 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f902c029e90 con 0x55a4f9fad530 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.416+0000 7f901effd700 1 -- 172.21.15.31:0/88962986 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c0043b0 con 0x55a4f9fad530 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 -- 172.21.15.31:0/88962986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 msgr2=0x55a4f9fbbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fbbd20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f902c009cb0 tx=0x7f902c004ad0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 -- 172.21.15.31:0/88962986 shutdown_connections 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fb5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fc44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 --2- 172.21.15.31:0/88962986 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fad530 0x55a4f9fbbd20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 -- 172.21.15.31:0/88962986 >> 172.21.15.31:0/88962986 conn(0x55a4f9fa23e0 msgr2=0x55a4f9fb5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 -- 172.21.15.31:0/88962986 shutdown_connections 2023-07-29T16:24:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.417+0000 7f90522fdb80 1 -- 172.21.15.31:0/88962986 wait complete. 2023-07-29T16:24:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.418+0000 7f90522fdb80 1 -- start start 2023-07-29T16:24:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.418+0000 7f90522fdb80 1 Processor -- start 2023-07-29T16:24:36.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fba960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9fbaea0 con 0x55a4f9fad530 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9fbb010 con 0x55a4f9fae2f0 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f9fbb1d0 con 0x55a4f9facbe0 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90522fdb80 10 client.0 _ll_register_callbacks cb 0x55a4f9ea3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:36.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f903cf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fba960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f903cf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fba960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48536/0 (socket says 172.21.15.31:48536) 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f903cf9c700 1 -- 172.21.15.31:0/1883542977 learned_addr learned my addr 172.21.15.31:0/1883542977 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55882/0 (socket says 172.21.15.31:55882) 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f90377fe700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 -- 172.21.15.31:0/1883542977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 msgr2=0x55a4f9fba960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fba960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 -- 172.21.15.31:0/1883542977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 msgr2=0x55a4f9fb9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f9037fff700 1 -- 172.21.15.31:0/1883542977 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c01d040 con 0x55a4f9facbe0 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.419+0000 7f903cf9c700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f9037fff700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb9410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f902809a290 tx=0x7f90280a26d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a94651a06a1b74b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90177fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90177fe700 1 -- 172.21.15.31:0/1883542977 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902809e5c0 con 0x55a4f9facbe0 2023-07-29T16:24:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90177fe700 1 -- 172.21.15.31:0/1883542977 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f90280a2950 con 0x55a4f9facbe0 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4f9f28d10 con 0x55a4f9facbe0 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90177fe700 1 -- 172.21.15.31:0/1883542977 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9028099a80 con 0x55a4f9facbe0 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 10 client.28234 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55a4f9d52250 con 0x55a4f9facbe0 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90177fe700 1 -- 172.21.15.31:0/1883542977 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a4f9f28d10 con 0x55a4f9facbe0 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90177fe700 1 -- 172.21.15.31:0/1883542977 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9028096580 con 0x55a4f9facbe0 2023-07-29T16:24:36.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 10 client.28234 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 10 client.28234 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 20 client.28234 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4fa013fd0 0x55a4fa0163e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 4 client.28234 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.420+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a4f9ef6a60 con 0x55a4fa013fd0 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.421+0000 7f90377fe700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4fa013fd0 0x55a4fa0163e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.421+0000 7f90377fe700 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4fa013fd0 0x55a4fa0163e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=316ed0e8fef0545b server_cookie=d813a86f04e2eb37 in_seq=0 out_seq=0 2023-07-29T16:24:36.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.421+0000 7f90177fe700 10 client.28234 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.421+0000 7f90177fe700 1 -- 172.21.15.31:0/1883542977 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a4f9ef6a60 con 0x55a4fa013fd0 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.421+0000 7f90177fe700 10 client.28234 handle_command_reply: tid=0 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.421+0000 7f90522fdb80 1 client.28234 shutdown 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 msgr2=0x55a4f9fb9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb9410 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f902809a290 tx=0x7f90280a26d0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 shutdown_connections 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55a4fa013fd0 0x55a4fa0163e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a4f9fae2f0 0x55a4f9fba960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a4f9facbe0 0x55a4f9fb9410 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90177fe700 0 client.28234 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 --2- 172.21.15.31:0/1883542977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a4f9fad530 0x55a4f9fb9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 >> 172.21.15.31:0/1883542977 conn(0x55a4f9fa23e0 msgr2=0x55a4f9fb5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 shutdown_connections 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 1 -- 172.21.15.31:0/1883542977 wait complete. 2023-07-29T16:24:36.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.422+0000 7f90522fdb80 20 client.28234 trim_cache size 0 max 16384 2023-07-29T16:24:36.429 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:24:36.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90200a0790 msgr2=0x7f90200a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90200a0790 0x7f90200a2c40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f903000f6e0 tx=0x7f9030009920 comp rx=0 tx=0).stop 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 msgr2=0x7f9038198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 0x7f9038198270 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9028000c00 tx=0x7f90280057c0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 shutdown_connections 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90200a0790 0x7f90200a2c40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9038101eb0 0x7f903819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9038100330 0x7f9038198270 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 --2- 172.21.15.31:0/3417495913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9038101520 0x7f90381987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 >> 172.21.15.31:0/3417495913 conn(0x7f90380fb970 msgr2=0x7f9038106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 shutdown_connections 2023-07-29T16:24:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.423+0000 7f903e9ff700 1 -- 172.21.15.31:0/3417495913 wait complete. 2023-07-29T16:24:36.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:24:36.665 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 -- 172.21.15.31:0/1546287872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 msgr2=0x7f18d0101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 --2- 172.21.15.31:0/1546287872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0101960 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009990 tx=0x7f18c0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 -- 172.21.15.31:0/1546287872 shutdown_connections 2023-07-29T16:24:36.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 --2- 172.21.15.31:0/1546287872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18d0101ea0 0x7f18d0108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 --2- 172.21.15.31:0/1546287872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18d0100320 0x7f18d0100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 --2- 172.21.15.31:0/1546287872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0101960 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 -- 172.21.15.31:0/1546287872 >> 172.21.15.31:0/1546287872 conn(0x7f18d00fb920 msgr2=0x7f18d00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 -- 172.21.15.31:0/1546287872 shutdown_connections 2023-07-29T16:24:36.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.666+0000 7f18d7fff700 1 -- 172.21.15.31:0/1546287872 wait complete. 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 Processor -- start 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 -- start start 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18d0100320 0x7f18d0198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18d0101ea0 0x7f18d019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d019fe80 con 0x7f18d0100320 2023-07-29T16:24:36.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d019ffc0 con 0x7f18d0101510 2023-07-29T16:24:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.668+0000 7f18d7fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d01a0150 con 0x7f18d0101ea0 2023-07-29T16:24:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d5d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18d0100320 0x7f18d0198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48550/0 (socket says 172.21.15.31:48550) 2023-07-29T16:24:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 -- 172.21.15.31:0/419189604 learned_addr learned my addr 172.21.15.31:0/419189604 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d659c700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18d0101ea0 0x7f18d019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 -- 172.21.15.31:0/419189604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18d0101ea0 msgr2=0x7f18d019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18d0101ea0 0x7f18d019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 -- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18d0100320 msgr2=0x7f18d0198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18d0100320 0x7f18d0198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 -- 172.21.15.31:0/419189604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c8009c40 con 0x7f18d0101510 2023-07-29T16:24:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.669+0000 7f18d559a700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0198840 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f18c000b620 tx=0x7f18c0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.670+0000 7f18c6ffd700 1 -- 172.21.15.31:0/419189604 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c001d070 con 0x7f18d0101510 2023-07-29T16:24:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.670+0000 7f18c6ffd700 1 -- 172.21.15.31:0/419189604 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18c0022470 con 0x7f18d0101510 2023-07-29T16:24:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.670+0000 7f18c6ffd700 1 -- 172.21.15.31:0/419189604 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0017650 con 0x7f18d0101510 2023-07-29T16:24:36.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.670+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c00095f0 con 0x7f18d0101510 2023-07-29T16:24:36.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.670+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d01a0610 con 0x7f18d0101510 2023-07-29T16:24:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.673+0000 7f18c6ffd700 1 -- 172.21.15.31:0/419189604 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f18c00225e0 con 0x7f18d0101510 2023-07-29T16:24:36.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.674+0000 7f18eb92db80 1 Processor -- start 2023-07-29T16:24:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 -- start start 2023-07-29T16:24:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f099be0 0x56046f099ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f09ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09b2f0 0x56046f09b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046eff1a10 con 0x56046f09b2f0 2023-07-29T16:24:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046efef3d0 con 0x56046f099be0 2023-07-29T16:24:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18eb92db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046efeacc0 con 0x56046f09a530 2023-07-29T16:24:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18d659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f099be0 0x56046f099ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18d659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f099be0 0x56046f099ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48566/0 (socket says 172.21.15.31:48566) 2023-07-29T16:24:36.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f09ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18d659c700 1 -- 172.21.15.31:0/4003504146 learned_addr learned my addr 172.21.15.31:0/4003504146 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.675+0000 7f18c6ffd700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18bc0a0810 0x7f18bc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18c6ffd700 1 -- 172.21.15.31:0/419189604 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18c00ccf00 con 0x7f18d0101510 2023-07-29T16:24:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4003504146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f099be0 msgr2=0x56046f099ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f099be0 0x56046f099ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d559a700 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09b2f0 0x56046f09b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4003504146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09b2f0 msgr2=0x56046f09b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d659c700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18bc0a0810 0x7f18bc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09b2f0 0x56046f09b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4003504146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56046ee18de0 con 0x56046f09a530 2023-07-29T16:24:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d559a700 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09b2f0 0x56046f09b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:36.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f09ad60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009c80 tx=0x7f18cc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be335e2b440c8d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.676+0000 7f18d659c700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18bc0a0810 0x7f18bc0a2cc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x56046f099980 tx=0x7f18c8004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.677+0000 7f18bbfff700 1 -- 172.21.15.31:0/4003504146 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc011cb0 con 0x56046f09a530 2023-07-29T16:24:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.677+0000 7f18bbfff700 1 -- 172.21.15.31:0/4003504146 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18cc011e10 con 0x56046f09a530 2023-07-29T16:24:36.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.677+0000 7f18bbfff700 1 -- 172.21.15.31:0/4003504146 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc005280 con 0x56046f09a530 2023-07-29T16:24:36.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.677+0000 7f18eb92db80 1 -- 172.21.15.31:0/4003504146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 msgr2=0x56046f09ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.677+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f09ad60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009c80 tx=0x7f18cc009f90 comp rx=0 tx=0).stop 2023-07-29T16:24:36.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 -- 172.21.15.31:0/4003504146 shutdown_connections 2023-07-29T16:24:36.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09b2f0 0x56046f09b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f099be0 0x56046f099ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4003504146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f09ad60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 -- 172.21.15.31:0/4003504146 >> 172.21.15.31:0/4003504146 conn(0x56046f08f3e0 msgr2=0x56046f094990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 -- 172.21.15.31:0/4003504146 shutdown_connections 2023-07-29T16:24:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 -- 172.21.15.31:0/4003504146 wait complete. 2023-07-29T16:24:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 -- start start 2023-07-29T16:24:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.678+0000 7f18eb92db80 1 Processor -- start 2023-07-29T16:24:36.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0aada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046ef87510 con 0x56046f09a530 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046ef871b0 con 0x56046f09b2f0 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046ee6e920 con 0x56046f099be0 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18eb92db80 10 client.0 _ll_register_callbacks cb 0x56046ef90620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:36.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55924/0 (socket says 172.21.15.31:55924) 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4124328650 learned_addr learned my addr 172.21.15.31:0/4124328650 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18d559a700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.679+0000 7f18d659c700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0aada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4124328650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 msgr2=0x56046f0aada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0aada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 msgr2=0x56046f0a3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18d5d9b700 1 -- 172.21.15.31:0/4124328650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc0098e0 con 0x56046f099be0 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a3820 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f18cc011720 tx=0x7f18cc011b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f51be16e1e82005 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.680+0000 7f18b97fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18b97fa700 1 -- 172.21.15.31:0/4124328650 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc005080 con 0x56046f099be0 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56046f015d10 con 0x56046f099be0 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18b97fa700 1 -- 172.21.15.31:0/4124328650 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18cc0051e0 con 0x56046f099be0 2023-07-29T16:24:36.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18b97fa700 1 -- 172.21.15.31:0/4124328650 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18cc0053e0 con 0x56046f099be0 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18eb92db80 10 client.28246 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56046ee01c60 con 0x56046f099be0 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18b97fa700 1 -- 172.21.15.31:0/4124328650 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56046f015d10 con 0x56046f099be0 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.681+0000 7f18b97fa700 1 -- 172.21.15.31:0/4124328650 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f18cc022400 con 0x56046f099be0 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18eb92db80 10 client.28246 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18eb92db80 10 client.28246 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:24:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18eb92db80 20 client.28246 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f0af3e0 0x56046f0b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18eb92db80 4 client.28246 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56046f068d70 con 0x56046f0af3e0 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.682+0000 7f18d559a700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f0af3e0 0x56046f0b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18b97fa700 10 client.28246 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18d559a700 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f0af3e0 0x56046f0b17f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8de5749d6f0227e server_cookie=d7514a3fd681a3da in_seq=0 out_seq=0 2023-07-29T16:24:36.694 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18b97fa700 1 -- 172.21.15.31:0/4124328650 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56046f068d70 con 0x56046f0af3e0 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18b97fa700 10 client.28246 handle_command_reply: tid=0 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18eb92db80 1 client.28246 shutdown 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 msgr2=0x56046f0a3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.683+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a3820 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f18cc011720 tx=0x7f18cc011b60 comp rx=0 tx=0).stop 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 shutdown_connections 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f0af3e0 0x56046f0b17f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0aada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a3820 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18b97fa700 0 client.28246 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:36.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 --2- 172.21.15.31:0/4124328650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 >> 172.21.15.31:0/4124328650 conn(0x56046f08f3e0 msgr2=0x56046f094060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 shutdown_connections 2023-07-29T16:24:36.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 1 -- 172.21.15.31:0/4124328650 wait complete. 2023-07-29T16:24:36.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.684+0000 7f18eb92db80 20 client.28246 trim_cache size 0 max 16384 2023-07-29T16:24:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 Processor -- start 2023-07-29T16:24:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 -- start start 2023-07-29T16:24:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046f0a33f0 con 0x56046f09a530 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046f08c770 con 0x56046f09b2f0 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18eb92db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046f0a49a0 con 0x56046f099be0 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18d659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18d559a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18d5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18d659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48586/0 (socket says 172.21.15.31:48586) 2023-07-29T16:24:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.698+0000 7f18d659c700 1 -- 172.21.15.31:0/2859957029 learned_addr learned my addr 172.21.15.31:0/2859957029 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d5d9b700 1 -- 172.21.15.31:0/2859957029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 msgr2=0x56046f0a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d5d9b700 1 -- 172.21.15.31:0/2859957029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 msgr2=0x56046f0a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d5d9b700 1 -- 172.21.15.31:0/2859957029 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56046ee3f250 con 0x56046f099be0 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d659c700 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d559a700 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a8d50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009bd0 tx=0x7f18cc0228e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5932c9dbc5fb4fb0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2859957029 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc004b00 con 0x56046f099be0 2023-07-29T16:24:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2859957029 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18cc004c60 con 0x56046f099be0 2023-07-29T16:24:36.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.699+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2859957029 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc00e770 con 0x56046f099be0 2023-07-29T16:24:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 -- 172.21.15.31:0/2859957029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 msgr2=0x56046f0a8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a8d50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009bd0 tx=0x7f18cc0228e0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 -- 172.21.15.31:0/2859957029 shutdown_connections 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f099be0 0x56046f0a8d50 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 --2- 172.21.15.31:0/2859957029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f09a530 0x56046f0a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 -- 172.21.15.31:0/2859957029 >> 172.21.15.31:0/2859957029 conn(0x56046f08f3e0 msgr2=0x56046f095690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.700+0000 7f18eb92db80 1 -- 172.21.15.31:0/2859957029 shutdown_connections 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.701+0000 7f18eb92db80 1 -- 172.21.15.31:0/2859957029 wait complete. 2023-07-29T16:24:36.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.701+0000 7f18eb92db80 1 -- start start 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.701+0000 7f18eb92db80 1 Processor -- start 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f099be0 0x56046f0a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f0a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046f0a8770 con 0x56046f099be0 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046f0a88e0 con 0x56046f09b2f0 2023-07-29T16:24:36.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56046f0a8aa0 con 0x56046f09a530 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18eb92db80 10 client.0 _ll_register_callbacks cb 0x56046ef90620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18d559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f0a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18d659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18d559a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f0a7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55942/0 (socket says 172.21.15.31:55942) 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18d559a700 1 -- 172.21.15.31:0/842480732 learned_addr learned my addr 172.21.15.31:0/842480732 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.702+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f099be0 0x56046f0a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18d559a700 1 -- 172.21.15.31:0/842480732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 msgr2=0x56046f0a8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18d559a700 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18d559a700 1 -- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f099be0 msgr2=0x56046f0a6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18d559a700 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f099be0 0x56046f0a6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18d559a700 1 -- 172.21.15.31:0/842480732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d01a0610 con 0x56046f09a530 2023-07-29T16:24:36.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18d559a700 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f0a7220 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f18c00a24c0 tx=0x7f18c00a0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=617baa60f5f04da1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18b97fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:36.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18b97fa700 1 -- 172.21.15.31:0/842480732 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c009e660 con 0x56046f09a530 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18b97fa700 1 -- 172.21.15.31:0/842480732 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18c00a29f0 con 0x56046f09a530 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18cc011e10 con 0x56046f09a530 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18b97fa700 1 -- 172.21.15.31:0/842480732 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18c00228c0 con 0x56046f09a530 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18eb92db80 10 client.28252 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.703+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f18c8003d90 con 0x56046f09a530 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18b97fa700 1 -- 172.21.15.31:0/842480732 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18c00ca030 con 0x56046f09a530 2023-07-29T16:24:36.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18b97fa700 1 -- 172.21.15.31:0/842480732 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f18c0099b20 con 0x56046f09a530 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18eb92db80 10 client.28252 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18eb92db80 10 client.28252 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18eb92db80 20 client.28252 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18eb92db80 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f101ce0 0x56046f1040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18eb92db80 4 client.28252 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56046efe3d40 con 0x56046f101ce0 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.704+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f101ce0 0x56046f1040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18d5d9b700 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f101ce0 0x56046f1040c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8cc65ba0b23bf9d6 server_cookie=bbd34126e56a9820 in_seq=0 out_seq=0 2023-07-29T16:24:36.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18b97fa700 10 client.28252 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18b97fa700 1 -- 172.21.15.31:0/842480732 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56046efe3d40 con 0x56046f101ce0 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18b97fa700 10 client.28252 handle_command_reply: tid=0 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 client.28252 shutdown 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 msgr2=0x56046f0a7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f0a7220 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f18c00a24c0 tx=0x7f18c00a0040 comp rx=0 tx=0).stop 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 shutdown_connections 2023-07-29T16:24:36.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56046f101ce0 0x56046f1040c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56046f09b2f0 0x56046f0a8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56046f099be0 0x56046f0a6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 --2- 172.21.15.31:0/842480732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56046f09a530 0x56046f0a7220 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18b97fa700 0 client.28252 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.705+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 >> 172.21.15.31:0/842480732 conn(0x56046f08f3e0 msgr2=0x56046f0a16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.706+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 shutdown_connections 2023-07-29T16:24:36.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.706+0000 7f18eb92db80 1 -- 172.21.15.31:0/842480732 wait complete. 2023-07-29T16:24:36.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.706+0000 7f18eb92db80 20 client.28252 trim_cache size 0 max 16384 2023-07-29T16:24:36.713 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18bc0a0810 msgr2=0x7f18bc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18bc0a0810 0x7f18bc0a2cc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x56046f099980 tx=0x7f18c8004b40 comp rx=0 tx=0).stop 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 msgr2=0x7f18d0198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0198840 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f18c000b620 tx=0x7f18c0005740 comp rx=0 tx=0).stop 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 shutdown_connections 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18bc0a0810 0x7f18bc0a2cc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18d0101ea0 0x7f18d019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18d0100320 0x7f18d0198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 --2- 172.21.15.31:0/419189604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101510 0x7f18d0198840 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 >> 172.21.15.31:0/419189604 conn(0x7f18d00fb920 msgr2=0x7f18d0106e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 shutdown_connections 2023-07-29T16:24:36.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.707+0000 7f18d7fff700 1 -- 172.21.15.31:0/419189604 wait complete. 2023-07-29T16:24:36.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:36 smithi031 ceph-mon[95504]: pgmap v644: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.6 MiB/s wr, 271 op/s 2023-07-29T16:24:36.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:36 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2503705825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:24:36.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:24:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:36 smithi039 ceph-mon[128858]: pgmap v644: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.6 MiB/s wr, 271 op/s 2023-07-29T16:24:36.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:36 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2503705825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:24:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:36 smithi156 ceph-mon[129245]: pgmap v644: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.6 MiB/s wr, 271 op/s 2023-07-29T16:24:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:36 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2503705825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:24:36.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 634575 8.95 MB/sec execute 261 sec latency 21.193 ms 2023-07-29T16:24:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 -- 172.21.15.31:0/3890229199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 msgr2=0x7fa4f4101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/3890229199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4101960 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc009990 tx=0x7fa4dc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 -- 172.21.15.31:0/3890229199 shutdown_connections 2023-07-29T16:24:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/3890229199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 0x7fa4f4108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/3890229199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4f4100320 0x7fa4f4100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/3890229199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4101960 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.949+0000 7fa4f8fff700 1 -- 172.21.15.31:0/3890229199 >> 172.21.15.31:0/3890229199 conn(0x7fa4f40fb920 msgr2=0x7fa4f40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.950+0000 7fa4f8fff700 1 -- 172.21.15.31:0/3890229199 shutdown_connections 2023-07-29T16:24:36.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.950+0000 7fa4f8fff700 1 -- 172.21.15.31:0/3890229199 wait complete. 2023-07-29T16:24:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 Processor -- start 2023-07-29T16:24:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 -- start start 2023-07-29T16:24:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4f4100320 0x7fa4f4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 0x7fa4f4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4073e20 con 0x7fa4f4101ea0 2023-07-29T16:24:36.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4073f90 con 0x7fa4f4101510 2023-07-29T16:24:36.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f8fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f41a0140 con 0x7fa4f4100320 2023-07-29T16:24:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4f4100320 0x7fa4f4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48632/0 (socket says 172.21.15.31:48632) 2023-07-29T16:24:36.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.951+0000 7fa4ebfff700 1 -- 172.21.15.31:0/4271003962 learned_addr learned my addr 172.21.15.31:0/4271003962 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 0x7fa4f4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4ebfff700 1 -- 172.21.15.31:0/4271003962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4f4100320 msgr2=0x7fa4f4074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4f4100320 0x7fa4f4074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4ebfff700 1 -- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 msgr2=0x7fa4f4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 0x7fa4f4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4ebfff700 1 -- 172.21.15.31:0/4271003962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc0095f0 con 0x7fa4f4101510 2023-07-29T16:24:36.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 0x7fa4f4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:36.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4073140 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00b620 tx=0x7fa4dc005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4eb7fe700 1 -- 172.21.15.31:0/4271003962 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc01d070 con 0x7fa4f4101510 2023-07-29T16:24:36.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.952+0000 7fa4eb7fe700 1 -- 172.21.15.31:0/4271003962 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4dc022470 con 0x7fa4f4101510 2023-07-29T16:24:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.953+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f41a03c0 con 0x7fa4f4101510 2023-07-29T16:24:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.953+0000 7fa4eb7fe700 1 -- 172.21.15.31:0/4271003962 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc017650 con 0x7fa4f4101510 2023-07-29T16:24:36.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.953+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f41a0840 con 0x7fa4f4101510 2023-07-29T16:24:36.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.956+0000 7fa4eb7fe700 1 -- 172.21.15.31:0/4271003962 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa4dc0225e0 con 0x7fa4f4101510 2023-07-29T16:24:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.957+0000 7fa50c903b80 1 Processor -- start 2023-07-29T16:24:36.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.957+0000 7fa4eb7fe700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4e00a0810 0x7fa4e00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.957+0000 7fa50c903b80 1 -- start start 2023-07-29T16:24:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.957+0000 7fa4eb7fe700 1 -- 172.21.15.31:0/4271003962 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa4dc0ccfb0 con 0x7fa4f4101510 2023-07-29T16:24:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be3be0 0x55dd31be3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 0x55dd31be4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31be5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa50c903b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31b3ba10 con 0x55dd31be52f0 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa50c903b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31b393d0 con 0x55dd31be4530 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa50c903b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31b34cc0 con 0x55dd31be3be0 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4e00a0810 0x7fa4e00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be3be0 0x55dd31be3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 0x55dd31be4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 0x55dd31be4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48642/0 (socket says 172.21.15.31:48642) 2023-07-29T16:24:36.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be3be0 0x55dd31be3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55972/0 (socket says 172.21.15.31:55972) 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 -- 172.21.15.31:0/3465243155 learned_addr learned my addr 172.21.15.31:0/3465243155 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31be5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 -- 172.21.15.31:0/3465243155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be3be0 msgr2=0x55dd31be3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be3be0 0x55dd31be3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 -- 172.21.15.31:0/3465243155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 msgr2=0x55dd31be5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31be5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4ebfff700 1 -- 172.21.15.31:0/3465243155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd31962de0 con 0x55dd31be4530 2023-07-29T16:24:36.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.958+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31be5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 0x55dd31be4d60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc096be0 tx=0x7fa4dc009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d849d6f64fc81c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4e00a0810 0x7fa4e00a2cc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4075160 tx=0x7fa4ec009890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa4e8ff9700 1 -- 172.21.15.31:0/3465243155 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc09e710 con 0x55dd31be4530 2023-07-29T16:24:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa4e8ff9700 1 -- 172.21.15.31:0/3465243155 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4dc0a2aa0 con 0x55dd31be4530 2023-07-29T16:24:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa4e8ff9700 1 -- 172.21.15.31:0/3465243155 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc0a2610 con 0x55dd31be4530 2023-07-29T16:24:36.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa50c903b80 1 -- 172.21.15.31:0/3465243155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 msgr2=0x55dd31be4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.959+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 0x55dd31be4d60 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc096be0 tx=0x7fa4dc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 -- 172.21.15.31:0/3465243155 shutdown_connections 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31be5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be3be0 0x55dd31be3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3465243155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be4530 0x55dd31be4d60 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 -- 172.21.15.31:0/3465243155 >> 172.21.15.31:0/3465243155 conn(0x55dd31bd93e0 msgr2=0x55dd31bde990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 -- 172.21.15.31:0/3465243155 shutdown_connections 2023-07-29T16:24:36.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 -- 172.21.15.31:0/3465243155 wait complete. 2023-07-29T16:24:36.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.960+0000 7fa50c903b80 1 -- start start 2023-07-29T16:24:36.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 Processor -- start 2023-07-29T16:24:36.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be3be0 0x55dd31bed490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be52f0 0x55dd31bf4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31ad1510 con 0x55dd31be3be0 2023-07-29T16:24:36.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31ad11b0 con 0x55dd31be52f0 2023-07-29T16:24:36.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd319b8920 con 0x55dd31be4530 2023-07-29T16:24:36.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa50c903b80 10 client.0 _ll_register_callbacks cb 0x55dd31ada620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:36.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa4f2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be52f0 0x55dd31bf4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be3be0 0x55dd31bed490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bed9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55978/0 (socket says 172.21.15.31:55978) 2023-07-29T16:24:36.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.961+0000 7fa4ebfff700 1 -- 172.21.15.31:0/2901449050 learned_addr learned my addr 172.21.15.31:0/2901449050 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4ebfff700 1 -- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be52f0 msgr2=0x55dd31bf4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be52f0 0x55dd31bf4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4ebfff700 1 -- 172.21.15.31:0/2901449050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be3be0 msgr2=0x55dd31bed490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be3be0 0x55dd31bed490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4ebfff700 1 -- 172.21.15.31:0/2901449050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f41a0840 con 0x55dd31be4530 2023-07-29T16:24:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bed9d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc09a390 tx=0x7fa4dc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58ae0f52b429488e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:36.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/2901449050 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc09aa00 con 0x55dd31be4530 2023-07-29T16:24:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/2901449050 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa4dc0a2610 con 0x55dd31be4530 2023-07-29T16:24:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd31b5fd10 con 0x55dd31be4530 2023-07-29T16:24:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/2901449050 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4dc0a27d0 con 0x55dd31be4530 2023-07-29T16:24:36.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa50c903b80 10 client.28264 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:36.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.962+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dd3194bc60 con 0x55dd31be4530 2023-07-29T16:24:36.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/2901449050 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dd31b5fd10 con 0x55dd31be4530 2023-07-29T16:24:36.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/2901449050 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa4dc0964d0 con 0x55dd31be4530 2023-07-29T16:24:36.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa50c903b80 10 client.28264 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa50c903b80 10 client.28264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa50c903b80 20 client.28264 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31bf8c70 0x55dd31bfb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa50c903b80 4 client.28264 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd31bb2d70 con 0x55dd31bf8c70 2023-07-29T16:24:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.963+0000 7fa4f259c700 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31bf8c70 0x55dd31bfb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.964+0000 7fa4f259c700 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31bf8c70 0x55dd31bfb050 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f533775e36f9aa78 server_cookie=328a083bbe1dcc7c in_seq=0 out_seq=0 2023-07-29T16:24:36.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.964+0000 7fa4d9ffb700 10 client.28264 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.964+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/2901449050 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55dd31bb2d70 con 0x55dd31bf8c70 2023-07-29T16:24:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.964+0000 7fa4d9ffb700 10 client.28264 handle_command_reply: tid=0 2023-07-29T16:24:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 client.28264 shutdown 2023-07-29T16:24:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 msgr2=0x55dd31bed9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:36.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bed9d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc09a390 tx=0x7fa4dc005fd0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 shutdown_connections 2023-07-29T16:24:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31bf8c70 0x55dd31bfb050 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be52f0 0x55dd31bf4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be3be0 0x55dd31bed490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2901449050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bed9d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 >> 172.21.15.31:0/2901449050 conn(0x55dd31bd93e0 msgr2=0x55dd31bde060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa4d9ffb700 0 client.28264 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 shutdown_connections 2023-07-29T16:24:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 1 -- 172.21.15.31:0/2901449050 wait complete. 2023-07-29T16:24:36.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.965+0000 7fa50c903b80 20 client.28264 trim_cache size 0 max 16384 2023-07-29T16:24:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.979+0000 7fa50c903b80 1 Processor -- start 2023-07-29T16:24:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.979+0000 7fa50c903b80 1 -- start start 2023-07-29T16:24:36.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bde060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bfac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa50c903b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31bd67e0 con 0x55dd31be52f0 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa50c903b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31bec940 con 0x55dd31be3be0 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa50c903b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31bed000 con 0x55dd31be4530 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bde060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48656/0 (socket says 172.21.15.31:48656) 2023-07-29T16:24:36.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 -- 172.21.15.31:0/2127956814 learned_addr learned my addr 172.21.15.31:0/2127956814 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:36.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bfac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:36.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 -- 172.21.15.31:0/2127956814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 msgr2=0x55dd31bde060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bde060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 -- 172.21.15.31:0/2127956814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 msgr2=0x55dd31bfac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bfac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f259c700 1 -- 172.21.15.31:0/2127956814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd31989250 con 0x55dd31be3be0 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bde060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.980+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bfac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.981+0000 7fa4f259c700 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf29c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4101370 tx=0x7fa4e400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e32c4ea1570ed8b2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.981+0000 7fa4e97fa700 1 -- 172.21.15.31:0/2127956814 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e401c040 con 0x55dd31be3be0 2023-07-29T16:24:36.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.981+0000 7fa4e97fa700 1 -- 172.21.15.31:0/2127956814 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4e400f040 con 0x55dd31be3be0 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.981+0000 7fa4e97fa700 1 -- 172.21.15.31:0/2127956814 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400e040 con 0x55dd31be3be0 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 -- 172.21.15.31:0/2127956814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 msgr2=0x55dd31bf29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf29c0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4101370 tx=0x7fa4e400b040 comp rx=0 tx=0).stop 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 -- 172.21.15.31:0/2127956814 shutdown_connections 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bfac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf29c0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 --2- 172.21.15.31:0/2127956814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bde060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 -- 172.21.15.31:0/2127956814 >> 172.21.15.31:0/2127956814 conn(0x55dd31bd93e0 msgr2=0x55dd31bdf690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 -- 172.21.15.31:0/2127956814 shutdown_connections 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 -- 172.21.15.31:0/2127956814 wait complete. 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 -- start start 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.982+0000 7fa50c903b80 1 Processor -- start 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:37.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bf0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:37.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bf1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:37.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31bf1850 con 0x55dd31be52f0 2023-07-29T16:24:37.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31bf19c0 con 0x55dd31be3be0 2023-07-29T16:24:37.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd31bf1b80 con 0x55dd31be4530 2023-07-29T16:24:37.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa50c903b80 10 client.0 _ll_register_callbacks cb 0x55dd31ada620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:37.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48662/0 (socket says 172.21.15.31:48662) 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 -- 172.21.15.31:0/3228812907 learned_addr learned my addr 172.21.15.31:0/3228812907 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bf1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bf0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 -- 172.21.15.31:0/3228812907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 msgr2=0x55dd31bf0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bf0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 -- 172.21.15.31:0/3228812907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 msgr2=0x55dd31bf1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:37.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bf1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f259c700 1 -- 172.21.15.31:0/3228812907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd3194bc60 con 0x55dd31be3be0 2023-07-29T16:24:37.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.983+0000 7fa4f2d9d700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bf1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:37.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa4f259c700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf25e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009310 tx=0x7fa4e40117c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=353c16a5a2f5f63e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:37.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa4d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:37.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/3228812907 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400e040 con 0x55dd31be3be0 2023-07-29T16:24:37.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4dc0a2aa0 con 0x55dd31be3be0 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/3228812907 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa4e4016810 con 0x55dd31be3be0 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/3228812907 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4e4003e90 con 0x55dd31be3be0 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa50c903b80 10 client.28262 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.984+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fa4e4011900 con 0x55dd31be3be0 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/3228812907 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4dc0a2aa0 con 0x55dd31be3be0 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/3228812907 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa4e402d020 con 0x55dd31be3be0 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa50c903b80 10 client.28262 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:37.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa50c903b80 10 client.28262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa50c903b80 20 client.28262 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31c4bd20 0x55dd31c4e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa50c903b80 4 client.28262 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd31b2dd40 con 0x55dd31c4bd20 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31c4bd20 0x55dd31c4e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa4d9ffb700 10 client.28262 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:37.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.985+0000 7fa4ebfff700 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31c4bd20 0x55dd31c4e130 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc732f76ea99727c server_cookie=7e550376855a6d95 in_seq=0 out_seq=0 2023-07-29T16:24:37.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa4d9ffb700 1 -- 172.21.15.31:0/3228812907 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd31b2dd40 con 0x55dd31c4bd20 2023-07-29T16:24:37.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa4d9ffb700 10 client.28262 handle_command_reply: tid=0 2023-07-29T16:24:37.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 client.28262 shutdown 2023-07-29T16:24:37.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 msgr2=0x55dd31bf25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:37.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf25e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4009310 tx=0x7fa4e40117c0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 shutdown_connections 2023-07-29T16:24:37.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55dd31c4bd20 0x55dd31c4e130 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd31be52f0 0x55dd31bf1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd31be3be0 0x55dd31bf25e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 --2- 172.21.15.31:0/3228812907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd31be4530 0x55dd31bf0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa4d9ffb700 0 client.28262 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:37.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 >> 172.21.15.31:0/3228812907 conn(0x55dd31bd93e0 msgr2=0x55dd31bde060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:37.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 shutdown_connections 2023-07-29T16:24:37.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 1 -- 172.21.15.31:0/3228812907 wait complete. 2023-07-29T16:24:37.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.986+0000 7fa50c903b80 20 client.28262 trim_cache size 0 max 16384 2023-07-29T16:24:37.010 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:24:37.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4e00a0810 msgr2=0x7fa4e00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:37.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4e00a0810 0x7fa4e00a2cc0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4075160 tx=0x7fa4ec009890 comp rx=0 tx=0).stop 2023-07-29T16:24:37.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 msgr2=0x7fa4f4073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:37.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4073140 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00b620 tx=0x7fa4dc005740 comp rx=0 tx=0).stop 2023-07-29T16:24:37.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 shutdown_connections 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa4e00a0810 0x7fa4e00a2cc0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa4f4101ea0 0x7fa4f4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa4f4100320 0x7fa4f4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 --2- 172.21.15.31:0/4271003962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4f4101510 0x7fa4f4073140 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 >> 172.21.15.31:0/4271003962 conn(0x7fa4f40fb920 msgr2=0x7fa4f40fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 shutdown_connections 2023-07-29T16:24:37.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:36.987+0000 7fa4f8fff700 1 -- 172.21.15.31:0/4271003962 wait complete. 2023-07-29T16:24:37.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 636095 8.96 MB/sec execute 262 sec latency 19.657 ms 2023-07-29T16:24:38.014 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:24:38.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:24:38.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 -- 172.21.15.31:0/3247283343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101520 msgr2=0x7f24b4101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 --2- 172.21.15.31:0/3247283343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101520 0x7f24b4101970 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009990 tx=0x7f24a8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 -- 172.21.15.31:0/3247283343 shutdown_connections 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 --2- 172.21.15.31:0/3247283343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4101eb0 0x7f24b4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 --2- 172.21.15.31:0/3247283343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 --2- 172.21.15.31:0/3247283343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101520 0x7f24b4101970 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.234+0000 7f24b99ff700 1 -- 172.21.15.31:0/3247283343 >> 172.21.15.31:0/3247283343 conn(0x7f24b40fb970 msgr2=0x7f24b40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.235+0000 7f24b99ff700 1 -- 172.21.15.31:0/3247283343 shutdown_connections 2023-07-29T16:24:38.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.235+0000 7f24b99ff700 1 -- 172.21.15.31:0/3247283343 wait complete. 2023-07-29T16:24:38.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 Processor -- start 2023-07-29T16:24:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 -- start start 2023-07-29T16:24:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4101520 0x7f24b4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101eb0 0x7f24b4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4073e20 con 0x7f24b4101520 2023-07-29T16:24:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4073f90 con 0x7f24b4100330 2023-07-29T16:24:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b99ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b41a00c0 con 0x7f24b4101eb0 2023-07-29T16:24:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48678/0 (socket says 172.21.15.31:48678) 2023-07-29T16:24:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b2ffd700 1 -- 172.21.15.31:0/766313094 learned_addr learned my addr 172.21.15.31:0/766313094 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b37fe700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101eb0 0x7f24b4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.236+0000 7f24b27fc700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4101520 0x7f24b4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b2ffd700 1 -- 172.21.15.31:0/766313094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101eb0 msgr2=0x7f24b4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101eb0 0x7f24b4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b2ffd700 1 -- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4101520 msgr2=0x7f24b4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4101520 0x7f24b4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b2ffd700 1 -- 172.21.15.31:0/766313094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4009520 con 0x7f24b4100330 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b37fe700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101eb0 0x7f24b4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:38.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4074b00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f24a400ea50 tx=0x7f24a400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b89fd700 1 -- 172.21.15.31:0/766313094 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a400cbd0 con 0x7f24b4100330 2023-07-29T16:24:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b89fd700 1 -- 172.21.15.31:0/766313094 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a400cd30 con 0x7f24b4100330 2023-07-29T16:24:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.237+0000 7f24b89fd700 1 -- 172.21.15.31:0/766313094 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4018680 con 0x7f24b4100330 2023-07-29T16:24:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.238+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a80095f0 con 0x7f24b4100330 2023-07-29T16:24:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.238+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b41a06d0 con 0x7f24b4100330 2023-07-29T16:24:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.241+0000 7f24b89fd700 1 -- 172.21.15.31:0/766313094 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f24a4010430 con 0x7f24b4100330 2023-07-29T16:24:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.243+0000 7f24cd254b80 1 Processor -- start 2023-07-29T16:24:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.243+0000 7f24cd254b80 1 -- start start 2023-07-29T16:24:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f188070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f188de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f1897c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24cd254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f0dfa90 con 0x55de1f1885b0 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24cd254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f0dd450 con 0x55de1f189370 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24cd254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f0d8d40 con 0x55de1f187c60 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b89fd700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0810 0x7f249c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b89fd700 1 -- 172.21.15.31:0/766313094 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f24a4014070 con 0x7f24b4100330 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f188de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f188070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f188de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48822/0 (socket says 172.21.15.31:48822) 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b2ffd700 1 -- 172.21.15.31:0/2349204492 learned_addr learned my addr 172.21.15.31:0/2349204492 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b27fc700 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f1897c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.244+0000 7f24b37fe700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0810 0x7f249c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b2ffd700 1 -- 172.21.15.31:0/2349204492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 msgr2=0x55de1f188070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f188070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b2ffd700 1 -- 172.21.15.31:0/2349204492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 msgr2=0x55de1f1897c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f1897c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b2ffd700 1 -- 172.21.15.31:0/2349204492 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de1ef06e60 con 0x55de1f1885b0 2023-07-29T16:24:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f188070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f188de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f24a400c5b0 tx=0x7f24a4010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eba1a329d351382 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f24b37fe700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0810 0x7f249c0a2cc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009550 tx=0x7f24ac00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f2499ffb700 1 -- 172.21.15.31:0/2349204492 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a409d700 con 0x55de1f1885b0 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f2499ffb700 1 -- 172.21.15.31:0/2349204492 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a40a1b20 con 0x55de1f1885b0 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.245+0000 7f2499ffb700 1 -- 172.21.15.31:0/2349204492 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4098e20 con 0x55de1f1885b0 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 -- 172.21.15.31:0/2349204492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 msgr2=0x55de1f188de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f188de0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f24a400c5b0 tx=0x7f24a4010c80 comp rx=0 tx=0).stop 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 -- 172.21.15.31:0/2349204492 shutdown_connections 2023-07-29T16:24:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f1897c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f188070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 --2- 172.21.15.31:0/2349204492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f188de0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.246+0000 7f24cd254b80 1 -- 172.21.15.31:0/2349204492 >> 172.21.15.31:0/2349204492 conn(0x55de1f17d460 msgr2=0x55de1f182a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.247+0000 7f24cd254b80 1 -- 172.21.15.31:0/2349204492 shutdown_connections 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.247+0000 7f24cd254b80 1 -- 172.21.15.31:0/2349204492 wait complete. 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.247+0000 7f24cd254b80 1 -- start start 2023-07-29T16:24:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.247+0000 7f24cd254b80 1 Processor -- start 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f191840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f191d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f075590 con 0x55de1f1885b0 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f075230 con 0x55de1f189370 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1ef5c9a0 con 0x55de1f187c60 2023-07-29T16:24:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24cd254b80 10 client.0 _ll_register_callbacks cb 0x55de1f07e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f191840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b27fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f191d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f191840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56020/0 (socket says 172.21.15.31:56020) 2023-07-29T16:24:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1847602730 learned_addr learned my addr 172.21.15.31:0/1847602730 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 msgr2=0x55de1f198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.248+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1847602730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 msgr2=0x55de1f191d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f191d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1847602730 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b41a06d0 con 0x55de1f187c60 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f191d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:38.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f191840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24a40b5aa0 tx=0x7f24a40cc070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b163be735522d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f2492ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f2492ffd700 1 -- 172.21.15.31:0/1847602730 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a409d700 con 0x55de1f187c60 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f2492ffd700 1 -- 172.21.15.31:0/1847602730 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f24a4098dd0 con 0x55de1f187c60 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de1f103d90 con 0x55de1f187c60 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f2492ffd700 1 -- 172.21.15.31:0/1847602730 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24a40a1690 con 0x55de1f187c60 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24cd254b80 10 client.28282 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:38.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.249+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de1eeefce0 con 0x55de1f187c60 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f2492ffd700 1 -- 172.21.15.31:0/1847602730 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55de1f103d90 con 0x55de1f187c60 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f2492ffd700 1 -- 172.21.15.31:0/1847602730 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f24a40106e0 con 0x55de1f187c60 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24cd254b80 10 client.28282 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24cd254b80 10 client.28282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24cd254b80 20 client.28282 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f19d200 0x55de1f19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24cd254b80 4 client.28282 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:38.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de1f156df0 con 0x55de1f19d200 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.250+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f19d200 0x55de1f19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.251+0000 7f2492ffd700 10 client.28282 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.251+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f19d200 0x55de1f19f5e0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=718d118202be524c server_cookie=7e1e7df00f4442a2 in_seq=0 out_seq=0 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.252+0000 7f2492ffd700 1 -- 172.21.15.31:0/1847602730 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55de1f156df0 con 0x55de1f19d200 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.252+0000 7f2492ffd700 10 client.28282 handle_command_reply: tid=0 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 client.28282 shutdown 2023-07-29T16:24:38.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 msgr2=0x55de1f191840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f191840 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24a40b5aa0 tx=0x7f24a40cc070 comp rx=0 tx=0).stop 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 shutdown_connections 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f19d200 0x55de1f19f5e0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f189370 0x55de1f198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f187c60 0x55de1f191840 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f2492ffd700 0 client.28282 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1847602730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f1885b0 0x55de1f191d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 >> 172.21.15.31:0/1847602730 conn(0x55de1f17d460 msgr2=0x55de1f1820e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 shutdown_connections 2023-07-29T16:24:38.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 1 -- 172.21.15.31:0/1847602730 wait complete. 2023-07-29T16:24:38.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.253+0000 7f24cd254b80 20 client.28282 trim_cache size 0 max 16384 2023-07-29T16:24:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.271+0000 7f24cd254b80 1 Processor -- start 2023-07-29T16:24:38.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.271+0000 7f24cd254b80 1 -- start start 2023-07-29T16:24:38.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f196d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f18f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f199650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24cd254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f19ce70 con 0x55de1f189370 2023-07-29T16:24:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24cd254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f17f4e0 con 0x55de1f187c60 2023-07-29T16:24:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24cd254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f182c10 con 0x55de1f1885b0 2023-07-29T16:24:38.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48708/0 (socket says 172.21.15.31:48708) 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1767930990 learned_addr learned my addr 172.21.15.31:0/1767930990 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f18f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1767930990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 msgr2=0x55de1f18f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f18f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1767930990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 msgr2=0x55de1f199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1767930990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de1ef2d2d0 con 0x55de1f187c60 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.272+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f196d70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f24a4005950 tx=0x7f24a4010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ecf060b57f00314 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f249a7fc700 1 -- 172.21.15.31:0/1767930990 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4099b80 con 0x55de1f187c60 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f249a7fc700 1 -- 172.21.15.31:0/1767930990 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a409d700 con 0x55de1f187c60 2023-07-29T16:24:38.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f249a7fc700 1 -- 172.21.15.31:0/1767930990 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a40a18d0 con 0x55de1f187c60 2023-07-29T16:24:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:37 smithi039 ceph-mon[128858]: pgmap v645: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2023-07-29T16:24:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 -- 172.21.15.31:0/1767930990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 msgr2=0x55de1f196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f196d70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f24a4005950 tx=0x7f24a4010c80 comp rx=0 tx=0).stop 2023-07-29T16:24:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 -- 172.21.15.31:0/1767930990 shutdown_connections 2023-07-29T16:24:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f196d70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1767930990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f18f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.273+0000 7f24cd254b80 1 -- 172.21.15.31:0/1767930990 >> 172.21.15.31:0/1767930990 conn(0x55de1f17d460 msgr2=0x55de1f1904d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.274+0000 7f24cd254b80 1 -- 172.21.15.31:0/1767930990 shutdown_connections 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.274+0000 7f24cd254b80 1 -- 172.21.15.31:0/1767930990 wait complete. 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.274+0000 7f24cd254b80 1 -- start start 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.274+0000 7f24cd254b80 1 Processor -- start 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f196100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f1eed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f1967e0 con 0x55de1f189370 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f1ef2b0 con 0x55de1f187c60 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de1f1ef3f0 con 0x55de1f1885b0 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24cd254b80 10 client.0 _ll_register_callbacks cb 0x55de1f07e6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b27fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48722/0 (socket says 172.21.15.31:48722) 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1635267675 learned_addr learned my addr 172.21.15.31:0/1635267675 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f196100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56028/0 (socket says 172.21.15.31:56028) 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f1eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1635267675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 msgr2=0x55de1f196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1635267675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 msgr2=0x55de1f1eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f1eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b2ffd700 1 -- 172.21.15.31:0/1635267675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de1eeefce0 con 0x55de1f187c60 2023-07-29T16:24:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.275+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f1eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:24:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24b2ffd700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f195bc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f24a400f010 tx=0x7f24a40109f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0d8f37d6353017e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f2492ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:38.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f2492ffd700 1 -- 172.21.15.31:0/1635267675 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4083030 con 0x55de1f187c60 2023-07-29T16:24:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f2492ffd700 1 -- 172.21.15.31:0/1635267675 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f24a4099b80 con 0x55de1f187c60 2023-07-29T16:24:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24a40a1b20 con 0x55de1f187c60 2023-07-29T16:24:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f2492ffd700 1 -- 172.21.15.31:0/1635267675 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24a40a18a0 con 0x55de1f187c60 2023-07-29T16:24:38.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 10 client.28280 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de1ef05330 con 0x55de1f187c60 2023-07-29T16:24:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f2492ffd700 1 -- 172.21.15.31:0/1635267675 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a40a1b20 con 0x55de1f187c60 2023-07-29T16:24:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f2492ffd700 1 -- 172.21.15.31:0/1635267675 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f24a4098630 con 0x55de1f187c60 2023-07-29T16:24:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 10 client.28280 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:38.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 10 client.28280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 20 client.28280 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f1f08e0 0x55de1f1f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 4 client.28280 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:24:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.276+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55de1f0d1dc0 con 0x55de1f1f08e0 2023-07-29T16:24:38.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.277+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f1f08e0 0x55de1f1f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.277+0000 7f24b27fc700 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f1f08e0 0x55de1f1f2cc0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f0aecaaf04648b3 server_cookie=ec78df51b53b06c1 in_seq=0 out_seq=0 2023-07-29T16:24:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.277+0000 7f2492ffd700 10 client.28280 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.278+0000 7f2492ffd700 1 -- 172.21.15.31:0/1635267675 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55de1f0d1dc0 con 0x55de1f1f08e0 2023-07-29T16:24:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.278+0000 7f2492ffd700 10 client.28280 handle_command_reply: tid=0 2023-07-29T16:24:38.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.278+0000 7f24cd254b80 1 client.28280 shutdown 2023-07-29T16:24:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 msgr2=0x55de1f195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f195bc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f24a400f010 tx=0x7f24a40109f0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 shutdown_connections 2023-07-29T16:24:38.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55de1f1f08e0 0x55de1f1f2cc0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de1f189370 0x55de1f1eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55de1f187c60 0x55de1f195bc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f2492ffd700 0 client.28280 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 --2- 172.21.15.31:0/1635267675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55de1f1885b0 0x55de1f196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 >> 172.21.15.31:0/1635267675 conn(0x55de1f17d460 msgr2=0x55de1f18f780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 shutdown_connections 2023-07-29T16:24:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 1 -- 172.21.15.31:0/1635267675 wait complete. 2023-07-29T16:24:38.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.279+0000 7f24cd254b80 20 client.28280 trim_cache size 0 max 16384 2023-07-29T16:24:38.290 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:24:38.290 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:24:38.290 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "979efd71-3c79-4ec0-8c64-031fd6e0eaf5", 2023-07-29T16:24:38.290 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:24:38.290 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:24:38.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0810 msgr2=0x7f249c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0810 0x7f249c0a2cc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009550 tx=0x7f24ac00b040 comp rx=0 tx=0).stop 2023-07-29T16:24:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 msgr2=0x7f24b4074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4074b00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f24a400ea50 tx=0x7f24a400ed60 comp rx=0 tx=0).stop 2023-07-29T16:24:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 shutdown_connections 2023-07-29T16:24:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0810 0x7f249c0a2cc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4101eb0 0x7f24b4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100330 0x7f24b4074b00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 --2- 172.21.15.31:0/766313094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4101520 0x7f24b4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 >> 172.21.15.31:0/766313094 conn(0x7f24b40fb970 msgr2=0x7f24b40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 shutdown_connections 2023-07-29T16:24:38.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.280+0000 7f24b99ff700 1 -- 172.21.15.31:0/766313094 wait complete. 2023-07-29T16:24:38.297 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:37 smithi031 ceph-mon[95504]: pgmap v645: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2023-07-29T16:24:38.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:24:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:37 smithi156 ceph-mon[129245]: pgmap v645: 97 pgs: 97 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.3 MiB/s wr, 244 op/s 2023-07-29T16:24:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/116770859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4100330 msgr2=0x7ffbc4100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/116770859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4100330 0x7ffbc4100740 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009990 tx=0x7ffbc0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/116770859 shutdown_connections 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/116770859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 0x7ffbc4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/116770859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4100330 0x7ffbc4100740 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/116770859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbc4101520 0x7ffbc4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/116770859 >> 172.21.15.31:0/116770859 conn(0x7ffbc40fb970 msgr2=0x7ffbc40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.522+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/116770859 shutdown_connections 2023-07-29T16:24:38.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.523+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/116770859 wait complete. 2023-07-29T16:24:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 Processor -- start 2023-07-29T16:24:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 -- start start 2023-07-29T16:24:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbc4100330 0x7ffbc4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4101520 0x7ffbc4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 0x7ffbc4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4073e20 con 0x7ffbc4101520 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc4073f90 con 0x7ffbc4101eb0 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbcc3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc41a0100 con 0x7ffbc4100330 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4101520 0x7ffbc4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4101520 0x7ffbc4073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48870/0 (socket says 172.21.15.31:48870) 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbc999a700 1 -- 172.21.15.31:0/2204338611 learned_addr learned my addr 172.21.15.31:0/2204338611 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.524+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 0x7ffbc4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca19b700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbc4100330 0x7ffbc4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca99c700 1 -- 172.21.15.31:0/2204338611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbc4100330 msgr2=0x7ffbc4074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbc4100330 0x7ffbc4074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca99c700 1 -- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4101520 msgr2=0x7ffbc4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4101520 0x7ffbc4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca99c700 1 -- 172.21.15.31:0/2204338611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc00095f0 con 0x7ffbc4101eb0 2023-07-29T16:24:38.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.525+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 0x7ffbc4073680 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00efc0 tx=0x7ffbbc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.526+0000 7ffbbb7fe700 1 -- 172.21.15.31:0/2204338611 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc018070 con 0x7ffbc4101eb0 2023-07-29T16:24:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.526+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc41a0380 con 0x7ffbc4101eb0 2023-07-29T16:24:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.526+0000 7ffbbb7fe700 1 -- 172.21.15.31:0/2204338611 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbbc00f040 con 0x7ffbc4101eb0 2023-07-29T16:24:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.526+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc41a08e0 con 0x7ffbc4101eb0 2023-07-29T16:24:38.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.526+0000 7ffbbb7fe700 1 -- 172.21.15.31:0/2204338611 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc00cc70 con 0x7ffbc4101eb0 2023-07-29T16:24:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.529+0000 7ffbbb7fe700 1 -- 172.21.15.31:0/2204338611 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ffbbc003a00 con 0x7ffbc4101eb0 2023-07-29T16:24:38.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.530+0000 7ffbdfd0fb80 1 Processor -- start 2023-07-29T16:24:38.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.530+0000 7ffbdfd0fb80 1 -- start start 2023-07-29T16:24:38.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b29be0 0x556d16b29ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b2ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbbb7fe700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffbb00a0810 0x7ffbb00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b2b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16a81a10 con 0x556d16b2a530 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16a7f3d0 con 0x556d16b2b2f0 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16a7acc0 con 0x556d16b29be0 2023-07-29T16:24:38.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbbb7fe700 1 -- 172.21.15.31:0/2204338611 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ffbbc0cb4e0 con 0x7ffbc4101eb0 2023-07-29T16:24:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b29be0 0x556d16b29ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b29be0 0x556d16b29ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56046/0 (socket says 172.21.15.31:56046) 2023-07-29T16:24:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbc999a700 1 -- 172.21.15.31:0/1731793170 learned_addr learned my addr 172.21.15.31:0/1731793170 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b2b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbca19b700 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b2ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbc999a700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffbb00a0810 0x7ffbb00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbca99c700 1 -- 172.21.15.31:0/1731793170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b29be0 msgr2=0x556d16b29ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.531+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b29be0 0x556d16b29ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbca99c700 1 -- 172.21.15.31:0/1731793170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 msgr2=0x556d16b2ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b2ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbca99c700 1 -- 172.21.15.31:0/1731793170 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d168a8de0 con 0x556d16b2b2f0 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b2b740 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc098ba0 tx=0x7ffbbc098640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d8f980a26746eb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbb8bfe700 1 -- 172.21.15.31:0/1731793170 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc09cea0 con 0x556d16b2b2f0 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbc999a700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffbb00a0810 0x7ffbb00a2cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x556d16b29980 tx=0x7ffbb4007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbb8bfe700 1 -- 172.21.15.31:0/1731793170 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbbc0a0fd0 con 0x556d16b2b2f0 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.532+0000 7ffbb8bfe700 1 -- 172.21.15.31:0/1731793170 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc098e70 con 0x556d16b2b2f0 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1731793170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 msgr2=0x556d16b2b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b2b740 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc098ba0 tx=0x7ffbbc098640 comp rx=0 tx=0).stop 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1731793170 shutdown_connections 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b2b740 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b29be0 0x556d16b29ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1731793170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b2ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1731793170 >> 172.21.15.31:0/1731793170 conn(0x556d16b1f3e0 msgr2=0x556d16b24990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1731793170 shutdown_connections 2023-07-29T16:24:38.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.533+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1731793170 wait complete. 2023-07-29T16:24:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 -- start start 2023-07-29T16:24:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 Processor -- start 2023-07-29T16:24:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b33830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b33d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16a17510 con 0x556d16b29be0 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16a171b0 con 0x556d16b2b2f0 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d168fe920 con 0x556d16b2a530 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.534+0000 7ffbdfd0fb80 10 client.0 _ll_register_callbacks cb 0x556d16a20620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b33d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b33d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56058/0 (socket says 172.21.15.31:56058) 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b33830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbc999a700 1 -- 172.21.15.31:0/1618257157 learned_addr learned my addr 172.21.15.31:0/1618257157 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 -- 172.21.15.31:0/1618257157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 msgr2=0x556d16b33d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b33d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 -- 172.21.15.31:0/1618257157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 msgr2=0x556d16b33830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b33830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 -- 172.21.15.31:0/1618257157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc41a08e0 con 0x556d16b2b2f0 2023-07-29T16:24:38.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbca99c700 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3adb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0a0aa0 tx=0x7ffbbc009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481ce1b39fd968f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbadffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbadffb700 1 -- 172.21.15.31:0/1618257157 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc096d50 con 0x556d16b2b2f0 2023-07-29T16:24:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbadffb700 1 -- 172.21.15.31:0/1618257157 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ffbbc0b1230 con 0x556d16b2b2f0 2023-07-29T16:24:38.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.535+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d16aa5d10 con 0x556d16b2b2f0 2023-07-29T16:24:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbadffb700 1 -- 172.21.15.31:0/1618257157 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbbc0a0b40 con 0x556d16b2b2f0 2023-07-29T16:24:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 10 client.28292 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556d16891c60 con 0x556d16b2b2f0 2023-07-29T16:24:38.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbadffb700 1 -- 172.21.15.31:0/1618257157 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d16aa5d10 con 0x556d16b2b2f0 2023-07-29T16:24:38.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbadffb700 1 -- 172.21.15.31:0/1618257157 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ffbbc012340 con 0x556d16b2b2f0 2023-07-29T16:24:38.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 10 client.28292 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:38.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 10 client.28292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 20 client.28292 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b3f000 0x556d16b413e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 4 client.28292 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d16af8d70 con 0x556d16b3f000 2023-07-29T16:24:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.536+0000 7ffbca19b700 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b3f000 0x556d16b413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.537+0000 7ffbca19b700 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b3f000 0x556d16b413e0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0770f682f501262 server_cookie=2262c062f8feec30 in_seq=0 out_seq=0 2023-07-29T16:24:38.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.537+0000 7ffbadffb700 10 client.28292 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbadffb700 1 -- 172.21.15.31:0/1618257157 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556d16af8d70 con 0x556d16b3f000 2023-07-29T16:24:38.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbadffb700 10 client.28292 handle_command_reply: tid=0 2023-07-29T16:24:38.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 client.28292 shutdown 2023-07-29T16:24:38.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 msgr2=0x556d16b3adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3adb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0a0aa0 tx=0x7ffbbc009f10 comp rx=0 tx=0).stop 2023-07-29T16:24:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 shutdown_connections 2023-07-29T16:24:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b3f000 0x556d16b413e0 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3adb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b33830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/1618257157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b33d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbadffb700 0 client.28292 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 >> 172.21.15.31:0/1618257157 conn(0x556d16b1f3e0 msgr2=0x556d16b24060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 shutdown_connections 2023-07-29T16:24:38.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.539+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/1618257157 wait complete. 2023-07-29T16:24:38.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.540+0000 7ffbdfd0fb80 20 client.28292 trim_cache size 0 max 16384 2023-07-29T16:24:38.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 Processor -- start 2023-07-29T16:24:38.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 -- start start 2023-07-29T16:24:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b25690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b3ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16b333a0 con 0x556d16b29be0 2023-07-29T16:24:38.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16b21460 con 0x556d16b2b2f0 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.559+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16b1c6e0 con 0x556d16b2a530 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbca19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b25690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b3ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbca99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b3ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56060/0 (socket says 172.21.15.31:56060) 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbca99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48766/0 (socket says 172.21.15.31:48766) 2023-07-29T16:24:38.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 -- 172.21.15.31:0/2047304081 learned_addr learned my addr 172.21.15.31:0/2047304081 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 -- 172.21.15.31:0/2047304081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 msgr2=0x556d16b3b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 -- 172.21.15.31:0/2047304081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 msgr2=0x556d16b25690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b25690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbc999a700 1 -- 172.21.15.31:0/2047304081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d168cf250 con 0x556d16b2a530 2023-07-29T16:24:38.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.560+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:38.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.561+0000 7ffbc999a700 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b3ad70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40107c0 tx=0x7ffbb40078b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30d9f50927652502 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.561+0000 7ffbb93ff700 1 -- 172.21.15.31:0/2047304081 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4004a30 con 0x556d16b2a530 2023-07-29T16:24:38.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.561+0000 7ffbb93ff700 1 -- 172.21.15.31:0/2047304081 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbb40054e0 con 0x556d16b2a530 2023-07-29T16:24:38.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.561+0000 7ffbb93ff700 1 -- 172.21.15.31:0/2047304081 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4021650 con 0x556d16b2a530 2023-07-29T16:24:38.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2047304081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 msgr2=0x556d16b3ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b3ad70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40107c0 tx=0x7ffbb40078b0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2047304081 shutdown_connections 2023-07-29T16:24:38.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b2b2f0 0x556d16b3b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b29be0 0x556d16b25690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2047304081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2a530 0x556d16b3ad70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2047304081 >> 172.21.15.31:0/2047304081 conn(0x556d16b1f3e0 msgr2=0x556d16b324e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2047304081 shutdown_connections 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2047304081 wait complete. 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 -- start start 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.562+0000 7ffbdfd0fb80 1 Processor -- start 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 0x556d16b3a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b3ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2b2f0 0x556d16b371e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16b378f0 con 0x556d16b2a530 2023-07-29T16:24:38.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16b37a60 con 0x556d16b29be0 2023-07-29T16:24:38.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d16b37c20 con 0x556d16b2b2f0 2023-07-29T16:24:38.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbdfd0fb80 10 client.0 _ll_register_callbacks cb 0x556d16a20620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:38.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbca19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 0x556d16b3a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbca19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 0x556d16b3a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48782/0 (socket says 172.21.15.31:48782) 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbc999a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b3ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2b2f0 0x556d16b371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.563+0000 7ffbca19b700 1 -- 172.21.15.31:0/2527023251 learned_addr learned my addr 172.21.15.31:0/2527023251 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca99c700 1 -- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 msgr2=0x556d16b3a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 0x556d16b3a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca99c700 1 -- 172.21.15.31:0/2527023251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 msgr2=0x556d16b3ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b3ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca99c700 1 -- 172.21.15.31:0/2527023251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d16891c60 con 0x556d16b2b2f0 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca19b700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 0x556d16b3a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbca99c700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2b2f0 0x556d16b371e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc095110 tx=0x7ffbbc098b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ea91151e0e76dc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbadffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbadffb700 1 -- 172.21.15.31:0/2527023251 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbbc098570 con 0x556d16b2b2f0 2023-07-29T16:24:38.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbadffb700 1 -- 172.21.15.31:0/2527023251 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ffbbc0a0b40 con 0x556d16b2b2f0 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffbbc0a0fd0 con 0x556d16b2b2f0 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbadffb700 1 -- 172.21.15.31:0/2527023251 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbbc097090 con 0x556d16b2b2f0 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbdfd0fb80 10 client.28306 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.564+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7ffbb4010500 con 0x556d16b2b2f0 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbadffb700 1 -- 172.21.15.31:0/2527023251 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffbbc0a0fd0 con 0x556d16b2b2f0 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbadffb700 1 -- 172.21.15.31:0/2527023251 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ffbbc0999f0 con 0x556d16b2b2f0 2023-07-29T16:24:38.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbdfd0fb80 10 client.28306 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:38.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbdfd0fb80 10 client.28306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:38.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbdfd0fb80 20 client.28306 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:38.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b91d30 0x556d16b94140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:38.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbdfd0fb80 4 client.28306 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:24:38.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556d16a73d40 con 0x556d16b91d30 2023-07-29T16:24:38.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbca19b700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b91d30 0x556d16b94140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:38.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbca19b700 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b91d30 0x556d16b94140 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b82a41965bc9c31a server_cookie=606facaa05a4924e in_seq=0 out_seq=0 2023-07-29T16:24:38.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.565+0000 7ffbadffb700 10 client.28306 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbadffb700 1 -- 172.21.15.31:0/2527023251 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556d16a73d40 con 0x556d16b91d30 2023-07-29T16:24:38.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbadffb700 10 client.28306 handle_command_reply: tid=0 2023-07-29T16:24:38.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 client.28306 shutdown 2023-07-29T16:24:38.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2b2f0 msgr2=0x556d16b371e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2b2f0 0x556d16b371e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc095110 tx=0x7ffbbc098b10 comp rx=0 tx=0).stop 2023-07-29T16:24:38.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 shutdown_connections 2023-07-29T16:24:38.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556d16b91d30 0x556d16b94140 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556d16b2b2f0 0x556d16b371e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556d16b29be0 0x556d16b3a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 --2- 172.21.15.31:0/2527023251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d16b2a530 0x556d16b3ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbadffb700 0 client.28306 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:38.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 >> 172.21.15.31:0/2527023251 conn(0x556d16b1f3e0 msgr2=0x556d16b25690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 shutdown_connections 2023-07-29T16:24:38.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.567+0000 7ffbdfd0fb80 1 -- 172.21.15.31:0/2527023251 wait complete. 2023-07-29T16:24:38.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbdfd0fb80 20 client.28306 trim_cache size 0 max 16384 2023-07-29T16:24:38.576 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:24:38.576 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:24:38.576 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:24:38.577 INFO:teuthology.orchestra.run.smithi031.stdout: "979efd71-3c79-4ec0-8c64-031fd6e0eaf5": { 2023-07-29T16:24:38.577 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:24:38.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "979efd71-3c79-4ec0-8c64-031fd6e0eaf5", 2023-07-29T16:24:38.577 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:24:38.577 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:24:38.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:24:38.578 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:24:38.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffbb00a0810 msgr2=0x7ffbb00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffbb00a0810 0x7ffbb00a2cc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x556d16b29980 tx=0x7ffbb4007480 comp rx=0 tx=0).stop 2023-07-29T16:24:38.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 msgr2=0x7ffbc4073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:38.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 0x7ffbc4073680 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc00efc0 tx=0x7ffbbc00c3a0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 shutdown_connections 2023-07-29T16:24:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ffbb00a0810 0x7ffbb00a2cc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffbc4101eb0 0x7ffbc4073680 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffbc4100330 0x7ffbc4074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 --2- 172.21.15.31:0/2204338611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffbc4101520 0x7ffbc4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 >> 172.21.15.31:0/2204338611 conn(0x7ffbc40fb970 msgr2=0x7ffbc4104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 shutdown_connections 2023-07-29T16:24:38.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:24:38.568+0000 7ffbcc3ff700 1 -- 172.21.15.31:0/2204338611 wait complete. 2023-07-29T16:24:38.587 INFO:tasks.cephfs.filesystem:scrub status for tag:979efd71-3c79-4ec0-8c64-031fd6e0eaf5 - {'path': '/', 'tag': '979efd71-3c79-4ec0-8c64-031fd6e0eaf5', 'options': 'recursive,force'} 2023-07-29T16:24:38.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 637914 8.95 MB/sec execute 263 sec latency 14.560 ms 2023-07-29T16:24:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:24:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:24:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:38 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:24:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:24:39.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:38 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:24:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:24:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:38 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:24:39.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 639583 8.97 MB/sec execute 264 sec latency 14.117 ms 2023-07-29T16:24:40.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:39 smithi039 ceph-mon[128858]: pgmap v646: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2023-07-29T16:24:40.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:39 smithi156 ceph-mon[129245]: pgmap v646: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2023-07-29T16:24:40.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:39 smithi031 ceph-mon[95504]: pgmap v646: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2023-07-29T16:24:40.881 INFO:tasks.workunit.client.0.smithi031.stdout: 1 640924 8.95 MB/sec execute 265 sec latency 50.276 ms 2023-07-29T16:24:41.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 643035 8.97 MB/sec execute 266 sec latency 41.635 ms 2023-07-29T16:24:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 -- 172.21.15.31:0/866375751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc071740 msgr2=0x7f10cc071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 --2- 172.21.15.31:0/866375751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc071740 0x7f10cc071b50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009990 tx=0x7f10bc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 -- 172.21.15.31:0/866375751 shutdown_connections 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 --2- 172.21.15.31:0/866375751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc10c5a0 0x7f10cc10ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 --2- 172.21.15.31:0/866375751 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc071740 0x7f10cc071b50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 --2- 172.21.15.31:0/866375751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc072090 0x7f10cc10c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.186+0000 7f10d17ff700 1 -- 172.21.15.31:0/866375751 >> 172.21.15.31:0/866375751 conn(0x7f10cc06d290 msgr2=0x7f10cc06f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.187+0000 7f10d17ff700 1 -- 172.21.15.31:0/866375751 shutdown_connections 2023-07-29T16:24:42.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.187+0000 7f10d17ff700 1 -- 172.21.15.31:0/866375751 wait complete. 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 Processor -- start 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 -- start start 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc071740 0x7f10cc1a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 0x7f10cc1a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc10c5a0 0x7f10cc1a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc1a6a10 con 0x7f10cc10c5a0 2023-07-29T16:24:42.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc1a6b80 con 0x7f10cc072090 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.188+0000 7f10d17ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc1a6d40 con 0x7f10cc071740 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 0x7f10cc1a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10caffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc071740 0x7f10cc1a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 0x7f10cc1a5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54552/0 (socket says 172.21.15.31:54552) 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10caffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc071740 0x7f10cc1a4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41826/0 (socket says 172.21.15.31:41826) 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2629722536 learned_addr learned my addr 172.21.15.31:0/2629722536 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:42.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc10c5a0 0x7f10cc1a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2629722536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc071740 msgr2=0x7f10cc1a4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc071740 0x7f10cc1a4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc10c5a0 msgr2=0x7f10cc1a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc10c5a0 0x7f10cc1a63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2629722536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc0095f0 con 0x7f10cc072090 2023-07-29T16:24:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.189+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 0x7f10cc1a5530 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f10c000d980 tx=0x7f10c000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.190+0000 7f10b3fff700 1 -- 172.21.15.31:0/2629722536 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0004020 con 0x7f10cc072090 2023-07-29T16:24:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.190+0000 7f10b3fff700 1 -- 172.21.15.31:0/2629722536 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10c0009730 con 0x7f10cc072090 2023-07-29T16:24:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.190+0000 7f10b3fff700 1 -- 172.21.15.31:0/2629722536 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0003a80 con 0x7f10cc072090 2023-07-29T16:24:42.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.190+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc1a7020 con 0x7f10cc072090 2023-07-29T16:24:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.190+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc1ad860 con 0x7f10cc072090 2023-07-29T16:24:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.194+0000 7f10b3fff700 1 -- 172.21.15.31:0/2629722536 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f10c00098a0 con 0x7f10cc072090 2023-07-29T16:24:42.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 Processor -- start 2023-07-29T16:24:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 -- start start 2023-07-29T16:24:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10b3fff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f10b40a0860 0x7f10b40a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10b3fff700 1 -- 172.21.15.31:0/2629722536 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f10c00cb700 con 0x7f10cc072090 2023-07-29T16:24:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 0x55a9241978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a924199160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9240ef9b0 con 0x55a923ff36b0 2023-07-29T16:24:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9240ed370 con 0x55a924198cf0 2023-07-29T16:24:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10e512bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9240e8c60 con 0x55a924197df0 2023-07-29T16:24:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.196+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f10b40a0860 0x7f10b40a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10caffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 0x55a9241978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10caffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 0x55a9241978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53094/0 (socket says 172.21.15.31:53094) 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10caffd700 1 -- 172.21.15.31:0/2035305969 learned_addr learned my addr 172.21.15.31:0/2035305969 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a924199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2035305969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 msgr2=0x55a924199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a924199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2035305969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 msgr2=0x55a9241978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 0x55a9241978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 -- 172.21.15.31:0/2035305969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a923f16d80 con 0x55a924197df0 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f10b40a0860 0x7f10b40a2d10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f10c40070c0 tx=0x7f10c4015970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10caffd700 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 0x55a9241978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.197+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241987b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f10c0000c00 tx=0x7f10c00987e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b937d8ab1e12b97 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10b17fa700 1 -- 172.21.15.31:0/2035305969 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c009ce30 con 0x55a924197df0 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10b17fa700 1 -- 172.21.15.31:0/2035305969 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10c00a1250 con 0x55a924197df0 2023-07-29T16:24:42.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10b17fa700 1 -- 172.21.15.31:0/2035305969 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00a0dc0 con 0x55a924197df0 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 -- 172.21.15.31:0/2035305969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 msgr2=0x55a9241987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241987b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f10c0000c00 tx=0x7f10c00987e0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 -- 172.21.15.31:0/2035305969 shutdown_connections 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a924199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a923ff36b0 0x55a9241978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2035305969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241987b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.198+0000 7f10e512bb80 1 -- 172.21.15.31:0/2035305969 >> 172.21.15.31:0/2035305969 conn(0x55a92418cfd0 msgr2=0x55a924192300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.199+0000 7f10e512bb80 1 -- 172.21.15.31:0/2035305969 shutdown_connections 2023-07-29T16:24:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.199+0000 7f10e512bb80 1 -- 172.21.15.31:0/2035305969 wait complete. 2023-07-29T16:24:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.199+0000 7f10e512bb80 1 -- start start 2023-07-29T16:24:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.199+0000 7f10e512bb80 1 Processor -- start 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 0x55a9241a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241a1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a9241a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9240854b0 con 0x55a924198cf0 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a924085150 con 0x55a923ff36b0 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a923f6c8c0 con 0x55a924197df0 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10e512bb80 10 client.0 _ll_register_callbacks cb 0x55a92408e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:42.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10caffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 0x55a9241a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10ca7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10caffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 0x55a9241a0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54562/0 (socket says 172.21.15.31:54562) 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10caffd700 1 -- 172.21.15.31:0/2062676653 learned_addr learned my addr 172.21.15.31:0/2062676653 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.200+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a9241a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10cb7fe700 1 -- 172.21.15.31:0/2062676653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 msgr2=0x55a9241a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241a1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10cb7fe700 1 -- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 msgr2=0x55a9241a0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 0x55a9241a0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10cb7fe700 1 -- 172.21.15.31:0/2062676653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc117e80 con 0x55a924198cf0 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10caffd700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 0x55a9241a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a9241a84b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f10c4015d70 tx=0x7f10c4004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696f5d0c76cfff82 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10aeff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10aeff5700 1 -- 172.21.15.31:0/2062676653 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4016aa0 con 0x55a924198cf0 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a924113cb0 con 0x55a924198cf0 2023-07-29T16:24:42.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.201+0000 7f10aeff5700 1 -- 172.21.15.31:0/2062676653 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f10c4016c00 con 0x55a924198cf0 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10aeff5700 1 -- 172.21.15.31:0/2062676653 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10c4016dc0 con 0x55a924198cf0 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 10 client.18243 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f10cc1ad860 con 0x55a924198cf0 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10aeff5700 1 -- 172.21.15.31:0/2062676653 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a924113cb0 con 0x55a924198cf0 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10aeff5700 1 -- 172.21.15.31:0/2062676653 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f10c4027020 con 0x55a924198cf0 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 10 client.18243 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 10 client.18243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:42.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 20 client.18243 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241acd20 0x55a9241af130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 4 client.18243 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a924166d10 con 0x55a9241acd20 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.202+0000 7f10caffd700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241acd20 0x55a9241af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.203+0000 7f10caffd700 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241acd20 0x55a9241af130 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e26e8f3c075aa1e server_cookie=2edae79b8fcc38cf in_seq=0 out_seq=0 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.203+0000 7f10aeff5700 10 client.18243 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:42.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10aeff5700 1 -- 172.21.15.31:0/2062676653 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a924166d10 con 0x55a9241acd20 2023-07-29T16:24:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10aeff5700 10 client.18243 handle_command_reply: tid=0 2023-07-29T16:24:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 client.18243 shutdown 2023-07-29T16:24:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 msgr2=0x55a9241a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a9241a84b0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f10c4015d70 tx=0x7f10c4004ce0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 shutdown_connections 2023-07-29T16:24:42.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241acd20 0x55a9241af130 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a9241a84b0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a923ff36b0 0x55a9241a0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2062676653 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a924197df0 0x55a9241a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 >> 172.21.15.31:0/2062676653 conn(0x55a92418cfd0 msgr2=0x55a924191a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10aeff5700 0 client.18243 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 shutdown_connections 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 1 -- 172.21.15.31:0/2062676653 wait complete. 2023-07-29T16:24:42.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.204+0000 7f10e512bb80 20 client.18243 trim_cache size 0 max 16384 2023-07-29T16:24:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.221+0000 7f10e512bb80 1 Processor -- start 2023-07-29T16:24:42.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.221+0000 7f10e512bb80 1 -- start start 2023-07-29T16:24:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924197df0 0x55a9241aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a92419bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10e512bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92418c9d0 con 0x55a924198cf0 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10e512bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9241a01e0 con 0x55a924197df0 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10e512bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a92419ef40 con 0x55a923ff36b0 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10ca7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924197df0 0x55a9241aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41850/0 (socket says 172.21.15.31:41850) 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 -- 172.21.15.31:0/3141826801 learned_addr learned my addr 172.21.15.31:0/3141826801 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a92419bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 -- 172.21.15.31:0/3141826801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924197df0 msgr2=0x55a9241aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924197df0 0x55a9241aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 -- 172.21.15.31:0/3141826801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 msgr2=0x55a92419bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a92419bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10caffd700 1 -- 172.21.15.31:0/3141826801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a923effc00 con 0x55a923ff36b0 2023-07-29T16:24:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924197df0 0x55a9241aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:42.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.222+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a92419bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:24:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.223+0000 7f10caffd700 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a63c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f10bc006010 tx=0x7f10bc01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6a9e5381d6ba8ed server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:42.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.223+0000 7f10b1ffb700 1 -- 172.21.15.31:0/3141826801 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc01d040 con 0x55a923ff36b0 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.223+0000 7f10b1ffb700 1 -- 172.21.15.31:0/3141826801 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10bc02d070 con 0x55a923ff36b0 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.223+0000 7f10b1ffb700 1 -- 172.21.15.31:0/3141826801 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc005310 con 0x55a923ff36b0 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 -- 172.21.15.31:0/3141826801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 msgr2=0x55a9241a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a63c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f10bc006010 tx=0x7f10bc01a040 comp rx=0 tx=0).stop 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 -- 172.21.15.31:0/3141826801 shutdown_connections 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924198cf0 0x55a92419bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a63c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 --2- 172.21.15.31:0/3141826801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924197df0 0x55a9241aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 -- 172.21.15.31:0/3141826801 >> 172.21.15.31:0/3141826801 conn(0x55a92418cfd0 msgr2=0x55a92419f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 -- 172.21.15.31:0/3141826801 shutdown_connections 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.224+0000 7f10e512bb80 1 -- 172.21.15.31:0/3141826801 wait complete. 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 -- start start 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 Processor -- start 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a4020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924197df0 0x55a9241a4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a9241a5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9241a5ab0 con 0x55a924197df0 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9241a5c20 con 0x55a924198cf0 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9241a5de0 con 0x55a923ff36b0 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10e512bb80 10 client.0 _ll_register_callbacks cb 0x55a92408e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.225+0000 7f10caffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a9241a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10caffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41854/0 (socket says 172.21.15.31:41854) 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10caffd700 1 -- 172.21.15.31:0/2493036082 learned_addr learned my addr 172.21.15.31:0/2493036082 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10ca7fc700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924197df0 0x55a9241a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 -- 172.21.15.31:0/2493036082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 msgr2=0x55a9241a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 -- 172.21.15.31:0/2493036082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924197df0 msgr2=0x55a9241a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924197df0 0x55a9241a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 -- 172.21.15.31:0/2493036082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c00b5040 con 0x55a924198cf0 2023-07-29T16:24:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10caffd700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10cb7fe700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a9241a5570 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f10c4007070 tx=0x7f10c4033be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddaf7b7b60ed5815 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10aeff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10aeff5700 1 -- 172.21.15.31:0/2493036082 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c4016ec0 con 0x55a924198cf0 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10aeff5700 1 -- 172.21.15.31:0/2493036082 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f10c40336d0 con 0x55a924198cf0 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.226+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10c00a1250 con 0x55a924198cf0 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10aeff5700 1 -- 172.21.15.31:0/2493036082 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10c4021460 con 0x55a924198cf0 2023-07-29T16:24:42.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 10 client.28316 fetch_fsmap learned FSMap version 28 2023-07-29T16:24:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f10bc00be00 con 0x55a924198cf0 2023-07-29T16:24:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10aeff5700 1 -- 172.21.15.31:0/2493036082 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10c00a1250 con 0x55a924198cf0 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10aeff5700 1 -- 172.21.15.31:0/2493036082 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 28) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f10c40163e0 con 0x55a924198cf0 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 10 client.28316 fetch_fsmap finished waiting for FSMap version 28 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 10 client.28316 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 20 client.28316 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241ff660 0x55a924201a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 4 client.28316 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:24:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.227+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a9240e1ce0 con 0x55a9241ff660 2023-07-29T16:24:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.228+0000 7f10caffd700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241ff660 0x55a924201a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.228+0000 7f10caffd700 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241ff660 0x55a924201a70 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d22e6e0783bd0cc0 server_cookie=af726395c0090ade in_seq=0 out_seq=0 2023-07-29T16:24:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.228+0000 7f10aeff5700 10 client.28316 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:42.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.306+0000 7f10aeff5700 1 -- 172.21.15.31:0/2493036082 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a9240e1ce0 con 0x55a9241ff660 2023-07-29T16:24:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.306+0000 7f10aeff5700 10 client.28316 handle_command_reply: tid=0 2023-07-29T16:24:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.306+0000 7f10e512bb80 1 client.28316 shutdown 2023-07-29T16:24:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 msgr2=0x55a9241a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a9241a5570 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f10c4007070 tx=0x7f10c4033be0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 shutdown_connections 2023-07-29T16:24:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a9241ff660 0x55a924201a70 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a924198cf0 0x55a9241a5570 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10aeff5700 0 client.28316 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a923ff36b0 0x55a9241a4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 --2- 172.21.15.31:0/2493036082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a924197df0 0x55a9241a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 >> 172.21.15.31:0/2493036082 conn(0x55a92418cfd0 msgr2=0x55a9241aed30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 shutdown_connections 2023-07-29T16:24:42.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 1 -- 172.21.15.31:0/2493036082 wait complete. 2023-07-29T16:24:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.307+0000 7f10e512bb80 20 client.28316 trim_cache size 0 max 16384 2023-07-29T16:24:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:24:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:24:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:24:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:24:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f10b40a0860 msgr2=0x7f10b40a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f10b40a0860 0x7f10b40a2d10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f10c40070c0 tx=0x7f10c4015970 comp rx=0 tx=0).stop 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 msgr2=0x7f10cc1a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 0x7f10cc1a5530 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f10c000d980 tx=0x7f10c000dc90 comp rx=0 tx=0).stop 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 shutdown_connections 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f10b40a0860 0x7f10b40a2d10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f10cc10c5a0 0x7f10cc1a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f10cc071740 0x7f10cc1a4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 --2- 172.21.15.31:0/2629722536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10cc072090 0x7f10cc1a5530 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 >> 172.21.15.31:0/2629722536 conn(0x7f10cc06d290 msgr2=0x7f10cc06e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 shutdown_connections 2023-07-29T16:24:42.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:42.308+0000 7f10d17ff700 1 -- 172.21.15.31:0/2629722536 wait complete. 2023-07-29T16:24:42.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:42 smithi039 ceph-mon[128858]: pgmap v647: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.9 MiB/s wr, 229 op/s 2023-07-29T16:24:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:42 smithi156 ceph-mon[129245]: pgmap v647: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.9 MiB/s wr, 229 op/s 2023-07-29T16:24:42.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 643780 8.94 MB/sec execute 267 sec latency 148.706 ms 2023-07-29T16:24:42.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:42 smithi031 ceph-mon[95504]: pgmap v647: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 1.9 MiB/s wr, 229 op/s 2023-07-29T16:24:43.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 646162 8.96 MB/sec execute 268 sec latency 16.603 ms 2023-07-29T16:24:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:44 smithi039 ceph-mon[128858]: pgmap v648: 97 pgs: 97 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:24:44.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:44 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:24:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:44 smithi156 ceph-mon[129245]: pgmap v648: 97 pgs: 97 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:24:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:44 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:24:44.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 646162 8.93 MB/sec execute 269 sec latency 1009.102 ms 2023-07-29T16:24:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:44 smithi031 ceph-mon[95504]: pgmap v648: 97 pgs: 97 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:24:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:44 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:24:45.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 646162 8.89 MB/sec execute 270 sec latency 2009.239 ms 2023-07-29T16:24:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:46 smithi156 ceph-mon[129245]: pgmap v649: 97 pgs: 97 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:24:46.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 646162 8.86 MB/sec execute 271 sec latency 3009.329 ms 2023-07-29T16:24:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:46 smithi031 ceph-mon[95504]: pgmap v649: 97 pgs: 97 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:24:47.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:46 smithi039 ceph-mon[128858]: pgmap v649: 97 pgs: 97 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:24:47.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:47 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:24:47.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 646649 8.84 MB/sec execute 272 sec latency 3560.834 ms 2023-07-29T16:24:47.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:47 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:24:48.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:47 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:24:48.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:48 smithi156 ceph-mon[129245]: pgmap v650: 97 pgs: 97 active+clean; 125 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2023-07-29T16:24:48.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:48 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:24:48.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:48 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:24:48.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:48 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:24:48.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 647999 8.83 MB/sec execute 273 sec latency 21.356 ms 2023-07-29T16:24:48.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:48 smithi031 ceph-mon[95504]: pgmap v650: 97 pgs: 97 active+clean; 125 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2023-07-29T16:24:48.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:48 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:24:48.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:48 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:24:48.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:48 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:24:49.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:48 smithi039 ceph-mon[128858]: pgmap v650: 97 pgs: 97 active+clean; 125 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 2.1 MiB/s wr, 244 op/s 2023-07-29T16:24:49.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:48 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:24:49.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:48 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:24:49.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:48 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:24:49.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 650032 8.85 MB/sec execute 274 sec latency 20.366 ms 2023-07-29T16:24:50.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: pgmap v651: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2023-07-29T16:24:50.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:50 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: pgmap v651: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2023-07-29T16:24:50.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:50 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: pgmap v651: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2023-07-29T16:24:50.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.824 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:50 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:50.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 650834 8.82 MB/sec execute 275 sec latency 75.513 ms 2023-07-29T16:24:51.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:24:51.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:24:51.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:51 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:51.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:24:51.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:24:51.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:51 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:51.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 652912 8.83 MB/sec execute 276 sec latency 25.227 ms 2023-07-29T16:24:51.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:24:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:24:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:51 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:24:52.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:52 smithi039 ceph-mon[128858]: pgmap v652: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2023-07-29T16:24:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:52 smithi156 ceph-mon[129245]: pgmap v652: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2023-07-29T16:24:52.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 654043 8.82 MB/sec execute 277 sec latency 30.127 ms 2023-07-29T16:24:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:52 smithi031 ceph-mon[95504]: pgmap v652: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 1.8 MiB/s wr, 178 op/s 2023-07-29T16:24:53.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 656353 8.84 MB/sec execute 278 sec latency 15.474 ms 2023-07-29T16:24:54.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:54 smithi039 ceph-mon[128858]: pgmap v653: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.5 MiB/s wr, 244 op/s 2023-07-29T16:24:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:54 smithi156 ceph-mon[129245]: pgmap v653: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.5 MiB/s wr, 244 op/s 2023-07-29T16:24:54.882 INFO:tasks.workunit.client.0.smithi031.stdout: 1 657585 8.83 MB/sec execute 279 sec latency 12.861 ms 2023-07-29T16:24:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:54 smithi031 ceph-mon[95504]: pgmap v653: 97 pgs: 97 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.5 MiB/s wr, 244 op/s 2023-07-29T16:24:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 -- 172.21.15.31:0/2895382536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc103780 msgr2=0x7f0edc105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/2895382536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc103780 0x7f0edc105b60 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009990 tx=0x7f0ed0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 -- 172.21.15.31:0/2895382536 shutdown_connections 2023-07-29T16:24:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/2895382536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0edc1060a0 0x7f0edc108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/2895382536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0edc100e60 0x7f0edc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/2895382536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc103780 0x7f0edc105b60 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 -- 172.21.15.31:0/2895382536 >> 172.21.15.31:0/2895382536 conn(0x7f0edc0fa840 msgr2=0x7f0edc0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 -- 172.21.15.31:0/2895382536 shutdown_connections 2023-07-29T16:24:55.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.476+0000 7f0ee21ff700 1 -- 172.21.15.31:0/2895382536 wait complete. 2023-07-29T16:24:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.477+0000 7f0ee21ff700 1 Processor -- start 2023-07-29T16:24:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.477+0000 7f0ee21ff700 1 -- start start 2023-07-29T16:24:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0ee21ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 0x7f0edc1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0ee21ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0edc103780 0x7f0edc1966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0ee21ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0edc1060a0 0x7f0edc197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0ee21ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc197bd0 con 0x7f0edc103780 2023-07-29T16:24:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0ee21ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc197d40 con 0x7f0edc100e60 2023-07-29T16:24:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0ee21ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc197f00 con 0x7f0edc1060a0 2023-07-29T16:24:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 0x7f0edc1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 0x7f0edc1961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59888/0 (socket says 172.21.15.31:59888) 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 -- 172.21.15.31:0/1490402392 learned_addr learned my addr 172.21.15.31:0/1490402392 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edaffd700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0edc103780 0x7f0edc1966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edbfff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0edc1060a0 0x7f0edc197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 -- 172.21.15.31:0/1490402392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0edc1060a0 msgr2=0x7f0edc197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0edc1060a0 0x7f0edc197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 -- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0edc103780 msgr2=0x7f0edc1966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0edc103780 0x7f0edc1966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.478+0000 7f0edb7fe700 1 -- 172.21.15.31:0/1490402392 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc009520 con 0x7f0edc100e60 2023-07-29T16:24:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.479+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 0x7f0edc1961b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00ea30 tx=0x7f0ecc00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:55.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.479+0000 7f0ed8ff9700 1 -- 172.21.15.31:0/1490402392 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00cbd0 con 0x7f0edc100e60 2023-07-29T16:24:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.479+0000 7f0ed8ff9700 1 -- 172.21.15.31:0/1490402392 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ecc00eed0 con 0x7f0edc100e60 2023-07-29T16:24:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.479+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed00095f0 con 0x7f0edc100e60 2023-07-29T16:24:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.479+0000 7f0ed8ff9700 1 -- 172.21.15.31:0/1490402392 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc003d20 con 0x7f0edc100e60 2023-07-29T16:24:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.479+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc0fee30 con 0x7f0edc100e60 2023-07-29T16:24:55.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.483+0000 7f0ed8ff9700 1 -- 172.21.15.31:0/1490402392 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0ecc003ee0 con 0x7f0edc100e60 2023-07-29T16:24:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.484+0000 7f0ed8ff9700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ec40a0810 0x7f0ec40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.484+0000 7f0ed8ff9700 1 -- 172.21.15.31:0/1490402392 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ecc014070 con 0x7f0edc100e60 2023-07-29T16:24:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.484+0000 7f0ef5a2bb80 1 Processor -- start 2023-07-29T16:24:55.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.484+0000 7f0edaffd700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ec40a0810 0x7f0ec40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 -- start start 2023-07-29T16:24:55.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adbc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adbd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adbe160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58ad149b0 con 0x55c58ac186b0 2023-07-29T16:24:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58ad12370 con 0x55c58adbcdf0 2023-07-29T16:24:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58ad0dc60 con 0x55c58adbdcf0 2023-07-29T16:24:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adbd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adbd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59904/0 (socket says 172.21.15.31:59904) 2023-07-29T16:24:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0edaffd700 1 -- 172.21.15.31:0/3831743947 learned_addr learned my addr 172.21.15.31:0/3831743947 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:55.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.485+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adbc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edbfff700 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adbe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ec40a0810 0x7f0ec40a2cc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b2e0 tx=0x7f0ed0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 -- 172.21.15.31:0/3831743947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 msgr2=0x55c58adbe160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adbe160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 -- 172.21.15.31:0/3831743947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 msgr2=0x55c58adbc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adbc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 -- 172.21.15.31:0/3831743947 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c58ab3bd80 con 0x55c58adbcdf0 2023-07-29T16:24:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0edaffd700 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adbd7b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0015000 tx=0x7f0ed0028000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f7c5c723d76c7d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0ec1ffb700 1 -- 172.21.15.31:0/3831743947 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0004700 con 0x55c58adbcdf0 2023-07-29T16:24:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0ec1ffb700 1 -- 172.21.15.31:0/3831743947 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ed001e040 con 0x55c58adbcdf0 2023-07-29T16:24:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.486+0000 7f0ec1ffb700 1 -- 172.21.15.31:0/3831743947 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000faf0 con 0x55c58adbcdf0 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/3831743947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 msgr2=0x55c58adbd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adbd7b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0015000 tx=0x7f0ed0028000 comp rx=0 tx=0).stop 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/3831743947 shutdown_connections 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adbe160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adbc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/3831743947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adbd7b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/3831743947 >> 172.21.15.31:0/3831743947 conn(0x55c58adb1fd0 msgr2=0x55c58adb7300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/3831743947 shutdown_connections 2023-07-29T16:24:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.487+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/3831743947 wait complete. 2023-07-29T16:24:55.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.488+0000 7f0ef5a2bb80 1 -- start start 2023-07-29T16:24:55.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.488+0000 7f0ef5a2bb80 1 Processor -- start 2023-07-29T16:24:55.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58acaa4b0 con 0x55c58ac186b0 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58acaa150 con 0x55c58adbcdf0 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58ab918c0 con 0x55c58adbdcf0 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0ef5a2bb80 10 client.0 _ll_register_callbacks cb 0x55c58acb35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59910/0 (socket says 172.21.15.31:59910) 2023-07-29T16:24:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 -- 172.21.15.31:0/2952762863 learned_addr learned my addr 172.21.15.31:0/2952762863 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edbfff700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 -- 172.21.15.31:0/2952762863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 msgr2=0x55c58adcd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 -- 172.21.15.31:0/2952762863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 msgr2=0x55c58adc5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.489+0000 7f0edaffd700 1 -- 172.21.15.31:0/2952762863 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc1090a0 con 0x55c58adbcdf0 2023-07-29T16:24:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0edaffd700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc63d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b680 tx=0x7f0ed00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a3e7fa291647ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ebaffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ebaffd700 1 -- 172.21.15.31:0/2952762863 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000fd50 con 0x55c58adbcdf0 2023-07-29T16:24:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c58ad38cb0 con 0x55c58adbcdf0 2023-07-29T16:24:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ebaffd700 1 -- 172.21.15.31:0/2952762863 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ed0004500 con 0x55c58adbcdf0 2023-07-29T16:24:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ebaffd700 1 -- 172.21.15.31:0/2952762863 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ed00046c0 con 0x55c58adbcdf0 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ef5a2bb80 10 client.28328 fetch_fsmap learned FSMap version 29 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.490+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c58ab24c00 con 0x55c58adbcdf0 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ebaffd700 1 -- 172.21.15.31:0/2952762863 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c58ad38cb0 con 0x55c58adbcdf0 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ebaffd700 1 -- 172.21.15.31:0/2952762863 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 29) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f0ed00378f0 con 0x55c58adbcdf0 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ef5a2bb80 10 client.28328 fetch_fsmap finished waiting for FSMap version 29 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ef5a2bb80 10 client.28328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ef5a2bb80 20 client.28328 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58add1890 0x55c58add3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ef5a2bb80 4 client.28328 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:24:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c58ad8bd10 con 0x55c58add1890 2023-07-29T16:24:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58add1890 0x55c58add3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.491+0000 7f0ebaffd700 10 client.28328 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.492+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58add1890 0x55c58add3ca0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933fe0dd408abb1d server_cookie=1a2d9c4668279c62 in_seq=0 out_seq=0 2023-07-29T16:24:55.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.492+0000 7f0ebaffd700 1 -- 172.21.15.31:0/2952762863 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c58ad8bd10 con 0x55c58add1890 2023-07-29T16:24:55.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.492+0000 7f0ebaffd700 10 client.28328 handle_command_reply: tid=0 2023-07-29T16:24:55.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.492+0000 7f0ef5a2bb80 1 client.28328 shutdown 2023-07-29T16:24:55.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 msgr2=0x55c58adc63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc63d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b680 tx=0x7f0ed00057f0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 shutdown_connections 2023-07-29T16:24:55.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58add1890 0x55c58add3ca0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2952762863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc63d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ebaffd700 0 client.28328 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:55.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 >> 172.21.15.31:0/2952762863 conn(0x55c58adb1fd0 msgr2=0x55c58adb6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:55.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 shutdown_connections 2023-07-29T16:24:55.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2952762863 wait complete. 2023-07-29T16:24:55.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.493+0000 7f0ef5a2bb80 20 client.28328 trim_cache size 0 max 16384 2023-07-29T16:24:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.511+0000 7f0ef5a2bb80 1 Processor -- start 2023-07-29T16:24:55.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.511+0000 7f0ef5a2bb80 1 -- start start 2023-07-29T16:24:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.511+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58ac186b0 0x55c58add85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.511+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58adbcdf0 0x55c58adcd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.511+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbdcf0 0x55c58adc4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58adc6b20 con 0x55c58adbcdf0 2023-07-29T16:24:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58add3ec0 con 0x55c58adbdcf0 2023-07-29T16:24:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58add4080 con 0x55c58ac186b0 2023-07-29T16:24:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58adbcdf0 0x55c58adcd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58ac186b0 0x55c58add85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbdcf0 0x55c58adc4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58adbcdf0 0x55c58adcd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48340/0 (socket says 172.21.15.31:48340) 2023-07-29T16:24:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edaffd700 1 -- 172.21.15.31:0/2568020626 learned_addr learned my addr 172.21.15.31:0/2568020626 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edb7fe700 1 -- 172.21.15.31:0/2568020626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbdcf0 msgr2=0x55c58adc4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbdcf0 0x55c58adc4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edb7fe700 1 -- 172.21.15.31:0/2568020626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58adbcdf0 msgr2=0x55c58adcd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58adbcdf0 0x55c58adcd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.512+0000 7f0edb7fe700 1 -- 172.21.15.31:0/2568020626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c58ab621f0 con 0x55c58ac186b0 2023-07-29T16:24:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.513+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58ac186b0 0x55c58add85c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0c98b0 tx=0x7f0ecc0c9780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb846f177923f0d4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.513+0000 7f0ec27fc700 1 -- 172.21.15.31:0/2568020626 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0a1960 con 0x55c58ac186b0 2023-07-29T16:24:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.513+0000 7f0ec27fc700 1 -- 172.21.15.31:0/2568020626 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ecc0b5070 con 0x55c58ac186b0 2023-07-29T16:24:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.513+0000 7f0ec27fc700 1 -- 172.21.15.31:0/2568020626 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc09d110 con 0x55c58ac186b0 2023-07-29T16:24:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2568020626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58ac186b0 msgr2=0x55c58add85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58ac186b0 0x55c58add85c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0c98b0 tx=0x7f0ecc0c9780 comp rx=0 tx=0).stop 2023-07-29T16:24:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2568020626 shutdown_connections 2023-07-29T16:24:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbdcf0 0x55c58adc4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58ac186b0 0x55c58add85c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/2568020626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58adbcdf0 0x55c58adcd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2568020626 >> 172.21.15.31:0/2568020626 conn(0x55c58adb1fd0 msgr2=0x55c58adb34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2568020626 shutdown_connections 2023-07-29T16:24:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/2568020626 wait complete. 2023-07-29T16:24:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 -- start start 2023-07-29T16:24:55.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.514+0000 7f0ef5a2bb80 1 Processor -- start 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58adccfb0 con 0x55c58ac186b0 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58adcd120 con 0x55c58adbcdf0 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58adcd2e0 con 0x55c58adbdcf0 2023-07-29T16:24:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0ef5a2bb80 10 client.0 _ll_register_callbacks cb 0x55c58acb35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:24:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0edb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0edaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59918/0 (socket says 172.21.15.31:59918) 2023-07-29T16:24:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0edaffd700 1 -- 172.21.15.31:0/527003139 learned_addr learned my addr 172.21.15.31:0/527003139 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:24:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.515+0000 7f0edbfff700 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0edaffd700 1 -- 172.21.15.31:0/527003139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 msgr2=0x55c58adcca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0edaffd700 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0edaffd700 1 -- 172.21.15.31:0/527003139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 msgr2=0x55c58adc9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:24:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0edaffd700 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0edaffd700 1 -- 172.21.15.31:0/527003139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc0fee30 con 0x55c58adbcdf0 2023-07-29T16:24:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0edaffd700 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc9aa0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0000c00 tx=0x7f0ed0037b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dd0a4a3a290461d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:24:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ebaffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:24:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ebaffd700 1 -- 172.21.15.31:0/527003139 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed000b310 con 0x55c58adbcdf0 2023-07-29T16:24:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ed001e040 con 0x55c58adbcdf0 2023-07-29T16:24:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ebaffd700 1 -- 172.21.15.31:0/527003139 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0ed000fc40 con 0x55c58adbcdf0 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ebaffd700 1 -- 172.21.15.31:0/527003139 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ed0018520 con 0x55c58adbcdf0 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ef5a2bb80 10 client.28334 fetch_fsmap learned FSMap version 29 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.516+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c58ab3a250 con 0x55c58adbcdf0 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ebaffd700 1 -- 172.21.15.31:0/527003139 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ed001e040 con 0x55c58adbcdf0 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ebaffd700 1 -- 172.21.15.31:0/527003139 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 29) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f0ed0004590 con 0x55c58adbcdf0 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ef5a2bb80 10 client.28334 fetch_fsmap finished waiting for FSMap version 29 2023-07-29T16:24:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ef5a2bb80 10 client.28334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:24:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ef5a2bb80 20 client.28334 populate_metadata read hostname 'smithi031' 2023-07-29T16:24:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58ae241b0 0x55c58ae265c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:24:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ef5a2bb80 4 client.28334 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:24:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c58ad06ce0 con 0x55c58ae241b0 2023-07-29T16:24:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.517+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58ae241b0 0x55c58ae265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:24:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.518+0000 7f0edb7fe700 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58ae241b0 0x55c58ae265c0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15343901da44d7fb server_cookie=4031f1f11d3f8b1c in_seq=0 out_seq=0 2023-07-29T16:24:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.518+0000 7f0ebaffd700 10 client.28334 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ebaffd700 1 -- 172.21.15.31:0/527003139 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c58ad06ce0 con 0x55c58ae241b0 2023-07-29T16:24:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ebaffd700 10 client.28334 handle_command_reply: tid=0 2023-07-29T16:24:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 client.28334 shutdown 2023-07-29T16:24:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 msgr2=0x55c58adc9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc9aa0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0000c00 tx=0x7f0ed0037b10 comp rx=0 tx=0).stop 2023-07-29T16:24:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 shutdown_connections 2023-07-29T16:24:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c58ae241b0 0x55c58ae265c0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c58adbdcf0 0x55c58adcca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c58ac186b0 0x55c58adc9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 --2- 172.21.15.31:0/527003139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c58adbcdf0 0x55c58adc9aa0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 >> 172.21.15.31:0/527003139 conn(0x55c58adb1fd0 msgr2=0x55c58adc0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.579+0000 7f0ebaffd700 0 client.28334 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 shutdown_connections 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ef5a2bb80 1 -- 172.21.15.31:0/527003139 wait complete. 2023-07-29T16:24:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ef5a2bb80 20 client.28334 trim_cache size 0 max 16384 2023-07-29T16:24:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:24:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:24:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:24:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:24:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ec40a0810 msgr2=0x7f0ec40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ec40a0810 0x7f0ec40a2cc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000b2e0 tx=0x7f0ed0005fb0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 msgr2=0x7f0edc1961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:24:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.580+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 0x7f0edc1961b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00ea30 tx=0x7f0ecc00ed40 comp rx=0 tx=0).stop 2023-07-29T16:24:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 shutdown_connections 2023-07-29T16:24:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0ec40a0810 0x7f0ec40a2cc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0edc1060a0 0x7f0edc197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0edc100e60 0x7f0edc1961b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 --2- 172.21.15.31:0/1490402392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0edc103780 0x7f0edc1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:24:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 >> 172.21.15.31:0/1490402392 conn(0x7f0edc0fa840 msgr2=0x7f0edc1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:24:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 shutdown_connections 2023-07-29T16:24:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:24:55.581+0000 7f0ee21ff700 1 -- 172.21.15.31:0/1490402392 wait complete. 2023-07-29T16:24:55.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 659477 8.83 MB/sec execute 280 sec latency 34.605 ms 2023-07-29T16:24:56.322 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:24:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: ignoring --setuser ceph since I am not root 2023-07-29T16:24:56.322 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:24:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: ignoring --setgroup ceph since I am not root 2023-07-29T16:24:56.322 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:24:55 smithi156 ceph-mds[155210]: -- 172.21.15.156:0/1064398477 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de233611e0 con 0x55de23337000 2023-07-29T16:24:56.322 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:24:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: starting mds.c at 2023-07-29T16:24:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:56 smithi039 ceph-mon[128858]: pgmap v654: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 2.1 MiB/s wr, 195 op/s 2023-07-29T16:24:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:56 smithi039 ceph-mon[128858]: daemon mds.c restarted 2023-07-29T16:24:56.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:56 smithi039 ceph-mon[128858]: osdmap e92: 12 total, 12 up, 12 in 2023-07-29T16:24:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:56 smithi156 ceph-mon[129245]: pgmap v654: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 2.1 MiB/s wr, 195 op/s 2023-07-29T16:24:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:56 smithi156 ceph-mon[129245]: daemon mds.c restarted 2023-07-29T16:24:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:56 smithi156 ceph-mon[129245]: osdmap e92: 12 total, 12 up, 12 in 2023-07-29T16:24:56.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 660910 8.83 MB/sec execute 281 sec latency 23.993 ms 2023-07-29T16:24:56.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:56 smithi031 ceph-mon[95504]: pgmap v654: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 2.1 MiB/s wr, 195 op/s 2023-07-29T16:24:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:56 smithi031 ceph-mon[95504]: daemon mds.c restarted 2023-07-29T16:24:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:56 smithi031 ceph-mon[95504]: osdmap e92: 12 total, 12 up, 12 in 2023-07-29T16:24:57.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:57 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6836/3963887865,v1:172.21.15.156:6837/3963887865] up:boot 2023-07-29T16:24:57.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:57 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2023-07-29T16:24:57.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:24:57.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:57 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:24:57.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:57 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6836/3963887865,v1:172.21.15.156:6837/3963887865] up:boot 2023-07-29T16:24:57.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:57 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2023-07-29T16:24:57.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:24:57.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:57 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:24:57.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 662824 8.83 MB/sec execute 282 sec latency 12.817 ms 2023-07-29T16:24:57.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:57 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6836/3963887865,v1:172.21.15.156:6837/3963887865] up:boot 2023-07-29T16:24:57.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:57 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 2 up:standby-replay 7 up:standby 2023-07-29T16:24:57.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:24:57.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:57 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:24:58.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:24:58 smithi156 ceph-mon[129245]: pgmap v656: 97 pgs: 97 active+clean; 124 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 2.2 MiB/s wr, 196 op/s 2023-07-29T16:24:58.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 664302 8.84 MB/sec execute 283 sec latency 137.544 ms 2023-07-29T16:24:58.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:24:58 smithi031 ceph-mon[95504]: pgmap v656: 97 pgs: 97 active+clean; 124 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 2.2 MiB/s wr, 196 op/s 2023-07-29T16:24:59.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:24:58 smithi039 ceph-mon[128858]: pgmap v656: 97 pgs: 97 active+clean; 124 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 2.2 MiB/s wr, 196 op/s 2023-07-29T16:24:59.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 665603 8.83 MB/sec execute 284 sec latency 16.663 ms 2023-07-29T16:25:00.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:00 smithi156 ceph-mon[129245]: pgmap v657: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2023-07-29T16:25:00.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 667707 8.84 MB/sec execute 285 sec latency 48.748 ms 2023-07-29T16:25:00.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:00 smithi031 ceph-mon[95504]: pgmap v657: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2023-07-29T16:25:01.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:00 smithi039 ceph-mon[128858]: pgmap v657: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2023-07-29T16:25:01.883 INFO:tasks.workunit.client.0.smithi031.stdout: 1 668559 8.82 MB/sec execute 286 sec latency 61.409 ms 2023-07-29T16:25:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:02 smithi156 ceph-mon[129245]: pgmap v658: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2023-07-29T16:25:02.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 670809 8.83 MB/sec execute 287 sec latency 19.813 ms 2023-07-29T16:25:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:02 smithi031 ceph-mon[95504]: pgmap v658: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2023-07-29T16:25:03.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:02 smithi039 ceph-mon[128858]: pgmap v658: 97 pgs: 97 active+clean; 126 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.4 MiB/s wr, 274 op/s 2023-07-29T16:25:03.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 671989 8.82 MB/sec execute 288 sec latency 13.214 ms 2023-07-29T16:25:04.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:04 smithi156 ceph-mon[129245]: pgmap v659: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 2.3 MiB/s wr, 279 op/s 2023-07-29T16:25:04.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 674191 8.83 MB/sec execute 289 sec latency 44.403 ms 2023-07-29T16:25:04.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:04 smithi031 ceph-mon[95504]: pgmap v659: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 2.3 MiB/s wr, 279 op/s 2023-07-29T16:25:05.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:04 smithi039 ceph-mon[128858]: pgmap v659: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 2.3 MiB/s wr, 279 op/s 2023-07-29T16:25:05.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 675499 8.83 MB/sec execute 290 sec latency 11.563 ms 2023-07-29T16:25:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:06 smithi156 ceph-mon[129245]: pgmap v660: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 2.5 MiB/s wr, 271 op/s 2023-07-29T16:25:06.834 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:06 smithi039 ceph-mon[128858]: pgmap v660: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 2.5 MiB/s wr, 271 op/s 2023-07-29T16:25:06.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 677529 8.83 MB/sec execute 291 sec latency 50.266 ms 2023-07-29T16:25:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:06 smithi031 ceph-mon[95504]: pgmap v660: 97 pgs: 97 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 2.5 MiB/s wr, 271 op/s 2023-07-29T16:25:07.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 678766 8.83 MB/sec execute 292 sec latency 23.092 ms 2023-07-29T16:25:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:07 smithi039 ceph-mon[128858]: pgmap v661: 97 pgs: 97 active+clean; 129 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.3 MiB/s wr, 241 op/s 2023-07-29T16:25:08.321 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:07 smithi156 ceph-mon[129245]: pgmap v661: 97 pgs: 97 active+clean; 129 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.3 MiB/s wr, 241 op/s 2023-07-29T16:25:08.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:07 smithi031 ceph-mon[95504]: pgmap v661: 97 pgs: 97 active+clean; 129 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.3 MiB/s wr, 241 op/s 2023-07-29T16:25:08.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:25:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 -- 172.21.15.31:0/910555998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec108200 msgr2=0x7ff9ec10a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/910555998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec108200 0x7ff9ec10a680 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff9e400b0a0 tx=0x7ff9e400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 -- 172.21.15.31:0/910555998 shutdown_connections 2023-07-29T16:25:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/910555998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec108200 0x7ff9ec10a680 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/910555998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9ec074d90 0x7ff9ec0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/910555998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 -- 172.21.15.31:0/910555998 >> 172.21.15.31:0/910555998 conn(0x7ff9ec0fb920 msgr2=0x7ff9ec0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 -- 172.21.15.31:0/910555998 shutdown_connections 2023-07-29T16:25:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.762+0000 7ff9f11ff700 1 -- 172.21.15.31:0/910555998 wait complete. 2023-07-29T16:25:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.763+0000 7ff9f11ff700 1 Processor -- start 2023-07-29T16:25:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.763+0000 7ff9f11ff700 1 -- start start 2023-07-29T16:25:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.763+0000 7ff9f11ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.763+0000 7ff9f11ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec074d90 0x7ff9ec198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9f11ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9ec108200 0x7ff9ec1998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9f11ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec199f00 con 0x7ff9ec108200 2023-07-29T16:25:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9f11ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec19a070 con 0x7ff9ec074d90 2023-07-29T16:25:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9f11ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec19a230 con 0x7ff9ec0737b0 2023-07-29T16:25:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57690/0 (socket says 172.21.15.31:57690) 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 -- 172.21.15.31:0/1069460196 learned_addr learned my addr 172.21.15.31:0/1069460196 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec074d90 0x7ff9ec198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9eb59e700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9ec108200 0x7ff9ec1998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 -- 172.21.15.31:0/1069460196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec074d90 msgr2=0x7ff9ec198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec074d90 0x7ff9ec198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 -- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9ec108200 msgr2=0x7ff9ec1998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9ec108200 0x7ff9ec1998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.764+0000 7ff9ead9d700 1 -- 172.21.15.31:0/1069460196 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e4009d00 con 0x7ff9ec0737b0 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.765+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec198530 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00e970 tx=0x7ff9dc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.765+0000 7ff9d3fff700 1 -- 172.21.15.31:0/1069460196 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc009790 con 0x7ff9ec0737b0 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.765+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec19a510 con 0x7ff9ec0737b0 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.765+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec1a0cd0 con 0x7ff9ec0737b0 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.765+0000 7ff9d3fff700 1 -- 172.21.15.31:0/1069460196 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9dc00eec0 con 0x7ff9ec0737b0 2023-07-29T16:25:08.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.765+0000 7ff9d3fff700 1 -- 172.21.15.31:0/1069460196 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc005500 con 0x7ff9ec0737b0 2023-07-29T16:25:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.769+0000 7ff9d3fff700 1 -- 172.21.15.31:0/1069460196 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff9dc00cbd0 con 0x7ff9ec0737b0 2023-07-29T16:25:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.771+0000 7ff9d3fff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9d40a0810 0x7ff9d40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.771+0000 7ff9d3fff700 1 -- 172.21.15.31:0/1069460196 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff9dc014070 con 0x7ff9ec0737b0 2023-07-29T16:25:08.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.771+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9d40a0810 0x7ff9d40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.772+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9d40a0810 0x7ff9d40a2cc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec074ab0 tx=0x7ff9e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.772+0000 7ffa04b0db80 1 Processor -- start 2023-07-29T16:25:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 -- start start 2023-07-29T16:25:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed12df0 0x56433ed137b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed14160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ec6a9b0 con 0x56433ed12df0 2023-07-29T16:25:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ec68370 con 0x56433ed13cf0 2023-07-29T16:25:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ec63c60 con 0x56433eb6e6b0 2023-07-29T16:25:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ff9ead9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ff9eb59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed14160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ff9ea59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed12df0 0x56433ed137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ff9eb59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed14160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39594/0 (socket says 172.21.15.31:39594) 2023-07-29T16:25:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.773+0000 7ff9eb59e700 1 -- 172.21.15.31:0/1224732487 learned_addr learned my addr 172.21.15.31:0/1224732487 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.774+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1224732487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 msgr2=0x56433ed128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.774+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.774+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1224732487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 msgr2=0x56433ed14160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.774+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed14160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.774+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1224732487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56433ea91d80 con 0x56433ed12df0 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.774+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed12df0 0x56433ed137b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0011c80 tx=0x7ff9e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d887a8850276340a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.775+0000 7ff9d17fa700 1 -- 172.21.15.31:0/1224732487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0004220 con 0x56433ed12df0 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.775+0000 7ff9d17fa700 1 -- 172.21.15.31:0/1224732487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9e0021070 con 0x56433ed12df0 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.775+0000 7ff9d17fa700 1 -- 172.21.15.31:0/1224732487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0003b70 con 0x56433ed12df0 2023-07-29T16:25:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1224732487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed12df0 msgr2=0x56433ed137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed12df0 0x56433ed137b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0011c80 tx=0x7ff9e0014000 comp rx=0 tx=0).stop 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1224732487 shutdown_connections 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed14160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1224732487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed12df0 0x56433ed137b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1224732487 >> 172.21.15.31:0/1224732487 conn(0x56433ed07fd0 msgr2=0x56433ed0d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1224732487 shutdown_connections 2023-07-29T16:25:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.776+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1224732487 wait complete. 2023-07-29T16:25:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.777+0000 7ffa04b0db80 1 -- start start 2023-07-29T16:25:08.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.777+0000 7ffa04b0db80 1 Processor -- start 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed246e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed24c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed2bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ec004b0 con 0x56433ed13cf0 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ec00150 con 0x56433ed12df0 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433eae78c0 con 0x56433eb6e6b0 2023-07-29T16:25:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ffa04b0db80 10 client.0 _ll_register_callbacks cb 0x56433ec095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed24c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed24c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39604/0 (socket says 172.21.15.31:39604) 2023-07-29T16:25:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1999626611 learned_addr learned my addr 172.21.15.31:0/1999626611 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1999626611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 msgr2=0x56433ed246e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:25:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed246e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1999626611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 msgr2=0x56433ed2bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed2bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.778+0000 7ff9ea59c700 1 -- 172.21.15.31:0/1999626611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e0017070 con 0x56433ed12df0 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed24c20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000e980 tx=0x7ff9e000ea60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df6abc4433342a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ff9ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ff9ceff5700 1 -- 172.21.15.31:0/1999626611 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0026680 con 0x56433ed12df0 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ff9ceff5700 1 -- 172.21.15.31:0/1999626611 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff9e00267e0 con 0x56433ed12df0 2023-07-29T16:25:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56433ec8ecb0 con 0x56433ed12df0 2023-07-29T16:25:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ff9ceff5700 1 -- 172.21.15.31:0/1999626611 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9e00269a0 con 0x56433ed12df0 2023-07-29T16:25:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ffa04b0db80 10 client.28352 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.779+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56433ea7ac00 con 0x56433ed12df0 2023-07-29T16:25:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ff9ceff5700 1 -- 172.21.15.31:0/1999626611 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56433ec8ecb0 con 0x56433ed12df0 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ff9ceff5700 1 -- 172.21.15.31:0/1999626611 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7ff9e0030cc0 con 0x56433ed12df0 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ffa04b0db80 10 client.28352 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ffa04b0db80 10 client.28352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ffa04b0db80 20 client.28352 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed303d0 0x56433ed327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ffa04b0db80 4 client.28352 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.780+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56433ece1d10 con 0x56433ed303d0 2023-07-29T16:25:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.781+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed303d0 0x56433ed327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.781+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed303d0 0x56433ed327e0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42f0b1fe04a9b8af server_cookie=8b080a124d7c795b in_seq=0 out_seq=0 2023-07-29T16:25:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.781+0000 7ff9ceff5700 10 client.28352 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.783+0000 7ff9ceff5700 1 -- 172.21.15.31:0/1999626611 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56433ece1d10 con 0x56433ed303d0 2023-07-29T16:25:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.783+0000 7ff9ceff5700 10 client.28352 handle_command_reply: tid=0 2023-07-29T16:25:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.783+0000 7ffa04b0db80 1 client.28352 shutdown 2023-07-29T16:25:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 msgr2=0x56433ed24c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed24c20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff9e000e980 tx=0x7ff9e000ea60 comp rx=0 tx=0).stop 2023-07-29T16:25:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 shutdown_connections 2023-07-29T16:25:08.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed303d0 0x56433ed327e0 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed2bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed246e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/1999626611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed24c20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 >> 172.21.15.31:0/1999626611 conn(0x56433ed07fd0 msgr2=0x56433ed0ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ff9ceff5700 0 client.28352 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 shutdown_connections 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 1 -- 172.21.15.31:0/1999626611 wait complete. 2023-07-29T16:25:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.784+0000 7ffa04b0db80 20 client.28352 trim_cache size 0 max 16384 2023-07-29T16:25:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.808+0000 7ffa04b0db80 1 Processor -- start 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.808+0000 7ffa04b0db80 1 -- start start 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433eb6e6b0 0x56433ed29b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433ed12df0 0x56433ed140c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed16ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ed30100 con 0x56433eb6e6b0 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ed24490 con 0x56433ed13cf0 2023-07-29T16:25:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ed0e140 con 0x56433ed12df0 2023-07-29T16:25:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed16ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9ead9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433eb6e6b0 0x56433ed29b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed16ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39610/0 (socket says 172.21.15.31:39610) 2023-07-29T16:25:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 -- 172.21.15.31:0/827168009 learned_addr learned my addr 172.21.15.31:0/827168009 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433ed12df0 0x56433ed140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 -- 172.21.15.31:0/827168009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433ed12df0 msgr2=0x56433ed140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433ed12df0 0x56433ed140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 -- 172.21.15.31:0/827168009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433eb6e6b0 msgr2=0x56433ed29b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433eb6e6b0 0x56433ed29b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9eb59e700 1 -- 172.21.15.31:0/827168009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56433eab81f0 con 0x56433ed13cf0 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.809+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433eb6e6b0 0x56433ed29b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.810+0000 7ff9eb59e700 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed16ac0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4007a60 tx=0x7ff9e4009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd6c9c5f556de831 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.810+0000 7ff9d1ffb700 1 -- 172.21.15.31:0/827168009 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400e040 con 0x56433ed13cf0 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.810+0000 7ff9d1ffb700 1 -- 172.21.15.31:0/827168009 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9e4009470 con 0x56433ed13cf0 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.810+0000 7ff9d1ffb700 1 -- 172.21.15.31:0/827168009 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40052a0 con 0x56433ed13cf0 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 -- 172.21.15.31:0/827168009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 msgr2=0x56433ed16ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed16ac0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4007a60 tx=0x7ff9e4009300 comp rx=0 tx=0).stop 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 -- 172.21.15.31:0/827168009 shutdown_connections 2023-07-29T16:25:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed13cf0 0x56433ed16ac0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433eb6e6b0 0x56433ed29b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/827168009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433ed12df0 0x56433ed140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 -- 172.21.15.31:0/827168009 >> 172.21.15.31:0/827168009 conn(0x56433ed07fd0 msgr2=0x56433ed323e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 -- 172.21.15.31:0/827168009 shutdown_connections 2023-07-29T16:25:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 -- 172.21.15.31:0/827168009 wait complete. 2023-07-29T16:25:08.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.811+0000 7ffa04b0db80 1 -- start start 2023-07-29T16:25:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 Processor -- start 2023-07-29T16:25:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed29860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed27e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed283b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ed288f0 con 0x56433ed13cf0 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ed28a60 con 0x56433ed12df0 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56433ed28bd0 con 0x56433eb6e6b0 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.812+0000 7ffa04b0db80 10 client.0 _ll_register_callbacks cb 0x56433ec095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ea59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed27e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ea59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed27e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39626/0 (socket says 172.21.15.31:39626) 2023-07-29T16:25:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ea59c700 1 -- 172.21.15.31:0/2516512683 learned_addr learned my addr 172.21.15.31:0/2516512683 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed29860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9eb59e700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 -- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 msgr2=0x56433ed27e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed27e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 -- 172.21.15.31:0/2516512683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 msgr2=0x56433ed283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed283b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 -- 172.21.15.31:0/2516512683 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec10b2a0 con 0x56433eb6e6b0 2023-07-29T16:25:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9eb59e700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ead9d700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed29860 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc005950 tx=0x7ff9dc021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e474ca17cd38e5d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.813+0000 7ff9ceff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:08.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ff9ceff5700 1 -- 172.21.15.31:0/2516512683 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc0a1920 con 0x56433eb6e6b0 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ff9ceff5700 1 -- 172.21.15.31:0/2516512683 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff9dc097870 con 0x56433eb6e6b0 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9e0021070 con 0x56433eb6e6b0 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ff9ceff5700 1 -- 172.21.15.31:0/2516512683 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9dc005660 con 0x56433eb6e6b0 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 10 client.28363 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56433ea90250 con 0x56433eb6e6b0 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ff9ceff5700 1 -- 172.21.15.31:0/2516512683 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9e0021070 con 0x56433eb6e6b0 2023-07-29T16:25:08.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ff9ceff5700 1 -- 172.21.15.31:0/2516512683 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7ff9dc099730 con 0x56433eb6e6b0 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 10 client.28363 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 10 client.28363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 20 client.28363 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed83670 0x56433ed85a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 4 client.28363 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.814+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56433ec5cce0 con 0x56433ed83670 2023-07-29T16:25:08.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.815+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed83670 0x56433ed85a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.815+0000 7ff9ceff5700 10 client.28363 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.815+0000 7ff9ea59c700 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed83670 0x56433ed85a80 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49f6adf11c51ed21 server_cookie=6f0307042378cd45 in_seq=0 out_seq=0 2023-07-29T16:25:08.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 680739 8.82 MB/sec execute 293 sec latency 19.174 ms 2023-07-29T16:25:08.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 -- 172.21.15.31:0/3670989739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 msgr2=0x7f039c0ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 --2- 172.21.15.31:0/3670989739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c0ff9b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f0388009990 tx=0x7f0388009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 -- 172.21.15.31:0/3670989739 shutdown_connections 2023-07-29T16:25:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 --2- 172.21.15.31:0/3670989739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f039c0ffef0 0x7f039c100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 --2- 172.21.15.31:0/3670989739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0febf0 0x7f039c0ff000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 --2- 172.21.15.31:0/3670989739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c0ff9b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 -- 172.21.15.31:0/3670989739 >> 172.21.15.31:0/3670989739 conn(0x7f039c0fa840 msgr2=0x7f039c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 -- 172.21.15.31:0/3670989739 shutdown_connections 2023-07-29T16:25:08.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.891+0000 7f03a35ff700 1 -- 172.21.15.31:0/3670989739 wait complete. 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.892+0000 7f03a35ff700 1 Processor -- start 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 -- start start 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f039c0febf0 0x7f039c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c19cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0ffef0 0x7f039c1a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1a4250 con 0x7f039c0ffef0 2023-07-29T16:25:08.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1a4390 con 0x7f039c0ff540 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a35ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1a4520 con 0x7f039c0febf0 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a0b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c19cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a0b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c19cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39648/0 (socket says 172.21.15.31:39648) 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a0b9a700 1 -- 172.21.15.31:0/2577189529 learned_addr learned my addr 172.21.15.31:0/2577189529 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a139b700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f039c0febf0 0x7f039c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.893+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0ffef0 0x7f039c1a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a0b9a700 1 -- 172.21.15.31:0/2577189529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f039c0febf0 msgr2=0x7f039c19c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f039c0febf0 0x7f039c19c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a0b9a700 1 -- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0ffef0 msgr2=0x7f039c1a3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0ffef0 0x7f039c1a3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a0b9a700 1 -- 172.21.15.31:0/2577189529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03880095f0 con 0x7f039c0ff540 2023-07-29T16:25:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0ffef0 0x7f039c1a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c19cc10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0388005910 tx=0x7f038800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f039a7fc700 1 -- 172.21.15.31:0/2577189529 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038801d070 con 0x7f039c0ff540 2023-07-29T16:25:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f039a7fc700 1 -- 172.21.15.31:0/2577189529 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f038800fb80 con 0x7f039c0ff540 2023-07-29T16:25:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f039a7fc700 1 -- 172.21.15.31:0/2577189529 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388017650 con 0x7f039c0ff540 2023-07-29T16:25:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c1a47a0 con 0x7f039c0ff540 2023-07-29T16:25:08.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.894+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c1a4b70 con 0x7f039c0ff540 2023-07-29T16:25:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.898+0000 7f039a7fc700 1 -- 172.21.15.31:0/2577189529 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f038800fcf0 con 0x7f039c0ff540 2023-07-29T16:25:08.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.899+0000 7f03b6f49b80 1 Processor -- start 2023-07-29T16:25:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.899+0000 7f03b6f49b80 1 -- start start 2023-07-29T16:25:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.899+0000 7f039a7fc700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f038c0a0810 0x7f038c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.899+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895668be0 0x55e895668ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895669d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89566a2f0 0x55e89566a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8955c0a10 con 0x55e895669530 2023-07-29T16:25:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8955be3d0 con 0x55e895668be0 2023-07-29T16:25:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8955b9cc0 con 0x55e89566a2f0 2023-07-29T16:25:08.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f039a7fc700 1 -- 172.21.15.31:0/2577189529 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f03880ccfe0 con 0x7f039c0ff540 2023-07-29T16:25:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03a139b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895668be0 0x55e895668ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03a1b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89566a2f0 0x55e89566a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03a139b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895668be0 0x55e895668ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39654/0 (socket says 172.21.15.31:39654) 2023-07-29T16:25:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03a139b700 1 -- 172.21.15.31:0/2239607968 learned_addr learned my addr 172.21.15.31:0/2239607968 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895669d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.900+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f038c0a0810 0x7f038c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.900+0000 7ff9ceff5700 1 -- 172.21.15.31:0/2516512683 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56433ec5cce0 con 0x56433ed83670 2023-07-29T16:25:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.900+0000 7ff9ceff5700 10 client.28363 handle_command_reply: tid=0 2023-07-29T16:25:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 client.28363 shutdown 2023-07-29T16:25:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03a1b9c700 1 -- 172.21.15.31:0/2239607968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895668be0 msgr2=0x55e895668ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895668be0 0x55e895668ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03a1b9c700 1 -- 172.21.15.31:0/2239607968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 msgr2=0x55e895669d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895669d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03a1b9c700 1 -- 172.21.15.31:0/2239607968 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8953e7de0 con 0x55e89566a2f0 2023-07-29T16:25:08.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89566a2f0 0x55e89566a740 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f03940137f0 tx=0x7f0394013b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614d89fb4e06ff9b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 msgr2=0x56433ed29860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed29860 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc005950 tx=0x7ff9dc021000 comp rx=0 tx=0).stop 2023-07-29T16:25:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 shutdown_connections 2023-07-29T16:25:08.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56433ed83670 0x56433ed85a80 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56433ed13cf0 0x56433ed283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56433eb6e6b0 0x56433ed29860 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 --2- 172.21.15.31:0/2516512683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56433ed12df0 0x56433ed27e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 >> 172.21.15.31:0/2516512683 conn(0x56433ed07fd0 msgr2=0x56433ed16ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ff9ceff5700 0 client.28363 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 shutdown_connections 2023-07-29T16:25:08.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 1 -- 172.21.15.31:0/2516512683 wait complete. 2023-07-29T16:25:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03877fe700 1 -- 172.21.15.31:0/2239607968 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039400ccb0 con 0x55e89566a2f0 2023-07-29T16:25:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03877fe700 1 -- 172.21.15.31:0/2239607968 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0394019070 con 0x55e89566a2f0 2023-07-29T16:25:08.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.901+0000 7f03877fe700 1 -- 172.21.15.31:0/2239607968 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0394005600 con 0x55e89566a2f0 2023-07-29T16:25:08.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.901+0000 7ffa04b0db80 20 client.28363 trim_cache size 0 max 16384 2023-07-29T16:25:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:25:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:25:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:25:08.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:25:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9d40a0810 msgr2=0x7ff9d40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9d40a0810 0x7ff9d40a2cc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec074ab0 tx=0x7ff9e0009260 comp rx=0 tx=0).stop 2023-07-29T16:25:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 msgr2=0x7ff9ec198530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec198530 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc00e970 tx=0x7ff9dc00ec80 comp rx=0 tx=0).stop 2023-07-29T16:25:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 shutdown_connections 2023-07-29T16:25:08.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9d40a0810 0x7ff9d40a2cc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9ec108200 0x7ff9ec1998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9ec0737b0 0x7ff9ec198530 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 --2- 172.21.15.31:0/1069460196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff9ec074d90 0x7ff9ec198a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 >> 172.21.15.31:0/1069460196 conn(0x7ff9ec0fb920 msgr2=0x7ff9ec0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2239607968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89566a2f0 msgr2=0x55e89566a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89566a2f0 0x55e89566a740 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f03940137f0 tx=0x7f0394013b00 comp rx=0 tx=0).stop 2023-07-29T16:25:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f038c0a0810 0x7f038c0a2cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0394020000 tx=0x7f039401d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 shutdown_connections 2023-07-29T16:25:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2239607968 shutdown_connections 2023-07-29T16:25:08.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89566a2f0 0x55e89566a740 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895668be0 0x55e895668ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2239607968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895669d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.902+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2239607968 >> 172.21.15.31:0/2239607968 conn(0x55e89565e3e0 msgr2=0x55e895663990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:08.902+0000 7ff9f11ff700 1 -- 172.21.15.31:0/1069460196 wait complete. 2023-07-29T16:25:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.903+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2239607968 shutdown_connections 2023-07-29T16:25:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.903+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2239607968 wait complete. 2023-07-29T16:25:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.903+0000 7f03b6f49b80 1 -- start start 2023-07-29T16:25:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 Processor -- start 2023-07-29T16:25:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e8956727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895669530 0x55e895672d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e89566a2f0 0x55e895679d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e895556510 con 0x55e89566a2f0 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8955561b0 con 0x55e895669530 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89543d920 con 0x55e895668be0 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.904+0000 7f03b6f49b80 10 client.0 _ll_register_callbacks cb 0x55e89555f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e8956727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e8956727d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57750/0 (socket says 172.21.15.31:57750) 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 -- 172.21.15.31:0/4213290789 learned_addr learned my addr 172.21.15.31:0/4213290789 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895669530 0x55e895672d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e89566a2f0 0x55e895679d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 -- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895669530 msgr2=0x55e895672d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895669530 0x55e895672d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 -- 172.21.15.31:0/4213290789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e89566a2f0 msgr2=0x55e895679d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e89566a2f0 0x55e895679d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 -- 172.21.15.31:0/4213290789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0394013460 con 0x55e895668be0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f03a139b700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e8956727d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f03900042c0 tx=0x7f03900042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6abd3fcdb28d8c3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.905+0000 7f0384ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f0384ff9700 1 -- 172.21.15.31:0/4213290789 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390004db0 con 0x55e895668be0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f0384ff9700 1 -- 172.21.15.31:0/4213290789 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0390007710 con 0x55e895668be0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e8955e4d10 con 0x55e895668be0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f0384ff9700 1 -- 172.21.15.31:0/4213290789 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03900078d0 con 0x55e895668be0 2023-07-29T16:25:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f03b6f49b80 10 client.28375 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e8953d0c60 con 0x55e895668be0 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f0384ff9700 1 -- 172.21.15.31:0/4213290789 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e8955e4d10 con 0x55e895668be0 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.906+0000 7f0384ff9700 1 -- 172.21.15.31:0/4213290789 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f0390016b70 con 0x55e895668be0 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03b6f49b80 10 client.28375 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03b6f49b80 10 client.28375 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03b6f49b80 20 client.28375 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e89567e5f0 0x55e895680a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03b6f49b80 4 client.28375 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e895637d70 con 0x55e89567e5f0 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e89567e5f0 0x55e895680a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e89567e5f0 0x55e895680a00 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42016bf14ea6d0f7 server_cookie=2bd064034d6b221b in_seq=0 out_seq=0 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.907+0000 7f0384ff9700 10 client.28375 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.909+0000 7f0384ff9700 1 -- 172.21.15.31:0/4213290789 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e895637d70 con 0x55e89567e5f0 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.909+0000 7f0384ff9700 10 client.28375 handle_command_reply: tid=0 2023-07-29T16:25:08.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.909+0000 7f03b6f49b80 1 client.28375 shutdown 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 msgr2=0x55e8956727d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e8956727d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f03900042c0 tx=0x7f03900042f0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 shutdown_connections 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e89567e5f0 0x55e895680a00 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e89566a2f0 0x55e895679d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e8956727d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/4213290789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e895669530 0x55e895672d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f0384ff9700 0 client.28375 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 >> 172.21.15.31:0/4213290789 conn(0x55e89565e3e0 msgr2=0x55e895663060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 shutdown_connections 2023-07-29T16:25:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 1 -- 172.21.15.31:0/4213290789 wait complete. 2023-07-29T16:25:08.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.910+0000 7f03b6f49b80 20 client.28375 trim_cache size 0 max 16384 2023-07-29T16:25:08.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.930+0000 7f03b6f49b80 1 Processor -- start 2023-07-29T16:25:08.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.930+0000 7f03b6f49b80 1 -- start start 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.930+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e895677c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.930+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895670650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e89566d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e895668fb0 con 0x55e895669530 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e895669120 con 0x55e89566a2f0 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e89565f930 con 0x55e895668be0 2023-07-29T16:25:08.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e895677c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a1b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e89566d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e895677c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57758/0 (socket says 172.21.15.31:57758) 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 -- 172.21.15.31:0/3939501858 learned_addr learned my addr 172.21.15.31:0/3939501858 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895670650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 -- 172.21.15.31:0/3939501858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 msgr2=0x55e89566d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e89566d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 -- 172.21.15.31:0/3939501858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 msgr2=0x55e895670650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895670650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 -- 172.21.15.31:0/3939501858 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e89540e250 con 0x55e895668be0 2023-07-29T16:25:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.931+0000 7f03a139b700 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e895677c60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0390003a90 tx=0x7f0390005aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96502228f4a74094 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.932+0000 7f0387fff700 1 -- 172.21.15.31:0/3939501858 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900159c0 con 0x55e895668be0 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.932+0000 7f0387fff700 1 -- 172.21.15.31:0/3939501858 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f039000f040 con 0x55e895668be0 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.932+0000 7f0387fff700 1 -- 172.21.15.31:0/3939501858 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390007e90 con 0x55e895668be0 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 -- 172.21.15.31:0/3939501858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 msgr2=0x55e895677c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e895677c60 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0390003a90 tx=0x7f0390005aa0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 -- 172.21.15.31:0/3939501858 shutdown_connections 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e89566d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895668be0 0x55e895677c60 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/3939501858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895669530 0x55e895670650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 -- 172.21.15.31:0/3939501858 >> 172.21.15.31:0/3939501858 conn(0x55e89565e3e0 msgr2=0x55e895664690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 -- 172.21.15.31:0/3939501858 shutdown_connections 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 -- 172.21.15.31:0/3939501858 wait complete. 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 -- start start 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.933+0000 7f03b6f49b80 1 Processor -- start 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895668be0 0x55e8956767f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895669530 0x55e895676d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e8956d0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e895677550 con 0x55e895668be0 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8956776c0 con 0x55e89566a2f0 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8956d0710 con 0x55e895669530 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03b6f49b80 10 client.0 _ll_register_callbacks cb 0x55e89555f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a139b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895668be0 0x55e8956767f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a1b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e8956d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a1b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e8956d0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39670/0 (socket says 172.21.15.31:39670) 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a1b9c700 1 -- 172.21.15.31:0/2985782872 learned_addr learned my addr 172.21.15.31:0/2985782872 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a0b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895669530 0x55e895676d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a139b700 1 -- 172.21.15.31:0/2985782872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895669530 msgr2=0x55e895676d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a139b700 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895669530 0x55e895676d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.934+0000 7f03a139b700 1 -- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 msgr2=0x55e8956d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f03a139b700 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e8956d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f03a139b700 1 -- 172.21.15.31:0/2985782872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e8953d0c60 con 0x55e895668be0 2023-07-29T16:25:08.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f03a1b9c700 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e8956d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:08.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f03a139b700 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895668be0 0x55e8956767f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0390007ae0 tx=0x7f0390009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a92be9285fea64 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f0384ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f0384ff9700 1 -- 172.21.15.31:0/2985782872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900159c0 con 0x55e895668be0 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f0384ff9700 1 -- 172.21.15.31:0/2985782872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0390015b20 con 0x55e895668be0 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.935+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0394019070 con 0x55e895668be0 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f0384ff9700 1 -- 172.21.15.31:0/2985782872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0390015ce0 con 0x55e895668be0 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 10 client.18303 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:08.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e8953e62b0 con 0x55e895668be0 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f0384ff9700 1 -- 172.21.15.31:0/2985782872 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0394019070 con 0x55e895668be0 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f0384ff9700 1 -- 172.21.15.31:0/2985782872 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f0390032030 con 0x55e895668be0 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 10 client.18303 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 10 client.18303 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 20 client.18303 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e8956d1c50 0x55e8956d4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 4 client.18303 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:25:08.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.936+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e8955b2d40 con 0x55e8956d1c50 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.937+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e8956d1c50 0x55e8956d4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.937+0000 7f03a0b9a700 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e8956d1c50 0x55e8956d4030 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b12f1c52531f2a49 server_cookie=46cea17e345f77a6 in_seq=0 out_seq=0 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.937+0000 7f0384ff9700 10 client.18303 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f0384ff9700 1 -- 172.21.15.31:0/2985782872 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e8955b2d40 con 0x55e8956d1c50 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f0384ff9700 10 client.18303 handle_command_reply: tid=0 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 client.18303 shutdown 2023-07-29T16:25:08.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895668be0 msgr2=0x55e8956767f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895668be0 0x55e8956767f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f0390007ae0 tx=0x7f0390009d90 comp rx=0 tx=0).stop 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 shutdown_connections 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e8956d1c50 0x55e8956d4030 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e89566a2f0 0x55e8956d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e895668be0 0x55e8956767f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 --2- 172.21.15.31:0/2985782872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e895669530 0x55e895676d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 >> 172.21.15.31:0/2985782872 conn(0x55e89565e3e0 msgr2=0x55e895670650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f0384ff9700 0 client.18303 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 shutdown_connections 2023-07-29T16:25:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 1 -- 172.21.15.31:0/2985782872 wait complete. 2023-07-29T16:25:08.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.940+0000 7f03b6f49b80 20 client.18303 trim_cache size 0 max 16384 2023-07-29T16:25:08.945 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:25:08.945 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:25:08.946 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:25:08.946 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:25:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f038c0a0810 msgr2=0x7f038c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f038c0a0810 0x7f038c0a2cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0394020000 tx=0x7f039401d000 comp rx=0 tx=0).stop 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 msgr2=0x7f039c19cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c19cc10 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f0388005910 tx=0x7f038800f690 comp rx=0 tx=0).stop 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 shutdown_connections 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f038c0a0810 0x7f038c0a2cc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f039c0ffef0 0x7f039c1a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f039c0febf0 0x7f039c19c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 --2- 172.21.15.31:0/2577189529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f039c0ff540 0x7f039c19cc10 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:08.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 >> 172.21.15.31:0/2577189529 conn(0x7f039c0fa840 msgr2=0x7f039c0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 shutdown_connections 2023-07-29T16:25:08.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:08.941+0000 7f03a35ff700 1 -- 172.21.15.31:0/2577189529 wait complete. 2023-07-29T16:25:08.960 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:25:08.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:25:09.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 -- 172.21.15.31:0/272533479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 msgr2=0x7fd254108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/272533479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254108780 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009f90 tx=0x7fd24c00b330 comp rx=0 tx=0).stop 2023-07-29T16:25:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 -- 172.21.15.31:0/272533479 shutdown_connections 2023-07-29T16:25:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/272533479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254108780 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/272533479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd254100340 0x7fd254100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/272533479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254101530 0x7fd254101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 -- 172.21.15.31:0/272533479 >> 172.21.15.31:0/272533479 conn(0x7fd2540fb920 msgr2=0x7fd2540fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 -- 172.21.15.31:0/272533479 shutdown_connections 2023-07-29T16:25:09.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.180+0000 7fd25c7ff700 1 -- 172.21.15.31:0/272533479 wait complete. 2023-07-29T16:25:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.181+0000 7fd25c7ff700 1 Processor -- start 2023-07-29T16:25:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 -- start start 2023-07-29T16:25:09.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 0x7fd254074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd254101530 0x7fd254073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd254073e20 con 0x7fd254101530 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd254073f90 con 0x7fd254101ec0 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25c7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2541a0190 con 0x7fd254100340 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25a59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 0x7fd254074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25a59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 0x7fd254074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57804/0 (socket says 172.21.15.31:57804) 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25a59b700 1 -- 172.21.15.31:0/3024259192 learned_addr learned my addr 172.21.15.31:0/3024259192 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd25ad9c700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.182+0000 7fd259d9a700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd254101530 0x7fd254073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25a59b700 1 -- 172.21.15.31:0/3024259192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 msgr2=0x7fd254073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25a59b700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25a59b700 1 -- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd254101530 msgr2=0x7fd254073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25a59b700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd254101530 0x7fd254073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25a59b700 1 -- 172.21.15.31:0/3024259192 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c009c40 con 0x7fd254100340 2023-07-29T16:25:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25ad9c700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25a59b700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 0x7fd254074b00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd25000e970 tx=0x7fd25000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250009790 con 0x7fd254100340 2023-07-29T16:25:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd25000c7c0 con 0x7fd254100340 2023-07-29T16:25:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250003c20 con 0x7fd254100340 2023-07-29T16:25:09.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2541a0470 con 0x7fd254100340 2023-07-29T16:25:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.183+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2541a0930 con 0x7fd254100340 2023-07-29T16:25:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.184+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd254107180 con 0x7fd254100340 2023-07-29T16:25:09.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.186+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd25000c930 con 0x7fd254100340 2023-07-29T16:25:09.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.188+0000 7fd24b7fe700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2400a07e0 0x7fd2400a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.188+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd250014070 con 0x7fd254100340 2023-07-29T16:25:09.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.188+0000 7fd259d9a700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2400a07e0 0x7fd2400a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.189+0000 7fd259d9a700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2400a07e0 0x7fd2400a2c90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd244005fd0 tx=0x7fd244009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.191+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fd2500959d0 con 0x7fd254100340 2023-07-29T16:25:09.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.495+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd25402cec0 con 0x7fd254100340 2023-07-29T16:25:09.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.496+0000 7fd24b7fe700 1 -- 172.21.15.31:0/3024259192 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10781 (secure 0 0 0) 0x7fd25001f340 con 0x7fd254100340 2023-07-29T16:25:09.496 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:25:09.497 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:24:56.527016+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:25:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2400a07e0 msgr2=0x7fd2400a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2400a07e0 0x7fd2400a2c90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd244005fd0 tx=0x7fd244009310 comp rx=0 tx=0).stop 2023-07-29T16:25:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 msgr2=0x7fd254074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 0x7fd254074b00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd25000e970 tx=0x7fd25000ec80 comp rx=0 tx=0).stop 2023-07-29T16:25:09.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 shutdown_connections 2023-07-29T16:25:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd2400a07e0 0x7fd2400a2c90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd254101ec0 0x7fd254073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd254100340 0x7fd254074b00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 --2- 172.21.15.31:0/3024259192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd254101530 0x7fd254073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 >> 172.21.15.31:0/3024259192 conn(0x7fd2540fb920 msgr2=0x7fd2540fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 shutdown_connections 2023-07-29T16:25:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.499+0000 7fd25c7ff700 1 -- 172.21.15.31:0/3024259192 wait complete. 2023-07-29T16:25:09.501 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2023-07-29T16:25:09.520 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:25:09.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 -- 172.21.15.31:0/2042437083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 msgr2=0x7f5684100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 --2- 172.21.15.31:0/2042437083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f5684100730 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5674009990 tx=0x7f5674009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 -- 172.21.15.31:0/2042437083 shutdown_connections 2023-07-29T16:25:09.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 --2- 172.21.15.31:0/2042437083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684101ea0 0x7f5684108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 --2- 172.21.15.31:0/2042437083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f5684100730 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 --2- 172.21.15.31:0/2042437083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5684101510 0x7f5684101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 -- 172.21.15.31:0/2042437083 >> 172.21.15.31:0/2042437083 conn(0x7f56840fb920 msgr2=0x7f56840fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 -- 172.21.15.31:0/2042437083 shutdown_connections 2023-07-29T16:25:09.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.742+0000 7f568a1ff700 1 -- 172.21.15.31:0/2042437083 wait complete. 2023-07-29T16:25:09.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.744+0000 7f568a1ff700 1 Processor -- start 2023-07-29T16:25:09.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.744+0000 7f568a1ff700 1 -- start start 2023-07-29T16:25:09.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.744+0000 7f568a1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f56841960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f568a1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684101510 0x7f5684196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f568a1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5684101ea0 0x7f568419d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f568a1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568419dc50 con 0x7f5684101510 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f568a1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568419dd90 con 0x7f5684100320 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f568a1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568419df20 con 0x7f5684101ea0 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f56841960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f56841960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39700/0 (socket says 172.21.15.31:39700) 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 -- 172.21.15.31:0/4072665933 learned_addr learned my addr 172.21.15.31:0/4072665933 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f567bfff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684101510 0x7f5684196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f5683fff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5684101ea0 0x7f568419d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 -- 172.21.15.31:0/4072665933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5684101ea0 msgr2=0x7f568419d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5684101ea0 0x7f568419d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 -- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684101510 msgr2=0x7f5684196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684101510 0x7f5684196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.745+0000 7f56837fe700 1 -- 172.21.15.31:0/4072665933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56740095f0 con 0x7f5684100320 2023-07-29T16:25:09.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.746+0000 7f56837fe700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f56841960d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f567400b620 tx=0x7f5674005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.746+0000 7f56817fa700 1 -- 172.21.15.31:0/4072665933 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567401d070 con 0x7f5684100320 2023-07-29T16:25:09.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.746+0000 7f56817fa700 1 -- 172.21.15.31:0/4072665933 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5674021410 con 0x7f5684100320 2023-07-29T16:25:09.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.746+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568419e1a0 con 0x7f5684100320 2023-07-29T16:25:09.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.746+0000 7f56817fa700 1 -- 172.21.15.31:0/4072665933 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5674017650 con 0x7f5684100320 2023-07-29T16:25:09.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.746+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568419e620 con 0x7f5684100320 2023-07-29T16:25:09.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.750+0000 7f56817fa700 1 -- 172.21.15.31:0/4072665933 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5674021580 con 0x7f5684100320 2023-07-29T16:25:09.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.751+0000 7f569dc16b80 1 Processor -- start 2023-07-29T16:25:09.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.751+0000 7f569dc16b80 1 -- start start 2023-07-29T16:25:09.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.751+0000 7f56817fa700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56700a0810 0x7f56700a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.751+0000 7f56817fa700 1 -- 172.21.15.31:0/4072665933 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f56740cce70 con 0x7f5684100320 2023-07-29T16:25:09.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 0x55902c207ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f5683fff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56700a0810 0x7f56700a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c208530 0x55902c208d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c209740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f569dc16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c15fa10 con 0x55902c2092f0 2023-07-29T16:25:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f569dc16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c15d3d0 con 0x55902c207be0 2023-07-29T16:25:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f569dc16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c158cc0 con 0x55902c208530 2023-07-29T16:25:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 0x55902c207ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f567bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c208530 0x55902c208d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 0x55902c207ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39702/0 (socket says 172.21.15.31:39702) 2023-07-29T16:25:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 -- 172.21.15.31:0/2648199661 learned_addr learned my addr 172.21.15.31:0/2648199661 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f5683fff700 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c209740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 -- 172.21.15.31:0/2648199661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c208530 msgr2=0x55902c208d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c208530 0x55902c208d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 -- 172.21.15.31:0/2648199661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 msgr2=0x55902c209740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c209740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f56837fe700 1 -- 172.21.15.31:0/2648199661 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902bf86de0 con 0x55902c207be0 2023-07-29T16:25:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.752+0000 7f567bfff700 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c208530 0x55902c208d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f56837fe700 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 0x55902c207ff0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5674096b00 tx=0x7f5674021c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5f70a5203b4c03 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f5683fff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56700a0810 0x7f56700a2cc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f567c009c90 tx=0x7f567c004320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f5679ffb700 1 -- 172.21.15.31:0/2648199661 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567409e630 con 0x55902c207be0 2023-07-29T16:25:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f5679ffb700 1 -- 172.21.15.31:0/2648199661 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56740a29c0 con 0x55902c207be0 2023-07-29T16:25:09.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f5679ffb700 1 -- 172.21.15.31:0/2648199661 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5674099d50 con 0x55902c207be0 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f569dc16b80 1 -- 172.21.15.31:0/2648199661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 msgr2=0x55902c207ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.753+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 0x55902c207ff0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5674096b00 tx=0x7f5674021c00 comp rx=0 tx=0).stop 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 -- 172.21.15.31:0/2648199661 shutdown_connections 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c209740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c207be0 0x55902c207ff0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2648199661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c208530 0x55902c208d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 -- 172.21.15.31:0/2648199661 >> 172.21.15.31:0/2648199661 conn(0x55902c1fd3e0 msgr2=0x55902c202990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 -- 172.21.15.31:0/2648199661 shutdown_connections 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 -- 172.21.15.31:0/2648199661 wait complete. 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.754+0000 7f569dc16b80 1 -- start start 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 Processor -- start 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c207be0 0x55902c215860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c215da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c2092f0 0x55902c21cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c0f5510 con 0x55902c207be0 2023-07-29T16:25:09.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c0f51b0 con 0x55902c208530 2023-07-29T16:25:09.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.755+0000 7f569dc16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902bfdc920 con 0x55902c2092f0 2023-07-29T16:25:09.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f569dc16b80 10 client.0 _ll_register_callbacks cb 0x55902c0fe620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:09.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f56837fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c207be0 0x55902c215860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f567bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c215da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c2092f0 0x55902c21cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f567bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c215da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39716/0 (socket says 172.21.15.31:39716) 2023-07-29T16:25:09.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f567bfff700 1 -- 172.21.15.31:0/2237334778 learned_addr learned my addr 172.21.15.31:0/2237334778 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:09.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 -- 172.21.15.31:0/2237334778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 msgr2=0x55902c215da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c215da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 -- 172.21.15.31:0/2237334778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c207be0 msgr2=0x55902c215860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c207be0 0x55902c215860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 -- 172.21.15.31:0/2237334778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c0098e0 con 0x55902c2092f0 2023-07-29T16:25:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f567bfff700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c215da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f56837fe700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c207be0 0x55902c215860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5683fff700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c2092f0 0x55902c21cde0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f567c004720 tx=0x7f567c004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f17cfe3ec726f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.756+0000 7f5662ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f5662ffd700 1 -- 172.21.15.31:0/2237334778 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567c01a670 con 0x55902c2092f0 2023-07-29T16:25:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f5662ffd700 1 -- 172.21.15.31:0/2237334778 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f567c01a7d0 con 0x55902c2092f0 2023-07-29T16:25:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55902c183d10 con 0x55902c2092f0 2023-07-29T16:25:09.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f5662ffd700 1 -- 172.21.15.31:0/2237334778 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f567c007e80 con 0x55902c2092f0 2023-07-29T16:25:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f569dc16b80 10 client.28399 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f568419e620 con 0x55902c2092f0 2023-07-29T16:25:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f5662ffd700 1 -- 172.21.15.31:0/2237334778 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55902c183d10 con 0x55902c2092f0 2023-07-29T16:25:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.757+0000 7f5662ffd700 1 -- 172.21.15.31:0/2237334778 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f567c025b40 con 0x55902c2092f0 2023-07-29T16:25:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f569dc16b80 10 client.28399 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:09.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f569dc16b80 10 client.28399 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f569dc16b80 20 client.28399 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c2215b0 0x55902c2239c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f569dc16b80 4 client.28399 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55902c1d6d70 con 0x55902c2215b0 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f56837fe700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c2215b0 0x55902c2239c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f5662ffd700 10 client.28399 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:09.771 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.758+0000 7f56837fe700 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c2215b0 0x55902c2239c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28db6cd89820ed4a server_cookie=2c08888e98591528 in_seq=0 out_seq=0 2023-07-29T16:25:09.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f5662ffd700 1 -- 172.21.15.31:0/2237334778 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55902c1d6d70 con 0x55902c2215b0 2023-07-29T16:25:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f5662ffd700 10 client.28399 handle_command_reply: tid=0 2023-07-29T16:25:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 client.28399 shutdown 2023-07-29T16:25:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c2092f0 msgr2=0x55902c21cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c2092f0 0x55902c21cde0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f567c004720 tx=0x7f567c004b40 comp rx=0 tx=0).stop 2023-07-29T16:25:09.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 shutdown_connections 2023-07-29T16:25:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c2092f0 0x55902c21cde0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c207be0 0x55902c215860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c2215b0 0x55902c2239c0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 --2- 172.21.15.31:0/2237334778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c215da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f5662ffd700 0 client.28399 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 >> 172.21.15.31:0/2237334778 conn(0x55902c1fd3e0 msgr2=0x55902c202060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.759+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 shutdown_connections 2023-07-29T16:25:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.760+0000 7f569dc16b80 1 -- 172.21.15.31:0/2237334778 wait complete. 2023-07-29T16:25:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.760+0000 7f569dc16b80 20 client.28399 trim_cache size 0 max 16384 2023-07-29T16:25:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 Processor -- start 2023-07-29T16:25:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 -- start start 2023-07-29T16:25:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c2096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c208530 0x55902c21cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c2092f0 0x55902c202900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c2153d0 con 0x55902c208530 2023-07-29T16:25:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.774+0000 7f569dc16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c1ff460 con 0x55902c2092f0 2023-07-29T16:25:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f569dc16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c2164e0 con 0x55902c207be0 2023-07-29T16:25:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c2096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c2096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57858/0 (socket says 172.21.15.31:57858) 2023-07-29T16:25:09.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 -- 172.21.15.31:0/3633681533 learned_addr learned my addr 172.21.15.31:0/3633681533 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f567bfff700 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c208530 0x55902c21cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f5683fff700 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c2092f0 0x55902c202900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 -- 172.21.15.31:0/3633681533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c2092f0 msgr2=0x55902c202900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c2092f0 0x55902c202900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 -- 172.21.15.31:0/3633681533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c208530 msgr2=0x55902c21cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c208530 0x55902c21cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 -- 172.21.15.31:0/3633681533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902bf6fc60 con 0x55902c207be0 2023-07-29T16:25:09.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f567bfff700 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c208530 0x55902c21cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.775+0000 7f56837fe700 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c2096c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f56740095c0 tx=0x7f5674009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=262970d93cba542b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f567a7fc700 1 -- 172.21.15.31:0/3633681533 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56740b8040 con 0x55902c207be0 2023-07-29T16:25:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f567a7fc700 1 -- 172.21.15.31:0/3633681533 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f567409aac0 con 0x55902c207be0 2023-07-29T16:25:09.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f567a7fc700 1 -- 172.21.15.31:0/3633681533 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567409e3d0 con 0x55902c207be0 2023-07-29T16:25:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 -- 172.21.15.31:0/3633681533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 msgr2=0x55902c2096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c2096c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f56740095c0 tx=0x7f5674009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 -- 172.21.15.31:0/3633681533 shutdown_connections 2023-07-29T16:25:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c2092f0 0x55902c202900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c2096c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 --2- 172.21.15.31:0/3633681533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c208530 0x55902c21cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 -- 172.21.15.31:0/3633681533 >> 172.21.15.31:0/3633681533 conn(0x55902c1fd3e0 msgr2=0x55902c2145c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 -- 172.21.15.31:0/3633681533 shutdown_connections 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.776+0000 7f569dc16b80 1 -- 172.21.15.31:0/3633681533 wait complete. 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.777+0000 7f569dc16b80 1 -- start start 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.777+0000 7f569dc16b80 1 Processor -- start 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c218d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c219280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c21c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c21c7e0 con 0x55902c2092f0 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c21c950 con 0x55902c208530 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902c21cb10 con 0x55902c207be0 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f569dc16b80 10 client.0 _ll_register_callbacks cb 0x55902c0fe620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f56837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c218d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f56837fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c218d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57872/0 (socket says 172.21.15.31:57872) 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f56837fe700 1 -- 172.21.15.31:0/493473063 learned_addr learned my addr 172.21.15.31:0/493473063 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f567bfff700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c219280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.778+0000 7f5683fff700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c21c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f56837fe700 1 -- 172.21.15.31:0/493473063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 msgr2=0x55902c219280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f56837fe700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c219280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f56837fe700 1 -- 172.21.15.31:0/493473063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 msgr2=0x55902c21c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f56837fe700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c21c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f56837fe700 1 -- 172.21.15.31:0/493473063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55902bf86de0 con 0x55902c207be0 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f56837fe700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c218d40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5674096380 tx=0x7f5674099f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ff0ea280377d238 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f5662ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f5662ffd700 1 -- 172.21.15.31:0/493473063 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56740b8040 con 0x55902c207be0 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f5662ffd700 1 -- 172.21.15.31:0/493473063 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f567409e170 con 0x55902c207be0 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56740a29c0 con 0x55902c207be0 2023-07-29T16:25:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f5662ffd700 1 -- 172.21.15.31:0/493473063 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56740a25b0 con 0x55902c207be0 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f569dc16b80 10 client.28405 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55902bfad250 con 0x55902c207be0 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.779+0000 7f5662ffd700 1 -- 172.21.15.31:0/493473063 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f567407f080 con 0x55902c207be0 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f5662ffd700 1 -- 172.21.15.31:0/493473063 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f5674099560 con 0x55902c207be0 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f569dc16b80 10 client.28405 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f569dc16b80 10 client.28405 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f569dc16b80 20 client.28405 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:09.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f569dc16b80 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c273d20 0x55902c276130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f569dc16b80 4 client.28405 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:25:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55902c151d40 con 0x55902c273d20 2023-07-29T16:25:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f567bfff700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c273d20 0x55902c276130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f567bfff700 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c273d20 0x55902c276130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f789ed71a263c901 server_cookie=1ed641b78d908879 in_seq=0 out_seq=0 2023-07-29T16:25:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.780+0000 7f5662ffd700 10 client.28405 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:09.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f5662ffd700 1 -- 172.21.15.31:0/493473063 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55902c151d40 con 0x55902c273d20 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f5662ffd700 10 client.28405 handle_command_reply: tid=0 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 client.28405 shutdown 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 msgr2=0x55902c218d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c218d40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5674096380 tx=0x7f5674099f60 comp rx=0 tx=0).stop 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 shutdown_connections 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55902c2092f0 0x55902c21c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55902c207be0 0x55902c218d40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55902c273d20 0x55902c276130 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f5662ffd700 0 client.28405 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 --2- 172.21.15.31:0/493473063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55902c208530 0x55902c219280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 >> 172.21.15.31:0/493473063 conn(0x55902c1fd3e0 msgr2=0x55902c20f640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 shutdown_connections 2023-07-29T16:25:09.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 1 -- 172.21.15.31:0/493473063 wait complete. 2023-07-29T16:25:09.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.781+0000 7f569dc16b80 20 client.28405 trim_cache size 0 max 16384 2023-07-29T16:25:09.797 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56700a0810 msgr2=0x7f56700a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56700a0810 0x7f56700a2cc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f567c009c90 tx=0x7f567c004320 comp rx=0 tx=0).stop 2023-07-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 msgr2=0x7f56841960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:09.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f56841960d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f567400b620 tx=0x7f5674005740 comp rx=0 tx=0).stop 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 shutdown_connections 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56700a0810 0x7f56700a2cc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5684101ea0 0x7f568419d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5684100320 0x7f56841960d0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 --2- 172.21.15.31:0/4072665933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5684101510 0x7f5684196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 >> 172.21.15.31:0/4072665933 conn(0x7f56840fb920 msgr2=0x7f56840fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:09.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 shutdown_connections 2023-07-29T16:25:09.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:09.782+0000 7f568a1ff700 1 -- 172.21.15.31:0/4072665933 wait complete. 2023-07-29T16:25:09.800 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:09 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3024259192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:25:09.801 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:25:09.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:09 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3024259192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:25:09.837 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:09 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3024259192' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:25:09.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 682176 8.83 MB/sec execute 294 sec latency 33.788 ms 2023-07-29T16:25:10.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 -- 172.21.15.31:0/3690023851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb01410c8f0 msgr2=0x7fb01410ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3690023851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb01410c8f0 0x7fb01410ecd0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00b0a0 tx=0x7fb00c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 -- 172.21.15.31:0/3690023851 shutdown_connections 2023-07-29T16:25:10.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3690023851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb01410c8f0 0x7fb01410ecd0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3690023851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 0x7fb0140731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3690023851 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0140737b0 0x7fb014073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 -- 172.21.15.31:0/3690023851 >> 172.21.15.31:0/3690023851 conn(0x7fb0140fb920 msgr2=0x7fb0140fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 -- 172.21.15.31:0/3690023851 shutdown_connections 2023-07-29T16:25:10.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.029+0000 7fb0195ff700 1 -- 172.21.15.31:0/3690023851 wait complete. 2023-07-29T16:25:10.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.030+0000 7fb0195ff700 1 Processor -- start 2023-07-29T16:25:10.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 -- start start 2023-07-29T16:25:10.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 0x7fb01419c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 0x7fb01419cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01410c8f0 0x7fb0141a3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0141a4350 con 0x7fb014074d90 2023-07-29T16:25:10.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0141a4490 con 0x7fb0140737b0 2023-07-29T16:25:10.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0195ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0141a4620 con 0x7fb01410c8f0 2023-07-29T16:25:10.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb012ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 0x7fb01419c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb012ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 0x7fb01419c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47476/0 (socket says 172.21.15.31:47476) 2023-07-29T16:25:10.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb012ffd700 1 -- 172.21.15.31:0/3907971410 learned_addr learned my addr 172.21.15.31:0/3907971410 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0127fc700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 0x7fb01419cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb0137fe700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01410c8f0 0x7fb0141a3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb012ffd700 1 -- 172.21.15.31:0/3907971410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01410c8f0 msgr2=0x7fb0141a3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.031+0000 7fb012ffd700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01410c8f0 0x7fb0141a3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb012ffd700 1 -- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 msgr2=0x7fb01419cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb012ffd700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 0x7fb01419cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb012ffd700 1 -- 172.21.15.31:0/3907971410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c009d00 con 0x7fb0140737b0 2023-07-29T16:25:10.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb0127fc700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 0x7fb01419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:10.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb012ffd700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 0x7fb01419c7d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb00400e970 tx=0x7fb00400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7faffbfff700 1 -- 172.21.15.31:0/3907971410 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004009790 con 0x7fb0140737b0 2023-07-29T16:25:10.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7faffbfff700 1 -- 172.21.15.31:0/3907971410 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb004004cf0 con 0x7fb0140737b0 2023-07-29T16:25:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7faffbfff700 1 -- 172.21.15.31:0/3907971410 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004005680 con 0x7fb0140737b0 2023-07-29T16:25:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0141a4900 con 0x7fb0140737b0 2023-07-29T16:25:10.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.032+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0141a4dc0 con 0x7fb0140737b0 2023-07-29T16:25:10.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.036+0000 7faffbfff700 1 -- 172.21.15.31:0/3907971410 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb004004750 con 0x7fb0140737b0 2023-07-29T16:25:10.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 Processor -- start 2023-07-29T16:25:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 -- start start 2023-07-29T16:25:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b720be0 0x56358b720ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7faffbfff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faffc0a0790 0x7faffc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b721d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b722740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b678a10 con 0x56358b7222f0 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b6763d0 con 0x56358b720be0 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.037+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b671cc0 con 0x56358b721530 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7faffbfff700 1 -- 172.21.15.31:0/3907971410 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb004014070 con 0x7fb0140737b0 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b721d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b721d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38662/0 (socket says 172.21.15.31:38662) 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 -- 172.21.15.31:0/2979097277 learned_addr learned my addr 172.21.15.31:0/2979097277 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b720be0 0x56358b720ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb012ffd700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faffc0a0790 0x7faffc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 -- 172.21.15.31:0/2979097277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b720be0 msgr2=0x56358b720ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b720be0 0x56358b720ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b722740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 -- 172.21.15.31:0/2979097277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 msgr2=0x56358b722740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b722740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.038+0000 7fb0127fc700 1 -- 172.21.15.31:0/2979097277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358b49fde0 con 0x56358b721530 2023-07-29T16:25:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.039+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b721d60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb008009d60 tx=0x7fb0080116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a8337f3ef91be08 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.039+0000 7fb012ffd700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faffc0a0790 0x7faffc0a2c40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb004096010 tx=0x7fb00400cbd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.039+0000 7faff97fa700 1 -- 172.21.15.31:0/2979097277 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008011e20 con 0x56358b721530 2023-07-29T16:25:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.039+0000 7faff97fa700 1 -- 172.21.15.31:0/2979097277 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb008012910 con 0x56358b721530 2023-07-29T16:25:10.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.039+0000 7faff97fa700 1 -- 172.21.15.31:0/2979097277 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800f840 con 0x56358b721530 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2979097277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 msgr2=0x56358b721d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b721d60 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb008009d60 tx=0x7fb0080116c0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2979097277 shutdown_connections 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b722740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b720be0 0x56358b720ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2979097277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b721d60 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2979097277 >> 172.21.15.31:0/2979097277 conn(0x56358b7163e0 msgr2=0x56358b71b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2979097277 shutdown_connections 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2979097277 wait complete. 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.040+0000 7fb02ce8eb80 1 -- start start 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 Processor -- start 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b72a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b72ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b7222f0 0x56358b731d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b60e510 con 0x56358b720be0 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b60e1b0 con 0x56358b7222f0 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b4f5920 con 0x56358b721530 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb02ce8eb80 10 client.0 _ll_register_callbacks cb 0x56358b617620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb0127fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b72ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb0127fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b72ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38670/0 (socket says 172.21.15.31:38670) 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.041+0000 7fb0127fc700 1 -- 172.21.15.31:0/2849110597 learned_addr learned my addr 172.21.15.31:0/2849110597 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b72a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b7222f0 0x56358b731d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0127fc700 1 -- 172.21.15.31:0/2849110597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b7222f0 msgr2=0x56358b731d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b7222f0 0x56358b731d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0127fc700 1 -- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 msgr2=0x56358b72a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b72a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0127fc700 1 -- 172.21.15.31:0/2849110597 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb008009990 con 0x56358b721530 2023-07-29T16:25:10.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b72ad40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb008009d30 tx=0x7fb008012df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e95028d8f6169b2f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.042+0000 7faff6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7faff6ff5700 1 -- 172.21.15.31:0/2849110597 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008012760 con 0x56358b721530 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7faff6ff5700 1 -- 172.21.15.31:0/2849110597 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb008005070 con 0x56358b721530 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56358b69cd10 con 0x56358b721530 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7faff6ff5700 1 -- 172.21.15.31:0/2849110597 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb008005230 con 0x56358b721530 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 10 client.28417 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56358b488c60 con 0x56358b721530 2023-07-29T16:25:10.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7faff6ff5700 1 -- 172.21.15.31:0/2849110597 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56358b69cd10 con 0x56358b721530 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7faff6ff5700 1 -- 172.21.15.31:0/2849110597 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fb008022070 con 0x56358b721530 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 10 client.28417 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 10 client.28417 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 20 client.28417 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.043+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b7363f0 0x56358b738800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.044+0000 7fb02ce8eb80 4 client.28417 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:10.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.044+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56358b6efd70 con 0x56358b7363f0 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.044+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b7363f0 0x56358b738800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.044+0000 7faff6ff5700 10 client.28417 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.044+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b7363f0 0x56358b738800 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=350b28cd8b1bad9c server_cookie=e98417f69875d26b in_seq=0 out_seq=0 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7faff6ff5700 1 -- 172.21.15.31:0/2849110597 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x7fb00c02a5f0 con 0x56358b7363f0 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7faff6ff5700 10 client.28417 handle_command_reply: tid=0 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 client.28417 shutdown 2023-07-29T16:25:10.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 msgr2=0x56358b72ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b72ad40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb008009d30 tx=0x7fb008012df0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 shutdown_connections 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b7363f0 0x56358b738800 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b7222f0 0x56358b731d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b72a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2849110597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b721530 0x56358b72ad40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7faff6ff5700 0 client.28417 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:10.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 >> 172.21.15.31:0/2849110597 conn(0x56358b7163e0 msgr2=0x56358b71b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 shutdown_connections 2023-07-29T16:25:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.045+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2849110597 wait complete. 2023-07-29T16:25:10.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.046+0000 7fb02ce8eb80 20 client.28417 trim_cache size 0 max 16384 2023-07-29T16:25:10.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 Processor -- start 2023-07-29T16:25:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 -- start start 2023-07-29T16:25:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 0x56358b72fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7287a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b7252d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b718460 con 0x56358b7222f0 2023-07-29T16:25:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b72b7f0 con 0x56358b721530 2023-07-29T16:25:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.060+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b713770 con 0x56358b720be0 2023-07-29T16:25:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 0x56358b72fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0127fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7287a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0127fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7287a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47510/0 (socket says 172.21.15.31:47510) 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0127fc700 1 -- 172.21.15.31:0/2848059849 learned_addr learned my addr 172.21.15.31:0/2848059849 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 0x56358b72fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38684/0 (socket says 172.21.15.31:38684) 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b7252d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 -- 172.21.15.31:0/2848059849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 msgr2=0x56358b7287a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7287a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 -- 172.21.15.31:0/2848059849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 msgr2=0x56358b7252d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b7252d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 -- 172.21.15.31:0/2848059849 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358b4c6250 con 0x56358b720be0 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0127fc700 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7287a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.061+0000 7fb0137fe700 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 0x56358b72fd30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009fd0 tx=0x7fb00c00b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d755dbab2e43956 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7faff9ffb700 1 -- 172.21.15.31:0/2848059849 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c00e040 con 0x56358b720be0 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7faff9ffb700 1 -- 172.21.15.31:0/2848059849 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb00c022070 con 0x56358b720be0 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7faff9ffb700 1 -- 172.21.15.31:0/2848059849 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c004ea0 con 0x56358b720be0 2023-07-29T16:25:10.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2848059849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 msgr2=0x56358b72fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 0x56358b72fd30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb00c009fd0 tx=0x7fb00c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2848059849 shutdown_connections 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b7222f0 0x56358b7252d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b720be0 0x56358b72fd30 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/2848059849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7287a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2848059849 >> 172.21.15.31:0/2848059849 conn(0x56358b7163e0 msgr2=0x56358b7294e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2848059849 shutdown_connections 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.062+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/2848059849 wait complete. 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.063+0000 7fb02ce8eb80 1 -- start start 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.063+0000 7fb02ce8eb80 1 Processor -- start 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b780470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7809b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b7222f0 0x56358b7879f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b787f30 con 0x56358b720be0 2023-07-29T16:25:10.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b788070 con 0x56358b721530 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358b788200 con 0x56358b7222f0 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb02ce8eb80 10 client.0 _ll_register_callbacks cb 0x56358b617620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b780470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b780470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44090/0 (socket says 172.21.15.31:44090) 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0137fe700 1 -- 172.21.15.31:0/14772888 learned_addr learned my addr 172.21.15.31:0/14772888 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0127fc700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7809b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0137fe700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b7222f0 0x56358b7879f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0127fc700 1 -- 172.21.15.31:0/14772888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b7222f0 msgr2=0x56358b7879f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0127fc700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b7222f0 0x56358b7879f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0127fc700 1 -- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 msgr2=0x56358b780470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0127fc700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b780470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0127fc700 1 -- 172.21.15.31:0/14772888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56358b488c60 con 0x56358b721530 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.064+0000 7fb0137fe700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b780470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7fb0127fc700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7809b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb008000c00 tx=0x7fb0080124c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92f53fa87a7a027e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7faff6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7faff6ff5700 1 -- 172.21.15.31:0/14772888 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008012760 con 0x56358b721530 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb008012910 con 0x56358b721530 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7faff6ff5700 1 -- 172.21.15.31:0/14772888 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb008000e40 con 0x56358b721530 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7faff6ff5700 1 -- 172.21.15.31:0/14772888 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb008009260 con 0x56358b721530 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7fb02ce8eb80 10 client.28418 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.065+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fb01410f680 con 0x56358b721530 2023-07-29T16:25:10.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7faff6ff5700 1 -- 172.21.15.31:0/14772888 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb008012910 con 0x56358b721530 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7faff6ff5700 1 -- 172.21.15.31:0/14772888 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fb00803d070 con 0x56358b721530 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb02ce8eb80 10 client.28418 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb02ce8eb80 10 client.28418 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb02ce8eb80 20 client.28418 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b789780 0x56358b78bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb02ce8eb80 4 client.28418 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56358b66ad40 con 0x56358b789780 2023-07-29T16:25:10.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.066+0000 7fb0137fe700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b789780 0x56358b78bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb0137fe700 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b789780 0x56358b78bb90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12b6239c1cdf73f0 server_cookie=c6abccc3d7749749 in_seq=0 out_seq=0 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7faff6ff5700 10 client.28418 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7faff6ff5700 1 -- 172.21.15.31:0/14772888 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56358b66ad40 con 0x56358b789780 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7faff6ff5700 10 client.28418 handle_command_reply: tid=0 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 client.28418 shutdown 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 msgr2=0x56358b7809b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7809b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb008000c00 tx=0x7fb0080124c0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 shutdown_connections 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56358b789780 0x56358b78bb90 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56358b7222f0 0x56358b7879f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56358b720be0 0x56358b780470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 --2- 172.21.15.31:0/14772888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56358b721530 0x56358b7809b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7faff6ff5700 0 client.28418 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:10.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 >> 172.21.15.31:0/14772888 conn(0x56358b7163e0 msgr2=0x56358b7287a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 shutdown_connections 2023-07-29T16:25:10.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.067+0000 7fb02ce8eb80 1 -- 172.21.15.31:0/14772888 wait complete. 2023-07-29T16:25:10.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb02ce8eb80 20 client.28418 trim_cache size 0 max 16384 2023-07-29T16:25:10.080 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:25:10.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faffc0a0790 msgr2=0x7faffc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faffc0a0790 0x7faffc0a2c40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb004096010 tx=0x7fb00400cbd0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 msgr2=0x7fb01419c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 0x7fb01419c7d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb00400e970 tx=0x7fb00400ed30 comp rx=0 tx=0).stop 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 shutdown_connections 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7faffc0a0790 0x7faffc0a2c40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb01410c8f0 0x7fb0141a3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0140737b0 0x7fb01419c7d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 --2- 172.21.15.31:0/3907971410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb014074d90 0x7fb01419cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.068+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 >> 172.21.15.31:0/3907971410 conn(0x7fb0140fb920 msgr2=0x7fb0140fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.069+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 shutdown_connections 2023-07-29T16:25:10.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.069+0000 7fb0195ff700 1 -- 172.21.15.31:0/3907971410 wait complete. 2023-07-29T16:25:10.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:25:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 -- 172.21.15.31:0/1078611140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 msgr2=0x7f49000731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 --2- 172.21.15.31:0/1078611140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f49000731e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f48f00098f0 tx=0x7f48f0009c00 comp rx=0 tx=0).stop 2023-07-29T16:25:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 -- 172.21.15.31:0/1078611140 shutdown_connections 2023-07-29T16:25:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 --2- 172.21.15.31:0/1078611140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f490010c900 0x7f490010ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 --2- 172.21.15.31:0/1078611140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f49000731e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 --2- 172.21.15.31:0/1078611140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49000737b0 0x7f4900073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 -- 172.21.15.31:0/1078611140 >> 172.21.15.31:0/1078611140 conn(0x7f49000fb970 msgr2=0x7f49000fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 -- 172.21.15.31:0/1078611140 shutdown_connections 2023-07-29T16:25:10.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.314+0000 7f49065ff700 1 -- 172.21.15.31:0/1078611140 wait complete. 2023-07-29T16:25:10.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 Processor -- start 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 -- start start 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49000737b0 0x7f490019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f490019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490010c900 0x7f49001a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001a42e0 con 0x7f490010c900 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001a4420 con 0x7f4900074d90 2023-07-29T16:25:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.315+0000 7f49065ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49001a45b0 con 0x7f49000737b0 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f490019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f490019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47538/0 (socket says 172.21.15.31:47538) 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48fffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49000737b0 0x7f490019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 -- 172.21.15.31:0/873593922 learned_addr learned my addr 172.21.15.31:0/873593922 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f4904b9c700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490010c900 0x7f49001a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 -- 172.21.15.31:0/873593922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49000737b0 msgr2=0x7f490019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49000737b0 0x7f490019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 -- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490010c900 msgr2=0x7f49001a3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490010c900 0x7f49001a3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 -- 172.21.15.31:0/873593922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f0009520 con 0x7f4900074d90 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48fffff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49000737b0 0x7f490019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.316+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f490019cca0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f48f400e970 tx=0x7f48f400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.317+0000 7f48fd7fa700 1 -- 172.21.15.31:0/873593922 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4009790 con 0x7f4900074d90 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.317+0000 7f48fd7fa700 1 -- 172.21.15.31:0/873593922 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f400c7e0 con 0x7f4900074d90 2023-07-29T16:25:10.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.317+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49001a4890 con 0x7f4900074d90 2023-07-29T16:25:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.317+0000 7f48fd7fa700 1 -- 172.21.15.31:0/873593922 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4003c70 con 0x7f4900074d90 2023-07-29T16:25:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.317+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49001a4df0 con 0x7f4900074d90 2023-07-29T16:25:10.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.320+0000 7f48fd7fa700 1 -- 172.21.15.31:0/873593922 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f48f400cc50 con 0x7f4900074d90 2023-07-29T16:25:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.322+0000 7f4919febb80 1 Processor -- start 2023-07-29T16:25:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.322+0000 7f4919febb80 1 -- start start 2023-07-29T16:25:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4919febb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4806ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4919febb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4807d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4919febb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 0x55f2e4808740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4919febb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e475ea10 con 0x55f2e4807530 2023-07-29T16:25:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4919febb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e475c3d0 con 0x55f2e4806be0 2023-07-29T16:25:10.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4919febb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e4757cc0 con 0x55f2e48082f0 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48fd7fa700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48e80a0810 0x7f48e80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 0x55f2e4808740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48fffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4807d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 0x55f2e4808740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38736/0 (socket says 172.21.15.31:38736) 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48ff7fe700 1 -- 172.21.15.31:0/1868677616 learned_addr learned my addr 172.21.15.31:0/1868677616 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4904b9c700 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4806ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48fd7fa700 1 -- 172.21.15.31:0/873593922 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f48f4014070 con 0x7f4900074d90 2023-07-29T16:25:10.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f4904b9c700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48e80a0810 0x7f48e80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48ff7fe700 1 -- 172.21.15.31:0/1868677616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 msgr2=0x55f2e4806ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.323+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4806ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48ff7fe700 1 -- 172.21.15.31:0/1868677616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 msgr2=0x55f2e4807d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4807d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48ff7fe700 1 -- 172.21.15.31:0/1868677616 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e4585de0 con 0x55f2e48082f0 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 0x55f2e4808740 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f48f4095180 tx=0x7f48f4003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b1575aa0f6bc7a3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f4904b9c700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48e80a0810 0x7f48e80a2cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x55f2e4806980 tx=0x7f48f800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48e67fc700 1 -- 172.21.15.31:0/1868677616 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40c7040 con 0x55f2e48082f0 2023-07-29T16:25:10.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48e67fc700 1 -- 172.21.15.31:0/1868677616 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f4099400 con 0x55f2e48082f0 2023-07-29T16:25:10.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.324+0000 7f48e67fc700 1 -- 172.21.15.31:0/1868677616 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4099660 con 0x55f2e48082f0 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 -- 172.21.15.31:0/1868677616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 msgr2=0x55f2e4808740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 0x55f2e4808740 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f48f4095180 tx=0x7f48f4003680 comp rx=0 tx=0).stop 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 -- 172.21.15.31:0/1868677616 shutdown_connections 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e48082f0 0x55f2e4808740 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4806ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 --2- 172.21.15.31:0/1868677616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4807d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 -- 172.21.15.31:0/1868677616 >> 172.21.15.31:0/1868677616 conn(0x55f2e47fc3e0 msgr2=0x55f2e4801990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 -- 172.21.15.31:0/1868677616 shutdown_connections 2023-07-29T16:25:10.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.325+0000 7f4919febb80 1 -- 172.21.15.31:0/1868677616 wait complete. 2023-07-29T16:25:10.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.326+0000 7f4919febb80 1 -- start start 2023-07-29T16:25:10.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.326+0000 7f4919febb80 1 Processor -- start 2023-07-29T16:25:10.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e4810820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4810d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e4817da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e46f4510 con 0x55f2e4807530 2023-07-29T16:25:10.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e46f41b0 con 0x55f2e48082f0 2023-07-29T16:25:10.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e45db920 con 0x55f2e4806be0 2023-07-29T16:25:10.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4919febb80 10 client.0 _ll_register_callbacks cb 0x55f2e46fd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4810d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4810d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44116/0 (socket says 172.21.15.31:44116) 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2641058664 learned_addr learned my addr 172.21.15.31:0/2641058664 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f4904b9c700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e4817da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.327+0000 7f48fffff700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e4810820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2641058664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 msgr2=0x55f2e4810820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e4810820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 msgr2=0x55f2e4817da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e4817da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2641058664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49001a4df0 con 0x55f2e4807530 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48fffff700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e4810820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4810d60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f48f4099390 tx=0x7f48f40c9700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f71cdd15373102 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48df7fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:25:10.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48df7fe700 1 -- 172.21.15.31:0/2641058664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40c98b0 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2e4782d10 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48df7fe700 1 -- 172.21.15.31:0/2641058664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f48f40a1850 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f48df7fe700 1 -- 172.21.15.31:0/2641058664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f4098b00 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f4919febb80 10 client.18354 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.328+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f2e456ec60 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f48df7fe700 1 -- 172.21.15.31:0/2641058664 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f407c080 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f48df7fe700 1 -- 172.21.15.31:0/2641058664 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f48f40a13c0 con 0x55f2e4807530 2023-07-29T16:25:10.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f4919febb80 10 client.18354 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f4919febb80 10 client.18354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f4919febb80 20 client.18354 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f4919febb80 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e481c240 0x55f2e481e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f4919febb80 4 client.18354 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2e47d5d70 con 0x55f2e481c240 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.329+0000 7f48fffff700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e481c240 0x55f2e481e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.330+0000 7f48fffff700 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e481c240 0x55f2e481e650 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24d029392947d26 server_cookie=1a979a8b43ea790 in_seq=0 out_seq=0 2023-07-29T16:25:10.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.330+0000 7f48df7fe700 10 client.18354 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:10.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.331+0000 7f48df7fe700 1 -- 172.21.15.31:0/2641058664 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f2e47d5d70 con 0x55f2e481c240 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.331+0000 7f48df7fe700 10 client.18354 handle_command_reply: tid=0 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.331+0000 7f4919febb80 1 client.18354 shutdown 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 msgr2=0x55f2e4810d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4810d60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f48f4099390 tx=0x7f48f40c9700 comp rx=0 tx=0).stop 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 shutdown_connections 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e481c240 0x55f2e481e650 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e4817da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e4810820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f48df7fe700 0 client.18354 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 --2- 172.21.15.31:0/2641058664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4810d60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 >> 172.21.15.31:0/2641058664 conn(0x55f2e47fc3e0 msgr2=0x55f2e4801060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 shutdown_connections 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 1 -- 172.21.15.31:0/2641058664 wait complete. 2023-07-29T16:25:10.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.332+0000 7f4919febb80 20 client.18354 trim_cache size 0 max 16384 2023-07-29T16:25:10.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.348+0000 7f4919febb80 1 Processor -- start 2023-07-29T16:25:10.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.348+0000 7f4919febb80 1 -- start start 2023-07-29T16:25:10.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4919febb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e481e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4919febb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4815d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4919febb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e48086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4919febb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e47fe480 con 0x55f2e4807530 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4919febb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e4802cc0 con 0x55f2e48082f0 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4919febb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e4802690 con 0x55f2e4806be0 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f48fffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e481e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4815d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f48fffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e481e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38742/0 (socket says 172.21.15.31:38742) 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f48fffff700 1 -- 172.21.15.31:0/2571010344 learned_addr learned my addr 172.21.15.31:0/2571010344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.349+0000 7f4904b9c700 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e48086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2571010344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 msgr2=0x55f2e481e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e481e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2571010344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 msgr2=0x55f2e48086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e48086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2571010344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e45ac250 con 0x55f2e4807530 2023-07-29T16:25:10.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f4904b9c700 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e48086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:10.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4815d50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f48f4004800 tx=0x7f48f4003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3107bcd1246171 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48e6ffd700 1 -- 172.21.15.31:0/2571010344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4098d60 con 0x55f2e4807530 2023-07-29T16:25:10.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48e6ffd700 1 -- 172.21.15.31:0/2571010344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f2e4782d10 con 0x55f2e4807530 2023-07-29T16:25:10.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.350+0000 7f48e6ffd700 1 -- 172.21.15.31:0/2571010344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f40a13c0 con 0x55f2e4807530 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 -- 172.21.15.31:0/2571010344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 msgr2=0x55f2e4815d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4815d50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f48f4004800 tx=0x7f48f4003680 comp rx=0 tx=0).stop 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 -- 172.21.15.31:0/2571010344 shutdown_connections 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e48082f0 0x55f2e48086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4806be0 0x55f2e481e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 --2- 172.21.15.31:0/2571010344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e4807530 0x55f2e4815d50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 -- 172.21.15.31:0/2571010344 >> 172.21.15.31:0/2571010344 conn(0x55f2e47fc3e0 msgr2=0x55f2e480f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 -- 172.21.15.31:0/2571010344 shutdown_connections 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.351+0000 7f4919febb80 1 -- 172.21.15.31:0/2571010344 wait complete. 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.352+0000 7f4919febb80 1 -- start start 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.352+0000 7f4919febb80 1 Processor -- start 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4813740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4807530 0x55f2e4813c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e48082f0 0x55f2e4814c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e48151d0 con 0x55f2e48082f0 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e4815340 con 0x55f2e4806be0 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e4815500 con 0x55f2e4807530 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4919febb80 10 client.0 _ll_register_callbacks cb 0x55f2e46fd620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4807530 0x55f2e4813c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4813740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4807530 0x55f2e4813c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38750/0 (socket says 172.21.15.31:38750) 2023-07-29T16:25:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48ff7fe700 1 -- 172.21.15.31:0/2436078469 learned_addr learned my addr 172.21.15.31:0/2436078469 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4904b9c700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e48082f0 0x55f2e4814c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48fffff700 1 -- 172.21.15.31:0/2436078469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4807530 msgr2=0x55f2e4813c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48fffff700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4807530 0x55f2e4813c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48fffff700 1 -- 172.21.15.31:0/2436078469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e48082f0 msgr2=0x55f2e4814c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48fffff700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e48082f0 0x55f2e4814c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f48fffff700 1 -- 172.21.15.31:0/2436078469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f8003a00 con 0x55f2e4806be0 2023-07-29T16:25:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.353+0000 7f4904b9c700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e48082f0 0x55f2e4814c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f48fffff700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4813740 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f48f000fa40 tx=0x7f48f00271b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29acf5828da16d45 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f48df7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f48df7fe700 1 -- 172.21.15.31:0/2436078469 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f001c040 con 0x55f2e4806be0 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f48df7fe700 1 -- 172.21.15.31:0/2436078469 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f48f0027d50 con 0x55f2e4806be0 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f4099400 con 0x55f2e4806be0 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f48df7fe700 1 -- 172.21.15.31:0/2436078469 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f0027490 con 0x55f2e4806be0 2023-07-29T16:25:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f4919febb80 10 client.28436 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f2e456ec60 con 0x55f2e4806be0 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.354+0000 7f48df7fe700 1 -- 172.21.15.31:0/2436078469 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f4099400 con 0x55f2e4806be0 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f48df7fe700 1 -- 172.21.15.31:0/2436078469 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f48f0009920 con 0x55f2e4806be0 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f4919febb80 10 client.28436 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f4919febb80 10 client.28436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f4919febb80 20 client.28436 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:10.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f4919febb80 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e486ea70 0x55f2e4870e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:10.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f4919febb80 4 client.28436 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:25:10.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f2e4750d40 con 0x55f2e486ea70 2023-07-29T16:25:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e486ea70 0x55f2e4870e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f48ff7fe700 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e486ea70 0x55f2e4870e80 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760e2dc1e185416b server_cookie=2c4344b5379b2c05 in_seq=0 out_seq=0 2023-07-29T16:25:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.355+0000 7f48df7fe700 10 client.28436 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f48df7fe700 1 -- 172.21.15.31:0/2436078469 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f2e4750d40 con 0x55f2e486ea70 2023-07-29T16:25:10.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f48df7fe700 10 client.28436 handle_command_reply: tid=0 2023-07-29T16:25:10.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 client.28436 shutdown 2023-07-29T16:25:10.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 msgr2=0x55f2e4813740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4813740 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f48f000fa40 tx=0x7f48f00271b0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 shutdown_connections 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f2e486ea70 0x55f2e4870e80 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f2e48082f0 0x55f2e4814c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f2e4806be0 0x55f2e4813740 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 --2- 172.21.15.31:0/2436078469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f2e4807530 0x55f2e4813c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f48df7fe700 0 client.28436 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 >> 172.21.15.31:0/2436078469 conn(0x55f2e47fc3e0 msgr2=0x55f2e480e740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 shutdown_connections 2023-07-29T16:25:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.357+0000 7f4919febb80 1 -- 172.21.15.31:0/2436078469 wait complete. 2023-07-29T16:25:10.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f4919febb80 20 client.28436 trim_cache size 0 max 16384 2023-07-29T16:25:10.365 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:25:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48e80a0810 msgr2=0x7f48e80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48e80a0810 0x7f48e80a2cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x55f2e4806980 tx=0x7f48f800b040 comp rx=0 tx=0).stop 2023-07-29T16:25:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 msgr2=0x7f490019cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f490019cca0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f48f400e970 tx=0x7f48f400ec80 comp rx=0 tx=0).stop 2023-07-29T16:25:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 shutdown_connections 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f48e80a0810 0x7f48e80a2cc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f490010c900 0x7f49001a3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.358+0000 7f49065ff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49000737b0 0x7f490019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.359+0000 7f49065ff700 1 --2- 172.21.15.31:0/873593922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4900074d90 0x7f490019cca0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.359+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 >> 172.21.15.31:0/873593922 conn(0x7f49000fb970 msgr2=0x7f49000fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.359+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 shutdown_connections 2023-07-29T16:25:10.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:10.359+0000 7f49065ff700 1 -- 172.21.15.31:0/873593922 wait complete. 2023-07-29T16:25:10.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:10 smithi031 ceph-mon[95504]: pgmap v662: 97 pgs: 97 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 297 op/s 2023-07-29T16:25:10.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:10 smithi156 ceph-mon[129245]: pgmap v662: 97 pgs: 97 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 297 op/s 2023-07-29T16:25:10.838 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:10 smithi039 ceph-mon[128858]: pgmap v662: 97 pgs: 97 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 297 op/s 2023-07-29T16:25:10.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 683749 8.82 MB/sec execute 295 sec latency 17.260 ms 2023-07-29T16:25:11.378 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:25:11.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:25:11.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 -- 172.21.15.31:0/1232103889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c1036c0 msgr2=0x7fa77c105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 --2- 172.21.15.31:0/1232103889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c1036c0 0x7fa77c105aa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa770009990 tx=0x7fa770009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 -- 172.21.15.31:0/1232103889 shutdown_connections 2023-07-29T16:25:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 --2- 172.21.15.31:0/1232103889 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 0x7fa77c108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 --2- 172.21.15.31:0/1232103889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa77c100da0 0x7fa77c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 --2- 172.21.15.31:0/1232103889 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c1036c0 0x7fa77c105aa0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 -- 172.21.15.31:0/1232103889 >> 172.21.15.31:0/1232103889 conn(0x7fa77c0fa840 msgr2=0x7fa77c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 -- 172.21.15.31:0/1232103889 shutdown_connections 2023-07-29T16:25:11.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.615+0000 7fa7829ff700 1 -- 172.21.15.31:0/1232103889 wait complete. 2023-07-29T16:25:11.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.616+0000 7fa7829ff700 1 Processor -- start 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 -- start start 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c100da0 0x7fa77c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa77c1036c0 0x7fa77c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 0x7fa77c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c19fe30 con 0x7fa77c1036c0 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c19ff70 con 0x7fa77c100da0 2023-07-29T16:25:11.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa7829ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c1a0100 con 0x7fa77c105fe0 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa77bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c100da0 0x7fa77c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa77c1036c0 0x7fa77c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa780f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 0x7fa77c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa77bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c100da0 0x7fa77c1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47610/0 (socket says 172.21.15.31:47610) 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.617+0000 7fa77bfff700 1 -- 172.21.15.31:0/3107675023 learned_addr learned my addr 172.21.15.31:0/3107675023 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa780f9c700 1 -- 172.21.15.31:0/3107675023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c100da0 msgr2=0x7fa77c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa780f9c700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c100da0 0x7fa77c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa780f9c700 1 -- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa77c1036c0 msgr2=0x7fa77c1987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa780f9c700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa77c1036c0 0x7fa77c1987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa780f9c700 1 -- 172.21.15.31:0/3107675023 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7700095f0 con 0x7fa77c105fe0 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa780f9c700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 0x7fa77c19f860 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa77400efc0 tx=0x7fa77400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa7797fa700 1 -- 172.21.15.31:0/3107675023 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774018070 con 0x7fa77c105fe0 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c1a0380 con 0x7fa77c105fe0 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa7797fa700 1 -- 172.21.15.31:0/3107675023 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa77400f040 con 0x7fa77c105fe0 2023-07-29T16:25:11.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.618+0000 7fa7797fa700 1 -- 172.21.15.31:0/3107675023 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77400cd40 con 0x7fa77c105fe0 2023-07-29T16:25:11.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.619+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c1a08e0 con 0x7fa77c105fe0 2023-07-29T16:25:11.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.623+0000 7fa7797fa700 1 -- 172.21.15.31:0/3107675023 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa774003a00 con 0x7fa77c105fe0 2023-07-29T16:25:11.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 Processor -- start 2023-07-29T16:25:11.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 -- start start 2023-07-29T16:25:11.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a6b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a6bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879a6c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638799c2a90 con 0x563879a6b5b0 2023-07-29T16:25:11.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638799c0450 con 0x563879a6ac60 2023-07-29T16:25:11.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.625+0000 7fa7963adb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638799bbd40 con 0x563879a6c370 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a6b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa7797fa700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7640a0810 0x7fa7640a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a6b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47614/0 (socket says 172.21.15.31:47614) 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3284748613 learned_addr learned my addr 172.21.15.31:0/3284748613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77bfff700 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a6bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879a6c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa7797fa700 1 -- 172.21.15.31:0/3107675023 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa7740cb570 con 0x7fa77c105fe0 2023-07-29T16:25:11.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3284748613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 msgr2=0x563879a6c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879a6c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3284748613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 msgr2=0x563879a6bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a6bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3284748613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5638797e9e60 con 0x563879a6ac60 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.626+0000 7fa780f9c700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7640a0810 0x7fa7640a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.627+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a6b070 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa770009990 tx=0x7fa770009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05870f73b62d511 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.627+0000 7fa7627fc700 1 -- 172.21.15.31:0/3284748613 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77000bda0 con 0x563879a6ac60 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.627+0000 7fa7627fc700 1 -- 172.21.15.31:0/3284748613 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa770004500 con 0x563879a6ac60 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.627+0000 7fa7627fc700 1 -- 172.21.15.31:0/3284748613 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77000f6a0 con 0x563879a6ac60 2023-07-29T16:25:11.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 -- 172.21.15.31:0/3284748613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 msgr2=0x563879a6b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a6b070 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa770009990 tx=0x7fa770009510 comp rx=0 tx=0).stop 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa780f9c700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7640a0810 0x7fa7640a2cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa7740090a0 tx=0x7fa774000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 -- 172.21.15.31:0/3284748613 shutdown_connections 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879a6c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a6b070 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3284748613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a6bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 -- 172.21.15.31:0/3284748613 >> 172.21.15.31:0/3284748613 conn(0x563879a60460 msgr2=0x563879a65a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 -- 172.21.15.31:0/3284748613 shutdown_connections 2023-07-29T16:25:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.628+0000 7fa7963adb80 1 -- 172.21.15.31:0/3284748613 wait complete. 2023-07-29T16:25:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.629+0000 7fa7963adb80 1 -- start start 2023-07-29T16:25:11.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.629+0000 7fa7963adb80 1 Processor -- start 2023-07-29T16:25:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a74860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a74da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a7bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879958590 con 0x563879a6ac60 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879958230 con 0x563879a6c370 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56387983f9a0 con 0x563879a6b5b0 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa7963adb80 10 client.0 _ll_register_callbacks cb 0x5638799616a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a74860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a74860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44168/0 (socket says 172.21.15.31:44168) 2023-07-29T16:25:11.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa77b7fe700 1 -- 172.21.15.31:0/2169808057 learned_addr learned my addr 172.21.15.31:0/2169808057 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa77bfff700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a74da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.630+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a7bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa77bfff700 1 -- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 msgr2=0x563879a7bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa77bfff700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a7bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa77bfff700 1 -- 172.21.15.31:0/2169808057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 msgr2=0x563879a74860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa77bfff700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a74860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa77bfff700 1 -- 172.21.15.31:0/2169808057 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa770020040 con 0x563879a6b5b0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa77bfff700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a74da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x563879a6b9c0 tx=0x7fa76c011770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b82407c12a650c82 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa75b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa75b7fe700 1 -- 172.21.15.31:0/2169808057 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c016070 con 0x563879a6b5b0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5638799e6d90 con 0x563879a6b5b0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa75b7fe700 1 -- 172.21.15.31:0/2169808057 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa76c004260 con 0x563879a6b5b0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.631+0000 7fa75b7fe700 1 -- 172.21.15.31:0/2169808057 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa76c004420 con 0x563879a6b5b0 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 10 client.28453 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:11.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5638797d2ce0 con 0x563879a6b5b0 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa75b7fe700 1 -- 172.21.15.31:0/2169808057 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5638799e6d90 con 0x563879a6b5b0 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa75b7fe700 1 -- 172.21.15.31:0/2169808057 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa76c01b750 con 0x563879a6b5b0 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 10 client.28453 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 10 client.28453 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 20 client.28453 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879a805e0 0x563879a829f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 4 client.28453 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563879a39df0 con 0x563879a805e0 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.632+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879a805e0 0x563879a829f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.633+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879a805e0 0x563879a829f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5df84ae43819fa8a server_cookie=66b1ec1f3b43cf8b in_seq=0 out_seq=0 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.633+0000 7fa75b7fe700 10 client.28453 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa75b7fe700 1 -- 172.21.15.31:0/2169808057 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563879a39df0 con 0x563879a805e0 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa75b7fe700 10 client.28453 handle_command_reply: tid=0 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 client.28453 shutdown 2023-07-29T16:25:11.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 msgr2=0x563879a74da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a74da0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x563879a6b9c0 tx=0x7fa76c011770 comp rx=0 tx=0).stop 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 shutdown_connections 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879a805e0 0x563879a829f0 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a74860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa75b7fe700 0 client.28453 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 --2- 172.21.15.31:0/2169808057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a74da0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 >> 172.21.15.31:0/2169808057 conn(0x563879a60460 msgr2=0x563879a650e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 shutdown_connections 2023-07-29T16:25:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.634+0000 7fa7963adb80 1 -- 172.21.15.31:0/2169808057 wait complete. 2023-07-29T16:25:11.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.635+0000 7fa7963adb80 20 client.28453 trim_cache size 0 max 16384 2023-07-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.654+0000 7fa7963adb80 1 Processor -- start 2023-07-29T16:25:11.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.654+0000 7fa7963adb80 1 -- start start 2023-07-29T16:25:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a72800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a7bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a73500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa7963adb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879a65880 con 0x563879a6ac60 2023-07-29T16:25:11.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa7963adb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879a725e0 con 0x563879a6c370 2023-07-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa7963adb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879a673c0 con 0x563879a6b5b0 2023-07-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a72800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a7bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a72800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44178/0 (socket says 172.21.15.31:44178) 2023-07-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77b7fe700 1 -- 172.21.15.31:0/928549414 learned_addr learned my addr 172.21.15.31:0/928549414 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a73500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77bfff700 1 -- 172.21.15.31:0/928549414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 msgr2=0x563879a73500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77bfff700 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a73500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77bfff700 1 -- 172.21.15.31:0/928549414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 msgr2=0x563879a72800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77bfff700 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a72800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77bfff700 1 -- 172.21.15.31:0/928549414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563879a6ca10 con 0x563879a6b5b0 2023-07-29T16:25:11.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.655+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a73500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:11.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.656+0000 7fa77bfff700 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a7bda0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa76c01bb60 tx=0x7fa76c01ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bb6ff69679a9824 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.656+0000 7fa762ffd700 1 -- 172.21.15.31:0/928549414 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c016040 con 0x563879a6b5b0 2023-07-29T16:25:11.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.656+0000 7fa762ffd700 1 -- 172.21.15.31:0/928549414 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa76c01b8e0 con 0x563879a6b5b0 2023-07-29T16:25:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.656+0000 7fa762ffd700 1 -- 172.21.15.31:0/928549414 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c003c20 con 0x563879a6b5b0 2023-07-29T16:25:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 -- 172.21.15.31:0/928549414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 msgr2=0x563879a7bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a7bda0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa76c01bb60 tx=0x7fa76c01ce60 comp rx=0 tx=0).stop 2023-07-29T16:25:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 -- 172.21.15.31:0/928549414 shutdown_connections 2023-07-29T16:25:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6c370 0x563879a73500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6ac60 0x563879a72800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 --2- 172.21.15.31:0/928549414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6b5b0 0x563879a7bda0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 -- 172.21.15.31:0/928549414 >> 172.21.15.31:0/928549414 conn(0x563879a60460 msgr2=0x563879a650e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 -- 172.21.15.31:0/928549414 shutdown_connections 2023-07-29T16:25:11.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 -- 172.21.15.31:0/928549414 wait complete. 2023-07-29T16:25:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 -- start start 2023-07-29T16:25:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.657+0000 7fa7963adb80 1 Processor -- start 2023-07-29T16:25:11.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a7b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a7b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879ad23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879ad2910 con 0x563879a6b5b0 2023-07-29T16:25:11.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879ad2a50 con 0x563879a6ac60 2023-07-29T16:25:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563879ad2be0 con 0x563879a6c370 2023-07-29T16:25:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa7963adb80 10 client.0 _ll_register_callbacks cb 0x5638799616a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:11.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a7b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa77b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a7b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47628/0 (socket says 172.21.15.31:47628) 2023-07-29T16:25:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3400381606 learned_addr learned my addr 172.21.15.31:0/3400381606 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa77bfff700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a7b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.658+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879ad23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3400381606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 msgr2=0x563879ad23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879ad23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3400381606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 msgr2=0x563879a7b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a7b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa77b7fe700 1 -- 172.21.15.31:0/3400381606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5638797d2ce0 con 0x563879a6ac60 2023-07-29T16:25:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa77b7fe700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a7b000 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa770009350 tx=0x7fa77000b850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53c2fc3d7d71becb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa75b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa75b7fe700 1 -- 172.21.15.31:0/3400381606 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7700053d0 con 0x563879a6ac60 2023-07-29T16:25:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa770004500 con 0x563879a6ac60 2023-07-29T16:25:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa75b7fe700 1 -- 172.21.15.31:0/3400381606 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa770005530 con 0x563879a6ac60 2023-07-29T16:25:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa75b7fe700 1 -- 172.21.15.31:0/3400381606 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7700056f0 con 0x563879a6ac60 2023-07-29T16:25:11.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa7963adb80 10 client.28454 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.659+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5638798102d0 con 0x563879a6ac60 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa75b7fe700 1 -- 172.21.15.31:0/3400381606 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa770044020 con 0x563879a6ac60 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa75b7fe700 1 -- 172.21.15.31:0/3400381606 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa77000f460 con 0x563879a6ac60 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa7963adb80 10 client.28454 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa7963adb80 10 client.28454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa7963adb80 20 client.28454 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:11.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879ad4030 0x563879ad6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa7963adb80 4 client.28454 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5638799b4dc0 con 0x563879ad4030 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.660+0000 7fa77bfff700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879ad4030 0x563879ad6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.661+0000 7fa77bfff700 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879ad4030 0x563879ad6440 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6248bcbfa1d28b4 server_cookie=102f88bd00b7d83 in_seq=0 out_seq=0 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.661+0000 7fa75b7fe700 10 client.28454 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.661+0000 7fa75b7fe700 1 -- 172.21.15.31:0/3400381606 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5638799b4dc0 con 0x563879ad4030 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.661+0000 7fa75b7fe700 10 client.28454 handle_command_reply: tid=0 2023-07-29T16:25:11.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 client.28454 shutdown 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 msgr2=0x563879a7b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a7b000 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fa770009350 tx=0x7fa77000b850 comp rx=0 tx=0).stop 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 shutdown_connections 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563879ad4030 0x563879ad6440 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563879a6c370 0x563879ad23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563879a6ac60 0x563879a7b000 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 --2- 172.21.15.31:0/3400381606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563879a6b5b0 0x563879a7b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 >> 172.21.15.31:0/3400381606 conn(0x563879a60460 msgr2=0x563879a72800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa75b7fe700 0 client.28454 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.662+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 shutdown_connections 2023-07-29T16:25:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.663+0000 7fa7963adb80 1 -- 172.21.15.31:0/3400381606 wait complete. 2023-07-29T16:25:11.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.663+0000 7fa7963adb80 20 client.28454 trim_cache size 0 max 16384 2023-07-29T16:25:11.681 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:25:11.682 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:25:11.682 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "afbb3b53-f9c4-4c5b-a50c-72fdf0913534", 2023-07-29T16:25:11.682 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:25:11.682 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:25:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.663+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7640a0810 msgr2=0x7fa7640a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.663+0000 7fa7829ff700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7640a0810 0x7fa7640a2cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa7740090a0 tx=0x7fa774000f40 comp rx=0 tx=0).stop 2023-07-29T16:25:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.663+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 msgr2=0x7fa77c19f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 0x7fa77c19f860 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa77400efc0 tx=0x7fa77400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 shutdown_connections 2023-07-29T16:25:11.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7640a0810 0x7fa7640a2cc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa77c105fe0 0x7fa77c19f860 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa77c100da0 0x7fa77c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 --2- 172.21.15.31:0/3107675023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa77c1036c0 0x7fa77c1987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 >> 172.21.15.31:0/3107675023 conn(0x7fa77c0fa840 msgr2=0x7fa77c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 shutdown_connections 2023-07-29T16:25:11.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.664+0000 7fa7829ff700 1 -- 172.21.15.31:0/3107675023 wait complete. 2023-07-29T16:25:11.686 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:25:11.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 685624 8.84 MB/sec execute 296 sec latency 61.310 ms 2023-07-29T16:25:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/3423456771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 msgr2=0x7fa7e8073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/3423456771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8073bc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc009990 tx=0x7fa7dc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/3423456771 shutdown_connections 2023-07-29T16:25:11.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/3423456771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e81081c0 0x7fa7e810a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/3423456771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e8074d30 0x7fa7e8073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/3423456771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8073bc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/3423456771 >> 172.21.15.31:0/3423456771 conn(0x7fa7e80fb970 msgr2=0x7fa7e80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/3423456771 shutdown_connections 2023-07-29T16:25:11.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.909+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/3423456771 wait complete. 2023-07-29T16:25:11.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.910+0000 7fa7ed3ff700 1 Processor -- start 2023-07-29T16:25:11.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.910+0000 7fa7ed3ff700 1 -- start start 2023-07-29T16:25:11.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7ed3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7ed3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e8074d30 0x7fa7e81987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7ed3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e81081c0 0x7fa7e819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7ed3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e819fe10 con 0x7fa7e81081c0 2023-07-29T16:25:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7ed3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e819ff50 con 0x7fa7e8074d30 2023-07-29T16:25:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7ed3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7e81a00e0 con 0x7fa7e8073750 2023-07-29T16:25:11.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38828/0 (socket says 172.21.15.31:38828) 2023-07-29T16:25:11.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e8074d30 0x7fa7e81987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/4066794097 learned_addr learned my addr 172.21.15.31:0/4066794097 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e81081c0 0x7fa7e819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/4066794097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e8074d30 msgr2=0x7fa7e81987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e8074d30 0x7fa7e81987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e81081c0 msgr2=0x7fa7e819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e81081c0 0x7fa7e819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/4066794097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7dc0095f0 con 0x7fa7e8073750 2023-07-29T16:25:11.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e8074d30 0x7fa7e81987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.911+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e81081c0 0x7fa7e819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.912+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8198290 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800e9a0 tx=0x7fa7d800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.912+0000 7fa7cffff700 1 -- 172.21.15.31:0/4066794097 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8009790 con 0x7fa7e8073750 2023-07-29T16:25:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.912+0000 7fa7cffff700 1 -- 172.21.15.31:0/4066794097 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7d8010430 con 0x7fa7e8073750 2023-07-29T16:25:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.912+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7e81a03c0 con 0x7fa7e8073750 2023-07-29T16:25:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.912+0000 7fa7cffff700 1 -- 172.21.15.31:0/4066794097 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8018680 con 0x7fa7e8073750 2023-07-29T16:25:11.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.912+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e81a08d0 con 0x7fa7e8073750 2023-07-29T16:25:11.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.916+0000 7fa7cffff700 1 -- 172.21.15.31:0/4066794097 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa7d80105a0 con 0x7fa7e8073750 2023-07-29T16:25:11.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.916+0000 7fa800d31b80 1 Processor -- start 2023-07-29T16:25:11.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 -- start start 2023-07-29T16:25:11.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724dbe0 0x557a3724dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 0x557a3724ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3724f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a371a5a10 con 0x557a3724dbe0 2023-07-29T16:25:11.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a371a33d0 con 0x557a3724e530 2023-07-29T16:25:11.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa800d31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3719ecc0 con 0x557a3724f2f0 2023-07-29T16:25:11.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa7cffff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7d00a0790 0x7fa7d00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.917+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 0x557a3724ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 0x557a3724ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47662/0 (socket says 172.21.15.31:47662) 2023-07-29T16:25:11.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/1084743425 learned_addr learned my addr 172.21.15.31:0/1084743425 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724dbe0 0x557a3724dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3724f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7cffff700 1 -- 172.21.15.31:0/4066794097 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa7d8014070 con 0x7fa7e8073750 2023-07-29T16:25:11.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7d00a0790 0x7fa7d00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/1084743425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 msgr2=0x557a3724f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3724f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/1084743425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724dbe0 msgr2=0x557a3724dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724dbe0 0x557a3724dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/1084743425 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a36fccde0 con 0x557a3724e530 2023-07-29T16:25:11.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724dbe0 0x557a3724dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:11.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.918+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 0x557a3724ed60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800c2d0 tx=0x7fa7d8099370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2825db1f7a132216 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.919+0000 7fa7cd7fa700 1 -- 172.21.15.31:0/1084743425 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8099480 con 0x557a3724e530 2023-07-29T16:25:11.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.919+0000 7fa7cd7fa700 1 -- 172.21.15.31:0/1084743425 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7d809d4b0 con 0x557a3724e530 2023-07-29T16:25:11.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.919+0000 7fa7cd7fa700 1 -- 172.21.15.31:0/1084743425 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8098970 con 0x557a3724e530 2023-07-29T16:25:11.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 -- 172.21.15.31:0/1084743425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 msgr2=0x557a3724ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 0x557a3724ed60 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800c2d0 tx=0x7fa7d8099370 comp rx=0 tx=0).stop 2023-07-29T16:25:11.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7d00a0790 0x7fa7d00a2c40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x557a3724d980 tx=0x7fa7e000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 -- 172.21.15.31:0/1084743425 shutdown_connections 2023-07-29T16:25:11.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3724f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724dbe0 0x557a3724dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 --2- 172.21.15.31:0/1084743425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724e530 0x557a3724ed60 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 -- 172.21.15.31:0/1084743425 >> 172.21.15.31:0/1084743425 conn(0x557a372433e0 msgr2=0x557a37248990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 -- 172.21.15.31:0/1084743425 shutdown_connections 2023-07-29T16:25:11.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 -- 172.21.15.31:0/1084743425 wait complete. 2023-07-29T16:25:11.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.920+0000 7fa800d31b80 1 -- start start 2023-07-29T16:25:11.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.921+0000 7fa800d31b80 1 Processor -- start 2023-07-29T16:25:11.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.921+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a37257890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.921+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a37257dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3725ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa800d31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3713b510 con 0x557a3724e530 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa800d31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3713b1b0 con 0x557a3724dbe0 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa800d31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a37022920 con 0x557a3724f2f0 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa800d31b80 10 client.0 _ll_register_callbacks cb 0x557a37144620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a37257890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a37257dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a37257890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47670/0 (socket says 172.21.15.31:47670) 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3510831301 learned_addr learned my addr 172.21.15.31:0/3510831301 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3725ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3510831301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 msgr2=0x557a3725ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3725ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3510831301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 msgr2=0x557a37257dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a37257dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3510831301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7e81a08d0 con 0x557a3724dbe0 2023-07-29T16:25:11.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a37257890 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80a13a0 tx=0x7fa7d8010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ba4cb4091cab1b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.922+0000 7fa7caff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:11.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa7caff5700 1 -- 172.21.15.31:0/3510831301 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d8097820 con 0x557a3724dbe0 2023-07-29T16:25:11.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa7caff5700 1 -- 172.21.15.31:0/3510831301 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa7d8099480 con 0x557a3724dbe0 2023-07-29T16:25:11.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a371c9d10 con 0x557a3724dbe0 2023-07-29T16:25:11.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa7caff5700 1 -- 172.21.15.31:0/3510831301 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7d80a1440 con 0x557a3724dbe0 2023-07-29T16:25:11.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 10 client.28466 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:11.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557a36fb5c60 con 0x557a3724dbe0 2023-07-29T16:25:11.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa7caff5700 1 -- 172.21.15.31:0/3510831301 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7d80b5080 con 0x557a3724dbe0 2023-07-29T16:25:11.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa7caff5700 1 -- 172.21.15.31:0/3510831301 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa7d80b1030 con 0x557a3724dbe0 2023-07-29T16:25:11.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 10 client.28466 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:11.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 10 client.28466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:11.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 20 client.28466 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:11.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.923+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a37263240 0x557a37265650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.924+0000 7fa800d31b80 4 client.28466 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:11.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.924+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a3721cd70 con 0x557a37263240 2023-07-29T16:25:11.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.924+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a37263240 0x557a37265650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.924+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a37263240 0x557a37265650 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3f967d8e63f79a6 server_cookie=875a6cad65c28909 in_seq=0 out_seq=0 2023-07-29T16:25:11.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.924+0000 7fa7caff5700 10 client.28466 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.925+0000 7fa7caff5700 1 -- 172.21.15.31:0/3510831301 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557a3721cd70 con 0x557a37263240 2023-07-29T16:25:11.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.925+0000 7fa7caff5700 10 client.28466 handle_command_reply: tid=0 2023-07-29T16:25:11.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.925+0000 7fa800d31b80 1 client.28466 shutdown 2023-07-29T16:25:11.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 msgr2=0x557a37257890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a37257890 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa7d80a13a0 tx=0x7fa7d8010ab0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 shutdown_connections 2023-07-29T16:25:11.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a37263240 0x557a37265650 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724f2f0 0x557a3725ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a37257890 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa7caff5700 0 client.28466 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3510831301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a37257dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 >> 172.21.15.31:0/3510831301 conn(0x557a372433e0 msgr2=0x557a37248060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 shutdown_connections 2023-07-29T16:25:11.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 1 -- 172.21.15.31:0/3510831301 wait complete. 2023-07-29T16:25:11.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.926+0000 7fa800d31b80 20 client.28466 trim_cache size 0 max 16384 2023-07-29T16:25:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.944+0000 7fa800d31b80 1 Processor -- start 2023-07-29T16:25:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.944+0000 7fa800d31b80 1 -- start start 2023-07-29T16:25:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724dbe0 0x557a37256f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a3725edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724f2f0 0x557a3724f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa800d31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a37265620 con 0x557a3724e530 2023-07-29T16:25:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa800d31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a37265790 con 0x557a3724f2f0 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa800d31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a37255700 con 0x557a3724dbe0 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a3725edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724dbe0 0x557a37256f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa7e67fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a3725edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44234/0 (socket says 172.21.15.31:44234) 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa7e67fc700 1 -- 172.21.15.31:0/2737671275 learned_addr learned my addr 172.21.15.31:0/2737671275 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.945+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724f2f0 0x557a3724f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e67fc700 1 -- 172.21.15.31:0/2737671275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724dbe0 msgr2=0x557a37256f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724dbe0 0x557a37256f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e67fc700 1 -- 172.21.15.31:0/2737671275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724f2f0 msgr2=0x557a3724f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724f2f0 0x557a3724f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e67fc700 1 -- 172.21.15.31:0/2737671275 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a36ff3250 con 0x557a3724e530 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724dbe0 0x557a37256f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a3725edd0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0095c0 tx=0x7fa7dc005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fd3303344bf63c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7cdffb700 1 -- 172.21.15.31:0/2737671275 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc01d040 con 0x557a3724e530 2023-07-29T16:25:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7cdffb700 1 -- 172.21.15.31:0/2737671275 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7dc01e040 con 0x557a3724e530 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.946+0000 7fa7cdffb700 1 -- 172.21.15.31:0/2737671275 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7dc0055b0 con 0x557a3724e530 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 -- 172.21.15.31:0/2737671275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 msgr2=0x557a3725edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a3725edd0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0095c0 tx=0x7fa7dc005fb0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 -- 172.21.15.31:0/2737671275 shutdown_connections 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724f2f0 0x557a3724f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724dbe0 0x557a37256f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 --2- 172.21.15.31:0/2737671275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724e530 0x557a3725edd0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 -- 172.21.15.31:0/2737671275 >> 172.21.15.31:0/2737671275 conn(0x557a372433e0 msgr2=0x557a37256510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 -- 172.21.15.31:0/2737671275 shutdown_connections 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.947+0000 7fa800d31b80 1 -- 172.21.15.31:0/2737671275 wait complete. 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 -- start start 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 Processor -- start 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a3725ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724e530 0x557a3725b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724f2f0 0x557a3725bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3725c2d0 con 0x557a3724f2f0 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a372b4de0 con 0x557a3724dbe0 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a372b4f20 con 0x557a3724e530 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.948+0000 7fa800d31b80 10 client.0 _ll_register_callbacks cb 0x557a37144620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a3725ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a3725ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47688/0 (socket says 172.21.15.31:47688) 2023-07-29T16:25:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3218656807 learned_addr learned my addr 172.21.15.31:0/3218656807 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724e530 0x557a3725b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e77fe700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724f2f0 0x557a3725bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3218656807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724e530 msgr2=0x557a3725b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724e530 0x557a3725b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3218656807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724f2f0 msgr2=0x557a3725bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724f2f0 0x557a3725bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 -- 172.21.15.31:0/3218656807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a36fb5c60 con 0x557a3724dbe0 2023-07-29T16:25:11.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7e6ffd700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a3725ec10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8099410 tx=0x7fa7d809d370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4416e672668c3f64 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7caff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7caff5700 1 -- 172.21.15.31:0/3218656807 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d80986f0 con 0x557a3724dbe0 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa7caff5700 1 -- 172.21.15.31:0/3218656807 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa7d80a1b20 con 0x557a3724dbe0 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.949+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7d809d4b0 con 0x557a3724dbe0 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa7caff5700 1 -- 172.21.15.31:0/3218656807 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7d80a1ce0 con 0x557a3724dbe0 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 10 client.28472 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:11.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fa7e810b260 con 0x557a3724dbe0 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa7caff5700 1 -- 172.21.15.31:0/3218656807 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7d809d4b0 con 0x557a3724dbe0 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa7caff5700 1 -- 172.21.15.31:0/3218656807 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa7d80996e0 con 0x557a3724dbe0 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 10 client.28472 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 10 client.28472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 20 client.28472 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a372b6450 0x557a372b8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 4 client.28472 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:25:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557a37197d40 con 0x557a372b6450 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.950+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a372b6450 0x557a372b8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.951+0000 7fa7caff5700 10 client.28472 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.951+0000 7fa7e67fc700 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a372b6450 0x557a372b8860 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87c39b705717952d server_cookie=f57a19957066b3a6 in_seq=0 out_seq=0 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa7caff5700 1 -- 172.21.15.31:0/3218656807 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557a37197d40 con 0x557a372b6450 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa7caff5700 10 client.28472 handle_command_reply: tid=0 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 client.28472 shutdown 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 msgr2=0x557a3725ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a3725ec10 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8099410 tx=0x7fa7d809d370 comp rx=0 tx=0).stop 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 shutdown_connections 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557a372b6450 0x557a372b8860 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a3724f2f0 0x557a3725bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a3724dbe0 0x557a3725ec10 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 --2- 172.21.15.31:0/3218656807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557a3724e530 0x557a3725b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 >> 172.21.15.31:0/3218656807 conn(0x557a372433e0 msgr2=0x557a37252250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa7caff5700 0 client.28472 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 shutdown_connections 2023-07-29T16:25:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 1 -- 172.21.15.31:0/3218656807 wait complete. 2023-07-29T16:25:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.952+0000 7fa800d31b80 20 client.28472 trim_cache size 0 max 16384 2023-07-29T16:25:11.964 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:25:11.964 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: "afbb3b53-f9c4-4c5b-a50c-72fdf0913534": { 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "afbb3b53-f9c4-4c5b-a50c-72fdf0913534", 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:25:11.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:25:11.966 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:25:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7d00a0790 msgr2=0x7fa7d00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7d00a0790 0x7fa7d00a2c40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x557a3724d980 tx=0x7fa7e000b040 comp rx=0 tx=0).stop 2023-07-29T16:25:11.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 msgr2=0x7fa7e8198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8198290 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa7d800e9a0 tx=0x7fa7d800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 shutdown_connections 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa7d00a0790 0x7fa7d00a2c40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7e81081c0 0x7fa7e819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7e8073750 0x7fa7e8198290 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 --2- 172.21.15.31:0/4066794097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7e8074d30 0x7fa7e81987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:11.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 >> 172.21.15.31:0/4066794097 conn(0x7fa7e80fb970 msgr2=0x7fa7e80fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 shutdown_connections 2023-07-29T16:25:11.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:11.953+0000 7fa7ed3ff700 1 -- 172.21.15.31:0/4066794097 wait complete. 2023-07-29T16:25:11.971 INFO:tasks.cephfs.filesystem:scrub status for tag:afbb3b53-f9c4-4c5b-a50c-72fdf0913534 - {'path': '/', 'tag': 'afbb3b53-f9c4-4c5b-a50c-72fdf0913534', 'options': 'recursive,force'} 2023-07-29T16:25:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:12 smithi156 ceph-mon[129245]: pgmap v663: 97 pgs: 97 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 2.1 MiB/s wr, 204 op/s 2023-07-29T16:25:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:12 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:12 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:12 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:12 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:12.840 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:12 smithi039 ceph-mon[128858]: pgmap v663: 97 pgs: 97 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 2.1 MiB/s wr, 204 op/s 2023-07-29T16:25:12.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:12 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:12.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:12 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:12.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:12 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:12.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:12 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:12.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 686909 8.83 MB/sec execute 297 sec latency 29.622 ms 2023-07-29T16:25:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:12 smithi031 ceph-mon[95504]: pgmap v663: 97 pgs: 97 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 2.1 MiB/s wr, 204 op/s 2023-07-29T16:25:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:12 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:12 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:12 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:12 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:13 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:25:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:13 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:13.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:13 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:13.840 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:13 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:25:13.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:13.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:13 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:13.841 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:13 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:13.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 689168 8.84 MB/sec execute 298 sec latency 14.296 ms 2023-07-29T16:25:13.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:13 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:25:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:13 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:13 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:14.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:14 smithi156 ceph-mon[129245]: pgmap v664: 97 pgs: 97 active+clean; 130 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2023-07-29T16:25:14.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 690373 8.83 MB/sec execute 299 sec latency 47.674 ms 2023-07-29T16:25:14.885 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:14 smithi039 ceph-mon[128858]: pgmap v664: 97 pgs: 97 active+clean; 130 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2023-07-29T16:25:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:14 smithi031 ceph-mon[95504]: pgmap v664: 97 pgs: 97 active+clean; 130 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 275 op/s 2023-07-29T16:25:15.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 692695 8.85 MB/sec execute 300 sec latency 12.833 ms 2023-07-29T16:25:16.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:16 smithi156 ceph-mon[129245]: pgmap v665: 97 pgs: 97 active+clean; 131 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-07-29T16:25:16.884 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:16 smithi039 ceph-mon[128858]: pgmap v665: 97 pgs: 97 active+clean; 131 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-07-29T16:25:16.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 694014 8.84 MB/sec execute 301 sec latency 31.499 ms 2023-07-29T16:25:16.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:16 smithi031 ceph-mon[95504]: pgmap v665: 97 pgs: 97 active+clean; 131 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.4 MiB/s wr, 269 op/s 2023-07-29T16:25:17.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 696254 8.85 MB/sec execute 302 sec latency 15.707 ms 2023-07-29T16:25:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:17 smithi039 ceph-mon[128858]: pgmap v666: 97 pgs: 97 active+clean; 130 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2023-07-29T16:25:18.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:17 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:25:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:17 smithi156 ceph-mon[129245]: pgmap v666: 97 pgs: 97 active+clean; 130 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2023-07-29T16:25:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:17 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:25:18.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:17 smithi031 ceph-mon[95504]: pgmap v666: 97 pgs: 97 active+clean; 130 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 2.0 MiB/s wr, 249 op/s 2023-07-29T16:25:18.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:17 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:25:18.884 INFO:tasks.workunit.client.0.smithi031.stdout: 1 697648 8.84 MB/sec execute 303 sec latency 24.890 ms 2023-07-29T16:25:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:18 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:25:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:18 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:25:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:18 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:25:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:18 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:25:19.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:18 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:25:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:18 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:25:19.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 699825 8.86 MB/sec execute 304 sec latency 15.019 ms 2023-07-29T16:25:20.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:19 smithi039 ceph-mon[128858]: pgmap v667: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 341 op/s 2023-07-29T16:25:20.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:19 smithi156 ceph-mon[129245]: pgmap v667: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 341 op/s 2023-07-29T16:25:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:19 smithi031 ceph-mon[95504]: pgmap v667: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 341 op/s 2023-07-29T16:25:20.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 701177 8.85 MB/sec execute 305 sec latency 79.769 ms 2023-07-29T16:25:21.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 703344 8.86 MB/sec execute 306 sec latency 26.677 ms 2023-07-29T16:25:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 -- 172.21.15.31:0/4049542719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0104100 msgr2=0x7fcea010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4049542719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0104100 0x7fcea010a9c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b0a0 tx=0x7fce9800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 -- 172.21.15.31:0/4049542719 shutdown_connections 2023-07-29T16:25:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4049542719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0104100 0x7fcea010a9c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4049542719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0102580 0x7fcea0102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4049542719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcea0103770 0x7fcea0103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 -- 172.21.15.31:0/4049542719 >> 172.21.15.31:0/4049542719 conn(0x7fcea00fdba0 msgr2=0x7fcea00fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 -- 172.21.15.31:0/4049542719 shutdown_connections 2023-07-29T16:25:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.071+0000 7fcea65ff700 1 -- 172.21.15.31:0/4049542719 wait complete. 2023-07-29T16:25:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.072+0000 7fcea65ff700 1 Processor -- start 2023-07-29T16:25:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.072+0000 7fcea65ff700 1 -- start start 2023-07-29T16:25:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.072+0000 7fcea65ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 0x7fcea019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fcea65ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0103770 0x7fcea019cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fcea65ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcea0104100 0x7fcea019dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fcea65ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea019e290 con 0x7fcea0103770 2023-07-29T16:25:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fcea65ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea019e400 con 0x7fcea0102580 2023-07-29T16:25:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fcea65ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea019e5c0 con 0x7fcea0104100 2023-07-29T16:25:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 0x7fcea019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 0x7fcea019c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52934/0 (socket says 172.21.15.31:52934) 2023-07-29T16:25:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 -- 172.21.15.31:0/4101961377 learned_addr learned my addr 172.21.15.31:0/4101961377 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0103770 0x7fcea019cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcea0104100 0x7fcea019dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 -- 172.21.15.31:0/4101961377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcea0104100 msgr2=0x7fcea019dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcea0104100 0x7fcea019dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 -- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0103770 msgr2=0x7fcea019cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0103770 0x7fcea019cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 -- 172.21.15.31:0/4101961377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94009520 con 0x7fcea0102580 2023-07-29T16:25:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.073+0000 7fce9ffff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 0x7fcea019c870 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fce9000e970 tx=0x7fce9000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.074+0000 7fce9d7fa700 1 -- 172.21.15.31:0/4101961377 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90009790 con 0x7fcea0102580 2023-07-29T16:25:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.074+0000 7fce9d7fa700 1 -- 172.21.15.31:0/4101961377 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce9000c7e0 con 0x7fcea0102580 2023-07-29T16:25:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.074+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce98009d00 con 0x7fcea0102580 2023-07-29T16:25:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.074+0000 7fce9d7fa700 1 -- 172.21.15.31:0/4101961377 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90003c70 con 0x7fcea0102580 2023-07-29T16:25:22.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.074+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01a4e80 con 0x7fcea0102580 2023-07-29T16:25:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.079+0000 7fce9d7fa700 1 -- 172.21.15.31:0/4101961377 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fce9000cc50 con 0x7fcea0102580 2023-07-29T16:25:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.080+0000 7fceb9f1db80 1 Processor -- start 2023-07-29T16:25:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.080+0000 7fceb9f1db80 1 -- start start 2023-07-29T16:25:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9192826b0 0x55d9194268b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 0x55d9194277b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d919428160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d91937e9b0 con 0x55d9192826b0 2023-07-29T16:25:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d91937c370 con 0x55d919427cf0 2023-07-29T16:25:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d919377c60 con 0x55d919426df0 2023-07-29T16:25:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9d7fa700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce880a0790 0x7fce880a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 0x55d9194277b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d919428160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d919428160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52936/0 (socket says 172.21.15.31:52936) 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 0x55d9194277b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58366/0 (socket says 172.21.15.31:58366) 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9f7fe700 1 -- 172.21.15.31:0/100755382 learned_addr learned my addr 172.21.15.31:0/100755382 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9d7fa700 1 -- 172.21.15.31:0/4101961377 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fce90014070 con 0x7fcea0102580 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9192826b0 0x55d9194268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce880a0790 0x7fce880a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 -- 172.21.15.31:0/100755382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 msgr2=0x55d919428160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d919428160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 -- 172.21.15.31:0/100755382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9192826b0 msgr2=0x55d9194268b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9192826b0 0x55d9194268b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.081+0000 7fce9ffff700 1 -- 172.21.15.31:0/100755382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9191a5d80 con 0x55d919426df0 2023-07-29T16:25:22.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.082+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d919428160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:22.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.082+0000 7fce9ffff700 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 0x55d9194277b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fce900c9860 tx=0x7fce90003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b9225cd41998106 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:22.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.082+0000 7fce867fc700 1 -- 172.21.15.31:0/100755382 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90098dd0 con 0x55d919426df0 2023-07-29T16:25:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.082+0000 7fce867fc700 1 -- 172.21.15.31:0/100755382 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce90003dd0 con 0x55d919426df0 2023-07-29T16:25:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.082+0000 7fce867fc700 1 -- 172.21.15.31:0/100755382 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90013190 con 0x55d919426df0 2023-07-29T16:25:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 -- 172.21.15.31:0/100755382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 msgr2=0x55d9194277b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce880a0790 0x7fce880a2c40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b780 tx=0x7fce98006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 0x55d9194277b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fce900c9860 tx=0x7fce90003680 comp rx=0 tx=0).stop 2023-07-29T16:25:22.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 -- 172.21.15.31:0/100755382 shutdown_connections 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d919428160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9192826b0 0x55d9194268b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/100755382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d919426df0 0x55d9194277b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 -- 172.21.15.31:0/100755382 >> 172.21.15.31:0/100755382 conn(0x55d91941bfd0 msgr2=0x55d919421300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 -- 172.21.15.31:0/100755382 shutdown_connections 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.083+0000 7fceb9f1db80 1 -- 172.21.15.31:0/100755382 wait complete. 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.084+0000 7fceb9f1db80 1 -- start start 2023-07-29T16:25:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.084+0000 7fceb9f1db80 1 Processor -- start 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d91942ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919426df0 0x55d919430440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919427cf0 0x55d919437480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9193144b0 con 0x55d919427cf0 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d919314150 con 0x55d919426df0 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9191fb8c0 con 0x55d9192826b0 2023-07-29T16:25:22.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fceb9f1db80 10 client.0 _ll_register_callbacks cb 0x55d91931d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d91942ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d91942ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58372/0 (socket says 172.21.15.31:58372) 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 -- 172.21.15.31:0/216825073 learned_addr learned my addr 172.21.15.31:0/216825073 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919426df0 0x55d919430440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919427cf0 0x55d919437480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 -- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919426df0 msgr2=0x55d919430440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919426df0 0x55d919430440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 -- 172.21.15.31:0/216825073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919427cf0 msgr2=0x55d919437480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919427cf0 0x55d919437480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 -- 172.21.15.31:0/216825073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce94009b70 con 0x55d9192826b0 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.085+0000 7fce9ffff700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d91942ff00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce9000efd0 tx=0x7fce90003710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f57d24d000199a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fce7f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fce7f7fe700 1 -- 172.21.15.31:0/216825073 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce900998d0 con 0x55d9192826b0 2023-07-29T16:25:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fce7f7fe700 1 -- 172.21.15.31:0/216825073 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fce9009d4a0 con 0x55d9192826b0 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9193a2cb0 con 0x55d9192826b0 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fce7f7fe700 1 -- 172.21.15.31:0/216825073 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce90098dd0 con 0x55d9192826b0 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 10 client.28489 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fcea01a4e80 con 0x55d9192826b0 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fce7f7fe700 1 -- 172.21.15.31:0/216825073 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce90063080 con 0x55d9192826b0 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fce7f7fe700 1 -- 172.21.15.31:0/216825073 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fce900b0030 con 0x55d9192826b0 2023-07-29T16:25:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 10 client.28489 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 10 client.28489 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 20 client.28489 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d91943dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 4 client.28489 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.086+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9193f5d10 con 0x55d91943b870 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.087+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d91943dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.087+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d91943dc80 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=276df0e8a8129df server_cookie=c5c203f6721fa418 in_seq=0 out_seq=0 2023-07-29T16:25:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.087+0000 7fce7f7fe700 10 client.28489 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:22.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fce7f7fe700 1 -- 172.21.15.31:0/216825073 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d9193f5d10 con 0x55d91943b870 2023-07-29T16:25:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fce7f7fe700 10 client.28489 handle_command_reply: tid=0 2023-07-29T16:25:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 client.28489 shutdown 2023-07-29T16:25:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 msgr2=0x55d91942ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d91942ff00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce9000efd0 tx=0x7fce90003710 comp rx=0 tx=0).stop 2023-07-29T16:25:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 shutdown_connections 2023-07-29T16:25:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d91943dc80 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919427cf0 0x55d919437480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d91942ff00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fce7f7fe700 0 client.28489 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/216825073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919426df0 0x55d919430440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 >> 172.21.15.31:0/216825073 conn(0x55d91941bfd0 msgr2=0x55d9194212d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.088+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 shutdown_connections 2023-07-29T16:25:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.089+0000 7fceb9f1db80 1 -- 172.21.15.31:0/216825073 wait complete. 2023-07-29T16:25:22.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.089+0000 7fceb9f1db80 20 client.28489 trim_cache size 0 max 16384 2023-07-29T16:25:22.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 Processor -- start 2023-07-29T16:25:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 -- start start 2023-07-29T16:25:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919435430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d91942ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d919422040 con 0x55d919426df0 2023-07-29T16:25:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d91942b6c0 con 0x55d919427cf0 2023-07-29T16:25:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d91943b5d0 con 0x55d9192826b0 2023-07-29T16:25:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.105+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919435430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919435430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58384/0 (socket says 172.21.15.31:58384) 2023-07-29T16:25:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 -- 172.21.15.31:0/3178416978 learned_addr learned my addr 172.21.15.31:0/3178416978 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d91942ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 -- 172.21.15.31:0/3178416978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 msgr2=0x55d91942ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d91942ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 -- 172.21.15.31:0/3178416978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 msgr2=0x55d9194280c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194280c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 -- 172.21.15.31:0/3178416978 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d91918ec00 con 0x55d9192826b0 2023-07-29T16:25:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce9ffff700 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919435430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce900c9b80 tx=0x7fce90095140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac6de4c893d97984 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce86ffd700 1 -- 172.21.15.31:0/3178416978 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90098dd0 con 0x55d9192826b0 2023-07-29T16:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce86ffd700 1 -- 172.21.15.31:0/3178416978 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9193a2cb0 con 0x55d9192826b0 2023-07-29T16:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.106+0000 7fce86ffd700 1 -- 172.21.15.31:0/3178416978 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90098bc0 con 0x55d9192826b0 2023-07-29T16:25:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 -- 172.21.15.31:0/3178416978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 msgr2=0x55d919435430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919435430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce900c9b80 tx=0x7fce90095140 comp rx=0 tx=0).stop 2023-07-29T16:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 -- 172.21.15.31:0/3178416978 shutdown_connections 2023-07-29T16:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d91942ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919435430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/3178416978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194280c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 -- 172.21.15.31:0/3178416978 >> 172.21.15.31:0/3178416978 conn(0x55d91941bfd0 msgr2=0x55d9194212d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 -- 172.21.15.31:0/3178416978 shutdown_connections 2023-07-29T16:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.107+0000 7fceb9f1db80 1 -- 172.21.15.31:0/3178416978 wait complete. 2023-07-29T16:25:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.108+0000 7fceb9f1db80 1 -- start start 2023-07-29T16:25:22.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.108+0000 7fceb9f1db80 1 Processor -- start 2023-07-29T16:25:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919433390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d9194348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d919434e20 con 0x55d919426df0 2023-07-29T16:25:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d919434f90 con 0x55d919427cf0 2023-07-29T16:25:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d919435150 con 0x55d9192826b0 2023-07-29T16:25:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fceb9f1db80 10 client.0 _ll_register_callbacks cb 0x55d91931d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919433390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fce9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919433390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58392/0 (socket says 172.21.15.31:58392) 2023-07-29T16:25:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fce9ffff700 1 -- 172.21.15.31:0/854213174 learned_addr learned my addr 172.21.15.31:0/854213174 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fcea4b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d9194348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fcea4b9c700 1 -- 172.21.15.31:0/854213174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 msgr2=0x55d919433390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919433390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fcea4b9c700 1 -- 172.21.15.31:0/854213174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 msgr2=0x55d9194338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fcea4b9c700 1 -- 172.21.15.31:0/854213174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01a4e80 con 0x55d919427cf0 2023-07-29T16:25:22.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fce9ffff700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919433390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:22.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.109+0000 7fce9f7fe700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:22.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fcea4b9c700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d9194348e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b070 tx=0x7fce980099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b87fbee6b3b3284 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fce7f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fce7f7fe700 1 -- 172.21.15.31:0/854213174 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9800e070 con 0x55d919427cf0 2023-07-29T16:25:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce90003dd0 con 0x55d919427cf0 2023-07-29T16:25:22.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fce7f7fe700 1 -- 172.21.15.31:0/854213174 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fce9801f850 con 0x55d919427cf0 2023-07-29T16:25:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fce7f7fe700 1 -- 172.21.15.31:0/854213174 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce98003e60 con 0x55d919427cf0 2023-07-29T16:25:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fceb9f1db80 10 client.28490 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.110+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d9191cc1f0 con 0x55d919427cf0 2023-07-29T16:25:22.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fce7f7fe700 1 -- 172.21.15.31:0/854213174 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce90003dd0 con 0x55d919427cf0 2023-07-29T16:25:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fce7f7fe700 1 -- 172.21.15.31:0/854213174 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fce980057e0 con 0x55d919427cf0 2023-07-29T16:25:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fceb9f1db80 10 client.28490 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fceb9f1db80 10 client.28490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:22.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fceb9f1db80 20 client.28490 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:22.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d9194903e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:22.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fceb9f1db80 4 client.28490 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:25:22.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d919370ce0 con 0x55d91943b870 2023-07-29T16:25:22.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.111+0000 7fce9ffff700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d9194903e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:22.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.112+0000 7fce9ffff700 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d9194903e0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b3d2aad4c869340 server_cookie=3a58239977b29604 in_seq=0 out_seq=0 2023-07-29T16:25:22.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.112+0000 7fce7f7fe700 10 client.28490 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fce7f7fe700 1 -- 172.21.15.31:0/854213174 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d919370ce0 con 0x55d91943b870 2023-07-29T16:25:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fce7f7fe700 10 client.28490 handle_command_reply: tid=0 2023-07-29T16:25:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 client.28490 shutdown 2023-07-29T16:25:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 msgr2=0x55d9194348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d9194348e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b070 tx=0x7fce980099e0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 shutdown_connections 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d91943b870 0x55d9194903e0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d919427cf0 0x55d9194348e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d9192826b0 0x55d919433390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 --2- 172.21.15.31:0/854213174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d919426df0 0x55d9194338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fce7f7fe700 0 client.28490 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 >> 172.21.15.31:0/854213174 conn(0x55d91941bfd0 msgr2=0x55d91942e020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 shutdown_connections 2023-07-29T16:25:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.180+0000 7fceb9f1db80 1 -- 172.21.15.31:0/854213174 wait complete. 2023-07-29T16:25:22.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.181+0000 7fceb9f1db80 20 client.28490 trim_cache size 0 max 16384 2023-07-29T16:25:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:25:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:25:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:25:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:25:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.181+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce880a0790 msgr2=0x7fce880a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.181+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce880a0790 0x7fce880a2c40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fce9800b780 tx=0x7fce98006210 comp rx=0 tx=0).stop 2023-07-29T16:25:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.181+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 msgr2=0x7fcea019c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.181+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 0x7fcea019c870 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fce9000e970 tx=0x7fce9000ec80 comp rx=0 tx=0).stop 2023-07-29T16:25:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 shutdown_connections 2023-07-29T16:25:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fce880a0790 0x7fce880a2c40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcea0104100 0x7fcea019dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcea0102580 0x7fcea019c870 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 --2- 172.21.15.31:0/4101961377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcea0103770 0x7fcea019cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 >> 172.21.15.31:0/4101961377 conn(0x7fcea00fdba0 msgr2=0x7fcea0104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 shutdown_connections 2023-07-29T16:25:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:22.182+0000 7fcea65ff700 1 -- 172.21.15.31:0/4101961377 wait complete. 2023-07-29T16:25:22.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:22 smithi039 ceph-mon[128858]: pgmap v668: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 1.8 MiB/s wr, 237 op/s 2023-07-29T16:25:22.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:22 smithi156 ceph-mon[129245]: pgmap v668: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 1.8 MiB/s wr, 237 op/s 2023-07-29T16:25:22.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 704602 8.85 MB/sec execute 307 sec latency 64.206 ms 2023-07-29T16:25:22.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:22 smithi031 ceph-mon[95504]: pgmap v668: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 1.8 MiB/s wr, 237 op/s 2023-07-29T16:25:23.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 706905 8.87 MB/sec execute 308 sec latency 13.337 ms 2023-07-29T16:25:24.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:24 smithi039 ceph-mon[128858]: pgmap v669: 97 pgs: 97 active+clean; 136 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 2.1 MiB/s wr, 293 op/s 2023-07-29T16:25:24.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:24 smithi156 ceph-mon[129245]: pgmap v669: 97 pgs: 97 active+clean; 136 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 2.1 MiB/s wr, 293 op/s 2023-07-29T16:25:24.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 708368 8.86 MB/sec execute 309 sec latency 11.249 ms 2023-07-29T16:25:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:24 smithi031 ceph-mon[95504]: pgmap v669: 97 pgs: 97 active+clean; 136 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 2.1 MiB/s wr, 293 op/s 2023-07-29T16:25:25.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 710514 8.87 MB/sec execute 310 sec latency 11.620 ms 2023-07-29T16:25:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:26 smithi156 ceph-mon[129245]: pgmap v670: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 1.9 MiB/s wr, 274 op/s 2023-07-29T16:25:26.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 711873 8.86 MB/sec execute 311 sec latency 60.991 ms 2023-07-29T16:25:26.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:26 smithi031 ceph-mon[95504]: pgmap v670: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 1.9 MiB/s wr, 274 op/s 2023-07-29T16:25:27.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:26 smithi039 ceph-mon[128858]: pgmap v670: 97 pgs: 97 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 1.9 MiB/s wr, 274 op/s 2023-07-29T16:25:27.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 714020 8.88 MB/sec execute 312 sec latency 24.802 ms 2023-07-29T16:25:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:27 smithi039 ceph-mon[128858]: pgmap v671: 97 pgs: 97 active+clean; 135 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.8 MiB/s wr, 218 op/s 2023-07-29T16:25:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:27 smithi156 ceph-mon[129245]: pgmap v671: 97 pgs: 97 active+clean; 135 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.8 MiB/s wr, 218 op/s 2023-07-29T16:25:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:27 smithi031 ceph-mon[95504]: pgmap v671: 97 pgs: 97 active+clean; 135 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.8 MiB/s wr, 218 op/s 2023-07-29T16:25:28.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 715626 8.87 MB/sec execute 313 sec latency 11.383 ms 2023-07-29T16:25:29.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 717681 8.88 MB/sec execute 314 sec latency 14.573 ms 2023-07-29T16:25:30.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:30 smithi039 ceph-mon[128858]: pgmap v672: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 313 op/s 2023-07-29T16:25:30.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:30 smithi156 ceph-mon[129245]: pgmap v672: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 313 op/s 2023-07-29T16:25:30.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 719754 8.88 MB/sec execute 315 sec latency 12.500 ms 2023-07-29T16:25:30.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:30 smithi031 ceph-mon[95504]: pgmap v672: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 313 op/s 2023-07-29T16:25:31.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 721196 8.88 MB/sec execute 316 sec latency 46.733 ms 2023-07-29T16:25:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:32 smithi156 ceph-mon[129245]: pgmap v673: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.8 MiB/s wr, 217 op/s 2023-07-29T16:25:32.885 INFO:tasks.workunit.client.0.smithi031.stdout: 1 723153 8.88 MB/sec execute 317 sec latency 18.490 ms 2023-07-29T16:25:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:32 smithi031 ceph-mon[95504]: pgmap v673: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.8 MiB/s wr, 217 op/s 2023-07-29T16:25:33.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:32 smithi039 ceph-mon[128858]: pgmap v673: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 1.8 MiB/s wr, 217 op/s 2023-07-29T16:25:33.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 724776 8.89 MB/sec execute 318 sec latency 25.958 ms 2023-07-29T16:25:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:34 smithi156 ceph-mon[129245]: pgmap v674: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 2.3 MiB/s wr, 274 op/s 2023-07-29T16:25:34.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 726773 8.88 MB/sec execute 319 sec latency 12.000 ms 2023-07-29T16:25:34.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:34 smithi031 ceph-mon[95504]: pgmap v674: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 2.3 MiB/s wr, 274 op/s 2023-07-29T16:25:35.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:34 smithi039 ceph-mon[128858]: pgmap v674: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 2.3 MiB/s wr, 274 op/s 2023-07-29T16:25:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/776251648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c881037c0 msgr2=0x7f6c88105ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/776251648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c881037c0 0x7f6c88105ba0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f6c780098f0 tx=0x7f6c78009c00 comp rx=0 tx=0).stop 2023-07-29T16:25:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/776251648 shutdown_connections 2023-07-29T16:25:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/776251648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c881060e0 0x7f6c881084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/776251648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c88100ea0 0x7f6c88103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/776251648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c881037c0 0x7f6c88105ba0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/776251648 >> 172.21.15.31:0/776251648 conn(0x7f6c880fa840 msgr2=0x7f6c880fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/776251648 shutdown_connections 2023-07-29T16:25:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.343+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/776251648 wait complete. 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.344+0000 7f6c8f3ff700 1 Processor -- start 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.344+0000 7f6c8f3ff700 1 -- start start 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8f3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 0x7f6c880fee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8f3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c881037c0 0x7f6c880ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8f3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c881060e0 0x7f6c880ff8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8f3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c881a01b0 con 0x7f6c881060e0 2023-07-29T16:25:35.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8f3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c881a02f0 con 0x7f6c88100ea0 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8f3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c881a0430 con 0x7f6c881037c0 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 0x7f6c880fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 0x7f6c880fee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56396/0 (socket says 172.21.15.31:56396) 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 -- 172.21.15.31:0/2175381641 learned_addr learned my addr 172.21.15.31:0/2175381641 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c881060e0 0x7f6c880ff8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8c99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c881037c0 0x7f6c880ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 -- 172.21.15.31:0/2175381641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c881037c0 msgr2=0x7f6c880ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c881037c0 0x7f6c880ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 -- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c881060e0 msgr2=0x7f6c880ff8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c881060e0 0x7f6c880ff8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8d19b700 1 -- 172.21.15.31:0/2175381641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78009520 con 0x7f6c88100ea0 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.345+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c881037c0 0x7f6c880ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.346+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 0x7f6c880fee50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400e9f0 tx=0x7f6c8400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.346+0000 7f6c7e7fc700 1 -- 172.21.15.31:0/2175381641 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84009690 con 0x7f6c88100ea0 2023-07-29T16:25:35.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.346+0000 7f6c7e7fc700 1 -- 172.21.15.31:0/2175381641 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c8400ce80 con 0x7f6c88100ea0 2023-07-29T16:25:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.346+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c881a06c0 con 0x7f6c88100ea0 2023-07-29T16:25:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.346+0000 7f6c7e7fc700 1 -- 172.21.15.31:0/2175381641 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84018420 con 0x7f6c88100ea0 2023-07-29T16:25:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.346+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c881a0bf0 con 0x7f6c88100ea0 2023-07-29T16:25:35.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.350+0000 7f6c7e7fc700 1 -- 172.21.15.31:0/2175381641 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6c8400c9c0 con 0x7f6c88100ea0 2023-07-29T16:25:35.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 Processor -- start 2023-07-29T16:25:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 -- start start 2023-07-29T16:25:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac152e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 0x55eac152f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152fcf0 0x55eac1530160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac14869b0 con 0x55eac138a6b0 2023-07-29T16:25:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac1484370 con 0x55eac152fcf0 2023-07-29T16:25:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac147fc60 con 0x55eac152edf0 2023-07-29T16:25:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.352+0000 7f6c7e7fc700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6c740a0810 0x7f6c740a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 0x55eac152f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 0x55eac152f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42588/0 (socket says 172.21.15.31:42588) 2023-07-29T16:25:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 -- 172.21.15.31:0/1254685772 learned_addr learned my addr 172.21.15.31:0/1254685772 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152fcf0 0x55eac1530160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac152e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 -- 172.21.15.31:0/1254685772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152fcf0 msgr2=0x55eac1530160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6c740a0810 0x7f6c740a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152fcf0 0x55eac1530160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 -- 172.21.15.31:0/1254685772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 msgr2=0x55eac152e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac152e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c7e7fc700 1 -- 172.21.15.31:0/2175381641 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6c84014070 con 0x7f6c88100ea0 2023-07-29T16:25:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8d19b700 1 -- 172.21.15.31:0/1254685772 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eac12add80 con 0x55eac152edf0 2023-07-29T16:25:35.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.353+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152fcf0 0x55eac1530160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.354+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 0x55eac152f7b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6c840994c0 tx=0x7f6c8400ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e84a45eaa742f5b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.354+0000 7f6c737fe700 1 -- 172.21.15.31:0/1254685772 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8409d330 con 0x55eac152edf0 2023-07-29T16:25:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.354+0000 7f6c737fe700 1 -- 172.21.15.31:0/1254685772 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c840a1750 con 0x55eac152edf0 2023-07-29T16:25:35.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.354+0000 7f6c737fe700 1 -- 172.21.15.31:0/1254685772 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c840996a0 con 0x55eac152edf0 2023-07-29T16:25:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/1254685772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 msgr2=0x55eac152f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6c740a0810 0x7f6c740a2cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009ec0 tx=0x7f6c8000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 0x55eac152f7b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6c840994c0 tx=0x7f6c8400ef80 comp rx=0 tx=0).stop 2023-07-29T16:25:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/1254685772 shutdown_connections 2023-07-29T16:25:35.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152fcf0 0x55eac1530160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac152e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/1254685772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152edf0 0x55eac152f7b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/1254685772 >> 172.21.15.31:0/1254685772 conn(0x55eac1523fd0 msgr2=0x55eac1529300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:35.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/1254685772 shutdown_connections 2023-07-29T16:25:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/1254685772 wait complete. 2023-07-29T16:25:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.355+0000 7f6ca2c82b80 1 -- start start 2023-07-29T16:25:35.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 Processor -- start 2023-07-29T16:25:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 0x55eac1537e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152edf0 0x55eac15383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac141c4b0 con 0x55eac152edf0 2023-07-29T16:25:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac141c150 con 0x55eac138a6b0 2023-07-29T16:25:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac13038c0 con 0x55eac152fcf0 2023-07-29T16:25:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6ca2c82b80 10 client.0 _ll_register_callbacks cb 0x55eac14255c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6c8c99a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152edf0 0x55eac15383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 0x55eac1537e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 0x55eac1537e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56420/0 (socket says 172.21.15.31:56420) 2023-07-29T16:25:35.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6c8d19b700 1 -- 172.21.15.31:0/799274943 learned_addr learned my addr 172.21.15.31:0/799274943 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.356+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c8d19b700 1 -- 172.21.15.31:0/799274943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 msgr2=0x55eac153f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c8d19b700 1 -- 172.21.15.31:0/799274943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152edf0 msgr2=0x55eac15383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152edf0 0x55eac15383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c8d19b700 1 -- 172.21.15.31:0/799274943 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c88109090 con 0x55eac138a6b0 2023-07-29T16:25:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 0x55eac1537e70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6c84000c00 tx=0x7f6c840cbce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d011b34b33e09a98 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:35.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c70ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c70ff9700 1 -- 172.21.15.31:0/799274943 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8409d170 con 0x55eac138a6b0 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6c70ff9700 1 -- 172.21.15.31:0/799274943 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6c840580d0 con 0x55eac138a6b0 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.357+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eac14aacb0 con 0x55eac138a6b0 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6c70ff9700 1 -- 172.21.15.31:0/799274943 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c8409d330 con 0x55eac138a6b0 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 10 client.28502 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f6c881a0bf0 con 0x55eac138a6b0 2023-07-29T16:25:35.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6c70ff9700 1 -- 172.21.15.31:0/799274943 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eac14aacb0 con 0x55eac138a6b0 2023-07-29T16:25:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6c70ff9700 1 -- 172.21.15.31:0/799274943 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6c840c9bf0 con 0x55eac138a6b0 2023-07-29T16:25:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 10 client.28502 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 10 client.28502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 20 client.28502 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1543810 0x55eac1545c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 4 client.28502 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eac14fdd10 con 0x55eac1543810 2023-07-29T16:25:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.358+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1543810 0x55eac1545c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.359+0000 7f6c70ff9700 10 client.28502 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.359+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1543810 0x55eac1545c20 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac234f5153ce5fd2 server_cookie=7c294f73879ad97d in_seq=0 out_seq=0 2023-07-29T16:25:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6c70ff9700 1 -- 172.21.15.31:0/799274943 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55eac14fdd10 con 0x55eac1543810 2023-07-29T16:25:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6c70ff9700 10 client.28502 handle_command_reply: tid=0 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 client.28502 shutdown 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 msgr2=0x55eac1537e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 0x55eac1537e70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6c84000c00 tx=0x7f6c840cbce0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 shutdown_connections 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1543810 0x55eac1545c20 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6c70ff9700 0 client.28502 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:35.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac138a6b0 0x55eac1537e70 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/799274943 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152edf0 0x55eac15383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 >> 172.21.15.31:0/799274943 conn(0x55eac1523fd0 msgr2=0x55eac15292d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 shutdown_connections 2023-07-29T16:25:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.360+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/799274943 wait complete. 2023-07-29T16:25:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.361+0000 7f6ca2c82b80 20 client.28502 trim_cache size 0 max 16384 2023-07-29T16:25:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 Processor -- start 2023-07-29T16:25:35.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 -- start start 2023-07-29T16:25:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac138a6b0 0x55eac153d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac15300c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 0x55eac1535ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac152aba0 con 0x55eac152fcf0 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac1530630 con 0x55eac152edf0 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac1523a00 con 0x55eac138a6b0 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6c8d99c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 0x55eac1535ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6c8c99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac15300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6c8d99c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 0x55eac1535ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50318/0 (socket says 172.21.15.31:50318) 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6c8d99c700 1 -- 172.21.15.31:0/632004194 learned_addr learned my addr 172.21.15.31:0/632004194 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:35.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.377+0000 7f6c8c99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac15300c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56434/0 (socket says 172.21.15.31:56434) 2023-07-29T16:25:35.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac138a6b0 0x55eac153d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d99c700 1 -- 172.21.15.31:0/632004194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac138a6b0 msgr2=0x55eac153d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac138a6b0 0x55eac153d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d99c700 1 -- 172.21.15.31:0/632004194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 msgr2=0x55eac15300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac15300c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d99c700 1 -- 172.21.15.31:0/632004194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eac1296c00 con 0x55eac152fcf0 2023-07-29T16:25:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac138a6b0 0x55eac153d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.378+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 0x55eac1535ff0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6c80018650 tx=0x7f6c80009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb753e21ced9345 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.379+0000 7f6c73fff700 1 -- 172.21.15.31:0/632004194 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8000e070 con 0x55eac152fcf0 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.379+0000 7f6c73fff700 1 -- 172.21.15.31:0/632004194 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c8000f040 con 0x55eac152fcf0 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.379+0000 7f6c73fff700 1 -- 172.21.15.31:0/632004194 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80016510 con 0x55eac152fcf0 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/632004194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 msgr2=0x55eac1535ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 0x55eac1535ff0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6c80018650 tx=0x7f6c80009920 comp rx=0 tx=0).stop 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/632004194 shutdown_connections 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac152fcf0 0x55eac1535ff0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac138a6b0 0x55eac153d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/632004194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac15300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/632004194 >> 172.21.15.31:0/632004194 conn(0x55eac1523fd0 msgr2=0x55eac1536bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/632004194 shutdown_connections 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/632004194 wait complete. 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 -- start start 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.380+0000 7f6ca2c82b80 1 Processor -- start 2023-07-29T16:25:35.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac153cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac153b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac153d7e0 con 0x55eac138a6b0 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac153c080 con 0x55eac152edf0 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eac153c240 con 0x55eac152fcf0 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6ca2c82b80 10 client.0 _ll_register_callbacks cb 0x55eac154aaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:35.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6c8d19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac153cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6c8c99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac153b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6c8c99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac153b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56448/0 (socket says 172.21.15.31:56448) 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.381+0000 7f6c8c99a700 1 -- 172.21.15.31:0/3989929436 learned_addr learned my addr 172.21.15.31:0/3989929436 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8d99c700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8c99a700 1 -- 172.21.15.31:0/3989929436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 msgr2=0x55eac153bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8c99a700 1 -- 172.21.15.31:0/3989929436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 msgr2=0x55eac153cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac153cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8c99a700 1 -- 172.21.15.31:0/3989929436 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c881a0bf0 con 0x55eac152edf0 2023-07-29T16:25:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac153cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c8c99a700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac153b5d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6c780098c0 tx=0x7f6c7800f960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e5b9bff8784bf5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c70ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c70ff9700 1 -- 172.21.15.31:0/3989929436 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7801c040 con 0x55eac152edf0 2023-07-29T16:25:35.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c840a1750 con 0x55eac152edf0 2023-07-29T16:25:35.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.382+0000 7f6c70ff9700 1 -- 172.21.15.31:0/3989929436 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6c78028bb0 con 0x55eac152edf0 2023-07-29T16:25:35.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6c70ff9700 1 -- 172.21.15.31:0/3989929436 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c78028d70 con 0x55eac152edf0 2023-07-29T16:25:35.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 10 client.28508 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:35.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f6c80021040 con 0x55eac152edf0 2023-07-29T16:25:35.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6c70ff9700 1 -- 172.21.15.31:0/3989929436 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c840a1750 con 0x55eac152edf0 2023-07-29T16:25:35.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6c70ff9700 1 -- 172.21.15.31:0/3989929436 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6c78009920 con 0x55eac152edf0 2023-07-29T16:25:35.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 10 client.28508 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 10 client.28508 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 20 client.28508 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1596af0 0x55eac1598f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 4 client.28508 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55eac1478ce0 con 0x55eac1596af0 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.383+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1596af0 0x55eac1598f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:35.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.384+0000 7f6c8d19b700 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1596af0 0x55eac1598f00 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96d4d0d400633f8a server_cookie=ab92c953b576a9d1 in_seq=0 out_seq=0 2023-07-29T16:25:35.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.384+0000 7f6c70ff9700 10 client.28508 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:35.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.523+0000 7f6c70ff9700 1 -- 172.21.15.31:0/3989929436 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55eac1478ce0 con 0x55eac1596af0 2023-07-29T16:25:35.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6c70ff9700 10 client.28508 handle_command_reply: tid=0 2023-07-29T16:25:35.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 client.28508 shutdown 2023-07-29T16:25:35.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 msgr2=0x55eac153b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac153b5d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6c780098c0 tx=0x7f6c7800f960 comp rx=0 tx=0).stop 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 shutdown_connections 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55eac1596af0 0x55eac1598f00 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55eac152fcf0 0x55eac153bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eac138a6b0 0x55eac153cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 --2- 172.21.15.31:0/3989929436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55eac152edf0 0x55eac153b5d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6c70ff9700 0 client.28508 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 >> 172.21.15.31:0/3989929436 conn(0x55eac1523fd0 msgr2=0x55eac1535ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 shutdown_connections 2023-07-29T16:25:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 1 -- 172.21.15.31:0/3989929436 wait complete. 2023-07-29T16:25:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.524+0000 7f6ca2c82b80 20 client.28508 trim_cache size 0 max 16384 2023-07-29T16:25:35.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:25:35.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:25:35.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:25:35.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:25:35.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6c740a0810 msgr2=0x7f6c740a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6c740a0810 0x7f6c740a2cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009ec0 tx=0x7f6c8000b040 comp rx=0 tx=0).stop 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 msgr2=0x7f6c880fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 0x7f6c880fee50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400e9f0 tx=0x7f6c8400ed00 comp rx=0 tx=0).stop 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 shutdown_connections 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6c740a0810 0x7f6c740a2cc0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c881060e0 0x7f6c880ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6c88100ea0 0x7f6c880fee50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 --2- 172.21.15.31:0/2175381641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c881037c0 0x7f6c880ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 >> 172.21.15.31:0/2175381641 conn(0x7f6c880fa840 msgr2=0x7f6c88102360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.525+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 shutdown_connections 2023-07-29T16:25:35.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:35.526+0000 7f6c8f3ff700 1 -- 172.21.15.31:0/2175381641 wait complete. 2023-07-29T16:25:35.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 728241 8.89 MB/sec execute 320 sec latency 132.977 ms 2023-07-29T16:25:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:36 smithi156 ceph-mon[129245]: pgmap v675: 97 pgs: 97 active+clean; 140 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.6 MiB/s wr, 262 op/s 2023-07-29T16:25:36.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 729584 8.88 MB/sec execute 321 sec latency 65.173 ms 2023-07-29T16:25:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:36 smithi031 ceph-mon[95504]: pgmap v675: 97 pgs: 97 active+clean; 140 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.6 MiB/s wr, 262 op/s 2023-07-29T16:25:37.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:36 smithi039 ceph-mon[128858]: pgmap v675: 97 pgs: 97 active+clean; 140 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 2.6 MiB/s wr, 262 op/s 2023-07-29T16:25:37.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 731716 8.90 MB/sec execute 322 sec latency 30.450 ms 2023-07-29T16:25:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:37 smithi039 ceph-mon[128858]: pgmap v676: 97 pgs: 97 active+clean; 141 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 2.4 MiB/s wr, 244 op/s 2023-07-29T16:25:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:37 smithi156 ceph-mon[129245]: pgmap v676: 97 pgs: 97 active+clean; 141 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 2.4 MiB/s wr, 244 op/s 2023-07-29T16:25:38.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:37 smithi031 ceph-mon[95504]: pgmap v676: 97 pgs: 97 active+clean; 141 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 2.4 MiB/s wr, 244 op/s 2023-07-29T16:25:38.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 733163 8.89 MB/sec execute 323 sec latency 13.089 ms 2023-07-29T16:25:39.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 735293 8.90 MB/sec execute 324 sec latency 13.679 ms 2023-07-29T16:25:40.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:40 smithi156 ceph-mon[129245]: pgmap v677: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 2.9 MiB/s wr, 330 op/s 2023-07-29T16:25:40.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 736741 8.89 MB/sec execute 325 sec latency 11.963 ms 2023-07-29T16:25:40.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:40 smithi031 ceph-mon[95504]: pgmap v677: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 2.9 MiB/s wr, 330 op/s 2023-07-29T16:25:41.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:40 smithi039 ceph-mon[128858]: pgmap v677: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 2.9 MiB/s wr, 330 op/s 2023-07-29T16:25:41.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 738818 8.90 MB/sec execute 326 sec latency 42.652 ms 2023-07-29T16:25:41.972 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:25:42.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.210+0000 7f6e2edff700 1 -- 172.21.15.31:0/2348143739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 msgr2=0x7f6e28073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.210+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2348143739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28073c20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009990 tx=0x7f6e14009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 -- 172.21.15.31:0/2348143739 shutdown_connections 2023-07-29T16:25:42.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2348143739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e281081e0 0x7f6e2810a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2348143739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e28074d90 0x7f6e280731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2348143739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28073c20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 -- 172.21.15.31:0/2348143739 >> 172.21.15.31:0/2348143739 conn(0x7f6e280fb920 msgr2=0x7f6e280fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 -- 172.21.15.31:0/2348143739 shutdown_connections 2023-07-29T16:25:42.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.211+0000 7f6e2edff700 1 -- 172.21.15.31:0/2348143739 wait complete. 2023-07-29T16:25:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 Processor -- start 2023-07-29T16:25:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 -- start start 2023-07-29T16:25:42.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e28074d90 0x7f6e28198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e281081e0 0x7f6e2819f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2819fe90 con 0x7f6e28074d90 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2819ffd0 con 0x7f6e280737b0 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.212+0000 7f6e2edff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e281a0160 con 0x7f6e281081e0 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46618/0 (socket says 172.21.15.31:46618) 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/2529967143 learned_addr learned my addr 172.21.15.31:0/2529967143 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2d39c700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e281081e0 0x7f6e2819f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e27fff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e28074d90 0x7f6e28198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/2529967143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e281081e0 msgr2=0x7f6e2819f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e281081e0 0x7f6e2819f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e28074d90 msgr2=0x7f6e28198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e28074d90 0x7f6e28198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/2529967143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c009c40 con 0x7f6e280737b0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.213+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28198310 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00c140 tx=0x7f6e1c00c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.214+0000 7f6e25ffb700 1 -- 172.21.15.31:0/2529967143 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c017070 con 0x7f6e280737b0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.214+0000 7f6e25ffb700 1 -- 172.21.15.31:0/2529967143 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e1c00f040 con 0x7f6e280737b0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.214+0000 7f6e25ffb700 1 -- 172.21.15.31:0/2529967143 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00e050 con 0x7f6e280737b0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.214+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e140095f0 con 0x7f6e280737b0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.214+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e281a0710 con 0x7f6e280737b0 2023-07-29T16:25:42.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.217+0000 7f6e25ffb700 1 -- 172.21.15.31:0/2529967143 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6e1c00cad0 con 0x7f6e280737b0 2023-07-29T16:25:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.219+0000 7f6e427bbb80 1 Processor -- start 2023-07-29T16:25:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.219+0000 7f6e427bbb80 1 -- start start 2023-07-29T16:25:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13551ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d13552d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13553740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d134a9a10 con 0x559d13551be0 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d134a73d0 con 0x559d13552530 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d134a2cc0 con 0x559d135532f0 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e25ffb700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e180a0810 0x7f6e180a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2d39c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13551ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d13552d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e27fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13553740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d13552d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46622/0 (socket says 172.21.15.31:46622) 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/804713253 learned_addr learned my addr 172.21.15.31:0/804713253 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e27fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13553740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50688/0 (socket says 172.21.15.31:50688) 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e25ffb700 1 -- 172.21.15.31:0/2529967143 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6e1c0cb410 con 0x7f6e280737b0 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2d39c700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e180a0810 0x7f6e180a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/804713253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 msgr2=0x559d13553740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13553740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/804713253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 msgr2=0x559d13551ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13551ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/804713253 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d132d0de0 con 0x559d13552530 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.220+0000 7f6e27fff700 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13553740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.221+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d13552d60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0c8e50 tx=0x7f6e1c098900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a10b7dbb6eb2665 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.221+0000 7f6e2d39c700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e180a0810 0x7f6e180a2cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6e200097a0 tx=0x7f6e20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.221+0000 7f6e12ffd700 1 -- 172.21.15.31:0/804713253 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c098410 con 0x559d13552530 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.221+0000 7f6e12ffd700 1 -- 172.21.15.31:0/804713253 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e1c0b1260 con 0x559d13552530 2023-07-29T16:25:42.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.221+0000 7f6e12ffd700 1 -- 172.21.15.31:0/804713253 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c096e50 con 0x559d13552530 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 -- 172.21.15.31:0/804713253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 msgr2=0x559d13552d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d13552d60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0c8e50 tx=0x7f6e1c098900 comp rx=0 tx=0).stop 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 -- 172.21.15.31:0/804713253 shutdown_connections 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13553740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13551ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/804713253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d13552d60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 -- 172.21.15.31:0/804713253 >> 172.21.15.31:0/804713253 conn(0x559d135473e0 msgr2=0x559d1354c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 -- 172.21.15.31:0/804713253 shutdown_connections 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 -- 172.21.15.31:0/804713253 wait complete. 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.222+0000 7f6e427bbb80 1 -- start start 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 Processor -- start 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 0x559d1355b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13552530 0x559d1355bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d135532f0 0x559d13562d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1343f510 con 0x559d135532f0 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1343f1b0 con 0x559d13551be0 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13326920 con 0x559d13552530 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e427bbb80 10 client.0 _ll_register_callbacks cb 0x559d13448620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 0x559d1355b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 0x559d1355b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46636/0 (socket says 172.21.15.31:46636) 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/4262345100 learned_addr learned my addr 172.21.15.31:0/4262345100 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:42.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.223+0000 7f6e27fff700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13552530 0x559d1355bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2d39c700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d135532f0 0x559d13562d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/4262345100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13552530 msgr2=0x559d1355bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13552530 0x559d1355bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/4262345100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d135532f0 msgr2=0x559d13562d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d135532f0 0x559d13562d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/4262345100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2810b280 con 0x559d13551be0 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 0x559d1355b790 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0987b0 tx=0x7f6e1c0986b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d002522450b7d1e8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e07fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e07fff700 1 -- 172.21.15.31:0/4262345100 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c098ab0 con 0x559d13551be0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d134cdd10 con 0x559d13551be0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e07fff700 1 -- 172.21.15.31:0/4262345100 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6e1c0092e0 con 0x559d13551be0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e07fff700 1 -- 172.21.15.31:0/4262345100 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e1c096e50 con 0x559d13551be0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e427bbb80 10 client.28520 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.224+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f6e281a0710 con 0x559d13551be0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e07fff700 1 -- 172.21.15.31:0/4262345100 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e1c0ac020 con 0x559d13551be0 2023-07-29T16:25:42.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e07fff700 1 -- 172.21.15.31:0/4262345100 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6e1c004ad0 con 0x559d13551be0 2023-07-29T16:25:42.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e427bbb80 10 client.28520 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:42.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e427bbb80 10 client.28520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:42.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e427bbb80 20 client.28520 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:42.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d13567090 0x559d135694a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e427bbb80 4 client.28520 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:42.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.225+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d13520d70 con 0x559d13567090 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.226+0000 7f6e27fff700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d13567090 0x559d135694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.226+0000 7f6e27fff700 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d13567090 0x559d135694a0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26647be38471e4f1 server_cookie=8de43fb55096a80e in_seq=0 out_seq=0 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.226+0000 7f6e07fff700 10 client.28520 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e07fff700 1 -- 172.21.15.31:0/4262345100 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559d13520d70 con 0x559d13567090 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e07fff700 10 client.28520 handle_command_reply: tid=0 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 client.28520 shutdown 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 msgr2=0x559d1355b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 0x559d1355b790 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c0987b0 tx=0x7f6e1c0986b0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 shutdown_connections 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d13567090 0x559d135694a0 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d135532f0 0x559d13562d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13551be0 0x559d1355b790 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/4262345100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13552530 0x559d1355bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 >> 172.21.15.31:0/4262345100 conn(0x559d135473e0 msgr2=0x559d1354c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e07fff700 0 client.28520 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:42.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.227+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 shutdown_connections 2023-07-29T16:25:42.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.228+0000 7f6e427bbb80 1 -- 172.21.15.31:0/4262345100 wait complete. 2023-07-29T16:25:42.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.228+0000 7f6e427bbb80 20 client.28520 trim_cache size 0 max 16384 2023-07-29T16:25:42.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 Processor -- start 2023-07-29T16:25:42.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 -- start start 2023-07-29T16:25:42.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13560cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d1354c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13559780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13549460 con 0x559d13551be0 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13546ea0 con 0x559d13552530 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.246+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d1354dcc0 con 0x559d135532f0 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13560cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e27fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d1354c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13560cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58694/0 (socket says 172.21.15.31:58694) 2023-07-29T16:25:42.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/553709958 learned_addr learned my addr 172.21.15.31:0/553709958 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:42.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e2d39c700 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13559780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e27fff700 1 -- 172.21.15.31:0/553709958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 msgr2=0x559d13559780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e27fff700 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13559780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e27fff700 1 -- 172.21.15.31:0/553709958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 msgr2=0x559d13560cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e27fff700 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13560cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.247+0000 7f6e27fff700 1 -- 172.21.15.31:0/553709958 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d132b9c60 con 0x559d13552530 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.248+0000 7f6e27fff700 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d1354c060 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6e140098e0 tx=0x7f6e14005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc468ef5c7aacfb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.248+0000 7f6e137fe700 1 -- 172.21.15.31:0/553709958 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1401d040 con 0x559d13552530 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.248+0000 7f6e137fe700 1 -- 172.21.15.31:0/553709958 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e1401e040 con 0x559d13552530 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.248+0000 7f6e137fe700 1 -- 172.21.15.31:0/553709958 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1400f460 con 0x559d13552530 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 -- 172.21.15.31:0/553709958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 msgr2=0x559d1354c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d1354c060 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6e140098e0 tx=0x7f6e14005fb0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 -- 172.21.15.31:0/553709958 shutdown_connections 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d135532f0 0x559d13559780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13551be0 0x559d13560cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/553709958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d13552530 0x559d1354c060 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 -- 172.21.15.31:0/553709958 >> 172.21.15.31:0/553709958 conn(0x559d135473e0 msgr2=0x559d1355a450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 -- 172.21.15.31:0/553709958 shutdown_connections 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 -- 172.21.15.31:0/553709958 wait complete. 2023-07-29T16:25:42.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 -- start start 2023-07-29T16:25:42.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.249+0000 7f6e427bbb80 1 Processor -- start 2023-07-29T16:25:42.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 0x559d135b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13552530 0x559d135b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d135532f0 0x559d135b8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d135b8a60 con 0x559d13552530 2023-07-29T16:25:42.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d135b8ba0 con 0x559d135532f0 2023-07-29T16:25:42.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d135b8d30 con 0x559d13551be0 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e427bbb80 10 client.0 _ll_register_callbacks cb 0x559d13448620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 0x559d135b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e2cb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 0x559d135b0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50714/0 (socket says 172.21.15.31:50714) 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/3567260462 learned_addr learned my addr 172.21.15.31:0/3567260462 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e2d39c700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d135532f0 0x559d135b8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.250+0000 7f6e27fff700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13552530 0x559d135b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d135532f0 msgr2=0x559d135b8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d135532f0 0x559d135b8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/3567260462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13552530 msgr2=0x559d135b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13552530 0x559d135b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e2cb9b700 1 -- 172.21.15.31:0/3567260462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e14009290 con 0x559d13551be0 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e2cb9b700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 0x559d135b0fa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c096e00 tx=0x7f6e1c003dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33000876c2580f03 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e07fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e07fff700 1 -- 172.21.15.31:0/3567260462 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c098ab0 con 0x559d13551be0 2023-07-29T16:25:42.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6e1c0b1260 con 0x559d13551be0 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.251+0000 7f6e07fff700 1 -- 172.21.15.31:0/3567260462 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6e1c0a0f00 con 0x559d13551be0 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e07fff700 1 -- 172.21.15.31:0/3567260462 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e1c0a0a70 con 0x559d13551be0 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 10 client.28531 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559d132f7250 con 0x559d13551be0 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e07fff700 1 -- 172.21.15.31:0/3567260462 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e1c0b1260 con 0x559d13551be0 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e07fff700 1 -- 172.21.15.31:0/3567260462 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6e1c097a10 con 0x559d13551be0 2023-07-29T16:25:42.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 10 client.28531 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 10 client.28531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 20 client.28531 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d135ba0c0 0x559d135bc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 4 client.28531 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559d1349bd40 con 0x559d135ba0c0 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.252+0000 7f6e27fff700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d135ba0c0 0x559d135bc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.253+0000 7f6e27fff700 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d135ba0c0 0x559d135bc4a0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c05c42c6d0fd589 server_cookie=f9c01b837fa010d4 in_seq=0 out_seq=0 2023-07-29T16:25:42.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.253+0000 7f6e07fff700 10 client.28531 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e07fff700 1 -- 172.21.15.31:0/3567260462 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559d1349bd40 con 0x559d135ba0c0 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e07fff700 10 client.28531 handle_command_reply: tid=0 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 client.28531 shutdown 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 msgr2=0x559d135b0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 0x559d135b0fa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c096e00 tx=0x7f6e1c003dc0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 shutdown_connections 2023-07-29T16:25:42.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d135ba0c0 0x559d135bc4a0 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d135532f0 0x559d135b8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d13551be0 0x559d135b0fa0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 --2- 172.21.15.31:0/3567260462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d13552530 0x559d135b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e07fff700 0 client.28531 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 >> 172.21.15.31:0/3567260462 conn(0x559d135473e0 msgr2=0x559d1354c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 shutdown_connections 2023-07-29T16:25:42.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 1 -- 172.21.15.31:0/3567260462 wait complete. 2023-07-29T16:25:42.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.254+0000 7f6e427bbb80 20 client.28531 trim_cache size 0 max 16384 2023-07-29T16:25:42.260 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:25:42.260 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:25:42.260 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:25:42.261 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:25:42.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e180a0810 msgr2=0x7f6e180a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e180a0810 0x7f6e180a2cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6e200097a0 tx=0x7f6e20009260 comp rx=0 tx=0).stop 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 msgr2=0x7f6e28198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28198310 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c00c140 tx=0x7f6e1c00c450 comp rx=0 tx=0).stop 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 shutdown_connections 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6e180a0810 0x7f6e180a2cc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e281081e0 0x7f6e2819f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6e280737b0 0x7f6e28198310 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 --2- 172.21.15.31:0/2529967143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6e28074d90 0x7f6e28198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 >> 172.21.15.31:0/2529967143 conn(0x7f6e280fb920 msgr2=0x7f6e280fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.255+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 shutdown_connections 2023-07-29T16:25:42.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.256+0000 7f6e2edff700 1 -- 172.21.15.31:0/2529967143 wait complete. 2023-07-29T16:25:42.281 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:25:42.282 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:25:42.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 -- 172.21.15.31:0/2486293663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101530 msgr2=0x7f90ac101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 --2- 172.21.15.31:0/2486293663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101530 0x7f90ac101980 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f909c009990 tx=0x7f909c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 -- 172.21.15.31:0/2486293663 shutdown_connections 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 --2- 172.21.15.31:0/2486293663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90ac101ec0 0x7f90ac108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 --2- 172.21.15.31:0/2486293663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 --2- 172.21.15.31:0/2486293663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101530 0x7f90ac101980 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 -- 172.21.15.31:0/2486293663 >> 172.21.15.31:0/2486293663 conn(0x7f90ac0fb920 msgr2=0x7f90ac0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.507+0000 7f90b43ff700 1 -- 172.21.15.31:0/2486293663 shutdown_connections 2023-07-29T16:25:42.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.508+0000 7f90b43ff700 1 -- 172.21.15.31:0/2486293663 wait complete. 2023-07-29T16:25:42.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.508+0000 7f90b43ff700 1 Processor -- start 2023-07-29T16:25:42.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 -- start start 2023-07-29T16:25:42.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90ac101530 0x7f90ac073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 0x7f90ac073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac073e20 con 0x7f90ac101530 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac073f90 con 0x7f90ac101ec0 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b43ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ac1a0110 con 0x7f90ac100340 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b199a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90ac101530 0x7f90ac073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b299c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 0x7f90ac073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b219b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50736/0 (socket says 172.21.15.31:50736) 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b299c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 0x7f90ac073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46692/0 (socket says 172.21.15.31:46692) 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.509+0000 7f90b219b700 1 -- 172.21.15.31:0/3846945768 learned_addr learned my addr 172.21.15.31:0/3846945768 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b299c700 1 -- 172.21.15.31:0/3846945768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 msgr2=0x7f90ac074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b299c700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b299c700 1 -- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90ac101530 msgr2=0x7f90ac073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b299c700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90ac101530 0x7f90ac073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b299c700 1 -- 172.21.15.31:0/3846945768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c0095f0 con 0x7f90ac101ec0 2023-07-29T16:25:42.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b219b700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:42.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b299c700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 0x7f90ac073680 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f90a400efc0 tx=0x7f90a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a4018070 con 0x7f90ac101ec0 2023-07-29T16:25:42.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90a400f040 con 0x7f90ac101ec0 2023-07-29T16:25:42.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a400cd40 con 0x7f90ac101ec0 2023-07-29T16:25:42.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac1a0390 con 0x7f90ac101ec0 2023-07-29T16:25:42.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.510+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac1a0850 con 0x7f90ac101ec0 2023-07-29T16:25:42.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.511+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac107870 con 0x7f90ac101ec0 2023-07-29T16:25:42.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.514+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f90a4003a00 con 0x7f90ac101ec0 2023-07-29T16:25:42.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.515+0000 7f90a37fe700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90980a0860 0x7f90980a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:42.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.515+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f90a40cbfe0 con 0x7f90ac101ec0 2023-07-29T16:25:42.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.515+0000 7f90b219b700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90980a0860 0x7f90980a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:42.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.516+0000 7f90b219b700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90980a0860 0x7f90980a2d10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005950 tx=0x7f90a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:42.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.517+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f90a4095a70 con 0x7f90ac101ec0 2023-07-29T16:25:42.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:42 smithi031 ceph-mon[95504]: pgmap v678: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.9 MiB/s wr, 229 op/s 2023-07-29T16:25:42.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.787+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90ac02cc30 con 0x7f90ac101ec0 2023-07-29T16:25:42.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.788+0000 7f90a37fe700 1 -- 172.21.15.31:0/3846945768 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10781 (secure 0 0 0) 0x7f90a4099090 con 0x7f90ac101ec0 2023-07-29T16:25:42.787 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:25:42.787 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:24:56.527016+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:25:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90980a0860 msgr2=0x7f90980a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90980a0860 0x7f90980a2d10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f90a8005950 tx=0x7f90a8009260 comp rx=0 tx=0).stop 2023-07-29T16:25:42.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 msgr2=0x7f90ac073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 0x7f90ac073680 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f90a400efc0 tx=0x7f90a400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 shutdown_connections 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f90980a0860 0x7f90980a2d10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90ac101ec0 0x7f90ac073680 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f90ac100340 0x7f90ac074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 --2- 172.21.15.31:0/3846945768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90ac101530 0x7f90ac073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:42.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 >> 172.21.15.31:0/3846945768 conn(0x7f90ac0fb920 msgr2=0x7f90ac1049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.790+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 shutdown_connections 2023-07-29T16:25:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:42.791+0000 7f90b43ff700 1 -- 172.21.15.31:0/3846945768 wait complete. 2023-07-29T16:25:42.791 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2023-07-29T16:25:42.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:25:42.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:42 smithi156 ceph-mon[129245]: pgmap v678: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.9 MiB/s wr, 229 op/s 2023-07-29T16:25:42.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 739765 8.88 MB/sec execute 327 sec latency 55.906 ms 2023-07-29T16:25:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 -- 172.21.15.31:0/3761598309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330100320 msgr2=0x7f1330100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 --2- 172.21.15.31:0/3761598309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330100320 0x7f1330100730 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f132c009990 tx=0x7f132c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 -- 172.21.15.31:0/3761598309 shutdown_connections 2023-07-29T16:25:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 --2- 172.21.15.31:0/3761598309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330101ea0 0x7f1330108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 --2- 172.21.15.31:0/3761598309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330100320 0x7f1330100730 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 --2- 172.21.15.31:0/3761598309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330101510 0x7f1330101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 -- 172.21.15.31:0/3761598309 >> 172.21.15.31:0/3761598309 conn(0x7f13300fb920 msgr2=0x7f13300fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 -- 172.21.15.31:0/3761598309 shutdown_connections 2023-07-29T16:25:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.024+0000 7f1337bff700 1 -- 172.21.15.31:0/3761598309 wait complete. 2023-07-29T16:25:43.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.026+0000 7f1337bff700 1 Processor -- start 2023-07-29T16:25:43.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.026+0000 7f1337bff700 1 -- start start 2023-07-29T16:25:43.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.026+0000 7f1337bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 0x7f13301982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.026+0000 7f1337bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330101510 0x7f13301987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f1337bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330101ea0 0x7f133019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f1337bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133019fe30 con 0x7f1330100320 2023-07-29T16:25:43.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f1337bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133019ff70 con 0x7f1330101ea0 2023-07-29T16:25:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f1337bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13301a0100 con 0x7f1330101510 2023-07-29T16:25:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 0x7f13301982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 0x7f13301982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58732/0 (socket says 172.21.15.31:58732) 2023-07-29T16:25:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 -- 172.21.15.31:0/419606671 learned_addr learned my addr 172.21.15.31:0/419606671 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133519a700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330101510 0x7f13301987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133619c700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330101ea0 0x7f133019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:42 smithi039 ceph-mon[128858]: pgmap v678: 97 pgs: 97 active+clean; 139 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 1.9 MiB/s wr, 229 op/s 2023-07-29T16:25:43.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 -- 172.21.15.31:0/419606671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330101510 msgr2=0x7f13301987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330101510 0x7f13301987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 -- 172.21.15.31:0/419606671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330101ea0 msgr2=0x7f133019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330101ea0 0x7f133019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.027+0000 7f133599b700 1 -- 172.21.15.31:0/419606671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1320009520 con 0x7f1330100320 2023-07-29T16:25:43.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.028+0000 7f133599b700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 0x7f13301982b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f132c005720 tx=0x7f132c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.028+0000 7f1326ffd700 1 -- 172.21.15.31:0/419606671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c01d070 con 0x7f1330100320 2023-07-29T16:25:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.028+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132c0095f0 con 0x7f1330100320 2023-07-29T16:25:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.028+0000 7f1326ffd700 1 -- 172.21.15.31:0/419606671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f132c021410 con 0x7f1330100320 2023-07-29T16:25:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.028+0000 7f1326ffd700 1 -- 172.21.15.31:0/419606671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c017650 con 0x7f1330100320 2023-07-29T16:25:43.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.028+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13301a06c0 con 0x7f1330100320 2023-07-29T16:25:43.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.032+0000 7f1326ffd700 1 -- 172.21.15.31:0/419606671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f132c021580 con 0x7f1330100320 2023-07-29T16:25:43.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.034+0000 7f1326ffd700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f131c0a0790 0x7f131c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.034+0000 7f1326ffd700 1 -- 172.21.15.31:0/419606671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f132c0ccd40 con 0x7f1330100320 2023-07-29T16:25:43.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.034+0000 7f133519a700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f131c0a0790 0x7f131c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.034+0000 7f134b4c0b80 1 Processor -- start 2023-07-29T16:25:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.034+0000 7f134b4c0b80 1 -- start start 2023-07-29T16:25:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b78783ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78784530 0x558b78784d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133519a700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f131c0a0790 0x7f131c0a2c40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1320009df0 tx=0x7f1320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b787852f0 0x558b78785740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b786dba10 con 0x558b787852f0 2023-07-29T16:25:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b786d93d0 con 0x558b78784530 2023-07-29T16:25:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b786d4cc0 con 0x558b78783be0 2023-07-29T16:25:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b78783ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b78783ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50764/0 (socket says 172.21.15.31:50764) 2023-07-29T16:25:43.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 -- 172.21.15.31:0/2784316507 learned_addr learned my addr 172.21.15.31:0/2784316507 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133519a700 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78784530 0x558b78784d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133619c700 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b787852f0 0x558b78785740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 -- 172.21.15.31:0/2784316507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78784530 msgr2=0x558b78784d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78784530 0x558b78784d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 -- 172.21.15.31:0/2784316507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b787852f0 msgr2=0x558b78785740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b787852f0 0x558b78785740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133599b700 1 -- 172.21.15.31:0/2784316507 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b78502de0 con 0x558b78783be0 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.035+0000 7f133519a700 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78784530 0x558b78784d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.036+0000 7f133599b700 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b78783ff0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f132c0a2360 tx=0x7f132c09a010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5adec95f457377ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.036+0000 7f131bfff700 1 -- 172.21.15.31:0/2784316507 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c005750 con 0x558b78783be0 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.036+0000 7f131bfff700 1 -- 172.21.15.31:0/2784316507 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f132c0af020 con 0x558b78783be0 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.036+0000 7f131bfff700 1 -- 172.21.15.31:0/2784316507 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c09a730 con 0x558b78783be0 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 -- 172.21.15.31:0/2784316507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 msgr2=0x558b78783ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b78783ff0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f132c0a2360 tx=0x7f132c09a010 comp rx=0 tx=0).stop 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 -- 172.21.15.31:0/2784316507 shutdown_connections 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b787852f0 0x558b78785740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b78783ff0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/2784316507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78784530 0x558b78784d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 -- 172.21.15.31:0/2784316507 >> 172.21.15.31:0/2784316507 conn(0x558b787793e0 msgr2=0x558b7877e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 -- 172.21.15.31:0/2784316507 shutdown_connections 2023-07-29T16:25:43.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.037+0000 7f134b4c0b80 1 -- 172.21.15.31:0/2784316507 wait complete. 2023-07-29T16:25:43.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.038+0000 7f134b4c0b80 1 -- start start 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.038+0000 7f134b4c0b80 1 Processor -- start 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 0x558b7878d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78784530 0x558b7878d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b78794980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b78671510 con 0x558b78783be0 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b786711b0 con 0x558b787852f0 2023-07-29T16:25:43.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b78558920 con 0x558b78784530 2023-07-29T16:25:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f134b4c0b80 10 client.0 _ll_register_callbacks cb 0x558b7867a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 0x558b7878d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 0x558b7878d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58750/0 (socket says 172.21.15.31:58750) 2023-07-29T16:25:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 -- 172.21.15.31:0/3884006071 learned_addr learned my addr 172.21.15.31:0/3884006071 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133619c700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b78794980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133519a700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78784530 0x558b7878d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 -- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78784530 msgr2=0x558b7878d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78784530 0x558b7878d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 -- 172.21.15.31:0/3884006071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 msgr2=0x558b78794980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b78794980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.039+0000 7f133599b700 1 -- 172.21.15.31:0/3884006071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1320017070 con 0x558b78783be0 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f133599b700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 0x558b7878d400 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f132c096250 tx=0x7f132c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb8f4fb48d3f1b3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f13197fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f13197fa700 1 -- 172.21.15.31:0/3884006071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c0b6070 con 0x558b78783be0 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f13197fa700 1 -- 172.21.15.31:0/3884006071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f132c0b0070 con 0x558b78783be0 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b786ffd10 con 0x558b78783be0 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f13197fa700 1 -- 172.21.15.31:0/3884006071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f132c005750 con 0x558b78783be0 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f134b4c0b80 10 client.18468 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.040+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f13301a06c0 con 0x558b78783be0 2023-07-29T16:25:43.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f13197fa700 1 -- 172.21.15.31:0/3884006071 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558b786ffd10 con 0x558b78783be0 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f13197fa700 1 -- 172.21.15.31:0/3884006071 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f132c0ca240 con 0x558b78783be0 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f134b4c0b80 10 client.18468 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f134b4c0b80 10 client.18468 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f134b4c0b80 20 client.18468 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b78798cf0 0x558b7879b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f134b4c0b80 4 client.18468 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b78752d70 con 0x558b78798cf0 2023-07-29T16:25:43.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.041+0000 7f133519a700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b78798cf0 0x558b7879b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.042+0000 7f13197fa700 10 client.18468 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.042+0000 7f133519a700 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b78798cf0 0x558b7879b0d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d49b94f31239f2d server_cookie=59f61ca271ccd159 in_seq=0 out_seq=0 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f13197fa700 1 -- 172.21.15.31:0/3884006071 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558b78752d70 con 0x558b78798cf0 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f13197fa700 10 client.18468 handle_command_reply: tid=0 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 client.18468 shutdown 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 msgr2=0x558b7878d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 0x558b7878d400 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f132c096250 tx=0x7f132c005ef0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 shutdown_connections 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b78798cf0 0x558b7879b0d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b78794980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78783be0 0x558b7878d400 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3884006071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78784530 0x558b7878d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f13197fa700 0 client.18468 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 >> 172.21.15.31:0/3884006071 conn(0x558b787793e0 msgr2=0x558b7877e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.043+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 shutdown_connections 2023-07-29T16:25:43.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.044+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3884006071 wait complete. 2023-07-29T16:25:43.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.044+0000 7f134b4c0b80 20 client.18468 trim_cache size 0 max 16384 2023-07-29T16:25:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.059+0000 7f134b4c0b80 1 Processor -- start 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.059+0000 7f134b4c0b80 1 -- start start 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b7878d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78794940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b7878d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b78776770 con 0x558b78784530 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b7878b3b0 con 0x558b787852f0 2023-07-29T16:25:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b78784b50 con 0x558b78783be0 2023-07-29T16:25:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b7878d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b7878d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50782/0 (socket says 172.21.15.31:50782) 2023-07-29T16:25:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 -- 172.21.15.31:0/3571262761 learned_addr learned my addr 172.21.15.31:0/3571262761 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133619c700 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b7878d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133519a700 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78794940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 -- 172.21.15.31:0/3571262761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 msgr2=0x558b7878d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b7878d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 -- 172.21.15.31:0/3571262761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 msgr2=0x558b78794940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78794940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.060+0000 7f133599b700 1 -- 172.21.15.31:0/3571262761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b784ebc60 con 0x558b78783be0 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.061+0000 7f133599b700 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b7878d1b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f132c0969d0 tx=0x7f132c09e170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4098c9dbdb76f4ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.061+0000 7f1324bff700 1 -- 172.21.15.31:0/3571262761 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c0b6070 con 0x558b78783be0 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.061+0000 7f1324bff700 1 -- 172.21.15.31:0/3571262761 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f132c005750 con 0x558b78783be0 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.061+0000 7f1324bff700 1 -- 172.21.15.31:0/3571262761 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c09a730 con 0x558b78783be0 2023-07-29T16:25:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3571262761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 msgr2=0x558b7878d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b7878d1b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f132c0969d0 tx=0x7f132c09e170 comp rx=0 tx=0).stop 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3571262761 shutdown_connections 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b787852f0 0x558b7878d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b78783be0 0x558b7878d1b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/3571262761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78794940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3571262761 >> 172.21.15.31:0/3571262761 conn(0x558b787793e0 msgr2=0x558b78788150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3571262761 shutdown_connections 2023-07-29T16:25:43.064 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 -- 172.21.15.31:0/3571262761 wait complete. 2023-07-29T16:25:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.062+0000 7f134b4c0b80 1 -- start start 2023-07-29T16:25:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 Processor -- start 2023-07-29T16:25:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 0x558b78790c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78791170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b787852f0 0x558b78794190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b787946d0 con 0x558b78784530 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b78794840 con 0x558b78783be0 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b78794a00 con 0x558b787852f0 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.063+0000 7f134b4c0b80 10 client.0 _ll_register_callbacks cb 0x558b7867a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 0x558b78790c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 0x558b78790c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46746/0 (socket says 172.21.15.31:46746) 2023-07-29T16:25:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 -- 172.21.15.31:0/265614454 learned_addr learned my addr 172.21.15.31:0/265614454 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133619c700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b787852f0 0x558b78794190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133519a700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78791170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 -- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b787852f0 msgr2=0x558b78794190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b787852f0 0x558b78794190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 -- 172.21.15.31:0/265614454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 msgr2=0x558b78791170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78791170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 -- 172.21.15.31:0/265614454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13301a06c0 con 0x558b78783be0 2023-07-29T16:25:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133519a700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78791170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f133599b700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 0x558b78790c30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f132c00b680 tx=0x7f132c09a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=969a35c2721877bf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.064+0000 7f13197fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f13197fa700 1 -- 172.21.15.31:0/265614454 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c0b6030 con 0x558b78783be0 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f13197fa700 1 -- 172.21.15.31:0/265614454 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f132c064070 con 0x558b78783be0 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f132c0af020 con 0x558b78783be0 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f13197fa700 1 -- 172.21.15.31:0/265614454 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f132c098940 con 0x558b78783be0 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 10 client.28550 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558b78529250 con 0x558b78783be0 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f13197fa700 1 -- 172.21.15.31:0/265614454 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f132c0af020 con 0x558b78783be0 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f13197fa700 1 -- 172.21.15.31:0/265614454 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f132c066310 con 0x558b78783be0 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 10 client.28550 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 10 client.28550 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 20 client.28550 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b787eb590 0x558b787ed970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 4 client.28550 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.065+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558b786cdd40 con 0x558b787eb590 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f133519a700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b787eb590 0x558b787ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f13197fa700 10 client.28550 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f133519a700 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b787eb590 0x558b787ed970 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87b09889171c8954 server_cookie=99fd5b20a8e0a40a in_seq=0 out_seq=0 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f13197fa700 1 -- 172.21.15.31:0/265614454 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558b786cdd40 con 0x558b787eb590 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f13197fa700 10 client.28550 handle_command_reply: tid=0 2023-07-29T16:25:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f134b4c0b80 1 client.28550 shutdown 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 msgr2=0x558b78790c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 0x558b78790c30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f132c00b680 tx=0x7f132c09a010 comp rx=0 tx=0).stop 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 shutdown_connections 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b787852f0 0x558b78794190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.066+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x558b787eb590 0x558b787ed970 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b78783be0 0x558b78790c30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f134b4c0b80 1 --2- 172.21.15.31:0/265614454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b78784530 0x558b78791170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f13197fa700 0 client.28550 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:25:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 >> 172.21.15.31:0/265614454 conn(0x558b787793e0 msgr2=0x558b7878d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 shutdown_connections 2023-07-29T16:25:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f134b4c0b80 1 -- 172.21.15.31:0/265614454 wait complete. 2023-07-29T16:25:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f134b4c0b80 20 client.28550 trim_cache size 0 max 16384 2023-07-29T16:25:43.074 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:25:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f131c0a0790 msgr2=0x7f131c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f1337bff700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f131c0a0790 0x7f131c0a2c40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1320009df0 tx=0x7f1320009260 comp rx=0 tx=0).stop 2023-07-29T16:25:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 msgr2=0x7f13301982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.067+0000 7f1337bff700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 0x7f13301982b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f132c005720 tx=0x7f132c009580 comp rx=0 tx=0).stop 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 shutdown_connections 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f131c0a0790 0x7f131c0a2c40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330101ea0 0x7f133019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100320 0x7f13301982b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 --2- 172.21.15.31:0/419606671 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330101510 0x7f13301987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 >> 172.21.15.31:0/419606671 conn(0x7f13300fb920 msgr2=0x7f1330106e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 shutdown_connections 2023-07-29T16:25:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.068+0000 7f1337bff700 1 -- 172.21.15.31:0/419606671 wait complete. 2023-07-29T16:25:43.086 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:25:43.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 -- 172.21.15.31:0/2081136305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c1040d0 msgr2=0x7f396c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 --2- 172.21.15.31:0/2081136305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c1040d0 0x7f396c10a990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f396400b0a0 tx=0x7f396400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 -- 172.21.15.31:0/2081136305 shutdown_connections 2023-07-29T16:25:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 --2- 172.21.15.31:0/2081136305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c1040d0 0x7f396c10a990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 --2- 172.21.15.31:0/2081136305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 --2- 172.21.15.31:0/2081136305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c103740 0x7f396c103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 -- 172.21.15.31:0/2081136305 >> 172.21.15.31:0/2081136305 conn(0x7f396c0fdb90 msgr2=0x7f396c0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 -- 172.21.15.31:0/2081136305 shutdown_connections 2023-07-29T16:25:43.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.332+0000 7f39731ff700 1 -- 172.21.15.31:0/2081136305 wait complete. 2023-07-29T16:25:43.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.333+0000 7f39731ff700 1 Processor -- start 2023-07-29T16:25:43.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 -- start start 2023-07-29T16:25:43.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c103740 0x7f396c1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c1040d0 0x7f396c19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c19ff00 con 0x7f396c1040d0 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1a0040 con 0x7f396c103740 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f39731ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1a01d0 con 0x7f396c102550 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f3970f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f3970f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50812/0 (socket says 172.21.15.31:50812) 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f3970f9b700 1 -- 172.21.15.31:0/2498917101 learned_addr learned my addr 172.21.15.31:0/2498917101 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.334+0000 7f397179c700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c1040d0 0x7f396c19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f396bfff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c103740 0x7f396c1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3970f9b700 1 -- 172.21.15.31:0/2498917101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c103740 msgr2=0x7f396c1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3970f9b700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c103740 0x7f396c1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3970f9b700 1 -- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c1040d0 msgr2=0x7f396c19f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3970f9b700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c1040d0 0x7f396c19f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3970f9b700 1 -- 172.21.15.31:0/2498917101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958009520 con 0x7f396c102550 2023-07-29T16:25:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f396bfff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c103740 0x7f396c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3970f9b700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c198380 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f396000e970 tx=0x7f396000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3969ffb700 1 -- 172.21.15.31:0/2498917101 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960009790 con 0x7f396c102550 2023-07-29T16:25:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3969ffb700 1 -- 172.21.15.31:0/2498917101 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f396000eec0 con 0x7f396c102550 2023-07-29T16:25:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f3969ffb700 1 -- 172.21.15.31:0/2498917101 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960003cf0 con 0x7f396c102550 2023-07-29T16:25:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.335+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3964009d00 con 0x7f396c102550 2023-07-29T16:25:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.336+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c1a0830 con 0x7f396c102550 2023-07-29T16:25:43.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.339+0000 7f3969ffb700 1 -- 172.21.15.31:0/2498917101 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f396000ccd0 con 0x7f396c102550 2023-07-29T16:25:43.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.340+0000 7f3986ac4b80 1 Processor -- start 2023-07-29T16:25:43.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.340+0000 7f3986ac4b80 1 -- start start 2023-07-29T16:25:43.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.340+0000 7f3969ffb700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f395c0a0810 0x7f395c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.340+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce144bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3969ffb700 1 -- 172.21.15.31:0/2498917101 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3960014070 con 0x7f396c102550 2023-07-29T16:25:43.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144c530 0x560ce144cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 0x560ce144d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce13a3a10 con 0x560ce144bbe0 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce13a13d0 con 0x560ce144d2f0 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce139ccc0 con 0x560ce144c530 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f397179c700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f395c0a0810 0x7f395c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f397179c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 0x560ce144d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3970f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce144bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f397179c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 0x560ce144d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46772/0 (socket says 172.21.15.31:46772) 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f397179c700 1 -- 172.21.15.31:0/3175689951 learned_addr learned my addr 172.21.15.31:0/3175689951 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f396bfff700 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144c530 0x560ce144cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3970f9b700 1 -- 172.21.15.31:0/3175689951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144c530 msgr2=0x560ce144cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3970f9b700 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144c530 0x560ce144cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3970f9b700 1 -- 172.21.15.31:0/3175689951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 msgr2=0x560ce144d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3970f9b700 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 0x560ce144d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f3970f9b700 1 -- 172.21.15.31:0/3175689951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ce11cade0 con 0x560ce144bbe0 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.341+0000 7f397179c700 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 0x560ce144d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.342+0000 7f3970f9b700 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce144bff0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3960099170 tx=0x7f3960099080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365ea2d8a4b09726 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.342+0000 7f397179c700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f395c0a0810 0x7f395c0a2cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3964009610 tx=0x7f39640097e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.342+0000 7f3956ffd700 1 -- 172.21.15.31:0/3175689951 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960099470 con 0x560ce144bbe0 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.342+0000 7f3956ffd700 1 -- 172.21.15.31:0/3175689951 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f396009d4a0 con 0x560ce144bbe0 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.342+0000 7f3956ffd700 1 -- 172.21.15.31:0/3175689951 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3960099930 con 0x560ce144bbe0 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3175689951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 msgr2=0x560ce144bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce144bff0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3960099170 tx=0x7f3960099080 comp rx=0 tx=0).stop 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3175689951 shutdown_connections 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144d2f0 0x560ce144d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce144bff0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3175689951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144c530 0x560ce144cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3175689951 >> 172.21.15.31:0/3175689951 conn(0x560ce14413e0 msgr2=0x560ce1446990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3175689951 shutdown_connections 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3175689951 wait complete. 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.343+0000 7f3986ac4b80 1 -- start start 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 Processor -- start 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce1455890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1455dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce145ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce1339510 con 0x560ce144bbe0 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce13391b0 con 0x560ce144c530 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce1220920 con 0x560ce144d2f0 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.344+0000 7f3986ac4b80 10 client.0 _ll_register_callbacks cb 0x560ce1342620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f3970f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce1455890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1455dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1455dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46786/0 (socket says 172.21.15.31:46786) 2023-07-29T16:25:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 -- 172.21.15.31:0/454015474 learned_addr learned my addr 172.21.15.31:0/454015474 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f397179c700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce145ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 -- 172.21.15.31:0/454015474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 msgr2=0x560ce145ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce145ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 -- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 msgr2=0x560ce1455890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce1455890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 -- 172.21.15.31:0/454015474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c10b2c0 con 0x560ce144c530 2023-07-29T16:25:43.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f397179c700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce145ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f3970f9b700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce1455890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f396bfff700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1455dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f39580128d0 tx=0x7f3958012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c8a2bd61460e570 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.345+0000 7f394bfff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f394bfff700 1 -- 172.21.15.31:0/454015474 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958017070 con 0x560ce144c530 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f394bfff700 1 -- 172.21.15.31:0/454015474 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39580118d0 con 0x560ce144c530 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ce13c7d10 con 0x560ce144c530 2023-07-29T16:25:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f394bfff700 1 -- 172.21.15.31:0/454015474 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3958011a90 con 0x560ce144c530 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 10 client.28562 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f396c1a0830 con 0x560ce144c530 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f394bfff700 1 -- 172.21.15.31:0/454015474 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560ce13c7d10 con 0x560ce144c530 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f394bfff700 1 -- 172.21.15.31:0/454015474 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f395800f460 con 0x560ce144c530 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 10 client.28562 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 10 client.28562 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:25:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 20 client.28562 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce1461670 0x560ce1463a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 4 client.28562 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.346+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ce141ad70 con 0x560ce1461670 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.347+0000 7f3970f9b700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce1461670 0x560ce1463a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.347+0000 7f3970f9b700 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce1461670 0x560ce1463a80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=779d0de5f184796c server_cookie=3ea8bb70f4b1c53c in_seq=0 out_seq=0 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.347+0000 7f394bfff700 10 client.28562 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:43.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f394bfff700 1 -- 172.21.15.31:0/454015474 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560ce141ad70 con 0x560ce1461670 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f394bfff700 10 client.28562 handle_command_reply: tid=0 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 client.28562 shutdown 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 msgr2=0x560ce1455dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1455dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f39580128d0 tx=0x7f3958012be0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 shutdown_connections 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce1461670 0x560ce1463a80 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce145ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce1455890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/454015474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1455dd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 >> 172.21.15.31:0/454015474 conn(0x560ce14413e0 msgr2=0x560ce1446060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f394bfff700 0 client.28562 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 shutdown_connections 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 1 -- 172.21.15.31:0/454015474 wait complete. 2023-07-29T16:25:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.348+0000 7f3986ac4b80 20 client.28562 trim_cache size 0 max 16384 2023-07-29T16:25:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.362+0000 7f3986ac4b80 1 Processor -- start 2023-07-29T16:25:43.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.362+0000 7f3986ac4b80 1 -- start start 2023-07-29T16:25:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144bbe0 0x560ce145adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce14501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144d2f0 0x560ce1450730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce145eb40 con 0x560ce144d2f0 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce1455400 con 0x560ce144c530 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce1443490 con 0x560ce144bbe0 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f396bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce14501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f396bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce14501f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46800/0 (socket says 172.21.15.31:46800) 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f396bfff700 1 -- 172.21.15.31:0/4251306381 learned_addr learned my addr 172.21.15.31:0/4251306381 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f397179c700 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144d2f0 0x560ce1450730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.363+0000 7f3970f9b700 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144bbe0 0x560ce145adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f396bfff700 1 -- 172.21.15.31:0/4251306381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144bbe0 msgr2=0x560ce145adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f396bfff700 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144bbe0 0x560ce145adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f396bfff700 1 -- 172.21.15.31:0/4251306381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144d2f0 msgr2=0x560ce1450730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f396bfff700 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144d2f0 0x560ce1450730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f396bfff700 1 -- 172.21.15.31:0/4251306381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ce11b3c60 con 0x560ce144c530 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f3970f9b700 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144bbe0 0x560ce145adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:43.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f396bfff700 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce14501f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3958000c00 tx=0x7f3958009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9fbaa6a2af6e050 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f39577fe700 1 -- 172.21.15.31:0/4251306381 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958017040 con 0x560ce144c530 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f39577fe700 1 -- 172.21.15.31:0/4251306381 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f395800feb0 con 0x560ce144c530 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.364+0000 7f39577fe700 1 -- 172.21.15.31:0/4251306381 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958005390 con 0x560ce144c530 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 -- 172.21.15.31:0/4251306381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 msgr2=0x560ce14501f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce14501f0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3958000c00 tx=0x7f3958009920 comp rx=0 tx=0).stop 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 -- 172.21.15.31:0/4251306381 shutdown_connections 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144d2f0 0x560ce1450730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144bbe0 0x560ce145adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/4251306381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce14501f0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 -- 172.21.15.31:0/4251306381 >> 172.21.15.31:0/4251306381 conn(0x560ce14413e0 msgr2=0x560ce14544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 -- 172.21.15.31:0/4251306381 shutdown_connections 2023-07-29T16:25:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 -- 172.21.15.31:0/4251306381 wait complete. 2023-07-29T16:25:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.365+0000 7f3986ac4b80 1 -- start start 2023-07-29T16:25:43.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.366+0000 7f3986ac4b80 1 Processor -- start 2023-07-29T16:25:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.366+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce145a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.366+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1458f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.366+0000 7f3986ac4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce1459470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce145b140 con 0x560ce144bbe0 2023-07-29T16:25:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce14599b0 con 0x560ce144c530 2023-07-29T16:25:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f3986ac4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ce1459b70 con 0x560ce144d2f0 2023-07-29T16:25:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f3986ac4b80 10 client.0 _ll_register_callbacks cb 0x560ce1342620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1458f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f3970f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce145a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1458f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46816/0 (socket says 172.21.15.31:46816) 2023-07-29T16:25:43.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 -- 172.21.15.31:0/3121683468 learned_addr learned my addr 172.21.15.31:0/3121683468 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f397179c700 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce1459470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 -- 172.21.15.31:0/3121683468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 msgr2=0x560ce1459470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce1459470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 -- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 msgr2=0x560ce145a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce145a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 -- 172.21.15.31:0/3121683468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c1a0830 con 0x560ce144c530 2023-07-29T16:25:43.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f396bfff700 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1458f30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f395800a010 tx=0x7f3958011ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f442c90bc8330e28 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.367+0000 7f394bfff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f394bfff700 1 -- 172.21.15.31:0/3121683468 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958017040 con 0x560ce144c530 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f396009d4a0 con 0x560ce144c530 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f394bfff700 1 -- 172.21.15.31:0/3121683468 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3958005110 con 0x560ce144c530 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f394bfff700 1 -- 172.21.15.31:0/3121683468 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39580052d0 con 0x560ce144c530 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 10 client.28568 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:43.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560ce11f1250 con 0x560ce144c530 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f394bfff700 1 -- 172.21.15.31:0/3121683468 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f396009d4a0 con 0x560ce144c530 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f394bfff700 1 -- 172.21.15.31:0/3121683468 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f3958026460 con 0x560ce144c530 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 10 client.28568 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 10 client.28568 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 20 client.28568 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce14b4300 0x560ce14b66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 4 client.28568 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.368+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560ce1395d40 con 0x560ce14b4300 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f3970f9b700 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce14b4300 0x560ce14b66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f394bfff700 10 client.28568 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f3970f9b700 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce14b4300 0x560ce14b66e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc2e9d7760110cf9 server_cookie=68cdf20dae669426 in_seq=0 out_seq=0 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f394bfff700 1 -- 172.21.15.31:0/3121683468 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560ce1395d40 con 0x560ce14b4300 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f394bfff700 10 client.28568 handle_command_reply: tid=0 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f3986ac4b80 1 client.28568 shutdown 2023-07-29T16:25:43.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 msgr2=0x560ce1458f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.369+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1458f30 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f395800a010 tx=0x7f3958011ee0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 shutdown_connections 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560ce14b4300 0x560ce14b66e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ce144d2f0 0x560ce1459470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ce144bbe0 0x560ce145a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 --2- 172.21.15.31:0/3121683468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ce144c530 0x560ce1458f30 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f394bfff700 0 client.28568 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:25:43.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 >> 172.21.15.31:0/3121683468 conn(0x560ce14413e0 msgr2=0x560ce14501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 shutdown_connections 2023-07-29T16:25:43.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 1 -- 172.21.15.31:0/3121683468 wait complete. 2023-07-29T16:25:43.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.370+0000 7f3986ac4b80 20 client.28568 trim_cache size 0 max 16384 2023-07-29T16:25:43.377 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:25:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f395c0a0810 msgr2=0x7f395c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f395c0a0810 0x7f395c0a2cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3964009610 tx=0x7f39640097e0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 msgr2=0x7f396c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c198380 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f396000e970 tx=0x7f396000ec80 comp rx=0 tx=0).stop 2023-07-29T16:25:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 shutdown_connections 2023-07-29T16:25:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f395c0a0810 0x7f395c0a2cc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f396c1040d0 0x7f396c19f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f396c102550 0x7f396c198380 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 --2- 172.21.15.31:0/2498917101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f396c103740 0x7f396c1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 >> 172.21.15.31:0/2498917101 conn(0x7f396c0fdb90 msgr2=0x7f396c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 shutdown_connections 2023-07-29T16:25:43.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.371+0000 7f39731ff700 1 -- 172.21.15.31:0/2498917101 wait complete. 2023-07-29T16:25:43.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:25:43.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 -- 172.21.15.31:0/3526550616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640737b0 msgr2=0x7fe464073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 --2- 172.21.15.31:0/3526550616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640737b0 0x7fe464073c20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe454009990 tx=0x7fe454009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 -- 172.21.15.31:0/3526550616 shutdown_connections 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 --2- 172.21.15.31:0/3526550616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 0x7fe46410ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 --2- 172.21.15.31:0/3526550616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe464074d90 0x7fe4640731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 --2- 172.21.15.31:0/3526550616 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4640737b0 0x7fe464073c20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 -- 172.21.15.31:0/3526550616 >> 172.21.15.31:0/3526550616 conn(0x7fe4640fb970 msgr2=0x7fe4640fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 -- 172.21.15.31:0/3526550616 shutdown_connections 2023-07-29T16:25:43.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.601+0000 7fe46bdff700 1 -- 172.21.15.31:0/3526550616 wait complete. 2023-07-29T16:25:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.602+0000 7fe46bdff700 1 Processor -- start 2023-07-29T16:25:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 -- start start 2023-07-29T16:25:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4640737b0 0x7fe46419c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe464074d90 0x7fe46419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 0x7fe4641a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641a4260 con 0x7fe4640737b0 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641a43a0 con 0x7fe46410c900 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46bdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641a4530 con 0x7fe464074d90 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46939a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe464074d90 0x7fe46419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4640737b0 0x7fe46419c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46939a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe464074d90 0x7fe46419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50842/0 (socket says 172.21.15.31:50842) 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46939a700 1 -- 172.21.15.31:0/902080578 learned_addr learned my addr 172.21.15.31:0/902080578 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.603+0000 7fe46a39c700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 0x7fe4641a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe469b9b700 1 -- 172.21.15.31:0/902080578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe464074d90 msgr2=0x7fe46419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe469b9b700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe464074d90 0x7fe46419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe469b9b700 1 -- 172.21.15.31:0/902080578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 msgr2=0x7fe4641a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe469b9b700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 0x7fe4641a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe469b9b700 1 -- 172.21.15.31:0/902080578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c009c40 con 0x7fe4640737b0 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe46a39c700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 0x7fe4641a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:43.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe469b9b700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4640737b0 0x7fe46419c6e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe46000e9c0 tx=0x7fe46000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe45affd700 1 -- 172.21.15.31:0/902080578 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46000c9c0 con 0x7fe4640737b0 2023-07-29T16:25:43.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe45affd700 1 -- 172.21.15.31:0/902080578 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe460010430 con 0x7fe4640737b0 2023-07-29T16:25:43.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4540095f0 con 0x7fe4640737b0 2023-07-29T16:25:43.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.604+0000 7fe45affd700 1 -- 172.21.15.31:0/902080578 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460018680 con 0x7fe4640737b0 2023-07-29T16:25:43.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.605+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4641a4be0 con 0x7fe4640737b0 2023-07-29T16:25:43.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.608+0000 7fe45affd700 1 -- 172.21.15.31:0/902080578 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe460010860 con 0x7fe4640737b0 2023-07-29T16:25:43.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.609+0000 7fe47f6beb80 1 Processor -- start 2023-07-29T16:25:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.609+0000 7fe47f6beb80 1 -- start start 2023-07-29T16:25:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe45affd700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4500a0810 0x7fe4500a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe45affd700 1 -- 172.21.15.31:0/902080578 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe460014070 con 0x7fe4640737b0 2023-07-29T16:25:43.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c92fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c930d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c931740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c887a10 con 0x556c0c930530 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c8853d0 con 0x556c0c9312f0 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c880cc0 con 0x556c0c92fbe0 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe46a39c700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4500a0810 0x7fe4500a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c92fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c92fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50846/0 (socket says 172.21.15.31:50846) 2023-07-29T16:25:43.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe469b9b700 1 -- 172.21.15.31:0/1291909733 learned_addr learned my addr 172.21.15.31:0/1291909733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe46939a700 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c930d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.610+0000 7fe46a39c700 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c931740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe469b9b700 1 -- 172.21.15.31:0/1291909733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 msgr2=0x556c0c931740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe469b9b700 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c931740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe469b9b700 1 -- 172.21.15.31:0/1291909733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 msgr2=0x556c0c930d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe469b9b700 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c930d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe469b9b700 1 -- 172.21.15.31:0/1291909733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c0c6aede0 con 0x556c0c92fbe0 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe469b9b700 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c92fff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe460095dd0 tx=0x7fe460099200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f49589b0b8b32d2f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe46a39c700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4500a0810 0x7fe4500a2cc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009550 tx=0x7fe45c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe44ffff700 1 -- 172.21.15.31:0/1291909733 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460099360 con 0x556c0c92fbe0 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe44ffff700 1 -- 172.21.15.31:0/1291909733 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe46009d390 con 0x556c0c92fbe0 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.611+0000 7fe44ffff700 1 -- 172.21.15.31:0/1291909733 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460098ab0 con 0x556c0c92fbe0 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1291909733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 msgr2=0x556c0c92fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c92fff0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe460095dd0 tx=0x7fe460099200 comp rx=0 tx=0).stop 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1291909733 shutdown_connections 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c931740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c92fff0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1291909733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c930d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1291909733 >> 172.21.15.31:0/1291909733 conn(0x556c0c9253e0 msgr2=0x556c0c92a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1291909733 shutdown_connections 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.612+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1291909733 wait complete. 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.613+0000 7fe47f6beb80 1 -- start start 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.613+0000 7fe47f6beb80 1 Processor -- start 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c939860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c930530 0x556c0c939da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c9312f0 0x556c0c940de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c81d510 con 0x556c0c9312f0 2023-07-29T16:25:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c81d1b0 con 0x556c0c930530 2023-07-29T16:25:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c704920 con 0x556c0c92fbe0 2023-07-29T16:25:43.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe47f6beb80 10 client.0 _ll_register_callbacks cb 0x556c0c826620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:43.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c939860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe46939a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c930530 0x556c0c939da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c939860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50852/0 (socket says 172.21.15.31:50852) 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe46939a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c930530 0x556c0c939da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46842/0 (socket says 172.21.15.31:46842) 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe46939a700 1 -- 172.21.15.31:0/1634507992 learned_addr learned my addr 172.21.15.31:0/1634507992 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe46a39c700 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c9312f0 0x556c0c940de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 -- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c930530 msgr2=0x556c0c939da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c930530 0x556c0c939da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 -- 172.21.15.31:0/1634507992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c9312f0 msgr2=0x556c0c940de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c9312f0 0x556c0c940de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.614+0000 7fe469b9b700 1 -- 172.21.15.31:0/1634507992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4641a4be0 con 0x556c0c92fbe0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe469b9b700 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c939860 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe460095860 tx=0x7fe460010bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1edcebcdf8be175 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe44d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe44d7fa700 1 -- 172.21.15.31:0/1634507992 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600a1520 con 0x556c0c92fbe0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe44d7fa700 1 -- 172.21.15.31:0/1634507992 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe4600c9bc0 con 0x556c0c92fbe0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556c0c8abd10 con 0x556c0c92fbe0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe44d7fa700 1 -- 172.21.15.31:0/1634507992 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4600993e0 con 0x556c0c92fbe0 2023-07-29T16:25:43.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe47f6beb80 10 client.28585 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:43.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.615+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556c0c697c60 con 0x556c0c92fbe0 2023-07-29T16:25:43.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe44d7fa700 1 -- 172.21.15.31:0/1634507992 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556c0c8abd10 con 0x556c0c92fbe0 2023-07-29T16:25:43.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe44d7fa700 1 -- 172.21.15.31:0/1634507992 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fe4600982c0 con 0x556c0c92fbe0 2023-07-29T16:25:43.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe47f6beb80 10 client.28585 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:43.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe47f6beb80 10 client.28585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:43.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe47f6beb80 20 client.28585 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c9451a0 0x556c0c9475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe47f6beb80 4 client.28585 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c0c8fed70 con 0x556c0c9451a0 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.616+0000 7fe46939a700 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c9451a0 0x556c0c9475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.617+0000 7fe44d7fa700 10 client.28585 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.617+0000 7fe46939a700 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c9451a0 0x556c0c9475b0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8db18b43cf986a8b server_cookie=51fcd9af967d15ed in_seq=0 out_seq=0 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.617+0000 7fe44d7fa700 1 -- 172.21.15.31:0/1634507992 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556c0c8fed70 con 0x556c0c9451a0 2023-07-29T16:25:43.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe44d7fa700 10 client.28585 handle_command_reply: tid=0 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 client.28585 shutdown 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 msgr2=0x556c0c939860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c939860 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe460095860 tx=0x7fe460010bd0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 shutdown_connections 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c9451a0 0x556c0c9475b0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c9312f0 0x556c0c940de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c92fbe0 0x556c0c939860 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe44d7fa700 0 client.28585 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:43.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/1634507992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c930530 0x556c0c939da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 >> 172.21.15.31:0/1634507992 conn(0x556c0c9253e0 msgr2=0x556c0c92a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 shutdown_connections 2023-07-29T16:25:43.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 1 -- 172.21.15.31:0/1634507992 wait complete. 2023-07-29T16:25:43.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.618+0000 7fe47f6beb80 20 client.28585 trim_cache size 0 max 16384 2023-07-29T16:25:43.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.632+0000 7fe47f6beb80 1 Processor -- start 2023-07-29T16:25:43.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.632+0000 7fe47f6beb80 1 -- start start 2023-07-29T16:25:43.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c92fbe0 0x556c0c9475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 0x556c0c940da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c938f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c927460 con 0x556c0c92fbe0 2023-07-29T16:25:43.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c92bcc0 con 0x556c0c9312f0 2023-07-29T16:25:43.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c9374d0 con 0x556c0c930530 2023-07-29T16:25:43.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c92fbe0 0x556c0c9475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 0x556c0c940da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 0x556c0c940da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50858/0 (socket says 172.21.15.31:50858) 2023-07-29T16:25:43.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 -- 172.21.15.31:0/889715688 learned_addr learned my addr 172.21.15.31:0/889715688 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46a39c700 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c938f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 -- 172.21.15.31:0/889715688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 msgr2=0x556c0c938f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c938f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 -- 172.21.15.31:0/889715688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c92fbe0 msgr2=0x556c0c9475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c92fbe0 0x556c0c9475b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.633+0000 7fe46939a700 1 -- 172.21.15.31:0/889715688 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c0c6d5250 con 0x556c0c930530 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.634+0000 7fe46939a700 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 0x556c0c940da0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x556c0c947e60 tx=0x7fe454005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58031fd38a31ce83 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.634+0000 7fe458bff700 1 -- 172.21.15.31:0/889715688 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45401d040 con 0x556c0c930530 2023-07-29T16:25:43.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.634+0000 7fe458bff700 1 -- 172.21.15.31:0/889715688 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe45401e040 con 0x556c0c930530 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.634+0000 7fe458bff700 1 -- 172.21.15.31:0/889715688 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454005050 con 0x556c0c930530 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 -- 172.21.15.31:0/889715688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 msgr2=0x556c0c940da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 0x556c0c940da0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x556c0c947e60 tx=0x7fe454005fb0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 -- 172.21.15.31:0/889715688 shutdown_connections 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c9312f0 0x556c0c938f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c92fbe0 0x556c0c9475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/889715688 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c930530 0x556c0c940da0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 -- 172.21.15.31:0/889715688 >> 172.21.15.31:0/889715688 conn(0x556c0c9253e0 msgr2=0x556c0c92a900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 -- 172.21.15.31:0/889715688 shutdown_connections 2023-07-29T16:25:43.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 -- 172.21.15.31:0/889715688 wait complete. 2023-07-29T16:25:43.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 -- start start 2023-07-29T16:25:43.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.635+0000 7fe47f6beb80 1 Processor -- start 2023-07-29T16:25:43.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 0x556c0c93c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c93cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c9312f0 0x556c0c93fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c9402e0 con 0x556c0c930530 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c940450 con 0x556c0c92fbe0 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c0c940610 con 0x556c0c9312f0 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe47f6beb80 10 client.0 _ll_register_callbacks cb 0x556c0c826620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:43.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 0x556c0c93c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe469b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 0x556c0c93c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46864/0 (socket says 172.21.15.31:46864) 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe469b9b700 1 -- 172.21.15.31:0/3301031202 learned_addr learned my addr 172.21.15.31:0/3301031202 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe46a39c700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c9312f0 0x556c0c93fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.636+0000 7fe46939a700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c93cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe469b9b700 1 -- 172.21.15.31:0/3301031202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c9312f0 msgr2=0x556c0c93fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe469b9b700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c9312f0 0x556c0c93fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe469b9b700 1 -- 172.21.15.31:0/3301031202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 msgr2=0x556c0c93cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe469b9b700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c93cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe469b9b700 1 -- 172.21.15.31:0/3301031202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe46410f690 con 0x556c0c92fbe0 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe469b9b700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 0x556c0c93c840 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe4600976b0 tx=0x7fe4600107d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56f358a828b739ff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe44d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe44d7fa700 1 -- 172.21.15.31:0/3301031202 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460099360 con 0x556c0c92fbe0 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe44d7fa700 1 -- 172.21.15.31:0/3301031202 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe460095150 con 0x556c0c92fbe0 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe46009d390 con 0x556c0c92fbe0 2023-07-29T16:25:43.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe44d7fa700 1 -- 172.21.15.31:0/3301031202 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe460095310 con 0x556c0c92fbe0 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe47f6beb80 10 client.28586 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.637+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556c0c6ad2b0 con 0x556c0c92fbe0 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe44d7fa700 1 -- 172.21.15.31:0/3301031202 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe46009d390 con 0x556c0c92fbe0 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe44d7fa700 1 -- 172.21.15.31:0/3301031202 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fe4600c9280 con 0x556c0c92fbe0 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe47f6beb80 10 client.28586 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe47f6beb80 10 client.28586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe47f6beb80 20 client.28586 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:43.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c997390 0x556c0c9997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe47f6beb80 4 client.28586 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556c0c879d40 con 0x556c0c997390 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe46939a700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c997390 0x556c0c9997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe46939a700 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c997390 0x556c0c9997a0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7d3135065e934a4 server_cookie=4d6d92ee73ddda76 in_seq=0 out_seq=0 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.638+0000 7fe44d7fa700 10 client.28586 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe44d7fa700 1 -- 172.21.15.31:0/3301031202 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556c0c879d40 con 0x556c0c997390 2023-07-29T16:25:43.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe44d7fa700 10 client.28586 handle_command_reply: tid=0 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 client.28586 shutdown 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 msgr2=0x556c0c93c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 0x556c0c93c840 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe4600976b0 tx=0x7fe4600107d0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 shutdown_connections 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x556c0c997390 0x556c0c9997a0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556c0c9312f0 0x556c0c93fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c0c92fbe0 0x556c0c93c840 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 --2- 172.21.15.31:0/3301031202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0c930530 0x556c0c93cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe44d7fa700 0 client.28586 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:43.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 >> 172.21.15.31:0/3301031202 conn(0x556c0c9253e0 msgr2=0x556c0c9384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 shutdown_connections 2023-07-29T16:25:43.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 1 -- 172.21.15.31:0/3301031202 wait complete. 2023-07-29T16:25:43.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.639+0000 7fe47f6beb80 20 client.28586 trim_cache size 0 max 16384 2023-07-29T16:25:43.646 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:25:43.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4500a0810 msgr2=0x7fe4500a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4500a0810 0x7fe4500a2cc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe45c009550 tx=0x7fe45c009920 comp rx=0 tx=0).stop 2023-07-29T16:25:43.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4640737b0 msgr2=0x7fe46419c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4640737b0 0x7fe46419c6e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe46000e9c0 tx=0x7fe46000ecd0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 shutdown_connections 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe4500a0810 0x7fe4500a2cc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe46410c900 0x7fe4641a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4640737b0 0x7fe46419c6e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 --2- 172.21.15.31:0/902080578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe464074d90 0x7fe46419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:43.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 >> 172.21.15.31:0/902080578 conn(0x7fe4640fb970 msgr2=0x7fe4640fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 shutdown_connections 2023-07-29T16:25:43.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:43.640+0000 7fe46bdff700 1 -- 172.21.15.31:0/902080578 wait complete. 2023-07-29T16:25:43.655 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:43 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3846945768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:25:43.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:43 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3846945768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:25:43.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 741923 8.90 MB/sec execute 328 sec latency 20.233 ms 2023-07-29T16:25:44.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:43 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3846945768' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:25:44.658 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:25:44.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:25:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:44 smithi156 ceph-mon[129245]: pgmap v679: 97 pgs: 97 active+clean; 141 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:25:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 -- 172.21.15.31:0/2389106485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934106040 msgr2=0x7f5934108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2389106485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934106040 0x7f5934108440 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f592c00b0a0 tx=0x7f592c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 -- 172.21.15.31:0/2389106485 shutdown_connections 2023-07-29T16:25:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2389106485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934106040 0x7f5934108440 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2389106485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934100e00 0x7f59341031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2389106485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5934103720 0x7f5934105b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 -- 172.21.15.31:0/2389106485 >> 172.21.15.31:0/2389106485 conn(0x7f59340fa840 msgr2=0x7f59340fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:44.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 -- 172.21.15.31:0/2389106485 shutdown_connections 2023-07-29T16:25:44.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.878+0000 7f593c3ff700 1 -- 172.21.15.31:0/2389106485 wait complete. 2023-07-29T16:25:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 Processor -- start 2023-07-29T16:25:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 -- start start 2023-07-29T16:25:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 0x7f5934196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5934103720 0x7f59341966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934106040 0x7f593419d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593419dd00 con 0x7f5934103720 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593419de40 con 0x7f5934106040 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.879+0000 7f593c3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593419dfd0 con 0x7f5934100e00 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 0x7f5934196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 0x7f5934196180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50898/0 (socket says 172.21.15.31:50898) 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 -- 172.21.15.31:0/2292290753 learned_addr learned my addr 172.21.15.31:0/2292290753 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a99c700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934106040 0x7f593419d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593999a700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5934103720 0x7f59341966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 -- 172.21.15.31:0/2292290753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934106040 msgr2=0x7f593419d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934106040 0x7f593419d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 -- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5934103720 msgr2=0x7f59341966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5934103720 0x7f59341966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 -- 172.21.15.31:0/2292290753 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592c009d00 con 0x7f5934100e00 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a99c700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934106040 0x7f593419d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:44.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.880+0000 7f593a19b700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 0x7f5934196180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f593000e9a0 tx=0x7f593000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:44.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.881+0000 7f592b7fe700 1 -- 172.21.15.31:0/2292290753 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930009790 con 0x7f5934100e00 2023-07-29T16:25:44.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.881+0000 7f592b7fe700 1 -- 172.21.15.31:0/2292290753 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5930010430 con 0x7f5934100e00 2023-07-29T16:25:44.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.881+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593419e2b0 con 0x7f5934100e00 2023-07-29T16:25:44.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.881+0000 7f592b7fe700 1 -- 172.21.15.31:0/2292290753 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5930018680 con 0x7f5934100e00 2023-07-29T16:25:44.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.881+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593419e810 con 0x7f5934100e00 2023-07-29T16:25:44.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.884+0000 7f592b7fe700 1 -- 172.21.15.31:0/2292290753 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f59300105a0 con 0x7f5934100e00 2023-07-29T16:25:44.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.886+0000 7f594fd0cb80 1 Processor -- start 2023-07-29T16:25:44.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.886+0000 7f594fd0cb80 1 -- start start 2023-07-29T16:25:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.886+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd8491b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd8491c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 0x55bd8491cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd8486ea90 con 0x55bd8491b070 2023-07-29T16:25:44.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd8486c450 con 0x55bd8491c780 2023-07-29T16:25:44.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84867d40 con 0x55bd8491b9c0 2023-07-29T16:25:44.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f592b7fe700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59200a0810 0x7f59200a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 0x55bd8491cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 0x55bd8491cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46886/0 (socket says 172.21.15.31:46886) 2023-07-29T16:25:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 -- 172.21.15.31:0/299980266 learned_addr learned my addr 172.21.15.31:0/299980266 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593a19b700 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd8491c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593a99c700 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd8491b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f592b7fe700 1 -- 172.21.15.31:0/2292290753 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5930014070 con 0x7f5934100e00 2023-07-29T16:25:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 -- 172.21.15.31:0/299980266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 msgr2=0x55bd8491c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd8491c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 -- 172.21.15.31:0/299980266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 msgr2=0x55bd8491b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd8491b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593999a700 1 -- 172.21.15.31:0/299980266 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd84695e60 con 0x55bd8491c780 2023-07-29T16:25:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593a99c700 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd8491b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:25:44.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.887+0000 7f593a99c700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59200a0810 0x7f59200a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 742955 8.88 MB/sec execute 329 sec latency 23.503 ms 2023-07-29T16:25:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f593999a700 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 0x55bd8491cbd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5924009d80 tx=0x7f5924011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4091b7733ff49 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f5928bfe700 1 -- 172.21.15.31:0/299980266 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924004230 con 0x55bd8491c780 2023-07-29T16:25:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f5928bfe700 1 -- 172.21.15.31:0/299980266 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5924004390 con 0x55bd8491c780 2023-07-29T16:25:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f5928bfe700 1 -- 172.21.15.31:0/299980266 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924019860 con 0x55bd8491c780 2023-07-29T16:25:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f594fd0cb80 1 -- 172.21.15.31:0/299980266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 msgr2=0x55bd8491cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f593a99c700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59200a0810 0x7f59200a2cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f592c00b780 tx=0x7f592c0099e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:44.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.888+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 0x55bd8491cbd0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f5924009d80 tx=0x7f5924011720 comp rx=0 tx=0).stop 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 -- 172.21.15.31:0/299980266 shutdown_connections 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491c780 0x55bd8491cbd0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd8491b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/299980266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd8491c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 -- 172.21.15.31:0/299980266 >> 172.21.15.31:0/299980266 conn(0x55bd8490c460 msgr2=0x55bd84911a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 -- 172.21.15.31:0/299980266 shutdown_connections 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 -- 172.21.15.31:0/299980266 wait complete. 2023-07-29T16:25:44.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 -- start start 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.889+0000 7f594fd0cb80 1 Processor -- start 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd84924c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84925190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491c780 0x55bd8492c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84804590 con 0x55bd8491b070 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84804230 con 0x55bd8491b9c0 2023-07-29T16:25:44.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd846eb9a0 con 0x55bd8491c780 2023-07-29T16:25:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f594fd0cb80 10 client.0 _ll_register_callbacks cb 0x55bd8480d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f593a99c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491c780 0x55bd8492c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f593999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84925190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f593a19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd84924c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f593a99c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491c780 0x55bd8492c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50916/0 (socket says 172.21.15.31:50916) 2023-07-29T16:25:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.890+0000 7f593a99c700 1 -- 172.21.15.31:0/1691092800 learned_addr learned my addr 172.21.15.31:0/1691092800 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f593999a700 1 -- 172.21.15.31:0/1691092800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491c780 msgr2=0x55bd8492c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f593999a700 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491c780 0x55bd8492c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f593999a700 1 -- 172.21.15.31:0/1691092800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 msgr2=0x55bd84924c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f593999a700 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd84924c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f593999a700 1 -- 172.21.15.31:0/1691092800 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59240099e0 con 0x55bd8491b9c0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f593999a700 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84925190 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f592401b040 tx=0x7f59240038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d706d29525beccc5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f591dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f591dffb700 1 -- 172.21.15.31:0/1691092800 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924005ce0 con 0x55bd8491b9c0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd84892d90 con 0x55bd8491b9c0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.891+0000 7f591dffb700 1 -- 172.21.15.31:0/1691092800 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5924005040 con 0x55bd8491b9c0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f591dffb700 1 -- 172.21.15.31:0/1691092800 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5924005200 con 0x55bd8491b9c0 2023-07-29T16:25:44.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 10 client.28598 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bd8467ece0 con 0x55bd8491b9c0 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f591dffb700 1 -- 172.21.15.31:0/1691092800 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bd84892d90 con 0x55bd8491b9c0 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f591dffb700 1 -- 172.21.15.31:0/1691092800 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f5924021040 con 0x55bd8491b9c0 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 10 client.28598 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 10 client.28598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 20 client.28598 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849308f0 0x55bd84932d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 4 client.28598 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd848e5df0 con 0x55bd849308f0 2023-07-29T16:25:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.892+0000 7f593a19b700 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849308f0 0x55bd84932d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.893+0000 7f593a19b700 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849308f0 0x55bd84932d00 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b186f25e5f0b2e1b server_cookie=b507ed4262b6639b in_seq=0 out_seq=0 2023-07-29T16:25:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.893+0000 7f591dffb700 10 client.28598 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f591dffb700 1 -- 172.21.15.31:0/1691092800 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bd848e5df0 con 0x55bd849308f0 2023-07-29T16:25:44.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f591dffb700 10 client.28598 handle_command_reply: tid=0 2023-07-29T16:25:44.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 client.28598 shutdown 2023-07-29T16:25:44.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 msgr2=0x55bd84925190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84925190 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f592401b040 tx=0x7f59240038b0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 shutdown_connections 2023-07-29T16:25:44.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849308f0 0x55bd84932d00 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491c780 0x55bd8492c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f591dffb700 0 client.28598 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491b070 0x55bd84924c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1691092800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84925190 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 >> 172.21.15.31:0/1691092800 conn(0x55bd8490c460 msgr2=0x55bd849110e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:44.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 shutdown_connections 2023-07-29T16:25:44.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1691092800 wait complete. 2023-07-29T16:25:44.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.894+0000 7f594fd0cb80 20 client.28598 trim_cache size 0 max 16384 2023-07-29T16:25:44.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 Processor -- start 2023-07-29T16:25:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 -- start start 2023-07-29T16:25:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 0x55bd8492a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84924c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84925150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd849097a0 con 0x55bd8491c780 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84924130 con 0x55bd8491b9c0 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd8490e4e0 con 0x55bd8491b070 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f593999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84924c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.912+0000 7f593a19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 0x55bd8492a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84924c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46902/0 (socket says 172.21.15.31:46902) 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593a19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 0x55bd8492a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50928/0 (socket says 172.21.15.31:50928) 2023-07-29T16:25:44.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 -- 172.21.15.31:0/1586743296 learned_addr learned my addr 172.21.15.31:0/1586743296 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593a99c700 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84925150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 -- 172.21.15.31:0/1586743296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 msgr2=0x55bd8492a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 0x55bd8492a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 -- 172.21.15.31:0/1586743296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 msgr2=0x55bd84925150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84925150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 -- 172.21.15.31:0/1586743296 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd846bc2d0 con 0x55bd8491b9c0 2023-07-29T16:25:44.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593a19b700 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 0x55bd8492a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f593999a700 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84924c10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5924000c00 tx=0x7f59240195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91601b9b33eaac77 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f59293ff700 1 -- 172.21.15.31:0/1586743296 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924019420 con 0x55bd8491b9c0 2023-07-29T16:25:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f59293ff700 1 -- 172.21.15.31:0/1586743296 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5924019e40 con 0x55bd8491b9c0 2023-07-29T16:25:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.913+0000 7f59293ff700 1 -- 172.21.15.31:0/1586743296 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924005530 con 0x55bd8491b9c0 2023-07-29T16:25:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1586743296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 msgr2=0x55bd84924c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84924c10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f5924000c00 tx=0x7f59240195f0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1586743296 shutdown_connections 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84925150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b070 0x55bd8492a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1586743296 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b9c0 0x55bd84924c10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1586743296 >> 172.21.15.31:0/1586743296 conn(0x55bd8490c460 msgr2=0x55bd8491f640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1586743296 shutdown_connections 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.914+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1586743296 wait complete. 2023-07-29T16:25:44.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 -- start start 2023-07-29T16:25:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 Processor -- start 2023-07-29T16:25:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b070 0x55bd84929290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd849297d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84982580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84982ac0 con 0x55bd8491c780 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84982c00 con 0x55bd8491b070 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd84982dc0 con 0x55bd8491b9c0 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.915+0000 7f594fd0cb80 10 client.0 _ll_register_callbacks cb 0x55bd8480d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd849297d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b070 0x55bd84929290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593999a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd849297d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50930/0 (socket says 172.21.15.31:50930) 2023-07-29T16:25:44.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593999a700 1 -- 172.21.15.31:0/1796186393 learned_addr learned my addr 172.21.15.31:0/1796186393 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a99c700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84982580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 -- 172.21.15.31:0/1796186393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 msgr2=0x55bd849297d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd849297d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 -- 172.21.15.31:0/1796186393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 msgr2=0x55bd84982580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84982580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 -- 172.21.15.31:0/1796186393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593419e810 con 0x55bd8491b070 2023-07-29T16:25:44.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a99c700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84982580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593999a700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd849297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f593a19b700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b070 0x55bd84929290 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5930089fd0 tx=0x7f5930095040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fe8f7de702ee433 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f591dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f591dffb700 1 -- 172.21.15.31:0/1796186393 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300ae070 con 0x55bd8491b070 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5924004390 con 0x55bd8491b070 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.916+0000 7f591dffb700 1 -- 172.21.15.31:0/1796186393 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f593009d460 con 0x55bd8491b070 2023-07-29T16:25:44.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f591dffb700 1 -- 172.21.15.31:0/1796186393 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59300a1880 con 0x55bd8491b070 2023-07-29T16:25:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 10 client.28604 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bd8467ece0 con 0x55bd8491b070 2023-07-29T16:25:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f591dffb700 1 -- 172.21.15.31:0/1796186393 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5924004390 con 0x55bd8491b070 2023-07-29T16:25:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f591dffb700 1 -- 172.21.15.31:0/1796186393 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f59300c9340 con 0x55bd8491b070 2023-07-29T16:25:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 10 client.28604 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:44.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 10 client.28604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:44.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 20 client.28604 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:44.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849847a0 0x55bd84986bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:44.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 4 client.28604 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:25:44.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55bd84860dc0 con 0x55bd849847a0 2023-07-29T16:25:44.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.917+0000 7f593999a700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849847a0 0x55bd84986bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.918+0000 7f593999a700 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849847a0 0x55bd84986bb0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34cd2d9b41a67ae9 server_cookie=5d52b6804d4de8f8 in_seq=0 out_seq=0 2023-07-29T16:25:44.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.918+0000 7f591dffb700 10 client.28604 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.919+0000 7f591dffb700 1 -- 172.21.15.31:0/1796186393 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bd84860dc0 con 0x55bd849847a0 2023-07-29T16:25:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.919+0000 7f591dffb700 10 client.28604 handle_command_reply: tid=0 2023-07-29T16:25:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.919+0000 7f594fd0cb80 1 client.28604 shutdown 2023-07-29T16:25:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b070 msgr2=0x55bd84929290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b070 0x55bd84929290 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f5930089fd0 tx=0x7f5930095040 comp rx=0 tx=0).stop 2023-07-29T16:25:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 shutdown_connections 2023-07-29T16:25:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bd849847a0 0x55bd84986bb0 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd8491c780 0x55bd84982580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f591dffb700 0 client.28604 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bd8491b070 0x55bd84929290 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 --2- 172.21.15.31:0/1796186393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bd8491b9c0 0x55bd849297d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 >> 172.21.15.31:0/1796186393 conn(0x55bd8490c460 msgr2=0x55bd84924bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 shutdown_connections 2023-07-29T16:25:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 1 -- 172.21.15.31:0/1796186393 wait complete. 2023-07-29T16:25:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.920+0000 7f594fd0cb80 20 client.28604 trim_cache size 0 max 16384 2023-07-29T16:25:44.929 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:25:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:25:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "3321c5c5-84a9-42f7-8ab5-e8160a960608", 2023-07-29T16:25:44.929 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:25:44.930 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:25:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59200a0810 msgr2=0x7f59200a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59200a0810 0x7f59200a2cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f592c00b780 tx=0x7f592c0099e0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 msgr2=0x7f5934196180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 0x7f5934196180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f593000e9a0 tx=0x7f593000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 shutdown_connections 2023-07-29T16:25:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59200a0810 0x7f59200a2cc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5934106040 0x7f593419d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5934100e00 0x7f5934196180 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 --2- 172.21.15.31:0/2292290753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5934103720 0x7f59341966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 >> 172.21.15.31:0/2292290753 conn(0x7f59340fa840 msgr2=0x7f5934101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 shutdown_connections 2023-07-29T16:25:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:44.921+0000 7f593c3ff700 1 -- 172.21.15.31:0/2292290753 wait complete. 2023-07-29T16:25:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:44 smithi031 ceph-mon[95504]: pgmap v679: 97 pgs: 97 active+clean; 141 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:25:44.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:25:45.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:44 smithi039 ceph-mon[128858]: pgmap v679: 97 pgs: 97 active+clean; 141 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:25:45.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 -- 172.21.15.31:0/1659422447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800100210 msgr2=0x7fe800100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 --2- 172.21.15.31:0/1659422447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800100210 0x7fe800100620 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009990 tx=0x7fe7f0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 -- 172.21.15.31:0/1659422447 shutdown_connections 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 --2- 172.21.15.31:0/1659422447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe800101d90 0x7fe800108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 --2- 172.21.15.31:0/1659422447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800100210 0x7fe800100620 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 --2- 172.21.15.31:0/1659422447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 -- 172.21.15.31:0/1659422447 >> 172.21.15.31:0/1659422447 conn(0x7fe8000fb850 msgr2=0x7fe8000fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 -- 172.21.15.31:0/1659422447 shutdown_connections 2023-07-29T16:25:45.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.161+0000 7fe80579d700 1 -- 172.21.15.31:0/1659422447 wait complete. 2023-07-29T16:25:45.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.162+0000 7fe80579d700 1 Processor -- start 2023-07-29T16:25:45.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.162+0000 7fe80579d700 1 -- start start 2023-07-29T16:25:45.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe80579d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe800100210 0x7fe800198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe80579d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe80579d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800101d90 0x7fe80019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe80579d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80019fb90 con 0x7fe800100210 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe80579d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80019fcd0 con 0x7fe800101400 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe80579d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80019fe60 con 0x7fe800101d90 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7fffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe800100210 0x7fe800198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46920/0 (socket says 172.21.15.31:46920) 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/235737502 learned_addr learned my addr 172.21.15.31:0/235737502 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe804f9c700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800101d90 0x7fe80019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/235737502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800101d90 msgr2=0x7fe80019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800101d90 0x7fe80019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe800100210 msgr2=0x7fe800198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe800100210 0x7fe800198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.163+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/235737502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f00095f0 con 0x7fe800101400 2023-07-29T16:25:45.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.164+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800198550 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400e9a0 tx=0x7fe7f400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:45.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.164+0000 7fe7fd7fa700 1 -- 172.21.15.31:0/235737502 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4009790 con 0x7fe800101400 2023-07-29T16:25:45.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.164+0000 7fe7fd7fa700 1 -- 172.21.15.31:0/235737502 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f400cbd0 con 0x7fe800101400 2023-07-29T16:25:45.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.164+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8001a0140 con 0x7fe800101400 2023-07-29T16:25:45.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.164+0000 7fe7fd7fa700 1 -- 172.21.15.31:0/235737502 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4018680 con 0x7fe800101400 2023-07-29T16:25:45.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.164+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8001a0650 con 0x7fe800101400 2023-07-29T16:25:45.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.168+0000 7fe7fd7fa700 1 -- 172.21.15.31:0/235737502 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe7f400cd40 con 0x7fe800101400 2023-07-29T16:25:45.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 Processor -- start 2023-07-29T16:25:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe7fd7fa700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe7e80a0830 0x7fe7e80a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 -- start start 2023-07-29T16:25:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe7fd7fa700 1 -- 172.21.15.31:0/235737502 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe7f4014070 con 0x7fe800101400 2023-07-29T16:25:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe804f9c700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe7e80a0830 0x7fe7e80a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac40150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 0x56420ac40f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac41920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ab97a10 con 0x56420ac3fd40 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ab953d0 con 0x56420ac40690 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.170+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ab90cc0 con 0x56420ac414d0 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 0x56420ac40f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7fffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac40150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 0x56420ac40f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46932/0 (socket says 172.21.15.31:46932) 2023-07-29T16:25:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/1080545835 learned_addr learned my addr 172.21.15.31:0/1080545835 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe804f9c700 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac41920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/1080545835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 msgr2=0x56420ac41920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac41920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/1080545835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 msgr2=0x56420ac40150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac40150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/1080545835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56420a9bede0 con 0x56420ac40690 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe804f9c700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe7e80a0830 0x7fe7e80a2ce0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80076f0 tx=0x7fe7f8007ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.171+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 0x56420ac40f40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40994a0 tx=0x7fe7f40990d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30faa4ab6ab8119d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe7e67fc700 1 -- 172.21.15.31:0/1080545835 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4099510 con 0x56420ac40690 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe7e67fc700 1 -- 172.21.15.31:0/1080545835 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f409d540 con 0x56420ac40690 2023-07-29T16:25:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe7e67fc700 1 -- 172.21.15.31:0/1080545835 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4098c60 con 0x56420ac40690 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 -- 172.21.15.31:0/1080545835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 msgr2=0x56420ac40f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 0x56420ac40f40 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40994a0 tx=0x7fe7f40990d0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 -- 172.21.15.31:0/1080545835 shutdown_connections 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac41920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac40150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/1080545835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac40690 0x56420ac40f40 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.172+0000 7fe81a41ab80 1 -- 172.21.15.31:0/1080545835 >> 172.21.15.31:0/1080545835 conn(0x56420ac35540 msgr2=0x56420ac3aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.173+0000 7fe81a41ab80 1 -- 172.21.15.31:0/1080545835 shutdown_connections 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.173+0000 7fe81a41ab80 1 -- 172.21.15.31:0/1080545835 wait complete. 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.173+0000 7fe81a41ab80 1 -- start start 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.173+0000 7fe81a41ab80 1 Processor -- start 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac49a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac49f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac414d0 0x56420ac50fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ab2d510 con 0x56420ac414d0 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ab2d1b0 con 0x56420ac3fd40 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420aa14920 con 0x56420ac40690 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac49a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac49a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46936/0 (socket says 172.21.15.31:46936) 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 -- 172.21.15.31:0/4249455734 learned_addr learned my addr 172.21.15.31:0/4249455734 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe81a41ab80 10 client.0 _ll_register_callbacks cb 0x56420ab36620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:45.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac49f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe804f9c700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac414d0 0x56420ac50fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 -- 172.21.15.31:0/4249455734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 msgr2=0x56420ac49f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac49f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 -- 172.21.15.31:0/4249455734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac414d0 msgr2=0x56420ac50fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac414d0 0x56420ac50fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7fffff700 1 -- 172.21.15.31:0/4249455734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8001a0650 con 0x56420ac3fd40 2023-07-29T16:25:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac49f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.174+0000 7fe804f9c700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac414d0 0x56420ac50fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe7fffff700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac49a40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe800101260 tx=0x7fe7f0009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4924600b41aab5cd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe7df7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe7df7fe700 1 -- 172.21.15.31:0/4249455734 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f001d030 con 0x56420ac3fd40 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe7df7fe700 1 -- 172.21.15.31:0/4249455734 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe7f0004700 con 0x56420ac3fd40 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56420abbbd10 con 0x56420ac3fd40 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe7df7fe700 1 -- 172.21.15.31:0/4249455734 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7f00048c0 con 0x56420ac3fd40 2023-07-29T16:25:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe81a41ab80 10 client.28616 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.175+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56420a9a7c60 con 0x56420ac3fd40 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe7df7fe700 1 -- 172.21.15.31:0/4249455734 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56420abbbd10 con 0x56420ac3fd40 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe7df7fe700 1 -- 172.21.15.31:0/4249455734 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fe7f000fb50 con 0x56420ac3fd40 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe81a41ab80 10 client.28616 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe81a41ab80 10 client.28616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe81a41ab80 20 client.28616 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:45.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420ac557e0 0x56420ac57bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe81a41ab80 4 client.28616 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56420ac0ed70 con 0x56420ac557e0 2023-07-29T16:25:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.176+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420ac557e0 0x56420ac57bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.177+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420ac557e0 0x56420ac57bf0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b76b402fe6dbe735 server_cookie=9b89d5c27e3cbccf in_seq=0 out_seq=0 2023-07-29T16:25:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.177+0000 7fe7df7fe700 10 client.28616 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:45.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.177+0000 7fe7df7fe700 1 -- 172.21.15.31:0/4249455734 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56420ac0ed70 con 0x56420ac557e0 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.177+0000 7fe7df7fe700 10 client.28616 handle_command_reply: tid=0 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 client.28616 shutdown 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 msgr2=0x56420ac49a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac49a40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe800101260 tx=0x7fe7f0009570 comp rx=0 tx=0).stop 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 shutdown_connections 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420ac557e0 0x56420ac57bf0 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac414d0 0x56420ac50fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac49a40 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe7df7fe700 0 client.28616 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4249455734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac49f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 >> 172.21.15.31:0/4249455734 conn(0x56420ac35540 msgr2=0x56420ac3a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 shutdown_connections 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4249455734 wait complete. 2023-07-29T16:25:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.178+0000 7fe81a41ab80 20 client.28616 trim_cache size 0 max 16384 2023-07-29T16:25:45.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 Processor -- start 2023-07-29T16:25:45.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 -- start start 2023-07-29T16:25:45.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac4eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac40690 0x56420ac478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac44410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ac47690 con 0x56420ac40690 2023-07-29T16:25:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ac40d20 con 0x56420ac3fd40 2023-07-29T16:25:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ac483f0 con 0x56420ac414d0 2023-07-29T16:25:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe7fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac4eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe7fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac4eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46944/0 (socket says 172.21.15.31:46944) 2023-07-29T16:25:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe7fffff700 1 -- 172.21.15.31:0/4242005084 learned_addr learned my addr 172.21.15.31:0/4242005084 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.198+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac40690 0x56420ac478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe804f9c700 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac44410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7fffff700 1 -- 172.21.15.31:0/4242005084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 msgr2=0x56420ac44410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7fffff700 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac44410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7fffff700 1 -- 172.21.15.31:0/4242005084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac40690 msgr2=0x56420ac478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7fffff700 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac40690 0x56420ac478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7fffff700 1 -- 172.21.15.31:0/4242005084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56420a9e5250 con 0x56420ac3fd40 2023-07-29T16:25:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac40690 0x56420ac478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:25:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7fffff700 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac4eed0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00098e0 tx=0x7fe7f0009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54c01335c95b8d24 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7e6ffd700 1 -- 172.21.15.31:0/4242005084 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f001d030 con 0x56420ac3fd40 2023-07-29T16:25:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7e6ffd700 1 -- 172.21.15.31:0/4242005084 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f001e040 con 0x56420ac3fd40 2023-07-29T16:25:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.199+0000 7fe7e6ffd700 1 -- 172.21.15.31:0/4242005084 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f000f960 con 0x56420ac3fd40 2023-07-29T16:25:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4242005084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 msgr2=0x56420ac4eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac4eed0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00098e0 tx=0x7fe7f0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4242005084 shutdown_connections 2023-07-29T16:25:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac414d0 0x56420ac44410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac3fd40 0x56420ac4eed0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/4242005084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac40690 0x56420ac478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4242005084 >> 172.21.15.31:0/4242005084 conn(0x56420ac35540 msgr2=0x56420ac486c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4242005084 shutdown_connections 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.200+0000 7fe81a41ab80 1 -- 172.21.15.31:0/4242005084 wait complete. 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.201+0000 7fe81a41ab80 1 -- start start 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.201+0000 7fe81a41ab80 1 Processor -- start 2023-07-29T16:25:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac4e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac4ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac414d0 0x56420ac4cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ac4d270 con 0x56420ac3fd40 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ac4d3e0 con 0x56420ac414d0 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56420ac4d550 con 0x56420ac40690 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe81a41ab80 10 client.0 _ll_register_callbacks cb 0x56420ab36620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac4ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac4ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50974/0 (socket says 172.21.15.31:50974) 2023-07-29T16:25:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/2633604585 learned_addr learned my addr 172.21.15.31:0/2633604585 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe804f9c700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac414d0 0x56420ac4cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7fffff700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac4e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac414d0 msgr2=0x56420ac4cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac414d0 0x56420ac4cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/2633604585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 msgr2=0x56420ac4e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac4e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.202+0000 7fe7ff7fe700 1 -- 172.21.15.31:0/2633604585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56420a9a7c60 con 0x56420ac40690 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7ff7fe700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac4ec60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4099210 tx=0x7fe7f401a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b3caed1cd09874 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7df7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7df7fe700 1 -- 172.21.15.31:0/2633604585 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40b7040 con 0x56420ac40690 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7df7fe700 1 -- 172.21.15.31:0/2633604585 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe7f409d170 con 0x56420ac40690 2023-07-29T16:25:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7f409d540 con 0x56420ac40690 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7df7fe700 1 -- 172.21.15.31:0/2633604585 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7f4099590 con 0x56420ac40690 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 10 client.28627 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56420a9bd2b0 con 0x56420ac40690 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7df7fe700 1 -- 172.21.15.31:0/2633604585 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7f409d540 con 0x56420ac40690 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe7df7fe700 1 -- 172.21.15.31:0/2633604585 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fe7f4098a90 con 0x56420ac40690 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 10 client.28627 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 10 client.28627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 20 client.28627 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420aca81c0 0x56420acaa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 4 client.28627 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.203+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56420ab89d40 con 0x56420aca81c0 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.204+0000 7fe7fffff700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420aca81c0 0x56420acaa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.204+0000 7fe7df7fe700 10 client.28627 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.204+0000 7fe7fffff700 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420aca81c0 0x56420acaa5d0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c4b41cd506ee53b server_cookie=7987ce401fce650d in_seq=0 out_seq=0 2023-07-29T16:25:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.204+0000 7fe7df7fe700 1 -- 172.21.15.31:0/2633604585 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56420ab89d40 con 0x56420aca81c0 2023-07-29T16:25:45.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.204+0000 7fe7df7fe700 10 client.28627 handle_command_reply: tid=0 2023-07-29T16:25:45.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.204+0000 7fe81a41ab80 1 client.28627 shutdown 2023-07-29T16:25:45.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 msgr2=0x56420ac4ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac4ec60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4099210 tx=0x7fe7f401a040 comp rx=0 tx=0).stop 2023-07-29T16:25:45.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 shutdown_connections 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56420aca81c0 0x56420acaa5d0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56420ac414d0 0x56420ac4cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56420ac3fd40 0x56420ac4e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 --2- 172.21.15.31:0/2633604585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56420ac40690 0x56420ac4ec60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe7df7fe700 0 client.28627 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 >> 172.21.15.31:0/2633604585 conn(0x56420ac35540 msgr2=0x56420ac478e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 shutdown_connections 2023-07-29T16:25:45.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 1 -- 172.21.15.31:0/2633604585 wait complete. 2023-07-29T16:25:45.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.205+0000 7fe81a41ab80 20 client.28627 trim_cache size 0 max 16384 2023-07-29T16:25:45.214 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:25:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:25:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:25:45.214 INFO:teuthology.orchestra.run.smithi031.stdout: "3321c5c5-84a9-42f7-8ab5-e8160a960608": { 2023-07-29T16:25:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:25:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "3321c5c5-84a9-42f7-8ab5-e8160a960608", 2023-07-29T16:25:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:25:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:25:45.215 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:25:45.216 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:25:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe7e80a0830 msgr2=0x7fe7e80a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe7e80a0830 0x7fe7e80a2ce0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80076f0 tx=0x7fe7f8007ab0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 msgr2=0x7fe800198550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800198550 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400e9a0 tx=0x7fe7f400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 shutdown_connections 2023-07-29T16:25:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe7e80a0830 0x7fe7e80a2ce0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe800101d90 0x7fe80019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe800100210 0x7fe800198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 --2- 172.21.15.31:0/235737502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe800101400 0x7fe800198550 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 >> 172.21.15.31:0/235737502 conn(0x7fe8000fb850 msgr2=0x7fe8000fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 shutdown_connections 2023-07-29T16:25:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:25:45.206+0000 7fe80579d700 1 -- 172.21.15.31:0/235737502 wait complete. 2023-07-29T16:25:45.224 INFO:tasks.cephfs.filesystem:scrub status for tag:3321c5c5-84a9-42f7-8ab5-e8160a960608 - {'path': '/', 'tag': '3321c5c5-84a9-42f7-8ab5-e8160a960608', 'options': 'recursive,force'} 2023-07-29T16:25:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:45 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:45 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:45 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:45 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:45 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:45.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:45 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:45.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 745297 8.90 MB/sec execute 330 sec latency 16.412 ms 2023-07-29T16:25:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:45 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:45 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:45.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:45 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:45.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:45 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:45.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:45 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:45.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:45 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:45 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:45 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:45 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:45 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:45 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:45 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:46 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:25:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:46 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:25:46.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:46 smithi156 ceph-mon[129245]: pgmap v680: 97 pgs: 97 active+clean; 142 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.5 MiB/s wr, 268 op/s 2023-07-29T16:25:46.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 746466 8.89 MB/sec execute 331 sec latency 12.163 ms 2023-07-29T16:25:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:46 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:25:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:46 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:25:46.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:46.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:46 smithi031 ceph-mon[95504]: pgmap v680: 97 pgs: 97 active+clean; 142 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.5 MiB/s wr, 268 op/s 2023-07-29T16:25:47.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:46 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:25:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:46 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:25:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:25:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:46 smithi039 ceph-mon[128858]: pgmap v680: 97 pgs: 97 active+clean; 142 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 2.5 MiB/s wr, 268 op/s 2023-07-29T16:25:47.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 748572 8.90 MB/sec execute 332 sec latency 43.163 ms 2023-07-29T16:25:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:47 smithi039 ceph-mon[128858]: pgmap v681: 97 pgs: 97 active+clean; 143 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.2 MiB/s wr, 270 op/s 2023-07-29T16:25:48.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:47 smithi156 ceph-mon[129245]: pgmap v681: 97 pgs: 97 active+clean; 143 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.2 MiB/s wr, 270 op/s 2023-07-29T16:25:48.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:47 smithi031 ceph-mon[95504]: pgmap v681: 97 pgs: 97 active+clean; 143 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.2 MiB/s wr, 270 op/s 2023-07-29T16:25:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.687+0000 7f98f3dff700 1 -- 172.21.15.31:0/1792859010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec100340 msgr2=0x7f98ec100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.687+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1792859010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec100340 0x7f98ec100750 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098f0 tx=0x7f98e8009c00 comp rx=0 tx=0).stop 2023-07-29T16:25:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 -- 172.21.15.31:0/1792859010 shutdown_connections 2023-07-29T16:25:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1792859010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98ec101ec0 0x7f98ec108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1792859010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec100340 0x7f98ec100750 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1792859010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec101530 0x7f98ec101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 -- 172.21.15.31:0/1792859010 >> 172.21.15.31:0/1792859010 conn(0x7f98ec0fb920 msgr2=0x7f98ec0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 -- 172.21.15.31:0/1792859010 shutdown_connections 2023-07-29T16:25:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.688+0000 7f98f3dff700 1 -- 172.21.15.31:0/1792859010 wait complete. 2023-07-29T16:25:48.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 Processor -- start 2023-07-29T16:25:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 -- start start 2023-07-29T16:25:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 0x7f98ec1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec101530 0x7f98ec198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98ec101ec0 0x7f98ec1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec199e00 con 0x7f98ec101ec0 2023-07-29T16:25:48.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec199f70 con 0x7f98ec100340 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.689+0000 7f98f3dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ec19a130 con 0x7f98ec101530 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 0x7f98ec1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f139a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec101530 0x7f98ec198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 0x7f98ec1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46976/0 (socket says 172.21.15.31:46976) 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1362583396 learned_addr learned my addr 172.21.15.31:0/1362583396 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f239c700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98ec101ec0 0x7f98ec1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1362583396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec101530 msgr2=0x7f98ec198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec101530 0x7f98ec198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98ec101ec0 msgr2=0x7f98ec1997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98ec101ec0 0x7f98ec1997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1362583396 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8009520 con 0x7f98ec100340 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.690+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 0x7f98ec1983e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009fd0 tx=0x7f98e800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.691+0000 7f98e2ffd700 1 -- 172.21.15.31:0/1362583396 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e801c070 con 0x7f98ec100340 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.691+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ec19a3b0 con 0x7f98ec100340 2023-07-29T16:25:48.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.691+0000 7f98e2ffd700 1 -- 172.21.15.31:0/1362583396 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98e800f7d0 con 0x7f98ec100340 2023-07-29T16:25:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.691+0000 7f98e2ffd700 1 -- 172.21.15.31:0/1362583396 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8003a80 con 0x7f98ec100340 2023-07-29T16:25:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.691+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec1a0bd0 con 0x7f98ec100340 2023-07-29T16:25:48.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.694+0000 7f98e2ffd700 1 -- 172.21.15.31:0/1362583396 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f98e8020030 con 0x7f98ec100340 2023-07-29T16:25:48.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.695+0000 7f990775db80 1 Processor -- start 2023-07-29T16:25:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 -- start start 2023-07-29T16:25:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98e2ffd700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98d80a0790 0x7f98d80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece22a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece22b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece22c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece1829b0 con 0x563ece22adf0 2023-07-29T16:25:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece180370 con 0x563ece0866b0 2023-07-29T16:25:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f990775db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece17bc60 con 0x563ece22bcf0 2023-07-29T16:25:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98e2ffd700 1 -- 172.21.15.31:0/1362583396 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f98e80ccb10 con 0x7f98ec100340 2023-07-29T16:25:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece22a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98f239c700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98d80a0790 0x7f98d80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece22a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46982/0 (socket says 172.21.15.31:46982) 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98f139a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece22b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.696+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1676792001 learned_addr learned my addr 172.21.15.31:0/1676792001 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f239c700 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece22c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1676792001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 msgr2=0x563ece22c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece22c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1676792001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 msgr2=0x563ece22b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece22b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1676792001 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ecdfa9d80 con 0x563ece0866b0 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece22a8b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f98e809a1d0 tx=0x7f98e8096600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d46eaf040b95ab4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98f239c700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98d80a0790 0x7f98d80a2c40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f98e4007b50 tx=0x7f98e4009200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98d7fff700 1 -- 172.21.15.31:0/1676792001 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8099ba0 con 0x563ece0866b0 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98d7fff700 1 -- 172.21.15.31:0/1676792001 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98e8064140 con 0x563ece0866b0 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.697+0000 7f98d7fff700 1 -- 172.21.15.31:0/1676792001 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e80a2600 con 0x563ece0866b0 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 -- 172.21.15.31:0/1676792001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 msgr2=0x563ece22a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece22a8b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f98e809a1d0 tx=0x7f98e8096600 comp rx=0 tx=0).stop 2023-07-29T16:25:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 -- 172.21.15.31:0/1676792001 shutdown_connections 2023-07-29T16:25:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece22c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece22a8b0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 --2- 172.21.15.31:0/1676792001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece22b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 -- 172.21.15.31:0/1676792001 >> 172.21.15.31:0/1676792001 conn(0x563ece21ffd0 msgr2=0x563ece225300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:48.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 -- 172.21.15.31:0/1676792001 shutdown_connections 2023-07-29T16:25:48.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.698+0000 7f990775db80 1 -- 172.21.15.31:0/1676792001 wait complete. 2023-07-29T16:25:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.699+0000 7f990775db80 1 -- start start 2023-07-29T16:25:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.699+0000 7f990775db80 1 Processor -- start 2023-07-29T16:25:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece233af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece234030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece23b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece1184b0 con 0x563ece22adf0 2023-07-29T16:25:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece118150 con 0x563ece0866b0 2023-07-29T16:25:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ecdfff8c0 con 0x563ece22bcf0 2023-07-29T16:25:48.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f990775db80 10 client.0 _ll_register_callbacks cb 0x563ece1215c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece233af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece233af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:46984/0 (socket says 172.21.15.31:46984) 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1842542979 learned_addr learned my addr 172.21.15.31:0/1842542979 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f98f239c700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece23b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.700+0000 7f98f139a700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece234030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1842542979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 msgr2=0x563ece23b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece23b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1842542979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 msgr2=0x563ece234030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece234030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1842542979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ec1a0bd0 con 0x563ece0866b0 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98d57fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece233af0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f98e80a20d0 tx=0x7f98e800ba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a787db04f25b977b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98d57fa700 1 -- 172.21.15.31:0/1842542979 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e80a0110 con 0x563ece0866b0 2023-07-29T16:25:48.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98d57fa700 1 -- 172.21.15.31:0/1842542979 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f98e809e040 con 0x563ece0866b0 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ece1a6cb0 con 0x563ece0866b0 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f98d57fa700 1 -- 172.21.15.31:0/1842542979 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98e80a2600 con 0x563ece0866b0 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f990775db80 10 client.28634 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.701+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563ecdf92c00 con 0x563ece0866b0 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f98d57fa700 1 -- 172.21.15.31:0/1842542979 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563ece1a6cb0 con 0x563ece0866b0 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f98d57fa700 1 -- 172.21.15.31:0/1842542979 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f98e809a4a0 con 0x563ece0866b0 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f990775db80 10 client.28634 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:48.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f990775db80 10 client.28634 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f990775db80 20 client.28634 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f990775db80 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece23f250 0x563ece241660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f990775db80 4 client.28634 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ece1f9d10 con 0x563ece23f250 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.702+0000 7f98f139a700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece23f250 0x563ece241660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.703+0000 7f98f139a700 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece23f250 0x563ece241660 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba03b56c2aebc2dc server_cookie=706c19bfd06e6d10 in_seq=0 out_seq=0 2023-07-29T16:25:48.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.703+0000 7f98d57fa700 10 client.28634 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f98d57fa700 1 -- 172.21.15.31:0/1842542979 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563ece1f9d10 con 0x563ece23f250 2023-07-29T16:25:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f98d57fa700 10 client.28634 handle_command_reply: tid=0 2023-07-29T16:25:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 client.28634 shutdown 2023-07-29T16:25:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 msgr2=0x563ece233af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece233af0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f98e80a20d0 tx=0x7f98e800ba60 comp rx=0 tx=0).stop 2023-07-29T16:25:48.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 shutdown_connections 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece23f250 0x563ece241660 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece22bcf0 0x563ece23b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece0866b0 0x563ece233af0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f98d57fa700 0 client.28634 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 --2- 172.21.15.31:0/1842542979 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22adf0 0x563ece234030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 >> 172.21.15.31:0/1842542979 conn(0x563ece21ffd0 msgr2=0x563ece224a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:48.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 shutdown_connections 2023-07-29T16:25:48.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 1 -- 172.21.15.31:0/1842542979 wait complete. 2023-07-29T16:25:48.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.704+0000 7f990775db80 20 client.28634 trim_cache size 0 max 16384 2023-07-29T16:25:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 Processor -- start 2023-07-29T16:25:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 -- start start 2023-07-29T16:25:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece225df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece23b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece2331a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece231c60 con 0x563ece22bcf0 2023-07-29T16:25:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece226710 con 0x563ece22adf0 2023-07-29T16:25:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.721+0000 7f990775db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece22f550 con 0x563ece0866b0 2023-07-29T16:25:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece23b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f239c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece2331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece225df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece23b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47000/0 (socket says 172.21.15.31:47000) 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 -- 172.21.15.31:0/3805039115 learned_addr learned my addr 172.21.15.31:0/3805039115 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 -- 172.21.15.31:0/3805039115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 msgr2=0x563ece225df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece225df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 -- 172.21.15.31:0/3805039115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 msgr2=0x563ece2331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece2331a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 -- 172.21.15.31:0/3805039115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ecdfd01f0 con 0x563ece22adf0 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98f139a700 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece23b030 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009840 tx=0x7f98dc009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c552ad265b3192 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98e0bff700 1 -- 172.21.15.31:0/3805039115 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc01f310 con 0x563ece22adf0 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98e0bff700 1 -- 172.21.15.31:0/3805039115 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98dc018040 con 0x563ece22adf0 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.722+0000 7f98e0bff700 1 -- 172.21.15.31:0/3805039115 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc005580 con 0x563ece22adf0 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 -- 172.21.15.31:0/3805039115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 msgr2=0x563ece23b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece23b030 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009840 tx=0x7f98dc009310 comp rx=0 tx=0).stop 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 -- 172.21.15.31:0/3805039115 shutdown_connections 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece2331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece225df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 --2- 172.21.15.31:0/3805039115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece23b030 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 -- 172.21.15.31:0/3805039115 >> 172.21.15.31:0/3805039115 conn(0x563ece21ffd0 msgr2=0x563ece22e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 -- 172.21.15.31:0/3805039115 shutdown_connections 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.723+0000 7f990775db80 1 -- 172.21.15.31:0/3805039115 wait complete. 2023-07-29T16:25:48.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 -- start start 2023-07-29T16:25:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 Processor -- start 2023-07-29T16:25:48.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece236d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece237270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece23a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece23a7d0 con 0x563ece22bcf0 2023-07-29T16:25:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece23a940 con 0x563ece22adf0 2023-07-29T16:25:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.724+0000 7f990775db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ece23ab00 con 0x563ece0866b0 2023-07-29T16:25:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f990775db80 10 client.0 _ll_register_callbacks cb 0x563ece1215c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece236d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece237270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f1b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece236d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51010/0 (socket says 172.21.15.31:51010) 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f1b9b700 1 -- 172.21.15.31:0/1467495094 learned_addr learned my addr 172.21.15.31:0/1467495094 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f239c700 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece23a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 -- 172.21.15.31:0/1467495094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 msgr2=0x563ece236d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece236d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 -- 172.21.15.31:0/1467495094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 msgr2=0x563ece23a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece23a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 -- 172.21.15.31:0/1467495094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc009ba0 con 0x563ece22adf0 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98f139a700 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece237270 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x563ece2221a0 tx=0x7f98dc01f250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65de112b14fba82a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.725+0000 7f98d57fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f98d57fa700 1 -- 172.21.15.31:0/1467495094 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc005710 con 0x563ece22adf0 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f98d57fa700 1 -- 172.21.15.31:0/1467495094 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f98dc0047c0 con 0x563ece22adf0 2023-07-29T16:25:48.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98e8064140 con 0x563ece22adf0 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f98d57fa700 1 -- 172.21.15.31:0/1467495094 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98dc01f400 con 0x563ece22adf0 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f990775db80 10 client.28640 fetch_fsmap learned FSMap version 31 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563ecdfa8250 con 0x563ece22adf0 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f98d57fa700 1 -- 172.21.15.31:0/1467495094 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98e8064140 con 0x563ece22adf0 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f98d57fa700 1 -- 172.21.15.31:0/1467495094 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f98dc02d070 con 0x563ece22adf0 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f990775db80 10 client.28640 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:25:48.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.726+0000 7f990775db80 10 client.28640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:25:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f990775db80 20 client.28640 populate_metadata read hostname 'smithi031' 2023-07-29T16:25:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f990775db80 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece291be0 0x563ece293ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:25:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f990775db80 4 client.28640 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:25:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563ece174ce0 con 0x563ece291be0 2023-07-29T16:25:48.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece291be0 0x563ece293ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:25:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f98f1b9b700 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece291be0 0x563ece293ff0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e5761d9dc8eda5 server_cookie=5bec698b0c5a0811 in_seq=0 out_seq=0 2023-07-29T16:25:48.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.727+0000 7f98d57fa700 10 client.28640 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.835+0000 7f98d57fa700 1 -- 172.21.15.31:0/1467495094 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563ece174ce0 con 0x563ece291be0 2023-07-29T16:25:48.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f98d57fa700 10 client.28640 handle_command_reply: tid=0 2023-07-29T16:25:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 client.28640 shutdown 2023-07-29T16:25:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 msgr2=0x563ece237270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece237270 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x563ece2221a0 tx=0x7f98dc01f250 comp rx=0 tx=0).stop 2023-07-29T16:25:48.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 shutdown_connections 2023-07-29T16:25:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ece291be0 0x563ece293ff0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ece22bcf0 0x563ece23a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ece0866b0 0x563ece236d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f98d57fa700 0 client.28640 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:25:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 --2- 172.21.15.31:0/1467495094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ece22adf0 0x563ece237270 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 >> 172.21.15.31:0/1467495094 conn(0x563ece21ffd0 msgr2=0x563ece22e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 shutdown_connections 2023-07-29T16:25:48.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 1 -- 172.21.15.31:0/1467495094 wait complete. 2023-07-29T16:25:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.836+0000 7f990775db80 20 client.28640 trim_cache size 0 max 16384 2023-07-29T16:25:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:25:48.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:25:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:25:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:25:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98d80a0790 msgr2=0x7f98d80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98d80a0790 0x7f98d80a2c40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f98e4007b50 tx=0x7f98e4009200 comp rx=0 tx=0).stop 2023-07-29T16:25:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 msgr2=0x7f98ec1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:25:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 0x7f98ec1983e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009fd0 tx=0x7f98e800b840 comp rx=0 tx=0).stop 2023-07-29T16:25:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 shutdown_connections 2023-07-29T16:25:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98d80a0790 0x7f98d80a2c40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f98ec101ec0 0x7f98ec1997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f98ec100340 0x7f98ec1983e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 --2- 172.21.15.31:0/1362583396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f98ec101530 0x7f98ec198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:25:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 >> 172.21.15.31:0/1362583396 conn(0x7f98ec0fb920 msgr2=0x7f98ec0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:25:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 shutdown_connections 2023-07-29T16:25:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:25:48.837+0000 7f98f3dff700 1 -- 172.21.15.31:0/1362583396 wait complete. 2023-07-29T16:25:48.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 749697 8.89 MB/sec execute 333 sec latency 62.190 ms 2023-07-29T16:25:49.886 INFO:tasks.workunit.client.0.smithi031.stdout: 1 751826 8.89 MB/sec execute 334 sec latency 13.148 ms 2023-07-29T16:25:50.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:50 smithi156 ceph-mon[129245]: pgmap v682: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:25:50.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 753245 8.89 MB/sec execute 335 sec latency 11.679 ms 2023-07-29T16:25:50.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:50 smithi031 ceph-mon[95504]: pgmap v682: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:25:51.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:50 smithi039 ceph-mon[128858]: pgmap v682: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:25:51.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 755267 8.89 MB/sec execute 336 sec latency 16.315 ms 2023-07-29T16:25:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:52 smithi156 ceph-mon[129245]: pgmap v683: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.0 MiB/s wr, 216 op/s 2023-07-29T16:25:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:25:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:52 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:52.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 756639 8.90 MB/sec execute 337 sec latency 52.497 ms 2023-07-29T16:25:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:52 smithi031 ceph-mon[95504]: pgmap v683: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.0 MiB/s wr, 216 op/s 2023-07-29T16:25:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:25:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:52 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:53.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:52 smithi039 ceph-mon[128858]: pgmap v683: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 2.0 MiB/s wr, 216 op/s 2023-07-29T16:25:53.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:25:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:52 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:25:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:25:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:25:53.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:25:53.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:25:53.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:53.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 757765 8.89 MB/sec execute 338 sec latency 102.469 ms 2023-07-29T16:25:53.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:25:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:25:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:25:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:25:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:25:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:54.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:25:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:25:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:25:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:25:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:25:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:25:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:54 smithi156 ceph-mon[129245]: pgmap v684: 97 pgs: 97 active+clean; 143 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2023-07-29T16:25:54.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 759985 8.90 MB/sec execute 339 sec latency 13.261 ms 2023-07-29T16:25:54.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:54 smithi031 ceph-mon[95504]: pgmap v684: 97 pgs: 97 active+clean; 143 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2023-07-29T16:25:55.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:54 smithi039 ceph-mon[128858]: pgmap v684: 97 pgs: 97 active+clean; 143 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2023-07-29T16:25:55.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 761291 8.89 MB/sec execute 340 sec latency 14.498 ms 2023-07-29T16:25:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:56 smithi156 ceph-mon[129245]: pgmap v685: 97 pgs: 97 active+clean; 145 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 2.5 MiB/s wr, 275 op/s 2023-07-29T16:25:56.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 763517 8.90 MB/sec execute 341 sec latency 14.495 ms 2023-07-29T16:25:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:56 smithi031 ceph-mon[95504]: pgmap v685: 97 pgs: 97 active+clean; 145 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 2.5 MiB/s wr, 275 op/s 2023-07-29T16:25:57.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:56 smithi039 ceph-mon[128858]: pgmap v685: 97 pgs: 97 active+clean; 145 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 2.5 MiB/s wr, 275 op/s 2023-07-29T16:25:57.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 764588 8.89 MB/sec execute 342 sec latency 50.917 ms 2023-07-29T16:25:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:25:57 smithi039 ceph-mon[128858]: pgmap v686: 97 pgs: 97 active+clean; 145 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 258 op/s 2023-07-29T16:25:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:25:57 smithi156 ceph-mon[129245]: pgmap v686: 97 pgs: 97 active+clean; 145 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 258 op/s 2023-07-29T16:25:58.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:25:57 smithi031 ceph-mon[95504]: pgmap v686: 97 pgs: 97 active+clean; 145 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.2 MiB/s wr, 258 op/s 2023-07-29T16:25:58.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 766778 8.90 MB/sec execute 343 sec latency 25.426 ms 2023-07-29T16:25:59.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 767828 8.89 MB/sec execute 344 sec latency 68.314 ms 2023-07-29T16:26:00.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:00 smithi156 ceph-mon[129245]: pgmap v687: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.8 MiB/s wr, 290 op/s 2023-07-29T16:26:00.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 770297 8.90 MB/sec execute 345 sec latency 16.720 ms 2023-07-29T16:26:00.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:00 smithi031 ceph-mon[95504]: pgmap v687: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.8 MiB/s wr, 290 op/s 2023-07-29T16:26:01.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:00 smithi039 ceph-mon[128858]: pgmap v687: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.8 MiB/s wr, 290 op/s 2023-07-29T16:26:01.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 771601 8.89 MB/sec execute 346 sec latency 11.304 ms 2023-07-29T16:26:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 -- 172.21.15.31:0/244966234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e410c920 msgr2=0x7f57e410ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 --2- 172.21.15.31:0/244966234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e410c920 0x7f57e410ed00 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009990 tx=0x7f57dc009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 -- 172.21.15.31:0/244966234 shutdown_connections 2023-07-29T16:26:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 --2- 172.21.15.31:0/244966234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e410c920 0x7f57e410ed00 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:01.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 --2- 172.21.15.31:0/244966234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 0x7f57e40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 --2- 172.21.15.31:0/244966234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57e40737b0 0x7f57e4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 -- 172.21.15.31:0/244966234 >> 172.21.15.31:0/244966234 conn(0x7f57e40fb970 msgr2=0x7f57e40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.994+0000 7f57eafff700 1 -- 172.21.15.31:0/244966234 shutdown_connections 2023-07-29T16:26:01.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.995+0000 7f57eafff700 1 -- 172.21.15.31:0/244966234 wait complete. 2023-07-29T16:26:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.995+0000 7f57eafff700 1 Processor -- start 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 -- start start 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 0x7f57e419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 0x7f57e419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57e410c920 0x7f57e419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e419e180 con 0x7f57e410c920 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e419e2f0 con 0x7f57e40737b0 2023-07-29T16:26:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57eafff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e419e4b0 con 0x7f57e4074d90 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 0x7f57e419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 0x7f57e419c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52552/0 (socket says 172.21.15.31:52552) 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57e8d9b700 1 -- 172.21.15.31:0/3462647061 learned_addr learned my addr 172.21.15.31:0/3462647061 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57e959c700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57e410c920 0x7f57e419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.996+0000 7f57e3fff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 0x7f57e419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e8d9b700 1 -- 172.21.15.31:0/3462647061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 msgr2=0x7f57e419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 0x7f57e419cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e8d9b700 1 -- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57e410c920 msgr2=0x7f57e419db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57e410c920 0x7f57e419db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e8d9b700 1 -- 172.21.15.31:0/3462647061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc0095f0 con 0x7f57e40737b0 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e3fff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 0x7f57e419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 0x7f57e419c760 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f57d800efc0 tx=0x7f57d800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e1ffb700 1 -- 172.21.15.31:0/3462647061 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d800ca70 con 0x7f57e40737b0 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e1ffb700 1 -- 172.21.15.31:0/3462647061 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57d800f040 con 0x7f57e40737b0 2023-07-29T16:26:02.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e419e730 con 0x7f57e40737b0 2023-07-29T16:26:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.997+0000 7f57e1ffb700 1 -- 172.21.15.31:0/3462647061 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8014480 con 0x7f57e40737b0 2023-07-29T16:26:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:01.998+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e41a4f50 con 0x7f57e40737b0 2023-07-29T16:26:02.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.001+0000 7f57e1ffb700 1 -- 172.21.15.31:0/3462647061 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f57d8007500 con 0x7f57e40737b0 2023-07-29T16:26:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.002+0000 7f57fe9ccb80 1 Processor -- start 2023-07-29T16:26:02.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 -- start start 2023-07-29T16:26:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e1ffb700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57d40a0790 0x7f57d40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e1ffb700 1 -- 172.21.15.31:0/3462647061 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f57d80cbe20 con 0x7f57e40737b0 2023-07-29T16:26:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46838b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4683df0 0x55bce46847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4685160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce45db9b0 con 0x55bce44df6b0 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce45d9370 con 0x55bce4684cf0 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce45d4c60 con 0x55bce4683df0 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e959c700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57d40a0790 0x7f57d40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46838b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4685160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4685160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52568/0 (socket says 172.21.15.31:52568) 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e959c700 1 -- 172.21.15.31:0/562679553 learned_addr learned my addr 172.21.15.31:0/562679553 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.003+0000 7f57e3fff700 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4683df0 0x55bce46847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 -- 172.21.15.31:0/562679553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4683df0 msgr2=0x55bce46847b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4683df0 0x55bce46847b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 -- 172.21.15.31:0/562679553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 msgr2=0x55bce46838b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46838b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 -- 172.21.15.31:0/562679553 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bce4402d80 con 0x55bce4684cf0 2023-07-29T16:26:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e3fff700 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4683df0 0x55bce46847b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57d40a0790 0x7f57d40a2c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f57dc015000 tx=0x7f57dc00b890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57e959c700 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4685160 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c010 tx=0x7f57dc029000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=554f87415b34fb8a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57ceffd700 1 -- 172.21.15.31:0/562679553 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc01d070 con 0x55bce4684cf0 2023-07-29T16:26:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57ceffd700 1 -- 172.21.15.31:0/562679553 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57dc005730 con 0x55bce4684cf0 2023-07-29T16:26:02.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.004+0000 7f57ceffd700 1 -- 172.21.15.31:0/562679553 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0297d0 con 0x55bce4684cf0 2023-07-29T16:26:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/562679553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 msgr2=0x55bce4685160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4685160 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c010 tx=0x7f57dc029000 comp rx=0 tx=0).stop 2023-07-29T16:26:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/562679553 shutdown_connections 2023-07-29T16:26:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4685160 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/562679553 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4683df0 0x55bce46847b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/562679553 >> 172.21.15.31:0/562679553 conn(0x55bce4678fd0 msgr2=0x55bce467e300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/562679553 shutdown_connections 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.005+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/562679553 wait complete. 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 -- start start 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 Processor -- start 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce468cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4683df0 0x55bce468d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4694340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce45714b0 con 0x55bce4683df0 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.006+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce4571150 con 0x55bce4684cf0 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce44588c0 con 0x55bce44df6b0 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57fe9ccb80 10 client.0 _ll_register_callbacks cb 0x55bce457a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4694340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce468cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4694340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52582/0 (socket says 172.21.15.31:52582) 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 -- 172.21.15.31:0/866488913 learned_addr learned my addr 172.21.15.31:0/866488913 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e3fff700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4683df0 0x55bce468d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 -- 172.21.15.31:0/866488913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 msgr2=0x55bce468cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce468cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 -- 172.21.15.31:0/866488913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4683df0 msgr2=0x55bce468d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4683df0 0x55bce468d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 -- 172.21.15.31:0/866488913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e41a4f50 con 0x55bce4684cf0 2023-07-29T16:26:02.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce468cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57e959c700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4694340 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57dc006010 tx=0x7f57dc005920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff941ed62c9c85cd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.007+0000 7f57c3fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57c3fff700 1 -- 172.21.15.31:0/866488913 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc01d040 con 0x55bce4684cf0 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bce45ffcb0 con 0x55bce4684cf0 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57c3fff700 1 -- 172.21.15.31:0/866488913 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f57dc029c90 con 0x55bce4684cf0 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57c3fff700 1 -- 172.21.15.31:0/866488913 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57dc004810 con 0x55bce4684cf0 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57fe9ccb80 10 client.28652 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f57d00098f0 con 0x55bce4684cf0 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57c3fff700 1 -- 172.21.15.31:0/866488913 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bce45ffcb0 con 0x55bce4684cf0 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.008+0000 7f57c3fff700 1 -- 172.21.15.31:0/866488913 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f57dc00fc60 con 0x55bce4684cf0 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57fe9ccb80 10 client.28652 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57fe9ccb80 10 client.28652 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57fe9ccb80 20 client.28652 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce469aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57fe9ccb80 4 client.28652 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bce4652d10 con 0x55bce4698660 2023-07-29T16:26:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce469aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce469aa40 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd6f5a0b3d4c5da server_cookie=db906bc8deea74cf in_seq=0 out_seq=0 2023-07-29T16:26:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.009+0000 7f57c3fff700 10 client.28652 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57c3fff700 1 -- 172.21.15.31:0/866488913 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bce4652d10 con 0x55bce4698660 2023-07-29T16:26:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57c3fff700 10 client.28652 handle_command_reply: tid=0 2023-07-29T16:26:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 client.28652 shutdown 2023-07-29T16:26:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 msgr2=0x55bce4694340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4694340 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f57dc006010 tx=0x7f57dc005920 comp rx=0 tx=0).stop 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 shutdown_connections 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce469aa40 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57c3fff700 0 client.28652 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4684cf0 0x55bce4694340 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce468cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/866488913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4683df0 0x55bce468d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 >> 172.21.15.31:0/866488913 conn(0x55bce4678fd0 msgr2=0x55bce467da40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:02.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 shutdown_connections 2023-07-29T16:26:02.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.011+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/866488913 wait complete. 2023-07-29T16:26:02.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.012+0000 7f57fe9ccb80 20 client.28652 trim_cache size 0 max 16384 2023-07-29T16:26:02.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 Processor -- start 2023-07-29T16:26:02.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 -- start start 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce469aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4684cf0 0x55bce46850c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce4684490 con 0x55bce4684cf0 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce4685600 con 0x55bce4683df0 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.029+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce46789d0 con 0x55bce44df6b0 2023-07-29T16:26:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce469aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce469aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51084/0 (socket says 172.21.15.31:51084) 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e959c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4684cf0 0x55bce46850c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 -- 172.21.15.31:0/1663394602 learned_addr learned my addr 172.21.15.31:0/1663394602 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e3fff700 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 -- 172.21.15.31:0/1663394602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 msgr2=0x55bce46922f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46922f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 -- 172.21.15.31:0/1663394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4684cf0 msgr2=0x55bce46850c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4684cf0 0x55bce46850c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 -- 172.21.15.31:0/1663394602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc020040 con 0x55bce44df6b0 2023-07-29T16:26:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce469aa40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57d8007fd0 tx=0x7f57d80cb040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fdd2763e116f7d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.030+0000 7f57cf7fe700 1 -- 172.21.15.31:0/1663394602 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d80b1ea0 con 0x55bce44df6b0 2023-07-29T16:26:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57cf7fe700 1 -- 172.21.15.31:0/1663394602 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57d80a1910 con 0x55bce44df6b0 2023-07-29T16:26:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57cf7fe700 1 -- 172.21.15.31:0/1663394602 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d809d320 con 0x55bce44df6b0 2023-07-29T16:26:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1663394602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 msgr2=0x55bce469aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce469aa40 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57d8007fd0 tx=0x7f57d80cb040 comp rx=0 tx=0).stop 2023-07-29T16:26:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1663394602 shutdown_connections 2023-07-29T16:26:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce4684cf0 0x55bce46850c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce44df6b0 0x55bce469aa40 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1663394602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1663394602 >> 172.21.15.31:0/1663394602 conn(0x55bce4678fd0 msgr2=0x55bce468ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.031+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1663394602 shutdown_connections 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.032+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1663394602 wait complete. 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.032+0000 7f57fe9ccb80 1 -- start start 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.032+0000 7f57fe9ccb80 1 Processor -- start 2023-07-29T16:26:02.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46e6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4684cf0 0x55bce46e86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce46e8e10 con 0x55bce44df6b0 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce46e8f80 con 0x55bce4683df0 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce46e9140 con 0x55bce4684cf0 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57fe9ccb80 10 client.0 _ll_register_callbacks cb 0x55bce457a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57e8d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46e6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:52600/0 (socket says 172.21.15.31:52600) 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57e3fff700 1 -- 172.21.15.31:0/1964675017 learned_addr learned my addr 172.21.15.31:0/1964675017 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.033+0000 7f57e959c700 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4684cf0 0x55bce46e86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57e3fff700 1 -- 172.21.15.31:0/1964675017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4684cf0 msgr2=0x55bce46e86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57e3fff700 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4684cf0 0x55bce46e86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57e3fff700 1 -- 172.21.15.31:0/1964675017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 msgr2=0x55bce46922f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57e3fff700 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46922f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57e3fff700 1 -- 172.21.15.31:0/1964675017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d8023070 con 0x55bce4683df0 2023-07-29T16:26:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57e3fff700 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46e6160 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f57d0012cb0 tx=0x7f57d0009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e584fdd8e65ae7a5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57c3fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57c3fff700 1 -- 172.21.15.31:0/1964675017 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0017040 con 0x55bce4683df0 2023-07-29T16:26:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57dc005730 con 0x55bce4683df0 2023-07-29T16:26:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57c3fff700 1 -- 172.21.15.31:0/1964675017 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f57d0011cc0 con 0x55bce4683df0 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57c3fff700 1 -- 172.21.15.31:0/1964675017 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57d0011e80 con 0x55bce4683df0 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.034+0000 7f57fe9ccb80 10 client.28658 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bce43ebc00 con 0x55bce4683df0 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57c3fff700 1 -- 172.21.15.31:0/1964675017 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57dc005730 con 0x55bce4683df0 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57c3fff700 1 -- 172.21.15.31:0/1964675017 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f57d000f600 con 0x55bce4683df0 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 10 client.28658 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 10 client.28658 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 20 client.28658 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce46ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 4 client.28658 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:26:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bce45cdce0 con 0x55bce4698660 2023-07-29T16:26:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.035+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce46ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.036+0000 7f57e8d9b700 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce46ecc00 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b14e2dde4238861 server_cookie=1c032b71382601b2 in_seq=0 out_seq=0 2023-07-29T16:26:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.036+0000 7f57c3fff700 10 client.28658 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57c3fff700 1 -- 172.21.15.31:0/1964675017 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bce45cdce0 con 0x55bce4698660 2023-07-29T16:26:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57c3fff700 10 client.28658 handle_command_reply: tid=0 2023-07-29T16:26:02.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 client.28658 shutdown 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 msgr2=0x55bce46e6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46e6160 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f57d0012cb0 tx=0x7f57d0009d40 comp rx=0 tx=0).stop 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 shutdown_connections 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bce4698660 0x55bce46ecc00 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce4684cf0 0x55bce46e86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57c3fff700 0 client.28658 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:02.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bce44df6b0 0x55bce46922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 --2- 172.21.15.31:0/1964675017 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bce4683df0 0x55bce46e6160 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 >> 172.21.15.31:0/1964675017 conn(0x55bce4678fd0 msgr2=0x55bce468b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 shutdown_connections 2023-07-29T16:26:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.112+0000 7f57fe9ccb80 1 -- 172.21.15.31:0/1964675017 wait complete. 2023-07-29T16:26:02.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57fe9ccb80 20 client.28658 trim_cache size 0 max 16384 2023-07-29T16:26:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:26:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:26:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:26:02.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57d40a0790 msgr2=0x7f57d40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57d40a0790 0x7f57d40a2c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f57dc015000 tx=0x7f57dc00b890 comp rx=0 tx=0).stop 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 msgr2=0x7f57e419c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 0x7f57e419c760 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f57d800efc0 tx=0x7f57d800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 shutdown_connections 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57d40a0790 0x7f57d40a2c40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57e410c920 0x7f57e419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57e40737b0 0x7f57e419c760 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 --2- 172.21.15.31:0/3462647061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57e4074d90 0x7f57e419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.113+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 >> 172.21.15.31:0/3462647061 conn(0x7f57e40fb970 msgr2=0x7f57e40ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.114+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 shutdown_connections 2023-07-29T16:26:02.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:02.114+0000 7f57eafff700 1 -- 172.21.15.31:0/3462647061 wait complete. 2023-07-29T16:26:02.821 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:02 smithi156 ceph-mon[129245]: pgmap v688: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2023-07-29T16:26:02.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 773493 8.90 MB/sec execute 347 sec latency 65.349 ms 2023-07-29T16:26:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:02 smithi031 ceph-mon[95504]: pgmap v688: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2023-07-29T16:26:03.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:02 smithi039 ceph-mon[128858]: pgmap v688: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2023-07-29T16:26:03.887 INFO:tasks.workunit.client.0.smithi031.stdout: 1 774770 8.89 MB/sec execute 348 sec latency 27.296 ms 2023-07-29T16:26:04.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:04 smithi156 ceph-mon[129245]: pgmap v689: 97 pgs: 97 active+clean; 150 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2023-07-29T16:26:04.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 777041 8.90 MB/sec execute 349 sec latency 26.446 ms 2023-07-29T16:26:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:04 smithi031 ceph-mon[95504]: pgmap v689: 97 pgs: 97 active+clean; 150 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2023-07-29T16:26:05.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:04 smithi039 ceph-mon[128858]: pgmap v689: 97 pgs: 97 active+clean; 150 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 298 op/s 2023-07-29T16:26:05.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 778279 8.90 MB/sec execute 350 sec latency 31.089 ms 2023-07-29T16:26:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:06 smithi156 ceph-mon[129245]: pgmap v690: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 2.8 MiB/s wr, 294 op/s 2023-07-29T16:26:06.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 780581 8.91 MB/sec execute 351 sec latency 12.825 ms 2023-07-29T16:26:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:06 smithi031 ceph-mon[95504]: pgmap v690: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 2.8 MiB/s wr, 294 op/s 2023-07-29T16:26:07.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:06 smithi039 ceph-mon[128858]: pgmap v690: 97 pgs: 97 active+clean; 146 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 2.8 MiB/s wr, 294 op/s 2023-07-29T16:26:07.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 781849 8.90 MB/sec execute 352 sec latency 44.239 ms 2023-07-29T16:26:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:07 smithi039 ceph-mon[128858]: pgmap v691: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.3 MiB/s wr, 252 op/s 2023-07-29T16:26:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:07 smithi156 ceph-mon[129245]: pgmap v691: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.3 MiB/s wr, 252 op/s 2023-07-29T16:26:08.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:07 smithi031 ceph-mon[95504]: pgmap v691: 97 pgs: 97 active+clean; 147 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.3 MiB/s wr, 252 op/s 2023-07-29T16:26:08.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 784070 8.91 MB/sec execute 353 sec latency 23.754 ms 2023-07-29T16:26:09.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 785436 8.90 MB/sec execute 354 sec latency 13.573 ms 2023-07-29T16:26:10.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:10 smithi156 ceph-mon[129245]: pgmap v692: 97 pgs: 97 active+clean; 148 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 307 op/s 2023-07-29T16:26:10.888 INFO:tasks.workunit.client.0.smithi031.stdout: 1 787652 8.91 MB/sec execute 355 sec latency 34.734 ms 2023-07-29T16:26:10.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:10 smithi031 ceph-mon[95504]: pgmap v692: 97 pgs: 97 active+clean; 148 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 307 op/s 2023-07-29T16:26:11.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:10 smithi039 ceph-mon[128858]: pgmap v692: 97 pgs: 97 active+clean; 148 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.8 MiB/s wr, 307 op/s 2023-07-29T16:26:11.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 788782 8.91 MB/sec execute 356 sec latency 131.502 ms 2023-07-29T16:26:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:12 smithi156 ceph-mon[129245]: pgmap v693: 97 pgs: 97 active+clean; 148 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.1 MiB/s wr, 229 op/s 2023-07-29T16:26:12.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 790942 8.91 MB/sec execute 357 sec latency 30.253 ms 2023-07-29T16:26:12.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:12 smithi031 ceph-mon[95504]: pgmap v693: 97 pgs: 97 active+clean; 148 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.1 MiB/s wr, 229 op/s 2023-07-29T16:26:13.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:12 smithi039 ceph-mon[128858]: pgmap v693: 97 pgs: 97 active+clean; 148 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.1 MiB/s wr, 229 op/s 2023-07-29T16:26:13.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 792258 8.91 MB/sec execute 358 sec latency 25.176 ms 2023-07-29T16:26:14.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:14 smithi156 ceph-mon[129245]: pgmap v694: 97 pgs: 97 active+clean; 152 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 312 op/s 2023-07-29T16:26:14.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 794490 8.92 MB/sec execute 359 sec latency 12.729 ms 2023-07-29T16:26:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:14 smithi031 ceph-mon[95504]: pgmap v694: 97 pgs: 97 active+clean; 152 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 312 op/s 2023-07-29T16:26:15.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:14 smithi039 ceph-mon[128858]: pgmap v694: 97 pgs: 97 active+clean; 152 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.7 MiB/s wr, 312 op/s 2023-07-29T16:26:15.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:26:15.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 -- 172.21.15.31:0/858521066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc105f20 msgr2=0x7f18dc1083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 --2- 172.21.15.31:0/858521066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc105f20 0x7f18dc1083d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f18d4009e00 tx=0x7f18d400b330 comp rx=0 tx=0).stop 2023-07-29T16:26:15.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 -- 172.21.15.31:0/858521066 shutdown_connections 2023-07-29T16:26:15.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 --2- 172.21.15.31:0/858521066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc105f20 0x7f18dc1083d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 --2- 172.21.15.31:0/858521066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc068e80 0x7f18dc102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 --2- 172.21.15.31:0/858521066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc103410 0x7f18dc105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 -- 172.21.15.31:0/858521066 >> 172.21.15.31:0/858521066 conn(0x7f18dc0fa840 msgr2=0x7f18dc0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 -- 172.21.15.31:0/858521066 shutdown_connections 2023-07-29T16:26:15.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.286+0000 7f18e37ff700 1 -- 172.21.15.31:0/858521066 wait complete. 2023-07-29T16:26:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.287+0000 7f18e37ff700 1 Processor -- start 2023-07-29T16:26:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.287+0000 7f18e37ff700 1 -- start start 2023-07-29T16:26:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e37ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc068e80 0x7f18dc1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e37ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 0x7f18dc1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e37ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc105f20 0x7f18dc199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e37ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc199ec0 con 0x7f18dc105f20 2023-07-29T16:26:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e37ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc19a030 con 0x7f18dc103410 2023-07-29T16:26:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e37ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc19a1f0 con 0x7f18dc068e80 2023-07-29T16:26:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 0x7f18dc1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc068e80 0x7f18dc1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 0x7f18dc1989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42810/0 (socket says 172.21.15.31:42810) 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e0d9a700 1 -- 172.21.15.31:0/2426871633 learned_addr learned my addr 172.21.15.31:0/2426871633 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.288+0000 7f18e159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc068e80 0x7f18dc1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52984/0 (socket says 172.21.15.31:52984) 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc105f20 0x7f18dc199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e0d9a700 1 -- 172.21.15.31:0/2426871633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc068e80 msgr2=0x7f18dc1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc068e80 0x7f18dc1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e0d9a700 1 -- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc105f20 msgr2=0x7f18dc199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc105f20 0x7f18dc199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e0d9a700 1 -- 172.21.15.31:0/2426871633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d80095f0 con 0x7f18dc103410 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc105f20 0x7f18dc199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 0x7f18dc1989e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00e9a0 tx=0x7f18cc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18d27fc700 1 -- 172.21.15.31:0/2426871633 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc009790 con 0x7f18dc103410 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18d27fc700 1 -- 172.21.15.31:0/2426871633 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18cc00cbd0 con 0x7f18dc103410 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.289+0000 7f18d27fc700 1 -- 172.21.15.31:0/2426871633 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc018680 con 0x7f18dc103410 2023-07-29T16:26:15.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.290+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d4005e70 con 0x7f18dc103410 2023-07-29T16:26:15.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.290+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc1a0ab0 con 0x7f18dc103410 2023-07-29T16:26:15.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.293+0000 7f18d27fc700 1 -- 172.21.15.31:0/2426871633 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f18cc00cd40 con 0x7f18dc103410 2023-07-29T16:26:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.296+0000 7f18d27fc700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18c80a0790 0x7f18c80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.296+0000 7f18f7023b80 1 Processor -- start 2023-07-29T16:26:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.296+0000 7f18d27fc700 1 -- 172.21.15.31:0/2426871633 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18cc014070 con 0x7f18dc103410 2023-07-29T16:26:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.296+0000 7f18f7023b80 1 -- start start 2023-07-29T16:26:15.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be537b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be54160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18f7023b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7bdaa9b0 con 0x55fb7bcae6b0 2023-07-29T16:26:15.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18f7023b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7bda8370 con 0x55fb7be52df0 2023-07-29T16:26:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18f7023b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7bda3c60 con 0x55fb7be53cf0 2023-07-29T16:26:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18e159b700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18c80a0790 0x7f18c80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18e159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18e1d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be54160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18e159b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be528b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42168/0 (socket says 172.21.15.31:42168) 2023-07-29T16:26:15.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18e159b700 1 -- 172.21.15.31:0/4252030416 learned_addr learned my addr 172.21.15.31:0/4252030416 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.297+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 -- 172.21.15.31:0/4252030416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 msgr2=0x55fb7be54160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be54160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 -- 172.21.15.31:0/4252030416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 msgr2=0x55fb7be537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 -- 172.21.15.31:0/4252030416 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb7bbd1d80 con 0x55fb7bcae6b0 2023-07-29T16:26:15.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be528b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f18d8011de0 tx=0x7f18d800f6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a9fc510d4117a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.298+0000 7f18e159b700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18c80a0790 0x7f18c80a2c40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f18d800f9f0 tx=0x7f18d8018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.299+0000 7f18c77fe700 1 -- 172.21.15.31:0/4252030416 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8009ad0 con 0x55fb7bcae6b0 2023-07-29T16:26:15.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.299+0000 7f18c77fe700 1 -- 172.21.15.31:0/4252030416 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18d800eb90 con 0x55fb7bcae6b0 2023-07-29T16:26:15.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.299+0000 7f18c77fe700 1 -- 172.21.15.31:0/4252030416 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d801fa30 con 0x55fb7bcae6b0 2023-07-29T16:26:15.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.299+0000 7f18f7023b80 1 -- 172.21.15.31:0/4252030416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 msgr2=0x55fb7be528b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.299+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be528b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f18d8011de0 tx=0x7f18d800f6a0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 -- 172.21.15.31:0/4252030416 shutdown_connections 2023-07-29T16:26:15.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be54160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be528b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4252030416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 -- 172.21.15.31:0/4252030416 >> 172.21.15.31:0/4252030416 conn(0x55fb7be47fd0 msgr2=0x55fb7be4d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 -- 172.21.15.31:0/4252030416 shutdown_connections 2023-07-29T16:26:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 -- 172.21.15.31:0/4252030416 wait complete. 2023-07-29T16:26:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.300+0000 7f18f7023b80 1 -- start start 2023-07-29T16:26:15.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 Processor -- start 2023-07-29T16:26:15.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be5bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be5c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be63470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7bd404b0 con 0x55fb7be53cf0 2023-07-29T16:26:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7bd40150 con 0x55fb7be52df0 2023-07-29T16:26:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7bc278c0 con 0x55fb7bcae6b0 2023-07-29T16:26:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.301+0000 7f18f7023b80 10 client.0 _ll_register_callbacks cb 0x55fb7bd495c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.303+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be5c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.303+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be5c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42822/0 (socket says 172.21.15.31:42822) 2023-07-29T16:26:15.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.303+0000 7f18e0d9a700 1 -- 172.21.15.31:0/2476269027 learned_addr learned my addr 172.21.15.31:0/2476269027 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.303+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be63470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.303+0000 7f18e159b700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be5bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.304+0000 7f18e1d9c700 1 -- 172.21.15.31:0/2476269027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 msgr2=0x55fb7be5bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.304+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be5bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.304+0000 7f18e1d9c700 1 -- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 msgr2=0x55fb7be5c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.304+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be5c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.304+0000 7f18e1d9c700 1 -- 172.21.15.31:0/2476269027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d8019040 con 0x55fb7be53cf0 2023-07-29T16:26:15.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.304+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be63470 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55fb7be52610 tx=0x7f18d4009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed173572e9131193 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.305+0000 7f18c4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.307+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2476269027 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d400e040 con 0x55fb7be53cf0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.307+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2476269027 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18d402a070 con 0x55fb7be53cf0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.307+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb7bdcecb0 con 0x55fb7be53cf0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.308+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2476269027 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18d4007e30 con 0x55fb7be53cf0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.308+0000 7f18f7023b80 10 client.18588 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.308+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fb7bbbac00 con 0x55fb7be53cf0 2023-07-29T16:26:15.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.309+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2476269027 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fb7bdcecb0 con 0x55fb7be53cf0 2023-07-29T16:26:15.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.309+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2476269027 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f18d400b050 con 0x55fb7be53cf0 2023-07-29T16:26:15.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.309+0000 7f18f7023b80 10 client.18588 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:15.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.309+0000 7f18f7023b80 10 client.18588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:15.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.310+0000 7f18f7023b80 20 client.18588 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:15.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.310+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7be67c80 0x55fb7be6a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.310+0000 7f18f7023b80 4 client.18588 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:15.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.310+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb7be21d10 con 0x55fb7be67c80 2023-07-29T16:26:15.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.311+0000 7f18e159b700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7be67c80 0x55fb7be6a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.311+0000 7f18c4ff9700 10 client.18588 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.312+0000 7f18e159b700 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7be67c80 0x55fb7be6a090 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9014b77a40103a6 server_cookie=388fe931fdf14c5 in_seq=0 out_seq=0 2023-07-29T16:26:15.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.313+0000 7f18c4ff9700 1 -- 172.21.15.31:0/2476269027 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55fb7be21d10 con 0x55fb7be67c80 2023-07-29T16:26:15.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.313+0000 7f18c4ff9700 10 client.18588 handle_command_reply: tid=0 2023-07-29T16:26:15.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.313+0000 7f18f7023b80 1 client.18588 shutdown 2023-07-29T16:26:15.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.314+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 msgr2=0x55fb7be63470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.314+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be63470 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55fb7be52610 tx=0x7f18d4009420 comp rx=0 tx=0).stop 2023-07-29T16:26:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 shutdown_connections 2023-07-29T16:26:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7be67c80 0x55fb7be6a090 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be63470 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be5bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2476269027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be5c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 >> 172.21.15.31:0/2476269027 conn(0x55fb7be47fd0 msgr2=0x55fb7be4ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.316+0000 7f18c4ff9700 0 client.18588 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.317+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 shutdown_connections 2023-07-29T16:26:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.317+0000 7f18f7023b80 1 -- 172.21.15.31:0/2476269027 wait complete. 2023-07-29T16:26:15.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.317+0000 7f18f7023b80 20 client.18588 trim_cache size 0 max 16384 2023-07-29T16:26:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.338+0000 7f18f7023b80 1 Processor -- start 2023-07-29T16:26:15.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.338+0000 7f18f7023b80 1 -- start start 2023-07-29T16:26:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be56ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be571e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18f7023b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7be5d110 con 0x55fb7be53cf0 2023-07-29T16:26:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18f7023b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7be5b3d0 con 0x55fb7be52df0 2023-07-29T16:26:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18f7023b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7be5b1c0 con 0x55fb7bcae6b0 2023-07-29T16:26:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e159b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be56ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42834/0 (socket says 172.21.15.31:42834) 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e0d9a700 1 -- 172.21.15.31:0/2159957449 learned_addr learned my addr 172.21.15.31:0/2159957449 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e159b700 1 -- 172.21.15.31:0/2159957449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 msgr2=0x55fb7be63430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e159b700 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e159b700 1 -- 172.21.15.31:0/2159957449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 msgr2=0x55fb7be571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e159b700 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be571e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.339+0000 7f18e159b700 1 -- 172.21.15.31:0/2159957449 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb7bbf81f0 con 0x55fb7bcae6b0 2023-07-29T16:26:15.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.340+0000 7f18e159b700 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be56ca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f18d8010010 tx=0x7f18d8005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0b4c1a5735a1aba server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.340+0000 7f18c7fff700 1 -- 172.21.15.31:0/2159957449 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8009310 con 0x55fb7bcae6b0 2023-07-29T16:26:15.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.340+0000 7f18c7fff700 1 -- 172.21.15.31:0/2159957449 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18d802f330 con 0x55fb7bcae6b0 2023-07-29T16:26:15.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.340+0000 7f18c7fff700 1 -- 172.21.15.31:0/2159957449 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8005670 con 0x55fb7bcae6b0 2023-07-29T16:26:15.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 -- 172.21.15.31:0/2159957449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 msgr2=0x55fb7be56ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be56ca0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f18d8010010 tx=0x7f18d8005a90 comp rx=0 tx=0).stop 2023-07-29T16:26:15.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 -- 172.21.15.31:0/2159957449 shutdown_connections 2023-07-29T16:26:15.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7be53cf0 0x55fb7be571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7bcae6b0 0x55fb7be56ca0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 --2- 172.21.15.31:0/2159957449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 -- 172.21.15.31:0/2159957449 >> 172.21.15.31:0/2159957449 conn(0x55fb7be47fd0 msgr2=0x55fb7be5a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 -- 172.21.15.31:0/2159957449 shutdown_connections 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.341+0000 7f18f7023b80 1 -- 172.21.15.31:0/2159957449 wait complete. 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 -- start start 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 Processor -- start 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be62d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be5f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7be5fec0 con 0x55fb7bcae6b0 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7be60030 con 0x55fb7be52df0 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb7be601f0 con 0x55fb7be53cf0 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18f7023b80 10 client.0 _ll_register_callbacks cb 0x55fb7bd495c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.342+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e0d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42846/0 (socket says 172.21.15.31:42846) 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e0d9a700 1 -- 172.21.15.31:0/4010529884 learned_addr learned my addr 172.21.15.31:0/4010529884 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e1d9c700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be5f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be62d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 -- 172.21.15.31:0/4010529884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 msgr2=0x55fb7be5f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be5f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 -- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 msgr2=0x55fb7be63280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 -- 172.21.15.31:0/4010529884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb7bbd1d80 con 0x55fb7bcae6b0 2023-07-29T16:26:15.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18e159b700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be62d40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f18d80052c0 tx=0x7f18d800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3475b883679c1fc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.343+0000 7f18c4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:26:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.344+0000 7f18c4ff9700 1 -- 172.21.15.31:0/4010529884 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d803d070 con 0x55fb7bcae6b0 2023-07-29T16:26:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.344+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18d800eb90 con 0x55fb7bcae6b0 2023-07-29T16:26:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.344+0000 7f18c4ff9700 1 -- 172.21.15.31:0/4010529884 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18d802d070 con 0x55fb7bcae6b0 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.344+0000 7f18c4ff9700 1 -- 172.21.15.31:0/4010529884 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18d8009310 con 0x55fb7bcae6b0 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.344+0000 7f18f7023b80 10 client.18594 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.344+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fb7bbd0250 con 0x55fb7bcae6b0 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18c4ff9700 1 -- 172.21.15.31:0/4010529884 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18d8047020 con 0x55fb7bcae6b0 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18c4ff9700 1 -- 172.21.15.31:0/4010529884 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f18d8040030 con 0x55fb7bcae6b0 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18f7023b80 10 client.18594 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18f7023b80 10 client.18594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18f7023b80 20 client.18594 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7beba980 0x55fb7bebcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18f7023b80 4 client.18594 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:26:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fb7bd9cce0 con 0x55fb7beba980 2023-07-29T16:26:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.345+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7beba980 0x55fb7bebcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.346+0000 7f18e0d9a700 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7beba980 0x55fb7bebcd60 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6cce09e0e2d868d server_cookie=984736937585a8e1 in_seq=0 out_seq=0 2023-07-29T16:26:15.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.346+0000 7f18c4ff9700 10 client.18594 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.413+0000 7f18c4ff9700 1 -- 172.21.15.31:0/4010529884 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fb7bd9cce0 con 0x55fb7beba980 2023-07-29T16:26:15.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.413+0000 7f18c4ff9700 10 client.18594 handle_command_reply: tid=0 2023-07-29T16:26:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.413+0000 7f18f7023b80 1 client.18594 shutdown 2023-07-29T16:26:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.413+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 msgr2=0x55fb7be62d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.413+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be62d40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f18d80052c0 tx=0x7f18d800fab0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 shutdown_connections 2023-07-29T16:26:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb7beba980 0x55fb7bebcd60 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb7be53cf0 0x55fb7be5f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb7bcae6b0 0x55fb7be62d40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 --2- 172.21.15.31:0/4010529884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb7be52df0 0x55fb7be63280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 >> 172.21.15.31:0/4010529884 conn(0x55fb7be47fd0 msgr2=0x55fb7be56ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18c4ff9700 0 client.18594 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 shutdown_connections 2023-07-29T16:26:15.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 1 -- 172.21.15.31:0/4010529884 wait complete. 2023-07-29T16:26:15.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.414+0000 7f18f7023b80 20 client.18594 trim_cache size 0 max 16384 2023-07-29T16:26:15.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:26:15.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:26:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:26:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:26:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18c80a0790 msgr2=0x7f18c80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18c80a0790 0x7f18c80a2c40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f18d800f9f0 tx=0x7f18d8018000 comp rx=0 tx=0).stop 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 msgr2=0x7f18dc1989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 0x7f18dc1989e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00e9a0 tx=0x7f18cc00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 shutdown_connections 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18c80a0790 0x7f18c80a2c40 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f18dc105f20 0x7f18dc199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18dc068e80 0x7f18dc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 --2- 172.21.15.31:0/2426871633 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18dc103410 0x7f18dc1989e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 >> 172.21.15.31:0/2426871633 conn(0x7f18dc0fa840 msgr2=0x7f18dc0fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 shutdown_connections 2023-07-29T16:26:15.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:15.415+0000 7f18e37ff700 1 -- 172.21.15.31:0/2426871633 wait complete. 2023-07-29T16:26:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 -- 172.21.15.31:0/3110102291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 msgr2=0x7f3550103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 --2- 172.21.15.31:0/3110102291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f3550103b90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f3544009990 tx=0x7f3544009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 -- 172.21.15.31:0/3110102291 shutdown_connections 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 --2- 172.21.15.31:0/3110102291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35501040d0 0x7f355010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 --2- 172.21.15.31:0/3110102291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3550102550 0x7f3550102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 --2- 172.21.15.31:0/3110102291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f3550103b90 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 -- 172.21.15.31:0/3110102291 >> 172.21.15.31:0/3110102291 conn(0x7f35500fdb90 msgr2=0x7f35500fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 -- 172.21.15.31:0/3110102291 shutdown_connections 2023-07-29T16:26:15.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.491+0000 7f35557ff700 1 -- 172.21.15.31:0/3110102291 wait complete. 2023-07-29T16:26:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 Processor -- start 2023-07-29T16:26:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 -- start start 2023-07-29T16:26:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3550102550 0x7f35501982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f35501987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35501040d0 0x7f355019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355019fe30 con 0x7f35501040d0 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355019ff70 con 0x7f3550103740 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f35557ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35501a0100 con 0x7f3550102550 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f35501987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.493+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f35501987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42876/0 (socket says 172.21.15.31:42876) 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 -- 172.21.15.31:0/3351966072 learned_addr learned my addr 172.21.15.31:0/3351966072 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354effd700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3550102550 0x7f35501982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354f7fe700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35501040d0 0x7f355019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 -- 172.21.15.31:0/3351966072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3550102550 msgr2=0x7f35501982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3550102550 0x7f35501982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 -- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35501040d0 msgr2=0x7f355019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35501040d0 0x7f355019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 -- 172.21.15.31:0/3351966072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35440095f0 con 0x7f3550103740 2023-07-29T16:26:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f354e7fc700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f35501987f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f354400b620 tx=0x7f3544005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f3537fff700 1 -- 172.21.15.31:0/3351966072 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354401d070 con 0x7f3550103740 2023-07-29T16:26:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.494+0000 7f3537fff700 1 -- 172.21.15.31:0/3351966072 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3544022470 con 0x7f3550103740 2023-07-29T16:26:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.495+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35501a0380 con 0x7f3550103740 2023-07-29T16:26:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.495+0000 7f3537fff700 1 -- 172.21.15.31:0/3351966072 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544017650 con 0x7f3550103740 2023-07-29T16:26:15.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.495+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35501a0800 con 0x7f3550103740 2023-07-29T16:26:15.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.499+0000 7f3537fff700 1 -- 172.21.15.31:0/3351966072 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f35440225e0 con 0x7f3550103740 2023-07-29T16:26:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.500+0000 7f3569047b80 1 Processor -- start 2023-07-29T16:26:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.500+0000 7f3569047b80 1 -- start start 2023-07-29T16:26:15.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3569047b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e0be0 0x56329d2e0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3569047b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3569047b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e22f0 0x56329d2e2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3569047b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d238a10 con 0x56329d2e22f0 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3569047b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2363d0 con 0x56329d2e0be0 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3569047b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d231cc0 con 0x56329d2e1530 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3537fff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35380a0810 0x7f35380a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f354effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f3537fff700 1 -- 172.21.15.31:0/3351966072 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f35440cce80 con 0x7f3550103740 2023-07-29T16:26:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f354effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2e1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53030/0 (socket says 172.21.15.31:53030) 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f354effd700 1 -- 172.21.15.31:0/2278860900 learned_addr learned my addr 172.21.15.31:0/2278860900 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f354f7fe700 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e0be0 0x56329d2e0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f354e7fc700 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e22f0 0x56329d2e2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.501+0000 7f354f7fe700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35380a0810 0x7f35380a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354effd700 1 -- 172.21.15.31:0/2278860900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e0be0 msgr2=0x56329d2e0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354effd700 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e0be0 0x56329d2e0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354effd700 1 -- 172.21.15.31:0/2278860900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e22f0 msgr2=0x56329d2e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354effd700 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e22f0 0x56329d2e2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354effd700 1 -- 172.21.15.31:0/2278860900 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56329d05fde0 con 0x56329d2e1530 2023-07-29T16:26:15.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354e7fc700 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e22f0 0x56329d2e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:15.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354f7fe700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35380a0810 0x7f35380a2cc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3548009ec0 tx=0x7f3548009e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f354effd700 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2e1d60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3540009d30 tx=0x7f3540011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31573617e3e4a0d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f35357fa700 1 -- 172.21.15.31:0/2278860900 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540004220 con 0x56329d2e1530 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f35357fa700 1 -- 172.21.15.31:0/2278860900 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3540004380 con 0x56329d2e1530 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.502+0000 7f35357fa700 1 -- 172.21.15.31:0/2278860900 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35400197a0 con 0x56329d2e1530 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 -- 172.21.15.31:0/2278860900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 msgr2=0x56329d2e1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2e1d60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3540009d30 tx=0x7f3540011720 comp rx=0 tx=0).stop 2023-07-29T16:26:15.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 -- 172.21.15.31:0/2278860900 shutdown_connections 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e22f0 0x56329d2e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e0be0 0x56329d2e0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 --2- 172.21.15.31:0/2278860900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2e1d60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 -- 172.21.15.31:0/2278860900 >> 172.21.15.31:0/2278860900 conn(0x56329d2d63e0 msgr2=0x56329d2db990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 -- 172.21.15.31:0/2278860900 shutdown_connections 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.503+0000 7f3569047b80 1 -- 172.21.15.31:0/2278860900 wait complete. 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.504+0000 7f3569047b80 1 -- start start 2023-07-29T16:26:15.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.504+0000 7f3569047b80 1 Processor -- start 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 0x56329d2ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e22f0 0x56329d2f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d1ce510 con 0x56329d2e0be0 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d1ce1b0 con 0x56329d2e1530 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d0b5920 con 0x56329d2e22f0 2023-07-29T16:26:15.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f3569047b80 10 client.0 _ll_register_callbacks cb 0x56329d1d7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 0x56329d2ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 0x56329d2ead40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42894/0 (socket says 172.21.15.31:42894) 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f354e7fc700 1 -- 172.21.15.31:0/152369886 learned_addr learned my addr 172.21.15.31:0/152369886 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f354f7fe700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e22f0 0x56329d2f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.505+0000 7f354effd700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f354e7fc700 1 -- 172.21.15.31:0/152369886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e22f0 msgr2=0x56329d2f1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f354e7fc700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e22f0 0x56329d2f1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f354e7fc700 1 -- 172.21.15.31:0/152369886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 msgr2=0x56329d2ea800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f354e7fc700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ea800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f354e7fc700 1 -- 172.21.15.31:0/152369886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35501a0800 con 0x56329d2e1530 2023-07-29T16:26:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f354e7fc700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 0x56329d2ead40 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3544000c00 tx=0x7f3544009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=796b47bcfeba386c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3532ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:15.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3532ff5700 1 -- 172.21.15.31:0/152369886 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354401bde0 con 0x56329d2e1530 2023-07-29T16:26:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3532ff5700 1 -- 172.21.15.31:0/152369886 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f354409a610 con 0x56329d2e1530 2023-07-29T16:26:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56329d25cd10 con 0x56329d2e1530 2023-07-29T16:26:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3532ff5700 1 -- 172.21.15.31:0/152369886 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3544099f70 con 0x56329d2e1530 2023-07-29T16:26:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3569047b80 10 client.28688 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.506+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f3540009960 con 0x56329d2e1530 2023-07-29T16:26:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3532ff5700 1 -- 172.21.15.31:0/152369886 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35440b8020 con 0x56329d2e1530 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3532ff5700 1 -- 172.21.15.31:0/152369886 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f3544099b00 con 0x56329d2e1530 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3569047b80 10 client.28688 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3569047b80 10 client.28688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3569047b80 20 client.28688 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3569047b80 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d2f6200 0x56329d2f8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3569047b80 4 client.28688 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56329d2afd70 con 0x56329d2f6200 2023-07-29T16:26:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.507+0000 7f354effd700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d2f6200 0x56329d2f8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.508+0000 7f354effd700 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d2f6200 0x56329d2f8610 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f14a700056d86b49 server_cookie=860ba65eb582c7db in_seq=0 out_seq=0 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.508+0000 7f3532ff5700 10 client.28688 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.508+0000 7f3532ff5700 1 -- 172.21.15.31:0/152369886 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56329d2afd70 con 0x56329d2f6200 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.508+0000 7f3532ff5700 10 client.28688 handle_command_reply: tid=0 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 client.28688 shutdown 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 msgr2=0x56329d2ead40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 0x56329d2ead40 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3544000c00 tx=0x7f3544009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 shutdown_connections 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d2f6200 0x56329d2f8610 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e22f0 0x56329d2f1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ea800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3532ff5700 0 client.28688 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 --2- 172.21.15.31:0/152369886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e1530 0x56329d2ead40 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 >> 172.21.15.31:0/152369886 conn(0x56329d2d63e0 msgr2=0x56329d2db060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 shutdown_connections 2023-07-29T16:26:15.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 1 -- 172.21.15.31:0/152369886 wait complete. 2023-07-29T16:26:15.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.509+0000 7f3569047b80 20 client.28688 trim_cache size 0 max 16384 2023-07-29T16:26:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.528+0000 7f3569047b80 1 Processor -- start 2023-07-29T16:26:15.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.528+0000 7f3569047b80 1 -- start start 2023-07-29T16:26:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.528+0000 7f3569047b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e0be0 0x56329d2e2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.528+0000 7f3569047b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e1530 0x56329d2efd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f3569047b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2e8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f3569047b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2ea370 con 0x56329d2e1530 2023-07-29T16:26:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f3569047b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2d8460 con 0x56329d2e22f0 2023-07-29T16:26:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f3569047b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2dd080 con 0x56329d2e0be0 2023-07-29T16:26:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e0be0 0x56329d2e2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e1530 0x56329d2efd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e0be0 0x56329d2e2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53052/0 (socket says 172.21.15.31:53052) 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354effd700 1 -- 172.21.15.31:0/2106641301 learned_addr learned my addr 172.21.15.31:0/2106641301 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354f7fe700 1 -- 172.21.15.31:0/2106641301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e0be0 msgr2=0x56329d2e2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354f7fe700 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e0be0 0x56329d2e2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354f7fe700 1 -- 172.21.15.31:0/2106641301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e1530 msgr2=0x56329d2efd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354f7fe700 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e1530 0x56329d2efd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.529+0000 7f354f7fe700 1 -- 172.21.15.31:0/2106641301 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56329d048c60 con 0x56329d2e22f0 2023-07-29T16:26:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.530+0000 7f354f7fe700 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2e8640 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x56329d2e0980 tx=0x7f354801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3192200c1aa41d9d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.530+0000 7f3535ffb700 1 -- 172.21.15.31:0/2106641301 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354802b070 con 0x56329d2e22f0 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.530+0000 7f3535ffb700 1 -- 172.21.15.31:0/2106641301 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f354800f040 con 0x56329d2e22f0 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.530+0000 7f3535ffb700 1 -- 172.21.15.31:0/2106641301 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354800e070 con 0x56329d2e22f0 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.530+0000 7f3569047b80 1 -- 172.21.15.31:0/2106641301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 msgr2=0x56329d2e8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.530+0000 7f3569047b80 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2e8640 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x56329d2e0980 tx=0x7f354801d000 comp rx=0 tx=0).stop 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 -- 172.21.15.31:0/2106641301 shutdown_connections 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2e8640 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e0be0 0x56329d2e2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 --2- 172.21.15.31:0/2106641301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e1530 0x56329d2efd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 -- 172.21.15.31:0/2106641301 >> 172.21.15.31:0/2106641301 conn(0x56329d2d63e0 msgr2=0x56329d2e9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 -- 172.21.15.31:0/2106641301 shutdown_connections 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 -- 172.21.15.31:0/2106641301 wait complete. 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 -- start start 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.531+0000 7f3569047b80 1 Processor -- start 2023-07-29T16:26:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ef890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2edea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2ee3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2f00b0 con 0x56329d2e0be0 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2ee920 con 0x56329d2e22f0 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56329d2eeae0 con 0x56329d2e1530 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f3569047b80 10 client.0 _ll_register_callbacks cb 0x56329d1d7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2edea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2ee3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354e7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2edea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53064/0 (socket says 172.21.15.31:53064) 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354e7fc700 1 -- 172.21.15.31:0/3969636680 learned_addr learned my addr 172.21.15.31:0/3969636680 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354effd700 1 -- 172.21.15.31:0/3969636680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 msgr2=0x56329d2edea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354effd700 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2edea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354effd700 1 -- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 msgr2=0x56329d2ee3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354effd700 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2ee3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354effd700 1 -- 172.21.15.31:0/3969636680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3548008030 con 0x56329d2e0be0 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.532+0000 7f354f7fe700 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f354effd700 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ef890 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3540005950 tx=0x7f35400058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea6db3b1e6cedec server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3532ff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3532ff5700 1 -- 172.21.15.31:0/3969636680 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540029040 con 0x56329d2e0be0 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3532ff5700 1 -- 172.21.15.31:0/3969636680 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f35400041b0 con 0x56329d2e0be0 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3540004380 con 0x56329d2e0be0 2023-07-29T16:26:15.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3532ff5700 1 -- 172.21.15.31:0/3969636680 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f354002a420 con 0x56329d2e0be0 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3569047b80 10 client.18612 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.533+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56329d086250 con 0x56329d2e0be0 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3532ff5700 1 -- 172.21.15.31:0/3969636680 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3540004380 con 0x56329d2e0be0 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3532ff5700 1 -- 172.21.15.31:0/3969636680 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f3540011440 con 0x56329d2e0be0 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3569047b80 10 client.18612 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3569047b80 10 client.18612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3569047b80 20 client.18612 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:15.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3569047b80 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d349280 0x56329d34b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3569047b80 4 client.18612 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:26:15.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56329d22ad40 con 0x56329d349280 2023-07-29T16:26:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f354e7fc700 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d349280 0x56329d34b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f354e7fc700 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d349280 0x56329d34b690 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b81324f50190069 server_cookie=8900f78298f1ab4c in_seq=0 out_seq=0 2023-07-29T16:26:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.534+0000 7f3532ff5700 10 client.18612 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3532ff5700 1 -- 172.21.15.31:0/3969636680 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56329d22ad40 con 0x56329d349280 2023-07-29T16:26:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3532ff5700 10 client.18612 handle_command_reply: tid=0 2023-07-29T16:26:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 client.18612 shutdown 2023-07-29T16:26:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 msgr2=0x56329d2ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ef890 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3540005950 tx=0x7f35400058e0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 shutdown_connections 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56329d349280 0x56329d34b690 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56329d2e22f0 0x56329d2ee3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56329d2e0be0 0x56329d2ef890 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 --2- 172.21.15.31:0/3969636680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56329d2e1530 0x56329d2edea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3532ff5700 0 client.18612 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:15.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 >> 172.21.15.31:0/3969636680 conn(0x56329d2d63e0 msgr2=0x56329d2e8640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 shutdown_connections 2023-07-29T16:26:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 1 -- 172.21.15.31:0/3969636680 wait complete. 2023-07-29T16:26:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.535+0000 7f3569047b80 20 client.18612 trim_cache size 0 max 16384 2023-07-29T16:26:15.544 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:26:15.545 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:26:15.545 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:26:15.545 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:26:15.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35380a0810 msgr2=0x7f35380a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35380a0810 0x7f35380a2cc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3548009ec0 tx=0x7f3548009e50 comp rx=0 tx=0).stop 2023-07-29T16:26:15.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 msgr2=0x7f35501987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f35501987f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f354400b620 tx=0x7f3544005740 comp rx=0 tx=0).stop 2023-07-29T16:26:15.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 shutdown_connections 2023-07-29T16:26:15.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35380a0810 0x7f35380a2cc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35501040d0 0x7f355019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3550102550 0x7f35501982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 --2- 172.21.15.31:0/3351966072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3550103740 0x7f35501987f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 >> 172.21.15.31:0/3351966072 conn(0x7f35500fdb90 msgr2=0x7f35500ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 shutdown_connections 2023-07-29T16:26:15.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.536+0000 7f35557ff700 1 -- 172.21.15.31:0/3351966072 wait complete. 2023-07-29T16:26:15.556 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:26:15.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:26:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 -- 172.21.15.31:0/3149198205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 msgr2=0x7ff19c103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/3149198205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c103170 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff198009990 tx=0x7ff198009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 -- 172.21.15.31:0/3149198205 shutdown_connections 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/3149198205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c105fd0 0x7ff19c108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/3149198205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c103170 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/3149198205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 0x7ff19c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 -- 172.21.15.31:0/3149198205 >> 172.21.15.31:0/3149198205 conn(0x7ff19c0fa810 msgr2=0x7ff19c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 -- 172.21.15.31:0/3149198205 shutdown_connections 2023-07-29T16:26:15.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.777+0000 7ff1a35ff700 1 -- 172.21.15.31:0/3149198205 wait complete. 2023-07-29T16:26:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.779+0000 7ff1a35ff700 1 Processor -- start 2023-07-29T16:26:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.779+0000 7ff1a35ff700 1 -- start start 2023-07-29T16:26:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.779+0000 7ff1a35ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.779+0000 7ff1a35ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 0x7ff19c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.779+0000 7ff1a35ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c105fd0 0x7ff19c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a35ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c19fd80 con 0x7ff19c1036b0 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a35ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c19fec0 con 0x7ff19c100d90 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a35ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c1a0050 con 0x7ff19c105fd0 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42934/0 (socket says 172.21.15.31:42934) 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 -- 172.21.15.31:0/23548168 learned_addr learned my addr 172.21.15.31:0/23548168 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a1b9c700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c105fd0 0x7ff19c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a0b9a700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 0x7ff19c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 -- 172.21.15.31:0/23548168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c105fd0 msgr2=0x7ff19c19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c105fd0 0x7ff19c19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 -- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 msgr2=0x7ff19c198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 0x7ff19c198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 -- 172.21.15.31:0/23548168 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1980095f0 con 0x7ff19c100d90 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a0b9a700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 0x7ff19c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.780+0000 7ff1a139b700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c198200 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b680 tx=0x7ff198005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.781+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19801d070 con 0x7ff19c100d90 2023-07-29T16:26:15.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.781+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff198022470 con 0x7ff19c100d90 2023-07-29T16:26:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.781+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198017650 con 0x7ff19c100d90 2023-07-29T16:26:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.781+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c1a02d0 con 0x7ff19c100d90 2023-07-29T16:26:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.781+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c1a06a0 con 0x7ff19c100d90 2023-07-29T16:26:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.782+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c107b40 con 0x7ff19c100d90 2023-07-29T16:26:15.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.784+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff1980225e0 con 0x7ff19c100d90 2023-07-29T16:26:15.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.786+0000 7ff1927fc700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1880a07e0 0x7ff1880a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.786+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff1980ccff0 con 0x7ff19c100d90 2023-07-29T16:26:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.786+0000 7ff1a0b9a700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1880a07e0 0x7ff1880a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:15.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.787+0000 7ff1a0b9a700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1880a07e0 0x7ff1880a2c90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009cb0 tx=0x7ff18c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:15.789+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7ff198096a60 con 0x7ff19c100d90 2023-07-29T16:26:15.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 795708 8.91 MB/sec execute 360 sec latency 77.841 ms 2023-07-29T16:26:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.068+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff19c02cc30 con 0x7ff19c100d90 2023-07-29T16:26:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.069+0000 7ff1927fc700 1 -- 172.21.15.31:0/23548168 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10781 (secure 0 0 0) 0x7ff198027090 con 0x7ff19c100d90 2023-07-29T16:26:16.070 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:26:16.070 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:24:56.527016+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:26:16.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1880a07e0 msgr2=0x7ff1880a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1880a07e0 0x7ff1880a2c90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009cb0 tx=0x7ff18c009260 comp rx=0 tx=0).stop 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 msgr2=0x7ff19c198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c198200 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b680 tx=0x7ff198005740 comp rx=0 tx=0).stop 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 shutdown_connections 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1880a07e0 0x7ff1880a2c90 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff19c105fd0 0x7ff19c19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff19c100d90 0x7ff19c198200 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 --2- 172.21.15.31:0/23548168 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff19c1036b0 0x7ff19c198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 >> 172.21.15.31:0/23548168 conn(0x7ff19c0fa810 msgr2=0x7ff19c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 shutdown_connections 2023-07-29T16:26:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.072+0000 7ff1a35ff700 1 -- 172.21.15.31:0/23548168 wait complete. 2023-07-29T16:26:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 31 2023-07-29T16:26:16.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:26:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 -- 172.21.15.31:0/2894971669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 msgr2=0x7f6dcc108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/2894971669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc108760 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009f90 tx=0x7f6dc400b330 comp rx=0 tx=0).stop 2023-07-29T16:26:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 -- 172.21.15.31:0/2894971669 shutdown_connections 2023-07-29T16:26:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/2894971669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc108760 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/2894971669 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc100320 0x7f6dcc100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/2894971669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc101510 0x7f6dcc101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 -- 172.21.15.31:0/2894971669 >> 172.21.15.31:0/2894971669 conn(0x7f6dcc0fb920 msgr2=0x7f6dcc0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 -- 172.21.15.31:0/2894971669 shutdown_connections 2023-07-29T16:26:16.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.325+0000 7f6dd23ff700 1 -- 172.21.15.31:0/2894971669 wait complete. 2023-07-29T16:26:16.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 Processor -- start 2023-07-29T16:26:16.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 -- start start 2023-07-29T16:26:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 0x7f6dcc196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101510 0x7f6dcc1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc19dd10 con 0x7f6dcc101510 2023-07-29T16:26:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc19de50 con 0x7f6dcc101ea0 2023-07-29T16:26:16.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.326+0000 7f6dd23ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc19dfe0 con 0x7f6dcc100320 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 0x7f6dcc196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101510 0x7f6dcc1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dd099c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 0x7f6dcc196190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53110/0 (socket says 172.21.15.31:53110) 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 -- 172.21.15.31:0/326319586 learned_addr learned my addr 172.21.15.31:0/326319586 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 -- 172.21.15.31:0/326319586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 msgr2=0x7f6dcc19d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc19d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 -- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101510 msgr2=0x7f6dcc1966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101510 0x7f6dcc1966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 -- 172.21.15.31:0/326319586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc4009c40 con 0x7f6dcc100320 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dd099c700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc19d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.327+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 0x7f6dcc196190 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00e9f0 tx=0x7f6dbc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.328+0000 7f6dc97fa700 1 -- 172.21.15.31:0/326319586 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc004220 con 0x7f6dcc100320 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.328+0000 7f6dc97fa700 1 -- 172.21.15.31:0/326319586 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6dbc004380 con 0x7f6dcc100320 2023-07-29T16:26:16.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.328+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc19e2c0 con 0x7f6dcc100320 2023-07-29T16:26:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.328+0000 7f6dc97fa700 1 -- 172.21.15.31:0/326319586 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc003be0 con 0x7f6dcc100320 2023-07-29T16:26:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.328+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc19e7d0 con 0x7f6dcc100320 2023-07-29T16:26:16.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.332+0000 7f6dc97fa700 1 -- 172.21.15.31:0/326319586 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6dbc003d40 con 0x7f6dcc100320 2023-07-29T16:26:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 Processor -- start 2023-07-29T16:26:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 -- start start 2023-07-29T16:26:16.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898dbe0 0x560f2898dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f2898ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898f2f0 0x560f2898f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f288e5a10 con 0x560f2898dbe0 2023-07-29T16:26:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f288e33d0 con 0x560f2898f2f0 2023-07-29T16:26:16.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f288decc0 con 0x560f2898e530 2023-07-29T16:26:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.333+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f2898ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dd099c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898dbe0 0x560f2898dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f2898ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53122/0 (socket says 172.21.15.31:53122) 2023-07-29T16:26:16.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcbfff700 1 -- 172.21.15.31:0/3243694071 learned_addr learned my addr 172.21.15.31:0/3243694071 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898f2f0 0x560f2898f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dc97fa700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6db40a0790 0x7f6db40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 -- 172.21.15.31:0/3243694071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 msgr2=0x560f2898ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f2898ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 -- 172.21.15.31:0/3243694071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898dbe0 msgr2=0x560f2898dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898dbe0 0x560f2898dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 -- 172.21.15.31:0/3243694071 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f2870cde0 con 0x560f2898f2f0 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dc97fa700 1 -- 172.21.15.31:0/326319586 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6dbc014070 con 0x7f6dcc100320 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dd099c700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6db40a0790 0x7f6db40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898f2f0 0x560f2898f740 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009cb0 tx=0x7f6dc0011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9622ff7fcedec08c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.334+0000 7f6db27fc700 1 -- 172.21.15.31:0/3243694071 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0011d20 con 0x560f2898f2f0 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6db27fc700 1 -- 172.21.15.31:0/3243694071 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6dc0011e80 con 0x560f2898f2f0 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6db27fc700 1 -- 172.21.15.31:0/3243694071 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0005280 con 0x560f2898f2f0 2023-07-29T16:26:16.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 -- 172.21.15.31:0/3243694071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898f2f0 msgr2=0x560f2898f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898f2f0 0x560f2898f740 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6dc0009cb0 tx=0x7f6dc0011440 comp rx=0 tx=0).stop 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6dd099c700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6db40a0790 0x7f6db40a2c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x560f2898d980 tx=0x7f6dc4009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 -- 172.21.15.31:0/3243694071 shutdown_connections 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898f2f0 0x560f2898f740 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898dbe0 0x560f2898dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/3243694071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f2898ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.335+0000 7f6de5d29b80 1 -- 172.21.15.31:0/3243694071 >> 172.21.15.31:0/3243694071 conn(0x560f289833e0 msgr2=0x560f28988990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.336+0000 7f6de5d29b80 1 -- 172.21.15.31:0/3243694071 shutdown_connections 2023-07-29T16:26:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.336+0000 7f6de5d29b80 1 -- 172.21.15.31:0/3243694071 wait complete. 2023-07-29T16:26:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.336+0000 7f6de5d29b80 1 -- start start 2023-07-29T16:26:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.336+0000 7f6de5d29b80 1 Processor -- start 2023-07-29T16:26:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289977f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f28997d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898f2f0 0x560f2899ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2887b510 con 0x560f2898f2f0 2023-07-29T16:26:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2887b1b0 con 0x560f2898dbe0 2023-07-29T16:26:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f28762920 con 0x560f2898e530 2023-07-29T16:26:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6de5d29b80 10 client.0 _ll_register_callbacks cb 0x560f28884620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f28997d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f28997d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53126/0 (socket says 172.21.15.31:53126) 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcb7fe700 1 -- 172.21.15.31:0/2906099474 learned_addr learned my addr 172.21.15.31:0/2906099474 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dd099c700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898f2f0 0x560f2899ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcbfff700 1 -- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 msgr2=0x560f28997d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f28997d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcbfff700 1 -- 172.21.15.31:0/2906099474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898f2f0 msgr2=0x560f2899ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898f2f0 0x560f2899ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcbfff700 1 -- 172.21.15.31:0/2906099474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc00098e0 con 0x560f2898dbe0 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f28997d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.337+0000 7f6dd099c700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898f2f0 0x560f2899ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289977f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc095840 tx=0x7f6dbc099240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265db8410c613071 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6dab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6dab7fe700 1 -- 172.21.15.31:0/2906099474 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc0d4030 con 0x560f2898dbe0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6dab7fe700 1 -- 172.21.15.31:0/2906099474 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6dbc0044f0 con 0x560f2898dbe0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f2895cd70 con 0x560f2898dbe0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6dab7fe700 1 -- 172.21.15.31:0/2906099474 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dbc098ca0 con 0x560f2898dbe0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6de5d29b80 10 client.28712 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560f286f5c60 con 0x560f2898dbe0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.338+0000 7f6dab7fe700 1 -- 172.21.15.31:0/2906099474 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6dbc0b7020 con 0x560f2898dbe0 2023-07-29T16:26:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6dab7fe700 1 -- 172.21.15.31:0/2906099474 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6dbc0b1d20 con 0x560f2898dbe0 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6de5d29b80 10 client.28712 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6de5d29b80 10 client.28712 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6de5d29b80 20 client.28712 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289a35f0 0x560f289a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6de5d29b80 4 client.28712 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f288d7d40 con 0x560f289a35f0 2023-07-29T16:26:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.339+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289a35f0 0x560f289a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.340+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289a35f0 0x560f289a59d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d8e24bdc7e3f3232 server_cookie=cc7144700ec67a92 in_seq=0 out_seq=0 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.340+0000 7f6dab7fe700 10 client.28712 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.340+0000 7f6dab7fe700 1 -- 172.21.15.31:0/2906099474 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560f288d7d40 con 0x560f289a35f0 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.340+0000 7f6dab7fe700 10 client.28712 handle_command_reply: tid=0 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 client.28712 shutdown 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 msgr2=0x560f289977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289977f0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc095840 tx=0x7f6dbc099240 comp rx=0 tx=0).stop 2023-07-29T16:26:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 shutdown_connections 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898f2f0 0x560f2899ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289a35f0 0x560f289a59d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289977f0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6dab7fe700 0 client.28712 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/2906099474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898e530 0x560f28997d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 >> 172.21.15.31:0/2906099474 conn(0x560f289833e0 msgr2=0x560f28988060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 shutdown_connections 2023-07-29T16:26:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 1 -- 172.21.15.31:0/2906099474 wait complete. 2023-07-29T16:26:16.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.341+0000 7f6de5d29b80 20 client.28712 trim_cache size 0 max 16384 2023-07-29T16:26:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.356+0000 7f6de5d29b80 1 Processor -- start 2023-07-29T16:26:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.356+0000 7f6de5d29b80 1 -- start start 2023-07-29T16:26:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.356+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.356+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.356+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2898f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f28995400 con 0x560f2898e530 2023-07-29T16:26:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f289a3390 con 0x560f2898dbe0 2023-07-29T16:26:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f289aa330 con 0x560f2898f2f0 2023-07-29T16:26:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42976/0 (socket says 172.21.15.31:42976) 2023-07-29T16:26:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 -- 172.21.15.31:0/1930384035 learned_addr learned my addr 172.21.15.31:0/1930384035 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dd099c700 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2898f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 -- 172.21.15.31:0/1930384035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 msgr2=0x560f2898f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2898f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 -- 172.21.15.31:0/1930384035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 msgr2=0x560f2899ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 -- 172.21.15.31:0/1930384035 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f28733250 con 0x560f2898dbe0 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289a59d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc0c9a50 tx=0x7f6dbc00ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a12b5d19c756e4a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.357+0000 7f6db2ffd700 1 -- 172.21.15.31:0/1930384035 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc0d4070 con 0x560f2898dbe0 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6db2ffd700 1 -- 172.21.15.31:0/1930384035 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6dbc0b7040 con 0x560f2898dbe0 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6db2ffd700 1 -- 172.21.15.31:0/1930384035 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc098a90 con 0x560f2898dbe0 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 -- 172.21.15.31:0/1930384035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 msgr2=0x560f289a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289a59d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc0c9a50 tx=0x7f6dbc00ef80 comp rx=0 tx=0).stop 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 -- 172.21.15.31:0/1930384035 shutdown_connections 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2898f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f289a59d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/1930384035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 -- 172.21.15.31:0/1930384035 >> 172.21.15.31:0/1930384035 conn(0x560f289833e0 msgr2=0x560f28996510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 -- 172.21.15.31:0/1930384035 shutdown_connections 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.358+0000 7f6de5d29b80 1 -- 172.21.15.31:0/1930384035 wait complete. 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 -- start start 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 Processor -- start 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f2899a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2899d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2899dd20 con 0x560f2898e530 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2899de90 con 0x560f2898dbe0 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f2899e050 con 0x560f2898f2f0 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.359+0000 7f6de5d29b80 10 client.0 _ll_register_callbacks cb 0x560f28884620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f2899a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dd099c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2899d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f2899a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42988/0 (socket says 172.21.15.31:42988) 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 -- 172.21.15.31:0/986784483 learned_addr learned my addr 172.21.15.31:0/986784483 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 -- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 msgr2=0x560f2899d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2899d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 -- 172.21.15.31:0/986784483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 msgr2=0x560f2899a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 -- 172.21.15.31:0/986784483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f286f5c60 con 0x560f2898dbe0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dcbfff700 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f2899a370 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00cfd0 tx=0x7f6dbc098ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79459a61ac208af8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dab7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dab7fe700 1 -- 172.21.15.31:0/986784483 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc0d4030 con 0x560f2898dbe0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6dab7fe700 1 -- 172.21.15.31:0/986784483 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6dbc0044f0 con 0x560f2898dbe0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.360+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f28909d10 con 0x560f2898dbe0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6dab7fe700 1 -- 172.21.15.31:0/986784483 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dbc098ca0 con 0x560f2898dbe0 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 10 client.28718 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560f2870b2b0 con 0x560f2898dbe0 2023-07-29T16:26:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6dab7fe700 1 -- 172.21.15.31:0/986784483 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560f28909d10 con 0x560f2898dbe0 2023-07-29T16:26:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6dab7fe700 1 -- 172.21.15.31:0/986784483 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6dbc0b1d20 con 0x560f2898dbe0 2023-07-29T16:26:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 10 client.28718 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 10 client.28718 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:26:16.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 20 client.28718 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289f5200 0x560f289f75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 4 client.28718 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:26:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560f288d7a60 con 0x560f289f5200 2023-07-29T16:26:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.361+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289f5200 0x560f289f75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6dcb7fe700 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289f5200 0x560f289f75e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62f6d150e878f11 server_cookie=101ef48800149980 in_seq=0 out_seq=0 2023-07-29T16:26:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6dab7fe700 10 client.28718 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6dab7fe700 1 -- 172.21.15.31:0/986784483 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560f288d7a60 con 0x560f289f5200 2023-07-29T16:26:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6dab7fe700 10 client.28718 handle_command_reply: tid=0 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 client.28718 shutdown 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 msgr2=0x560f2899a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f2899a370 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00cfd0 tx=0x7f6dbc098ae0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 shutdown_connections 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560f2898f2f0 0x560f2899d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x560f289f5200 0x560f289f75e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560f2898dbe0 0x560f2899a370 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 --2- 172.21.15.31:0/986784483 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560f2898e530 0x560f2899a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6dab7fe700 0 client.28718 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.362+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 >> 172.21.15.31:0/986784483 conn(0x560f289833e0 msgr2=0x560f28989690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.363+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 shutdown_connections 2023-07-29T16:26:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.363+0000 7f6de5d29b80 1 -- 172.21.15.31:0/986784483 wait complete. 2023-07-29T16:26:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.363+0000 7f6de5d29b80 20 client.28718 trim_cache size 0 max 16384 2023-07-29T16:26:16.372 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:26:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6db40a0790 msgr2=0x7f6db40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6db40a0790 0x7f6db40a2c40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x560f2898d980 tx=0x7f6dc4009920 comp rx=0 tx=0).stop 2023-07-29T16:26:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 msgr2=0x7f6dcc196190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 0x7f6dcc196190 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00e9f0 tx=0x7f6dbc00ed00 comp rx=0 tx=0).stop 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 shutdown_connections 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6db40a0790 0x7f6db40a2c40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dcc101ea0 0x7f6dcc19d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6dcc100320 0x7f6dcc196190 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 --2- 172.21.15.31:0/326319586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6dcc101510 0x7f6dcc1966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 >> 172.21.15.31:0/326319586 conn(0x7f6dcc0fb920 msgr2=0x7f6dcc0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 shutdown_connections 2023-07-29T16:26:16.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.364+0000 7f6dd23ff700 1 -- 172.21.15.31:0/326319586 wait complete. 2023-07-29T16:26:16.383 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:26:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 -- 172.21.15.31:0/3730406864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0100390 msgr2=0x7f37c01007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 --2- 172.21.15.31:0/3730406864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0100390 0x7f37c01007a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009990 tx=0x7f37b0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 -- 172.21.15.31:0/3730406864 shutdown_connections 2023-07-29T16:26:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 --2- 172.21.15.31:0/3730406864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c0101f10 0x7f37c01087d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 --2- 172.21.15.31:0/3730406864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0100390 0x7f37c01007a0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 --2- 172.21.15.31:0/3730406864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0101580 0x7f37c01019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.603+0000 7f37c69ff700 1 -- 172.21.15.31:0/3730406864 >> 172.21.15.31:0/3730406864 conn(0x7f37c00fb9b0 msgr2=0x7f37c00fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.604+0000 7f37c69ff700 1 -- 172.21.15.31:0/3730406864 shutdown_connections 2023-07-29T16:26:16.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.604+0000 7f37c69ff700 1 -- 172.21.15.31:0/3730406864 wait complete. 2023-07-29T16:26:16.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 Processor -- start 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 -- start start 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0100390 0x7f37c01982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c0101580 0x7f37c0198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 0x7f37c019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c019fe40 con 0x7f37c0100390 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c019ff80 con 0x7f37c0101f10 2023-07-29T16:26:16.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37c69ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c01a0110 con 0x7f37c0101580 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.605+0000 7f37bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0100390 0x7f37c01982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 0x7f37c019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 0x7f37c019f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43008/0 (socket says 172.21.15.31:43008) 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37b7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c0101580 0x7f37c0198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0100390 0x7f37c01982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42354/0 (socket says 172.21.15.31:42354) 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 -- 172.21.15.31:0/695826868 learned_addr learned my addr 172.21.15.31:0/695826868 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 -- 172.21.15.31:0/695826868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c0101580 msgr2=0x7f37c0198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c0101580 0x7f37c0198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 -- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0100390 msgr2=0x7f37c01982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0100390 0x7f37c01982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 -- 172.21.15.31:0/695826868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b00095f0 con 0x7f37c0101f10 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 0x7f37c019f870 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f37b800ef30 tx=0x7f37b800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37bdffb700 1 -- 172.21.15.31:0/695826868 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8018070 con 0x7f37c0101f10 2023-07-29T16:26:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.606+0000 7f37bdffb700 1 -- 172.21.15.31:0/695826868 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37b800f040 con 0x7f37c0101f10 2023-07-29T16:26:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.607+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c01a0390 con 0x7f37c0101f10 2023-07-29T16:26:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.607+0000 7f37bdffb700 1 -- 172.21.15.31:0/695826868 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b800cd00 con 0x7f37c0101f10 2023-07-29T16:26:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.607+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c01a08a0 con 0x7f37c0101f10 2023-07-29T16:26:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.610+0000 7f37bdffb700 1 -- 172.21.15.31:0/695826868 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f37b8003a00 con 0x7f37c0101f10 2023-07-29T16:26:16.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.611+0000 7f37da2e7b80 1 Processor -- start 2023-07-29T16:26:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 -- start start 2023-07-29T16:26:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37bdffb700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f37ac0a0790 0x7f37ac0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37bdffb700 1 -- 172.21.15.31:0/695826868 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f37b80cb620 con 0x7f37c0101f10 2023-07-29T16:26:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a3231f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a323148a10 con 0x55a3231f0be0 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3231463d0 con 0x55a3231f1530 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a323141cc0 con 0x55a3231f22f0 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.612+0000 7f37b7fff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f37ac0a0790 0x7f37ac0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53176/0 (socket says 172.21.15.31:53176) 2023-07-29T16:26:16.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 -- 172.21.15.31:0/542004301 learned_addr learned my addr 172.21.15.31:0/542004301 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37b7fff700 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a3231f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 -- 172.21.15.31:0/542004301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 msgr2=0x55a3231f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a3231f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 -- 172.21.15.31:0/542004301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 msgr2=0x55a3231f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 -- 172.21.15.31:0/542004301 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a322f6fde0 con 0x55a3231f22f0 2023-07-29T16:26:16.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37b7fff700 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a3231f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37bffff700 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37b7fff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f37ac0a0790 0x7f37ac0a2c40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005950 tx=0x7f37a80149f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f2740 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f37b80090a0 tx=0x7f37b80951c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac305320808738e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37b67fc700 1 -- 172.21.15.31:0/542004301 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b800ce60 con 0x55a3231f22f0 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37b67fc700 1 -- 172.21.15.31:0/542004301 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37b8098d50 con 0x55a3231f22f0 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.613+0000 7f37b67fc700 1 -- 172.21.15.31:0/542004301 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b80991b0 con 0x55a3231f22f0 2023-07-29T16:26:16.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 -- 172.21.15.31:0/542004301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 msgr2=0x55a3231f2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f2740 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f37b80090a0 tx=0x7f37b80951c0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 -- 172.21.15.31:0/542004301 shutdown_connections 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f2740 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/542004301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a3231f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 -- 172.21.15.31:0/542004301 >> 172.21.15.31:0/542004301 conn(0x55a3231e63e0 msgr2=0x55a3231eb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 -- 172.21.15.31:0/542004301 shutdown_connections 2023-07-29T16:26:16.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.614+0000 7f37da2e7b80 1 -- 172.21.15.31:0/542004301 wait complete. 2023-07-29T16:26:16.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.615+0000 7f37da2e7b80 1 -- start start 2023-07-29T16:26:16.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.615+0000 7f37da2e7b80 1 Processor -- start 2023-07-29T16:26:16.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.615+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f0be0 0x55a3231fa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 0x55a323201dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3230de510 con 0x55a3231f1530 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3230de1b0 con 0x55a3231f22f0 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a322fc5920 con 0x55a3231f0be0 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37da2e7b80 10 client.0 _ll_register_callbacks cb 0x55a3230e7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 0x55a323201dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37b7fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37bffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f0be0 0x55a3231fa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 0x55a323201dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43026/0 (socket says 172.21.15.31:43026) 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 -- 172.21.15.31:0/534388885 learned_addr learned my addr 172.21.15.31:0/534388885 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 -- 172.21.15.31:0/534388885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f0be0 msgr2=0x55a3231fa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f0be0 0x55a3231fa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 -- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 msgr2=0x55a3231fad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.616+0000 7f37c4f9c700 1 -- 172.21.15.31:0/534388885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c0109100 con 0x55a3231f22f0 2023-07-29T16:26:16.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 0x55a323201dc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f37b8097340 tx=0x7f37b8004cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=180559162e229e4c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f379f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:16.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f379f7fe700 1 -- 172.21.15.31:0/534388885 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b809cd80 con 0x55a3231f22f0 2023-07-29T16:26:16.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f379f7fe700 1 -- 172.21.15.31:0/534388885 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f37b80a1110 con 0x55a3231f22f0 2023-07-29T16:26:16.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a32316cd10 con 0x55a3231f22f0 2023-07-29T16:26:16.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f379f7fe700 1 -- 172.21.15.31:0/534388885 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37b80a0c80 con 0x55a3231f22f0 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f37da2e7b80 10 client.28730 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f37c01a08a0 con 0x55a3231f22f0 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f379f7fe700 1 -- 172.21.15.31:0/534388885 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a32316cd10 con 0x55a3231f22f0 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.617+0000 7f379f7fe700 1 -- 172.21.15.31:0/534388885 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f37b80cb1e0 con 0x55a3231f22f0 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37da2e7b80 10 client.28730 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37da2e7b80 10 client.28730 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37da2e7b80 20 client.28730 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:16.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323206010 0x55a3232083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37da2e7b80 4 client.28730 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3231bfd70 con 0x55a323206010 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.618+0000 7f37bffff700 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323206010 0x55a3232083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.619+0000 7f37bffff700 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323206010 0x55a3232083f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc7bcdc2f617a59b server_cookie=7d8479cc92a0e541 in_seq=0 out_seq=0 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.619+0000 7f379f7fe700 10 client.28730 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.619+0000 7f379f7fe700 1 -- 172.21.15.31:0/534388885 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a3231bfd70 con 0x55a323206010 2023-07-29T16:26:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.619+0000 7f379f7fe700 10 client.28730 handle_command_reply: tid=0 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.619+0000 7f37da2e7b80 1 client.28730 shutdown 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 msgr2=0x55a323201dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 0x55a323201dc0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f37b8097340 tx=0x7f37b8004cc0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 shutdown_connections 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323206010 0x55a3232083f0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f22f0 0x55a323201dc0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f0be0 0x55a3231fa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/534388885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f379f7fe700 0 client.28730 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:16.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 >> 172.21.15.31:0/534388885 conn(0x55a3231e63e0 msgr2=0x55a3231eb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 shutdown_connections 2023-07-29T16:26:16.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 1 -- 172.21.15.31:0/534388885 wait complete. 2023-07-29T16:26:16.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.620+0000 7f37da2e7b80 20 client.28730 trim_cache size 0 max 16384 2023-07-29T16:26:16.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.634+0000 7f37da2e7b80 1 Processor -- start 2023-07-29T16:26:16.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.634+0000 7f37da2e7b80 1 -- start start 2023-07-29T16:26:16.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a323201d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3231e5ea0 con 0x55a3231f0be0 2023-07-29T16:26:16.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3231f9cf0 con 0x55a3231f1530 2023-07-29T16:26:16.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3231fa3e0 con 0x55a3231f22f0 2023-07-29T16:26:16.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a323201d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a323201d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43042/0 (socket says 172.21.15.31:43042) 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 -- 172.21.15.31:0/3076167592 learned_addr learned my addr 172.21.15.31:0/3076167592 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37bffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 -- 172.21.15.31:0/3076167592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 msgr2=0x55a3231f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 -- 172.21.15.31:0/3076167592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 msgr2=0x55a3231f8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.635+0000 7f37b7fff700 1 -- 172.21.15.31:0/3076167592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a322f58c60 con 0x55a3231f1530 2023-07-29T16:26:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.636+0000 7f37b7fff700 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a323201d80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f37a8014340 tx=0x7f37a800b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82ba5b17267d6b49 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.636+0000 7f37b6ffd700 1 -- 172.21.15.31:0/3076167592 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8011d90 con 0x55a3231f1530 2023-07-29T16:26:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.636+0000 7f37b6ffd700 1 -- 172.21.15.31:0/3076167592 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37a801a070 con 0x55a3231f1530 2023-07-29T16:26:16.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.636+0000 7f37b6ffd700 1 -- 172.21.15.31:0/3076167592 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8005690 con 0x55a3231f1530 2023-07-29T16:26:16.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3076167592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 msgr2=0x55a323201d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a323201d80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f37a8014340 tx=0x7f37a800b410 comp rx=0 tx=0).stop 2023-07-29T16:26:16.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3076167592 shutdown_connections 2023-07-29T16:26:16.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3231f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f0be0 0x55a3231f8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3076167592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f1530 0x55a323201d80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3076167592 >> 172.21.15.31:0/3076167592 conn(0x55a3231e63e0 msgr2=0x55a3231f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3076167592 shutdown_connections 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.637+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3076167592 wait complete. 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 -- start start 2023-07-29T16:26:16.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 Processor -- start 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 0x55a3231fdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3232014f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a323201a30 con 0x55a3231f1530 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a323201ba0 con 0x55a3231f0be0 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.638+0000 7f37da2e7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a323201d60 con 0x55a3231f22f0 2023-07-29T16:26:16.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37da2e7b80 10 client.0 _ll_register_callbacks cb 0x55a32320d270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 0x55a3231fdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 0x55a3231fdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43050/0 (socket says 172.21.15.31:43050) 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 -- 172.21.15.31:0/3316889443 learned_addr learned my addr 172.21.15.31:0/3316889443 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37b7fff700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37c4f9c700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3232014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 -- 172.21.15.31:0/3316889443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 msgr2=0x55a3232014f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3232014f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 -- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 msgr2=0x55a3231fe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.639+0000 7f37bffff700 1 -- 172.21.15.31:0/3316889443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a8014080 con 0x55a3231f0be0 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f37bffff700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 0x55a3231fdf90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009990 tx=0x7f37b0005290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee40243bb1475f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f379f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f379f7fe700 1 -- 172.21.15.31:0/3316889443 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b001d040 con 0x55a3231f0be0 2023-07-29T16:26:16.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37b8098d50 con 0x55a3231f0be0 2023-07-29T16:26:16.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f379f7fe700 1 -- 172.21.15.31:0/3316889443 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f37b000b9f0 con 0x55a3231f0be0 2023-07-29T16:26:16.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f379f7fe700 1 -- 172.21.15.31:0/3316889443 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37b000bbb0 con 0x55a3231f0be0 2023-07-29T16:26:16.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f37da2e7b80 10 client.28736 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:16.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.640+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a322f96250 con 0x55a3231f0be0 2023-07-29T16:26:16.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f379f7fe700 1 -- 172.21.15.31:0/3316889443 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37b8098d50 con 0x55a3231f0be0 2023-07-29T16:26:16.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f379f7fe700 1 -- 172.21.15.31:0/3316889443 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f37b000f460 con 0x55a3231f0be0 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37da2e7b80 10 client.28736 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37da2e7b80 10 client.28736 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37da2e7b80 20 client.28736 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323258ff0 0x55a32325b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37da2e7b80 4 client.28736 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a32313ad40 con 0x55a323258ff0 2023-07-29T16:26:16.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.641+0000 7f37b7fff700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323258ff0 0x55a32325b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f379f7fe700 10 client.28736 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37b7fff700 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323258ff0 0x55a32325b400 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=786a81cc4f36ee60 server_cookie=294ea323ae37006e in_seq=0 out_seq=0 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f379f7fe700 1 -- 172.21.15.31:0/3316889443 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a32313ad40 con 0x55a323258ff0 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f379f7fe700 10 client.28736 handle_command_reply: tid=0 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 client.28736 shutdown 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 msgr2=0x55a3231fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 0x55a3231fdf90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f37b0009990 tx=0x7f37b0005290 comp rx=0 tx=0).stop 2023-07-29T16:26:16.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 shutdown_connections 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a323258ff0 0x55a32325b400 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3231f22f0 0x55a3232014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a3231f0be0 0x55a3231fdf90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f379f7fe700 0 client.28736 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 --2- 172.21.15.31:0/3316889443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a3231f1530 0x55a3231fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 >> 172.21.15.31:0/3316889443 conn(0x55a3231e63e0 msgr2=0x55a3231f8710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 shutdown_connections 2023-07-29T16:26:16.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.642+0000 7f37da2e7b80 1 -- 172.21.15.31:0/3316889443 wait complete. 2023-07-29T16:26:16.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37da2e7b80 20 client.28736 trim_cache size 0 max 16384 2023-07-29T16:26:16.650 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:26:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f37ac0a0790 msgr2=0x7f37ac0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f37ac0a0790 0x7f37ac0a2c40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f37a8005950 tx=0x7f37a80149f0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 msgr2=0x7f37c019f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 0x7f37c019f870 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f37b800ef30 tx=0x7f37b800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 shutdown_connections 2023-07-29T16:26:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f37ac0a0790 0x7f37ac0a2c40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f37c0101f10 0x7f37c019f870 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37c0100390 0x7f37c01982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 --2- 172.21.15.31:0/695826868 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c0101580 0x7f37c0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 >> 172.21.15.31:0/695826868 conn(0x7f37c00fb9b0 msgr2=0x7f37c00fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.643+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 shutdown_connections 2023-07-29T16:26:16.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.644+0000 7f37c69ff700 1 -- 172.21.15.31:0/695826868 wait complete. 2023-07-29T16:26:16.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:16 smithi031 ceph-mon[95504]: pgmap v695: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.6 MiB/s wr, 256 op/s 2023-07-29T16:26:16.662 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:16 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/23548168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:26:16.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:26:16.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:16 smithi156 ceph-mon[129245]: pgmap v695: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.6 MiB/s wr, 256 op/s 2023-07-29T16:26:16.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:16 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/23548168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:26:16.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 797522 8.91 MB/sec execute 361 sec latency 15.621 ms 2023-07-29T16:26:16.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 -- 172.21.15.31:0/2912075800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101eb0 msgr2=0x7fa98c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 --2- 172.21.15.31:0/2912075800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101eb0 0x7fa98c108770 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fa984009f90 tx=0x7fa98400b330 comp rx=0 tx=0).stop 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 -- 172.21.15.31:0/2912075800 shutdown_connections 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 --2- 172.21.15.31:0/2912075800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101eb0 0x7fa98c108770 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 --2- 172.21.15.31:0/2912075800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c100330 0x7fa98c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 --2- 172.21.15.31:0/2912075800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa98c101520 0x7fa98c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 -- 172.21.15.31:0/2912075800 >> 172.21.15.31:0/2912075800 conn(0x7fa98c0fb970 msgr2=0x7fa98c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 -- 172.21.15.31:0/2912075800 shutdown_connections 2023-07-29T16:26:16.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.898+0000 7fa9909ff700 1 -- 172.21.15.31:0/2912075800 wait complete. 2023-07-29T16:26:16.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 Processor -- start 2023-07-29T16:26:16.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 -- start start 2023-07-29T16:26:16.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa98c100330 0x7fa98c196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 0x7fa98c1966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c101eb0 0x7fa98c19d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c19dcf0 con 0x7fa98c100330 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c19de30 con 0x7fa98c101520 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.900+0000 7fa9909ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c19dfc0 con 0x7fa98c101eb0 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 0x7fa98c1966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 0x7fa98c1966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43072/0 (socket says 172.21.15.31:43072) 2023-07-29T16:26:16.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa98c100330 0x7fa98c196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 -- 172.21.15.31:0/3396733327 learned_addr learned my addr 172.21.15.31:0/3396733327 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa98ad9d700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c101eb0 0x7fa98c19d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 -- 172.21.15.31:0/3396733327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c101eb0 msgr2=0x7fa98c19d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c101eb0 0x7fa98c19d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 -- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa98c100330 msgr2=0x7fa98c196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa98c100330 0x7fa98c196170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 -- 172.21.15.31:0/3396733327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c009520 con 0x7fa98c101520 2023-07-29T16:26:16.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.901+0000 7fa989d9b700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 0x7fa98c1966b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa98000ec10 tx=0x7fa98000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.902+0000 7fa97b7fe700 1 -- 172.21.15.31:0/3396733327 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98000cc90 con 0x7fa98c101520 2023-07-29T16:26:16.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.902+0000 7fa97b7fe700 1 -- 172.21.15.31:0/3396733327 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9800044e0 con 0x7fa98c101520 2023-07-29T16:26:16.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.902+0000 7fa97b7fe700 1 -- 172.21.15.31:0/3396733327 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980010430 con 0x7fa98c101520 2023-07-29T16:26:16.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.902+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9800095f0 con 0x7fa98c101520 2023-07-29T16:26:16.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.902+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa984009c40 con 0x7fa98c101520 2023-07-29T16:26:16.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.906+0000 7fa97b7fe700 1 -- 172.21.15.31:0/3396733327 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa980003fb0 con 0x7fa98c101520 2023-07-29T16:26:16.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.907+0000 7fa9a4381b80 1 Processor -- start 2023-07-29T16:26:16.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.907+0000 7fa9a4381b80 1 -- start start 2023-07-29T16:26:16.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.907+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e23450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e24140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e246d0 0x561493e24b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493d76a10 con 0x561493e23040 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493d743d0 con 0x561493e246d0 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493d6fcc0 con 0x561493e23990 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e24140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa98ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e23450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e24140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53230/0 (socket says 172.21.15.31:53230) 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa98a59c700 1 -- 172.21.15.31:0/1846876787 learned_addr learned my addr 172.21.15.31:0/1846876787 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa97b7fe700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9740a0790 0x7fa9740a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e246d0 0x561493e24b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa97b7fe700 1 -- 172.21.15.31:0/3396733327 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa980014070 con 0x7fa98c101520 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa989d9b700 1 -- 172.21.15.31:0/1846876787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 msgr2=0x561493e24140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e24140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa989d9b700 1 -- 172.21.15.31:0/1846876787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 msgr2=0x561493e23450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e23450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa98ad9d700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9740a0790 0x7fa9740a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.908+0000 7fa989d9b700 1 -- 172.21.15.31:0/1846876787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561493b9dde0 con 0x561493e246d0 2023-07-29T16:26:16.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.909+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e246d0 0x561493e24b40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa9800c9b40 tx=0x7fa980095c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dbd912b3a16b03d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.909+0000 7fa978ff9700 1 -- 172.21.15.31:0/1846876787 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980003680 con 0x561493e246d0 2023-07-29T16:26:16.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.909+0000 7fa978ff9700 1 -- 172.21.15.31:0/1846876787 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa980099310 con 0x561493e246d0 2023-07-29T16:26:16.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.909+0000 7fa978ff9700 1 -- 172.21.15.31:0/1846876787 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980097c90 con 0x561493e246d0 2023-07-29T16:26:16.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1846876787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e246d0 msgr2=0x561493e24b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e246d0 0x561493e24b40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fa9800c9b40 tx=0x7fa980095c40 comp rx=0 tx=0).stop 2023-07-29T16:26:16.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa98ad9d700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9740a0790 0x7fa9740a2c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa984009550 tx=0x7fa98400b330 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1846876787 shutdown_connections 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e246d0 0x561493e24b40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e23450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1846876787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e24140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1846876787 >> 172.21.15.31:0/1846876787 conn(0x561493e143e0 msgr2=0x561493e198a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1846876787 shutdown_connections 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1846876787 wait complete. 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.910+0000 7fa9a4381b80 1 -- start start 2023-07-29T16:26:16.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 Processor -- start 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e2cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 0x561493e2d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e340e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493d0c510 con 0x561493e23040 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493d0c1b0 con 0x561493e23990 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493bf3920 con 0x561493e246d0 2023-07-29T16:26:16.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa9a4381b80 10 client.0 _ll_register_callbacks cb 0x561493d15620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa989d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 0x561493e2d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa989d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 0x561493e2d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43084/0 (socket says 172.21.15.31:43084) 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa989d9b700 1 -- 172.21.15.31:0/2447292844 learned_addr learned my addr 172.21.15.31:0/2447292844 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.911+0000 7fa98a59c700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e2cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa98ad9d700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa989d9b700 1 -- 172.21.15.31:0/2447292844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 msgr2=0x561493e340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa989d9b700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa989d9b700 1 -- 172.21.15.31:0/2447292844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 msgr2=0x561493e2cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa989d9b700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e2cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa989d9b700 1 -- 172.21.15.31:0/2447292844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa984009c40 con 0x561493e23990 2023-07-29T16:26:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa98a59c700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e2cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa989d9b700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 0x561493e2d0a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa9800c9ee0 tx=0x7fa980003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d282c049c3b91c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa971ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:16.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa971ffb700 1 -- 172.21.15.31:0/2447292844 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980099c30 con 0x561493e23990 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa971ffb700 1 -- 172.21.15.31:0/2447292844 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa98009d800 con 0x561493e23990 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561493d9ad10 con 0x561493e23990 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa971ffb700 1 -- 172.21.15.31:0/2447292844 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa980098cc0 con 0x561493e23990 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa9a4381b80 10 client.28748 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.912+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x561493b86c60 con 0x561493e23990 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa971ffb700 1 -- 172.21.15.31:0/2447292844 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561493d9ad10 con 0x561493e23990 2023-07-29T16:26:16.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa971ffb700 1 -- 172.21.15.31:0/2447292844 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa980098730 con 0x561493e23990 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa9a4381b80 10 client.28748 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa9a4381b80 10 client.28748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa9a4381b80 20 client.28748 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e384d0 0x561493e3a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa9a4381b80 4 client.28748 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561493dedd70 con 0x561493e384d0 2023-07-29T16:26:16.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.913+0000 7fa98a59c700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e384d0 0x561493e3a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.914+0000 7fa98a59c700 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e384d0 0x561493e3a8b0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f186d44b4f9fca35 server_cookie=606f2a9bc2f18e9c in_seq=0 out_seq=0 2023-07-29T16:26:16.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.914+0000 7fa971ffb700 10 client.28748 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:16.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa971ffb700 1 -- 172.21.15.31:0/2447292844 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561493dedd70 con 0x561493e384d0 2023-07-29T16:26:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa971ffb700 10 client.28748 handle_command_reply: tid=0 2023-07-29T16:26:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 client.28748 shutdown 2023-07-29T16:26:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 msgr2=0x561493e2d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 0x561493e2d0a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fa9800c9ee0 tx=0x7fa980003a10 comp rx=0 tx=0).stop 2023-07-29T16:26:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 shutdown_connections 2023-07-29T16:26:16.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e384d0 0x561493e3a8b0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23040 0x561493e2cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/2447292844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23990 0x561493e2d0a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa971ffb700 0 client.28748 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 >> 172.21.15.31:0/2447292844 conn(0x561493e143e0 msgr2=0x561493e19060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 shutdown_connections 2023-07-29T16:26:16.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.915+0000 7fa9a4381b80 1 -- 172.21.15.31:0/2447292844 wait complete. 2023-07-29T16:26:16.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.916+0000 7fa9a4381b80 20 client.28748 trim_cache size 0 max 16384 2023-07-29T16:26:16.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 Processor -- start 2023-07-29T16:26:16.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 -- start start 2023-07-29T16:26:16.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e1a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e246d0 0x561493e24ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c075050 con 0x561493e246d0 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493e23fb0 con 0x561493e23040 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493e2c6d0 con 0x561493e23990 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e1a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa989d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e1a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43086/0 (socket says 172.21.15.31:43086) 2023-07-29T16:26:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa98a59c700 1 -- 172.21.15.31:0/730500857 learned_addr learned my addr 172.21.15.31:0/730500857 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.930+0000 7fa98ad9d700 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e246d0 0x561493e24ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa98a59c700 1 -- 172.21.15.31:0/730500857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 msgr2=0x561493e340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa98a59c700 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e340a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa98a59c700 1 -- 172.21.15.31:0/730500857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e246d0 msgr2=0x561493e24ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa98a59c700 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e246d0 0x561493e24ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa98a59c700 1 -- 172.21.15.31:0/730500857 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561493bc4250 con 0x561493e23040 2023-07-29T16:26:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa989d9b700 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:16.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa98a59c700 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e1a8e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x561493e3b1b0 tx=0x7fa97c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cceb46d8a146a6c4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa9797fa700 1 -- 172.21.15.31:0/730500857 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c01e070 con 0x561493e23040 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa9797fa700 1 -- 172.21.15.31:0/730500857 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa97c017040 con 0x561493e23040 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.931+0000 7fa9797fa700 1 -- 172.21.15.31:0/730500857 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c0053a0 con 0x561493e23040 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 -- 172.21.15.31:0/730500857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 msgr2=0x561493e1a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e1a8e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x561493e3b1b0 tx=0x7fa97c009260 comp rx=0 tx=0).stop 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 -- 172.21.15.31:0/730500857 shutdown_connections 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e246d0 0x561493e24ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e1a8e0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/730500857 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e23990 0x561493e340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 -- 172.21.15.31:0/730500857 >> 172.21.15.31:0/730500857 conn(0x561493e143e0 msgr2=0x561493e2b850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 -- 172.21.15.31:0/730500857 shutdown_connections 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.932+0000 7fa9a4381b80 1 -- 172.21.15.31:0/730500857 wait complete. 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 -- start start 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 Processor -- start 2023-07-29T16:26:16.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e302d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23990 0x561493e30810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e33420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493e33960 con 0x561493e23990 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493e33ad0 con 0x561493e23040 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561493e33c90 con 0x561493e246d0 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.933+0000 7fa9a4381b80 10 client.0 _ll_register_callbacks cb 0x561493d15620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e302d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43088/0 (socket says 172.21.15.31:43088) 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 -- 172.21.15.31:0/1372685041 learned_addr learned my addr 172.21.15.31:0/1372685041 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e33420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23990 0x561493e30810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 -- 172.21.15.31:0/1372685041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 msgr2=0x561493e33420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e33420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 -- 172.21.15.31:0/1372685041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23990 msgr2=0x561493e30810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23990 0x561493e30810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 -- 172.21.15.31:0/1372685041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561493b86c60 con 0x561493e23040 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa98a59c700 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e302d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00a010 tx=0x7fa97c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42376cd280b676de server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.934+0000 7fa971ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa971ffb700 1 -- 172.21.15.31:0/1372685041 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c005120 con 0x561493e23040 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa980099310 con 0x561493e23040 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa971ffb700 1 -- 172.21.15.31:0/1372685041 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa97c005280 con 0x561493e23040 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa971ffb700 1 -- 172.21.15.31:0/1372685041 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa97c005440 con 0x561493e23040 2023-07-29T16:26:16.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 10 client.28754 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fa97c004120 con 0x561493e23040 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa971ffb700 1 -- 172.21.15.31:0/1372685041 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa980099310 con 0x561493e23040 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa971ffb700 1 -- 172.21.15.31:0/1372685041 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa97c036070 con 0x561493e23040 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 10 client.28754 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 10 client.28754 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 20 client.28754 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:16.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e8afc0 0x561493e8d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 4 client.28754 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:26:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.935+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561493d68d40 con 0x561493e8afc0 2023-07-29T16:26:16.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.936+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e8afc0 0x561493e8d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:16.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.936+0000 7fa989d9b700 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e8afc0 0x561493e8d3d0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58957aae18fde1d server_cookie=2ead090b89a63300 in_seq=0 out_seq=0 2023-07-29T16:26:16.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.936+0000 7fa971ffb700 10 client.28754 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:16.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.936+0000 7fa971ffb700 1 -- 172.21.15.31:0/1372685041 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561493d68d40 con 0x561493e8afc0 2023-07-29T16:26:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.936+0000 7fa971ffb700 10 client.28754 handle_command_reply: tid=0 2023-07-29T16:26:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.936+0000 7fa9a4381b80 1 client.28754 shutdown 2023-07-29T16:26:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 msgr2=0x561493e302d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e302d0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa97c00a010 tx=0x7fa97c0047c0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 shutdown_connections 2023-07-29T16:26:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561493e8afc0 0x561493e8d3d0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561493e246d0 0x561493e33420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561493e23040 0x561493e302d0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 --2- 172.21.15.31:0/1372685041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561493e23990 0x561493e30810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa971ffb700 0 client.28754 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:16.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 >> 172.21.15.31:0/1372685041 conn(0x561493e143e0 msgr2=0x561493e2aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 shutdown_connections 2023-07-29T16:26:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 1 -- 172.21.15.31:0/1372685041 wait complete. 2023-07-29T16:26:16.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.937+0000 7fa9a4381b80 20 client.28754 trim_cache size 0 max 16384 2023-07-29T16:26:16.948 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:26:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9740a0790 msgr2=0x7fa9740a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9740a0790 0x7fa9740a2c40 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa984009550 tx=0x7fa98400b330 comp rx=0 tx=0).stop 2023-07-29T16:26:16.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 msgr2=0x7fa98c1966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 0x7fa98c1966b0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fa98000ec10 tx=0x7fa98000ef20 comp rx=0 tx=0).stop 2023-07-29T16:26:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 shutdown_connections 2023-07-29T16:26:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa9740a0790 0x7fa9740a2c40 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa98c101eb0 0x7fa98c19d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa98c100330 0x7fa98c196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 --2- 172.21.15.31:0/3396733327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa98c101520 0x7fa98c1966b0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 >> 172.21.15.31:0/3396733327 conn(0x7fa98c0fb970 msgr2=0x7fa98c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 shutdown_connections 2023-07-29T16:26:16.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:16.938+0000 7fa9909ff700 1 -- 172.21.15.31:0/3396733327 wait complete. 2023-07-29T16:26:17.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:16 smithi039 ceph-mon[128858]: pgmap v695: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.6 MiB/s wr, 256 op/s 2023-07-29T16:26:17.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:16 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/23548168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:26:17.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 799166 8.92 MB/sec execute 362 sec latency 40.251 ms 2023-07-29T16:26:17.958 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:26:17.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:26:18.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 -- 172.21.15.31:0/1636077373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 msgr2=0x7fd5f4108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/1636077373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f4108440 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec009f90 tx=0x7fd5ec00b330 comp rx=0 tx=0).stop 2023-07-29T16:26:18.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 -- 172.21.15.31:0/1636077373 shutdown_connections 2023-07-29T16:26:18.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/1636077373 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f4108440 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/1636077373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f4100da0 0x7fd5f4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/1636077373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f41036c0 0x7fd5f4105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 -- 172.21.15.31:0/1636077373 >> 172.21.15.31:0/1636077373 conn(0x7fd5f40fa840 msgr2=0x7fd5f40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 -- 172.21.15.31:0/1636077373 shutdown_connections 2023-07-29T16:26:18.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.191+0000 7fd5f9bff700 1 -- 172.21.15.31:0/1636077373 wait complete. 2023-07-29T16:26:18.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.192+0000 7fd5f9bff700 1 Processor -- start 2023-07-29T16:26:18.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.192+0000 7fd5f9bff700 1 -- start start 2023-07-29T16:26:18.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f9bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 0x7fd5f4198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f9bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f41036c0 0x7fd5f41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f9bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f419f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f9bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f419ff10 con 0x7fd5f41036c0 2023-07-29T16:26:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f9bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f41a0050 con 0x7fd5f4100da0 2023-07-29T16:26:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f9bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f41a01e0 con 0x7fd5f4105fe0 2023-07-29T16:26:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 0x7fd5f4198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f419f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 0x7fd5f4198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43104/0 (socket says 172.21.15.31:43104) 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f37fe700 1 -- 172.21.15.31:0/633191123 learned_addr learned my addr 172.21.15.31:0/633191123 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f41036c0 0x7fd5f41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f3fff700 1 -- 172.21.15.31:0/633191123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 msgr2=0x7fd5f4198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f3fff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 0x7fd5f4198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f3fff700 1 -- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f41036c0 msgr2=0x7fd5f41988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f3fff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f41036c0 0x7fd5f41988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.193+0000 7fd5f3fff700 1 -- 172.21.15.31:0/633191123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec009c40 con 0x7fd5f4105fe0 2023-07-29T16:26:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 0x7fd5f4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f3fff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f419f940 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00b6c0 tx=0x7fd5ec004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f0ff9700 1 -- 172.21.15.31:0/633191123 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec00e040 con 0x7fd5f4105fe0 2023-07-29T16:26:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f0ff9700 1 -- 172.21.15.31:0/633191123 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5ec00bbf0 con 0x7fd5f4105fe0 2023-07-29T16:26:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f41a0460 con 0x7fd5f4105fe0 2023-07-29T16:26:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f0ff9700 1 -- 172.21.15.31:0/633191123 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec005040 con 0x7fd5f4105fe0 2023-07-29T16:26:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.194+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f41a08e0 con 0x7fd5f4105fe0 2023-07-29T16:26:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.198+0000 7fd5f0ff9700 1 -- 172.21.15.31:0/633191123 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd5ec01d470 con 0x7fd5f4105fe0 2023-07-29T16:26:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.200+0000 7fd60d445b80 1 Processor -- start 2023-07-29T16:26:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.200+0000 7fd60d445b80 1 -- start start 2023-07-29T16:26:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907af070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907afde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd60d445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca90706a90 con 0x55ca907b0370 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd60d445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca90704450 con 0x55ca907af5b0 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd60d445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca906ffd40 con 0x55ca907aec60 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f0ff9700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd5dc0a0810 0x7fd5dc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43106/0 (socket says 172.21.15.31:43106) 2023-07-29T16:26:18.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f37fe700 1 -- 172.21.15.31:0/958476821 learned_addr learned my addr 172.21.15.31:0/958476821 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f0ff9700 1 -- 172.21.15.31:0/633191123 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd5ec0ccf00 con 0x7fd5f4105fe0 2023-07-29T16:26:18.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f37fe700 1 -- 172.21.15.31:0/958476821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 msgr2=0x55ca907af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f3fff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd5dc0a0810 0x7fd5dc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.201+0000 7fd5f37fe700 1 -- 172.21.15.31:0/958476821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 msgr2=0x55ca907b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5f37fe700 1 -- 172.21.15.31:0/958476821 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca9052de60 con 0x55ca907af5b0 2023-07-29T16:26:18.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:26:18.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907afde0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40117e0 tx=0x7fd5e4011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc57563ca263f99c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5d9ffb700 1 -- 172.21.15.31:0/958476821 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400f480 con 0x55ca907af5b0 2023-07-29T16:26:18.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5f3fff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd5dc0a0810 0x7fd5dc0a2cc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec09a620 tx=0x7fd5ec0bb040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5d9ffb700 1 -- 172.21.15.31:0/958476821 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5e400faa0 con 0x55ca907af5b0 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.202+0000 7fd5d9ffb700 1 -- 172.21.15.31:0/958476821 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4019b60 con 0x55ca907af5b0 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 -- 172.21.15.31:0/958476821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 msgr2=0x55ca907afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907afde0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40117e0 tx=0x7fd5e4011af0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 -- 172.21.15.31:0/958476821 shutdown_connections 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 --2- 172.21.15.31:0/958476821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907afde0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.203+0000 7fd60d445b80 1 -- 172.21.15.31:0/958476821 >> 172.21.15.31:0/958476821 conn(0x55ca907a4460 msgr2=0x55ca907a9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.204+0000 7fd60d445b80 1 -- 172.21.15.31:0/958476821 shutdown_connections 2023-07-29T16:26:18.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.204+0000 7fd60d445b80 1 -- 172.21.15.31:0/958476821 wait complete. 2023-07-29T16:26:18.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.205+0000 7fd60d445b80 1 -- start start 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.205+0000 7fd60d445b80 1 Processor -- start 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907aec60 0x55ca907b88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 0x55ca907b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca9069c590 con 0x55ca907b0370 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca9069c230 con 0x55ca907aec60 2023-07-29T16:26:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca905839a0 con 0x55ca907af5b0 2023-07-29T16:26:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd60d445b80 10 client.0 _ll_register_callbacks cb 0x55ca906a56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 0x55ca907b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 0x55ca907b8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53304/0 (socket says 172.21.15.31:53304) 2023-07-29T16:26:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 -- 172.21.15.31:0/2894224465 learned_addr learned my addr 172.21.15.31:0/2894224465 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907aec60 0x55ca907b88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 -- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907aec60 msgr2=0x55ca907b88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907aec60 0x55ca907b88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 -- 172.21.15.31:0/2894224465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 msgr2=0x55ca907bfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907bfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.206+0000 7fd5f37fe700 1 -- 172.21.15.31:0/2894224465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e4011440 con 0x55ca907af5b0 2023-07-29T16:26:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 0x55ca907b8e10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd5e401c040 tx=0x7fd5e400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c7aa25f7c8c3a28 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd5d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/2894224465 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e400f920 con 0x55ca907af5b0 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/2894224465 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd5e4005ce0 con 0x55ca907af5b0 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ca9072ad90 con 0x55ca907af5b0 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/2894224465 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5e4004250 con 0x55ca907af5b0 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd60d445b80 10 client.28768 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.207+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ca90516ce0 con 0x55ca907af5b0 2023-07-29T16:26:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/2894224465 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ca9072ad90 con 0x55ca907af5b0 2023-07-29T16:26:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/2894224465 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fd5e40051f0 con 0x55ca907af5b0 2023-07-29T16:26:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd60d445b80 10 client.28768 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd60d445b80 10 client.28768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd60d445b80 20 client.28768 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:18.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd60d445b80 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca907c44a0 0x55ca907c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd60d445b80 4 client.28768 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:18.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca9077ddf0 con 0x55ca907c44a0 2023-07-29T16:26:18.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.208+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca907c44a0 0x55ca907c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.209+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca907c44a0 0x55ca907c68b0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=206b6390d2fe5dd2 server_cookie=1df552cbb248b35a in_seq=0 out_seq=0 2023-07-29T16:26:18.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.209+0000 7fd5d2ffd700 10 client.28768 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/2894224465 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ca9077ddf0 con 0x55ca907c44a0 2023-07-29T16:26:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd5d2ffd700 10 client.28768 handle_command_reply: tid=0 2023-07-29T16:26:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 client.28768 shutdown 2023-07-29T16:26:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 msgr2=0x55ca907b8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 0x55ca907b8e10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd5e401c040 tx=0x7fd5e400f730 comp rx=0 tx=0).stop 2023-07-29T16:26:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 shutdown_connections 2023-07-29T16:26:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca907c44a0 0x55ca907c68b0 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca907bfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907aec60 0x55ca907b88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd5d2ffd700 0 client.28768 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 --2- 172.21.15.31:0/2894224465 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907af5b0 0x55ca907b8e10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 >> 172.21.15.31:0/2894224465 conn(0x55ca907a4460 msgr2=0x55ca907a90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 shutdown_connections 2023-07-29T16:26:18.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 1 -- 172.21.15.31:0/2894224465 wait complete. 2023-07-29T16:26:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.210+0000 7fd60d445b80 20 client.28768 trim_cache size 0 max 16384 2023-07-29T16:26:18.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 Processor -- start 2023-07-29T16:26:18.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 -- start start 2023-07-29T16:26:18.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907aec60 0x55ca907bde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907b0370 0x55ca907b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca907b84a0 con 0x55ca907aec60 2023-07-29T16:26:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca907a64e0 con 0x55ca907af5b0 2023-07-29T16:26:18.232 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.229+0000 7fd60d445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca907afc20 con 0x55ca907b0370 2023-07-29T16:26:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907b32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43122/0 (socket says 172.21.15.31:43122) 2023-07-29T16:26:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3442908015 learned_addr learned my addr 172.21.15.31:0/3442908015 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907aec60 0x55ca907bde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907b0370 0x55ca907b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3442908015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907b0370 msgr2=0x55ca907b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907b0370 0x55ca907b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3442908015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907aec60 msgr2=0x55ca907bde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907aec60 0x55ca907bde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3442908015 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca905542d0 con 0x55ca907af5b0 2023-07-29T16:26:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907aec60 0x55ca907bde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.230+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907b32d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4011730 tx=0x7fd5e4005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72ac600f5afd70da server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.231+0000 7fd5da7fc700 1 -- 172.21.15.31:0/3442908015 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4005ce0 con 0x55ca907af5b0 2023-07-29T16:26:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.231+0000 7fd5da7fc700 1 -- 172.21.15.31:0/3442908015 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5e400f7a0 con 0x55ca907af5b0 2023-07-29T16:26:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.231+0000 7fd5da7fc700 1 -- 172.21.15.31:0/3442908015 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e402b890 con 0x55ca907af5b0 2023-07-29T16:26:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 -- 172.21.15.31:0/3442908015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 msgr2=0x55ca907b32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907b32d0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4011730 tx=0x7fd5e4005720 comp rx=0 tx=0).stop 2023-07-29T16:26:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 -- 172.21.15.31:0/3442908015 shutdown_connections 2023-07-29T16:26:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907b0370 0x55ca907b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907aec60 0x55ca907bde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3442908015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907b32d0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 -- 172.21.15.31:0/3442908015 >> 172.21.15.31:0/3442908015 conn(0x55ca907a4460 msgr2=0x55ca907b88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 -- 172.21.15.31:0/3442908015 shutdown_connections 2023-07-29T16:26:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 -- 172.21.15.31:0/3442908015 wait complete. 2023-07-29T16:26:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.232+0000 7fd60d445b80 1 -- start start 2023-07-29T16:26:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 Processor -- start 2023-07-29T16:26:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907bd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca90815fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca90816520 con 0x55ca907b0370 2023-07-29T16:26:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca90816660 con 0x55ca907af5b0 2023-07-29T16:26:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca908167a0 con 0x55ca907aec60 2023-07-29T16:26:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd60d445b80 10 client.0 _ll_register_callbacks cb 0x55ca906a56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:18.242 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907bd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd5f37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907bd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43138/0 (socket says 172.21.15.31:43138) 2023-07-29T16:26:18.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd5f2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.233+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3495797281 learned_addr learned my addr 172.21.15.31:0/3495797281 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca90815fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3495797281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 msgr2=0x55ca907bcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907bcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3495797281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 msgr2=0x55ca90815fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca90815fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f37fe700 1 -- 172.21.15.31:0/3495797281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca90516ce0 con 0x55ca907af5b0 2023-07-29T16:26:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca90815fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5f37fe700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907bd2f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4005670 tx=0x7fd5e40038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2023b2418eb4830 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5d2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/3495797281 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4005ce0 con 0x55ca907af5b0 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/3495797281 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd5e4003d60 con 0x55ca907af5b0 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.234+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5e400faa0 con 0x55ca907af5b0 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/3495797281 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5e400f910 con 0x55ca907af5b0 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 10 client.28772 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ca9052c330 con 0x55ca907af5b0 2023-07-29T16:26:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/3495797281 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5e404b080 con 0x55ca907af5b0 2023-07-29T16:26:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/3495797281 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fd5e4033070 con 0x55ca907af5b0 2023-07-29T16:26:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 10 client.28772 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 10 client.28772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 20 client.28772 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca90817d90 0x55ca9081a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 4 client.28772 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:26:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.235+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ca906f8dc0 con 0x55ca90817d90 2023-07-29T16:26:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.236+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca90817d90 0x55ca9081a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.236+0000 7fd5f2ffd700 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca90817d90 0x55ca9081a1a0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31eae55ba1a607 server_cookie=2f6710a1714e271b in_seq=0 out_seq=0 2023-07-29T16:26:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.236+0000 7fd5d2ffd700 10 client.28772 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.243+0000 7fd5d2ffd700 1 -- 172.21.15.31:0/3495797281 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ca906f8dc0 con 0x55ca90817d90 2023-07-29T16:26:18.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.243+0000 7fd5d2ffd700 10 client.28772 handle_command_reply: tid=0 2023-07-29T16:26:18.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.243+0000 7fd60d445b80 1 client.28772 shutdown 2023-07-29T16:26:18.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.243+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 msgr2=0x55ca907bd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.243+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907bd2f0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4005670 tx=0x7fd5e40038b0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 shutdown_connections 2023-07-29T16:26:18.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ca90817d90 0x55ca9081a1a0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ca907b0370 0x55ca90815fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd5d2ffd700 0 client.28772 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ca907aec60 0x55ca907bcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 --2- 172.21.15.31:0/3495797281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ca907af5b0 0x55ca907bd2f0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 >> 172.21.15.31:0/3495797281 conn(0x55ca907a4460 msgr2=0x55ca907b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 shutdown_connections 2023-07-29T16:26:18.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 1 -- 172.21.15.31:0/3495797281 wait complete. 2023-07-29T16:26:18.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.244+0000 7fd60d445b80 20 client.28772 trim_cache size 0 max 16384 2023-07-29T16:26:18.253 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:26:18.253 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:26:18.254 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "7f3ee09d-429c-42d8-8daa-1d28c3fb5007", 2023-07-29T16:26:18.254 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:26:18.254 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:26:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd5dc0a0810 msgr2=0x7fd5dc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd5dc0a0810 0x7fd5dc0a2cc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec09a620 tx=0x7fd5ec0bb040 comp rx=0 tx=0).stop 2023-07-29T16:26:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 msgr2=0x7fd5f419f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f419f940 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00b6c0 tx=0x7fd5ec004c60 comp rx=0 tx=0).stop 2023-07-29T16:26:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 shutdown_connections 2023-07-29T16:26:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd5dc0a0810 0x7fd5dc0a2cc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5f4105fe0 0x7fd5f419f940 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5f4100da0 0x7fd5f4198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 --2- 172.21.15.31:0/633191123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f41036c0 0x7fd5f41988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 >> 172.21.15.31:0/633191123 conn(0x7fd5f40fa840 msgr2=0x7fd5f40ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 shutdown_connections 2023-07-29T16:26:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.245+0000 7fd5f9bff700 1 -- 172.21.15.31:0/633191123 wait complete. 2023-07-29T16:26:18.265 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:17 smithi031 ceph-mon[95504]: pgmap v696: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 2.1 MiB/s wr, 221 op/s 2023-07-29T16:26:18.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:26:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:17 smithi039 ceph-mon[128858]: pgmap v696: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 2.1 MiB/s wr, 221 op/s 2023-07-29T16:26:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:17 smithi156 ceph-mon[129245]: pgmap v696: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 2.1 MiB/s wr, 221 op/s 2023-07-29T16:26:18.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 -- 172.21.15.31:0/2841992554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 msgr2=0x7f9598103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 --2- 172.21.15.31:0/2841992554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f9598103180 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9588009990 tx=0x7f9588009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 -- 172.21.15.31:0/2841992554 shutdown_connections 2023-07-29T16:26:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 --2- 172.21.15.31:0/2841992554 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 0x7f9598108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 --2- 172.21.15.31:0/2841992554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f9598103180 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 --2- 172.21.15.31:0/2841992554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95981036c0 0x7f9598105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.518+0000 7f959dfff700 1 -- 172.21.15.31:0/2841992554 >> 172.21.15.31:0/2841992554 conn(0x7f95980fa840 msgr2=0x7f95980fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.519+0000 7f959dfff700 1 -- 172.21.15.31:0/2841992554 shutdown_connections 2023-07-29T16:26:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.519+0000 7f959dfff700 1 -- 172.21.15.31:0/2841992554 wait complete. 2023-07-29T16:26:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 Processor -- start 2023-07-29T16:26:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 -- start start 2023-07-29T16:26:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f95981982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95981036c0 0x7f95981987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 0x7f959819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959819fe20 con 0x7f9598105fe0 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959819ff60 con 0x7f9598100da0 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f959dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95981a00f0 con 0x7f95981036c0 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f95977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f95981982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f95977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f95981982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43156/0 (socket says 172.21.15.31:43156) 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f95977fe700 1 -- 172.21.15.31:0/3055486646 learned_addr learned my addr 172.21.15.31:0/3055486646 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.520+0000 7f9596ffd700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95981036c0 0x7f95981987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f9597fff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 0x7f959819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f95977fe700 1 -- 172.21.15.31:0/3055486646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95981036c0 msgr2=0x7f95981987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f95977fe700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95981036c0 0x7f95981987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f95977fe700 1 -- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 msgr2=0x7f959819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f95977fe700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 0x7f959819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f95977fe700 1 -- 172.21.15.31:0/3055486646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95880095f0 con 0x7f9598100da0 2023-07-29T16:26:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f9597fff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 0x7f959819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:26:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f95977fe700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f95981982a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f958800b680 tx=0x7f9588005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f9594ff9700 1 -- 172.21.15.31:0/3055486646 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958801d070 con 0x7f9598100da0 2023-07-29T16:26:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f9594ff9700 1 -- 172.21.15.31:0/3055486646 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9588022470 con 0x7f9598100da0 2023-07-29T16:26:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95981a0370 con 0x7f9598100da0 2023-07-29T16:26:18.525 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f9594ff9700 1 -- 172.21.15.31:0/3055486646 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9588017650 con 0x7f9598100da0 2023-07-29T16:26:18.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.521+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95981a07f0 con 0x7f9598100da0 2023-07-29T16:26:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.525+0000 7f9594ff9700 1 -- 172.21.15.31:0/3055486646 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f95880225e0 con 0x7f9598100da0 2023-07-29T16:26:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.526+0000 7f95b1a10b80 1 Processor -- start 2023-07-29T16:26:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.526+0000 7f95b1a10b80 1 -- start start 2023-07-29T16:26:18.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.526+0000 7f9594ff9700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95800a0810 0x7f95800a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f86899be0 0x563f86899ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f8689ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f8689b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f867f1a10 con 0x563f86899be0 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f867ef3d0 con 0x563f8689a530 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f867eacc0 con 0x563f8689b2f0 2023-07-29T16:26:18.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9594ff9700 1 -- 172.21.15.31:0/3055486646 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f95880ccfa0 con 0x7f9598100da0 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9597fff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95800a0810 0x7f95800a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f8689ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f8689ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43170/0 (socket says 172.21.15.31:43170) 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 -- 172.21.15.31:0/2585820934 learned_addr learned my addr 172.21.15.31:0/2585820934 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9597fff700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f8689b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95977fe700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f86899be0 0x563f86899ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 -- 172.21.15.31:0/2585820934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 msgr2=0x563f8689b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f8689b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 -- 172.21.15.31:0/2585820934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f86899be0 msgr2=0x563f86899ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f86899be0 0x563f86899ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9596ffd700 1 -- 172.21.15.31:0/2585820934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f86618de0 con 0x563f8689a530 2023-07-29T16:26:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f95977fe700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f86899be0 0x563f86899ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:18.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.527+0000 7f9597fff700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f8689b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:18.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.528+0000 7f9596ffd700 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f8689ad60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f958c009d80 tx=0x7f958c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d02f675c6fab40e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.528+0000 7f9597fff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95800a0810 0x7f95800a2cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9590009fd0 tx=0x7f9590009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.528+0000 7f957dffb700 1 -- 172.21.15.31:0/2585820934 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c004230 con 0x563f8689a530 2023-07-29T16:26:18.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.528+0000 7f957dffb700 1 -- 172.21.15.31:0/2585820934 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f958c004390 con 0x563f8689a530 2023-07-29T16:26:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.528+0000 7f957dffb700 1 -- 172.21.15.31:0/2585820934 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c019880 con 0x563f8689a530 2023-07-29T16:26:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2585820934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 msgr2=0x563f8689ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f8689ad60 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f958c009d80 tx=0x7f958c011720 comp rx=0 tx=0).stop 2023-07-29T16:26:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2585820934 shutdown_connections 2023-07-29T16:26:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f8689b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f86899be0 0x563f86899ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2585820934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f8689ad60 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2585820934 >> 172.21.15.31:0/2585820934 conn(0x563f8688f3e0 msgr2=0x563f86894990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2585820934 shutdown_connections 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2585820934 wait complete. 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.529+0000 7f95b1a10b80 1 -- start start 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 Processor -- start 2023-07-29T16:26:18.536 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 0x563f868a3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f868aad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f86787510 con 0x563f8689a530 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f867871b0 con 0x563f86899be0 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f8666e920 con 0x563f8689b2f0 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95b1a10b80 10 client.0 _ll_register_callbacks cb 0x563f86790620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:18.537 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 0x563f868a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 0x563f868a3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43182/0 (socket says 172.21.15.31:43182) 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.530+0000 7f95977fe700 1 -- 172.21.15.31:0/3617424395 learned_addr learned my addr 172.21.15.31:0/3617424395 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f9596ffd700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f9597fff700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f868aad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95977fe700 1 -- 172.21.15.31:0/3617424395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 msgr2=0x563f868aad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95977fe700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f868aad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.538 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95977fe700 1 -- 172.21.15.31:0/3617424395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 msgr2=0x563f868a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95977fe700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95977fe700 1 -- 172.21.15.31:0/3617424395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c0099e0 con 0x563f86899be0 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f9596ffd700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95977fe700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 0x563f868a3800 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9588097140 tx=0x7f958809a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a358e31733e91e64 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f9576ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f9576ffd700 1 -- 172.21.15.31:0/3617424395 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880caea0 con 0x563f86899be0 2023-07-29T16:26:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f9576ffd700 1 -- 172.21.15.31:0/3617424395 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f958809a6d0 con 0x563f86899be0 2023-07-29T16:26:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.531+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f86815d10 con 0x563f86899be0 2023-07-29T16:26:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f9576ffd700 1 -- 172.21.15.31:0/3617424395 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9588022ac0 con 0x563f86899be0 2023-07-29T16:26:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 10 client.28784 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563f86601c60 con 0x563f86899be0 2023-07-29T16:26:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f9576ffd700 1 -- 172.21.15.31:0/3617424395 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563f86815d10 con 0x563f86899be0 2023-07-29T16:26:18.540 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f9576ffd700 1 -- 172.21.15.31:0/3617424395 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f958809a9c0 con 0x563f86899be0 2023-07-29T16:26:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 10 client.28784 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 10 client.28784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 20 client.28784 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f868af220 0x563f868b1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 4 client.28784 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.532+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f86868d70 con 0x563f868af220 2023-07-29T16:26:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.533+0000 7f9596ffd700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f868af220 0x563f868b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.533+0000 7f9596ffd700 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f868af220 0x563f868b1630 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=802b396c2ae32c3b server_cookie=65851991f490ca95 in_seq=0 out_seq=0 2023-07-29T16:26:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.533+0000 7f9576ffd700 10 client.28784 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.534+0000 7f9576ffd700 1 -- 172.21.15.31:0/3617424395 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563f86868d70 con 0x563f868af220 2023-07-29T16:26:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.534+0000 7f9576ffd700 10 client.28784 handle_command_reply: tid=0 2023-07-29T16:26:18.543 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.534+0000 7f95b1a10b80 1 client.28784 shutdown 2023-07-29T16:26:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.534+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 msgr2=0x563f868a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.534+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 0x563f868a3800 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f9588097140 tx=0x7f958809a340 comp rx=0 tx=0).stop 2023-07-29T16:26:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 shutdown_connections 2023-07-29T16:26:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f868af220 0x563f868b1630 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.544 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f8689b2f0 0x563f868aad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f9576ffd700 0 client.28784 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f86899be0 0x563f868a3800 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/3617424395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 >> 172.21.15.31:0/3617424395 conn(0x563f8688f3e0 msgr2=0x563f86894060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 shutdown_connections 2023-07-29T16:26:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 1 -- 172.21.15.31:0/3617424395 wait complete. 2023-07-29T16:26:18.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.535+0000 7f95b1a10b80 20 client.28784 trim_cache size 0 max 16384 2023-07-29T16:26:18.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.553+0000 7f95b1a10b80 1 Processor -- start 2023-07-29T16:26:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.553+0000 7f95b1a10b80 1 -- start start 2023-07-29T16:26:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 0x563f8689b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f8689bc00 con 0x563f8689a530 2023-07-29T16:26:18.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f868a2cb0 con 0x563f8689b2f0 2023-07-29T16:26:18.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f8689abc0 con 0x563f86899be0 2023-07-29T16:26:18.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f9597fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 0x563f8689b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f9596ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f9597fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 0x563f8689b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43184/0 (socket says 172.21.15.31:43184) 2023-07-29T16:26:18.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f9597fff700 1 -- 172.21.15.31:0/2504112632 learned_addr learned my addr 172.21.15.31:0/2504112632 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95977fe700 1 -- 172.21.15.31:0/2504112632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 msgr2=0x563f8689b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95977fe700 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 0x563f8689b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95977fe700 1 -- 172.21.15.31:0/2504112632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 msgr2=0x563f868a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95977fe700 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f95977fe700 1 -- 172.21.15.31:0/2504112632 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f8663f250 con 0x563f86899be0 2023-07-29T16:26:18.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f9597fff700 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 0x563f8689b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:18.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.554+0000 7f9596ffd700 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:18.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.555+0000 7f95977fe700 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a8d30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9588098a20 tx=0x7f9588009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a842ad9e4220c374 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.555+0000 7f957e7fc700 1 -- 172.21.15.31:0/2504112632 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880caea0 con 0x563f86899be0 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.555+0000 7f957e7fc700 1 -- 172.21.15.31:0/2504112632 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f958809a6d0 con 0x563f86899be0 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.555+0000 7f957e7fc700 1 -- 172.21.15.31:0/2504112632 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95880a28b0 con 0x563f86899be0 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2504112632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 msgr2=0x563f868a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a8d30 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9588098a20 tx=0x7f9588009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2504112632 shutdown_connections 2023-07-29T16:26:18.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689b2f0 0x563f8689b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a8d30 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/2504112632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689a530 0x563f868a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2504112632 >> 172.21.15.31:0/2504112632 conn(0x563f8688f3e0 msgr2=0x563f868a24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2504112632 shutdown_connections 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 -- 172.21.15.31:0/2504112632 wait complete. 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 -- start start 2023-07-29T16:26:18.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.556+0000 7f95b1a10b80 1 Processor -- start 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f868a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689b2f0 0x563f868a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f868a8380 con 0x563f8689b2f0 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f868a84f0 con 0x563f8689a530 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f868a86b0 con 0x563f86899be0 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95b1a10b80 10 client.0 _ll_register_callbacks cb 0x563f86790620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:18.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9596ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f868a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9597fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689b2f0 0x563f868a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9596ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f868a6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43186/0 (socket says 172.21.15.31:43186) 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9596ffd700 1 -- 172.21.15.31:0/1821488567 learned_addr learned my addr 172.21.15.31:0/1821488567 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f95977fe700 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9596ffd700 1 -- 172.21.15.31:0/1821488567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 msgr2=0x563f868a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9596ffd700 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.557+0000 7f9596ffd700 1 -- 172.21.15.31:0/1821488567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689b2f0 msgr2=0x563f868a7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9596ffd700 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689b2f0 0x563f868a7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9596ffd700 1 -- 172.21.15.31:0/1821488567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f86618de0 con 0x563f8689a530 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9596ffd700 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f868a6e30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f958c019460 tx=0x7f958c005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c007a6696c7de37 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9576ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9576ffd700 1 -- 172.21.15.31:0/1821488567 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c02a040 con 0x563f8689a530 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9576ffd700 1 -- 172.21.15.31:0/1821488567 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f958c005240 con 0x563f8689a530 2023-07-29T16:26:18.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f958c004390 con 0x563f8689a530 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f9576ffd700 1 -- 172.21.15.31:0/1821488567 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f958c02b450 con 0x563f8689a530 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f95b1a10b80 10 client.28790 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.558+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f9590007d80 con 0x563f8689a530 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f9576ffd700 1 -- 172.21.15.31:0/1821488567 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f958c039020 con 0x563f8689a530 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f9576ffd700 1 -- 172.21.15.31:0/1821488567 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f958c004600 con 0x563f8689a530 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95b1a10b80 10 client.28790 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95b1a10b80 10 client.28790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:18.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95b1a10b80 20 client.28790 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f86901af0 0x563f86903f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95b1a10b80 4 client.28790 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563f867e3d40 con 0x563f86901af0 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95977fe700 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f86901af0 0x563f86903f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f95977fe700 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f86901af0 0x563f86903f00 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa66d9cb8331ea4f server_cookie=c0df756b99b149e7 in_seq=0 out_seq=0 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.559+0000 7f9576ffd700 10 client.28790 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f9576ffd700 1 -- 172.21.15.31:0/1821488567 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563f867e3d40 con 0x563f86901af0 2023-07-29T16:26:18.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f9576ffd700 10 client.28790 handle_command_reply: tid=0 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 client.28790 shutdown 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 msgr2=0x563f868a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f868a6e30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f958c019460 tx=0x7f958c005b20 comp rx=0 tx=0).stop 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 shutdown_connections 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563f86901af0 0x563f86903f00 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563f8689b2f0 0x563f868a7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f86899be0 0x563f868a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 --2- 172.21.15.31:0/1821488567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563f8689a530 0x563f868a6e30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 >> 172.21.15.31:0/1821488567 conn(0x563f8688f3e0 msgr2=0x563f868a16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f9576ffd700 0 client.28790 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:18.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 shutdown_connections 2023-07-29T16:26:18.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 1 -- 172.21.15.31:0/1821488567 wait complete. 2023-07-29T16:26:18.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.560+0000 7f95b1a10b80 20 client.28790 trim_cache size 0 max 16384 2023-07-29T16:26:18.573 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:26:18.573 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:26:18.573 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:26:18.573 INFO:teuthology.orchestra.run.smithi031.stdout: "7f3ee09d-429c-42d8-8daa-1d28c3fb5007": { 2023-07-29T16:26:18.573 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:26:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "7f3ee09d-429c-42d8-8daa-1d28c3fb5007", 2023-07-29T16:26:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:26:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:26:18.574 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:26:18.574 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:26:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95800a0810 msgr2=0x7f95800a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95800a0810 0x7f95800a2cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9590009fd0 tx=0x7f9590009920 comp rx=0 tx=0).stop 2023-07-29T16:26:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 msgr2=0x7f95981982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f95981982a0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f958800b680 tx=0x7f9588005740 comp rx=0 tx=0).stop 2023-07-29T16:26:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 shutdown_connections 2023-07-29T16:26:18.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f95800a0810 0x7f95800a2cc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9598105fe0 0x7f959819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9598100da0 0x7f95981982a0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 --2- 172.21.15.31:0/3055486646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95981036c0 0x7f95981987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:18.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 >> 172.21.15.31:0/3055486646 conn(0x7f95980fa840 msgr2=0x7f95980ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:18.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 shutdown_connections 2023-07-29T16:26:18.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:18.561+0000 7f959dfff700 1 -- 172.21.15.31:0/3055486646 wait complete. 2023-07-29T16:26:18.580 INFO:tasks.cephfs.filesystem:scrub status for tag:7f3ee09d-429c-42d8-8daa-1d28c3fb5007 - {'path': '/', 'tag': '7f3ee09d-429c-42d8-8daa-1d28c3fb5007', 'options': 'recursive,force'} 2023-07-29T16:26:18.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 800511 8.91 MB/sec execute 363 sec latency 19.834 ms 2023-07-29T16:26:19.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:26:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:19.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:19.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:19.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:18 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:26:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:18 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:26:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:19.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:19.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:18 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:19.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 802637 8.92 MB/sec execute 364 sec latency 14.016 ms 2023-07-29T16:26:20.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:19 smithi039 ceph-mon[128858]: pgmap v697: 97 pgs: 97 active+clean; 150 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:26:20.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:19 smithi156 ceph-mon[129245]: pgmap v697: 97 pgs: 97 active+clean; 150 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:26:20.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:19 smithi031 ceph-mon[95504]: pgmap v697: 97 pgs: 97 active+clean; 150 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:26:20.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 804051 8.91 MB/sec execute 365 sec latency 13.725 ms 2023-07-29T16:26:21.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 806189 8.92 MB/sec execute 366 sec latency 16.967 ms 2023-07-29T16:26:22.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:22 smithi156 ceph-mon[129245]: pgmap v698: 97 pgs: 97 active+clean; 150 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 2.1 MiB/s wr, 239 op/s 2023-07-29T16:26:22.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 807364 8.91 MB/sec execute 367 sec latency 53.875 ms 2023-07-29T16:26:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:22 smithi031 ceph-mon[95504]: pgmap v698: 97 pgs: 97 active+clean; 150 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 2.1 MiB/s wr, 239 op/s 2023-07-29T16:26:23.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:22 smithi039 ceph-mon[128858]: pgmap v698: 97 pgs: 97 active+clean; 150 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 2.1 MiB/s wr, 239 op/s 2023-07-29T16:26:23.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 809587 8.92 MB/sec execute 368 sec latency 17.966 ms 2023-07-29T16:26:24.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:24 smithi156 ceph-mon[129245]: pgmap v699: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:26:24.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 810680 8.91 MB/sec execute 369 sec latency 56.858 ms 2023-07-29T16:26:24.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:24 smithi031 ceph-mon[95504]: pgmap v699: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:26:25.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:24 smithi039 ceph-mon[128858]: pgmap v699: 97 pgs: 97 active+clean; 152 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2023-07-29T16:26:25.889 INFO:tasks.workunit.client.0.smithi031.stdout: 1 812889 8.92 MB/sec execute 370 sec latency 33.182 ms 2023-07-29T16:26:26.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:26 smithi156 ceph-mon[129245]: pgmap v700: 97 pgs: 97 active+clean; 156 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.4 MiB/s wr, 260 op/s 2023-07-29T16:26:26.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 814034 8.91 MB/sec execute 371 sec latency 30.110 ms 2023-07-29T16:26:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:26 smithi031 ceph-mon[95504]: pgmap v700: 97 pgs: 97 active+clean; 156 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.4 MiB/s wr, 260 op/s 2023-07-29T16:26:27.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:26 smithi039 ceph-mon[128858]: pgmap v700: 97 pgs: 97 active+clean; 156 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.4 MiB/s wr, 260 op/s 2023-07-29T16:26:27.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 816031 8.92 MB/sec execute 372 sec latency 66.723 ms 2023-07-29T16:26:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:27 smithi039 ceph-mon[128858]: pgmap v701: 97 pgs: 97 active+clean; 157 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 2.0 MiB/s wr, 242 op/s 2023-07-29T16:26:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:27 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:26:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:27 smithi156 ceph-mon[129245]: pgmap v701: 97 pgs: 97 active+clean; 157 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 2.0 MiB/s wr, 242 op/s 2023-07-29T16:26:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:27 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:26:28.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:27 smithi031 ceph-mon[95504]: pgmap v701: 97 pgs: 97 active+clean; 157 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 2.0 MiB/s wr, 242 op/s 2023-07-29T16:26:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:27 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:26:28.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 -- 172.21.15.31:0/4090945438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 msgr2=0x7f34e4073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/4090945438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e4073c20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f34d40098f0 tx=0x7f34d4009c00 comp rx=0 tx=0).stop 2023-07-29T16:26:28.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 -- 172.21.15.31:0/4090945438 shutdown_connections 2023-07-29T16:26:28.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/4090945438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34e410c920 0x7f34e410ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/4090945438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 0x7f34e40731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/4090945438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e4073c20 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 -- 172.21.15.31:0/4090945438 >> 172.21.15.31:0/4090945438 conn(0x7f34e40fb970 msgr2=0x7f34e40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:28.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 -- 172.21.15.31:0/4090945438 shutdown_connections 2023-07-29T16:26:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.576+0000 7f34eb7ff700 1 -- 172.21.15.31:0/4090945438 wait complete. 2023-07-29T16:26:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.577+0000 7f34eb7ff700 1 Processor -- start 2023-07-29T16:26:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.577+0000 7f34eb7ff700 1 -- start start 2023-07-29T16:26:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34eb7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34eb7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 0x7f34e419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34eb7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34e410c920 0x7f34e419dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34eb7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e419e290 con 0x7f34e4074d90 2023-07-29T16:26:28.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34eb7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e419e400 con 0x7f34e40737b0 2023-07-29T16:26:28.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34eb7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e419e5c0 con 0x7f34e410c920 2023-07-29T16:26:28.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e959b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e959b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56804/0 (socket says 172.21.15.31:56804) 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e959b700 1 -- 172.21.15.31:0/1444410764 learned_addr learned my addr 172.21.15.31:0/1444410764 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34e410c920 0x7f34e419dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 0x7f34e419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e959b700 1 -- 172.21.15.31:0/1444410764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34e410c920 msgr2=0x7f34e419dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e959b700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34e410c920 0x7f34e419dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.578+0000 7f34e959b700 1 -- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 msgr2=0x7f34e419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34e959b700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 0x7f34e419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34e959b700 1 -- 172.21.15.31:0/1444410764 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e00095f0 con 0x7f34e40737b0 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 0x7f34e419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34e959b700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e419c870 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f34e000ec70 tx=0x7f34e000ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34da7fc700 1 -- 172.21.15.31:0/1444410764 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e00099d0 con 0x7f34e40737b0 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34da7fc700 1 -- 172.21.15.31:0/1444410764 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34e00044e0 con 0x7f34e40737b0 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34da7fc700 1 -- 172.21.15.31:0/1444410764 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0010430 con 0x7f34e40737b0 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4009520 con 0x7f34e40737b0 2023-07-29T16:26:28.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.579+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e41a4e80 con 0x7f34e40737b0 2023-07-29T16:26:28.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.584+0000 7f34da7fc700 1 -- 172.21.15.31:0/1444410764 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f34e0003680 con 0x7f34e40737b0 2023-07-29T16:26:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.585+0000 7f34ff033b80 1 Processor -- start 2023-07-29T16:26:28.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 -- start start 2023-07-29T16:26:28.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56523172c6b0 0x5652318d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318d17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318d2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652318289b0 con 0x56523172c6b0 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x565231826370 con 0x5652318d1cf0 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34ff033b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x565231821c60 con 0x5652318d0df0 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.586+0000 7f34da7fc700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f34d00a0810 0x7f34d00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318d2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318d2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56806/0 (socket says 172.21.15.31:56806) 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 -- 172.21.15.31:0/1850581595 learned_addr learned my addr 172.21.15.31:0/1850581595 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56523172c6b0 0x5652318d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34da7fc700 1 -- 172.21.15.31:0/1444410764 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f34e0014070 con 0x7f34e40737b0 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e959b700 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318d17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f34d00a0810 0x7f34d00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 -- 172.21.15.31:0/1850581595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 msgr2=0x5652318d17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318d17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 -- 172.21.15.31:0/1850581595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56523172c6b0 msgr2=0x5652318d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56523172c6b0 0x5652318d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 -- 172.21.15.31:0/1850581595 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56523164fd80 con 0x5652318d1cf0 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.587+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318d2160 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f34e4074ab0 tx=0x7f34d400bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56c7799128e2471 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34cf7fe700 1 -- 172.21.15.31:0/1850581595 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d401c040 con 0x5652318d1cf0 2023-07-29T16:26:28.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34cf7fe700 1 -- 172.21.15.31:0/1850581595 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34d400f760 con 0x5652318d1cf0 2023-07-29T16:26:28.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34cf7fe700 1 -- 172.21.15.31:0/1850581595 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d40046d0 con 0x5652318d1cf0 2023-07-29T16:26:28.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f34d00a0810 0x7f34d00a2cc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009ec0 tx=0x7f34dc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:28.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 -- 172.21.15.31:0/1850581595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 msgr2=0x5652318d2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318d2160 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f34e4074ab0 tx=0x7f34d400bef0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 -- 172.21.15.31:0/1850581595 shutdown_connections 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318d2160 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56523172c6b0 0x5652318d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1850581595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318d17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.588+0000 7f34ff033b80 1 -- 172.21.15.31:0/1850581595 >> 172.21.15.31:0/1850581595 conn(0x5652318c5fd0 msgr2=0x5652318cb300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.589+0000 7f34ff033b80 1 -- 172.21.15.31:0/1850581595 shutdown_connections 2023-07-29T16:26:28.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.589+0000 7f34ff033b80 1 -- 172.21.15.31:0/1850581595 wait complete. 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.589+0000 7f34ff033b80 1 -- start start 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.589+0000 7f34ff033b80 1 Processor -- start 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d0df0 0x5652318da3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318e13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652317be4b0 con 0x5652318d0df0 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652317be150 con 0x5652318d1cf0 2023-07-29T16:26:28.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652316a58c0 con 0x56523172c6b0 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34ff033b80 10 client.0 _ll_register_callbacks cb 0x5652317c75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318e13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e8d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d0df0 0x5652318da3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318e13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56816/0 (socket says 172.21.15.31:56816) 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 -- 172.21.15.31:0/1727739844 learned_addr learned my addr 172.21.15.31:0/1727739844 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e959b700 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 -- 172.21.15.31:0/1727739844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 msgr2=0x5652318d9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 -- 172.21.15.31:0/1727739844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d0df0 msgr2=0x5652318da3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.590+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d0df0 0x5652318da3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34e9d9c700 1 -- 172.21.15.31:0/1727739844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e410f6b0 con 0x5652318d1cf0 2023-07-29T16:26:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318e13f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f34dc019b00 tx=0x7f34dc004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c30eb9a36b436aaf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ccff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ccff9700 1 -- 172.21.15.31:0/1727739844 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc00e070 con 0x5652318d1cf0 2023-07-29T16:26:28.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ccff9700 1 -- 172.21.15.31:0/1727739844 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f34dc019e10 con 0x5652318d1cf0 2023-07-29T16:26:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56523184ccb0 con 0x5652318d1cf0 2023-07-29T16:26:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ccff9700 1 -- 172.21.15.31:0/1727739844 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34dc0192c0 con 0x5652318d1cf0 2023-07-29T16:26:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ff033b80 10 client.28802 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.591+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x565231638c00 con 0x5652318d1cf0 2023-07-29T16:26:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ccff9700 1 -- 172.21.15.31:0/1727739844 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34dc032020 con 0x5652318d1cf0 2023-07-29T16:26:28.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ccff9700 1 -- 172.21.15.31:0/1727739844 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f34dc024990 con 0x5652318d1cf0 2023-07-29T16:26:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ff033b80 10 client.28802 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ff033b80 10 client.28802 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:28.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ff033b80 20 client.28802 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652318e5d10 0x5652318e8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ff033b80 4 client.28802 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56523189fd10 con 0x5652318e5d10 2023-07-29T16:26:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.592+0000 7f34e959b700 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652318e5d10 0x5652318e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.593+0000 7f34e959b700 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652318e5d10 0x5652318e8120 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89fb026bd4ce0ea6 server_cookie=1579fe5ea26e3a7f in_seq=0 out_seq=0 2023-07-29T16:26:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.593+0000 7f34ccff9700 10 client.28802 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.593+0000 7f34ccff9700 1 -- 172.21.15.31:0/1727739844 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56523189fd10 con 0x5652318e5d10 2023-07-29T16:26:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ccff9700 10 client.28802 handle_command_reply: tid=0 2023-07-29T16:26:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 client.28802 shutdown 2023-07-29T16:26:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 msgr2=0x5652318e13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318e13f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f34dc019b00 tx=0x7f34dc004ad0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 shutdown_connections 2023-07-29T16:26:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5652318e5d10 0x5652318e8120 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d1cf0 0x5652318e13f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ccff9700 0 client.28802 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 --2- 172.21.15.31:0/1727739844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d0df0 0x5652318da3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 >> 172.21.15.31:0/1727739844 conn(0x5652318c5fd0 msgr2=0x5652318caa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 shutdown_connections 2023-07-29T16:26:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 1 -- 172.21.15.31:0/1727739844 wait complete. 2023-07-29T16:26:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.594+0000 7f34ff033b80 20 client.28802 trim_cache size 0 max 16384 2023-07-29T16:26:28.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 Processor -- start 2023-07-29T16:26:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 -- start start 2023-07-29T16:26:28.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d0df0 0x5652318e13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318d5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652318cc5f0 con 0x5652318d1cf0 2023-07-29T16:26:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652318d9320 con 0x5652318d0df0 2023-07-29T16:26:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34ff033b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652318cb880 con 0x56523172c6b0 2023-07-29T16:26:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34e959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34e8d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d0df0 0x5652318e13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34e959b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44182/0 (socket says 172.21.15.31:44182) 2023-07-29T16:26:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34e959b700 1 -- 172.21.15.31:0/4211611676 learned_addr learned my addr 172.21.15.31:0/4211611676 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.611+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e959b700 1 -- 172.21.15.31:0/4211611676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d0df0 msgr2=0x5652318e13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e959b700 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d0df0 0x5652318e13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e959b700 1 -- 172.21.15.31:0/4211611676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 msgr2=0x5652318d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e959b700 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e959b700 1 -- 172.21.15.31:0/4211611676 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652316761f0 con 0x56523172c6b0 2023-07-29T16:26:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34e959b700 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d4b00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f34e00c9b20 tx=0x7f34e009c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f18dc9a96142be4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34cffff700 1 -- 172.21.15.31:0/4211611676 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e00a1bd0 con 0x56523172c6b0 2023-07-29T16:26:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34cffff700 1 -- 172.21.15.31:0/4211611676 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34e00b73e0 con 0x56523172c6b0 2023-07-29T16:26:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.612+0000 7f34cffff700 1 -- 172.21.15.31:0/4211611676 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e009abe0 con 0x56523172c6b0 2023-07-29T16:26:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 -- 172.21.15.31:0/4211611676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 msgr2=0x5652318d4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d4b00 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f34e00c9b20 tx=0x7f34e009c040 comp rx=0 tx=0).stop 2023-07-29T16:26:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 -- 172.21.15.31:0/4211611676 shutdown_connections 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56523172c6b0 0x5652318d4b00 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 --2- 172.21.15.31:0/4211611676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5652318d0df0 0x5652318e13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 -- 172.21.15.31:0/4211611676 >> 172.21.15.31:0/4211611676 conn(0x5652318c5fd0 msgr2=0x5652318cb170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 -- 172.21.15.31:0/4211611676 shutdown_connections 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.613+0000 7f34ff033b80 1 -- 172.21.15.31:0/4211611676 wait complete. 2023-07-29T16:26:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 -- start start 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 Processor -- start 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56523172c6b0 0x5652318e1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318ddd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318de270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652318de980 con 0x5652318d1cf0 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x565231937850 con 0x56523172c6b0 2023-07-29T16:26:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.614+0000 7f34ff033b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x565231937a10 con 0x5652318d0df0 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34ff033b80 10 client.0 _ll_register_callbacks cb 0x5652317c75c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318ddd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e9d9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318de270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318ddd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44188/0 (socket says 172.21.15.31:44188) 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e959b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56523172c6b0 0x5652318e1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 -- 172.21.15.31:0/2394673275 learned_addr learned my addr 172.21.15.31:0/2394673275 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 -- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56523172c6b0 msgr2=0x5652318e1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56523172c6b0 0x5652318e1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 -- 172.21.15.31:0/2394673275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 msgr2=0x5652318de270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318de270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 -- 172.21.15.31:0/2394673275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e00b3040 con 0x5652318d0df0 2023-07-29T16:26:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e9d9c700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318de270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:28.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e959b700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56523172c6b0 0x5652318e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34e8d9a700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318ddd30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f34d4025040 tx=0x7f34d400b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af03e690961f57fb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.615+0000 7f34ccff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ccff9700 1 -- 172.21.15.31:0/2394673275 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d401c040 con 0x5652318d0df0 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34d400f760 con 0x5652318d0df0 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ccff9700 1 -- 172.21.15.31:0/2394673275 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f34d400b900 con 0x5652318d0df0 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ccff9700 1 -- 172.21.15.31:0/2394673275 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34d4005b00 con 0x5652318d0df0 2023-07-29T16:26:28.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ff033b80 10 client.28810 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56523164e250 con 0x5652318d0df0 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ccff9700 1 -- 172.21.15.31:0/2394673275 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34d400f760 con 0x5652318d0df0 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.616+0000 7f34ccff9700 1 -- 172.21.15.31:0/2394673275 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f34d4015020 con 0x5652318d0df0 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ff033b80 10 client.28810 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ff033b80 10 client.28810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ff033b80 20 client.28810 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ff033b80 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x565231938ff0 0x56523193b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ff033b80 4 client.28810 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:26:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56523181ace0 con 0x565231938ff0 2023-07-29T16:26:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34e959b700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x565231938ff0 0x56523193b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34e959b700 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x565231938ff0 0x56523193b3d0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2464bdbc126e9c6 server_cookie=f173c2b1081aa3f8 in_seq=0 out_seq=0 2023-07-29T16:26:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.617+0000 7f34ccff9700 10 client.28810 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ccff9700 1 -- 172.21.15.31:0/2394673275 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56523181ace0 con 0x565231938ff0 2023-07-29T16:26:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ccff9700 10 client.28810 handle_command_reply: tid=0 2023-07-29T16:26:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 client.28810 shutdown 2023-07-29T16:26:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 msgr2=0x5652318ddd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318ddd30 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f34d4025040 tx=0x7f34d400b790 comp rx=0 tx=0).stop 2023-07-29T16:26:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 shutdown_connections 2023-07-29T16:26:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x565231938ff0 0x56523193b3d0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5652318d1cf0 0x5652318de270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56523172c6b0 0x5652318e1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ccff9700 0 client.28810 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 --2- 172.21.15.31:0/2394673275 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5652318d0df0 0x5652318ddd30 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 >> 172.21.15.31:0/2394673275 conn(0x5652318c5fd0 msgr2=0x5652318d4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 shutdown_connections 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.678+0000 7f34ff033b80 1 -- 172.21.15.31:0/2394673275 wait complete. 2023-07-29T16:26:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.679+0000 7f34ff033b80 20 client.28810 trim_cache size 0 max 16384 2023-07-29T16:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:26:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:26:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.679+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f34d00a0810 msgr2=0x7f34d00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.679+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f34d00a0810 0x7f34d00a2cc0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009ec0 tx=0x7f34dc00b040 comp rx=0 tx=0).stop 2023-07-29T16:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.679+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 msgr2=0x7f34e419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.679+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e419c870 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f34e000ec70 tx=0x7f34e000ef80 comp rx=0 tx=0).stop 2023-07-29T16:26:28.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 shutdown_connections 2023-07-29T16:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f34d00a0810 0x7f34d00a2cc0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34e410c920 0x7f34e419dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34e40737b0 0x7f34e419c870 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 --2- 172.21.15.31:0/1444410764 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f34e4074d90 0x7f34e419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 >> 172.21.15.31:0/1444410764 conn(0x7f34e40fb970 msgr2=0x7f34e40fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:28.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 shutdown_connections 2023-07-29T16:26:28.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:28.680+0000 7f34eb7ff700 1 -- 172.21.15.31:0/1444410764 wait complete. 2023-07-29T16:26:28.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 817028 8.91 MB/sec execute 373 sec latency 52.002 ms 2023-07-29T16:26:29.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:28 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:26:29.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:28 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:26:29.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:28 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:26:29.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:28 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:26:29.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:28 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:26:29.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:28 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:26:29.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 819044 8.91 MB/sec execute 374 sec latency 12.798 ms 2023-07-29T16:26:30.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:29 smithi039 ceph-mon[128858]: pgmap v702: 97 pgs: 97 active+clean; 152 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2023-07-29T16:26:30.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:29 smithi156 ceph-mon[129245]: pgmap v702: 97 pgs: 97 active+clean; 152 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2023-07-29T16:26:30.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:29 smithi031 ceph-mon[95504]: pgmap v702: 97 pgs: 97 active+clean; 152 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2023-07-29T16:26:30.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 820550 8.91 MB/sec execute 375 sec latency 12.811 ms 2023-07-29T16:26:31.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 822525 8.91 MB/sec execute 376 sec latency 15.246 ms 2023-07-29T16:26:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:32 smithi156 ceph-mon[129245]: pgmap v703: 97 pgs: 97 active+clean; 152 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.0 MiB/s wr, 201 op/s 2023-07-29T16:26:32.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 823988 8.92 MB/sec execute 377 sec latency 35.092 ms 2023-07-29T16:26:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:32 smithi031 ceph-mon[95504]: pgmap v703: 97 pgs: 97 active+clean; 152 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.0 MiB/s wr, 201 op/s 2023-07-29T16:26:33.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:32 smithi039 ceph-mon[128858]: pgmap v703: 97 pgs: 97 active+clean; 152 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.0 MiB/s wr, 201 op/s 2023-07-29T16:26:33.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 825293 8.91 MB/sec execute 378 sec latency 51.297 ms 2023-07-29T16:26:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:34 smithi156 ceph-mon[129245]: pgmap v704: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 292 op/s 2023-07-29T16:26:34.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 827391 8.92 MB/sec execute 379 sec latency 20.273 ms 2023-07-29T16:26:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:34 smithi031 ceph-mon[95504]: pgmap v704: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 292 op/s 2023-07-29T16:26:35.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:34 smithi039 ceph-mon[128858]: pgmap v704: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 292 op/s 2023-07-29T16:26:35.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 828675 8.91 MB/sec execute 380 sec latency 12.568 ms 2023-07-29T16:26:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:36 smithi156 ceph-mon[129245]: pgmap v705: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 2.6 MiB/s wr, 264 op/s 2023-07-29T16:26:36.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 830861 8.92 MB/sec execute 381 sec latency 15.189 ms 2023-07-29T16:26:36.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:36 smithi031 ceph-mon[95504]: pgmap v705: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 2.6 MiB/s wr, 264 op/s 2023-07-29T16:26:37.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:36 smithi039 ceph-mon[128858]: pgmap v705: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 2.6 MiB/s wr, 264 op/s 2023-07-29T16:26:37.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 831811 8.91 MB/sec execute 382 sec latency 41.614 ms 2023-07-29T16:26:38.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:37 smithi039 ceph-mon[128858]: pgmap v706: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2023-07-29T16:26:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:37 smithi156 ceph-mon[129245]: pgmap v706: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2023-07-29T16:26:38.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:37 smithi031 ceph-mon[95504]: pgmap v706: 97 pgs: 97 active+clean; 154 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2023-07-29T16:26:38.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 833704 8.91 MB/sec execute 383 sec latency 177.818 ms 2023-07-29T16:26:39.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 834910 8.90 MB/sec execute 384 sec latency 22.031 ms 2023-07-29T16:26:40.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:40 smithi156 ceph-mon[129245]: pgmap v707: 97 pgs: 97 active+clean; 162 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2023-07-29T16:26:40.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 837272 8.91 MB/sec execute 385 sec latency 15.001 ms 2023-07-29T16:26:40.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:40 smithi031 ceph-mon[95504]: pgmap v707: 97 pgs: 97 active+clean; 162 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2023-07-29T16:26:41.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:40 smithi039 ceph-mon[128858]: pgmap v707: 97 pgs: 97 active+clean; 162 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.0 MiB/s wr, 298 op/s 2023-07-29T16:26:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.839+0000 7f87313ff700 1 -- 172.21.15.31:0/4223631899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c074d90 msgr2=0x7f872c0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.839+0000 7f87313ff700 1 --2- 172.21.15.31:0/4223631899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c074d90 0x7f872c0731e0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f871c009990 tx=0x7f871c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 -- 172.21.15.31:0/4223631899 shutdown_connections 2023-07-29T16:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 --2- 172.21.15.31:0/4223631899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c108210 0x7f872c10a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 --2- 172.21.15.31:0/4223631899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c074d90 0x7f872c0731e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 --2- 172.21.15.31:0/4223631899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872c0737b0 0x7f872c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 -- 172.21.15.31:0/4223631899 >> 172.21.15.31:0/4223631899 conn(0x7f872c0fb970 msgr2=0x7f872c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 -- 172.21.15.31:0/4223631899 shutdown_connections 2023-07-29T16:26:41.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.840+0000 7f87313ff700 1 -- 172.21.15.31:0/4223631899 wait complete. 2023-07-29T16:26:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 Processor -- start 2023-07-29T16:26:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 -- start start 2023-07-29T16:26:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 0x7f872c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872c074d90 0x7f872c1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c108210 0x7f872c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c199d90 con 0x7f872c074d90 2023-07-29T16:26:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c199f00 con 0x7f872c108210 2023-07-29T16:26:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f87313ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872c19a0c0 con 0x7f872c0737b0 2023-07-29T16:26:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f872affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 0x7f872c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f872affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 0x7f872c198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47114/0 (socket says 172.21.15.31:47114) 2023-07-29T16:26:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f872affd700 1 -- 172.21.15.31:0/1166159228 learned_addr learned my addr 172.21.15.31:0/1166159228 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:41.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.841+0000 7f872a7fc700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872c074d90 0x7f872c1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872b7fe700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c108210 0x7f872c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872affd700 1 -- 172.21.15.31:0/1166159228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c108210 msgr2=0x7f872c199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872affd700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c108210 0x7f872c199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872affd700 1 -- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872c074d90 msgr2=0x7f872c1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872affd700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872c074d90 0x7f872c1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872affd700 1 -- 172.21.15.31:0/1166159228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724009c40 con 0x7f872c0737b0 2023-07-29T16:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f872affd700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 0x7f872c198370 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f871c005720 tx=0x7f871c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f8713fff700 1 -- 172.21.15.31:0/1166159228 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c01d070 con 0x7f872c0737b0 2023-07-29T16:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c0095f0 con 0x7f872c0737b0 2023-07-29T16:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f8713fff700 1 -- 172.21.15.31:0/1166159228 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f871c021410 con 0x7f872c0737b0 2023-07-29T16:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.842+0000 7f8713fff700 1 -- 172.21.15.31:0/1166159228 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c017650 con 0x7f872c0737b0 2023-07-29T16:26:41.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.843+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c1a0980 con 0x7f872c0737b0 2023-07-29T16:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.847+0000 7f8713fff700 1 -- 172.21.15.31:0/1166159228 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f871c021580 con 0x7f872c0737b0 2023-07-29T16:26:41.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.848+0000 7f8744d68b80 1 Processor -- start 2023-07-29T16:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.848+0000 7f8713fff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f87140a0790 0x7f87140a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.848+0000 7f8713fff700 1 -- 172.21.15.31:0/1166159228 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f871c0ccc60 con 0x7f872c0737b0 2023-07-29T16:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.848+0000 7f8744d68b80 1 -- start start 2023-07-29T16:26:41.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872b7fe700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f87140a0790 0x7f87140a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x5616479858b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647985df0 0x5616479867b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647986cf0 0x561647987160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f8744d68b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616478dd9b0 con 0x5616477e16b0 2023-07-29T16:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f8744d68b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616478db370 con 0x561647986cf0 2023-07-29T16:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f8744d68b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616478d6c60 con 0x561647985df0 2023-07-29T16:26:41.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x5616479858b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872b7fe700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f87140a0790 0x7f87140a2c40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8724009fd0 tx=0x7f872400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x5616479858b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48242/0 (socket says 172.21.15.31:48242) 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872a7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647985df0 0x5616479867b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872affd700 1 -- 172.21.15.31:0/3772702486 learned_addr learned my addr 172.21.15.31:0/3772702486 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.849+0000 7f872b7fe700 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647986cf0 0x561647987160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872a7fc700 1 -- 172.21.15.31:0/3772702486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647986cf0 msgr2=0x561647987160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872a7fc700 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647986cf0 0x561647987160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872a7fc700 1 -- 172.21.15.31:0/3772702486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 msgr2=0x5616479858b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872a7fc700 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x5616479858b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872a7fc700 1 -- 172.21.15.31:0/3772702486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561647704d80 con 0x561647985df0 2023-07-29T16:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872affd700 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x5616479858b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f872a7fc700 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647985df0 0x5616479867b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8720009c80 tx=0x7f8720009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7592ecc22ba1b892 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:41.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f87117fa700 1 -- 172.21.15.31:0/3772702486 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720011cb0 con 0x561647985df0 2023-07-29T16:26:41.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f87117fa700 1 -- 172.21.15.31:0/3772702486 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8720011e10 con 0x561647985df0 2023-07-29T16:26:41.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.850+0000 7f87117fa700 1 -- 172.21.15.31:0/3772702486 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720005280 con 0x561647985df0 2023-07-29T16:26:41.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 -- 172.21.15.31:0/3772702486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647985df0 msgr2=0x5616479867b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647985df0 0x5616479867b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f8720009c80 tx=0x7f8720009f90 comp rx=0 tx=0).stop 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 -- 172.21.15.31:0/3772702486 shutdown_connections 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647986cf0 0x561647987160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x5616479858b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3772702486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647985df0 0x5616479867b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 -- 172.21.15.31:0/3772702486 >> 172.21.15.31:0/3772702486 conn(0x56164797afd0 msgr2=0x561647980300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:41.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 -- 172.21.15.31:0/3772702486 shutdown_connections 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.851+0000 7f8744d68b80 1 -- 172.21.15.31:0/3772702486 wait complete. 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.852+0000 7f8744d68b80 1 -- start start 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.852+0000 7f8744d68b80 1 Processor -- start 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x561647997710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x561647997c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x56164799ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616478734b0 con 0x5616477e16b0 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561647873150 con 0x561647985df0 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164775a8c0 con 0x561647986cf0 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f8744d68b80 10 client.0 _ll_register_callbacks cb 0x56164787c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f872a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x561647997c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f872affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x561647997710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f872a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x561647997c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50892/0 (socket says 172.21.15.31:50892) 2023-07-29T16:26:41.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f872a7fc700 1 -- 172.21.15.31:0/3130196967 learned_addr learned my addr 172.21.15.31:0/3130196967 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.853+0000 7f872b7fe700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x56164799ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872a7fc700 1 -- 172.21.15.31:0/3130196967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 msgr2=0x56164799ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872a7fc700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x56164799ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872a7fc700 1 -- 172.21.15.31:0/3130196967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 msgr2=0x561647997710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872a7fc700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x561647997710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872a7fc700 1 -- 172.21.15.31:0/3130196967 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872c1a0980 con 0x561647985df0 2023-07-29T16:26:41.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872affd700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x561647997710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f872a7fc700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x561647997c50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8720005c50 tx=0x7f8720011b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aac52b8742476661 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f870eff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f870eff5700 1 -- 172.21.15.31:0/3130196967 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8720005080 con 0x561647985df0 2023-07-29T16:26:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561647901cb0 con 0x561647985df0 2023-07-29T16:26:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.854+0000 7f870eff5700 1 -- 172.21.15.31:0/3130196967 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f87200051e0 con 0x561647985df0 2023-07-29T16:26:41.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f870eff5700 1 -- 172.21.15.31:0/3130196967 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87200053e0 con 0x561647985df0 2023-07-29T16:26:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 10 client.28820 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f87200098e0 con 0x561647985df0 2023-07-29T16:26:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f870eff5700 1 -- 172.21.15.31:0/3130196967 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561647901cb0 con 0x561647985df0 2023-07-29T16:26:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f870eff5700 1 -- 172.21.15.31:0/3130196967 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f8720022400 con 0x561647985df0 2023-07-29T16:26:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 10 client.28820 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:41.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 10 client.28820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 20 client.28820 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479a3350 0x5616479a5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 4 client.28820 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.855+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561647954d10 con 0x5616479a3350 2023-07-29T16:26:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.856+0000 7f872affd700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479a3350 0x5616479a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.856+0000 7f870eff5700 10 client.28820 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.856+0000 7f872affd700 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479a3350 0x5616479a5760 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab043b9799ba7ce7 server_cookie=48d97647d664187d in_seq=0 out_seq=0 2023-07-29T16:26:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f870eff5700 1 -- 172.21.15.31:0/3130196967 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561647954d10 con 0x5616479a3350 2023-07-29T16:26:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f870eff5700 10 client.28820 handle_command_reply: tid=0 2023-07-29T16:26:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 client.28820 shutdown 2023-07-29T16:26:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 msgr2=0x561647997c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x561647997c50 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f8720005c50 tx=0x7f8720011b40 comp rx=0 tx=0).stop 2023-07-29T16:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 shutdown_connections 2023-07-29T16:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479a3350 0x5616479a5760 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x56164799ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616477e16b0 0x561647997710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 --2- 172.21.15.31:0/3130196967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x561647997c50 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f870eff5700 0 client.28820 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.857+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 >> 172.21.15.31:0/3130196967 conn(0x56164797afd0 msgr2=0x56164797fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.858+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 shutdown_connections 2023-07-29T16:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.858+0000 7f8744d68b80 1 -- 172.21.15.31:0/3130196967 wait complete. 2023-07-29T16:26:41.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.858+0000 7f8744d68b80 20 client.28820 trim_cache size 0 max 16384 2023-07-29T16:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 Processor -- start 2023-07-29T16:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 -- start start 2023-07-29T16:26:41.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 0x5616479a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647985df0 0x56164799ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x5616479964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561647997280 con 0x561647985df0 2023-07-29T16:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561647980df0 con 0x5616477e16b0 2023-07-29T16:26:41.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f8744d68b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164798da00 con 0x561647986cf0 2023-07-29T16:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.873+0000 7f872affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 0x5616479a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872a7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647985df0 0x56164799ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 0x5616479a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50894/0 (socket says 172.21.15.31:50894) 2023-07-29T16:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 -- 172.21.15.31:0/1539073463 learned_addr learned my addr 172.21.15.31:0/1539073463 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872b7fe700 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x5616479964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 -- 172.21.15.31:0/1539073463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 msgr2=0x5616479964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x5616479964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 -- 172.21.15.31:0/1539073463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647985df0 msgr2=0x56164799ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647985df0 0x56164799ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 -- 172.21.15.31:0/1539073463 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616476edc00 con 0x5616477e16b0 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872b7fe700 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x5616479964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872a7fc700 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647985df0 0x56164799ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f872affd700 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 0x5616479a5730 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f871c096db0 tx=0x7f871c0cc040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c32812016acf0ff5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:41.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f8711ffb700 1 -- 172.21.15.31:0/1539073463 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c09e3c0 con 0x5616477e16b0 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f8711ffb700 1 -- 172.21.15.31:0/1539073463 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f871c0a2750 con 0x5616477e16b0 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.874+0000 7f8711ffb700 1 -- 172.21.15.31:0/1539073463 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c09a850 con 0x5616477e16b0 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 -- 172.21.15.31:0/1539073463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 msgr2=0x5616479a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 0x5616479a5730 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f871c096db0 tx=0x7f871c0cc040 comp rx=0 tx=0).stop 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 -- 172.21.15.31:0/1539073463 shutdown_connections 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561647986cf0 0x5616479964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5616477e16b0 0x5616479a5730 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 --2- 172.21.15.31:0/1539073463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647985df0 0x56164799ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 -- 172.21.15.31:0/1539073463 >> 172.21.15.31:0/1539073463 conn(0x56164797afd0 msgr2=0x56164797fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 -- 172.21.15.31:0/1539073463 shutdown_connections 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.875+0000 7f8744d68b80 1 -- 172.21.15.31:0/1539073463 wait complete. 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 -- start start 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 Processor -- start 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616477e16b0 0x56164799a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x56164799a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647986cf0 0x56164799d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164799dce0 con 0x561647986cf0 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164799de50 con 0x561647985df0 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56164799e010 con 0x5616477e16b0 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f8744d68b80 10 client.0 _ll_register_callbacks cb 0x56164787c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f872a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x56164799a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f872affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616477e16b0 0x56164799a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f872b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647986cf0 0x56164799d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f872a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x56164799a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50910/0 (socket says 172.21.15.31:50910) 2023-07-29T16:26:41.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.876+0000 7f872a7fc700 1 -- 172.21.15.31:0/673163203 learned_addr learned my addr 172.21.15.31:0/673163203 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:41.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872affd700 1 -- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 msgr2=0x56164799a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872affd700 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x56164799a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872affd700 1 -- 172.21.15.31:0/673163203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647986cf0 msgr2=0x56164799d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872affd700 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647986cf0 0x56164799d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872affd700 1 -- 172.21.15.31:0/673163203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87200098e0 con 0x5616477e16b0 2023-07-29T16:26:41.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872a7fc700 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x56164799a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f872affd700 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616477e16b0 0x56164799a290 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f871c00b520 tx=0x7f871c09e1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d586680d001c2645 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f870eff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f870eff5700 1 -- 172.21.15.31:0/673163203 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871c099a80 con 0x5616477e16b0 2023-07-29T16:26:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f870eff5700 1 -- 172.21.15.31:0/673163203 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f871c09a5f0 con 0x5616477e16b0 2023-07-29T16:26:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.877+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8720011e10 con 0x5616477e16b0 2023-07-29T16:26:41.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f870eff5700 1 -- 172.21.15.31:0/673163203 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f871c09a7d0 con 0x5616477e16b0 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 10 client.28828 fetch_fsmap learned FSMap version 31 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f871c0095f0 con 0x5616477e16b0 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f870eff5700 1 -- 172.21.15.31:0/673163203 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8720011e10 con 0x5616477e16b0 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f870eff5700 1 -- 172.21.15.31:0/673163203 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 31) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f871c00fac0 con 0x5616477e16b0 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 10 client.28828 fetch_fsmap finished waiting for FSMap version 31 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 10 client.28828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:41.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 20 client.28828 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:41.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479f5340 0x5616479f7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:41.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 4 client.28828 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:26:41.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5616478cfce0 con 0x5616479f5340 2023-07-29T16:26:41.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.878+0000 7f872a7fc700 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479f5340 0x5616479f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:41.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.879+0000 7f872a7fc700 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479f5340 0x5616479f7750 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=515c02a999f6db8c server_cookie=b164c84659a0bb08 in_seq=0 out_seq=0 2023-07-29T16:26:41.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.879+0000 7f870eff5700 10 client.28828 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:41.890 INFO:tasks.workunit.client.0.smithi031.stdout: 1 838459 8.91 MB/sec execute 386 sec latency 21.105 ms 2023-07-29T16:26:41.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f870eff5700 1 -- 172.21.15.31:0/673163203 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5616478cfce0 con 0x5616479f5340 2023-07-29T16:26:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f870eff5700 10 client.28828 handle_command_reply: tid=0 2023-07-29T16:26:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 client.28828 shutdown 2023-07-29T16:26:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616477e16b0 msgr2=0x56164799a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616477e16b0 0x56164799a290 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f871c00b520 tx=0x7f871c09e1b0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 shutdown_connections 2023-07-29T16:26:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5616479f5340 0x5616479f7750 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561647986cf0 0x56164799d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5616477e16b0 0x56164799a290 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f870eff5700 0 client.28828 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 --2- 172.21.15.31:0/673163203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561647985df0 0x56164799a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 >> 172.21.15.31:0/673163203 conn(0x56164797afd0 msgr2=0x56164798cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 shutdown_connections 2023-07-29T16:26:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.967+0000 7f8744d68b80 1 -- 172.21.15.31:0/673163203 wait complete. 2023-07-29T16:26:41.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f8744d68b80 20 client.28828 trim_cache size 0 max 16384 2023-07-29T16:26:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:26:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:26:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:26:41.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:26:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f87140a0790 msgr2=0x7f87140a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f87140a0790 0x7f87140a2c40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8724009fd0 tx=0x7f872400b040 comp rx=0 tx=0).stop 2023-07-29T16:26:41.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 msgr2=0x7f872c198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 0x7f872c198370 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f871c005720 tx=0x7f871c009580 comp rx=0 tx=0).stop 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 shutdown_connections 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f87140a0790 0x7f87140a2c40 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f872c108210 0x7f872c199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f872c0737b0 0x7f872c198370 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 --2- 172.21.15.31:0/1166159228 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f872c074d90 0x7f872c1988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:41.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.968+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 >> 172.21.15.31:0/1166159228 conn(0x7f872c0fb970 msgr2=0x7f872c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.969+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 shutdown_connections 2023-07-29T16:26:41.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:41.969+0000 7f87313ff700 1 -- 172.21.15.31:0/1166159228 wait complete. 2023-07-29T16:26:42.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:42 smithi156 ceph-mon[129245]: pgmap v708: 97 pgs: 97 active+clean; 162 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.0 MiB/s wr, 209 op/s 2023-07-29T16:26:42.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 840489 8.91 MB/sec execute 387 sec latency 48.155 ms 2023-07-29T16:26:42.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:42 smithi031 ceph-mon[95504]: pgmap v708: 97 pgs: 97 active+clean; 162 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.0 MiB/s wr, 209 op/s 2023-07-29T16:26:43.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:42 smithi039 ceph-mon[128858]: pgmap v708: 97 pgs: 97 active+clean; 162 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.0 MiB/s wr, 209 op/s 2023-07-29T16:26:43.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 841776 8.91 MB/sec execute 388 sec latency 81.380 ms 2023-07-29T16:26:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:44 smithi156 ceph-mon[129245]: pgmap v709: 97 pgs: 97 active+clean; 158 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 2.5 MiB/s wr, 288 op/s 2023-07-29T16:26:44.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 843404 8.90 MB/sec execute 389 sec latency 17.095 ms 2023-07-29T16:26:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:44 smithi031 ceph-mon[95504]: pgmap v709: 97 pgs: 97 active+clean; 158 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 2.5 MiB/s wr, 288 op/s 2023-07-29T16:26:45.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:44 smithi039 ceph-mon[128858]: pgmap v709: 97 pgs: 97 active+clean; 158 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 2.5 MiB/s wr, 288 op/s 2023-07-29T16:26:45.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 845289 8.91 MB/sec execute 390 sec latency 12.936 ms 2023-07-29T16:26:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:46 smithi156 ceph-mon[129245]: pgmap v710: 97 pgs: 97 active+clean; 160 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 2.4 MiB/s wr, 243 op/s 2023-07-29T16:26:46.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 846579 8.91 MB/sec execute 391 sec latency 48.314 ms 2023-07-29T16:26:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:46 smithi031 ceph-mon[95504]: pgmap v710: 97 pgs: 97 active+clean; 160 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 2.4 MiB/s wr, 243 op/s 2023-07-29T16:26:47.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:46 smithi039 ceph-mon[128858]: pgmap v710: 97 pgs: 97 active+clean; 160 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 2.4 MiB/s wr, 243 op/s 2023-07-29T16:26:47.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 848722 8.92 MB/sec execute 392 sec latency 33.731 ms 2023-07-29T16:26:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:47 smithi039 ceph-mon[128858]: pgmap v711: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-07-29T16:26:48.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:47 smithi156 ceph-mon[129245]: pgmap v711: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-07-29T16:26:48.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:47 smithi031 ceph-mon[95504]: pgmap v711: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 2.4 MiB/s wr, 273 op/s 2023-07-29T16:26:48.581 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:26:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 -- 172.21.15.31:0/2041082520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec073730 msgr2=0x7f61ec073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 --2- 172.21.15.31:0/2041082520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec073730 0x7f61ec073ba0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009990 tx=0x7f61e0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 -- 172.21.15.31:0/2041082520 shutdown_connections 2023-07-29T16:26:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 --2- 172.21.15.31:0/2041082520 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61ec1081e0 0x7f61ec10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 --2- 172.21.15.31:0/2041082520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 --2- 172.21.15.31:0/2041082520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec073730 0x7f61ec073ba0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 -- 172.21.15.31:0/2041082520 >> 172.21.15.31:0/2041082520 conn(0x7f61ec0fb970 msgr2=0x7f61ec0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 -- 172.21.15.31:0/2041082520 shutdown_connections 2023-07-29T16:26:48.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.796+0000 7f61f1dff700 1 -- 172.21.15.31:0/2041082520 wait complete. 2023-07-29T16:26:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.797+0000 7f61f1dff700 1 Processor -- start 2023-07-29T16:26:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 -- start start 2023-07-29T16:26:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61ec073730 0x7f61ec198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 0x7f61ec19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec19fe00 con 0x7f61ec073730 2023-07-29T16:26:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec19ff40 con 0x7f61ec1081e0 2023-07-29T16:26:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61f1dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec1a00d0 con 0x7f61ec074d10 2023-07-29T16:26:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61eaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 0x7f61ec19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.798+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61ec073730 0x7f61ec198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 0x7f61ec19f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50932/0 (socket says 172.21.15.31:50932) 2023-07-29T16:26:48.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61eaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47184/0 (socket says 172.21.15.31:47184) 2023-07-29T16:26:48.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 -- 172.21.15.31:0/3902343987 learned_addr learned my addr 172.21.15.31:0/3902343987 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 -- 172.21.15.31:0/3902343987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 msgr2=0x7f61ec1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 -- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61ec073730 msgr2=0x7f61ec198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61ec073730 0x7f61ec198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 -- 172.21.15.31:0/3902343987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e00095f0 con 0x7f61ec1081e0 2023-07-29T16:26:48.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61eaffd700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:48.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61ebfff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 0x7f61ec19f830 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f61e400ef30 tx=0x7f61e400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61e8ff9700 1 -- 172.21.15.31:0/3902343987 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4018070 con 0x7f61ec1081e0 2023-07-29T16:26:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61e8ff9700 1 -- 172.21.15.31:0/3902343987 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61e400f040 con 0x7f61ec1081e0 2023-07-29T16:26:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61e8ff9700 1 -- 172.21.15.31:0/3902343987 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e400cd00 con 0x7f61ec1081e0 2023-07-29T16:26:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.799+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec1a0350 con 0x7f61ec1081e0 2023-07-29T16:26:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.800+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec1a0810 con 0x7f61ec1081e0 2023-07-29T16:26:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.803+0000 7f61e8ff9700 1 -- 172.21.15.31:0/3902343987 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f61e4003a00 con 0x7f61ec1081e0 2023-07-29T16:26:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.804+0000 7f6205732b80 1 Processor -- start 2023-07-29T16:26:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.804+0000 7f6205732b80 1 -- start start 2023-07-29T16:26:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.804+0000 7f61e8ff9700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61d40a0790 0x7f61d40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61e8ff9700 1 -- 172.21.15.31:0/3902343987 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f61e40cb5c0 con 0x7f61ec1081e0 2023-07-29T16:26:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f6205732b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f6205732b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f6205732b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b42f0 0x555fc36b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f6205732b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc360aa10 con 0x555fc36b2be0 2023-07-29T16:26:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f6205732b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc36083d0 con 0x555fc36b42f0 2023-07-29T16:26:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f6205732b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc3603cc0 con 0x555fc36b3530 2023-07-29T16:26:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eaffd700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61d40a0790 0x7f61d40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48286/0 (socket says 172.21.15.31:48286) 2023-07-29T16:26:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 -- 172.21.15.31:0/4268549844 learned_addr learned my addr 172.21.15.31:0/4268549844 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eaffd700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61ebfff700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b42f0 0x555fc36b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 -- 172.21.15.31:0/4268549844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 msgr2=0x555fc36b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 -- 172.21.15.31:0/4268549844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b42f0 msgr2=0x555fc36b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.805+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b42f0 0x555fc36b4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61eb7fe700 1 -- 172.21.15.31:0/4268549844 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fc3431de0 con 0x555fc36b2be0 2023-07-29T16:26:48.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61eaffd700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:48.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61ebfff700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b42f0 0x555fc36b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b2ff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009d80 tx=0x7f61dc011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f878f7ca39c8110b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61eaffd700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61d40a0790 0x7f61d40a2c40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x555fc36b3940 tx=0x7f61e001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61d1ffb700 1 -- 172.21.15.31:0/4268549844 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc004230 con 0x555fc36b2be0 2023-07-29T16:26:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61d1ffb700 1 -- 172.21.15.31:0/4268549844 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61dc004390 con 0x555fc36b2be0 2023-07-29T16:26:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.806+0000 7f61d1ffb700 1 -- 172.21.15.31:0/4268549844 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc019860 con 0x555fc36b2be0 2023-07-29T16:26:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 -- 172.21.15.31:0/4268549844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 msgr2=0x555fc36b2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b2ff0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009d80 tx=0x7f61dc011720 comp rx=0 tx=0).stop 2023-07-29T16:26:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 -- 172.21.15.31:0/4268549844 shutdown_connections 2023-07-29T16:26:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b42f0 0x555fc36b4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b2ff0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 --2- 172.21.15.31:0/4268549844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 -- 172.21.15.31:0/4268549844 >> 172.21.15.31:0/4268549844 conn(0x555fc36a83e0 msgr2=0x555fc36ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:48.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 -- 172.21.15.31:0/4268549844 shutdown_connections 2023-07-29T16:26:48.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.807+0000 7f6205732b80 1 -- 172.21.15.31:0/4268549844 wait complete. 2023-07-29T16:26:48.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 -- start start 2023-07-29T16:26:48.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 Processor -- start 2023-07-29T16:26:48.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 0x555fc36bc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b3530 0x555fc36bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b42f0 0x555fc36c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc35a0510 con 0x555fc36b3530 2023-07-29T16:26:48.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc35a01b0 con 0x555fc36b2be0 2023-07-29T16:26:48.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc3487920 con 0x555fc36b42f0 2023-07-29T16:26:48.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.808+0000 7f6205732b80 10 client.0 _ll_register_callbacks cb 0x555fc35a9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:48.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 0x555fc36bc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 0x555fc36bc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50936/0 (socket says 172.21.15.31:50936) 2023-07-29T16:26:48.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61ebfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b42f0 0x555fc36c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 -- 172.21.15.31:0/1819131223 learned_addr learned my addr 172.21.15.31:0/1819131223 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:48.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eaffd700 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b3530 0x555fc36bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 -- 172.21.15.31:0/1819131223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b42f0 msgr2=0x555fc36c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b42f0 0x555fc36c3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 -- 172.21.15.31:0/1819131223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b3530 msgr2=0x555fc36bcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b3530 0x555fc36bcd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 -- 172.21.15.31:0/1819131223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec1a0810 con 0x555fc36b2be0 2023-07-29T16:26:48.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 0x555fc36bc7c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f61dc01b040 tx=0x7f61dc0038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d07f0e694bddfe server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.809+0000 7f61caffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f61caffd700 1 -- 172.21.15.31:0/1819131223 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc005240 con 0x555fc36b2be0 2023-07-29T16:26:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555fc362ed10 con 0x555fc36b2be0 2023-07-29T16:26:48.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f61caffd700 1 -- 172.21.15.31:0/1819131223 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f61dc004500 con 0x555fc36b2be0 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f61caffd700 1 -- 172.21.15.31:0/1819131223 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61dc003bf0 con 0x555fc36b2be0 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f6205732b80 10 client.28838 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f61dc0099e0 con 0x555fc36b2be0 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f61caffd700 1 -- 172.21.15.31:0/1819131223 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555fc362ed10 con 0x555fc36b2be0 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f61caffd700 1 -- 172.21.15.31:0/1819131223 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f61dc021040 con 0x555fc36b2be0 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f6205732b80 10 client.28838 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:48.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.810+0000 7f6205732b80 10 client.28838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f6205732b80 20 client.28838 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f6205732b80 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36c84f0 0x555fc36ca900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f6205732b80 4 client.28838 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555fc3681d70 con 0x555fc36c84f0 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f61eaffd700 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36c84f0 0x555fc36ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f61eaffd700 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36c84f0 0x555fc36ca900 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb23da0870f22d1b server_cookie=384b790064bcf8b0 in_seq=0 out_seq=0 2023-07-29T16:26:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.811+0000 7f61caffd700 10 client.28838 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f61caffd700 1 -- 172.21.15.31:0/1819131223 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555fc3681d70 con 0x555fc36c84f0 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f61caffd700 10 client.28838 handle_command_reply: tid=0 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 client.28838 shutdown 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 msgr2=0x555fc36bc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 0x555fc36bc7c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f61dc01b040 tx=0x7f61dc0038b0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 shutdown_connections 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36c84f0 0x555fc36ca900 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b42f0 0x555fc36c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b2be0 0x555fc36bc7c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f61caffd700 0 client.28838 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 --2- 172.21.15.31:0/1819131223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b3530 0x555fc36bcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 >> 172.21.15.31:0/1819131223 conn(0x555fc36a83e0 msgr2=0x555fc36ad060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 shutdown_connections 2023-07-29T16:26:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 1 -- 172.21.15.31:0/1819131223 wait complete. 2023-07-29T16:26:48.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.813+0000 7f6205732b80 20 client.28838 trim_cache size 0 max 16384 2023-07-29T16:26:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.830+0000 7f6205732b80 1 Processor -- start 2023-07-29T16:26:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.830+0000 7f6205732b80 1 -- start start 2023-07-29T16:26:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f6205732b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 0x555fc36c1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f6205732b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f6205732b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f6205732b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc36af080 con 0x555fc36b2be0 2023-07-29T16:26:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f6205732b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc36b3bc0 con 0x555fc36a83e0 2023-07-29T16:26:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f6205732b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc36aa460 con 0x555fc36b3530 2023-07-29T16:26:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 0x555fc36c1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 0x555fc36c1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50940/0 (socket says 172.21.15.31:50940) 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 -- 172.21.15.31:0/770408035 learned_addr learned my addr 172.21.15.31:0/770408035 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61ebfff700 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eaffd700 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 -- 172.21.15.31:0/770408035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 msgr2=0x555fc36b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 -- 172.21.15.31:0/770408035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 msgr2=0x555fc36b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61eb7fe700 1 -- 172.21.15.31:0/770408035 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fc341ac60 con 0x555fc36a83e0 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.831+0000 7f61ebfff700 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 0x555fc36c1cf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f61dc000be0 tx=0x7f61dc011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699df142da1ebaf3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f61d27fc700 1 -- 172.21.15.31:0/770408035 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc0196f0 con 0x555fc36a83e0 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f61d27fc700 1 -- 172.21.15.31:0/770408035 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61dc019e10 con 0x555fc36a83e0 2023-07-29T16:26:48.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f61d27fc700 1 -- 172.21.15.31:0/770408035 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc005500 con 0x555fc36a83e0 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 -- 172.21.15.31:0/770408035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 msgr2=0x555fc36c1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 0x555fc36c1cf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f61dc000be0 tx=0x7f61dc011720 comp rx=0 tx=0).stop 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 -- 172.21.15.31:0/770408035 shutdown_connections 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b3530 0x555fc36b76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36a83e0 0x555fc36c1cf0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 --2- 172.21.15.31:0/770408035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36b2be0 0x555fc36b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.832+0000 7f6205732b80 1 -- 172.21.15.31:0/770408035 >> 172.21.15.31:0/770408035 conn(0x555fc36bb870 msgr2=0x555fc36ba660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.833+0000 7f6205732b80 1 -- 172.21.15.31:0/770408035 shutdown_connections 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.833+0000 7f6205732b80 1 -- 172.21.15.31:0/770408035 wait complete. 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.833+0000 7f6205732b80 1 -- start start 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.833+0000 7f6205732b80 1 Processor -- start 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36a83e0 0x555fc36c07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b2be0 0x555fc36c0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 0x555fc371a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc36c14a0 con 0x555fc36a83e0 2023-07-29T16:26:48.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc36c1610 con 0x555fc36b3530 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fc371a570 con 0x555fc36b2be0 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f6205732b80 10 client.0 _ll_register_callbacks cb 0x555fc35a9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 0x555fc371a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61eb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36a83e0 0x555fc36c07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 0x555fc371a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50954/0 (socket says 172.21.15.31:50954) 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 -- 172.21.15.31:0/2771294053 learned_addr learned my addr 172.21.15.31:0/2771294053 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:48.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61eaffd700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b2be0 0x555fc36c0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 -- 172.21.15.31:0/2771294053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b2be0 msgr2=0x555fc36c0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b2be0 0x555fc36c0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 -- 172.21.15.31:0/2771294053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36a83e0 msgr2=0x555fc36c07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36a83e0 0x555fc36c07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61ebfff700 1 -- 172.21.15.31:0/2771294053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec1a0350 con 0x555fc36b3530 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.834+0000 7f61eaffd700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b2be0 0x555fc36c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:48.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61ebfff700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 0x555fc371a030 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f61e400c6f0 tx=0x7f61e4004dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83c650b392a18119 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61caffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61caffd700 1 -- 172.21.15.31:0/2771294053 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e40b1070 con 0x555fc36b3530 2023-07-29T16:26:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61dc004390 con 0x555fc36b3530 2023-07-29T16:26:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61caffd700 1 -- 172.21.15.31:0/2771294053 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f61e4098d50 con 0x555fc36b3530 2023-07-29T16:26:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61caffd700 1 -- 172.21.15.31:0/2771294053 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61e409cd80 con 0x555fc36b3530 2023-07-29T16:26:48.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 10 client.28844 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f61dc0099e0 con 0x555fc36b3530 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61caffd700 1 -- 172.21.15.31:0/2771294053 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61dc004390 con 0x555fc36b3530 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f61caffd700 1 -- 172.21.15.31:0/2771294053 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f61e40b11d0 con 0x555fc36b3530 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 10 client.28844 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 10 client.28844 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 20 client.28844 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.835+0000 7f6205732b80 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36b42f0 0x555fc371dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:48.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f6205732b80 4 client.28844 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555fc35fcd40 con 0x555fc36b42f0 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36b42f0 0x555fc371dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f61eb7fe700 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36b42f0 0x555fc371dec0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b9ae1fbbc6bb200 server_cookie=a7e276ed6a54a28d in_seq=0 out_seq=0 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f61caffd700 10 client.28844 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f61caffd700 1 -- 172.21.15.31:0/2771294053 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555fc35fcd40 con 0x555fc36b42f0 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f61caffd700 10 client.28844 handle_command_reply: tid=0 2023-07-29T16:26:48.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.836+0000 7f6205732b80 1 client.28844 shutdown 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 msgr2=0x555fc371a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 0x555fc371a030 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f61e400c6f0 tx=0x7f61e4004dc0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 shutdown_connections 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555fc36b42f0 0x555fc371dec0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555fc36b3530 0x555fc371a030 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f61caffd700 0 client.28844 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:48.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555fc36a83e0 0x555fc36c07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 --2- 172.21.15.31:0/2771294053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555fc36b2be0 0x555fc36c0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 >> 172.21.15.31:0/2771294053 conn(0x555fc36bb870 msgr2=0x555fc36b7190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 shutdown_connections 2023-07-29T16:26:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 1 -- 172.21.15.31:0/2771294053 wait complete. 2023-07-29T16:26:48.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.837+0000 7f6205732b80 20 client.28844 trim_cache size 0 max 16384 2023-07-29T16:26:48.850 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:26:48.850 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:26:48.850 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:26:48.851 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:26:48.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61d40a0790 msgr2=0x7f61d40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61d40a0790 0x7f61d40a2c40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x555fc36b3940 tx=0x7f61e001a040 comp rx=0 tx=0).stop 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 msgr2=0x7f61ec19f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 0x7f61ec19f830 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f61e400ef30 tx=0x7f61e400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 shutdown_connections 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61d40a0790 0x7f61d40a2c40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61ec1081e0 0x7f61ec19f830 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61ec073730 0x7f61ec198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 --2- 172.21.15.31:0/3902343987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61ec074d10 0x7f61ec1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 >> 172.21.15.31:0/3902343987 conn(0x7f61ec0fb970 msgr2=0x7f61ec0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 shutdown_connections 2023-07-29T16:26:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:48.838+0000 7f61f1dff700 1 -- 172.21.15.31:0/3902343987 wait complete. 2023-07-29T16:26:48.854 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:26:48.855 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:26:48.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 849802 8.90 MB/sec execute 393 sec latency 20.059 ms 2023-07-29T16:26:49.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 -- 172.21.15.31:0/787815708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928100340 msgr2=0x7f4928100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 --2- 172.21.15.31:0/787815708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928100340 0x7f4928100750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4924009990 tx=0x7f4924009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 -- 172.21.15.31:0/787815708 shutdown_connections 2023-07-29T16:26:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 --2- 172.21.15.31:0/787815708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4928101ec0 0x7f4928108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 --2- 172.21.15.31:0/787815708 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928100340 0x7f4928100750 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 --2- 172.21.15.31:0/787815708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101530 0x7f4928101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 -- 172.21.15.31:0/787815708 >> 172.21.15.31:0/787815708 conn(0x7f49280fb920 msgr2=0x7f49280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 -- 172.21.15.31:0/787815708 shutdown_connections 2023-07-29T16:26:49.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.070+0000 7f49307ff700 1 -- 172.21.15.31:0/787815708 wait complete. 2023-07-29T16:26:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.071+0000 7f49307ff700 1 Processor -- start 2023-07-29T16:26:49.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.071+0000 7f49307ff700 1 -- start start 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f49307ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4928100340 0x7f4928195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f49307ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928101530 0x7f4928196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f49307ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 0x7f492819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f49307ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492819db60 con 0x7f4928100340 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f49307ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492819dca0 con 0x7f4928101ec0 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f49307ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492819de30 con 0x7f4928101530 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f492e59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4928100340 0x7f4928195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f492ed9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 0x7f492819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f492ed9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 0x7f492819d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50980/0 (socket says 172.21.15.31:50980) 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f492ed9c700 1 -- 172.21.15.31:0/2352829495 learned_addr learned my addr 172.21.15.31:0/2352829495 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.072+0000 7f492dd9a700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928101530 0x7f4928196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f492ed9c700 1 -- 172.21.15.31:0/2352829495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928101530 msgr2=0x7f4928196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f492ed9c700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928101530 0x7f4928196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f492ed9c700 1 -- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4928100340 msgr2=0x7f4928195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f492ed9c700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4928100340 0x7f4928195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f492ed9c700 1 -- 172.21.15.31:0/2352829495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49240095f0 con 0x7f4928101ec0 2023-07-29T16:26:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f492ed9c700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 0x7f492819d590 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4920007ee0 tx=0x7f492000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492000dd20 con 0x7f4928101ec0 2023-07-29T16:26:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f492000f040 con 0x7f4928101ec0 2023-07-29T16:26:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492819e110 con 0x7f4928101ec0 2023-07-29T16:26:49.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200048a0 con 0x7f4928101ec0 2023-07-29T16:26:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.073+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492819e5d0 con 0x7f4928101ec0 2023-07-29T16:26:49.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.074+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49281078d0 con 0x7f4928101ec0 2023-07-29T16:26:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.076+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4920003a00 con 0x7f4928101ec0 2023-07-29T16:26:49.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.078+0000 7f491f7fe700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49140a07e0 0x7f49140a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.078+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f49200cb360 con 0x7f4928101ec0 2023-07-29T16:26:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.078+0000 7f492e59b700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49140a07e0 0x7f49140a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.079+0000 7f492e59b700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49140a07e0 0x7f49140a2c90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4924006010 tx=0x7f4924005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.082+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f492004e720 con 0x7f4928101ec0 2023-07-29T16:26:49.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:48 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:26:49.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:48 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:26:49.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:48 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:26:49.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.428+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f492802cc30 con 0x7f4928101ec0 2023-07-29T16:26:49.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.429+0000 7f491f7fe700 1 -- 172.21.15.31:0/2352829495 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10780 (secure 0 0 0) 0x7f4920094d40 con 0x7f4928101ec0 2023-07-29T16:26:49.428 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:26:49.429 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:26:47.968454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:26:49.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49140a07e0 msgr2=0x7f49140a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49140a07e0 0x7f49140a2c90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4924006010 tx=0x7f4924005960 comp rx=0 tx=0).stop 2023-07-29T16:26:49.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 msgr2=0x7f492819d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 0x7f492819d590 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4920007ee0 tx=0x7f492000d3b0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 shutdown_connections 2023-07-29T16:26:49.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f49140a07e0 0x7f49140a2c90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4928101ec0 0x7f492819d590 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4928100340 0x7f4928195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 --2- 172.21.15.31:0/2352829495 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4928101530 0x7f4928196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 >> 172.21.15.31:0/2352829495 conn(0x7f49280fb920 msgr2=0x7f4928104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 shutdown_connections 2023-07-29T16:26:49.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.431+0000 7f49307ff700 1 -- 172.21.15.31:0/2352829495 wait complete. 2023-07-29T16:26:49.433 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2023-07-29T16:26:49.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:26:49.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 -- 172.21.15.31:0/523353169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101eb0 msgr2=0x7f08d8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 --2- 172.21.15.31:0/523353169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101eb0 0x7f08d8108770 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f08d000b0a0 tx=0x7f08d000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 -- 172.21.15.31:0/523353169 shutdown_connections 2023-07-29T16:26:49.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 --2- 172.21.15.31:0/523353169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101eb0 0x7f08d8108770 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 --2- 172.21.15.31:0/523353169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 0x7f08d8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 --2- 172.21.15.31:0/523353169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101520 0x7f08d8101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.702+0000 7f08df7ff700 1 -- 172.21.15.31:0/523353169 >> 172.21.15.31:0/523353169 conn(0x7f08d80fb970 msgr2=0x7f08d80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.703+0000 7f08df7ff700 1 -- 172.21.15.31:0/523353169 shutdown_connections 2023-07-29T16:26:49.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.703+0000 7f08df7ff700 1 -- 172.21.15.31:0/523353169 wait complete. 2023-07-29T16:26:49.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 Processor -- start 2023-07-29T16:26:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 -- start start 2023-07-29T16:26:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 0x7f08d8198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101520 0x7f08d81986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101eb0 0x7f08d819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d819fcf0 con 0x7f08d8100330 2023-07-29T16:26:49.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d819fe30 con 0x7f08d8101eb0 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08df7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d819ffc0 con 0x7f08d8101520 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.704+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 0x7f08d8198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101eb0 0x7f08d819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 0x7f08d8198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48358/0 (socket says 172.21.15.31:48358) 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08dcd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101520 0x7f08d81986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08dd59b700 1 -- 172.21.15.31:0/3242257802 learned_addr learned my addr 172.21.15.31:0/3242257802 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 -- 172.21.15.31:0/3242257802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101520 msgr2=0x7f08d81986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101520 0x7f08d81986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 -- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 msgr2=0x7f08d8198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 0x7f08d8198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 -- 172.21.15.31:0/3242257802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0009d00 con 0x7f08d8101eb0 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101eb0 0x7f08d819f720 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009fd0 tx=0x7f08d000bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ce7fc700 1 -- 172.21.15.31:0/3242257802 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d000e040 con 0x7f08d8101eb0 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ce7fc700 1 -- 172.21.15.31:0/3242257802 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08d00094c0 con 0x7f08d8101eb0 2023-07-29T16:26:49.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d81a0240 con 0x7f08d8101eb0 2023-07-29T16:26:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.705+0000 7f08ce7fc700 1 -- 172.21.15.31:0/3242257802 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d00052a0 con 0x7f08d8101eb0 2023-07-29T16:26:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.706+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d81a06c0 con 0x7f08d8101eb0 2023-07-29T16:26:49.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.709+0000 7f08ce7fc700 1 -- 172.21.15.31:0/3242257802 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f08d001d470 con 0x7f08d8101eb0 2023-07-29T16:26:49.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.710+0000 7f08f321eb80 1 Processor -- start 2023-07-29T16:26:49.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.710+0000 7f08ce7fc700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f08c40a0790 0x7f08c40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.710+0000 7f08f321eb80 1 -- start start 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.710+0000 7f08ce7fc700 1 -- 172.21.15.31:0/3242257802 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f08d00cd100 con 0x7f08d8101eb0 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.710+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf510ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f08c40a0790 0x7f08c40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf511d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf512740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08f321eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf468a10 con 0x55adcf511530 2023-07-29T16:26:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08f321eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf4663d0 con 0x55adcf5122f0 2023-07-29T16:26:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08f321eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf461cc0 con 0x55adcf510be0 2023-07-29T16:26:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf510ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dcd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf511d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf510ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47288/0 (socket says 172.21.15.31:47288) 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 -- 172.21.15.31:0/801058762 learned_addr learned my addr 172.21.15.31:0/801058762 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf512740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 -- 172.21.15.31:0/801058762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 msgr2=0x55adcf512740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf512740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 -- 172.21.15.31:0/801058762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 msgr2=0x55adcf511d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf511d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.711+0000 7f08dd59b700 1 -- 172.21.15.31:0/801058762 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adcf28fde0 con 0x55adcf510be0 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.712+0000 7f08dd59b700 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf510ff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009df0 tx=0x7f08d4011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755a5930a7c35a23 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.712+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f08c40a0790 0x7f08c40a2c40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009fd0 tx=0x7f08c8009b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.712+0000 7f08c37fe700 1 -- 172.21.15.31:0/801058762 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004230 con 0x55adcf510be0 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.712+0000 7f08c37fe700 1 -- 172.21.15.31:0/801058762 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08d4004390 con 0x55adcf510be0 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.712+0000 7f08c37fe700 1 -- 172.21.15.31:0/801058762 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d40197a0 con 0x55adcf510be0 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 -- 172.21.15.31:0/801058762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 msgr2=0x55adcf510ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf510ff0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009df0 tx=0x7f08d4011720 comp rx=0 tx=0).stop 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 -- 172.21.15.31:0/801058762 shutdown_connections 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf512740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf510ff0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 --2- 172.21.15.31:0/801058762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf511d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 -- 172.21.15.31:0/801058762 >> 172.21.15.31:0/801058762 conn(0x55adcf5063e0 msgr2=0x55adcf50b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 -- 172.21.15.31:0/801058762 shutdown_connections 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 -- 172.21.15.31:0/801058762 wait complete. 2023-07-29T16:26:49.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.713+0000 7f08f321eb80 1 -- start start 2023-07-29T16:26:49.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 Processor -- start 2023-07-29T16:26:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 0x55adcf51a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf5122f0 0x55adcf521d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf3fe510 con 0x55adcf5122f0 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf3fe1b0 con 0x55adcf510be0 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf2e5920 con 0x55adcf511530 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08f321eb80 10 client.0 _ll_register_callbacks cb 0x55adcf407620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 0x55adcf51a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.714+0000 7f08dcd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 0x55adcf51a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51016/0 (socket says 172.21.15.31:51016) 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 -- 172.21.15.31:0/3487695138 learned_addr learned my addr 172.21.15.31:0/3487695138 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf5122f0 0x55adcf521d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 -- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 msgr2=0x55adcf51ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 -- 172.21.15.31:0/3487695138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf5122f0 msgr2=0x55adcf521d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf5122f0 0x55adcf521d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 -- 172.21.15.31:0/3487695138 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d4009a50 con 0x55adcf510be0 2023-07-29T16:26:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 0x55adcf51a7e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f08d4000c00 tx=0x7f08d40038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=236f2a2ba0b71ab8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.715+0000 7f08c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:49.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3487695138 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4005460 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3487695138 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f08d4005ce0 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55adcf48cd10 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3487695138 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08d40055c0 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08f321eb80 10 client.28862 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55adcf278c60 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3487695138 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55adcf48cd10 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3487695138 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f08d4021040 con 0x55adcf510be0 2023-07-29T16:26:49.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08f321eb80 10 client.28862 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.716+0000 7f08f321eb80 10 client.28862 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08f321eb80 20 client.28862 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf526460 0x55adcf528870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08f321eb80 4 client.28862 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55adcf4dfd70 con 0x55adcf526460 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf526460 0x55adcf528870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf526460 0x55adcf528870 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19550ff7d9c2e153 server_cookie=804ad71d61b85485 in_seq=0 out_seq=0 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.717+0000 7f08c0ff9700 10 client.28862 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.718+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3487695138 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55adcf4dfd70 con 0x55adcf526460 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.718+0000 7f08c0ff9700 10 client.28862 handle_command_reply: tid=0 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.718+0000 7f08f321eb80 1 client.28862 shutdown 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.718+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 msgr2=0x55adcf51a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.718+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 0x55adcf51a7e0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f08d4000c00 tx=0x7f08d40038b0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 shutdown_connections 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf5122f0 0x55adcf521d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf526460 0x55adcf528870 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf510be0 0x55adcf51a7e0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3487695138 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 >> 172.21.15.31:0/3487695138 conn(0x55adcf5063e0 msgr2=0x55adcf50b060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08c0ff9700 0 client.28862 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 shutdown_connections 2023-07-29T16:26:49.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 1 -- 172.21.15.31:0/3487695138 wait complete. 2023-07-29T16:26:49.726 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.719+0000 7f08f321eb80 20 client.28862 trim_cache size 0 max 16384 2023-07-29T16:26:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 Processor -- start 2023-07-29T16:26:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 -- start start 2023-07-29T16:26:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf528870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf521d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf519e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf508460 con 0x55adcf511530 2023-07-29T16:26:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf518490 con 0x55adcf5122f0 2023-07-29T16:26:49.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08f321eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf52d1d0 con 0x55adcf510be0 2023-07-29T16:26:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf528870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08dcd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf521d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf528870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47314/0 (socket says 172.21.15.31:47314) 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.733+0000 7f08dd59b700 1 -- 172.21.15.31:0/3101223200 learned_addr learned my addr 172.21.15.31:0/3101223200 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf519e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08dd59b700 1 -- 172.21.15.31:0/3101223200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 msgr2=0x55adcf519e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf519e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08dd59b700 1 -- 172.21.15.31:0/3101223200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 msgr2=0x55adcf521d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf521d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08dd59b700 1 -- 172.21.15.31:0/3101223200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adcf2b6250 con 0x55adcf510be0 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf528870 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f08d40096b0 tx=0x7f08d4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb900ebcef32cdcf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.734+0000 7f08c3fff700 1 -- 172.21.15.31:0/3101223200 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4019420 con 0x55adcf510be0 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08c3fff700 1 -- 172.21.15.31:0/3101223200 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08d4005720 con 0x55adcf510be0 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08c3fff700 1 -- 172.21.15.31:0/3101223200 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4004930 con 0x55adcf510be0 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 -- 172.21.15.31:0/3101223200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 msgr2=0x55adcf528870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf528870 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f08d40096b0 tx=0x7f08d4009260 comp rx=0 tx=0).stop 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 -- 172.21.15.31:0/3101223200 shutdown_connections 2023-07-29T16:26:49.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf519e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf510be0 0x55adcf528870 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3101223200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf511530 0x55adcf521d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 -- 172.21.15.31:0/3101223200 >> 172.21.15.31:0/3101223200 conn(0x55adcf5063e0 msgr2=0x55adcf519540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 -- 172.21.15.31:0/3101223200 shutdown_connections 2023-07-29T16:26:49.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.735+0000 7f08f321eb80 1 -- 172.21.15.31:0/3101223200 wait complete. 2023-07-29T16:26:49.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.736+0000 7f08f321eb80 1 -- start start 2023-07-29T16:26:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.736+0000 7f08f321eb80 1 Processor -- start 2023-07-29T16:26:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.736+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 0x55adcf51d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.736+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08f321eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf520a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08f321eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf520fb0 con 0x55adcf510be0 2023-07-29T16:26:49.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08f321eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf521120 con 0x55adcf5122f0 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08f321eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adcf5212e0 con 0x55adcf511530 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08f321eb80 10 client.0 _ll_register_callbacks cb 0x55adcf407620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 0x55adcf51d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08ddd9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf520a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dd59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 0x55adcf51d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48390/0 (socket says 172.21.15.31:48390) 2023-07-29T16:26:49.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dd59b700 1 -- 172.21.15.31:0/3653041575 learned_addr learned my addr 172.21.15.31:0/3653041575 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47316/0 (socket says 172.21.15.31:47316) 2023-07-29T16:26:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 -- 172.21.15.31:0/3653041575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 msgr2=0x55adcf520a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf520a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 -- 172.21.15.31:0/3653041575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 msgr2=0x55adcf51d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 0x55adcf51d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dcd9a700 1 -- 172.21.15.31:0/3653041575 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55adcf278c60 con 0x55adcf511530 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08ddd9c700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf520a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.737+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 0x55adcf51d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08dcd9a700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51da50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55adcf529120 tx=0x7f08c80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f107ad0c9ba43 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3653041575 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8026040 con 0x55adcf511530 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08d4004390 con 0x55adcf511530 2023-07-29T16:26:49.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3653041575 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f08c8011bb0 con 0x55adcf511530 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3653041575 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08c8011db0 con 0x55adcf511530 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08f321eb80 10 client.28870 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.738+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55adcf28e2b0 con 0x55adcf511530 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3653041575 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08d4004390 con 0x55adcf511530 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3653041575 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f08c8003980 con 0x55adcf511530 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08f321eb80 10 client.28870 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08f321eb80 10 client.28870 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:26:49.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08f321eb80 20 client.28870 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf578a10 0x55adcf57ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08f321eb80 4 client.28870 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55adcf45ad40 con 0x55adcf578a10 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.739+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf578a10 0x55adcf57ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08dd59b700 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf578a10 0x55adcf57ae20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5f36e8c19efbf658 server_cookie=e125ea1bfbc5b92d in_seq=0 out_seq=0 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08c0ff9700 10 client.28870 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08c0ff9700 1 -- 172.21.15.31:0/3653041575 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55adcf45ad40 con 0x55adcf578a10 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08c0ff9700 10 client.28870 handle_command_reply: tid=0 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 client.28870 shutdown 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 msgr2=0x55adcf51da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51da50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55adcf529120 tx=0x7f08c80047c0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 shutdown_connections 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55adcf578a10 0x55adcf57ae20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adcf5122f0 0x55adcf520a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55adcf510be0 0x55adcf51d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 --2- 172.21.15.31:0/3653041575 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55adcf511530 0x55adcf51da50 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08c0ff9700 0 client.28870 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:26:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 >> 172.21.15.31:0/3653041575 conn(0x55adcf5063e0 msgr2=0x55adcf5186e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 shutdown_connections 2023-07-29T16:26:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.740+0000 7f08f321eb80 1 -- 172.21.15.31:0/3653041575 wait complete. 2023-07-29T16:26:49.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08f321eb80 20 client.28870 trim_cache size 0 max 16384 2023-07-29T16:26:49.748 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:26:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f08c40a0790 msgr2=0x7f08c40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f08c40a0790 0x7f08c40a2c40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f08c8009fd0 tx=0x7f08c8009b40 comp rx=0 tx=0).stop 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101eb0 msgr2=0x7f08d819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101eb0 0x7f08d819f720 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009fd0 tx=0x7f08d000bce0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 shutdown_connections 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f08c40a0790 0x7f08c40a2c40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08d8101eb0 0x7f08d819f720 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08d8100330 0x7f08d8198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 --2- 172.21.15.31:0/3242257802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08d8101520 0x7f08d81986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.741+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 >> 172.21.15.31:0/3242257802 conn(0x7f08d80fb970 msgr2=0x7f08d80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.742+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 shutdown_connections 2023-07-29T16:26:49.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.742+0000 7f08df7ff700 1 -- 172.21.15.31:0/3242257802 wait complete. 2023-07-29T16:26:49.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:26:49.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 851584 8.91 MB/sec execute 394 sec latency 34.772 ms 2023-07-29T16:26:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 -- 172.21.15.31:0/1748060738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 msgr2=0x7efd40101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 --2- 172.21.15.31:0/1748060738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40101970 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7efd34009990 tx=0x7efd34009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 -- 172.21.15.31:0/1748060738 shutdown_connections 2023-07-29T16:26:49.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 --2- 172.21.15.31:0/1748060738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd40101eb0 0x7efd40108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 --2- 172.21.15.31:0/1748060738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd40100330 0x7efd40100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 --2- 172.21.15.31:0/1748060738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40101970 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.979+0000 7efd469ff700 1 -- 172.21.15.31:0/1748060738 >> 172.21.15.31:0/1748060738 conn(0x7efd400fb970 msgr2=0x7efd400fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.980+0000 7efd469ff700 1 -- 172.21.15.31:0/1748060738 shutdown_connections 2023-07-29T16:26:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.980+0000 7efd469ff700 1 -- 172.21.15.31:0/1748060738 wait complete. 2023-07-29T16:26:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.981+0000 7efd469ff700 1 Processor -- start 2023-07-29T16:26:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 -- start start 2023-07-29T16:26:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd40100330 0x7efd40196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd40101eb0 0x7efd4019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4019dba0 con 0x7efd40101eb0 2023-07-29T16:26:49.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4019dce0 con 0x7efd40100330 2023-07-29T16:26:49.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.982+0000 7efd469ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4019de70 con 0x7efd40101520 2023-07-29T16:26:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57200/0 (socket says 172.21.15.31:57200) 2023-07-29T16:26:49.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 -- 172.21.15.31:0/1330034614 learned_addr learned my addr 172.21.15.31:0/1330034614 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd44f9c700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd40101eb0 0x7efd4019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3ffff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd40100330 0x7efd40196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 -- 172.21.15.31:0/1330034614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd40100330 msgr2=0x7efd40196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd40100330 0x7efd40196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 -- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd40101eb0 msgr2=0x7efd4019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd40101eb0 0x7efd4019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 -- 172.21.15.31:0/1330034614 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd340095f0 con 0x7efd40101520 2023-07-29T16:26:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.983+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40196560 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efd34004740 tx=0x7efd34009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.984+0000 7efd3d7fa700 1 -- 172.21.15.31:0/1330034614 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3401d070 con 0x7efd40101520 2023-07-29T16:26:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.984+0000 7efd3d7fa700 1 -- 172.21.15.31:0/1330034614 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd34004830 con 0x7efd40101520 2023-07-29T16:26:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.984+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4019e0f0 con 0x7efd40101520 2023-07-29T16:26:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.984+0000 7efd3d7fa700 1 -- 172.21.15.31:0/1330034614 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340053e0 con 0x7efd40101520 2023-07-29T16:26:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.984+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4019e510 con 0x7efd40101520 2023-07-29T16:26:49.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.988+0000 7efd3d7fa700 1 -- 172.21.15.31:0/1330034614 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7efd34005540 con 0x7efd40101520 2023-07-29T16:26:49.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.989+0000 7efd5a2bab80 1 Processor -- start 2023-07-29T16:26:49.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.989+0000 7efd5a2bab80 1 -- start start 2023-07-29T16:26:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a12400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a13170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282965a10 con 0x564282a13700 2023-07-29T16:26:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642829633d0 con 0x564282a11ff0 2023-07-29T16:26:49.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56428295ecc0 con 0x564282a12940 2023-07-29T16:26:49.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd44f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a12400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a13170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a13170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57216/0 (socket says 172.21.15.31:57216) 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 -- 172.21.15.31:0/970035954 learned_addr learned my addr 172.21.15.31:0/970035954 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3d7fa700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd280a0790 0x7efd280a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 -- 172.21.15.31:0/970035954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 msgr2=0x564282a12400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a12400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 -- 172.21.15.31:0/970035954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 msgr2=0x564282a13b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.990+0000 7efd3ffff700 1 -- 172.21.15.31:0/970035954 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56428278cde0 con 0x564282a12940 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.991+0000 7efd3d7fa700 1 -- 172.21.15.31:0/1330034614 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efd340ccb10 con 0x7efd40101520 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.991+0000 7efd44f9c700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd280a0790 0x7efd280a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.991+0000 7efd3ffff700 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a13170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7efd30009d30 tx=0x7efd30011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6199817cc3a3cfd8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.991+0000 7efd267fc700 1 -- 172.21.15.31:0/970035954 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd30004220 con 0x564282a12940 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.991+0000 7efd267fc700 1 -- 172.21.15.31:0/970035954 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd30004380 con 0x564282a12940 2023-07-29T16:26:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.991+0000 7efd267fc700 1 -- 172.21.15.31:0/970035954 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd300198d0 con 0x564282a12940 2023-07-29T16:26:49.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 -- 172.21.15.31:0/970035954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 msgr2=0x564282a13170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:49.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a13170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7efd30009d30 tx=0x7efd30011720 comp rx=0 tx=0).stop 2023-07-29T16:26:49.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd44f9c700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd280a0790 0x7efd280a2c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x564282a11d90 tx=0x7efd38009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 -- 172.21.15.31:0/970035954 shutdown_connections 2023-07-29T16:26:49.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a12400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/970035954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a13170 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.992+0000 7efd5a2bab80 1 -- 172.21.15.31:0/970035954 >> 172.21.15.31:0/970035954 conn(0x564282a033e0 msgr2=0x564282a08990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.993+0000 7efd5a2bab80 1 -- 172.21.15.31:0/970035954 shutdown_connections 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.993+0000 7efd5a2bab80 1 -- 172.21.15.31:0/970035954 wait complete. 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.993+0000 7efd5a2bab80 1 -- start start 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 Processor -- start 2023-07-29T16:26:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a1bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a1c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a23220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642828fb510 con 0x564282a13700 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642828fb1b0 con 0x564282a11ff0 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642827e2920 con 0x564282a12940 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd5a2bab80 10 client.0 _ll_register_callbacks cb 0x564282904620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:49.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd3ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a1bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd3f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a1c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd3ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a1bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39520/0 (socket says 172.21.15.31:39520) 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd3f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a1c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57222/0 (socket says 172.21.15.31:57222) 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd3ffff700 1 -- 172.21.15.31:0/2210414583 learned_addr learned my addr 172.21.15.31:0/2210414583 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.994+0000 7efd44f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a23220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3f7fe700 1 -- 172.21.15.31:0/2210414583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 msgr2=0x564282a1bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a1bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3f7fe700 1 -- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 msgr2=0x564282a23220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a23220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3f7fe700 1 -- 172.21.15.31:0/2210414583 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd30009960 con 0x564282a12940 2023-07-29T16:26:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3ffff700 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a1bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a1c1e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efd340ca600 tx=0x7efd3409a7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2af36a0cf9df89c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd1f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:49.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd1f7fe700 1 -- 172.21.15.31:0/2210414583 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3409e2d0 con 0x564282a12940 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd1f7fe700 1 -- 172.21.15.31:0/2210414583 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efd340a2660 con 0x564282a12940 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.995+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564282989d10 con 0x564282a12940 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd1f7fe700 1 -- 172.21.15.31:0/2210414583 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd340a0070 con 0x564282a12940 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 10 client.28882 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x564282775c60 con 0x564282a12940 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd1f7fe700 1 -- 172.21.15.31:0/2210414583 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564282989d10 con 0x564282a12940 2023-07-29T16:26:50.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd1f7fe700 1 -- 172.21.15.31:0/2210414583 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7efd34099200 con 0x564282a12940 2023-07-29T16:26:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 10 client.28882 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 10 client.28882 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:26:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 20 client.28882 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a275a0 0x564282a29980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 4 client.28882 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:50.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642829dcd70 con 0x564282a275a0 2023-07-29T16:26:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.996+0000 7efd3ffff700 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a275a0 0x564282a29980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.997+0000 7efd3ffff700 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a275a0 0x564282a29980 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a8c1f8e20a7ca0c server_cookie=8792907cb04fd018 in_seq=0 out_seq=0 2023-07-29T16:26:50.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.997+0000 7efd1f7fe700 10 client.28882 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd1f7fe700 1 -- 172.21.15.31:0/2210414583 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5642829dcd70 con 0x564282a275a0 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd1f7fe700 10 client.28882 handle_command_reply: tid=0 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 client.28882 shutdown 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 msgr2=0x564282a1c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a1c1e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efd340ca600 tx=0x7efd3409a7e0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 shutdown_connections 2023-07-29T16:26:50.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a275a0 0x564282a29980 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a23220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a11ff0 0x564282a1bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/2210414583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a12940 0x564282a1c1e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd1f7fe700 0 client.28882 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.998+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 >> 172.21.15.31:0/2210414583 conn(0x564282a033e0 msgr2=0x564282a08060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.999+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 shutdown_connections 2023-07-29T16:26:50.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.999+0000 7efd5a2bab80 1 -- 172.21.15.31:0/2210414583 wait complete. 2023-07-29T16:26:50.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:49.999+0000 7efd5a2bab80 20 client.28882 trim_cache size 0 max 16384 2023-07-29T16:26:50.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 Processor -- start 2023-07-29T16:26:50.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 -- start start 2023-07-29T16:26:50.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a29980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 0x564282a231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282a1b810 con 0x564282a13700 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282a006e0 con 0x564282a12940 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282a05460 con 0x564282a11ff0 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd3f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 0x564282a231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd3f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 0x564282a231e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39526/0 (socket says 172.21.15.31:39526) 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd3f7fe700 1 -- 172.21.15.31:0/3012546066 learned_addr learned my addr 172.21.15.31:0/3012546066 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.013+0000 7efd44f9c700 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3ffff700 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a29980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3f7fe700 1 -- 172.21.15.31:0/3012546066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 msgr2=0x564282a29980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a29980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3f7fe700 1 -- 172.21.15.31:0/3012546066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 msgr2=0x564282a13ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3f7fe700 1 -- 172.21.15.31:0/3012546066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642827b3250 con 0x564282a12940 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 0x564282a231e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efd340985f0 tx=0x7efd34099a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c076ce06edc83749 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd26ffd700 1 -- 172.21.15.31:0/3012546066 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd34099580 con 0x564282a12940 2023-07-29T16:26:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd26ffd700 1 -- 172.21.15.31:0/3012546066 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd3409e040 con 0x564282a12940 2023-07-29T16:26:50.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.014+0000 7efd26ffd700 1 -- 172.21.15.31:0/3012546066 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd340a21d0 con 0x564282a12940 2023-07-29T16:26:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 -- 172.21.15.31:0/3012546066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 msgr2=0x564282a231e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 0x564282a231e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efd340985f0 tx=0x7efd34099a70 comp rx=0 tx=0).stop 2023-07-29T16:26:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 -- 172.21.15.31:0/3012546066 shutdown_connections 2023-07-29T16:26:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a13700 0x564282a13ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a29980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/3012546066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a12940 0x564282a231e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 -- 172.21.15.31:0/3012546066 >> 172.21.15.31:0/3012546066 conn(0x564282a033e0 msgr2=0x564282a08900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 -- 172.21.15.31:0/3012546066 shutdown_connections 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.015+0000 7efd5a2bab80 1 -- 172.21.15.31:0/3012546066 wait complete. 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.016+0000 7efd5a2bab80 1 -- start start 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.016+0000 7efd5a2bab80 1 Processor -- start 2023-07-29T16:26:50.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.016+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a0ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a12940 0x564282a0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd5a2bab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 0x564282a23140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282a102d0 con 0x564282a12940 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282a10440 con 0x564282a13700 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd5a2bab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564282a10600 con 0x564282a11ff0 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd5a2bab80 10 client.0 _ll_register_callbacks cb 0x564282904620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:50.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 0x564282a23140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 0x564282a23140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39540/0 (socket says 172.21.15.31:39540) 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 -- 172.21.15.31:0/582519151 learned_addr learned my addr 172.21.15.31:0/582519151 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a12940 0x564282a0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd3ffff700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a0ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 -- 172.21.15.31:0/582519151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 msgr2=0x564282a0ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a0ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 -- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a12940 msgr2=0x564282a0f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a12940 0x564282a0f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 -- 172.21.15.31:0/582519151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564282775c60 con 0x564282a13700 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd3f7fe700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a12940 0x564282a0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd44f9c700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 0x564282a23140 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7efd38017aa0 tx=0x7efd38004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9c143361745e1ee server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.017+0000 7efd1f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd1f7fe700 1 -- 172.21.15.31:0/582519151 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3800e040 con 0x564282a13700 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd30004380 con 0x564282a13700 2023-07-29T16:26:50.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd1f7fe700 1 -- 172.21.15.31:0/582519151 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efd3801c040 con 0x564282a13700 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd1f7fe700 1 -- 172.21.15.31:0/582519151 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd38003e10 con 0x564282a13700 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd5a2bab80 10 client.28886 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56428278b2b0 con 0x564282a13700 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd1f7fe700 1 -- 172.21.15.31:0/582519151 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd30004380 con 0x564282a13700 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.018+0000 7efd1f7fe700 1 -- 172.21.15.31:0/582519151 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7efd38031070 con 0x564282a13700 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd5a2bab80 10 client.28886 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd5a2bab80 10 client.28886 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:26:50.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd5a2bab80 20 client.28886 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:50.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a752a0 0x564282a776b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd5a2bab80 4 client.28886 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:26:50.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564282957d40 con 0x564282a752a0 2023-07-29T16:26:50.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd3ffff700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a752a0 0x564282a776b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd3ffff700 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a752a0 0x564282a776b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa38eb22f4cab3eb server_cookie=d642f37731f3f350 in_seq=0 out_seq=0 2023-07-29T16:26:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.019+0000 7efd1f7fe700 10 client.28886 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd1f7fe700 1 -- 172.21.15.31:0/582519151 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564282957d40 con 0x564282a752a0 2023-07-29T16:26:50.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd1f7fe700 10 client.28886 handle_command_reply: tid=0 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 client.28886 shutdown 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 msgr2=0x564282a23140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 0x564282a23140 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7efd38017aa0 tx=0x7efd38004ad0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 shutdown_connections 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x564282a752a0 0x564282a776b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564282a13700 0x564282a23140 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564282a11ff0 0x564282a0ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 --2- 172.21.15.31:0/582519151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564282a12940 0x564282a0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd1f7fe700 0 client.28886 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.020+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 >> 172.21.15.31:0/582519151 conn(0x564282a033e0 msgr2=0x564282a16660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 shutdown_connections 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd5a2bab80 1 -- 172.21.15.31:0/582519151 wait complete. 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd5a2bab80 20 client.28886 trim_cache size 0 max 16384 2023-07-29T16:26:50.027 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:26:50.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd280a0790 msgr2=0x7efd280a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd280a0790 0x7efd280a2c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x564282a11d90 tx=0x7efd38009040 comp rx=0 tx=0).stop 2023-07-29T16:26:50.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 msgr2=0x7efd40196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40196560 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7efd34004740 tx=0x7efd34009580 comp rx=0 tx=0).stop 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 shutdown_connections 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd280a0790 0x7efd280a2c40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd40101eb0 0x7efd4019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd40100330 0x7efd40196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 --2- 172.21.15.31:0/1330034614 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd40101520 0x7efd40196560 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 >> 172.21.15.31:0/1330034614 conn(0x7efd400fb970 msgr2=0x7efd400fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 shutdown_connections 2023-07-29T16:26:50.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.021+0000 7efd469ff700 1 -- 172.21.15.31:0/1330034614 wait complete. 2023-07-29T16:26:50.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:49 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2352829495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:26:50.036 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:49 smithi031 ceph-mon[95504]: pgmap v712: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 2.8 MiB/s wr, 315 op/s 2023-07-29T16:26:50.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:26:50.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 -- 172.21.15.31:0/3106723157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90074d30 msgr2=0x7f3c90073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3106723157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90074d30 0x7f3c90073180 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3c800098f0 tx=0x7f3c80009c00 comp rx=0 tx=0).stop 2023-07-29T16:26:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 -- 172.21.15.31:0/3106723157 shutdown_connections 2023-07-29T16:26:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3106723157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90108220 0x7f3c9010a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3106723157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90074d30 0x7f3c90073180 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3106723157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90073750 0x7f3c90073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.260+0000 7f3c94fff700 1 -- 172.21.15.31:0/3106723157 >> 172.21.15.31:0/3106723157 conn(0x7f3c900fb9b0 msgr2=0x7f3c900fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.261+0000 7f3c94fff700 1 -- 172.21.15.31:0/3106723157 shutdown_connections 2023-07-29T16:26:50.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.261+0000 7f3c94fff700 1 -- 172.21.15.31:0/3106723157 wait complete. 2023-07-29T16:26:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.261+0000 7f3c94fff700 1 Processor -- start 2023-07-29T16:26:50.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 -- start start 2023-07-29T16:26:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 0x7f3c901982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90074d30 0x7f3c90198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 0x7f3c9019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9019fe50 con 0x7f3c90074d30 2023-07-29T16:26:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9019ff90 con 0x7f3c90108220 2023-07-29T16:26:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c94fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c901a0120 con 0x7f3c90073750 2023-07-29T16:26:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 0x7f3c901982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c8ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 0x7f3c9019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 0x7f3c901982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57262/0 (socket says 172.21.15.31:57262) 2023-07-29T16:26:50.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c8ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 0x7f3c9019f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39546/0 (socket says 172.21.15.31:39546) 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c8ed9d700 1 -- 172.21.15.31:0/3230986115 learned_addr learned my addr 172.21.15.31:0/3230986115 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.262+0000 7f3c87fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90074d30 0x7f3c90198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3230986115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 msgr2=0x7f3c9019f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 0x7f3c9019f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90074d30 msgr2=0x7f3c90198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90074d30 0x7f3c90198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3230986115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80009520 con 0x7f3c90073750 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 0x7f3c9019f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c87fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90074d30 0x7f3c90198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 0x7f3c901982d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009230 tx=0x7f3c8000b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c877fe700 1 -- 172.21.15.31:0/3230986115 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8001c070 con 0x7f3c90073750 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c877fe700 1 -- 172.21.15.31:0/3230986115 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c8000f7d0 con 0x7f3c90073750 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c877fe700 1 -- 172.21.15.31:0/3230986115 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80003ac0 con 0x7f3c90073750 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c901a03a0 con 0x7f3c90073750 2023-07-29T16:26:50.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.263+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c901a07c0 con 0x7f3c90073750 2023-07-29T16:26:50.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.268+0000 7f3c877fe700 1 -- 172.21.15.31:0/3230986115 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3c80020030 con 0x7f3c90073750 2023-07-29T16:26:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.269+0000 7f3ca898ab80 1 Processor -- start 2023-07-29T16:26:50.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.269+0000 7f3ca898ab80 1 -- start start 2023-07-29T16:26:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ab6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ab7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ab8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94a0ea10 con 0x559b94ab6be0 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94a0c3d0 con 0x559b94ab7530 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94a07cc0 con 0x559b94ab82f0 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ab7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c8ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ab6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c877fe700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3c7c0a0810 0x7f3c7c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c87fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ab8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ab7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39560/0 (socket says 172.21.15.31:39560) 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c8e59c700 1 -- 172.21.15.31:0/2958535733 learned_addr learned my addr 172.21.15.31:0/2958535733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c87fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ab8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57268/0 (socket says 172.21.15.31:57268) 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c877fe700 1 -- 172.21.15.31:0/3230986115 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3c800cc9b0 con 0x7f3c90073750 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.270+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3c7c0a0810 0x7f3c7c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c87fff700 1 -- 172.21.15.31:0/2958535733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 msgr2=0x559b94ab7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c87fff700 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ab7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c87fff700 1 -- 172.21.15.31:0/2958535733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 msgr2=0x559b94ab6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c87fff700 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ab6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c87fff700 1 -- 172.21.15.31:0/2958535733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b94835de0 con 0x559b94ab82f0 2023-07-29T16:26:50.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c87fff700 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ab8740 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009cc0 tx=0x7f3c780113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f4e269f4a5f65a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3c7c0a0810 0x7f3c7c0a2cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009fd0 tx=0x7f3c88004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c84ff9700 1 -- 172.21.15.31:0/2958535733 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78011c20 con 0x559b94ab82f0 2023-07-29T16:26:50.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c84ff9700 1 -- 172.21.15.31:0/2958535733 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c78011d80 con 0x559b94ab82f0 2023-07-29T16:26:50.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.271+0000 7f3c84ff9700 1 -- 172.21.15.31:0/2958535733 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7800f7b0 con 0x559b94ab82f0 2023-07-29T16:26:50.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2958535733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 msgr2=0x559b94ab8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ab8740 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f3c78009cc0 tx=0x7f3c780113e0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2958535733 shutdown_connections 2023-07-29T16:26:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ab8740 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ab6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2958535733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ab7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2958535733 >> 172.21.15.31:0/2958535733 conn(0x559b94aac3e0 msgr2=0x559b94ab1990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2958535733 shutdown_connections 2023-07-29T16:26:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.272+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2958535733 wait complete. 2023-07-29T16:26:50.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.273+0000 7f3ca898ab80 1 -- start start 2023-07-29T16:26:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.273+0000 7f3ca898ab80 1 Processor -- start 2023-07-29T16:26:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.273+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ac0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.273+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.273+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ac7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.273+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b949a4510 con 0x559b94ab6be0 2023-07-29T16:26:50.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b949a41b0 con 0x559b94ab7530 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b9488b920 con 0x559b94ab82f0 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3ca898ab80 10 client.0 _ll_register_callbacks cb 0x559b949ad620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39564/0 (socket says 172.21.15.31:39564) 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 -- 172.21.15.31:0/2679572452 learned_addr learned my addr 172.21.15.31:0/2679572452 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ac7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ac0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 -- 172.21.15.31:0/2679572452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 msgr2=0x559b94ac7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ac7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 -- 172.21.15.31:0/2679572452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 msgr2=0x559b94ac0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ac0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.274+0000 7f3c87fff700 1 -- 172.21.15.31:0/2679572452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c78009920 con 0x559b94ab7530 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c87fff700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac0cc0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3c78000c00 tx=0x7f3c780127b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d57c5bf259031712 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:50.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c75ffb700 1 -- 172.21.15.31:0/2679572452 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c78005450 con 0x559b94ab7530 2023-07-29T16:26:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b94a32d10 con 0x559b94ab7530 2023-07-29T16:26:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c75ffb700 1 -- 172.21.15.31:0/2679572452 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3c780055b0 con 0x559b94ab7530 2023-07-29T16:26:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c75ffb700 1 -- 172.21.15.31:0/2679572452 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c78004020 con 0x559b94ab7530 2023-07-29T16:26:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3ca898ab80 10 client.28898 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559b9481ec60 con 0x559b94ab7530 2023-07-29T16:26:50.283 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c75ffb700 1 -- 172.21.15.31:0/2679572452 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b94a32d10 con 0x559b94ab7530 2023-07-29T16:26:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.275+0000 7f3c75ffb700 1 -- 172.21.15.31:0/2679572452 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3c78004b90 con 0x559b94ab7530 2023-07-29T16:26:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3ca898ab80 10 client.28898 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3ca898ab80 10 client.28898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:50.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3ca898ab80 20 client.28898 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94acc410 0x559b94ace820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3ca898ab80 4 client.28898 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b94a85d70 con 0x559b94acc410 2023-07-29T16:26:50.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94acc410 0x559b94ace820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94acc410 0x559b94ace820 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d6e46ba701347d8 server_cookie=27b20db4967bffb1 in_seq=0 out_seq=0 2023-07-29T16:26:50.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.276+0000 7f3c75ffb700 10 client.28898 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:50.287 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:49 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2352829495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:26:50.287 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:49 smithi039 ceph-mon[128858]: pgmap v712: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 2.8 MiB/s wr, 315 op/s 2023-07-29T16:26:50.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.277+0000 7f3c75ffb700 1 -- 172.21.15.31:0/2679572452 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559b94a85d70 con 0x559b94acc410 2023-07-29T16:26:50.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.277+0000 7f3c75ffb700 10 client.28898 handle_command_reply: tid=0 2023-07-29T16:26:50.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.277+0000 7f3ca898ab80 1 client.28898 shutdown 2023-07-29T16:26:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 msgr2=0x559b94ac0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac0cc0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3c78000c00 tx=0x7f3c780127b0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 shutdown_connections 2023-07-29T16:26:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94acc410 0x559b94ace820 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab82f0 0x559b94ac7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab6be0 0x559b94ac0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/2679572452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac0cc0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3c75ffb700 0 client.28898 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 >> 172.21.15.31:0/2679572452 conn(0x559b94aac3e0 msgr2=0x559b94ab1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 shutdown_connections 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 1 -- 172.21.15.31:0/2679572452 wait complete. 2023-07-29T16:26:50.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.278+0000 7f3ca898ab80 20 client.28898 trim_cache size 0 max 16384 2023-07-29T16:26:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.292+0000 7f3ca898ab80 1 Processor -- start 2023-07-29T16:26:50.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 -- start start 2023-07-29T16:26:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 0x559b94ab2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab7530 0x559b94ac7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ab1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94abe450 con 0x559b94ab82f0 2023-07-29T16:26:50.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94ac18d0 con 0x559b94ab6be0 2023-07-29T16:26:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94abf210 con 0x559b94ab7530 2023-07-29T16:26:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 0x559b94ab2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 0x559b94ab2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39568/0 (socket says 172.21.15.31:39568) 2023-07-29T16:26:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3c8e59c700 1 -- 172.21.15.31:0/821073169 learned_addr learned my addr 172.21.15.31:0/821073169 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3c87fff700 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab7530 0x559b94ac7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.293+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ab1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c8e59c700 1 -- 172.21.15.31:0/821073169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab7530 msgr2=0x559b94ac7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab7530 0x559b94ac7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c8e59c700 1 -- 172.21.15.31:0/821073169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 msgr2=0x559b94ab1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ab1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c8e59c700 1 -- 172.21.15.31:0/821073169 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b9485c250 con 0x559b94ab6be0 2023-07-29T16:26:50.295 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c87fff700 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab7530 0x559b94ac7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:50.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 0x559b94ab2690 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3c800098c0 tx=0x7f3c80019040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4143e8981da84d7b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c857fa700 1 -- 172.21.15.31:0/821073169 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c800a24a0 con 0x559b94ab6be0 2023-07-29T16:26:50.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c857fa700 1 -- 172.21.15.31:0/821073169 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c8000b9b0 con 0x559b94ab6be0 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.294+0000 7f3c857fa700 1 -- 172.21.15.31:0/821073169 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80021c50 con 0x559b94ab6be0 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 -- 172.21.15.31:0/821073169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 msgr2=0x559b94ab2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 0x559b94ab2690 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3c800098c0 tx=0x7f3c80019040 comp rx=0 tx=0).stop 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 -- 172.21.15.31:0/821073169 shutdown_connections 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ab1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab6be0 0x559b94ab2690 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/821073169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab7530 0x559b94ac7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 -- 172.21.15.31:0/821073169 >> 172.21.15.31:0/821073169 conn(0x559b94aac3e0 msgr2=0x559b94abf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 -- 172.21.15.31:0/821073169 shutdown_connections 2023-07-29T16:26:50.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 -- 172.21.15.31:0/821073169 wait complete. 2023-07-29T16:26:50.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.295+0000 7f3ca898ab80 1 -- start start 2023-07-29T16:26:50.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 Processor -- start 2023-07-29T16:26:50.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 0x559b94ac7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ac3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94ac45e0 con 0x559b94ab82f0 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94ac4720 con 0x559b94ab7530 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b94ac48e0 con 0x559b94ab6be0 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3ca898ab80 10 client.0 _ll_register_callbacks cb 0x559b949ad620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 0x559b94ac7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3c87fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3c8e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 0x559b94ac7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57302/0 (socket says 172.21.15.31:57302) 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3452067230 learned_addr learned my addr 172.21.15.31:0/3452067230 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.296+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ac3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 msgr2=0x559b94ac7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3452067230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 msgr2=0x559b94ac3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ac3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8e59c700 1 -- 172.21.15.31:0/3452067230 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b9481ec60 con 0x559b94ab6be0 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c87fff700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8ed9d700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ac3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c8e59c700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 0x559b94ac7520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c80099de0 tx=0x7f3c80009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b0735ed20ae8cf1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c75ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c75ffb700 1 -- 172.21.15.31:0/3452067230 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c800b8300 con 0x559b94ab6be0 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c75ffb700 1 -- 172.21.15.31:0/3452067230 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3c80021c50 con 0x559b94ab6be0 2023-07-29T16:26:50.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c78011d80 con 0x559b94ab6be0 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3c75ffb700 1 -- 172.21.15.31:0/3452067230 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c800b8460 con 0x559b94ab6be0 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3ca898ab80 10 client.28906 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.297+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f3c901a07c0 con 0x559b94ab6be0 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3c75ffb700 1 -- 172.21.15.31:0/3452067230 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c78011d80 con 0x559b94ab6be0 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3c75ffb700 1 -- 172.21.15.31:0/3452067230 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3c8009a340 con 0x559b94ab6be0 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3ca898ab80 10 client.28906 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:50.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3ca898ab80 10 client.28906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3ca898ab80 20 client.28906 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94b1f120 0x559b94b21530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3ca898ab80 4 client.28906 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559b94a00d40 con 0x559b94b1f120 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.298+0000 7f3c87fff700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94b1f120 0x559b94b21530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3c87fff700 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94b1f120 0x559b94b21530 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f66228f47358c30a server_cookie=1e0914bf543e5e6 in_seq=0 out_seq=0 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3c75ffb700 10 client.28906 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:50.305 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3c75ffb700 1 -- 172.21.15.31:0/3452067230 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559b94a00d40 con 0x559b94b1f120 2023-07-29T16:26:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3c75ffb700 10 client.28906 handle_command_reply: tid=0 2023-07-29T16:26:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 client.28906 shutdown 2023-07-29T16:26:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 msgr2=0x559b94ac7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 0x559b94ac7520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c80099de0 tx=0x7f3c80009f50 comp rx=0 tx=0).stop 2023-07-29T16:26:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 shutdown_connections 2023-07-29T16:26:50.306 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b94b1f120 0x559b94b21530 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b94ab82f0 0x559b94ac3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b94ab6be0 0x559b94ac7520 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 --2- 172.21.15.31:0/3452067230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b94ab7530 0x559b94ac7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3c75ffb700 0 client.28906 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 >> 172.21.15.31:0/3452067230 conn(0x559b94aac3e0 msgr2=0x559b94abf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 shutdown_connections 2023-07-29T16:26:50.307 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 1 -- 172.21.15.31:0/3452067230 wait complete. 2023-07-29T16:26:50.308 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.299+0000 7f3ca898ab80 20 client.28906 trim_cache size 0 max 16384 2023-07-29T16:26:50.308 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3c7c0a0810 msgr2=0x7f3c7c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3c7c0a0810 0x7f3c7c0a2cc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009fd0 tx=0x7f3c88004b40 comp rx=0 tx=0).stop 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 msgr2=0x7f3c901982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 0x7f3c901982d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3c80009230 tx=0x7f3c8000b790 comp rx=0 tx=0).stop 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 shutdown_connections 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3c7c0a0810 0x7f3c7c0a2cc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c90108220 0x7f3c9019f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c90073750 0x7f3c901982d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 --2- 172.21.15.31:0/3230986115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c90074d30 0x7f3c90198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 >> 172.21.15.31:0/3230986115 conn(0x7f3c900fb9b0 msgr2=0x7f3c900fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 shutdown_connections 2023-07-29T16:26:50.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:50.300+0000 7f3c94fff700 1 -- 172.21.15.31:0/3230986115 wait complete. 2023-07-29T16:26:50.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:49 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2352829495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:26:50.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:49 smithi156 ceph-mon[129245]: pgmap v712: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 2.8 MiB/s wr, 315 op/s 2023-07-29T16:26:50.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 852833 8.90 MB/sec execute 395 sec latency 13.232 ms 2023-07-29T16:26:51.319 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:26:51.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:26:51.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.556+0000 7f78055ff700 1 -- 172.21.15.31:0/2083966313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7800102550 msgr2=0x7f7800102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.556+0000 7f78055ff700 1 --2- 172.21.15.31:0/2083966313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7800102550 0x7f7800102960 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f77f0009990 tx=0x7f77f0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 -- 172.21.15.31:0/2083966313 shutdown_connections 2023-07-29T16:26:51.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 --2- 172.21.15.31:0/2083966313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78001040d0 0x7f780010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 --2- 172.21.15.31:0/2083966313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7800102550 0x7f7800102960 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 --2- 172.21.15.31:0/2083966313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f7800103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 -- 172.21.15.31:0/2083966313 >> 172.21.15.31:0/2083966313 conn(0x7f78000fdb90 msgr2=0x7f78000fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 -- 172.21.15.31:0/2083966313 shutdown_connections 2023-07-29T16:26:51.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.557+0000 7f78055ff700 1 -- 172.21.15.31:0/2083966313 wait complete. 2023-07-29T16:26:51.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 Processor -- start 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 -- start start 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7800102550 0x7f78001982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f78001987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78001040d0 0x7f780019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780019fe30 con 0x7f78001040d0 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780019ff70 con 0x7f7800103740 2023-07-29T16:26:51.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f78055ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78001a0100 con 0x7f7800102550 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.558+0000 7f77fe7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f78001987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78001040d0 0x7f780019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77feffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7800102550 0x7f78001982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f78001987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39586/0 (socket says 172.21.15.31:39586) 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 -- 172.21.15.31:0/1716883020 learned_addr learned my addr 172.21.15.31:0/1716883020 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 -- 172.21.15.31:0/1716883020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7800102550 msgr2=0x7f78001982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7800102550 0x7f78001982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 -- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78001040d0 msgr2=0x7f780019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78001040d0 0x7f780019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 -- 172.21.15.31:0/1716883020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f00095f0 con 0x7f7800103740 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78001040d0 0x7f780019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77feffd700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7800102550 0x7f78001982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f78001987f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f77f400e9a0 tx=0x7f77f400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.559+0000 7f77e7fff700 1 -- 172.21.15.31:0/1716883020 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4009790 con 0x7f7800103740 2023-07-29T16:26:51.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.560+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78001a03e0 con 0x7f7800103740 2023-07-29T16:26:51.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.560+0000 7f77e7fff700 1 -- 172.21.15.31:0/1716883020 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f4010430 con 0x7f7800103740 2023-07-29T16:26:51.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.560+0000 7f77e7fff700 1 -- 172.21.15.31:0/1716883020 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4018680 con 0x7f7800103740 2023-07-29T16:26:51.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.560+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78001a0940 con 0x7f7800103740 2023-07-29T16:26:51.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.563+0000 7f77e7fff700 1 -- 172.21.15.31:0/1716883020 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f77f40105a0 con 0x7f7800103740 2023-07-29T16:26:51.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.564+0000 7f7818e31b80 1 Processor -- start 2023-07-29T16:26:51.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.564+0000 7f7818e31b80 1 -- start start 2023-07-29T16:26:51.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.564+0000 7f77e7fff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f77e80a0790 0x7f77e80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77e7fff700 1 -- 172.21.15.31:0/1716883020 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f77f4014070 con 0x7f7800103740 2023-07-29T16:26:51.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 0x560ecf14c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf14cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14d370 0x560ecf14d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f7818e31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf0a3a90 con 0x560ecf14d370 2023-07-29T16:26:51.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f7818e31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf0a1450 con 0x560ecf14c5b0 2023-07-29T16:26:51.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f7818e31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf09cd40 con 0x560ecf14bc60 2023-07-29T16:26:51.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f77e80a0790 0x7f77e80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 0x560ecf14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 0x560ecf14c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57328/0 (socket says 172.21.15.31:57328) 2023-07-29T16:26:51.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 -- 172.21.15.31:0/3069905305 learned_addr learned my addr 172.21.15.31:0/3069905305 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf14cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14d370 0x560ecf14d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 -- 172.21.15.31:0/3069905305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 msgr2=0x560ecf14cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf14cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 -- 172.21.15.31:0/3069905305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14d370 msgr2=0x560ecf14d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14d370 0x560ecf14d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.565+0000 7f77feffd700 1 -- 172.21.15.31:0/3069905305 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eceecae60 con 0x560ecf14bc60 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf14cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14d370 0x560ecf14d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:26:51.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77feffd700 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 0x560ecf14c070 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f78001035a0 tx=0x7f77f0005ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eec923847354d8df server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f77e80a0790 0x7f77e80a2c40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009e90 tx=0x7f77f8009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77e57fa700 1 -- 172.21.15.31:0/3069905305 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f001d070 con 0x560ecf14bc60 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77e57fa700 1 -- 172.21.15.31:0/3069905305 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f001e040 con 0x560ecf14bc60 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.566+0000 7f77e57fa700 1 -- 172.21.15.31:0/3069905305 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0004e50 con 0x560ecf14bc60 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 -- 172.21.15.31:0/3069905305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 msgr2=0x560ecf14c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 0x560ecf14c070 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f78001035a0 tx=0x7f77f0005ee0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 -- 172.21.15.31:0/3069905305 shutdown_connections 2023-07-29T16:26:51.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14d370 0x560ecf14d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14bc60 0x560ecf14c070 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3069905305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf14cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 -- 172.21.15.31:0/3069905305 >> 172.21.15.31:0/3069905305 conn(0x560ecf141460 msgr2=0x560ecf146a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 -- 172.21.15.31:0/3069905305 shutdown_connections 2023-07-29T16:26:51.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.567+0000 7f7818e31b80 1 -- 172.21.15.31:0/3069905305 wait complete. 2023-07-29T16:26:51.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.568+0000 7f7818e31b80 1 -- start start 2023-07-29T16:26:51.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.568+0000 7f7818e31b80 1 Processor -- start 2023-07-29T16:26:51.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14c5b0 0x560ecf155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14d370 0x560ecf15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf039590 con 0x560ecf14bc60 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf039230 con 0x560ecf14d370 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecef209a0 con 0x560ecf14c5b0 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f7818e31b80 10 client.0 _ll_register_callbacks cb 0x560ecf0426a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f77feffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f77ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14d370 0x560ecf15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f77feffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf1557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34300/0 (socket says 172.21.15.31:34300) 2023-07-29T16:26:51.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f77feffd700 1 -- 172.21.15.31:0/3300067172 learned_addr learned my addr 172.21.15.31:0/3300067172 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.569+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14c5b0 0x560ecf155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77ff7fe700 1 -- 172.21.15.31:0/3300067172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14c5b0 msgr2=0x560ecf155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14c5b0 0x560ecf155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77ff7fe700 1 -- 172.21.15.31:0/3300067172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 msgr2=0x560ecf1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf1557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77ff7fe700 1 -- 172.21.15.31:0/3300067172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780010b2c0 con 0x560ecf14d370 2023-07-29T16:26:51.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14d370 0x560ecf15cd40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009fd0 tx=0x7f77f8004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771e4f0d713884d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77e2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77e2ff5700 1 -- 172.21.15.31:0/3300067172 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f801abb0 con 0x560ecf14d370 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77e2ff5700 1 -- 172.21.15.31:0/3300067172 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f77f801ad10 con 0x560ecf14d370 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ecf0c7d90 con 0x560ecf14d370 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f77e2ff5700 1 -- 172.21.15.31:0/3300067172 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77f8018440 con 0x560ecf14d370 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f7818e31b80 10 client.28916 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.570+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x560eceeb3ce0 con 0x560ecf14d370 2023-07-29T16:26:51.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f77e2ff5700 1 -- 172.21.15.31:0/3300067172 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560ecf0c7d90 con 0x560ecf14d370 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f77e2ff5700 1 -- 172.21.15.31:0/3300067172 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f77f80244f0 con 0x560ecf14d370 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f7818e31b80 10 client.28916 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f7818e31b80 10 client.28916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f7818e31b80 20 client.28916 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1615c0 0x560ecf1639d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f7818e31b80 4 client.28916 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:51.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ecf11adf0 con 0x560ecf1615c0 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.571+0000 7f77feffd700 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1615c0 0x560ecf1639d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.572+0000 7f77feffd700 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1615c0 0x560ecf1639d0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65d874b3344c4dc server_cookie=97b957f8b39a4027 in_seq=0 out_seq=0 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.572+0000 7f77e2ff5700 10 client.28916 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f77e2ff5700 1 -- 172.21.15.31:0/3300067172 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560ecf11adf0 con 0x560ecf1615c0 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f77e2ff5700 10 client.28916 handle_command_reply: tid=0 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 client.28916 shutdown 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14d370 msgr2=0x560ecf15cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14d370 0x560ecf15cd40 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009fd0 tx=0x7f77f8004ad0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 shutdown_connections 2023-07-29T16:26:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1615c0 0x560ecf1639d0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14d370 0x560ecf15cd40 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3300067172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14c5b0 0x560ecf155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f77e2ff5700 0 client.28916 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 >> 172.21.15.31:0/3300067172 conn(0x560ecf141460 msgr2=0x560ecf1460e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 shutdown_connections 2023-07-29T16:26:51.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 1 -- 172.21.15.31:0/3300067172 wait complete. 2023-07-29T16:26:51.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.573+0000 7f7818e31b80 20 client.28916 trim_cache size 0 max 16384 2023-07-29T16:26:51.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.592+0000 7f7818e31b80 1 Processor -- start 2023-07-29T16:26:51.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.592+0000 7f7818e31b80 1 -- start start 2023-07-29T16:26:51.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 0x560ecf150270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14c5b0 0x560ecf15cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf1507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f7818e31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf1434e0 con 0x560ecf14c5b0 2023-07-29T16:26:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f7818e31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf1612b0 con 0x560ecf14bc60 2023-07-29T16:26:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f7818e31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf154c40 con 0x560ecf14d370 2023-07-29T16:26:51.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 0x560ecf150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 0x560ecf150270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39596/0 (socket says 172.21.15.31:39596) 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 -- 172.21.15.31:0/3215848542 learned_addr learned my addr 172.21.15.31:0/3215848542 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf1507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14c5b0 0x560ecf15cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 -- 172.21.15.31:0/3215848542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 msgr2=0x560ecf1507b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf1507b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 -- 172.21.15.31:0/3215848542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14c5b0 msgr2=0x560ecf15cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14c5b0 0x560ecf15cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77feffd700 1 -- 172.21.15.31:0/3215848542 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eceef12d0 con 0x560ecf14bc60 2023-07-29T16:26:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.593+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14c5b0 0x560ecf15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:51.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.594+0000 7f77feffd700 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 0x560ecf150270 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f77f0005f50 tx=0x7f77f001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d71cad59db13840 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.594+0000 7f77e5ffb700 1 -- 172.21.15.31:0/3215848542 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f0032070 con 0x560ecf14bc60 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.594+0000 7f77e5ffb700 1 -- 172.21.15.31:0/3215848542 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f000be80 con 0x560ecf14bc60 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.594+0000 7f77e5ffb700 1 -- 172.21.15.31:0/3215848542 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f001d040 con 0x560ecf14bc60 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 -- 172.21.15.31:0/3215848542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 msgr2=0x560ecf150270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 0x560ecf150270 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f77f0005f50 tx=0x7f77f001a040 comp rx=0 tx=0).stop 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 -- 172.21.15.31:0/3215848542 shutdown_connections 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf1507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14bc60 0x560ecf150270 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 --2- 172.21.15.31:0/3215848542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14c5b0 0x560ecf15cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 -- 172.21.15.31:0/3215848542 >> 172.21.15.31:0/3215848542 conn(0x560ecf141460 msgr2=0x560ecf153720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 -- 172.21.15.31:0/3215848542 shutdown_connections 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 -- 172.21.15.31:0/3215848542 wait complete. 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.595+0000 7f7818e31b80 1 -- start start 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 Processor -- start 2023-07-29T16:26:51.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf15aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf15bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf15c040 con 0x560ecf14bc60 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf15c1b0 con 0x560ecf14c5b0 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ecf15c370 con 0x560ecf14d370 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f7818e31b80 10 client.0 _ll_register_callbacks cb 0x560ecf0426a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:51.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f77fe7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf15aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f77feffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f77fe7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf15aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39604/0 (socket says 172.21.15.31:39604) 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f77fe7fc700 1 -- 172.21.15.31:0/2474512395 learned_addr learned my addr 172.21.15.31:0/2474512395 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f77feffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf15a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34312/0 (socket says 172.21.15.31:34312) 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.596+0000 7f77ff7fe700 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf15bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77fe7fc700 1 -- 172.21.15.31:0/2474512395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 msgr2=0x560ecf15bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf15bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77fe7fc700 1 -- 172.21.15.31:0/2474512395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 msgr2=0x560ecf15a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf15a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77fe7fc700 1 -- 172.21.15.31:0/2474512395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560eceecae60 con 0x560ecf14c5b0 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77fe7fc700 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf15aaf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f77f4099410 tx=0x7f77f4099480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f50aff9f77aa84 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77e2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77e2ff5700 1 -- 172.21.15.31:0/2474512395 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f40a18d0 con 0x560ecf14c5b0 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77e2ff5700 1 -- 172.21.15.31:0/2474512395 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f77f40b0050 con 0x560ecf14c5b0 2023-07-29T16:26:51.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77f001e040 con 0x560ecf14c5b0 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f77e2ff5700 1 -- 172.21.15.31:0/2474512395 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77f4010880 con 0x560ecf14c5b0 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f7818e31b80 10 client.28922 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.597+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x560eceec9330 con 0x560ecf14c5b0 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f77e2ff5700 1 -- 172.21.15.31:0/2474512395 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77f4063080 con 0x560ecf14c5b0 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f77e2ff5700 1 -- 172.21.15.31:0/2474512395 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f77f40a1440 con 0x560ecf14c5b0 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f7818e31b80 10 client.28922 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:51.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f7818e31b80 10 client.28922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f7818e31b80 20 client.28922 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f7818e31b80 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1b5b10 0x560ecf1b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f7818e31b80 4 client.28922 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560ecf095dc0 con 0x560ecf1b5b10 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f77feffd700 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1b5b10 0x560ecf1b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f77e2ff5700 10 client.28922 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.598+0000 7f77feffd700 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1b5b10 0x560ecf1b7f20 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba4056b4c31acb5 server_cookie=42e12f049028f50 in_seq=0 out_seq=0 2023-07-29T16:26:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f77e2ff5700 1 -- 172.21.15.31:0/2474512395 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560ecf095dc0 con 0x560ecf1b5b10 2023-07-29T16:26:51.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f77e2ff5700 10 client.28922 handle_command_reply: tid=0 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 client.28922 shutdown 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 msgr2=0x560ecf15aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf15aaf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f77f4099410 tx=0x7f77f4099480 comp rx=0 tx=0).stop 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 shutdown_connections 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560ecf1b5b10 0x560ecf1b7f20 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560ecf14d370 0x560ecf15bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ecf14bc60 0x560ecf15a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f77e2ff5700 0 client.28922 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 --2- 172.21.15.31:0/2474512395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560ecf14c5b0 0x560ecf15aaf0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 >> 172.21.15.31:0/2474512395 conn(0x560ecf141460 msgr2=0x560ecf153720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 shutdown_connections 2023-07-29T16:26:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 1 -- 172.21.15.31:0/2474512395 wait complete. 2023-07-29T16:26:51.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.600+0000 7f7818e31b80 20 client.28922 trim_cache size 0 max 16384 2023-07-29T16:26:51.608 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:26:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:26:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "3abf07da-9149-4b2a-8222-3ba61faecd17", 2023-07-29T16:26:51.608 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:26:51.608 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:26:51.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f77e80a0790 msgr2=0x7f77e80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f77e80a0790 0x7f77e80a2c40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f77f8009e90 tx=0x7f77f8009920 comp rx=0 tx=0).stop 2023-07-29T16:26:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 msgr2=0x7f78001987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f78001987f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f77f400e9a0 tx=0x7f77f400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 shutdown_connections 2023-07-29T16:26:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f77e80a0790 0x7f77e80a2c40 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f78001040d0 0x7f780019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7800102550 0x7f78001982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 --2- 172.21.15.31:0/1716883020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7800103740 0x7f78001987f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 >> 172.21.15.31:0/1716883020 conn(0x7f78000fdb90 msgr2=0x7f78000ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 shutdown_connections 2023-07-29T16:26:51.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.601+0000 7f78055ff700 1 -- 172.21.15.31:0/1716883020 wait complete. 2023-07-29T16:26:51.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:26:51.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 -- 172.21.15.31:0/4098673909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b41036c0 msgr2=0x7f24b4105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/4098673909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b41036c0 0x7f24b4105aa0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009990 tx=0x7f24a8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 -- 172.21.15.31:0/4098673909 shutdown_connections 2023-07-29T16:26:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/4098673909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 0x7f24b4108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/4098673909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b4100da0 0x7f24b4103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/4098673909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b41036c0 0x7f24b4105aa0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 -- 172.21.15.31:0/4098673909 >> 172.21.15.31:0/4098673909 conn(0x7f24b40fa840 msgr2=0x7f24b40fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 -- 172.21.15.31:0/4098673909 shutdown_connections 2023-07-29T16:26:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.836+0000 7f24ba5ff700 1 -- 172.21.15.31:0/4098673909 wait complete. 2023-07-29T16:26:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.837+0000 7f24ba5ff700 1 Processor -- start 2023-07-29T16:26:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.837+0000 7f24ba5ff700 1 -- start start 2023-07-29T16:26:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24ba5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 0x7f24b41982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24ba5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b41036c0 0x7f24b41987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24ba5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 0x7f24b419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24ba5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b419fe30 con 0x7f24b4105fe0 2023-07-29T16:26:51.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24ba5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b419ff70 con 0x7f24b4100da0 2023-07-29T16:26:51.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24ba5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b41a0100 con 0x7f24b41036c0 2023-07-29T16:26:51.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 0x7f24b41982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 0x7f24b41982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39614/0 (socket says 172.21.15.31:39614) 2023-07-29T16:26:51.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 -- 172.21.15.31:0/533094488 learned_addr learned my addr 172.21.15.31:0/533094488 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b37fe700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b41036c0 0x7f24b41987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b8b9c700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 0x7f24b419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 -- 172.21.15.31:0/533094488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b41036c0 msgr2=0x7f24b41987f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b41036c0 0x7f24b41987f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 -- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 msgr2=0x7f24b419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 0x7f24b419f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b3fff700 1 -- 172.21.15.31:0/533094488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80095f0 con 0x7f24b4100da0 2023-07-29T16:26:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.838+0000 7f24b8b9c700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 0x7f24b419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.839+0000 7f24b3fff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 0x7f24b41982b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e9a0 tx=0x7f24a400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.839+0000 7f24b17fa700 1 -- 172.21.15.31:0/533094488 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4009790 con 0x7f24b4100da0 2023-07-29T16:26:51.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.839+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b41a03e0 con 0x7f24b4100da0 2023-07-29T16:26:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.839+0000 7f24b17fa700 1 -- 172.21.15.31:0/533094488 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a400cbd0 con 0x7f24b4100da0 2023-07-29T16:26:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.839+0000 7f24b17fa700 1 -- 172.21.15.31:0/533094488 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4018680 con 0x7f24b4100da0 2023-07-29T16:26:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.839+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b41a08f0 con 0x7f24b4100da0 2023-07-29T16:26:51.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.842+0000 7f24b17fa700 1 -- 172.21.15.31:0/533094488 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f24a400cd40 con 0x7f24b4100da0 2023-07-29T16:26:51.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.844+0000 7f24b17fa700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0790 0x7f249c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.844+0000 7f24cdf5bb80 1 Processor -- start 2023-07-29T16:26:51.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.844+0000 7f24b17fa700 1 -- 172.21.15.31:0/533094488 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f24a4014070 con 0x7f24b4100da0 2023-07-29T16:26:51.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.844+0000 7f24b37fe700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0790 0x7f249c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.855 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 -- start start 2023-07-29T16:26:51.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd651ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd652d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd653740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd5a9a10 con 0x563efd6532f0 2023-07-29T16:26:51.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd5a73d0 con 0x563efd652530 2023-07-29T16:26:51.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd5a2cc0 con 0x563efd651be0 2023-07-29T16:26:51.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24b37fe700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0790 0x7f249c0a2c40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f24a800b2e0 tx=0x7f24a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd651ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd651ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57380/0 (socket says 172.21.15.31:57380) 2023-07-29T16:26:51.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24b3fff700 1 -- 172.21.15.31:0/1160666185 learned_addr learned my addr 172.21.15.31:0/1160666185 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd652d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.845+0000 7f24b8b9c700 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd653740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f24b3fff700 1 -- 172.21.15.31:0/1160666185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 msgr2=0x563efd652d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f24b3fff700 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd652d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.859 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f24b3fff700 1 -- 172.21.15.31:0/1160666185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 msgr2=0x563efd653740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f24b3fff700 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd653740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f24b3fff700 1 -- 172.21.15.31:0/1160666185 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563efd3d0de0 con 0x563efd651be0 2023-07-29T16:26:51.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f24b3fff700 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd651ff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f24a40994a0 tx=0x7f24a40990d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=604ff170bfaa070e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f249a7fc700 1 -- 172.21.15.31:0/1160666185 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4099510 con 0x563efd651be0 2023-07-29T16:26:51.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f249a7fc700 1 -- 172.21.15.31:0/1160666185 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a409d540 con 0x563efd651be0 2023-07-29T16:26:51.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.846+0000 7f249a7fc700 1 -- 172.21.15.31:0/1160666185 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4098c60 con 0x563efd651be0 2023-07-29T16:26:51.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1160666185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 msgr2=0x563efd651ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.861 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd651ff0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f24a40994a0 tx=0x7f24a40990d0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1160666185 shutdown_connections 2023-07-29T16:26:51.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd653740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd651ff0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1160666185 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd652d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1160666185 >> 172.21.15.31:0/1160666185 conn(0x563efd6473e0 msgr2=0x563efd64c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1160666185 shutdown_connections 2023-07-29T16:26:51.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.847+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1160666185 wait complete. 2023-07-29T16:26:51.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 -- start start 2023-07-29T16:26:51.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 Processor -- start 2023-07-29T16:26:51.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd65b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd662d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd53f510 con 0x563efd6532f0 2023-07-29T16:26:51.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd53f1b0 con 0x563efd652530 2023-07-29T16:26:51.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.848+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd426920 con 0x563efd651be0 2023-07-29T16:26:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24cdf5bb80 10 client.0 _ll_register_callbacks cb 0x563efd548620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd65b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd65b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57382/0 (socket says 172.21.15.31:57382) 2023-07-29T16:26:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 -- 172.21.15.31:0/1835960771 learned_addr learned my addr 172.21.15.31:0/1835960771 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b8b9c700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd662d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 -- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 msgr2=0x563efd65bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 -- 172.21.15.31:0/1835960771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 msgr2=0x563efd662d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd662d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 -- 172.21.15.31:0/1835960771 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b41a08f0 con 0x563efd651be0 2023-07-29T16:26:51.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b8b9c700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd662d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:26:51.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24b3fff700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd65b810 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f24a4095f80 tx=0x7f24a40a1960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23d9350602ec0dce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24937fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:51.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.849+0000 7f24937fe700 1 -- 172.21.15.31:0/1835960771 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a40a16d0 con 0x563efd651be0 2023-07-29T16:26:51.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563efd5cdd10 con 0x563efd651be0 2023-07-29T16:26:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24937fe700 1 -- 172.21.15.31:0/1835960771 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f24a4095300 con 0x563efd651be0 2023-07-29T16:26:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24937fe700 1 -- 172.21.15.31:0/1835960771 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24a40954c0 con 0x563efd651be0 2023-07-29T16:26:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 10 client.28936 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:51.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563efd3b9c60 con 0x563efd651be0 2023-07-29T16:26:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24937fe700 1 -- 172.21.15.31:0/1835960771 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563efd5cdd10 con 0x563efd651be0 2023-07-29T16:26:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24937fe700 1 -- 172.21.15.31:0/1835960771 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f24a4099c10 con 0x563efd651be0 2023-07-29T16:26:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 10 client.28936 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:51.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 10 client.28936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:51.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 20 client.28936 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:51.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd667220 0x563efd669630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 4 client.28936 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.850+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563efd620d70 con 0x563efd667220 2023-07-29T16:26:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.851+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd667220 0x563efd669630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.851+0000 7f24b37fe700 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd667220 0x563efd669630 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2db162b2fb02443 server_cookie=17463945299b781e in_seq=0 out_seq=0 2023-07-29T16:26:51.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.851+0000 7f24937fe700 10 client.28936 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24937fe700 1 -- 172.21.15.31:0/1835960771 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563efd620d70 con 0x563efd667220 2023-07-29T16:26:51.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24937fe700 10 client.28936 handle_command_reply: tid=0 2023-07-29T16:26:51.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 client.28936 shutdown 2023-07-29T16:26:51.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 msgr2=0x563efd65b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd65b810 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f24a4095f80 tx=0x7f24a40a1960 comp rx=0 tx=0).stop 2023-07-29T16:26:51.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 shutdown_connections 2023-07-29T16:26:51.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd667220 0x563efd669630 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd662d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd65b810 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24937fe700 0 client.28936 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/1835960771 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 >> 172.21.15.31:0/1835960771 conn(0x563efd6473e0 msgr2=0x563efd64c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 shutdown_connections 2023-07-29T16:26:51.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.852+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/1835960771 wait complete. 2023-07-29T16:26:51.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.853+0000 7f24cdf5bb80 20 client.28936 trim_cache size 0 max 16384 2023-07-29T16:26:51.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 Processor -- start 2023-07-29T16:26:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 -- start start 2023-07-29T16:26:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd651be0 0x563efd64c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd662d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd6532f0 0x563efd659740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd5cdd10 con 0x563efd651be0 2023-07-29T16:26:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd65ca00 con 0x563efd652530 2023-07-29T16:26:51.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd65acc0 con 0x563efd6532f0 2023-07-29T16:26:51.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd662d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24b8b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd6532f0 0x563efd659740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd662d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39636/0 (socket says 172.21.15.31:39636) 2023-07-29T16:26:51.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24b8b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd6532f0 0x563efd659740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57384/0 (socket says 172.21.15.31:57384) 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd651be0 0x563efd64c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.871+0000 7f24b37fe700 1 -- 172.21.15.31:0/2648739810 learned_addr learned my addr 172.21.15.31:0/2648739810 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f24b37fe700 1 -- 172.21.15.31:0/2648739810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd6532f0 msgr2=0x563efd659740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd6532f0 0x563efd659740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f24b37fe700 1 -- 172.21.15.31:0/2648739810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd651be0 msgr2=0x563efd64c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd651be0 0x563efd64c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f24b37fe700 1 -- 172.21.15.31:0/2648739810 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563efd3f7250 con 0x563efd652530 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd662d50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009990 tx=0x7f24a8020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ddb5bcd419a65e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f249affd700 1 -- 172.21.15.31:0/2648739810 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8005030 con 0x563efd652530 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f249affd700 1 -- 172.21.15.31:0/2648739810 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a801e040 con 0x563efd652530 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.872+0000 7f249affd700 1 -- 172.21.15.31:0/2648739810 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a800f560 con 0x563efd652530 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2648739810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 msgr2=0x563efd662d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd662d50 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009990 tx=0x7f24a8020000 comp rx=0 tx=0).stop 2023-07-29T16:26:51.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2648739810 shutdown_connections 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd6532f0 0x563efd659740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd651be0 0x563efd64c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2648739810 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd662d50 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.873+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2648739810 >> 172.21.15.31:0/2648739810 conn(0x563efd6473e0 msgr2=0x563efd65a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.874+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2648739810 shutdown_connections 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.874+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2648739810 wait complete. 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.874+0000 7f24cdf5bb80 1 -- start start 2023-07-29T16:26:51.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.874+0000 7f24cdf5bb80 1 Processor -- start 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd662cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd65fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd660380 con 0x563efd6532f0 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd6b8da0 con 0x563efd652530 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563efd6b8ee0 con 0x563efd651be0 2023-07-29T16:26:51.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24cdf5bb80 10 client.0 _ll_register_callbacks cb 0x563efd548620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd662cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd662cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57396/0 (socket says 172.21.15.31:57396) 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 -- 172.21.15.31:0/2650429797 learned_addr learned my addr 172.21.15.31:0/2650429797 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b8b9c700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd65fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 -- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 msgr2=0x563efd65f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 -- 172.21.15.31:0/2650429797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 msgr2=0x563efd65fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd65fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b3fff700 1 -- 172.21.15.31:0/2650429797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b4109030 con 0x563efd651be0 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.875+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24b3fff700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd662cc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f24a4099fd0 tx=0x7f24a409d3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d86ebb453c6c69 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24937fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24937fe700 1 -- 172.21.15.31:0/2650429797 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a409a2f0 con 0x563efd651be0 2023-07-29T16:26:51.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24937fe700 1 -- 172.21.15.31:0/2650429797 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f24a409a450 con 0x563efd651be0 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24a409d540 con 0x563efd651be0 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24937fe700 1 -- 172.21.15.31:0/2650429797 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24a409a610 con 0x563efd651be0 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 10 client.28942 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563efd3cf2b0 con 0x563efd651be0 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24937fe700 1 -- 172.21.15.31:0/2650429797 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f24a409d540 con 0x563efd651be0 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24937fe700 1 -- 172.21.15.31:0/2650429797 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f24a40c39f0 con 0x563efd651be0 2023-07-29T16:26:51.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 10 client.28942 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 10 client.28942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 20 client.28942 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd6ba3c0 0x563efd6bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 4 client.28942 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.876+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563efd59bd40 con 0x563efd6ba3c0 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.877+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd6ba3c0 0x563efd6bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.877+0000 7f24b37fe700 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd6ba3c0 0x563efd6bc7a0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6244e2b427ed5113 server_cookie=6ba1a51ad4384174 in_seq=0 out_seq=0 2023-07-29T16:26:51.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.877+0000 7f24937fe700 10 client.28942 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.878+0000 7f24937fe700 1 -- 172.21.15.31:0/2650429797 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563efd59bd40 con 0x563efd6ba3c0 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.878+0000 7f24937fe700 10 client.28942 handle_command_reply: tid=0 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.878+0000 7f24cdf5bb80 1 client.28942 shutdown 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.878+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 msgr2=0x563efd662cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.878+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd662cc0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f24a4099fd0 tx=0x7f24a409d3c0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 shutdown_connections 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563efd6ba3c0 0x563efd6bc7a0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563efd6532f0 0x563efd65fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563efd651be0 0x563efd662cc0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 --2- 172.21.15.31:0/2650429797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563efd652530 0x563efd65f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24937fe700 0 client.28942 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 >> 172.21.15.31:0/2650429797 conn(0x563efd6473e0 msgr2=0x563efd64c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 shutdown_connections 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 1 -- 172.21.15.31:0/2650429797 wait complete. 2023-07-29T16:26:51.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24cdf5bb80 20 client.28942 trim_cache size 0 max 16384 2023-07-29T16:26:51.897 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:26:51.897 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:26:51.897 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:26:51.897 INFO:teuthology.orchestra.run.smithi031.stdout: "3abf07da-9149-4b2a-8222-3ba61faecd17": { 2023-07-29T16:26:51.897 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:26:51.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "3abf07da-9149-4b2a-8222-3ba61faecd17", 2023-07-29T16:26:51.898 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:26:51.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:26:51.898 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:26:51.898 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:26:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0790 msgr2=0x7f249c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0790 0x7f249c0a2c40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f24a800b2e0 tx=0x7f24a8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 msgr2=0x7f24b41982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.879+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 0x7f24b41982b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e9a0 tx=0x7f24a400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 shutdown_connections 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f249c0a0790 0x7f249c0a2c40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24b4105fe0 0x7f24b419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24b4100da0 0x7f24b41982b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 --2- 172.21.15.31:0/533094488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24b41036c0 0x7f24b41987f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 >> 172.21.15.31:0/533094488 conn(0x7f24b40fa840 msgr2=0x7f24b40ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 shutdown_connections 2023-07-29T16:26:51.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:26:51.880+0000 7f24ba5ff700 1 -- 172.21.15.31:0/533094488 wait complete. 2023-07-29T16:26:51.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 855011 8.91 MB/sec execute 396 sec latency 13.853 ms 2023-07-29T16:26:51.901 INFO:tasks.cephfs.filesystem:scrub status for tag:3abf07da-9149-4b2a-8222-3ba61faecd17 - {'path': '/', 'tag': '3abf07da-9149-4b2a-8222-3ba61faecd17', 'options': 'recursive,force'} 2023-07-29T16:26:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: pgmap v713: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.8 MiB/s wr, 224 op/s 2023-07-29T16:26:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:26:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:52.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:52.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:52.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:52 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:52.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 856249 8.90 MB/sec execute 397 sec latency 19.665 ms 2023-07-29T16:26:52.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: pgmap v713: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.8 MiB/s wr, 224 op/s 2023-07-29T16:26:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:26:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:52 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:53.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: pgmap v713: 97 pgs: 97 active+clean; 161 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 1.8 MiB/s wr, 224 op/s 2023-07-29T16:26:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:26:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:53.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:52 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:53 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:26:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:53 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:53.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:53 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:26:53.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 858260 8.90 MB/sec execute 398 sec latency 45.768 ms 2023-07-29T16:26:53.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:53 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:26:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:53 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:53.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:53 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:26:54.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:53 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:26:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:53 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:26:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:53 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:26:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: pgmap v714: 97 pgs: 97 active+clean; 163 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.4 MiB/s wr, 290 op/s 2023-07-29T16:26:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:26:54.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:26:54.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:54 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 859581 8.91 MB/sec execute 399 sec latency 30.855 ms 2023-07-29T16:26:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: pgmap v714: 97 pgs: 97 active+clean; 163 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.4 MiB/s wr, 290 op/s 2023-07-29T16:26:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:26:54.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:26:54.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:54 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:55.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: pgmap v714: 97 pgs: 97 active+clean; 163 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.4 MiB/s wr, 290 op/s 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:26:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:54 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:26:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 -- 172.21.15.31:0/1427917570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0100e60 msgr2=0x7fa0c0103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/1427917570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0100e60 0x7fa0c0103240 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009990 tx=0x7fa0b0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 -- 172.21.15.31:0/1427917570 shutdown_connections 2023-07-29T16:26:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/1427917570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 0x7fa0c0108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/1427917570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0100e60 0x7fa0c0103240 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/1427917570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0103780 0x7fa0c0105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 -- 172.21.15.31:0/1427917570 >> 172.21.15.31:0/1427917570 conn(0x7fa0c00fa840 msgr2=0x7fa0c00fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 -- 172.21.15.31:0/1427917570 shutdown_connections 2023-07-29T16:26:55.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.124+0000 7fa0c4bff700 1 -- 172.21.15.31:0/1427917570 wait complete. 2023-07-29T16:26:55.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.125+0000 7fa0c4bff700 1 Processor -- start 2023-07-29T16:26:55.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 -- start start 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 0x7fa0c01961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0103780 0x7fa0c0196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 0x7fa0c0197580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0197be0 con 0x7fa0c0103780 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0197d50 con 0x7fa0c01060a0 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0c4bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0197f10 con 0x7fa0c0100e60 2023-07-29T16:26:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 0x7fa0c01961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0bed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 0x7fa0c0197580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0bdd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0103780 0x7fa0c0196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 0x7fa0c01961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57422/0 (socket says 172.21.15.31:57422) 2023-07-29T16:26:55.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.126+0000 7fa0be59c700 1 -- 172.21.15.31:0/507107584 learned_addr learned my addr 172.21.15.31:0/507107584 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:55.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0be59c700 1 -- 172.21.15.31:0/507107584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 msgr2=0x7fa0c0197580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0be59c700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 0x7fa0c0197580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0be59c700 1 -- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0103780 msgr2=0x7fa0c0196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0be59c700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0103780 0x7fa0c0196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0be59c700 1 -- 172.21.15.31:0/507107584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00095f0 con 0x7fa0c0100e60 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 0x7fa0c0197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0103780 0x7fa0c0196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0be59c700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 0x7fa0c01961c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0005720 tx=0x7fa0b0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0af7fe700 1 -- 172.21.15.31:0/507107584 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b001d070 con 0x7fa0c0100e60 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c0198190 con 0x7fa0c0100e60 2023-07-29T16:26:55.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0af7fe700 1 -- 172.21.15.31:0/507107584 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0b0021410 con 0x7fa0c0100e60 2023-07-29T16:26:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.127+0000 7fa0af7fe700 1 -- 172.21.15.31:0/507107584 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0017650 con 0x7fa0c0100e60 2023-07-29T16:26:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.128+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c019e9b0 con 0x7fa0c0100e60 2023-07-29T16:26:55.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.131+0000 7fa0af7fe700 1 -- 172.21.15.31:0/507107584 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa0b0021580 con 0x7fa0c0100e60 2023-07-29T16:26:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.132+0000 7fa0d8560b80 1 Processor -- start 2023-07-29T16:26:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.132+0000 7fa0d8560b80 1 -- start start 2023-07-29T16:26:55.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.132+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7f48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7f6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec74c9b0 con 0x55a8ec7f4df0 2023-07-29T16:26:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec74a370 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec745c60 con 0x55a8ec6506b0 2023-07-29T16:26:55.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0af7fe700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa0a80a0790 0x7fa0a80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7f57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34402/0 (socket says 172.21.15.31:34402) 2023-07-29T16:26:55.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0be59c700 1 -- 172.21.15.31:0/4251716782 learned_addr learned my addr 172.21.15.31:0/4251716782 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:55.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7f48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0af7fe700 1 -- 172.21.15.31:0/507107584 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa0b00ccc60 con 0x7fa0c0100e60 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.133+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa0a80a0790 0x7fa0a80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 -- 172.21.15.31:0/4251716782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 msgr2=0x55a8ec7f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 -- 172.21.15.31:0/4251716782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 msgr2=0x55a8ec7f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 -- 172.21.15.31:0/4251716782 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8ec573d80 con 0x55a8ec6506b0 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0be59c700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7f48b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009eb0 tx=0x7fa0b8019c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b814d0465c9b11e4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa0a80a0790 0x7fa0a80a2c40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8011000 tx=0x7fa0b8022000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0acff9700 1 -- 172.21.15.31:0/4251716782 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8019050 con 0x55a8ec6506b0 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0acff9700 1 -- 172.21.15.31:0/4251716782 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0b801f070 con 0x55a8ec6506b0 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.134+0000 7fa0acff9700 1 -- 172.21.15.31:0/4251716782 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800e4f0 con 0x55a8ec6506b0 2023-07-29T16:26:55.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 -- 172.21.15.31:0/4251716782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 msgr2=0x55a8ec7f48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7f48b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009eb0 tx=0x7fa0b8019c80 comp rx=0 tx=0).stop 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 -- 172.21.15.31:0/4251716782 shutdown_connections 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7f48b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/4251716782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 -- 172.21.15.31:0/4251716782 >> 172.21.15.31:0/4251716782 conn(0x55a8ec7e9fd0 msgr2=0x55a8ec7ef300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:55.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 -- 172.21.15.31:0/4251716782 shutdown_connections 2023-07-29T16:26:55.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 -- 172.21.15.31:0/4251716782 wait complete. 2023-07-29T16:26:55.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.135+0000 7fa0d8560b80 1 -- start start 2023-07-29T16:26:55.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 Processor -- start 2023-07-29T16:26:55.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7fde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7fe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec805400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec6e24b0 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec6e2150 con 0x55a8ec7f4df0 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec5c98c0 con 0x55a8ec6506b0 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0d8560b80 10 client.0 _ll_register_callbacks cb 0x55a8ec6eb5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7fde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.136+0000 7fa0bdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7fe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7fde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57440/0 (socket says 172.21.15.31:57440) 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0bdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7fe3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39690/0 (socket says 172.21.15.31:39690) 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 -- 172.21.15.31:0/1312241394 learned_addr learned my addr 172.21.15.31:0/1312241394 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec805400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 -- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f4df0 msgr2=0x55a8ec7fe3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7fe3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 -- 172.21.15.31:0/1312241394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f5cf0 msgr2=0x55a8ec805400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec805400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 -- 172.21.15.31:0/1312241394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c019e9b0 con 0x55a8ec6506b0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0be59c700 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7fde80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00ccec0 tx=0x7fa0b00a0390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b8e4c4c282f4ca9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/1312241394 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b009a390 con 0x55a8ec6506b0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/1312241394 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa0b009e3c0 con 0x55a8ec6506b0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.137+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8ec770cb0 con 0x55a8ec6506b0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/1312241394 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0b00a2750 con 0x55a8ec6506b0 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 10 client.28954 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:55.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7fa0b4009c90 con 0x55a8ec6506b0 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/1312241394 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a8ec770cb0 con 0x55a8ec6506b0 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/1312241394 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa0b00999a0 con 0x55a8ec6506b0 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 10 client.28954 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 10 client.28954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 20 client.28954 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec809850 0x55a8ec80bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 4 client.28954 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8ec7c3d10 con 0x55a8ec809850 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.138+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec809850 0x55a8ec80bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.139+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec809850 0x55a8ec80bc60 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff53a2129a206a88 server_cookie=805ef645b37b0fcf in_seq=0 out_seq=0 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.139+0000 7fa0a5ffb700 10 client.28954 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/1312241394 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a8ec7c3d10 con 0x55a8ec809850 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0a5ffb700 10 client.28954 handle_command_reply: tid=0 2023-07-29T16:26:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 client.28954 shutdown 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 msgr2=0x55a8ec7fde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7fde80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fa0b00ccec0 tx=0x7fa0b00a0390 comp rx=0 tx=0).stop 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 shutdown_connections 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec809850 0x55a8ec80bc60 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec805400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0a5ffb700 0 client.28954 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec7fde80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/1312241394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7fe3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 >> 172.21.15.31:0/1312241394 conn(0x55a8ec7e9fd0 msgr2=0x55a8ec7eea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 shutdown_connections 2023-07-29T16:26:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.140+0000 7fa0d8560b80 1 -- 172.21.15.31:0/1312241394 wait complete. 2023-07-29T16:26:55.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.141+0000 7fa0d8560b80 20 client.28954 trim_cache size 0 max 16384 2023-07-29T16:26:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 Processor -- start 2023-07-29T16:26:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 -- start start 2023-07-29T16:26:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec6506b0 0x55a8ec8033b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7ef170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec7c3d40 con 0x55a8ec6506b0 2023-07-29T16:26:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec770ce0 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec7fd330 con 0x55a8ec7f4df0 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0bed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0bed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39706/0 (socket says 172.21.15.31:39706) 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec6506b0 0x55a8ec8033b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.157+0000 7fa0bed9d700 1 -- 172.21.15.31:0/2649435223 learned_addr learned my addr 172.21.15.31:0/2649435223 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7ef170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bed9d700 1 -- 172.21.15.31:0/2649435223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec7f4df0 msgr2=0x55a8ec7ef170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7ef170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bed9d700 1 -- 172.21.15.31:0/2649435223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec6506b0 msgr2=0x55a8ec8033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec6506b0 0x55a8ec8033b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bed9d700 1 -- 172.21.15.31:0/2649435223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8005e70 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7fc0d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8008650 tx=0x7fa0b8008fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7875ef992be8e30 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0ad7fa700 1 -- 172.21.15.31:0/2649435223 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8031040 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0ad7fa700 1 -- 172.21.15.31:0/2649435223 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0b800f040 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.158+0000 7fa0ad7fa700 1 -- 172.21.15.31:0/2649435223 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800e4f0 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 -- 172.21.15.31:0/2649435223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 msgr2=0x55a8ec7fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7fc0d0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8008650 tx=0x7fa0b8008fe0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 -- 172.21.15.31:0/2649435223 shutdown_connections 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec7fc0d0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec6506b0 0x55a8ec8033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/2649435223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec7f4df0 0x55a8ec7ef170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 -- 172.21.15.31:0/2649435223 >> 172.21.15.31:0/2649435223 conn(0x55a8ec7e9fd0 msgr2=0x55a8ec7fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.159+0000 7fa0d8560b80 1 -- 172.21.15.31:0/2649435223 shutdown_connections 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.160+0000 7fa0d8560b80 1 -- 172.21.15.31:0/2649435223 wait complete. 2023-07-29T16:26:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.160+0000 7fa0d8560b80 1 -- start start 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.160+0000 7fa0d8560b80 1 Processor -- start 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec803370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec801980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec801f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec8026d0 con 0x55a8ec7f4df0 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec802840 con 0x55a8ec7f5cf0 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8ec85b450 con 0x55a8ec6506b0 2023-07-29T16:26:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0d8560b80 10 client.0 _ll_register_callbacks cb 0x55a8ec6eb5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec803370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0bdd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec801980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec803370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57454/0 (socket says 172.21.15.31:57454) 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0be59c700 1 -- 172.21.15.31:0/92274674 learned_addr learned my addr 172.21.15.31:0/92274674 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.161+0000 7fa0bed9d700 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec801f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0be59c700 1 -- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 msgr2=0x55a8ec801f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0be59c700 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec801f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0be59c700 1 -- 172.21.15.31:0/92274674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 msgr2=0x55a8ec801980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:26:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0be59c700 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec801980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0be59c700 1 -- 172.21.15.31:0/92274674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a8ec573d80 con 0x55a8ec6506b0 2023-07-29T16:26:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0be59c700 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec803370 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa0b009a320 tx=0x7fa0b009a820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6983d03d73ab5ec1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:26:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:26:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/92274674 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0099ed0 con 0x55a8ec6506b0 2023-07-29T16:26:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/92274674 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa0b009e3c0 con 0x55a8ec6506b0 2023-07-29T16:26:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0b801f070 con 0x55a8ec6506b0 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/92274674 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0b009a390 con 0x55a8ec6506b0 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0d8560b80 10 client.28960 fetch_fsmap learned FSMap version 32 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.162+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a8ec55cc00 con 0x55a8ec6506b0 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/92274674 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0b801f070 con 0x55a8ec6506b0 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/92274674 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa0b00cabb0 con 0x55a8ec6506b0 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0d8560b80 10 client.28960 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0d8560b80 10 client.28960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:26:55.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0d8560b80 20 client.28960 populate_metadata read hostname 'smithi031' 2023-07-29T16:26:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec85caa0 0x55a8ec85eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:26:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0d8560b80 4 client.28960 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:26:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a8ec73ece0 con 0x55a8ec85caa0 2023-07-29T16:26:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.163+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec85caa0 0x55a8ec85eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:26:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.164+0000 7fa0bdd9b700 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec85caa0 0x55a8ec85eeb0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ce15c6acb7cbb server_cookie=ad263db09ea1e551 in_seq=0 out_seq=0 2023-07-29T16:26:55.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.164+0000 7fa0a5ffb700 10 client.28960 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.232+0000 7fa0a5ffb700 1 -- 172.21.15.31:0/92274674 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a8ec73ece0 con 0x55a8ec85caa0 2023-07-29T16:26:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.232+0000 7fa0a5ffb700 10 client.28960 handle_command_reply: tid=0 2023-07-29T16:26:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 client.28960 shutdown 2023-07-29T16:26:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 msgr2=0x55a8ec803370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec803370 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fa0b009a320 tx=0x7fa0b009a820 comp rx=0 tx=0).stop 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 shutdown_connections 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a8ec85caa0 0x55a8ec85eeb0 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8ec7f5cf0 0x55a8ec801f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a8ec6506b0 0x55a8ec803370 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 --2- 172.21.15.31:0/92274674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8ec7f4df0 0x55a8ec801980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0a5ffb700 0 client.28960 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:26:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 >> 172.21.15.31:0/92274674 conn(0x55a8ec7e9fd0 msgr2=0x55a8ec7ef0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 shutdown_connections 2023-07-29T16:26:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 1 -- 172.21.15.31:0/92274674 wait complete. 2023-07-29T16:26:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.233+0000 7fa0d8560b80 20 client.28960 trim_cache size 0 max 16384 2023-07-29T16:26:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:26:55.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:26:55.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:26:55.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:26:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa0a80a0790 msgr2=0x7fa0a80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa0a80a0790 0x7fa0a80a2c40 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8011000 tx=0x7fa0b8022000 comp rx=0 tx=0).stop 2023-07-29T16:26:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 msgr2=0x7fa0c01961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:26:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 0x7fa0c01961c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0005720 tx=0x7fa0b0009580 comp rx=0 tx=0).stop 2023-07-29T16:26:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 shutdown_connections 2023-07-29T16:26:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa0a80a0790 0x7fa0a80a2c40 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa0c01060a0 0x7fa0c0197580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0c0100e60 0x7fa0c01961c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 --2- 172.21.15.31:0/507107584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0c0103780 0x7fa0c0196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:26:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 >> 172.21.15.31:0/507107584 conn(0x7fa0c00fa840 msgr2=0x7fa0c0101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:26:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 shutdown_connections 2023-07-29T16:26:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:26:55.234+0000 7fa0c4bff700 1 -- 172.21.15.31:0/507107584 wait complete. 2023-07-29T16:26:55.891 INFO:tasks.workunit.client.0.smithi031.stdout: 1 861537 8.90 MB/sec execute 400 sec latency 46.214 ms 2023-07-29T16:26:56.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:56 smithi156 ceph-mon[129245]: pgmap v715: 97 pgs: 97 active+clean; 165 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.4 MiB/s wr, 243 op/s 2023-07-29T16:26:56.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 863139 8.91 MB/sec execute 401 sec latency 13.598 ms 2023-07-29T16:26:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:56 smithi031 ceph-mon[95504]: pgmap v715: 97 pgs: 97 active+clean; 165 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.4 MiB/s wr, 243 op/s 2023-07-29T16:26:57.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:56 smithi039 ceph-mon[128858]: pgmap v715: 97 pgs: 97 active+clean; 165 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 2.4 MiB/s wr, 243 op/s 2023-07-29T16:26:57.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 865186 8.90 MB/sec execute 402 sec latency 10.131 ms 2023-07-29T16:26:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:57 smithi039 ceph-mon[128858]: pgmap v716: 97 pgs: 97 active+clean; 160 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2023-07-29T16:26:58.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:57 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:26:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:57 smithi156 ceph-mon[129245]: pgmap v716: 97 pgs: 97 active+clean; 160 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2023-07-29T16:26:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:57 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:26:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:57 smithi031 ceph-mon[95504]: pgmap v716: 97 pgs: 97 active+clean; 160 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 215 op/s 2023-07-29T16:26:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:57 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:26:58.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 866653 8.91 MB/sec execute 403 sec latency 69.112 ms 2023-07-29T16:26:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:58 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:26:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:58 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:26:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:58 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:26:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:58 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:26:59.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:58 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:26:59.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:58 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:26:59.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 868497 8.90 MB/sec execute 404 sec latency 17.558 ms 2023-07-29T16:27:00.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:26:59 smithi039 ceph-mon[128858]: pgmap v717: 97 pgs: 97 active+clean; 164 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.6 MiB/s wr, 281 op/s 2023-07-29T16:27:00.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:26:59 smithi156 ceph-mon[129245]: pgmap v717: 97 pgs: 97 active+clean; 164 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.6 MiB/s wr, 281 op/s 2023-07-29T16:27:00.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:26:59 smithi031 ceph-mon[95504]: pgmap v717: 97 pgs: 97 active+clean; 164 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.6 MiB/s wr, 281 op/s 2023-07-29T16:27:00.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 870222 8.92 MB/sec execute 405 sec latency 37.439 ms 2023-07-29T16:27:01.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 872267 8.91 MB/sec execute 406 sec latency 11.630 ms 2023-07-29T16:27:02.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:02 smithi156 ceph-mon[129245]: pgmap v718: 97 pgs: 97 active+clean; 164 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2023-07-29T16:27:02.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 873846 8.92 MB/sec execute 407 sec latency 13.318 ms 2023-07-29T16:27:02.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:02 smithi031 ceph-mon[95504]: pgmap v718: 97 pgs: 97 active+clean; 164 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2023-07-29T16:27:03.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:02 smithi039 ceph-mon[128858]: pgmap v718: 97 pgs: 97 active+clean; 164 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 2.0 MiB/s wr, 232 op/s 2023-07-29T16:27:03.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 875883 8.92 MB/sec execute 408 sec latency 32.043 ms 2023-07-29T16:27:04.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:04 smithi156 ceph-mon[129245]: pgmap v719: 97 pgs: 97 active+clean; 166 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2023-07-29T16:27:04.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 877354 8.92 MB/sec execute 409 sec latency 31.207 ms 2023-07-29T16:27:04.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:04 smithi031 ceph-mon[95504]: pgmap v719: 97 pgs: 97 active+clean; 166 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2023-07-29T16:27:05.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:04 smithi039 ceph-mon[128858]: pgmap v719: 97 pgs: 97 active+clean; 166 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 300 op/s 2023-07-29T16:27:05.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 879428 8.92 MB/sec execute 410 sec latency 14.272 ms 2023-07-29T16:27:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:06 smithi156 ceph-mon[129245]: pgmap v720: 97 pgs: 97 active+clean; 167 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:27:06.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 880929 8.92 MB/sec execute 411 sec latency 13.913 ms 2023-07-29T16:27:06.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:06 smithi031 ceph-mon[95504]: pgmap v720: 97 pgs: 97 active+clean; 167 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:27:07.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:06 smithi039 ceph-mon[128858]: pgmap v720: 97 pgs: 97 active+clean; 167 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.4 MiB/s wr, 263 op/s 2023-07-29T16:27:07.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 883058 8.92 MB/sec execute 412 sec latency 14.091 ms 2023-07-29T16:27:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:07 smithi039 ceph-mon[128858]: pgmap v721: 97 pgs: 97 active+clean; 167 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 2.1 MiB/s wr, 237 op/s 2023-07-29T16:27:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:07 smithi156 ceph-mon[129245]: pgmap v721: 97 pgs: 97 active+clean; 167 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 2.1 MiB/s wr, 237 op/s 2023-07-29T16:27:08.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 -- 172.21.15.31:0/2357486482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30101ed0 msgr2=0x7fcc30108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 --2- 172.21.15.31:0/2357486482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30101ed0 0x7fcc30108790 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009f90 tx=0x7fcc2800b330 comp rx=0 tx=0).stop 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 -- 172.21.15.31:0/2357486482 shutdown_connections 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 --2- 172.21.15.31:0/2357486482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30101ed0 0x7fcc30108790 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 --2- 172.21.15.31:0/2357486482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30100350 0x7fcc30100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 --2- 172.21.15.31:0/2357486482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc30101540 0x7fcc30101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 -- 172.21.15.31:0/2357486482 >> 172.21.15.31:0/2357486482 conn(0x7fcc300fb970 msgr2=0x7fcc300fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 -- 172.21.15.31:0/2357486482 shutdown_connections 2023-07-29T16:27:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.396+0000 7fcc34bff700 1 -- 172.21.15.31:0/2357486482 wait complete. 2023-07-29T16:27:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.397+0000 7fcc34bff700 1 Processor -- start 2023-07-29T16:27:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 -- start start 2023-07-29T16:27:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30100350 0x7fcc301984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 0x7fcc301989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc30101ed0 0x7fcc30199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30199ec0 con 0x7fcc30100350 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3019a030 con 0x7fcc30101ed0 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc34bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3019a1f0 con 0x7fcc30101540 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30100350 0x7fcc301984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc2dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 0x7fcc301989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30100350 0x7fcc301984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57368/0 (socket says 172.21.15.31:57368) 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc2e59c700 1 -- 172.21.15.31:0/1053097766 learned_addr learned my addr 172.21.15.31:0/1053097766 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc2dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 0x7fcc301989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36814/0 (socket says 172.21.15.31:36814) 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.398+0000 7fcc2ed9d700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc30101ed0 0x7fcc30199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/1053097766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc30101ed0 msgr2=0x7fcc30199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc30101ed0 0x7fcc30199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30100350 msgr2=0x7fcc301984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30100350 0x7fcc301984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/1053097766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24009520 con 0x7fcc30101540 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 0x7fcc301989e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e9f0 tx=0x7fcc2400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc1f7fe700 1 -- 172.21.15.31:0/1053097766 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2400cb70 con 0x7fcc30101540 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc1f7fe700 1 -- 172.21.15.31:0/1053097766 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc2400ee90 con 0x7fcc30101540 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc28009c40 con 0x7fcc30101540 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc1f7fe700 1 -- 172.21.15.31:0/1053097766 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24003cf0 con 0x7fcc30101540 2023-07-29T16:27:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.399+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc301a0ab0 con 0x7fcc30101540 2023-07-29T16:27:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.403+0000 7fcc1f7fe700 1 -- 172.21.15.31:0/1053097766 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fcc2400ccd0 con 0x7fcc30101540 2023-07-29T16:27:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc1f7fe700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc180a0790 0x7fcc180a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 Processor -- start 2023-07-29T16:27:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc1f7fe700 1 -- 172.21.15.31:0/1053097766 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcc24014070 con 0x7fcc30101540 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 -- start start 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc180a0790 0x7fcc180a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 0x55f8031a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031a77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a7cf0 0x55f8031a8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8030fe9b0 con 0x55f8031a6df0 2023-07-29T16:27:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8030fc370 con 0x55f8031a7cf0 2023-07-29T16:27:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.405+0000 7fcc48536b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8030f7c60 con 0x55f8030026b0 2023-07-29T16:27:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 0x55f8031a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 0x55f8031a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36822/0 (socket says 172.21.15.31:36822) 2023-07-29T16:27:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2070069891 learned_addr learned my addr 172.21.15.31:0/2070069891 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2ed9d700 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a7cf0 0x55f8031a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031a77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2070069891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a7cf0 msgr2=0x55f8031a8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a7cf0 0x55f8031a8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2070069891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 msgr2=0x55f8031a77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031a77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2070069891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f802f25d80 con 0x55f8030026b0 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc180a0790 0x7fcc180a2c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcc200116b0 tx=0x7fcc20011a70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.406+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 0x55f8031a68b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcc20011e70 tx=0x7fcc20020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b8842ac0f83a463 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.407+0000 7fcc1cff9700 1 -- 172.21.15.31:0/2070069891 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20004020 con 0x55f8030026b0 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.407+0000 7fcc1cff9700 1 -- 172.21.15.31:0/2070069891 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc20003680 con 0x55f8030026b0 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.407+0000 7fcc1cff9700 1 -- 172.21.15.31:0/2070069891 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20020690 con 0x55f8030026b0 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.407+0000 7fcc48536b80 1 -- 172.21.15.31:0/2070069891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 msgr2=0x55f8031a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.407+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 0x55f8031a68b0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcc20011e70 tx=0x7fcc20020000 comp rx=0 tx=0).stop 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 -- 172.21.15.31:0/2070069891 shutdown_connections 2023-07-29T16:27:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a7cf0 0x55f8031a8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8030026b0 0x55f8031a68b0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2070069891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031a77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 -- 172.21.15.31:0/2070069891 >> 172.21.15.31:0/2070069891 conn(0x55f80319bfd0 msgr2=0x55f8031a1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 -- 172.21.15.31:0/2070069891 shutdown_connections 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 -- 172.21.15.31:0/2070069891 wait complete. 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 -- start start 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.408+0000 7fcc48536b80 1 Processor -- start 2023-07-29T16:27:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8030026b0 0x55f8031afe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 0x55f8031b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8030944b0 con 0x55f8030026b0 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f803094150 con 0x55f8031a6df0 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f802f7b8c0 con 0x55f8031a7cf0 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc48536b80 10 client.0 _ll_register_callbacks cb 0x55f80309d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 0x55f8031b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 0x55f8031b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60118/0 (socket says 172.21.15.31:60118) 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/857359422 learned_addr learned my addr 172.21.15.31:0/857359422 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8030026b0 0x55f8031afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/857359422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 msgr2=0x55f8031b7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/857359422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8030026b0 msgr2=0x55f8031afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8030026b0 0x55f8031afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2dd9b700 1 -- 172.21.15.31:0/857359422 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2000f4a0 con 0x55f8031a6df0 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.409+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8030026b0 0x55f8031afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 0x55f8031b03c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c5b0 tx=0x7fcc2401e360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ce4229f97ac4eba server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc15ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc15ffb700 1 -- 172.21.15.31:0/857359422 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240999d0 con 0x55f8031a6df0 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc15ffb700 1 -- 172.21.15.31:0/857359422 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcc2409d5a0 con 0x55f8031a6df0 2023-07-29T16:27:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f803122cb0 con 0x55f8031a6df0 2023-07-29T16:27:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc15ffb700 1 -- 172.21.15.31:0/857359422 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc240a19c0 con 0x55f8031a6df0 2023-07-29T16:27:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc48536b80 10 client.28970 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f802f0ec00 con 0x55f8031a6df0 2023-07-29T16:27:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.410+0000 7fcc15ffb700 1 -- 172.21.15.31:0/857359422 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc24063020 con 0x55f8031a6df0 2023-07-29T16:27:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc15ffb700 1 -- 172.21.15.31:0/857359422 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fcc240984d0 con 0x55f8031a6df0 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc48536b80 10 client.28970 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc48536b80 10 client.28970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc48536b80 20 client.28970 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc48536b80 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8031bbc30 0x55f8031be040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc48536b80 4 client.28970 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f803175d10 con 0x55f8031bbc30 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.411+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8031bbc30 0x55f8031be040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.412+0000 7fcc15ffb700 10 client.28970 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.412+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8031bbc30 0x55f8031be040 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38315c01805a19df server_cookie=85bc38c17975c128 in_seq=0 out_seq=0 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.412+0000 7fcc15ffb700 1 -- 172.21.15.31:0/857359422 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f803175d10 con 0x55f8031bbc30 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.412+0000 7fcc15ffb700 10 client.28970 handle_command_reply: tid=0 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.412+0000 7fcc48536b80 1 client.28970 shutdown 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 msgr2=0x55f8031b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 0x55f8031b03c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c5b0 tx=0x7fcc2401e360 comp rx=0 tx=0).stop 2023-07-29T16:27:08.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 shutdown_connections 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f8031bbc30 0x55f8031be040 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8030026b0 0x55f8031afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 --2- 172.21.15.31:0/857359422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8031a6df0 0x55f8031b03c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc15ffb700 0 client.28970 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 >> 172.21.15.31:0/857359422 conn(0x55f80319bfd0 msgr2=0x55f8031a0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 shutdown_connections 2023-07-29T16:27:08.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 1 -- 172.21.15.31:0/857359422 wait complete. 2023-07-29T16:27:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.413+0000 7fcc48536b80 20 client.28970 trim_cache size 0 max 16384 2023-07-29T16:27:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.429+0000 7fcc48536b80 1 Processor -- start 2023-07-29T16:27:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.429+0000 7fcc48536b80 1 -- start start 2023-07-29T16:27:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031ae060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc48536b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80319b9d0 con 0x55f8031a6df0 2023-07-29T16:27:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc48536b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8031a1f70 con 0x55f8030026b0 2023-07-29T16:27:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc48536b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8031ab700 con 0x55f8031a7cf0 2023-07-29T16:27:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60130/0 (socket says 172.21.15.31:60130) 2023-07-29T16:27:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 -- 172.21.15.31:0/3835506339 learned_addr learned my addr 172.21.15.31:0/3835506339 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2ed9d700 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 -- 172.21.15.31:0/3835506339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 msgr2=0x55f8031ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031ae060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 -- 172.21.15.31:0/3835506339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 msgr2=0x55f8031b73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.430+0000 7fcc2e59c700 1 -- 172.21.15.31:0/3835506339 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f802f4c1f0 con 0x55f8030026b0 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.431+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031a0a40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009740 tx=0x7fcc200099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c89797780e578eef server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.431+0000 7fcc1d7fa700 1 -- 172.21.15.31:0/3835506339 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20020450 con 0x55f8030026b0 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.431+0000 7fcc1d7fa700 1 -- 172.21.15.31:0/3835506339 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc20030040 con 0x55f8030026b0 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.431+0000 7fcc1d7fa700 1 -- 172.21.15.31:0/3835506339 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20004550 con 0x55f8030026b0 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.431+0000 7fcc48536b80 1 -- 172.21.15.31:0/3835506339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 msgr2=0x55f8031a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.431+0000 7fcc48536b80 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031a0a40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009740 tx=0x7fcc200099b0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 -- 172.21.15.31:0/3835506339 shutdown_connections 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031a0a40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 --2- 172.21.15.31:0/3835506339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 -- 172.21.15.31:0/3835506339 >> 172.21.15.31:0/3835506339 conn(0x55f80319bfd0 msgr2=0x55f8031a12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 -- 172.21.15.31:0/3835506339 shutdown_connections 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 -- 172.21.15.31:0/3835506339 wait complete. 2023-07-29T16:27:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 -- start start 2023-07-29T16:27:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.432+0000 7fcc48536b80 1 Processor -- start 2023-07-29T16:27:08.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8031b44e0 con 0x55f8031a6df0 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8031b4650 con 0x55f8030026b0 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f80320d7b0 con 0x55f8031a7cf0 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc48536b80 10 client.0 _ll_register_callbacks cb 0x55f8031c2ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031b6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60144/0 (socket says 172.21.15.31:60144) 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2134434406 learned_addr learned my addr 172.21.15.31:0/2134434406 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2ed9d700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2134434406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 msgr2=0x55f8031b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2134434406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 msgr2=0x55f8031b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.433+0000 7fcc2e59c700 1 -- 172.21.15.31:0/2134434406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f802f25d80 con 0x55f8030026b0 2023-07-29T16:27:08.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:07 smithi031 ceph-mon[95504]: pgmap v721: 97 pgs: 97 active+clean; 167 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 2.1 MiB/s wr, 237 op/s 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc2e59c700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031b6fc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fcc20014040 tx=0x7fcc20005c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23b97babbde72a04 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc15ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc15ffb700 1 -- 172.21.15.31:0/2134434406 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20020450 con 0x55f8030026b0 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc15ffb700 1 -- 172.21.15.31:0/2134434406 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcc200205b0 con 0x55f8030026b0 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc20003680 con 0x55f8030026b0 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc15ffb700 1 -- 172.21.15.31:0/2134434406 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc20009b10 con 0x55f8030026b0 2023-07-29T16:27:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc48536b80 10 client.28976 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.434+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f802f24250 con 0x55f8030026b0 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc15ffb700 1 -- 172.21.15.31:0/2134434406 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc20003680 con 0x55f8030026b0 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc15ffb700 1 -- 172.21.15.31:0/2134434406 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fcc200368d0 con 0x55f8030026b0 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc48536b80 10 client.28976 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc48536b80 10 client.28976 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc48536b80 20 client.28976 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f80320ece0 0x55f8032110f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc48536b80 4 client.28976 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:27:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f8030f0ce0 con 0x55f80320ece0 2023-07-29T16:27:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.435+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f80320ece0 0x55f8032110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.436+0000 7fcc2dd9b700 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f80320ece0 0x55f8032110f0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ad6ef975ebc7789 server_cookie=ac908d3cda11c8d5 in_seq=0 out_seq=0 2023-07-29T16:27:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.436+0000 7fcc15ffb700 10 client.28976 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc15ffb700 1 -- 172.21.15.31:0/2134434406 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f8030f0ce0 con 0x55f80320ece0 2023-07-29T16:27:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc15ffb700 10 client.28976 handle_command_reply: tid=0 2023-07-29T16:27:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 client.28976 shutdown 2023-07-29T16:27:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 msgr2=0x55f8031b6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031b6fc0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fcc20014040 tx=0x7fcc20005c60 comp rx=0 tx=0).stop 2023-07-29T16:27:08.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 shutdown_connections 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f80320ece0 0x55f8032110f0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8031a7cf0 0x55f8031b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8030026b0 0x55f8031b6fc0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 --2- 172.21.15.31:0/2134434406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8031a6df0 0x55f8031b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc15ffb700 0 client.28976 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 >> 172.21.15.31:0/2134434406 conn(0x55f80319bfd0 msgr2=0x55f8031aabc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 shutdown_connections 2023-07-29T16:27:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.517+0000 7fcc48536b80 1 -- 172.21.15.31:0/2134434406 wait complete. 2023-07-29T16:27:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc48536b80 20 client.28976 trim_cache size 0 max 16384 2023-07-29T16:27:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:27:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:27:08.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:27:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:27:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc180a0790 msgr2=0x7fcc180a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc180a0790 0x7fcc180a2c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcc200116b0 tx=0x7fcc20011a70 comp rx=0 tx=0).stop 2023-07-29T16:27:08.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 msgr2=0x7fcc301989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 0x7fcc301989e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e9f0 tx=0x7fcc2400ed00 comp rx=0 tx=0).stop 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 shutdown_connections 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc180a0790 0x7fcc180a2c40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc30101ed0 0x7fcc30199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc30100350 0x7fcc301984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.518+0000 7fcc34bff700 1 --2- 172.21.15.31:0/1053097766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc30101540 0x7fcc301989e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:08.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.519+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 >> 172.21.15.31:0/1053097766 conn(0x7fcc300fb970 msgr2=0x7fcc300fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.519+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 shutdown_connections 2023-07-29T16:27:08.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:08.519+0000 7fcc34bff700 1 -- 172.21.15.31:0/1053097766 wait complete. 2023-07-29T16:27:08.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 884392 8.93 MB/sec execute 413 sec latency 70.215 ms 2023-07-29T16:27:09.892 INFO:tasks.workunit.client.0.smithi031.stdout: 1 886108 8.92 MB/sec execute 414 sec latency 39.835 ms 2023-07-29T16:27:10.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:10 smithi156 ceph-mon[129245]: pgmap v722: 97 pgs: 97 active+clean; 170 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 2.7 MiB/s wr, 327 op/s 2023-07-29T16:27:10.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 887823 8.93 MB/sec execute 415 sec latency 110.463 ms 2023-07-29T16:27:10.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:10 smithi031 ceph-mon[95504]: pgmap v722: 97 pgs: 97 active+clean; 170 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 2.7 MiB/s wr, 327 op/s 2023-07-29T16:27:11.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:10 smithi039 ceph-mon[128858]: pgmap v722: 97 pgs: 97 active+clean; 170 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 2.7 MiB/s wr, 327 op/s 2023-07-29T16:27:11.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 888773 8.91 MB/sec execute 416 sec latency 210.757 ms 2023-07-29T16:27:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:12 smithi156 ceph-mon[129245]: pgmap v723: 97 pgs: 97 active+clean; 170 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2023-07-29T16:27:12.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 891182 8.93 MB/sec execute 417 sec latency 23.602 ms 2023-07-29T16:27:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:12 smithi031 ceph-mon[95504]: pgmap v723: 97 pgs: 97 active+clean; 170 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2023-07-29T16:27:13.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:12 smithi039 ceph-mon[128858]: pgmap v723: 97 pgs: 97 active+clean; 170 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.9 MiB/s wr, 211 op/s 2023-07-29T16:27:13.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 892360 8.92 MB/sec execute 418 sec latency 11.405 ms 2023-07-29T16:27:14.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:14 smithi156 ceph-mon[129245]: pgmap v724: 97 pgs: 97 active+clean; 167 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-07-29T16:27:14.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 894471 8.93 MB/sec execute 419 sec latency 32.140 ms 2023-07-29T16:27:14.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:14 smithi031 ceph-mon[95504]: pgmap v724: 97 pgs: 97 active+clean; 167 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-07-29T16:27:15.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:14 smithi039 ceph-mon[128858]: pgmap v724: 97 pgs: 97 active+clean; 167 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.6 MiB/s wr, 288 op/s 2023-07-29T16:27:15.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 895600 8.92 MB/sec execute 420 sec latency 18.836 ms 2023-07-29T16:27:16.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:16 smithi156 ceph-mon[129245]: pgmap v725: 97 pgs: 97 active+clean; 167 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 256 op/s 2023-07-29T16:27:16.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 897680 8.92 MB/sec execute 421 sec latency 34.423 ms 2023-07-29T16:27:16.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:16 smithi031 ceph-mon[95504]: pgmap v725: 97 pgs: 97 active+clean; 167 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 256 op/s 2023-07-29T16:27:17.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:16 smithi039 ceph-mon[128858]: pgmap v725: 97 pgs: 97 active+clean; 167 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 2.3 MiB/s wr, 256 op/s 2023-07-29T16:27:17.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 898921 8.92 MB/sec execute 422 sec latency 19.867 ms 2023-07-29T16:27:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:17 smithi039 ceph-mon[128858]: pgmap v726: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 2.1 MiB/s wr, 243 op/s 2023-07-29T16:27:18.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:17 smithi156 ceph-mon[129245]: pgmap v726: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 2.1 MiB/s wr, 243 op/s 2023-07-29T16:27:18.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:17 smithi031 ceph-mon[95504]: pgmap v726: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 2.1 MiB/s wr, 243 op/s 2023-07-29T16:27:18.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 901179 8.93 MB/sec execute 423 sec latency 16.322 ms 2023-07-29T16:27:19.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 902315 8.92 MB/sec execute 424 sec latency 31.622 ms 2023-07-29T16:27:20.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:20 smithi156 ceph-mon[129245]: pgmap v727: 97 pgs: 97 active+clean; 169 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:27:20.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 904516 8.93 MB/sec execute 425 sec latency 22.629 ms 2023-07-29T16:27:20.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:20 smithi031 ceph-mon[95504]: pgmap v727: 97 pgs: 97 active+clean; 169 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:27:21.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:20 smithi039 ceph-mon[128858]: pgmap v727: 97 pgs: 97 active+clean; 169 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 2.8 MiB/s wr, 316 op/s 2023-07-29T16:27:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.674+0000 7ff97b9ff700 1 -- 172.21.15.31:0/1749682754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974100340 msgr2=0x7ff974100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.674+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/1749682754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974100340 0x7ff974100750 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff970009990 tx=0x7ff970009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 -- 172.21.15.31:0/1749682754 shutdown_connections 2023-07-29T16:27:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/1749682754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff974101ec0 0x7ff974108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/1749682754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974100340 0x7ff974100750 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/1749682754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974101530 0x7ff974101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 -- 172.21.15.31:0/1749682754 >> 172.21.15.31:0/1749682754 conn(0x7ff9740fb920 msgr2=0x7ff9740fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 -- 172.21.15.31:0/1749682754 shutdown_connections 2023-07-29T16:27:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.675+0000 7ff97b9ff700 1 -- 172.21.15.31:0/1749682754 wait complete. 2023-07-29T16:27:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.676+0000 7ff97b9ff700 1 Processor -- start 2023-07-29T16:27:21.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.676+0000 7ff97b9ff700 1 -- start start 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97b9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 0x7ff974073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97b9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff974101530 0x7ff974073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97b9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974101ec0 0x7ff974073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97b9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9741a0130 con 0x7ff974101530 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97b9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9741a0270 con 0x7ff974100340 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97b9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9741a03b0 con 0x7ff974101ec0 2023-07-29T16:27:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 0x7ff974073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 0x7ff974073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38838/0 (socket says 172.21.15.31:38838) 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 -- 172.21.15.31:0/2587261494 learned_addr learned my addr 172.21.15.31:0/2587261494 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff979f9c700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974101ec0 0x7ff974073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff978f9a700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff974101530 0x7ff974073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 -- 172.21.15.31:0/2587261494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974101ec0 msgr2=0x7ff974073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974101ec0 0x7ff974073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 -- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff974101530 msgr2=0x7ff974073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff974101530 0x7ff974073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.677+0000 7ff97979b700 1 -- 172.21.15.31:0/2587261494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700095f0 con 0x7ff974100340 2023-07-29T16:27:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.678+0000 7ff97979b700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 0x7ff974073140 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff970004710 tx=0x7ff970009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.678+0000 7ff96a7fc700 1 -- 172.21.15.31:0/2587261494 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97001d070 con 0x7ff974100340 2023-07-29T16:27:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.678+0000 7ff96a7fc700 1 -- 172.21.15.31:0/2587261494 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff970021410 con 0x7ff974100340 2023-07-29T16:27:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.678+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9741a05e0 con 0x7ff974100340 2023-07-29T16:27:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.678+0000 7ff96a7fc700 1 -- 172.21.15.31:0/2587261494 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970017650 con 0x7ff974100340 2023-07-29T16:27:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.678+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9741a0ab0 con 0x7ff974100340 2023-07-29T16:27:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.681+0000 7ff96a7fc700 1 -- 172.21.15.31:0/2587261494 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff970021580 con 0x7ff974100340 2023-07-29T16:27:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff96a7fc700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9600a0810 0x7ff9600a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff98f3ddb80 1 Processor -- start 2023-07-29T16:27:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff98f3ddb80 1 -- start start 2023-07-29T16:27:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff96a7fc700 1 -- 172.21.15.31:0/2587261494 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff9700ccde0 con 0x7ff974100340 2023-07-29T16:27:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff978f9a700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9600a0810 0x7ff9600a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf2e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2edf0 0x55894cf2f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.684+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 0x55894cf30160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894ce869b0 con 0x55894cf2edf0 2023-07-29T16:27:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894ce84370 con 0x55894cd8a6b0 2023-07-29T16:27:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894ce7fc60 con 0x55894cf2fcf0 2023-07-29T16:27:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 0x55894cf30160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 0x55894cf30160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52362/0 (socket says 172.21.15.31:52362) 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff97979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf2e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 -- 172.21.15.31:0/4268218736 learned_addr learned my addr 172.21.15.31:0/4268218736 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff978f9a700 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2edf0 0x55894cf2f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff978f9a700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9600a0810 0x7ff9600a2cc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff974074440 tx=0x7ff964011720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 -- 172.21.15.31:0/4268218736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 msgr2=0x55894cf2e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf2e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 -- 172.21.15.31:0/4268218736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2edf0 msgr2=0x55894cf2f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2edf0 0x55894cf2f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff979f9c700 1 -- 172.21.15.31:0/4268218736 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55894ccadd80 con 0x55894cf2fcf0 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.685+0000 7ff97979b700 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf2e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.686+0000 7ff979f9c700 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 0x55894cf30160 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff96c011680 tx=0x7ff96c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba85633bac592fdc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.686+0000 7ff95f7fe700 1 -- 172.21.15.31:0/4268218736 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c00e040 con 0x55894cf2fcf0 2023-07-29T16:27:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.686+0000 7ff95f7fe700 1 -- 172.21.15.31:0/4268218736 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96c00f040 con 0x55894cf2fcf0 2023-07-29T16:27:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.686+0000 7ff95f7fe700 1 -- 172.21.15.31:0/4268218736 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c0048c0 con 0x55894cf2fcf0 2023-07-29T16:27:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/4268218736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 msgr2=0x55894cf30160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 0x55894cf30160 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff96c011680 tx=0x7ff96c009920 comp rx=0 tx=0).stop 2023-07-29T16:27:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/4268218736 shutdown_connections 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2fcf0 0x55894cf30160 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf2e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/4268218736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2edf0 0x55894cf2f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/4268218736 >> 172.21.15.31:0/4268218736 conn(0x55894cf23fd0 msgr2=0x55894cf29300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/4268218736 shutdown_connections 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/4268218736 wait complete. 2023-07-29T16:27:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.687+0000 7ff98f3ddb80 1 -- start start 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 Processor -- start 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf37ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf38420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894ce1c4b0 con 0x55894cf2fcf0 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894ce1c150 con 0x55894cd8a6b0 2023-07-29T16:27:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cd038c0 con 0x55894cf2edf0 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff98f3ddb80 10 client.0 _ll_register_callbacks cb 0x55894ce255c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff978f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf38420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff97979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf37ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff978f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf38420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52374/0 (socket says 172.21.15.31:52374) 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff978f9a700 1 -- 172.21.15.31:0/267302263 learned_addr learned my addr 172.21.15.31:0/267302263 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.688+0000 7ff979f9c700 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff978f9a700 1 -- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 msgr2=0x55894cf37ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff978f9a700 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf37ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff978f9a700 1 -- 172.21.15.31:0/267302263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 msgr2=0x55894cf3f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff978f9a700 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff978f9a700 1 -- 172.21.15.31:0/267302263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9741a0ab0 con 0x55894cf2edf0 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff978f9a700 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf38420 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff96400f740 tx=0x7ff9640047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9998d135996dfc5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff95cff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff95cff9700 1 -- 172.21.15.31:0/267302263 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964004830 con 0x55894cf2edf0 2023-07-29T16:27:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55894ceaacb0 con 0x55894cf2edf0 2023-07-29T16:27:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.689+0000 7ff95cff9700 1 -- 172.21.15.31:0/267302263 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff964004990 con 0x55894cf2edf0 2023-07-29T16:27:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff95cff9700 1 -- 172.21.15.31:0/267302263 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff964004b50 con 0x55894cf2edf0 2023-07-29T16:27:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 10 client.28990 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7ff96c011330 con 0x55894cf2edf0 2023-07-29T16:27:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff95cff9700 1 -- 172.21.15.31:0/267302263 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55894ceaacb0 con 0x55894cf2edf0 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff95cff9700 1 -- 172.21.15.31:0/267302263 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff96401d810 con 0x55894cf2edf0 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 10 client.28990 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 10 client.28990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 20 client.28990 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf43d90 0x55894cf461a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 4 client.28990 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55894cefdd10 con 0x55894cf43d90 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.690+0000 7ff97979b700 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf43d90 0x55894cf461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.691+0000 7ff97979b700 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf43d90 0x55894cf461a0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=155e3eda6768523b server_cookie=c14a71b577b7d413 in_seq=0 out_seq=0 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.691+0000 7ff95cff9700 10 client.28990 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff95cff9700 1 -- 172.21.15.31:0/267302263 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55894cefdd10 con 0x55894cf43d90 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff95cff9700 10 client.28990 handle_command_reply: tid=0 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 client.28990 shutdown 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 msgr2=0x55894cf38420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf38420 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff96400f740 tx=0x7ff9640047c0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 shutdown_connections 2023-07-29T16:27:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf43d90 0x55894cf461a0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf37ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff95cff9700 0 client.28990 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/267302263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf38420 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 >> 172.21.15.31:0/267302263 conn(0x55894cf23fd0 msgr2=0x55894cf28a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 shutdown_connections 2023-07-29T16:27:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/267302263 wait complete. 2023-07-29T16:27:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.692+0000 7ff98f3ddb80 20 client.28990 trim_cache size 0 max 16384 2023-07-29T16:27:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 Processor -- start 2023-07-29T16:27:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 -- start start 2023-07-29T16:27:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cd8a6b0 0x55894cf3d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cf2fcf0 0x55894cf330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cf41150 con 0x55894cd8a6b0 2023-07-29T16:27:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cf37390 con 0x55894cf2fcf0 2023-07-29T16:27:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cf37a50 con 0x55894cf2edf0 2023-07-29T16:27:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff97979b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cd8a6b0 0x55894cf3d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff978f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff97979b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cd8a6b0 0x55894cf3d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36162/0 (socket says 172.21.15.31:36162) 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.708+0000 7ff97979b700 1 -- 172.21.15.31:0/3271236 learned_addr learned my addr 172.21.15.31:0/3271236 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff979f9c700 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cf2fcf0 0x55894cf330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff978f9a700 1 -- 172.21.15.31:0/3271236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cf2fcf0 msgr2=0x55894cf330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff978f9a700 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cf2fcf0 0x55894cf330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff978f9a700 1 -- 172.21.15.31:0/3271236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cd8a6b0 msgr2=0x55894cf3d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff978f9a700 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cd8a6b0 0x55894cf3d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff978f9a700 1 -- 172.21.15.31:0/3271236 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55894cc96c00 con 0x55894cf2edf0 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff978f9a700 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf32ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff96401dc70 tx=0x7ff964009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcd10d5dce486a2a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff95ffff700 1 -- 172.21.15.31:0/3271236 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964034070 con 0x55894cf2edf0 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff95ffff700 1 -- 172.21.15.31:0/3271236 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff964004830 con 0x55894cf2edf0 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.709+0000 7ff95ffff700 1 -- 172.21.15.31:0/3271236 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964005450 con 0x55894cf2edf0 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/3271236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 msgr2=0x55894cf32ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf32ba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff96401dc70 tx=0x7ff964009260 comp rx=0 tx=0).stop 2023-07-29T16:27:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/3271236 shutdown_connections 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cf2fcf0 0x55894cf330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cd8a6b0 0x55894cf3d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/3271236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf32ba0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/3271236 >> 172.21.15.31:0/3271236 conn(0x55894cf23fd0 msgr2=0x55894cf36060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/3271236 shutdown_connections 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.710+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/3271236 wait complete. 2023-07-29T16:27:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 -- start start 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 Processor -- start 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf3cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cf3d770 con 0x55894cf2fcf0 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cf3bfe0 con 0x55894cd8a6b0 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55894cf3c1a0 con 0x55894cf2edf0 2023-07-29T16:27:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.711+0000 7ff98f3ddb80 10 client.0 _ll_register_callbacks cb 0x55894ce255c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff97979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf3cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf3b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52392/0 (socket says 172.21.15.31:52392) 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 -- 172.21.15.31:0/1762107729 learned_addr learned my addr 172.21.15.31:0/1762107729 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff979f9c700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 -- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 msgr2=0x55894cf3cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf3cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 -- 172.21.15.31:0/1762107729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 msgr2=0x55894cf3baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 -- 172.21.15.31:0/1762107729 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96402b040 con 0x55894cf2edf0 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff979f9c700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff97979b700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf3cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff978f9a700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf3b560 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff96401db70 tx=0x7ff96400fdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=825a577468dff4c8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff95cff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.712+0000 7ff95cff9700 1 -- 172.21.15.31:0/1762107729 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964034070 con 0x55894cf2edf0 2023-07-29T16:27:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff96c00f040 con 0x55894cf2edf0 2023-07-29T16:27:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff95cff9700 1 -- 172.21.15.31:0/1762107729 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff964004d70 con 0x55894cf2edf0 2023-07-29T16:27:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff95cff9700 1 -- 172.21.15.31:0/1762107729 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9640051d0 con 0x55894cf2edf0 2023-07-29T16:27:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 10 client.28996 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55894ccd41f0 con 0x55894cf2edf0 2023-07-29T16:27:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff95cff9700 1 -- 172.21.15.31:0/1762107729 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96c00f040 con 0x55894cf2edf0 2023-07-29T16:27:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff95cff9700 1 -- 172.21.15.31:0/1762107729 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff964043a10 con 0x55894cf2edf0 2023-07-29T16:27:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 10 client.28996 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 10 client.28996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 20 client.28996 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf96b90 0x55894cf98fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 4 client.28996 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.713+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55894ce78ce0 con 0x55894cf96b90 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.714+0000 7ff97979b700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf96b90 0x55894cf98fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.714+0000 7ff97979b700 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf96b90 0x55894cf98fa0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f4b33c2473a41f2 server_cookie=ba869163474ebdc8 in_seq=0 out_seq=0 2023-07-29T16:27:21.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.714+0000 7ff95cff9700 10 client.28996 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:21.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff95cff9700 1 -- 172.21.15.31:0/1762107729 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55894ce78ce0 con 0x55894cf96b90 2023-07-29T16:27:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff95cff9700 10 client.28996 handle_command_reply: tid=0 2023-07-29T16:27:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 client.28996 shutdown 2023-07-29T16:27:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 msgr2=0x55894cf3b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf3b560 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff96401db70 tx=0x7ff96400fdb0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 shutdown_connections 2023-07-29T16:27:21.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55894cf96b90 0x55894cf98fa0 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55894cf2fcf0 0x55894cf3baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff95cff9700 0 client.28996 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55894cd8a6b0 0x55894cf3cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 --2- 172.21.15.31:0/1762107729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55894cf2edf0 0x55894cf3b560 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 >> 172.21.15.31:0/1762107729 conn(0x55894cf23fd0 msgr2=0x55894cf36060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 shutdown_connections 2023-07-29T16:27:21.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.788+0000 7ff98f3ddb80 1 -- 172.21.15.31:0/1762107729 wait complete. 2023-07-29T16:27:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff98f3ddb80 20 client.28996 trim_cache size 0 max 16384 2023-07-29T16:27:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:27:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:27:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:27:21.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:27:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9600a0810 msgr2=0x7ff9600a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9600a0810 0x7ff9600a2cc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff974074440 tx=0x7ff964011720 comp rx=0 tx=0).stop 2023-07-29T16:27:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 msgr2=0x7ff974073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:21.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 0x7ff974073140 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff970004710 tx=0x7ff970009580 comp rx=0 tx=0).stop 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 shutdown_connections 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff9600a0810 0x7ff9600a2cc0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff974101ec0 0x7ff974073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff974100340 0x7ff974073140 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 --2- 172.21.15.31:0/2587261494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff974101530 0x7ff974073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.789+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 >> 172.21.15.31:0/2587261494 conn(0x7ff9740fb920 msgr2=0x7ff9740fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:21.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.790+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 shutdown_connections 2023-07-29T16:27:21.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:21.790+0000 7ff97b9ff700 1 -- 172.21.15.31:0/2587261494 wait complete. 2023-07-29T16:27:21.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 905717 8.92 MB/sec execute 426 sec latency 49.754 ms 2023-07-29T16:27:21.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:27:22.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 -- 172.21.15.31:0/1232018716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e0100410 msgr2=0x7f50e0100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 --2- 172.21.15.31:0/1232018716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e0100410 0x7f50e0100860 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009990 tx=0x7f50d4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 -- 172.21.15.31:0/1232018716 shutdown_connections 2023-07-29T16:27:22.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 --2- 172.21.15.31:0/1232018716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100da0 0x7f50e0107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 --2- 172.21.15.31:0/1232018716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e00ff220 0x7f50e00ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 --2- 172.21.15.31:0/1232018716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e0100410 0x7f50e0100860 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 -- 172.21.15.31:0/1232018716 >> 172.21.15.31:0/1232018716 conn(0x7f50e00fa840 msgr2=0x7f50e00fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.125+0000 7f50e53ff700 1 -- 172.21.15.31:0/1232018716 shutdown_connections 2023-07-29T16:27:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.126+0000 7f50e53ff700 1 -- 172.21.15.31:0/1232018716 wait complete. 2023-07-29T16:27:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 Processor -- start 2023-07-29T16:27:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 -- start start 2023-07-29T16:27:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 0x7f50e0198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 0x7f50e0198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e0100da0 0x7f50e019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e019fe90 con 0x7f50e0100da0 2023-07-29T16:27:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e019ffd0 con 0x7f50e00ff220 2023-07-29T16:27:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50e53ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e01a0160 con 0x7f50e0100410 2023-07-29T16:27:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50deffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 0x7f50e0198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.127+0000 7f50de7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 0x7f50e0198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 0x7f50e0198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38892/0 (socket says 172.21.15.31:38892) 2023-07-29T16:27:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50de7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 0x7f50e0198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52414/0 (socket says 172.21.15.31:52414) 2023-07-29T16:27:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 -- 172.21.15.31:0/735041765 learned_addr learned my addr 172.21.15.31:0/735041765 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50df7fe700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e0100da0 0x7f50e019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 -- 172.21.15.31:0/735041765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 msgr2=0x7f50e0198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 0x7f50e0198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 -- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e0100da0 msgr2=0x7f50e019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e0100da0 0x7f50e019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 -- 172.21.15.31:0/735041765 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40095f0 con 0x7f50e00ff220 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50de7fc700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 0x7f50e0198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.128+0000 7f50deffd700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 0x7f50e0198310 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f50d000e970 tx=0x7f50d000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.129+0000 7f50c7fff700 1 -- 172.21.15.31:0/735041765 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0009790 con 0x7f50e00ff220 2023-07-29T16:27:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.129+0000 7f50c7fff700 1 -- 172.21.15.31:0/735041765 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50d000c7e0 con 0x7f50e00ff220 2023-07-29T16:27:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.129+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e01a0440 con 0x7f50e00ff220 2023-07-29T16:27:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.129+0000 7f50c7fff700 1 -- 172.21.15.31:0/735041765 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0003c70 con 0x7f50e00ff220 2023-07-29T16:27:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.129+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e01a09a0 con 0x7f50e00ff220 2023-07-29T16:27:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.132+0000 7f50c7fff700 1 -- 172.21.15.31:0/735041765 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f50d000cc50 con 0x7f50e00ff220 2023-07-29T16:27:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 Processor -- start 2023-07-29T16:27:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 -- start start 2023-07-29T16:27:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab38ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab39d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadab3a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaa90a10 con 0x55aadab3a2f0 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaa8e3d0 con 0x55aadab38be0 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.134+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaa89cc0 con 0x55aadab39530 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab38ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50c7fff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50c80a0810 0x7f50c80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab38ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38906/0 (socket says 172.21.15.31:38906) 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50df7fe700 1 -- 172.21.15.31:0/1579804340 learned_addr learned my addr 172.21.15.31:0/1579804340 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab39d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50de7fc700 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadab3a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50c7fff700 1 -- 172.21.15.31:0/735041765 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f50d0014070 con 0x7f50e00ff220 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50df7fe700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50c80a0810 0x7f50c80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 -- 172.21.15.31:0/1579804340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 msgr2=0x55aadab38ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab38ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 -- 172.21.15.31:0/1579804340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 msgr2=0x55aadab3a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadab3a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 -- 172.21.15.31:0/1579804340 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aada8b7de0 con 0x55aadab39530 2023-07-29T16:27:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.135+0000 7f50deffd700 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab39d60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f50d00c9740 tx=0x7f50d0003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67b771a8da34576 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.136+0000 7f50df7fe700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50c80a0810 0x7f50c80a2cc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x55aadab38980 tx=0x7f50d8004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.136+0000 7f50c57fa700 1 -- 172.21.15.31:0/1579804340 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0099400 con 0x55aadab39530 2023-07-29T16:27:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.136+0000 7f50c57fa700 1 -- 172.21.15.31:0/1579804340 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50d009d430 con 0x55aadab39530 2023-07-29T16:27:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.136+0000 7f50c57fa700 1 -- 172.21.15.31:0/1579804340 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d00988f0 con 0x55aadab39530 2023-07-29T16:27:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.136+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/1579804340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 msgr2=0x55aadab39d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.136+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab39d60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f50d00c9740 tx=0x7f50d0003680 comp rx=0 tx=0).stop 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/1579804340 shutdown_connections 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadab3a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab38ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/1579804340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab39d60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/1579804340 >> 172.21.15.31:0/1579804340 conn(0x55aadab2e3e0 msgr2=0x55aadab33990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/1579804340 shutdown_connections 2023-07-29T16:27:22.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/1579804340 wait complete. 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 -- start start 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.137+0000 7f50f8d1ab80 1 Processor -- start 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab38be0 0x55aadab42830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab42d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 0x55aadab49db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaa26510 con 0x55aadab39530 2023-07-29T16:27:22.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaa261b0 con 0x55aadab3a2f0 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aada90d920 con 0x55aadab38be0 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50f8d1ab80 10 client.0 _ll_register_callbacks cb 0x55aadaa2f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50de7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab42d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 0x55aadab49db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50df7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 0x55aadab49db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38908/0 (socket says 172.21.15.31:38908) 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50df7fe700 1 -- 172.21.15.31:0/41131353 learned_addr learned my addr 172.21.15.31:0/41131353 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.138+0000 7f50deffd700 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab38be0 0x55aadab42830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50df7fe700 1 -- 172.21.15.31:0/41131353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab38be0 msgr2=0x55aadab42830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50df7fe700 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab38be0 0x55aadab42830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50df7fe700 1 -- 172.21.15.31:0/41131353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 msgr2=0x55aadab42d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50df7fe700 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab42d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50df7fe700 1 -- 172.21.15.31:0/41131353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e01a09a0 con 0x55aadab3a2f0 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50df7fe700 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 0x55aadab49db0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f50d8018c20 tx=0x7f50d8004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29586dca33bec819 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50c2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:22.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50c2ff5700 1 -- 172.21.15.31:0/41131353 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d800e040 con 0x55aadab3a2f0 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50c2ff5700 1 -- 172.21.15.31:0/41131353 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f50d8007da0 con 0x55aadab3a2f0 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aadaab4d10 con 0x55aadab3a2f0 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.139+0000 7f50c2ff5700 1 -- 172.21.15.31:0/41131353 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50d801a470 con 0x55aadab3a2f0 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 10 client.29006 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55aada8a0c60 con 0x55aadab3a2f0 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50c2ff5700 1 -- 172.21.15.31:0/41131353 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50d8033080 con 0x55aadab3a2f0 2023-07-29T16:27:22.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50c2ff5700 1 -- 172.21.15.31:0/41131353 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f50d8025520 con 0x55aadab3a2f0 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 10 client.29006 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 10 client.29006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 20 client.29006 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadab4e620 0x55aadab50a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 4 client.29006 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aadab07d70 con 0x55aadab4e620 2023-07-29T16:27:22.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.140+0000 7f50deffd700 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadab4e620 0x55aadab50a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.141+0000 7f50deffd700 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadab4e620 0x55aadab50a30 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33ef660284677ccb server_cookie=d3afe008b07e8595 in_seq=0 out_seq=0 2023-07-29T16:27:22.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.141+0000 7f50c2ff5700 10 client.29006 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.141+0000 7f50c2ff5700 1 -- 172.21.15.31:0/41131353 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55aadab07d70 con 0x55aadab4e620 2023-07-29T16:27:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.141+0000 7f50c2ff5700 10 client.29006 handle_command_reply: tid=0 2023-07-29T16:27:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.141+0000 7f50f8d1ab80 1 client.29006 shutdown 2023-07-29T16:27:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 msgr2=0x55aadab49db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 0x55aadab49db0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f50d8018c20 tx=0x7f50d8004ad0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 shutdown_connections 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadab4e620 0x55aadab50a30 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab3a2f0 0x55aadab49db0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab38be0 0x55aadab42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50c2ff5700 0 client.29006 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/41131353 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab42d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 >> 172.21.15.31:0/41131353 conn(0x55aadab2e3e0 msgr2=0x55aadab33060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 shutdown_connections 2023-07-29T16:27:22.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/41131353 wait complete. 2023-07-29T16:27:22.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.142+0000 7f50f8d1ab80 20 client.29006 trim_cache size 0 max 16384 2023-07-29T16:27:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 Processor -- start 2023-07-29T16:27:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 -- start start 2023-07-29T16:27:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab3d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab49d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 0x55aadab3d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadab423a0 con 0x55aadab39530 2023-07-29T16:27:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadab30460 con 0x55aadab38be0 2023-07-29T16:27:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.160+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadab439b0 con 0x55aadab3a2f0 2023-07-29T16:27:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 0x55aadab3d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50de7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab49d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 0x55aadab3d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52446/0 (socket says 172.21.15.31:52446) 2023-07-29T16:27:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 -- 172.21.15.31:0/3082572660 learned_addr learned my addr 172.21.15.31:0/3082572660 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50deffd700 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab3d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 -- 172.21.15.31:0/3082572660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 msgr2=0x55aadab3d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab3d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 -- 172.21.15.31:0/3082572660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 msgr2=0x55aadab49d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab49d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 -- 172.21.15.31:0/3082572660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aada8de250 con 0x55aadab3a2f0 2023-07-29T16:27:22.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.161+0000 7f50df7fe700 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 0x55aadab3d6f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f50d8018bf0 tx=0x7f50d8009df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6641779a0f2211a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50c5ffb700 1 -- 172.21.15.31:0/3082572660 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d800e070 con 0x55aadab3a2f0 2023-07-29T16:27:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50c5ffb700 1 -- 172.21.15.31:0/3082572660 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aadaab4d10 con 0x55aadab3a2f0 2023-07-29T16:27:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50c5ffb700 1 -- 172.21.15.31:0/3082572660 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8025660 con 0x55aadab3a2f0 2023-07-29T16:27:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3082572660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 msgr2=0x55aadab3d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 0x55aadab3d6f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f50d8018bf0 tx=0x7f50d8009df0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3082572660 shutdown_connections 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab3a2f0 0x55aadab3d6f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab3d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3082572660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab39530 0x55aadab49d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3082572660 >> 172.21.15.31:0/3082572660 conn(0x55aadab2e3e0 msgr2=0x55aadab40680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3082572660 shutdown_connections 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.162+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3082572660 wait complete. 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.163+0000 7f50f8d1ab80 1 -- start start 2023-07-29T16:27:22.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.163+0000 7f50f8d1ab80 1 Processor -- start 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.163+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab46b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.163+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab47080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50f8d1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadaba01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadab476a0 con 0x55aadab3a2f0 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaba06e0 con 0x55aadab38be0 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50f8d1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadaba0820 con 0x55aadab39530 2023-07-29T16:27:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50f8d1ab80 10 client.0 _ll_register_callbacks cb 0x55aadaa2f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab46b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab46b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38934/0 (socket says 172.21.15.31:38934) 2023-07-29T16:27:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 -- 172.21.15.31:0/3467801231 learned_addr learned my addr 172.21.15.31:0/3467801231 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50df7fe700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadaba01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50de7fc700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab47080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 -- 172.21.15.31:0/3467801231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 msgr2=0x55aadab47080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab47080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 -- 172.21.15.31:0/3467801231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 msgr2=0x55aadaba01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadaba01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 -- 172.21.15.31:0/3467801231 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aada8a0c60 con 0x55aadab38be0 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50df7fe700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadaba01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50deffd700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab46b40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f50d00036c0 tx=0x7f50d0095740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30fc5b9c4bd44649 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.164+0000 7f50c2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50c2ff5700 1 -- 172.21.15.31:0/3467801231 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d0095c80 con 0x55aadab38be0 2023-07-29T16:27:22.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50d009d430 con 0x55aadab38be0 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50c2ff5700 1 -- 172.21.15.31:0/3467801231 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f50d00a1850 con 0x55aadab38be0 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50c2ff5700 1 -- 172.21.15.31:0/3467801231 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50d00c9820 con 0x55aadab38be0 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50f8d1ab80 10 client.29012 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55aada8b62b0 con 0x55aadab38be0 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50c2ff5700 1 -- 172.21.15.31:0/3467801231 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50d009d430 con 0x55aadab38be0 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.165+0000 7f50c2ff5700 1 -- 172.21.15.31:0/3467801231 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f50d00a13c0 con 0x55aadab38be0 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50f8d1ab80 10 client.29012 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:22.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50f8d1ab80 10 client.29012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50f8d1ab80 20 client.29012 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadaba1d80 0x55aadaba4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50f8d1ab80 4 client.29012 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55aadaa82d40 con 0x55aadaba1d80 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50de7fc700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadaba1d80 0x55aadaba4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50de7fc700 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadaba1d80 0x55aadaba4190 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb9221ecdf9dd948 server_cookie=b43828c4d51b5dff in_seq=0 out_seq=0 2023-07-29T16:27:22.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.166+0000 7f50c2ff5700 10 client.29012 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.167+0000 7f50c2ff5700 1 -- 172.21.15.31:0/3467801231 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55aadaa82d40 con 0x55aadaba1d80 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.167+0000 7f50c2ff5700 10 client.29012 handle_command_reply: tid=0 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.167+0000 7f50f8d1ab80 1 client.29012 shutdown 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 msgr2=0x55aadab46b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab46b40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f50d00036c0 tx=0x7f50d0095740 comp rx=0 tx=0).stop 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 shutdown_connections 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55aadaba1d80 0x55aadaba4190 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55aadab3a2f0 0x55aadaba01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aadab38be0 0x55aadab46b40 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50c2ff5700 0 client.29012 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 --2- 172.21.15.31:0/3467801231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aadab39530 0x55aadab47080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 >> 172.21.15.31:0/3467801231 conn(0x55aadab2e3e0 msgr2=0x55aadab40680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 shutdown_connections 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 1 -- 172.21.15.31:0/3467801231 wait complete. 2023-07-29T16:27:22.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.168+0000 7f50f8d1ab80 20 client.29012 trim_cache size 0 max 16384 2023-07-29T16:27:22.176 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:27:22.176 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:27:22.176 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:27:22.176 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:27:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50c80a0810 msgr2=0x7f50c80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50c80a0810 0x7f50c80a2cc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x55aadab38980 tx=0x7f50d8004b40 comp rx=0 tx=0).stop 2023-07-29T16:27:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 msgr2=0x7f50e0198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 0x7f50e0198310 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f50d000e970 tx=0x7f50d000ec80 comp rx=0 tx=0).stop 2023-07-29T16:27:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 shutdown_connections 2023-07-29T16:27:22.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f50c80a0810 0x7f50c80a2cc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e0100da0 0x7f50e019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f50e00ff220 0x7f50e0198310 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 --2- 172.21.15.31:0/735041765 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50e0100410 0x7f50e0198850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 >> 172.21.15.31:0/735041765 conn(0x7f50e00fa840 msgr2=0x7f50e0067ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 shutdown_connections 2023-07-29T16:27:22.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.169+0000 7f50e53ff700 1 -- 172.21.15.31:0/735041765 wait complete. 2023-07-29T16:27:22.189 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:27:22.189 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:27:22.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 -- 172.21.15.31:0/3833787152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b81036c0 msgr2=0x7f09b8105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 --2- 172.21.15.31:0/3833787152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b81036c0 0x7f09b8105aa0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f09ac009990 tx=0x7f09ac009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 -- 172.21.15.31:0/3833787152 shutdown_connections 2023-07-29T16:27:22.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 --2- 172.21.15.31:0/3833787152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09b8105fe0 0x7f09b8108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 --2- 172.21.15.31:0/3833787152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09b8100da0 0x7f09b8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 --2- 172.21.15.31:0/3833787152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b81036c0 0x7f09b8105aa0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.409+0000 7f09be7ff700 1 -- 172.21.15.31:0/3833787152 >> 172.21.15.31:0/3833787152 conn(0x7f09b80fa840 msgr2=0x7f09b80fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.410+0000 7f09be7ff700 1 -- 172.21.15.31:0/3833787152 shutdown_connections 2023-07-29T16:27:22.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.410+0000 7f09be7ff700 1 -- 172.21.15.31:0/3833787152 wait complete. 2023-07-29T16:27:22.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 Processor -- start 2023-07-29T16:27:22.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 -- start start 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 0x7f09b8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09b81036c0 0x7f09b8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09b8105fe0 0x7f09b819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b819fd80 con 0x7f09b8105fe0 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b819fec0 con 0x7f09b81036c0 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.411+0000 7f09be7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b81a0050 con 0x7f09b8100da0 2023-07-29T16:27:22.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 0x7f09b8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 0x7f09b8198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52458/0 (socket says 172.21.15.31:52458) 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 -- 172.21.15.31:0/145894514 learned_addr learned my addr 172.21.15.31:0/145894514 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09bcd9c700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09b8105fe0 0x7f09b819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b77fe700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09b81036c0 0x7f09b8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 -- 172.21.15.31:0/145894514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09b81036c0 msgr2=0x7f09b8198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09b81036c0 0x7f09b8198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 -- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09b8105fe0 msgr2=0x7f09b819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09b8105fe0 0x7f09b819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.412+0000 7f09b7fff700 1 -- 172.21.15.31:0/145894514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0095f0 con 0x7f09b8100da0 2023-07-29T16:27:22.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.413+0000 7f09b7fff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 0x7f09b8198200 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f09a800e9a0 tx=0x7f09a800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.413+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a800ca10 con 0x7f09b8100da0 2023-07-29T16:27:22.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.413+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09a800cb70 con 0x7f09b8100da0 2023-07-29T16:27:22.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.413+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b81a0330 con 0x7f09b8100da0 2023-07-29T16:27:22.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.413+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a8018680 con 0x7f09b8100da0 2023-07-29T16:27:22.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.413+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b81a0890 con 0x7f09b8100da0 2023-07-29T16:27:22.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.414+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b8107bd0 con 0x7f09b8100da0 2023-07-29T16:27:22.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.416+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f09a800cce0 con 0x7f09b8100da0 2023-07-29T16:27:22.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.418+0000 7f09b57fa700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09a00a0860 0x7f09a00a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.418+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f09a8014070 con 0x7f09b8100da0 2023-07-29T16:27:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.418+0000 7f09b77fe700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09a00a0860 0x7f09a00a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.419+0000 7f09b77fe700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09a00a0860 0x7f09a00a2d10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00b2e0 tx=0x7f09ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.422+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f09a8051720 con 0x7f09b8100da0 2023-07-29T16:27:22.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.708+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b802cc30 con 0x7f09b8100da0 2023-07-29T16:27:22.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.709+0000 7f09b57fa700 1 -- 172.21.15.31:0/145894514 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10780 (secure 0 0 0) 0x7f09a8095780 con 0x7f09b8100da0 2023-07-29T16:27:22.709 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:27:22.709 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:26:47.968454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:27:22.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09a00a0860 msgr2=0x7f09a00a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09a00a0860 0x7f09a00a2d10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00b2e0 tx=0x7f09ac005fb0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 msgr2=0x7f09b8198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 0x7f09b8198200 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f09a800e9a0 tx=0x7f09a800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 shutdown_connections 2023-07-29T16:27:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f09a00a0860 0x7f09a00a2d10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09b8105fe0 0x7f09b819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09b8100da0 0x7f09b8198200 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 --2- 172.21.15.31:0/145894514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09b81036c0 0x7f09b8198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 >> 172.21.15.31:0/145894514 conn(0x7f09b80fa840 msgr2=0x7f09b8101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 shutdown_connections 2023-07-29T16:27:22.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.712+0000 7f09be7ff700 1 -- 172.21.15.31:0/145894514 wait complete. 2023-07-29T16:27:22.714 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 32 2023-07-29T16:27:22.733 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:27:22.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:22 smithi156 ceph-mon[129245]: pgmap v728: 97 pgs: 97 active+clean; 169 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.0 MiB/s wr, 207 op/s 2023-07-29T16:27:22.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 907619 8.92 MB/sec execute 427 sec latency 16.395 ms 2023-07-29T16:27:22.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:22 smithi031 ceph-mon[95504]: pgmap v728: 97 pgs: 97 active+clean; 169 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.0 MiB/s wr, 207 op/s 2023-07-29T16:27:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 -- 172.21.15.31:0/3349850724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074d30 msgr2=0x7f0a28073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/3349850724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074d30 0x7f0a28073180 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009990 tx=0x7f0a24009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.955 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 -- 172.21.15.31:0/3349850724 shutdown_connections 2023-07-29T16:27:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/3349850724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a281080c0 0x7f0a2810a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/3349850724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a28074d30 0x7f0a28073180 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/3349850724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a28073750 0x7f0a28073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 -- 172.21.15.31:0/3349850724 >> 172.21.15.31:0/3349850724 conn(0x7f0a280fb850 msgr2=0x7f0a280fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 -- 172.21.15.31:0/3349850724 shutdown_connections 2023-07-29T16:27:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.955+0000 7f0a2e39d700 1 -- 172.21.15.31:0/3349850724 wait complete. 2023-07-29T16:27:22.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 Processor -- start 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 -- start start 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a28073750 0x7f0a28197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a28074d30 0x7f0a28198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 0x7f0a2819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2819fb70 con 0x7f0a28073750 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2819fcb0 con 0x7f0a281080c0 2023-07-29T16:27:22.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2e39d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2819fe40 con 0x7f0a28074d30 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2db9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 0x7f0a2819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2cb9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a28074d30 0x7f0a28198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2db9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 0x7f0a2819f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38962/0 (socket says 172.21.15.31:38962) 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2cb9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a28074d30 0x7f0a28198530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52462/0 (socket says 172.21.15.31:52462) 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a28073750 0x7f0a28197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.957+0000 7f0a2db9c700 1 -- 172.21.15.31:0/695887945 learned_addr learned my addr 172.21.15.31:0/695887945 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2db9c700 1 -- 172.21.15.31:0/695887945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a28074d30 msgr2=0x7f0a28198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a28074d30 0x7f0a28198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2db9c700 1 -- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a28073750 msgr2=0x7f0a28197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a28073750 0x7f0a28197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2db9c700 1 -- 172.21.15.31:0/695887945 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a18009520 con 0x7f0a281080c0 2023-07-29T16:27:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 0x7f0a2819f5a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000ef30 tx=0x7f0a2000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a1e7fc700 1 -- 172.21.15.31:0/695887945 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20018070 con 0x7f0a281080c0 2023-07-29T16:27:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a1e7fc700 1 -- 172.21.15.31:0/695887945 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a2000f040 con 0x7f0a281080c0 2023-07-29T16:27:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a1e7fc700 1 -- 172.21.15.31:0/695887945 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000cd00 con 0x7f0a281080c0 2023-07-29T16:27:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a240095f0 con 0x7f0a281080c0 2023-07-29T16:27:22.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.958+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a281a0440 con 0x7f0a281080c0 2023-07-29T16:27:22.962 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.962+0000 7f0a1e7fc700 1 -- 172.21.15.31:0/695887945 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0a20003a00 con 0x7f0a281080c0 2023-07-29T16:27:22.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 Processor -- start 2023-07-29T16:27:22.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 -- start start 2023-07-29T16:27:22.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd02e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a1e7fc700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a140a08b0 0x7f0a140a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd02ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd02f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8ccf85a10 con 0x55f8cd02f4d0 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8ccf833d0 con 0x55f8cd02e690 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.964+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8ccf7ecc0 con 0x55f8cd02dd40 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a1e7fc700 1 -- 172.21.15.31:0/695887945 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0a200cb5a0 con 0x7f0a281080c0 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd02e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd02e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52468/0 (socket says 172.21.15.31:52468) 2023-07-29T16:27:22.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 -- 172.21.15.31:0/415900304 learned_addr learned my addr 172.21.15.31:0/415900304 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a140a08b0 0x7f0a140a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd02f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd02ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 -- 172.21.15.31:0/415900304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 msgr2=0x55f8cd02ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd02ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 -- 172.21.15.31:0/415900304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 msgr2=0x55f8cd02f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd02f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2d39b700 1 -- 172.21.15.31:0/415900304 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8ccdacde0 con 0x55f8cd02dd40 2023-07-29T16:27:22.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.965+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd02ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:22.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.966+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd02e150 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0a24015000 tx=0x7f0a24005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45341dc7ac4effc0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.966+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a140a08b0 0x7f0a140a2d60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55f8cd02eaa0 tx=0x7f0a18009a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.966+0000 7f0a137fe700 1 -- 172.21.15.31:0/415900304 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2401d070 con 0x55f8cd02dd40 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.966+0000 7f0a137fe700 1 -- 172.21.15.31:0/415900304 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a2401e040 con 0x55f8cd02dd40 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.966+0000 7f0a137fe700 1 -- 172.21.15.31:0/415900304 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a24004980 con 0x55f8cd02dd40 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 -- 172.21.15.31:0/415900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 msgr2=0x55f8cd02e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd02e150 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0a24015000 tx=0x7f0a24005ea0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 -- 172.21.15.31:0/415900304 shutdown_connections 2023-07-29T16:27:22.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd02f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd02e150 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/415900304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd02ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 -- 172.21.15.31:0/415900304 >> 172.21.15.31:0/415900304 conn(0x55f8cd023540 msgr2=0x55f8cd028ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 -- 172.21.15.31:0/415900304 shutdown_connections 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 -- 172.21.15.31:0/415900304 wait complete. 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.967+0000 7f0a42e86b80 1 -- start start 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 Processor -- start 2023-07-29T16:27:22.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd037900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd037e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd03ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8ccf1b510 con 0x55f8cd02f4d0 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8ccf1b1b0 con 0x55f8cd02e690 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cce02920 con 0x55f8cd02dd40 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a42e86b80 10 client.0 _ll_register_callbacks cb 0x55f8ccf24620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:22.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd037900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a2cb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd037e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a2cb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd037e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38984/0 (socket says 172.21.15.31:38984) 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd037900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52480/0 (socket says 172.21.15.31:52480) 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a2cb9a700 1 -- 172.21.15.31:0/4162073063 learned_addr learned my addr 172.21.15.31:0/4162073063 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.968+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd03ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2d39b700 1 -- 172.21.15.31:0/4162073063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 msgr2=0x55f8cd037e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd037e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2d39b700 1 -- 172.21.15.31:0/4162073063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 msgr2=0x55f8cd03ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd03ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2d39b700 1 -- 172.21.15.31:0/4162073063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a2810b160 con 0x55f8cd02dd40 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd037e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd037900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400b680 tx=0x7f0a24022ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83fbfc1098d5a132 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a10ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.969+0000 7f0a10ff9700 1 -- 172.21.15.31:0/4162073063 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2401d030 con 0x55f8cd02dd40 2023-07-29T16:27:22.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8ccfa9d10 con 0x55f8cd02dd40 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a10ff9700 1 -- 172.21.15.31:0/4162073063 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0a240058e0 con 0x55f8cd02dd40 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a10ff9700 1 -- 172.21.15.31:0/4162073063 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a24005bc0 con 0x55f8cd02dd40 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 10 client.29032 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f8ccd95c60 con 0x55f8cd02dd40 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a10ff9700 1 -- 172.21.15.31:0/4162073063 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8ccfa9d10 con 0x55f8cd02dd40 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a10ff9700 1 -- 172.21.15.31:0/4162073063 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0a2402c400 con 0x55f8cd02dd40 2023-07-29T16:27:22.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 10 client.29032 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:22.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 10 client.29032 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:27:22.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 20 client.29032 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:22.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd043180 0x55f8cd045590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 4 client.29032 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:22.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.970+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8ccffcd70 con 0x55f8cd043180 2023-07-29T16:27:22.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.971+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd043180 0x55f8cd045590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.971+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd043180 0x55f8cd045590 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d77aa2ff0adf083a server_cookie=261b2ef96ce5eb02 in_seq=0 out_seq=0 2023-07-29T16:27:22.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.971+0000 7f0a10ff9700 10 client.29032 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a10ff9700 1 -- 172.21.15.31:0/4162073063 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f8ccffcd70 con 0x55f8cd043180 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a10ff9700 10 client.29032 handle_command_reply: tid=0 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 client.29032 shutdown 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 msgr2=0x55f8cd037900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd037900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400b680 tx=0x7f0a24022ba0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 shutdown_connections 2023-07-29T16:27:22.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd03ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd037900 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd043180 0x55f8cd045590 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/4162073063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd037e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a10ff9700 0 client.29032 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 >> 172.21.15.31:0/4162073063 conn(0x55f8cd023540 msgr2=0x55f8cd0281c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 shutdown_connections 2023-07-29T16:27:22.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 1 -- 172.21.15.31:0/4162073063 wait complete. 2023-07-29T16:27:22.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.972+0000 7f0a42e86b80 20 client.29032 trim_cache size 0 max 16384 2023-07-29T16:27:22.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.986+0000 7f0a42e86b80 1 Processor -- start 2023-07-29T16:27:22.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 -- start start 2023-07-29T16:27:22.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 0x55f8cd03ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02e690 0x55f8cd032330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd032870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd038a80 con 0x55f8cd02f4d0 2023-07-29T16:27:22.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd042f30 con 0x55f8cd02dd40 2023-07-29T16:27:22.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd035590 con 0x55f8cd02e690 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 0x55f8cd03ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 0x55f8cd03ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38992/0 (socket says 172.21.15.31:38992) 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 -- 172.21.15.31:0/1074961297 learned_addr learned my addr 172.21.15.31:0/1074961297 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02e690 0x55f8cd032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd032870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 -- 172.21.15.31:0/1074961297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02e690 msgr2=0x55f8cd032330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02e690 0x55f8cd032330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 -- 172.21.15.31:0/1074961297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 msgr2=0x55f8cd032870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.987+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd032870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.988+0000 7f0a2d39b700 1 -- 172.21.15.31:0/1074961297 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8ccdd3250 con 0x55f8cd02dd40 2023-07-29T16:27:22.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.988+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 0x55f8cd03ce30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0a24000c00 tx=0x7f0a24009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5e4851d0444620 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.988+0000 7f0a13fff700 1 -- 172.21.15.31:0/1074961297 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2401d070 con 0x55f8cd02dd40 2023-07-29T16:27:22.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.988+0000 7f0a13fff700 1 -- 172.21.15.31:0/1074961297 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a24005bc0 con 0x55f8cd02dd40 2023-07-29T16:27:22.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.988+0000 7f0a13fff700 1 -- 172.21.15.31:0/1074961297 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2400bdf0 con 0x55f8cd02dd40 2023-07-29T16:27:22.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1074961297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 msgr2=0x55f8cd03ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 0x55f8cd03ce30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f0a24000c00 tx=0x7f0a24009310 comp rx=0 tx=0).stop 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1074961297 shutdown_connections 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd032870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02dd40 0x55f8cd03ce30 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1074961297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02e690 0x55f8cd032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1074961297 >> 172.21.15.31:0/1074961297 conn(0x55f8cd023540 msgr2=0x55f8cd024a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:22.993 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1074961297 shutdown_connections 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1074961297 wait complete. 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 -- start start 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.989+0000 7f0a42e86b80 1 Processor -- start 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd0974c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd099a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd09bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.994 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd09c4a0 con 0x55f8cd02f4d0 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd09c5e0 con 0x55f8cd02e690 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8cd09c720 con 0x55f8cd02dd40 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a42e86b80 10 client.0 _ll_register_callbacks cb 0x55f8ccf24620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39000/0 (socket says 172.21.15.31:39000) 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 -- 172.21.15.31:0/1408030219 learned_addr learned my addr 172.21.15.31:0/1408030219 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:22.995 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2db9c700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd09bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd0974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 -- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 msgr2=0x55f8cd0974c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd0974c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 -- 172.21.15.31:0/1408030219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 msgr2=0x55f8cd09bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.990+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd09bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a2cb9a700 1 -- 172.21.15.31:0/1408030219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8ccd95c60 con 0x55f8cd02e690 2023-07-29T16:27:22.996 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a2cb9a700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd099a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x55f8cd045e80 tx=0x7f0a18004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50322df25c05b875 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a10ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a10ff9700 1 -- 172.21.15.31:0/1408030219 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18024040 con 0x55f8cd02e690 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8ccf77d40 con 0x55f8cd02e690 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a10ff9700 1 -- 172.21.15.31:0/1408030219 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0a18030070 con 0x55f8cd02e690 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a10ff9700 1 -- 172.21.15.31:0/1408030219 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a1800f7a0 con 0x55f8cd02e690 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a42e86b80 10 client.29036 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.991+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f8ccdab2b0 con 0x55f8cd02e690 2023-07-29T16:27:22.997 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a10ff9700 1 -- 172.21.15.31:0/1408030219 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f8ccf77d40 con 0x55f8cd02e690 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a10ff9700 1 -- 172.21.15.31:0/1408030219 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f0a18003bf0 con 0x55f8cd02e690 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a42e86b80 10 client.29036 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a42e86b80 10 client.29036 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a42e86b80 20 client.29036 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd09e110 0x55f8cd0a0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a42e86b80 4 client.29036 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f8ccf77a60 con 0x55f8cd09e110 2023-07-29T16:27:22.998 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.992+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd09e110 0x55f8cd0a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:22.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a2d39b700 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd09e110 0x55f8cd0a0520 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c219c4d94b314950 server_cookie=64a2aab0d45f57fd in_seq=0 out_seq=0 2023-07-29T16:27:22.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a10ff9700 10 client.29036 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a10ff9700 1 -- 172.21.15.31:0/1408030219 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f8ccf77a60 con 0x55f8cd09e110 2023-07-29T16:27:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a10ff9700 10 client.29036 handle_command_reply: tid=0 2023-07-29T16:27:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 client.29036 shutdown 2023-07-29T16:27:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 msgr2=0x55f8cd099a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd099a10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x55f8cd045e80 tx=0x7f0a18004750 comp rx=0 tx=0).stop 2023-07-29T16:27:23.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 shutdown_connections 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f8cd02f4d0 0x55f8cd09bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f8cd02dd40 0x55f8cd0974c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55f8cd09e110 0x55f8cd0a0520 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 --2- 172.21.15.31:0/1408030219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8cd02e690 0x55f8cd099a10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 >> 172.21.15.31:0/1408030219 conn(0x55f8cd023540 msgr2=0x55f8cd032330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a10ff9700 0 client.29036 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:23.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 shutdown_connections 2023-07-29T16:27:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.993+0000 7f0a42e86b80 1 -- 172.21.15.31:0/1408030219 wait complete. 2023-07-29T16:27:23.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a42e86b80 20 client.29036 trim_cache size 0 max 16384 2023-07-29T16:27:23.002 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a140a08b0 msgr2=0x7f0a140a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a140a08b0 0x7f0a140a2d60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55f8cd02eaa0 tx=0x7f0a18009a60 comp rx=0 tx=0).stop 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 msgr2=0x7f0a2819f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 0x7f0a2819f5a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000ef30 tx=0x7f0a2000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 shutdown_connections 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a140a08b0 0x7f0a140a2d60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a281080c0 0x7f0a2819f5a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a28073750 0x7f0a28197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 --2- 172.21.15.31:0/695887945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a28074d30 0x7f0a28198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 >> 172.21.15.31:0/695887945 conn(0x7f0a280fb850 msgr2=0x7f0a2810a420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 shutdown_connections 2023-07-29T16:27:23.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:22.994+0000 7f0a2e39d700 1 -- 172.21.15.31:0/695887945 wait complete. 2023-07-29T16:27:23.012 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:27:23.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:22 smithi039 ceph-mon[128858]: pgmap v728: 97 pgs: 97 active+clean; 169 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.0 MiB/s wr, 207 op/s 2023-07-29T16:27:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 -- 172.21.15.31:0/1645431795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 msgr2=0x7fcaa0101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/1645431795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 0x7fcaa0101970 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fca94009990 tx=0x7fca94009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 -- 172.21.15.31:0/1645431795 shutdown_connections 2023-07-29T16:27:23.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/1645431795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0101eb0 0x7fcaa0108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/1645431795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaa0100330 0x7fcaa0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/1645431795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 0x7fcaa0101970 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 -- 172.21.15.31:0/1645431795 >> 172.21.15.31:0/1645431795 conn(0x7fcaa00fb970 msgr2=0x7fcaa00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 -- 172.21.15.31:0/1645431795 shutdown_connections 2023-07-29T16:27:23.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.230+0000 7fcaa69ff700 1 -- 172.21.15.31:0/1645431795 wait complete. 2023-07-29T16:27:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 Processor -- start 2023-07-29T16:27:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 -- start start 2023-07-29T16:27:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 0x7fcaa01960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 0x7fcaa0196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaa0101eb0 0x7fcaa019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa019dc50 con 0x7fcaa0101eb0 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.231+0000 7fcaa69ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa019dd90 con 0x7fcaa0101520 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fcaa69ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa019df20 con 0x7fcaa0100330 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 0x7fcaa01960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 0x7fcaa01960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52522/0 (socket says 172.21.15.31:52522) 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 -- 172.21.15.31:0/2886456181 learned_addr learned my addr 172.21.15.31:0/2886456181 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fcaa4f9c700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaa0101eb0 0x7fcaa019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 0x7fcaa0196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 -- 172.21.15.31:0/2886456181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 msgr2=0x7fcaa0196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 0x7fcaa0196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 -- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaa0101eb0 msgr2=0x7fcaa019d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaa0101eb0 0x7fcaa019d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.232+0000 7fca9ffff700 1 -- 172.21.15.31:0/2886456181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940095f0 con 0x7fcaa0100330 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.233+0000 7fca9ffff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 0x7fcaa01960d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fca9000e970 tx=0x7fca9000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.233+0000 7fca9d7fa700 1 -- 172.21.15.31:0/2886456181 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90009790 con 0x7fcaa0100330 2023-07-29T16:27:23.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.233+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa019e200 con 0x7fcaa0100330 2023-07-29T16:27:23.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.233+0000 7fca9d7fa700 1 -- 172.21.15.31:0/2886456181 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca9000c7e0 con 0x7fcaa0100330 2023-07-29T16:27:23.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.233+0000 7fca9d7fa700 1 -- 172.21.15.31:0/2886456181 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90003c70 con 0x7fcaa0100330 2023-07-29T16:27:23.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.233+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa019e710 con 0x7fcaa0100330 2023-07-29T16:27:23.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.238+0000 7fca9d7fa700 1 -- 172.21.15.31:0/2886456181 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fca9000cc50 con 0x7fcaa0100330 2023-07-29T16:27:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.239+0000 7fcaba24cb80 1 Processor -- start 2023-07-29T16:27:23.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.239+0000 7fcaba24cb80 1 -- start start 2023-07-29T16:27:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2305400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b2306170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2306b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b2258a10 con 0x5585b2306700 2023-07-29T16:27:23.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b22563d0 con 0x5585b2305940 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b2251cc0 con 0x5585b2304ff0 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaa4f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2305400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9d7fa700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca880a0790 0x7fca880a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b2306170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaa4f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2305400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52532/0 (socket says 172.21.15.31:52532) 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaa4f9c700 1 -- 172.21.15.31:0/274476513 learned_addr learned my addr 172.21.15.31:0/274476513 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2306b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fcaa4f9c700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca880a0790 0x7fca880a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9d7fa700 1 -- 172.21.15.31:0/2886456181 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca90014070 con 0x7fcaa0100330 2023-07-29T16:27:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9ffff700 1 -- 172.21.15.31:0/274476513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 msgr2=0x5585b2305400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9ffff700 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2305400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9ffff700 1 -- 172.21.15.31:0/274476513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 msgr2=0x5585b2306b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.240+0000 7fca9ffff700 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2306b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fca9ffff700 1 -- 172.21.15.31:0/274476513 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585b207fde0 con 0x5585b2305940 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2306b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fca9ffff700 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b2306170 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fca90095ee0 tx=0x7fca9000cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2175b4b6e7aca31 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fca867fc700 1 -- 172.21.15.31:0/274476513 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90095d60 con 0x5585b2305940 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fcaa4f9c700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca880a0790 0x7fca880a2c40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x5585b2304d90 tx=0x7fca98004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fca867fc700 1 -- 172.21.15.31:0/274476513 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca900d0040 con 0x5585b2305940 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.241+0000 7fca867fc700 1 -- 172.21.15.31:0/274476513 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca900a1950 con 0x5585b2305940 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 -- 172.21.15.31:0/274476513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 msgr2=0x5585b2306170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b2306170 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fca90095ee0 tx=0x7fca9000cf50 comp rx=0 tx=0).stop 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 -- 172.21.15.31:0/274476513 shutdown_connections 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2306b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2305400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/274476513 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b2306170 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 -- 172.21.15.31:0/274476513 >> 172.21.15.31:0/274476513 conn(0x5585b22f63e0 msgr2=0x5585b22fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 -- 172.21.15.31:0/274476513 shutdown_connections 2023-07-29T16:27:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.242+0000 7fcaba24cb80 1 -- 172.21.15.31:0/274476513 wait complete. 2023-07-29T16:27:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.243+0000 7fcaba24cb80 1 -- start start 2023-07-29T16:27:23.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.243+0000 7fcaba24cb80 1 Processor -- start 2023-07-29T16:27:23.248 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b230ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2305940 0x5585b230f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2306700 0x5585b2316160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b21ee510 con 0x5585b2305940 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b21ee1b0 con 0x5585b2306700 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b20d5920 con 0x5585b2304ff0 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaba24cb80 10 client.0 _ll_register_callbacks cb 0x5585b21f7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:23.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b230ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b230ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52544/0 (socket says 172.21.15.31:52544) 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 -- 172.21.15.31:0/991836191 learned_addr learned my addr 172.21.15.31:0/991836191 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fcaa4f9c700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2306700 0x5585b2316160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2305940 0x5585b230f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 -- 172.21.15.31:0/991836191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2306700 msgr2=0x5585b2316160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2306700 0x5585b2316160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 -- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2305940 msgr2=0x5585b230f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2305940 0x5585b230f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 -- 172.21.15.31:0/991836191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa019e710 con 0x5585b2304ff0 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2305940 0x5585b230f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca9ffff700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b230ebe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fca900951f0 tx=0x7fca90098dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42dcae91c12035bd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.244+0000 7fca7f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fca7f7fe700 1 -- 172.21.15.31:0/991836191 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca90095b80 con 0x5585b2304ff0 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585b227cd10 con 0x5585b2304ff0 2023-07-29T16:27:23.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fca7f7fe700 1 -- 172.21.15.31:0/991836191 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca90099470 con 0x5585b2304ff0 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fca7f7fe700 1 -- 172.21.15.31:0/991836191 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca90095d40 con 0x5585b2304ff0 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fcaba24cb80 10 client.29050 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5585b2068c60 con 0x5585b2304ff0 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fca7f7fe700 1 -- 172.21.15.31:0/991836191 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5585b227cd10 con 0x5585b2304ff0 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fca7f7fe700 1 -- 172.21.15.31:0/991836191 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca900a1530 con 0x5585b2304ff0 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fcaba24cb80 10 client.29050 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:23.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.245+0000 7fcaba24cb80 10 client.29050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fcaba24cb80 20 client.29050 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b231a460 0x5585b231c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fcaba24cb80 4 client.29050 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585b22cfd70 con 0x5585b231a460 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b231a460 0x5585b231c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b231a460 0x5585b231c870 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=866f7ca7e9065601 server_cookie=7a3d3630a86e82cc in_seq=0 out_seq=0 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.246+0000 7fca7f7fe700 10 client.29050 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:23.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fca7f7fe700 1 -- 172.21.15.31:0/991836191 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5585b22cfd70 con 0x5585b231a460 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fca7f7fe700 10 client.29050 handle_command_reply: tid=0 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fcaba24cb80 1 client.29050 shutdown 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 msgr2=0x5585b230ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b230ebe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fca900951f0 tx=0x7fca90098dd0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 shutdown_connections 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.247+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b231a460 0x5585b231c870 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2306700 0x5585b2316160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b230ebe0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/991836191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2305940 0x5585b230f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 >> 172.21.15.31:0/991836191 conn(0x5585b22f63e0 msgr2=0x5585b22fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fca7f7fe700 0 client.29050 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:23.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 shutdown_connections 2023-07-29T16:27:23.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 1 -- 172.21.15.31:0/991836191 wait complete. 2023-07-29T16:27:23.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.248+0000 7fcaba24cb80 20 client.29050 trim_cache size 0 max 16384 2023-07-29T16:27:23.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.262+0000 7fcaba24cb80 1 Processor -- start 2023-07-29T16:27:23.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 -- start start 2023-07-29T16:27:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2314110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b230cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b230d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b22f84d0 con 0x5585b2306700 2023-07-29T16:27:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b230fd90 con 0x5585b2305940 2023-07-29T16:27:23.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b230c8f0 con 0x5585b2304ff0 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fca9f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b230cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2314110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fca9f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b230cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39032/0 (socket says 172.21.15.31:39032) 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2314110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52560/0 (socket says 172.21.15.31:52560) 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fca9f7fe700 1 -- 172.21.15.31:0/4052780340 learned_addr learned my addr 172.21.15.31:0/4052780340 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.263+0000 7fcaa4f9c700 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b230d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9f7fe700 1 -- 172.21.15.31:0/4052780340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 msgr2=0x5585b2314110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2314110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9f7fe700 1 -- 172.21.15.31:0/4052780340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 msgr2=0x5585b230d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b230d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9f7fe700 1 -- 172.21.15.31:0/4052780340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585b20a6250 con 0x5585b2305940 2023-07-29T16:27:23.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9ffff700 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2314110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:23.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b230cb10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fca94009990 tx=0x7fca94009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2b8073d74e9a6da server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca86ffd700 1 -- 172.21.15.31:0/4052780340 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9401d030 con 0x5585b2305940 2023-07-29T16:27:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca86ffd700 1 -- 172.21.15.31:0/4052780340 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca9401e040 con 0x5585b2305940 2023-07-29T16:27:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.264+0000 7fca86ffd700 1 -- 172.21.15.31:0/4052780340 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9400be20 con 0x5585b2305940 2023-07-29T16:27:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 -- 172.21.15.31:0/4052780340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 msgr2=0x5585b230cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b230cb10 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fca94009990 tx=0x7fca94009310 comp rx=0 tx=0).stop 2023-07-29T16:27:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 -- 172.21.15.31:0/4052780340 shutdown_connections 2023-07-29T16:27:23.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b230d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2304ff0 0x5585b2314110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/4052780340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2305940 0x5585b230cb10 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 -- 172.21.15.31:0/4052780340 >> 172.21.15.31:0/4052780340 conn(0x5585b22f63e0 msgr2=0x5585b2309660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 -- 172.21.15.31:0/4052780340 shutdown_connections 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.265+0000 7fcaba24cb80 1 -- 172.21.15.31:0/4052780340 wait complete. 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 -- start start 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 Processor -- start 2023-07-29T16:27:23.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2304ff0 0x5585b2313ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2305940 0x5585b2312570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2312ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b2312ff0 con 0x5585b2306700 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b2313160 con 0x5585b2304ff0 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585b23132d0 con 0x5585b2305940 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.266+0000 7fcaba24cb80 10 client.0 _ll_register_callbacks cb 0x5585b21f7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2305940 0x5585b2312570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2304ff0 0x5585b2313ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2305940 0x5585b2312570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52564/0 (socket says 172.21.15.31:52564) 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9f7fe700 1 -- 172.21.15.31:0/3411585592 learned_addr learned my addr 172.21.15.31:0/3411585592 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fcaa4f9c700 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2312ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 -- 172.21.15.31:0/3411585592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2305940 msgr2=0x5585b2312570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2305940 0x5585b2312570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 -- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 msgr2=0x5585b2312ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2312ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 -- 172.21.15.31:0/3411585592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5585b2068c60 con 0x5585b2304ff0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca9ffff700 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2304ff0 0x5585b2313ea0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fca90095970 tx=0x7fca9000cf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b1ab63ee0fd4571 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca7f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca7f7fe700 1 -- 172.21.15.31:0/3411585592 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca900955d0 con 0x5585b2304ff0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.267+0000 7fca7f7fe700 1 -- 172.21.15.31:0/3411585592 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca90099470 con 0x5585b2304ff0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca900d0040 con 0x5585b2304ff0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fca7f7fe700 1 -- 172.21.15.31:0/3411585592 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca900a18c0 con 0x5585b2304ff0 2023-07-29T16:27:23.273 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 10 client.29054 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7fcaa01090a0 con 0x5585b2304ff0 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fca7f7fe700 1 -- 172.21.15.31:0/3411585592 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca900c7080 con 0x5585b2304ff0 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fca7f7fe700 1 -- 172.21.15.31:0/3411585592 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca900984a0 con 0x5585b2304ff0 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 10 client.29054 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 10 client.29054 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 20 client.29054 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b236d270 0x5585b236f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 4 client.29054 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:27:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.268+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5585b224ad40 con 0x5585b236d270 2023-07-29T16:27:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.269+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b236d270 0x5585b236f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.269+0000 7fca9f7fe700 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b236d270 0x5585b236f650 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24228e97a0f566a8 server_cookie=297b5ce29e316ed2 in_seq=0 out_seq=0 2023-07-29T16:27:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.269+0000 7fca7f7fe700 10 client.29054 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fca7f7fe700 1 -- 172.21.15.31:0/3411585592 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5585b224ad40 con 0x5585b236d270 2023-07-29T16:27:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fca7f7fe700 10 client.29054 handle_command_reply: tid=0 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 client.29054 shutdown 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2304ff0 msgr2=0x5585b2313ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2304ff0 0x5585b2313ea0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fca90095970 tx=0x7fca9000cf50 comp rx=0 tx=0).stop 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 shutdown_connections 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x5585b236d270 0x5585b236f650 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585b2306700 0x5585b2312ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585b2304ff0 0x5585b2313ea0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 --2- 172.21.15.31:0/3411585592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585b2305940 0x5585b2312570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 >> 172.21.15.31:0/3411585592 conn(0x5585b22f63e0 msgr2=0x5585b230cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fca7f7fe700 0 client.29054 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 shutdown_connections 2023-07-29T16:27:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 1 -- 172.21.15.31:0/3411585592 wait complete. 2023-07-29T16:27:23.278 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.270+0000 7fcaba24cb80 20 client.29054 trim_cache size 0 max 16384 2023-07-29T16:27:23.279 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:27:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca880a0790 msgr2=0x7fca880a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca880a0790 0x7fca880a2c40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x5585b2304d90 tx=0x7fca98004b40 comp rx=0 tx=0).stop 2023-07-29T16:27:23.279 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 msgr2=0x7fcaa01960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 0x7fcaa01960d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fca9000e970 tx=0x7fca9000ec80 comp rx=0 tx=0).stop 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 shutdown_connections 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca880a0790 0x7fca880a2c40 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcaa0101eb0 0x7fcaa019d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcaa0100330 0x7fcaa01960d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 --2- 172.21.15.31:0/2886456181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcaa0101520 0x7fcaa0196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 >> 172.21.15.31:0/2886456181 conn(0x7fcaa00fb970 msgr2=0x7fcaa00fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 shutdown_connections 2023-07-29T16:27:23.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.271+0000 7fcaa69ff700 1 -- 172.21.15.31:0/2886456181 wait complete. 2023-07-29T16:27:23.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:27:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 -- 172.21.15.31:0/3708273993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 msgr2=0x7f58e4100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 --2- 172.21.15.31:0/3708273993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4100730 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009990 tx=0x7f58d4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 -- 172.21.15.31:0/3708273993 shutdown_connections 2023-07-29T16:27:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 --2- 172.21.15.31:0/3708273993 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58e4101ea0 0x7f58e4108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 --2- 172.21.15.31:0/3708273993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4100730 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 --2- 172.21.15.31:0/3708273993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58e4101510 0x7f58e4101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 -- 172.21.15.31:0/3708273993 >> 172.21.15.31:0/3708273993 conn(0x7f58e40fb920 msgr2=0x7f58e40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 -- 172.21.15.31:0/3708273993 shutdown_connections 2023-07-29T16:27:23.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.511+0000 7f58e9bff700 1 -- 172.21.15.31:0/3708273993 wait complete. 2023-07-29T16:27:23.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.512+0000 7f58e9bff700 1 Processor -- start 2023-07-29T16:27:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 -- start start 2023-07-29T16:27:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58e4101510 0x7f58e4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58e4101ea0 0x7f58e4073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e4073e20 con 0x7f58e4101510 2023-07-29T16:27:23.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e4073f90 con 0x7f58e4100320 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e9bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e41a0130 con 0x7f58e4101ea0 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39068/0 (socket says 172.21.15.31:39068) 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.513+0000 7f58e37fe700 1 -- 172.21.15.31:0/2530900180 learned_addr learned my addr 172.21.15.31:0/2530900180 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e3fff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58e4101ea0 0x7f58e4073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58e4101510 0x7f58e4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e37fe700 1 -- 172.21.15.31:0/2530900180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58e4101ea0 msgr2=0x7f58e4073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e37fe700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58e4101ea0 0x7f58e4073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e37fe700 1 -- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58e4101510 msgr2=0x7f58e4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e37fe700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58e4101510 0x7f58e4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e37fe700 1 -- 172.21.15.31:0/2530900180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40095f0 con 0x7f58e4100320 2023-07-29T16:27:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e37fe700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4074b00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f58d4004730 tx=0x7f58d4004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e0ff9700 1 -- 172.21.15.31:0/2530900180 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d401d070 con 0x7f58e4100320 2023-07-29T16:27:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e41a03b0 con 0x7f58e4100320 2023-07-29T16:27:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e0ff9700 1 -- 172.21.15.31:0/2530900180 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58d4021410 con 0x7f58e4100320 2023-07-29T16:27:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.514+0000 7f58e0ff9700 1 -- 172.21.15.31:0/2530900180 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4017650 con 0x7f58e4100320 2023-07-29T16:27:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.515+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e41a07e0 con 0x7f58e4100320 2023-07-29T16:27:23.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.518+0000 7f58e0ff9700 1 -- 172.21.15.31:0/2530900180 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f58d4021580 con 0x7f58e4100320 2023-07-29T16:27:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.519+0000 7f58fd4d3b80 1 Processor -- start 2023-07-29T16:27:23.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 -- start start 2023-07-29T16:27:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72baff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72bbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72bc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58e0ff9700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58cc0a0790 0x7f58cc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c7212a10 con 0x5600c72bc2f0 2023-07-29T16:27:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72103d0 con 0x5600c72babe0 2023-07-29T16:27:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c720bcc0 con 0x5600c72bb530 2023-07-29T16:27:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58e0ff9700 1 -- 172.21.15.31:0/2530900180 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58d40cced0 con 0x7f58e4100320 2023-07-29T16:27:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72baff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72bbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72baff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39084/0 (socket says 172.21.15.31:39084) 2023-07-29T16:27:23.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.520+0000 7f58e3fff700 1 -- 172.21.15.31:0/2869032891 learned_addr learned my addr 172.21.15.31:0/2869032891 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58cc0a0790 0x7f58cc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e37fe700 1 -- 172.21.15.31:0/2869032891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 msgr2=0x5600c72baff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e37fe700 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72baff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e37fe700 1 -- 172.21.15.31:0/2869032891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 msgr2=0x5600c72bc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e3fff700 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72bc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e37fe700 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72bc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e37fe700 1 -- 172.21.15.31:0/2869032891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600c7039de0 con 0x5600c72bb530 2023-07-29T16:27:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e3fff700 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:27:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58e37fe700 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72bbd60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f58d4096b00 tx=0x7f58d4009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=280ac52415ff14c7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.521+0000 7f58c9ffb700 1 -- 172.21.15.31:0/2869032891 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d406ccc0 con 0x5600c72bb530 2023-07-29T16:27:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.522+0000 7f58c9ffb700 1 -- 172.21.15.31:0/2869032891 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58d406ceb0 con 0x5600c72bb530 2023-07-29T16:27:23.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.522+0000 7f58c9ffb700 1 -- 172.21.15.31:0/2869032891 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d409a860 con 0x5600c72bb530 2023-07-29T16:27:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/2869032891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 msgr2=0x5600c72bbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72bbd60 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f58d4096b00 tx=0x7f58d4009310 comp rx=0 tx=0).stop 2023-07-29T16:27:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58cc0a0790 0x7f58cc0a2c40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009c60 tx=0x7f58d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/2869032891 shutdown_connections 2023-07-29T16:27:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72bc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72baff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/2869032891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72bbd60 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/2869032891 >> 172.21.15.31:0/2869032891 conn(0x5600c72b03e0 msgr2=0x5600c72b5990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/2869032891 shutdown_connections 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.523+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/2869032891 wait complete. 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.524+0000 7f58fd4d3b80 1 -- start start 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.524+0000 7f58fd4d3b80 1 Processor -- start 2023-07-29T16:27:23.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bb530 0x5600c72c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bc2f0 0x5600c72cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c71a8510 con 0x5600c72bb530 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c71a81b0 con 0x5600c72babe0 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c708f920 con 0x5600c72bc2f0 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58fd4d3b80 10 client.0 _ll_register_callbacks cb 0x5600c71b1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39094/0 (socket says 172.21.15.31:39094) 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58e37fe700 1 -- 172.21.15.31:0/1557756033 learned_addr learned my addr 172.21.15.31:0/1557756033 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bb530 0x5600c72c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58e37fe700 1 -- 172.21.15.31:0/1557756033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bc2f0 msgr2=0x5600c72cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.525+0000 7f58e37fe700 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bc2f0 0x5600c72cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58e37fe700 1 -- 172.21.15.31:0/1557756033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bb530 msgr2=0x5600c72c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58e37fe700 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bb530 0x5600c72c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58e37fe700 1 -- 172.21.15.31:0/1557756033 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e41a07e0 con 0x5600c72babe0 2023-07-29T16:27:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58e37fe700 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72c44b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f58d4000c00 tx=0x7f58d406cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7ee06f358601b47 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58c2ffd700 1 -- 172.21.15.31:0/1557756033 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d409a860 con 0x5600c72babe0 2023-07-29T16:27:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58c2ffd700 1 -- 172.21.15.31:0/1557756033 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58d409a9c0 con 0x5600c72babe0 2023-07-29T16:27:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600c7236d10 con 0x5600c72babe0 2023-07-29T16:27:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58c2ffd700 1 -- 172.21.15.31:0/1557756033 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58d40a2530 con 0x5600c72babe0 2023-07-29T16:27:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58fd4d3b80 10 client.29066 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.526+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5600c7022c60 con 0x5600c72babe0 2023-07-29T16:27:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58c2ffd700 1 -- 172.21.15.31:0/1557756033 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5600c7236d10 con 0x5600c72babe0 2023-07-29T16:27:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58c2ffd700 1 -- 172.21.15.31:0/1557756033 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f58d4099560 con 0x5600c72babe0 2023-07-29T16:27:23.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58fd4d3b80 10 client.29066 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58fd4d3b80 10 client.29066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58fd4d3b80 20 client.29066 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c72cfd20 0x5600c72d2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58fd4d3b80 4 client.29066 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600c7289d70 con 0x5600c72cfd20 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.527+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c72cfd20 0x5600c72d2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.528+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c72cfd20 0x5600c72d2100 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc0fb15064494775 server_cookie=8edd589b90558d89 in_seq=0 out_seq=0 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.528+0000 7f58c2ffd700 10 client.29066 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.528+0000 7f58c2ffd700 1 -- 172.21.15.31:0/1557756033 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5600c7289d70 con 0x5600c72cfd20 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.528+0000 7f58c2ffd700 10 client.29066 handle_command_reply: tid=0 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.528+0000 7f58fd4d3b80 1 client.29066 shutdown 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 msgr2=0x5600c72c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72c44b0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f58d4000c00 tx=0x7f58d406cd70 comp rx=0 tx=0).stop 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 shutdown_connections 2023-07-29T16:27:23.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c72cfd20 0x5600c72d2100 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bc2f0 0x5600c72cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72babe0 0x5600c72c44b0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1557756033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bb530 0x5600c72c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 >> 172.21.15.31:0/1557756033 conn(0x5600c72b03e0 msgr2=0x5600c72b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58c2ffd700 0 client.29066 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 shutdown_connections 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1557756033 wait complete. 2023-07-29T16:27:23.535 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.529+0000 7f58fd4d3b80 20 client.29066 trim_cache size 0 max 16384 2023-07-29T16:27:23.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.546+0000 7f58fd4d3b80 1 Processor -- start 2023-07-29T16:27:23.546 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.546+0000 7f58fd4d3b80 1 -- start start 2023-07-29T16:27:23.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72babe0 0x5600c72b5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 0x5600c72bf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.547 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72c3960 con 0x5600c72babe0 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72b6cc0 con 0x5600c72bc2f0 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72b2460 con 0x5600c72bb530 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 0x5600c72bf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 0x5600c72bf1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39102/0 (socket says 172.21.15.31:39102) 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 -- 172.21.15.31:0/1513999960 learned_addr learned my addr 172.21.15.31:0/1513999960 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e37fe700 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72babe0 0x5600c72b5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 -- 172.21.15.31:0/1513999960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 msgr2=0x5600c72cb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72cb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 -- 172.21.15.31:0/1513999960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72babe0 msgr2=0x5600c72b5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72babe0 0x5600c72b5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.549 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.547+0000 7f58e3fff700 1 -- 172.21.15.31:0/1513999960 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600c7060250 con 0x5600c72bc2f0 2023-07-29T16:27:23.550 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.548+0000 7f58e3fff700 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 0x5600c72bf1f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007560 tx=0x7f58dc007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b29c3fad95ed46 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.548+0000 7f58ca7fc700 1 -- 172.21.15.31:0/1513999960 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc00e040 con 0x5600c72bc2f0 2023-07-29T16:27:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.548+0000 7f58ca7fc700 1 -- 172.21.15.31:0/1513999960 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58dc00f040 con 0x5600c72bc2f0 2023-07-29T16:27:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.548+0000 7f58ca7fc700 1 -- 172.21.15.31:0/1513999960 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc018670 con 0x5600c72bc2f0 2023-07-29T16:27:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1513999960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 msgr2=0x5600c72bf1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 0x5600c72bf1f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f58dc007560 tx=0x7f58dc007870 comp rx=0 tx=0).stop 2023-07-29T16:27:23.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1513999960 shutdown_connections 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bc2f0 0x5600c72bf1f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72babe0 0x5600c72b5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/1513999960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72bb530 0x5600c72cb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1513999960 >> 172.21.15.31:0/1513999960 conn(0x5600c72b03e0 msgr2=0x5600c72b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1513999960 shutdown_connections 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/1513999960 wait complete. 2023-07-29T16:27:23.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.549+0000 7f58fd4d3b80 1 -- start start 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 Processor -- start 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 0x5600c72cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bb530 0x5600c72cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72c8340 con 0x5600c72bc2f0 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72c8480 con 0x5600c72bb530 2023-07-29T16:27:23.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600c72c8640 con 0x5600c72babe0 2023-07-29T16:27:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58fd4d3b80 10 client.0 _ll_register_callbacks cb 0x5600c71b1620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.550+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 0x5600c72cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e37fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 0x5600c72cb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52628/0 (socket says 172.21.15.31:52628) 2023-07-29T16:27:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bb530 0x5600c72cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e37fe700 1 -- 172.21.15.31:0/97571962 learned_addr learned my addr 172.21.15.31:0/97571962 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e3fff700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 -- 172.21.15.31:0/97571962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 msgr2=0x5600c72cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 0x5600c72cb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 -- 172.21.15.31:0/97571962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 msgr2=0x5600c72c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 -- 172.21.15.31:0/97571962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc0092a0 con 0x5600c72bb530 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e37fe700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 0x5600c72cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58e2ffd700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bb530 0x5600c72cb7c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x5600c72d2990 tx=0x7f58d8013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d85a181aa47fe94 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.551+0000 7f58c2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58c2ffd700 1 -- 172.21.15.31:0/97571962 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d80251b0 con 0x5600c72bb530 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58d406ceb0 con 0x5600c72bb530 2023-07-29T16:27:23.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58c2ffd700 1 -- 172.21.15.31:0/97571962 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58d8025310 con 0x5600c72bb530 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58c2ffd700 1 -- 172.21.15.31:0/97571962 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58d8003680 con 0x5600c72bb530 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58fd4d3b80 10 client.29072 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5600c70382b0 con 0x5600c72bb530 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58c2ffd700 1 -- 172.21.15.31:0/97571962 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58d406ceb0 con 0x5600c72bb530 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58c2ffd700 1 -- 172.21.15.31:0/97571962 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f58d8025a40 con 0x5600c72bb530 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58fd4d3b80 10 client.29072 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.552+0000 7f58fd4d3b80 10 client.29072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:23.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58fd4d3b80 20 client.29072 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c7322cc0 0x5600c73250d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58fd4d3b80 4 client.29072 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5600c7204d40 con 0x5600c7322cc0 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58e37fe700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c7322cc0 0x5600c73250d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58e37fe700 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c7322cc0 0x5600c73250d0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=454ccc51d4c211bd server_cookie=54cbe604d9a6ed6 in_seq=0 out_seq=0 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58c2ffd700 10 client.29072 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:23.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58c2ffd700 1 -- 172.21.15.31:0/97571962 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5600c7204d40 con 0x5600c7322cc0 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58c2ffd700 10 client.29072 handle_command_reply: tid=0 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.553+0000 7f58fd4d3b80 1 client.29072 shutdown 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bb530 msgr2=0x5600c72cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bb530 0x5600c72cb7c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x5600c72d2990 tx=0x7f58d8013040 comp rx=0 tx=0).stop 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 shutdown_connections 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5600c7322cc0 0x5600c73250d0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5600c72bc2f0 0x5600c72c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5600c72babe0 0x5600c72cb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 --2- 172.21.15.31:0/97571962 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5600c72bb530 0x5600c72cb7c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58c2ffd700 0 client.29072 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 >> 172.21.15.31:0/97571962 conn(0x5600c72b03e0 msgr2=0x5600c72d1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 shutdown_connections 2023-07-29T16:27:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 1 -- 172.21.15.31:0/97571962 wait complete. 2023-07-29T16:27:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.554+0000 7f58fd4d3b80 20 client.29072 trim_cache size 0 max 16384 2023-07-29T16:27:23.562 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:27:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58cc0a0790 msgr2=0x7f58cc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58cc0a0790 0x7f58cc0a2c40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f58d8009c60 tx=0x7f58d8009260 comp rx=0 tx=0).stop 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 msgr2=0x7f58e4074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4074b00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f58d4004730 tx=0x7f58d4004760 comp rx=0 tx=0).stop 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 shutdown_connections 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58cc0a0790 0x7f58cc0a2c40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58e4101ea0 0x7f58e4073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58e4100320 0x7f58e4074b00 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 --2- 172.21.15.31:0/2530900180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58e4101510 0x7f58e4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:23.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 >> 172.21.15.31:0/2530900180 conn(0x7f58e40fb920 msgr2=0x7f58e40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 shutdown_connections 2023-07-29T16:27:23.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:23.555+0000 7f58e9bff700 1 -- 172.21.15.31:0/2530900180 wait complete. 2023-07-29T16:27:23.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/145894514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:27:23.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 909080 8.93 MB/sec execute 428 sec latency 38.276 ms 2023-07-29T16:27:23.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/145894514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:27:24.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/145894514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:27:24.575 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:27:24.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:27:24.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 -- 172.21.15.31:0/614037668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 msgr2=0x7f53f8073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/614037668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 0x7f53f8073160 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009990 tx=0x7f53e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 -- 172.21.15.31:0/614037668 shutdown_connections 2023-07-29T16:27:24.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/614037668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 0x7f53f810a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/614037668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 0x7f53f8073160 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/614037668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.799+0000 7f53fd1ff700 1 -- 172.21.15.31:0/614037668 >> 172.21.15.31:0/614037668 conn(0x7f53f80fb960 msgr2=0x7f53f80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:24.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.800+0000 7f53fd1ff700 1 -- 172.21.15.31:0/614037668 shutdown_connections 2023-07-29T16:27:24.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.800+0000 7f53fd1ff700 1 -- 172.21.15.31:0/614037668 wait complete. 2023-07-29T16:27:24.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.801+0000 7f53fd1ff700 1 Processor -- start 2023-07-29T16:27:24.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.801+0000 7f53fd1ff700 1 -- start start 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53fd1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53fd1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 0x7f53f81987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53fd1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 0x7f53f819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53fd1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f819fe10 con 0x7f53f81081f0 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53fd1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f819ff50 con 0x7f53f8074d10 2023-07-29T16:27:24.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53fd1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f81a00e0 con 0x7f53f8073730 2023-07-29T16:27:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52644/0 (socket says 172.21.15.31:52644) 2023-07-29T16:27:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 -- 172.21.15.31:0/983152811 learned_addr learned my addr 172.21.15.31:0/983152811 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f659c700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 0x7f53f81987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f759e700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 0x7f53f819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 -- 172.21.15.31:0/983152811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 msgr2=0x7f53f81987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 0x7f53f81987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 -- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 msgr2=0x7f53f819f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 0x7f53f819f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f6d9d700 1 -- 172.21.15.31:0/983152811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f0009c40 con 0x7f53f8073730 2023-07-29T16:27:24.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.802+0000 7f53f759e700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 0x7f53f819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.803+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8198290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f53e800b680 tx=0x7f53e8005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.803+0000 7f53dffff700 1 -- 172.21.15.31:0/983152811 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e801d070 con 0x7f53f8073730 2023-07-29T16:27:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.803+0000 7f53dffff700 1 -- 172.21.15.31:0/983152811 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53e8022470 con 0x7f53f8073730 2023-07-29T16:27:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.803+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e80095f0 con 0x7f53f8073730 2023-07-29T16:27:24.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.803+0000 7f53dffff700 1 -- 172.21.15.31:0/983152811 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8017650 con 0x7f53f8073730 2023-07-29T16:27:24.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.803+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f81a0650 con 0x7f53f8073730 2023-07-29T16:27:24.808 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.808+0000 7f53dffff700 1 -- 172.21.15.31:0/983152811 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f53e80225e0 con 0x7f53f8073730 2023-07-29T16:27:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 Processor -- start 2023-07-29T16:27:24.809 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 -- start start 2023-07-29T16:27:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d667070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d667de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.811 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d6687c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d5bea90 con 0x55696d666c60 2023-07-29T16:27:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d5bc450 con 0x55696d668370 2023-07-29T16:27:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.809+0000 7f5410a54b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d5b7d40 con 0x55696d6675b0 2023-07-29T16:27:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d667de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d6687c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.812 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d6687c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39148/0 (socket says 172.21.15.31:39148) 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 -- 172.21.15.31:0/1803005060 learned_addr learned my addr 172.21.15.31:0/1803005060 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f759e700 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d667070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 -- 172.21.15.31:0/1803005060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 msgr2=0x55696d667de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d667de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 -- 172.21.15.31:0/1803005060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 msgr2=0x55696d667070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d667070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 -- 172.21.15.31:0/1803005060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55696d3e5e60 con 0x55696d668370 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f759e700 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d667070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53dffff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53e00a0810 0x7f53e00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.810+0000 7f53f659c700 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d6687c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009d80 tx=0x7f53ec011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ac3f394e24c01f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.811+0000 7f53dffff700 1 -- 172.21.15.31:0/983152811 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f53e80ccfa0 con 0x7f53f8073730 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.811+0000 7f53f759e700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53e00a0810 0x7f53e00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.811+0000 7f53dd7fa700 1 -- 172.21.15.31:0/1803005060 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec004230 con 0x55696d668370 2023-07-29T16:27:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.811+0000 7f53dd7fa700 1 -- 172.21.15.31:0/1803005060 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53ec004390 con 0x55696d668370 2023-07-29T16:27:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.811+0000 7f53dd7fa700 1 -- 172.21.15.31:0/1803005060 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec019880 con 0x55696d668370 2023-07-29T16:27:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 -- 172.21.15.31:0/1803005060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 msgr2=0x55696d6687c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f53f759e700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53e00a0810 0x7f53e00a2cc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009fd0 tx=0x7f53f0009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d6687c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009d80 tx=0x7f53ec011720 comp rx=0 tx=0).stop 2023-07-29T16:27:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 -- 172.21.15.31:0/1803005060 shutdown_connections 2023-07-29T16:27:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d6687c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d667070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 --2- 172.21.15.31:0/1803005060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d667de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 -- 172.21.15.31:0/1803005060 >> 172.21.15.31:0/1803005060 conn(0x55696d65c460 msgr2=0x55696d661a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 -- 172.21.15.31:0/1803005060 shutdown_connections 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 -- 172.21.15.31:0/1803005060 wait complete. 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.812+0000 7f5410a54b80 1 -- start start 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 Processor -- start 2023-07-29T16:27:24.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d6708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d6675b0 0x55696d670e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d677e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d554590 con 0x55696d6675b0 2023-07-29T16:27:24.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d554230 con 0x55696d668370 2023-07-29T16:27:24.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d43b9a0 con 0x55696d666c60 2023-07-29T16:27:24.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f5410a54b80 10 client.0 _ll_register_callbacks cb 0x55696d55d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d6708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d6708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52658/0 (socket says 172.21.15.31:52658) 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f53f6d9d700 1 -- 172.21.15.31:0/4180067010 learned_addr learned my addr 172.21.15.31:0/4180067010 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.813+0000 7f53f759e700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d677e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f659c700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d6675b0 0x55696d670e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f6d9d700 1 -- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 msgr2=0x55696d677e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d677e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f6d9d700 1 -- 172.21.15.31:0/4180067010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d6675b0 msgr2=0x55696d670e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d6675b0 0x55696d670e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f6d9d700 1 -- 172.21.15.31:0/4180067010 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ec0099e0 con 0x55696d666c60 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d6708d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f53e809a3d0 tx=0x7f53e8005ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf3aa3fce8eb852 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53daff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53daff5700 1 -- 172.21.15.31:0/4180067010 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e80caea0 con 0x55696d666c60 2023-07-29T16:27:24.822 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53daff5700 1 -- 172.21.15.31:0/4180067010 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f53e809a6d0 con 0x55696d666c60 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55696d5e2d90 con 0x55696d666c60 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.814+0000 7f53daff5700 1 -- 172.21.15.31:0/4180067010 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53e809a4a0 con 0x55696d666c60 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 10 client.29083 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55696d3cece0 con 0x55696d666c60 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f53daff5700 1 -- 172.21.15.31:0/4180067010 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55696d5e2d90 con 0x55696d666c60 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f53daff5700 1 -- 172.21.15.31:0/4180067010 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f53e8099bc0 con 0x55696d666c60 2023-07-29T16:27:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 10 client.29083 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 10 client.29083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 20 client.29083 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d67c240 0x55696d67e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 4 client.29083 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55696d635df0 con 0x55696d67c240 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.815+0000 7f53f659c700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d67c240 0x55696d67e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.816+0000 7f53f659c700 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d67c240 0x55696d67e650 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a188806b8b3959 server_cookie=58f2df727e8d906f in_seq=0 out_seq=0 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.816+0000 7f53daff5700 10 client.29083 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.816+0000 7f53daff5700 1 -- 172.21.15.31:0/4180067010 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55696d635df0 con 0x55696d67c240 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.816+0000 7f53daff5700 10 client.29083 handle_command_reply: tid=0 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.816+0000 7f5410a54b80 1 client.29083 shutdown 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 msgr2=0x55696d6708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d6708d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f53e809a3d0 tx=0x7f53e8005ad0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 shutdown_connections 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d67c240 0x55696d67e650 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d677e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d6708d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 --2- 172.21.15.31:0/4180067010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d6675b0 0x55696d670e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f53daff5700 0 client.29083 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 >> 172.21.15.31:0/4180067010 conn(0x55696d65c460 msgr2=0x55696d6610e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:24.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 shutdown_connections 2023-07-29T16:27:24.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 1 -- 172.21.15.31:0/4180067010 wait complete. 2023-07-29T16:27:24.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.817+0000 7f5410a54b80 20 client.29083 trim_cache size 0 max 16384 2023-07-29T16:27:24.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:24 smithi156 ceph-mon[129245]: pgmap v729: 97 pgs: 97 active+clean; 166 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 2.4 MiB/s wr, 255 op/s 2023-07-29T16:27:24.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 Processor -- start 2023-07-29T16:27:24.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 -- start start 2023-07-29T16:27:24.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d670680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d677e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d670bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d662710 con 0x55696d666c60 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d65bf20 con 0x55696d668370 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.835+0000 7f5410a54b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d66fd80 con 0x55696d6675b0 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d677e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d677e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52674/0 (socket says 172.21.15.31:52674) 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 -- 172.21.15.31:0/2976574849 learned_addr learned my addr 172.21.15.31:0/2976574849 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d670680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f759e700 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d670bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 -- 172.21.15.31:0/2976574849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 msgr2=0x55696d670bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d670bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 -- 172.21.15.31:0/2976574849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 msgr2=0x55696d670680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d670680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 -- 172.21.15.31:0/2976574849 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55696d40c2d0 con 0x55696d6675b0 2023-07-29T16:27:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.836+0000 7f53f659c700 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d677e10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x55696d65c3e0 tx=0x7f53ec011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40ff9c991706d87f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:24.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.837+0000 7f53ddffb700 1 -- 172.21.15.31:0/2976574849 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec005040 con 0x55696d6675b0 2023-07-29T16:27:24.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.837+0000 7f53ddffb700 1 -- 172.21.15.31:0/2976574849 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53ec02f070 con 0x55696d6675b0 2023-07-29T16:27:24.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.837+0000 7f53ddffb700 1 -- 172.21.15.31:0/2976574849 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec02bcd0 con 0x55696d6675b0 2023-07-29T16:27:24.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.837+0000 7f5410a54b80 1 -- 172.21.15.31:0/2976574849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 msgr2=0x55696d677e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.837+0000 7f5410a54b80 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d677e10 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x55696d65c3e0 tx=0x7f53ec011720 comp rx=0 tx=0).stop 2023-07-29T16:27:24.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.837+0000 7f5410a54b80 1 -- 172.21.15.31:0/2976574849 shutdown_connections 2023-07-29T16:27:24.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d668370 0x55696d670bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d666c60 0x55696d670680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 --2- 172.21.15.31:0/2976574849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d6675b0 0x55696d677e10 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 -- 172.21.15.31:0/2976574849 >> 172.21.15.31:0/2976574849 conn(0x55696d65c460 msgr2=0x55696d66b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 -- 172.21.15.31:0/2976574849 shutdown_connections 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 -- 172.21.15.31:0/2976574849 wait complete. 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 -- start start 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.838+0000 7f5410a54b80 1 Processor -- start 2023-07-29T16:27:24.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d675000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d6675b0 0x55696d675540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d668370 0x55696d676550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d676a90 con 0x55696d668370 2023-07-29T16:27:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d676c00 con 0x55696d6675b0 2023-07-29T16:27:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55696d676dc0 con 0x55696d666c60 2023-07-29T16:27:24.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f5410a54b80 10 client.0 _ll_register_callbacks cb 0x55696d55d6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d675000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d6675b0 0x55696d675540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d675000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52678/0 (socket says 172.21.15.31:52678) 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 -- 172.21.15.31:0/3182994989 learned_addr learned my addr 172.21.15.31:0/3182994989 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f759e700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d668370 0x55696d676550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 -- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d6675b0 msgr2=0x55696d675540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d6675b0 0x55696d675540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 -- 172.21.15.31:0/3182994989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d668370 msgr2=0x55696d676550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:24.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d668370 0x55696d676550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f6d9d700 1 -- 172.21.15.31:0/3182994989 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55696d3e5e60 con 0x55696d666c60 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.839+0000 7f53f659c700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d6675b0 0x55696d675540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53f6d9d700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d675000 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f53e8005b40 tx=0x7f53e809ab60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b22017b1e97b238 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53daff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53daff5700 1 -- 172.21.15.31:0/3182994989 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e80caea0 con 0x55696d666c60 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53daff5700 1 -- 172.21.15.31:0/3182994989 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f53e809b4b0 con 0x55696d666c60 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53ec004390 con 0x55696d666c60 2023-07-29T16:27:24.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53daff5700 1 -- 172.21.15.31:0/3182994989 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53e8009290 con 0x55696d666c60 2023-07-29T16:27:24.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f5410a54b80 10 client.29089 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:24.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55696d3e4330 con 0x55696d666c60 2023-07-29T16:27:24.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53daff5700 1 -- 172.21.15.31:0/3182994989 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53ec004390 con 0x55696d666c60 2023-07-29T16:27:24.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f53daff5700 1 -- 172.21.15.31:0/3182994989 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f53e809bb30 con 0x55696d666c60 2023-07-29T16:27:24.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f5410a54b80 10 client.29089 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:24.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f5410a54b80 10 client.29089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:24.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.840+0000 7f5410a54b80 20 client.29089 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:24.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.841+0000 7f5410a54b80 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d6cfdd0 0x55696d6d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:24.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.841+0000 7f5410a54b80 4 client.29089 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:27:24.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.841+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55696d5b0dc0 con 0x55696d6cfdd0 2023-07-29T16:27:24.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.841+0000 7f53f659c700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d6cfdd0 0x55696d6d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:24.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.841+0000 7f53f659c700 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d6cfdd0 0x55696d6d21b0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ee5e060f045d691 server_cookie=837efaac7cfd0dca in_seq=0 out_seq=0 2023-07-29T16:27:24.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.841+0000 7f53daff5700 10 client.29089 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:24.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f53daff5700 1 -- 172.21.15.31:0/3182994989 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55696d5b0dc0 con 0x55696d6cfdd0 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f53daff5700 10 client.29089 handle_command_reply: tid=0 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 client.29089 shutdown 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 msgr2=0x55696d675000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d675000 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f53e8005b40 tx=0x7f53e809ab60 comp rx=0 tx=0).stop 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 shutdown_connections 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55696d6cfdd0 0x55696d6d21b0 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55696d668370 0x55696d676550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55696d666c60 0x55696d675000 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f53daff5700 0 client.29089 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 --2- 172.21.15.31:0/3182994989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55696d6675b0 0x55696d675540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 >> 172.21.15.31:0/3182994989 conn(0x55696d65c460 msgr2=0x55696d670680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 shutdown_connections 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.844+0000 7f5410a54b80 1 -- 172.21.15.31:0/3182994989 wait complete. 2023-07-29T16:27:24.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f5410a54b80 20 client.29089 trim_cache size 0 max 16384 2023-07-29T16:27:24.855 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:27:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:27:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "a3307c46-c7e9-4d74-9293-bdc634cffedd", 2023-07-29T16:27:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:27:24.855 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:27:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53e00a0810 msgr2=0x7f53e00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53e00a0810 0x7f53e00a2cc0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009fd0 tx=0x7f53f0009040 comp rx=0 tx=0).stop 2023-07-29T16:27:24.856 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 msgr2=0x7f53f8198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8198290 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f53e800b680 tx=0x7f53e8005740 comp rx=0 tx=0).stop 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 shutdown_connections 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f53e00a0810 0x7f53e00a2cc0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f53f81081f0 0x7f53f819f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53f8073730 0x7f53f8198290 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 --2- 172.21.15.31:0/983152811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53f8074d10 0x7f53f81987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 >> 172.21.15.31:0/983152811 conn(0x7f53f80fb960 msgr2=0x7f53f80fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 shutdown_connections 2023-07-29T16:27:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:24.845+0000 7f53fd1ff700 1 -- 172.21.15.31:0/983152811 wait complete. 2023-07-29T16:27:24.864 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:24 smithi031 ceph-mon[95504]: pgmap v729: 97 pgs: 97 active+clean; 166 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 2.4 MiB/s wr, 255 op/s 2023-07-29T16:27:24.867 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:27:24.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 910071 8.91 MB/sec execute 429 sec latency 43.639 ms 2023-07-29T16:27:25.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:24 smithi039 ceph-mon[128858]: pgmap v729: 97 pgs: 97 active+clean; 166 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 2.4 MiB/s wr, 255 op/s 2023-07-29T16:27:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 -- 172.21.15.31:0/3014059639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 msgr2=0x7f02c0101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 --2- 172.21.15.31:0/3014059639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0101980 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009990 tx=0x7f02b0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 -- 172.21.15.31:0/3014059639 shutdown_connections 2023-07-29T16:27:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 --2- 172.21.15.31:0/3014059639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0101ec0 0x7f02c0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 --2- 172.21.15.31:0/3014059639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c0100340 0x7f02c0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 --2- 172.21.15.31:0/3014059639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0101980 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 -- 172.21.15.31:0/3014059639 >> 172.21.15.31:0/3014059639 conn(0x7f02c00fb960 msgr2=0x7f02c00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:25.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 -- 172.21.15.31:0/3014059639 shutdown_connections 2023-07-29T16:27:25.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.087+0000 7f02c77ff700 1 -- 172.21.15.31:0/3014059639 wait complete. 2023-07-29T16:27:25.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 Processor -- start 2023-07-29T16:27:25.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 -- start start 2023-07-29T16:27:25.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0100340 0x7f02c01982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c0101ec0 0x7f02c019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c019fe40 con 0x7f02c0101ec0 2023-07-29T16:27:25.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c019ff80 con 0x7f02c0101530 2023-07-29T16:27:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.089+0000 7f02c77ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c01a0110 con 0x7f02c0100340 2023-07-29T16:27:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0100340 0x7f02c01982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39196/0 (socket says 172.21.15.31:39196) 2023-07-29T16:27:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c4d9a700 1 -- 172.21.15.31:0/708630937 learned_addr learned my addr 172.21.15.31:0/708630937 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:25.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c5d9c700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c0101ec0 0x7f02c019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 -- 172.21.15.31:0/708630937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 msgr2=0x7f02c0198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 -- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c0101ec0 msgr2=0x7f02c019f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c0101ec0 0x7f02c019f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 -- 172.21.15.31:0/708630937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b00095f0 con 0x7f02c0100340 2023-07-29T16:27:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:25.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.090+0000 7f02c559b700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0100340 0x7f02c01982c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00eee0 tx=0x7f02bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:25.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.091+0000 7f02b67fc700 1 -- 172.21.15.31:0/708630937 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc018070 con 0x7f02c0100340 2023-07-29T16:27:25.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.091+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c01a0390 con 0x7f02c0100340 2023-07-29T16:27:25.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.091+0000 7f02b67fc700 1 -- 172.21.15.31:0/708630937 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02bc00f040 con 0x7f02c0100340 2023-07-29T16:27:25.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.091+0000 7f02b67fc700 1 -- 172.21.15.31:0/708630937 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc004280 con 0x7f02c0100340 2023-07-29T16:27:25.094 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.091+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c01a08f0 con 0x7f02c0100340 2023-07-29T16:27:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.094+0000 7f02b67fc700 1 -- 172.21.15.31:0/708630937 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f02bc003a00 con 0x7f02c0100340 2023-07-29T16:27:25.095 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 Processor -- start 2023-07-29T16:27:25.096 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 -- start start 2023-07-29T16:27:25.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02b67fc700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0810 0x7f02ac0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02b67fc700 1 -- 172.21.15.31:0/708630937 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02bc0cb9a0 con 0x7f02c0100340 2023-07-29T16:27:25.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f2be0 0x562fb69f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb69f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69f4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.097 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb694aa10 con 0x562fb69f3530 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb69483d0 con 0x562fb69f2be0 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.096+0000 7f02db097b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb6943cc0 con 0x562fb69f42f0 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c5d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c5d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69f4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52704/0 (socket says 172.21.15.31:52704) 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c5d9c700 1 -- 172.21.15.31:0/935564701 learned_addr learned my addr 172.21.15.31:0/935564701 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f2be0 0x562fb69f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.098 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c5d9c700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0810 0x7f02ac0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb69f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 -- 172.21.15.31:0/935564701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 msgr2=0x562fb69f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 -- 172.21.15.31:0/935564701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 msgr2=0x562fb69f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb69f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 -- 172.21.15.31:0/935564701 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fb6771de0 con 0x562fb69f2be0 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb69f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c559b700 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f2be0 0x562fb69f2ff0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0c93e0 tx=0x7f02bc0092e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f5cdc3d958bc5bf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02ab7fe700 1 -- 172.21.15.31:0/935564701 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc009370 con 0x562fb69f2be0 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02c5d9c700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0810 0x7f02ac0a2cc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009fd0 tx=0x7f02b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02ab7fe700 1 -- 172.21.15.31:0/935564701 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02bc0990d0 con 0x562fb69f2be0 2023-07-29T16:27:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.097+0000 7f02ab7fe700 1 -- 172.21.15.31:0/935564701 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc099590 con 0x562fb69f2be0 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.098+0000 7f02db097b80 1 -- 172.21.15.31:0/935564701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f2be0 msgr2=0x562fb69f2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.098+0000 7f02db097b80 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f2be0 0x562fb69f2ff0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0c93e0 tx=0x7f02bc0092e0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.098+0000 7f02db097b80 1 -- 172.21.15.31:0/935564701 shutdown_connections 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.098+0000 7f02db097b80 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.098+0000 7f02db097b80 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f2be0 0x562fb69f2ff0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.099+0000 7f02db097b80 1 --2- 172.21.15.31:0/935564701 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb69f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.099+0000 7f02db097b80 1 -- 172.21.15.31:0/935564701 >> 172.21.15.31:0/935564701 conn(0x562fb69e83e0 msgr2=0x562fb69ed990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:25.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.099+0000 7f02db097b80 1 -- 172.21.15.31:0/935564701 shutdown_connections 2023-07-29T16:27:25.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.099+0000 7f02db097b80 1 -- 172.21.15.31:0/935564701 wait complete. 2023-07-29T16:27:25.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.099+0000 7f02db097b80 1 -- start start 2023-07-29T16:27:25.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 Processor -- start 2023-07-29T16:27:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69fc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb69fcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb6a03dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb68e0510 con 0x562fb69f2be0 2023-07-29T16:27:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb68e01b0 con 0x562fb69f3530 2023-07-29T16:27:25.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb67c7920 con 0x562fb69f42f0 2023-07-29T16:27:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.100+0000 7f02db097b80 10 client.0 _ll_register_callbacks cb 0x562fb68e9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb69fcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c559b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c559b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69fc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36560/0 (socket says 172.21.15.31:36560) 2023-07-29T16:27:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c559b700 1 -- 172.21.15.31:0/1765648787 learned_addr learned my addr 172.21.15.31:0/1765648787 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:25.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c5d9c700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb6a03dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 -- 172.21.15.31:0/1765648787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 msgr2=0x562fb6a03dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb6a03dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 -- 172.21.15.31:0/1765648787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 msgr2=0x562fb69fc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69fc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 -- 172.21.15.31:0/1765648787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c01a08f0 con 0x562fb69f3530 2023-07-29T16:27:25.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c559b700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb69fcd80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x562fb69f3940 tx=0x7f02b0005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c07dcbd80413ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.101+0000 7f02a8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1765648787 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0004910 con 0x562fb69f3530 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1765648787 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02b00056c0 con 0x562fb69f3530 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562fb696ed10 con 0x562fb69f3530 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1765648787 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02b0004a70 con 0x562fb69f3530 2023-07-29T16:27:25.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02db097b80 10 client.29102 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x562fb675ac60 con 0x562fb69f3530 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1765648787 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b0031020 con 0x562fb69f3530 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.102+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1765648787 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f02b000fbc0 con 0x562fb69f3530 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02db097b80 10 client.29102 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02db097b80 10 client.29102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02db097b80 20 client.29102 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:25.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02db097b80 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a08500 0x562fb6a0a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02db097b80 4 client.29102 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562fb69c1d70 con 0x562fb6a08500 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02c559b700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a08500 0x562fb6a0a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02c559b700 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a08500 0x562fb6a0a910 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a6e47f292b44d8c server_cookie=de3cac49c9b71fd7 in_seq=0 out_seq=0 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.103+0000 7f02a8ff9700 10 client.29102 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.104+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1765648787 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562fb69c1d70 con 0x562fb6a08500 2023-07-29T16:27:25.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.104+0000 7f02a8ff9700 10 client.29102 handle_command_reply: tid=0 2023-07-29T16:27:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.104+0000 7f02db097b80 1 client.29102 shutdown 2023-07-29T16:27:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.104+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 msgr2=0x562fb69fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.104+0000 7f02db097b80 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb69fcd80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x562fb69f3940 tx=0x7f02b0005270 comp rx=0 tx=0).stop 2023-07-29T16:27:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.104+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 shutdown_connections 2023-07-29T16:27:25.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a08500 0x562fb6a0a910 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb6a03dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 --2- 172.21.15.31:0/1765648787 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb69fcd80 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02a8ff9700 0 client.29102 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:25.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 >> 172.21.15.31:0/1765648787 conn(0x562fb69e83e0 msgr2=0x562fb69ed060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:25.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 shutdown_connections 2023-07-29T16:27:25.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 1 -- 172.21.15.31:0/1765648787 wait complete. 2023-07-29T16:27:25.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.105+0000 7f02db097b80 20 client.29102 trim_cache size 0 max 16384 2023-07-29T16:27:25.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 Processor -- start 2023-07-29T16:27:25.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 -- start start 2023-07-29T16:27:25.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb6a03d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69fa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb69f3bc0 con 0x562fb69f2be0 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb69ea460 con 0x562fb69f3530 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.123+0000 7f02db097b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb69fd4c0 con 0x562fb69f42f0 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb6a03d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c5d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69fa760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c559b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb6a03d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39236/0 (socket says 172.21.15.31:39236) 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 -- 172.21.15.31:0/3218281376 learned_addr learned my addr 172.21.15.31:0/3218281376 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 -- 172.21.15.31:0/3218281376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 msgr2=0x562fb69fa760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69fa760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 -- 172.21.15.31:0/3218281376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 msgr2=0x562fb69f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69f46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 -- 172.21.15.31:0/3218281376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fb6798250 con 0x562fb69f3530 2023-07-29T16:27:25.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c5d9c700 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69fa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:25.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.124+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb6a03d80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f02b000c010 tx=0x7f02b0005cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15f86deffa11ef90 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:25.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.125+0000 7f02abfff700 1 -- 172.21.15.31:0/3218281376 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0023620 con 0x562fb69f3530 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.125+0000 7f02abfff700 1 -- 172.21.15.31:0/3218281376 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b0031030 con 0x562fb69f3530 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.125+0000 7f02abfff700 1 -- 172.21.15.31:0/3218281376 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b000ea50 con 0x562fb69f3530 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 -- 172.21.15.31:0/3218281376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 msgr2=0x562fb6a03d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb6a03d80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f02b000c010 tx=0x7f02b0005cc0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 -- 172.21.15.31:0/3218281376 shutdown_connections 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f42f0 0x562fb69fa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f2be0 0x562fb69f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 --2- 172.21.15.31:0/3218281376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f3530 0x562fb6a03d80 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 -- 172.21.15.31:0/3218281376 >> 172.21.15.31:0/3218281376 conn(0x562fb69e83e0 msgr2=0x562fb69fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 -- 172.21.15.31:0/3218281376 shutdown_connections 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 -- 172.21.15.31:0/3218281376 wait complete. 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 -- start start 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.126+0000 7f02db097b80 1 Processor -- start 2023-07-29T16:27:25.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 0x562fb6a01a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb6a01f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f42f0 0x562fb6a02f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb6a034d0 con 0x562fb69f3530 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb6a03640 con 0x562fb69f42f0 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb6a03800 con 0x562fb69f2be0 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02db097b80 10 client.0 _ll_register_callbacks cb 0x562fb68e9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:25.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb6a01f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02c559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 0x562fb6a01a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02c559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 0x562fb6a01a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:52734/0 (socket says 172.21.15.31:52734) 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02c4d9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb6a01f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36586/0 (socket says 172.21.15.31:36586) 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02c559b700 1 -- 172.21.15.31:0/1487508339 learned_addr learned my addr 172.21.15.31:0/1487508339 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.127+0000 7f02c5d9c700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f42f0 0x562fb6a02f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c4d9a700 1 -- 172.21.15.31:0/1487508339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 msgr2=0x562fb6a01a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 0x562fb6a01a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c4d9a700 1 -- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f42f0 msgr2=0x562fb6a02f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:25.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f42f0 0x562fb6a02f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c4d9a700 1 -- 172.21.15.31:0/1487508339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b8019040 con 0x562fb69f3530 2023-07-29T16:27:25.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c559b700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 0x562fb6a01a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:25.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02c4d9a700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb6a01f80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f02b0038040 tx=0x7f02b000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba674322b9371d83 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02a8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1487508339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0004910 con 0x562fb69f3530 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f02bc0990d0 con 0x562fb69f3530 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.128+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1487508339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f02b0000e40 con 0x562fb69f3530 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1487508339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02b000ee80 con 0x562fb69f3530 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 10 client.19008 fetch_fsmap learned FSMap version 32 2023-07-29T16:27:25.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x562fb675ac60 con 0x562fb69f3530 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1487508339 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02b004c020 con 0x562fb69f3530 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1487508339 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 32) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f02b0034070 con 0x562fb69f3530 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 10 client.19008 fetch_fsmap finished waiting for FSMap version 32 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 10 client.19008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 20 client.19008 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a5ab30 0x562fb6a5cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:25.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 4 client.19008 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:27:25.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.129+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562fb693cd40 con 0x562fb6a5ab30 2023-07-29T16:27:25.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02c559b700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a5ab30 0x562fb6a5cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:25.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02c559b700 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a5ab30 0x562fb6a5cf40 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec9244a43b6b8834 server_cookie=59988b09753f6fbf in_seq=0 out_seq=0 2023-07-29T16:27:25.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02a8ff9700 10 client.19008 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:25.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02a8ff9700 1 -- 172.21.15.31:0/1487508339 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562fb693cd40 con 0x562fb6a5ab30 2023-07-29T16:27:25.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02a8ff9700 10 client.19008 handle_command_reply: tid=0 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02db097b80 1 client.19008 shutdown 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 msgr2=0x562fb6a01f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.130+0000 7f02db097b80 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb6a01f80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f02b0038040 tx=0x7f02b000ed10 comp rx=0 tx=0).stop 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 shutdown_connections 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562fb6a5ab30 0x562fb6a5cf40 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562fb69f42f0 0x562fb6a02f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562fb69f2be0 0x562fb6a01a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 --2- 172.21.15.31:0/1487508339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562fb69f3530 0x562fb6a01f80 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 >> 172.21.15.31:0/1487508339 conn(0x562fb69e83e0 msgr2=0x562fb69fa760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02a8ff9700 0 client.19008 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 shutdown_connections 2023-07-29T16:27:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 1 -- 172.21.15.31:0/1487508339 wait complete. 2023-07-29T16:27:25.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02db097b80 20 client.19008 trim_cache size 0 max 16384 2023-07-29T16:27:25.141 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:27:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:27:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:27:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "a3307c46-c7e9-4d74-9293-bdc634cffedd": { 2023-07-29T16:27:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:27:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "a3307c46-c7e9-4d74-9293-bdc634cffedd", 2023-07-29T16:27:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:27:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:27:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:27:25.142 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:27:25.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0810 msgr2=0x7f02ac0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02c77ff700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0810 0x7f02ac0a2cc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009fd0 tx=0x7f02b8009260 comp rx=0 tx=0).stop 2023-07-29T16:27:25.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0100340 msgr2=0x7f02c01982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:25.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02c77ff700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0100340 0x7f02c01982c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00eee0 tx=0x7f02bc00c3a0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.131+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 shutdown_connections 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f02ac0a0810 0x7f02ac0a2cc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f02c0101ec0 0x7f02c019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c0100340 0x7f02c01982c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 --2- 172.21.15.31:0/708630937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c0101530 0x7f02c0198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 >> 172.21.15.31:0/708630937 conn(0x7f02c00fb960 msgr2=0x7f02c0106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 shutdown_connections 2023-07-29T16:27:25.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:25.132+0000 7f02c77ff700 1 -- 172.21.15.31:0/708630937 wait complete. 2023-07-29T16:27:25.154 INFO:tasks.cephfs.filesystem:scrub status for tag:a3307c46-c7e9-4d74-9293-bdc634cffedd - {'path': '/', 'tag': 'a3307c46-c7e9-4d74-9293-bdc634cffedd', 'options': 'recursive,force'} 2023-07-29T16:27:25.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 912346 8.93 MB/sec execute 430 sec latency 21.919 ms 2023-07-29T16:27:26.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:27:26.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:26.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:26 smithi156 ceph-mon[129245]: pgmap v730: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.3 MiB/s wr, 260 op/s 2023-07-29T16:27:26.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:27:26.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:26.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:26.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:26.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:26.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.685 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:26 smithi031 ceph-mon[95504]: pgmap v730: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.3 MiB/s wr, 260 op/s 2023-07-29T16:27:26.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:26.777 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:26 smithi039 ceph-mon[128858]: pgmap v730: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.3 MiB/s wr, 260 op/s 2023-07-29T16:27:26.893 INFO:tasks.workunit.client.0.smithi031.stdout: 1 913507 8.91 MB/sec execute 431 sec latency 11.714 ms 2023-07-29T16:27:27.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 915872 8.93 MB/sec execute 432 sec latency 20.358 ms 2023-07-29T16:27:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:27 smithi039 ceph-mon[128858]: pgmap v731: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 2.0 MiB/s wr, 241 op/s 2023-07-29T16:27:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:27 smithi156 ceph-mon[129245]: pgmap v731: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 2.0 MiB/s wr, 241 op/s 2023-07-29T16:27:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:27 smithi031 ceph-mon[95504]: pgmap v731: 97 pgs: 97 active+clean; 168 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 2.0 MiB/s wr, 241 op/s 2023-07-29T16:27:28.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 916689 8.91 MB/sec execute 433 sec latency 40.869 ms 2023-07-29T16:27:29.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 918200 8.95 MB/sec execute 434 sec latency 71.740 ms 2023-07-29T16:27:30.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:30 smithi156 ceph-mon[129245]: pgmap v732: 97 pgs: 97 active+clean; 171 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 2.8 MiB/s wr, 313 op/s 2023-07-29T16:27:30.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 920384 8.96 MB/sec execute 435 sec latency 12.398 ms 2023-07-29T16:27:30.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:30 smithi031 ceph-mon[95504]: pgmap v732: 97 pgs: 97 active+clean; 171 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 2.8 MiB/s wr, 313 op/s 2023-07-29T16:27:31.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:30 smithi039 ceph-mon[128858]: pgmap v732: 97 pgs: 97 active+clean; 171 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 2.8 MiB/s wr, 313 op/s 2023-07-29T16:27:31.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 921695 8.96 MB/sec execute 436 sec latency 13.866 ms 2023-07-29T16:27:32.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:32 smithi156 ceph-mon[129245]: pgmap v733: 97 pgs: 97 active+clean; 171 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.0 MiB/s wr, 234 op/s 2023-07-29T16:27:32.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 923929 8.96 MB/sec execute 437 sec latency 14.432 ms 2023-07-29T16:27:32.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:32 smithi031 ceph-mon[95504]: pgmap v733: 97 pgs: 97 active+clean; 171 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.0 MiB/s wr, 234 op/s 2023-07-29T16:27:33.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:32 smithi039 ceph-mon[128858]: pgmap v733: 97 pgs: 97 active+clean; 171 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.0 MiB/s wr, 234 op/s 2023-07-29T16:27:33.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:33 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:27:33.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:33 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:27:33.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:33 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:27:33.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:33 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:27:33.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 924858 8.95 MB/sec execute 438 sec latency 81.429 ms 2023-07-29T16:27:33.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:33 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:27:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:33 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:27:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:33 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:27:33.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:33 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:27:34.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:33 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:27:34.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:33 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:27:34.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:33 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:27:34.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:33 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:27:34.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:34 smithi156 ceph-mon[129245]: pgmap v734: 97 pgs: 97 active+clean; 164 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.4 MiB/s wr, 261 op/s 2023-07-29T16:27:34.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 926835 8.96 MB/sec execute 439 sec latency 47.462 ms 2023-07-29T16:27:34.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:34 smithi031 ceph-mon[95504]: pgmap v734: 97 pgs: 97 active+clean; 164 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.4 MiB/s wr, 261 op/s 2023-07-29T16:27:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 -- 172.21.15.31:0/1928427543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8108210 msgr2=0x7f36a810a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 --2- 172.21.15.31:0/1928427543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8108210 0x7f36a810a690 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f36a000b0a0 tx=0x7f36a000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 -- 172.21.15.31:0/1928427543 shutdown_connections 2023-07-29T16:27:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 --2- 172.21.15.31:0/1928427543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8108210 0x7f36a810a690 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 --2- 172.21.15.31:0/1928427543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a8074d90 0x7f36a80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 --2- 172.21.15.31:0/1928427543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a80737b0 0x7f36a8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 -- 172.21.15.31:0/1928427543 >> 172.21.15.31:0/1928427543 conn(0x7f36a80fb970 msgr2=0x7f36a80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:34.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 -- 172.21.15.31:0/1928427543 shutdown_connections 2023-07-29T16:27:34.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.954+0000 7f36b03ff700 1 -- 172.21.15.31:0/1928427543 wait complete. 2023-07-29T16:27:34.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.955+0000 7f36b03ff700 1 Processor -- start 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.955+0000 7f36b03ff700 1 -- start start 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36b03ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a80737b0 0x7f36a81984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36b03ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8074d90 0x7f36a8198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36b03ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a8108210 0x7f36a81998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36b03ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a8199f00 con 0x7f36a80737b0 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36b03ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a819a070 con 0x7f36a8108210 2023-07-29T16:27:34.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36b03ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a819a230 con 0x7f36a8074d90 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8074d90 0x7f36a8198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8074d90 0x7f36a8198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54444/0 (socket says 172.21.15.31:54444) 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ad99a700 1 -- 172.21.15.31:0/2963133990 learned_addr learned my addr 172.21.15.31:0/2963133990 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae99c700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a8108210 0x7f36a81998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae19b700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a80737b0 0x7f36a81984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae99c700 1 -- 172.21.15.31:0/2963133990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8074d90 msgr2=0x7f36a8198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae99c700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8074d90 0x7f36a8198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae99c700 1 -- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a80737b0 msgr2=0x7f36a81984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae99c700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a80737b0 0x7f36a81984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.956+0000 7f36ae99c700 1 -- 172.21.15.31:0/2963133990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0009d00 con 0x7f36a8108210 2023-07-29T16:27:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.957+0000 7f36ae99c700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a8108210 0x7f36a81998a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009fd0 tx=0x7f36a0007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:34.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.957+0000 7f369f7fe700 1 -- 172.21.15.31:0/2963133990 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a000e040 con 0x7f36a8108210 2023-07-29T16:27:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.957+0000 7f369f7fe700 1 -- 172.21.15.31:0/2963133990 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36a0004c50 con 0x7f36a8108210 2023-07-29T16:27:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.957+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a819a4b0 con 0x7f36a8108210 2023-07-29T16:27:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.957+0000 7f369f7fe700 1 -- 172.21.15.31:0/2963133990 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a001b6c0 con 0x7f36a8108210 2023-07-29T16:27:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.957+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a81a0cd0 con 0x7f36a8108210 2023-07-29T16:27:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.961+0000 7f369f7fe700 1 -- 172.21.15.31:0/2963133990 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f36a001d470 con 0x7f36a8108210 2023-07-29T16:27:34.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.962+0000 7f36c3c79b80 1 Processor -- start 2023-07-29T16:27:34.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 -- start start 2023-07-29T16:27:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea031f9b0 con 0x555ea03c7df0 2023-07-29T16:27:34.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea031d370 con 0x555ea03c8cf0 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea0318c60 con 0x555ea02236b0 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03c78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54450/0 (socket says 172.21.15.31:54450) 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f36ad99a700 1 -- 172.21.15.31:0/2726948875 learned_addr learned my addr 172.21.15.31:0/2726948875 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.963+0000 7f369f7fe700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36940a0790 0x7f36940a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ae19b700 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f369f7fe700 1 -- 172.21.15.31:0/2963133990 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f36a00ccac0 con 0x7f36a8108210 2023-07-29T16:27:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 -- 172.21.15.31:0/2726948875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 msgr2=0x555ea03c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 -- 172.21.15.31:0/2726948875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 msgr2=0x555ea03c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03c87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 -- 172.21.15.31:0/2726948875 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ea0146d80 con 0x555ea02236b0 2023-07-29T16:27:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ae99c700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36940a0790 0x7f36940a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03c78b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x555ea03c7610 tx=0x7f3698009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5494f5f82cee5f0b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f369cbfe700 1 -- 172.21.15.31:0/2726948875 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36980131b0 con 0x555ea02236b0 2023-07-29T16:27:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.964+0000 7f369cbfe700 1 -- 172.21.15.31:0/2726948875 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3698009e80 con 0x555ea02236b0 2023-07-29T16:27:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f369cbfe700 1 -- 172.21.15.31:0/2726948875 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698003e90 con 0x555ea02236b0 2023-07-29T16:27:34.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36ae99c700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36940a0790 0x7f36940a2c40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f36a0096cc0 tx=0x7f36a000b3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2726948875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 msgr2=0x555ea03c78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03c78b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x555ea03c7610 tx=0x7f3698009600 comp rx=0 tx=0).stop 2023-07-29T16:27:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2726948875 shutdown_connections 2023-07-29T16:27:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03c78b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2726948875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.965+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2726948875 >> 172.21.15.31:0/2726948875 conn(0x555ea03bcfd0 msgr2=0x555ea03c2300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.966+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2726948875 shutdown_connections 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.966+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2726948875 wait complete. 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.966+0000 7f36c3c79b80 1 -- start start 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.966+0000 7f36c3c79b80 1 Processor -- start 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 0x555ea03d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea03c7df0 0x555ea03d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea03d84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea02b54b0 con 0x555ea03c8cf0 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea02b5150 con 0x555ea02236b0 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea019c8c0 con 0x555ea03c7df0 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36c3c79b80 10 client.0 _ll_register_callbacks cb 0x555ea02be5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 0x555ea03d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36ae19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea03c7df0 0x555ea03d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 0x555ea03d0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40242/0 (socket says 172.21.15.31:40242) 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.967+0000 7f36ad99a700 1 -- 172.21.15.31:0/556017482 learned_addr learned my addr 172.21.15.31:0/556017482 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ad99a700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea03d84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ae19b700 1 -- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 msgr2=0x555ea03d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ae19b700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 0x555ea03d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ae19b700 1 -- 172.21.15.31:0/556017482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 msgr2=0x555ea03d84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ae19b700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea03d84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ae19b700 1 -- 172.21.15.31:0/556017482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369801b040 con 0x555ea03c7df0 2023-07-29T16:27:34.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ad99a700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 0x555ea03d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36ae19b700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea03c7df0 0x555ea03d1490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f36a4012740 tx=0x7f36a4012b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2adf62a0b40b532b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f3691ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:34.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f3691ffb700 1 -- 172.21.15.31:0/556017482 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4017070 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.968+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ea0396d10 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f3691ffb700 1 -- 172.21.15.31:0/556017482 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f36a400f5c0 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f3691ffb700 1 -- 172.21.15.31:0/556017482 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36a400f780 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 10 client.29119 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555ea012fc00 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f3691ffb700 1 -- 172.21.15.31:0/556017482 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555ea0396d10 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f3691ffb700 1 -- 172.21.15.31:0/556017482 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f36a40269b0 con 0x555ea03c7df0 2023-07-29T16:27:34.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 10 client.29119 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 10 client.29119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 20 client.29119 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea03dcd20 0x555ea03df130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 4 client.29119 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.969+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ea0311ce0 con 0x555ea03dcd20 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.970+0000 7f36ad99a700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea03dcd20 0x555ea03df130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.970+0000 7f36ad99a700 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea03dcd20 0x555ea03df130 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd27908e2c6c2808 server_cookie=e8a5101536f525d7 in_seq=0 out_seq=0 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.970+0000 7f3691ffb700 10 client.29119 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f3691ffb700 1 -- 172.21.15.31:0/556017482 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555ea0311ce0 con 0x555ea03dcd20 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f3691ffb700 10 client.29119 handle_command_reply: tid=0 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 client.29119 shutdown 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea03c7df0 msgr2=0x555ea03d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea03c7df0 0x555ea03d1490 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f36a4012740 tx=0x7f36a4012b00 comp rx=0 tx=0).stop 2023-07-29T16:27:34.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 shutdown_connections 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea03dcd20 0x555ea03df130 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea03d84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea02236b0 0x555ea03d0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/556017482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea03c7df0 0x555ea03d1490 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 >> 172.21.15.31:0/556017482 conn(0x555ea03bcfd0 msgr2=0x555ea03c1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f3691ffb700 0 client.29119 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 shutdown_connections 2023-07-29T16:27:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.971+0000 7f36c3c79b80 1 -- 172.21.15.31:0/556017482 wait complete. 2023-07-29T16:27:34.978 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.972+0000 7f36c3c79b80 20 client.29119 trim_cache size 0 max 16384 2023-07-29T16:27:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.988+0000 7f36c3c79b80 1 Processor -- start 2023-07-29T16:27:34.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.988+0000 7f36c3c79b80 1 -- start start 2023-07-29T16:27:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.988+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03d63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.988+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03ded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea03c81c0 con 0x555ea03c7df0 2023-07-29T16:27:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea03d2140 con 0x555ea03c8cf0 2023-07-29T16:27:34.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea03cf820 con 0x555ea02236b0 2023-07-29T16:27:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ae19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03d63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03d63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54468/0 (socket says 172.21.15.31:54468) 2023-07-29T16:27:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ae19b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03ded30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37356/0 (socket says 172.21.15.31:37356) 2023-07-29T16:27:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 -- 172.21.15.31:0/164557091 learned_addr learned my addr 172.21.15.31:0/164557091 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 -- 172.21.15.31:0/164557091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 msgr2=0x555ea03cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 -- 172.21.15.31:0/164557091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 msgr2=0x555ea03ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03ded30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.989+0000 7f36ad99a700 1 -- 172.21.15.31:0/164557091 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ea03c9390 con 0x555ea02236b0 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.990+0000 7f36ad99a700 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03d63e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3698001eb0 tx=0x7f3698022040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ede80db104194e55 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.990+0000 7f369d3ff700 1 -- 172.21.15.31:0/164557091 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698026040 con 0x555ea02236b0 2023-07-29T16:27:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.990+0000 7f369d3ff700 1 -- 172.21.15.31:0/164557091 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3698003df0 con 0x555ea02236b0 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.990+0000 7f369d3ff700 1 -- 172.21.15.31:0/164557091 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3698005270 con 0x555ea02236b0 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 -- 172.21.15.31:0/164557091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 msgr2=0x555ea03d63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03d63e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3698001eb0 tx=0x7f3698022040 comp rx=0 tx=0).stop 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 -- 172.21.15.31:0/164557091 shutdown_connections 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c8cf0 0x555ea03cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea03d63e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/164557091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c7df0 0x555ea03ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 -- 172.21.15.31:0/164557091 >> 172.21.15.31:0/164557091 conn(0x555ea03bcfd0 msgr2=0x555ea03cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 -- 172.21.15.31:0/164557091 shutdown_connections 2023-07-29T16:27:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 -- 172.21.15.31:0/164557091 wait complete. 2023-07-29T16:27:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 -- start start 2023-07-29T16:27:34.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.991+0000 7f36c3c79b80 1 Processor -- start 2023-07-29T16:27:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea0428870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 0x555ea042adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea042d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea03d67e0 con 0x555ea03c8cf0 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea042d880 con 0x555ea03c7df0 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ea042da40 con 0x555ea02236b0 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36c3c79b80 10 client.0 _ll_register_callbacks cb 0x555ea02be5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea0428870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36ae19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 0x555ea042adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36ad99a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea0428870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54478/0 (socket says 172.21.15.31:54478) 2023-07-29T16:27:34.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36ad99a700 1 -- 172.21.15.31:0/2691487994 learned_addr learned my addr 172.21.15.31:0/2691487994 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36ae19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 0x555ea042adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40254/0 (socket says 172.21.15.31:40254) 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.992+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea042d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ae19b700 1 -- 172.21.15.31:0/2691487994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 msgr2=0x555ea0428870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ae19b700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea0428870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ae19b700 1 -- 172.21.15.31:0/2691487994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 msgr2=0x555ea042d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ae19b700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea042d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:34.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ae19b700 1 -- 172.21.15.31:0/2691487994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3698017040 con 0x555ea03c7df0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea0428870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36ae19b700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 0x555ea042adc0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f36a4000c00 tx=0x7f36a4009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b048926817ea595 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f3691ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f3691ffb700 1 -- 172.21.15.31:0/2691487994 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a4017040 con 0x555ea03c7df0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ea0343cb0 con 0x555ea03c7df0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.993+0000 7f3691ffb700 1 -- 172.21.15.31:0/2691487994 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f36a402b070 con 0x555ea03c7df0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f3691ffb700 1 -- 172.21.15.31:0/2691487994 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36a40054d0 con 0x555ea03c7df0 2023-07-29T16:27:34.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 10 client.29126 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x555ea016d1f0 con 0x555ea03c7df0 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f3691ffb700 1 -- 172.21.15.31:0/2691487994 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555ea0343cb0 con 0x555ea03c7df0 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f3691ffb700 1 -- 172.21.15.31:0/2691487994 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f36a4010620 con 0x555ea03c7df0 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 10 client.29126 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 10 client.29126 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 20 client.29126 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea042f3e0 0x555ea04317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:34.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 4 client.29126 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:27:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.994+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555ea0311a00 con 0x555ea042f3e0 2023-07-29T16:27:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.995+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea042f3e0 0x555ea04317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.995+0000 7f36ad99a700 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea042f3e0 0x555ea04317f0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3556a0163ef68404 server_cookie=bd7796ab76ae1f9c in_seq=0 out_seq=0 2023-07-29T16:27:35.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:34.995+0000 7f3691ffb700 10 client.29126 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:35.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:34 smithi039 ceph-mon[128858]: pgmap v734: 97 pgs: 97 active+clean; 164 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 2.4 MiB/s wr, 261 op/s 2023-07-29T16:27:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.073+0000 7f3691ffb700 1 -- 172.21.15.31:0/2691487994 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555ea0311a00 con 0x555ea042f3e0 2023-07-29T16:27:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.073+0000 7f3691ffb700 10 client.29126 handle_command_reply: tid=0 2023-07-29T16:27:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 client.29126 shutdown 2023-07-29T16:27:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 msgr2=0x555ea042adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:35.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 0x555ea042adc0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f36a4000c00 tx=0x7f36a4009ec0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 shutdown_connections 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555ea042f3e0 0x555ea04317f0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555ea03c8cf0 0x555ea042d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ea02236b0 0x555ea0428870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f3691ffb700 0 client.29126 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 --2- 172.21.15.31:0/2691487994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ea03c7df0 0x555ea042adc0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 >> 172.21.15.31:0/2691487994 conn(0x555ea03bcfd0 msgr2=0x555ea03ded30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 shutdown_connections 2023-07-29T16:27:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 1 -- 172.21.15.31:0/2691487994 wait complete. 2023-07-29T16:27:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.074+0000 7f36c3c79b80 20 client.29126 trim_cache size 0 max 16384 2023-07-29T16:27:35.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:27:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:27:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:27:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:27:35.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36940a0790 msgr2=0x7f36940a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36940a0790 0x7f36940a2c40 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f36a0096cc0 tx=0x7f36a000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a8108210 msgr2=0x7f36a81998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a8108210 0x7f36a81998a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f36a0009fd0 tx=0x7f36a0007930 comp rx=0 tx=0).stop 2023-07-29T16:27:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 shutdown_connections 2023-07-29T16:27:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f36940a0790 0x7f36940a2c40 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f36a8108210 0x7f36a81998a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36a80737b0 0x7f36a81984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 --2- 172.21.15.31:0/2963133990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36a8074d90 0x7f36a8198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 >> 172.21.15.31:0/2963133990 conn(0x7f36a80fb970 msgr2=0x7f36a8104110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 shutdown_connections 2023-07-29T16:27:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:35.075+0000 7f36b03ff700 1 -- 172.21.15.31:0/2963133990 wait complete. 2023-07-29T16:27:35.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 927939 8.95 MB/sec execute 440 sec latency 68.835 ms 2023-07-29T16:27:36.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:36 smithi156 ceph-mon[129245]: pgmap v735: 97 pgs: 97 active+clean; 160 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:27:36.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 930114 8.95 MB/sec execute 441 sec latency 20.990 ms 2023-07-29T16:27:36.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:36 smithi031 ceph-mon[95504]: pgmap v735: 97 pgs: 97 active+clean; 160 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:27:37.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:36 smithi039 ceph-mon[128858]: pgmap v735: 97 pgs: 97 active+clean; 160 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 2.6 MiB/s wr, 291 op/s 2023-07-29T16:27:37.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 931439 8.95 MB/sec execute 442 sec latency 12.219 ms 2023-07-29T16:27:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:37 smithi039 ceph-mon[128858]: pgmap v736: 97 pgs: 97 active+clean; 160 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 2.2 MiB/s wr, 231 op/s 2023-07-29T16:27:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:37 smithi156 ceph-mon[129245]: pgmap v736: 97 pgs: 97 active+clean; 160 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 2.2 MiB/s wr, 231 op/s 2023-07-29T16:27:38.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:37 smithi031 ceph-mon[95504]: pgmap v736: 97 pgs: 97 active+clean; 160 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 2.2 MiB/s wr, 231 op/s 2023-07-29T16:27:38.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 933373 8.95 MB/sec execute 443 sec latency 32.406 ms 2023-07-29T16:27:39.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 934819 8.96 MB/sec execute 444 sec latency 59.982 ms 2023-07-29T16:27:40.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:40 smithi156 ceph-mon[129245]: pgmap v737: 97 pgs: 97 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 3.1 MiB/s wr, 328 op/s 2023-07-29T16:27:40.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 936806 8.95 MB/sec execute 445 sec latency 16.425 ms 2023-07-29T16:27:40.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:40 smithi031 ceph-mon[95504]: pgmap v737: 97 pgs: 97 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 3.1 MiB/s wr, 328 op/s 2023-07-29T16:27:41.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:40 smithi039 ceph-mon[128858]: pgmap v737: 97 pgs: 97 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 3.1 MiB/s wr, 328 op/s 2023-07-29T16:27:41.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 938354 8.96 MB/sec execute 446 sec latency 14.205 ms 2023-07-29T16:27:42.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:42 smithi156 ceph-mon[129245]: pgmap v738: 97 pgs: 97 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 2.2 MiB/s wr, 240 op/s 2023-07-29T16:27:42.894 INFO:tasks.workunit.client.0.smithi031.stdout: 1 940075 8.95 MB/sec execute 447 sec latency 15.627 ms 2023-07-29T16:27:42.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:42 smithi031 ceph-mon[95504]: pgmap v738: 97 pgs: 97 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 2.2 MiB/s wr, 240 op/s 2023-07-29T16:27:43.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:42 smithi039 ceph-mon[128858]: pgmap v738: 97 pgs: 97 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 2.2 MiB/s wr, 240 op/s 2023-07-29T16:27:43.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 941835 8.96 MB/sec execute 448 sec latency 14.103 ms 2023-07-29T16:27:44.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:44 smithi156 ceph-mon[129245]: pgmap v739: 97 pgs: 97 active+clean; 155 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 2.6 MiB/s wr, 264 op/s 2023-07-29T16:27:44.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 943157 8.95 MB/sec execute 449 sec latency 36.195 ms 2023-07-29T16:27:44.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:44 smithi031 ceph-mon[95504]: pgmap v739: 97 pgs: 97 active+clean; 155 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 2.6 MiB/s wr, 264 op/s 2023-07-29T16:27:45.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:44 smithi039 ceph-mon[128858]: pgmap v739: 97 pgs: 97 active+clean; 155 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 2.6 MiB/s wr, 264 op/s 2023-07-29T16:27:45.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 945263 8.96 MB/sec execute 450 sec latency 17.038 ms 2023-07-29T16:27:46.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:46 smithi156 ceph-mon[129245]: pgmap v740: 97 pgs: 97 active+clean; 157 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:27:46.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 946593 8.95 MB/sec execute 451 sec latency 12.973 ms 2023-07-29T16:27:46.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:46 smithi031 ceph-mon[95504]: pgmap v740: 97 pgs: 97 active+clean; 157 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:27:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:46 smithi039 ceph-mon[128858]: pgmap v740: 97 pgs: 97 active+clean; 157 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2023-07-29T16:27:47.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 948740 8.96 MB/sec execute 452 sec latency 13.243 ms 2023-07-29T16:27:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 -- 172.21.15.31:0/3268047325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac103470 msgr2=0x7f1dac105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 --2- 172.21.15.31:0/3268047325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac103470 0x7f1dac105890 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f1da0009990 tx=0x7f1da0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 -- 172.21.15.31:0/3268047325 shutdown_connections 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 --2- 172.21.15.31:0/3268047325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac105f20 0x7f1dac1083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 --2- 172.21.15.31:0/3268047325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac068e20 0x7f1dac102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 --2- 172.21.15.31:0/3268047325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac103470 0x7f1dac105890 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 -- 172.21.15.31:0/3268047325 >> 172.21.15.31:0/3268047325 conn(0x7f1dac0fa840 msgr2=0x7f1dac0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 -- 172.21.15.31:0/3268047325 shutdown_connections 2023-07-29T16:27:48.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.230+0000 7f1db13ff700 1 -- 172.21.15.31:0/3268047325 wait complete. 2023-07-29T16:27:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.231+0000 7f1db13ff700 1 Processor -- start 2023-07-29T16:27:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 -- start start 2023-07-29T16:27:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 0x7f1dac1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac103470 0x7f1dac1966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac105f20 0x7f1dac197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac197bd0 con 0x7f1dac105f20 2023-07-29T16:27:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac197d40 con 0x7f1dac068e20 2023-07-29T16:27:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1db13ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dac197f00 con 0x7f1dac103470 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 0x7f1dac1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 0x7f1dac1961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51978/0 (socket says 172.21.15.31:51978) 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1daaffd700 1 -- 172.21.15.31:0/1858768027 learned_addr learned my addr 172.21.15.31:0/1858768027 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac103470 0x7f1dac1966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.232+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac105f20 0x7f1dac197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daaffd700 1 -- 172.21.15.31:0/1858768027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac103470 msgr2=0x7f1dac1966f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daaffd700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac103470 0x7f1dac1966f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daaffd700 1 -- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac105f20 msgr2=0x7f1dac197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daaffd700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac105f20 0x7f1dac197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daaffd700 1 -- 172.21.15.31:0/1858768027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da00095f0 con 0x7f1dac068e20 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac103470 0x7f1dac1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac105f20 0x7f1dac197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1daaffd700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 0x7f1dac1961b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e970 tx=0x7f1d9c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1d93fff700 1 -- 172.21.15.31:0/1858768027 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c009790 con 0x7f1dac068e20 2023-07-29T16:27:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1d93fff700 1 -- 172.21.15.31:0/1858768027 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d9c00eec0 con 0x7f1dac068e20 2023-07-29T16:27:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1d93fff700 1 -- 172.21.15.31:0/1858768027 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c005590 con 0x7f1dac068e20 2023-07-29T16:27:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dac1981e0 con 0x7f1dac068e20 2023-07-29T16:27:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.233+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac19e9a0 con 0x7f1dac068e20 2023-07-29T16:27:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.236+0000 7f1d93fff700 1 -- 172.21.15.31:0/1858768027 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1d9c00cbd0 con 0x7f1dac068e20 2023-07-29T16:27:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.238+0000 7f1dc4c86b80 1 Processor -- start 2023-07-29T16:27:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 -- start start 2023-07-29T16:27:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1d93fff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d940a0810 0x7f1d940a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 0x55f120dc88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dc97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc9cf0 0x55f120dca160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120d209b0 con 0x55f120dc8df0 2023-07-29T16:27:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120d1e370 con 0x55f120c246b0 2023-07-29T16:27:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.239+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120d19c60 con 0x55f120dc9cf0 2023-07-29T16:27:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 0x55f120dc88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1d93fff700 1 -- 172.21.15.31:0/1858768027 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1d9c014070 con 0x7f1dac068e20 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 0x55f120dc88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51986/0 (socket says 172.21.15.31:51986) 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 -- 172.21.15.31:0/1950355624 learned_addr learned my addr 172.21.15.31:0/1950355624 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dc97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d940a0810 0x7f1d940a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc9cf0 0x55f120dca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 -- 172.21.15.31:0/1950355624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc9cf0 msgr2=0x55f120dca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc9cf0 0x55f120dca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 -- 172.21.15.31:0/1950355624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 msgr2=0x55f120dc97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dc97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 -- 172.21.15.31:0/1950355624 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f120b47d80 con 0x55f120c246b0 2023-07-29T16:27:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dc97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:48.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1daaffd700 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 0x55f120dc88b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c095230 tx=0x7f1d9c0993b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=513b5096d563ea92 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d940a0810 0x7f1d940a2cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x55f120dc8610 tx=0x7f1da4009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.240+0000 7f1d917fa700 1 -- 172.21.15.31:0/1950355624 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c09d070 con 0x55f120c246b0 2023-07-29T16:27:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.241+0000 7f1d917fa700 1 -- 172.21.15.31:0/1950355624 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d9c0994b0 con 0x55f120c246b0 2023-07-29T16:27:48.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.241+0000 7f1d917fa700 1 -- 172.21.15.31:0/1950355624 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c0a1900 con 0x55f120c246b0 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.241+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/1950355624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 msgr2=0x55f120dc88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.241+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 0x55f120dc88b0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c095230 tx=0x7f1d9c0993b0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/1950355624 shutdown_connections 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc9cf0 0x55f120dca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120c246b0 0x55f120dc88b0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/1950355624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dc97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/1950355624 >> 172.21.15.31:0/1950355624 conn(0x55f120dbdfd0 msgr2=0x55f120dc3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/1950355624 shutdown_connections 2023-07-29T16:27:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/1950355624 wait complete. 2023-07-29T16:27:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 -- start start 2023-07-29T16:27:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.242+0000 7f1dc4c86b80 1 Processor -- start 2023-07-29T16:27:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120dd62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dd6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120cb64b0 con 0x55f120dc8df0 2023-07-29T16:27:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120cb6150 con 0x55f120dc9cf0 2023-07-29T16:27:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120b9d8c0 con 0x55f120c246b0 2023-07-29T16:27:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dc4c86b80 10 client.0 _ll_register_callbacks cb 0x55f120cbf5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:48.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120dd62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dab7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120dd62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40134/0 (socket says 172.21.15.31:40134) 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1daaffd700 1 -- 172.21.15.31:0/763981344 learned_addr learned my addr 172.21.15.31:0/763981344 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51996/0 (socket says 172.21.15.31:51996) 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dd6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1daaffd700 1 -- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 msgr2=0x55f120ddd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.243+0000 7f1daaffd700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1daaffd700 1 -- 172.21.15.31:0/763981344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 msgr2=0x55f120dd6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1daaffd700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dd6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1daaffd700 1 -- 172.21.15.31:0/763981344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac19e9a0 con 0x55f120c246b0 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1daaffd700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120dd62f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c095960 tx=0x7f1d9c0951c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5894c411be0894 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1d8eff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1d8eff5700 1 -- 172.21.15.31:0/763981344 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c0b3ea0 con 0x55f120c246b0 2023-07-29T16:27:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f120d44cb0 con 0x55f120c246b0 2023-07-29T16:27:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1d8eff5700 1 -- 172.21.15.31:0/763981344 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1d9c0a1900 con 0x55f120c246b0 2023-07-29T16:27:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1d8eff5700 1 -- 172.21.15.31:0/763981344 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d9c09d0c0 con 0x55f120c246b0 2023-07-29T16:27:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1dc4c86b80 10 client.29137 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.244+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f120b30c00 con 0x55f120c246b0 2023-07-29T16:27:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1d8eff5700 1 -- 172.21.15.31:0/763981344 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f120d44cb0 con 0x55f120c246b0 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1d8eff5700 1 -- 172.21.15.31:0/763981344 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f1d9c0b0020 con 0x55f120c246b0 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1dc4c86b80 10 client.29137 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1dc4c86b80 10 client.29137 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1dc4c86b80 20 client.29137 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120de1c60 0x55f120de4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1dc4c86b80 4 client.29137 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f120d97d10 con 0x55f120de1c60 2023-07-29T16:27:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.245+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120de1c60 0x55f120de4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.246+0000 7f1d8eff5700 10 client.29137 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.246+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120de1c60 0x55f120de4070 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7a3e7fa15195989 server_cookie=4dfa9913c6c690b8 in_seq=0 out_seq=0 2023-07-29T16:27:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.246+0000 7f1d8eff5700 1 -- 172.21.15.31:0/763981344 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f120d97d10 con 0x55f120de1c60 2023-07-29T16:27:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.246+0000 7f1d8eff5700 10 client.29137 handle_command_reply: tid=0 2023-07-29T16:27:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 client.29137 shutdown 2023-07-29T16:27:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 msgr2=0x55f120dd62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120dd62f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c095960 tx=0x7f1d9c0951c0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 shutdown_connections 2023-07-29T16:27:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120de1c60 0x55f120de4070 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120dd62f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/763981344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120dd6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1d8eff5700 0 client.29137 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 >> 172.21.15.31:0/763981344 conn(0x55f120dbdfd0 msgr2=0x55f120dc2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 shutdown_connections 2023-07-29T16:27:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/763981344 wait complete. 2023-07-29T16:27:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.247+0000 7f1dc4c86b80 20 client.29137 trim_cache size 0 max 16384 2023-07-29T16:27:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 Processor -- start 2023-07-29T16:27:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 -- start start 2023-07-29T16:27:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120de4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120ddd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120dca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120dc3ed0 con 0x55f120dc8df0 2023-07-29T16:27:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120dcfe40 con 0x55f120dc9cf0 2023-07-29T16:27:48.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120dcd730 con 0x55f120c246b0 2023-07-29T16:27:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120de4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1daa7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120ddd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1daaffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120de4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40144/0 (socket says 172.21.15.31:40144) 2023-07-29T16:27:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120dca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.263+0000 7f1daaffd700 1 -- 172.21.15.31:0/156559880 learned_addr learned my addr 172.21.15.31:0/156559880 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daaffd700 1 -- 172.21.15.31:0/156559880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 msgr2=0x55f120dca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daaffd700 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120dca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daaffd700 1 -- 172.21.15.31:0/156559880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 msgr2=0x55f120ddd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daaffd700 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120ddd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daaffd700 1 -- 172.21.15.31:0/156559880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f120b6e1f0 con 0x55f120c246b0 2023-07-29T16:27:48.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120ddd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:48.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1daaffd700 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120de4040 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c095230 tx=0x7f1d9c00cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e30c1be2d3da23 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:48.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1d91ffb700 1 -- 172.21.15.31:0/156559880 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c0a1900 con 0x55f120c246b0 2023-07-29T16:27:48.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1d91ffb700 1 -- 172.21.15.31:0/156559880 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d9c0b1e90 con 0x55f120c246b0 2023-07-29T16:27:48.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.264+0000 7f1d91ffb700 1 -- 172.21.15.31:0/156559880 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c09d310 con 0x55f120c246b0 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/156559880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 msgr2=0x55f120de4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120de4040 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c095230 tx=0x7f1d9c00cf40 comp rx=0 tx=0).stop 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/156559880 shutdown_connections 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120dca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120c246b0 0x55f120de4040 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/156559880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120dc8df0 0x55f120ddd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/156559880 >> 172.21.15.31:0/156559880 conn(0x55f120dbdfd0 msgr2=0x55f120dd0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/156559880 shutdown_connections 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.265+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/156559880 wait complete. 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 -- start start 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 Processor -- start 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120c246b0 0x55f120dd9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 0x55f120dd96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120ddcbe0 con 0x55f120c246b0 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120ddcd50 con 0x55f120dc9cf0 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f120ddcf10 con 0x55f120dc8df0 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.266+0000 7f1dc4c86b80 10 client.0 _ll_register_callbacks cb 0x55f120cbf5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 0x55f120dd96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 0x55f120dd96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40152/0 (socket says 172.21.15.31:40152) 2023-07-29T16:27:48.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 -- 172.21.15.31:0/2177897427 learned_addr learned my addr 172.21.15.31:0/2177897427 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1dab7fe700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daaffd700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120c246b0 0x55f120dd9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 -- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 msgr2=0x55f120ddc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 -- 172.21.15.31:0/2177897427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120c246b0 msgr2=0x55f120dd9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120c246b0 0x55f120dd9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 -- 172.21.15.31:0/2177897427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dac109090 con 0x55f120dc8df0 2023-07-29T16:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1daa7fc700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 0x55f120dd96b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1da000b680 tx=0x7f1da0028040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4027f37c7d261e3f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.267+0000 7f1d8eff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1d8eff5700 1 -- 172.21.15.31:0/2177897427 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da001d040 con 0x55f120dc8df0 2023-07-29T16:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d9c0994b0 con 0x55f120dc8df0 2023-07-29T16:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1d8eff5700 1 -- 172.21.15.31:0/2177897427 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1da0028b40 con 0x55f120dc8df0 2023-07-29T16:27:48.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1d8eff5700 1 -- 172.21.15.31:0/2177897427 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1da0028470 con 0x55f120dc8df0 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 10 client.29143 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f120b30c00 con 0x55f120dc8df0 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1d8eff5700 1 -- 172.21.15.31:0/2177897427 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d9c0994b0 con 0x55f120dc8df0 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1d8eff5700 1 -- 172.21.15.31:0/2177897427 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f1da0004740 con 0x55f120dc8df0 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 10 client.29143 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 10 client.29143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:48.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 20 client.29143 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120e343a0 0x55f120e367b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 4 client.29143 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.268+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f120d12ce0 con 0x55f120e343a0 2023-07-29T16:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.269+0000 7f1daaffd700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120e343a0 0x55f120e367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.269+0000 7f1daaffd700 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120e343a0 0x55f120e367b0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ba34937ebac265f server_cookie=31908d9c4c40bae1 in_seq=0 out_seq=0 2023-07-29T16:27:48.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.269+0000 7f1d8eff5700 10 client.29143 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:48.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:47 smithi039 ceph-mon[128858]: pgmap v741: 97 pgs: 97 active+clean; 158 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 2.3 MiB/s wr, 246 op/s 2023-07-29T16:27:48.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:47 smithi156 ceph-mon[129245]: pgmap v741: 97 pgs: 97 active+clean; 158 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 2.3 MiB/s wr, 246 op/s 2023-07-29T16:27:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.350+0000 7f1d8eff5700 1 -- 172.21.15.31:0/2177897427 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f120d12ce0 con 0x55f120e343a0 2023-07-29T16:27:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.350+0000 7f1d8eff5700 10 client.29143 handle_command_reply: tid=0 2023-07-29T16:27:48.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.350+0000 7f1dc4c86b80 1 client.29143 shutdown 2023-07-29T16:27:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 msgr2=0x55f120dd96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 0x55f120dd96b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1da000b680 tx=0x7f1da0028040 comp rx=0 tx=0).stop 2023-07-29T16:27:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 shutdown_connections 2023-07-29T16:27:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f120e343a0 0x55f120e367b0 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f120dc9cf0 0x55f120ddc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f120c246b0 0x55f120dd9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 --2- 172.21.15.31:0/2177897427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f120dc8df0 0x55f120dd96b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 >> 172.21.15.31:0/2177897427 conn(0x55f120dbdfd0 msgr2=0x55f120dd0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1d8eff5700 0 client.29143 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 shutdown_connections 2023-07-29T16:27:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 1 -- 172.21.15.31:0/2177897427 wait complete. 2023-07-29T16:27:48.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.351+0000 7f1dc4c86b80 20 client.29143 trim_cache size 0 max 16384 2023-07-29T16:27:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:27:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:27:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:27:48.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:27:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d940a0810 msgr2=0x7f1d940a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d940a0810 0x7f1d940a2cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x55f120dc8610 tx=0x7f1da4009920 comp rx=0 tx=0).stop 2023-07-29T16:27:48.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 msgr2=0x7f1dac1961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 0x7f1dac1961b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e970 tx=0x7f1d9c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:27:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 shutdown_connections 2023-07-29T16:27:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d940a0810 0x7f1d940a2cc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1dac105f20 0x7f1dac197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1dac068e20 0x7f1dac1961b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 --2- 172.21.15.31:0/1858768027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dac103470 0x7f1dac1966f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 >> 172.21.15.31:0/1858768027 conn(0x7f1dac0fa840 msgr2=0x7f1dac1017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 shutdown_connections 2023-07-29T16:27:48.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:27:48.352+0000 7f1db13ff700 1 -- 172.21.15.31:0/1858768027 wait complete. 2023-07-29T16:27:48.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:47 smithi031 ceph-mon[95504]: pgmap v741: 97 pgs: 97 active+clean; 158 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 2.3 MiB/s wr, 246 op/s 2023-07-29T16:27:48.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 949632 8.95 MB/sec execute 453 sec latency 49.223 ms 2023-07-29T16:27:49.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 951674 8.95 MB/sec execute 454 sec latency 51.729 ms 2023-07-29T16:27:50.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:50 smithi156 ceph-mon[129245]: pgmap v742: 97 pgs: 97 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 3.0 MiB/s wr, 315 op/s 2023-07-29T16:27:50.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 952765 8.95 MB/sec execute 455 sec latency 26.393 ms 2023-07-29T16:27:50.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:50 smithi031 ceph-mon[95504]: pgmap v742: 97 pgs: 97 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 3.0 MiB/s wr, 315 op/s 2023-07-29T16:27:51.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:50 smithi039 ceph-mon[128858]: pgmap v742: 97 pgs: 97 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 3.0 MiB/s wr, 315 op/s 2023-07-29T16:27:51.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 954954 8.95 MB/sec execute 456 sec latency 19.287 ms 2023-07-29T16:27:52.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:52 smithi156 ceph-mon[129245]: pgmap v743: 97 pgs: 97 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.0 MiB/s wr, 201 op/s 2023-07-29T16:27:52.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 956174 8.95 MB/sec execute 457 sec latency 14.481 ms 2023-07-29T16:27:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:52 smithi031 ceph-mon[95504]: pgmap v743: 97 pgs: 97 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.0 MiB/s wr, 201 op/s 2023-07-29T16:27:53.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:52 smithi039 ceph-mon[128858]: pgmap v743: 97 pgs: 97 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 2.0 MiB/s wr, 201 op/s 2023-07-29T16:27:53.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 958187 8.95 MB/sec execute 458 sec latency 14.405 ms 2023-07-29T16:27:54.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:54 smithi156 ceph-mon[129245]: pgmap v744: 97 pgs: 97 active+clean; 158 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 2.5 MiB/s wr, 280 op/s 2023-07-29T16:27:54.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 959645 8.95 MB/sec execute 459 sec latency 15.699 ms 2023-07-29T16:27:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:54 smithi031 ceph-mon[95504]: pgmap v744: 97 pgs: 97 active+clean; 158 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 2.5 MiB/s wr, 280 op/s 2023-07-29T16:27:55.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:54 smithi039 ceph-mon[128858]: pgmap v744: 97 pgs: 97 active+clean; 158 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 2.5 MiB/s wr, 280 op/s 2023-07-29T16:27:55.155 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:27:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.414+0000 7fa39e3ff700 1 -- 172.21.15.31:0/2311453327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 msgr2=0x7fa398101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.414+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/2311453327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398101960 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009990 tx=0x7fa38c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.414+0000 7fa39e3ff700 1 -- 172.21.15.31:0/2311453327 shutdown_connections 2023-07-29T16:27:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.414+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/2311453327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa398101ea0 0x7fa398108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.414+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/2311453327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 0x7fa398100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.414+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/2311453327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398101960 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.415+0000 7fa39e3ff700 1 -- 172.21.15.31:0/2311453327 >> 172.21.15.31:0/2311453327 conn(0x7fa3980fb920 msgr2=0x7fa3980fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.415+0000 7fa39e3ff700 1 -- 172.21.15.31:0/2311453327 shutdown_connections 2023-07-29T16:27:55.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.415+0000 7fa39e3ff700 1 -- 172.21.15.31:0/2311453327 wait complete. 2023-07-29T16:27:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.416+0000 7fa39e3ff700 1 Processor -- start 2023-07-29T16:27:55.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 -- start start 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 0x7fa398196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa398101ea0 0x7fa39819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39819dbc0 con 0x7fa398100320 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39819dd00 con 0x7fa398101ea0 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39819de90 con 0x7fa398101510 2023-07-29T16:27:55.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa397fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 0x7fa398196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59262/0 (socket says 172.21.15.31:59262) 2023-07-29T16:27:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa39c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa398101ea0 0x7fa39819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa397fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 0x7fa398196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49486/0 (socket says 172.21.15.31:49486) 2023-07-29T16:27:55.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.417+0000 7fa3977fe700 1 -- 172.21.15.31:0/3478495154 learned_addr learned my addr 172.21.15.31:0/3478495154 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3977fe700 1 -- 172.21.15.31:0/3478495154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa398101ea0 msgr2=0x7fa39819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3977fe700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa398101ea0 0x7fa39819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3977fe700 1 -- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 msgr2=0x7fa398196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3977fe700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 0x7fa398196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3977fe700 1 -- 172.21.15.31:0/3478495154 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0095f0 con 0x7fa398101510 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3977fe700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398196580 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa38c004740 tx=0x7fa38c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3957fa700 1 -- 172.21.15.31:0/3478495154 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c01d070 con 0x7fa398101510 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3957fa700 1 -- 172.21.15.31:0/3478495154 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa38c004830 con 0x7fa398101510 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa3957fa700 1 -- 172.21.15.31:0/3478495154 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c005330 con 0x7fa398101510 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.418+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39819e110 con 0x7fa398101510 2023-07-29T16:27:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.419+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39819e530 con 0x7fa398101510 2023-07-29T16:27:55.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.422+0000 7fa3957fa700 1 -- 172.21.15.31:0/3478495154 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa38c0054f0 con 0x7fa398101510 2023-07-29T16:27:55.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.424+0000 7fa3957fa700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3800a0810 0x7fa3800a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.424+0000 7fa3957fa700 1 -- 172.21.15.31:0/3478495154 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa38c0ccba0 con 0x7fa398101510 2023-07-29T16:27:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.424+0000 7fa397fff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3800a0810 0x7fa3800a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.425+0000 7fa397fff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3800a0810 0x7fa3800a2cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa398101370 tx=0x7fa388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.425+0000 7fa3b1c23b80 1 Processor -- start 2023-07-29T16:27:55.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 -- start start 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e31be0 0x561946e31ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 0x561946e32d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e33740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946d89a10 con 0x561946e31be0 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946d873d0 con 0x561946e332f0 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946d82cc0 con 0x561946e32530 2023-07-29T16:27:55.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 0x561946e32d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 0x561946e32d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59270/0 (socket says 172.21.15.31:59270) 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa3977fe700 1 -- 172.21.15.31:0/1599742421 learned_addr learned my addr 172.21.15.31:0/1599742421 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa397fff700 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e31be0 0x561946e31ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.426+0000 7fa39c99c700 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e33740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa3977fe700 1 -- 172.21.15.31:0/1599742421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 msgr2=0x561946e33740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa3977fe700 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e33740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa3977fe700 1 -- 172.21.15.31:0/1599742421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e31be0 msgr2=0x561946e31ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa3977fe700 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e31be0 0x561946e31ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa3977fe700 1 -- 172.21.15.31:0/1599742421 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561946bb0de0 con 0x561946e32530 2023-07-29T16:27:55.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa397fff700 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e31be0 0x561946e31ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:55.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa3977fe700 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 0x561946e32d60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa38c096050 tx=0x7fa38c099f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba4fb5f42529827a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa37e7fc700 1 -- 172.21.15.31:0/1599742421 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c098410 con 0x561946e32530 2023-07-29T16:27:55.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa37e7fc700 1 -- 172.21.15.31:0/1599742421 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa38c098670 con 0x561946e32530 2023-07-29T16:27:55.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.427+0000 7fa37e7fc700 1 -- 172.21.15.31:0/1599742421 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0a2200 con 0x561946e32530 2023-07-29T16:27:55.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/1599742421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 msgr2=0x561946e32d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 0x561946e32d60 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa38c096050 tx=0x7fa38c099f40 comp rx=0 tx=0).stop 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/1599742421 shutdown_connections 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e33740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e31be0 0x561946e31ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/1599742421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e32530 0x561946e32d60 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/1599742421 >> 172.21.15.31:0/1599742421 conn(0x561946e273e0 msgr2=0x561946e2c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/1599742421 shutdown_connections 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.428+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/1599742421 wait complete. 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.429+0000 7fa3b1c23b80 1 -- start start 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.429+0000 7fa3b1c23b80 1 Processor -- start 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.429+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e3b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.429+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e3b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.429+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.429+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946d1f510 con 0x561946e32530 2023-07-29T16:27:55.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946d1f1b0 con 0x561946e332f0 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946c06920 con 0x561946e31be0 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa3b1c23b80 10 client.0 _ll_register_callbacks cb 0x561946d28620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e3b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e3b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59278/0 (socket says 172.21.15.31:59278) 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 -- 172.21.15.31:0/4160871646 learned_addr learned my addr 172.21.15.31:0/4160871646 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa39c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa3977fe700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e3b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 -- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 msgr2=0x561946e429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 -- 172.21.15.31:0/4160871646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 msgr2=0x561946e3b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e3b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa397fff700 1 -- 172.21.15.31:0/4160871646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39819e530 con 0x561946e31be0 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.430+0000 7fa39c99c700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa397fff700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e3b460 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa388009b20 tx=0x7fa3880047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b6b38d94b161926 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa3777fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:55.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa3777fe700 1 -- 172.21.15.31:0/4160871646 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38801c840 con 0x561946e31be0 2023-07-29T16:27:55.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa3777fe700 1 -- 172.21.15.31:0/4160871646 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa38801c9a0 con 0x561946e31be0 2023-07-29T16:27:55.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561946dadd10 con 0x561946e31be0 2023-07-29T16:27:55.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa3777fe700 1 -- 172.21.15.31:0/4160871646 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa38801cb60 con 0x561946e31be0 2023-07-29T16:27:55.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.431+0000 7fa3b1c23b80 10 client.29155 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561946b99c60 con 0x561946e31be0 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3777fe700 1 -- 172.21.15.31:0/4160871646 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561946dadd10 con 0x561946e31be0 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3777fe700 1 -- 172.21.15.31:0/4160871646 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa38800e3b0 con 0x561946e31be0 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 10 client.29155 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 10 client.29155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 20 client.29155 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:55.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e471e0 0x561946e495f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 4 client.29155 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561946e00d70 con 0x561946e471e0 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.432+0000 7fa3977fe700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e471e0 0x561946e495f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.433+0000 7fa3977fe700 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e471e0 0x561946e495f0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49da311f8036a3f5 server_cookie=d3b7e649ce03d90d in_seq=0 out_seq=0 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.433+0000 7fa3777fe700 10 client.29155 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3777fe700 1 -- 172.21.15.31:0/4160871646 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561946e00d70 con 0x561946e471e0 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3777fe700 10 client.29155 handle_command_reply: tid=0 2023-07-29T16:27:55.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 client.29155 shutdown 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 msgr2=0x561946e3b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e3b460 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa388009b20 tx=0x7fa3880047c0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 shutdown_connections 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e471e0 0x561946e495f0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e3b460 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/4160871646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e3b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 >> 172.21.15.31:0/4160871646 conn(0x561946e273e0 msgr2=0x561946e2c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3777fe700 0 client.29155 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.434+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 shutdown_connections 2023-07-29T16:27:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.435+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/4160871646 wait complete. 2023-07-29T16:27:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.435+0000 7fa3b1c23b80 20 client.29155 trim_cache size 0 max 16384 2023-07-29T16:27:55.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.465+0000 7fa3b1c23b80 1 Processor -- start 2023-07-29T16:27:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 -- start start 2023-07-29T16:27:55.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e36110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e429a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e36650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946e294a0 con 0x561946e32530 2023-07-29T16:27:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946e26ed0 con 0x561946e332f0 2023-07-29T16:27:55.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.466+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946e32bc0 con 0x561946e31be0 2023-07-29T16:27:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.467+0000 7fa39c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e36650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.467+0000 7fa39c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e36650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50330/0 (socket says 172.21.15.31:50330) 2023-07-29T16:27:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.467+0000 7fa39c99c700 1 -- 172.21.15.31:0/2870486996 learned_addr learned my addr 172.21.15.31:0/2870486996 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.467+0000 7fa397fff700 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e36110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.467+0000 7fa3977fe700 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e429a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa39c99c700 1 -- 172.21.15.31:0/2870486996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 msgr2=0x561946e36110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa39c99c700 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e36110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa39c99c700 1 -- 172.21.15.31:0/2870486996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 msgr2=0x561946e429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa39c99c700 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e429a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa39c99c700 1 -- 172.21.15.31:0/2870486996 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561946bd7250 con 0x561946e332f0 2023-07-29T16:27:55.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa39c99c700 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e36650 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa390011c30 tx=0x7fa390009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8774cd6c6ce43d5a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.468+0000 7fa37effd700 1 -- 172.21.15.31:0/2870486996 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39000e040 con 0x561946e332f0 2023-07-29T16:27:55.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.469+0000 7fa37effd700 1 -- 172.21.15.31:0/2870486996 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa39000f040 con 0x561946e332f0 2023-07-29T16:27:55.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.469+0000 7fa37effd700 1 -- 172.21.15.31:0/2870486996 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa390004030 con 0x561946e332f0 2023-07-29T16:27:55.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.470+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/2870486996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 msgr2=0x561946e36650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.470+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e36650 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa390011c30 tx=0x7fa390009920 comp rx=0 tx=0).stop 2023-07-29T16:27:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.471+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/2870486996 shutdown_connections 2023-07-29T16:27:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.471+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e332f0 0x561946e36650 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.471+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e36110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.471+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/2870486996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e32530 0x561946e429a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.471+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/2870486996 >> 172.21.15.31:0/2870486996 conn(0x561946e273e0 msgr2=0x561946e39600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.472+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/2870486996 shutdown_connections 2023-07-29T16:27:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.472+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/2870486996 wait complete. 2023-07-29T16:27:55.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.472+0000 7fa3b1c23b80 1 -- start start 2023-07-29T16:27:55.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 Processor -- start 2023-07-29T16:27:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e428b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 0x561946e3f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e332f0 0x561946e3f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946e98d60 con 0x561946e332f0 2023-07-29T16:27:55.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946e98ed0 con 0x561946e32530 2023-07-29T16:27:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.473+0000 7fa3b1c23b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561946e99090 con 0x561946e31be0 2023-07-29T16:27:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.474+0000 7fa3b1c23b80 10 client.0 _ll_register_callbacks cb 0x561946d28620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.474+0000 7fa3977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 0x561946e3f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.474+0000 7fa3977fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 0x561946e3f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50334/0 (socket says 172.21.15.31:50334) 2023-07-29T16:27:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.474+0000 7fa3977fe700 1 -- 172.21.15.31:0/290239347 learned_addr learned my addr 172.21.15.31:0/290239347 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:55.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.474+0000 7fa397fff700 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3977fe700 1 -- 172.21.15.31:0/290239347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 msgr2=0x561946e428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3977fe700 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3977fe700 1 -- 172.21.15.31:0/290239347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e332f0 msgr2=0x561946e3f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3977fe700 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e332f0 0x561946e3f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3977fe700 1 -- 172.21.15.31:0/290239347 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3900118e0 con 0x561946e32530 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3977fe700 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 0x561946e3f350 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa38c099e80 tx=0x7fa38c09e480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1419859adef8f90 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3777fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3777fe700 1 -- 172.21.15.31:0/290239347 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c098410 con 0x561946e32530 2023-07-29T16:27:55.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3777fe700 1 -- 172.21.15.31:0/290239347 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa38c0997c0 con 0x561946e32530 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.475+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa38c098670 con 0x561946e32530 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3777fe700 1 -- 172.21.15.31:0/290239347 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa38c099980 con 0x561946e32530 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 10 client.29162 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561946baf2b0 con 0x561946e32530 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3777fe700 1 -- 172.21.15.31:0/290239347 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa38c0ca030 con 0x561946e32530 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3777fe700 1 -- 172.21.15.31:0/290239347 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fa38c0a2320 con 0x561946e32530 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 10 client.29162 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:55.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 10 client.29162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:55.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 20 client.29162 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:55.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e9a550 0x561946e9c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 4 client.29162 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:27:55.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.476+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561946d7bd40 con 0x561946e9a550 2023-07-29T16:27:55.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.477+0000 7fa397fff700 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e9a550 0x561946e9c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.478+0000 7fa397fff700 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e9a550 0x561946e9c960 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed4afa9c75ee4c47 server_cookie=28120883cf926ce9 in_seq=0 out_seq=0 2023-07-29T16:27:55.479 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.478+0000 7fa3777fe700 10 client.29162 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:55.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.481+0000 7fa3777fe700 1 -- 172.21.15.31:0/290239347 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561946d7bd40 con 0x561946e9a550 2023-07-29T16:27:55.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.481+0000 7fa3777fe700 10 client.29162 handle_command_reply: tid=0 2023-07-29T16:27:55.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.482+0000 7fa3b1c23b80 1 client.29162 shutdown 2023-07-29T16:27:55.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.482+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 msgr2=0x561946e3f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.482+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 0x561946e3f350 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa38c099e80 tx=0x7fa38c09e480 comp rx=0 tx=0).stop 2023-07-29T16:27:55.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 shutdown_connections 2023-07-29T16:27:55.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561946e9a550 0x561946e9c960 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561946e332f0 0x561946e3f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561946e31be0 0x561946e428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 --2- 172.21.15.31:0/290239347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561946e32530 0x561946e3f350 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 >> 172.21.15.31:0/290239347 conn(0x561946e273e0 msgr2=0x561946e36110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3777fe700 0 client.29162 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.483+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 shutdown_connections 2023-07-29T16:27:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.484+0000 7fa3b1c23b80 1 -- 172.21.15.31:0/290239347 wait complete. 2023-07-29T16:27:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.484+0000 7fa3b1c23b80 20 client.29162 trim_cache size 0 max 16384 2023-07-29T16:27:55.484 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:27:55.484 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:27:55.484 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:27:55.484 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:27:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3800a0810 msgr2=0x7fa3800a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3800a0810 0x7fa3800a2cc0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa398101370 tx=0x7fa388009260 comp rx=0 tx=0).stop 2023-07-29T16:27:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 msgr2=0x7fa398196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398196580 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa38c004740 tx=0x7fa38c009580 comp rx=0 tx=0).stop 2023-07-29T16:27:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 shutdown_connections 2023-07-29T16:27:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa3800a0810 0x7fa3800a2cc0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa398101ea0 0x7fa39819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa398100320 0x7fa398196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 --2- 172.21.15.31:0/3478495154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa398101510 0x7fa398196580 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.486+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 >> 172.21.15.31:0/3478495154 conn(0x7fa3980fb920 msgr2=0x7fa3980fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.487+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 shutdown_connections 2023-07-29T16:27:55.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.487+0000 7fa39e3ff700 1 -- 172.21.15.31:0/3478495154 wait complete. 2023-07-29T16:27:55.512 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:27:55.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:27:55.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 -- 172.21.15.31:0/1886406079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c101ec0 msgr2=0x7fdd1c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 --2- 172.21.15.31:0/1886406079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c101ec0 0x7fdd1c108780 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdd1400b0a0 tx=0x7fdd1400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 -- 172.21.15.31:0/1886406079 shutdown_connections 2023-07-29T16:27:55.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 --2- 172.21.15.31:0/1886406079 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c101ec0 0x7fdd1c108780 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 --2- 172.21.15.31:0/1886406079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd1c100340 0x7fdd1c100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 --2- 172.21.15.31:0/1886406079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd1c101530 0x7fdd1c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 -- 172.21.15.31:0/1886406079 >> 172.21.15.31:0/1886406079 conn(0x7fdd1c0fb920 msgr2=0x7fdd1c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:55.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.759+0000 7fdd221ff700 1 -- 172.21.15.31:0/1886406079 shutdown_connections 2023-07-29T16:27:55.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.760+0000 7fdd221ff700 1 -- 172.21.15.31:0/1886406079 wait complete. 2023-07-29T16:27:55.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.761+0000 7fdd221ff700 1 Processor -- start 2023-07-29T16:27:55.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.761+0000 7fdd221ff700 1 -- start start 2023-07-29T16:27:55.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd221ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 0x7fdd1c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd221ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd1c101530 0x7fdd1c198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd221ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd1c101ec0 0x7fdd1c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd221ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c19fe80 con 0x7fdd1c101530 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd221ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c19ffc0 con 0x7fdd1c100340 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd221ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1a0150 con 0x7fdd1c101ec0 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 0x7fdd1c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 0x7fdd1c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50356/0 (socket says 172.21.15.31:50356) 2023-07-29T16:27:55.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1b7fe700 1 -- 172.21.15.31:0/603051585 learned_addr learned my addr 172.21.15.31:0/603051585 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1affd700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd1c101530 0x7fdd1c198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1bfff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd1c101ec0 0x7fdd1c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1b7fe700 1 -- 172.21.15.31:0/603051585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd1c101ec0 msgr2=0x7fdd1c19f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.762+0000 7fdd1b7fe700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd1c101ec0 0x7fdd1c19f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd1b7fe700 1 -- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd1c101530 msgr2=0x7fdd1c198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd1b7fe700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd1c101530 0x7fdd1c198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:55.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd1b7fe700 1 -- 172.21.15.31:0/603051585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd14009d00 con 0x7fdd1c100340 2023-07-29T16:27:55.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd1b7fe700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 0x7fdd1c198300 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e970 tx=0x7fdd0c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c009790 con 0x7fdd1c100340 2023-07-29T16:27:55.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd0c00c7e0 con 0x7fdd1c100340 2023-07-29T16:27:55.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c003c70 con 0x7fdd1c100340 2023-07-29T16:27:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c1a0430 con 0x7fdd1c100340 2023-07-29T16:27:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.763+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c1a08f0 con 0x7fdd1c100340 2023-07-29T16:27:55.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.764+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1c10bb30 con 0x7fdd1c100340 2023-07-29T16:27:55.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.767+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdd0c00cc50 con 0x7fdd1c100340 2023-07-29T16:27:55.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.768+0000 7fdd18ff9700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdd040a0860 0x7fdd040a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:55.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.768+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdd0c014070 con 0x7fdd1c100340 2023-07-29T16:27:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.769+0000 7fdd1affd700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdd040a0860 0x7fdd040a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:55.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.769+0000 7fdd1affd700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdd040a0860 0x7fdd040a2d10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009c60 tx=0x7fdd10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:55.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:55.771+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fdd0c095950 con 0x7fdd1c100340 2023-07-29T16:27:55.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:55 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:27:55.895 INFO:tasks.workunit.client.0.smithi031.stdout: 1 960198 8.94 MB/sec execute 460 sec latency 119.290 ms 2023-07-29T16:27:55.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:55 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:27:56.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:55 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:27:56.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.051+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd1c10bd20 con 0x7fdd1c100340 2023-07-29T16:27:56.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.052+0000 7fdd18ff9700 1 -- 172.21.15.31:0/603051585 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10781 (secure 0 0 0) 0x7fdd0c098f70 con 0x7fdd1c100340 2023-07-29T16:27:56.051 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:27:56.051 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:27:32.973421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.054+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdd040a0860 msgr2=0x7fdd040a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.054+0000 7fdd221ff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdd040a0860 0x7fdd040a2d10 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fdd10009c60 tx=0x7fdd10009260 comp rx=0 tx=0).stop 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.054+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 msgr2=0x7fdd1c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.054+0000 7fdd221ff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 0x7fdd1c198300 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00e970 tx=0x7fdd0c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 shutdown_connections 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdd040a0860 0x7fdd040a2d10 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd1c101ec0 0x7fdd1c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd1c100340 0x7fdd1c198300 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 --2- 172.21.15.31:0/603051585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd1c101530 0x7fdd1c198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 >> 172.21.15.31:0/603051585 conn(0x7fdd1c0fb920 msgr2=0x7fdd1c106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 shutdown_connections 2023-07-29T16:27:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.055+0000 7fdd221ff700 1 -- 172.21.15.31:0/603051585 wait complete. 2023-07-29T16:27:56.055 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2023-07-29T16:27:56.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:27:56.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/954430393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc681040d0 msgr2=0x7fcc6810a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/954430393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc681040d0 0x7fcc6810a990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009f90 tx=0x7fcc6000b330 comp rx=0 tx=0).stop 2023-07-29T16:27:56.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/954430393 shutdown_connections 2023-07-29T16:27:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/954430393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc681040d0 0x7fcc6810a990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/954430393 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc68102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/954430393 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc68103740 0x7fcc68103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.324+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/954430393 >> 172.21.15.31:0/954430393 conn(0x7fcc680fdb90 msgr2=0x7fcc680fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.325+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/954430393 shutdown_connections 2023-07-29T16:27:56.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.325+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/954430393 wait complete. 2023-07-29T16:27:56.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.326+0000 7fcc6d9ff700 1 Processor -- start 2023-07-29T16:27:56.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.327+0000 7fcc6d9ff700 1 -- start start 2023-07-29T16:27:56.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.327+0000 7fcc6d9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc6819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc6d9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 0x7fcc6819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc6d9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc681040d0 0x7fcc681a3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc6d9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681a4220 con 0x7fcc68102550 2023-07-29T16:27:56.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc6d9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681a4360 con 0x7fcc68103740 2023-07-29T16:27:56.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc6d9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc681a44f0 con 0x7fcc681040d0 2023-07-29T16:27:56.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc5ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 0x7fcc6819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc6819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc5ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 0x7fcc6819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50370/0 (socket says 172.21.15.31:50370) 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc5ffff700 1 -- 172.21.15.31:0/1798454379 learned_addr learned my addr 172.21.15.31:0/1798454379 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.328+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc6819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49554/0 (socket says 172.21.15.31:49554) 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc681040d0 0x7fcc681a3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc5ffff700 1 -- 172.21.15.31:0/1798454379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc681040d0 msgr2=0x7fcc681a3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc681040d0 0x7fcc681a3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc5ffff700 1 -- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 msgr2=0x7fcc6819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc6819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc5ffff700 1 -- 172.21.15.31:0/1798454379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc60009c40 con 0x7fcc68103740 2023-07-29T16:27:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc681040d0 0x7fcc681a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.329+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc6819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.330+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 0x7fcc6819cbe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000d950 tx=0x7fcc5000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.331+0000 7fcc6c9fd700 1 -- 172.21.15.31:0/1798454379 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50009b40 con 0x7fcc68103740 2023-07-29T16:27:56.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.331+0000 7fcc6c9fd700 1 -- 172.21.15.31:0/1798454379 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc50009ca0 con 0x7fcc68103740 2023-07-29T16:27:56.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.331+0000 7fcc6c9fd700 1 -- 172.21.15.31:0/1798454379 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000f620 con 0x7fcc68103740 2023-07-29T16:27:56.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.331+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc681a47d0 con 0x7fcc68103740 2023-07-29T16:27:56.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.331+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc681a4c90 con 0x7fcc68103740 2023-07-29T16:27:56.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.336+0000 7fcc6c9fd700 1 -- 172.21.15.31:0/1798454379 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fcc50010460 con 0x7fcc68103740 2023-07-29T16:27:56.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 Processor -- start 2023-07-29T16:27:56.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 -- start start 2023-07-29T16:27:56.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc6c9fd700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc540a0910 0x7fcc540a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c804ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc6c9fd700 1 -- 172.21.15.31:0/1798454379 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcc500cbd00 con 0x7fcc68103740 2023-07-29T16:27:56.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 0x55d76c805d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c8062f0 0x55d76c806740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c75ca10 con 0x55d76c8062f0 2023-07-29T16:27:56.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c75a3d0 con 0x55d76c804be0 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.338+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c755cc0 con 0x55d76c805530 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc540a0910 0x7fcc540a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 0x55d76c805d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 0x55d76c805d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59344/0 (socket says 172.21.15.31:59344) 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 -- 172.21.15.31:0/371835184 learned_addr learned my addr 172.21.15.31:0/371835184 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc677fe700 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c8062f0 0x55d76c806740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c804ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 -- 172.21.15.31:0/371835184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 msgr2=0x55d76c804ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c804ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 -- 172.21.15.31:0/371835184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c8062f0 msgr2=0x55d76c806740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c8062f0 0x55d76c806740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 -- 172.21.15.31:0/371835184 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d76c583de0 con 0x55d76c805530 2023-07-29T16:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.339+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 0x55d76c805d60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcc50095e10 tx=0x7fcc50099220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1db3eeaaa924de8d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.340+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc540a0910 0x7fcc540a2dc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009040 tx=0x7fcc600094b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.340+0000 7fcc5dffb700 1 -- 172.21.15.31:0/371835184 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5009d3d0 con 0x55d76c805530 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.340+0000 7fcc5dffb700 1 -- 172.21.15.31:0/371835184 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc500af020 con 0x55d76c805530 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.340+0000 7fcc5dffb700 1 -- 172.21.15.31:0/371835184 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50010970 con 0x55d76c805530 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.340+0000 7fcc8123eb80 1 -- 172.21.15.31:0/371835184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 msgr2=0x55d76c805d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.340+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 0x55d76c805d60 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcc50095e10 tx=0x7fcc50099220 comp rx=0 tx=0).stop 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 -- 172.21.15.31:0/371835184 shutdown_connections 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c8062f0 0x55d76c806740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c804ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/371835184 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c805530 0x55d76c805d60 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 -- 172.21.15.31:0/371835184 >> 172.21.15.31:0/371835184 conn(0x55d76c7fa3e0 msgr2=0x55d76c7ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 -- 172.21.15.31:0/371835184 shutdown_connections 2023-07-29T16:27:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 -- 172.21.15.31:0/371835184 wait complete. 2023-07-29T16:27:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.341+0000 7fcc8123eb80 1 -- start start 2023-07-29T16:27:56.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 Processor -- start 2023-07-29T16:27:56.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 0x55d76c80e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c80ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c8062f0 0x55d76c815d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c6f2510 con 0x55d76c805530 2023-07-29T16:27:56.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c6f21b0 con 0x55d76c8062f0 2023-07-29T16:27:56.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c5d9920 con 0x55d76c804be0 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc8123eb80 10 client.0 _ll_register_callbacks cb 0x55d76c6fb620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.342+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 0x55d76c80e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 0x55d76c80e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59346/0 (socket says 172.21.15.31:59346) 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1474872780 learned_addr learned my addr 172.21.15.31:0/1474872780 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c8062f0 0x55d76c815d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c80ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1474872780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c8062f0 msgr2=0x55d76c815d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c8062f0 0x55d76c815d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1474872780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 msgr2=0x55d76c80ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c80ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1474872780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc681a4c90 con 0x55d76c804be0 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c80ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 0x55d76c80e800 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcc58003a50 tx=0x7fcc58003d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de058d8c850d4457 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.343+0000 7fcc46ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:27:56.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1474872780 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58011ad0 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1474872780 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcc58011c30 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d76c780d10 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1474872780 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc58011df0 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc8123eb80 10 client.29179 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d76c56cc60 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1474872780 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d76c780d10 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1474872780 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fcc58012680 con 0x55d76c804be0 2023-07-29T16:27:56.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc8123eb80 10 client.29179 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.344+0000 7fcc8123eb80 10 client.29179 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc8123eb80 20 client.29179 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c81a580 0x55d76c81c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc8123eb80 4 client.29179 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d76c7d3d70 con 0x55d76c81a580 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c81a580 0x55d76c81c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c81a580 0x55d76c81c990 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4df3bb0ec12dd43f server_cookie=a925e3674856bc0 in_seq=0 out_seq=0 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.345+0000 7fcc46ffd700 10 client.29179 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1474872780 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d76c7d3d70 con 0x55d76c81a580 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc46ffd700 10 client.29179 handle_command_reply: tid=0 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 client.29179 shutdown 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 msgr2=0x55d76c80e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 0x55d76c80e800 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcc58003a50 tx=0x7fcc58003d60 comp rx=0 tx=0).stop 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 shutdown_connections 2023-07-29T16:27:56.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c81a580 0x55d76c81c990 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c8062f0 0x55d76c815d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c804be0 0x55d76c80e800 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1474872780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c80ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc46ffd700 0 client.29179 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.346+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 >> 172.21.15.31:0/1474872780 conn(0x55d76c7fa3e0 msgr2=0x55d76c7ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.347+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 shutdown_connections 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.347+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1474872780 wait complete. 2023-07-29T16:27:56.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.347+0000 7fcc8123eb80 20 client.29179 trim_cache size 0 max 16384 2023-07-29T16:27:56.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 Processor -- start 2023-07-29T16:27:56.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 -- start start 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c804be0 0x55d76c813d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c805530 0x55d76c809230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c809770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c80dcb0 con 0x55d76c804be0 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c7fc460 con 0x55d76c805530 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.361+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c7f7770 con 0x55d76c8062f0 2023-07-29T16:27:56.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c804be0 0x55d76c813d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c809770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc5ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c805530 0x55d76c809230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c809770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59362/0 (socket says 172.21.15.31:59362) 2023-07-29T16:27:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c804be0 0x55d76c813d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49582/0 (socket says 172.21.15.31:49582) 2023-07-29T16:27:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 -- 172.21.15.31:0/1181351911 learned_addr learned my addr 172.21.15.31:0/1181351911 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc5ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c805530 0x55d76c809230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50406/0 (socket says 172.21.15.31:50406) 2023-07-29T16:27:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 -- 172.21.15.31:0/1181351911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c805530 msgr2=0x55d76c809230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c805530 0x55d76c809230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 -- 172.21.15.31:0/1181351911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c804be0 msgr2=0x55d76c813d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c804be0 0x55d76c813d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 -- 172.21.15.31:0/1181351911 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d76c80d780 con 0x55d76c8062f0 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.362+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c809770 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcc60015000 tx=0x7fcc6001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d60490a4b359aab7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.363+0000 7fcc5e7fc700 1 -- 172.21.15.31:0/1181351911 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc60030070 con 0x55d76c8062f0 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.363+0000 7fcc5e7fc700 1 -- 172.21.15.31:0/1181351911 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc6000ba40 con 0x55d76c8062f0 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.363+0000 7fcc5e7fc700 1 -- 172.21.15.31:0/1181351911 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6002cb10 con 0x55d76c8062f0 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.363+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1181351911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 msgr2=0x55d76c809770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.363+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c809770 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcc60015000 tx=0x7fcc6001d000 comp rx=0 tx=0).stop 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1181351911 shutdown_connections 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c809770 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c804be0 0x55d76c813d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1181351911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c805530 0x55d76c809230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1181351911 >> 172.21.15.31:0/1181351911 conn(0x55d76c7fa3e0 msgr2=0x55d76c80c720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1181351911 shutdown_connections 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1181351911 wait complete. 2023-07-29T16:27:56.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 -- start start 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.364+0000 7fcc8123eb80 1 Processor -- start 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c812c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c813170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c86c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c813800 con 0x55d76c805530 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c86c640 con 0x55d76c804be0 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d76c86c780 con 0x55d76c8062f0 2023-07-29T16:27:56.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc8123eb80 10 client.0 _ll_register_callbacks cb 0x55d76c6fb620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:56.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c812c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc66ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c812c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50422/0 (socket says 172.21.15.31:50422) 2023-07-29T16:27:56.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1676798179 learned_addr learned my addr 172.21.15.31:0/1676798179 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c813170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.365+0000 7fcc677fe700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c86c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 msgr2=0x55d76c86c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c86c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1676798179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 msgr2=0x55d76c813170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c813170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc66ffd700 1 -- 172.21.15.31:0/1676798179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6810b2c0 con 0x55d76c804be0 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c813170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc66ffd700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c812c30 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fcc580129e0 tx=0x7fcc58012eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795fcf349cbbc911 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc46ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1676798179 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc58012710 con 0x55d76c804be0 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc500af020 con 0x55d76c804be0 2023-07-29T16:27:56.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1676798179 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcc58000e40 con 0x55d76c804be0 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.366+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1676798179 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc580118d0 con 0x55d76c804be0 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 10 client.29186 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55d76c5aa250 con 0x55d76c804be0 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1676798179 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc5803d020 con 0x55d76c804be0 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1676798179 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fcc58004710 con 0x55d76c804be0 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 10 client.29186 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:56.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 10 client.29186 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 20 client.29186 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c86e160 0x55d76c870570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 4 client.29186 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d76c74ed40 con 0x55d76c86e160 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.367+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c86e160 0x55d76c870570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc46ffd700 10 client.29186 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc5ffff700 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c86e160 0x55d76c870570 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f52aaf71b621da92 server_cookie=4eacc2cf25a19422 in_seq=0 out_seq=0 2023-07-29T16:27:56.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc46ffd700 1 -- 172.21.15.31:0/1676798179 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d76c74ed40 con 0x55d76c86e160 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc46ffd700 10 client.29186 handle_command_reply: tid=0 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 client.29186 shutdown 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 msgr2=0x55d76c812c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c812c30 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fcc580129e0 tx=0x7fcc58012eb0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 shutdown_connections 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d76c8062f0 0x55d76c86c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55d76c86e160 0x55d76c870570 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d76c804be0 0x55d76c812c30 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc8123eb80 1 --2- 172.21.15.31:0/1676798179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76c805530 0x55d76c813170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 >> 172.21.15.31:0/1676798179 conn(0x55d76c7fa3e0 msgr2=0x55d76c809230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.368+0000 7fcc46ffd700 0 client.29186 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 shutdown_connections 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc8123eb80 1 -- 172.21.15.31:0/1676798179 wait complete. 2023-07-29T16:27:56.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc8123eb80 20 client.29186 trim_cache size 0 max 16384 2023-07-29T16:27:56.376 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:27:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc540a0910 msgr2=0x7fcc540a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc540a0910 0x7fcc540a2dc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fcc60009040 tx=0x7fcc600094b0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 msgr2=0x7fcc6819cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.369+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 0x7fcc6819cbe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000d950 tx=0x7fcc5000dd10 comp rx=0 tx=0).stop 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 shutdown_connections 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcc540a0910 0x7fcc540a2dc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc681040d0 0x7fcc681a3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc68102550 0x7fcc6819c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 --2- 172.21.15.31:0/1798454379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcc68103740 0x7fcc6819cbe0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 >> 172.21.15.31:0/1798454379 conn(0x7fcc680fdb90 msgr2=0x7fcc68078da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 shutdown_connections 2023-07-29T16:27:56.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.370+0000 7fcc6d9ff700 1 -- 172.21.15.31:0/1798454379 wait complete. 2023-07-29T16:27:56.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:27:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 -- 172.21.15.31:0/864848940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8100320 msgr2=0x7f39d8100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 --2- 172.21.15.31:0/864848940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8100320 0x7f39d8100730 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f39c8009990 tx=0x7f39c8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 -- 172.21.15.31:0/864848940 shutdown_connections 2023-07-29T16:27:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 --2- 172.21.15.31:0/864848940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101ea0 0x7f39d8108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 --2- 172.21.15.31:0/864848940 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8100320 0x7f39d8100730 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 --2- 172.21.15.31:0/864848940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8101510 0x7f39d8101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.614+0000 7f39de1ff700 1 -- 172.21.15.31:0/864848940 >> 172.21.15.31:0/864848940 conn(0x7f39d80fb920 msgr2=0x7f39d80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.615+0000 7f39de1ff700 1 -- 172.21.15.31:0/864848940 shutdown_connections 2023-07-29T16:27:56.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.615+0000 7f39de1ff700 1 -- 172.21.15.31:0/864848940 wait complete. 2023-07-29T16:27:56.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 Processor -- start 2023-07-29T16:27:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 -- start start 2023-07-29T16:27:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8100320 0x7f39d8074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 0x7f39d8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8101ea0 0x7f39d8073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d8073e20 con 0x7f39d8101ea0 2023-07-29T16:27:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d8073f90 con 0x7f39d8101510 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39de1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d81a0130 con 0x7f39d8100320 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.616+0000 7f39d6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 0x7f39d8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8100320 0x7f39d8074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 0x7f39d8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50440/0 (socket says 172.21.15.31:50440) 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d6ffd700 1 -- 172.21.15.31:0/798253836 learned_addr learned my addr 172.21.15.31:0/798253836 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d7fff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8101ea0 0x7f39d8073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 -- 172.21.15.31:0/798253836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 msgr2=0x7f39d8073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 0x7f39d8073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 -- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8101ea0 msgr2=0x7f39d8073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8101ea0 0x7f39d8073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 -- 172.21.15.31:0/798253836 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c80095f0 con 0x7f39d8100320 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d7fff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8101ea0 0x7f39d8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 0x7f39d8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d77fe700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8100320 0x7f39d8074b00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f39c800b620 tx=0x7f39c8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d4ff9700 1 -- 172.21.15.31:0/798253836 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c801d070 con 0x7f39d8100320 2023-07-29T16:27:56.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.617+0000 7f39d4ff9700 1 -- 172.21.15.31:0/798253836 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c8021410 con 0x7f39d8100320 2023-07-29T16:27:56.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.618+0000 7f39d4ff9700 1 -- 172.21.15.31:0/798253836 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c8017650 con 0x7f39d8100320 2023-07-29T16:27:56.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.618+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d81a03b0 con 0x7f39d8100320 2023-07-29T16:27:56.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.618+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d81a0780 con 0x7f39d8100320 2023-07-29T16:27:56.620 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.621+0000 7f39d4ff9700 1 -- 172.21.15.31:0/798253836 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f39c8021810 con 0x7f39d8100320 2023-07-29T16:27:56.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.623+0000 7f39d4ff9700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39c00a0810 0x7f39c00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.623+0000 7f39f1c0cb80 1 Processor -- start 2023-07-29T16:27:56.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.623+0000 7f39d4ff9700 1 -- 172.21.15.31:0/798253836 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39c80ccf00 con 0x7f39d8100320 2023-07-29T16:27:56.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.623+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39c00a0810 0x7f39c00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.623+0000 7f39f1c0cb80 1 -- start start 2023-07-29T16:27:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b2a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b2b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b2bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69a79a10 con 0x55ff69b2ad90 2023-07-29T16:27:56.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69a773d0 con 0x55ff69b2a440 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69a72cc0 con 0x55ff69b2bad0 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b2a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b2a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50450/0 (socket says 172.21.15.31:50450) 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 -- 172.21.15.31:0/2107398953 learned_addr learned my addr 172.21.15.31:0/2107398953 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d7fff700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b2bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b2b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 -- 172.21.15.31:0/2107398953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 msgr2=0x55ff69b2bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b2bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 -- 172.21.15.31:0/2107398953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 msgr2=0x55ff69b2b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b2b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 -- 172.21.15.31:0/2107398953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff698a0de0 con 0x55ff69b2a440 2023-07-29T16:27:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d7fff700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b2bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:27:56.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39c00a0810 0x7f39c00a2cc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f39cc011550 tx=0x7f39cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.624+0000 7f39d77fe700 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b2a850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f39c8096b30 tx=0x7f39c809a240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ad433b837d07522 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.625+0000 7f39bdffb700 1 -- 172.21.15.31:0/2107398953 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c809e660 con 0x55ff69b2a440 2023-07-29T16:27:56.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.625+0000 7f39bdffb700 1 -- 172.21.15.31:0/2107398953 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c80a29f0 con 0x55ff69b2a440 2023-07-29T16:27:56.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.625+0000 7f39bdffb700 1 -- 172.21.15.31:0/2107398953 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c8099d60 con 0x55ff69b2a440 2023-07-29T16:27:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2107398953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 msgr2=0x55ff69b2a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b2a850 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f39c8096b30 tx=0x7f39c809a240 comp rx=0 tx=0).stop 2023-07-29T16:27:56.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2107398953 shutdown_connections 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b2bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b2a850 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2107398953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b2b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2107398953 >> 172.21.15.31:0/2107398953 conn(0x55ff69b173e0 msgr2=0x55ff69b1c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2107398953 shutdown_connections 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.626+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2107398953 wait complete. 2023-07-29T16:27:56.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.627+0000 7f39f1c0cb80 1 -- start start 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.627+0000 7f39f1c0cb80 1 Processor -- start 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.627+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b33c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b341c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b3b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69a0f510 con 0x55ff69b2ad90 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69a0f1b0 con 0x55ff69b2a440 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff698f6920 con 0x55ff69b2bad0 2023-07-29T16:27:56.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39f1c0cb80 10 client.0 _ll_register_callbacks cb 0x55ff69a18620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b33c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b33c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50452/0 (socket says 172.21.15.31:50452) 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 -- 172.21.15.31:0/254486306 learned_addr learned my addr 172.21.15.31:0/254486306 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d7fff700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b3b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 -- 172.21.15.31:0/254486306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 msgr2=0x55ff69b3b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b3b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 -- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 msgr2=0x55ff69b341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 -- 172.21.15.31:0/254486306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d81a0780 con 0x55ff69b2a440 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39d77fe700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b33c80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f39c80a24c0 tx=0x7f39c809a0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0a0f003895a8197 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.628+0000 7f39b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:56.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39b6ffd700 1 -- 172.21.15.31:0/254486306 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c80a2560 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39b6ffd700 1 -- 172.21.15.31:0/254486306 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39c80a26c0 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff69a9dd10 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39b6ffd700 1 -- 172.21.15.31:0/254486306 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39c809e170 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 10 client.29198 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ff69889c60 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39b6ffd700 1 -- 172.21.15.31:0/254486306 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ff69a9dd10 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39b6ffd700 1 -- 172.21.15.31:0/254486306 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f39c8099590 con 0x55ff69b2a440 2023-07-29T16:27:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 10 client.29198 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 10 client.29198 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 20 client.29198 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b3f5a0 0x55ff69b419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 4 client.29198 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.629+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff69af0d70 con 0x55ff69b3f5a0 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.630+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b3f5a0 0x55ff69b419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.630+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b3f5a0 0x55ff69b419b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4339404cba75072a server_cookie=a1f322e03b23680f in_seq=0 out_seq=0 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.630+0000 7f39b6ffd700 10 client.29198 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39b6ffd700 1 -- 172.21.15.31:0/254486306 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ff69af0d70 con 0x55ff69b3f5a0 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39b6ffd700 10 client.29198 handle_command_reply: tid=0 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 client.29198 shutdown 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 msgr2=0x55ff69b33c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b33c80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f39c80a24c0 tx=0x7f39c809a0c0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 shutdown_connections 2023-07-29T16:27:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b3f5a0 0x55ff69b419b0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b3b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b33c80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39b6ffd700 0 client.29198 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/254486306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 >> 172.21.15.31:0/254486306 conn(0x55ff69b173e0 msgr2=0x55ff69b1c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 shutdown_connections 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.631+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/254486306 wait complete. 2023-07-29T16:27:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.632+0000 7f39f1c0cb80 20 client.29198 trim_cache size 0 max 16384 2023-07-29T16:27:56.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.646+0000 7f39f1c0cb80 1 Processor -- start 2023-07-29T16:27:56.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.646+0000 7f39f1c0cb80 1 -- start start 2023-07-29T16:27:56.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b32980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b3b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b1c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69a9dd10 con 0x55ff69b2ad90 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69b34e70 con 0x55ff69b2a440 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69b19460 con 0x55ff69b2bad0 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b32980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b3b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b32980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50468/0 (socket says 172.21.15.31:50468) 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d77fe700 1 -- 172.21.15.31:0/3186325634 learned_addr learned my addr 172.21.15.31:0/3186325634 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d7fff700 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b1c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d6ffd700 1 -- 172.21.15.31:0/3186325634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 msgr2=0x55ff69b1c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b1c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d6ffd700 1 -- 172.21.15.31:0/3186325634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 msgr2=0x55ff69b32980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b32980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d6ffd700 1 -- 172.21.15.31:0/3186325634 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff698c7250 con 0x55ff69b2ad90 2023-07-29T16:27:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d7fff700 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b1c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.647+0000 7f39d77fe700 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b32980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.648+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b3b1c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x55ff69b41f90 tx=0x7f39cc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ebd14240409479 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.648+0000 7f39be7fc700 1 -- 172.21.15.31:0/3186325634 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc021040 con 0x55ff69b2ad90 2023-07-29T16:27:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.648+0000 7f39be7fc700 1 -- 172.21.15.31:0/3186325634 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39cc004cf0 con 0x55ff69b2ad90 2023-07-29T16:27:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.648+0000 7f39be7fc700 1 -- 172.21.15.31:0/3186325634 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc005020 con 0x55ff69b2ad90 2023-07-29T16:27:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/3186325634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 msgr2=0x55ff69b3b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b3b1c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x55ff69b41f90 tx=0x7f39cc014000 comp rx=0 tx=0).stop 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/3186325634 shutdown_connections 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2bad0 0x55ff69b1c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2a440 0x55ff69b32980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/3186325634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2ad90 0x55ff69b3b1c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/3186325634 >> 172.21.15.31:0/3186325634 conn(0x55ff69b173e0 msgr2=0x55ff69b1c870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/3186325634 shutdown_connections 2023-07-29T16:27:56.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/3186325634 wait complete. 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.649+0000 7f39f1c0cb80 1 -- start start 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 Processor -- start 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 0x55ff69b3aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2ad90 0x55ff69b3aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2bad0 0x55ff69b37580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69b37b60 con 0x55ff69b2a440 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69b37cd0 con 0x55ff69b2bad0 2023-07-29T16:27:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff69b37e90 con 0x55ff69b2ad90 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39f1c0cb80 10 client.0 _ll_register_callbacks cb 0x55ff69a18620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 0x55ff69b3aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39d7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2bad0 0x55ff69b37580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39d7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2bad0 0x55ff69b37580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50478/0 (socket says 172.21.15.31:50478) 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39d77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 0x55ff69b3aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49636/0 (socket says 172.21.15.31:49636) 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39d6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2ad90 0x55ff69b3aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.650+0000 7f39d7fff700 1 -- 172.21.15.31:0/2986483317 learned_addr learned my addr 172.21.15.31:0/2986483317 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39d77fe700 1 -- 172.21.15.31:0/2986483317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2ad90 msgr2=0x55ff69b3aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39d77fe700 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2ad90 0x55ff69b3aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39d77fe700 1 -- 172.21.15.31:0/2986483317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2bad0 msgr2=0x55ff69b37580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39d77fe700 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2bad0 0x55ff69b37580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39d77fe700 1 -- 172.21.15.31:0/2986483317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff69889c60 con 0x55ff69b2a440 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39d77fe700 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 0x55ff69b3aab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f39c8096b30 tx=0x7f39c8005a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bb545cb253950 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39b6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:27:56.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39b6ffd700 1 -- 172.21.15.31:0/2986483317 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c800fad0 con 0x55ff69b2a440 2023-07-29T16:27:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39b6ffd700 1 -- 172.21.15.31:0/2986483317 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f39c800fc30 con 0x55ff69b2a440 2023-07-29T16:27:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39c80a29f0 con 0x55ff69b2a440 2023-07-29T16:27:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39b6ffd700 1 -- 172.21.15.31:0/2986483317 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39c800fdf0 con 0x55ff69b2a440 2023-07-29T16:27:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39f1c0cb80 10 client.19095 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.651+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f39cc017070 con 0x55ff69b2a440 2023-07-29T16:27:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39b6ffd700 1 -- 172.21.15.31:0/2986483317 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39c80a29f0 con 0x55ff69b2a440 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39b6ffd700 1 -- 172.21.15.31:0/2986483317 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f39c809e170 con 0x55ff69b2a440 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39f1c0cb80 10 client.19095 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39f1c0cb80 10 client.19095 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39f1c0cb80 20 client.19095 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b92270 0x55ff69b94680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39f1c0cb80 4 client.19095 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:27:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ff69a6bd40 con 0x55ff69b92270 2023-07-29T16:27:56.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.652+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b92270 0x55ff69b94680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39d6ffd700 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b92270 0x55ff69b94680 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=affee292cda17936 server_cookie=de481bc727145e4 in_seq=0 out_seq=0 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39b6ffd700 10 client.19095 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39b6ffd700 1 -- 172.21.15.31:0/2986483317 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ff69a6bd40 con 0x55ff69b92270 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39b6ffd700 10 client.19095 handle_command_reply: tid=0 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 client.19095 shutdown 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 msgr2=0x55ff69b3aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 0x55ff69b3aab0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f39c8096b30 tx=0x7f39c8005a90 comp rx=0 tx=0).stop 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 shutdown_connections 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55ff69b92270 0x55ff69b94680 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff69b2bad0 0x55ff69b37580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ff69b2a440 0x55ff69b3aab0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39b6ffd700 0 client.19095 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 --2- 172.21.15.31:0/2986483317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ff69b2ad90 0x55ff69b3aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 >> 172.21.15.31:0/2986483317 conn(0x55ff69b173e0 msgr2=0x55ff69b31c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 shutdown_connections 2023-07-29T16:27:56.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.653+0000 7f39f1c0cb80 1 -- 172.21.15.31:0/2986483317 wait complete. 2023-07-29T16:27:56.661 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39f1c0cb80 20 client.19095 trim_cache size 0 max 16384 2023-07-29T16:27:56.661 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:27:56.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39c00a0810 msgr2=0x7f39c00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39c00a0810 0x7f39c00a2cc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f39cc011550 tx=0x7f39cc009260 comp rx=0 tx=0).stop 2023-07-29T16:27:56.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8100320 msgr2=0x7f39d8074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8100320 0x7f39d8074b00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f39c800b620 tx=0x7f39c8004730 comp rx=0 tx=0).stop 2023-07-29T16:27:56.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 shutdown_connections 2023-07-29T16:27:56.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39c00a0810 0x7f39c00a2cc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39d8101ea0 0x7f39d8073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39d8100320 0x7f39d8074b00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 --2- 172.21.15.31:0/798253836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39d8101510 0x7f39d8073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 >> 172.21.15.31:0/798253836 conn(0x7f39d80fb920 msgr2=0x7f39d80fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.654+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 shutdown_connections 2023-07-29T16:27:56.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.655+0000 7f39de1ff700 1 -- 172.21.15.31:0/798253836 wait complete. 2023-07-29T16:27:56.669 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:56 smithi031 ceph-mon[95504]: pgmap v745: 97 pgs: 97 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 286 op/s 2023-07-29T16:27:56.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:27:56.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:27:56.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:27:56.670 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:56 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/603051585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:27:56.672 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:27:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 -- 172.21.15.31:0/3152489414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 msgr2=0x7f04bc073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 --2- 172.21.15.31:0/3152489414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc073bc0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098f0 tx=0x7f04ac009c00 comp rx=0 tx=0).stop 2023-07-29T16:27:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 -- 172.21.15.31:0/3152489414 shutdown_connections 2023-07-29T16:27:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 --2- 172.21.15.31:0/3152489414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04bc1081c0 0x7f04bc10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 --2- 172.21.15.31:0/3152489414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc074d30 0x7f04bc073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 --2- 172.21.15.31:0/3152489414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc073bc0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.886+0000 7f04c19ff700 1 -- 172.21.15.31:0/3152489414 >> 172.21.15.31:0/3152489414 conn(0x7f04bc0fb970 msgr2=0x7f04bc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.887+0000 7f04c19ff700 1 -- 172.21.15.31:0/3152489414 shutdown_connections 2023-07-29T16:27:56.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.887+0000 7f04c19ff700 1 -- 172.21.15.31:0/3152489414 wait complete. 2023-07-29T16:27:56.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 Processor -- start 2023-07-29T16:27:56.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 -- start start 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc1982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04bc074d30 0x7f04bc198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc1081c0 0x7f04bc19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc19fe50 con 0x7f04bc074d30 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc19ff90 con 0x7f04bc073750 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.888+0000 7f04c19ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04bc1a0120 con 0x7f04bc1081c0 2023-07-29T16:27:56.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04baffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc1982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04baffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc1982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50502/0 (socket says 172.21.15.31:50502) 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04baffd700 1 -- 172.21.15.31:0/150845588 learned_addr learned my addr 172.21.15.31:0/150845588 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04bb7fe700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc1081c0 0x7f04bc19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04bc074d30 0x7f04bc198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04bb7fe700 1 -- 172.21.15.31:0/150845588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 msgr2=0x7f04bc1982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04bb7fe700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc1982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04bb7fe700 1 -- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04bc074d30 msgr2=0x7f04bc198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04bb7fe700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04bc074d30 0x7f04bc198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04bb7fe700 1 -- 172.21.15.31:0/150845588 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009c40 con 0x7f04bc1081c0 2023-07-29T16:27:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.889+0000 7f04baffd700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.890+0000 7f04bb7fe700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc1081c0 0x7f04bc19f880 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f04b400efb0 tx=0x7f04b400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.890+0000 7f04c09fd700 1 -- 172.21.15.31:0/150845588 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4003a00 con 0x7f04bc1081c0 2023-07-29T16:27:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.890+0000 7f04c09fd700 1 -- 172.21.15.31:0/150845588 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04b400f040 con 0x7f04bc1081c0 2023-07-29T16:27:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.890+0000 7f04c09fd700 1 -- 172.21.15.31:0/150845588 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b4004540 con 0x7f04bc1081c0 2023-07-29T16:27:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.890+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac009520 con 0x7f04bc1081c0 2023-07-29T16:27:56.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.890+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04bc1a0730 con 0x7f04bc1081c0 2023-07-29T16:27:56.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.893+0000 7f04c09fd700 1 -- 172.21.15.31:0/150845588 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f04b4003bf0 con 0x7f04bc1081c0 2023-07-29T16:27:56.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.895+0000 7f04d52d1b80 1 Processor -- start 2023-07-29T16:27:56.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.895+0000 7f04d52d1b80 1 -- start start 2023-07-29T16:27:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c732cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c732dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c732e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c7284a10 con 0x55a6c732d530 2023-07-29T16:27:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c72823d0 con 0x55a6c732cbe0 2023-07-29T16:27:56.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c727dcc0 con 0x55a6c732e2f0 2023-07-29T16:27:56.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c732cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c732cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50514/0 (socket says 172.21.15.31:50514) 2023-07-29T16:27:56.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2280636337 learned_addr learned my addr 172.21.15.31:0/2280636337 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04c09fd700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f04a80a0790 0x7f04a80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04baffd700 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c732dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.896+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c732e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04c09fd700 1 -- 172.21.15.31:0/150845588 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f04b40cb700 con 0x7f04bc1081c0 2023-07-29T16:27:56.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 962214 8.94 MB/sec execute 461 sec latency 21.548 ms 2023-07-29T16:27:56.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2280636337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 msgr2=0x55a6c732e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c732e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2280636337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 msgr2=0x55a6c732dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c732dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2280636337 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6c70abde0 con 0x55a6c732cbe0 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04bb7fe700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f04a80a0790 0x7f04a80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c732cff0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098f0 tx=0x7f04ac019040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c47fe1183d3cad0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04b1ffb700 1 -- 172.21.15.31:0/2280636337 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00f460 con 0x55a6c732cbe0 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04b1ffb700 1 -- 172.21.15.31:0/2280636337 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04ac00ba20 con 0x55a6c732cbe0 2023-07-29T16:27:56.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.897+0000 7f04b1ffb700 1 -- 172.21.15.31:0/2280636337 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac005290 con 0x55a6c732cbe0 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2280636337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 msgr2=0x55a6c732cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04bb7fe700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f04a80a0790 0x7f04a80a2c40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f04b4003f40 tx=0x7f04b4003b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c732cff0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098f0 tx=0x7f04ac019040 comp rx=0 tx=0).stop 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2280636337 shutdown_connections 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c732e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c732cff0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2280636337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c732dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2280636337 >> 172.21.15.31:0/2280636337 conn(0x55a6c73223e0 msgr2=0x55a6c7327990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2280636337 shutdown_connections 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.898+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2280636337 wait complete. 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.899+0000 7f04d52d1b80 1 -- start start 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.899+0000 7f04d52d1b80 1 Processor -- start 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c7336800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7336d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c721a510 con 0x55a6c732d530 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c721a1b0 con 0x55a6c732cbe0 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c7101920 con 0x55a6c732e2f0 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04d52d1b80 10 client.0 _ll_register_callbacks cb 0x55a6c7223620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c7336800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c7336800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50518/0 (socket says 172.21.15.31:50518) 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 -- 172.21.15.31:0/4164718389 learned_addr learned my addr 172.21.15.31:0/4164718389 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04baffd700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7336d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 -- 172.21.15.31:0/4164718389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 msgr2=0x55a6c733dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 -- 172.21.15.31:0/4164718389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 msgr2=0x55a6c7336d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7336d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 -- 172.21.15.31:0/4164718389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac01f040 con 0x55a6c732cbe0 2023-07-29T16:27:56.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c7336800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098c0 tx=0x7f04ac009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6af45083f00baa9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.900+0000 7f049affd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:56.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f049affd700 1 -- 172.21.15.31:0/4164718389 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac02c020 con 0x55a6c732cbe0 2023-07-29T16:27:56.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f049affd700 1 -- 172.21.15.31:0/4164718389 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f04ac00f5a0 con 0x55a6c732cbe0 2023-07-29T16:27:56.906 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a6c72a8d10 con 0x55a6c732cbe0 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f049affd700 1 -- 172.21.15.31:0/4164718389 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04ac00b560 con 0x55a6c732cbe0 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f04d52d1b80 10 client.29216 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a6c7094c60 con 0x55a6c732cbe0 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f049affd700 1 -- 172.21.15.31:0/4164718389 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a6c72a8d10 con 0x55a6c732cbe0 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.901+0000 7f049affd700 1 -- 172.21.15.31:0/4164718389 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f04ac015020 con 0x55a6c732cbe0 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04d52d1b80 10 client.29216 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04d52d1b80 10 client.29216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:56.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04d52d1b80 20 client.29216 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c73420b0 0x55a6c73444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04d52d1b80 4 client.29216 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6c72fbd70 con 0x55a6c73420b0 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04baffd700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c73420b0 0x55a6c73444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f04baffd700 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c73420b0 0x55a6c73444c0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9798dca90b1caae5 server_cookie=25f4afa82e9e68cb in_seq=0 out_seq=0 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.902+0000 7f049affd700 10 client.29216 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:56.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.903+0000 7f049affd700 1 -- 172.21.15.31:0/4164718389 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a6c72fbd70 con 0x55a6c73420b0 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.903+0000 7f049affd700 10 client.29216 handle_command_reply: tid=0 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.903+0000 7f04d52d1b80 1 client.29216 shutdown 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.903+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 msgr2=0x55a6c7336800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.903+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c7336800 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0098c0 tx=0x7f04ac009c00 comp rx=0 tx=0).stop 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 shutdown_connections 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c73420b0 0x55a6c73444c0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732cbe0 0x55a6c7336800 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/4164718389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7336d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f049affd700 0 client.29216 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 >> 172.21.15.31:0/4164718389 conn(0x55a6c73223e0 msgr2=0x55a6c7327060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 shutdown_connections 2023-07-29T16:27:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 1 -- 172.21.15.31:0/4164718389 wait complete. 2023-07-29T16:27:56.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.904+0000 7f04d52d1b80 20 client.29216 trim_cache size 0 max 16384 2023-07-29T16:27:56.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 Processor -- start 2023-07-29T16:27:56.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 -- start start 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 0x55a6c733bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7327060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732e2f0 0x55a6c7334720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c732dbf0 con 0x55a6c732d530 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c73379f0 con 0x55a6c732e2f0 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c731f6e0 con 0x55a6c732cbe0 2023-07-29T16:27:56.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 0x55a6c733bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 0x55a6c733bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59478/0 (socket says 172.21.15.31:59478) 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2420390485 learned_addr learned my addr 172.21.15.31:0/2420390485 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732e2f0 0x55a6c7334720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.918+0000 7f04baffd700 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7327060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2420390485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732e2f0 msgr2=0x55a6c7334720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732e2f0 0x55a6c7334720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2420390485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 msgr2=0x55a6c7327060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7327060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2420390485 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6c70d2250 con 0x55a6c732cbe0 2023-07-29T16:27:56.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 0x55a6c733bd30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009810 tx=0x7f04ac005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8d2ccefebe465e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04b27fc700 1 -- 172.21.15.31:0/2420390485 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac02c030 con 0x55a6c732cbe0 2023-07-29T16:27:56.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04b27fc700 1 -- 172.21.15.31:0/2420390485 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04ac02d040 con 0x55a6c732cbe0 2023-07-29T16:27:56.922 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.919+0000 7f04b27fc700 1 -- 172.21.15.31:0/2420390485 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ac00f7c0 con 0x55a6c732cbe0 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2420390485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 msgr2=0x55a6c733bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 0x55a6c733bd30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009810 tx=0x7f04ac005dc0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2420390485 shutdown_connections 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732e2f0 0x55a6c7334720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732cbe0 0x55a6c733bd30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2420390485 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732d530 0x55a6c7327060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.923 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2420390485 >> 172.21.15.31:0/2420390485 conn(0x55a6c73223e0 msgr2=0x55a6c73354b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2420390485 shutdown_connections 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2420390485 wait complete. 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.920+0000 7f04d52d1b80 1 -- start start 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 Processor -- start 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732cbe0 0x55a6c7339f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 0x55a6c733a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c733b8f0 con 0x55a6c732cbe0 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c733ba60 con 0x55a6c732d530 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6c733bc20 con 0x55a6c732e2f0 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04d52d1b80 10 client.0 _ll_register_callbacks cb 0x55a6c7223620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732cbe0 0x55a6c7339f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04baffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 0x55a6c733a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04ba7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732cbe0 0x55a6c7339f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49682/0 (socket says 172.21.15.31:49682) 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04baffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 0x55a6c733a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50532/0 (socket says 172.21.15.31:50532) 2023-07-29T16:27:56.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.921+0000 7f04ba7fc700 1 -- 172.21.15.31:0/2823694239 learned_addr learned my addr 172.21.15.31:0/2823694239 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04baffd700 1 -- 172.21.15.31:0/2823694239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 msgr2=0x55a6c733b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04baffd700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04baffd700 1 -- 172.21.15.31:0/2823694239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732cbe0 msgr2=0x55a6c7339f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04baffd700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732cbe0 0x55a6c7339f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04baffd700 1 -- 172.21.15.31:0/2823694239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6c7094c60 con 0x55a6c732d530 2023-07-29T16:27:56.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04baffd700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 0x55a6c733a470 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x55a6c7344d30 tx=0x7f04a4004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3da515f34c4ba5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f049affd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f049affd700 1 -- 172.21.15.31:0/2823694239 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4017050 con 0x55a6c732d530 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04ac00ba20 con 0x55a6c732d530 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.922+0000 7f049affd700 1 -- 172.21.15.31:0/2823694239 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f04a4013e00 con 0x55a6c732d530 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f049affd700 1 -- 172.21.15.31:0/2823694239 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04a40122e0 con 0x55a6c732d530 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 10 client.29222 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:56.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55a6c70aa2b0 con 0x55a6c732d530 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f049affd700 1 -- 172.21.15.31:0/2823694239 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04ac00ba20 con 0x55a6c732d530 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f049affd700 1 -- 172.21.15.31:0/2823694239 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f04a40037a0 con 0x55a6c732d530 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 10 client.29222 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 10 client.29222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 20 client.29222 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c739d6c0 0x55a6c739fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:56.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 4 client.29222 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:27:56.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a6c7276d40 con 0x55a6c739d6c0 2023-07-29T16:27:56.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.923+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c739d6c0 0x55a6c739fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:56.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f049affd700 10 client.29222 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:56.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04ba7fc700 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c739d6c0 0x55a6c739fad0 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66f463fed207af2 server_cookie=60eb6d10fcdee593 in_seq=0 out_seq=0 2023-07-29T16:27:56.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f049affd700 1 -- 172.21.15.31:0/2823694239 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a6c7276d40 con 0x55a6c739d6c0 2023-07-29T16:27:56.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f049affd700 10 client.29222 handle_command_reply: tid=0 2023-07-29T16:27:56.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 client.29222 shutdown 2023-07-29T16:27:56.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 msgr2=0x55a6c733a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 0x55a6c733a470 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x55a6c7344d30 tx=0x7f04a4004410 comp rx=0 tx=0).stop 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 shutdown_connections 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a6c739d6c0 0x55a6c739fad0 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a6c732e2f0 0x55a6c733b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a6c732cbe0 0x55a6c7339f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 --2- 172.21.15.31:0/2823694239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a6c732d530 0x55a6c733a470 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 >> 172.21.15.31:0/2823694239 conn(0x55a6c73223e0 msgr2=0x55a6c7327060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.924+0000 7f049affd700 0 client.29222 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 shutdown_connections 2023-07-29T16:27:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04d52d1b80 1 -- 172.21.15.31:0/2823694239 wait complete. 2023-07-29T16:27:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04d52d1b80 20 client.29222 trim_cache size 0 max 16384 2023-07-29T16:27:56.932 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:27:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f04a80a0790 msgr2=0x7f04a80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f04a80a0790 0x7f04a80a2c40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f04b4003f40 tx=0x7f04b4003b60 comp rx=0 tx=0).stop 2023-07-29T16:27:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc1081c0 msgr2=0x7f04bc19f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc1081c0 0x7f04bc19f880 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f04b400efb0 tx=0x7f04b400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 shutdown_connections 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f04a80a0790 0x7f04a80a2c40 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04bc1081c0 0x7f04bc19f880 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04bc073750 0x7f04bc1982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.925+0000 7f04c19ff700 1 --2- 172.21.15.31:0/150845588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04bc074d30 0x7f04bc198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.926+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 >> 172.21.15.31:0/150845588 conn(0x7f04bc0fb970 msgr2=0x7f04bc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.926+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 shutdown_connections 2023-07-29T16:27:56.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:56.926+0000 7f04c19ff700 1 -- 172.21.15.31:0/150845588 wait complete. 2023-07-29T16:27:57.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:56 smithi039 ceph-mon[128858]: pgmap v745: 97 pgs: 97 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 286 op/s 2023-07-29T16:27:57.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:27:57.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:27:57.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:27:57.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:56 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/603051585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:27:57.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:56 smithi156 ceph-mon[129245]: pgmap v745: 97 pgs: 97 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 2.5 MiB/s wr, 286 op/s 2023-07-29T16:27:57.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:27:57.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:27:57.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:27:57.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:56 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/603051585' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:27:57.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 963304 8.94 MB/sec execute 462 sec latency 14.193 ms 2023-07-29T16:27:57.944 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:27:57.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:27:58.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.175+0000 7f611cbff700 1 -- 172.21.15.31:0/1178259335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118101ea0 msgr2=0x7f6118108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.175+0000 7f611cbff700 1 --2- 172.21.15.31:0/1178259335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118101ea0 0x7f6118108760 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f611000b0a0 tx=0x7f611000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 -- 172.21.15.31:0/1178259335 shutdown_connections 2023-07-29T16:27:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 --2- 172.21.15.31:0/1178259335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118101ea0 0x7f6118108760 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 --2- 172.21.15.31:0/1178259335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6118100320 0x7f6118100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 --2- 172.21.15.31:0/1178259335 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6118101510 0x7f6118101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 -- 172.21.15.31:0/1178259335 >> 172.21.15.31:0/1178259335 conn(0x7f61180fb920 msgr2=0x7f61180fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 -- 172.21.15.31:0/1178259335 shutdown_connections 2023-07-29T16:27:58.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.176+0000 7f611cbff700 1 -- 172.21.15.31:0/1178259335 wait complete. 2023-07-29T16:27:58.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 Processor -- start 2023-07-29T16:27:58.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 -- start start 2023-07-29T16:27:58.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 0x7f6118198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6118101510 0x7f61181988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6118101ea0 0x7f611819f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611819ff00 con 0x7f6118101510 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61181a0040 con 0x7f6118100320 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.177+0000 7f611cbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61181a01d0 con 0x7f6118101ea0 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 0x7f6118198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 0x7f6118198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50538/0 (socket says 172.21.15.31:50538) 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 -- 172.21.15.31:0/3015729130 learned_addr learned my addr 172.21.15.31:0/3015729130 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f6116d9d700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6118101ea0 0x7f611819f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f6115d9b700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6118101510 0x7f61181988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 -- 172.21.15.31:0/3015729130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6118101ea0 msgr2=0x7f611819f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6118101ea0 0x7f611819f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 -- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6118101510 msgr2=0x7f61181988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6118101510 0x7f61181988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 -- 172.21.15.31:0/3015729130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6110009d00 con 0x7f6118100320 2023-07-29T16:27:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.178+0000 7f611659c700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 0x7f6118198380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f610800e970 tx=0x7f610800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.179+0000 7f61077fe700 1 -- 172.21.15.31:0/3015729130 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108009790 con 0x7f6118100320 2023-07-29T16:27:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.179+0000 7f61077fe700 1 -- 172.21.15.31:0/3015729130 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f610800c7e0 con 0x7f6118100320 2023-07-29T16:27:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.179+0000 7f61077fe700 1 -- 172.21.15.31:0/3015729130 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108003c70 con 0x7f6118100320 2023-07-29T16:27:58.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.179+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61181a04b0 con 0x7f6118100320 2023-07-29T16:27:58.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.179+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61181a09c0 con 0x7f6118100320 2023-07-29T16:27:58.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.182+0000 7f61077fe700 1 -- 172.21.15.31:0/3015729130 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f610800cc50 con 0x7f6118100320 2023-07-29T16:27:58.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.183+0000 7f6130462b80 1 Processor -- start 2023-07-29T16:27:58.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.183+0000 7f6130462b80 1 -- start start 2023-07-29T16:27:58.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f61077fe700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61000a0810 0x7f61000a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6130462b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661bf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6130462b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6130462b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661c07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6130462b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562066116a90 con 0x5620661c0370 2023-07-29T16:27:58.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6130462b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562066114450 con 0x5620661bf5b0 2023-07-29T16:27:58.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6130462b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56206610fd40 con 0x5620661bec60 2023-07-29T16:27:58.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f61077fe700 1 -- 172.21.15.31:0/3015729130 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6108014070 con 0x7f6118100320 2023-07-29T16:27:58.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661bfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50550/0 (socket says 172.21.15.31:50550) 2023-07-29T16:27:58.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 -- 172.21.15.31:0/3163643540 learned_addr learned my addr 172.21.15.31:0/3163643540 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f611659c700 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661bf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6116d9d700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61000a0810 0x7f61000a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6116d9d700 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661c07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 -- 172.21.15.31:0/3163643540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 msgr2=0x5620661bf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661bf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 -- 172.21.15.31:0/3163643540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 msgr2=0x5620661c07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.184+0000 7f6115d9b700 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661c07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.185+0000 7f6115d9b700 1 -- 172.21.15.31:0/3163643540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562065f3de60 con 0x5620661bf5b0 2023-07-29T16:27:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.185+0000 7f6115d9b700 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661bfde0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f610c009d80 tx=0x7f610c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f5044c120655365 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.185+0000 7f6116d9d700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61000a0810 0x7f61000a2cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f611000b780 tx=0x7f6110006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.185+0000 7f6104ff9700 1 -- 172.21.15.31:0/3163643540 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c004230 con 0x5620661bf5b0 2023-07-29T16:27:58.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.185+0000 7f6104ff9700 1 -- 172.21.15.31:0/3163643540 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f610c004390 con 0x5620661bf5b0 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.185+0000 7f6104ff9700 1 -- 172.21.15.31:0/3163643540 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c019880 con 0x5620661bf5b0 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 -- 172.21.15.31:0/3163643540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 msgr2=0x5620661bfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661bfde0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f610c009d80 tx=0x7f610c011720 comp rx=0 tx=0).stop 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 -- 172.21.15.31:0/3163643540 shutdown_connections 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661c07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661bf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 --2- 172.21.15.31:0/3163643540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661bfde0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 -- 172.21.15.31:0/3163643540 >> 172.21.15.31:0/3163643540 conn(0x5620661b4460 msgr2=0x5620661b9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 -- 172.21.15.31:0/3163643540 shutdown_connections 2023-07-29T16:27:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.186+0000 7f6130462b80 1 -- 172.21.15.31:0/3163643540 wait complete. 2023-07-29T16:27:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.187+0000 7f6130462b80 1 -- start start 2023-07-29T16:27:58.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.187+0000 7f6130462b80 1 Processor -- start 2023-07-29T16:27:58.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661c8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620660ac590 con 0x5620661c0370 2023-07-29T16:27:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620660ac230 con 0x5620661bf5b0 2023-07-29T16:27:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562065f939a0 con 0x5620661bec60 2023-07-29T16:27:58.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6130462b80 10 client.0 _ll_register_callbacks cb 0x5620660b56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661c8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50562/0 (socket says 172.21.15.31:50562) 2023-07-29T16:27:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 -- 172.21.15.31:0/182792785 learned_addr learned my addr 172.21.15.31:0/182792785 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f611659c700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6116d9d700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 -- 172.21.15.31:0/182792785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 msgr2=0x5620661c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 -- 172.21.15.31:0/182792785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 msgr2=0x5620661cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f6115d9b700 1 -- 172.21.15.31:0/182792785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610c0099e0 con 0x5620661bf5b0 2023-07-29T16:27:58.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.188+0000 7f611659c700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f6115d9b700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661c8d70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f610c009770 tx=0x7f610c0038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc592ff6b57a8b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f60fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f60fdffb700 1 -- 172.21.15.31:0/182792785 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c005450 con 0x5620661bf5b0 2023-07-29T16:27:58.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56206613ad90 con 0x5620661bf5b0 2023-07-29T16:27:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f60fdffb700 1 -- 172.21.15.31:0/182792785 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f610c005ce0 con 0x5620661bf5b0 2023-07-29T16:27:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f60fdffb700 1 -- 172.21.15.31:0/182792785 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f610c009ed0 con 0x5620661bf5b0 2023-07-29T16:27:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f6130462b80 10 client.29234 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:58.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.189+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562065f26ce0 con 0x5620661bf5b0 2023-07-29T16:27:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f60fdffb700 1 -- 172.21.15.31:0/182792785 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56206613ad90 con 0x5620661bf5b0 2023-07-29T16:27:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f60fdffb700 1 -- 172.21.15.31:0/182792785 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f610c021040 con 0x5620661bf5b0 2023-07-29T16:27:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f6130462b80 10 client.29234 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f6130462b80 10 client.29234 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f6130462b80 20 client.29234 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f6130462b80 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5620661d44e0 0x5620661d68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f6130462b80 4 client.29234 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:58.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56206618ddf0 con 0x5620661d44e0 2023-07-29T16:27:58.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.190+0000 7f611659c700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5620661d44e0 0x5620661d68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.191+0000 7f60fdffb700 10 client.29234 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.191+0000 7f611659c700 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5620661d44e0 0x5620661d68f0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1835a3527214c7d5 server_cookie=2704b4cc2fe1524a in_seq=0 out_seq=0 2023-07-29T16:27:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f60fdffb700 1 -- 172.21.15.31:0/182792785 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56206618ddf0 con 0x5620661d44e0 2023-07-29T16:27:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f60fdffb700 10 client.29234 handle_command_reply: tid=0 2023-07-29T16:27:58.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 client.29234 shutdown 2023-07-29T16:27:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 msgr2=0x5620661c8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661c8d70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f610c009770 tx=0x7f610c0038b0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 shutdown_connections 2023-07-29T16:27:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5620661d44e0 0x5620661d68f0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661c0370 0x5620661cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661bec60 0x5620661c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 --2- 172.21.15.31:0/182792785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bf5b0 0x5620661c8d70 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f60fdffb700 0 client.29234 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 >> 172.21.15.31:0/182792785 conn(0x5620661b4460 msgr2=0x5620661b90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.192+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 shutdown_connections 2023-07-29T16:27:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.193+0000 7f6130462b80 1 -- 172.21.15.31:0/182792785 wait complete. 2023-07-29T16:27:58.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.193+0000 7f6130462b80 20 client.29234 trim_cache size 0 max 16384 2023-07-29T16:27:58.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 Processor -- start 2023-07-29T16:27:58.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 -- start start 2023-07-29T16:27:58.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620661c94b0 con 0x5620661bf5b0 2023-07-29T16:27:58.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620661b1760 con 0x5620661bec60 2023-07-29T16:27:58.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6130462b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620661c6490 con 0x5620661c0370 2023-07-29T16:27:58.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f6115d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.212+0000 7f611659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50572/0 (socket says 172.21.15.31:50572) 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f6116d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 -- 172.21.15.31:0/3637068069 learned_addr learned my addr 172.21.15.31:0/3637068069 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 -- 172.21.15.31:0/3637068069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 msgr2=0x5620661c66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661c66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 -- 172.21.15.31:0/3637068069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 msgr2=0x5620661cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 -- 172.21.15.31:0/3637068069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620661c0a10 con 0x5620661bec60 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f6116d9d700 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f611659c700 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661d0640 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6108000c00 tx=0x7f6108095840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a452ec83f0bcba server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f61057fa700 1 -- 172.21.15.31:0/3637068069 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610809d400 con 0x5620661bec60 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f61057fa700 1 -- 172.21.15.31:0/3637068069 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61080a1820 con 0x5620661bec60 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.213+0000 7f61057fa700 1 -- 172.21.15.31:0/3637068069 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610805bb70 con 0x5620661bec60 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 -- 172.21.15.31:0/3637068069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 msgr2=0x5620661d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661d0640 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f6108000c00 tx=0x7f6108095840 comp rx=0 tx=0).stop 2023-07-29T16:27:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 -- 172.21.15.31:0/3637068069 shutdown_connections 2023-07-29T16:27:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661c66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661d0640 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 --2- 172.21.15.31:0/3637068069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 -- 172.21.15.31:0/3637068069 >> 172.21.15.31:0/3637068069 conn(0x5620661b4460 msgr2=0x5620661b99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 -- 172.21.15.31:0/3637068069 shutdown_connections 2023-07-29T16:27:58.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.214+0000 7f6130462b80 1 -- 172.21.15.31:0/3637068069 wait complete. 2023-07-29T16:27:58.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.215+0000 7f6130462b80 1 -- start start 2023-07-29T16:27:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.215+0000 7f6130462b80 1 Processor -- start 2023-07-29T16:27:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562066226040 con 0x5620661bf5b0 2023-07-29T16:27:58.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562066226180 con 0x5620661bec60 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620662262c0 con 0x5620661c0370 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6130462b80 10 client.0 _ll_register_callbacks cb 0x5620660b56a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50574/0 (socket says 172.21.15.31:50574) 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 -- 172.21.15.31:0/69422109 learned_addr learned my addr 172.21.15.31:0/69422109 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6115d9b700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6116d9d700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 -- 172.21.15.31:0/69422109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 msgr2=0x5620661ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 -- 172.21.15.31:0/69422109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 msgr2=0x5620661cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f611659c700 1 -- 172.21.15.31:0/69422109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6118109090 con 0x5620661bec60 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6115d9b700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:58.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.216+0000 7f6116d9d700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f611659c700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661cfcf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6108005950 tx=0x7f610809d4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ec743795d2b927 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f60fdffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f60fdffb700 1 -- 172.21.15.31:0/69422109 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108099630 con 0x5620661bec60 2023-07-29T16:27:58.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f610c004390 con 0x5620661bec60 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f60fdffb700 1 -- 172.21.15.31:0/69422109 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6108099790 con 0x5620661bec60 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f60fdffb700 1 -- 172.21.15.31:0/69422109 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61080988c0 con 0x5620661bec60 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f6130462b80 10 client.29240 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f61181a09c0 con 0x5620661bec60 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.217+0000 7f60fdffb700 1 -- 172.21.15.31:0/69422109 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f610c004390 con 0x5620661bec60 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f60fdffb700 1 -- 172.21.15.31:0/69422109 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f61080951c0 con 0x5620661bec60 2023-07-29T16:27:58.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6130462b80 10 client.29240 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6130462b80 10 client.29240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6130462b80 20 client.29240 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6130462b80 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562066227730 0x562066229b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6130462b80 4 client.29240 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562066108dc0 con 0x562066227730 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6115d9b700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562066227730 0x562066229b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f6115d9b700 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562066227730 0x562066229b10 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1f105b425997555 server_cookie=387d82c88e3524b3 in_seq=0 out_seq=0 2023-07-29T16:27:58.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.218+0000 7f60fdffb700 10 client.29240 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.220+0000 7f60fdffb700 1 -- 172.21.15.31:0/69422109 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562066108dc0 con 0x562066227730 2023-07-29T16:27:58.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.220+0000 7f60fdffb700 10 client.29240 handle_command_reply: tid=0 2023-07-29T16:27:58.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.220+0000 7f6130462b80 1 client.29240 shutdown 2023-07-29T16:27:58.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 msgr2=0x5620661cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661cfcf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6108005950 tx=0x7f610809d4b0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 shutdown_connections 2023-07-29T16:27:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562066227730 0x562066229b10 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5620661c0370 0x5620661ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5620661bec60 0x5620661cfcf0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 --2- 172.21.15.31:0/69422109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620661bf5b0 0x5620661cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f60fdffb700 0 client.29240 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.227 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 >> 172.21.15.31:0/69422109 conn(0x5620661b4460 msgr2=0x5620661d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 shutdown_connections 2023-07-29T16:27:58.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 1 -- 172.21.15.31:0/69422109 wait complete. 2023-07-29T16:27:58.228 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.221+0000 7f6130462b80 20 client.29240 trim_cache size 0 max 16384 2023-07-29T16:27:58.228 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:27:58.228 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:27:58.229 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "f4156fd2-bbca-49c1-a42f-93cc09910517", 2023-07-29T16:27:58.229 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:27:58.229 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:27:58.229 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61000a0810 msgr2=0x7f61000a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61000a0810 0x7f61000a2cc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f611000b780 tx=0x7f6110006000 comp rx=0 tx=0).stop 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 msgr2=0x7f6118198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 0x7f6118198380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f610800e970 tx=0x7f610800ec80 comp rx=0 tx=0).stop 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 shutdown_connections 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61000a0810 0x7f61000a2cc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6118101ea0 0x7f611819f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.230 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6118100320 0x7f6118198380 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 --2- 172.21.15.31:0/3015729130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6118101510 0x7f61181988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 >> 172.21.15.31:0/3015729130 conn(0x7f61180fb920 msgr2=0x7f61180fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 shutdown_connections 2023-07-29T16:27:58.231 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.222+0000 7f611cbff700 1 -- 172.21.15.31:0/3015729130 wait complete. 2023-07-29T16:27:58.241 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:57 smithi031 ceph-mon[95504]: pgmap v746: 97 pgs: 97 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.1 MiB/s wr, 219 op/s 2023-07-29T16:27:58.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:27:58.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:57 smithi039 ceph-mon[128858]: pgmap v746: 97 pgs: 97 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.1 MiB/s wr, 219 op/s 2023-07-29T16:27:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:57 smithi156 ceph-mon[129245]: pgmap v746: 97 pgs: 97 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 2.1 MiB/s wr, 219 op/s 2023-07-29T16:27:58.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 -- 172.21.15.31:0/1516676404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938101eb0 msgr2=0x7f5938108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 --2- 172.21.15.31:0/1516676404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938101eb0 0x7f5938108770 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f593000b0a0 tx=0x7f593000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 -- 172.21.15.31:0/1516676404 shutdown_connections 2023-07-29T16:27:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 --2- 172.21.15.31:0/1516676404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938101eb0 0x7f5938108770 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 --2- 172.21.15.31:0/1516676404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5938100330 0x7f5938100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 --2- 172.21.15.31:0/1516676404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5938101520 0x7f5938101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 -- 172.21.15.31:0/1516676404 >> 172.21.15.31:0/1516676404 conn(0x7f59380fb970 msgr2=0x7f59380fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 -- 172.21.15.31:0/1516676404 shutdown_connections 2023-07-29T16:27:58.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.461+0000 7f593f7ff700 1 -- 172.21.15.31:0/1516676404 wait complete. 2023-07-29T16:27:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.462+0000 7f593f7ff700 1 Processor -- start 2023-07-29T16:27:58.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 -- start start 2023-07-29T16:27:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 0x7f5938074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5938101520 0x7f5938073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5938101eb0 0x7f5938073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938073e20 con 0x7f5938101520 2023-07-29T16:27:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938073f90 con 0x7f5938100330 2023-07-29T16:27:58.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593f7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381a01d0 con 0x7f5938101eb0 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593d59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 0x7f5938074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593d59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 0x7f5938074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50600/0 (socket says 172.21.15.31:50600) 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593d59b700 1 -- 172.21.15.31:0/2859481576 learned_addr learned my addr 172.21.15.31:0/2859481576 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.463+0000 7f593cd9a700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5938101520 0x7f5938073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593dd9c700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5938101eb0 0x7f5938073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593d59b700 1 -- 172.21.15.31:0/2859481576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5938101eb0 msgr2=0x7f5938073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593d59b700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5938101eb0 0x7f5938073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593d59b700 1 -- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5938101520 msgr2=0x7f5938073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593d59b700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5938101520 0x7f5938073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593d59b700 1 -- 172.21.15.31:0/2859481576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5930009d00 con 0x7f5938100330 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593d59b700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 0x7f5938074b00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f593400e970 tx=0x7f593400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f592e7fc700 1 -- 172.21.15.31:0/2859481576 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934009790 con 0x7f5938100330 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f592e7fc700 1 -- 172.21.15.31:0/2859481576 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f593400eec0 con 0x7f5938100330 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59381a04b0 con 0x7f5938100330 2023-07-29T16:27:58.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f592e7fc700 1 -- 172.21.15.31:0/2859481576 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934003cf0 con 0x7f5938100330 2023-07-29T16:27:58.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.464+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59381a0a10 con 0x7f5938100330 2023-07-29T16:27:58.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.469+0000 7f592e7fc700 1 -- 172.21.15.31:0/2859481576 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f593400ccd0 con 0x7f5938100330 2023-07-29T16:27:58.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 Processor -- start 2023-07-29T16:27:58.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 -- start start 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f592e7fc700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59240a0810 0x7f59240a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd84ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd85530 0x56477dd85d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd862f0 0x56477dd86740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dcdca10 con 0x56477dd862f0 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dcda3d0 con 0x56477dd84be0 2023-07-29T16:27:58.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f595302bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dcd5cc0 con 0x56477dd85530 2023-07-29T16:27:58.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f592e7fc700 1 -- 172.21.15.31:0/2859481576 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5934014070 con 0x7f5938100330 2023-07-29T16:27:58.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f593d59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd84ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.470+0000 7f593d59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd84ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50602/0 (socket says 172.21.15.31:50602) 2023-07-29T16:27:58.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 -- 172.21.15.31:0/2723574969 learned_addr learned my addr 172.21.15.31:0/2723574969 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593dd9c700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59240a0810 0x7f59240a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593cd9a700 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd85530 0x56477dd85d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 -- 172.21.15.31:0/2723574969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd85530 msgr2=0x56477dd85d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd85530 0x56477dd85d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593dd9c700 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd862f0 0x56477dd86740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 -- 172.21.15.31:0/2723574969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd862f0 msgr2=0x56477dd86740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd862f0 0x56477dd86740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 -- 172.21.15.31:0/2723574969 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56477db03de0 con 0x56477dd84be0 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593dd9c700 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd862f0 0x56477dd86740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593d59b700 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd84ff0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5934095970 tx=0x7f593400cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d65443f113236300 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f593dd9c700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59240a0810 0x7f59240a2cc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5938075160 tx=0x7f5930009320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f59237fe700 1 -- 172.21.15.31:0/2723574969 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934098dd0 con 0x56477dd84be0 2023-07-29T16:27:58.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f59237fe700 1 -- 172.21.15.31:0/2723574969 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5934016330 con 0x56477dd84be0 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.471+0000 7f59237fe700 1 -- 172.21.15.31:0/2723574969 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340a1950 con 0x56477dd84be0 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 -- 172.21.15.31:0/2723574969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 msgr2=0x56477dd84ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd84ff0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f5934095970 tx=0x7f593400cbd0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 -- 172.21.15.31:0/2723574969 shutdown_connections 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd862f0 0x56477dd86740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd84ff0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 --2- 172.21.15.31:0/2723574969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd85530 0x56477dd85d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 -- 172.21.15.31:0/2723574969 >> 172.21.15.31:0/2723574969 conn(0x56477dd7a3e0 msgr2=0x56477dd7f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 -- 172.21.15.31:0/2723574969 shutdown_connections 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.472+0000 7f595302bb80 1 -- 172.21.15.31:0/2723574969 wait complete. 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.473+0000 7f595302bb80 1 -- start start 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.473+0000 7f595302bb80 1 Processor -- start 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd8e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd95d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dc72510 con 0x56477dd84be0 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dc721b0 con 0x56477dd85530 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477db59920 con 0x56477dd862f0 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f595302bb80 10 client.0 _ll_register_callbacks cb 0x56477dc7b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f593cd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f593cd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50606/0 (socket says 172.21.15.31:50606) 2023-07-29T16:27:58.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f593cd9a700 1 -- 172.21.15.31:0/896616120 learned_addr learned my addr 172.21.15.31:0/896616120 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f593dd9c700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd95d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.474+0000 7f593d59b700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd8e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593cd9a700 1 -- 172.21.15.31:0/896616120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 msgr2=0x56477dd95d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593cd9a700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd95d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593cd9a700 1 -- 172.21.15.31:0/896616120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 msgr2=0x56477dd8e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593cd9a700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd8e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.480 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593cd9a700 1 -- 172.21.15.31:0/896616120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59381a0a10 con 0x56477dd85530 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593d59b700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd8e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f593cd9a700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8ed40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f59280126d0 tx=0x7f59280129e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e13990ad8fb90be server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f5920ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f5920ff9700 1 -- 172.21.15.31:0/896616120 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928017070 con 0x56477dd85530 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56477dd00d10 con 0x56477dd85530 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f5920ff9700 1 -- 172.21.15.31:0/896616120 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f592800f480 con 0x56477dd85530 2023-07-29T16:27:58.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f5920ff9700 1 -- 172.21.15.31:0/896616120 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f592800f640 con 0x56477dd85530 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f595302bb80 10 client.29252 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.475+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56477daecc60 con 0x56477dd85530 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f5920ff9700 1 -- 172.21.15.31:0/896616120 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56477dd00d10 con 0x56477dd85530 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f5920ff9700 1 -- 172.21.15.31:0/896616120 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f5928026860 con 0x56477dd85530 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f595302bb80 10 client.29252 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f595302bb80 10 client.29252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:58.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f595302bb80 20 client.29252 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f595302bb80 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477dd9a520 0x56477dd9c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f595302bb80 4 client.29252 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56477dd53d70 con 0x56477dd9a520 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.476+0000 7f593d59b700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477dd9a520 0x56477dd9c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.477+0000 7f593d59b700 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477dd9a520 0x56477dd9c930 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e3a45f08ed77d5 server_cookie=6efcf09b043e97e5 in_seq=0 out_seq=0 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.477+0000 7f5920ff9700 10 client.29252 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f5920ff9700 1 -- 172.21.15.31:0/896616120 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56477dd53d70 con 0x56477dd9a520 2023-07-29T16:27:58.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f5920ff9700 10 client.29252 handle_command_reply: tid=0 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 client.29252 shutdown 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 msgr2=0x56477dd8ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8ed40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f59280126d0 tx=0x7f59280129e0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 shutdown_connections 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477dd9a520 0x56477dd9c930 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd95d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd8e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 --2- 172.21.15.31:0/896616120 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8ed40 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f5920ff9700 0 client.29252 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 >> 172.21.15.31:0/896616120 conn(0x56477dd7a3e0 msgr2=0x56477dd7f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 shutdown_connections 2023-07-29T16:27:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.479+0000 7f595302bb80 1 -- 172.21.15.31:0/896616120 wait complete. 2023-07-29T16:27:58.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.480+0000 7f595302bb80 20 client.29252 trim_cache size 0 max 16384 2023-07-29T16:27:58.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.497+0000 7f595302bb80 1 Processor -- start 2023-07-29T16:27:58.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.497+0000 7f595302bb80 1 -- start start 2023-07-29T16:27:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.497+0000 7f595302bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd93d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f595302bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f595302bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd89250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f595302bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dd7c460 con 0x56477dd84be0 2023-07-29T16:27:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f595302bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dd8f480 con 0x56477dd85530 2023-07-29T16:27:58.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f595302bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477dd8c4a0 con 0x56477dd862f0 2023-07-29T16:27:58.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593d59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd93d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593dd9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd89250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50612/0 (socket says 172.21.15.31:50612) 2023-07-29T16:27:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593d59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd93d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49760/0 (socket says 172.21.15.31:49760) 2023-07-29T16:27:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 -- 172.21.15.31:0/1989003730 learned_addr learned my addr 172.21.15.31:0/1989003730 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 -- 172.21.15.31:0/1989003730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 msgr2=0x56477dd89250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd89250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 -- 172.21.15.31:0/1989003730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 msgr2=0x56477dd93d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd93d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593cd9a700 1 -- 172.21.15.31:0/1989003730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56477db2a250 con 0x56477dd85530 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.498+0000 7f593d59b700 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd93d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.499+0000 7f593cd9a700 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8c6f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5928026c40 tx=0x7f59280092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d6e363507a60d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.499+0000 7f5923fff700 1 -- 172.21.15.31:0/1989003730 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592802b070 con 0x56477dd85530 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.499+0000 7f5923fff700 1 -- 172.21.15.31:0/1989003730 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5928010dd0 con 0x56477dd85530 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.499+0000 7f5923fff700 1 -- 172.21.15.31:0/1989003730 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928017040 con 0x56477dd85530 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.499+0000 7f595302bb80 1 -- 172.21.15.31:0/1989003730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 msgr2=0x56477dd8c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.499+0000 7f595302bb80 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8c6f0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5928026c40 tx=0x7f59280092d0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 -- 172.21.15.31:0/1989003730 shutdown_connections 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd89250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd84be0 0x56477dd93d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 --2- 172.21.15.31:0/1989003730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd85530 0x56477dd8c6f0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 -- 172.21.15.31:0/1989003730 >> 172.21.15.31:0/1989003730 conn(0x56477dd7a3e0 msgr2=0x56477dd80690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 -- 172.21.15.31:0/1989003730 shutdown_connections 2023-07-29T16:27:58.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 -- 172.21.15.31:0/1989003730 wait complete. 2023-07-29T16:27:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 -- start start 2023-07-29T16:27:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.500+0000 7f595302bb80 1 Processor -- start 2023-07-29T16:27:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd85530 0x56477dd92c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd93170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477ddec100 con 0x56477dd85530 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477ddec270 con 0x56477dd84be0 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477ddec430 con 0x56477dd862f0 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f595302bb80 10 client.0 _ll_register_callbacks cb 0x56477dc7b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593cd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd85530 0x56477dd92c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd926f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50616/0 (socket says 172.21.15.31:50616) 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 -- 172.21.15.31:0/1450192204 learned_addr learned my addr 172.21.15.31:0/1450192204 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593dd9c700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd93170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 -- 172.21.15.31:0/1450192204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 msgr2=0x56477dd93170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd93170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 -- 172.21.15.31:0/1450192204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd85530 msgr2=0x56477dd92c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd85530 0x56477dd92c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593d59b700 1 -- 172.21.15.31:0/1450192204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56477db03de0 con 0x56477dd84be0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.501+0000 7f593cd9a700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd85530 0x56477dd92c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f593d59b700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd926f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5934005950 tx=0x7f593400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3063b260011964af server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f5920ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f5920ff9700 1 -- 172.21.15.31:0/1450192204 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340b6040 con 0x56477dd84be0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f5920ff9700 1 -- 172.21.15.31:0/1450192204 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5934099470 con 0x56477dd84be0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5934016330 con 0x56477dd84be0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f5920ff9700 1 -- 172.21.15.31:0/1450192204 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f593409d510 con 0x56477dd84be0 2023-07-29T16:27:58.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f595302bb80 10 client.29258 fetch_fsmap learned FSMap version 33 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56477daecc60 con 0x56477dd84be0 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.502+0000 7f5920ff9700 1 -- 172.21.15.31:0/1450192204 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5934016330 con 0x56477dd84be0 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f5920ff9700 1 -- 172.21.15.31:0/1450192204 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f5934098960 con 0x56477dd84be0 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f595302bb80 10 client.29258 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f595302bb80 10 client.29258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f595302bb80 20 client.29258 populate_metadata read hostname 'smithi031' 2023-07-29T16:27:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f595302bb80 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477ddedae0 0x56477ddefef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:27:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f595302bb80 4 client.29258 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:27:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56477dcced40 con 0x56477ddedae0 2023-07-29T16:27:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f593cd9a700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477ddedae0 0x56477ddefef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:27:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f593cd9a700 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477ddedae0 0x56477ddefef0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ccf15a28bef2a25 server_cookie=6a853f74f9feec54 in_seq=0 out_seq=0 2023-07-29T16:27:58.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.503+0000 7f5920ff9700 10 client.29258 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f5920ff9700 1 -- 172.21.15.31:0/1450192204 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56477dcced40 con 0x56477ddedae0 2023-07-29T16:27:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f5920ff9700 10 client.29258 handle_command_reply: tid=0 2023-07-29T16:27:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 client.29258 shutdown 2023-07-29T16:27:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 msgr2=0x56477dd926f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd926f0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5934005950 tx=0x7f593400cc40 comp rx=0 tx=0).stop 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 shutdown_connections 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56477ddedae0 0x56477ddefef0 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56477dd862f0 0x56477dd93170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f5920ff9700 0 client.29258 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56477dd84be0 0x56477dd926f0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 --2- 172.21.15.31:0/1450192204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56477dd85530 0x56477dd92c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 >> 172.21.15.31:0/1450192204 conn(0x56477dd7a3e0 msgr2=0x56477dd8c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 shutdown_connections 2023-07-29T16:27:58.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 1 -- 172.21.15.31:0/1450192204 wait complete. 2023-07-29T16:27:58.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.506+0000 7f595302bb80 20 client.29258 trim_cache size 0 max 16384 2023-07-29T16:27:58.512 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:27:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:27:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:27:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "f4156fd2-bbca-49c1-a42f-93cc09910517": { 2023-07-29T16:27:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:27:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "f4156fd2-bbca-49c1-a42f-93cc09910517", 2023-07-29T16:27:58.513 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:27:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:27:58.514 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:27:58.514 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:27:58.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59240a0810 msgr2=0x7f59240a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59240a0810 0x7f59240a2cc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f5938075160 tx=0x7f5930009320 comp rx=0 tx=0).stop 2023-07-29T16:27:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 msgr2=0x7f5938074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:27:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 0x7f5938074b00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f593400e970 tx=0x7f593400ec80 comp rx=0 tx=0).stop 2023-07-29T16:27:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 shutdown_connections 2023-07-29T16:27:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f59240a0810 0x7f59240a2cc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5938101eb0 0x7f5938073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5938100330 0x7f5938074b00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 --2- 172.21.15.31:0/2859481576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5938101520 0x7f5938073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:27:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 >> 172.21.15.31:0/2859481576 conn(0x7f59380fb970 msgr2=0x7f59380fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:27:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 shutdown_connections 2023-07-29T16:27:58.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:27:58.507+0000 7f593f7ff700 1 -- 172.21.15.31:0/2859481576 wait complete. 2023-07-29T16:27:58.525 INFO:tasks.cephfs.filesystem:scrub status for tag:f4156fd2-bbca-49c1-a42f-93cc09910517 - {'path': '/', 'tag': 'f4156fd2-bbca-49c1-a42f-93cc09910517', 'options': 'recursive,force'} 2023-07-29T16:27:58.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 965181 8.93 MB/sec execute 463 sec latency 30.379 ms 2023-07-29T16:27:59.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:27:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:27:59.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:58 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:27:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:27:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:58 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:27:59.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:59.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:59.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:27:59.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:58 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:27:59.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 966687 8.94 MB/sec execute 464 sec latency 13.130 ms 2023-07-29T16:28:00.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:27:59 smithi039 ceph-mon[128858]: pgmap v747: 97 pgs: 97 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 2.6 MiB/s wr, 286 op/s 2023-07-29T16:28:00.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:27:59 smithi156 ceph-mon[129245]: pgmap v747: 97 pgs: 97 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 2.6 MiB/s wr, 286 op/s 2023-07-29T16:28:00.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:27:59 smithi031 ceph-mon[95504]: pgmap v747: 97 pgs: 97 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 2.6 MiB/s wr, 286 op/s 2023-07-29T16:28:00.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 967972 8.93 MB/sec execute 465 sec latency 77.789 ms 2023-07-29T16:28:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.506+0000 7fc4dfdff700 1 -- 172.21.15.31:0/2136434623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 msgr2=0x7fc4d8100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.506+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/2136434623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d8100760 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009990 tx=0x7fc4d4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 -- 172.21.15.31:0/2136434623 shutdown_connections 2023-07-29T16:28:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/2136434623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101ed0 0x7fc4d8108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/2136434623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d8100760 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/2136434623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4d8101540 0x7fc4d8101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 -- 172.21.15.31:0/2136434623 >> 172.21.15.31:0/2136434623 conn(0x7fc4d80fb970 msgr2=0x7fc4d80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 -- 172.21.15.31:0/2136434623 shutdown_connections 2023-07-29T16:28:01.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.507+0000 7fc4dfdff700 1 -- 172.21.15.31:0/2136434623 wait complete. 2023-07-29T16:28:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 Processor -- start 2023-07-29T16:28:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 -- start start 2023-07-29T16:28:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d81961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4d8101540 0x7fc4d81966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101ed0 0x7fc4d8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8197bc0 con 0x7fc4d8101540 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8197d30 con 0x7fc4d8100350 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.508+0000 7fc4dfdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d8197ef0 con 0x7fc4d8101ed0 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d81961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4de39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101ed0 0x7fc4d8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d81961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33082/0 (socket says 172.21.15.31:33082) 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 -- 172.21.15.31:0/845514519 learned_addr learned my addr 172.21.15.31:0/845514519 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4d8101540 0x7fc4d81966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 -- 172.21.15.31:0/845514519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101ed0 msgr2=0x7fc4d8197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101ed0 0x7fc4d8197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 -- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4d8101540 msgr2=0x7fc4d81966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4d8101540 0x7fc4d81966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 -- 172.21.15.31:0/845514519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d40095f0 con 0x7fc4d8100350 2023-07-29T16:28:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d81961a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4005740 tx=0x7fc4d4005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.509+0000 7fc4ceffd700 1 -- 172.21.15.31:0/845514519 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d401d070 con 0x7fc4d8100350 2023-07-29T16:28:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.510+0000 7fc4ceffd700 1 -- 172.21.15.31:0/845514519 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4d4021410 con 0x7fc4d8100350 2023-07-29T16:28:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.510+0000 7fc4ceffd700 1 -- 172.21.15.31:0/845514519 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4017650 con 0x7fc4d8100350 2023-07-29T16:28:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.510+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d8198170 con 0x7fc4d8100350 2023-07-29T16:28:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.510+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d819e990 con 0x7fc4d8100350 2023-07-29T16:28:01.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.515+0000 7fc4ceffd700 1 -- 172.21.15.31:0/845514519 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc4d4021580 con 0x7fc4d8100350 2023-07-29T16:28:01.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 Processor -- start 2023-07-29T16:28:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 -- start start 2023-07-29T16:28:01.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da59e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da59f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5a0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da4f69b0 con 0x5624da59edf0 2023-07-29T16:28:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da4f4370 con 0x5624da59fcf0 2023-07-29T16:28:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da4efc60 con 0x5624da3fa6b0 2023-07-29T16:28:01.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.516+0000 7fc4ceffd700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4c40a0810 0x7fc4c40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da59e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da59e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58270/0 (socket says 172.21.15.31:58270) 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 -- 172.21.15.31:0/2100193053 learned_addr learned my addr 172.21.15.31:0/2100193053 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4ceffd700 1 -- 172.21.15.31:0/845514519 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4d40ccb60 con 0x7fc4d8100350 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da59f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4c40a0810 0x7fc4c40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 -- 172.21.15.31:0/2100193053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 msgr2=0x5624da5a0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5a0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 -- 172.21.15.31:0/2100193053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 msgr2=0x5624da59f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da59f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.517+0000 7fc4de39c700 1 -- 172.21.15.31:0/2100193053 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624da31dd80 con 0x5624da3fa6b0 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.518+0000 7fc4de39c700 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da59e8b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009fd0 tx=0x7fc4d0017e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=940681561e12c83d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.518+0000 7fc4de39c700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4c40a0810 0x7fc4c40a2cc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0011000 tx=0x7fc4d0022000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.518+0000 7fc4c3fff700 1 -- 172.21.15.31:0/2100193053 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d000e050 con 0x5624da3fa6b0 2023-07-29T16:28:01.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.518+0000 7fc4c3fff700 1 -- 172.21.15.31:0/2100193053 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4d000f040 con 0x5624da3fa6b0 2023-07-29T16:28:01.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.518+0000 7fc4c3fff700 1 -- 172.21.15.31:0/2100193053 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d0022b40 con 0x5624da3fa6b0 2023-07-29T16:28:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2100193053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 msgr2=0x5624da59e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da59e8b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009fd0 tx=0x7fc4d0017e80 comp rx=0 tx=0).stop 2023-07-29T16:28:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2100193053 shutdown_connections 2023-07-29T16:28:01.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5a0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da59e8b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2100193053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da59f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2100193053 >> 172.21.15.31:0/2100193053 conn(0x5624da593fd0 msgr2=0x5624da599300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2100193053 shutdown_connections 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2100193053 wait complete. 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.519+0000 7fc4f36edb80 1 -- start start 2023-07-29T16:28:01.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 Processor -- start 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5a7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5a8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5af480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da48c4b0 con 0x5624da59fcf0 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da48c150 con 0x5624da59edf0 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da3738c0 con 0x5624da3fa6b0 2023-07-29T16:28:01.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4f36edb80 10 client.0 _ll_register_callbacks cb 0x5624da4955c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4ddb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5a7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4dd39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4dd39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33112/0 (socket says 172.21.15.31:33112) 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4dd39a700 1 -- 172.21.15.31:0/2613529543 learned_addr learned my addr 172.21.15.31:0/2613529543 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.520+0000 7fc4de39c700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4dd39a700 1 -- 172.21.15.31:0/2613529543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 msgr2=0x5624da5a7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5a7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4dd39a700 1 -- 172.21.15.31:0/2613529543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 msgr2=0x5624da5af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4dd39a700 1 -- 172.21.15.31:0/2613529543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d0017800 con 0x5624da59edf0 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5a7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5a8440 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80117c0 tx=0x7fc4c8011ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=535c391849e166b0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4c17fa700 1 -- 172.21.15.31:0/2613529543 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8016070 con 0x5624da59edf0 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624da51acb0 con 0x5624da59edf0 2023-07-29T16:28:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.521+0000 7fc4c17fa700 1 -- 172.21.15.31:0/2613529543 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4c800f7b0 con 0x5624da59edf0 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.522+0000 7fc4c17fa700 1 -- 172.21.15.31:0/2613529543 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4c800f970 con 0x5624da59edf0 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.522+0000 7fc4f36edb80 10 client.29270 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.522+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5624da306c00 con 0x5624da59edf0 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.522+0000 7fc4c17fa700 1 -- 172.21.15.31:0/2613529543 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4c8028020 con 0x5624da59edf0 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4c17fa700 1 -- 172.21.15.31:0/2613529543 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fc4c800fad0 con 0x5624da59edf0 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4f36edb80 10 client.29270 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:01.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4f36edb80 10 client.29270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4f36edb80 20 client.29270 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da5b3c80 0x5624da5b6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4f36edb80 4 client.29270 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624da56dd10 con 0x5624da5b3c80 2023-07-29T16:28:01.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da5b3c80 0x5624da5b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4c17fa700 10 client.29270 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.523+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da5b3c80 0x5624da5b6090 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca370ccb1f78f397 server_cookie=1080603d765b9cdb in_seq=0 out_seq=0 2023-07-29T16:28:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4c17fa700 1 -- 172.21.15.31:0/2613529543 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5624da56dd10 con 0x5624da5b3c80 2023-07-29T16:28:01.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4c17fa700 10 client.29270 handle_command_reply: tid=0 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 client.29270 shutdown 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 msgr2=0x5624da5a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5a8440 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fc4c80117c0 tx=0x7fc4c8011ad0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 shutdown_connections 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da5b3c80 0x5624da5b6090 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5a7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/2613529543 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5a8440 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4c17fa700 0 client.29270 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.524+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 >> 172.21.15.31:0/2613529543 conn(0x5624da593fd0 msgr2=0x5624da598a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.525+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 shutdown_connections 2023-07-29T16:28:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.525+0000 7fc4f36edb80 1 -- 172.21.15.31:0/2613529543 wait complete. 2023-07-29T16:28:01.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.525+0000 7fc4f36edb80 20 client.29270 trim_cache size 0 max 16384 2023-07-29T16:28:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 Processor -- start 2023-07-29T16:28:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 -- start start 2023-07-29T16:28:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da599190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da5a7a70 con 0x5624da59fcf0 2023-07-29T16:28:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da59f490 con 0x5624da59edf0 2023-07-29T16:28:01.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da5a5f30 con 0x5624da3fa6b0 2023-07-29T16:28:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4dd39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4ddb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da599190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4dd39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33114/0 (socket says 172.21.15.31:33114) 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4dd39a700 1 -- 172.21.15.31:0/772901647 learned_addr learned my addr 172.21.15.31:0/772901647 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.541+0000 7fc4de39c700 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4dd39a700 1 -- 172.21.15.31:0/772901647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 msgr2=0x5624da599190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da599190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4dd39a700 1 -- 172.21.15.31:0/772901647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 msgr2=0x5624da5a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5a6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4dd39a700 1 -- 172.21.15.31:0/772901647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624da3441f0 con 0x5624da59edf0 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5af440 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800fe30 tx=0x7fc4c8009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b78cd0fa9da76160 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4ccbff700 1 -- 172.21.15.31:0/772901647 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8016040 con 0x5624da59edf0 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4ccbff700 1 -- 172.21.15.31:0/772901647 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4c8028030 con 0x5624da59edf0 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.542+0000 7fc4ccbff700 1 -- 172.21.15.31:0/772901647 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8003a40 con 0x5624da59edf0 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 -- 172.21.15.31:0/772901647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 msgr2=0x5624da5af440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5af440 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800fe30 tx=0x7fc4c8009750 comp rx=0 tx=0).stop 2023-07-29T16:28:01.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 -- 172.21.15.31:0/772901647 shutdown_connections 2023-07-29T16:28:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59fcf0 0x5624da5a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da599190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/772901647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59edf0 0x5624da5af440 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 -- 172.21.15.31:0/772901647 >> 172.21.15.31:0/772901647 conn(0x5624da593fd0 msgr2=0x5624da5a6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 -- 172.21.15.31:0/772901647 shutdown_connections 2023-07-29T16:28:01.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.543+0000 7fc4f36edb80 1 -- 172.21.15.31:0/772901647 wait complete. 2023-07-29T16:28:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 -- start start 2023-07-29T16:28:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 Processor -- start 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5aecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da5af230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5ab790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da5abdb0 con 0x5624da59edf0 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da5abf20 con 0x5624da59fcf0 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624da5ac0e0 con 0x5624da3fa6b0 2023-07-29T16:28:01.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.544+0000 7fc4f36edb80 10 client.0 _ll_register_callbacks cb 0x5624da4955c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4de39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5ab790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4ddb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5aecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da5af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4de39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5ab790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33124/0 (socket says 172.21.15.31:33124) 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4de39c700 1 -- 172.21.15.31:0/3066676082 learned_addr learned my addr 172.21.15.31:0/3066676082 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 -- 172.21.15.31:0/3066676082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 msgr2=0x5624da5aecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5aecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 -- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 msgr2=0x5624da5ab790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5ab790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 -- 172.21.15.31:0/3066676082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624da306c00 con 0x5624da59edf0 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4dd39a700 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da5af230 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x5624da59f780 tx=0x7fc4c800fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b423a1dace22a046 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4c17fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.545+0000 7fc4c17fa700 1 -- 172.21.15.31:0/3066676082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8016070 con 0x5624da59edf0 2023-07-29T16:28:01.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4d000f040 con 0x5624da59edf0 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4c17fa700 1 -- 172.21.15.31:0/3066676082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4c8000e40 con 0x5624da59edf0 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4c17fa700 1 -- 172.21.15.31:0/3066676082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4c80054d0 con 0x5624da59edf0 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 10 client.19164 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5624da31c250 con 0x5624da59edf0 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4c17fa700 1 -- 172.21.15.31:0/3066676082 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4d000f040 con 0x5624da59edf0 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4c17fa700 1 -- 172.21.15.31:0/3066676082 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fc4c803d070 con 0x5624da59edf0 2023-07-29T16:28:01.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 10 client.19164 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 10 client.19164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 20 client.19164 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da606960 0x5624da608d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 4 client.19164 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.546+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5624da4e8ce0 con 0x5624da606960 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.547+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da606960 0x5624da608d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:01.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.547+0000 7fc4c17fa700 10 client.19164 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:01.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.547+0000 7fc4ddb9b700 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da606960 0x5624da608d40 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8fb63237c9f2c7 server_cookie=52a5cd673c0ab1cb in_seq=0 out_seq=0 2023-07-29T16:28:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.621+0000 7fc4c17fa700 1 -- 172.21.15.31:0/3066676082 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5624da4e8ce0 con 0x5624da606960 2023-07-29T16:28:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.621+0000 7fc4c17fa700 10 client.19164 handle_command_reply: tid=0 2023-07-29T16:28:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 client.19164 shutdown 2023-07-29T16:28:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 msgr2=0x5624da5af230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da5af230 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x5624da59f780 tx=0x7fc4c800fa60 comp rx=0 tx=0).stop 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 shutdown_connections 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624da606960 0x5624da608d40 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5624da59fcf0 0x5624da5ab790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624da3fa6b0 0x5624da5aecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 --2- 172.21.15.31:0/3066676082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5624da59edf0 0x5624da5af230 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 >> 172.21.15.31:0/3066676082 conn(0x5624da593fd0 msgr2=0x5624da5990c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:01.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4c17fa700 0 client.19164 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 shutdown_connections 2023-07-29T16:28:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.622+0000 7fc4f36edb80 1 -- 172.21.15.31:0/3066676082 wait complete. 2023-07-29T16:28:01.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4f36edb80 20 client.19164 trim_cache size 0 max 16384 2023-07-29T16:28:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:28:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:28:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:28:01.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:28:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4c40a0810 msgr2=0x7fc4c40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4c40a0810 0x7fc4c40a2cc0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0011000 tx=0x7fc4d0022000 comp rx=0 tx=0).stop 2023-07-29T16:28:01.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 msgr2=0x7fc4d81961a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d81961a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4005740 tx=0x7fc4d4005770 comp rx=0 tx=0).stop 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 shutdown_connections 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4c40a0810 0x7fc4c40a2cc0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4d8101ed0 0x7fc4d8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4d8100350 0x7fc4d81961a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 --2- 172.21.15.31:0/845514519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4d8101540 0x7fc4d81966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.623+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 >> 172.21.15.31:0/845514519 conn(0x7fc4d80fb970 msgr2=0x7fc4d80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:01.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.624+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 shutdown_connections 2023-07-29T16:28:01.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:01.624+0000 7fc4dfdff700 1 -- 172.21.15.31:0/845514519 wait complete. 2023-07-29T16:28:01.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 970063 8.94 MB/sec execute 466 sec latency 66.950 ms 2023-07-29T16:28:02.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:02 smithi156 ceph-mon[129245]: pgmap v748: 97 pgs: 97 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.8 MiB/s wr, 196 op/s 2023-07-29T16:28:02.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 971334 8.93 MB/sec execute 467 sec latency 15.656 ms 2023-07-29T16:28:02.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:02 smithi031 ceph-mon[95504]: pgmap v748: 97 pgs: 97 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.8 MiB/s wr, 196 op/s 2023-07-29T16:28:03.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:02 smithi039 ceph-mon[128858]: pgmap v748: 97 pgs: 97 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 1.8 MiB/s wr, 196 op/s 2023-07-29T16:28:03.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 973553 8.94 MB/sec execute 468 sec latency 17.463 ms 2023-07-29T16:28:04.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:04 smithi156 ceph-mon[129245]: pgmap v749: 97 pgs: 97 active+clean; 170 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 3.4 MiB/s wr, 272 op/s 2023-07-29T16:28:04.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 974689 8.93 MB/sec execute 469 sec latency 14.346 ms 2023-07-29T16:28:04.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:04 smithi031 ceph-mon[95504]: pgmap v749: 97 pgs: 97 active+clean; 170 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 3.4 MiB/s wr, 272 op/s 2023-07-29T16:28:05.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:04 smithi039 ceph-mon[128858]: pgmap v749: 97 pgs: 97 active+clean; 170 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 3.4 MiB/s wr, 272 op/s 2023-07-29T16:28:05.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 976673 8.94 MB/sec execute 470 sec latency 43.401 ms 2023-07-29T16:28:06.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:06 smithi156 ceph-mon[129245]: pgmap v750: 97 pgs: 97 active+clean; 178 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 3.9 MiB/s wr, 245 op/s 2023-07-29T16:28:06.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 977806 8.93 MB/sec execute 471 sec latency 24.375 ms 2023-07-29T16:28:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:06 smithi031 ceph-mon[95504]: pgmap v750: 97 pgs: 97 active+clean; 178 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 3.9 MiB/s wr, 245 op/s 2023-07-29T16:28:07.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:06 smithi039 ceph-mon[128858]: pgmap v750: 97 pgs: 97 active+clean; 178 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 3.9 MiB/s wr, 245 op/s 2023-07-29T16:28:07.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 980026 8.94 MB/sec execute 472 sec latency 18.163 ms 2023-07-29T16:28:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:07 smithi039 ceph-mon[128858]: pgmap v751: 97 pgs: 97 active+clean; 179 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 3.7 MiB/s wr, 258 op/s 2023-07-29T16:28:08.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:07 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:28:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:07 smithi156 ceph-mon[129245]: pgmap v751: 97 pgs: 97 active+clean; 179 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 3.7 MiB/s wr, 258 op/s 2023-07-29T16:28:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:07 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:28:08.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:07 smithi031 ceph-mon[95504]: pgmap v751: 97 pgs: 97 active+clean; 179 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 3.7 MiB/s wr, 258 op/s 2023-07-29T16:28:08.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:07 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:28:08.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 981222 8.93 MB/sec execute 473 sec latency 15.426 ms 2023-07-29T16:28:09.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:08 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:28:09.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:08 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:28:09.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:08 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:28:09.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:08 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:28:09.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:08 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:28:09.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:08 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:28:09.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 983436 8.94 MB/sec execute 474 sec latency 49.265 ms 2023-07-29T16:28:10.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:09 smithi039 ceph-mon[128858]: pgmap v752: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 4.4 MiB/s wr, 308 op/s 2023-07-29T16:28:10.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:09 smithi156 ceph-mon[129245]: pgmap v752: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 4.4 MiB/s wr, 308 op/s 2023-07-29T16:28:10.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:09 smithi031 ceph-mon[95504]: pgmap v752: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 4.4 MiB/s wr, 308 op/s 2023-07-29T16:28:10.896 INFO:tasks.workunit.client.0.smithi031.stdout: 1 984524 8.93 MB/sec execute 475 sec latency 46.066 ms 2023-07-29T16:28:11.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 986525 8.93 MB/sec execute 476 sec latency 19.413 ms 2023-07-29T16:28:12.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:12 smithi156 ceph-mon[129245]: pgmap v753: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 3.7 MiB/s wr, 227 op/s 2023-07-29T16:28:12.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 988020 8.94 MB/sec execute 477 sec latency 18.453 ms 2023-07-29T16:28:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:12 smithi031 ceph-mon[95504]: pgmap v753: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 3.7 MiB/s wr, 227 op/s 2023-07-29T16:28:13.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:12 smithi039 ceph-mon[128858]: pgmap v753: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 3.7 MiB/s wr, 227 op/s 2023-07-29T16:28:13.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 989826 8.93 MB/sec execute 478 sec latency 13.253 ms 2023-07-29T16:28:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 -- 172.21.15.31:0/1280189413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 msgr2=0x7f4e1c10a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 --2- 172.21.15.31:0/1280189413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 0x7f4e1c10a660 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b0a0 tx=0x7f4e1400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 -- 172.21.15.31:0/1280189413 shutdown_connections 2023-07-29T16:28:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 --2- 172.21.15.31:0/1280189413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 0x7f4e1c10a660 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 --2- 172.21.15.31:0/1280189413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c074d30 0x7f4e1c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 --2- 172.21.15.31:0/1280189413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c073750 0x7f4e1c073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 -- 172.21.15.31:0/1280189413 >> 172.21.15.31:0/1280189413 conn(0x7f4e1c0fb970 msgr2=0x7f4e1c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 -- 172.21.15.31:0/1280189413 shutdown_connections 2023-07-29T16:28:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.776+0000 7f4e247ff700 1 -- 172.21.15.31:0/1280189413 wait complete. 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.777+0000 7f4e247ff700 1 Processor -- start 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 -- start start 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 0x7f4e1c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c074d30 0x7f4e1c1989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 0x7f4e1c199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c199ec0 con 0x7f4e1c074d30 2023-07-29T16:28:14.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19a030 con 0x7f4e1c073750 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e247ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1c19a1f0 con 0x7f4e1c1081e0 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 0x7f4e1c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 0x7f4e1c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57476/0 (socket says 172.21.15.31:57476) 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 -- 172.21.15.31:0/2796154604 learned_addr learned my addr 172.21.15.31:0/2796154604 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 0x7f4e1c199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c074d30 0x7f4e1c1989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 -- 172.21.15.31:0/2796154604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 msgr2=0x7f4e1c199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 0x7f4e1c199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 -- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c074d30 msgr2=0x7f4e1c1989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c074d30 0x7f4e1c1989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e2259b700 1 -- 172.21.15.31:0/2796154604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e14009d00 con 0x7f4e1c073750 2023-07-29T16:28:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.778+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c074d30 0x7f4e1c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:14.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.779+0000 7f4e2259b700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 0x7f4e1c1984a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800e9a0 tx=0x7f4e1800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.779+0000 7f4e137fe700 1 -- 172.21.15.31:0/2796154604 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18009790 con 0x7f4e1c073750 2023-07-29T16:28:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.779+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1c19a4d0 con 0x7f4e1c073750 2023-07-29T16:28:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.779+0000 7f4e137fe700 1 -- 172.21.15.31:0/2796154604 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1800cbd0 con 0x7f4e1c073750 2023-07-29T16:28:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.779+0000 7f4e137fe700 1 -- 172.21.15.31:0/2796154604 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18018680 con 0x7f4e1c073750 2023-07-29T16:28:14.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.779+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c1a0c90 con 0x7f4e1c073750 2023-07-29T16:28:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.782+0000 7f4e137fe700 1 -- 172.21.15.31:0/2796154604 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4e1800cd40 con 0x7f4e1c073750 2023-07-29T16:28:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 Processor -- start 2023-07-29T16:28:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 -- start start 2023-07-29T16:28:14.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e137fe700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4e080a0790 0x7f4e080a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e137fe700 1 -- 172.21.15.31:0/2796154604 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4e18014070 con 0x7f4e1c073750 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371968b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371977b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x559737198160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597370ee9b0 con 0x559737196df0 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597370ec370 con 0x559736ff26b0 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597370e7c60 con 0x559737197cf0 2023-07-29T16:28:14.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.784+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4e080a0790 0x7f4e080a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371968b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57488/0 (socket says 172.21.15.31:57488) 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 -- 172.21.15.31:0/482130951 learned_addr learned my addr 172.21.15.31:0/482130951 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x559737198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 -- 172.21.15.31:0/482130951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 msgr2=0x559737198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x559737198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 -- 172.21.15.31:0/482130951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 msgr2=0x5597371977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 -- 172.21.15.31:0/482130951 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559736f15d80 con 0x559736ff26b0 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4e080a0790 0x7f4e080a2c40 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b780 tx=0x7f4e14007720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e2259b700 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371968b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4e180994a0 tx=0x7f4e180990d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5441cd1306f75d84 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e10ff9700 1 -- 172.21.15.31:0/482130951 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18099510 con 0x559736ff26b0 2023-07-29T16:28:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e10ff9700 1 -- 172.21.15.31:0/482130951 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1809d540 con 0x559736ff26b0 2023-07-29T16:28:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.785+0000 7f4e10ff9700 1 -- 172.21.15.31:0/482130951 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18098c60 con 0x559736ff26b0 2023-07-29T16:28:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 -- 172.21.15.31:0/482130951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 msgr2=0x5597371968b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371968b0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4e180994a0 tx=0x7f4e180990d0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 -- 172.21.15.31:0/482130951 shutdown_connections 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x559737198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371968b0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/482130951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 -- 172.21.15.31:0/482130951 >> 172.21.15.31:0/482130951 conn(0x55973718bfd0 msgr2=0x559737191300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 -- 172.21.15.31:0/482130951 shutdown_connections 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.786+0000 7f4e380d1b80 1 -- 172.21.15.31:0/482130951 wait complete. 2023-07-29T16:28:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.787+0000 7f4e380d1b80 1 -- start start 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.787+0000 7f4e380d1b80 1 Processor -- start 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.787+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x55973719ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.787+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x5597371a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x5597371a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597370844b0 con 0x559736ff26b0 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559737084150 con 0x559737197cf0 2023-07-29T16:28:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559736f6b8c0 con 0x559737196df0 2023-07-29T16:28:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e380d1b80 10 client.0 _ll_register_callbacks cb 0x55973708d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x5597371a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x55973719ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x5597371a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:49058/0 (socket says 172.21.15.31:49058) 2023-07-29T16:28:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 -- 172.21.15.31:0/31137563 learned_addr learned my addr 172.21.15.31:0/31137563 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x5597371a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 -- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 msgr2=0x5597371a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x5597371a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 -- 172.21.15.31:0/31137563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 msgr2=0x55973719ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x55973719ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e21d9a700 1 -- 172.21.15.31:0/31137563 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e1c1a0c90 con 0x559737196df0 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.788+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x5597371a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x5597371a0460 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c013790 tx=0x7f4e0c013aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=816370a64965d23f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:14.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e05ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e05ffb700 1 -- 172.21.15.31:0/31137563 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c011910 con 0x559737196df0 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559737112cb0 con 0x559737196df0 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e05ffb700 1 -- 172.21.15.31:0/31137563 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4e0c011a70 con 0x559737196df0 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e05ffb700 1 -- 172.21.15.31:0/31137563 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e0c011c30 con 0x559737196df0 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e380d1b80 10 client.29284 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559736efec00 con 0x559737196df0 2023-07-29T16:28:14.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e05ffb700 1 -- 172.21.15.31:0/31137563 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e0c02b080 con 0x559737196df0 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.789+0000 7f4e05ffb700 1 -- 172.21.15.31:0/31137563 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f4e0c0273f0 con 0x559737196df0 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e380d1b80 10 client.29284 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e380d1b80 10 client.29284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e380d1b80 20 client.29284 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371abd70 0x5597371ae180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e380d1b80 4 client.29284 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559737165d10 con 0x5597371abd70 2023-07-29T16:28:14.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e2259b700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371abd70 0x5597371ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e2259b700 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371abd70 0x5597371ae180 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48fd025b9158d26a server_cookie=81426924d902622e in_seq=0 out_seq=0 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.790+0000 7f4e05ffb700 10 client.29284 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.791+0000 7f4e05ffb700 1 -- 172.21.15.31:0/31137563 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559737165d10 con 0x5597371abd70 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.791+0000 7f4e05ffb700 10 client.29284 handle_command_reply: tid=0 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.791+0000 7f4e380d1b80 1 client.29284 shutdown 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.791+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 msgr2=0x5597371a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.791+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x5597371a0460 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c013790 tx=0x7f4e0c013aa0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 shutdown_connections 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371abd70 0x5597371ae180 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x5597371a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x55973719ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/31137563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x5597371a0460 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e05ffb700 0 client.29284 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 >> 172.21.15.31:0/31137563 conn(0x55973718bfd0 msgr2=0x559737190a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:14.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 shutdown_connections 2023-07-29T16:28:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 1 -- 172.21.15.31:0/31137563 wait complete. 2023-07-29T16:28:14.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.792+0000 7f4e380d1b80 20 client.29284 trim_cache size 0 max 16384 2023-07-29T16:28:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.807+0000 7f4e380d1b80 1 Processor -- start 2023-07-29T16:28:14.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.807+0000 7f4e380d1b80 1 -- start start 2023-07-29T16:28:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x5597371a5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x55973719ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x55973719b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559737191fc0 con 0x559736ff26b0 2023-07-29T16:28:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371921c0 con 0x559737197cf0 2023-07-29T16:28:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559737192df0 con 0x559737196df0 2023-07-29T16:28:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x5597371a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x5597371a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59714/0 (socket says 172.21.15.31:59714) 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 -- 172.21.15.31:0/3769738304 learned_addr learned my addr 172.21.15.31:0/3769738304 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x55973719b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x55973719ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 -- 172.21.15.31:0/3769738304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 msgr2=0x55973719ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x55973719ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 -- 172.21.15.31:0/3769738304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 msgr2=0x55973719b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x55973719b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e2259b700 1 -- 172.21.15.31:0/3769738304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559736f3c1f0 con 0x559736ff26b0 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.808+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x55973719ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e2259b700 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x5597371a5450 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4e180996d0 tx=0x7f4e1801a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d9b92912b1e200 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e117fa700 1 -- 172.21.15.31:0/3769738304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e180b7040 con 0x559736ff26b0 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e117fa700 1 -- 172.21.15.31:0/3769738304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e18095850 con 0x559736ff26b0 2023-07-29T16:28:14.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e117fa700 1 -- 172.21.15.31:0/3769738304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e180a1730 con 0x559736ff26b0 2023-07-29T16:28:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 -- 172.21.15.31:0/3769738304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 msgr2=0x5597371a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x5597371a5450 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4e180996d0 tx=0x7f4e1801a040 comp rx=0 tx=0).stop 2023-07-29T16:28:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 -- 172.21.15.31:0/3769738304 shutdown_connections 2023-07-29T16:28:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559737197cf0 0x55973719b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559736ff26b0 0x5597371a5450 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/3769738304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737196df0 0x55973719ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.809+0000 7f4e380d1b80 1 -- 172.21.15.31:0/3769738304 >> 172.21.15.31:0/3769738304 conn(0x55973718bfd0 msgr2=0x55973719e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.810+0000 7f4e380d1b80 1 -- 172.21.15.31:0/3769738304 shutdown_connections 2023-07-29T16:28:14.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.810+0000 7f4e380d1b80 1 -- 172.21.15.31:0/3769738304 wait complete. 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.810+0000 7f4e380d1b80 1 -- start start 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.810+0000 7f4e380d1b80 1 Processor -- start 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371a3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371a3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x5597371a4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371a4cb0 con 0x559737196df0 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371a4e20 con 0x559736ff26b0 2023-07-29T16:28:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597371a4fe0 con 0x559737197cf0 2023-07-29T16:28:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e380d1b80 10 client.0 _ll_register_callbacks cb 0x55973708d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e22d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x5597371a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e2259b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371a3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57506/0 (socket says 172.21.15.31:57506) 2023-07-29T16:28:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e2259b700 1 -- 172.21.15.31:0/1064380643 learned_addr learned my addr 172.21.15.31:0/1064380643 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e21d9a700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e22d9c700 1 -- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 msgr2=0x5597371a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371a3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e22d9c700 1 -- 172.21.15.31:0/1064380643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 msgr2=0x5597371a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e22d9c700 1 -- 172.21.15.31:0/1064380643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559736f15d80 con 0x559737197cf0 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.811+0000 7f4e2259b700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e22d9c700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x5597371a4770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009fd0 tx=0x7f4e1400b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4940e23f60893698 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e05ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e05ffb700 1 -- 172.21.15.31:0/1064380643 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1400e040 con 0x559737197cf0 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e1809d540 con 0x559737197cf0 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e05ffb700 1 -- 172.21.15.31:0/1064380643 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4e1402d890 con 0x559737197cf0 2023-07-29T16:28:14.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e05ffb700 1 -- 172.21.15.31:0/1064380643 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e1402da50 con 0x559737197cf0 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e380d1b80 10 client.29290 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x559736f14250 con 0x559737197cf0 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.812+0000 7f4e05ffb700 1 -- 172.21.15.31:0/1064380643 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1809d540 con 0x559737197cf0 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e05ffb700 1 -- 172.21.15.31:0/1064380643 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f4e14005040 con 0x559737197cf0 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e380d1b80 10 client.29290 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e380d1b80 10 client.29290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:14.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e380d1b80 20 client.29290 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371fe910 0x559737200d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e380d1b80 4 client.29290 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:28:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5597370e0ce0 con 0x5597371fe910 2023-07-29T16:28:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e2259b700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371fe910 0x559737200d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e2259b700 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371fe910 0x559737200d20 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ec2ffc3cd3fedec server_cookie=de760fe6384700cf in_seq=0 out_seq=0 2023-07-29T16:28:14.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.813+0000 7f4e05ffb700 10 client.29290 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:14.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:14 smithi156 ceph-mon[129245]: pgmap v754: 97 pgs: 97 active+clean; 182 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 4.2 MiB/s wr, 293 op/s 2023-07-29T16:28:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.885+0000 7f4e05ffb700 1 -- 172.21.15.31:0/1064380643 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5597370e0ce0 con 0x5597371fe910 2023-07-29T16:28:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.885+0000 7f4e05ffb700 10 client.29290 handle_command_reply: tid=0 2023-07-29T16:28:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.885+0000 7f4e380d1b80 1 client.29290 shutdown 2023-07-29T16:28:14.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.885+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 msgr2=0x5597371a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.885+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x5597371a4770 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009fd0 tx=0x7f4e1400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 shutdown_connections 2023-07-29T16:28:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597371fe910 0x559737200d20 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559737197cf0 0x5597371a4770 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559736ff26b0 0x5597371a3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 --2- 172.21.15.31:0/1064380643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559737196df0 0x5597371a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e05ffb700 0 client.29290 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 >> 172.21.15.31:0/1064380643 conn(0x55973718bfd0 msgr2=0x55973719e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 shutdown_connections 2023-07-29T16:28:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 1 -- 172.21.15.31:0/1064380643 wait complete. 2023-07-29T16:28:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.886+0000 7f4e380d1b80 20 client.29290 trim_cache size 0 max 16384 2023-07-29T16:28:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:28:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:28:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:28:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:28:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4e080a0790 msgr2=0x7f4e080a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4e080a0790 0x7f4e080a2c40 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b780 tx=0x7f4e14007720 comp rx=0 tx=0).stop 2023-07-29T16:28:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 msgr2=0x7f4e1c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 0x7f4e1c1984a0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800e9a0 tx=0x7f4e1800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 shutdown_connections 2023-07-29T16:28:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4e080a0790 0x7f4e080a2c40 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1c1081e0 0x7f4e1c199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e1c073750 0x7f4e1c1984a0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 --2- 172.21.15.31:0/2796154604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e1c074d30 0x7f4e1c1989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 >> 172.21.15.31:0/2796154604 conn(0x7f4e1c0fb970 msgr2=0x7f4e1c10a420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 shutdown_connections 2023-07-29T16:28:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:14.887+0000 7f4e247ff700 1 -- 172.21.15.31:0/2796154604 wait complete. 2023-07-29T16:28:14.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 991473 8.94 MB/sec execute 479 sec latency 65.007 ms 2023-07-29T16:28:14.933 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:14 smithi031 ceph-mon[95504]: pgmap v754: 97 pgs: 97 active+clean; 182 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 4.2 MiB/s wr, 293 op/s 2023-07-29T16:28:15.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:14 smithi039 ceph-mon[128858]: pgmap v754: 97 pgs: 97 active+clean; 182 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 4.2 MiB/s wr, 293 op/s 2023-07-29T16:28:15.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 992641 8.93 MB/sec execute 480 sec latency 30.577 ms 2023-07-29T16:28:16.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:16 smithi156 ceph-mon[129245]: pgmap v755: 97 pgs: 97 active+clean; 183 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.9 MiB/s wr, 239 op/s 2023-07-29T16:28:16.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 994783 8.94 MB/sec execute 481 sec latency 31.433 ms 2023-07-29T16:28:16.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:16 smithi031 ceph-mon[95504]: pgmap v755: 97 pgs: 97 active+clean; 183 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.9 MiB/s wr, 239 op/s 2023-07-29T16:28:17.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:16 smithi039 ceph-mon[128858]: pgmap v755: 97 pgs: 97 active+clean; 183 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.9 MiB/s wr, 239 op/s 2023-07-29T16:28:17.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 995485 8.92 MB/sec execute 482 sec latency 173.623 ms 2023-07-29T16:28:18.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:17 smithi039 ceph-mon[128858]: pgmap v756: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 2.2 MiB/s wr, 228 op/s 2023-07-29T16:28:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:17 smithi156 ceph-mon[129245]: pgmap v756: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 2.2 MiB/s wr, 228 op/s 2023-07-29T16:28:18.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:17 smithi031 ceph-mon[95504]: pgmap v756: 97 pgs: 97 active+clean; 181 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 2.2 MiB/s wr, 228 op/s 2023-07-29T16:28:18.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 997699 8.93 MB/sec execute 483 sec latency 15.879 ms 2023-07-29T16:28:19.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 998859 8.93 MB/sec execute 484 sec latency 17.027 ms 2023-07-29T16:28:20.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1000822 8.92 MB/sec execute 485 sec latency 50.773 ms 2023-07-29T16:28:20.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:20 smithi031 ceph-mon[95504]: pgmap v757: 97 pgs: 97 active+clean; 182 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.8 MiB/s wr, 281 op/s 2023-07-29T16:28:21.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:20 smithi039 ceph-mon[128858]: pgmap v757: 97 pgs: 97 active+clean; 182 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.8 MiB/s wr, 281 op/s 2023-07-29T16:28:21.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:20 smithi156 ceph-mon[129245]: pgmap v757: 97 pgs: 97 active+clean; 182 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 2.8 MiB/s wr, 281 op/s 2023-07-29T16:28:21.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1002222 8.93 MB/sec execute 486 sec latency 31.273 ms 2023-07-29T16:28:22.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1003690 8.92 MB/sec execute 487 sec latency 25.477 ms 2023-07-29T16:28:22.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:22 smithi031 ceph-mon[95504]: pgmap v758: 97 pgs: 97 active+clean; 182 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.9 MiB/s wr, 226 op/s 2023-07-29T16:28:23.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:22 smithi039 ceph-mon[128858]: pgmap v758: 97 pgs: 97 active+clean; 182 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.9 MiB/s wr, 226 op/s 2023-07-29T16:28:23.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:22 smithi156 ceph-mon[129245]: pgmap v758: 97 pgs: 97 active+clean; 182 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 1.9 MiB/s wr, 226 op/s 2023-07-29T16:28:23.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1005639 8.93 MB/sec execute 488 sec latency 15.602 ms 2023-07-29T16:28:24.897 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1006907 8.92 MB/sec execute 489 sec latency 14.431 ms 2023-07-29T16:28:24.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:24 smithi031 ceph-mon[95504]: pgmap v759: 97 pgs: 97 active+clean; 185 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 306 op/s 2023-07-29T16:28:25.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:24 smithi039 ceph-mon[128858]: pgmap v759: 97 pgs: 97 active+clean; 185 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 306 op/s 2023-07-29T16:28:25.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:24 smithi156 ceph-mon[129245]: pgmap v759: 97 pgs: 97 active+clean; 185 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 2.5 MiB/s wr, 306 op/s 2023-07-29T16:28:25.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1009020 8.93 MB/sec execute 490 sec latency 24.711 ms 2023-07-29T16:28:26.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1009931 8.92 MB/sec execute 491 sec latency 26.576 ms 2023-07-29T16:28:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:26 smithi031 ceph-mon[95504]: pgmap v760: 97 pgs: 97 active+clean; 185 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.4 MiB/s wr, 256 op/s 2023-07-29T16:28:27.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:26 smithi039 ceph-mon[128858]: pgmap v760: 97 pgs: 97 active+clean; 185 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.4 MiB/s wr, 256 op/s 2023-07-29T16:28:27.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:26 smithi156 ceph-mon[129245]: pgmap v760: 97 pgs: 97 active+clean; 185 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.4 MiB/s wr, 256 op/s 2023-07-29T16:28:27.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1012314 8.93 MB/sec execute 492 sec latency 15.582 ms 2023-07-29T16:28:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 -- 172.21.15.31:0/4079929291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103570 msgr2=0x7f392c1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 --2- 172.21.15.31:0/4079929291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103570 0x7f392c1039c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f391c009990 tx=0x7f391c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 -- 172.21.15.31:0/4079929291 shutdown_connections 2023-07-29T16:28:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 --2- 172.21.15.31:0/4079929291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c103f00 0x7f392c10a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 --2- 172.21.15.31:0/4079929291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392c102380 0x7f392c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 --2- 172.21.15.31:0/4079929291 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103570 0x7f392c1039c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 -- 172.21.15.31:0/4079929291 >> 172.21.15.31:0/4079929291 conn(0x7f392c0fd9c0 msgr2=0x7f392c0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 -- 172.21.15.31:0/4079929291 shutdown_connections 2023-07-29T16:28:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.046+0000 7f3932f9d700 1 -- 172.21.15.31:0/4079929291 wait complete. 2023-07-29T16:28:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.047+0000 7f3932f9d700 1 Processor -- start 2023-07-29T16:28:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.047+0000 7f3932f9d700 1 -- start start 2023-07-29T16:28:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3932f9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 0x7f392c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3932f9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392c103570 0x7f392c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3932f9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103f00 0x7f392c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3932f9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c19fc00 con 0x7f392c103570 2023-07-29T16:28:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3932f9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c19fd40 con 0x7f392c103f00 2023-07-29T16:28:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3932f9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c19fed0 con 0x7f392c102380 2023-07-29T16:28:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 0x7f392c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 0x7f392c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42878/0 (socket says 172.21.15.31:42878) 2023-07-29T16:28:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 -- 172.21.15.31:0/2601787431 learned_addr learned my addr 172.21.15.31:0/2601787431 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f393279c700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103f00 0x7f392c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f393179a700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392c103570 0x7f392c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 -- 172.21.15.31:0/2601787431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103f00 msgr2=0x7f392c19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103f00 0x7f392c19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 -- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392c103570 msgr2=0x7f392c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392c103570 0x7f392c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f3931f9b700 1 -- 172.21.15.31:0/2601787431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0095f0 con 0x7f392c102380 2023-07-29T16:28:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.048+0000 7f393279c700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103f00 0x7f392c19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.049+0000 7f3931f9b700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 0x7f392c198080 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f392800e970 tx=0x7f392800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.049+0000 7f3922ffd700 1 -- 172.21.15.31:0/2601787431 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928009790 con 0x7f392c102380 2023-07-29T16:28:28.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.049+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c1a01b0 con 0x7f392c102380 2023-07-29T16:28:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.049+0000 7f3922ffd700 1 -- 172.21.15.31:0/2601787431 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f392800eec0 con 0x7f392c102380 2023-07-29T16:28:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.049+0000 7f3922ffd700 1 -- 172.21.15.31:0/2601787431 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928003cf0 con 0x7f392c102380 2023-07-29T16:28:28.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.049+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c1a06c0 con 0x7f392c102380 2023-07-29T16:28:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.053+0000 7f3922ffd700 1 -- 172.21.15.31:0/2601787431 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f392800ccd0 con 0x7f392c102380 2023-07-29T16:28:28.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.054+0000 7f3947a61b80 1 Processor -- start 2023-07-29T16:28:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.054+0000 7f3922ffd700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39180a0790 0x7f39180a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.054+0000 7f3947a61b80 1 -- start start 2023-07-29T16:28:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.054+0000 7f3922ffd700 1 -- 172.21.15.31:0/2601787431 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3928014070 con 0x7f392c102380 2023-07-29T16:28:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.054+0000 7f393179a700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39180a0790 0x7f39180a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589acca2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3947a61b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acbfa9b0 con 0x5589acca3150 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3947a61b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acbf8370 con 0x5589acca3f00 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3947a61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acbf3c60 con 0x5589acafe6b0 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589acca2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589acca2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42884/0 (socket says 172.21.15.31:42884) 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f393279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 -- 172.21.15.31:0/318606918 learned_addr learned my addr 172.21.15.31:0/318606918 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f393179a700 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 -- 172.21.15.31:0/318606918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 msgr2=0x5589acca43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 -- 172.21.15.31:0/318606918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 msgr2=0x5589acca39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.055+0000 7f3931f9b700 1 -- 172.21.15.31:0/318606918 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589aca21d80 con 0x5589acafe6b0 2023-07-29T16:28:28.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f393179a700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39180a0790 0x7f39180a2c40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f391c00b2e0 tx=0x7f391c005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f3931f9b700 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589acca2c10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3928095f80 tx=0x7f3928099120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7d266b14d614d8e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f3917fff700 1 -- 172.21.15.31:0/318606918 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928099510 con 0x5589acafe6b0 2023-07-29T16:28:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f3917fff700 1 -- 172.21.15.31:0/318606918 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f392809d540 con 0x5589acafe6b0 2023-07-29T16:28:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f3917fff700 1 -- 172.21.15.31:0/318606918 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928099990 con 0x5589acafe6b0 2023-07-29T16:28:28.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f3947a61b80 1 -- 172.21.15.31:0/318606918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 msgr2=0x5589acca2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.056+0000 7f3947a61b80 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589acca2c10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3928095f80 tx=0x7f3928099120 comp rx=0 tx=0).stop 2023-07-29T16:28:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 -- 172.21.15.31:0/318606918 shutdown_connections 2023-07-29T16:28:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589acca2c10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 --2- 172.21.15.31:0/318606918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 -- 172.21.15.31:0/318606918 >> 172.21.15.31:0/318606918 conn(0x5589acc98380 msgr2=0x5589acc9d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 -- 172.21.15.31:0/318606918 shutdown_connections 2023-07-29T16:28:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 -- 172.21.15.31:0/318606918 wait complete. 2023-07-29T16:28:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.057+0000 7f3947a61b80 1 -- start start 2023-07-29T16:28:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 Processor -- start 2023-07-29T16:28:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accac630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acb904b0 con 0x5589acca3f00 2023-07-29T16:28:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acb90150 con 0x5589acca3150 2023-07-29T16:28:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589aca778c0 con 0x5589acafe6b0 2023-07-29T16:28:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.058+0000 7f3947a61b80 10 client.0 _ll_register_callbacks cb 0x5589acb995c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:28.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f393179a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f393179a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51838/0 (socket says 172.21.15.31:51838) 2023-07-29T16:28:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f393179a700 1 -- 172.21.15.31:0/752654418 learned_addr learned my addr 172.21.15.31:0/752654418 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f393279c700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 -- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 msgr2=0x5589accac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accac630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 -- 172.21.15.31:0/752654418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 msgr2=0x5589accb3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 -- 172.21.15.31:0/752654418 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c1a06c0 con 0x5589acafe6b0 2023-07-29T16:28:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f393279c700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f393179a700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f3931f9b700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accac0f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f39280959c0 tx=0x7f3928098dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3257ab50ea02af5a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f39157fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f39157fa700 1 -- 172.21.15.31:0/752654418 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928099510 con 0x5589acafe6b0 2023-07-29T16:28:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.059+0000 7f39157fa700 1 -- 172.21.15.31:0/752654418 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3928099770 con 0x5589acafe6b0 2023-07-29T16:28:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5589acc1ecb0 con 0x5589acafe6b0 2023-07-29T16:28:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f39157fa700 1 -- 172.21.15.31:0/752654418 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3928099930 con 0x5589acafe6b0 2023-07-29T16:28:28.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 10 client.29302 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5589aca0ac00 con 0x5589acafe6b0 2023-07-29T16:28:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f39157fa700 1 -- 172.21.15.31:0/752654418 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5589acc1ecb0 con 0x5589acafe6b0 2023-07-29T16:28:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f39157fa700 1 -- 172.21.15.31:0/752654418 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f39280af070 con 0x5589acafe6b0 2023-07-29T16:28:28.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 10 client.29302 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 10 client.29302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 20 client.29302 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589accb7a70 0x5589accb9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 4 client.29302 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.060+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589acc71d10 con 0x5589accb7a70 2023-07-29T16:28:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.061+0000 7f393179a700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589accb7a70 0x5589accb9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.061+0000 7f393179a700 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589accb7a70 0x5589accb9e50 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3956d22a00a4fc41 server_cookie=7e35fdc01c76a4da in_seq=0 out_seq=0 2023-07-29T16:28:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.061+0000 7f39157fa700 10 client.29302 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f39157fa700 1 -- 172.21.15.31:0/752654418 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5589acc71d10 con 0x5589accb7a70 2023-07-29T16:28:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f39157fa700 10 client.29302 handle_command_reply: tid=0 2023-07-29T16:28:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 client.29302 shutdown 2023-07-29T16:28:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 msgr2=0x5589accac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accac0f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f39280959c0 tx=0x7f3928098dc0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 shutdown_connections 2023-07-29T16:28:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589accb7a70 0x5589accb9e50 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accac0f0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 --2- 172.21.15.31:0/752654418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f39157fa700 0 client.29302 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 >> 172.21.15.31:0/752654418 conn(0x5589acc98380 msgr2=0x5589acc9cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 shutdown_connections 2023-07-29T16:28:28.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.062+0000 7f3947a61b80 1 -- 172.21.15.31:0/752654418 wait complete. 2023-07-29T16:28:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.063+0000 7f3947a61b80 20 client.29302 trim_cache size 0 max 16384 2023-07-29T16:28:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 Processor -- start 2023-07-29T16:28:28.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 -- start start 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accb1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acc95490 con 0x5589acca3150 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589accab5a0 con 0x5589acca3f00 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.079+0000 7f3947a61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589acc9f020 con 0x5589acafe6b0 2023-07-29T16:28:28.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accb1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f393179a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accb1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42900/0 (socket says 172.21.15.31:42900) 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f393279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 -- 172.21.15.31:0/1452239427 learned_addr learned my addr 172.21.15.31:0/1452239427 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 -- 172.21.15.31:0/1452239427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 msgr2=0x5589acca7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 -- 172.21.15.31:0/1452239427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 msgr2=0x5589acca6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 -- 172.21.15.31:0/1452239427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589aca481f0 con 0x5589acafe6b0 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3931f9b700 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accb1620 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3928003730 tx=0x7f3928097ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3538babb6a27aeb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3920ff9700 1 -- 172.21.15.31:0/1452239427 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392809d380 con 0x5589acafe6b0 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3920ff9700 1 -- 172.21.15.31:0/1452239427 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3928099510 con 0x5589acafe6b0 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.080+0000 7f3920ff9700 1 -- 172.21.15.31:0/1452239427 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280a14d0 con 0x5589acafe6b0 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 -- 172.21.15.31:0/1452239427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 msgr2=0x5589accb1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accb1620 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f3928003730 tx=0x7f3928097ac0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 -- 172.21.15.31:0/1452239427 shutdown_connections 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3f00 0x5589acca7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accb1620 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 --2- 172.21.15.31:0/1452239427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3150 0x5589acca6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 -- 172.21.15.31:0/1452239427 >> 172.21.15.31:0/1452239427 conn(0x5589acc98380 msgr2=0x5589accaa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 -- 172.21.15.31:0/1452239427 shutdown_connections 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.081+0000 7f3947a61b80 1 -- 172.21.15.31:0/1452239427 wait complete. 2023-07-29T16:28:28.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 -- start start 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 Processor -- start 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accaf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accaf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589accb0e80 con 0x5589acca3f00 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589accb0ff0 con 0x5589acca3150 2023-07-29T16:28:28.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589accb11b0 con 0x5589acafe6b0 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3947a61b80 10 client.0 _ll_register_callbacks cb 0x5589acb995c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accaf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f393179a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accaf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3931f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accaf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42906/0 (socket says 172.21.15.31:42906) 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.082+0000 7f3931f9b700 1 -- 172.21.15.31:0/3407223796 learned_addr learned my addr 172.21.15.31:0/3407223796 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f393279c700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3931f9b700 1 -- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 msgr2=0x5589accaf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3931f9b700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accaf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3931f9b700 1 -- 172.21.15.31:0/3407223796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 msgr2=0x5589accb0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3931f9b700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3931f9b700 1 -- 172.21.15.31:0/3407223796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5589aca0ac00 con 0x5589acafe6b0 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f393279c700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3931f9b700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accaf3f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f392801b040 tx=0x7f39280092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=149f231b64f99c8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f39157fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:28.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f39157fa700 1 -- 172.21.15.31:0/3407223796 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928098710 con 0x5589acafe6b0 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f39157fa700 1 -- 172.21.15.31:0/3407223796 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3928098470 con 0x5589acafe6b0 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.083+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f392809d540 con 0x5589acafe6b0 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f39157fa700 1 -- 172.21.15.31:0/3407223796 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3928099980 con 0x5589acafe6b0 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 10 client.29308 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5589aca20250 con 0x5589acafe6b0 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f39157fa700 1 -- 172.21.15.31:0/3407223796 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f392809d540 con 0x5589acafe6b0 2023-07-29T16:28:28.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f39157fa700 1 -- 172.21.15.31:0/3407223796 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f39280af070 con 0x5589acafe6b0 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 10 client.29308 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 10 client.29308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 20 client.29308 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589acd09fb0 0x5589acd0c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 4 client.29308 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5589acbecce0 con 0x5589acd09fb0 2023-07-29T16:28:28.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.084+0000 7f393179a700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589acd09fb0 0x5589acd0c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.085+0000 7f393179a700 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589acd09fb0 0x5589acd0c390 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a132b74b8c77929 server_cookie=e805b835feeea1be in_seq=0 out_seq=0 2023-07-29T16:28:28.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.085+0000 7f39157fa700 10 client.29308 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.160+0000 7f39157fa700 1 -- 172.21.15.31:0/3407223796 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5589acbecce0 con 0x5589acd09fb0 2023-07-29T16:28:28.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.160+0000 7f39157fa700 10 client.29308 handle_command_reply: tid=0 2023-07-29T16:28:28.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.160+0000 7f3947a61b80 1 client.29308 shutdown 2023-07-29T16:28:28.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 msgr2=0x5589accaf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accaf3f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f392801b040 tx=0x7f39280092d0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 shutdown_connections 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5589acd09fb0 0x5589acd0c390 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589acca3f00 0x5589accb0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5589acafe6b0 0x5589accaf3f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 --2- 172.21.15.31:0/3407223796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5589acca3150 0x5589accaf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f39157fa700 0 client.29308 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 >> 172.21.15.31:0/3407223796 conn(0x5589acc98380 msgr2=0x5589accaa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 shutdown_connections 2023-07-29T16:28:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 1 -- 172.21.15.31:0/3407223796 wait complete. 2023-07-29T16:28:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.161+0000 7f3947a61b80 20 client.29308 trim_cache size 0 max 16384 2023-07-29T16:28:28.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:28:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:28:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:28:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:28:28.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39180a0790 msgr2=0x7f39180a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39180a0790 0x7f39180a2c40 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f391c00b2e0 tx=0x7f391c005c90 comp rx=0 tx=0).stop 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 msgr2=0x7f392c198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 0x7f392c198080 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f392800e970 tx=0x7f392800ec80 comp rx=0 tx=0).stop 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 shutdown_connections 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f39180a0790 0x7f39180a2c40 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f392c103f00 0x7f392c19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f392c102380 0x7f392c198080 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 --2- 172.21.15.31:0/2601787431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f392c103570 0x7f392c1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 >> 172.21.15.31:0/2601787431 conn(0x7f392c0fd9c0 msgr2=0x7f392c0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 shutdown_connections 2023-07-29T16:28:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:28.162+0000 7f3932f9d700 1 -- 172.21.15.31:0/2601787431 wait complete. 2023-07-29T16:28:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:27 smithi039 ceph-mon[128858]: pgmap v761: 97 pgs: 97 active+clean; 186 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 257 op/s 2023-07-29T16:28:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:27 smithi156 ceph-mon[129245]: pgmap v761: 97 pgs: 97 active+clean; 186 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 257 op/s 2023-07-29T16:28:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:27 smithi031 ceph-mon[95504]: pgmap v761: 97 pgs: 97 active+clean; 186 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.2 MiB/s wr, 257 op/s 2023-07-29T16:28:28.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:28:28.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 -- 172.21.15.31:0/1494337836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4100330 msgr2=0x7f06f4100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 --2- 172.21.15.31:0/1494337836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4100330 0x7f06f4100740 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009990 tx=0x7f06e4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 -- 172.21.15.31:0/1494337836 shutdown_connections 2023-07-29T16:28:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 --2- 172.21.15.31:0/1494337836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4101eb0 0x7f06f4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 --2- 172.21.15.31:0/1494337836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4100330 0x7f06f4100740 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 --2- 172.21.15.31:0/1494337836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 0x7f06f4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 -- 172.21.15.31:0/1494337836 >> 172.21.15.31:0/1494337836 conn(0x7f06f40fb970 msgr2=0x7f06f40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 -- 172.21.15.31:0/1494337836 shutdown_connections 2023-07-29T16:28:28.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.743+0000 7f06f8bff700 1 -- 172.21.15.31:0/1494337836 wait complete. 2023-07-29T16:28:28.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.744+0000 7f06f8bff700 1 Processor -- start 2023-07-29T16:28:28.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.744+0000 7f06f8bff700 1 -- start start 2023-07-29T16:28:28.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f8bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4100330 0x7f06f4196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f8bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 0x7f06f4196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f8bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4101eb0 0x7f06f419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f8bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f419dbc0 con 0x7f06f4100330 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f8bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f419dd00 con 0x7f06f4101520 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f8bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f419de90 con 0x7f06f4101eb0 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4100330 0x7f06f4196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4101eb0 0x7f06f419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 0x7f06f4196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4100330 0x7f06f4196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46860/0 (socket says 172.21.15.31:46860) 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4101eb0 0x7f06f419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42930/0 (socket says 172.21.15.31:42930) 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f259c700 1 -- 172.21.15.31:0/512305480 learned_addr learned my addr 172.21.15.31:0/512305480 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f1d9b700 1 -- 172.21.15.31:0/512305480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4101eb0 msgr2=0x7f06f419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4101eb0 0x7f06f419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f1d9b700 1 -- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4100330 msgr2=0x7f06f4196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4100330 0x7f06f4196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.745+0000 7f06f1d9b700 1 -- 172.21.15.31:0/512305480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ec009c40 con 0x7f06f4101520 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.746+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 0x7f06f4196580 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f06e800e9f0 tx=0x7f06e800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.746+0000 7f06e37fe700 1 -- 172.21.15.31:0/512305480 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80096b0 con 0x7f06f4101520 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.746+0000 7f06e37fe700 1 -- 172.21.15.31:0/512305480 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06e8004cf0 con 0x7f06f4101520 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.746+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e40095f0 con 0x7f06f4101520 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.746+0000 7f06e37fe700 1 -- 172.21.15.31:0/512305480 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e800cdb0 con 0x7f06f4101520 2023-07-29T16:28:28.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.746+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f4073310 con 0x7f06f4101520 2023-07-29T16:28:28.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.750+0000 7f06e37fe700 1 -- 172.21.15.31:0/512305480 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f06e8004320 con 0x7f06f4101520 2023-07-29T16:28:28.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.751+0000 7f070c439b80 1 Processor -- start 2023-07-29T16:28:28.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.751+0000 7f070c439b80 1 -- start start 2023-07-29T16:28:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.751+0000 7f06e37fe700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06dc0a0790 0x7f06dc0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.751+0000 7f070c439b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x562115393ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f070c439b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115394530 0x562115394d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f070c439b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115395740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f070c439b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621152eba10 con 0x562115394530 2023-07-29T16:28:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f070c439b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621152e93d0 con 0x5621153952f0 2023-07-29T16:28:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f070c439b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621152e4cc0 con 0x562115393be0 2023-07-29T16:28:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06e37fe700 1 -- 172.21.15.31:0/512305480 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f06e8014070 con 0x7f06f4101520 2023-07-29T16:28:28.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06f2d9d700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06dc0a0790 0x7f06dc0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115395740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115395740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51872/0 (socket says 172.21.15.31:51872) 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06f2d9d700 1 -- 172.21.15.31:0/3900085057 learned_addr learned my addr 172.21.15.31:0/3900085057 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06f259c700 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x562115393ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.752+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115394530 0x562115394d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f259c700 1 -- 172.21.15.31:0/3900085057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 msgr2=0x562115395740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f259c700 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115395740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f259c700 1 -- 172.21.15.31:0/3900085057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115394530 msgr2=0x562115394d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f259c700 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115394530 0x562115394d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f259c700 1 -- 172.21.15.31:0/3900085057 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562115112de0 con 0x562115393be0 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f2d9d700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06dc0a0790 0x7f06dc0a2c40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f06ec008e50 tx=0x7f06ec008510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06f259c700 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x562115393ff0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006010 tx=0x7f06e4005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11c6dd1ae6a10eb1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06e0ff9700 1 -- 172.21.15.31:0/3900085057 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4021050 con 0x562115393be0 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06e0ff9700 1 -- 172.21.15.31:0/3900085057 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06e4022040 con 0x562115393be0 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.753+0000 7f06e0ff9700 1 -- 172.21.15.31:0/3900085057 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e40055a0 con 0x562115393be0 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 -- 172.21.15.31:0/3900085057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 msgr2=0x562115393ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x562115393ff0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f06e4006010 tx=0x7f06e4005c90 comp rx=0 tx=0).stop 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 -- 172.21.15.31:0/3900085057 shutdown_connections 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115395740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x562115393ff0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 --2- 172.21.15.31:0/3900085057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115394530 0x562115394d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 -- 172.21.15.31:0/3900085057 >> 172.21.15.31:0/3900085057 conn(0x5621153893e0 msgr2=0x56211538e990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 -- 172.21.15.31:0/3900085057 shutdown_connections 2023-07-29T16:28:28.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.754+0000 7f070c439b80 1 -- 172.21.15.31:0/3900085057 wait complete. 2023-07-29T16:28:28.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.755+0000 7f070c439b80 1 -- start start 2023-07-29T16:28:28.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.755+0000 7f070c439b80 1 Processor -- start 2023-07-29T16:28:28.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x56211539dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621153952f0 0x5621153a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562115281510 con 0x562115393be0 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621152811b0 con 0x562115394530 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562115168920 con 0x5621153952f0 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f070c439b80 10 client.0 _ll_register_callbacks cb 0x56211528a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621153952f0 0x5621153a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46872/0 (socket says 172.21.15.31:46872) 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 -- 172.21.15.31:0/3851044367 learned_addr learned my addr 172.21.15.31:0/3851044367 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x56211539dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 -- 172.21.15.31:0/3851044367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621153952f0 msgr2=0x5621153a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621153952f0 0x5621153a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 -- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 msgr2=0x56211539dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x56211539dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.756+0000 7f06f259c700 1 -- 172.21.15.31:0/3851044367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f41090a0 con 0x562115393be0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f06f259c700 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539d7d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009ce0 tx=0x7f06e4005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702f9c8b4ae05e25 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f06d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f06d9ffb700 1 -- 172.21.15.31:0/3851044367 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e400bc60 con 0x562115393be0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56211530fd10 con 0x562115393be0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f06d9ffb700 1 -- 172.21.15.31:0/3851044367 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f06e400bdc0 con 0x562115393be0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f06d9ffb700 1 -- 172.21.15.31:0/3851044367 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06e4027460 con 0x562115393be0 2023-07-29T16:28:28.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f070c439b80 10 client.19212 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.757+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5621150fbc60 con 0x562115393be0 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f06d9ffb700 1 -- 172.21.15.31:0/3851044367 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56211530fd10 con 0x562115393be0 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f06d9ffb700 1 -- 172.21.15.31:0/3851044367 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f06e400f600 con 0x562115393be0 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f070c439b80 10 client.19212 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f070c439b80 10 client.19212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f070c439b80 20 client.19212 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f070c439b80 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153a9080 0x5621153ab490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f070c439b80 4 client.19212 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562115362d70 con 0x5621153a9080 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.758+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153a9080 0x5621153ab490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.759+0000 7f06d9ffb700 10 client.19212 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.759+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153a9080 0x5621153ab490 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d72d0a31bf3ff5a9 server_cookie=767ae018a033125b in_seq=0 out_seq=0 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f06d9ffb700 1 -- 172.21.15.31:0/3851044367 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562115362d70 con 0x5621153a9080 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f06d9ffb700 10 client.19212 handle_command_reply: tid=0 2023-07-29T16:28:28.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 client.19212 shutdown 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 msgr2=0x56211539d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539d7d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009ce0 tx=0x7f06e4005550 comp rx=0 tx=0).stop 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 shutdown_connections 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153a9080 0x5621153ab490 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621153952f0 0x5621153a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539d7d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f06d9ffb700 0 client.19212 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 --2- 172.21.15.31:0/3851044367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x56211539dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 >> 172.21.15.31:0/3851044367 conn(0x5621153893e0 msgr2=0x56211538e060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 shutdown_connections 2023-07-29T16:28:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 1 -- 172.21.15.31:0/3851044367 wait complete. 2023-07-29T16:28:28.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.760+0000 7f070c439b80 20 client.19212 trim_cache size 0 max 16384 2023-07-29T16:28:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 Processor -- start 2023-07-29T16:28:28.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 -- start start 2023-07-29T16:28:28.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115394530 0x5621153a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115398190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56211539d3a0 con 0x562115393be0 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56211538b460 con 0x5621153952f0 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f070c439b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562115390310 con 0x562115394530 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115398190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115398190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51882/0 (socket says 172.21.15.31:51882) 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f06f2d9d700 1 -- 172.21.15.31:0/340053110 learned_addr learned my addr 172.21.15.31:0/340053110 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.778+0000 7f06f259c700 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115394530 0x5621153a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f2d9d700 1 -- 172.21.15.31:0/340053110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115394530 msgr2=0x5621153a4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f2d9d700 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115394530 0x5621153a4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f2d9d700 1 -- 172.21.15.31:0/340053110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 msgr2=0x56211539b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f2d9d700 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f2d9d700 1 -- 172.21.15.31:0/340053110 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562115139250 con 0x5621153952f0 2023-07-29T16:28:28.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06f2d9d700 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115398190 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f06ec01abe0 tx=0x7f06ec01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c629821e85989412 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06e17fa700 1 -- 172.21.15.31:0/340053110 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec00e070 con 0x5621153952f0 2023-07-29T16:28:28.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06e17fa700 1 -- 172.21.15.31:0/340053110 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06ec00f040 con 0x5621153952f0 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.779+0000 7f06e17fa700 1 -- 172.21.15.31:0/340053110 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0044a0 con 0x5621153952f0 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 -- 172.21.15.31:0/340053110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 msgr2=0x562115398190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115398190 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f06ec01abe0 tx=0x7f06ec01d000 comp rx=0 tx=0).stop 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 -- 172.21.15.31:0/340053110 shutdown_connections 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621153952f0 0x562115398190 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562115393be0 0x56211539b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 --2- 172.21.15.31:0/340053110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115394530 0x5621153a4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.780+0000 7f070c439b80 1 -- 172.21.15.31:0/340053110 >> 172.21.15.31:0/340053110 conn(0x5621153893e0 msgr2=0x56211539c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.781+0000 7f070c439b80 1 -- 172.21.15.31:0/340053110 shutdown_connections 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.781+0000 7f070c439b80 1 -- 172.21.15.31:0/340053110 wait complete. 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.781+0000 7f070c439b80 1 -- start start 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.781+0000 7f070c439b80 1 Processor -- start 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x5621153a4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x5621153a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621153952f0 0x5621153a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621153a1e30 con 0x5621153952f0 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621153a1fa0 con 0x562115394530 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621153fabe0 con 0x562115393be0 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f070c439b80 10 client.0 _ll_register_callbacks cb 0x56211528a620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f06f1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x5621153a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f06f1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x5621153a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51892/0 (socket says 172.21.15.31:51892) 2023-07-29T16:28:28.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f06f1d9b700 1 -- 172.21.15.31:0/4127686668 learned_addr learned my addr 172.21.15.31:0/4127686668 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f06f2d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621153952f0 0x5621153a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.782+0000 7f06f259c700 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x5621153a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06f1d9b700 1 -- 172.21.15.31:0/4127686668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 msgr2=0x5621153a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x5621153a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06f1d9b700 1 -- 172.21.15.31:0/4127686668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621153952f0 msgr2=0x5621153a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621153952f0 0x5621153a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06f1d9b700 1 -- 172.21.15.31:0/4127686668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec009040 con 0x562115394530 2023-07-29T16:28:28.787 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06f1d9b700 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x5621153a13b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f06e80958a0 tx=0x7f06e800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b0df2d0b54ae9e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06d9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06d9ffb700 1 -- 172.21.15.31:0/4127686668 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80993a0 con 0x562115394530 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06d9ffb700 1 -- 172.21.15.31:0/4127686668 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f06e809d3d0 con 0x562115394530 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06e4022040 con 0x562115394530 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f06d9ffb700 1 -- 172.21.15.31:0/4127686668 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06e80a17f0 con 0x562115394530 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f070c439b80 10 client.29330 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:28.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.783+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5621151112b0 con 0x562115394530 2023-07-29T16:28:28.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f06d9ffb700 1 -- 172.21.15.31:0/4127686668 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06e4022040 con 0x562115394530 2023-07-29T16:28:28.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f06d9ffb700 1 -- 172.21.15.31:0/4127686668 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f06e80652a0 con 0x562115394530 2023-07-29T16:28:28.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f070c439b80 10 client.29330 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:28.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f070c439b80 10 client.29330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:28.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f070c439b80 20 client.29330 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:28.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f070c439b80 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153fc370 0x5621153fe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:28.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f070c439b80 4 client.29330 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5621152ddd40 con 0x5621153fc370 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.784+0000 7f06f259c700 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153fc370 0x5621153fe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.785+0000 7f06f259c700 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153fc370 0x5621153fe780 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34317503e52d97fe server_cookie=2815f84d5469d9ac in_seq=0 out_seq=0 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.785+0000 7f06d9ffb700 10 client.29330 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f06d9ffb700 1 -- 172.21.15.31:0/4127686668 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5621152ddd40 con 0x5621153fc370 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f06d9ffb700 10 client.29330 handle_command_reply: tid=0 2023-07-29T16:28:28.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 client.29330 shutdown 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 msgr2=0x5621153a13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x5621153a13b0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f06e80958a0 tx=0x7f06e800c850 comp rx=0 tx=0).stop 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 shutdown_connections 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621153fc370 0x5621153fe780 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621153952f0 0x5621153a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.786+0000 7f070c439b80 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562115393be0 0x5621153a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f070c439b80 1 --2- 172.21.15.31:0/4127686668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562115394530 0x5621153a13b0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f06d9ffb700 0 client.29330 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:28.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 >> 172.21.15.31:0/4127686668 conn(0x5621153893e0 msgr2=0x56211539b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 shutdown_connections 2023-07-29T16:28:28.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f070c439b80 1 -- 172.21.15.31:0/4127686668 wait complete. 2023-07-29T16:28:28.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f070c439b80 20 client.29330 trim_cache size 0 max 16384 2023-07-29T16:28:28.794 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:28:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:28:28.795 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:28:28.795 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:28:28.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06dc0a0790 msgr2=0x7f06dc0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f06f8bff700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06dc0a0790 0x7f06dc0a2c40 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f06ec008e50 tx=0x7f06ec008510 comp rx=0 tx=0).stop 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 msgr2=0x7f06f4196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.787+0000 7f06f8bff700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 0x7f06f4196580 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f06e800e9f0 tx=0x7f06e800ed00 comp rx=0 tx=0).stop 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 shutdown_connections 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06dc0a0790 0x7f06dc0a2c40 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06f4101eb0 0x7f06f419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06f4100330 0x7f06f4196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 --2- 172.21.15.31:0/512305480 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06f4101520 0x7f06f4196580 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:28.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 >> 172.21.15.31:0/512305480 conn(0x7f06f40fb970 msgr2=0x7f06f4104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:28.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 shutdown_connections 2023-07-29T16:28:28.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:28.788+0000 7f06f8bff700 1 -- 172.21.15.31:0/512305480 wait complete. 2023-07-29T16:28:28.806 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:28:28.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:28:28.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1013307 8.92 MB/sec execute 493 sec latency 68.601 ms 2023-07-29T16:28:29.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 -- 172.21.15.31:0/1944287099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0100340 msgr2=0x7f6fb0100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/1944287099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0100340 0x7f6fb0100750 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f6fac009990 tx=0x7f6fac009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 -- 172.21.15.31:0/1944287099 shutdown_connections 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/1944287099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fb0101ec0 0x7f6fb0108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/1944287099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0100340 0x7f6fb0100750 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/1944287099 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0101530 0x7f6fb0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 -- 172.21.15.31:0/1944287099 >> 172.21.15.31:0/1944287099 conn(0x7f6fb00fb920 msgr2=0x7f6fb00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 -- 172.21.15.31:0/1944287099 shutdown_connections 2023-07-29T16:28:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.027+0000 7f6fb7bff700 1 -- 172.21.15.31:0/1944287099 wait complete. 2023-07-29T16:28:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.028+0000 7f6fb7bff700 1 Processor -- start 2023-07-29T16:28:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 -- start start 2023-07-29T16:28:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 0x7f6fb0074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fb0101530 0x7f6fb0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0101ec0 0x7f6fb0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0073e20 con 0x7f6fb0101530 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0073f90 con 0x7f6fb0101ec0 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb7bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb01a00e0 con 0x7f6fb0100340 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 0x7f6fb0074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb599b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 0x7f6fb0074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42980/0 (socket says 172.21.15.31:42980) 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb599b700 1 -- 172.21.15.31:0/2842169375 learned_addr learned my addr 172.21.15.31:0/2842169375 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb519a700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fb0101530 0x7f6fb0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.029+0000 7f6fb619c700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0101ec0 0x7f6fb0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb599b700 1 -- 172.21.15.31:0/2842169375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0101ec0 msgr2=0x7f6fb0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb599b700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0101ec0 0x7f6fb0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb599b700 1 -- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fb0101530 msgr2=0x7f6fb0073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb599b700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fb0101530 0x7f6fb0073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb599b700 1 -- 172.21.15.31:0/2842169375 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac0095f0 con 0x7f6fb0100340 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb619c700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0101ec0 0x7f6fb0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:29.032 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb599b700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 0x7f6fb0074b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00b620 tx=0x7f6fac005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac01d070 con 0x7f6fb0100340 2023-07-29T16:28:29.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fac021410 con 0x7f6fb0100340 2023-07-29T16:28:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb01a0360 con 0x7f6fb0100340 2023-07-29T16:28:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.030+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac017650 con 0x7f6fb0100340 2023-07-29T16:28:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.031+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb01a07e0 con 0x7f6fb0100340 2023-07-29T16:28:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.032+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb01078b0 con 0x7f6fb0100340 2023-07-29T16:28:29.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.034+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6fac021580 con 0x7f6fb0100340 2023-07-29T16:28:29.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.035+0000 7f6fa6ffd700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f9c0a0860 0x7f6f9c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.035+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6fac0cd080 con 0x7f6fb0100340 2023-07-29T16:28:29.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.035+0000 7f6fb519a700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f9c0a0860 0x7f6f9c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.036+0000 7f6fb519a700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f9c0a0860 0x7f6f9c0a2d10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009c60 tx=0x7f6fa0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.037+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f6fac096af0 con 0x7f6fb0100340 2023-07-29T16:28:29.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.298+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fb002cc30 con 0x7f6fb0100340 2023-07-29T16:28:29.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.298+0000 7f6fa6ffd700 1 -- 172.21.15.31:0/2842169375 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10781 (secure 0 0 0) 0x7f6fac026090 con 0x7f6fb0100340 2023-07-29T16:28:29.299 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:28:29.299 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:27:32.973421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:28:29.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f9c0a0860 msgr2=0x7f6f9c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f9c0a0860 0x7f6f9c0a2d10 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009c60 tx=0x7f6fa0009260 comp rx=0 tx=0).stop 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 msgr2=0x7f6fb0074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 0x7f6fb0074b00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00b620 tx=0x7f6fac005740 comp rx=0 tx=0).stop 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 shutdown_connections 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6f9c0a0860 0x7f6f9c0a2d10 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6fb0101ec0 0x7f6fb0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fb0100340 0x7f6fb0074b00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 --2- 172.21.15.31:0/2842169375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6fb0101530 0x7f6fb0073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.301+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 >> 172.21.15.31:0/2842169375 conn(0x7f6fb00fb920 msgr2=0x7f6fb01049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.302+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 shutdown_connections 2023-07-29T16:28:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.302+0000 7f6fb7bff700 1 -- 172.21.15.31:0/2842169375 wait complete. 2023-07-29T16:28:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2023-07-29T16:28:29.324 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:28:29.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 -- 172.21.15.31:0/186413814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 msgr2=0x7f7218100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 --2- 172.21.15.31:0/186413814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218100620 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f720c009990 tx=0x7f720c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 -- 172.21.15.31:0/186413814 shutdown_connections 2023-07-29T16:28:29.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 --2- 172.21.15.31:0/186413814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7218101e50 0x7f72181086f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 --2- 172.21.15.31:0/186413814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218100620 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 --2- 172.21.15.31:0/186413814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f72181014a0 0x7f7218101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 -- 172.21.15.31:0/186413814 >> 172.21.15.31:0/186413814 conn(0x7f72180fb850 msgr2=0x7f72180fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.557+0000 7f721df9d700 1 -- 172.21.15.31:0/186413814 shutdown_connections 2023-07-29T16:28:29.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.558+0000 7f721df9d700 1 -- 172.21.15.31:0/186413814 wait complete. 2023-07-29T16:28:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.558+0000 7f721df9d700 1 Processor -- start 2023-07-29T16:28:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 -- start start 2023-07-29T16:28:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72181014a0 0x7f7218073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7218101e50 0x7f7218073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7218073e20 con 0x7f72181014a0 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7218073f90 con 0x7f7218101e50 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721df9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721819fec0 con 0x7f7218100210 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f7217fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72181014a0 0x7f7218073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42996/0 (socket says 172.21.15.31:42996) 2023-07-29T16:28:29.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721d79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7218101e50 0x7f7218073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.559+0000 7f721cf9b700 1 -- 172.21.15.31:0/518627258 learned_addr learned my addr 172.21.15.31:0/518627258 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721cf9b700 1 -- 172.21.15.31:0/518627258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7218101e50 msgr2=0x7f7218073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721cf9b700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7218101e50 0x7f7218073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721cf9b700 1 -- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72181014a0 msgr2=0x7f7218073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721cf9b700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72181014a0 0x7f7218073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721cf9b700 1 -- 172.21.15.31:0/518627258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c0095f0 con 0x7f7218100210 2023-07-29T16:28:29.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721d79c700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7218101e50 0x7f7218073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:29.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721cf9b700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218074b00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f720c00b680 tx=0x7f720c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f7215ffb700 1 -- 172.21.15.31:0/518627258 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c01d070 con 0x7f7218100210 2023-07-29T16:28:29.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f7215ffb700 1 -- 172.21.15.31:0/518627258 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720c022470 con 0x7f7218100210 2023-07-29T16:28:29.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f7215ffb700 1 -- 172.21.15.31:0/518627258 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c017650 con 0x7f7218100210 2023-07-29T16:28:29.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72181a0140 con 0x7f7218100210 2023-07-29T16:28:29.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.560+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72181a0510 con 0x7f7218100210 2023-07-29T16:28:29.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.564+0000 7f7215ffb700 1 -- 172.21.15.31:0/518627258 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f720c0225e0 con 0x7f7218100210 2023-07-29T16:28:29.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.565+0000 7f7232a38b80 1 Processor -- start 2023-07-29T16:28:29.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.565+0000 7f7215ffb700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f72080a08b0 0x7f72080a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.565+0000 7f7232a38b80 1 -- start start 2023-07-29T16:28:29.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.565+0000 7f7215ffb700 1 -- 172.21.15.31:0/518627258 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f720c0ccef0 con 0x7f7218100210 2023-07-29T16:28:29.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721d79c700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f72080a08b0 0x7f72080a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d272150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d272690 0x561c3d272f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d2734d0 0x561c3d273920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7232a38b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d1c9a10 con 0x561c3d272690 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7232a38b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d1c73d0 con 0x561c3d271d40 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7232a38b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d1c2cc0 con 0x561c3d2734d0 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d272150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d272150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51936/0 (socket says 172.21.15.31:51936) 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721cf9b700 1 -- 172.21.15.31:0/3935089268 learned_addr learned my addr 172.21.15.31:0/3935089268 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721d79c700 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d2734d0 0x561c3d273920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f7217fff700 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d272690 0x561c3d272f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721cf9b700 1 -- 172.21.15.31:0/3935089268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d2734d0 msgr2=0x561c3d273920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.566+0000 7f721cf9b700 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d2734d0 0x561c3d273920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f721cf9b700 1 -- 172.21.15.31:0/3935089268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d272690 msgr2=0x561c3d272f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f721cf9b700 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d272690 0x561c3d272f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f721cf9b700 1 -- 172.21.15.31:0/3935089268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c3cff0de0 con 0x561c3d271d40 2023-07-29T16:28:29.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f721d79c700 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d2734d0 0x561c3d273920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:29.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f721d79c700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f72080a08b0 0x7f72080a2d60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7218075160 tx=0x7f7210009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f721cf9b700 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d272150 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f720c005f50 tx=0x7f720c022c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d212949a3b895fd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f7202ffd700 1 -- 172.21.15.31:0/3935089268 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c09a620 con 0x561c3d271d40 2023-07-29T16:28:29.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f7202ffd700 1 -- 172.21.15.31:0/3935089268 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720c09e650 con 0x561c3d271d40 2023-07-29T16:28:29.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.567+0000 7f7202ffd700 1 -- 172.21.15.31:0/3935089268 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c099b10 con 0x561c3d271d40 2023-07-29T16:28:29.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 -- 172.21.15.31:0/3935089268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 msgr2=0x561c3d272150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d272150 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f720c005f50 tx=0x7f720c022c80 comp rx=0 tx=0).stop 2023-07-29T16:28:29.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 -- 172.21.15.31:0/3935089268 shutdown_connections 2023-07-29T16:28:29.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d2734d0 0x561c3d273920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d272150 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 --2- 172.21.15.31:0/3935089268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d272690 0x561c3d272f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 -- 172.21.15.31:0/3935089268 >> 172.21.15.31:0/3935089268 conn(0x561c3d267540 msgr2=0x561c3d26cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 -- 172.21.15.31:0/3935089268 shutdown_connections 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.568+0000 7f7232a38b80 1 -- 172.21.15.31:0/3935089268 wait complete. 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 -- start start 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 Processor -- start 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d271d40 0x561c3d27b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 0x561c3d27bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d282bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.574 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d15f510 con 0x561c3d2734d0 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d15f1b0 con 0x561c3d272690 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d046920 con 0x561c3d271d40 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7232a38b80 10 client.0 _ll_register_callbacks cb 0x561c3d168620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.569+0000 7f7217fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 0x561c3d27bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d271d40 0x561c3d27b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 0x561c3d27bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51938/0 (socket says 172.21.15.31:51938) 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 -- 172.21.15.31:0/313733595 learned_addr learned my addr 172.21.15.31:0/313733595 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.575 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f721d79c700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d282bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 -- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d271d40 msgr2=0x561c3d27b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d271d40 0x561c3d27b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 -- 172.21.15.31:0/313733595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 msgr2=0x561c3d282bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d282bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 -- 172.21.15.31:0/313733595 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72181a0510 con 0x561c3d272690 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f721cf9b700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d271d40 0x561c3d27b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:29.576 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f7217fff700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 0x561c3d27bb70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f72040126d0 tx=0x7f72040129e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac23cb0e134399cf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.570+0000 7f71f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f71f7fff700 1 -- 172.21.15.31:0/313733595 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204017070 con 0x561c3d272690 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f71f7fff700 1 -- 172.21.15.31:0/313733595 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7204004050 con 0x561c3d272690 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c3d1edd10 con 0x561c3d272690 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f71f7fff700 1 -- 172.21.15.31:0/313733595 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7204004210 con 0x561c3d272690 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f7232a38b80 10 client.29348 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:29.577 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561c3cfd9c60 con 0x561c3d272690 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f71f7fff700 1 -- 172.21.15.31:0/313733595 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561c3d1edd10 con 0x561c3d272690 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f71f7fff700 1 -- 172.21.15.31:0/313733595 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f7204010430 con 0x561c3d272690 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f7232a38b80 10 client.29348 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.571+0000 7f7232a38b80 10 client.29348 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f7232a38b80 20 client.29348 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f7232a38b80 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2873f0 0x561c3d289800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f7232a38b80 4 client.29348 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:29.578 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c3d240d70 con 0x561c3d2873f0 2023-07-29T16:28:29.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f721cf9b700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2873f0 0x561c3d289800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f721cf9b700 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2873f0 0x561c3d289800 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4abb593cf8d355e3 server_cookie=e22f067b71ec8ab2 in_seq=0 out_seq=0 2023-07-29T16:28:29.579 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.572+0000 7f71f7fff700 10 client.29348 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:28:29.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f71f7fff700 1 -- 172.21.15.31:0/313733595 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561c3d240d70 con 0x561c3d2873f0 2023-07-29T16:28:29.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f71f7fff700 10 client.29348 handle_command_reply: tid=0 2023-07-29T16:28:29.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 client.29348 shutdown 2023-07-29T16:28:29.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 msgr2=0x561c3d27bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.580 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 0x561c3d27bb70 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f72040126d0 tx=0x7f72040129e0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 shutdown_connections 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d282bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d271d40 0x561c3d27b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2873f0 0x561c3d289800 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 --2- 172.21.15.31:0/313733595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d272690 0x561c3d27bb70 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f71f7fff700 0 client.29348 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 >> 172.21.15.31:0/313733595 conn(0x561c3d267540 msgr2=0x561c3d26c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.581 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.573+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 shutdown_connections 2023-07-29T16:28:29.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.574+0000 7f7232a38b80 1 -- 172.21.15.31:0/313733595 wait complete. 2023-07-29T16:28:29.582 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.574+0000 7f7232a38b80 20 client.29348 trim_cache size 0 max 16384 2023-07-29T16:28:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.587+0000 7f7232a38b80 1 Processor -- start 2023-07-29T16:28:29.587 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.587+0000 7f7232a38b80 1 -- start start 2023-07-29T16:28:29.588 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.587+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.587+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d276370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.587+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d2768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7232a38b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d27b1a0 con 0x561c3d2734d0 2023-07-29T16:28:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7232a38b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d2648d0 con 0x561c3d271d40 2023-07-29T16:28:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7232a38b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d26e250 con 0x561c3d272690 2023-07-29T16:28:29.589 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d276370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d276370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:43010/0 (socket says 172.21.15.31:43010) 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 -- 172.21.15.31:0/1336100811 learned_addr learned my addr 172.21.15.31:0/1336100811 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f721d79c700 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d2768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 -- 172.21.15.31:0/1336100811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 msgr2=0x561c3d280ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 -- 172.21.15.31:0/1336100811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 msgr2=0x561c3d2768b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d2768b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 -- 172.21.15.31:0/1336100811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c3d017250 con 0x561c3d272690 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f721d79c700 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d2768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f721cf9b700 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.588+0000 7f7217fff700 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d276370 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f72040107c0 tx=0x7f7204005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17010662ac90dade server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.589+0000 7f72037fe700 1 -- 172.21.15.31:0/1336100811 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7204017070 con 0x561c3d272690 2023-07-29T16:28:29.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.589+0000 7f72037fe700 1 -- 172.21.15.31:0/1336100811 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7204018040 con 0x561c3d272690 2023-07-29T16:28:29.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.589+0000 7f72037fe700 1 -- 172.21.15.31:0/1336100811 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72040038e0 con 0x561c3d272690 2023-07-29T16:28:29.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.589+0000 7f7232a38b80 1 -- 172.21.15.31:0/1336100811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 msgr2=0x561c3d276370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.589+0000 7f7232a38b80 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d276370 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f72040107c0 tx=0x7f7204005dc0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 -- 172.21.15.31:0/1336100811 shutdown_connections 2023-07-29T16:28:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d2768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 --2- 172.21.15.31:0/1336100811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d276370 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 -- 172.21.15.31:0/1336100811 >> 172.21.15.31:0/1336100811 conn(0x561c3d267540 msgr2=0x561c3d27b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 -- 172.21.15.31:0/1336100811 shutdown_connections 2023-07-29T16:28:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 -- 172.21.15.31:0/1336100811 wait complete. 2023-07-29T16:28:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 -- start start 2023-07-29T16:28:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.590+0000 7f7232a38b80 1 Processor -- start 2023-07-29T16:28:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d27ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d27f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d27fa20 con 0x561c3d2734d0 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d27fb90 con 0x561c3d271d40 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c3d27fd50 con 0x561c3d272690 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7232a38b80 10 client.0 _ll_register_callbacks cb 0x561c3d168620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f721cf9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51942/0 (socket says 172.21.15.31:51942) 2023-07-29T16:28:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f721cf9b700 1 -- 172.21.15.31:0/530575066 learned_addr learned my addr 172.21.15.31:0/530575066 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f7217fff700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d27ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.591+0000 7f721d79c700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d27f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721cf9b700 1 -- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 msgr2=0x561c3d27ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721cf9b700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d27ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721cf9b700 1 -- 172.21.15.31:0/530575066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 msgr2=0x561c3d27f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721cf9b700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d27f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721cf9b700 1 -- 172.21.15.31:0/530575066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c3cff0de0 con 0x561c3d271d40 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721d79c700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d27f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f721cf9b700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280850 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f720c022890 tx=0x7f720c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7fbc7b0a5fa6824 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f71f7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f71f7fff700 1 -- 172.21.15.31:0/530575066 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720c099b10 con 0x561c3d271d40 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f71f7fff700 1 -- 172.21.15.31:0/530575066 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f720c099c70 con 0x561c3d271d40 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f720c09e650 con 0x561c3d271d40 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f71f7fff700 1 -- 172.21.15.31:0/530575066 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f720c09a880 con 0x561c3d271d40 2023-07-29T16:28:29.599 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f7232a38b80 10 client.29354 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.592+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x561c3cfd9c60 con 0x561c3d271d40 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f71f7fff700 1 -- 172.21.15.31:0/530575066 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720c09e650 con 0x561c3d271d40 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f71f7fff700 1 -- 172.21.15.31:0/530575066 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f720c096410 con 0x561c3d271d40 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7232a38b80 10 client.29354 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7232a38b80 10 client.29354 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7232a38b80 20 client.29354 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7232a38b80 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2da2c0 0x561c3d2dc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7232a38b80 4 client.29354 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:28:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c3d1bbd40 con 0x561c3d2da2c0 2023-07-29T16:28:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.593+0000 7f7217fff700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2da2c0 0x561c3d2dc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7217fff700 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2da2c0 0x561c3d2dc6d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cfdd301e4592ea06 server_cookie=d2fe2e718f40ea39 in_seq=0 out_seq=0 2023-07-29T16:28:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f71f7fff700 10 client.29354 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:28:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f71f7fff700 1 -- 172.21.15.31:0/530575066 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c3d1bbd40 con 0x561c3d2da2c0 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f71f7fff700 10 client.29354 handle_command_reply: tid=0 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 client.29354 shutdown 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 msgr2=0x561c3d280850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280850 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f720c022890 tx=0x7f720c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 shutdown_connections 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561c3d2734d0 0x561c3d27f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x561c3d2da2c0 0x561c3d2dc6d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561c3d271d40 0x561c3d280850 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 --2- 172.21.15.31:0/530575066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561c3d272690 0x561c3d27ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f71f7fff700 0 client.29354 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:28:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 >> 172.21.15.31:0/530575066 conn(0x561c3d267540 msgr2=0x561c3d27b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 shutdown_connections 2023-07-29T16:28:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 1 -- 172.21.15.31:0/530575066 wait complete. 2023-07-29T16:28:29.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.594+0000 7f7232a38b80 20 client.29354 trim_cache size 0 max 16384 2023-07-29T16:28:29.604 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:28:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f72080a08b0 msgr2=0x7f72080a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f72080a08b0 0x7f72080a2d60 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7218075160 tx=0x7f7210009920 comp rx=0 tx=0).stop 2023-07-29T16:28:29.604 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 msgr2=0x7f7218074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218074b00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f720c00b680 tx=0x7f720c005740 comp rx=0 tx=0).stop 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 shutdown_connections 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f72080a08b0 0x7f72080a2d60 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7218101e50 0x7f7218073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7218100210 0x7f7218074b00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 --2- 172.21.15.31:0/518627258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72181014a0 0x7f7218073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 >> 172.21.15.31:0/518627258 conn(0x7f72180fb850 msgr2=0x7f72181048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 shutdown_connections 2023-07-29T16:28:29.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.595+0000 7f721df9d700 1 -- 172.21.15.31:0/518627258 wait complete. 2023-07-29T16:28:29.610 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:29 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2842169375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:28:29.613 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:28:29.775 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:29 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2842169375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:28:29.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:29 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2842169375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:28:29.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 -- 172.21.15.31:0/2215727883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 msgr2=0x7f01c8073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 --2- 172.21.15.31:0/2215727883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c8073c20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f01b8009990 tx=0x7f01b8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 -- 172.21.15.31:0/2215727883 shutdown_connections 2023-07-29T16:28:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 --2- 172.21.15.31:0/2215727883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c81081f0 0x7f01c810a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 --2- 172.21.15.31:0/2215727883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c8074d90 0x7f01c80731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 --2- 172.21.15.31:0/2215727883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c8073c20 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 -- 172.21.15.31:0/2215727883 >> 172.21.15.31:0/2215727883 conn(0x7f01c80fb970 msgr2=0x7f01c80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 -- 172.21.15.31:0/2215727883 shutdown_connections 2023-07-29T16:28:29.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.830+0000 7f01d07ff700 1 -- 172.21.15.31:0/2215727883 wait complete. 2023-07-29T16:28:29.833 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.831+0000 7f01d07ff700 1 Processor -- start 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.831+0000 7f01d07ff700 1 -- start start 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01d07ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c81982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01d07ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c8074d90 0x7f01c8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01d07ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c81081f0 0x7f01c819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01d07ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c819fe50 con 0x7f01c8074d90 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01d07ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c819ff90 con 0x7f01c80737b0 2023-07-29T16:28:29.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01d07ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c81a0120 con 0x7f01c81081f0 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c81982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c81081f0 0x7f01c819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01cdd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c8074d90 0x7f01c8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c81982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51962/0 (socket says 172.21.15.31:51962) 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 -- 172.21.15.31:0/900919313 learned_addr learned my addr 172.21.15.31:0/900919313 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c81081f0 0x7f01c819f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:43044/0 (socket says 172.21.15.31:43044) 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 -- 172.21.15.31:0/900919313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c81081f0 msgr2=0x7f01c819f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c81081f0 0x7f01c819f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 -- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c8074d90 msgr2=0x7f01c8198810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.832+0000 7f01ce59b700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c8074d90 0x7f01c8198810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01ce59b700 1 -- 172.21.15.31:0/900919313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80095f0 con 0x7f01c80737b0 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01ce59b700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c81982d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f01c400e970 tx=0x7f01c400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01bf7fe700 1 -- 172.21.15.31:0/900919313 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4009790 con 0x7f01c80737b0 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01bf7fe700 1 -- 172.21.15.31:0/900919313 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01c400c7e0 con 0x7f01c80737b0 2023-07-29T16:28:29.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c81a0400 con 0x7f01c80737b0 2023-07-29T16:28:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01bf7fe700 1 -- 172.21.15.31:0/900919313 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c40054e0 con 0x7f01c80737b0 2023-07-29T16:28:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.833+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c81a0960 con 0x7f01c80737b0 2023-07-29T16:28:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.836+0000 7f01bf7fe700 1 -- 172.21.15.31:0/900919313 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f01c400c950 con 0x7f01c80737b0 2023-07-29T16:28:29.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.837+0000 7f01e4147b80 1 Processor -- start 2023-07-29T16:28:29.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.837+0000 7f01e4147b80 1 -- start start 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x560822025ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822026d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822027740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01e4147b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560821f7da10 con 0x5608220272f0 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01e4147b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560821f7b3d0 con 0x560822025be0 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01e4147b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560821f76cc0 con 0x560822026530 2023-07-29T16:28:29.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x560822025ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x560822025ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44380/0 (socket says 172.21.15.31:44380) 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01ced9c700 1 -- 172.21.15.31:0/531652257 learned_addr learned my addr 172.21.15.31:0/531652257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01ce59b700 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822026d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.838+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822027740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ced9c700 1 -- 172.21.15.31:0/531652257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 msgr2=0x560822026d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ced9c700 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822026d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ced9c700 1 -- 172.21.15.31:0/531652257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 msgr2=0x560822027740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ced9c700 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822027740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ced9c700 1 -- 172.21.15.31:0/531652257 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560821da4de0 con 0x560822025be0 2023-07-29T16:28:29.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01bf7fe700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f01b40a0810 0x7f01b40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ced9c700 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x560822025ff0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011680 tx=0x7f01c0009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cd7fdad7d40d48b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01bf7fe700 1 -- 172.21.15.31:0/900919313 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f01c4014070 con 0x7f01c80737b0 2023-07-29T16:28:29.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01ce59b700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f01b40a0810 0x7f01b40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01bcff9700 1 -- 172.21.15.31:0/531652257 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c000e040 con 0x560822025be0 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01bcff9700 1 -- 172.21.15.31:0/531652257 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01c000f040 con 0x560822025be0 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.839+0000 7f01bcff9700 1 -- 172.21.15.31:0/531652257 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c00048c0 con 0x560822025be0 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 -- 172.21.15.31:0/531652257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 msgr2=0x560822025ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01ce59b700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f01b40a0810 0x7f01b40a2cc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f01c40a13e0 tx=0x7f01c40cf660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x560822025ff0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011680 tx=0x7f01c0009920 comp rx=0 tx=0).stop 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 -- 172.21.15.31:0/531652257 shutdown_connections 2023-07-29T16:28:29.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822027740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x560822025ff0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 --2- 172.21.15.31:0/531652257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822026d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 -- 172.21.15.31:0/531652257 >> 172.21.15.31:0/531652257 conn(0x56082201b3e0 msgr2=0x560822020990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 -- 172.21.15.31:0/531652257 shutdown_connections 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.840+0000 7f01e4147b80 1 -- 172.21.15.31:0/531652257 wait complete. 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.841+0000 7f01e4147b80 1 -- start start 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.841+0000 7f01e4147b80 1 Processor -- start 2023-07-29T16:28:29.845 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822025be0 0x56082202f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 0x56082202fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608220272f0 0x560822036d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560821f13510 con 0x560822025be0 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560821f131b0 con 0x560822026530 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560821dfa920 con 0x5608220272f0 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01e4147b80 10 client.0 _ll_register_callbacks cb 0x560821f1c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:29.846 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822025be0 0x56082202f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 0x56082202fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 0x56082202fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44386/0 (socket says 172.21.15.31:44386) 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 -- 172.21.15.31:0/2932163432 learned_addr learned my addr 172.21.15.31:0/2932163432 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01ced9c700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608220272f0 0x560822036d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 -- 172.21.15.31:0/2932163432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608220272f0 msgr2=0x560822036d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608220272f0 0x560822036d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 -- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822025be0 msgr2=0x56082202f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.847 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822025be0 0x56082202f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01cdd9a700 1 -- 172.21.15.31:0/2932163432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c0011330 con 0x560822026530 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.842+0000 7f01ced9c700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822025be0 0x56082202f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 0x56082202fd40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f01c8074ab0 tx=0x7f01b800bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9f274e4d1eacbb7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01b1ffb700 1 -- 172.21.15.31:0/2932163432 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b801d030 con 0x560822026530 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560821fa1d10 con 0x560822026530 2023-07-29T16:28:29.848 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01b1ffb700 1 -- 172.21.15.31:0/2932163432 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f01b8004500 con 0x560822026530 2023-07-29T16:28:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01b1ffb700 1 -- 172.21.15.31:0/2932163432 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b80046c0 con 0x560822026530 2023-07-29T16:28:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01e4147b80 10 client.29366 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:29.849 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.843+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560821d8dc60 con 0x560822026530 2023-07-29T16:28:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01b1ffb700 1 -- 172.21.15.31:0/2932163432 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01b8032020 con 0x560822026530 2023-07-29T16:28:29.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01b1ffb700 1 -- 172.21.15.31:0/2932163432 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f01b800f7e0 con 0x560822026530 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01e4147b80 10 client.29366 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01e4147b80 10 client.29366 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01e4147b80 20 client.29366 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01e4147b80 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082203b5d0 0x56082203d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01e4147b80 4 client.29366 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560821ff4d70 con 0x56082203b5d0 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.844+0000 7f01ced9c700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082203b5d0 0x56082203d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.851 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01ced9c700 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082203b5d0 0x56082203d9e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=976a542aadd0bc4a server_cookie=b67745f1474794aa in_seq=0 out_seq=0 2023-07-29T16:28:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01b1ffb700 10 client.29366 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:28:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01b1ffb700 1 -- 172.21.15.31:0/2932163432 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560821ff4d70 con 0x56082203b5d0 2023-07-29T16:28:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01b1ffb700 10 client.29366 handle_command_reply: tid=0 2023-07-29T16:28:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01e4147b80 1 client.29366 shutdown 2023-07-29T16:28:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 msgr2=0x56082202fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.852 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.845+0000 7f01e4147b80 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 0x56082202fd40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f01c8074ab0 tx=0x7f01b800bf60 comp rx=0 tx=0).stop 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 shutdown_connections 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082203b5d0 0x56082203d9e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5608220272f0 0x560822036d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822025be0 0x56082202f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01b1ffb700 0 client.29366 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 --2- 172.21.15.31:0/2932163432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822026530 0x56082202fd40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 >> 172.21.15.31:0/2932163432 conn(0x56082201b3e0 msgr2=0x560822020060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.853 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 shutdown_connections 2023-07-29T16:28:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 1 -- 172.21.15.31:0/2932163432 wait complete. 2023-07-29T16:28:29.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.846+0000 7f01e4147b80 20 client.29366 trim_cache size 0 max 16384 2023-07-29T16:28:29.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 Processor -- start 2023-07-29T16:28:29.860 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 -- start start 2023-07-29T16:28:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 0x560822042340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822026530 0x560822036d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608220272f0 0x5608220276c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560822030970 con 0x560822026530 2023-07-29T16:28:29.862 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082202ec20 con 0x5608220272f0 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.860+0000 7f01e4147b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608220186e0 con 0x560822025be0 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 0x560822042340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 0x560822042340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39600/0 (socket says 172.21.15.31:39600) 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 -- 172.21.15.31:0/3120714340 learned_addr learned my addr 172.21.15.31:0/3120714340 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822026530 0x560822036d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608220272f0 0x5608220276c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.863 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 -- 172.21.15.31:0/3120714340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608220272f0 msgr2=0x5608220276c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608220272f0 0x5608220276c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 -- 172.21.15.31:0/3120714340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822026530 msgr2=0x560822036d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822026530 0x560822036d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 -- 172.21.15.31:0/3120714340 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560821dcb250 con 0x560822025be0 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822026530 0x560822036d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01ced9c700 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 0x560822042340 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011620 tx=0x7f01c0004140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bc7f5565a588ca7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.864 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.861+0000 7f01bd7fa700 1 -- 172.21.15.31:0/3120714340 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c0026070 con 0x560822025be0 2023-07-29T16:28:29.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01bd7fa700 1 -- 172.21.15.31:0/3120714340 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01c0027040 con 0x560822025be0 2023-07-29T16:28:29.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01bd7fa700 1 -- 172.21.15.31:0/3120714340 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c000e040 con 0x560822025be0 2023-07-29T16:28:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 -- 172.21.15.31:0/3120714340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 msgr2=0x560822042340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 0x560822042340 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f01c0011620 tx=0x7f01c0004140 comp rx=0 tx=0).stop 2023-07-29T16:28:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 -- 172.21.15.31:0/3120714340 shutdown_connections 2023-07-29T16:28:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5608220272f0 0x5608220276c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822025be0 0x560822042340 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 --2- 172.21.15.31:0/3120714340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560822026530 0x560822036d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 -- 172.21.15.31:0/3120714340 >> 172.21.15.31:0/3120714340 conn(0x56082201b3e0 msgr2=0x56082202e510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 -- 172.21.15.31:0/3120714340 shutdown_connections 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.862+0000 7f01e4147b80 1 -- 172.21.15.31:0/3120714340 wait complete. 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 -- start start 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 Processor -- start 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x5608220368a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822033340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822033880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560822033f60 con 0x5608220272f0 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560822036f20 con 0x560822025be0 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.863+0000 7f01e4147b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608220370b0 con 0x560822026530 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01e4147b80 10 client.0 _ll_register_callbacks cb 0x560821f1c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822033340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822033340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39616/0 (socket says 172.21.15.31:39616) 2023-07-29T16:28:29.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 -- 172.21.15.31:0/1810650470 learned_addr learned my addr 172.21.15.31:0/1810650470 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01ced9c700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x5608220368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01ced9c700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822033880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 -- 172.21.15.31:0/1810650470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 msgr2=0x5608220368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x5608220368a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 -- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 msgr2=0x560822033880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822033880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 -- 172.21.15.31:0/1810650470 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560821da4de0 con 0x560822026530 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01ced9c700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822033880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01cdd9a700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822033340 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f01b800bd00 tx=0x7f01b8009950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d73474560cb51018 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.864+0000 7f01b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01b1ffb700 1 -- 172.21.15.31:0/1810650470 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b801d040 con 0x560822026530 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01c000f040 con 0x560822026530 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01b1ffb700 1 -- 172.21.15.31:0/1810650470 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f01b8033080 con 0x560822026530 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01b1ffb700 1 -- 172.21.15.31:0/1810650470 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b80233e0 con 0x560822026530 2023-07-29T16:28:29.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01e4147b80 10 client.29368 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560821da32b0 con 0x560822026530 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01b1ffb700 1 -- 172.21.15.31:0/1810650470 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01b8045020 con 0x560822026530 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.865+0000 7f01b1ffb700 1 -- 172.21.15.31:0/1810650470 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f01b8035040 con 0x560822026530 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01e4147b80 10 client.29368 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01e4147b80 10 client.29368 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01e4147b80 20 client.29368 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:29.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01e4147b80 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082208e4a0 0x560822090880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01e4147b80 4 client.29368 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560821f6fd40 con 0x56082208e4a0 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01ced9c700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082208e4a0 0x560822090880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01ced9c700 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082208e4a0 0x560822090880 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dbb48a09e4d676be server_cookie=e6c7c18f6dce20d7 in_seq=0 out_seq=0 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.866+0000 7f01b1ffb700 10 client.29368 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01b1ffb700 1 -- 172.21.15.31:0/1810650470 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560821f6fd40 con 0x56082208e4a0 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01b1ffb700 10 client.29368 handle_command_reply: tid=0 2023-07-29T16:28:29.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 client.29368 shutdown 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 msgr2=0x560822033340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822033340 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f01b800bd00 tx=0x7f01b8009950 comp rx=0 tx=0).stop 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 shutdown_connections 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x56082208e4a0 0x560822090880 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608220272f0 0x560822033880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01b1ffb700 0 client.29368 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560822025be0 0x5608220368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 --2- 172.21.15.31:0/1810650470 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560822026530 0x560822033340 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 >> 172.21.15.31:0/1810650470 conn(0x56082201b3e0 msgr2=0x56082202d670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 shutdown_connections 2023-07-29T16:28:29.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 1 -- 172.21.15.31:0/1810650470 wait complete. 2023-07-29T16:28:29.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.867+0000 7f01e4147b80 20 client.29368 trim_cache size 0 max 16384 2023-07-29T16:28:29.876 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:28:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f01b40a0810 msgr2=0x7f01b40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f01b40a0810 0x7f01b40a2cc0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f01c40a13e0 tx=0x7f01c40cf660 comp rx=0 tx=0).stop 2023-07-29T16:28:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 msgr2=0x7f01c81982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:29.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c81982d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f01c400e970 tx=0x7f01c400ec80 comp rx=0 tx=0).stop 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 shutdown_connections 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f01b40a0810 0x7f01b40a2cc0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01c81081f0 0x7f01c819f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01c80737b0 0x7f01c81982d0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 --2- 172.21.15.31:0/900919313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f01c8074d90 0x7f01c8198810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 >> 172.21.15.31:0/900919313 conn(0x7f01c80fb970 msgr2=0x7f01c80fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 shutdown_connections 2023-07-29T16:28:29.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:29.868+0000 7f01d07ff700 1 -- 172.21.15.31:0/900919313 wait complete. 2023-07-29T16:28:29.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:28:29.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1015435 8.93 MB/sec execute 494 sec latency 20.120 ms 2023-07-29T16:28:30.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 -- 172.21.15.31:0/1470224699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240fec10 msgr2=0x7f16240ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 --2- 172.21.15.31:0/1470224699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240fec10 0x7f16240ff020 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f1614009990 tx=0x7f1614009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 -- 172.21.15.31:0/1470224699 shutdown_connections 2023-07-29T16:28:30.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 --2- 172.21.15.31:0/1470224699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fff10 0x7f1624100380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 --2- 172.21.15.31:0/1470224699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240fec10 0x7f16240ff020 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 --2- 172.21.15.31:0/1470224699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240ff560 0x7f16240ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 -- 172.21.15.31:0/1470224699 >> 172.21.15.31:0/1470224699 conn(0x7f16240fa840 msgr2=0x7f16240fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:30.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 -- 172.21.15.31:0/1470224699 shutdown_connections 2023-07-29T16:28:30.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.107+0000 7f162a3ff700 1 -- 172.21.15.31:0/1470224699 wait complete. 2023-07-29T16:28:30.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 Processor -- start 2023-07-29T16:28:30.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 -- start start 2023-07-29T16:28:30.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 0x7f1624198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240ff560 0x7f1624198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240fff10 0x7f162419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162419fdd0 con 0x7f16240ff560 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162419ff10 con 0x7f16240fec10 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.109+0000 7f162a3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16241a00a0 con 0x7f16240fff10 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 0x7f1624198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 0x7f1624198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44424/0 (socket says 172.21.15.31:44424) 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 -- 172.21.15.31:0/2261108978 learned_addr learned my addr 172.21.15.31:0/2261108978 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:30.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f16237fe700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240ff560 0x7f1624198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f162899c700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240fff10 0x7f162419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 -- 172.21.15.31:0/2261108978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240fff10 msgr2=0x7f162419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240fff10 0x7f162419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 -- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240ff560 msgr2=0x7f1624198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240ff560 0x7f1624198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 -- 172.21.15.31:0/2261108978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f161c009c40 con 0x7f16240fec10 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.110+0000 7f1623fff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 0x7f1624198250 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1614005740 tx=0x7f1614005770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:30.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.111+0000 7f16217fa700 1 -- 172.21.15.31:0/2261108978 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161401d070 con 0x7f16240fec10 2023-07-29T16:28:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.111+0000 7f16217fa700 1 -- 172.21.15.31:0/2261108978 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1614005a00 con 0x7f16240fec10 2023-07-29T16:28:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.111+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16140095f0 con 0x7f16240fec10 2023-07-29T16:28:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.111+0000 7f16217fa700 1 -- 172.21.15.31:0/2261108978 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16140173f0 con 0x7f16240fec10 2023-07-29T16:28:30.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.111+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16241a0690 con 0x7f16240fec10 2023-07-29T16:28:30.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.116+0000 7f16217fa700 1 -- 172.21.15.31:0/2261108978 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f161400fcf0 con 0x7f16240fec10 2023-07-29T16:28:30.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 Processor -- start 2023-07-29T16:28:30.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 -- start start 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd590d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd591740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd4e7a10 con 0x5609cd590530 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd4e53d0 con 0x5609cd5912f0 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.117+0000 7f163dd75b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd4e0cc0 con 0x5609cd58fbe0 2023-07-29T16:28:30.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16217fa700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f160c0a0810 0x7f160c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd591740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f162899c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd591740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44426/0 (socket says 172.21.15.31:44426) 2023-07-29T16:28:30.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 -- 172.21.15.31:0/2941683080 learned_addr learned my addr 172.21.15.31:0/2941683080 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:30.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16217fa700 1 -- 172.21.15.31:0/2261108978 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f16140ccda0 con 0x7f16240fec10 2023-07-29T16:28:30.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 -- 172.21.15.31:0/2941683080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 msgr2=0x5609cd58fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 -- 172.21.15.31:0/2941683080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 msgr2=0x5609cd590d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f162899c700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f160c0a0810 0x7f160c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f1623fff700 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd590d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd590d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f16237fe700 1 -- 172.21.15.31:0/2941683080 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609cd30ede0 con 0x5609cd5912f0 2023-07-29T16:28:30.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.118+0000 7f1623fff700 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd590d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:28:30.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.119+0000 7f16237fe700 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd591740 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1618009d80 tx=0x7f1618011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64cf2ecaa8d6d5af server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:30.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.119+0000 7f162899c700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f160c0a0810 0x7f160c0a2cc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x5609cd58f980 tx=0x7f161c004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:30.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.119+0000 7f160a7fc700 1 -- 172.21.15.31:0/2941683080 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618004230 con 0x5609cd5912f0 2023-07-29T16:28:30.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.119+0000 7f160a7fc700 1 -- 172.21.15.31:0/2941683080 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1618004390 con 0x5609cd5912f0 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.119+0000 7f160a7fc700 1 -- 172.21.15.31:0/2941683080 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618019880 con 0x5609cd5912f0 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 -- 172.21.15.31:0/2941683080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 msgr2=0x5609cd591740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd591740 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1618009d80 tx=0x7f1618011720 comp rx=0 tx=0).stop 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 -- 172.21.15.31:0/2941683080 shutdown_connections 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd591740 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 --2- 172.21.15.31:0/2941683080 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd590d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 -- 172.21.15.31:0/2941683080 >> 172.21.15.31:0/2941683080 conn(0x5609cd5853e0 msgr2=0x5609cd58a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 -- 172.21.15.31:0/2941683080 shutdown_connections 2023-07-29T16:28:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 -- 172.21.15.31:0/2941683080 wait complete. 2023-07-29T16:28:30.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.120+0000 7f163dd75b80 1 -- start start 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 Processor -- start 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 0x5609cd599830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd590530 0x5609cd599d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd47d510 con 0x5609cd58fbe0 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd47d1b0 con 0x5609cd590530 2023-07-29T16:28:30.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd364920 con 0x5609cd5912f0 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f163dd75b80 10 client.0 _ll_register_callbacks cb 0x5609cd486620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.121+0000 7f1623fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 0x5609cd599830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 0x5609cd599830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35190/0 (socket says 172.21.15.31:35190) 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 -- 172.21.15.31:0/4086713960 learned_addr learned my addr 172.21.15.31:0/4086713960 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f16237fe700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd590530 0x5609cd599d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f162899c700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.128 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 -- 172.21.15.31:0/4086713960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 msgr2=0x5609cd5a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 -- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd590530 msgr2=0x5609cd599d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd590530 0x5609cd599d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 -- 172.21.15.31:0/4086713960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16180099e0 con 0x5609cd58fbe0 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f16237fe700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd590530 0x5609cd599d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f162899c700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f1623fff700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 0x5609cd599830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f16140cabb0 tx=0x7f161409aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7a0155b5f934c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:30.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.122+0000 7f16037fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f16037fe700 1 -- 172.21.15.31:0/4086713960 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161409e370 con 0x5609cd58fbe0 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609cd50bd10 con 0x5609cd58fbe0 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f16037fe700 1 -- 172.21.15.31:0/4086713960 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f161409a530 con 0x5609cd58fbe0 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f16037fe700 1 -- 172.21.15.31:0/4086713960 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1614099e90 con 0x5609cd58fbe0 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 10 client.19269 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5609cd2f7c60 con 0x5609cd58fbe0 2023-07-29T16:28:30.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f16037fe700 1 -- 172.21.15.31:0/4086713960 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5609cd50bd10 con 0x5609cd58fbe0 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f16037fe700 1 -- 172.21.15.31:0/4086713960 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f1614096320 con 0x5609cd58fbe0 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 10 client.19269 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 10 client.19269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 20 client.19269 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5a5030 0x5609cd5a7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 4 client.19269 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:30.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.123+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609cd55ed70 con 0x5609cd5a5030 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.124+0000 7f16237fe700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5a5030 0x5609cd5a7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.124+0000 7f16237fe700 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5a5030 0x5609cd5a7410 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca97da6344a929ea server_cookie=db7fe1dc638e90e1 in_seq=0 out_seq=0 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.124+0000 7f16037fe700 10 client.19269 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.125+0000 7f16037fe700 1 -- 172.21.15.31:0/4086713960 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5609cd55ed70 con 0x5609cd5a5030 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.125+0000 7f16037fe700 10 client.19269 handle_command_reply: tid=0 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.125+0000 7f163dd75b80 1 client.19269 shutdown 2023-07-29T16:28:30.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.125+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 msgr2=0x5609cd599830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.125+0000 7f163dd75b80 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 0x5609cd599830 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f16140cabb0 tx=0x7f161409aa70 comp rx=0 tx=0).stop 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.125+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 shutdown_connections 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5a5030 0x5609cd5a7410 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd58fbe0 0x5609cd599830 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 --2- 172.21.15.31:0/4086713960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd590530 0x5609cd599d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f16037fe700 0 client.19269 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:30.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 >> 172.21.15.31:0/4086713960 conn(0x5609cd5853e0 msgr2=0x5609cd58a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 shutdown_connections 2023-07-29T16:28:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 1 -- 172.21.15.31:0/4086713960 wait complete. 2023-07-29T16:28:30.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.126+0000 7f163dd75b80 20 client.19269 trim_cache size 0 max 16384 2023-07-29T16:28:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.140+0000 7f163dd75b80 1 Processor -- start 2023-07-29T16:28:30.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.140+0000 7f163dd75b80 1 -- start start 2023-07-29T16:28:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd5a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd58a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f163dd75b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd598ce0 con 0x5609cd590530 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f163dd75b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd5993a0 con 0x5609cd5912f0 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f163dd75b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd587490 con 0x5609cd58fbe0 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f1623fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f16237fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd5a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f1623fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39668/0 (socket says 172.21.15.31:39668) 2023-07-29T16:28:30.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f1623fff700 1 -- 172.21.15.31:0/3074933046 learned_addr learned my addr 172.21.15.31:0/3074933046 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f162899c700 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd58a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f16237fe700 1 -- 172.21.15.31:0/3074933046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 msgr2=0x5609cd58b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f16237fe700 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f16237fe700 1 -- 172.21.15.31:0/3074933046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 msgr2=0x5609cd58a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.143 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.141+0000 7f16237fe700 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd58a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.142+0000 7f16237fe700 1 -- 172.21.15.31:0/3074933046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609cd335250 con 0x5609cd590530 2023-07-29T16:28:30.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.142+0000 7f16237fe700 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd5a0d70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1618009770 tx=0x7f16180058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b426c4bfeba53 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:30.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.142+0000 7f160affd700 1 -- 172.21.15.31:0/3074933046 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1618029070 con 0x5609cd590530 2023-07-29T16:28:30.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.142+0000 7f160affd700 1 -- 172.21.15.31:0/3074933046 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1618004500 con 0x5609cd590530 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.142+0000 7f160affd700 1 -- 172.21.15.31:0/3074933046 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16180052a0 con 0x5609cd590530 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 -- 172.21.15.31:0/3074933046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 msgr2=0x5609cd5a0d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd5a0d70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1618009770 tx=0x7f16180058e0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 -- 172.21.15.31:0/3074933046 shutdown_connections 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd5912f0 0x5609cd58a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd58fbe0 0x5609cd58b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3074933046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd5a0d70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 -- 172.21.15.31:0/3074933046 >> 172.21.15.31:0/3074933046 conn(0x5609cd5853e0 msgr2=0x5609cd5984e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 -- 172.21.15.31:0/3074933046 shutdown_connections 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.143+0000 7f163dd75b80 1 -- 172.21.15.31:0/3074933046 wait complete. 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 -- start start 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 Processor -- start 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 0x5609cd59ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd59d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd5a08f0 con 0x5609cd590530 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd5a0a60 con 0x5609cd58fbe0 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609cd5a0c20 con 0x5609cd5912f0 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.144+0000 7f163dd75b80 10 client.0 _ll_register_callbacks cb 0x5609cd486620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 0x5609cd59ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f16237fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd59d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f162899c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 0x5609cd59ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44454/0 (socket says 172.21.15.31:44454) 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 -- 172.21.15.31:0/3039625672 learned_addr learned my addr 172.21.15.31:0/3039625672 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 -- 172.21.15.31:0/3039625672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 msgr2=0x5609cd5a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 -- 172.21.15.31:0/3039625672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 msgr2=0x5609cd59d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd59d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 -- 172.21.15.31:0/3039625672 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609cd30ede0 con 0x5609cd58fbe0 2023-07-29T16:28:30.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f16237fe700 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd59d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f1623fff700 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 0x5609cd59ce50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f161400fca0 tx=0x7f16140057e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968e29d6c4838320 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.145+0000 7f16037fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:30.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f16037fe700 1 -- 172.21.15.31:0/3039625672 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614021780 con 0x5609cd58fbe0 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f16037fe700 1 -- 172.21.15.31:0/3039625672 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f16140c8070 con 0x5609cd58fbe0 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1618004390 con 0x5609cd58fbe0 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f16037fe700 1 -- 172.21.15.31:0/3039625672 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f161409e380 con 0x5609cd58fbe0 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 10 client.29390 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5609cd30d2b0 con 0x5609cd58fbe0 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f16037fe700 1 -- 172.21.15.31:0/3039625672 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1618004390 con 0x5609cd58fbe0 2023-07-29T16:28:30.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f16037fe700 1 -- 172.21.15.31:0/3039625672 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f16140a00b0 con 0x5609cd58fbe0 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 10 client.29390 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 10 client.29390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 20 client.29390 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5f79a0 0x5609cd5f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 4 client.29390 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.146+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5609cd4d9d40 con 0x5609cd5f79a0 2023-07-29T16:28:30.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.147+0000 7f16237fe700 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5f79a0 0x5609cd5f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.147+0000 7f16037fe700 10 client.29390 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.147+0000 7f16237fe700 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5f79a0 0x5609cd5f9db0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df36d3b8943453ee server_cookie=a9375cc7be7da2ee in_seq=0 out_seq=0 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f16037fe700 1 -- 172.21.15.31:0/3039625672 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5609cd4d9d40 con 0x5609cd5f79a0 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f16037fe700 10 client.29390 handle_command_reply: tid=0 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 client.29390 shutdown 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 msgr2=0x5609cd59ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 0x5609cd59ce50 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f161400fca0 tx=0x7f16140057e0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 shutdown_connections 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609cd5f79a0 0x5609cd5f9db0 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609cd5912f0 0x5609cd5a03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5609cd58fbe0 0x5609cd59ce50 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 --2- 172.21.15.31:0/3039625672 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609cd590530 0x5609cd59d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 >> 172.21.15.31:0/3039625672 conn(0x5609cd5853e0 msgr2=0x5609cd5984e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f16037fe700 0 client.29390 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:30.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 shutdown_connections 2023-07-29T16:28:30.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 1 -- 172.21.15.31:0/3039625672 wait complete. 2023-07-29T16:28:30.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.148+0000 7f163dd75b80 20 client.29390 trim_cache size 0 max 16384 2023-07-29T16:28:30.156 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:28:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f160c0a0810 msgr2=0x7f160c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f160c0a0810 0x7f160c0a2cc0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x5609cd58f980 tx=0x7f161c004ad0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 msgr2=0x7f1624198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 0x7f1624198250 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1614005740 tx=0x7f1614005770 comp rx=0 tx=0).stop 2023-07-29T16:28:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 shutdown_connections 2023-07-29T16:28:30.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f160c0a0810 0x7f160c0a2cc0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16240fff10 0x7f162419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16240fec10 0x7f1624198250 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 --2- 172.21.15.31:0/2261108978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16240ff560 0x7f1624198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 >> 172.21.15.31:0/2261108978 conn(0x7f16240fa840 msgr2=0x7f16240fc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 shutdown_connections 2023-07-29T16:28:30.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:30.149+0000 7f162a3ff700 1 -- 172.21.15.31:0/2261108978 wait complete. 2023-07-29T16:28:30.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:30 smithi031 ceph-mon[95504]: pgmap v762: 97 pgs: 97 active+clean; 188 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.7 MiB/s wr, 296 op/s 2023-07-29T16:28:30.776 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:30 smithi039 ceph-mon[128858]: pgmap v762: 97 pgs: 97 active+clean; 188 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.7 MiB/s wr, 296 op/s 2023-07-29T16:28:30.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:30 smithi156 ceph-mon[129245]: pgmap v762: 97 pgs: 97 active+clean; 188 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.7 MiB/s wr, 296 op/s 2023-07-29T16:28:30.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1016535 8.92 MB/sec execute 495 sec latency 35.997 ms 2023-07-29T16:28:31.168 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:28:31.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:28:31.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 -- 172.21.15.31:0/3279250617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 msgr2=0x7f51cc0731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 --2- 172.21.15.31:0/3279250617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc0731e0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f51c80098f0 tx=0x7f51c8009c00 comp rx=0 tx=0).stop 2023-07-29T16:28:31.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 -- 172.21.15.31:0/3279250617 shutdown_connections 2023-07-29T16:28:31.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 --2- 172.21.15.31:0/3279250617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51cc1081f0 0x7f51cc10a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 --2- 172.21.15.31:0/3279250617 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc0731e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 --2- 172.21.15.31:0/3279250617 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51cc0737b0 0x7f51cc073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 -- 172.21.15.31:0/3279250617 >> 172.21.15.31:0/3279250617 conn(0x7f51cc0fb970 msgr2=0x7f51cc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 -- 172.21.15.31:0/3279250617 shutdown_connections 2023-07-29T16:28:31.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.410+0000 7f51d41ff700 1 -- 172.21.15.31:0/3279250617 wait complete. 2023-07-29T16:28:31.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.411+0000 7f51d41ff700 1 Processor -- start 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 -- start start 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51cc0737b0 0x7f51cc198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51cc1081f0 0x7f51cc19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19ff10 con 0x7f51cc0737b0 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc1a0050 con 0x7f51cc1081f0 2023-07-29T16:28:31.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d41ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc1a01e0 con 0x7f51cc074d90 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d179a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51cc1081f0 0x7f51cc19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51cc0737b0 0x7f51cc198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d179a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc1988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39694/0 (socket says 172.21.15.31:39694) 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d179a700 1 -- 172.21.15.31:0/2731321615 learned_addr learned my addr 172.21.15.31:0/2731321615 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d179a700 1 -- 172.21.15.31:0/2731321615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51cc1081f0 msgr2=0x7f51cc19f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.412+0000 7f51d179a700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51cc1081f0 0x7f51cc19f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51d179a700 1 -- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51cc0737b0 msgr2=0x7f51cc198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51d179a700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51cc0737b0 0x7f51cc198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51d179a700 1 -- 172.21.15.31:0/2731321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c8009520 con 0x7f51cc074d90 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51d179a700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc1988d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00e970 tx=0x7f51bc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51c2ffd700 1 -- 172.21.15.31:0/2731321615 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc009790 con 0x7f51cc074d90 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51c2ffd700 1 -- 172.21.15.31:0/2731321615 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51bc00c7e0 con 0x7f51cc074d90 2023-07-29T16:28:31.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc1a04c0 con 0x7f51cc074d90 2023-07-29T16:28:31.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51c2ffd700 1 -- 172.21.15.31:0/2731321615 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc003c70 con 0x7f51cc074d90 2023-07-29T16:28:31.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.413+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc1a0a20 con 0x7f51cc074d90 2023-07-29T16:28:31.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.417+0000 7f51c2ffd700 1 -- 172.21.15.31:0/2731321615 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f51bc00cc50 con 0x7f51cc074d90 2023-07-29T16:28:31.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.418+0000 7f51e7af7b80 1 Processor -- start 2023-07-29T16:28:31.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.418+0000 7f51e7af7b80 1 -- start start 2023-07-29T16:28:31.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b2c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 0x563ae7b2cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.419 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b2d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7a83a90 con 0x563ae7b2d370 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7a81450 con 0x563ae7b2c5b0 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7a7cd40 con 0x563ae7b2bc60 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b2c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 0x563ae7b2cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51c2ffd700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51b80a0810 0x7f51b80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b2c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39706/0 (socket says 172.21.15.31:39706) 2023-07-29T16:28:31.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d279c700 1 -- 172.21.15.31:0/3503252336 learned_addr learned my addr 172.21.15.31:0/3503252336 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 0x563ae7b2cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44474/0 (socket says 172.21.15.31:44474) 2023-07-29T16:28:31.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d179a700 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b2d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51c2ffd700 1 -- 172.21.15.31:0/2731321615 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f51bc014070 con 0x7f51cc074d90 2023-07-29T16:28:31.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 -- 172.21.15.31:0/3503252336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 msgr2=0x563ae7b2c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b2c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 -- 172.21.15.31:0/3503252336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 msgr2=0x563ae7b2d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b2d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.419+0000 7f51d1f9b700 1 -- 172.21.15.31:0/3503252336 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ae78aae60 con 0x563ae7b2c5b0 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.420+0000 7f51d279c700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51b80a0810 0x7f51b80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.420+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 0x563ae7b2cde0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f51c8000c00 tx=0x7f51c8005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbe7721fe050bee5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.420+0000 7f51b7fff700 1 -- 172.21.15.31:0/3503252336 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800f720 con 0x563ae7b2c5b0 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.420+0000 7f51b7fff700 1 -- 172.21.15.31:0/3503252336 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c801d040 con 0x563ae7b2c5b0 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.420+0000 7f51b7fff700 1 -- 172.21.15.31:0/3503252336 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c80055d0 con 0x563ae7b2c5b0 2023-07-29T16:28:31.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.420+0000 7f51d279c700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51b80a0810 0x7f51b80a2cc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x563ae7b2ba00 tx=0x7f51c4004ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 -- 172.21.15.31:0/3503252336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 msgr2=0x563ae7b2cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 0x563ae7b2cde0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f51c8000c00 tx=0x7f51c8005dc0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 -- 172.21.15.31:0/3503252336 shutdown_connections 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b2d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b2c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/3503252336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2c5b0 0x563ae7b2cde0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 -- 172.21.15.31:0/3503252336 >> 172.21.15.31:0/3503252336 conn(0x563ae7b21460 msgr2=0x563ae7b26a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.421+0000 7f51e7af7b80 1 -- 172.21.15.31:0/3503252336 shutdown_connections 2023-07-29T16:28:31.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.422+0000 7f51e7af7b80 1 -- 172.21.15.31:0/3503252336 wait complete. 2023-07-29T16:28:31.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.422+0000 7f51e7af7b80 1 -- start start 2023-07-29T16:28:31.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.422+0000 7f51e7af7b80 1 Processor -- start 2023-07-29T16:28:31.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b35830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b35d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b3cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7a19590 con 0x563ae7b2d370 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7a19230 con 0x563ae7b2bc60 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae79009a0 con 0x563ae7b2c5b0 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51e7af7b80 10 client.0 _ll_register_callbacks cb 0x563ae7a226a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b35830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51d179a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b35d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b35830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44490/0 (socket says 172.21.15.31:44490) 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51d1f9b700 1 -- 172.21.15.31:0/110298008 learned_addr learned my addr 172.21.15.31:0/110298008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51d179a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b35d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39714/0 (socket says 172.21.15.31:39714) 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.423+0000 7f51d279c700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b3cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d1f9b700 1 -- 172.21.15.31:0/110298008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 msgr2=0x563ae7b35d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b35d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d1f9b700 1 -- 172.21.15.31:0/110298008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 msgr2=0x563ae7b3cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b3cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d1f9b700 1 -- 172.21.15.31:0/110298008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc10b290 con 0x563ae7b2bc60 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d179a700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b35d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b35830 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009810 tx=0x7f51c8005580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=462bea62c1a3768c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.424+0000 7f51b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51b57fa700 1 -- 172.21.15.31:0/110298008 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800b970 con 0x563ae7b2bc60 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51b57fa700 1 -- 172.21.15.31:0/110298008 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f51c800bad0 con 0x563ae7b2bc60 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ae7aa7d90 con 0x563ae7b2bc60 2023-07-29T16:28:31.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51b57fa700 1 -- 172.21.15.31:0/110298008 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51c80056f0 con 0x563ae7b2bc60 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51e7af7b80 10 client.29402 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563ae7893ce0 con 0x563ae7b2bc60 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51b57fa700 1 -- 172.21.15.31:0/110298008 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563ae7aa7d90 con 0x563ae7b2bc60 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.425+0000 7f51b57fa700 1 -- 172.21.15.31:0/110298008 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f51c8015020 con 0x563ae7b2bc60 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51e7af7b80 10 client.29402 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51e7af7b80 10 client.29402 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:31.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51e7af7b80 20 client.29402 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b41050 0x563ae7b43460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51e7af7b80 4 client.29402 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ae7afadf0 con 0x563ae7b41050 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51d179a700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b41050 0x563ae7b43460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51d179a700 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b41050 0x563ae7b43460 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69f34a4b1f598f9a server_cookie=ac8d3ae925fb0089 in_seq=0 out_seq=0 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.426+0000 7f51b57fa700 10 client.29402 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51b57fa700 1 -- 172.21.15.31:0/110298008 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563ae7afadf0 con 0x563ae7b41050 2023-07-29T16:28:31.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51b57fa700 10 client.29402 handle_command_reply: tid=0 2023-07-29T16:28:31.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 client.29402 shutdown 2023-07-29T16:28:31.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 msgr2=0x563ae7b35830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b35830 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009810 tx=0x7f51c8005580 comp rx=0 tx=0).stop 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 shutdown_connections 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b41050 0x563ae7b43460 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b3cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b35830 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/110298008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b35d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 >> 172.21.15.31:0/110298008 conn(0x563ae7b21460 msgr2=0x563ae7b260e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.427+0000 7f51b57fa700 0 client.29402 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.428+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 shutdown_connections 2023-07-29T16:28:31.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.428+0000 7f51e7af7b80 1 -- 172.21.15.31:0/110298008 wait complete. 2023-07-29T16:28:31.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.428+0000 7f51e7af7b80 20 client.29402 trim_cache size 0 max 16384 2023-07-29T16:28:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 Processor -- start 2023-07-29T16:28:31.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 -- start start 2023-07-29T16:28:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b3ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 0x563ae7b355e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2d370 0x563ae7b35b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7b36a20 con 0x563ae7b2c5b0 2023-07-29T16:28:31.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7b34ce0 con 0x563ae7b2d370 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.448+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7b23510 con 0x563ae7b2bc60 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d179a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 0x563ae7b355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b3ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d179a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 0x563ae7b355e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35260/0 (socket says 172.21.15.31:35260) 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d179a700 1 -- 172.21.15.31:0/2398105805 learned_addr learned my addr 172.21.15.31:0/2398105805 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d279c700 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2d370 0x563ae7b35b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.450 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 -- 172.21.15.31:0/2398105805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2d370 msgr2=0x563ae7b35b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2d370 0x563ae7b35b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 -- 172.21.15.31:0/2398105805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 msgr2=0x563ae7b355e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 0x563ae7b355e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 -- 172.21.15.31:0/2398105805 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ae78d12d0 con 0x563ae7b2bc60 2023-07-29T16:28:31.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d179a700 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 0x563ae7b355e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.449+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b3ad60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f51c800a010 tx=0x7f51c8005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=959a9be15c7962a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51c0ff9700 1 -- 172.21.15.31:0/2398105805 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c802aa60 con 0x563ae7b2bc60 2023-07-29T16:28:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51c0ff9700 1 -- 172.21.15.31:0/2398105805 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c802a420 con 0x563ae7b2bc60 2023-07-29T16:28:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51c0ff9700 1 -- 172.21.15.31:0/2398105805 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800bab0 con 0x563ae7b2bc60 2023-07-29T16:28:31.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 -- 172.21.15.31:0/2398105805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 msgr2=0x563ae7b3ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b3ad60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f51c800a010 tx=0x7f51c8005dc0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 -- 172.21.15.31:0/2398105805 shutdown_connections 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2d370 0x563ae7b35b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2bc60 0x563ae7b3ad60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/2398105805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2c5b0 0x563ae7b355e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.450+0000 7f51e7af7b80 1 -- 172.21.15.31:0/2398105805 >> 172.21.15.31:0/2398105805 conn(0x563ae7b21460 msgr2=0x563ae7b30310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.451+0000 7f51e7af7b80 1 -- 172.21.15.31:0/2398105805 shutdown_connections 2023-07-29T16:28:31.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.451+0000 7f51e7af7b80 1 -- 172.21.15.31:0/2398105805 wait complete. 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.451+0000 7f51e7af7b80 1 -- start start 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.451+0000 7f51e7af7b80 1 Processor -- start 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b8b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b8ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b92a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7b92fa0 con 0x563ae7b2d370 2023-07-29T16:28:31.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7b930e0 con 0x563ae7b2bc60 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ae7b93270 con 0x563ae7b2c5b0 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51e7af7b80 10 client.0 _ll_register_callbacks cb 0x563ae7a226a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b8b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51d1f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b8b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44512/0 (socket says 172.21.15.31:44512) 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51d1f9b700 1 -- 172.21.15.31:0/4007913188 learned_addr learned my addr 172.21.15.31:0/4007913188 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51d179a700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b8ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.452+0000 7f51d279c700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b92a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51d1f9b700 1 -- 172.21.15.31:0/4007913188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 msgr2=0x563ae7b8ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b8ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51d1f9b700 1 -- 172.21.15.31:0/4007913188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 msgr2=0x563ae7b92a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b92a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51d1f9b700 1 -- 172.21.15.31:0/4007913188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc1a0a20 con 0x563ae7b2bc60 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51d1f9b700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b8b510 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f51c8005a80 tx=0x7f51c800f8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caaf96acf0df7567 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51b57fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.453+0000 7f51b57fa700 1 -- 172.21.15.31:0/4007913188 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800ba60 con 0x563ae7b2bc60 2023-07-29T16:28:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51c801d040 con 0x563ae7b2bc60 2023-07-29T16:28:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51b57fa700 1 -- 172.21.15.31:0/4007913188 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f51c8000e40 con 0x563ae7b2bc60 2023-07-29T16:28:31.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51b57fa700 1 -- 172.21.15.31:0/4007913188 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51c800bbc0 con 0x563ae7b2bc60 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 10 client.29408 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563ae7893ce0 con 0x563ae7b2bc60 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51b57fa700 1 -- 172.21.15.31:0/4007913188 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c801d040 con 0x563ae7b2bc60 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51b57fa700 1 -- 172.21.15.31:0/4007913188 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f51c8009920 con 0x563ae7b2bc60 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 10 client.29408 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 10 client.29408 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:31.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 20 client.29408 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b947c0 0x563ae7b96bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 4 client.29408 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.454+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563ae7a75dc0 con 0x563ae7b947c0 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.455+0000 7f51d179a700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b947c0 0x563ae7b96bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.455+0000 7f51d179a700 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b947c0 0x563ae7b96bd0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd5cf88dd0145912 server_cookie=691ab6010cdcd358 in_seq=0 out_seq=0 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.455+0000 7f51b57fa700 10 client.29408 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.456+0000 7f51b57fa700 1 -- 172.21.15.31:0/4007913188 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563ae7a75dc0 con 0x563ae7b947c0 2023-07-29T16:28:31.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.456+0000 7f51b57fa700 10 client.29408 handle_command_reply: tid=0 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.456+0000 7f51e7af7b80 1 client.29408 shutdown 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 msgr2=0x563ae7b8b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b8b510 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f51c8005a80 tx=0x7f51c800f8a0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 shutdown_connections 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563ae7b947c0 0x563ae7b96bd0 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ae7b2d370 0x563ae7b92a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ae7b2bc60 0x563ae7b8b510 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 --2- 172.21.15.31:0/4007913188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ae7b2c5b0 0x563ae7b8ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 >> 172.21.15.31:0/4007913188 conn(0x563ae7b21460 msgr2=0x563ae7b355e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51b57fa700 0 client.29408 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 shutdown_connections 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 1 -- 172.21.15.31:0/4007913188 wait complete. 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.457+0000 7f51e7af7b80 20 client.29408 trim_cache size 0 max 16384 2023-07-29T16:28:31.463 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:28:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:28:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "21e7b9cb-68b1-4061-b9f8-c8ff27f2c049", 2023-07-29T16:28:31.464 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:28:31.464 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:28:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51b80a0810 msgr2=0x7f51b80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51b80a0810 0x7f51b80a2cc0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x563ae7b2ba00 tx=0x7f51c4004ad0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 msgr2=0x7f51cc1988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc1988d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00e970 tx=0x7f51bc00ec80 comp rx=0 tx=0).stop 2023-07-29T16:28:31.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 shutdown_connections 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f51b80a0810 0x7f51b80a2cc0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51cc1081f0 0x7f51cc19f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51cc0737b0 0x7f51cc198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 --2- 172.21.15.31:0/2731321615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51cc074d90 0x7f51cc1988d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 >> 172.21.15.31:0/2731321615 conn(0x7f51cc0fb970 msgr2=0x7f51cc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 shutdown_connections 2023-07-29T16:28:31.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.458+0000 7f51d41ff700 1 -- 172.21.15.31:0/2731321615 wait complete. 2023-07-29T16:28:31.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:28:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.738+0000 7efd57dff700 1 -- 172.21.15.31:0/910557057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50103700 msgr2=0x7efd50105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.738+0000 7efd57dff700 1 --2- 172.21.15.31:0/910557057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50103700 0x7efd50105ae0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7efd40009990 tx=0x7efd40009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 -- 172.21.15.31:0/910557057 shutdown_connections 2023-07-29T16:28:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 --2- 172.21.15.31:0/910557057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50106020 0x7efd50108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 --2- 172.21.15.31:0/910557057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd50100de0 0x7efd501031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 --2- 172.21.15.31:0/910557057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50103700 0x7efd50105ae0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 -- 172.21.15.31:0/910557057 >> 172.21.15.31:0/910557057 conn(0x7efd500fa860 msgr2=0x7efd500fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 -- 172.21.15.31:0/910557057 shutdown_connections 2023-07-29T16:28:31.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.739+0000 7efd57dff700 1 -- 172.21.15.31:0/910557057 wait complete. 2023-07-29T16:28:31.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.740+0000 7efd57dff700 1 Processor -- start 2023-07-29T16:28:31.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.740+0000 7efd57dff700 1 -- start start 2023-07-29T16:28:31.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd57dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 0x7efd50100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd57dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd50103700 0x7efd500fed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd57dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50106020 0x7efd500ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd57dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500ffa70 con 0x7efd50100de0 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd57dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd500ffbe0 con 0x7efd50106020 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd57dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd501a0100 con 0x7efd50103700 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 0x7efd50100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 0x7efd50100750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35280/0 (socket says 172.21.15.31:35280) 2023-07-29T16:28:31.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd55b9b700 1 -- 172.21.15.31:0/1928041535 learned_addr learned my addr 172.21.15.31:0/1928041535 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd5539a700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd50103700 0x7efd500fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.741+0000 7efd5639c700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50106020 0x7efd500ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd5639c700 1 -- 172.21.15.31:0/1928041535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd50103700 msgr2=0x7efd500fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd5639c700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd50103700 0x7efd500fed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd5639c700 1 -- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 msgr2=0x7efd50100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd5639c700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 0x7efd50100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd5639c700 1 -- 172.21.15.31:0/1928041535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd400095f0 con 0x7efd50106020 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd55b9b700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 0x7efd50100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd5639c700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50106020 0x7efd500ff2d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7efd4800efc0 tx=0x7efd4800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd46ffd700 1 -- 172.21.15.31:0/1928041535 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd48018070 con 0x7efd50106020 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd46ffd700 1 -- 172.21.15.31:0/1928041535 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd4800f040 con 0x7efd50106020 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd501a0380 con 0x7efd50106020 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.742+0000 7efd46ffd700 1 -- 172.21.15.31:0/1928041535 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4800cd40 con 0x7efd50106020 2023-07-29T16:28:31.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.743+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd501a08e0 con 0x7efd50106020 2023-07-29T16:28:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.747+0000 7efd46ffd700 1 -- 172.21.15.31:0/1928041535 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7efd48003a00 con 0x7efd50106020 2023-07-29T16:28:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.747+0000 7efd6b742b80 1 Processor -- start 2023-07-29T16:28:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 -- start start 2023-07-29T16:28:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a489bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a489cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a489d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a47f3a10 con 0x5637a489c530 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a47f13d0 con 0x5637a489bbe0 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd6b742b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a47eccc0 con 0x5637a489d2f0 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd5539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a489bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd5539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a489bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44542/0 (socket says 172.21.15.31:44542) 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd5539a700 1 -- 172.21.15.31:0/4051472743 learned_addr learned my addr 172.21.15.31:0/4051472743 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd5539a700 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a489d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.748+0000 7efd55b9b700 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a489cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd5539a700 1 -- 172.21.15.31:0/4051472743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 msgr2=0x5637a489d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd5539a700 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a489d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd5539a700 1 -- 172.21.15.31:0/4051472743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 msgr2=0x5637a489cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd5539a700 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a489cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd5539a700 1 -- 172.21.15.31:0/4051472743 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637a461ade0 con 0x5637a489bbe0 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd46ffd700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd3c0a0810 0x7efd3c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd5539a700 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a489bff0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efd40005cd0 tx=0x7efd4001a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aee7bb7ba3ef24cc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd46ffd700 1 -- 172.21.15.31:0/1928041535 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efd480cb570 con 0x7efd50106020 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd55b9b700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd3c0a0810 0x7efd3c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd3bfff700 1 -- 172.21.15.31:0/4051472743 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4000bc20 con 0x5637a489bbe0 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd3bfff700 1 -- 172.21.15.31:0/4051472743 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd4000bd80 con 0x5637a489bbe0 2023-07-29T16:28:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.749+0000 7efd3bfff700 1 -- 172.21.15.31:0/4051472743 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd40017650 con 0x5637a489bbe0 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd55b9b700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd3c0a0810 0x7efd3c0a2cc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x5637a489c940 tx=0x7efd4c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 -- 172.21.15.31:0/4051472743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 msgr2=0x5637a489bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a489bff0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efd40005cd0 tx=0x7efd4001a040 comp rx=0 tx=0).stop 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 -- 172.21.15.31:0/4051472743 shutdown_connections 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a489d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a489bff0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 --2- 172.21.15.31:0/4051472743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a489cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.750+0000 7efd6b742b80 1 -- 172.21.15.31:0/4051472743 >> 172.21.15.31:0/4051472743 conn(0x5637a48913e0 msgr2=0x5637a4896990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.751+0000 7efd6b742b80 1 -- 172.21.15.31:0/4051472743 shutdown_connections 2023-07-29T16:28:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.751+0000 7efd6b742b80 1 -- 172.21.15.31:0/4051472743 wait complete. 2023-07-29T16:28:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.751+0000 7efd6b742b80 1 -- start start 2023-07-29T16:28:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.751+0000 7efd6b742b80 1 Processor -- start 2023-07-29T16:28:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48a5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a48a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a48acec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a4789510 con 0x5637a489c530 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a47891b0 con 0x5637a489bbe0 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a4670920 con 0x5637a489d2f0 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd6b742b80 10 client.0 _ll_register_callbacks cb 0x5637a4792620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48a5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd5539a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a48a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48a5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44552/0 (socket says 172.21.15.31:44552) 2023-07-29T16:28:31.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd55b9b700 1 -- 172.21.15.31:0/2302304519 learned_addr learned my addr 172.21.15.31:0/2302304519 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.752+0000 7efd5639c700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a48acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5639c700 1 -- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 msgr2=0x5637a48a5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5639c700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48a5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5639c700 1 -- 172.21.15.31:0/2302304519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 msgr2=0x5637a48a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5639c700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a48a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5639c700 1 -- 172.21.15.31:0/2302304519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd40020040 con 0x5637a489d2f0 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5539a700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a48a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:31.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd5639c700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a48acec0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efd480c9380 tx=0x7efd48098bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2873dd5e7bc6059c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd397fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd397fa700 1 -- 172.21.15.31:0/2302304519 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd480983f0 con 0x5637a489d2f0 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd397fa700 1 -- 172.21.15.31:0/2302304519 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efd48004ad0 con 0x5637a489d2f0 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.753+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5637a4817d10 con 0x5637a489d2f0 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd397fa700 1 -- 172.21.15.31:0/2302304519 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd48098d00 con 0x5637a489d2f0 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 10 client.29416 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:31.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5637a4603c60 con 0x5637a489d2f0 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd397fa700 1 -- 172.21.15.31:0/2302304519 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5637a4817d10 con 0x5637a489d2f0 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd397fa700 1 -- 172.21.15.31:0/2302304519 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7efd480b11a0 con 0x5637a489d2f0 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 10 client.29416 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 10 client.29416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 20 client.29416 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a48b16e0 0x5637a48b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 4 client.29416 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5637a486ad70 con 0x5637a48b16e0 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.754+0000 7efd55b9b700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a48b16e0 0x5637a48b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.755+0000 7efd55b9b700 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a48b16e0 0x5637a48b3af0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6739ccfa6ad3097 server_cookie=c5d9fd92bdc89f48 in_seq=0 out_seq=0 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.755+0000 7efd397fa700 10 client.29416 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.755+0000 7efd397fa700 1 -- 172.21.15.31:0/2302304519 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5637a486ad70 con 0x5637a48b16e0 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.755+0000 7efd397fa700 10 client.29416 handle_command_reply: tid=0 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 client.29416 shutdown 2023-07-29T16:28:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 msgr2=0x5637a48acec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a48acec0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7efd480c9380 tx=0x7efd48098bc0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 shutdown_connections 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a48b16e0 0x5637a48b3af0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489d2f0 0x5637a48acec0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48a5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 --2- 172.21.15.31:0/2302304519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489c530 0x5637a48a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 >> 172.21.15.31:0/2302304519 conn(0x5637a48913e0 msgr2=0x5637a4896060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd397fa700 0 client.29416 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 shutdown_connections 2023-07-29T16:28:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 1 -- 172.21.15.31:0/2302304519 wait complete. 2023-07-29T16:28:31.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.756+0000 7efd6b742b80 20 client.29416 trim_cache size 0 max 16384 2023-07-29T16:28:31.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.779+0000 7efd6b742b80 1 Processor -- start 2023-07-29T16:28:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 -- start start 2023-07-29T16:28:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489bbe0 0x5637a489d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489d2f0 0x5637a4896060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a48aebb0 con 0x5637a489bbe0 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a48a4df0 con 0x5637a489d2f0 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd6b742b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a48a54b0 con 0x5637a489c530 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd5539a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd5639c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489d2f0 0x5637a4896060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd5539a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39774/0 (socket says 172.21.15.31:39774) 2023-07-29T16:28:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489bbe0 0x5637a489d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.782 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.780+0000 7efd5539a700 1 -- 172.21.15.31:0/3306721964 learned_addr learned my addr 172.21.15.31:0/3306721964 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5539a700 1 -- 172.21.15.31:0/3306721964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489d2f0 msgr2=0x5637a4896060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5539a700 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489d2f0 0x5637a4896060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5539a700 1 -- 172.21.15.31:0/3306721964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489bbe0 msgr2=0x5637a489d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5539a700 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489bbe0 0x5637a489d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.783 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5539a700 1 -- 172.21.15.31:0/3306721964 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5637a4641250 con 0x5637a489c530 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5639c700 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489d2f0 0x5637a4896060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd5539a700 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48ace80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efd40005820 tx=0x7efd400057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38739dd0fb2e4e9e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd44bff700 1 -- 172.21.15.31:0/3306721964 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4000bc20 con 0x5637a489c530 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd44bff700 1 -- 172.21.15.31:0/3306721964 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd4000f920 con 0x5637a489c530 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.781+0000 7efd44bff700 1 -- 172.21.15.31:0/3306721964 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd400299b0 con 0x5637a489c530 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 -- 172.21.15.31:0/3306721964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 msgr2=0x5637a48ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.784 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48ace80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efd40005820 tx=0x7efd400057b0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 -- 172.21.15.31:0/3306721964 shutdown_connections 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489d2f0 0x5637a4896060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489bbe0 0x5637a489d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3306721964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48ace80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 -- 172.21.15.31:0/3306721964 >> 172.21.15.31:0/3306721964 conn(0x5637a48913e0 msgr2=0x5637a4896900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 -- 172.21.15.31:0/3306721964 shutdown_connections 2023-07-29T16:28:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.782+0000 7efd6b742b80 1 -- 172.21.15.31:0/3306721964 wait complete. 2023-07-29T16:28:31.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.783+0000 7efd6b742b80 1 -- start start 2023-07-29T16:28:31.786 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.783+0000 7efd6b742b80 1 Processor -- start 2023-07-29T16:28:31.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48aa820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48aad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489d2f0 0x5637a48abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a48ac2b0 con 0x5637a489d2f0 2023-07-29T16:28:31.788 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a48ac420 con 0x5637a489bbe0 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637a48ac5e0 con 0x5637a489c530 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd6b742b80 10 client.0 _ll_register_callbacks cb 0x7efd48095040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48aa820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44570/0 (socket says 172.21.15.31:44570) 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 -- 172.21.15.31:0/3615360094 learned_addr learned my addr 172.21.15.31:0/3615360094 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd5639c700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489d2f0 0x5637a48abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.789 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd5539a700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48aad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 -- 172.21.15.31:0/3615360094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 msgr2=0x5637a48aad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48aad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 -- 172.21.15.31:0/3615360094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489d2f0 msgr2=0x5637a48abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489d2f0 0x5637a48abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd55b9b700 1 -- 172.21.15.31:0/3615360094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5637a4603c60 con 0x5637a489bbe0 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd5539a700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.784+0000 7efd5639c700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489d2f0 0x5637a48abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:28:31.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd55b9b700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48aa820 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x5637a48b43d0 tx=0x7efd4c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e942c8c69a8348 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd397fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd397fa700 1 -- 172.21.15.31:0/3615360094 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd4c025040 con 0x5637a489bbe0 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd4000bd80 con 0x5637a489bbe0 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd397fa700 1 -- 172.21.15.31:0/3615360094 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7efd4c003cf0 con 0x5637a489bbe0 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd397fa700 1 -- 172.21.15.31:0/3615360094 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd4c017070 con 0x5637a489bbe0 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd6b742b80 10 client.29426 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:31.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.785+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x5637a461ade0 con 0x5637a489bbe0 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd397fa700 1 -- 172.21.15.31:0/3615360094 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd4000bd80 con 0x5637a489bbe0 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd397fa700 1 -- 172.21.15.31:0/3615360094 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7efd4c02ab00 con 0x5637a489bbe0 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd6b742b80 10 client.29426 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd6b742b80 10 client.29426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd6b742b80 20 client.29426 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a4903dd0 0x5637a49061e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd6b742b80 4 client.29426 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:28:31.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5637a47e5d40 con 0x5637a4903dd0 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.786+0000 7efd5539a700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a4903dd0 0x5637a49061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd5539a700 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a4903dd0 0x5637a49061e0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3cf96e054399617 server_cookie=fe623b3a4cb7e57b in_seq=0 out_seq=0 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd397fa700 10 client.29426 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd397fa700 1 -- 172.21.15.31:0/3615360094 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5637a47e5d40 con 0x5637a4903dd0 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd397fa700 10 client.29426 handle_command_reply: tid=0 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 client.29426 shutdown 2023-07-29T16:28:31.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 msgr2=0x5637a48aa820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48aa820 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x5637a48b43d0 tx=0x7efd4c004750 comp rx=0 tx=0).stop 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 shutdown_connections 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5637a4903dd0 0x5637a49061e0 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5637a489d2f0 0x5637a48abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5637a489bbe0 0x5637a48aa820 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 --2- 172.21.15.31:0/3615360094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5637a489c530 0x5637a48aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 >> 172.21.15.31:0/3615360094 conn(0x5637a48913e0 msgr2=0x5637a48ad6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd397fa700 0 client.29426 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:31.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 shutdown_connections 2023-07-29T16:28:31.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.787+0000 7efd6b742b80 1 -- 172.21.15.31:0/3615360094 wait complete. 2023-07-29T16:28:31.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd6b742b80 20 client.29426 trim_cache size 0 max 16384 2023-07-29T16:28:31.795 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:28:31.795 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: "21e7b9cb-68b1-4061-b9f8-c8ff27f2c049": { 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "21e7b9cb-68b1-4061-b9f8-c8ff27f2c049", 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:28:31.796 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:28:31.797 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:28:31.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd3c0a0810 msgr2=0x7efd3c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd3c0a0810 0x7efd3c0a2cc0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x5637a489c940 tx=0x7efd4c009260 comp rx=0 tx=0).stop 2023-07-29T16:28:31.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50106020 msgr2=0x7efd500ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50106020 0x7efd500ff2d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7efd4800efc0 tx=0x7efd4800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 shutdown_connections 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7efd3c0a0810 0x7efd3c0a2cc0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd50106020 0x7efd500ff2d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efd50100de0 0x7efd50100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 --2- 172.21.15.31:0/1928041535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efd50103700 0x7efd500fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:31.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.788+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 >> 172.21.15.31:0/1928041535 conn(0x7efd500fa860 msgr2=0x7efd50102220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:31.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.789+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 shutdown_connections 2023-07-29T16:28:31.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:28:31.789+0000 7efd57dff700 1 -- 172.21.15.31:0/1928041535 wait complete. 2023-07-29T16:28:31.808 INFO:tasks.cephfs.filesystem:scrub status for tag:21e7b9cb-68b1-4061-b9f8-c8ff27f2c049 - {'path': '/', 'tag': '21e7b9cb-68b1-4061-b9f8-c8ff27f2c049', 'options': 'recursive,force'} 2023-07-29T16:28:31.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1018484 8.92 MB/sec execute 496 sec latency 19.696 ms 2023-07-29T16:28:31.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:31 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:28:31.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:31 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:28:31.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:31 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:31.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:31 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:31 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:28:32.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:31 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:28:32.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:31 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:31 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:31 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:28:32.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:31 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:28:32.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:31 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:31 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1019975 8.92 MB/sec execute 497 sec latency 17.440 ms 2023-07-29T16:28:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:32 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:28:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:32 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:32 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:28:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:32 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:32 smithi031 ceph-mon[95504]: pgmap v763: 97 pgs: 97 active+clean; 188 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.9 MiB/s wr, 200 op/s 2023-07-29T16:28:33.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:32 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:28:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:32 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:32 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:28:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:32 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:32 smithi039 ceph-mon[128858]: pgmap v763: 97 pgs: 97 active+clean; 188 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.9 MiB/s wr, 200 op/s 2023-07-29T16:28:33.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:32 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:28:33.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:32 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:33.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:32 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:28:33.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:32 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:28:33.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:32 smithi156 ceph-mon[129245]: pgmap v763: 97 pgs: 97 active+clean; 188 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 1.9 MiB/s wr, 200 op/s 2023-07-29T16:28:33.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1021976 8.92 MB/sec execute 498 sec latency 15.333 ms 2023-07-29T16:28:34.898 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1023467 8.93 MB/sec execute 499 sec latency 23.076 ms 2023-07-29T16:28:34.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:34 smithi031 ceph-mon[95504]: pgmap v764: 97 pgs: 97 active+clean; 186 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.4 MiB/s wr, 290 op/s 2023-07-29T16:28:35.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:34 smithi039 ceph-mon[128858]: pgmap v764: 97 pgs: 97 active+clean; 186 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.4 MiB/s wr, 290 op/s 2023-07-29T16:28:35.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:34 smithi156 ceph-mon[129245]: pgmap v764: 97 pgs: 97 active+clean; 186 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 2.4 MiB/s wr, 290 op/s 2023-07-29T16:28:35.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1025322 8.92 MB/sec execute 500 sec latency 12.279 ms 2023-07-29T16:28:36.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1026921 8.93 MB/sec execute 501 sec latency 48.860 ms 2023-07-29T16:28:36.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:36 smithi031 ceph-mon[95504]: pgmap v765: 97 pgs: 97 active+clean; 187 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-07-29T16:28:37.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:36 smithi039 ceph-mon[128858]: pgmap v765: 97 pgs: 97 active+clean; 187 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-07-29T16:28:37.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:36 smithi156 ceph-mon[129245]: pgmap v765: 97 pgs: 97 active+clean; 187 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.2 MiB/s wr, 234 op/s 2023-07-29T16:28:37.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1028326 8.92 MB/sec execute 502 sec latency 14.157 ms 2023-07-29T16:28:38.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:37 smithi039 ceph-mon[128858]: pgmap v766: 97 pgs: 97 active+clean; 189 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.3 MiB/s wr, 279 op/s 2023-07-29T16:28:38.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:37 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:28:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:37 smithi156 ceph-mon[129245]: pgmap v766: 97 pgs: 97 active+clean; 189 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.3 MiB/s wr, 279 op/s 2023-07-29T16:28:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:37 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:28:38.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:37 smithi031 ceph-mon[95504]: pgmap v766: 97 pgs: 97 active+clean; 189 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.3 MiB/s wr, 279 op/s 2023-07-29T16:28:38.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:37 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:28:38.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1030453 8.93 MB/sec execute 503 sec latency 14.860 ms 2023-07-29T16:28:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:38 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:28:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:38 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:28:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:38 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:28:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:38 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:28:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:38 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:28:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:38 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:28:39.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1031304 8.92 MB/sec execute 504 sec latency 77.810 ms 2023-07-29T16:28:40.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:39 smithi039 ceph-mon[128858]: pgmap v767: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 2.8 MiB/s wr, 295 op/s 2023-07-29T16:28:40.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:39 smithi156 ceph-mon[129245]: pgmap v767: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 2.8 MiB/s wr, 295 op/s 2023-07-29T16:28:40.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:39 smithi031 ceph-mon[95504]: pgmap v767: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 2.8 MiB/s wr, 295 op/s 2023-07-29T16:28:40.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1033580 8.92 MB/sec execute 505 sec latency 18.828 ms 2023-07-29T16:28:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/1722906668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101540 msgr2=0x7f3e24101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/1722906668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101540 0x7f3e24101990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009990 tx=0x7f3e14009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/1722906668 shutdown_connections 2023-07-29T16:28:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/1722906668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24101ed0 0x7f3e24108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/1722906668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24100350 0x7f3e24100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/1722906668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101540 0x7f3e24101990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/1722906668 >> 172.21.15.31:0/1722906668 conn(0x7f3e240fb970 msgr2=0x7f3e240fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/1722906668 shutdown_connections 2023-07-29T16:28:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.316+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/1722906668 wait complete. 2023-07-29T16:28:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.317+0000 7f3e2c5ff700 1 Processor -- start 2023-07-29T16:28:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 -- start start 2023-07-29T16:28:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24100350 0x7f3e24198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 0x7f3e24198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101ed0 0x7f3e241996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24199d50 con 0x7f3e24100350 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24199ec0 con 0x7f3e24101540 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2c5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2419a080 con 0x7f3e24101ed0 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24100350 0x7f3e24198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e29b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 0x7f3e24198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24100350 0x7f3e24198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41112/0 (socket says 172.21.15.31:41112) 2023-07-29T16:28:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e29b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 0x7f3e24198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58358/0 (socket says 172.21.15.31:58358) 2023-07-29T16:28:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.318+0000 7f3e2a39b700 1 -- 172.21.15.31:0/2546192442 learned_addr learned my addr 172.21.15.31:0/2546192442 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e2ab9c700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101ed0 0x7f3e241996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e29b9a700 1 -- 172.21.15.31:0/2546192442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101ed0 msgr2=0x7f3e241996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101ed0 0x7f3e241996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e29b9a700 1 -- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24100350 msgr2=0x7f3e24198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24100350 0x7f3e24198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e29b9a700 1 -- 172.21.15.31:0/2546192442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e140095f0 con 0x7f3e24101540 2023-07-29T16:28:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 0x7f3e24198870 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b680 tx=0x7f3e14009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e1b7fe700 1 -- 172.21.15.31:0/2546192442 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1401d070 con 0x7f3e24101540 2023-07-29T16:28:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2419a300 con 0x7f3e24101540 2023-07-29T16:28:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e1b7fe700 1 -- 172.21.15.31:0/2546192442 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e14021410 con 0x7f3e24101540 2023-07-29T16:28:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.319+0000 7f3e1b7fe700 1 -- 172.21.15.31:0/2546192442 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14017650 con 0x7f3e24101540 2023-07-29T16:28:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.320+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e241a0b20 con 0x7f3e24101540 2023-07-29T16:28:41.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.323+0000 7f3e1b7fe700 1 -- 172.21.15.31:0/2546192442 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3e14021810 con 0x7f3e24101540 2023-07-29T16:28:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.324+0000 7f3e3fe93b80 1 Processor -- start 2023-07-29T16:28:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 -- start start 2023-07-29T16:28:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e1b7fe700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3e100a0790 0x7f3e100a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e1b7fe700 1 -- 172.21.15.31:0/2546192442 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3e140cce10 con 0x7f3e24101540 2023-07-29T16:28:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 0x55b1053c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c6cf0 0x55b1053c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b10531d9b0 con 0x55b1052216b0 2023-07-29T16:28:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b10531b370 con 0x55b1053c5df0 2023-07-29T16:28:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b105316c60 con 0x55b1053c6cf0 2023-07-29T16:28:41.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e2ab9c700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3e100a0790 0x7f3e100a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 0x55b1053c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 0x55b1053c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41124/0 (socket says 172.21.15.31:41124) 2023-07-29T16:28:41.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e2a39b700 1 -- 172.21.15.31:0/3657636318 learned_addr learned my addr 172.21.15.31:0/3657636318 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e2ab9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c6cf0 0x55b1053c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.325+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e29b9a700 1 -- 172.21.15.31:0/3657636318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c6cf0 msgr2=0x55b1053c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c6cf0 0x55b1053c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e29b9a700 1 -- 172.21.15.31:0/3657636318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 msgr2=0x55b1053c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 0x55b1053c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e29b9a700 1 -- 172.21.15.31:0/3657636318 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b105144d80 con 0x55b1053c5df0 2023-07-29T16:28:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 0x55b1053c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e2ab9c700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3e100a0790 0x7f3e100a2c40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009ec0 tx=0x7f3e1c0134a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053c67b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b2e0 tx=0x7f3e1409a400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d89e7075ba1b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e18bfe700 1 -- 172.21.15.31:0/3657636318 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1409a540 con 0x55b1053c5df0 2023-07-29T16:28:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e18bfe700 1 -- 172.21.15.31:0/3657636318 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e1409e570 con 0x55b1053c5df0 2023-07-29T16:28:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.326+0000 7f3e18bfe700 1 -- 172.21.15.31:0/3657636318 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14099c90 con 0x55b1053c5df0 2023-07-29T16:28:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3657636318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 msgr2=0x55b1053c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053c67b0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b2e0 tx=0x7f3e1409a400 comp rx=0 tx=0).stop 2023-07-29T16:28:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3657636318 shutdown_connections 2023-07-29T16:28:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c6cf0 0x55b1053c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1052216b0 0x55b1053c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3657636318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053c67b0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3657636318 >> 172.21.15.31:0/3657636318 conn(0x55b1053bafd0 msgr2=0x55b1053c0300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3657636318 shutdown_connections 2023-07-29T16:28:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.327+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3657636318 wait complete. 2023-07-29T16:28:41.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.328+0000 7f3e3fe93b80 1 -- start start 2023-07-29T16:28:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.328+0000 7f3e3fe93b80 1 Processor -- start 2023-07-29T16:28:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053ceb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1052b34b0 con 0x55b1053c6cf0 2023-07-29T16:28:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1052b3150 con 0x55b1053c5df0 2023-07-29T16:28:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b10519a8c0 con 0x55b1052216b0 2023-07-29T16:28:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e3fe93b80 10 client.0 _ll_register_callbacks cb 0x55b1052bc5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053ceb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053ceb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:33132/0 (socket says 172.21.15.31:33132) 2023-07-29T16:28:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e2a39b700 1 -- 172.21.15.31:0/2722642146 learned_addr learned my addr 172.21.15.31:0/2722642146 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e2ab9c700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e29b9a700 1 -- 172.21.15.31:0/2722642146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 msgr2=0x55b1053ceb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053ceb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e29b9a700 1 -- 172.21.15.31:0/2722642146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 msgr2=0x55b1053d60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.329+0000 7f3e29b9a700 1 -- 172.21.15.31:0/2722642146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e20009a50 con 0x55b1053c5df0 2023-07-29T16:28:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053cf0b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b2e0 tx=0x7f3e140ca7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e0bb0026a72eb2a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e0dffb700 1 -- 172.21.15.31:0/2722642146 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e140b8070 con 0x55b1053c5df0 2023-07-29T16:28:41.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e0dffb700 1 -- 172.21.15.31:0/2722642146 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3e1409a540 con 0x55b1053c5df0 2023-07-29T16:28:41.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b105341cb0 con 0x55b1053c5df0 2023-07-29T16:28:41.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e0dffb700 1 -- 172.21.15.31:0/2722642146 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e14099c30 con 0x55b1053c5df0 2023-07-29T16:28:41.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e3fe93b80 10 client.29438 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.330+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f3e241a0b20 con 0x55b1053c5df0 2023-07-29T16:28:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e0dffb700 1 -- 172.21.15.31:0/2722642146 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b105341cb0 con 0x55b1053c5df0 2023-07-29T16:28:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e0dffb700 1 -- 172.21.15.31:0/2722642146 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f3e140964a0 con 0x55b1053c5df0 2023-07-29T16:28:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e3fe93b80 10 client.29438 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e3fe93b80 10 client.29438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e3fe93b80 20 client.29438 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b1053da470 0x55b1053dc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e3fe93b80 4 client.29438 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b105394d10 con 0x55b1053da470 2023-07-29T16:28:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.331+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b1053da470 0x55b1053dc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.332+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b1053da470 0x55b1053dc880 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1332eccb1be4a906 server_cookie=bc09ed9fef7a5e52 in_seq=0 out_seq=0 2023-07-29T16:28:41.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.332+0000 7f3e0dffb700 10 client.29438 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e0dffb700 1 -- 172.21.15.31:0/2722642146 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b105394d10 con 0x55b1053da470 2023-07-29T16:28:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e0dffb700 10 client.29438 handle_command_reply: tid=0 2023-07-29T16:28:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 client.29438 shutdown 2023-07-29T16:28:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 msgr2=0x55b1053cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053cf0b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b2e0 tx=0x7f3e140ca7f0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 shutdown_connections 2023-07-29T16:28:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b1053da470 0x55b1053dc880 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053ceb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/2722642146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053cf0b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e0dffb700 0 client.29438 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.333+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 >> 172.21.15.31:0/2722642146 conn(0x55b1053bafd0 msgr2=0x55b1053bfa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:41.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.334+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 shutdown_connections 2023-07-29T16:28:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.334+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/2722642146 wait complete. 2023-07-29T16:28:41.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.334+0000 7f3e3fe93b80 20 client.29438 trim_cache size 0 max 16384 2023-07-29T16:28:41.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.361+0000 7f3e3fe93b80 1 Processor -- start 2023-07-29T16:28:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.361+0000 7f3e3fe93b80 1 -- start start 2023-07-29T16:28:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1053cfd60 con 0x55b1053c6cf0 2023-07-29T16:28:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1053baa20 con 0x55b1053c5df0 2023-07-29T16:28:41.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1053cfa70 con 0x55b1052216b0 2023-07-29T16:28:41.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2ab9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053c6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:33134/0 (socket says 172.21.15.31:33134) 2023-07-29T16:28:41.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 -- 172.21.15.31:0/894372757 learned_addr learned my addr 172.21.15.31:0/894372757 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:41.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 -- 172.21.15.31:0/894372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 msgr2=0x55b1053d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053d60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 -- 172.21.15.31:0/894372757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 msgr2=0x55b1053bfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053bfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.362+0000 7f3e2a39b700 1 -- 172.21.15.31:0/894372757 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b10512dc00 con 0x55b1052216b0 2023-07-29T16:28:41.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.363+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053c6490 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x55b1053dd160 tx=0x7f3e20009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=706774f8eec4f685 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:41.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.363+0000 7f3e193ff700 1 -- 172.21.15.31:0/894372757 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20016040 con 0x55b1052216b0 2023-07-29T16:28:41.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.363+0000 7f3e193ff700 1 -- 172.21.15.31:0/894372757 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e20017040 con 0x55b1052216b0 2023-07-29T16:28:41.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.363+0000 7f3e193ff700 1 -- 172.21.15.31:0/894372757 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20004c50 con 0x55b1052216b0 2023-07-29T16:28:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/894372757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 msgr2=0x55b1053c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053c6490 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x55b1053dd160 tx=0x7f3e20009260 comp rx=0 tx=0).stop 2023-07-29T16:28:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/894372757 shutdown_connections 2023-07-29T16:28:41.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053bfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1052216b0 0x55b1053c6490 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/894372757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1053c5df0 0x55b1053d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/894372757 >> 172.21.15.31:0/894372757 conn(0x55b1053bafd0 msgr2=0x55b1053bc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:41.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/894372757 shutdown_connections 2023-07-29T16:28:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/894372757 wait complete. 2023-07-29T16:28:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.364+0000 7f3e3fe93b80 1 -- start start 2023-07-29T16:28:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 Processor -- start 2023-07-29T16:28:41.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 0x55b1053d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c5df0 0x55b1053d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1053d5ca0 con 0x55b1053c6cf0 2023-07-29T16:28:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1053d5e10 con 0x55b1052216b0 2023-07-29T16:28:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b1053d5fd0 con 0x55b1053c5df0 2023-07-29T16:28:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e3fe93b80 10 client.0 _ll_register_callbacks cb 0x55b1052bc5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:41.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 0x55b1053d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e2a39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 0x55b1053d2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58410/0 (socket says 172.21.15.31:58410) 2023-07-29T16:28:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e2a39b700 1 -- 172.21.15.31:0/3107692817 learned_addr learned my addr 172.21.15.31:0/3107692817 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e29b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c5df0 0x55b1053d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.365+0000 7f3e2ab9c700 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e2a39b700 1 -- 172.21.15.31:0/3107692817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c5df0 msgr2=0x55b1053d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c5df0 0x55b1053d2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e2a39b700 1 -- 172.21.15.31:0/3107692817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 msgr2=0x55b1053d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:41.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e2a39b700 1 -- 172.21.15.31:0/3107692817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e20009e80 con 0x55b1052216b0 2023-07-29T16:28:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e2a39b700 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 0x55b1053d2200 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3e200096b0 tx=0x7f3e20010e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d80b0d799b1a7fb7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:41.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.366+0000 7f3e0dffb700 1 -- 172.21.15.31:0/3107692817 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20016050 con 0x55b1052216b0 2023-07-29T16:28:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e1409e570 con 0x55b1052216b0 2023-07-29T16:28:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e0dffb700 1 -- 172.21.15.31:0/3107692817 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3e20004bd0 con 0x55b1052216b0 2023-07-29T16:28:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e0dffb700 1 -- 172.21.15.31:0/3107692817 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e20004d90 con 0x55b1052216b0 2023-07-29T16:28:41.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 10 client.29444 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55b10516b1f0 con 0x55b1052216b0 2023-07-29T16:28:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e0dffb700 1 -- 172.21.15.31:0/3107692817 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e1409e570 con 0x55b1052216b0 2023-07-29T16:28:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e0dffb700 1 -- 172.21.15.31:0/3107692817 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f3e20024020 con 0x55b1052216b0 2023-07-29T16:28:41.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 10 client.29444 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 10 client.29444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 20 client.29444 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b10542d1f0 0x55b10542f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:41.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 4 client.29444 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:28:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.367+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b10530fce0 con 0x55b10542d1f0 2023-07-29T16:28:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.368+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b10542d1f0 0x55b10542f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.368+0000 7f3e29b9a700 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b10542d1f0 0x55b10542f600 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd7f0c5b6a0f86a server_cookie=3e65e65c2e56af8b in_seq=0 out_seq=0 2023-07-29T16:28:41.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.368+0000 7f3e0dffb700 10 client.29444 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.442+0000 7f3e0dffb700 1 -- 172.21.15.31:0/3107692817 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b10530fce0 con 0x55b10542d1f0 2023-07-29T16:28:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.442+0000 7f3e0dffb700 10 client.29444 handle_command_reply: tid=0 2023-07-29T16:28:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.442+0000 7f3e3fe93b80 1 client.29444 shutdown 2023-07-29T16:28:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 msgr2=0x55b1053d2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 0x55b1053d2200 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3e200096b0 tx=0x7f3e20010e40 comp rx=0 tx=0).stop 2023-07-29T16:28:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 shutdown_connections 2023-07-29T16:28:41.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b10542d1f0 0x55b10542f600 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1053c6cf0 0x55b1053d5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b1052216b0 0x55b1053d2200 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 --2- 172.21.15.31:0/3107692817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b1053c5df0 0x55b1053d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e0dffb700 0 client.29444 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:41.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 >> 172.21.15.31:0/3107692817 conn(0x55b1053bafd0 msgr2=0x55b1053c6490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 shutdown_connections 2023-07-29T16:28:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 1 -- 172.21.15.31:0/3107692817 wait complete. 2023-07-29T16:28:41.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.443+0000 7f3e3fe93b80 20 client.29444 trim_cache size 0 max 16384 2023-07-29T16:28:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:28:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:28:41.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:28:41.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:28:41.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3e100a0790 msgr2=0x7f3e100a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3e100a0790 0x7f3e100a2c40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009ec0 tx=0x7f3e1c0134a0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 msgr2=0x7f3e24198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 0x7f3e24198870 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3e1400b680 tx=0x7f3e14009580 comp rx=0 tx=0).stop 2023-07-29T16:28:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 shutdown_connections 2023-07-29T16:28:41.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3e100a0790 0x7f3e100a2c40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e24101ed0 0x7f3e241996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e24100350 0x7f3e24198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 --2- 172.21.15.31:0/2546192442 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e24101540 0x7f3e24198870 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 >> 172.21.15.31:0/2546192442 conn(0x7f3e240fb970 msgr2=0x7f3e24106e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 shutdown_connections 2023-07-29T16:28:41.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:41.444+0000 7f3e2c5ff700 1 -- 172.21.15.31:0/2546192442 wait complete. 2023-07-29T16:28:41.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1034245 8.92 MB/sec execute 506 sec latency 69.735 ms 2023-07-29T16:28:42.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:42 smithi156 ceph-mon[129245]: pgmap v768: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 2.0 MiB/s wr, 215 op/s 2023-07-29T16:28:42.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1036374 8.91 MB/sec execute 507 sec latency 12.731 ms 2023-07-29T16:28:42.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:42 smithi031 ceph-mon[95504]: pgmap v768: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 2.0 MiB/s wr, 215 op/s 2023-07-29T16:28:43.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:42 smithi039 ceph-mon[128858]: pgmap v768: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 2.0 MiB/s wr, 215 op/s 2023-07-29T16:28:43.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1037713 8.92 MB/sec execute 508 sec latency 18.181 ms 2023-07-29T16:28:44.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1039159 8.91 MB/sec execute 509 sec latency 103.815 ms 2023-07-29T16:28:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:44 smithi031 ceph-mon[95504]: pgmap v769: 97 pgs: 97 active+clean; 188 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.7 MiB/s wr, 295 op/s 2023-07-29T16:28:45.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:44 smithi039 ceph-mon[128858]: pgmap v769: 97 pgs: 97 active+clean; 188 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.7 MiB/s wr, 295 op/s 2023-07-29T16:28:45.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:44 smithi156 ceph-mon[129245]: pgmap v769: 97 pgs: 97 active+clean; 188 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 2.7 MiB/s wr, 295 op/s 2023-07-29T16:28:45.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1041109 8.92 MB/sec execute 510 sec latency 21.718 ms 2023-07-29T16:28:46.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1042130 8.91 MB/sec execute 511 sec latency 59.604 ms 2023-07-29T16:28:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:46 smithi031 ceph-mon[95504]: pgmap v770: 97 pgs: 97 active+clean; 188 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 2.5 MiB/s wr, 241 op/s 2023-07-29T16:28:47.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:46 smithi039 ceph-mon[128858]: pgmap v770: 97 pgs: 97 active+clean; 188 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 2.5 MiB/s wr, 241 op/s 2023-07-29T16:28:47.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:46 smithi156 ceph-mon[129245]: pgmap v770: 97 pgs: 97 active+clean; 188 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 2.5 MiB/s wr, 241 op/s 2023-07-29T16:28:47.901 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1044381 8.92 MB/sec execute 512 sec latency 27.792 ms 2023-07-29T16:28:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:47 smithi039 ceph-mon[128858]: pgmap v771: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.4 MiB/s wr, 272 op/s 2023-07-29T16:28:48.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:47 smithi156 ceph-mon[129245]: pgmap v771: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.4 MiB/s wr, 272 op/s 2023-07-29T16:28:48.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:47 smithi031 ceph-mon[95504]: pgmap v771: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.4 MiB/s wr, 272 op/s 2023-07-29T16:28:48.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1045242 8.91 MB/sec execute 513 sec latency 16.158 ms 2023-07-29T16:28:49.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1047438 8.91 MB/sec execute 514 sec latency 13.343 ms 2023-07-29T16:28:50.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1048445 8.91 MB/sec execute 515 sec latency 31.289 ms 2023-07-29T16:28:50.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:50 smithi031 ceph-mon[95504]: pgmap v772: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.5 MiB/s wr, 247 op/s 2023-07-29T16:28:51.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:50 smithi039 ceph-mon[128858]: pgmap v772: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.5 MiB/s wr, 247 op/s 2023-07-29T16:28:51.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:50 smithi156 ceph-mon[129245]: pgmap v772: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.5 MiB/s wr, 247 op/s 2023-07-29T16:28:51.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1050414 8.91 MB/sec execute 516 sec latency 32.710 ms 2023-07-29T16:28:52.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1051834 8.91 MB/sec execute 517 sec latency 17.327 ms 2023-07-29T16:28:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:52 smithi031 ceph-mon[95504]: pgmap v773: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2023-07-29T16:28:53.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:52 smithi039 ceph-mon[128858]: pgmap v773: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2023-07-29T16:28:53.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:52 smithi156 ceph-mon[129245]: pgmap v773: 97 pgs: 97 active+clean; 190 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2023-07-29T16:28:53.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1053423 8.90 MB/sec execute 518 sec latency 16.983 ms 2023-07-29T16:28:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3062493156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44100e60 msgr2=0x7f5d44103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3062493156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44100e60 0x7f5d44103240 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f5d40009990 tx=0x7f5d40009ca0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3062493156 shutdown_connections 2023-07-29T16:28:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3062493156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d441060a0 0x7f5d44108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3062493156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44100e60 0x7f5d44103240 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3062493156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d44103780 0x7f5d44105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3062493156 >> 172.21.15.31:0/3062493156 conn(0x7f5d440fa840 msgr2=0x7f5d440fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3062493156 shutdown_connections 2023-07-29T16:28:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.613+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3062493156 wait complete. 2023-07-29T16:28:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.614+0000 7f5d4bdff700 1 Processor -- start 2023-07-29T16:28:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.614+0000 7f5d4bdff700 1 -- start start 2023-07-29T16:28:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4bdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 0x7f5d440fee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4bdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44103780 0x7f5d440ff350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4bdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d441060a0 0x7f5d440ff890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4bdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d441a00e0 con 0x7f5d44100e60 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4bdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d441a0220 con 0x7f5d44103780 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4bdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d441a0360 con 0x7f5d441060a0 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 0x7f5d440fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 0x7f5d440fee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37666/0 (socket says 172.21.15.31:37666) 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 -- 172.21.15.31:0/3420185955 learned_addr learned my addr 172.21.15.31:0/3420185955 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4939a700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44103780 0x7f5d440ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d441060a0 0x7f5d440ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 -- 172.21.15.31:0/3420185955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d441060a0 msgr2=0x7f5d440ff890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d441060a0 0x7f5d440ff890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 -- 172.21.15.31:0/3420185955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44103780 msgr2=0x7f5d440ff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44103780 0x7f5d440ff350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d49b9b700 1 -- 172.21.15.31:0/3420185955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d400095f0 con 0x7f5d44100e60 2023-07-29T16:28:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.615+0000 7f5d4939a700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44103780 0x7f5d440ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:28:54.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.616+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 0x7f5d440fee10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5d40000c00 tx=0x7f5d40005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:54.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.616+0000 7f5d3affd700 1 -- 172.21.15.31:0/3420185955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4001d070 con 0x7f5d44100e60 2023-07-29T16:28:54.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.616+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d441a0590 con 0x7f5d44100e60 2023-07-29T16:28:54.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.616+0000 7f5d3affd700 1 -- 172.21.15.31:0/3420185955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d40022470 con 0x7f5d44100e60 2023-07-29T16:28:54.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.616+0000 7f5d3affd700 1 -- 172.21.15.31:0/3420185955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40017650 con 0x7f5d44100e60 2023-07-29T16:28:54.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.616+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d441a09c0 con 0x7f5d44100e60 2023-07-29T16:28:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.619+0000 7f5d3affd700 1 -- 172.21.15.31:0/3420185955 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5d400225e0 con 0x7f5d44100e60 2023-07-29T16:28:54.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 Processor -- start 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d3affd700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5d300a0790 0x7f5d300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 -- start start 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d3affd700 1 -- 172.21.15.31:0/3420185955 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5d400ccd70 con 0x7f5d44100e60 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5d300a0790 0x7f5d300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5239b0 con 0x55582d5cbdf0 2023-07-29T16:28:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d521370 con 0x55582d4276b0 2023-07-29T16:28:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.621+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d51cc60 con 0x55582d5cccf0 2023-07-29T16:28:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37416/0 (socket says 172.21.15.31:37416) 2023-07-29T16:28:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 -- 172.21.15.31:0/2933008798 learned_addr learned my addr 172.21.15.31:0/2933008798 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d4939a700 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 -- 172.21.15.31:0/2933008798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 msgr2=0x55582d5cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 -- 172.21.15.31:0/2933008798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 msgr2=0x55582d5cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 -- 172.21.15.31:0/2933008798 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55582d34ad80 con 0x55582d4276b0 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cb8b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5d40096f10 tx=0x7f5d4009a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eea726548505963 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5d300a0790 0x7f5d300a2c40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009fd0 tx=0x7f5d3c00fc20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d2ffff700 1 -- 172.21.15.31:0/2933008798 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4009a4a0 con 0x55582d4276b0 2023-07-29T16:28:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d2ffff700 1 -- 172.21.15.31:0/2933008798 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d4009e4d0 con 0x55582d4276b0 2023-07-29T16:28:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.622+0000 7f5d2ffff700 1 -- 172.21.15.31:0/2933008798 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40099b90 con 0x55582d4276b0 2023-07-29T16:28:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/2933008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 msgr2=0x55582d5cb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cb8b0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f5d40096f10 tx=0x7f5d4009a360 comp rx=0 tx=0).stop 2023-07-29T16:28:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/2933008798 shutdown_connections 2023-07-29T16:28:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cb8b0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/2933008798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/2933008798 >> 172.21.15.31:0/2933008798 conn(0x55582d5c0fd0 msgr2=0x55582d5c6300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/2933008798 shutdown_connections 2023-07-29T16:28:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.623+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/2933008798 wait complete. 2023-07-29T16:28:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.624+0000 7f5d5f6a4b80 1 -- start start 2023-07-29T16:28:54.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.624+0000 7f5d5f6a4b80 1 Processor -- start 2023-07-29T16:28:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d4276b0 0x55582d5d4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 0x55582d5d50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5dc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d4b94b0 con 0x55582d4276b0 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d4b9150 con 0x55582d5cbdf0 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d3a08c0 con 0x55582d5cccf0 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 0x55582d5d50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d5f6a4b80 10 client.0 _ll_register_callbacks cb 0x55582d4c25c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 0x55582d5d50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37430/0 (socket says 172.21.15.31:37430) 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 -- 172.21.15.31:0/505050836 learned_addr learned my addr 172.21.15.31:0/505050836 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5dc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d4276b0 0x55582d5d4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 -- 172.21.15.31:0/505050836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 msgr2=0x55582d5dc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5dc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 -- 172.21.15.31:0/505050836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d4276b0 msgr2=0x55582d5d4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d4276b0 0x55582d5d4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.625+0000 7f5d4939a700 1 -- 172.21.15.31:0/505050836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d441a09c0 con 0x55582d5cbdf0 2023-07-29T16:28:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d4939a700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 0x55582d5d50a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5d34012cf0 tx=0x7f5d34009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc2c0b66eebe60a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d2d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/505050836 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34017070 con 0x55582d5cbdf0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55582d547cb0 con 0x55582d5cbdf0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/505050836 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5d34004830 con 0x55582d5cbdf0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/505050836 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d340122e0 con 0x55582d5cbdf0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d5f6a4b80 10 client.29456 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55582d333c00 con 0x55582d5cbdf0 2023-07-29T16:28:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/505050836 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55582d547cb0 con 0x55582d5cbdf0 2023-07-29T16:28:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.626+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/505050836 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f5d3401c9f0 con 0x55582d5cbdf0 2023-07-29T16:28:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d5f6a4b80 10 client.29456 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d5f6a4b80 10 client.29456 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d5f6a4b80 20 client.29456 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d5e0980 0x55582d5e2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d5f6a4b80 4 client.29456 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:28:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55582d59ad10 con 0x55582d5e0980 2023-07-29T16:28:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d5e0980 0x55582d5e2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d5e0980 0x55582d5e2d90 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bcb60a59b4046be server_cookie=7238df9c1edff98 in_seq=0 out_seq=0 2023-07-29T16:28:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.627+0000 7f5d2d7fa700 10 client.29456 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/505050836 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55582d59ad10 con 0x55582d5e0980 2023-07-29T16:28:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d2d7fa700 10 client.29456 handle_command_reply: tid=0 2023-07-29T16:28:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 client.29456 shutdown 2023-07-29T16:28:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 msgr2=0x55582d5d50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 0x55582d5d50a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f5d34012cf0 tx=0x7f5d34009e30 comp rx=0 tx=0).stop 2023-07-29T16:28:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 shutdown_connections 2023-07-29T16:28:54.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d5e0980 0x55582d5e2d90 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5dc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d4276b0 0x55582d5d4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/505050836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cbdf0 0x55582d5d50a0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 >> 172.21.15.31:0/505050836 conn(0x55582d5c0fd0 msgr2=0x55582d5c5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d2d7fa700 0 client.29456 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 shutdown_connections 2023-07-29T16:28:54.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/505050836 wait complete. 2023-07-29T16:28:54.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.629+0000 7f5d5f6a4b80 20 client.29456 trim_cache size 0 max 16384 2023-07-29T16:28:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.644+0000 7f5d5f6a4b80 1 Processor -- start 2023-07-29T16:28:54.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.644+0000 7f5d5f6a4b80 1 -- start start 2023-07-29T16:28:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5dc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5c7740 con 0x55582d5cbdf0 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5cc4f0 con 0x55582d4276b0 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5d57e0 con 0x55582d5cccf0 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37438/0 (socket says 172.21.15.31:37438) 2023-07-29T16:28:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 -- 172.21.15.31:0/1799122503 learned_addr learned my addr 172.21.15.31:0/1799122503 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d4939a700 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5dc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 -- 172.21.15.31:0/1799122503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 msgr2=0x55582d5d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 -- 172.21.15.31:0/1799122503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 msgr2=0x55582d5dc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5dc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d49b9b700 1 -- 172.21.15.31:0/1799122503 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55582d3711f0 con 0x55582d4276b0 2023-07-29T16:28:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.645+0000 7f5d4939a700 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5dc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:28:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.646+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cfb00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5d400969a0 tx=0x7f5d400228e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f3bfad038b09a0e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.646+0000 7f5d38bff700 1 -- 172.21.15.31:0/1799122503 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4009a270 con 0x55582d4276b0 2023-07-29T16:28:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.646+0000 7f5d38bff700 1 -- 172.21.15.31:0/1799122503 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d40009290 con 0x55582d4276b0 2023-07-29T16:28:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.646+0000 7f5d38bff700 1 -- 172.21.15.31:0/1799122503 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4009e2a0 con 0x55582d4276b0 2023-07-29T16:28:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/1799122503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 msgr2=0x55582d5cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cfb00 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5d400969a0 tx=0x7f5d400228e0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/1799122503 shutdown_connections 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d5cccf0 0x55582d5d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d4276b0 0x55582d5cfb00 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/1799122503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5dc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/1799122503 >> 172.21.15.31:0/1799122503 conn(0x55582d5c0fd0 msgr2=0x55582d5d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/1799122503 shutdown_connections 2023-07-29T16:28:54.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/1799122503 wait complete. 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 -- start start 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.647+0000 7f5d5f6a4b80 1 Processor -- start 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 0x55582d5d82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5d8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cccf0 0x55582d5db750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5dbc90 con 0x55582d5cbdf0 2023-07-29T16:28:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5dbe00 con 0x55582d5cccf0 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55582d5dbfc0 con 0x55582d4276b0 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d5f6a4b80 10 client.0 _ll_register_callbacks cb 0x55582d4c25c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 0x55582d5d82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 0x55582d5d82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54464/0 (socket says 172.21.15.31:54464) 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 -- 172.21.15.31:0/554936360 learned_addr learned my addr 172.21.15.31:0/554936360 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d4a39c700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cccf0 0x55582d5db750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d4939a700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5d8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 -- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cccf0 msgr2=0x55582d5db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cccf0 0x55582d5db750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 -- 172.21.15.31:0/554936360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 msgr2=0x55582d5d8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5d8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.648+0000 7f5d49b9b700 1 -- 172.21.15.31:0/554936360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55582d34ad80 con 0x55582d4276b0 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d49b9b700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 0x55582d5d82d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000b680 tx=0x7f5d4009aac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b63c6de399b5f661 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d2d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:28:54.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/554936360 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4009a270 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/554936360 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5d40099990 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d4009e4d0 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/554936360 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d4009a4a0 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 10 client.29455 fetch_fsmap learned FSMap version 33 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55582d349250 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/554936360 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5d4009e4d0 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/554936360 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f5d400a00d0 con 0x55582d4276b0 2023-07-29T16:28:54.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 10 client.29455 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:28:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 10 client.29455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:28:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 20 client.29455 populate_metadata read hostname 'smithi031' 2023-07-29T16:28:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d6333d0 0x55582d6357e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:28:54.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 4 client.29455 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:28:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.649+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55582d515ce0 con 0x55582d6333d0 2023-07-29T16:28:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.650+0000 7f5d4939a700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d6333d0 0x55582d6357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:28:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.650+0000 7f5d4939a700 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d6333d0 0x55582d6357e0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b14d1a1934a71948 server_cookie=73a239daf0818929 in_seq=0 out_seq=0 2023-07-29T16:28:54.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.650+0000 7f5d2d7fa700 10 client.29455 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d2d7fa700 1 -- 172.21.15.31:0/554936360 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55582d515ce0 con 0x55582d6333d0 2023-07-29T16:28:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d2d7fa700 10 client.29455 handle_command_reply: tid=0 2023-07-29T16:28:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 client.29455 shutdown 2023-07-29T16:28:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 msgr2=0x55582d5d82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 0x55582d5d82d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000b680 tx=0x7f5d4009aac0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 shutdown_connections 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55582d6333d0 0x55582d6357e0 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55582d5cccf0 0x55582d5db750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55582d4276b0 0x55582d5d82d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 --2- 172.21.15.31:0/554936360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55582d5cbdf0 0x55582d5d8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d2d7fa700 0 client.29455 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:28:54.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.733+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 >> 172.21.15.31:0/554936360 conn(0x55582d5c0fd0 msgr2=0x55582d5d4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 shutdown_connections 2023-07-29T16:28:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d5f6a4b80 1 -- 172.21.15.31:0/554936360 wait complete. 2023-07-29T16:28:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d5f6a4b80 20 client.29455 trim_cache size 0 max 16384 2023-07-29T16:28:54.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:28:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:28:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:28:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:28:54.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5d300a0790 msgr2=0x7f5d300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5d300a0790 0x7f5d300a2c40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c009fd0 tx=0x7f5d3c00fc20 comp rx=0 tx=0).stop 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 msgr2=0x7f5d440fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.734+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 0x7f5d440fee10 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5d40000c00 tx=0x7f5d40005720 comp rx=0 tx=0).stop 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 shutdown_connections 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5d300a0790 0x7f5d300a2c40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d441060a0 0x7f5d440ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5d44100e60 0x7f5d440fee10 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 --2- 172.21.15.31:0/3420185955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d44103780 0x7f5d440ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:28:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 >> 172.21.15.31:0/3420185955 conn(0x7f5d440fa840 msgr2=0x7f5d44102320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:28:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 shutdown_connections 2023-07-29T16:28:54.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:28:54.735+0000 7f5d4bdff700 1 -- 172.21.15.31:0/3420185955 wait complete. 2023-07-29T16:28:54.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1055265 8.91 MB/sec execute 519 sec latency 51.832 ms 2023-07-29T16:28:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:54 smithi031 ceph-mon[95504]: pgmap v774: 97 pgs: 97 active+clean; 193 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 2.5 MiB/s wr, 278 op/s 2023-07-29T16:28:55.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:54 smithi039 ceph-mon[128858]: pgmap v774: 97 pgs: 97 active+clean; 193 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 2.5 MiB/s wr, 278 op/s 2023-07-29T16:28:55.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:54 smithi156 ceph-mon[129245]: pgmap v774: 97 pgs: 97 active+clean; 193 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 2.5 MiB/s wr, 278 op/s 2023-07-29T16:28:55.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1056243 8.90 MB/sec execute 520 sec latency 42.726 ms 2023-07-29T16:28:56.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1058044 8.90 MB/sec execute 521 sec latency 44.094 ms 2023-07-29T16:28:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:56 smithi031 ceph-mon[95504]: pgmap v775: 97 pgs: 97 active+clean; 194 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.3 MiB/s wr, 220 op/s 2023-07-29T16:28:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:56 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:28:57.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:56 smithi039 ceph-mon[128858]: pgmap v775: 97 pgs: 97 active+clean; 194 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.3 MiB/s wr, 220 op/s 2023-07-29T16:28:57.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:56 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:28:57.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:56 smithi156 ceph-mon[129245]: pgmap v775: 97 pgs: 97 active+clean; 194 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.3 MiB/s wr, 220 op/s 2023-07-29T16:28:57.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:56 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:28:57.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1059064 8.90 MB/sec execute 522 sec latency 21.463 ms 2023-07-29T16:28:57.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:28:57.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:28:57.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:57 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:28:58.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:28:58.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:28:58.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:57 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:28:58.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:28:58.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:28:58.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:57 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:28:58.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1061089 8.90 MB/sec execute 523 sec latency 13.583 ms 2023-07-29T16:28:58.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:28:58 smithi031 ceph-mon[95504]: pgmap v776: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 2.2 MiB/s wr, 245 op/s 2023-07-29T16:28:59.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:28:58 smithi039 ceph-mon[128858]: pgmap v776: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 2.2 MiB/s wr, 245 op/s 2023-07-29T16:28:59.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:28:58 smithi156 ceph-mon[129245]: pgmap v776: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 2.2 MiB/s wr, 245 op/s 2023-07-29T16:28:59.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1062491 8.90 MB/sec execute 524 sec latency 19.089 ms 2023-07-29T16:29:00.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1063974 8.89 MB/sec execute 525 sec latency 14.943 ms 2023-07-29T16:29:00.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:00 smithi031 ceph-mon[95504]: pgmap v777: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 2.6 MiB/s wr, 256 op/s 2023-07-29T16:29:01.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:00 smithi039 ceph-mon[128858]: pgmap v777: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 2.6 MiB/s wr, 256 op/s 2023-07-29T16:29:01.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:00 smithi156 ceph-mon[129245]: pgmap v777: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 2.6 MiB/s wr, 256 op/s 2023-07-29T16:29:01.809 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:29:01.902 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1065769 8.90 MB/sec execute 526 sec latency 43.979 ms 2023-07-29T16:29:02.033 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 -- 172.21.15.31:0/3420697591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 msgr2=0x7f0578108440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 --2- 172.21.15.31:0/3420697591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f0578108440 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f057000b0a0 tx=0x7f057000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 -- 172.21.15.31:0/3420697591 shutdown_connections 2023-07-29T16:29:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 --2- 172.21.15.31:0/3420697591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f0578108440 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 --2- 172.21.15.31:0/3420697591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0578100da0 0x7f0578103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 --2- 172.21.15.31:0/3420697591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f05781036c0 0x7f0578105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.034 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 -- 172.21.15.31:0/3420697591 >> 172.21.15.31:0/3420697591 conn(0x7f05780fa840 msgr2=0x7f05780fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 -- 172.21.15.31:0/3420697591 shutdown_connections 2023-07-29T16:29:02.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.034+0000 7f057e5ff700 1 -- 172.21.15.31:0/3420697591 wait complete. 2023-07-29T16:29:02.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.036+0000 7f057e5ff700 1 Processor -- start 2023-07-29T16:29:02.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.036+0000 7f057e5ff700 1 -- start start 2023-07-29T16:29:02.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.036+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 0x7f05781961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.036 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05781036c0 0x7f0578196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f057819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057819dd40 con 0x7f0578100da0 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057819de80 con 0x7f0578105fe0 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057819e010 con 0x7f05781036c0 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f0577fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 0x7f05781961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f05777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05781036c0 0x7f0578196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057cb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f057819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f0577fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 0x7f05781961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51730/0 (socket says 172.21.15.31:51730) 2023-07-29T16:29:02.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f0577fff700 1 -- 172.21.15.31:0/193470541 learned_addr learned my addr 172.21.15.31:0/193470541 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057cb9c700 1 -- 172.21.15.31:0/193470541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05781036c0 msgr2=0x7f0578196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057cb9c700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05781036c0 0x7f0578196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057cb9c700 1 -- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 msgr2=0x7f05781961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057cb9c700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 0x7f05781961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f057cb9c700 1 -- 172.21.15.31:0/193470541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570009d00 con 0x7f0578105fe0 2023-07-29T16:29:02.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f05777fe700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05781036c0 0x7f0578196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.037+0000 7f0577fff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 0x7f05781961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.038+0000 7f057cb9c700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f057819d770 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0570009fd0 tx=0x7f0570007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.038+0000 7f05757fa700 1 -- 172.21.15.31:0/193470541 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057000e040 con 0x7f0578105fe0 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.038+0000 7f05757fa700 1 -- 172.21.15.31:0/193470541 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0570004c50 con 0x7f0578105fe0 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.038+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057819e290 con 0x7f0578105fe0 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.038+0000 7f05757fa700 1 -- 172.21.15.31:0/193470541 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057001b6c0 con 0x7f0578105fe0 2023-07-29T16:29:02.040 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.038+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057819e6b0 con 0x7f0578105fe0 2023-07-29T16:29:02.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.042+0000 7f05757fa700 1 -- 172.21.15.31:0/193470541 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f057001d470 con 0x7f0578105fe0 2023-07-29T16:29:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 Processor -- start 2023-07-29T16:29:02.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 -- start start 2023-07-29T16:29:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f05757fa700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05600a0810 0x7f05600a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabbbe0 0x5597aaabbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 0x5597aaabcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaabd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaa13a10 con 0x5597aaabbbe0 2023-07-29T16:29:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaa113d0 con 0x5597aaabd2f0 2023-07-29T16:29:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.043+0000 7f0591f98b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaa0ccc0 con 0x5597aaabc530 2023-07-29T16:29:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05757fa700 1 -- 172.21.15.31:0/193470541 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f05700ccc40 con 0x7f0578105fe0 2023-07-29T16:29:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 0x5597aaabcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f057cb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaabd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f0577fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabbbe0 0x5597aaabbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 0x5597aaabcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48318/0 (socket says 172.21.15.31:48318) 2023-07-29T16:29:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05777fe700 1 -- 172.21.15.31:0/1202847397 learned_addr learned my addr 172.21.15.31:0/1202847397 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f057cb9c700 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaabd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41732/0 (socket says 172.21.15.31:41732) 2023-07-29T16:29:02.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05777fe700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05600a0810 0x7f05600a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05777fe700 1 -- 172.21.15.31:0/1202847397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 msgr2=0x5597aaabd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.044+0000 7f05777fe700 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaabd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f05777fe700 1 -- 172.21.15.31:0/1202847397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabbbe0 msgr2=0x5597aaabbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f05777fe700 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabbbe0 0x5597aaabbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f05777fe700 1 -- 172.21.15.31:0/1202847397 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597aa83ade0 con 0x5597aaabc530 2023-07-29T16:29:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f0577fff700 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabbbe0 0x5597aaabbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f057cb9c700 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaabd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f05777fe700 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 0x5597aaabcd60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f056c011fb0 tx=0x7f056c00f6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e0eaa5af16b98da server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f05777fe700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05600a0810 0x7f05600a2cc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f056c00f9f0 tx=0x7f056c018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f055e7fc700 1 -- 172.21.15.31:0/1202847397 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c011050 con 0x5597aaabc530 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f055e7fc700 1 -- 172.21.15.31:0/1202847397 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f056c0111b0 con 0x5597aaabc530 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.045+0000 7f055e7fc700 1 -- 172.21.15.31:0/1202847397 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0107f0 con 0x5597aaabc530 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 -- 172.21.15.31:0/1202847397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 msgr2=0x5597aaabcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 0x5597aaabcd60 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f056c011fb0 tx=0x7f056c00f6a0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 -- 172.21.15.31:0/1202847397 shutdown_connections 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaabd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabbbe0 0x5597aaabbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1202847397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabc530 0x5597aaabcd60 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 -- 172.21.15.31:0/1202847397 >> 172.21.15.31:0/1202847397 conn(0x5597aaab13e0 msgr2=0x5597aaab6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 -- 172.21.15.31:0/1202847397 shutdown_connections 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.046+0000 7f0591f98b80 1 -- 172.21.15.31:0/1202847397 wait complete. 2023-07-29T16:29:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.047+0000 7f0591f98b80 1 -- start start 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.047+0000 7f0591f98b80 1 Processor -- start 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaac9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabc530 0x5597aaaca220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaad1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aa9a9510 con 0x5597aaabc530 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aa9a91b0 con 0x5597aaabd2f0 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aa890920 con 0x5597aaabbbe0 2023-07-29T16:29:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0591f98b80 10 client.0 _ll_register_callbacks cb 0x5597aa9b2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaad1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaad1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41740/0 (socket says 172.21.15.31:41740) 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0577fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaac9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 -- 172.21.15.31:0/3066628375 learned_addr learned my addr 172.21.15.31:0/3066628375 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f05777fe700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabc530 0x5597aaaca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 -- 172.21.15.31:0/3066628375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 msgr2=0x5597aaac9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaac9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 -- 172.21.15.31:0/3066628375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabc530 msgr2=0x5597aaaca220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabc530 0x5597aaaca220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f057cb9c700 1 -- 172.21.15.31:0/3066628375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0568009d60 con 0x5597aaabd2f0 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.048+0000 7f0577fff700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaac9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f057cb9c700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaad1260 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f05700caa50 tx=0x7f05700097e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60fb9b0cc913e2d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f05577fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:02.057 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f05577fe700 1 -- 172.21.15.31:0/3066628375 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057007e500 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f05577fe700 1 -- 172.21.15.31:0/3066628375 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0570004dc0 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597aaa37d10 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f05577fe700 1 -- 172.21.15.31:0/3066628375 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f057009a3d0 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f0591f98b80 10 client.29474 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f057819e6b0 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.049+0000 7f05577fe700 1 -- 172.21.15.31:0/3066628375 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5597aaa37d10 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f05577fe700 1 -- 172.21.15.31:0/3066628375 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f05700a0170 con 0x5597aaabd2f0 2023-07-29T16:29:02.058 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0591f98b80 10 client.29474 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0591f98b80 10 client.29474 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0591f98b80 20 client.29474 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0591f98b80 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aaad5770 0x5597aaad7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0591f98b80 4 client.29474 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597aaa8ad70 con 0x5597aaad5770 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0577fff700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aaad5770 0x5597aaad7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f0577fff700 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aaad5770 0x5597aaad7b80 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3308b84536d18c3d server_cookie=8819f838a18900fb in_seq=0 out_seq=0 2023-07-29T16:29:02.059 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.050+0000 7f05577fe700 10 client.29474 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f05577fe700 1 -- 172.21.15.31:0/3066628375 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5597aaa8ad70 con 0x5597aaad5770 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f05577fe700 10 client.29474 handle_command_reply: tid=0 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f0591f98b80 1 client.29474 shutdown 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 msgr2=0x5597aaad1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f0591f98b80 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaad1260 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f05700caa50 tx=0x7f05700097e0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 shutdown_connections 2023-07-29T16:29:02.060 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.051+0000 7f0591f98b80 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aaad5770 0x5597aaad7b80 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabd2f0 0x5597aaad1260 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaac9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 1 --2- 172.21.15.31:0/3066628375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabc530 0x5597aaaca220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f05577fe700 0 client.29474 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 >> 172.21.15.31:0/3066628375 conn(0x5597aaab13e0 msgr2=0x5597aaab6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 shutdown_connections 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 1 -- 172.21.15.31:0/3066628375 wait complete. 2023-07-29T16:29:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.052+0000 7f0591f98b80 20 client.29474 trim_cache size 0 max 16384 2023-07-29T16:29:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 Processor -- start 2023-07-29T16:29:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 -- start start 2023-07-29T16:29:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaabd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaad1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaab6060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaab0ea0 con 0x5597aaabd2f0 2023-07-29T16:29:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaac4d80 con 0x5597aaabc530 2023-07-29T16:29:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0591f98b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaab3460 con 0x5597aaabbbe0 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f05777fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaad1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f05777fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaad1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41754/0 (socket says 172.21.15.31:41754) 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f05777fe700 1 -- 172.21.15.31:0/2169444560 learned_addr learned my addr 172.21.15.31:0/2169444560 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f0577fff700 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaabd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.071+0000 7f057cb9c700 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaab6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f05777fe700 1 -- 172.21.15.31:0/2169444560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 msgr2=0x5597aaabd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f05777fe700 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaabd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f05777fe700 1 -- 172.21.15.31:0/2169444560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 msgr2=0x5597aaab6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f05777fe700 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaab6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f05777fe700 1 -- 172.21.15.31:0/2169444560 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c019040 con 0x5597aaabc530 2023-07-29T16:29:02.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f0577fff700 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaabd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f05777fe700 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaad1220 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f056c011f80 tx=0x7f056c011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb70c1c6e1fea12 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f055effd700 1 -- 172.21.15.31:0/2169444560 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c005ce0 con 0x5597aaabc530 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f055effd700 1 -- 172.21.15.31:0/2169444560 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f056c011320 con 0x5597aaabc530 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.072+0000 7f055effd700 1 -- 172.21.15.31:0/2169444560 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c00e990 con 0x5597aaabc530 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 -- 172.21.15.31:0/2169444560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 msgr2=0x5597aaad1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaad1220 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f056c011f80 tx=0x7f056c011900 comp rx=0 tx=0).stop 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 -- 172.21.15.31:0/2169444560 shutdown_connections 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaab6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaabd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 --2- 172.21.15.31:0/2169444560 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaad1220 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 -- 172.21.15.31:0/2169444560 >> 172.21.15.31:0/2169444560 conn(0x5597aaab13e0 msgr2=0x5597aaab6900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 -- 172.21.15.31:0/2169444560 shutdown_connections 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 -- 172.21.15.31:0/2169444560 wait complete. 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.073+0000 7f0591f98b80 1 -- start start 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 Processor -- start 2023-07-29T16:29:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaacf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaacf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaad0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaad0c90 con 0x5597aaabd2f0 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaad0e00 con 0x5597aaabc530 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597aaad0fc0 con 0x5597aaabbbe0 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0591f98b80 10 client.0 _ll_register_callbacks cb 0x5597aa9b2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f05777fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaacf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f0577fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaacf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f05777fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaacf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41756/0 (socket says 172.21.15.31:41756) 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.074+0000 7f05777fe700 1 -- 172.21.15.31:0/1022730780 learned_addr learned my addr 172.21.15.31:0/1022730780 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f057cb9c700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaad0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05777fe700 1 -- 172.21.15.31:0/1022730780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 msgr2=0x5597aaacf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05777fe700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaacf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05777fe700 1 -- 172.21.15.31:0/1022730780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 msgr2=0x5597aaad0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05777fe700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaad0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05777fe700 1 -- 172.21.15.31:0/1022730780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0568012da0 con 0x5597aaabc530 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f057cb9c700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaad0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f0577fff700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaacf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05777fe700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaacf740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f056c013ff0 tx=0x7f056c009db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3ff406025d6681 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05577fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05577fe700 1 -- 172.21.15.31:0/1022730780 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c005ce0 con 0x5597aaabc530 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f05577fe700 1 -- 172.21.15.31:0/1022730780 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f056c011a90 con 0x5597aaabc530 2023-07-29T16:29:02.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.075+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f056c0111b0 con 0x5597aaabc530 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f05577fe700 1 -- 172.21.15.31:0/1022730780 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f056c010e60 con 0x5597aaabc530 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 10 client.29480 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f056c02a040 con 0x5597aaabc530 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f05577fe700 1 -- 172.21.15.31:0/1022730780 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f056c0111b0 con 0x5597aaabc530 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f05577fe700 1 -- 172.21.15.31:0/1022730780 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f056c02d040 con 0x5597aaabc530 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 10 client.29480 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:02.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 10 client.29480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 20 client.29480 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aab28020 0x5597aab2a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 4 client.29480 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:29:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5597aaa05d40 con 0x5597aab28020 2023-07-29T16:29:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.076+0000 7f0577fff700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aab28020 0x5597aab2a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0577fff700 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aab28020 0x5597aab2a430 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12be8fe2eeb7157f server_cookie=6a8681f11f61982d in_seq=0 out_seq=0 2023-07-29T16:29:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f05577fe700 10 client.29480 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f05577fe700 1 -- 172.21.15.31:0/1022730780 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5597aaa05d40 con 0x5597aab28020 2023-07-29T16:29:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f05577fe700 10 client.29480 handle_command_reply: tid=0 2023-07-29T16:29:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 client.29480 shutdown 2023-07-29T16:29:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 msgr2=0x5597aaacf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaacf740 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f056c013ff0 tx=0x7f056c009db0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 shutdown_connections 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5597aab28020 0x5597aab2a430 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597aaabd2f0 0x5597aaad0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f05577fe700 0 client.29480 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5597aaabbbe0 0x5597aaacf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.077+0000 7f0591f98b80 1 --2- 172.21.15.31:0/1022730780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597aaabc530 0x5597aaacf740 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 >> 172.21.15.31:0/1022730780 conn(0x5597aaab13e0 msgr2=0x5597aaac37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 shutdown_connections 2023-07-29T16:29:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f0591f98b80 1 -- 172.21.15.31:0/1022730780 wait complete. 2023-07-29T16:29:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f0591f98b80 20 client.29480 trim_cache size 0 max 16384 2023-07-29T16:29:02.086 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:29:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:29:02.086 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:29:02.086 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:29:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05600a0810 msgr2=0x7f05600a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05600a0810 0x7f05600a2cc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f056c00f9f0 tx=0x7f056c018000 comp rx=0 tx=0).stop 2023-07-29T16:29:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 msgr2=0x7f057819d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f057819d770 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0570009fd0 tx=0x7f0570007930 comp rx=0 tx=0).stop 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 shutdown_connections 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05600a0810 0x7f05600a2cc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0578105fe0 0x7f057819d770 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0578100da0 0x7f05781961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 --2- 172.21.15.31:0/193470541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05781036c0 0x7f0578196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.078+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 >> 172.21.15.31:0/193470541 conn(0x7f05780fa840 msgr2=0x7f0578101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.079+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 shutdown_connections 2023-07-29T16:29:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.079+0000 7f057e5ff700 1 -- 172.21.15.31:0/193470541 wait complete. 2023-07-29T16:29:02.096 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:29:02.097 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:29:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 -- 172.21.15.31:0/2727370785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a00737b0 msgr2=0x7f48a0073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2727370785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a00737b0 0x7f48a0073c20 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4890009990 tx=0x7f4890009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 -- 172.21.15.31:0/2727370785 shutdown_connections 2023-07-29T16:29:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2727370785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a0108200 0x7f48a010a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2727370785 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a0074d90 0x7f48a00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2727370785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a00737b0 0x7f48a0073c20 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 -- 172.21.15.31:0/2727370785 >> 172.21.15.31:0/2727370785 conn(0x7f48a00fb920 msgr2=0x7f48a00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 -- 172.21.15.31:0/2727370785 shutdown_connections 2023-07-29T16:29:02.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.315+0000 7f48a73ff700 1 -- 172.21.15.31:0/2727370785 wait complete. 2023-07-29T16:29:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.316+0000 7f48a73ff700 1 Processor -- start 2023-07-29T16:29:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 -- start start 2023-07-29T16:29:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 0x7f48a0198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a0074d90 0x7f48a01987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a0108200 0x7f48a019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a019fde0 con 0x7f48a0074d90 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a019ff20 con 0x7f48a0108200 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a73ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a01a00b0 con 0x7f48a00737b0 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a519b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 0x7f48a0198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a519b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 0x7f48a0198260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48344/0 (socket says 172.21.15.31:48344) 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.317+0000 7f48a519b700 1 -- 172.21.15.31:0/2553771406 learned_addr learned my addr 172.21.15.31:0/2553771406 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a599c700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a0108200 0x7f48a019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a499a700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a0074d90 0x7f48a01987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a519b700 1 -- 172.21.15.31:0/2553771406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a0108200 msgr2=0x7f48a019f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a519b700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a0108200 0x7f48a019f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a519b700 1 -- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a0074d90 msgr2=0x7f48a01987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a519b700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a0074d90 0x7f48a01987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a519b700 1 -- 172.21.15.31:0/2553771406 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48900095f0 con 0x7f48a00737b0 2023-07-29T16:29:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48a519b700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 0x7f48a0198260 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f489c00e970 tx=0x7f489c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.318+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c009790 con 0x7f48a00737b0 2023-07-29T16:29:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.319+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f489c00eec0 con 0x7f48a00737b0 2023-07-29T16:29:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.319+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a01a0390 con 0x7f48a00737b0 2023-07-29T16:29:02.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.319+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489c005590 con 0x7f48a00737b0 2023-07-29T16:29:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.319+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a01a08f0 con 0x7f48a00737b0 2023-07-29T16:29:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.320+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a01068b0 con 0x7f48a00737b0 2023-07-29T16:29:02.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.322+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f489c00cbd0 con 0x7f48a00737b0 2023-07-29T16:29:02.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.323+0000 7f48967fc700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f488c0a0860 0x7f488c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.323+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f489c014070 con 0x7f48a00737b0 2023-07-29T16:29:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.323+0000 7f48a499a700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f488c0a0860 0x7f488c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.324+0000 7f48a499a700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f488c0a0860 0x7f488c0a2d10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f489000b2e0 tx=0x7f4890005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.325+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f489c095990 con 0x7f48a00737b0 2023-07-29T16:29:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.631+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a002d0c0 con 0x7f48a00737b0 2023-07-29T16:29:02.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.632+0000 7f48967fc700 1 -- 172.21.15.31:0/2553771406 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+10781 (secure 0 0 0) 0x7f489c004ef0 con 0x7f48a00737b0 2023-07-29T16:29:02.632 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:29:02.632 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:27:32.973421+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:29:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f488c0a0860 msgr2=0x7f488c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f488c0a0860 0x7f488c0a2d10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f489000b2e0 tx=0x7f4890005fb0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 msgr2=0x7f48a0198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 0x7f48a0198260 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f489c00e970 tx=0x7f489c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 shutdown_connections 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f488c0a0860 0x7f488c0a2d10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48a0108200 0x7f48a019f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f48a00737b0 0x7f48a0198260 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 --2- 172.21.15.31:0/2553771406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f48a0074d90 0x7f48a01987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 >> 172.21.15.31:0/2553771406 conn(0x7f48a00fb920 msgr2=0x7f48a00fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 shutdown_connections 2023-07-29T16:29:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.635+0000 7f48a73ff700 1 -- 172.21.15.31:0/2553771406 wait complete. 2023-07-29T16:29:02.636 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 33 2023-07-29T16:29:02.654 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:29:02.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 -- 172.21.15.31:0/384300651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 msgr2=0x7fd4600ff630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 --2- 172.21.15.31:0/384300651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 0x7fd4600ff630 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd450009990 tx=0x7fd450009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 -- 172.21.15.31:0/384300651 shutdown_connections 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 --2- 172.21.15.31:0/384300651 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100da0 0x7fd460107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 --2- 172.21.15.31:0/384300651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 0x7fd4600ff630 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 --2- 172.21.15.31:0/384300651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd460100410 0x7fd460100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 -- 172.21.15.31:0/384300651 >> 172.21.15.31:0/384300651 conn(0x7fd4600fa840 msgr2=0x7fd4600fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 -- 172.21.15.31:0/384300651 shutdown_connections 2023-07-29T16:29:02.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.871+0000 7fd464bff700 1 -- 172.21.15.31:0/384300651 wait complete. 2023-07-29T16:29:02.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 Processor -- start 2023-07-29T16:29:02.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 -- start start 2023-07-29T16:29:02.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 0x7fd4601982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 0x7fd4601987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd460100da0 0x7fd46019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46019fe30 con 0x7fd460100410 2023-07-29T16:29:02.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46019ff70 con 0x7fd460100da0 2023-07-29T16:29:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd464bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4601a0100 con 0x7fd4600ff220 2023-07-29T16:29:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.873+0000 7fd45dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 0x7fd4601987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 0x7fd4601987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51792/0 (socket says 172.21.15.31:51792) 2023-07-29T16:29:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 -- 172.21.15.31:0/994307908 learned_addr learned my addr 172.21.15.31:0/994307908 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45e59c700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 0x7fd4601982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd460100da0 0x7fd46019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 -- 172.21.15.31:0/994307908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 msgr2=0x7fd4601982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 0x7fd4601982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 -- 172.21.15.31:0/994307908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd460100da0 msgr2=0x7fd46019f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd460100da0 0x7fd46019f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 -- 172.21.15.31:0/994307908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4500095f0 con 0x7fd460100410 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 0x7fd4601987f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd45400e9a0 tx=0x7fd45400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.874+0000 7fd44f7fe700 1 -- 172.21.15.31:0/994307908 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454009790 con 0x7fd460100410 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.875+0000 7fd44f7fe700 1 -- 172.21.15.31:0/994307908 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd454010430 con 0x7fd460100410 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.875+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4601a03e0 con 0x7fd460100410 2023-07-29T16:29:02.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.875+0000 7fd44f7fe700 1 -- 172.21.15.31:0/994307908 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454018680 con 0x7fd460100410 2023-07-29T16:29:02.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.875+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4601a08f0 con 0x7fd460100410 2023-07-29T16:29:02.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.878+0000 7fd44f7fe700 1 -- 172.21.15.31:0/994307908 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd4540105a0 con 0x7fd460100410 2023-07-29T16:29:02.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.879+0000 7fd47842bb80 1 Processor -- start 2023-07-29T16:29:02.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.879+0000 7fd44f7fe700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd4480a0790 0x7fd4480a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.879+0000 7fd47842bb80 1 -- start start 2023-07-29T16:29:02.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.879+0000 7fd44f7fe700 1 -- 172.21.15.31:0/994307908 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd454014070 con 0x7fd460100410 2023-07-29T16:29:02.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991df6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991df8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd47842bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991d4ea10 con 0x562991df7530 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd47842bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991d4c3d0 con 0x562991df82f0 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd47842bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991d47cc0 con 0x562991df6be0 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd4480a0790 0x7fd4480a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991df6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991df6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48364/0 (socket says 172.21.15.31:48364) 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 -- 172.21.15.31:0/3901929461 learned_addr learned my addr 172.21.15.31:0/3901929461 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 -- 172.21.15.31:0/3901929461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 msgr2=0x562991df8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991df8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 -- 172.21.15.31:0/3901929461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 msgr2=0x562991df7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.880+0000 7fd45e59c700 1 -- 172.21.15.31:0/3901929461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562991b75de0 con 0x562991df6be0 2023-07-29T16:29:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.881+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd4480a0790 0x7fd4480a2c40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd45800f540 tx=0x7fd458009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.881+0000 7fd45e59c700 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991df6ff0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd460100270 tx=0x7fd450005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1302837bc802caff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.881+0000 7fd44cff9700 1 -- 172.21.15.31:0/3901929461 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45001d070 con 0x562991df6be0 2023-07-29T16:29:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.881+0000 7fd44cff9700 1 -- 172.21.15.31:0/3901929461 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd45001e040 con 0x562991df6be0 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.881+0000 7fd44cff9700 1 -- 172.21.15.31:0/3901929461 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4500048f0 con 0x562991df6be0 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 -- 172.21.15.31:0/3901929461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 msgr2=0x562991df6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991df6ff0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd460100270 tx=0x7fd450005ea0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 -- 172.21.15.31:0/3901929461 shutdown_connections 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991df8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991df6ff0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3901929461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 -- 172.21.15.31:0/3901929461 >> 172.21.15.31:0/3901929461 conn(0x562991dec3e0 msgr2=0x562991df1990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 -- 172.21.15.31:0/3901929461 shutdown_connections 2023-07-29T16:29:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.882+0000 7fd47842bb80 1 -- 172.21.15.31:0/3901929461 wait complete. 2023-07-29T16:29:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 -- start start 2023-07-29T16:29:02.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 Processor -- start 2023-07-29T16:29:02.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e00730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e00c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991e07cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991ce4510 con 0x562991df7530 2023-07-29T16:29:02.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991ce41b0 con 0x562991df82f0 2023-07-29T16:29:02.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991bcb920 con 0x562991df6be0 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.883+0000 7fd47842bb80 10 client.0 _ll_register_callbacks cb 0x562991ced620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e00730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e00730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48380/0 (socket says 172.21.15.31:48380) 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 -- 172.21.15.31:0/3208197713 learned_addr learned my addr 172.21.15.31:0/3208197713 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991e07cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e00c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 -- 172.21.15.31:0/3208197713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 msgr2=0x562991e07cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991e07cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 -- 172.21.15.31:0/3208197713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 msgr2=0x562991e00c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e00c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 -- 172.21.15.31:0/3208197713 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd460107f90 con 0x562991df6be0 2023-07-29T16:29:02.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd45e59c700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e00730 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd450009350 tx=0x7fd4500226b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1640a527836fd17 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.884+0000 7fd445ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:02.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd445ffb700 1 -- 172.21.15.31:0/3208197713 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45001d030 con 0x562991df6be0 2023-07-29T16:29:02.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd445ffb700 1 -- 172.21.15.31:0/3208197713 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd450022cb0 con 0x562991df6be0 2023-07-29T16:29:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562991d72d10 con 0x562991df6be0 2023-07-29T16:29:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd445ffb700 1 -- 172.21.15.31:0/3208197713 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd450022e70 con 0x562991df6be0 2023-07-29T16:29:02.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd47842bb80 10 client.29491 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562991b5ec60 con 0x562991df6be0 2023-07-29T16:29:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.885+0000 7fd445ffb700 1 -- 172.21.15.31:0/3208197713 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562991d72d10 con 0x562991df6be0 2023-07-29T16:29:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd445ffb700 1 -- 172.21.15.31:0/3208197713 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fd45002c5c0 con 0x562991df6be0 2023-07-29T16:29:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd47842bb80 10 client.29491 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd47842bb80 10 client.29491 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd47842bb80 20 client.29491 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e0c050 0x562991e0e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd47842bb80 4 client.29491 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562991dc5d70 con 0x562991e0c050 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e0c050 0x562991e0e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e0c050 0x562991e0e460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cdad27268f196817 server_cookie=66cfe79b1b9ef582 in_seq=0 out_seq=0 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.886+0000 7fd445ffb700 10 client.29491 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:02.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd445ffb700 1 -- 172.21.15.31:0/3208197713 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562991dc5d70 con 0x562991e0c050 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd445ffb700 10 client.29491 handle_command_reply: tid=0 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 client.29491 shutdown 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 msgr2=0x562991e00730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e00730 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd450009350 tx=0x7fd4500226b0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 shutdown_connections 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e0c050 0x562991e0e460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991e07cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e00730 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd445ffb700 0 client.29491 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 --2- 172.21.15.31:0/3208197713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e00c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.887+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 >> 172.21.15.31:0/3208197713 conn(0x562991dec3e0 msgr2=0x562991df1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.888+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 shutdown_connections 2023-07-29T16:29:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.888+0000 7fd47842bb80 1 -- 172.21.15.31:0/3208197713 wait complete. 2023-07-29T16:29:02.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.888+0000 7fd47842bb80 20 client.29491 trim_cache size 0 max 16384 2023-07-29T16:29:02.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.901+0000 7fd47842bb80 1 Processor -- start 2023-07-29T16:29:02.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.901+0000 7fd47842bb80 1 -- start start 2023-07-29T16:29:02.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e05c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991dfe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd47842bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991dffc10 con 0x562991df7530 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd47842bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991dee490 con 0x562991df82f0 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd47842bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991e01930 con 0x562991df6be0 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e05c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e05c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48394/0 (socket says 172.21.15.31:48394) 2023-07-29T16:29:02.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45e59c700 1 -- 172.21.15.31:0/564044970 learned_addr learned my addr 172.21.15.31:0/564044970 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991dfe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45dd9b700 1 -- 172.21.15.31:0/564044970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 msgr2=0x562991e05c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e05c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45dd9b700 1 -- 172.21.15.31:0/564044970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 msgr2=0x562991dfe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991dfe660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.902+0000 7fd45dd9b700 1 -- 172.21.15.31:0/564044970 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562991b9c250 con 0x562991df7530 2023-07-29T16:29:02.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.903+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df86c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd45400e970 tx=0x7fd4540c96f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab742cb002aa084 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.903+0000 7fd44d7fa700 1 -- 172.21.15.31:0/564044970 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4540b5070 con 0x562991df7530 2023-07-29T16:29:02.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.903+0000 7fd44d7fa700 1 -- 172.21.15.31:0/564044970 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd454099480 con 0x562991df7530 2023-07-29T16:29:02.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.903+0000 7fd44d7fa700 1 -- 172.21.15.31:0/564044970 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454010880 con 0x562991df7530 2023-07-29T16:29:02.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1066359 8.89 MB/sec execute 527 sec latency 30.842 ms 2023-07-29T16:29:02.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.903+0000 7fd47842bb80 1 -- 172.21.15.31:0/564044970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 msgr2=0x562991df86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.903+0000 7fd47842bb80 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df86c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd45400e970 tx=0x7fd4540c96f0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 -- 172.21.15.31:0/564044970 shutdown_connections 2023-07-29T16:29:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df82f0 0x562991dfe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df6be0 0x562991e05c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 --2- 172.21.15.31:0/564044970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991df86c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 -- 172.21.15.31:0/564044970 >> 172.21.15.31:0/564044970 conn(0x562991dec3e0 msgr2=0x562991dff480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 -- 172.21.15.31:0/564044970 shutdown_connections 2023-07-29T16:29:02.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 -- 172.21.15.31:0/564044970 wait complete. 2023-07-29T16:29:02.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 -- start start 2023-07-29T16:29:02.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.904+0000 7fd47842bb80 1 Processor -- start 2023-07-29T16:29:02.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 0x562991e05920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df82f0 0x562991e04600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991e04b40 con 0x562991df7530 2023-07-29T16:29:02.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991e04cb0 con 0x562991df6be0 2023-07-29T16:29:02.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562991e04e70 con 0x562991df82f0 2023-07-29T16:29:02.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd47842bb80 10 client.0 _ll_register_callbacks cb 0x562991ced620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:02.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 0x562991e05920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 0x562991e05920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41836/0 (socket says 172.21.15.31:41836) 2023-07-29T16:29:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 -- 172.21.15.31:0/1146181882 learned_addr learned my addr 172.21.15.31:0/1146181882 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df82f0 0x562991e04600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 -- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df82f0 msgr2=0x562991e04600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df82f0 0x562991e04600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 -- 172.21.15.31:0/1146181882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 msgr2=0x562991e040c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.905+0000 7fd45e59c700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e040c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd45e59c700 1 -- 172.21.15.31:0/1146181882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4601a08f0 con 0x562991df6be0 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd45ed9d700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df82f0 0x562991e04600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd45e59c700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 0x562991e05920 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd45002c970 tx=0x7fd45002c610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe365bd14cd0bf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd445ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd445ffb700 1 -- 172.21.15.31:0/1146181882 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45001d070 con 0x562991df6be0 2023-07-29T16:29:02.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd45001e040 con 0x562991df6be0 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd445ffb700 1 -- 172.21.15.31:0/1146181882 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd45002cc60 con 0x562991df6be0 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd445ffb700 1 -- 172.21.15.31:0/1146181882 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd45002cea0 con 0x562991df6be0 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd47842bb80 10 client.29501 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.906+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x562991b742b0 con 0x562991df6be0 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd445ffb700 1 -- 172.21.15.31:0/1146181882 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd45001e040 con 0x562991df6be0 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd445ffb700 1 -- 172.21.15.31:0/1146181882 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fd450004730 con 0x562991df6be0 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd47842bb80 10 client.29501 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:02.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd47842bb80 10 client.29501 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd47842bb80 20 client.29501 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd47842bb80 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e5ee00 0x562991e61210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd47842bb80 4 client.29501 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562991d40d40 con 0x562991e5ee00 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.907+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e5ee00 0x562991e61210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd445ffb700 10 client.29501 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:02.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd45dd9b700 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e5ee00 0x562991e61210 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ed51f8798bf2d99 server_cookie=c51f1c44e67893fc in_seq=0 out_seq=0 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd445ffb700 1 -- 172.21.15.31:0/1146181882 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562991d40d40 con 0x562991e5ee00 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd445ffb700 10 client.29501 handle_command_reply: tid=0 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 client.29501 shutdown 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 msgr2=0x562991e05920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 0x562991e05920 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fd45002c970 tx=0x7fd45002c610 comp rx=0 tx=0).stop 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 shutdown_connections 2023-07-29T16:29:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562991df82f0 0x562991e04600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x562991e5ee00 0x562991e61210 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562991df6be0 0x562991e05920 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 --2- 172.21.15.31:0/1146181882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562991df7530 0x562991e040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd445ffb700 0 client.29501 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 >> 172.21.15.31:0/1146181882 conn(0x562991dec3e0 msgr2=0x562991dfe660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 shutdown_connections 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 1 -- 172.21.15.31:0/1146181882 wait complete. 2023-07-29T16:29:02.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.908+0000 7fd47842bb80 20 client.29501 trim_cache size 0 max 16384 2023-07-29T16:29:02.919 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:29:02.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd4480a0790 msgr2=0x7fd4480a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd4480a0790 0x7fd4480a2c40 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd45800f540 tx=0x7fd458009920 comp rx=0 tx=0).stop 2023-07-29T16:29:02.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 msgr2=0x7fd4601987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:02.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 0x7fd4601987f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd45400e9a0 tx=0x7fd45400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 shutdown_connections 2023-07-29T16:29:02.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd4480a0790 0x7fd4480a2c40 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd460100da0 0x7fd46019f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4600ff220 0x7fd4601982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 --2- 172.21.15.31:0/994307908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd460100410 0x7fd4601987f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:02.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 >> 172.21.15.31:0/994307908 conn(0x7fd4600fa840 msgr2=0x7fd460067e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:02.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 shutdown_connections 2023-07-29T16:29:02.921 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:02.909+0000 7fd464bff700 1 -- 172.21.15.31:0/994307908 wait complete. 2023-07-29T16:29:02.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:02 smithi031 ceph-mon[95504]: pgmap v778: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-07-29T16:29:02.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:29:03.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:02 smithi039 ceph-mon[128858]: pgmap v778: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-07-29T16:29:03.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:02 smithi156 ceph-mon[129245]: pgmap v778: 97 pgs: 97 active+clean; 192 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 2.0 MiB/s wr, 220 op/s 2023-07-29T16:29:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 -- 172.21.15.31:0/316514522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8101520 msgr2=0x7f75c8101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 --2- 172.21.15.31:0/316514522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8101520 0x7f75c8101970 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009990 tx=0x7f75b8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 -- 172.21.15.31:0/316514522 shutdown_connections 2023-07-29T16:29:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 --2- 172.21.15.31:0/316514522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75c8101eb0 0x7f75c8108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 --2- 172.21.15.31:0/316514522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75c8100330 0x7f75c8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 --2- 172.21.15.31:0/316514522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8101520 0x7f75c8101970 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 -- 172.21.15.31:0/316514522 >> 172.21.15.31:0/316514522 conn(0x7f75c80fb970 msgr2=0x7f75c80fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.145 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 -- 172.21.15.31:0/316514522 shutdown_connections 2023-07-29T16:29:03.146 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.145+0000 7f75cfdff700 1 -- 172.21.15.31:0/316514522 wait complete. 2023-07-29T16:29:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.146+0000 7f75cfdff700 1 Processor -- start 2023-07-29T16:29:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 -- start start 2023-07-29T16:29:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 0x7f75c81960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75c8101520 0x7f75c8196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75c8101eb0 0x7f75c819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.147 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c819dc50 con 0x7f75c8101520 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c819dd90 con 0x7f75c8100330 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cfdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c819df20 con 0x7f75c8101eb0 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 0x7f75c81960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 0x7f75c81960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41844/0 (socket says 172.21.15.31:41844) 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1282456452 learned_addr learned my addr 172.21.15.31:0/1282456452 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75cd39a700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75c8101520 0x7f75c8196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.147+0000 7f75ce39c700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75c8101eb0 0x7f75c819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1282456452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75c8101eb0 msgr2=0x7f75c819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75c8101eb0 0x7f75c819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75c8101520 msgr2=0x7f75c8196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75c8101520 0x7f75c8196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1282456452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b80095f0 con 0x7f75c8100330 2023-07-29T16:29:03.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 0x7f75c81960d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f75c400efc0 tx=0x7f75c400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75beffd700 1 -- 172.21.15.31:0/1282456452 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4018070 con 0x7f75c8100330 2023-07-29T16:29:03.150 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75beffd700 1 -- 172.21.15.31:0/1282456452 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75c400f040 con 0x7f75c8100330 2023-07-29T16:29:03.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75beffd700 1 -- 172.21.15.31:0/1282456452 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4004280 con 0x7f75c8100330 2023-07-29T16:29:03.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.148+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c819e1a0 con 0x7f75c8100330 2023-07-29T16:29:03.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.149+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c819e660 con 0x7f75c8100330 2023-07-29T16:29:03.151 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.152+0000 7f75beffd700 1 -- 172.21.15.31:0/1282456452 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f75c4003a00 con 0x7f75c8100330 2023-07-29T16:29:03.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 Processor -- start 2023-07-29T16:29:03.152 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 -- start start 2023-07-29T16:29:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52e9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe523fa10 con 0x563fe52e7be0 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe523d3d0 con 0x563fe52e92f0 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.153+0000 7f75e378db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe5238cc0 con 0x563fe52e8530 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75beffd700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75b40a0810 0x7f75b40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52e8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48422/0 (socket says 172.21.15.31:48422) 2023-07-29T16:29:03.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cdb9b700 1 -- 172.21.15.31:0/2177745445 learned_addr learned my addr 172.21.15.31:0/2177745445 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52e9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75ce39c700 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75beffd700 1 -- 172.21.15.31:0/1282456452 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f75c40cb930 con 0x7f75c8100330 2023-07-29T16:29:03.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75ce39c700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75b40a0810 0x7f75b40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cd39a700 1 -- 172.21.15.31:0/2177745445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 msgr2=0x563fe52e8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52e8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cd39a700 1 -- 172.21.15.31:0/2177745445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 msgr2=0x563fe52e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52e7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.156 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.154+0000 7f75cd39a700 1 -- 172.21.15.31:0/2177745445 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563fe5066de0 con 0x563fe52e92f0 2023-07-29T16:29:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.155+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52e9740 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000c00 tx=0x7f75b8005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ce82c8fe9ec59fe server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.155+0000 7f75b3fff700 1 -- 172.21.15.31:0/2177745445 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b801d070 con 0x563fe52e92f0 2023-07-29T16:29:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.155+0000 7f75b3fff700 1 -- 172.21.15.31:0/2177745445 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75b801e040 con 0x563fe52e92f0 2023-07-29T16:29:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.155+0000 7f75ce39c700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75b40a0810 0x7f75b40a2cc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005950 tx=0x7f75c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.155+0000 7f75b3fff700 1 -- 172.21.15.31:0/2177745445 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8004860 con 0x563fe52e92f0 2023-07-29T16:29:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 -- 172.21.15.31:0/2177745445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 msgr2=0x563fe52e9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52e9740 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f75b8000c00 tx=0x7f75b8005ea0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 -- 172.21.15.31:0/2177745445 shutdown_connections 2023-07-29T16:29:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52e9740 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 --2- 172.21.15.31:0/2177745445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52e8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 -- 172.21.15.31:0/2177745445 >> 172.21.15.31:0/2177745445 conn(0x563fe52dd3e0 msgr2=0x563fe52e2990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 -- 172.21.15.31:0/2177745445 shutdown_connections 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 -- 172.21.15.31:0/2177745445 wait complete. 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.156+0000 7f75e378db80 1 -- start start 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 Processor -- start 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52f1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe51d5510 con 0x563fe52e7be0 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe51d51b0 con 0x563fe52e92f0 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe50bc920 con 0x563fe52e8530 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.157+0000 7f75e378db80 10 client.0 _ll_register_callbacks cb 0x563fe51de620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52f1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52f1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48432/0 (socket says 172.21.15.31:48432) 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 -- 172.21.15.31:0/775617506 learned_addr learned my addr 172.21.15.31:0/775617506 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75ce39c700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 -- 172.21.15.31:0/775617506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 msgr2=0x563fe52f8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 -- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 msgr2=0x563fe52f1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52f1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 -- 172.21.15.31:0/775617506 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81090a0 con 0x563fe52e8530 2023-07-29T16:29:03.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75cd39a700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52f1cd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009350 tx=0x7f75b800ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8200d8f2037240a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.158+0000 7f75b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75b17fa700 1 -- 172.21.15.31:0/775617506 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b801d030 con 0x563fe52e8530 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75b17fa700 1 -- 172.21.15.31:0/775617506 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f75b800bd00 con 0x563fe52e8530 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563fe5263d10 con 0x563fe52e8530 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75b17fa700 1 -- 172.21.15.31:0/775617506 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75b8005950 con 0x563fe52e8530 2023-07-29T16:29:03.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75e378db80 10 client.29509 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563fe504fc60 con 0x563fe52e8530 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75b17fa700 1 -- 172.21.15.31:0/775617506 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563fe5263d10 con 0x563fe52e8530 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75b17fa700 1 -- 172.21.15.31:0/775617506 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f75b800fbd0 con 0x563fe52e8530 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75e378db80 10 client.29509 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75e378db80 10 client.29509 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.159+0000 7f75e378db80 20 client.29509 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:03.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.160+0000 7f75e378db80 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe52fd020 0x563fe52ff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.160+0000 7f75e378db80 4 client.29509 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:03.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.160+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563fe52b6d70 con 0x563fe52fd020 2023-07-29T16:29:03.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.160+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe52fd020 0x563fe52ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.160+0000 7f75b17fa700 10 client.29509 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:03.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.160+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe52fd020 0x563fe52ff400 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39383e25d21026fa server_cookie=5472db7f2f8a641 in_seq=0 out_seq=0 2023-07-29T16:29:03.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75b17fa700 1 -- 172.21.15.31:0/775617506 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563fe52b6d70 con 0x563fe52fd020 2023-07-29T16:29:03.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75b17fa700 10 client.29509 handle_command_reply: tid=0 2023-07-29T16:29:03.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 client.29509 shutdown 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 msgr2=0x563fe52f1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52f1cd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009350 tx=0x7f75b800ba20 comp rx=0 tx=0).stop 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 shutdown_connections 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe52fd020 0x563fe52ff400 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75b17fa700 0 client.29509 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:03.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e7be0 0x563fe52f1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 --2- 172.21.15.31:0/775617506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e8530 0x563fe52f1cd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 >> 172.21.15.31:0/775617506 conn(0x563fe52dd3e0 msgr2=0x563fe52e2060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.161+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 shutdown_connections 2023-07-29T16:29:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.162+0000 7f75e378db80 1 -- 172.21.15.31:0/775617506 wait complete. 2023-07-29T16:29:03.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.162+0000 7f75e378db80 20 client.29509 trim_cache size 0 max 16384 2023-07-29T16:29:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.175+0000 7f75e378db80 1 Processor -- start 2023-07-29T16:29:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.175+0000 7f75e378db80 1 -- start start 2023-07-29T16:29:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75e378db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75e378db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 0x563fe52e3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75e378db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e92f0 0x563fe52e2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75e378db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe52f0c40 con 0x563fe52e92f0 2023-07-29T16:29:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75e378db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe52f1300 con 0x563fe52e8530 2023-07-29T16:29:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75e378db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe52df490 con 0x563fe52e7be0 2023-07-29T16:29:03.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75cd39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 0x563fe52e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75cd39a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 0x563fe52e3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41858/0 (socket says 172.21.15.31:41858) 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75cd39a700 1 -- 172.21.15.31:0/2169017488 learned_addr learned my addr 172.21.15.31:0/2169017488 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.176+0000 7f75ce39c700 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e92f0 0x563fe52e2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75cd39a700 1 -- 172.21.15.31:0/2169017488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 msgr2=0x563fe52f6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75cd39a700 1 -- 172.21.15.31:0/2169017488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e92f0 msgr2=0x563fe52e2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e92f0 0x563fe52e2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75cd39a700 1 -- 172.21.15.31:0/2169017488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563fe508d250 con 0x563fe52e8530 2023-07-29T16:29:03.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75ce39c700 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e92f0 0x563fe52e2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75cd39a700 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 0x563fe52e3840 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f75b80098e0 tx=0x7f75b8009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4060156946bc33fb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75bcbff700 1 -- 172.21.15.31:0/2169017488 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b801d040 con 0x563fe52e8530 2023-07-29T16:29:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75bcbff700 1 -- 172.21.15.31:0/2169017488 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75b802c610 con 0x563fe52e8530 2023-07-29T16:29:03.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.177+0000 7f75bcbff700 1 -- 172.21.15.31:0/2169017488 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8004d00 con 0x563fe52e8530 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 -- 172.21.15.31:0/2169017488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 msgr2=0x563fe52e3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 0x563fe52e3840 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f75b80098e0 tx=0x7f75b8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 -- 172.21.15.31:0/2169017488 shutdown_connections 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e92f0 0x563fe52e2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 --2- 172.21.15.31:0/2169017488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e8530 0x563fe52e3840 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 -- 172.21.15.31:0/2169017488 >> 172.21.15.31:0/2169017488 conn(0x563fe52dd3e0 msgr2=0x563fe52f04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 -- 172.21.15.31:0/2169017488 shutdown_connections 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 -- 172.21.15.31:0/2169017488 wait complete. 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.178+0000 7f75e378db80 1 -- start start 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 Processor -- start 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e8530 0x563fe52f5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe52f67c0 con 0x563fe52e8530 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe52f6930 con 0x563fe52e92f0 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fe52f6af0 con 0x563fe52e7be0 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75e378db80 10 client.0 _ll_register_callbacks cb 0x563fe51de620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75cdb9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48438/0 (socket says 172.21.15.31:48438) 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1240524770 learned_addr learned my addr 172.21.15.31:0/1240524770 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75ce39c700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.179+0000 7f75cd39a700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e8530 0x563fe52f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1240524770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 msgr2=0x563fe52f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e8530 msgr2=0x563fe52f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e8530 0x563fe52f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75cdb9b700 1 -- 172.21.15.31:0/1240524770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563fe504fc60 con 0x563fe52e7be0 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75cdb9b700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f4e00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c6f0 tx=0x7f75c400cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d04ecd9c8a21f90 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75b17fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75b17fa700 1 -- 172.21.15.31:0/1240524770 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c40953a0 con 0x563fe52e7be0 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75b17fa700 1 -- 172.21.15.31:0/1240524770 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f75c4099060 con 0x563fe52e7be0 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75b801e040 con 0x563fe52e7be0 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75b17fa700 1 -- 172.21.15.31:0/1240524770 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75c409d090 con 0x563fe52e7be0 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75e378db80 10 client.29515 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.180+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x563fe50652b0 con 0x563fe52e7be0 2023-07-29T16:29:03.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75b17fa700 1 -- 172.21.15.31:0/1240524770 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75b801e040 con 0x563fe52e7be0 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75b17fa700 1 -- 172.21.15.31:0/1240524770 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f75c4098550 con 0x563fe52e7be0 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75e378db80 10 client.29515 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75e378db80 10 client.29515 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75e378db80 20 client.29515 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75e378db80 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe5354260 0x563fe5356670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75e378db80 4 client.29515 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563fe5231d40 con 0x563fe5354260 2023-07-29T16:29:03.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.181+0000 7f75cd39a700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe5354260 0x563fe5356670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75b17fa700 10 client.29515 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75cd39a700 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe5354260 0x563fe5356670 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4680c278c4bd2f3f server_cookie=6eee2c8f71a80a03 in_seq=0 out_seq=0 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75b17fa700 1 -- 172.21.15.31:0/1240524770 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563fe5231d40 con 0x563fe5354260 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75b17fa700 10 client.29515 handle_command_reply: tid=0 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 client.29515 shutdown 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 msgr2=0x563fe52f4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f4e00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f75c400c6f0 tx=0x7f75c400cc70 comp rx=0 tx=0).stop 2023-07-29T16:29:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 shutdown_connections 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563fe5354260 0x563fe5356670 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563fe52e92f0 0x563fe52f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563fe52e7be0 0x563fe52f4e00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75b17fa700 0 client.29515 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 --2- 172.21.15.31:0/1240524770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563fe52e8530 0x563fe52f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 >> 172.21.15.31:0/1240524770 conn(0x563fe52dd3e0 msgr2=0x563fe52e2060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 shutdown_connections 2023-07-29T16:29:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.182+0000 7f75e378db80 1 -- 172.21.15.31:0/1240524770 wait complete. 2023-07-29T16:29:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75e378db80 20 client.29515 trim_cache size 0 max 16384 2023-07-29T16:29:03.190 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75b40a0810 msgr2=0x7f75b40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75b40a0810 0x7f75b40a2cc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005950 tx=0x7f75c0009260 comp rx=0 tx=0).stop 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 msgr2=0x7f75c81960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 0x7f75c81960d0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f75c400efc0 tx=0x7f75c400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 shutdown_connections 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75b40a0810 0x7f75b40a2cc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75c8101eb0 0x7f75c819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75c8100330 0x7f75c81960d0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 --2- 172.21.15.31:0/1282456452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75c8101520 0x7f75c8196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.183+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 >> 172.21.15.31:0/1282456452 conn(0x7f75c80fb970 msgr2=0x7f75c80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.184+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 shutdown_connections 2023-07-29T16:29:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.184+0000 7f75cfdff700 1 -- 172.21.15.31:0/1282456452 wait complete. 2023-07-29T16:29:03.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:29:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/658874463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28101eb0 msgr2=0x7f4a28108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/658874463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28101eb0 0x7f4a28108770 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000b0a0 tx=0x7f4a2000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/658874463 shutdown_connections 2023-07-29T16:29:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/658874463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28101eb0 0x7f4a28108770 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/658874463 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a28100330 0x7f4a28100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/658874463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a28101520 0x7f4a28101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.421+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/658874463 >> 172.21.15.31:0/658874463 conn(0x7f4a280fb970 msgr2=0x7f4a280fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.422+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/658874463 shutdown_connections 2023-07-29T16:29:03.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.422+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/658874463 wait complete. 2023-07-29T16:29:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 Processor -- start 2023-07-29T16:29:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 -- start start 2023-07-29T16:29:03.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 0x7f4a281983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a28101520 0x7f4a281988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a28101eb0 0x7f4a2819f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2819ff20 con 0x7f4a28101eb0 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281a0060 con 0x7f4a28100330 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.423+0000 7f4a2d5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a281a01f0 con 0x7f4a28101520 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 0x7f4a281983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 0x7f4a281983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41878/0 (socket says 172.21.15.31:41878) 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 -- 172.21.15.31:0/2984967612 learned_addr learned my addr 172.21.15.31:0/2984967612 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a267fc700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a28101520 0x7f4a281988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a277fe700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a28101eb0 0x7f4a2819f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 -- 172.21.15.31:0/2984967612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a28101520 msgr2=0x7f4a281988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a28101520 0x7f4a281988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 -- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a28101eb0 msgr2=0x7f4a2819f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a28101eb0 0x7f4a2819f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 -- 172.21.15.31:0/2984967612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a20009d00 con 0x7f4a28100330 2023-07-29T16:29:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.424+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 0x7f4a281983a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800e970 tx=0x7f4a1800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.425+0000 7f4a0ffff700 1 -- 172.21.15.31:0/2984967612 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18009790 con 0x7f4a28100330 2023-07-29T16:29:03.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.425+0000 7f4a0ffff700 1 -- 172.21.15.31:0/2984967612 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a1800c7e0 con 0x7f4a28100330 2023-07-29T16:29:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.425+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a281a04d0 con 0x7f4a28100330 2023-07-29T16:29:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.425+0000 7f4a0ffff700 1 -- 172.21.15.31:0/2984967612 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a18003c70 con 0x7f4a28100330 2023-07-29T16:29:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.425+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a281a0a30 con 0x7f4a28100330 2023-07-29T16:29:03.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.428+0000 7f4a0ffff700 1 -- 172.21.15.31:0/2984967612 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4a1800cc50 con 0x7f4a28100330 2023-07-29T16:29:03.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.430+0000 7f4a0ffff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a100a0810 0x7f4a100a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.430+0000 7f4a0ffff700 1 -- 172.21.15.31:0/2984967612 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a18014070 con 0x7f4a28100330 2023-07-29T16:29:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.430+0000 7f4a267fc700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a100a0810 0x7f4a100a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.430+0000 7f4a40fd6b80 1 Processor -- start 2023-07-29T16:29:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 -- start start 2023-07-29T16:29:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a267fc700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a100a0810 0x7f4a100a2cc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009c60 tx=0x7f4a1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x558816426ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816427d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816428740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881637ea10 con 0x558816426be0 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881637c3d0 con 0x558816427530 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558816377cc0 con 0x5588164282f0 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816427d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816427d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41890/0 (socket says 172.21.15.31:41890) 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a267fc700 1 -- 172.21.15.31:0/1841304836 learned_addr learned my addr 172.21.15.31:0/1841304836 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.431+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x558816426ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a277fe700 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816428740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a267fc700 1 -- 172.21.15.31:0/1841304836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 msgr2=0x558816428740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a267fc700 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816428740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a267fc700 1 -- 172.21.15.31:0/1841304836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 msgr2=0x558816426ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a267fc700 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x558816426ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a267fc700 1 -- 172.21.15.31:0/1841304836 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588161a5de0 con 0x558816427530 2023-07-29T16:29:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a267fc700 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816427d60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00e4e0 tx=0x7f4a1c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48e7b983a6eb3be4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a0d7fa700 1 -- 172.21.15.31:0/1841304836 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c00ee00 con 0x558816427530 2023-07-29T16:29:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a0d7fa700 1 -- 172.21.15.31:0/1841304836 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a1c021070 con 0x558816427530 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.432+0000 7f4a0d7fa700 1 -- 172.21.15.31:0/1841304836 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c01d810 con 0x558816427530 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/1841304836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 msgr2=0x558816427d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816427d60 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00e4e0 tx=0x7f4a1c014000 comp rx=0 tx=0).stop 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/1841304836 shutdown_connections 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816428740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x558816426ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/1841304836 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816427d60 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/1841304836 >> 172.21.15.31:0/1841304836 conn(0x55881641c3e0 msgr2=0x558816421990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/1841304836 shutdown_connections 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.433+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/1841304836 wait complete. 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.434+0000 7f4a40fd6b80 1 -- start start 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.434+0000 7f4a40fd6b80 1 Processor -- start 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.434+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x5588164307f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.434+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816430d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.434+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816437d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558816314510 con 0x558816426be0 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588163141b0 con 0x558816427530 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588161fb920 con 0x5588164282f0 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a40fd6b80 10 client.0 _ll_register_callbacks cb 0x55881631d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816437d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816430d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a26ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x5588164307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816437d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48472/0 (socket says 172.21.15.31:48472) 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 -- 172.21.15.31:0/663571582 learned_addr learned my addr 172.21.15.31:0/663571582 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 -- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 msgr2=0x558816430d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816430d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 -- 172.21.15.31:0/663571582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 msgr2=0x5588164307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x5588164307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.435+0000 7f4a277fe700 1 -- 172.21.15.31:0/663571582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1c017070 con 0x5588164282f0 2023-07-29T16:29:03.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a277fe700 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816437d70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000b0a0 tx=0x7f4a200074e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd7a2fdfebd4da02 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a0aff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a0aff5700 1 -- 172.21.15.31:0/663571582 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2000e040 con 0x5588164282f0 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588163a2d10 con 0x5588164282f0 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a0aff5700 1 -- 172.21.15.31:0/663571582 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a2000bc50 con 0x5588164282f0 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a0aff5700 1 -- 172.21.15.31:0/663571582 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a20009b80 con 0x5588164282f0 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a40fd6b80 10 client.29527 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:03.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.436+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55881618ec60 con 0x5588164282f0 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a0aff5700 1 -- 172.21.15.31:0/663571582 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5588163a2d10 con 0x5588164282f0 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a0aff5700 1 -- 172.21.15.31:0/663571582 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f4a20023030 con 0x5588164282f0 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a40fd6b80 10 client.29527 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a40fd6b80 10 client.29527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a40fd6b80 20 client.29527 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881643c540 0x55881643e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a40fd6b80 4 client.29527 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588163f5d70 con 0x55881643c540 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.437+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881643c540 0x55881643e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.438+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881643c540 0x55881643e950 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d020a786df2de800 server_cookie=f0e1fefe46d3ba4a in_seq=0 out_seq=0 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.438+0000 7f4a0aff5700 10 client.29527 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a0aff5700 1 -- 172.21.15.31:0/663571582 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5588163f5d70 con 0x55881643c540 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a0aff5700 10 client.29527 handle_command_reply: tid=0 2023-07-29T16:29:03.443 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 client.29527 shutdown 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 msgr2=0x558816437d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816437d70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000b0a0 tx=0x7f4a200074e0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 shutdown_connections 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881643c540 0x55881643e950 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588164282f0 0x558816437d70 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558816426be0 0x5588164307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a0aff5700 0 client.29527 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:03.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/663571582 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816430d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 >> 172.21.15.31:0/663571582 conn(0x55881641c3e0 msgr2=0x558816421060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 shutdown_connections 2023-07-29T16:29:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/663571582 wait complete. 2023-07-29T16:29:03.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.439+0000 7f4a40fd6b80 20 client.29527 trim_cache size 0 max 16384 2023-07-29T16:29:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 Processor -- start 2023-07-29T16:29:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 -- start start 2023-07-29T16:29:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x55881642b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x55881642b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881642fca0 con 0x5588164282f0 2023-07-29T16:29:03.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558816430360 con 0x558816427530 2023-07-29T16:29:03.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.454+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881641e460 con 0x558816426be0 2023-07-29T16:29:03.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x55881642b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x55881642b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41910/0 (socket says 172.21.15.31:41910) 2023-07-29T16:29:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 -- 172.21.15.31:0/3184602240 learned_addr learned my addr 172.21.15.31:0/3184602240 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a277fe700 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x55881642b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 -- 172.21.15.31:0/3184602240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 msgr2=0x558816435d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 -- 172.21.15.31:0/3184602240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 msgr2=0x55881642b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x55881642b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 -- 172.21.15.31:0/3184602240 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588161cc250 con 0x558816427530 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.455+0000 7f4a267fc700 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x55881642b1f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x558816427940 tx=0x7f4a1c0092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2d9408b9ed77397 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a0dffb700 1 -- 172.21.15.31:0/3184602240 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c00ee30 con 0x558816427530 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a0dffb700 1 -- 172.21.15.31:0/3184602240 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a1c005500 con 0x558816427530 2023-07-29T16:29:03.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a0dffb700 1 -- 172.21.15.31:0/3184602240 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c01c410 con 0x558816427530 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/3184602240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 msgr2=0x55881642b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x55881642b1f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x558816427940 tx=0x7f4a1c0092d0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/3184602240 shutdown_connections 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x55881642b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/3184602240 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x55881642b1f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.456+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/3184602240 >> 172.21.15.31:0/3184602240 conn(0x55881641c3e0 msgr2=0x55881642e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.457+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/3184602240 shutdown_connections 2023-07-29T16:29:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.457+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/3184602240 wait complete. 2023-07-29T16:29:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.457+0000 7f4a40fd6b80 1 -- start start 2023-07-29T16:29:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.457+0000 7f4a40fd6b80 1 Processor -- start 2023-07-29T16:29:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816434310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x5588164348b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558816434fc0 con 0x5588164282f0 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558816435130 con 0x558816427530 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55881648e0c0 con 0x558816426be0 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a40fd6b80 10 client.0 _ll_register_callbacks cb 0x55881631d620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816434310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a26ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816434310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41924/0 (socket says 172.21.15.31:41924) 2023-07-29T16:29:03.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 -- 172.21.15.31:0/2902127811 learned_addr learned my addr 172.21.15.31:0/2902127811 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a277fe700 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x5588164348b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 -- 172.21.15.31:0/2902127811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 msgr2=0x558816435d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 -- 172.21.15.31:0/2902127811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 msgr2=0x5588164348b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x5588164348b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.458+0000 7f4a267fc700 1 -- 172.21.15.31:0/2902127811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a281a0a30 con 0x558816427530 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.459+0000 7f4a267fc700 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816434310 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00ead0 tx=0x7f4a1c00eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a434e92c8abb526 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:03.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.459+0000 7f4a0aff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.459+0000 7f4a0aff5700 1 -- 172.21.15.31:0/2902127811 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c003e70 con 0x558816427530 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.459+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a1c021070 con 0x558816427530 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.459+0000 7f4a0aff5700 1 -- 172.21.15.31:0/2902127811 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4a1c0050c0 con 0x558816427530 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a0aff5700 1 -- 172.21.15.31:0/2902127811 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a1c0052c0 con 0x558816427530 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 10 client.29537 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f4a1c02a040 con 0x558816427530 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a0aff5700 1 -- 172.21.15.31:0/2902127811 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a1c047020 con 0x558816427530 2023-07-29T16:29:03.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a0aff5700 1 -- 172.21.15.31:0/2902127811 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f4a1c02d070 con 0x558816427530 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 10 client.29537 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 10 client.29537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 20 client.29537 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881648f6e0 0x558816491af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 4 client.29537 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.460+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558816370d40 con 0x55881648f6e0 2023-07-29T16:29:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881648f6e0 0x558816491af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a26ffd700 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881648f6e0 0x558816491af0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e071d92c230e0314 server_cookie=8c7020a5f924131 in_seq=0 out_seq=0 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a0aff5700 10 client.29537 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a0aff5700 1 -- 172.21.15.31:0/2902127811 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558816370d40 con 0x55881648f6e0 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a0aff5700 10 client.29537 handle_command_reply: tid=0 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 client.29537 shutdown 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 msgr2=0x558816434310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816434310 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c00ead0 tx=0x7f4a1c00eec0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 shutdown_connections 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55881648f6e0 0x558816491af0 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5588164282f0 0x5588164348b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558816426be0 0x558816435d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a0aff5700 0 client.29537 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 --2- 172.21.15.31:0/2902127811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558816427530 0x558816434310 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.461+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 >> 172.21.15.31:0/2902127811 conn(0x55881641c3e0 msgr2=0x55881642b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 shutdown_connections 2023-07-29T16:29:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a40fd6b80 1 -- 172.21.15.31:0/2902127811 wait complete. 2023-07-29T16:29:03.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a40fd6b80 20 client.29537 trim_cache size 0 max 16384 2023-07-29T16:29:03.468 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:29:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a100a0810 msgr2=0x7f4a100a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a100a0810 0x7f4a100a2cc0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c009c60 tx=0x7f4a1c009260 comp rx=0 tx=0).stop 2023-07-29T16:29:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 msgr2=0x7f4a281983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 0x7f4a281983a0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800e970 tx=0x7f4a1800ec80 comp rx=0 tx=0).stop 2023-07-29T16:29:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 shutdown_connections 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4a100a0810 0x7f4a100a2cc0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a28101eb0 0x7f4a2819f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4a28100330 0x7f4a281983a0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 --2- 172.21.15.31:0/2984967612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a28101520 0x7f4a281988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.462+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 >> 172.21.15.31:0/2984967612 conn(0x7f4a280fb970 msgr2=0x7f4a28106d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.463+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 shutdown_connections 2023-07-29T16:29:03.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:03.463+0000 7f4a2d5ff700 1 -- 172.21.15.31:0/2984967612 wait complete. 2023-07-29T16:29:03.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1068296 8.89 MB/sec execute 528 sec latency 16.618 ms 2023-07-29T16:29:03.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2553771406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:29:04.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2553771406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:29:04.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2553771406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:29:04.481 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:29:04.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:29:04.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/2104714830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 msgr2=0x7f0d54108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/2104714830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d54108770 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00b0a0 tx=0x7f0d4c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/2104714830 shutdown_connections 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/2104714830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d54108770 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/2104714830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d54100330 0x7f0d54100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/2104714830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d54101520 0x7f0d54101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/2104714830 >> 172.21.15.31:0/2104714830 conn(0x7f0d540fb970 msgr2=0x7f0d540fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/2104714830 shutdown_connections 2023-07-29T16:29:04.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.708+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/2104714830 wait complete. 2023-07-29T16:29:04.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.709+0000 7f0d5a3ff700 1 Processor -- start 2023-07-29T16:29:04.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 -- start start 2023-07-29T16:29:04.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d54100330 0x7f0d541961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d54101520 0x7f0d541966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.710 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d5419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5419dd20 con 0x7f0d54101520 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5419de60 con 0x7f0d54101eb0 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.710+0000 7f0d5a3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5419dff0 con 0x7f0d54100330 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d5419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d537fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d54101520 0x7f0d541966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d5419d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41944/0 (socket says 172.21.15.31:41944) 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 -- 172.21.15.31:0/1588287376 learned_addr learned my addr 172.21.15.31:0/1588287376 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:04.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d54100330 0x7f0d541961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 -- 172.21.15.31:0/1588287376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d54100330 msgr2=0x7f0d541961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d54100330 0x7f0d541961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 -- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d54101520 msgr2=0x7f0d541966e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d54101520 0x7f0d541966e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 -- 172.21.15.31:0/1588287376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44009520 con 0x7f0d54101eb0 2023-07-29T16:29:04.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d5899c700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d5419d750 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00d010 tx=0x7f0d4c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:04.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d517fa700 1 -- 172.21.15.31:0/1588287376 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00e040 con 0x7f0d54101eb0 2023-07-29T16:29:04.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.711+0000 7f0d517fa700 1 -- 172.21.15.31:0/1588287376 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d4c003e10 con 0x7f0d54101eb0 2023-07-29T16:29:04.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.712+0000 7f0d517fa700 1 -- 172.21.15.31:0/1588287376 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0052a0 con 0x7f0d54101eb0 2023-07-29T16:29:04.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.712+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c009d00 con 0x7f0d54101eb0 2023-07-29T16:29:04.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.712+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5419e500 con 0x7f0d54101eb0 2023-07-29T16:29:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.715+0000 7f0d517fa700 1 -- 172.21.15.31:0/1588287376 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0d4c01d480 con 0x7f0d54101eb0 2023-07-29T16:29:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.716+0000 7f0d6dcb4b80 1 Processor -- start 2023-07-29T16:29:04.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.716+0000 7f0d6dcb4b80 1 -- start start 2023-07-29T16:29:04.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 0x55ee55d60070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d605b0 0x55ee55d60de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d61370 0x55ee55d617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55cb7a90 con 0x55ee55d605b0 2023-07-29T16:29:04.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55cb5450 con 0x55ee55d5fc60 2023-07-29T16:29:04.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55cb0d40 con 0x55ee55d61370 2023-07-29T16:29:04.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 0x55ee55d60070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 0x55ee55d60070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41960/0 (socket says 172.21.15.31:41960) 2023-07-29T16:29:04.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 -- 172.21.15.31:0/406196552 learned_addr learned my addr 172.21.15.31:0/406196552 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:04.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d53fff700 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d605b0 0x55ee55d60de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d61370 0x55ee55d617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 -- 172.21.15.31:0/406196552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d61370 msgr2=0x55ee55d617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d61370 0x55ee55d617c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 -- 172.21.15.31:0/406196552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d605b0 msgr2=0x55ee55d60de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d605b0 0x55ee55d60de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.717+0000 7f0d537fe700 1 -- 172.21.15.31:0/406196552 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee55adee60 con 0x55ee55d5fc60 2023-07-29T16:29:04.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d517fa700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d3c0a0810 0x7f0d3c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d517fa700 1 -- 172.21.15.31:0/1588287376 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0d4c0ccbb0 con 0x7f0d54101eb0 2023-07-29T16:29:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d537fe700 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 0x55ee55d60070 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0d48013910 tx=0x7f0d48009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd271b1e7031e5a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d3c0a0810 0x7f0d3c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d3a7fc700 1 -- 172.21.15.31:0/406196552 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48013170 con 0x55ee55d5fc60 2023-07-29T16:29:04.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d3a7fc700 1 -- 172.21.15.31:0/406196552 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d480132d0 con 0x55ee55d5fc60 2023-07-29T16:29:04.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.718+0000 7f0d3a7fc700 1 -- 172.21.15.31:0/406196552 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d48005550 con 0x55ee55d5fc60 2023-07-29T16:29:04.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/406196552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 msgr2=0x55ee55d60070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.727 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d3c0a0810 0x7f0d3c0a2cc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x55ee55d609c0 tx=0x7f0d44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 0x55ee55d60070 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0d48013910 tx=0x7f0d48009260 comp rx=0 tx=0).stop 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/406196552 shutdown_connections 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d61370 0x55ee55d617c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d5fc60 0x55ee55d60070 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/406196552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d605b0 0x55ee55d60de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/406196552 >> 172.21.15.31:0/406196552 conn(0x55ee55d55460 msgr2=0x55ee55d5aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:04.728 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/406196552 shutdown_connections 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.719+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/406196552 wait complete. 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.720+0000 7f0d6dcb4b80 1 -- start start 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.720+0000 7f0d6dcb4b80 1 Processor -- start 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.720+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d69930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d69e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d70eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55c4d590 con 0x55ee55d61370 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55c4d230 con 0x55ee55d605b0 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55b349a0 con 0x55ee55d5fc60 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d6dcb4b80 10 client.0 _ll_register_callbacks cb 0x55ee55c566a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d69930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d69930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48514/0 (socket says 172.21.15.31:48514) 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 -- 172.21.15.31:0/3753852258 learned_addr learned my addr 172.21.15.31:0/3753852258 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d5899c700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d70eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d537fe700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d69e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 -- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 msgr2=0x55ee55d69e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d69e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 -- 172.21.15.31:0/3753852258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 msgr2=0x55ee55d70eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d70eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d53fff700 1 -- 172.21.15.31:0/3753852258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4801b040 con 0x55ee55d5fc60 2023-07-29T16:29:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.721+0000 7f0d5899c700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d70eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d53fff700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d69930 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009850 tx=0x7f0d440047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84a3503f6947c2ce server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d337fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d337fe700 1 -- 172.21.15.31:0/3753852258 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44017070 con 0x55ee55d5fc60 2023-07-29T16:29:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee55cdbd90 con 0x55ee55d5fc60 2023-07-29T16:29:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d337fe700 1 -- 172.21.15.31:0/3753852258 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0d44012e40 con 0x55ee55d5fc60 2023-07-29T16:29:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d337fe700 1 -- 172.21.15.31:0/3753852258 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d4401e470 con 0x55ee55d5fc60 2023-07-29T16:29:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d6dcb4b80 10 client.29545 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.722+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ee55ac7ce0 con 0x55ee55d5fc60 2023-07-29T16:29:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d337fe700 1 -- 172.21.15.31:0/3753852258 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ee55cdbd90 con 0x55ee55d5fc60 2023-07-29T16:29:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d337fe700 1 -- 172.21.15.31:0/3753852258 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f0d44032780 con 0x55ee55d5fc60 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d6dcb4b80 10 client.29545 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d6dcb4b80 10 client.29545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d6dcb4b80 20 client.29545 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55d75730 0x55ee55d77b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d6dcb4b80 4 client.29545 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee55d2edf0 con 0x55ee55d75730 2023-07-29T16:29:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.723+0000 7f0d537fe700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55d75730 0x55ee55d77b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.724+0000 7f0d537fe700 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55d75730 0x55ee55d77b40 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ead47f3e9fbb1a server_cookie=28ef93a66dcfd0a in_seq=0 out_seq=0 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.724+0000 7f0d337fe700 10 client.29545 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d337fe700 1 -- 172.21.15.31:0/3753852258 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ee55d2edf0 con 0x55ee55d75730 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d337fe700 10 client.29545 handle_command_reply: tid=0 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 client.29545 shutdown 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 msgr2=0x55ee55d69930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d69930 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d44009850 tx=0x7f0d440047c0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.735 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 shutdown_connections 2023-07-29T16:29:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55d75730 0x55ee55d77b40 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d70eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d69930 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d337fe700 0 client.29545 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/3753852258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d69e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.736 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 >> 172.21.15.31:0/3753852258 conn(0x55ee55d55460 msgr2=0x55ee55d5a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:04.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 shutdown_connections 2023-07-29T16:29:04.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/3753852258 wait complete. 2023-07-29T16:29:04.737 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.725+0000 7f0d6dcb4b80 20 client.29545 trim_cache size 0 max 16384 2023-07-29T16:29:04.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.744+0000 7f0d6dcb4b80 1 Processor -- start 2023-07-29T16:29:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 -- start start 2023-07-29T16:29:04.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d6ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d696e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d69c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55d60c70 con 0x55ee55d61370 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55d6aab0 con 0x55ee55d605b0 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55d5bd50 con 0x55ee55d5fc60 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d6ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d537fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d6ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48524/0 (socket says 172.21.15.31:48524) 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d53fff700 1 -- 172.21.15.31:0/1169440494 learned_addr learned my addr 172.21.15.31:0/1169440494 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.745+0000 7f0d5899c700 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d69c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d53fff700 1 -- 172.21.15.31:0/1169440494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 msgr2=0x55ee55d696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d53fff700 1 -- 172.21.15.31:0/1169440494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 msgr2=0x55ee55d69c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d69c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d53fff700 1 -- 172.21.15.31:0/1169440494 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee55b052d0 con 0x55ee55d5fc60 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d537fe700 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d5899c700 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d69c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d6ee60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0d44000c00 tx=0x7f0d44009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f68928d87390671a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d3affd700 1 -- 172.21.15.31:0/1169440494 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44032620 con 0x55ee55d5fc60 2023-07-29T16:29:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d3affd700 1 -- 172.21.15.31:0/1169440494 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d44012ec0 con 0x55ee55d5fc60 2023-07-29T16:29:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.746+0000 7f0d3affd700 1 -- 172.21.15.31:0/1169440494 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4401f6b0 con 0x55ee55d5fc60 2023-07-29T16:29:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1169440494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 msgr2=0x55ee55d6ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d6ee60 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0d44000c00 tx=0x7f0d44009cc0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1169440494 shutdown_connections 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55d69c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55d6ee60 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1169440494 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55d696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1169440494 >> 172.21.15.31:0/1169440494 conn(0x55ee55d55460 msgr2=0x55ee55d64340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1169440494 shutdown_connections 2023-07-29T16:29:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1169440494 wait complete. 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.747+0000 7f0d6dcb4b80 1 -- start start 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 Processor -- start 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55dc4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55dc46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55dcb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55dcbc30 con 0x55ee55d61370 2023-07-29T16:29:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55dcbd70 con 0x55ee55d605b0 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee55dcbf00 con 0x55ee55d5fc60 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d6dcb4b80 10 client.0 _ll_register_callbacks cb 0x55ee55c566a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55dc4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d5899c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55dcb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55dc4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48534/0 (socket says 172.21.15.31:48534) 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.748+0000 7f0d53fff700 1 -- 172.21.15.31:0/1552967606 learned_addr learned my addr 172.21.15.31:0/1552967606 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:04.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d537fe700 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55dc46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d53fff700 1 -- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 msgr2=0x55ee55dc46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55dc46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d53fff700 1 -- 172.21.15.31:0/1552967606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 msgr2=0x55ee55dcb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55dcb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d53fff700 1 -- 172.21.15.31:0/1552967606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d48017040 con 0x55ee55d5fc60 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d53fff700 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55dc4170 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0d44005cb0 tx=0x7f0d4401e6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b242e30dc18caa8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:04.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.749+0000 7f0d337fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d337fe700 1 -- 172.21.15.31:0/1552967606 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d44032ab0 con 0x55ee55d5fc60 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d337fe700 1 -- 172.21.15.31:0/1552967606 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0d44032870 con 0x55ee55d5fc60 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee55ca9dc0 con 0x55ee55d5fc60 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d337fe700 1 -- 172.21.15.31:0/1552967606 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d44005050 con 0x55ee55d5fc60 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d6dcb4b80 10 client.29551 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55ee55ac7ce0 con 0x55ee55d5fc60 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.750+0000 7f0d337fe700 1 -- 172.21.15.31:0/1552967606 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ee55ca9dc0 con 0x55ee55d5fc60 2023-07-29T16:29:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d337fe700 1 -- 172.21.15.31:0/1552967606 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f0d44004c80 con 0x55ee55d5fc60 2023-07-29T16:29:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d6dcb4b80 10 client.29551 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d6dcb4b80 10 client.29551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d6dcb4b80 20 client.29551 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55dcd580 0x55ee55dcf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d6dcb4b80 4 client.29551 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:29:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ee55ca9ae0 con 0x55ee55dcd580 2023-07-29T16:29:04.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.751+0000 7f0d537fe700 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55dcd580 0x55ee55dcf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.752+0000 7f0d337fe700 10 client.29551 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.752+0000 7f0d537fe700 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55dcd580 0x55ee55dcf990 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7bfafa9345053c3 server_cookie=1564f75e59c5346a in_seq=0 out_seq=0 2023-07-29T16:29:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d337fe700 1 -- 172.21.15.31:0/1552967606 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ee55ca9ae0 con 0x55ee55dcd580 2023-07-29T16:29:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d337fe700 10 client.29551 handle_command_reply: tid=0 2023-07-29T16:29:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 client.29551 shutdown 2023-07-29T16:29:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 msgr2=0x55ee55dc4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55dc4170 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0d44005cb0 tx=0x7f0d4401e6a0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 shutdown_connections 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ee55dcd580 0x55ee55dcf990 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee55d61370 0x55ee55dcb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ee55d5fc60 0x55ee55dc4170 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d337fe700 0 client.29551 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 --2- 172.21.15.31:0/1552967606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee55d605b0 0x55ee55dc46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 >> 172.21.15.31:0/1552967606 conn(0x55ee55d55460 msgr2=0x55ee55d696e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.753+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 shutdown_connections 2023-07-29T16:29:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d6dcb4b80 1 -- 172.21.15.31:0/1552967606 wait complete. 2023-07-29T16:29:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d6dcb4b80 20 client.29551 trim_cache size 0 max 16384 2023-07-29T16:29:04.760 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:29:04.761 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:29:04.761 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "a178a74c-853d-4bfe-bade-0185ff3eff6e", 2023-07-29T16:29:04.761 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:29:04.761 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:29:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d3c0a0810 msgr2=0x7f0d3c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d3c0a0810 0x7f0d3c0a2cc0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x55ee55d609c0 tx=0x7f0d44009260 comp rx=0 tx=0).stop 2023-07-29T16:29:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 msgr2=0x7f0d5419d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d5419d750 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c00d010 tx=0x7f0d4c00bc80 comp rx=0 tx=0).stop 2023-07-29T16:29:04.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 shutdown_connections 2023-07-29T16:29:04.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0d3c0a0810 0x7f0d3c0a2cc0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0d54101eb0 0x7f0d5419d750 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.763 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d54100330 0x7f0d541961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 --2- 172.21.15.31:0/1588287376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d54101520 0x7f0d541966e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:04.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 >> 172.21.15.31:0/1588287376 conn(0x7f0d540fb970 msgr2=0x7f0d540fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:04.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.754+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 shutdown_connections 2023-07-29T16:29:04.764 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:04.755+0000 7f0d5a3ff700 1 -- 172.21.15.31:0/1588287376 wait complete. 2023-07-29T16:29:04.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:29:04.840 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:04 smithi031 ceph-mon[95504]: pgmap v779: 97 pgs: 97 active+clean; 195 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 2.6 MiB/s wr, 283 op/s 2023-07-29T16:29:04.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1069606 8.89 MB/sec execute 529 sec latency 21.901 ms 2023-07-29T16:29:04.999 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 -- 172.21.15.31:0/1093878397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 msgr2=0x7ff234073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 --2- 172.21.15.31:0/1093878397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff234073160 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff224009990 tx=0x7ff224009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 -- 172.21.15.31:0/1093878397 shutdown_connections 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 --2- 172.21.15.31:0/1093878397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2341081e0 0x7ff23410a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 --2- 172.21.15.31:0/1093878397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff234073160 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 --2- 172.21.15.31:0/1093878397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 0x7ff234073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 -- 172.21.15.31:0/1093878397 >> 172.21.15.31:0/1093878397 conn(0x7ff2340fb970 msgr2=0x7ff2340fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:05.000 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 -- 172.21.15.31:0/1093878397 shutdown_connections 2023-07-29T16:29:05.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.000+0000 7ff239fff700 1 -- 172.21.15.31:0/1093878397 wait complete. 2023-07-29T16:29:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.001+0000 7ff239fff700 1 Processor -- start 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.001+0000 7ff239fff700 1 -- start start 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.001+0000 7ff239fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 0x7ff234198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.001+0000 7ff239fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff2341987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.001+0000 7ff239fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2341081e0 0x7ff23419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff239fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23419fdf0 con 0x7ff2341081e0 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff239fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23419ff30 con 0x7ff234073730 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff239fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2341a00c0 con 0x7ff234074d10 2023-07-29T16:29:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 0x7ff234198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 0x7ff234198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42016/0 (socket says 172.21.15.31:42016) 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff2341987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff2337fe700 1 -- 172.21.15.31:0/940466143 learned_addr learned my addr 172.21.15.31:0/940466143 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff233fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2341081e0 0x7ff23419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff232ffd700 1 -- 172.21.15.31:0/940466143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 msgr2=0x7ff234198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff232ffd700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 0x7ff234198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff232ffd700 1 -- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2341081e0 msgr2=0x7ff23419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff232ffd700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2341081e0 0x7ff23419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.002+0000 7ff232ffd700 1 -- 172.21.15.31:0/940466143 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240095f0 con 0x7ff234074d10 2023-07-29T16:29:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.003+0000 7ff232ffd700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff2341987b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff22800e9f0 tx=0x7ff22800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.003+0000 7ff230ff9700 1 -- 172.21.15.31:0/940466143 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22800c9c0 con 0x7ff234074d10 2023-07-29T16:29:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.003+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2341a03a0 con 0x7ff234074d10 2023-07-29T16:29:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.003+0000 7ff230ff9700 1 -- 172.21.15.31:0/940466143 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff228010430 con 0x7ff234074d10 2023-07-29T16:29:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.003+0000 7ff230ff9700 1 -- 172.21.15.31:0/940466143 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228018680 con 0x7ff234074d10 2023-07-29T16:29:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.003+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2341a0900 con 0x7ff234074d10 2023-07-29T16:29:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.006+0000 7ff230ff9700 1 -- 172.21.15.31:0/940466143 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff228010860 con 0x7ff234074d10 2023-07-29T16:29:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.007+0000 7ff24d9a5b80 1 Processor -- start 2023-07-29T16:29:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 -- start start 2023-07-29T16:29:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff230ff9700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff21c0a0810 0x7ff21c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff230ff9700 1 -- 172.21.15.31:0/940466143 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff228014070 con 0x7ff234074d10 2023-07-29T16:29:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be4700ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4701d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff233fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff21c0a0810 0x7ff21c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4702740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be4658a10 con 0x558be4700be0 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be46563d0 con 0x558be47022f0 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be4651cc0 con 0x558be4701530 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4701d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff233fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4702740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be4700ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4701d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48568/0 (socket says 172.21.15.31:48568) 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.008+0000 7ff232ffd700 1 -- 172.21.15.31:0/1747476900 learned_addr learned my addr 172.21.15.31:0/1747476900 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff232ffd700 1 -- 172.21.15.31:0/1747476900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 msgr2=0x558be4702740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff232ffd700 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4702740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff232ffd700 1 -- 172.21.15.31:0/1747476900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 msgr2=0x558be4700ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff232ffd700 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be4700ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff232ffd700 1 -- 172.21.15.31:0/1747476900 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558be447fde0 con 0x558be4701530 2023-07-29T16:29:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff232ffd700 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4701d60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff228099310 tx=0x7ff228098ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55ffdb72c942a70e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff233fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff21c0a0810 0x7ff21c0a2cc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009ec0 tx=0x7ff22c013480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff219ffb700 1 -- 172.21.15.31:0/1747476900 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22809d1b0 con 0x558be4701530 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff219ffb700 1 -- 172.21.15.31:0/1747476900 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff228099380 con 0x558be4701530 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.009+0000 7ff219ffb700 1 -- 172.21.15.31:0/1747476900 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228097840 con 0x558be4701530 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1747476900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 msgr2=0x558be4701d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4701d60 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7ff228099310 tx=0x7ff228098ec0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1747476900 shutdown_connections 2023-07-29T16:29:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4702740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be4700ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1747476900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4701d60 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1747476900 >> 172.21.15.31:0/1747476900 conn(0x558be46f63e0 msgr2=0x558be46fb990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1747476900 shutdown_connections 2023-07-29T16:29:05.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.010+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1747476900 wait complete. 2023-07-29T16:29:05.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.011+0000 7ff24d9a5b80 1 -- start start 2023-07-29T16:29:05.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.011+0000 7ff24d9a5b80 1 Processor -- start 2023-07-29T16:29:05.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be470ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4711dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be45ee510 con 0x558be4700be0 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be45ee1b0 con 0x558be47022f0 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be44d5920 con 0x558be4701530 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff24d9a5b80 10 client.0 _ll_register_callbacks cb 0x558be45f7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be470ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.017 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be470ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48582/0 (socket says 172.21.15.31:48582) 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff232ffd700 1 -- 172.21.15.31:0/1838846827 learned_addr learned my addr 172.21.15.31:0/1838846827 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff233fff700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4711dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff2337fe700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff233fff700 1 -- 172.21.15.31:0/1838846827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 msgr2=0x558be470ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff233fff700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be470ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff233fff700 1 -- 172.21.15.31:0/1838846827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 msgr2=0x558be470a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff233fff700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff233fff700 1 -- 172.21.15.31:0/1838846827 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2341a0900 con 0x558be47022f0 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.012+0000 7ff232ffd700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be470ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.013+0000 7ff233fff700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4711dc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff22c01aa50 tx=0x7ff22c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb0b40217315d3f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.013+0000 7ff212ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.013+0000 7ff212ffd700 1 -- 172.21.15.31:0/1838846827 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22c00e040 con 0x558be47022f0 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.013+0000 7ff212ffd700 1 -- 172.21.15.31:0/1838846827 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff22c0046e0 con 0x558be47022f0 2023-07-29T16:29:05.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.013+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558be467cd10 con 0x558be47022f0 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.013+0000 7ff212ffd700 1 -- 172.21.15.31:0/1838846827 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff22c01aec0 con 0x558be47022f0 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 10 client.29567 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558be4468c60 con 0x558be47022f0 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff212ffd700 1 -- 172.21.15.31:0/1838846827 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558be467cd10 con 0x558be47022f0 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff212ffd700 1 -- 172.21.15.31:0/1838846827 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7ff22c008ab0 con 0x558be47022f0 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 10 client.29567 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 10 client.29567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:05.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 20 client.29567 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be47165c0 0x558be47189d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 4 client.29567 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558be46cfd70 con 0x558be47165c0 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.014+0000 7ff2337fe700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be47165c0 0x558be47189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.015+0000 7ff2337fe700 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be47165c0 0x558be47189d0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26f636b2c129d25c server_cookie=3ddcf04a70d457ca in_seq=0 out_seq=0 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.015+0000 7ff212ffd700 10 client.29567 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:05.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.016+0000 7ff212ffd700 1 -- 172.21.15.31:0/1838846827 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558be46cfd70 con 0x558be47165c0 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.016+0000 7ff212ffd700 10 client.29567 handle_command_reply: tid=0 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.016+0000 7ff24d9a5b80 1 client.29567 shutdown 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 msgr2=0x558be4711dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4711dc0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7ff22c01aa50 tx=0x7ff22c004b40 comp rx=0 tx=0).stop 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 shutdown_connections 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be47165c0 0x558be47189d0 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be4711dc0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff212ffd700 0 client.29567 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:05.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/1838846827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be470ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 >> 172.21.15.31:0/1838846827 conn(0x558be46f63e0 msgr2=0x558be46fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:05.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 shutdown_connections 2023-07-29T16:29:05.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/1838846827 wait complete. 2023-07-29T16:29:05.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.017+0000 7ff24d9a5b80 20 client.29567 trim_cache size 0 max 16384 2023-07-29T16:29:05.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:04 smithi039 ceph-mon[128858]: pgmap v779: 97 pgs: 97 active+clean; 195 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 2.6 MiB/s wr, 283 op/s 2023-07-29T16:29:05.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 Processor -- start 2023-07-29T16:29:05.035 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 -- start start 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be47094e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4711d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be46fb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be46f8460 con 0x558be4700be0 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be470b4c0 con 0x558be47022f0 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.036+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be46f36e0 con 0x558be4701530 2023-07-29T16:29:05.037 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4711d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff233fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be46fb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4711d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48584/0 (socket says 172.21.15.31:48584) 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff233fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be46fb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42038/0 (socket says 172.21.15.31:42038) 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 -- 172.21.15.31:0/4114221248 learned_addr learned my addr 172.21.15.31:0/4114221248 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff2337fe700 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be47094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 -- 172.21.15.31:0/4114221248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 msgr2=0x558be46fb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.038 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be46fb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 -- 172.21.15.31:0/4114221248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 msgr2=0x558be47094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be47094e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 -- 172.21.15.31:0/4114221248 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558be44a6250 con 0x558be4701530 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff233fff700 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be46fb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff232ffd700 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4711d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff228095880 tx=0x7ff228009790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc40e5f3e071c6f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.037+0000 7ff21a7fc700 1 -- 172.21.15.31:0/4114221248 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280974c0 con 0x558be4701530 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff21a7fc700 1 -- 172.21.15.31:0/4114221248 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff22809d1b0 con 0x558be4701530 2023-07-29T16:29:05.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff21a7fc700 1 -- 172.21.15.31:0/4114221248 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228098ad0 con 0x558be4701530 2023-07-29T16:29:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/4114221248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 msgr2=0x558be4711d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4711d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7ff228095880 tx=0x7ff228009790 comp rx=0 tx=0).stop 2023-07-29T16:29:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/4114221248 shutdown_connections 2023-07-29T16:29:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be47022f0 0x558be46fb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be47094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.042 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/4114221248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be4701530 0x558be4711d80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/4114221248 >> 172.21.15.31:0/4114221248 conn(0x558be46f63e0 msgr2=0x558be46fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/4114221248 shutdown_connections 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.038+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/4114221248 wait complete. 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.039+0000 7ff24d9a5b80 1 -- start start 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.039+0000 7ff24d9a5b80 1 Processor -- start 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.043 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be4701530 0x558be470dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be47022f0 0x558be4711010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be4711550 con 0x558be4700be0 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be47116c0 con 0x558be4701530 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558be4711880 con 0x558be47022f0 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff24d9a5b80 10 client.0 _ll_register_callbacks cb 0x558be45f7620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff232ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be4701530 0x558be470dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.044 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff2337fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52014/0 (socket says 172.21.15.31:52014) 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff2337fe700 1 -- 172.21.15.31:0/2931386402 learned_addr learned my addr 172.21.15.31:0/2931386402 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.040+0000 7ff233fff700 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be47022f0 0x558be4711010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff2337fe700 1 -- 172.21.15.31:0/2931386402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be47022f0 msgr2=0x558be4711010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff2337fe700 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be47022f0 0x558be4711010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff2337fe700 1 -- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be4701530 msgr2=0x558be470dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff2337fe700 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be4701530 0x558be470dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.045 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff2337fe700 1 -- 172.21.15.31:0/2931386402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558be4468c60 con 0x558be4700be0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff2337fe700 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470dab0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff224005910 tx=0x7ff22400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba553804784f4b1f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff212ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff212ffd700 1 -- 172.21.15.31:0/2931386402 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff22401d040 con 0x558be4700be0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff228099380 con 0x558be4700be0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.041+0000 7ff212ffd700 1 -- 172.21.15.31:0/2931386402 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff2240292b0 con 0x558be4700be0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff212ffd700 1 -- 172.21.15.31:0/2931386402 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff224029470 con 0x558be4700be0 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 10 client.19455 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:05.046 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x558be447fde0 con 0x558be4700be0 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff212ffd700 1 -- 172.21.15.31:0/2931386402 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff228099380 con 0x558be4700be0 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff212ffd700 1 -- 172.21.15.31:0/2931386402 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7ff224004500 con 0x558be4700be0 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 10 client.19455 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 10 client.19455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 20 client.19455 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be4768fb0 0x558be476b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:05.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 4 client.19455 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:29:05.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.042+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558be464ad40 con 0x558be4768fb0 2023-07-29T16:29:05.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.043+0000 7ff232ffd700 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be4768fb0 0x558be476b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:05.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.043+0000 7ff232ffd700 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be4768fb0 0x558be476b3c0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=637fa03e11e5ba3a server_cookie=e671639aa2c298ac in_seq=0 out_seq=0 2023-07-29T16:29:05.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.043+0000 7ff212ffd700 10 client.19455 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:05.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.043+0000 7ff212ffd700 1 -- 172.21.15.31:0/2931386402 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558be464ad40 con 0x558be4768fb0 2023-07-29T16:29:05.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.043+0000 7ff212ffd700 10 client.19455 handle_command_reply: tid=0 2023-07-29T16:29:05.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.043+0000 7ff24d9a5b80 1 client.19455 shutdown 2023-07-29T16:29:05.049 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 msgr2=0x558be470dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470dab0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff224005910 tx=0x7ff22400ba40 comp rx=0 tx=0).stop 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 shutdown_connections 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558be4768fb0 0x558be476b3c0 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558be47022f0 0x558be4711010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff212ffd700 0 client.19455 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558be4700be0 0x558be470dab0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 --2- 172.21.15.31:0/2931386402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558be4701530 0x558be470dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 >> 172.21.15.31:0/2931386402 conn(0x558be46f63e0 msgr2=0x558be47094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:05.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 shutdown_connections 2023-07-29T16:29:05.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 1 -- 172.21.15.31:0/2931386402 wait complete. 2023-07-29T16:29:05.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.044+0000 7ff24d9a5b80 20 client.19455 trim_cache size 0 max 16384 2023-07-29T16:29:05.051 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:29:05.051 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:29:05.051 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout: "a178a74c-853d-4bfe-bade-0185ff3eff6e": { 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "a178a74c-853d-4bfe-bade-0185ff3eff6e", 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:29:05.052 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:29:05.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff21c0a0810 msgr2=0x7ff21c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff21c0a0810 0x7ff21c0a2cc0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff22c009ec0 tx=0x7ff22c013480 comp rx=0 tx=0).stop 2023-07-29T16:29:05.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 msgr2=0x7ff2341987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:05.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff2341987b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff22800e9f0 tx=0x7ff22800ed00 comp rx=0 tx=0).stop 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 shutdown_connections 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff21c0a0810 0x7ff21c0a2cc0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2341081e0 0x7ff23419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff234073730 0x7ff234198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 --2- 172.21.15.31:0/940466143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff234074d10 0x7ff2341987b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 >> 172.21.15.31:0/940466143 conn(0x7ff2340fb970 msgr2=0x7ff2340fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:05.054 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 shutdown_connections 2023-07-29T16:29:05.055 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:05.045+0000 7ff239fff700 1 -- 172.21.15.31:0/940466143 wait complete. 2023-07-29T16:29:05.062 INFO:tasks.cephfs.filesystem:scrub status for tag:a178a74c-853d-4bfe-bade-0185ff3eff6e - {'path': '/', 'tag': 'a178a74c-853d-4bfe-bade-0185ff3eff6e', 'options': 'recursive,force'} 2023-07-29T16:29:05.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:04 smithi156 ceph-mon[129245]: pgmap v779: 97 pgs: 97 active+clean; 195 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 2.6 MiB/s wr, 283 op/s 2023-07-29T16:29:05.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1071194 8.88 MB/sec execute 530 sec latency 26.036 ms 2023-07-29T16:29:05.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:05 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:05.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:05 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:05.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:05 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:05.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:05 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:05.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:05 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:05.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:05 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:05 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:06.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:05 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:06.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:05 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:05 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:05 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:06.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:05 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:05 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:06.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:05 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:06.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:05 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:05 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:05 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:06.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:05 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1072777 8.89 MB/sec execute 531 sec latency 63.319 ms 2023-07-29T16:29:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:06 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:29:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:06 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:06.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:06 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:06.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:06 smithi031 ceph-mon[95504]: pgmap v780: 97 pgs: 97 active+clean; 195 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.3 MiB/s wr, 243 op/s 2023-07-29T16:29:07.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:06 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:29:07.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:07.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:06 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:07.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:06 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:07.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:06 smithi039 ceph-mon[128858]: pgmap v780: 97 pgs: 97 active+clean; 195 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.3 MiB/s wr, 243 op/s 2023-07-29T16:29:07.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:06 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:29:07.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:07.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:06 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:07.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:06 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:07.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:06 smithi156 ceph-mon[129245]: pgmap v780: 97 pgs: 97 active+clean; 195 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.3 MiB/s wr, 243 op/s 2023-07-29T16:29:07.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 -- 172.21.15.31:0/1830492188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe888074d30 msgr2=0x7fe888073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/1830492188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe888074d30 0x7fe888073180 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fe878009990 tx=0x7fe878009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 -- 172.21.15.31:0/1830492188 shutdown_connections 2023-07-29T16:29:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/1830492188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe8881081e0 0x7fe88810a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/1830492188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe888074d30 0x7fe888073180 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/1830492188 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 0x7fe888073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 -- 172.21.15.31:0/1830492188 >> 172.21.15.31:0/1830492188 conn(0x7fe8880fb970 msgr2=0x7fe8880fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 -- 172.21.15.31:0/1830492188 shutdown_connections 2023-07-29T16:29:07.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.888+0000 7fe88d5ff700 1 -- 172.21.15.31:0/1830492188 wait complete. 2023-07-29T16:29:07.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.889+0000 7fe88d5ff700 1 Processor -- start 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 -- start start 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 0x7fe888198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 0x7fe888198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8881081e0 0x7fe888199700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe888199d60 con 0x7fe8881081e0 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe888199ed0 con 0x7fe888074d30 2023-07-29T16:29:07.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe88d5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88819a090 con 0x7fe888073750 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 0x7fe888198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 0x7fe888198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 0x7fe888198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48600/0 (socket says 172.21.15.31:48600) 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 0x7fe888198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42062/0 (socket says 172.21.15.31:42062) 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe886ffd700 1 -- 172.21.15.31:0/429484021 learned_addr learned my addr 172.21.15.31:0/429484021 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.890+0000 7fe8877fe700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8881081e0 0x7fe888199700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8867fc700 1 -- 172.21.15.31:0/429484021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 msgr2=0x7fe888198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8867fc700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 0x7fe888198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8867fc700 1 -- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8881081e0 msgr2=0x7fe888199700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8867fc700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8881081e0 0x7fe888199700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8867fc700 1 -- 172.21.15.31:0/429484021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8780095f0 con 0x7fe888074d30 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8877fe700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8881081e0 0x7fe888199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe8867fc700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 0x7fe888198880 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00e9f0 tx=0x7fe87c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe86ffff700 1 -- 172.21.15.31:0/429484021 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c00eee0 con 0x7fe888074d30 2023-07-29T16:29:07.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe86ffff700 1 -- 172.21.15.31:0/429484021 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe87c00cbd0 con 0x7fe888074d30 2023-07-29T16:29:07.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88819a370 con 0x7fe888074d30 2023-07-29T16:29:07.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe86ffff700 1 -- 172.21.15.31:0/429484021 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c018680 con 0x7fe888074d30 2023-07-29T16:29:07.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.891+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8881a0b30 con 0x7fe888074d30 2023-07-29T16:29:07.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.895+0000 7fe86ffff700 1 -- 172.21.15.31:0/429484021 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe87c00cd40 con 0x7fe888074d30 2023-07-29T16:29:07.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.896+0000 7fe8a0e78b80 1 Processor -- start 2023-07-29T16:29:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.896+0000 7fe8a0e78b80 1 -- start start 2023-07-29T16:29:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102ac978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102ac987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac98cf0 0x56102ac99160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102abef9b0 con 0x56102aaf36b0 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102abed370 con 0x56102ac98cf0 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102abe8c60 con 0x56102ac97df0 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102ac978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac98cf0 0x56102ac99160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8877fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102ac978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52042/0 (socket says 172.21.15.31:52042) 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102ac987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe8877fe700 1 -- 172.21.15.31:0/3368502064 learned_addr learned my addr 172.21.15.31:0/3368502064 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.897+0000 7fe86ffff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe8700a0790 0x7fe8700a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe886ffd700 1 -- 172.21.15.31:0/3368502064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac98cf0 msgr2=0x56102ac99160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe886ffd700 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac98cf0 0x56102ac99160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe886ffd700 1 -- 172.21.15.31:0/3368502064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 msgr2=0x56102ac978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe886ffd700 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102ac978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe886ffd700 1 -- 172.21.15.31:0/3368502064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56102aa16d80 con 0x56102ac97df0 2023-07-29T16:29:07.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe86ffff700 1 -- 172.21.15.31:0/429484021 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe87c014070 con 0x7fe888074d30 2023-07-29T16:29:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe8877fe700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe8700a0790 0x7fe8700a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe886ffd700 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102ac987b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe8780059d0 tx=0x7fe878005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca1d71c07dbbc2f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:07.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe86d7fa700 1 -- 172.21.15.31:0/3368502064 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87801d070 con 0x56102ac97df0 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe86d7fa700 1 -- 172.21.15.31:0/3368502064 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe87801e040 con 0x56102ac97df0 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.898+0000 7fe86d7fa700 1 -- 172.21.15.31:0/3368502064 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878004940 con 0x56102ac97df0 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3368502064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 msgr2=0x56102ac987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8877fe700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe8700a0790 0x7fe8700a2c40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe8800076a0 tx=0x7fe880009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102ac987b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe8780059d0 tx=0x7fe878005bc0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3368502064 shutdown_connections 2023-07-29T16:29:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac98cf0 0x56102ac99160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102ac978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3368502064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102ac987b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3368502064 >> 172.21.15.31:0/3368502064 conn(0x56102ac8cfd0 msgr2=0x56102ac92300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3368502064 shutdown_connections 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.899+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3368502064 wait complete. 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.900+0000 7fe8a0e78b80 1 -- start start 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.900+0000 7fe8a0e78b80 1 Processor -- start 2023-07-29T16:29:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.900+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102aaf36b0 0x56102aca0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.900+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.900+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102aca8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102ab854b0 con 0x56102ac98cf0 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102ab85150 con 0x56102ac97df0 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102aa6c8c0 con 0x56102aaf36b0 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8a0e78b80 10 client.0 _ll_register_callbacks cb 0x56102ab8e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102aaf36b0 0x56102aca0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42082/0 (socket says 172.21.15.31:42082) 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102aaf36b0 0x56102aca0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48628/0 (socket says 172.21.15.31:48628) 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 -- 172.21.15.31:0/4000679317 learned_addr learned my addr 172.21.15.31:0/4000679317 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8877fe700 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102aca8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 -- 172.21.15.31:0/4000679317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102aaf36b0 msgr2=0x56102aca0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102aaf36b0 0x56102aca0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 -- 172.21.15.31:0/4000679317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 msgr2=0x56102aca8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102aca8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 -- 172.21.15.31:0/4000679317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88810b280 con 0x56102ac97df0 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe8867fc700 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca1330 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0991a0 tx=0x7fe87c0999e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf7304c132d8724 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.901+0000 7fe86aff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe86aff5700 1 -- 172.21.15.31:0/4000679317 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c099270 con 0x56102ac97df0 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe86aff5700 1 -- 172.21.15.31:0/4000679317 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe87c0994a0 con 0x56102ac97df0 2023-07-29T16:29:07.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56102ac13cb0 con 0x56102ac97df0 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe86aff5700 1 -- 172.21.15.31:0/4000679317 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe87c098fe0 con 0x56102ac97df0 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe8a0e78b80 10 client.29585 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56102a9ffc00 con 0x56102ac97df0 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe86aff5700 1 -- 172.21.15.31:0/4000679317 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe87c0b5070 con 0x56102ac97df0 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.902+0000 7fe86aff5700 1 -- 172.21.15.31:0/4000679317 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fe87c0a1580 con 0x56102ac97df0 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe8a0e78b80 10 client.29585 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe8a0e78b80 10 client.29585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe8a0e78b80 20 client.29585 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acacc90 0x56102acaf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe8a0e78b80 4 client.29585 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:07.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56102ac66d10 con 0x56102acacc90 2023-07-29T16:29:07.910 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1073664 8.88 MB/sec execute 532 sec latency 27.234 ms 2023-07-29T16:29:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.903+0000 7fe886ffd700 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acacc90 0x56102acaf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.904+0000 7fe886ffd700 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acacc90 0x56102acaf0a0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a17c459f4598ee7 server_cookie=2a8caaa3c7fbacb4 in_seq=0 out_seq=0 2023-07-29T16:29:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.904+0000 7fe86aff5700 10 client.29585 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.904+0000 7fe86aff5700 1 -- 172.21.15.31:0/4000679317 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56102ac66d10 con 0x56102acacc90 2023-07-29T16:29:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.904+0000 7fe86aff5700 10 client.29585 handle_command_reply: tid=0 2023-07-29T16:29:07.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.904+0000 7fe8a0e78b80 1 client.29585 shutdown 2023-07-29T16:29:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 msgr2=0x56102aca1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca1330 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe87c0991a0 tx=0x7fe87c0999e0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 shutdown_connections 2023-07-29T16:29:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acacc90 0x56102acaf0a0 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102aca8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102aaf36b0 0x56102aca0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/4000679317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca1330 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe86aff5700 0 client.29585 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 >> 172.21.15.31:0/4000679317 conn(0x56102ac8cfd0 msgr2=0x56102ac922d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 shutdown_connections 2023-07-29T16:29:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/4000679317 wait complete. 2023-07-29T16:29:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.905+0000 7fe8a0e78b80 20 client.29585 trim_cache size 0 max 16384 2023-07-29T16:29:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 Processor -- start 2023-07-29T16:29:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 -- start start 2023-07-29T16:29:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 0x56102ac91a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102aca8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102ac9f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102ac8ca10 con 0x56102ac98cf0 2023-07-29T16:29:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102aca1a70 con 0x56102aaf36b0 2023-07-29T16:29:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.920+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102ac9edb0 con 0x56102ac97df0 2023-07-29T16:29:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 0x56102ac91a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe8867fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102aca8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 0x56102ac91a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42094/0 (socket says 172.21.15.31:42094) 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 -- 172.21.15.31:0/3789744397 learned_addr learned my addr 172.21.15.31:0/3789744397 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe8877fe700 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102ac9f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 -- 172.21.15.31:0/3789744397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 msgr2=0x56102aca8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102aca8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 -- 172.21.15.31:0/3789744397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 msgr2=0x56102ac9f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102ac9f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 -- 172.21.15.31:0/3789744397 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56102aa3d1f0 con 0x56102aaf36b0 2023-07-29T16:29:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe8877fe700 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102ac9f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe8867fc700 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102aca8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.921+0000 7fe886ffd700 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 0x56102ac91a40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe878006010 tx=0x7fe878005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baa1346ef070c1b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe86dffb700 1 -- 172.21.15.31:0/3789744397 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87801d040 con 0x56102aaf36b0 2023-07-29T16:29:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe86dffb700 1 -- 172.21.15.31:0/3789744397 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe878005500 con 0x56102aaf36b0 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe86dffb700 1 -- 172.21.15.31:0/3789744397 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878004cb0 con 0x56102aaf36b0 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3789744397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 msgr2=0x56102ac91a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 0x56102ac91a40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe878006010 tx=0x7fe878005e80 comp rx=0 tx=0).stop 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3789744397 shutdown_connections 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ac98cf0 0x56102ac9f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102aaf36b0 0x56102ac91a40 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/3789744397 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac97df0 0x56102aca8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3789744397 >> 172.21.15.31:0/3789744397 conn(0x56102ac8cfd0 msgr2=0x56102ac922d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3789744397 shutdown_connections 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.922+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/3789744397 wait complete. 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 -- start start 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 Processor -- start 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102aca7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac98cf0 0x56102aca45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102aca4b70 con 0x56102aaf36b0 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102aca4ce0 con 0x56102ac97df0 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.923+0000 7fe8a0e78b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56102aca4ea0 con 0x56102ac98cf0 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe8a0e78b80 10 client.0 _ll_register_callbacks cb 0x56102ab8e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102aca7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102aca7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52068/0 (socket says 172.21.15.31:52068) 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 -- 172.21.15.31:0/1761684718 learned_addr learned my addr 172.21.15.31:0/1761684718 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe8867fc700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe8877fe700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac98cf0 0x56102aca45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 -- 172.21.15.31:0/1761684718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac98cf0 msgr2=0x56102aca45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac98cf0 0x56102aca45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 -- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 msgr2=0x56102aca8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 -- 172.21.15.31:0/1761684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56102a9ffc00 con 0x56102aaf36b0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe8867fc700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe886ffd700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102aca7b50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe878005c00 tx=0x7fe8780057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25b574cac161557 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.924+0000 7fe86aff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe86aff5700 1 -- 172.21.15.31:0/1761684718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87801d050 con 0x56102aaf36b0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe86aff5700 1 -- 172.21.15.31:0/1761684718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe878031040 con 0x56102aaf36b0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe87801e040 con 0x56102aaf36b0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe86aff5700 1 -- 172.21.15.31:0/1761684718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe878005240 con 0x56102aaf36b0 2023-07-29T16:29:07.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 10 client.19473 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x56102aa16d80 con 0x56102aaf36b0 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe86aff5700 1 -- 172.21.15.31:0/1761684718 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe87801e040 con 0x56102aaf36b0 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe86aff5700 1 -- 172.21.15.31:0/1761684718 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fe878045070 con 0x56102aaf36b0 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 10 client.19473 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 10 client.19473 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 20 client.19473 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acff930 0x56102ad01d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 4 client.19473 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:29:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.925+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56102abe1ce0 con 0x56102acff930 2023-07-29T16:29:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.926+0000 7fe8867fc700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acff930 0x56102ad01d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.926+0000 7fe8867fc700 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acff930 0x56102ad01d40 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b3e27f2ae6a496f server_cookie=9fa139f825475a9e in_seq=0 out_seq=0 2023-07-29T16:29:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:07.926+0000 7fe86aff5700 10 client.19473 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.006+0000 7fe86aff5700 1 -- 172.21.15.31:0/1761684718 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56102abe1ce0 con 0x56102acff930 2023-07-29T16:29:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.006+0000 7fe86aff5700 10 client.19473 handle_command_reply: tid=0 2023-07-29T16:29:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.006+0000 7fe8a0e78b80 1 client.19473 shutdown 2023-07-29T16:29:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 msgr2=0x56102aca7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:08.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102aca7b50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fe878005c00 tx=0x7fe8780057d0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 shutdown_connections 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56102acff930 0x56102ad01d40 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56102ac98cf0 0x56102aca45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102aaf36b0 0x56102aca7b50 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe86aff5700 0 client.19473 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 --2- 172.21.15.31:0/1761684718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56102ac97df0 0x56102aca8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 >> 172.21.15.31:0/1761684718 conn(0x56102ac8cfd0 msgr2=0x56102ac9f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 shutdown_connections 2023-07-29T16:29:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 1 -- 172.21.15.31:0/1761684718 wait complete. 2023-07-29T16:29:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.007+0000 7fe8a0e78b80 20 client.19473 trim_cache size 0 max 16384 2023-07-29T16:29:08.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:29:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:29:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:29:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:29:08.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe8700a0790 msgr2=0x7fe8700a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe8700a0790 0x7fe8700a2c40 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe8800076a0 tx=0x7fe880009040 comp rx=0 tx=0).stop 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 msgr2=0x7fe888198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 0x7fe888198880 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe87c00e9f0 tx=0x7fe87c00ed00 comp rx=0 tx=0).stop 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 shutdown_connections 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe8700a0790 0x7fe8700a2c40 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8881081e0 0x7fe888199700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe888073750 0x7fe888198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 --2- 172.21.15.31:0/429484021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe888074d30 0x7fe888198880 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 >> 172.21.15.31:0/429484021 conn(0x7fe8880fb970 msgr2=0x7fe8880fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 shutdown_connections 2023-07-29T16:29:08.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:08.008+0000 7fe88d5ff700 1 -- 172.21.15.31:0/429484021 wait complete. 2023-07-29T16:29:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:08 smithi039 ceph-mon[128858]: pgmap v781: 97 pgs: 97 active+clean; 197 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.1 MiB/s wr, 260 op/s 2023-07-29T16:29:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:08 smithi156 ceph-mon[129245]: pgmap v781: 97 pgs: 97 active+clean; 197 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.1 MiB/s wr, 260 op/s 2023-07-29T16:29:08.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:08 smithi031 ceph-mon[95504]: pgmap v781: 97 pgs: 97 active+clean; 197 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.1 MiB/s wr, 260 op/s 2023-07-29T16:29:08.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1075709 8.88 MB/sec execute 533 sec latency 29.384 ms 2023-07-29T16:29:09.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1076863 8.88 MB/sec execute 534 sec latency 24.313 ms 2023-07-29T16:29:10.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1078910 8.88 MB/sec execute 535 sec latency 17.812 ms 2023-07-29T16:29:10.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:10 smithi031 ceph-mon[95504]: pgmap v782: 97 pgs: 97 active+clean; 198 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.6 MiB/s wr, 273 op/s 2023-07-29T16:29:11.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:10 smithi039 ceph-mon[128858]: pgmap v782: 97 pgs: 97 active+clean; 198 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.6 MiB/s wr, 273 op/s 2023-07-29T16:29:11.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:10 smithi156 ceph-mon[129245]: pgmap v782: 97 pgs: 97 active+clean; 198 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 2.6 MiB/s wr, 273 op/s 2023-07-29T16:29:11.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1080242 8.88 MB/sec execute 536 sec latency 37.653 ms 2023-07-29T16:29:12.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1081566 8.87 MB/sec execute 537 sec latency 17.730 ms 2023-07-29T16:29:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:12 smithi031 ceph-mon[95504]: pgmap v783: 97 pgs: 97 active+clean; 198 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.0 MiB/s wr, 207 op/s 2023-07-29T16:29:13.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:12 smithi039 ceph-mon[128858]: pgmap v783: 97 pgs: 97 active+clean; 198 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.0 MiB/s wr, 207 op/s 2023-07-29T16:29:13.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:12 smithi156 ceph-mon[129245]: pgmap v783: 97 pgs: 97 active+clean; 198 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 2.0 MiB/s wr, 207 op/s 2023-07-29T16:29:13.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1083574 8.88 MB/sec execute 538 sec latency 17.649 ms 2023-07-29T16:29:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:29:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:29:13.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:13 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:29:14.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:29:14.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:29:14.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:13 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:29:14.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:29:14.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:29:14.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:13 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:29:14.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1084608 8.87 MB/sec execute 539 sec latency 15.850 ms 2023-07-29T16:29:14.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:14 smithi031 ceph-mon[95504]: pgmap v784: 97 pgs: 97 active+clean; 199 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.4 MiB/s wr, 251 op/s 2023-07-29T16:29:15.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:14 smithi039 ceph-mon[128858]: pgmap v784: 97 pgs: 97 active+clean; 199 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.4 MiB/s wr, 251 op/s 2023-07-29T16:29:15.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:14 smithi156 ceph-mon[129245]: pgmap v784: 97 pgs: 97 active+clean; 199 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.4 MiB/s wr, 251 op/s 2023-07-29T16:29:15.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1086768 8.88 MB/sec execute 540 sec latency 16.456 ms 2023-07-29T16:29:16.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1087522 8.87 MB/sec execute 541 sec latency 96.599 ms 2023-07-29T16:29:16.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:16 smithi031 ceph-mon[95504]: pgmap v785: 97 pgs: 97 active+clean; 202 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 2.3 MiB/s wr, 258 op/s 2023-07-29T16:29:17.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:16 smithi039 ceph-mon[128858]: pgmap v785: 97 pgs: 97 active+clean; 202 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 2.3 MiB/s wr, 258 op/s 2023-07-29T16:29:17.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:16 smithi156 ceph-mon[129245]: pgmap v785: 97 pgs: 97 active+clean; 202 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 2.3 MiB/s wr, 258 op/s 2023-07-29T16:29:17.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1089362 8.87 MB/sec execute 542 sec latency 34.096 ms 2023-07-29T16:29:18.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:18 smithi039 ceph-mon[128858]: pgmap v786: 97 pgs: 97 active+clean; 201 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.1 MiB/s wr, 232 op/s 2023-07-29T16:29:18.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:18 smithi156 ceph-mon[129245]: pgmap v786: 97 pgs: 97 active+clean; 201 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.1 MiB/s wr, 232 op/s 2023-07-29T16:29:18.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:18 smithi031 ceph-mon[95504]: pgmap v786: 97 pgs: 97 active+clean; 201 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.1 MiB/s wr, 232 op/s 2023-07-29T16:29:18.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1090736 8.87 MB/sec execute 543 sec latency 54.264 ms 2023-07-29T16:29:19.903 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1091920 8.86 MB/sec execute 544 sec latency 13.390 ms 2023-07-29T16:29:20.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1093996 8.87 MB/sec execute 545 sec latency 14.532 ms 2023-07-29T16:29:20.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:20 smithi031 ceph-mon[95504]: pgmap v787: 97 pgs: 97 active+clean; 205 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.6 MiB/s wr, 274 op/s 2023-07-29T16:29:21.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:20 smithi039 ceph-mon[128858]: pgmap v787: 97 pgs: 97 active+clean; 205 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.6 MiB/s wr, 274 op/s 2023-07-29T16:29:21.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:20 smithi156 ceph-mon[129245]: pgmap v787: 97 pgs: 97 active+clean; 205 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 2.6 MiB/s wr, 274 op/s 2023-07-29T16:29:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.167+0000 7f7257fff700 1 -- 172.21.15.31:0/2417703041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 msgr2=0x7f7250108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.167+0000 7f7257fff700 1 --2- 172.21.15.31:0/2417703041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 0x7f7250108780 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f724800b0a0 tx=0x7f724800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.167+0000 7f7257fff700 1 -- 172.21.15.31:0/2417703041 shutdown_connections 2023-07-29T16:29:21.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.167+0000 7f7257fff700 1 --2- 172.21.15.31:0/2417703041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 0x7f7250108780 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.167+0000 7f7257fff700 1 --2- 172.21.15.31:0/2417703041 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7250100340 0x7f7250100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.167+0000 7f7257fff700 1 --2- 172.21.15.31:0/2417703041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.168+0000 7f7257fff700 1 -- 172.21.15.31:0/2417703041 >> 172.21.15.31:0/2417703041 conn(0x7f72500fb920 msgr2=0x7f72500fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.168+0000 7f7257fff700 1 -- 172.21.15.31:0/2417703041 shutdown_connections 2023-07-29T16:29:21.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.168+0000 7f7257fff700 1 -- 172.21.15.31:0/2417703041 wait complete. 2023-07-29T16:29:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 Processor -- start 2023-07-29T16:29:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 -- start start 2023-07-29T16:29:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7250100340 0x7f7250073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 0x7f7250073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725019fff0 con 0x7f7250100340 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72501a0130 con 0x7f7250101530 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.169+0000 7f7257fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72501a0270 con 0x7f7250101ec0 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7250100340 0x7f7250073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33854/0 (socket says 172.21.15.31:33854) 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 -- 172.21.15.31:0/1522347131 learned_addr learned my addr 172.21.15.31:0/1522347131 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725659c700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 0x7f7250073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 -- 172.21.15.31:0/1522347131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 msgr2=0x7f7250073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 0x7f7250073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 -- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7250100340 msgr2=0x7f7250073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7250100340 0x7f7250073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 -- 172.21.15.31:0/1522347131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7248009d00 con 0x7f7250101530 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.170+0000 7f725559a700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250073680 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f724000e9a0 tx=0x7f724000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.171+0000 7f7246ffd700 1 -- 172.21.15.31:0/1522347131 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240009790 con 0x7f7250101530 2023-07-29T16:29:21.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.171+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72501a0500 con 0x7f7250101530 2023-07-29T16:29:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.171+0000 7f7246ffd700 1 -- 172.21.15.31:0/1522347131 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7240010430 con 0x7f7250101530 2023-07-29T16:29:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.171+0000 7f7246ffd700 1 -- 172.21.15.31:0/1522347131 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240018680 con 0x7f7250101530 2023-07-29T16:29:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.171+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72501a0a10 con 0x7f7250101530 2023-07-29T16:29:21.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.174+0000 7f7246ffd700 1 -- 172.21.15.31:0/1522347131 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f72400105a0 con 0x7f7250101530 2023-07-29T16:29:21.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.176+0000 7f7246ffd700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f723c0a0790 0x7f723c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.176+0000 7f7246ffd700 1 -- 172.21.15.31:0/1522347131 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f7240014070 con 0x7f7250101530 2023-07-29T16:29:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.176+0000 7f726b9d7b80 1 Processor -- start 2023-07-29T16:29:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f7255d9b700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f723c0a0790 0x7f723c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 -- start start 2023-07-29T16:29:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27d006b0 0x560e27ea48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27ea57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 0x560e27ea6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27dfc9b0 con 0x560e27ea4df0 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27dfa370 con 0x560e27d006b0 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.177+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27df5c60 con 0x560e27ea5cf0 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 0x560e27ea6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 0x560e27ea6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42092/0 (socket says 172.21.15.31:42092) 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 -- 172.21.15.31:0/3420126724 learned_addr learned my addr 172.21.15.31:0/3420126724 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f7255d9b700 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27d006b0 0x560e27ea48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f7255d9b700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f723c0a0790 0x7f723c0a2c40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7250074a50 tx=0x7f724c007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 -- 172.21.15.31:0/3420126724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27d006b0 msgr2=0x560e27ea48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27d006b0 0x560e27ea48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 -- 172.21.15.31:0/3420126724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 msgr2=0x560e27ea57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27ea57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 -- 172.21.15.31:0/3420126724 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e27c23d80 con 0x560e27ea5cf0 2023-07-29T16:29:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.178+0000 7f725659c700 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 0x560e27ea6160 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f724800b0a0 tx=0x7f7248009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daffab3e228becb2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.179+0000 7f723bfff700 1 -- 172.21.15.31:0/3420126724 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724800e040 con 0x560e27ea5cf0 2023-07-29T16:29:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.179+0000 7f723bfff700 1 -- 172.21.15.31:0/3420126724 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f724800bdb0 con 0x560e27ea5cf0 2023-07-29T16:29:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.179+0000 7f723bfff700 1 -- 172.21.15.31:0/3420126724 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724801c800 con 0x560e27ea5cf0 2023-07-29T16:29:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.179+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3420126724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 msgr2=0x560e27ea6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.179+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 0x560e27ea6160 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f724800b0a0 tx=0x7f7248009500 comp rx=0 tx=0).stop 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3420126724 shutdown_connections 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27ea5cf0 0x560e27ea6160 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27d006b0 0x560e27ea48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3420126724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27ea57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3420126724 >> 172.21.15.31:0/3420126724 conn(0x560e27e99fd0 msgr2=0x560e27e9f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3420126724 shutdown_connections 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3420126724 wait complete. 2023-07-29T16:29:21.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.180+0000 7f726b9d7b80 1 -- start start 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 Processor -- start 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eadf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27eae480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 0x560e27eb54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27d924b0 con 0x560e27ea4df0 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27d92150 con 0x560e27ea5cf0 2023-07-29T16:29:21.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27c798c0 con 0x560e27d006b0 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f726b9d7b80 10 client.0 _ll_register_callbacks cb 0x560e27d9b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eadf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f725659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 0x560e27eb54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eadf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42094/0 (socket says 172.21.15.31:42094) 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f725659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 0x560e27eb54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33878/0 (socket says 172.21.15.31:33878) 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f7255d9b700 1 -- 172.21.15.31:0/1040612472 learned_addr learned my addr 172.21.15.31:0/1040612472 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.181+0000 7f725559a700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27eae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f725659c700 1 -- 172.21.15.31:0/1040612472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 msgr2=0x560e27eadf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f725659c700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eadf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f725659c700 1 -- 172.21.15.31:0/1040612472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 msgr2=0x560e27eae480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f725659c700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27eae480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f725659c700 1 -- 172.21.15.31:0/1040612472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72501090b0 con 0x560e27ea5cf0 2023-07-29T16:29:21.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f7255d9b700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eadf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:21.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f725659c700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 0x560e27eb54c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7248007ab0 tx=0x7f724800b020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a51def7b8c3622c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:21.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f72397fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:21.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f72397fa700 1 -- 172.21.15.31:0/1040612472 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724800e070 con 0x560e27ea5cf0 2023-07-29T16:29:21.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560e27e20cb0 con 0x560e27ea5cf0 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.182+0000 7f72397fa700 1 -- 172.21.15.31:0/1040612472 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f724801c750 con 0x560e27ea5cf0 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f72397fa700 1 -- 172.21.15.31:0/1040612472 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724801c910 con 0x560e27ea5cf0 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 10 client.29603 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560e27c0cc00 con 0x560e27ea5cf0 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f72397fa700 1 -- 172.21.15.31:0/1040612472 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560e27e20cb0 con 0x560e27ea5cf0 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f72397fa700 1 -- 172.21.15.31:0/1040612472 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f7248019030 con 0x560e27ea5cf0 2023-07-29T16:29:21.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 10 client.29603 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 10 client.29603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 20 client.29603 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27eb97d0 0x560e27ebbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 4 client.29603 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.183+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560e27e73d10 con 0x560e27eb97d0 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.184+0000 7f7255d9b700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27eb97d0 0x560e27ebbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.184+0000 7f7255d9b700 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27eb97d0 0x560e27ebbbe0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1252bc0f22265dc8 server_cookie=8e61722585968cee in_seq=0 out_seq=0 2023-07-29T16:29:21.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.184+0000 7f72397fa700 10 client.29603 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f72397fa700 1 -- 172.21.15.31:0/1040612472 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560e27e73d10 con 0x560e27eb97d0 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f72397fa700 10 client.29603 handle_command_reply: tid=0 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 client.29603 shutdown 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 msgr2=0x560e27eb54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 0x560e27eb54c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7248007ab0 tx=0x7f724800b020 comp rx=0 tx=0).stop 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 shutdown_connections 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27eb97d0 0x560e27ebbbe0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea5cf0 0x560e27eb54c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eadf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/1040612472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea4df0 0x560e27eae480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 >> 172.21.15.31:0/1040612472 conn(0x560e27e99fd0 msgr2=0x560e27e9ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f72397fa700 0 client.29603 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 shutdown_connections 2023-07-29T16:29:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.185+0000 7f726b9d7b80 1 -- 172.21.15.31:0/1040612472 wait complete. 2023-07-29T16:29:21.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.186+0000 7f726b9d7b80 20 client.29603 trim_cache size 0 max 16384 2023-07-29T16:29:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 Processor -- start 2023-07-29T16:29:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 -- start start 2023-07-29T16:29:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27e9ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27ebb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27eadab0 con 0x560e27ea5cf0 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27ea0df0 con 0x560e27ea4df0 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27eaf0c0 con 0x560e27d006b0 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.202+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27e9ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27e9ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42100/0 (socket says 172.21.15.31:42100) 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f7255d9b700 1 -- 172.21.15.31:0/3596002931 learned_addr learned my addr 172.21.15.31:0/3596002931 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725659c700 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27ebb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 -- 172.21.15.31:0/3596002931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 msgr2=0x560e27e9ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27e9ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 -- 172.21.15.31:0/3596002931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 msgr2=0x560e27ebb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27ebb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 -- 172.21.15.31:0/3596002931 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e27c4a1f0 con 0x560e27ea4df0 2023-07-29T16:29:21.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f725559a700 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb5480 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7240099380 tx=0x7f72400cb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9533ff5d3cb70e1e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f7244ff9700 1 -- 172.21.15.31:0/3596002931 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724009d420 con 0x560e27ea4df0 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f7244ff9700 1 -- 172.21.15.31:0/3596002931 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72400a1840 con 0x560e27ea4df0 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.203+0000 7f7244ff9700 1 -- 172.21.15.31:0/3596002931 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7240098b40 con 0x560e27ea4df0 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3596002931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 msgr2=0x560e27eb5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb5480 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7240099380 tx=0x7f72400cb040 comp rx=0 tx=0).stop 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3596002931 shutdown_connections 2023-07-29T16:29:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27ebb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27e9ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3596002931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb5480 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3596002931 >> 172.21.15.31:0/3596002931 conn(0x560e27e99fd0 msgr2=0x560e27eacc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3596002931 shutdown_connections 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.204+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3596002931 wait complete. 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 -- start start 2023-07-29T16:29:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 Processor -- start 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eb16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27eb4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27eb4e70 con 0x560e27ea5cf0 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27eb4fe0 con 0x560e27ea4df0 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e27eb51a0 con 0x560e27d006b0 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.205+0000 7f726b9d7b80 10 client.0 _ll_register_callbacks cb 0x560e27d9b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eb16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f725659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27eb4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f725559a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eb16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42110/0 (socket says 172.21.15.31:42110) 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 -- 172.21.15.31:0/3680925121 learned_addr learned my addr 172.21.15.31:0/3680925121 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 -- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 msgr2=0x560e27eb1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 -- 172.21.15.31:0/3680925121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 msgr2=0x560e27eb4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27eb4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 -- 172.21.15.31:0/3680925121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72501a0a10 con 0x560e27d006b0 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f725559a700 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f7255d9b700 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eb16b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f724c007610 tx=0x7f724c013070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4056f6bb99668d4b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f72397fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.206+0000 7f72397fa700 1 -- 172.21.15.31:0/3680925121 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c024040 con 0x560e27d006b0 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f724800bdb0 con 0x560e27d006b0 2023-07-29T16:29:21.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f72397fa700 1 -- 172.21.15.31:0/3680925121 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f724c017070 con 0x560e27d006b0 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f72397fa700 1 -- 172.21.15.31:0/3680925121 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724c004230 con 0x560e27d006b0 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 10 client.29605 fetch_fsmap learned FSMap version 33 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x560e27c22250 con 0x560e27d006b0 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f72397fa700 1 -- 172.21.15.31:0/3680925121 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f724800bdb0 con 0x560e27d006b0 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f72397fa700 1 -- 172.21.15.31:0/3680925121 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f724c011440 con 0x560e27d006b0 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 10 client.29605 fetch_fsmap finished waiting for FSMap version 33 2023-07-29T16:29:21.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 10 client.29605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 20 client.29605 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27f0c4e0 0x560e27f0e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 4 client.29605 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:29:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.207+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560e27deece0 con 0x560e27f0c4e0 2023-07-29T16:29:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.208+0000 7f725559a700 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27f0c4e0 0x560e27f0e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.208+0000 7f725559a700 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27f0c4e0 0x560e27f0e8f0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91097b1646bc7852 server_cookie=d031b5ef9e7b8bc6 in_seq=0 out_seq=0 2023-07-29T16:29:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.208+0000 7f72397fa700 10 client.29605 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.285+0000 7f72397fa700 1 -- 172.21.15.31:0/3680925121 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560e27deece0 con 0x560e27f0c4e0 2023-07-29T16:29:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.285+0000 7f72397fa700 10 client.29605 handle_command_reply: tid=0 2023-07-29T16:29:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.285+0000 7f726b9d7b80 1 client.29605 shutdown 2023-07-29T16:29:21.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 msgr2=0x560e27eb16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eb16b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f724c007610 tx=0x7f724c013070 comp rx=0 tx=0).stop 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 shutdown_connections 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x560e27f0c4e0 0x560e27f0e8f0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560e27ea5cf0 0x560e27eb4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e27d006b0 0x560e27eb16b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 --2- 172.21.15.31:0/3680925121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e27ea4df0 0x560e27eb1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f72397fa700 0 client.29605 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:21.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 >> 172.21.15.31:0/3680925121 conn(0x560e27e99fd0 msgr2=0x560e27ebb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 shutdown_connections 2023-07-29T16:29:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 1 -- 172.21.15.31:0/3680925121 wait complete. 2023-07-29T16:29:21.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.286+0000 7f726b9d7b80 20 client.29605 trim_cache size 0 max 16384 2023-07-29T16:29:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:29:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:29:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:29:21.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:29:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f723c0a0790 msgr2=0x7f723c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f723c0a0790 0x7f723c0a2c40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7250074a50 tx=0x7f724c007400 comp rx=0 tx=0).stop 2023-07-29T16:29:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 msgr2=0x7f7250073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250073680 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f724000e9a0 tx=0x7f724000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 shutdown_connections 2023-07-29T16:29:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f723c0a0790 0x7f723c0a2c40 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7250101ec0 0x7f7250073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7250100340 0x7f7250073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 --2- 172.21.15.31:0/1522347131 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7250101530 0x7f7250073680 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:21.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 >> 172.21.15.31:0/1522347131 conn(0x7f72500fb920 msgr2=0x7f7250104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 shutdown_connections 2023-07-29T16:29:21.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:21.287+0000 7f7257fff700 1 -- 172.21.15.31:0/1522347131 wait complete. 2023-07-29T16:29:21.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1094775 8.86 MB/sec execute 546 sec latency 65.185 ms 2023-07-29T16:29:22.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1096483 8.86 MB/sec execute 547 sec latency 42.301 ms 2023-07-29T16:29:22.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:22 smithi031 ceph-mon[95504]: pgmap v788: 97 pgs: 97 active+clean; 205 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.8 MiB/s wr, 200 op/s 2023-07-29T16:29:23.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:22 smithi039 ceph-mon[128858]: pgmap v788: 97 pgs: 97 active+clean; 205 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.8 MiB/s wr, 200 op/s 2023-07-29T16:29:23.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:22 smithi156 ceph-mon[129245]: pgmap v788: 97 pgs: 97 active+clean; 205 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.8 MiB/s wr, 200 op/s 2023-07-29T16:29:23.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1097854 8.86 MB/sec execute 548 sec latency 39.569 ms 2023-07-29T16:29:24.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:24 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:29:24.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:24 smithi156 ceph-mon[129245]: pgmap v789: 97 pgs: 97 active+clean; 201 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.2 MiB/s wr, 247 op/s 2023-07-29T16:29:24.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:24 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:29:24.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:24 smithi031 ceph-mon[95504]: pgmap v789: 97 pgs: 97 active+clean; 201 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.2 MiB/s wr, 247 op/s 2023-07-29T16:29:24.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:24 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:29:24.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:24 smithi039 ceph-mon[128858]: pgmap v789: 97 pgs: 97 active+clean; 201 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 2.2 MiB/s wr, 247 op/s 2023-07-29T16:29:24.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1099072 8.85 MB/sec execute 549 sec latency 15.059 ms 2023-07-29T16:29:25.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1101109 8.86 MB/sec execute 550 sec latency 14.010 ms 2023-07-29T16:29:26.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1101951 8.85 MB/sec execute 551 sec latency 14.870 ms 2023-07-29T16:29:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:26 smithi031 ceph-mon[95504]: pgmap v790: 97 pgs: 97 active+clean; 202 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.3 MiB/s wr, 270 op/s 2023-07-29T16:29:27.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:26 smithi039 ceph-mon[128858]: pgmap v790: 97 pgs: 97 active+clean; 202 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.3 MiB/s wr, 270 op/s 2023-07-29T16:29:27.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:26 smithi156 ceph-mon[129245]: pgmap v790: 97 pgs: 97 active+clean; 202 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 2.3 MiB/s wr, 270 op/s 2023-07-29T16:29:27.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1103745 8.85 MB/sec execute 552 sec latency 34.405 ms 2023-07-29T16:29:28.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:28 smithi039 ceph-mon[128858]: pgmap v791: 97 pgs: 97 active+clean; 204 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2023-07-29T16:29:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:28 smithi156 ceph-mon[129245]: pgmap v791: 97 pgs: 97 active+clean; 204 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2023-07-29T16:29:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:28 smithi031 ceph-mon[95504]: pgmap v791: 97 pgs: 97 active+clean; 204 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2023-07-29T16:29:28.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1105072 8.85 MB/sec execute 553 sec latency 20.008 ms 2023-07-29T16:29:29.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1106673 8.84 MB/sec execute 554 sec latency 14.900 ms 2023-07-29T16:29:30.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1108487 8.85 MB/sec execute 555 sec latency 13.686 ms 2023-07-29T16:29:30.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:30 smithi031 ceph-mon[95504]: pgmap v792: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.5 MiB/s wr, 274 op/s 2023-07-29T16:29:31.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:30 smithi039 ceph-mon[128858]: pgmap v792: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.5 MiB/s wr, 274 op/s 2023-07-29T16:29:31.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:30 smithi156 ceph-mon[129245]: pgmap v792: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.5 MiB/s wr, 274 op/s 2023-07-29T16:29:31.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1109741 8.85 MB/sec execute 556 sec latency 16.889 ms 2023-07-29T16:29:32.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1111599 8.85 MB/sec execute 557 sec latency 45.736 ms 2023-07-29T16:29:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:32 smithi031 ceph-mon[95504]: pgmap v793: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2023-07-29T16:29:33.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:32 smithi039 ceph-mon[128858]: pgmap v793: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2023-07-29T16:29:33.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:32 smithi156 ceph-mon[129245]: pgmap v793: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 1.8 MiB/s wr, 192 op/s 2023-07-29T16:29:33.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1112597 8.84 MB/sec execute 558 sec latency 14.226 ms 2023-07-29T16:29:34.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3237310854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68100350 msgr2=0x7f2f68100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3237310854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68100350 0x7f2f68100760 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2f58009990 tx=0x7f2f58009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3237310854 shutdown_connections 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3237310854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f68101ed0 0x7f2f68108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3237310854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68100350 0x7f2f68100760 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3237310854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68101540 0x7f2f68101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3237310854 >> 172.21.15.31:0/3237310854 conn(0x7f2f680fb970 msgr2=0x7f2f680fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:34.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3237310854 shutdown_connections 2023-07-29T16:29:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.456+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3237310854 wait complete. 2023-07-29T16:29:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.457+0000 7f2f6cbff700 1 Processor -- start 2023-07-29T16:29:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 -- start start 2023-07-29T16:29:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68100350 0x7f2f681983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68101540 0x7f2f68198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f68101ed0 0x7f2f681997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68199e00 con 0x7f2f68101ed0 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f68199f70 con 0x7f2f68101540 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6cbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6819a130 con 0x7f2f68100350 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68100350 0x7f2f681983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f5ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68101540 0x7f2f68198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68100350 0x7f2f681983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54902/0 (socket says 172.21.15.31:54902) 2023-07-29T16:29:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f6659c700 1 -- 172.21.15.31:0/3239709860 learned_addr learned my addr 172.21.15.31:0/3239709860 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.458+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f68101ed0 0x7f2f681997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5ffff700 1 -- 172.21.15.31:0/3239709860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68100350 msgr2=0x7f2f681983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68100350 0x7f2f681983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5ffff700 1 -- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f68101ed0 msgr2=0x7f2f681997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f68101ed0 0x7f2f681997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5ffff700 1 -- 172.21.15.31:0/3239709860 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50009520 con 0x7f2f68101540 2023-07-29T16:29:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68101540 0x7f2f68198920 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000e9f0 tx=0x7f2f5000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5f7fe700 1 -- 172.21.15.31:0/3239709860 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5000cb70 con 0x7f2f68101540 2023-07-29T16:29:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f580095f0 con 0x7f2f68101540 2023-07-29T16:29:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5f7fe700 1 -- 172.21.15.31:0/3239709860 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f5000ee90 con 0x7f2f68101540 2023-07-29T16:29:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f5f7fe700 1 -- 172.21.15.31:0/3239709860 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50003cf0 con 0x7f2f68101540 2023-07-29T16:29:34.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.459+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f681a09f0 con 0x7f2f68101540 2023-07-29T16:29:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.463+0000 7f2f5f7fe700 1 -- 172.21.15.31:0/3239709860 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2f5000ccd0 con 0x7f2f68101540 2023-07-29T16:29:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.464+0000 7f2f804c1b80 1 Processor -- start 2023-07-29T16:29:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.464+0000 7f2f804c1b80 1 -- start start 2023-07-29T16:29:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.464+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651363566b0 0x5651364fa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.471 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x5651364fb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fbcf0 0x5651364fc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651364529b0 con 0x5651364fbcf0 2023-07-29T16:29:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136450370 con 0x5651363566b0 2023-07-29T16:29:34.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56513644bc60 con 0x5651364fadf0 2023-07-29T16:29:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f5f7fe700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2f540a0790 0x7f2f540a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x5651364fb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x5651364fb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54910/0 (socket says 172.21.15.31:54910) 2023-07-29T16:29:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 -- 172.21.15.31:0/1018566260 learned_addr learned my addr 172.21.15.31:0/1018566260 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651363566b0 0x5651364fa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f5f7fe700 1 -- 172.21.15.31:0/3239709860 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2f50014070 con 0x7f2f68101540 2023-07-29T16:29:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fbcf0 0x5651364fc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2f540a0790 0x7f2f540a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 -- 172.21.15.31:0/1018566260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651363566b0 msgr2=0x5651364fa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651363566b0 0x5651364fa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 -- 172.21.15.31:0/1018566260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fbcf0 msgr2=0x5651364fc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fbcf0 0x5651364fc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.465+0000 7f2f6659c700 1 -- 172.21.15.31:0/1018566260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565136279d80 con 0x5651364fadf0 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.466+0000 7f2f6659c700 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x5651364fb7b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2f580050f0 tx=0x7f2f58005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56c99d4dc1c7c22b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.466+0000 7f2f5cff9700 1 -- 172.21.15.31:0/1018566260 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5801d070 con 0x5651364fadf0 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.466+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2f540a0790 0x7f2f540a2c40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2f60007820 tx=0x7f2f60009130 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.466+0000 7f2f5cff9700 1 -- 172.21.15.31:0/1018566260 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f5801e040 con 0x5651364fadf0 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.466+0000 7f2f5cff9700 1 -- 172.21.15.31:0/1018566260 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58004af0 con 0x5651364fadf0 2023-07-29T16:29:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1018566260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 msgr2=0x5651364fb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x5651364fb7b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2f580050f0 tx=0x7f2f58005c90 comp rx=0 tx=0).stop 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1018566260 shutdown_connections 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fbcf0 0x5651364fc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651363566b0 0x5651364fa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1018566260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x5651364fb7b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1018566260 >> 172.21.15.31:0/1018566260 conn(0x5651364effd0 msgr2=0x5651364f5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1018566260 shutdown_connections 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.467+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1018566260 wait complete. 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.468+0000 7f2f804c1b80 1 -- start start 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.468+0000 7f2f804c1b80 1 Processor -- start 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.468+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 0x565136503e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fadf0 0x565136504360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x56513650b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651363e84b0 con 0x5651364fadf0 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651363e8150 con 0x5651364fbcf0 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651362cf8c0 con 0x5651363566b0 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f804c1b80 10 client.0 _ll_register_callbacks cb 0x5651363f15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 0x565136503e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 0x565136503e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54924/0 (socket says 172.21.15.31:54924) 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 -- 172.21.15.31:0/1568114363 learned_addr learned my addr 172.21.15.31:0/1568114363 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fadf0 0x565136504360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 -- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 msgr2=0x56513650b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x56513650b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 -- 172.21.15.31:0/1568114363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fadf0 msgr2=0x565136504360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fadf0 0x565136504360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 -- 172.21.15.31:0/1568114363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f681090c0 con 0x5651363566b0 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fadf0 0x565136504360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f6659c700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 0x565136503e20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2f58005a60 tx=0x7f2f58004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60ca68bb8e1ceae server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.469+0000 7f2f4dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f4dffb700 1 -- 172.21.15.31:0/1568114363 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5801d030 con 0x5651363566b0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f4dffb700 1 -- 172.21.15.31:0/1568114363 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2f58004c40 con 0x5651363566b0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565136476cb0 con 0x5651363566b0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f4dffb700 1 -- 172.21.15.31:0/1568114363 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f5800bd00 con 0x5651363566b0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f804c1b80 10 client.29617 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x565136262c00 con 0x5651363566b0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f4dffb700 1 -- 172.21.15.31:0/1568114363 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565136476cb0 con 0x5651363566b0 2023-07-29T16:29:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.470+0000 7f2f4dffb700 1 -- 172.21.15.31:0/1568114363 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2f5802c490 con 0x5651363566b0 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f804c1b80 10 client.29617 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f804c1b80 10 client.29617 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f804c1b80 20 client.29617 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56513650f720 0x565136511b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f804c1b80 4 client.29617 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651364c9d10 con 0x56513650f720 2023-07-29T16:29:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56513650f720 0x565136511b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f5ffff700 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56513650f720 0x565136511b00 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aec7af4db470155f server_cookie=f707f3ce7a12f4b8 in_seq=0 out_seq=0 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.471+0000 7f2f4dffb700 10 client.29617 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.472+0000 7f2f4dffb700 1 -- 172.21.15.31:0/1568114363 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5651364c9d10 con 0x56513650f720 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.472+0000 7f2f4dffb700 10 client.29617 handle_command_reply: tid=0 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.472+0000 7f2f804c1b80 1 client.29617 shutdown 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 msgr2=0x565136503e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 0x565136503e20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f2f58005a60 tx=0x7f2f58004570 comp rx=0 tx=0).stop 2023-07-29T16:29:34.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 shutdown_connections 2023-07-29T16:29:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56513650f720 0x565136511b00 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x56513650b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651363566b0 0x565136503e20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/1568114363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651364fadf0 0x565136504360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f4dffb700 0 client.29617 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 >> 172.21.15.31:0/1568114363 conn(0x5651364effd0 msgr2=0x5651364f52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 shutdown_connections 2023-07-29T16:29:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 1 -- 172.21.15.31:0/1568114363 wait complete. 2023-07-29T16:29:34.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.473+0000 7f2f804c1b80 20 client.29617 trim_cache size 0 max 16384 2023-07-29T16:29:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 Processor -- start 2023-07-29T16:29:34.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 -- start start 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x565136509350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136502bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x5651364fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651364f5fb0 con 0x5651363566b0 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136503120 con 0x5651364fbcf0 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.491+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651364f6bb0 con 0x5651364fadf0 2023-07-29T16:29:34.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f5ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136502bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x565136509350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f5ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136502bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54934/0 (socket says 172.21.15.31:54934) 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f5ffff700 1 -- 172.21.15.31:0/486315550 learned_addr learned my addr 172.21.15.31:0/486315550 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x5651364fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 -- 172.21.15.31:0/486315550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 msgr2=0x565136502bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136502bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 -- 172.21.15.31:0/486315550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 msgr2=0x5651364fc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x5651364fc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 -- 172.21.15.31:0/486315550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5651362a01f0 con 0x5651363566b0 2023-07-29T16:29:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x5651364fc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.492+0000 7f2f6659c700 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x565136509350 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2f5802c8a0 tx=0x7f2f58005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8506cb5eee45f74 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.493+0000 7f2f5d7fa700 1 -- 172.21.15.31:0/486315550 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5801d070 con 0x5651363566b0 2023-07-29T16:29:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.493+0000 7f2f5d7fa700 1 -- 172.21.15.31:0/486315550 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f58005c40 con 0x5651363566b0 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.493+0000 7f2f5d7fa700 1 -- 172.21.15.31:0/486315550 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5800f6a0 con 0x5651363566b0 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.493+0000 7f2f804c1b80 1 -- 172.21.15.31:0/486315550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 msgr2=0x565136509350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.493+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x565136509350 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2f5802c8a0 tx=0x7f2f58005fd0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 -- 172.21.15.31:0/486315550 shutdown_connections 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x5651364fc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x565136509350 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/486315550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136502bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 -- 172.21.15.31:0/486315550 >> 172.21.15.31:0/486315550 conn(0x5651364effd0 msgr2=0x5651364feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 -- 172.21.15.31:0/486315550 shutdown_connections 2023-07-29T16:29:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 -- 172.21.15.31:0/486315550 wait complete. 2023-07-29T16:29:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 -- start start 2023-07-29T16:29:34.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.494+0000 7f2f804c1b80 1 Processor -- start 2023-07-29T16:29:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x5651365074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136507a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x565136508860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136508da0 con 0x5651363566b0 2023-07-29T16:29:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x565136508f10 con 0x5651364fbcf0 2023-07-29T16:29:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651365090d0 con 0x5651364fadf0 2023-07-29T16:29:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f804c1b80 10 client.0 _ll_register_callbacks cb 0x5651363f15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f6659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x5651365074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f66d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x565136508860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f66d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x565136508860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39406/0 (socket says 172.21.15.31:39406) 2023-07-29T16:29:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f5ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136507a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.495+0000 7f2f66d9d700 1 -- 172.21.15.31:0/719394462 learned_addr learned my addr 172.21.15.31:0/719394462 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f66d9d700 1 -- 172.21.15.31:0/719394462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 msgr2=0x565136507a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136507a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f66d9d700 1 -- 172.21.15.31:0/719394462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 msgr2=0x5651365074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x5651365074f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f66d9d700 1 -- 172.21.15.31:0/719394462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f60011040 con 0x5651364fbcf0 2023-07-29T16:29:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f66d9d700 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x565136508860 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2f600196f0 tx=0x7f2f60009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907c0ebd3f699b07 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f4dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f4dffb700 1 -- 172.21.15.31:0/719394462 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f60004bb0 con 0x5651364fbcf0 2023-07-29T16:29:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f4dffb700 1 -- 172.21.15.31:0/719394462 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2f60004d10 con 0x5651364fbcf0 2023-07-29T16:29:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.496+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565136444ce0 con 0x5651364fbcf0 2023-07-29T16:29:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f4dffb700 1 -- 172.21.15.31:0/719394462 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f60025600 con 0x5651364fbcf0 2023-07-29T16:29:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 10 client.29624 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x565136262c00 con 0x5651364fbcf0 2023-07-29T16:29:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f4dffb700 1 -- 172.21.15.31:0/719394462 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565136444ce0 con 0x5651364fbcf0 2023-07-29T16:29:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f4dffb700 1 -- 172.21.15.31:0/719394462 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2f60024070 con 0x5651364fbcf0 2023-07-29T16:29:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 10 client.29624 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:34.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 10 client.29624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 20 client.29624 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651365625b0 0x5651365649c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 4 client.29624 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:29:34.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x565136444a00 con 0x5651365625b0 2023-07-29T16:29:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.497+0000 7f2f6659c700 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651365625b0 0x5651365649c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:34.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.498+0000 7f2f6659c700 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651365625b0 0x5651365649c0 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcea55d645ad0699 server_cookie=44c49085f6c7a5df in_seq=0 out_seq=0 2023-07-29T16:29:34.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.498+0000 7f2f4dffb700 10 client.29624 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.582+0000 7f2f4dffb700 1 -- 172.21.15.31:0/719394462 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x565136444a00 con 0x5651365625b0 2023-07-29T16:29:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.582+0000 7f2f4dffb700 10 client.29624 handle_command_reply: tid=0 2023-07-29T16:29:34.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.583+0000 7f2f804c1b80 1 client.29624 shutdown 2023-07-29T16:29:34.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.585+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 msgr2=0x565136508860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.585+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x565136508860 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2f600196f0 tx=0x7f2f60009bf0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 shutdown_connections 2023-07-29T16:29:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5651365625b0 0x5651365649c0 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5651364fbcf0 0x565136508860 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5651363566b0 0x5651365074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f804c1b80 1 --2- 172.21.15.31:0/719394462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5651364fadf0 0x565136507a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 >> 172.21.15.31:0/719394462 conn(0x5651364effd0 msgr2=0x5651364feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:34.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.586+0000 7f2f4dffb700 0 client.29624 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.587+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 shutdown_connections 2023-07-29T16:29:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.587+0000 7f2f804c1b80 1 -- 172.21.15.31:0/719394462 wait complete. 2023-07-29T16:29:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.588+0000 7f2f804c1b80 20 client.29624 trim_cache size 0 max 16384 2023-07-29T16:29:34.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:29:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:29:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:29:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:29:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.590+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2f540a0790 msgr2=0x7f2f540a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.590+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2f540a0790 0x7f2f540a2c40 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2f60007820 tx=0x7f2f60009130 comp rx=0 tx=0).stop 2023-07-29T16:29:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.590+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68101540 msgr2=0x7f2f68198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.590+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68101540 0x7f2f68198920 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000e9f0 tx=0x7f2f5000ed00 comp rx=0 tx=0).stop 2023-07-29T16:29:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 shutdown_connections 2023-07-29T16:29:34.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2f540a0790 0x7f2f540a2c40 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2f68101ed0 0x7f2f681997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2f68100350 0x7f2f681983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 --2- 172.21.15.31:0/3239709860 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2f68101540 0x7f2f68198920 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 >> 172.21.15.31:0/3239709860 conn(0x7f2f680fb970 msgr2=0x7f2f680fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 shutdown_connections 2023-07-29T16:29:34.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:34.591+0000 7f2f6cbff700 1 -- 172.21.15.31:0/3239709860 wait complete. 2023-07-29T16:29:34.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1114542 8.84 MB/sec execute 559 sec latency 72.559 ms 2023-07-29T16:29:34.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:34 smithi031 ceph-mon[95504]: pgmap v794: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.5 MiB/s wr, 263 op/s 2023-07-29T16:29:35.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:34 smithi039 ceph-mon[128858]: pgmap v794: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.5 MiB/s wr, 263 op/s 2023-07-29T16:29:35.064 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:29:35.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:34 smithi156 ceph-mon[129245]: pgmap v794: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.5 MiB/s wr, 263 op/s 2023-07-29T16:29:35.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.280+0000 7f1d689ff700 1 -- 172.21.15.31:0/2682627586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 msgr2=0x7f1d640731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.280+0000 7f1d689ff700 1 --2- 172.21.15.31:0/2682627586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 0x7f1d640731e0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1d54009990 tx=0x7f1d54009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.281 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 -- 172.21.15.31:0/2682627586 shutdown_connections 2023-07-29T16:29:35.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 --2- 172.21.15.31:0/2682627586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d6410c900 0x7f1d6410ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 --2- 172.21.15.31:0/2682627586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 0x7f1d640731e0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 --2- 172.21.15.31:0/2682627586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d640737b0 0x7f1d64073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 -- 172.21.15.31:0/2682627586 >> 172.21.15.31:0/2682627586 conn(0x7f1d640fb970 msgr2=0x7f1d640fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 -- 172.21.15.31:0/2682627586 shutdown_connections 2023-07-29T16:29:35.282 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.281+0000 7f1d689ff700 1 -- 172.21.15.31:0/2682627586 wait complete. 2023-07-29T16:29:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 Processor -- start 2023-07-29T16:29:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 -- start start 2023-07-29T16:29:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 0x7f1d6419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 0x7f1d6419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.284 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d6410c900 0x7f1d641a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d641a42f0 con 0x7f1d6410c900 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d641a4430 con 0x7f1d64074d90 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d689ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d641a45c0 con 0x7f1d640737b0 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 0x7f1d6419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 0x7f1d6419c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54946/0 (socket says 172.21.15.31:54946) 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.282+0000 7f1d6259c700 1 -- 172.21.15.31:0/1529593759 learned_addr learned my addr 172.21.15.31:0/1529593759 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:35.285 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 0x7f1d6419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d6410c900 0x7f1d641a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d6259c700 1 -- 172.21.15.31:0/1529593759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 msgr2=0x7f1d6419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d6259c700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 0x7f1d6419ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d6259c700 1 -- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d6410c900 msgr2=0x7f1d641a3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d6259c700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d6410c900 0x7f1d641a3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d6259c700 1 -- 172.21.15.31:0/1529593759 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c009c40 con 0x7f1d640737b0 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d6259c700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 0x7f1d6419c770 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1d54005720 tx=0x7f1d54009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.286 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d537fe700 1 -- 172.21.15.31:0/1529593759 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5401d070 con 0x7f1d640737b0 2023-07-29T16:29:35.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d537fe700 1 -- 172.21.15.31:0/1529593759 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d54021410 con 0x7f1d640737b0 2023-07-29T16:29:35.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d537fe700 1 -- 172.21.15.31:0/1529593759 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54017650 con 0x7f1d640737b0 2023-07-29T16:29:35.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.283+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d540095f0 con 0x7f1d640737b0 2023-07-29T16:29:35.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.284+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d641a4a80 con 0x7f1d640737b0 2023-07-29T16:29:35.287 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.287+0000 7f1d537fe700 1 -- 172.21.15.31:0/1529593759 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1d54021580 con 0x7f1d640737b0 2023-07-29T16:29:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.288+0000 7f1d7c383b80 1 Processor -- start 2023-07-29T16:29:35.288 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.288+0000 7f1d7c383b80 1 -- start start 2023-07-29T16:29:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.288+0000 7f1d537fe700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d4c0a0810 0x7f1d4c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.288+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d5be0 0x561bfd9d5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.288+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9d6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9d7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd92da10 con 0x561bfd9d5be0 2023-07-29T16:29:35.289 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd92b3d0 con 0x561bfd9d72f0 2023-07-29T16:29:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd926cc0 con 0x561bfd9d6530 2023-07-29T16:29:35.290 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d537fe700 1 -- 172.21.15.31:0/1529593759 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1d540cccd0 con 0x7f1d640737b0 2023-07-29T16:29:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d5be0 0x561bfd9d5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d62d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9d7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d61d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9d6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.291 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d5be0 0x561bfd9d5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42042/0 (socket says 172.21.15.31:42042) 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d62d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9d7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39438/0 (socket says 172.21.15.31:39438) 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d6259c700 1 -- 172.21.15.31:0/6258459 learned_addr learned my addr 172.21.15.31:0/6258459 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d4c0a0810 0x7f1d4c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d61d9b700 1 -- 172.21.15.31:0/6258459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 msgr2=0x561bfd9d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9d7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d61d9b700 1 -- 172.21.15.31:0/6258459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d5be0 msgr2=0x561bfd9d5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.292 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d5be0 0x561bfd9d5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.289+0000 7f1d61d9b700 1 -- 172.21.15.31:0/6258459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bfd754de0 con 0x561bfd9d6530 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.290+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d4c0a0810 0x7f1d4c0a2cc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c012f50 tx=0x7f1d5c012280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.290+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9d6d60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009c80 tx=0x7f1d58009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bd108dd61f73ce5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.290+0000 7f1d50ff9700 1 -- 172.21.15.31:0/6258459 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58011cb0 con 0x561bfd9d6530 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.290+0000 7f1d50ff9700 1 -- 172.21.15.31:0/6258459 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d58011e10 con 0x561bfd9d6530 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.290+0000 7f1d50ff9700 1 -- 172.21.15.31:0/6258459 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58005280 con 0x561bfd9d6530 2023-07-29T16:29:35.293 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 -- 172.21.15.31:0/6258459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 msgr2=0x561bfd9d6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9d6d60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009c80 tx=0x7f1d58009f90 comp rx=0 tx=0).stop 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 -- 172.21.15.31:0/6258459 shutdown_connections 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9d7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d5be0 0x561bfd9d5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/6258459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9d6d60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 -- 172.21.15.31:0/6258459 >> 172.21.15.31:0/6258459 conn(0x561bfd9cb3e0 msgr2=0x561bfd9d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 -- 172.21.15.31:0/6258459 shutdown_connections 2023-07-29T16:29:35.294 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.291+0000 7f1d7c383b80 1 -- 172.21.15.31:0/6258459 wait complete. 2023-07-29T16:29:35.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 -- start start 2023-07-29T16:29:35.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 Processor -- start 2023-07-29T16:29:35.297 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 0x561bfd9df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9dfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd8c3510 con 0x561bfd9d6530 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd8c31b0 con 0x561bfd9d72f0 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.292+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd7aa920 con 0x561bfd9d5be0 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d7c383b80 10 client.0 _ll_register_callbacks cb 0x561bfd8cc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 0x561bfd9df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.298 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 0x561bfd9df800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54962/0 (socket says 172.21.15.31:54962) 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d62d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 -- 172.21.15.31:0/2423846562 learned_addr learned my addr 172.21.15.31:0/2423846562 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9dfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 -- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 msgr2=0x561bfd9e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9e6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 -- 172.21.15.31:0/2423846562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 msgr2=0x561bfd9dfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.299 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9dfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 -- 172.21.15.31:0/2423846562 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d580098e0 con 0x561bfd9d5be0 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d6259c700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 0x561bfd9df800 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400b680 tx=0x7f1d5400ff30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b91d0132da023cd0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.293+0000 7f1d49ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d49ffb700 1 -- 172.21.15.31:0/2423846562 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5409a400 con 0x561bfd9d5be0 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561bfd951d10 con 0x561bfd9d5be0 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d49ffb700 1 -- 172.21.15.31:0/2423846562 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1d5409e430 con 0x561bfd9d5be0 2023-07-29T16:29:35.300 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d49ffb700 1 -- 172.21.15.31:0/2423846562 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d540a27c0 con 0x561bfd9d5be0 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 10 client.29635 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x561bfd73dc60 con 0x561bfd9d5be0 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d49ffb700 1 -- 172.21.15.31:0/2423846562 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561bfd951d10 con 0x561bfd9d5be0 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d49ffb700 1 -- 172.21.15.31:0/2423846562 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f1d5409a660 con 0x561bfd9d5be0 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 10 client.29635 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 10 client.29635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:35.301 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 20 client.29635 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfd9eb260 0x561bfd9ed640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 4 client.29635 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.294+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561bfd9a4d70 con 0x561bfd9eb260 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.295+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfd9eb260 0x561bfd9ed640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.295+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfd9eb260 0x561bfd9ed640 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b40ae85da3edd9 server_cookie=a0e7fd8cb9d0e25a in_seq=0 out_seq=0 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.295+0000 7f1d49ffb700 10 client.29635 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d49ffb700 1 -- 172.21.15.31:0/2423846562 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561bfd9a4d70 con 0x561bfd9eb260 2023-07-29T16:29:35.302 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d49ffb700 10 client.29635 handle_command_reply: tid=0 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 client.29635 shutdown 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 msgr2=0x561bfd9df800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 0x561bfd9df800 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1d5400b680 tx=0x7f1d5400ff30 comp rx=0 tx=0).stop 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 shutdown_connections 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfd9eb260 0x561bfd9ed640 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d72f0 0x561bfd9e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.303 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d5be0 0x561bfd9df800 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2423846562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9dfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d49ffb700 0 client.29635 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.296+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 >> 172.21.15.31:0/2423846562 conn(0x561bfd9cb3e0 msgr2=0x561bfd9d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.297+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 shutdown_connections 2023-07-29T16:29:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.297+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2423846562 wait complete. 2023-07-29T16:29:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.297+0000 7f1d7c383b80 20 client.29635 trim_cache size 0 max 16384 2023-07-29T16:29:35.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.315+0000 7f1d7c383b80 1 Processor -- start 2023-07-29T16:29:35.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.315+0000 7f1d7c383b80 1 -- start start 2023-07-29T16:29:35.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.315+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.315+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d72f0 0x561bfd9d1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd9dd440 con 0x561bfd9d6530 2023-07-29T16:29:35.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd9eae30 con 0x561bfd9d5be0 2023-07-29T16:29:35.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd9de210 con 0x561bfd9d72f0 2023-07-29T16:29:35.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39456/0 (socket says 172.21.15.31:39456) 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 -- 172.21.15.31:0/2404483835 learned_addr learned my addr 172.21.15.31:0/2404483835 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d72f0 0x561bfd9d1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 -- 172.21.15.31:0/2404483835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d72f0 msgr2=0x561bfd9d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d72f0 0x561bfd9d1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 -- 172.21.15.31:0/2404483835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 msgr2=0x561bfd9d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 -- 172.21.15.31:0/2404483835 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bfd77b250 con 0x561bfd9d5be0 2023-07-29T16:29:35.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.316+0000 7f1d6259c700 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4d30 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1d54021830 tx=0x7f1d540989b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdafdcb5d2c9be5f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d517fa700 1 -- 172.21.15.31:0/2404483835 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54099f40 con 0x561bfd9d5be0 2023-07-29T16:29:35.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d517fa700 1 -- 172.21.15.31:0/2404483835 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d5409e430 con 0x561bfd9d5be0 2023-07-29T16:29:35.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d517fa700 1 -- 172.21.15.31:0/2404483835 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54099a40 con 0x561bfd9d5be0 2023-07-29T16:29:35.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2404483835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 msgr2=0x561bfd9e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4d30 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f1d54021830 tx=0x7f1d540989b0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2404483835 shutdown_connections 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d72f0 0x561bfd9d1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4d30 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/2404483835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d6530 0x561bfd9d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2404483835 >> 172.21.15.31:0/2404483835 conn(0x561bfd9cb3e0 msgr2=0x561bfd9de4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.317+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2404483835 shutdown_connections 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.318+0000 7f1d7c383b80 1 -- 172.21.15.31:0/2404483835 wait complete. 2023-07-29T16:29:35.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.318+0000 7f1d7c383b80 1 -- start start 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.318+0000 7f1d7c383b80 1 Processor -- start 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d72f0 0x561bfd9e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd9e3060 con 0x561bfd9d72f0 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd9e31d0 con 0x561bfd9d5be0 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bfd9e3390 con 0x561bfd9d6530 2023-07-29T16:29:35.324 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d7c383b80 10 client.0 _ll_register_callbacks cb 0x561bfd8cc620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d61d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39464/0 (socket says 172.21.15.31:39464) 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 -- 172.21.15.31:0/1269395087 learned_addr learned my addr 172.21.15.31:0/1269395087 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d61d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9e4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54984/0 (socket says 172.21.15.31:54984) 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d72f0 0x561bfd9e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 -- 172.21.15.31:0/1269395087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 msgr2=0x561bfd9e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 -- 172.21.15.31:0/1269395087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d72f0 msgr2=0x561bfd9e2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d72f0 0x561bfd9e2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 -- 172.21.15.31:0/1269395087 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561bfd73dc60 con 0x561bfd9d5be0 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d62d9d700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d72f0 0x561bfd9e2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d6259c700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4510 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1d540ca710 tx=0x7f1d5400fac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613f760c15c1ef77 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.319+0000 7f1d49ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d49ffb700 1 -- 172.21.15.31:0/1269395087 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d54099f40 con 0x561bfd9d5be0 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d49ffb700 1 -- 172.21.15.31:0/1269395087 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1d540a2330 con 0x561bfd9d5be0 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d58011e10 con 0x561bfd9d5be0 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d49ffb700 1 -- 172.21.15.31:0/1269395087 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d540a2550 con 0x561bfd9d5be0 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d7c383b80 10 client.29642 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x561bfd7532b0 con 0x561bfd9d5be0 2023-07-29T16:29:35.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d49ffb700 1 -- 172.21.15.31:0/1269395087 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d58011e10 con 0x561bfd9d5be0 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d49ffb700 1 -- 172.21.15.31:0/1269395087 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f1d54099980 con 0x561bfd9d5be0 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d7c383b80 10 client.29642 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d7c383b80 10 client.29642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.320+0000 7f1d7c383b80 20 client.29642 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfda3d9b0 0x561bfda3fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d7c383b80 4 client.29642 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561bfd91fd40 con 0x561bfda3d9b0 2023-07-29T16:29:35.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfda3d9b0 0x561bfda3fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d49ffb700 10 client.29642 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:35.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d61d9b700 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfda3d9b0 0x561bfda3fd90 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13446dc995f62e29 server_cookie=aee368996c458974 in_seq=0 out_seq=0 2023-07-29T16:29:35.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d49ffb700 1 -- 172.21.15.31:0/1269395087 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561bfd91fd40 con 0x561bfda3d9b0 2023-07-29T16:29:35.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d49ffb700 10 client.29642 handle_command_reply: tid=0 2023-07-29T16:29:35.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.321+0000 7f1d7c383b80 1 client.29642 shutdown 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 msgr2=0x561bfd9e4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4510 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1d540ca710 tx=0x7f1d5400fac0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 shutdown_connections 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561bfda3d9b0 0x561bfda3fd90 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561bfd9d72f0 0x561bfd9e2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561bfd9d5be0 0x561bfd9e4510 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 --2- 172.21.15.31:0/1269395087 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bfd9d6530 0x561bfd9e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d49ffb700 0 client.29642 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 >> 172.21.15.31:0/1269395087 conn(0x561bfd9cb3e0 msgr2=0x561bfd9d08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 shutdown_connections 2023-07-29T16:29:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 1 -- 172.21.15.31:0/1269395087 wait complete. 2023-07-29T16:29:35.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d7c383b80 20 client.29642 trim_cache size 0 max 16384 2023-07-29T16:29:35.331 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:29:35.331 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:29:35.332 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:29:35.332 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:29:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d4c0a0810 msgr2=0x7f1d4c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d689ff700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d4c0a0810 0x7f1d4c0a2cc0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c012f50 tx=0x7f1d5c012280 comp rx=0 tx=0).stop 2023-07-29T16:29:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 msgr2=0x7f1d6419c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.322+0000 7f1d689ff700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 0x7f1d6419c770 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1d54005720 tx=0x7f1d54009580 comp rx=0 tx=0).stop 2023-07-29T16:29:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 shutdown_connections 2023-07-29T16:29:35.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f1d4c0a0810 0x7f1d4c0a2cc0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d6410c900 0x7f1d641a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d640737b0 0x7f1d6419c770 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 --2- 172.21.15.31:0/1529593759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d64074d90 0x7f1d6419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 >> 172.21.15.31:0/1529593759 conn(0x7f1d640fb970 msgr2=0x7f1d640fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 shutdown_connections 2023-07-29T16:29:35.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.323+0000 7f1d689ff700 1 -- 172.21.15.31:0/1529593759 wait complete. 2023-07-29T16:29:35.340 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:29:35.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:29:35.551 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 -- 172.21.15.31:0/2256423091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101530 msgr2=0x7f9380101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 --2- 172.21.15.31:0/2256423091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101530 0x7f9380101980 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9374009990 tx=0x7f9374009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 -- 172.21.15.31:0/2256423091 shutdown_connections 2023-07-29T16:29:35.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 --2- 172.21.15.31:0/2256423091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380101ec0 0x7f9380108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 --2- 172.21.15.31:0/2256423091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380100340 0x7f9380100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 --2- 172.21.15.31:0/2256423091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101530 0x7f9380101980 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 -- 172.21.15.31:0/2256423091 >> 172.21.15.31:0/2256423091 conn(0x7f93800fb920 msgr2=0x7f93800fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 -- 172.21.15.31:0/2256423091 shutdown_connections 2023-07-29T16:29:35.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.551+0000 7f9385bff700 1 -- 172.21.15.31:0/2256423091 wait complete. 2023-07-29T16:29:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.552+0000 7f9385bff700 1 Processor -- start 2023-07-29T16:29:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 -- start start 2023-07-29T16:29:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380100340 0x7f9380196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 0x7f9380196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101ec0 0x7f938019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938019dbc0 con 0x7f9380101530 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938019dd00 con 0x7f9380100340 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f9385bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938019de90 con 0x7f9380101ec0 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f937effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 0x7f9380196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f937effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 0x7f9380196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42082/0 (socket says 172.21.15.31:42082) 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f937effd700 1 -- 172.21.15.31:0/2897573811 learned_addr learned my addr 172.21.15.31:0/2897573811 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f937f7fe700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380100340 0x7f9380196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.553+0000 7f937ffff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101ec0 0x7f938019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937effd700 1 -- 172.21.15.31:0/2897573811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101ec0 msgr2=0x7f938019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937effd700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101ec0 0x7f938019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937effd700 1 -- 172.21.15.31:0/2897573811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380100340 msgr2=0x7f9380196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937effd700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380100340 0x7f9380196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937effd700 1 -- 172.21.15.31:0/2897573811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9370009520 con 0x7f9380101530 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937ffff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101ec0 0x7f938019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:35.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937f7fe700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380100340 0x7f9380196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937effd700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 0x7f9380196580 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f937400b680 tx=0x7f9374005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937401d070 con 0x7f9380101530 2023-07-29T16:29:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9374022470 con 0x7f9380101530 2023-07-29T16:29:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9374017650 con 0x7f9380101530 2023-07-29T16:29:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.554+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9378009c40 con 0x7f9380101530 2023-07-29T16:29:35.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.555+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93740095f0 con 0x7f9380101530 2023-07-29T16:29:35.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.556+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9380107870 con 0x7f9380101530 2023-07-29T16:29:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.558+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f93740225e0 con 0x7f9380101530 2023-07-29T16:29:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.559+0000 7f937cff9700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f93680a07e0 0x7f93680a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.559+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f93740ccf30 con 0x7f9380101530 2023-07-29T16:29:35.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.559+0000 7f937f7fe700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f93680a07e0 0x7f93680a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:35.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.560+0000 7f937f7fe700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f93680a07e0 0x7f93680a2c90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9370009fd0 tx=0x7f9370009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.563+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f93740969a0 con 0x7f9380101530 2023-07-29T16:29:35.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.827+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f938002cc30 con 0x7f9380101530 2023-07-29T16:29:35.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.828+0000 7f937cff9700 1 -- 172.21.15.31:0/2897573811 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+10780 (secure 0 0 0) 0x7f9374099fc0 con 0x7f9380101530 2023-07-29T16:29:35.828 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:29:35.829 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:29:23.015310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:29:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f93680a07e0 msgr2=0x7f93680a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f93680a07e0 0x7f93680a2c90 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9370009fd0 tx=0x7f9370009260 comp rx=0 tx=0).stop 2023-07-29T16:29:35.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 msgr2=0x7f9380196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 0x7f9380196580 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f937400b680 tx=0x7f9374005740 comp rx=0 tx=0).stop 2023-07-29T16:29:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 shutdown_connections 2023-07-29T16:29:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f93680a07e0 0x7f93680a2c90 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9380101ec0 0x7f938019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9380100340 0x7f9380196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 --2- 172.21.15.31:0/2897573811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9380101530 0x7f9380196580 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.830+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 >> 172.21.15.31:0/2897573811 conn(0x7f93800fb920 msgr2=0x7f93801049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.831+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 shutdown_connections 2023-07-29T16:29:35.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:35.831+0000 7f9385bff700 1 -- 172.21.15.31:0/2897573811 wait complete. 2023-07-29T16:29:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 34 2023-07-29T16:29:35.850 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:29:35.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1115768 8.84 MB/sec execute 560 sec latency 14.701 ms 2023-07-29T16:29:36.061 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 -- 172.21.15.31:0/3848618877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538074d10 msgr2=0x7fc538073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/3848618877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538074d10 0x7fc538073160 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc528009990 tx=0x7fc528009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 -- 172.21.15.31:0/3848618877 shutdown_connections 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/3848618877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc538108240 0x7fc53810a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/3848618877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538074d10 0x7fc538073160 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/3848618877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538073730 0x7fc538073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 -- 172.21.15.31:0/3848618877 >> 172.21.15.31:0/3848618877 conn(0x7fc5380fb9b0 msgr2=0x7fc5380fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.062 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 -- 172.21.15.31:0/3848618877 shutdown_connections 2023-07-29T16:29:36.063 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.061+0000 7fc53e3ff700 1 -- 172.21.15.31:0/3848618877 wait complete. 2023-07-29T16:29:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.062+0000 7fc53e3ff700 1 Processor -- start 2023-07-29T16:29:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 -- start start 2023-07-29T16:29:36.065 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 0x7fc538103730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc538074d10 0x7fc538103c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538108240 0x7fc5381002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.066 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538100a60 con 0x7fc538074d10 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538100bd0 con 0x7fc538073730 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc53e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538100d90 con 0x7fc538108240 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc537fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 0x7fc538103730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc537fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 0x7fc538103730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39500/0 (socket says 172.21.15.31:39500) 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc537fff700 1 -- 172.21.15.31:0/2780252563 learned_addr learned my addr 172.21.15.31:0/2780252563 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.063+0000 7fc5377fe700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc538074d10 0x7fc538103c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538108240 0x7fc5381002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.067 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc537fff700 1 -- 172.21.15.31:0/2780252563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538108240 msgr2=0x7fc5381002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc537fff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538108240 0x7fc5381002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc537fff700 1 -- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc538074d10 msgr2=0x7fc538103c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc537fff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc538074d10 0x7fc538103c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc537fff700 1 -- 172.21.15.31:0/2780252563 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280095f0 con 0x7fc538073730 2023-07-29T16:29:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538108240 0x7fc5381002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:36.068 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc537fff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 0x7fc538103730 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc52800b680 tx=0x7fc528005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc5357fa700 1 -- 172.21.15.31:0/2780252563 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52801d070 con 0x7fc538073730 2023-07-29T16:29:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc5357fa700 1 -- 172.21.15.31:0/2780252563 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc528022470 con 0x7fc538073730 2023-07-29T16:29:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5381a4790 con 0x7fc538073730 2023-07-29T16:29:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.064+0000 7fc5357fa700 1 -- 172.21.15.31:0/2780252563 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc528017650 con 0x7fc538073730 2023-07-29T16:29:36.069 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.065+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5381a4bb0 con 0x7fc538073730 2023-07-29T16:29:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.068+0000 7fc5357fa700 1 -- 172.21.15.31:0/2780252563 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc5280225e0 con 0x7fc538073730 2023-07-29T16:29:36.072 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 Processor -- start 2023-07-29T16:29:36.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 -- start start 2023-07-29T16:29:36.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c15ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c16d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c17740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.073 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5b6da10 con 0x5619c5c172f0 2023-07-29T16:29:36.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5b6b3d0 con 0x5619c5c15be0 2023-07-29T16:29:36.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.069+0000 7fc551de8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5b66cc0 con 0x5619c5c16530 2023-07-29T16:29:36.074 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc5357fa700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5200a0790 0x7fc5200a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc5357fa700 1 -- 172.21.15.31:0/2780252563 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc5280ccff0 con 0x7fc538073730 2023-07-29T16:29:36.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c15ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.075 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c15ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39510/0 (socket says 172.21.15.31:39510) 2023-07-29T16:29:36.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc537fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c16d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.076 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 -- 172.21.15.31:0/2102201012 learned_addr learned my addr 172.21.15.31:0/2102201012 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc5377fe700 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c17740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5200a0790 0x7fc5200a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.077 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 -- 172.21.15.31:0/2102201012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 msgr2=0x5619c5c16d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c16d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 -- 172.21.15.31:0/2102201012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 msgr2=0x5619c5c17740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c17740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.070+0000 7fc53c99c700 1 -- 172.21.15.31:0/2102201012 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619c5994de0 con 0x5619c5c15be0 2023-07-29T16:29:36.078 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.071+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c15ff0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc530012cc0 tx=0x7fc530017350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79431ad264213cfa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.071+0000 7fc53c99c700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5200a0790 0x7fc5200a2c40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc530010000 tx=0x7fc53001e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.071+0000 7fc51e7fc700 1 -- 172.21.15.31:0/2102201012 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53000e040 con 0x5619c5c15be0 2023-07-29T16:29:36.079 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.071+0000 7fc51e7fc700 1 -- 172.21.15.31:0/2102201012 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5300125e0 con 0x5619c5c15be0 2023-07-29T16:29:36.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.071+0000 7fc51e7fc700 1 -- 172.21.15.31:0/2102201012 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc530007870 con 0x5619c5c15be0 2023-07-29T16:29:36.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 -- 172.21.15.31:0/2102201012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 msgr2=0x5619c5c15ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c15ff0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fc530012cc0 tx=0x7fc530017350 comp rx=0 tx=0).stop 2023-07-29T16:29:36.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 -- 172.21.15.31:0/2102201012 shutdown_connections 2023-07-29T16:29:36.080 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c17740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c15ff0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 --2- 172.21.15.31:0/2102201012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c16d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 -- 172.21.15.31:0/2102201012 >> 172.21.15.31:0/2102201012 conn(0x5619c5c0b3e0 msgr2=0x5619c5c10990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 -- 172.21.15.31:0/2102201012 shutdown_connections 2023-07-29T16:29:36.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 -- 172.21.15.31:0/2102201012 wait complete. 2023-07-29T16:29:36.081 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.072+0000 7fc551de8b80 1 -- start start 2023-07-29T16:29:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 Processor -- start 2023-07-29T16:29:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c1f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.085 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c1fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c26d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5b03510 con 0x5619c5c15be0 2023-07-29T16:29:36.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5b031b0 con 0x5619c5c16530 2023-07-29T16:29:36.086 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c59ea920 con 0x5619c5c172f0 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc551de8b80 10 client.0 _ll_register_callbacks cb 0x5619c5b0c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc5377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c1fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc5377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c1fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39526/0 (socket says 172.21.15.31:39526) 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc5377fe700 1 -- 172.21.15.31:0/3081492340 learned_addr learned my addr 172.21.15.31:0/3081492340 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.073+0000 7fc53c99c700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c26d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc537fff700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c1f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.087 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5377fe700 1 -- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 msgr2=0x5619c5c26d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5377fe700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c26d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5377fe700 1 -- 172.21.15.31:0/3081492340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 msgr2=0x5619c5c1f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5377fe700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c1f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5377fe700 1 -- 172.21.15.31:0/3081492340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5300128a0 con 0x5619c5c16530 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc537fff700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c1f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5377fe700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c1fd50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0117c0 tx=0x7fc52c011ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=976527458658516d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.088 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5177fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5177fe700 1 -- 172.21.15.31:0/3081492340 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c016070 con 0x5619c5c16530 2023-07-29T16:29:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5177fe700 1 -- 172.21.15.31:0/3081492340 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc52c00f810 con 0x5619c5c16530 2023-07-29T16:29:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619c5be4d70 con 0x5619c5c16530 2023-07-29T16:29:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc5177fe700 1 -- 172.21.15.31:0/3081492340 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc52c00f9d0 con 0x5619c5c16530 2023-07-29T16:29:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.074+0000 7fc551de8b80 10 client.29660 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:36.089 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5619c597dc60 con 0x5619c5c16530 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc5177fe700 1 -- 172.21.15.31:0/3081492340 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc52c028020 con 0x5619c5c16530 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc5177fe700 1 -- 172.21.15.31:0/3081492340 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc52c01abf0 con 0x5619c5c16530 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 10 client.29660 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 10 client.29660 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 20 client.29660 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c2b640 0x5619c5c2da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.090 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 4 client.29660 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619c5b5fd40 con 0x5619c5c2b640 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.075+0000 7fc537fff700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c2b640 0x5619c5c2da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.076+0000 7fc537fff700 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c2b640 0x5619c5c2da50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82fc560c7cfb9704 server_cookie=701b7f5afa201c8b in_seq=0 out_seq=0 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.076+0000 7fc5177fe700 10 client.29660 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.076+0000 7fc5177fe700 1 -- 172.21.15.31:0/3081492340 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5619c5b5fd40 con 0x5619c5c2b640 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc5177fe700 10 client.29660 handle_command_reply: tid=0 2023-07-29T16:29:36.091 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 client.29660 shutdown 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 msgr2=0x5619c5c1fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c1fd50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0117c0 tx=0x7fc52c011ad0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 shutdown_connections 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c26d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c1f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c2b640 0x5619c5c2da50 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.092 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3081492340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c1fd50 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 >> 172.21.15.31:0/3081492340 conn(0x5619c5c0b3e0 msgr2=0x5619c5c10060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc5177fe700 0 client.29660 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:36.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 shutdown_connections 2023-07-29T16:29:36.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 1 -- 172.21.15.31:0/3081492340 wait complete. 2023-07-29T16:29:36.093 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.077+0000 7fc551de8b80 20 client.29660 trim_cache size 0 max 16384 2023-07-29T16:29:36.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.091+0000 7fc551de8b80 1 Processor -- start 2023-07-29T16:29:36.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.091+0000 7fc551de8b80 1 -- start start 2023-07-29T16:29:36.100 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c24d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c1a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c1a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc551de8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5b5fd40 con 0x5619c5c172f0 2023-07-29T16:29:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc551de8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5c16bf0 con 0x5619c5c15be0 2023-07-29T16:29:36.101 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc551de8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5c28a80 con 0x5619c5c16530 2023-07-29T16:29:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c1a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c1a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55068/0 (socket says 172.21.15.31:55068) 2023-07-29T16:29:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 -- 172.21.15.31:0/1924180219 learned_addr learned my addr 172.21.15.31:0/1924180219 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.102 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc537fff700 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c24d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc53c99c700 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c1a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 -- 172.21.15.31:0/1924180219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 msgr2=0x5619c5c24d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c24d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 -- 172.21.15.31:0/1924180219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 msgr2=0x5619c5c1a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c1a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.092+0000 7fc5377fe700 1 -- 172.21.15.31:0/1924180219 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619c59bb250 con 0x5619c5c16530 2023-07-29T16:29:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.093+0000 7fc53c99c700 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c1a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:36.104 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.093+0000 7fc5377fe700 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c1a2f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc52c01af20 tx=0x7fc52c012f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f357171b68038c31 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.093+0000 7fc51effd700 1 -- 172.21.15.31:0/1924180219 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c016040 con 0x5619c5c16530 2023-07-29T16:29:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.093+0000 7fc51effd700 1 -- 172.21.15.31:0/1924180219 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc52c028070 con 0x5619c5c16530 2023-07-29T16:29:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.093+0000 7fc51effd700 1 -- 172.21.15.31:0/1924180219 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c0052d0 con 0x5619c5c16530 2023-07-29T16:29:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 -- 172.21.15.31:0/1924180219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 msgr2=0x5619c5c1a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c1a2f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc52c01af20 tx=0x7fc52c012f60 comp rx=0 tx=0).stop 2023-07-29T16:29:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 -- 172.21.15.31:0/1924180219 shutdown_connections 2023-07-29T16:29:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c172f0 0x5619c5c1a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c15be0 0x5619c5c24d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 --2- 172.21.15.31:0/1924180219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c16530 0x5619c5c1a2f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.106 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 -- 172.21.15.31:0/1924180219 >> 172.21.15.31:0/1924180219 conn(0x5619c5c0b3e0 msgr2=0x5619c5c1e450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 -- 172.21.15.31:0/1924180219 shutdown_connections 2023-07-29T16:29:36.107 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 -- 172.21.15.31:0/1924180219 wait complete. 2023-07-29T16:29:36.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.094+0000 7fc551de8b80 1 -- start start 2023-07-29T16:29:36.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 Processor -- start 2023-07-29T16:29:36.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c753d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.108 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c75910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c7c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5c7ce90 con 0x5619c5c15be0 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5c7cfd0 con 0x5619c5c16530 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619c5c7d160 con 0x5619c5c172f0 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc551de8b80 10 client.0 _ll_register_callbacks cb 0x5619c5b0c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc5377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c75910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.109 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc5377fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c75910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39536/0 (socket says 172.21.15.31:39536) 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc5377fe700 1 -- 172.21.15.31:0/3367821738 learned_addr learned my addr 172.21.15.31:0/3367821738 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc537fff700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.095+0000 7fc53c99c700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c7c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5377fe700 1 -- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 msgr2=0x5619c5c7c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5377fe700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c7c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5377fe700 1 -- 172.21.15.31:0/3367821738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 msgr2=0x5619c5c753d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5377fe700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c753d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5377fe700 1 -- 172.21.15.31:0/3367821738 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619c597dc60 con 0x5619c5c16530 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5377fe700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c75910 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00fd90 tx=0x7fc52c009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21ca40d843a5ec65 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5177fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5177fe700 1 -- 172.21.15.31:0/3367821738 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c016040 con 0x5619c5c16530 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619c5b91d10 con 0x5619c5c16530 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.096+0000 7fc5177fe700 1 -- 172.21.15.31:0/3367821738 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc52c000e40 con 0x5619c5c16530 2023-07-29T16:29:36.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc5177fe700 1 -- 172.21.15.31:0/3367821738 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc52c022510 con 0x5619c5c16530 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 10 client.29666 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5619c59932b0 con 0x5619c5c16530 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc5177fe700 1 -- 172.21.15.31:0/3367821738 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5619c5b91d10 con 0x5619c5c16530 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc5177fe700 1 -- 172.21.15.31:0/3367821738 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc52c003680 con 0x5619c5c16530 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 10 client.29666 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 10 client.29666 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:29:36.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 20 client.29666 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c7e7b0 0x5619c5c80bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 4 client.29666 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:29:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5619c5b5fa60 con 0x5619c5c7e7b0 2023-07-29T16:29:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.097+0000 7fc537fff700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c7e7b0 0x5619c5c80bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc5177fe700 10 client.29666 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:36.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc537fff700 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c7e7b0 0x5619c5c80bc0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=808851f46959dbea server_cookie=b16dc675f64c834c in_seq=0 out_seq=0 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc5177fe700 1 -- 172.21.15.31:0/3367821738 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5619c5b5fa60 con 0x5619c5c7e7b0 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc5177fe700 10 client.29666 handle_command_reply: tid=0 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 client.29666 shutdown 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 msgr2=0x5619c5c75910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c75910 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00fd90 tx=0x7fc52c009f00 comp rx=0 tx=0).stop 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 shutdown_connections 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5619c5c172f0 0x5619c5c7c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5619c5c15be0 0x5619c5c753d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x5619c5c7e7b0 0x5619c5c80bc0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 --2- 172.21.15.31:0/3367821738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5619c5c16530 0x5619c5c75910 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 >> 172.21.15.31:0/3367821738 conn(0x5619c5c0b3e0 msgr2=0x5619c5c1a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc5177fe700 0 client.29666 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 shutdown_connections 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.098+0000 7fc551de8b80 1 -- 172.21.15.31:0/3367821738 wait complete. 2023-07-29T16:29:36.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc551de8b80 20 client.29666 trim_cache size 0 max 16384 2023-07-29T16:29:36.116 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:29:36.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5200a0790 msgr2=0x7fc5200a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5200a0790 0x7fc5200a2c40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc530010000 tx=0x7fc53001e000 comp rx=0 tx=0).stop 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 msgr2=0x7fc538103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 0x7fc538103730 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc52800b680 tx=0x7fc528005740 comp rx=0 tx=0).stop 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 shutdown_connections 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5200a0790 0x7fc5200a2c40 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc538108240 0x7fc5381002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc538073730 0x7fc538103730 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 --2- 172.21.15.31:0/2780252563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc538074d10 0x7fc538103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 >> 172.21.15.31:0/2780252563 conn(0x7fc5380fb9b0 msgr2=0x7fc5380fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 shutdown_connections 2023-07-29T16:29:36.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.099+0000 7fc53e3ff700 1 -- 172.21.15.31:0/2780252563 wait complete. 2023-07-29T16:29:36.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:29:36.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 -- 172.21.15.31:0/2524136455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc10c900 msgr2=0x7f6ecc10ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/2524136455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc10c900 0x7f6ecc10ece0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b0a0 tx=0x7f6ec400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 -- 172.21.15.31:0/2524136455 shutdown_connections 2023-07-29T16:29:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/2524136455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc10c900 0x7f6ecc10ece0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/2524136455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ecc074d90 0x7f6ecc0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/2524136455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.336+0000 7f6ed11ff700 1 -- 172.21.15.31:0/2524136455 >> 172.21.15.31:0/2524136455 conn(0x7f6ecc0fb970 msgr2=0x7f6ecc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.337+0000 7f6ed11ff700 1 -- 172.21.15.31:0/2524136455 shutdown_connections 2023-07-29T16:29:36.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.337+0000 7f6ed11ff700 1 -- 172.21.15.31:0/2524136455 wait complete. 2023-07-29T16:29:36.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 Processor -- start 2023-07-29T16:29:36.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 -- start start 2023-07-29T16:29:36.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ecc074d90 0x7f6ecc19ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ecc10c900 0x7f6ecc1a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1a4310 con 0x7f6ecc10c900 2023-07-29T16:29:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1a4450 con 0x7f6ecc0737b0 2023-07-29T16:29:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ed11ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1a45e0 con 0x7f6ecc074d90 2023-07-29T16:29:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ecad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ecad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc19c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39554/0 (socket says 172.21.15.31:39554) 2023-07-29T16:29:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ecad9d700 1 -- 172.21.15.31:0/1789483831 learned_addr learned my addr 172.21.15.31:0/1789483831 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6eca59c700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ecc074d90 0x7f6ecc19ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.338+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ecc10c900 0x7f6ecc1a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ecad9d700 1 -- 172.21.15.31:0/1789483831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ecc074d90 msgr2=0x7f6ecc19ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ecc074d90 0x7f6ecc19ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ecad9d700 1 -- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ecc10c900 msgr2=0x7f6ecc1a3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ecc10c900 0x7f6ecc1a3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ecad9d700 1 -- 172.21.15.31:0/1789483831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec4009d00 con 0x7f6ecc0737b0 2023-07-29T16:29:36.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc19c790 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00e970 tx=0x7f6ebc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6eb3fff700 1 -- 172.21.15.31:0/1789483831 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc009790 con 0x7f6ecc0737b0 2023-07-29T16:29:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc1a48c0 con 0x7f6ecc0737b0 2023-07-29T16:29:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6eb3fff700 1 -- 172.21.15.31:0/1789483831 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ebc00c7e0 con 0x7f6ecc0737b0 2023-07-29T16:29:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6eb3fff700 1 -- 172.21.15.31:0/1789483831 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc0054e0 con 0x7f6ecc0737b0 2023-07-29T16:29:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.339+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc1a4dd0 con 0x7f6ecc0737b0 2023-07-29T16:29:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.343+0000 7f6eb3fff700 1 -- 172.21.15.31:0/1789483831 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6ebc00c950 con 0x7f6ecc0737b0 2023-07-29T16:29:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 Processor -- start 2023-07-29T16:29:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 -- start start 2023-07-29T16:29:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341928be0 0x55a341928ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341929530 0x55a341929d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a34192a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a341880a10 con 0x55a341929530 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34187e3d0 con 0x55a34192a2f0 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a341879cc0 con 0x55a341928be0 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6eb3fff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6eb40a0790 0x7f6eb40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a34192a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a34192a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39560/0 (socket says 172.21.15.31:39560) 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.344+0000 7f6eca59c700 1 -- 172.21.15.31:0/3683471221 learned_addr learned my addr 172.21.15.31:0/3683471221 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341928be0 0x55a341928ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eb3fff700 1 -- 172.21.15.31:0/1789483831 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6ebc014070 con 0x7f6ecc0737b0 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341929530 0x55a341929d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eca59c700 1 -- 172.21.15.31:0/3683471221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341928be0 msgr2=0x55a341928ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eca59c700 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341928be0 0x55a341928ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6eb40a0790 0x7f6eb40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eca59c700 1 -- 172.21.15.31:0/3683471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341929530 msgr2=0x55a341929d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eca59c700 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341929530 0x55a341929d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eca59c700 1 -- 172.21.15.31:0/3683471221 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3416a7de0 con 0x55a34192a2f0 2023-07-29T16:29:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eca59c700 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a34192a740 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009d30 tx=0x7f6ec0011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c3c2df75604e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eb17fa700 1 -- 172.21.15.31:0/3683471221 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0004220 con 0x55a34192a2f0 2023-07-29T16:29:36.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eb17fa700 1 -- 172.21.15.31:0/3683471221 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ec0004380 con 0x55a34192a2f0 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.345+0000 7f6eb17fa700 1 -- 172.21.15.31:0/3683471221 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec00198d0 con 0x55a34192a2f0 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3683471221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 msgr2=0x55a34192a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a34192a740 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009d30 tx=0x7f6ec0011720 comp rx=0 tx=0).stop 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6eb40a0790 0x7f6eb40a2c40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b780 tx=0x7f6ec4006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3683471221 shutdown_connections 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a34192a740 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341928be0 0x55a341928ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3683471221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341929530 0x55a341929d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3683471221 >> 172.21.15.31:0/3683471221 conn(0x55a34191e3e0 msgr2=0x55a341923990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3683471221 shutdown_connections 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.346+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3683471221 wait complete. 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 -- start start 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 Processor -- start 2023-07-29T16:29:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a3419327f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341932d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a341939d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a341816510 con 0x55a341928be0 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3418161b0 con 0x55a34192a2f0 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.347+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3416fd920 con 0x55a341929530 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ee4b01b80 10 client.0 _ll_register_callbacks cb 0x55a34181f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a341939d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a3419327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a341939d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39562/0 (socket says 172.21.15.31:39562) 2023-07-29T16:29:36.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 -- 172.21.15.31:0/1444340759 learned_addr learned my addr 172.21.15.31:0/1444340759 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6eca59c700 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341932d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 -- 172.21.15.31:0/1444340759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 msgr2=0x55a341932d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341932d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 -- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 msgr2=0x55a3419327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a3419327f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 -- 172.21.15.31:0/1444340759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec0009960 con 0x55a34192a2f0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a341939d70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4015000 tx=0x7f6ec40099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d443bbf004dba89 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.348+0000 7f6eaeff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6eaeff5700 1 -- 172.21.15.31:0/1444340759 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4027070 con 0x55a34192a2f0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6eaeff5700 1 -- 172.21.15.31:0/1444340759 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6ec4020450 con 0x55a34192a2f0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a3418a4d10 con 0x55a34192a2f0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6eaeff5700 1 -- 172.21.15.31:0/1444340759 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ec400e040 con 0x55a34192a2f0 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6ee4b01b80 10 client.29678 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:36.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a341690c60 con 0x55a34192a2f0 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6eaeff5700 1 -- 172.21.15.31:0/1444340759 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a3418a4d10 con 0x55a34192a2f0 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6eaeff5700 1 -- 172.21.15.31:0/1444340759 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f6ec4033070 con 0x55a34192a2f0 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6ee4b01b80 10 client.29678 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.349+0000 7f6ee4b01b80 10 client.29678 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6ee4b01b80 20 client.29678 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a34193e550 0x55a341940960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6ee4b01b80 4 client.29678 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3418f7d70 con 0x55a34193e550 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a34193e550 0x55a341940960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6eaeff5700 10 client.29678 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.350+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a34193e550 0x55a341940960 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=600a4fea89e1c40b server_cookie=7e05af7c56733900 in_seq=0 out_seq=0 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6eaeff5700 1 -- 172.21.15.31:0/1444340759 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a3418f7d70 con 0x55a34193e550 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6eaeff5700 10 client.29678 handle_command_reply: tid=0 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 client.29678 shutdown 2023-07-29T16:29:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 msgr2=0x55a341939d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a341939d70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4015000 tx=0x7f6ec40099e0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 shutdown_connections 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a34193e550 0x55a341940960 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a34192a2f0 0x55a341939d70 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a3419327f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6eaeff5700 0 client.29678 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/1444340759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341932d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 >> 172.21.15.31:0/1444340759 conn(0x55a34191e3e0 msgr2=0x55a341923060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 shutdown_connections 2023-07-29T16:29:36.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.351+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/1444340759 wait complete. 2023-07-29T16:29:36.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.352+0000 7f6ee4b01b80 20 client.29678 trim_cache size 0 max 16384 2023-07-29T16:29:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 Processor -- start 2023-07-29T16:29:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 -- start start 2023-07-29T16:29:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a341940930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 0x55a341939d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a34192a2f0 0x55a34192a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a341931210 con 0x55a341928be0 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a34193e270 con 0x55a341929530 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.366+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3419452e0 con 0x55a34192a2f0 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a341940930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 0x55a341939d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a341940930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42200/0 (socket says 172.21.15.31:42200) 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 0x55a341939d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39564/0 (socket says 172.21.15.31:39564) 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 -- 172.21.15.31:0/3834519377 learned_addr learned my addr 172.21.15.31:0/3834519377 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a34192a2f0 0x55a34192a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 -- 172.21.15.31:0/3834519377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a34192a2f0 msgr2=0x55a34192a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a34192a2f0 0x55a34192a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 -- 172.21.15.31:0/3834519377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 msgr2=0x55a341939d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 0x55a341939d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 -- 172.21.15.31:0/3834519377 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3416ce250 con 0x55a341928be0 2023-07-29T16:29:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6eca59c700 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 0x55a341939d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.367+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a341940930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc095980 tx=0x7f6ebc095150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b3a944038e7581 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6eb1ffb700 1 -- 172.21.15.31:0/3834519377 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc0994d0 con 0x55a341928be0 2023-07-29T16:29:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6eb1ffb700 1 -- 172.21.15.31:0/3834519377 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ebc09d500 con 0x55a341928be0 2023-07-29T16:29:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6eb1ffb700 1 -- 172.21.15.31:0/3834519377 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc0056f0 con 0x55a341928be0 2023-07-29T16:29:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3834519377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 msgr2=0x55a341940930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a341940930 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc095980 tx=0x7f6ebc095150 comp rx=0 tx=0).stop 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3834519377 shutdown_connections 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a34192a2f0 0x55a34192a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a341928be0 0x55a341940930 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/3834519377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341929530 0x55a341939d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3834519377 >> 172.21.15.31:0/3834519377 conn(0x55a34191e3e0 msgr2=0x55a341923900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3834519377 shutdown_connections 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.368+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/3834519377 wait complete. 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 -- start start 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 Processor -- start 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341928be0 0x55a341935910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341935e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a34192a2f0 0x55a341938e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3419393b0 con 0x55a34192a2f0 2023-07-29T16:29:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a341939520 con 0x55a341928be0 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3419396e0 con 0x55a341929530 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.369+0000 7f6ee4b01b80 10 client.0 _ll_register_callbacks cb 0x55a34181f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341935e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341928be0 0x55a341935910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6eca59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341935e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55120/0 (socket says 172.21.15.31:55120) 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6eca59c700 1 -- 172.21.15.31:0/54941908 learned_addr learned my addr 172.21.15.31:0/54941908 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.376 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a34192a2f0 0x55a341938e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 -- 172.21.15.31:0/54941908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 msgr2=0x55a341935e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341935e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 -- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a34192a2f0 msgr2=0x55a341938e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a34192a2f0 0x55a341938e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.377 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 -- 172.21.15.31:0/54941908 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3416a7de0 con 0x55a341928be0 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecb59e700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a34192a2f0 0x55a341938e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ecad9d700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341928be0 0x55a341935910 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc095980 tx=0x7f6ebc0a1920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10d69e2930f905c6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6eaeff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6eaeff5700 1 -- 172.21.15.31:0/54941908 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc0994d0 con 0x55a341928be0 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6eaeff5700 1 -- 172.21.15.31:0/54941908 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6ebc095be0 con 0x55a341928be0 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.370+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ec0004380 con 0x55a341928be0 2023-07-29T16:29:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6eaeff5700 1 -- 172.21.15.31:0/54941908 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ebc095da0 con 0x55a341928be0 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 10 client.29684 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7f6ecc1a4dd0 con 0x55a341928be0 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6eaeff5700 1 -- 172.21.15.31:0/54941908 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ebc07c080 con 0x55a341928be0 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6eaeff5700 1 -- 172.21.15.31:0/54941908 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f6ebc0989c0 con 0x55a341928be0 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 10 client.29684 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 10 client.29684 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:29:36.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 20 client.29684 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a341990940 0x55a341992d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 4 client.29684 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a341872d40 con 0x55a341990940 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.371+0000 7f6eca59c700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a341990940 0x55a341992d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6eca59c700 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a341990940 0x55a341992d50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d1f4da7efcad0ae server_cookie=97bd30902c8c6276 in_seq=0 out_seq=0 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6eaeff5700 10 client.29684 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:36.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6eaeff5700 1 -- 172.21.15.31:0/54941908 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a341872d40 con 0x55a341990940 2023-07-29T16:29:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6eaeff5700 10 client.29684 handle_command_reply: tid=0 2023-07-29T16:29:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 client.29684 shutdown 2023-07-29T16:29:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341928be0 msgr2=0x55a341935910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341928be0 0x55a341935910 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc095980 tx=0x7f6ebc0a1920 comp rx=0 tx=0).stop 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 shutdown_connections 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x55a341990940 0x55a341992d50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a34192a2f0 0x55a341938e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6eaeff5700 0 client.29684 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a341928be0 0x55a341935910 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 --2- 172.21.15.31:0/54941908 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a341929530 0x55a341935e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 >> 172.21.15.31:0/54941908 conn(0x55a34191e3e0 msgr2=0x55a341930680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.372+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 shutdown_connections 2023-07-29T16:29:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ee4b01b80 1 -- 172.21.15.31:0/54941908 wait complete. 2023-07-29T16:29:36.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ee4b01b80 20 client.29684 trim_cache size 0 max 16384 2023-07-29T16:29:36.383 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:29:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6eb40a0790 msgr2=0x7f6eb40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6eb40a0790 0x7f6eb40a2c40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400b780 tx=0x7f6ec4006230 comp rx=0 tx=0).stop 2023-07-29T16:29:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 msgr2=0x7f6ecc19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc19c790 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00e970 tx=0x7f6ebc00ec80 comp rx=0 tx=0).stop 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 shutdown_connections 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6eb40a0790 0x7f6eb40a2c40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ecc10c900 0x7f6ecc1a3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ecc0737b0 0x7f6ecc19c790 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 --2- 172.21.15.31:0/1789483831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ecc074d90 0x7f6ecc19ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 >> 172.21.15.31:0/1789483831 conn(0x7f6ecc0fb970 msgr2=0x7f6ecc0ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 shutdown_connections 2023-07-29T16:29:36.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.373+0000 7f6ed11ff700 1 -- 172.21.15.31:0/1789483831 wait complete. 2023-07-29T16:29:36.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:29:36.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 -- 172.21.15.31:0/1670195156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c1036c0 msgr2=0x7f942c105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 --2- 172.21.15.31:0/1670195156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c1036c0 0x7f942c105aa0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9420009990 tx=0x7f9420009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 -- 172.21.15.31:0/1670195156 shutdown_connections 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 --2- 172.21.15.31:0/1670195156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c105fe0 0x7f942c108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 --2- 172.21.15.31:0/1670195156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 --2- 172.21.15.31:0/1670195156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c1036c0 0x7f942c105aa0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 -- 172.21.15.31:0/1670195156 >> 172.21.15.31:0/1670195156 conn(0x7f942c0fa840 msgr2=0x7f942c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 -- 172.21.15.31:0/1670195156 shutdown_connections 2023-07-29T16:29:36.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.606+0000 7f9430fff700 1 -- 172.21.15.31:0/1670195156 wait complete. 2023-07-29T16:29:36.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.607+0000 7f9430fff700 1 Processor -- start 2023-07-29T16:29:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 -- start start 2023-07-29T16:29:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c1960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c1036c0 0x7f942c1965e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c105fe0 0x7f942c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c19dc20 con 0x7f942c100da0 2023-07-29T16:29:36.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c19dd60 con 0x7f942c105fe0 2023-07-29T16:29:36.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f9430fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c19def0 con 0x7f942c1036c0 2023-07-29T16:29:36.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f942a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c1960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f942a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c1960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42236/0 (socket says 172.21.15.31:42236) 2023-07-29T16:29:36.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.608+0000 7f942a59c700 1 -- 172.21.15.31:0/1527178314 learned_addr learned my addr 172.21.15.31:0/1527178314 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942ad9d700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c105fe0 0x7f942c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f9429d9b700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c1036c0 0x7f942c1965e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942a59c700 1 -- 172.21.15.31:0/1527178314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c1036c0 msgr2=0x7f942c1965e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942a59c700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c1036c0 0x7f942c1965e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942a59c700 1 -- 172.21.15.31:0/1527178314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c105fe0 msgr2=0x7f942c19d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942a59c700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c105fe0 0x7f942c19d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942a59c700 1 -- 172.21.15.31:0/1527178314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200095f0 con 0x7f942c100da0 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942ad9d700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c105fe0 0x7f942c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f9429d9b700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c1036c0 0x7f942c1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f942a59c700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c1960a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f941c00e9a0 tx=0x7f941c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.609+0000 7f941b7fe700 1 -- 172.21.15.31:0/1527178314 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c009790 con 0x7f942c100da0 2023-07-29T16:29:36.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.610+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942c19e1d0 con 0x7f942c100da0 2023-07-29T16:29:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.610+0000 7f941b7fe700 1 -- 172.21.15.31:0/1527178314 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f941c00cbd0 con 0x7f942c100da0 2023-07-29T16:29:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.610+0000 7f941b7fe700 1 -- 172.21.15.31:0/1527178314 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c018680 con 0x7f942c100da0 2023-07-29T16:29:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.610+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c19e730 con 0x7f942c100da0 2023-07-29T16:29:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.613+0000 7f941b7fe700 1 -- 172.21.15.31:0/1527178314 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f941c00cd40 con 0x7f942c100da0 2023-07-29T16:29:36.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.614+0000 7f94449d7b80 1 Processor -- start 2023-07-29T16:29:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 -- start start 2023-07-29T16:29:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 0x55c0d8aa1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d89f3a10 con 0x55c0d8a9fff0 2023-07-29T16:29:36.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d89f13d0 con 0x55c0d8aa0940 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f94449d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d89eccc0 con 0x55c0d8aa1700 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f941b7fe700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94140a0810 0x7f94140a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f942a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 0x55c0d8aa1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f942a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 0x55c0d8aa1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39608/0 (socket says 172.21.15.31:39608) 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f942a59c700 1 -- 172.21.15.31:0/2082319907 learned_addr learned my addr 172.21.15.31:0/2082319907 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f942ad9d700 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.615+0000 7f941b7fe700 1 -- 172.21.15.31:0/1527178314 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f941c014070 con 0x7f942c100da0 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f9429d9b700 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942a59c700 1 -- 172.21.15.31:0/2082319907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa1700 msgr2=0x55c0d8aa1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942a59c700 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942ad9d700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94140a0810 0x7f94140a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942a59c700 1 -- 172.21.15.31:0/2082319907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 msgr2=0x55c0d8aa0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942a59c700 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942a59c700 1 -- 172.21.15.31:0/2082319907 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0d881ade0 con 0x55c0d8aa0940 2023-07-29T16:29:36.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942a59c700 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 0x55c0d8aa1170 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f941c00c5b0 tx=0x7f941c06b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7be7fa630630d0f5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f9418ff9700 1 -- 172.21.15.31:0/2082319907 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c099490 con 0x55c0d8aa0940 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f942ad9d700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94140a0810 0x7f94140a2cc0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x55c0d8a9fd90 tx=0x7f9424009130 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f9418ff9700 1 -- 172.21.15.31:0/2082319907 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f941c09d4c0 con 0x55c0d8aa0940 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.616+0000 7f9418ff9700 1 -- 172.21.15.31:0/2082319907 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0996f0 con 0x55c0d8aa0940 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 -- 172.21.15.31:0/2082319907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 msgr2=0x55c0d8aa1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 0x55c0d8aa1170 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f941c00c5b0 tx=0x7f941c06b8a0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 -- 172.21.15.31:0/2082319907 shutdown_connections 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2082319907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa0940 0x55c0d8aa1170 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 -- 172.21.15.31:0/2082319907 >> 172.21.15.31:0/2082319907 conn(0x55c0d8a913e0 msgr2=0x55c0d8a96990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 -- 172.21.15.31:0/2082319907 shutdown_connections 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.617+0000 7f94449d7b80 1 -- 172.21.15.31:0/2082319907 wait complete. 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.618+0000 7f94449d7b80 1 -- start start 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.618+0000 7f94449d7b80 1 Processor -- start 2023-07-29T16:29:36.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.618+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa0940 0x55c0d8aaa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8ab1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f94449d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8989510 con 0x55c0d8aa0940 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f94449d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d89891b0 con 0x55c0d8aa1700 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f94449d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8870920 con 0x55c0d8a9fff0 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f94449d7b80 10 client.0 _ll_register_callbacks cb 0x55c0d8992620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:36.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa9ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55164/0 (socket says 172.21.15.31:55164) 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 -- 172.21.15.31:0/2985331450 learned_addr learned my addr 172.21.15.31:0/2985331450 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942ad9d700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8ab1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f9429d9b700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa0940 0x55c0d8aaa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 -- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 msgr2=0x55c0d8ab1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8ab1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 -- 172.21.15.31:0/2985331450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa0940 msgr2=0x55c0d8aaa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa0940 0x55c0d8aaa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 -- 172.21.15.31:0/2985331450 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c19e730 con 0x55c0d8a9fff0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f942a59c700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa9ca0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f941c095990 tx=0x7f941c09d3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abf86fb556fce88e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.619+0000 7f9411ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f9411ffb700 1 -- 172.21.15.31:0/2985331450 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c099490 con 0x55c0d8a9fff0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f9411ffb700 1 -- 172.21.15.31:0/2985331450 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f941c0ae070 con 0x55c0d8a9fff0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c0d8a17d10 con 0x55c0d8a9fff0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f9411ffb700 1 -- 172.21.15.31:0/2985331450 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f941c098b90 con 0x55c0d8a9fff0 2023-07-29T16:29:36.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 10 client.29695 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55c0d8803c60 con 0x55c0d8a9fff0 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f9411ffb700 1 -- 172.21.15.31:0/2985331450 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c0d8a17d10 con 0x55c0d8a9fff0 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f9411ffb700 1 -- 172.21.15.31:0/2985331450 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f941c068d20 con 0x55c0d8a9fff0 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 10 client.29695 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 10 client.29695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 20 client.29695 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:36.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8ab5540 0x55c0d8ab7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 4 client.29695 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.620+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0d8a6ad70 con 0x55c0d8ab5540 2023-07-29T16:29:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.621+0000 7f9429d9b700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8ab5540 0x55c0d8ab7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.621+0000 7f9429d9b700 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8ab5540 0x55c0d8ab7950 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b411a4ad9461288c server_cookie=25c983410a6addd9 in_seq=0 out_seq=0 2023-07-29T16:29:36.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.621+0000 7f9411ffb700 10 client.29695 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f9411ffb700 1 -- 172.21.15.31:0/2985331450 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c0d8a6ad70 con 0x55c0d8ab5540 2023-07-29T16:29:36.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f9411ffb700 10 client.29695 handle_command_reply: tid=0 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 client.29695 shutdown 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 msgr2=0x55c0d8aa9ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa9ca0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f941c095990 tx=0x7f941c09d3e0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 shutdown_connections 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8ab5540 0x55c0d8ab7950 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8ab1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa9ca0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 --2- 172.21.15.31:0/2985331450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa0940 0x55c0d8aaa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f9411ffb700 0 client.29695 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 >> 172.21.15.31:0/2985331450 conn(0x55c0d8a913e0 msgr2=0x55c0d8a96060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 shutdown_connections 2023-07-29T16:29:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.622+0000 7f94449d7b80 1 -- 172.21.15.31:0/2985331450 wait complete. 2023-07-29T16:29:36.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.623+0000 7f94449d7b80 20 client.29695 trim_cache size 0 max 16384 2023-07-29T16:29:36.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.640+0000 7f94449d7b80 1 Processor -- start 2023-07-29T16:29:36.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.640+0000 7f94449d7b80 1 -- start start 2023-07-29T16:29:36.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8ab11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f94449d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8aa8e60 con 0x55c0d8aa1700 2023-07-29T16:29:36.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f94449d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8aa8fa0 con 0x55c0d8a9fff0 2023-07-29T16:29:36.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f94449d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8abc540 con 0x55c0d8aa0940 2023-07-29T16:29:36.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f942a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f942a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39624/0 (socket says 172.21.15.31:39624) 2023-07-29T16:29:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f9429d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8ab11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f942a59c700 1 -- 172.21.15.31:0/3870057369 learned_addr learned my addr 172.21.15.31:0/3870057369 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.641+0000 7f942ad9d700 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f9429d9b700 1 -- 172.21.15.31:0/3870057369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 msgr2=0x55c0d8aa1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f9429d9b700 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f9429d9b700 1 -- 172.21.15.31:0/3870057369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa1700 msgr2=0x55c0d8aa8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f9429d9b700 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f9429d9b700 1 -- 172.21.15.31:0/3870057369 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0d8841250 con 0x55c0d8aa0940 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f942a59c700 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f9429d9b700 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8ab11e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9420000c00 tx=0x7f9420005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=619e524b3be1d70e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f94197fa700 1 -- 172.21.15.31:0/3870057369 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942002d070 con 0x55c0d8aa0940 2023-07-29T16:29:36.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f94197fa700 1 -- 172.21.15.31:0/3870057369 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f942001e040 con 0x55c0d8aa0940 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.642+0000 7f94197fa700 1 -- 172.21.15.31:0/3870057369 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420005040 con 0x55c0d8aa0940 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 -- 172.21.15.31:0/3870057369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 msgr2=0x55c0d8ab11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8ab11e0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9420000c00 tx=0x7f9420005e80 comp rx=0 tx=0).stop 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 -- 172.21.15.31:0/3870057369 shutdown_connections 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8aa1700 0x55c0d8aa8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8a9fff0 0x55c0d8aa1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 --2- 172.21.15.31:0/3870057369 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8ab11e0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 -- 172.21.15.31:0/3870057369 >> 172.21.15.31:0/3870057369 conn(0x55c0d8a913e0 msgr2=0x55c0d8a97730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 -- 172.21.15.31:0/3870057369 shutdown_connections 2023-07-29T16:29:36.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.643+0000 7f94449d7b80 1 -- 172.21.15.31:0/3870057369 wait complete. 2023-07-29T16:29:36.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.644+0000 7f94449d7b80 1 -- start start 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.644+0000 7f94449d7b80 1 Processor -- start 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8ab0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8aad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8aadcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8aae3d0 con 0x55c0d8a9fff0 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8ab1390 con 0x55c0d8aa1700 2023-07-29T16:29:36.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0d8ab1520 con 0x55c0d8aa0940 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f94449d7b80 10 client.0 _ll_register_callbacks cb 0x55c0d8992620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8aadcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8ab0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8aadcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39634/0 (socket says 172.21.15.31:39634) 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 -- 172.21.15.31:0/4138781556 learned_addr learned my addr 172.21.15.31:0/4138781556 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f9429d9b700 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8aad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 -- 172.21.15.31:0/4138781556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 msgr2=0x55c0d8aad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8aad7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 -- 172.21.15.31:0/4138781556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 msgr2=0x55c0d8ab0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8ab0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.645+0000 7f942ad9d700 1 -- 172.21.15.31:0/4138781556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0d8803c60 con 0x55c0d8aa1700 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f942ad9d700 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8aadcf0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f942401a910 tx=0x7f9424004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7643d23ae458fa7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f9411ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f9411ffb700 1 -- 172.21.15.31:0/4138781556 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94240041e0 con 0x55c0d8aa1700 2023-07-29T16:29:36.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f9411ffb700 1 -- 172.21.15.31:0/4138781556 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9424004340 con 0x55c0d8aa1700 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f941c09d4c0 con 0x55c0d8aa1700 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f9411ffb700 1 -- 172.21.15.31:0/4138781556 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9424004500 con 0x55c0d8aa1700 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f94449d7b80 10 client.29702 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.646+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7f942c109030 con 0x55c0d8aa1700 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f9411ffb700 1 -- 172.21.15.31:0/4138781556 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f941c09d4c0 con 0x55c0d8aa1700 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f9411ffb700 1 -- 172.21.15.31:0/4138781556 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9424025050 con 0x55c0d8aa1700 2023-07-29T16:29:36.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f94449d7b80 10 client.29702 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f94449d7b80 10 client.29702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f94449d7b80 20 client.29702 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f94449d7b80 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8b08950 0x55c0d8b0ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f94449d7b80 4 client.29702 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c0d89e5d40 con 0x55c0d8b08950 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f942a59c700 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8b08950 0x55c0d8b0ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f942a59c700 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8b08950 0x55c0d8b0ad60 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=716e703dcd5de885 server_cookie=5cbbf6817d7aeba7 in_seq=0 out_seq=0 2023-07-29T16:29:36.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.647+0000 7f9411ffb700 10 client.29702 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f9411ffb700 1 -- 172.21.15.31:0/4138781556 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c0d89e5d40 con 0x55c0d8b08950 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f9411ffb700 10 client.29702 handle_command_reply: tid=0 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 client.29702 shutdown 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 msgr2=0x55c0d8aadcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8aadcf0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f942401a910 tx=0x7f9424004ad0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 shutdown_connections 2023-07-29T16:29:36.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c0d8b08950 0x55c0d8b0ad60 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0d8aa1700 0x55c0d8aadcf0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c0d8a9fff0 0x55c0d8ab0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 --2- 172.21.15.31:0/4138781556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c0d8aa0940 0x55c0d8aad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 >> 172.21.15.31:0/4138781556 conn(0x55c0d8a913e0 msgr2=0x55c0d8aa7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f9411ffb700 0 client.29702 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 shutdown_connections 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 1 -- 172.21.15.31:0/4138781556 wait complete. 2023-07-29T16:29:36.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.648+0000 7f94449d7b80 20 client.29702 trim_cache size 0 max 16384 2023-07-29T16:29:36.657 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:29:36.657 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94140a0810 msgr2=0x7f94140a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94140a0810 0x7f94140a2cc0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x55c0d8a9fd90 tx=0x7f9424009130 comp rx=0 tx=0).stop 2023-07-29T16:29:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 msgr2=0x7f942c1960a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c1960a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f941c00e9a0 tx=0x7f941c00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 shutdown_connections 2023-07-29T16:29:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f94140a0810 0x7f94140a2cc0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.658 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f942c105fe0 0x7f942c19d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f942c100da0 0x7f942c1960a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 --2- 172.21.15.31:0/1527178314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f942c1036c0 0x7f942c1965e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:36.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 >> 172.21.15.31:0/1527178314 conn(0x7f942c0fa840 msgr2=0x7f942c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:36.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 shutdown_connections 2023-07-29T16:29:36.659 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:36.649+0000 7f9430fff700 1 -- 172.21.15.31:0/1527178314 wait complete. 2023-07-29T16:29:36.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:36 smithi031 ceph-mon[95504]: pgmap v795: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.4 MiB/s wr, 230 op/s 2023-07-29T16:29:36.667 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:36 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2897573811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:29:36.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1117186 8.84 MB/sec execute 561 sec latency 30.002 ms 2023-07-29T16:29:37.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:36 smithi039 ceph-mon[128858]: pgmap v795: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.4 MiB/s wr, 230 op/s 2023-07-29T16:29:37.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:36 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2897573811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:29:37.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:36 smithi156 ceph-mon[129245]: pgmap v795: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.4 MiB/s wr, 230 op/s 2023-07-29T16:29:37.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:36 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2897573811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:29:37.670 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:29:37.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:29:37.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 -- 172.21.15.31:0/4010463117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 msgr2=0x7fc5b81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/4010463117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 0x7fc5b81031f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009990 tx=0x7fc5a8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 -- 172.21.15.31:0/4010463117 shutdown_connections 2023-07-29T16:29:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/4010463117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5b8106050 0x7fc5b81084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/4010463117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 0x7fc5b81031f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/4010463117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b8105b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 -- 172.21.15.31:0/4010463117 >> 172.21.15.31:0/4010463117 conn(0x7fc5b80fa8b0 msgr2=0x7fc5b80fcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 -- 172.21.15.31:0/4010463117 shutdown_connections 2023-07-29T16:29:37.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.889+0000 7fc5bdbff700 1 -- 172.21.15.31:0/4010463117 wait complete. 2023-07-29T16:29:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.890+0000 7fc5bdbff700 1 Processor -- start 2023-07-29T16:29:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.890+0000 7fc5bdbff700 1 -- start start 2023-07-29T16:29:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5bdbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 0x7fc5b8198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5bdbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b81987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5bdbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5b8106050 0x7fc5b819f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5bdbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b819fe00 con 0x7fc5b8100e10 2023-07-29T16:29:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5bdbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b819ff40 con 0x7fc5b8103730 2023-07-29T16:29:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5bdbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b81a00d0 con 0x7fc5b8106050 2023-07-29T16:29:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b81987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b81987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39656/0 (socket says 172.21.15.31:39656) 2023-07-29T16:29:37.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/822153341 learned_addr learned my addr 172.21.15.31:0/822153341 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 0x7fc5b8198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5b8106050 0x7fc5b819f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/822153341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5b8106050 msgr2=0x7fc5b819f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5b8106050 0x7fc5b819f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 msgr2=0x7fc5b8198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 0x7fc5b8198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/822153341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a80095f0 con 0x7fc5b8103730 2023-07-29T16:29:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.891+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b81987c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00e9a0 tx=0x7fc5ac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.892+0000 7fc5b4ff9700 1 -- 172.21.15.31:0/822153341 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac009790 con 0x7fc5b8103730 2023-07-29T16:29:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.892+0000 7fc5b4ff9700 1 -- 172.21.15.31:0/822153341 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5ac010430 con 0x7fc5b8103730 2023-07-29T16:29:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.892+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b81a03b0 con 0x7fc5b8103730 2023-07-29T16:29:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.892+0000 7fc5b4ff9700 1 -- 172.21.15.31:0/822153341 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac018680 con 0x7fc5b8103730 2023-07-29T16:29:37.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.892+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b81a08c0 con 0x7fc5b8103730 2023-07-29T16:29:37.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.897+0000 7fc5b4ff9700 1 -- 172.21.15.31:0/822153341 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc5ac0105a0 con 0x7fc5b8103730 2023-07-29T16:29:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 Processor -- start 2023-07-29T16:29:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 -- start start 2023-07-29T16:29:37.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c100f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c100fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c10107c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c0f66a90 con 0x5621c100f5b0 2023-07-29T16:29:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c0f64450 con 0x5621c1010370 2023-07-29T16:29:37.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c0f5fd40 con 0x5621c100ec60 2023-07-29T16:29:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c10107c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c100fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c10107c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39668/0 (socket says 172.21.15.31:39668) 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/1340893194 learned_addr learned my addr 172.21.15.31:0/1340893194 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.898+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c100f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/1340893194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 msgr2=0x5621c100f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c100f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/1340893194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 msgr2=0x5621c100fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c100fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/1340893194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621c0d8de60 con 0x5621c1010370 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c100f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b4ff9700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5a00a0790 0x7fc5a00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c10107c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0cfb90 tx=0x7fc5ac0cfea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6e5457430d48091 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5a00a0790 0x7fc5a00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc5b4ff9700 1 -- 172.21.15.31:0/822153341 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc5ac014070 con 0x7fc5b8103730 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.899+0000 7fc59dffb700 1 -- 172.21.15.31:0/1340893194 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0957c0 con 0x5621c1010370 2023-07-29T16:29:37.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc59dffb700 1 -- 172.21.15.31:0/1340893194 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5ac099250 con 0x5621c1010370 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc59dffb700 1 -- 172.21.15.31:0/1340893194 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac099480 con 0x5621c1010370 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1340893194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 msgr2=0x5621c10107c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c10107c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac0cfb90 tx=0x7fc5ac0cfea0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5a00a0790 0x7fc5a00a2c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x5621c100ea00 tx=0x7fc5b000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1340893194 shutdown_connections 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c10107c0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c100f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1340893194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c100fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1340893194 >> 172.21.15.31:0/1340893194 conn(0x5621c1004460 msgr2=0x5621c1009a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1340893194 shutdown_connections 2023-07-29T16:29:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.900+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1340893194 wait complete. 2023-07-29T16:29:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.901+0000 7fc5d1515b80 1 -- start start 2023-07-29T16:29:37.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.901+0000 7fc5d1515b80 1 Processor -- start 2023-07-29T16:29:37.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c10187d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c1018d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c101fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c0efc590 con 0x5621c100f5b0 2023-07-29T16:29:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c0efc230 con 0x5621c1010370 2023-07-29T16:29:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c0de39a0 con 0x5621c100ec60 2023-07-29T16:29:37.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5d1515b80 10 client.0 _ll_register_callbacks cb 0x5621c0f056a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c10187d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c101fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c10187d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55200/0 (socket says 172.21.15.31:55200) 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 -- 172.21.15.31:0/698885620 learned_addr learned my addr 172.21.15.31:0/698885620 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c1018d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 -- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 msgr2=0x5621c101fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c101fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 -- 172.21.15.31:0/698885620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 msgr2=0x5621c1018d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c1018d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b77fe700 1 -- 172.21.15.31:0/698885620 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b0011230 con 0x5621c100ec60 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c101fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.902+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c1018d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c10187d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009990 tx=0x7fc5a8009570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41d4ef4cfe02dd62 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc596ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:37.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc596ffd700 1 -- 172.21.15.31:0/698885620 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a801d030 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621c0f8ad90 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc596ffd700 1 -- 172.21.15.31:0/698885620 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc5a8004500 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc596ffd700 1 -- 172.21.15.31:0/698885620 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5a80046c0 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc5d1515b80 10 client.29713 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.903+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7fc5b81a08c0 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc596ffd700 1 -- 172.21.15.31:0/698885620 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5621c0f8ad90 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc596ffd700 1 -- 172.21.15.31:0/698885620 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc5a800f9c0 con 0x5621c100ec60 2023-07-29T16:29:37.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5d1515b80 10 client.29713 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5d1515b80 10 client.29713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5d1515b80 20 client.29713 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c10245b0 0x5621c10269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5d1515b80 4 client.29713 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:37.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621c0fdddf0 con 0x5621c10245b0 2023-07-29T16:29:37.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c10245b0 0x5621c10269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.913 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1118979 8.84 MB/sec execute 562 sec latency 46.214 ms 2023-07-29T16:29:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c10245b0 0x5621c10269c0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36f952567934cc31 server_cookie=33e8f23f3551163 in_seq=0 out_seq=0 2023-07-29T16:29:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.904+0000 7fc596ffd700 10 client.29713 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.905+0000 7fc596ffd700 1 -- 172.21.15.31:0/698885620 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5621c0fdddf0 con 0x5621c10245b0 2023-07-29T16:29:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.905+0000 7fc596ffd700 10 client.29713 handle_command_reply: tid=0 2023-07-29T16:29:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.905+0000 7fc5d1515b80 1 client.29713 shutdown 2023-07-29T16:29:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 msgr2=0x5621c10187d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c10187d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009990 tx=0x7fc5a8009570 comp rx=0 tx=0).stop 2023-07-29T16:29:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 shutdown_connections 2023-07-29T16:29:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c10245b0 0x5621c10269c0 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c1010370 0x5621c101fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c10187d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc596ffd700 0 client.29713 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/698885620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100f5b0 0x5621c1018d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 >> 172.21.15.31:0/698885620 conn(0x5621c1004460 msgr2=0x5621c10090e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 shutdown_connections 2023-07-29T16:29:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 1 -- 172.21.15.31:0/698885620 wait complete. 2023-07-29T16:29:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.906+0000 7fc5d1515b80 20 client.29713 trim_cache size 0 max 16384 2023-07-29T16:29:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.923+0000 7fc5d1515b80 1 Processor -- start 2023-07-29T16:29:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.923+0000 7fc5d1515b80 1 -- start start 2023-07-29T16:29:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100ec60 0x5621c101dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c1018580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.924 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c1010370 0x5621c1018ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c10183a0 con 0x5621c100ec60 2023-07-29T16:29:37.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c10017f0 con 0x5621c100f5b0 2023-07-29T16:29:37.925 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c1016500 con 0x5621c1010370 2023-07-29T16:29:37.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100ec60 0x5621c101dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.926 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c1018580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c1018580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39686/0 (socket says 172.21.15.31:39686) 2023-07-29T16:29:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/3267444341 learned_addr learned my addr 172.21.15.31:0/3267444341 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c1010370 0x5621c1018ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/3267444341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c1010370 msgr2=0x5621c1018ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c1010370 0x5621c1018ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/3267444341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100ec60 msgr2=0x5621c101dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100ec60 0x5621c101dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.924+0000 7fc5b6ffd700 1 -- 172.21.15.31:0/3267444341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621c0d76ce0 con 0x5621c100f5b0 2023-07-29T16:29:37.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.925+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c1018580 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac095ea0 tx=0x7fc5ac010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efd5304f6b5df79 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:37.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.925+0000 7fc59e7fc700 1 -- 172.21.15.31:0/3267444341 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0cbe80 con 0x5621c100f5b0 2023-07-29T16:29:37.928 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.925+0000 7fc59e7fc700 1 -- 172.21.15.31:0/3267444341 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5ac0cf170 con 0x5621c100f5b0 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.925+0000 7fc59e7fc700 1 -- 172.21.15.31:0/3267444341 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ac0a18d0 con 0x5621c100f5b0 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.925+0000 7fc5d1515b80 1 -- 172.21.15.31:0/3267444341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 msgr2=0x5621c1018580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c1018580 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac095ea0 tx=0x7fc5ac010c80 comp rx=0 tx=0).stop 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 -- 172.21.15.31:0/3267444341 shutdown_connections 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c1010370 0x5621c1018ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c100ec60 0x5621c101dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/3267444341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c1018580 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 -- 172.21.15.31:0/3267444341 >> 172.21.15.31:0/3267444341 conn(0x5621c1004460 msgr2=0x5621c10132d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 -- 172.21.15.31:0/3267444341 shutdown_connections 2023-07-29T16:29:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 -- 172.21.15.31:0/3267444341 wait complete. 2023-07-29T16:29:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 -- start start 2023-07-29T16:29:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.926+0000 7fc5d1515b80 1 Processor -- start 2023-07-29T16:29:37.932 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c101d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c101d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c1010370 0x5621c10764f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c1076a30 con 0x5621c1010370 2023-07-29T16:29:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c1076b70 con 0x5621c100f5b0 2023-07-29T16:29:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621c1076d30 con 0x5621c100ec60 2023-07-29T16:29:37.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5d1515b80 10 client.0 _ll_register_callbacks cb 0x5621c0f056a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c101d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c101d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c101d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55214/0 (socket says 172.21.15.31:55214) 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 -- 172.21.15.31:0/1078060602 learned_addr learned my addr 172.21.15.31:0/1078060602 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b7fff700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c1010370 0x5621c10764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 -- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 msgr2=0x5621c101d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c101d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 -- 172.21.15.31:0/1078060602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c1010370 msgr2=0x5621c10764f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.927+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c1010370 0x5621c10764f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc5b77fe700 1 -- 172.21.15.31:0/1078060602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621c0d8de60 con 0x5621c100ec60 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c101d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc5b77fe700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c101d0d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800b680 tx=0x7fc5a800fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5ada19b1de67379 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc596ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:29:37.935 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc596ffd700 1 -- 172.21.15.31:0/1078060602 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a801d070 con 0x5621c100ec60 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621c0f58dc0 con 0x5621c100ec60 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc596ffd700 1 -- 172.21.15.31:0/1078060602 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc5a8000e40 con 0x5621c100ec60 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc596ffd700 1 -- 172.21.15.31:0/1078060602 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5a8024450 con 0x5621c100ec60 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc5d1515b80 10 client.29719 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.928+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5621c0db42d0 con 0x5621c100ec60 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc596ffd700 1 -- 172.21.15.31:0/1078060602 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5621c0f58dc0 con 0x5621c100ec60 2023-07-29T16:29:37.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc596ffd700 1 -- 172.21.15.31:0/1078060602 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fc5a8004730 con 0x5621c100ec60 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5d1515b80 10 client.29719 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5d1515b80 10 client.29719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5d1515b80 20 client.29719 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c1078250 0x5621c107a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5d1515b80 4 client.29719 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5621c0f58ae0 con 0x5621c1078250 2023-07-29T16:29:37.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.929+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c1078250 0x5621c107a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.930+0000 7fc5b6ffd700 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c1078250 0x5621c107a630 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e04c2f05d233487 server_cookie=cdbeef17ad8d86d3 in_seq=0 out_seq=0 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.930+0000 7fc596ffd700 10 client.29719 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc596ffd700 1 -- 172.21.15.31:0/1078060602 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5621c0f58ae0 con 0x5621c1078250 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc596ffd700 10 client.29719 handle_command_reply: tid=0 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 client.29719 shutdown 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 msgr2=0x5621c101d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.938 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c101d0d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800b680 tx=0x7fc5a800fb40 comp rx=0 tx=0).stop 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 shutdown_connections 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5621c1078250 0x5621c107a630 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621c1010370 0x5621c10764f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5621c100ec60 0x5621c101d0d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 --2- 172.21.15.31:0/1078060602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5621c100f5b0 0x5621c101d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc596ffd700 0 client.29719 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.931+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 >> 172.21.15.31:0/1078060602 conn(0x5621c1004460 msgr2=0x5621c1018580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 shutdown_connections 2023-07-29T16:29:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5d1515b80 1 -- 172.21.15.31:0/1078060602 wait complete. 2023-07-29T16:29:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5d1515b80 20 client.29719 trim_cache size 0 max 16384 2023-07-29T16:29:37.940 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:29:37.940 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:29:37.940 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "11d8bcdb-25ad-45d9-9864-b2aeec38d3eb", 2023-07-29T16:29:37.941 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:29:37.941 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:29:37.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5a00a0790 msgr2=0x7fc5a00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5a00a0790 0x7fc5a00a2c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x5621c100ea00 tx=0x7fc5b000b040 comp rx=0 tx=0).stop 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 msgr2=0x7fc5b81987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.932+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b81987c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00e9a0 tx=0x7fc5ac00ecb0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 shutdown_connections 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc5a00a0790 0x7fc5a00a2c40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc5b8106050 0x7fc5b819f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.942 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5b8100e10 0x7fc5b8198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 --2- 172.21.15.31:0/822153341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc5b8103730 0x7fc5b81987c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 >> 172.21.15.31:0/822153341 conn(0x7fc5b80fa8b0 msgr2=0x7fc5b80ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 shutdown_connections 2023-07-29T16:29:37.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:37.933+0000 7fc5bdbff700 1 -- 172.21.15.31:0/822153341 wait complete. 2023-07-29T16:29:37.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:29:38.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 -- 172.21.15.31:0/3258312608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450101ea0 msgr2=0x7f8450108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 --2- 172.21.15.31:0/3258312608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450101ea0 0x7f8450108760 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f844800b0a0 tx=0x7f844800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 -- 172.21.15.31:0/3258312608 shutdown_connections 2023-07-29T16:29:38.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 --2- 172.21.15.31:0/3258312608 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450101ea0 0x7f8450108760 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 --2- 172.21.15.31:0/3258312608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450100320 0x7f8450100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 --2- 172.21.15.31:0/3258312608 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8450101510 0x7f8450101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 -- 172.21.15.31:0/3258312608 >> 172.21.15.31:0/3258312608 conn(0x7f84500fb920 msgr2=0x7f84500fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:38.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 -- 172.21.15.31:0/3258312608 shutdown_connections 2023-07-29T16:29:38.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.167+0000 7f84553ff700 1 -- 172.21.15.31:0/3258312608 wait complete. 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.168+0000 7f84553ff700 1 Processor -- start 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.168+0000 7f84553ff700 1 -- start start 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f84553ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 0x7f8450074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f84553ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8450101510 0x7f8450073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f84553ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450101ea0 0x7f8450073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f84553ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450073e20 con 0x7f8450101ea0 2023-07-29T16:29:38.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f84553ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450073f90 con 0x7f8450100320 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f84553ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84501a01f0 con 0x7f8450101510 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 0x7f8450074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 0x7f8450074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39708/0 (socket says 172.21.15.31:39708) 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 -- 172.21.15.31:0/3009707976 learned_addr learned my addr 172.21.15.31:0/3009707976 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844e7fc700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8450101510 0x7f8450073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844f7fe700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450101ea0 0x7f8450073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 -- 172.21.15.31:0/3009707976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8450101510 msgr2=0x7f8450073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8450101510 0x7f8450073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 -- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450101ea0 msgr2=0x7f8450073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.169+0000 7f844effd700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450101ea0 0x7f8450073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f844effd700 1 -- 172.21.15.31:0/3009707976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8448009d00 con 0x7f8450100320 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f844f7fe700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450101ea0 0x7f8450073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f844effd700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 0x7f8450074b00 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f844000e970 tx=0x7f844000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f8437fff700 1 -- 172.21.15.31:0/3009707976 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440009790 con 0x7f8450100320 2023-07-29T16:29:38.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f8437fff700 1 -- 172.21.15.31:0/3009707976 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f844000eec0 con 0x7f8450100320 2023-07-29T16:29:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f8437fff700 1 -- 172.21.15.31:0/3009707976 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440003cf0 con 0x7f8450100320 2023-07-29T16:29:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84501a04d0 con 0x7f8450100320 2023-07-29T16:29:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.170+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84501a09e0 con 0x7f8450100320 2023-07-29T16:29:38.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.173+0000 7f8437fff700 1 -- 172.21.15.31:0/3009707976 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f844000ccd0 con 0x7f8450100320 2023-07-29T16:29:38.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.175+0000 7f8468c88b80 1 Processor -- start 2023-07-29T16:29:38.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.175+0000 7f8468c88b80 1 -- start start 2023-07-29T16:29:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449395be0 0x562449395ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 0x562449396d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x562449397740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8468c88b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624492eda10 con 0x562449395be0 2023-07-29T16:29:38.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8468c88b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624492eb3d0 con 0x562449396530 2023-07-29T16:29:38.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8468c88b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624492e6cc0 con 0x5624493972f0 2023-07-29T16:29:38.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8437fff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84380a0810 0x7f84380a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 0x562449396d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449395be0 0x562449395ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844e7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x562449397740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 0x562449396d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39716/0 (socket says 172.21.15.31:39716) 2023-07-29T16:29:38.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 -- 172.21.15.31:0/3082205246 learned_addr learned my addr 172.21.15.31:0/3082205246 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:38.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844e7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x562449397740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55242/0 (socket says 172.21.15.31:55242) 2023-07-29T16:29:38.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f8437fff700 1 -- 172.21.15.31:0/3009707976 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8440014070 con 0x7f8450100320 2023-07-29T16:29:38.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844f7fe700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84380a0810 0x7f84380a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.181 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 -- 172.21.15.31:0/3082205246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 msgr2=0x562449397740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x562449397740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 -- 172.21.15.31:0/3082205246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449395be0 msgr2=0x562449395ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449395be0 0x562449395ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844effd700 1 -- 172.21.15.31:0/3082205246 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562449114de0 con 0x562449396530 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844e7fc700 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x562449397740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.176+0000 7f844f7fe700 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449395be0 0x562449395ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:38.182 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.177+0000 7f844effd700 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 0x562449396d60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f84400a1390 tx=0x7f84400992c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d791930fce7c83c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.177+0000 7f84357fa700 1 -- 172.21.15.31:0/3082205246 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440099470 con 0x562449396530 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.177+0000 7f844f7fe700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84380a0810 0x7f84380a2cc0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8450075160 tx=0x7f844800b3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.177+0000 7f84357fa700 1 -- 172.21.15.31:0/3082205246 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8440016330 con 0x562449396530 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.177+0000 7f84357fa700 1 -- 172.21.15.31:0/3082205246 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84400a1430 con 0x562449396530 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 -- 172.21.15.31:0/3082205246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 msgr2=0x562449396d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 0x562449396d60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f84400a1390 tx=0x7f84400992c0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.183 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 -- 172.21.15.31:0/3082205246 shutdown_connections 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x562449397740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449395be0 0x562449395ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3082205246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449396530 0x562449396d60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 -- 172.21.15.31:0/3082205246 >> 172.21.15.31:0/3082205246 conn(0x56244938b3e0 msgr2=0x562449390990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 -- 172.21.15.31:0/3082205246 shutdown_connections 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 -- 172.21.15.31:0/3082205246 wait complete. 2023-07-29T16:29:38.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.178+0000 7f8468c88b80 1 -- start start 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 Processor -- start 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x56244939fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562449283510 con 0x562449396530 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624492831b0 con 0x562449395be0 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56244916a920 con 0x5624493972f0 2023-07-29T16:29:38.185 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f8468c88b80 10 client.0 _ll_register_callbacks cb 0x56244928c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.179+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39728/0 (socket says 172.21.15.31:39728) 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 -- 172.21.15.31:0/2999126578 learned_addr learned my addr 172.21.15.31:0/2999126578 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x56244939fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844f7fe700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 -- 172.21.15.31:0/2999126578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 msgr2=0x5624493a6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 -- 172.21.15.31:0/2999126578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 msgr2=0x56244939fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x56244939fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 -- 172.21.15.31:0/2999126578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8450109090 con 0x562449395be0 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x56244939fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f844effd700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939f7d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8440099170 tx=0x7f8440098e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc852581e0d8479 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f8432ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f8432ff5700 1 -- 172.21.15.31:0/2999126578 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440099470 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f8432ff5700 1 -- 172.21.15.31:0/2999126578 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f84400a18c0 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.180+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562449311d10 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8432ff5700 1 -- 172.21.15.31:0/2999126578 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f844009d140 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 10 client.29732 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7f84440098e0 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8432ff5700 1 -- 172.21.15.31:0/2999126578 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562449311d10 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8432ff5700 1 -- 172.21.15.31:0/2999126578 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f84400983d0 con 0x562449395be0 2023-07-29T16:29:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 10 client.29732 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:38.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 10 client.29732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:38.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 20 client.29732 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:38.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493ab0d0 0x5624493ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 4 client.29732 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:38.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562449364d70 con 0x5624493ab0d0 2023-07-29T16:29:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.181+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493ab0d0 0x5624493ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.182+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493ab0d0 0x5624493ad4b0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c98613e1dd02b16 server_cookie=7bf4a50d052f2426 in_seq=0 out_seq=0 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.182+0000 7f8432ff5700 10 client.29732 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8432ff5700 1 -- 172.21.15.31:0/2999126578 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562449364d70 con 0x5624493ab0d0 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8432ff5700 10 client.29732 handle_command_reply: tid=0 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 client.29732 shutdown 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 msgr2=0x56244939f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939f7d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f8440099170 tx=0x7f8440098e90 comp rx=0 tx=0).stop 2023-07-29T16:29:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 shutdown_connections 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493ab0d0 0x5624493ad4b0 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939f7d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2999126578 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x56244939fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8432ff5700 0 client.29732 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 >> 172.21.15.31:0/2999126578 conn(0x56244938b3e0 msgr2=0x562449390060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:38.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 shutdown_connections 2023-07-29T16:29:38.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.183+0000 7f8468c88b80 1 -- 172.21.15.31:0/2999126578 wait complete. 2023-07-29T16:29:38.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.184+0000 7f8468c88b80 20 client.29732 trim_cache size 0 max 16384 2023-07-29T16:29:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.201+0000 7f8468c88b80 1 Processor -- start 2023-07-29T16:29:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.201+0000 7f8468c88b80 1 -- start start 2023-07-29T16:29:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x56244939a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f8468c88b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56244938aea0 con 0x562449396530 2023-07-29T16:29:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f8468c88b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56244938d460 con 0x562449395be0 2023-07-29T16:29:38.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f8468c88b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562449391cc0 con 0x5624493972f0 2023-07-29T16:29:38.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39732/0 (socket says 172.21.15.31:39732) 2023-07-29T16:29:38.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 -- 172.21.15.31:0/3639776352 learned_addr learned my addr 172.21.15.31:0/3639776352 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844f7fe700 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x56244939a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 -- 172.21.15.31:0/3639776352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 msgr2=0x56244939a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x56244939a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 -- 172.21.15.31:0/3639776352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 msgr2=0x5624493a6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844effd700 1 -- 172.21.15.31:0/3639776352 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84501a09e0 con 0x562449395be0 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.202+0000 7f844e7fc700 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f844effd700 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939a330 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f84400977c0 tx=0x7f8440098f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb56f4f2267db2f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8435ffb700 1 -- 172.21.15.31:0/3639776352 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844009d4a0 con 0x562449395be0 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8435ffb700 1 -- 172.21.15.31:0/3639776352 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8440098960 con 0x562449395be0 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8435ffb700 1 -- 172.21.15.31:0/3639776352 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844009d140 con 0x562449395be0 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 -- 172.21.15.31:0/3639776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 msgr2=0x56244939a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939a330 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f84400977c0 tx=0x7f8440098f00 comp rx=0 tx=0).stop 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 -- 172.21.15.31:0/3639776352 shutdown_connections 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x56244939a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x56244939a330 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 --2- 172.21.15.31:0/3639776352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.203+0000 7f8468c88b80 1 -- 172.21.15.31:0/3639776352 >> 172.21.15.31:0/3639776352 conn(0x56244938b3e0 msgr2=0x5624493a75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.204+0000 7f8468c88b80 1 -- 172.21.15.31:0/3639776352 shutdown_connections 2023-07-29T16:29:38.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.204+0000 7f8468c88b80 1 -- 172.21.15.31:0/3639776352 wait complete. 2023-07-29T16:29:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.204+0000 7f8468c88b80 1 -- start start 2023-07-29T16:29:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.204+0000 7f8468c88b80 1 Processor -- start 2023-07-29T16:29:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x5624493a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624493a7110 con 0x562449396530 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624493a59b0 con 0x562449395be0 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624493a5b70 con 0x5624493972f0 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8468c88b80 10 client.0 _ll_register_callbacks cb 0x56244928c620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x5624493a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x5624493a68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39736/0 (socket says 172.21.15.31:39736) 2023-07-29T16:29:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 -- 172.21.15.31:0/2466173203 learned_addr learned my addr 172.21.15.31:0/2466173203 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844f7fe700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 -- 172.21.15.31:0/2466173203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 msgr2=0x5624493a5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 -- 172.21.15.31:0/2466173203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 msgr2=0x5624493a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:38.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 -- 172.21.15.31:0/2466173203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844401f8b0 con 0x562449395be0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f844effd700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x5624493a68f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8440099170 tx=0x7f84400164a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5555772366ae413 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.205+0000 7f8432ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8432ff5700 1 -- 172.21.15.31:0/2466173203 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844009d4a0 con 0x562449395be0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8432ff5700 1 -- 172.21.15.31:0/2466173203 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f84400a18c0 con 0x562449395be0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8440016330 con 0x562449395be0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8432ff5700 1 -- 172.21.15.31:0/2466173203 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84400984d0 con 0x562449395be0 2023-07-29T16:29:38.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 10 client.29738 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7f84440098e0 con 0x562449395be0 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8432ff5700 1 -- 172.21.15.31:0/2466173203 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8440016330 con 0x562449395be0 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8432ff5700 1 -- 172.21.15.31:0/2466173203 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f84400c92c0 con 0x562449395be0 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 10 client.29738 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 10 client.29738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 20 client.29738 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:38.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493fdd20 0x562449400130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 4 client.29738 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.206+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5624492dfd40 con 0x5624493fdd20 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493fdd20 0x562449400130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f844e7fc700 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493fdd20 0x562449400130 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee119151ce74be0a server_cookie=9d042f12074e3dce in_seq=0 out_seq=0 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8432ff5700 10 client.29738 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8432ff5700 1 -- 172.21.15.31:0/2466173203 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5624492dfd40 con 0x5624493fdd20 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8432ff5700 10 client.29738 handle_command_reply: tid=0 2023-07-29T16:29:38.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8468c88b80 1 client.29738 shutdown 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 msgr2=0x5624493a68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x5624493a68f0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8440099170 tx=0x7f84400164a0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 shutdown_connections 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.207+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5624493fdd20 0x562449400130 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5624493972f0 0x5624493a5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562449395be0 0x5624493a68f0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 1 --2- 172.21.15.31:0/2466173203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562449396530 0x5624493a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 >> 172.21.15.31:0/2466173203 conn(0x56244938b3e0 msgr2=0x56244939a330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8432ff5700 0 client.29738 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 shutdown_connections 2023-07-29T16:29:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 1 -- 172.21.15.31:0/2466173203 wait complete. 2023-07-29T16:29:38.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f8468c88b80 20 client.29738 trim_cache size 0 max 16384 2023-07-29T16:29:38.219 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:29:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:29:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:29:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "11d8bcdb-25ad-45d9-9864-b2aeec38d3eb": { 2023-07-29T16:29:38.219 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:29:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "11d8bcdb-25ad-45d9-9864-b2aeec38d3eb", 2023-07-29T16:29:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:29:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:29:38.220 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:29:38.220 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:29:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84380a0810 msgr2=0x7f84380a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f84553ff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84380a0810 0x7f84380a2cc0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8450075160 tx=0x7f844800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 msgr2=0x7f8450074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.208+0000 7f84553ff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 0x7f8450074b00 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f844000e970 tx=0x7f844000ec80 comp rx=0 tx=0).stop 2023-07-29T16:29:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 shutdown_connections 2023-07-29T16:29:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f84380a0810 0x7f84380a2cc0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8450101ea0 0x7f8450073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8450100320 0x7f8450074b00 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 --2- 172.21.15.31:0/3009707976 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8450101510 0x7f8450073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 >> 172.21.15.31:0/3009707976 conn(0x7f84500fb920 msgr2=0x7f84500fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 shutdown_connections 2023-07-29T16:29:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:29:38.209+0000 7f84553ff700 1 -- 172.21.15.31:0/3009707976 wait complete. 2023-07-29T16:29:38.227 INFO:tasks.cephfs.filesystem:scrub status for tag:11d8bcdb-25ad-45d9-9864-b2aeec38d3eb - {'path': '/', 'tag': '11d8bcdb-25ad-45d9-9864-b2aeec38d3eb', 'options': 'recursive,force'} 2023-07-29T16:29:38.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:38 smithi039 ceph-mon[128858]: pgmap v796: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 2.0 MiB/s wr, 206 op/s 2023-07-29T16:29:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:38 smithi156 ceph-mon[129245]: pgmap v796: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 2.0 MiB/s wr, 206 op/s 2023-07-29T16:29:38.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:38 smithi031 ceph-mon[95504]: pgmap v796: 97 pgs: 97 active+clean; 207 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 2.0 MiB/s wr, 206 op/s 2023-07-29T16:29:38.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1119842 8.83 MB/sec execute 563 sec latency 21.654 ms 2023-07-29T16:29:39.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:29:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:29:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:39 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:29:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:29:39.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:39 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:29:39.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:39 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:29:39.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1122049 8.84 MB/sec execute 564 sec latency 22.553 ms 2023-07-29T16:29:40.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:40 smithi039 ceph-mon[128858]: pgmap v797: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 2.4 MiB/s wr, 245 op/s 2023-07-29T16:29:40.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:40 smithi156 ceph-mon[129245]: pgmap v797: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 2.4 MiB/s wr, 245 op/s 2023-07-29T16:29:40.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:40 smithi031 ceph-mon[95504]: pgmap v797: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 2.4 MiB/s wr, 245 op/s 2023-07-29T16:29:40.904 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1123020 8.83 MB/sec execute 565 sec latency 50.457 ms 2023-07-29T16:29:41.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1125173 8.83 MB/sec execute 566 sec latency 15.113 ms 2023-07-29T16:29:42.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1126385 8.83 MB/sec execute 567 sec latency 30.145 ms 2023-07-29T16:29:42.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:42 smithi031 ceph-mon[95504]: pgmap v798: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2023-07-29T16:29:43.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:42 smithi039 ceph-mon[128858]: pgmap v798: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2023-07-29T16:29:43.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:42 smithi156 ceph-mon[129245]: pgmap v798: 97 pgs: 97 active+clean; 206 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 1.9 MiB/s wr, 201 op/s 2023-07-29T16:29:43.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1128227 8.83 MB/sec execute 568 sec latency 30.876 ms 2023-07-29T16:29:44.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1129822 8.84 MB/sec execute 569 sec latency 18.215 ms 2023-07-29T16:29:44.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:44 smithi031 ceph-mon[95504]: pgmap v799: 97 pgs: 97 active+clean; 209 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-07-29T16:29:45.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:44 smithi039 ceph-mon[128858]: pgmap v799: 97 pgs: 97 active+clean; 209 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-07-29T16:29:45.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:44 smithi156 ceph-mon[129245]: pgmap v799: 97 pgs: 97 active+clean; 209 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 2.5 MiB/s wr, 289 op/s 2023-07-29T16:29:45.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1131210 8.83 MB/sec execute 570 sec latency 50.005 ms 2023-07-29T16:29:46.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1133332 8.84 MB/sec execute 571 sec latency 14.720 ms 2023-07-29T16:29:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:46 smithi031 ceph-mon[95504]: pgmap v800: 97 pgs: 97 active+clean; 208 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.0 MiB/s wr, 231 op/s 2023-07-29T16:29:47.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:46 smithi039 ceph-mon[128858]: pgmap v800: 97 pgs: 97 active+clean; 208 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.0 MiB/s wr, 231 op/s 2023-07-29T16:29:47.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:46 smithi156 ceph-mon[129245]: pgmap v800: 97 pgs: 97 active+clean; 208 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 2.0 MiB/s wr, 231 op/s 2023-07-29T16:29:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 -- 172.21.15.31:0/1563953266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101540 msgr2=0x7f2174101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 --2- 172.21.15.31:0/1563953266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101540 0x7f2174101990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2160009990 tx=0x7f2160009ca0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 -- 172.21.15.31:0/1563953266 shutdown_connections 2023-07-29T16:29:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 --2- 172.21.15.31:0/1563953266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2174101ed0 0x7f2174108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 --2- 172.21.15.31:0/1563953266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 --2- 172.21.15.31:0/1563953266 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101540 0x7f2174101990 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 -- 172.21.15.31:0/1563953266 >> 172.21.15.31:0/1563953266 conn(0x7f21740fb970 msgr2=0x7f21740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 -- 172.21.15.31:0/1563953266 shutdown_connections 2023-07-29T16:29:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.754+0000 7f217afff700 1 -- 172.21.15.31:0/1563953266 wait complete. 2023-07-29T16:29:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.755+0000 7f217afff700 1 Processor -- start 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 -- start start 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2174101540 0x7f2174073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101ed0 0x7f2174073bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21741a00f0 con 0x7f2174101540 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21741a0230 con 0x7f2174101ed0 2023-07-29T16:29:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217afff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21741a0370 con 0x7f2174100350 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41682/0 (socket says 172.21.15.31:41682) 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f2178d9b700 1 -- 172.21.15.31:0/4075604602 learned_addr learned my addr 172.21.15.31:0/4075604602 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f2173fff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2174101540 0x7f2174073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.756+0000 7f217959c700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101ed0 0x7f2174073bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2178d9b700 1 -- 172.21.15.31:0/4075604602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101ed0 msgr2=0x7f2174073bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2178d9b700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101ed0 0x7f2174073bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2178d9b700 1 -- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2174101540 msgr2=0x7f2174073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2178d9b700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2174101540 0x7f2174073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2178d9b700 1 -- 172.21.15.31:0/4075604602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21600095f0 con 0x7f2174100350 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2178d9b700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174073140 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f216800e970 tx=0x7f216800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2171ffb700 1 -- 172.21.15.31:0/4075604602 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168009790 con 0x7f2174100350 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2171ffb700 1 -- 172.21.15.31:0/4075604602 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f216800c7e0 con 0x7f2174100350 2023-07-29T16:29:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21741a0600 con 0x7f2174100350 2023-07-29T16:29:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f2171ffb700 1 -- 172.21.15.31:0/4075604602 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168003c70 con 0x7f2174100350 2023-07-29T16:29:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.757+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741a0b10 con 0x7f2174100350 2023-07-29T16:29:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.761+0000 7f2171ffb700 1 -- 172.21.15.31:0/4075604602 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f216800cc50 con 0x7f2174100350 2023-07-29T16:29:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.762+0000 7f2171ffb700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f21640a0790 0x7f21640a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.762+0000 7f2171ffb700 1 -- 172.21.15.31:0/4075604602 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2168014070 con 0x7f2174100350 2023-07-29T16:29:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.762+0000 7f218e9b4b80 1 Processor -- start 2023-07-29T16:29:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f2173fff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f21640a0790 0x7f21640a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 -- start start 2023-07-29T16:29:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed727b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed73160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ecc99b0 con 0x55833ebcd6b0 2023-07-29T16:29:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ecc7370 con 0x55833ed71df0 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ecc2c60 con 0x55833ed72cf0 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.763+0000 7f2173fff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f21640a0790 0x7f21640a2c40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2174074440 tx=0x7f2160005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2173fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f217959c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed73160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f217959c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed73160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41684/0 (socket says 172.21.15.31:41684) 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f217959c700 1 -- 172.21.15.31:0/2200434413 learned_addr learned my addr 172.21.15.31:0/2200434413 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2173fff700 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33072/0 (socket says 172.21.15.31:33072) 2023-07-29T16:29:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 -- 172.21.15.31:0/2200434413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 msgr2=0x55833ed73160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed73160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 -- 172.21.15.31:0/2200434413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 msgr2=0x55833ed727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 -- 172.21.15.31:0/2200434413 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55833eaf0d80 con 0x55833ebcd6b0 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed718b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2168099170 tx=0x7f2168003710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3f8a399a1d1320 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.764+0000 7f215effd700 1 -- 172.21.15.31:0/2200434413 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216809d4a0 con 0x55833ebcd6b0 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f215effd700 1 -- 172.21.15.31:0/2200434413 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21680a18c0 con 0x55833ebcd6b0 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f215effd700 1 -- 172.21.15.31:0/2200434413 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680998f0 con 0x55833ebcd6b0 2023-07-29T16:29:47.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2200434413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 msgr2=0x55833ed718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed718b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2168099170 tx=0x7f2168003710 comp rx=0 tx=0).stop 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2200434413 shutdown_connections 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed73160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed718b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2200434413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2200434413 >> 172.21.15.31:0/2200434413 conn(0x55833ed66fd0 msgr2=0x55833ed6c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:47.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.765+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2200434413 shutdown_connections 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.766+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2200434413 wait complete. 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.766+0000 7f218e9b4b80 1 -- start start 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.766+0000 7f218e9b4b80 1 Processor -- start 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed7af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 0x55833ed82480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ec5f4b0 con 0x55833ebcd6b0 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ec5f150 con 0x55833ed72cf0 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833eb468c0 con 0x55833ed71df0 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f218e9b4b80 10 client.0 _ll_register_callbacks cb 0x55833ec685c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f217959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 0x55833ed82480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed7af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f217959c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 0x55833ed82480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33088/0 (socket says 172.21.15.31:33088) 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f217959c700 1 -- 172.21.15.31:0/2871374290 learned_addr learned my addr 172.21.15.31:0/2871374290 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:47.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f2173fff700 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.767+0000 7f217959c700 1 -- 172.21.15.31:0/2871374290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 msgr2=0x55833ed7b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f217959c700 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f217959c700 1 -- 172.21.15.31:0/2871374290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 msgr2=0x55833ed7af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f217959c700 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed7af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f217959c700 1 -- 172.21.15.31:0/2871374290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21741a0b10 con 0x55833ed72cf0 2023-07-29T16:29:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f217959c700 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 0x55833ed82480 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f216c004660 tx=0x7f216c004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=693eb0238042c8f6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f2153fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f2153fff700 1 -- 172.21.15.31:0/2871374290 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c00e040 con 0x55833ed72cf0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55833ecedcb0 con 0x55833ed72cf0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f2153fff700 1 -- 172.21.15.31:0/2871374290 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f216c01a070 con 0x55833ed72cf0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f2153fff700 1 -- 172.21.15.31:0/2871374290 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f216c0092e0 con 0x55833ed72cf0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f218e9b4b80 10 client.29750 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.768+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55833ead9c00 con 0x55833ed72cf0 2023-07-29T16:29:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f2153fff700 1 -- 172.21.15.31:0/2871374290 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55833ecedcb0 con 0x55833ed72cf0 2023-07-29T16:29:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f2153fff700 1 -- 172.21.15.31:0/2871374290 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f216c01ebd0 con 0x55833ed72cf0 2023-07-29T16:29:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f218e9b4b80 10 client.29750 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f218e9b4b80 10 client.29750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f218e9b4b80 20 client.29750 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833ed86d00 0x55833ed89110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f218e9b4b80 4 client.29750 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:29:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55833ed40d10 con 0x55833ed86d00 2023-07-29T16:29:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.769+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833ed86d00 0x55833ed89110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.770+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833ed86d00 0x55833ed89110 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af6b9916c473c6c7 server_cookie=17314e8cef733533 in_seq=0 out_seq=0 2023-07-29T16:29:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.770+0000 7f2153fff700 10 client.29750 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f2153fff700 1 -- 172.21.15.31:0/2871374290 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55833ed40d10 con 0x55833ed86d00 2023-07-29T16:29:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f2153fff700 10 client.29750 handle_command_reply: tid=0 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 client.29750 shutdown 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 msgr2=0x55833ed82480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 0x55833ed82480 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f216c004660 tx=0x7f216c004970 comp rx=0 tx=0).stop 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 shutdown_connections 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833ed86d00 0x55833ed89110 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed72cf0 0x55833ed82480 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed7af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2871374290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 >> 172.21.15.31:0/2871374290 conn(0x55833ed66fd0 msgr2=0x55833ed6ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f2153fff700 0 client.29750 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.771+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 shutdown_connections 2023-07-29T16:29:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.772+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2871374290 wait complete. 2023-07-29T16:29:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.772+0000 7f218e9b4b80 20 client.29750 trim_cache size 0 max 16384 2023-07-29T16:29:47.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.787+0000 7f218e9b4b80 1 Processor -- start 2023-07-29T16:29:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.787+0000 7f218e9b4b80 1 -- start start 2023-07-29T16:29:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.787+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed790f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.787+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed79630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed82440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ed6d3f0 con 0x55833ebcd6b0 2023-07-29T16:29:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ed78ea0 con 0x55833ed71df0 2023-07-29T16:29:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ed6b620 con 0x55833ed72cf0 2023-07-29T16:29:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed790f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f217959c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed82440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed790f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38398/0 (socket says 172.21.15.31:38398) 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 -- 172.21.15.31:0/2610674587 learned_addr learned my addr 172.21.15.31:0/2610674587 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2173fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed79630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 -- 172.21.15.31:0/2610674587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 msgr2=0x55833ed82440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed82440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 -- 172.21.15.31:0/2610674587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 msgr2=0x55833ed79630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed79630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 -- 172.21.15.31:0/2610674587 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55833eb171f0 con 0x55833ebcd6b0 2023-07-29T16:29:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.788+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed790f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2168095e90 tx=0x7f21680b3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7b8973bd0d7e05 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.789+0000 7f215f7fe700 1 -- 172.21.15.31:0/2610674587 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168097810 con 0x55833ebcd6b0 2023-07-29T16:29:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.789+0000 7f215f7fe700 1 -- 172.21.15.31:0/2610674587 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2168095b80 con 0x55833ebcd6b0 2023-07-29T16:29:47.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.789+0000 7f215f7fe700 1 -- 172.21.15.31:0/2610674587 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21680996d0 con 0x55833ebcd6b0 2023-07-29T16:29:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.789+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2610674587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 msgr2=0x55833ed790f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.789+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed790f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2168095e90 tx=0x7f21680b3040 comp rx=0 tx=0).stop 2023-07-29T16:29:47.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2610674587 shutdown_connections 2023-07-29T16:29:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed72cf0 0x55833ed82440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ebcd6b0 0x55833ed790f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2610674587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ed71df0 0x55833ed79630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2610674587 >> 172.21.15.31:0/2610674587 conn(0x55833ed66fd0 msgr2=0x55833ed88d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2610674587 shutdown_connections 2023-07-29T16:29:47.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2610674587 wait complete. 2023-07-29T16:29:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 -- start start 2023-07-29T16:29:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.790+0000 7f218e9b4b80 1 Processor -- start 2023-07-29T16:29:47.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 0x55833ed7e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ed72cf0 0x55833ed81b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ed820b0 con 0x55833ed72cf0 2023-07-29T16:29:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ed82220 con 0x55833ebcd6b0 2023-07-29T16:29:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ed823e0 con 0x55833ed71df0 2023-07-29T16:29:47.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f218e9b4b80 10 client.0 _ll_register_callbacks cb 0x55833ec685c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 0x55833ed7e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 0x55833ed7e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33118/0 (socket says 172.21.15.31:33118) 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 -- 172.21.15.31:0/2935256136 learned_addr learned my addr 172.21.15.31:0/2935256136 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2173fff700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f217959c700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ed72cf0 0x55833ed81b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 -- 172.21.15.31:0/2935256136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 msgr2=0x55833ed7ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 -- 172.21.15.31:0/2935256136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ed72cf0 msgr2=0x55833ed81b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:29:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ed72cf0 0x55833ed81b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.791+0000 7f2178d9b700 1 -- 172.21.15.31:0/2935256136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55833eaf0d80 con 0x55833ebcd6b0 2023-07-29T16:29:47.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2178d9b700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 0x55833ed7e670 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2168099170 tx=0x7f21680958e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0cde7cf7926f307 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:29:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2153fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:29:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2153fff700 1 -- 172.21.15.31:0/2935256136 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2168095cf0 con 0x55833ebcd6b0 2023-07-29T16:29:47.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2153fff700 1 -- 172.21.15.31:0/2935256136 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2168098ad0 con 0x55833ebcd6b0 2023-07-29T16:29:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21680a18c0 con 0x55833ebcd6b0 2023-07-29T16:29:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2153fff700 1 -- 172.21.15.31:0/2935256136 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f216809d150 con 0x55833ebcd6b0 2023-07-29T16:29:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 10 client.29756 fetch_fsmap learned FSMap version 34 2023-07-29T16:29:47.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55833eaef250 con 0x55833ebcd6b0 2023-07-29T16:29:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2153fff700 1 -- 172.21.15.31:0/2935256136 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21680a18c0 con 0x55833ebcd6b0 2023-07-29T16:29:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f2153fff700 1 -- 172.21.15.31:0/2935256136 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f21680b0030 con 0x55833ebcd6b0 2023-07-29T16:29:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 10 client.29756 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:29:47.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 10 client.29756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:29:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 20 client.29756 populate_metadata read hostname 'smithi031' 2023-07-29T16:29:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.792+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833edd9660 0x55833eddba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:29:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.793+0000 7f218e9b4b80 4 client.29756 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:29:47.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.793+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55833ecbbce0 con 0x55833edd9660 2023-07-29T16:29:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.793+0000 7f2173fff700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833edd9660 0x55833eddba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:29:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.793+0000 7f2173fff700 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833edd9660 0x55833eddba70 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e7fe7a94485146f server_cookie=123f6aaab9eaaacd in_seq=0 out_seq=0 2023-07-29T16:29:47.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.793+0000 7f2153fff700 10 client.29756 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.882+0000 7f2153fff700 1 -- 172.21.15.31:0/2935256136 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55833ecbbce0 con 0x55833edd9660 2023-07-29T16:29:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.882+0000 7f2153fff700 10 client.29756 handle_command_reply: tid=0 2023-07-29T16:29:47.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.882+0000 7f218e9b4b80 1 client.29756 shutdown 2023-07-29T16:29:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 msgr2=0x55833ed7e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 0x55833ed7e670 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f2168099170 tx=0x7f21680958e0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 shutdown_connections 2023-07-29T16:29:47.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55833edd9660 0x55833eddba70 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55833ed72cf0 0x55833ed81b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55833ebcd6b0 0x55833ed7e670 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 --2- 172.21.15.31:0/2935256136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55833ed71df0 0x55833ed7ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f2153fff700 0 client.29756 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:29:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 >> 172.21.15.31:0/2935256136 conn(0x55833ed66fd0 msgr2=0x55833ed790f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 shutdown_connections 2023-07-29T16:29:47.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 1 -- 172.21.15.31:0/2935256136 wait complete. 2023-07-29T16:29:47.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.883+0000 7f218e9b4b80 20 client.29756 trim_cache size 0 max 16384 2023-07-29T16:29:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:29:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:29:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:29:47.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:29:47.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f21640a0790 msgr2=0x7f21640a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f21640a0790 0x7f21640a2c40 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2174074440 tx=0x7f2160005960 comp rx=0 tx=0).stop 2023-07-29T16:29:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 msgr2=0x7f2174073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:29:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174073140 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f216800e970 tx=0x7f216800ec80 comp rx=0 tx=0).stop 2023-07-29T16:29:47.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 shutdown_connections 2023-07-29T16:29:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f21640a0790 0x7f21640a2c40 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174101ed0 0x7f2174073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2174100350 0x7f2174073140 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 --2- 172.21.15.31:0/4075604602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2174101540 0x7f2174073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:29:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 >> 172.21.15.31:0/4075604602 conn(0x7f21740fb970 msgr2=0x7f2174104a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:29:47.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 shutdown_connections 2023-07-29T16:29:47.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:29:47.884+0000 7f217afff700 1 -- 172.21.15.31:0/4075604602 wait complete. 2023-07-29T16:29:47.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1134274 8.83 MB/sec execute 572 sec latency 52.298 ms 2023-07-29T16:29:48.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:48 smithi039 ceph-mon[128858]: pgmap v801: 97 pgs: 97 active+clean; 208 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2023-07-29T16:29:48.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:48 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:29:48.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:48 smithi156 ceph-mon[129245]: pgmap v801: 97 pgs: 97 active+clean; 208 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2023-07-29T16:29:48.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:48 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:29:48.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:48 smithi031 ceph-mon[95504]: pgmap v801: 97 pgs: 97 active+clean; 208 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 1.9 MiB/s wr, 223 op/s 2023-07-29T16:29:48.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:48 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:29:48.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1136373 8.83 MB/sec execute 573 sec latency 18.245 ms 2023-07-29T16:29:49.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:49 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:29:49.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:49 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:29:49.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:49 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:29:49.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:49 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:29:49.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:49 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:29:49.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:49 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:29:49.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1137349 8.83 MB/sec execute 574 sec latency 13.989 ms 2023-07-29T16:29:50.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:50 smithi039 ceph-mon[128858]: pgmap v802: 97 pgs: 97 active+clean; 214 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.5 MiB/s wr, 286 op/s 2023-07-29T16:29:50.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:50 smithi156 ceph-mon[129245]: pgmap v802: 97 pgs: 97 active+clean; 214 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.5 MiB/s wr, 286 op/s 2023-07-29T16:29:50.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:50 smithi031 ceph-mon[95504]: pgmap v802: 97 pgs: 97 active+clean; 214 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 2.5 MiB/s wr, 286 op/s 2023-07-29T16:29:50.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1139235 8.82 MB/sec execute 575 sec latency 34.027 ms 2023-07-29T16:29:51.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1140613 8.83 MB/sec execute 576 sec latency 16.798 ms 2023-07-29T16:29:52.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1142495 8.82 MB/sec execute 577 sec latency 36.001 ms 2023-07-29T16:29:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:52 smithi031 ceph-mon[95504]: pgmap v803: 97 pgs: 97 active+clean; 214 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.8 MiB/s wr, 212 op/s 2023-07-29T16:29:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:52 smithi039 ceph-mon[128858]: pgmap v803: 97 pgs: 97 active+clean; 214 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.8 MiB/s wr, 212 op/s 2023-07-29T16:29:53.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:52 smithi156 ceph-mon[129245]: pgmap v803: 97 pgs: 97 active+clean; 214 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.8 MiB/s wr, 212 op/s 2023-07-29T16:29:53.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1144047 8.83 MB/sec execute 578 sec latency 25.009 ms 2023-07-29T16:29:54.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1145616 8.83 MB/sec execute 579 sec latency 13.854 ms 2023-07-29T16:29:54.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:54 smithi031 ceph-mon[95504]: pgmap v804: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.4 MiB/s wr, 295 op/s 2023-07-29T16:29:55.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:54 smithi039 ceph-mon[128858]: pgmap v804: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.4 MiB/s wr, 295 op/s 2023-07-29T16:29:55.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:54 smithi156 ceph-mon[129245]: pgmap v804: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.4 MiB/s wr, 295 op/s 2023-07-29T16:29:55.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1147596 8.83 MB/sec execute 580 sec latency 10.909 ms 2023-07-29T16:29:56.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1148987 8.83 MB/sec execute 581 sec latency 45.258 ms 2023-07-29T16:29:56.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:56 smithi031 ceph-mon[95504]: pgmap v805: 97 pgs: 97 active+clean; 212 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 2.0 MiB/s wr, 219 op/s 2023-07-29T16:29:57.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:56 smithi039 ceph-mon[128858]: pgmap v805: 97 pgs: 97 active+clean; 212 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 2.0 MiB/s wr, 219 op/s 2023-07-29T16:29:57.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:56 smithi156 ceph-mon[129245]: pgmap v805: 97 pgs: 97 active+clean; 212 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 2.0 MiB/s wr, 219 op/s 2023-07-29T16:29:57.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1150952 8.83 MB/sec execute 582 sec latency 47.640 ms 2023-07-29T16:29:58.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:29:58.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:58 smithi039 ceph-mon[128858]: pgmap v806: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.9 MiB/s wr, 253 op/s 2023-07-29T16:29:58.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:29:58.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:29:58.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:29:58 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:29:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:29:58.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:58 smithi156 ceph-mon[129245]: pgmap v806: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.9 MiB/s wr, 253 op/s 2023-07-29T16:29:58.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:29:58.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:29:58.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:29:58 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:29:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:29:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:58 smithi031 ceph-mon[95504]: pgmap v806: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 1.9 MiB/s wr, 253 op/s 2023-07-29T16:29:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:29:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:29:58.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:29:58 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:29:58.905 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1152015 8.82 MB/sec execute 583 sec latency 22.984 ms 2023-07-29T16:29:59.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1154388 8.84 MB/sec execute 584 sec latency 12.434 ms 2023-07-29T16:30:00.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1155572 8.83 MB/sec execute 585 sec latency 11.621 ms 2023-07-29T16:30:00.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:00 smithi031 ceph-mon[95504]: pgmap v807: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 2.5 MiB/s wr, 301 op/s 2023-07-29T16:30:00.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:00 smithi031 ceph-mon[95504]: overall HEALTH_OK 2023-07-29T16:30:01.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:00 smithi039 ceph-mon[128858]: pgmap v807: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 2.5 MiB/s wr, 301 op/s 2023-07-29T16:30:01.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:00 smithi039 ceph-mon[128858]: overall HEALTH_OK 2023-07-29T16:30:01.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 -- 172.21.15.31:0/182989091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644074d10 msgr2=0x7f8644073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 --2- 172.21.15.31:0/182989091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644074d10 0x7f8644073160 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f8640009990 tx=0x7f8640009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 -- 172.21.15.31:0/182989091 shutdown_connections 2023-07-29T16:30:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 --2- 172.21.15.31:0/182989091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644108230 0x7f864410a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 --2- 172.21.15.31:0/182989091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644074d10 0x7f8644073160 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 --2- 172.21.15.31:0/182989091 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 0x7f8644073ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 -- 172.21.15.31:0/182989091 >> 172.21.15.31:0/182989091 conn(0x7f86440fb9a0 msgr2=0x7f86440fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 -- 172.21.15.31:0/182989091 shutdown_connections 2023-07-29T16:30:01.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.041+0000 7f864bfff700 1 -- 172.21.15.31:0/182989091 wait complete. 2023-07-29T16:30:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 Processor -- start 2023-07-29T16:30:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 -- start start 2023-07-29T16:30:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 0x7f8644198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 0x7f8644198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644108230 0x7f86441996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644199d30 con 0x7f8644073730 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644199ea0 con 0x7f8644074d10 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864bfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864419a060 con 0x7f8644108230 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 0x7f8644198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f8649d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 0x7f8644198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f8649d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 0x7f8644198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59172/0 (socket says 172.21.15.31:59172) 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f8649d9b700 1 -- 172.21.15.31:0/1137305898 learned_addr learned my addr 172.21.15.31:0/1137305898 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:01.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.043+0000 7f864959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 0x7f8644198850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34414/0 (socket says 172.21.15.31:34414) 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864a59c700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644108230 0x7f86441996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864959a700 1 -- 172.21.15.31:0/1137305898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644108230 msgr2=0x7f86441996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864959a700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644108230 0x7f86441996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864959a700 1 -- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 msgr2=0x7f8644198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864959a700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 0x7f8644198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864959a700 1 -- 172.21.15.31:0/1137305898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86400095f0 con 0x7f8644074d10 2023-07-29T16:30:01.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864959a700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 0x7f8644198850 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f863400e9f0 tx=0x7f863400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f863affd700 1 -- 172.21.15.31:0/1137305898 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863400c9c0 con 0x7f8644074d10 2023-07-29T16:30:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f863affd700 1 -- 172.21.15.31:0/1137305898 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8634010430 con 0x7f8644074d10 2023-07-29T16:30:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f863affd700 1 -- 172.21.15.31:0/1137305898 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634018680 con 0x7f8644074d10 2023-07-29T16:30:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.044+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864419a340 con 0x7f8644074d10 2023-07-29T16:30:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.045+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86441a0b00 con 0x7f8644074d10 2023-07-29T16:30:01.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.048+0000 7f863affd700 1 -- 172.21.15.31:0/1137305898 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8634010860 con 0x7f8644074d10 2023-07-29T16:30:01.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.049+0000 7f865f962b80 1 Processor -- start 2023-07-29T16:30:01.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.049+0000 7f863affd700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86300a0790 0x7f86300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f863affd700 1 -- 172.21.15.31:0/1137305898 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8634014070 con 0x7f8644074d10 2023-07-29T16:30:01.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 -- start start 2023-07-29T16:30:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f864a59c700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86300a0790 0x7f86300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 0x55b06dbfa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06dbfadf0 0x55b06dbfb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbfc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06db529b0 con 0x55b06dbfadf0 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06db50370 con 0x55b06dbfbcf0 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f865f962b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06db4bc60 con 0x55b06da566b0 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f8649d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 0x55b06dbfa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f8649d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 0x55b06dbfa8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36946/0 (socket says 172.21.15.31:36946) 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f8649d9b700 1 -- 172.21.15.31:0/642271830 learned_addr learned my addr 172.21.15.31:0/642271830 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:01.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.050+0000 7f864a59c700 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbfc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f8649d9b700 1 -- 172.21.15.31:0/642271830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 msgr2=0x55b06dbfc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f8649d9b700 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbfc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f8649d9b700 1 -- 172.21.15.31:0/642271830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06dbfadf0 msgr2=0x55b06dbfb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f8649d9b700 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06dbfadf0 0x55b06dbfb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f8649d9b700 1 -- 172.21.15.31:0/642271830 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b06d979d80 con 0x55b06da566b0 2023-07-29T16:30:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f864a59c700 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbfc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f864a59c700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86300a0790 0x7f86300a2c40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f863c00f590 tx=0x7f863c009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:01.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f8649d9b700 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 0x55b06dbfa8b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8640000c00 tx=0x7f8640005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f149729954e37682 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f862ffff700 1 -- 172.21.15.31:0/642271830 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864001d070 con 0x55b06da566b0 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f862ffff700 1 -- 172.21.15.31:0/642271830 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f864001e040 con 0x55b06da566b0 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.051+0000 7f862ffff700 1 -- 172.21.15.31:0/642271830 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640004ad0 con 0x55b06da566b0 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 -- 172.21.15.31:0/642271830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 msgr2=0x55b06dbfa8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 0x55b06dbfa8b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f8640000c00 tx=0x7f8640005bc0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 -- 172.21.15.31:0/642271830 shutdown_connections 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbfc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06da566b0 0x55b06dbfa8b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 --2- 172.21.15.31:0/642271830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06dbfadf0 0x55b06dbfb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 -- 172.21.15.31:0/642271830 >> 172.21.15.31:0/642271830 conn(0x55b06dbeffd0 msgr2=0x55b06dbf5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.052+0000 7f865f962b80 1 -- 172.21.15.31:0/642271830 shutdown_connections 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.053+0000 7f865f962b80 1 -- 172.21.15.31:0/642271830 wait complete. 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.053+0000 7f865f962b80 1 -- start start 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.053+0000 7f865f962b80 1 Processor -- start 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc03f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 0x55b06dc04440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 0x55b06dc0b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dae84b0 con 0x55b06da566b0 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dae8150 con 0x55b06dbfadf0 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06d9cf8c0 con 0x55b06dbfbcf0 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f865f962b80 10 client.0 _ll_register_callbacks cb 0x55b06daf15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f864959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 0x55b06dc04440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f864a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 0x55b06dc0b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f864959a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 0x55b06dc04440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34430/0 (socket says 172.21.15.31:34430) 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f864959a700 1 -- 172.21.15.31:0/3599867171 learned_addr learned my addr 172.21.15.31:0/3599867171 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.054+0000 7f864a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 0x55b06dc0b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36962/0 (socket says 172.21.15.31:36962) 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f8649d9b700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc03f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864a59c700 1 -- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 msgr2=0x55b06dc04440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864a59c700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 0x55b06dc04440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864a59c700 1 -- 172.21.15.31:0/3599867171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 msgr2=0x55b06dc03f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864a59c700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc03f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864a59c700 1 -- 172.21.15.31:0/3599867171 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864410b2b0 con 0x55b06dbfbcf0 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864959a700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 0x55b06dc04440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f864a59c700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 0x55b06dc0b480 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f863c009fd0 tx=0x7f863c004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0627cf9626e8ebd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f862d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.055+0000 7f862d7fa700 1 -- 172.21.15.31:0/3599867171 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00e040 con 0x55b06dbfbcf0 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f862d7fa700 1 -- 172.21.15.31:0/3599867171 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f863c00fe40 con 0x55b06dbfbcf0 2023-07-29T16:30:01.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b06db76cb0 con 0x55b06dbfbcf0 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f862d7fa700 1 -- 172.21.15.31:0/3599867171 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f863c017cb0 con 0x55b06dbfbcf0 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 10 client.29767 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55b06d962c00 con 0x55b06dbfbcf0 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f862d7fa700 1 -- 172.21.15.31:0/3599867171 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b06db76cb0 con 0x55b06dbfbcf0 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f862d7fa700 1 -- 172.21.15.31:0/3599867171 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f863c0045f0 con 0x55b06dbfbcf0 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 10 client.29767 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:01.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 10 client.29767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 20 client.29767 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc0fd40 0x55b06dc12150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 4 client.29767 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.056+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b06dbc9d10 con 0x55b06dc0fd40 2023-07-29T16:30:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.057+0000 7f8649d9b700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc0fd40 0x55b06dc12150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.057+0000 7f8649d9b700 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc0fd40 0x55b06dc12150 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ef6a00b2744049 server_cookie=f1cee4a744d39917 in_seq=0 out_seq=0 2023-07-29T16:30:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.057+0000 7f862d7fa700 10 client.29767 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f862d7fa700 1 -- 172.21.15.31:0/3599867171 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b06dbc9d10 con 0x55b06dc0fd40 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f862d7fa700 10 client.29767 handle_command_reply: tid=0 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 client.29767 shutdown 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 msgr2=0x55b06dc0b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 0x55b06dc0b480 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f863c009fd0 tx=0x7f863c004b40 comp rx=0 tx=0).stop 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 shutdown_connections 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc0fd40 0x55b06dc12150 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfbcf0 0x55b06dc0b480 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc03f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 --2- 172.21.15.31:0/3599867171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfadf0 0x55b06dc04440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 >> 172.21.15.31:0/3599867171 conn(0x55b06dbeffd0 msgr2=0x55b06dbf4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.058+0000 7f862d7fa700 0 client.29767 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.059+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 shutdown_connections 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.059+0000 7f865f962b80 1 -- 172.21.15.31:0/3599867171 wait complete. 2023-07-29T16:30:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.059+0000 7f865f962b80 20 client.29767 trim_cache size 0 max 16384 2023-07-29T16:30:01.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:00 smithi156 ceph-mon[129245]: pgmap v807: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 2.5 MiB/s wr, 301 op/s 2023-07-29T16:30:01.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:00 smithi156 ceph-mon[129245]: overall HEALTH_OK 2023-07-29T16:30:01.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.076+0000 7f865f962b80 1 Processor -- start 2023-07-29T16:30:01.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.076+0000 7f865f962b80 1 -- start start 2023-07-29T16:30:01.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.076+0000 7f865f962b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dbf63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f865f962b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc09430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f865f962b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbf14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f865f962b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dc03a70 con 0x55b06da566b0 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f865f962b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dbf5f90 con 0x55b06dbfbcf0 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f865f962b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dc04b80 con 0x55b06dbfadf0 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc09430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbf14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc09430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36970/0 (socket says 172.21.15.31:36970) 2023-07-29T16:30:01.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 -- 172.21.15.31:0/3345640360 learned_addr learned my addr 172.21.15.31:0/3345640360 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f8649d9b700 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dbf63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 -- 172.21.15.31:0/3345640360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 msgr2=0x55b06dbf14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbf14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 -- 172.21.15.31:0/3345640360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 msgr2=0x55b06dbf63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dbf63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 -- 172.21.15.31:0/3345640360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b06d9a01f0 con 0x55b06dbfadf0 2023-07-29T16:30:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.077+0000 7f864959a700 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc09430 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8634095e00 tx=0x7f8634009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fcdc5eb7dc31ad9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f8638ff9700 1 -- 172.21.15.31:0/3345640360 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634095720 con 0x55b06dbfadf0 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f8638ff9700 1 -- 172.21.15.31:0/3345640360 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86340993e0 con 0x55b06dbfadf0 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f8638ff9700 1 -- 172.21.15.31:0/3345640360 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86340a13a0 con 0x55b06dbfadf0 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 -- 172.21.15.31:0/3345640360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 msgr2=0x55b06dc09430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc09430 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8634095e00 tx=0x7f8634009ef0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 -- 172.21.15.31:0/3345640360 shutdown_connections 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dbf14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dbf63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 --2- 172.21.15.31:0/3345640360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc09430 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.078+0000 7f865f962b80 1 -- 172.21.15.31:0/3345640360 >> 172.21.15.31:0/3345640360 conn(0x55b06dbeffd0 msgr2=0x55b06dbf52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.079+0000 7f865f962b80 1 -- 172.21.15.31:0/3345640360 shutdown_connections 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.079+0000 7f865f962b80 1 -- 172.21.15.31:0/3345640360 wait complete. 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.079+0000 7f865f962b80 1 -- start start 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.079+0000 7f865f962b80 1 Processor -- start 2023-07-29T16:30:01.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc640a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc665f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dc68b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dc09770 con 0x55b06da566b0 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dc69080 con 0x55b06dbfbcf0 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b06dc691c0 con 0x55b06dbfadf0 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f865f962b80 10 client.0 _ll_register_callbacks cb 0x55b06daf15c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f8649d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc640a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc665f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc665f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36972/0 (socket says 172.21.15.31:36972) 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f8649d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc640a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59210/0 (socket says 172.21.15.31:59210) 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 -- 172.21.15.31:0/1932071694 learned_addr learned my addr 172.21.15.31:0/1932071694 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864a59c700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dc68b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 -- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 msgr2=0x55b06dc68b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dc68b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 -- 172.21.15.31:0/1932071694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 msgr2=0x55b06dc640a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc640a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f864959a700 1 -- 172.21.15.31:0/1932071694 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86441a0b00 con 0x55b06dbfadf0 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.080+0000 7f8649d9b700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f864959a700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc665f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8634098eb0 tx=0x7f86340999e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dcc4fa889d49d33 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f862d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f862d7fa700 1 -- 172.21.15.31:0/1932071694 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86340a1830 con 0x55b06dbfadf0 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f862d7fa700 1 -- 172.21.15.31:0/1932071694 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f863409d410 con 0x55b06dbfadf0 2023-07-29T16:30:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f864001e040 con 0x55b06dbfadf0 2023-07-29T16:30:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f862d7fa700 1 -- 172.21.15.31:0/1932071694 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8634065240 con 0x55b06dbfadf0 2023-07-29T16:30:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f865f962b80 10 client.29773 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55b06d978250 con 0x55b06dbfadf0 2023-07-29T16:30:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.081+0000 7f862d7fa700 1 -- 172.21.15.31:0/1932071694 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f864001e040 con 0x55b06dbfadf0 2023-07-29T16:30:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f862d7fa700 1 -- 172.21.15.31:0/1932071694 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8634098340 con 0x55b06dbfadf0 2023-07-29T16:30:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f865f962b80 10 client.29773 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f865f962b80 10 client.29773 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f865f962b80 20 client.29773 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:01.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f865f962b80 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc6a820 0x55b06dc6cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f865f962b80 4 client.29773 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:30:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b06db44ce0 con 0x55b06dc6a820 2023-07-29T16:30:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f8649d9b700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc6a820 0x55b06dc6cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f8649d9b700 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc6a820 0x55b06dc6cc30 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55273a22b0181510 server_cookie=88653a76ba2fe134 in_seq=0 out_seq=0 2023-07-29T16:30:01.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.082+0000 7f862d7fa700 10 client.29773 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.163+0000 7f862d7fa700 1 -- 172.21.15.31:0/1932071694 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b06db44ce0 con 0x55b06dc6a820 2023-07-29T16:30:01.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.163+0000 7f862d7fa700 10 client.29773 handle_command_reply: tid=0 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.163+0000 7f865f962b80 1 client.29773 shutdown 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 msgr2=0x55b06dc665f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc665f0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8634098eb0 tx=0x7f86340999e0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 shutdown_connections 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b06dc6a820 0x55b06dc6cc30 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b06dbfbcf0 0x55b06dc68b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b06da566b0 0x55b06dc640a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f862d7fa700 0 client.29773 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 --2- 172.21.15.31:0/1932071694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b06dbfadf0 0x55b06dc665f0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 >> 172.21.15.31:0/1932071694 conn(0x55b06dbeffd0 msgr2=0x55b06dc02bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 shutdown_connections 2023-07-29T16:30:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 1 -- 172.21.15.31:0/1932071694 wait complete. 2023-07-29T16:30:01.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.164+0000 7f865f962b80 20 client.29773 trim_cache size 0 max 16384 2023-07-29T16:30:01.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:30:01.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:30:01.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:30:01.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:30:01.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86300a0790 msgr2=0x7f86300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86300a0790 0x7f86300a2c40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f863c00f590 tx=0x7f863c009920 comp rx=0 tx=0).stop 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 msgr2=0x7f8644198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 0x7f8644198850 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f863400e9f0 tx=0x7f863400ed00 comp rx=0 tx=0).stop 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 shutdown_connections 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f86300a0790 0x7f86300a2c40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8644108230 0x7f86441996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8644073730 0x7f8644198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 --2- 172.21.15.31:0/1137305898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8644074d10 0x7f8644198850 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 >> 172.21.15.31:0/1137305898 conn(0x7f86440fb9a0 msgr2=0x7f86440fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 shutdown_connections 2023-07-29T16:30:01.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:01.165+0000 7f864bfff700 1 -- 172.21.15.31:0/1137305898 wait complete. 2023-07-29T16:30:01.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1157732 8.83 MB/sec execute 586 sec latency 63.884 ms 2023-07-29T16:30:02.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1158718 8.83 MB/sec execute 587 sec latency 34.555 ms 2023-07-29T16:30:02.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:02 smithi031 ceph-mon[95504]: pgmap v808: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2023-07-29T16:30:03.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:02 smithi039 ceph-mon[128858]: pgmap v808: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2023-07-29T16:30:03.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:02 smithi156 ceph-mon[129245]: pgmap v808: 97 pgs: 97 active+clean; 213 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 1.8 MiB/s wr, 195 op/s 2023-07-29T16:30:03.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1160826 8.83 MB/sec execute 588 sec latency 38.216 ms 2023-07-29T16:30:04.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1162039 8.83 MB/sec execute 589 sec latency 21.835 ms 2023-07-29T16:30:04.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:04 smithi031 ceph-mon[95504]: pgmap v809: 97 pgs: 97 active+clean; 212 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.4 MiB/s wr, 296 op/s 2023-07-29T16:30:05.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:04 smithi039 ceph-mon[128858]: pgmap v809: 97 pgs: 97 active+clean; 212 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.4 MiB/s wr, 296 op/s 2023-07-29T16:30:05.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:04 smithi156 ceph-mon[129245]: pgmap v809: 97 pgs: 97 active+clean; 212 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 2.4 MiB/s wr, 296 op/s 2023-07-29T16:30:05.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1164076 8.83 MB/sec execute 590 sec latency 14.004 ms 2023-07-29T16:30:06.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1165443 8.83 MB/sec execute 591 sec latency 18.426 ms 2023-07-29T16:30:06.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:06 smithi031 ceph-mon[95504]: pgmap v810: 97 pgs: 97 active+clean; 214 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.0 MiB/s wr, 242 op/s 2023-07-29T16:30:07.025 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:06 smithi039 ceph-mon[128858]: pgmap v810: 97 pgs: 97 active+clean; 214 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.0 MiB/s wr, 242 op/s 2023-07-29T16:30:07.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:06 smithi156 ceph-mon[129245]: pgmap v810: 97 pgs: 97 active+clean; 214 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.0 MiB/s wr, 242 op/s 2023-07-29T16:30:07.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1166973 8.83 MB/sec execute 592 sec latency 43.153 ms 2023-07-29T16:30:08.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:30:08.275 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:08 smithi039 ceph-mon[128858]: pgmap v811: 97 pgs: 97 active+clean; 215 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 2.2 MiB/s wr, 258 op/s 2023-07-29T16:30:08.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:08 smithi031 ceph-mon[95504]: pgmap v811: 97 pgs: 97 active+clean; 215 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 2.2 MiB/s wr, 258 op/s 2023-07-29T16:30:08.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:08 smithi156 ceph-mon[129245]: pgmap v811: 97 pgs: 97 active+clean; 215 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 2.2 MiB/s wr, 258 op/s 2023-07-29T16:30:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 -- 172.21.15.31:0/1360011822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c1081c0 msgr2=0x7ff82c10a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 --2- 172.21.15.31:0/1360011822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c1081c0 0x7ff82c10a640 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff82400b0a0 tx=0x7ff82400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 -- 172.21.15.31:0/1360011822 shutdown_connections 2023-07-29T16:30:08.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 --2- 172.21.15.31:0/1360011822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c1081c0 0x7ff82c10a640 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 --2- 172.21.15.31:0/1360011822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff82c074d30 0x7ff82c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 --2- 172.21.15.31:0/1360011822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c073750 0x7ff82c073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.452+0000 7ff833bff700 1 -- 172.21.15.31:0/1360011822 >> 172.21.15.31:0/1360011822 conn(0x7ff82c0fb970 msgr2=0x7ff82c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.453+0000 7ff833bff700 1 -- 172.21.15.31:0/1360011822 shutdown_connections 2023-07-29T16:30:08.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.453+0000 7ff833bff700 1 -- 172.21.15.31:0/1360011822 wait complete. 2023-07-29T16:30:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 Processor -- start 2023-07-29T16:30:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 -- start start 2023-07-29T16:30:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 0x7ff82c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff82c074d30 0x7ff82c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c1081c0 0x7ff82c19f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c19fed0 con 0x7ff82c1081c0 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1a0010 con 0x7ff82c074d30 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff833bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1a01a0 con 0x7ff82c073750 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff83199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 0x7ff82c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff83199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 0x7ff82c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36984/0 (socket says 172.21.15.31:36984) 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff83199b700 1 -- 172.21.15.31:0/2948258114 learned_addr learned my addr 172.21.15.31:0/2948258114 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:08.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff83119a700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff82c074d30 0x7ff82c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.454+0000 7ff83219c700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c1081c0 0x7ff82c19f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83199b700 1 -- 172.21.15.31:0/2948258114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff82c074d30 msgr2=0x7ff82c198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83199b700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff82c074d30 0x7ff82c198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83199b700 1 -- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c1081c0 msgr2=0x7ff82c19f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83199b700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c1081c0 0x7ff82c19f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83199b700 1 -- 172.21.15.31:0/2948258114 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff824009d00 con 0x7ff82c073750 2023-07-29T16:30:08.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83219c700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c1081c0 0x7ff82c19f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff83199b700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 0x7ff82c198350 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff82800e9a0 tx=0x7ff82800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff822ffd700 1 -- 172.21.15.31:0/2948258114 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828009790 con 0x7ff82c073750 2023-07-29T16:30:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff822ffd700 1 -- 172.21.15.31:0/2948258114 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff82800cbd0 con 0x7ff82c073750 2023-07-29T16:30:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff822ffd700 1 -- 172.21.15.31:0/2948258114 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828018680 con 0x7ff82c073750 2023-07-29T16:30:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82c1a0480 con 0x7ff82c073750 2023-07-29T16:30:08.459 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.455+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c1a0940 con 0x7ff82c073750 2023-07-29T16:30:08.460 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.460+0000 7ff822ffd700 1 -- 172.21.15.31:0/2948258114 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff82800cd40 con 0x7ff82c073750 2023-07-29T16:30:08.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.461+0000 7ff822ffd700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8180a0810 0x7ff8180a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.461+0000 7ff847445b80 1 Processor -- start 2023-07-29T16:30:08.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.462+0000 7ff822ffd700 1 -- 172.21.15.31:0/2948258114 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff828014070 con 0x7ff82c073750 2023-07-29T16:30:08.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.462+0000 7ff847445b80 1 -- start start 2023-07-29T16:30:08.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.462+0000 7ff83119a700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8180a0810 0x7ff8180a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.462+0000 7ff847445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 0x561f41324ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.462+0000 7ff847445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f41325d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.462+0000 7ff847445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f413262f0 0x561f41326740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff847445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f4127ca10 con 0x561f413262f0 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff847445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f4127a3d0 con 0x561f41324be0 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff847445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f41275cc0 con 0x561f41325530 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 0x561f41324ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 0x561f41324ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34468/0 (socket says 172.21.15.31:34468) 2023-07-29T16:30:08.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 -- 172.21.15.31:0/4103910416 learned_addr learned my addr 172.21.15.31:0/4103910416 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83219c700 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f413262f0 0x561f41326740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83119a700 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f41325d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 -- 172.21.15.31:0/4103910416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 msgr2=0x561f41325d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f41325d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 -- 172.21.15.31:0/4103910416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f413262f0 msgr2=0x561f41326740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f413262f0 0x561f41326740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83199b700 1 -- 172.21.15.31:0/4103910416 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f410a3de0 con 0x561f41324be0 2023-07-29T16:30:08.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83119a700 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f41325d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:08.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.463+0000 7ff83119a700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8180a0810 0x7ff8180a2cc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009cb0 tx=0x7ff81c011720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.464+0000 7ff83199b700 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 0x561f41324ff0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff828099420 tx=0x7ff828099050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d718584d223c5a61 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.464+0000 7ff817fff700 1 -- 172.21.15.31:0/4103910416 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff828099490 con 0x561f41324be0 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.464+0000 7ff817fff700 1 -- 172.21.15.31:0/4103910416 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff82809d4c0 con 0x561f41324be0 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.464+0000 7ff817fff700 1 -- 172.21.15.31:0/4103910416 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a1450 con 0x561f41324be0 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.464+0000 7ff847445b80 1 -- 172.21.15.31:0/4103910416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 msgr2=0x561f41324ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.464+0000 7ff847445b80 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 0x561f41324ff0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff828099420 tx=0x7ff828099050 comp rx=0 tx=0).stop 2023-07-29T16:30:08.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.465+0000 7ff847445b80 1 -- 172.21.15.31:0/4103910416 shutdown_connections 2023-07-29T16:30:08.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.465+0000 7ff847445b80 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f413262f0 0x561f41326740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.465+0000 7ff847445b80 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41324be0 0x561f41324ff0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.465+0000 7ff847445b80 1 --2- 172.21.15.31:0/4103910416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f41325d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.465+0000 7ff847445b80 1 -- 172.21.15.31:0/4103910416 >> 172.21.15.31:0/4103910416 conn(0x561f4131a3e0 msgr2=0x561f4131f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.467+0000 7ff847445b80 1 -- 172.21.15.31:0/4103910416 shutdown_connections 2023-07-29T16:30:08.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.467+0000 7ff847445b80 1 -- 172.21.15.31:0/4103910416 wait complete. 2023-07-29T16:30:08.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.467+0000 7ff847445b80 1 -- start start 2023-07-29T16:30:08.469 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.468+0000 7ff847445b80 1 Processor -- start 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 0x561f41337050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41325530 0x561f41337590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f413262f0 0x561f4133e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f41212510 con 0x561f41325530 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f412121b0 con 0x561f413262f0 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f410f9920 con 0x561f41324be0 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff847445b80 10 client.0 _ll_register_callbacks cb 0x561f4121b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:08.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 0x561f41337050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 0x561f41337050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36998/0 (socket says 172.21.15.31:36998) 2023-07-29T16:30:08.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 -- 172.21.15.31:0/3880831555 learned_addr learned my addr 172.21.15.31:0/3880831555 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:08.471 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83219c700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f413262f0 0x561f4133e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83119a700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41325530 0x561f41337590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 -- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f413262f0 msgr2=0x561f4133e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f413262f0 0x561f4133e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 -- 172.21.15.31:0/3880831555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41325530 msgr2=0x561f41337590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41325530 0x561f41337590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.469+0000 7ff83199b700 1 -- 172.21.15.31:0/3880831555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c1a0940 con 0x561f41324be0 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.470+0000 7ff83199b700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 0x561f41337050 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff8280a13b0 tx=0x7ff82809d2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f3fac2a5d8247f6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.470+0000 7ff8157fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:08.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.470+0000 7ff8157fa700 1 -- 172.21.15.31:0/3880831555 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a1450 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.470+0000 7ff8157fa700 1 -- 172.21.15.31:0/3880831555 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff8280ae020 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.470+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f412a0d10 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff8157fa700 1 -- 172.21.15.31:0/3880831555 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8280a15b0 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 10 client.29782 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x561f4108cc60 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff8157fa700 1 -- 172.21.15.31:0/3880831555 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561f412a0d10 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff8157fa700 1 -- 172.21.15.31:0/3880831555 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff8280983f0 con 0x561f41324be0 2023-07-29T16:30:08.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 10 client.29782 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:08.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 10 client.29782 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:08.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 20 client.29782 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:08.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413428c0 0x561f41344cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 4 client.29782 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:08.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.471+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f412f3d70 con 0x561f413428c0 2023-07-29T16:30:08.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.472+0000 7ff83119a700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413428c0 0x561f41344cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.473+0000 7ff83119a700 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413428c0 0x561f41344cd0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=986bfb1050b26c29 server_cookie=d55e2e285edb7c8f in_seq=0 out_seq=0 2023-07-29T16:30:08.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.473+0000 7ff8157fa700 10 client.29782 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.474+0000 7ff8157fa700 1 -- 172.21.15.31:0/3880831555 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561f412f3d70 con 0x561f413428c0 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.474+0000 7ff8157fa700 10 client.29782 handle_command_reply: tid=0 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.474+0000 7ff847445b80 1 client.29782 shutdown 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.474+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 msgr2=0x561f41337050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.474+0000 7ff847445b80 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 0x561f41337050 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7ff8280a13b0 tx=0x7ff82809d2a0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 shutdown_connections 2023-07-29T16:30:08.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413428c0 0x561f41344cd0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f413262f0 0x561f4133e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41324be0 0x561f41337050 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 --2- 172.21.15.31:0/3880831555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41325530 0x561f41337590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 >> 172.21.15.31:0/3880831555 conn(0x561f4131a3e0 msgr2=0x561f4131f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff8157fa700 0 client.29782 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 shutdown_connections 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.475+0000 7ff847445b80 1 -- 172.21.15.31:0/3880831555 wait complete. 2023-07-29T16:30:08.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.476+0000 7ff847445b80 20 client.29782 trim_cache size 0 max 16384 2023-07-29T16:30:08.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.498+0000 7ff847445b80 1 Processor -- start 2023-07-29T16:30:08.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 -- start start 2023-07-29T16:30:08.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f4131a3e0 0x561f4133c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4131f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 0x561f413448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f412a0d10 con 0x561f41324be0 2023-07-29T16:30:08.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f41338240 con 0x561f41325530 2023-07-29T16:30:08.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff847445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f4131c460 con 0x561f4131a3e0 2023-07-29T16:30:08.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff83219c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 0x561f413448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff83199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f4131a3e0 0x561f4133c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff83219c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 0x561f413448d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34492/0 (socket says 172.21.15.31:34492) 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff83219c700 1 -- 172.21.15.31:0/842614965 learned_addr learned my addr 172.21.15.31:0/842614965 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff83199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f4131a3e0 0x561f4133c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37008/0 (socket says 172.21.15.31:37008) 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.499+0000 7ff83119a700 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4131f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff83219c700 1 -- 172.21.15.31:0/842614965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f4131a3e0 msgr2=0x561f4133c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff83219c700 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f4131a3e0 0x561f4133c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.502 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff83219c700 1 -- 172.21.15.31:0/842614965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 msgr2=0x561f4131f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff83219c700 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4131f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff83219c700 1 -- 172.21.15.31:0/842614965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f410ca250 con 0x561f41325530 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff83219c700 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 0x561f413448d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff82400b0a0 tx=0x7ff8240093a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3117d66de39efb74 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff820bff700 1 -- 172.21.15.31:0/842614965 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff82400e040 con 0x561f41325530 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff820bff700 1 -- 172.21.15.31:0/842614965 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff824021070 con 0x561f41325530 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.500+0000 7ff820bff700 1 -- 172.21.15.31:0/842614965 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff824004ea0 con 0x561f41325530 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 -- 172.21.15.31:0/842614965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 msgr2=0x561f413448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.503 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 0x561f413448d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff82400b0a0 tx=0x7ff8240093a0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 -- 172.21.15.31:0/842614965 shutdown_connections 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f41325530 0x561f413448d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f4131a3e0 0x561f4133c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 --2- 172.21.15.31:0/842614965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4131f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 -- 172.21.15.31:0/842614965 >> 172.21.15.31:0/842614965 conn(0x561f41320aa0 msgr2=0x561f4131f900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 -- 172.21.15.31:0/842614965 shutdown_connections 2023-07-29T16:30:08.504 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.501+0000 7ff847445b80 1 -- 172.21.15.31:0/842614965 wait complete. 2023-07-29T16:30:08.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 -- start start 2023-07-29T16:30:08.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 Processor -- start 2023-07-29T16:30:08.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 0x561f4133c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4133a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f4133aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f4133c970 con 0x561f41324be0 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f4133b210 con 0x561f4131a3e0 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f4133b3d0 con 0x561f41325530 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.502+0000 7ff847445b80 10 client.0 _ll_register_callbacks cb 0x561f4121b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 0x561f4133c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83219c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f4133aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 0x561f4133c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34496/0 (socket says 172.21.15.31:34496) 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 -- 172.21.15.31:0/3824887031 learned_addr learned my addr 172.21.15.31:0/3824887031 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83119a700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4133a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 -- 172.21.15.31:0/3824887031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 msgr2=0x561f4133aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f4133aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 -- 172.21.15.31:0/3824887031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 msgr2=0x561f4133a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4133a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 -- 172.21.15.31:0/3824887031 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c10b260 con 0x561f4131a3e0 2023-07-29T16:30:08.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83219c700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f4133aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:08.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff83199b700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 0x561f4133c150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff828098820 tx=0x7ff8280c98d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33877dc9ba89728c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff8157fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:08.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.503+0000 7ff8157fa700 1 -- 172.21.15.31:0/3824887031 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8280a1650 con 0x561f4131a3e0 2023-07-29T16:30:08.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff8157fa700 1 -- 172.21.15.31:0/3824887031 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff8280ae040 con 0x561f4131a3e0 2023-07-29T16:30:08.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff82809d4c0 con 0x561f4131a3e0 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff8157fa700 1 -- 172.21.15.31:0/3824887031 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff828098b90 con 0x561f4131a3e0 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 10 client.29792 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x561f410a22b0 con 0x561f4131a3e0 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff8157fa700 1 -- 172.21.15.31:0/3824887031 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff82809d4c0 con 0x561f4131a3e0 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff8157fa700 1 -- 172.21.15.31:0/3824887031 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff8280c9270 con 0x561f4131a3e0 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 10 client.29792 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:08.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 10 client.29792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 20 client.29792 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413262f0 0x561f413975c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 4 client.29792 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561f4126ed40 con 0x561f413262f0 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.504+0000 7ff83119a700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413262f0 0x561f413975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff83119a700 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413262f0 0x561f413975c0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd5f3161b3ccdfe4 server_cookie=179b05fa26df43c7 in_seq=0 out_seq=0 2023-07-29T16:30:08.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff8157fa700 10 client.29792 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff8157fa700 1 -- 172.21.15.31:0/3824887031 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561f4126ed40 con 0x561f413262f0 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff8157fa700 10 client.29792 handle_command_reply: tid=0 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff847445b80 1 client.29792 shutdown 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 msgr2=0x561f4133c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff847445b80 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 0x561f4133c150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff828098820 tx=0x7ff8280c98d0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.505+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 shutdown_connections 2023-07-29T16:30:08.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561f413262f0 0x561f413975c0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f41325530 0x561f4133aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561f4131a3e0 0x561f4133c150 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 --2- 172.21.15.31:0/3824887031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561f41324be0 0x561f4133a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 >> 172.21.15.31:0/3824887031 conn(0x561f41320aa0 msgr2=0x561f4132c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff8157fa700 0 client.29792 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 shutdown_connections 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 1 -- 172.21.15.31:0/3824887031 wait complete. 2023-07-29T16:30:08.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.506+0000 7ff847445b80 20 client.29792 trim_cache size 0 max 16384 2023-07-29T16:30:08.515 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:30:08.515 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:30:08.515 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:30:08.515 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:30:08.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8180a0810 msgr2=0x7ff8180a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8180a0810 0x7ff8180a2cc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff81c009cb0 tx=0x7ff81c011720 comp rx=0 tx=0).stop 2023-07-29T16:30:08.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 msgr2=0x7ff82c198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 0x7ff82c198350 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff82800e9a0 tx=0x7ff82800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 shutdown_connections 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8180a0810 0x7ff8180a2cc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff82c1081c0 0x7ff82c19f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff82c073750 0x7ff82c198350 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 --2- 172.21.15.31:0/2948258114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff82c074d30 0x7ff82c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 >> 172.21.15.31:0/2948258114 conn(0x7ff82c0fb970 msgr2=0x7ff82c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 shutdown_connections 2023-07-29T16:30:08.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.507+0000 7ff833bff700 1 -- 172.21.15.31:0/2948258114 wait complete. 2023-07-29T16:30:08.526 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:30:08.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:30:08.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.747+0000 7fc72c1ff700 1 -- 172.21.15.31:0/1637864576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724074d90 msgr2=0x7fc7240731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.747+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/1637864576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724074d90 0x7fc7240731e0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc720009990 tx=0x7fc720009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 -- 172.21.15.31:0/1637864576 shutdown_connections 2023-07-29T16:30:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/1637864576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc724108200 0x7fc72410a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/1637864576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724074d90 0x7fc7240731e0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/1637864576 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc724073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 -- 172.21.15.31:0/1637864576 >> 172.21.15.31:0/1637864576 conn(0x7fc7240fb920 msgr2=0x7fc7240fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 -- 172.21.15.31:0/1637864576 shutdown_connections 2023-07-29T16:30:08.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.748+0000 7fc72c1ff700 1 -- 172.21.15.31:0/1637864576 wait complete. 2023-07-29T16:30:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.749+0000 7fc72c1ff700 1 Processor -- start 2023-07-29T16:30:08.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 -- start start 2023-07-29T16:30:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc7241982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc724074d90 0x7fc724198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724108200 0x7fc72419f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72419fe40 con 0x7fc724074d90 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72419ff80 con 0x7fc7240737b0 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc72c1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7241a0110 con 0x7fc724108200 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc729f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc7241982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc729f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc7241982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34512/0 (socket says 172.21.15.31:34512) 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.750+0000 7fc729f9b700 1 -- 172.21.15.31:0/700577155 learned_addr learned my addr 172.21.15.31:0/700577155 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc72979a700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc724074d90 0x7fc724198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc72a79c700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724108200 0x7fc72419f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc729f9b700 1 -- 172.21.15.31:0/700577155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724108200 msgr2=0x7fc72419f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc729f9b700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724108200 0x7fc72419f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc729f9b700 1 -- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc724074d90 msgr2=0x7fc724198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc729f9b700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc724074d90 0x7fc724198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc729f9b700 1 -- 172.21.15.31:0/700577155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7200095f0 con 0x7fc7240737b0 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc72a79c700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724108200 0x7fc72419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:08.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc729f9b700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc7241982c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc720005720 tx=0x7fc720009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72001d070 con 0x7fc7240737b0 2023-07-29T16:30:08.755 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7241a0390 con 0x7fc7240737b0 2023-07-29T16:30:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc720021410 con 0x7fc7240737b0 2023-07-29T16:30:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.751+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720017650 con 0x7fc7240737b0 2023-07-29T16:30:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.752+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7241a07c0 con 0x7fc7240737b0 2023-07-29T16:30:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.753+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7241068b0 con 0x7fc7240737b0 2023-07-29T16:30:08.756 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.755+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc720021580 con 0x7fc7240737b0 2023-07-29T16:30:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.757+0000 7fc71affd700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7100a07e0 0x7fc7100a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.757+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc7200cd1b0 con 0x7fc7240737b0 2023-07-29T16:30:08.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.757+0000 7fc72979a700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7100a07e0 0x7fc7100a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.758+0000 7fc72979a700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7100a07e0 0x7fc7100a2c90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc724074ab0 tx=0x7fc714009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:08.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:08.758+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc720096c20 con 0x7fc7240737b0 2023-07-29T16:30:08.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1168651 8.83 MB/sec execute 593 sec latency 40.703 ms 2023-07-29T16:30:09.047 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.047+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc72402cea0 con 0x7fc7240737b0 2023-07-29T16:30:09.048 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.048+0000 7fc71affd700 1 -- 172.21.15.31:0/700577155 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+10780 (secure 0 0 0) 0x7fc72009a240 con 0x7fc7240737b0 2023-07-29T16:30:09.049 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:30:09.049 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:29:23.015310+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:30:09.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7100a07e0 msgr2=0x7fc7100a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7100a07e0 0x7fc7100a2c90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc724074ab0 tx=0x7fc714009260 comp rx=0 tx=0).stop 2023-07-29T16:30:09.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 msgr2=0x7fc7241982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc7241982c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fc720005720 tx=0x7fc720009580 comp rx=0 tx=0).stop 2023-07-29T16:30:09.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 shutdown_connections 2023-07-29T16:30:09.051 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc7100a07e0 0x7fc7100a2c90 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc724108200 0x7fc72419f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.050+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc7240737b0 0x7fc7241982c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.051+0000 7fc72c1ff700 1 --2- 172.21.15.31:0/700577155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc724074d90 0x7fc724198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.052 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.051+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 >> 172.21.15.31:0/700577155 conn(0x7fc7240fb920 msgr2=0x7fc7240fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.051+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 shutdown_connections 2023-07-29T16:30:09.053 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.051+0000 7fc72c1ff700 1 -- 172.21.15.31:0/700577155 wait complete. 2023-07-29T16:30:09.053 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 34 2023-07-29T16:30:09.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:30:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 -- 172.21.15.31:0/3796269336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 msgr2=0x7f836c103b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 --2- 172.21.15.31:0/3796269336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c103b90 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8360009990 tx=0x7f8360009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 -- 172.21.15.31:0/3796269336 shutdown_connections 2023-07-29T16:30:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 --2- 172.21.15.31:0/3796269336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f836c1040d0 0x7f836c10a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 --2- 172.21.15.31:0/3796269336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f836c102550 0x7f836c102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 --2- 172.21.15.31:0/3796269336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c103b90 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 -- 172.21.15.31:0/3796269336 >> 172.21.15.31:0/3796269336 conn(0x7f836c0fdb90 msgr2=0x7f836c0fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 -- 172.21.15.31:0/3796269336 shutdown_connections 2023-07-29T16:30:09.310 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.308+0000 7f83711ff700 1 -- 172.21.15.31:0/3796269336 wait complete. 2023-07-29T16:30:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.309+0000 7f83711ff700 1 Processor -- start 2023-07-29T16:30:09.312 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.309+0000 7f83711ff700 1 -- start start 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f83711ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f836c102550 0x7f836c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f83711ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f83711ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f836c1040d0 0x7f836c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f83711ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c19fe30 con 0x7f836c1040d0 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f83711ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c19ff70 con 0x7f836c103740 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f83711ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c1a0100 con 0x7f836c102550 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f836a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.313 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f836a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34522/0 (socket says 172.21.15.31:34522) 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f836a59c700 1 -- 172.21.15.31:0/3699174329 learned_addr learned my addr 172.21.15.31:0/3699174329 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f836ad9d700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f836c102550 0x7f836c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.310+0000 7f836b59e700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f836c1040d0 0x7f836c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f836a59c700 1 -- 172.21.15.31:0/3699174329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f836c102550 msgr2=0x7f836c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f836a59c700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f836c102550 0x7f836c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f836a59c700 1 -- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f836c1040d0 msgr2=0x7f836c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.314 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f836a59c700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f836c1040d0 0x7f836c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f836a59c700 1 -- 172.21.15.31:0/3699174329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83600095f0 con 0x7f836c103740 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f836a59c700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c1987f0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f836000b620 tx=0x7f8360005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f8353fff700 1 -- 172.21.15.31:0/3699174329 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836001d070 con 0x7f836c103740 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836c1a0380 con 0x7f836c103740 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f8353fff700 1 -- 172.21.15.31:0/3699174329 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8360022470 con 0x7f836c103740 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f8353fff700 1 -- 172.21.15.31:0/3699174329 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360017650 con 0x7f836c103740 2023-07-29T16:30:09.315 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.311+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c1a07a0 con 0x7f836c103740 2023-07-29T16:30:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.315+0000 7f8353fff700 1 -- 172.21.15.31:0/3699174329 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f83600225e0 con 0x7f836c103740 2023-07-29T16:30:09.316 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.316+0000 7f8384b4fb80 1 Processor -- start 2023-07-29T16:30:09.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.316+0000 7f8384b4fb80 1 -- start start 2023-07-29T16:30:09.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.316+0000 7f8353fff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83540a0790 0x7f83540a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f6be0 0x55dd8c6f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c6f7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8353fff700 1 -- 172.21.15.31:0/3699174329 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f83600ccf80 con 0x7f836c103740 2023-07-29T16:30:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f82f0 0x55dd8c6f8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c64ea10 con 0x55dd8c6f82f0 2023-07-29T16:30:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c64c3d0 con 0x55dd8c6f6be0 2023-07-29T16:30:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c647cc0 con 0x55dd8c6f7530 2023-07-29T16:30:09.318 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836b59e700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83540a0790 0x7f83540a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f6be0 0x55dd8c6f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f82f0 0x55dd8c6f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.319 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c6f7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f6be0 0x55dd8c6f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34528/0 (socket says 172.21.15.31:34528) 2023-07-29T16:30:09.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836ad9d700 1 -- 172.21.15.31:0/3771423110 learned_addr learned my addr 172.21.15.31:0/3771423110 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836a59c700 1 -- 172.21.15.31:0/3771423110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f6be0 msgr2=0x55dd8c6f6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836a59c700 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f6be0 0x55dd8c6f6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836a59c700 1 -- 172.21.15.31:0/3771423110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f82f0 msgr2=0x55dd8c6f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836a59c700 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f82f0 0x55dd8c6f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.317+0000 7f836a59c700 1 -- 172.21.15.31:0/3771423110 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd8c475de0 con 0x55dd8c6f7530 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.318+0000 7f836a59c700 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c6f7d60 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8360022890 tx=0x7f8360022c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aed18fa34516f046 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.318+0000 7f83517fa700 1 -- 172.21.15.31:0/3771423110 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836009a6b0 con 0x55dd8c6f7530 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.318+0000 7f836b59e700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83540a0790 0x7f83540a2c40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8364009ec0 tx=0x7f83640092f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.318+0000 7f83517fa700 1 -- 172.21.15.31:0/3771423110 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f836009e6e0 con 0x55dd8c6f7530 2023-07-29T16:30:09.321 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.318+0000 7f83517fa700 1 -- 172.21.15.31:0/3771423110 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360099ba0 con 0x55dd8c6f7530 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 -- 172.21.15.31:0/3771423110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 msgr2=0x55dd8c6f7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c6f7d60 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8360022890 tx=0x7f8360022c80 comp rx=0 tx=0).stop 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 -- 172.21.15.31:0/3771423110 shutdown_connections 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f82f0 0x55dd8c6f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f6be0 0x55dd8c6f6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/3771423110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c6f7d60 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 -- 172.21.15.31:0/3771423110 >> 172.21.15.31:0/3771423110 conn(0x55dd8c6ec3e0 msgr2=0x55dd8c6f1990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.322 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 -- 172.21.15.31:0/3771423110 shutdown_connections 2023-07-29T16:30:09.323 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 -- 172.21.15.31:0/3771423110 wait complete. 2023-07-29T16:30:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.319+0000 7f8384b4fb80 1 -- start start 2023-07-29T16:30:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 Processor -- start 2023-07-29T16:30:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c700430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c700970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.325 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c5e4510 con 0x55dd8c6f6be0 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c5e41b0 con 0x55dd8c6f82f0 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c4cb920 con 0x55dd8c6f7530 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f8384b4fb80 10 client.0 _ll_register_callbacks cb 0x55dd8c5ed620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f836ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c700430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f836a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c700970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f836a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c700970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37058/0 (socket says 172.21.15.31:37058) 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.320+0000 7f836a59c700 1 -- 172.21.15.31:0/4077058428 learned_addr learned my addr 172.21.15.31:0/4077058428 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836b59e700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836a59c700 1 -- 172.21.15.31:0/4077058428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 msgr2=0x55dd8c7079b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836a59c700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7079b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836a59c700 1 -- 172.21.15.31:0/4077058428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 msgr2=0x55dd8c700430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836a59c700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c700430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836a59c700 1 -- 172.21.15.31:0/4077058428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c1a07a0 con 0x55dd8c6f7530 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836ad9d700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c700430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f836a59c700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c700970 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8360096bb0 tx=0x7f836009a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de4111e28af9c916 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f834eff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f834eff5700 1 -- 172.21.15.31:0/4077058428 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83600a27e0 con 0x55dd8c6f7530 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd8c672d10 con 0x55dd8c6f7530 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.321+0000 7f834eff5700 1 -- 172.21.15.31:0/4077058428 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8360098a50 con 0x55dd8c6f7530 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.322+0000 7f834eff5700 1 -- 172.21.15.31:0/4077058428 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f836009a6b0 con 0x55dd8c6f7530 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.322+0000 7f8384b4fb80 10 client.29806 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.322+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55dd8c45ec60 con 0x55dd8c6f7530 2023-07-29T16:30:09.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.322+0000 7f834eff5700 1 -- 172.21.15.31:0/4077058428 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dd8c672d10 con 0x55dd8c6f7530 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f834eff5700 1 -- 172.21.15.31:0/4077058428 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f836009e330 con 0x55dd8c6f7530 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f8384b4fb80 10 client.29806 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f8384b4fb80 10 client.29806 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f8384b4fb80 20 client.29806 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c70bcd0 0x55dd8c70e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f8384b4fb80 4 client.29806 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd8c6c5d70 con 0x55dd8c70bcd0 2023-07-29T16:30:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f836ad9d700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c70bcd0 0x55dd8c70e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f836ad9d700 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c70bcd0 0x55dd8c70e0b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=535262c43e5c765e server_cookie=eae24cfdb2d64d6 in_seq=0 out_seq=0 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.323+0000 7f834eff5700 10 client.29806 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f834eff5700 1 -- 172.21.15.31:0/4077058428 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55dd8c6c5d70 con 0x55dd8c70bcd0 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f834eff5700 10 client.29806 handle_command_reply: tid=0 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 client.29806 shutdown 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 msgr2=0x55dd8c700970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c700970 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8360096bb0 tx=0x7f836009a0d0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 shutdown_connections 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c70bcd0 0x55dd8c70e0b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c700430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/4077058428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c700970 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f834eff5700 0 client.29806 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.324+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 >> 172.21.15.31:0/4077058428 conn(0x55dd8c6ec3e0 msgr2=0x55dd8c6f1060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.325+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 shutdown_connections 2023-07-29T16:30:09.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.325+0000 7f8384b4fb80 1 -- 172.21.15.31:0/4077058428 wait complete. 2023-07-29T16:30:09.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.325+0000 7f8384b4fb80 20 client.29806 trim_cache size 0 max 16384 2023-07-29T16:30:09.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 Processor -- start 2023-07-29T16:30:09.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 -- start start 2023-07-29T16:30:09.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c708240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c707970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7003f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c6f3110 con 0x55dd8c6f6be0 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c7010b0 con 0x55dd8c6f82f0 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.339+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c6e96e0 con 0x55dd8c6f7530 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c707970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7003f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34540/0 (socket says 172.21.15.31:34540) 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c707970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37060/0 (socket says 172.21.15.31:37060) 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c708240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 -- 172.21.15.31:0/1734476212 learned_addr learned my addr 172.21.15.31:0/1734476212 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 -- 172.21.15.31:0/1734476212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 msgr2=0x55dd8c707970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c707970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 -- 172.21.15.31:0/1734476212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 msgr2=0x55dd8c708240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c708240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836b59e700 1 -- 172.21.15.31:0/1734476212 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd8c49c250 con 0x55dd8c6f82f0 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836a59c700 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c707970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.340+0000 7f836ad9d700 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c708240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.341+0000 7f836b59e700 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7003f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f836401a900 tx=0x7f836401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b983bc9609bcfe4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.341+0000 7f8351ffb700 1 -- 172.21.15.31:0/1734476212 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836400e070 con 0x55dd8c6f82f0 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.341+0000 7f8351ffb700 1 -- 172.21.15.31:0/1734476212 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f836400f040 con 0x55dd8c6f82f0 2023-07-29T16:30:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.341+0000 7f8351ffb700 1 -- 172.21.15.31:0/1734476212 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364026a80 con 0x55dd8c6f82f0 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1734476212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 msgr2=0x55dd8c7003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7003f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f836401a900 tx=0x7f836401d000 comp rx=0 tx=0).stop 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1734476212 shutdown_connections 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f82f0 0x55dd8c7003f0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c708240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1734476212 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f7530 0x55dd8c707970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1734476212 >> 172.21.15.31:0/1734476212 conn(0x55dd8c6ec3e0 msgr2=0x55dd8c6f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1734476212 shutdown_connections 2023-07-29T16:30:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1734476212 wait complete. 2023-07-29T16:30:09.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 -- start start 2023-07-29T16:30:09.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.342+0000 7f8384b4fb80 1 Processor -- start 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c703ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 0x55dd8c7040e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f82f0 0x55dd8c706f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c707470 con 0x55dd8c6f6be0 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c7075e0 con 0x55dd8c6f7530 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd8c7077a0 con 0x55dd8c6f82f0 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f8384b4fb80 10 client.0 _ll_register_callbacks cb 0x55dd8c5ed620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f836ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c703ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f836a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 0x55dd8c7040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f836a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 0x55dd8c7040e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34550/0 (socket says 172.21.15.31:34550) 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f836b59e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f82f0 0x55dd8c706f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f836ad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c703ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59312/0 (socket says 172.21.15.31:59312) 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.343+0000 7f836a59c700 1 -- 172.21.15.31:0/1018441607 learned_addr learned my addr 172.21.15.31:0/1018441607 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836a59c700 1 -- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f82f0 msgr2=0x55dd8c706f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836a59c700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f82f0 0x55dd8c706f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836a59c700 1 -- 172.21.15.31:0/1018441607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 msgr2=0x55dd8c703ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836a59c700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c703ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836a59c700 1 -- 172.21.15.31:0/1018441607 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835c004410 con 0x55dd8c6f7530 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836b59e700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f82f0 0x55dd8c706f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836ad9d700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c703ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f836a59c700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 0x55dd8c7040e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8360000c00 tx=0x7f8360009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38341546e619331 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f834eff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f834eff5700 1 -- 172.21.15.31:0/1018441607 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836009a910 con 0x55dd8c6f7530 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f834eff5700 1 -- 172.21.15.31:0/1018441607 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8360096800 con 0x55dd8c6f7530 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f836009e6e0 con 0x55dd8c6f7530 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f834eff5700 1 -- 172.21.15.31:0/1018441607 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8360009290 con 0x55dd8c6f7530 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f8384b4fb80 10 client.29816 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.344+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55dd8c45ec60 con 0x55dd8c6f7530 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f834eff5700 1 -- 172.21.15.31:0/1018441607 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f836009e6e0 con 0x55dd8c6f7530 2023-07-29T16:30:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f834eff5700 1 -- 172.21.15.31:0/1018441607 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f83600c4b00 con 0x55dd8c6f7530 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f8384b4fb80 10 client.29816 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f8384b4fb80 10 client.29816 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f8384b4fb80 20 client.29816 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c75e580 0x55dd8c760990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f8384b4fb80 4 client.29816 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd8c640d40 con 0x55dd8c75e580 2023-07-29T16:30:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.345+0000 7f836ad9d700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c75e580 0x55dd8c760990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f836ad9d700 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c75e580 0x55dd8c760990 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58b772fa04de8bdd server_cookie=7e7c7720c0f21c87 in_seq=0 out_seq=0 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f834eff5700 10 client.29816 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f834eff5700 1 -- 172.21.15.31:0/1018441607 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd8c640d40 con 0x55dd8c75e580 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f834eff5700 10 client.29816 handle_command_reply: tid=0 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 client.29816 shutdown 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 msgr2=0x55dd8c7040e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 0x55dd8c7040e0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8360000c00 tx=0x7f8360009910 comp rx=0 tx=0).stop 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 shutdown_connections 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55dd8c6f82f0 0x55dd8c706f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dd8c6f6be0 0x55dd8c703ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55dd8c75e580 0x55dd8c760990 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 --2- 172.21.15.31:0/1018441607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55dd8c6f7530 0x55dd8c7040e0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f834eff5700 0 client.29816 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:09.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 >> 172.21.15.31:0/1018441607 conn(0x55dd8c6ec3e0 msgr2=0x55dd8c708240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 shutdown_connections 2023-07-29T16:30:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.346+0000 7f8384b4fb80 1 -- 172.21.15.31:0/1018441607 wait complete. 2023-07-29T16:30:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f8384b4fb80 20 client.29816 trim_cache size 0 max 16384 2023-07-29T16:30:09.357 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:30:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83540a0790 msgr2=0x7f83540a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83540a0790 0x7f83540a2c40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8364009ec0 tx=0x7f83640092f0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 msgr2=0x7f836c1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c1987f0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f836000b620 tx=0x7f8360005740 comp rx=0 tx=0).stop 2023-07-29T16:30:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 shutdown_connections 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f83540a0790 0x7f83540a2c40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f836c1040d0 0x7f836c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f836c102550 0x7f836c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 --2- 172.21.15.31:0/3699174329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f836c103740 0x7f836c1987f0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 >> 172.21.15.31:0/3699174329 conn(0x7f836c0fdb90 msgr2=0x7f836c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 shutdown_connections 2023-07-29T16:30:09.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.347+0000 7f83711ff700 1 -- 172.21.15.31:0/3699174329 wait complete. 2023-07-29T16:30:09.364 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:09 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/700577155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:30:09.366 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:30:09.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:09 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/700577155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:30:09.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:09 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/700577155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:30:09.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2044591318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb88102550 msgr2=0x7fbb88102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2044591318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb88102550 0x7fbb88102960 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fbb78009990 tx=0x7fbb78009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.590 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2044591318 shutdown_connections 2023-07-29T16:30:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2044591318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb881040d0 0x7fbb8810a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2044591318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb88102550 0x7fbb88102960 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2044591318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb88103740 0x7fbb88103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2044591318 >> 172.21.15.31:0/2044591318 conn(0x7fbb880fdb90 msgr2=0x7fbb880fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2044591318 shutdown_connections 2023-07-29T16:30:09.591 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.590+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2044591318 wait complete. 2023-07-29T16:30:09.592 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.591+0000 7fbb8d9ff700 1 Processor -- start 2023-07-29T16:30:09.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 -- start start 2023-07-29T16:30:09.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 0x7fbb88198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb88103740 0x7fbb88198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb881040d0 0x7fbb8819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8819fea0 con 0x7fbb88103740 2023-07-29T16:30:09.593 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8819ffe0 con 0x7fbb88102550 2023-07-29T16:30:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb8d9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881a0170 con 0x7fbb881040d0 2023-07-29T16:30:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb867fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb88103740 0x7fbb88198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 0x7fbb88198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb877fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb881040d0 0x7fbb8819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 0x7fbb88198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34578/0 (socket says 172.21.15.31:34578) 2023-07-29T16:30:09.594 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.592+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2851303544 learned_addr learned my addr 172.21.15.31:0/2851303544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2851303544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb881040d0 msgr2=0x7fbb8819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb881040d0 0x7fbb8819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.595 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb88103740 msgr2=0x7fbb88198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb88103740 0x7fbb88198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2851303544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80009c40 con 0x7fbb88102550 2023-07-29T16:30:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 0x7fbb88198320 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbb78005720 tx=0x7fbb78009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb8c9fd700 1 -- 172.21.15.31:0/2851303544 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7801d070 con 0x7fbb88102550 2023-07-29T16:30:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb8c9fd700 1 -- 172.21.15.31:0/2851303544 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb78021410 con 0x7fbb88102550 2023-07-29T16:30:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb8c9fd700 1 -- 172.21.15.31:0/2851303544 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78017650 con 0x7fbb88102550 2023-07-29T16:30:09.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.593+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb780095f0 con 0x7fbb88102550 2023-07-29T16:30:09.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.594+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881a0690 con 0x7fbb88102550 2023-07-29T16:30:09.597 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.597+0000 7fbb8c9fd700 1 -- 172.21.15.31:0/2851303544 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbb78021580 con 0x7fbb88102550 2023-07-29T16:30:09.598 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.598+0000 7fbba127db80 1 Processor -- start 2023-07-29T16:30:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.598+0000 7fbba127db80 1 -- start start 2023-07-29T16:30:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb8c9fd700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb700a0790 0x7fbb700a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbba127db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af6664ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbba127db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6665d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbba127db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af66662f0 0x563af6666740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbba127db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af65bca10 con 0x563af6665530 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbba127db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af65ba3d0 con 0x563af6664be0 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbba127db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af65b5cc0 con 0x563af66662f0 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb8c9fd700 1 -- 172.21.15.31:0/2851303544 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbb780ccda0 con 0x7fbb88102550 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af6664ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af6664ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34586/0 (socket says 172.21.15.31:34586) 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2117669020 learned_addr learned my addr 172.21.15.31:0/2117669020 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb700a0790 0x7fbb700a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2117669020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af66662f0 msgr2=0x563af6666740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af66662f0 0x563af6666740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af66662f0 0x563af6666740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2117669020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 msgr2=0x563af6665d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6665d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2117669020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563af63e3de0 con 0x563af6664be0 2023-07-29T16:30:09.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.599+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af66662f0 0x563af6666740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:09.603 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.600+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af6664ff0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbb780ca7e0 tx=0x7fbb7809a320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=935c9ef07e1d65b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.605 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.600+0000 7fbb6dffb700 1 -- 172.21.15.31:0/2117669020 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7809a4d0 con 0x563af6664be0 2023-07-29T16:30:09.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.600+0000 7fbb6dffb700 1 -- 172.21.15.31:0/2117669020 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb780af070 con 0x563af6664be0 2023-07-29T16:30:09.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.600+0000 7fbb6dffb700 1 -- 172.21.15.31:0/2117669020 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb780a2920 con 0x563af6664be0 2023-07-29T16:30:09.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 -- 172.21.15.31:0/2117669020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 msgr2=0x563af6664ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af6664ff0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbb780ca7e0 tx=0x7fbb7809a320 comp rx=0 tx=0).stop 2023-07-29T16:30:09.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb700a0790 0x7fbb700a2c40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbb80009fd0 tx=0x7fbb80009920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.606 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 -- 172.21.15.31:0/2117669020 shutdown_connections 2023-07-29T16:30:09.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af66662f0 0x563af6666740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af6664ff0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 --2- 172.21.15.31:0/2117669020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6665d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.607 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 -- 172.21.15.31:0/2117669020 >> 172.21.15.31:0/2117669020 conn(0x563af665a3e0 msgr2=0x563af665f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 -- 172.21.15.31:0/2117669020 shutdown_connections 2023-07-29T16:30:09.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 -- 172.21.15.31:0/2117669020 wait complete. 2023-07-29T16:30:09.608 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.601+0000 7fbba127db80 1 -- start start 2023-07-29T16:30:09.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 Processor -- start 2023-07-29T16:30:09.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af666e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.609 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af666e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af66662f0 0x563af6675a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af6552510 con 0x563af66662f0 2023-07-29T16:30:09.610 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af65521b0 con 0x563af6664be0 2023-07-29T16:30:09.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af6439920 con 0x563af6665530 2023-07-29T16:30:09.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbba127db80 10 client.0 _ll_register_callbacks cb 0x563af655b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:09.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.602+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af666e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.611 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af666e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34592/0 (socket says 172.21.15.31:34592) 2023-07-29T16:30:09.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 -- 172.21.15.31:0/3491611565 learned_addr learned my addr 172.21.15.31:0/3491611565 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb877fe700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af66662f0 0x563af6675a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb867fc700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af666e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.612 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 -- 172.21.15.31:0/3491611565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 msgr2=0x563af666e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af666e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 -- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af66662f0 msgr2=0x563af6675a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af66662f0 0x563af6675a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 -- 172.21.15.31:0/3491611565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881a0690 con 0x563af6664be0 2023-07-29T16:30:09.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af666e490 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800b2e0 tx=0x7fbb7809a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b41d9a7937f05c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.613 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.603+0000 7fbb66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:09.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbb66ffd700 1 -- 172.21.15.31:0/3491611565 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb780a2900 con 0x563af6664be0 2023-07-29T16:30:09.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563af65e0d10 con 0x563af6664be0 2023-07-29T16:30:09.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbb66ffd700 1 -- 172.21.15.31:0/3491611565 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbb7809a4d0 con 0x563af6664be0 2023-07-29T16:30:09.614 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbb66ffd700 1 -- 172.21.15.31:0/3491611565 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb7809e500 con 0x563af6664be0 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 10 client.29828 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x563af63ccc60 con 0x563af6664be0 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbb66ffd700 1 -- 172.21.15.31:0/3491611565 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb780c8080 con 0x563af6664be0 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbb66ffd700 1 -- 172.21.15.31:0/3491611565 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbb7800fac0 con 0x563af6664be0 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 10 client.29828 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 10 client.29828 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 20 client.29828 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:09.615 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af6679bd0 0x563af667bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 4 client.29828 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:09.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.604+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563af6633d70 con 0x563af6679bd0 2023-07-29T16:30:09.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.605+0000 7fbb867fc700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af6679bd0 0x563af667bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.605+0000 7fbb66ffd700 10 client.29828 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:09.616 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.605+0000 7fbb867fc700 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af6679bd0 0x563af667bfe0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=421cb0d6caca2bea server_cookie=a8753f41b145876b in_seq=0 out_seq=0 2023-07-29T16:30:09.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbb66ffd700 1 -- 172.21.15.31:0/3491611565 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563af6633d70 con 0x563af6679bd0 2023-07-29T16:30:09.617 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbb66ffd700 10 client.29828 handle_command_reply: tid=0 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 client.29828 shutdown 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 msgr2=0x563af666e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af666e490 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbb7800b2e0 tx=0x7fbb7809a110 comp rx=0 tx=0).stop 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 shutdown_connections 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af6679bd0 0x563af667bfe0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af66662f0 0x563af6675a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af6664be0 0x563af666e490 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbb66ffd700 0 client.29828 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:09.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 --2- 172.21.15.31:0/3491611565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af666e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 >> 172.21.15.31:0/3491611565 conn(0x563af665a3e0 msgr2=0x563af665f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.606+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 shutdown_connections 2023-07-29T16:30:09.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.607+0000 7fbba127db80 1 -- 172.21.15.31:0/3491611565 wait complete. 2023-07-29T16:30:09.619 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.607+0000 7fbba127db80 20 client.29828 trim_cache size 0 max 16384 2023-07-29T16:30:09.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 Processor -- start 2023-07-29T16:30:09.621 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 -- start start 2023-07-29T16:30:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6664be0 0x563af665f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af66759d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af66691b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af6633d70 con 0x563af6664be0 2023-07-29T16:30:09.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af66577e0 con 0x563af66662f0 2023-07-29T16:30:09.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.621+0000 7fbba127db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af666f4d0 con 0x563af6665530 2023-07-29T16:30:09.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6664be0 0x563af665f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af66691b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb867fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af66759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af66691b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34598/0 (socket says 172.21.15.31:34598) 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb867fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af66759d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37132/0 (socket says 172.21.15.31:37132) 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 -- 172.21.15.31:0/467398469 learned_addr learned my addr 172.21.15.31:0/467398469 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 -- 172.21.15.31:0/467398469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 msgr2=0x563af66759d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af66759d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 -- 172.21.15.31:0/467398469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6664be0 msgr2=0x563af665f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6664be0 0x563af665f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.622+0000 7fbb877fe700 1 -- 172.21.15.31:0/467398469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563af640a250 con 0x563af66662f0 2023-07-29T16:30:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.623+0000 7fbb877fe700 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af66691b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbb80003d50 tx=0x7fbb80015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b2788e8d4a1b6d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.623+0000 7fbb6e7fc700 1 -- 172.21.15.31:0/467398469 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000e040 con 0x563af66662f0 2023-07-29T16:30:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.623+0000 7fbb6e7fc700 1 -- 172.21.15.31:0/467398469 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb8000f040 con 0x563af66662f0 2023-07-29T16:30:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.623+0000 7fbb6e7fc700 1 -- 172.21.15.31:0/467398469 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80017860 con 0x563af66662f0 2023-07-29T16:30:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 -- 172.21.15.31:0/467398469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 msgr2=0x563af66691b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af66691b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fbb80003d50 tx=0x7fbb80015000 comp rx=0 tx=0).stop 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 -- 172.21.15.31:0/467398469 shutdown_connections 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af66691b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6664be0 0x563af665f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 --2- 172.21.15.31:0/467398469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6665530 0x563af66759d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 -- 172.21.15.31:0/467398469 >> 172.21.15.31:0/467398469 conn(0x563af665a3e0 msgr2=0x563af665f900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 -- 172.21.15.31:0/467398469 shutdown_connections 2023-07-29T16:30:09.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 -- 172.21.15.31:0/467398469 wait complete. 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.624+0000 7fbba127db80 1 -- start start 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 Processor -- start 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6664be0 0x563af6675600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6672230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af6672770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af6672cb0 con 0x563af6665530 2023-07-29T16:30:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af6672e20 con 0x563af66662f0 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563af66cb710 con 0x563af6664be0 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbba127db80 10 client.0 _ll_register_callbacks cb 0x563af655b620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6664be0 0x563af6675600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbb867fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6672230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbb877fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af6672770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbb86ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6664be0 0x563af6675600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37140/0 (socket says 172.21.15.31:37140) 2023-07-29T16:30:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.625+0000 7fbb86ffd700 1 -- 172.21.15.31:0/2065368922 learned_addr learned my addr 172.21.15.31:0/2065368922 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb877fe700 1 -- 172.21.15.31:0/2065368922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6664be0 msgr2=0x563af6675600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6664be0 0x563af6675600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb877fe700 1 -- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 msgr2=0x563af6672230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6672230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb877fe700 1 -- 172.21.15.31:0/2065368922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb80003a00 con 0x563af66662f0 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb877fe700 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af6672770 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbb80010010 tx=0x7fbb80011c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a73cd4b878729f91 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb66ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:09.631 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb66ffd700 1 -- 172.21.15.31:0/2065368922 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000e050 con 0x563af66662f0 2023-07-29T16:30:09.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbb66ffd700 1 -- 172.21.15.31:0/2065368922 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbb8001d040 con 0x563af66662f0 2023-07-29T16:30:09.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.626+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb780af070 con 0x563af66662f0 2023-07-29T16:30:09.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbb66ffd700 1 -- 172.21.15.31:0/2065368922 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb80017560 con 0x563af66662f0 2023-07-29T16:30:09.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 10 client.29834 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:09.632 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x563af63e22b0 con 0x563af66662f0 2023-07-29T16:30:09.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbb66ffd700 1 -- 172.21.15.31:0/2065368922 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb80041020 con 0x563af66662f0 2023-07-29T16:30:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbb66ffd700 1 -- 172.21.15.31:0/2065368922 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbb80023710 con 0x563af66662f0 2023-07-29T16:30:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 10 client.29834 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 10 client.29834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:30:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 20 client.29834 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af66cce50 0x563af66cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 4 client.29834 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.627+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563af65aed40 con 0x563af66cce50 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.628+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af66cce50 0x563af66cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.628+0000 7fbb86ffd700 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af66cce50 0x563af66cf260 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ae72a71f18dfba server_cookie=c7d207c73cb471fa in_seq=0 out_seq=0 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.628+0000 7fbb66ffd700 10 client.29834 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.628+0000 7fbb66ffd700 1 -- 172.21.15.31:0/2065368922 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563af65aed40 con 0x563af66cce50 2023-07-29T16:30:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.628+0000 7fbb66ffd700 10 client.29834 handle_command_reply: tid=0 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.628+0000 7fbba127db80 1 client.29834 shutdown 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 msgr2=0x563af6672770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af6672770 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fbb80010010 tx=0x7fbb80011c00 comp rx=0 tx=0).stop 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 shutdown_connections 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x563af66cce50 0x563af66cf260 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563af66662f0 0x563af6672770 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563af6664be0 0x563af6675600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 --2- 172.21.15.31:0/2065368922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563af6665530 0x563af6672230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbb66ffd700 0 client.29834 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 >> 172.21.15.31:0/2065368922 conn(0x563af665a3e0 msgr2=0x563af666e240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 shutdown_connections 2023-07-29T16:30:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 1 -- 172.21.15.31:0/2065368922 wait complete. 2023-07-29T16:30:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.629+0000 7fbba127db80 20 client.29834 trim_cache size 0 max 16384 2023-07-29T16:30:09.638 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:30:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb700a0790 msgr2=0x7fbb700a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb700a0790 0x7fbb700a2c40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbb80009fd0 tx=0x7fbb80009920 comp rx=0 tx=0).stop 2023-07-29T16:30:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 msgr2=0x7fbb88198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 0x7fbb88198320 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbb78005720 tx=0x7fbb78009580 comp rx=0 tx=0).stop 2023-07-29T16:30:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 shutdown_connections 2023-07-29T16:30:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbb700a0790 0x7fbb700a2c40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb881040d0 0x7fbb8819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb88102550 0x7fbb88198320 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 --2- 172.21.15.31:0/2851303544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb88103740 0x7fbb88198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 >> 172.21.15.31:0/2851303544 conn(0x7fbb880fdb90 msgr2=0x7fbb88106c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 shutdown_connections 2023-07-29T16:30:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.630+0000 7fbb8d9ff700 1 -- 172.21.15.31:0/2851303544 wait complete. 2023-07-29T16:30:09.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:30:09.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2661400947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc074d30 msgr2=0x7f3dbc073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2661400947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc074d30 0x7f3dbc073180 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009990 tx=0x7f3dac009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2661400947 shutdown_connections 2023-07-29T16:30:09.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2661400947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dbc1081c0 0x7f3dbc10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2661400947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc074d30 0x7f3dbc073180 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2661400947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 0x7f3dbc073bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2661400947 >> 172.21.15.31:0/2661400947 conn(0x7f3dbc0fb970 msgr2=0x7f3dbc0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2661400947 shutdown_connections 2023-07-29T16:30:09.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.870+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2661400947 wait complete. 2023-07-29T16:30:09.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.871+0000 7f3dc09ff700 1 Processor -- start 2023-07-29T16:30:09.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 -- start start 2023-07-29T16:30:09.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 0x7f3dbc198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dbc074d30 0x7f3dbc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc1081c0 0x7f3dbc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc19fe10 con 0x7f3dbc074d30 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc19ff50 con 0x7f3dbc1081c0 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dc09ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc1a00e0 con 0x7f3dbc073750 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 0x7f3dbc198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc1081c0 0x7f3dbc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 0x7f3dbc198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59474/0 (socket says 172.21.15.31:59474) 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3db9d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dbc074d30 0x7f3dbc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.872+0000 7f3dba59c700 1 -- 172.21.15.31:0/2106332892 learned_addr learned my addr 172.21.15.31:0/2106332892 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dbad9d700 1 -- 172.21.15.31:0/2106332892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 msgr2=0x7f3dbc198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 0x7f3dbc198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dbad9d700 1 -- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dbc074d30 msgr2=0x7f3dbc1987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dbc074d30 0x7f3dbc1987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dbad9d700 1 -- 172.21.15.31:0/2106332892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac0095f0 con 0x7f3dbc1081c0 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc1081c0 0x7f3dbc19f840 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3db400efc0 tx=0x7f3db400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dab7fe700 1 -- 172.21.15.31:0/2106332892 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4018070 con 0x7f3dbc1081c0 2023-07-29T16:30:09.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dab7fe700 1 -- 172.21.15.31:0/2106332892 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3db400f040 con 0x7f3dbc1081c0 2023-07-29T16:30:09.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dab7fe700 1 -- 172.21.15.31:0/2106332892 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db400cd40 con 0x7f3dbc1081c0 2023-07-29T16:30:09.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc1a0360 con 0x7f3dbc1081c0 2023-07-29T16:30:09.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.873+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc1a0870 con 0x7f3dbc1081c0 2023-07-29T16:30:09.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dab7fe700 1 -- 172.21.15.31:0/2106332892 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3db4003a00 con 0x7f3dbc1081c0 2023-07-29T16:30:09.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 Processor -- start 2023-07-29T16:30:09.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 -- start start 2023-07-29T16:30:09.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bbbe0 0x559d576bbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576bcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bd2f0 0x559d576bd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d57613a10 con 0x559d576bbbe0 2023-07-29T16:30:09.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576113d0 con 0x559d576bc530 2023-07-29T16:30:09.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.878+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d5760ccc0 con 0x559d576bd2f0 2023-07-29T16:30:09.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576bcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576bcd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44342/0 (socket says 172.21.15.31:44342) 2023-07-29T16:30:09.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 -- 172.21.15.31:0/2045229330 learned_addr learned my addr 172.21.15.31:0/2045229330 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bbbe0 0x559d576bbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bd2f0 0x559d576bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 -- 172.21.15.31:0/2045229330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bd2f0 msgr2=0x559d576bd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bd2f0 0x559d576bd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 -- 172.21.15.31:0/2045229330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bbbe0 msgr2=0x559d576bbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bbbe0 0x559d576bbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 -- 172.21.15.31:0/2045229330 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d5743ade0 con 0x559d576bc530 2023-07-29T16:30:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.879+0000 7f3dba59c700 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576bcd60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005f40 tx=0x7f3dac005bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=507d80e87d3f95de server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3dab7fe700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3da40a0810 0x7f3da40a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3da8ff9700 1 -- 172.21.15.31:0/2045229330 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac01d070 con 0x559d576bc530 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3da8ff9700 1 -- 172.21.15.31:0/2045229330 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dac01e040 con 0x559d576bc530 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3da8ff9700 1 -- 172.21.15.31:0/2045229330 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac004aa0 con 0x559d576bc530 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3dab7fe700 1 -- 172.21.15.31:0/2106332892 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3db40cb570 con 0x7f3dbc1081c0 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3da40a0810 0x7f3da40a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3dd4422b80 1 -- 172.21.15.31:0/2045229330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 msgr2=0x559d576bcd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3da40a0810 0x7f3da40a2cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x559d576bb980 tx=0x7f3db0015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.880+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576bcd60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005f40 tx=0x7f3dac005bc0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 -- 172.21.15.31:0/2045229330 shutdown_connections 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bd2f0 0x559d576bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bbbe0 0x559d576bbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/2045229330 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576bcd60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 -- 172.21.15.31:0/2045229330 >> 172.21.15.31:0/2045229330 conn(0x559d576b13e0 msgr2=0x559d576b6990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 -- 172.21.15.31:0/2045229330 shutdown_connections 2023-07-29T16:30:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 -- 172.21.15.31:0/2045229330 wait complete. 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 -- start start 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.881+0000 7f3dd4422b80 1 Processor -- start 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bc530 0x559d576c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 0x559d576ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d575a9510 con 0x559d576bc530 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d575a91b0 con 0x559d576bd2f0 2023-07-29T16:30:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d57490920 con 0x559d576bbbe0 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dd4422b80 10 client.0 _ll_register_callbacks cb 0x559d575b2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 0x559d576ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dbad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 0x559d576ccd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44356/0 (socket says 172.21.15.31:44356) 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dbad9d700 1 -- 172.21.15.31:0/3912368098 learned_addr learned my addr 172.21.15.31:0/3912368098 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bc530 0x559d576c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.882+0000 7f3dba59c700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dbad9d700 1 -- 172.21.15.31:0/3912368098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 msgr2=0x559d576c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dbad9d700 1 -- 172.21.15.31:0/3912368098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bc530 msgr2=0x559d576c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bc530 0x559d576c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dbad9d700 1 -- 172.21.15.31:0/3912368098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc10b260 con 0x559d576bd2f0 2023-07-29T16:30:09.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bc530 0x559d576c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 0x559d576ccd10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3db4095770 tx=0x7f3db4098b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f1fe2847021fb70 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3da1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3da1ffb700 1 -- 172.21.15.31:0/3912368098 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db40983f0 con 0x559d576bd2f0 2023-07-29T16:30:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d57637d10 con 0x559d576bd2f0 2023-07-29T16:30:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3da1ffb700 1 -- 172.21.15.31:0/3912368098 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3db4098d00 con 0x559d576bd2f0 2023-07-29T16:30:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3da1ffb700 1 -- 172.21.15.31:0/3912368098 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3db409cd30 con 0x559d576bd2f0 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dd4422b80 10 client.29846 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x559d57423c60 con 0x559d576bd2f0 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.883+0000 7f3da1ffb700 1 -- 172.21.15.31:0/3912368098 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559d57637d10 con 0x559d576bd2f0 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3da1ffb700 1 -- 172.21.15.31:0/3912368098 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3db40b11a0 con 0x559d576bd2f0 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dd4422b80 10 client.29846 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dd4422b80 10 client.29846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:09.891 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dd4422b80 20 client.29846 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d576d1550 0x559d576d3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dd4422b80 4 client.29846 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d5768ad70 con 0x559d576d1550 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.884+0000 7f3dba59c700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d576d1550 0x559d576d3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.885+0000 7f3dba59c700 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d576d1550 0x559d576d3960 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5577009799b5108e server_cookie=22afd9c103435bab in_seq=0 out_seq=0 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.885+0000 7f3da1ffb700 10 client.29846 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.885+0000 7f3da1ffb700 1 -- 172.21.15.31:0/3912368098 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559d5768ad70 con 0x559d576d1550 2023-07-29T16:30:09.892 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.885+0000 7f3da1ffb700 10 client.29846 handle_command_reply: tid=0 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 client.29846 shutdown 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 msgr2=0x559d576ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 0x559d576ccd10 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3db4095770 tx=0x7f3db4098b50 comp rx=0 tx=0).stop 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 shutdown_connections 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d576d1550 0x559d576d3960 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bd2f0 0x559d576ccd10 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/3912368098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bc530 0x559d576c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3da1ffb700 0 client.29846 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 >> 172.21.15.31:0/3912368098 conn(0x559d576b13e0 msgr2=0x559d576b6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 shutdown_connections 2023-07-29T16:30:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 1 -- 172.21.15.31:0/3912368098 wait complete. 2023-07-29T16:30:09.894 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.886+0000 7f3dd4422b80 20 client.29846 trim_cache size 0 max 16384 2023-07-29T16:30:09.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 Processor -- start 2023-07-29T16:30:09.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 -- start start 2023-07-29T16:30:09.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576b6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576c6910 con 0x559d576bd2f0 2023-07-29T16:30:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576ae6e0 con 0x559d576bc530 2023-07-29T16:30:09.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.900+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576c34f0 con 0x559d576bbbe0 2023-07-29T16:30:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576bd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59502/0 (socket says 172.21.15.31:59502) 2023-07-29T16:30:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 -- 172.21.15.31:0/1530428334 learned_addr learned my addr 172.21.15.31:0/1530428334 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 -- 172.21.15.31:0/1530428334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 msgr2=0x559d576cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 -- 172.21.15.31:0/1530428334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 msgr2=0x559d576b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 -- 172.21.15.31:0/1530428334 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d57461250 con 0x559d576bbbe0 2023-07-29T16:30:09.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:09.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.901+0000 7f3dba59c700 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576bd6c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005ac0 tx=0x7f3dac004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bcacab48e59b64e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3da97fa700 1 -- 172.21.15.31:0/1530428334 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac031070 con 0x559d576bbbe0 2023-07-29T16:30:09.907 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3da97fa700 1 -- 172.21.15.31:0/1530428334 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dac00bb60 con 0x559d576bbbe0 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3da97fa700 1 -- 172.21.15.31:0/1530428334 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac01d040 con 0x559d576bbbe0 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1530428334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 msgr2=0x559d576bd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576bd6c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005ac0 tx=0x7f3dac004500 comp rx=0 tx=0).stop 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1530428334 shutdown_connections 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576bd6c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.908 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1530428334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.902+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1530428334 >> 172.21.15.31:0/1530428334 conn(0x559d576b13e0 msgr2=0x559d576c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.903+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1530428334 shutdown_connections 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.903+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1530428334 wait complete. 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.903+0000 7f3dd4422b80 1 -- start start 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.903+0000 7f3dd4422b80 1 Processor -- start 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.909 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576cc710 con 0x559d576bd2f0 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576cc880 con 0x559d576bc530 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d576cca40 con 0x559d576bbbe0 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dd4422b80 10 client.0 _ll_register_callbacks cb 0x559d575b2620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3db9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dba59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.910 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3db9d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44382/0 (socket says 172.21.15.31:44382) 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3db9d9b700 1 -- 172.21.15.31:0/1507638749 learned_addr learned my addr 172.21.15.31:0/1507638749 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.904+0000 7f3dbad9d700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dba59c700 1 -- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 msgr2=0x559d576c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dba59c700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dba59c700 1 -- 172.21.15.31:0/1507638749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 msgr2=0x559d576cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dba59c700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dba59c700 1 -- 172.21.15.31:0/1507638749 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d5743ade0 con 0x559d576bbbe0 2023-07-29T16:30:09.911 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dba59c700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c8c70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x559d576c4c60 tx=0x7f3dac00bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea483f39b1407592 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3da1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3da1ffb700 1 -- 172.21.15.31:0/1507638749 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac031070 con 0x559d576bbbe0 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3da1ffb700 1 -- 172.21.15.31:0/1507638749 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3dac005050 con 0x559d576bbbe0 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d57605d40 con 0x559d576bbbe0 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3da1ffb700 1 -- 172.21.15.31:0/1507638749 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dac01d040 con 0x559d576bbbe0 2023-07-29T16:30:09.912 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dd4422b80 10 client.29845 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.905+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x559d574392b0 con 0x559d576bbbe0 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3da1ffb700 1 -- 172.21.15.31:0/1507638749 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559d57605d40 con 0x559d576bbbe0 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3da1ffb700 1 -- 172.21.15.31:0/1507638749 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f3dac03a060 con 0x559d576bbbe0 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3dd4422b80 10 client.29845 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3dd4422b80 10 client.29845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3dd4422b80 20 client.29845 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d57723d00 0x559d577260e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:09.913 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3dd4422b80 4 client.29845 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:30:09.914 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d57605a60 con 0x559d57723d00 2023-07-29T16:30:09.914 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1169380 8.82 MB/sec execute 594 sec latency 25.168 ms 2023-07-29T16:30:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.906+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d57723d00 0x559d577260e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3db9d9b700 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d57723d00 0x559d577260e0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ac19e91d057a2d server_cookie=d14196fd8d3cc4e9 in_seq=0 out_seq=0 2023-07-29T16:30:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3da1ffb700 10 client.29845 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:09.915 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3da1ffb700 1 -- 172.21.15.31:0/1507638749 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d57605a60 con 0x559d57723d00 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3da1ffb700 10 client.29845 handle_command_reply: tid=0 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 client.29845 shutdown 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 msgr2=0x559d576c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c8c70 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x559d576c4c60 tx=0x7f3dac00bc40 comp rx=0 tx=0).stop 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 shutdown_connections 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559d57723d00 0x559d577260e0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.916 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559d576bd2f0 0x559d576cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559d576bbbe0 0x559d576c8c70 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3da1ffb700 0 client.29845 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 --2- 172.21.15.31:0/1507638749 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559d576bc530 0x559d576c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.907+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 >> 172.21.15.31:0/1507638749 conn(0x559d576b13e0 msgr2=0x559d576c3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 shutdown_connections 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dd4422b80 1 -- 172.21.15.31:0/1507638749 wait complete. 2023-07-29T16:30:09.917 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dd4422b80 20 client.29845 trim_cache size 0 max 16384 2023-07-29T16:30:09.918 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:30:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3da40a0810 msgr2=0x7f3da40a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.918 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3da40a0810 0x7f3da40a2cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x559d576bb980 tx=0x7f3db0015040 comp rx=0 tx=0).stop 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc1081c0 msgr2=0x7f3dbc19f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.908+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc1081c0 0x7f3dbc19f840 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3db400efc0 tx=0x7f3db400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 shutdown_connections 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3da40a0810 0x7f3da40a2cc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc1081c0 0x7f3dbc19f840 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc073750 0x7f3dbc198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.919 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 --2- 172.21.15.31:0/2106332892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3dbc074d30 0x7f3dbc1987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 >> 172.21.15.31:0/2106332892 conn(0x7f3dbc0fb970 msgr2=0x7f3dbc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 shutdown_connections 2023-07-29T16:30:09.920 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:09.909+0000 7f3dc09ff700 1 -- 172.21.15.31:0/2106332892 wait complete. 2023-07-29T16:30:10.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:10 smithi031 ceph-mon[95504]: pgmap v812: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 2.8 MiB/s wr, 278 op/s 2023-07-29T16:30:10.525 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:10 smithi039 ceph-mon[128858]: pgmap v812: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 2.8 MiB/s wr, 278 op/s 2023-07-29T16:30:10.572 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:10 smithi156 ceph-mon[129245]: pgmap v812: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 2.8 MiB/s wr, 278 op/s 2023-07-29T16:30:10.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1171481 8.82 MB/sec execute 595 sec latency 31.193 ms 2023-07-29T16:30:10.928 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:30:10.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:30:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 -- 172.21.15.31:0/2726751026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c073750 msgr2=0x7f565c073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.154 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 --2- 172.21.15.31:0/2726751026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c073750 0x7f565c073bc0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5650009990 tx=0x7f5650009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 -- 172.21.15.31:0/2726751026 shutdown_connections 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 --2- 172.21.15.31:0/2726751026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f565c1081c0 0x7f565c10a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 --2- 172.21.15.31:0/2726751026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 0x7f565c073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 --2- 172.21.15.31:0/2726751026 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c073750 0x7f565c073bc0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 -- 172.21.15.31:0/2726751026 >> 172.21.15.31:0/2726751026 conn(0x7f565c0fb970 msgr2=0x7f565c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 -- 172.21.15.31:0/2726751026 shutdown_connections 2023-07-29T16:30:11.155 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.154+0000 7f5660bff700 1 -- 172.21.15.31:0/2726751026 wait complete. 2023-07-29T16:30:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 Processor -- start 2023-07-29T16:30:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 -- start start 2023-07-29T16:30:11.157 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f565c073750 0x7f565c198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 0x7f565c1987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 0x7f565c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c19fde0 con 0x7f565c074d30 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c19ff20 con 0x7f565c1081c0 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.155+0000 7f5660bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c1a00b0 con 0x7f565c073750 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 0x7f565c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.158 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 0x7f565c19f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44404/0 (socket says 172.21.15.31:44404) 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f565c073750 0x7f565c198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 -- 172.21.15.31:0/600332194 learned_addr learned my addr 172.21.15.31:0/600332194 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f5659d9b700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 0x7f565c1987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 -- 172.21.15.31:0/600332194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f565c073750 msgr2=0x7f565c198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f565c073750 0x7f565c198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 -- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 msgr2=0x7f565c1987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 0x7f565c1987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.159 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 -- 172.21.15.31:0/600332194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500095f0 con 0x7f565c1081c0 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f5659d9b700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 0x7f565c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f565ad9d700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 0x7f565c19f810 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f565400ef30 tx=0x7f565400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f564b7fe700 1 -- 172.21.15.31:0/600332194 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654018070 con 0x7f565c1081c0 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.156+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c1a0330 con 0x7f565c1081c0 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.157+0000 7f564b7fe700 1 -- 172.21.15.31:0/600332194 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f565400f040 con 0x7f565c1081c0 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.157+0000 7f564b7fe700 1 -- 172.21.15.31:0/600332194 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565400cdb0 con 0x7f565c1081c0 2023-07-29T16:30:11.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.157+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c1a0890 con 0x7f565c1081c0 2023-07-29T16:30:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.160+0000 7f564b7fe700 1 -- 172.21.15.31:0/600332194 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5654003a00 con 0x7f565c1081c0 2023-07-29T16:30:11.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.161+0000 7f56745b5b80 1 Processor -- start 2023-07-29T16:30:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.161+0000 7f56745b5b80 1 -- start start 2023-07-29T16:30:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb65c60 0x55f99cb66070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 0x55f99cb66de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb677c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f56745b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cabda90 con 0x55f99cb65c60 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f56745b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cabb450 con 0x55f99cb665b0 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f56745b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cab6d40 con 0x55f99cb67370 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f565a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 0x55f99cb66de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb65c60 0x55f99cb66070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f565a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 0x55f99cb66de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44416/0 (socket says 172.21.15.31:44416) 2023-07-29T16:30:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f565a59c700 1 -- 172.21.15.31:0/3874986249 learned_addr learned my addr 172.21.15.31:0/3874986249 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f5659d9b700 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.162+0000 7f564b7fe700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56440a0790 0x7f56440a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f564b7fe700 1 -- 172.21.15.31:0/600332194 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f56540cb6b0 con 0x7f565c1081c0 2023-07-29T16:30:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565ad9d700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56440a0790 0x7f56440a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565a59c700 1 -- 172.21.15.31:0/3874986249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 msgr2=0x55f99cb677c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565a59c700 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb677c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565a59c700 1 -- 172.21.15.31:0/3874986249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb65c60 msgr2=0x55f99cb66070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565a59c700 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb65c60 0x55f99cb66070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565a59c700 1 -- 172.21.15.31:0/3874986249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f99c8e4e60 con 0x55f99cb665b0 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f5659d9b700 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb65c60 0x55f99cb66070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.163+0000 7f565a59c700 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 0x55f99cb66de0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f564c009d80 tx=0x7f564c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c645cf8d8966c345 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.164+0000 7f565ad9d700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56440a0790 0x7f56440a2c40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5654009c10 tx=0x7f565401c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.164+0000 7f5648ff9700 1 -- 172.21.15.31:0/3874986249 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c004230 con 0x55f99cb665b0 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.164+0000 7f5648ff9700 1 -- 172.21.15.31:0/3874986249 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f564c004390 con 0x55f99cb665b0 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.164+0000 7f5648ff9700 1 -- 172.21.15.31:0/3874986249 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c0198c0 con 0x55f99cb665b0 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 -- 172.21.15.31:0/3874986249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 msgr2=0x55f99cb66de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 0x55f99cb66de0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f564c009d80 tx=0x7f564c011720 comp rx=0 tx=0).stop 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 -- 172.21.15.31:0/3874986249 shutdown_connections 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb677c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb65c60 0x55f99cb66070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3874986249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb665b0 0x55f99cb66de0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 -- 172.21.15.31:0/3874986249 >> 172.21.15.31:0/3874986249 conn(0x55f99cb5b460 msgr2=0x55f99cb60a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 -- 172.21.15.31:0/3874986249 shutdown_connections 2023-07-29T16:30:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 -- 172.21.15.31:0/3874986249 wait complete. 2023-07-29T16:30:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.165+0000 7f56745b5b80 1 -- start start 2023-07-29T16:30:11.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 Processor -- start 2023-07-29T16:30:11.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb6f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb6fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb76e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99ca53590 con 0x55f99cb665b0 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99ca53230 con 0x55f99cb65c60 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99c93a9a0 con 0x55f99cb67370 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f56745b5b80 10 client.0 _ll_register_callbacks cb 0x55f99ca5c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb6f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb6f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44426/0 (socket says 172.21.15.31:44426) 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.166+0000 7f5659d9b700 1 -- 172.21.15.31:0/3957572750 learned_addr learned my addr 172.21.15.31:0/3957572750 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb76e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f565a59c700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb6fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 -- 172.21.15.31:0/3957572750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 msgr2=0x55f99cb76e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb76e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 -- 172.21.15.31:0/3957572750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 msgr2=0x55f99cb6fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb6fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 -- 172.21.15.31:0/3957572750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c10b260 con 0x55f99cb65c60 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f565a59c700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb6fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:11.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5659d9b700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb6f8c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f56500098e0 tx=0x7f5650005ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=812a35636f8767a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5641ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5641ffb700 1 -- 172.21.15.31:0/3957572750 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565001d070 con 0x55f99cb65c60 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f99cae1d90 con 0x55f99cb65c60 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.167+0000 7f5641ffb700 1 -- 172.21.15.31:0/3957572750 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f56500049a0 con 0x55f99cb65c60 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f5641ffb700 1 -- 172.21.15.31:0/3957572750 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5650004b60 con 0x55f99cb65c60 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 10 client.29864 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7f564c0099e0 con 0x55f99cb65c60 2023-07-29T16:30:11.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f5641ffb700 1 -- 172.21.15.31:0/3957572750 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f99cae1d90 con 0x55f99cb65c60 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f5641ffb700 1 -- 172.21.15.31:0/3957572750 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f565000f460 con 0x55f99cb65c60 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 10 client.29864 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 10 client.29864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 20 client.29864 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cb7b590 0x55f99cb7d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 4 client.29864 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:11.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.168+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f99cb34df0 con 0x55f99cb7b590 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.169+0000 7f565a59c700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cb7b590 0x55f99cb7d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.169+0000 7f565a59c700 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cb7b590 0x55f99cb7d9a0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a581bb213a44b17f server_cookie=b2f5a6b4ef539414 in_seq=0 out_seq=0 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.169+0000 7f5641ffb700 10 client.29864 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f5641ffb700 1 -- 172.21.15.31:0/3957572750 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f99cb34df0 con 0x55f99cb7b590 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f5641ffb700 10 client.29864 handle_command_reply: tid=0 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 client.29864 shutdown 2023-07-29T16:30:11.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 msgr2=0x55f99cb6f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb6f8c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f56500098e0 tx=0x7f5650005ab0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 shutdown_connections 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cb7b590 0x55f99cb7d9a0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb67370 0x55f99cb76e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb6f8c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 --2- 172.21.15.31:0/3957572750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb6fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f5641ffb700 0 client.29864 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.170+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 >> 172.21.15.31:0/3957572750 conn(0x55f99cb5b460 msgr2=0x55f99cb600e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.171+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 shutdown_connections 2023-07-29T16:30:11.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.171+0000 7f56745b5b80 1 -- 172.21.15.31:0/3957572750 wait complete. 2023-07-29T16:30:11.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.172+0000 7f56745b5b80 20 client.29864 trim_cache size 0 max 16384 2023-07-29T16:30:11.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.190+0000 7f56745b5b80 1 Processor -- start 2023-07-29T16:30:11.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 -- start start 2023-07-29T16:30:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb665b0 0x55f99cb76e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb67370 0x55f99cb6a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cb5af20 con 0x55f99cb67370 2023-07-29T16:30:11.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cb6ed70 con 0x55f99cb65c60 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f56745b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cb5d4e0 con 0x55f99cb665b0 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f565a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb665b0 0x55f99cb76e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb600e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44430/0 (socket says 172.21.15.31:44430) 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f5659d9b700 1 -- 172.21.15.31:0/1316500950 learned_addr learned my addr 172.21.15.31:0/1316500950 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.191+0000 7f5659d9b700 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb67370 0x55f99cb6a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f565a59c700 1 -- 172.21.15.31:0/1316500950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 msgr2=0x55f99cb600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f565a59c700 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f565a59c700 1 -- 172.21.15.31:0/1316500950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb67370 msgr2=0x55f99cb6a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f565a59c700 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb67370 0x55f99cb6a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f565a59c700 1 -- 172.21.15.31:0/1316500950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f99c8cdce0 con 0x55f99cb665b0 2023-07-29T16:30:11.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f565a59c700 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb665b0 0x55f99cb76e00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f564c019490 tx=0x7f564c005fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19d2f4cfea419044 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f56497fa700 1 -- 172.21.15.31:0/1316500950 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c02a070 con 0x55f99cb665b0 2023-07-29T16:30:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f56497fa700 1 -- 172.21.15.31:0/1316500950 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f564c02e070 con 0x55f99cb665b0 2023-07-29T16:30:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.192+0000 7f56497fa700 1 -- 172.21.15.31:0/1316500950 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564c0052a0 con 0x55f99cb665b0 2023-07-29T16:30:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 -- 172.21.15.31:0/1316500950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb665b0 msgr2=0x55f99cb76e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb665b0 0x55f99cb76e00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f564c019490 tx=0x7f564c005fa0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 -- 172.21.15.31:0/1316500950 shutdown_connections 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb67370 0x55f99cb6a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb65c60 0x55f99cb600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 --2- 172.21.15.31:0/1316500950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb665b0 0x55f99cb76e00 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 -- 172.21.15.31:0/1316500950 >> 172.21.15.31:0/1316500950 conn(0x55f99cb5b460 msgr2=0x55f99cb6d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 -- 172.21.15.31:0/1316500950 shutdown_connections 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.193+0000 7f56745b5b80 1 -- 172.21.15.31:0/1316500950 wait complete. 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 -- start start 2023-07-29T16:30:11.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 Processor -- start 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 0x55f99cb760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb765f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb67370 0x55f99cbd5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cbd6120 con 0x55f99cb665b0 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cbd6260 con 0x55f99cb67370 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.194+0000 7f56745b5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f99cbd63f0 con 0x55f99cb65c60 2023-07-29T16:30:11.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f56745b5b80 10 client.0 _ll_register_callbacks cb 0x55f99ca5c6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 0x55f99cb760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 0x55f99cb760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59560/0 (socket says 172.21.15.31:59560) 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 -- 172.21.15.31:0/2112279281 learned_addr learned my addr 172.21.15.31:0/2112279281 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb67370 0x55f99cbd5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f565a59c700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb765f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 -- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb67370 msgr2=0x55f99cbd5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb67370 0x55f99cbd5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 -- 172.21.15.31:0/2112279281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 msgr2=0x55f99cb765f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb765f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 -- 172.21.15.31:0/2112279281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f99c8e4e60 con 0x55f99cb65c60 2023-07-29T16:30:11.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5659d9b700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 0x55f99cb760b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f565000f820 tx=0x7f565000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e5e626c13a866db server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.195+0000 7f5641ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:11.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f5641ffb700 1 -- 172.21.15.31:0/2112279281 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565001d040 con 0x55f99cb65c60 2023-07-29T16:30:11.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f5641ffb700 1 -- 172.21.15.31:0/2112279281 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f565003d030 con 0x55f99cb65c60 2023-07-29T16:30:11.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f564c004390 con 0x55f99cb65c60 2023-07-29T16:30:11.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f5641ffb700 1 -- 172.21.15.31:0/2112279281 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5650028cd0 con 0x55f99cb65c60 2023-07-29T16:30:11.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f56745b5b80 10 client.29863 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:11.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55f99c90b2d0 con 0x55f99cb65c60 2023-07-29T16:30:11.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f5641ffb700 1 -- 172.21.15.31:0/2112279281 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f564c004390 con 0x55f99cb65c60 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f5641ffb700 1 -- 172.21.15.31:0/2112279281 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5650004d00 con 0x55f99cb65c60 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.196+0000 7f56745b5b80 10 client.29863 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f56745b5b80 10 client.29863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f56745b5b80 20 client.29863 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f56745b5b80 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cbd7900 0x55f99cbd9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f56745b5b80 4 client.29863 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:30:11.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f99caafdc0 con 0x55f99cbd7900 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f565a59c700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cbd7900 0x55f99cbd9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f565a59c700 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cbd7900 0x55f99cbd9ce0 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d3fdfbfe411197 server_cookie=d3b6a8b42b2449af in_seq=0 out_seq=0 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.197+0000 7f5641ffb700 10 client.29863 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.198+0000 7f5641ffb700 1 -- 172.21.15.31:0/2112279281 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f99caafdc0 con 0x55f99cbd7900 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.198+0000 7f5641ffb700 10 client.29863 handle_command_reply: tid=0 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.198+0000 7f56745b5b80 1 client.29863 shutdown 2023-07-29T16:30:11.204 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 msgr2=0x55f99cb760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 0x55f99cb760b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f565000f820 tx=0x7f565000b8f0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 shutdown_connections 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f99cbd7900 0x55f99cbd9ce0 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f99cb67370 0x55f99cbd5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f99cb65c60 0x55f99cb760b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f5641ffb700 0 client.29863 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.205 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 --2- 172.21.15.31:0/2112279281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f99cb665b0 0x55f99cb765f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 >> 172.21.15.31:0/2112279281 conn(0x55f99cb5b460 msgr2=0x55f99cb6d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 shutdown_connections 2023-07-29T16:30:11.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 1 -- 172.21.15.31:0/2112279281 wait complete. 2023-07-29T16:30:11.206 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.199+0000 7f56745b5b80 20 client.29863 trim_cache size 0 max 16384 2023-07-29T16:30:11.206 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:30:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:30:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "b0acc086-6e57-4627-aeff-19214b861863", 2023-07-29T16:30:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:30:11.207 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:30:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56440a0790 msgr2=0x7f56440a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56440a0790 0x7f56440a2c40 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5654009c10 tx=0x7f565401c000 comp rx=0 tx=0).stop 2023-07-29T16:30:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 msgr2=0x7f565c19f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 0x7f565c19f810 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f565400ef30 tx=0x7f565400c3a0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 shutdown_connections 2023-07-29T16:30:11.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f56440a0790 0x7f56440a2c40 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f565c1081c0 0x7f565c19f810 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f565c073750 0x7f565c198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 --2- 172.21.15.31:0/600332194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f565c074d30 0x7f565c1987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 >> 172.21.15.31:0/600332194 conn(0x7f565c0fb970 msgr2=0x7f565c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 shutdown_connections 2023-07-29T16:30:11.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.200+0000 7f5660bff700 1 -- 172.21.15.31:0/600332194 wait complete. 2023-07-29T16:30:11.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:30:11.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 -- 172.21.15.31:0/3947196565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590100350 msgr2=0x7f4590100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 --2- 172.21.15.31:0/3947196565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590100350 0x7f4590100760 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4580009990 tx=0x7f4580009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 -- 172.21.15.31:0/3947196565 shutdown_connections 2023-07-29T16:30:11.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 --2- 172.21.15.31:0/3947196565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590101ed0 0x7f4590108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 --2- 172.21.15.31:0/3947196565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590100350 0x7f4590100760 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 --2- 172.21.15.31:0/3947196565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4590101540 0x7f4590101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 -- 172.21.15.31:0/3947196565 >> 172.21.15.31:0/3947196565 conn(0x7f45900fb970 msgr2=0x7f45900fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 -- 172.21.15.31:0/3947196565 shutdown_connections 2023-07-29T16:30:11.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.439+0000 7f4594fff700 1 -- 172.21.15.31:0/3947196565 wait complete. 2023-07-29T16:30:11.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.440+0000 7f4594fff700 1 Processor -- start 2023-07-29T16:30:11.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 -- start start 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 0x7f4590198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4590101540 0x7f45901987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590101ed0 0x7f459019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459019fe10 con 0x7f4590101540 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459019ff50 con 0x7f4590100350 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f4594fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45901a00e0 con 0x7f4590101ed0 2023-07-29T16:30:11.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f458e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 0x7f4590198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4590101540 0x7f45901987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f458e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 0x7f4590198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44454/0 (socket says 172.21.15.31:44454) 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f458e59c700 1 -- 172.21.15.31:0/1616502791 learned_addr learned my addr 172.21.15.31:0/1616502791 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.441+0000 7f458ed9d700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590101ed0 0x7f459019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458e59c700 1 -- 172.21.15.31:0/1616502791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590101ed0 msgr2=0x7f459019f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458e59c700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590101ed0 0x7f459019f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458e59c700 1 -- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4590101540 msgr2=0x7f45901987d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458e59c700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4590101540 0x7f45901987d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458e59c700 1 -- 172.21.15.31:0/1616502791 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588009c40 con 0x7f4590100350 2023-07-29T16:30:11.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458ed9d700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590101ed0 0x7f459019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:11.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f458e59c700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 0x7f4590198290 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4580005720 tx=0x7f4580009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f457f7fe700 1 -- 172.21.15.31:0/1616502791 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458001d070 con 0x7f4590100350 2023-07-29T16:30:11.448 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f457f7fe700 1 -- 172.21.15.31:0/1616502791 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4580021410 con 0x7f4590100350 2023-07-29T16:30:11.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45800095f0 con 0x7f4590100350 2023-07-29T16:30:11.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.442+0000 7f457f7fe700 1 -- 172.21.15.31:0/1616502791 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580017650 con 0x7f4590100350 2023-07-29T16:30:11.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.443+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45901a0650 con 0x7f4590100350 2023-07-29T16:30:11.451 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.446+0000 7f457f7fe700 1 -- 172.21.15.31:0/1616502791 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4580021580 con 0x7f4590100350 2023-07-29T16:30:11.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.447+0000 7f45a89e4b80 1 Processor -- start 2023-07-29T16:30:11.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 -- start start 2023-07-29T16:30:11.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f457f7fe700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f45780a0790 0x7f45780a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895dbe0 0x55fa1895dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa1895ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.452 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895f2f0 0x55fa1895f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa188b5a10 con 0x55fa1895dbe0 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa188b33d0 con 0x55fa1895e530 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa188aecc0 con 0x55fa1895f2f0 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f457f7fe700 1 -- 172.21.15.31:0/1616502791 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f45800ccd10 con 0x7f4590100350 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f458ed9d700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f45780a0790 0x7f45780a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa1895ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f458ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895f2f0 0x55fa1895f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa1895ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44458/0 (socket says 172.21.15.31:44458) 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f458dd9b700 1 -- 172.21.15.31:0/2137218724 learned_addr learned my addr 172.21.15.31:0/2137218724 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.448+0000 7f458e59c700 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895dbe0 0x55fa1895dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458dd9b700 1 -- 172.21.15.31:0/2137218724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895f2f0 msgr2=0x55fa1895f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458dd9b700 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895f2f0 0x55fa1895f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458dd9b700 1 -- 172.21.15.31:0/2137218724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895dbe0 msgr2=0x55fa1895dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.454 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458dd9b700 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895dbe0 0x55fa1895dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458dd9b700 1 -- 172.21.15.31:0/2137218724 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa186dcde0 con 0x55fa1895e530 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458e59c700 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895dbe0 0x55fa1895dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458ed9d700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f45780a0790 0x7f45780a2c40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4588009550 tx=0x7f45880094e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f458dd9b700 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa1895ed60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4584009d80 tx=0x7f4584011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8957145342ccda7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f457cff9700 1 -- 172.21.15.31:0/2137218724 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584004230 con 0x55fa1895e530 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f457cff9700 1 -- 172.21.15.31:0/2137218724 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4584004390 con 0x55fa1895e530 2023-07-29T16:30:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.449+0000 7f457cff9700 1 -- 172.21.15.31:0/2137218724 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45840197e0 con 0x55fa1895e530 2023-07-29T16:30:11.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2137218724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 msgr2=0x55fa1895ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa1895ed60 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4584009d80 tx=0x7f4584011720 comp rx=0 tx=0).stop 2023-07-29T16:30:11.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2137218724 shutdown_connections 2023-07-29T16:30:11.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895f2f0 0x55fa1895f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.456 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895dbe0 0x55fa1895dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2137218724 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa1895ed60 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2137218724 >> 172.21.15.31:0/2137218724 conn(0x55fa189533e0 msgr2=0x55fa18958990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2137218724 shutdown_connections 2023-07-29T16:30:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.450+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2137218724 wait complete. 2023-07-29T16:30:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.451+0000 7f45a89e4b80 1 -- start start 2023-07-29T16:30:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.451+0000 7f45a89e4b80 1 Processor -- start 2023-07-29T16:30:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa189677d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18967d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa1884b510 con 0x55fa1895f2f0 2023-07-29T16:30:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa1884b1b0 con 0x55fa1895e530 2023-07-29T16:30:11.461 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa18732920 con 0x55fa1895dbe0 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f45a89e4b80 10 client.0 _ll_register_callbacks cb 0x55fa18854620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa189677d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18967d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa189677d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59588/0 (socket says 172.21.15.31:59588) 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 -- 172.21.15.31:0/1292078008 learned_addr learned my addr 172.21.15.31:0/1292078008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18967d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44462/0 (socket says 172.21.15.31:44462) 2023-07-29T16:30:11.462 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458ed9d700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 -- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 msgr2=0x55fa18967d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18967d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 -- 172.21.15.31:0/1292078008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 msgr2=0x55fa1896ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458e59c700 1 -- 172.21.15.31:0/1292078008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45840099e0 con 0x55fa1895dbe0 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458dd9b700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18967d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:11.463 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.452+0000 7f458ed9d700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f458e59c700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa189677d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4580000c00 tx=0x7f4580099fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d52ec3558d7ca260 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f4575ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f4575ffb700 1 -- 172.21.15.31:0/1292078008 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580021a80 con 0x55fa1895dbe0 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f4575ffb700 1 -- 172.21.15.31:0/1292078008 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f458009a4a0 con 0x55fa1895dbe0 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa188d9d10 con 0x55fa1895dbe0 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f4575ffb700 1 -- 172.21.15.31:0/1292078008 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45800010b0 con 0x55fa1895dbe0 2023-07-29T16:30:11.464 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f45a89e4b80 10 client.29875 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55fa186c5c60 con 0x55fa1895dbe0 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f4575ffb700 1 -- 172.21.15.31:0/1292078008 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fa188d9d10 con 0x55fa1895dbe0 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f4575ffb700 1 -- 172.21.15.31:0/1292078008 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f458009e0b0 con 0x55fa1895dbe0 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f45a89e4b80 10 client.29875 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f45a89e4b80 10 client.29875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.453+0000 7f45a89e4b80 20 client.29875 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:11.465 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.454+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa18973190 0x55fa18975570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.454+0000 7f45a89e4b80 4 client.29875 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.454+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa1892cd70 con 0x55fa18973190 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.454+0000 7f458dd9b700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa18973190 0x55fa18975570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.454+0000 7f458dd9b700 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa18973190 0x55fa18975570 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7e618826710a70c server_cookie=cb691c6015eab4e3 in_seq=0 out_seq=0 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.454+0000 7f4575ffb700 10 client.29875 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f4575ffb700 1 -- 172.21.15.31:0/1292078008 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55fa1892cd70 con 0x55fa18973190 2023-07-29T16:30:11.466 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f4575ffb700 10 client.29875 handle_command_reply: tid=0 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 client.29875 shutdown 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 msgr2=0x55fa189677d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa189677d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f4580000c00 tx=0x7f4580099fe0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 shutdown_connections 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa18973190 0x55fa18975570 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.467 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa189677d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f4575ffb700 0 client.29875 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/1292078008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18967d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.455+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 >> 172.21.15.31:0/1292078008 conn(0x55fa189533e0 msgr2=0x55fa18958060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.456+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 shutdown_connections 2023-07-29T16:30:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.456+0000 7f45a89e4b80 1 -- 172.21.15.31:0/1292078008 wait complete. 2023-07-29T16:30:11.468 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.456+0000 7f45a89e4b80 20 client.29875 trim_cache size 0 max 16384 2023-07-29T16:30:11.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 Processor -- start 2023-07-29T16:30:11.474 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 -- start start 2023-07-29T16:30:11.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa1896cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18959690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa18958060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa189673a0 con 0x55fa1895f2f0 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa18972e00 con 0x55fa1895e530 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa1895ebc0 con 0x55fa1895dbe0 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f458e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa1896cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f458e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa1896cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59594/0 (socket says 172.21.15.31:59594) 2023-07-29T16:30:11.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f458e59c700 1 -- 172.21.15.31:0/433527318 learned_addr learned my addr 172.21.15.31:0/433527318 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.474+0000 7f458dd9b700 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18959690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458e59c700 1 -- 172.21.15.31:0/433527318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 msgr2=0x55fa18959690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458e59c700 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18959690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458e59c700 1 -- 172.21.15.31:0/433527318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 msgr2=0x55fa18958060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458ed9d700 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa18958060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458e59c700 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa18958060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458e59c700 1 -- 172.21.15.31:0/433527318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa18703250 con 0x55fa1895dbe0 2023-07-29T16:30:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458ed9d700 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa18958060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f458e59c700 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa1896cd00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4580096f10 tx=0x7f4580098a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47b054ff7478a07d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f457d7fa700 1 -- 172.21.15.31:0/433527318 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45800ca070 con 0x55fa1895dbe0 2023-07-29T16:30:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f457d7fa700 1 -- 172.21.15.31:0/433527318 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45800b4e90 con 0x55fa1895dbe0 2023-07-29T16:30:11.478 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.475+0000 7f457d7fa700 1 -- 172.21.15.31:0/433527318 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45800a2860 con 0x55fa1895dbe0 2023-07-29T16:30:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 -- 172.21.15.31:0/433527318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 msgr2=0x55fa1896cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.481 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa1896cd00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f4580096f10 tx=0x7f4580098a50 comp rx=0 tx=0).stop 2023-07-29T16:30:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 -- 172.21.15.31:0/433527318 shutdown_connections 2023-07-29T16:30:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa18958060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895dbe0 0x55fa1896cd00 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/433527318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895e530 0x55fa18959690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 -- 172.21.15.31:0/433527318 >> 172.21.15.31:0/433527318 conn(0x55fa189533e0 msgr2=0x55fa18958900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.482 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 -- 172.21.15.31:0/433527318 shutdown_connections 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.476+0000 7f45a89e4b80 1 -- 172.21.15.31:0/433527318 wait complete. 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 -- start start 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 Processor -- start 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895dbe0 0x55fa1896aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 0x55fa1896b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.483 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa1896c630 con 0x55fa1895f2f0 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa1896c7a0 con 0x55fa1895dbe0 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa1896c960 con 0x55fa1895e530 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.477+0000 7f45a89e4b80 10 client.0 _ll_register_callbacks cb 0x55fa18854620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 0x55fa1896b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895dbe0 0x55fa1896aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 0x55fa1896b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59602/0 (socket says 172.21.15.31:59602) 2023-07-29T16:30:11.484 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 -- 172.21.15.31:0/2189814895 learned_addr learned my addr 172.21.15.31:0/2189814895 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458ed9d700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 -- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895dbe0 msgr2=0x55fa1896aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895dbe0 0x55fa1896aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 -- 172.21.15.31:0/2189814895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 msgr2=0x55fa1896c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 -- 172.21.15.31:0/2189814895 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa186c5c60 con 0x55fa1895e530 2023-07-29T16:30:11.485 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458ed9d700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458e59c700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895dbe0 0x55fa1896aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f458dd9b700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 0x55fa1896b0e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f458401b040 tx=0x7f4584011ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d35097e15307b4ee server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.478+0000 7f4575ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f4575ffb700 1 -- 172.21.15.31:0/2189814895 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584003880 con 0x55fa1895e530 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4584004390 con 0x55fa1895e530 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f4575ffb700 1 -- 172.21.15.31:0/2189814895 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f458402e070 con 0x55fa1895e530 2023-07-29T16:30:11.486 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f4575ffb700 1 -- 172.21.15.31:0/2189814895 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f458402a480 con 0x55fa1895e530 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 10 client.29881 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55fa186db2b0 con 0x55fa1895e530 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f4575ffb700 1 -- 172.21.15.31:0/2189814895 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4584004390 con 0x55fa1895e530 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f4575ffb700 1 -- 172.21.15.31:0/2189814895 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f458403a070 con 0x55fa1895e530 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 10 client.29881 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 10 client.29881 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:11.487 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 20 client.29881 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa189c5cc0 0x55fa189c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 4 client.29881 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.479+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fa188a7d40 con 0x55fa189c5cc0 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f458e59c700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa189c5cc0 0x55fa189c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f458e59c700 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa189c5cc0 0x55fa189c80d0 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21e1198f7b267808 server_cookie=9e60773724e64a8b in_seq=0 out_seq=0 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f4575ffb700 10 client.29881 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.488 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f4575ffb700 1 -- 172.21.15.31:0/2189814895 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fa188a7d40 con 0x55fa189c5cc0 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f4575ffb700 10 client.29881 handle_command_reply: tid=0 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f45a89e4b80 1 client.29881 shutdown 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 msgr2=0x55fa1896b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.480+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 0x55fa1896b0e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f458401b040 tx=0x7f4584011ba0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 shutdown_connections 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fa189c5cc0 0x55fa189c80d0 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.489 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fa1895f2f0 0x55fa1896c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fa1895dbe0 0x55fa1896aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 --2- 172.21.15.31:0/2189814895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fa1895e530 0x55fa1896b0e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f4575ffb700 0 client.29881 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 >> 172.21.15.31:0/2189814895 conn(0x55fa189533e0 msgr2=0x55fa189656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 shutdown_connections 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 1 -- 172.21.15.31:0/2189814895 wait complete. 2023-07-29T16:30:11.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f45a89e4b80 20 client.29881 trim_cache size 0 max 16384 2023-07-29T16:30:11.491 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:30:11.491 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:30:11.491 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:30:11.491 INFO:teuthology.orchestra.run.smithi031.stdout: "b0acc086-6e57-4627-aeff-19214b861863": { 2023-07-29T16:30:11.492 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:30:11.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "b0acc086-6e57-4627-aeff-19214b861863", 2023-07-29T16:30:11.492 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:30:11.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:30:11.492 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:30:11.492 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:30:11.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f45780a0790 msgr2=0x7f45780a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f4594fff700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f45780a0790 0x7f45780a2c40 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4588009550 tx=0x7f45880094e0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 msgr2=0x7f4590198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.481+0000 7f4594fff700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 0x7f4590198290 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f4580005720 tx=0x7f4580009580 comp rx=0 tx=0).stop 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 shutdown_connections 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f45780a0790 0x7f45780a2c40 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4590101ed0 0x7f459019f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4590100350 0x7f4590198290 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 --2- 172.21.15.31:0/1616502791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4590101540 0x7f45901987d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:11.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 >> 172.21.15.31:0/1616502791 conn(0x7f45900fb970 msgr2=0x7f4590106e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 shutdown_connections 2023-07-29T16:30:11.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:11.482+0000 7f4594fff700 1 -- 172.21.15.31:0/1616502791 wait complete. 2023-07-29T16:30:11.500 INFO:tasks.cephfs.filesystem:scrub status for tag:b0acc086-6e57-4627-aeff-19214b861863 - {'path': '/', 'tag': 'b0acc086-6e57-4627-aeff-19214b861863', 'options': 'recursive,force'} 2023-07-29T16:30:11.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1172523 8.82 MB/sec execute 596 sec latency 25.136 ms 2023-07-29T16:30:11.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:11 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:11.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:11 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:11.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:11 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:11.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:11 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:11.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:11 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:11.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:11 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:11 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:12.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:11 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:11 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:12.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:11 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:12.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:11 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:11 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:11 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:12.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:11 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:11 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:12.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:11 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:12.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:11 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:11 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.602 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:12 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:30:12.602 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:12 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.602 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:12 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:12.603 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:12 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.603 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:12 smithi039 ceph-mon[128858]: pgmap v813: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 2.0 MiB/s wr, 223 op/s 2023-07-29T16:30:12.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1173983 8.82 MB/sec execute 597 sec latency 46.762 ms 2023-07-29T16:30:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:12 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:30:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:12 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:12 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:12 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:12.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:12 smithi031 ceph-mon[95504]: pgmap v813: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 2.0 MiB/s wr, 223 op/s 2023-07-29T16:30:13.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:12 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:30:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:12 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:12 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:12 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:12 smithi156 ceph-mon[129245]: pgmap v813: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 2.0 MiB/s wr, 223 op/s 2023-07-29T16:30:13.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1175800 8.82 MB/sec execute 598 sec latency 38.546 ms 2023-07-29T16:30:14.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 -- 172.21.15.31:0/300259946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 msgr2=0x7fbe44102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/300259946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44102d40 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbe400098f0 tx=0x7fbe40009c00 comp rx=0 tx=0).stop 2023-07-29T16:30:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 -- 172.21.15.31:0/300259946 shutdown_connections 2023-07-29T16:30:14.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/300259946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44105d50 0x7fbe44108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/300259946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44102d40 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/300259946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44103280 0x7fbe441056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 -- 172.21.15.31:0/300259946 >> 172.21.15.31:0/300259946 conn(0x7fbe440fa690 msgr2=0x7fbe440fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 -- 172.21.15.31:0/300259946 shutdown_connections 2023-07-29T16:30:14.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.322+0000 7fbe4b59d700 1 -- 172.21.15.31:0/300259946 wait complete. 2023-07-29T16:30:14.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 Processor -- start 2023-07-29T16:30:14.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 -- start start 2023-07-29T16:30:14.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44103280 0x7fbe44196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 0x7fbe4419d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4419dad0 con 0x7fbe44068e40 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4419dc10 con 0x7fbe44103280 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.323+0000 7fbe4b59d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe4419dda0 con 0x7fbe44105d50 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe49d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44103280 0x7fbe44196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 0x7fbe4419d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 0x7fbe4419d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59628/0 (socket says 172.21.15.31:59628) 2023-07-29T16:30:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4a59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe49d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44103280 0x7fbe44196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44498/0 (socket says 172.21.15.31:44498) 2023-07-29T16:30:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2731837410 learned_addr learned my addr 172.21.15.31:0/2731837410 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2731837410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44103280 msgr2=0x7fbe44196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44103280 0x7fbe44196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 msgr2=0x7fbe44195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2731837410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3c009d00 con 0x7fbe44105d50 2023-07-29T16:30:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.324+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 0x7fbe4419d500 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00e3c0 tx=0x7fbe3c00e6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:14.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.325+0000 7fbe3b7fe700 1 -- 172.21.15.31:0/2731837410 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0090d0 con 0x7fbe44105d50 2023-07-29T16:30:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.325+0000 7fbe3b7fe700 1 -- 172.21.15.31:0/2731837410 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe3c00f040 con 0x7fbe44105d50 2023-07-29T16:30:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.325+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe40009520 con 0x7fbe44105d50 2023-07-29T16:30:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.325+0000 7fbe3b7fe700 1 -- 172.21.15.31:0/2731837410 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c007c20 con 0x7fbe44105d50 2023-07-29T16:30:14.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.325+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe440feeb0 con 0x7fbe44105d50 2023-07-29T16:30:14.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.328+0000 7fbe3b7fe700 1 -- 172.21.15.31:0/2731837410 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbe3c003a00 con 0x7fbe44105d50 2023-07-29T16:30:14.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.330+0000 7fbe60057b80 1 Processor -- start 2023-07-29T16:30:14.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.330+0000 7fbe60057b80 1 -- start start 2023-07-29T16:30:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f6bc6b0 0x56529f860c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe3b7fe700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbe300a0790 0x7fbe300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f861150 0x56529f8619c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f8623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe60057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f7b89b0 con 0x56529f6bc6b0 2023-07-29T16:30:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe60057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f7b6370 con 0x56529f861150 2023-07-29T16:30:14.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe60057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f7b1c60 con 0x56529f861f00 2023-07-29T16:30:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe3b7fe700 1 -- 172.21.15.31:0/2731837410 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbe3c0cb910 con 0x7fbe44105d50 2023-07-29T16:30:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbe300a0790 0x7fbe300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f8623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f8623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59630/0 (socket says 172.21.15.31:59630) 2023-07-29T16:30:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/1841082190 learned_addr learned my addr 172.21.15.31:0/1841082190 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.331+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f861150 0x56529f8619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f6bc6b0 0x56529f860c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/1841082190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f861150 msgr2=0x56529f8619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f861150 0x56529f8619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/1841082190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f6bc6b0 msgr2=0x56529f860c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f6bc6b0 0x56529f860c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/1841082190 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529f5dfd80 con 0x56529f861f00 2023-07-29T16:30:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f8623a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c095540 tx=0x7fbe3c098f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4514bb4813d54845 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:14.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe38ff9700 1 -- 172.21.15.31:0/1841082190 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c09d070 con 0x56529f861f00 2023-07-29T16:30:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe38ff9700 1 -- 172.21.15.31:0/1841082190 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe3c0a1400 con 0x56529f861f00 2023-07-29T16:30:14.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.332+0000 7fbe38ff9700 1 -- 172.21.15.31:0/1841082190 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0092c0 con 0x56529f861f00 2023-07-29T16:30:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 -- 172.21.15.31:0/1841082190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 msgr2=0x56529f8623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbe300a0790 0x7fbe300a2c40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009fd0 tx=0x7fbe40019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f8623a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c095540 tx=0x7fbe3c098f40 comp rx=0 tx=0).stop 2023-07-29T16:30:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 -- 172.21.15.31:0/1841082190 shutdown_connections 2023-07-29T16:30:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f8623a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f6bc6b0 0x56529f860c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1841082190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f861150 0x56529f8619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 -- 172.21.15.31:0/1841082190 >> 172.21.15.31:0/1841082190 conn(0x56529f856380 msgr2=0x56529f85b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 -- 172.21.15.31:0/1841082190 shutdown_connections 2023-07-29T16:30:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.333+0000 7fbe60057b80 1 -- 172.21.15.31:0/1841082190 wait complete. 2023-07-29T16:30:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 -- start start 2023-07-29T16:30:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 Processor -- start 2023-07-29T16:30:14.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 0x56529f86a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861f00 0x56529f871660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f74e4b0 con 0x56529f861f00 2023-07-29T16:30:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f74e150 con 0x56529f6bc6b0 2023-07-29T16:30:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f6358c0 con 0x56529f861150 2023-07-29T16:30:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.334+0000 7fbe60057b80 10 client.0 _ll_register_callbacks cb 0x56529f7575c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 0x56529f86a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe4a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 0x56529f86a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59644/0 (socket says 172.21.15.31:59644) 2023-07-29T16:30:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861f00 0x56529f871660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 -- 172.21.15.31:0/1779953522 learned_addr learned my addr 172.21.15.31:0/1779953522 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:14.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 -- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 msgr2=0x56529f86a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 -- 172.21.15.31:0/1779953522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861f00 msgr2=0x56529f871660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861f00 0x56529f871660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 -- 172.21.15.31:0/1779953522 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe440feeb0 con 0x56529f861150 2023-07-29T16:30:14.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 0x56529f86a620 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009db0 tx=0x7fbe3400b5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4c8c68f5fa84f25 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:14.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.335+0000 7fbe2dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:14.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe2dffb700 1 -- 172.21.15.31:0/1779953522 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe34015070 con 0x56529f861150 2023-07-29T16:30:14.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe2dffb700 1 -- 172.21.15.31:0/1779953522 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbe34004030 con 0x56529f861150 2023-07-29T16:30:14.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56529f7dccb0 con 0x56529f861150 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe2dffb700 1 -- 172.21.15.31:0/1779953522 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe340041f0 con 0x56529f861150 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 10 client.29893 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x56529f5c8c00 con 0x56529f861150 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe2dffb700 1 -- 172.21.15.31:0/1779953522 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe3402b020 con 0x56529f861150 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe2dffb700 1 -- 172.21.15.31:0/1779953522 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbe3401d400 con 0x56529f861150 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 10 client.29893 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 10 client.29893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 20 client.29893 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.336+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f875f50 0x56529f878360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.337+0000 7fbe60057b80 4 client.29893 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.337+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56529f82fd10 con 0x56529f875f50 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.337+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f875f50 0x56529f878360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.337+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f875f50 0x56529f878360 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be1ccbad308f78da server_cookie=a2b95233dea4d17d in_seq=0 out_seq=0 2023-07-29T16:30:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.337+0000 7fbe2dffb700 10 client.29893 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe2dffb700 1 -- 172.21.15.31:0/1779953522 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56529f82fd10 con 0x56529f875f50 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe2dffb700 10 client.29893 handle_command_reply: tid=0 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 client.29893 shutdown 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 msgr2=0x56529f86a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 0x56529f86a620 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009db0 tx=0x7fbe3400b5a0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 shutdown_connections 2023-07-29T16:30:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f875f50 0x56529f878360 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861f00 0x56529f871660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe2dffb700 0 client.29893 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 --2- 172.21.15.31:0/1779953522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861150 0x56529f86a620 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 >> 172.21.15.31:0/1779953522 conn(0x56529f856380 msgr2=0x56529f85adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 shutdown_connections 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.338+0000 7fbe60057b80 1 -- 172.21.15.31:0/1779953522 wait complete. 2023-07-29T16:30:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.339+0000 7fbe60057b80 20 client.29893 trim_cache size 0 max 16384 2023-07-29T16:30:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 Processor -- start 2023-07-29T16:30:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 -- start start 2023-07-29T16:30:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f868dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f85adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f869590 con 0x56529f861150 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f861810 con 0x56529f6bc6b0 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe60057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f8536e0 con 0x56529f861f00 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe4a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f85adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe4a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44534/0 (socket says 172.21.15.31:44534) 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe4ad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f85adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59652/0 (socket says 172.21.15.31:59652) 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.355+0000 7fbe4a59b700 1 -- 172.21.15.31:0/2086889223 learned_addr learned my addr 172.21.15.31:0/2086889223 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f868dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2086889223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 msgr2=0x56529f86f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2086889223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 msgr2=0x56529f868dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f868dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4ad9c700 1 -- 172.21.15.31:0/2086889223 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56529f6061f0 con 0x56529f861f00 2023-07-29T16:30:14.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:14.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f85adf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0095b0 tx=0x7fbe3c098cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=836fb11ea5ee20ae server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:14.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe397fa700 1 -- 172.21.15.31:0/2086889223 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c09d070 con 0x56529f861f00 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe397fa700 1 -- 172.21.15.31:0/2086889223 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe3c0092c0 con 0x56529f861f00 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.356+0000 7fbe397fa700 1 -- 172.21.15.31:0/2086889223 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c0c9520 con 0x56529f861f00 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 -- 172.21.15.31:0/2086889223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 msgr2=0x56529f85adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f85adf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0095b0 tx=0x7fbe3c098cb0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 -- 172.21.15.31:0/2086889223 shutdown_connections 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f85adf0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 --2- 172.21.15.31:0/2086889223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f868dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 -- 172.21.15.31:0/2086889223 >> 172.21.15.31:0/2086889223 conn(0x56529f856380 msgr2=0x56529f85b670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 -- 172.21.15.31:0/2086889223 shutdown_connections 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.357+0000 7fbe60057b80 1 -- 172.21.15.31:0/2086889223 wait complete. 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 -- start start 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 Processor -- start 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f86f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f86d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f86db80 con 0x56529f861150 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f86dcf0 con 0x56529f6bc6b0 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56529f86deb0 con 0x56529f861f00 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.358+0000 7fbe60057b80 10 client.0 _ll_register_callbacks cb 0x56529f7575c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44538/0 (socket says 172.21.15.31:44538) 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 -- 172.21.15.31:0/947083872 learned_addr learned my addr 172.21.15.31:0/947083872 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f86f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4ad9c700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f86d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 -- 172.21.15.31:0/947083872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 msgr2=0x56529f86d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f86d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 -- 172.21.15.31:0/947083872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 msgr2=0x56529f86f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f86f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 -- 172.21.15.31:0/947083872 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe44108ec0 con 0x56529f6bc6b0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe4a59b700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86eee0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c010 tx=0x7fbe40009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7db3eaf1e73d8923 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.359+0000 7fbe2dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe2dffb700 1 -- 172.21.15.31:0/947083872 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400318f0 con 0x56529f6bc6b0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe2dffb700 1 -- 172.21.15.31:0/947083872 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbe40031a50 con 0x56529f6bc6b0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe3c0a1400 con 0x56529f6bc6b0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe2dffb700 1 -- 172.21.15.31:0/947083872 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe40031c10 con 0x56529f6bc6b0 2023-07-29T16:30:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 10 client.29906 fetch_fsmap learned FSMap version 34 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x56529f5dfd80 con 0x56529f6bc6b0 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe2dffb700 1 -- 172.21.15.31:0/947083872 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe3c0a1400 con 0x56529f6bc6b0 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe2dffb700 1 -- 172.21.15.31:0/947083872 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 34) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fbe400038e0 con 0x56529f6bc6b0 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 10 client.29906 fetch_fsmap finished waiting for FSMap version 34 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 10 client.29906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 20 client.29906 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f8c8e70 0x56529f8cb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 4 client.29906 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:30:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.360+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56529f7aace0 con 0x56529f8c8e70 2023-07-29T16:30:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.361+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f8c8e70 0x56529f8cb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.361+0000 7fbe49d9a700 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f8c8e70 0x56529f8cb280 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa604b65266e8ecd server_cookie=ad32949372a92bfc in_seq=0 out_seq=0 2023-07-29T16:30:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.361+0000 7fbe2dffb700 10 client.29906 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe2dffb700 1 -- 172.21.15.31:0/947083872 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56529f7aace0 con 0x56529f8c8e70 2023-07-29T16:30:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe2dffb700 10 client.29906 handle_command_reply: tid=0 2023-07-29T16:30:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 client.29906 shutdown 2023-07-29T16:30:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 msgr2=0x56529f86eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86eee0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000c010 tx=0x7fbe40009260 comp rx=0 tx=0).stop 2023-07-29T16:30:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 shutdown_connections 2023-07-29T16:30:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56529f8c8e70 0x56529f8cb280 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56529f861f00 0x56529f86d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56529f6bc6b0 0x56529f86eee0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 --2- 172.21.15.31:0/947083872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56529f861150 0x56529f86f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe2dffb700 0 client.29906 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 >> 172.21.15.31:0/947083872 conn(0x56529f856380 msgr2=0x56529f8682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 shutdown_connections 2023-07-29T16:30:14.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 1 -- 172.21.15.31:0/947083872 wait complete. 2023-07-29T16:30:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.438+0000 7fbe60057b80 20 client.29906 trim_cache size 0 max 16384 2023-07-29T16:30:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:30:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:30:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:30:14.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:30:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbe300a0790 msgr2=0x7fbe300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbe300a0790 0x7fbe300a2c40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fbe40009fd0 tx=0x7fbe40019040 comp rx=0 tx=0).stop 2023-07-29T16:30:14.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 msgr2=0x7fbe4419d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 0x7fbe4419d500 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c00e3c0 tx=0x7fbe3c00e6d0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 shutdown_connections 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbe300a0790 0x7fbe300a2c40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe44105d50 0x7fbe4419d500 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe44068e40 0x7fbe44195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 --2- 172.21.15.31:0/2731837410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe44103280 0x7fbe44196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:14.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 >> 172.21.15.31:0/2731837410 conn(0x7fbe440fa690 msgr2=0x7fbe441015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 shutdown_connections 2023-07-29T16:30:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:14.439+0000 7fbe4b59d700 1 -- 172.21.15.31:0/2731837410 wait complete. 2023-07-29T16:30:14.906 INFO:tasks.workunit.client.0.smithi031.stdout: 1 1176693 8.81 MB/sec execute 599 sec latency 68.273 ms 2023-07-29T16:30:14.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:14 smithi031 ceph-mon[95504]: pgmap v814: 97 pgs: 97 active+clean; 217 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.5 MiB/s wr, 296 op/s 2023-07-29T16:30:15.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:14 smithi039 ceph-mon[128858]: pgmap v814: 97 pgs: 97 active+clean; 217 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.5 MiB/s wr, 296 op/s 2023-07-29T16:30:15.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:14 smithi156 ceph-mon[129245]: pgmap v814: 97 pgs: 97 active+clean; 217 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 2.5 MiB/s wr, 296 op/s 2023-07-29T16:30:15.907 INFO:tasks.workunit.client.0.smithi031.stdout: 1 cleanup 600 sec 2023-07-29T16:30:16.112 INFO:tasks.workunit.client.0.smithi031.stdout: 0 cleanup 600 sec 2023-07-29T16:30:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: 2023-07-29T16:30:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: Operation Count AvgLat MaxLat 2023-07-29T16:30:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: ---------------------------------------- 2023-07-29T16:30:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: NTCreateX 168460 1.542 3560.824 2023-07-29T16:30:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: Close 123765 0.004 0.261 2023-07-29T16:30:16.113 INFO:tasks.workunit.client.0.smithi031.stdout: Rename 7136 5.856 221.266 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Unlink 34006 1.972 5030.527 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Deltree 4 84.163 106.688 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Mkdir 2 0.015 0.016 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Qpathinfo 152741 1.045 177.806 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Qfileinfo 26771 0.002 0.074 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Qfsinfo 27990 0.308 25.463 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Sfileinfo 13724 0.038 11.051 2023-07-29T16:30:16.114 INFO:tasks.workunit.client.0.smithi031.stdout: Find 59038 0.223 85.744 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout: WriteX 83974 0.028 2.081 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout: ReadX 264158 0.010 15.240 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout: LockX 550 0.638 11.228 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout: UnlockX 550 0.635 10.861 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout: Flush 11805 3.342 75.504 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout: 2023-07-29T16:30:16.115 INFO:tasks.workunit.client.0.smithi031.stdout:Throughput 8.81184 MB/sec 1 clients 1 procs max_latency=5030.543 ms 2023-07-29T16:30:16.116 INFO:tasks.workunit.client.0.smithi031.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2023-07-29T16:30:16.116 INFO:tasks.workunit.client.0.smithi031.stdout: 2023-07-29T16:30:16.116 INFO:tasks.workunit.client.0.smithi031.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2023-07-29T16:30:16.117 INFO:tasks.workunit.client.0.smithi031.stdout:9 of 10 processes prepared for launch 0 sec 2023-07-29T16:30:16.218 INFO:tasks.workunit.client.0.smithi031.stdout:10 of 10 processes prepared for launch 0 sec 2023-07-29T16:30:16.218 INFO:tasks.workunit.client.0.smithi031.stdout:releasing clients 2023-07-29T16:30:16.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:16 smithi031 ceph-mon[95504]: pgmap v815: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.2 MiB/s wr, 209 op/s 2023-07-29T16:30:17.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:16 smithi039 ceph-mon[128858]: pgmap v815: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.2 MiB/s wr, 209 op/s 2023-07-29T16:30:17.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:16 smithi156 ceph-mon[129245]: pgmap v815: 97 pgs: 97 active+clean; 216 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.2 MiB/s wr, 209 op/s 2023-07-29T16:30:17.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 260 98.46 MB/sec warmup 1 sec latency 73.055 ms 2023-07-29T16:30:18.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 472 82.65 MB/sec warmup 2 sec latency 70.837 ms 2023-07-29T16:30:18.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:18 smithi039 ceph-mon[128858]: pgmap v816: 97 pgs: 97 active+clean; 212 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2023-07-29T16:30:18.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:18 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:30:18.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:18 smithi156 ceph-mon[129245]: pgmap v816: 97 pgs: 97 active+clean; 212 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2023-07-29T16:30:18.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:18 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:30:18.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:18 smithi031 ceph-mon[95504]: pgmap v816: 97 pgs: 97 active+clean; 212 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 2.0 MiB/s wr, 191 op/s 2023-07-29T16:30:18.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:18 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:30:19.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 709 69.48 MB/sec warmup 3 sec latency 303.235 ms 2023-07-29T16:30:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:19 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:30:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:19 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:30:19.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:19 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:19 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:30:19.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:19 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:30:19.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:19 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:19 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:30:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:19 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:30:19.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:19 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:20.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 920 54.30 MB/sec warmup 4 sec latency 179.917 ms 2023-07-29T16:30:20.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:20 smithi039 ceph-mon[128858]: pgmap v817: 97 pgs: 97 active+clean; 226 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 3.6 MiB/s wr, 283 op/s 2023-07-29T16:30:20.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:20 smithi156 ceph-mon[129245]: pgmap v817: 97 pgs: 97 active+clean; 226 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 3.6 MiB/s wr, 283 op/s 2023-07-29T16:30:20.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:20 smithi031 ceph-mon[95504]: pgmap v817: 97 pgs: 97 active+clean; 226 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 3.6 MiB/s wr, 283 op/s 2023-07-29T16:30:21.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1310 48.49 MB/sec warmup 5 sec latency 51.428 ms 2023-07-29T16:30:21.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:21 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:21.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:21 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:21.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:21 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:22.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 1892 44.20 MB/sec warmup 6 sec latency 79.375 ms 2023-07-29T16:30:22.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:22 smithi156 ceph-mon[129245]: pgmap v818: 97 pgs: 97 active+clean; 226 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 2.8 MiB/s wr, 219 op/s 2023-07-29T16:30:22.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:22 smithi031 ceph-mon[95504]: pgmap v818: 97 pgs: 97 active+clean; 226 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 2.8 MiB/s wr, 219 op/s 2023-07-29T16:30:22.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:22 smithi039 ceph-mon[128858]: pgmap v818: 97 pgs: 97 active+clean; 226 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 2.8 MiB/s wr, 219 op/s 2023-07-29T16:30:23.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2404 40.20 MB/sec warmup 7 sec latency 78.151 ms 2023-07-29T16:30:24.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2545 36.25 MB/sec warmup 8 sec latency 413.412 ms 2023-07-29T16:30:24.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:24 smithi031 ceph-mon[95504]: pgmap v819: 97 pgs: 97 active+clean; 214 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 5.0 MiB/s wr, 359 op/s 2023-07-29T16:30:25.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:24 smithi039 ceph-mon[128858]: pgmap v819: 97 pgs: 97 active+clean; 214 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 5.0 MiB/s wr, 359 op/s 2023-07-29T16:30:25.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:24 smithi156 ceph-mon[129245]: pgmap v819: 97 pgs: 97 active+clean; 214 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 5.0 MiB/s wr, 359 op/s 2023-07-29T16:30:25.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 2959 36.70 MB/sec warmup 9 sec latency 303.185 ms 2023-07-29T16:30:26.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3439 37.26 MB/sec warmup 10 sec latency 62.291 ms 2023-07-29T16:30:26.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:26 smithi031 ceph-mon[95504]: pgmap v820: 97 pgs: 97 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 9.7 MiB/s wr, 345 op/s 2023-07-29T16:30:27.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:26 smithi039 ceph-mon[128858]: pgmap v820: 97 pgs: 97 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 9.7 MiB/s wr, 345 op/s 2023-07-29T16:30:27.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:26 smithi156 ceph-mon[129245]: pgmap v820: 97 pgs: 97 active+clean; 260 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 9.7 MiB/s wr, 345 op/s 2023-07-29T16:30:27.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3711 35.48 MB/sec warmup 11 sec latency 81.075 ms 2023-07-29T16:30:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 -- 172.21.15.31:0/3946585859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20101ed0 msgr2=0x7f8d20108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 --2- 172.21.15.31:0/3946585859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20101ed0 0x7f8d20108790 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800b0a0 tx=0x7f8d1800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 -- 172.21.15.31:0/3946585859 shutdown_connections 2023-07-29T16:30:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 --2- 172.21.15.31:0/3946585859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20101ed0 0x7f8d20108790 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 --2- 172.21.15.31:0/3946585859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d20100350 0x7f8d20100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 --2- 172.21.15.31:0/3946585859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d20101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 -- 172.21.15.31:0/3946585859 >> 172.21.15.31:0/3946585859 conn(0x7f8d200fb970 msgr2=0x7f8d200fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 -- 172.21.15.31:0/3946585859 shutdown_connections 2023-07-29T16:30:27.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.598+0000 7f8d287ff700 1 -- 172.21.15.31:0/3946585859 wait complete. 2023-07-29T16:30:27.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.599+0000 7f8d287ff700 1 Processor -- start 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.599+0000 7f8d287ff700 1 -- start start 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d287ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100350 0x7f8d20198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d287ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d201989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d287ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d20101ed0 0x7f8d20199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d287ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d20199ea0 con 0x7f8d20101ed0 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d287ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2019a010 con 0x7f8d20100350 2023-07-29T16:30:27.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d287ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2019a1d0 con 0x7f8d20101540 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100350 0x7f8d20198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d201989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100350 0x7f8d20198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57094/0 (socket says 172.21.15.31:57094) 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d201989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53516/0 (socket says 172.21.15.31:53516) 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d2659b700 1 -- 172.21.15.31:0/1245523636 learned_addr learned my addr 172.21.15.31:0/1245523636 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d20101ed0 0x7f8d20199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 -- 172.21.15.31:0/1245523636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100350 msgr2=0x7f8d20198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100350 0x7f8d20198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 -- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d20101ed0 msgr2=0x7f8d20199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d20101ed0 0x7f8d20199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.600+0000 7f8d25d9a700 1 -- 172.21.15.31:0/1245523636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18009d00 con 0x7f8d20101540 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.601+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d201989c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000d950 tx=0x7f8d1000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.601+0000 7f8d177fe700 1 -- 172.21.15.31:0/1245523636 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10009930 con 0x7f8d20101540 2023-07-29T16:30:27.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.601+0000 7f8d177fe700 1 -- 172.21.15.31:0/1245523636 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d10010460 con 0x7f8d20101540 2023-07-29T16:30:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.601+0000 7f8d177fe700 1 -- 172.21.15.31:0/1245523636 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000b5d0 con 0x7f8d20101540 2023-07-29T16:30:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.601+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2019a4b0 con 0x7f8d20101540 2023-07-29T16:30:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.601+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d201a0c70 con 0x7f8d20101540 2023-07-29T16:30:27.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.604+0000 7f8d177fe700 1 -- 172.21.15.31:0/1245523636 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8d10010820 con 0x7f8d20101540 2023-07-29T16:30:27.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.606+0000 7f8d3c1aab80 1 Processor -- start 2023-07-29T16:30:27.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.606+0000 7f8d177fe700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8d0c0a0790 0x7f8d0c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.606+0000 7f8d3c1aab80 1 -- start start 2023-07-29T16:30:27.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.606+0000 7f8d177fe700 1 -- 172.21.15.31:0/1245523636 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8d100cb720 con 0x7f8d20101540 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8d0c0a0790 0x7f8d0c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c798b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41bd19b0 con 0x55ec41c7acf0 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41bcf370 con 0x55ec41ad56b0 2023-07-29T16:30:27.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41bcac60 con 0x55ec41c79df0 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c798b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c798b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57108/0 (socket says 172.21.15.31:57108) 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 -- 172.21.15.31:0/3697387108 learned_addr learned my addr 172.21.15.31:0/3697387108 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 -- 172.21.15.31:0/3697387108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 msgr2=0x55ec41c7a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 -- 172.21.15.31:0/3697387108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 msgr2=0x55ec41c7b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.607+0000 7f8d2659b700 1 -- 172.21.15.31:0/3697387108 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec419f8d80 con 0x55ec41ad56b0 2023-07-29T16:30:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.608+0000 7f8d2659b700 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c798b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009df0 tx=0x7f8d1c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb00886553f62ad9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.608+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8d0c0a0790 0x7f8d0c0a2c40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009cd0 tx=0x7f8d18007db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.608+0000 7f8d14ff9700 1 -- 172.21.15.31:0/3697387108 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c004230 con 0x55ec41ad56b0 2023-07-29T16:30:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.608+0000 7f8d14ff9700 1 -- 172.21.15.31:0/3697387108 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d1c004390 con 0x55ec41ad56b0 2023-07-29T16:30:27.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.608+0000 7f8d14ff9700 1 -- 172.21.15.31:0/3697387108 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c019800 con 0x55ec41ad56b0 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3697387108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 msgr2=0x55ec41c798b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c798b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c009df0 tx=0x7f8d1c011720 comp rx=0 tx=0).stop 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3697387108 shutdown_connections 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c798b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3697387108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3697387108 >> 172.21.15.31:0/3697387108 conn(0x55ec41c6efd0 msgr2=0x55ec41c74300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3697387108 shutdown_connections 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3697387108 wait complete. 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.609+0000 7f8d3c1aab80 1 -- start start 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 Processor -- start 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41ad56b0 0x55ec41c82f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 0x55ec41c83490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c7acf0 0x55ec41c8a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41b674b0 con 0x55ec41ad56b0 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41b67150 con 0x55ec41c79df0 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41a4e8c0 con 0x55ec41c7acf0 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d3c1aab80 10 client.0 _ll_register_callbacks cb 0x55ec41b705c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d25d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 0x55ec41c83490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41ad56b0 0x55ec41c82f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d25d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 0x55ec41c83490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57124/0 (socket says 172.21.15.31:57124) 2023-07-29T16:30:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d25d9a700 1 -- 172.21.15.31:0/951676923 learned_addr learned my addr 172.21.15.31:0/951676923 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.610+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c7acf0 0x55ec41c8a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d25d9a700 1 -- 172.21.15.31:0/951676923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c7acf0 msgr2=0x55ec41c8a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c7acf0 0x55ec41c8a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d25d9a700 1 -- 172.21.15.31:0/951676923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41ad56b0 msgr2=0x55ec41c82f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41ad56b0 0x55ec41c82f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d25d9a700 1 -- 172.21.15.31:0/951676923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c009a50 con 0x55ec41c79df0 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 0x55ec41c83490 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009520 tx=0x7f8d1009ce70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bae186a96d22dfa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d09ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d09ffb700 1 -- 172.21.15.31:0/951676923 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10099020 con 0x55ec41c79df0 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d09ffb700 1 -- 172.21.15.31:0/951676923 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8d10099180 con 0x55ec41c79df0 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec41bf5cb0 con 0x55ec41c79df0 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d09ffb700 1 -- 172.21.15.31:0/951676923 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d100982b0 con 0x55ec41c79df0 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d3c1aab80 10 client.29918 fetch_fsmap learned FSMap version 36 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.611+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55ec419e1c00 con 0x55ec41c79df0 2023-07-29T16:30:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d09ffb700 1 -- 172.21.15.31:0/951676923 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec41bf5cb0 con 0x55ec41c79df0 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d09ffb700 1 -- 172.21.15.31:0/951676923 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 36) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f8d10094bb0 con 0x55ec41c79df0 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d3c1aab80 10 client.29918 fetch_fsmap finished waiting for FSMap version 36 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d3c1aab80 10 client.29918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d3c1aab80 20 client.29918 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41c8ed10 0x55ec41c91120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d3c1aab80 4 client.29918 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec41c48d10 con 0x55ec41c8ed10 2023-07-29T16:30:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.612+0000 7f8d2659b700 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41c8ed10 0x55ec41c91120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.613+0000 7f8d2659b700 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41c8ed10 0x55ec41c91120 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ff8ae5b7f45a9c server_cookie=16e0b60f9a1ee819 in_seq=0 out_seq=0 2023-07-29T16:30:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.613+0000 7f8d09ffb700 10 client.29918 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.618+0000 7f8d09ffb700 1 -- 172.21.15.31:0/951676923 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55ec41c48d10 con 0x55ec41c8ed10 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.618+0000 7f8d09ffb700 10 client.29918 handle_command_reply: tid=0 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.618+0000 7f8d3c1aab80 1 client.29918 shutdown 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 msgr2=0x55ec41c83490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 0x55ec41c83490 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009520 tx=0x7f8d1009ce70 comp rx=0 tx=0).stop 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 shutdown_connections 2023-07-29T16:30:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41c8ed10 0x55ec41c91120 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c7acf0 0x55ec41c8a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41ad56b0 0x55ec41c82f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/951676923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c79df0 0x55ec41c83490 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d09ffb700 0 client.29918 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 >> 172.21.15.31:0/951676923 conn(0x55ec41c6efd0 msgr2=0x55ec41c73a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 shutdown_connections 2023-07-29T16:30:27.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/951676923 wait complete. 2023-07-29T16:30:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.619+0000 7f8d3c1aab80 20 client.29918 trim_cache size 0 max 16384 2023-07-29T16:30:27.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.636+0000 7f8d3c1aab80 1 Processor -- start 2023-07-29T16:30:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.636+0000 7f8d3c1aab80 1 -- start start 2023-07-29T16:30:27.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c88480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41c82400 con 0x55ec41c7acf0 2023-07-29T16:30:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41c82ac0 con 0x55ec41ad56b0 2023-07-29T16:30:27.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41c8eb00 con 0x55ec41c79df0 2023-07-29T16:30:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c88480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c88480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57136/0 (socket says 172.21.15.31:57136) 2023-07-29T16:30:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.637+0000 7f8d2659b700 1 -- 172.21.15.31:0/1820961580 learned_addr learned my addr 172.21.15.31:0/1820961580 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d2659b700 1 -- 172.21.15.31:0/1820961580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 msgr2=0x55ec41c7dc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:30:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d2659b700 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7dc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d2659b700 1 -- 172.21.15.31:0/1820961580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 msgr2=0x55ec41c7e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d2659b700 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d2659b700 1 -- 172.21.15.31:0/1820961580 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec41a1f1f0 con 0x55ec41ad56b0 2023-07-29T16:30:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.638+0000 7f8d2659b700 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c88480 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c003d70 tx=0x7f8d1c0098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e61cc0653acae32 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.639+0000 7f8d157fa700 1 -- 172.21.15.31:0/1820961580 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c003da0 con 0x55ec41ad56b0 2023-07-29T16:30:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.639+0000 7f8d157fa700 1 -- 172.21.15.31:0/1820961580 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d1c02c070 con 0x55ec41ad56b0 2023-07-29T16:30:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.639+0000 7f8d157fa700 1 -- 172.21.15.31:0/1820961580 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c004880 con 0x55ec41ad56b0 2023-07-29T16:30:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.640+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/1820961580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 msgr2=0x55ec41c88480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:27.641 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.640+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c88480 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c003d70 tx=0x7f8d1c0098f0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.641+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/1820961580 shutdown_connections 2023-07-29T16:30:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.641+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c7acf0 0x55ec41c7e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.641+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41ad56b0 0x55ec41c88480 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.641+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/1820961580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41c79df0 0x55ec41c7dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.642 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.641+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/1820961580 >> 172.21.15.31:0/1820961580 conn(0x55ec41c6efd0 msgr2=0x55ec41c810f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.641+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/1820961580 shutdown_connections 2023-07-29T16:30:27.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.642+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/1820961580 wait complete. 2023-07-29T16:30:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.642+0000 7f8d3c1aab80 1 -- start start 2023-07-29T16:30:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 Processor -- start 2023-07-29T16:30:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ad56b0 0x55ec41c87f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c79df0 0x55ec41c884c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c7acf0 0x55ec41c86680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.644 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41c86d60 con 0x55ec41c79df0 2023-07-29T16:30:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41c86ea0 con 0x55ec41c7acf0 2023-07-29T16:30:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec41c87060 con 0x55ec41ad56b0 2023-07-29T16:30:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.643+0000 7f8d3c1aab80 10 client.0 _ll_register_callbacks cb 0x55ec41b705c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.644+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ad56b0 0x55ec41c87f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.644+0000 7f8d2659b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ad56b0 0x55ec41c87f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53556/0 (socket says 172.21.15.31:53556) 2023-07-29T16:30:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.644+0000 7f8d2659b700 1 -- 172.21.15.31:0/3288421152 learned_addr learned my addr 172.21.15.31:0/3288421152 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.644+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c79df0 0x55ec41c884c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.645+0000 7f8d26d9c700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c7acf0 0x55ec41c86680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.645+0000 7f8d25d9a700 1 -- 172.21.15.31:0/3288421152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ad56b0 msgr2=0x55ec41c87f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.645+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ad56b0 0x55ec41c87f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.646+0000 7f8d25d9a700 1 -- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c7acf0 msgr2=0x55ec41c86680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.646+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c7acf0 0x55ec41c86680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.646+0000 7f8d25d9a700 1 -- 172.21.15.31:0/3288421152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec419f8d80 con 0x55ec41c79df0 2023-07-29T16:30:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.646+0000 7f8d25d9a700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c79df0 0x55ec41c884c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8d10005950 tx=0x7f8d10010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b632fcaec2171e5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.646+0000 7f8d09ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:30:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.647+0000 7f8d09ffb700 1 -- 172.21.15.31:0/3288421152 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10012ee0 con 0x55ec41c79df0 2023-07-29T16:30:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.647+0000 7f8d09ffb700 1 -- 172.21.15.31:0/3288421152 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8d100994c0 con 0x55ec41c79df0 2023-07-29T16:30:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.647+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d1c004390 con 0x55ec41c79df0 2023-07-29T16:30:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.648+0000 7f8d09ffb700 1 -- 172.21.15.31:0/3288421152 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d100982b0 con 0x55ec41c79df0 2023-07-29T16:30:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.648+0000 7f8d3c1aab80 10 client.19788 fetch_fsmap learned FSMap version 36 2023-07-29T16:30:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.648+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55ec419f7250 con 0x55ec41c79df0 2023-07-29T16:30:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d09ffb700 1 -- 172.21.15.31:0/3288421152 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d10063020 con 0x55ec41c79df0 2023-07-29T16:30:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d09ffb700 1 -- 172.21.15.31:0/3288421152 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 36) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f8d10099020 con 0x55ec41c79df0 2023-07-29T16:30:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d3c1aab80 10 client.19788 fetch_fsmap finished waiting for FSMap version 36 2023-07-29T16:30:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d3c1aab80 10 client.19788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d3c1aab80 20 client.19788 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41ce18c0 0x55ec41ce3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d3c1aab80 4 client.19788 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:30:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ec41bc3ce0 con 0x55ec41ce18c0 2023-07-29T16:30:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.649+0000 7f8d2659b700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41ce18c0 0x55ec41ce3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.651+0000 7f8d2659b700 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41ce18c0 0x55ec41ce3cd0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56f6609a36a85bf6 server_cookie=d586703e844fd914 in_seq=0 out_seq=0 2023-07-29T16:30:27.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:27.652+0000 7f8d09ffb700 10 client.19788 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:28.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3808 33.04 MB/sec warmup 12 sec latency 279.079 ms 2023-07-29T16:30:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.261+0000 7f8d09ffb700 1 -- 172.21.15.31:0/3288421152 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ec41bc3ce0 con 0x55ec41ce18c0 2023-07-29T16:30:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d09ffb700 10 client.19788 handle_command_reply: tid=0 2023-07-29T16:30:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 client.19788 shutdown 2023-07-29T16:30:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c79df0 msgr2=0x55ec41c884c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c79df0 0x55ec41c884c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8d10005950 tx=0x7f8d10010c20 comp rx=0 tx=0).stop 2023-07-29T16:30:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 shutdown_connections 2023-07-29T16:30:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55ec41ce18c0 0x55ec41ce3cd0 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ec41c7acf0 0x55ec41c86680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ec41ad56b0 0x55ec41c87f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 --2- 172.21.15.31:0/3288421152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ec41c79df0 0x55ec41c884c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 >> 172.21.15.31:0/3288421152 conn(0x55ec41c6efd0 msgr2=0x55ec41c7dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.262+0000 7f8d09ffb700 0 client.19788 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.263+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 shutdown_connections 2023-07-29T16:30:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.263+0000 7f8d3c1aab80 1 -- 172.21.15.31:0/3288421152 wait complete. 2023-07-29T16:30:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.263+0000 7f8d3c1aab80 20 client.19788 trim_cache size 0 max 16384 2023-07-29T16:30:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:30:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:30:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:30:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:30:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8d0c0a0790 msgr2=0x7f8d0c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8d0c0a0790 0x7f8d0c0a2c40 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009cd0 tx=0x7f8d18007db0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 msgr2=0x7f8d201989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d201989c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000d950 tx=0x7f8d1000dc60 comp rx=0 tx=0).stop 2023-07-29T16:30:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 shutdown_connections 2023-07-29T16:30:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8d0c0a0790 0x7f8d0c0a2c40 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d20101ed0 0x7f8d20199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d20100350 0x7f8d20198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 --2- 172.21.15.31:0/1245523636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8d20101540 0x7f8d201989c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:28.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 >> 172.21.15.31:0/1245523636 conn(0x7f8d200fb970 msgr2=0x7f8d20106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:28.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 shutdown_connections 2023-07-29T16:30:28.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:28.264+0000 7f8d287ff700 1 -- 172.21.15.31:0/1245523636 wait complete. 2023-07-29T16:30:28.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:28 smithi156 ceph-mon[129245]: pgmap v821: 97 pgs: 97 active+clean; 286 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 12 MiB/s wr, 349 op/s 2023-07-29T16:30:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:28 smithi031 ceph-mon[95504]: pgmap v821: 97 pgs: 97 active+clean; 286 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 12 MiB/s wr, 349 op/s 2023-07-29T16:30:28.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:28 smithi039 ceph-mon[128858]: pgmap v821: 97 pgs: 97 active+clean; 286 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 12 MiB/s wr, 349 op/s 2023-07-29T16:30:29.218 INFO:tasks.workunit.client.0.smithi031.stdout: 10 3941 30.72 MB/sec warmup 13 sec latency 96.394 ms 2023-07-29T16:30:30.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4080 28.62 MB/sec warmup 14 sec latency 178.487 ms 2023-07-29T16:30:30.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:30 smithi031 ceph-mon[95504]: pgmap v822: 97 pgs: 97 active+clean; 364 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 19 MiB/s wr, 571 op/s 2023-07-29T16:30:31.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:30 smithi039 ceph-mon[128858]: pgmap v822: 97 pgs: 97 active+clean; 364 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 19 MiB/s wr, 571 op/s 2023-07-29T16:30:31.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:30 smithi156 ceph-mon[129245]: pgmap v822: 97 pgs: 97 active+clean; 364 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 19 MiB/s wr, 571 op/s 2023-07-29T16:30:31.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4255 26.89 MB/sec warmup 15 sec latency 82.748 ms 2023-07-29T16:30:32.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4521 25.77 MB/sec warmup 16 sec latency 110.256 ms 2023-07-29T16:30:32.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:32 smithi031 ceph-mon[95504]: pgmap v823: 97 pgs: 97 active+clean; 364 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 17 MiB/s wr, 450 op/s 2023-07-29T16:30:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:32 smithi039 ceph-mon[128858]: pgmap v823: 97 pgs: 97 active+clean; 364 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 17 MiB/s wr, 450 op/s 2023-07-29T16:30:33.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:32 smithi156 ceph-mon[129245]: pgmap v823: 97 pgs: 97 active+clean; 364 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 17 MiB/s wr, 450 op/s 2023-07-29T16:30:33.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 4884 25.60 MB/sec warmup 17 sec latency 58.424 ms 2023-07-29T16:30:34.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5390 25.23 MB/sec warmup 18 sec latency 69.860 ms 2023-07-29T16:30:34.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:34 smithi031 ceph-mon[95504]: pgmap v824: 97 pgs: 97 active+clean; 363 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 18 MiB/s wr, 629 op/s 2023-07-29T16:30:35.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:34 smithi039 ceph-mon[128858]: pgmap v824: 97 pgs: 97 active+clean; 363 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 18 MiB/s wr, 629 op/s 2023-07-29T16:30:35.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:34 smithi156 ceph-mon[129245]: pgmap v824: 97 pgs: 97 active+clean; 363 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 18 MiB/s wr, 629 op/s 2023-07-29T16:30:35.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 5895 24.81 MB/sec warmup 19 sec latency 57.925 ms 2023-07-29T16:30:36.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6308 25.03 MB/sec warmup 20 sec latency 67.755 ms 2023-07-29T16:30:36.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:36 smithi031 ceph-mon[95504]: pgmap v825: 97 pgs: 97 active+clean; 365 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 16 MiB/s wr, 503 op/s 2023-07-29T16:30:37.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:36 smithi039 ceph-mon[128858]: pgmap v825: 97 pgs: 97 active+clean; 365 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 16 MiB/s wr, 503 op/s 2023-07-29T16:30:37.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:36 smithi156 ceph-mon[129245]: pgmap v825: 97 pgs: 97 active+clean; 365 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 16 MiB/s wr, 503 op/s 2023-07-29T16:30:37.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 6736 25.72 MB/sec warmup 21 sec latency 56.115 ms 2023-07-29T16:30:38.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7105 26.02 MB/sec warmup 22 sec latency 141.913 ms 2023-07-29T16:30:38.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:38 smithi039 ceph-mon[128858]: pgmap v826: 97 pgs: 97 active+clean; 366 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 11 MiB/s wr, 450 op/s 2023-07-29T16:30:38.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:38 smithi156 ceph-mon[129245]: pgmap v826: 97 pgs: 97 active+clean; 366 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 11 MiB/s wr, 450 op/s 2023-07-29T16:30:38.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:38 smithi031 ceph-mon[95504]: pgmap v826: 97 pgs: 97 active+clean; 366 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 11 MiB/s wr, 450 op/s 2023-07-29T16:30:39.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 7548 25.77 MB/sec warmup 23 sec latency 161.393 ms 2023-07-29T16:30:39.276 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:39 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:39.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:39 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:39.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:39 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:40.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8160 25.96 MB/sec warmup 24 sec latency 60.617 ms 2023-07-29T16:30:40.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:40 smithi156 ceph-mon[129245]: pgmap v827: 97 pgs: 97 active+clean; 372 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.3 MiB/s wr, 510 op/s 2023-07-29T16:30:40.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:40 smithi031 ceph-mon[95504]: pgmap v827: 97 pgs: 97 active+clean; 372 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.3 MiB/s wr, 510 op/s 2023-07-29T16:30:40.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:40 smithi039 ceph-mon[128858]: pgmap v827: 97 pgs: 97 active+clean; 372 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.3 MiB/s wr, 510 op/s 2023-07-29T16:30:41.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 8575 25.48 MB/sec warmup 25 sec latency 149.911 ms 2023-07-29T16:30:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 -- 172.21.15.31:0/2476114843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c0febf0 msgr2=0x7fca5c101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 --2- 172.21.15.31:0/2476114843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c0febf0 0x7fca5c101010 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009990 tx=0x7fca4c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 -- 172.21.15.31:0/2476114843 shutdown_connections 2023-07-29T16:30:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 --2- 172.21.15.31:0/2476114843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c103f10 0x7fca5c104380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 --2- 172.21.15.31:0/2476114843 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c0febf0 0x7fca5c101010 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 --2- 172.21.15.31:0/2476114843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca5c101550 0x7fca5c1039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 -- 172.21.15.31:0/2476114843 >> 172.21.15.31:0/2476114843 conn(0x7fca5c0fa8a0 msgr2=0x7fca5c0fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 -- 172.21.15.31:0/2476114843 shutdown_connections 2023-07-29T16:30:41.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.480+0000 7fca629ff700 1 -- 172.21.15.31:0/2476114843 wait complete. 2023-07-29T16:30:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.483+0000 7fca629ff700 1 Processor -- start 2023-07-29T16:30:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.483+0000 7fca629ff700 1 -- start start 2023-07-29T16:30:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca629ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c0febf0 0x7fca5c19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca629ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 0x7fca5c19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca629ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca5c103f10 0x7fca5c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca629ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c19e150 con 0x7fca5c0febf0 2023-07-29T16:30:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca629ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c19e2c0 con 0x7fca5c103f10 2023-07-29T16:30:41.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca629ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5c19e480 con 0x7fca5c101550 2023-07-29T16:30:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca5b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 0x7fca5c19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca5b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 0x7fca5c19cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59130/0 (socket says 172.21.15.31:59130) 2023-07-29T16:30:41.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca5b7fe700 1 -- 172.21.15.31:0/3012574420 learned_addr learned my addr 172.21.15.31:0/3012574420 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca5bfff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c0febf0 0x7fca5c19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.484+0000 7fca60f9c700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca5c103f10 0x7fca5c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5b7fe700 1 -- 172.21.15.31:0/3012574420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca5c103f10 msgr2=0x7fca5c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca5c103f10 0x7fca5c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5b7fe700 1 -- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c0febf0 msgr2=0x7fca5c19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c0febf0 0x7fca5c19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5b7fe700 1 -- 172.21.15.31:0/3012574420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4c0095f0 con 0x7fca5c101550 2023-07-29T16:30:41.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5bfff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c0febf0 0x7fca5c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:41.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 0x7fca5c19cc70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fca5000e9a0 tx=0x7fca5000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca597fa700 1 -- 172.21.15.31:0/3012574420 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50009790 con 0x7fca5c101550 2023-07-29T16:30:41.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca597fa700 1 -- 172.21.15.31:0/3012574420 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca50010430 con 0x7fca5c101550 2023-07-29T16:30:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca597fa700 1 -- 172.21.15.31:0/3012574420 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50018680 con 0x7fca5c101550 2023-07-29T16:30:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5c19e760 con 0x7fca5c101550 2023-07-29T16:30:41.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.485+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c1a4f20 con 0x7fca5c101550 2023-07-29T16:30:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.490+0000 7fca597fa700 1 -- 172.21.15.31:0/3012574420 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fca500105a0 con 0x7fca5c101550 2023-07-29T16:30:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.492+0000 7fca597fa700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca440a0810 0x7fca440a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.493+0000 7fca597fa700 1 -- 172.21.15.31:0/3012574420 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca50014070 con 0x7fca5c101550 2023-07-29T16:30:41.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.493+0000 7fca5bfff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca440a0810 0x7fca440a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.493+0000 7fca76377b80 1 Processor -- start 2023-07-29T16:30:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 -- start start 2023-07-29T16:30:41.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab095e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab095f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab0960160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab08b69b0 con 0x558ab095fcf0 2023-07-29T16:30:41.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab08b4370 con 0x558ab07ba6b0 2023-07-29T16:30:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca76377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab08afc60 con 0x558ab095edf0 2023-07-29T16:30:41.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca5bfff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca440a0810 0x7fca440a2cc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00b2e0 tx=0x7fca4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.494+0000 7fca5bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab095e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab095e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37912/0 (socket says 172.21.15.31:37912) 2023-07-29T16:30:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 -- 172.21.15.31:0/202323423 learned_addr learned my addr 172.21.15.31:0/202323423 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca60f9c700 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab0960160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab095f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 -- 172.21.15.31:0/202323423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 msgr2=0x558ab095f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab095f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 -- 172.21.15.31:0/202323423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 msgr2=0x558ab0960160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab0960160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 -- 172.21.15.31:0/202323423 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ab06ddd80 con 0x558ab07ba6b0 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca60f9c700 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab0960160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.495+0000 7fca5bfff700 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab095e8b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fca4c015000 tx=0x7fca4c020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b484520be43ea1d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.496+0000 7fca427fc700 1 -- 172.21.15.31:0/202323423 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c02b070 con 0x558ab07ba6b0 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.496+0000 7fca427fc700 1 -- 172.21.15.31:0/202323423 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca4c01e040 con 0x558ab07ba6b0 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.496+0000 7fca427fc700 1 -- 172.21.15.31:0/202323423 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c02f660 con 0x558ab07ba6b0 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- 172.21.15.31:0/202323423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 msgr2=0x558ab095e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab095e8b0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fca4c015000 tx=0x7fca4c020000 comp rx=0 tx=0).stop 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- 172.21.15.31:0/202323423 shutdown_connections 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab0960160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab095e8b0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- 172.21.15.31:0/202323423 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab095f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- 172.21.15.31:0/202323423 >> 172.21.15.31:0/202323423 conn(0x558ab0953fd0 msgr2=0x558ab0959300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- 172.21.15.31:0/202323423 shutdown_connections 2023-07-29T16:30:41.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- 172.21.15.31:0/202323423 wait complete. 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- start start 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 Processor -- start 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab0967e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab0968380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab084c4b0 con 0x558ab095fcf0 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab084c150 con 0x558ab07ba6b0 2023-07-29T16:30:41.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab07338c0 con 0x558ab095edf0 2023-07-29T16:30:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.497+0000 7fca76377b80 10 client.0 _ll_register_callbacks cb 0x558ab08555c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab0968380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab0968380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59150/0 (socket says 172.21.15.31:59150) 2023-07-29T16:30:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 -- 172.21.15.31:0/908533968 learned_addr learned my addr 172.21.15.31:0/908533968 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5bfff700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab0967e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca60f9c700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 -- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 msgr2=0x558ab0967e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab0967e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 -- 172.21.15.31:0/908533968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 msgr2=0x558ab096f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 -- 172.21.15.31:0/908533968 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5c104eb0 con 0x558ab095edf0 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca60f9c700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab0968380 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fca50095930 tx=0x7fca50099970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a655017d945bb41 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca3b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.499+0000 7fca3b7fe700 1 -- 172.21.15.31:0/908533968 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca500ae030 con 0x558ab095edf0 2023-07-29T16:30:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.500+0000 7fca3b7fe700 1 -- 172.21.15.31:0/908533968 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca500af070 con 0x558ab095edf0 2023-07-29T16:30:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.500+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ab08dacb0 con 0x558ab095edf0 2023-07-29T16:30:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.500+0000 7fca3b7fe700 1 -- 172.21.15.31:0/908533968 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca500a1880 con 0x558ab095edf0 2023-07-29T16:30:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.500+0000 7fca76377b80 10 client.29926 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.500+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x558ab06c6c00 con 0x558ab095edf0 2023-07-29T16:30:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca3b7fe700 1 -- 172.21.15.31:0/908533968 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558ab08dacb0 con 0x558ab095edf0 2023-07-29T16:30:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca3b7fe700 1 -- 172.21.15.31:0/908533968 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7fca50095220 con 0x558ab095edf0 2023-07-29T16:30:41.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca76377b80 10 client.29926 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca76377b80 10 client.29926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca76377b80 20 client.29926 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:41.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca76377b80 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab0973c00 0x558ab0976010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca76377b80 4 client.29926 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:41.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ab092dd10 con 0x558ab0973c00 2023-07-29T16:30:41.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca5bfff700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab0973c00 0x558ab0976010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca5bfff700 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab0973c00 0x558ab0976010 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49285930cb4eeb28 server_cookie=83fa3df2340b10df in_seq=0 out_seq=0 2023-07-29T16:30:41.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.501+0000 7fca3b7fe700 10 client.29926 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:30:41.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca3b7fe700 1 -- 172.21.15.31:0/908533968 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558ab092dd10 con 0x558ab0973c00 2023-07-29T16:30:41.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca3b7fe700 10 client.29926 handle_command_reply: tid=0 2023-07-29T16:30:41.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca76377b80 1 client.29926 shutdown 2023-07-29T16:30:41.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 msgr2=0x558ab0968380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca76377b80 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab0968380 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fca50095930 tx=0x7fca50099970 comp rx=0 tx=0).stop 2023-07-29T16:30:41.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 shutdown_connections 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.503+0000 7fca76377b80 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab0973c00 0x558ab0976010 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab07ba6b0 0x558ab0967e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 1 --2- 172.21.15.31:0/908533968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095edf0 0x558ab0968380 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca3b7fe700 0 client.29926 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 >> 172.21.15.31:0/908533968 conn(0x558ab0953fd0 msgr2=0x558ab0958a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 shutdown_connections 2023-07-29T16:30:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 1 -- 172.21.15.31:0/908533968 wait complete. 2023-07-29T16:30:41.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.505+0000 7fca76377b80 20 client.29926 trim_cache size 0 max 16384 2023-07-29T16:30:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.526+0000 7fca76377b80 1 Processor -- start 2023-07-29T16:30:41.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.526+0000 7fca76377b80 1 -- start start 2023-07-29T16:30:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca76377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab07ba6b0 0x558ab0966b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca76377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca76377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 0x558ab0959170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca76377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab0967a10 con 0x558ab07ba6b0 2023-07-29T16:30:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca76377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab08dacb0 con 0x558ab095edf0 2023-07-29T16:30:41.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca76377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab0968d40 con 0x558ab095fcf0 2023-07-29T16:30:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca60f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 0x558ab0959170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca60f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 0x558ab0959170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59164/0 (socket says 172.21.15.31:59164) 2023-07-29T16:30:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca60f9c700 1 -- 172.21.15.31:0/2493103277 learned_addr learned my addr 172.21.15.31:0/2493103277 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.527+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.528+0000 7fca60f9c700 1 -- 172.21.15.31:0/2493103277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 msgr2=0x558ab096f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.528+0000 7fca60f9c700 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.528+0000 7fca60f9c700 1 -- 172.21.15.31:0/2493103277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab07ba6b0 msgr2=0x558ab0966b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.528+0000 7fca60f9c700 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab07ba6b0 0x558ab0966b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.528+0000 7fca60f9c700 1 -- 172.21.15.31:0/2493103277 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ab07041f0 con 0x558ab095fcf0 2023-07-29T16:30:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.529+0000 7fca60f9c700 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 0x558ab0959170 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca54006070 tx=0x7fca54007950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a65bb1410238379f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.530+0000 7fca42ffd700 1 -- 172.21.15.31:0/2493103277 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54012e50 con 0x558ab095fcf0 2023-07-29T16:30:41.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.530+0000 7fca42ffd700 1 -- 172.21.15.31:0/2493103277 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca54016920 con 0x558ab095fcf0 2023-07-29T16:30:41.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.530+0000 7fca42ffd700 1 -- 172.21.15.31:0/2493103277 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca54015920 con 0x558ab095fcf0 2023-07-29T16:30:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 -- 172.21.15.31:0/2493103277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 msgr2=0x558ab0959170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 0x558ab0959170 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fca54006070 tx=0x7fca54007950 comp rx=0 tx=0).stop 2023-07-29T16:30:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 -- 172.21.15.31:0/2493103277 shutdown_connections 2023-07-29T16:30:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab095fcf0 0x558ab0959170 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab07ba6b0 0x558ab0966b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 --2- 172.21.15.31:0/2493103277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.532+0000 7fca76377b80 1 -- 172.21.15.31:0/2493103277 >> 172.21.15.31:0/2493103277 conn(0x558ab0953fd0 msgr2=0x558ab0958a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.533+0000 7fca76377b80 1 -- 172.21.15.31:0/2493103277 shutdown_connections 2023-07-29T16:30:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.533+0000 7fca76377b80 1 -- 172.21.15.31:0/2493103277 wait complete. 2023-07-29T16:30:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.533+0000 7fca76377b80 1 -- start start 2023-07-29T16:30:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.534+0000 7fca76377b80 1 Processor -- start 2023-07-29T16:30:41.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.534+0000 7fca76377b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab07ba6b0 0x558ab096ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.534+0000 7fca76377b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca76377b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca76377b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab096be50 con 0x558ab095fcf0 2023-07-29T16:30:41.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca76377b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab096bfc0 con 0x558ab095edf0 2023-07-29T16:30:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca76377b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ab096c180 con 0x558ab07ba6b0 2023-07-29T16:30:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca76377b80 10 client.0 _ll_register_callbacks cb 0x558ab08555c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca5b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca5b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37946/0 (socket says 172.21.15.31:37946) 2023-07-29T16:30:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.535+0000 7fca5b7fe700 1 -- 172.21.15.31:0/1275263667 learned_addr learned my addr 172.21.15.31:0/1275263667 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.536+0000 7fca5bfff700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab07ba6b0 0x558ab096ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.536+0000 7fca60f9c700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.536+0000 7fca5b7fe700 1 -- 172.21.15.31:0/1275263667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab07ba6b0 msgr2=0x558ab096ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.536+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab07ba6b0 0x558ab096ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.537+0000 7fca5b7fe700 1 -- 172.21.15.31:0/1275263667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 msgr2=0x558ab096b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.537+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.537+0000 7fca5b7fe700 1 -- 172.21.15.31:0/1275263667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca54007dd0 con 0x558ab095edf0 2023-07-29T16:30:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.537+0000 7fca5b7fe700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f220 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fca5000c2d0 tx=0x7fca50010ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea4db8fc802470f6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.537+0000 7fca3b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.538+0000 7fca3b7fe700 1 -- 172.21.15.31:0/1275263667 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca500ae020 con 0x558ab095edf0 2023-07-29T16:30:41.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.538+0000 7fca3b7fe700 1 -- 172.21.15.31:0/1275263667 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca500a1880 con 0x558ab095edf0 2023-07-29T16:30:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.538+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca4c01e040 con 0x558ab095edf0 2023-07-29T16:30:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.538+0000 7fca3b7fe700 1 -- 172.21.15.31:0/1275263667 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca500955d0 con 0x558ab095edf0 2023-07-29T16:30:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 10 client.29939 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:41.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x558ab06dc250 con 0x558ab095edf0 2023-07-29T16:30:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca3b7fe700 1 -- 172.21.15.31:0/1275263667 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca4c01e040 con 0x558ab095edf0 2023-07-29T16:30:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca3b7fe700 1 -- 172.21.15.31:0/1275263667 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7fca50080070 con 0x558ab095edf0 2023-07-29T16:30:41.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 10 client.29939 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 10 client.29939 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 20 client.29939 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab09c69f0 0x558ab09c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 4 client.29939 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:30:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.539+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558ab08a8ce0 con 0x558ab09c69f0 2023-07-29T16:30:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.540+0000 7fca5bfff700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab09c69f0 0x558ab09c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.540+0000 7fca5bfff700 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab09c69f0 0x558ab09c8dd0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0ae6ee0d52bf276 server_cookie=bef707891c465b52 in_seq=0 out_seq=0 2023-07-29T16:30:41.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.540+0000 7fca3b7fe700 10 client.29939 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 -- 172.21.15.31:0/1199192796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100410 msgr2=0x7f23ec100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 --2- 172.21.15.31:0/1199192796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100410 0x7f23ec100860 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009990 tx=0x7f23e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 -- 172.21.15.31:0/1199192796 shutdown_connections 2023-07-29T16:30:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 --2- 172.21.15.31:0/1199192796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23ec100da0 0x7f23ec107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 --2- 172.21.15.31:0/1199192796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec0ff220 0x7f23ec0ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 --2- 172.21.15.31:0/1199192796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100410 0x7f23ec100860 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 -- 172.21.15.31:0/1199192796 >> 172.21.15.31:0/1199192796 conn(0x7f23ec0fa840 msgr2=0x7f23ec0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 -- 172.21.15.31:0/1199192796 shutdown_connections 2023-07-29T16:30:41.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.866+0000 7f23f39ff700 1 -- 172.21.15.31:0/1199192796 wait complete. 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.867+0000 7f23f39ff700 1 Processor -- start 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.867+0000 7f23f39ff700 1 -- start start 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f39ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23ec0ff220 0x7f23ec06f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f39ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 0x7f23ec06fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f39ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100da0 0x7f23ec1a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f39ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec072bf0 con 0x7f23ec0ff220 2023-07-29T16:30:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f39ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec072d60 con 0x7f23ec100da0 2023-07-29T16:30:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f39ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec067eb0 con 0x7f23ec100410 2023-07-29T16:30:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 0x7f23ec06fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 0x7f23ec06fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59196/0 (socket says 172.21.15.31:59196) 2023-07-29T16:30:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 -- 172.21.15.31:0/3567964461 learned_addr learned my addr 172.21.15.31:0/3567964461 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100da0 0x7f23ec1a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 -- 172.21.15.31:0/3567964461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100da0 msgr2=0x7f23ec1a51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100da0 0x7f23ec1a51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 -- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23ec0ff220 msgr2=0x7f23ec06f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.868+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23ec0ff220 0x7f23ec06f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.870 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23f0f9a700 1 -- 172.21.15.31:0/3567964461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e80095f0 con 0x7f23ec100410 2023-07-29T16:30:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 0x7f23ec06fb10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f23e8005740 tx=0x7f23e8005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23de7fc700 1 -- 172.21.15.31:0/3567964461 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e801d070 con 0x7f23ec100410 2023-07-29T16:30:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23de7fc700 1 -- 172.21.15.31:0/3567964461 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e8022470 con 0x7f23ec100410 2023-07-29T16:30:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23de7fc700 1 -- 172.21.15.31:0/3567964461 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8017650 con 0x7f23ec100410 2023-07-29T16:30:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23ec068130 con 0x7f23ec100410 2023-07-29T16:30:41.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.869+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23ec0685b0 con 0x7f23ec100410 2023-07-29T16:30:41.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.874+0000 7f2407272b80 1 Processor -- start 2023-07-29T16:30:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.874+0000 7f2407272b80 1 -- start start 2023-07-29T16:30:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f2407272b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2b5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f2407272b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b5d40 0x562d0d2b6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f2407272b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b6b00 0x562d0d2b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f2407272b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d204a10 con 0x562d0d2b53f0 2023-07-29T16:30:41.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f2407272b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2023d0 con 0x562d0d2b6b00 2023-07-29T16:30:41.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f2407272b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d1fdcc0 con 0x562d0d2b5d40 2023-07-29T16:30:41.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f23f1f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f23f1f9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2b5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42602/0 (socket says 172.21.15.31:42602) 2023-07-29T16:30:41.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.875+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1283970995 learned_addr learned my addr 172.21.15.31:0/1283970995 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.876+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1283970995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b5d40 msgr2=0x562d0d2b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.876+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b5d40 0x562d0d2b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.876+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1283970995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b6b00 msgr2=0x562d0d2b6f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:30:41.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.876+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b6b00 0x562d0d2b6f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.876+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1283970995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d0d02bde0 con 0x562d0d2b53f0 2023-07-29T16:30:41.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.876+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2b5800 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a430 tx=0x7f23e4009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2f993c952ff708 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.877+0000 7f23d77fe700 1 -- 172.21.15.31:0/1283970995 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4010040 con 0x562d0d2b53f0 2023-07-29T16:30:41.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.877+0000 7f23d77fe700 1 -- 172.21.15.31:0/1283970995 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e4013a50 con 0x562d0d2b53f0 2023-07-29T16:30:41.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.877+0000 7f23d77fe700 1 -- 172.21.15.31:0/1283970995 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4014b90 con 0x562d0d2b53f0 2023-07-29T16:30:41.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 -- 172.21.15.31:0/1283970995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 msgr2=0x562d0d2b5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2b5800 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a430 tx=0x7f23e4009650 comp rx=0 tx=0).stop 2023-07-29T16:30:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 -- 172.21.15.31:0/1283970995 shutdown_connections 2023-07-29T16:30:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b6b00 0x562d0d2b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f23de7fc700 1 -- 172.21.15.31:0/3567964461 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f23e80177b0 con 0x7f23ec100410 2023-07-29T16:30:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2b5800 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 --2- 172.21.15.31:0/1283970995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b5d40 0x562d0d2b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 -- 172.21.15.31:0/1283970995 >> 172.21.15.31:0/1283970995 conn(0x562d0d2a23e0 msgr2=0x562d0d2a7990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.878+0000 7f2407272b80 1 -- 172.21.15.31:0/1283970995 shutdown_connections 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.879+0000 7f2407272b80 1 -- 172.21.15.31:0/1283970995 wait complete. 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.880+0000 7f2407272b80 1 -- start start 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.880+0000 7f2407272b80 1 Processor -- start 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2ae150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ae690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2b36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d19a510 con 0x562d0d2b53f0 2023-07-29T16:30:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d19a1b0 con 0x562d0d2b5d40 2023-07-29T16:30:41.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d081920 con 0x562d0d2b6b00 2023-07-29T16:30:41.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f2407272b80 10 client.0 _ll_register_callbacks cb 0x562d0d1a3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:41.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f23f0f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2b36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f23f0f9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2b36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59202/0 (socket says 172.21.15.31:59202) 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f23f0f9a700 1 -- 172.21.15.31:0/1835758690 learned_addr learned my addr 172.21.15.31:0/1835758690 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.881+0000 7f23f179b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1835758690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 msgr2=0x562d0d2b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2b36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 msgr2=0x562d0d2ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f1f9c700 1 -- 172.21.15.31:0/1835758690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e4009c60 con 0x562d0d2b53f0 2023-07-29T16:30:41.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.882+0000 7f23f179b700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:41.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.883+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2ae150 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a400 tx=0x7f23e4013f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde43f0f70429d13 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.883+0000 7f23d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:30:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.883+0000 7f23de7fc700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f23d80a07a0 0x7f23d80a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f23de7fc700 1 -- 172.21.15.31:0/3567964461 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f23e80ccdf0 con 0x7f23ec100410 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f23d4ff9700 1 -- 172.21.15.31:0/1835758690 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4010070 con 0x562d0d2b53f0 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f23d4ff9700 1 -- 172.21.15.31:0/1835758690 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f23e4006e40 con 0x562d0d2b53f0 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f23f179b700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f23d80a07a0 0x7f23d80a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d0d228d10 con 0x562d0d2b53f0 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f23d4ff9700 1 -- 172.21.15.31:0/1835758690 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23e4014930 con 0x562d0d2b53f0 2023-07-29T16:30:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f2407272b80 10 client.19803 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.884+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x562d0d014c60 con 0x562d0d2b53f0 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f23d4ff9700 1 -- 172.21.15.31:0/1835758690 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562d0d228d10 con 0x562d0d2b53f0 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f23d4ff9700 1 -- 172.21.15.31:0/1835758690 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f23e4024020 con 0x562d0d2b53f0 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f2407272b80 10 client.19803 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f2407272b80 10 client.19803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f2407272b80 20 client.19803 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:41.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f2407272b80 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d2d3480 0x562d0d2d5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f2407272b80 4 client.19803 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.885+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d0d27bd70 con 0x562d0d2d3480 2023-07-29T16:30:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.886+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d2d3480 0x562d0d2d5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.886+0000 7f23f179b700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f23d80a07a0 0x7f23d80a2c50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f23e0005950 tx=0x7f23e00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.886+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d2d3480 0x562d0d2d5890 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f8a727b569ca8fb server_cookie=f6fdead35493eb20 in_seq=0 out_seq=0 2023-07-29T16:30:41.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.886+0000 7f23d4ff9700 10 client.19803 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.934+0000 7f23d4ff9700 1 -- 172.21.15.31:0/1835758690 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562d0d27bd70 con 0x562d0d2d3480 2023-07-29T16:30:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.934+0000 7f23d4ff9700 10 client.19803 handle_command_reply: tid=0 2023-07-29T16:30:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.934+0000 7f2407272b80 1 client.19803 shutdown 2023-07-29T16:30:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.935+0000 7fca3b7fe700 1 -- 172.21.15.31:0/1275263667 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558ab08a8ce0 con 0x558ab09c69f0 2023-07-29T16:30:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.935+0000 7fca3b7fe700 10 client.29939 handle_command_reply: tid=0 2023-07-29T16:30:41.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.935+0000 7fca76377b80 1 client.29939 shutdown 2023-07-29T16:30:41.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.935+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 msgr2=0x562d0d2ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.935+0000 7f2407272b80 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2ae150 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a400 tx=0x7f23e4013f20 comp rx=0 tx=0).stop 2023-07-29T16:30:41.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 shutdown_connections 2023-07-29T16:30:41.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d2d3480 0x562d0d2d5890 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2b36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.936 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2ae150 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 --2- 172.21.15.31:0/1835758690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 >> 172.21.15.31:0/1835758690 conn(0x562d0d2a23e0 msgr2=0x562d0d2a7060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f23d4ff9700 0 client.19803 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 shutdown_connections 2023-07-29T16:30:41.937 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.936+0000 7f2407272b80 1 -- 172.21.15.31:0/1835758690 wait complete. 2023-07-29T16:30:41.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.937+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 msgr2=0x558ab096f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.937+0000 7fca76377b80 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f220 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fca5000c2d0 tx=0x7fca50010ab0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.940 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.937+0000 7f2407272b80 20 client.19803 trim_cache size 0 max 16384 2023-07-29T16:30:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 shutdown_connections 2023-07-29T16:30:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558ab09c69f0 0x558ab09c8dd0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558ab095fcf0 0x558ab096b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558ab07ba6b0 0x558ab096ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 --2- 172.21.15.31:0/1275263667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558ab095edf0 0x558ab096f220 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 >> 172.21.15.31:0/1275263667 conn(0x558ab0953fd0 msgr2=0x558ab0966090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca3b7fe700 0 client.29939 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 shutdown_connections 2023-07-29T16:30:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 1 -- 172.21.15.31:0/1275263667 wait complete. 2023-07-29T16:30:41.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.938+0000 7fca76377b80 20 client.29939 trim_cache size 0 max 16384 2023-07-29T16:30:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:30:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:30:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:30:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:30:41.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.939+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca440a0810 msgr2=0x7fca440a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.939+0000 7fca629ff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca440a0810 0x7fca440a2cc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00b2e0 tx=0x7fca4c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.939+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 msgr2=0x7fca5c19cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.939+0000 7fca629ff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 0x7fca5c19cc70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fca5000e9a0 tx=0x7fca5000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.941+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 shutdown_connections 2023-07-29T16:30:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.941+0000 7fca629ff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca440a0810 0x7fca440a2cc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.941+0000 7fca629ff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca5c103f10 0x7fca5c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.941+0000 7fca629ff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca5c0febf0 0x7fca5c19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.941+0000 7fca629ff700 1 --2- 172.21.15.31:0/3012574420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca5c101550 0x7fca5c19cc70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.941+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 >> 172.21.15.31:0/3012574420 conn(0x7fca5c0fa8a0 msgr2=0x7fca5c102250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.942+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 shutdown_connections 2023-07-29T16:30:41.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:41.942+0000 7fca629ff700 1 -- 172.21.15.31:0/3012574420 wait complete. 2023-07-29T16:30:41.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 Processor -- start 2023-07-29T16:30:41.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 -- start start 2023-07-29T16:30:41.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2af1a0 con 0x562d0d2b53f0 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2ad600 con 0x562d0d2b5d40 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.968+0000 7f2407272b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2a4460 con 0x562d0d2b6b00 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f0f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f0f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37978/0 (socket says 172.21.15.31:37978) 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59206/0 (socket says 172.21.15.31:59206) 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f0f9a700 1 -- 172.21.15.31:0/3500568720 learned_addr learned my addr 172.21.15.31:0/3500568720 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 -- 172.21.15.31:0/3500568720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 msgr2=0x562d0d2ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 -- 172.21.15.31:0/3500568720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 msgr2=0x562d0d2e2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.969+0000 7f23f1f9c700 1 -- 172.21.15.31:0/3500568720 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d0d052250 con 0x562d0d2b6b00 2023-07-29T16:30:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.970+0000 7f23f1f9c700 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2ace80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a430 tx=0x7f23e400aad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eabd1b429fb1153 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.970+0000 7f23d7fff700 1 -- 172.21.15.31:0/3500568720 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4010040 con 0x562d0d2b6b00 2023-07-29T16:30:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.970+0000 7f23d7fff700 1 -- 172.21.15.31:0/3500568720 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e4030cf0 con 0x562d0d2b6b00 2023-07-29T16:30:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.970+0000 7f23d7fff700 1 -- 172.21.15.31:0/3500568720 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e401d5f0 con 0x562d0d2b6b00 2023-07-29T16:30:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 -- 172.21.15.31:0/3500568720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 msgr2=0x562d0d2ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2ace80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f23e400a430 tx=0x7f23e400aad0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 -- 172.21.15.31:0/3500568720 shutdown_connections 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b6b00 0x562d0d2ace80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 --2- 172.21.15.31:0/3500568720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 -- 172.21.15.31:0/3500568720 >> 172.21.15.31:0/3500568720 conn(0x562d0d2a23e0 msgr2=0x562d0d2a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 -- 172.21.15.31:0/3500568720 shutdown_connections 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.971+0000 7f2407272b80 1 -- 172.21.15.31:0/3500568720 wait complete. 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.972+0000 7f2407272b80 1 -- start start 2023-07-29T16:30:41.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.972+0000 7f2407272b80 1 Processor -- start 2023-07-29T16:30:41.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2e0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b6b00 0x562d0d2e14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2e3180 con 0x562d0d2b6b00 2023-07-29T16:30:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2e19f0 con 0x562d0d2b5d40 2023-07-29T16:30:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d0d2e1bb0 con 0x562d0d2b53f0 2023-07-29T16:30:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f2407272b80 10 client.0 _ll_register_callbacks cb 0x562d0d1a3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59222/0 (socket says 172.21.15.31:59222) 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 -- 172.21.15.31:0/2689117257 learned_addr learned my addr 172.21.15.31:0/2689117257 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2e0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 -- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 msgr2=0x562d0d2e0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2e0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 -- 172.21.15.31:0/2689117257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b6b00 msgr2=0x562d0d2e14b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:30:41.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b6b00 0x562d0d2e14b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.973+0000 7f23f179b700 1 -- 172.21.15.31:0/2689117257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d0d014c60 con 0x562d0d2b53f0 2023-07-29T16:30:41.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.974+0000 7f23f179b700 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f23ec100270 tx=0x7f23e0004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d46179bf45858d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:41.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.974+0000 7f23d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:41.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.975+0000 7f23d4ff9700 1 -- 172.21.15.31:0/2689117257 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e0027460 con 0x562d0d2b53f0 2023-07-29T16:30:41.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.975+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23e4013a50 con 0x562d0d2b53f0 2023-07-29T16:30:41.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.975+0000 7f23d4ff9700 1 -- 172.21.15.31:0/2689117257 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f23e0018070 con 0x562d0d2b53f0 2023-07-29T16:30:41.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.975+0000 7f23d4ff9700 1 -- 172.21.15.31:0/2689117257 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23e00275c0 con 0x562d0d2b53f0 2023-07-29T16:30:41.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 10 client.29947 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:41.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x562d0d02a2b0 con 0x562d0d2b53f0 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f23d4ff9700 1 -- 172.21.15.31:0/2689117257 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e4013a50 con 0x562d0d2b53f0 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f23d4ff9700 1 -- 172.21.15.31:0/2689117257 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f23e00319a0 con 0x562d0d2b53f0 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 10 client.29947 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 10 client.29947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 20 client.29947 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d326b80 0x562d0d328f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:41.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 4 client.29947 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:30:41.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562d0d1f6d40 con 0x562d0d326b80 2023-07-29T16:30:41.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.976+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d326b80 0x562d0d328f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:41.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.977+0000 7f23d4ff9700 10 client.29947 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.977+0000 7f23f0f9a700 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d326b80 0x562d0d328f90 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a12d91a4e8b5f991 server_cookie=fdaae4af466fbf91 in_seq=0 out_seq=0 2023-07-29T16:30:41.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f23d4ff9700 1 -- 172.21.15.31:0/2689117257 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562d0d1f6d40 con 0x562d0d326b80 2023-07-29T16:30:41.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f23d4ff9700 10 client.29947 handle_command_reply: tid=0 2023-07-29T16:30:41.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 client.29947 shutdown 2023-07-29T16:30:41.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 msgr2=0x562d0d2e2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f23ec100270 tx=0x7f23e0004750 comp rx=0 tx=0).stop 2023-07-29T16:30:41.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 shutdown_connections 2023-07-29T16:30:41.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562d0d326b80 0x562d0d328f90 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562d0d2b6b00 0x562d0d2e14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562d0d2b53f0 0x562d0d2e2960 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 --2- 172.21.15.31:0/2689117257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d0d2b5d40 0x562d0d2e0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 >> 172.21.15.31:0/2689117257 conn(0x562d0d2a23e0 msgr2=0x562d0d2bcee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f23d4ff9700 0 client.29947 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.983+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 shutdown_connections 2023-07-29T16:30:41.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.984+0000 7f2407272b80 1 -- 172.21.15.31:0/2689117257 wait complete. 2023-07-29T16:30:41.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.984+0000 7f2407272b80 20 client.29947 trim_cache size 0 max 16384 2023-07-29T16:30:41.985 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:30:41.985 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:30:41.986 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:30:41.986 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:30:41.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.984+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f23d80a07a0 msgr2=0x7f23d80a2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.984+0000 7f23f39ff700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f23d80a07a0 0x7f23d80a2c50 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f23e0005950 tx=0x7f23e00058e0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 msgr2=0x7f23ec06fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 0x7f23ec06fb10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f23e8005740 tx=0x7f23e8005f10 comp rx=0 tx=0).stop 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 shutdown_connections 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f23d80a07a0 0x7f23d80a2c50 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23ec100da0 0x7f23ec1a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f23ec0ff220 0x7f23ec06f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 --2- 172.21.15.31:0/3567964461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23ec100410 0x7f23ec06fb10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:41.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 >> 172.21.15.31:0/3567964461 conn(0x7f23ec0fa840 msgr2=0x7f23ec0fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:41.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 shutdown_connections 2023-07-29T16:30:41.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:41.985+0000 7f23f39ff700 1 -- 172.21.15.31:0/3567964461 wait complete. 2023-07-29T16:30:42.009 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:30:42.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:30:42.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9127 25.29 MB/sec warmup 26 sec latency 330.067 ms 2023-07-29T16:30:42.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 -- 172.21.15.31:0/1724236611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 msgr2=0x7f5ea4108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 --2- 172.21.15.31:0/1724236611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 0x7f5ea4108430 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c00b0a0 tx=0x7f5e9c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 -- 172.21.15.31:0/1724236611 shutdown_connections 2023-07-29T16:30:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 --2- 172.21.15.31:0/1724236611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 0x7f5ea4108430 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 --2- 172.21.15.31:0/1724236611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea4100d90 0x7f5ea4103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 --2- 172.21.15.31:0/1724236611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea41036b0 0x7f5ea4105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.329+0000 7f5eabbff700 1 -- 172.21.15.31:0/1724236611 >> 172.21.15.31:0/1724236611 conn(0x7f5ea40fa810 msgr2=0x7f5ea40fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.330+0000 7f5eabbff700 1 -- 172.21.15.31:0/1724236611 shutdown_connections 2023-07-29T16:30:42.330 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.330+0000 7f5eabbff700 1 -- 172.21.15.31:0/1724236611 wait complete. 2023-07-29T16:30:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.331+0000 7f5eabbff700 1 Processor -- start 2023-07-29T16:30:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.331+0000 7f5eabbff700 1 -- start start 2023-07-29T16:30:42.331 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.331+0000 7f5eabbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 0x7f5ea4193e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.331+0000 7f5eabbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea41036b0 0x7f5ea4194340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.331+0000 7f5eabbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 0x7f5ea419b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5eabbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea419b980 con 0x7f5ea41036b0 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5eabbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea419baf0 con 0x7f5ea4105fd0 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5eabbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea419bcb0 con 0x7f5ea4100d90 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 0x7f5ea4193e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:42.332 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 0x7f5ea4193e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59252/0 (socket says 172.21.15.31:59252) 2023-07-29T16:30:42.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 -- 172.21.15.31:0/942904139 learned_addr learned my addr 172.21.15.31:0/942904139 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:42.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea919a700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea41036b0 0x7f5ea4194340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:42.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5eaa19c700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 0x7f5ea419b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 -- 172.21.15.31:0/942904139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 msgr2=0x7f5ea419b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 0x7f5ea419b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 -- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea41036b0 msgr2=0x7f5ea4194340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea41036b0 0x7f5ea4194340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.332+0000 7f5ea999b700 1 -- 172.21.15.31:0/942904139 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c009d00 con 0x7f5ea4100d90 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.333+0000 7f5ea999b700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 0x7f5ea4193e00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000d950 tx=0x7f5ea000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:42.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.333+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000f890 con 0x7f5ea4100d90 2023-07-29T16:30:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.333+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ea0010460 con 0x7f5ea4100d90 2023-07-29T16:30:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.333+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000e610 con 0x7f5ea4100d90 2023-07-29T16:30:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.333+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea419bf30 con 0x7f5ea4100d90 2023-07-29T16:30:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.333+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea419c3f0 con 0x7f5ea4100d90 2023-07-29T16:30:42.335 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.335+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea410bbb0 con 0x7f5ea4100d90 2023-07-29T16:30:42.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.337+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5ea00108d0 con 0x7f5ea4100d90 2023-07-29T16:30:42.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.339+0000 7f5e9affd700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e900a0860 0x7f5e900a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:42.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.339+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5ea00cb8a0 con 0x7f5ea4100d90 2023-07-29T16:30:42.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.340+0000 7f5ea919a700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e900a0860 0x7f5e900a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:42.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.341+0000 7f5ea919a700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e900a0860 0x7f5e900a2d10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5e94005950 tx=0x7f5e9400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:42.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.341+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5ea0095280 con 0x7f5ea4100d90 2023-07-29T16:30:42.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.749+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ea419dc70 con 0x7f5ea4100d90 2023-07-29T16:30:42.756 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:30:42.756 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:30:38.018321+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:30:42.757 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.756+0000 7f5e9affd700 1 -- 172.21.15.31:0/942904139 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+10783 (secure 0 0 0) 0x7f5ea0020020 con 0x7f5ea4100d90 2023-07-29T16:30:42.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.761+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e900a0860 msgr2=0x7f5e900a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:42.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.761+0000 7f5eabbff700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e900a0860 0x7f5e900a2d10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5e94005950 tx=0x7f5e9400b410 comp rx=0 tx=0).stop 2023-07-29T16:30:42.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.761+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 msgr2=0x7f5ea4193e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:42.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.761+0000 7f5eabbff700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 0x7f5ea4193e00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000d950 tx=0x7f5ea000dc60 comp rx=0 tx=0).stop 2023-07-29T16:30:42.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.764+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 shutdown_connections 2023-07-29T16:30:42.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.764+0000 7f5eabbff700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e900a0860 0x7f5e900a2d10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.764+0000 7f5eabbff700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5ea4105fd0 0x7f5ea419b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.764+0000 7f5eabbff700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5ea4100d90 0x7f5ea4193e00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.764+0000 7f5eabbff700 1 --2- 172.21.15.31:0/942904139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ea41036b0 0x7f5ea4194340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:42.765 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.764+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 >> 172.21.15.31:0/942904139 conn(0x7f5ea40fa810 msgr2=0x7f5ea40ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:42.766 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.767+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 shutdown_connections 2023-07-29T16:30:42.767 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:42.767+0000 7f5eabbff700 1 -- 172.21.15.31:0/942904139 wait complete. 2023-07-29T16:30:42.770 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 37 2023-07-29T16:30:42.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:30:42.890 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:42 smithi031 ceph-mon[95504]: pgmap v828: 97 pgs: 97 active+clean; 372 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 277 op/s 2023-07-29T16:30:43.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:42 smithi039 ceph-mon[128858]: pgmap v828: 97 pgs: 97 active+clean; 372 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 277 op/s 2023-07-29T16:30:43.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:42 smithi156 ceph-mon[129245]: pgmap v828: 97 pgs: 97 active+clean; 372 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.6 MiB/s wr, 277 op/s 2023-07-29T16:30:43.110 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.110+0000 7f89423ff700 1 -- 172.21.15.31:0/2573902427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c071e60 msgr2=0x7f893c1110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.110+0000 7f89423ff700 1 --2- 172.21.15.31:0/2573902427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c071e60 0x7f893c1110a0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f893400b0a0 tx=0x7f893400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 -- 172.21.15.31:0/2573902427 shutdown_connections 2023-07-29T16:30:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 --2- 172.21.15.31:0/2573902427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c071e60 0x7f893c1110a0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 --2- 172.21.15.31:0/2573902427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 0x7f893c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 --2- 172.21.15.31:0/2573902427 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893c0714b0 0x7f893c071920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 -- 172.21.15.31:0/2573902427 >> 172.21.15.31:0/2573902427 conn(0x7f893c06d2d0 msgr2=0x7f893c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 -- 172.21.15.31:0/2573902427 shutdown_connections 2023-07-29T16:30:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.111+0000 7f89423ff700 1 -- 172.21.15.31:0/2573902427 wait complete. 2023-07-29T16:30:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.112+0000 7f89423ff700 1 Processor -- start 2023-07-29T16:30:43.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.112+0000 7f89423ff700 1 -- start start 2023-07-29T16:30:43.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.113+0000 7f89423ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c0714b0 0x7f893c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.113+0000 7f89423ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893c071e60 0x7f893c1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.113 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.113+0000 7f89423ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 0x7f893c1a8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.113+0000 7f89423ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a8740 con 0x7f893c108260 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.113+0000 7f89423ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a8880 con 0x7f893c071e60 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.113+0000 7f89423ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c1a8a10 con 0x7f893c0714b0 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f894099c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 0x7f893c1a8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f893bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c0714b0 0x7f893c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f893bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c0714b0 0x7f893c1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59258/0 (socket says 172.21.15.31:59258) 2023-07-29T16:30:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f893bfff700 1 -- 172.21.15.31:0/2353340939 learned_addr learned my addr 172.21.15.31:0/2353340939 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f894099c700 1 -- 172.21.15.31:0/2353340939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c0714b0 msgr2=0x7f893c1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f894099c700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c0714b0 0x7f893c1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f894099c700 1 -- 172.21.15.31:0/2353340939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893c071e60 msgr2=0x7f893c1a1100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f894099c700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893c071e60 0x7f893c1a1100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.114+0000 7f894099c700 1 -- 172.21.15.31:0/2353340939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8934009d00 con 0x7f893c108260 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.115+0000 7f894099c700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 0x7f893c1a8170 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f893400b070 tx=0x7f8934004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.115+0000 7f89397fa700 1 -- 172.21.15.31:0/2353340939 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893400e040 con 0x7f893c108260 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.115+0000 7f89397fa700 1 -- 172.21.15.31:0/2353340939 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f893400ce90 con 0x7f893c108260 2023-07-29T16:30:43.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.115+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c1115e0 con 0x7f893c108260 2023-07-29T16:30:43.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.115+0000 7f89397fa700 1 -- 172.21.15.31:0/2353340939 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340057c0 con 0x7f893c108260 2023-07-29T16:30:43.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.115+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c111bc0 con 0x7f893c108260 2023-07-29T16:30:43.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.122+0000 7f89397fa700 1 -- 172.21.15.31:0/2353340939 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8934005920 con 0x7f893c108260 2023-07-29T16:30:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.122+0000 7f8955d4eb80 1 Processor -- start 2023-07-29T16:30:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.122+0000 7f8955d4eb80 1 -- start start 2023-07-29T16:30:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 0x55e8d3858ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3859530 0x55e8d3859d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d385a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d37b0a10 con 0x55e8d3859530 2023-07-29T16:30:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d37ae3d0 con 0x55e8d3858be0 2023-07-29T16:30:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d37a9cc0 con 0x55e8d385a2f0 2023-07-29T16:30:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 0x55e8d3858ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 0x55e8d3858ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38030/0 (socket says 172.21.15.31:38030) 2023-07-29T16:30:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f893b7fe700 1 -- 172.21.15.31:0/827551545 learned_addr learned my addr 172.21.15.31:0/827551545 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f893b7fe700 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d385a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.123+0000 7f893bfff700 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3859530 0x55e8d3859d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f893b7fe700 1 -- 172.21.15.31:0/827551545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 msgr2=0x55e8d385a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f893b7fe700 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d385a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f893b7fe700 1 -- 172.21.15.31:0/827551545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3859530 msgr2=0x55e8d3859d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f893b7fe700 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3859530 0x55e8d3859d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f893b7fe700 1 -- 172.21.15.31:0/827551545 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8d35d7de0 con 0x55e8d3858be0 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f893b7fe700 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 0x55e8d3858ff0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8930013780 tx=0x7f893000a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9bb57b4ce0c6926 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f89397fa700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89240a0860 0x7f89240a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.124+0000 7f89227fc700 1 -- 172.21.15.31:0/827551545 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930004220 con 0x55e8d3858be0 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.125+0000 7f89397fa700 1 -- 172.21.15.31:0/2353340939 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f89340cd5f0 con 0x7f893c108260 2023-07-29T16:30:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.125+0000 7f89227fc700 1 -- 172.21.15.31:0/827551545 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8930005ea0 con 0x55e8d3858be0 2023-07-29T16:30:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.125+0000 7f89227fc700 1 -- 172.21.15.31:0/827551545 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930010870 con 0x55e8d3858be0 2023-07-29T16:30:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 -- 172.21.15.31:0/827551545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 msgr2=0x55e8d3858ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 0x55e8d3858ff0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8930013780 tx=0x7f893000a250 comp rx=0 tx=0).stop 2023-07-29T16:30:43.127 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f893bfff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89240a0860 0x7f89240a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 -- 172.21.15.31:0/827551545 shutdown_connections 2023-07-29T16:30:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d385a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3858be0 0x55e8d3858ff0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/827551545 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3859530 0x55e8d3859d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.126+0000 7f8955d4eb80 1 -- 172.21.15.31:0/827551545 >> 172.21.15.31:0/827551545 conn(0x55e8d384e3e0 msgr2=0x55e8d3853990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.127+0000 7f8955d4eb80 1 -- 172.21.15.31:0/827551545 shutdown_connections 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.128+0000 7f893bfff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89240a0860 0x7f89240a2d10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x55e8d384e360 tx=0x7f892c009290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.128+0000 7f8955d4eb80 1 -- 172.21.15.31:0/827551545 wait complete. 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.128+0000 7f8955d4eb80 1 -- start start 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 Processor -- start 2023-07-29T16:30:43.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3862890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3862dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d3869e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d3746510 con 0x55e8d385a2f0 2023-07-29T16:30:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d37461b0 con 0x55e8d3859530 2023-07-29T16:30:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.129+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d362d920 con 0x55e8d3858be0 2023-07-29T16:30:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.130+0000 7f8955d4eb80 10 client.0 _ll_register_callbacks cb 0x55e8d374f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.130+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3862dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.130+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3862dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38044/0 (socket says 172.21.15.31:38044) 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.130+0000 7f893b7fe700 1 -- 172.21.15.31:0/3228710070 learned_addr learned my addr 172.21.15.31:0/3228710070 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.130+0000 7f894099c700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d3869e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f893b7fe700 1 -- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 msgr2=0x55e8d3862890 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f893b7fe700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3862890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f893b7fe700 1 -- 172.21.15.31:0/3228710070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 msgr2=0x55e8d3869e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f893b7fe700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d3869e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f893b7fe700 1 -- 172.21.15.31:0/3228710070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8930013460 con 0x55e8d3859530 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f894099c700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d3869e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f893b7fe700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3862dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8930003e10 tx=0x7f893000aaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d6e9707b9423729 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f891b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f891b7fe700 1 -- 172.21.15.31:0/3228710070 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930003b50 con 0x55e8d3859530 2023-07-29T16:30:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f891b7fe700 1 -- 172.21.15.31:0/3228710070 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8930013030 con 0x55e8d3859530 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.131+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e8d37d4d10 con 0x55e8d3859530 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.132+0000 7f891b7fe700 1 -- 172.21.15.31:0/3228710070 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89300131f0 con 0x55e8d3859530 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.132+0000 7f8955d4eb80 10 client.29966 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.132+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55e8d35c0c60 con 0x55e8d3859530 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.132+0000 7f891b7fe700 1 -- 172.21.15.31:0/3228710070 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e8d37d4d10 con 0x55e8d3859530 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f891b7fe700 1 -- 172.21.15.31:0/3228710070 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f893001d5e0 con 0x55e8d3859530 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f8955d4eb80 10 client.29966 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f8955d4eb80 10 client.29966 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:30:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f8955d4eb80 20 client.29966 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d386e4d0 0x55e8d38708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f8955d4eb80 4 client.29966 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:43.136 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.133+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e8d3827d70 con 0x55e8d386e4d0 2023-07-29T16:30:43.137 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.134+0000 7f893bfff700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d386e4d0 0x55e8d38708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.135+0000 7f893bfff700 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d386e4d0 0x55e8d38708e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f802d24e2272c6ba server_cookie=f180312b683b42ee in_seq=0 out_seq=0 2023-07-29T16:30:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.135+0000 7f891b7fe700 10 client.29966 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.135+0000 7f891b7fe700 1 -- 172.21.15.31:0/3228710070 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e8d3827d70 con 0x55e8d386e4d0 2023-07-29T16:30:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.135+0000 7f891b7fe700 10 client.29966 handle_command_reply: tid=0 2023-07-29T16:30:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 client.29966 shutdown 2023-07-29T16:30:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 msgr2=0x55e8d3862dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3862dd0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8930003e10 tx=0x7f893000aaa0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 shutdown_connections 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d3869e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3862890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d386e4d0 0x55e8d38708e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/3228710070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3862dd0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.136+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 >> 172.21.15.31:0/3228710070 conn(0x55e8d384e3e0 msgr2=0x55e8d3853060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.137+0000 7f891b7fe700 0 client.29966 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.137+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 shutdown_connections 2023-07-29T16:30:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.137+0000 7f8955d4eb80 1 -- 172.21.15.31:0/3228710070 wait complete. 2023-07-29T16:30:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.137+0000 7f8955d4eb80 20 client.29966 trim_cache size 0 max 16384 2023-07-29T16:30:43.160 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 Processor -- start 2023-07-29T16:30:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 -- start start 2023-07-29T16:30:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3867dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3860710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d385d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.161 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d3859bc0 con 0x55e8d385a2f0 2023-07-29T16:30:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d3861b60 con 0x55e8d3859530 2023-07-29T16:30:43.162 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.161+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d38604c0 con 0x55e8d3858be0 2023-07-29T16:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3867dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3860710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3867dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59270/0 (socket says 172.21.15.31:59270) 2023-07-29T16:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893bfff700 1 -- 172.21.15.31:0/1156015989 learned_addr learned my addr 172.21.15.31:0/1156015989 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.163 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f894099c700 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d385d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 -- 172.21.15.31:0/1156015989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 msgr2=0x55e8d3867dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3867dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 -- 172.21.15.31:0/1156015989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 msgr2=0x55e8d385d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d385d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 -- 172.21.15.31:0/1156015989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8d35fe250 con 0x55e8d3859530 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.162+0000 7f893b7fe700 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3860710 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f8930001eb0 tx=0x7f8930000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e536910dae0393a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.164 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.163+0000 7f8922ffd700 1 -- 172.21.15.31:0/1156015989 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930003e40 con 0x55e8d3859530 2023-07-29T16:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.163+0000 7f8922ffd700 1 -- 172.21.15.31:0/1156015989 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f893001dc00 con 0x55e8d3859530 2023-07-29T16:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.163+0000 7f8922ffd700 1 -- 172.21.15.31:0/1156015989 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930003950 con 0x55e8d3859530 2023-07-29T16:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 -- 172.21.15.31:0/1156015989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 msgr2=0x55e8d3860710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.165 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3860710 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f8930001eb0 tx=0x7f8930000bc0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 -- 172.21.15.31:0/1156015989 shutdown_connections 2023-07-29T16:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d385a2f0 0x55e8d385d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d3858be0 0x55e8d3867dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.166 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/1156015989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3860710 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 -- 172.21.15.31:0/1156015989 >> 172.21.15.31:0/1156015989 conn(0x55e8d384e3e0 msgr2=0x55e8d38704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 -- 172.21.15.31:0/1156015989 shutdown_connections 2023-07-29T16:30:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 -- 172.21.15.31:0/1156015989 wait complete. 2023-07-29T16:30:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.164+0000 7f8955d4eb80 1 -- start start 2023-07-29T16:30:43.167 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.165+0000 7f8955d4eb80 1 Processor -- start 2023-07-29T16:30:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3858be0 0x55e8d38656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3865be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d3866bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.168 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d3867130 con 0x55e8d3858be0 2023-07-29T16:30:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d38672a0 con 0x55e8d3859530 2023-07-29T16:30:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e8d3867460 con 0x55e8d385a2f0 2023-07-29T16:30:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.167+0000 7f8955d4eb80 10 client.0 _ll_register_callbacks cb 0x55e8d374f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3865be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3865be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38064/0 (socket says 172.21.15.31:38064) 2023-07-29T16:30:43.169 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f893b7fe700 1 -- 172.21.15.31:0/113662835 learned_addr learned my addr 172.21.15.31:0/113662835 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f894099c700 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d3866bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f894099c700 1 -- 172.21.15.31:0/113662835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 msgr2=0x55e8d3865be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f894099c700 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3865be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f894099c700 1 -- 172.21.15.31:0/113662835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3858be0 msgr2=0x55e8d38656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f894099c700 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3858be0 0x55e8d38656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.168+0000 7f894099c700 1 -- 172.21.15.31:0/113662835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e8d35c0c60 con 0x55e8d385a2f0 2023-07-29T16:30:43.170 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.169+0000 7f894099c700 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d3866bf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f893400b0a0 tx=0x7f8934006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=256235bc0fe4eff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.169+0000 7f891b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.169+0000 7f891b7fe700 1 -- 172.21.15.31:0/113662835 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893409edb0 con 0x55e8d385a2f0 2023-07-29T16:30:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.169+0000 7f891b7fe700 1 -- 172.21.15.31:0/113662835 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f89340a3140 con 0x55e8d385a2f0 2023-07-29T16:30:43.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.169+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8930005ea0 con 0x55e8d385a2f0 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.169+0000 7f891b7fe700 1 -- 172.21.15.31:0/113662835 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8934099120 con 0x55e8d385a2f0 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.170+0000 7f8955d4eb80 10 client.29968 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.170+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55e8d35d62b0 con 0x55e8d385a2f0 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.170+0000 7f891b7fe700 1 -- 172.21.15.31:0/113662835 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8930005ea0 con 0x55e8d385a2f0 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f891b7fe700 1 -- 172.21.15.31:0/113662835 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f89340a2cb0 con 0x55e8d385a2f0 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f8955d4eb80 10 client.29968 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:43.173 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f8955d4eb80 10 client.29968 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:30:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f8955d4eb80 20 client.29968 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d38c08c0 0x55e8d38c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f8955d4eb80 4 client.29968 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:30:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e8d37a2d40 con 0x55e8d38c08c0 2023-07-29T16:30:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.171+0000 7f893bfff700 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d38c08c0 0x55e8d38c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.174 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f893bfff700 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d38c08c0 0x55e8d38c2cd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d95bfab6e5eec8b server_cookie=f54de94ac6ea9e70 in_seq=0 out_seq=0 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f891b7fe700 10 client.29968 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f891b7fe700 1 -- 172.21.15.31:0/113662835 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e8d37a2d40 con 0x55e8d38c08c0 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f891b7fe700 10 client.29968 handle_command_reply: tid=0 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f8955d4eb80 1 client.29968 shutdown 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 msgr2=0x55e8d3866bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.172+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d3866bf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f893400b0a0 tx=0x7f8934006000 comp rx=0 tx=0).stop 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 shutdown_connections 2023-07-29T16:30:43.175 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e8d385a2f0 0x55e8d3866bf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8d3858be0 0x55e8d38656a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55e8d38c08c0 0x55e8d38c2cd0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 --2- 172.21.15.31:0/113662835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e8d3859530 0x55e8d3865be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 >> 172.21.15.31:0/113662835 conn(0x55e8d384e3e0 msgr2=0x55e8d3860710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f891b7fe700 0 client.29968 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:30:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 shutdown_connections 2023-07-29T16:30:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.173+0000 7f8955d4eb80 1 -- 172.21.15.31:0/113662835 wait complete. 2023-07-29T16:30:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.174+0000 7f8955d4eb80 20 client.29968 trim_cache size 0 max 16384 2023-07-29T16:30:43.178 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:30:43.178 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.174+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89240a0860 msgr2=0x7f89240a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.174+0000 7f89423ff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89240a0860 0x7f89240a2d10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x55e8d384e360 tx=0x7f892c009290 comp rx=0 tx=0).stop 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.174+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 msgr2=0x7f893c1a8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.174+0000 7f89423ff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 0x7f893c1a8170 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f893400b070 tx=0x7f8934004c80 comp rx=0 tx=0).stop 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.175+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 shutdown_connections 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.175+0000 7f89423ff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f89240a0860 0x7f89240a2d10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.175+0000 7f89423ff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f893c108260 0x7f893c1a8170 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.175+0000 7f89423ff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f893c0714b0 0x7f893c1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.175+0000 7f89423ff700 1 --2- 172.21.15.31:0/2353340939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f893c071e60 0x7f893c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.175+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 >> 172.21.15.31:0/2353340939 conn(0x7f893c06d2d0 msgr2=0x7f893c070510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.177+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 shutdown_connections 2023-07-29T16:30:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.177+0000 7f89423ff700 1 -- 172.21.15.31:0/2353340939 wait complete. 2023-07-29T16:30:43.208 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:30:43.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 9841 26.24 MB/sec warmup 27 sec latency 78.086 ms 2023-07-29T16:30:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 -- 172.21.15.31:0/3037889332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba01014b0 msgr2=0x7f6ba0103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/3037889332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba01014b0 0x7f6ba0103930 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009990 tx=0x7f6b94009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 -- 172.21.15.31:0/3037889332 shutdown_connections 2023-07-29T16:30:43.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/3037889332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103e70 0x7f6ba01042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/3037889332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 0x7f6ba0100f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/3037889332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba01014b0 0x7f6ba0103930 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.505+0000 7f6ba4bff700 1 -- 172.21.15.31:0/3037889332 >> 172.21.15.31:0/3037889332 conn(0x7f6ba00fa7e0 msgr2=0x7f6ba00fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.506+0000 7f6ba4bff700 1 -- 172.21.15.31:0/3037889332 shutdown_connections 2023-07-29T16:30:43.506 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.506+0000 7f6ba4bff700 1 -- 172.21.15.31:0/3037889332 wait complete. 2023-07-29T16:30:43.507 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.507+0000 7f6ba4bff700 1 Processor -- start 2023-07-29T16:30:43.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.507+0000 7f6ba4bff700 1 -- start start 2023-07-29T16:30:43.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6ba4bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 0x7f6ba019c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6ba4bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba01014b0 0x7f6ba019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6ba4bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba0103e70 0x7f6ba01a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6ba4bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba01a41b0 con 0x7f6ba01014b0 2023-07-29T16:30:43.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6ba4bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba006a770 con 0x7f6ba0103e70 2023-07-29T16:30:43.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6ba4bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba006a900 con 0x7f6ba00feb50 2023-07-29T16:30:43.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba01014b0 0x7f6ba019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6b9ed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba0103e70 0x7f6ba01a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6b9e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 0x7f6ba019c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba01014b0 0x7f6ba019cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42716/0 (socket says 172.21.15.31:42716) 2023-07-29T16:30:43.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.508+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/926532677 learned_addr learned my addr 172.21.15.31:0/926532677 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9ed9d700 1 -- 172.21.15.31:0/926532677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 msgr2=0x7f6ba019c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9ed9d700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 0x7f6ba019c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9ed9d700 1 -- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba01014b0 msgr2=0x7f6ba019cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9ed9d700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba01014b0 0x7f6ba019cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9ed9d700 1 -- 172.21.15.31:0/926532677 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b940095f0 con 0x7f6ba0103e70 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 0x7f6ba019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.509+0000 7f6b9ed9d700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba0103e70 0x7f6ba01a3be0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800efc0 tx=0x7f6b9800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.510+0000 7f6b8f7fe700 1 -- 172.21.15.31:0/926532677 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b98018070 con 0x7f6ba0103e70 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.510+0000 7f6b8f7fe700 1 -- 172.21.15.31:0/926532677 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b9800f040 con 0x7f6ba0103e70 2023-07-29T16:30:43.511 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.510+0000 7f6b8f7fe700 1 -- 172.21.15.31:0/926532677 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9800cd40 con 0x7f6ba0103e70 2023-07-29T16:30:43.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.510+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba006ab80 con 0x7f6ba0103e70 2023-07-29T16:30:43.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.510+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba006b040 con 0x7f6ba0103e70 2023-07-29T16:30:43.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.516+0000 7f6b8f7fe700 1 -- 172.21.15.31:0/926532677 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6b98003a00 con 0x7f6ba0103e70 2023-07-29T16:30:43.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.516+0000 7f6bb859eb80 1 Processor -- start 2023-07-29T16:30:43.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 -- start start 2023-07-29T16:30:43.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419efcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efd530 0x556419efdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 0x556419efe740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419e54a10 con 0x556419efcbe0 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419e523d0 con 0x556419efd530 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.518+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419e4dcc0 con 0x556419efe2f0 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 0x556419efe740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 0x556419efe740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59304/0 (socket says 172.21.15.31:59304) 2023-07-29T16:30:43.519 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3510519206 learned_addr learned my addr 172.21.15.31:0/3510519206 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b8f7fe700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6b880a0890 0x7f6b880a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3510519206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efd530 msgr2=0x556419efdd60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:30:43.520 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efd530 0x556419efdd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3510519206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 msgr2=0x556419efcff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419efcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3510519206 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556419c7bde0 con 0x556419efe2f0 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.519+0000 7f6b8f7fe700 1 -- 172.21.15.31:0/926532677 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6b980cb660 con 0x7f6ba0103e70 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.520+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 0x556419efe740 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6b94005ef0 tx=0x7f6b94005510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fcd398395c3498f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.520+0000 7f6b9ed9d700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6b880a0890 0x7f6b880a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.520+0000 7f6b8cff9700 1 -- 172.21.15.31:0/3510519206 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400b7b0 con 0x556419efe2f0 2023-07-29T16:30:43.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.521+0000 7f6b8cff9700 1 -- 172.21.15.31:0/3510519206 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b9400bdd0 con 0x556419efe2f0 2023-07-29T16:30:43.522 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.521+0000 7f6b8cff9700 1 -- 172.21.15.31:0/3510519206 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400f460 con 0x556419efe2f0 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.521+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3510519206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 msgr2=0x556419efe740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.521+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 0x556419efe740 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6b94005ef0 tx=0x7f6b94005510 comp rx=0 tx=0).stop 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.521+0000 7f6b9ed9d700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6b880a0890 0x7f6b880a2d40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009fd0 tx=0x7f6b980092e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.522+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3510519206 shutdown_connections 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.522+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efe2f0 0x556419efe740 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.522+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419efcff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.522+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3510519206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efd530 0x556419efdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.522+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3510519206 >> 172.21.15.31:0/3510519206 conn(0x556419ef23e0 msgr2=0x556419ef7990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.523+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3510519206 shutdown_connections 2023-07-29T16:30:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.523+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3510519206 wait complete. 2023-07-29T16:30:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.523+0000 7f6bb859eb80 1 -- start start 2023-07-29T16:30:43.524 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.523+0000 7f6bb859eb80 1 Processor -- start 2023-07-29T16:30:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 0x556419f06800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efd530 0x556419f06d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419dea510 con 0x556419efd530 2023-07-29T16:30:43.526 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419dea1b0 con 0x556419efe2f0 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419cd1920 con 0x556419efcbe0 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6bb859eb80 10 client.0 _ll_register_callbacks cb 0x556419df3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 0x556419f06800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 0x556419f06800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59314/0 (socket says 172.21.15.31:59314) 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/355256078 learned_addr learned my addr 172.21.15.31:0/355256078 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.524+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/355256078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 msgr2=0x556419f0dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efd530 msgr2=0x556419f06d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efd530 0x556419f06d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/355256078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b94020040 con 0x556419efcbe0 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 0x556419f06800 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6b94003990 tx=0x7f6b94005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55d07767fb5e1aaf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b85ffb700 1 -- 172.21.15.31:0/355256078 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400f460 con 0x556419efcbe0 2023-07-29T16:30:43.528 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.525+0000 7f6b85ffb700 1 -- 172.21.15.31:0/355256078 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6b9400f5c0 con 0x556419efcbe0 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.526+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556419e78d10 con 0x556419efcbe0 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.526+0000 7f6b85ffb700 1 -- 172.21.15.31:0/355256078 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b9400f780 con 0x556419efcbe0 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.526+0000 7f6bb859eb80 10 client.29980 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.526+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x556419c64c60 con 0x556419efcbe0 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.526+0000 7f6b85ffb700 1 -- 172.21.15.31:0/355256078 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556419e78d10 con 0x556419efcbe0 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.526+0000 7f6b85ffb700 1 -- 172.21.15.31:0/355256078 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f6b9402b020 con 0x556419efcbe0 2023-07-29T16:30:43.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6bb859eb80 10 client.29980 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6bb859eb80 10 client.29980 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:30:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6bb859eb80 20 client.29980 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f11fb0 0x556419f143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6bb859eb80 4 client.29980 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:43.530 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556419ecbd70 con 0x556419f11fb0 2023-07-29T16:30:43.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.527+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f11fb0 0x556419f143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.528+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f11fb0 0x556419f143c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c19c572377cb7033 server_cookie=6815eddf4f119278 in_seq=0 out_seq=0 2023-07-29T16:30:43.531 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.528+0000 7f6b85ffb700 10 client.29980 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.530+0000 7f6b85ffb700 1 -- 172.21.15.31:0/355256078 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x556419ecbd70 con 0x556419f11fb0 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.530+0000 7f6b85ffb700 10 client.29980 handle_command_reply: tid=0 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 client.29980 shutdown 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 msgr2=0x556419f06800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 0x556419f06800 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6b94003990 tx=0x7f6b94005a00 comp rx=0 tx=0).stop 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 shutdown_connections 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f11fb0 0x556419f143c0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.532 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efcbe0 0x556419f06800 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/355256078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efd530 0x556419f06d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 >> 172.21.15.31:0/355256078 conn(0x556419ef23e0 msgr2=0x556419ef7060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6b85ffb700 0 client.29980 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.531+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 shutdown_connections 2023-07-29T16:30:43.533 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.532+0000 7f6bb859eb80 1 -- 172.21.15.31:0/355256078 wait complete. 2023-07-29T16:30:43.534 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.532+0000 7f6bb859eb80 20 client.29980 trim_cache size 0 max 16384 2023-07-29T16:30:43.552 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.553+0000 7f6bb859eb80 1 Processor -- start 2023-07-29T16:30:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.553+0000 7f6bb859eb80 1 -- start start 2023-07-29T16:30:43.553 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.553+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f0bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.553+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419efe6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f054b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419ef4460 con 0x556419efcbe0 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419f07700 con 0x556419efe2f0 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419ef9380 con 0x556419efd530 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f0bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.554 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6b9dd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f0bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42730/0 (socket says 172.21.15.31:42730) 2023-07-29T16:30:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3748007352 learned_addr learned my addr 172.21.15.31:0/3748007352 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419efe6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.555 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.554+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3748007352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 msgr2=0x556419efe6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419efe6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3748007352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 msgr2=0x556419f0bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f0bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b9dd9b700 1 -- 172.21.15.31:0/3748007352 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556419ca2250 con 0x556419efe2f0 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f054b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6b94003960 tx=0x7f6b9400b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab64d066a1cb6e20 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.556 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.555+0000 7f6b8d7fa700 1 -- 172.21.15.31:0/3748007352 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94005bc0 con 0x556419efe2f0 2023-07-29T16:30:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.556+0000 7f6b8d7fa700 1 -- 172.21.15.31:0/3748007352 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b940039c0 con 0x556419efe2f0 2023-07-29T16:30:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.556+0000 7f6b8d7fa700 1 -- 172.21.15.31:0/3748007352 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94017b70 con 0x556419efe2f0 2023-07-29T16:30:43.557 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.556+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3748007352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 msgr2=0x556419f054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.556+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f054b0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f6b94003960 tx=0x7f6b9400b260 comp rx=0 tx=0).stop 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3748007352 shutdown_connections 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f054b0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f0bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/3748007352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419efe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3748007352 >> 172.21.15.31:0/3748007352 conn(0x556419ef23e0 msgr2=0x556419ef7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.558 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3748007352 shutdown_connections 2023-07-29T16:30:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.557+0000 7f6bb859eb80 1 -- 172.21.15.31:0/3748007352 wait complete. 2023-07-29T16:30:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.558+0000 7f6bb859eb80 1 -- start start 2023-07-29T16:30:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.558+0000 7f6bb859eb80 1 Processor -- start 2023-07-29T16:30:43.559 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f09f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419f0a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419f0b3f0 con 0x556419efcbe0 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419f0b560 con 0x556419efe2f0 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x556419f0b720 con 0x556419efd530 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.559+0000 7f6bb859eb80 10 client.0 _ll_register_callbacks cb 0x556419df3620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.560+0000 7f6b9e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419f0a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.560+0000 7f6b9e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419f0a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59332/0 (socket says 172.21.15.31:59332) 2023-07-29T16:30:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.560+0000 7f6b9e59c700 1 -- 172.21.15.31:0/994786326 learned_addr learned my addr 172.21.15.31:0/994786326 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.560+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.560+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f09f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b9e59c700 1 -- 172.21.15.31:0/994786326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 msgr2=0x556419f0aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b9e59c700 1 -- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 msgr2=0x556419f09f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f09f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b9e59c700 1 -- 172.21.15.31:0/994786326 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556419c64c60 con 0x556419efd530 2023-07-29T16:30:43.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b9e59c700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419f0a4b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x556419efd940 tx=0x7f6b90019d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b0b8901b8809bf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.561+0000 7f6b85ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.562+0000 7f6b85ffb700 1 -- 172.21.15.31:0/994786326 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90016040 con 0x556419efd530 2023-07-29T16:30:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.562+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b9400bdd0 con 0x556419efd530 2023-07-29T16:30:43.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.562+0000 7f6b85ffb700 1 -- 172.21.15.31:0/994786326 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6b90004020 con 0x556419efd530 2023-07-29T16:30:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6b85ffb700 1 -- 172.21.15.31:0/994786326 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b90019490 con 0x556419efd530 2023-07-29T16:30:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 10 client.29986 fetch_fsmap learned FSMap version 37 2023-07-29T16:30:43.564 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x556419c7a2b0 con 0x556419efd530 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6b85ffb700 1 -- 172.21.15.31:0/994786326 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b9400bdd0 con 0x556419efd530 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6b85ffb700 1 -- 172.21.15.31:0/994786326 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 37) v1 ==== 7466+0+0 (secure 0 0 0) 0x7f6b90005140 con 0x556419efd530 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 10 client.29986 fetch_fsmap finished waiting for FSMap version 37 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 10 client.29986 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 20 client.29986 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f64960 0x556419f66d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 4 client.29986 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:30:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.563+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556419e46d40 con 0x556419f64960 2023-07-29T16:30:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.564+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f64960 0x556419f66d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.564+0000 7f6b85ffb700 10 client.29986 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:43.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.564+0000 7f6b9dd9b700 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f64960 0x556419f66d70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f21f4623fdf27ce6 server_cookie=6dcc92fdec5c304b in_seq=0 out_seq=0 2023-07-29T16:30:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6b85ffb700 1 -- 172.21.15.31:0/994786326 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556419e46d40 con 0x556419f64960 2023-07-29T16:30:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6b85ffb700 10 client.29986 handle_command_reply: tid=0 2023-07-29T16:30:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 client.29986 shutdown 2023-07-29T16:30:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 msgr2=0x556419f0a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419f0a4b0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x556419efd940 tx=0x7f6b90019d00 comp rx=0 tx=0).stop 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 shutdown_connections 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x556419f64960 0x556419f66d70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556419efe2f0 0x556419f0aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556419efcbe0 0x556419f09f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6b85ffb700 0 client.29986 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 --2- 172.21.15.31:0/994786326 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556419efd530 0x556419f0a4b0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.567+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 >> 172.21.15.31:0/994786326 conn(0x556419ef23e0 msgr2=0x556419f054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 shutdown_connections 2023-07-29T16:30:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6bb859eb80 1 -- 172.21.15.31:0/994786326 wait complete. 2023-07-29T16:30:43.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6bb859eb80 20 client.29986 trim_cache size 0 max 16384 2023-07-29T16:30:43.569 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:30:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6b880a0890 msgr2=0x7f6b880a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6b880a0890 0x7f6b880a2d40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009fd0 tx=0x7f6b980092e0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba0103e70 msgr2=0x7f6ba01a3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.568+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba0103e70 0x7f6ba01a3be0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f6b9800efc0 tx=0x7f6b9800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 shutdown_connections 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f6b880a0890 0x7f6b880a2d40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ba0103e70 0x7f6ba01a3be0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ba00feb50 0x7f6ba019c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 --2- 172.21.15.31:0/926532677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba01014b0 0x7f6ba019cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 >> 172.21.15.31:0/926532677 conn(0x7f6ba00fa7e0 msgr2=0x7f6ba00ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 shutdown_connections 2023-07-29T16:30:43.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.569+0000 7f6ba4bff700 1 -- 172.21.15.31:0/926532677 wait complete. 2023-07-29T16:30:43.594 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:30:43.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:43 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/942904139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:30:43.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 -- 172.21.15.31:0/2006484416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 msgr2=0x7fa67c0ff630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2006484416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c0ff630 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fa66c01a9a0 tx=0x7fa66c01acb0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 -- 172.21.15.31:0/2006484416 shutdown_connections 2023-07-29T16:30:43.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2006484416 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa67c100da0 0x7fa67c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2006484416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c0ff630 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2006484416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa67c100410 0x7fa67c100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 -- 172.21.15.31:0/2006484416 >> 172.21.15.31:0/2006484416 conn(0x7fa67c0fa840 msgr2=0x7fa67c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.966 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.965+0000 7fa6833ff700 1 -- 172.21.15.31:0/2006484416 shutdown_connections 2023-07-29T16:30:43.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.966+0000 7fa6833ff700 1 -- 172.21.15.31:0/2006484416 wait complete. 2023-07-29T16:30:43.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.967+0000 7fa6833ff700 1 Processor -- start 2023-07-29T16:30:43.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.967+0000 7fa6833ff700 1 -- start start 2023-07-29T16:30:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa6833ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c106170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa6833ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa67c100410 0x7fa67c1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa6833ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa67c100da0 0x7fa67c192f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa6833ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c106f30 con 0x7fa67c100410 2023-07-29T16:30:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa6833ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c193460 con 0x7fa67c0ff220 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa6833ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c193620 con 0x7fa67c100da0 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c106170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c106170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38136/0 (socket says 172.21.15.31:38136) 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 -- 172.21.15.31:0/2469463351 learned_addr learned my addr 172.21.15.31:0/2469463351 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68199c700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa67c100da0 0x7fa67c192f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68099a700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa67c100410 0x7fa67c1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 -- 172.21.15.31:0/2469463351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa67c100da0 msgr2=0x7fa67c192f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa67c100da0 0x7fa67c192f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 -- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa67c100410 msgr2=0x7fa67c1066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa67c100410 0x7fa67c1066b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.968+0000 7fa68119b700 1 -- 172.21.15.31:0/2469463351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c01a600 con 0x7fa67c0ff220 2023-07-29T16:30:43.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.969+0000 7fa68119b700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c106170 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa66c003790 tx=0x7fa66c01cec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.969+0000 7fa6727fc700 1 -- 172.21.15.31:0/2469463351 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c02d070 con 0x7fa67c0ff220 2023-07-29T16:30:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.969+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c1938a0 con 0x7fa67c0ff220 2023-07-29T16:30:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.969+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c193da0 con 0x7fa67c0ff220 2023-07-29T16:30:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.969+0000 7fa6727fc700 1 -- 172.21.15.31:0/2469463351 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa66c031410 con 0x7fa67c0ff220 2023-07-29T16:30:43.971 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.969+0000 7fa6727fc700 1 -- 172.21.15.31:0/2469463351 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0283f0 con 0x7fa67c0ff220 2023-07-29T16:30:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.975+0000 7fa696c85b80 1 Processor -- start 2023-07-29T16:30:43.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.975+0000 7fa696c85b80 1 -- start start 2023-07-29T16:30:43.975 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa6727fc700 1 -- 172.21.15.31:0/2469463351 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa66c003b70 con 0x7fa67c0ff220 2023-07-29T16:30:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f31ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f32d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f33740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa696c85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838e89a10 con 0x555838f32530 2023-07-29T16:30:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa696c85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838e873d0 con 0x555838f332f0 2023-07-29T16:30:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa696c85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838e82cc0 con 0x555838f31be0 2023-07-29T16:30:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa68099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f33740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa68099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f33740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38152/0 (socket says 172.21.15.31:38152) 2023-07-29T16:30:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.976+0000 7fa68099a700 1 -- 172.21.15.31:0/273258681 learned_addr learned my addr 172.21.15.31:0/273258681 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68099a700 1 -- 172.21.15.31:0/273258681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 msgr2=0x555838f31ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68099a700 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f31ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68119b700 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f32d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68099a700 1 -- 172.21.15.31:0/273258681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 msgr2=0x555838f32d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68099a700 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f32d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68099a700 1 -- 172.21.15.31:0/273258681 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555838cb0de0 con 0x555838f332f0 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68119b700 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f32d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:43.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.977+0000 7fa68099a700 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f33740 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa678009f50 tx=0x7fa678009f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93db1647d9bdac05 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.978+0000 7fa6677fe700 1 -- 172.21.15.31:0/273258681 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678011070 con 0x555838f332f0 2023-07-29T16:30:43.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.978+0000 7fa6677fe700 1 -- 172.21.15.31:0/273258681 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa678015410 con 0x555838f332f0 2023-07-29T16:30:43.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.978+0000 7fa6677fe700 1 -- 172.21.15.31:0/273258681 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67800a660 con 0x555838f332f0 2023-07-29T16:30:43.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.978+0000 7fa6727fc700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6680a0810 0x7fa6680a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa6727fc700 1 -- 172.21.15.31:0/2469463351 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa66c0dd240 con 0x7fa67c0ff220 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 -- 172.21.15.31:0/273258681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 msgr2=0x555838f33740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f33740 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa678009f50 tx=0x7fa678009f80 comp rx=0 tx=0).stop 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 -- 172.21.15.31:0/273258681 shutdown_connections 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f33740 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f31ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.980 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 --2- 172.21.15.31:0/273258681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f32d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.979+0000 7fa696c85b80 1 -- 172.21.15.31:0/273258681 >> 172.21.15.31:0/273258681 conn(0x555838f273e0 msgr2=0x555838f2c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.980+0000 7fa696c85b80 1 -- 172.21.15.31:0/273258681 shutdown_connections 2023-07-29T16:30:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.980+0000 7fa696c85b80 1 -- 172.21.15.31:0/273258681 wait complete. 2023-07-29T16:30:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.980+0000 7fa68199c700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6680a0810 0x7fa6680a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.981 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.980+0000 7fa696c85b80 1 -- start start 2023-07-29T16:30:43.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 Processor -- start 2023-07-29T16:30:43.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 0x555838f3b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f32530 0x555838f3bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa68199c700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6680a0810 0x7fa6680a2cc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x555838f31980 tx=0x7fa67400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.982 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f42d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838e1f510 con 0x555838f31be0 2023-07-29T16:30:43.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838e1f1b0 con 0x555838f32530 2023-07-29T16:30:43.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.981+0000 7fa696c85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838d06920 con 0x555838f332f0 2023-07-29T16:30:43.983 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.982+0000 7fa696c85b80 10 client.0 _ll_register_callbacks cb 0x555838e28620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:43.984 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f32530 0x555838f3bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68199c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f42d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 0x555838f3b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 0x555838f3b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42764/0 (socket says 172.21.15.31:42764) 2023-07-29T16:30:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68199c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f42d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59374/0 (socket says 172.21.15.31:59374) 2023-07-29T16:30:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 -- 172.21.15.31:0/1930903281 learned_addr learned my addr 172.21.15.31:0/1930903281 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 -- 172.21.15.31:0/1930903281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 msgr2=0x555838f42d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f42d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 -- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f32530 msgr2=0x555838f3bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f32530 0x555838f3bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.983+0000 7fa68119b700 1 -- 172.21.15.31:0/1930903281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa678009c30 con 0x555838f31be0 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.984+0000 7fa68119b700 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 0x555838f3b800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0aa670 tx=0x7fa66c0aa590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e3b11de179aaf server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.984+0000 7fa664ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.984+0000 7fa664ff9700 1 -- 172.21.15.31:0/1930903281 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0ae9a0 con 0x555838f31be0 2023-07-29T16:30:43.986 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.984+0000 7fa664ff9700 1 -- 172.21.15.31:0/1930903281 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa66c0b2d30 con 0x555838f31be0 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.984+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555838eadd10 con 0x555838f31be0 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa664ff9700 1 -- 172.21.15.31:0/1930903281 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa66c0a9e60 con 0x555838f31be0 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 10 client.19839 fetch_fsmap learned FSMap version 38 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555838c99c60 con 0x555838f31be0 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa664ff9700 1 -- 172.21.15.31:0/1930903281 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555838eadd10 con 0x555838f31be0 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa664ff9700 1 -- 172.21.15.31:0/1930903281 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa66c0a6760 con 0x555838f31be0 2023-07-29T16:30:43.987 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 10 client.19839 fetch_fsmap finished waiting for FSMap version 38 2023-07-29T16:30:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 10 client.19839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 20 client.19839 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f47240 0x555838f49650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:43.988 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 4 client.19839 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:43.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.985+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555838f00d70 con 0x555838f47240 2023-07-29T16:30:43.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.986+0000 7fa68099a700 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f47240 0x555838f49650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:43.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.986+0000 7fa664ff9700 10 client.19839 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:43.989 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.987+0000 7fa68099a700 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f47240 0x555838f49650 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c4a8dd614238b3e server_cookie=b494a5cf43d3e2e7 in_seq=0 out_seq=0 2023-07-29T16:30:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa664ff9700 1 -- 172.21.15.31:0/1930903281 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x555838f00d70 con 0x555838f47240 2023-07-29T16:30:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa664ff9700 10 client.19839 handle_command_reply: tid=0 2023-07-29T16:30:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 client.19839 shutdown 2023-07-29T16:30:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 msgr2=0x555838f3b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:43.990 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 0x555838f3b800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0aa670 tx=0x7fa66c0aa590 comp rx=0 tx=0).stop 2023-07-29T16:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 shutdown_connections 2023-07-29T16:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f47240 0x555838f49650 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f42d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f31be0 0x555838f3b800 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1930903281 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f32530 0x555838f3bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:43.991 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.990+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 >> 172.21.15.31:0/1930903281 conn(0x555838f273e0 msgr2=0x555838f2c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:43.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.991+0000 7fa664ff9700 0 client.19839 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:43.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.991+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 shutdown_connections 2023-07-29T16:30:43.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.991+0000 7fa696c85b80 1 -- 172.21.15.31:0/1930903281 wait complete. 2023-07-29T16:30:43.992 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:43.991+0000 7fa696c85b80 20 client.19839 trim_cache size 0 max 16384 2023-07-29T16:30:44.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.010+0000 7fa696c85b80 1 Processor -- start 2023-07-29T16:30:44.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 -- start start 2023-07-29T16:30:44.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f31be0 0x555838f40d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f2c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f49250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838f3c9f0 con 0x555838f32530 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838f3acd0 con 0x555838f31be0 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa696c85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838f3b370 con 0x555838f332f0 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa68199c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f49250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa68199c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f49250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59386/0 (socket says 172.21.15.31:59386) 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.011+0000 7fa68199c700 1 -- 172.21.15.31:0/1026957878 learned_addr learned my addr 172.21.15.31:0/1026957878 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:44.012 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68119b700 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f31be0 0x555838f40d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:44.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68199c700 1 -- 172.21.15.31:0/1026957878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f31be0 msgr2=0x555838f40d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:44.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68199c700 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f31be0 0x555838f40d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68199c700 1 -- 172.21.15.31:0/1026957878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 msgr2=0x555838f2c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:44.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68199c700 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f2c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68199c700 1 -- 172.21.15.31:0/1026957878 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555838cd7250 con 0x555838f332f0 2023-07-29T16:30:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.012+0000 7fa68199c700 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f49250 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa674003c40 tx=0x7fa674013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a96f2e175ebc1448 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.013+0000 7fa667fff700 1 -- 172.21.15.31:0/1026957878 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa674004a30 con 0x555838f332f0 2023-07-29T16:30:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.013+0000 7fa667fff700 1 -- 172.21.15.31:0/1026957878 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa674015070 con 0x555838f332f0 2023-07-29T16:30:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.013+0000 7fa667fff700 1 -- 172.21.15.31:0/1026957878 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6740119f0 con 0x555838f332f0 2023-07-29T16:30:44.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 -- 172.21.15.31:0/1026957878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 msgr2=0x555838f49250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:44.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f49250 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fa674003c40 tx=0x7fa674013000 comp rx=0 tx=0).stop 2023-07-29T16:30:44.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 -- 172.21.15.31:0/1026957878 shutdown_connections 2023-07-29T16:30:44.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f332f0 0x555838f49250 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f31be0 0x555838f40d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 --2- 172.21.15.31:0/1026957878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f2c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 -- 172.21.15.31:0/1026957878 >> 172.21.15.31:0/1026957878 conn(0x555838f273e0 msgr2=0x555838f2c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.014+0000 7fa696c85b80 1 -- 172.21.15.31:0/1026957878 shutdown_connections 2023-07-29T16:30:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.015+0000 7fa696c85b80 1 -- 172.21.15.31:0/1026957878 wait complete. 2023-07-29T16:30:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.015+0000 7fa696c85b80 1 -- start start 2023-07-29T16:30:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 Processor -- start 2023-07-29T16:30:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f40470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f40a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f3ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838f3f2c0 con 0x555838f32530 2023-07-29T16:30:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838f3f430 con 0x555838f332f0 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x555838f3f5f0 con 0x555838f31be0 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa696c85b80 10 client.0 _ll_register_callbacks cb 0x555838e28620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.016+0000 7fa68119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f40470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f40470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59402/0 (socket says 172.21.15.31:59402) 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 -- 172.21.15.31:0/3765152359 learned_addr learned my addr 172.21.15.31:0/3765152359 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68199c700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f3ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68099a700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f40a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 -- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 msgr2=0x555838f3ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f3ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 -- 172.21.15.31:0/3765152359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 msgr2=0x555838f40a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f40a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 -- 172.21.15.31:0/3765152359 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6740038f0 con 0x555838f31be0 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68199c700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f3ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68099a700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f40a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa68119b700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f40470 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa66c005910 tx=0x7fa66c0a98d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=983bd7e5a0c9822f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa664ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa664ff9700 1 -- 172.21.15.31:0/3765152359 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0a9c70 con 0x555838f31be0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa664ff9700 1 -- 172.21.15.31:0/3765152359 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa66c0ae9a0 con 0x555838f31be0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.017+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa678015410 con 0x555838f31be0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa664ff9700 1 -- 172.21.15.31:0/3765152359 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa66c0b28a0 con 0x555838f31be0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 10 client.30001 fetch_fsmap learned FSMap version 38 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555838caf2b0 con 0x555838f31be0 2023-07-29T16:30:44.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa664ff9700 1 -- 172.21.15.31:0/3765152359 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa678015410 con 0x555838f31be0 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa664ff9700 1 -- 172.21.15.31:0/3765152359 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa66c070b80 con 0x555838f31be0 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 10 client.30001 fetch_fsmap finished waiting for FSMap version 38 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 10 client.30001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 20 client.30001 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f99a60 0x555838f9be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 4 client.30001 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555838e7bd40 con 0x555838f99a60 2023-07-29T16:30:44.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.018+0000 7fa68099a700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f99a60 0x555838f9be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:44.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.019+0000 7fa68099a700 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f99a60 0x555838f9be40 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96894dbd467ad73c server_cookie=940dc70158d3ae58 in_seq=0 out_seq=0 2023-07-29T16:30:44.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.019+0000 7fa664ff9700 10 client.30001 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.025+0000 7fa664ff9700 1 -- 172.21.15.31:0/3765152359 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555838e7bd40 con 0x555838f99a60 2023-07-29T16:30:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.025+0000 7fa664ff9700 10 client.30001 handle_command_reply: tid=0 2023-07-29T16:30:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.026+0000 7fa696c85b80 1 client.30001 shutdown 2023-07-29T16:30:44.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:43 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/942904139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:30:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.026+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 msgr2=0x555838f40470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.026+0000 7fa696c85b80 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f40470 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fa66c005910 tx=0x7fa66c0a98d0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 shutdown_connections 2023-07-29T16:30:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x555838f99a60 0x555838f9be40 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555838f332f0 0x555838f3ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555838f31be0 0x555838f40470 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa664ff9700 0 client.30001 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 --2- 172.21.15.31:0/3765152359 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555838f32530 0x555838f40a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 >> 172.21.15.31:0/3765152359 conn(0x555838f273e0 msgr2=0x555838f39670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 shutdown_connections 2023-07-29T16:30:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 1 -- 172.21.15.31:0/3765152359 wait complete. 2023-07-29T16:30:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.027+0000 7fa696c85b80 20 client.30001 trim_cache size 0 max 16384 2023-07-29T16:30:44.028 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:30:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.028+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6680a0810 msgr2=0x7fa6680a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.028+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6680a0810 0x7fa6680a2cc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x555838f31980 tx=0x7fa67400b390 comp rx=0 tx=0).stop 2023-07-29T16:30:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.028+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 msgr2=0x7fa67c106170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:44.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.028+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c106170 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa66c003790 tx=0x7fa66c01cec0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.028+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 shutdown_connections 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.028+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa6680a0810 0x7fa6680a2cc0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.029+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa67c100da0 0x7fa67c192f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.029+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa67c0ff220 0x7fa67c106170 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.029+0000 7fa6833ff700 1 --2- 172.21.15.31:0/2469463351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa67c100410 0x7fa67c1066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.029+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 >> 172.21.15.31:0/2469463351 conn(0x7fa67c0fa840 msgr2=0x7fa67c0fb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.029+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 shutdown_connections 2023-07-29T16:30:44.030 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:44.029+0000 7fa6833ff700 1 -- 172.21.15.31:0/2469463351 wait complete. 2023-07-29T16:30:44.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:43 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/942904139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:30:44.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 10586 26.15 MB/sec warmup 28 sec latency 149.345 ms 2023-07-29T16:30:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:44 smithi031 ceph-mon[95504]: pgmap v829: 97 pgs: 97 active+clean; 372 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 436 op/s 2023-07-29T16:30:44.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:44 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:45.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:44 smithi039 ceph-mon[128858]: pgmap v829: 97 pgs: 97 active+clean; 372 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 436 op/s 2023-07-29T16:30:45.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:44 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:45.057 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-07-29T16:30:45.058 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-07-29T16:30:45.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:44 smithi156 ceph-mon[129245]: pgmap v829: 97 pgs: 97 active+clean; 372 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.4 MiB/s wr, 436 op/s 2023-07-29T16:30:45.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:44 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:45.219 INFO:tasks.workunit.client.0.smithi031.stdout: 10 11558 27.71 MB/sec warmup 29 sec latency 34.447 ms 2023-07-29T16:30:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.365+0000 7fb0a03ff700 1 -- 172.21.15.31:0/3053437392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb098100450 msgr2=0x7fb0981008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.365+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/3053437392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb098100450 0x7fb0981008a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fb088009990 tx=0x7fb088009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.367+0000 7fb0a03ff700 1 -- 172.21.15.31:0/3053437392 shutdown_connections 2023-07-29T16:30:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.367+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/3053437392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb098100de0 0x7fb0981076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.367+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/3053437392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0980ff260 0x7fb0980ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.367+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/3053437392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb098100450 0x7fb0981008a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.367+0000 7fb0a03ff700 1 -- 172.21.15.31:0/3053437392 >> 172.21.15.31:0/3053437392 conn(0x7fb0980fa860 msgr2=0x7fb0980fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.368+0000 7fb0a03ff700 1 -- 172.21.15.31:0/3053437392 shutdown_connections 2023-07-29T16:30:45.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.369+0000 7fb0a03ff700 1 -- 172.21.15.31:0/3053437392 wait complete. 2023-07-29T16:30:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.370+0000 7fb0a03ff700 1 Processor -- start 2023-07-29T16:30:45.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 -- start start 2023-07-29T16:30:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 0x7fb09819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb098100450 0x7fb09819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb098100de0 0x7fb0981a3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0981a4250 con 0x7fb098100de0 2023-07-29T16:30:45.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0981a4390 con 0x7fb0980ff260 2023-07-29T16:30:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb0a03ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0981a4520 con 0x7fb098100450 2023-07-29T16:30:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb09e19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 0x7fb09819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb09e19b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 0x7fb09819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38214/0 (socket says 172.21.15.31:38214) 2023-07-29T16:30:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.371+0000 7fb09e19b700 1 -- 172.21.15.31:0/681306602 learned_addr learned my addr 172.21.15.31:0/681306602 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09d99a700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb098100450 0x7fb09819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e99c700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb098100de0 0x7fb0981a3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e19b700 1 -- 172.21.15.31:0/681306602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb098100450 msgr2=0x7fb09819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e19b700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb098100450 0x7fb09819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e19b700 1 -- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb098100de0 msgr2=0x7fb0981a3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e19b700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb098100de0 0x7fb0981a3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e19b700 1 -- 172.21.15.31:0/681306602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090009c40 con 0x7fb0980ff260 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.372+0000 7fb09e19b700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 0x7fb09819c6d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb0940099f0 tx=0x7fb094009d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.373+0000 7fb08f7fe700 1 -- 172.21.15.31:0/681306602 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094012070 con 0x7fb0980ff260 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.373+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880095f0 con 0x7fb0980ff260 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.373+0000 7fb08f7fe700 1 -- 172.21.15.31:0/681306602 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb09400ce30 con 0x7fb0980ff260 2023-07-29T16:30:45.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.373+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0981a4bd0 con 0x7fb0980ff260 2023-07-29T16:30:45.375 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.373+0000 7fb08f7fe700 1 -- 172.21.15.31:0/681306602 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0940038e0 con 0x7fb0980ff260 2023-07-29T16:30:45.379 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.379+0000 7fb0b3db5b80 1 Processor -- start 2023-07-29T16:30:45.380 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 -- start start 2023-07-29T16:30:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd0b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd0bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd0c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dc62a90 con 0x559a7dd0ac60 2023-07-29T16:30:45.381 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dc60450 con 0x559a7dd0c370 2023-07-29T16:30:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.381+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dc5bd40 con 0x559a7dd0b5b0 2023-07-29T16:30:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd0c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.382 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd0c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38226/0 (socket says 172.21.15.31:38226) 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 -- 172.21.15.31:0/3196788132 learned_addr learned my addr 172.21.15.31:0/3196788132 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09e99c700 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd0b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 -- 172.21.15.31:0/3196788132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 msgr2=0x559a7dd0bde0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd0bde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 -- 172.21.15.31:0/3196788132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 msgr2=0x559a7dd0b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd0b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.383 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.382+0000 7fb09d99a700 1 -- 172.21.15.31:0/3196788132 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a7da89e60 con 0x559a7dd0c370 2023-07-29T16:30:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.383+0000 7fb08f7fe700 1 -- 172.21.15.31:0/681306602 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb09400c970 con 0x7fb0980ff260 2023-07-29T16:30:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.383+0000 7fb09d99a700 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd0c7c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb088009990 tx=0x7fb088005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d958a2c52ef78d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.384 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.383+0000 7fb08cbfe700 1 -- 172.21.15.31:0/3196788132 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088003c10 con 0x559a7dd0c370 2023-07-29T16:30:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.383+0000 7fb08cbfe700 1 -- 172.21.15.31:0/3196788132 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0880050b0 con 0x559a7dd0c370 2023-07-29T16:30:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.383+0000 7fb08cbfe700 1 -- 172.21.15.31:0/3196788132 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08800b6f0 con 0x559a7dd0c370 2023-07-29T16:30:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3196788132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 msgr2=0x559a7dd0c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.385 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd0c7c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb088009990 tx=0x7fb088005fd0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3196788132 shutdown_connections 2023-07-29T16:30:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd0c7c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.386 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd0b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3196788132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd0bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3196788132 >> 172.21.15.31:0/3196788132 conn(0x559a7dd00460 msgr2=0x559a7dd05a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.385+0000 7fb08f7fe700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb0840a0890 0x7fb0840a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.386+0000 7fb08f7fe700 1 -- 172.21.15.31:0/681306602 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb0940cadb0 con 0x7fb0980ff260 2023-07-29T16:30:45.387 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.386+0000 7fb09e99c700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb0840a0890 0x7fb0840a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.386+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3196788132 shutdown_connections 2023-07-29T16:30:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.386+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3196788132 wait complete. 2023-07-29T16:30:45.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.387+0000 7fb09e99c700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb0840a0890 0x7fb0840a2d40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x559a7dd0aa00 tx=0x7fb09000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.387+0000 7fb0b3db5b80 1 -- start start 2023-07-29T16:30:45.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.387+0000 7fb0b3db5b80 1 Processor -- start 2023-07-29T16:30:45.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd14df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.389 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd1be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dbf8590 con 0x559a7dd0ac60 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dbf8230 con 0x559a7dd0c370 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dadf9a0 con 0x559a7dd0b5b0 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.388+0000 7fb0b3db5b80 10 client.0 _ll_register_callbacks cb 0x559a7dc016a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd1be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd1be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38228/0 (socket says 172.21.15.31:38228) 2023-07-29T16:30:45.390 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 -- 172.21.15.31:0/4060126864 learned_addr learned my addr 172.21.15.31:0/4060126864 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09d99a700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd14df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 -- 172.21.15.31:0/4060126864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 msgr2=0x559a7dd14df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd14df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 -- 172.21.15.31:0/4060126864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 msgr2=0x559a7dd148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.389+0000 7fb09e99c700 1 -- 172.21.15.31:0/4060126864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb098107fd0 con 0x559a7dd0c370 2023-07-29T16:30:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.390+0000 7fb09d99a700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.390+0000 7fb09e99c700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd1be30 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb0900042b0 tx=0x7fb090004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e3a96307805f9d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.390+0000 7fb081ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:30:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.390+0000 7fb081ffb700 1 -- 172.21.15.31:0/4060126864 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09001c040 con 0x559a7dd0c370 2023-07-29T16:30:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.390+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a7dc86d90 con 0x559a7dd0c370 2023-07-29T16:30:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.390+0000 7fb081ffb700 1 -- 172.21.15.31:0/4060126864 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb090016810 con 0x559a7dd0c370 2023-07-29T16:30:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb081ffb700 1 -- 172.21.15.31:0/4060126864 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb09000e040 con 0x559a7dd0c370 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 10 client.30014 fetch_fsmap learned FSMap version 38 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x559a7da72ce0 con 0x559a7dd0c370 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb081ffb700 1 -- 172.21.15.31:0/4060126864 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559a7dc86d90 con 0x559a7dd0c370 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb081ffb700 1 -- 172.21.15.31:0/4060126864 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb090022040 con 0x559a7dd0c370 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 10 client.30014 fetch_fsmap finished waiting for FSMap version 38 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 10 client.30014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 20 client.30014 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd20770 0x559a7dd22b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 4 client.30014 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.391+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a7dcd9df0 con 0x559a7dd20770 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.392+0000 7fb09e19b700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd20770 0x559a7dd22b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.392+0000 7fb081ffb700 10 client.30014 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.392+0000 7fb09e19b700 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd20770 0x559a7dd22b80 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd9929d0ceaf6ccc server_cookie=15d6d820c0b3b498 in_seq=0 out_seq=0 2023-07-29T16:30:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.393+0000 7fb081ffb700 1 -- 172.21.15.31:0/4060126864 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559a7dcd9df0 con 0x559a7dd20770 2023-07-29T16:30:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.393+0000 7fb081ffb700 10 client.30014 handle_command_reply: tid=0 2023-07-29T16:30:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.393+0000 7fb0b3db5b80 1 client.30014 shutdown 2023-07-29T16:30:45.396 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 msgr2=0x559a7dd1be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd1be30 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb0900042b0 tx=0x7fb090004b40 comp rx=0 tx=0).stop 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 shutdown_connections 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd20770 0x559a7dd22b80 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0c370 0x559a7dd1be30 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/4060126864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0b5b0 0x559a7dd14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.397 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 >> 172.21.15.31:0/4060126864 conn(0x559a7dd00460 msgr2=0x559a7dd050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb081ffb700 0 client.30014 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 shutdown_connections 2023-07-29T16:30:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.394+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/4060126864 wait complete. 2023-07-29T16:30:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.395+0000 7fb0b3db5b80 20 client.30014 trim_cache size 0 max 16384 2023-07-29T16:30:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.422+0000 7fb0b3db5b80 1 Processor -- start 2023-07-29T16:30:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.422+0000 7fb0b3db5b80 1 -- start start 2023-07-29T16:30:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd19d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.423 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0c370 0x559a7dd14df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dd205a0 con 0x559a7dd0ac60 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dd024e0 con 0x559a7dd0b5b0 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dcfff50 con 0x559a7dd0c370 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb09d99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb09d99a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd148b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38234/0 (socket says 172.21.15.31:38234) 2023-07-29T16:30:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb09d99a700 1 -- 172.21.15.31:0/387808223 learned_addr learned my addr 172.21.15.31:0/387808223 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.423+0000 7fb09e99c700 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0c370 0x559a7dd14df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb09d99a700 1 -- 172.21.15.31:0/387808223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0c370 msgr2=0x559a7dd14df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb09d99a700 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0c370 0x559a7dd14df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb09d99a700 1 -- 172.21.15.31:0/387808223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 msgr2=0x559a7dd19d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb09d99a700 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd19d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb09d99a700 1 -- 172.21.15.31:0/387808223 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a7dab02d0 con 0x559a7dd0b5b0 2023-07-29T16:30:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb09d99a700 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd148b0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb0880106b0 tx=0x7fb088024040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=467ea5cc2f749fe0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb08d3ff700 1 -- 172.21.15.31:0/387808223 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088003a10 con 0x559a7dd0b5b0 2023-07-29T16:30:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb08d3ff700 1 -- 172.21.15.31:0/387808223 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb088003b70 con 0x559a7dd0b5b0 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.424+0000 7fb08d3ff700 1 -- 172.21.15.31:0/387808223 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb088018790 con 0x559a7dd0b5b0 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/387808223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 msgr2=0x559a7dd148b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd148b0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb0880106b0 tx=0x7fb088024040 comp rx=0 tx=0).stop 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/387808223 shutdown_connections 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0c370 0x559a7dd14df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0ac60 0x559a7dd19d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/387808223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd148b0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/387808223 >> 172.21.15.31:0/387808223 conn(0x559a7dd00460 msgr2=0x559a7dd0f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/387808223 shutdown_connections 2023-07-29T16:30:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/387808223 wait complete. 2023-07-29T16:30:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.426+0000 7fb0b3db5b80 1 -- start start 2023-07-29T16:30:45.428 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.427+0000 7fb0b3db5b80 1 Processor -- start 2023-07-29T16:30:45.429 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 0x559a7dd18bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd19110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0c370 0x559a7dd72310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dd197e0 con 0x559a7dd0c370 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dd72850 con 0x559a7dd0b5b0 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a7dd839f0 con 0x559a7dd0ac60 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb09e19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 0x559a7dd18bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.430 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb0b3db5b80 10 client.0 _ll_register_callbacks cb 0x559a7dc016a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb09e19b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 0x559a7dd18bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59448/0 (socket says 172.21.15.31:59448) 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb09e19b700 1 -- 172.21.15.31:0/3819487899 learned_addr learned my addr 172.21.15.31:0/3819487899 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb09e19b700 1 -- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 msgr2=0x559a7dd19110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb09e19b700 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd19110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.428+0000 7fb09e19b700 1 -- 172.21.15.31:0/3819487899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0c370 msgr2=0x559a7dd72310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.429+0000 7fb09e19b700 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0c370 0x559a7dd72310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.429+0000 7fb09e19b700 1 -- 172.21.15.31:0/3819487899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a7da89e60 con 0x559a7dd0ac60 2023-07-29T16:30:45.431 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.429+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0880050b0 con 0x559a7dd0ac60 2023-07-29T16:30:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.429+0000 7fb09e19b700 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 0x559a7dd18bd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb0940c87f0 tx=0x7fb094097e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c70d47eb55c319a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.429+0000 7fb081ffb700 10 client.30016 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.430+0000 7fb081ffb700 1 -- 172.21.15.31:0/3819487899 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09409c480 con 0x559a7dd0ac60 2023-07-29T16:30:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.430+0000 7fb081ffb700 1 -- 172.21.15.31:0/3819487899 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb0940a08a0 con 0x559a7dd0ac60 2023-07-29T16:30:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.430+0000 7fb081ffb700 1 -- 172.21.15.31:0/3819487899 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0940c8900 con 0x559a7dd0ac60 2023-07-29T16:30:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.431+0000 7fb0b3db5b80 10 client.30016 fetch_fsmap learned FSMap version 38 2023-07-29T16:30:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.431+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=38}) v3 -- 0x559a7da88330 con 0x559a7dd0ac60 2023-07-29T16:30:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb081ffb700 1 -- 172.21.15.31:0/3819487899 <== mon.1 v2:172.21.15.156:3300/0 4 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb0940986b0 con 0x559a7dd0ac60 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb0b3db5b80 10 client.30016 fetch_fsmap finished waiting for FSMap version 38 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb0b3db5b80 10 client.30016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb0b3db5b80 20 client.30016 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd852b0 0x559a7dd876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb0b3db5b80 4 client.30016 mds_command: new command op to 24581 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.432+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559a7dc54dc0 con 0x559a7dd852b0 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.433+0000 7fb09d99a700 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd852b0 0x559a7dd876c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.433+0000 7fb081ffb700 10 client.30016 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.433+0000 7fb09d99a700 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd852b0 0x559a7dd876c0 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54471838bdf38886 server_cookie=3cc6af23bd0ece0c in_seq=0 out_seq=0 2023-07-29T16:30:45.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.439+0000 7fb081ffb700 1 -- 172.21.15.31:0/3819487899 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559a7dc54dc0 con 0x559a7dd852b0 2023-07-29T16:30:45.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.439+0000 7fb081ffb700 10 client.30016 handle_command_reply: tid=0 2023-07-29T16:30:45.439 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.439+0000 7fb0b3db5b80 1 client.30016 shutdown 2023-07-29T16:30:45.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 msgr2=0x559a7dd18bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.440 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 0x559a7dd18bd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb0940c87f0 tx=0x7fb094097e60 comp rx=0 tx=0).stop 2023-07-29T16:30:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 shutdown_connections 2023-07-29T16:30:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559a7dd852b0 0x559a7dd876c0 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559a7dd0c370 0x559a7dd72310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559a7dd0ac60 0x559a7dd18bd0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 --2- 172.21.15.31:0/3819487899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559a7dd0b5b0 0x559a7dd19110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.440+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 >> 172.21.15.31:0/3819487899 conn(0x559a7dd00460 msgr2=0x559a7dd14850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.442+0000 7fb081ffb700 0 client.30016 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.442+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 shutdown_connections 2023-07-29T16:30:45.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.442+0000 7fb0b3db5b80 1 -- 172.21.15.31:0/3819487899 wait complete. 2023-07-29T16:30:45.442 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.442+0000 7fb0b3db5b80 20 client.30016 trim_cache size 0 max 16384 2023-07-29T16:30:45.443 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:30:45.443 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2023-07-29T16:30:45.443 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "fb2849eb-979a-4326-b9ec-44a72fd1dbe4", 2023-07-29T16:30:45.443 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2023-07-29T16:30:45.443 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:30:45.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb0840a0890 msgr2=0x7fb0840a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb0840a0890 0x7fb0840a2d40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x559a7dd0aa00 tx=0x7fb09000b040 comp rx=0 tx=0).stop 2023-07-29T16:30:45.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 msgr2=0x7fb09819c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.444 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 0x7fb09819c6d0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb0940099f0 tx=0x7fb094009d00 comp rx=0 tx=0).stop 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 shutdown_connections 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb0840a0890 0x7fb0840a2d40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb098100de0 0x7fb0981a3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0980ff260 0x7fb09819c6d0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 --2- 172.21.15.31:0/681306602 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb098100450 0x7fb09819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 >> 172.21.15.31:0/681306602 conn(0x7fb0980fa860 msgr2=0x7fb0980fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.445 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 shutdown_connections 2023-07-29T16:30:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.443+0000 7fb0a03ff700 1 -- 172.21.15.31:0/681306602 wait complete. 2023-07-29T16:30:45.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:30:45.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 -- 172.21.15.31:0/1193466552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 msgr2=0x7fb8981038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 --2- 172.21.15.31:0/1193466552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 0x7fb8981038f0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009990 tx=0x7fb88c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 -- 172.21.15.31:0/1193466552 shutdown_connections 2023-07-29T16:30:45.813 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 --2- 172.21.15.31:0/1193466552 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb898103e30 0x7fb8981042b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 --2- 172.21.15.31:0/1193466552 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8980feb10 0x7fb898100f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 --2- 172.21.15.31:0/1193466552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 0x7fb8981038f0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 -- 172.21.15.31:0/1193466552 >> 172.21.15.31:0/1193466552 conn(0x7fb8980fa700 msgr2=0x7fb8980fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.814 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.813+0000 7fb897fff700 1 -- 172.21.15.31:0/1193466552 shutdown_connections 2023-07-29T16:30:45.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.815+0000 7fb897fff700 1 -- 172.21.15.31:0/1193466552 wait complete. 2023-07-29T16:30:45.815 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.816+0000 7fb897fff700 1 Processor -- start 2023-07-29T16:30:45.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.816+0000 7fb897fff700 1 -- start start 2023-07-29T16:30:45.816 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb897fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8980feb10 0x7fb89806f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb897fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 0x7fb89806fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb897fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 0x7fb8981ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb897fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898072b60 con 0x7fb8980feb10 2023-07-29T16:30:45.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb897fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898072cd0 con 0x7fb898101470 2023-07-29T16:30:45.817 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb897fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981acbf0 con 0x7fb898103e30 2023-07-29T16:30:45.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb8977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 0x7fb8981ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb8977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 0x7fb8981ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59452/0 (socket says 172.21.15.31:59452) 2023-07-29T16:30:45.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb8977fe700 1 -- 172.21.15.31:0/3838267382 learned_addr learned my addr 172.21.15.31:0/3838267382 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb8967fc700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 0x7fb89806fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.818 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.817+0000 7fb896ffd700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8980feb10 0x7fb89806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.818+0000 7fb8977fe700 1 -- 172.21.15.31:0/3838267382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 msgr2=0x7fb89806fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.818+0000 7fb8977fe700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 0x7fb89806fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.818+0000 7fb8977fe700 1 -- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8980feb10 msgr2=0x7fb89806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.818+0000 7fb8977fe700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8980feb10 0x7fb89806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.818+0000 7fb8977fe700 1 -- 172.21.15.31:0/3838267382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c0095f0 con 0x7fb898103e30 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.818+0000 7fb8977fe700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 0x7fb8981ac6b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb89000ef30 tx=0x7fb89000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.819+0000 7fb87ffff700 1 -- 172.21.15.31:0/3838267382 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb890018070 con 0x7fb898103e30 2023-07-29T16:30:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.819+0000 7fb87ffff700 1 -- 172.21.15.31:0/3838267382 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb89000f040 con 0x7fb898103e30 2023-07-29T16:30:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.819+0000 7fb87ffff700 1 -- 172.21.15.31:0/3838267382 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89000cca0 con 0x7fb898103e30 2023-07-29T16:30:45.820 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.819+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8981acd30 con 0x7fb898103e30 2023-07-29T16:30:45.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.819+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8981ad190 con 0x7fb898103e30 2023-07-29T16:30:45.822 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:45 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:45.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:45 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:45.823 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:45 smithi156 ceph-mon[129245]: Scrub error on inode 0x100000045b8 (/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902/client.0/tmp/clients/client3/~dmtmp/PWRPNT/NEWPCB.PPT) see mds.i log and `damage ls` output for details 2023-07-29T16:30:45.823 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:30:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: 2023-07-29T16:30:45.507+0000 7f8057901700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000045b8 [2,head] /volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902/client.0/tmp/clients/client3/~dmtmp/PWRPNT/NEWPCB.PPT auth v320 ap=1 s=85026 n(v0 rc2023-07-29T16:30:41.341462+0000 b85026 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={25186=0-4194304@1} caps={25186=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25186 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x56152bf7db80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(3)0x100000045b8:[<0x10000003af7/NEWPCB.PPT v320>,<0x10000003aa3/PWRPNT v1732>,<0x10000003aa2/~dmtmp v92>,<0x10000000007/client3 v2510>,<0x10000000006/clients v1742>,<0x10000000005/tmp v1696>,<0x100000001fb/client.0 v1686>,<0x100000001fa/5354c809-3264-4aed-854c-cd3a8f125902 v1690>,<0x10000000001/sv_1 v1687>,<0x10000000000/_nogroup v1643>,<0x1/volumes v1581>]//[]","memoryvalue":"(3)0x100000045b8:[<0x10000003af7/NEWPCB.PPT v320>,<0x10000003aa3/PWRPNT v1760>,<0x10000003aa2/~dmtmp v107>,<0x10000000007/client3 v2577>,<0x10000000006/clients v1756>,<0x10000000005/tmp v1688>,<0x100000001fb/client.0 v1689>,<0x100000001fa/5354c809-3264-4aed-854c-cd3a8f125902 v1693>,<0x10000000001/sv_1 v1690>,<0x10000000000/_nogroup v1645>,<0x1/volumes v1584>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2023-07-29T16:30:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.825+0000 7fb87ffff700 1 -- 172.21.15.31:0/3838267382 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb890003a00 con 0x7fb898103e30 2023-07-29T16:30:45.824 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.825+0000 7fb8b0d6eb80 1 Processor -- start 2023-07-29T16:30:45.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.825+0000 7fb8b0d6eb80 1 -- start start 2023-07-29T16:30:45.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469ca150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.825 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469caf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469cb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d546921a10 con 0x55d5469cb4d0 2023-07-29T16:30:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54691f3d0 con 0x55d5469c9d40 2023-07-29T16:30:45.826 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54691acc0 con 0x55d5469ca690 2023-07-29T16:30:45.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59468/0 (socket says 172.21.15.31:59468) 2023-07-29T16:30:45.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb896ffd700 1 -- 172.21.15.31:0/210268488 learned_addr learned my addr 172.21.15.31:0/210268488 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.827 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8967fc700 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb8967fc700 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.826+0000 7fb896ffd700 1 -- 172.21.15.31:0/210268488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 msgr2=0x55d5469ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb896ffd700 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb896ffd700 1 -- 172.21.15.31:0/210268488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 msgr2=0x55d5469cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb896ffd700 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb896ffd700 1 -- 172.21.15.31:0/210268488 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d546748de0 con 0x55d5469ca690 2023-07-29T16:30:45.828 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb8967fc700 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:45.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb896ffd700 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469caf40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb88800bd40 tx=0x7fb8880106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf6eae399929f36 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb87d7fa700 1 -- 172.21.15.31:0/210268488 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88800be90 con 0x55d5469ca690 2023-07-29T16:30:45.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb87d7fa700 1 -- 172.21.15.31:0/210268488 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb888015070 con 0x55d5469ca690 2023-07-29T16:30:45.829 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.827+0000 7fb87d7fa700 1 -- 172.21.15.31:0/210268488 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888005280 con 0x55d5469ca690 2023-07-29T16:30:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.828+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/210268488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 msgr2=0x55d5469caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.828+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469caf40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb88800bd40 tx=0x7fb8880106b0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/210268488 shutdown_connections 2023-07-29T16:30:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.830 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/210268488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469caf40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/210268488 >> 172.21.15.31:0/210268488 conn(0x55d5469bf540 msgr2=0x55d5469c4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb87ffff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8800a0a20 0x7fb8800a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.831 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb87ffff700 1 -- 172.21.15.31:0/3838267382 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb8900cb470 con 0x7fb898103e30 2023-07-29T16:30:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/210268488 shutdown_connections 2023-07-29T16:30:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/210268488 wait complete. 2023-07-29T16:30:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.829+0000 7fb896ffd700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8800a0a20 0x7fb8800a2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.830+0000 7fb8b0d6eb80 1 -- start start 2023-07-29T16:30:45.832 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.830+0000 7fb8b0d6eb80 1 Processor -- start 2023-07-29T16:30:45.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb896ffd700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8800a0a20 0x7fb8800a2ed0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb888004c30 tx=0x7fb88800b650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469daf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.834 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5468b7510 con 0x55d5469cb4d0 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5468b71b0 con 0x55d5469ca690 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d54679e920 con 0x55d5469c9d40 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8b0d6eb80 10 client.0 _ll_register_callbacks cb 0x55d5468c0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59476/0 (socket says 172.21.15.31:59476) 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb896ffd700 1 -- 172.21.15.31:0/1813927613 learned_addr learned my addr 172.21.15.31:0/1813927613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8967fc700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.835 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.831+0000 7fb8977fe700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469daf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.832+0000 7fb896ffd700 1 -- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 msgr2=0x55d5469d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.832+0000 7fb896ffd700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.832+0000 7fb896ffd700 1 -- 172.21.15.31:0/1813927613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 msgr2=0x55d5469daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.832+0000 7fb896ffd700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469daf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.832+0000 7fb896ffd700 1 -- 172.21.15.31:0/1813927613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88800b9a0 con 0x55d5469c9d40 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb896ffd700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d3990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb888019000 tx=0x7fb8880106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=991c5f38c6f01a13 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb87aff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb87aff5700 1 -- 172.21.15.31:0/1813927613 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888020070 con 0x55d5469c9d40 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb87aff5700 1 -- 172.21.15.31:0/1813927613 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb888017070 con 0x55d5469c9d40 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d546945d10 con 0x55d5469c9d40 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb87aff5700 1 -- 172.21.15.31:0/1813927613 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb888003890 con 0x55d5469c9d40 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb8b0d6eb80 10 client.30028 fetch_fsmap learned FSMap version 38 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.833+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55d546731c60 con 0x55d5469c9d40 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb87aff5700 1 -- 172.21.15.31:0/1813927613 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d546945d10 con 0x55d5469c9d40 2023-07-29T16:30:45.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb87aff5700 1 -- 172.21.15.31:0/1813927613 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb888005450 con 0x55d5469c9d40 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8b0d6eb80 10 client.30028 fetch_fsmap finished waiting for FSMap version 38 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8b0d6eb80 10 client.30028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8b0d6eb80 20 client.30028 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5469df750 0x55d5469e1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8b0d6eb80 4 client.30028 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d546998d70 con 0x55d5469df750 2023-07-29T16:30:45.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.834+0000 7fb8967fc700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5469df750 0x55d5469e1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.837+0000 7fb8967fc700 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5469df750 0x55d5469e1b60 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ccc5fbfc81507d2 server_cookie=9e8e8c01c6dbecc3 in_seq=0 out_seq=0 2023-07-29T16:30:45.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.837+0000 7fb87aff5700 10 client.30028 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.839 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.839+0000 7fb87aff5700 1 -- 172.21.15.31:0/1813927613 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d546998d70 con 0x55d5469df750 2023-07-29T16:30:45.840 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.839+0000 7fb87aff5700 10 client.30028 handle_command_reply: tid=0 2023-07-29T16:30:45.841 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 client.30028 shutdown 2023-07-29T16:30:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 msgr2=0x55d5469d3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d3990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb888019000 tx=0x7fb8880106b0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 shutdown_connections 2023-07-29T16:30:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5469df750 0x55d5469e1b60 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469daf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d3990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1813927613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.842+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 >> 172.21.15.31:0/1813927613 conn(0x55d5469bf540 msgr2=0x55d5469c41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.843+0000 7fb87aff5700 0 client.30028 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.843+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 shutdown_connections 2023-07-29T16:30:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.843+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1813927613 wait complete. 2023-07-29T16:30:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.843+0000 7fb8b0d6eb80 20 client.30028 trim_cache size 0 max 16384 2023-07-29T16:30:45.865 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.866+0000 7fb8b0d6eb80 1 Processor -- start 2023-07-29T16:30:45.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.866+0000 7fb8b0d6eb80 1 -- start start 2023-07-29T16:30:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.866+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469d8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469d1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5469d49c0 con 0x55d5469cb4d0 2023-07-29T16:30:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5469c3b90 con 0x55d5469c9d40 2023-07-29T16:30:45.867 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5469e1e70 con 0x55d5469ca690 2023-07-29T16:30:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469d8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469d8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38300/0 (socket says 172.21.15.31:38300) 2023-07-29T16:30:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 -- 172.21.15.31:0/4183559922 learned_addr learned my addr 172.21.15.31:0/4183559922 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8967fc700 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 -- 172.21.15.31:0/4183559922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 msgr2=0x55d5469d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8977fe700 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 -- 172.21.15.31:0/4183559922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 msgr2=0x55d5469ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb896ffd700 1 -- 172.21.15.31:0/4183559922 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d54676f250 con 0x55d5469c9d40 2023-07-29T16:30:45.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.867+0000 7fb8977fe700 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:45.871 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.868+0000 7fb896ffd700 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469d8ec0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb88800bc90 tx=0x7fb8880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c235541b6d777880 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.868+0000 7fb87dffb700 1 -- 172.21.15.31:0/4183559922 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88803f070 con 0x55d5469c9d40 2023-07-29T16:30:45.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.868+0000 7fb87dffb700 1 -- 172.21.15.31:0/4183559922 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb888032dd0 con 0x55d5469c9d40 2023-07-29T16:30:45.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.868+0000 7fb87dffb700 1 -- 172.21.15.31:0/4183559922 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb888004f30 con 0x55d5469c9d40 2023-07-29T16:30:45.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/4183559922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 msgr2=0x55d5469d8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469d8ec0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb88800bc90 tx=0x7fb8880046c0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/4183559922 shutdown_connections 2023-07-29T16:30:45.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469c9d40 0x55d5469d8ec0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/4183559922 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469ca690 0x55d5469d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/4183559922 >> 172.21.15.31:0/4183559922 conn(0x55d5469bf540 msgr2=0x55d5469e1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/4183559922 shutdown_connections 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/4183559922 wait complete. 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.869+0000 7fb8b0d6eb80 1 -- start start 2023-07-29T16:30:45.874 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 Processor -- start 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469d8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5469d87d0 con 0x55d5469cb4d0 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5469d8940 con 0x55d5469ca690 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5469d8b00 con 0x55d5469c9d40 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb8b0d6eb80 10 client.0 _ll_register_callbacks cb 0x55d5468c0620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:45.875 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb896ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59486/0 (socket says 172.21.15.31:59486) 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.870+0000 7fb896ffd700 1 -- 172.21.15.31:0/1637707090 learned_addr learned my addr 172.21.15.31:0/1637707090 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb8967fc700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb896ffd700 1 -- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 msgr2=0x55d5469d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb8977fe700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb896ffd700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.876 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb896ffd700 1 -- 172.21.15.31:0/1637707090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 msgr2=0x55d5469d8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb896ffd700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469d8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb896ffd700 1 -- 172.21.15.31:0/1637707090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d546731c60 con 0x55d5469c9d40 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb8977fe700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469d8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb896ffd700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d6d40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb888005780 tx=0x7fb888003ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9555eed3a40fa85c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb87aff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.871+0000 7fb87aff5700 1 -- 172.21.15.31:0/1637707090 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88803f040 con 0x55d5469c9d40 2023-07-29T16:30:45.877 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.872+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb888015070 con 0x55d5469c9d40 2023-07-29T16:30:45.878 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.872+0000 7fb87aff5700 1 -- 172.21.15.31:0/1637707090 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb888017040 con 0x55d5469c9d40 2023-07-29T16:30:45.879 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.872+0000 7fb87aff5700 1 -- 172.21.15.31:0/1637707090 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb888004cd0 con 0x55d5469c9d40 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.872+0000 7fb8b0d6eb80 10 client.30034 fetch_fsmap learned FSMap version 38 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.872+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55d5467472b0 con 0x55d5469c9d40 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb87aff5700 1 -- 172.21.15.31:0/1637707090 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb888015070 con 0x55d5469c9d40 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb87aff5700 1 -- 172.21.15.31:0/1637707090 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb888012780 con 0x55d5469c9d40 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb8b0d6eb80 10 client.30034 fetch_fsmap finished waiting for FSMap version 38 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb8b0d6eb80 10 client.30034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb8b0d6eb80 20 client.30034 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:45.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d546a31da0 0x55d546a341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:45.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb8b0d6eb80 4 client.30034 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:30:45.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.873+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d546913d40 con 0x55d546a31da0 2023-07-29T16:30:45.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.874+0000 7fb8967fc700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d546a31da0 0x55d546a341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:45.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.874+0000 7fb8967fc700 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d546a31da0 0x55d546a341b0 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6cfad53fdbdd52c server_cookie=ca55196c089e7a0 in_seq=0 out_seq=0 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.874+0000 7fb87aff5700 10 client.30034 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.878+0000 7fb87aff5700 1 -- 172.21.15.31:0/1637707090 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d546913d40 con 0x55d546a31da0 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.878+0000 7fb87aff5700 10 client.30034 handle_command_reply: tid=0 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.878+0000 7fb8b0d6eb80 1 client.30034 shutdown 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.878+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 msgr2=0x55d5469d6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.878+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d6d40 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb888005780 tx=0x7fb888003ee0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.882 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 shutdown_connections 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d546a31da0 0x55d546a341b0 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5469cb4d0 0x55d5469d8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d5469c9d40 0x55d5469d6d40 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 --2- 172.21.15.31:0/1637707090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d5469ca690 0x55d5469d7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 >> 172.21.15.31:0/1637707090 conn(0x55d5469bf540 msgr2=0x55d5469d1800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb87aff5700 0 client.30034 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 shutdown_connections 2023-07-29T16:30:45.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.879+0000 7fb8b0d6eb80 1 -- 172.21.15.31:0/1637707090 wait complete. 2023-07-29T16:30:45.884 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:30:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2023-07-29T16:30:45.884 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "fb2849eb-979a-4326-b9ec-44a72fd1dbe4": { 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "fb2849eb-979a-4326-b9ec-44a72fd1dbe4", 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:30:45.885 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:30:45.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.881+0000 7fb8b0d6eb80 20 client.30034 trim_cache size 0 max 16384 2023-07-29T16:30:45.886 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.882+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8800a0a20 msgr2=0x7fb8800a2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.882+0000 7fb897fff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8800a0a20 0x7fb8800a2ed0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb888004c30 tx=0x7fb88800b650 comp rx=0 tx=0).stop 2023-07-29T16:30:45.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.882+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 msgr2=0x7fb8981ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:45.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.882+0000 7fb897fff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 0x7fb8981ac6b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fb89000ef30 tx=0x7fb89000c3a0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.884+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 shutdown_connections 2023-07-29T16:30:45.887 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.884+0000 7fb897fff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb8800a0a20 0x7fb8800a2ed0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.884+0000 7fb897fff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb898103e30 0x7fb8981ac6b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.884+0000 7fb897fff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8980feb10 0x7fb89806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.884+0000 7fb897fff700 1 --2- 172.21.15.31:0/3838267382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb898101470 0x7fb89806fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:45.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.884+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 >> 172.21.15.31:0/3838267382 conn(0x7fb8980fa700 msgr2=0x7fb8980ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:45.888 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.885+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 shutdown_connections 2023-07-29T16:30:45.889 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:30:45.885+0000 7fb897fff700 1 -- 172.21.15.31:0/3838267382 wait complete. 2023-07-29T16:30:45.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:45 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:45.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:45 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:45.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:45 smithi031 ceph-mon[95504]: Scrub error on inode 0x100000045b8 (/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902/client.0/tmp/clients/client3/~dmtmp/PWRPNT/NEWPCB.PPT) see mds.i log and `damage ls` output for details 2023-07-29T16:30:45.913 INFO:tasks.cephfs.filesystem:scrub status for tag:fb2849eb-979a-4326-b9ec-44a72fd1dbe4 - {'path': '/', 'tag': 'fb2849eb-979a-4326-b9ec-44a72fd1dbe4', 'options': 'recursive,force'} 2023-07-29T16:30:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:45 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:46.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:45 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:46.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:45 smithi039 ceph-mon[128858]: Scrub error on inode 0x100000045b8 (/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902/client.0/tmp/clients/client3/~dmtmp/PWRPNT/NEWPCB.PPT) see mds.i log and `damage ls` output for details 2023-07-29T16:30:46.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12157 27.60 MB/sec warmup 30 sec latency 293.970 ms 2023-07-29T16:30:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:46 smithi031 ceph-mon[95504]: scrub queued for path: / 2023-07-29T16:30:46.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:46 smithi031 ceph-mon[95504]: scrub summary: active paths [/] 2023-07-29T16:30:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:46 smithi031 ceph-mon[95504]: pgmap v830: 97 pgs: 97 active+clean; 380 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 412 op/s 2023-07-29T16:30:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:46 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:46 smithi039 ceph-mon[128858]: scrub queued for path: / 2023-07-29T16:30:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:46 smithi039 ceph-mon[128858]: scrub summary: active paths [/] 2023-07-29T16:30:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:46 smithi039 ceph-mon[128858]: pgmap v830: 97 pgs: 97 active+clean; 380 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 412 op/s 2023-07-29T16:30:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:46 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:46 smithi156 ceph-mon[129245]: scrub queued for path: / 2023-07-29T16:30:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:46 smithi156 ceph-mon[129245]: scrub summary: active paths [/] 2023-07-29T16:30:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:46 smithi156 ceph-mon[129245]: pgmap v830: 97 pgs: 97 active+clean; 380 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 412 op/s 2023-07-29T16:30:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:46 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:47.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 12875 28.28 MB/sec warmup 31 sec latency 705.341 ms 2023-07-29T16:30:47.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:47 smithi031 ceph-mon[95504]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:48.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:47 smithi039 ceph-mon[128858]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:48.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:47 smithi156 ceph-mon[129245]: scrub summary: idle+waiting paths [/] 2023-07-29T16:30:48.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 13522 28.18 MB/sec warmup 32 sec latency 41.398 ms 2023-07-29T16:30:48.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:48 smithi031 ceph-mon[95504]: pgmap v831: 97 pgs: 97 active+clean; 386 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 502 op/s 2023-07-29T16:30:49.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:48 smithi039 ceph-mon[128858]: pgmap v831: 97 pgs: 97 active+clean; 386 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 502 op/s 2023-07-29T16:30:49.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:48 smithi156 ceph-mon[129245]: pgmap v831: 97 pgs: 97 active+clean; 386 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 502 op/s 2023-07-29T16:30:49.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 14419 28.64 MB/sec warmup 33 sec latency 69.787 ms 2023-07-29T16:30:50.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 15316 29.23 MB/sec warmup 34 sec latency 71.655 ms 2023-07-29T16:30:50.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:50 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:30:50.644+0000 7fa43fcae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-07-29T16:30:50.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:50 smithi031 ceph-mon[95504]: pgmap v832: 97 pgs: 97 active+clean; 393 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2023-07-29T16:30:51.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:50 smithi039 ceph-mon[128858]: pgmap v832: 97 pgs: 97 active+clean; 393 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2023-07-29T16:30:51.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:50 smithi156 ceph-mon[129245]: pgmap v832: 97 pgs: 97 active+clean; 393 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 610 op/s 2023-07-29T16:30:51.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16018 29.74 MB/sec warmup 35 sec latency 330.831 ms 2023-07-29T16:30:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:51 smithi031 ceph-mon[95504]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-07-29T16:30:51.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:51 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:active 2023-07-29T16:30:51.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:51 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:52.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:51 smithi039 ceph-mon[128858]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-07-29T16:30:52.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:51 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:active 2023-07-29T16:30:52.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:51 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:52.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:51 smithi156 ceph-mon[129245]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2023-07-29T16:30:52.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:51 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] up:active 2023-07-29T16:30:52.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:51 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:30:52.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 16724 30.00 MB/sec warmup 36 sec latency 64.521 ms 2023-07-29T16:30:52.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:52 smithi031 ceph-mon[95504]: pgmap v833: 97 pgs: 97 active+clean; 393 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 533 op/s 2023-07-29T16:30:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:52 smithi039 ceph-mon[128858]: pgmap v833: 97 pgs: 97 active+clean; 393 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 533 op/s 2023-07-29T16:30:53.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:52 smithi156 ceph-mon[129245]: pgmap v833: 97 pgs: 97 active+clean; 393 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 533 op/s 2023-07-29T16:30:53.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 17439 30.03 MB/sec warmup 37 sec latency 61.177 ms 2023-07-29T16:30:53.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:30:53.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:30:53.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:53 smithi031 ceph-mon[95504]: scrub summary: idle 2023-07-29T16:30:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:30:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:30:54.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:53 smithi039 ceph-mon[128858]: scrub summary: idle 2023-07-29T16:30:54.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:30:54.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:30:54.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:53 smithi156 ceph-mon[129245]: scrub summary: idle 2023-07-29T16:30:54.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 18335 30.56 MB/sec warmup 38 sec latency 47.939 ms 2023-07-29T16:30:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:54 smithi039 ceph-mon[128858]: pgmap v834: 97 pgs: 97 active+clean; 423 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 6.0 MiB/s wr, 739 op/s 2023-07-29T16:30:55.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:54 smithi156 ceph-mon[129245]: pgmap v834: 97 pgs: 97 active+clean; 423 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 6.0 MiB/s wr, 739 op/s 2023-07-29T16:30:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 -- 172.21.15.31:0/2007044260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 msgr2=0x7f9ff810ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/2007044260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 0x7f9ff810ea50 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f9ff001aed0 tx=0x7f9ff001c330 comp rx=0 tx=0).stop 2023-07-29T16:30:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 -- 172.21.15.31:0/2007044260 shutdown_connections 2023-07-29T16:30:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/2007044260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 0x7f9ff810ea50 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/2007044260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8071740 0x7f9ff8071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/2007044260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8072090 0x7f9ff810c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.174+0000 7f9ffcfff700 1 -- 172.21.15.31:0/2007044260 >> 172.21.15.31:0/2007044260 conn(0x7f9ff806d290 msgr2=0x7f9ff806f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.176+0000 7f9ffcfff700 1 -- 172.21.15.31:0/2007044260 shutdown_connections 2023-07-29T16:30:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.176+0000 7f9ffcfff700 1 -- 172.21.15.31:0/2007044260 wait complete. 2023-07-29T16:30:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.177+0000 7f9ffcfff700 1 Processor -- start 2023-07-29T16:30:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 -- start start 2023-07-29T16:30:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 0x7f9ff819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8072090 0x7f9ff819d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 0x7f9ff819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff819e2c0 con 0x7f9ff8072090 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff819e430 con 0x7f9ff8071740 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ffcfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff819e5f0 con 0x7f9ff810c5a0 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ff659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 0x7f9ff819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ff659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 0x7f9ff819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55172/0 (socket says 172.21.15.31:55172) 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.178+0000 7f9ff659c700 1 -- 172.21.15.31:0/937331762 learned_addr learned my addr 172.21.15.31:0/937331762 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.179+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8072090 0x7f9ff819d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff659c700 1 -- 172.21.15.31:0/937331762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 msgr2=0x7f9ff819db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff659c700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 0x7f9ff819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff659c700 1 -- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8072090 msgr2=0x7f9ff819d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff659c700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8072090 0x7f9ff819d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff659c700 1 -- 172.21.15.31:0/937331762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0005e70 con 0x7f9ff8071740 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8072090 0x7f9ff819d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ff659c700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 0x7f9ff819d000 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800b530 tx=0x7f9fe800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:55.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9fef7fe700 1 -- 172.21.15.31:0/937331762 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8011830 con 0x7f9ff8071740 2023-07-29T16:30:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9fef7fe700 1 -- 172.21.15.31:0/937331762 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fe8011e50 con 0x7f9ff8071740 2023-07-29T16:30:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9fef7fe700 1 -- 172.21.15.31:0/937331762 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe800f2e0 con 0x7f9ff8071740 2023-07-29T16:30:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.180+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff819e8d0 con 0x7f9ff8071740 2023-07-29T16:30:55.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.181+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff81a50c0 con 0x7f9ff8071740 2023-07-29T16:30:55.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:54 smithi031 ceph-mon[95504]: pgmap v834: 97 pgs: 97 active+clean; 423 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 6.0 MiB/s wr, 739 op/s 2023-07-29T16:30:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.186+0000 7f9fef7fe700 1 -- 172.21.15.31:0/937331762 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9fe8010380 con 0x7f9ff8071740 2023-07-29T16:30:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.187+0000 7fa010a15b80 1 Processor -- start 2023-07-29T16:30:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.187+0000 7fa010a15b80 1 -- start start 2023-07-29T16:30:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.187+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c54f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5507b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 0x55928c551160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7fa010a15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c4a79b0 con 0x55928c3ab6b0 2023-07-29T16:30:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7fa010a15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c4a5370 con 0x55928c550cf0 2023-07-29T16:30:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7fa010a15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c4a0c60 con 0x55928c54fdf0 2023-07-29T16:30:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 0x55928c551160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 0x55928c551160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55180/0 (socket says 172.21.15.31:55180) 2023-07-29T16:30:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1055453419 learned_addr learned my addr 172.21.15.31:0/1055453419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c54f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1055453419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 msgr2=0x55928c5507b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:30:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5507b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1055453419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 msgr2=0x55928c54f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c54f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1055453419 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55928c2ced80 con 0x55928c550cf0 2023-07-29T16:30:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.188+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c54f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:30:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.189+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 0x55928c551160 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009e50 tx=0x7f9fe400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb84bbff2094e3e5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:55.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.189+0000 7f9fef7fe700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fdc0a0810 0x7f9fdc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.189+0000 7f9fecff9700 1 -- 172.21.15.31:0/1055453419 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe400ecd0 con 0x55928c550cf0 2023-07-29T16:30:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.189+0000 7f9fef7fe700 1 -- 172.21.15.31:0/937331762 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9fe80cb580 con 0x7f9ff8071740 2023-07-29T16:30:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.189+0000 7f9fecff9700 1 -- 172.21.15.31:0/1055453419 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fe400ee30 con 0x55928c550cf0 2023-07-29T16:30:55.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.189+0000 7f9fecff9700 1 -- 172.21.15.31:0/1055453419 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4012790 con 0x55928c550cf0 2023-07-29T16:30:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.190+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fdc0a0810 0x7f9fdc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.190+0000 7fa010a15b80 1 -- 172.21.15.31:0/1055453419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 msgr2=0x55928c551160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 0x55928c551160 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009e50 tx=0x7f9fe400e4b0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fdc0a0810 0x7f9fdc0a2cc0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9ff001afd0 tx=0x7f9ff001c750 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:55.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 -- 172.21.15.31:0/1055453419 shutdown_connections 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c550cf0 0x55928c551160 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c54f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1055453419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5507b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 -- 172.21.15.31:0/1055453419 >> 172.21.15.31:0/1055453419 conn(0x55928c544fd0 msgr2=0x55928c54a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 -- 172.21.15.31:0/1055453419 shutdown_connections 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.191+0000 7fa010a15b80 1 -- 172.21.15.31:0/1055453419 wait complete. 2023-07-29T16:30:55.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 -- start start 2023-07-29T16:30:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 Processor -- start 2023-07-29T16:30:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c3ab6b0 0x55928c558f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5594a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c5604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c43d4b0 con 0x55928c550cf0 2023-07-29T16:30:55.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c43d150 con 0x55928c3ab6b0 2023-07-29T16:30:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c3248c0 con 0x55928c54fdf0 2023-07-29T16:30:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.192+0000 7fa010a15b80 10 client.0 _ll_register_callbacks cb 0x55928c4465c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.193+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.193+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:38150/0 (socket says 172.21.15.31:38150) 2023-07-29T16:30:55.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.193+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1716091538 learned_addr learned my addr 172.21.15.31:0/1716091538 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:55.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.193+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c3ab6b0 msgr2=0x55928c558f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c3ab6b0 0x55928c558f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1716091538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 msgr2=0x55928c5604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c5604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/1716091538 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe4009a80 con 0x55928c54fdf0 2023-07-29T16:30:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5594a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009390 tx=0x7f9fe4013c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faf1f19158911c1a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:55.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9fd9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/1716091538 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4006dc0 con 0x55928c54fdf0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.194+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/1716091538 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9fe40126e0 con 0x55928c54fdf0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.195+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55928c4cbcb0 con 0x55928c54fdf0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.195+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/1716091538 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fe40128a0 con 0x55928c54fdf0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.195+0000 7fa010a15b80 10 client.30040 fetch_fsmap learned FSMap version 39 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.195+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55928c2b7c00 con 0x55928c54fdf0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.195+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/1716091538 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55928c4cbcb0 con 0x55928c54fdf0 2023-07-29T16:30:55.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.195+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/1716091538 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9fe402c070 con 0x55928c54fdf0 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7fa010a15b80 10 client.30040 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7fa010a15b80 10 client.30040 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7fa010a15b80 20 client.30040 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c564c70 0x55928c567080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7fa010a15b80 4 client.30040 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55928c51ed10 con 0x55928c564c70 2023-07-29T16:30:55.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7f9ff659c700 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c564c70 0x55928c567080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7f9ff659c700 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c564c70 0x55928c567080 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c67d655a0b1fb61b server_cookie=772bf9e5f3986f6 in_seq=0 out_seq=0 2023-07-29T16:30:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.196+0000 7f9fd9ffb700 10 client.30040 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.199+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/1716091538 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55928c51ed10 con 0x55928c564c70 2023-07-29T16:30:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.199+0000 7f9fd9ffb700 10 client.30040 handle_command_reply: tid=0 2023-07-29T16:30:55.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.199+0000 7fa010a15b80 1 client.30040 shutdown 2023-07-29T16:30:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.199+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 msgr2=0x55928c5594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.199+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5594a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009390 tx=0x7f9fe4013c30 comp rx=0 tx=0).stop 2023-07-29T16:30:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 shutdown_connections 2023-07-29T16:30:55.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c564c70 0x55928c567080 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c5604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c3ab6b0 0x55928c558f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 --2- 172.21.15.31:0/1716091538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c54fdf0 0x55928c5594a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 >> 172.21.15.31:0/1716091538 conn(0x55928c544fd0 msgr2=0x55928c549a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7f9fd9ffb700 0 client.30040 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 shutdown_connections 2023-07-29T16:30:55.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 1 -- 172.21.15.31:0/1716091538 wait complete. 2023-07-29T16:30:55.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.200+0000 7fa010a15b80 20 client.30040 trim_cache size 0 max 16384 2023-07-29T16:30:55.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 19021 30.74 MB/sec warmup 39 sec latency 372.127 ms 2023-07-29T16:30:55.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.227+0000 7fa010a15b80 1 Processor -- start 2023-07-29T16:30:55.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 -- start start 2023-07-29T16:30:55.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c3ab6b0 0x55928c5510e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c5604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c566c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c5546f0 con 0x55928c550cf0 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c564a00 con 0x55928c54fdf0 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7fa010a15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c56bc30 con 0x55928c3ab6b0 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.228+0000 7f9ff6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c566c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c566c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42834/0 (socket says 172.21.15.31:42834) 2023-07-29T16:30:55.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3246715122 learned_addr learned my addr 172.21.15.31:0/3246715122 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff5d9b700 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c5604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff659c700 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c3ab6b0 0x55928c5510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3246715122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c3ab6b0 msgr2=0x55928c5510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c3ab6b0 0x55928c5510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3246715122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 msgr2=0x55928c5604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c5604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.229+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3246715122 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55928c2f51f0 con 0x55928c550cf0 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.230+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c566c80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0018540 tx=0x7f9ff001af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c64425ea32815e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:55.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.231+0000 7f9fed7fa700 1 -- 172.21.15.31:0/3246715122 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff001cc80 con 0x55928c550cf0 2023-07-29T16:30:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.231+0000 7f9fed7fa700 1 -- 172.21.15.31:0/3246715122 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ff001cde0 con 0x55928c550cf0 2023-07-29T16:30:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.231+0000 7f9fed7fa700 1 -- 172.21.15.31:0/3246715122 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff0039680 con 0x55928c550cf0 2023-07-29T16:30:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 -- 172.21.15.31:0/3246715122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 msgr2=0x55928c566c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c566c80 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f9ff0018540 tx=0x7f9ff001af60 comp rx=0 tx=0).stop 2023-07-29T16:30:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 -- 172.21.15.31:0/3246715122 shutdown_connections 2023-07-29T16:30:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c550cf0 0x55928c566c80 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c3ab6b0 0x55928c5510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3246715122 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c5604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.232+0000 7fa010a15b80 1 -- 172.21.15.31:0/3246715122 >> 172.21.15.31:0/3246715122 conn(0x55928c544fd0 msgr2=0x55928c557c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.234+0000 7fa010a15b80 1 -- 172.21.15.31:0/3246715122 shutdown_connections 2023-07-29T16:30:55.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.234+0000 7fa010a15b80 1 -- 172.21.15.31:0/3246715122 wait complete. 2023-07-29T16:30:55.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.234+0000 7fa010a15b80 1 -- start start 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 Processor -- start 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c55c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c55cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c550cf0 0x55928c55fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c560130 con 0x55928c3ab6b0 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c5602a0 con 0x55928c54fdf0 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55928c560460 con 0x55928c550cf0 2023-07-29T16:30:55.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.235+0000 7fa010a15b80 10 client.0 _ll_register_callbacks cb 0x55928c4465c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.236+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c55cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.236+0000 7f9ff5d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c55cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55208/0 (socket says 172.21.15.31:55208) 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.236+0000 7f9ff5d9b700 1 -- 172.21.15.31:0/3746524586 learned_addr learned my addr 172.21.15.31:0/3746524586 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.236+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c550cf0 0x55928c55fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.237+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 msgr2=0x55928c55cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.237+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c55cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.237+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3746524586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 msgr2=0x55928c55c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:30:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.237+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c55c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.237+0000 7f9ff6d9d700 1 -- 172.21.15.31:0/3746524586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff8117e80 con 0x55928c550cf0 2023-07-29T16:30:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.238+0000 7f9ff6d9d700 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c550cf0 0x55928c55fbf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9ff002cdb0 tx=0x7f9ff001a790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5979e38a3cad07ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:30:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.238+0000 7f9fd9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:30:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.238+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/3746524586 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00186d0 con 0x55928c550cf0 2023-07-29T16:30:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.238+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/3746524586 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9ff0034050 con 0x55928c550cf0 2023-07-29T16:30:55.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.239+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fe400ee30 con 0x55928c550cf0 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.239+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/3746524586 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ff002cae0 con 0x55928c550cf0 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.239+0000 7fa010a15b80 10 client.30046 fetch_fsmap learned FSMap version 39 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.239+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55928c2cd250 con 0x55928c550cf0 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/3746524586 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fe400ee30 con 0x55928c550cf0 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/3746524586 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9ff00425e0 con 0x55928c550cf0 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7fa010a15b80 10 client.30046 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:30:55.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7fa010a15b80 10 client.30046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:30:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7fa010a15b80 20 client.30046 populate_metadata read hostname 'smithi031' 2023-07-29T16:30:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c5b77b0 0x55928c5b9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:30:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7fa010a15b80 4 client.30046 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:30:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.240+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55928c499ce0 con 0x55928c5b77b0 2023-07-29T16:30:55.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.241+0000 7f9ff659c700 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c5b77b0 0x55928c5b9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:30:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.241+0000 7f9ff659c700 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c5b77b0 0x55928c5b9bc0 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b55733660b4bd4 server_cookie=98f12b291ecf1eec in_seq=0 out_seq=0 2023-07-29T16:30:55.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.241+0000 7f9fd9ffb700 10 client.30046 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.516+0000 7f9fd9ffb700 1 -- 172.21.15.31:0/3746524586 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55928c499ce0 con 0x55928c5b77b0 2023-07-29T16:30:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.516+0000 7f9fd9ffb700 10 client.30046 handle_command_reply: tid=0 2023-07-29T16:30:55.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.516+0000 7fa010a15b80 1 client.30046 shutdown 2023-07-29T16:30:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.517+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c550cf0 msgr2=0x55928c55fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.517+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c550cf0 0x55928c55fbf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f9ff002cdb0 tx=0x7f9ff001a790 comp rx=0 tx=0).stop 2023-07-29T16:30:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.520+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 shutdown_connections 2023-07-29T16:30:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.520+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55928c5b77b0 0x55928c5b9bc0 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.520+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55928c550cf0 0x55928c55fbf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.520+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55928c3ab6b0 0x55928c55c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.520+0000 7fa010a15b80 1 --2- 172.21.15.31:0/3746524586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55928c54fdf0 0x55928c55cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.520+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 >> 172.21.15.31:0/3746524586 conn(0x55928c544fd0 msgr2=0x55928c553b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.521+0000 7f9fd9ffb700 0 client.30046 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:30:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.522+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 shutdown_connections 2023-07-29T16:30:55.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.522+0000 7fa010a15b80 1 -- 172.21.15.31:0/3746524586 wait complete. 2023-07-29T16:30:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.523+0000 7fa010a15b80 20 client.30046 trim_cache size 0 max 16384 2023-07-29T16:30:55.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:30:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:30:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:30:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:30:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fdc0a0810 msgr2=0x7f9fdc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fdc0a0810 0x7f9fdc0a2cc0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f9ff001afd0 tx=0x7f9ff001c750 comp rx=0 tx=0).stop 2023-07-29T16:30:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 msgr2=0x7f9ff819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:30:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 0x7f9ff819d000 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800b530 tx=0x7f9fe800b840 comp rx=0 tx=0).stop 2023-07-29T16:30:55.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 shutdown_connections 2023-07-29T16:30:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fdc0a0810 0x7f9fdc0a2cc0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ff810c5a0 0x7f9ff819db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ff8071740 0x7f9ff819d000 secure :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800b530 tx=0x7f9fe800b840 comp rx=0 tx=0).stop 2023-07-29T16:30:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 --2- 172.21.15.31:0/937331762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9ff8072090 0x7f9ff819d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:30:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.524+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 >> 172.21.15.31:0/937331762 conn(0x7f9ff806d290 msgr2=0x7f9ff806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:30:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.525+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 shutdown_connections 2023-07-29T16:30:55.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:30:55.525+0000 7f9ffcfff700 1 -- 172.21.15.31:0/937331762 wait complete. 2023-07-29T16:30:56.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 19789 30.81 MB/sec warmup 40 sec latency 138.422 ms 2023-07-29T16:30:57.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:56 smithi039 ceph-mon[128858]: pgmap v835: 97 pgs: 97 active+clean; 445 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.2 MiB/s wr, 607 op/s 2023-07-29T16:30:57.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:56 smithi156 ceph-mon[129245]: pgmap v835: 97 pgs: 97 active+clean; 445 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.2 MiB/s wr, 607 op/s 2023-07-29T16:30:57.184 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:56 smithi031 ceph-mon[95504]: pgmap v835: 97 pgs: 97 active+clean; 445 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.2 MiB/s wr, 607 op/s 2023-07-29T16:30:57.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 20734 31.52 MB/sec warmup 41 sec latency 64.715 ms 2023-07-29T16:30:58.220 INFO:tasks.workunit.client.0.smithi031.stdout: 10 21567 31.98 MB/sec warmup 42 sec latency 109.856 ms 2023-07-29T16:30:58.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:58 smithi156 ceph-mon[129245]: pgmap v836: 97 pgs: 97 active+clean; 442 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.2 MiB/s wr, 542 op/s 2023-07-29T16:30:58.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:58 smithi031 ceph-mon[95504]: pgmap v836: 97 pgs: 97 active+clean; 442 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.2 MiB/s wr, 542 op/s 2023-07-29T16:30:58.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:58 smithi039 ceph-mon[128858]: pgmap v836: 97 pgs: 97 active+clean; 442 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 7.2 MiB/s wr, 542 op/s 2023-07-29T16:30:59.221 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22208 31.87 MB/sec warmup 43 sec latency 92.175 ms 2023-07-29T16:30:59.322 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:30:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:30:59.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:30:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:30:59.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:30:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:31:00.221 INFO:tasks.workunit.client.0.smithi031.stdout: 10 22964 32.03 MB/sec warmup 44 sec latency 149.477 ms 2023-07-29T16:31:00.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:31:00.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:31:00.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:31:00.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:00 smithi156 ceph-mon[129245]: pgmap v837: 97 pgs: 97 active+clean; 450 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.6 MiB/s wr, 747 op/s 2023-07-29T16:31:00.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:31:00.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:31:00.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:31:00.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:00 smithi031 ceph-mon[95504]: pgmap v837: 97 pgs: 97 active+clean; 450 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.6 MiB/s wr, 747 op/s 2023-07-29T16:31:00.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:31:00.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:31:00.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:31:00.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:00 smithi039 ceph-mon[128858]: pgmap v837: 97 pgs: 97 active+clean; 450 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.6 MiB/s wr, 747 op/s 2023-07-29T16:31:01.221 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24103 32.82 MB/sec warmup 45 sec latency 53.160 ms 2023-07-29T16:31:02.221 INFO:tasks.workunit.client.0.smithi031.stdout: 10 24913 33.08 MB/sec warmup 46 sec latency 70.963 ms 2023-07-29T16:31:02.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:02 smithi031 ceph-mon[95504]: pgmap v838: 97 pgs: 97 active+clean; 450 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.7 MiB/s wr, 632 op/s 2023-07-29T16:31:03.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:02 smithi039 ceph-mon[128858]: pgmap v838: 97 pgs: 97 active+clean; 450 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.7 MiB/s wr, 632 op/s 2023-07-29T16:31:03.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:02 smithi156 ceph-mon[129245]: pgmap v838: 97 pgs: 97 active+clean; 450 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 7.7 MiB/s wr, 632 op/s 2023-07-29T16:31:03.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 25722 33.21 MB/sec warmup 47 sec latency 58.254 ms 2023-07-29T16:31:04.221 INFO:tasks.workunit.client.0.smithi031.stdout: 10 26704 33.56 MB/sec warmup 48 sec latency 46.573 ms 2023-07-29T16:31:04.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:04 smithi031 ceph-mon[95504]: pgmap v839: 97 pgs: 97 active+clean; 464 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 800 op/s 2023-07-29T16:31:05.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:04 smithi039 ceph-mon[128858]: pgmap v839: 97 pgs: 97 active+clean; 464 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 800 op/s 2023-07-29T16:31:05.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:04 smithi156 ceph-mon[129245]: pgmap v839: 97 pgs: 97 active+clean; 464 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 800 op/s 2023-07-29T16:31:05.221 INFO:tasks.workunit.client.0.smithi031.stdout: 10 27399 33.72 MB/sec warmup 49 sec latency 245.269 ms 2023-07-29T16:31:06.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 28238 33.79 MB/sec warmup 50 sec latency 227.282 ms 2023-07-29T16:31:06.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:06 smithi031 ceph-mon[95504]: pgmap v840: 97 pgs: 97 active+clean; 470 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.3 MiB/s wr, 682 op/s 2023-07-29T16:31:07.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:06 smithi039 ceph-mon[128858]: pgmap v840: 97 pgs: 97 active+clean; 470 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.3 MiB/s wr, 682 op/s 2023-07-29T16:31:07.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:06 smithi156 ceph-mon[129245]: pgmap v840: 97 pgs: 97 active+clean; 470 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.3 MiB/s wr, 682 op/s 2023-07-29T16:31:07.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 29210 34.25 MB/sec warmup 51 sec latency 39.666 ms 2023-07-29T16:31:08.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30010 34.52 MB/sec warmup 52 sec latency 62.842 ms 2023-07-29T16:31:08.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:08 smithi156 ceph-mon[129245]: pgmap v841: 97 pgs: 97 active+clean; 476 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.0 MiB/s wr, 740 op/s 2023-07-29T16:31:08.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:08 smithi031 ceph-mon[95504]: pgmap v841: 97 pgs: 97 active+clean; 476 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.0 MiB/s wr, 740 op/s 2023-07-29T16:31:08.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:08 smithi039 ceph-mon[128858]: pgmap v841: 97 pgs: 97 active+clean; 476 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.0 MiB/s wr, 740 op/s 2023-07-29T16:31:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.767+0000 7f607dbff700 1 -- 172.21.15.31:0/2578793882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6078107f60 msgr2=0x7f60781083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:08.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.767+0000 7f607dbff700 1 --2- 172.21.15.31:0/2578793882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6078107f60 0x7f60781083b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f606000ba80 tx=0x7f606000bd90 comp rx=0 tx=0).stop 2023-07-29T16:31:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.770+0000 7f607dbff700 1 -- 172.21.15.31:0/2578793882 shutdown_connections 2023-07-29T16:31:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.770+0000 7f607dbff700 1 --2- 172.21.15.31:0/2578793882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60781088f0 0x7f607810f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.770+0000 7f607dbff700 1 --2- 172.21.15.31:0/2578793882 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078071fa0 0x7f60780723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.770+0000 7f607dbff700 1 --2- 172.21.15.31:0/2578793882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6078107f60 0x7f60781083b0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.770+0000 7f607dbff700 1 -- 172.21.15.31:0/2578793882 >> 172.21.15.31:0/2578793882 conn(0x7f607806d5c0 msgr2=0x7f607806f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.772+0000 7f607dbff700 1 -- 172.21.15.31:0/2578793882 shutdown_connections 2023-07-29T16:31:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.772+0000 7f607dbff700 1 -- 172.21.15.31:0/2578793882 wait complete. 2023-07-29T16:31:08.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.774+0000 7f607dbff700 1 Processor -- start 2023-07-29T16:31:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.774+0000 7f607dbff700 1 -- start start 2023-07-29T16:31:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.774+0000 7f607dbff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6078071fa0 0x7f60781a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f607dbff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078107f60 0x7f60781a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f607dbff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60781088f0 0x7f60781a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f607dbff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60781a6a50 con 0x7f6078107f60 2023-07-29T16:31:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f607dbff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60781a6bc0 con 0x7f60781088f0 2023-07-29T16:31:08.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f607dbff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60781a6d80 con 0x7f6078071fa0 2023-07-29T16:31:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f60777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6078071fa0 0x7f60781a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f60777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6078071fa0 0x7f60781a5030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58822/0 (socket says 172.21.15.31:58822) 2023-07-29T16:31:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f60777fe700 1 -- 172.21.15.31:0/2957160016 learned_addr learned my addr 172.21.15.31:0/2957160016 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f6076ffd700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078107f60 0x7f60781a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.775+0000 7f6077fff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60781088f0 0x7f60781a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6076ffd700 1 -- 172.21.15.31:0/2957160016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6078071fa0 msgr2=0x7f60781a5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6076ffd700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6078071fa0 0x7f60781a5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6076ffd700 1 -- 172.21.15.31:0/2957160016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60781088f0 msgr2=0x7f60781a63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6076ffd700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60781088f0 0x7f60781a63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6076ffd700 1 -- 172.21.15.31:0/2957160016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f606000b730 con 0x7f6078107f60 2023-07-29T16:31:08.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6076ffd700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078107f60 0x7f60781a5570 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f606000ba80 tx=0x7f6060014750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.776+0000 7f6074ff9700 1 -- 172.21.15.31:0/2957160016 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060010040 con 0x7f6078107f60 2023-07-29T16:31:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.777+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60781a7000 con 0x7f6078107f60 2023-07-29T16:31:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.777+0000 7f6074ff9700 1 -- 172.21.15.31:0/2957160016 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6060005700 con 0x7f6078107f60 2023-07-29T16:31:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.777+0000 7f6074ff9700 1 -- 172.21.15.31:0/2957160016 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606001f6a0 con 0x7f6078107f60 2023-07-29T16:31:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.777+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60781ad820 con 0x7f6078107f60 2023-07-29T16:31:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.785+0000 7f6091502b80 1 Processor -- start 2023-07-29T16:31:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.786+0000 7f6074ff9700 1 -- 172.21.15.31:0/2957160016 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f606001f800 con 0x7f6078107f60 2023-07-29T16:31:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.786+0000 7f6091502b80 1 -- start start 2023-07-29T16:31:08.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.787+0000 7f6091502b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56afb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.787+0000 7f6091502b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afbdf0 0x557b56afc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.787+0000 7f6091502b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56afd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.787+0000 7f6091502b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56a539b0 con 0x557b569576b0 2023-07-29T16:31:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.787+0000 7f6091502b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56a51370 con 0x557b56afccf0 2023-07-29T16:31:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.787+0000 7f6091502b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56a4cc60 con 0x557b56afbdf0 2023-07-29T16:31:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.788+0000 7f6077fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56afb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.788+0000 7f6077fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56afb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40094/0 (socket says 172.21.15.31:40094) 2023-07-29T16:31:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.788+0000 7f6077fff700 1 -- 172.21.15.31:0/1807078247 learned_addr learned my addr 172.21.15.31:0/1807078247 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.789+0000 7f60777fe700 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afbdf0 0x557b56afc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.789+0000 7f6076ffd700 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56afd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.789+0000 7f6074ff9700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60640a0770 0x7f60640a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6077fff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60640a0770 0x7f60640a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6074ff9700 1 -- 172.21.15.31:0/2957160016 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f60600cd8e0 con 0x7f6078107f60 2023-07-29T16:31:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6077fff700 1 -- 172.21.15.31:0/1807078247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afbdf0 msgr2=0x557b56afc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6077fff700 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afbdf0 0x557b56afc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6077fff700 1 -- 172.21.15.31:0/1807078247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 msgr2=0x557b56afd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6077fff700 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56afd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.790+0000 7f6077fff700 1 -- 172.21.15.31:0/1807078247 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b5687ad80 con 0x557b569576b0 2023-07-29T16:31:08.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.791+0000 7f6077fff700 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56afb8b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f607001a550 tx=0x7f607001a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8289a690ac0868b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.792+0000 7f606dffb700 1 -- 172.21.15.31:0/1807078247 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070028070 con 0x557b569576b0 2023-07-29T16:31:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.792+0000 7f6077fff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60640a0770 0x7f60640a2c20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6070023000 tx=0x7f607001f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.792+0000 7f606dffb700 1 -- 172.21.15.31:0/1807078247 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6070003e20 con 0x557b569576b0 2023-07-29T16:31:08.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.792+0000 7f606dffb700 1 -- 172.21.15.31:0/1807078247 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607000f040 con 0x557b569576b0 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 -- 172.21.15.31:0/1807078247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 msgr2=0x557b56afb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56afb8b0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f607001a550 tx=0x7f607001a860 comp rx=0 tx=0).stop 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 -- 172.21.15.31:0/1807078247 shutdown_connections 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56afd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56afb8b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 --2- 172.21.15.31:0/1807078247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afbdf0 0x557b56afc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.793+0000 7f6091502b80 1 -- 172.21.15.31:0/1807078247 >> 172.21.15.31:0/1807078247 conn(0x557b56af0fd0 msgr2=0x557b56af6300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:08.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.794+0000 7f6091502b80 1 -- 172.21.15.31:0/1807078247 shutdown_connections 2023-07-29T16:31:08.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.794+0000 7f6091502b80 1 -- 172.21.15.31:0/1807078247 wait complete. 2023-07-29T16:31:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.795+0000 7f6091502b80 1 -- start start 2023-07-29T16:31:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.795+0000 7f6091502b80 1 Processor -- start 2023-07-29T16:31:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b15ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b16400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b569e94b0 con 0x557b56afbdf0 2023-07-29T16:31:08.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b569e9150 con 0x557b56afccf0 2023-07-29T16:31:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b568d08c0 con 0x557b569576b0 2023-07-29T16:31:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6091502b80 10 client.0 _ll_register_callbacks cb 0x557b569f25c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f60777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b15ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f60777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b15ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58842/0 (socket says 172.21.15.31:58842) 2023-07-29T16:31:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f60777fe700 1 -- 172.21.15.31:0/3269412660 learned_addr learned my addr 172.21.15.31:0/3269412660 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6076ffd700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b16400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f6077fff700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.796+0000 7f60777fe700 1 -- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 msgr2=0x557b56b1d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f60777fe700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f60777fe700 1 -- 172.21.15.31:0/3269412660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 msgr2=0x557b56b16400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f60777fe700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b16400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f60777fe700 1 -- 172.21.15.31:0/3269412660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60700047f0 con 0x557b569576b0 2023-07-29T16:31:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f60777fe700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b15ec0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f60680118b0 tx=0x7f6068011bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b8043d075c9a668 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f6056ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f6056ffd700 1 -- 172.21.15.31:0/3269412660 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60680136a0 con 0x557b569576b0 2023-07-29T16:31:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b56a77cb0 con 0x557b569576b0 2023-07-29T16:31:08.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.797+0000 7f6056ffd700 1 -- 172.21.15.31:0/3269412660 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6068013800 con 0x557b569576b0 2023-07-29T16:31:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6056ffd700 1 -- 172.21.15.31:0/3269412660 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60680139c0 con 0x557b569576b0 2023-07-29T16:31:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 10 client.30058 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x557b56863c00 con 0x557b569576b0 2023-07-29T16:31:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6056ffd700 1 -- 172.21.15.31:0/3269412660 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557b56a77cb0 con 0x557b569576b0 2023-07-29T16:31:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6056ffd700 1 -- 172.21.15.31:0/3269412660 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f606801b020 con 0x557b569576b0 2023-07-29T16:31:08.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 10 client.30058 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 10 client.30058 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 20 client.30058 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b21d60 0x557b56b24170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 4 client.30058 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b56acad10 con 0x557b56b21d60 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.798+0000 7f6076ffd700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b21d60 0x557b56b24170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.799+0000 7f6076ffd700 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b21d60 0x557b56b24170 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffc3867c7c5a6df9 server_cookie=9e2b2cd5d06ee5a5 in_seq=0 out_seq=0 2023-07-29T16:31:08.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.799+0000 7f6056ffd700 10 client.30058 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.804+0000 7f6056ffd700 1 -- 172.21.15.31:0/3269412660 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557b56acad10 con 0x557b56b21d60 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.804+0000 7f6056ffd700 10 client.30058 handle_command_reply: tid=0 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 client.30058 shutdown 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 msgr2=0x557b56b15ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b15ec0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f60680118b0 tx=0x7f6068011bc0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 shutdown_connections 2023-07-29T16:31:08.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b21d60 0x557b56b24170 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b15ec0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 --2- 172.21.15.31:0/3269412660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b16400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.805+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 >> 172.21.15.31:0/3269412660 conn(0x557b56af0fd0 msgr2=0x557b56af5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.806+0000 7f6056ffd700 0 client.30058 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.806+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 shutdown_connections 2023-07-29T16:31:08.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.806+0000 7f6091502b80 1 -- 172.21.15.31:0/3269412660 wait complete. 2023-07-29T16:31:08.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.806+0000 7f6091502b80 20 client.30058 trim_cache size 0 max 16384 2023-07-29T16:31:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 Processor -- start 2023-07-29T16:31:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 -- start start 2023-07-29T16:31:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56b1b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 0x557b56b23d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afccf0 0x557b56b03030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56b14a10 con 0x557b569576b0 2023-07-29T16:31:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56b1e3e0 con 0x557b56afbdf0 2023-07-29T16:31:08.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6091502b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56af7b30 con 0x557b56afccf0 2023-07-29T16:31:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6076ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 0x557b56b23d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6076ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 0x557b56b23d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:36374/0 (socket says 172.21.15.31:36374) 2023-07-29T16:31:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.830+0000 7f6076ffd700 1 -- 172.21.15.31:0/4258805547 learned_addr learned my addr 172.21.15.31:0/4258805547 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6077fff700 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afccf0 0x557b56b03030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f60777fe700 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56b1b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6076ffd700 1 -- 172.21.15.31:0/4258805547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afccf0 msgr2=0x557b56b03030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6076ffd700 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afccf0 0x557b56b03030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6076ffd700 1 -- 172.21.15.31:0/4258805547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 msgr2=0x557b56b1b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6076ffd700 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56b1b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6076ffd700 1 -- 172.21.15.31:0/4258805547 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b568a11f0 con 0x557b56afbdf0 2023-07-29T16:31:08.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f60777fe700 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56b1b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:08.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f6076ffd700 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 0x557b56b23d70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6060005e90 tx=0x7f6060016040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8440ff407d6a1d59 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:08.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f606e7fc700 1 -- 172.21.15.31:0/4258805547 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6060013ea0 con 0x557b56afbdf0 2023-07-29T16:31:08.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f606e7fc700 1 -- 172.21.15.31:0/4258805547 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60600a3430 con 0x557b56afbdf0 2023-07-29T16:31:08.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.831+0000 7f606e7fc700 1 -- 172.21.15.31:0/4258805547 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60600cdba0 con 0x557b56afbdf0 2023-07-29T16:31:08.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 -- 172.21.15.31:0/4258805547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 msgr2=0x557b56b23d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:08.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 0x557b56b23d70 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6060005e90 tx=0x7f6060016040 comp rx=0 tx=0).stop 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 -- 172.21.15.31:0/4258805547 shutdown_connections 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b56afccf0 0x557b56b03030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b569576b0 0x557b56b1b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 --2- 172.21.15.31:0/4258805547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afbdf0 0x557b56b23d70 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 -- 172.21.15.31:0/4258805547 >> 172.21.15.31:0/4258805547 conn(0x557b56af0fd0 msgr2=0x557b56b1dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.832+0000 7f6091502b80 1 -- 172.21.15.31:0/4258805547 shutdown_connections 2023-07-29T16:31:08.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.833+0000 7f6091502b80 1 -- 172.21.15.31:0/4258805547 wait complete. 2023-07-29T16:31:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.833+0000 7f6091502b80 1 -- start start 2023-07-29T16:31:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.833+0000 7f6091502b80 1 Processor -- start 2023-07-29T16:31:08.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b191c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b19700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56b1ac50 con 0x557b56afbdf0 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56b1adc0 con 0x557b56afccf0 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b56b1af80 con 0x557b569576b0 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.834+0000 7f6091502b80 10 client.0 _ll_register_callbacks cb 0x557b569f25c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:08.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.835+0000 7f60777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.835+0000 7f60777fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b191c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58858/0 (socket says 172.21.15.31:58858) 2023-07-29T16:31:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.835+0000 7f60777fe700 1 -- 172.21.15.31:0/896025893 learned_addr learned my addr 172.21.15.31:0/896025893 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.835+0000 7f6077fff700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.836+0000 7f6076ffd700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b19700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.836+0000 7f60777fe700 1 -- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 msgr2=0x557b56b1a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.836+0000 7f60777fe700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.836+0000 7f60777fe700 1 -- 172.21.15.31:0/896025893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 msgr2=0x557b56b19700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.836+0000 7f60777fe700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b19700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.836+0000 7f60777fe700 1 -- 172.21.15.31:0/896025893 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60781ad820 con 0x557b569576b0 2023-07-29T16:31:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f60777fe700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b191c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6068000c00 tx=0x7f6068028cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7b0ce5692d95de server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6056ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6056ffd700 1 -- 172.21.15.31:0/896025893 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068014850 con 0x557b569576b0 2023-07-29T16:31:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6056ffd700 1 -- 172.21.15.31:0/896025893 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f6068000e40 con 0x557b569576b0 2023-07-29T16:31:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6070003e20 con 0x557b569576b0 2023-07-29T16:31:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6056ffd700 1 -- 172.21.15.31:0/896025893 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6068027490 con 0x557b569576b0 2023-07-29T16:31:08.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6091502b80 10 client.30064 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.837+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x557b56879250 con 0x557b569576b0 2023-07-29T16:31:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6056ffd700 1 -- 172.21.15.31:0/896025893 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6070003e20 con 0x557b569576b0 2023-07-29T16:31:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6056ffd700 1 -- 172.21.15.31:0/896025893 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f606801b050 con 0x557b569576b0 2023-07-29T16:31:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6091502b80 10 client.30064 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6091502b80 10 client.30064 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:08.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6091502b80 20 client.30064 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6091502b80 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b748c0 0x557b56b76cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6091502b80 4 client.30064 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:31:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557b56a45ce0 con 0x557b56b748c0 2023-07-29T16:31:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.838+0000 7f6076ffd700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b748c0 0x557b56b76cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.839+0000 7f6056ffd700 10 client.30064 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:08.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:08.839+0000 7f6076ffd700 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b748c0 0x557b56b76cd0 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42497952c28cc39 server_cookie=6bb8ff4ece20aeee in_seq=0 out_seq=0 2023-07-29T16:31:09.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 30587 34.40 MB/sec warmup 53 sec latency 347.871 ms 2023-07-29T16:31:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6056ffd700 1 -- 172.21.15.31:0/896025893 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557b56a45ce0 con 0x557b56b748c0 2023-07-29T16:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6056ffd700 10 client.30064 handle_command_reply: tid=0 2023-07-29T16:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 client.30064 shutdown 2023-07-29T16:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 msgr2=0x557b56b191c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b191c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6068000c00 tx=0x7f6068028cd0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 shutdown_connections 2023-07-29T16:31:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557b56b748c0 0x557b56b76cd0 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557b56afccf0 0x557b56b1a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557b569576b0 0x557b56b191c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 --2- 172.21.15.31:0/896025893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557b56afbdf0 0x557b56b19700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6056ffd700 0 client.30064 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 >> 172.21.15.31:0/896025893 conn(0x557b56af0fd0 msgr2=0x557b56b23d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 shutdown_connections 2023-07-29T16:31:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 1 -- 172.21.15.31:0/896025893 wait complete. 2023-07-29T16:31:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.309+0000 7f6091502b80 20 client.30064 trim_cache size 0 max 16384 2023-07-29T16:31:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:31:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:31:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:31:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:31:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60640a0770 msgr2=0x7f60640a2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60640a0770 0x7f60640a2c20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6070023000 tx=0x7f607001f000 comp rx=0 tx=0).stop 2023-07-29T16:31:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078107f60 msgr2=0x7f60781a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078107f60 0x7f60781a5570 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f606000ba80 tx=0x7f6060014750 comp rx=0 tx=0).stop 2023-07-29T16:31:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 shutdown_connections 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f60640a0770 0x7f60640a2c20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60781088f0 0x7f60781a63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6078071fa0 0x7f60781a5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 --2- 172.21.15.31:0/2957160016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6078107f60 0x7f60781a5570 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 >> 172.21.15.31:0/2957160016 conn(0x7f607806d5c0 msgr2=0x7f607806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 shutdown_connections 2023-07-29T16:31:09.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:09.310+0000 7f607dbff700 1 -- 172.21.15.31:0/2957160016 wait complete. 2023-07-29T16:31:10.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 31280 34.40 MB/sec warmup 54 sec latency 386.183 ms 2023-07-29T16:31:10.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:10 smithi031 ceph-mon[95504]: pgmap v842: 97 pgs: 97 active+clean; 486 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.8 MiB/s wr, 857 op/s 2023-07-29T16:31:11.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:10 smithi039 ceph-mon[128858]: pgmap v842: 97 pgs: 97 active+clean; 486 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.8 MiB/s wr, 857 op/s 2023-07-29T16:31:11.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:10 smithi156 ceph-mon[129245]: pgmap v842: 97 pgs: 97 active+clean; 486 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.8 MiB/s wr, 857 op/s 2023-07-29T16:31:11.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32062 34.65 MB/sec warmup 55 sec latency 195.827 ms 2023-07-29T16:31:12.222 INFO:tasks.workunit.client.0.smithi031.stdout: 10 32970 34.73 MB/sec warmup 56 sec latency 41.388 ms 2023-07-29T16:31:12.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:12 smithi031 ceph-mon[95504]: pgmap v843: 97 pgs: 97 active+clean; 486 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 5.7 MiB/s wr, 549 op/s 2023-07-29T16:31:13.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:12 smithi039 ceph-mon[128858]: pgmap v843: 97 pgs: 97 active+clean; 486 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 5.7 MiB/s wr, 549 op/s 2023-07-29T16:31:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:12 smithi156 ceph-mon[129245]: pgmap v843: 97 pgs: 97 active+clean; 486 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 5.7 MiB/s wr, 549 op/s 2023-07-29T16:31:13.223 INFO:tasks.workunit.client.0.smithi031.stdout: 10 33978 35.10 MB/sec warmup 57 sec latency 44.849 ms 2023-07-29T16:31:14.223 INFO:tasks.workunit.client.0.smithi031.stdout: 10 34888 35.46 MB/sec warmup 58 sec latency 56.262 ms 2023-07-29T16:31:14.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:14 smithi031 ceph-mon[95504]: pgmap v844: 97 pgs: 97 active+clean; 489 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.5 MiB/s wr, 815 op/s 2023-07-29T16:31:15.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:14 smithi039 ceph-mon[128858]: pgmap v844: 97 pgs: 97 active+clean; 489 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.5 MiB/s wr, 815 op/s 2023-07-29T16:31:15.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:14 smithi156 ceph-mon[129245]: pgmap v844: 97 pgs: 97 active+clean; 489 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.5 MiB/s wr, 815 op/s 2023-07-29T16:31:15.223 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35490 35.41 MB/sec warmup 59 sec latency 96.993 ms 2023-07-29T16:31:15.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-07-29T16:31:16.223 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35515 34.83 MB/sec warmup 60 sec latency 819.488 ms 2023-07-29T16:31:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.338+0000 7fecbc7ff700 1 -- 172.21.15.31:0/4223641565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb41037c0 msgr2=0x7fecb4103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.339 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.338+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/4223641565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb41037c0 0x7fecb4103c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7feca40098f0 tx=0x7feca4009c00 comp rx=0 tx=0).stop 2023-07-29T16:31:16.340 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 -- 172.21.15.31:0/4223641565 shutdown_connections 2023-07-29T16:31:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/4223641565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecb4104150 0x7fecb410aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/4223641565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41025d0 0x7fecb41029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/4223641565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb41037c0 0x7fecb4103c10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.341 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 -- 172.21.15.31:0/4223641565 >> 172.21.15.31:0/4223641565 conn(0x7fecb40fdbd0 msgr2=0x7fecb4100020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 -- 172.21.15.31:0/4223641565 shutdown_connections 2023-07-29T16:31:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.340+0000 7fecbc7ff700 1 -- 172.21.15.31:0/4223641565 wait complete. 2023-07-29T16:31:16.342 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 Processor -- start 2023-07-29T16:31:16.343 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 -- start start 2023-07-29T16:31:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecb41025d0 0x7fecb4193e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41037c0 0x7fecb41943d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb4104150 0x7fecb419b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb419ba10 con 0x7fecb41037c0 2023-07-29T16:31:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb406a770 con 0x7fecb41025d0 2023-07-29T16:31:16.344 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.342+0000 7fecbc7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb406a900 con 0x7fecb4104150 2023-07-29T16:31:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.343+0000 7fecbad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb4104150 0x7fecb419b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.343+0000 7fecbad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb4104150 0x7fecb419b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34360/0 (socket says 172.21.15.31:34360) 2023-07-29T16:31:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.343+0000 7fecbad9c700 1 -- 172.21.15.31:0/1006040728 learned_addr learned my addr 172.21.15.31:0/1006040728 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:16.345 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.343+0000 7fecba59b700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecb41025d0 0x7fecb4193e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.343+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41037c0 0x7fecb41943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.344+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1006040728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb4104150 msgr2=0x7fecb419b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.344+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb4104150 0x7fecb419b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.344+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1006040728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecb41025d0 msgr2=0x7fecb4193e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.344+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecb41025d0 0x7fecb4193e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.344+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1006040728 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca4009520 con 0x7fecb41037c0 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.345+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41037c0 0x7fecb41943d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7feca4009230 tx=0x7feca400b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.345+0000 7fecab7fe700 1 -- 172.21.15.31:0/1006040728 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca401c070 con 0x7fecb41037c0 2023-07-29T16:31:16.347 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.345+0000 7fecab7fe700 1 -- 172.21.15.31:0/1006040728 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feca400f7d0 con 0x7fecb41037c0 2023-07-29T16:31:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.345+0000 7fecab7fe700 1 -- 172.21.15.31:0/1006040728 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca4003a00 con 0x7fecb41037c0 2023-07-29T16:31:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.347+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb406ab80 con 0x7fecb41037c0 2023-07-29T16:31:16.348 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.347+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb406b010 con 0x7fecb41037c0 2023-07-29T16:31:16.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.353+0000 7fecd0049b80 1 Processor -- start 2023-07-29T16:31:16.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 -- start start 2023-07-29T16:31:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6ed9fbe0 0x559c6ed9fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda0530 0x559c6eda0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda12f0 0x559c6eda1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ecf7a10 con 0x559c6eda0530 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ecf53d0 con 0x559c6eda12f0 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.356+0000 7fecd0049b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ecf0cc0 con 0x559c6ed9fbe0 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecba59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda0530 0x559c6eda0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecbad9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6ed9fbe0 0x559c6ed9fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecba59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda0530 0x559c6eda0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47268/0 (socket says 172.21.15.31:47268) 2023-07-29T16:31:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecba59b700 1 -- 172.21.15.31:0/1189816433 learned_addr learned my addr 172.21.15.31:0/1189816433 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecbad9c700 1 -- 172.21.15.31:0/1189816433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda12f0 msgr2=0x559c6eda1740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:31:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecbad9c700 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda12f0 0x559c6eda1740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecbad9c700 1 -- 172.21.15.31:0/1189816433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda0530 msgr2=0x559c6eda0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecbad9c700 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda0530 0x559c6eda0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.357+0000 7fecbad9c700 1 -- 172.21.15.31:0/1189816433 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6eb1ede0 con 0x559c6ed9fbe0 2023-07-29T16:31:16.360 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.358+0000 7fecbad9c700 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6ed9fbe0 0x559c6ed9fff0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fecac009c10 tx=0x7fecac009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2f321a784842bd9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.359+0000 7fecab7fe700 1 -- 172.21.15.31:0/1006040728 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7feca4020030 con 0x7fecb41037c0 2023-07-29T16:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.359+0000 7feca8ff9700 1 -- 172.21.15.31:0/1189816433 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac00e040 con 0x559c6ed9fbe0 2023-07-29T16:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.359+0000 7feca8ff9700 1 -- 172.21.15.31:0/1189816433 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecac00f040 con 0x559c6ed9fbe0 2023-07-29T16:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.359+0000 7feca8ff9700 1 -- 172.21.15.31:0/1189816433 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac0047a0 con 0x559c6ed9fbe0 2023-07-29T16:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.360+0000 7fecd0049b80 1 -- 172.21.15.31:0/1189816433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6ed9fbe0 msgr2=0x559c6ed9fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.360+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6ed9fbe0 0x559c6ed9fff0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fecac009c10 tx=0x7fecac009920 comp rx=0 tx=0).stop 2023-07-29T16:31:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecab7fe700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feca00a0810 0x7feca00a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecab7fe700 1 -- 172.21.15.31:0/1006040728 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feca40cc840 con 0x7fecb41037c0 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecbad9c700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feca00a0810 0x7feca00a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecd0049b80 1 -- 172.21.15.31:0/1189816433 shutdown_connections 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda12f0 0x559c6eda1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6ed9fbe0 0x559c6ed9fff0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1189816433 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda0530 0x559c6eda0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.361+0000 7fecd0049b80 1 -- 172.21.15.31:0/1189816433 >> 172.21.15.31:0/1189816433 conn(0x559c6ed953e0 msgr2=0x559c6ed9a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.363 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.362+0000 7fecd0049b80 1 -- 172.21.15.31:0/1189816433 shutdown_connections 2023-07-29T16:31:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.362+0000 7fecbad9c700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feca00a0810 0x7feca00a2cc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fecac004a20 tx=0x7fecac00e1a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.362+0000 7fecd0049b80 1 -- 172.21.15.31:0/1189816433 wait complete. 2023-07-29T16:31:16.364 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.362+0000 7fecd0049b80 1 -- start start 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 Processor -- start 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ec8d510 con 0x559c6ed9fbe0 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ec8d1b0 con 0x559c6eda0530 2023-07-29T16:31:16.365 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6eb74920 con 0x559c6eda12f0 2023-07-29T16:31:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.363+0000 7fecd0049b80 10 client.0 _ll_register_callbacks cb 0x559c6ec96620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41176/0 (socket says 172.21.15.31:41176) 2023-07-29T16:31:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1355997742 learned_addr learned my addr 172.21.15.31:0/1355997742 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:16.366 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecbad9c700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecba59b700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1355997742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 msgr2=0x559c6edb0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1355997742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 msgr2=0x559c6eda9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.368 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 -- 172.21.15.31:0/1355997742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac00e330 con 0x559c6eda0530 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecba59b700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e40 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7feca40a1e60 tx=0x7feca4099be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd917f90407b3f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.364+0000 7fec9dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fec9dffb700 1 -- 172.21.15.31:0/1355997742 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca40a2390 con 0x559c6eda0530 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fec9dffb700 1 -- 172.21.15.31:0/1355997742 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feca40966e0 con 0x559c6eda0530 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c6ed1bd10 con 0x559c6eda0530 2023-07-29T16:31:16.369 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fec9dffb700 1 -- 172.21.15.31:0/1355997742 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7feca40ca050 con 0x559c6eda0530 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fecd0049b80 10 client.30071 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x559c6eb07c60 con 0x559c6eda0530 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.365+0000 7fec9dffb700 1 -- 172.21.15.31:0/1355997742 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559c6ed1bd10 con 0x559c6eda0530 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fec9dffb700 1 -- 172.21.15.31:0/1355997742 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feca4094be0 con 0x559c6eda0530 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecd0049b80 10 client.30071 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecd0049b80 10 client.30071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:16.370 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecd0049b80 20 client.30071 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6edb5270 0x559c6edb7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecd0049b80 4 client.30071 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c6ed6ed70 con 0x559c6edb5270 2023-07-29T16:31:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.366+0000 7fecba59b700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6edb5270 0x559c6edb7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.367+0000 7fecba59b700 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6edb5270 0x559c6edb7680 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=776a45a0e65b4646 server_cookie=8fa72b1611611541 in_seq=0 out_seq=0 2023-07-29T16:31:16.371 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.367+0000 7fec9dffb700 10 client.30071 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fec9dffb700 1 -- 172.21.15.31:0/1355997742 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559c6ed6ed70 con 0x559c6edb5270 2023-07-29T16:31:16.372 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fec9dffb700 10 client.30071 handle_command_reply: tid=0 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 client.30071 shutdown 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 msgr2=0x559c6eda9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e40 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7feca40a1e60 tx=0x7feca4099be0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 shutdown_connections 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6edb5270 0x559c6edb7680 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.373 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fec9dffb700 0 client.30071 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 --2- 172.21.15.31:0/1355997742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e40 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 >> 172.21.15.31:0/1355997742 conn(0x559c6ed953e0 msgr2=0x559c6ed9a060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.368+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 shutdown_connections 2023-07-29T16:31:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.369+0000 7fecd0049b80 1 -- 172.21.15.31:0/1355997742 wait complete. 2023-07-29T16:31:16.374 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.370+0000 7fecd0049b80 20 client.30071 trim_cache size 0 max 16384 2023-07-29T16:31:16.400 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 Processor -- start 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 -- start start 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ed9b8e0 con 0x559c6ed9fbe0 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ed1bd10 con 0x559c6eda0530 2023-07-29T16:31:16.401 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.399+0000 7fecd0049b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6edaa800 con 0x559c6eda12f0 2023-07-29T16:31:16.402 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.400+0000 7fecba59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.400+0000 7fecba59b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47292/0 (socket says 172.21.15.31:47292) 2023-07-29T16:31:16.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.400+0000 7fecba59b700 1 -- 172.21.15.31:0/2113214581 learned_addr learned my addr 172.21.15.31:0/2113214581 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:16.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7fecba59b700 1 -- 172.21.15.31:0/2113214581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 msgr2=0x559c6edb0e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:31:16.403 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7fecba59b700 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7fecba59b700 1 -- 172.21.15.31:0/2113214581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 msgr2=0x559c6eda9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:16.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7fecba59b700 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7fecba59b700 1 -- 172.21.15.31:0/2113214581 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6eb45250 con 0x559c6ed9fbe0 2023-07-29T16:31:16.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7fecba59b700 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda98e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x559c6ed95360 tx=0x7fecb001eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1366bf032fa5a0d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.404 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7feca97fa700 1 -- 172.21.15.31:0/2113214581 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb001e330 con 0x559c6ed9fbe0 2023-07-29T16:31:16.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7feca97fa700 1 -- 172.21.15.31:0/2113214581 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecb0022070 con 0x559c6ed9fbe0 2023-07-29T16:31:16.405 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.401+0000 7feca97fa700 1 -- 172.21.15.31:0/2113214581 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb0010430 con 0x559c6ed9fbe0 2023-07-29T16:31:16.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.402+0000 7fecd0049b80 1 -- 172.21.15.31:0/2113214581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 msgr2=0x559c6eda98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.406 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.402+0000 7fecd0049b80 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda98e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x559c6ed95360 tx=0x7fecb001eba0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 -- 172.21.15.31:0/2113214581 shutdown_connections 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda12f0 0x559c6edb0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6ed9fbe0 0x559c6eda98e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 --2- 172.21.15.31:0/2113214581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6eda0530 0x559c6eda9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 -- 172.21.15.31:0/2113214581 >> 172.21.15.31:0/2113214581 conn(0x559c6ed953e0 msgr2=0x559c6eda7700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 -- 172.21.15.31:0/2113214581 shutdown_connections 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 -- 172.21.15.31:0/2113214581 wait complete. 2023-07-29T16:31:16.407 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.403+0000 7fecd0049b80 1 -- start start 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.404+0000 7fecd0049b80 1 Processor -- start 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.404+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6ed9fbe0 0x559c6edadbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.404+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 0x559c6edae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.404+0000 7fecd0049b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda12f0 0x559c6ee06db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecd0049b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6edb1120 con 0x559c6eda12f0 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecd0049b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6edb1290 con 0x559c6ed9fbe0 2023-07-29T16:31:16.408 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecd0049b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6ee072f0 con 0x559c6eda0530 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecd0049b80 10 client.0 _ll_register_callbacks cb 0x559c6ec96620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecb9d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 0x559c6edae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecb9d9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 0x559c6edae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34392/0 (socket says 172.21.15.31:34392) 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecb9d9a700 1 -- 172.21.15.31:0/3259977021 learned_addr learned my addr 172.21.15.31:0/3259977021 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.405+0000 7fecbad9c700 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda12f0 0x559c6ee06db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fecb9d9a700 1 -- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6ed9fbe0 msgr2=0x559c6edadbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:16.409 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6ed9fbe0 0x559c6edadbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fecb9d9a700 1 -- 172.21.15.31:0/3259977021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda12f0 msgr2=0x559c6ee06db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:16.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda12f0 0x559c6ee06db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fecb9d9a700 1 -- 172.21.15.31:0/3259977021 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb0009d50 con 0x559c6eda0530 2023-07-29T16:31:16.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fecb9d9a700 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 0x559c6edae100 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7feca40094b0 tx=0x7feca40b8480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a0c4672af9a4726 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.410 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.406+0000 7fec9dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.409+0000 7fec9dffb700 1 -- 172.21.15.31:0/3259977021 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca40ca190 con 0x559c6eda0530 2023-07-29T16:31:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.409+0000 7fec9dffb700 1 -- 172.21.15.31:0/3259977021 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7feca409a470 con 0x559c6eda0530 2023-07-29T16:31:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.409+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecac00f040 con 0x559c6eda0530 2023-07-29T16:31:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.409+0000 7fec9dffb700 1 -- 172.21.15.31:0/3259977021 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7feca409a280 con 0x559c6eda0530 2023-07-29T16:31:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.409+0000 7fecd0049b80 10 client.30079 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:16.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.409+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x559c6eb1d2b0 con 0x559c6eda0530 2023-07-29T16:31:16.412 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fec9dffb700 1 -- 172.21.15.31:0/3259977021 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecac00f040 con 0x559c6eda0530 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fec9dffb700 1 -- 172.21.15.31:0/3259977021 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7feca40994c0 con 0x559c6eda0530 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecd0049b80 10 client.30079 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecd0049b80 10 client.30079 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecd0049b80 20 client.30079 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecd0049b80 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6ee087a0 0x559c6ee0ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecd0049b80 4 client.30079 mds_command: new command op to 24581 tid=0[{"prefix": "scrub status"}] 2023-07-29T16:31:16.413 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559c6ece9d40 con 0x559c6ee087a0 2023-07-29T16:31:16.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.411+0000 7fecba59b700 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6ee087a0 0x559c6ee0ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.412+0000 7fecba59b700 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6ee087a0 0x559c6ee0ab80 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5cb3057867b8775 server_cookie=616eadea4cfeafd3 in_seq=0 out_seq=0 2023-07-29T16:31:16.414 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.412+0000 7fec9dffb700 10 client.30079 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:16.415 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.412+0000 7fec9dffb700 1 -- 172.21.15.31:0/3259977021 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559c6ece9d40 con 0x559c6ee087a0 2023-07-29T16:31:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.412+0000 7fec9dffb700 10 client.30079 handle_command_reply: tid=0 2023-07-29T16:31:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.412+0000 7fecd0049b80 1 client.30079 shutdown 2023-07-29T16:31:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 msgr2=0x559c6edae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 0x559c6edae100 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7feca40094b0 tx=0x7feca40b8480 comp rx=0 tx=0).stop 2023-07-29T16:31:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 shutdown_connections 2023-07-29T16:31:16.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6ee087a0 0x559c6ee0ab80 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6eda12f0 0x559c6ee06db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fec9dffb700 0 client.30079 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6ed9fbe0 0x559c6edadbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 --2- 172.21.15.31:0/3259977021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6eda0530 0x559c6edae100 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 >> 172.21.15.31:0/3259977021 conn(0x559c6ed953e0 msgr2=0x559c6eda4230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.417 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 shutdown_connections 2023-07-29T16:31:16.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.413+0000 7fecd0049b80 1 -- 172.21.15.31:0/3259977021 wait complete. 2023-07-29T16:31:16.418 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.414+0000 7fecd0049b80 20 client.30079 trim_cache size 0 max 16384 2023-07-29T16:31:16.418 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:31:16.419 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2023-07-29T16:31:16.420 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2023-07-29T16:31:16.420 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:31:16.420 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feca00a0810 msgr2=0x7feca00a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feca00a0810 0x7feca00a2cc0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fecac004a20 tx=0x7fecac00e1a0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41037c0 msgr2=0x7fecb41943d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41037c0 0x7fecb41943d0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7feca4009230 tx=0x7feca400b790 comp rx=0 tx=0).stop 2023-07-29T16:31:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 shutdown_connections 2023-07-29T16:31:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7feca00a0810 0x7feca00a2cc0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.421 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fecb4104150 0x7fecb419b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fecb41025d0 0x7fecb4193e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 --2- 172.21.15.31:0/1006040728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fecb41037c0 0x7fecb41943d0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 >> 172.21.15.31:0/1006040728 conn(0x7fecb40fdbd0 msgr2=0x7fecb40771a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 shutdown_connections 2023-07-29T16:31:16.422 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.415+0000 7fecbc7ff700 1 -- 172.21.15.31:0/1006040728 wait complete. 2023-07-29T16:31:16.447 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-07-29T16:31:16.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:31:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/2192821686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 msgr2=0x7f5bc8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/2192821686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc8105a90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009990 tx=0x7f5bb8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/2192821686 shutdown_connections 2023-07-29T16:31:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/2192821686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bc8105fd0 0x7f5bc8108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/2192821686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8100d90 0x7f5bc8103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/2192821686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc8105a90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.766+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/2192821686 >> 172.21.15.31:0/2192821686 conn(0x7f5bc80fa810 msgr2=0x7f5bc80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.767+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/2192821686 shutdown_connections 2023-07-29T16:31:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.767+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/2192821686 wait complete. 2023-07-29T16:31:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.768+0000 7f5bcf9ff700 1 Processor -- start 2023-07-29T16:31:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.768+0000 7f5bcf9ff700 1 -- start start 2023-07-29T16:31:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcf9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bc8100d90 0x7f5bc819c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcf9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc819cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcf9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8105fd0 0x7f5bc81a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcf9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc81a41b0 con 0x7f5bc8100d90 2023-07-29T16:31:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcf9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc81a42f0 con 0x7f5bc8105fd0 2023-07-29T16:31:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcf9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc81a4480 con 0x7f5bc81036b0 2023-07-29T16:31:16.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bccf9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc819cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bcdf9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8105fd0 0x7f5bc81a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bccf9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc819cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34412/0 (socket says 172.21.15.31:34412) 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.769+0000 7f5bccf9a700 1 -- 172.21.15.31:0/643106975 learned_addr learned my addr 172.21.15.31:0/643106975 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bcdf9c700 1 -- 172.21.15.31:0/643106975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 msgr2=0x7f5bc819cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bcdf9c700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc819cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bcdf9c700 1 -- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bc8100d90 msgr2=0x7f5bc819c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:16.774 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bcdf9c700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bc8100d90 0x7f5bc819c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bcdf9c700 1 -- 172.21.15.31:0/643106975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80095f0 con 0x7f5bc8105fd0 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bcdf9c700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8105fd0 0x7f5bc81a3be0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0007f10 tx=0x7f5bc000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.770+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc000ddd0 con 0x7f5bc8105fd0 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.771+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bc000f040 con 0x7f5bc8105fd0 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.771+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0015400 con 0x7f5bc8105fd0 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.771+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc81a4760 con 0x7f5bc8105fd0 2023-07-29T16:31:16.775 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.771+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc81a4c20 con 0x7f5bc8105fd0 2023-07-29T16:31:16.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.772+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc8109980 con 0x7f5bc8105fd0 2023-07-29T16:31:16.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.775+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5bc0004c40 con 0x7f5bc8105fd0 2023-07-29T16:31:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.777+0000 7f5bbe7fc700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5bb40a0840 0x7f5bb40a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:16.779 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.777+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5bc00cb580 con 0x7f5bc8105fd0 2023-07-29T16:31:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.778+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5bc0094f60 con 0x7f5bc8105fd0 2023-07-29T16:31:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.780+0000 7f5bcd79b700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5bb40a0840 0x7f5bb40a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:16.781+0000 7f5bcd79b700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5bb40a0840 0x7f5bb40a2cf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4005950 tx=0x7f5bc40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:16.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:16 smithi031 ceph-mon[95504]: pgmap v845: 97 pgs: 97 active+clean; 496 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.9 MiB/s wr, 793 op/s 2023-07-29T16:31:17.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:16 smithi039 ceph-mon[128858]: pgmap v845: 97 pgs: 97 active+clean; 496 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.9 MiB/s wr, 793 op/s 2023-07-29T16:31:17.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:16 smithi156 ceph-mon[129245]: pgmap v845: 97 pgs: 97 active+clean; 496 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 7.9 MiB/s wr, 793 op/s 2023-07-29T16:31:17.227 INFO:tasks.workunit.client.0.smithi031.stdout: 10 35966 34.73 MB/sec warmup 61 sec latency 442.959 ms 2023-07-29T16:31:17.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:31:17.237 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":39,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:30:50.645149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":282,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_28348":{"gid":28348,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6837/3963887865","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":3963887865},{"type":"v1","addr":"172.21.15.156:6837","nonce":3963887865}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:31:17.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.233+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bc8109b70 con 0x7f5bc8105fd0 2023-07-29T16:31:17.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.235+0000 7f5bbe7fc700 1 -- 172.21.15.31:0/643106975 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+10787 (secure 0 0 0) 0x7f5bc0098580 con 0x7f5bc8105fd0 2023-07-29T16:31:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.241+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5bb40a0840 msgr2=0x7f5bb40a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.241+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5bb40a0840 0x7f5bb40a2cf0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4005950 tx=0x7f5bc40058e0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.241+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8105fd0 msgr2=0x7f5bc81a3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.243 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.241+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8105fd0 0x7f5bc81a3be0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0007f10 tx=0x7f5bc000d3b0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 shutdown_connections 2023-07-29T16:31:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5bb40a0840 0x7f5bb40a2cf0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5bc8105fd0 0x7f5bc81a3be0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5bc8100d90 0x7f5bc819c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 --2- 172.21.15.31:0/643106975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bc81036b0 0x7f5bc819cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 >> 172.21.15.31:0/643106975 conn(0x7f5bc80fa810 msgr2=0x7f5bc81021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 shutdown_connections 2023-07-29T16:31:17.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.248+0000 7f5bcf9ff700 1 -- 172.21.15.31:0/643106975 wait complete. 2023-07-29T16:31:17.259 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 39 2023-07-29T16:31:17.284 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-07-29T16:31:17.696 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 -- 172.21.15.31:0/220243492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100410 msgr2=0x7fcf30100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 --2- 172.21.15.31:0/220243492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100410 0x7fcf30100860 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcf24009990 tx=0x7fcf24009ca0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 -- 172.21.15.31:0/220243492 shutdown_connections 2023-07-29T16:31:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 --2- 172.21.15.31:0/220243492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf30100da0 0x7fcf30107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 --2- 172.21.15.31:0/220243492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf300ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.697 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 --2- 172.21.15.31:0/220243492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100410 0x7fcf30100860 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 -- 172.21.15.31:0/220243492 >> 172.21.15.31:0/220243492 conn(0x7fcf300fa840 msgr2=0x7fcf300fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.694+0000 7fcf365ff700 1 -- 172.21.15.31:0/220243492 shutdown_connections 2023-07-29T16:31:17.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.695+0000 7fcf365ff700 1 -- 172.21.15.31:0/220243492 wait complete. 2023-07-29T16:31:17.698 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.696+0000 7fcf365ff700 1 Processor -- start 2023-07-29T16:31:17.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 -- start start 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf3010f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf30100410 0x7fcf3010f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100da0 0x7fcf301168c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30116e90 con 0x7fcf300ff220 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30116fd0 con 0x7fcf30100410 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.697+0000 7fcf365ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf30117160 con 0x7fcf30100da0 2023-07-29T16:31:17.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.698+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf3010f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.698+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf3010f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47332/0 (socket says 172.21.15.31:47332) 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.698+0000 7fcf2ffff700 1 -- 172.21.15.31:0/4200462163 learned_addr learned my addr 172.21.15.31:0/4200462163 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf30100410 0x7fcf3010f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100da0 0x7fcf301168c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf2ffff700 1 -- 172.21.15.31:0/4200462163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100da0 msgr2=0x7fcf301168c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100da0 0x7fcf301168c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf2ffff700 1 -- 172.21.15.31:0/4200462163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf30100410 msgr2=0x7fcf3010f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf30100410 0x7fcf3010f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.699+0000 7fcf2ffff700 1 -- 172.21.15.31:0/4200462163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240095f0 con 0x7fcf300ff220 2023-07-29T16:31:17.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.702+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf3010f310 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000b500 tx=0x7fcf2000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.703+0000 7fcf2d7fa700 1 -- 172.21.15.31:0/4200462163 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20011830 con 0x7fcf300ff220 2023-07-29T16:31:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.703+0000 7fcf2d7fa700 1 -- 172.21.15.31:0/4200462163 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf20011e50 con 0x7fcf300ff220 2023-07-29T16:31:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.703+0000 7fcf2d7fa700 1 -- 172.21.15.31:0/4200462163 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20010dc0 con 0x7fcf300ff220 2023-07-29T16:31:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.704+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf30117440 con 0x7fcf300ff220 2023-07-29T16:31:17.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.704+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3006a770 con 0x7fcf300ff220 2023-07-29T16:31:17.711 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.711+0000 7fcf49f5db80 1 Processor -- start 2023-07-29T16:31:17.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.711+0000 7fcf49f5db80 1 -- start start 2023-07-29T16:31:17.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c802ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c803d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c804740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c75aa10 con 0x55be3c8042f0 2023-07-29T16:31:17.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c7583d0 con 0x55be3c803530 2023-07-29T16:31:17.713 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c753cc0 con 0x55be3c802be0 2023-07-29T16:31:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf34b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c802ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf34b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c802ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34426/0 (socket says 172.21.15.31:34426) 2023-07-29T16:31:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.712+0000 7fcf34b9c700 1 -- 172.21.15.31:0/3013018078 learned_addr learned my addr 172.21.15.31:0/3013018078 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.713+0000 7fcf34b9c700 1 -- 172.21.15.31:0/3013018078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 msgr2=0x55be3c803d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:31:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.713+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c803d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.714 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.713+0000 7fcf34b9c700 1 -- 172.21.15.31:0/3013018078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 msgr2=0x55be3c804740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.713+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c804740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.713+0000 7fcf34b9c700 1 -- 172.21.15.31:0/3013018078 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be3c581de0 con 0x55be3c802be0 2023-07-29T16:31:17.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.714+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c802ff0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800a4d0 tx=0x7fcf28009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2288576ac0256b01 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:17.716 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.715+0000 7fcf2d7fa700 1 -- 172.21.15.31:0/4200462163 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fcf20010380 con 0x7fcf300ff220 2023-07-29T16:31:17.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.716+0000 7fcf167fc700 1 -- 172.21.15.31:0/3013018078 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28010040 con 0x55be3c802be0 2023-07-29T16:31:17.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.716+0000 7fcf167fc700 1 -- 172.21.15.31:0/3013018078 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf28014ae0 con 0x55be3c802be0 2023-07-29T16:31:17.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.716+0000 7fcf167fc700 1 -- 172.21.15.31:0/3013018078 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280139f0 con 0x55be3c802be0 2023-07-29T16:31:17.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 -- 172.21.15.31:0/3013018078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 msgr2=0x55be3c802ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c802ff0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800a4d0 tx=0x7fcf28009650 comp rx=0 tx=0).stop 2023-07-29T16:31:17.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 -- 172.21.15.31:0/3013018078 shutdown_connections 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c804740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c802ff0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/3013018078 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c803d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.717+0000 7fcf49f5db80 1 -- 172.21.15.31:0/3013018078 >> 172.21.15.31:0/3013018078 conn(0x55be3c7f83e0 msgr2=0x55be3c7fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.718+0000 7fcf2d7fa700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcf180a0910 0x7fcf180a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.718+0000 7fcf2d7fa700 1 -- 172.21.15.31:0/4200462163 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcf200cb430 con 0x7fcf300ff220 2023-07-29T16:31:17.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.718+0000 7fcf49f5db80 1 -- 172.21.15.31:0/3013018078 shutdown_connections 2023-07-29T16:31:17.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.718+0000 7fcf49f5db80 1 -- 172.21.15.31:0/3013018078 wait complete. 2023-07-29T16:31:17.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.719+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcf180a0910 0x7fcf180a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.719+0000 7fcf49f5db80 1 -- start start 2023-07-29T16:31:17.720 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.719+0000 7fcf49f5db80 1 Processor -- start 2023-07-29T16:31:17.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.721+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcf180a0910 0x7fcf180a2dc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55be3c7f8360 tx=0x7fcf24009f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c80c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c80ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c813e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c6f0510 con 0x55be3c8042f0 2023-07-29T16:31:17.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c6f01b0 con 0x55be3c803530 2023-07-29T16:31:17.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c5d7920 con 0x55be3c802be0 2023-07-29T16:31:17.741 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.729+0000 7fcf49f5db80 10 client.0 _ll_register_callbacks cb 0x55be3c6f9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:17.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.730+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c80c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.730+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c80c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34432/0 (socket says 172.21.15.31:34432) 2023-07-29T16:31:17.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.730+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2780972456 learned_addr learned my addr 172.21.15.31:0/2780972456 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.741+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c813e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c80ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.751 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2780972456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 msgr2=0x55be3c80ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c80ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2780972456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 msgr2=0x55be3c813e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.752 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c813e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2780972456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf28009d00 con 0x55be3c802be0 2023-07-29T16:31:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c80ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c80c910 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf20098a60 tx=0x7fcf20098740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e3925425a9e51c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf0f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.742+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2780972456 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2009cb00 con 0x55be3c802be0 2023-07-29T16:31:17.753 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.743+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2780972456 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcf200a0f20 con 0x55be3c802be0 2023-07-29T16:31:17.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.749+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be3c77ed10 con 0x55be3c802be0 2023-07-29T16:31:17.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.749+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2780972456 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf20097fc0 con 0x55be3c802be0 2023-07-29T16:31:17.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.749+0000 7fcf49f5db80 10 client.30097 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:17.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.749+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55be3c56ac60 con 0x55be3c802be0 2023-07-29T16:31:17.754 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2780972456 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55be3c77ed10 con 0x55be3c802be0 2023-07-29T16:31:17.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2780972456 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcf20094a80 con 0x55be3c802be0 2023-07-29T16:31:17.758 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf49f5db80 10 client.30097 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf49f5db80 10 client.30097 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf49f5db80 20 client.30097 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c818660 0x55be3c81aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf49f5db80 4 client.30097 mds_command: new command op to 24550 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.750+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be3c7d1d70 con 0x55be3c818660 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.751+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c818660 0x55be3c81aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.759 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.752+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c818660 0x55be3c81aa70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6497f13e54602559 server_cookie=39b2cadcab7e795e in_seq=0 out_seq=0 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.752+0000 7fcf0f7fe700 10 client.30097 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.755+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2780972456 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55be3c7d1d70 con 0x55be3c818660 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.755+0000 7fcf0f7fe700 10 client.30097 handle_command_reply: tid=0 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.755+0000 7fcf49f5db80 1 client.30097 shutdown 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 msgr2=0x55be3c80c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c80c910 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf20098a60 tx=0x7fcf20098740 comp rx=0 tx=0).stop 2023-07-29T16:31:17.760 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 shutdown_connections 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c813e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c802be0 0x55be3c80c910 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c818660 0x55be3c81aa70 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2780972456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c803530 0x55be3c80ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 >> 172.21.15.31:0/2780972456 conn(0x55be3c7f83e0 msgr2=0x55be3c7fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.756+0000 7fcf0f7fe700 0 client.30097 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.757+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 shutdown_connections 2023-07-29T16:31:17.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.757+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2780972456 wait complete. 2023-07-29T16:31:17.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.757+0000 7fcf49f5db80 20 client.30097 trim_cache size 0 max 16384 2023-07-29T16:31:17.790 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.779+0000 7fcf49f5db80 1 Processor -- start 2023-07-29T16:31:17.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 -- start start 2023-07-29T16:31:17.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 0x55be3c81a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c813e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c807330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.791 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c803c20 con 0x55be3c8042f0 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c80db30 con 0x55be3c802be0 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.781+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c7ff3e0 con 0x55be3c803530 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c813e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 0x55be3c81a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 0x55be3c81a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41242/0 (socket says 172.21.15.31:41242) 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c813e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34448/0 (socket says 172.21.15.31:34448) 2023-07-29T16:31:17.792 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2882257238 learned_addr learned my addr 172.21.15.31:0/2882257238 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c807330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2882257238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 msgr2=0x55be3c813e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c813e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2882257238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 msgr2=0x55be3c807330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c807330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.782+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2882257238 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be3c5a8250 con 0x55be3c802be0 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.783+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 0x55be3c81a670 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcf20010700 tx=0x7fcf200984e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76de9cb8c68dbaa7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:17.793 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.783+0000 7fcf16ffd700 1 -- 172.21.15.31:0/2882257238 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf200a0f20 con 0x55be3c802be0 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.783+0000 7fcf16ffd700 1 -- 172.21.15.31:0/2882257238 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf20097bf0 con 0x55be3c802be0 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.783+0000 7fcf16ffd700 1 -- 172.21.15.31:0/2882257238 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20098220 con 0x55be3c802be0 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2882257238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 msgr2=0x55be3c81a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 0x55be3c81a670 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fcf20010700 tx=0x7fcf200984e0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2882257238 shutdown_connections 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c8042f0 0x55be3c807330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.794 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c802be0 0x55be3c81a670 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2882257238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c813e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2882257238 >> 172.21.15.31:0/2882257238 conn(0x55be3c7f83e0 msgr2=0x55be3c80b510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2882257238 shutdown_connections 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2882257238 wait complete. 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.784+0000 7fcf49f5db80 1 -- start start 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 Processor -- start 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 0x55be3c811b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.795 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c812090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c8042f0 0x55be3c8130a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c8135e0 con 0x55be3c802be0 2023-07-29T16:31:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c813750 con 0x55be3c8042f0 2023-07-29T16:31:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be3c813910 con 0x55be3c803530 2023-07-29T16:31:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.785+0000 7fcf49f5db80 10 client.0 _ll_register_callbacks cb 0x55be3c6f9620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:17.796 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 0x55be3c811b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 0x55be3c811b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47384/0 (socket says 172.21.15.31:47384) 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2ffff700 1 -- 172.21.15.31:0/2596265997 learned_addr learned my addr 172.21.15.31:0/2596265997 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf34b9c700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c8042f0 0x55be3c8130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c812090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2f7fe700 1 -- 172.21.15.31:0/2596265997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c8042f0 msgr2=0x55be3c8130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c8042f0 0x55be3c8130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2f7fe700 1 -- 172.21.15.31:0/2596265997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 msgr2=0x55be3c811b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:17.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 0x55be3c811b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2f7fe700 1 -- 172.21.15.31:0/2596265997 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55be3c56ac60 con 0x55be3c803530 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.786+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 0x55be3c811b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf2f7fe700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c812090 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcf24003790 tx=0x7fcf24005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c1476af36a68c19 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf0f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2596265997 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24003960 con 0x55be3c803530 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf28014ae0 con 0x55be3c803530 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2596265997 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fcf24003ac0 con 0x55be3c803530 2023-07-29T16:31:17.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2596265997 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf24003c80 con 0x55be3c803530 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf49f5db80 10 client.30103 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.787+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55be3c5802b0 con 0x55be3c803530 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.788+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2596265997 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf28014ae0 con 0x55be3c803530 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.788+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2596265997 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcf24005400 con 0x55be3c803530 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.788+0000 7fcf49f5db80 10 client.30103 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.788+0000 7fcf49f5db80 10 client.30103 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-07-29T16:31:17.799 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.788+0000 7fcf49f5db80 20 client.30103 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.788+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c86b0f0 0x55be3c86d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.789+0000 7fcf49f5db80 4 client.30103 mds_command: new command op to 24550 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:31:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.789+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 --> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55be3c74cd40 con 0x55be3c86b0f0 2023-07-29T16:31:17.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.789+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c86b0f0 0x55be3c86d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.791+0000 7fcf2ffff700 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c86b0f0 0x55be3c86d500 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37ce2d5fa2d1fa4b server_cookie=792601d977f099e2 in_seq=0 out_seq=0 2023-07-29T16:31:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.791+0000 7fcf0f7fe700 10 client.30103 ms_handle_connect on v2:172.21.15.156:6838/2888902152 2023-07-29T16:31:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf0f7fe700 1 -- 172.21.15.31:0/2596265997 <== mds.2 v2:172.21.15.156:6838/2888902152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+240 (crc 0 0 0) 0x55be3c74cd40 con 0x55be3c86b0f0 2023-07-29T16:31:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf0f7fe700 10 client.30103 handle_command_reply: tid=0 2023-07-29T16:31:17.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 client.30103 shutdown 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 msgr2=0x55be3c812090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c812090 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcf24003790 tx=0x7fcf24005fd0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 shutdown_connections 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:6838/2888902152,v1:172.21.15.156:6839/2888902152] conn(0x55be3c86b0f0 0x55be3c86d500 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be3c8042f0 0x55be3c8130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be3c802be0 0x55be3c811b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 --2- 172.21.15.31:0/2596265997 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be3c803530 0x55be3c812090 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 >> 172.21.15.31:0/2596265997 conn(0x55be3c7f83e0 msgr2=0x55be3c81a670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.797+0000 7fcf0f7fe700 0 client.30103 ms_handle_reset on v2:172.21.15.156:6838/2888902152 2023-07-29T16:31:17.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.798+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 shutdown_connections 2023-07-29T16:31:17.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.798+0000 7fcf49f5db80 1 -- 172.21.15.31:0/2596265997 wait complete. 2023-07-29T16:31:17.803 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "damage_type": "backtrace", 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 3731810628, 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 1099511645624, 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902/client.0/tmp/clients/client3/~dmtmp/PWRPNT/NEWPCB.PPT" 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:31:17.804 INFO:teuthology.orchestra.run.smithi031.stdout:] 2023-07-29T16:31:17.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.798+0000 7fcf49f5db80 20 client.30103 trim_cache size 0 max 16384 2023-07-29T16:31:17.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.800+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcf180a0910 msgr2=0x7fcf180a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.800+0000 7fcf365ff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcf180a0910 0x7fcf180a2dc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55be3c7f8360 tx=0x7fcf24009f60 comp rx=0 tx=0).stop 2023-07-29T16:31:17.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.800+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 msgr2=0x7fcf3010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:17.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.800+0000 7fcf365ff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf3010f310 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000b500 tx=0x7fcf2000b810 comp rx=0 tx=0).stop 2023-07-29T16:31:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 shutdown_connections 2023-07-29T16:31:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fcf180a0910 0x7fcf180a2dc0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf30100da0 0x7fcf301168c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf300ff220 0x7fcf3010f310 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 --2- 172.21.15.31:0/4200462163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf30100410 0x7fcf3010f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 >> 172.21.15.31:0/4200462163 conn(0x7fcf300fa840 msgr2=0x7fcf30103640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 shutdown_connections 2023-07-29T16:31:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:17.801+0000 7fcf365ff700 1 -- 172.21.15.31:0/4200462163 wait complete. 2023-07-29T16:31:17.837 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-07-29T16:31:18.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:18 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/643106975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:31:18.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:18 smithi031 ceph-mon[95504]: pgmap v846: 97 pgs: 97 active+clean; 509 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 8.7 MiB/s wr, 748 op/s 2023-07-29T16:31:18.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.193+0000 7f3d2efff700 1 -- 172.21.15.31:0/3321666301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d281036c0 msgr2=0x7f3d28105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.193+0000 7f3d2efff700 1 --2- 172.21.15.31:0/3321666301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d281036c0 0x7f3d28105aa0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b0a0 tx=0x7f3d1800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.194+0000 7f3d2efff700 1 -- 172.21.15.31:0/3321666301 shutdown_connections 2023-07-29T16:31:18.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.194+0000 7f3d2efff700 1 --2- 172.21.15.31:0/3321666301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d28108440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.194+0000 7f3d2efff700 1 --2- 172.21.15.31:0/3321666301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d28100da0 0x7f3d28103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.194+0000 7f3d2efff700 1 --2- 172.21.15.31:0/3321666301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d281036c0 0x7f3d28105aa0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.194+0000 7f3d2efff700 1 -- 172.21.15.31:0/3321666301 >> 172.21.15.31:0/3321666301 conn(0x7f3d280fa840 msgr2=0x7f3d280fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.194+0000 7f3d2efff700 1 -- 172.21.15.31:0/3321666301 shutdown_connections 2023-07-29T16:31:18.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.196+0000 7f3d2efff700 1 -- 172.21.15.31:0/3321666301 wait complete. 2023-07-29T16:31:18.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.197+0000 7f3d2efff700 1 Processor -- start 2023-07-29T16:31:18.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.197+0000 7f3d2efff700 1 -- start start 2023-07-29T16:31:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d28100da0 0x7f3d28196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d281036c0 0x7f3d281965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d2819d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819dc10 con 0x7f3d281036c0 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819dd50 con 0x7f3d28100da0 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2819dee0 con 0x7f3d28105fe0 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2d59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d2819d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2d59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d2819d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34466/0 (socket says 172.21.15.31:34466) 2023-07-29T16:31:18.200 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2d59c700 1 -- 172.21.15.31:0/697230516 learned_addr learned my addr 172.21.15.31:0/697230516 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.198+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d28100da0 0x7f3d28196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d27fff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d281036c0 0x7f3d281965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d2d59c700 1 -- 172.21.15.31:0/697230516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d28100da0 msgr2=0x7f3d28196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d2d59c700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d28100da0 0x7f3d28196090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d2d59c700 1 -- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d281036c0 msgr2=0x7f3d281965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d2d59c700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d281036c0 0x7f3d281965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.201 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d2d59c700 1 -- 172.21.15.31:0/697230516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d18009d00 con 0x7f3d28105fe0 2023-07-29T16:31:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.199+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d28100da0 0x7f3d28196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:31:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.201+0000 7f3d2d59c700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d2819d640 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00e940 tx=0x7f3d1c00ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.202 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.201+0000 7f3d25ffb700 1 -- 172.21.15.31:0/697230516 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0097b0 con 0x7f3d28105fe0 2023-07-29T16:31:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.201+0000 7f3d25ffb700 1 -- 172.21.15.31:0/697230516 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d1c00ee90 con 0x7f3d28105fe0 2023-07-29T16:31:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.201+0000 7f3d25ffb700 1 -- 172.21.15.31:0/697230516 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c005520 con 0x7f3d28105fe0 2023-07-29T16:31:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.201+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d280febf0 con 0x7f3d28105fe0 2023-07-29T16:31:18.203 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.201+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d280ff0b0 con 0x7f3d28105fe0 2023-07-29T16:31:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.207+0000 7f3d25ffb700 1 -- 172.21.15.31:0/697230516 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3d1c004830 con 0x7f3d28105fe0 2023-07-29T16:31:18.209 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.208+0000 7f3d428a3b80 1 Processor -- start 2023-07-29T16:31:18.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 -- start start 2023-07-29T16:31:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 0x560b6cf85800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf86570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf86f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6ced4a10 con 0x560b6cf85d40 2023-07-29T16:31:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6ced23d0 con 0x560b6cf853f0 2023-07-29T16:31:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cecdcc0 con 0x560b6cf86b00 2023-07-29T16:31:18.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.209+0000 7f3d25ffb700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d100a0810 0x7f3d100a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 0x560b6cf85800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 0x560b6cf85800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41268/0 (socket says 172.21.15.31:41268) 2023-07-29T16:31:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 -- 172.21.15.31:0/2462238159 learned_addr learned my addr 172.21.15.31:0/2462238159 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d25ffb700 1 -- 172.21.15.31:0/697230516 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3d1c014070 con 0x7f3d28105fe0 2023-07-29T16:31:18.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf86f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf86570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 -- 172.21.15.31:0/2462238159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 msgr2=0x560b6cf86f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf86f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 -- 172.21.15.31:0/2462238159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 msgr2=0x560b6cf86570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf86570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 -- 172.21.15.31:0/2462238159 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b6ccfbde0 con 0x560b6cf853f0 2023-07-29T16:31:18.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.210+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 0x560b6cf85800 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b0a0 tx=0x7f3d180096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8681ecb5ddb1a27a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.211+0000 7f3d0effd700 1 -- 172.21.15.31:0/2462238159 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800e070 con 0x560b6cf853f0 2023-07-29T16:31:18.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.211+0000 7f3d0effd700 1 -- 172.21.15.31:0/2462238159 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d18007960 con 0x560b6cf853f0 2023-07-29T16:31:18.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.211+0000 7f3d0effd700 1 -- 172.21.15.31:0/2462238159 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18005660 con 0x560b6cf853f0 2023-07-29T16:31:18.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.211+0000 7f3d2d59c700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d100a0810 0x7f3d100a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2462238159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 msgr2=0x560b6cf85800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 0x560b6cf85800 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800b0a0 tx=0x7f3d180096a0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d2d59c700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d100a0810 0x7f3d100a2cc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c018000 tx=0x7f3d1c0047c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2462238159 shutdown_connections 2023-07-29T16:31:18.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf86f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf853f0 0x560b6cf85800 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2462238159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf86570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.212+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2462238159 >> 172.21.15.31:0/2462238159 conn(0x560b6cf723e0 msgr2=0x560b6cf77990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.213+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2462238159 shutdown_connections 2023-07-29T16:31:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.214+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2462238159 wait complete. 2023-07-29T16:31:18.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.215+0000 7f3d428a3b80 1 -- start start 2023-07-29T16:31:18.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.215+0000 7f3d428a3b80 1 Processor -- start 2023-07-29T16:31:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf853f0 0x560b6cf8f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf85d40 0x560b6cf8f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf96640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6ce6a510 con 0x560b6cf853f0 2023-07-29T16:31:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6ce6a1b0 con 0x560b6cf85d40 2023-07-29T16:31:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cd51920 con 0x560b6cf86b00 2023-07-29T16:31:18.225 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.216+0000 7f3d428a3b80 10 client.0 _ll_register_callbacks cb 0x560b6ce73620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:18.233 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.217+0000 7f3d2cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf85d40 0x560b6cf8f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.217+0000 7f3d2cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf85d40 0x560b6cf8f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41284/0 (socket says 172.21.15.31:41284) 2023-07-29T16:31:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.217+0000 7f3d2cd9b700 1 -- 172.21.15.31:0/2973913933 learned_addr learned my addr 172.21.15.31:0/2973913933 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.217+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf96640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.217+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf853f0 0x560b6cf8f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d27fff700 1 -- 172.21.15.31:0/2973913933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 msgr2=0x560b6cf96640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf96640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.235 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d27fff700 1 -- 172.21.15.31:0/2973913933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf85d40 msgr2=0x560b6cf8f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf85d40 0x560b6cf8f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d27fff700 1 -- 172.21.15.31:0/2973913933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d28109030 con 0x560b6cf853f0 2023-07-29T16:31:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d27fff700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf853f0 0x560b6cf8f0c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009fd0 tx=0x7f3d1800b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370c2c932456e48 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d07fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:31:18.237 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.218+0000 7f3d07fff700 1 -- 172.21.15.31:0/2973913933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18026070 con 0x560b6cf853f0 2023-07-29T16:31:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.219+0000 7f3d07fff700 1 -- 172.21.15.31:0/2973913933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3d18005570 con 0x560b6cf853f0 2023-07-29T16:31:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.219+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b6cef8d10 con 0x560b6cf853f0 2023-07-29T16:31:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.219+0000 7f3d07fff700 1 -- 172.21.15.31:0/2973913933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d1800e070 con 0x560b6cf853f0 2023-07-29T16:31:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.219+0000 7f3d428a3b80 10 client.19923 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:18.238 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.219+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x560b6cce4c60 con 0x560b6cf853f0 2023-07-29T16:31:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d07fff700 1 -- 172.21.15.31:0/2973913933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560b6cef8d10 con 0x560b6cf853f0 2023-07-29T16:31:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d07fff700 1 -- 172.21.15.31:0/2973913933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d18019030 con 0x560b6cf853f0 2023-07-29T16:31:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d428a3b80 10 client.19923 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:18.239 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d428a3b80 10 client.19923 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:31:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d428a3b80 20 client.19923 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cf807e0 0x560b6cf82bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.240 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d428a3b80 4 client.19923 mds_command: new command op to 24551 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.220+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b6cf4bd70 con 0x560b6cf807e0 2023-07-29T16:31:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.221+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cf807e0 0x560b6cf82bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.222+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cf807e0 0x560b6cf82bf0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f9f76121583a0ab9 server_cookie=dd7f6e65bc578f2d in_seq=0 out_seq=0 2023-07-29T16:31:18.241 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.222+0000 7f3d07fff700 10 client.19923 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:31:18.242 INFO:tasks.workunit.client.0.smithi031.stdout: 10 36567 34.50 MB/sec warmup 62 sec latency 88.885 ms 2023-07-29T16:31:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.224+0000 7f3d07fff700 1 -- 172.21.15.31:0/2973913933 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x560b6cf4bd70 con 0x560b6cf807e0 2023-07-29T16:31:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.224+0000 7f3d07fff700 10 client.19923 handle_command_reply: tid=0 2023-07-29T16:31:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.224+0000 7f3d428a3b80 1 client.19923 shutdown 2023-07-29T16:31:18.244 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf853f0 msgr2=0x560b6cf8f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf853f0 0x560b6cf8f0c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009fd0 tx=0x7f3d1800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 shutdown_connections 2023-07-29T16:31:18.245 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cf807e0 0x560b6cf82bf0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf96640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf853f0 0x560b6cf8f0c0 secure :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009fd0 tx=0x7f3d1800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/2973913933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf85d40 0x560b6cf8f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.246 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 >> 172.21.15.31:0/2973913933 conn(0x560b6cf723e0 msgr2=0x560b6cf77060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.225+0000 7f3d07fff700 0 client.19923 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:31:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.227+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 shutdown_connections 2023-07-29T16:31:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.227+0000 7f3d428a3b80 1 -- 172.21.15.31:0/2973913933 wait complete. 2023-07-29T16:31:18.247 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.227+0000 7f3d428a3b80 20 client.19923 trim_cache size 0 max 16384 2023-07-29T16:31:18.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.251+0000 7f3d428a3b80 1 Processor -- start 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 -- start start 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf945c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf89a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86ed0 0x560b6cf89fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cef8d10 con 0x560b6cf85d40 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cf8ec30 con 0x560b6cf86ed0 2023-07-29T16:31:18.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cf8e3a0 con 0x560b6cf86b00 2023-07-29T16:31:18.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.252+0000 7f3d2cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf89a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf89a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34478/0 (socket says 172.21.15.31:34478) 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 -- 172.21.15.31:0/3276049233 learned_addr learned my addr 172.21.15.31:0/3276049233 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 -- 172.21.15.31:0/3276049233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86ed0 msgr2=0x560b6cf89fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86ed0 0x560b6cf89fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 -- 172.21.15.31:0/3276049233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 msgr2=0x560b6cf945c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf945c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 -- 172.21.15.31:0/3276049233 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b6cd22250 con 0x560b6cf86b00 2023-07-29T16:31:18.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf89a90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3d20005950 tx=0x7f3d2000b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548d1e67275adc19 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.253+0000 7f3d0f7fe700 1 -- 172.21.15.31:0/3276049233 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20017040 con 0x560b6cf86b00 2023-07-29T16:31:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d0f7fe700 1 -- 172.21.15.31:0/3276049233 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d20019040 con 0x560b6cf86b00 2023-07-29T16:31:18.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d0f7fe700 1 -- 172.21.15.31:0/3276049233 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d200103d0 con 0x560b6cf86b00 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 -- 172.21.15.31:0/3276049233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 msgr2=0x560b6cf89a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf89a90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3d20005950 tx=0x7f3d2000b410 comp rx=0 tx=0).stop 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 -- 172.21.15.31:0/3276049233 shutdown_connections 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86ed0 0x560b6cf89fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf85d40 0x560b6cf945c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/3276049233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf86b00 0x560b6cf89a90 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.254+0000 7f3d428a3b80 1 -- 172.21.15.31:0/3276049233 >> 172.21.15.31:0/3276049233 conn(0x560b6cf723e0 msgr2=0x560b6cf827f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.255+0000 7f3d428a3b80 1 -- 172.21.15.31:0/3276049233 shutdown_connections 2023-07-29T16:31:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.255+0000 7f3d428a3b80 1 -- 172.21.15.31:0/3276049233 wait complete. 2023-07-29T16:31:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.255+0000 7f3d428a3b80 1 -- start start 2023-07-29T16:31:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.255+0000 7f3d428a3b80 1 Processor -- start 2023-07-29T16:31:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf85d40 0x560b6cfe0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86b00 0x560b6cfe0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf86ed0 0x560b6cf94550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cfe34c0 con 0x560b6cf86ed0 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cfe3630 con 0x560b6cf86b00 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b6cfe37f0 con 0x560b6cf85d40 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d428a3b80 10 client.0 _ll_register_callbacks cb 0x560b6ce73620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d2cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86b00 0x560b6cfe0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d2cd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86b00 0x560b6cfe0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41312/0 (socket says 172.21.15.31:41312) 2023-07-29T16:31:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d2cd9b700 1 -- 172.21.15.31:0/1394322419 learned_addr learned my addr 172.21.15.31:0/1394322419 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.256+0000 7f3d27fff700 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf85d40 0x560b6cfe0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d27fff700 1 -- 172.21.15.31:0/1394322419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86b00 msgr2=0x560b6cfe0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d27fff700 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86b00 0x560b6cfe0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d27fff700 1 -- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf86ed0 msgr2=0x560b6cf94550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d27fff700 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf86ed0 0x560b6cf94550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d27fff700 1 -- 172.21.15.31:0/1394322419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d20011560 con 0x560b6cf85d40 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d27fff700 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf85d40 0x560b6cfe0040 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800d010 tx=0x7f3d18006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2c2fa4392703a47 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d07fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d07fff700 1 -- 172.21.15.31:0/1394322419 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800e070 con 0x560b6cf85d40 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.257+0000 7f3d07fff700 1 -- 172.21.15.31:0/1394322419 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3d180095b0 con 0x560b6cf85d40 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.258+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d18007960 con 0x560b6cf85d40 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.258+0000 7f3d07fff700 1 -- 172.21.15.31:0/1394322419 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d18024070 con 0x560b6cf85d40 2023-07-29T16:31:18.262 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.258+0000 7f3d428a3b80 10 client.30121 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.258+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x560b6ccfa2b0 con 0x560b6cf85d40 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.258+0000 7f3d07fff700 1 -- 172.21.15.31:0/1394322419 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d1805b030 con 0x560b6cf85d40 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.258+0000 7f3d07fff700 1 -- 172.21.15.31:0/1394322419 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d1800b0d0 con 0x560b6cf85d40 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.259+0000 7f3d428a3b80 10 client.30121 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.259+0000 7f3d428a3b80 10 client.30121 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.259+0000 7f3d428a3b80 20 client.30121 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.259+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cff5b80 0x560b6cff7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.263 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.259+0000 7f3d428a3b80 4 client.30121 mds_command: new command op to 24551 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:31:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.259+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 --> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560b6cec6d40 con 0x560b6cff5b80 2023-07-29T16:31:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.261+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cff5b80 0x560b6cff7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.264 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.261+0000 7f3d2cd9b700 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cff5b80 0x560b6cff7f60 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58ff7dbbf732ee3b server_cookie=17214a5438a9134e in_seq=0 out_seq=0 2023-07-29T16:31:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.261+0000 7f3d07fff700 10 client.30121 ms_handle_connect on v2:172.21.15.31:6838/3329463675 2023-07-29T16:31:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.263+0000 7f3d07fff700 1 -- 172.21.15.31:0/1394322419 <== mds.1 v2:172.21.15.31:6838/3329463675 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560b6cec6d40 con 0x560b6cff5b80 2023-07-29T16:31:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.263+0000 7f3d07fff700 10 client.30121 handle_command_reply: tid=0 2023-07-29T16:31:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.263+0000 7f3d428a3b80 1 client.30121 shutdown 2023-07-29T16:31:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.263+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf85d40 msgr2=0x560b6cfe0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.265 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.263+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf85d40 0x560b6cfe0040 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800d010 tx=0x7f3d18006000 comp rx=0 tx=0).stop 2023-07-29T16:31:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 shutdown_connections 2023-07-29T16:31:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:6838/3329463675,v1:172.21.15.31:6839/3329463675] conn(0x560b6cff5b80 0x560b6cff7f60 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560b6cf86ed0 0x560b6cf94550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560b6cf85d40 0x560b6cfe0040 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.266 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 --2- 172.21.15.31:0/1394322419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560b6cf86b00 0x560b6cfe0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 >> 172.21.15.31:0/1394322419 conn(0x560b6cf723e0 msgr2=0x560b6cf89a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d07fff700 0 client.30121 ms_handle_reset on v2:172.21.15.31:6838/3329463675 2023-07-29T16:31:18.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 shutdown_connections 2023-07-29T16:31:18.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 1 -- 172.21.15.31:0/1394322419 wait complete. 2023-07-29T16:31:18.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.265+0000 7f3d428a3b80 20 client.30121 trim_cache size 0 max 16384 2023-07-29T16:31:18.267 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:31:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.268+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d100a0810 msgr2=0x7f3d100a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.268 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.268+0000 7f3d2efff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d100a0810 0x7f3d100a2cc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c018000 tx=0x7f3d1c0047c0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.268+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 msgr2=0x7f3d2819d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.268+0000 7f3d2efff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d2819d640 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00e940 tx=0x7f3d1c00ec50 comp rx=0 tx=0).stop 2023-07-29T16:31:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.269+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 shutdown_connections 2023-07-29T16:31:18.269 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.269+0000 7f3d2efff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f3d100a0810 0x7f3d100a2cc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.269+0000 7f3d2efff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3d28105fe0 0x7f3d2819d640 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.269+0000 7f3d2efff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d28100da0 0x7f3d28196090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.269+0000 7f3d2efff700 1 --2- 172.21.15.31:0/697230516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3d281036c0 0x7f3d281965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.269+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 >> 172.21.15.31:0/697230516 conn(0x7f3d280fa840 msgr2=0x7f3d28101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.271+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 shutdown_connections 2023-07-29T16:31:18.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.271+0000 7f3d2efff700 1 -- 172.21.15.31:0/697230516 wait complete. 2023-07-29T16:31:18.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-07-29T16:31:18.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:18 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/643106975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:31:18.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:18 smithi156 ceph-mon[129245]: pgmap v846: 97 pgs: 97 active+clean; 509 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 8.7 MiB/s wr, 748 op/s 2023-07-29T16:31:18.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:18 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/643106975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:31:18.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:18 smithi039 ceph-mon[128858]: pgmap v846: 97 pgs: 97 active+clean; 509 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 8.7 MiB/s wr, 748 op/s 2023-07-29T16:31:18.622 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 -- 172.21.15.31:0/3094053411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 msgr2=0x7ff210110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 --2- 172.21.15.31:0/3094053411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 0x7ff210110da0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff20800faf0 tx=0x7ff20800fe00 comp rx=0 tx=0).stop 2023-07-29T16:31:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 -- 172.21.15.31:0/3094053411 shutdown_connections 2023-07-29T16:31:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 --2- 172.21.15.31:0/3094053411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 0x7ff210110da0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 --2- 172.21.15.31:0/3094053411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff2100693b0 0x7ff2100697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 --2- 172.21.15.31:0/3094053411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff21010c0a0 0x7ff21010e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.623 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.621+0000 7ff2161ff700 1 -- 172.21.15.31:0/3094053411 >> 172.21.15.31:0/3094053411 conn(0x7ff21006e280 msgr2=0x7ff2100706b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.622+0000 7ff2161ff700 1 -- 172.21.15.31:0/3094053411 shutdown_connections 2023-07-29T16:31:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.622+0000 7ff2161ff700 1 -- 172.21.15.31:0/3094053411 wait complete. 2023-07-29T16:31:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.623+0000 7ff2161ff700 1 Processor -- start 2023-07-29T16:31:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 -- start start 2023-07-29T16:31:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 0x7ff21019c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff21010c0a0 0x7ff21019cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 0x7ff2101a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101a42a0 con 0x7ff21010c0a0 2023-07-29T16:31:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101a43e0 con 0x7ff2100693b0 2023-07-29T16:31:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff2161ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101a4570 con 0x7ff21010e9c0 2023-07-29T16:31:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.624+0000 7ff20f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 0x7ff21019c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.626 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 0x7ff21019c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41320/0 (socket says 172.21.15.31:41320) 2023-07-29T16:31:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 -- 172.21.15.31:0/371980103 learned_addr learned my addr 172.21.15.31:0/371980103 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20effd700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff21010c0a0 0x7ff21019cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 -- 172.21.15.31:0/371980103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 msgr2=0x7ff2101a3cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:31:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 0x7ff2101a3cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 -- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff21010c0a0 msgr2=0x7ff21019cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff21010c0a0 0x7ff21019cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.625+0000 7ff20f7fe700 1 -- 172.21.15.31:0/371980103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20800f750 con 0x7ff2100693b0 2023-07-29T16:31:18.628 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.626+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 0x7ff21019c720 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff20000ba50 tx=0x7ff20000bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.628+0000 7ff20cff9700 1 -- 172.21.15.31:0/371980103 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000c9b0 con 0x7ff2100693b0 2023-07-29T16:31:18.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.628+0000 7ff20cff9700 1 -- 172.21.15.31:0/371980103 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff200014440 con 0x7ff2100693b0 2023-07-29T16:31:18.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.628+0000 7ff20cff9700 1 -- 172.21.15.31:0/371980103 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2000075d0 con 0x7ff2100693b0 2023-07-29T16:31:18.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.628+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2101a4850 con 0x7ff2100693b0 2023-07-29T16:31:18.629 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.628+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2101a4d90 con 0x7ff2100693b0 2023-07-29T16:31:18.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.634+0000 7ff229c1ab80 1 Processor -- start 2023-07-29T16:31:18.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 -- start start 2023-07-29T16:31:18.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ad3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad4530 0x561352ad4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad52f0 0x561352ad5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352a2ba10 con 0x561352ad4530 2023-07-29T16:31:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352a293d0 con 0x561352ad52f0 2023-07-29T16:31:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.635+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352a24cc0 con 0x561352ad3be0 2023-07-29T16:31:18.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20cff9700 1 -- 172.21.15.31:0/371980103 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff2000145b0 con 0x7ff2100693b0 2023-07-29T16:31:18.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ad3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ad3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34512/0 (socket says 172.21.15.31:34512) 2023-07-29T16:31:18.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 -- 172.21.15.31:0/2121661501 learned_addr learned my addr 172.21.15.31:0/2121661501 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 -- 172.21.15.31:0/2121661501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad52f0 msgr2=0x561352ad5740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:31:18.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad52f0 0x561352ad5740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 -- 172.21.15.31:0/2121661501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad4530 msgr2=0x561352ad4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad4530 0x561352ad4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 -- 172.21.15.31:0/2121661501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561352852de0 con 0x561352ad3be0 2023-07-29T16:31:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.636+0000 7ff20ffff700 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ad3ff0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff208005910 tx=0x7ff208005ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b25dc3df8e36de server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.637+0000 7ff1f5ffb700 1 -- 172.21.15.31:0/2121661501 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080050e0 con 0x561352ad3be0 2023-07-29T16:31:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.637+0000 7ff1f5ffb700 1 -- 172.21.15.31:0/2121661501 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff208027070 con 0x561352ad3be0 2023-07-29T16:31:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.637+0000 7ff1f5ffb700 1 -- 172.21.15.31:0/2121661501 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2080038e0 con 0x561352ad3be0 2023-07-29T16:31:18.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 -- 172.21.15.31:0/2121661501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 msgr2=0x561352ad3ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ad3ff0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff208005910 tx=0x7ff208005ab0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff20cff9700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1f80a0990 0x7ff1f80a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.644 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 -- 172.21.15.31:0/2121661501 shutdown_connections 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad52f0 0x561352ad5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ad3ff0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/2121661501 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad4530 0x561352ad4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff229c1ab80 1 -- 172.21.15.31:0/2121661501 >> 172.21.15.31:0/2121661501 conn(0x561352ac93e0 msgr2=0x561352ace990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff20cff9700 1 -- 172.21.15.31:0/371980103 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff200006e40 con 0x7ff2100693b0 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.638+0000 7ff20ffff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1f80a0990 0x7ff1f80a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.639+0000 7ff229c1ab80 1 -- 172.21.15.31:0/2121661501 shutdown_connections 2023-07-29T16:31:18.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.640+0000 7ff229c1ab80 1 -- 172.21.15.31:0/2121661501 wait complete. 2023-07-29T16:31:18.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.640+0000 7ff229c1ab80 1 -- start start 2023-07-29T16:31:18.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.640+0000 7ff20ffff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1f80a0990 0x7ff1f80a2e40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff208018cf0 tx=0x7ff208018f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.640+0000 7ff229c1ab80 1 Processor -- start 2023-07-29T16:31:18.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad3be0 0x561352add840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352addd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.646 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad52f0 0x561352ae4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613529c1510 con 0x561352ad3be0 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613529c11b0 con 0x561352ad4530 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613528a8920 con 0x561352ad52f0 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff229c1ab80 10 client.0 _ll_register_callbacks cb 0x5613529ca620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff20effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352addd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff20effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352addd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41346/0 (socket says 172.21.15.31:41346) 2023-07-29T16:31:18.647 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.641+0000 7ff20effd700 1 -- 172.21.15.31:0/1112978243 learned_addr learned my addr 172.21.15.31:0/1112978243 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20ffff700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad52f0 0x561352ae4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20effd700 1 -- 172.21.15.31:0/1112978243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad52f0 msgr2=0x561352ae4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20effd700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad52f0 0x561352ae4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad3be0 0x561352add840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20effd700 1 -- 172.21.15.31:0/1112978243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad3be0 msgr2=0x561352add840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20effd700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad3be0 0x561352add840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20effd700 1 -- 172.21.15.31:0/1112978243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff210111890 con 0x561352ad4530 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad3be0 0x561352add840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff20effd700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352addd80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff2040116c0 tx=0x7ff204011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10617e1e1fb82120 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.642+0000 7ff1eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.643+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1112978243 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20400e740 con 0x561352ad4530 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.643+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1112978243 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff20400e8a0 con 0x561352ad4530 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.643+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561352a4fd10 con 0x561352ad4530 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.643+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1112978243 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff20400ea60 con 0x561352ad4530 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 10 client.30110 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x56135283bc60 con 0x561352ad4530 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1112978243 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561352a4fd10 con 0x561352ad4530 2023-07-29T16:31:18.651 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1112978243 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff204019c50 con 0x561352ad4530 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 10 client.30110 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 10 client.30110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 20 client.30110 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352ae9660 0x561352aeba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 4 client.30110 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.644+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561352aa2d70 con 0x561352ae9660 2023-07-29T16:31:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.645+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352ae9660 0x561352aeba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.645+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352ae9660 0x561352aeba70 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7646aeec5c1ad55 server_cookie=15aa5b8c70193df in_seq=0 out_seq=0 2023-07-29T16:31:18.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.645+0000 7ff1eeffd700 10 client.30110 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:18.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.648+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1112978243 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561352aa2d70 con 0x561352ae9660 2023-07-29T16:31:18.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.648+0000 7ff1eeffd700 10 client.30110 handle_command_reply: tid=0 2023-07-29T16:31:18.653 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.648+0000 7ff229c1ab80 1 client.30110 shutdown 2023-07-29T16:31:18.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 msgr2=0x561352addd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352addd80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff2040116c0 tx=0x7ff204011a80 comp rx=0 tx=0).stop 2023-07-29T16:31:18.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 shutdown_connections 2023-07-29T16:31:18.654 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352ae9660 0x561352aeba70 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad52f0 0x561352ae4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad3be0 0x561352add840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1112978243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352addd80 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.649+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 >> 172.21.15.31:0/1112978243 conn(0x561352ac93e0 msgr2=0x561352ace060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.650+0000 7ff1eeffd700 0 client.30110 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.650+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 shutdown_connections 2023-07-29T16:31:18.655 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.650+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1112978243 wait complete. 2023-07-29T16:31:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.653+0000 7ff229c1ab80 20 client.30110 trim_cache size 0 max 16384 2023-07-29T16:31:18.674 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.673+0000 7ff229c1ab80 1 Processor -- start 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 -- start start 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad3be0 0x561352aeb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 0x561352ae4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352adb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352ac66e0 con 0x561352ad52f0 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352ace770 con 0x561352ad3be0 2023-07-29T16:31:18.675 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352ade4c0 con 0x561352ad4530 2023-07-29T16:31:18.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.674+0000 7ff20effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 0x561352ae4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 0x561352ae4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:34532/0 (socket says 172.21.15.31:34532) 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 -- 172.21.15.31:0/3454957012 learned_addr learned my addr 172.21.15.31:0/3454957012 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad3be0 0x561352aeb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 -- 172.21.15.31:0/3454957012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad3be0 msgr2=0x561352aeb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad3be0 0x561352aeb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 -- 172.21.15.31:0/3454957012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 msgr2=0x561352adb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352adb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 -- 172.21.15.31:0/3454957012 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561352879250 con 0x561352ad4530 2023-07-29T16:31:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad3be0 0x561352aeb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.675+0000 7ff20effd700 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 0x561352ae4d80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff204019000 tx=0x7ff204009350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fddb4fb29b3658a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.676+0000 7ff1f67fc700 1 -- 172.21.15.31:0/3454957012 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040226c0 con 0x561352ad4530 2023-07-29T16:31:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff1f67fc700 1 -- 172.21.15.31:0/3454957012 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff204012e90 con 0x561352ad4530 2023-07-29T16:31:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff1f67fc700 1 -- 172.21.15.31:0/3454957012 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204005720 con 0x561352ad4530 2023-07-29T16:31:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 -- 172.21.15.31:0/3454957012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 msgr2=0x561352ae4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 0x561352ae4d80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff204019000 tx=0x7ff204009350 comp rx=0 tx=0).stop 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 -- 172.21.15.31:0/3454957012 shutdown_connections 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352adb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad3be0 0x561352aeb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/3454957012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad4530 0x561352ae4d80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.677+0000 7ff229c1ab80 1 -- 172.21.15.31:0/3454957012 >> 172.21.15.31:0/3454957012 conn(0x561352ac93e0 msgr2=0x561352ace060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.678+0000 7ff229c1ab80 1 -- 172.21.15.31:0/3454957012 shutdown_connections 2023-07-29T16:31:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.678+0000 7ff229c1ab80 1 -- 172.21.15.31:0/3454957012 wait complete. 2023-07-29T16:31:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.678+0000 7ff229c1ab80 1 -- start start 2023-07-29T16:31:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.678+0000 7ff229c1ab80 1 Processor -- start 2023-07-29T16:31:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ae29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352ae2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352ae3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352ae4450 con 0x561352ad52f0 2023-07-29T16:31:18.682 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352ae45c0 con 0x561352ad4530 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561352ae4780 con 0x561352ad3be0 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff229c1ab80 10 client.0 _ll_register_callbacks cb 0x5613529ca620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.679+0000 7ff20f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ae29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352ae2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352ae2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41370/0 (socket says 172.21.15.31:41370) 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 -- 172.21.15.31:0/1129477380 learned_addr learned my addr 172.21.15.31:0/1129477380 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:18.683 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20ffff700 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352ae3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 -- 172.21.15.31:0/1129477380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 msgr2=0x561352ae29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ae29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 -- 172.21.15.31:0/1129477380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 msgr2=0x561352ae3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352ae3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 -- 172.21.15.31:0/1129477380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56135283bc60 con 0x561352ad4530 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff20effd700 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352ae2f00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ff2040092a0 tx=0x7ff204005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4636315086e7c163 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.680+0000 7ff1eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:31:18.684 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.681+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1129477380 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204022a60 con 0x561352ad4530 2023-07-29T16:31:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.681+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff208027070 con 0x561352ad4530 2023-07-29T16:31:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.681+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1129477380 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff204022bc0 con 0x561352ad4530 2023-07-29T16:31:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.681+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1129477380 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff204022da0 con 0x561352ad4530 2023-07-29T16:31:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.681+0000 7ff229c1ab80 10 client.30113 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:18.685 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.681+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5613528512b0 con 0x561352ad4530 2023-07-29T16:31:18.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1129477380 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff208027070 con 0x561352ad4530 2023-07-29T16:31:18.686 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1129477380 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff204034070 con 0x561352ad4530 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff229c1ab80 10 client.30113 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff229c1ab80 10 client.30113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff229c1ab80 20 client.30113 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352b3bbd0 0x561352b3dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff229c1ab80 4 client.30113 mds_command: new command op to 24581 tid=0[{"prefix": "damage ls"}] 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561352a1dd40 con 0x561352b3bbd0 2023-07-29T16:31:18.687 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.682+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352b3bbd0 0x561352b3dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.683+0000 7ff20f7fe700 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352b3bbd0 0x561352b3dfe0 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c42fcea1ac91b8e9 server_cookie=e573b3818850e56e in_seq=0 out_seq=0 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.683+0000 7ff1eeffd700 10 client.30113 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff1eeffd700 1 -- 172.21.15.31:0/1129477380 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561352a1dd40 con 0x561352b3bbd0 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff1eeffd700 10 client.30113 handle_command_reply: tid=0 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 client.30113 shutdown 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 msgr2=0x561352ae2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.688 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352ae2f00 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7ff2040092a0 tx=0x7ff204005c30 comp rx=0 tx=0).stop 2023-07-29T16:31:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 shutdown_connections 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561352b3bbd0 0x561352b3dfe0 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561352ad52f0 0x561352ae3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff1eeffd700 0 client.30113 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561352ad3be0 0x561352ae29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 --2- 172.21.15.31:0/1129477380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561352ad4530 0x561352ae2f00 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 >> 172.21.15.31:0/1129477380 conn(0x561352ac93e0 msgr2=0x561352aeb670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 shutdown_connections 2023-07-29T16:31:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.684+0000 7ff229c1ab80 1 -- 172.21.15.31:0/1129477380 wait complete. 2023-07-29T16:31:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.685+0000 7ff229c1ab80 20 client.30113 trim_cache size 0 max 16384 2023-07-29T16:31:18.691 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2023-07-29T16:31:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.686+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1f80a0990 msgr2=0x7ff1f80a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.686+0000 7ff2161ff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1f80a0990 0x7ff1f80a2e40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff208018cf0 tx=0x7ff208018f90 comp rx=0 tx=0).stop 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.686+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 msgr2=0x7ff21019c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.686+0000 7ff2161ff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 0x7ff21019c720 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7ff20000ba50 tx=0x7ff20000bd60 comp rx=0 tx=0).stop 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 shutdown_connections 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff1f80a0990 0x7ff1f80a2e40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff21010e9c0 0x7ff2101a3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff2100693b0 0x7ff21019c720 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 --2- 172.21.15.31:0/371980103 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff21010c0a0 0x7ff21019cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 >> 172.21.15.31:0/371980103 conn(0x7ff21006e280 msgr2=0x7ff21010cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 shutdown_connections 2023-07-29T16:31:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:31:18.687+0000 7ff2161ff700 1 -- 172.21.15.31:0/371980103 wait complete. 2023-07-29T16:31:18.718 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 3731810628, 'ino': 1099511645624, 'path': '/volumes/_nogroup/sv_1/5354c809-3264-4aed-854c-cd3a8f125902/client.0/tmp/clients/client3/~dmtmp/PWRPNT/NEWPCB.PPT'}] 2023-07-29T16:31:18.718 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2023-07-29T16:31:19.224 INFO:tasks.workunit.client.0.smithi031.stdout: 10 37517 34.71 MB/sec warmup 63 sec latency 40.370 ms 2023-07-29T16:31:20.224 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38378 35.10 MB/sec warmup 64 sec latency 50.814 ms 2023-07-29T16:31:20.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:20 smithi031 ceph-mon[95504]: pgmap v847: 97 pgs: 97 active+clean; 511 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 15 MiB/s wr, 1.02k op/s 2023-07-29T16:31:21.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:20 smithi039 ceph-mon[128858]: pgmap v847: 97 pgs: 97 active+clean; 511 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 15 MiB/s wr, 1.02k op/s 2023-07-29T16:31:21.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:20 smithi156 ceph-mon[129245]: pgmap v847: 97 pgs: 97 active+clean; 511 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 15 MiB/s wr, 1.02k op/s 2023-07-29T16:31:21.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38787 34.99 MB/sec warmup 65 sec latency 446.387 ms 2023-07-29T16:31:22.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 38882 34.50 MB/sec warmup 66 sec latency 901.545 ms 2023-07-29T16:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3399169293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e141015b0 msgr2=0x7f5e14103a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3399169293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e141015b0 0x7f5e14103a30 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f5e10009990 tx=0x7f5e10009ca0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3399169293 shutdown_connections 2023-07-29T16:31:22.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3399169293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e14103f70 0x7f5e141043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3399169293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e14101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3399169293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e141015b0 0x7f5e14103a30 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.546+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3399169293 >> 172.21.15.31:0/3399169293 conn(0x7f5e140fa840 msgr2=0x7f5e140fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:22.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.548+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3399169293 shutdown_connections 2023-07-29T16:31:22.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.549+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3399169293 wait complete. 2023-07-29T16:31:22.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.550+0000 7f5e1c1ff700 1 Processor -- start 2023-07-29T16:31:22.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.551+0000 7f5e1c1ff700 1 -- start start 2023-07-29T16:31:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.552+0000 7f5e1c1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e1419c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.552+0000 7f5e1c1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e141015b0 0x7f5e1419cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.552+0000 7f5e1c1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e14103f70 0x7f5e1419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.552+0000 7f5e1c1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1419e1c0 con 0x7f5e140fec50 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.552+0000 7f5e1c1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1419e330 con 0x7f5e14103f70 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.552+0000 7f5e1c1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1419e4f0 con 0x7f5e141015b0 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.553+0000 7f5e19f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e1419c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.553+0000 7f5e19f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e1419c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38452/0 (socket says 172.21.15.31:38452) 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.553+0000 7f5e19f9b700 1 -- 172.21.15.31:0/3362771147 learned_addr learned my addr 172.21.15.31:0/3362771147 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.554+0000 7f5e19f9b700 1 -- 172.21.15.31:0/3362771147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e141015b0 msgr2=0x7f5e1419cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.554+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e141015b0 0x7f5e1419cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.554+0000 7f5e19f9b700 1 -- 172.21.15.31:0/3362771147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e14103f70 msgr2=0x7f5e1419db60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:31:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.554+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e14103f70 0x7f5e1419db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.554+0000 7f5e19f9b700 1 -- 172.21.15.31:0/3362771147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e100095f0 con 0x7f5e140fec50 2023-07-29T16:31:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.554+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e1419c7a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400e970 tx=0x7f5e0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.555+0000 7f5e0affd700 1 -- 172.21.15.31:0/3362771147 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04009790 con 0x7f5e140fec50 2023-07-29T16:31:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.555+0000 7f5e0affd700 1 -- 172.21.15.31:0/3362771147 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e040044e0 con 0x7f5e140fec50 2023-07-29T16:31:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.555+0000 7f5e0affd700 1 -- 172.21.15.31:0/3362771147 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04003e00 con 0x7f5e140fec50 2023-07-29T16:31:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.555+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1419e7d0 con 0x7f5e140fec50 2023-07-29T16:31:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.555+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1406a950 con 0x7f5e140fec50 2023-07-29T16:31:22.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.563+0000 7f5e2fb85b80 1 Processor -- start 2023-07-29T16:31:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.564+0000 7f5e2fb85b80 1 -- start start 2023-07-29T16:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.565+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebd68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.565+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebd77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.565+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebd8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.565+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeeb2e9b0 con 0x557eeebd6df0 2023-07-29T16:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.565+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeeb2c370 con 0x557eeea326b0 2023-07-29T16:31:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.565+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeeb27c60 con 0x557eeebd7cf0 2023-07-29T16:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebd8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebd8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37410/0 (socket says 172.21.15.31:37410) 2023-07-29T16:31:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 -- 172.21.15.31:0/2457985565 learned_addr learned my addr 172.21.15.31:0/2457985565 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 -- 172.21.15.31:0/2457985565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 msgr2=0x557eeebd68b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:31:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebd68b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 -- 172.21.15.31:0/2457985565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 msgr2=0x557eeebd77b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:31:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebd77b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.566+0000 7f5e1979a700 1 -- 172.21.15.31:0/2457985565 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557eee955d80 con 0x557eeebd7cf0 2023-07-29T16:31:22.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.567+0000 7f5e1979a700 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebd8160 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000b680 tx=0x7f5e10009550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=500cb63ca4eefbb2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.568+0000 7f5e0affd700 1 -- 172.21.15.31:0/3362771147 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5e04003f60 con 0x7f5e140fec50 2023-07-29T16:31:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.570+0000 7f5dfffff700 1 -- 172.21.15.31:0/2457985565 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1001d070 con 0x557eeebd7cf0 2023-07-29T16:31:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.570+0000 7f5dfffff700 1 -- 172.21.15.31:0/2457985565 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e1000bd90 con 0x557eeebd7cf0 2023-07-29T16:31:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.570+0000 7f5dfffff700 1 -- 172.21.15.31:0/2457985565 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e10021410 con 0x557eeebd7cf0 2023-07-29T16:31:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.570+0000 7f5e0affd700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e000a0990 0x7f5e000a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.570+0000 7f5e0affd700 1 -- 172.21.15.31:0/3362771147 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5e04025070 con 0x7f5e140fec50 2023-07-29T16:31:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/2457985565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 msgr2=0x557eeebd8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebd8160 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000b680 tx=0x7f5e10009550 comp rx=0 tx=0).stop 2023-07-29T16:31:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e1a79c700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e000a0990 0x7f5e000a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/2457985565 shutdown_connections 2023-07-29T16:31:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebd8160 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebd68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/2457985565 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebd77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.571+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/2457985565 >> 172.21.15.31:0/2457985565 conn(0x557eeebcbfd0 msgr2=0x557eeebd1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.573+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/2457985565 shutdown_connections 2023-07-29T16:31:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.573+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/2457985565 wait complete. 2023-07-29T16:31:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.574+0000 7f5e2fb85b80 1 -- start start 2023-07-29T16:31:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.574+0000 7f5e2fb85b80 1 Processor -- start 2023-07-29T16:31:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.576+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebdff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.576+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.576+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebe74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.576+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeeac44b0 con 0x557eeebd6df0 2023-07-29T16:31:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.576+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeeac4150 con 0x557eeea326b0 2023-07-29T16:31:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.576+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eee9ab8c0 con 0x557eeebd7cf0 2023-07-29T16:31:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.577+0000 7f5e2fb85b80 10 client.0 _ll_register_callbacks cb 0x557eeeacd5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.577+0000 7f5e19f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebdff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.577+0000 7f5e19f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebdff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:51788/0 (socket says 172.21.15.31:51788) 2023-07-29T16:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.577+0000 7f5e19f9b700 1 -- 172.21.15.31:0/556056218 learned_addr learned my addr 172.21.15.31:0/556056218 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.577+0000 7f5e1a79c700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e000a0990 0x7f5e000a2e40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009510 tx=0x7f5e0c009300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.577+0000 7f5e1a79c700 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebe74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5e19f9b700 1 -- 172.21.15.31:0/556056218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 msgr2=0x557eeebe74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebe74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5e19f9b700 1 -- 172.21.15.31:0/556056218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 msgr2=0x557eeebe0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5e19f9b700 1 -- 172.21.15.31:0/556056218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e14104ee0 con 0x557eeea326b0 2023-07-29T16:31:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebdff20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f5e040b23e0 tx=0x7f5e040aa080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b63b4c46852ee7fe server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5dfd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/556056218 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e040c2ea0 con 0x557eeea326b0 2023-07-29T16:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.578+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/556056218 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5e040aa4c0 con 0x557eeea326b0 2023-07-29T16:31:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.579+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557eeeb52cb0 con 0x557eeea326b0 2023-07-29T16:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.579+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/556056218 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e040ae4f0 con 0x557eeea326b0 2023-07-29T16:31:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.579+0000 7f5e2fb85b80 10 client.30119 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.579+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x557eee93ec00 con 0x557eeea326b0 2023-07-29T16:31:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.579+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/556056218 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557eeeb52cb0 con 0x557eeea326b0 2023-07-29T16:31:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.580+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/556056218 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e040aa720 con 0x557eeea326b0 2023-07-29T16:31:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.580+0000 7f5e2fb85b80 10 client.30119 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.580+0000 7f5e2fb85b80 10 client.30119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.581+0000 7f5e2fb85b80 20 client.30119 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.581+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeebebc70 0x557eeebee080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.581+0000 7f5e2fb85b80 4 client.30119 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.581+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557eeeba5d10 con 0x557eeebebc70 2023-07-29T16:31:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.582+0000 7f5e1979a700 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeebebc70 0x557eeebee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.584+0000 7f5e1979a700 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeebebc70 0x557eeebee080 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18625fee6387211b server_cookie=9e92e95403dfdc7b in_seq=0 out_seq=0 2023-07-29T16:31:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.584+0000 7f5dfd7fa700 10 client.30119 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.586+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/556056218 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557eeeba5d10 con 0x557eeebebc70 2023-07-29T16:31:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.586+0000 7f5dfd7fa700 10 client.30119 handle_command_reply: tid=0 2023-07-29T16:31:22.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.586+0000 7f5e2fb85b80 1 client.30119 shutdown 2023-07-29T16:31:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 msgr2=0x557eeebdff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebdff20 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f5e040b23e0 tx=0x7f5e040aa080 comp rx=0 tx=0).stop 2023-07-29T16:31:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 shutdown_connections 2023-07-29T16:31:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeebebc70 0x557eeebee080 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeebd7cf0 0x557eeebe74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeea326b0 0x557eeebdff20 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/556056218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 >> 172.21.15.31:0/556056218 conn(0x557eeebcbfd0 msgr2=0x557eeebd0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.587+0000 7f5dfd7fa700 0 client.30119 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:22.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.588+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 shutdown_connections 2023-07-29T16:31:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.588+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/556056218 wait complete. 2023-07-29T16:31:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.588+0000 7f5e2fb85b80 20 client.30119 trim_cache size 0 max 16384 2023-07-29T16:31:22.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.608+0000 7f5e2fb85b80 1 Processor -- start 2023-07-29T16:31:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.608+0000 7f5e2fb85b80 1 -- start start 2023-07-29T16:31:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebd80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd7cf0 0x557eeebedc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeebd8600 con 0x557eeebd6df0 2023-07-29T16:31:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeebcb9d0 con 0x557eeebd7cf0 2023-07-29T16:31:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeebdde40 con 0x557eeea326b0 2023-07-29T16:31:22.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e19f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebd80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e19f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebd80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37422/0 (socket says 172.21.15.31:37422) 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.609+0000 7f5e19f9b700 1 -- 172.21.15.31:0/10385915 learned_addr learned my addr 172.21.15.31:0/10385915 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e1979a700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e1a79c700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd7cf0 0x557eeebedc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e19f9b700 1 -- 172.21.15.31:0/10385915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd7cf0 msgr2=0x557eeebedc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd7cf0 0x557eeebedc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e19f9b700 1 -- 172.21.15.31:0/10385915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 msgr2=0x557eeebe7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e19f9b700 1 -- 172.21.15.31:0/10385915 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557eee97c1f0 con 0x557eeea326b0 2023-07-29T16:31:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e1a79c700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd7cf0 0x557eeebedc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:31:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e1979a700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:31:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebd80c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5e04004800 tx=0x7f5e040a8a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f0c61e82feadb3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:22.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.610+0000 7f5e08ff9700 1 -- 172.21.15.31:0/10385915 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e040c2ea0 con 0x557eeea326b0 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e08ff9700 1 -- 172.21.15.31:0/10385915 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e040ae2f0 con 0x557eeea326b0 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e08ff9700 1 -- 172.21.15.31:0/10385915 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e040aa720 con 0x557eeea326b0 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/10385915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 msgr2=0x557eeebd80c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebd80c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5e04004800 tx=0x7f5e040a8a70 comp rx=0 tx=0).stop 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/10385915 shutdown_connections 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd7cf0 0x557eeebedc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebd80c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/10385915 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd6df0 0x557eeebe7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.611+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/10385915 >> 172.21.15.31:0/10385915 conn(0x557eeebcbfd0 msgr2=0x557eeebdec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.612+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/10385915 shutdown_connections 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.612+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/10385915 wait complete. 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.612+0000 7f5e2fb85b80 1 -- start start 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.612+0000 7f5e2fb85b80 1 Processor -- start 2023-07-29T16:31:22.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebe2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd6df0 0x557eeebe3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 0x557eeebe6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeebe6690 con 0x557eeebd7cf0 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeebe6800 con 0x557eeebd6df0 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557eeebe69c0 con 0x557eeea326b0 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e2fb85b80 10 client.0 _ll_register_callbacks cb 0x557eeeacd5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e1a79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 0x557eeebe6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e1979a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd6df0 0x557eeebe3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e1a79c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 0x557eeebe6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38468/0 (socket says 172.21.15.31:38468) 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.613+0000 7f5e1a79c700 1 -- 172.21.15.31:0/1775960486 learned_addr learned my addr 172.21.15.31:0/1775960486 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebe2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1979a700 1 -- 172.21.15.31:0/1775960486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 msgr2=0x557eeebe2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1979a700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebe2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1979a700 1 -- 172.21.15.31:0/1775960486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 msgr2=0x557eeebe6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:22.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1979a700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 0x557eeebe6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1979a700 1 -- 172.21.15.31:0/1775960486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557eee93ec00 con 0x557eeebd6df0 2023-07-29T16:31:22.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1a79c700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 0x557eeebe6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:31:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.614+0000 7f5e1979a700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd6df0 0x557eeebe3270 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000fb70 tx=0x7f5e10005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb7d4bf8d01b2e7c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5dfd7fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:31:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/1775960486 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1001d040 con 0x557eeebd6df0 2023-07-29T16:31:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/1775960486 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5e1002b410 con 0x557eeebd6df0 2023-07-29T16:31:22.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e1000bd90 con 0x557eeebd6df0 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/1775960486 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e1000f9e0 con 0x557eeebd6df0 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5e2fb85b80 10 client.30125 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.615+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x557eee954250 con 0x557eeebd6df0 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.616+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/1775960486 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e1000bd90 con 0x557eeebd6df0 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/1775960486 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e1002b830 con 0x557eeebd6df0 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e2fb85b80 10 client.30125 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e2fb85b80 10 client.30125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e2fb85b80 20 client.30125 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeec3dcf0 0x557eeec40100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e2fb85b80 4 client.30125 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:31:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557eeeb20ce0 con 0x557eeec3dcf0 2023-07-29T16:31:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.617+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeec3dcf0 0x557eeec40100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.618+0000 7f5e19f9b700 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeec3dcf0 0x557eeec40100 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8208341a490aae70 server_cookie=f4acd052a8923c75 in_seq=0 out_seq=0 2023-07-29T16:31:22.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.618+0000 7f5dfd7fa700 10 client.30125 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:22.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:22 smithi031 ceph-mon[95504]: pgmap v848: 97 pgs: 97 active+clean; 511 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 817 op/s 2023-07-29T16:31:22.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.965+0000 7f5dfd7fa700 1 -- 172.21.15.31:0/1775960486 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557eeeb20ce0 con 0x557eeec3dcf0 2023-07-29T16:31:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.965+0000 7f5dfd7fa700 10 client.30125 handle_command_reply: tid=0 2023-07-29T16:31:22.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.965+0000 7f5e2fb85b80 1 client.30125 shutdown 2023-07-29T16:31:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd6df0 msgr2=0x557eeebe3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd6df0 0x557eeebe3270 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000fb70 tx=0x7f5e10005960 comp rx=0 tx=0).stop 2023-07-29T16:31:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 shutdown_connections 2023-07-29T16:31:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557eeec3dcf0 0x557eeec40100 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557eeebd7cf0 0x557eeebe6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557eeea326b0 0x557eeebe2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5dfd7fa700 0 client.30125 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 --2- 172.21.15.31:0/1775960486 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557eeebd6df0 0x557eeebe3270 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 >> 172.21.15.31:0/1775960486 conn(0x557eeebcbfd0 msgr2=0x557eeebedc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 shutdown_connections 2023-07-29T16:31:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.966+0000 7f5e2fb85b80 1 -- 172.21.15.31:0/1775960486 wait complete. 2023-07-29T16:31:22.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.967+0000 7f5e2fb85b80 20 client.30125 trim_cache size 0 max 16384 2023-07-29T16:31:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:31:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:31:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:31:22.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:31:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.967+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e000a0990 msgr2=0x7f5e000a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.967+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e000a0990 0x7f5e000a2e40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c009510 tx=0x7f5e0c009300 comp rx=0 tx=0).stop 2023-07-29T16:31:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.967+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 msgr2=0x7f5e1419c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.967+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e1419c7a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400e970 tx=0x7f5e0400ed30 comp rx=0 tx=0).stop 2023-07-29T16:31:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.968+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 shutdown_connections 2023-07-29T16:31:22.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.968+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5e000a0990 0x7f5e000a2e40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.968+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e14103f70 0x7f5e1419db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.968+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e140fec50 0x7f5e1419c7a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.968+0000 7f5e1c1ff700 1 --2- 172.21.15.31:0/3362771147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e141015b0 0x7f5e1419cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.968+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 >> 172.21.15.31:0/3362771147 conn(0x7f5e140fa840 msgr2=0x7f5e140ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.969+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 shutdown_connections 2023-07-29T16:31:22.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:22.969+0000 7f5e1c1ff700 1 -- 172.21.15.31:0/3362771147 wait complete. 2023-07-29T16:31:23.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:22 smithi039 ceph-mon[128858]: pgmap v848: 97 pgs: 97 active+clean; 511 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 817 op/s 2023-07-29T16:31:23.072 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:22 smithi156 ceph-mon[129245]: pgmap v848: 97 pgs: 97 active+clean; 511 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 817 op/s 2023-07-29T16:31:23.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39264 34.17 MB/sec warmup 67 sec latency 255.931 ms 2023-07-29T16:31:24.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 39925 34.10 MB/sec warmup 68 sec latency 46.811 ms 2023-07-29T16:31:24.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:24 smithi031 ceph-mon[95504]: pgmap v849: 97 pgs: 97 active+clean; 522 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 21 MiB/s wr, 1.15k op/s 2023-07-29T16:31:25.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:24 smithi039 ceph-mon[128858]: pgmap v849: 97 pgs: 97 active+clean; 522 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 21 MiB/s wr, 1.15k op/s 2023-07-29T16:31:25.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:24 smithi156 ceph-mon[129245]: pgmap v849: 97 pgs: 97 active+clean; 522 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 21 MiB/s wr, 1.15k op/s 2023-07-29T16:31:25.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 40829 34.25 MB/sec warmup 69 sec latency 110.052 ms 2023-07-29T16:31:26.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 41851 34.74 MB/sec warmup 70 sec latency 51.237 ms 2023-07-29T16:31:26.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:26 smithi031 ceph-mon[95504]: pgmap v850: 97 pgs: 97 active+clean; 517 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 23 MiB/s wr, 1.11k op/s 2023-07-29T16:31:27.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:26 smithi039 ceph-mon[128858]: pgmap v850: 97 pgs: 97 active+clean; 517 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 23 MiB/s wr, 1.11k op/s 2023-07-29T16:31:27.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:26 smithi156 ceph-mon[129245]: pgmap v850: 97 pgs: 97 active+clean; 517 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 23 MiB/s wr, 1.11k op/s 2023-07-29T16:31:27.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 42565 34.91 MB/sec warmup 71 sec latency 54.192 ms 2023-07-29T16:31:28.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43212 34.80 MB/sec warmup 72 sec latency 60.940 ms 2023-07-29T16:31:28.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:28 smithi156 ceph-mon[129245]: pgmap v851: 97 pgs: 97 active+clean; 497 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 22 MiB/s wr, 1.08k op/s 2023-07-29T16:31:28.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:28 smithi031 ceph-mon[95504]: pgmap v851: 97 pgs: 97 active+clean; 497 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 22 MiB/s wr, 1.08k op/s 2023-07-29T16:31:28.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:28 smithi039 ceph-mon[128858]: pgmap v851: 97 pgs: 97 active+clean; 497 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 22 MiB/s wr, 1.08k op/s 2023-07-29T16:31:29.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 43983 34.72 MB/sec warmup 73 sec latency 48.320 ms 2023-07-29T16:31:30.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 44837 35.06 MB/sec warmup 74 sec latency 168.982 ms 2023-07-29T16:31:30.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:30 smithi031 ceph-mon[95504]: pgmap v852: 97 pgs: 97 active+clean; 485 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 22 MiB/s wr, 1.23k op/s 2023-07-29T16:31:31.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:30 smithi039 ceph-mon[128858]: pgmap v852: 97 pgs: 97 active+clean; 485 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 22 MiB/s wr, 1.23k op/s 2023-07-29T16:31:31.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:30 smithi156 ceph-mon[129245]: pgmap v852: 97 pgs: 97 active+clean; 485 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 22 MiB/s wr, 1.23k op/s 2023-07-29T16:31:31.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 45712 35.07 MB/sec warmup 75 sec latency 83.078 ms 2023-07-29T16:31:32.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 46543 35.31 MB/sec warmup 76 sec latency 54.085 ms 2023-07-29T16:31:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:32 smithi031 ceph-mon[95504]: pgmap v853: 97 pgs: 97 active+clean; 485 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 864 op/s 2023-07-29T16:31:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:32 smithi039 ceph-mon[128858]: pgmap v853: 97 pgs: 97 active+clean; 485 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 864 op/s 2023-07-29T16:31:33.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:32 smithi156 ceph-mon[129245]: pgmap v853: 97 pgs: 97 active+clean; 485 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 864 op/s 2023-07-29T16:31:33.228 INFO:tasks.workunit.client.0.smithi031.stdout: 10 47304 35.43 MB/sec warmup 77 sec latency 70.188 ms 2023-07-29T16:31:34.229 INFO:tasks.workunit.client.0.smithi031.stdout: 10 48129 35.45 MB/sec warmup 78 sec latency 91.922 ms 2023-07-29T16:31:34.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:34 smithi031 ceph-mon[95504]: pgmap v854: 97 pgs: 97 active+clean; 493 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 15 MiB/s wr, 959 op/s 2023-07-29T16:31:35.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:34 smithi039 ceph-mon[128858]: pgmap v854: 97 pgs: 97 active+clean; 493 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 15 MiB/s wr, 959 op/s 2023-07-29T16:31:35.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:34 smithi156 ceph-mon[129245]: pgmap v854: 97 pgs: 97 active+clean; 493 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 15 MiB/s wr, 959 op/s 2023-07-29T16:31:35.229 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49016 35.72 MB/sec warmup 79 sec latency 82.086 ms 2023-07-29T16:31:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 -- 172.21.15.31:0/873370264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101f00 msgr2=0x7f2a2c1087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 --2- 172.21.15.31:0/873370264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101f00 0x7f2a2c1087c0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009f90 tx=0x7f2a2400b330 comp rx=0 tx=0).stop 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 -- 172.21.15.31:0/873370264 shutdown_connections 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 --2- 172.21.15.31:0/873370264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101f00 0x7f2a2c1087c0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 --2- 172.21.15.31:0/873370264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a2c100380 0x7f2a2c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 --2- 172.21.15.31:0/873370264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a2c101570 0x7f2a2c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 -- 172.21.15.31:0/873370264 >> 172.21.15.31:0/873370264 conn(0x7f2a2c0fb9a0 msgr2=0x7f2a2c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 -- 172.21.15.31:0/873370264 shutdown_connections 2023-07-29T16:31:36.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.209+0000 7f2a30dff700 1 -- 172.21.15.31:0/873370264 wait complete. 2023-07-29T16:31:36.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.211+0000 7f2a30dff700 1 Processor -- start 2023-07-29T16:31:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 -- start start 2023-07-29T16:31:36.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a2c100380 0x7f2a2c102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 0x7f2a2c103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a2c101f00 0x7f2a2c104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c104670 con 0x7f2a2c101f00 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c1047e0 con 0x7f2a2c101570 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a30dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2c1049a0 con 0x7f2a2c100380 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a29d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 0x7f2a2c103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a29d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 0x7f2a2c103140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33564/0 (socket says 172.21.15.31:33564) 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.212+0000 7f2a29d9b700 1 -- 172.21.15.31:0/411563742 learned_addr learned my addr 172.21.15.31:0/411563742 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.213+0000 7f2a29d9b700 1 -- 172.21.15.31:0/411563742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a2c100380 msgr2=0x7f2a2c102c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.213+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a2c100380 0x7f2a2c102c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.213+0000 7f2a29d9b700 1 -- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a2c101f00 msgr2=0x7f2a2c104010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.213+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a2c101f00 0x7f2a2c104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.213+0000 7f2a29d9b700 1 -- 172.21.15.31:0/411563742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24009c40 con 0x7f2a2c101570 2023-07-29T16:31:36.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.214+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 0x7f2a2c103140 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000e9a0 tx=0x7f2a2000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.214+0000 7f2a1b7fe700 1 -- 172.21.15.31:0/411563742 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20009790 con 0x7f2a2c101570 2023-07-29T16:31:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.214+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2c104c80 con 0x7f2a2c101570 2023-07-29T16:31:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.214+0000 7f2a1b7fe700 1 -- 172.21.15.31:0/411563742 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a2000cbd0 con 0x7f2a2c101570 2023-07-29T16:31:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.214+0000 7f2a1b7fe700 1 -- 172.21.15.31:0/411563742 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20018680 con 0x7f2a2c101570 2023-07-29T16:31:36.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.215+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c105430 con 0x7f2a2c101570 2023-07-29T16:31:36.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.219+0000 7f2a1b7fe700 1 -- 172.21.15.31:0/411563742 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2a2000cd40 con 0x7f2a2c101570 2023-07-29T16:31:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.221+0000 7f2a1b7fe700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a140a0890 0x7f2a140a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.221+0000 7f2a1b7fe700 1 -- 172.21.15.31:0/411563742 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2a20014070 con 0x7f2a2c101570 2023-07-29T16:31:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.222+0000 7f2a447e4b80 1 Processor -- start 2023-07-29T16:31:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a140a0890 0x7f2a140a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 -- start start 2023-07-29T16:31:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb737b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 0x55edefb74160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefaca9b0 con 0x55edef9ce6b0 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefac8370 con 0x55edefb73cf0 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.223+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefac3c60 con 0x55edefb72df0 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 0x55edefb74160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 0x55edefb74160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33574/0 (socket says 172.21.15.31:33574) 2023-07-29T16:31:36.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/2984397522 learned_addr learned my addr 172.21.15.31:0/2984397522 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/2984397522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 msgr2=0x55edefb737b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/2984397522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 msgr2=0x55edefb728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/2984397522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edef8f1d80 con 0x55edefb73cf0 2023-07-29T16:31:36.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:36.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 0x55edefb74160 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009350 tx=0x7f2a24009380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5e8aa0d245cd8d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.224+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a140a0890 0x7f2a140a2d40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c102970 tx=0x7f2a1c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.225+0000 7f2a18ff9700 1 -- 172.21.15.31:0/2984397522 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400e040 con 0x55edefb73cf0 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.225+0000 7f2a18ff9700 1 -- 172.21.15.31:0/2984397522 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a24007620 con 0x55edefb73cf0 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.225+0000 7f2a18ff9700 1 -- 172.21.15.31:0/2984397522 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400bdd0 con 0x55edefb73cf0 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 -- 172.21.15.31:0/2984397522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 msgr2=0x55edefb74160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 0x55edefb74160 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009350 tx=0x7f2a24009380 comp rx=0 tx=0).stop 2023-07-29T16:31:36.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 -- 172.21.15.31:0/2984397522 shutdown_connections 2023-07-29T16:31:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb73cf0 0x55edefb74160 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/2984397522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb737b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 -- 172.21.15.31:0/2984397522 >> 172.21.15.31:0/2984397522 conn(0x55edefb67fd0 msgr2=0x55edefb6d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 -- 172.21.15.31:0/2984397522 shutdown_connections 2023-07-29T16:31:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 -- 172.21.15.31:0/2984397522 wait complete. 2023-07-29T16:31:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.226+0000 7f2a447e4b80 1 -- start start 2023-07-29T16:31:36.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.227+0000 7f2a447e4b80 1 Processor -- start 2023-07-29T16:31:36.231 INFO:tasks.workunit.client.0.smithi031.stdout: 10 49857 35.68 MB/sec warmup 80 sec latency 48.489 ms 2023-07-29T16:31:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb72df0 0x55edefb7c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb834d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefa604b0 con 0x55edefb72df0 2023-07-29T16:31:36.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefa60150 con 0x55edef9ce6b0 2023-07-29T16:31:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edef9478c0 con 0x55edefb73cf0 2023-07-29T16:31:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.228+0000 7f2a447e4b80 10 client.0 _ll_register_callbacks cb 0x55edefa695c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33582/0 (socket says 172.21.15.31:33582) 2023-07-29T16:31:36.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2a59c700 1 -- 172.21.15.31:0/3415345658 learned_addr learned my addr 172.21.15.31:0/3415345658 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb834d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 msgr2=0x55edefb7bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/3415345658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb72df0 msgr2=0x55edefb7c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb72df0 0x55edefb7c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.229+0000 7f2a2ad9d700 1 -- 172.21.15.31:0/3415345658 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2c1090f0 con 0x55edefb73cf0 2023-07-29T16:31:36.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb834d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009320 tx=0x7f2a2400bd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91e0bbe42b45995 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a11ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3415345658 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2400e040 con 0x55edefb73cf0 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55edefaeecb0 con 0x55edefb73cf0 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3415345658 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2a24007d10 con 0x55edefb73cf0 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3415345658 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a240050c0 con 0x55edefb73cf0 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a447e4b80 10 client.30157 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:36.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.230+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55edef8dac00 con 0x55edefb73cf0 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3415345658 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55edefaeecb0 con 0x55edefb73cf0 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3415345658 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2a24029030 con 0x55edefb73cf0 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a447e4b80 10 client.30157 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a447e4b80 10 client.30157 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a447e4b80 20 client.30157 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefb98840 0x55edefb9ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a447e4b80 4 client.30157 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:36.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.231+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55edefb41d10 con 0x55edefb98840 2023-07-29T16:31:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.232+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefb98840 0x55edefb9ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.233+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefb98840 0x55edefb9ac50 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97544dec6240f2a2 server_cookie=2a407053301b9fa1 in_seq=0 out_seq=0 2023-07-29T16:31:36.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.233+0000 7f2a11ffb700 10 client.30157 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.238+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3415345658 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55edefb41d10 con 0x55edefb98840 2023-07-29T16:31:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.238+0000 7f2a11ffb700 10 client.30157 handle_command_reply: tid=0 2023-07-29T16:31:36.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.238+0000 7f2a447e4b80 1 client.30157 shutdown 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.238+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 msgr2=0x55edefb834d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.238+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb834d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009320 tx=0x7f2a2400bd40 comp rx=0 tx=0).stop 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.238+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 shutdown_connections 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefb98840 0x55edefb9ac50 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb834d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3415345658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb72df0 0x55edefb7c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a11ffb700 0 client.30157 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 >> 172.21.15.31:0/3415345658 conn(0x55edefb67fd0 msgr2=0x55edefb6ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:36.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 shutdown_connections 2023-07-29T16:31:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3415345658 wait complete. 2023-07-29T16:31:36.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.239+0000 7f2a447e4b80 20 client.30157 trim_cache size 0 max 16384 2023-07-29T16:31:36.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.266+0000 7f2a447e4b80 1 Processor -- start 2023-07-29T16:31:36.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 -- start start 2023-07-29T16:31:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb83490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb73cf0 0x55edefb6ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefaeecb0 con 0x55edefb73cf0 2023-07-29T16:31:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefb7d140 con 0x55edef9ce6b0 2023-07-29T16:31:36.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefb6e5f0 con 0x55edefb72df0 2023-07-29T16:31:36.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a2a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a2a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33592/0 (socket says 172.21.15.31:33592) 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.269+0000 7f2a2a59c700 1 -- 172.21.15.31:0/67131439 learned_addr learned my addr 172.21.15.31:0/67131439 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2ad9d700 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb73cf0 0x55edefb6ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2a59c700 1 -- 172.21.15.31:0/67131439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 msgr2=0x55edefb83490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb83490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2a59c700 1 -- 172.21.15.31:0/67131439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb73cf0 msgr2=0x55edefb6ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:36.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb73cf0 0x55edefb6ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2a59c700 1 -- 172.21.15.31:0/67131439 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55edef9181f0 con 0x55edef9ce6b0 2023-07-29T16:31:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7abb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x55edefb9b4f0 tx=0x7f2a1c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78fdcf24da5f940c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.270+0000 7f2a197fa700 1 -- 172.21.15.31:0/67131439 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c021040 con 0x55edef9ce6b0 2023-07-29T16:31:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.271+0000 7f2a197fa700 1 -- 172.21.15.31:0/67131439 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a1c00ee80 con 0x55edef9ce6b0 2023-07-29T16:31:36.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.271+0000 7f2a197fa700 1 -- 172.21.15.31:0/67131439 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c01cea0 con 0x55edef9ce6b0 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 -- 172.21.15.31:0/67131439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 msgr2=0x55edefb7abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7abb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x55edefb9b4f0 tx=0x7f2a1c016000 comp rx=0 tx=0).stop 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 -- 172.21.15.31:0/67131439 shutdown_connections 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edefb73cf0 0x55edefb6ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edef9ce6b0 0x55edefb7abb0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/67131439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb72df0 0x55edefb83490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.272+0000 7f2a447e4b80 1 -- 172.21.15.31:0/67131439 >> 172.21.15.31:0/67131439 conn(0x55edefb67fd0 msgr2=0x55edefb694a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:36.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.275+0000 7f2a447e4b80 1 -- 172.21.15.31:0/67131439 shutdown_connections 2023-07-29T16:31:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.277+0000 7f2a447e4b80 1 -- 172.21.15.31:0/67131439 wait complete. 2023-07-29T16:31:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.277+0000 7f2a447e4b80 1 -- start start 2023-07-29T16:31:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.277+0000 7f2a447e4b80 1 Processor -- start 2023-07-29T16:31:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb7f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 0x55edefb7f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb82910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefb82e50 con 0x55edef9ce6b0 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefb82fc0 con 0x55edefb72df0 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55edefb83180 con 0x55edefb73cf0 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a447e4b80 10 client.0 _ll_register_callbacks cb 0x55edefa695c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.278+0000 7f2a29d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 0x55edefb7f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 0x55edefb7f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33594/0 (socket says 172.21.15.31:33594) 2023-07-29T16:31:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 -- 172.21.15.31:0/3805646034 learned_addr learned my addr 172.21.15.31:0/3805646034 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 -- 172.21.15.31:0/3805646034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 msgr2=0x55edefb82910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:31:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb82910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 -- 172.21.15.31:0/3805646034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 msgr2=0x55edefb7f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb7f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 -- 172.21.15.31:0/3805646034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c00ebb0 con 0x55edefb72df0 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a29d9b700 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 0x55edefb7f8f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a200a14e0 tx=0x7f2a20099230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2dc7e33c8547fd0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a11ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.279+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3805646034 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a200995c0 con 0x55edefb72df0 2023-07-29T16:31:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.280+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3805646034 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2a2009d5f0 con 0x55edefb72df0 2023-07-29T16:31:36.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.280+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a24007620 con 0x55edefb72df0 2023-07-29T16:31:36.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.280+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3805646034 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a200a1a10 con 0x55edefb72df0 2023-07-29T16:31:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.280+0000 7f2a447e4b80 10 client.30143 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.280+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55edef8f0250 con 0x55edefb72df0 2023-07-29T16:31:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.281+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3805646034 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a24007620 con 0x55edefb72df0 2023-07-29T16:31:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3805646034 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2a20098520 con 0x55edefb72df0 2023-07-29T16:31:36.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a447e4b80 10 client.30143 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a447e4b80 10 client.30143 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a447e4b80 20 client.30143 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefbeb580 0x55edefbed990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a447e4b80 4 client.30143 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.282+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55edefabcce0 con 0x55edefbeb580 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.283+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefbeb580 0x55edefbed990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:36.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.283+0000 7f2a2a59c700 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefbeb580 0x55edefbed990 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96fb9c15659c8521 server_cookie=8109ca1d4f4351b in_seq=0 out_seq=0 2023-07-29T16:31:36.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.283+0000 7f2a11ffb700 10 client.30143 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:36.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.603+0000 7f2a11ffb700 1 -- 172.21.15.31:0/3805646034 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55edefabcce0 con 0x55edefbeb580 2023-07-29T16:31:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.603+0000 7f2a11ffb700 10 client.30143 handle_command_reply: tid=0 2023-07-29T16:31:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.606+0000 7f2a447e4b80 1 client.30143 shutdown 2023-07-29T16:31:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 msgr2=0x55edefb7f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 0x55edefb7f8f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f2a200a14e0 tx=0x7f2a20099230 comp rx=0 tx=0).stop 2023-07-29T16:31:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 shutdown_connections 2023-07-29T16:31:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55edefbeb580 0x55edefbed990 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edefb73cf0 0x55edefb82910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55edef9ce6b0 0x55edefb7f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 --2- 172.21.15.31:0/3805646034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55edefb72df0 0x55edefb7f8f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 >> 172.21.15.31:0/3805646034 conn(0x55edefb67fd0 msgr2=0x55edefb7abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.607+0000 7f2a11ffb700 0 client.30143 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:36.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.608+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 shutdown_connections 2023-07-29T16:31:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.609+0000 7f2a447e4b80 1 -- 172.21.15.31:0/3805646034 wait complete. 2023-07-29T16:31:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.610+0000 7f2a447e4b80 20 client.30143 trim_cache size 0 max 16384 2023-07-29T16:31:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:31:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:31:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:31:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:31:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a140a0890 msgr2=0x7f2a140a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a140a0890 0x7f2a140a2d40 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c102970 tx=0x7f2a1c009310 comp rx=0 tx=0).stop 2023-07-29T16:31:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 msgr2=0x7f2a2c103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 0x7f2a2c103140 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f2a2000e9a0 tx=0x7f2a2000ed60 comp rx=0 tx=0).stop 2023-07-29T16:31:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 shutdown_connections 2023-07-29T16:31:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f2a140a0890 0x7f2a140a2d40 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a2c101f00 0x7f2a2c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a2c100380 0x7f2a2c102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 --2- 172.21.15.31:0/411563742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a2c101570 0x7f2a2c103140 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 >> 172.21.15.31:0/411563742 conn(0x7f2a2c0fb9a0 msgr2=0x7f2a2c106d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:36.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 shutdown_connections 2023-07-29T16:31:36.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:36.611+0000 7f2a30dff700 1 -- 172.21.15.31:0/411563742 wait complete. 2023-07-29T16:31:36.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:36 smithi031 ceph-mon[95504]: pgmap v855: 97 pgs: 97 active+clean; 503 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.7 MiB/s wr, 745 op/s 2023-07-29T16:31:37.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:36 smithi039 ceph-mon[128858]: pgmap v855: 97 pgs: 97 active+clean; 503 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.7 MiB/s wr, 745 op/s 2023-07-29T16:31:37.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:36 smithi156 ceph-mon[129245]: pgmap v855: 97 pgs: 97 active+clean; 503 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.7 MiB/s wr, 745 op/s 2023-07-29T16:31:37.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 50640 35.86 MB/sec warmup 81 sec latency 153.412 ms 2023-07-29T16:31:38.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 51551 36.07 MB/sec warmup 82 sec latency 41.717 ms 2023-07-29T16:31:38.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:38 smithi156 ceph-mon[129245]: pgmap v856: 97 pgs: 97 active+clean; 502 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.1 MiB/s wr, 560 op/s 2023-07-29T16:31:38.434 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:38 smithi031 ceph-mon[95504]: pgmap v856: 97 pgs: 97 active+clean; 502 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.1 MiB/s wr, 560 op/s 2023-07-29T16:31:38.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:38 smithi039 ceph-mon[128858]: pgmap v856: 97 pgs: 97 active+clean; 502 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.1 MiB/s wr, 560 op/s 2023-07-29T16:31:39.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 52291 36.07 MB/sec warmup 83 sec latency 134.493 ms 2023-07-29T16:31:40.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53034 36.14 MB/sec warmup 84 sec latency 91.636 ms 2023-07-29T16:31:40.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:40 smithi031 ceph-mon[95504]: pgmap v857: 97 pgs: 97 active+clean; 510 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.3 MiB/s wr, 740 op/s 2023-07-29T16:31:41.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:40 smithi039 ceph-mon[128858]: pgmap v857: 97 pgs: 97 active+clean; 510 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.3 MiB/s wr, 740 op/s 2023-07-29T16:31:41.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:40 smithi156 ceph-mon[129245]: pgmap v857: 97 pgs: 97 active+clean; 510 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.3 MiB/s wr, 740 op/s 2023-07-29T16:31:41.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 53692 36.11 MB/sec warmup 85 sec latency 68.332 ms 2023-07-29T16:31:42.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54071 35.89 MB/sec warmup 86 sec latency 280.115 ms 2023-07-29T16:31:42.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:42 smithi031 ceph-mon[95504]: pgmap v858: 97 pgs: 97 active+clean; 510 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 553 op/s 2023-07-29T16:31:43.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:42 smithi039 ceph-mon[128858]: pgmap v858: 97 pgs: 97 active+clean; 510 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 553 op/s 2023-07-29T16:31:43.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:42 smithi156 ceph-mon[129245]: pgmap v858: 97 pgs: 97 active+clean; 510 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 553 op/s 2023-07-29T16:31:43.230 INFO:tasks.workunit.client.0.smithi031.stdout: 10 54906 35.94 MB/sec warmup 87 sec latency 110.653 ms 2023-07-29T16:31:44.234 INFO:tasks.workunit.client.0.smithi031.stdout: 10 55823 36.21 MB/sec warmup 88 sec latency 80.588 ms 2023-07-29T16:31:44.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:44 smithi031 ceph-mon[95504]: pgmap v859: 97 pgs: 97 active+clean; 522 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 5.8 MiB/s wr, 710 op/s 2023-07-29T16:31:45.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:44 smithi039 ceph-mon[128858]: pgmap v859: 97 pgs: 97 active+clean; 522 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 5.8 MiB/s wr, 710 op/s 2023-07-29T16:31:45.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:44 smithi156 ceph-mon[129245]: pgmap v859: 97 pgs: 97 active+clean; 522 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 5.8 MiB/s wr, 710 op/s 2023-07-29T16:31:45.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56259 36.11 MB/sec warmup 89 sec latency 321.666 ms 2023-07-29T16:31:46.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 56799 35.99 MB/sec warmup 90 sec latency 96.451 ms 2023-07-29T16:31:46.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:46 smithi031 ceph-mon[95504]: pgmap v860: 97 pgs: 97 active+clean; 524 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.7 MiB/s wr, 691 op/s 2023-07-29T16:31:47.031 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:46 smithi039 ceph-mon[128858]: pgmap v860: 97 pgs: 97 active+clean; 524 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.7 MiB/s wr, 691 op/s 2023-07-29T16:31:47.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:46 smithi156 ceph-mon[129245]: pgmap v860: 97 pgs: 97 active+clean; 524 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.7 MiB/s wr, 691 op/s 2023-07-29T16:31:47.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57236 35.75 MB/sec warmup 91 sec latency 364.221 ms 2023-07-29T16:31:48.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57299 35.39 MB/sec warmup 92 sec latency 994.676 ms 2023-07-29T16:31:48.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:48 smithi031 ceph-mon[95504]: pgmap v861: 97 pgs: 97 active+clean; 539 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 7.8 MiB/s wr, 674 op/s 2023-07-29T16:31:48.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:48 smithi039 ceph-mon[128858]: pgmap v861: 97 pgs: 97 active+clean; 539 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 7.8 MiB/s wr, 674 op/s 2023-07-29T16:31:48.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:48 smithi156 ceph-mon[129245]: pgmap v861: 97 pgs: 97 active+clean; 539 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 7.8 MiB/s wr, 674 op/s 2023-07-29T16:31:49.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 57978 35.47 MB/sec warmup 93 sec latency 179.473 ms 2023-07-29T16:31:49.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 -- 172.21.15.31:0/3736155910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06bf50 msgr2=0x7f541c06c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 --2- 172.21.15.31:0/3736155910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06bf50 0x7f541c06c3c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f540c009990 tx=0x7f540c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 -- 172.21.15.31:0/3736155910 shutdown_connections 2023-07-29T16:31:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 --2- 172.21.15.31:0/3736155910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 0x7f541c0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 --2- 172.21.15.31:0/3736155910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06ac70 0x7f541c06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 --2- 172.21.15.31:0/3736155910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06bf50 0x7f541c06c3c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.856+0000 7f54241ff700 1 -- 172.21.15.31:0/3736155910 >> 172.21.15.31:0/3736155910 conn(0x7f541c0fa840 msgr2=0x7f541c0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.857+0000 7f54241ff700 1 -- 172.21.15.31:0/3736155910 shutdown_connections 2023-07-29T16:31:49.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.857+0000 7f54241ff700 1 -- 172.21.15.31:0/3736155910 wait complete. 2023-07-29T16:31:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.858+0000 7f54241ff700 1 Processor -- start 2023-07-29T16:31:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 -- start start 2023-07-29T16:31:49.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 0x7f541c116bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06bf50 0x7f541c117110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 0x7f541c117fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c118640 con 0x7f541c06c900 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c1187b0 con 0x7f541c06ac70 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.859+0000 7f54241ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c118970 con 0x7f541c06bf50 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 0x7f541c116bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f542179a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06bf50 0x7f541c117110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 0x7f541c116bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60300/0 (socket says 172.21.15.31:60300) 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 -- 172.21.15.31:0/965747544 learned_addr learned my addr 172.21.15.31:0/965747544 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f542279c700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 0x7f541c117fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 -- 172.21.15.31:0/965747544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06bf50 msgr2=0x7f541c117110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06bf50 0x7f541c117110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 -- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 msgr2=0x7f541c117fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:49.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 0x7f541c117fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 -- 172.21.15.31:0/965747544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0095f0 con 0x7f541c06ac70 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f542179a700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06bf50 0x7f541c117110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f542279c700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 0x7f541c117fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.860+0000 7f5421f9b700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 0x7f541c116bd0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f541800da70 tx=0x7f541800dd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.861+0000 7f5412ffd700 1 -- 172.21.15.31:0/965747544 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541800c9d0 con 0x7f541c06ac70 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.861+0000 7f5412ffd700 1 -- 172.21.15.31:0/965747544 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5418004020 con 0x7f541c06ac70 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.861+0000 7f5412ffd700 1 -- 172.21.15.31:0/965747544 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5418003d50 con 0x7f541c06ac70 2023-07-29T16:31:49.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.861+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c118c50 con 0x7f541c06ac70 2023-07-29T16:31:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.861+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c11f430 con 0x7f541c06ac70 2023-07-29T16:31:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.868+0000 7f5437b15b80 1 Processor -- start 2023-07-29T16:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 -- start start 2023-07-29T16:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d377b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d38160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2c8e9b0 con 0x55d3a2d36df0 2023-07-29T16:31:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2c8c370 con 0x55d3a2d37cf0 2023-07-29T16:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f5437b15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2c87c60 con 0x55d3a2b926b0 2023-07-29T16:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f542279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f542279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d368b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59404/0 (socket says 172.21.15.31:59404) 2023-07-29T16:31:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f542279c700 1 -- 172.21.15.31:0/2954813135 learned_addr learned my addr 172.21.15.31:0/2954813135 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.871+0000 7f542179a700 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d38160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f5412ffd700 1 -- 172.21.15.31:0/965747544 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5418004190 con 0x7f541c06ac70 2023-07-29T16:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f5421f9b700 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f542279c700 1 -- 172.21.15.31:0/2954813135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 msgr2=0x55d3a2d38160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f542279c700 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d38160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f542279c700 1 -- 172.21.15.31:0/2954813135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 msgr2=0x55d3a2d377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f542279c700 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f542279c700 1 -- 172.21.15.31:0/2954813135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3a2ab5d80 con 0x55d3a2b926b0 2023-07-29T16:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.872+0000 7f5421f9b700 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:31:49.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.873+0000 7f542279c700 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d368b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5414009c40 tx=0x7f5414009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f8f2539bf6f8deb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:49.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.874+0000 7f5407fff700 1 -- 172.21.15.31:0/2954813135 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541403a070 con 0x55d3a2b926b0 2023-07-29T16:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.874+0000 7f5407fff700 1 -- 172.21.15.31:0/2954813135 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5414007d20 con 0x55d3a2b926b0 2023-07-29T16:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.874+0000 7f5407fff700 1 -- 172.21.15.31:0/2954813135 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140353c0 con 0x55d3a2b926b0 2023-07-29T16:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.875+0000 7f5412ffd700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54080a08e0 0x7f54080a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.875+0000 7f5412ffd700 1 -- 172.21.15.31:0/965747544 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f54180cb030 con 0x7f541c06ac70 2023-07-29T16:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.876+0000 7f5437b15b80 1 -- 172.21.15.31:0/2954813135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 msgr2=0x55d3a2d368b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:49.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.876+0000 7f5437b15b80 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d368b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f5414009c40 tx=0x7f5414009f50 comp rx=0 tx=0).stop 2023-07-29T16:31:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.881+0000 7f5421f9b700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54080a08e0 0x7f54080a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.881+0000 7f5437b15b80 1 -- 172.21.15.31:0/2954813135 shutdown_connections 2023-07-29T16:31:49.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.881+0000 7f5437b15b80 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d38160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.881+0000 7f5437b15b80 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d368b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.881+0000 7f5437b15b80 1 --2- 172.21.15.31:0/2954813135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.881+0000 7f5437b15b80 1 -- 172.21.15.31:0/2954813135 >> 172.21.15.31:0/2954813135 conn(0x55d3a2d2bfd0 msgr2=0x55d3a2d31300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.883+0000 7f5437b15b80 1 -- 172.21.15.31:0/2954813135 shutdown_connections 2023-07-29T16:31:49.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.883+0000 7f5437b15b80 1 -- 172.21.15.31:0/2954813135 wait complete. 2023-07-29T16:31:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.884+0000 7f5437b15b80 1 -- start start 2023-07-29T16:31:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 Processor -- start 2023-07-29T16:31:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d3ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d40450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d47490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2c244b0 con 0x55d3a2d36df0 2023-07-29T16:31:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2c24150 con 0x55d3a2d37cf0 2023-07-29T16:31:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2b0b8c0 con 0x55d3a2b926b0 2023-07-29T16:31:49.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.885+0000 7f5437b15b80 10 client.0 _ll_register_callbacks cb 0x55d3a2c2d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.886+0000 7f542279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d3ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.886+0000 7f542279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d3ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59416/0 (socket says 172.21.15.31:59416) 2023-07-29T16:31:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.886+0000 7f542279c700 1 -- 172.21.15.31:0/685727403 learned_addr learned my addr 172.21.15.31:0/685727403 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:49.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.886+0000 7f542279c700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d47490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.887+0000 7f542179a700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d40450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.887+0000 7f542279c700 1 -- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 msgr2=0x55d3a2d47490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.887+0000 7f542279c700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d47490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.887+0000 7f542279c700 1 -- 172.21.15.31:0/685727403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 msgr2=0x55d3a2d40450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.887+0000 7f542279c700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d40450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.887+0000 7f542279c700 1 -- 172.21.15.31:0/685727403 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5414032040 con 0x55d3a2b926b0 2023-07-29T16:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.888+0000 7f542279c700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d3ff10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5414006130 tx=0x7f5414009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2b8ddb882e8e978 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.888+0000 7f54057fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.888+0000 7f54057fa700 1 -- 172.21.15.31:0/685727403 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541403a040 con 0x55d3a2b926b0 2023-07-29T16:31:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.888+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3a2cb2cb0 con 0x55d3a2b926b0 2023-07-29T16:31:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.888+0000 7f54057fa700 1 -- 172.21.15.31:0/685727403 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f541403f7c0 con 0x55d3a2b926b0 2023-07-29T16:31:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.888+0000 7f54057fa700 1 -- 172.21.15.31:0/685727403 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f541403f980 con 0x55d3a2b926b0 2023-07-29T16:31:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 10 client.30169 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55d3a2a9ec00 con 0x55d3a2b926b0 2023-07-29T16:31:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5421f9b700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54080a08e0 0x7f54080a2d90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f54180cb2d0 tx=0x7f541800df60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f54057fa700 1 -- 172.21.15.31:0/685727403 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5414053020 con 0x55d3a2b926b0 2023-07-29T16:31:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f54057fa700 1 -- 172.21.15.31:0/685727403 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5414043a60 con 0x55d3a2b926b0 2023-07-29T16:31:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 10 client.30169 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 10 client.30169 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 20 client.30169 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d4bbb0 0x55d3a2d4dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 4 client.30169 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:31:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.889+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3a2d05d10 con 0x55d3a2d4bbb0 2023-07-29T16:31:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.890+0000 7f542179a700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d4bbb0 0x55d3a2d4dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.891+0000 7f542179a700 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d4bbb0 0x55d3a2d4dfc0 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be3633fcb8484e6f server_cookie=fd37a567abbdddf0 in_seq=0 out_seq=0 2023-07-29T16:31:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.891+0000 7f54057fa700 10 client.30169 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.897+0000 7f54057fa700 1 -- 172.21.15.31:0/685727403 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d3a2d05d10 con 0x55d3a2d4bbb0 2023-07-29T16:31:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.897+0000 7f54057fa700 10 client.30169 handle_command_reply: tid=0 2023-07-29T16:31:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.897+0000 7f5437b15b80 1 client.30169 shutdown 2023-07-29T16:31:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 msgr2=0x55d3a2d3ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d3ff10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5414006130 tx=0x7f5414009040 comp rx=0 tx=0).stop 2023-07-29T16:31:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 shutdown_connections 2023-07-29T16:31:49.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d4bbb0 0x55d3a2d4dfc0 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d47490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d3ff10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f54057fa700 0 client.30169 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 --2- 172.21.15.31:0/685727403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d40450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 >> 172.21.15.31:0/685727403 conn(0x55d3a2d2bfd0 msgr2=0x55d3a2d30a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:49.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 shutdown_connections 2023-07-29T16:31:49.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.898+0000 7f5437b15b80 1 -- 172.21.15.31:0/685727403 wait complete. 2023-07-29T16:31:49.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.899+0000 7f5437b15b80 20 client.30169 trim_cache size 0 max 16384 2023-07-29T16:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 Processor -- start 2023-07-29T16:31:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 -- start start 2023-07-29T16:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 0x55d3a2d45440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d380c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d3e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2d38600 con 0x55d3a2d36df0 2023-07-29T16:31:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2d31fc0 con 0x55d3a2b926b0 2023-07-29T16:31:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.924+0000 7f5437b15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2d32df0 con 0x55d3a2d37cf0 2023-07-29T16:31:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 0x55d3a2d45440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 0x55d3a2d45440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60316/0 (socket says 172.21.15.31:60316) 2023-07-29T16:31:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 -- 172.21.15.31:0/3705966593 learned_addr learned my addr 172.21.15.31:0/3705966593 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d3e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 -- 172.21.15.31:0/3705966593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2d37cf0 msgr2=0x55d3a2d3e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d3e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 -- 172.21.15.31:0/3705966593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 msgr2=0x55d3a2d380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.925+0000 7f542279c700 1 -- 172.21.15.31:0/3705966593 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3a2adc1f0 con 0x55d3a2b926b0 2023-07-29T16:31:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.926+0000 7f542279c700 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 0x55d3a2d45440 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f54140095b0 tx=0x7f5414009540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1ce0d3b36790d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.927+0000 7f5410ff9700 1 -- 172.21.15.31:0/3705966593 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541403a070 con 0x55d3a2b926b0 2023-07-29T16:31:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.927+0000 7f5410ff9700 1 -- 172.21.15.31:0/3705966593 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5414053070 con 0x55d3a2b926b0 2023-07-29T16:31:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.927+0000 7f5410ff9700 1 -- 172.21.15.31:0/3705966593 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54140418c0 con 0x55d3a2b926b0 2023-07-29T16:31:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 -- 172.21.15.31:0/3705966593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 msgr2=0x55d3a2d45440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 0x55d3a2d45440 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f54140095b0 tx=0x7f5414009540 comp rx=0 tx=0).stop 2023-07-29T16:31:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 -- 172.21.15.31:0/3705966593 shutdown_connections 2023-07-29T16:31:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d3e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2b926b0 0x55d3a2d45440 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 --2- 172.21.15.31:0/3705966593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d380c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.928+0000 7f5437b15b80 1 -- 172.21.15.31:0/3705966593 >> 172.21.15.31:0/3705966593 conn(0x55d3a2d2bfd0 msgr2=0x55d3a2d3ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:49.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.929+0000 7f5437b15b80 1 -- 172.21.15.31:0/3705966593 shutdown_connections 2023-07-29T16:31:49.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.929+0000 7f5437b15b80 1 -- 172.21.15.31:0/3705966593 wait complete. 2023-07-29T16:31:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.929+0000 7f5437b15b80 1 -- start start 2023-07-29T16:31:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.929+0000 7f5437b15b80 1 Processor -- start 2023-07-29T16:31:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d44c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d45140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d43430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2d43970 con 0x55d3a2d36df0 2023-07-29T16:31:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2d43ae0 con 0x55d3a2d37cf0 2023-07-29T16:31:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a2d43ca0 con 0x55d3a2b926b0 2023-07-29T16:31:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f5437b15b80 10 client.0 _ll_register_callbacks cb 0x55d3a2c2d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:31:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f542279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d44c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f542279c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d44c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59430/0 (socket says 172.21.15.31:59430) 2023-07-29T16:31:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.930+0000 7f542279c700 1 -- 172.21.15.31:0/30414033 learned_addr learned my addr 172.21.15.31:0/30414033 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:31:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.931+0000 7f542279c700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d43430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.932+0000 7f542179a700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d45140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.932+0000 7f542279c700 1 -- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 msgr2=0x55d3a2d43430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.932+0000 7f542279c700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d43430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.932+0000 7f542279c700 1 -- 172.21.15.31:0/30414033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 msgr2=0x55d3a2d45140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.932+0000 7f542279c700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d45140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.932+0000 7f542279c700 1 -- 172.21.15.31:0/30414033 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3a2a9ec00 con 0x55d3a2b926b0 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f542279c700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d44c00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5414036630 tx=0x7f5414004ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2567edcafee34f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f54057fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:31:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f54057fa700 1 -- 172.21.15.31:0/30414033 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541403a070 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5414007d20 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f54057fa700 1 -- 172.21.15.31:0/30414033 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5414054080 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f54057fa700 1 -- 172.21.15.31:0/30414033 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5414043930 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.933+0000 7f5437b15b80 10 client.30175 fetch_fsmap learned FSMap version 39 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55d3a2ab4250 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f54057fa700 1 -- 172.21.15.31:0/30414033 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5414007d20 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f54057fa700 1 -- 172.21.15.31:0/30414033 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f541403f820 con 0x55d3a2b926b0 2023-07-29T16:31:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 10 client.30175 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:31:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 10 client.30175 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:31:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 20 client.30175 populate_metadata read hostname 'smithi031' 2023-07-29T16:31:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d9e360 0x55d3a2da0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:31:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 4 client.30175 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:31:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.934+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d3a2c80ce0 con 0x55d3a2d9e360 2023-07-29T16:31:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.935+0000 7f542179a700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d9e360 0x55d3a2da0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:31:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.936+0000 7f542179a700 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d9e360 0x55d3a2da0770 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743acc922fafd951 server_cookie=a460b24cff33a70f in_seq=0 out_seq=0 2023-07-29T16:31:49.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:49.936+0000 7f54057fa700 10 client.30175 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:50.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.164+0000 7f54057fa700 1 -- 172.21.15.31:0/30414033 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d3a2c80ce0 con 0x55d3a2d9e360 2023-07-29T16:31:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.164+0000 7f54057fa700 10 client.30175 handle_command_reply: tid=0 2023-07-29T16:31:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.164+0000 7f5437b15b80 1 client.30175 shutdown 2023-07-29T16:31:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.164+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 msgr2=0x55d3a2d44c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.164+0000 7f5437b15b80 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d44c00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5414036630 tx=0x7f5414004ca0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 shutdown_connections 2023-07-29T16:31:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f5437b15b80 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d3a2d9e360 0x55d3a2da0770 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f5437b15b80 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d3a2d37cf0 0x55d3a2d43430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f5437b15b80 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d3a2b926b0 0x55d3a2d44c00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f5437b15b80 1 --2- 172.21.15.31:0/30414033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3a2d36df0 0x55d3a2d45140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 >> 172.21.15.31:0/30414033 conn(0x55d3a2d2bfd0 msgr2=0x55d3a2d3e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.169+0000 7f54057fa700 0 client.30175 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:31:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.170+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 shutdown_connections 2023-07-29T16:31:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.170+0000 7f5437b15b80 1 -- 172.21.15.31:0/30414033 wait complete. 2023-07-29T16:31:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.171+0000 7f5437b15b80 20 client.30175 trim_cache size 0 max 16384 2023-07-29T16:31:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:31:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:31:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:31:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:31:50.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54080a08e0 msgr2=0x7f54080a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54080a08e0 0x7f54080a2d90 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f54180cb2d0 tx=0x7f541800df60 comp rx=0 tx=0).stop 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 msgr2=0x7f541c116bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 0x7f541c116bd0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f541800da70 tx=0x7f541800dd80 comp rx=0 tx=0).stop 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 shutdown_connections 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f54080a08e0 0x7f54080a2d90 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f541c06c900 0x7f541c117fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f541c06ac70 0x7f541c116bd0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 --2- 172.21.15.31:0/965747544 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f541c06bf50 0x7f541c117110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:31:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.172+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 >> 172.21.15.31:0/965747544 conn(0x7f541c0fa840 msgr2=0x7f541c06f400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:31:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.173+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 shutdown_connections 2023-07-29T16:31:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:31:50.173+0000 7f54241ff700 1 -- 172.21.15.31:0/965747544 wait complete. 2023-07-29T16:31:50.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 58527 35.34 MB/sec warmup 94 sec latency 117.702 ms 2023-07-29T16:31:50.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:50 smithi031 ceph-mon[95504]: pgmap v862: 97 pgs: 97 active+clean; 628 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 17 MiB/s wr, 914 op/s 2023-07-29T16:31:51.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:50 smithi039 ceph-mon[128858]: pgmap v862: 97 pgs: 97 active+clean; 628 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 17 MiB/s wr, 914 op/s 2023-07-29T16:31:51.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:50 smithi156 ceph-mon[129245]: pgmap v862: 97 pgs: 97 active+clean; 628 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 17 MiB/s wr, 914 op/s 2023-07-29T16:31:51.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 59351 35.48 MB/sec warmup 95 sec latency 55.769 ms 2023-07-29T16:31:52.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60137 35.55 MB/sec warmup 96 sec latency 81.195 ms 2023-07-29T16:31:52.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:52 smithi031 ceph-mon[95504]: pgmap v863: 97 pgs: 97 active+clean; 628 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 15 MiB/s wr, 620 op/s 2023-07-29T16:31:53.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:52 smithi039 ceph-mon[128858]: pgmap v863: 97 pgs: 97 active+clean; 628 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 15 MiB/s wr, 620 op/s 2023-07-29T16:31:53.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:52 smithi156 ceph-mon[129245]: pgmap v863: 97 pgs: 97 active+clean; 628 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 15 MiB/s wr, 620 op/s 2023-07-29T16:31:53.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 60911 35.73 MB/sec warmup 97 sec latency 57.333 ms 2023-07-29T16:31:54.232 INFO:tasks.workunit.client.0.smithi031.stdout: 10 61673 35.73 MB/sec warmup 98 sec latency 51.164 ms 2023-07-29T16:31:54.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:54 smithi031 ceph-mon[95504]: pgmap v864: 97 pgs: 97 active+clean; 566 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 17 MiB/s wr, 843 op/s 2023-07-29T16:31:55.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:54 smithi039 ceph-mon[128858]: pgmap v864: 97 pgs: 97 active+clean; 566 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 17 MiB/s wr, 843 op/s 2023-07-29T16:31:55.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:54 smithi156 ceph-mon[129245]: pgmap v864: 97 pgs: 97 active+clean; 566 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 17 MiB/s wr, 843 op/s 2023-07-29T16:31:55.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 62338 35.73 MB/sec warmup 99 sec latency 153.097 ms 2023-07-29T16:31:56.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 63084 35.71 MB/sec warmup 100 sec latency 62.236 ms 2023-07-29T16:31:56.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:56 smithi031 ceph-mon[95504]: pgmap v865: 97 pgs: 97 active+clean; 518 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 16 MiB/s wr, 909 op/s 2023-07-29T16:31:57.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:56 smithi039 ceph-mon[128858]: pgmap v865: 97 pgs: 97 active+clean; 518 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 16 MiB/s wr, 909 op/s 2023-07-29T16:31:57.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:56 smithi156 ceph-mon[129245]: pgmap v865: 97 pgs: 97 active+clean; 518 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 16 MiB/s wr, 909 op/s 2023-07-29T16:31:57.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 64100 35.83 MB/sec warmup 101 sec latency 44.818 ms 2023-07-29T16:31:58.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65182 36.18 MB/sec warmup 102 sec latency 50.278 ms 2023-07-29T16:31:58.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:58 smithi156 ceph-mon[129245]: pgmap v866: 97 pgs: 97 active+clean; 524 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 15 MiB/s wr, 898 op/s 2023-07-29T16:31:58.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:58 smithi031 ceph-mon[95504]: pgmap v866: 97 pgs: 97 active+clean; 524 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 15 MiB/s wr, 898 op/s 2023-07-29T16:31:58.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:58 smithi039 ceph-mon[128858]: pgmap v866: 97 pgs: 97 active+clean; 524 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 15 MiB/s wr, 898 op/s 2023-07-29T16:31:59.233 INFO:tasks.workunit.client.0.smithi031.stdout: 10 65841 36.20 MB/sec warmup 103 sec latency 63.028 ms 2023-07-29T16:31:59.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:31:59 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:32:00.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:31:59 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:32:00.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:31:59 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:32:00.235 INFO:tasks.workunit.client.0.smithi031.stdout: 10 66413 36.14 MB/sec warmup 104 sec latency 88.623 ms 2023-07-29T16:32:00.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: pgmap v867: 97 pgs: 97 active+clean; 532 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 14 MiB/s wr, 933 op/s 2023-07-29T16:32:00.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:00.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:00 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: pgmap v867: 97 pgs: 97 active+clean; 532 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 14 MiB/s wr, 933 op/s 2023-07-29T16:32:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:00 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: pgmap v867: 97 pgs: 97 active+clean; 532 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 14 MiB/s wr, 933 op/s 2023-07-29T16:32:01.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:00 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67020 36.06 MB/sec warmup 105 sec latency 60.720 ms 2023-07-29T16:32:01.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:32:01.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:32:01.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:32:01.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:32:01.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:01.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:32:01.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:32:01.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:32:02.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:32:02.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:32:02.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:32:02.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:32:02.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:32:02.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:32:02.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:32:02.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:32:02.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:32:02.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:32:02.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 67828 36.08 MB/sec warmup 106 sec latency 50.446 ms 2023-07-29T16:32:02.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:02 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:32:02.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:02 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:32:02.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:02 smithi031 ceph-mon[95504]: pgmap v868: 97 pgs: 97 active+clean; 532 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 644 op/s 2023-07-29T16:32:03.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:02 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:32:03.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:02 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:32:03.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:02 smithi039 ceph-mon[128858]: pgmap v868: 97 pgs: 97 active+clean; 532 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 644 op/s 2023-07-29T16:32:03.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:02 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:32:03.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:02 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:32:03.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:02 smithi156 ceph-mon[129245]: pgmap v868: 97 pgs: 97 active+clean; 532 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 644 op/s 2023-07-29T16:32:03.238 INFO:tasks.workunit.client.0.smithi031.stdout: 10 68684 36.18 MB/sec warmup 107 sec latency 71.975 ms 2023-07-29T16:32:03.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.387+0000 7f79b05ff700 1 -- 172.21.15.31:0/4252389181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101540 msgr2=0x7f79a8101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.387+0000 7f79b05ff700 1 --2- 172.21.15.31:0/4252389181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101540 0x7f79a8101990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f7998009990 tx=0x7f7998009ca0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 -- 172.21.15.31:0/4252389181 shutdown_connections 2023-07-29T16:32:03.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 --2- 172.21.15.31:0/4252389181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8101ed0 0x7f79a8108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 --2- 172.21.15.31:0/4252389181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8100350 0x7f79a8100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 --2- 172.21.15.31:0/4252389181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101540 0x7f79a8101990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 -- 172.21.15.31:0/4252389181 >> 172.21.15.31:0/4252389181 conn(0x7f79a80fb970 msgr2=0x7f79a80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 -- 172.21.15.31:0/4252389181 shutdown_connections 2023-07-29T16:32:03.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.388+0000 7f79b05ff700 1 -- 172.21.15.31:0/4252389181 wait complete. 2023-07-29T16:32:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.389+0000 7f79b05ff700 1 Processor -- start 2023-07-29T16:32:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 -- start start 2023-07-29T16:32:03.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 0x7f79a8193f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8101540 0x7f79a8194490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101ed0 0x7f79a8195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8195970 con 0x7f79a8101540 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8195ae0 con 0x7f79a8101ed0 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79b05ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a8195ca0 con 0x7f79a8100350 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79ae39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 0x7f79a8193f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79ae39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 0x7f79a8193f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42796/0 (socket says 172.21.15.31:42796) 2023-07-29T16:32:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79ae39b700 1 -- 172.21.15.31:0/350761166 learned_addr learned my addr 172.21.15.31:0/350761166 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.390+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101ed0 0x7f79a8195310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79adb9a700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8101540 0x7f79a8194490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79ae39b700 1 -- 172.21.15.31:0/350761166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101ed0 msgr2=0x7f79a8195310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79ae39b700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101ed0 0x7f79a8195310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79ae39b700 1 -- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8101540 msgr2=0x7f79a8194490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79ae39b700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8101540 0x7f79a8194490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79ae39b700 1 -- 172.21.15.31:0/350761166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980095f0 con 0x7f79a8100350 2023-07-29T16:32:03.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79adb9a700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8101540 0x7f79a8194490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:32:03.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79ae39b700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 0x7f79a8193f50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009c70 tx=0x7f79a4009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:03.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f799f7fe700 1 -- 172.21.15.31:0/350761166 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a401d070 con 0x7f79a8100350 2023-07-29T16:32:03.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a8195f80 con 0x7f79a8100350 2023-07-29T16:32:03.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.391+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a819c740 con 0x7f79a8100350 2023-07-29T16:32:03.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.392+0000 7f799f7fe700 1 -- 172.21.15.31:0/350761166 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79a4015040 con 0x7f79a8100350 2023-07-29T16:32:03.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.392+0000 7f799f7fe700 1 -- 172.21.15.31:0/350761166 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40039e0 con 0x7f79a8100350 2023-07-29T16:32:03.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.396+0000 7f799f7fe700 1 -- 172.21.15.31:0/350761166 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f79a4003c40 con 0x7f79a8100350 2023-07-29T16:32:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.398+0000 7f79c3fd7b80 1 Processor -- start 2023-07-29T16:32:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.399+0000 7f799f7fe700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f79940a0890 0x7f79940a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.399+0000 7f799f7fe700 1 -- 172.21.15.31:0/350761166 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f79a40cef20 con 0x7f79a8100350 2023-07-29T16:32:03.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.400+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f79940a0890 0x7f79940a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 -- start start 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 0x563917e468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e46df0 0x563917e477b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e48160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917d9e9b0 con 0x563917e47cf0 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917d9c370 con 0x563917ca26b0 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917d97c60 con 0x563917e46df0 2023-07-29T16:32:03.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79ae39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 0x563917e468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79ae39b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 0x563917e468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43812/0 (socket says 172.21.15.31:43812) 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79adb9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e46df0 0x563917e477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.401+0000 7f79ae39b700 1 -- 172.21.15.31:0/425956077 learned_addr learned my addr 172.21.15.31:0/425956077 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79ae39b700 1 -- 172.21.15.31:0/425956077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e46df0 msgr2=0x563917e477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79ae39b700 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e46df0 0x563917e477b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79ae39b700 1 -- 172.21.15.31:0/425956077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 msgr2=0x563917e48160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:03.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79ae39b700 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e48160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79ae39b700 1 -- 172.21.15.31:0/425956077 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563917bc5d80 con 0x563917ca26b0 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79ae39b700 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 0x563917e468b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f79a409d360 tx=0x7f79a409d220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14ab69f130c38b2a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f79940a0890 0x7f79940a2d40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f79a000f3c0 tx=0x7f79a0007950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f799cbfe700 1 -- 172.21.15.31:0/425956077 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40a1690 con 0x563917ca26b0 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f799cbfe700 1 -- 172.21.15.31:0/425956077 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79a40a5a20 con 0x563917ca26b0 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.402+0000 7f799cbfe700 1 -- 172.21.15.31:0/425956077 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a409cdb0 con 0x563917ca26b0 2023-07-29T16:32:03.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.403+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/425956077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 msgr2=0x563917e468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.403+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 0x563917e468b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f79a409d360 tx=0x7f79a409d220 comp rx=0 tx=0).stop 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/425956077 shutdown_connections 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e48160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917ca26b0 0x563917e468b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/425956077 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e46df0 0x563917e477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/425956077 >> 172.21.15.31:0/425956077 conn(0x563917e3bfd0 msgr2=0x563917e41300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:03.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/425956077 shutdown_connections 2023-07-29T16:32:03.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.404+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/425956077 wait complete. 2023-07-29T16:32:03.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.405+0000 7f79c3fd7b80 1 -- start start 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.405+0000 7f79c3fd7b80 1 Processor -- start 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.405+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e4ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.405+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e50460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917d344b0 con 0x563917e47cf0 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917d34150 con 0x563917e46df0 2023-07-29T16:32:03.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917c1b8c0 con 0x563917ca26b0 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e4ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e4ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42824/0 (socket says 172.21.15.31:42824) 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 -- 172.21.15.31:0/1596206461 learned_addr learned my addr 172.21.15.31:0/1596206461 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79c3fd7b80 10 client.0 _ll_register_callbacks cb 0x563917d3d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79adb9a700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e50460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 -- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 msgr2=0x563917e50460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e50460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 -- 172.21.15.31:0/1596206461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 msgr2=0x563917e574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.406+0000 7f79ae39b700 1 -- 172.21.15.31:0/1596206461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a819c740 con 0x563917ca26b0 2023-07-29T16:32:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f79ae39b700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e4ff20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f79a409a0d0 tx=0x7f79a409ce10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=284ec791f6e612d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f7991ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f7991ffb700 1 -- 172.21.15.31:0/1596206461 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40a1690 con 0x563917ca26b0 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f7991ffb700 1 -- 172.21.15.31:0/1596206461 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f79a40a1380 con 0x563917ca26b0 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563917dc2cb0 con 0x563917ca26b0 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f7991ffb700 1 -- 172.21.15.31:0/1596206461 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79a40c6b90 con 0x563917ca26b0 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f79c3fd7b80 10 client.30184 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.407+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x563917baec00 con 0x563917ca26b0 2023-07-29T16:32:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f7991ffb700 1 -- 172.21.15.31:0/1596206461 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79a40ca020 con 0x563917ca26b0 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f7991ffb700 1 -- 172.21.15.31:0/1596206461 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f79a4007a30 con 0x563917ca26b0 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79c3fd7b80 10 client.30184 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79c3fd7b80 10 client.30184 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79c3fd7b80 20 client.30184 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917e5a2b0 0x563917e5c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79c3fd7b80 4 client.30184 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563917e15d10 con 0x563917e5a2b0 2023-07-29T16:32:03.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.408+0000 7f79adb9a700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917e5a2b0 0x563917e5c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.409+0000 7f7991ffb700 10 client.30184 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:03.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.409+0000 7f79adb9a700 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917e5a2b0 0x563917e5c6c0 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=288d1d63eed6e1c server_cookie=a1e0f8996a40da76 in_seq=0 out_seq=0 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.412+0000 7f7991ffb700 1 -- 172.21.15.31:0/1596206461 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563917e15d10 con 0x563917e5a2b0 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.412+0000 7f7991ffb700 10 client.30184 handle_command_reply: tid=0 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.412+0000 7f79c3fd7b80 1 client.30184 shutdown 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 msgr2=0x563917e4ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e4ff20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f79a409a0d0 tx=0x7f79a409ce10 comp rx=0 tx=0).stop 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 shutdown_connections 2023-07-29T16:32:03.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917e5a2b0 0x563917e5c6c0 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e4ff20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1596206461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e50460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 >> 172.21.15.31:0/1596206461 conn(0x563917e3bfd0 msgr2=0x563917e40a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f7991ffb700 0 client.30184 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 shutdown_connections 2023-07-29T16:32:03.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.415+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1596206461 wait complete. 2023-07-29T16:32:03.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.416+0000 7f79c3fd7b80 20 client.30184 trim_cache size 0 max 16384 2023-07-29T16:32:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.438+0000 7f79c3fd7b80 1 Processor -- start 2023-07-29T16:32:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.438+0000 7f79c3fd7b80 1 -- start start 2023-07-29T16:32:03.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.439+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e40a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.439+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e57460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.439+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e4ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.439+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917e4b6c0 con 0x563917e47cf0 2023-07-29T16:32:03.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.439+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917e5a010 con 0x563917e46df0 2023-07-29T16:32:03.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.439+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917e42d10 con 0x563917ca26b0 2023-07-29T16:32:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.440+0000 7f79aeb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e4ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.440+0000 7f79aeb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e4ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38612/0 (socket says 172.21.15.31:38612) 2023-07-29T16:32:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.440+0000 7f79aeb9c700 1 -- 172.21.15.31:0/3741964704 learned_addr learned my addr 172.21.15.31:0/3741964704 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:03.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.441+0000 7f79aeb9c700 1 -- 172.21.15.31:0/3741964704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 msgr2=0x563917e40a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:32:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.441+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e40a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.441+0000 7f79aeb9c700 1 -- 172.21.15.31:0/3741964704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 msgr2=0x563917e57460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:32:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.441+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e57460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.441+0000 7f79aeb9c700 1 -- 172.21.15.31:0/3741964704 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563917bec1f0 con 0x563917e47cf0 2023-07-29T16:32:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.442+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e4ec70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f79a00155c0 tx=0x7f79a0004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420020d124d5b9b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.442+0000 7f799d3ff700 1 -- 172.21.15.31:0/3741964704 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0015d40 con 0x563917e47cf0 2023-07-29T16:32:03.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.442+0000 7f799d3ff700 1 -- 172.21.15.31:0/3741964704 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79a00218e0 con 0x563917e47cf0 2023-07-29T16:32:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.443+0000 7f799d3ff700 1 -- 172.21.15.31:0/3741964704 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a00148b0 con 0x563917e47cf0 2023-07-29T16:32:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.443+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/3741964704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 msgr2=0x563917e4ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.443+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e4ec70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f79a00155c0 tx=0x7f79a0004b40 comp rx=0 tx=0).stop 2023-07-29T16:32:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.444+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/3741964704 shutdown_connections 2023-07-29T16:32:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.444+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917e47cf0 0x563917e4ec70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.444+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917ca26b0 0x563917e40a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.444+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/3741964704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e57460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.444+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/3741964704 >> 172.21.15.31:0/3741964704 conn(0x563917e3bfd0 msgr2=0x563917e3d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.445+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/3741964704 shutdown_connections 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.445+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/3741964704 wait complete. 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.445+0000 7f79c3fd7b80 1 -- start start 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 Processor -- start 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917ca26b0 0x563917e52e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e53380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 0x563917e563a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917e568e0 con 0x563917ca26b0 2023-07-29T16:32:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917e56a50 con 0x563917e46df0 2023-07-29T16:32:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563917e56c10 con 0x563917e47cf0 2023-07-29T16:32:03.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.446+0000 7f79c3fd7b80 10 client.0 _ll_register_callbacks cb 0x563917d3d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.447+0000 7f79aeb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 0x563917e563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.447+0000 7f79aeb9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 0x563917e563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42828/0 (socket says 172.21.15.31:42828) 2023-07-29T16:32:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.447+0000 7f79aeb9c700 1 -- 172.21.15.31:0/1464935899 learned_addr learned my addr 172.21.15.31:0/1464935899 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.448+0000 7f79ae39b700 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917ca26b0 0x563917e52e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f79aeb9c700 1 -- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 msgr2=0x563917e53380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:32:03.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e53380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f79aeb9c700 1 -- 172.21.15.31:0/1464935899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917ca26b0 msgr2=0x563917e52e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917ca26b0 0x563917e52e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f79aeb9c700 1 -- 172.21.15.31:0/1464935899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a000feb0 con 0x563917e47cf0 2023-07-29T16:32:03.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f79aeb9c700 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 0x563917e563a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79a0021680 tx=0x7f79a0021760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5372e2d46949ab1f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.449+0000 7f7991ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.450+0000 7f7991ffb700 1 -- 172.21.15.31:0/1464935899 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a00147f0 con 0x563917e47cf0 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.450+0000 7f7991ffb700 1 -- 172.21.15.31:0/1464935899 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f79a0014950 con 0x563917e47cf0 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.450+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79a40a5a20 con 0x563917e47cf0 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.450+0000 7f7991ffb700 1 -- 172.21.15.31:0/1464935899 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79a0021de0 con 0x563917e47cf0 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.450+0000 7f79c3fd7b80 10 client.30187 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:03.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.450+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x563917bc4250 con 0x563917e47cf0 2023-07-29T16:32:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f7991ffb700 1 -- 172.21.15.31:0/1464935899 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79a40a5a20 con 0x563917e47cf0 2023-07-29T16:32:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f7991ffb700 1 -- 172.21.15.31:0/1464935899 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f79a003ec70 con 0x563917e47cf0 2023-07-29T16:32:03.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79c3fd7b80 10 client.30187 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79c3fd7b80 10 client.30187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79c3fd7b80 20 client.30187 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917eaeed0 0x563917eb12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79c3fd7b80 4 client.30187 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:32:03.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563917d90ce0 con 0x563917eaeed0 2023-07-29T16:32:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.453+0000 7f79ae39b700 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917eaeed0 0x563917eb12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:03.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.454+0000 7f7991ffb700 10 client.30187 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:03.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.454+0000 7f79ae39b700 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917eaeed0 0x563917eb12e0 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4167dc568dc73e31 server_cookie=71483dc4b67f24be in_seq=0 out_seq=0 2023-07-29T16:32:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.756+0000 7f7991ffb700 1 -- 172.21.15.31:0/1464935899 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f79a40ca040 con 0x563917eaeed0 2023-07-29T16:32:03.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.756+0000 7f7991ffb700 10 client.30187 handle_command_reply: tid=0 2023-07-29T16:32:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.756+0000 7f79c3fd7b80 1 client.30187 shutdown 2023-07-29T16:32:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.756+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 msgr2=0x563917e563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.756+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 0x563917e563a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79a0021680 tx=0x7f79a0021760 comp rx=0 tx=0).stop 2023-07-29T16:32:03.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.757+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 shutdown_connections 2023-07-29T16:32:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.757+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563917eaeed0 0x563917eb12e0 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.757+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563917e47cf0 0x563917e563a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.757+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563917ca26b0 0x563917e52e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.757+0000 7f79c3fd7b80 1 --2- 172.21.15.31:0/1464935899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563917e46df0 0x563917e53380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.757+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 >> 172.21.15.31:0/1464935899 conn(0x563917e3bfd0 msgr2=0x563917e4ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.759+0000 7f7991ffb700 0 client.30187 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:03.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.760+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 shutdown_connections 2023-07-29T16:32:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.760+0000 7f79c3fd7b80 1 -- 172.21.15.31:0/1464935899 wait complete. 2023-07-29T16:32:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.761+0000 7f79c3fd7b80 20 client.30187 trim_cache size 0 max 16384 2023-07-29T16:32:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:32:03.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:32:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:32:03.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:32:03.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.761+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f79940a0890 msgr2=0x7f79940a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.761+0000 7f79b05ff700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f79940a0890 0x7f79940a2d40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f79a000f3c0 tx=0x7f79a0007950 comp rx=0 tx=0).stop 2023-07-29T16:32:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.761+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 msgr2=0x7f79a8193f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.761+0000 7f79b05ff700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 0x7f79a8193f50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009c70 tx=0x7f79a4009f80 comp rx=0 tx=0).stop 2023-07-29T16:32:03.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 shutdown_connections 2023-07-29T16:32:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f79940a0890 0x7f79940a2d40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f79a8101ed0 0x7f79a8195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79a8100350 0x7f79a8193f50 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 --2- 172.21.15.31:0/350761166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79a8101540 0x7f79a8194490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 >> 172.21.15.31:0/350761166 conn(0x7f79a80fb970 msgr2=0x7f79a8104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 shutdown_connections 2023-07-29T16:32:03.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:03.762+0000 7f79b05ff700 1 -- 172.21.15.31:0/350761166 wait complete. 2023-07-29T16:32:04.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 69304 36.18 MB/sec warmup 108 sec latency 210.750 ms 2023-07-29T16:32:04.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:04 smithi031 ceph-mon[95504]: pgmap v869: 97 pgs: 97 active+clean; 544 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.7 MiB/s wr, 773 op/s 2023-07-29T16:32:05.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:04 smithi039 ceph-mon[128858]: pgmap v869: 97 pgs: 97 active+clean; 544 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.7 MiB/s wr, 773 op/s 2023-07-29T16:32:05.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:04 smithi156 ceph-mon[129245]: pgmap v869: 97 pgs: 97 active+clean; 544 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.7 MiB/s wr, 773 op/s 2023-07-29T16:32:05.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70046 36.28 MB/sec warmup 109 sec latency 57.438 ms 2023-07-29T16:32:06.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 70768 36.27 MB/sec warmup 110 sec latency 66.172 ms 2023-07-29T16:32:06.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:06 smithi031 ceph-mon[95504]: pgmap v870: 97 pgs: 97 active+clean; 546 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.5 MiB/s wr, 660 op/s 2023-07-29T16:32:07.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:06 smithi039 ceph-mon[128858]: pgmap v870: 97 pgs: 97 active+clean; 546 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.5 MiB/s wr, 660 op/s 2023-07-29T16:32:07.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:06 smithi156 ceph-mon[129245]: pgmap v870: 97 pgs: 97 active+clean; 546 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.5 MiB/s wr, 660 op/s 2023-07-29T16:32:07.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 71666 36.35 MB/sec warmup 111 sec latency 40.237 ms 2023-07-29T16:32:08.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 72522 36.50 MB/sec warmup 112 sec latency 77.579 ms 2023-07-29T16:32:08.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:08 smithi031 ceph-mon[95504]: pgmap v871: 97 pgs: 97 active+clean; 543 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.6 MiB/s wr, 484 op/s 2023-07-29T16:32:08.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:08 smithi039 ceph-mon[128858]: pgmap v871: 97 pgs: 97 active+clean; 543 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.6 MiB/s wr, 484 op/s 2023-07-29T16:32:08.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:08 smithi156 ceph-mon[129245]: pgmap v871: 97 pgs: 97 active+clean; 543 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.6 MiB/s wr, 484 op/s 2023-07-29T16:32:09.237 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73056 36.41 MB/sec warmup 113 sec latency 318.252 ms 2023-07-29T16:32:10.238 INFO:tasks.workunit.client.0.smithi031.stdout: 10 73780 36.35 MB/sec warmup 114 sec latency 211.639 ms 2023-07-29T16:32:10.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:10 smithi031 ceph-mon[95504]: pgmap v872: 97 pgs: 97 active+clean; 557 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 663 op/s 2023-07-29T16:32:11.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:10 smithi039 ceph-mon[128858]: pgmap v872: 97 pgs: 97 active+clean; 557 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 663 op/s 2023-07-29T16:32:11.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:10 smithi156 ceph-mon[129245]: pgmap v872: 97 pgs: 97 active+clean; 557 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 663 op/s 2023-07-29T16:32:11.238 INFO:tasks.workunit.client.0.smithi031.stdout: 10 74672 36.47 MB/sec warmup 115 sec latency 49.771 ms 2023-07-29T16:32:12.238 INFO:tasks.workunit.client.0.smithi031.stdout: 10 75462 36.61 MB/sec warmup 116 sec latency 69.865 ms 2023-07-29T16:32:12.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:12 smithi031 ceph-mon[95504]: pgmap v873: 97 pgs: 97 active+clean; 557 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 531 op/s 2023-07-29T16:32:13.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:12 smithi039 ceph-mon[128858]: pgmap v873: 97 pgs: 97 active+clean; 557 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 531 op/s 2023-07-29T16:32:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:12 smithi156 ceph-mon[129245]: pgmap v873: 97 pgs: 97 active+clean; 557 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 531 op/s 2023-07-29T16:32:13.238 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76124 36.59 MB/sec warmup 117 sec latency 88.081 ms 2023-07-29T16:32:14.238 INFO:tasks.workunit.client.0.smithi031.stdout: 10 76670 36.47 MB/sec warmup 118 sec latency 90.302 ms 2023-07-29T16:32:14.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:14 smithi031 ceph-mon[95504]: pgmap v874: 97 pgs: 97 active+clean; 567 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.9 MiB/s wr, 667 op/s 2023-07-29T16:32:15.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:14 smithi039 ceph-mon[128858]: pgmap v874: 97 pgs: 97 active+clean; 567 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.9 MiB/s wr, 667 op/s 2023-07-29T16:32:15.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:14 smithi156 ceph-mon[129245]: pgmap v874: 97 pgs: 97 active+clean; 567 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.9 MiB/s wr, 667 op/s 2023-07-29T16:32:15.239 INFO:tasks.workunit.client.0.smithi031.stdout: 10 77410 36.53 MB/sec warmup 119 sec latency 137.992 ms 2023-07-29T16:32:16.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:16 smithi031 ceph-mon[95504]: pgmap v875: 97 pgs: 97 active+clean; 571 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.8 MiB/s wr, 611 op/s 2023-07-29T16:32:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.986+0000 7f13371ff700 1 -- 172.21.15.31:0/2376030858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 msgr2=0x7f1330105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.986+0000 7f13371ff700 1 --2- 172.21.15.31:0/2376030858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f1330105b60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f131c0098f0 tx=0x7f131c009c00 comp rx=0 tx=0).stop 2023-07-29T16:32:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.987+0000 7f13371ff700 1 -- 172.21.15.31:0/2376030858 shutdown_connections 2023-07-29T16:32:16.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.987+0000 7f13371ff700 1 --2- 172.21.15.31:0/2376030858 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f13301060a0 0x7f1330108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.987+0000 7f13371ff700 1 --2- 172.21.15.31:0/2376030858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1330100e60 0x7f1330103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.987+0000 7f13371ff700 1 --2- 172.21.15.31:0/2376030858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f1330105b60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.987+0000 7f13371ff700 1 -- 172.21.15.31:0/2376030858 >> 172.21.15.31:0/2376030858 conn(0x7f13300fa840 msgr2=0x7f13300fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.987+0000 7f13371ff700 1 -- 172.21.15.31:0/2376030858 shutdown_connections 2023-07-29T16:32:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.988+0000 7f13371ff700 1 -- 172.21.15.31:0/2376030858 wait complete. 2023-07-29T16:32:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.989+0000 7f13371ff700 1 Processor -- start 2023-07-29T16:32:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.989+0000 7f13371ff700 1 -- start start 2023-07-29T16:32:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f13371ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330100e60 0x7f133019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f13371ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f133019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f13371ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13301060a0 0x7f133019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f13371ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133019e220 con 0x7f13301060a0 2023-07-29T16:32:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f13371ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133019e390 con 0x7f1330100e60 2023-07-29T16:32:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f13371ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133019e550 con 0x7f1330103780 2023-07-29T16:32:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330100e60 0x7f133019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f132ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f133019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330100e60 0x7f133019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48510/0 (socket says 172.21.15.31:48510) 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f1334f9b700 1 -- 172.21.15.31:0/3158517468 learned_addr learned my addr 172.21.15.31:0/3158517468 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.990+0000 7f132ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f133019cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57106/0 (socket says 172.21.15.31:57106) 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f133579c700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13301060a0 0x7f133019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f132ffff700 1 -- 172.21.15.31:0/3158517468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330100e60 msgr2=0x7f133019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f132ffff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330100e60 0x7f133019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f132ffff700 1 -- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13301060a0 msgr2=0x7f133019dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f132ffff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13301060a0 0x7f133019dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f132ffff700 1 -- 172.21.15.31:0/3158517468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328005e70 con 0x7f1330103780 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f133579c700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13301060a0 0x7f133019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.991+0000 7f132ffff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f133019cd40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f131c009230 tx=0x7f131c00b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.992+0000 7f132dffb700 1 -- 172.21.15.31:0/3158517468 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c01c070 con 0x7f1330103780 2023-07-29T16:32:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.992+0000 7f132dffb700 1 -- 172.21.15.31:0/3158517468 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f131c00f860 con 0x7f1330103780 2023-07-29T16:32:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.992+0000 7f132dffb700 1 -- 172.21.15.31:0/3158517468 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c003af0 con 0x7f1330103780 2023-07-29T16:32:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.992+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c009520 con 0x7f1330103780 2023-07-29T16:32:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.992+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133019ea10 con 0x7f1330103780 2023-07-29T16:32:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.998+0000 7f132dffb700 1 -- 172.21.15.31:0/3158517468 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f131c020070 con 0x7f1330103780 2023-07-29T16:32:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.998+0000 7f134ab9bb80 1 Processor -- start 2023-07-29T16:32:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:16.999+0000 7f134ab9bb80 1 -- start start 2023-07-29T16:32:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 0x5648ad5cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad5269b0 con 0x5648ad5cfcf0 2023-07-29T16:32:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad524370 con 0x5648ad42a6b0 2023-07-29T16:32:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad51fc60 con 0x5648ad5cedf0 2023-07-29T16:32:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 0x5648ad5cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 0x5648ad5cf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57120/0 (socket says 172.21.15.31:57120) 2023-07-29T16:32:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.000+0000 7f1334f9b700 1 -- 172.21.15.31:0/4149323201 learned_addr learned my addr 172.21.15.31:0/4149323201 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:17.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f1334f9b700 1 -- 172.21.15.31:0/4149323201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 msgr2=0x5648ad5ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f1334f9b700 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f1334f9b700 1 -- 172.21.15.31:0/4149323201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5cfcf0 msgr2=0x5648ad5d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f1334f9b700 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f1334f9b700 1 -- 172.21.15.31:0/4149323201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648ad34dd80 con 0x5648ad5cedf0 2023-07-29T16:32:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f1334f9b700 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 0x5648ad5cf7b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1324007dc0 tx=0x7f13240106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=296ef5d5c1dffe03 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.001+0000 7f131affd700 1 -- 172.21.15.31:0/4149323201 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324010ec0 con 0x5648ad5cedf0 2023-07-29T16:32:17.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.002+0000 7f131affd700 1 -- 172.21.15.31:0/4149323201 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1324015070 con 0x5648ad5cedf0 2023-07-29T16:32:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.002+0000 7f131affd700 1 -- 172.21.15.31:0/4149323201 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324019670 con 0x5648ad5cedf0 2023-07-29T16:32:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.002+0000 7f132dffb700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13200a0810 0x7f13200a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.002+0000 7f132dffb700 1 -- 172.21.15.31:0/3158517468 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f131c0cc950 con 0x7f1330103780 2023-07-29T16:32:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.002+0000 7f133579c700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13200a0810 0x7f13200a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 -- 172.21.15.31:0/4149323201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 msgr2=0x5648ad5cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 0x5648ad5cf7b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1324007dc0 tx=0x7f13240106b0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 -- 172.21.15.31:0/4149323201 shutdown_connections 2023-07-29T16:32:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/4149323201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cedf0 0x5648ad5cf7b0 secure :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1324007dc0 tx=0x7f13240106b0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.003+0000 7f134ab9bb80 1 -- 172.21.15.31:0/4149323201 >> 172.21.15.31:0/4149323201 conn(0x5648ad5c3fd0 msgr2=0x5648ad5c9300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.004+0000 7f133579c700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13200a0810 0x7f13200a2cc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1328009fd0 tx=0x7f13280099e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.004+0000 7f134ab9bb80 1 -- 172.21.15.31:0/4149323201 shutdown_connections 2023-07-29T16:32:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.004+0000 7f134ab9bb80 1 -- 172.21.15.31:0/4149323201 wait complete. 2023-07-29T16:32:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.005+0000 7f134ab9bb80 1 -- start start 2023-07-29T16:32:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.006+0000 7f134ab9bb80 1 Processor -- start 2023-07-29T16:32:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad5d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 0x5648ad5dfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad4bc4b0 con 0x5648ad42a6b0 2023-07-29T16:32:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad4bc150 con 0x5648ad5dd710 2023-07-29T16:32:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad3a38c0 con 0x5648ad5cfcf0 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.007+0000 7f134ab9bb80 10 client.0 _ll_register_callbacks cb 0x5648ad4c55c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 0x5648ad5dfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 0x5648ad5dfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48526/0 (socket says 172.21.15.31:48526) 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 -- 172.21.15.31:0/2492176965 learned_addr learned my addr 172.21.15.31:0/2492176965 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f132ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f1334f9b700 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad5d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 -- 172.21.15.31:0/2492176965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 msgr2=0x5648ad5d86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 -- 172.21.15.31:0/2492176965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 msgr2=0x5648ad5d81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad5d81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.008+0000 7f133579c700 1 -- 172.21.15.31:0/2492176965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13240079f0 con 0x5648ad5dd710 2023-07-29T16:32:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.009+0000 7f133579c700 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 0x5648ad5dfaf0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1328019b10 tx=0x7f132800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54759cf89956b8c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.009+0000 7f130ffff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:32:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.010+0000 7f130ffff700 1 -- 172.21.15.31:0/2492176965 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132801e070 con 0x5648ad5dd710 2023-07-29T16:32:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.010+0000 7f130ffff700 1 -- 172.21.15.31:0/2492176965 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1328022410 con 0x5648ad5dd710 2023-07-29T16:32:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.010+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648ad54acb0 con 0x5648ad5dd710 2023-07-29T16:32:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.010+0000 7f130ffff700 1 -- 172.21.15.31:0/2492176965 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13280225d0 con 0x5648ad5dd710 2023-07-29T16:32:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.011+0000 7f134ab9bb80 10 client.30182 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:17.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.011+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5648ad336c00 con 0x5648ad5dd710 2023-07-29T16:32:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.011+0000 7f130ffff700 1 -- 172.21.15.31:0/2492176965 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5648ad54acb0 con 0x5648ad5dd710 2023-07-29T16:32:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.011+0000 7f130ffff700 1 -- 172.21.15.31:0/2492176965 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1328017690 con 0x5648ad5dd710 2023-07-29T16:32:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f134ab9bb80 10 client.30182 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f134ab9bb80 10 client.30182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:17.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f134ab9bb80 20 client.30182 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad5e43e0 0x5648ad5e67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f134ab9bb80 4 client.30182 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:32:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648ad59dd10 con 0x5648ad5e43e0 2023-07-29T16:32:17.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.012+0000 7f1334f9b700 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad5e43e0 0x5648ad5e67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.013+0000 7f1334f9b700 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad5e43e0 0x5648ad5e67f0 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56bda63c9ee4628f server_cookie=ae97ce41266fa211 in_seq=0 out_seq=0 2023-07-29T16:32:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.013+0000 7f130ffff700 10 client.30182 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.017+0000 7f130ffff700 1 -- 172.21.15.31:0/2492176965 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5648ad59dd10 con 0x5648ad5e43e0 2023-07-29T16:32:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.017+0000 7f130ffff700 10 client.30182 handle_command_reply: tid=0 2023-07-29T16:32:17.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.018+0000 7f134ab9bb80 1 client.30182 shutdown 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 msgr2=0x5648ad5dfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 0x5648ad5dfaf0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f1328019b10 tx=0x7f132800b040 comp rx=0 tx=0).stop 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 shutdown_connections 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad5e43e0 0x5648ad5e67f0 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5dd710 0x5648ad5dfaf0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad5d81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f130ffff700 0 client.30182 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:17.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2492176965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5d86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 >> 172.21.15.31:0/2492176965 conn(0x5648ad5c3fd0 msgr2=0x5648ad5c92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:17.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.019+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 shutdown_connections 2023-07-29T16:32:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.020+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2492176965 wait complete. 2023-07-29T16:32:17.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.020+0000 7f134ab9bb80 20 client.30182 trim_cache size 0 max 16384 2023-07-29T16:32:17.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:16 smithi039 ceph-mon[128858]: pgmap v875: 97 pgs: 97 active+clean; 571 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.8 MiB/s wr, 611 op/s 2023-07-29T16:32:17.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.048+0000 7f134ab9bb80 1 Processor -- start 2023-07-29T16:32:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.048+0000 7f134ab9bb80 1 -- start start 2023-07-29T16:32:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5dd710 0x5648ad5d2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad54acb0 con 0x5648ad5dd710 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad5d7d10 con 0x5648ad42a6b0 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad5e4200 con 0x5648ad5cfcf0 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48540/0 (socket says 172.21.15.31:48540) 2023-07-29T16:32:17.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f1334f9b700 1 -- 172.21.15.31:0/2206304388 learned_addr learned my addr 172.21.15.31:0/2206304388 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.049+0000 7f132ffff700 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.050+0000 7f132ffff700 1 -- 172.21.15.31:0/2206304388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 msgr2=0x5648ad5f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.050+0000 7f132ffff700 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.050+0000 7f132ffff700 1 -- 172.21.15.31:0/2206304388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5dd710 msgr2=0x5648ad5d2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.050+0000 7f132ffff700 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5dd710 0x5648ad5d2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.050+0000 7f132ffff700 1 -- 172.21.15.31:0/2206304388 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648ad3741f0 con 0x5648ad5cfcf0 2023-07-29T16:32:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.051+0000 7f132ffff700 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5c7fa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x5648ad5ce610 tx=0x7f131c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b149d37a069da17 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:17.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.051+0000 7f131b7fe700 1 -- 172.21.15.31:0/2206304388 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c0b1070 con 0x5648ad5cfcf0 2023-07-29T16:32:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.054+0000 7f131b7fe700 1 -- 172.21.15.31:0/2206304388 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f131c0a24a0 con 0x5648ad5cfcf0 2023-07-29T16:32:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.054+0000 7f131b7fe700 1 -- 172.21.15.31:0/2206304388 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c00baf0 con 0x5648ad5cfcf0 2023-07-29T16:32:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.055+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2206304388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 msgr2=0x5648ad5c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.055+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5c7fa0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x5648ad5ce610 tx=0x7f131c000bc0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.056+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2206304388 shutdown_connections 2023-07-29T16:32:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.056+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad5dd710 0x5648ad5d2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.056+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad42a6b0 0x5648ad5f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.056+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/2206304388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5cfcf0 0x5648ad5c7fa0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.056+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2206304388 >> 172.21.15.31:0/2206304388 conn(0x5648ad5c3fd0 msgr2=0x5648ad5cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:17.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.059+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2206304388 shutdown_connections 2023-07-29T16:32:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.059+0000 7f134ab9bb80 1 -- 172.21.15.31:0/2206304388 wait complete. 2023-07-29T16:32:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.060+0000 7f134ab9bb80 1 -- start start 2023-07-29T16:32:17.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.060+0000 7f134ab9bb80 1 Processor -- start 2023-07-29T16:32:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad641540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5cfcf0 0x5648ad643a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5dd710 0x5648ad645fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad646520 con 0x5648ad42a6b0 2023-07-29T16:32:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad646660 con 0x5648ad5cfcf0 2023-07-29T16:32:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648ad6467a0 con 0x5648ad5dd710 2023-07-29T16:32:17.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.061+0000 7f134ab9bb80 10 client.0 _ll_register_callbacks cb 0x5648ad4c55c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.064+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad641540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.064+0000 7f1334f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad641540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59962/0 (socket says 172.21.15.31:59962) 2023-07-29T16:32:17.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.064+0000 7f1334f9b700 1 -- 172.21.15.31:0/808701740 learned_addr learned my addr 172.21.15.31:0/808701740 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:17.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f133579c700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5dd710 0x5648ad645fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f1334f9b700 1 -- 172.21.15.31:0/808701740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5dd710 msgr2=0x5648ad645fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f1334f9b700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5dd710 0x5648ad645fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f1334f9b700 1 -- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5cfcf0 msgr2=0x5648ad643a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f132ffff700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5cfcf0 0x5648ad643a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f1334f9b700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5cfcf0 0x5648ad643a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f1334f9b700 1 -- 172.21.15.31:0/808701740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133019ea10 con 0x5648ad42a6b0 2023-07-29T16:32:17.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f133579c700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5dd710 0x5648ad645fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:32:17.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.065+0000 7f132ffff700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5cfcf0 0x5648ad643a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:32:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.071+0000 7f1334f9b700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad641540 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x5648ad5ca110 tx=0x7f1324011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aace4640ed964e2f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:17.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.071+0000 7f130ffff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:32:17.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:16 smithi156 ceph-mon[129245]: pgmap v875: 97 pgs: 97 active+clean; 571 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.8 MiB/s wr, 611 op/s 2023-07-29T16:32:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.072+0000 7f130ffff700 1 -- 172.21.15.31:0/808701740 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132402e070 con 0x5648ad42a6b0 2023-07-29T16:32:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.072+0000 7f130ffff700 1 -- 172.21.15.31:0/808701740 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f1324029130 con 0x5648ad42a6b0 2023-07-29T16:32:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.072+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1324015070 con 0x5648ad42a6b0 2023-07-29T16:32:17.074 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.073+0000 7f130ffff700 1 -- 172.21.15.31:0/808701740 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13240292f0 con 0x5648ad42a6b0 2023-07-29T16:32:17.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.076+0000 7f134ab9bb80 10 client.19980 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:17.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.076+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5648ad34c250 con 0x5648ad42a6b0 2023-07-29T16:32:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.076+0000 7f130ffff700 1 -- 172.21.15.31:0/808701740 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1324015070 con 0x5648ad42a6b0 2023-07-29T16:32:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f130ffff700 1 -- 172.21.15.31:0/808701740 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1324031020 con 0x5648ad42a6b0 2023-07-29T16:32:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f134ab9bb80 10 client.19980 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f134ab9bb80 10 client.19980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f134ab9bb80 20 client.19980 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:17.078 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad648100 0x5648ad64a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f134ab9bb80 4 client.19980 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:32:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.077+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5648ad518ce0 con 0x5648ad648100 2023-07-29T16:32:17.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.079+0000 7f132ffff700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad648100 0x5648ad64a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:17.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.082+0000 7f132ffff700 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad648100 0x5648ad64a510 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e7ece79b45184cb server_cookie=548397fbc4d14022 in_seq=0 out_seq=0 2023-07-29T16:32:17.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.083+0000 7f130ffff700 10 client.19980 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:17.239 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79049 41.68 MB/sec execute 1 sec latency 216.098 ms 2023-07-29T16:32:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.437+0000 7f130ffff700 1 -- 172.21.15.31:0/808701740 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5648ad518ce0 con 0x5648ad648100 2023-07-29T16:32:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.437+0000 7f130ffff700 10 client.19980 handle_command_reply: tid=0 2023-07-29T16:32:17.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.437+0000 7f134ab9bb80 1 client.19980 shutdown 2023-07-29T16:32:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.443+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 msgr2=0x5648ad641540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.443+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad641540 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x5648ad5ca110 tx=0x7f1324011e20 comp rx=0 tx=0).stop 2023-07-29T16:32:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.447+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 shutdown_connections 2023-07-29T16:32:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.447+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5648ad648100 0x5648ad64a510 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.447+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5648ad5dd710 0x5648ad645fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.447+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648ad42a6b0 0x5648ad641540 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.447+0000 7f134ab9bb80 1 --2- 172.21.15.31:0/808701740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5648ad5cfcf0 0x5648ad643a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.447+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 >> 172.21.15.31:0/808701740 conn(0x5648ad5c3fd0 msgr2=0x5648ad5c7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.448+0000 7f130ffff700 0 client.19980 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.448+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 shutdown_connections 2023-07-29T16:32:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.448+0000 7f134ab9bb80 1 -- 172.21.15.31:0/808701740 wait complete. 2023-07-29T16:32:17.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.449+0000 7f134ab9bb80 20 client.19980 trim_cache size 0 max 16384 2023-07-29T16:32:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:32:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:32:17.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:32:17.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:32:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.452+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13200a0810 msgr2=0x7f13200a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:17.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.452+0000 7f13371ff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13200a0810 0x7f13200a2cc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1328009fd0 tx=0x7f13280099e0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.452+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 msgr2=0x7f133019cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.452+0000 7f13371ff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f133019cd40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f131c009230 tx=0x7f131c00b790 comp rx=0 tx=0).stop 2023-07-29T16:32:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.456+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 shutdown_connections 2023-07-29T16:32:17.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.456+0000 7f13371ff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f13200a0810 0x7f13200a2cc0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.456+0000 7f13371ff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13301060a0 0x7f133019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.456+0000 7f13371ff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1330100e60 0x7f133019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.456+0000 7f13371ff700 1 --2- 172.21.15.31:0/3158517468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1330103780 0x7f133019cd40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:17.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.456+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 >> 172.21.15.31:0/3158517468 conn(0x7f13300fa840 msgr2=0x7f1330101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.458+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 shutdown_connections 2023-07-29T16:32:17.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:17.458+0000 7f13371ff700 1 -- 172.21.15.31:0/3158517468 wait complete. 2023-07-29T16:32:18.239 INFO:tasks.workunit.client.0.smithi031.stdout: 10 79694 38.64 MB/sec execute 2 sec latency 278.492 ms 2023-07-29T16:32:18.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:18 smithi031 ceph-mon[95504]: pgmap v876: 97 pgs: 97 active+clean; 572 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.6 MiB/s wr, 540 op/s 2023-07-29T16:32:18.530 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:18 smithi039 ceph-mon[128858]: pgmap v876: 97 pgs: 97 active+clean; 572 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.6 MiB/s wr, 540 op/s 2023-07-29T16:32:18.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:18 smithi156 ceph-mon[129245]: pgmap v876: 97 pgs: 97 active+clean; 572 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.6 MiB/s wr, 540 op/s 2023-07-29T16:32:19.241 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80134 34.22 MB/sec execute 3 sec latency 355.671 ms 2023-07-29T16:32:20.239 INFO:tasks.workunit.client.0.smithi031.stdout: 10 80705 32.86 MB/sec execute 4 sec latency 176.015 ms 2023-07-29T16:32:20.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:20 smithi031 ceph-mon[95504]: pgmap v877: 97 pgs: 97 active+clean; 598 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 819 op/s 2023-07-29T16:32:21.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:20 smithi039 ceph-mon[128858]: pgmap v877: 97 pgs: 97 active+clean; 598 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 819 op/s 2023-07-29T16:32:21.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:20 smithi156 ceph-mon[129245]: pgmap v877: 97 pgs: 97 active+clean; 598 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.5 MiB/s wr, 819 op/s 2023-07-29T16:32:21.239 INFO:tasks.workunit.client.0.smithi031.stdout: 10 81432 33.54 MB/sec execute 5 sec latency 380.084 ms 2023-07-29T16:32:22.240 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82263 35.57 MB/sec execute 6 sec latency 71.364 ms 2023-07-29T16:32:22.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:22 smithi031 ceph-mon[95504]: pgmap v878: 97 pgs: 97 active+clean; 598 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.8 MiB/s wr, 574 op/s 2023-07-29T16:32:23.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:22 smithi039 ceph-mon[128858]: pgmap v878: 97 pgs: 97 active+clean; 598 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.8 MiB/s wr, 574 op/s 2023-07-29T16:32:23.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:22 smithi156 ceph-mon[129245]: pgmap v878: 97 pgs: 97 active+clean; 598 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.8 MiB/s wr, 574 op/s 2023-07-29T16:32:23.239 INFO:tasks.workunit.client.0.smithi031.stdout: 10 82962 35.34 MB/sec execute 7 sec latency 96.084 ms 2023-07-29T16:32:24.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83381 33.29 MB/sec execute 8 sec latency 507.725 ms 2023-07-29T16:32:24.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:24 smithi031 ceph-mon[95504]: pgmap v879: 97 pgs: 97 active+clean; 593 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.9 MiB/s wr, 779 op/s 2023-07-29T16:32:25.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:24 smithi039 ceph-mon[128858]: pgmap v879: 97 pgs: 97 active+clean; 593 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.9 MiB/s wr, 779 op/s 2023-07-29T16:32:25.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:24 smithi156 ceph-mon[129245]: pgmap v879: 97 pgs: 97 active+clean; 593 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 9.9 MiB/s wr, 779 op/s 2023-07-29T16:32:25.243 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83462 30.09 MB/sec execute 9 sec latency 935.539 ms 2023-07-29T16:32:26.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 83917 29.45 MB/sec execute 10 sec latency 218.712 ms 2023-07-29T16:32:26.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:26 smithi031 ceph-mon[95504]: pgmap v880: 97 pgs: 97 active+clean; 615 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 782 op/s 2023-07-29T16:32:27.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:26 smithi039 ceph-mon[128858]: pgmap v880: 97 pgs: 97 active+clean; 615 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 782 op/s 2023-07-29T16:32:27.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:26 smithi156 ceph-mon[129245]: pgmap v880: 97 pgs: 97 active+clean; 615 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 782 op/s 2023-07-29T16:32:27.243 INFO:tasks.workunit.client.0.smithi031.stdout: 10 84646 31.63 MB/sec execute 11 sec latency 58.625 ms 2023-07-29T16:32:28.243 INFO:tasks.workunit.client.0.smithi031.stdout: 10 85400 32.47 MB/sec execute 12 sec latency 59.556 ms 2023-07-29T16:32:28.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:28 smithi031 ceph-mon[95504]: pgmap v881: 97 pgs: 97 active+clean; 618 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 15 MiB/s wr, 891 op/s 2023-07-29T16:32:28.526 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:28 smithi039 ceph-mon[128858]: pgmap v881: 97 pgs: 97 active+clean; 618 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 15 MiB/s wr, 891 op/s 2023-07-29T16:32:28.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:28 smithi156 ceph-mon[129245]: pgmap v881: 97 pgs: 97 active+clean; 618 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 15 MiB/s wr, 891 op/s 2023-07-29T16:32:29.243 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86023 32.02 MB/sec execute 13 sec latency 72.091 ms 2023-07-29T16:32:30.243 INFO:tasks.workunit.client.0.smithi031.stdout: 10 86738 32.68 MB/sec execute 14 sec latency 118.943 ms 2023-07-29T16:32:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 -- 172.21.15.31:0/1252545734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35280719e0 msgr2=0x7f3528071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:30.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 --2- 172.21.15.31:0/1252545734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35280719e0 0x7f3528071df0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3518009990 tx=0x7f3518009ca0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 -- 172.21.15.31:0/1252545734 shutdown_connections 2023-07-29T16:32:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 --2- 172.21.15.31:0/1252545734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f352810d8c0 0x7f352810fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 --2- 172.21.15.31:0/1252545734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f35280719e0 0x7f3528071df0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 --2- 172.21.15.31:0/1252545734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810afa0 0x7f352810d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 -- 172.21.15.31:0/1252545734 >> 172.21.15.31:0/1252545734 conn(0x7f352806d290 msgr2=0x7f352806f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 -- 172.21.15.31:0/1252545734 shutdown_connections 2023-07-29T16:32:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.703+0000 7f352cfff700 1 -- 172.21.15.31:0/1252545734 wait complete. 2023-07-29T16:32:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 Processor -- start 2023-07-29T16:32:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 -- start start 2023-07-29T16:32:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35280719e0 0x7f352810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f352810afa0 0x7f352810e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 0x7f352810e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352810f9f0 con 0x7f35280719e0 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352810fb60 con 0x7f352810d8c0 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.705+0000 7f352cfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f352810fd20 con 0x7f352810afa0 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.706+0000 7f3526d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 0x7f352810e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.706+0000 7f3526d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 0x7f352810e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50116/0 (socket says 172.21.15.31:50116) 2023-07-29T16:32:30.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.706+0000 7f3526d9d700 1 -- 172.21.15.31:0/2464192530 learned_addr learned my addr 172.21.15.31:0/2464192530 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.706+0000 7f352659c700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35280719e0 0x7f352810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.706+0000 7f3525d9b700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f352810afa0 0x7f352810e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3526d9d700 1 -- 172.21.15.31:0/2464192530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f352810afa0 msgr2=0x7f352810e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3526d9d700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f352810afa0 0x7f352810e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3526d9d700 1 -- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35280719e0 msgr2=0x7f352810de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3526d9d700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35280719e0 0x7f352810de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3526d9d700 1 -- 172.21.15.31:0/2464192530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180095f0 con 0x7f352810d8c0 2023-07-29T16:32:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3525d9b700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f352810afa0 0x7f352810e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:32:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f3526d9d700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 0x7f352810e8d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f352000ef60 tx=0x7f352000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f35177fe700 1 -- 172.21.15.31:0/2464192530 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520018070 con 0x7f352810d8c0 2023-07-29T16:32:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f35177fe700 1 -- 172.21.15.31:0/2464192530 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f352000f040 con 0x7f352810d8c0 2023-07-29T16:32:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.707+0000 7f35177fe700 1 -- 172.21.15.31:0/2464192530 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520007c80 con 0x7f352810d8c0 2023-07-29T16:32:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.708+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35281ad220 con 0x7f352810d8c0 2023-07-29T16:32:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.708+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35281ad6b0 con 0x7f352810d8c0 2023-07-29T16:32:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.714+0000 7f354088db80 1 Processor -- start 2023-07-29T16:32:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.714+0000 7f35177fe700 1 -- 172.21.15.31:0/2464192530 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f352000c9e0 con 0x7f352810d8c0 2023-07-29T16:32:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 -- start start 2023-07-29T16:32:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 0x55644b0f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f8df0 0x55644b0f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b0fa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b0509b0 con 0x55644b0f8df0 2023-07-29T16:32:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b04e370 con 0x55644af546b0 2023-07-29T16:32:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.715+0000 7f354088db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b049c60 con 0x55644b0f9cf0 2023-07-29T16:32:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 0x55644b0f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 0x55644b0f88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50120/0 (socket says 172.21.15.31:50120) 2023-07-29T16:32:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 -- 172.21.15.31:0/501325426 learned_addr learned my addr 172.21.15.31:0/501325426 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b0fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 -- 172.21.15.31:0/501325426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 msgr2=0x55644b0fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b0fa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 -- 172.21.15.31:0/501325426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f8df0 msgr2=0x55644b0f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f8df0 0x55644b0f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.716+0000 7f3525d9b700 1 -- 172.21.15.31:0/501325426 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55644ae77d80 con 0x55644af546b0 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.717+0000 7f35177fe700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35100a0960 0x7f35100a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.717+0000 7f35177fe700 1 -- 172.21.15.31:0/2464192530 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f35200cbc30 con 0x7f352810d8c0 2023-07-29T16:32:30.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.718+0000 7f352659c700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35100a0960 0x7f35100a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.719+0000 7f352659c700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35100a0960 0x7f35100a2e10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f351800b2e0 tx=0x7f35180058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:30.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.720+0000 7f3525d9b700 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 0x55644b0f88b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f351c0095f0 tx=0x7f351c0097b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d36b25ee71656bcb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:30.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.721+0000 7f3514ff9700 1 -- 172.21.15.31:0/501325426 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c00f450 con 0x55644af546b0 2023-07-29T16:32:30.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.721+0000 7f3514ff9700 1 -- 172.21.15.31:0/501325426 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f351c00fa70 con 0x55644af546b0 2023-07-29T16:32:30.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.721+0000 7f3514ff9700 1 -- 172.21.15.31:0/501325426 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c01e920 con 0x55644af546b0 2023-07-29T16:32:30.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 -- 172.21.15.31:0/501325426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 msgr2=0x55644b0f88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:30.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 0x55644b0f88b0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f351c0095f0 tx=0x7f351c0097b0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 -- 172.21.15.31:0/501325426 shutdown_connections 2023-07-29T16:32:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b0fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644af546b0 0x55644b0f88b0 secure :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f351c0095f0 tx=0x7f351c0097b0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 --2- 172.21.15.31:0/501325426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f8df0 0x55644b0f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.722+0000 7f354088db80 1 -- 172.21.15.31:0/501325426 >> 172.21.15.31:0/501325426 conn(0x55644b0edfd0 msgr2=0x55644b0f3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.723+0000 7f354088db80 1 -- 172.21.15.31:0/501325426 shutdown_connections 2023-07-29T16:32:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.723+0000 7f354088db80 1 -- 172.21.15.31:0/501325426 wait complete. 2023-07-29T16:32:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.724+0000 7f354088db80 1 -- start start 2023-07-29T16:32:30.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.724+0000 7f354088db80 1 Processor -- start 2023-07-29T16:32:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b0f8df0 0x55644b1021b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b1026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b107720 0x55644b109b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644afe64b0 con 0x55644b107720 2023-07-29T16:32:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644afe6150 con 0x55644b0f8df0 2023-07-29T16:32:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644aecd8c0 con 0x55644b0f9cf0 2023-07-29T16:32:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f354088db80 10 client.0 _ll_register_callbacks cb 0x55644afef5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f352659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b0f8df0 0x55644b1021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f352659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b0f8df0 0x55644b1021b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50122/0 (socket says 172.21.15.31:50122) 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f352659c700 1 -- 172.21.15.31:0/1129751733 learned_addr learned my addr 172.21.15.31:0/1129751733 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.725+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b107720 0x55644b109b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3525d9b700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b1026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3526d9d700 1 -- 172.21.15.31:0/1129751733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 msgr2=0x55644b1026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b1026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3526d9d700 1 -- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b0f8df0 msgr2=0x55644b1021b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b0f8df0 0x55644b1021b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3526d9d700 1 -- 172.21.15.31:0/1129751733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c01a040 con 0x55644b107720 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b107720 0x55644b109b00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3520099060 tx=0x7f3520098cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36516784db81b676 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f350dffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f350dffb700 1 -- 172.21.15.31:0/1129751733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200b0070 con 0x55644b107720 2023-07-29T16:32:30.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.726+0000 7f350dffb700 1 -- 172.21.15.31:0/1129751733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f3520099360 con 0x55644b107720 2023-07-29T16:32:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.727+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55644b074cb0 con 0x55644b107720 2023-07-29T16:32:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.727+0000 7f350dffb700 1 -- 172.21.15.31:0/1129751733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f352009d1b0 con 0x55644b107720 2023-07-29T16:32:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.727+0000 7f354088db80 10 client.19989 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.727+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55644ae60c00 con 0x55644b107720 2023-07-29T16:32:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f350dffb700 1 -- 172.21.15.31:0/1129751733 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55644b074cb0 con 0x55644b107720 2023-07-29T16:32:30.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f350dffb700 1 -- 172.21.15.31:0/1129751733 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3520098850 con 0x55644b107720 2023-07-29T16:32:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f354088db80 10 client.19989 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f354088db80 10 client.19989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f354088db80 20 client.19989 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f354088db80 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b10e160 0x55644b110570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f354088db80 4 client.19989 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:32:30.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.728+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55644b0c7d10 con 0x55644b10e160 2023-07-29T16:32:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.730+0000 7f352659c700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b10e160 0x55644b110570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.731+0000 7f352659c700 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b10e160 0x55644b110570 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e40575e034e96ec7 server_cookie=b418768b13be2b1a in_seq=0 out_seq=0 2023-07-29T16:32:30.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.731+0000 7f350dffb700 10 client.19989 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.736+0000 7f350dffb700 1 -- 172.21.15.31:0/1129751733 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55644b0c7d10 con 0x55644b10e160 2023-07-29T16:32:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.736+0000 7f350dffb700 10 client.19989 handle_command_reply: tid=0 2023-07-29T16:32:30.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.736+0000 7f354088db80 1 client.19989 shutdown 2023-07-29T16:32:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b107720 msgr2=0x55644b109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b107720 0x55644b109b00 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3520099060 tx=0x7f3520098cc0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 shutdown_connections 2023-07-29T16:32:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b10e160 0x55644b110570 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b107720 0x55644b109b00 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f350dffb700 0 client.19989 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:30.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b0f8df0 0x55644b1021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 --2- 172.21.15.31:0/1129751733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f9cf0 0x55644b1026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 >> 172.21.15.31:0/1129751733 conn(0x55644b0edfd0 msgr2=0x55644b0f2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 shutdown_connections 2023-07-29T16:32:30.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 1 -- 172.21.15.31:0/1129751733 wait complete. 2023-07-29T16:32:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.737+0000 7f354088db80 20 client.19989 trim_cache size 0 max 16384 2023-07-29T16:32:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 Processor -- start 2023-07-29T16:32:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 -- start start 2023-07-29T16:32:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b0fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b100c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b0c7d10 con 0x55644b0f9cf0 2023-07-29T16:32:30.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b074cb0 con 0x55644b107720 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.760+0000 7f354088db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b0f47a0 con 0x55644b0f8df0 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f3526d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b100c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35176/0 (socket says 172.21.15.31:35176) 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 -- 172.21.15.31:0/4133100503 learned_addr learned my addr 172.21.15.31:0/4133100503 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f3525d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b0fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 -- 172.21.15.31:0/4133100503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 msgr2=0x55644b100c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b100c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 -- 172.21.15.31:0/4133100503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 msgr2=0x55644b0fa0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b0fa0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.761+0000 7f352659c700 1 -- 172.21.15.31:0/4133100503 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55644ae9e1f0 con 0x55644b0f8df0 2023-07-29T16:32:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.762+0000 7f352659c700 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11dad0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3518000c00 tx=0x7f3518023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9f7f20ee28d1568 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.762+0000 7f35157fa700 1 -- 172.21.15.31:0/4133100503 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351802b070 con 0x55644b0f8df0 2023-07-29T16:32:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.762+0000 7f35157fa700 1 -- 172.21.15.31:0/4133100503 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f351802c040 con 0x55644b0f8df0 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.762+0000 7f35157fa700 1 -- 172.21.15.31:0/4133100503 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3518032e20 con 0x55644b0f8df0 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 -- 172.21.15.31:0/4133100503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 msgr2=0x55644b11dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11dad0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3518000c00 tx=0x7f3518023000 comp rx=0 tx=0).stop 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 -- 172.21.15.31:0/4133100503 shutdown_connections 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11dad0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 --2- 172.21.15.31:0/4133100503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b0fa0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 -- 172.21.15.31:0/4133100503 >> 172.21.15.31:0/4133100503 conn(0x55644b0edfd0 msgr2=0x55644b10a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 -- 172.21.15.31:0/4133100503 shutdown_connections 2023-07-29T16:32:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.763+0000 7f354088db80 1 -- 172.21.15.31:0/4133100503 wait complete. 2023-07-29T16:32:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.764+0000 7f354088db80 1 -- start start 2023-07-29T16:32:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.764+0000 7f354088db80 1 Processor -- start 2023-07-29T16:32:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b11c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b11c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b11cf70 con 0x55644b0f9cf0 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b11d0e0 con 0x55644b107720 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55644b11d2a0 con 0x55644b0f8df0 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f354088db80 10 client.0 _ll_register_callbacks cb 0x55644afef5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f3526d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b11c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f3526d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b11c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50140/0 (socket says 172.21.15.31:50140) 2023-07-29T16:32:30.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.765+0000 7f3526d9d700 1 -- 172.21.15.31:0/1455383328 learned_addr learned my addr 172.21.15.31:0/1455383328 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f352659c700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3525d9b700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b11c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3526d9d700 1 -- 172.21.15.31:0/1455383328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 msgr2=0x55644b11bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3526d9d700 1 -- 172.21.15.31:0/1455383328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 msgr2=0x55644b11c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b11c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3526d9d700 1 -- 172.21.15.31:0/1455383328 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3528110790 con 0x55644b107720 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f3526d9d700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b11c860 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f35200a12b0 tx=0x7f35200957b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=166796387bd0ae77 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.766+0000 7f350dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f350dffb700 1 -- 172.21.15.31:0/1455383328 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35200b0060 con 0x55644b107720 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f350dffb700 1 -- 172.21.15.31:0/1455383328 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f35200a1720 con 0x55644b107720 2023-07-29T16:32:30.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f351c00fa70 con 0x55644b107720 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f350dffb700 1 -- 172.21.15.31:0/1455383328 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f352009d1b0 con 0x55644b107720 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f354088db80 10 client.30203 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55644ae76250 con 0x55644b107720 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.767+0000 7f350dffb700 1 -- 172.21.15.31:0/1455383328 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f351c00fa70 con 0x55644b107720 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f350dffb700 1 -- 172.21.15.31:0/1455383328 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f35200c9b30 con 0x55644b107720 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f354088db80 10 client.30203 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:30.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f354088db80 10 client.30203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f354088db80 20 client.30203 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f354088db80 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b171940 0x55644b173d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f354088db80 4 client.30203 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55644b042ce0 con 0x55644b171940 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.768+0000 7f352659c700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b171940 0x55644b173d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.769+0000 7f352659c700 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b171940 0x55644b173d20 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ad05f54c1051f43 server_cookie=b9ab5220d68f6c2e in_seq=0 out_seq=0 2023-07-29T16:32:30.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:30.769+0000 7f350dffb700 10 client.30203 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:30.942 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:30 smithi031 ceph-mon[95504]: pgmap v882: 97 pgs: 97 active+clean; 580 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 22 MiB/s wr, 1.16k op/s 2023-07-29T16:32:31.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:30 smithi039 ceph-mon[128858]: pgmap v882: 97 pgs: 97 active+clean; 580 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 22 MiB/s wr, 1.16k op/s 2023-07-29T16:32:31.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:30 smithi156 ceph-mon[129245]: pgmap v882: 97 pgs: 97 active+clean; 580 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 22 MiB/s wr, 1.16k op/s 2023-07-29T16:32:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.111+0000 7f350dffb700 1 -- 172.21.15.31:0/1455383328 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55644b042ce0 con 0x55644b171940 2023-07-29T16:32:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.111+0000 7f350dffb700 10 client.30203 handle_command_reply: tid=0 2023-07-29T16:32:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.111+0000 7f354088db80 1 client.30203 shutdown 2023-07-29T16:32:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 msgr2=0x55644b11c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b11c860 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f35200a12b0 tx=0x7f35200957b0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 shutdown_connections 2023-07-29T16:32:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55644b171940 0x55644b173d20 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55644b107720 0x55644b11c860 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55644b0f8df0 0x55644b11bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 --2- 172.21.15.31:0/1455383328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55644b0f9cf0 0x55644b11c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 >> 172.21.15.31:0/1455383328 conn(0x55644b0edfd0 msgr2=0x55644b100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f350dffb700 0 client.30203 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 shutdown_connections 2023-07-29T16:32:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.112+0000 7f354088db80 1 -- 172.21.15.31:0/1455383328 wait complete. 2023-07-29T16:32:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.113+0000 7f354088db80 20 client.30203 trim_cache size 0 max 16384 2023-07-29T16:32:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:32:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:32:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:32:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:32:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.117+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35100a0960 msgr2=0x7f35100a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.117+0000 7f352cfff700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35100a0960 0x7f35100a2e10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f351800b2e0 tx=0x7f35180058e0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.117+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 msgr2=0x7f352810e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.117+0000 7f352cfff700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 0x7f352810e8d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f352000ef60 tx=0x7f352000ef90 comp rx=0 tx=0).stop 2023-07-29T16:32:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.121+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 shutdown_connections 2023-07-29T16:32:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.121+0000 7f352cfff700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f35100a0960 0x7f35100a2e10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.121+0000 7f352cfff700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f352810d8c0 0x7f352810e8d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.121+0000 7f352cfff700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f35280719e0 0x7f352810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.121+0000 7f352cfff700 1 --2- 172.21.15.31:0/2464192530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f352810afa0 0x7f352810e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.121+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 >> 172.21.15.31:0/2464192530 conn(0x7f352806d290 msgr2=0x7f352806f670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.122+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 shutdown_connections 2023-07-29T16:32:31.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:31.122+0000 7f352cfff700 1 -- 172.21.15.31:0/2464192530 wait complete. 2023-07-29T16:32:31.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 87228 31.63 MB/sec execute 15 sec latency 311.358 ms 2023-07-29T16:32:32.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 88389 33.27 MB/sec execute 16 sec latency 86.346 ms 2023-07-29T16:32:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:32 smithi031 ceph-mon[95504]: pgmap v883: 97 pgs: 97 active+clean; 563 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 18 MiB/s wr, 878 op/s 2023-07-29T16:32:33.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:32 smithi039 ceph-mon[128858]: pgmap v883: 97 pgs: 97 active+clean; 563 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 18 MiB/s wr, 878 op/s 2023-07-29T16:32:33.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:32 smithi156 ceph-mon[129245]: pgmap v883: 97 pgs: 97 active+clean; 563 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 18 MiB/s wr, 878 op/s 2023-07-29T16:32:33.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 89415 35.64 MB/sec execute 17 sec latency 63.507 ms 2023-07-29T16:32:34.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90093 35.58 MB/sec execute 18 sec latency 61.939 ms 2023-07-29T16:32:34.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:34 smithi031 ceph-mon[95504]: pgmap v884: 97 pgs: 97 active+clean; 564 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 19 MiB/s wr, 1.10k op/s 2023-07-29T16:32:35.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:34 smithi039 ceph-mon[128858]: pgmap v884: 97 pgs: 97 active+clean; 564 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 19 MiB/s wr, 1.10k op/s 2023-07-29T16:32:35.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:34 smithi156 ceph-mon[129245]: pgmap v884: 97 pgs: 97 active+clean; 564 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 19 MiB/s wr, 1.10k op/s 2023-07-29T16:32:35.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 90770 35.44 MB/sec execute 19 sec latency 150.980 ms 2023-07-29T16:32:36.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 91525 35.41 MB/sec execute 20 sec latency 65.516 ms 2023-07-29T16:32:36.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:36 smithi031 ceph-mon[95504]: pgmap v885: 97 pgs: 97 active+clean; 569 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 17 MiB/s wr, 1.00k op/s 2023-07-29T16:32:37.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:36 smithi039 ceph-mon[128858]: pgmap v885: 97 pgs: 97 active+clean; 569 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 17 MiB/s wr, 1.00k op/s 2023-07-29T16:32:37.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:36 smithi156 ceph-mon[129245]: pgmap v885: 97 pgs: 97 active+clean; 569 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 17 MiB/s wr, 1.00k op/s 2023-07-29T16:32:37.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 92438 36.27 MB/sec execute 21 sec latency 89.204 ms 2023-07-29T16:32:38.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 93382 36.74 MB/sec execute 22 sec latency 51.719 ms 2023-07-29T16:32:38.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:38 smithi031 ceph-mon[95504]: pgmap v886: 97 pgs: 97 active+clean; 575 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 932 op/s 2023-07-29T16:32:38.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:38 smithi039 ceph-mon[128858]: pgmap v886: 97 pgs: 97 active+clean; 575 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 932 op/s 2023-07-29T16:32:38.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:38 smithi156 ceph-mon[129245]: pgmap v886: 97 pgs: 97 active+clean; 575 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 14 MiB/s wr, 932 op/s 2023-07-29T16:32:39.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 94182 37.05 MB/sec execute 23 sec latency 51.928 ms 2023-07-29T16:32:40.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95054 37.56 MB/sec execute 24 sec latency 78.386 ms 2023-07-29T16:32:40.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:40 smithi031 ceph-mon[95504]: pgmap v887: 97 pgs: 97 active+clean; 580 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 842 op/s 2023-07-29T16:32:41.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:40 smithi039 ceph-mon[128858]: pgmap v887: 97 pgs: 97 active+clean; 580 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 842 op/s 2023-07-29T16:32:41.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:40 smithi156 ceph-mon[129245]: pgmap v887: 97 pgs: 97 active+clean; 580 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 11 MiB/s wr, 842 op/s 2023-07-29T16:32:41.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 95721 37.62 MB/sec execute 25 sec latency 100.955 ms 2023-07-29T16:32:42.244 INFO:tasks.workunit.client.0.smithi031.stdout: 10 96430 37.57 MB/sec execute 26 sec latency 72.140 ms 2023-07-29T16:32:42.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:42 smithi031 ceph-mon[95504]: pgmap v888: 97 pgs: 97 active+clean; 583 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.6 MiB/s wr, 585 op/s 2023-07-29T16:32:43.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:42 smithi039 ceph-mon[128858]: pgmap v888: 97 pgs: 97 active+clean; 583 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.6 MiB/s wr, 585 op/s 2023-07-29T16:32:43.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:42 smithi156 ceph-mon[129245]: pgmap v888: 97 pgs: 97 active+clean; 583 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.6 MiB/s wr, 585 op/s 2023-07-29T16:32:43.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 97434 37.92 MB/sec execute 27 sec latency 45.021 ms 2023-07-29T16:32:44.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98287 38.37 MB/sec execute 28 sec latency 67.256 ms 2023-07-29T16:32:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 -- 172.21.15.31:0/383860740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8101570 msgr2=0x7fd3a81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 --2- 172.21.15.31:0/383860740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8101570 0x7fd3a81019c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0098f0 tx=0x7fd39c009c00 comp rx=0 tx=0).stop 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 -- 172.21.15.31:0/383860740 shutdown_connections 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 --2- 172.21.15.31:0/383860740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3a8101f00 0x7fd3a81087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 --2- 172.21.15.31:0/383860740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8100380 0x7fd3a8100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 --2- 172.21.15.31:0/383860740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8101570 0x7fd3a81019c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 -- 172.21.15.31:0/383860740 >> 172.21.15.31:0/383860740 conn(0x7fd3a80fb9a0 msgr2=0x7fd3a80fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 -- 172.21.15.31:0/383860740 shutdown_connections 2023-07-29T16:32:44.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.357+0000 7fd3adfff700 1 -- 172.21.15.31:0/383860740 wait complete. 2023-07-29T16:32:44.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.358+0000 7fd3adfff700 1 Processor -- start 2023-07-29T16:32:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.358+0000 7fd3adfff700 1 -- start start 2023-07-29T16:32:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3adfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8100380 0x7fd3a806f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3adfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 0x7fd3a806fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3adfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3a8101f00 0x7fd3a8070a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3adfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80710e0 con 0x7fd3a8100380 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3adfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8071250 con 0x7fd3a8101f00 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3adfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8071410 con 0x7fd3a8101570 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 0x7fd3a806fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3a77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8100380 0x7fd3a806f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3a8101f00 0x7fd3a8070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 0x7fd3a806fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42724/0 (socket says 172.21.15.31:42724) 2023-07-29T16:32:44.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.359+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/794823551 learned_addr learned my addr 172.21.15.31:0/794823551 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/794823551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3a8101f00 msgr2=0x7fd3a8070a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3a8101f00 0x7fd3a8070a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8100380 msgr2=0x7fd3a806f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8100380 0x7fd3a806f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/794823551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c009520 con 0x7fd3a8101570 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 0x7fd3a806fbb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005e90 tx=0x7fd39c00b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.360+0000 7fd3a4ff9700 1 -- 172.21.15.31:0/794823551 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c01c070 con 0x7fd3a8101570 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.361+0000 7fd3a4ff9700 1 -- 172.21.15.31:0/794823551 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd39c00f7f0 con 0x7fd3a8101570 2023-07-29T16:32:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.361+0000 7fd3a4ff9700 1 -- 172.21.15.31:0/794823551 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c004a40 con 0x7fd3a8101570 2023-07-29T16:32:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.361+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8071690 con 0x7fd3a8101570 2023-07-29T16:32:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.362+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8071ea0 con 0x7fd3a8101570 2023-07-29T16:32:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.368+0000 7fd3c1997b80 1 Processor -- start 2023-07-29T16:32:44.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3a4ff9700 1 -- 172.21.15.31:0/794823551 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd39c020070 con 0x7fd3a8101570 2023-07-29T16:32:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 -- start start 2023-07-29T16:32:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab396c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab396d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab38c39b0 con 0x562ab396ccf0 2023-07-29T16:32:44.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab38c1370 con 0x562ab396bdf0 2023-07-29T16:32:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.369+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab38bcc60 con 0x562ab37c76b0 2023-07-29T16:32:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42728/0 (socket says 172.21.15.31:42728) 2023-07-29T16:32:44.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 -- 172.21.15.31:0/809325667 learned_addr learned my addr 172.21.15.31:0/809325667 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab396c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 -- 172.21.15.31:0/809325667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 msgr2=0x562ab396c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab396c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 -- 172.21.15.31:0/809325667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 msgr2=0x562ab396d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab396d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 -- 172.21.15.31:0/809325667 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ab36ead80 con 0x562ab37c76b0 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.370+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396b8b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0011680 tx=0x7fd3a0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d639690d4d70ee5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.371+0000 7fd38dffb700 1 -- 172.21.15.31:0/809325667 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000e040 con 0x562ab37c76b0 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.371+0000 7fd38dffb700 1 -- 172.21.15.31:0/809325667 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd3a000f040 con 0x562ab37c76b0 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.371+0000 7fd38dffb700 1 -- 172.21.15.31:0/809325667 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00048c0 con 0x562ab37c76b0 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.371+0000 7fd3a4ff9700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3900a07f0 0x7fd3900a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.371+0000 7fd3a4ff9700 1 -- 172.21.15.31:0/794823551 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd39c0cc7a0 con 0x7fd3a8101570 2023-07-29T16:32:44.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.372+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3900a07f0 0x7fd3900a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.372+0000 7fd3c1997b80 1 -- 172.21.15.31:0/809325667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 msgr2=0x562ab396b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396b8b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0011680 tx=0x7fd3a0009920 comp rx=0 tx=0).stop 2023-07-29T16:32:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3c1997b80 1 -- 172.21.15.31:0/809325667 shutdown_connections 2023-07-29T16:32:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab396d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396b8b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/809325667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab396c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3c1997b80 1 -- 172.21.15.31:0/809325667 >> 172.21.15.31:0/809325667 conn(0x562ab3960fd0 msgr2=0x562ab3966300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.373+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3900a07f0 0x7fd3900a2ca0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd3a806f3e0 tx=0x7fd398009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.374+0000 7fd3c1997b80 1 -- 172.21.15.31:0/809325667 shutdown_connections 2023-07-29T16:32:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.375+0000 7fd3c1997b80 1 -- 172.21.15.31:0/809325667 wait complete. 2023-07-29T16:32:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.376+0000 7fd3c1997b80 1 -- start start 2023-07-29T16:32:44.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.376+0000 7fd3c1997b80 1 Processor -- start 2023-07-29T16:32:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab37c76b0 0x562ab3974f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 0x562ab3975450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab397c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab38594b0 con 0x562ab396ccf0 2023-07-29T16:32:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab3859150 con 0x562ab37c76b0 2023-07-29T16:32:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab37408c0 con 0x562ab396bdf0 2023-07-29T16:32:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3c1997b80 10 client.0 _ll_register_callbacks cb 0x562ab38625c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 0x562ab3975450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 0x562ab3975450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42734/0 (socket says 172.21.15.31:42734) 2023-07-29T16:32:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.377+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/1184070920 learned_addr learned my addr 172.21.15.31:0/1184070920 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab397c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab37c76b0 msgr2=0x562ab3974f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab37c76b0 0x562ab3974f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/1184070920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 msgr2=0x562ab397c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab397c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a6ffd700 1 -- 172.21.15.31:0/1184070920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a0011330 con 0x562ab396bdf0 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 0x562ab3975450 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd39c099bd0 tx=0x7fd39c0998b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61301b9db60b3ff6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.378+0000 7fd386ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.379+0000 7fd386ffd700 1 -- 172.21.15.31:0/1184070920 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0a2290 con 0x562ab396bdf0 2023-07-29T16:32:44.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.379+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ab38e7cb0 con 0x562ab396bdf0 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.379+0000 7fd386ffd700 1 -- 172.21.15.31:0/1184070920 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd39c0993c0 con 0x562ab396bdf0 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.379+0000 7fd386ffd700 1 -- 172.21.15.31:0/1184070920 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd39c099580 con 0x562ab396bdf0 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.379+0000 7fd3c1997b80 10 client.30229 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.379+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x562ab36d3c00 con 0x562ab396bdf0 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd386ffd700 1 -- 172.21.15.31:0/1184070920 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ab38e7cb0 con 0x562ab396bdf0 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd386ffd700 1 -- 172.21.15.31:0/1184070920 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd39c004ba0 con 0x562ab396bdf0 2023-07-29T16:32:44.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd3c1997b80 10 client.30229 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd3c1997b80 10 client.30229 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd3c1997b80 20 client.30229 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab3980940 0x562ab3982d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd3c1997b80 4 client.30229 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:32:44.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.380+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ab393ad10 con 0x562ab3980940 2023-07-29T16:32:44.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.381+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab3980940 0x562ab3982d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.381+0000 7fd386ffd700 10 client.30229 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:44.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.381+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab3980940 0x562ab3982d50 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ccf1748ecc8cdf5 server_cookie=e227689b09ef47eb in_seq=0 out_seq=0 2023-07-29T16:32:44.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.387+0000 7fd386ffd700 1 -- 172.21.15.31:0/1184070920 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562ab393ad10 con 0x562ab3980940 2023-07-29T16:32:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.387+0000 7fd386ffd700 10 client.30229 handle_command_reply: tid=0 2023-07-29T16:32:44.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.387+0000 7fd3c1997b80 1 client.30229 shutdown 2023-07-29T16:32:44.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 msgr2=0x562ab3975450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 0x562ab3975450 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd39c099bd0 tx=0x7fd39c0998b0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 shutdown_connections 2023-07-29T16:32:44.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab3980940 0x562ab3982d50 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab397c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab37c76b0 0x562ab3974f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1184070920 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab396bdf0 0x562ab3975450 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 >> 172.21.15.31:0/1184070920 conn(0x562ab3960fd0 msgr2=0x562ab3965a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:44.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd386ffd700 0 client.30229 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:44.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 shutdown_connections 2023-07-29T16:32:44.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.388+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1184070920 wait complete. 2023-07-29T16:32:44.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.389+0000 7fd3c1997b80 20 client.30229 trim_cache size 0 max 16384 2023-07-29T16:32:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 Processor -- start 2023-07-29T16:32:44.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 -- start start 2023-07-29T16:32:44.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab397c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab39700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab38e7cb0 con 0x562ab396ccf0 2023-07-29T16:32:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab396c1c0 con 0x562ab396bdf0 2023-07-29T16:32:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.407+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab3976100 con 0x562ab37c76b0 2023-07-29T16:32:44.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab397c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42744/0 (socket says 172.21.15.31:42744) 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a6ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab397c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35678/0 (socket says 172.21.15.31:35678) 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 -- 172.21.15.31:0/3595509497 learned_addr learned my addr 172.21.15.31:0/3595509497 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab39700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 -- 172.21.15.31:0/3595509497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 msgr2=0x562ab397c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab397c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 -- 172.21.15.31:0/3595509497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 msgr2=0x562ab39700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab39700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a77fe700 1 -- 172.21.15.31:0/3595509497 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ab37111f0 con 0x562ab37c76b0 2023-07-29T16:32:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a6ffd700 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab397c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:32:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.408+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab39700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:32:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.409+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396fba0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd398012cd0 tx=0x7fd398004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1389ec4539c738f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.409+0000 7fd38e7fc700 1 -- 172.21.15.31:0/3595509497 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398023040 con 0x562ab37c76b0 2023-07-29T16:32:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.409+0000 7fd38e7fc700 1 -- 172.21.15.31:0/3595509497 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd398028470 con 0x562ab37c76b0 2023-07-29T16:32:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.409+0000 7fd38e7fc700 1 -- 172.21.15.31:0/3595509497 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398017070 con 0x562ab37c76b0 2023-07-29T16:32:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 -- 172.21.15.31:0/3595509497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 msgr2=0x562ab396fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396fba0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fd398012cd0 tx=0x7fd398004750 comp rx=0 tx=0).stop 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 -- 172.21.15.31:0/3595509497 shutdown_connections 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396ccf0 0x562ab39700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab396fba0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/3595509497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396bdf0 0x562ab397c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 -- 172.21.15.31:0/3595509497 >> 172.21.15.31:0/3595509497 conn(0x562ab3960fd0 msgr2=0x562ab3982950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.410+0000 7fd3c1997b80 1 -- 172.21.15.31:0/3595509497 shutdown_connections 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.411+0000 7fd3c1997b80 1 -- 172.21.15.31:0/3595509497 wait complete. 2023-07-29T16:32:44.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.411+0000 7fd3c1997b80 1 -- start start 2023-07-29T16:32:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 Processor -- start 2023-07-29T16:32:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab397bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396bdf0 0x562ab397c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 0x562ab3978830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab3978ed0 con 0x562ab396bdf0 2023-07-29T16:32:44.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab3979040 con 0x562ab396ccf0 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ab3979200 con 0x562ab37c76b0 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.412+0000 7fd3c1997b80 10 client.0 _ll_register_callbacks cb 0x562ab38625c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.413+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 0x562ab3978830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.413+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 0x562ab3978830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35682/0 (socket says 172.21.15.31:35682) 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.413+0000 7fd3a7fff700 1 -- 172.21.15.31:0/1449206127 learned_addr learned my addr 172.21.15.31:0/1449206127 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.413+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab397bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.413+0000 7fd3a7fff700 1 -- 172.21.15.31:0/1449206127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 msgr2=0x562ab397bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:44.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab397bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd3a7fff700 1 -- 172.21.15.31:0/1449206127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396bdf0 msgr2=0x562ab397c2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:32:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396bdf0 0x562ab397c2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd3a7fff700 1 -- 172.21.15.31:0/1449206127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ab36d3c00 con 0x562ab396ccf0 2023-07-29T16:32:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab397bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:32:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd3a7fff700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 0x562ab3978830 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0011620 tx=0x7fd3a0007850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c1e966ae7d33b6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:44.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd386ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:32:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.414+0000 7fd386ffd700 1 -- 172.21.15.31:0/1449206127 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000e040 con 0x562ab396ccf0 2023-07-29T16:32:44.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.415+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ab38b5ce0 con 0x562ab396ccf0 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.415+0000 7fd386ffd700 1 -- 172.21.15.31:0/1449206127 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd3a0007d40 con 0x562ab396ccf0 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.415+0000 7fd386ffd700 1 -- 172.21.15.31:0/1449206127 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3a0027470 con 0x562ab396ccf0 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.415+0000 7fd3c1997b80 10 client.30218 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.415+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x562ab36ead80 con 0x562ab396ccf0 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.415+0000 7fd386ffd700 1 -- 172.21.15.31:0/1449206127 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562ab38b5ce0 con 0x562ab396ccf0 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd386ffd700 1 -- 172.21.15.31:0/1449206127 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd3a0015720 con 0x562ab396ccf0 2023-07-29T16:32:44.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3c1997b80 10 client.30218 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3c1997b80 10 client.30218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3c1997b80 20 client.30218 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab39d3a00 0x562ab39d5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3c1997b80 4 client.30218 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562ab38b5a00 con 0x562ab39d3a00 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.416+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab39d3a00 0x562ab39d5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.417+0000 7fd3a77fe700 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab39d3a00 0x562ab39d5e10 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae8225090a950fcb server_cookie=d3511a3b95e170f2 in_seq=0 out_seq=0 2023-07-29T16:32:44.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.417+0000 7fd386ffd700 10 client.30218 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.798+0000 7fd386ffd700 1 -- 172.21.15.31:0/1449206127 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562ab38b5a00 con 0x562ab39d3a00 2023-07-29T16:32:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.798+0000 7fd386ffd700 10 client.30218 handle_command_reply: tid=0 2023-07-29T16:32:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.799+0000 7fd3c1997b80 1 client.30218 shutdown 2023-07-29T16:32:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.800+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 msgr2=0x562ab3978830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.800+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 0x562ab3978830 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0011620 tx=0x7fd3a0007850 comp rx=0 tx=0).stop 2023-07-29T16:32:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 shutdown_connections 2023-07-29T16:32:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562ab39d3a00 0x562ab39d5e10 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562ab396ccf0 0x562ab3978830 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ab37c76b0 0x562ab397bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 --2- 172.21.15.31:0/1449206127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562ab396bdf0 0x562ab397c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 >> 172.21.15.31:0/1449206127 conn(0x562ab3960fd0 msgr2=0x562ab396fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd386ffd700 0 client.30218 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 shutdown_connections 2023-07-29T16:32:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.801+0000 7fd3c1997b80 1 -- 172.21.15.31:0/1449206127 wait complete. 2023-07-29T16:32:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.802+0000 7fd3c1997b80 20 client.30218 trim_cache size 0 max 16384 2023-07-29T16:32:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:32:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:32:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:32:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:32:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.804+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3900a07f0 msgr2=0x7fd3900a2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.804+0000 7fd3adfff700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3900a07f0 0x7fd3900a2ca0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd3a806f3e0 tx=0x7fd398009310 comp rx=0 tx=0).stop 2023-07-29T16:32:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.804+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 msgr2=0x7fd3a806fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.804+0000 7fd3adfff700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 0x7fd3a806fbb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fd39c005e90 tx=0x7fd39c00b730 comp rx=0 tx=0).stop 2023-07-29T16:32:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.806+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 shutdown_connections 2023-07-29T16:32:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.806+0000 7fd3adfff700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd3900a07f0 0x7fd3900a2ca0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.806+0000 7fd3adfff700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3a8101f00 0x7fd3a8070a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.806+0000 7fd3adfff700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd3a8100380 0x7fd3a806f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.806+0000 7fd3adfff700 1 --2- 172.21.15.31:0/794823551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3a8101570 0x7fd3a806fbb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.806+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 >> 172.21.15.31:0/794823551 conn(0x7fd3a80fb9a0 msgr2=0x7fd3a8104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.809+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 shutdown_connections 2023-07-29T16:32:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:44.809+0000 7fd3adfff700 1 -- 172.21.15.31:0/794823551 wait complete. 2023-07-29T16:32:45.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:44 smithi039 ceph-mon[128858]: pgmap v889: 97 pgs: 97 active+clean; 596 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.1 MiB/s wr, 688 op/s 2023-07-29T16:32:45.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:44 smithi156 ceph-mon[129245]: pgmap v889: 97 pgs: 97 active+clean; 596 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.1 MiB/s wr, 688 op/s 2023-07-29T16:32:45.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:44 smithi031 ceph-mon[95504]: pgmap v889: 97 pgs: 97 active+clean; 596 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.1 MiB/s wr, 688 op/s 2023-07-29T16:32:45.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 98889 38.29 MB/sec execute 29 sec latency 193.689 ms 2023-07-29T16:32:46.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 99583 38.40 MB/sec execute 30 sec latency 63.378 ms 2023-07-29T16:32:47.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:46 smithi039 ceph-mon[128858]: pgmap v890: 97 pgs: 97 active+clean; 603 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 4.5 MiB/s wr, 572 op/s 2023-07-29T16:32:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:46 smithi156 ceph-mon[129245]: pgmap v890: 97 pgs: 97 active+clean; 603 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 4.5 MiB/s wr, 572 op/s 2023-07-29T16:32:47.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:46 smithi031 ceph-mon[95504]: pgmap v890: 97 pgs: 97 active+clean; 603 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 4.5 MiB/s wr, 572 op/s 2023-07-29T16:32:47.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 100301 38.36 MB/sec execute 31 sec latency 65.044 ms 2023-07-29T16:32:48.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 101090 38.15 MB/sec execute 32 sec latency 75.503 ms 2023-07-29T16:32:48.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:48 smithi031 ceph-mon[95504]: pgmap v891: 97 pgs: 97 active+clean; 599 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.2 MiB/s wr, 551 op/s 2023-07-29T16:32:48.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:48 smithi039 ceph-mon[128858]: pgmap v891: 97 pgs: 97 active+clean; 599 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.2 MiB/s wr, 551 op/s 2023-07-29T16:32:48.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:48 smithi156 ceph-mon[129245]: pgmap v891: 97 pgs: 97 active+clean; 599 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.2 MiB/s wr, 551 op/s 2023-07-29T16:32:49.245 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102122 38.88 MB/sec execute 33 sec latency 48.088 ms 2023-07-29T16:32:50.247 INFO:tasks.workunit.client.0.smithi031.stdout: 10 102858 38.91 MB/sec execute 34 sec latency 100.823 ms 2023-07-29T16:32:50.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:50 smithi031 ceph-mon[95504]: pgmap v892: 97 pgs: 97 active+clean; 603 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.5 MiB/s wr, 659 op/s 2023-07-29T16:32:51.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:50 smithi039 ceph-mon[128858]: pgmap v892: 97 pgs: 97 active+clean; 603 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.5 MiB/s wr, 659 op/s 2023-07-29T16:32:51.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:50 smithi156 ceph-mon[129245]: pgmap v892: 97 pgs: 97 active+clean; 603 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.5 MiB/s wr, 659 op/s 2023-07-29T16:32:51.247 INFO:tasks.workunit.client.0.smithi031.stdout: 10 103580 38.95 MB/sec execute 35 sec latency 115.853 ms 2023-07-29T16:32:52.247 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104408 39.19 MB/sec execute 36 sec latency 50.021 ms 2023-07-29T16:32:52.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:52 smithi031 ceph-mon[95504]: pgmap v893: 97 pgs: 97 active+clean; 608 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 625 op/s 2023-07-29T16:32:53.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:52 smithi039 ceph-mon[128858]: pgmap v893: 97 pgs: 97 active+clean; 608 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 625 op/s 2023-07-29T16:32:53.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:52 smithi156 ceph-mon[129245]: pgmap v893: 97 pgs: 97 active+clean; 608 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 625 op/s 2023-07-29T16:32:53.247 INFO:tasks.workunit.client.0.smithi031.stdout: 10 104996 38.98 MB/sec execute 37 sec latency 94.695 ms 2023-07-29T16:32:54.247 INFO:tasks.workunit.client.0.smithi031.stdout: 10 105670 38.74 MB/sec execute 38 sec latency 85.181 ms 2023-07-29T16:32:54.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:54 smithi031 ceph-mon[95504]: pgmap v894: 97 pgs: 97 active+clean; 618 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.2 MiB/s wr, 714 op/s 2023-07-29T16:32:55.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:54 smithi039 ceph-mon[128858]: pgmap v894: 97 pgs: 97 active+clean; 618 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.2 MiB/s wr, 714 op/s 2023-07-29T16:32:55.078 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:54 smithi156 ceph-mon[129245]: pgmap v894: 97 pgs: 97 active+clean; 618 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 5.2 MiB/s wr, 714 op/s 2023-07-29T16:32:55.248 INFO:tasks.workunit.client.0.smithi031.stdout: 10 106352 38.71 MB/sec execute 39 sec latency 216.277 ms 2023-07-29T16:32:56.248 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107020 38.60 MB/sec execute 40 sec latency 231.761 ms 2023-07-29T16:32:56.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:56 smithi031 ceph-mon[95504]: pgmap v895: 97 pgs: 97 active+clean; 621 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.9 MiB/s wr, 660 op/s 2023-07-29T16:32:57.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:56 smithi039 ceph-mon[128858]: pgmap v895: 97 pgs: 97 active+clean; 621 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.9 MiB/s wr, 660 op/s 2023-07-29T16:32:57.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:56 smithi156 ceph-mon[129245]: pgmap v895: 97 pgs: 97 active+clean; 621 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 4.9 MiB/s wr, 660 op/s 2023-07-29T16:32:57.248 INFO:tasks.workunit.client.0.smithi031.stdout: 10 107841 38.78 MB/sec execute 41 sec latency 62.637 ms 2023-07-29T16:32:58.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 -- 172.21.15.31:0/804059646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 msgr2=0x7f8300108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 --2- 172.21.15.31:0/804059646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f8300108480 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f82f8009ec0 tx=0x7f82f800b330 comp rx=0 tx=0).stop 2023-07-29T16:32:58.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 -- 172.21.15.31:0/804059646 shutdown_connections 2023-07-29T16:32:58.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 --2- 172.21.15.31:0/804059646 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f8300108480 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 --2- 172.21.15.31:0/804059646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8300100e60 0x7f8300103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 --2- 172.21.15.31:0/804059646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 0x7f8300105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.032+0000 7f83065ff700 1 -- 172.21.15.31:0/804059646 >> 172.21.15.31:0/804059646 conn(0x7f83000fa840 msgr2=0x7f83000fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:58.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.035+0000 7f83065ff700 1 -- 172.21.15.31:0/804059646 shutdown_connections 2023-07-29T16:32:58.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.035+0000 7f83065ff700 1 -- 172.21.15.31:0/804059646 wait complete. 2023-07-29T16:32:58.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.036+0000 7f83065ff700 1 Processor -- start 2023-07-29T16:32:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 -- start start 2023-07-29T16:32:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8300100e60 0x7f83000fee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 0x7f83000ff350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f83000ff890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83001a62e0 con 0x7f8300100e60 2023-07-29T16:32:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83001a6450 con 0x7f83001060a0 2023-07-29T16:32:58.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.037+0000 7f83065ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83001a6610 con 0x7f8300103780 2023-07-29T16:32:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.038+0000 7f8304b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f83000ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.038+0000 7f8304b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f83000ff890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44754/0 (socket says 172.21.15.31:44754) 2023-07-29T16:32:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.038+0000 7f8304b9c700 1 -- 172.21.15.31:0/3319525522 learned_addr learned my addr 172.21.15.31:0/3319525522 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f8304b9c700 1 -- 172.21.15.31:0/3319525522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 msgr2=0x7f83000ff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 0x7f83000ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f8304b9c700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 0x7f83000ff350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f8304b9c700 1 -- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8300100e60 msgr2=0x7f83000fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f8304b9c700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8300100e60 0x7f83000fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f8304b9c700 1 -- 172.21.15.31:0/3319525522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f8005e70 con 0x7f83001060a0 2023-07-29T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 0x7f83000ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.039+0000 7f8304b9c700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f83000ff890 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f82f80077d0 tx=0x7f82f800b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.040+0000 7f82fd7fa700 1 -- 172.21.15.31:0/3319525522 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f800e040 con 0x7f83001060a0 2023-07-29T16:32:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.040+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8300071170 con 0x7f83001060a0 2023-07-29T16:32:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.040+0000 7f82fd7fa700 1 -- 172.21.15.31:0/3319525522 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82f801db40 con 0x7f83001060a0 2023-07-29T16:32:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.040+0000 7f82fd7fa700 1 -- 172.21.15.31:0/3319525522 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f801bc20 con 0x7f83001060a0 2023-07-29T16:32:58.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.040+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83000716d0 con 0x7f83001060a0 2023-07-29T16:32:58.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.046+0000 7f82fd7fa700 1 -- 172.21.15.31:0/3319525522 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f82f801d680 con 0x7f83001060a0 2023-07-29T16:32:58.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.047+0000 7f8319f03b80 1 Processor -- start 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 -- start start 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b534608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b534617b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b53462160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b533b89b0 con 0x558b53460df0 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b533b6370 con 0x558b532bc6b0 2023-07-29T16:32:58.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f8319f03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b533b1c60 con 0x558b53461cf0 2023-07-29T16:32:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b534608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b534608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44766/0 (socket says 172.21.15.31:44766) 2023-07-29T16:32:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.048+0000 7f82ff7fe700 1 -- 172.21.15.31:0/3403316505 learned_addr learned my addr 172.21.15.31:0/3403316505 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:58.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b53462160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82fd7fa700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82e80a0810 0x7f82e80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82fffff700 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b534617b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 -- 172.21.15.31:0/3403316505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 msgr2=0x558b53462160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b53462160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 -- 172.21.15.31:0/3403316505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 msgr2=0x558b534617b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b534617b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82fd7fa700 1 -- 172.21.15.31:0/3319525522 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f82f80cd800 con 0x7f83001060a0 2023-07-29T16:32:58.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 -- 172.21.15.31:0/3403316505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b531dfd80 con 0x558b532bc6b0 2023-07-29T16:32:58.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b534608b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f82f4015a10 tx=0x7f82f40094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26505ceb32c7c787 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:58.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.049+0000 7f8304b9c700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82e80a0810 0x7f82e80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.050+0000 7f82e67fc700 1 -- 172.21.15.31:0/3403316505 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4004a30 con 0x558b532bc6b0 2023-07-29T16:32:58.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.050+0000 7f82e67fc700 1 -- 172.21.15.31:0/3403316505 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82f40054e0 con 0x558b532bc6b0 2023-07-29T16:32:58.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.050+0000 7f82e67fc700 1 -- 172.21.15.31:0/3403316505 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f400c870 con 0x558b532bc6b0 2023-07-29T16:32:58.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 -- 172.21.15.31:0/3403316505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 msgr2=0x558b534608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8304b9c700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82e80a0810 0x7f82e80a2cc0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2dc0 tx=0x7f82f8009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b534608b0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f82f4015a10 tx=0x7f82f40094b0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 -- 172.21.15.31:0/3403316505 shutdown_connections 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b53462160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b534608b0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 --2- 172.21.15.31:0/3403316505 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b534617b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 -- 172.21.15.31:0/3403316505 >> 172.21.15.31:0/3403316505 conn(0x558b53455fd0 msgr2=0x558b5345b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:58.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 -- 172.21.15.31:0/3403316505 shutdown_connections 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.051+0000 7f8319f03b80 1 -- 172.21.15.31:0/3403316505 wait complete. 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.052+0000 7f8319f03b80 1 -- start start 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.052+0000 7f8319f03b80 1 Processor -- start 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 0x558b53469fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b5346a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53461cf0 0x558b53471520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5334e4b0 con 0x558b53460df0 2023-07-29T16:32:58.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5334e150 con 0x558b53461cf0 2023-07-29T16:32:58.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b532358c0 con 0x558b532bc6b0 2023-07-29T16:32:58.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f8319f03b80 10 client.0 _ll_register_callbacks cb 0x558b533575c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 0x558b53469fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.053+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 0x558b53469fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:57698/0 (socket says 172.21.15.31:57698) 2023-07-29T16:32:58.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 -- 172.21.15.31:0/4253179135 learned_addr learned my addr 172.21.15.31:0/4253179135 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:58.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53461cf0 0x558b53471520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82fffff700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b5346a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 -- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53461cf0 msgr2=0x558b53471520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53461cf0 0x558b53471520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 -- 172.21.15.31:0/4253179135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 msgr2=0x558b5346a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b5346a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 -- 172.21.15.31:0/4253179135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40156f0 con 0x558b532bc6b0 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.054+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 0x558b53469fa0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009f10 tx=0x7f82f4009be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eddcd32841e647af server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f82df7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f82df7fe700 1 -- 172.21.15.31:0/4253179135 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4015030 con 0x558b532bc6b0 2023-07-29T16:32:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b533dccb0 con 0x558b532bc6b0 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f82df7fe700 1 -- 172.21.15.31:0/4253179135 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f82f4015190 con 0x558b532bc6b0 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f82df7fe700 1 -- 172.21.15.31:0/4253179135 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82f4015350 con 0x558b532bc6b0 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f8319f03b80 10 client.30241 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x558b531c8c00 con 0x558b532bc6b0 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.055+0000 7f82df7fe700 1 -- 172.21.15.31:0/4253179135 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82f4033020 con 0x558b532bc6b0 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f82df7fe700 1 -- 172.21.15.31:0/4253179135 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f82f402d030 con 0x558b532bc6b0 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f8319f03b80 10 client.30241 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:58.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f8319f03b80 10 client.30241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:58.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f8319f03b80 20 client.30241 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:58.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f8319f03b80 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b53475c10 0x558b53478020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f8319f03b80 4 client.30241 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:32:58.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b5342fd10 con 0x558b53475c10 2023-07-29T16:32:58.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.056+0000 7f82fffff700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b53475c10 0x558b53478020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.057+0000 7f82fffff700 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b53475c10 0x558b53478020 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6c51ccd92bb3ac0 server_cookie=b12a25a747765bd6 in_seq=0 out_seq=0 2023-07-29T16:32:58.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.057+0000 7f82df7fe700 10 client.30241 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.058+0000 7f82df7fe700 1 -- 172.21.15.31:0/4253179135 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558b5342fd10 con 0x558b53475c10 2023-07-29T16:32:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.058+0000 7f82df7fe700 10 client.30241 handle_command_reply: tid=0 2023-07-29T16:32:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.058+0000 7f8319f03b80 1 client.30241 shutdown 2023-07-29T16:32:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 msgr2=0x558b53469fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 0x558b53469fa0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009f10 tx=0x7f82f4009be0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 shutdown_connections 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b53475c10 0x558b53478020 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53461cf0 0x558b53471520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b532bc6b0 0x558b53469fa0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 --2- 172.21.15.31:0/4253179135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53460df0 0x558b5346a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 >> 172.21.15.31:0/4253179135 conn(0x558b53455fd0 msgr2=0x558b5345aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f82df7fe700 0 client.30241 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 shutdown_connections 2023-07-29T16:32:58.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 1 -- 172.21.15.31:0/4253179135 wait complete. 2023-07-29T16:32:58.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.059+0000 7f8319f03b80 20 client.30241 trim_cache size 0 max 16384 2023-07-29T16:32:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 Processor -- start 2023-07-29T16:32:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 -- start start 2023-07-29T16:32:58.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b5346f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53460df0 0x558b53468070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53461cf0 0x558b534685b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5346b190 con 0x558b53461cf0 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5345c600 con 0x558b532bc6b0 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.082+0000 7f8319f03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5345cd90 con 0x558b53460df0 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b5346f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b5346f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44770/0 (socket says 172.21.15.31:44770) 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 -- 172.21.15.31:0/737036100 learned_addr learned my addr 172.21.15.31:0/737036100 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82fffff700 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53460df0 0x558b53468070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 -- 172.21.15.31:0/737036100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53460df0 msgr2=0x558b53468070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53460df0 0x558b53468070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 -- 172.21.15.31:0/737036100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53461cf0 msgr2=0x558b534685b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53461cf0 0x558b534685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.083+0000 7f82ff7fe700 1 -- 172.21.15.31:0/737036100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b532061f0 con 0x558b532bc6b0 2023-07-29T16:32:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.084+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b5346f4a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001eb0 tx=0x7f82f4005650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6b39e2579dc540f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:58.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.084+0000 7f82e6ffd700 1 -- 172.21.15.31:0/737036100 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4021720 con 0x558b532bc6b0 2023-07-29T16:32:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.084+0000 7f82e6ffd700 1 -- 172.21.15.31:0/737036100 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82f4033040 con 0x558b532bc6b0 2023-07-29T16:32:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.084+0000 7f82e6ffd700 1 -- 172.21.15.31:0/737036100 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4015360 con 0x558b532bc6b0 2023-07-29T16:32:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 -- 172.21.15.31:0/737036100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 msgr2=0x558b5346f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b5346f4a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001eb0 tx=0x7f82f4005650 comp rx=0 tx=0).stop 2023-07-29T16:32:58.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 -- 172.21.15.31:0/737036100 shutdown_connections 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b53461cf0 0x558b534685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b532bc6b0 0x558b5346f4a0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 --2- 172.21.15.31:0/737036100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53460df0 0x558b53468070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 -- 172.21.15.31:0/737036100 >> 172.21.15.31:0/737036100 conn(0x558b53455fd0 msgr2=0x558b53464ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.085+0000 7f8319f03b80 1 -- 172.21.15.31:0/737036100 shutdown_connections 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.086+0000 7f8319f03b80 1 -- 172.21.15.31:0/737036100 wait complete. 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.086+0000 7f8319f03b80 1 -- start start 2023-07-29T16:32:58.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.086+0000 7f8319f03b80 1 Processor -- start 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b532bc6b0 0x558b5346f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 0x558b5346d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b5346dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5346e200 con 0x558b532bc6b0 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5346e370 con 0x558b53460df0 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5346e4e0 con 0x558b53461cf0 2023-07-29T16:32:58.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f8319f03b80 10 client.0 _ll_register_callbacks cb 0x558b533575c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:32:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f82fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 0x558b5346d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f82fffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 0x558b5346d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:44772/0 (socket says 172.21.15.31:44772) 2023-07-29T16:32:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.087+0000 7f82fffff700 1 -- 172.21.15.31:0/641367654 learned_addr learned my addr 172.21.15.31:0/641367654 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:32:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b532bc6b0 0x558b5346f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b5346dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82fffff700 1 -- 172.21.15.31:0/641367654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 msgr2=0x558b5346dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82fffff700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b5346dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82fffff700 1 -- 172.21.15.31:0/641367654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b532bc6b0 msgr2=0x558b5346f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:32:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82fffff700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b532bc6b0 0x558b5346f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82fffff700 1 -- 172.21.15.31:0/641367654 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b531c8c00 con 0x558b53460df0 2023-07-29T16:32:58.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82fffff700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 0x558b5346d780 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f82f0000c00 tx=0x7f82f0019500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9803accf528089f4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:32:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.088+0000 7f82df7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:32:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.089+0000 7f82df7fe700 1 -- 172.21.15.31:0/641367654 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0017040 con 0x558b53460df0 2023-07-29T16:32:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.089+0000 7f82df7fe700 1 -- 172.21.15.31:0/641367654 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f82f0019dc0 con 0x558b53460df0 2023-07-29T16:32:58.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.089+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82f40054e0 con 0x558b53460df0 2023-07-29T16:32:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.100+0000 7f82df7fe700 1 -- 172.21.15.31:0/641367654 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82f0004050 con 0x558b53460df0 2023-07-29T16:32:58.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.100+0000 7f8319f03b80 10 client.30236 fetch_fsmap learned FSMap version 39 2023-07-29T16:32:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.100+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x558b531de250 con 0x558b53460df0 2023-07-29T16:32:58.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.100+0000 7f82df7fe700 1 -- 172.21.15.31:0/641367654 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82f40054e0 con 0x558b53460df0 2023-07-29T16:32:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.101+0000 7f82df7fe700 1 -- 172.21.15.31:0/641367654 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f82f0012450 con 0x558b53460df0 2023-07-29T16:32:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.102+0000 7f8319f03b80 10 client.30236 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:32:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.102+0000 7f8319f03b80 10 client.30236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:32:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.102+0000 7f8319f03b80 20 client.30236 populate_metadata read hostname 'smithi031' 2023-07-29T16:32:58.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.102+0000 7f8319f03b80 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b534c8f00 0x558b534cb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:32:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.102+0000 7f8319f03b80 4 client.30236 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:32:58.103 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.102+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558b533aace0 con 0x558b534c8f00 2023-07-29T16:32:58.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.104+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b534c8f00 0x558b534cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:32:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.105+0000 7f82df7fe700 10 client.30236 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:58.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.105+0000 7f82ff7fe700 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b534c8f00 0x558b534cb310 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21ed60bc475091 server_cookie=817f7e1bac64d6d3 in_seq=0 out_seq=0 2023-07-29T16:32:58.248 INFO:tasks.workunit.client.0.smithi031.stdout: 10 108578 38.89 MB/sec execute 42 sec latency 100.599 ms 2023-07-29T16:32:58.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:32:58 smithi031 ceph-mon[95504]: pgmap v896: 97 pgs: 97 active+clean; 625 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.0 MiB/s wr, 625 op/s 2023-07-29T16:32:58.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:32:58 smithi039 ceph-mon[128858]: pgmap v896: 97 pgs: 97 active+clean; 625 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.0 MiB/s wr, 625 op/s 2023-07-29T16:32:58.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.557+0000 7f82df7fe700 1 -- 172.21.15.31:0/641367654 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558b533aace0 con 0x558b534c8f00 2023-07-29T16:32:58.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.557+0000 7f82df7fe700 10 client.30236 handle_command_reply: tid=0 2023-07-29T16:32:58.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.557+0000 7f8319f03b80 1 client.30236 shutdown 2023-07-29T16:32:58.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.558+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 msgr2=0x558b5346d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.558+0000 7f8319f03b80 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 0x558b5346d780 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f82f0000c00 tx=0x7f82f0019500 comp rx=0 tx=0).stop 2023-07-29T16:32:58.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 shutdown_connections 2023-07-29T16:32:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f8319f03b80 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b534c8f00 0x558b534cb310 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f8319f03b80 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b53461cf0 0x558b5346dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f8319f03b80 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b532bc6b0 0x558b5346f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f8319f03b80 1 --2- 172.21.15.31:0/641367654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b53460df0 0x558b5346d780 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 >> 172.21.15.31:0/641367654 conn(0x558b53455fd0 msgr2=0x558b53468070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:58.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.559+0000 7f82df7fe700 0 client.30236 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:32:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.560+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 shutdown_connections 2023-07-29T16:32:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.560+0000 7f8319f03b80 1 -- 172.21.15.31:0/641367654 wait complete. 2023-07-29T16:32:58.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.560+0000 7f8319f03b80 20 client.30236 trim_cache size 0 max 16384 2023-07-29T16:32:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:32:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:32:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:32:58.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:32:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82e80a0810 msgr2=0x7f82e80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82e80a0810 0x7f82e80a2cc0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f82f80a2dc0 tx=0x7f82f8009040 comp rx=0 tx=0).stop 2023-07-29T16:32:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 msgr2=0x7f83000ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:32:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f83000ff890 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f82f80077d0 tx=0x7f82f800b9d0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 shutdown_connections 2023-07-29T16:32:58.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f82e80a0810 0x7f82e80a2cc0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83001060a0 0x7f83000ff890 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8300100e60 0x7f83000fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 --2- 172.21.15.31:0/3319525522 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8300103780 0x7f83000ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:32:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 >> 172.21.15.31:0/3319525522 conn(0x7f83000fa840 msgr2=0x7f8300101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:32:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 shutdown_connections 2023-07-29T16:32:58.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:32:58.562+0000 7f83065ff700 1 -- 172.21.15.31:0/3319525522 wait complete. 2023-07-29T16:32:58.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:32:58 smithi156 ceph-mon[129245]: pgmap v896: 97 pgs: 97 active+clean; 625 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 5.0 MiB/s wr, 625 op/s 2023-07-29T16:32:59.249 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109102 38.78 MB/sec execute 43 sec latency 237.199 ms 2023-07-29T16:33:00.249 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109569 38.38 MB/sec execute 44 sec latency 190.668 ms 2023-07-29T16:33:00.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:00 smithi031 ceph-mon[95504]: pgmap v897: 97 pgs: 97 active+clean; 630 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.5 MiB/s wr, 719 op/s 2023-07-29T16:33:01.026 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:00 smithi039 ceph-mon[128858]: pgmap v897: 97 pgs: 97 active+clean; 630 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.5 MiB/s wr, 719 op/s 2023-07-29T16:33:01.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:00 smithi156 ceph-mon[129245]: pgmap v897: 97 pgs: 97 active+clean; 630 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.5 MiB/s wr, 719 op/s 2023-07-29T16:33:01.251 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109579 37.53 MB/sec execute 45 sec latency 1182.608 ms 2023-07-29T16:33:02.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:01 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:33:02.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:01 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:33:02.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:01 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:33:02.251 INFO:tasks.workunit.client.0.smithi031.stdout: 10 109715 36.87 MB/sec execute 46 sec latency 1708.460 ms 2023-07-29T16:33:03.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:02 smithi039 ceph-mon[128858]: pgmap v898: 97 pgs: 97 active+clean; 630 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.4 MiB/s wr, 649 op/s 2023-07-29T16:33:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:33:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:33:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:33:03.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:02 smithi156 ceph-mon[129245]: pgmap v898: 97 pgs: 97 active+clean; 630 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.4 MiB/s wr, 649 op/s 2023-07-29T16:33:03.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:33:03.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:33:03.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:33:03.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:02 smithi031 ceph-mon[95504]: pgmap v898: 97 pgs: 97 active+clean; 630 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.4 MiB/s wr, 649 op/s 2023-07-29T16:33:03.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:33:03.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:33:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:33:03.252 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110090 36.39 MB/sec execute 47 sec latency 234.334 ms 2023-07-29T16:33:04.252 INFO:tasks.workunit.client.0.smithi031.stdout: 10 110571 36.22 MB/sec execute 48 sec latency 79.472 ms 2023-07-29T16:33:05.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:04 smithi156 ceph-mon[129245]: pgmap v899: 97 pgs: 97 active+clean; 680 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 15 MiB/s wr, 874 op/s 2023-07-29T16:33:05.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:04 smithi031 ceph-mon[95504]: pgmap v899: 97 pgs: 97 active+clean; 680 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 15 MiB/s wr, 874 op/s 2023-07-29T16:33:05.252 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111448 36.29 MB/sec execute 49 sec latency 45.514 ms 2023-07-29T16:33:05.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:04 smithi039 ceph-mon[128858]: pgmap v899: 97 pgs: 97 active+clean; 680 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 15 MiB/s wr, 874 op/s 2023-07-29T16:33:06.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:05 smithi031 ceph-mon[95504]: pgmap v900: 97 pgs: 97 active+clean; 677 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 20 MiB/s wr, 977 op/s 2023-07-29T16:33:06.252 INFO:tasks.workunit.client.0.smithi031.stdout: 10 111753 35.97 MB/sec execute 50 sec latency 644.860 ms 2023-07-29T16:33:06.277 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:05 smithi039 ceph-mon[128858]: pgmap v900: 97 pgs: 97 active+clean; 677 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 20 MiB/s wr, 977 op/s 2023-07-29T16:33:06.323 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:05 smithi156 ceph-mon[129245]: pgmap v900: 97 pgs: 97 active+clean; 677 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 20 MiB/s wr, 977 op/s 2023-07-29T16:33:07.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 112503 36.14 MB/sec execute 51 sec latency 182.347 ms 2023-07-29T16:33:08.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 113328 36.33 MB/sec execute 52 sec latency 40.715 ms 2023-07-29T16:33:08.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:08 smithi031 ceph-mon[95504]: pgmap v901: 97 pgs: 97 active+clean; 640 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 21 MiB/s wr, 1.08k op/s 2023-07-29T16:33:08.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:08 smithi039 ceph-mon[128858]: pgmap v901: 97 pgs: 97 active+clean; 640 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 21 MiB/s wr, 1.08k op/s 2023-07-29T16:33:08.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:08 smithi156 ceph-mon[129245]: pgmap v901: 97 pgs: 97 active+clean; 640 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 21 MiB/s wr, 1.08k op/s 2023-07-29T16:33:09.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114026 36.41 MB/sec execute 53 sec latency 74.286 ms 2023-07-29T16:33:10.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 114659 36.33 MB/sec execute 54 sec latency 73.228 ms 2023-07-29T16:33:10.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:10 smithi031 ceph-mon[95504]: pgmap v902: 97 pgs: 97 active+clean; 608 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 22 MiB/s wr, 1.15k op/s 2023-07-29T16:33:11.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:10 smithi039 ceph-mon[128858]: pgmap v902: 97 pgs: 97 active+clean; 608 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 22 MiB/s wr, 1.15k op/s 2023-07-29T16:33:11.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:10 smithi156 ceph-mon[129245]: pgmap v902: 97 pgs: 97 active+clean; 608 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 22 MiB/s wr, 1.15k op/s 2023-07-29T16:33:11.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 115364 36.24 MB/sec execute 55 sec latency 66.441 ms 2023-07-29T16:33:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 -- 172.21.15.31:0/368064500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450108940 msgr2=0x7f1450071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 --2- 172.21.15.31:0/368064500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450108940 0x7f1450071370 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f144c009990 tx=0x7f144c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 -- 172.21.15.31:0/368064500 shutdown_connections 2023-07-29T16:33:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 --2- 172.21.15.31:0/368064500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14500718b0 0x7f1450071d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 --2- 172.21.15.31:0/368064500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1450107ff0 0x7f1450108400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 --2- 172.21.15.31:0/368064500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450108940 0x7f1450071370 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.792+0000 7f145679d700 1 -- 172.21.15.31:0/368064500 >> 172.21.15.31:0/368064500 conn(0x7f145006d150 msgr2=0x7f145006f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.793+0000 7f145679d700 1 -- 172.21.15.31:0/368064500 shutdown_connections 2023-07-29T16:33:11.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.793+0000 7f145679d700 1 -- 172.21.15.31:0/368064500 wait complete. 2023-07-29T16:33:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.794+0000 7f145679d700 1 Processor -- start 2023-07-29T16:33:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 -- start start 2023-07-29T16:33:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 0x7f145010e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450107ff0 0x7f145010ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1450108940 0x7f145010b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145010bd10 con 0x7f14500718b0 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145010be80 con 0x7f1450107ff0 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.795+0000 7f145679d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145010c040 con 0x7f1450108940 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.796+0000 7f145579b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 0x7f145010e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.796+0000 7f145579b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 0x7f145010e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58162/0 (socket says 172.21.15.31:58162) 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.796+0000 7f145579b700 1 -- 172.21.15.31:0/3435637042 learned_addr learned my addr 172.21.15.31:0/3435637042 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:11.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.796+0000 7f1455f9c700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1450108940 0x7f145010b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.796+0000 7f1454f9a700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450107ff0 0x7f145010ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.798+0000 7f145579b700 1 -- 172.21.15.31:0/3435637042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1450108940 msgr2=0x7f145010b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.798+0000 7f145579b700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1450108940 0x7f145010b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.798+0000 7f145579b700 1 -- 172.21.15.31:0/3435637042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450107ff0 msgr2=0x7f145010ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.798+0000 7f145579b700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450107ff0 0x7f145010ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.798+0000 7f145579b700 1 -- 172.21.15.31:0/3435637042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0095f0 con 0x7f14500718b0 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.798+0000 7f1455f9c700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1450108940 0x7f145010b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.799+0000 7f145579b700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 0x7f145010e9e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f144000b500 tx=0x7f144000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.799+0000 7f14467fc700 1 -- 172.21.15.31:0/3435637042 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440011830 con 0x7f14500718b0 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.799+0000 7f14467fc700 1 -- 172.21.15.31:0/3435637042 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1440011e50 con 0x7f14500718b0 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.799+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14501a0670 con 0x7f14500718b0 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.800+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14501a0b80 con 0x7f14500718b0 2023-07-29T16:33:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.800+0000 7f14467fc700 1 -- 172.21.15.31:0/3435637042 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144000f2e0 con 0x7f14500718b0 2023-07-29T16:33:11.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.804+0000 7f14467fc700 1 -- 172.21.15.31:0/3435637042 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f144000f440 con 0x7f14500718b0 2023-07-29T16:33:11.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.807+0000 7f14467fc700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f143c0a0990 0x7f143c0a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.807+0000 7f14467fc700 1 -- 172.21.15.31:0/3435637042 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f14400cb490 con 0x7f14500718b0 2023-07-29T16:33:11.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.807+0000 7f1454f9a700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f143c0a0990 0x7f143c0a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.809+0000 7f1454f9a700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f143c0a0990 0x7f143c0a2e40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f144c00b5c0 tx=0x7f144c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:11.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.814+0000 7f146b370b80 1 Processor -- start 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 -- start start 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951918c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919150 0x5609519199c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x56095191a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609518709b0 con 0x5609517746b0 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56095186e370 con 0x560951919f00 2023-07-29T16:33:11.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.816+0000 7f146b370b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951869c60 con 0x560951919150 2023-07-29T16:33:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.817+0000 7f145579b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951918c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.817+0000 7f1455f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x56095191a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.817+0000 7f145579b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951918c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58176/0 (socket says 172.21.15.31:58176) 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.817+0000 7f145579b700 1 -- 172.21.15.31:0/1780052826 learned_addr learned my addr 172.21.15.31:0/1780052826 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.817+0000 7f1454f9a700 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919150 0x5609519199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f1455f9c700 1 -- 172.21.15.31:0/1780052826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919150 msgr2=0x5609519199c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f1455f9c700 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919150 0x5609519199c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f1455f9c700 1 -- 172.21.15.31:0/1780052826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 msgr2=0x560951918c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f1455f9c700 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951918c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f1455f9c700 1 -- 172.21.15.31:0/1780052826 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560951697d80 con 0x560951919f00 2023-07-29T16:33:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f1455f9c700 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x56095191a3a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1448009d90 tx=0x7f14480077e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3e4ece38568a029 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f143b7fe700 1 -- 172.21.15.31:0/1780052826 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448014610 con 0x560951919f00 2023-07-29T16:33:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f143b7fe700 1 -- 172.21.15.31:0/1780052826 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144800f050 con 0x560951919f00 2023-07-29T16:33:11.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.818+0000 7f143b7fe700 1 -- 172.21.15.31:0/1780052826 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448013960 con 0x560951919f00 2023-07-29T16:33:11.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.821+0000 7f146b370b80 1 -- 172.21.15.31:0/1780052826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 msgr2=0x56095191a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.821+0000 7f146b370b80 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x56095191a3a0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f1448009d90 tx=0x7f14480077e0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 -- 172.21.15.31:0/1780052826 shutdown_connections 2023-07-29T16:33:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x56095191a3a0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951918c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 --2- 172.21.15.31:0/1780052826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919150 0x5609519199c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 -- 172.21.15.31:0/1780052826 >> 172.21.15.31:0/1780052826 conn(0x56095190e380 msgr2=0x5609519136a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 -- 172.21.15.31:0/1780052826 shutdown_connections 2023-07-29T16:33:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.822+0000 7f146b370b80 1 -- 172.21.15.31:0/1780052826 wait complete. 2023-07-29T16:33:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.823+0000 7f146b370b80 1 -- start start 2023-07-29T16:33:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.823+0000 7f146b370b80 1 Processor -- start 2023-07-29T16:33:11.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951922100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951922640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951929680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609518064b0 con 0x5609517746b0 2023-07-29T16:33:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951806150 con 0x560951919150 2023-07-29T16:33:11.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609516ed8c0 con 0x560951919f00 2023-07-29T16:33:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.824+0000 7f146b370b80 10 client.0 _ll_register_callbacks cb 0x56095180f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951922100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951922100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58186/0 (socket says 172.21.15.31:58186) 2023-07-29T16:33:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 -- 172.21.15.31:0/1819119352 learned_addr learned my addr 172.21.15.31:0/1819119352 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f1455f9c700 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951929680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 -- 172.21.15.31:0/1819119352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 msgr2=0x560951929680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951929680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 -- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 msgr2=0x560951922640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951922640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.825+0000 7f145579b700 1 -- 172.21.15.31:0/1819119352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14480099c0 con 0x5609517746b0 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.826+0000 7f145579b700 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951922100 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f144000b4d0 tx=0x7f1440098760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2e0aee6ae99f29 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.826+0000 7f1438ff9700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:33:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.826+0000 7f1438ff9700 1 -- 172.21.15.31:0/1819119352 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144009cb60 con 0x5609517746b0 2023-07-29T16:33:11.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.826+0000 7f1438ff9700 1 -- 172.21.15.31:0/1819119352 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f14400a0f80 con 0x5609517746b0 2023-07-29T16:33:11.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.826+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560951894cb0 con 0x5609517746b0 2023-07-29T16:33:11.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.827+0000 7f1438ff9700 1 -- 172.21.15.31:0/1819119352 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1440098020 con 0x5609517746b0 2023-07-29T16:33:11.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.827+0000 7f146b370b80 10 client.20025 fetch_fsmap learned FSMap version 39 2023-07-29T16:33:11.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.827+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x560951680c00 con 0x5609517746b0 2023-07-29T16:33:11.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.827+0000 7f1438ff9700 1 -- 172.21.15.31:0/1819119352 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560951894cb0 con 0x5609517746b0 2023-07-29T16:33:11.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.827+0000 7f1438ff9700 1 -- 172.21.15.31:0/1819119352 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1440017b90 con 0x5609517746b0 2023-07-29T16:33:11.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.828+0000 7f146b370b80 10 client.20025 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:33:11.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.828+0000 7f146b370b80 10 client.20025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:11.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.828+0000 7f146b370b80 20 client.20025 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.828+0000 7f146b370b80 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56095194ffe0 0x5609519523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.828+0000 7f146b370b80 4 client.20025 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:33:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.828+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609518e7d10 con 0x56095194ffe0 2023-07-29T16:33:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.830+0000 7f1454f9a700 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56095194ffe0 0x5609519523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.831+0000 7f1454f9a700 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56095194ffe0 0x5609519523f0 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db7113ff4beef7f0 server_cookie=8ad33c30b8de9176 in_seq=0 out_seq=0 2023-07-29T16:33:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.831+0000 7f1438ff9700 10 client.20025 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:11.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.832+0000 7f1438ff9700 1 -- 172.21.15.31:0/1819119352 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5609518e7d10 con 0x56095194ffe0 2023-07-29T16:33:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.832+0000 7f1438ff9700 10 client.20025 handle_command_reply: tid=0 2023-07-29T16:33:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.832+0000 7f146b370b80 1 client.20025 shutdown 2023-07-29T16:33:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.832+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 msgr2=0x560951922100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:11.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.832+0000 7f146b370b80 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951922100 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f144000b4d0 tx=0x7f1440098760 comp rx=0 tx=0).stop 2023-07-29T16:33:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 shutdown_connections 2023-07-29T16:33:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f146b370b80 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56095194ffe0 0x5609519523f0 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f146b370b80 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951929680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f146b370b80 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951922100 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f146b370b80 1 --2- 172.21.15.31:0/1819119352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951922640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 >> 172.21.15.31:0/1819119352 conn(0x56095190e380 msgr2=0x560951912df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.833+0000 7f1438ff9700 0 client.20025 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.834+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 shutdown_connections 2023-07-29T16:33:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.834+0000 7f146b370b80 1 -- 172.21.15.31:0/1819119352 wait complete. 2023-07-29T16:33:11.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.837+0000 7f146b370b80 20 client.20025 trim_cache size 0 max 16384 2023-07-29T16:33:11.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.860+0000 7f146b370b80 1 Processor -- start 2023-07-29T16:33:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 -- start start 2023-07-29T16:33:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609517746b0 0x560951920270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 0x560951929610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x5609519207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609519215b0 con 0x560951919150 2023-07-29T16:33:11.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951914750 con 0x560951919f00 2023-07-29T16:33:11.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.862+0000 7f146b370b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951919810 con 0x5609517746b0 2023-07-29T16:33:11.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 0x560951929610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 0x560951929610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58192/0 (socket says 172.21.15.31:58192) 2023-07-29T16:33:11.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f145579b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609517746b0 0x560951920270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 -- 172.21.15.31:0/3747574207 learned_addr learned my addr 172.21.15.31:0/3747574207 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:11.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 -- 172.21.15.31:0/3747574207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609517746b0 msgr2=0x560951920270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609517746b0 0x560951920270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 -- 172.21.15.31:0/3747574207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 msgr2=0x5609519207b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x5609519207b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.863+0000 7f1454f9a700 1 -- 172.21.15.31:0/3747574207 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609516be1f0 con 0x560951919150 2023-07-29T16:33:11.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.864+0000 7f1454f9a700 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 0x560951929610 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f144c000c00 tx=0x7f144c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218af544cdad9901 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.864+0000 7f143bfff700 1 -- 172.21.15.31:0/3747574207 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c031a40 con 0x560951919150 2023-07-29T16:33:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.864+0000 7f143bfff700 1 -- 172.21.15.31:0/3747574207 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144c02b040 con 0x560951919150 2023-07-29T16:33:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.864+0000 7f143bfff700 1 -- 172.21.15.31:0/3747574207 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c031e00 con 0x560951919150 2023-07-29T16:33:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.866+0000 7f146b370b80 1 -- 172.21.15.31:0/3747574207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 msgr2=0x560951929610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:11.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.866+0000 7f146b370b80 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 0x560951929610 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f144c000c00 tx=0x7f144c020000 comp rx=0 tx=0).stop 2023-07-29T16:33:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.867+0000 7f146b370b80 1 -- 172.21.15.31:0/3747574207 shutdown_connections 2023-07-29T16:33:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.867+0000 7f146b370b80 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919f00 0x5609519207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.867+0000 7f146b370b80 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5609517746b0 0x560951920270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.867+0000 7f146b370b80 1 --2- 172.21.15.31:0/3747574207 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560951919150 0x560951929610 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.867+0000 7f146b370b80 1 -- 172.21.15.31:0/3747574207 >> 172.21.15.31:0/3747574207 conn(0x56095190e380 msgr2=0x560951951ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:11.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.868+0000 7f146b370b80 1 -- 172.21.15.31:0/3747574207 shutdown_connections 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.868+0000 7f146b370b80 1 -- 172.21.15.31:0/3747574207 wait complete. 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.868+0000 7f146b370b80 1 -- start start 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.868+0000 7f146b370b80 1 Processor -- start 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951925330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951925870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951928890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951928dd0 con 0x5609517746b0 2023-07-29T16:33:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951928f40 con 0x560951919150 2023-07-29T16:33:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x560951929100 con 0x560951919f00 2023-07-29T16:33:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f146b370b80 10 client.0 _ll_register_callbacks cb 0x56095180f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f1454f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951925870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f1454f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951925870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45408/0 (socket says 172.21.15.31:45408) 2023-07-29T16:33:11.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.869+0000 7f1454f9a700 1 -- 172.21.15.31:0/4001224144 learned_addr learned my addr 172.21.15.31:0/4001224144 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1454f9a700 1 -- 172.21.15.31:0/4001224144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 msgr2=0x560951928890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1454f9a700 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951928890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1454f9a700 1 -- 172.21.15.31:0/4001224144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 msgr2=0x560951925330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1454f9a700 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951925330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1454f9a700 1 -- 172.21.15.31:0/4001224144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1450111730 con 0x560951919150 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1454f9a700 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951925870 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f144c005850 tx=0x7f144c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1101071870ede9ad server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:11.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1438ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.870+0000 7f1438ff9700 1 -- 172.21.15.31:0/4001224144 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c031840 con 0x560951919150 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.871+0000 7f1438ff9700 1 -- 172.21.15.31:0/4001224144 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f144c0319a0 con 0x560951919150 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.871+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f144800f050 con 0x560951919150 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.871+0000 7f1438ff9700 1 -- 172.21.15.31:0/4001224144 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f144c031b60 con 0x560951919150 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.871+0000 7f146b370b80 10 client.30248 fetch_fsmap learned FSMap version 39 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.871+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x560951696250 con 0x560951919150 2023-07-29T16:33:11.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.871+0000 7f1438ff9700 1 -- 172.21.15.31:0/4001224144 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f144c046030 con 0x560951919150 2023-07-29T16:33:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f1438ff9700 1 -- 172.21.15.31:0/4001224144 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 39) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f144c024410 con 0x560951919150 2023-07-29T16:33:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f146b370b80 10 client.30248 fetch_fsmap finished waiting for FSMap version 39 2023-07-29T16:33:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f146b370b80 10 client.30248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f146b370b80 20 client.30248 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:11.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f146b370b80 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609519a26d0 0x5609519a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f146b370b80 4 client.30248 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:33:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560951862ce0 con 0x5609519a26d0 2023-07-29T16:33:11.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.872+0000 7f145579b700 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609519a26d0 0x5609519a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.873+0000 7f1438ff9700 10 client.30248 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:11.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:11.873+0000 7f145579b700 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609519a26d0 0x5609519a4ae0 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81a114b4e62aaa2f server_cookie=cb929aac5d8a5cab in_seq=0 out_seq=0 2023-07-29T16:33:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.181+0000 7f1438ff9700 1 -- 172.21.15.31:0/4001224144 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560951862ce0 con 0x5609519a26d0 2023-07-29T16:33:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.181+0000 7f1438ff9700 10 client.30248 handle_command_reply: tid=0 2023-07-29T16:33:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.181+0000 7f146b370b80 1 client.30248 shutdown 2023-07-29T16:33:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.183+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 msgr2=0x560951925870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951925870 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f144c005850 tx=0x7f144c00bab0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 shutdown_connections 2023-07-29T16:33:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5609519a26d0 0x5609519a4ae0 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560951919f00 0x560951928890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5609517746b0 0x560951925330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 --2- 172.21.15.31:0/4001224144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560951919150 0x560951925870 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 >> 172.21.15.31:0/4001224144 conn(0x56095190e380 msgr2=0x560951920270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f1438ff9700 0 client.30248 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 shutdown_connections 2023-07-29T16:33:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.184+0000 7f146b370b80 1 -- 172.21.15.31:0/4001224144 wait complete. 2023-07-29T16:33:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.185+0000 7f146b370b80 20 client.30248 trim_cache size 0 max 16384 2023-07-29T16:33:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:33:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:33:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:33:12.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:33:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.185+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f143c0a0990 msgr2=0x7f143c0a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.185+0000 7f145679d700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f143c0a0990 0x7f143c0a2e40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f144c00b5c0 tx=0x7f144c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.185+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 msgr2=0x7f145010e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:12.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.185+0000 7f145679d700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 0x7f145010e9e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f144000b500 tx=0x7f144000b810 comp rx=0 tx=0).stop 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 shutdown_connections 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f143c0a0990 0x7f143c0a2e40 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1450108940 0x7f145010b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f14500718b0 0x7f145010e9e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 --2- 172.21.15.31:0/3435637042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1450107ff0 0x7f145010ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 >> 172.21.15.31:0/3435637042 conn(0x7f145006d150 msgr2=0x7f145006fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.187+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 shutdown_connections 2023-07-29T16:33:12.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:12.188+0000 7f145679d700 1 -- 172.21.15.31:0/3435637042 wait complete. 2023-07-29T16:33:12.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 116083 36.26 MB/sec execute 56 sec latency 176.832 ms 2023-07-29T16:33:12.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:12 smithi031 ceph-mon[95504]: pgmap v903: 97 pgs: 97 active+clean; 608 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 22 MiB/s wr, 1.01k op/s 2023-07-29T16:33:13.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:12 smithi039 ceph-mon[128858]: pgmap v903: 97 pgs: 97 active+clean; 608 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 22 MiB/s wr, 1.01k op/s 2023-07-29T16:33:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:12 smithi156 ceph-mon[129245]: pgmap v903: 97 pgs: 97 active+clean; 608 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 22 MiB/s wr, 1.01k op/s 2023-07-29T16:33:13.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 117011 36.44 MB/sec execute 57 sec latency 42.619 ms 2023-07-29T16:33:14.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 117940 36.91 MB/sec execute 58 sec latency 48.539 ms 2023-07-29T16:33:14.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:14 smithi031 ceph-mon[95504]: pgmap v904: 97 pgs: 97 active+clean; 621 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 22 MiB/s wr, 1.15k op/s 2023-07-29T16:33:15.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:14 smithi039 ceph-mon[128858]: pgmap v904: 97 pgs: 97 active+clean; 621 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 22 MiB/s wr, 1.15k op/s 2023-07-29T16:33:15.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:14 smithi156 ceph-mon[129245]: pgmap v904: 97 pgs: 97 active+clean; 621 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 22 MiB/s wr, 1.15k op/s 2023-07-29T16:33:15.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 118542 36.88 MB/sec execute 59 sec latency 99.701 ms 2023-07-29T16:33:16.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 119047 36.61 MB/sec execute 60 sec latency 117.446 ms 2023-07-29T16:33:16.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:16 smithi031 ceph-mon[95504]: pgmap v905: 97 pgs: 97 active+clean; 622 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 14 MiB/s wr, 942 op/s 2023-07-29T16:33:17.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:16 smithi039 ceph-mon[128858]: pgmap v905: 97 pgs: 97 active+clean; 622 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 14 MiB/s wr, 942 op/s 2023-07-29T16:33:17.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:16 smithi156 ceph-mon[129245]: pgmap v905: 97 pgs: 97 active+clean; 622 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 14 MiB/s wr, 942 op/s 2023-07-29T16:33:17.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 119918 36.71 MB/sec execute 61 sec latency 93.184 ms 2023-07-29T16:33:18.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 120909 36.97 MB/sec execute 62 sec latency 61.662 ms 2023-07-29T16:33:18.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:18 smithi031 ceph-mon[95504]: pgmap v906: 97 pgs: 97 active+clean; 618 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.5 MiB/s wr, 756 op/s 2023-07-29T16:33:18.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:18 smithi039 ceph-mon[128858]: pgmap v906: 97 pgs: 97 active+clean; 618 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.5 MiB/s wr, 756 op/s 2023-07-29T16:33:18.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:18 smithi156 ceph-mon[129245]: pgmap v906: 97 pgs: 97 active+clean; 618 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.5 MiB/s wr, 756 op/s 2023-07-29T16:33:19.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 121697 37.23 MB/sec execute 63 sec latency 91.336 ms 2023-07-29T16:33:19.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:19 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:33:19.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:19 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:33:19.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:19 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:33:20.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 122272 37.11 MB/sec execute 64 sec latency 89.888 ms 2023-07-29T16:33:20.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:20 smithi031 ceph-mon[95504]: pgmap v907: 97 pgs: 97 active+clean; 624 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.3 MiB/s wr, 631 op/s 2023-07-29T16:33:20.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:20 smithi039 ceph-mon[128858]: pgmap v907: 97 pgs: 97 active+clean; 624 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.3 MiB/s wr, 631 op/s 2023-07-29T16:33:20.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:20 smithi156 ceph-mon[129245]: pgmap v907: 97 pgs: 97 active+clean; 624 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.3 MiB/s wr, 631 op/s 2023-07-29T16:33:21.253 INFO:tasks.workunit.client.0.smithi031.stdout: 10 123167 37.21 MB/sec execute 65 sec latency 58.809 ms 2023-07-29T16:33:22.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 123895 37.22 MB/sec execute 66 sec latency 60.986 ms 2023-07-29T16:33:22.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:22 smithi031 ceph-mon[95504]: pgmap v908: 97 pgs: 97 active+clean; 632 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.9 MiB/s wr, 598 op/s 2023-07-29T16:33:23.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:22 smithi039 ceph-mon[128858]: pgmap v908: 97 pgs: 97 active+clean; 632 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.9 MiB/s wr, 598 op/s 2023-07-29T16:33:23.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:22 smithi156 ceph-mon[129245]: pgmap v908: 97 pgs: 97 active+clean; 632 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 4.9 MiB/s wr, 598 op/s 2023-07-29T16:33:23.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 124754 37.28 MB/sec execute 67 sec latency 53.509 ms 2023-07-29T16:33:24.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 125814 37.63 MB/sec execute 68 sec latency 76.420 ms 2023-07-29T16:33:24.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:24 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:33:24.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:24 smithi031 ceph-mon[95504]: pgmap v909: 97 pgs: 97 active+clean; 645 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 5.1 MiB/s wr, 702 op/s 2023-07-29T16:33:24.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:24 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:33:24.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:24 smithi039 ceph-mon[128858]: pgmap v909: 97 pgs: 97 active+clean; 645 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 5.1 MiB/s wr, 702 op/s 2023-07-29T16:33:24.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:24 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:33:24.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:24 smithi156 ceph-mon[129245]: pgmap v909: 97 pgs: 97 active+clean; 645 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 5.1 MiB/s wr, 702 op/s 2023-07-29T16:33:25.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 126573 37.77 MB/sec execute 69 sec latency 70.637 ms 2023-07-29T16:33:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.413+0000 7fb46b1ff700 1 -- 172.21.15.31:0/3787806697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 msgr2=0x7fb464101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.413+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/3787806697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 0x7fb464101070 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fb45c00b0a0 tx=0x7fb45c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.417+0000 7fb46b1ff700 1 -- 172.21.15.31:0/3787806697 shutdown_connections 2023-07-29T16:33:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.417+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/3787806697 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb464103f70 0x7fb4641043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.417+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/3787806697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 0x7fb464101070 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.417+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/3787806697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb464103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.417+0000 7fb46b1ff700 1 -- 172.21.15.31:0/3787806697 >> 172.21.15.31:0/3787806697 conn(0x7fb4640fa840 msgr2=0x7fb4640fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.417+0000 7fb46b1ff700 1 -- 172.21.15.31:0/3787806697 shutdown_connections 2023-07-29T16:33:25.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.418+0000 7fb46b1ff700 1 -- 172.21.15.31:0/3787806697 wait complete. 2023-07-29T16:33:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.419+0000 7fb46b1ff700 1 Processor -- start 2023-07-29T16:33:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.419+0000 7fb46b1ff700 1 -- start start 2023-07-29T16:33:25.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.419+0000 7fb46b1ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 0x7fb46410a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb46b1ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb46410a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb46b1ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb464103f70 0x7fb46410b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb46b1ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46410bbc0 con 0x7fb4640fec50 2023-07-29T16:33:25.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb46b1ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46410bd30 con 0x7fb464103f70 2023-07-29T16:33:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb46b1ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46410bef0 con 0x7fb4641015b0 2023-07-29T16:33:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb46410a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb46410a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42494/0 (socket says 172.21.15.31:42494) 2023-07-29T16:33:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 -- 172.21.15.31:0/832216611 learned_addr learned my addr 172.21.15.31:0/832216611 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb46979c700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb464103f70 0x7fb46410b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 -- 172.21.15.31:0/832216611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb464103f70 msgr2=0x7fb46410b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb464103f70 0x7fb46410b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 -- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 msgr2=0x7fb46410a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.420+0000 7fb463fff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 0x7fb46410a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.421+0000 7fb463fff700 1 -- 172.21.15.31:0/832216611 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c009d00 con 0x7fb4641015b0 2023-07-29T16:33:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.421+0000 7fb463fff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb46410a690 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb45800c920 tx=0x7fb45800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.421+0000 7fb461ffb700 1 -- 172.21.15.31:0/832216611 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458007b00 con 0x7fb4641015b0 2023-07-29T16:33:25.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.421+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46410c1d0 con 0x7fb4641015b0 2023-07-29T16:33:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.421+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46410ca00 con 0x7fb4641015b0 2023-07-29T16:33:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.422+0000 7fb461ffb700 1 -- 172.21.15.31:0/832216611 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb45800ce70 con 0x7fb4641015b0 2023-07-29T16:33:25.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.422+0000 7fb461ffb700 1 -- 172.21.15.31:0/832216611 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458005580 con 0x7fb4641015b0 2023-07-29T16:33:25.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.431+0000 7fb47ea29b80 1 Processor -- start 2023-07-29T16:33:25.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.431+0000 7fb47ea29b80 1 -- start start 2023-07-29T16:33:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcdf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdce07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdce1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdc379b0 con 0x5633fdce0cf0 2023-07-29T16:33:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdc35370 con 0x5633fdb3b6b0 2023-07-29T16:33:25.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdc30c60 con 0x5633fdcdfdf0 2023-07-29T16:33:25.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb46979c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcdf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb46979c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcdf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60080/0 (socket says 172.21.15.31:60080) 2023-07-29T16:33:25.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.432+0000 7fb46979c700 1 -- 172.21.15.31:0/3253162584 learned_addr learned my addr 172.21.15.31:0/3253162584 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.435+0000 7fb468f9b700 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdce07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.435+0000 7fb461ffb700 1 -- 172.21.15.31:0/832216611 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb45800f450 con 0x7fb4641015b0 2023-07-29T16:33:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.436+0000 7fb46979c700 1 -- 172.21.15.31:0/3253162584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 msgr2=0x5633fdce07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:25.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.436+0000 7fb46979c700 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdce07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.436+0000 7fb46979c700 1 -- 172.21.15.31:0/3253162584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 msgr2=0x5633fdce1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.436+0000 7fb46979c700 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdce1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.436+0000 7fb46979c700 1 -- 172.21.15.31:0/3253162584 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633fda5ed80 con 0x5633fdb3b6b0 2023-07-29T16:33:25.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.436+0000 7fb46979c700 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcdf8b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb45400b500 tx=0x7fb454009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9129634e2404b88 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:25.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.437+0000 7fb44affd700 1 -- 172.21.15.31:0/3253162584 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454010040 con 0x5633fdb3b6b0 2023-07-29T16:33:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.437+0000 7fb44affd700 1 -- 172.21.15.31:0/3253162584 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb45400be20 con 0x5633fdb3b6b0 2023-07-29T16:33:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.437+0000 7fb44affd700 1 -- 172.21.15.31:0/3253162584 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454014c00 con 0x5633fdb3b6b0 2023-07-29T16:33:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb461ffb700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb44c0a0810 0x7fb44c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 -- 172.21.15.31:0/3253162584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 msgr2=0x5633fdcdf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcdf8b0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb45400b500 tx=0x7fb454009630 comp rx=0 tx=0).stop 2023-07-29T16:33:25.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb461ffb700 1 -- 172.21.15.31:0/832216611 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb4580cb890 con 0x7fb4641015b0 2023-07-29T16:33:25.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 -- 172.21.15.31:0/3253162584 shutdown_connections 2023-07-29T16:33:25.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdce1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcdf8b0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/3253162584 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdce07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.438+0000 7fb47ea29b80 1 -- 172.21.15.31:0/3253162584 >> 172.21.15.31:0/3253162584 conn(0x5633fdcd4fd0 msgr2=0x5633fdcda300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.443+0000 7fb47ea29b80 1 -- 172.21.15.31:0/3253162584 shutdown_connections 2023-07-29T16:33:25.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.443+0000 7fb47ea29b80 1 -- 172.21.15.31:0/3253162584 wait complete. 2023-07-29T16:33:25.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.444+0000 7fb47ea29b80 1 -- start start 2023-07-29T16:33:25.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.444+0000 7fb47ea29b80 1 Processor -- start 2023-07-29T16:33:25.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcfa010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdcfa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdd01590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdbcd4b0 con 0x5633fdce0cf0 2023-07-29T16:33:25.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdbcd150 con 0x5633fdb3b6b0 2023-07-29T16:33:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdab48c0 con 0x5633fdcdfdf0 2023-07-29T16:33:25.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.445+0000 7fb47ea29b80 10 client.0 _ll_register_callbacks cb 0x5633fdbd65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.446+0000 7fb468f9b700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb44c0a0810 0x7fb44c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.446+0000 7fb46979c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdd01590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.446+0000 7fb46979c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdd01590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50626/0 (socket says 172.21.15.31:50626) 2023-07-29T16:33:25.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.446+0000 7fb46979c700 1 -- 172.21.15.31:0/1841837327 learned_addr learned my addr 172.21.15.31:0/1841837327 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.447+0000 7fb46979c700 1 -- 172.21.15.31:0/1841837327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 msgr2=0x5633fdcfa550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.447+0000 7fb46979c700 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdcfa550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.447+0000 7fb46979c700 1 -- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 msgr2=0x5633fdcfa010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.447+0000 7fb46979c700 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcfa010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.447+0000 7fb46979c700 1 -- 172.21.15.31:0/1841837327 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb454009ce0 con 0x5633fdce0cf0 2023-07-29T16:33:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.449+0000 7fb468f9b700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb44c0a0810 0x7fb44c0a2cc0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009fd0 tx=0x7fb45c00b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:25.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.450+0000 7fb46979c700 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdd01590 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb454015c40 tx=0x7fb454015d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc408e88b3a3ef3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.450+0000 7fb443fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:33:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.451+0000 7fb443fff700 1 -- 172.21.15.31:0/1841837327 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454010070 con 0x5633fdce0cf0 2023-07-29T16:33:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.451+0000 7fb443fff700 1 -- 172.21.15.31:0/1841837327 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb45400a7f0 con 0x5633fdce0cf0 2023-07-29T16:33:25.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.451+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633fdc5bcb0 con 0x5633fdce0cf0 2023-07-29T16:33:25.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.451+0000 7fb443fff700 1 -- 172.21.15.31:0/1841837327 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4540145a0 con 0x5633fdce0cf0 2023-07-29T16:33:25.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 10 client.20034 fetch_fsmap learned FSMap version 41 2023-07-29T16:33:25.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x5633fda47c00 con 0x5633fdce0cf0 2023-07-29T16:33:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb443fff700 1 -- 172.21.15.31:0/1841837327 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633fdc5bcb0 con 0x5633fdce0cf0 2023-07-29T16:33:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb443fff700 1 -- 172.21.15.31:0/1841837327 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fb454031c30 con 0x5633fdce0cf0 2023-07-29T16:33:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 10 client.20034 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:33:25.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 10 client.20034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 20 client.20034 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd05ce0 0x5633fdd080f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 4 client.20034 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:33:25.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.452+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5633fdcaed10 con 0x5633fdd05ce0 2023-07-29T16:33:25.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.457+0000 7fb468f9b700 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd05ce0 0x5633fdd080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.458+0000 7fb468f9b700 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd05ce0 0x5633fdd080f0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74859a2d0c8557c6 server_cookie=8de8f4f6bab34fdc in_seq=0 out_seq=0 2023-07-29T16:33:25.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.458+0000 7fb443fff700 10 client.20034 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:25.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.462+0000 7fb443fff700 1 -- 172.21.15.31:0/1841837327 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5633fdcaed10 con 0x5633fdd05ce0 2023-07-29T16:33:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.462+0000 7fb443fff700 10 client.20034 handle_command_reply: tid=0 2023-07-29T16:33:25.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.462+0000 7fb47ea29b80 1 client.20034 shutdown 2023-07-29T16:33:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 msgr2=0x5633fdd01590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdd01590 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb454015c40 tx=0x7fb454015d20 comp rx=0 tx=0).stop 2023-07-29T16:33:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 shutdown_connections 2023-07-29T16:33:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd05ce0 0x5633fdd080f0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdce0cf0 0x5633fdd01590 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcfa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/1841837327 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdcdfdf0 0x5633fdcfa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 >> 172.21.15.31:0/1841837327 conn(0x5633fdcd4fd0 msgr2=0x5633fdcde540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb443fff700 0 client.20034 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 shutdown_connections 2023-07-29T16:33:25.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.463+0000 7fb47ea29b80 1 -- 172.21.15.31:0/1841837327 wait complete. 2023-07-29T16:33:25.467 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.464+0000 7fb47ea29b80 20 client.20034 trim_cache size 0 max 16384 2023-07-29T16:33:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.481+0000 7fb47ea29b80 1 Processor -- start 2023-07-29T16:33:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 -- start start 2023-07-29T16:33:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdb3b6b0 0x5633fdcf96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 0x5633fdd01550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd07cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdce1680 con 0x5633fdb3b6b0 2023-07-29T16:33:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdce7850 con 0x5633fdcdfdf0 2023-07-29T16:33:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdce45d0 con 0x5633fdce0cf0 2023-07-29T16:33:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb468f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdb3b6b0 0x5633fdcf96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb463fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 0x5633fdd01550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.482+0000 7fb46979c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd07cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb463fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 0x5633fdd01550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60094/0 (socket says 172.21.15.31:60094) 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd07cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42532/0 (socket says 172.21.15.31:42532) 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb463fff700 1 -- 172.21.15.31:0/687622691 learned_addr learned my addr 172.21.15.31:0/687622691 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 -- 172.21.15.31:0/687622691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 msgr2=0x5633fdd01550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 0x5633fdd01550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 -- 172.21.15.31:0/687622691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdb3b6b0 msgr2=0x5633fdcf96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdb3b6b0 0x5633fdcf96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 -- 172.21.15.31:0/687622691 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633fda851f0 con 0x5633fdce0cf0 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb463fff700 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 0x5633fdd01550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.483+0000 7fb46979c700 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd07cf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x5633fdcdb190 tx=0x7fb454016f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa3c22c11b2ee340 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.484+0000 7fb44b7fe700 1 -- 172.21.15.31:0/687622691 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454010040 con 0x5633fdce0cf0 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.484+0000 7fb44b7fe700 1 -- 172.21.15.31:0/687622691 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb454022070 con 0x5633fdce0cf0 2023-07-29T16:33:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.484+0000 7fb44b7fe700 1 -- 172.21.15.31:0/687622691 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454014b40 con 0x5633fdce0cf0 2023-07-29T16:33:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.485+0000 7fb47ea29b80 1 -- 172.21.15.31:0/687622691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 msgr2=0x5633fdd07cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.485+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd07cf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x5633fdcdb190 tx=0x7fb454016f80 comp rx=0 tx=0).stop 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 -- 172.21.15.31:0/687622691 shutdown_connections 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd07cf0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdb3b6b0 0x5633fdcf96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/687622691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdcdfdf0 0x5633fdd01550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 -- 172.21.15.31:0/687622691 >> 172.21.15.31:0/687622691 conn(0x5633fdcd4fd0 msgr2=0x5633fdce7140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 -- 172.21.15.31:0/687622691 shutdown_connections 2023-07-29T16:33:25.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.486+0000 7fb47ea29b80 1 -- 172.21.15.31:0/687622691 wait complete. 2023-07-29T16:33:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.487+0000 7fb47ea29b80 1 -- start start 2023-07-29T16:33:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.487+0000 7fb47ea29b80 1 Processor -- start 2023-07-29T16:33:25.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdcdfdf0 0x5633fdcff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd005c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdd00b00 con 0x5633fdcdfdf0 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdd00c70 con 0x5633fdb3b6b0 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633fdd00e30 con 0x5633fdce0cf0 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb47ea29b80 10 client.0 _ll_register_callbacks cb 0x5633fdbd65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:25.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb468f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.488+0000 7fb468f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcff070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:60100/0 (socket says 172.21.15.31:60100) 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb468f9b700 1 -- 172.21.15.31:0/865857394 learned_addr learned my addr 172.21.15.31:0/865857394 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb46979c700 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb46979c700 1 -- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 msgr2=0x5633fdcff070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb46979c700 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcff070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb46979c700 1 -- 172.21.15.31:0/865857394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdcdfdf0 msgr2=0x5633fdcff5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:25.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb46979c700 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdcdfdf0 0x5633fdcff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.489+0000 7fb46979c700 1 -- 172.21.15.31:0/865857394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46410ca00 con 0x5633fdce0cf0 2023-07-29T16:33:25.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.490+0000 7fb46979c700 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd005c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x5633fdce0780 tx=0x7fb45401d400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=400d2f98c32c13e1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.490+0000 7fb443fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:33:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb443fff700 1 -- 172.21.15.31:0/865857394 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454010050 con 0x5633fdce0cf0 2023-07-29T16:33:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb443fff700 1 -- 172.21.15.31:0/865857394 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb454027c70 con 0x5633fdce0cf0 2023-07-29T16:33:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb45400be20 con 0x5633fdce0cf0 2023-07-29T16:33:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb443fff700 1 -- 172.21.15.31:0/865857394 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb454003950 con 0x5633fdce0cf0 2023-07-29T16:33:25.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb47ea29b80 10 client.30271 fetch_fsmap learned FSMap version 41 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x5633fda47c00 con 0x5633fdce0cf0 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.491+0000 7fb443fff700 1 -- 172.21.15.31:0/865857394 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb45400be20 con 0x5633fdce0cf0 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb443fff700 1 -- 172.21.15.31:0/865857394 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fb454038070 con 0x5633fdce0cf0 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb47ea29b80 10 client.30271 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb47ea29b80 10 client.30271 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb47ea29b80 20 client.30271 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd57fb0 0x5633fdd5a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:25.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb47ea29b80 4 client.30271 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:33:25.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5633fdc29ce0 con 0x5633fdd57fb0 2023-07-29T16:33:25.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.492+0000 7fb468f9b700 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd57fb0 0x5633fdd5a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:25.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.493+0000 7fb468f9b700 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd57fb0 0x5633fdd5a390 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7247713b97cc2e5 server_cookie=98fffa513718788b in_seq=0 out_seq=0 2023-07-29T16:33:25.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.493+0000 7fb443fff700 10 client.30271 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.878+0000 7fb443fff700 1 -- 172.21.15.31:0/865857394 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5633fdc29ce0 con 0x5633fdd57fb0 2023-07-29T16:33:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.878+0000 7fb443fff700 10 client.30271 handle_command_reply: tid=0 2023-07-29T16:33:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.878+0000 7fb47ea29b80 1 client.30271 shutdown 2023-07-29T16:33:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.878+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 msgr2=0x5633fdd005c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.878+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd005c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x5633fdce0780 tx=0x7fb45401d400 comp rx=0 tx=0).stop 2023-07-29T16:33:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 shutdown_connections 2023-07-29T16:33:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5633fdd57fb0 0x5633fdd5a390 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633fdce0cf0 0x5633fdd005c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5633fdb3b6b0 0x5633fdcff070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 --2- 172.21.15.31:0/865857394 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633fdcdfdf0 0x5633fdcff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 >> 172.21.15.31:0/865857394 conn(0x5633fdcd4fd0 msgr2=0x5633fdd07cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb443fff700 0 client.30271 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 shutdown_connections 2023-07-29T16:33:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 1 -- 172.21.15.31:0/865857394 wait complete. 2023-07-29T16:33:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.879+0000 7fb47ea29b80 20 client.30271 trim_cache size 0 max 16384 2023-07-29T16:33:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:33:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:33:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:33:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb44c0a0810 msgr2=0x7fb44c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb44c0a0810 0x7fb44c0a2cc0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009fd0 tx=0x7fb45c00b7b0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 msgr2=0x7fb46410a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb46410a690 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb45800c920 tx=0x7fb45800cc30 comp rx=0 tx=0).stop 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 shutdown_connections 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb44c0a0810 0x7fb44c0a2cc0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb464103f70 0x7fb46410b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb4640fec50 0x7fb46410a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 --2- 172.21.15.31:0/832216611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4641015b0 0x7fb46410a690 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.880+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 >> 172.21.15.31:0/832216611 conn(0x7fb4640fa840 msgr2=0x7fb4641022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.881+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 shutdown_connections 2023-07-29T16:33:25.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:25.881+0000 7fb46b1ff700 1 -- 172.21.15.31:0/832216611 wait complete. 2023-07-29T16:33:26.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 127083 37.64 MB/sec execute 70 sec latency 190.830 ms 2023-07-29T16:33:27.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:26 smithi039 ceph-mon[128858]: pgmap v910: 97 pgs: 97 active+clean; 645 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 486 op/s 2023-07-29T16:33:27.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:26 smithi156 ceph-mon[129245]: pgmap v910: 97 pgs: 97 active+clean; 645 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 486 op/s 2023-07-29T16:33:27.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:26 smithi031 ceph-mon[95504]: pgmap v910: 97 pgs: 97 active+clean; 645 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.0 MiB/s wr, 486 op/s 2023-07-29T16:33:27.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 127807 37.63 MB/sec execute 71 sec latency 62.384 ms 2023-07-29T16:33:28.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 128567 37.60 MB/sec execute 72 sec latency 59.234 ms 2023-07-29T16:33:28.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:28 smithi031 ceph-mon[95504]: pgmap v911: 97 pgs: 97 active+clean; 648 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2023-07-29T16:33:28.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:28 smithi039 ceph-mon[128858]: pgmap v911: 97 pgs: 97 active+clean; 648 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2023-07-29T16:33:28.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:28 smithi156 ceph-mon[129245]: pgmap v911: 97 pgs: 97 active+clean; 648 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2023-07-29T16:33:29.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 129452 37.70 MB/sec execute 73 sec latency 56.521 ms 2023-07-29T16:33:30.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 130400 37.95 MB/sec execute 74 sec latency 42.477 ms 2023-07-29T16:33:30.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:30 smithi031 ceph-mon[95504]: pgmap v912: 97 pgs: 97 active+clean; 653 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.7 MiB/s wr, 654 op/s 2023-07-29T16:33:31.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:30 smithi039 ceph-mon[128858]: pgmap v912: 97 pgs: 97 active+clean; 653 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.7 MiB/s wr, 654 op/s 2023-07-29T16:33:31.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:30 smithi156 ceph-mon[129245]: pgmap v912: 97 pgs: 97 active+clean; 653 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.7 MiB/s wr, 654 op/s 2023-07-29T16:33:31.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 131280 38.12 MB/sec execute 75 sec latency 72.503 ms 2023-07-29T16:33:32.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 131667 37.97 MB/sec execute 76 sec latency 278.750 ms 2023-07-29T16:33:32.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:32 smithi031 ceph-mon[95504]: pgmap v913: 97 pgs: 97 active+clean; 655 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.2 MiB/s wr, 625 op/s 2023-07-29T16:33:33.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:32 smithi039 ceph-mon[128858]: pgmap v913: 97 pgs: 97 active+clean; 655 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.2 MiB/s wr, 625 op/s 2023-07-29T16:33:33.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:32 smithi156 ceph-mon[129245]: pgmap v913: 97 pgs: 97 active+clean; 655 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 4.2 MiB/s wr, 625 op/s 2023-07-29T16:33:33.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 132205 37.84 MB/sec execute 77 sec latency 62.806 ms 2023-07-29T16:33:34.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 132839 37.61 MB/sec execute 78 sec latency 96.191 ms 2023-07-29T16:33:34.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:34 smithi031 ceph-mon[95504]: pgmap v914: 97 pgs: 97 active+clean; 669 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.4 MiB/s wr, 714 op/s 2023-07-29T16:33:35.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:34 smithi039 ceph-mon[128858]: pgmap v914: 97 pgs: 97 active+clean; 669 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.4 MiB/s wr, 714 op/s 2023-07-29T16:33:35.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:34 smithi156 ceph-mon[129245]: pgmap v914: 97 pgs: 97 active+clean; 669 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 6.4 MiB/s wr, 714 op/s 2023-07-29T16:33:35.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 133966 37.90 MB/sec execute 79 sec latency 36.840 ms 2023-07-29T16:33:36.254 INFO:tasks.workunit.client.0.smithi031.stdout: 10 134909 38.09 MB/sec execute 80 sec latency 49.530 ms 2023-07-29T16:33:37.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:36 smithi039 ceph-mon[128858]: pgmap v915: 97 pgs: 97 active+clean; 672 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.0 MiB/s wr, 658 op/s 2023-07-29T16:33:37.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:36 smithi156 ceph-mon[129245]: pgmap v915: 97 pgs: 97 active+clean; 672 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.0 MiB/s wr, 658 op/s 2023-07-29T16:33:37.198 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:36 smithi031 ceph-mon[95504]: pgmap v915: 97 pgs: 97 active+clean; 672 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.0 MiB/s wr, 658 op/s 2023-07-29T16:33:37.257 INFO:tasks.workunit.client.0.smithi031.stdout: 10 135040 37.73 MB/sec execute 81 sec latency 798.558 ms 2023-07-29T16:33:38.257 INFO:tasks.workunit.client.0.smithi031.stdout: 10 135108 37.31 MB/sec execute 82 sec latency 1392.180 ms 2023-07-29T16:33:38.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:38 smithi031 ceph-mon[95504]: pgmap v916: 97 pgs: 97 active+clean; 703 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.7 MiB/s wr, 762 op/s 2023-07-29T16:33:38.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:38 smithi039 ceph-mon[128858]: pgmap v916: 97 pgs: 97 active+clean; 703 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.7 MiB/s wr, 762 op/s 2023-07-29T16:33:38.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:38 smithi156 ceph-mon[129245]: pgmap v916: 97 pgs: 97 active+clean; 703 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 9.7 MiB/s wr, 762 op/s 2023-07-29T16:33:39.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.110+0000 7f3233bff700 1 -- 172.21.15.31:0/705512759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c075710 msgr2=0x7f322c075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.110+0000 7f3233bff700 1 --2- 172.21.15.31:0/705512759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c075710 0x7f322c075b20 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f322400b0a0 tx=0x7f322400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.121+0000 7f3233bff700 1 -- 172.21.15.31:0/705512759 shutdown_connections 2023-07-29T16:33:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.121+0000 7f3233bff700 1 --2- 172.21.15.31:0/705512759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c077300 0x7f322c077b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.121+0000 7f3233bff700 1 --2- 172.21.15.31:0/705512759 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c075710 0x7f322c075b20 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.121+0000 7f3233bff700 1 --2- 172.21.15.31:0/705512759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c076950 0x7f322c076dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.121+0000 7f3233bff700 1 -- 172.21.15.31:0/705512759 >> 172.21.15.31:0/705512759 conn(0x7f322c0fdba0 msgr2=0x7f322c0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:39.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.132+0000 7f3233bff700 1 -- 172.21.15.31:0/705512759 shutdown_connections 2023-07-29T16:33:39.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.134+0000 7f3233bff700 1 -- 172.21.15.31:0/705512759 wait complete. 2023-07-29T16:33:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.136+0000 7f3233bff700 1 Processor -- start 2023-07-29T16:33:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.136+0000 7f3233bff700 1 -- start start 2023-07-29T16:33:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.137+0000 7f3233bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c075710 0x7f322c06f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.137+0000 7f3233bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 0x7f322c06fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.137+0000 7f3233bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c077300 0x7f322c070b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.137+0000 7f3233bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0711a0 con 0x7f322c075710 2023-07-29T16:33:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.137+0000 7f3233bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c071310 con 0x7f322c077300 2023-07-29T16:33:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.137+0000 7f3233bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0714d0 con 0x7f322c076950 2023-07-29T16:33:39.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 0x7f322c06fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 0x7f322c06fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44260/0 (socket says 172.21.15.31:44260) 2023-07-29T16:33:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 -- 172.21.15.31:0/2389561331 learned_addr learned my addr 172.21.15.31:0/2389561331 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 -- 172.21.15.31:0/2389561331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c077300 msgr2=0x7f322c070b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:33:39.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c077300 0x7f322c070b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 -- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c075710 msgr2=0x7f322c06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c075710 0x7f322c06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.138+0000 7f323119a700 1 -- 172.21.15.31:0/2389561331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224009d00 con 0x7f322c076950 2023-07-29T16:33:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.139+0000 7f323119a700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 0x7f322c06fc70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f322800d950 tx=0x7f322800dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:39.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.139+0000 7f3222ffd700 1 -- 172.21.15.31:0/2389561331 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3228009b40 con 0x7f322c076950 2023-07-29T16:33:39.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.139+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c0717b0 con 0x7f322c076950 2023-07-29T16:33:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.139+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c071f60 con 0x7f322c076950 2023-07-29T16:33:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.145+0000 7f3222ffd700 1 -- 172.21.15.31:0/2389561331 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3228009ca0 con 0x7f322c076950 2023-07-29T16:33:39.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.145+0000 7f3222ffd700 1 -- 172.21.15.31:0/2389561331 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800f620 con 0x7f322c076950 2023-07-29T16:33:39.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.147+0000 7f32474fbb80 1 Processor -- start 2023-07-29T16:33:39.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 -- start start 2023-07-29T16:33:39.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d778b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d77df0 0x55c3f4d787b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d79160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4ccf9b0 con 0x55c3f4bd36b0 2023-07-29T16:33:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4ccd370 con 0x55c3f4d77df0 2023-07-29T16:33:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.148+0000 7f32474fbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4cc8c60 con 0x55c3f4d78cf0 2023-07-29T16:33:39.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.149+0000 7f323199b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d77df0 0x55c3f4d787b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.149+0000 7f323219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d778b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.149+0000 7f323219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d778b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49220/0 (socket says 172.21.15.31:49220) 2023-07-29T16:33:39.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.149+0000 7f323219c700 1 -- 172.21.15.31:0/3133685409 learned_addr learned my addr 172.21.15.31:0/3133685409 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.150+0000 7f323199b700 1 -- 172.21.15.31:0/3133685409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d78cf0 msgr2=0x55c3f4d79160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.150+0000 7f323199b700 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d79160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.150+0000 7f323199b700 1 -- 172.21.15.31:0/3133685409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4bd36b0 msgr2=0x55c3f4d778b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.150+0000 7f323199b700 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d778b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.150+0000 7f323199b700 1 -- 172.21.15.31:0/3133685409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3f4af6d80 con 0x55c3f4d77df0 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.151+0000 7f323199b700 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d77df0 0x55c3f4d787b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f322400b070 tx=0x7f3224009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc545f159d11a09 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:39.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.151+0000 7f3222ffd700 1 -- 172.21.15.31:0/2389561331 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f3228010460 con 0x7f322c076950 2023-07-29T16:33:39.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.151+0000 7f3217fff700 1 -- 172.21.15.31:0/3133685409 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322400e040 con 0x55c3f4d77df0 2023-07-29T16:33:39.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.151+0000 7f3217fff700 1 -- 172.21.15.31:0/3133685409 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f322400bab0 con 0x55c3f4d77df0 2023-07-29T16:33:39.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.151+0000 7f3217fff700 1 -- 172.21.15.31:0/3133685409 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322401b6c0 con 0x55c3f4d77df0 2023-07-29T16:33:39.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.152+0000 7f3222ffd700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f32180a08e0 0x7f32180a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.153+0000 7f3222ffd700 1 -- 172.21.15.31:0/2389561331 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f32280cbd10 con 0x7f322c076950 2023-07-29T16:33:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.153+0000 7f32474fbb80 1 -- 172.21.15.31:0/3133685409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d77df0 msgr2=0x55c3f4d787b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.153+0000 7f32474fbb80 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d77df0 0x55c3f4d787b0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f322400b070 tx=0x7f3224009c40 comp rx=0 tx=0).stop 2023-07-29T16:33:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.153+0000 7f323219c700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f32180a08e0 0x7f32180a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.154+0000 7f32474fbb80 1 -- 172.21.15.31:0/3133685409 shutdown_connections 2023-07-29T16:33:39.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.154+0000 7f32474fbb80 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d79160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.154+0000 7f32474fbb80 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d778b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.154+0000 7f32474fbb80 1 --2- 172.21.15.31:0/3133685409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d77df0 0x55c3f4d787b0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.154+0000 7f32474fbb80 1 -- 172.21.15.31:0/3133685409 >> 172.21.15.31:0/3133685409 conn(0x55c3f4d6cfd0 msgr2=0x55c3f4d72300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.154+0000 7f32474fbb80 1 -- 172.21.15.31:0/3133685409 shutdown_connections 2023-07-29T16:33:39.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.155+0000 7f32474fbb80 1 -- 172.21.15.31:0/3133685409 wait complete. 2023-07-29T16:33:39.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.155+0000 7f32474fbb80 1 -- start start 2023-07-29T16:33:39.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.156+0000 7f32474fbb80 1 Processor -- start 2023-07-29T16:33:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.156+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d80f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.157+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d81470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.157+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d884b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.157+0000 7f32474fbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4c654b0 con 0x55c3f4d77df0 2023-07-29T16:33:39.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.157+0000 7f32474fbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4c65150 con 0x55c3f4d78cf0 2023-07-29T16:33:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.157+0000 7f32474fbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4b4c8c0 con 0x55c3f4bd36b0 2023-07-29T16:33:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.157+0000 7f32474fbb80 10 client.0 _ll_register_callbacks cb 0x55c3f4c6e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d80f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d80f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:44270/0 (socket says 172.21.15.31:44270) 2023-07-29T16:33:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 -- 172.21.15.31:0/1174340623 learned_addr learned my addr 172.21.15.31:0/1174340623 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 -- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 msgr2=0x55c3f4d884b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d884b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 -- 172.21.15.31:0/1174340623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 msgr2=0x55c3f4d81470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d81470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.158+0000 7f323199b700 1 -- 172.21.15.31:0/1174340623 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c078480 con 0x55c3f4bd36b0 2023-07-29T16:33:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.159+0000 7f323199b700 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d80f30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3224009fd0 tx=0x7f3224007980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdee81b41353fdc1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:39.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.159+0000 7f32157fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:33:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.159+0000 7f32157fa700 1 -- 172.21.15.31:0/1174340623 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224029070 con 0x55c3f4bd36b0 2023-07-29T16:33:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.159+0000 7f32157fa700 1 -- 172.21.15.31:0/1174340623 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f32240052d0 con 0x55c3f4bd36b0 2023-07-29T16:33:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.159+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3f4cf3cb0 con 0x55c3f4bd36b0 2023-07-29T16:33:39.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.161+0000 7f323219c700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f32180a08e0 0x7f32180a2d90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f321c009d60 tx=0x7f321c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.164+0000 7f32157fa700 1 -- 172.21.15.31:0/1174340623 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f322400e040 con 0x55c3f4bd36b0 2023-07-29T16:33:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.165+0000 7f32474fbb80 10 client.30280 fetch_fsmap learned FSMap version 41 2023-07-29T16:33:39.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.165+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x55c3f4adfc00 con 0x55c3f4bd36b0 2023-07-29T16:33:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32157fa700 1 -- 172.21.15.31:0/1174340623 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c3f4cf3cb0 con 0x55c3f4bd36b0 2023-07-29T16:33:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32157fa700 1 -- 172.21.15.31:0/1174340623 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f3224022070 con 0x55c3f4bd36b0 2023-07-29T16:33:39.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32474fbb80 10 client.30280 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:33:39.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32474fbb80 10 client.30280 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:39.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32474fbb80 20 client.30280 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:39.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32474fbb80 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4d9d810 0x55c3f4d9fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32474fbb80 4 client.30280 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:33:39.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.166+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3f4d46d10 con 0x55c3f4d9d810 2023-07-29T16:33:39.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.167+0000 7f323119a700 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4d9d810 0x55c3f4d9fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.168+0000 7f323119a700 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4d9d810 0x55c3f4d9fc20 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=171722d083811b1b server_cookie=f38ad65971d58e52 in_seq=0 out_seq=0 2023-07-29T16:33:39.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.168+0000 7f32157fa700 10 client.30280 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.169+0000 7f32157fa700 1 -- 172.21.15.31:0/1174340623 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c3f4d46d10 con 0x55c3f4d9d810 2023-07-29T16:33:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.169+0000 7f32157fa700 10 client.30280 handle_command_reply: tid=0 2023-07-29T16:33:39.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.171+0000 7f32474fbb80 1 client.30280 shutdown 2023-07-29T16:33:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 msgr2=0x55c3f4d80f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d80f30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3224009fd0 tx=0x7f3224007980 comp rx=0 tx=0).stop 2023-07-29T16:33:39.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 shutdown_connections 2023-07-29T16:33:39.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4d9d810 0x55c3f4d9fc20 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d884b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4bd36b0 0x55c3f4d80f30 secure :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3224009fd0 tx=0x7f3224007980 comp rx=0 tx=0).stop 2023-07-29T16:33:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.178+0000 7f32474fbb80 1 --2- 172.21.15.31:0/1174340623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d81470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.179+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 >> 172.21.15.31:0/1174340623 conn(0x55c3f4d6cfd0 msgr2=0x55c3f4d71a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.179+0000 7f32157fa700 0 client.30280 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.179+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 shutdown_connections 2023-07-29T16:33:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.179+0000 7f32474fbb80 1 -- 172.21.15.31:0/1174340623 wait complete. 2023-07-29T16:33:39.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.180+0000 7f32474fbb80 20 client.30280 trim_cache size 0 max 16384 2023-07-29T16:33:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.214+0000 7f32474fbb80 1 Processor -- start 2023-07-29T16:33:39.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 -- start start 2023-07-29T16:33:39.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d77df0 0x55c3f4d863c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d9f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d7f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4d6f080 con 0x55c3f4d9fd60 2023-07-29T16:33:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4d73620 con 0x55c3f4d78cf0 2023-07-29T16:33:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.215+0000 7f32474fbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4d731b0 con 0x55c3f4d77df0 2023-07-29T16:33:39.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.216+0000 7f323199b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d77df0 0x55c3f4d863c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.216+0000 7f323219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d7f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.216+0000 7f323219c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d7f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49246/0 (socket says 172.21.15.31:49246) 2023-07-29T16:33:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.216+0000 7f323219c700 1 -- 172.21.15.31:0/925696955 learned_addr learned my addr 172.21.15.31:0/925696955 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.217+0000 7f323219c700 1 -- 172.21.15.31:0/925696955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d77df0 msgr2=0x55c3f4d863c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:39.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.217+0000 7f323219c700 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d77df0 0x55c3f4d863c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.217+0000 7f323219c700 1 -- 172.21.15.31:0/925696955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 msgr2=0x55c3f4d9f820 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:39.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.217+0000 7f323219c700 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d9f820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.217+0000 7f323219c700 1 -- 172.21.15.31:0/925696955 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3f4b1d1f0 con 0x55c3f4d9fd60 2023-07-29T16:33:39.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.218+0000 7f323219c700 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d7f0d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f321c011c40 tx=0x7f321c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbead2d77c28b5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.222+0000 7f3220bff700 1 -- 172.21.15.31:0/925696955 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c003680 con 0x55c3f4d9fd60 2023-07-29T16:33:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.222+0000 7f3220bff700 1 -- 172.21.15.31:0/925696955 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f321c003ca0 con 0x55c3f4d9fd60 2023-07-29T16:33:39.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.222+0000 7f3220bff700 1 -- 172.21.15.31:0/925696955 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c026d70 con 0x55c3f4d9fd60 2023-07-29T16:33:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 -- 172.21.15.31:0/925696955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 msgr2=0x55c3f4d7f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d7f0d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f321c011c40 tx=0x7f321c014000 comp rx=0 tx=0).stop 2023-07-29T16:33:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 -- 172.21.15.31:0/925696955 shutdown_connections 2023-07-29T16:33:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d7f0d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d77df0 0x55c3f4d863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 --2- 172.21.15.31:0/925696955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d9f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.224+0000 7f32474fbb80 1 -- 172.21.15.31:0/925696955 >> 172.21.15.31:0/925696955 conn(0x55c3f4d6cfd0 msgr2=0x55c3f4d72190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:39.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.228+0000 7f32474fbb80 1 -- 172.21.15.31:0/925696955 shutdown_connections 2023-07-29T16:33:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.228+0000 7f32474fbb80 1 -- 172.21.15.31:0/925696955 wait complete. 2023-07-29T16:33:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.229+0000 7f32474fbb80 1 -- start start 2023-07-29T16:33:39.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.229+0000 7f32474fbb80 1 Processor -- start 2023-07-29T16:33:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d86370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d84980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d84f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4d856a0 con 0x55c3f4d77df0 2023-07-29T16:33:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4d85810 con 0x55c3f4d78cf0 2023-07-29T16:33:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3f4def3a0 con 0x55c3f4d9fd60 2023-07-29T16:33:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.230+0000 7f32474fbb80 10 client.0 _ll_register_callbacks cb 0x55c3f4c6e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.235+0000 7f323199b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d86370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.235+0000 7f323199b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d86370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49250/0 (socket says 172.21.15.31:49250) 2023-07-29T16:33:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.235+0000 7f323199b700 1 -- 172.21.15.31:0/391123040 learned_addr learned my addr 172.21.15.31:0/391123040 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.239+0000 7f323219c700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d84f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.239+0000 7f323119a700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d84980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.241+0000 7f323219c700 1 -- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 msgr2=0x55c3f4d84980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.241+0000 7f323219c700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d84980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.241+0000 7f323219c700 1 -- 172.21.15.31:0/391123040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 msgr2=0x55c3f4d86370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.241+0000 7f323219c700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d86370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.241+0000 7f323219c700 1 -- 172.21.15.31:0/391123040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321c017070 con 0x55c3f4d9fd60 2023-07-29T16:33:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.246+0000 7f323219c700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d84f90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f321c009940 tx=0x7f321c0266e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=228af84e993c98ec server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.246+0000 7f32157fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:33:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.248+0000 7f32157fa700 1 -- 172.21.15.31:0/391123040 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c02b070 con 0x55c3f4d9fd60 2023-07-29T16:33:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.249+0000 7f32157fa700 1 -- 172.21.15.31:0/391123040 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f321c026d00 con 0x55c3f4d9fd60 2023-07-29T16:33:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.251+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f322400bab0 con 0x55c3f4d9fd60 2023-07-29T16:33:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.254+0000 7f32157fa700 1 -- 172.21.15.31:0/391123040 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f321c003680 con 0x55c3f4d9fd60 2023-07-29T16:33:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.257+0000 7f32474fbb80 10 client.30283 fetch_fsmap learned FSMap version 41 2023-07-29T16:33:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.257+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x55c3f4af5250 con 0x55c3f4d9fd60 2023-07-29T16:33:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32157fa700 1 -- 172.21.15.31:0/391123040 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f322400bab0 con 0x55c3f4d9fd60 2023-07-29T16:33:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32157fa700 1 -- 172.21.15.31:0/391123040 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f321c037070 con 0x55c3f4d9fd60 2023-07-29T16:33:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32474fbb80 10 client.30283 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:33:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32474fbb80 10 client.30283 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32474fbb80 20 client.30283 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32474fbb80 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4df0b60 0x55c3f4df2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32474fbb80 4 client.30283 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:33:39.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.260+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c3f4cc1ce0 con 0x55c3f4df0b60 2023-07-29T16:33:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.261+0000 7f323199b700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4df0b60 0x55c3f4df2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:39.263 INFO:tasks.workunit.client.0.smithi031.stdout: 10 135574 37.20 MB/sec execute 83 sec latency 141.347 ms 2023-07-29T16:33:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.268+0000 7f323199b700 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4df0b60 0x55c3f4df2f70 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52da42fda360b31 server_cookie=b55da76459bd97c7 in_seq=0 out_seq=0 2023-07-29T16:33:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.271+0000 7f32157fa700 10 client.30283 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:39.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.609+0000 7f32157fa700 1 -- 172.21.15.31:0/391123040 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c3f4cc1ce0 con 0x55c3f4df0b60 2023-07-29T16:33:39.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.609+0000 7f32157fa700 10 client.30283 handle_command_reply: tid=0 2023-07-29T16:33:39.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.609+0000 7f32474fbb80 1 client.30283 shutdown 2023-07-29T16:33:39.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d9fd60 msgr2=0x55c3f4d84f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d84f90 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f321c009940 tx=0x7f321c0266e0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 shutdown_connections 2023-07-29T16:33:39.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c3f4df0b60 0x55c3f4df2f70 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c3f4d9fd60 0x55c3f4d84f90 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c3f4d77df0 0x55c3f4d86370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 --2- 172.21.15.31:0/391123040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c3f4d78cf0 0x55c3f4d84980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 >> 172.21.15.31:0/391123040 conn(0x55c3f4d6cfd0 msgr2=0x55c3f4d7f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:39.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32157fa700 0 client.30283 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:39.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.610+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 shutdown_connections 2023-07-29T16:33:39.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.611+0000 7f32474fbb80 1 -- 172.21.15.31:0/391123040 wait complete. 2023-07-29T16:33:39.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.612+0000 7f32474fbb80 20 client.30283 trim_cache size 0 max 16384 2023-07-29T16:33:39.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:33:39.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:33:39.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:33:39.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:33:39.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.612+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f32180a08e0 msgr2=0x7f32180a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.612+0000 7f3233bff700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f32180a08e0 0x7f32180a2d90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f321c009d60 tx=0x7f321c009260 comp rx=0 tx=0).stop 2023-07-29T16:33:39.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.612+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 msgr2=0x7f322c06fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:39.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.612+0000 7f3233bff700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 0x7f322c06fc70 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f322800d950 tx=0x7f322800dd10 comp rx=0 tx=0).stop 2023-07-29T16:33:39.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 shutdown_connections 2023-07-29T16:33:39.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f32180a08e0 0x7f32180a2d90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f322c077300 0x7f322c070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f322c075710 0x7f322c06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 --2- 172.21.15.31:0/2389561331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f322c076950 0x7f322c06fc70 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 >> 172.21.15.31:0/2389561331 conn(0x7f322c0fdba0 msgr2=0x7f322c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 shutdown_connections 2023-07-29T16:33:39.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:39.614+0000 7f3233bff700 1 -- 172.21.15.31:0/2389561331 wait complete. 2023-07-29T16:33:40.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 135983 37.04 MB/sec execute 84 sec latency 239.926 ms 2023-07-29T16:33:40.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:40 smithi031 ceph-mon[95504]: pgmap v917: 97 pgs: 97 active+clean; 772 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 17 MiB/s wr, 885 op/s 2023-07-29T16:33:41.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:40 smithi039 ceph-mon[128858]: pgmap v917: 97 pgs: 97 active+clean; 772 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 17 MiB/s wr, 885 op/s 2023-07-29T16:33:41.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:40 smithi156 ceph-mon[129245]: pgmap v917: 97 pgs: 97 active+clean; 772 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 17 MiB/s wr, 885 op/s 2023-07-29T16:33:41.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 136579 36.91 MB/sec execute 85 sec latency 288.823 ms 2023-07-29T16:33:42.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 137290 36.95 MB/sec execute 86 sec latency 171.201 ms 2023-07-29T16:33:43.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:42 smithi039 ceph-mon[128858]: pgmap v918: 97 pgs: 97 active+clean; 781 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 19 MiB/s wr, 825 op/s 2023-07-29T16:33:43.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:42 smithi156 ceph-mon[129245]: pgmap v918: 97 pgs: 97 active+clean; 781 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 19 MiB/s wr, 825 op/s 2023-07-29T16:33:43.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:42 smithi031 ceph-mon[95504]: pgmap v918: 97 pgs: 97 active+clean; 781 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 19 MiB/s wr, 825 op/s 2023-07-29T16:33:43.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 137989 37.02 MB/sec execute 87 sec latency 112.573 ms 2023-07-29T16:33:44.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 138774 36.98 MB/sec execute 88 sec latency 57.070 ms 2023-07-29T16:33:45.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:44 smithi039 ceph-mon[128858]: pgmap v919: 97 pgs: 97 active+clean; 681 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2023-07-29T16:33:45.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:44 smithi156 ceph-mon[129245]: pgmap v919: 97 pgs: 97 active+clean; 681 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2023-07-29T16:33:45.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:44 smithi031 ceph-mon[95504]: pgmap v919: 97 pgs: 97 active+clean; 681 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 19 MiB/s wr, 1.01k op/s 2023-07-29T16:33:45.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 139546 37.02 MB/sec execute 89 sec latency 64.287 ms 2023-07-29T16:33:46.264 INFO:tasks.workunit.client.0.smithi031.stdout: 10 140269 37.13 MB/sec execute 90 sec latency 168.613 ms 2023-07-29T16:33:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:46 smithi039 ceph-mon[128858]: pgmap v920: 97 pgs: 97 active+clean; 650 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 17 MiB/s wr, 977 op/s 2023-07-29T16:33:47.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:46 smithi156 ceph-mon[129245]: pgmap v920: 97 pgs: 97 active+clean; 650 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 17 MiB/s wr, 977 op/s 2023-07-29T16:33:47.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:46 smithi031 ceph-mon[95504]: pgmap v920: 97 pgs: 97 active+clean; 650 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 17 MiB/s wr, 977 op/s 2023-07-29T16:33:47.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 140825 37.06 MB/sec execute 91 sec latency 375.650 ms 2023-07-29T16:33:48.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 141222 36.88 MB/sec execute 92 sec latency 376.597 ms 2023-07-29T16:33:48.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:48 smithi031 ceph-mon[95504]: pgmap v921: 97 pgs: 97 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 16 MiB/s wr, 908 op/s 2023-07-29T16:33:48.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:48 smithi039 ceph-mon[128858]: pgmap v921: 97 pgs: 97 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 16 MiB/s wr, 908 op/s 2023-07-29T16:33:48.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:48 smithi156 ceph-mon[129245]: pgmap v921: 97 pgs: 97 active+clean; 653 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 16 MiB/s wr, 908 op/s 2023-07-29T16:33:49.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 141811 36.87 MB/sec execute 93 sec latency 86.377 ms 2023-07-29T16:33:50.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 142444 36.75 MB/sec execute 94 sec latency 80.917 ms 2023-07-29T16:33:50.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:50 smithi031 ceph-mon[95504]: pgmap v922: 97 pgs: 97 active+clean; 660 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 15 MiB/s wr, 888 op/s 2023-07-29T16:33:51.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:50 smithi039 ceph-mon[128858]: pgmap v922: 97 pgs: 97 active+clean; 660 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 15 MiB/s wr, 888 op/s 2023-07-29T16:33:51.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:50 smithi156 ceph-mon[129245]: pgmap v922: 97 pgs: 97 active+clean; 660 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 15 MiB/s wr, 888 op/s 2023-07-29T16:33:51.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 143257 36.82 MB/sec execute 95 sec latency 60.660 ms 2023-07-29T16:33:52.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 144049 36.90 MB/sec execute 96 sec latency 73.268 ms 2023-07-29T16:33:52.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.854+0000 7fd1b07ff700 1 -- 172.21.15.31:0/2701123072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a8100e80 msgr2=0x7fd1a8107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:52.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.854+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/2701123072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a8100e80 0x7fd1a8107720 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fd198009be0 tx=0x7fd198009ef0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 -- 172.21.15.31:0/2701123072 shutdown_connections 2023-07-29T16:33:52.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/2701123072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a8100e80 0x7fd1a8107720 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/2701123072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a80ff240 0x7fd1a80ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/2701123072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a81004d0 0x7fd1a8100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 -- 172.21.15.31:0/2701123072 >> 172.21.15.31:0/2701123072 conn(0x7fd1a80fa840 msgr2=0x7fd1a80fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:52.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 -- 172.21.15.31:0/2701123072 shutdown_connections 2023-07-29T16:33:52.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.855+0000 7fd1b07ff700 1 -- 172.21.15.31:0/2701123072 wait complete. 2023-07-29T16:33:52.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 Processor -- start 2023-07-29T16:33:52.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 -- start start 2023-07-29T16:33:52.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a80ff240 0x7fd1a8193f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a81004d0 0x7fd1a81944c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 0x7fd1a8195390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a81959f0 con 0x7fd1a81004d0 2023-07-29T16:33:52.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8195b60 con 0x7fd1a80ff240 2023-07-29T16:33:52.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1b07ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8195d20 con 0x7fd1a8100e80 2023-07-29T16:33:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1ae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a80ff240 0x7fd1a8193f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.857+0000 7fd1aed9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 0x7fd1a8195390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1ae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a80ff240 0x7fd1a8193f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35964/0 (socket says 172.21.15.31:35964) 2023-07-29T16:33:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1ae59b700 1 -- 172.21.15.31:0/3903611345 learned_addr learned my addr 172.21.15.31:0/3903611345 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 0x7fd1a8195390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35996/0 (socket says 172.21.15.31:35996) 2023-07-29T16:33:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 -- 172.21.15.31:0/3903611345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a80ff240 msgr2=0x7fd1a8193f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a80ff240 0x7fd1a8193f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 -- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a81004d0 msgr2=0x7fd1a81944c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a81004d0 0x7fd1a81944c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 -- 172.21.15.31:0/3903611345 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd198009820 con 0x7fd1a8100e80 2023-07-29T16:33:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.858+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 0x7fd1a8195390 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd198006010 tx=0x7fd198005930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:52.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.859+0000 7fd19f7fe700 1 -- 172.21.15.31:0/3903611345 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19800bb90 con 0x7fd1a8100e80 2023-07-29T16:33:52.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.859+0000 7fd19f7fe700 1 -- 172.21.15.31:0/3903611345 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd19800bcf0 con 0x7fd1a8100e80 2023-07-29T16:33:52.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.859+0000 7fd19f7fe700 1 -- 172.21.15.31:0/3903611345 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198010660 con 0x7fd1a8100e80 2023-07-29T16:33:52.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.859+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a8195fa0 con 0x7fd1a8100e80 2023-07-29T16:33:52.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.859+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a81965d0 con 0x7fd1a8100e80 2023-07-29T16:33:52.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.865+0000 7fd1c40eab80 1 Processor -- start 2023-07-29T16:33:52.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.865+0000 7fd19f7fe700 1 -- 172.21.15.31:0/3903611345 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd1980107c0 con 0x7fd1a8100e80 2023-07-29T16:33:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.865+0000 7fd1c40eab80 1 -- start start 2023-07-29T16:33:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 0x5626cb1a97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1009b0 con 0x5626cb0046b0 2023-07-29T16:33:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb0fe370 con 0x5626cb1a8df0 2023-07-29T16:33:52.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb0f9c60 con 0x5626cb1a9cf0 2023-07-29T16:33:52.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.866+0000 7fd1ae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 0x5626cb1a97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 0x5626cb1a97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35980/0 (socket says 172.21.15.31:35980) 2023-07-29T16:33:52.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 -- 172.21.15.31:0/1681183780 learned_addr learned my addr 172.21.15.31:0/1681183780 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:52.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 -- 172.21.15.31:0/1681183780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a9cf0 msgr2=0x5626cb1aa160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:33:52.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 -- 172.21.15.31:0/1681183780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 msgr2=0x5626cb1a88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 -- 172.21.15.31:0/1681183780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626caf27d80 con 0x5626cb1a8df0 2023-07-29T16:33:52.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 0x5626cb1a97b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e270 tx=0x7fd1a0009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87daf2d54f8f8777 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:52.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.867+0000 7fd19cff9700 1 -- 172.21.15.31:0/1681183780 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0014680 con 0x5626cb1a8df0 2023-07-29T16:33:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.868+0000 7fd19cff9700 1 -- 172.21.15.31:0/1681183780 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1a000f040 con 0x5626cb1a8df0 2023-07-29T16:33:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.868+0000 7fd19cff9700 1 -- 172.21.15.31:0/1681183780 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a00139e0 con 0x5626cb1a8df0 2023-07-29T16:33:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.868+0000 7fd19f7fe700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1940a0840 0x7fd1940a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.868+0000 7fd19f7fe700 1 -- 172.21.15.31:0/3903611345 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd1980cd290 con 0x7fd1a8100e80 2023-07-29T16:33:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 -- 172.21.15.31:0/1681183780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 msgr2=0x5626cb1a97b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 0x5626cb1a97b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e270 tx=0x7fd1a0009770 comp rx=0 tx=0).stop 2023-07-29T16:33:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 -- 172.21.15.31:0/1681183780 shutdown_connections 2023-07-29T16:33:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1a88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/1681183780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a8df0 0x5626cb1a97b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 -- 172.21.15.31:0/1681183780 >> 172.21.15.31:0/1681183780 conn(0x5626cb19dfd0 msgr2=0x5626cb1a3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:52.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1add9a700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1940a0840 0x7fd1940a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 -- 172.21.15.31:0/1681183780 shutdown_connections 2023-07-29T16:33:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.869+0000 7fd1c40eab80 1 -- 172.21.15.31:0/1681183780 wait complete. 2023-07-29T16:33:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.870+0000 7fd1c40eab80 1 -- start start 2023-07-29T16:33:52.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.870+0000 7fd1add9a700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1940a0840 0x7fd1940a2cf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4005d90 tx=0x7fd1a4009d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.870+0000 7fd1c40eab80 1 Processor -- start 2023-07-29T16:33:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb0964b0 con 0x5626cb0046b0 2023-07-29T16:33:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb096150 con 0x5626cb1a9cf0 2023-07-29T16:33:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626caf7d8c0 con 0x5626cb1a8df0 2023-07-29T16:33:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1c40eab80 10 client.0 _ll_register_callbacks cb 0x5626cb09f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1add9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1add9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36008/0 (socket says 172.21.15.31:36008) 2023-07-29T16:33:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.871+0000 7fd1add9a700 1 -- 172.21.15.31:0/2624937830 learned_addr learned my addr 172.21.15.31:0/2624937830 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2624937830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 msgr2=0x5626cb1b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2624937830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 msgr2=0x5626cb1b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1b1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2624937830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a0009da0 con 0x5626cb1a9cf0 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b93f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd1980cad80 tx=0x7fd19809a470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3721a746287125 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.872+0000 7fd191ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd191ffb700 1 -- 172.21.15.31:0/2624937830 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19809a110 con 0x5626cb1a9cf0 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd191ffb700 1 -- 172.21.15.31:0/2624937830 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd19809aa20 con 0x5626cb1a9cf0 2023-07-29T16:33:52.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5626cb124cb0 con 0x5626cb1a9cf0 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd191ffb700 1 -- 172.21.15.31:0/2624937830 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1980c4e70 con 0x5626cb1a9cf0 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd1c40eab80 10 client.30281 fetch_fsmap learned FSMap version 41 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x5626caf10c00 con 0x5626cb1a9cf0 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd191ffb700 1 -- 172.21.15.31:0/2624937830 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5626cb124cb0 con 0x5626cb1a9cf0 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd191ffb700 1 -- 172.21.15.31:0/2624937830 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fd1980a2a70 con 0x5626cb1a9cf0 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd1c40eab80 10 client.30281 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:33:52.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd1c40eab80 10 client.30281 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.873+0000 7fd1c40eab80 20 client.30281 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.874+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb1bd810 0x5626cb1bfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.874+0000 7fd1c40eab80 4 client.30281 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.874+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5626cb177d10 con 0x5626cb1bd810 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.874+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb1bd810 0x5626cb1bfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.874+0000 7fd191ffb700 10 client.30281 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:52.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.875+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb1bd810 0x5626cb1bfbf0 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b32bf81dcaafd7a6 server_cookie=d9b1bf1ce233990 in_seq=0 out_seq=0 2023-07-29T16:33:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.877+0000 7fd191ffb700 1 -- 172.21.15.31:0/2624937830 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5626cb177d10 con 0x5626cb1bd810 2023-07-29T16:33:52.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd191ffb700 10 client.30281 handle_command_reply: tid=0 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 client.30281 shutdown 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 msgr2=0x5626cb1b93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b93f0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fd1980cad80 tx=0x7fd19809a470 comp rx=0 tx=0).stop 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 shutdown_connections 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb1bd810 0x5626cb1bfbf0 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b93f0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb0046b0 0x5626cb1b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2624937830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 >> 172.21.15.31:0/2624937830 conn(0x5626cb19dfd0 msgr2=0x5626cb1a2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:52.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.878+0000 7fd191ffb700 0 client.30281 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:52.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.879+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 shutdown_connections 2023-07-29T16:33:52.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.879+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2624937830 wait complete. 2023-07-29T16:33:52.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.879+0000 7fd1c40eab80 20 client.30281 trim_cache size 0 max 16384 2023-07-29T16:33:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 Processor -- start 2023-07-29T16:33:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 -- start start 2023-07-29T16:33:52.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb0046b0 0x5626cb1b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1a3f50 con 0x5626cb1a9cf0 2023-07-29T16:33:52.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1a0050 con 0x5626cb0046b0 2023-07-29T16:33:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.900+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1a45f0 con 0x5626cb1a8df0 2023-07-29T16:33:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60504/0 (socket says 172.21.15.31:60504) 2023-07-29T16:33:52.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2626432932 learned_addr learned my addr 172.21.15.31:0/2626432932 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1add9a700 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2626432932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 msgr2=0x5626cb1b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2626432932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb0046b0 msgr2=0x5626cb1b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb0046b0 0x5626cb1b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.901+0000 7fd1aed9c700 1 -- 172.21.15.31:0/2626432932 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5626caf4e1f0 con 0x5626cb1a9cf0 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.902+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa0c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd19809a720 tx=0x7fd198098fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46dd34f1ae0a3198 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.902+0000 7fd19d7fa700 1 -- 172.21.15.31:0/2626432932 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19809aa20 con 0x5626cb1a9cf0 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.902+0000 7fd19d7fa700 1 -- 172.21.15.31:0/2626432932 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd19809ea50 con 0x5626cb1a9cf0 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.902+0000 7fd19d7fa700 1 -- 172.21.15.31:0/2626432932 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1980a2950 con 0x5626cb1a9cf0 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2626432932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 msgr2=0x5626cb1aa0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa0c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd19809a720 tx=0x7fd198098fd0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2626432932 shutdown_connections 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a9cf0 0x5626cb1aa0c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb0046b0 0x5626cb1b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/2626432932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb1a8df0 0x5626cb1b93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.903+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2626432932 >> 172.21.15.31:0/2626432932 conn(0x5626cb19dfd0 msgr2=0x5626cb1a32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.904+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2626432932 shutdown_connections 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.904+0000 7fd1c40eab80 1 -- 172.21.15.31:0/2626432932 wait complete. 2023-07-29T16:33:52.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.904+0000 7fd1c40eab80 1 -- start start 2023-07-29T16:33:52.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.904+0000 7fd1c40eab80 1 Processor -- start 2023-07-29T16:33:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 0x5626cb1b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a8df0 0x5626cb1b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1b8bf0 con 0x5626cb1a8df0 2023-07-29T16:33:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1b8d60 con 0x5626cb1a9cf0 2023-07-29T16:33:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626cb1b8f20 con 0x5626cb0046b0 2023-07-29T16:33:52.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1c40eab80 10 client.0 _ll_register_callbacks cb 0x5626cb09f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:33:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1ae59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 0x5626cb1b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1ae59b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 0x5626cb1b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36032/0 (socket says 172.21.15.31:36032) 2023-07-29T16:33:52.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1ae59b700 1 -- 172.21.15.31:0/491840939 learned_addr learned my addr 172.21.15.31:0/491840939 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:33:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.905+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1add9a700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a8df0 0x5626cb1b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1ae59b700 1 -- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 msgr2=0x5626cb1b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1ae59b700 1 -- 172.21.15.31:0/491840939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a8df0 msgr2=0x5626cb1b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:33:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a8df0 0x5626cb1b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1ae59b700 1 -- 172.21.15.31:0/491840939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a0009da0 con 0x5626cb0046b0 2023-07-29T16:33:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1aed9c700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:33:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1add9a700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a8df0 0x5626cb1b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:33:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd1ae59b700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 0x5626cb1b3530 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009d70 tx=0x7fd1a00134b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bae05238cf00b66 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:33:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.906+0000 7fd191ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:33:52.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd191ffb700 1 -- 172.21.15.31:0/491840939 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0003ac0 con 0x5626cb0046b0 2023-07-29T16:33:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd191ffb700 1 -- 172.21.15.31:0/491840939 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd1a0022040 con 0x5626cb0046b0 2023-07-29T16:33:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1a000f040 con 0x5626cb0046b0 2023-07-29T16:33:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd191ffb700 1 -- 172.21.15.31:0/491840939 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1a0003c20 con 0x5626cb0046b0 2023-07-29T16:33:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd1c40eab80 10 client.30295 fetch_fsmap learned FSMap version 41 2023-07-29T16:33:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x5626caf26250 con 0x5626cb0046b0 2023-07-29T16:33:52.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.907+0000 7fd191ffb700 1 -- 172.21.15.31:0/491840939 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1a000f040 con 0x5626cb0046b0 2023-07-29T16:33:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd191ffb700 1 -- 172.21.15.31:0/491840939 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7fd1a0013ac0 con 0x5626cb0046b0 2023-07-29T16:33:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1c40eab80 10 client.30295 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:33:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1c40eab80 10 client.30295 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:33:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1c40eab80 20 client.30295 populate_metadata read hostname 'smithi031' 2023-07-29T16:33:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb2211f0 0x5626cb223600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:33:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1c40eab80 4 client.30295 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:33:52.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5626cb0f2ce0 con 0x5626cb2211f0 2023-07-29T16:33:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.908+0000 7fd1add9a700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb2211f0 0x5626cb223600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:33:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.909+0000 7fd1add9a700 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb2211f0 0x5626cb223600 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e55ffe5deced5e97 server_cookie=ffc07421e912d04d in_seq=0 out_seq=0 2023-07-29T16:33:52.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:52.909+0000 7fd191ffb700 10 client.30295 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:53.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:52 smithi039 ceph-mon[128858]: pgmap v923: 97 pgs: 97 active+clean; 665 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 9.1 MiB/s wr, 722 op/s 2023-07-29T16:33:53.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:52 smithi156 ceph-mon[129245]: pgmap v923: 97 pgs: 97 active+clean; 665 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 9.1 MiB/s wr, 722 op/s 2023-07-29T16:33:53.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:52 smithi031 ceph-mon[95504]: pgmap v923: 97 pgs: 97 active+clean; 665 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 9.1 MiB/s wr, 722 op/s 2023-07-29T16:33:53.266 INFO:tasks.workunit.client.0.smithi031.stdout: 10 144263 36.67 MB/sec execute 97 sec latency 424.509 ms 2023-07-29T16:33:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.281+0000 7fd191ffb700 1 -- 172.21.15.31:0/491840939 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5626cb0f2ce0 con 0x5626cb2211f0 2023-07-29T16:33:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.281+0000 7fd191ffb700 10 client.30295 handle_command_reply: tid=0 2023-07-29T16:33:53.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.281+0000 7fd1c40eab80 1 client.30295 shutdown 2023-07-29T16:33:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 msgr2=0x5626cb1b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 0x5626cb1b3530 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009d70 tx=0x7fd1a00134b0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 shutdown_connections 2023-07-29T16:33:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5626cb2211f0 0x5626cb223600 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5626cb1a9cf0 0x5626cb1b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626cb0046b0 0x5626cb1b3530 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 --2- 172.21.15.31:0/491840939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626cb1a8df0 0x5626cb1b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 >> 172.21.15.31:0/491840939 conn(0x5626cb19dfd0 msgr2=0x5626cb1acac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd191ffb700 0 client.30295 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:33:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 shutdown_connections 2023-07-29T16:33:53.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.282+0000 7fd1c40eab80 1 -- 172.21.15.31:0/491840939 wait complete. 2023-07-29T16:33:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.283+0000 7fd1c40eab80 20 client.30295 trim_cache size 0 max 16384 2023-07-29T16:33:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:33:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:33:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:33:53.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:33:53.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.283+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1940a0840 msgr2=0x7fd1940a2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:53.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.283+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1940a0840 0x7fd1940a2cf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4005d90 tx=0x7fd1a4009d80 comp rx=0 tx=0).stop 2023-07-29T16:33:53.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.283+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 msgr2=0x7fd1a8195390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:33:53.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.283+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 0x7fd1a8195390 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd198006010 tx=0x7fd198005930 comp rx=0 tx=0).stop 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 shutdown_connections 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd1940a0840 0x7fd1940a2cf0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1a8100e80 0x7fd1a8195390 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1a80ff240 0x7fd1a8193f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 --2- 172.21.15.31:0/3903611345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1a81004d0 0x7fd1a81944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 >> 172.21.15.31:0/3903611345 conn(0x7fd1a80fa840 msgr2=0x7fd1a8103960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:33:53.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 shutdown_connections 2023-07-29T16:33:53.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:33:53.284+0000 7fd1b07ff700 1 -- 172.21.15.31:0/3903611345 wait complete. 2023-07-29T16:33:54.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 144843 36.62 MB/sec execute 98 sec latency 198.493 ms 2023-07-29T16:33:55.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:54 smithi039 ceph-mon[128858]: pgmap v924: 97 pgs: 97 active+clean; 679 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.0 MiB/s wr, 806 op/s 2023-07-29T16:33:55.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:54 smithi156 ceph-mon[129245]: pgmap v924: 97 pgs: 97 active+clean; 679 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.0 MiB/s wr, 806 op/s 2023-07-29T16:33:55.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:54 smithi031 ceph-mon[95504]: pgmap v924: 97 pgs: 97 active+clean; 679 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 9.0 MiB/s wr, 806 op/s 2023-07-29T16:33:55.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 145559 36.64 MB/sec execute 99 sec latency 69.692 ms 2023-07-29T16:33:56.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 146208 36.62 MB/sec execute 100 sec latency 97.371 ms 2023-07-29T16:33:57.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:56 smithi039 ceph-mon[128858]: pgmap v925: 97 pgs: 97 active+clean; 682 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 10 MiB/s wr, 747 op/s 2023-07-29T16:33:57.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:56 smithi156 ceph-mon[129245]: pgmap v925: 97 pgs: 97 active+clean; 682 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 10 MiB/s wr, 747 op/s 2023-07-29T16:33:57.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:56 smithi031 ceph-mon[95504]: pgmap v925: 97 pgs: 97 active+clean; 682 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 10 MiB/s wr, 747 op/s 2023-07-29T16:33:57.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 146777 36.55 MB/sec execute 101 sec latency 100.050 ms 2023-07-29T16:33:58.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 147509 36.56 MB/sec execute 102 sec latency 97.332 ms 2023-07-29T16:33:58.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:33:58 smithi031 ceph-mon[95504]: pgmap v926: 97 pgs: 97 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 673 op/s 2023-07-29T16:33:58.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:33:58 smithi039 ceph-mon[128858]: pgmap v926: 97 pgs: 97 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 673 op/s 2023-07-29T16:33:58.573 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:33:58 smithi156 ceph-mon[129245]: pgmap v926: 97 pgs: 97 active+clean; 679 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 673 op/s 2023-07-29T16:33:59.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 148251 36.51 MB/sec execute 103 sec latency 184.756 ms 2023-07-29T16:34:00.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 149001 36.68 MB/sec execute 104 sec latency 59.531 ms 2023-07-29T16:34:00.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:00 smithi031 ceph-mon[95504]: pgmap v927: 97 pgs: 97 active+clean; 656 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 11 MiB/s wr, 784 op/s 2023-07-29T16:34:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:00 smithi039 ceph-mon[128858]: pgmap v927: 97 pgs: 97 active+clean; 656 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 11 MiB/s wr, 784 op/s 2023-07-29T16:34:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:00 smithi156 ceph-mon[129245]: pgmap v927: 97 pgs: 97 active+clean; 656 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 11 MiB/s wr, 784 op/s 2023-07-29T16:34:01.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 149628 36.66 MB/sec execute 105 sec latency 74.738 ms 2023-07-29T16:34:02.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 150211 36.56 MB/sec execute 106 sec latency 101.557 ms 2023-07-29T16:34:03.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:02 smithi039 ceph-mon[128858]: pgmap v928: 97 pgs: 97 active+clean; 658 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 701 op/s 2023-07-29T16:34:03.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:02 smithi156 ceph-mon[129245]: pgmap v928: 97 pgs: 97 active+clean; 658 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 701 op/s 2023-07-29T16:34:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:02 smithi031 ceph-mon[95504]: pgmap v928: 97 pgs: 97 active+clean; 658 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 8.3 MiB/s wr, 701 op/s 2023-07-29T16:34:03.267 INFO:tasks.workunit.client.0.smithi031.stdout: 10 151157 36.68 MB/sec execute 107 sec latency 59.971 ms 2023-07-29T16:34:04.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:03 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:34:04.086 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:03 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:34:04.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:03 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:34:04.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 151760 36.68 MB/sec execute 108 sec latency 83.431 ms 2023-07-29T16:34:05.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:04 smithi039 ceph-mon[128858]: pgmap v929: 97 pgs: 97 active+clean; 669 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 8.0 MiB/s wr, 756 op/s 2023-07-29T16:34:05.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:04 smithi156 ceph-mon[129245]: pgmap v929: 97 pgs: 97 active+clean; 669 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 8.0 MiB/s wr, 756 op/s 2023-07-29T16:34:05.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:04 smithi031 ceph-mon[95504]: pgmap v929: 97 pgs: 97 active+clean; 669 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 8.0 MiB/s wr, 756 op/s 2023-07-29T16:34:05.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 152465 36.63 MB/sec execute 109 sec latency 57.272 ms 2023-07-29T16:34:06.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:34:06.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:34:06.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:05 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:34:06.103 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:34:06.103 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:34:06.104 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:05 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:34:06.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:34:06.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:34:06.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:05 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:34:06.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 153407 36.80 MB/sec execute 110 sec latency 42.239 ms 2023-07-29T16:34:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.500+0000 7f634b9ff700 1 -- 172.21.15.31:0/120257336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344107b80 msgr2=0x7f6344107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.500+0000 7f634b9ff700 1 --2- 172.21.15.31:0/120257336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344107b80 0x7f6344107fd0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f63380098f0 tx=0x7f6338009c00 comp rx=0 tx=0).stop 2023-07-29T16:34:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 -- 172.21.15.31:0/120257336 shutdown_connections 2023-07-29T16:34:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 --2- 172.21.15.31:0/120257336 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 0x7f634410edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 --2- 172.21.15.31:0/120257336 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6344106990 0x7f6344106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 --2- 172.21.15.31:0/120257336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344107b80 0x7f6344107fd0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 -- 172.21.15.31:0/120257336 >> 172.21.15.31:0/120257336 conn(0x7f63440759d0 msgr2=0x7f6344077e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 -- 172.21.15.31:0/120257336 shutdown_connections 2023-07-29T16:34:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.501+0000 7f634b9ff700 1 -- 172.21.15.31:0/120257336 wait complete. 2023-07-29T16:34:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.503+0000 7f634b9ff700 1 Processor -- start 2023-07-29T16:34:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.503+0000 7f634b9ff700 1 -- start start 2023-07-29T16:34:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.503+0000 7f634b9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 0x7f634419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.503+0000 7f634b9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6344107b80 0x7f634419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.503+0000 7f634b9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 0x7f634419db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634b9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634419e1b0 con 0x7f6344108510 2023-07-29T16:34:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634b9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634419e320 con 0x7f6344107b80 2023-07-29T16:34:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634b9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634419e4e0 con 0x7f6344106990 2023-07-29T16:34:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 0x7f634419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 0x7f634419c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48676/0 (socket says 172.21.15.31:48676) 2023-07-29T16:34:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 -- 172.21.15.31:0/3443244624 learned_addr learned my addr 172.21.15.31:0/3443244624 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f6348f9a700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6344107b80 0x7f634419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f6349f9c700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 0x7f634419db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 -- 172.21.15.31:0/3443244624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6344107b80 msgr2=0x7f634419ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6344107b80 0x7f634419ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 -- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 msgr2=0x7f634419db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 0x7f634419db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f634979b700 1 -- 172.21.15.31:0/3443244624 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6338009520 con 0x7f6344106990 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.504+0000 7f6349f9c700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 0x7f634419db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:34:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.505+0000 7f634979b700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 0x7f634419c790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f633c00efb0 tx=0x7f633c00c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.505+0000 7f63427fc700 1 -- 172.21.15.31:0/3443244624 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c018070 con 0x7f6344106990 2023-07-29T16:34:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.505+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634419e760 con 0x7f6344106990 2023-07-29T16:34:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.505+0000 7f63427fc700 1 -- 172.21.15.31:0/3443244624 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f633c00f040 con 0x7f6344106990 2023-07-29T16:34:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.505+0000 7f63427fc700 1 -- 172.21.15.31:0/3443244624 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c004410 con 0x7f6344106990 2023-07-29T16:34:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.505+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63441a4f80 con 0x7f6344106990 2023-07-29T16:34:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 Processor -- start 2023-07-29T16:34:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f63427fc700 1 -- 172.21.15.31:0/3443244624 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f633c003a90 con 0x7f6344106990 2023-07-29T16:34:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 -- start start 2023-07-29T16:34:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625978286b0 0x5625979cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979cdcf0 0x5625979ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979249b0 con 0x5625979cdcf0 2023-07-29T16:34:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562597922370 con 0x5625979ccdf0 2023-07-29T16:34:06.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.512+0000 7f635f406b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56259791dc60 con 0x5625978286b0 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.513+0000 7f634979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.513+0000 7f634979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47806/0 (socket says 172.21.15.31:47806) 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.513+0000 7f634979b700 1 -- 172.21.15.31:0/2734194530 learned_addr learned my addr 172.21.15.31:0/2734194530 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.513+0000 7f6348f9a700 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979cdcf0 0x5625979ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.513+0000 7f6349f9c700 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625978286b0 0x5625979cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.514+0000 7f634979b700 1 -- 172.21.15.31:0/2734194530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625978286b0 msgr2=0x5625979cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.514+0000 7f634979b700 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625978286b0 0x5625979cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.514+0000 7f634979b700 1 -- 172.21.15.31:0/2734194530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979cdcf0 msgr2=0x5625979ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:06.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.514+0000 7f634979b700 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979cdcf0 0x5625979ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.514+0000 7f634979b700 1 -- 172.21.15.31:0/2734194530 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56259774bd80 con 0x5625979ccdf0 2023-07-29T16:34:06.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.514+0000 7f634979b700 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979cd7b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f633c0c9410 tx=0x7f633c0d1d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0bf02b6b59cde54 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:06.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.515+0000 7f63427fc700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f632c0a0860 0x7f632c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.515+0000 7f63427fc700 1 -- 172.21.15.31:0/3443244624 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f633c0cba30 con 0x7f6344106990 2023-07-29T16:34:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.515+0000 7f632b7fe700 1 -- 172.21.15.31:0/2734194530 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c095500 con 0x5625979ccdf0 2023-07-29T16:34:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.515+0000 7f632b7fe700 1 -- 172.21.15.31:0/2734194530 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f633c0d0050 con 0x5625979ccdf0 2023-07-29T16:34:06.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.515+0000 7f632b7fe700 1 -- 172.21.15.31:0/2734194530 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c098d00 con 0x5625979ccdf0 2023-07-29T16:34:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.516+0000 7f6349f9c700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f632c0a0860 0x7f632c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.516+0000 7f635f406b80 1 -- 172.21.15.31:0/2734194530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 msgr2=0x5625979cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.516+0000 7f635f406b80 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979cd7b0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f633c0c9410 tx=0x7f633c0d1d60 comp rx=0 tx=0).stop 2023-07-29T16:34:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 -- 172.21.15.31:0/2734194530 shutdown_connections 2023-07-29T16:34:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979cdcf0 0x5625979ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625978286b0 0x5625979cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 --2- 172.21.15.31:0/2734194530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979cd7b0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 -- 172.21.15.31:0/2734194530 >> 172.21.15.31:0/2734194530 conn(0x5625979c1fd0 msgr2=0x5625979c7300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f6349f9c700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f632c0a0860 0x7f632c0a2d10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6334009d10 tx=0x7f6334009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 -- 172.21.15.31:0/2734194530 shutdown_connections 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.517+0000 7f635f406b80 1 -- 172.21.15.31:0/2734194530 wait complete. 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.518+0000 7f635f406b80 1 -- start start 2023-07-29T16:34:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.518+0000 7f635f406b80 1 Processor -- start 2023-07-29T16:34:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625978ba4b0 con 0x5625978286b0 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625978ba150 con 0x5625979ccdf0 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625977a18c0 con 0x5625979cdcf0 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f635f406b80 10 client.0 _ll_register_callbacks cb 0x5625978c35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f6348f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f6348f9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979d62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47820/0 (socket says 172.21.15.31:47820) 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f6348f9a700 1 -- 172.21.15.31:0/3644843953 learned_addr learned my addr 172.21.15.31:0/3644843953 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:06.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.519+0000 7f6349f9c700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f634979b700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6348f9a700 1 -- 172.21.15.31:0/3644843953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 msgr2=0x5625979dd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6348f9a700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979dd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6348f9a700 1 -- 172.21.15.31:0/3644843953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 msgr2=0x5625979d5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6348f9a700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6348f9a700 1 -- 172.21.15.31:0/3644843953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63441a4f80 con 0x5625979ccdf0 2023-07-29T16:34:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6348f9a700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979d62c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f63380098f0 tx=0x7f633800b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd0c44b3acb76b43 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.520+0000 7f6328ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.521+0000 7f6328ff9700 1 -- 172.21.15.31:0/3644843953 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6338005240 con 0x5625979ccdf0 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.521+0000 7f6328ff9700 1 -- 172.21.15.31:0/3644843953 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f633800bbc0 con 0x5625979ccdf0 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.521+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562597948cb0 con 0x5625979ccdf0 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.521+0000 7f6328ff9700 1 -- 172.21.15.31:0/3644843953 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63380053a0 con 0x5625979ccdf0 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.521+0000 7f635f406b80 10 client.30296 fetch_fsmap learned FSMap version 41 2023-07-29T16:34:06.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.521+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x562597734c00 con 0x5625979ccdf0 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f6328ff9700 1 -- 172.21.15.31:0/3644843953 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562597948cb0 con 0x5625979ccdf0 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f6328ff9700 1 -- 172.21.15.31:0/3644843953 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6338015020 con 0x5625979ccdf0 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f635f406b80 10 client.30296 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f635f406b80 10 client.30296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f635f406b80 20 client.30296 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f635f406b80 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625979e1bf0 0x5625979e4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f635f406b80 4 client.30296 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:34:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56259799bd10 con 0x5625979e1bf0 2023-07-29T16:34:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.522+0000 7f634979b700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625979e1bf0 0x5625979e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.523+0000 7f6328ff9700 10 client.30296 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.523+0000 7f634979b700 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625979e1bf0 0x5625979e4000 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4860a44c6030d4f server_cookie=71a8d5193fdb62b4 in_seq=0 out_seq=0 2023-07-29T16:34:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.526+0000 7f6328ff9700 1 -- 172.21.15.31:0/3644843953 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56259799bd10 con 0x5625979e1bf0 2023-07-29T16:34:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.526+0000 7f6328ff9700 10 client.30296 handle_command_reply: tid=0 2023-07-29T16:34:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.526+0000 7f635f406b80 1 client.30296 shutdown 2023-07-29T16:34:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.526+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 msgr2=0x5625979d62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.526+0000 7f635f406b80 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979d62c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f63380098f0 tx=0x7f633800b790 comp rx=0 tx=0).stop 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 shutdown_connections 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5625979e1bf0 0x5625979e4000 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979dd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 --2- 172.21.15.31:0/3644843953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979ccdf0 0x5625979d62c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 >> 172.21.15.31:0/3644843953 conn(0x5625979c1fd0 msgr2=0x5625979c6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f6328ff9700 0 client.30296 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 shutdown_connections 2023-07-29T16:34:06.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.527+0000 7f635f406b80 1 -- 172.21.15.31:0/3644843953 wait complete. 2023-07-29T16:34:06.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.528+0000 7f635f406b80 20 client.30296 trim_cache size 0 max 16384 2023-07-29T16:34:06.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.551+0000 7f635f406b80 1 Processor -- start 2023-07-29T16:34:06.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 -- start start 2023-07-29T16:34:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 0x5625979db210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979ccdf0 0x5625979d3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979cd2c0 con 0x5625979ccdf0 2023-07-29T16:34:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979c4050 con 0x5625978286b0 2023-07-29T16:34:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.552+0000 7f635f406b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979d58f0 con 0x5625979cdcf0 2023-07-29T16:34:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.553+0000 7f634979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 0x5625979db210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.553+0000 7f634979b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 0x5625979db210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47822/0 (socket says 172.21.15.31:47822) 2023-07-29T16:34:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.553+0000 7f634979b700 1 -- 172.21.15.31:0/3236668905 learned_addr learned my addr 172.21.15.31:0/3236668905 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f634979b700 1 -- 172.21.15.31:0/3236668905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 msgr2=0x5625979d4510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:34:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f634979b700 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979d4510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f634979b700 1 -- 172.21.15.31:0/3236668905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979ccdf0 msgr2=0x5625979d3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f634979b700 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979ccdf0 0x5625979d3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f634979b700 1 -- 172.21.15.31:0/3236668905 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625977721f0 con 0x5625978286b0 2023-07-29T16:34:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f634979b700 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 0x5625979db210 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f633c095c30 tx=0x7f633c0cefd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8734da39a0f4d75a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.554+0000 7f632bfff700 1 -- 172.21.15.31:0/3236668905 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c09f0b0 con 0x5625978286b0 2023-07-29T16:34:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.555+0000 7f632bfff700 1 -- 172.21.15.31:0/3236668905 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f633c095500 con 0x5625978286b0 2023-07-29T16:34:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.555+0000 7f632bfff700 1 -- 172.21.15.31:0/3236668905 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f633c09d1f0 con 0x5625978286b0 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 -- 172.21.15.31:0/3236668905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 msgr2=0x5625979db210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 0x5625979db210 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f633c095c30 tx=0x7f633c0cefd0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 -- 172.21.15.31:0/3236668905 shutdown_connections 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979cdcf0 0x5625979d4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625978286b0 0x5625979db210 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 --2- 172.21.15.31:0/3236668905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625979ccdf0 0x5625979d3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 -- 172.21.15.31:0/3236668905 >> 172.21.15.31:0/3236668905 conn(0x5625979c1fd0 msgr2=0x5625979e3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.556+0000 7f635f406b80 1 -- 172.21.15.31:0/3236668905 shutdown_connections 2023-07-29T16:34:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.557+0000 7f635f406b80 1 -- 172.21.15.31:0/3236668905 wait complete. 2023-07-29T16:34:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.557+0000 7f635f406b80 1 -- start start 2023-07-29T16:34:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.557+0000 7f635f406b80 1 Processor -- start 2023-07-29T16:34:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979ccdf0 0x5625979d95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 0x5625979da600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979dab40 con 0x5625978286b0 2023-07-29T16:34:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979dacb0 con 0x5625979cdcf0 2023-07-29T16:34:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625979dae70 con 0x5625979ccdf0 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f635f406b80 10 client.0 _ll_register_callbacks cb 0x5625978c35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.558+0000 7f6349f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 0x5625979da600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 0x5625979da600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47834/0 (socket says 172.21.15.31:47834) 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 -- 172.21.15.31:0/2025532833 learned_addr learned my addr 172.21.15.31:0/2025532833 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 -- 172.21.15.31:0/2025532833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979ccdf0 msgr2=0x5625979d95f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979ccdf0 0x5625979d95f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 -- 172.21.15.31:0/2025532833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 msgr2=0x5625979d90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 -- 172.21.15.31:0/2025532833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56259774bd80 con 0x5625979cdcf0 2023-07-29T16:34:06.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6349f9c700 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 0x5625979da600 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6334011c30 tx=0x7f6334015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237200e7956b6113 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:06.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.559+0000 7f6328ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:34:06.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.560+0000 7f6328ff9700 1 -- 172.21.15.31:0/2025532833 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6334023040 con 0x5625979cdcf0 2023-07-29T16:34:06.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.560+0000 7f6328ff9700 1 -- 172.21.15.31:0/2025532833 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f633401dd80 con 0x5625979cdcf0 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.560+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f633c0d0050 con 0x5625979cdcf0 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f6328ff9700 1 -- 172.21.15.31:0/2025532833 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6334004830 con 0x5625979cdcf0 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 10 client.30302 fetch_fsmap learned FSMap version 41 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x56259774a250 con 0x5625979cdcf0 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f6328ff9700 1 -- 172.21.15.31:0/2025532833 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f633c0d0050 con 0x5625979cdcf0 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f6328ff9700 1 -- 172.21.15.31:0/2025532833 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 41) v1 ==== 7458+0+0 (secure 0 0 0) 0x7f6334027520 con 0x5625979cdcf0 2023-07-29T16:34:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 10 client.30302 fetch_fsmap finished waiting for FSMap version 41 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 10 client.30302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 20 client.30302 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562597a348b0 0x562597a36cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 4 client.30302 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.561+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562597916ce0 con 0x562597a348b0 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.562+0000 7f634979b700 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562597a348b0 0x562597a36cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.562+0000 7f634979b700 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562597a348b0 0x562597a36cc0 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6632a184734b2ee0 server_cookie=1efd1302f5e2694e in_seq=0 out_seq=0 2023-07-29T16:34:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.562+0000 7f6328ff9700 10 client.30302 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:06.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.944+0000 7f6328ff9700 1 -- 172.21.15.31:0/2025532833 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562597916ce0 con 0x562597a348b0 2023-07-29T16:34:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.944+0000 7f6328ff9700 10 client.30302 handle_command_reply: tid=0 2023-07-29T16:34:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 client.30302 shutdown 2023-07-29T16:34:06.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 msgr2=0x5625979da600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 0x5625979da600 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6334011c30 tx=0x7f6334015040 comp rx=0 tx=0).stop 2023-07-29T16:34:06.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 shutdown_connections 2023-07-29T16:34:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562597a348b0 0x562597a36cc0 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5625979cdcf0 0x5625979da600 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625978286b0 0x5625979d90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 --2- 172.21.15.31:0/2025532833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5625979ccdf0 0x5625979d95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.945+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 >> 172.21.15.31:0/2025532833 conn(0x5625979c1fd0 msgr2=0x5625979d3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.946+0000 7f6328ff9700 0 client.30302 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.946+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 shutdown_connections 2023-07-29T16:34:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.946+0000 7f635f406b80 1 -- 172.21.15.31:0/2025532833 wait complete. 2023-07-29T16:34:06.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.946+0000 7f635f406b80 20 client.30302 trim_cache size 0 max 16384 2023-07-29T16:34:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:34:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:34:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:34:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f632c0a0860 msgr2=0x7f632c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f632c0a0860 0x7f632c0a2d10 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6334009d10 tx=0x7f6334009260 comp rx=0 tx=0).stop 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 msgr2=0x7f634419c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 0x7f634419c790 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f633c00efb0 tx=0x7f633c00c450 comp rx=0 tx=0).stop 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 shutdown_connections 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f632c0a0860 0x7f632c0a2d10 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6344108510 0x7f634419db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6344106990 0x7f634419c790 secure :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f633c00efb0 tx=0x7f633c00c450 comp rx=0 tx=0).stop 2023-07-29T16:34:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 --2- 172.21.15.31:0/3443244624 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6344107b80 0x7f634419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 >> 172.21.15.31:0/3443244624 conn(0x7f63440759d0 msgr2=0x7f6344077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.947+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 shutdown_connections 2023-07-29T16:34:06.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:06.948+0000 7f634b9ff700 1 -- 172.21.15.31:0/3443244624 wait complete. 2023-07-29T16:34:07.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:06 smithi039 ceph-mon[128858]: pgmap v930: 97 pgs: 97 active+clean; 671 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.0 MiB/s wr, 652 op/s 2023-07-29T16:34:07.139 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:06 smithi156 ceph-mon[129245]: pgmap v930: 97 pgs: 97 active+clean; 671 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.0 MiB/s wr, 652 op/s 2023-07-29T16:34:07.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:06 smithi031 ceph-mon[95504]: pgmap v930: 97 pgs: 97 active+clean; 671 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.0 MiB/s wr, 652 op/s 2023-07-29T16:34:07.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 154207 36.91 MB/sec execute 111 sec latency 202.738 ms 2023-07-29T16:34:08.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 154792 36.88 MB/sec execute 112 sec latency 140.731 ms 2023-07-29T16:34:08.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:08 smithi031 ceph-mon[95504]: pgmap v931: 97 pgs: 97 active+clean; 666 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.4 MiB/s wr, 552 op/s 2023-07-29T16:34:08.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:08 smithi039 ceph-mon[128858]: pgmap v931: 97 pgs: 97 active+clean; 666 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.4 MiB/s wr, 552 op/s 2023-07-29T16:34:08.669 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:08 smithi156 ceph-mon[129245]: pgmap v931: 97 pgs: 97 active+clean; 666 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.4 MiB/s wr, 552 op/s 2023-07-29T16:34:09.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 155327 36.76 MB/sec execute 113 sec latency 141.644 ms 2023-07-29T16:34:10.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 156149 36.75 MB/sec execute 114 sec latency 40.842 ms 2023-07-29T16:34:10.935 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:10 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:34:10.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:10 smithi031 ceph-mon[95504]: pgmap v932: 97 pgs: 97 active+clean; 673 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 665 op/s 2023-07-29T16:34:11.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:10 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:34:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:10 smithi039 ceph-mon[128858]: pgmap v932: 97 pgs: 97 active+clean; 673 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 665 op/s 2023-07-29T16:34:11.194 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:10 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:34:11.194 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:10 smithi156 ceph-mon[129245]: pgmap v932: 97 pgs: 97 active+clean; 673 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.4 MiB/s wr, 665 op/s 2023-07-29T16:34:11.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 157040 36.87 MB/sec execute 115 sec latency 41.347 ms 2023-07-29T16:34:12.268 INFO:tasks.workunit.client.0.smithi031.stdout: 10 158013 37.05 MB/sec execute 116 sec latency 38.403 ms 2023-07-29T16:34:13.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:12 smithi039 ceph-mon[128858]: pgmap v933: 97 pgs: 97 active+clean; 676 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 592 op/s 2023-07-29T16:34:13.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:12 smithi156 ceph-mon[129245]: pgmap v933: 97 pgs: 97 active+clean; 676 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 592 op/s 2023-07-29T16:34:13.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:12 smithi031 ceph-mon[95504]: pgmap v933: 97 pgs: 97 active+clean; 676 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.9 MiB/s wr, 592 op/s 2023-07-29T16:34:13.269 INFO:tasks.workunit.client.0.smithi031.stdout: 10 158815 37.14 MB/sec execute 117 sec latency 93.466 ms 2023-07-29T16:34:14.269 INFO:tasks.workunit.client.0.smithi031.stdout: 10 159372 37.10 MB/sec execute 118 sec latency 58.724 ms 2023-07-29T16:34:15.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:14 smithi039 ceph-mon[128858]: pgmap v934: 97 pgs: 97 active+clean; 686 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 682 op/s 2023-07-29T16:34:15.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:14 smithi156 ceph-mon[129245]: pgmap v934: 97 pgs: 97 active+clean; 686 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 682 op/s 2023-07-29T16:34:15.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:14 smithi031 ceph-mon[95504]: pgmap v934: 97 pgs: 97 active+clean; 686 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 682 op/s 2023-07-29T16:34:15.270 INFO:tasks.workunit.client.0.smithi031.stdout: 10 159874 36.94 MB/sec execute 119 sec latency 82.958 ms 2023-07-29T16:34:16.270 INFO:tasks.workunit.client.0.smithi031.stdout: 10 160549 36.87 MB/sec execute 120 sec latency 129.244 ms 2023-07-29T16:34:17.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:16 smithi039 ceph-mon[128858]: pgmap v935: 97 pgs: 97 active+clean; 690 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 630 op/s 2023-07-29T16:34:17.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:16 smithi156 ceph-mon[129245]: pgmap v935: 97 pgs: 97 active+clean; 690 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 630 op/s 2023-07-29T16:34:17.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:16 smithi031 ceph-mon[95504]: pgmap v935: 97 pgs: 97 active+clean; 690 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 630 op/s 2023-07-29T16:34:17.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 161560 37.03 MB/sec execute 121 sec latency 48.925 ms 2023-07-29T16:34:18.270 INFO:tasks.workunit.client.0.smithi031.stdout: 10 162458 37.21 MB/sec execute 122 sec latency 35.887 ms 2023-07-29T16:34:18.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:18 smithi031 ceph-mon[95504]: pgmap v936: 97 pgs: 97 active+clean; 695 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.0 MiB/s wr, 612 op/s 2023-07-29T16:34:18.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:18 smithi039 ceph-mon[128858]: pgmap v936: 97 pgs: 97 active+clean; 695 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.0 MiB/s wr, 612 op/s 2023-07-29T16:34:18.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:18 smithi156 ceph-mon[129245]: pgmap v936: 97 pgs: 97 active+clean; 695 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.0 MiB/s wr, 612 op/s 2023-07-29T16:34:19.270 INFO:tasks.workunit.client.0.smithi031.stdout: 10 162980 37.13 MB/sec execute 123 sec latency 111.807 ms 2023-07-29T16:34:20.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 -- 172.21.15.31:0/2325601675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0101570 msgr2=0x7f5af01019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 --2- 172.21.15.31:0/2325601675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0101570 0x7f5af01019c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00098f0 tx=0x7f5ae0009c00 comp rx=0 tx=0).stop 2023-07-29T16:34:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 -- 172.21.15.31:0/2325601675 shutdown_connections 2023-07-29T16:34:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 --2- 172.21.15.31:0/2325601675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af0101f00 0x7f5af01087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 --2- 172.21.15.31:0/2325601675 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0100380 0x7f5af0100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 --2- 172.21.15.31:0/2325601675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0101570 0x7f5af01019c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.152+0000 7f5af7dff700 1 -- 172.21.15.31:0/2325601675 >> 172.21.15.31:0/2325601675 conn(0x7f5af00fb9a0 msgr2=0x7f5af00fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.153+0000 7f5af7dff700 1 -- 172.21.15.31:0/2325601675 shutdown_connections 2023-07-29T16:34:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.153+0000 7f5af7dff700 1 -- 172.21.15.31:0/2325601675 wait complete. 2023-07-29T16:34:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 Processor -- start 2023-07-29T16:34:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 -- start start 2023-07-29T16:34:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 0x7f5af019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af0101570 0x7f5af019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0101f00 0x7f5af019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af019e220 con 0x7f5af0101f00 2023-07-29T16:34:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af019e390 con 0x7f5af0100380 2023-07-29T16:34:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.154+0000 7f5af7dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af019e550 con 0x7f5af0101570 2023-07-29T16:34:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 0x7f5af019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 0x7f5af019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54280/0 (socket says 172.21.15.31:54280) 2023-07-29T16:34:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 -- 172.21.15.31:0/3583866715 learned_addr learned my addr 172.21.15.31:0/3583866715 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af639c700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0101f00 0x7f5af019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af539a700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af0101570 0x7f5af019cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 -- 172.21.15.31:0/3583866715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af0101570 msgr2=0x7f5af019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af0101570 0x7f5af019cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 -- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0101f00 msgr2=0x7f5af019dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0101f00 0x7f5af019dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af5b9b700 1 -- 172.21.15.31:0/3583866715 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8009d00 con 0x7f5af0100380 2023-07-29T16:34:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.155+0000 7f5af639c700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0101f00 0x7f5af019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:34:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.156+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 0x7f5af019c800 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e970 tx=0x7f5aec00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.156+0000 7f5ae6ffd700 1 -- 172.21.15.31:0/3583866715 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec009790 con 0x7f5af0100380 2023-07-29T16:34:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.156+0000 7f5ae6ffd700 1 -- 172.21.15.31:0/3583866715 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5aec00c7e0 con 0x7f5af0100380 2023-07-29T16:34:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.156+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae0009520 con 0x7f5af0100380 2023-07-29T16:34:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.156+0000 7f5ae6ffd700 1 -- 172.21.15.31:0/3583866715 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec003c70 con 0x7f5af0100380 2023-07-29T16:34:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.156+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af019ea20 con 0x7f5af0100380 2023-07-29T16:34:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.160+0000 7f5ae6ffd700 1 -- 172.21.15.31:0/3583866715 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5aec00cc50 con 0x7f5af0100380 2023-07-29T16:34:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.162+0000 7f5ae6ffd700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5adc0a0810 0x7f5adc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.162+0000 7f5ae6ffd700 1 -- 172.21.15.31:0/3583866715 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5aec014070 con 0x7f5af0100380 2023-07-29T16:34:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.163+0000 7f5af539a700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5adc0a0810 0x7f5adc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.164+0000 7f5b0b6b6b80 1 Processor -- start 2023-07-29T16:34:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.164+0000 7f5af539a700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5adc0a0810 0x7f5adc0a2cc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005e90 tx=0x7f5ae0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.164+0000 7f5b0b6b6b80 1 -- start start 2023-07-29T16:34:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.164+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 0x55617b7af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.164+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7afdf0 0x55617b7b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7079b0 con 0x55617b7afdf0 2023-07-29T16:34:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b705370 con 0x55617b60b6b0 2023-07-29T16:34:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b700c60 con 0x55617b7b0cf0 2023-07-29T16:34:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5af639c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5af5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 0x55617b7af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5af5b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 0x55617b7af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54292/0 (socket says 172.21.15.31:54292) 2023-07-29T16:34:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5af5b9b700 1 -- 172.21.15.31:0/1228068719 learned_addr learned my addr 172.21.15.31:0/1228068719 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.165+0000 7f5af539a700 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7afdf0 0x55617b7b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.166+0000 7f5af5b9b700 1 -- 172.21.15.31:0/1228068719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 msgr2=0x55617b7b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.166+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.166+0000 7f5af5b9b700 1 -- 172.21.15.31:0/1228068719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7afdf0 msgr2=0x55617b7b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.166+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7afdf0 0x55617b7b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.166+0000 7f5af5b9b700 1 -- 172.21.15.31:0/1228068719 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55617b52ed80 con 0x55617b60b6b0 2023-07-29T16:34:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.166+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 0x55617b7af8b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0a1320 tx=0x7f5aec003710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83fb98bf65e63722 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.167+0000 7f5adbfff700 1 -- 172.21.15.31:0/1228068719 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec09d430 con 0x55617b60b6b0 2023-07-29T16:34:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.167+0000 7f5adbfff700 1 -- 172.21.15.31:0/1228068719 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5aec0a1850 con 0x55617b60b6b0 2023-07-29T16:34:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.167+0000 7f5adbfff700 1 -- 172.21.15.31:0/1228068719 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0998c0 con 0x55617b60b6b0 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/1228068719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 msgr2=0x55617b7af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 0x55617b7af8b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5aec0a1320 tx=0x7f5aec003710 comp rx=0 tx=0).stop 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/1228068719 shutdown_connections 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b60b6b0 0x55617b7af8b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/1228068719 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7afdf0 0x55617b7b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.168+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/1228068719 >> 172.21.15.31:0/1228068719 conn(0x55617b7a4fd0 msgr2=0x55617b7aa300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.169+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/1228068719 shutdown_connections 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.169+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/1228068719 wait complete. 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.169+0000 7f5b0b6b6b80 1 -- start start 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 Processor -- start 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7d1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b69d4b0 con 0x55617b60b6b0 2023-07-29T16:34:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b69d150 con 0x55617b7afdf0 2023-07-29T16:34:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b5848c0 con 0x55617b7b0cf0 2023-07-29T16:34:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5b0b6b6b80 10 client.0 _ll_register_callbacks cb 0x55617b6a65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5af539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5af539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54296/0 (socket says 172.21.15.31:54296) 2023-07-29T16:34:20.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.170+0000 7f5af539a700 1 -- 172.21.15.31:0/3931657647 learned_addr learned my addr 172.21.15.31:0/3931657647 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:20.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af639c700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7d1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af539a700 1 -- 172.21.15.31:0/3931657647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 msgr2=0x55617b7d1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af539a700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7d1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af539a700 1 -- 172.21.15.31:0/3931657647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 msgr2=0x55617b7c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af539a700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7c9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af539a700 1 -- 172.21.15.31:0/3931657647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af019ea20 con 0x55617b7afdf0 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5af539a700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7ca4f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009810 tx=0x7f5ae0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48756c8c087e5564 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5ad97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5ad97fa700 1 -- 172.21.15.31:0/3931657647 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae002b070 con 0x55617b7afdf0 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.171+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55617b72bcb0 con 0x55617b7afdf0 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.172+0000 7f5ad97fa700 1 -- 172.21.15.31:0/3931657647 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5ae0005040 con 0x55617b7afdf0 2023-07-29T16:34:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.172+0000 7f5ad97fa700 1 -- 172.21.15.31:0/3931657647 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ae0005200 con 0x55617b7afdf0 2023-07-29T16:34:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.172+0000 7f5b0b6b6b80 10 client.30314 fetch_fsmap learned FSMap version 42 2023-07-29T16:34:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.172+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55617b517c00 con 0x55617b7afdf0 2023-07-29T16:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.172+0000 7f5ad97fa700 1 -- 172.21.15.31:0/3931657647 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55617b72bcb0 con 0x55617b7afdf0 2023-07-29T16:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5ad97fa700 1 -- 172.21.15.31:0/3931657647 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5ae0025430 con 0x55617b7afdf0 2023-07-29T16:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5b0b6b6b80 10 client.30314 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:34:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5b0b6b6b80 10 client.30314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5b0b6b6b80 20 client.30314 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b7d5d50 0x55617b7d8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5b0b6b6b80 4 client.30314 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55617b77ed10 con 0x55617b7d5d50 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.173+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b7d5d50 0x55617b7d8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.174+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b7d5d50 0x55617b7d8160 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1aac157bc4fb577 server_cookie=8c363ddfc9599c78 in_seq=0 out_seq=0 2023-07-29T16:34:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.174+0000 7f5ad97fa700 10 client.30314 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5ad97fa700 1 -- 172.21.15.31:0/3931657647 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55617b77ed10 con 0x55617b7d5d50 2023-07-29T16:34:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5ad97fa700 10 client.30314 handle_command_reply: tid=0 2023-07-29T16:34:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 client.30314 shutdown 2023-07-29T16:34:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 msgr2=0x55617b7ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7ca4f0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009810 tx=0x7f5ae0009260 comp rx=0 tx=0).stop 2023-07-29T16:34:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 shutdown_connections 2023-07-29T16:34:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b7d5d50 0x55617b7d8160 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7b0cf0 0x55617b7d1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3931657647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7ca4f0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.179+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 >> 172.21.15.31:0/3931657647 conn(0x55617b7a4fd0 msgr2=0x55617b7a9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.180+0000 7f5ad97fa700 0 client.30314 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.181+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 shutdown_connections 2023-07-29T16:34:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.181+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3931657647 wait complete. 2023-07-29T16:34:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.181+0000 7f5b0b6b6b80 20 client.30314 trim_cache size 0 max 16384 2023-07-29T16:34:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.201+0000 7f5b0b6b6b80 1 Processor -- start 2023-07-29T16:34:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.201+0000 7f5b0b6b6b80 1 -- start start 2023-07-29T16:34:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b60b6b0 0x55617b7cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7b0cf0 0x55617b7b7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7ab600 con 0x55617b7b0cf0 2023-07-29T16:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7a4a20 con 0x55617b7afdf0 2023-07-29T16:34:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7c8a10 con 0x55617b60b6b0 2023-07-29T16:34:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5af539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5af539a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54312/0 (socket says 172.21.15.31:54312) 2023-07-29T16:34:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5af539a700 1 -- 172.21.15.31:0/3887681280 learned_addr learned my addr 172.21.15.31:0/3887681280 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b60b6b0 0x55617b7cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5af639c700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7b0cf0 0x55617b7b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.202+0000 7f5af539a700 1 -- 172.21.15.31:0/3887681280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b60b6b0 msgr2=0x55617b7cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af539a700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b60b6b0 0x55617b7cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af539a700 1 -- 172.21.15.31:0/3887681280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7b0cf0 msgr2=0x55617b7b7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af539a700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7b0cf0 0x55617b7b7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af539a700 1 -- 172.21.15.31:0/3887681280 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55617b5551f0 con 0x55617b7afdf0 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af639c700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7b0cf0 0x55617b7b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b60b6b0 0x55617b7cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5af539a700 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7a9a40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00258b0 tx=0x7f5ae000bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aad9330287e5c2a9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5ae4bff700 1 -- 172.21.15.31:0/3887681280 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae002b070 con 0x55617b7afdf0 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5ae4bff700 1 -- 172.21.15.31:0/3887681280 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ae002c040 con 0x55617b7afdf0 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.203+0000 7f5ae4bff700 1 -- 172.21.15.31:0/3887681280 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0005680 con 0x55617b7afdf0 2023-07-29T16:34:20.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3887681280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 msgr2=0x55617b7a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7a9a40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00258b0 tx=0x7f5ae000bbf0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3887681280 shutdown_connections 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b7b0cf0 0x55617b7b7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b60b6b0 0x55617b7cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/3887681280 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7afdf0 0x55617b7a9a40 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.204+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3887681280 >> 172.21.15.31:0/3887681280 conn(0x55617b7a4fd0 msgr2=0x55617b7c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.205+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3887681280 shutdown_connections 2023-07-29T16:34:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.205+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/3887681280 wait complete. 2023-07-29T16:34:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.205+0000 7f5b0b6b6b80 1 -- start start 2023-07-29T16:34:20.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 Processor -- start 2023-07-29T16:34:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7cd140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7afdf0 0x55617b7cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 0x55617b7ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7cebd0 con 0x55617b60b6b0 2023-07-29T16:34:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7ced40 con 0x55617b7b0cf0 2023-07-29T16:34:20.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55617b7cef00 con 0x55617b7afdf0 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.206+0000 7f5b0b6b6b80 10 client.0 _ll_register_callbacks cb 0x55617b6a65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 0x55617b7ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 0x55617b7ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54316/0 (socket says 172.21.15.31:54316) 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 -- 172.21.15.31:0/4078464910 learned_addr learned my addr 172.21.15.31:0/4078464910 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af539a700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7afdf0 0x55617b7cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7cd140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 -- 172.21.15.31:0/4078464910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7afdf0 msgr2=0x55617b7cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7afdf0 0x55617b7cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 -- 172.21.15.31:0/4078464910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 msgr2=0x55617b7cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7cd140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 -- 172.21.15.31:0/4078464910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55617b52ed80 con 0x55617b7b0cf0 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5af639c700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 0x55617b7ce690 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800edb0 tx=0x7f5ae80078a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3947e4d4591a4d5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.207+0000 7f5ad97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:34:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5ad97fa700 1 -- 172.21.15.31:0/4078464910 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae800e040 con 0x55617b7b0cf0 2023-07-29T16:34:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5ad97fa700 1 -- 172.21.15.31:0/4078464910 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5ae8017820 con 0x55617b7b0cf0 2023-07-29T16:34:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5aec0a1850 con 0x55617b7b0cf0 2023-07-29T16:34:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5ad97fa700 1 -- 172.21.15.31:0/4078464910 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ae80179e0 con 0x55617b7b0cf0 2023-07-29T16:34:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5b0b6b6b80 10 client.30320 fetch_fsmap learned FSMap version 42 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55617b517c00 con 0x55617b7b0cf0 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.208+0000 7f5ad97fa700 1 -- 172.21.15.31:0/4078464910 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5aec0a1850 con 0x55617b7b0cf0 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5ad97fa700 1 -- 172.21.15.31:0/4078464910 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5ae8021ca0 con 0x55617b7b0cf0 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5b0b6b6b80 10 client.30320 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5b0b6b6b80 10 client.30320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5b0b6b6b80 20 client.30320 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b828880 0x55617b82ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5b0b6b6b80 4 client.30320 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:34:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55617b6f9ce0 con 0x55617b828880 2023-07-29T16:34:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.209+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b828880 0x55617b82ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.210+0000 7f5ad97fa700 10 client.30320 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.210+0000 7f5af5b9b700 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b828880 0x55617b82ac90 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad2bfb65105e1904 server_cookie=12c5dab8ac1df7bd in_seq=0 out_seq=0 2023-07-29T16:34:20.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 163687 37.15 MB/sec execute 124 sec latency 63.003 ms 2023-07-29T16:34:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.681+0000 7f5ad97fa700 1 -- 172.21.15.31:0/4078464910 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55617b6f9ce0 con 0x55617b828880 2023-07-29T16:34:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.681+0000 7f5ad97fa700 10 client.30320 handle_command_reply: tid=0 2023-07-29T16:34:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.681+0000 7f5b0b6b6b80 1 client.30320 shutdown 2023-07-29T16:34:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 msgr2=0x55617b7ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 0x55617b7ce690 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f5ae800edb0 tx=0x7f5ae80078a0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 shutdown_connections 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55617b828880 0x55617b82ac90 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55617b7b0cf0 0x55617b7ce690 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55617b60b6b0 0x55617b7cd140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5ad97fa700 0 client.30320 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 --2- 172.21.15.31:0/4078464910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55617b7afdf0 0x55617b7cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.682+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 >> 172.21.15.31:0/4078464910 conn(0x55617b7a4fd0 msgr2=0x55617b7a8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.683+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 shutdown_connections 2023-07-29T16:34:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.683+0000 7f5b0b6b6b80 1 -- 172.21.15.31:0/4078464910 wait complete. 2023-07-29T16:34:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.684+0000 7f5b0b6b6b80 20 client.30320 trim_cache size 0 max 16384 2023-07-29T16:34:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:34:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:34:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:34:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:34:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5adc0a0810 msgr2=0x7f5adc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5adc0a0810 0x7f5adc0a2cc0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005e90 tx=0x7f5ae0019040 comp rx=0 tx=0).stop 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 msgr2=0x7f5af019c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 0x7f5af019c800 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e970 tx=0x7f5aec00ec80 comp rx=0 tx=0).stop 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 shutdown_connections 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5adc0a0810 0x7f5adc0a2cc0 secure :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005e90 tx=0x7f5ae0019040 comp rx=0 tx=0).stop 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5af0101f00 0x7f5af019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5af0100380 0x7f5af019c800 secure :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00e970 tx=0x7f5aec00ec80 comp rx=0 tx=0).stop 2023-07-29T16:34:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 --2- 172.21.15.31:0/3583866715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5af0101570 0x7f5af019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.685+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 >> 172.21.15.31:0/3583866715 conn(0x7f5af00fb9a0 msgr2=0x7f5af0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.688+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 shutdown_connections 2023-07-29T16:34:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:20.688+0000 7f5af7dff700 1 -- 172.21.15.31:0/3583866715 wait complete. 2023-07-29T16:34:21.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:20 smithi039 ceph-mon[128858]: pgmap v937: 97 pgs: 97 active+clean; 701 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 618 op/s 2023-07-29T16:34:21.073 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:20 smithi156 ceph-mon[129245]: pgmap v937: 97 pgs: 97 active+clean; 701 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 618 op/s 2023-07-29T16:34:21.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:20 smithi031 ceph-mon[95504]: pgmap v937: 97 pgs: 97 active+clean; 701 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 4.6 MiB/s wr, 618 op/s 2023-07-29T16:34:21.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 164206 37.04 MB/sec execute 125 sec latency 258.395 ms 2023-07-29T16:34:22.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 164892 37.07 MB/sec execute 126 sec latency 53.140 ms 2023-07-29T16:34:23.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:22 smithi039 ceph-mon[128858]: pgmap v938: 97 pgs: 97 active+clean; 705 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 471 op/s 2023-07-29T16:34:23.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:22 smithi156 ceph-mon[129245]: pgmap v938: 97 pgs: 97 active+clean; 705 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 471 op/s 2023-07-29T16:34:23.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:22 smithi031 ceph-mon[95504]: pgmap v938: 97 pgs: 97 active+clean; 705 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.2 MiB/s wr, 471 op/s 2023-07-29T16:34:23.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 165402 36.93 MB/sec execute 127 sec latency 357.070 ms 2023-07-29T16:34:24.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 165580 36.72 MB/sec execute 128 sec latency 689.199 ms 2023-07-29T16:34:25.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:24 smithi039 ceph-mon[128858]: pgmap v939: 97 pgs: 97 active+clean; 700 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 5.4 MiB/s wr, 639 op/s 2023-07-29T16:34:25.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:24 smithi156 ceph-mon[129245]: pgmap v939: 97 pgs: 97 active+clean; 700 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 5.4 MiB/s wr, 639 op/s 2023-07-29T16:34:25.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:24 smithi031 ceph-mon[95504]: pgmap v939: 97 pgs: 97 active+clean; 700 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 5.4 MiB/s wr, 639 op/s 2023-07-29T16:34:25.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 165874 36.57 MB/sec execute 129 sec latency 781.460 ms 2023-07-29T16:34:26.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 166449 36.53 MB/sec execute 130 sec latency 95.543 ms 2023-07-29T16:34:27.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:26 smithi039 ceph-mon[128858]: pgmap v940: 97 pgs: 97 active+clean; 750 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.0 MiB/s wr, 657 op/s 2023-07-29T16:34:27.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:26 smithi156 ceph-mon[129245]: pgmap v940: 97 pgs: 97 active+clean; 750 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.0 MiB/s wr, 657 op/s 2023-07-29T16:34:27.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:26 smithi031 ceph-mon[95504]: pgmap v940: 97 pgs: 97 active+clean; 750 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.0 MiB/s wr, 657 op/s 2023-07-29T16:34:27.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 167207 36.53 MB/sec execute 131 sec latency 101.681 ms 2023-07-29T16:34:28.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 168097 36.71 MB/sec execute 132 sec latency 64.778 ms 2023-07-29T16:34:28.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:28 smithi031 ceph-mon[95504]: pgmap v941: 97 pgs: 97 active+clean; 748 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 10 MiB/s wr, 651 op/s 2023-07-29T16:34:28.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:28 smithi039 ceph-mon[128858]: pgmap v941: 97 pgs: 97 active+clean; 748 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 10 MiB/s wr, 651 op/s 2023-07-29T16:34:28.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:28 smithi156 ceph-mon[129245]: pgmap v941: 97 pgs: 97 active+clean; 748 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 10 MiB/s wr, 651 op/s 2023-07-29T16:34:29.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 168602 36.63 MB/sec execute 133 sec latency 99.889 ms 2023-07-29T16:34:30.271 INFO:tasks.workunit.client.0.smithi031.stdout: 10 168650 36.38 MB/sec execute 134 sec latency 733.823 ms 2023-07-29T16:34:31.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:30 smithi039 ceph-mon[128858]: pgmap v942: 97 pgs: 97 active+clean; 749 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 765 op/s 2023-07-29T16:34:31.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:30 smithi156 ceph-mon[129245]: pgmap v942: 97 pgs: 97 active+clean; 749 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 765 op/s 2023-07-29T16:34:31.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:30 smithi031 ceph-mon[95504]: pgmap v942: 97 pgs: 97 active+clean; 749 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 765 op/s 2023-07-29T16:34:31.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 169046 36.22 MB/sec execute 135 sec latency 358.111 ms 2023-07-29T16:34:32.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 169641 36.15 MB/sec execute 136 sec latency 128.667 ms 2023-07-29T16:34:33.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:32 smithi039 ceph-mon[128858]: pgmap v943: 97 pgs: 97 active+clean; 724 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 14 MiB/s wr, 851 op/s 2023-07-29T16:34:33.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:32 smithi156 ceph-mon[129245]: pgmap v943: 97 pgs: 97 active+clean; 724 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 14 MiB/s wr, 851 op/s 2023-07-29T16:34:33.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:32 smithi031 ceph-mon[95504]: pgmap v943: 97 pgs: 97 active+clean; 724 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 14 MiB/s wr, 851 op/s 2023-07-29T16:34:33.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 170572 36.28 MB/sec execute 137 sec latency 61.989 ms 2023-07-29T16:34:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 -- 172.21.15.31:0/4167518950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8100340 msgr2=0x7f9fd8100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4167518950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8100340 0x7f9fd8100750 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009990 tx=0x7f9fd4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 -- 172.21.15.31:0/4167518950 shutdown_connections 2023-07-29T16:34:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4167518950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fd8101ec0 0x7f9fd8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4167518950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8100340 0x7f9fd8100750 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4167518950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fd8101530 0x7f9fd8101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 -- 172.21.15.31:0/4167518950 >> 172.21.15.31:0/4167518950 conn(0x7f9fd80fb920 msgr2=0x7f9fd80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 -- 172.21.15.31:0/4167518950 shutdown_connections 2023-07-29T16:34:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.906+0000 7f9fdffff700 1 -- 172.21.15.31:0/4167518950 wait complete. 2023-07-29T16:34:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.907+0000 7f9fdffff700 1 Processor -- start 2023-07-29T16:34:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 -- start start 2023-07-29T16:34:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fd8100340 0x7f9fd81961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 0x7f9fd81966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fd8101ec0 0x7f9fd8197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8197bd0 con 0x7f9fd8100340 2023-07-29T16:34:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8197d40 con 0x7f9fd8101ec0 2023-07-29T16:34:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdffff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8197f00 con 0x7f9fd8101530 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdd59a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 0x7f9fd81966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdd59a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 0x7f9fd81966f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:33172/0 (socket says 172.21.15.31:33172) 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fdd59a700 1 -- 172.21.15.31:0/4083209754 learned_addr learned my addr 172.21.15.31:0/4083209754 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.908+0000 7f9fde59c700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fd8101ec0 0x7f9fd8197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fd8100340 0x7f9fd81961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdd59a700 1 -- 172.21.15.31:0/4083209754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fd8101ec0 msgr2=0x7f9fd8197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fd8101ec0 0x7f9fd8197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdd59a700 1 -- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fd8100340 msgr2=0x7f9fd81961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fd8100340 0x7f9fd81961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdd59a700 1 -- 172.21.15.31:0/4083209754 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40095f0 con 0x7f9fd8101530 2023-07-29T16:34:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 0x7f9fd81966f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800e9a0 tx=0x7f9fc800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fceffd700 1 -- 172.21.15.31:0/4083209754 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8009790 con 0x7f9fd8101530 2023-07-29T16:34:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fceffd700 1 -- 172.21.15.31:0/4083209754 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fc800cbd0 con 0x7f9fd8101530 2023-07-29T16:34:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.909+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd81981e0 con 0x7f9fd8101530 2023-07-29T16:34:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.910+0000 7f9fceffd700 1 -- 172.21.15.31:0/4083209754 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8018680 con 0x7f9fd8101530 2023-07-29T16:34:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.910+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd819e9a0 con 0x7f9fd8101530 2023-07-29T16:34:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.913+0000 7f9fceffd700 1 -- 172.21.15.31:0/4083209754 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9fc800cd40 con 0x7f9fd8101530 2023-07-29T16:34:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.915+0000 7f9fceffd700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fc40a0890 0x7f9fc40a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.915+0000 7f9ff385ab80 1 Processor -- start 2023-07-29T16:34:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.915+0000 7f9fceffd700 1 -- 172.21.15.31:0/4083209754 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9fc8014070 con 0x7f9fd8101530 2023-07-29T16:34:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 -- start start 2023-07-29T16:34:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa107f6b0 0x562aa122d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1223550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 0x562aa1223f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa117b9b0 con 0x562aa107f6b0 2023-07-29T16:34:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa1179370 con 0x562aa1223a90 2023-07-29T16:34:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa1174c60 con 0x562aa122d8c0 2023-07-29T16:34:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fc40a0890 0x7f9fc40a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9fdd59a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1223550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.916+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 0x562aa1223f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 0x562aa1223f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40986/0 (socket says 172.21.15.31:40986) 2023-07-29T16:34:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 -- 172.21.15.31:0/2373519664 learned_addr learned my addr 172.21.15.31:0/2373519664 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa107f6b0 0x562aa122d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 -- 172.21.15.31:0/2373519664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 msgr2=0x562aa1223550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1223550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 -- 172.21.15.31:0/2373519664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa107f6b0 msgr2=0x562aa122d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa107f6b0 0x562aa122d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 -- 172.21.15.31:0/2373519664 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562aa0fa2d80 con 0x562aa1223a90 2023-07-29T16:34:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.917+0000 7f9fde59c700 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 0x562aa1223f00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00098f0 tx=0x7f9fd0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3a5339d772e5015 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.918+0000 7f9fc3fff700 1 -- 172.21.15.31:0/2373519664 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0019070 con 0x562aa1223a90 2023-07-29T16:34:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.918+0000 7f9fc3fff700 1 -- 172.21.15.31:0/2373519664 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fd000f040 con 0x562aa1223a90 2023-07-29T16:34:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.918+0000 7f9fc3fff700 1 -- 172.21.15.31:0/2373519664 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0007aa0 con 0x562aa1223a90 2023-07-29T16:34:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2373519664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 msgr2=0x562aa1223f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 0x562aa1223f00 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00098f0 tx=0x7f9fd0009c00 comp rx=0 tx=0).stop 2023-07-29T16:34:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fc40a0890 0x7f9fc40a2d40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400b2e0 tx=0x7f9fd401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2373519664 shutdown_connections 2023-07-29T16:34:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa1223a90 0x562aa1223f00 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa107f6b0 0x562aa122d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2373519664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1223550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.919+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2373519664 >> 172.21.15.31:0/2373519664 conn(0x562aa1218fd0 msgr2=0x562aa121e310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.923+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2373519664 shutdown_connections 2023-07-29T16:34:33.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.923+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2373519664 wait complete. 2023-07-29T16:34:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.924+0000 7f9ff385ab80 1 -- start start 2023-07-29T16:34:33.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.924+0000 7f9ff385ab80 1 Processor -- start 2023-07-29T16:34:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.925+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1246920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.925+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa1246e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.925+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa124dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.925+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa11114b0 con 0x562aa1223a90 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.925+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa1111150 con 0x562aa107f6b0 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.925+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa0ff88c0 con 0x562aa122d8c0 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9ff385ab80 10 client.0 _ll_register_callbacks cb 0x562aa111a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1246920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa124dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1246920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40994/0 (socket says 172.21.15.31:40994) 2023-07-29T16:34:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fddd9b700 1 -- 172.21.15.31:0/1877958351 learned_addr learned my addr 172.21.15.31:0/1877958351 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa1246e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fde59c700 1 -- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 msgr2=0x562aa1246920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fde59c700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1246920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fde59c700 1 -- 172.21.15.31:0/1877958351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 msgr2=0x562aa1246e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fde59c700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa1246e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fde59c700 1 -- 172.21.15.31:0/1877958351 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd0003c50 con 0x562aa122d8c0 2023-07-29T16:34:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1246920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:34:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.926+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa1246e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:34:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.927+0000 7f9fde59c700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa124dc90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0004fa0 tx=0x7f9fd0004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63454d41289494c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.927+0000 7f9fc17fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:34:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.927+0000 7f9fc17fa700 1 -- 172.21.15.31:0/1877958351 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0019040 con 0x562aa122d8c0 2023-07-29T16:34:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.927+0000 7f9fc17fa700 1 -- 172.21.15.31:0/1877958351 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9fd0007c30 con 0x562aa122d8c0 2023-07-29T16:34:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.927+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562aa119fcb0 con 0x562aa122d8c0 2023-07-29T16:34:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.928+0000 7f9fc17fa700 1 -- 172.21.15.31:0/1877958351 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fd0007e10 con 0x562aa122d8c0 2023-07-29T16:34:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.928+0000 7f9ff385ab80 10 client.30337 fetch_fsmap learned FSMap version 42 2023-07-29T16:34:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.928+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x562aa0f8bc00 con 0x562aa122d8c0 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.928+0000 7f9fc17fa700 1 -- 172.21.15.31:0/1877958351 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562aa119fcb0 con 0x562aa122d8c0 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9fc17fa700 1 -- 172.21.15.31:0/1877958351 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9fd00294c0 con 0x562aa122d8c0 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9ff385ab80 10 client.30337 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9ff385ab80 10 client.30337 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9ff385ab80 20 client.30337 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12525f0 0x562aa1254a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9ff385ab80 4 client.30337 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:34:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562aa11f2d10 con 0x562aa12525f0 2023-07-29T16:34:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.929+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12525f0 0x562aa1254a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.930+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12525f0 0x562aa1254a00 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7af99d0947b8516 server_cookie=e119070d5fdb69c9 in_seq=0 out_seq=0 2023-07-29T16:34:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.930+0000 7f9fc17fa700 10 client.30337 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:33.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.935+0000 7f9fc17fa700 1 -- 172.21.15.31:0/1877958351 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x562aa11f2d10 con 0x562aa12525f0 2023-07-29T16:34:33.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.935+0000 7f9fc17fa700 10 client.30337 handle_command_reply: tid=0 2023-07-29T16:34:33.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.935+0000 7f9ff385ab80 1 client.30337 shutdown 2023-07-29T16:34:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 msgr2=0x562aa124dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa124dc90 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0004fa0 tx=0x7f9fd0004700 comp rx=0 tx=0).stop 2023-07-29T16:34:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 shutdown_connections 2023-07-29T16:34:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12525f0 0x562aa1254a00 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa124dc90 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1246920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/1877958351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa1246e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 >> 172.21.15.31:0/1877958351 conn(0x562aa1218fd0 msgr2=0x562aa121da40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9fc17fa700 0 client.30337 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.936+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 shutdown_connections 2023-07-29T16:34:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.937+0000 7f9ff385ab80 1 -- 172.21.15.31:0/1877958351 wait complete. 2023-07-29T16:34:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.937+0000 7f9ff385ab80 20 client.30337 trim_cache size 0 max 16384 2023-07-29T16:34:33.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.957+0000 7f9ff385ab80 1 Processor -- start 2023-07-29T16:34:33.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.957+0000 7f9ff385ab80 1 -- start start 2023-07-29T16:34:33.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1223550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1233910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa121f1d0 con 0x562aa1223a90 2023-07-29T16:34:33.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa122dc90 con 0x562aa107f6b0 2023-07-29T16:34:33.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa12189d0 con 0x562aa122d8c0 2023-07-29T16:34:33.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9fddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1223550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.958+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1233910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1223550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41006/0 (socket says 172.21.15.31:41006) 2023-07-29T16:34:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 -- 172.21.15.31:0/2786862104 learned_addr learned my addr 172.21.15.31:0/2786862104 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 -- 172.21.15.31:0/2786862104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 msgr2=0x562aa1233910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1233910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 -- 172.21.15.31:0/2786862104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 msgr2=0x562aa124bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 -- 172.21.15.31:0/2786862104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562aa0fc91f0 con 0x562aa107f6b0 2023-07-29T16:34:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fde59c700 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1233910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:34:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.959+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1223550 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4005850 tx=0x7f9fd4021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2aea18fc3274dc29 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:33.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.960+0000 7f9fccff9700 1 -- 172.21.15.31:0/2786862104 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd400be70 con 0x562aa107f6b0 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.960+0000 7f9fccff9700 1 -- 172.21.15.31:0/2786862104 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fd402d040 con 0x562aa107f6b0 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.960+0000 7f9fccff9700 1 -- 172.21.15.31:0/2786862104 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4032890 con 0x562aa107f6b0 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2786862104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 msgr2=0x562aa1223550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1223550 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4005850 tx=0x7f9fd4021000 comp rx=0 tx=0).stop 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2786862104 shutdown_connections 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa122d8c0 0x562aa1233910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa107f6b0 0x562aa1223550 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/2786862104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.961+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2786862104 >> 172.21.15.31:0/2786862104 conn(0x562aa1218fd0 msgr2=0x562aa1254600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:33.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.962+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2786862104 shutdown_connections 2023-07-29T16:34:33.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.963+0000 7f9ff385ab80 1 -- 172.21.15.31:0/2786862104 wait complete. 2023-07-29T16:34:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.964+0000 7f9ff385ab80 1 -- start start 2023-07-29T16:34:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.964+0000 7f9ff385ab80 1 Processor -- start 2023-07-29T16:34:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa107f6b0 0x562aa124b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa122d8c0 0x562aa1249970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa1249eb0 con 0x562aa1223a90 2023-07-29T16:34:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa124a020 con 0x562aa122d8c0 2023-07-29T16:34:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x562aa124a1e0 con 0x562aa107f6b0 2023-07-29T16:34:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9ff385ab80 10 client.0 _ll_register_callbacks cb 0x562aa111a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa122d8c0 0x562aa1249970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.965+0000 7f9fde59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa122d8c0 0x562aa1249970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:41016/0 (socket says 172.21.15.31:41016) 2023-07-29T16:34:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fde59c700 1 -- 172.21.15.31:0/3065691151 learned_addr learned my addr 172.21.15.31:0/3065691151 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa107f6b0 0x562aa124b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 -- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa122d8c0 msgr2=0x562aa1249970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa122d8c0 0x562aa1249970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 -- 172.21.15.31:0/3065691151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 msgr2=0x562aa124b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 -- 172.21.15.31:0/3065691151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562aa0f8bc00 con 0x562aa107f6b0 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fddd9b700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa107f6b0 0x562aa124b360 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4032470 tx=0x7f9fd4005610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15fe18b4e2181b88 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.966+0000 7f9fc17fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.967+0000 7f9fc17fa700 1 -- 172.21.15.31:0/3065691151 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4033a40 con 0x562aa107f6b0 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.967+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fd000f040 con 0x562aa107f6b0 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.967+0000 7f9fc17fa700 1 -- 172.21.15.31:0/3065691151 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9fd4033ba0 con 0x562aa107f6b0 2023-07-29T16:34:33.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.967+0000 7f9fc17fa700 1 -- 172.21.15.31:0/3065691151 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fd4033480 con 0x562aa107f6b0 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.967+0000 7f9ff385ab80 10 client.30343 fetch_fsmap learned FSMap version 42 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.967+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x7f9fd81090b0 con 0x562aa107f6b0 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9fc17fa700 1 -- 172.21.15.31:0/3065691151 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fd000f040 con 0x562aa107f6b0 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9fc17fa700 1 -- 172.21.15.31:0/3065691151 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9fd4047420 con 0x562aa107f6b0 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9ff385ab80 10 client.30343 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9ff385ab80 10 client.30343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:33.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9ff385ab80 20 client.30343 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12a4e90 0x562aa12a7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9ff385ab80 4 client.30343 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:34:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562aa116dce0 con 0x562aa12a4e90 2023-07-29T16:34:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.968+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12a4e90 0x562aa12a7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:33.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.970+0000 7f9fdd59a700 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12a4e90 0x562aa12a7270 crc :-1 s=READY pgs=460 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f449afcc525fe83 server_cookie=746b3bcc869b31a in_seq=0 out_seq=0 2023-07-29T16:34:33.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:33.970+0000 7f9fc17fa700 10 client.30343 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:34.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 170960 36.20 MB/sec execute 138 sec latency 396.020 ms 2023-07-29T16:34:34.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.304+0000 7f9fc17fa700 1 -- 172.21.15.31:0/3065691151 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562aa116dce0 con 0x562aa12a4e90 2023-07-29T16:34:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.304+0000 7f9fc17fa700 10 client.30343 handle_command_reply: tid=0 2023-07-29T16:34:34.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.306+0000 7f9ff385ab80 1 client.30343 shutdown 2023-07-29T16:34:34.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.306+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa107f6b0 msgr2=0x562aa124b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.306+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa107f6b0 0x562aa124b360 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4032470 tx=0x7f9fd4005610 comp rx=0 tx=0).stop 2023-07-29T16:34:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.306+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 shutdown_connections 2023-07-29T16:34:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.306+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x562aa12a4e90 0x562aa12a7270 crc :-1 s=READY pgs=460 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562aa122d8c0 0x562aa1249970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562aa107f6b0 0x562aa124b360 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 1 --2- 172.21.15.31:0/3065691151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562aa1223a90 0x562aa124b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 >> 172.21.15.31:0/3065691151 conn(0x562aa1218fd0 msgr2=0x562aa1233910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9fc17fa700 0 client.30343 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 shutdown_connections 2023-07-29T16:34:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 1 -- 172.21.15.31:0/3065691151 wait complete. 2023-07-29T16:34:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.307+0000 7f9ff385ab80 20 client.30343 trim_cache size 0 max 16384 2023-07-29T16:34:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:34:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:34:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:34:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:34:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.308+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fc40a0890 msgr2=0x7f9fc40a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:34.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.308+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fc40a0890 0x7f9fc40a2d40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400b2e0 tx=0x7f9fd401a040 comp rx=0 tx=0).stop 2023-07-29T16:34:34.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.308+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 msgr2=0x7f9fd81966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:34.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.308+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 0x7f9fd81966f0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f9fc800e9a0 tx=0x7f9fc800ecb0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.312+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 shutdown_connections 2023-07-29T16:34:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.312+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9fc40a0890 0x7f9fc40a2d40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.312+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9fd8101ec0 0x7f9fd8197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.312+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9fd8100340 0x7f9fd81961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.312+0000 7f9fdffff700 1 --2- 172.21.15.31:0/4083209754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd8101530 0x7f9fd81966f0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:34.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.313+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 >> 172.21.15.31:0/4083209754 conn(0x7f9fd80fb920 msgr2=0x7f9fd80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:34.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.315+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 shutdown_connections 2023-07-29T16:34:34.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:34.316+0000 7f9fdffff700 1 -- 172.21.15.31:0/4083209754 wait complete. 2023-07-29T16:34:35.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:34 smithi039 ceph-mon[128858]: pgmap v944: 97 pgs: 97 active+clean; 713 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 19 MiB/s wr, 1.06k op/s 2023-07-29T16:34:35.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:34 smithi156 ceph-mon[129245]: pgmap v944: 97 pgs: 97 active+clean; 713 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 19 MiB/s wr, 1.06k op/s 2023-07-29T16:34:35.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:34 smithi031 ceph-mon[95504]: pgmap v944: 97 pgs: 97 active+clean; 713 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 19 MiB/s wr, 1.06k op/s 2023-07-29T16:34:35.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 171647 36.20 MB/sec execute 139 sec latency 120.481 ms 2023-07-29T16:34:36.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 172203 36.19 MB/sec execute 140 sec latency 93.770 ms 2023-07-29T16:34:37.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:36 smithi039 ceph-mon[128858]: pgmap v945: 97 pgs: 97 active+clean; 679 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 20 MiB/s wr, 991 op/s 2023-07-29T16:34:37.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:36 smithi156 ceph-mon[129245]: pgmap v945: 97 pgs: 97 active+clean; 679 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 20 MiB/s wr, 991 op/s 2023-07-29T16:34:37.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:36 smithi031 ceph-mon[95504]: pgmap v945: 97 pgs: 97 active+clean; 679 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 20 MiB/s wr, 991 op/s 2023-07-29T16:34:37.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 172841 36.14 MB/sec execute 141 sec latency 64.036 ms 2023-07-29T16:34:38.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 173540 36.14 MB/sec execute 142 sec latency 102.134 ms 2023-07-29T16:34:38.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:38 smithi031 ceph-mon[95504]: pgmap v946: 97 pgs: 97 active+clean; 681 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 16 MiB/s wr, 956 op/s 2023-07-29T16:34:38.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:38 smithi039 ceph-mon[128858]: pgmap v946: 97 pgs: 97 active+clean; 681 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 16 MiB/s wr, 956 op/s 2023-07-29T16:34:38.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:38 smithi156 ceph-mon[129245]: pgmap v946: 97 pgs: 97 active+clean; 681 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 16 MiB/s wr, 956 op/s 2023-07-29T16:34:39.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 174213 36.12 MB/sec execute 143 sec latency 68.137 ms 2023-07-29T16:34:40.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 174938 36.21 MB/sec execute 144 sec latency 104.550 ms 2023-07-29T16:34:41.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:40 smithi039 ceph-mon[128858]: pgmap v947: 97 pgs: 97 active+clean; 687 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 14 MiB/s wr, 956 op/s 2023-07-29T16:34:41.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:40 smithi156 ceph-mon[129245]: pgmap v947: 97 pgs: 97 active+clean; 687 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 14 MiB/s wr, 956 op/s 2023-07-29T16:34:41.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:40 smithi031 ceph-mon[95504]: pgmap v947: 97 pgs: 97 active+clean; 687 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 14 MiB/s wr, 956 op/s 2023-07-29T16:34:41.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 175566 36.15 MB/sec execute 145 sec latency 77.408 ms 2023-07-29T16:34:42.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 176235 36.18 MB/sec execute 146 sec latency 97.467 ms 2023-07-29T16:34:43.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:42 smithi039 ceph-mon[128858]: pgmap v948: 97 pgs: 97 active+clean; 685 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 801 op/s 2023-07-29T16:34:43.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:42 smithi156 ceph-mon[129245]: pgmap v948: 97 pgs: 97 active+clean; 685 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 801 op/s 2023-07-29T16:34:43.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:42 smithi031 ceph-mon[95504]: pgmap v948: 97 pgs: 97 active+clean; 685 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 801 op/s 2023-07-29T16:34:43.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 176770 36.10 MB/sec execute 147 sec latency 120.761 ms 2023-07-29T16:34:44.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 177593 36.17 MB/sec execute 148 sec latency 71.734 ms 2023-07-29T16:34:45.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:44 smithi039 ceph-mon[128858]: pgmap v949: 97 pgs: 97 active+clean; 693 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 10 MiB/s wr, 810 op/s 2023-07-29T16:34:45.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:44 smithi156 ceph-mon[129245]: pgmap v949: 97 pgs: 97 active+clean; 693 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 10 MiB/s wr, 810 op/s 2023-07-29T16:34:45.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:44 smithi031 ceph-mon[95504]: pgmap v949: 97 pgs: 97 active+clean; 693 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 10 MiB/s wr, 810 op/s 2023-07-29T16:34:45.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 178456 36.27 MB/sec execute 149 sec latency 52.964 ms 2023-07-29T16:34:46.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 179055 36.21 MB/sec execute 150 sec latency 128.560 ms 2023-07-29T16:34:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:46 smithi039 ceph-mon[128858]: pgmap v950: 97 pgs: 97 active+clean; 697 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.2 MiB/s wr, 590 op/s 2023-07-29T16:34:47.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:46 smithi156 ceph-mon[129245]: pgmap v950: 97 pgs: 97 active+clean; 697 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.2 MiB/s wr, 590 op/s 2023-07-29T16:34:47.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:46 smithi031 ceph-mon[95504]: pgmap v950: 97 pgs: 97 active+clean; 697 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.2 MiB/s wr, 590 op/s 2023-07-29T16:34:47.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 179751 36.21 MB/sec execute 151 sec latency 60.822 ms 2023-07-29T16:34:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 -- 172.21.15.31:0/520800805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0109a20 msgr2=0x7f47d0071950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 --2- 172.21.15.31:0/520800805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0109a20 0x7f47d0071950 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f47c0009990 tx=0x7f47c0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 -- 172.21.15.31:0/520800805 shutdown_connections 2023-07-29T16:34:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 --2- 172.21.15.31:0/520800805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47d0072010 0x7f47d00724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 --2- 172.21.15.31:0/520800805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0109040 0x7f47d0109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 --2- 172.21.15.31:0/520800805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0109a20 0x7f47d0071950 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 -- 172.21.15.31:0/520800805 >> 172.21.15.31:0/520800805 conn(0x7f47d006d5c0 msgr2=0x7f47d006f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 -- 172.21.15.31:0/520800805 shutdown_connections 2023-07-29T16:34:47.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.561+0000 7f47d77ff700 1 -- 172.21.15.31:0/520800805 wait complete. 2023-07-29T16:34:47.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.562+0000 7f47d77ff700 1 Processor -- start 2023-07-29T16:34:47.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 -- start start 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 0x7f47d019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47d0109040 0x7f47d019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0109a20 0x7f47d019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d01a03a0 con 0x7f47d0109a20 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d01a0510 con 0x7f47d0109040 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d77ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d01a06d0 con 0x7f47d0072010 2023-07-29T16:34:47.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.563+0000 7f47d559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 0x7f47d019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47d0109040 0x7f47d019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 0x7f47d019e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37202/0 (socket says 172.21.15.31:37202) 2023-07-29T16:34:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 -- 172.21.15.31:0/983649865 learned_addr learned my addr 172.21.15.31:0/983649865 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:47.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0109a20 0x7f47d019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 -- 172.21.15.31:0/983649865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47d0109040 msgr2=0x7f47d019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47d0109040 0x7f47d019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 -- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0109a20 msgr2=0x7f47d019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0109a20 0x7f47d019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 -- 172.21.15.31:0/983649865 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c00095f0 con 0x7f47d0072010 2023-07-29T16:34:47.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47d559b700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 0x7f47d019e980 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00e970 tx=0x7f47cc00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.564+0000 7f47c67fc700 1 -- 172.21.15.31:0/983649865 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc009790 con 0x7f47d0072010 2023-07-29T16:34:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.565+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d01a09b0 con 0x7f47d0072010 2023-07-29T16:34:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.565+0000 7f47c67fc700 1 -- 172.21.15.31:0/983649865 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47cc00eec0 con 0x7f47d0072010 2023-07-29T16:34:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.565+0000 7f47c67fc700 1 -- 172.21.15.31:0/983649865 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc003cf0 con 0x7f47d0072010 2023-07-29T16:34:47.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.565+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d01107b0 con 0x7f47d0072010 2023-07-29T16:34:47.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.569+0000 7f47c67fc700 1 -- 172.21.15.31:0/983649865 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f47cc00ccd0 con 0x7f47d0072010 2023-07-29T16:34:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.572+0000 7f47eb13fb80 1 Processor -- start 2023-07-29T16:34:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.572+0000 7f47c67fc700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47bc0a0810 0x7f47bc0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.573+0000 7f47c67fc700 1 -- 172.21.15.31:0/983649865 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f47cc014070 con 0x7f47d0072010 2023-07-29T16:34:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.573+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47bc0a0810 0x7f47bc0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.573+0000 7f47eb13fb80 1 -- start start 2023-07-29T16:34:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd2376b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2376c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 0x55cd2376d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236c39b0 con 0x55cd2376bdf0 2023-07-29T16:34:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236c1370 con 0x55cd235c76b0 2023-07-29T16:34:47.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236bcc60 con 0x55cd2376ccf0 2023-07-29T16:34:47.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47d5d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 0x55cd2376d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47d5d9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 0x55cd2376d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37218/0 (socket says 172.21.15.31:37218) 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.574+0000 7f47d5d9c700 1 -- 172.21.15.31:0/1543879770 learned_addr learned my addr 172.21.15.31:0/1543879770 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47bc0a0810 0x7f47bc0a2cc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f47c800f270 tx=0x7f47c800a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 -- 172.21.15.31:0/1543879770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 msgr2=0x55cd2376b8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd2376b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2376c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 -- 172.21.15.31:0/1543879770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 msgr2=0x55cd2376c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2376c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 -- 172.21.15.31:0/1543879770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd234ead80 con 0x55cd2376ccf0 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2376c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 0x55cd2376d160 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f47c800fdb0 tx=0x7f47c8012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20737e157ae7d5a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47bb7fe700 1 -- 172.21.15.31:0/1543879770 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8016c50 con 0x55cd2376ccf0 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47bb7fe700 1 -- 172.21.15.31:0/1543879770 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47c8016db0 con 0x55cd2376ccf0 2023-07-29T16:34:47.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.575+0000 7f47bb7fe700 1 -- 172.21.15.31:0/1543879770 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8014840 con 0x55cd2376ccf0 2023-07-29T16:34:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 -- 172.21.15.31:0/1543879770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 msgr2=0x55cd2376d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 0x55cd2376d160 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f47c800fdb0 tx=0x7f47c8012000 comp rx=0 tx=0).stop 2023-07-29T16:34:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 -- 172.21.15.31:0/1543879770 shutdown_connections 2023-07-29T16:34:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376ccf0 0x55cd2376d160 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd2376b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/1543879770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2376c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.576+0000 7f47eb13fb80 1 -- 172.21.15.31:0/1543879770 >> 172.21.15.31:0/1543879770 conn(0x55cd23760fd0 msgr2=0x55cd23766300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.577+0000 7f47eb13fb80 1 -- 172.21.15.31:0/1543879770 shutdown_connections 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.577+0000 7f47eb13fb80 1 -- 172.21.15.31:0/1543879770 wait complete. 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.577+0000 7f47eb13fb80 1 -- start start 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.577+0000 7f47eb13fb80 1 Processor -- start 2023-07-29T16:34:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd2377d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2377dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 0x55cd23784d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236594b0 con 0x55cd2376bdf0 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd23659150 con 0x55cd2376ccf0 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd235408c0 con 0x55cd235c76b0 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47eb13fb80 10 client.0 _ll_register_callbacks cb 0x55cd236625c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.578+0000 7f47d559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd2377d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d5d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 0x55cd23784d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d5d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 0x55cd23784d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54900/0 (socket says 172.21.15.31:54900) 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d5d9c700 1 -- 172.21.15.31:0/3285721183 learned_addr learned my addr 172.21.15.31:0/3285721183 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d559b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd2377d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:37232/0 (socket says 172.21.15.31:37232) 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d559b700 1 -- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 msgr2=0x55cd23784d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d559b700 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 0x55cd23784d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d559b700 1 -- 172.21.15.31:0/3285721183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 msgr2=0x55cd2377dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d559b700 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2377dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d559b700 1 -- 172.21.15.31:0/3285721183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c8010040 con 0x55cd235c76b0 2023-07-29T16:34:47.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.579+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 0x55cd23784d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:34:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47d559b700 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd2377d790 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0991c0 tx=0x7f47cc098e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea4a237ca99ab9f2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47b8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:34:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3285721183 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47cc0b1ea0 con 0x55cd235c76b0 2023-07-29T16:34:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3285721183 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f47cc0994c0 con 0x55cd235c76b0 2023-07-29T16:34:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cd236e7cb0 con 0x55cd235c76b0 2023-07-29T16:34:47.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3285721183 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47cc09d4f0 con 0x55cd235c76b0 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47eb13fb80 10 client.30355 fetch_fsmap learned FSMap version 42 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55cd234d3c00 con 0x55cd235c76b0 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.580+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3285721183 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cd236e7cb0 con 0x55cd235c76b0 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3285721183 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f47cc0c9d20 con 0x55cd235c76b0 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47eb13fb80 10 client.30355 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47eb13fb80 10 client.30355 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47eb13fb80 20 client.30355 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:47.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd237893d0 0x55cd2378b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47eb13fb80 4 client.30355 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:34:47.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd2373ad10 con 0x55cd237893d0 2023-07-29T16:34:47.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.581+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd237893d0 0x55cd2378b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.582+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd237893d0 0x55cd2378b7e0 crc :-1 s=READY pgs=461 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f48ddca9490c87d server_cookie=cd4bcee4a2cc1e9f in_seq=0 out_seq=0 2023-07-29T16:34:47.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.582+0000 7f47b8ff9700 10 client.30355 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.586+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3285721183 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55cd2373ad10 con 0x55cd237893d0 2023-07-29T16:34:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.586+0000 7f47b8ff9700 10 client.30355 handle_command_reply: tid=0 2023-07-29T16:34:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 client.30355 shutdown 2023-07-29T16:34:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 msgr2=0x55cd2377d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd2377d790 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0991c0 tx=0x7f47cc098e20 comp rx=0 tx=0).stop 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 shutdown_connections 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd237893d0 0x55cd2378b7e0 crc :-1 s=READY pgs=461 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd2376ccf0 0x55cd23784d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd2377d790 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3285721183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd2377dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 >> 172.21.15.31:0/3285721183 conn(0x55cd23760fd0 msgr2=0x55cd23765a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:47.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47b8ff9700 0 client.30355 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.587+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 shutdown_connections 2023-07-29T16:34:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.588+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3285721183 wait complete. 2023-07-29T16:34:47.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.588+0000 7f47eb13fb80 20 client.30355 trim_cache size 0 max 16384 2023-07-29T16:34:47.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 Processor -- start 2023-07-29T16:34:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 -- start start 2023-07-29T16:34:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd23760fd0 0x55cd2378b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd23784cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376bdf0 0x55cd2376fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd236e7cb0 con 0x55cd23760fd0 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd2375e2e0 con 0x55cd235c76b0 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.612+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd2377e980 con 0x55cd2376bdf0 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd23784cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd23784cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54910/0 (socket says 172.21.15.31:54910) 2023-07-29T16:34:47.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 -- 172.21.15.31:0/722355725 learned_addr learned my addr 172.21.15.31:0/722355725 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376bdf0 0x55cd2376fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 -- 172.21.15.31:0/722355725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376bdf0 msgr2=0x55cd2376fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376bdf0 0x55cd2376fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d559b700 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd23760fd0 0x55cd2378b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 -- 172.21.15.31:0/722355725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd23760fd0 msgr2=0x55cd2378b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd23760fd0 0x55cd2378b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 -- 172.21.15.31:0/722355725 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd235111f0 con 0x55cd235c76b0 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d5d9c700 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376bdf0 0x55cd2376fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.613+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd23784cd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f47c0000c00 tx=0x7f47c000ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e090a5f0f0333 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.614+0000 7f47bbfff700 1 -- 172.21.15.31:0/722355725 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000bd90 con 0x55cd235c76b0 2023-07-29T16:34:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.614+0000 7f47bbfff700 1 -- 172.21.15.31:0/722355725 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47c001e070 con 0x55cd235c76b0 2023-07-29T16:34:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.614+0000 7f47bbfff700 1 -- 172.21.15.31:0/722355725 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c00054c0 con 0x55cd235c76b0 2023-07-29T16:34:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 -- 172.21.15.31:0/722355725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 msgr2=0x55cd23784cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd23784cd0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f47c0000c00 tx=0x7f47c000ba20 comp rx=0 tx=0).stop 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 -- 172.21.15.31:0/722355725 shutdown_connections 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd2376bdf0 0x55cd2376fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd23760fd0 0x55cd2378b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/722355725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd235c76b0 0x55cd23784cd0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.615+0000 7f47eb13fb80 1 -- 172.21.15.31:0/722355725 >> 172.21.15.31:0/722355725 conn(0x55cd2376d3f0 msgr2=0x55cd23773110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:47.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.616+0000 7f47eb13fb80 1 -- 172.21.15.31:0/722355725 shutdown_connections 2023-07-29T16:34:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.616+0000 7f47eb13fb80 1 -- 172.21.15.31:0/722355725 wait complete. 2023-07-29T16:34:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.616+0000 7f47eb13fb80 1 -- start start 2023-07-29T16:34:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.616+0000 7f47eb13fb80 1 Processor -- start 2023-07-29T16:34:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd23780f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 0x55cd23781440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd23783d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd237842a0 con 0x55cd2376bdf0 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd23784410 con 0x55cd23760fd0 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd237845d0 con 0x55cd235c76b0 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47eb13fb80 10 client.0 _ll_register_callbacks cb 0x55cd236625c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47d4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 0x55cd23781440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47d4d9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 0x55cd23781440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54920/0 (socket says 172.21.15.31:54920) 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.617+0000 7f47d4d9a700 1 -- 172.21.15.31:0/3670999279 learned_addr learned my addr 172.21.15.31:0/3670999279 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d559b700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd23780f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d4d9a700 1 -- 172.21.15.31:0/3670999279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 msgr2=0x55cd23780f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd23780f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d4d9a700 1 -- 172.21.15.31:0/3670999279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 msgr2=0x55cd23783d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd23783d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d4d9a700 1 -- 172.21.15.31:0/3670999279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd234ead80 con 0x55cd23760fd0 2023-07-29T16:34:47.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d559b700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd23780f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47d4d9a700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 0x55cd23781440 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f47c0005f30 tx=0x7f47c0009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3d4ec2910689837 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47b8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3670999279 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c000bee0 con 0x55cd23760fd0 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.618+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47c8016db0 con 0x55cd23760fd0 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3670999279 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f47c0005440 con 0x55cd23760fd0 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3670999279 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47c0005600 con 0x55cd23760fd0 2023-07-29T16:34:47.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 10 client.30350 fetch_fsmap learned FSMap version 42 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x7f47d0107f10 con 0x55cd23760fd0 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3670999279 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f47c8016db0 con 0x55cd23760fd0 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3670999279 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f47c0004a30 con 0x55cd23760fd0 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 10 client.30350 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 10 client.30350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 20 client.30350 populate_metadata read hostname 'smithi031' 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd2376ccf0 0x55cd237dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:34:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 4 client.30350 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:34:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.619+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cd236b5ce0 con 0x55cd2376ccf0 2023-07-29T16:34:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.620+0000 7f47d559b700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd2376ccf0 0x55cd237dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:34:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.620+0000 7f47b8ff9700 10 client.30350 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:47.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.620+0000 7f47d559b700 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd2376ccf0 0x55cd237dd8d0 crc :-1 s=READY pgs=462 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ba93e32d9c0d43d server_cookie=885138b4f7815ab1 in_seq=0 out_seq=0 2023-07-29T16:34:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47b8ff9700 1 -- 172.21.15.31:0/3670999279 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cd236b5ce0 con 0x55cd2376ccf0 2023-07-29T16:34:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47b8ff9700 10 client.30350 handle_command_reply: tid=0 2023-07-29T16:34:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 client.30350 shutdown 2023-07-29T16:34:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 msgr2=0x55cd23781440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 0x55cd23781440 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f47c0005f30 tx=0x7f47c0009290 comp rx=0 tx=0).stop 2023-07-29T16:34:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 shutdown_connections 2023-07-29T16:34:47.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cd2376ccf0 0x55cd237dd8d0 crc :-1 s=READY pgs=462 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd2376bdf0 0x55cd23783d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cd235c76b0 0x55cd23780f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 --2- 172.21.15.31:0/3670999279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cd23760fd0 0x55cd23781440 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 >> 172.21.15.31:0/3670999279 conn(0x55cd2376d3f0 msgr2=0x55cd2378b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.958+0000 7f47b8ff9700 0 client.30350 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.959+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 shutdown_connections 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.959+0000 7f47eb13fb80 1 -- 172.21.15.31:0/3670999279 wait complete. 2023-07-29T16:34:47.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.959+0000 7f47eb13fb80 20 client.30350 trim_cache size 0 max 16384 2023-07-29T16:34:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:34:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:34:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:34:47.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47bc0a0810 msgr2=0x7f47bc0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47bc0a0810 0x7f47bc0a2cc0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f47c800f270 tx=0x7f47c800a040 comp rx=0 tx=0).stop 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 msgr2=0x7f47d019e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 0x7f47d019e980 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00e970 tx=0x7f47cc00ec80 comp rx=0 tx=0).stop 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 shutdown_connections 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f47bc0a0810 0x7f47bc0a2cc0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f47d0109a20 0x7f47d019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47d0072010 0x7f47d019e980 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 --2- 172.21.15.31:0/983649865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f47d0109040 0x7f47d019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:34:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 >> 172.21.15.31:0/983649865 conn(0x7f47d006d5c0 msgr2=0x7f47d006f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:34:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 shutdown_connections 2023-07-29T16:34:47.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:34:47.960+0000 7f47d77ff700 1 -- 172.21.15.31:0/983649865 wait complete. 2023-07-29T16:34:48.272 INFO:tasks.workunit.client.0.smithi031.stdout: 10 180369 36.17 MB/sec execute 152 sec latency 260.029 ms 2023-07-29T16:34:48.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:48 smithi031 ceph-mon[95504]: pgmap v951: 97 pgs: 97 active+clean; 698 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 449 op/s 2023-07-29T16:34:48.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:48 smithi039 ceph-mon[128858]: pgmap v951: 97 pgs: 97 active+clean; 698 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 449 op/s 2023-07-29T16:34:48.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:48 smithi156 ceph-mon[129245]: pgmap v951: 97 pgs: 97 active+clean; 698 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 449 op/s 2023-07-29T16:34:49.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 181172 36.24 MB/sec execute 153 sec latency 60.186 ms 2023-07-29T16:34:50.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 181916 36.30 MB/sec execute 154 sec latency 53.020 ms 2023-07-29T16:34:51.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:50 smithi039 ceph-mon[128858]: pgmap v952: 97 pgs: 97 active+clean; 708 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 493 op/s 2023-07-29T16:34:51.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:50 smithi156 ceph-mon[129245]: pgmap v952: 97 pgs: 97 active+clean; 708 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 493 op/s 2023-07-29T16:34:51.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:50 smithi031 ceph-mon[95504]: pgmap v952: 97 pgs: 97 active+clean; 708 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 493 op/s 2023-07-29T16:34:51.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 182453 36.25 MB/sec execute 155 sec latency 48.098 ms 2023-07-29T16:34:52.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 183073 36.21 MB/sec execute 156 sec latency 62.905 ms 2023-07-29T16:34:53.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:52 smithi039 ceph-mon[128858]: pgmap v953: 97 pgs: 97 active+clean; 708 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.9 MiB/s wr, 536 op/s 2023-07-29T16:34:53.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:52 smithi156 ceph-mon[129245]: pgmap v953: 97 pgs: 97 active+clean; 708 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.9 MiB/s wr, 536 op/s 2023-07-29T16:34:53.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:52 smithi031 ceph-mon[95504]: pgmap v953: 97 pgs: 97 active+clean; 708 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.9 MiB/s wr, 536 op/s 2023-07-29T16:34:53.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 183780 36.18 MB/sec execute 157 sec latency 76.821 ms 2023-07-29T16:34:54.273 INFO:tasks.workunit.client.0.smithi031.stdout: 10 184626 36.26 MB/sec execute 158 sec latency 71.847 ms 2023-07-29T16:34:55.031 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:54 smithi039 ceph-mon[128858]: pgmap v954: 97 pgs: 97 active+clean; 704 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.0 MiB/s wr, 576 op/s 2023-07-29T16:34:55.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:54 smithi156 ceph-mon[129245]: pgmap v954: 97 pgs: 97 active+clean; 704 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.0 MiB/s wr, 576 op/s 2023-07-29T16:34:55.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:54 smithi031 ceph-mon[95504]: pgmap v954: 97 pgs: 97 active+clean; 704 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.0 MiB/s wr, 576 op/s 2023-07-29T16:34:55.274 INFO:tasks.workunit.client.0.smithi031.stdout: 10 185305 36.29 MB/sec execute 159 sec latency 60.757 ms 2023-07-29T16:34:56.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 185915 36.28 MB/sec execute 160 sec latency 52.331 ms 2023-07-29T16:34:57.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:56 smithi039 ceph-mon[128858]: pgmap v955: 97 pgs: 97 active+clean; 716 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.9 MiB/s wr, 554 op/s 2023-07-29T16:34:57.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:56 smithi156 ceph-mon[129245]: pgmap v955: 97 pgs: 97 active+clean; 716 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.9 MiB/s wr, 554 op/s 2023-07-29T16:34:57.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:56 smithi031 ceph-mon[95504]: pgmap v955: 97 pgs: 97 active+clean; 716 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.9 MiB/s wr, 554 op/s 2023-07-29T16:34:57.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 186700 36.33 MB/sec execute 161 sec latency 47.596 ms 2023-07-29T16:34:58.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 187423 36.33 MB/sec execute 162 sec latency 114.996 ms 2023-07-29T16:34:58.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:34:58 smithi031 ceph-mon[95504]: pgmap v956: 97 pgs: 97 active+clean; 719 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 548 op/s 2023-07-29T16:34:58.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:34:58 smithi039 ceph-mon[128858]: pgmap v956: 97 pgs: 97 active+clean; 719 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 548 op/s 2023-07-29T16:34:58.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:34:58 smithi156 ceph-mon[129245]: pgmap v956: 97 pgs: 97 active+clean; 719 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 548 op/s 2023-07-29T16:34:59.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 188016 36.28 MB/sec execute 163 sec latency 83.356 ms 2023-07-29T16:35:00.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 188756 36.27 MB/sec execute 164 sec latency 143.149 ms 2023-07-29T16:35:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:00 smithi039 ceph-mon[128858]: pgmap v957: 97 pgs: 97 active+clean; 718 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 568 op/s 2023-07-29T16:35:01.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:00 smithi156 ceph-mon[129245]: pgmap v957: 97 pgs: 97 active+clean; 718 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 568 op/s 2023-07-29T16:35:01.185 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:00 smithi031 ceph-mon[95504]: pgmap v957: 97 pgs: 97 active+clean; 718 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 568 op/s 2023-07-29T16:35:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.206+0000 7f76a29ff700 1 -- 172.21.15.31:0/437976641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c0714d0 msgr2=0x7f769c071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.206+0000 7f76a29ff700 1 --2- 172.21.15.31:0/437976641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c0714d0 0x7f769c071940 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7690009990 tx=0x7f7690009ca0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.207+0000 7f76a29ff700 1 -- 172.21.15.31:0/437976641 shutdown_connections 2023-07-29T16:35:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.207+0000 7f76a29ff700 1 --2- 172.21.15.31:0/437976641 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f769c071e80 0x7f769c1110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.207+0000 7f76a29ff700 1 --2- 172.21.15.31:0/437976641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f769c108260 0x7f769c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.207+0000 7f76a29ff700 1 --2- 172.21.15.31:0/437976641 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c0714d0 0x7f769c071940 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.207+0000 7f76a29ff700 1 -- 172.21.15.31:0/437976641 >> 172.21.15.31:0/437976641 conn(0x7f769c06d290 msgr2=0x7f769c06f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.207+0000 7f76a29ff700 1 -- 172.21.15.31:0/437976641 shutdown_connections 2023-07-29T16:35:01.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.208+0000 7f76a29ff700 1 -- 172.21.15.31:0/437976641 wait complete. 2023-07-29T16:35:01.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.209+0000 7f76a29ff700 1 Processor -- start 2023-07-29T16:35:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.210+0000 7f76a29ff700 1 -- start start 2023-07-29T16:35:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.210+0000 7f76a29ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f769c0714d0 0x7f769c1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.210+0000 7f76a29ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 0x7f769c1a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.210+0000 7f76a29ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f769c108260 0x7f769c1a1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.211+0000 7f76a29ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1a25f0 con 0x7f769c108260 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.211+0000 7f76a29ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1a2760 con 0x7f769c0714d0 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.211+0000 7f76a29ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c1a2920 con 0x7f769c071e80 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.211+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 0x7f769c1a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.211+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 0x7f769c1a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45322/0 (socket says 172.21.15.31:45322) 2023-07-29T16:35:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.211+0000 7f769b7fe700 1 -- 172.21.15.31:0/1390422960 learned_addr learned my addr 172.21.15.31:0/1390422960 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.212+0000 7f769b7fe700 1 -- 172.21.15.31:0/1390422960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f769c0714d0 msgr2=0x7f769c1a0bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.212+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f769c0714d0 0x7f769c1a0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.212+0000 7f769b7fe700 1 -- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f769c108260 msgr2=0x7f769c1a1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.212+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f769c108260 0x7f769c1a1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.212+0000 7f769b7fe700 1 -- 172.21.15.31:0/1390422960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76900095f0 con 0x7f769c071e80 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.212+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 0x7f769c1a1110 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f769000b680 tx=0x7f769000f460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.213+0000 7f76997fa700 1 -- 172.21.15.31:0/1390422960 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769001d070 con 0x7f769c071e80 2023-07-29T16:35:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.213+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c1a2b40 con 0x7f769c071e80 2023-07-29T16:35:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.213+0000 7f76997fa700 1 -- 172.21.15.31:0/1390422960 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f769000fc30 con 0x7f769c071e80 2023-07-29T16:35:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.213+0000 7f76997fa700 1 -- 172.21.15.31:0/1390422960 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690017650 con 0x7f769c071e80 2023-07-29T16:35:01.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.213+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c0731e0 con 0x7f769c071e80 2023-07-29T16:35:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.219+0000 7f76997fa700 1 -- 172.21.15.31:0/1390422960 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f7690021410 con 0x7f769c071e80 2023-07-29T16:35:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.220+0000 7f76b6327b80 1 Processor -- start 2023-07-29T16:35:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 -- start start 2023-07-29T16:35:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f03b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f039e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f03a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898ef899b0 con 0x55898f03b770 2023-07-29T16:35:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898ef87370 con 0x55898ee8d6b0 2023-07-29T16:35:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.221+0000 7f76b6327b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898ef82c60 con 0x55898f03a380 2023-07-29T16:35:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f03a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f03a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45336/0 (socket says 172.21.15.31:45336) 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 -- 172.21.15.31:0/3678699488 learned_addr learned my addr 172.21.15.31:0/3678699488 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769bfff700 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f039e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f76a0f9c700 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f03b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f76997fa700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f76840c28f0 0x7f76840c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 -- 172.21.15.31:0/3678699488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 msgr2=0x55898f03b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f03b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 -- 172.21.15.31:0/3678699488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 msgr2=0x55898f039e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f039e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f769b7fe700 1 -- 172.21.15.31:0/3678699488 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55898edb0d80 con 0x55898f03a380 2023-07-29T16:35:01.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.222+0000 7f76a0f9c700 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f03b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:35:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.223+0000 7f76997fa700 1 -- 172.21.15.31:0/1390422960 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f76900cce40 con 0x7f769c071e80 2023-07-29T16:35:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.223+0000 7f76a0f9c700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f76840c28f0 0x7f76840c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.223+0000 7f769b7fe700 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f03a820 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f76900bc5c0 tx=0x7f769006c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed19cb2abc6f3f20 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.223+0000 7f76827fc700 1 -- 172.21.15.31:0/3678699488 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769009a340 con 0x55898f03a380 2023-07-29T16:35:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.224+0000 7f76827fc700 1 -- 172.21.15.31:0/3678699488 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f769009a0b0 con 0x55898f03a380 2023-07-29T16:35:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.224+0000 7f76827fc700 1 -- 172.21.15.31:0/3678699488 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76900a5750 con 0x55898f03a380 2023-07-29T16:35:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.224+0000 7f76a0f9c700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f76840c28f0 0x7f76840c4da0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7694009470 tx=0x7f7694009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.224+0000 7f76b6327b80 1 -- 172.21.15.31:0/3678699488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 msgr2=0x55898f03a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.224+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f03a820 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f76900bc5c0 tx=0x7f769006c980 comp rx=0 tx=0).stop 2023-07-29T16:35:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 -- 172.21.15.31:0/3678699488 shutdown_connections 2023-07-29T16:35:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f03a820 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f03b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3678699488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f039e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 -- 172.21.15.31:0/3678699488 >> 172.21.15.31:0/3678699488 conn(0x55898f026fd0 msgr2=0x55898f02c310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 -- 172.21.15.31:0/3678699488 shutdown_connections 2023-07-29T16:35:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 -- 172.21.15.31:0/3678699488 wait complete. 2023-07-29T16:35:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.225+0000 7f76b6327b80 1 -- start start 2023-07-29T16:35:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.226+0000 7f76b6327b80 1 Processor -- start 2023-07-29T16:35:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.226+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f0434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.226+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f043a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.226+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03b770 0x55898f04a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f76b6327b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898ef1f4b0 con 0x55898f03a380 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f76b6327b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898ef1f150 con 0x55898ee8d6b0 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f76b6327b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898ee068c0 con 0x55898f03b770 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f76b6327b80 10 client.0 _ll_register_callbacks cb 0x55898ef285c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f043a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f043a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55532/0 (socket says 172.21.15.31:55532) 2023-07-29T16:35:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f769b7fe700 1 -- 172.21.15.31:0/1860842859 learned_addr learned my addr 172.21.15.31:0/1860842859 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f76a0f9c700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03b770 0x55898f04a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f769bfff700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f0434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.227+0000 7f769bfff700 1 -- 172.21.15.31:0/1860842859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03b770 msgr2=0x55898f04a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f769bfff700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03b770 0x55898f04a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f769bfff700 1 -- 172.21.15.31:0/1860842859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 msgr2=0x55898f043a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f769bfff700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f043a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f769bfff700 1 -- 172.21.15.31:0/1860842859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c0731e0 con 0x55898ee8d6b0 2023-07-29T16:35:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f043a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:35:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f769bfff700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f0434c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f768c00d160 tx=0x7f768c00d470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7e517ab0358b5f6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f767b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:35:01.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f767b7fe700 1 -- 172.21.15.31:0/1860842859 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00dd00 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f767b7fe700 1 -- 172.21.15.31:0/1860842859 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f768c014430 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.228+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55898efadcb0 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f767b7fe700 1 -- 172.21.15.31:0/1860842859 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f768c00de60 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 10 client.30359 fetch_fsmap learned FSMap version 42 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55898ed99c00 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f767b7fe700 1 -- 172.21.15.31:0/1860842859 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55898efadcb0 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f767b7fe700 1 -- 172.21.15.31:0/1860842859 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f768c0127b0 con 0x55898ee8d6b0 2023-07-29T16:35:01.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 10 client.30359 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 10 client.30359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 20 client.30359 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f04f3b0 0x55898f0517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 4 client.30359 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55898f000d10 con 0x55898f04f3b0 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.229+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f04f3b0 0x55898f0517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.230+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f04f3b0 0x55898f0517c0 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d04ca95570d3195e server_cookie=6ebb49d658edd555 in_seq=0 out_seq=0 2023-07-29T16:35:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.230+0000 7f767b7fe700 10 client.30359 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:01.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f767b7fe700 1 -- 172.21.15.31:0/1860842859 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55898f000d10 con 0x55898f04f3b0 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f767b7fe700 10 client.30359 handle_command_reply: tid=0 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 client.30359 shutdown 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 msgr2=0x55898f0434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f0434c0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f768c00d160 tx=0x7f768c00d470 comp rx=0 tx=0).stop 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 shutdown_connections 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f04f3b0 0x55898f0517c0 crc :-1 s=READY pgs=463 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03b770 0x55898f04a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f0434c0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1860842859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f043a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 >> 172.21.15.31:0/1860842859 conn(0x55898f026fd0 msgr2=0x55898f02ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f767b7fe700 0 client.30359 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.231+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 shutdown_connections 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.232+0000 7f76b6327b80 1 -- 172.21.15.31:0/1860842859 wait complete. 2023-07-29T16:35:01.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.232+0000 7f76b6327b80 20 client.30359 trim_cache size 0 max 16384 2023-07-29T16:35:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 Processor -- start 2023-07-29T16:35:01.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 -- start start 2023-07-29T16:35:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 0x55898f043270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f04a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898f03b770 0x55898f0437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898f042e80 con 0x55898f03a380 2023-07-29T16:35:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898f03be70 con 0x55898f03b770 2023-07-29T16:35:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.253+0000 7f76b6327b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898f04ae30 con 0x55898ee8d6b0 2023-07-29T16:35:01.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 0x55898f043270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 0x55898f043270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45356/0 (socket says 172.21.15.31:45356) 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 -- 172.21.15.31:0/3784308183 learned_addr learned my addr 172.21.15.31:0/3784308183 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769b7fe700 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f04a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f76a0f9c700 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898f03b770 0x55898f0437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 -- 172.21.15.31:0/3784308183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898f03b770 msgr2=0x55898f0437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898f03b770 0x55898f0437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 -- 172.21.15.31:0/3784308183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 msgr2=0x55898f04a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f04a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 -- 172.21.15.31:0/3784308183 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55898edd71f0 con 0x55898ee8d6b0 2023-07-29T16:35:01.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.254+0000 7f769bfff700 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 0x55898f043270 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f768c006b30 tx=0x7f768c00d8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb464bfc9b373b6c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.255+0000 7f7682ffd700 1 -- 172.21.15.31:0/3784308183 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00a070 con 0x55898ee8d6b0 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.255+0000 7f7682ffd700 1 -- 172.21.15.31:0/3784308183 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f768c0128a0 con 0x55898ee8d6b0 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.255+0000 7f7682ffd700 1 -- 172.21.15.31:0/3784308183 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c01e8e0 con 0x55898ee8d6b0 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 -- 172.21.15.31:0/3784308183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 msgr2=0x55898f043270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 0x55898f043270 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f768c006b30 tx=0x7f768c00d8c0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 -- 172.21.15.31:0/3784308183 shutdown_connections 2023-07-29T16:35:01.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898f03b770 0x55898f0437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898ee8d6b0 0x55898f043270 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 --2- 172.21.15.31:0/3784308183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03a380 0x55898f04a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.256+0000 7f76b6327b80 1 -- 172.21.15.31:0/3784308183 >> 172.21.15.31:0/3784308183 conn(0x55898f026fd0 msgr2=0x55898f03e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.257+0000 7f76b6327b80 1 -- 172.21.15.31:0/3784308183 shutdown_connections 2023-07-29T16:35:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.257+0000 7f76b6327b80 1 -- 172.21.15.31:0/3784308183 wait complete. 2023-07-29T16:35:01.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.258+0000 7f76b6327b80 1 -- start start 2023-07-29T16:35:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.258+0000 7f76b6327b80 1 Processor -- start 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f046970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f046eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f049ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898f04a410 con 0x55898f03b770 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898f04a580 con 0x55898ee8d6b0 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55898f04a740 con 0x55898f03a380 2023-07-29T16:35:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f76b6327b80 10 client.0 _ll_register_callbacks cb 0x55898ef285c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f046eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f769bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f046970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f769b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f046eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45372/0 (socket says 172.21.15.31:45372) 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f769bfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f046970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34578/0 (socket says 172.21.15.31:34578) 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.259+0000 7f769b7fe700 1 -- 172.21.15.31:0/1361933703 learned_addr learned my addr 172.21.15.31:0/1361933703 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769b7fe700 1 -- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 msgr2=0x55898f046970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f046970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769b7fe700 1 -- 172.21.15.31:0/1361933703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 msgr2=0x55898f049ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f049ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769b7fe700 1 -- 172.21.15.31:0/1361933703 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55898ed99c00 con 0x55898f03a380 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769bfff700 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f046970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f769b7fe700 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f046eb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7690005910 tx=0x7f7690096810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9cc6c9850ad3ff3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f767b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f767b7fe700 1 -- 172.21.15.31:0/1361933703 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769006df30 con 0x55898f03a380 2023-07-29T16:35:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f767b7fe700 1 -- 172.21.15.31:0/1361933703 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f769009a570 con 0x55898f03a380 2023-07-29T16:35:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f769009a0b0 con 0x55898f03a380 2023-07-29T16:35:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.260+0000 7f767b7fe700 1 -- 172.21.15.31:0/1361933703 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f769009e5a0 con 0x55898f03a380 2023-07-29T16:35:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 10 client.30376 fetch_fsmap learned FSMap version 42 2023-07-29T16:35:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55898edaf250 con 0x55898f03a380 2023-07-29T16:35:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f767b7fe700 1 -- 172.21.15.31:0/1361933703 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f769009a0b0 con 0x55898f03a380 2023-07-29T16:35:01.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f767b7fe700 1 -- 172.21.15.31:0/1361933703 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 42) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f76900ca050 con 0x55898f03a380 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 10 client.30376 fetch_fsmap finished waiting for FSMap version 42 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 10 client.30376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 20 client.30376 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f033560 0x55898f035970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 4 client.30376 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.261+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55898ef7bce0 con 0x55898f033560 2023-07-29T16:35:01.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.262+0000 7f769bfff700 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f033560 0x55898f035970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.262+0000 7f769bfff700 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f033560 0x55898f035970 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78aba2c8b5ebad20 server_cookie=3029c19dca268660 in_seq=0 out_seq=0 2023-07-29T16:35:01.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.262+0000 7f767b7fe700 10 client.30376 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:01.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 189097 36.14 MB/sec execute 165 sec latency 466.419 ms 2023-07-29T16:35:01.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.648+0000 7f767b7fe700 1 -- 172.21.15.31:0/1361933703 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55898ef7bce0 con 0x55898f033560 2023-07-29T16:35:01.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.648+0000 7f767b7fe700 10 client.30376 handle_command_reply: tid=0 2023-07-29T16:35:01.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.648+0000 7f76b6327b80 1 client.30376 shutdown 2023-07-29T16:35:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 msgr2=0x55898f046eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f046eb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7690005910 tx=0x7f7690096810 comp rx=0 tx=0).stop 2023-07-29T16:35:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 shutdown_connections 2023-07-29T16:35:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55898f033560 0x55898f035970 crc :-1 s=READY pgs=464 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55898f03b770 0x55898f049ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55898ee8d6b0 0x55898f046970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 --2- 172.21.15.31:0/1361933703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55898f03a380 0x55898f046eb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 >> 172.21.15.31:0/1361933703 conn(0x55898f026fd0 msgr2=0x55898f043270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.649+0000 7f767b7fe700 0 client.30376 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.650+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 shutdown_connections 2023-07-29T16:35:01.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.650+0000 7f76b6327b80 1 -- 172.21.15.31:0/1361933703 wait complete. 2023-07-29T16:35:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.650+0000 7f76b6327b80 20 client.30376 trim_cache size 0 max 16384 2023-07-29T16:35:01.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:35:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:35:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:35:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:35:01.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f76840c28f0 msgr2=0x7f76840c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f76840c28f0 0x7f76840c4da0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7694009470 tx=0x7f7694009040 comp rx=0 tx=0).stop 2023-07-29T16:35:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 msgr2=0x7f769c1a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 0x7f769c1a1110 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f769000b680 tx=0x7f769000f460 comp rx=0 tx=0).stop 2023-07-29T16:35:01.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 shutdown_connections 2023-07-29T16:35:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f76840c28f0 0x7f76840c4da0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f769c108260 0x7f769c1a1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f769c0714d0 0x7f769c1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 --2- 172.21.15.31:0/1390422960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f769c071e80 0x7f769c1a1110 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.653+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 >> 172.21.15.31:0/1390422960 conn(0x7f769c06d290 msgr2=0x7f769c10f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.654+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 shutdown_connections 2023-07-29T16:35:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:01.654+0000 7f76a29ff700 1 -- 172.21.15.31:0/1390422960 wait complete. 2023-07-29T16:35:02.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 189568 36.09 MB/sec execute 166 sec latency 207.843 ms 2023-07-29T16:35:02.324 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:35:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: ignoring --setuser ceph since I am not root 2023-07-29T16:35:02.324 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:35:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: ignoring --setgroup ceph since I am not root 2023-07-29T16:35:02.324 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:35:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: starting mds.c at 2023-07-29T16:35:03.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:02 smithi039 ceph-mon[128858]: pgmap v958: 97 pgs: 97 active+clean; 726 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.1 MiB/s wr, 502 op/s 2023-07-29T16:35:03.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:02 smithi039 ceph-mon[128858]: daemon mds.c restarted 2023-07-29T16:35:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:02 smithi039 ceph-mon[128858]: osdmap e93: 12 total, 12 up, 12 in 2023-07-29T16:35:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:02 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6836/1692916896,v1:172.21.15.156:6837/1692916896] up:boot 2023-07-29T16:35:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:02 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:35:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:02 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:35:03.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:02 smithi156 ceph-mon[129245]: pgmap v958: 97 pgs: 97 active+clean; 726 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.1 MiB/s wr, 502 op/s 2023-07-29T16:35:03.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:02 smithi156 ceph-mon[129245]: daemon mds.c restarted 2023-07-29T16:35:03.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:02 smithi156 ceph-mon[129245]: osdmap e93: 12 total, 12 up, 12 in 2023-07-29T16:35:03.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:02 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6836/1692916896,v1:172.21.15.156:6837/1692916896] up:boot 2023-07-29T16:35:03.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:02 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:35:03.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:02 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:35:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:02 smithi031 ceph-mon[95504]: pgmap v958: 97 pgs: 97 active+clean; 726 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.1 MiB/s wr, 502 op/s 2023-07-29T16:35:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:02 smithi031 ceph-mon[95504]: daemon mds.c restarted 2023-07-29T16:35:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:02 smithi031 ceph-mon[95504]: osdmap e93: 12 total, 12 up, 12 in 2023-07-29T16:35:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:02 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6836/1692916896,v1:172.21.15.156:6837/1692916896] up:boot 2023-07-29T16:35:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:02 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:35:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:02 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-29T16:35:03.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 190442 36.22 MB/sec execute 167 sec latency 51.413 ms 2023-07-29T16:35:04.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 190983 36.19 MB/sec execute 168 sec latency 365.037 ms 2023-07-29T16:35:05.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:04 smithi039 ceph-mon[128858]: pgmap v960: 97 pgs: 97 active+clean; 723 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.8 MiB/s wr, 673 op/s 2023-07-29T16:35:05.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:04 smithi156 ceph-mon[129245]: pgmap v960: 97 pgs: 97 active+clean; 723 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.8 MiB/s wr, 673 op/s 2023-07-29T16:35:05.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:04 smithi031 ceph-mon[95504]: pgmap v960: 97 pgs: 97 active+clean; 723 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 7.8 MiB/s wr, 673 op/s 2023-07-29T16:35:05.275 INFO:tasks.workunit.client.0.smithi031.stdout: 10 191600 36.16 MB/sec execute 169 sec latency 97.475 ms 2023-07-29T16:35:06.278 INFO:tasks.workunit.client.0.smithi031.stdout: 10 192100 36.11 MB/sec execute 170 sec latency 238.940 ms 2023-07-29T16:35:07.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:06 smithi039 ceph-mon[128858]: pgmap v961: 97 pgs: 97 active+clean; 717 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 9.2 MiB/s wr, 687 op/s 2023-07-29T16:35:07.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:06 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:35:07.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:06 smithi156 ceph-mon[129245]: pgmap v961: 97 pgs: 97 active+clean; 717 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 9.2 MiB/s wr, 687 op/s 2023-07-29T16:35:07.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:06 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:35:07.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:06 smithi031 ceph-mon[95504]: pgmap v961: 97 pgs: 97 active+clean; 717 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 9.2 MiB/s wr, 687 op/s 2023-07-29T16:35:07.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:06 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:35:07.278 INFO:tasks.workunit.client.0.smithi031.stdout: 10 192120 35.90 MB/sec execute 171 sec latency 950.975 ms 2023-07-29T16:35:08.278 INFO:tasks.workunit.client.0.smithi031.stdout: 10 192296 35.76 MB/sec execute 172 sec latency 742.251 ms 2023-07-29T16:35:08.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:08 smithi031 ceph-mon[95504]: pgmap v962: 97 pgs: 97 active+clean; 745 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 730 op/s 2023-07-29T16:35:08.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:08 smithi039 ceph-mon[128858]: pgmap v962: 97 pgs: 97 active+clean; 745 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 730 op/s 2023-07-29T16:35:08.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:08 smithi156 ceph-mon[129245]: pgmap v962: 97 pgs: 97 active+clean; 745 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 730 op/s 2023-07-29T16:35:09.279 INFO:tasks.workunit.client.0.smithi031.stdout: 10 192583 35.61 MB/sec execute 173 sec latency 194.139 ms 2023-07-29T16:35:10.279 INFO:tasks.workunit.client.0.smithi031.stdout: 10 193032 35.51 MB/sec execute 174 sec latency 97.090 ms 2023-07-29T16:35:11.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:10 smithi039 ceph-mon[128858]: pgmap v963: 97 pgs: 97 active+clean; 778 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 19 MiB/s wr, 943 op/s 2023-07-29T16:35:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:10 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:10 smithi156 ceph-mon[129245]: pgmap v963: 97 pgs: 97 active+clean; 778 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 19 MiB/s wr, 943 op/s 2023-07-29T16:35:11.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.078 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:10 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:10 smithi031 ceph-mon[95504]: pgmap v963: 97 pgs: 97 active+clean; 778 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 19 MiB/s wr, 943 op/s 2023-07-29T16:35:11.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:10 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:11.280 INFO:tasks.workunit.client.0.smithi031.stdout: 10 193818 35.54 MB/sec execute 175 sec latency 64.956 ms 2023-07-29T16:35:12.280 INFO:tasks.workunit.client.0.smithi031.stdout: 10 194750 35.68 MB/sec execute 176 sec latency 58.400 ms 2023-07-29T16:35:13.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:12 smithi039 ceph-mon[128858]: pgmap v964: 97 pgs: 97 active+clean; 770 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 22 MiB/s wr, 1.00k op/s 2023-07-29T16:35:13.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:12 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:13.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:12 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:13.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:12 smithi156 ceph-mon[129245]: pgmap v964: 97 pgs: 97 active+clean; 770 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 22 MiB/s wr, 1.00k op/s 2023-07-29T16:35:13.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:12 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:13.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:12 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:13.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:12 smithi031 ceph-mon[95504]: pgmap v964: 97 pgs: 97 active+clean; 770 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 22 MiB/s wr, 1.00k op/s 2023-07-29T16:35:13.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:12 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:13.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:12 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:13.280 INFO:tasks.workunit.client.0.smithi031.stdout: 10 195445 35.69 MB/sec execute 177 sec latency 60.569 ms 2023-07-29T16:35:14.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:35:14.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:35:14.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:14.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:35:14.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:35:14.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:14.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:35:14.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:35:14.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:35:14.280 INFO:tasks.workunit.client.0.smithi031.stdout: 10 195993 35.67 MB/sec execute 178 sec latency 81.333 ms 2023-07-29T16:35:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.882+0000 7f9be3fff700 1 -- 172.21.15.31:0/824357247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 msgr2=0x7f9bdc108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.882+0000 7f9be3fff700 1 --2- 172.21.15.31:0/824357247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 0x7f9bdc108670 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8009990 tx=0x7f9bd8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 -- 172.21.15.31:0/824357247 shutdown_connections 2023-07-29T16:35:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 --2- 172.21.15.31:0/824357247 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 --2- 172.21.15.31:0/824357247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 0x7f9bdc108670 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 --2- 172.21.15.31:0/824357247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc0714d0 0x7f9bdc071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 -- 172.21.15.31:0/824357247 >> 172.21.15.31:0/824357247 conn(0x7f9bdc06d290 msgr2=0x7f9bdc06f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 -- 172.21.15.31:0/824357247 shutdown_connections 2023-07-29T16:35:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.883+0000 7f9be3fff700 1 -- 172.21.15.31:0/824357247 wait complete. 2023-07-29T16:35:14.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 Processor -- start 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 -- start start 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bdc0714d0 0x7f9bdc1a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 0x7f9bdc1a62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc1a6930 con 0x7f9bdc108260 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc1a6aa0 con 0x7f9bdc071e80 2023-07-29T16:35:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.885+0000 7f9be3fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bdc1a6c60 con 0x7f9bdc0714d0 2023-07-29T16:35:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1a5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33532/0 (socket says 172.21.15.31:33532) 2023-07-29T16:35:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 -- 172.21.15.31:0/1004249825 learned_addr learned my addr 172.21.15.31:0/1004249825 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bdc0714d0 0x7f9bdc1a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be259c700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 0x7f9bdc1a62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 -- 172.21.15.31:0/1004249825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bdc0714d0 msgr2=0x7f9bdc1a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bdc0714d0 0x7f9bdc1a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 -- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 msgr2=0x7f9bdc1a62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 0x7f9bdc1a62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.886+0000 7f9be159a700 1 -- 172.21.15.31:0/1004249825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd80095f0 con 0x7f9bdc071e80 2023-07-29T16:35:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.887+0000 7f9be159a700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1a5450 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c9b0 tx=0x7f9bcc00cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.887+0000 7f9bd2ffd700 1 -- 172.21.15.31:0/1004249825 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc007b00 con 0x7f9bdc071e80 2023-07-29T16:35:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.887+0000 7f9bd2ffd700 1 -- 172.21.15.31:0/1004249825 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bcc007c60 con 0x7f9bdc071e80 2023-07-29T16:35:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.887+0000 7f9bd2ffd700 1 -- 172.21.15.31:0/1004249825 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc018660 con 0x7f9bdc071e80 2023-07-29T16:35:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.888+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bdc1a6f40 con 0x7f9bdc071e80 2023-07-29T16:35:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.888+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bdc199c40 con 0x7f9bdc071e80 2023-07-29T16:35:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.893+0000 7f9bf78f2b80 1 Processor -- start 2023-07-29T16:35:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.893+0000 7f9bf78f2b80 1 -- start start 2023-07-29T16:35:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.893+0000 7f9bd2ffd700 1 -- 172.21.15.31:0/1004249825 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9bcc018880 con 0x7f9bdc071e80 2023-07-29T16:35:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 0x559b4917b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4917c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b4917d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b490d39b0 con 0x559b4917bdf0 2023-07-29T16:35:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b490d1370 con 0x559b4917ccf0 2023-07-29T16:35:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b490ccc60 con 0x559b48fd76b0 2023-07-29T16:35:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 0x559b4917b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 0x559b4917b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:60974/0 (socket says 172.21.15.31:60974) 2023-07-29T16:35:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 -- 172.21.15.31:0/1758859738 learned_addr learned my addr 172.21.15.31:0/1758859738 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:14.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be159a700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b4917d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4917c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 -- 172.21.15.31:0/1758859738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 msgr2=0x559b4917d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b4917d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 -- 172.21.15.31:0/1758859738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 msgr2=0x559b4917c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.894+0000 7f9be259c700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4917c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9be259c700 1 -- 172.21.15.31:0/1758859738 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b48efad80 con 0x559b48fd76b0 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9be159a700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b4917d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4917c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:35:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9be259c700 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 0x559b4917b8b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009c30 tx=0x7f9bd4009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6cbb821161febd3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9bc7fff700 1 -- 172.21.15.31:0/1758859738 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4014410 con 0x559b48fd76b0 2023-07-29T16:35:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9bc7fff700 1 -- 172.21.15.31:0/1758859738 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bd400f050 con 0x559b48fd76b0 2023-07-29T16:35:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.895+0000 7f9bc7fff700 1 -- 172.21.15.31:0/1758859738 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4013770 con 0x559b48fd76b0 2023-07-29T16:35:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.896+0000 7f9bd2ffd700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bc80a0810 0x7f9bc80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.896+0000 7f9bd2ffd700 1 -- 172.21.15.31:0/1004249825 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9bcc0cbbc0 con 0x7f9bdc071e80 2023-07-29T16:35:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.896+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1758859738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 msgr2=0x559b4917b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.896+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 0x559b4917b8b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009c30 tx=0x7f9bd4009f40 comp rx=0 tx=0).stop 2023-07-29T16:35:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.897+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bc80a0810 0x7f9bc80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1758859738 shutdown_connections 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b4917d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b48fd76b0 0x559b4917b8b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1758859738 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4917c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1758859738 >> 172.21.15.31:0/1758859738 conn(0x559b49170fd0 msgr2=0x559b49176300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bc80a0810 0x7f9bc80a2cc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800b5c0 tx=0x7f9bd801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1758859738 shutdown_connections 2023-07-29T16:35:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1758859738 wait complete. 2023-07-29T16:35:14.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.898+0000 7f9bf78f2b80 1 -- start start 2023-07-29T16:35:14.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.899+0000 7f9bf78f2b80 1 Processor -- start 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b49184e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b49185370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b4918c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b490694b0 con 0x559b4917bdf0 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b49069150 con 0x559b48fd76b0 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b48f508c0 con 0x559b4917ccf0 2023-07-29T16:35:14.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9bf78f2b80 10 client.0 _ll_register_callbacks cb 0x559b490725c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9be159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b49185370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9be159a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b49185370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44232/0 (socket says 172.21.15.31:44232) 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.900+0000 7f9be159a700 1 -- 172.21.15.31:0/3159610965 learned_addr learned my addr 172.21.15.31:0/3159610965 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.901+0000 7f9be159a700 1 -- 172.21.15.31:0/3159610965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 msgr2=0x559b4918c3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.901+0000 7f9be159a700 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b4918c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.901+0000 7f9be159a700 1 -- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 msgr2=0x559b49184e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:35:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.901+0000 7f9be159a700 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b49184e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.901+0000 7f9be159a700 1 -- 172.21.15.31:0/3159610965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd4005e70 con 0x559b4917bdf0 2023-07-29T16:35:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9be159a700 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b49185370 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0991f0 tx=0x7f9bcc007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b585e3f5fc7839f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bc57fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:35:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bc57fa700 1 -- 172.21.15.31:0/3159610965 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc098950 con 0x559b4917bdf0 2023-07-29T16:35:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bc57fa700 1 -- 172.21.15.31:0/3159610965 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9bcc099260 con 0x559b4917bdf0 2023-07-29T16:35:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b490f7cb0 con 0x559b4917bdf0 2023-07-29T16:35:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bc57fa700 1 -- 172.21.15.31:0/3159610965 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bcc09d290 con 0x559b4917bdf0 2023-07-29T16:35:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bf78f2b80 10 client.20136 fetch_fsmap learned FSMap version 43 2023-07-29T16:35:14.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.902+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559b48ee3c00 con 0x559b4917bdf0 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bc57fa700 1 -- 172.21.15.31:0/3159610965 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bcc0b5030 con 0x559b4917bdf0 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bc57fa700 1 -- 172.21.15.31:0/3159610965 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9bcc0950e0 con 0x559b4917bdf0 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bf78f2b80 10 client.20136 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bf78f2b80 10 client.20136 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bf78f2b80 20 client.20136 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b49190b30 0x559b49192f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bf78f2b80 4 client.20136 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:35:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b4914ad10 con 0x559b49190b30 2023-07-29T16:35:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.903+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b49190b30 0x559b49192f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.904+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b49190b30 0x559b49192f40 crc :-1 s=READY pgs=465 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8dde1a2daa86a0 server_cookie=8e8961b680b90001 in_seq=0 out_seq=0 2023-07-29T16:35:14.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.904+0000 7f9bc57fa700 10 client.20136 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bc57fa700 1 -- 172.21.15.31:0/3159610965 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559b4914ad10 con 0x559b49190b30 2023-07-29T16:35:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bc57fa700 10 client.20136 handle_command_reply: tid=0 2023-07-29T16:35:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bf78f2b80 1 client.20136 shutdown 2023-07-29T16:35:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 msgr2=0x559b49185370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b49185370 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0991f0 tx=0x7f9bcc007e80 comp rx=0 tx=0).stop 2023-07-29T16:35:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 shutdown_connections 2023-07-29T16:35:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.917+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b49190b30 0x559b49192f40 crc :-1 s=READY pgs=465 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b4918c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b49184e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/3159610965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b49185370 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bc57fa700 0 client.20136 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 >> 172.21.15.31:0/3159610965 conn(0x559b49170fd0 msgr2=0x559b49175a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 shutdown_connections 2023-07-29T16:35:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/3159610965 wait complete. 2023-07-29T16:35:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.918+0000 7f9bf78f2b80 20 client.20136 trim_cache size 0 max 16384 2023-07-29T16:35:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 Processor -- start 2023-07-29T16:35:14.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 -- start start 2023-07-29T16:35:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b4917d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4918c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b49182f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4918e040 con 0x559b4917bdf0 2023-07-29T16:35:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b49184310 con 0x559b48fd76b0 2023-07-29T16:35:14.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.938+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b4917d600 con 0x559b4917ccf0 2023-07-29T16:35:14.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b49182f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b49182f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:60990/0 (socket says 172.21.15.31:60990) 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be259c700 1 -- 172.21.15.31:0/2705418125 learned_addr learned my addr 172.21.15.31:0/2705418125 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be159a700 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4918c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b4917d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be259c700 1 -- 172.21.15.31:0/2705418125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 msgr2=0x559b4917d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.939+0000 7f9be259c700 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b4917d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9be259c700 1 -- 172.21.15.31:0/2705418125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 msgr2=0x559b4918c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:14.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9be259c700 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4918c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9be259c700 1 -- 172.21.15.31:0/2705418125 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b48f211f0 con 0x559b4917ccf0 2023-07-29T16:35:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9be259c700 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b49182f70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4000f80 tx=0x7f9bd4006080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18950c28e04ed9b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9bd0ff9700 1 -- 172.21.15.31:0/2705418125 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4014610 con 0x559b4917ccf0 2023-07-29T16:35:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9bd0ff9700 1 -- 172.21.15.31:0/2705418125 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bd4014c30 con 0x559b4917ccf0 2023-07-29T16:35:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.940+0000 7f9bd0ff9700 1 -- 172.21.15.31:0/2705418125 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd401ec50 con 0x559b4917ccf0 2023-07-29T16:35:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/2705418125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 msgr2=0x559b49182f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b49182f70 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4000f80 tx=0x7f9bd4006080 comp rx=0 tx=0).stop 2023-07-29T16:35:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/2705418125 shutdown_connections 2023-07-29T16:35:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917ccf0 0x559b49182f70 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b48fd76b0 0x559b4917d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/2705418125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b4917bdf0 0x559b4918c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/2705418125 >> 172.21.15.31:0/2705418125 conn(0x559b49170fd0 msgr2=0x559b4917fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/2705418125 shutdown_connections 2023-07-29T16:35:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.942+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/2705418125 wait complete. 2023-07-29T16:35:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.943+0000 7f9bf78f2b80 1 -- start start 2023-07-29T16:35:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.943+0000 7f9bf78f2b80 1 Processor -- start 2023-07-29T16:35:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b48fd76b0 0x559b4918c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917bdf0 0x559b49188c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b491891b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b49189800 con 0x559b48fd76b0 2023-07-29T16:35:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b491e26f0 con 0x559b4917ccf0 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b491e2830 con 0x559b4917bdf0 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9bf78f2b80 10 client.0 _ll_register_callbacks cb 0x559b490725c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9be259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b491891b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9be259c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b491891b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33566/0 (socket says 172.21.15.31:33566) 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.944+0000 7f9be259c700 1 -- 172.21.15.31:0/1375723333 learned_addr learned my addr 172.21.15.31:0/1375723333 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be159a700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917bdf0 0x559b49188c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b48fd76b0 0x559b4918c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be259c700 1 -- 172.21.15.31:0/1375723333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917bdf0 msgr2=0x559b49188c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be259c700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917bdf0 0x559b49188c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be259c700 1 -- 172.21.15.31:0/1375723333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b48fd76b0 msgr2=0x559b4918c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be259c700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b48fd76b0 0x559b4918c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be259c700 1 -- 172.21.15.31:0/1375723333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b48efad80 con 0x559b4917ccf0 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9be259c700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b491891b0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9bd401efb0 tx=0x7f9bd401e460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f927c30e5e529f66 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:14.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.945+0000 7f9bc57fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:35:14.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.946+0000 7f9bc57fa700 1 -- 172.21.15.31:0/1375723333 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4014820 con 0x559b4917ccf0 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.946+0000 7f9bc57fa700 1 -- 172.21.15.31:0/1375723333 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9bd4014980 con 0x559b4917ccf0 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.946+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9bd400f050 con 0x559b4917ccf0 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.946+0000 7f9bc57fa700 1 -- 172.21.15.31:0/1375723333 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bd4032440 con 0x559b4917ccf0 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.946+0000 7f9bf78f2b80 10 client.30377 fetch_fsmap learned FSMap version 43 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.946+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559b48ef9250 con 0x559b4917ccf0 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bc57fa700 1 -- 172.21.15.31:0/1375723333 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9bd400f050 con 0x559b4917ccf0 2023-07-29T16:35:14.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bc57fa700 1 -- 172.21.15.31:0/1375723333 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9bd4026030 con 0x559b4917ccf0 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bf78f2b80 10 client.30377 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bf78f2b80 10 client.30377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bf78f2b80 20 client.30377 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b491e3f70 0x559b491e6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bf78f2b80 4 client.30377 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559b490c5ce0 con 0x559b491e3f70 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.947+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b491e3f70 0x559b491e6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.948+0000 7f9be1d9b700 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b491e3f70 0x559b491e6380 crc :-1 s=READY pgs=466 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc686929a40256b7 server_cookie=7f29ef4c37951c4c in_seq=0 out_seq=0 2023-07-29T16:35:14.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:14.948+0000 7f9bc57fa700 10 client.30377 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:14 smithi039 ceph-mon[128858]: pgmap v965: 97 pgs: 97 active+clean; 711 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 20 MiB/s wr, 1.07k op/s 2023-07-29T16:35:15.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:14 smithi156 ceph-mon[129245]: pgmap v965: 97 pgs: 97 active+clean; 711 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 20 MiB/s wr, 1.07k op/s 2023-07-29T16:35:15.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:14 smithi031 ceph-mon[95504]: pgmap v965: 97 pgs: 97 active+clean; 711 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 20 MiB/s wr, 1.07k op/s 2023-07-29T16:35:15.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 196392 35.59 MB/sec execute 179 sec latency 284.065 ms 2023-07-29T16:35:15.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.364+0000 7f9bc57fa700 1 -- 172.21.15.31:0/1375723333 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559b490c5ce0 con 0x559b491e3f70 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.364+0000 7f9bc57fa700 10 client.30377 handle_command_reply: tid=0 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.364+0000 7f9bf78f2b80 1 client.30377 shutdown 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.364+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 msgr2=0x559b491891b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.364+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b491891b0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9bd401efb0 tx=0x7f9bd401e460 comp rx=0 tx=0).stop 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 shutdown_connections 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559b491e3f70 0x559b491e6380 crc :-1 s=READY pgs=466 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559b4917ccf0 0x559b491891b0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559b48fd76b0 0x559b4918c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 --2- 172.21.15.31:0/1375723333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559b4917bdf0 0x559b49188c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 >> 172.21.15.31:0/1375723333 conn(0x559b49170fd0 msgr2=0x559b49182f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:15.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bc57fa700 0 client.30377 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:15.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.365+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 shutdown_connections 2023-07-29T16:35:15.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.366+0000 7f9bf78f2b80 1 -- 172.21.15.31:0/1375723333 wait complete. 2023-07-29T16:35:15.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.366+0000 7f9bf78f2b80 20 client.30377 trim_cache size 0 max 16384 2023-07-29T16:35:15.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:35:15.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:35:15.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:35:15.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:35:15.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.368+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bc80a0810 msgr2=0x7f9bc80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:15.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.368+0000 7f9be3fff700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bc80a0810 0x7f9bc80a2cc0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800b5c0 tx=0x7f9bd801a040 comp rx=0 tx=0).stop 2023-07-29T16:35:15.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.368+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 msgr2=0x7f9bdc1a5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:15.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.368+0000 7f9be3fff700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1a5450 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c9b0 tx=0x7f9bcc00cd70 comp rx=0 tx=0).stop 2023-07-29T16:35:15.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.369+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 shutdown_connections 2023-07-29T16:35:15.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.369+0000 7f9be3fff700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f9bc80a0810 0x7f9bc80a2cc0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.369+0000 7f9be3fff700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9bdc108260 0x7f9bdc1a62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.369+0000 7f9be3fff700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9bdc0714d0 0x7f9bdc1a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:15.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.369+0000 7f9be3fff700 1 --2- 172.21.15.31:0/1004249825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9bdc071e80 0x7f9bdc1a5450 secure :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc00c9b0 tx=0x7f9bcc00cd70 comp rx=0 tx=0).stop 2023-07-29T16:35:15.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.369+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 >> 172.21.15.31:0/1004249825 conn(0x7f9bdc06d290 msgr2=0x7f9bdc10f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:15.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.370+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 shutdown_connections 2023-07-29T16:35:15.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:15.370+0000 7f9be3fff700 1 -- 172.21.15.31:0/1004249825 wait complete. 2023-07-29T16:35:16.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 197043 35.53 MB/sec execute 180 sec latency 240.838 ms 2023-07-29T16:35:17.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:16 smithi039 ceph-mon[128858]: pgmap v966: 97 pgs: 97 active+clean; 710 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 16 MiB/s wr, 871 op/s 2023-07-29T16:35:17.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:16 smithi156 ceph-mon[129245]: pgmap v966: 97 pgs: 97 active+clean; 710 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 16 MiB/s wr, 871 op/s 2023-07-29T16:35:17.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:16 smithi031 ceph-mon[95504]: pgmap v966: 97 pgs: 97 active+clean; 710 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 16 MiB/s wr, 871 op/s 2023-07-29T16:35:17.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 197935 35.56 MB/sec execute 181 sec latency 65.397 ms 2023-07-29T16:35:18.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 198911 35.75 MB/sec execute 182 sec latency 45.726 ms 2023-07-29T16:35:18.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:18 smithi031 ceph-mon[95504]: pgmap v967: 97 pgs: 97 active+clean; 705 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 786 op/s 2023-07-29T16:35:18.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:18 smithi039 ceph-mon[128858]: pgmap v967: 97 pgs: 97 active+clean; 705 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 786 op/s 2023-07-29T16:35:18.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:18 smithi156 ceph-mon[129245]: pgmap v967: 97 pgs: 97 active+clean; 705 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 14 MiB/s wr, 786 op/s 2023-07-29T16:35:19.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 199485 35.74 MB/sec execute 183 sec latency 77.027 ms 2023-07-29T16:35:20.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 200048 35.66 MB/sec execute 184 sec latency 86.555 ms 2023-07-29T16:35:21.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:20 smithi039 ceph-mon[128858]: pgmap v968: 97 pgs: 97 active+clean; 700 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 13 MiB/s wr, 826 op/s 2023-07-29T16:35:21.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:20 smithi156 ceph-mon[129245]: pgmap v968: 97 pgs: 97 active+clean; 700 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 13 MiB/s wr, 826 op/s 2023-07-29T16:35:21.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:20 smithi031 ceph-mon[95504]: pgmap v968: 97 pgs: 97 active+clean; 700 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 13 MiB/s wr, 826 op/s 2023-07-29T16:35:21.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 200803 35.67 MB/sec execute 185 sec latency 103.120 ms 2023-07-29T16:35:22.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 201308 35.64 MB/sec execute 186 sec latency 390.861 ms 2023-07-29T16:35:23.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:22 smithi039 ceph-mon[128858]: pgmap v969: 97 pgs: 97 active+clean; 703 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 6.5 MiB/s wr, 683 op/s 2023-07-29T16:35:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:22 smithi156 ceph-mon[129245]: pgmap v969: 97 pgs: 97 active+clean; 703 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 6.5 MiB/s wr, 683 op/s 2023-07-29T16:35:23.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:22 smithi031 ceph-mon[95504]: pgmap v969: 97 pgs: 97 active+clean; 703 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 6.5 MiB/s wr, 683 op/s 2023-07-29T16:35:23.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 201831 35.62 MB/sec execute 187 sec latency 419.226 ms 2023-07-29T16:35:24.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 202424 35.58 MB/sec execute 188 sec latency 108.479 ms 2023-07-29T16:35:25.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:24 smithi039 ceph-mon[128858]: pgmap v970: 97 pgs: 97 active+clean; 709 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 706 op/s 2023-07-29T16:35:25.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:24 smithi156 ceph-mon[129245]: pgmap v970: 97 pgs: 97 active+clean; 709 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 706 op/s 2023-07-29T16:35:25.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:24 smithi031 ceph-mon[95504]: pgmap v970: 97 pgs: 97 active+clean; 709 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.5 MiB/s wr, 706 op/s 2023-07-29T16:35:25.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 203021 35.53 MB/sec execute 189 sec latency 305.625 ms 2023-07-29T16:35:26.282 INFO:tasks.workunit.client.0.smithi031.stdout: 10 203825 35.62 MB/sec execute 190 sec latency 39.521 ms 2023-07-29T16:35:27.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:26 smithi039 ceph-mon[128858]: pgmap v971: 97 pgs: 97 active+clean; 700 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.9 MiB/s wr, 554 op/s 2023-07-29T16:35:27.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:26 smithi156 ceph-mon[129245]: pgmap v971: 97 pgs: 97 active+clean; 700 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.9 MiB/s wr, 554 op/s 2023-07-29T16:35:27.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:26 smithi031 ceph-mon[95504]: pgmap v971: 97 pgs: 97 active+clean; 700 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.9 MiB/s wr, 554 op/s 2023-07-29T16:35:27.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 204418 35.62 MB/sec execute 191 sec latency 127.591 ms 2023-07-29T16:35:28.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 204963 35.57 MB/sec execute 192 sec latency 76.011 ms 2023-07-29T16:35:28.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:28 smithi031 ceph-mon[95504]: pgmap v972: 97 pgs: 97 active+clean; 688 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.5 MiB/s wr, 520 op/s 2023-07-29T16:35:28.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:28 smithi039 ceph-mon[128858]: pgmap v972: 97 pgs: 97 active+clean; 688 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.5 MiB/s wr, 520 op/s 2023-07-29T16:35:28.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:28 smithi156 ceph-mon[129245]: pgmap v972: 97 pgs: 97 active+clean; 688 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 3.5 MiB/s wr, 520 op/s 2023-07-29T16:35:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 -- 172.21.15.31:0/3043679166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5744103f70 msgr2=0x7f57441043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 --2- 172.21.15.31:0/3043679166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5744103f70 0x7f57441043f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f573c009e00 tx=0x7f573c00b330 comp rx=0 tx=0).stop 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 -- 172.21.15.31:0/3043679166 shutdown_connections 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 --2- 172.21.15.31:0/3043679166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5744103f70 0x7f57441043f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 --2- 172.21.15.31:0/3043679166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57440fec50 0x7f5744101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 --2- 172.21.15.31:0/3043679166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57441015b0 0x7f5744103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 -- 172.21.15.31:0/3043679166 >> 172.21.15.31:0/3043679166 conn(0x7f57440fa840 msgr2=0x7f57440fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.599+0000 7f5749dff700 1 -- 172.21.15.31:0/3043679166 shutdown_connections 2023-07-29T16:35:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.600+0000 7f5749dff700 1 -- 172.21.15.31:0/3043679166 wait complete. 2023-07-29T16:35:28.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.601+0000 7f5749dff700 1 Processor -- start 2023-07-29T16:35:28.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.601+0000 7f5749dff700 1 -- start start 2023-07-29T16:35:28.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5749dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57440fec50 0x7f574406f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5749dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57441015b0 0x7f574406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5749dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5744103f70 0x7f5744070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5749dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5744071230 con 0x7f57440fec50 2023-07-29T16:35:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5749dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57440713a0 con 0x7f5744103f70 2023-07-29T16:35:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5749dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5744071560 con 0x7f57441015b0 2023-07-29T16:35:28.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57441015b0 0x7f574406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57441015b0 0x7f574406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51478/0 (socket says 172.21.15.31:51478) 2023-07-29T16:35:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5742ffd700 1 -- 172.21.15.31:0/655552704 learned_addr learned my addr 172.21.15.31:0/655552704 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:28.606 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.602+0000 7f5743fff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5744103f70 0x7f5744070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f57437fe700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57440fec50 0x7f574406f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f5743fff700 1 -- 172.21.15.31:0/655552704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57441015b0 msgr2=0x7f574406fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f5743fff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57441015b0 0x7f574406fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f5743fff700 1 -- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57440fec50 msgr2=0x7f574406f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f5743fff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57440fec50 0x7f574406f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f5743fff700 1 -- 172.21.15.31:0/655552704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c005e70 con 0x7f5744103f70 2023-07-29T16:35:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.603+0000 7f5743fff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5744103f70 0x7f5744070bd0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f573c00d010 tx=0x7f573c00bbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:28.608 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.604+0000 7f5740ff9700 1 -- 172.21.15.31:0/655552704 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00e040 con 0x7f5744103f70 2023-07-29T16:35:28.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.604+0000 7f5740ff9700 1 -- 172.21.15.31:0/655552704 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f573c01dd50 con 0x7f5744103f70 2023-07-29T16:35:28.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.604+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57440717e0 con 0x7f5744103f70 2023-07-29T16:35:28.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.604+0000 7f5740ff9700 1 -- 172.21.15.31:0/655552704 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c01bd00 con 0x7f5744103f70 2023-07-29T16:35:28.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.604+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5744071ff0 con 0x7f5744103f70 2023-07-29T16:35:28.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.609+0000 7f5740ff9700 1 -- 172.21.15.31:0/655552704 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f573c01b450 con 0x7f5744103f70 2023-07-29T16:35:28.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.610+0000 7f575d817b80 1 Processor -- start 2023-07-29T16:35:28.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.612+0000 7f5740ff9700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57300a08e0 0x7f57300a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.612+0000 7f5740ff9700 1 -- 172.21.15.31:0/655552704 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f573c0cd270 con 0x7f5744103f70 2023-07-29T16:35:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.613+0000 7f575d817b80 1 -- start start 2023-07-29T16:35:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f5742ffd700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57300a08e0 0x7f57300a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f575d817b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604daea76b0 0x5604db04b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f575d817b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db04c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f575d817b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604db04ccf0 0x5604db04d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f575d817b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604dafa39b0 con 0x5604daea76b0 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f575d817b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604dafa1370 con 0x5604db04bdf0 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f575d817b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604daf9cc60 con 0x5604db04ccf0 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db04c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db04c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33212/0 (socket says 172.21.15.31:33212) 2023-07-29T16:35:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.614+0000 7f5742ffd700 1 -- 172.21.15.31:0/236879176 learned_addr learned my addr 172.21.15.31:0/236879176 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.615+0000 7f5742ffd700 1 -- 172.21.15.31:0/236879176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604db04ccf0 msgr2=0x5604db04d160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:35:28.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.615+0000 7f5742ffd700 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604db04ccf0 0x5604db04d160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.615+0000 7f5742ffd700 1 -- 172.21.15.31:0/236879176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604daea76b0 msgr2=0x5604db04b8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.615+0000 7f5742ffd700 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604daea76b0 0x5604db04b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.615+0000 7f5742ffd700 1 -- 172.21.15.31:0/236879176 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604dadcad80 con 0x5604db04bdf0 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.615+0000 7f5742ffd700 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db04c7b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f572c013b40 tx=0x7f572c013e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d070cadff881a51 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.616+0000 7f5739ffb700 1 -- 172.21.15.31:0/236879176 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c00fe30 con 0x5604db04bdf0 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.616+0000 7f5742ffd700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57300a08e0 0x7f57300a2d90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f572c00f640 tx=0x7f572c01e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:28.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.616+0000 7f5739ffb700 1 -- 172.21.15.31:0/236879176 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f572c013630 con 0x5604db04bdf0 2023-07-29T16:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.616+0000 7f5739ffb700 1 -- 172.21.15.31:0/236879176 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0049b0 con 0x5604db04bdf0 2023-07-29T16:35:28.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.617+0000 7f575d817b80 1 -- 172.21.15.31:0/236879176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 msgr2=0x5604db04c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.617+0000 7f575d817b80 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db04c7b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f572c013b40 tx=0x7f572c013e50 comp rx=0 tx=0).stop 2023-07-29T16:35:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.617+0000 7f575d817b80 1 -- 172.21.15.31:0/236879176 shutdown_connections 2023-07-29T16:35:28.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.617+0000 7f575d817b80 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604db04ccf0 0x5604db04d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.617+0000 7f575d817b80 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604daea76b0 0x5604db04b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.618+0000 7f575d817b80 1 --2- 172.21.15.31:0/236879176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db04c7b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.618+0000 7f575d817b80 1 -- 172.21.15.31:0/236879176 >> 172.21.15.31:0/236879176 conn(0x5604db040fd0 msgr2=0x5604db046300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:28.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.618+0000 7f575d817b80 1 -- 172.21.15.31:0/236879176 shutdown_connections 2023-07-29T16:35:28.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.618+0000 7f575d817b80 1 -- 172.21.15.31:0/236879176 wait complete. 2023-07-29T16:35:28.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.619+0000 7f575d817b80 1 -- start start 2023-07-29T16:35:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 Processor -- start 2023-07-29T16:35:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db054ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db055410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db05c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604daf394b0 con 0x5604db04ccf0 2023-07-29T16:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604daf39150 con 0x5604db04bdf0 2023-07-29T16:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604dae208c0 con 0x5604daea76b0 2023-07-29T16:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.620+0000 7f575d817b80 10 client.0 _ll_register_callbacks cb 0x5604daf425c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:28.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db055410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db055410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33228/0 (socket says 172.21.15.31:33228) 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 -- 172.21.15.31:0/284448265 learned_addr learned my addr 172.21.15.31:0/284448265 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5743fff700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db05c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 -- 172.21.15.31:0/284448265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 msgr2=0x5604db054ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f57437fe700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db054ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db054ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 -- 172.21.15.31:0/284448265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 msgr2=0x5604db05c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db05c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.621+0000 7f5742ffd700 1 -- 172.21.15.31:0/284448265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c01b040 con 0x5604db04bdf0 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.622+0000 7f5742ffd700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db055410 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f572c009a40 tx=0x7f572c009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cece4f95da6027fb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.622+0000 7f5722ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.622+0000 7f5722ffd700 1 -- 172.21.15.31:0/284448265 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0133a0 con 0x5604db04bdf0 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.622+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604dafc7cb0 con 0x5604db04bdf0 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.622+0000 7f5722ffd700 1 -- 172.21.15.31:0/284448265 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f572c0046d0 con 0x5604db04bdf0 2023-07-29T16:35:28.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.623+0000 7f5722ffd700 1 -- 172.21.15.31:0/284448265 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f572c004890 con 0x5604db04bdf0 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.623+0000 7f575d817b80 10 client.30389 fetch_fsmap learned FSMap version 43 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.623+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5604dadb3c00 con 0x5604db04bdf0 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.623+0000 7f5722ffd700 1 -- 172.21.15.31:0/284448265 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f572c03e080 con 0x5604db04bdf0 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f5722ffd700 1 -- 172.21.15.31:0/284448265 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f572c026070 con 0x5604db04bdf0 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f575d817b80 10 client.30389 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f575d817b80 10 client.30389 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:28.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f575d817b80 20 client.30389 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f575d817b80 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db060ce0 0x5604db0630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f575d817b80 4 client.30389 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604db01ad10 con 0x5604db060ce0 2023-07-29T16:35:28.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.624+0000 7f57437fe700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db060ce0 0x5604db0630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.625+0000 7f57437fe700 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db060ce0 0x5604db0630c0 crc :-1 s=READY pgs=467 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b027e8335764a48 server_cookie=8e3db1eaafe3d16 in_seq=0 out_seq=0 2023-07-29T16:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.625+0000 7f5722ffd700 10 client.30389 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.628+0000 7f5722ffd700 1 -- 172.21.15.31:0/284448265 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5604db01ad10 con 0x5604db060ce0 2023-07-29T16:35:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.628+0000 7f5722ffd700 10 client.30389 handle_command_reply: tid=0 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.628+0000 7f575d817b80 1 client.30389 shutdown 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 msgr2=0x5604db055410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db055410 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f572c009a40 tx=0x7f572c009520 comp rx=0 tx=0).stop 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 shutdown_connections 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db060ce0 0x5604db0630c0 crc :-1 s=READY pgs=467 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db05c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db054ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 --2- 172.21.15.31:0/284448265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db055410 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 >> 172.21.15.31:0/284448265 conn(0x5604db040fd0 msgr2=0x5604db045a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f5722ffd700 0 client.30389 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 shutdown_connections 2023-07-29T16:35:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.629+0000 7f575d817b80 1 -- 172.21.15.31:0/284448265 wait complete. 2023-07-29T16:35:28.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.630+0000 7f575d817b80 20 client.30389 trim_cache size 0 max 16384 2023-07-29T16:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.649+0000 7f575d817b80 1 Processor -- start 2023-07-29T16:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.649+0000 7f575d817b80 1 -- start start 2023-07-29T16:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f575d817b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db04fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f575d817b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04bdf0 0x5604db050100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f575d817b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04ccf0 0x5604db05c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f575d817b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604db055d70 con 0x5604db04bdf0 2023-07-29T16:35:28.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f575d817b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604db054320 con 0x5604db04ccf0 2023-07-29T16:35:28.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f575d817b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604db047b30 con 0x5604daea76b0 2023-07-29T16:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db04fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db04fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51504/0 (socket says 172.21.15.31:51504) 2023-07-29T16:35:28.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 -- 172.21.15.31:0/2983952723 learned_addr learned my addr 172.21.15.31:0/2983952723 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f5742ffd700 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04bdf0 0x5604db050100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f5743fff700 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04ccf0 0x5604db05c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 -- 172.21.15.31:0/2983952723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04ccf0 msgr2=0x5604db05c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04ccf0 0x5604db05c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 -- 172.21.15.31:0/2983952723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04bdf0 msgr2=0x5604db050100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04bdf0 0x5604db050100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.650+0000 7f57437fe700 1 -- 172.21.15.31:0/2983952723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604dadf11f0 con 0x5604daea76b0 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.651+0000 7f57437fe700 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db04fbc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5734006fd0 tx=0x7f5734017c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d8bf6c6e9e52a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.651+0000 7f573a7fc700 1 -- 172.21.15.31:0/2983952723 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734017360 con 0x5604daea76b0 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.651+0000 7f573a7fc700 1 -- 172.21.15.31:0/2983952723 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5734009da0 con 0x5604daea76b0 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.651+0000 7f573a7fc700 1 -- 172.21.15.31:0/2983952723 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5734020dd0 con 0x5604daea76b0 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 -- 172.21.15.31:0/2983952723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 msgr2=0x5604db04fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db04fbc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5734006fd0 tx=0x7f5734017c30 comp rx=0 tx=0).stop 2023-07-29T16:35:28.656 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 -- 172.21.15.31:0/2983952723 shutdown_connections 2023-07-29T16:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04ccf0 0x5604db05c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db04fbc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 --2- 172.21.15.31:0/2983952723 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04bdf0 0x5604db050100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 -- 172.21.15.31:0/2983952723 >> 172.21.15.31:0/2983952723 conn(0x5604db040fd0 msgr2=0x5604db053c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 -- 172.21.15.31:0/2983952723 shutdown_connections 2023-07-29T16:35:28.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 -- 172.21.15.31:0/2983952723 wait complete. 2023-07-29T16:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.652+0000 7f575d817b80 1 -- start start 2023-07-29T16:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 Processor -- start 2023-07-29T16:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db05bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db05c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db058c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604db059380 con 0x5604db04ccf0 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604db0594f0 con 0x5604db04bdf0 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604db0596b0 con 0x5604daea76b0 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.653+0000 7f575d817b80 10 client.0 _ll_register_callbacks cb 0x5604daf425c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db05c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db05c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:33246/0 (socket says 172.21.15.31:33246) 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 -- 172.21.15.31:0/736762789 learned_addr learned my addr 172.21.15.31:0/736762789 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5743fff700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db058c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f57437fe700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db05bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 -- 172.21.15.31:0/736762789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 msgr2=0x5604db05bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db05bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 -- 172.21.15.31:0/736762789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 msgr2=0x5604db058c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db058c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:28.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.654+0000 7f5742ffd700 1 -- 172.21.15.31:0/736762789 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573400e040 con 0x5604db04bdf0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.655+0000 7f5742ffd700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db05c430 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f572c005e30 tx=0x7f572c010cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f38d01438e71584 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.655+0000 7f5722ffd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.655+0000 7f5722ffd700 1 -- 172.21.15.31:0/736762789 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572c0046d0 con 0x5604db04bdf0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.655+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f572c013630 con 0x5604db04bdf0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.655+0000 7f5722ffd700 1 -- 172.21.15.31:0/736762789 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f572c004830 con 0x5604db04bdf0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.656+0000 7f5722ffd700 1 -- 172.21.15.31:0/736762789 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f572c0106b0 con 0x5604db04bdf0 2023-07-29T16:35:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.656+0000 7f575d817b80 10 client.30395 fetch_fsmap learned FSMap version 43 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.656+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5604dadc9250 con 0x5604db04bdf0 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.656+0000 7f5722ffd700 1 -- 172.21.15.31:0/736762789 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f572c013630 con 0x5604db04bdf0 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f5722ffd700 1 -- 172.21.15.31:0/736762789 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f572c00f360 con 0x5604db04bdf0 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f575d817b80 10 client.30395 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f575d817b80 10 client.30395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f575d817b80 20 client.30395 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f575d817b80 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db0b3d00 0x5604db0b6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f575d817b80 4 client.30395 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:35:28.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5604daf95ce0 con 0x5604db0b3d00 2023-07-29T16:35:28.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.657+0000 7f57437fe700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db0b3d00 0x5604db0b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:28.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.658+0000 7f57437fe700 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db0b3d00 0x5604db0b6110 crc :-1 s=READY pgs=468 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33f3ceff47674cf2 server_cookie=aa3e375d8c539558 in_seq=0 out_seq=0 2023-07-29T16:35:28.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:28.658+0000 7f5722ffd700 10 client.30395 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:29.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.103+0000 7f5722ffd700 1 -- 172.21.15.31:0/736762789 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5604daf95ce0 con 0x5604db0b3d00 2023-07-29T16:35:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.103+0000 7f5722ffd700 10 client.30395 handle_command_reply: tid=0 2023-07-29T16:35:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.103+0000 7f575d817b80 1 client.30395 shutdown 2023-07-29T16:35:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 msgr2=0x5604db05c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:29.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db05c430 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f572c005e30 tx=0x7f572c010cf0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 shutdown_connections 2023-07-29T16:35:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5604db0b3d00 0x5604db0b6110 crc :-1 s=READY pgs=468 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604db04ccf0 0x5604db058c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5604daea76b0 0x5604db05bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 --2- 172.21.15.31:0/736762789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604db04bdf0 0x5604db05c430 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 >> 172.21.15.31:0/736762789 conn(0x5604db040fd0 msgr2=0x5604db04fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.104+0000 7f5722ffd700 0 client.30395 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.105+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 shutdown_connections 2023-07-29T16:35:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.105+0000 7f575d817b80 1 -- 172.21.15.31:0/736762789 wait complete. 2023-07-29T16:35:29.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.105+0000 7f575d817b80 20 client.30395 trim_cache size 0 max 16384 2023-07-29T16:35:29.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:35:29.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:35:29.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:35:29.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:35:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57300a08e0 msgr2=0x7f57300a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57300a08e0 0x7f57300a2d90 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f572c00f640 tx=0x7f572c01e000 comp rx=0 tx=0).stop 2023-07-29T16:35:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5744103f70 msgr2=0x7f5744070bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5744103f70 0x7f5744070bd0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f573c00d010 tx=0x7f573c00bbf0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 shutdown_connections 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57300a08e0 0x7f57300a2d90 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5744103f70 0x7f5744070bd0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57440fec50 0x7f574406f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 --2- 172.21.15.31:0/655552704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57441015b0 0x7f574406fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 >> 172.21.15.31:0/655552704 conn(0x7f57440fa840 msgr2=0x7f57440ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 shutdown_connections 2023-07-29T16:35:29.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:29.106+0000 7f5749dff700 1 -- 172.21.15.31:0/655552704 wait complete. 2023-07-29T16:35:29.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 205369 35.48 MB/sec execute 193 sec latency 266.855 ms 2023-07-29T16:35:30.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 206109 35.49 MB/sec execute 194 sec latency 59.513 ms 2023-07-29T16:35:31.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:30 smithi039 ceph-mon[128858]: pgmap v973: 97 pgs: 97 active+clean; 691 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 535 op/s 2023-07-29T16:35:31.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:30 smithi156 ceph-mon[129245]: pgmap v973: 97 pgs: 97 active+clean; 691 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 535 op/s 2023-07-29T16:35:31.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:30 smithi031 ceph-mon[95504]: pgmap v973: 97 pgs: 97 active+clean; 691 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.9 MiB/s wr, 535 op/s 2023-07-29T16:35:31.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 206970 35.56 MB/sec execute 195 sec latency 71.935 ms 2023-07-29T16:35:32.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 207773 35.60 MB/sec execute 196 sec latency 71.149 ms 2023-07-29T16:35:33.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:32 smithi039 ceph-mon[128858]: pgmap v974: 97 pgs: 97 active+clean; 692 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.7 MiB/s wr, 525 op/s 2023-07-29T16:35:33.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:32 smithi156 ceph-mon[129245]: pgmap v974: 97 pgs: 97 active+clean; 692 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.7 MiB/s wr, 525 op/s 2023-07-29T16:35:33.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:32 smithi031 ceph-mon[95504]: pgmap v974: 97 pgs: 97 active+clean; 692 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.7 MiB/s wr, 525 op/s 2023-07-29T16:35:33.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 208391 35.62 MB/sec execute 197 sec latency 92.585 ms 2023-07-29T16:35:34.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 208956 35.60 MB/sec execute 198 sec latency 128.998 ms 2023-07-29T16:35:35.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:34 smithi039 ceph-mon[128858]: pgmap v975: 97 pgs: 97 active+clean; 696 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 648 op/s 2023-07-29T16:35:35.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:34 smithi156 ceph-mon[129245]: pgmap v975: 97 pgs: 97 active+clean; 696 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 648 op/s 2023-07-29T16:35:35.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:34 smithi031 ceph-mon[95504]: pgmap v975: 97 pgs: 97 active+clean; 696 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.8 MiB/s wr, 648 op/s 2023-07-29T16:35:35.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 209462 35.53 MB/sec execute 199 sec latency 94.620 ms 2023-07-29T16:35:36.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 210164 35.53 MB/sec execute 200 sec latency 79.699 ms 2023-07-29T16:35:37.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:36 smithi039 ceph-mon[128858]: pgmap v976: 97 pgs: 97 active+clean; 700 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 547 op/s 2023-07-29T16:35:37.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:36 smithi156 ceph-mon[129245]: pgmap v976: 97 pgs: 97 active+clean; 700 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 547 op/s 2023-07-29T16:35:37.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:36 smithi031 ceph-mon[95504]: pgmap v976: 97 pgs: 97 active+clean; 700 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 547 op/s 2023-07-29T16:35:37.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 210980 35.52 MB/sec execute 201 sec latency 50.309 ms 2023-07-29T16:35:38.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 211457 35.52 MB/sec execute 202 sec latency 266.488 ms 2023-07-29T16:35:38.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:38 smithi031 ceph-mon[95504]: pgmap v977: 97 pgs: 97 active+clean; 705 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 571 op/s 2023-07-29T16:35:38.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:38 smithi039 ceph-mon[128858]: pgmap v977: 97 pgs: 97 active+clean; 705 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 571 op/s 2023-07-29T16:35:38.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:38 smithi156 ceph-mon[129245]: pgmap v977: 97 pgs: 97 active+clean; 705 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 571 op/s 2023-07-29T16:35:39.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 212182 35.54 MB/sec execute 203 sec latency 65.199 ms 2023-07-29T16:35:40.284 INFO:tasks.workunit.client.0.smithi031.stdout: 10 212872 35.58 MB/sec execute 204 sec latency 74.383 ms 2023-07-29T16:35:41.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:40 smithi039 ceph-mon[128858]: pgmap v978: 97 pgs: 97 active+clean; 704 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.9 MiB/s wr, 594 op/s 2023-07-29T16:35:41.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:40 smithi156 ceph-mon[129245]: pgmap v978: 97 pgs: 97 active+clean; 704 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.9 MiB/s wr, 594 op/s 2023-07-29T16:35:41.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:40 smithi031 ceph-mon[95504]: pgmap v978: 97 pgs: 97 active+clean; 704 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.9 MiB/s wr, 594 op/s 2023-07-29T16:35:41.283 INFO:tasks.workunit.client.0.smithi031.stdout: 10 213481 35.54 MB/sec execute 205 sec latency 89.179 ms 2023-07-29T16:35:42.284 INFO:tasks.workunit.client.0.smithi031.stdout: 10 214230 35.56 MB/sec execute 206 sec latency 89.540 ms 2023-07-29T16:35:43.033 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:42 smithi039 ceph-mon[128858]: pgmap v979: 97 pgs: 97 active+clean; 708 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.6 MiB/s wr, 594 op/s 2023-07-29T16:35:43.081 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:42 smithi156 ceph-mon[129245]: pgmap v979: 97 pgs: 97 active+clean; 708 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.6 MiB/s wr, 594 op/s 2023-07-29T16:35:43.286 INFO:tasks.workunit.client.0.smithi031.stdout: 10 214757 35.53 MB/sec execute 207 sec latency 348.221 ms 2023-07-29T16:35:44.287 INFO:tasks.workunit.client.0.smithi031.stdout: 10 215049 35.44 MB/sec execute 208 sec latency 620.813 ms 2023-07-29T16:35:44.401 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:42 smithi031 ceph-mon[95504]: pgmap v979: 97 pgs: 97 active+clean; 708 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.6 MiB/s wr, 594 op/s 2023-07-29T16:35:45.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:44 smithi039 ceph-mon[128858]: pgmap v980: 97 pgs: 97 active+clean; 728 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.6 MiB/s wr, 687 op/s 2023-07-29T16:35:45.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:44 smithi156 ceph-mon[129245]: pgmap v980: 97 pgs: 97 active+clean; 728 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.6 MiB/s wr, 687 op/s 2023-07-29T16:35:45.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:44 smithi031 ceph-mon[95504]: pgmap v980: 97 pgs: 97 active+clean; 728 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.6 MiB/s wr, 687 op/s 2023-07-29T16:35:45.287 INFO:tasks.workunit.client.0.smithi031.stdout: 10 215412 35.34 MB/sec execute 209 sec latency 378.116 ms 2023-07-29T16:35:46.287 INFO:tasks.workunit.client.0.smithi031.stdout: 10 216136 35.37 MB/sec execute 210 sec latency 49.297 ms 2023-07-29T16:35:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:46 smithi039 ceph-mon[128858]: pgmap v981: 97 pgs: 97 active+clean; 750 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 10 MiB/s wr, 583 op/s 2023-07-29T16:35:47.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:46 smithi156 ceph-mon[129245]: pgmap v981: 97 pgs: 97 active+clean; 750 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 10 MiB/s wr, 583 op/s 2023-07-29T16:35:47.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:46 smithi031 ceph-mon[95504]: pgmap v981: 97 pgs: 97 active+clean; 750 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 10 MiB/s wr, 583 op/s 2023-07-29T16:35:47.287 INFO:tasks.workunit.client.0.smithi031.stdout: 10 216857 35.36 MB/sec execute 211 sec latency 92.563 ms 2023-07-29T16:35:48.288 INFO:tasks.workunit.client.0.smithi031.stdout: 10 217602 35.42 MB/sec execute 212 sec latency 61.673 ms 2023-07-29T16:35:48.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:48 smithi039 ceph-mon[128858]: pgmap v982: 97 pgs: 97 active+clean; 753 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 663 op/s 2023-07-29T16:35:48.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:48 smithi156 ceph-mon[129245]: pgmap v982: 97 pgs: 97 active+clean; 753 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 663 op/s 2023-07-29T16:35:49.288 INFO:tasks.workunit.client.0.smithi031.stdout: 10 217797 35.34 MB/sec execute 213 sec latency 748.789 ms 2023-07-29T16:35:49.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:48 smithi031 ceph-mon[95504]: pgmap v982: 97 pgs: 97 active+clean; 753 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 663 op/s 2023-07-29T16:35:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 -- 172.21.15.31:0/891429474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 msgr2=0x7f209810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 --2- 172.21.15.31:0/891429474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 0x7f209810aa20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f209000b0a0 tx=0x7f209000b3b0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 -- 172.21.15.31:0/891429474 shutdown_connections 2023-07-29T16:35:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 --2- 172.21.15.31:0/891429474 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 0x7f209810aa20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 --2- 172.21.15.31:0/891429474 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20981025e0 0x7f20981029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 --2- 172.21.15.31:0/891429474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.029+0000 7f209e3ff700 1 -- 172.21.15.31:0/891429474 >> 172.21.15.31:0/891429474 conn(0x7f20980fdc00 msgr2=0x7f2098100030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:50.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.030+0000 7f209e3ff700 1 -- 172.21.15.31:0/891429474 shutdown_connections 2023-07-29T16:35:50.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.030+0000 7f209e3ff700 1 -- 172.21.15.31:0/891429474 wait complete. 2023-07-29T16:35:50.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.032+0000 7f209e3ff700 1 Processor -- start 2023-07-29T16:35:50.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.032+0000 7f209e3ff700 1 -- start start 2023-07-29T16:35:50.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.032+0000 7f209e3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20981025e0 0x7f2098194040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f209e3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098194580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f209e3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 0x7f2098195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f209e3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098195ab0 con 0x7f2098104160 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f209e3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098195c20 con 0x7f20981037d0 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f209e3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098195de0 con 0x7f20981025e0 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f208f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098194580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f208f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098194580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47634/0 (socket says 172.21.15.31:47634) 2023-07-29T16:35:50.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.033+0000 7f208f7fe700 1 -- 172.21.15.31:0/724656526 learned_addr learned my addr 172.21.15.31:0/724656526 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:50.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.034+0000 7f208f7fe700 1 -- 172.21.15.31:0/724656526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20981025e0 msgr2=0x7f2098194040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.034+0000 7f208f7fe700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20981025e0 0x7f2098194040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.034+0000 7f208f7fe700 1 -- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 msgr2=0x7f2098195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.034+0000 7f208f7fe700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 0x7f2098195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.034+0000 7f208f7fe700 1 -- 172.21.15.31:0/724656526 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090009d00 con 0x7f20981037d0 2023-07-29T16:35:50.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.034+0000 7f208f7fe700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098194580 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f208000d950 tx=0x7f208000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:50.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.036+0000 7f2095ffb700 1 -- 172.21.15.31:0/724656526 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080009b40 con 0x7f20981037d0 2023-07-29T16:35:50.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.036+0000 7f2095ffb700 1 -- 172.21.15.31:0/724656526 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2080009ca0 con 0x7f20981037d0 2023-07-29T16:35:50.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.036+0000 7f2095ffb700 1 -- 172.21.15.31:0/724656526 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f208000f620 con 0x7f20981037d0 2023-07-29T16:35:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.037+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20981960c0 con 0x7f20981037d0 2023-07-29T16:35:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.037+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209819a890 con 0x7f20981037d0 2023-07-29T16:35:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.043+0000 7f20b1e20b80 1 Processor -- start 2023-07-29T16:35:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.043+0000 7f20b1e20b80 1 -- start start 2023-07-29T16:35:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 0x561b53f887b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f89160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53edf9b0 con 0x561b53de36b0 2023-07-29T16:35:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53edd370 con 0x561b53f87df0 2023-07-29T16:35:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53ed8c60 con 0x561b53f88cf0 2023-07-29T16:35:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f209c99c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f209c99c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f878b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59112/0 (socket says 172.21.15.31:59112) 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f209c99c700 1 -- 172.21.15.31:0/150076725 learned_addr learned my addr 172.21.15.31:0/150076725 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.044+0000 7f208f7fe700 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f89160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f2095ffb700 1 -- 172.21.15.31:0/724656526 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2080010460 con 0x7f20981037d0 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f209c99c700 1 -- 172.21.15.31:0/150076725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 msgr2=0x561b53f89160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f209c99c700 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f89160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f209c99c700 1 -- 172.21.15.31:0/150076725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 msgr2=0x561b53f887b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f209c99c700 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 0x561b53f887b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f209c99c700 1 -- 172.21.15.31:0/150076725 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b53d06d80 con 0x561b53de36b0 2023-07-29T16:35:50.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.045+0000 7f209c99c700 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f878b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2090009fd0 tx=0x7f20900092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e4e27d8213178a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.046+0000 7f208e7fc700 1 -- 172.21.15.31:0/150076725 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209000e040 con 0x561b53de36b0 2023-07-29T16:35:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.046+0000 7f208e7fc700 1 -- 172.21.15.31:0/150076725 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2090009490 con 0x561b53de36b0 2023-07-29T16:35:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.046+0000 7f208e7fc700 1 -- 172.21.15.31:0/150076725 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090003c60 con 0x561b53de36b0 2023-07-29T16:35:50.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.046+0000 7f2095ffb700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f20840a0960 0x7f20840a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.046+0000 7f2095ffb700 1 -- 172.21.15.31:0/724656526 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f20800cb990 con 0x7f20981037d0 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 -- 172.21.15.31:0/150076725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 msgr2=0x561b53f878b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f878b0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2090009fd0 tx=0x7f20900092e0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f2097fff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f20840a0960 0x7f20840a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 -- 172.21.15.31:0/150076725 shutdown_connections 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f89160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f878b0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/150076725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 0x561b53f887b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 -- 172.21.15.31:0/150076725 >> 172.21.15.31:0/150076725 conn(0x561b53f7cfd0 msgr2=0x561b53f82300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 -- 172.21.15.31:0/150076725 shutdown_connections 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.047+0000 7f20b1e20b80 1 -- 172.21.15.31:0/150076725 wait complete. 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.048+0000 7f2097fff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f20840a0960 0x7f20840a2e10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f20880094f0 tx=0x7f2088009460 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.048+0000 7f20b1e20b80 1 -- start start 2023-07-29T16:35:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.048+0000 7f20b1e20b80 1 Processor -- start 2023-07-29T16:35:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f90e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f913d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f98410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53e754b0 con 0x561b53de36b0 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53e75150 con 0x561b53f88cf0 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53d5c8c0 con 0x561b53f87df0 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f20b1e20b80 10 client.0 _ll_register_callbacks cb 0x561b53e7e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f2097fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f90e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f2097fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f90e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59122/0 (socket says 172.21.15.31:59122) 2023-07-29T16:35:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.049+0000 7f2097fff700 1 -- 172.21.15.31:0/1366636596 learned_addr learned my addr 172.21.15.31:0/1366636596 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f208f7fe700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f913d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f208f7fe700 1 -- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 msgr2=0x561b53f98410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f208f7fe700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f98410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f208f7fe700 1 -- 172.21.15.31:0/1366636596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 msgr2=0x561b53f90e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f208f7fe700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f90e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f208f7fe700 1 -- 172.21.15.31:0/1366636596 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209810b350 con 0x561b53f87df0 2023-07-29T16:35:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.050+0000 7f2097fff700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f90e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:35:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f208f7fe700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f913d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x561b53f87610 tx=0x7f2080010cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f557472a94a91fa server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20777fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20777fe700 1 -- 172.21.15.31:0/1366636596 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080098720 con 0x561b53f87df0 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20777fe700 1 -- 172.21.15.31:0/1366636596 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2080099030 con 0x561b53f87df0 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b53f03cb0 con 0x561b53f87df0 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20777fe700 1 -- 172.21.15.31:0/1366636596 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2080098990 con 0x561b53f87df0 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20b1e20b80 10 client.30415 fetch_fsmap learned FSMap version 43 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x561b53cefc00 con 0x561b53f87df0 2023-07-29T16:35:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.052+0000 7f20777fe700 1 -- 172.21.15.31:0/1366636596 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561b53f03cb0 con 0x561b53f87df0 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20777fe700 1 -- 172.21.15.31:0/1366636596 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2080010710 con 0x561b53f87df0 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20b1e20b80 10 client.30415 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20b1e20b80 10 client.30415 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20b1e20b80 20 client.30415 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53f9cb90 0x561b53f9efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20b1e20b80 4 client.30415 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:35:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b53f56d10 con 0x561b53f9cb90 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.053+0000 7f2097fff700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53f9cb90 0x561b53f9efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.054+0000 7f2097fff700 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53f9cb90 0x561b53f9efa0 crc :-1 s=READY pgs=469 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed948c9a30e73859 server_cookie=4347e804be84aa69 in_seq=0 out_seq=0 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.054+0000 7f20777fe700 10 client.30415 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20777fe700 1 -- 172.21.15.31:0/1366636596 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x561b53f56d10 con 0x561b53f9cb90 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20777fe700 10 client.30415 handle_command_reply: tid=0 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 client.30415 shutdown 2023-07-29T16:35:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 msgr2=0x561b53f913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f913d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x561b53f87610 tx=0x7f2080010cb0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 shutdown_connections 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53f9cb90 0x561b53f9efa0 crc :-1 s=READY pgs=469 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f98410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f90e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20777fe700 0 client.30415 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1366636596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f913d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.056+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 >> 172.21.15.31:0/1366636596 conn(0x561b53f7cfd0 msgr2=0x561b53f81a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.057+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 shutdown_connections 2023-07-29T16:35:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.057+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1366636596 wait complete. 2023-07-29T16:35:50.063 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.057+0000 7f20b1e20b80 20 client.30415 trim_cache size 0 max 16384 2023-07-29T16:35:50.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.079+0000 7f20b1e20b80 1 Processor -- start 2023-07-29T16:35:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.079+0000 7f20b1e20b80 1 -- start start 2023-07-29T16:35:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.079+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f963c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.079+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f8bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f8c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53f90a50 con 0x561b53de36b0 2023-07-29T16:35:50.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53f831f0 con 0x561b53f88cf0 2023-07-29T16:35:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53f82ff0 con 0x561b53f87df0 2023-07-29T16:35:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f209c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f8c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f209c99c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f8c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:47660/0 (socket says 172.21.15.31:47660) 2023-07-29T16:35:50.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f209c99c700 1 -- 172.21.15.31:0/2480056298 learned_addr learned my addr 172.21.15.31:0/2480056298 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.080+0000 7f208f7fe700 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f8bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.081+0000 7f209c99c700 1 -- 172.21.15.31:0/2480056298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 msgr2=0x561b53f8bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.081+0000 7f209c99c700 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f8bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.081+0000 7f209c99c700 1 -- 172.21.15.31:0/2480056298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 msgr2=0x561b53f963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.081+0000 7f209c99c700 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.081+0000 7f209c99c700 1 -- 172.21.15.31:0/2480056298 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b53d2d1f0 con 0x561b53f88cf0 2023-07-29T16:35:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.081+0000 7f209c99c700 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f8c140 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f20900060e0 tx=0x7f2090009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de805840fe463a82 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.082+0000 7f208effd700 1 -- 172.21.15.31:0/2480056298 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209000e040 con 0x561b53f88cf0 2023-07-29T16:35:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.082+0000 7f208effd700 1 -- 172.21.15.31:0/2480056298 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f209000ba50 con 0x561b53f88cf0 2023-07-29T16:35:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.082+0000 7f208effd700 1 -- 172.21.15.31:0/2480056298 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20900246c0 con 0x561b53f88cf0 2023-07-29T16:35:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 -- 172.21.15.31:0/2480056298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 msgr2=0x561b53f8c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f8c140 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f20900060e0 tx=0x7f2090009b80 comp rx=0 tx=0).stop 2023-07-29T16:35:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 -- 172.21.15.31:0/2480056298 shutdown_connections 2023-07-29T16:35:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f88cf0 0x561b53f8c140 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/2480056298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f87df0 0x561b53f8bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.083+0000 7f20b1e20b80 1 -- 172.21.15.31:0/2480056298 >> 172.21.15.31:0/2480056298 conn(0x561b53f7cfd0 msgr2=0x561b53f8f080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.084+0000 7f20b1e20b80 1 -- 172.21.15.31:0/2480056298 shutdown_connections 2023-07-29T16:35:50.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.084+0000 7f20b1e20b80 1 -- 172.21.15.31:0/2480056298 wait complete. 2023-07-29T16:35:50.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.084+0000 7f20b1e20b80 1 -- start start 2023-07-29T16:35:50.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 Processor -- start 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f94a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 0x561b53f94f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f95490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53f96660 con 0x561b53de36b0 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53fee730 con 0x561b53f87df0 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b53fee870 con 0x561b53f88cf0 2023-07-29T16:35:50.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.085+0000 7f20b1e20b80 10 client.0 _ll_register_callbacks cb 0x561b53e7e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f94a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f94a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59146/0 (socket says 172.21.15.31:59146) 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 -- 172.21.15.31:0/1145976372 learned_addr learned my addr 172.21.15.31:0/1145976372 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 -- 172.21.15.31:0/1145976372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 msgr2=0x561b53f95490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f95490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 -- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 msgr2=0x561b53f94f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:35:50.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 0x561b53f94f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.086+0000 7f2097fff700 1 -- 172.21.15.31:0/1145976372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b53d06d80 con 0x561b53de36b0 2023-07-29T16:35:50.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.087+0000 7f2097fff700 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f94a10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x561b53f9f870 tx=0x7f2088009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34b8e877dc94ec5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.087+0000 7f20777fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.088+0000 7f20777fe700 1 -- 172.21.15.31:0/1145976372 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2088022040 con 0x561b53de36b0 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.088+0000 7f20777fe700 1 -- 172.21.15.31:0/1145976372 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f2088024d60 con 0x561b53de36b0 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.088+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2090009490 con 0x561b53de36b0 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.088+0000 7f20777fe700 1 -- 172.21.15.31:0/1145976372 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f208802d430 con 0x561b53de36b0 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.088+0000 7f20b1e20b80 10 client.20166 fetch_fsmap learned FSMap version 43 2023-07-29T16:35:50.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.088+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x561b53d05250 con 0x561b53de36b0 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20777fe700 1 -- 172.21.15.31:0/1145976372 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2090009490 con 0x561b53de36b0 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20777fe700 1 -- 172.21.15.31:0/1145976372 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f20880106f0 con 0x561b53de36b0 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20b1e20b80 10 client.20166 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20b1e20b80 10 client.20166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20b1e20b80 20 client.20166 populate_metadata read hostname 'smithi031' 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53ff01c0 0x561b53ff25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:35:50.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20b1e20b80 4 client.20166 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:35:50.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.089+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561b53ed1ce0 con 0x561b53ff01c0 2023-07-29T16:35:50.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.093+0000 7f208f7fe700 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53ff01c0 0x561b53ff25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:35:50.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.096+0000 7f208f7fe700 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53ff01c0 0x561b53ff25d0 crc :-1 s=READY pgs=470 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f296f6995fe88cb server_cookie=1754052f3cf127d0 in_seq=0 out_seq=0 2023-07-29T16:35:50.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.096+0000 7f20777fe700 10 client.20166 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:50.291 INFO:tasks.workunit.client.0.smithi031.stdout: 10 217928 35.19 MB/sec execute 214 sec latency 1164.998 ms 2023-07-29T16:35:50.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20777fe700 1 -- 172.21.15.31:0/1145976372 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561b53ed1ce0 con 0x561b53ff01c0 2023-07-29T16:35:50.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20777fe700 10 client.20166 handle_command_reply: tid=0 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 client.20166 shutdown 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 msgr2=0x561b53f94a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f94a10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x561b53f9f870 tx=0x7f2088009260 comp rx=0 tx=0).stop 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 shutdown_connections 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x561b53ff01c0 0x561b53ff25d0 crc :-1 s=READY pgs=470 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561b53f88cf0 0x561b53f95490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b53de36b0 0x561b53f94a10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 --2- 172.21.15.31:0/1145976372 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561b53f87df0 0x561b53f94f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 >> 172.21.15.31:0/1145976372 conn(0x561b53f7cfd0 msgr2=0x561b53f8bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:50.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20777fe700 0 client.20166 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:35:50.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.667+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 shutdown_connections 2023-07-29T16:35:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.668+0000 7f20b1e20b80 1 -- 172.21.15.31:0/1145976372 wait complete. 2023-07-29T16:35:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.670+0000 7f20b1e20b80 20 client.20166 trim_cache size 0 max 16384 2023-07-29T16:35:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:35:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:35:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:35:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:35:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.673+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f20840a0960 msgr2=0x7f20840a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.673+0000 7f209e3ff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f20840a0960 0x7f20840a2e10 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f20880094f0 tx=0x7f2088009460 comp rx=0 tx=0).stop 2023-07-29T16:35:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.673+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 msgr2=0x7f2098194580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:35:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.673+0000 7f209e3ff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098194580 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f208000d950 tx=0x7f208000dd10 comp rx=0 tx=0).stop 2023-07-29T16:35:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.674+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 shutdown_connections 2023-07-29T16:35:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.674+0000 7f209e3ff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f20840a0960 0x7f20840a2e10 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.674+0000 7f209e3ff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2098104160 0x7f2098195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.674+0000 7f209e3ff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20981025e0 0x7f2098194040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.674+0000 7f209e3ff700 1 --2- 172.21.15.31:0/724656526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20981037d0 0x7f2098194580 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:35:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.674+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 >> 172.21.15.31:0/724656526 conn(0x7f20980fdc00 msgr2=0x7f20980ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:35:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.676+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 shutdown_connections 2023-07-29T16:35:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:35:50.676+0000 7f209e3ff700 1 -- 172.21.15.31:0/724656526 wait complete. 2023-07-29T16:35:51.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:50 smithi039 ceph-mon[128858]: pgmap v983: 97 pgs: 97 active+clean; 737 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 12 MiB/s wr, 668 op/s 2023-07-29T16:35:51.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:50 smithi156 ceph-mon[129245]: pgmap v983: 97 pgs: 97 active+clean; 737 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 12 MiB/s wr, 668 op/s 2023-07-29T16:35:51.196 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:50 smithi031 ceph-mon[95504]: pgmap v983: 97 pgs: 97 active+clean; 737 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 12 MiB/s wr, 668 op/s 2023-07-29T16:35:51.291 INFO:tasks.workunit.client.0.smithi031.stdout: 10 218232 35.10 MB/sec execute 215 sec latency 366.717 ms 2023-07-29T16:35:52.291 INFO:tasks.workunit.client.0.smithi031.stdout: 10 218833 35.09 MB/sec execute 216 sec latency 67.477 ms 2023-07-29T16:35:53.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:52 smithi039 ceph-mon[128858]: pgmap v984: 97 pgs: 97 active+clean; 748 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 15 MiB/s wr, 725 op/s 2023-07-29T16:35:53.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:52 smithi156 ceph-mon[129245]: pgmap v984: 97 pgs: 97 active+clean; 748 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 15 MiB/s wr, 725 op/s 2023-07-29T16:35:53.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:52 smithi031 ceph-mon[95504]: pgmap v984: 97 pgs: 97 active+clean; 748 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 15 MiB/s wr, 725 op/s 2023-07-29T16:35:53.291 INFO:tasks.workunit.client.0.smithi031.stdout: 10 219438 35.07 MB/sec execute 217 sec latency 49.706 ms 2023-07-29T16:35:54.299 INFO:tasks.workunit.client.0.smithi031.stdout: 10 219672 34.96 MB/sec execute 218 sec latency 323.437 ms 2023-07-29T16:35:55.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:54 smithi039 ceph-mon[128858]: pgmap v985: 97 pgs: 97 active+clean; 747 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 19 MiB/s wr, 948 op/s 2023-07-29T16:35:55.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:54 smithi156 ceph-mon[129245]: pgmap v985: 97 pgs: 97 active+clean; 747 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 19 MiB/s wr, 948 op/s 2023-07-29T16:35:55.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:54 smithi031 ceph-mon[95504]: pgmap v985: 97 pgs: 97 active+clean; 747 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 19 MiB/s wr, 948 op/s 2023-07-29T16:35:55.300 INFO:tasks.workunit.client.0.smithi031.stdout: 10 220291 34.89 MB/sec execute 219 sec latency 131.118 ms 2023-07-29T16:35:56.299 INFO:tasks.workunit.client.0.smithi031.stdout: 10 221115 34.97 MB/sec execute 220 sec latency 59.707 ms 2023-07-29T16:35:57.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:56 smithi039 ceph-mon[128858]: pgmap v986: 97 pgs: 97 active+clean; 693 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 19 MiB/s wr, 956 op/s 2023-07-29T16:35:57.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:56 smithi156 ceph-mon[129245]: pgmap v986: 97 pgs: 97 active+clean; 693 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 19 MiB/s wr, 956 op/s 2023-07-29T16:35:57.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:56 smithi031 ceph-mon[95504]: pgmap v986: 97 pgs: 97 active+clean; 693 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 19 MiB/s wr, 956 op/s 2023-07-29T16:35:57.300 INFO:tasks.workunit.client.0.smithi031.stdout: 10 221883 35.03 MB/sec execute 221 sec latency 96.604 ms 2023-07-29T16:35:58.301 INFO:tasks.workunit.client.0.smithi031.stdout: 10 222441 35.02 MB/sec execute 222 sec latency 51.410 ms 2023-07-29T16:35:58.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:35:58 smithi031 ceph-mon[95504]: pgmap v987: 97 pgs: 97 active+clean; 691 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 18 MiB/s wr, 992 op/s 2023-07-29T16:35:58.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:35:58 smithi039 ceph-mon[128858]: pgmap v987: 97 pgs: 97 active+clean; 691 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 18 MiB/s wr, 992 op/s 2023-07-29T16:35:58.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:35:58 smithi156 ceph-mon[129245]: pgmap v987: 97 pgs: 97 active+clean; 691 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 18 MiB/s wr, 992 op/s 2023-07-29T16:35:59.301 INFO:tasks.workunit.client.0.smithi031.stdout: 10 222993 34.97 MB/sec execute 223 sec latency 56.480 ms 2023-07-29T16:36:00.301 INFO:tasks.workunit.client.0.smithi031.stdout: 10 223712 34.96 MB/sec execute 224 sec latency 56.542 ms 2023-07-29T16:36:01.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:00 smithi039 ceph-mon[128858]: pgmap v988: 97 pgs: 97 active+clean; 689 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 17 MiB/s wr, 930 op/s 2023-07-29T16:36:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:00 smithi156 ceph-mon[129245]: pgmap v988: 97 pgs: 97 active+clean; 689 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 17 MiB/s wr, 930 op/s 2023-07-29T16:36:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:00 smithi031 ceph-mon[95504]: pgmap v988: 97 pgs: 97 active+clean; 689 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 17 MiB/s wr, 930 op/s 2023-07-29T16:36:01.302 INFO:tasks.workunit.client.0.smithi031.stdout: 10 224523 35.00 MB/sec execute 225 sec latency 56.078 ms 2023-07-29T16:36:02.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 225143 35.05 MB/sec execute 226 sec latency 68.951 ms 2023-07-29T16:36:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:02 smithi039 ceph-mon[128858]: pgmap v989: 97 pgs: 97 active+clean; 689 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 16 MiB/s wr, 905 op/s 2023-07-29T16:36:03.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:02 smithi156 ceph-mon[129245]: pgmap v989: 97 pgs: 97 active+clean; 689 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 16 MiB/s wr, 905 op/s 2023-07-29T16:36:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:02 smithi031 ceph-mon[95504]: pgmap v989: 97 pgs: 97 active+clean; 689 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 16 MiB/s wr, 905 op/s 2023-07-29T16:36:03.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 225702 35.01 MB/sec execute 227 sec latency 73.032 ms 2023-07-29T16:36:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 -- 172.21.15.31:0/801706125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c100e60 msgr2=0x7f8f6c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 --2- 172.21.15.31:0/801706125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c100e60 0x7f8f6c103240 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009990 tx=0x7f8f5c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 -- 172.21.15.31:0/801706125 shutdown_connections 2023-07-29T16:36:04.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 --2- 172.21.15.31:0/801706125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 --2- 172.21.15.31:0/801706125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c100e60 0x7f8f6c103240 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 --2- 172.21.15.31:0/801706125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f6c103780 0x7f8f6c105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 -- 172.21.15.31:0/801706125 >> 172.21.15.31:0/801706125 conn(0x7f8f6c0fa840 msgr2=0x7f8f6c0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:04.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 -- 172.21.15.31:0/801706125 shutdown_connections 2023-07-29T16:36:04.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.120+0000 7f8f721ff700 1 -- 172.21.15.31:0/801706125 wait complete. 2023-07-29T16:36:04.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.122+0000 7f8f721ff700 1 Processor -- start 2023-07-29T16:36:04.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.122+0000 7f8f721ff700 1 -- start start 2023-07-29T16:36:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f721ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c100e60 0x7f8f6c0fee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f721ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f6c103780 0x7f8f6c0ff350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f721ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c0ff890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f721ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c071190 con 0x7f8f6c103780 2023-07-29T16:36:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f721ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c071300 con 0x7f8f6c100e60 2023-07-29T16:36:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f721ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c0714c0 con 0x7f8f6c1060a0 2023-07-29T16:36:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f6bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c0ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f6bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c0ff890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46740/0 (socket says 172.21.15.31:46740) 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f6bfff700 1 -- 172.21.15.31:0/4079999847 learned_addr learned my addr 172.21.15.31:0/4079999847 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c100e60 0x7f8f6c0fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.123+0000 7f8f6affd700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f6c103780 0x7f8f6c0ff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6bfff700 1 -- 172.21.15.31:0/4079999847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c100e60 msgr2=0x7f8f6c0fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c100e60 0x7f8f6c0fee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6bfff700 1 -- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f6c103780 msgr2=0x7f8f6c0ff350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f6c103780 0x7f8f6c0ff350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6bfff700 1 -- 172.21.15.31:0/4079999847 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0095f0 con 0x7f8f6c1060a0 2023-07-29T16:36:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c100e60 0x7f8f6c0fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:36:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.124+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c0ff890 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8f64007f10 tx=0x7f8f6400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.125+0000 7f8f68ff9700 1 -- 172.21.15.31:0/4079999847 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64017070 con 0x7f8f6c1060a0 2023-07-29T16:36:04.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.125+0000 7f8f68ff9700 1 -- 172.21.15.31:0/4079999847 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f6400f040 con 0x7f8f6c1060a0 2023-07-29T16:36:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.125+0000 7f8f68ff9700 1 -- 172.21.15.31:0/4079999847 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64012350 con 0x7f8f6c1060a0 2023-07-29T16:36:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.125+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c0717a0 con 0x7f8f6c1060a0 2023-07-29T16:36:04.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.125+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c071c60 con 0x7f8f6c1060a0 2023-07-29T16:36:04.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.130+0000 7f8f68ff9700 1 -- 172.21.15.31:0/4079999847 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8f6400dbd0 con 0x7f8f6c1060a0 2023-07-29T16:36:04.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.133+0000 7f8f68ff9700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f540a0960 0x7f8f540a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.133+0000 7f8f68ff9700 1 -- 172.21.15.31:0/4079999847 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f640cb460 con 0x7f8f6c1060a0 2023-07-29T16:36:04.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.133+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f540a0960 0x7f8f540a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.135+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f540a0960 0x7f8f540a2e10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c100720 tx=0x7f8f5c005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.147+0000 7f8f85bdbb80 1 Processor -- start 2023-07-29T16:36:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 -- start start 2023-07-29T16:36:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328ba0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328bafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328bb500 0x55f0328bb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0328099b0 con 0x55f0328ba600 2023-07-29T16:36:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f032807370 con 0x55f0328bb500 2023-07-29T16:36:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.148+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f032802c60 con 0x55f03270d6b0 2023-07-29T16:36:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.149+0000 7f8f6affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328bafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.150+0000 7f8f6affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328bafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58540/0 (socket says 172.21.15.31:58540) 2023-07-29T16:36:04.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.150+0000 7f8f6affd700 1 -- 172.21.15.31:0/3414227130 learned_addr learned my addr 172.21.15.31:0/3414227130 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.151+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328ba0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.151+0000 7f8f6affd700 1 -- 172.21.15.31:0/3414227130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 msgr2=0x55f0328ba0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.151+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328ba0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.151+0000 7f8f6affd700 1 -- 172.21.15.31:0/3414227130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328bb500 msgr2=0x55f0328bb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.152+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328bb500 0x55f0328bb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.152+0000 7f8f6affd700 1 -- 172.21.15.31:0/3414227130 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f032630d80 con 0x55f0328ba600 2023-07-29T16:36:04.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.152+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328bafc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8f60008c10 tx=0x7f8f60008f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7734ee4c8b3fca22 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:04.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.153+0000 7f8f51ffb700 1 -- 172.21.15.31:0/3414227130 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6000edc0 con 0x55f0328ba600 2023-07-29T16:36:04.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.153+0000 7f8f51ffb700 1 -- 172.21.15.31:0/3414227130 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f60019920 con 0x55f0328ba600 2023-07-29T16:36:04.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.153+0000 7f8f51ffb700 1 -- 172.21.15.31:0/3414227130 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60010970 con 0x55f0328ba600 2023-07-29T16:36:04.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3414227130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 msgr2=0x55f0328bafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328bafc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8f60008c10 tx=0x7f8f60008f20 comp rx=0 tx=0).stop 2023-07-29T16:36:04.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3414227130 shutdown_connections 2023-07-29T16:36:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328bb500 0x55f0328bb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328ba0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3414227130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328bafc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.154+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3414227130 >> 172.21.15.31:0/3414227130 conn(0x55f0328a6fd0 msgr2=0x55f0328ac300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.155+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3414227130 shutdown_connections 2023-07-29T16:36:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.155+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3414227130 wait complete. 2023-07-29T16:36:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 -- start start 2023-07-29T16:36:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 Processor -- start 2023-07-29T16:36:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 0x55f0328b2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328bb500 0x55f0328b8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f03279f4b0 con 0x55f0328ba600 2023-07-29T16:36:04.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f03279f150 con 0x55f03270d6b0 2023-07-29T16:36:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.156+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0326868c0 con 0x55f0328bb500 2023-07-29T16:36:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f85bdbb80 10 client.0 _ll_register_callbacks cb 0x55f0327a85c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 0x55f0328b2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 0x55f0328b2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:54968/0 (socket says 172.21.15.31:54968) 2023-07-29T16:36:04.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/3266297807 learned_addr learned my addr 172.21.15.31:0/3266297807 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328bb500 0x55f0328b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/3266297807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328bb500 msgr2=0x55f0328b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328bb500 0x55f0328b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/3266297807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 msgr2=0x55f0328b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/3266297807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f60008870 con 0x55f03270d6b0 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.157+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.158+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 0x55f0328b2b90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c010 tx=0x7f8f5c009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf9716c2525bea6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.158+0000 7f8f4affd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:36:04.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.158+0000 7f8f4affd700 1 -- 172.21.15.31:0/3266297807 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00fa40 con 0x55f03270d6b0 2023-07-29T16:36:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.158+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f03282dcb0 con 0x55f03270d6b0 2023-07-29T16:36:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.158+0000 7f8f4affd700 1 -- 172.21.15.31:0/3266297807 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f5c00fba0 con 0x55f03270d6b0 2023-07-29T16:36:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f4affd700 1 -- 172.21.15.31:0/3266297807 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f5c00fd60 con 0x55f03270d6b0 2023-07-29T16:36:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 10 client.30413 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55f032619c00 con 0x55f03270d6b0 2023-07-29T16:36:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f4affd700 1 -- 172.21.15.31:0/3266297807 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f03282dcb0 con 0x55f03270d6b0 2023-07-29T16:36:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f4affd700 1 -- 172.21.15.31:0/3266297807 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8f5c02f650 con 0x55f03270d6b0 2023-07-29T16:36:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 10 client.30413 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 10 client.30413 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 20 client.30413 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328d7ce0 0x55f0328da0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 4 client.30413 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:36:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.159+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f032880d10 con 0x55f0328d7ce0 2023-07-29T16:36:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.160+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328d7ce0 0x55f0328da0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.160+0000 7f8f4affd700 10 client.30413 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:04.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.160+0000 7f8f6affd700 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328d7ce0 0x55f0328da0f0 crc :-1 s=READY pgs=471 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f086d65186a970ea server_cookie=40a52331b0bfc9b7 in_seq=0 out_seq=0 2023-07-29T16:36:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f4affd700 1 -- 172.21.15.31:0/3266297807 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55f032880d10 con 0x55f0328d7ce0 2023-07-29T16:36:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f4affd700 10 client.30413 handle_command_reply: tid=0 2023-07-29T16:36:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 client.30413 shutdown 2023-07-29T16:36:04.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 msgr2=0x55f0328b2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 0x55f0328b2b90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00c010 tx=0x7f8f5c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 shutdown_connections 2023-07-29T16:36:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328d7ce0 0x55f0328da0f0 crc :-1 s=READY pgs=471 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328bb500 0x55f0328b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f03270d6b0 0x55f0328b2b90 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.164+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/3266297807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328ba600 0x55f0328b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.165+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 >> 172.21.15.31:0/3266297807 conn(0x55f0328a6fd0 msgr2=0x55f0328aba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:04.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.165+0000 7f8f4affd700 0 client.30413 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.165+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 shutdown_connections 2023-07-29T16:36:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.165+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/3266297807 wait complete. 2023-07-29T16:36:04.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.165+0000 7f8f85bdbb80 20 client.30413 trim_cache size 0 max 16384 2023-07-29T16:36:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.184+0000 7f8f85bdbb80 1 Processor -- start 2023-07-29T16:36:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 -- start start 2023-07-29T16:36:04.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328a6fd0 0x55f0328e7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 0x55f0328aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328d9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0328bad00 con 0x55f03270d6b0 2023-07-29T16:36:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f032880d10 con 0x55f0328ba600 2023-07-29T16:36:04.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f03282dcb0 con 0x55f0328a6fd0 2023-07-29T16:36:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.185+0000 7f8f6affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 0x55f0328aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328a6fd0 0x55f0328e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 0x55f0328aba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58560/0 (socket says 172.21.15.31:58560) 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6affd700 1 -- 172.21.15.31:0/1762534691 learned_addr learned my addr 172.21.15.31:0/1762534691 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328d9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/1762534691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 msgr2=0x55f0328d9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328d9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/1762534691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 msgr2=0x55f0328aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 0x55f0328aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/1762534691 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0326571f0 con 0x55f0328a6fd0 2023-07-29T16:36:04.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6affd700 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 0x55f0328aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:36:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.186+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328a6fd0 0x55f0328e7650 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c005b20 tx=0x7f8f5c02fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebde6880fdc7fc5e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.187+0000 7f8f527fc700 1 -- 172.21.15.31:0/1762534691 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c02f5f0 con 0x55f0328a6fd0 2023-07-29T16:36:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.187+0000 7f8f527fc700 1 -- 172.21.15.31:0/1762534691 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f5c02b040 con 0x55f0328a6fd0 2023-07-29T16:36:04.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.187+0000 7f8f527fc700 1 -- 172.21.15.31:0/1762534691 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c024620 con 0x55f0328a6fd0 2023-07-29T16:36:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/1762534691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328a6fd0 msgr2=0x55f0328e7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328a6fd0 0x55f0328e7650 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c005b20 tx=0x7f8f5c02fc00 comp rx=0 tx=0).stop 2023-07-29T16:36:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/1762534691 shutdown_connections 2023-07-29T16:36:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328d9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f0328a6fd0 0x55f0328e7650 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/1762534691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f03270d6b0 0x55f0328aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/1762534691 >> 172.21.15.31:0/1762534691 conn(0x55f0328bbc00 msgr2=0x55f0328b8990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.188+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/1762534691 shutdown_connections 2023-07-29T16:36:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.189+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/1762534691 wait complete. 2023-07-29T16:36:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.189+0000 7f8f85bdbb80 1 -- start start 2023-07-29T16:36:04.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.189+0000 7f8f85bdbb80 1 Processor -- start 2023-07-29T16:36:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328e4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328a6fd0 0x55f0328e5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328e6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0328e6700 con 0x55f0328a6fd0 2023-07-29T16:36:04.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0328e6870 con 0x55f0328ba600 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0328e6a30 con 0x55f03270d6b0 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f85bdbb80 10 client.0 _ll_register_callbacks cb 0x55f0327a85c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328e4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f6b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328e4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46764/0 (socket says 172.21.15.31:46764) 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f6b7fe700 1 -- 172.21.15.31:0/517988585 learned_addr learned my addr 172.21.15.31:0/517988585 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328e6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.191+0000 7f8f6affd700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328a6fd0 0x55f0328e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6bfff700 1 -- 172.21.15.31:0/517988585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 msgr2=0x55f0328e4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328e4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6bfff700 1 -- 172.21.15.31:0/517988585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328a6fd0 msgr2=0x55f0328e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328a6fd0 0x55f0328e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6bfff700 1 -- 172.21.15.31:0/517988585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f032630d80 con 0x55f0328ba600 2023-07-29T16:36:04.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328e4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f6bfff700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328e6150 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x55f0328b9e20 tx=0x7f8f64098750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45edaf3fd763752e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f4affd700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.192+0000 7f8f4affd700 1 -- 172.21.15.31:0/517988585 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f640a0f50 con 0x55f0328ba600 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.193+0000 7f8f4affd700 1 -- 172.21.15.31:0/517988585 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8f64098d60 con 0x55f0328ba600 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.193+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f60019920 con 0x55f0328ba600 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.193+0000 7f8f4affd700 1 -- 172.21.15.31:0/517988585 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f64098f20 con 0x55f0328ba600 2023-07-29T16:36:04.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.193+0000 7f8f85bdbb80 10 client.30419 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.193+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55f032619c00 con 0x55f0328ba600 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f4affd700 1 -- 172.21.15.31:0/517988585 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f60019920 con 0x55f0328ba600 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f4affd700 1 -- 172.21.15.31:0/517988585 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8f64097a60 con 0x55f0328ba600 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f85bdbb80 10 client.30419 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f85bdbb80 10 client.30419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f85bdbb80 20 client.30419 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328bb500 0x55f03292c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:04.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f85bdbb80 4 client.30419 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:36:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f0327fbce0 con 0x55f0328bb500 2023-07-29T16:36:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.194+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328bb500 0x55f03292c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.195+0000 7f8f4affd700 10 client.30419 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:04.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.195+0000 7f8f6b7fe700 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328bb500 0x55f03292c6c0 crc :-1 s=READY pgs=472 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7c86c7a0f74828f server_cookie=ea4a8e6ae071ae64 in_seq=0 out_seq=0 2023-07-29T16:36:04.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 226238 34.96 MB/sec execute 228 sec latency 92.137 ms 2023-07-29T16:36:04.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.614+0000 7f8f4affd700 1 -- 172.21.15.31:0/517988585 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f0327fbce0 con 0x55f0328bb500 2023-07-29T16:36:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f4affd700 10 client.30419 handle_command_reply: tid=0 2023-07-29T16:36:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 client.30419 shutdown 2023-07-29T16:36:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 msgr2=0x55f0328e6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328e6150 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x55f0328b9e20 tx=0x7f8f64098750 comp rx=0 tx=0).stop 2023-07-29T16:36:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 shutdown_connections 2023-07-29T16:36:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55f0328bb500 0x55f03292c6c0 crc :-1 s=READY pgs=472 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0328ba600 0x55f0328e6150 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f03270d6b0 0x55f0328e4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 --2- 172.21.15.31:0/517988585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f0328a6fd0 0x55f0328e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 >> 172.21.15.31:0/517988585 conn(0x55f0328bbc00 msgr2=0x55f0328aafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.615+0000 7f8f4affd700 0 client.30419 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.616+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 shutdown_connections 2023-07-29T16:36:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.616+0000 7f8f85bdbb80 1 -- 172.21.15.31:0/517988585 wait complete. 2023-07-29T16:36:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.618+0000 7f8f85bdbb80 20 client.30419 trim_cache size 0 max 16384 2023-07-29T16:36:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:36:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:36:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:36:04.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:36:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.619+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f540a0960 msgr2=0x7f8f540a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.619+0000 7f8f721ff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f540a0960 0x7f8f540a2e10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c100720 tx=0x7f8f5c005a90 comp rx=0 tx=0).stop 2023-07-29T16:36:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.619+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 msgr2=0x7f8f6c0ff890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.619+0000 7f8f721ff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c0ff890 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8f64007f10 tx=0x7f8f6400d3b0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 shutdown_connections 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8f540a0960 0x7f8f540a2e10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f6c1060a0 0x7f8f6c0ff890 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f6c100e60 0x7f8f6c0fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 --2- 172.21.15.31:0/4079999847 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8f6c103780 0x7f8f6c0ff350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 >> 172.21.15.31:0/4079999847 conn(0x7f8f6c0fa840 msgr2=0x7f8f6c101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 shutdown_connections 2023-07-29T16:36:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:04.620+0000 7f8f721ff700 1 -- 172.21.15.31:0/4079999847 wait complete. 2023-07-29T16:36:05.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:04 smithi039 ceph-mon[128858]: pgmap v990: 97 pgs: 97 active+clean; 692 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 867 op/s 2023-07-29T16:36:05.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:04 smithi156 ceph-mon[129245]: pgmap v990: 97 pgs: 97 active+clean; 692 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 867 op/s 2023-07-29T16:36:05.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:04 smithi031 ceph-mon[95504]: pgmap v990: 97 pgs: 97 active+clean; 692 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 867 op/s 2023-07-29T16:36:05.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 226965 34.98 MB/sec execute 229 sec latency 236.510 ms 2023-07-29T16:36:06.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 227751 35.03 MB/sec execute 230 sec latency 117.825 ms 2023-07-29T16:36:07.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:06 smithi039 ceph-mon[128858]: pgmap v991: 97 pgs: 97 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.9 MiB/s wr, 696 op/s 2023-07-29T16:36:07.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:06 smithi156 ceph-mon[129245]: pgmap v991: 97 pgs: 97 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.9 MiB/s wr, 696 op/s 2023-07-29T16:36:07.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:06 smithi031 ceph-mon[95504]: pgmap v991: 97 pgs: 97 active+clean; 696 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 7.9 MiB/s wr, 696 op/s 2023-07-29T16:36:07.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 228415 35.06 MB/sec execute 231 sec latency 83.199 ms 2023-07-29T16:36:08.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 229075 34.99 MB/sec execute 232 sec latency 80.389 ms 2023-07-29T16:36:08.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:08 smithi031 ceph-mon[95504]: pgmap v992: 97 pgs: 97 active+clean; 695 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.7 MiB/s wr, 610 op/s 2023-07-29T16:36:08.527 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:08 smithi039 ceph-mon[128858]: pgmap v992: 97 pgs: 97 active+clean; 695 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.7 MiB/s wr, 610 op/s 2023-07-29T16:36:08.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:08 smithi156 ceph-mon[129245]: pgmap v992: 97 pgs: 97 active+clean; 695 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.7 MiB/s wr, 610 op/s 2023-07-29T16:36:09.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 229849 35.05 MB/sec execute 233 sec latency 57.378 ms 2023-07-29T16:36:10.303 INFO:tasks.workunit.client.0.smithi031.stdout: 10 230586 35.05 MB/sec execute 234 sec latency 80.904 ms 2023-07-29T16:36:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:10 smithi039 ceph-mon[128858]: pgmap v993: 97 pgs: 97 active+clean; 694 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.5 MiB/s wr, 562 op/s 2023-07-29T16:36:11.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:10 smithi156 ceph-mon[129245]: pgmap v993: 97 pgs: 97 active+clean; 694 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.5 MiB/s wr, 562 op/s 2023-07-29T16:36:11.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:10 smithi031 ceph-mon[95504]: pgmap v993: 97 pgs: 97 active+clean; 694 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.5 MiB/s wr, 562 op/s 2023-07-29T16:36:11.304 INFO:tasks.workunit.client.0.smithi031.stdout: 10 231237 35.08 MB/sec execute 235 sec latency 128.560 ms 2023-07-29T16:36:12.304 INFO:tasks.workunit.client.0.smithi031.stdout: 10 231954 35.11 MB/sec execute 236 sec latency 70.281 ms 2023-07-29T16:36:13.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:12 smithi039 ceph-mon[128858]: pgmap v994: 97 pgs: 97 active+clean; 693 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.6 MiB/s wr, 521 op/s 2023-07-29T16:36:13.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:12 smithi156 ceph-mon[129245]: pgmap v994: 97 pgs: 97 active+clean; 693 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.6 MiB/s wr, 521 op/s 2023-07-29T16:36:13.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:12 smithi031 ceph-mon[95504]: pgmap v994: 97 pgs: 97 active+clean; 693 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.6 MiB/s wr, 521 op/s 2023-07-29T16:36:13.304 INFO:tasks.workunit.client.0.smithi031.stdout: 10 232545 35.10 MB/sec execute 237 sec latency 103.768 ms 2023-07-29T16:36:14.031 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:13 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:36:14.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:13 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:36:14.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:13 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:36:14.311 INFO:tasks.workunit.client.0.smithi031.stdout: 10 232938 35.00 MB/sec execute 238 sec latency 379.776 ms 2023-07-29T16:36:15.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:14 smithi039 ceph-mon[128858]: pgmap v995: 97 pgs: 97 active+clean; 702 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.2 MiB/s wr, 614 op/s 2023-07-29T16:36:15.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:14 smithi156 ceph-mon[129245]: pgmap v995: 97 pgs: 97 active+clean; 702 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.2 MiB/s wr, 614 op/s 2023-07-29T16:36:15.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:14 smithi031 ceph-mon[95504]: pgmap v995: 97 pgs: 97 active+clean; 702 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.2 MiB/s wr, 614 op/s 2023-07-29T16:36:15.311 INFO:tasks.workunit.client.0.smithi031.stdout: 10 232952 34.85 MB/sec execute 239 sec latency 1211.068 ms 2023-07-29T16:36:16.314 INFO:tasks.workunit.client.0.smithi031.stdout: 10 233473 34.85 MB/sec execute 240 sec latency 457.017 ms 2023-07-29T16:36:16.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:16.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:16 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:16.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:16 smithi039 ceph-mon[128858]: pgmap v996: 97 pgs: 97 active+clean; 743 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.6 MiB/s wr, 648 op/s 2023-07-29T16:36:16.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:16.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:16 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:16.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:16 smithi156 ceph-mon[129245]: pgmap v996: 97 pgs: 97 active+clean; 743 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.6 MiB/s wr, 648 op/s 2023-07-29T16:36:16.695 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:16.695 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:16 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:16.695 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:16 smithi031 ceph-mon[95504]: pgmap v996: 97 pgs: 97 active+clean; 743 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 7.6 MiB/s wr, 648 op/s 2023-07-29T16:36:17.314 INFO:tasks.workunit.client.0.smithi031.stdout: 10 233881 34.78 MB/sec execute 241 sec latency 152.834 ms 2023-07-29T16:36:18.313 INFO:tasks.workunit.client.0.smithi031.stdout: 10 234541 34.79 MB/sec execute 242 sec latency 94.346 ms 2023-07-29T16:36:18.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:18 smithi031 ceph-mon[95504]: pgmap v997: 97 pgs: 97 active+clean; 731 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 9.4 MiB/s wr, 631 op/s 2023-07-29T16:36:18.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:18 smithi039 ceph-mon[128858]: pgmap v997: 97 pgs: 97 active+clean; 731 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 9.4 MiB/s wr, 631 op/s 2023-07-29T16:36:18.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:18 smithi156 ceph-mon[129245]: pgmap v997: 97 pgs: 97 active+clean; 731 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 9.4 MiB/s wr, 631 op/s 2023-07-29T16:36:19.313 INFO:tasks.workunit.client.0.smithi031.stdout: 10 235252 34.82 MB/sec execute 243 sec latency 246.292 ms 2023-07-29T16:36:20.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.125+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2260564512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 msgr2=0x7f68b4105330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.125+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2260564512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 0x7f68b4105330 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009990 tx=0x7f68a4009ca0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.129+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2260564512 shutdown_connections 2023-07-29T16:36:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.129+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2260564512 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68b4108190 0x7f68b410a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.129+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2260564512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 0x7f68b4105330 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.129+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2260564512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b4107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.129+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2260564512 >> 172.21.15.31:0/2260564512 conn(0x7f68b40fa840 msgr2=0x7f68b40fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:20.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.129+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2260564512 shutdown_connections 2023-07-29T16:36:20.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.130+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2260564512 wait complete. 2023-07-29T16:36:20.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.131+0000 7f68bc5ff700 1 Processor -- start 2023-07-29T16:36:20.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.131+0000 7f68bc5ff700 1 -- start start 2023-07-29T16:36:20.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68bc5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 0x7f68b406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68bc5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b406f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68bc5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68b4108190 0x7f68b406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68bc5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4070540 con 0x7f68b4068d90 2023-07-29T16:36:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68bc5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40706b0 con 0x7f68b4105870 2023-07-29T16:36:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68bc5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4070870 con 0x7f68b4108190 2023-07-29T16:36:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68ba39b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 0x7f68b406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b406f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35872/0 (socket says 172.21.15.31:35872) 2023-07-29T16:36:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.132+0000 7f68b9b9a700 1 -- 172.21.15.31:0/2358663380 learned_addr learned my addr 172.21.15.31:0/2358663380 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.133+0000 7f68bab9c700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68b4108190 0x7f68b406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.137+0000 7f68b9b9a700 1 -- 172.21.15.31:0/2358663380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68b4108190 msgr2=0x7f68b406fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.137+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68b4108190 0x7f68b406fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.137+0000 7f68b9b9a700 1 -- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 msgr2=0x7f68b406ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.137+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 0x7f68b406ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.137+0000 7f68b9b9a700 1 -- 172.21.15.31:0/2358663380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a40095f0 con 0x7f68b4105870 2023-07-29T16:36:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.138+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b406f010 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00ba50 tx=0x7f68ac00bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.139+0000 7f68ab7fe700 1 -- 172.21.15.31:0/2358663380 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00c9b0 con 0x7f68b4105870 2023-07-29T16:36:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.139+0000 7f68ab7fe700 1 -- 172.21.15.31:0/2358663380 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68ac014440 con 0x7f68b4105870 2023-07-29T16:36:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.139+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4070b50 con 0x7f68b4105870 2023-07-29T16:36:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.139+0000 7f68ab7fe700 1 -- 172.21.15.31:0/2358663380 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac0075d0 con 0x7f68b4105870 2023-07-29T16:36:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.139+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b4071380 con 0x7f68b4105870 2023-07-29T16:36:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.147+0000 7f68ab7fe700 1 -- 172.21.15.31:0/2358663380 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f68ac007730 con 0x7f68b4105870 2023-07-29T16:36:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.159+0000 7f68ab7fe700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68a00a0990 0x7f68a00a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.159+0000 7f68ab7fe700 1 -- 172.21.15.31:0/2358663380 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f68ac006e40 con 0x7f68b4105870 2023-07-29T16:36:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.161+0000 7f68cff0db80 1 Processor -- start 2023-07-29T16:36:20.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.162+0000 7f68cff0db80 1 -- start start 2023-07-29T16:36:20.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f12546b0 0x5585f13f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f13f8df0 0x5585f13f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f13fa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68cff0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f13509b0 con 0x5585f12546b0 2023-07-29T16:36:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68cff0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f134e370 con 0x5585f13f9cf0 2023-07-29T16:36:20.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68cff0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f1349c60 con 0x5585f13f8df0 2023-07-29T16:36:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.163+0000 7f68bab9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f13fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f13fa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35888/0 (socket says 172.21.15.31:35888) 2023-07-29T16:36:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 -- 172.21.15.31:0/2032283987 learned_addr learned my addr 172.21.15.31:0/2032283987 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f13f8df0 0x5585f13f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 -- 172.21.15.31:0/2032283987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f13f8df0 msgr2=0x5585f13f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f13f8df0 0x5585f13f97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 -- 172.21.15.31:0/2032283987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f12546b0 msgr2=0x5585f13f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f12546b0 0x5585f13f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.164+0000 7f68bab9c700 1 -- 172.21.15.31:0/2032283987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585f1177d80 con 0x5585f13f9cf0 2023-07-29T16:36:20.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.165+0000 7f68bab9c700 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f13fa160 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f68b000a4d0 tx=0x7f68b0009650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fbde8a67af7c5a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.166+0000 7f68a8bfe700 1 -- 172.21.15.31:0/2032283987 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0010040 con 0x5585f13f9cf0 2023-07-29T16:36:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.166+0000 7f68a8bfe700 1 -- 172.21.15.31:0/2032283987 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68b0013c50 con 0x5585f13f9cf0 2023-07-29T16:36:20.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.166+0000 7f68a8bfe700 1 -- 172.21.15.31:0/2032283987 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0014d20 con 0x5585f13f9cf0 2023-07-29T16:36:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.167+0000 7f68cff0db80 1 -- 172.21.15.31:0/2032283987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 msgr2=0x5585f13fa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.167+0000 7f68cff0db80 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f13fa160 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f68b000a4d0 tx=0x7f68b0009650 comp rx=0 tx=0).stop 2023-07-29T16:36:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68ba39b700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68a00a0990 0x7f68a00a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 -- 172.21.15.31:0/2032283987 shutdown_connections 2023-07-29T16:36:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f13fa160 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f12546b0 0x5585f13f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 --2- 172.21.15.31:0/2032283987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f13f8df0 0x5585f13f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 -- 172.21.15.31:0/2032283987 >> 172.21.15.31:0/2032283987 conn(0x5585f13edfd0 msgr2=0x5585f13f3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 -- 172.21.15.31:0/2032283987 shutdown_connections 2023-07-29T16:36:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.168+0000 7f68cff0db80 1 -- 172.21.15.31:0/2032283987 wait complete. 2023-07-29T16:36:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.169+0000 7f68cff0db80 1 -- start start 2023-07-29T16:36:20.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.169+0000 7f68cff0db80 1 Processor -- start 2023-07-29T16:36:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1401ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1402420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1409460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f12e64b0 con 0x5585f13f9cf0 2023-07-29T16:36:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f12e6150 con 0x5585f13f8df0 2023-07-29T16:36:20.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f11cd8c0 con 0x5585f12546b0 2023-07-29T16:36:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.170+0000 7f68cff0db80 10 client.0 _ll_register_callbacks cb 0x5585f12ef5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.171+0000 7f68ba39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1401ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68ba39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1401ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58684/0 (socket says 172.21.15.31:58684) 2023-07-29T16:36:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68ba39b700 1 -- 172.21.15.31:0/1139324290 learned_addr learned my addr 172.21.15.31:0/1139324290 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68bab9c700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1409460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1402420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68bab9c700 1 -- 172.21.15.31:0/1139324290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 msgr2=0x5585f1401ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68bab9c700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1401ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68bab9c700 1 -- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 msgr2=0x5585f1402420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68bab9c700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1402420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68bab9c700 1 -- 172.21.15.31:0/1139324290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b0009d00 con 0x5585f13f9cf0 2023-07-29T16:36:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.173+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1402420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:36:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.174+0000 7f68bab9c700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1409460 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f68b00093b0 tx=0x7f68b000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d11b2fff5b92aa8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:20.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.174+0000 7f689dffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:36:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.175+0000 7f689dffb700 1 -- 172.21.15.31:0/1139324290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b0010040 con 0x5585f13f9cf0 2023-07-29T16:36:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.175+0000 7f689dffb700 1 -- 172.21.15.31:0/1139324290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f68b0013590 con 0x5585f13f9cf0 2023-07-29T16:36:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.175+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585f1374cb0 con 0x5585f13f9cf0 2023-07-29T16:36:20.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.175+0000 7f689dffb700 1 -- 172.21.15.31:0/1139324290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68b0014e70 con 0x5585f13f9cf0 2023-07-29T16:36:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.175+0000 7f68cff0db80 10 client.20190 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.175+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5585f1160c00 con 0x5585f13f9cf0 2023-07-29T16:36:20.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f689dffb700 1 -- 172.21.15.31:0/1139324290 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5585f1374cb0 con 0x5585f13f9cf0 2023-07-29T16:36:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f689dffb700 1 -- 172.21.15.31:0/1139324290 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f68b0008030 con 0x5585f13f9cf0 2023-07-29T16:36:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f68cff0db80 10 client.20190 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f68cff0db80 10 client.20190 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f68cff0db80 20 client.20190 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f68cff0db80 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f141ebe0 0x5585f1420ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f68cff0db80 4 client.20190 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:36:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.176+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585f13c7d10 con 0x5585f141ebe0 2023-07-29T16:36:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.177+0000 7f68ba39b700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f141ebe0 0x5585f1420ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.177+0000 7f68ba39b700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68a00a0990 0x7f68a00a2e40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009fd0 tx=0x7f68a4009ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.180+0000 7f68ba39b700 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f141ebe0 0x5585f1420ff0 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0ab84ba36823dc3 server_cookie=e459f1240868dcc6 in_seq=0 out_seq=0 2023-07-29T16:36:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.180+0000 7f689dffb700 10 client.20190 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.184+0000 7f689dffb700 1 -- 172.21.15.31:0/1139324290 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5585f13c7d10 con 0x5585f141ebe0 2023-07-29T16:36:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.184+0000 7f689dffb700 10 client.20190 handle_command_reply: tid=0 2023-07-29T16:36:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.184+0000 7f68cff0db80 1 client.20190 shutdown 2023-07-29T16:36:20.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 msgr2=0x5585f1409460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1409460 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f68b00093b0 tx=0x7f68b000beb0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 shutdown_connections 2023-07-29T16:36:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f141ebe0 0x5585f1420ff0 crc :-1 s=READY pgs=473 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1409460 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1401ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 --2- 172.21.15.31:0/1139324290 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1402420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 >> 172.21.15.31:0/1139324290 conn(0x5585f13edfd0 msgr2=0x5585f13f2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f689dffb700 0 client.20190 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 shutdown_connections 2023-07-29T16:36:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.185+0000 7f68cff0db80 1 -- 172.21.15.31:0/1139324290 wait complete. 2023-07-29T16:36:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.186+0000 7f68cff0db80 20 client.20190 trim_cache size 0 max 16384 2023-07-29T16:36:20.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.213+0000 7f68cff0db80 1 Processor -- start 2023-07-29T16:36:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.213+0000 7f68cff0db80 1 -- start start 2023-07-29T16:36:20.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f141ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f8df0 0x5585f1409420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f1420bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68cff0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f13f41d0 con 0x5585f13f8df0 2023-07-29T16:36:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68cff0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f1403060 con 0x5585f13f9cf0 2023-07-29T16:36:20.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68cff0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f13f3f80 con 0x5585f12546b0 2023-07-29T16:36:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68ba39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f141ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68ba39b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f141ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:58688/0 (socket says 172.21.15.31:58688) 2023-07-29T16:36:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68ba39b700 1 -- 172.21.15.31:0/4216071995 learned_addr learned my addr 172.21.15.31:0/4216071995 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:20.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.214+0000 7f68bab9c700 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f1420bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.215+0000 7f68ba39b700 1 -- 172.21.15.31:0/4216071995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 msgr2=0x5585f1420bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.215+0000 7f68ba39b700 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f1420bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.215+0000 7f68ba39b700 1 -- 172.21.15.31:0/4216071995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f8df0 msgr2=0x5585f1409420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.215+0000 7f68ba39b700 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f8df0 0x5585f1409420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.215+0000 7f68ba39b700 1 -- 172.21.15.31:0/4216071995 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585f119e1f0 con 0x5585f12546b0 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.215+0000 7f68ba39b700 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f141ad00 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f68a400bc60 tx=0x7f68a4026000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcaba01c1c8e5095 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.216+0000 7f68a93ff700 1 -- 172.21.15.31:0/4216071995 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a402e040 con 0x5585f12546b0 2023-07-29T16:36:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.216+0000 7f68a93ff700 1 -- 172.21.15.31:0/4216071995 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68a4030d90 con 0x5585f12546b0 2023-07-29T16:36:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.216+0000 7f68a93ff700 1 -- 172.21.15.31:0/4216071995 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a401d070 con 0x5585f12546b0 2023-07-29T16:36:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 -- 172.21.15.31:0/4216071995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 msgr2=0x5585f141ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f141ad00 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f68a400bc60 tx=0x7f68a4026000 comp rx=0 tx=0).stop 2023-07-29T16:36:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 -- 172.21.15.31:0/4216071995 shutdown_connections 2023-07-29T16:36:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f9cf0 0x5585f1420bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f141ad00 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 --2- 172.21.15.31:0/4216071995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f8df0 0x5585f1409420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 -- 172.21.15.31:0/4216071995 >> 172.21.15.31:0/4216071995 conn(0x5585f13edfd0 msgr2=0x5585f13f2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 -- 172.21.15.31:0/4216071995 shutdown_connections 2023-07-29T16:36:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.217+0000 7f68cff0db80 1 -- 172.21.15.31:0/4216071995 wait complete. 2023-07-29T16:36:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.218+0000 7f68cff0db80 1 -- start start 2023-07-29T16:36:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.218+0000 7f68cff0db80 1 Processor -- start 2023-07-29T16:36:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.218+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1405540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.218+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1405a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.218+0000 7f68cff0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1408aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68cff0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f1408fe0 con 0x5585f13f9cf0 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68cff0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f1409150 con 0x5585f13f8df0 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68cff0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585f1409310 con 0x5585f12546b0 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68cff0db80 10 client.0 _ll_register_callbacks cb 0x5585f12ef5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1405a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68b9b9a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1405a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:35916/0 (socket says 172.21.15.31:35916) 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68b9b9a700 1 -- 172.21.15.31:0/3962862568 learned_addr learned my addr 172.21.15.31:0/3962862568 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.219+0000 7f68ba39b700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1405540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68b9b9a700 1 -- 172.21.15.31:0/3962862568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 msgr2=0x5585f1405540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1405540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68b9b9a700 1 -- 172.21.15.31:0/3962862568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 msgr2=0x5585f1408aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1408aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68b9b9a700 1 -- 172.21.15.31:0/3962862568 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b41010e0 con 0x5585f13f8df0 2023-07-29T16:36:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68ba39b700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1405540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:36:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68b9b9a700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1405a80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0961a0 tx=0x7f68ac004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d0342924e03fe53 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f689dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:36:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f689dffb700 1 -- 172.21.15.31:0/3962862568 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac09ad90 con 0x5585f13f8df0 2023-07-29T16:36:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f689dffb700 1 -- 172.21.15.31:0/3962862568 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f68ac099730 con 0x5585f13f8df0 2023-07-29T16:36:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.220+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68b0013c50 con 0x5585f13f8df0 2023-07-29T16:36:20.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.227+0000 7f689dffb700 1 -- 172.21.15.31:0/3962862568 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68ac09d760 con 0x5585f13f8df0 2023-07-29T16:36:20.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.227+0000 7f68cff0db80 10 client.30434 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:20.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.227+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5585f1176250 con 0x5585f13f8df0 2023-07-29T16:36:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f689dffb700 1 -- 172.21.15.31:0/3962862568 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68ac0a5080 con 0x5585f13f8df0 2023-07-29T16:36:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f689dffb700 1 -- 172.21.15.31:0/3962862568 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f68ac098690 con 0x5585f13f8df0 2023-07-29T16:36:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f68cff0db80 10 client.30434 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f68cff0db80 10 client.30434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f68cff0db80 20 client.30434 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f68cff0db80 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f1471a50 0x5585f1473e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f68cff0db80 4 client.30434 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:36:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.228+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5585f1342ce0 con 0x5585f1471a50 2023-07-29T16:36:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.229+0000 7f68ba39b700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f1471a50 0x5585f1473e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.230+0000 7f68ba39b700 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f1471a50 0x5585f1473e60 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbc11dd5381a000f server_cookie=df7aeff98259c1fe in_seq=0 out_seq=0 2023-07-29T16:36:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.230+0000 7f689dffb700 10 client.30434 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:20.313 INFO:tasks.workunit.client.0.smithi031.stdout: 10 235938 34.82 MB/sec execute 244 sec latency 75.959 ms 2023-07-29T16:36:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.564+0000 7f689dffb700 1 -- 172.21.15.31:0/3962862568 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5585f1342ce0 con 0x5585f1471a50 2023-07-29T16:36:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.564+0000 7f689dffb700 10 client.30434 handle_command_reply: tid=0 2023-07-29T16:36:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.564+0000 7f68cff0db80 1 client.30434 shutdown 2023-07-29T16:36:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 msgr2=0x5585f1405a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1405a80 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0961a0 tx=0x7f68ac004ad0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 shutdown_connections 2023-07-29T16:36:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5585f1471a50 0x5585f1473e60 crc :-1 s=READY pgs=474 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5585f13f9cf0 0x5585f1408aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5585f12546b0 0x5585f1405540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 --2- 172.21.15.31:0/3962862568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585f13f8df0 0x5585f1405a80 secure :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0961a0 tx=0x7f68ac004ad0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 >> 172.21.15.31:0/3962862568 conn(0x5585f13edfd0 msgr2=0x5585f1420bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.565+0000 7f689dffb700 0 client.30434 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.567+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 shutdown_connections 2023-07-29T16:36:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.567+0000 7f68cff0db80 1 -- 172.21.15.31:0/3962862568 wait complete. 2023-07-29T16:36:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.567+0000 7f68cff0db80 20 client.30434 trim_cache size 0 max 16384 2023-07-29T16:36:20.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:36:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:36:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:36:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:36:20.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68a00a0990 msgr2=0x7f68a00a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68a00a0990 0x7f68a00a2e40 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f68a4009fd0 tx=0x7f68a4009ee0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 msgr2=0x7f68b406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b406f010 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00ba50 tx=0x7f68ac00bd60 comp rx=0 tx=0).stop 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 shutdown_connections 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f68a00a0990 0x7f68a00a2e40 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f68b4108190 0x7f68b406fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68b4068d90 0x7f68b406ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 --2- 172.21.15.31:0/2358663380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f68b4105870 0x7f68b406f010 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 >> 172.21.15.31:0/2358663380 conn(0x7f68b40fa840 msgr2=0x7f68b41085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 shutdown_connections 2023-07-29T16:36:20.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:20.568+0000 7f68bc5ff700 1 -- 172.21.15.31:0/2358663380 wait complete. 2023-07-29T16:36:21.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:20 smithi039 ceph-mon[128858]: pgmap v998: 97 pgs: 97 active+clean; 731 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 12 MiB/s wr, 651 op/s 2023-07-29T16:36:21.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:21.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:20 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:21.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:20 smithi156 ceph-mon[129245]: pgmap v998: 97 pgs: 97 active+clean; 731 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 12 MiB/s wr, 651 op/s 2023-07-29T16:36:21.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:21.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:20 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:21.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:20 smithi031 ceph-mon[95504]: pgmap v998: 97 pgs: 97 active+clean; 731 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 12 MiB/s wr, 651 op/s 2023-07-29T16:36:21.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:21.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:20 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:21.313 INFO:tasks.workunit.client.0.smithi031.stdout: 10 236554 34.81 MB/sec execute 245 sec latency 181.924 ms 2023-07-29T16:36:22.313 INFO:tasks.workunit.client.0.smithi031.stdout: 10 237207 34.86 MB/sec execute 246 sec latency 84.192 ms 2023-07-29T16:36:23.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:22 smithi039 ceph-mon[128858]: pgmap v999: 97 pgs: 97 active+clean; 692 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 684 op/s 2023-07-29T16:36:23.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:36:23.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:36:23.030 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:22 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:22 smithi156 ceph-mon[129245]: pgmap v999: 97 pgs: 97 active+clean; 692 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 684 op/s 2023-07-29T16:36:23.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:36:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:36:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:22 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:22 smithi031 ceph-mon[95504]: pgmap v999: 97 pgs: 97 active+clean; 692 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 13 MiB/s wr, 684 op/s 2023-07-29T16:36:23.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:36:23.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:36:23.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:22 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:36:23.313 INFO:tasks.workunit.client.0.smithi031.stdout: 10 237678 34.76 MB/sec execute 247 sec latency 92.406 ms 2023-07-29T16:36:24.331 INFO:tasks.workunit.client.0.smithi031.stdout: 10 238383 34.78 MB/sec execute 248 sec latency 68.802 ms 2023-07-29T16:36:25.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:24 smithi156 ceph-mon[129245]: pgmap v1000: 97 pgs: 97 active+clean; 687 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 13 MiB/s wr, 831 op/s 2023-07-29T16:36:25.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:24 smithi031 ceph-mon[95504]: pgmap v1000: 97 pgs: 97 active+clean; 687 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 13 MiB/s wr, 831 op/s 2023-07-29T16:36:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:24 smithi039 ceph-mon[128858]: pgmap v1000: 97 pgs: 97 active+clean; 687 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 13 MiB/s wr, 831 op/s 2023-07-29T16:36:25.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 238722 34.69 MB/sec execute 249 sec latency 516.606 ms 2023-07-29T16:36:26.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 239370 34.71 MB/sec execute 250 sec latency 78.871 ms 2023-07-29T16:36:27.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:26 smithi156 ceph-mon[129245]: pgmap v1001: 97 pgs: 97 active+clean; 698 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 14 MiB/s wr, 806 op/s 2023-07-29T16:36:27.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:26 smithi031 ceph-mon[95504]: pgmap v1001: 97 pgs: 97 active+clean; 698 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 14 MiB/s wr, 806 op/s 2023-07-29T16:36:27.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:26 smithi039 ceph-mon[128858]: pgmap v1001: 97 pgs: 97 active+clean; 698 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 14 MiB/s wr, 806 op/s 2023-07-29T16:36:27.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 240226 34.79 MB/sec execute 251 sec latency 44.015 ms 2023-07-29T16:36:28.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 240684 34.75 MB/sec execute 252 sec latency 89.890 ms 2023-07-29T16:36:28.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:28 smithi039 ceph-mon[128858]: pgmap v1002: 97 pgs: 97 active+clean; 696 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 12 MiB/s wr, 751 op/s 2023-07-29T16:36:28.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:28 smithi156 ceph-mon[129245]: pgmap v1002: 97 pgs: 97 active+clean; 696 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 12 MiB/s wr, 751 op/s 2023-07-29T16:36:28.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:28 smithi031 ceph-mon[95504]: pgmap v1002: 97 pgs: 97 active+clean; 696 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 12 MiB/s wr, 751 op/s 2023-07-29T16:36:29.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 241310 34.73 MB/sec execute 253 sec latency 72.305 ms 2023-07-29T16:36:30.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 242026 34.74 MB/sec execute 254 sec latency 83.971 ms 2023-07-29T16:36:31.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:30 smithi039 ceph-mon[128858]: pgmap v1003: 97 pgs: 97 active+clean; 704 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 11 MiB/s wr, 772 op/s 2023-07-29T16:36:31.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:30 smithi156 ceph-mon[129245]: pgmap v1003: 97 pgs: 97 active+clean; 704 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 11 MiB/s wr, 772 op/s 2023-07-29T16:36:31.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:30 smithi031 ceph-mon[95504]: pgmap v1003: 97 pgs: 97 active+clean; 704 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 11 MiB/s wr, 772 op/s 2023-07-29T16:36:31.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 242689 34.76 MB/sec execute 255 sec latency 63.975 ms 2023-07-29T16:36:32.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 243311 34.71 MB/sec execute 256 sec latency 60.254 ms 2023-07-29T16:36:33.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:32 smithi039 ceph-mon[128858]: pgmap v1004: 97 pgs: 97 active+clean; 689 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.3 MiB/s wr, 717 op/s 2023-07-29T16:36:33.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:32 smithi156 ceph-mon[129245]: pgmap v1004: 97 pgs: 97 active+clean; 689 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.3 MiB/s wr, 717 op/s 2023-07-29T16:36:33.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:32 smithi031 ceph-mon[95504]: pgmap v1004: 97 pgs: 97 active+clean; 689 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.3 MiB/s wr, 717 op/s 2023-07-29T16:36:33.330 INFO:tasks.workunit.client.0.smithi031.stdout: 10 243902 34.71 MB/sec execute 257 sec latency 192.551 ms 2023-07-29T16:36:33.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 -- 172.21.15.31:0/901928786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e241004d0 msgr2=0x7f8e24100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:33.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 --2- 172.21.15.31:0/901928786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e241004d0 0x7f8e24100940 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f8e18009990 tx=0x7f8e18009ca0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 -- 172.21.15.31:0/901928786 shutdown_connections 2023-07-29T16:36:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 --2- 172.21.15.31:0/901928786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e24100e80 0x7f8e24107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 --2- 172.21.15.31:0/901928786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e240ff240 0x7f8e240ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 --2- 172.21.15.31:0/901928786 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e241004d0 0x7f8e24100940 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 -- 172.21.15.31:0/901928786 >> 172.21.15.31:0/901928786 conn(0x7f8e240fa840 msgr2=0x7f8e240fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 -- 172.21.15.31:0/901928786 shutdown_connections 2023-07-29T16:36:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.826+0000 7f8e293ff700 1 -- 172.21.15.31:0/901928786 wait complete. 2023-07-29T16:36:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 Processor -- start 2023-07-29T16:36:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 -- start start 2023-07-29T16:36:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 0x7f8e24193f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e241004d0 0x7f8e24194450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e24100e80 0x7f8e2419c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e2419cbf0 con 0x7f8e241004d0 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e24195320 con 0x7f8e240ff240 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e293ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e241954e0 con 0x7f8e24100e80 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e22ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 0x7f8e24193f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e22ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 0x7f8e24193f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48338/0 (socket says 172.21.15.31:48338) 2023-07-29T16:36:33.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.828+0000 7f8e22ffd700 1 -- 172.21.15.31:0/2000472846 learned_addr learned my addr 172.21.15.31:0/2000472846 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e237fe700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e24100e80 0x7f8e2419c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e227fc700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e241004d0 0x7f8e24194450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e22ffd700 1 -- 172.21.15.31:0/2000472846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e24100e80 msgr2=0x7f8e2419c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e24100e80 0x7f8e2419c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e22ffd700 1 -- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e241004d0 msgr2=0x7f8e24194450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e241004d0 0x7f8e24194450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e22ffd700 1 -- 172.21.15.31:0/2000472846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e180095f0 con 0x7f8e240ff240 2023-07-29T16:36:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.829+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 0x7f8e24193f10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9a0 tx=0x7f8e1400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.830+0000 7f8e0bfff700 1 -- 172.21.15.31:0/2000472846 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14009790 con 0x7f8e240ff240 2023-07-29T16:36:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.830+0000 7f8e0bfff700 1 -- 172.21.15.31:0/2000472846 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e1400cbd0 con 0x7f8e240ff240 2023-07-29T16:36:33.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.830+0000 7f8e0bfff700 1 -- 172.21.15.31:0/2000472846 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14018680 con 0x7f8e240ff240 2023-07-29T16:36:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.830+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e241957c0 con 0x7f8e240ff240 2023-07-29T16:36:33.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.830+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e24195c80 con 0x7f8e240ff240 2023-07-29T16:36:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.835+0000 7f8e0bfff700 1 -- 172.21.15.31:0/2000472846 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f8e1400cd40 con 0x7f8e240ff240 2023-07-29T16:36:33.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.837+0000 7f8e3cdc8b80 1 Processor -- start 2023-07-29T16:36:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.838+0000 7f8e3cdc8b80 1 -- start start 2023-07-29T16:36:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6e988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6e997b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 0x564ea6e9a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6df09b0 con 0x564ea6cf46b0 2023-07-29T16:36:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6dee370 con 0x564ea6e98df0 2023-07-29T16:36:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6de9c60 con 0x564ea6e99cf0 2023-07-29T16:36:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e0bfff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8e0c0a0960 0x7f8e0c0a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.839+0000 7f8e0bfff700 1 -- 172.21.15.31:0/2000472846 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8e14014070 con 0x7f8e240ff240 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 0x564ea6e9a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 0x564ea6e9a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51362/0 (socket says 172.21.15.31:51362) 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 -- 172.21.15.31:0/1652058111 learned_addr learned my addr 172.21.15.31:0/1652058111 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 -- 172.21.15.31:0/1652058111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 msgr2=0x564ea6e997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6e997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6e997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 -- 172.21.15.31:0/1652058111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 msgr2=0x564ea6e988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6e988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.840+0000 7f8e227fc700 1 -- 172.21.15.31:0/1652058111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ea6c17d80 con 0x564ea6e99cf0 2023-07-29T16:36:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.841+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6e997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:36:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.841+0000 7f8e237fe700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8e0c0a0960 0x7f8e0c0a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.842+0000 7f8e227fc700 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 0x564ea6e9a160 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005b40 tx=0x7f8e18005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c352a1b1bed0cc server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.842+0000 7f8e237fe700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8e0c0a0960 0x7f8e0c0a2e10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e220 tx=0x7f8e1c0077c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.842+0000 7f8e097fa700 1 -- 172.21.15.31:0/1652058111 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18005040 con 0x564ea6e99cf0 2023-07-29T16:36:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.842+0000 7f8e097fa700 1 -- 172.21.15.31:0/1652058111 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e1801e040 con 0x564ea6e99cf0 2023-07-29T16:36:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.842+0000 7f8e097fa700 1 -- 172.21.15.31:0/1652058111 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e18004d20 con 0x564ea6e99cf0 2023-07-29T16:36:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.843+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1652058111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 msgr2=0x564ea6e9a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.843+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 0x564ea6e9a160 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8e18005b40 tx=0x7f8e18005c00 comp rx=0 tx=0).stop 2023-07-29T16:36:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1652058111 shutdown_connections 2023-07-29T16:36:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e99cf0 0x564ea6e9a160 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6e988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1652058111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6e997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1652058111 >> 172.21.15.31:0/1652058111 conn(0x564ea6e8dfd0 msgr2=0x564ea6e93300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1652058111 shutdown_connections 2023-07-29T16:36:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.844+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1652058111 wait complete. 2023-07-29T16:36:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.845+0000 7f8e3cdc8b80 1 -- start start 2023-07-29T16:36:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.845+0000 7f8e3cdc8b80 1 Processor -- start 2023-07-29T16:36:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6ea1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6ea9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6d864b0 con 0x564ea6e99cf0 2023-07-29T16:36:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6d86150 con 0x564ea6e98df0 2023-07-29T16:36:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6c6d8c0 con 0x564ea6cf46b0 2023-07-29T16:36:33.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e3cdc8b80 10 client.0 _ll_register_callbacks cb 0x564ea6d8f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e227fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e227fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48346/0 (socket says 172.21.15.31:48346) 2023-07-29T16:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.846+0000 7f8e227fc700 1 -- 172.21.15.31:0/1960637831 learned_addr learned my addr 172.21.15.31:0/1960637831 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:33.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6ea9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6ea1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e237fe700 1 -- 172.21.15.31:0/1960637831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 msgr2=0x564ea6ea1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6ea1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e237fe700 1 -- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 msgr2=0x564ea6ea23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.847+0000 7f8e237fe700 1 -- 172.21.15.31:0/1960637831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e24107f90 con 0x564ea6e99cf0 2023-07-29T16:36:33.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.848+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6ea9400 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c014700 tx=0x7f8e1c004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee88bb12eb706d3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.848+0000 7f8e06ff5700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:36:33.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.849+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1960637831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0136b0 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.849+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1960637831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8e1c013810 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.849+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ea6e14cb0 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.849+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1960637831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e1c0139d0 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.849+0000 7f8e3cdc8b80 10 client.20196 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.849+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x564ea6c00c00 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1960637831 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564ea6e14cb0 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1960637831 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8e1c027070 con 0x564ea6e99cf0 2023-07-29T16:36:33.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e3cdc8b80 10 client.20196 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e3cdc8b80 10 client.20196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e3cdc8b80 20 client.20196 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6eadac0 0x564ea6eafed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e3cdc8b80 4 client.20196 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:36:33.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.850+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ea6e67d10 con 0x564ea6eadac0 2023-07-29T16:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.851+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6eadac0 0x564ea6eafed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.851+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6eadac0 0x564ea6eafed0 crc :-1 s=READY pgs=475 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5886d6aa98c4e4 server_cookie=e09aa6ab9001a899 in_seq=0 out_seq=0 2023-07-29T16:36:33.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.851+0000 7f8e06ff5700 10 client.20196 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.855+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1960637831 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x564ea6e67d10 con 0x564ea6eadac0 2023-07-29T16:36:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.855+0000 7f8e06ff5700 10 client.20196 handle_command_reply: tid=0 2023-07-29T16:36:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.855+0000 7f8e3cdc8b80 1 client.20196 shutdown 2023-07-29T16:36:33.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.856+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 msgr2=0x564ea6ea9400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.856+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6ea9400 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c014700 tx=0x7f8e1c004d50 comp rx=0 tx=0).stop 2023-07-29T16:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.856+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 shutdown_connections 2023-07-29T16:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.856+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6eadac0 0x564ea6eafed0 crc :-1 s=READY pgs=475 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.856+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6ea9400 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.857+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6ea1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.857+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1960637831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.857+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 >> 172.21.15.31:0/1960637831 conn(0x564ea6e8dfd0 msgr2=0x564ea6e92a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.857+0000 7f8e06ff5700 0 client.20196 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.857+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 shutdown_connections 2023-07-29T16:36:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.857+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1960637831 wait complete. 2023-07-29T16:36:33.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.860+0000 7f8e3cdc8b80 20 client.20196 trim_cache size 0 max 16384 2023-07-29T16:36:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.888+0000 7f8e3cdc8b80 1 Processor -- start 2023-07-29T16:36:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 -- start start 2023-07-29T16:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6e92a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6eafad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6ea1330 con 0x564ea6e99cf0 2023-07-29T16:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6ea19f0 con 0x564ea6e98df0 2023-07-29T16:36:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.889+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6e94b60 con 0x564ea6cf46b0 2023-07-29T16:36:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e22ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6e92a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e22ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6e92a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:51384/0 (socket says 172.21.15.31:51384) 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e22ffd700 1 -- 172.21.15.31:0/1002647613 learned_addr learned my addr 172.21.15.31:0/1002647613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e227fc700 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e22ffd700 1 -- 172.21.15.31:0/1002647613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 msgr2=0x564ea6ea93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.890+0000 7f8e22ffd700 1 -- 172.21.15.31:0/1002647613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 msgr2=0x564ea6eafad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.891+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6eafad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.891+0000 7f8e22ffd700 1 -- 172.21.15.31:0/1002647613 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ea6c3e1f0 con 0x564ea6cf46b0 2023-07-29T16:36:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.891+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6e92a40 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e970 tx=0x7f8e14023040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e17baf9578d6ea server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.891+0000 7f8e09ffb700 1 -- 172.21.15.31:0/1002647613 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e140a1aa0 con 0x564ea6cf46b0 2023-07-29T16:36:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.892+0000 7f8e09ffb700 1 -- 172.21.15.31:0/1002647613 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e140979f0 con 0x564ea6cf46b0 2023-07-29T16:36:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.892+0000 7f8e09ffb700 1 -- 172.21.15.31:0/1002647613 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e140a1870 con 0x564ea6cf46b0 2023-07-29T16:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1002647613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 msgr2=0x564ea6e92a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6e92a40 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e970 tx=0x7f8e14023040 comp rx=0 tx=0).stop 2023-07-29T16:36:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1002647613 shutdown_connections 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6e99cf0 0x564ea6eafad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6cf46b0 0x564ea6e92a40 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1002647613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e98df0 0x564ea6ea93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1002647613 >> 172.21.15.31:0/1002647613 conn(0x564ea6e8dfd0 msgr2=0x564ea6ea0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.893+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1002647613 shutdown_connections 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.894+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1002647613 wait complete. 2023-07-29T16:36:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.894+0000 7f8e3cdc8b80 1 -- start start 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 Processor -- start 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6ea8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e98df0 0x564ea6ea5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 0x564ea6ea5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6ea65a0 con 0x564ea6cf46b0 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6ea9560 con 0x564ea6e99cf0 2023-07-29T16:36:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ea6ea96f0 con 0x564ea6e98df0 2023-07-29T16:36:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.895+0000 7f8e3cdc8b80 10 client.0 _ll_register_callbacks cb 0x564ea6d8f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 0x564ea6ea5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e227fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e98df0 0x564ea6ea5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 0x564ea6ea5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:48354/0 (socket says 172.21.15.31:48354) 2023-07-29T16:36:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 -- 172.21.15.31:0/1816936092 learned_addr learned my addr 172.21.15.31:0/1816936092 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6ea8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 -- 172.21.15.31:0/1816936092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e98df0 msgr2=0x564ea6ea5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e98df0 0x564ea6ea5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 -- 172.21.15.31:0/1816936092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 msgr2=0x564ea6ea8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6ea8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e237fe700 1 -- 172.21.15.31:0/1816936092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e24195c80 con 0x564ea6e99cf0 2023-07-29T16:36:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.896+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6ea8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.897+0000 7f8e237fe700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 0x564ea6ea5ec0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c007e60 tx=0x7f8e1c01eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36b3cc8f51a74b64 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.897+0000 7f8e06ff5700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.897+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1816936092 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0283a0 con 0x564ea6e99cf0 2023-07-29T16:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.897+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e1801e040 con 0x564ea6e99cf0 2023-07-29T16:36:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.897+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1816936092 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f8e1c028500 con 0x564ea6e99cf0 2023-07-29T16:36:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.898+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1816936092 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e1c0286c0 con 0x564ea6e99cf0 2023-07-29T16:36:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.902+0000 7f8e3cdc8b80 10 client.30446 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.902+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x564ea6c16250 con 0x564ea6e99cf0 2023-07-29T16:36:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1816936092 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e1801e040 con 0x564ea6e99cf0 2023-07-29T16:36:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1816936092 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f8e1c014730 con 0x564ea6e99cf0 2023-07-29T16:36:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e3cdc8b80 10 client.30446 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e3cdc8b80 10 client.30446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:33.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e3cdc8b80 20 client.30446 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:33.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6f00da0 0x564ea6f031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:33.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e3cdc8b80 4 client.30446 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:36:33.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564ea6de2ce0 con 0x564ea6f00da0 2023-07-29T16:36:33.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.904+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6f00da0 0x564ea6f031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.905+0000 7f8e06ff5700 10 client.30446 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:33.906+0000 7f8e22ffd700 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6f00da0 0x564ea6f031b0 crc :-1 s=READY pgs=476 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a7da54293e37b9b server_cookie=3d5381747562cb61 in_seq=0 out_seq=0 2023-07-29T16:36:34.331 INFO:tasks.workunit.client.0.smithi031.stdout: 10 244399 34.67 MB/sec execute 258 sec latency 369.567 ms 2023-07-29T16:36:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.013+0000 7f8e06ff5700 1 -- 172.21.15.31:0/1816936092 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564ea6de2ce0 con 0x564ea6f00da0 2023-07-29T16:36:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.013+0000 7f8e06ff5700 10 client.30446 handle_command_reply: tid=0 2023-07-29T16:36:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.013+0000 7f8e3cdc8b80 1 client.30446 shutdown 2023-07-29T16:36:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 msgr2=0x564ea6ea5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 0x564ea6ea5ec0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c007e60 tx=0x7f8e1c01eb10 comp rx=0 tx=0).stop 2023-07-29T16:36:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 shutdown_connections 2023-07-29T16:36:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x564ea6f00da0 0x564ea6f031b0 crc :-1 s=READY pgs=476 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564ea6e99cf0 0x564ea6ea5ec0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ea6cf46b0 0x564ea6ea8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 --2- 172.21.15.31:0/1816936092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564ea6e98df0 0x564ea6ea5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 >> 172.21.15.31:0/1816936092 conn(0x564ea6e8dfd0 msgr2=0x564ea6e91fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.014+0000 7f8e06ff5700 0 client.30446 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.015+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 shutdown_connections 2023-07-29T16:36:35.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.019+0000 7f8e3cdc8b80 1 -- 172.21.15.31:0/1816936092 wait complete. 2023-07-29T16:36:35.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.020+0000 7f8e3cdc8b80 20 client.30446 trim_cache size 0 max 16384 2023-07-29T16:36:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:36:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:36:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:36:35.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:36:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.023+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8e0c0a0960 msgr2=0x7f8e0c0a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:35.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.023+0000 7f8e293ff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8e0c0a0960 0x7f8e0c0a2e10 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c00e220 tx=0x7f8e1c0077c0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.023+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 msgr2=0x7f8e24193f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.023+0000 7f8e293ff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 0x7f8e24193f10 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9a0 tx=0x7f8e1400ed60 comp rx=0 tx=0).stop 2023-07-29T16:36:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.024+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 shutdown_connections 2023-07-29T16:36:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.024+0000 7f8e293ff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f8e0c0a0960 0x7f8e0c0a2e10 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.024+0000 7f8e293ff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e24100e80 0x7f8e2419c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.024+0000 7f8e293ff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8e240ff240 0x7f8e24193f10 secure :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400e9a0 tx=0x7f8e1400ed60 comp rx=0 tx=0).stop 2023-07-29T16:36:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.024+0000 7f8e293ff700 1 --2- 172.21.15.31:0/2000472846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e241004d0 0x7f8e24194450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.024+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 >> 172.21.15.31:0/2000472846 conn(0x7f8e240fa840 msgr2=0x7f8e24068220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:35.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:34 smithi039 ceph-mon[128858]: pgmap v1005: 97 pgs: 97 active+clean; 688 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 677 op/s 2023-07-29T16:36:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.025+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 shutdown_connections 2023-07-29T16:36:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:35.025+0000 7f8e293ff700 1 -- 172.21.15.31:0/2000472846 wait complete. 2023-07-29T16:36:35.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:34 smithi156 ceph-mon[129245]: pgmap v1005: 97 pgs: 97 active+clean; 688 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 677 op/s 2023-07-29T16:36:35.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:34 smithi031 ceph-mon[95504]: pgmap v1005: 97 pgs: 97 active+clean; 688 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 8.0 MiB/s wr, 677 op/s 2023-07-29T16:36:35.331 INFO:tasks.workunit.client.0.smithi031.stdout: 10 244616 34.57 MB/sec execute 259 sec latency 799.089 ms 2023-07-29T16:36:36.331 INFO:tasks.workunit.client.0.smithi031.stdout: 10 245115 34.60 MB/sec execute 260 sec latency 71.395 ms 2023-07-29T16:36:37.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:36 smithi156 ceph-mon[129245]: pgmap v1006: 97 pgs: 97 active+clean; 712 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 682 op/s 2023-07-29T16:36:37.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:36 smithi031 ceph-mon[95504]: pgmap v1006: 97 pgs: 97 active+clean; 712 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 682 op/s 2023-07-29T16:36:37.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:36 smithi039 ceph-mon[128858]: pgmap v1006: 97 pgs: 97 active+clean; 712 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 10 MiB/s wr, 682 op/s 2023-07-29T16:36:37.331 INFO:tasks.workunit.client.0.smithi031.stdout: 10 245650 34.59 MB/sec execute 261 sec latency 107.068 ms 2023-07-29T16:36:38.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 246100 34.53 MB/sec execute 262 sec latency 75.900 ms 2023-07-29T16:36:38.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:38 smithi031 ceph-mon[95504]: pgmap v1007: 97 pgs: 97 active+clean; 702 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.3 MiB/s wr, 625 op/s 2023-07-29T16:36:38.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:38 smithi039 ceph-mon[128858]: pgmap v1007: 97 pgs: 97 active+clean; 702 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.3 MiB/s wr, 625 op/s 2023-07-29T16:36:38.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:38 smithi156 ceph-mon[129245]: pgmap v1007: 97 pgs: 97 active+clean; 702 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.3 MiB/s wr, 625 op/s 2023-07-29T16:36:39.331 INFO:tasks.workunit.client.0.smithi031.stdout: 10 246663 34.48 MB/sec execute 263 sec latency 76.165 ms 2023-07-29T16:36:40.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 247370 34.50 MB/sec execute 264 sec latency 79.032 ms 2023-07-29T16:36:41.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:40 smithi039 ceph-mon[128858]: pgmap v1008: 97 pgs: 97 active+clean; 699 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 10 MiB/s wr, 703 op/s 2023-07-29T16:36:41.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:40 smithi156 ceph-mon[129245]: pgmap v1008: 97 pgs: 97 active+clean; 699 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 10 MiB/s wr, 703 op/s 2023-07-29T16:36:41.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:40 smithi031 ceph-mon[95504]: pgmap v1008: 97 pgs: 97 active+clean; 699 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 10 MiB/s wr, 703 op/s 2023-07-29T16:36:41.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 248205 34.53 MB/sec execute 265 sec latency 45.723 ms 2023-07-29T16:36:42.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 249024 34.60 MB/sec execute 266 sec latency 79.427 ms 2023-07-29T16:36:43.031 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:42 smithi039 ceph-mon[128858]: pgmap v1009: 97 pgs: 97 active+clean; 679 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 657 op/s 2023-07-29T16:36:43.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:42 smithi156 ceph-mon[129245]: pgmap v1009: 97 pgs: 97 active+clean; 679 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 657 op/s 2023-07-29T16:36:43.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:42 smithi031 ceph-mon[95504]: pgmap v1009: 97 pgs: 97 active+clean; 679 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 8.1 MiB/s wr, 657 op/s 2023-07-29T16:36:43.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 249649 34.60 MB/sec execute 267 sec latency 73.731 ms 2023-07-29T16:36:44.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 250190 34.60 MB/sec execute 268 sec latency 82.010 ms 2023-07-29T16:36:45.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:44 smithi039 ceph-mon[128858]: pgmap v1010: 97 pgs: 97 active+clean; 683 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.7 MiB/s wr, 732 op/s 2023-07-29T16:36:45.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:44 smithi156 ceph-mon[129245]: pgmap v1010: 97 pgs: 97 active+clean; 683 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.7 MiB/s wr, 732 op/s 2023-07-29T16:36:45.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:44 smithi031 ceph-mon[95504]: pgmap v1010: 97 pgs: 97 active+clean; 683 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 8.7 MiB/s wr, 732 op/s 2023-07-29T16:36:45.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 250592 34.53 MB/sec execute 269 sec latency 201.542 ms 2023-07-29T16:36:46.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 251315 34.51 MB/sec execute 270 sec latency 62.708 ms 2023-07-29T16:36:47.027 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:46 smithi039 ceph-mon[128858]: pgmap v1011: 97 pgs: 97 active+clean; 675 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.6 MiB/s wr, 701 op/s 2023-07-29T16:36:47.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:46 smithi156 ceph-mon[129245]: pgmap v1011: 97 pgs: 97 active+clean; 675 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.6 MiB/s wr, 701 op/s 2023-07-29T16:36:47.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:46 smithi031 ceph-mon[95504]: pgmap v1011: 97 pgs: 97 active+clean; 675 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.6 MiB/s wr, 701 op/s 2023-07-29T16:36:47.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 252202 34.56 MB/sec execute 271 sec latency 53.205 ms 2023-07-29T16:36:48.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 252933 34.60 MB/sec execute 272 sec latency 54.630 ms 2023-07-29T16:36:48.530 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:48 smithi039 ceph-mon[128858]: pgmap v1012: 97 pgs: 97 active+clean; 680 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.0 MiB/s wr, 577 op/s 2023-07-29T16:36:48.578 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:48 smithi156 ceph-mon[129245]: pgmap v1012: 97 pgs: 97 active+clean; 680 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.0 MiB/s wr, 577 op/s 2023-07-29T16:36:48.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:48 smithi031 ceph-mon[95504]: pgmap v1012: 97 pgs: 97 active+clean; 680 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.0 MiB/s wr, 577 op/s 2023-07-29T16:36:49.332 INFO:tasks.workunit.client.0.smithi031.stdout: 10 253621 34.63 MB/sec execute 273 sec latency 74.164 ms 2023-07-29T16:36:49.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 -- 172.21.15.31:0/1247854728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10101570 msgr2=0x7f5e101019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 --2- 172.21.15.31:0/1247854728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10101570 0x7f5e101019c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009990 tx=0x7f5e00009ca0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 -- 172.21.15.31:0/1247854728 shutdown_connections 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 --2- 172.21.15.31:0/1247854728 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e10101f00 0x7f5e101087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 --2- 172.21.15.31:0/1247854728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e10100380 0x7f5e10100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 --2- 172.21.15.31:0/1247854728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10101570 0x7f5e101019c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 -- 172.21.15.31:0/1247854728 >> 172.21.15.31:0/1247854728 conn(0x7f5e100fb9a0 msgr2=0x7f5e100fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:49.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 -- 172.21.15.31:0/1247854728 shutdown_connections 2023-07-29T16:36:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.686+0000 7f5e181ff700 1 -- 172.21.15.31:0/1247854728 wait complete. 2023-07-29T16:36:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.688+0000 7f5e181ff700 1 Processor -- start 2023-07-29T16:36:49.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.688+0000 7f5e181ff700 1 -- start start 2023-07-29T16:36:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e181ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 0x7f5e10198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e181ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e10101570 0x7f5e101988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e181ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e10101f00 0x7f5e10199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e181ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10199d80 con 0x7f5e10100380 2023-07-29T16:36:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e181ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e10199ef0 con 0x7f5e10101570 2023-07-29T16:36:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e181ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1019a0b0 con 0x7f5e10101f00 2023-07-29T16:36:49.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 0x7f5e10198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 0x7f5e10198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47020/0 (socket says 172.21.15.31:47020) 2023-07-29T16:36:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2629724818 learned_addr learned my addr 172.21.15.31:0/2629724818 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:49.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e1579a700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e10101570 0x7f5e101988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e1679c700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e10101f00 0x7f5e10199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2629724818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e10101f00 msgr2=0x7f5e10199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:49.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e10101f00 0x7f5e10199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2629724818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e10101570 msgr2=0x7f5e101988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.689+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e10101570 0x7f5e101988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.690+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2629724818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e000095f0 con 0x7f5e10100380 2023-07-29T16:36:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.690+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 0x7f5e10198360 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00e970 tx=0x7f5e0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.691+0000 7f5e06ffd700 1 -- 172.21.15.31:0/2629724818 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c009790 con 0x7f5e10100380 2023-07-29T16:36:49.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.691+0000 7f5e06ffd700 1 -- 172.21.15.31:0/2629724818 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e0c00eec0 con 0x7f5e10100380 2023-07-29T16:36:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.691+0000 7f5e06ffd700 1 -- 172.21.15.31:0/2629724818 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c003cf0 con 0x7f5e10100380 2023-07-29T16:36:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.691+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1019a390 con 0x7f5e10100380 2023-07-29T16:36:49.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.692+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e101a0b50 con 0x7f5e10100380 2023-07-29T16:36:49.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.697+0000 7f5e06ffd700 1 -- 172.21.15.31:0/2629724818 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5e0c00ccd0 con 0x7f5e10100380 2023-07-29T16:36:49.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.698+0000 7f5e2bbbab80 1 Processor -- start 2023-07-29T16:36:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 -- start start 2023-07-29T16:36:49.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90b926b0 0x563e90d3f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d3f600 0x563e90d3ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d40500 0x563e90d40970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90c8e9b0 con 0x563e90d40500 2023-07-29T16:36:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90c8c370 con 0x563e90d3f600 2023-07-29T16:36:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90c87c60 con 0x563e90b926b0 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e06ffd700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5dfc0a08e0 0x7f5dfc0a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e1679c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90b926b0 0x563e90d3f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e1679c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90b926b0 0x563e90d3f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54980/0 (socket says 172.21.15.31:54980) 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e1679c700 1 -- 172.21.15.31:0/3726468814 learned_addr learned my addr 172.21.15.31:0/3726468814 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d3f600 0x563e90d3ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.699+0000 7f5e1579a700 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d40500 0x563e90d40970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e06ffd700 1 -- 172.21.15.31:0/2629724818 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5e0c014070 con 0x7f5e10100380 2023-07-29T16:36:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e15f9b700 1 -- 172.21.15.31:0/3726468814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90b926b0 msgr2=0x563e90d3f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90b926b0 0x563e90d3f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e15f9b700 1 -- 172.21.15.31:0/3726468814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d40500 msgr2=0x563e90d40970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d40500 0x563e90d40970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e15f9b700 1 -- 172.21.15.31:0/3726468814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e90ab5d80 con 0x563e90d3f600 2023-07-29T16:36:49.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e1679c700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5dfc0a08e0 0x7f5dfc0a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.700+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d3f600 0x563e90d3ffc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c005950 tx=0x7f5e0c095880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e65529cda905fc4d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.701+0000 7f5dfbfff700 1 -- 172.21.15.31:0/3726468814 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0b1ea0 con 0x563e90d3f600 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.701+0000 7f5dfbfff700 1 -- 172.21.15.31:0/3726468814 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e0c0994c0 con 0x563e90d3f600 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.701+0000 7f5dfbfff700 1 -- 172.21.15.31:0/3726468814 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0989b0 con 0x563e90d3f600 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e1679c700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5dfc0a08e0 0x7f5dfc0a2d90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009510 tx=0x7f5e08004b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3726468814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d3f600 msgr2=0x563e90d3ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d3f600 0x563e90d3ffc0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c005950 tx=0x7f5e0c095880 comp rx=0 tx=0).stop 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3726468814 shutdown_connections 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d40500 0x563e90d40970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90b926b0 0x563e90d3f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3726468814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d3f600 0x563e90d3ffc0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3726468814 >> 172.21.15.31:0/3726468814 conn(0x563e90d2bfd0 msgr2=0x563e90d31300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3726468814 shutdown_connections 2023-07-29T16:36:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.702+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3726468814 wait complete. 2023-07-29T16:36:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.703+0000 7f5e2bbbab80 1 -- start start 2023-07-29T16:36:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.703+0000 7f5e2bbbab80 1 Processor -- start 2023-07-29T16:36:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90b926b0 0x563e90d48740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d3f600 0x563e90d48c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 0x563e90d4fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90c244b0 con 0x563e90b926b0 2023-07-29T16:36:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90c24150 con 0x563e90d40500 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90b0b8c0 con 0x563e90d3f600 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.704+0000 7f5e2bbbab80 10 client.0 _ll_register_callbacks cb 0x563e90c2d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 0x563e90d4fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 0x563e90d4fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50244/0 (socket says 172.21.15.31:50244) 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 -- 172.21.15.31:0/3591431268 learned_addr learned my addr 172.21.15.31:0/3591431268 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90b926b0 0x563e90d48740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 -- 172.21.15.31:0/3591431268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d3f600 msgr2=0x563e90d48c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:36:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d3f600 0x563e90d48c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 -- 172.21.15.31:0/3591431268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90b926b0 msgr2=0x563e90d48740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90b926b0 0x563e90d48740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.705+0000 7f5e1679c700 1 -- 172.21.15.31:0/3591431268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e101a0b50 con 0x563e90d40500 2023-07-29T16:36:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.706+0000 7f5e1679c700 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 0x563e90d4fcc0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5e08017de0 tx=0x7f5e08004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60fe9846518581f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.706+0000 7f5df97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:36:49.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.706+0000 7f5df97fa700 1 -- 172.21.15.31:0/3591431268 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0800e040 con 0x563e90d40500 2023-07-29T16:36:49.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.706+0000 7f5df97fa700 1 -- 172.21.15.31:0/3591431268 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5e0801e040 con 0x563e90d40500 2023-07-29T16:36:49.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.706+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563e90cb2cb0 con 0x563e90d40500 2023-07-29T16:36:49.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5df97fa700 1 -- 172.21.15.31:0/3591431268 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e08017ec0 con 0x563e90d40500 2023-07-29T16:36:49.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 10 client.30458 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:49.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x563e90a9ec00 con 0x563e90d40500 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5df97fa700 1 -- 172.21.15.31:0/3591431268 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563e90cb2cb0 con 0x563e90d40500 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5df97fa700 1 -- 172.21.15.31:0/3591431268 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5e08023c20 con 0x563e90d40500 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 10 client.30458 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 10 client.30458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 20 client.30458 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90d545e0 0x563e90d569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 4 client.30458 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:36:49.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.707+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e90d05d10 con 0x563e90d545e0 2023-07-29T16:36:49.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.708+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90d545e0 0x563e90d569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.709+0000 7f5df97fa700 10 client.30458 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:49.715 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.709+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90d545e0 0x563e90d569f0 crc :-1 s=READY pgs=477 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa299bfffaddd3bc server_cookie=33906cc7cf6a8201 in_seq=0 out_seq=0 2023-07-29T16:36:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.714+0000 7f5df97fa700 1 -- 172.21.15.31:0/3591431268 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x563e90d05d10 con 0x563e90d545e0 2023-07-29T16:36:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.714+0000 7f5df97fa700 10 client.30458 handle_command_reply: tid=0 2023-07-29T16:36:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.714+0000 7f5e2bbbab80 1 client.30458 shutdown 2023-07-29T16:36:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 msgr2=0x563e90d4fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 0x563e90d4fcc0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f5e08017de0 tx=0x7f5e08004ad0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 shutdown_connections 2023-07-29T16:36:49.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90d545e0 0x563e90d569f0 crc :-1 s=READY pgs=477 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90d40500 0x563e90d4fcc0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90b926b0 0x563e90d48740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/3591431268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d3f600 0x563e90d48c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 >> 172.21.15.31:0/3591431268 conn(0x563e90d2bfd0 msgr2=0x563e90d312d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5df97fa700 0 client.30458 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 shutdown_connections 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/3591431268 wait complete. 2023-07-29T16:36:49.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.715+0000 7f5e2bbbab80 20 client.30458 trim_cache size 0 max 16384 2023-07-29T16:36:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 Processor -- start 2023-07-29T16:36:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 -- start start 2023-07-29T16:36:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d48740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d48c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90d32040 con 0x563e90d3f600 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90d30620 con 0x563e90b926b0 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.734+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90d43f00 con 0x563e90d40500 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e1579a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d48740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50250/0 (socket says 172.21.15.31:50250) 2023-07-29T16:36:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 -- 172.21.15.31:0/234547042 learned_addr learned my addr 172.21.15.31:0/234547042 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e1679c700 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d48c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 -- 172.21.15.31:0/234547042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 msgr2=0x563e90d48c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d48c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 -- 172.21.15.31:0/234547042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 msgr2=0x563e90d48740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d48740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 -- 172.21.15.31:0/234547042 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e90adc1f0 con 0x563e90b926b0 2023-07-29T16:36:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.735+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4dc70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c0991c0 tx=0x7f5e0c003780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcb7820634985c45 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e04ff9700 1 -- 172.21.15.31:0/234547042 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0c8070 con 0x563e90b926b0 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e04ff9700 1 -- 172.21.15.31:0/234547042 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e0c09d570 con 0x563e90b926b0 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e04ff9700 1 -- 172.21.15.31:0/234547042 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0b1ea0 con 0x563e90b926b0 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/234547042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 msgr2=0x563e90d4dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4dc70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c0991c0 tx=0x7f5e0c003780 comp rx=0 tx=0).stop 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/234547042 shutdown_connections 2023-07-29T16:36:49.742 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d48c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4dc70 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.736+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/234547042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d48740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.737+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/234547042 >> 172.21.15.31:0/234547042 conn(0x563e90d2bfd0 msgr2=0x563e90d47420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.737+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/234547042 shutdown_connections 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.737+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/234547042 wait complete. 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.737+0000 7f5e2bbbab80 1 -- start start 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.737+0000 7f5e2bbbab80 1 Processor -- start 2023-07-29T16:36:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d4c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d4d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90d4d6e0 con 0x563e90d3f600 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90d4d850 con 0x563e90b926b0 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e90d4da10 con 0x563e90d40500 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e2bbbab80 10 client.0 _ll_register_callbacks cb 0x563e90c2d5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:36:49.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e15f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e15f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50256/0 (socket says 172.21.15.31:50256) 2023-07-29T16:36:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e1679c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d4d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2471312813 learned_addr learned my addr 172.21.15.31:0/2471312813 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:36:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.738+0000 7f5e1579a700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d4c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2471312813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 msgr2=0x563e90d4d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d4d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2471312813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 msgr2=0x563e90d4c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d4c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e15f9b700 1 -- 172.21.15.31:0/2471312813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e90ab5d80 con 0x563e90b926b0 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e1579a700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d4c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:36:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e15f9b700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4bc50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c099450 tx=0x7f5e0c00c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2b45a42f2562b75 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5df97fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5df97fa700 1 -- 172.21.15.31:0/2471312813 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0b1ea0 con 0x563e90b926b0 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5df97fa700 1 -- 172.21.15.31:0/2471312813 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5e0c0a1910 con 0x563e90b926b0 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e0c0994c0 con 0x563e90b926b0 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5df97fa700 1 -- 172.21.15.31:0/2471312813 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e0c0c8070 con 0x563e90b926b0 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e2bbbab80 10 client.30464 fetch_fsmap learned FSMap version 43 2023-07-29T16:36:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.739+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x563e90ab4250 con 0x563e90b926b0 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5df97fa700 1 -- 172.21.15.31:0/2471312813 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e0c0994c0 con 0x563e90b926b0 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5df97fa700 1 -- 172.21.15.31:0/2471312813 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5e0c0c9d20 con 0x563e90b926b0 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e2bbbab80 10 client.30464 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e2bbbab80 10 client.30464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e2bbbab80 20 client.30464 populate_metadata read hostname 'smithi031' 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90da6dc0 0x563e90da91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:36:49.749 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e2bbbab80 4 client.30464 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:36:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563e90c80ce0 con 0x563e90da6dc0 2023-07-29T16:36:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.740+0000 7f5e1579a700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90da6dc0 0x563e90da91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:36:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.741+0000 7f5e1579a700 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90da6dc0 0x563e90da91d0 crc :-1 s=READY pgs=478 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=154fb576f2e6b0e9 server_cookie=9289214687b0279 in_seq=0 out_seq=0 2023-07-29T16:36:49.750 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:49.741+0000 7f5df97fa700 10 client.30464 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:50.334 INFO:tasks.workunit.client.0.smithi031.stdout: 10 253990 34.56 MB/sec execute 274 sec latency 469.025 ms 2023-07-29T16:36:51.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:50 smithi156 ceph-mon[129245]: pgmap v1013: 97 pgs: 97 active+clean; 686 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 5.3 MiB/s wr, 564 op/s 2023-07-29T16:36:51.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:50 smithi031 ceph-mon[95504]: pgmap v1013: 97 pgs: 97 active+clean; 686 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 5.3 MiB/s wr, 564 op/s 2023-07-29T16:36:51.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:50 smithi039 ceph-mon[128858]: pgmap v1013: 97 pgs: 97 active+clean; 686 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 5.3 MiB/s wr, 564 op/s 2023-07-29T16:36:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:36:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:36:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:36:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:36:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.295+0000 7f5df97fa700 1 -- 172.21.15.31:0/2471312813 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563e90c80ce0 con 0x563e90da6dc0 2023-07-29T16:36:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.295+0000 7f5df97fa700 10 client.30464 handle_command_reply: tid=0 2023-07-29T16:36:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.295+0000 7f5e2bbbab80 1 client.30464 shutdown 2023-07-29T16:36:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 msgr2=0x563e90d4bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4bc50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c099450 tx=0x7f5e0c00c750 comp rx=0 tx=0).stop 2023-07-29T16:36:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 shutdown_connections 2023-07-29T16:36:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x563e90da6dc0 0x563e90da91d0 crc :-1 s=READY pgs=478 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563e90d40500 0x563e90d4d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563e90b926b0 0x563e90d4bc50 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 --2- 172.21.15.31:0/2471312813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e90d3f600 0x563e90d4c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5df97fa700 0 client.30464 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 >> 172.21.15.31:0/2471312813 conn(0x563e90d2bfd0 msgr2=0x563e90d46860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 shutdown_connections 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 1 -- 172.21.15.31:0/2471312813 wait complete. 2023-07-29T16:36:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.296+0000 7f5e2bbbab80 20 client.30464 trim_cache size 0 max 16384 2023-07-29T16:36:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5dfc0a08e0 msgr2=0x7f5dfc0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5dfc0a08e0 0x7f5dfc0a2d90 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009510 tx=0x7f5e08004b40 comp rx=0 tx=0).stop 2023-07-29T16:36:51.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 msgr2=0x7f5e10198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 0x7f5e10198360 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00e970 tx=0x7f5e0c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 shutdown_connections 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5dfc0a08e0 0x7f5dfc0a2d90 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e10101f00 0x7f5e10199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5e10100380 0x7f5e10198360 secure :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00e970 tx=0x7f5e0c00ec80 comp rx=0 tx=0).stop 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 --2- 172.21.15.31:0/2629724818 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5e10101570 0x7f5e101988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:36:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.303+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 >> 172.21.15.31:0/2629724818 conn(0x7f5e100fb9a0 msgr2=0x7f5e100fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:36:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.313+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 shutdown_connections 2023-07-29T16:36:51.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:36:51.318+0000 7f5e181ff700 1 -- 172.21.15.31:0/2629724818 wait complete. 2023-07-29T16:36:51.335 INFO:tasks.workunit.client.0.smithi031.stdout: 10 254069 34.46 MB/sec execute 275 sec latency 739.426 ms 2023-07-29T16:36:52.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 254504 34.41 MB/sec execute 276 sec latency 197.021 ms 2023-07-29T16:36:53.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:52 smithi156 ceph-mon[129245]: pgmap v1014: 97 pgs: 97 active+clean; 702 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2023-07-29T16:36:53.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:52 smithi031 ceph-mon[95504]: pgmap v1014: 97 pgs: 97 active+clean; 702 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2023-07-29T16:36:53.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:52 smithi039 ceph-mon[128858]: pgmap v1014: 97 pgs: 97 active+clean; 702 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2023-07-29T16:36:53.335 INFO:tasks.workunit.client.0.smithi031.stdout: 10 255045 34.39 MB/sec execute 277 sec latency 155.126 ms 2023-07-29T16:36:54.335 INFO:tasks.workunit.client.0.smithi031.stdout: 10 255702 34.39 MB/sec execute 278 sec latency 55.225 ms 2023-07-29T16:36:55.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:54 smithi156 ceph-mon[129245]: pgmap v1015: 97 pgs: 97 active+clean; 699 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.9 MiB/s wr, 703 op/s 2023-07-29T16:36:55.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:54 smithi031 ceph-mon[95504]: pgmap v1015: 97 pgs: 97 active+clean; 699 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.9 MiB/s wr, 703 op/s 2023-07-29T16:36:55.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:54 smithi039 ceph-mon[128858]: pgmap v1015: 97 pgs: 97 active+clean; 699 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 8.9 MiB/s wr, 703 op/s 2023-07-29T16:36:55.335 INFO:tasks.workunit.client.0.smithi031.stdout: 10 256386 34.39 MB/sec execute 279 sec latency 73.256 ms 2023-07-29T16:36:56.335 INFO:tasks.workunit.client.0.smithi031.stdout: 10 257065 34.42 MB/sec execute 280 sec latency 78.018 ms 2023-07-29T16:36:57.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:56 smithi156 ceph-mon[129245]: pgmap v1016: 97 pgs: 97 active+clean; 689 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.6 MiB/s wr, 655 op/s 2023-07-29T16:36:57.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:56 smithi031 ceph-mon[95504]: pgmap v1016: 97 pgs: 97 active+clean; 689 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.6 MiB/s wr, 655 op/s 2023-07-29T16:36:57.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:56 smithi039 ceph-mon[128858]: pgmap v1016: 97 pgs: 97 active+clean; 689 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.6 MiB/s wr, 655 op/s 2023-07-29T16:36:57.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 257773 34.42 MB/sec execute 281 sec latency 102.297 ms 2023-07-29T16:36:58.335 INFO:tasks.workunit.client.0.smithi031.stdout: 10 258436 34.44 MB/sec execute 282 sec latency 87.951 ms 2023-07-29T16:36:58.436 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:36:58 smithi031 ceph-mon[95504]: pgmap v1017: 97 pgs: 97 active+clean; 676 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 725 op/s 2023-07-29T16:36:58.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:36:58 smithi039 ceph-mon[128858]: pgmap v1017: 97 pgs: 97 active+clean; 676 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 725 op/s 2023-07-29T16:36:58.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:36:58 smithi156 ceph-mon[129245]: pgmap v1017: 97 pgs: 97 active+clean; 676 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 10 MiB/s wr, 725 op/s 2023-07-29T16:36:59.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 259120 34.44 MB/sec execute 283 sec latency 64.445 ms 2023-07-29T16:37:00.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 259918 34.47 MB/sec execute 284 sec latency 57.092 ms 2023-07-29T16:37:01.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:00 smithi039 ceph-mon[128858]: pgmap v1018: 97 pgs: 97 active+clean; 681 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 11 MiB/s wr, 769 op/s 2023-07-29T16:37:01.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:00 smithi156 ceph-mon[129245]: pgmap v1018: 97 pgs: 97 active+clean; 681 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 11 MiB/s wr, 769 op/s 2023-07-29T16:37:01.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:00 smithi031 ceph-mon[95504]: pgmap v1018: 97 pgs: 97 active+clean; 681 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 11 MiB/s wr, 769 op/s 2023-07-29T16:37:01.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 260779 34.49 MB/sec execute 285 sec latency 56.242 ms 2023-07-29T16:37:02.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 261538 34.53 MB/sec execute 286 sec latency 57.959 ms 2023-07-29T16:37:03.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:02 smithi039 ceph-mon[128858]: pgmap v1019: 97 pgs: 97 active+clean; 684 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 742 op/s 2023-07-29T16:37:03.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:02 smithi156 ceph-mon[129245]: pgmap v1019: 97 pgs: 97 active+clean; 684 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 742 op/s 2023-07-29T16:37:03.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:02 smithi031 ceph-mon[95504]: pgmap v1019: 97 pgs: 97 active+clean; 684 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 742 op/s 2023-07-29T16:37:03.336 INFO:tasks.workunit.client.0.smithi031.stdout: 10 262233 34.55 MB/sec execute 287 sec latency 80.780 ms 2023-07-29T16:37:04.337 INFO:tasks.workunit.client.0.smithi031.stdout: 10 262818 34.53 MB/sec execute 288 sec latency 66.994 ms 2023-07-29T16:37:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 -- 172.21.15.31:0/1410447230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c103da0 msgr2=0x7ff81c104220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 --2- 172.21.15.31:0/1410447230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c103da0 0x7ff81c104220 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff80c009f90 tx=0x7ff80c00b330 comp rx=0 tx=0).stop 2023-07-29T16:37:04.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 -- 172.21.15.31:0/1410447230 shutdown_connections 2023-07-29T16:37:04.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 --2- 172.21.15.31:0/1410447230 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c103da0 0x7ff81c104220 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 --2- 172.21.15.31:0/1410447230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff81c0fea80 0x7ff81c100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 --2- 172.21.15.31:0/1410447230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff81c1013e0 0x7ff81c103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.558+0000 7ff8209fe700 1 -- 172.21.15.31:0/1410447230 >> 172.21.15.31:0/1410447230 conn(0x7ff81c0fa690 msgr2=0x7ff81c0fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:04.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.559+0000 7ff8209fe700 1 -- 172.21.15.31:0/1410447230 shutdown_connections 2023-07-29T16:37:04.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.559+0000 7ff8209fe700 1 -- 172.21.15.31:0/1410447230 wait complete. 2023-07-29T16:37:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.560+0000 7ff8209fe700 1 Processor -- start 2023-07-29T16:37:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.560+0000 7ff8209fe700 1 -- start start 2023-07-29T16:37:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff8209fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 0x7ff81c19c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff8209fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff81c1013e0 0x7ff81c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff8209fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff81c103da0 0x7ff81c1a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff8209fe700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c1a41c0 con 0x7ff81c103da0 2023-07-29T16:37:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff8209fe700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c1a4300 con 0x7ff81c1013e0 2023-07-29T16:37:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff8209fe700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81c1a4490 con 0x7ff81c0fea80 2023-07-29T16:37:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff81ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 0x7ff81c19c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff81ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 0x7ff81c19c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54252/0 (socket says 172.21.15.31:54252) 2023-07-29T16:37:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff81ad9d700 1 -- 172.21.15.31:0/230186448 learned_addr learned my addr 172.21.15.31:0/230186448 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.561+0000 7ff81b59e700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff81c103da0 0x7ff81c1a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81a59c700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff81c1013e0 0x7ff81c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81ad9d700 1 -- 172.21.15.31:0/230186448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff81c1013e0 msgr2=0x7ff81c19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff81c1013e0 0x7ff81c19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81ad9d700 1 -- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff81c103da0 msgr2=0x7ff81c1a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff81c103da0 0x7ff81c1a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81ad9d700 1 -- 172.21.15.31:0/230186448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c009c40 con 0x7ff81c0fea80 2023-07-29T16:37:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 0x7ff81c19c640 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e9f0 tx=0x7ff81400edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.562+0000 7ff813fff700 1 -- 172.21.15.31:0/230186448 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81400cbd0 con 0x7ff81c0fea80 2023-07-29T16:37:04.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.563+0000 7ff813fff700 1 -- 172.21.15.31:0/230186448 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8140044e0 con 0x7ff81c0fea80 2023-07-29T16:37:04.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.563+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81c1a4770 con 0x7ff81c0fea80 2023-07-29T16:37:04.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.563+0000 7ff813fff700 1 -- 172.21.15.31:0/230186448 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff814003d90 con 0x7ff81c0fea80 2023-07-29T16:37:04.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.563+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c1a4d50 con 0x7ff81c0fea80 2023-07-29T16:37:04.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.570+0000 7ff834c04b80 1 Processor -- start 2023-07-29T16:37:04.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.588+0000 7ff813fff700 1 -- 172.21.15.31:0/230186448 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff814004000 con 0x7ff81c0fea80 2023-07-29T16:37:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.591+0000 7ff813fff700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8080a0890 0x7ff8080a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.592+0000 7ff813fff700 1 -- 172.21.15.31:0/230186448 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff814014070 con 0x7ff81c0fea80 2023-07-29T16:37:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.592+0000 7ff834c04b80 1 -- start start 2023-07-29T16:37:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.593+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a325c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.593+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a3269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.593+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3273a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.593+0000 7ff834c04b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a27d9b0 con 0x55b26a326f00 2023-07-29T16:37:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.593+0000 7ff834c04b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a27b370 con 0x55b26a326150 2023-07-29T16:37:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.593+0000 7ff834c04b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a276c60 con 0x55b26a1816b0 2023-07-29T16:37:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.597+0000 7ff81ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a325c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.598+0000 7ff81ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a325c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54268/0 (socket says 172.21.15.31:54268) 2023-07-29T16:37:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.598+0000 7ff81ad9d700 1 -- 172.21.15.31:0/3435587110 learned_addr learned my addr 172.21.15.31:0/3435587110 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:04.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.605+0000 7ff81a59c700 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a3269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.610+0000 7ff81ad9d700 1 -- 172.21.15.31:0/3435587110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 msgr2=0x55b26a3269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.610+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a3269c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.610+0000 7ff81ad9d700 1 -- 172.21.15.31:0/3435587110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 msgr2=0x55b26a3273a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.610+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3273a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.610+0000 7ff81ad9d700 1 -- 172.21.15.31:0/3435587110 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b26a0a4d80 con 0x55b26a1816b0 2023-07-29T16:37:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.611+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a325c10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff8140a14d0 tx=0x7ff814099480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a7e6cfeda5e276 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.612+0000 7ff81b59e700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8080a0890 0x7ff8080a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.617+0000 7ff8117fa700 1 -- 172.21.15.31:0/3435587110 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8140995b0 con 0x55b26a1816b0 2023-07-29T16:37:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.617+0000 7ff8117fa700 1 -- 172.21.15.31:0/3435587110 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff81409d5e0 con 0x55b26a1816b0 2023-07-29T16:37:04.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.617+0000 7ff8117fa700 1 -- 172.21.15.31:0/3435587110 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff814097a70 con 0x55b26a1816b0 2023-07-29T16:37:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.618+0000 7ff834c04b80 1 -- 172.21.15.31:0/3435587110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 msgr2=0x55b26a325c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:04.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.618+0000 7ff834c04b80 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a325c10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ff8140a14d0 tx=0x7ff814099480 comp rx=0 tx=0).stop 2023-07-29T16:37:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.623+0000 7ff834c04b80 1 -- 172.21.15.31:0/3435587110 shutdown_connections 2023-07-29T16:37:04.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.623+0000 7ff834c04b80 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3273a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.623+0000 7ff834c04b80 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a325c10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.623+0000 7ff834c04b80 1 --2- 172.21.15.31:0/3435587110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a3269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.623+0000 7ff834c04b80 1 -- 172.21.15.31:0/3435587110 >> 172.21.15.31:0/3435587110 conn(0x55b26a31b380 msgr2=0x55b26a3206a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:04.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.643+0000 7ff81b59e700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8080a0890 0x7ff8080a2d40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0060b0 tx=0x7ff80c009ac0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:04.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.646+0000 7ff834c04b80 1 -- 172.21.15.31:0/3435587110 shutdown_connections 2023-07-29T16:37:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.656+0000 7ff834c04b80 1 -- 172.21.15.31:0/3435587110 wait complete. 2023-07-29T16:37:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.657+0000 7ff834c04b80 1 -- start start 2023-07-29T16:37:04.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.657+0000 7ff834c04b80 1 Processor -- start 2023-07-29T16:37:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a1816b0 0x55b26a3400c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a340600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a347640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a2134b0 con 0x55b26a1816b0 2023-07-29T16:37:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a213150 con 0x55b26a326150 2023-07-29T16:37:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a0fa8c0 con 0x55b26a326f00 2023-07-29T16:37:04.661 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.658+0000 7ff834c04b80 10 client.0 _ll_register_callbacks cb 0x55b26a21c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.664+0000 7ff81b59e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a347640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.664+0000 7ff81b59e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a347640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54280/0 (socket says 172.21.15.31:54280) 2023-07-29T16:37:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.664+0000 7ff81b59e700 1 -- 172.21.15.31:0/1866151529 learned_addr learned my addr 172.21.15.31:0/1866151529 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.670+0000 7ff81b59e700 1 -- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 msgr2=0x55b26a340600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.670+0000 7ff81b59e700 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a340600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.670+0000 7ff81b59e700 1 -- 172.21.15.31:0/1866151529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a1816b0 msgr2=0x55b26a3400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.670+0000 7ff81b59e700 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a1816b0 0x55b26a3400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.670+0000 7ff81b59e700 1 -- 172.21.15.31:0/1866151529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81c1a4d50 con 0x55b26a326f00 2023-07-29T16:37:04.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.682+0000 7ff81b59e700 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a347640 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff80c015000 tx=0x7ff80c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=526631c99d05dbdd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:04.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.682+0000 7ff8027fc700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:37:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.700+0000 7ff8027fc700 1 -- 172.21.15.31:0/1866151529 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c00be80 con 0x55b26a326f00 2023-07-29T16:37:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.700+0000 7ff8027fc700 1 -- 172.21.15.31:0/1866151529 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff80c004e30 con 0x55b26a326f00 2023-07-29T16:37:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.700+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b26a2a1cb0 con 0x55b26a326f00 2023-07-29T16:37:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.703+0000 7ff8027fc700 1 -- 172.21.15.31:0/1866151529 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff80c005250 con 0x55b26a326f00 2023-07-29T16:37:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.717+0000 7ff834c04b80 10 client.30487 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:04.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.717+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55b26a08dc00 con 0x55b26a326f00 2023-07-29T16:37:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff8027fc700 1 -- 172.21.15.31:0/1866151529 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b26a2a1cb0 con 0x55b26a326f00 2023-07-29T16:37:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff8027fc700 1 -- 172.21.15.31:0/1866151529 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff80c0268c0 con 0x55b26a326f00 2023-07-29T16:37:04.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff834c04b80 10 client.30487 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff834c04b80 10 client.30487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff834c04b80 20 client.30487 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a35cf40 0x55b26a35f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff834c04b80 4 client.30487 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:37:04.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.728+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b26a2f4d10 con 0x55b26a35cf40 2023-07-29T16:37:04.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.729+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a35cf40 0x55b26a35f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.756+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a35cf40 0x55b26a35f350 crc :-1 s=READY pgs=479 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5258fec3d8194bb9 server_cookie=d41a3458385fd523 in_seq=0 out_seq=0 2023-07-29T16:37:04.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.757+0000 7ff8027fc700 10 client.30487 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.783+0000 7ff8027fc700 1 -- 172.21.15.31:0/1866151529 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55b26a2f4d10 con 0x55b26a35cf40 2023-07-29T16:37:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.783+0000 7ff8027fc700 10 client.30487 handle_command_reply: tid=0 2023-07-29T16:37:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.783+0000 7ff834c04b80 1 client.30487 shutdown 2023-07-29T16:37:04.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.792+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 msgr2=0x55b26a347640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:04.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.792+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a347640 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff80c015000 tx=0x7ff80c009920 comp rx=0 tx=0).stop 2023-07-29T16:37:04.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 shutdown_connections 2023-07-29T16:37:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a35cf40 0x55b26a35f350 crc :-1 s=READY pgs=479 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a347640 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a1816b0 0x55b26a3400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1866151529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a340600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 >> 172.21.15.31:0/1866151529 conn(0x55b26a31b380 msgr2=0x55b26a31fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff8027fc700 0 client.30487 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:04.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.802+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 shutdown_connections 2023-07-29T16:37:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.803+0000 7ff834c04b80 1 -- 172.21.15.31:0/1866151529 wait complete. 2023-07-29T16:37:04.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.803+0000 7ff834c04b80 20 client.30487 trim_cache size 0 max 16384 2023-07-29T16:37:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.834+0000 7ff834c04b80 1 Processor -- start 2023-07-29T16:37:04.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.834+0000 7ff834c04b80 1 -- start start 2023-07-29T16:37:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.834+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a345550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.834+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a32d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.835+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3400c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.835+0000 7ff834c04b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a35a340 con 0x55b26a326f00 2023-07-29T16:37:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.835+0000 7ff834c04b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a35cd30 con 0x55b26a326150 2023-07-29T16:37:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.835+0000 7ff834c04b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a321380 con 0x55b26a1816b0 2023-07-29T16:37:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.835+0000 7ff81a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a32d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.835+0000 7ff81b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.836+0000 7ff81b59e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3400c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52770/0 (socket says 172.21.15.31:52770) 2023-07-29T16:37:04.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.836+0000 7ff81b59e700 1 -- 172.21.15.31:0/2660913219 learned_addr learned my addr 172.21.15.31:0/2660913219 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.841+0000 7ff81b59e700 1 -- 172.21.15.31:0/2660913219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 msgr2=0x55b26a345550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.841+0000 7ff81b59e700 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a345550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.841+0000 7ff81b59e700 1 -- 172.21.15.31:0/2660913219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 msgr2=0x55b26a32d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.841+0000 7ff81b59e700 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a32d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.841+0000 7ff81b59e700 1 -- 172.21.15.31:0/2660913219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b26a0cb1f0 con 0x55b26a326f00 2023-07-29T16:37:04.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.879+0000 7ff81b59e700 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3400c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff80c01c410 tx=0x7ff80c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50df67adb2dabff server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.887+0000 7ff811ffb700 1 -- 172.21.15.31:0/2660913219 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c00be00 con 0x55b26a326f00 2023-07-29T16:37:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.887+0000 7ff811ffb700 1 -- 172.21.15.31:0/2660913219 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff80c004e30 con 0x55b26a326f00 2023-07-29T16:37:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.887+0000 7ff811ffb700 1 -- 172.21.15.31:0/2660913219 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0057b0 con 0x55b26a326f00 2023-07-29T16:37:04.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.888+0000 7ff834c04b80 1 -- 172.21.15.31:0/2660913219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 msgr2=0x55b26a3400c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:04.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.888+0000 7ff834c04b80 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3400c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff80c01c410 tx=0x7ff80c01c440 comp rx=0 tx=0).stop 2023-07-29T16:37:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.893+0000 7ff834c04b80 1 -- 172.21.15.31:0/2660913219 shutdown_connections 2023-07-29T16:37:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.893+0000 7ff834c04b80 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326f00 0x55b26a3400c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.893+0000 7ff834c04b80 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a1816b0 0x55b26a345550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.893+0000 7ff834c04b80 1 --2- 172.21.15.31:0/2660913219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a326150 0x55b26a32d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.893+0000 7ff834c04b80 1 -- 172.21.15.31:0/2660913219 >> 172.21.15.31:0/2660913219 conn(0x55b26a31b380 msgr2=0x55b26a35ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:04.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.911+0000 7ff834c04b80 1 -- 172.21.15.31:0/2660913219 shutdown_connections 2023-07-29T16:37:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.911+0000 7ff834c04b80 1 -- 172.21.15.31:0/2660913219 wait complete. 2023-07-29T16:37:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.911+0000 7ff834c04b80 1 -- start start 2023-07-29T16:37:04.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 Processor -- start 2023-07-29T16:37:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a1816b0 0x55b26a343520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326150 0x55b26a343a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a344a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a344fb0 con 0x55b26a326150 2023-07-29T16:37:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a345120 con 0x55b26a1816b0 2023-07-29T16:37:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.912+0000 7ff834c04b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a3452e0 con 0x55b26a326f00 2023-07-29T16:37:04.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.913+0000 7ff834c04b80 10 client.0 _ll_register_callbacks cb 0x55b26a21c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:04.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.915+0000 7ff81b59e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a344a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.915+0000 7ff81b59e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a344a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54292/0 (socket says 172.21.15.31:54292) 2023-07-29T16:37:04.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.915+0000 7ff81b59e700 1 -- 172.21.15.31:0/1855124083 learned_addr learned my addr 172.21.15.31:0/1855124083 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.934+0000 7ff81b59e700 1 -- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a1816b0 msgr2=0x55b26a343520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.934+0000 7ff81b59e700 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a1816b0 0x55b26a343520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.934+0000 7ff81b59e700 1 -- 172.21.15.31:0/1855124083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326150 msgr2=0x55b26a343a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.934+0000 7ff81b59e700 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326150 0x55b26a343a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:04.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.934+0000 7ff81b59e700 1 -- 172.21.15.31:0/1855124083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b26a08dc00 con 0x55b26a326f00 2023-07-29T16:37:04.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.940+0000 7ff81b59e700 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a344a70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff80c026c50 tx=0x7ff80c005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d845176bdf1de84 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:04.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.941+0000 7ff8027fc700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:37:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.954+0000 7ff8027fc700 1 -- 172.21.15.31:0/1855124083 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c03d070 con 0x55b26a326f00 2023-07-29T16:37:04.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.954+0000 7ff8027fc700 1 -- 172.21.15.31:0/1855124083 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff80c026420 con 0x55b26a326f00 2023-07-29T16:37:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.958+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff81409d5e0 con 0x55b26a326f00 2023-07-29T16:37:04.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.965+0000 7ff8027fc700 1 -- 172.21.15.31:0/1855124083 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff80c0265e0 con 0x55b26a326f00 2023-07-29T16:37:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.967+0000 7ff834c04b80 10 client.30490 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.967+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55b26a0a3250 con 0x55b26a326f00 2023-07-29T16:37:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.969+0000 7ff8027fc700 1 -- 172.21.15.31:0/1855124083 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff81409d5e0 con 0x55b26a326f00 2023-07-29T16:37:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.969+0000 7ff8027fc700 1 -- 172.21.15.31:0/1855124083 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff80c040020 con 0x55b26a326f00 2023-07-29T16:37:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.970+0000 7ff834c04b80 10 client.30490 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.970+0000 7ff834c04b80 10 client.30490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.970+0000 7ff834c04b80 20 client.30490 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.970+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a3af710 0x55b26a3b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.970+0000 7ff834c04b80 4 client.30490 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:37:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.970+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b26a26fce0 con 0x55b26a3af710 2023-07-29T16:37:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.971+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a3af710 0x55b26a3b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.996+0000 7ff81ad9d700 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a3af710 0x55b26a3b1b20 crc :-1 s=READY pgs=480 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=765fe756cb1c25d1 server_cookie=447fb8682849d2ec in_seq=0 out_seq=0 2023-07-29T16:37:05.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:04.999+0000 7ff8027fc700 10 client.30490 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:05.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:04 smithi031 ceph-mon[95504]: pgmap v1020: 97 pgs: 97 active+clean; 689 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 9.7 MiB/s wr, 802 op/s 2023-07-29T16:37:05.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:04 smithi039 ceph-mon[128858]: pgmap v1020: 97 pgs: 97 active+clean; 689 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 9.7 MiB/s wr, 802 op/s 2023-07-29T16:37:05.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:04 smithi156 ceph-mon[129245]: pgmap v1020: 97 pgs: 97 active+clean; 689 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 9.7 MiB/s wr, 802 op/s 2023-07-29T16:37:05.340 INFO:tasks.workunit.client.0.smithi031.stdout: 10 262933 34.43 MB/sec execute 289 sec latency 731.304 ms 2023-07-29T16:37:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.040+0000 7ff8027fc700 1 -- 172.21.15.31:0/1855124083 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b26a26fce0 con 0x55b26a3af710 2023-07-29T16:37:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.040+0000 7ff8027fc700 10 client.30490 handle_command_reply: tid=0 2023-07-29T16:37:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.040+0000 7ff834c04b80 1 client.30490 shutdown 2023-07-29T16:37:06.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.041+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 msgr2=0x55b26a344a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:06.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.041+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a344a70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff80c026c50 tx=0x7ff80c005b20 comp rx=0 tx=0).stop 2023-07-29T16:37:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 shutdown_connections 2023-07-29T16:37:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55b26a3af710 0x55b26a3b1b20 crc :-1 s=READY pgs=480 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b26a326f00 0x55b26a344a70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b26a1816b0 0x55b26a343520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff834c04b80 1 --2- 172.21.15.31:0/1855124083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b26a326150 0x55b26a343a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 >> 172.21.15.31:0/1855124083 conn(0x55b26a31b380 msgr2=0x55b26a32d280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.043+0000 7ff8027fc700 0 client.30490 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.044+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 shutdown_connections 2023-07-29T16:37:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.046+0000 7ff834c04b80 1 -- 172.21.15.31:0/1855124083 wait complete. 2023-07-29T16:37:06.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.048+0000 7ff834c04b80 20 client.30490 trim_cache size 0 max 16384 2023-07-29T16:37:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:37:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:37:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:37:06.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:37:06.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8080a0890 msgr2=0x7ff8080a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:06.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8080a0890 0x7ff8080a2d40 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0060b0 tx=0x7ff80c009ac0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 msgr2=0x7ff81c19c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:06.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 0x7ff81c19c640 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e9f0 tx=0x7ff81400edb0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 shutdown_connections 2023-07-29T16:37:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff8080a0890 0x7ff8080a2d40 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff81c103da0 0x7ff81c1a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff81c0fea80 0x7ff81c19c640 secure :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e9f0 tx=0x7ff81400edb0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 --2- 172.21.15.31:0/230186448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff81c1013e0 0x7ff81c19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:06.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.051+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 >> 172.21.15.31:0/230186448 conn(0x7ff81c0fa690 msgr2=0x7ff81c0ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.052+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 shutdown_connections 2023-07-29T16:37:06.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:06.052+0000 7ff8209fe700 1 -- 172.21.15.31:0/230186448 wait complete. 2023-07-29T16:37:06.341 INFO:tasks.workunit.client.0.smithi031.stdout: 10 263275 34.38 MB/sec execute 290 sec latency 709.450 ms 2023-07-29T16:37:07.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:06 smithi031 ceph-mon[95504]: pgmap v1021: 97 pgs: 97 active+clean; 697 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.4 MiB/s wr, 565 op/s 2023-07-29T16:37:07.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:06 smithi039 ceph-mon[128858]: pgmap v1021: 97 pgs: 97 active+clean; 697 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.4 MiB/s wr, 565 op/s 2023-07-29T16:37:07.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:06 smithi156 ceph-mon[129245]: pgmap v1021: 97 pgs: 97 active+clean; 697 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.4 MiB/s wr, 565 op/s 2023-07-29T16:37:07.341 INFO:tasks.workunit.client.0.smithi031.stdout: 10 263786 34.34 MB/sec execute 291 sec latency 123.189 ms 2023-07-29T16:37:08.341 INFO:tasks.workunit.client.0.smithi031.stdout: 10 264579 34.38 MB/sec execute 292 sec latency 62.089 ms 2023-07-29T16:37:08.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:08 smithi031 ceph-mon[95504]: pgmap v1022: 97 pgs: 97 active+clean; 713 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 8.8 MiB/s wr, 638 op/s 2023-07-29T16:37:08.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:08 smithi039 ceph-mon[128858]: pgmap v1022: 97 pgs: 97 active+clean; 713 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 8.8 MiB/s wr, 638 op/s 2023-07-29T16:37:08.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:08 smithi156 ceph-mon[129245]: pgmap v1022: 97 pgs: 97 active+clean; 713 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 8.8 MiB/s wr, 638 op/s 2023-07-29T16:37:09.341 INFO:tasks.workunit.client.0.smithi031.stdout: 10 265172 34.38 MB/sec execute 293 sec latency 64.698 ms 2023-07-29T16:37:10.341 INFO:tasks.workunit.client.0.smithi031.stdout: 10 265899 34.39 MB/sec execute 294 sec latency 77.257 ms 2023-07-29T16:37:11.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:10 smithi039 ceph-mon[128858]: pgmap v1023: 97 pgs: 97 active+clean; 708 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 10 MiB/s wr, 690 op/s 2023-07-29T16:37:11.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:10 smithi156 ceph-mon[129245]: pgmap v1023: 97 pgs: 97 active+clean; 708 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 10 MiB/s wr, 690 op/s 2023-07-29T16:37:11.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:10 smithi031 ceph-mon[95504]: pgmap v1023: 97 pgs: 97 active+clean; 708 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 10 MiB/s wr, 690 op/s 2023-07-29T16:37:11.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 266607 34.40 MB/sec execute 295 sec latency 82.688 ms 2023-07-29T16:37:12.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 267232 34.42 MB/sec execute 296 sec latency 104.033 ms 2023-07-29T16:37:13.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:12 smithi039 ceph-mon[128858]: pgmap v1024: 97 pgs: 97 active+clean; 685 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 10 MiB/s wr, 703 op/s 2023-07-29T16:37:13.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:12 smithi156 ceph-mon[129245]: pgmap v1024: 97 pgs: 97 active+clean; 685 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 10 MiB/s wr, 703 op/s 2023-07-29T16:37:13.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:12 smithi031 ceph-mon[95504]: pgmap v1024: 97 pgs: 97 active+clean; 685 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 10 MiB/s wr, 703 op/s 2023-07-29T16:37:13.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 267832 34.40 MB/sec execute 297 sec latency 66.414 ms 2023-07-29T16:37:14.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 268571 34.43 MB/sec execute 298 sec latency 46.168 ms 2023-07-29T16:37:15.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:14 smithi039 ceph-mon[128858]: pgmap v1025: 97 pgs: 97 active+clean; 681 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 735 op/s 2023-07-29T16:37:15.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:14 smithi156 ceph-mon[129245]: pgmap v1025: 97 pgs: 97 active+clean; 681 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 735 op/s 2023-07-29T16:37:15.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:14 smithi031 ceph-mon[95504]: pgmap v1025: 97 pgs: 97 active+clean; 681 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 735 op/s 2023-07-29T16:37:15.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 269311 34.42 MB/sec execute 299 sec latency 67.394 ms 2023-07-29T16:37:16.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 269993 34.44 MB/sec execute 300 sec latency 83.532 ms 2023-07-29T16:37:17.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:16 smithi039 ceph-mon[128858]: pgmap v1026: 97 pgs: 97 active+clean; 684 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 10 MiB/s wr, 680 op/s 2023-07-29T16:37:17.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:16 smithi156 ceph-mon[129245]: pgmap v1026: 97 pgs: 97 active+clean; 684 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 10 MiB/s wr, 680 op/s 2023-07-29T16:37:17.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:16 smithi031 ceph-mon[95504]: pgmap v1026: 97 pgs: 97 active+clean; 684 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 10 MiB/s wr, 680 op/s 2023-07-29T16:37:17.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 270705 34.45 MB/sec execute 301 sec latency 81.168 ms 2023-07-29T16:37:18.342 INFO:tasks.workunit.client.0.smithi031.stdout: 10 271530 34.48 MB/sec execute 302 sec latency 53.662 ms 2023-07-29T16:37:18.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:18 smithi039 ceph-mon[128858]: pgmap v1027: 97 pgs: 97 active+clean; 687 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 9.6 MiB/s wr, 678 op/s 2023-07-29T16:37:18.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:18 smithi156 ceph-mon[129245]: pgmap v1027: 97 pgs: 97 active+clean; 687 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 9.6 MiB/s wr, 678 op/s 2023-07-29T16:37:18.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:18 smithi031 ceph-mon[95504]: pgmap v1027: 97 pgs: 97 active+clean; 687 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 9.6 MiB/s wr, 678 op/s 2023-07-29T16:37:19.354 INFO:tasks.workunit.client.0.smithi031.stdout: 10 272270 34.52 MB/sec execute 303 sec latency 158.507 ms 2023-07-29T16:37:20.343 INFO:tasks.workunit.client.0.smithi031.stdout: 10 272354 34.42 MB/sec execute 304 sec latency 475.714 ms 2023-07-29T16:37:21.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:20 smithi156 ceph-mon[129245]: pgmap v1028: 97 pgs: 97 active+clean; 688 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.2 MiB/s wr, 622 op/s 2023-07-29T16:37:21.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:20 smithi031 ceph-mon[95504]: pgmap v1028: 97 pgs: 97 active+clean; 688 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.2 MiB/s wr, 622 op/s 2023-07-29T16:37:21.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:20 smithi039 ceph-mon[128858]: pgmap v1028: 97 pgs: 97 active+clean; 688 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.2 MiB/s wr, 622 op/s 2023-07-29T16:37:21.344 INFO:tasks.workunit.client.0.smithi031.stdout: 10 272688 34.36 MB/sec execute 305 sec latency 131.188 ms 2023-07-29T16:37:22.344 INFO:tasks.workunit.client.0.smithi031.stdout: 10 273235 34.34 MB/sec execute 306 sec latency 114.365 ms 2023-07-29T16:37:22.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.889+0000 7fa84ebff700 1 -- 172.21.15.31:0/17118878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848106990 msgr2=0x7fa848106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.889+0000 7fa84ebff700 1 --2- 172.21.15.31:0/17118878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848106990 0x7fa848106da0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa838009990 tx=0x7fa838009ca0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 -- 172.21.15.31:0/17118878 shutdown_connections 2023-07-29T16:37:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 --2- 172.21.15.31:0/17118878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa848108510 0x7fa84810edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 --2- 172.21.15.31:0/17118878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848106990 0x7fa848106da0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 --2- 172.21.15.31:0/17118878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848107b80 0x7fa848107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 -- 172.21.15.31:0/17118878 >> 172.21.15.31:0/17118878 conn(0x7fa8480759d0 msgr2=0x7fa848077e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:22.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 -- 172.21.15.31:0/17118878 shutdown_connections 2023-07-29T16:37:22.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.890+0000 7fa84ebff700 1 -- 172.21.15.31:0/17118878 wait complete. 2023-07-29T16:37:22.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.891+0000 7fa84ebff700 1 Processor -- start 2023-07-29T16:37:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 -- start start 2023-07-29T16:37:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa848106990 0x7fa84806f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 0x7fa84806fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108510 0x7fa848070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa848071140 con 0x7fa848108510 2023-07-29T16:37:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8480712b0 con 0x7fa848107b80 2023-07-29T16:37:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.892+0000 7fa84ebff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa848071470 con 0x7fa848106990 2023-07-29T16:37:22.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.893+0000 7fa847fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 0x7fa84806fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.893+0000 7fa847fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 0x7fa84806fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58532/0 (socket says 172.21.15.31:58532) 2023-07-29T16:37:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.893+0000 7fa847fff700 1 -- 172.21.15.31:0/1169843489 learned_addr learned my addr 172.21.15.31:0/1169843489 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.893+0000 7fa84c99b700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa848106990 0x7fa84806f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.893+0000 7fa84d19c700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108510 0x7fa848070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa847fff700 1 -- 172.21.15.31:0/1169843489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa848106990 msgr2=0x7fa84806f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:22.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa847fff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa848106990 0x7fa84806f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa847fff700 1 -- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108510 msgr2=0x7fa848070ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa847fff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108510 0x7fa848070ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa847fff700 1 -- 172.21.15.31:0/1169843489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8380095f0 con 0x7fa848107b80 2023-07-29T16:37:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa84d19c700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108510 0x7fa848070ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:37:22.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa847fff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 0x7fa84806fc10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00b530 tx=0x7fa83c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.894+0000 7fa845ffb700 1 -- 172.21.15.31:0/1169843489 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c011830 con 0x7fa848107b80 2023-07-29T16:37:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.895+0000 7fa845ffb700 1 -- 172.21.15.31:0/1169843489 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa83c011e50 con 0x7fa848107b80 2023-07-29T16:37:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.895+0000 7fa845ffb700 1 -- 172.21.15.31:0/1169843489 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c00f2e0 con 0x7fa848107b80 2023-07-29T16:37:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.895+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa848071750 con 0x7fa848107b80 2023-07-29T16:37:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.895+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa848071f80 con 0x7fa848107b80 2023-07-29T16:37:22.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.901+0000 7fa845ffb700 1 -- 172.21.15.31:0/1169843489 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fa83c010380 con 0x7fa848107b80 2023-07-29T16:37:22.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.901+0000 7fa862554b80 1 Processor -- start 2023-07-29T16:37:22.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.902+0000 7fa862554b80 1 -- start start 2023-07-29T16:37:22.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa862554b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa862554b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3bfdf0 0x5602be3c07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa862554b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa862554b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3179b0 con 0x5602be3bfdf0 2023-07-29T16:37:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa862554b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be315370 con 0x5602be3c0cf0 2023-07-29T16:37:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa862554b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be310c60 con 0x5602be21b6b0 2023-07-29T16:37:22.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa847fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.903+0000 7fa84d19c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3bf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3bf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42722/0 (socket says 172.21.15.31:42722) 2023-07-29T16:37:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 -- 172.21.15.31:0/4201947299 learned_addr learned my addr 172.21.15.31:0/4201947299 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:22.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa847fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58540/0 (socket says 172.21.15.31:58540) 2023-07-29T16:37:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 -- 172.21.15.31:0/4201947299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 msgr2=0x5602be3c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 -- 172.21.15.31:0/4201947299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3bfdf0 msgr2=0x5602be3c07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3bfdf0 0x5602be3c07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 -- 172.21.15.31:0/4201947299 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602be13ed80 con 0x5602be21b6b0 2023-07-29T16:37:22.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa845ffb700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8300a0890 0x7fa8300a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa845ffb700 1 -- 172.21.15.31:0/1169843489 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa83c0cb5e0 con 0x7fa848107b80 2023-07-29T16:37:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa84d19c700 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3bf8b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa840007f30 tx=0x7fa84000c5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e2f96b373e24b2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:22.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.904+0000 7fa847fff700 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.905+0000 7fa82effd700 1 -- 172.21.15.31:0/4201947299 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84000f440 con 0x5602be21b6b0 2023-07-29T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.905+0000 7fa82effd700 1 -- 172.21.15.31:0/4201947299 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa84000fa60 con 0x5602be21b6b0 2023-07-29T16:37:22.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.905+0000 7fa82effd700 1 -- 172.21.15.31:0/4201947299 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84000ea00 con 0x5602be21b6b0 2023-07-29T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.905+0000 7fa84c99b700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8300a0890 0x7fa8300a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 -- 172.21.15.31:0/4201947299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 msgr2=0x5602be3bf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3bf8b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fa840007f30 tx=0x7fa84000c5a0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa84c99b700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8300a0890 0x7fa8300a2d40 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa8380036f0 tx=0x7fa838009ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 -- 172.21.15.31:0/4201947299 shutdown_connections 2023-07-29T16:37:22.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3bf8b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 --2- 172.21.15.31:0/4201947299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3bfdf0 0x5602be3c07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 -- 172.21.15.31:0/4201947299 >> 172.21.15.31:0/4201947299 conn(0x5602be3b4fd0 msgr2=0x5602be3ba300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 -- 172.21.15.31:0/4201947299 shutdown_connections 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.906+0000 7fa862554b80 1 -- 172.21.15.31:0/4201947299 wait complete. 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.907+0000 7fa862554b80 1 -- start start 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.907+0000 7fa862554b80 1 Processor -- start 2023-07-29T16:37:22.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be21b6b0 0x5602be3c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be2ad4b0 con 0x5602be3c0cf0 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be2ad150 con 0x5602be21b6b0 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be1948c0 con 0x5602be3bfdf0 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa862554b80 10 client.0 _ll_register_callbacks cb 0x5602be2b65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:22.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa84c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be21b6b0 0x5602be3c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa84c99b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be21b6b0 0x5602be3c8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58550/0 (socket says 172.21.15.31:58550) 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.908+0000 7fa84c99b700 1 -- 172.21.15.31:0/3474112007 learned_addr learned my addr 172.21.15.31:0/3474112007 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa84d19c700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 -- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be21b6b0 msgr2=0x5602be3c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be21b6b0 0x5602be3c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 -- 172.21.15.31:0/3474112007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 msgr2=0x5602be3d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 -- 172.21.15.31:0/3474112007 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa840007b60 con 0x5602be3bfdf0 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa847fff700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c9490 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x5602be3bf610 tx=0x7fa83c0985e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3e7f7a6f4cb7520 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.909+0000 7fa827fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa827fff700 1 -- 172.21.15.31:0/3474112007 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c09ccb0 con 0x5602be3bfdf0 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa827fff700 1 -- 172.21.15.31:0/3474112007 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa83c0a10d0 con 0x5602be3bfdf0 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602be33bcb0 con 0x5602be3bfdf0 2023-07-29T16:37:22.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa827fff700 1 -- 172.21.15.31:0/3474112007 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa83c0b1240 con 0x5602be3bfdf0 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa862554b80 10 client.30499 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5602be127c00 con 0x5602be3bfdf0 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.910+0000 7fa827fff700 1 -- 172.21.15.31:0/3474112007 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5602be33bcb0 con 0x5602be3bfdf0 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa827fff700 1 -- 172.21.15.31:0/3474112007 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa83c0a0c40 con 0x5602be3bfdf0 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa862554b80 10 client.30499 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa862554b80 10 client.30499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:22.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa862554b80 20 client.30499 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa862554b80 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be3d4c80 0x5602be3d7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa862554b80 4 client.30499 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:37:22.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602be38ed10 con 0x5602be3d4c80 2023-07-29T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.911+0000 7fa84c99b700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be3d4c80 0x5602be3d7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.912+0000 7fa84c99b700 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be3d4c80 0x5602be3d7090 crc :-1 s=READY pgs=481 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d1907b9d8311b53 server_cookie=3d595b81f5a8bbce in_seq=0 out_seq=0 2023-07-29T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.912+0000 7fa827fff700 10 client.30499 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.915+0000 7fa827fff700 1 -- 172.21.15.31:0/3474112007 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5602be38ed10 con 0x5602be3d4c80 2023-07-29T16:37:22.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.915+0000 7fa827fff700 10 client.30499 handle_command_reply: tid=0 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.915+0000 7fa862554b80 1 client.30499 shutdown 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 msgr2=0x5602be3c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c9490 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x5602be3bf610 tx=0x7fa83c0985e0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 shutdown_connections 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be3d4c80 0x5602be3d7090 crc :-1 s=READY pgs=481 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be21b6b0 0x5602be3c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 --2- 172.21.15.31:0/3474112007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c9490 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 >> 172.21.15.31:0/3474112007 conn(0x5602be3b4fd0 msgr2=0x5602be3b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa827fff700 0 client.30499 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.916+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 shutdown_connections 2023-07-29T16:37:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.917+0000 7fa862554b80 1 -- 172.21.15.31:0/3474112007 wait complete. 2023-07-29T16:37:22.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.917+0000 7fa862554b80 20 client.30499 trim_cache size 0 max 16384 2023-07-29T16:37:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.939+0000 7fa862554b80 1 Processor -- start 2023-07-29T16:37:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 -- start start 2023-07-29T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be21b6b0 0x5602be3ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3ca140 con 0x5602be21b6b0 2023-07-29T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3bb600 con 0x5602be3c0cf0 2023-07-29T16:37:22.942 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.940+0000 7fa862554b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3c7820 con 0x5602be3bfdf0 2023-07-29T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58566/0 (socket says 172.21.15.31:58566) 2023-07-29T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 -- 172.21.15.31:0/2317585803 learned_addr learned my addr 172.21.15.31:0/2317585803 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:22.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84c99b700 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be21b6b0 0x5602be3ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 -- 172.21.15.31:0/2317585803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 msgr2=0x5602be3c3c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c3c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 -- 172.21.15.31:0/2317585803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be21b6b0 msgr2=0x5602be3ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be21b6b0 0x5602be3ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84d19c700 1 -- 172.21.15.31:0/2317585803 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602be1651f0 con 0x5602be3c0cf0 2023-07-29T16:37:22.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.941+0000 7fa84c99b700 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be21b6b0 0x5602be3ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:37:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.942+0000 7fa84d19c700 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c41d0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa840016000 tx=0x7fa840007600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2345e562f6969f19 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.942+0000 7fa82f7fe700 1 -- 172.21.15.31:0/2317585803 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84001e050 con 0x5602be3c0cf0 2023-07-29T16:37:22.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.942+0000 7fa82f7fe700 1 -- 172.21.15.31:0/2317585803 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa84001f040 con 0x5602be3c0cf0 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.942+0000 7fa82f7fe700 1 -- 172.21.15.31:0/2317585803 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84000f8f0 con 0x5602be3c0cf0 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.943+0000 7fa862554b80 1 -- 172.21.15.31:0/2317585803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 msgr2=0x5602be3c41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.943+0000 7fa862554b80 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c41d0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa840016000 tx=0x7fa840007600 comp rx=0 tx=0).stop 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 -- 172.21.15.31:0/2317585803 shutdown_connections 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3c0cf0 0x5602be3c41d0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be21b6b0 0x5602be3ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 --2- 172.21.15.31:0/2317585803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be3bfdf0 0x5602be3c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 -- 172.21.15.31:0/2317585803 >> 172.21.15.31:0/2317585803 conn(0x5602be3b4fd0 msgr2=0x5602be3c7110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 -- 172.21.15.31:0/2317585803 shutdown_connections 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.944+0000 7fa862554b80 1 -- 172.21.15.31:0/2317585803 wait complete. 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.945+0000 7fa862554b80 1 -- start start 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.945+0000 7fa862554b80 1 Processor -- start 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3bfdf0 0x5602be3ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3cc9b0 con 0x5602be3c0cf0 2023-07-29T16:37:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3ccb20 con 0x5602be3bfdf0 2023-07-29T16:37:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602be3ccce0 con 0x5602be21b6b0 2023-07-29T16:37:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa862554b80 10 client.0 _ll_register_callbacks cb 0x5602be3dbf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa84c99b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa84c99b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:42756/0 (socket says 172.21.15.31:42756) 2023-07-29T16:37:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.946+0000 7fa84c99b700 1 -- 172.21.15.31:0/1840002378 learned_addr learned my addr 172.21.15.31:0/1840002378 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa84d19c700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa847fff700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3bfdf0 0x5602be3ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa84c99b700 1 -- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3bfdf0 msgr2=0x5602be3ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa84c99b700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3bfdf0 0x5602be3ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa84c99b700 1 -- 172.21.15.31:0/1840002378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 msgr2=0x5602be3cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa84c99b700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.947+0000 7fa84c99b700 1 -- 172.21.15.31:0/1840002378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84810f700 con 0x5602be21b6b0 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.948+0000 7fa84c99b700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3cdd00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x5602be3d79a0 tx=0x7fa83800b540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ea3e55f19c24851 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.948+0000 7fa827fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:37:22.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.948+0000 7fa827fff700 1 -- 172.21.15.31:0/1840002378 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa838035e80 con 0x5602be21b6b0 2023-07-29T16:37:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.948+0000 7fa827fff700 1 -- 172.21.15.31:0/1840002378 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fa8380352c0 con 0x5602be21b6b0 2023-07-29T16:37:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.948+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa84000fa60 con 0x5602be21b6b0 2023-07-29T16:37:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa827fff700 1 -- 172.21.15.31:0/1840002378 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa83800b7f0 con 0x5602be21b6b0 2023-07-29T16:37:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 10 client.30502 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:22.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5602be13ed80 con 0x5602be21b6b0 2023-07-29T16:37:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa827fff700 1 -- 172.21.15.31:0/1840002378 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa838045030 con 0x5602be21b6b0 2023-07-29T16:37:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa827fff700 1 -- 172.21.15.31:0/1840002378 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fa838003e90 con 0x5602be21b6b0 2023-07-29T16:37:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 10 client.30502 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:22.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 10 client.30502 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 20 client.30502 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be427b10 0x5602be429f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 4 client.30502 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5602be309ce0 con 0x5602be427b10 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.949+0000 7fa847fff700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be427b10 0x5602be429f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.950+0000 7fa847fff700 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be427b10 0x5602be429f20 crc :-1 s=READY pgs=482 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61a8e521a84d7de server_cookie=3299bac4cb038326 in_seq=0 out_seq=0 2023-07-29T16:37:22.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:22.950+0000 7fa827fff700 10 client.30502 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:22 smithi156 ceph-mon[129245]: pgmap v1029: 97 pgs: 97 active+clean; 714 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.5 MiB/s wr, 538 op/s 2023-07-29T16:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.186+0000 7fa827fff700 1 -- 172.21.15.31:0/1840002378 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5602be309ce0 con 0x5602be427b10 2023-07-29T16:37:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.186+0000 7fa827fff700 10 client.30502 handle_command_reply: tid=0 2023-07-29T16:37:23.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:22 smithi031 ceph-mon[95504]: pgmap v1029: 97 pgs: 97 active+clean; 714 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.5 MiB/s wr, 538 op/s 2023-07-29T16:37:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.187+0000 7fa862554b80 1 client.30502 shutdown 2023-07-29T16:37:23.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.187+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 msgr2=0x5602be3cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.187+0000 7fa862554b80 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3cdd00 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x5602be3d79a0 tx=0x7fa83800b540 comp rx=0 tx=0).stop 2023-07-29T16:37:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 shutdown_connections 2023-07-29T16:37:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5602be427b10 0x5602be429f20 crc :-1 s=READY pgs=482 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5602be3c0cf0 0x5602be3cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602be21b6b0 0x5602be3cdd00 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 --2- 172.21.15.31:0/1840002378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602be3bfdf0 0x5602be3ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 >> 172.21.15.31:0/1840002378 conn(0x5602be3b4fd0 msgr2=0x5602be3c3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa827fff700 0 client.30502 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 shutdown_connections 2023-07-29T16:37:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.188+0000 7fa862554b80 1 -- 172.21.15.31:0/1840002378 wait complete. 2023-07-29T16:37:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.189+0000 7fa862554b80 20 client.30502 trim_cache size 0 max 16384 2023-07-29T16:37:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:37:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:37:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:37:23.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:37:23.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8300a0890 msgr2=0x7fa8300a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:23.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8300a0890 0x7fa8300a2d40 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fa8380036f0 tx=0x7fa838009ee0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 msgr2=0x7fa84806fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:23.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 0x7fa84806fc10 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00b530 tx=0x7fa83c00b840 comp rx=0 tx=0).stop 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 shutdown_connections 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fa8300a0890 0x7fa8300a2d40 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa848108510 0x7fa848070ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa848106990 0x7fa84806f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 --2- 172.21.15.31:0/1169843489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa848107b80 0x7fa84806fc10 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 >> 172.21.15.31:0/1169843489 conn(0x7fa8480759d0 msgr2=0x7fa84810b130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.191+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 shutdown_connections 2023-07-29T16:37:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:23.192+0000 7fa84ebff700 1 -- 172.21.15.31:0/1169843489 wait complete. 2023-07-29T16:37:23.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:22 smithi039 ceph-mon[128858]: pgmap v1029: 97 pgs: 97 active+clean; 714 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.5 MiB/s wr, 538 op/s 2023-07-29T16:37:23.344 INFO:tasks.workunit.client.0.smithi031.stdout: 10 273820 34.33 MB/sec execute 307 sec latency 155.350 ms 2023-07-29T16:37:24.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:37:24.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:23 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:24.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:37:24.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:23 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:24.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:37:24.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:23 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:24.344 INFO:tasks.workunit.client.0.smithi031.stdout: 10 274289 34.30 MB/sec execute 308 sec latency 237.593 ms 2023-07-29T16:37:25.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:24 smithi156 ceph-mon[129245]: pgmap v1030: 97 pgs: 97 active+clean; 710 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 11 MiB/s wr, 629 op/s 2023-07-29T16:37:25.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:24 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:24 smithi031 ceph-mon[95504]: pgmap v1030: 97 pgs: 97 active+clean; 710 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 11 MiB/s wr, 629 op/s 2023-07-29T16:37:25.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:24 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:24 smithi039 ceph-mon[128858]: pgmap v1030: 97 pgs: 97 active+clean; 710 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 11 MiB/s wr, 629 op/s 2023-07-29T16:37:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:24 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:25.345 INFO:tasks.workunit.client.0.smithi031.stdout: 10 274956 34.29 MB/sec execute 309 sec latency 99.863 ms 2023-07-29T16:37:26.345 INFO:tasks.workunit.client.0.smithi031.stdout: 10 275719 34.29 MB/sec execute 310 sec latency 111.558 ms 2023-07-29T16:37:26.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:37:26.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:37:26.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.440 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:26 smithi031 ceph-mon[95504]: pgmap v1031: 97 pgs: 97 active+clean; 698 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 679 op/s 2023-07-29T16:37:26.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:37:26.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:37:26.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:26 smithi039 ceph-mon[128858]: pgmap v1031: 97 pgs: 97 active+clean; 698 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 679 op/s 2023-07-29T16:37:26.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:37:26.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:37:26.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:37:26.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:26 smithi156 ceph-mon[129245]: pgmap v1031: 97 pgs: 97 active+clean; 698 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 11 MiB/s wr, 679 op/s 2023-07-29T16:37:27.346 INFO:tasks.workunit.client.0.smithi031.stdout: 10 276576 34.35 MB/sec execute 311 sec latency 52.849 ms 2023-07-29T16:37:28.346 INFO:tasks.workunit.client.0.smithi031.stdout: 10 277264 34.39 MB/sec execute 312 sec latency 95.044 ms 2023-07-29T16:37:28.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:28 smithi039 ceph-mon[128858]: pgmap v1032: 97 pgs: 97 active+clean; 680 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 700 op/s 2023-07-29T16:37:28.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:28 smithi156 ceph-mon[129245]: pgmap v1032: 97 pgs: 97 active+clean; 680 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 700 op/s 2023-07-29T16:37:28.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:28 smithi031 ceph-mon[95504]: pgmap v1032: 97 pgs: 97 active+clean; 680 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 700 op/s 2023-07-29T16:37:29.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 277697 34.35 MB/sec execute 313 sec latency 123.292 ms 2023-07-29T16:37:30.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 278250 34.31 MB/sec execute 314 sec latency 67.422 ms 2023-07-29T16:37:31.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:30 smithi039 ceph-mon[128858]: pgmap v1033: 97 pgs: 97 active+clean; 676 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 723 op/s 2023-07-29T16:37:31.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:30 smithi156 ceph-mon[129245]: pgmap v1033: 97 pgs: 97 active+clean; 676 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 723 op/s 2023-07-29T16:37:31.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:30 smithi031 ceph-mon[95504]: pgmap v1033: 97 pgs: 97 active+clean; 676 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 12 MiB/s wr, 723 op/s 2023-07-29T16:37:31.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 279059 34.32 MB/sec execute 315 sec latency 57.152 ms 2023-07-29T16:37:32.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 280104 34.42 MB/sec execute 316 sec latency 36.013 ms 2023-07-29T16:37:33.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:32 smithi039 ceph-mon[128858]: pgmap v1034: 97 pgs: 97 active+clean; 676 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 12 MiB/s wr, 700 op/s 2023-07-29T16:37:33.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:32 smithi156 ceph-mon[129245]: pgmap v1034: 97 pgs: 97 active+clean; 676 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 12 MiB/s wr, 700 op/s 2023-07-29T16:37:33.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:32 smithi031 ceph-mon[95504]: pgmap v1034: 97 pgs: 97 active+clean; 676 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 12 MiB/s wr, 700 op/s 2023-07-29T16:37:33.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 280717 34.43 MB/sec execute 317 sec latency 81.208 ms 2023-07-29T16:37:34.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 281195 34.39 MB/sec execute 318 sec latency 154.292 ms 2023-07-29T16:37:35.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:34 smithi039 ceph-mon[128858]: pgmap v1035: 97 pgs: 97 active+clean; 681 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 739 op/s 2023-07-29T16:37:35.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:34 smithi156 ceph-mon[129245]: pgmap v1035: 97 pgs: 97 active+clean; 681 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 739 op/s 2023-07-29T16:37:35.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:34 smithi031 ceph-mon[95504]: pgmap v1035: 97 pgs: 97 active+clean; 681 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 739 op/s 2023-07-29T16:37:35.347 INFO:tasks.workunit.client.0.smithi031.stdout: 10 281798 34.38 MB/sec execute 319 sec latency 95.140 ms 2023-07-29T16:37:36.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 282566 34.39 MB/sec execute 320 sec latency 72.861 ms 2023-07-29T16:37:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.454+0000 7f55ae7ff700 1 -- 172.21.15.31:0/1301204073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a80fe1e0 msgr2=0x7f55a80fe5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.454+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/1301204073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a80fe1e0 0x7f55a80fe5f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f559800b0a0 tx=0x7f559800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.454+0000 7f55ae7ff700 1 -- 172.21.15.31:0/1301204073 shutdown_connections 2023-07-29T16:37:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.454+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/1301204073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a8104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.454+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/1301204073 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a80fe1e0 0x7f55a80fe5f0 secure :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f559800b0a0 tx=0x7f559800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.455+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/1301204073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 0x7f55a80ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.455+0000 7f55ae7ff700 1 -- 172.21.15.31:0/1301204073 >> 172.21.15.31:0/1301204073 conn(0x7f55a80f9b70 msgr2=0x7f55a80fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.466+0000 7f55ae7ff700 1 -- 172.21.15.31:0/1301204073 shutdown_connections 2023-07-29T16:37:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.469+0000 7f55ae7ff700 1 -- 172.21.15.31:0/1301204073 wait complete. 2023-07-29T16:37:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.470+0000 7f55ae7ff700 1 Processor -- start 2023-07-29T16:37:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 -- start start 2023-07-29T16:37:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 0x7f55a810fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a81100d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a810e040 0x7f55a810e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a810ecb0 con 0x7f55a80ff310 2023-07-29T16:37:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a810ee20 con 0x7f55a80ffcc0 2023-07-29T16:37:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.471+0000 7f55ae7ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a81ad870 con 0x7f55a810e040 2023-07-29T16:37:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.472+0000 7f559f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a81100d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.472+0000 7f559f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a81100d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55442/0 (socket says 172.21.15.31:55442) 2023-07-29T16:37:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.472+0000 7f559f7fe700 1 -- 172.21.15.31:0/568227008 learned_addr learned my addr 172.21.15.31:0/568227008 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.472+0000 7f55acd9c700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a810e040 0x7f55a810e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f55a7fff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 0x7f55a810fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f559f7fe700 1 -- 172.21.15.31:0/568227008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a810e040 msgr2=0x7f55a810e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f559f7fe700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a810e040 0x7f55a810e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f559f7fe700 1 -- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 msgr2=0x7f55a810fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f559f7fe700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 0x7f55a810fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f559f7fe700 1 -- 172.21.15.31:0/568227008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5598009d00 con 0x7f55a80ffcc0 2023-07-29T16:37:36.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.473+0000 7f55a7fff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 0x7f55a810fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:37:36.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.476+0000 7f559f7fe700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a81100d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f559000b530 tx=0x7f559000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.476+0000 7f55a5ffb700 1 -- 172.21.15.31:0/568227008 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5590011830 con 0x7f55a80ffcc0 2023-07-29T16:37:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.476+0000 7f55a5ffb700 1 -- 172.21.15.31:0/568227008 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5590011e50 con 0x7f55a80ffcc0 2023-07-29T16:37:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.476+0000 7f55a5ffb700 1 -- 172.21.15.31:0/568227008 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559000f2e0 con 0x7f55a80ffcc0 2023-07-29T16:37:36.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.476+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a81adb50 con 0x7f55a80ffcc0 2023-07-29T16:37:36.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.477+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a81adfe0 con 0x7f55a80ffcc0 2023-07-29T16:37:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.483+0000 7f55c2158b80 1 Processor -- start 2023-07-29T16:37:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.491+0000 7f55c2158b80 1 -- start start 2023-07-29T16:37:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.492+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f71b6b0 0x559e4f8bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.492+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8c07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.492+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8c1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.492+0000 7f55c2158b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8179b0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.492+0000 7f55c2158b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f815370 con 0x559e4f8bfdf0 2023-07-29T16:37:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.492+0000 7f55c2158b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f810c60 con 0x559e4f71b6b0 2023-07-29T16:37:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.493+0000 7f55a5ffb700 1 -- 172.21.15.31:0/568227008 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5590010380 con 0x7f55a80ffcc0 2023-07-29T16:37:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.493+0000 7f55a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8c07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8c07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55454/0 (socket says 172.21.15.31:55454) 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 -- 172.21.15.31:0/3290078414 learned_addr learned my addr 172.21.15.31:0/3290078414 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 -- 172.21.15.31:0/3290078414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f71b6b0 msgr2=0x559e4f8bf8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f71b6b0 0x559e4f8bf8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 -- 172.21.15.31:0/3290078414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 msgr2=0x559e4f8c1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8c1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.495+0000 7f55a7fff700 1 -- 172.21.15.31:0/3290078414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e4f63ed80 con 0x559e4f8bfdf0 2023-07-29T16:37:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.496+0000 7f55a5ffb700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55940a08e0 0x7f55940a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.496+0000 7f55a7fff700 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8c07b0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f5598009fd0 tx=0x7f5598007e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8777aada65078929 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.496+0000 7f55a5ffb700 1 -- 172.21.15.31:0/568227008 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f55900cb5e0 con 0x7f55a80ffcc0 2023-07-29T16:37:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.499+0000 7f55acd9c700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55940a08e0 0x7f55940a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.500+0000 7f559e7fc700 1 -- 172.21.15.31:0/3290078414 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559800e040 con 0x559e4f8bfdf0 2023-07-29T16:37:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.501+0000 7f559e7fc700 1 -- 172.21.15.31:0/3290078414 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5598007860 con 0x559e4f8bfdf0 2023-07-29T16:37:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.501+0000 7f559e7fc700 1 -- 172.21.15.31:0/3290078414 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559801c860 con 0x559e4f8bfdf0 2023-07-29T16:37:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.502+0000 7f55c2158b80 1 -- 172.21.15.31:0/3290078414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 msgr2=0x559e4f8c07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.502+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8c07b0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f5598009fd0 tx=0x7f5598007e80 comp rx=0 tx=0).stop 2023-07-29T16:37:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.502+0000 7f55acd9c700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55940a08e0 0x7f55940a2d90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009970 tx=0x7f55a0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:36.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.507+0000 7f55c2158b80 1 -- 172.21.15.31:0/3290078414 shutdown_connections 2023-07-29T16:37:36.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.507+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8c1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.507+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f71b6b0 0x559e4f8bf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.507+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3290078414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8c07b0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.507+0000 7f55c2158b80 1 -- 172.21.15.31:0/3290078414 >> 172.21.15.31:0/3290078414 conn(0x559e4f8b4fd0 msgr2=0x559e4f8ba300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:36.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.511+0000 7f55c2158b80 1 -- 172.21.15.31:0/3290078414 shutdown_connections 2023-07-29T16:37:36.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.511+0000 7f55c2158b80 1 -- 172.21.15.31:0/3290078414 wait complete. 2023-07-29T16:37:36.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.512+0000 7f55c2158b80 1 -- start start 2023-07-29T16:37:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.512+0000 7f55c2158b80 1 Processor -- start 2023-07-29T16:37:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8d9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8da410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f7ad4b0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f7ad150 con 0x559e4f71b6b0 2023-07-29T16:37:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f6948c0 con 0x559e4f8bfdf0 2023-07-29T16:37:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.513+0000 7f55c2158b80 10 client.0 _ll_register_callbacks cb 0x559e4f7b65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.517+0000 7f55acd9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.517+0000 7f55acd9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8e1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37952/0 (socket says 172.21.15.31:37952) 2023-07-29T16:37:36.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.517+0000 7f55acd9c700 1 -- 172.21.15.31:0/4273771755 learned_addr learned my addr 172.21.15.31:0/4273771755 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.518+0000 7f55acd9c700 1 -- 172.21.15.31:0/4273771755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 msgr2=0x559e4f8da410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:37:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.518+0000 7f55acd9c700 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8da410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.518+0000 7f55acd9c700 1 -- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 msgr2=0x559e4f8d9ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:37:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.518+0000 7f55acd9c700 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8d9ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.518+0000 7f55acd9c700 1 -- 172.21.15.31:0/4273771755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a8104e60 con 0x559e4f8c0cf0 2023-07-29T16:37:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.518+0000 7f55acd9c700 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8e1450 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f55a000eff0 tx=0x7f55a0004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cdfb49f5244911 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.519+0000 7f55877fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:37:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.519+0000 7f55877fe700 1 -- 172.21.15.31:0/4273771755 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0015a70 con 0x559e4f8c0cf0 2023-07-29T16:37:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.519+0000 7f55877fe700 1 -- 172.21.15.31:0/4273771755 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f55a0015bd0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.519+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e4f83bcb0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.521+0000 7f55877fe700 1 -- 172.21.15.31:0/4273771755 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55a0015d90 con 0x559e4f8c0cf0 2023-07-29T16:37:36.523 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.521+0000 7f55c2158b80 10 client.20238 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:36.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.521+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559e4f627c00 con 0x559e4f8c0cf0 2023-07-29T16:37:36.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55877fe700 1 -- 172.21.15.31:0/4273771755 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e4f83bcb0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.524 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55877fe700 1 -- 172.21.15.31:0/4273771755 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f55a0028520 con 0x559e4f8c0cf0 2023-07-29T16:37:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55c2158b80 10 client.20238 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55c2158b80 10 client.20238 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55c2158b80 20 client.20238 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55c2158b80 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8e5d40 0x559e4f8e8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55c2158b80 4 client.20238 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:37:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e4f88ed10 con 0x559e4f8e5d40 2023-07-29T16:37:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.522+0000 7f55a7fff700 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8e5d40 0x559e4f8e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.524+0000 7f55a7fff700 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8e5d40 0x559e4f8e8150 crc :-1 s=READY pgs=483 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c64b2253c44c0286 server_cookie=96fb52963fee5d4 in_seq=0 out_seq=0 2023-07-29T16:37:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.527+0000 7f55877fe700 10 client.20238 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.527+0000 7f55877fe700 1 -- 172.21.15.31:0/4273771755 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559e4f88ed10 con 0x559e4f8e5d40 2023-07-29T16:37:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.527+0000 7f55877fe700 10 client.20238 handle_command_reply: tid=0 2023-07-29T16:37:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.527+0000 7f55c2158b80 1 client.20238 shutdown 2023-07-29T16:37:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.527+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 msgr2=0x559e4f8e1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.527+0000 7f55c2158b80 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8e1450 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f55a000eff0 tx=0x7f55a0004860 comp rx=0 tx=0).stop 2023-07-29T16:37:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 shutdown_connections 2023-07-29T16:37:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55c2158b80 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8e5d40 0x559e4f8e8150 crc :-1 s=READY pgs=483 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55c2158b80 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8c0cf0 0x559e4f8e1450 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55c2158b80 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8d9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55c2158b80 1 --2- 172.21.15.31:0/4273771755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8da410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 >> 172.21.15.31:0/4273771755 conn(0x559e4f8b4fd0 msgr2=0x559e4f8b9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.528+0000 7f55877fe700 0 client.20238 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:36.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.529+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 shutdown_connections 2023-07-29T16:37:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.529+0000 7f55c2158b80 1 -- 172.21.15.31:0/4273771755 wait complete. 2023-07-29T16:37:36.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.530+0000 7f55c2158b80 20 client.20238 trim_cache size 0 max 16384 2023-07-29T16:37:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.557+0000 7f55c2158b80 1 Processor -- start 2023-07-29T16:37:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.557+0000 7f55c2158b80 1 -- start start 2023-07-29T16:37:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.557+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 0x559e4f8df360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f71b6b0 0x559e4f8b9a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55c2158b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8c0490 con 0x559e4f71b6b0 2023-07-29T16:37:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55c2158b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8c01c0 con 0x559e4f8bfdf0 2023-07-29T16:37:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55c2158b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8bb600 con 0x559e4f8b4fd0 2023-07-29T16:37:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55a7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 0x559e4f8df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55a7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 0x559e4f8df360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41240/0 (socket says 172.21.15.31:41240) 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55a7fff700 1 -- 172.21.15.31:0/3969196744 learned_addr learned my addr 172.21.15.31:0/3969196744 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.558+0000 7f55acd9c700 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f559f7fe700 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f71b6b0 0x559e4f8b9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f55a7fff700 1 -- 172.21.15.31:0/3969196744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 msgr2=0x559e4f8e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f55a7fff700 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8e7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f55a7fff700 1 -- 172.21.15.31:0/3969196744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f71b6b0 msgr2=0x559e4f8b9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f55a7fff700 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f71b6b0 0x559e4f8b9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f55a7fff700 1 -- 172.21.15.31:0/3969196744 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e4f6651f0 con 0x559e4f8b4fd0 2023-07-29T16:37:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f559f7fe700 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f71b6b0 0x559e4f8b9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:37:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f55a7fff700 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 0x559e4f8df360 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f55980062a0 tx=0x7f5598007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d704016534066a1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f559effd700 1 -- 172.21.15.31:0/3969196744 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559800e040 con 0x559e4f8b4fd0 2023-07-29T16:37:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f559effd700 1 -- 172.21.15.31:0/3969196744 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5598007660 con 0x559e4f8b4fd0 2023-07-29T16:37:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.559+0000 7f559effd700 1 -- 172.21.15.31:0/3969196744 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559801c860 con 0x559e4f8b4fd0 2023-07-29T16:37:36.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 -- 172.21.15.31:0/3969196744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 msgr2=0x559e4f8df360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 0x559e4f8df360 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f55980062a0 tx=0x7f5598007e80 comp rx=0 tx=0).stop 2023-07-29T16:37:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 -- 172.21.15.31:0/3969196744 shutdown_connections 2023-07-29T16:37:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f8bfdf0 0x559e4f8e7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8b4fd0 0x559e4f8df360 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 --2- 172.21.15.31:0/3969196744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f71b6b0 0x559e4f8b9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.560+0000 7f55c2158b80 1 -- 172.21.15.31:0/3969196744 >> 172.21.15.31:0/3969196744 conn(0x559e4f8c13f0 msgr2=0x559e4f8d8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.561+0000 7f55c2158b80 1 -- 172.21.15.31:0/3969196744 shutdown_connections 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.561+0000 7f55c2158b80 1 -- 172.21.15.31:0/3969196744 wait complete. 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.561+0000 7f55c2158b80 1 -- start start 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 Processor -- start 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8dd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8b4fd0 0x559e4f8ddae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8de690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8debd0 con 0x559e4f8b4fd0 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8ded40 con 0x559e4f71b6b0 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e4f8def00 con 0x559e4f8bfdf0 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.562+0000 7f55c2158b80 10 client.0 _ll_register_callbacks cb 0x559e4f7b65c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8de690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8de690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:41256/0 (socket says 172.21.15.31:41256) 2023-07-29T16:37:36.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 -- 172.21.15.31:0/2730561949 learned_addr learned my addr 172.21.15.31:0/2730561949 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55a7fff700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8dd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f559f7fe700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8b4fd0 0x559e4f8ddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 -- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 msgr2=0x559e4f8dd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8dd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 -- 172.21.15.31:0/2730561949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8b4fd0 msgr2=0x559e4f8ddae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:36.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8b4fd0 0x559e4f8ddae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 -- 172.21.15.31:0/2730561949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e4f63ed80 con 0x559e4f8bfdf0 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f559f7fe700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8b4fd0 0x559e4f8ddae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55acd9c700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8de690 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a0015cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a095e46a1f38f68d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.563+0000 7f55877fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.564+0000 7f55877fe700 1 -- 172.21.15.31:0/2730561949 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0013860 con 0x559e4f8bfdf0 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.564+0000 7f55877fe700 1 -- 172.21.15.31:0/2730561949 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f55a00139c0 con 0x559e4f8bfdf0 2023-07-29T16:37:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.564+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5598007860 con 0x559e4f8bfdf0 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.565+0000 7f55877fe700 1 -- 172.21.15.31:0/2730561949 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55a0013b80 con 0x559e4f8bfdf0 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.565+0000 7f55c2158b80 10 client.30514 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.565+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559e4f63d250 con 0x559e4f8bfdf0 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.565+0000 7f55877fe700 1 -- 172.21.15.31:0/2730561949 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5598007860 con 0x559e4f8bfdf0 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55877fe700 1 -- 172.21.15.31:0/2730561949 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f55a0044cf0 con 0x559e4f8bfdf0 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55c2158b80 10 client.30514 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:36.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55c2158b80 10 client.30514 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55c2158b80 20 client.30514 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55c2158b80 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8c0cf0 0x559e4f93a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55c2158b80 4 client.30514 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559e4f809ce0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.566+0000 7f55a7fff700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8c0cf0 0x559e4f93a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.567+0000 7f55a7fff700 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8c0cf0 0x559e4f93a790 crc :-1 s=READY pgs=484 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cacf3942aa5809f server_cookie=15901aebf96ee3fd in_seq=0 out_seq=0 2023-07-29T16:37:36.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.567+0000 7f55877fe700 10 client.30514 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:36.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.952+0000 7f55877fe700 1 -- 172.21.15.31:0/2730561949 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559e4f809ce0 con 0x559e4f8c0cf0 2023-07-29T16:37:36.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.952+0000 7f55877fe700 10 client.30514 handle_command_reply: tid=0 2023-07-29T16:37:36.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.953+0000 7f55c2158b80 1 client.30514 shutdown 2023-07-29T16:37:36.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.955+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 msgr2=0x559e4f8de690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.955+0000 7f55c2158b80 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8de690 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f55a0000c00 tx=0x7f55a0015cd0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 shutdown_connections 2023-07-29T16:37:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559e4f8c0cf0 0x559e4f93a790 crc :-1 s=READY pgs=484 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e4f8bfdf0 0x559e4f8de690 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559e4f71b6b0 0x559e4f8dd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 --2- 172.21.15.31:0/2730561949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559e4f8b4fd0 0x559e4f8ddae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 >> 172.21.15.31:0/2730561949 conn(0x559e4f8c13f0 msgr2=0x559e4f8b8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:36.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55877fe700 0 client.30514 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:36.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 shutdown_connections 2023-07-29T16:37:36.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 1 -- 172.21.15.31:0/2730561949 wait complete. 2023-07-29T16:37:36.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.956+0000 7f55c2158b80 20 client.30514 trim_cache size 0 max 16384 2023-07-29T16:37:36.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:37:36.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:37:36.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:37:36.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:37:36.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.957+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55940a08e0 msgr2=0x7f55940a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.957+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55940a08e0 0x7f55940a2d90 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009970 tx=0x7f55a0009110 comp rx=0 tx=0).stop 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.957+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 msgr2=0x7f55a81100d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.957+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a81100d0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f559000b530 tx=0x7f559000b840 comp rx=0 tx=0).stop 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 shutdown_connections 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55940a08e0 0x7f55940a2d90 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a810e040 0x7f55a810e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a80ff310 0x7f55a810fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 --2- 172.21.15.31:0/568227008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a80ffcc0 0x7f55a81100d0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 >> 172.21.15.31:0/568227008 conn(0x7f55a80f9b70 msgr2=0x7f55a80fbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 shutdown_connections 2023-07-29T16:37:36.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:36.958+0000 7f55ae7ff700 1 -- 172.21.15.31:0/568227008 wait complete. 2023-07-29T16:37:37.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:36 smithi031 ceph-mon[95504]: pgmap v1036: 97 pgs: 97 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 541 op/s 2023-07-29T16:37:37.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:36 smithi039 ceph-mon[128858]: pgmap v1036: 97 pgs: 97 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 541 op/s 2023-07-29T16:37:37.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:36 smithi156 ceph-mon[129245]: pgmap v1036: 97 pgs: 97 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.6 MiB/s wr, 541 op/s 2023-07-29T16:37:37.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 283119 34.38 MB/sec execute 321 sec latency 218.544 ms 2023-07-29T16:37:38.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 283905 34.42 MB/sec execute 322 sec latency 58.371 ms 2023-07-29T16:37:38.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:38 smithi039 ceph-mon[128858]: pgmap v1037: 97 pgs: 97 active+clean; 691 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.6 MiB/s wr, 564 op/s 2023-07-29T16:37:38.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:38 smithi156 ceph-mon[129245]: pgmap v1037: 97 pgs: 97 active+clean; 691 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.6 MiB/s wr, 564 op/s 2023-07-29T16:37:38.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:38 smithi031 ceph-mon[95504]: pgmap v1037: 97 pgs: 97 active+clean; 691 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.6 MiB/s wr, 564 op/s 2023-07-29T16:37:39.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 284671 34.45 MB/sec execute 323 sec latency 47.773 ms 2023-07-29T16:37:40.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 285228 34.42 MB/sec execute 324 sec latency 87.680 ms 2023-07-29T16:37:41.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:40 smithi039 ceph-mon[128858]: pgmap v1038: 97 pgs: 97 active+clean; 686 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.2 MiB/s wr, 480 op/s 2023-07-29T16:37:41.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:40 smithi156 ceph-mon[129245]: pgmap v1038: 97 pgs: 97 active+clean; 686 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.2 MiB/s wr, 480 op/s 2023-07-29T16:37:41.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:40 smithi031 ceph-mon[95504]: pgmap v1038: 97 pgs: 97 active+clean; 686 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.2 MiB/s wr, 480 op/s 2023-07-29T16:37:41.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 285990 34.43 MB/sec execute 325 sec latency 81.764 ms 2023-07-29T16:37:42.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 286707 34.45 MB/sec execute 326 sec latency 82.558 ms 2023-07-29T16:37:43.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:42 smithi156 ceph-mon[129245]: pgmap v1039: 97 pgs: 97 active+clean; 690 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 571 op/s 2023-07-29T16:37:43.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:42 smithi031 ceph-mon[95504]: pgmap v1039: 97 pgs: 97 active+clean; 690 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 571 op/s 2023-07-29T16:37:43.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:42 smithi039 ceph-mon[128858]: pgmap v1039: 97 pgs: 97 active+clean; 690 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 571 op/s 2023-07-29T16:37:43.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 287496 34.46 MB/sec execute 327 sec latency 66.389 ms 2023-07-29T16:37:44.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 288239 34.50 MB/sec execute 328 sec latency 45.014 ms 2023-07-29T16:37:45.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:44 smithi156 ceph-mon[129245]: pgmap v1040: 97 pgs: 97 active+clean; 696 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 566 op/s 2023-07-29T16:37:45.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:44 smithi031 ceph-mon[95504]: pgmap v1040: 97 pgs: 97 active+clean; 696 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 566 op/s 2023-07-29T16:37:45.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:44 smithi039 ceph-mon[128858]: pgmap v1040: 97 pgs: 97 active+clean; 696 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 566 op/s 2023-07-29T16:37:45.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 288936 34.51 MB/sec execute 329 sec latency 81.090 ms 2023-07-29T16:37:46.348 INFO:tasks.workunit.client.0.smithi031.stdout: 10 289502 34.50 MB/sec execute 330 sec latency 68.192 ms 2023-07-29T16:37:47.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:46 smithi156 ceph-mon[129245]: pgmap v1041: 97 pgs: 97 active+clean; 692 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 503 op/s 2023-07-29T16:37:47.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:46 smithi031 ceph-mon[95504]: pgmap v1041: 97 pgs: 97 active+clean; 692 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 503 op/s 2023-07-29T16:37:47.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:46 smithi039 ceph-mon[128858]: pgmap v1041: 97 pgs: 97 active+clean; 692 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 503 op/s 2023-07-29T16:37:47.349 INFO:tasks.workunit.client.0.smithi031.stdout: 10 290164 34.50 MB/sec execute 331 sec latency 108.285 ms 2023-07-29T16:37:48.349 INFO:tasks.workunit.client.0.smithi031.stdout: 10 290807 34.50 MB/sec execute 332 sec latency 110.298 ms 2023-07-29T16:37:48.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:48 smithi039 ceph-mon[128858]: pgmap v1042: 97 pgs: 97 active+clean; 697 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.0 MiB/s wr, 551 op/s 2023-07-29T16:37:48.574 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:48 smithi156 ceph-mon[129245]: pgmap v1042: 97 pgs: 97 active+clean; 697 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.0 MiB/s wr, 551 op/s 2023-07-29T16:37:48.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:48 smithi031 ceph-mon[95504]: pgmap v1042: 97 pgs: 97 active+clean; 697 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.0 MiB/s wr, 551 op/s 2023-07-29T16:37:49.349 INFO:tasks.workunit.client.0.smithi031.stdout: 10 291547 34.49 MB/sec execute 333 sec latency 45.675 ms 2023-07-29T16:37:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.197+0000 7f756adff700 1 -- 172.21.15.31:0/2464052098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 msgr2=0x7f7564101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.197+0000 7f756adff700 1 --2- 172.21.15.31:0/2464052098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564101070 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f755400b0a0 tx=0x7f755400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 -- 172.21.15.31:0/2464052098 shutdown_connections 2023-07-29T16:37:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 --2- 172.21.15.31:0/2464052098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7564103f70 0x7f75641043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 --2- 172.21.15.31:0/2464052098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564101070 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 --2- 172.21.15.31:0/2464052098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75641015b0 0x7f7564103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 -- 172.21.15.31:0/2464052098 >> 172.21.15.31:0/2464052098 conn(0x7f75640fa840 msgr2=0x7f75640fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 -- 172.21.15.31:0/2464052098 shutdown_connections 2023-07-29T16:37:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.198+0000 7f756adff700 1 -- 172.21.15.31:0/2464052098 wait complete. 2023-07-29T16:37:50.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 Processor -- start 2023-07-29T16:37:50.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 -- start start 2023-07-29T16:37:50.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75641015b0 0x7f7564071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7564103f70 0x7f7564071e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75640725e0 con 0x7f7564103f70 2023-07-29T16:37:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564072750 con 0x7f75640fec50 2023-07-29T16:37:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.200+0000 7f756adff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7564072910 con 0x7f75641015b0 2023-07-29T16:37:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564070bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40650/0 (socket says 172.21.15.31:40650) 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 -- 172.21.15.31:0/3943130186 learned_addr learned my addr 172.21.15.31:0/3943130186 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 -- 172.21.15.31:0/3943130186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75641015b0 msgr2=0x7f7564071100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75641015b0 0x7f7564071100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 -- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7564103f70 msgr2=0x7f7564071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7564103f70 0x7f7564071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.201+0000 7f7568b9b700 1 -- 172.21.15.31:0/3943130186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554009d00 con 0x7f75640fec50 2023-07-29T16:37:50.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.202+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564070bc0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f755400b780 tx=0x7f755400be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:50.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.202+0000 7f75627fc700 1 -- 172.21.15.31:0/3943130186 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755400e040 con 0x7f75640fec50 2023-07-29T16:37:50.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.202+0000 7f75627fc700 1 -- 172.21.15.31:0/3943130186 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7554004e60 con 0x7f75640fec50 2023-07-29T16:37:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.202+0000 7f75627fc700 1 -- 172.21.15.31:0/3943130186 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540053c0 con 0x7f75640fec50 2023-07-29T16:37:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.202+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7564072b90 con 0x7f75640fec50 2023-07-29T16:37:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.202+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756410b5c0 con 0x7f75640fec50 2023-07-29T16:37:50.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.210+0000 7f757e75fb80 1 Processor -- start 2023-07-29T16:37:50.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.210+0000 7f757e75fb80 1 -- start start 2023-07-29T16:37:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc094646b0 0x55bc096088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09608df0 0x55bc096097b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc0960a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f757e75fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc095609b0 con 0x55bc094646b0 2023-07-29T16:37:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f757e75fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0955e370 con 0x55bc09608df0 2023-07-29T16:37:50.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f757e75fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc09559c60 con 0x55bc09609cf0 2023-07-29T16:37:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f75627fc700 1 -- 172.21.15.31:0/3943130186 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f755401daa0 con 0x7f75640fec50 2023-07-29T16:37:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f755bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc0960a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f755bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc0960a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53132/0 (socket says 172.21.15.31:53132) 2023-07-29T16:37:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.211+0000 7f755bfff700 1 -- 172.21.15.31:0/314895574 learned_addr learned my addr 172.21.15.31:0/314895574 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f7568b9b700 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09608df0 0x55bc096097b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f756939c700 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc094646b0 0x55bc096088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f755bfff700 1 -- 172.21.15.31:0/314895574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09608df0 msgr2=0x55bc096097b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f755bfff700 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09608df0 0x55bc096097b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f755bfff700 1 -- 172.21.15.31:0/314895574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc094646b0 msgr2=0x55bc096088b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f755bfff700 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc094646b0 0x55bc096088b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f755bfff700 1 -- 172.21.15.31:0/314895574 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc09387d80 con 0x55bc09609cf0 2023-07-29T16:37:50.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.212+0000 7f755bfff700 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc0960a160 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f754c009c00 tx=0x7f754c009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e89615e20898ba0a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:50.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.213+0000 7f755affd700 1 -- 172.21.15.31:0/314895574 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c009750 con 0x55bc09609cf0 2023-07-29T16:37:50.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.213+0000 7f755affd700 1 -- 172.21.15.31:0/314895574 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f754c004cf0 con 0x55bc09609cf0 2023-07-29T16:37:50.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.213+0000 7f755affd700 1 -- 172.21.15.31:0/314895574 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754c0056c0 con 0x55bc09609cf0 2023-07-29T16:37:50.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f75627fc700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75500a0960 0x7f75500a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f75627fc700 1 -- 172.21.15.31:0/3943130186 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f75540cc9b0 con 0x7f75640fec50 2023-07-29T16:37:50.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 -- 172.21.15.31:0/314895574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 msgr2=0x55bc0960a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc0960a160 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f754c009c00 tx=0x7f754c009f10 comp rx=0 tx=0).stop 2023-07-29T16:37:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f756939c700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75500a0960 0x7f75500a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 -- 172.21.15.31:0/314895574 shutdown_connections 2023-07-29T16:37:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc0960a160 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc094646b0 0x55bc096088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.219 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 --2- 172.21.15.31:0/314895574 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09608df0 0x55bc096097b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.214+0000 7f757e75fb80 1 -- 172.21.15.31:0/314895574 >> 172.21.15.31:0/314895574 conn(0x55bc095fdfd0 msgr2=0x55bc09603300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.215+0000 7f757e75fb80 1 -- 172.21.15.31:0/314895574 shutdown_connections 2023-07-29T16:37:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.215+0000 7f757e75fb80 1 -- 172.21.15.31:0/314895574 wait complete. 2023-07-29T16:37:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.216+0000 7f756939c700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75500a0960 0x7f75500a2e10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f755c009e00 tx=0x7f755c009430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:50.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.216+0000 7f757e75fb80 1 -- start start 2023-07-29T16:37:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.216+0000 7f757e75fb80 1 Processor -- start 2023-07-29T16:37:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc094646b0 0x55bc09611f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc096124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 0x55bc096194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc094f64b0 con 0x55bc09608df0 2023-07-29T16:37:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc094f6150 con 0x55bc09609cf0 2023-07-29T16:37:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc093dd8c0 con 0x55bc094646b0 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f757e75fb80 10 client.0 _ll_register_callbacks cb 0x55bc094ff5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f756939c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 0x55bc096194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f756939c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 0x55bc096194f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40656/0 (socket says 172.21.15.31:40656) 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.217+0000 7f756939c700 1 -- 172.21.15.31:0/3972011587 learned_addr learned my addr 172.21.15.31:0/3972011587 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc094646b0 0x55bc09611f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f756939c700 1 -- 172.21.15.31:0/3972011587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc094646b0 msgr2=0x55bc09611f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f756939c700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc094646b0 0x55bc09611f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f756939c700 1 -- 172.21.15.31:0/3972011587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 msgr2=0x55bc096124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f756939c700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc096124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f756939c700 1 -- 172.21.15.31:0/3972011587 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c0098b0 con 0x55bc09609cf0 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc094646b0 0x55bc09611f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.218+0000 7f756939c700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 0x55bc096194f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f755c0176c0 tx=0x7f755c004a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be369fadef830299 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f7543fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:37:50.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f7543fff700 1 -- 172.21.15.31:0/3972011587 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c01c070 con 0x55bc09609cf0 2023-07-29T16:37:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f7543fff700 1 -- 172.21.15.31:0/3972011587 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f755c022420 con 0x55bc09609cf0 2023-07-29T16:37:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc09584cb0 con 0x55bc09609cf0 2023-07-29T16:37:50.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f7543fff700 1 -- 172.21.15.31:0/3972011587 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f755c011070 con 0x55bc09609cf0 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f757e75fb80 10 client.30506 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.219+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55bc09370c00 con 0x55bc09609cf0 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.220+0000 7f7543fff700 1 -- 172.21.15.31:0/3972011587 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bc09584cb0 con 0x55bc09609cf0 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f7543fff700 1 -- 172.21.15.31:0/3972011587 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f755c0166a0 con 0x55bc09609cf0 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f757e75fb80 10 client.30506 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f757e75fb80 10 client.30506 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:50.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f757e75fb80 20 client.30506 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc0961dd30 0x55bc09620140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f757e75fb80 4 client.30506 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:37:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.221+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc095d7d10 con 0x55bc0961dd30 2023-07-29T16:37:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.222+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc0961dd30 0x55bc09620140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.223+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc0961dd30 0x55bc09620140 crc :-1 s=READY pgs=485 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b39b2ff6547e087 server_cookie=fb2e29f33ad38eac in_seq=0 out_seq=0 2023-07-29T16:37:50.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.223+0000 7f7543fff700 10 client.30506 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.226+0000 7f7543fff700 1 -- 172.21.15.31:0/3972011587 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55bc095d7d10 con 0x55bc0961dd30 2023-07-29T16:37:50.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.226+0000 7f7543fff700 10 client.30506 handle_command_reply: tid=0 2023-07-29T16:37:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.226+0000 7f757e75fb80 1 client.30506 shutdown 2023-07-29T16:37:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.226+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 msgr2=0x55bc096194f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.226+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 0x55bc096194f0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f755c0176c0 tx=0x7f755c004a70 comp rx=0 tx=0).stop 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 shutdown_connections 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc0961dd30 0x55bc09620140 crc :-1 s=READY pgs=485 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc09609cf0 0x55bc096194f0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc094646b0 0x55bc09611f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3972011587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc096124b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f7543fff700 0 client.30506 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 >> 172.21.15.31:0/3972011587 conn(0x55bc095fdfd0 msgr2=0x55bc09602a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:50.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.227+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 shutdown_connections 2023-07-29T16:37:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.228+0000 7f757e75fb80 1 -- 172.21.15.31:0/3972011587 wait complete. 2023-07-29T16:37:50.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.229+0000 7f757e75fb80 20 client.30506 trim_cache size 0 max 16384 2023-07-29T16:37:50.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.252+0000 7f757e75fb80 1 Processor -- start 2023-07-29T16:37:50.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.252+0000 7f757e75fb80 1 -- start start 2023-07-29T16:37:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09602a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc096194b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc096100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f757e75fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc095fda60 con 0x55bc09608df0 2023-07-29T16:37:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f757e75fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0961b210 con 0x55bc094646b0 2023-07-29T16:37:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f757e75fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc096045f0 con 0x55bc09609cf0 2023-07-29T16:37:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f7568b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09602a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f7568b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09602a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40662/0 (socket says 172.21.15.31:40662) 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f7568b9b700 1 -- 172.21.15.31:0/1483319110 learned_addr learned my addr 172.21.15.31:0/1483319110 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.253+0000 7f756939c700 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc096100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f756939c700 1 -- 172.21.15.31:0/1483319110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 msgr2=0x55bc09602a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f756939c700 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09602a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f756939c700 1 -- 172.21.15.31:0/1483319110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 msgr2=0x55bc096194b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f756939c700 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc096194b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f756939c700 1 -- 172.21.15.31:0/1483319110 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc093ae1f0 con 0x55bc09609cf0 2023-07-29T16:37:50.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f7568b9b700 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09602a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:37:50.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f756939c700 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc096100f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f755c017690 tx=0x7f755c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7c79a7423eb180 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:50.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f755b7fe700 1 -- 172.21.15.31:0/1483319110 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c01c070 con 0x55bc09609cf0 2023-07-29T16:37:50.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f755b7fe700 1 -- 172.21.15.31:0/1483319110 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f755c016860 con 0x55bc09609cf0 2023-07-29T16:37:50.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.254+0000 7f755b7fe700 1 -- 172.21.15.31:0/1483319110 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c022930 con 0x55bc09609cf0 2023-07-29T16:37:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 -- 172.21.15.31:0/1483319110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 msgr2=0x55bc096100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc096100f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f755c017690 tx=0x7f755c009560 comp rx=0 tx=0).stop 2023-07-29T16:37:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 -- 172.21.15.31:0/1483319110 shutdown_connections 2023-07-29T16:37:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc096100f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09602a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 --2- 172.21.15.31:0/1483319110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc096194b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.255+0000 7f757e75fb80 1 -- 172.21.15.31:0/1483319110 >> 172.21.15.31:0/1483319110 conn(0x55bc095fdfd0 msgr2=0x55bc09610c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.256+0000 7f757e75fb80 1 -- 172.21.15.31:0/1483319110 shutdown_connections 2023-07-29T16:37:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.256+0000 7f757e75fb80 1 -- 172.21.15.31:0/1483319110 wait complete. 2023-07-29T16:37:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.256+0000 7f757e75fb80 1 -- start start 2023-07-29T16:37:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 Processor -- start 2023-07-29T16:37:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09619110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc09615d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc09616280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc096167c0 con 0x55bc09608df0 2023-07-29T16:37:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc09616930 con 0x55bc094646b0 2023-07-29T16:37:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0966f8b0 con 0x55bc09609cf0 2023-07-29T16:37:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.257+0000 7f757e75fb80 10 client.0 _ll_register_callbacks cb 0x55bc094ff5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:37:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f7568b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09619110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f7568b9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09619110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40676/0 (socket says 172.21.15.31:40676) 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f7568b9b700 1 -- 172.21.15.31:0/3260716181 learned_addr learned my addr 172.21.15.31:0/3260716181 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f756939c700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc09616280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f755bfff700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc09615d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f756939c700 1 -- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 msgr2=0x55bc09619110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f756939c700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09619110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f756939c700 1 -- 172.21.15.31:0/3260716181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 msgr2=0x55bc09615d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f756939c700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc09615d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.258+0000 7f756939c700 1 -- 172.21.15.31:0/3260716181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f756410b5c0 con 0x55bc09609cf0 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09619110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f755bfff700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc09615d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f756939c700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc09616280 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f755c0095f0 tx=0x7f755c0164d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d481baf2d02dcdb6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f7543fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f7543fff700 1 -- 172.21.15.31:0/3260716181 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755c03b070 con 0x55bc09609cf0 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc09552ce0 con 0x55bc09609cf0 2023-07-29T16:37:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f7543fff700 1 -- 172.21.15.31:0/3260716181 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f755c0228b0 con 0x55bc09609cf0 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f7543fff700 1 -- 172.21.15.31:0/3260716181 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f755c01c070 con 0x55bc09609cf0 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f757e75fb80 10 client.30526 fetch_fsmap learned FSMap version 43 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.259+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55bc09370c00 con 0x55bc09609cf0 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f7543fff700 1 -- 172.21.15.31:0/3260716181 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f755c045020 con 0x55bc09609cf0 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f7543fff700 1 -- 172.21.15.31:0/3260716181 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f755c03e030 con 0x55bc09609cf0 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f757e75fb80 10 client.30526 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f757e75fb80 10 client.30526 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:37:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f757e75fb80 20 client.30526 populate_metadata read hostname 'smithi031' 2023-07-29T16:37:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc09670ea0 0x55bc096732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:37:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f757e75fb80 4 client.30526 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:37:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bc09552a00 con 0x55bc09670ea0 2023-07-29T16:37:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.260+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc09670ea0 0x55bc096732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:37:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.261+0000 7f7543fff700 10 client.30526 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.261+0000 7f7568b9b700 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc09670ea0 0x55bc096732b0 crc :-1 s=READY pgs=486 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be166d946cb20c0 server_cookie=cd1eeeb69f1bc725 in_seq=0 out_seq=0 2023-07-29T16:37:50.349 INFO:tasks.workunit.client.0.smithi031.stdout: 10 292394 34.53 MB/sec execute 334 sec latency 69.732 ms 2023-07-29T16:37:50.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.681+0000 7f7543fff700 1 -- 172.21.15.31:0/3260716181 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bc09552a00 con 0x55bc09670ea0 2023-07-29T16:37:50.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.681+0000 7f7543fff700 10 client.30526 handle_command_reply: tid=0 2023-07-29T16:37:50.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.681+0000 7f757e75fb80 1 client.30526 shutdown 2023-07-29T16:37:50.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 msgr2=0x55bc09616280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc09616280 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f755c0095f0 tx=0x7f755c0164d0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 shutdown_connections 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55bc09670ea0 0x55bc096732b0 crc :-1 s=READY pgs=486 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bc09609cf0 0x55bc09616280 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc094646b0 0x55bc09619110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f7543fff700 0 client.30526 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 --2- 172.21.15.31:0/3260716181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bc09608df0 0x55bc09615d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 >> 172.21.15.31:0/3260716181 conn(0x55bc095fdfd0 msgr2=0x55bc09601fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:50.686 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 shutdown_connections 2023-07-29T16:37:50.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 1 -- 172.21.15.31:0/3260716181 wait complete. 2023-07-29T16:37:50.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.682+0000 7f757e75fb80 20 client.30526 trim_cache size 0 max 16384 2023-07-29T16:37:50.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:37:50.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:37:50.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:37:50.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:37:50.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75500a0960 msgr2=0x7f75500a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75500a0960 0x7f75500a2e10 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f755c009e00 tx=0x7f755c009430 comp rx=0 tx=0).stop 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 msgr2=0x7f7564070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564070bc0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f755400b780 tx=0x7f755400be10 comp rx=0 tx=0).stop 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 shutdown_connections 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f75500a0960 0x7f75500a2e10 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7564103f70 0x7f7564071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75640fec50 0x7f7564070bc0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 --2- 172.21.15.31:0/3943130186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75641015b0 0x7f7564071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:37:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.684+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 >> 172.21.15.31:0/3943130186 conn(0x7f75640fa840 msgr2=0x7f75641022f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:37:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.685+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 shutdown_connections 2023-07-29T16:37:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:37:50.686+0000 7f756adff700 1 -- 172.21.15.31:0/3943130186 wait complete. 2023-07-29T16:37:51.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:50 smithi039 ceph-mon[128858]: pgmap v1043: 97 pgs: 97 active+clean; 705 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.8 MiB/s wr, 515 op/s 2023-07-29T16:37:51.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:50 smithi156 ceph-mon[129245]: pgmap v1043: 97 pgs: 97 active+clean; 705 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.8 MiB/s wr, 515 op/s 2023-07-29T16:37:51.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:50 smithi031 ceph-mon[95504]: pgmap v1043: 97 pgs: 97 active+clean; 705 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 3.8 MiB/s wr, 515 op/s 2023-07-29T16:37:51.354 INFO:tasks.workunit.client.0.smithi031.stdout: 10 292958 34.55 MB/sec execute 335 sec latency 275.302 ms 2023-07-29T16:37:52.349 INFO:tasks.workunit.client.0.smithi031.stdout: 10 293609 34.55 MB/sec execute 336 sec latency 89.681 ms 2023-07-29T16:37:53.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:52 smithi039 ceph-mon[128858]: pgmap v1044: 97 pgs: 97 active+clean; 710 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 547 op/s 2023-07-29T16:37:53.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:52 smithi156 ceph-mon[129245]: pgmap v1044: 97 pgs: 97 active+clean; 710 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 547 op/s 2023-07-29T16:37:53.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:52 smithi031 ceph-mon[95504]: pgmap v1044: 97 pgs: 97 active+clean; 710 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 547 op/s 2023-07-29T16:37:53.350 INFO:tasks.workunit.client.0.smithi031.stdout: 10 294154 34.52 MB/sec execute 337 sec latency 85.269 ms 2023-07-29T16:37:54.350 INFO:tasks.workunit.client.0.smithi031.stdout: 10 294706 34.52 MB/sec execute 338 sec latency 230.375 ms 2023-07-29T16:37:55.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:54 smithi156 ceph-mon[129245]: pgmap v1045: 97 pgs: 97 active+clean; 709 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 552 op/s 2023-07-29T16:37:55.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:54 smithi031 ceph-mon[95504]: pgmap v1045: 97 pgs: 97 active+clean; 709 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 552 op/s 2023-07-29T16:37:55.284 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:54 smithi039 ceph-mon[128858]: pgmap v1045: 97 pgs: 97 active+clean; 709 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.9 MiB/s wr, 552 op/s 2023-07-29T16:37:55.350 INFO:tasks.workunit.client.0.smithi031.stdout: 10 295098 34.47 MB/sec execute 339 sec latency 302.656 ms 2023-07-29T16:37:56.350 INFO:tasks.workunit.client.0.smithi031.stdout: 10 295106 34.37 MB/sec execute 340 sec latency 1266.211 ms 2023-07-29T16:37:57.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:56 smithi031 ceph-mon[95504]: pgmap v1046: 97 pgs: 97 active+clean; 715 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.9 MiB/s wr, 593 op/s 2023-07-29T16:37:57.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:56 smithi039 ceph-mon[128858]: pgmap v1046: 97 pgs: 97 active+clean; 715 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.9 MiB/s wr, 593 op/s 2023-07-29T16:37:57.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:56 smithi156 ceph-mon[129245]: pgmap v1046: 97 pgs: 97 active+clean; 715 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.9 MiB/s wr, 593 op/s 2023-07-29T16:37:57.351 INFO:tasks.workunit.client.0.smithi031.stdout: 10 295417 34.31 MB/sec execute 341 sec latency 1371.673 ms 2023-07-29T16:37:58.351 INFO:tasks.workunit.client.0.smithi031.stdout: 10 295788 34.25 MB/sec execute 342 sec latency 182.654 ms 2023-07-29T16:37:58.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:37:58 smithi039 ceph-mon[128858]: pgmap v1047: 97 pgs: 97 active+clean; 761 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 9.3 MiB/s wr, 693 op/s 2023-07-29T16:37:58.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:37:58 smithi156 ceph-mon[129245]: pgmap v1047: 97 pgs: 97 active+clean; 761 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 9.3 MiB/s wr, 693 op/s 2023-07-29T16:37:58.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:37:58 smithi031 ceph-mon[95504]: pgmap v1047: 97 pgs: 97 active+clean; 761 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 9.3 MiB/s wr, 693 op/s 2023-07-29T16:37:59.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 296363 34.25 MB/sec execute 343 sec latency 61.249 ms 2023-07-29T16:38:00.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 297233 34.29 MB/sec execute 344 sec latency 73.307 ms 2023-07-29T16:38:01.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:00 smithi031 ceph-mon[95504]: pgmap v1048: 97 pgs: 97 active+clean; 773 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 763 op/s 2023-07-29T16:38:01.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:00 smithi039 ceph-mon[128858]: pgmap v1048: 97 pgs: 97 active+clean; 773 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 763 op/s 2023-07-29T16:38:01.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:00 smithi156 ceph-mon[129245]: pgmap v1048: 97 pgs: 97 active+clean; 773 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 763 op/s 2023-07-29T16:38:01.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 297656 34.28 MB/sec execute 345 sec latency 282.816 ms 2023-07-29T16:38:02.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 298101 34.26 MB/sec execute 346 sec latency 180.456 ms 2023-07-29T16:38:03.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:02 smithi031 ceph-mon[95504]: pgmap v1049: 97 pgs: 97 active+clean; 742 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 16 MiB/s wr, 829 op/s 2023-07-29T16:38:03.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:02 smithi039 ceph-mon[128858]: pgmap v1049: 97 pgs: 97 active+clean; 742 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 16 MiB/s wr, 829 op/s 2023-07-29T16:38:03.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:02 smithi156 ceph-mon[129245]: pgmap v1049: 97 pgs: 97 active+clean; 742 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 16 MiB/s wr, 829 op/s 2023-07-29T16:38:03.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 298652 34.24 MB/sec execute 347 sec latency 70.974 ms 2023-07-29T16:38:04.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 298961 34.17 MB/sec execute 348 sec latency 301.187 ms 2023-07-29T16:38:05.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:04 smithi031 ceph-mon[95504]: pgmap v1050: 97 pgs: 97 active+clean; 694 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 18 MiB/s wr, 980 op/s 2023-07-29T16:38:05.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:04 smithi039 ceph-mon[128858]: pgmap v1050: 97 pgs: 97 active+clean; 694 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 18 MiB/s wr, 980 op/s 2023-07-29T16:38:05.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:04 smithi156 ceph-mon[129245]: pgmap v1050: 97 pgs: 97 active+clean; 694 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 18 MiB/s wr, 980 op/s 2023-07-29T16:38:05.352 INFO:tasks.workunit.client.0.smithi031.stdout: 10 299429 34.14 MB/sec execute 349 sec latency 138.467 ms 2023-07-29T16:38:06.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 300277 34.16 MB/sec execute 350 sec latency 51.350 ms 2023-07-29T16:38:06.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.482+0000 7f9840dff700 1 -- 172.21.15.31:0/1127725193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c100e60 msgr2=0x7f983c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.482+0000 7f9840dff700 1 --2- 172.21.15.31:0/1127725193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c100e60 0x7f983c103240 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f982c009990 tx=0x7f982c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 -- 172.21.15.31:0/1127725193 shutdown_connections 2023-07-29T16:38:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 --2- 172.21.15.31:0/1127725193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1060a0 0x7f983c108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 --2- 172.21.15.31:0/1127725193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c100e60 0x7f983c103240 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 --2- 172.21.15.31:0/1127725193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c103780 0x7f983c105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 -- 172.21.15.31:0/1127725193 >> 172.21.15.31:0/1127725193 conn(0x7f983c0fa840 msgr2=0x7f983c0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 -- 172.21.15.31:0/1127725193 shutdown_connections 2023-07-29T16:38:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.483+0000 7f9840dff700 1 -- 172.21.15.31:0/1127725193 wait complete. 2023-07-29T16:38:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.484+0000 7f9840dff700 1 Processor -- start 2023-07-29T16:38:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 -- start start 2023-07-29T16:38:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 0x7f983c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c103780 0x7f983c198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1060a0 0x7f983c199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c199df0 con 0x7f983c100e60 2023-07-29T16:38:06.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c199f60 con 0x7f983c103780 2023-07-29T16:38:06.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.485+0000 7f9840dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c19a120 con 0x7f983c1060a0 2023-07-29T16:38:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.486+0000 7f983a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 0x7f983c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.486+0000 7f983a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 0x7f983c1983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48088/0 (socket says 172.21.15.31:48088) 2023-07-29T16:38:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.486+0000 7f983a59c700 1 -- 172.21.15.31:0/3369790285 learned_addr learned my addr 172.21.15.31:0/3369790285 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.486+0000 7f9839d9b700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c103780 0x7f983c198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.486+0000 7f983ad9d700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1060a0 0x7f983c199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.487+0000 7f983a59c700 1 -- 172.21.15.31:0/3369790285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1060a0 msgr2=0x7f983c199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.487+0000 7f983a59c700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1060a0 0x7f983c199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.487+0000 7f983a59c700 1 -- 172.21.15.31:0/3369790285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c103780 msgr2=0x7f983c198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.487+0000 7f983a59c700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c103780 0x7f983c198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.487+0000 7f983a59c700 1 -- 172.21.15.31:0/3369790285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c0095f0 con 0x7f983c100e60 2023-07-29T16:38:06.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.487+0000 7f983a59c700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 0x7f983c1983d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f982c00b680 tx=0x7f982c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.488+0000 7f982b7fe700 1 -- 172.21.15.31:0/3369790285 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c01d070 con 0x7f983c100e60 2023-07-29T16:38:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.488+0000 7f982b7fe700 1 -- 172.21.15.31:0/3369790285 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f982c021410 con 0x7f983c100e60 2023-07-29T16:38:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.488+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c19a3a0 con 0x7f983c100e60 2023-07-29T16:38:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.488+0000 7f982b7fe700 1 -- 172.21.15.31:0/3369790285 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c017650 con 0x7f983c100e60 2023-07-29T16:38:06.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.488+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c1a0bc0 con 0x7f983c100e60 2023-07-29T16:38:06.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.493+0000 7f982b7fe700 1 -- 172.21.15.31:0/3369790285 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f982c021580 con 0x7f983c100e60 2023-07-29T16:38:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.494+0000 7f9854670b80 1 Processor -- start 2023-07-29T16:38:06.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.495+0000 7f982b7fe700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98240a0810 0x7f98240a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.495+0000 7f982b7fe700 1 -- 172.21.15.31:0/3369790285 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f982c0ccf40 con 0x7f983c100e60 2023-07-29T16:38:06.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.495+0000 7f983ad9d700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98240a0810 0x7f98240a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 -- start start 2023-07-29T16:38:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a1a46b0 0x558b3a352230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a350e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 0x558b3a351820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a2a09b0 con 0x558b3a352770 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a29e370 con 0x558b3a351380 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f9854670b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a299c60 con 0x558b3a1a46b0 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f983ad9d700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98240a0810 0x7f98240a2cc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f98340093a0 tx=0x7f983400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f983ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 0x558b3a351820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f983ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 0x558b3a351820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56870/0 (socket says 172.21.15.31:56870) 2023-07-29T16:38:06.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f983ad9d700 1 -- 172.21.15.31:0/2544067192 learned_addr learned my addr 172.21.15.31:0/2544067192 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:06.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.496+0000 7f983a59c700 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a1a46b0 0x558b3a352230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.497+0000 7f9839d9b700 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a350e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.497+0000 7f983ad9d700 1 -- 172.21.15.31:0/2544067192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a1a46b0 msgr2=0x558b3a352230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.497+0000 7f983ad9d700 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a1a46b0 0x558b3a352230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.497+0000 7f983ad9d700 1 -- 172.21.15.31:0/2544067192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 msgr2=0x558b3a350e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.497+0000 7f983ad9d700 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a350e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.497+0000 7f983ad9d700 1 -- 172.21.15.31:0/2544067192 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b3a0c7d80 con 0x558b3a351380 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.498+0000 7f983ad9d700 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 0x558b3a351820 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9834009690 tx=0x7f9834012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d01d99584e0a030c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.498+0000 7f9828ff9700 1 -- 172.21.15.31:0/2544067192 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834007e20 con 0x558b3a351380 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.498+0000 7f9828ff9700 1 -- 172.21.15.31:0/2544067192 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98340218c0 con 0x558b3a351380 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.498+0000 7f9828ff9700 1 -- 172.21.15.31:0/2544067192 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834014980 con 0x558b3a351380 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 -- 172.21.15.31:0/2544067192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 msgr2=0x558b3a351820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 0x558b3a351820 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f9834009690 tx=0x7f9834012000 comp rx=0 tx=0).stop 2023-07-29T16:38:06.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 -- 172.21.15.31:0/2544067192 shutdown_connections 2023-07-29T16:38:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a351380 0x558b3a351820 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a1a46b0 0x558b3a352230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 --2- 172.21.15.31:0/2544067192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a350e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 -- 172.21.15.31:0/2544067192 >> 172.21.15.31:0/2544067192 conn(0x558b3a33dfd0 msgr2=0x558b3a343310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 -- 172.21.15.31:0/2544067192 shutdown_connections 2023-07-29T16:38:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.499+0000 7f9854670b80 1 -- 172.21.15.31:0/2544067192 wait complete. 2023-07-29T16:38:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.500+0000 7f9854670b80 1 -- start start 2023-07-29T16:38:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.500+0000 7f9854670b80 1 Processor -- start 2023-07-29T16:38:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a349bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a34a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a34ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a2364b0 con 0x558b3a352770 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a236150 con 0x558b3a1a46b0 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a11d8c0 con 0x558b3a351380 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9854670b80 10 client.0 _ll_register_callbacks cb 0x558b3a23f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9839d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a34a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9839d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a34a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55458/0 (socket says 172.21.15.31:55458) 2023-07-29T16:38:06.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.501+0000 7f9839d9b700 1 -- 172.21.15.31:0/1687412808 learned_addr learned my addr 172.21.15.31:0/1687412808 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.502+0000 7f9839d9b700 1 -- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 msgr2=0x558b3a349bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:38:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.502+0000 7f9839d9b700 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a349bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.502+0000 7f9839d9b700 1 -- 172.21.15.31:0/1687412808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 msgr2=0x558b3a34ef10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:38:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.502+0000 7f9839d9b700 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a34ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.502+0000 7f9839d9b700 1 -- 172.21.15.31:0/1687412808 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9834005e70 con 0x558b3a351380 2023-07-29T16:38:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9839d9b700 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a34a0f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f983000bac0 tx=0x7f983000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff73de4e2244465c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:06.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9821ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:38:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9821ffb700 1 -- 172.21.15.31:0/1687412808 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983001b440 con 0x558b3a351380 2023-07-29T16:38:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b3a2c4cb0 con 0x558b3a351380 2023-07-29T16:38:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9821ffb700 1 -- 172.21.15.31:0/1687412808 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f983001b5a0 con 0x558b3a351380 2023-07-29T16:38:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9821ffb700 1 -- 172.21.15.31:0/1687412808 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f983001b760 con 0x558b3a351380 2023-07-29T16:38:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9854670b80 10 client.30535 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:06.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.503+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x558b3a0b0c00 con 0x558b3a351380 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.504+0000 7f9821ffb700 1 -- 172.21.15.31:0/1687412808 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558b3a2c4cb0 con 0x558b3a351380 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.504+0000 7f9821ffb700 1 -- 172.21.15.31:0/1687412808 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9830011a10 con 0x558b3a351380 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.504+0000 7f9854670b80 10 client.30535 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.504+0000 7f9854670b80 10 client.30535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.504+0000 7f9854670b80 20 client.30535 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.504+0000 7f9854670b80 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a37fec0 0x558b3a3822d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.505+0000 7f9854670b80 4 client.30535 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:38:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.505+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b3a317d10 con 0x558b3a37fec0 2023-07-29T16:38:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.505+0000 7f983a59c700 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a37fec0 0x558b3a3822d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.505+0000 7f983a59c700 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a37fec0 0x558b3a3822d0 crc :-1 s=READY pgs=487 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f29d15de83a00ca server_cookie=92d54a7e16c86494 in_seq=0 out_seq=0 2023-07-29T16:38:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.505+0000 7f9821ffb700 10 client.30535 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9821ffb700 1 -- 172.21.15.31:0/1687412808 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558b3a317d10 con 0x558b3a37fec0 2023-07-29T16:38:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9821ffb700 10 client.30535 handle_command_reply: tid=0 2023-07-29T16:38:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 client.30535 shutdown 2023-07-29T16:38:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 msgr2=0x558b3a34a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a34a0f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f983000bac0 tx=0x7f983000be80 comp rx=0 tx=0).stop 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 shutdown_connections 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a37fec0 0x558b3a3822d0 crc :-1 s=READY pgs=487 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a34ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a349bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 --2- 172.21.15.31:0/1687412808 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a34a0f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 >> 172.21.15.31:0/1687412808 conn(0x558b3a33dfd0 msgr2=0x558b3a342a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:06.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.508+0000 7f9821ffb700 0 client.30535 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.509+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 shutdown_connections 2023-07-29T16:38:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.509+0000 7f9854670b80 1 -- 172.21.15.31:0/1687412808 wait complete. 2023-07-29T16:38:06.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.509+0000 7f9854670b80 20 client.30535 trim_cache size 0 max 16384 2023-07-29T16:38:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.528+0000 7f9854670b80 1 Processor -- start 2023-07-29T16:38:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 -- start start 2023-07-29T16:38:06.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a3521f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a351380 0x558b3a350db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a352770 0x558b3a34ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a348e50 con 0x558b3a351380 2023-07-29T16:38:06.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a352e70 con 0x558b3a1a46b0 2023-07-29T16:38:06.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9854670b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a344b70 con 0x558b3a352770 2023-07-29T16:38:06.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f983ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a352770 0x558b3a34ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f983a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a3521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f983ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a352770 0x558b3a34ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55470/0 (socket says 172.21.15.31:55470) 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f983ad9d700 1 -- 172.21.15.31:0/4050994324 learned_addr learned my addr 172.21.15.31:0/4050994324 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f983a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a3521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:56900/0 (socket says 172.21.15.31:56900) 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.529+0000 7f9839d9b700 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a351380 0x558b3a350db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f983a59c700 1 -- 172.21.15.31:0/4050994324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a352770 msgr2=0x558b3a34ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:06.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f983a59c700 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a352770 0x558b3a34ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f983a59c700 1 -- 172.21.15.31:0/4050994324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a351380 msgr2=0x558b3a350db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f983a59c700 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a351380 0x558b3a350db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f983a59c700 1 -- 172.21.15.31:0/4050994324 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b3a0ee1f0 con 0x558b3a1a46b0 2023-07-29T16:38:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f9839d9b700 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a351380 0x558b3a350db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:38:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f983a59c700 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a3521f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f982c0ca9e0 tx=0x7f982c0a0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18a6a2dc6583ce2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f98297fa700 1 -- 172.21.15.31:0/4050994324 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c0caea0 con 0x558b3a1a46b0 2023-07-29T16:38:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f98297fa700 1 -- 172.21.15.31:0/4050994324 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f982c0a2a90 con 0x558b3a1a46b0 2023-07-29T16:38:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.530+0000 7f98297fa700 1 -- 172.21.15.31:0/4050994324 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c099e20 con 0x558b3a1a46b0 2023-07-29T16:38:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 -- 172.21.15.31:0/4050994324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 msgr2=0x558b3a3521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a3521f0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f982c0ca9e0 tx=0x7f982c0a0040 comp rx=0 tx=0).stop 2023-07-29T16:38:06.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 -- 172.21.15.31:0/4050994324 shutdown_connections 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a352770 0x558b3a34ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a3521f0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 --2- 172.21.15.31:0/4050994324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a351380 0x558b3a350db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.531+0000 7f9854670b80 1 -- 172.21.15.31:0/4050994324 >> 172.21.15.31:0/4050994324 conn(0x558b3a33dfd0 msgr2=0x558b3a342a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.532+0000 7f9854670b80 1 -- 172.21.15.31:0/4050994324 shutdown_connections 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.532+0000 7f9854670b80 1 -- 172.21.15.31:0/4050994324 wait complete. 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.532+0000 7f9854670b80 1 -- start start 2023-07-29T16:38:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 Processor -- start 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a34eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a3cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a3d0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a3d0770 con 0x558b3a352770 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a3d08e0 con 0x558b3a1a46b0 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b3a3d0aa0 con 0x558b3a351380 2023-07-29T16:38:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.533+0000 7f9854670b80 10 client.0 _ll_register_callbacks cb 0x558b3a23f5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a3cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a3cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55476/0 (socket says 172.21.15.31:55476) 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 -- 172.21.15.31:0/1371788966 learned_addr learned my addr 172.21.15.31:0/1371788966 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f983a59c700 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a34eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 -- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 msgr2=0x558b3a34eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a34eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 -- 172.21.15.31:0/1371788966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 msgr2=0x558b3a3d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a3d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 -- 172.21.15.31:0/1371788966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c1a0bc0 con 0x558b3a351380 2023-07-29T16:38:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9839d9b700 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a3cda40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x558b3a340070 tx=0x7f9830003960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce567defe7759b84 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.534+0000 7f9821ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:38:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.535+0000 7f9821ffb700 1 -- 172.21.15.31:0/1371788966 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983002a070 con 0x558b3a351380 2023-07-29T16:38:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.536+0000 7f9821ffb700 1 -- 172.21.15.31:0/1371788966 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f9830025ca0 con 0x558b3a351380 2023-07-29T16:38:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.536+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98340218c0 con 0x558b3a351380 2023-07-29T16:38:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.536+0000 7f9821ffb700 1 -- 172.21.15.31:0/1371788966 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9830025e60 con 0x558b3a351380 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.536+0000 7f9854670b80 10 client.30541 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.536+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x558b3a0c6250 con 0x558b3a351380 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.536+0000 7f9821ffb700 1 -- 172.21.15.31:0/1371788966 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98340218c0 con 0x558b3a351380 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9821ffb700 1 -- 172.21.15.31:0/1371788966 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f9830004af0 con 0x558b3a351380 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9854670b80 10 client.30541 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9854670b80 10 client.30541 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9854670b80 20 client.30541 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9854670b80 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a3d2410 0x558b3a3d4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9854670b80 4 client.30541 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:38:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.537+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558b3a292ce0 con 0x558b3a3d2410 2023-07-29T16:38:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.538+0000 7f983a59c700 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a3d2410 0x558b3a3d4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.538+0000 7f983a59c700 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a3d2410 0x558b3a3d4820 crc :-1 s=READY pgs=488 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b6ee91a2f938bc server_cookie=66054df2346eda7 in_seq=0 out_seq=0 2023-07-29T16:38:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.538+0000 7f9821ffb700 10 client.30541 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:06.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.859+0000 7f9821ffb700 1 -- 172.21.15.31:0/1371788966 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558b3a292ce0 con 0x558b3a3d2410 2023-07-29T16:38:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.859+0000 7f9821ffb700 10 client.30541 handle_command_reply: tid=0 2023-07-29T16:38:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.859+0000 7f9854670b80 1 client.30541 shutdown 2023-07-29T16:38:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 msgr2=0x558b3a3cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a3cda40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x558b3a340070 tx=0x7f9830003960 comp rx=0 tx=0).stop 2023-07-29T16:38:06.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 shutdown_connections 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558b3a3d2410 0x558b3a3d4820 crc :-1 s=READY pgs=488 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558b3a352770 0x558b3a3d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558b3a1a46b0 0x558b3a34eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 --2- 172.21.15.31:0/1371788966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b3a351380 0x558b3a3cda40 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 >> 172.21.15.31:0/1371788966 conn(0x558b3a33dfd0 msgr2=0x558b3a3583b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9821ffb700 0 client.30541 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:06.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 shutdown_connections 2023-07-29T16:38:06.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 1 -- 172.21.15.31:0/1371788966 wait complete. 2023-07-29T16:38:06.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.860+0000 7f9854670b80 20 client.30541 trim_cache size 0 max 16384 2023-07-29T16:38:06.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:38:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:38:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:38:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:38:06.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98240a0810 msgr2=0x7f98240a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98240a0810 0x7f98240a2cc0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f98340093a0 tx=0x7f983400b040 comp rx=0 tx=0).stop 2023-07-29T16:38:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 msgr2=0x7f983c1983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 0x7f983c1983d0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f982c00b680 tx=0x7f982c005740 comp rx=0 tx=0).stop 2023-07-29T16:38:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 shutdown_connections 2023-07-29T16:38:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f98240a0810 0x7f98240a2cc0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f983c1060a0 0x7f983c199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f983c100e60 0x7f983c1983d0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 --2- 172.21.15.31:0/3369790285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f983c103780 0x7f983c198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 >> 172.21.15.31:0/3369790285 conn(0x7f983c0fa840 msgr2=0x7f983c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.861+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 shutdown_connections 2023-07-29T16:38:06.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:06.862+0000 7f9840dff700 1 -- 172.21.15.31:0/3369790285 wait complete. 2023-07-29T16:38:07.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:06 smithi031 ceph-mon[95504]: pgmap v1051: 97 pgs: 97 active+clean; 693 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 18 MiB/s wr, 883 op/s 2023-07-29T16:38:07.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:06 smithi039 ceph-mon[128858]: pgmap v1051: 97 pgs: 97 active+clean; 693 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 18 MiB/s wr, 883 op/s 2023-07-29T16:38:07.324 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:06 smithi156 ceph-mon[129245]: pgmap v1051: 97 pgs: 97 active+clean; 693 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 18 MiB/s wr, 883 op/s 2023-07-29T16:38:07.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 300996 34.20 MB/sec execute 351 sec latency 131.238 ms 2023-07-29T16:38:08.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 301623 34.20 MB/sec execute 352 sec latency 73.264 ms 2023-07-29T16:38:08.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:08 smithi039 ceph-mon[128858]: pgmap v1052: 97 pgs: 97 active+clean; 691 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 19 MiB/s wr, 900 op/s 2023-07-29T16:38:08.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:08 smithi156 ceph-mon[129245]: pgmap v1052: 97 pgs: 97 active+clean; 691 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 19 MiB/s wr, 900 op/s 2023-07-29T16:38:08.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:08 smithi031 ceph-mon[95504]: pgmap v1052: 97 pgs: 97 active+clean; 691 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 19 MiB/s wr, 900 op/s 2023-07-29T16:38:09.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 302207 34.19 MB/sec execute 353 sec latency 58.835 ms 2023-07-29T16:38:10.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 302857 34.19 MB/sec execute 354 sec latency 81.122 ms 2023-07-29T16:38:11.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:10 smithi039 ceph-mon[128858]: pgmap v1053: 97 pgs: 97 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 15 MiB/s wr, 872 op/s 2023-07-29T16:38:11.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:10 smithi156 ceph-mon[129245]: pgmap v1053: 97 pgs: 97 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 15 MiB/s wr, 872 op/s 2023-07-29T16:38:11.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:10 smithi031 ceph-mon[95504]: pgmap v1053: 97 pgs: 97 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 15 MiB/s wr, 872 op/s 2023-07-29T16:38:11.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 303630 34.20 MB/sec execute 355 sec latency 54.637 ms 2023-07-29T16:38:12.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 304365 34.24 MB/sec execute 356 sec latency 59.779 ms 2023-07-29T16:38:13.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:12 smithi156 ceph-mon[129245]: pgmap v1054: 97 pgs: 97 active+clean; 680 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 798 op/s 2023-07-29T16:38:13.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:12 smithi031 ceph-mon[95504]: pgmap v1054: 97 pgs: 97 active+clean; 680 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 798 op/s 2023-07-29T16:38:13.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:12 smithi039 ceph-mon[128858]: pgmap v1054: 97 pgs: 97 active+clean; 680 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 11 MiB/s wr, 798 op/s 2023-07-29T16:38:13.354 INFO:tasks.workunit.client.0.smithi031.stdout: 10 304935 34.22 MB/sec execute 357 sec latency 76.360 ms 2023-07-29T16:38:14.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 305570 34.22 MB/sec execute 358 sec latency 75.200 ms 2023-07-29T16:38:15.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:14 smithi156 ceph-mon[129245]: pgmap v1055: 97 pgs: 97 active+clean; 679 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.8 MiB/s wr, 742 op/s 2023-07-29T16:38:15.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:14 smithi031 ceph-mon[95504]: pgmap v1055: 97 pgs: 97 active+clean; 679 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.8 MiB/s wr, 742 op/s 2023-07-29T16:38:15.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:14 smithi039 ceph-mon[128858]: pgmap v1055: 97 pgs: 97 active+clean; 679 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 8.8 MiB/s wr, 742 op/s 2023-07-29T16:38:15.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 306281 34.22 MB/sec execute 359 sec latency 127.597 ms 2023-07-29T16:38:16.354 INFO:tasks.workunit.client.0.smithi031.stdout: 10 307033 34.23 MB/sec execute 360 sec latency 60.487 ms 2023-07-29T16:38:17.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:16 smithi156 ceph-mon[129245]: pgmap v1056: 97 pgs: 97 active+clean; 686 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.3 MiB/s wr, 597 op/s 2023-07-29T16:38:17.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:16 smithi031 ceph-mon[95504]: pgmap v1056: 97 pgs: 97 active+clean; 686 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.3 MiB/s wr, 597 op/s 2023-07-29T16:38:17.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:16 smithi039 ceph-mon[128858]: pgmap v1056: 97 pgs: 97 active+clean; 686 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 7.3 MiB/s wr, 597 op/s 2023-07-29T16:38:17.353 INFO:tasks.workunit.client.0.smithi031.stdout: 10 307742 34.26 MB/sec execute 361 sec latency 55.899 ms 2023-07-29T16:38:18.354 INFO:tasks.workunit.client.0.smithi031.stdout: 10 308337 34.25 MB/sec execute 362 sec latency 98.631 ms 2023-07-29T16:38:18.530 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:18 smithi039 ceph-mon[128858]: pgmap v1057: 97 pgs: 97 active+clean; 692 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.7 MiB/s wr, 615 op/s 2023-07-29T16:38:18.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:18 smithi156 ceph-mon[129245]: pgmap v1057: 97 pgs: 97 active+clean; 692 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.7 MiB/s wr, 615 op/s 2023-07-29T16:38:18.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:18 smithi031 ceph-mon[95504]: pgmap v1057: 97 pgs: 97 active+clean; 692 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.7 MiB/s wr, 615 op/s 2023-07-29T16:38:19.354 INFO:tasks.workunit.client.0.smithi031.stdout: 10 309080 34.26 MB/sec execute 363 sec latency 49.059 ms 2023-07-29T16:38:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 -- 172.21.15.31:0/2966632231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28381060a0 msgr2=0x7f2838108480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 --2- 172.21.15.31:0/2966632231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28381060a0 0x7f2838108480 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2830009ec0 tx=0x7f283000b330 comp rx=0 tx=0).stop 2023-07-29T16:38:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 -- 172.21.15.31:0/2966632231 shutdown_connections 2023-07-29T16:38:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 --2- 172.21.15.31:0/2966632231 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28381060a0 0x7f2838108480 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 --2- 172.21.15.31:0/2966632231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2838100e60 0x7f2838103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 --2- 172.21.15.31:0/2966632231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838103780 0x7f2838105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 -- 172.21.15.31:0/2966632231 >> 172.21.15.31:0/2966632231 conn(0x7f28380fa840 msgr2=0x7f28380fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:20.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 -- 172.21.15.31:0/2966632231 shutdown_connections 2023-07-29T16:38:20.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.086+0000 7f28401ff700 1 -- 172.21.15.31:0/2966632231 wait complete. 2023-07-29T16:38:20.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.087+0000 7f28401ff700 1 Processor -- start 2023-07-29T16:38:20.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 -- start start 2023-07-29T16:38:20.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838100e60 0x7f2838198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2838103780 0x7f28381989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28381060a0 0x7f2838199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2838199e90 con 0x7f2838100e60 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819a000 con 0x7f28381060a0 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f28401ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283819a1c0 con 0x7f2838103780 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f283df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838100e60 0x7f2838198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f283d79a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2838103780 0x7f28381989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f283df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838100e60 0x7f2838198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34396/0 (socket says 172.21.15.31:34396) 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.088+0000 7f283df9b700 1 -- 172.21.15.31:0/1013846947 learned_addr learned my addr 172.21.15.31:0/1013846947 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283e79c700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28381060a0 0x7f2838199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283d79a700 1 -- 172.21.15.31:0/1013846947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28381060a0 msgr2=0x7f2838199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283d79a700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28381060a0 0x7f2838199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283d79a700 1 -- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838100e60 msgr2=0x7f2838198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283d79a700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838100e60 0x7f2838198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283d79a700 1 -- 172.21.15.31:0/1013846947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2830005e70 con 0x7f2838103780 2023-07-29T16:38:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283e79c700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28381060a0 0x7f2838199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:38:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.089+0000 7f283d79a700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2838103780 0x7f28381989b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f282800b450 tx=0x7f282800b760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:20.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.090+0000 7f282effd700 1 -- 172.21.15.31:0/1013846947 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828011830 con 0x7f2838103780 2023-07-29T16:38:20.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.090+0000 7f282effd700 1 -- 172.21.15.31:0/1013846947 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2828011e50 con 0x7f2838103780 2023-07-29T16:38:20.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.090+0000 7f282effd700 1 -- 172.21.15.31:0/1013846947 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828010de0 con 0x7f2838103780 2023-07-29T16:38:20.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.090+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283819a4a0 con 0x7f2838103780 2023-07-29T16:38:20.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.090+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28381a0c60 con 0x7f2838103780 2023-07-29T16:38:20.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.094+0000 7f282effd700 1 -- 172.21.15.31:0/1013846947 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f2828010480 con 0x7f2838103780 2023-07-29T16:38:20.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.097+0000 7f282effd700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28240a0810 0x7f28240a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.097+0000 7f282effd700 1 -- 172.21.15.31:0/1013846947 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f28280cb500 con 0x7f2838103780 2023-07-29T16:38:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.097+0000 7f2853b76b80 1 Processor -- start 2023-07-29T16:38:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.097+0000 7f283df9b700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28240a0810 0x7f28240a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.098+0000 7f2853b76b80 1 -- start start 2023-07-29T16:38:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.098+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.098+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d28e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d29820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f2853b76b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745c789b0 con 0x55d745b7c6b0 2023-07-29T16:38:20.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f2853b76b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745c76370 con 0x55d745d29380 2023-07-29T16:38:20.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f2853b76b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745c71c60 con 0x55d745d2a770 2023-07-29T16:38:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f283df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f283df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34410/0 (socket says 172.21.15.31:34410) 2023-07-29T16:38:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f283df9b700 1 -- 172.21.15.31:0/167257546 learned_addr learned my addr 172.21.15.31:0/167257546 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:20.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.099+0000 7f283d79a700 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d28e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283e79c700 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d29820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283df9b700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28240a0810 0x7f28240a2cc0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2834005950 tx=0x7f2834009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283d79a700 1 -- 172.21.15.31:0/167257546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 msgr2=0x55d745d29820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283d79a700 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d29820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283d79a700 1 -- 172.21.15.31:0/167257546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 msgr2=0x55d745d2a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283d79a700 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283d79a700 1 -- 172.21.15.31:0/167257546 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d745a9fd80 con 0x55d745d2a770 2023-07-29T16:38:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f283d79a700 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d28e40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2828009fd0 tx=0x7f2828010880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3da06aa2e724486 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:20.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f2823fff700 1 -- 172.21.15.31:0/167257546 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282809cbd0 con 0x55d745d2a770 2023-07-29T16:38:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f2823fff700 1 -- 172.21.15.31:0/167257546 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28280a0ff0 con 0x55d745d2a770 2023-07-29T16:38:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.100+0000 7f2823fff700 1 -- 172.21.15.31:0/167257546 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2828098090 con 0x55d745d2a770 2023-07-29T16:38:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.101+0000 7f2853b76b80 1 -- 172.21.15.31:0/167257546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 msgr2=0x55d745d28e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.101+0000 7f2853b76b80 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d28e40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f2828009fd0 tx=0x7f2828010880 comp rx=0 tx=0).stop 2023-07-29T16:38:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.101+0000 7f2853b76b80 1 -- 172.21.15.31:0/167257546 shutdown_connections 2023-07-29T16:38:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.101+0000 7f2853b76b80 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d29820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.101+0000 7f2853b76b80 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.101+0000 7f2853b76b80 1 --2- 172.21.15.31:0/167257546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d28e40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.102+0000 7f2853b76b80 1 -- 172.21.15.31:0/167257546 >> 172.21.15.31:0/167257546 conn(0x55d745d15fd0 msgr2=0x55d745d1b310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.102+0000 7f2853b76b80 1 -- 172.21.15.31:0/167257546 shutdown_connections 2023-07-29T16:38:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.102+0000 7f2853b76b80 1 -- 172.21.15.31:0/167257546 wait complete. 2023-07-29T16:38:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.102+0000 7f2853b76b80 1 -- start start 2023-07-29T16:38:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.103+0000 7f2853b76b80 1 Processor -- start 2023-07-29T16:38:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d32920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d32e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 0x55d745d39c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745c0e4b0 con 0x55d745b7c6b0 2023-07-29T16:38:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745c0e150 con 0x55d745d2a770 2023-07-29T16:38:20.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745af58c0 con 0x55d745d29380 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f2853b76b80 10 client.0 _ll_register_callbacks cb 0x55d745c175c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f283e79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 0x55d745d39c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f283e79c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 0x55d745d39c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:42350/0 (socket says 172.21.15.31:42350) 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f283e79c700 1 -- 172.21.15.31:0/336841363 learned_addr learned my addr 172.21.15.31:0/336841363 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.104+0000 7f283d79a700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d32e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283df9b700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d32920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283e79c700 1 -- 172.21.15.31:0/336841363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 msgr2=0x55d745d32e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283e79c700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d32e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283e79c700 1 -- 172.21.15.31:0/336841363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 msgr2=0x55d745d32920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283e79c700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d32920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283e79c700 1 -- 172.21.15.31:0/336841363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28381a0c60 con 0x55d745d2a770 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f283e79c700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 0x55d745d39c90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f283000b700 tx=0x7f2830007c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7114de145ddb0a8 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f28217fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f28217fa700 1 -- 172.21.15.31:0/336841363 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283000e040 con 0x55d745d2a770 2023-07-29T16:38:20.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f28217fa700 1 -- 172.21.15.31:0/336841363 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f283000bbe0 con 0x55d745d2a770 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.105+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d745c9ccb0 con 0x55d745d2a770 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.106+0000 7f28217fa700 1 -- 172.21.15.31:0/336841363 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2830025410 con 0x55d745d2a770 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.106+0000 7f2853b76b80 10 client.30536 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.106+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55d745a88c00 con 0x55d745d2a770 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f28217fa700 1 -- 172.21.15.31:0/336841363 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d745c9ccb0 con 0x55d745d2a770 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f28217fa700 1 -- 172.21.15.31:0/336841363 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2830023030 con 0x55d745d2a770 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f2853b76b80 10 client.30536 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:20.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f2853b76b80 10 client.30536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f2853b76b80 20 client.30536 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f2853b76b80 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d40b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f2853b76b80 4 client.30536 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:38:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d745cefd10 con 0x55d745d3e730 2023-07-29T16:38:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.107+0000 7f283df9b700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d40b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.108+0000 7f283df9b700 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d40b40 crc :-1 s=READY pgs=489 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c1a59201b3bd090 server_cookie=e8240a660bc0f3bd in_seq=0 out_seq=0 2023-07-29T16:38:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.108+0000 7f28217fa700 10 client.30536 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f28217fa700 1 -- 172.21.15.31:0/336841363 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d745cefd10 con 0x55d745d3e730 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f28217fa700 10 client.30536 handle_command_reply: tid=0 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f2853b76b80 1 client.30536 shutdown 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 msgr2=0x55d745d39c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f2853b76b80 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 0x55d745d39c90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f283000b700 tx=0x7f2830007c70 comp rx=0 tx=0).stop 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 shutdown_connections 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.110+0000 7f2853b76b80 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d40b40 crc :-1 s=READY pgs=489 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d2a770 0x55d745d39c90 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f28217fa700 0 client.30536 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d32920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 1 --2- 172.21.15.31:0/336841363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d32e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 >> 172.21.15.31:0/336841363 conn(0x55d745d15fd0 msgr2=0x55d745d1aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 shutdown_connections 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 1 -- 172.21.15.31:0/336841363 wait complete. 2023-07-29T16:38:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.111+0000 7f2853b76b80 20 client.30536 trim_cache size 0 max 16384 2023-07-29T16:38:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.132+0000 7f2853b76b80 1 Processor -- start 2023-07-29T16:38:20.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 -- start start 2023-07-29T16:38:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d39c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d2d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745c9ccb0 con 0x55d745b7c6b0 2023-07-29T16:38:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745d1c120 con 0x55d745d29380 2023-07-29T16:38:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f2853b76b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745d33900 con 0x55d745d2a770 2023-07-29T16:38:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f283e79c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d2d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f283df9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f283e79c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d2d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53088/0 (socket says 172.21.15.31:53088) 2023-07-29T16:38:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.133+0000 7f283e79c700 1 -- 172.21.15.31:0/2931812774 learned_addr learned my addr 172.21.15.31:0/2931812774 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:20.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283e79c700 1 -- 172.21.15.31:0/2931812774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 msgr2=0x55d745d39c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283e79c700 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d39c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283e79c700 1 -- 172.21.15.31:0/2931812774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 msgr2=0x55d745d2d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283e79c700 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283e79c700 1 -- 172.21.15.31:0/2931812774 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d745ac61f0 con 0x55d745d2a770 2023-07-29T16:38:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283df9b700 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:38:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f283e79c700 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d2d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f283000d010 tx=0x7f283000afe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5750e10b208b0e57 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f282cff9700 1 -- 172.21.15.31:0/2931812774 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283000e040 con 0x55d745d2a770 2023-07-29T16:38:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f282cff9700 1 -- 172.21.15.31:0/2931812774 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f283001f070 con 0x55d745d2a770 2023-07-29T16:38:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.134+0000 7f282cff9700 1 -- 172.21.15.31:0/2931812774 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830025b30 con 0x55d745d2a770 2023-07-29T16:38:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 -- 172.21.15.31:0/2931812774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 msgr2=0x55d745d2d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d2d990 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f283000d010 tx=0x7f283000afe0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 -- 172.21.15.31:0/2931812774 shutdown_connections 2023-07-29T16:38:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d2a770 0x55d745d2d990 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745b7c6b0 0x55d745d2d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 --2- 172.21.15.31:0/2931812774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745d29380 0x55d745d39c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.135+0000 7f2853b76b80 1 -- 172.21.15.31:0/2931812774 >> 172.21.15.31:0/2931812774 conn(0x55d745d15fd0 msgr2=0x55d745d308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.136+0000 7f2853b76b80 1 -- 172.21.15.31:0/2931812774 shutdown_connections 2023-07-29T16:38:20.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.136+0000 7f2853b76b80 1 -- 172.21.15.31:0/2931812774 wait complete. 2023-07-29T16:38:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.136+0000 7f2853b76b80 1 -- start start 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.137+0000 7f2853b76b80 1 Processor -- start 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745b7c6b0 0x55d745d35d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d36240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745d2a770 0x55d745d39260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745d397a0 con 0x55d745d2a770 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745d39910 con 0x55d745b7c6b0 2023-07-29T16:38:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d745d39ad0 con 0x55d745d29380 2023-07-29T16:38:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.138+0000 7f2853b76b80 10 client.0 _ll_register_callbacks cb 0x55d745c175c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283d79a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d36240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283d79a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d36240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53098/0 (socket says 172.21.15.31:53098) 2023-07-29T16:38:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283d79a700 1 -- 172.21.15.31:0/959856838 learned_addr learned my addr 172.21.15.31:0/959856838 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283df9b700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745b7c6b0 0x55d745d35d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283e79c700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745d2a770 0x55d745d39260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283df9b700 1 -- 172.21.15.31:0/959856838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 msgr2=0x55d745d36240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283df9b700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d36240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283df9b700 1 -- 172.21.15.31:0/959856838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745d2a770 msgr2=0x55d745d39260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283df9b700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745d2a770 0x55d745d39260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.139+0000 7f283df9b700 1 -- 172.21.15.31:0/959856838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d745a88c00 con 0x55d745b7c6b0 2023-07-29T16:38:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f283df9b700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745b7c6b0 0x55d745d35d00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x55d745d413f0 tx=0x7f2834013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d846baaab60b4e5 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f28217fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:38:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f283d79a700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d36240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:38:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f283e79c700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745d2a770 0x55d745d39260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:38:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f28217fa700 1 -- 172.21.15.31:0/959856838 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834024040 con 0x55d745b7c6b0 2023-07-29T16:38:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f28217fa700 1 -- 172.21.15.31:0/959856838 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f28340297c0 con 0x55d745b7c6b0 2023-07-29T16:38:20.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.140+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28280a0ff0 con 0x55d745b7c6b0 2023-07-29T16:38:20.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f28217fa700 1 -- 172.21.15.31:0/959856838 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2834029980 con 0x55d745b7c6b0 2023-07-29T16:38:20.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 10 client.30539 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:20.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55d745a9e250 con 0x55d745b7c6b0 2023-07-29T16:38:20.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f28217fa700 1 -- 172.21.15.31:0/959856838 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f28280a0ff0 con 0x55d745b7c6b0 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f28217fa700 1 -- 172.21.15.31:0/959856838 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f2834032c00 con 0x55d745b7c6b0 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 10 client.30539 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 10 client.30539 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 20 client.30539 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d934f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 4 client.30539 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:38:20.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.141+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d745c6ace0 con 0x55d745d3e730 2023-07-29T16:38:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.142+0000 7f283d79a700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d934f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.142+0000 7f283d79a700 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d934f0 crc :-1 s=READY pgs=490 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b605ec02a57e481f server_cookie=2fcdb3f1148b0aff in_seq=0 out_seq=0 2023-07-29T16:38:20.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.142+0000 7f28217fa700 10 client.30539 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:20.357 INFO:tasks.workunit.client.0.smithi031.stdout: 10 309758 34.27 MB/sec execute 364 sec latency 198.138 ms 2023-07-29T16:38:20.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.594+0000 7f28217fa700 1 -- 172.21.15.31:0/959856838 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d745c6ace0 con 0x55d745d3e730 2023-07-29T16:38:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.594+0000 7f28217fa700 10 client.30539 handle_command_reply: tid=0 2023-07-29T16:38:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 client.30539 shutdown 2023-07-29T16:38:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745b7c6b0 msgr2=0x55d745d35d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745b7c6b0 0x55d745d35d00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x55d745d413f0 tx=0x7f2834013040 comp rx=0 tx=0).stop 2023-07-29T16:38:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 shutdown_connections 2023-07-29T16:38:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d745d3e730 0x55d745d934f0 crc :-1 s=READY pgs=490 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d745d2a770 0x55d745d39260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d745b7c6b0 0x55d745d35d00 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 --2- 172.21.15.31:0/959856838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d745d29380 0x55d745d36240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 >> 172.21.15.31:0/959856838 conn(0x55d745d15fd0 msgr2=0x55d745d308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.595+0000 7f28217fa700 0 client.30539 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.596+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 shutdown_connections 2023-07-29T16:38:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.597+0000 7f2853b76b80 1 -- 172.21.15.31:0/959856838 wait complete. 2023-07-29T16:38:20.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.597+0000 7f2853b76b80 20 client.30539 trim_cache size 0 max 16384 2023-07-29T16:38:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:38:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:38:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:38:20.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:38:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28240a0810 msgr2=0x7f28240a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28240a0810 0x7f28240a2cc0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2834005950 tx=0x7f2834009260 comp rx=0 tx=0).stop 2023-07-29T16:38:20.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2838103780 msgr2=0x7f28381989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2838103780 0x7f28381989b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f282800b450 tx=0x7f282800b760 comp rx=0 tx=0).stop 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 shutdown_connections 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f28240a0810 0x7f28240a2cc0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f28381060a0 0x7f2838199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2838100e60 0x7f2838198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 --2- 172.21.15.31:0/1013846947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2838103780 0x7f28381989b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:20.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.598+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 >> 172.21.15.31:0/1013846947 conn(0x7f28380fa840 msgr2=0x7f28380ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.599+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 shutdown_connections 2023-07-29T16:38:20.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:20.599+0000 7f28401ff700 1 -- 172.21.15.31:0/1013846947 wait complete. 2023-07-29T16:38:21.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:20 smithi039 ceph-mon[128858]: pgmap v1058: 97 pgs: 97 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.9 MiB/s wr, 491 op/s 2023-07-29T16:38:21.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:20 smithi156 ceph-mon[129245]: pgmap v1058: 97 pgs: 97 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.9 MiB/s wr, 491 op/s 2023-07-29T16:38:21.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:20 smithi031 ceph-mon[95504]: pgmap v1058: 97 pgs: 97 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.9 MiB/s wr, 491 op/s 2023-07-29T16:38:21.357 INFO:tasks.workunit.client.0.smithi031.stdout: 10 310378 34.28 MB/sec execute 365 sec latency 295.362 ms 2023-07-29T16:38:22.357 INFO:tasks.workunit.client.0.smithi031.stdout: 10 310852 34.25 MB/sec execute 366 sec latency 79.806 ms 2023-07-29T16:38:23.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:22 smithi156 ceph-mon[129245]: pgmap v1059: 97 pgs: 97 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.4 MiB/s wr, 467 op/s 2023-07-29T16:38:23.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:22 smithi031 ceph-mon[95504]: pgmap v1059: 97 pgs: 97 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.4 MiB/s wr, 467 op/s 2023-07-29T16:38:23.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:22 smithi039 ceph-mon[128858]: pgmap v1059: 97 pgs: 97 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.4 MiB/s wr, 467 op/s 2023-07-29T16:38:23.357 INFO:tasks.workunit.client.0.smithi031.stdout: 10 311427 34.23 MB/sec execute 367 sec latency 113.740 ms 2023-07-29T16:38:24.358 INFO:tasks.workunit.client.0.smithi031.stdout: 10 312221 34.25 MB/sec execute 368 sec latency 124.200 ms 2023-07-29T16:38:25.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:24 smithi156 ceph-mon[129245]: pgmap v1060: 97 pgs: 97 active+clean; 699 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.1 MiB/s wr, 599 op/s 2023-07-29T16:38:25.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:24 smithi031 ceph-mon[95504]: pgmap v1060: 97 pgs: 97 active+clean; 699 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.1 MiB/s wr, 599 op/s 2023-07-29T16:38:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:24 smithi039 ceph-mon[128858]: pgmap v1060: 97 pgs: 97 active+clean; 699 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.1 MiB/s wr, 599 op/s 2023-07-29T16:38:25.359 INFO:tasks.workunit.client.0.smithi031.stdout: 10 313074 34.29 MB/sec execute 369 sec latency 56.811 ms 2023-07-29T16:38:26.074 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:25 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:38:26.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:25 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:38:26.287 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:25 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:38:26.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 313662 34.27 MB/sec execute 370 sec latency 89.737 ms 2023-07-29T16:38:27.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:26 smithi156 ceph-mon[129245]: pgmap v1061: 97 pgs: 97 active+clean; 704 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.8 MiB/s wr, 566 op/s 2023-07-29T16:38:27.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:38:27.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:38:27.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:26 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:38:27.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:26 smithi031 ceph-mon[95504]: pgmap v1061: 97 pgs: 97 active+clean; 704 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.8 MiB/s wr, 566 op/s 2023-07-29T16:38:27.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:38:27.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:38:27.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:26 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:38:27.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:26 smithi039 ceph-mon[128858]: pgmap v1061: 97 pgs: 97 active+clean; 704 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.8 MiB/s wr, 566 op/s 2023-07-29T16:38:27.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:38:27.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:38:27.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:26 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:38:27.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 314334 34.29 MB/sec execute 371 sec latency 65.955 ms 2023-07-29T16:38:28.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 315061 34.33 MB/sec execute 372 sec latency 60.588 ms 2023-07-29T16:38:28.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:28 smithi039 ceph-mon[128858]: pgmap v1062: 97 pgs: 97 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 535 op/s 2023-07-29T16:38:28.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:28 smithi156 ceph-mon[129245]: pgmap v1062: 97 pgs: 97 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 535 op/s 2023-07-29T16:38:28.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:28 smithi031 ceph-mon[95504]: pgmap v1062: 97 pgs: 97 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 535 op/s 2023-07-29T16:38:29.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 315637 34.28 MB/sec execute 373 sec latency 59.464 ms 2023-07-29T16:38:30.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 316333 34.31 MB/sec execute 374 sec latency 56.753 ms 2023-07-29T16:38:31.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:30 smithi156 ceph-mon[129245]: pgmap v1063: 97 pgs: 97 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 517 op/s 2023-07-29T16:38:31.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:30 smithi031 ceph-mon[95504]: pgmap v1063: 97 pgs: 97 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 517 op/s 2023-07-29T16:38:31.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:30 smithi039 ceph-mon[128858]: pgmap v1063: 97 pgs: 97 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 517 op/s 2023-07-29T16:38:31.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 317022 34.31 MB/sec execute 375 sec latency 63.930 ms 2023-07-29T16:38:32.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 317669 34.31 MB/sec execute 376 sec latency 74.040 ms 2023-07-29T16:38:33.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:32 smithi156 ceph-mon[129245]: pgmap v1064: 97 pgs: 97 active+clean; 710 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.0 MiB/s wr, 585 op/s 2023-07-29T16:38:33.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:32 smithi031 ceph-mon[95504]: pgmap v1064: 97 pgs: 97 active+clean; 710 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.0 MiB/s wr, 585 op/s 2023-07-29T16:38:33.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:32 smithi039 ceph-mon[128858]: pgmap v1064: 97 pgs: 97 active+clean; 710 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.0 MiB/s wr, 585 op/s 2023-07-29T16:38:33.360 INFO:tasks.workunit.client.0.smithi031.stdout: 10 318155 34.27 MB/sec execute 377 sec latency 105.403 ms 2023-07-29T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.877+0000 7fec03dff700 1 -- 172.21.15.31:0/3847858262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc105830 msgr2=0x7febfc107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.877+0000 7fec03dff700 1 --2- 172.21.15.31:0/3847858262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc105830 0x7febfc107c10 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7febe8009990 tx=0x7febe8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 -- 172.21.15.31:0/3847858262 shutdown_connections 2023-07-29T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 --2- 172.21.15.31:0/3847858262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc108150 0x7febfc10a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 --2- 172.21.15.31:0/3847858262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febfc068e60 0x7febfc1052f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 --2- 172.21.15.31:0/3847858262 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc105830 0x7febfc107c10 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 -- 172.21.15.31:0/3847858262 >> 172.21.15.31:0/3847858262 conn(0x7febfc0fa840 msgr2=0x7febfc0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 -- 172.21.15.31:0/3847858262 shutdown_connections 2023-07-29T16:38:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.878+0000 7fec03dff700 1 -- 172.21.15.31:0/3847858262 wait complete. 2023-07-29T16:38:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.879+0000 7fec03dff700 1 Processor -- start 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.879+0000 7fec03dff700 1 -- start start 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec03dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 0x7febfc19c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec03dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febfc105830 0x7febfc19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec03dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc108150 0x7febfc19db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec03dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc19e170 con 0x7febfc105830 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec03dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc19e2e0 con 0x7febfc108150 2023-07-29T16:38:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec03dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc19e4a0 con 0x7febfc068e60 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 0x7febfc19c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec0239c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc108150 0x7febfc19db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 0x7febfc19c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40654/0 (socket says 172.21.15.31:40654) 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec01b9b700 1 -- 172.21.15.31:0/1138366757 learned_addr learned my addr 172.21.15.31:0/1138366757 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec0139a700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febfc105830 0x7febfc19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec0239c700 1 -- 172.21.15.31:0/1138366757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 msgr2=0x7febfc19c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec0239c700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 0x7febfc19c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec0239c700 1 -- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febfc105830 msgr2=0x7febfc19cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.880+0000 7fec0239c700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febfc105830 0x7febfc19cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7fec0239c700 1 -- 172.21.15.31:0/1138366757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe80095f0 con 0x7febfc108150 2023-07-29T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7fec01b9b700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 0x7febfc19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:38:33.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7fec0239c700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc108150 0x7febfc19db10 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7febf400eab0 tx=0x7febf400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7febfaffd700 1 -- 172.21.15.31:0/1138366757 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf40098c0 con 0x7febfc108150 2023-07-29T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc19e780 con 0x7febfc108150 2023-07-29T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7febfaffd700 1 -- 172.21.15.31:0/1138366757 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febf400ce30 con 0x7febfc108150 2023-07-29T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7febfaffd700 1 -- 172.21.15.31:0/1138366757 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4004280 con 0x7febfc108150 2023-07-29T16:38:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.881+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc1a4f40 con 0x7febfc108150 2023-07-29T16:38:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.886+0000 7febfaffd700 1 -- 172.21.15.31:0/1138366757 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7febf400c970 con 0x7febfc108150 2023-07-29T16:38:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.887+0000 7fec1780eb80 1 Processor -- start 2023-07-29T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.887+0000 7fec1780eb80 1 -- start start 2023-07-29T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd435a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd435b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd435c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec1780eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd42b29b0 con 0x559dd41b66b0 2023-07-29T16:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec1780eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd42b0370 con 0x559dd435bcf0 2023-07-29T16:38:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec1780eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd42abc60 con 0x559dd435adf0 2023-07-29T16:38:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd435b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd435b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40662/0 (socket says 172.21.15.31:40662) 2023-07-29T16:38:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.888+0000 7fec01b9b700 1 -- 172.21.15.31:0/2344366135 learned_addr learned my addr 172.21.15.31:0/2344366135 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec0139a700 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd435c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec0139a700 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd435a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec01b9b700 1 -- 172.21.15.31:0/2344366135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 msgr2=0x559dd435c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec01b9b700 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd435c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec01b9b700 1 -- 172.21.15.31:0/2344366135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 msgr2=0x559dd435a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec01b9b700 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd435a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.889+0000 7fec01b9b700 1 -- 172.21.15.31:0/2344366135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dd40d9d80 con 0x559dd435adf0 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7febfaffd700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7febec0a0810 0x7febec0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7febfaffd700 1 -- 172.21.15.31:0/1138366757 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7febf4014070 con 0x7febfc108150 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7fec01b9b700 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd435b7b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7febf000c850 tx=0x7febf0007a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5126d6c257b965 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7febe7fff700 1 -- 172.21.15.31:0/2344366135 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0014660 con 0x559dd435adf0 2023-07-29T16:38:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7fec0139a700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7febec0a0810 0x7febec0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7febe7fff700 1 -- 172.21.15.31:0/2344366135 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febf0014c80 con 0x559dd435adf0 2023-07-29T16:38:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.890+0000 7febe7fff700 1 -- 172.21.15.31:0/2344366135 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0015e30 con 0x559dd435adf0 2023-07-29T16:38:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.891+0000 7fec1780eb80 1 -- 172.21.15.31:0/2344366135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 msgr2=0x559dd435b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.891+0000 7fec0139a700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7febec0a0810 0x7febec0a2cc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x559dd435a610 tx=0x7febe8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.891+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd435b7b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7febf000c850 tx=0x7febf0007a70 comp rx=0 tx=0).stop 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 -- 172.21.15.31:0/2344366135 shutdown_connections 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd435c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd435a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2344366135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd435b7b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 -- 172.21.15.31:0/2344366135 >> 172.21.15.31:0/2344366135 conn(0x559dd434ffd0 msgr2=0x559dd4355300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 -- 172.21.15.31:0/2344366135 shutdown_connections 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.892+0000 7fec1780eb80 1 -- 172.21.15.31:0/2344366135 wait complete. 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.893+0000 7fec1780eb80 1 -- start start 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.893+0000 7fec1780eb80 1 Processor -- start 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd4363ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd4364420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd436b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd42484b0 con 0x559dd41b66b0 2023-07-29T16:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd4248150 con 0x559dd435bcf0 2023-07-29T16:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd412f8c0 con 0x559dd435adf0 2023-07-29T16:38:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.894+0000 7fec1780eb80 10 client.0 _ll_register_callbacks cb 0x559dd42515c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.896+0000 7fec0239c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd436b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.896+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd4363ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.896+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd4363ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44202/0 (socket says 172.21.15.31:44202) 2023-07-29T16:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.896+0000 7fec01b9b700 1 -- 172.21.15.31:0/480492051 learned_addr learned my addr 172.21.15.31:0/480492051 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:33.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.897+0000 7fec0139a700 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd4364420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.897+0000 7fec0239c700 1 -- 172.21.15.31:0/480492051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 msgr2=0x559dd4364420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.897+0000 7fec0239c700 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd4364420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.897+0000 7fec0239c700 1 -- 172.21.15.31:0/480492051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 msgr2=0x559dd4363ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.897+0000 7fec0239c700 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd4363ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.897+0000 7fec0239c700 1 -- 172.21.15.31:0/480492051 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf000c4b0 con 0x559dd435bcf0 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7fec0239c700 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd436b460 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7febf40c95f0 tx=0x7febf4099790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc1e0011bab9100 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:33.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7febe57fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7febe57fa700 1 -- 172.21.15.31:0/480492051 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf4099250 con 0x559dd435bcf0 2023-07-29T16:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559dd42d6cb0 con 0x559dd435bcf0 2023-07-29T16:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7febe57fa700 1 -- 172.21.15.31:0/480492051 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7febf409d280 con 0x559dd435bcf0 2023-07-29T16:38:33.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7febe57fa700 1 -- 172.21.15.31:0/480492051 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7febf40a16a0 con 0x559dd435bcf0 2023-07-29T16:38:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7fec1780eb80 10 client.30551 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559dd40c2c00 con 0x559dd435bcf0 2023-07-29T16:38:33.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.898+0000 7febe57fa700 1 -- 172.21.15.31:0/480492051 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559dd42d6cb0 con 0x559dd435bcf0 2023-07-29T16:38:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7febe57fa700 1 -- 172.21.15.31:0/480492051 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7febf40950d0 con 0x559dd435bcf0 2023-07-29T16:38:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec1780eb80 10 client.30551 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec1780eb80 10 client.30551 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:33.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec1780eb80 20 client.30551 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec1780eb80 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd436fc50 0x559dd4372060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec1780eb80 4 client.30551 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:38:33.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559dd4329d10 con 0x559dd436fc50 2023-07-29T16:38:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.899+0000 7fec01b9b700 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd436fc50 0x559dd4372060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.900+0000 7febe57fa700 10 client.30551 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.900+0000 7fec01b9b700 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd436fc50 0x559dd4372060 crc :-1 s=READY pgs=491 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ee64b1b47023064 server_cookie=c060ffb63ee7626e in_seq=0 out_seq=0 2023-07-29T16:38:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.901+0000 7febe57fa700 1 -- 172.21.15.31:0/480492051 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559dd4329d10 con 0x559dd436fc50 2023-07-29T16:38:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.901+0000 7febe57fa700 10 client.30551 handle_command_reply: tid=0 2023-07-29T16:38:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.901+0000 7fec1780eb80 1 client.30551 shutdown 2023-07-29T16:38:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 msgr2=0x559dd436b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:33.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd436b460 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7febf40c95f0 tx=0x7febf4099790 comp rx=0 tx=0).stop 2023-07-29T16:38:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 shutdown_connections 2023-07-29T16:38:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd436fc50 0x559dd4372060 crc :-1 s=READY pgs=491 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd436b460 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd41b66b0 0x559dd4363ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 --2- 172.21.15.31:0/480492051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd435adf0 0x559dd4364420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7febe57fa700 0 client.30551 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 >> 172.21.15.31:0/480492051 conn(0x559dd434ffd0 msgr2=0x559dd4354a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 shutdown_connections 2023-07-29T16:38:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 1 -- 172.21.15.31:0/480492051 wait complete. 2023-07-29T16:38:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.902+0000 7fec1780eb80 20 client.30551 trim_cache size 0 max 16384 2023-07-29T16:38:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.927+0000 7fec1780eb80 1 Processor -- start 2023-07-29T16:38:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.927+0000 7fec1780eb80 1 -- start start 2023-07-29T16:38:33.929 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4372060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435adf0 0x559dd4369410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 0x559dd435c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec1780eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd4363a50 con 0x559dd435bcf0 2023-07-29T16:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec1780eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd43565f0 con 0x559dd435adf0 2023-07-29T16:38:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec1780eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd4356b30 con 0x559dd41b66b0 2023-07-29T16:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec0239c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 0x559dd435c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec0239c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 0x559dd435c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44218/0 (socket says 172.21.15.31:44218) 2023-07-29T16:38:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.928+0000 7fec0239c700 1 -- 172.21.15.31:0/2545064796 learned_addr learned my addr 172.21.15.31:0/2545064796 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.929+0000 7fec01b9b700 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4372060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.929+0000 7fec0239c700 1 -- 172.21.15.31:0/2545064796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 msgr2=0x559dd4372060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.929+0000 7fec0239c700 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4372060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.929+0000 7fec0239c700 1 -- 172.21.15.31:0/2545064796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435adf0 msgr2=0x559dd4369410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.929+0000 7fec0239c700 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435adf0 0x559dd4369410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.929+0000 7fec0239c700 1 -- 172.21.15.31:0/2545064796 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dd41001f0 con 0x559dd435bcf0 2023-07-29T16:38:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.930+0000 7fec0239c700 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 0x559dd435c0c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7febf400f010 tx=0x7febf4098a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d45255eb849589 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:33.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.930+0000 7febf8bff700 1 -- 172.21.15.31:0/2545064796 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf409d280 con 0x559dd435bcf0 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.930+0000 7febf8bff700 1 -- 172.21.15.31:0/2545064796 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febf4099250 con 0x559dd435bcf0 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.930+0000 7febf8bff700 1 -- 172.21.15.31:0/2545064796 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf40a1210 con 0x559dd435bcf0 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 -- 172.21.15.31:0/2545064796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 msgr2=0x559dd435c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 0x559dd435c0c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7febf400f010 tx=0x7febf4098a20 comp rx=0 tx=0).stop 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 -- 172.21.15.31:0/2545064796 shutdown_connections 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435bcf0 0x559dd435c0c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.934 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4372060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 --2- 172.21.15.31:0/2545064796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435adf0 0x559dd4369410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.931+0000 7fec1780eb80 1 -- 172.21.15.31:0/2545064796 >> 172.21.15.31:0/2545064796 conn(0x559dd434ffd0 msgr2=0x559dd4362c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.936+0000 7fec1780eb80 1 -- 172.21.15.31:0/2545064796 shutdown_connections 2023-07-29T16:38:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.938+0000 7fec1780eb80 1 -- 172.21.15.31:0/2545064796 wait complete. 2023-07-29T16:38:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.938+0000 7fec1780eb80 1 -- start start 2023-07-29T16:38:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.939+0000 7fec1780eb80 1 Processor -- start 2023-07-29T16:38:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.939+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4369410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435adf0 0x559dd43bd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec1780eb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd43bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec1780eb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd43c05c0 con 0x559dd435adf0 2023-07-29T16:38:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec1780eb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd43c0730 con 0x559dd435bcf0 2023-07-29T16:38:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec1780eb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dd43c08f0 con 0x559dd41b66b0 2023-07-29T16:38:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec1780eb80 10 client.0 _ll_register_callbacks cb 0x559dd42515c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4369410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec01b9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4369410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40688/0 (socket says 172.21.15.31:40688) 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec01b9b700 1 -- 172.21.15.31:0/1455147527 learned_addr learned my addr 172.21.15.31:0/1455147527 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.940+0000 7fec0239c700 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd43bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec01b9b700 1 -- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 msgr2=0x559dd43bfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec01b9b700 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd43bfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec01b9b700 1 -- 172.21.15.31:0/1455147527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435adf0 msgr2=0x559dd43bd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec01b9b700 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435adf0 0x559dd43bd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec01b9b700 1 -- 172.21.15.31:0/1455147527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559dd40c2c00 con 0x559dd41b66b0 2023-07-29T16:38:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec01b9b700 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4369410 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7febf000c850 tx=0x7febf0004a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6cc5e62cb636ed9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7febe57fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:38:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7febe57fa700 1 -- 172.21.15.31:0/1455147527 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf000b070 con 0x559dd41b66b0 2023-07-29T16:38:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7febe57fa700 1 -- 172.21.15.31:0/1455147527 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7febf002a230 con 0x559dd41b66b0 2023-07-29T16:38:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.941+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7febf0014c80 con 0x559dd41b66b0 2023-07-29T16:38:33.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.942+0000 7febe57fa700 1 -- 172.21.15.31:0/1455147527 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7febf0023040 con 0x559dd41b66b0 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.942+0000 7fec1780eb80 10 client.30577 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.942+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559dd40d8250 con 0x559dd41b66b0 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.942+0000 7febe57fa700 1 -- 172.21.15.31:0/1455147527 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7febf0014c80 con 0x559dd41b66b0 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.942+0000 7febe57fa700 1 -- 172.21.15.31:0/1455147527 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7febf0012070 con 0x559dd41b66b0 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec1780eb80 10 client.30577 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec1780eb80 10 client.30577 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:33.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec1780eb80 20 client.30577 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec1780eb80 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd43c22d0 0x559dd43c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec1780eb80 4 client.30577 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:38:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559dd42a4ce0 con 0x559dd43c22d0 2023-07-29T16:38:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.943+0000 7fec0139a700 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd43c22d0 0x559dd43c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.944+0000 7fec0139a700 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd43c22d0 0x559dd43c46e0 crc :-1 s=READY pgs=492 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e487fcb1a82961 server_cookie=be1437c0ca62b0f4 in_seq=0 out_seq=0 2023-07-29T16:38:33.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:33.944+0000 7febe57fa700 10 client.30577 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:34.361 INFO:tasks.workunit.client.0.smithi031.stdout: 10 318511 34.23 MB/sec execute 378 sec latency 319.704 ms 2023-07-29T16:38:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.361+0000 7febe57fa700 1 -- 172.21.15.31:0/1455147527 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559dd42a4ce0 con 0x559dd43c22d0 2023-07-29T16:38:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.361+0000 7febe57fa700 10 client.30577 handle_command_reply: tid=0 2023-07-29T16:38:34.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.361+0000 7fec1780eb80 1 client.30577 shutdown 2023-07-29T16:38:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 msgr2=0x559dd4369410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4369410 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7febf000c850 tx=0x7febf0004a70 comp rx=0 tx=0).stop 2023-07-29T16:38:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 shutdown_connections 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559dd43c22d0 0x559dd43c46e0 crc :-1 s=READY pgs=492 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559dd435bcf0 0x559dd43bfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559dd41b66b0 0x559dd4369410 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 --2- 172.21.15.31:0/1455147527 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559dd435adf0 0x559dd43bd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 >> 172.21.15.31:0/1455147527 conn(0x559dd434ffd0 msgr2=0x559dd435eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7febe57fa700 0 client.30577 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 shutdown_connections 2023-07-29T16:38:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.362+0000 7fec1780eb80 1 -- 172.21.15.31:0/1455147527 wait complete. 2023-07-29T16:38:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.363+0000 7fec1780eb80 20 client.30577 trim_cache size 0 max 16384 2023-07-29T16:38:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:38:34.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:38:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:38:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:38:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.364+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7febec0a0810 msgr2=0x7febec0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:34.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.364+0000 7fec03dff700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7febec0a0810 0x7febec0a2cc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x559dd435a610 tx=0x7febe8005fb0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.364+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc108150 msgr2=0x7febfc19db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.364+0000 7fec03dff700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc108150 0x7febfc19db10 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7febf400eab0 tx=0x7febf400edc0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 shutdown_connections 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7febec0a0810 0x7febec0a2cc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febfc108150 0x7febfc19db10 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febfc068e60 0x7febfc19c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 --2- 172.21.15.31:0/1138366757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7febfc105830 0x7febfc19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 >> 172.21.15.31:0/1138366757 conn(0x7febfc0fa840 msgr2=0x7febfc106490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 shutdown_connections 2023-07-29T16:38:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:34.365+0000 7fec03dff700 1 -- 172.21.15.31:0/1138366757 wait complete. 2023-07-29T16:38:35.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:34 smithi156 ceph-mon[129245]: pgmap v1065: 97 pgs: 97 active+clean; 716 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 563 op/s 2023-07-29T16:38:35.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:34 smithi031 ceph-mon[95504]: pgmap v1065: 97 pgs: 97 active+clean; 716 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 563 op/s 2023-07-29T16:38:35.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:34 smithi039 ceph-mon[128858]: pgmap v1065: 97 pgs: 97 active+clean; 716 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.0 MiB/s wr, 563 op/s 2023-07-29T16:38:35.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 319042 34.23 MB/sec execute 379 sec latency 359.800 ms 2023-07-29T16:38:36.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 319180 34.16 MB/sec execute 380 sec latency 795.914 ms 2023-07-29T16:38:37.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:36 smithi031 ceph-mon[95504]: pgmap v1066: 97 pgs: 97 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 463 op/s 2023-07-29T16:38:37.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:36 smithi039 ceph-mon[128858]: pgmap v1066: 97 pgs: 97 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 463 op/s 2023-07-29T16:38:37.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:36 smithi156 ceph-mon[129245]: pgmap v1066: 97 pgs: 97 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 463 op/s 2023-07-29T16:38:37.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 319822 34.15 MB/sec execute 381 sec latency 102.938 ms 2023-07-29T16:38:38.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 320278 34.16 MB/sec execute 382 sec latency 78.899 ms 2023-07-29T16:38:38.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:38 smithi039 ceph-mon[128858]: pgmap v1067: 97 pgs: 97 active+clean; 741 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.6 MiB/s wr, 575 op/s 2023-07-29T16:38:38.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:38 smithi156 ceph-mon[129245]: pgmap v1067: 97 pgs: 97 active+clean; 741 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.6 MiB/s wr, 575 op/s 2023-07-29T16:38:38.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:38 smithi031 ceph-mon[95504]: pgmap v1067: 97 pgs: 97 active+clean; 741 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 7.6 MiB/s wr, 575 op/s 2023-07-29T16:38:39.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 320807 34.14 MB/sec execute 383 sec latency 80.690 ms 2023-07-29T16:38:40.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 321151 34.10 MB/sec execute 384 sec latency 136.196 ms 2023-07-29T16:38:41.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:40 smithi031 ceph-mon[95504]: pgmap v1068: 97 pgs: 97 active+clean; 736 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 11 MiB/s wr, 706 op/s 2023-07-29T16:38:41.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:40 smithi039 ceph-mon[128858]: pgmap v1068: 97 pgs: 97 active+clean; 736 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 11 MiB/s wr, 706 op/s 2023-07-29T16:38:41.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:40 smithi156 ceph-mon[129245]: pgmap v1068: 97 pgs: 97 active+clean; 736 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 11 MiB/s wr, 706 op/s 2023-07-29T16:38:41.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 321278 34.04 MB/sec execute 385 sec latency 734.802 ms 2023-07-29T16:38:42.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 321650 34.02 MB/sec execute 386 sec latency 838.105 ms 2023-07-29T16:38:43.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:42 smithi031 ceph-mon[95504]: pgmap v1069: 97 pgs: 97 active+clean; 727 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 754 op/s 2023-07-29T16:38:43.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:42 smithi039 ceph-mon[128858]: pgmap v1069: 97 pgs: 97 active+clean; 727 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 754 op/s 2023-07-29T16:38:43.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:42 smithi156 ceph-mon[129245]: pgmap v1069: 97 pgs: 97 active+clean; 727 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 754 op/s 2023-07-29T16:38:43.362 INFO:tasks.workunit.client.0.smithi031.stdout: 10 322276 33.99 MB/sec execute 387 sec latency 124.472 ms 2023-07-29T16:38:44.363 INFO:tasks.workunit.client.0.smithi031.stdout: 10 323221 34.05 MB/sec execute 388 sec latency 39.221 ms 2023-07-29T16:38:45.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:44 smithi031 ceph-mon[95504]: pgmap v1070: 97 pgs: 97 active+clean; 708 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 875 op/s 2023-07-29T16:38:45.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:44 smithi039 ceph-mon[128858]: pgmap v1070: 97 pgs: 97 active+clean; 708 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 875 op/s 2023-07-29T16:38:45.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:44 smithi156 ceph-mon[129245]: pgmap v1070: 97 pgs: 97 active+clean; 708 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 875 op/s 2023-07-29T16:38:45.363 INFO:tasks.workunit.client.0.smithi031.stdout: 10 323828 34.07 MB/sec execute 389 sec latency 211.870 ms 2023-07-29T16:38:46.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 324276 34.06 MB/sec execute 390 sec latency 89.473 ms 2023-07-29T16:38:47.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:46 smithi031 ceph-mon[95504]: pgmap v1071: 97 pgs: 97 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 17 MiB/s wr, 886 op/s 2023-07-29T16:38:48.022 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:46 smithi156 ceph-mon[129245]: pgmap v1071: 97 pgs: 97 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 17 MiB/s wr, 886 op/s 2023-07-29T16:38:48.023 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:46 smithi039 ceph-mon[128858]: pgmap v1071: 97 pgs: 97 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 17 MiB/s wr, 886 op/s 2023-07-29T16:38:48.023 INFO:tasks.workunit.client.0.smithi031.stdout: 10 324772 34.02 MB/sec execute 391 sec latency 73.708 ms 2023-07-29T16:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 -- 172.21.15.31:0/785823277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101530 msgr2=0x7fba28101980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 --2- 172.21.15.31:0/785823277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101530 0x7fba28101980 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fba1c009990 tx=0x7fba1c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 -- 172.21.15.31:0/785823277 shutdown_connections 2023-07-29T16:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 --2- 172.21.15.31:0/785823277 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba28101ec0 0x7fba28108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 --2- 172.21.15.31:0/785823277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba28100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 --2- 172.21.15.31:0/785823277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101530 0x7fba28101980 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 -- 172.21.15.31:0/785823277 >> 172.21.15.31:0/785823277 conn(0x7fba280fb920 msgr2=0x7fba280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.603+0000 7fba2cdff700 1 -- 172.21.15.31:0/785823277 shutdown_connections 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.604+0000 7fba2cdff700 1 -- 172.21.15.31:0/785823277 wait complete. 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.604+0000 7fba2cdff700 1 Processor -- start 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2cdff700 1 -- start start 2023-07-29T16:38:48.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2cdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba2810e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2cdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba28101530 0x7fba2810e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba2810e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba2810e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37166/0 (socket says 172.21.15.31:37166) 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2cdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101ec0 0x7fba2810f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2659c700 1 -- 172.21.15.31:0/2398104853 learned_addr learned my addr 172.21.15.31:0/2398104853 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2810fc50 con 0x7fba28100340 2023-07-29T16:38:48.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.605+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2810fdc0 con 0x7fba28101ec0 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2810ff80 con 0x7fba28101530 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba26d9d700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101ec0 0x7fba2810f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba25d9b700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba28101530 0x7fba2810e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba2659c700 1 -- 172.21.15.31:0/2398104853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba28101530 msgr2=0x7fba2810e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba2659c700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba28101530 0x7fba2810e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba2659c700 1 -- 172.21.15.31:0/2398104853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101ec0 msgr2=0x7fba2810f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.606+0000 7fba2659c700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101ec0 0x7fba2810f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.607+0000 7fba2659c700 1 -- 172.21.15.31:0/2398104853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c0095f0 con 0x7fba28100340 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.607+0000 7fba2659c700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba2810e1e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fba1800e970 tx=0x7fba1800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.608+0000 7fba177fe700 1 -- 172.21.15.31:0/2398104853 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18009790 con 0x7fba28100340 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.608+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28110260 con 0x7fba28100340 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.608+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba28110a10 con 0x7fba28100340 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.609+0000 7fba177fe700 1 -- 172.21.15.31:0/2398104853 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba1800c7e0 con 0x7fba28100340 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.609+0000 7fba177fe700 1 -- 172.21.15.31:0/2398104853 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18003c70 con 0x7fba28100340 2023-07-29T16:38:48.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.614+0000 7fba177fe700 1 -- 172.21.15.31:0/2398104853 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fba18004220 con 0x7fba28100340 2023-07-29T16:38:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.617+0000 7fba177fe700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba100a0870 0x7fba100a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.617+0000 7fba4072ab80 1 Processor -- start 2023-07-29T16:38:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.617+0000 7fba177fe700 1 -- 172.21.15.31:0/2398104853 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fba18014070 con 0x7fba28100340 2023-07-29T16:38:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 -- start start 2023-07-29T16:38:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0d97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0da160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc0309b0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc02e370 con 0x55a0fc0d8df0 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.618+0000 7fba4072ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc029c60 con 0x55a0fbf346b0 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.619+0000 7fba2659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.619+0000 7fba2659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0d88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45100/0 (socket says 172.21.15.31:45100) 2023-07-29T16:38:48.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.619+0000 7fba2659c700 1 -- 172.21.15.31:0/2674775901 learned_addr learned my addr 172.21.15.31:0/2674775901 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba26d9d700 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0da160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba25d9b700 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0d97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba25d9b700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba100a0870 0x7fba100a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba2659c700 1 -- 172.21.15.31:0/2674775901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 msgr2=0x55a0fc0d97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba2659c700 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0d97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba2659c700 1 -- 172.21.15.31:0/2674775901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d9cf0 msgr2=0x55a0fc0da160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba2659c700 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0da160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba2659c700 1 -- 172.21.15.31:0/2674775901 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0fbe57d80 con 0x55a0fbf346b0 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.620+0000 7fba2659c700 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0d88b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fba18095880 tx=0x7fba18098d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8baa70f79715a8d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.621+0000 7fba14ff9700 1 -- 172.21.15.31:0/2674775901 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1809d170 con 0x55a0fbf346b0 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.621+0000 7fba14ff9700 1 -- 172.21.15.31:0/2674775901 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba180a17d0 con 0x55a0fbf346b0 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.621+0000 7fba14ff9700 1 -- 172.21.15.31:0/2674775901 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18099840 con 0x55a0fbf346b0 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.622+0000 7fba4072ab80 1 -- 172.21.15.31:0/2674775901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 msgr2=0x55a0fc0d88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.622+0000 7fba4072ab80 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0d88b0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fba18095880 tx=0x7fba18098d60 comp rx=0 tx=0).stop 2023-07-29T16:38:48.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.623+0000 7fba4072ab80 1 -- 172.21.15.31:0/2674775901 shutdown_connections 2023-07-29T16:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.623+0000 7fba4072ab80 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0da160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.623+0000 7fba4072ab80 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0d88b0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.623+0000 7fba4072ab80 1 --2- 172.21.15.31:0/2674775901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0d97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.623+0000 7fba4072ab80 1 -- 172.21.15.31:0/2674775901 >> 172.21.15.31:0/2674775901 conn(0x55a0fc0cdfd0 msgr2=0x55a0fc0d3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.624+0000 7fba25d9b700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba100a0870 0x7fba100a2d20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00b2e0 tx=0x7fba1c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.625+0000 7fba4072ab80 1 -- 172.21.15.31:0/2674775901 shutdown_connections 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.625+0000 7fba4072ab80 1 -- 172.21.15.31:0/2674775901 wait complete. 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.625+0000 7fba4072ab80 1 -- start start 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.626+0000 7fba4072ab80 1 Processor -- start 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.626+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.626+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.627+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.627+0000 7fba4072ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fbfc64b0 con 0x55a0fc0d8df0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.627+0000 7fba4072ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fbfc6150 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.627+0000 7fba4072ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fbead8c0 con 0x55a0fbf346b0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.627+0000 7fba4072ab80 10 client.0 _ll_register_callbacks cb 0x55a0fbfcf5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.628+0000 7fba2659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.628+0000 7fba26d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.628+0000 7fba25d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.628+0000 7fba25d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37186/0 (socket says 172.21.15.31:37186) 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.628+0000 7fba25d9b700 1 -- 172.21.15.31:0/3743161020 learned_addr learned my addr 172.21.15.31:0/3743161020 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba26d9d700 1 -- 172.21.15.31:0/3743161020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 msgr2=0x55a0fc0e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba26d9d700 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba26d9d700 1 -- 172.21.15.31:0/3743161020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 msgr2=0x55a0fc0e2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba26d9d700 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba26d9d700 1 -- 172.21.15.31:0/3743161020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba28110a10 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba26d9d700 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e94d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fba20007af0 tx=0x7fba20007b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cc4780f1a3c6b35 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.629+0000 7fba0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.630+0000 7fba0dffb700 1 -- 172.21.15.31:0/3743161020 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2000e040 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.630+0000 7fba0dffb700 1 -- 172.21.15.31:0/3743161020 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fba20003c00 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.630+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0fc054cb0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.630+0000 7fba0dffb700 1 -- 172.21.15.31:0/3743161020 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba20003de0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.630+0000 7fba4072ab80 10 client.30563 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:48.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.630+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55a0fbe40c00 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba0dffb700 1 -- 172.21.15.31:0/3743161020 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a0fc054cb0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba0dffb700 1 -- 172.21.15.31:0/3743161020 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fba200043d0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba4072ab80 10 client.30563 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba4072ab80 10 client.30563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba4072ab80 20 client.30563 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba4072ab80 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc0eddb0 0x55a0fc0f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba4072ab80 4 client.30563 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0fc0a7d10 con 0x55a0fc0eddb0 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.631+0000 7fba2659c700 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc0eddb0 0x55a0fc0f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.632+0000 7fba2659c700 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc0eddb0 0x55a0fc0f01c0 crc :-1 s=READY pgs=493 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7457a7c3fc318a3d server_cookie=420e8d648a9096e8 in_seq=0 out_seq=0 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.632+0000 7fba0dffb700 10 client.30563 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba0dffb700 1 -- 172.21.15.31:0/3743161020 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a0fc0a7d10 con 0x55a0fc0eddb0 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba0dffb700 10 client.30563 handle_command_reply: tid=0 2023-07-29T16:38:48.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 client.30563 shutdown 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 msgr2=0x55a0fc0e94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e94d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fba20007af0 tx=0x7fba20007b20 comp rx=0 tx=0).stop 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 shutdown_connections 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc0eddb0 0x55a0fc0f01c0 crc :-1 s=READY pgs=493 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e94d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 --2- 172.21.15.31:0/3743161020 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 >> 172.21.15.31:0/3743161020 conn(0x55a0fc0cdfd0 msgr2=0x55a0fc0d2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba0dffb700 0 client.30563 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.637+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 shutdown_connections 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.638+0000 7fba4072ab80 1 -- 172.21.15.31:0/3743161020 wait complete. 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.638+0000 7fba4072ab80 20 client.30563 trim_cache size 0 max 16384 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.664+0000 7fba4072ab80 1 Processor -- start 2023-07-29T16:38:48.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.664+0000 7fba4072ab80 1 -- start start 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.665+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.665+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.665+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.665+0000 7fba4072ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc0d4df0 con 0x55a0fc0d8df0 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.665+0000 7fba4072ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc0d43a0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.665+0000 7fba4072ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc0dfea0 con 0x55a0fbf346b0 2023-07-29T16:38:48.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.666+0000 7fba26d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.666+0000 7fba26d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50258/0 (socket says 172.21.15.31:50258) 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.666+0000 7fba26d9d700 1 -- 172.21.15.31:0/4223058172 learned_addr learned my addr 172.21.15.31:0/4223058172 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.666+0000 7fba25d9b700 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba2659c700 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba25d9b700 1 -- 172.21.15.31:0/4223058172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 msgr2=0x55a0fc0e73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba25d9b700 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba25d9b700 1 -- 172.21.15.31:0/4223058172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 msgr2=0x55a0fc0e0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba25d9b700 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba25d9b700 1 -- 172.21.15.31:0/4223058172 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0fbe7e1f0 con 0x55a0fc0d8df0 2023-07-29T16:38:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba2659c700 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:38:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.667+0000 7fba25d9b700 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e00f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00b620 tx=0x7fba1c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51e298329f9c605 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.669+0000 7fba157fa700 1 -- 172.21.15.31:0/4223058172 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c00be40 con 0x55a0fc0d8df0 2023-07-29T16:38:48.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.669+0000 7fba157fa700 1 -- 172.21.15.31:0/4223058172 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba1c00fbd0 con 0x55a0fc0d8df0 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.669+0000 7fba157fa700 1 -- 172.21.15.31:0/4223058172 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1c030d70 con 0x55a0fc0d8df0 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.670+0000 7fba4072ab80 1 -- 172.21.15.31:0/4223058172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 msgr2=0x55a0fc0e00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.670+0000 7fba4072ab80 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e00f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00b620 tx=0x7fba1c01a040 comp rx=0 tx=0).stop 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 -- 172.21.15.31:0/4223058172 shutdown_connections 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc0e0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fbf346b0 0x55a0fc0e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 --2- 172.21.15.31:0/4223058172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e00f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 -- 172.21.15.31:0/4223058172 >> 172.21.15.31:0/4223058172 conn(0x55a0fc0cdfd0 msgr2=0x55a0fc0e0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 -- 172.21.15.31:0/4223058172 shutdown_connections 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.673+0000 7fba4072ab80 1 -- 172.21.15.31:0/4223058172 wait complete. 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.674+0000 7fba4072ab80 1 -- start start 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.674+0000 7fba4072ab80 1 Processor -- start 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fbf346b0 0x55a0fc13d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc13bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc13c200 con 0x55a0fbf346b0 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc13c370 con 0x55a0fc0d8df0 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0fc13c530 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.675+0000 7fba4072ab80 10 client.0 _ll_register_callbacks cb 0x55a0fbfcf5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba25d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba25d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:50264/0 (socket says 172.21.15.31:50264) 2023-07-29T16:38:48.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba25d9b700 1 -- 172.21.15.31:0/362486898 learned_addr learned my addr 172.21.15.31:0/362486898 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba26d9d700 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc13bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba26d9d700 1 -- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 msgr2=0x55a0fc0e73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba26d9d700 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba26d9d700 1 -- 172.21.15.31:0/362486898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fbf346b0 msgr2=0x55a0fc13d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba26d9d700 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fbf346b0 0x55a0fc13d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.676+0000 7fba26d9d700 1 -- 172.21.15.31:0/362486898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1c02c040 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.677+0000 7fba26d9d700 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc13bac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x55a0fc0eab00 tx=0x7fba200164e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b7acac7e744ac4e server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.677+0000 7fba0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.677+0000 7fba0dffb700 1 -- 172.21.15.31:0/362486898 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2000e040 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.677+0000 7fba0dffb700 1 -- 172.21.15.31:0/362486898 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fba20016be0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.677+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba180a17d0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.681+0000 7fba0dffb700 1 -- 172.21.15.31:0/362486898 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba20016de0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.681+0000 7fba4072ab80 10 client.30589 fetch_fsmap learned FSMap version 43 2023-07-29T16:38:48.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.681+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55a0fbe56250 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba0dffb700 1 -- 172.21.15.31:0/362486898 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba180a17d0 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba0dffb700 1 -- 172.21.15.31:0/362486898 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fba20033830 con 0x55a0fc0d9cf0 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba4072ab80 10 client.30589 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba4072ab80 10 client.30589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba4072ab80 20 client.30589 populate_metadata read hostname 'smithi031' 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba4072ab80 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc151d20 0x55a0fc154130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba4072ab80 4 client.30589 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:38:48.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.683+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a0fc022ce0 con 0x55a0fc151d20 2023-07-29T16:38:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.684+0000 7fba2659c700 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc151d20 0x55a0fc154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:38:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.685+0000 7fba0dffb700 10 client.30589 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:48.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:47.685+0000 7fba2659c700 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc151d20 0x55a0fc154130 crc :-1 s=READY pgs=494 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9d4c57f98942129 server_cookie=bb85edebbd7957c3 in_seq=0 out_seq=0 2023-07-29T16:38:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba0dffb700 1 -- 172.21.15.31:0/362486898 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a0fc022ce0 con 0x55a0fc151d20 2023-07-29T16:38:48.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba0dffb700 10 client.30589 handle_command_reply: tid=0 2023-07-29T16:38:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 client.30589 shutdown 2023-07-29T16:38:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fc0d9cf0 msgr2=0x55a0fc13bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc13bac0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x55a0fc0eab00 tx=0x7fba200164e0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 shutdown_connections 2023-07-29T16:38:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a0fc151d20 0x55a0fc154130 crc :-1 s=READY pgs=494 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a0fc0d9cf0 0x55a0fc13bac0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a0fbf346b0 0x55a0fc13d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 --2- 172.21.15.31:0/362486898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a0fc0d8df0 0x55a0fc0e73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 >> 172.21.15.31:0/362486898 conn(0x55a0fc0cdfd0 msgr2=0x55a0fc0e00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:48.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba0dffb700 0 client.30589 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:38:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 shutdown_connections 2023-07-29T16:38:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.166+0000 7fba4072ab80 1 -- 172.21.15.31:0/362486898 wait complete. 2023-07-29T16:38:48.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.167+0000 7fba4072ab80 20 client.30589 trim_cache size 0 max 16384 2023-07-29T16:38:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:38:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:38:48.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:38:48.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:38:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.169+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba100a0870 msgr2=0x7fba100a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.169+0000 7fba2cdff700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba100a0870 0x7fba100a2d20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fba1c00b2e0 tx=0x7fba1c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 msgr2=0x7fba2810e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba2810e1e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fba1800e970 tx=0x7fba1800ec80 comp rx=0 tx=0).stop 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 shutdown_connections 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba100a0870 0x7fba100a2d20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba28101ec0 0x7fba2810f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba28100340 0x7fba2810e1e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 --2- 172.21.15.31:0/2398104853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba28101530 0x7fba2810e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:38:48.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 >> 172.21.15.31:0/2398104853 conn(0x7fba280fb920 msgr2=0x7fba280fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:38:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 shutdown_connections 2023-07-29T16:38:48.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:38:48.170+0000 7fba2cdff700 1 -- 172.21.15.31:0/2398104853 wait complete. 2023-07-29T16:38:48.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 325202 33.98 MB/sec execute 392 sec latency 337.454 ms 2023-07-29T16:38:48.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:48 smithi039 ceph-mon[128858]: pgmap v1072: 97 pgs: 97 active+clean; 659 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 17 MiB/s wr, 883 op/s 2023-07-29T16:38:48.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:48 smithi156 ceph-mon[129245]: pgmap v1072: 97 pgs: 97 active+clean; 659 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 17 MiB/s wr, 883 op/s 2023-07-29T16:38:48.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:48 smithi031 ceph-mon[95504]: pgmap v1072: 97 pgs: 97 active+clean; 659 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 17 MiB/s wr, 883 op/s 2023-07-29T16:38:49.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 325911 34.01 MB/sec execute 393 sec latency 74.603 ms 2023-07-29T16:38:50.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 326520 33.99 MB/sec execute 394 sec latency 105.713 ms 2023-07-29T16:38:51.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:50 smithi156 ceph-mon[129245]: pgmap v1073: 97 pgs: 97 active+clean; 646 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 833 op/s 2023-07-29T16:38:51.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:50 smithi031 ceph-mon[95504]: pgmap v1073: 97 pgs: 97 active+clean; 646 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 833 op/s 2023-07-29T16:38:51.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:50 smithi039 ceph-mon[128858]: pgmap v1073: 97 pgs: 97 active+clean; 646 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 833 op/s 2023-07-29T16:38:51.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 327152 33.99 MB/sec execute 395 sec latency 90.129 ms 2023-07-29T16:38:52.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 327878 34.00 MB/sec execute 396 sec latency 63.100 ms 2023-07-29T16:38:53.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:52 smithi156 ceph-mon[129245]: pgmap v1074: 97 pgs: 97 active+clean; 643 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.9 MiB/s wr, 726 op/s 2023-07-29T16:38:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:52 smithi031 ceph-mon[95504]: pgmap v1074: 97 pgs: 97 active+clean; 643 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.9 MiB/s wr, 726 op/s 2023-07-29T16:38:53.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:52 smithi039 ceph-mon[128858]: pgmap v1074: 97 pgs: 97 active+clean; 643 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 9.9 MiB/s wr, 726 op/s 2023-07-29T16:38:53.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 328648 34.02 MB/sec execute 397 sec latency 61.582 ms 2023-07-29T16:38:54.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 329289 34.05 MB/sec execute 398 sec latency 51.743 ms 2023-07-29T16:38:55.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:54 smithi156 ceph-mon[129245]: pgmap v1075: 97 pgs: 97 active+clean; 653 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.8 MiB/s wr, 774 op/s 2023-07-29T16:38:55.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:54 smithi031 ceph-mon[95504]: pgmap v1075: 97 pgs: 97 active+clean; 653 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.8 MiB/s wr, 774 op/s 2023-07-29T16:38:55.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:54 smithi039 ceph-mon[128858]: pgmap v1075: 97 pgs: 97 active+clean; 653 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 8.8 MiB/s wr, 774 op/s 2023-07-29T16:38:55.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 329763 34.01 MB/sec execute 399 sec latency 74.478 ms 2023-07-29T16:38:56.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 330449 34.01 MB/sec execute 400 sec latency 91.981 ms 2023-07-29T16:38:57.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:56 smithi156 ceph-mon[129245]: pgmap v1076: 97 pgs: 97 active+clean; 656 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 578 op/s 2023-07-29T16:38:57.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:56 smithi031 ceph-mon[95504]: pgmap v1076: 97 pgs: 97 active+clean; 656 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 578 op/s 2023-07-29T16:38:57.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:56 smithi039 ceph-mon[128858]: pgmap v1076: 97 pgs: 97 active+clean; 656 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 578 op/s 2023-07-29T16:38:57.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 331067 33.99 MB/sec execute 401 sec latency 111.876 ms 2023-07-29T16:38:58.366 INFO:tasks.workunit.client.0.smithi031.stdout: 10 331890 34.05 MB/sec execute 402 sec latency 48.379 ms 2023-07-29T16:38:58.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:38:58 smithi039 ceph-mon[128858]: pgmap v1077: 97 pgs: 97 active+clean; 663 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 565 op/s 2023-07-29T16:38:58.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:38:58 smithi156 ceph-mon[129245]: pgmap v1077: 97 pgs: 97 active+clean; 663 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 565 op/s 2023-07-29T16:38:58.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:38:58 smithi031 ceph-mon[95504]: pgmap v1077: 97 pgs: 97 active+clean; 663 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 565 op/s 2023-07-29T16:38:59.367 INFO:tasks.workunit.client.0.smithi031.stdout: 10 332532 34.04 MB/sec execute 403 sec latency 90.283 ms 2023-07-29T16:39:00.367 INFO:tasks.workunit.client.0.smithi031.stdout: 10 333236 34.04 MB/sec execute 404 sec latency 62.708 ms 2023-07-29T16:39:01.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:00 smithi156 ceph-mon[129245]: pgmap v1078: 97 pgs: 97 active+clean; 671 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.0 MiB/s wr, 583 op/s 2023-07-29T16:39:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:00 smithi031 ceph-mon[95504]: pgmap v1078: 97 pgs: 97 active+clean; 671 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.0 MiB/s wr, 583 op/s 2023-07-29T16:39:01.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:00 smithi039 ceph-mon[128858]: pgmap v1078: 97 pgs: 97 active+clean; 671 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.0 MiB/s wr, 583 op/s 2023-07-29T16:39:01.367 INFO:tasks.workunit.client.0.smithi031.stdout: 10 333960 34.07 MB/sec execute 405 sec latency 57.521 ms 2023-07-29T16:39:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 -- 172.21.15.31:0/2464966726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd10100e60 msgr2=0x7fdd10103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 --2- 172.21.15.31:0/2464966726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd10100e60 0x7fdd10103240 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fdd000098f0 tx=0x7fdd00009c00 comp rx=0 tx=0).stop 2023-07-29T16:39:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 -- 172.21.15.31:0/2464966726 shutdown_connections 2023-07-29T16:39:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 --2- 172.21.15.31:0/2464966726 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd101060a0 0x7fdd10108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 --2- 172.21.15.31:0/2464966726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd10100e60 0x7fdd10103240 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 --2- 172.21.15.31:0/2464966726 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd10103780 0x7fdd10105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 -- 172.21.15.31:0/2464966726 >> 172.21.15.31:0/2464966726 conn(0x7fdd100fa840 msgr2=0x7fdd100fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 -- 172.21.15.31:0/2464966726 shutdown_connections 2023-07-29T16:39:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.385+0000 7fdd167ff700 1 -- 172.21.15.31:0/2464966726 wait complete. 2023-07-29T16:39:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.386+0000 7fdd167ff700 1 Processor -- start 2023-07-29T16:39:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.386+0000 7fdd167ff700 1 -- start start 2023-07-29T16:39:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd167ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd10100e60 0x7fdd10198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd167ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd10103780 0x7fdd101989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd167ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 0x7fdd10199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd167ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10199e90 con 0x7fdd10100e60 2023-07-29T16:39:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd167ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1019a000 con 0x7fdd101060a0 2023-07-29T16:39:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd167ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1019a1c0 con 0x7fdd10103780 2023-07-29T16:39:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd14d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 0x7fdd10199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.387+0000 7fdd14d9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 0x7fdd10199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34166/0 (socket says 172.21.15.31:34166) 2023-07-29T16:39:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 -- 172.21.15.31:0/4149905260 learned_addr learned my addr 172.21.15.31:0/4149905260 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd10103780 0x7fdd101989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd10100e60 0x7fdd10198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 -- 172.21.15.31:0/4149905260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd10103780 msgr2=0x7fdd101989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd10103780 0x7fdd101989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 -- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd10100e60 msgr2=0x7fdd10198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd10100e60 0x7fdd10198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 -- 172.21.15.31:0/4149905260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08005e70 con 0x7fdd101060a0 2023-07-29T16:39:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.388+0000 7fdd14d9c700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 0x7fdd10199830 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800efc0 tx=0x7fdd0800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.389+0000 7fdd0d7fa700 1 -- 172.21.15.31:0/4149905260 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08018070 con 0x7fdd101060a0 2023-07-29T16:39:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.389+0000 7fdd0d7fa700 1 -- 172.21.15.31:0/4149905260 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd0800f040 con 0x7fdd101060a0 2023-07-29T16:39:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.389+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd00009520 con 0x7fdd101060a0 2023-07-29T16:39:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.389+0000 7fdd0d7fa700 1 -- 172.21.15.31:0/4149905260 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08007c70 con 0x7fdd101060a0 2023-07-29T16:39:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.389+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd101a0a80 con 0x7fdd101060a0 2023-07-29T16:39:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.393+0000 7fdd0d7fa700 1 -- 172.21.15.31:0/4149905260 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fdd0800c930 con 0x7fdd101060a0 2023-07-29T16:39:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.394+0000 7fdd2a04bb80 1 Processor -- start 2023-07-29T16:39:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 -- start start 2023-07-29T16:39:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411e18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411e27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411e3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411399b0 con 0x5592411e1df0 2023-07-29T16:39:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559241137370 con 0x55924103d6b0 2023-07-29T16:39:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559241132c60 con 0x5592411e2cf0 2023-07-29T16:39:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd0d7fa700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdcf80a0810 0x7fdcf80a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411e18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411e18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34172/0 (socket says 172.21.15.31:34172) 2023-07-29T16:39:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/3811291909 learned_addr learned my addr 172.21.15.31:0/3811291909 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.395+0000 7fdd0ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411e27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0d7fa700 1 -- 172.21.15.31:0/4149905260 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdd080cbad0 con 0x7fdd101060a0 2023-07-29T16:39:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd14d9c700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdcf80a0810 0x7fdcf80a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/3811291909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 msgr2=0x5592411e3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411e3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/3811291909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 msgr2=0x5592411e27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411e27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/3811291909 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559240f60d80 con 0x55924103d6b0 2023-07-29T16:39:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:39:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd14d9c700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdcf80a0810 0x7fdcf80a2cc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdd08099190 tx=0x7fdd08009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.396+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411e18b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005950 tx=0x7fdd040058e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8636e0dde0f0adbd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.397+0000 7fdcf67fc700 1 -- 172.21.15.31:0/3811291909 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04004030 con 0x55924103d6b0 2023-07-29T16:39:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.397+0000 7fdcf67fc700 1 -- 172.21.15.31:0/3811291909 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd04004830 con 0x55924103d6b0 2023-07-29T16:39:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.397+0000 7fdcf67fc700 1 -- 172.21.15.31:0/3811291909 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04005260 con 0x55924103d6b0 2023-07-29T16:39:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.397+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/3811291909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 msgr2=0x5592411e18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.397+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411e18b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005950 tx=0x7fdd040058e0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/3811291909 shutdown_connections 2023-07-29T16:39:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411e18b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/3811291909 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411e27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/3811291909 >> 172.21.15.31:0/3811291909 conn(0x5592411d6fd0 msgr2=0x5592411dc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:01.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/3811291909 shutdown_connections 2023-07-29T16:39:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/3811291909 wait complete. 2023-07-29T16:39:01.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.398+0000 7fdd2a04bb80 1 -- start start 2023-07-29T16:39:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 Processor -- start 2023-07-29T16:39:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 0x5592411eaf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e2cf0 0x5592411f2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592410cf4b0 con 0x5592411e1df0 2023-07-29T16:39:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592410cf150 con 0x5592411e2cf0 2023-07-29T16:39:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.399+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559240fb68c0 con 0x55924103d6b0 2023-07-29T16:39:01.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd2a04bb80 10 client.0 _ll_register_callbacks cb 0x5592410d85c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 0x5592411eaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 0x5592411eaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:54502/0 (socket says 172.21.15.31:54502) 2023-07-29T16:39:01.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/1728739052 learned_addr learned my addr 172.21.15.31:0/1728739052 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e2cf0 0x5592411f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.413 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e2cf0 msgr2=0x5592411f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e2cf0 0x5592411f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/1728739052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 msgr2=0x5592411eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.400+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/1728739052 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd101090a0 con 0x55924103d6b0 2023-07-29T16:39:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.401+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 0x5592411eaf10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdd040098d0 tx=0x7fdd04004f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9de9d118cb98a9c0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.401+0000 7fdcef7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:39:01.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.401+0000 7fdcef7fe700 1 -- 172.21.15.31:0/1728739052 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04004a50 con 0x55924103d6b0 2023-07-29T16:39:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.401+0000 7fdcef7fe700 1 -- 172.21.15.31:0/1728739052 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdd04004bb0 con 0x55924103d6b0 2023-07-29T16:39:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.401+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55924115dcb0 con 0x55924103d6b0 2023-07-29T16:39:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdcef7fe700 1 -- 172.21.15.31:0/1728739052 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd04004d70 con 0x55924103d6b0 2023-07-29T16:39:01.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 10 client.30601 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x7fdd04019040 con 0x55924103d6b0 2023-07-29T16:39:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdcef7fe700 1 -- 172.21.15.31:0/1728739052 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55924115dcb0 con 0x55924103d6b0 2023-07-29T16:39:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdcef7fe700 1 -- 172.21.15.31:0/1728739052 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdd0401d070 con 0x55924103d6b0 2023-07-29T16:39:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 10 client.30601 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 10 client.30601 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 20 client.30601 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592411f6af0 0x5592411f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 4 client.30601 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:39:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.402+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592411b0d10 con 0x5592411f6af0 2023-07-29T16:39:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.405+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592411f6af0 0x5592411f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.406+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592411f6af0 0x5592411f8f00 crc :-1 s=READY pgs=495 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c46640ee18cbe8be server_cookie=94a57994a50e8671 in_seq=0 out_seq=0 2023-07-29T16:39:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.406+0000 7fdcef7fe700 10 client.30601 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.409+0000 7fdcef7fe700 1 -- 172.21.15.31:0/1728739052 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x5592411b0d10 con 0x5592411f6af0 2023-07-29T16:39:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdcef7fe700 10 client.30601 handle_command_reply: tid=0 2023-07-29T16:39:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 client.30601 shutdown 2023-07-29T16:39:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 msgr2=0x5592411eaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 0x5592411eaf10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fdd040098d0 tx=0x7fdd04004f60 comp rx=0 tx=0).stop 2023-07-29T16:39:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 shutdown_connections 2023-07-29T16:39:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592411f6af0 0x5592411f8f00 crc :-1 s=READY pgs=495 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e2cf0 0x5592411f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55924103d6b0 0x5592411eaf10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1728739052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 >> 172.21.15.31:0/1728739052 conn(0x5592411d6fd0 msgr2=0x5592411dba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdcef7fe700 0 client.30601 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 shutdown_connections 2023-07-29T16:39:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.410+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1728739052 wait complete. 2023-07-29T16:39:01.425 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.411+0000 7fdd2a04bb80 20 client.30601 trim_cache size 0 max 16384 2023-07-29T16:39:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.432+0000 7fdd2a04bb80 1 Processor -- start 2023-07-29T16:39:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.432+0000 7fdd2a04bb80 1 -- start start 2023-07-29T16:39:01.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55924103d6b0 0x5592411e1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 0x5592411f2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411ea5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411d69d0 con 0x55924103d6b0 2023-07-29T16:39:01.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411e66d0 con 0x5592411e1df0 2023-07-29T16:39:01.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411fd860 con 0x5592411e2cf0 2023-07-29T16:39:01.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55924103d6b0 0x5592411e1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd0ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 0x5592411f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd0ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 0x5592411f2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34188/0 (socket says 172.21.15.31:34188) 2023-07-29T16:39:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd0ffff700 1 -- 172.21.15.31:0/1597661111 learned_addr learned my addr 172.21.15.31:0/1597661111 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.433+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411ea5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdd0ffff700 1 -- 172.21.15.31:0/1597661111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 msgr2=0x5592411ea5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411ea5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdd0ffff700 1 -- 172.21.15.31:0/1597661111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55924103d6b0 msgr2=0x5592411e1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55924103d6b0 0x5592411e1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdd0ffff700 1 -- 172.21.15.31:0/1597661111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559240f49c00 con 0x5592411e1df0 2023-07-29T16:39:01.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 0x5592411f2450 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000fa40 tx=0x7fdd00005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6701dfc1bc46dc3 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdcf6ffd700 1 -- 172.21.15.31:0/1597661111 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00025230 con 0x5592411e1df0 2023-07-29T16:39:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdcf6ffd700 1 -- 172.21.15.31:0/1597661111 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd0001d040 con 0x5592411e1df0 2023-07-29T16:39:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.434+0000 7fdcf6ffd700 1 -- 172.21.15.31:0/1597661111 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000b960 con 0x5592411e1df0 2023-07-29T16:39:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.435+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1597661111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 msgr2=0x5592411f2450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.435+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 0x5592411f2450 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fdd0000fa40 tx=0x7fdd00005dc0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1597661111 shutdown_connections 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411ea5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55924103d6b0 0x5592411e1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/1597661111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5592411e1df0 0x5592411f2450 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1597661111 >> 172.21.15.31:0/1597661111 conn(0x5592411d6fd0 msgr2=0x5592411e9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1597661111 shutdown_connections 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.436+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/1597661111 wait complete. 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.437+0000 7fdd2a04bb80 1 -- start start 2023-07-29T16:39:01.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.437+0000 7fdd2a04bb80 1 Processor -- start 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411ee680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411f1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411f1c40 con 0x5592411e1df0 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411f1db0 con 0x55924103d6b0 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd0f7fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:34192/0 (socket says 172.21.15.31:34192) 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/4290268 learned_addr learned my addr 172.21.15.31:0/4290268 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592411f1f70 con 0x5592411e2cf0 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd2a04bb80 10 client.0 _ll_register_callbacks cb 0x5592410d85c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.438+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411f1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.439+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.439+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/4290268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 msgr2=0x5592411f1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.439+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411f1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.440+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/4290268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 msgr2=0x5592411eebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.440+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.440+0000 7fdd0f7fe700 1 -- 172.21.15.31:0/4290268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0000f640 con 0x55924103d6b0 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.440+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.440+0000 7fdd0f7fe700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411ee680 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005690 tx=0x7fdd0403d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3078f1396666dc24 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.440+0000 7fdcef7fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:39:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.441+0000 7fdcef7fe700 1 -- 172.21.15.31:0/4290268 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04040070 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.441+0000 7fdcef7fe700 1 -- 172.21.15.31:0/4290268 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fdd04000e40 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.441+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd04004830 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.441+0000 7fdcef7fe700 1 -- 172.21.15.31:0/4290268 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd04004c60 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.441+0000 7fdd2a04bb80 10 client.30587 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.441+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559240f871f0 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdcef7fe700 1 -- 172.21.15.31:0/4290268 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd04004830 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdcef7fe700 1 -- 172.21.15.31:0/4290268 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fdd0401d050 con 0x55924103d6b0 2023-07-29T16:39:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd2a04bb80 10 client.30587 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:01.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd2a04bb80 10 client.30587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:01.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd2a04bb80 20 client.30587 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:01.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592412452c0 0x5592412476d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:01.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd2a04bb80 4 client.30587 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:39:01.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55924112bce0 con 0x5592412452c0 2023-07-29T16:39:01.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.442+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592412452c0 0x5592412476d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:01.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.443+0000 7fdd0ffff700 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592412452c0 0x5592412476d0 crc :-1 s=READY pgs=496 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1f71767626a84c0 server_cookie=89ef9a547b3e32f5 in_seq=0 out_seq=0 2023-07-29T16:39:01.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.443+0000 7fdcef7fe700 10 client.30587 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:01.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.957+0000 7fdcef7fe700 1 -- 172.21.15.31:0/4290268 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55924112bce0 con 0x5592412452c0 2023-07-29T16:39:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.957+0000 7fdcef7fe700 10 client.30587 handle_command_reply: tid=0 2023-07-29T16:39:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.957+0000 7fdd2a04bb80 1 client.30587 shutdown 2023-07-29T16:39:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 msgr2=0x5592411ee680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411ee680 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005690 tx=0x7fdd0403d040 comp rx=0 tx=0).stop 2023-07-29T16:39:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 shutdown_connections 2023-07-29T16:39:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5592412452c0 0x5592412476d0 crc :-1 s=READY pgs=496 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5592411e2cf0 0x5592411f1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55924103d6b0 0x5592411ee680 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 --2- 172.21.15.31:0/4290268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592411e1df0 0x5592411eebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 >> 172.21.15.31:0/4290268 conn(0x5592411d6fd0 msgr2=0x5592411e9010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdcef7fe700 0 client.30587 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 shutdown_connections 2023-07-29T16:39:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.958+0000 7fdd2a04bb80 1 -- 172.21.15.31:0/4290268 wait complete. 2023-07-29T16:39:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.959+0000 7fdd2a04bb80 20 client.30587 trim_cache size 0 max 16384 2023-07-29T16:39:01.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:39:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:39:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:39:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:39:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.959+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdcf80a0810 msgr2=0x7fdcf80a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.959+0000 7fdd167ff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdcf80a0810 0x7fdcf80a2cc0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fdd08099190 tx=0x7fdd08009630 comp rx=0 tx=0).stop 2023-07-29T16:39:01.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.959+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 msgr2=0x7fdd10199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:01.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.959+0000 7fdd167ff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 0x7fdd10199830 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800efc0 tx=0x7fdd0800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 shutdown_connections 2023-07-29T16:39:01.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fdcf80a0810 0x7fdcf80a2cc0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd101060a0 0x7fdd10199830 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdd10100e60 0x7fdd10198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 --2- 172.21.15.31:0/4149905260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd10103780 0x7fdd101989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 >> 172.21.15.31:0/4149905260 conn(0x7fdd100fa840 msgr2=0x7fdd100ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 shutdown_connections 2023-07-29T16:39:01.971 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:01.960+0000 7fdd167ff700 1 -- 172.21.15.31:0/4149905260 wait complete. 2023-07-29T16:39:02.367 INFO:tasks.workunit.client.0.smithi031.stdout: 10 334245 34.03 MB/sec execute 406 sec latency 328.787 ms 2023-07-29T16:39:03.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:02 smithi031 ceph-mon[95504]: pgmap v1079: 97 pgs: 97 active+clean; 673 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 449 op/s 2023-07-29T16:39:03.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:02 smithi039 ceph-mon[128858]: pgmap v1079: 97 pgs: 97 active+clean; 673 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 449 op/s 2023-07-29T16:39:03.337 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:02 smithi156 ceph-mon[129245]: pgmap v1079: 97 pgs: 97 active+clean; 673 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 449 op/s 2023-07-29T16:39:03.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 334519 33.99 MB/sec execute 407 sec latency 615.703 ms 2023-07-29T16:39:04.376 INFO:tasks.workunit.client.0.smithi031.stdout: 10 334557 33.91 MB/sec execute 408 sec latency 1497.976 ms 2023-07-29T16:39:05.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:04 smithi031 ceph-mon[95504]: pgmap v1080: 97 pgs: 97 active+clean; 690 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.4 MiB/s wr, 621 op/s 2023-07-29T16:39:05.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:04 smithi039 ceph-mon[128858]: pgmap v1080: 97 pgs: 97 active+clean; 690 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.4 MiB/s wr, 621 op/s 2023-07-29T16:39:05.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:04 smithi156 ceph-mon[129245]: pgmap v1080: 97 pgs: 97 active+clean; 690 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.4 MiB/s wr, 621 op/s 2023-07-29T16:39:05.374 INFO:tasks.workunit.client.0.smithi031.stdout: 10 334907 33.85 MB/sec execute 409 sec latency 950.046 ms 2023-07-29T16:39:06.374 INFO:tasks.workunit.client.0.smithi031.stdout: 10 335210 33.80 MB/sec execute 410 sec latency 127.100 ms 2023-07-29T16:39:07.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:06 smithi031 ceph-mon[95504]: pgmap v1081: 97 pgs: 97 active+clean; 720 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.7 MiB/s wr, 576 op/s 2023-07-29T16:39:07.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:06 smithi039 ceph-mon[128858]: pgmap v1081: 97 pgs: 97 active+clean; 720 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.7 MiB/s wr, 576 op/s 2023-07-29T16:39:07.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:06 smithi156 ceph-mon[129245]: pgmap v1081: 97 pgs: 97 active+clean; 720 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 8.7 MiB/s wr, 576 op/s 2023-07-29T16:39:07.376 INFO:tasks.workunit.client.0.smithi031.stdout: 10 335805 33.79 MB/sec execute 411 sec latency 94.065 ms 2023-07-29T16:39:08.377 INFO:tasks.workunit.client.0.smithi031.stdout: 10 336322 33.79 MB/sec execute 412 sec latency 274.362 ms 2023-07-29T16:39:08.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:08 smithi039 ceph-mon[128858]: pgmap v1082: 97 pgs: 97 active+clean; 739 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 778 op/s 2023-07-29T16:39:08.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:08 smithi156 ceph-mon[129245]: pgmap v1082: 97 pgs: 97 active+clean; 739 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 778 op/s 2023-07-29T16:39:08.686 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:08 smithi031 ceph-mon[95504]: pgmap v1082: 97 pgs: 97 active+clean; 739 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 778 op/s 2023-07-29T16:39:09.377 INFO:tasks.workunit.client.0.smithi031.stdout: 10 337031 33.83 MB/sec execute 413 sec latency 68.615 ms 2023-07-29T16:39:10.377 INFO:tasks.workunit.client.0.smithi031.stdout: 10 337529 33.79 MB/sec execute 414 sec latency 67.338 ms 2023-07-29T16:39:10.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:10 smithi031 ceph-mon[95504]: pgmap v1083: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 16 MiB/s wr, 844 op/s 2023-07-29T16:39:11.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:10 smithi156 ceph-mon[129245]: pgmap v1083: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 16 MiB/s wr, 844 op/s 2023-07-29T16:39:11.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:10 smithi039 ceph-mon[128858]: pgmap v1083: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 16 MiB/s wr, 844 op/s 2023-07-29T16:39:11.377 INFO:tasks.workunit.client.0.smithi031.stdout: 10 337882 33.76 MB/sec execute 415 sec latency 82.712 ms 2023-07-29T16:39:12.377 INFO:tasks.workunit.client.0.smithi031.stdout: 10 338467 33.74 MB/sec execute 416 sec latency 56.615 ms 2023-07-29T16:39:13.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:12 smithi156 ceph-mon[129245]: pgmap v1084: 97 pgs: 97 active+clean; 671 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 16 MiB/s wr, 846 op/s 2023-07-29T16:39:13.126 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:12 smithi031 ceph-mon[95504]: pgmap v1084: 97 pgs: 97 active+clean; 671 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 16 MiB/s wr, 846 op/s 2023-07-29T16:39:13.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:12 smithi039 ceph-mon[128858]: pgmap v1084: 97 pgs: 97 active+clean; 671 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 16 MiB/s wr, 846 op/s 2023-07-29T16:39:13.378 INFO:tasks.workunit.client.0.smithi031.stdout: 10 338942 33.70 MB/sec execute 417 sec latency 338.100 ms 2023-07-29T16:39:14.378 INFO:tasks.workunit.client.0.smithi031.stdout: 10 339317 33.69 MB/sec execute 418 sec latency 544.677 ms 2023-07-29T16:39:15.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:14 smithi156 ceph-mon[129245]: pgmap v1085: 97 pgs: 97 active+clean; 673 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 17 MiB/s wr, 898 op/s 2023-07-29T16:39:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 -- 172.21.15.31:0/3326651706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8100e60 msgr2=0x7f57a8103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 --2- 172.21.15.31:0/3326651706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8100e60 0x7f57a8103240 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5798009990 tx=0x7f5798009ca0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 -- 172.21.15.31:0/3326651706 shutdown_connections 2023-07-29T16:39:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 --2- 172.21.15.31:0/3326651706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57a81060a0 0x7f57a8108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 --2- 172.21.15.31:0/3326651706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8100e60 0x7f57a8103240 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 --2- 172.21.15.31:0/3326651706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57a8103780 0x7f57a8105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.184+0000 7f57acfff700 1 -- 172.21.15.31:0/3326651706 >> 172.21.15.31:0/3326651706 conn(0x7f57a80fa840 msgr2=0x7f57a80fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.185+0000 7f57acfff700 1 -- 172.21.15.31:0/3326651706 shutdown_connections 2023-07-29T16:39:15.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.185+0000 7f57acfff700 1 -- 172.21.15.31:0/3326651706 wait complete. 2023-07-29T16:39:15.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:14 smithi031 ceph-mon[95504]: pgmap v1085: 97 pgs: 97 active+clean; 673 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 17 MiB/s wr, 898 op/s 2023-07-29T16:39:15.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.187+0000 7f57acfff700 1 Processor -- start 2023-07-29T16:39:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.187+0000 7f57acfff700 1 -- start start 2023-07-29T16:39:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f57acfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57a8100e60 0x7f57a8196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f57acfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 0x7f57a8196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f57acfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57a81060a0 0x7f57a8197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f57acfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a8197b70 con 0x7f57a8100e60 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f57acfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a8197ce0 con 0x7f57a81060a0 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f57acfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a8197ea0 con 0x7f57a8103780 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f579ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 0x7f57a8196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f579ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 0x7f57a8196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47044/0 (socket says 172.21.15.31:47044) 2023-07-29T16:39:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.188+0000 7f579ffff700 1 -- 172.21.15.31:0/1736685324 learned_addr learned my addr 172.21.15.31:0/1736685324 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579ffff700 1 -- 172.21.15.31:0/1736685324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57a81060a0 msgr2=0x7f57a8197510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-29T16:39:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579ffff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57a81060a0 0x7f57a8197510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579ffff700 1 -- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57a8100e60 msgr2=0x7f57a8196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579ffff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57a8100e60 0x7f57a8196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579ffff700 1 -- 172.21.15.31:0/1736685324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57980095f0 con 0x7f57a8103780 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579ffff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 0x7f57a8196690 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f579000e9a0 tx=0x7f579000ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.189+0000 7f579f7fe700 1 -- 172.21.15.31:0/1736685324 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579000cbd0 con 0x7f57a8103780 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.190+0000 7f579f7fe700 1 -- 172.21.15.31:0/1736685324 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f579000cd30 con 0x7f57a8103780 2023-07-29T16:39:15.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.190+0000 7f579f7fe700 1 -- 172.21.15.31:0/1736685324 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5790018710 con 0x7f57a8103780 2023-07-29T16:39:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.190+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a8198180 con 0x7f57a8103780 2023-07-29T16:39:15.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.191+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a806a950 con 0x7f57a8103780 2023-07-29T16:39:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.196+0000 7f579f7fe700 1 -- 172.21.15.31:0/1736685324 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5790010980 con 0x7f57a8103780 2023-07-29T16:39:15.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.197+0000 7f57c084fb80 1 Processor -- start 2023-07-29T16:39:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 -- start start 2023-07-29T16:39:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b789380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b7898c0 0x559c6b77f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b77ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b6d79b0 con 0x559c6b77fa90 2023-07-29T16:39:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b6d5370 con 0x559c6b7898c0 2023-07-29T16:39:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.198+0000 7f57c084fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b6d0c60 con 0x559c6b5db6b0 2023-07-29T16:39:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.199+0000 7f57a6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b789380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.199+0000 7f57a6d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b789380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47052/0 (socket says 172.21.15.31:47052) 2023-07-29T16:39:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.199+0000 7f57a6d9d700 1 -- 172.21.15.31:0/1272229711 learned_addr learned my addr 172.21.15.31:0/1272229711 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.199+0000 7f579ffff700 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b77ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.199+0000 7f57a659c700 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b7898c0 0x559c6b77f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f57a6d9d700 1 -- 172.21.15.31:0/1272229711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b7898c0 msgr2=0x559c6b77f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f57a6d9d700 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b7898c0 0x559c6b77f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f57a6d9d700 1 -- 172.21.15.31:0/1272229711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 msgr2=0x559c6b77ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f57a6d9d700 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b77ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f57a6d9d700 1 -- 172.21.15.31:0/1272229711 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6b4fed80 con 0x559c6b5db6b0 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f579ffff700 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b77ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f57a6d9d700 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b789380 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f57a0008de0 tx=0x7f57a000d3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=214f12f3978eba5a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.200+0000 7f579f7fe700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57940a0810 0x7f57940a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.201+0000 7f579cff9700 1 -- 172.21.15.31:0/1272229711 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a000dd70 con 0x559c6b5db6b0 2023-07-29T16:39:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.201+0000 7f579f7fe700 1 -- 172.21.15.31:0/1736685324 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5790014070 con 0x7f57a8103780 2023-07-29T16:39:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.201+0000 7f579cff9700 1 -- 172.21.15.31:0/1272229711 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57a0008a20 con 0x559c6b5db6b0 2023-07-29T16:39:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.201+0000 7f579cff9700 1 -- 172.21.15.31:0/1272229711 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57a0012740 con 0x559c6b5db6b0 2023-07-29T16:39:15.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.201+0000 7f57a659c700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57940a0810 0x7f57940a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 -- 172.21.15.31:0/1272229711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 msgr2=0x559c6b789380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b789380 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f57a0008de0 tx=0x7f57a000d3c0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 -- 172.21.15.31:0/1272229711 shutdown_connections 2023-07-29T16:39:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b77ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b789380 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 --2- 172.21.15.31:0/1272229711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b7898c0 0x559c6b77f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 -- 172.21.15.31:0/1272229711 >> 172.21.15.31:0/1272229711 conn(0x559c6b774fd0 msgr2=0x559c6b77a310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 -- 172.21.15.31:0/1272229711 shutdown_connections 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.202+0000 7f57c084fb80 1 -- 172.21.15.31:0/1272229711 wait complete. 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.203+0000 7f57c084fb80 1 -- start start 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.203+0000 7f57c084fb80 1 Processor -- start 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b791990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b791ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 0x559c6b798d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b66d4b0 con 0x559c6b7898c0 2023-07-29T16:39:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b66d150 con 0x559c6b77fa90 2023-07-29T16:39:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b5548c0 con 0x559c6b5db6b0 2023-07-29T16:39:15.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57c084fb80 10 client.0 _ll_register_callbacks cb 0x559c6b6765c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f579ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b791ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57a659c700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57940a0810 0x7f57940a2cc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5798006010 tx=0x7f579800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f579ffff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b791ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43520/0 (socket says 172.21.15.31:43520) 2023-07-29T16:39:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f579ffff700 1 -- 172.21.15.31:0/2378927621 learned_addr learned my addr 172.21.15.31:0/2378927621 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.204+0000 7f57a659c700 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b791990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f579ffff700 1 -- 172.21.15.31:0/2378927621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 msgr2=0x559c6b791990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f579ffff700 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b791990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f579ffff700 1 -- 172.21.15.31:0/2378927621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 msgr2=0x559c6b798d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f579ffff700 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 0x559c6b798d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f579ffff700 1 -- 172.21.15.31:0/2378927621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a0005e70 con 0x559c6b77fa90 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f579ffff700 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b791ed0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f5790099490 tx=0x7f5790099220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d5032fb8353a7f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f578dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:39:15.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f578dffb700 1 -- 172.21.15.31:0/2378927621 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57900a1950 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f578dffb700 1 -- 172.21.15.31:0/2378927621 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5790099790 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c6b74ed10 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.205+0000 7f578dffb700 1 -- 172.21.15.31:0/2378927621 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f579009d7c0 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.206+0000 7f57c084fb80 10 client.30593 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.206+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x559c6b4e7c00 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.206+0000 7f578dffb700 1 -- 172.21.15.31:0/2378927621 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559c6b74ed10 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.206+0000 7f578dffb700 1 -- 172.21.15.31:0/2378927621 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5790098810 con 0x559c6b77fa90 2023-07-29T16:39:15.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57c084fb80 10 client.30593 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57c084fb80 10 client.30593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57c084fb80 20 client.30593 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57c084fb80 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b79d6e0 0x559c6b79faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57c084fb80 4 client.30593 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c6b6c9ce0 con 0x559c6b79d6e0 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57a659c700 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b79d6e0 0x559c6b79faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f578dffb700 10 client.30593 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:15.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.207+0000 7f57a659c700 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b79d6e0 0x559c6b79faf0 crc :-1 s=READY pgs=497 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f967505a75ff5b9f server_cookie=c7562622b46e0a6e in_seq=0 out_seq=0 2023-07-29T16:39:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.210+0000 7f578dffb700 1 -- 172.21.15.31:0/2378927621 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x559c6b6c9ce0 con 0x559c6b79d6e0 2023-07-29T16:39:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.210+0000 7f578dffb700 10 client.30593 handle_command_reply: tid=0 2023-07-29T16:39:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 client.30593 shutdown 2023-07-29T16:39:15.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 msgr2=0x559c6b791ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b791ed0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f5790099490 tx=0x7f5790099220 comp rx=0 tx=0).stop 2023-07-29T16:39:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 shutdown_connections 2023-07-29T16:39:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b79d6e0 0x559c6b79faf0 crc :-1 s=READY pgs=497 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 0x559c6b798d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b791990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 --2- 172.21.15.31:0/2378927621 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b791ed0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 >> 172.21.15.31:0/2378927621 conn(0x559c6b774fd0 msgr2=0x559c6b779a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f578dffb700 0 client.30593 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 shutdown_connections 2023-07-29T16:39:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.211+0000 7f57c084fb80 1 -- 172.21.15.31:0/2378927621 wait complete. 2023-07-29T16:39:15.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.212+0000 7f57c084fb80 20 client.30593 trim_cache size 0 max 16384 2023-07-29T16:39:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 Processor -- start 2023-07-29T16:39:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 -- start start 2023-07-29T16:39:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 0x559c6b77f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b798cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b7898c0 0x559c6b78c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b791190 con 0x559c6b77fa90 2023-07-29T16:39:15.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b7902a0 con 0x559c6b5db6b0 2023-07-29T16:39:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57c084fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b790c30 con 0x559c6b7898c0 2023-07-29T16:39:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57a659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 0x559c6b77f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57a659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 0x559c6b77f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:43534/0 (socket says 172.21.15.31:43534) 2023-07-29T16:39:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57a659c700 1 -- 172.21.15.31:0/596040382 learned_addr learned my addr 172.21.15.31:0/596040382 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f57a6d9d700 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b7898c0 0x559c6b78c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.236+0000 7f579ffff700 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b798cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f57a659c700 1 -- 172.21.15.31:0/596040382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b7898c0 msgr2=0x559c6b78c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f57a659c700 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b7898c0 0x559c6b78c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f57a659c700 1 -- 172.21.15.31:0/596040382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 msgr2=0x559c6b798cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f57a659c700 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b798cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f57a659c700 1 -- 172.21.15.31:0/596040382 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c6b5251f0 con 0x559c6b5db6b0 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f579ffff700 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b798cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.237+0000 7f57a659c700 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 0x559c6b77f550 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f57980093d0 tx=0x7f5798020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec047faf86a77b76 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:15.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.238+0000 7f579d7fa700 1 -- 172.21.15.31:0/596040382 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579800b9b0 con 0x559c6b5db6b0 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.238+0000 7f579d7fa700 1 -- 172.21.15.31:0/596040382 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f579801e040 con 0x559c6b5db6b0 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.238+0000 7f579d7fa700 1 -- 172.21.15.31:0/596040382 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798003c40 con 0x559c6b5db6b0 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.238+0000 7f57c084fb80 1 -- 172.21.15.31:0/596040382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 msgr2=0x559c6b77f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.238+0000 7f57c084fb80 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 0x559c6b77f550 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f57980093d0 tx=0x7f5798020000 comp rx=0 tx=0).stop 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 -- 172.21.15.31:0/596040382 shutdown_connections 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b7898c0 0x559c6b78c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b5db6b0 0x559c6b77f550 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 --2- 172.21.15.31:0/596040382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b77fa90 0x559c6b798cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 -- 172.21.15.31:0/596040382 >> 172.21.15.31:0/596040382 conn(0x559c6b774fd0 msgr2=0x559c6b7997d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 -- 172.21.15.31:0/596040382 shutdown_connections 2023-07-29T16:39:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 -- 172.21.15.31:0/596040382 wait complete. 2023-07-29T16:39:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.239+0000 7f57c084fb80 1 -- start start 2023-07-29T16:39:15.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 Processor -- start 2023-07-29T16:39:15.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b793540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b798a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 0x559c6b793a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b794060 con 0x559c6b7898c0 2023-07-29T16:39:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b782950 con 0x559c6b77fa90 2023-07-29T16:39:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c6b782b10 con 0x559c6b5db6b0 2023-07-29T16:39:15.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57c084fb80 10 client.0 _ll_register_callbacks cb 0x559c6b6765c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57a659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b793540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57a659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b793540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:47084/0 (socket says 172.21.15.31:47084) 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.240+0000 7f57a659c700 1 -- 172.21.15.31:0/3373057165 learned_addr learned my addr 172.21.15.31:0/3373057165 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.241+0000 7f579ffff700 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b798a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.241+0000 7f579ffff700 1 -- 172.21.15.31:0/3373057165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 msgr2=0x559c6b793540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.241+0000 7f579ffff700 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b793540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.241+0000 7f579ffff700 1 -- 172.21.15.31:0/3373057165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 msgr2=0x559c6b793a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.241+0000 7f579ffff700 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 0x559c6b793a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.241+0000 7f579ffff700 1 -- 172.21.15.31:0/3373057165 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c6b4e7c00 con 0x559c6b77fa90 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.242+0000 7f579ffff700 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b798a80 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f5790000c00 tx=0x7f57900a1ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc2bddea1740d95 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.242+0000 7f578dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.242+0000 7f578dffb700 1 -- 172.21.15.31:0/3373057165 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579009d7c0 con 0x559c6b77fa90 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.242+0000 7f578dffb700 1 -- 172.21.15.31:0/3373057165 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f57900a1be0 con 0x559c6b77fa90 2023-07-29T16:39:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.243+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c6b6fbcb0 con 0x559c6b77fa90 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.243+0000 7f578dffb700 1 -- 172.21.15.31:0/3373057165 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f579009d2e0 con 0x559c6b77fa90 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.243+0000 7f57c084fb80 10 client.30599 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.243+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x7f57a81090a0 con 0x559c6b77fa90 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.243+0000 7f578dffb700 1 -- 172.21.15.31:0/3373057165 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559c6b6fbcb0 con 0x559c6b77fa90 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f578dffb700 1 -- 172.21.15.31:0/3373057165 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f57900cbd70 con 0x559c6b77fa90 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57c084fb80 10 client.30599 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57c084fb80 10 client.30599 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57c084fb80 20 client.30599 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57c084fb80 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b783f10 0x559c6b786320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57c084fb80 4 client.30599 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:39:15.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559c6b6c9a00 con 0x559c6b783f10 2023-07-29T16:39:15.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.244+0000 7f57a659c700 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b783f10 0x559c6b786320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:15.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.245+0000 7f57a659c700 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b783f10 0x559c6b786320 crc :-1 s=READY pgs=498 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b256e9558688f756 server_cookie=8684086df27d0f52 in_seq=0 out_seq=0 2023-07-29T16:39:15.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.245+0000 7f578dffb700 10 client.30599 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:15.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:14 smithi039 ceph-mon[128858]: pgmap v1085: 97 pgs: 97 active+clean; 673 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 17 MiB/s wr, 898 op/s 2023-07-29T16:39:15.379 INFO:tasks.workunit.client.0.smithi031.stdout: 10 339970 33.70 MB/sec execute 419 sec latency 106.140 ms 2023-07-29T16:39:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f578dffb700 1 -- 172.21.15.31:0/3373057165 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559c6b6c9a00 con 0x559c6b783f10 2023-07-29T16:39:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f578dffb700 10 client.30599 handle_command_reply: tid=0 2023-07-29T16:39:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 client.30599 shutdown 2023-07-29T16:39:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 msgr2=0x559c6b798a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b798a80 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f5790000c00 tx=0x7f57900a1ab0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 shutdown_connections 2023-07-29T16:39:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x559c6b783f10 0x559c6b786320 crc :-1 s=READY pgs=498 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559c6b7898c0 0x559c6b793a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559c6b5db6b0 0x559c6b793540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.549+0000 7f57c084fb80 1 --2- 172.21.15.31:0/3373057165 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c6b77fa90 0x559c6b798a80 secure :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f5790000c00 tx=0x7f57900a1ab0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.550+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 >> 172.21.15.31:0/3373057165 conn(0x559c6b774fd0 msgr2=0x559c6b78c540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.550+0000 7f578dffb700 0 client.30599 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.552+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 shutdown_connections 2023-07-29T16:39:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.552+0000 7f57c084fb80 1 -- 172.21.15.31:0/3373057165 wait complete. 2023-07-29T16:39:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.552+0000 7f57c084fb80 20 client.30599 trim_cache size 0 max 16384 2023-07-29T16:39:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:39:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:39:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:39:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:39:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.553+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57940a0810 msgr2=0x7f57940a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.553+0000 7f57acfff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57940a0810 0x7f57940a2cc0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5798006010 tx=0x7f579800b260 comp rx=0 tx=0).stop 2023-07-29T16:39:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.553+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 msgr2=0x7f57a8196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.553+0000 7f57acfff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 0x7f57a8196690 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f579000e9a0 tx=0x7f579000ed60 comp rx=0 tx=0).stop 2023-07-29T16:39:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 shutdown_connections 2023-07-29T16:39:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f57940a0810 0x7f57940a2cc0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57a81060a0 0x7f57a8197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57a8100e60 0x7f57a8196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 --2- 172.21.15.31:0/1736685324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57a8103780 0x7f57a8196690 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 >> 172.21.15.31:0/1736685324 conn(0x7f57a80fa840 msgr2=0x7f57a8101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.554+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 shutdown_connections 2023-07-29T16:39:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:15.557+0000 7f57acfff700 1 -- 172.21.15.31:0/1736685324 wait complete. 2023-07-29T16:39:16.379 INFO:tasks.workunit.client.0.smithi031.stdout: 10 340550 33.70 MB/sec execute 420 sec latency 192.966 ms 2023-07-29T16:39:17.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:16 smithi031 ceph-mon[95504]: pgmap v1086: 97 pgs: 97 active+clean; 677 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 680 op/s 2023-07-29T16:39:17.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:16 smithi039 ceph-mon[128858]: pgmap v1086: 97 pgs: 97 active+clean; 677 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 680 op/s 2023-07-29T16:39:17.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:16 smithi156 ceph-mon[129245]: pgmap v1086: 97 pgs: 97 active+clean; 677 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 680 op/s 2023-07-29T16:39:17.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 341141 33.70 MB/sec execute 421 sec latency 78.648 ms 2023-07-29T16:39:18.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 341756 33.70 MB/sec execute 422 sec latency 97.510 ms 2023-07-29T16:39:18.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:18 smithi039 ceph-mon[128858]: pgmap v1087: 97 pgs: 97 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 14 MiB/s wr, 789 op/s 2023-07-29T16:39:18.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:18 smithi156 ceph-mon[129245]: pgmap v1087: 97 pgs: 97 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 14 MiB/s wr, 789 op/s 2023-07-29T16:39:18.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:18 smithi031 ceph-mon[95504]: pgmap v1087: 97 pgs: 97 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 14 MiB/s wr, 789 op/s 2023-07-29T16:39:19.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 342284 33.67 MB/sec execute 423 sec latency 90.935 ms 2023-07-29T16:39:20.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 342946 33.67 MB/sec execute 424 sec latency 64.575 ms 2023-07-29T16:39:21.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:20 smithi156 ceph-mon[129245]: pgmap v1088: 97 pgs: 97 active+clean; 667 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.4 MiB/s wr, 624 op/s 2023-07-29T16:39:21.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:20 smithi031 ceph-mon[95504]: pgmap v1088: 97 pgs: 97 active+clean; 667 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.4 MiB/s wr, 624 op/s 2023-07-29T16:39:21.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:20 smithi039 ceph-mon[128858]: pgmap v1088: 97 pgs: 97 active+clean; 667 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 7.4 MiB/s wr, 624 op/s 2023-07-29T16:39:21.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 343732 33.68 MB/sec execute 425 sec latency 65.549 ms 2023-07-29T16:39:22.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 344590 33.73 MB/sec execute 426 sec latency 36.747 ms 2023-07-29T16:39:23.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:22 smithi156 ceph-mon[129245]: pgmap v1089: 97 pgs: 97 active+clean; 662 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.0 MiB/s wr, 518 op/s 2023-07-29T16:39:23.174 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:22 smithi031 ceph-mon[95504]: pgmap v1089: 97 pgs: 97 active+clean; 662 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.0 MiB/s wr, 518 op/s 2023-07-29T16:39:23.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:22 smithi039 ceph-mon[128858]: pgmap v1089: 97 pgs: 97 active+clean; 662 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 6.0 MiB/s wr, 518 op/s 2023-07-29T16:39:23.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 345135 33.74 MB/sec execute 427 sec latency 88.648 ms 2023-07-29T16:39:24.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 345603 33.70 MB/sec execute 428 sec latency 69.172 ms 2023-07-29T16:39:25.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:24 smithi031 ceph-mon[95504]: pgmap v1090: 97 pgs: 97 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.6 MiB/s wr, 552 op/s 2023-07-29T16:39:25.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:24 smithi039 ceph-mon[128858]: pgmap v1090: 97 pgs: 97 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.6 MiB/s wr, 552 op/s 2023-07-29T16:39:25.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:24 smithi156 ceph-mon[129245]: pgmap v1090: 97 pgs: 97 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.6 MiB/s wr, 552 op/s 2023-07-29T16:39:25.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 346117 33.68 MB/sec execute 429 sec latency 95.246 ms 2023-07-29T16:39:26.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 346703 33.67 MB/sec execute 430 sec latency 90.269 ms 2023-07-29T16:39:27.143 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:26 smithi031 ceph-mon[95504]: pgmap v1091: 97 pgs: 97 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.9 MiB/s wr, 533 op/s 2023-07-29T16:39:27.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:26 smithi039 ceph-mon[128858]: pgmap v1091: 97 pgs: 97 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.9 MiB/s wr, 533 op/s 2023-07-29T16:39:27.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:26 smithi156 ceph-mon[129245]: pgmap v1091: 97 pgs: 97 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.9 MiB/s wr, 533 op/s 2023-07-29T16:39:27.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 347431 33.69 MB/sec execute 431 sec latency 62.936 ms 2023-07-29T16:39:28.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:27 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:39:28.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:27 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:39:28.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:27 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:39:28.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 348098 33.69 MB/sec execute 432 sec latency 59.912 ms 2023-07-29T16:39:28.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 -- 172.21.15.31:0/1781936477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 msgr2=0x7f42c01007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:28.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 --2- 172.21.15.31:0/1781936477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01007f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009990 tx=0x7f42b0009ca0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 -- 172.21.15.31:0/1781936477 shutdown_connections 2023-07-29T16:39:28.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 --2- 172.21.15.31:0/1781936477 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c0101f60 0x7f42c0108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 --2- 172.21.15.31:0/1781936477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01007f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 --2- 172.21.15.31:0/1781936477 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c01015d0 0x7f42c0101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 -- 172.21.15.31:0/1781936477 >> 172.21.15.31:0/1781936477 conn(0x7f42c00fba00 msgr2=0x7f42c00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:28.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 -- 172.21.15.31:0/1781936477 shutdown_connections 2023-07-29T16:39:28.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.770+0000 7f42c4fff700 1 -- 172.21.15.31:0/1781936477 wait complete. 2023-07-29T16:39:28.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 Processor -- start 2023-07-29T16:39:28.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 -- start start 2023-07-29T16:39:28.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c01015d0 0x7f42c01966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c0101f60 0x7f42c0197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0197bc0 con 0x7f42c0101f60 2023-07-29T16:39:28.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0197d30 con 0x7f42c01015d0 2023-07-29T16:39:28.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.772+0000 7f42c4fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0197ef0 con 0x7f42c01003e0 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c01015d0 0x7f42c01966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45228/0 (socket says 172.21.15.31:45228) 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42be59c700 1 -- 172.21.15.31:0/3318785999 learned_addr learned my addr 172.21.15.31:0/3318785999 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c0101f60 0x7f42c0197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 -- 172.21.15.31:0/3318785999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 msgr2=0x7f42c01961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 -- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c0101f60 msgr2=0x7f42c0197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c0101f60 0x7f42c0197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 -- 172.21.15.31:0/3318785999 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b00095f0 con 0x7f42c01015d0 2023-07-29T16:39:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.773+0000 7f42bdd9b700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c01015d0 0x7f42c01966e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f42b400e9f0 tx=0x7f42b400edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42be59c700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:39:28.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c0101f60 0x7f42c0197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:39:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42af7fe700 1 -- 172.21.15.31:0/3318785999 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400cbd0 con 0x7f42c01015d0 2023-07-29T16:39:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42af7fe700 1 -- 172.21.15.31:0/3318785999 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42b400cd30 con 0x7f42c01015d0 2023-07-29T16:39:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42af7fe700 1 -- 172.21.15.31:0/3318785999 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4018710 con 0x7f42c01015d0 2023-07-29T16:39:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c01981d0 con 0x7f42c01015d0 2023-07-29T16:39:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.774+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c019e990 con 0x7f42c01015d0 2023-07-29T16:39:28.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.780+0000 7f42af7fe700 1 -- 172.21.15.31:0/3318785999 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f42b4018870 con 0x7f42c01015d0 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.780+0000 7f42d8906b80 1 Processor -- start 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 -- start start 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dcc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dcd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773dce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773d249b0 con 0x557773dcdcf0 2023-07-29T16:39:28.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773d22370 con 0x557773dccda0 2023-07-29T16:39:28.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.781+0000 7f42d8906b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773d1dc60 con 0x557773c286b0 2023-07-29T16:39:28.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dcc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dcc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45242/0 (socket says 172.21.15.31:45242) 2023-07-29T16:39:28.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 -- 172.21.15.31:0/2376815657 learned_addr learned my addr 172.21.15.31:0/2376815657 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:28.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42be59c700 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dcd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42af7fe700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42a80a0790 0x7f42a80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 -- 172.21.15.31:0/2376815657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 msgr2=0x557773dcd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dcd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 -- 172.21.15.31:0/2376815657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 msgr2=0x557773dce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773dce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42bed9d700 1 -- 172.21.15.31:0/2376815657 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557773b4bd80 con 0x557773c286b0 2023-07-29T16:39:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42be59c700 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dcd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:39:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.782+0000 7f42af7fe700 1 -- 172.21.15.31:0/3318785999 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f42b4014070 con 0x7f42c01015d0 2023-07-29T16:39:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.783+0000 7f42bed9d700 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dcc860 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009d50 tx=0x7f42b80077e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=311b8e34cbbceffe server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.783+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42a80a0790 0x7f42a80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.783+0000 7f42acff9700 1 -- 172.21.15.31:0/2376815657 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8019070 con 0x557773c286b0 2023-07-29T16:39:28.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.783+0000 7f42acff9700 1 -- 172.21.15.31:0/2376815657 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42b8014960 con 0x557773c286b0 2023-07-29T16:39:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.783+0000 7f42acff9700 1 -- 172.21.15.31:0/2376815657 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8013b60 con 0x557773c286b0 2023-07-29T16:39:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.784+0000 7f42d8906b80 1 -- 172.21.15.31:0/2376815657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 msgr2=0x557773dcc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.784+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42a80a0790 0x7f42a80a2c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f42b8013fd0 tx=0x7f42b8013da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:28.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.784+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dcc860 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f42b8009d50 tx=0x7f42b80077e0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 -- 172.21.15.31:0/2376815657 shutdown_connections 2023-07-29T16:39:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773dce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dcc860 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2376815657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dcd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 -- 172.21.15.31:0/2376815657 >> 172.21.15.31:0/2376815657 conn(0x557773dc1fd0 msgr2=0x557773dc7300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 -- 172.21.15.31:0/2376815657 shutdown_connections 2023-07-29T16:39:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.785+0000 7f42d8906b80 1 -- 172.21.15.31:0/2376815657 wait complete. 2023-07-29T16:39:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.786+0000 7f42d8906b80 1 -- start start 2023-07-29T16:39:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.786+0000 7f42d8906b80 1 Processor -- start 2023-07-29T16:39:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773c286b0 0x557773dde800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dded40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 0x557773de5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773cba4b0 con 0x557773c286b0 2023-07-29T16:39:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773cba150 con 0x557773dccda0 2023-07-29T16:39:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773ba18c0 con 0x557773dcdcf0 2023-07-29T16:39:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42d8906b80 10 client.0 _ll_register_callbacks cb 0x557773cc35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42bed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 0x557773de5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42bed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 0x557773de5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45254/0 (socket says 172.21.15.31:45254) 2023-07-29T16:39:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.787+0000 7f42bed9d700 1 -- 172.21.15.31:0/3805454089 learned_addr learned my addr 172.21.15.31:0/3805454089 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.788+0000 7f42be59c700 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773c286b0 0x557773dde800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.788+0000 7f42bed9d700 1 -- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 msgr2=0x557773dded40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:39:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.788+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dded40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.788+0000 7f42bed9d700 1 -- 172.21.15.31:0/3805454089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773c286b0 msgr2=0x557773dde800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.788+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773c286b0 0x557773dde800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.788+0000 7f42bed9d700 1 -- 172.21.15.31:0/3805454089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b8009980 con 0x557773dcdcf0 2023-07-29T16:39:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.789+0000 7f42bed9d700 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 0x557773de5d80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f42b8014000 tx=0x7f42b8015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ca5be831867c225 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.789+0000 7f42a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:39:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.789+0000 7f42a5ffb700 1 -- 172.21.15.31:0/3805454089 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b8019040 con 0x557773dcdcf0 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.789+0000 7f42a5ffb700 1 -- 172.21.15.31:0/3805454089 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f42b80090b0 con 0x557773dcdcf0 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.789+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557773d48cb0 con 0x557773dcdcf0 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.790+0000 7f42a5ffb700 1 -- 172.21.15.31:0/3805454089 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42b8014ad0 con 0x557773dcdcf0 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.790+0000 7f42d8906b80 10 client.30637 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.790+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x557773b34c00 con 0x557773dcdcf0 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.790+0000 7f42a5ffb700 1 -- 172.21.15.31:0/3805454089 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557773d48cb0 con 0x557773dcdcf0 2023-07-29T16:39:28.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42a5ffb700 1 -- 172.21.15.31:0/3805454089 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f42b8004d00 con 0x557773dcdcf0 2023-07-29T16:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42d8906b80 10 client.30637 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42d8906b80 10 client.30637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42d8906b80 20 client.30637 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42d8906b80 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773dd4b70 0x557773dd6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42d8906b80 4 client.30637 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:39:28.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.791+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557773d9bd10 con 0x557773dd4b70 2023-07-29T16:39:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.799+0000 7f42be59c700 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773dd4b70 0x557773dd6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.808+0000 7f42be59c700 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773dd4b70 0x557773dd6f80 crc :-1 s=READY pgs=499 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72a773c454a6c227 server_cookie=3a5df6f37488bc0e in_seq=0 out_seq=0 2023-07-29T16:39:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.808+0000 7f42a5ffb700 10 client.30637 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.809+0000 7f42a5ffb700 1 -- 172.21.15.31:0/3805454089 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557773d9bd10 con 0x557773dd4b70 2023-07-29T16:39:28.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.809+0000 7f42a5ffb700 10 client.30637 handle_command_reply: tid=0 2023-07-29T16:39:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 client.30637 shutdown 2023-07-29T16:39:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 msgr2=0x557773de5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 0x557773de5d80 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f42b8014000 tx=0x7f42b8015000 comp rx=0 tx=0).stop 2023-07-29T16:39:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 shutdown_connections 2023-07-29T16:39:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773dd4b70 0x557773dd6f80 crc :-1 s=READY pgs=499 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dcdcf0 0x557773de5d80 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773c286b0 0x557773dde800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42a5ffb700 0 client.30637 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 --2- 172.21.15.31:0/3805454089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dccda0 0x557773dded40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 >> 172.21.15.31:0/3805454089 conn(0x557773dc1fd0 msgr2=0x557773dc6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.810+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 shutdown_connections 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.811+0000 7f42d8906b80 1 -- 172.21.15.31:0/3805454089 wait complete. 2023-07-29T16:39:28.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.811+0000 7f42d8906b80 20 client.30637 trim_cache size 0 max 16384 2023-07-29T16:39:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 Processor -- start 2023-07-29T16:39:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 -- start start 2023-07-29T16:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dc83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dccda0 0x557773de5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dcdcf0 0x557773dddeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773d48cb0 con 0x557773dccda0 2023-07-29T16:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773dc7df0 con 0x557773dcdcf0 2023-07-29T16:39:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42d8906b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773dddce0 con 0x557773c286b0 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dc83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42bed9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dcdcf0 0x557773dddeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42be59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dc83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45264/0 (socket says 172.21.15.31:45264) 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42bdd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dccda0 0x557773de5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.832+0000 7f42be59c700 1 -- 172.21.15.31:0/2136381947 learned_addr learned my addr 172.21.15.31:0/2136381947 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42be59c700 1 -- 172.21.15.31:0/2136381947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dcdcf0 msgr2=0x557773dddeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42be59c700 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dcdcf0 0x557773dddeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42be59c700 1 -- 172.21.15.31:0/2136381947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dccda0 msgr2=0x557773de5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42be59c700 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dccda0 0x557773de5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42be59c700 1 -- 172.21.15.31:0/2136381947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557773b721f0 con 0x557773c286b0 2023-07-29T16:39:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42be59c700 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dc83a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009350 tx=0x7f42b0009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2eae95a6d91140 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42ad7fa700 1 -- 172.21.15.31:0/2136381947 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002b030 con 0x557773c286b0 2023-07-29T16:39:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42ad7fa700 1 -- 172.21.15.31:0/2136381947 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42b000bbb0 con 0x557773c286b0 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.833+0000 7f42ad7fa700 1 -- 172.21.15.31:0/2136381947 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0005040 con 0x557773c286b0 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 -- 172.21.15.31:0/2136381947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 msgr2=0x557773dc83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dc83a0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009350 tx=0x7f42b0009530 comp rx=0 tx=0).stop 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 -- 172.21.15.31:0/2136381947 shutdown_connections 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773dcdcf0 0x557773dddeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773c286b0 0x557773dc83a0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 --2- 172.21.15.31:0/2136381947 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dccda0 0x557773de5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.834+0000 7f42d8906b80 1 -- 172.21.15.31:0/2136381947 >> 172.21.15.31:0/2136381947 conn(0x557773dc1fd0 msgr2=0x557773dc72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.835+0000 7f42d8906b80 1 -- 172.21.15.31:0/2136381947 shutdown_connections 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.835+0000 7f42d8906b80 1 -- 172.21.15.31:0/2136381947 wait complete. 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.835+0000 7f42d8906b80 1 -- start start 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.835+0000 7f42d8906b80 1 Processor -- start 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773c286b0 0x557773ddab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 0x557773de5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773ddb060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773ddb6b0 con 0x557773dcdcf0 2023-07-29T16:39:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773ddb820 con 0x557773c286b0 2023-07-29T16:39:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557773ddb9e0 con 0x557773dccda0 2023-07-29T16:39:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42d8906b80 10 client.0 _ll_register_callbacks cb 0x557773cc35c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42bdd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 0x557773de5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42bdd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 0x557773de5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:45270/0 (socket says 172.21.15.31:45270) 2023-07-29T16:39:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42bdd9b700 1 -- 172.21.15.31:0/129939016 learned_addr learned my addr 172.21.15.31:0/129939016 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.836+0000 7f42bed9d700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773ddb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42be59c700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773c286b0 0x557773ddab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42bdd9b700 1 -- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773c286b0 msgr2=0x557773ddab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42bdd9b700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773c286b0 0x557773ddab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42bdd9b700 1 -- 172.21.15.31:0/129939016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 msgr2=0x557773ddb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42bdd9b700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773ddb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42bdd9b700 1 -- 172.21.15.31:0/129939016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b0020040 con 0x557773dccda0 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.837+0000 7f42bed9d700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773ddb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.838+0000 7f42bdd9b700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 0x557773de5cf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f42b400c5b0 tx=0x7f42b4099ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e9da25443a3f40c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.838+0000 7f42a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:39:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.838+0000 7f42a5ffb700 1 -- 172.21.15.31:0/129939016 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b4098f00 con 0x557773dccda0 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.838+0000 7f42a5ffb700 1 -- 172.21.15.31:0/129939016 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f42b40995a0 con 0x557773dccda0 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.838+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42b8014960 con 0x557773dccda0 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.840+0000 7f42a5ffb700 1 -- 172.21.15.31:0/129939016 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42b409d5d0 con 0x557773dccda0 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 10 client.30643 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x557773b4a250 con 0x557773dccda0 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42a5ffb700 1 -- 172.21.15.31:0/129939016 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42b8014960 con 0x557773dccda0 2023-07-29T16:39:28.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42a5ffb700 1 -- 172.21.15.31:0/129939016 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f42b4095390 con 0x557773dccda0 2023-07-29T16:39:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 10 client.30643 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 10 client.30643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 20 client.30643 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:28.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773e45960 0x557773e47d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 4 client.30643 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:39:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.841+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557773d16ce0 con 0x557773e45960 2023-07-29T16:39:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.842+0000 7f42be59c700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773e45960 0x557773e47d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.842+0000 7f42a5ffb700 10 client.30643 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:28.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:28.842+0000 7f42be59c700 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773e45960 0x557773e47d70 crc :-1 s=READY pgs=500 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fca308747491895 server_cookie=6525467e4fda8a6a in_seq=0 out_seq=0 2023-07-29T16:39:29.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:28 smithi031 ceph-mon[95504]: pgmap v1092: 97 pgs: 97 active+clean; 676 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 574 op/s 2023-07-29T16:39:29.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:28 smithi039 ceph-mon[128858]: pgmap v1092: 97 pgs: 97 active+clean; 676 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 574 op/s 2023-07-29T16:39:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.307+0000 7f42a5ffb700 1 -- 172.21.15.31:0/129939016 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557773d16ce0 con 0x557773e45960 2023-07-29T16:39:29.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.307+0000 7f42a5ffb700 10 client.30643 handle_command_reply: tid=0 2023-07-29T16:39:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.307+0000 7f42d8906b80 1 client.30643 shutdown 2023-07-29T16:39:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.307+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 msgr2=0x557773de5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.307+0000 7f42d8906b80 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 0x557773de5cf0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f42b400c5b0 tx=0x7f42b4099ae0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 shutdown_connections 2023-07-29T16:39:29.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557773e45960 0x557773e47d70 crc :-1 s=READY pgs=500 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557773dcdcf0 0x557773ddb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557773c286b0 0x557773ddab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 --2- 172.21.15.31:0/129939016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557773dccda0 0x557773de5cf0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42a5ffb700 0 client.30643 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:29.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 >> 172.21.15.31:0/129939016 conn(0x557773dc1fd0 msgr2=0x557773dde7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:29.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 shutdown_connections 2023-07-29T16:39:29.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 1 -- 172.21.15.31:0/129939016 wait complete. 2023-07-29T16:39:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.309+0000 7f42d8906b80 20 client.30643 trim_cache size 0 max 16384 2023-07-29T16:39:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:39:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:39:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:39:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:39:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42a80a0790 msgr2=0x7f42a80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42a80a0790 0x7f42a80a2c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f42b8013fd0 tx=0x7f42b8013da0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c01015d0 msgr2=0x7f42c01966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c01015d0 0x7f42c01966e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f42b400e9f0 tx=0x7f42b400edb0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 shutdown_connections 2023-07-29T16:39:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f42a80a0790 0x7f42a80a2c40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42c0101f60 0x7f42c0197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42c01003e0 0x7f42c01961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.311+0000 7f42c4fff700 1 --2- 172.21.15.31:0/3318785999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f42c01015d0 0x7f42c01966e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.312+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 >> 172.21.15.31:0/3318785999 conn(0x7f42c00fba00 msgr2=0x7f42c00fdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.312+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 shutdown_connections 2023-07-29T16:39:29.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:29.312+0000 7f42c4fff700 1 -- 172.21.15.31:0/3318785999 wait complete. 2023-07-29T16:39:29.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:28 smithi156 ceph-mon[129245]: pgmap v1092: 97 pgs: 97 active+clean; 676 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.6 MiB/s wr, 574 op/s 2023-07-29T16:39:29.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 348613 33.69 MB/sec execute 433 sec latency 359.359 ms 2023-07-29T16:39:30.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:39:30.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:39:30.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:39:30.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:39:30.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:39:30.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:39:30.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:39:30.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:39:30.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:39:30.380 INFO:tasks.workunit.client.0.smithi031.stdout: 10 349292 33.69 MB/sec execute 434 sec latency 126.004 ms 2023-07-29T16:39:31.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:30 smithi031 ceph-mon[95504]: pgmap v1093: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.5 MiB/s wr, 468 op/s 2023-07-29T16:39:31.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:30 smithi039 ceph-mon[128858]: pgmap v1093: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.5 MiB/s wr, 468 op/s 2023-07-29T16:39:31.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:30 smithi156 ceph-mon[129245]: pgmap v1093: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.5 MiB/s wr, 468 op/s 2023-07-29T16:39:31.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 349898 33.70 MB/sec execute 435 sec latency 67.631 ms 2023-07-29T16:39:32.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 350627 33.71 MB/sec execute 436 sec latency 65.828 ms 2023-07-29T16:39:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:32 smithi031 ceph-mon[95504]: pgmap v1094: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.4 MiB/s wr, 446 op/s 2023-07-29T16:39:33.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:32 smithi039 ceph-mon[128858]: pgmap v1094: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.4 MiB/s wr, 446 op/s 2023-07-29T16:39:33.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:32 smithi156 ceph-mon[129245]: pgmap v1094: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.4 MiB/s wr, 446 op/s 2023-07-29T16:39:33.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 351263 33.71 MB/sec execute 437 sec latency 60.626 ms 2023-07-29T16:39:34.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 351833 33.70 MB/sec execute 438 sec latency 91.611 ms 2023-07-29T16:39:35.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:34 smithi031 ceph-mon[95504]: pgmap v1095: 97 pgs: 97 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 598 op/s 2023-07-29T16:39:35.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:34 smithi039 ceph-mon[128858]: pgmap v1095: 97 pgs: 97 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 598 op/s 2023-07-29T16:39:35.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:34 smithi156 ceph-mon[129245]: pgmap v1095: 97 pgs: 97 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.1 MiB/s wr, 598 op/s 2023-07-29T16:39:35.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 352497 33.70 MB/sec execute 439 sec latency 80.866 ms 2023-07-29T16:39:36.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 353242 33.75 MB/sec execute 440 sec latency 82.495 ms 2023-07-29T16:39:37.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:36 smithi031 ceph-mon[95504]: pgmap v1096: 97 pgs: 97 active+clean; 687 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.4 MiB/s wr, 512 op/s 2023-07-29T16:39:37.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:36 smithi039 ceph-mon[128858]: pgmap v1096: 97 pgs: 97 active+clean; 687 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.4 MiB/s wr, 512 op/s 2023-07-29T16:39:37.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:36 smithi156 ceph-mon[129245]: pgmap v1096: 97 pgs: 97 active+clean; 687 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.4 MiB/s wr, 512 op/s 2023-07-29T16:39:37.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 353941 33.77 MB/sec execute 441 sec latency 58.986 ms 2023-07-29T16:39:38.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 354538 33.77 MB/sec execute 442 sec latency 85.052 ms 2023-07-29T16:39:38.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:38 smithi039 ceph-mon[128858]: pgmap v1097: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 547 op/s 2023-07-29T16:39:38.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:38 smithi156 ceph-mon[129245]: pgmap v1097: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 547 op/s 2023-07-29T16:39:38.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:38 smithi031 ceph-mon[95504]: pgmap v1097: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.7 MiB/s wr, 547 op/s 2023-07-29T16:39:39.381 INFO:tasks.workunit.client.0.smithi031.stdout: 10 355130 33.77 MB/sec execute 443 sec latency 52.531 ms 2023-07-29T16:39:40.382 INFO:tasks.workunit.client.0.smithi031.stdout: 10 355751 33.75 MB/sec execute 444 sec latency 58.569 ms 2023-07-29T16:39:41.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:40 smithi156 ceph-mon[129245]: pgmap v1098: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.4 MiB/s wr, 502 op/s 2023-07-29T16:39:41.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:40 smithi031 ceph-mon[95504]: pgmap v1098: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.4 MiB/s wr, 502 op/s 2023-07-29T16:39:41.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:40 smithi039 ceph-mon[128858]: pgmap v1098: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.4 MiB/s wr, 502 op/s 2023-07-29T16:39:41.382 INFO:tasks.workunit.client.0.smithi031.stdout: 10 356337 33.75 MB/sec execute 445 sec latency 137.224 ms 2023-07-29T16:39:42.382 INFO:tasks.workunit.client.0.smithi031.stdout: 10 356964 33.74 MB/sec execute 446 sec latency 70.718 ms 2023-07-29T16:39:43.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:42 smithi031 ceph-mon[95504]: pgmap v1099: 97 pgs: 97 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.3 MiB/s wr, 486 op/s 2023-07-29T16:39:43.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:42 smithi039 ceph-mon[128858]: pgmap v1099: 97 pgs: 97 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.3 MiB/s wr, 486 op/s 2023-07-29T16:39:43.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:42 smithi156 ceph-mon[129245]: pgmap v1099: 97 pgs: 97 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.3 MiB/s wr, 486 op/s 2023-07-29T16:39:43.382 INFO:tasks.workunit.client.0.smithi031.stdout: 10 357536 33.75 MB/sec execute 447 sec latency 316.644 ms 2023-07-29T16:39:44.382 INFO:tasks.workunit.client.0.smithi031.stdout: 10 357952 33.72 MB/sec execute 448 sec latency 342.357 ms 2023-07-29T16:39:45.191 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:44 smithi031 ceph-mon[95504]: pgmap v1100: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.7 MiB/s wr, 506 op/s 2023-07-29T16:39:45.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:44 smithi039 ceph-mon[128858]: pgmap v1100: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.7 MiB/s wr, 506 op/s 2023-07-29T16:39:45.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:44 smithi156 ceph-mon[129245]: pgmap v1100: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.7 MiB/s wr, 506 op/s 2023-07-29T16:39:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.329+0000 7fb5981ff700 1 -- 172.21.15.31:0/802465854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 msgr2=0x7fb5900723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:45.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.329+0000 7fb5981ff700 1 --2- 172.21.15.31:0/802465854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb5900723b0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb5800074e0 tx=0x7fb58000af90 comp rx=0 tx=0).stop 2023-07-29T16:39:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.335+0000 7fb5981ff700 1 -- 172.21.15.31:0/802465854 shutdown_connections 2023-07-29T16:39:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.335+0000 7fb5981ff700 1 --2- 172.21.15.31:0/802465854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5901088f0 0x7fb59010f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.335+0000 7fb5981ff700 1 --2- 172.21.15.31:0/802465854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb5900723b0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.335+0000 7fb5981ff700 1 --2- 172.21.15.31:0/802465854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb590107f60 0x7fb5901083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.335+0000 7fb5981ff700 1 -- 172.21.15.31:0/802465854 >> 172.21.15.31:0/802465854 conn(0x7fb59006d5c0 msgr2=0x7fb59006f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:45.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.336+0000 7fb5981ff700 1 -- 172.21.15.31:0/802465854 shutdown_connections 2023-07-29T16:39:45.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.338+0000 7fb5981ff700 1 -- 172.21.15.31:0/802465854 wait complete. 2023-07-29T16:39:45.352 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.349+0000 7fb5981ff700 1 Processor -- start 2023-07-29T16:39:45.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.349+0000 7fb5981ff700 1 -- start start 2023-07-29T16:39:45.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb5981ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb59019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb5981ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb590107f60 0x7fb59019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb5981ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5901088f0 0x7fb59019ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb5981ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901a0640 con 0x7fb590071fa0 2023-07-29T16:39:45.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb5981ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901a07b0 con 0x7fb5901088f0 2023-07-29T16:39:45.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb5981ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5901a0970 con 0x7fb590107f60 2023-07-29T16:39:45.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.350+0000 7fb595f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb59019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.351+0000 7fb595f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb59019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57252/0 (socket says 172.21.15.31:57252) 2023-07-29T16:39:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.351+0000 7fb595f9b700 1 -- 172.21.15.31:0/2876153431 learned_addr learned my addr 172.21.15.31:0/2876153431 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.352+0000 7fb595f9b700 1 -- 172.21.15.31:0/2876153431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb590107f60 msgr2=0x7fb59019f160 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-29T16:39:45.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.352+0000 7fb595f9b700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb590107f60 0x7fb59019f160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.352+0000 7fb595f9b700 1 -- 172.21.15.31:0/2876153431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5901088f0 msgr2=0x7fb59019ffe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-29T16:39:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.352+0000 7fb595f9b700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5901088f0 0x7fb59019ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.352+0000 7fb595f9b700 1 -- 172.21.15.31:0/2876153431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5800071c0 con 0x7fb590071fa0 2023-07-29T16:39:45.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.353+0000 7fb595f9b700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb59019ec20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb580016040 tx=0x7fb580003960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.354+0000 7fb586ffd700 1 -- 172.21.15.31:0/2876153431 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58000f070 con 0x7fb590071fa0 2023-07-29T16:39:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.354+0000 7fb586ffd700 1 -- 172.21.15.31:0/2876153431 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb58000ccf0 con 0x7fb590071fa0 2023-07-29T16:39:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.354+0000 7fb586ffd700 1 -- 172.21.15.31:0/2876153431 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58000a7a0 con 0x7fb590071fa0 2023-07-29T16:39:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.354+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5901a0bf0 con 0x7fb590071fa0 2023-07-29T16:39:45.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.354+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59019c750 con 0x7fb590071fa0 2023-07-29T16:39:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.361+0000 7fb5abc0db80 1 Processor -- start 2023-07-29T16:39:45.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.361+0000 7fb5abc0db80 1 -- start start 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da13df0 0x557d7da147b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da15160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d96b9b0 con 0x557d7da13df0 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d969370 con 0x557d7da14cf0 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d964c60 con 0x557d7d86f6b0 2023-07-29T16:39:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb59579a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da15160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb59579a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da15160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38414/0 (socket says 172.21.15.31:38414) 2023-07-29T16:39:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.362+0000 7fb59579a700 1 -- 172.21.15.31:0/1304507679 learned_addr learned my addr 172.21.15.31:0/1304507679 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.363+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da13df0 0x557d7da147b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.363+0000 7fb595f9b700 1 -- 172.21.15.31:0/1304507679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 msgr2=0x557d7da138b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:39:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.363+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da138b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.364+0000 7fb595f9b700 1 -- 172.21.15.31:0/1304507679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 msgr2=0x557d7da15160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.364+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da15160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.364+0000 7fb595f9b700 1 -- 172.21.15.31:0/1304507679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d7d792d80 con 0x557d7da13df0 2023-07-29T16:39:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.364+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da13df0 0x557d7da147b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb5800c9b20 tx=0x7fb5800d41e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=608a3699a1e60024 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.365+0000 7fb586ffd700 1 -- 172.21.15.31:0/2876153431 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb580008040 con 0x7fb590071fa0 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.366+0000 7fb57bfff700 1 -- 172.21.15.31:0/1304507679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800ce2e0 con 0x557d7da13df0 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.366+0000 7fb57bfff700 1 -- 172.21.15.31:0/1304507679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5800d4e20 con 0x557d7da13df0 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.366+0000 7fb57bfff700 1 -- 172.21.15.31:0/1304507679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800cce30 con 0x557d7da13df0 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.366+0000 7fb586ffd700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb57c0a91a0 0x7fb57c0ab650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.367+0000 7fb586ffd700 1 -- 172.21.15.31:0/2876153431 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb5800cc140 con 0x7fb590071fa0 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.367+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1304507679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da13df0 msgr2=0x557d7da147b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:45.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.367+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da13df0 0x557d7da147b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb5800c9b20 tx=0x7fb5800d41e0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.367+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1304507679 shutdown_connections 2023-07-29T16:39:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.368+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da15160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.368+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.368+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1304507679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da13df0 0x557d7da147b0 secure :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb5800c9b20 tx=0x7fb5800d41e0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.368+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1304507679 >> 172.21.15.31:0/1304507679 conn(0x557d7da08fd0 msgr2=0x557d7da0e300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:45.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.368+0000 7fb59679c700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb57c0a91a0 0x7fb57c0ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.370+0000 7fb59679c700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb57c0a91a0 0x7fb57c0ab650 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb58800aab0 tx=0x7fb588009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.371+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1304507679 shutdown_connections 2023-07-29T16:39:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.371+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1304507679 wait complete. 2023-07-29T16:39:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.372+0000 7fb5abc0db80 1 -- start start 2023-07-29T16:39:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.372+0000 7fb5abc0db80 1 Processor -- start 2023-07-29T16:39:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7d86f6b0 0x557d7da1d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da14cf0 0x557d7da1d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da24960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d9014b0 con 0x557d7da22580 2023-07-29T16:39:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d901150 con 0x557d7d86f6b0 2023-07-29T16:39:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d7e88c0 con 0x557d7da14cf0 2023-07-29T16:39:45.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.373+0000 7fb5abc0db80 10 client.0 _ll_register_callbacks cb 0x557d7d90a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.376+0000 7fb59579a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da14cf0 0x557d7da1d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.376+0000 7fb595f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7d86f6b0 0x557d7da1d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.376+0000 7fb59679c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da24960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb59679c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da24960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57260/0 (socket says 172.21.15.31:57260) 2023-07-29T16:39:45.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb59679c700 1 -- 172.21.15.31:0/1161008613 learned_addr learned my addr 172.21.15.31:0/1161008613 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb595f9b700 1 -- 172.21.15.31:0/1161008613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da14cf0 msgr2=0x557d7da1d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da14cf0 0x557d7da1d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb595f9b700 1 -- 172.21.15.31:0/1161008613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 msgr2=0x557d7da24960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da24960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.377+0000 7fb595f9b700 1 -- 172.21.15.31:0/1161008613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59019c750 con 0x557d7d86f6b0 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.378+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7d86f6b0 0x557d7da1d010 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb5800cd000 tx=0x7fb5800d45e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb46e9488be2e975 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.379+0000 7fb5797fa700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.379+0000 7fb5797fa700 1 -- 172.21.15.31:0/1161008613 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800cf790 con 0x557d7d86f6b0 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.379+0000 7fb5797fa700 1 -- 172.21.15.31:0/1161008613 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb58009a870 con 0x557d7d86f6b0 2023-07-29T16:39:45.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.379+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d7d98fcb0 con 0x557d7d86f6b0 2023-07-29T16:39:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.380+0000 7fb5797fa700 1 -- 172.21.15.31:0/1161008613 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb580096bb0 con 0x557d7d86f6b0 2023-07-29T16:39:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.380+0000 7fb5abc0db80 10 client.30620 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.380+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x557d7d77bc00 con 0x557d7d86f6b0 2023-07-29T16:39:45.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.380+0000 7fb5797fa700 1 -- 172.21.15.31:0/1161008613 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d7d98fcb0 con 0x557d7d86f6b0 2023-07-29T16:39:45.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5797fa700 1 -- 172.21.15.31:0/1161008613 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb5800cc530 con 0x557d7d86f6b0 2023-07-29T16:39:45.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5abc0db80 10 client.30620 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:45.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5abc0db80 10 client.30620 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:45.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5abc0db80 20 client.30620 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:45.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da39c20 0x557d7da3c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5abc0db80 4 client.30620 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:39:45.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d7d9e2d10 con 0x557d7da39c20 2023-07-29T16:39:45.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.382+0000 7fb59579a700 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da39c20 0x557d7da3c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.387 INFO:tasks.workunit.client.0.smithi031.stdout: 10 357986 33.65 MB/sec execute 449 sec latency 940.176 ms 2023-07-29T16:39:45.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.385+0000 7fb59579a700 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da39c20 0x557d7da3c000 crc :-1 s=READY pgs=501 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8483f4aed45c5969 server_cookie=ea3fb1a6ce55500b in_seq=0 out_seq=0 2023-07-29T16:39:45.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.385+0000 7fb5797fa700 10 client.30620 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.390+0000 7fb5797fa700 1 -- 172.21.15.31:0/1161008613 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x557d7d9e2d10 con 0x557d7da39c20 2023-07-29T16:39:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.390+0000 7fb5797fa700 10 client.30620 handle_command_reply: tid=0 2023-07-29T16:39:45.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.390+0000 7fb5abc0db80 1 client.30620 shutdown 2023-07-29T16:39:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7d86f6b0 msgr2=0x557d7da1d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:45.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7d86f6b0 0x557d7da1d010 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb5800cd000 tx=0x7fb5800d45e0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 shutdown_connections 2023-07-29T16:39:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da39c20 0x557d7da3c000 crc :-1 s=READY pgs=501 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da24960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7d86f6b0 0x557d7da1d010 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1161008613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da14cf0 0x557d7da1d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 >> 172.21.15.31:0/1161008613 conn(0x557d7da08fd0 msgr2=0x557d7da0da40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5797fa700 0 client.30620 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.391+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 shutdown_connections 2023-07-29T16:39:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.392+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1161008613 wait complete. 2023-07-29T16:39:45.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.392+0000 7fb5abc0db80 20 client.30620 trim_cache size 0 max 16384 2023-07-29T16:39:45.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 Processor -- start 2023-07-29T16:39:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 -- start start 2023-07-29T16:39:45.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7d86f6b0 0x557d7da49560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da0da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da22580 0x557d7da36170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7da0edf0 con 0x557d7d86f6b0 2023-07-29T16:39:45.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7da08a20 con 0x557d7da14cf0 2023-07-29T16:39:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.424+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7d9e2d10 con 0x557d7da22580 2023-07-29T16:39:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb59579a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da0da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb59679c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da22580 0x557d7da36170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb59579a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da0da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38418/0 (socket says 172.21.15.31:38418) 2023-07-29T16:39:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb59579a700 1 -- 172.21.15.31:0/1638516743 learned_addr learned my addr 172.21.15.31:0/1638516743 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:45.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7d86f6b0 0x557d7da49560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb595f9b700 1 -- 172.21.15.31:0/1638516743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da22580 msgr2=0x557d7da36170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da22580 0x557d7da36170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb595f9b700 1 -- 172.21.15.31:0/1638516743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 msgr2=0x557d7da0da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da0da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.425+0000 7fb595f9b700 1 -- 172.21.15.31:0/1638516743 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d7d7b91f0 con 0x557d7d86f6b0 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.426+0000 7fb595f9b700 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7d86f6b0 0x557d7da49560 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb58009e040 tx=0x7fb580005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5065a957c30c5481 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.426+0000 7fb584ff9700 1 -- 172.21.15.31:0/1638516743 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb580096bb0 con 0x557d7d86f6b0 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.426+0000 7fb584ff9700 1 -- 172.21.15.31:0/1638516743 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb58009a640 con 0x557d7d86f6b0 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.426+0000 7fb584ff9700 1 -- 172.21.15.31:0/1638516743 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800a2c30 con 0x557d7d86f6b0 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1638516743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7d86f6b0 msgr2=0x557d7da49560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:45.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7d86f6b0 0x557d7da49560 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb58009e040 tx=0x7fb580005c00 comp rx=0 tx=0).stop 2023-07-29T16:39:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1638516743 shutdown_connections 2023-07-29T16:39:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7da22580 0x557d7da36170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7d86f6b0 0x557d7da49560 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/1638516743 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da0da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1638516743 >> 172.21.15.31:0/1638516743 conn(0x557d7da08fd0 msgr2=0x557d7da0a4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:45.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.427+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1638516743 shutdown_connections 2023-07-29T16:39:45.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.428+0000 7fb5abc0db80 1 -- 172.21.15.31:0/1638516743 wait complete. 2023-07-29T16:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.428+0000 7fb5abc0db80 1 -- start start 2023-07-29T16:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.428+0000 7fb5abc0db80 1 Processor -- start 2023-07-29T16:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da48b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da490b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da8a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7da8a800 con 0x557d7da22580 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7da8a940 con 0x557d7da14cf0 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d7da8aad0 con 0x557d7d86f6b0 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb5abc0db80 10 client.0 _ll_register_callbacks cb 0x557d7d90a5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb595f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da48b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb595f9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da48b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:48862/0 (socket says 172.21.15.31:48862) 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb595f9b700 1 -- 172.21.15.31:0/4107840489 learned_addr learned my addr 172.21.15.31:0/4107840489 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:45.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.432+0000 7fb59579a700 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb595f9b700 1 -- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 msgr2=0x557d7da490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb595f9b700 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da490b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb595f9b700 1 -- 172.21.15.31:0/4107840489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 msgr2=0x557d7da8a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb595f9b700 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da8a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb595f9b700 1 -- 172.21.15.31:0/4107840489 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d7d77bc00 con 0x557d7d86f6b0 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb595f9b700 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da48b70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb5800ce0a0 tx=0x7fb580005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e406e071eedab157 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:45.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb5797fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:39:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb5797fa700 1 -- 172.21.15.31:0/4107840489 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58000ce60 con 0x557d7d86f6b0 2023-07-29T16:39:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb5797fa700 1 -- 172.21.15.31:0/4107840489 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb5800a2c30 con 0x557d7d86f6b0 2023-07-29T16:39:45.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.433+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5800d4e20 con 0x557d7d86f6b0 2023-07-29T16:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.436+0000 7fb5797fa700 1 -- 172.21.15.31:0/4107840489 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb58009a3b0 con 0x557d7d86f6b0 2023-07-29T16:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.436+0000 7fb5abc0db80 10 client.30649 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.436+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x557d7d791250 con 0x557d7d86f6b0 2023-07-29T16:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5797fa700 1 -- 172.21.15.31:0/4107840489 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb5800c7020 con 0x557d7d86f6b0 2023-07-29T16:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5797fa700 1 -- 172.21.15.31:0/4107840489 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fb5800cf790 con 0x557d7d86f6b0 2023-07-29T16:39:45.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5abc0db80 10 client.30649 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5abc0db80 10 client.30649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5abc0db80 20 client.30649 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da8bda0 0x557d7da8e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5abc0db80 4 client.30649 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:39:45.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557d7d95dce0 con 0x557d7da8bda0 2023-07-29T16:39:45.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.437+0000 7fb59579a700 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da8bda0 0x557d7da8e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.446+0000 7fb59579a700 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da8bda0 0x557d7da8e180 crc :-1 s=READY pgs=502 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba121fdd0fae68b server_cookie=4e51136ea062805f in_seq=0 out_seq=0 2023-07-29T16:39:45.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:45.446+0000 7fb5797fa700 10 client.30649 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.111+0000 7fb5797fa700 1 -- 172.21.15.31:0/4107840489 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557d7d95dce0 con 0x557d7da8bda0 2023-07-29T16:39:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.111+0000 7fb5797fa700 10 client.30649 handle_command_reply: tid=0 2023-07-29T16:39:46.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.111+0000 7fb5abc0db80 1 client.30649 shutdown 2023-07-29T16:39:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 msgr2=0x557d7da48b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:46.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da48b70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb5800ce0a0 tx=0x7fb580005c70 comp rx=0 tx=0).stop 2023-07-29T16:39:46.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 shutdown_connections 2023-07-29T16:39:46.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x557d7da8bda0 0x557d7da8e180 crc :-1 s=READY pgs=502 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557d7da22580 0x557d7da8a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5797fa700 0 client.30649 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:46.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x557d7d86f6b0 0x557d7da48b70 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 --2- 172.21.15.31:0/4107840489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7da14cf0 0x557d7da490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.114+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 >> 172.21.15.31:0/4107840489 conn(0x557d7da08fd0 msgr2=0x557d7da0cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.115+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 shutdown_connections 2023-07-29T16:39:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.115+0000 7fb5abc0db80 1 -- 172.21.15.31:0/4107840489 wait complete. 2023-07-29T16:39:46.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.115+0000 7fb5abc0db80 20 client.30649 trim_cache size 0 max 16384 2023-07-29T16:39:46.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:39:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:39:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:39:46.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:39:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb57c0a91a0 msgr2=0x7fb57c0ab650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb57c0a91a0 0x7fb57c0ab650 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb58800aab0 tx=0x7fb588009250 comp rx=0 tx=0).stop 2023-07-29T16:39:46.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 msgr2=0x7fb59019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb59019ec20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb580016040 tx=0x7fb580003960 comp rx=0 tx=0).stop 2023-07-29T16:39:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 shutdown_connections 2023-07-29T16:39:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb57c0a91a0 0x7fb57c0ab650 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5901088f0 0x7fb59019ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb590071fa0 0x7fb59019ec20 secure :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb580016040 tx=0x7fb580003960 comp rx=0 tx=0).stop 2023-07-29T16:39:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 --2- 172.21.15.31:0/2876153431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb590107f60 0x7fb59019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:46.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.118+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 >> 172.21.15.31:0/2876153431 conn(0x7fb59006d5c0 msgr2=0x7fb59010b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.122+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 shutdown_connections 2023-07-29T16:39:46.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:46.122+0000 7fb5981ff700 1 -- 172.21.15.31:0/2876153431 wait complete. 2023-07-29T16:39:46.389 INFO:tasks.workunit.client.0.smithi031.stdout: 10 358179 33.59 MB/sec execute 450 sec latency 400.715 ms 2023-07-29T16:39:47.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:46 smithi031 ceph-mon[95504]: pgmap v1101: 97 pgs: 97 active+clean; 709 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 429 op/s 2023-07-29T16:39:47.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:46 smithi039 ceph-mon[128858]: pgmap v1101: 97 pgs: 97 active+clean; 709 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 429 op/s 2023-07-29T16:39:47.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:46 smithi156 ceph-mon[129245]: pgmap v1101: 97 pgs: 97 active+clean; 709 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 429 op/s 2023-07-29T16:39:47.388 INFO:tasks.workunit.client.0.smithi031.stdout: 10 358408 33.53 MB/sec execute 451 sec latency 139.055 ms 2023-07-29T16:39:48.388 INFO:tasks.workunit.client.0.smithi031.stdout: 10 359169 33.55 MB/sec execute 452 sec latency 107.211 ms 2023-07-29T16:39:48.535 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:48 smithi039 ceph-mon[128858]: pgmap v1102: 97 pgs: 97 active+clean; 698 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.0 MiB/s wr, 533 op/s 2023-07-29T16:39:48.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:48 smithi156 ceph-mon[129245]: pgmap v1102: 97 pgs: 97 active+clean; 698 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.0 MiB/s wr, 533 op/s 2023-07-29T16:39:48.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:48 smithi031 ceph-mon[95504]: pgmap v1102: 97 pgs: 97 active+clean; 698 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.0 MiB/s wr, 533 op/s 2023-07-29T16:39:49.388 INFO:tasks.workunit.client.0.smithi031.stdout: 10 359891 33.58 MB/sec execute 453 sec latency 60.947 ms 2023-07-29T16:39:50.389 INFO:tasks.workunit.client.0.smithi031.stdout: 10 360120 33.54 MB/sec execute 454 sec latency 673.686 ms 2023-07-29T16:39:51.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:50 smithi031 ceph-mon[95504]: pgmap v1103: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 11 MiB/s wr, 611 op/s 2023-07-29T16:39:51.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:50 smithi039 ceph-mon[128858]: pgmap v1103: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 11 MiB/s wr, 611 op/s 2023-07-29T16:39:51.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:50 smithi156 ceph-mon[129245]: pgmap v1103: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 11 MiB/s wr, 611 op/s 2023-07-29T16:39:51.390 INFO:tasks.workunit.client.0.smithi031.stdout: 10 360417 33.52 MB/sec execute 455 sec latency 807.139 ms 2023-07-29T16:39:52.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 360667 33.46 MB/sec execute 456 sec latency 128.812 ms 2023-07-29T16:39:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:52 smithi031 ceph-mon[95504]: pgmap v1104: 97 pgs: 97 active+clean; 686 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 14 MiB/s wr, 769 op/s 2023-07-29T16:39:53.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:52 smithi039 ceph-mon[128858]: pgmap v1104: 97 pgs: 97 active+clean; 686 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 14 MiB/s wr, 769 op/s 2023-07-29T16:39:53.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:52 smithi156 ceph-mon[129245]: pgmap v1104: 97 pgs: 97 active+clean; 686 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 14 MiB/s wr, 769 op/s 2023-07-29T16:39:53.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 361075 33.41 MB/sec execute 457 sec latency 164.086 ms 2023-07-29T16:39:54.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 361710 33.42 MB/sec execute 458 sec latency 81.239 ms 2023-07-29T16:39:55.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:54 smithi031 ceph-mon[95504]: pgmap v1105: 97 pgs: 97 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 17 MiB/s wr, 824 op/s 2023-07-29T16:39:55.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:54 smithi039 ceph-mon[128858]: pgmap v1105: 97 pgs: 97 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 17 MiB/s wr, 824 op/s 2023-07-29T16:39:55.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:54 smithi156 ceph-mon[129245]: pgmap v1105: 97 pgs: 97 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 17 MiB/s wr, 824 op/s 2023-07-29T16:39:55.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 362556 33.44 MB/sec execute 459 sec latency 43.125 ms 2023-07-29T16:39:56.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 363322 33.47 MB/sec execute 460 sec latency 63.759 ms 2023-07-29T16:39:57.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:56 smithi031 ceph-mon[95504]: pgmap v1106: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 18 MiB/s wr, 872 op/s 2023-07-29T16:39:57.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:56 smithi039 ceph-mon[128858]: pgmap v1106: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 18 MiB/s wr, 872 op/s 2023-07-29T16:39:57.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:56 smithi156 ceph-mon[129245]: pgmap v1106: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 18 MiB/s wr, 872 op/s 2023-07-29T16:39:57.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 363929 33.47 MB/sec execute 461 sec latency 74.503 ms 2023-07-29T16:39:58.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 364365 33.44 MB/sec execute 462 sec latency 182.419 ms 2023-07-29T16:39:58.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:39:58 smithi039 ceph-mon[128858]: pgmap v1107: 97 pgs: 97 active+clean; 682 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 16 MiB/s wr, 850 op/s 2023-07-29T16:39:58.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:39:58 smithi156 ceph-mon[129245]: pgmap v1107: 97 pgs: 97 active+clean; 682 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 16 MiB/s wr, 850 op/s 2023-07-29T16:39:58.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:39:58 smithi031 ceph-mon[95504]: pgmap v1107: 97 pgs: 97 active+clean; 682 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 16 MiB/s wr, 850 op/s 2023-07-29T16:39:59.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/2316235218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18103780 msgr2=0x7f4b18105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/2316235218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18103780 0x7f4b18105b60 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0098f0 tx=0x7f4b0c009c00 comp rx=0 tx=0).stop 2023-07-29T16:39:59.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/2316235218 shutdown_connections 2023-07-29T16:39:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/2316235218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b181060a0 0x7f4b18108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/2316235218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b18100e60 0x7f4b18103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/2316235218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18103780 0x7f4b18105b60 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.366+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/2316235218 >> 172.21.15.31:0/2316235218 conn(0x7f4b180fa840 msgr2=0x7f4b180fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.367+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/2316235218 shutdown_connections 2023-07-29T16:39:59.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.367+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/2316235218 wait complete. 2023-07-29T16:39:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.368+0000 7f4b1e5ff700 1 Processor -- start 2023-07-29T16:39:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.368+0000 7f4b1e5ff700 1 -- start start 2023-07-29T16:39:59.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b1e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 0x7f4b18198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b1e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b18103780 0x7f4b18198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b1e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b181060a0 0x7f4b181997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b1e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18199e50 con 0x7f4b181060a0 2023-07-29T16:39:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b1e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b18199fc0 con 0x7f4b18103780 2023-07-29T16:39:59.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b1e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1819a180 con 0x7f4b18100e60 2023-07-29T16:39:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 0x7f4b18198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 0x7f4b18198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39812/0 (socket says 172.21.15.31:39812) 2023-07-29T16:39:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b177fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b18103780 0x7f4b18198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.369+0000 7f4b17fff700 1 -- 172.21.15.31:0/3956294515 learned_addr learned my addr 172.21.15.31:0/3956294515 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b181060a0 0x7f4b181997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b17fff700 1 -- 172.21.15.31:0/3956294515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b18103780 msgr2=0x7f4b18198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b17fff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b18103780 0x7f4b18198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b17fff700 1 -- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b181060a0 msgr2=0x7f4b181997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b17fff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b181060a0 0x7f4b181997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b17fff700 1 -- 172.21.15.31:0/3956294515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c009520 con 0x7f4b18100e60 2023-07-29T16:39:59.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b17fff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 0x7f4b18198430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800e960 tx=0x7f4b0800ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b157fa700 1 -- 172.21.15.31:0/3956294515 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08004220 con 0x7f4b18100e60 2023-07-29T16:39:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b157fa700 1 -- 172.21.15.31:0/3956294515 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b08004cf0 con 0x7f4b18100e60 2023-07-29T16:39:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1819a460 con 0x7f4b18100e60 2023-07-29T16:39:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.370+0000 7f4b157fa700 1 -- 172.21.15.31:0/3956294515 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08005720 con 0x7f4b18100e60 2023-07-29T16:39:59.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.371+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b181a0c20 con 0x7f4b18100e60 2023-07-29T16:39:59.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.377+0000 7f4b157fa700 1 -- 172.21.15.31:0/3956294515 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f4b08004380 con 0x7f4b18100e60 2023-07-29T16:39:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.378+0000 7f4b31ef2b80 1 Processor -- start 2023-07-29T16:39:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.380+0000 7f4b157fa700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4b000a0790 0x7f4b000a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.380+0000 7f4b157fa700 1 -- 172.21.15.31:0/3956294515 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4b080cb920 con 0x7f4b18100e60 2023-07-29T16:39:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.381+0000 7f4b31ef2b80 1 -- start start 2023-07-29T16:39:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.381+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be755916b0 0x55be75733a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.381+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be757325c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.381+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75732f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be7568d9b0 con 0x55be755916b0 2023-07-29T16:39:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be7568b370 con 0x55be75733f90 2023-07-29T16:39:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be75686c60 con 0x55be75732b00 2023-07-29T16:39:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b177fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be757325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4b000a0790 0x7f4b000a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b177fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be757325c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45938/0 (socket says 172.21.15.31:45938) 2023-07-29T16:39:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b177fe700 1 -- 172.21.15.31:0/385239546 learned_addr learned my addr 172.21.15.31:0/385239546 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:59.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.382+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75732f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.383+0000 7f4b177fe700 1 -- 172.21.15.31:0/385239546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 msgr2=0x55be75732f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.383+0000 7f4b177fe700 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75732f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.383+0000 7f4b177fe700 1 -- 172.21.15.31:0/385239546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be755916b0 msgr2=0x55be75733a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.383+0000 7f4b177fe700 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be755916b0 0x55be75733a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.383+0000 7f4b177fe700 1 -- 172.21.15.31:0/385239546 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be754b4d80 con 0x55be75733f90 2023-07-29T16:39:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.384+0000 7f4b177fe700 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be757325c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0098c0 tx=0x7f4b0c005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d0de297e0f9a42b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.384+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4b000a0790 0x7f4b000a2c40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4b10007720 tx=0x7f4b100094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:59.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.384+0000 7f4afe7fc700 1 -- 172.21.15.31:0/385239546 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c01c050 con 0x55be75733f90 2023-07-29T16:39:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.384+0000 7f4afe7fc700 1 -- 172.21.15.31:0/385239546 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b0c01d040 con 0x55be75733f90 2023-07-29T16:39:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.384+0000 7f4afe7fc700 1 -- 172.21.15.31:0/385239546 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0c00b870 con 0x55be75733f90 2023-07-29T16:39:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.385+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/385239546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 msgr2=0x55be757325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.385+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be757325c0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0098c0 tx=0x7f4b0c005dc0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/385239546 shutdown_connections 2023-07-29T16:39:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75732f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be755916b0 0x55be75733a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/385239546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be757325c0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/385239546 >> 172.21.15.31:0/385239546 conn(0x55be7572afd0 msgr2=0x55be75730310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/385239546 shutdown_connections 2023-07-29T16:39:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.386+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/385239546 wait complete. 2023-07-29T16:39:59.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.387+0000 7f4b31ef2b80 1 -- start start 2023-07-29T16:39:59.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.387+0000 7f4b31ef2b80 1 Processor -- start 2023-07-29T16:39:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be7573f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75732b00 0x55be7573f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be75746480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be756234b0 con 0x55be75733f90 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be75623150 con 0x55be75732b00 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be7550a8c0 con 0x55be755916b0 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.388+0000 7f4b31ef2b80 10 client.0 _ll_register_callbacks cb 0x55be7562c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be75746480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be75746480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52652/0 (socket says 172.21.15.31:52652) 2023-07-29T16:39:59.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/1024672736 learned_addr learned my addr 172.21.15.31:0/1024672736 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b17fff700 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be7573f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/1024672736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 msgr2=0x55be7573f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be7573f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75732b00 msgr2=0x55be7573f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75732b00 0x55be7573f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.389+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/1024672736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b181090a0 con 0x55be75733f90 2023-07-29T16:39:59.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.390+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be75746480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4b10019d10 tx=0x7f4b10004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eacfee8dc7312065 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:59.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.390+0000 7f4af77fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:39:59.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 365007 33.46 MB/sec execute 463 sec latency 83.207 ms 2023-07-29T16:39:59.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.391+0000 7f4af77fe700 1 -- 172.21.15.31:0/1024672736 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10018d80 con 0x55be75733f90 2023-07-29T16:39:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.391+0000 7f4af77fe700 1 -- 172.21.15.31:0/1024672736 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4b10024440 con 0x55be75733f90 2023-07-29T16:39:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.391+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be756b1cb0 con 0x55be75733f90 2023-07-29T16:39:59.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.392+0000 7f4af77fe700 1 -- 172.21.15.31:0/1024672736 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b10024600 con 0x55be75733f90 2023-07-29T16:39:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.392+0000 7f4b31ef2b80 10 client.20346 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.392+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55be7549dc00 con 0x55be75733f90 2023-07-29T16:39:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4af77fe700 1 -- 172.21.15.31:0/1024672736 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55be756b1cb0 con 0x55be75733f90 2023-07-29T16:39:59.401 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4af77fe700 1 -- 172.21.15.31:0/1024672736 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4b1002c070 con 0x55be75733f90 2023-07-29T16:39:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4b31ef2b80 10 client.20346 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4b31ef2b80 10 client.20346 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4b31ef2b80 20 client.20346 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:59.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be7574af30 0x55be7574d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4b31ef2b80 4 client.20346 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:39:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.393+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be75704d10 con 0x55be7574af30 2023-07-29T16:39:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.394+0000 7f4b17fff700 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be7574af30 0x55be7574d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.394+0000 7f4b17fff700 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be7574af30 0x55be7574d340 crc :-1 s=READY pgs=503 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecb65346b954ead2 server_cookie=ec61c09559d1f052 in_seq=0 out_seq=0 2023-07-29T16:39:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.394+0000 7f4af77fe700 10 client.20346 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:59.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.396+0000 7f4af77fe700 1 -- 172.21.15.31:0/1024672736 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55be75704d10 con 0x55be7574af30 2023-07-29T16:39:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.396+0000 7f4af77fe700 10 client.20346 handle_command_reply: tid=0 2023-07-29T16:39:59.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.396+0000 7f4b31ef2b80 1 client.20346 shutdown 2023-07-29T16:39:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 msgr2=0x55be75746480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be75746480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4b10019d10 tx=0x7f4b10004d50 comp rx=0 tx=0).stop 2023-07-29T16:39:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 shutdown_connections 2023-07-29T16:39:59.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be7574af30 0x55be7574d340 crc :-1 s=READY pgs=503 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be75746480 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be7573f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1024672736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75732b00 0x55be7573f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.398+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 >> 172.21.15.31:0/1024672736 conn(0x55be7572afd0 msgr2=0x55be7572fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.399+0000 7f4af77fe700 0 client.20346 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.399+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 shutdown_connections 2023-07-29T16:39:59.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.399+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1024672736 wait complete. 2023-07-29T16:39:59.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.400+0000 7f4b31ef2b80 20 client.20346 trim_cache size 0 max 16384 2023-07-29T16:39:59.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.423+0000 7f4b31ef2b80 1 Processor -- start 2023-07-29T16:39:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 -- start start 2023-07-29T16:39:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be757463a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75732b00 0x55be75746f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be7574cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be7573ea70 con 0x55be75732b00 2023-07-29T16:39:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be7572a9d0 con 0x55be75733f90 2023-07-29T16:39:59.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.424+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be7573a950 con 0x55be755916b0 2023-07-29T16:39:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be7574cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be757463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be757463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39838/0 (socket says 172.21.15.31:39838) 2023-07-29T16:39:59.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b17fff700 1 -- 172.21.15.31:0/3356367178 learned_addr learned my addr 172.21.15.31:0/3356367178 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be7574cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45952/0 (socket says 172.21.15.31:45952) 2023-07-29T16:39:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b177fe700 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75732b00 0x55be75746f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.431 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/3356367178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 msgr2=0x55be757463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be757463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/3356367178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75732b00 msgr2=0x55be75746f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.432 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75732b00 0x55be75746f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.425+0000 7f4b1cb9c700 1 -- 172.21.15.31:0/3356367178 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be754db1f0 con 0x55be75733f90 2023-07-29T16:39:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.426+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be7574cf40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x55be7573fc90 tx=0x7f4b10004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6277a01dad850e46 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:59.433 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.426+0000 7f4afeffd700 1 -- 172.21.15.31:0/3356367178 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10007dc0 con 0x55be75733f90 2023-07-29T16:39:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.426+0000 7f4afeffd700 1 -- 172.21.15.31:0/3356367178 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b1000f030 con 0x55be75733f90 2023-07-29T16:39:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.426+0000 7f4afeffd700 1 -- 172.21.15.31:0/3356367178 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100233a0 con 0x55be75733f90 2023-07-29T16:39:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/3356367178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 msgr2=0x55be7574cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be7574cf40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x55be7573fc90 tx=0x7f4b10004cd0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/3356367178 shutdown_connections 2023-07-29T16:39:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be75733f90 0x55be7574cf40 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be755916b0 0x55be757463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/3356367178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75732b00 0x55be75746f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/3356367178 >> 172.21.15.31:0/3356367178 conn(0x55be7572afd0 msgr2=0x55be7572c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.427+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/3356367178 shutdown_connections 2023-07-29T16:39:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.428+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/3356367178 wait complete. 2023-07-29T16:39:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.428+0000 7f4b31ef2b80 1 -- start start 2023-07-29T16:39:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.428+0000 7f4b31ef2b80 1 Processor -- start 2023-07-29T16:39:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 0x55be75736a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75738f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be757cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be75739740 con 0x55be75733f90 2023-07-29T16:39:59.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be757cf120 con 0x55be755916b0 2023-07-29T16:39:59.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be757cf2e0 con 0x55be75732b00 2023-07-29T16:39:59.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b31ef2b80 10 client.0 _ll_register_callbacks cb 0x55be7562c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:39:59.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 0x55be75736a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b17fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 0x55be75736a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:45956/0 (socket says 172.21.15.31:45956) 2023-07-29T16:39:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b17fff700 1 -- 172.21.15.31:0/1655133469 learned_addr learned my addr 172.21.15.31:0/1655133469 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:39:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.429+0000 7f4b177fe700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75738f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be757cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b17fff700 1 -- 172.21.15.31:0/1655133469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 msgr2=0x55be75738f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b17fff700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75738f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b17fff700 1 -- 172.21.15.31:0/1655133469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 msgr2=0x55be757cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:39:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b17fff700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be757cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b17fff700 1 -- 172.21.15.31:0/1655133469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55be7549dc00 con 0x55be755916b0 2023-07-29T16:39:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b1cb9c700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be757cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:39:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b17fff700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 0x55be75736a30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f4b080a0ee0 tx=0x7f4b080990e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8fc5ab5fb611bbb server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:39:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4af77fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:39:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4af77fe700 1 -- 172.21.15.31:0/1655133469 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b080b72f0 con 0x55be755916b0 2023-07-29T16:39:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4af77fe700 1 -- 172.21.15.31:0/1655133469 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f4b080956d0 con 0x55be755916b0 2023-07-29T16:39:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b0c01d040 con 0x55be755916b0 2023-07-29T16:39:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4af77fe700 1 -- 172.21.15.31:0/1655133469 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b08095890 con 0x55be755916b0 2023-07-29T16:39:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.430+0000 7f4b31ef2b80 10 client.30641 fetch_fsmap learned FSMap version 43 2023-07-29T16:39:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55be754b3250 con 0x55be755916b0 2023-07-29T16:39:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4af77fe700 1 -- 172.21.15.31:0/1655133469 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b0c01d040 con 0x55be755916b0 2023-07-29T16:39:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4af77fe700 1 -- 172.21.15.31:0/1655133469 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f4b08093bd0 con 0x55be755916b0 2023-07-29T16:39:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 10 client.30641 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:39:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 10 client.30641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:39:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 20 client.30641 populate_metadata read hostname 'smithi031' 2023-07-29T16:39:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be757d0d00 0x55be757d3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:39:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 4 client.30641 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:39:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55be7567fce0 con 0x55be757d0d00 2023-07-29T16:39:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.431+0000 7f4b177fe700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be757d0d00 0x55be757d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:39:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.432+0000 7f4b177fe700 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be757d0d00 0x55be757d3110 crc :-1 s=READY pgs=504 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da8abe382b7cc65 server_cookie=71ed5a5563d79a52 in_seq=0 out_seq=0 2023-07-29T16:39:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.432+0000 7f4af77fe700 10 client.30641 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.855+0000 7f4af77fe700 1 -- 172.21.15.31:0/1655133469 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55be7567fce0 con 0x55be757d0d00 2023-07-29T16:39:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.855+0000 7f4af77fe700 10 client.30641 handle_command_reply: tid=0 2023-07-29T16:39:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.855+0000 7f4b31ef2b80 1 client.30641 shutdown 2023-07-29T16:39:59.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 msgr2=0x55be75736a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 0x55be75736a30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f4b080a0ee0 tx=0x7f4b080990e0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 shutdown_connections 2023-07-29T16:39:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55be757d0d00 0x55be757d3110 crc :-1 s=READY pgs=504 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be75733f90 0x55be757cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55be755916b0 0x55be75736a30 secure :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f4b080a0ee0 tx=0x7f4b080990e0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 --2- 172.21.15.31:0/1655133469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55be75732b00 0x55be75738f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 >> 172.21.15.31:0/1655133469 conn(0x55be7572afd0 msgr2=0x55be7574cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4af77fe700 0 client.30641 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:39:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 shutdown_connections 2023-07-29T16:39:59.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.856+0000 7f4b31ef2b80 1 -- 172.21.15.31:0/1655133469 wait complete. 2023-07-29T16:39:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.857+0000 7f4b31ef2b80 20 client.30641 trim_cache size 0 max 16384 2023-07-29T16:39:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:39:59.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:39:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:39:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:39:59.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.859+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4b000a0790 msgr2=0x7f4b000a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.859+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4b000a0790 0x7f4b000a2c40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4b10007720 tx=0x7f4b100094f0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.859+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 msgr2=0x7f4b18198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:39:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.859+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 0x7f4b18198430 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4b0800e960 tx=0x7f4b0800ec70 comp rx=0 tx=0).stop 2023-07-29T16:39:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 shutdown_connections 2023-07-29T16:39:59.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f4b000a0790 0x7f4b000a2c40 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b181060a0 0x7f4b181997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b18100e60 0x7f4b18198430 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 --2- 172.21.15.31:0/3956294515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b18103780 0x7f4b18198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:39:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 >> 172.21.15.31:0/3956294515 conn(0x7f4b180fa840 msgr2=0x7f4b18101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:39:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 shutdown_connections 2023-07-29T16:39:59.868 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:39:59.861+0000 7f4b1e5ff700 1 -- 172.21.15.31:0/3956294515 wait complete. 2023-07-29T16:40:00.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 365621 33.42 MB/sec execute 464 sec latency 282.962 ms 2023-07-29T16:40:00.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:39:59.998+0000 7fa43fcae700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-07-29T16:40:00.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:39:59.998+0000 7fa43fcae700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-07-29T16:40:00.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:39:59.998+0000 7fa43fcae700 -1 log_channel(cluster) log [ERR] : mds.i(mds.2): Metadata damage detected 2023-07-29T16:40:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-mon[95504]: pgmap v1108: 97 pgs: 97 active+clean; 687 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 14 MiB/s wr, 751 op/s 2023-07-29T16:40:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-mon[95504]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-07-29T16:40:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-mon[95504]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-07-29T16:40:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:00 smithi031 ceph-mon[95504]: mds.i(mds.2): Metadata damage detected 2023-07-29T16:40:01.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:00 smithi039 ceph-mon[128858]: pgmap v1108: 97 pgs: 97 active+clean; 687 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 14 MiB/s wr, 751 op/s 2023-07-29T16:40:01.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:00 smithi039 ceph-mon[128858]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-07-29T16:40:01.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:00 smithi039 ceph-mon[128858]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-07-29T16:40:01.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:00 smithi039 ceph-mon[128858]: mds.i(mds.2): Metadata damage detected 2023-07-29T16:40:01.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:00 smithi156 ceph-mon[129245]: pgmap v1108: 97 pgs: 97 active+clean; 687 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 14 MiB/s wr, 751 op/s 2023-07-29T16:40:01.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:00 smithi156 ceph-mon[129245]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2023-07-29T16:40:01.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:00 smithi156 ceph-mon[129245]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2023-07-29T16:40:01.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:00 smithi156 ceph-mon[129245]: mds.i(mds.2): Metadata damage detected 2023-07-29T16:40:01.394 INFO:tasks.workunit.client.0.smithi031.stdout: 10 366482 33.48 MB/sec execute 465 sec latency 45.379 ms 2023-07-29T16:40:02.393 INFO:tasks.workunit.client.0.smithi031.stdout: 10 367221 33.48 MB/sec execute 466 sec latency 78.670 ms 2023-07-29T16:40:03.133 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:02 smithi156 ceph-mon[129245]: pgmap v1109: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 10 MiB/s wr, 646 op/s 2023-07-29T16:40:03.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:02 smithi031 ceph-mon[95504]: pgmap v1109: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 10 MiB/s wr, 646 op/s 2023-07-29T16:40:03.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:02 smithi039 ceph-mon[128858]: pgmap v1109: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 10 MiB/s wr, 646 op/s 2023-07-29T16:40:03.394 INFO:tasks.workunit.client.0.smithi031.stdout: 10 367833 33.50 MB/sec execute 467 sec latency 88.612 ms 2023-07-29T16:40:04.395 INFO:tasks.workunit.client.0.smithi031.stdout: 10 368410 33.47 MB/sec execute 468 sec latency 88.608 ms 2023-07-29T16:40:05.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:04 smithi031 ceph-mon[95504]: pgmap v1110: 97 pgs: 97 active+clean; 692 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 8.2 MiB/s wr, 653 op/s 2023-07-29T16:40:05.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:04 smithi039 ceph-mon[128858]: pgmap v1110: 97 pgs: 97 active+clean; 692 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 8.2 MiB/s wr, 653 op/s 2023-07-29T16:40:05.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:04 smithi156 ceph-mon[129245]: pgmap v1110: 97 pgs: 97 active+clean; 692 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 8.2 MiB/s wr, 653 op/s 2023-07-29T16:40:05.396 INFO:tasks.workunit.client.0.smithi031.stdout: 10 369111 33.48 MB/sec execute 469 sec latency 62.059 ms 2023-07-29T16:40:06.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 369886 33.51 MB/sec execute 470 sec latency 95.803 ms 2023-07-29T16:40:07.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:06 smithi031 ceph-mon[95504]: pgmap v1111: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.5 MiB/s wr, 557 op/s 2023-07-29T16:40:07.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:06 smithi039 ceph-mon[128858]: pgmap v1111: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.5 MiB/s wr, 557 op/s 2023-07-29T16:40:07.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:06 smithi156 ceph-mon[129245]: pgmap v1111: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.5 MiB/s wr, 557 op/s 2023-07-29T16:40:07.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 370422 33.51 MB/sec execute 471 sec latency 61.358 ms 2023-07-29T16:40:08.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 371044 33.50 MB/sec execute 472 sec latency 66.030 ms 2023-07-29T16:40:08.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:08 smithi039 ceph-mon[128858]: pgmap v1112: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.5 MiB/s wr, 478 op/s 2023-07-29T16:40:08.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:08 smithi156 ceph-mon[129245]: pgmap v1112: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.5 MiB/s wr, 478 op/s 2023-07-29T16:40:08.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:08 smithi031 ceph-mon[95504]: pgmap v1112: 97 pgs: 97 active+clean; 691 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.5 MiB/s wr, 478 op/s 2023-07-29T16:40:09.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 371686 33.50 MB/sec execute 473 sec latency 58.148 ms 2023-07-29T16:40:10.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 372371 33.50 MB/sec execute 474 sec latency 98.036 ms 2023-07-29T16:40:11.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:10 smithi031 ceph-mon[95504]: pgmap v1113: 97 pgs: 97 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 483 op/s 2023-07-29T16:40:11.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:10 smithi039 ceph-mon[128858]: pgmap v1113: 97 pgs: 97 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 483 op/s 2023-07-29T16:40:11.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:10 smithi156 ceph-mon[129245]: pgmap v1113: 97 pgs: 97 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.5 MiB/s wr, 483 op/s 2023-07-29T16:40:11.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 372973 33.51 MB/sec execute 475 sec latency 119.257 ms 2023-07-29T16:40:12.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 373429 33.49 MB/sec execute 476 sec latency 122.056 ms 2023-07-29T16:40:13.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:12 smithi031 ceph-mon[95504]: pgmap v1114: 97 pgs: 97 active+clean; 699 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.9 MiB/s wr, 523 op/s 2023-07-29T16:40:13.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:12 smithi039 ceph-mon[128858]: pgmap v1114: 97 pgs: 97 active+clean; 699 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.9 MiB/s wr, 523 op/s 2023-07-29T16:40:13.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:12 smithi156 ceph-mon[129245]: pgmap v1114: 97 pgs: 97 active+clean; 699 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.9 MiB/s wr, 523 op/s 2023-07-29T16:40:13.397 INFO:tasks.workunit.client.0.smithi031.stdout: 10 374067 33.47 MB/sec execute 477 sec latency 198.462 ms 2023-07-29T16:40:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 -- 172.21.15.31:0/1099425534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c1037d0 msgr2=0x7fca1c103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 --2- 172.21.15.31:0/1099425534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c1037d0 0x7fca1c103c20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009990 tx=0x7fca0c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 -- 172.21.15.31:0/1099425534 shutdown_connections 2023-07-29T16:40:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 --2- 172.21.15.31:0/1099425534 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1c104160 0x7fca1c10aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 --2- 172.21.15.31:0/1099425534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c1029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 --2- 172.21.15.31:0/1099425534 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c1037d0 0x7fca1c103c20 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.829+0000 7fca22bff700 1 -- 172.21.15.31:0/1099425534 >> 172.21.15.31:0/1099425534 conn(0x7fca1c0fdc00 msgr2=0x7fca1c100030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.830+0000 7fca22bff700 1 -- 172.21.15.31:0/1099425534 shutdown_connections 2023-07-29T16:40:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.830+0000 7fca22bff700 1 -- 172.21.15.31:0/1099425534 wait complete. 2023-07-29T16:40:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.832+0000 7fca22bff700 1 Processor -- start 2023-07-29T16:40:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.832+0000 7fca22bff700 1 -- start start 2023-07-29T16:40:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca22bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c075600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca22bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1c1037d0 0x7fca1c075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca22bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c104160 0x7fca1c076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca22bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c077070 con 0x7fca1c1037d0 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca22bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0771e0 con 0x7fca1c104160 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca22bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1c0773a0 con 0x7fca1c1025e0 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca2099b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c075600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca2099b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c075600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46706/0 (socket says 172.21.15.31:46706) 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca2099b700 1 -- 172.21.15.31:0/2296711960 learned_addr learned my addr 172.21.15.31:0/2296711960 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca2119c700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c104160 0x7fca1c076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.836 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.833+0000 7fca1bfff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1c1037d0 0x7fca1c075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2099b700 1 -- 172.21.15.31:0/2296711960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c104160 msgr2=0x7fca1c076a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2099b700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c104160 0x7fca1c076a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2099b700 1 -- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1c1037d0 msgr2=0x7fca1c075b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2099b700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1c1037d0 0x7fca1c075b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2099b700 1 -- 172.21.15.31:0/2296711960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0095f0 con 0x7fca1c1025e0 2023-07-29T16:40:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2119c700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c104160 0x7fca1c076a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:40:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca2099b700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c075600 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b500 tx=0x7fca1000b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca19ffb700 1 -- 172.21.15.31:0/2296711960 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10011830 con 0x7fca1c1025e0 2023-07-29T16:40:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1c077680 con 0x7fca1c1025e0 2023-07-29T16:40:13.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.834+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c077eb0 con 0x7fca1c1025e0 2023-07-29T16:40:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.835+0000 7fca19ffb700 1 -- 172.21.15.31:0/2296711960 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca10011e50 con 0x7fca1c1025e0 2023-07-29T16:40:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.835+0000 7fca19ffb700 1 -- 172.21.15.31:0/2296711960 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1000f2e0 con 0x7fca1c1025e0 2023-07-29T16:40:13.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.840+0000 7fca19ffb700 1 -- 172.21.15.31:0/2296711960 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fca10011990 con 0x7fca1c1025e0 2023-07-29T16:40:13.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.841+0000 7fca3642bb80 1 Processor -- start 2023-07-29T16:40:13.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 -- start start 2023-07-29T16:40:13.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e4177f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4177f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e4177fa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4177509b0 con 0x55e4176546b0 2023-07-29T16:40:13.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e41774e370 con 0x55e4177f8df0 2023-07-29T16:40:13.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.842+0000 7fca3642bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417749c60 con 0x55e4177f9cf0 2023-07-29T16:40:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.843+0000 7fca1bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e4177fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.843+0000 7fca1bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e4177fa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46720/0 (socket says 172.21.15.31:46720) 2023-07-29T16:40:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.843+0000 7fca1bfff700 1 -- 172.21.15.31:0/2162225417 learned_addr learned my addr 172.21.15.31:0/2162225417 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.843+0000 7fca2119c700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e4177f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.843+0000 7fca19ffb700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca040a0890 0x7fca040a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.843+0000 7fca19ffb700 1 -- 172.21.15.31:0/2296711960 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca100cb420 con 0x7fca1c1025e0 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2099b700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4177f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 -- 172.21.15.31:0/2162225417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 msgr2=0x55e4177fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e4177fa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 -- 172.21.15.31:0/2162225417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 msgr2=0x55e4177f97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4177f97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 -- 172.21.15.31:0/2162225417 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e417577d80 con 0x55e4176546b0 2023-07-29T16:40:13.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2099b700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4177f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:40:13.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca1bfff700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e4177fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:40:13.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e4177f88b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fca14003e60 tx=0x7fca14004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533587eff20a7753 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.844+0000 7fca2119c700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca040a0890 0x7fca040a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.845+0000 7fca02ffd700 1 -- 172.21.15.31:0/2162225417 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14007500 con 0x55e4176546b0 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.845+0000 7fca02ffd700 1 -- 172.21.15.31:0/2162225417 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca1400f040 con 0x55e4176546b0 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.845+0000 7fca02ffd700 1 -- 172.21.15.31:0/2162225417 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca1400eb20 con 0x55e4176546b0 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 -- 172.21.15.31:0/2162225417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 msgr2=0x55e4177f88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e4177f88b0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fca14003e60 tx=0x7fca14004320 comp rx=0 tx=0).stop 2023-07-29T16:40:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 -- 172.21.15.31:0/2162225417 shutdown_connections 2023-07-29T16:40:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e4177fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e4177f88b0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2162225417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4177f97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 -- 172.21.15.31:0/2162225417 >> 172.21.15.31:0/2162225417 conn(0x55e4177edfd0 msgr2=0x55e4177f3300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 -- 172.21.15.31:0/2162225417 shutdown_connections 2023-07-29T16:40:13.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.846+0000 7fca3642bb80 1 -- 172.21.15.31:0/2162225417 wait complete. 2023-07-29T16:40:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.847+0000 7fca3642bb80 1 -- start start 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.847+0000 7fca3642bb80 1 Processor -- start 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417801f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e417802440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e417809480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4176e64b0 con 0x55e4176546b0 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4176e6150 con 0x55e4177f8df0 2023-07-29T16:40:13.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4175cd8c0 con 0x55e4177f9cf0 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.848+0000 7fca3642bb80 10 client.0 _ll_register_callbacks cb 0x55e4176ef5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e417809480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e417809480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46722/0 (socket says 172.21.15.31:46722) 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 -- 172.21.15.31:0/1328700254 learned_addr learned my addr 172.21.15.31:0/1328700254 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2099b700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417801f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca040a0890 0x7fca040a2d40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fca140060b0 tx=0x7fca14004320 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:13.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca1bfff700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e417802440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 -- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 msgr2=0x55e417802440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e417802440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 -- 172.21.15.31:0/1328700254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 msgr2=0x55e417801f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417801f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2119c700 1 -- 172.21.15.31:0/1328700254 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1c10b350 con 0x55e4177f9cf0 2023-07-29T16:40:13.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.849+0000 7fca2099b700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417801f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:40:13.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fca2119c700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e417809480 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fca1400ef60 tx=0x7fca1400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0080f89224aee11 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:13.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fc9fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:40:13.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fc9fbfff700 1 -- 172.21.15.31:0/1328700254 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca14004db0 con 0x55e4177f9cf0 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e417774cb0 con 0x55e4177f9cf0 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fc9fbfff700 1 -- 172.21.15.31:0/1328700254 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca14026070 con 0x55e4177f9cf0 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fc9fbfff700 1 -- 172.21.15.31:0/1328700254 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca14007d20 con 0x55e4177f9cf0 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fca3642bb80 10 client.30676 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x7fca14003b40 con 0x55e4177f9cf0 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.850+0000 7fc9fbfff700 1 -- 172.21.15.31:0/1328700254 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e417774cb0 con 0x55e4177f9cf0 2023-07-29T16:40:13.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fc9fbfff700 1 -- 172.21.15.31:0/1328700254 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca1402c470 con 0x55e4177f9cf0 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca3642bb80 10 client.30676 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca3642bb80 10 client.30676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca3642bb80 20 client.30676 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca3642bb80 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e41780dc40 0x55e417810050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca3642bb80 4 client.30676 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4177c7d10 con 0x55e41780dc40 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.851+0000 7fca2099b700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e41780dc40 0x55e417810050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.852+0000 7fca2099b700 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e41780dc40 0x55e417810050 crc :-1 s=READY pgs=505 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2508e7e8433f6a59 server_cookie=b5e2338453c2798f in_seq=0 out_seq=0 2023-07-29T16:40:13.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.852+0000 7fc9fbfff700 10 client.30676 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:13.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.860+0000 7fc9fbfff700 1 -- 172.21.15.31:0/1328700254 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55e4177c7d10 con 0x55e41780dc40 2023-07-29T16:40:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fc9fbfff700 10 client.30676 handle_command_reply: tid=0 2023-07-29T16:40:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 client.30676 shutdown 2023-07-29T16:40:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 msgr2=0x55e417809480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:13.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e417809480 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fca1400ef60 tx=0x7fca1400b040 comp rx=0 tx=0).stop 2023-07-29T16:40:13.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 shutdown_connections 2023-07-29T16:40:13.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e41780dc40 0x55e417810050 crc :-1 s=READY pgs=505 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f9cf0 0x55e417809480 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417801f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fc9fbfff700 0 client.30676 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 --2- 172.21.15.31:0/1328700254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e417802440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.861+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 >> 172.21.15.31:0/1328700254 conn(0x55e4177edfd0 msgr2=0x55e4177f2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.862+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 shutdown_connections 2023-07-29T16:40:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.862+0000 7fca3642bb80 1 -- 172.21.15.31:0/1328700254 wait complete. 2023-07-29T16:40:13.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.862+0000 7fca3642bb80 20 client.30676 trim_cache size 0 max 16384 2023-07-29T16:40:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.880+0000 7fca3642bb80 1 Processor -- start 2023-07-29T16:40:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 -- start start 2023-07-29T16:40:13.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417807390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 0x55e4177f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f9cf0 0x55e417809d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417801a70 con 0x55e4176546b0 2023-07-29T16:40:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4177f41c0 con 0x55e4177f9cf0 2023-07-29T16:40:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca3642bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4177ed9d0 con 0x55e4177f8df0 2023-07-29T16:40:13.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca1bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 0x55e4177f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca2099b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417807390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca2119c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f9cf0 0x55e417809d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca1bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 0x55e4177f2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46738/0 (socket says 172.21.15.31:46738) 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.881+0000 7fca1bfff700 1 -- 172.21.15.31:0/28093051 learned_addr learned my addr 172.21.15.31:0/28093051 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca1bfff700 1 -- 172.21.15.31:0/28093051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f9cf0 msgr2=0x55e417809d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca1bfff700 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f9cf0 0x55e417809d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca1bfff700 1 -- 172.21.15.31:0/28093051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 msgr2=0x55e417807390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca1bfff700 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417807390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca1bfff700 1 -- 172.21.15.31:0/28093051 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e417560c00 con 0x55e4177f8df0 2023-07-29T16:40:13.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca1bfff700 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 0x55e4177f2a40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009990 tx=0x7fca0c00b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c73209922f75e4d server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca037fe700 1 -- 172.21.15.31:0/28093051 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c005240 con 0x55e4177f8df0 2023-07-29T16:40:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca037fe700 1 -- 172.21.15.31:0/28093051 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca0c01e040 con 0x55e4177f8df0 2023-07-29T16:40:13.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.882+0000 7fca037fe700 1 -- 172.21.15.31:0/28093051 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c004d20 con 0x55e4177f8df0 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.883+0000 7fca3642bb80 1 -- 172.21.15.31:0/28093051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 msgr2=0x55e4177f2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.883+0000 7fca3642bb80 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 0x55e4177f2a40 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fca0c009990 tx=0x7fca0c00b940 comp rx=0 tx=0).stop 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.883+0000 7fca3642bb80 1 -- 172.21.15.31:0/28093051 shutdown_connections 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.883+0000 7fca3642bb80 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f9cf0 0x55e417809d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4176546b0 0x55e417807390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 --2- 172.21.15.31:0/28093051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4177f8df0 0x55e4177f2a40 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 -- 172.21.15.31:0/28093051 >> 172.21.15.31:0/28093051 conn(0x55e4177edfd0 msgr2=0x55e4177f32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 -- 172.21.15.31:0/28093051 shutdown_connections 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 -- 172.21.15.31:0/28093051 wait complete. 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 -- start start 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.884+0000 7fca3642bb80 1 Processor -- start 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 0x55e417807020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4178057b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4177f9cf0 0x55e417805cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417806230 con 0x55e4177f9cf0 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4178063a0 con 0x55e4177f8df0 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417806560 con 0x55e4176546b0 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca3642bb80 10 client.0 _ll_register_callbacks cb 0x55e4176ef5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.885+0000 7fca2099b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 0x55e417807020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 0x55e417807020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:46752/0 (socket says 172.21.15.31:46752) 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 -- 172.21.15.31:0/2258827684 learned_addr learned my addr 172.21.15.31:0/2258827684 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2119c700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4177f9cf0 0x55e417805cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca1bfff700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4178057b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 -- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 msgr2=0x55e4178057b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4178057b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 -- 172.21.15.31:0/2258827684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4177f9cf0 msgr2=0x55e417805cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4177f9cf0 0x55e417805cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca2099b700 1 -- 172.21.15.31:0/2258827684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c005da0 con 0x55e4176546b0 2023-07-29T16:40:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.886+0000 7fca1bfff700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4178057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:40:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fca2099b700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 0x55e417807020 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b4d0 tx=0x7fca1009b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=953bd15bbf6efd37 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fc9fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:40:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fc9fbfff700 1 -- 172.21.15.31:0/2258827684 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10096e60 con 0x55e4176546b0 2023-07-29T16:40:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fc9fbfff700 1 -- 172.21.15.31:0/2258827684 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fca100b1040 con 0x55e4176546b0 2023-07-29T16:40:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fca1400f040 con 0x55e4176546b0 2023-07-29T16:40:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fc9fbfff700 1 -- 172.21.15.31:0/2258827684 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fca100c8f70 con 0x55e4176546b0 2023-07-29T16:40:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fca3642bb80 10 client.30682 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55e41759e1f0 con 0x55e4176546b0 2023-07-29T16:40:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.887+0000 7fc9fbfff700 1 -- 172.21.15.31:0/2258827684 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca1400f040 con 0x55e4176546b0 2023-07-29T16:40:13.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fc9fbfff700 1 -- 172.21.15.31:0/2258827684 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fca10097a20 con 0x55e4176546b0 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca3642bb80 10 client.30682 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca3642bb80 10 client.30682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca3642bb80 20 client.30682 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e417860e60 0x55e417863270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca3642bb80 4 client.30682 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e417742ce0 con 0x55e417860e60 2023-07-29T16:40:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.888+0000 7fca1bfff700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e417860e60 0x55e417863270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.889+0000 7fca1bfff700 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e417860e60 0x55e417863270 crc :-1 s=READY pgs=506 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c829faf957ba4f6 server_cookie=f75f8a56656c1062 in_seq=0 out_seq=0 2023-07-29T16:40:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:13.889+0000 7fc9fbfff700 10 client.30682 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:14.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 374447 33.44 MB/sec execute 478 sec latency 340.166 ms 2023-07-29T16:40:14.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fc9fbfff700 1 -- 172.21.15.31:0/2258827684 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e417742ce0 con 0x55e417860e60 2023-07-29T16:40:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fc9fbfff700 10 client.30682 handle_command_reply: tid=0 2023-07-29T16:40:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 client.30682 shutdown 2023-07-29T16:40:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 msgr2=0x55e417807020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 0x55e417807020 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b4d0 tx=0x7fca1009b040 comp rx=0 tx=0).stop 2023-07-29T16:40:14.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 shutdown_connections 2023-07-29T16:40:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55e417860e60 0x55e417863270 crc :-1 s=READY pgs=506 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e4177f9cf0 0x55e417805cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e4176546b0 0x55e417807020 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 --2- 172.21.15.31:0/2258827684 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e4177f8df0 0x55e4178057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.479+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 >> 172.21.15.31:0/2258827684 conn(0x55e4177edfd0 msgr2=0x55e4177fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.480+0000 7fc9fbfff700 0 client.30682 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.480+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 shutdown_connections 2023-07-29T16:40:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.480+0000 7fca3642bb80 1 -- 172.21.15.31:0/2258827684 wait complete. 2023-07-29T16:40:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.480+0000 7fca3642bb80 20 client.30682 trim_cache size 0 max 16384 2023-07-29T16:40:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:40:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:40:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:40:14.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:40:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca040a0890 msgr2=0x7fca040a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca040a0890 0x7fca040a2d40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fca140060b0 tx=0x7fca14004320 comp rx=0 tx=0).stop 2023-07-29T16:40:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 msgr2=0x7fca1c075600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c075600 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fca1000b500 tx=0x7fca1000b810 comp rx=0 tx=0).stop 2023-07-29T16:40:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 shutdown_connections 2023-07-29T16:40:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fca040a0890 0x7fca040a2d40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca1c104160 0x7fca1c076a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca1c1025e0 0x7fca1c075600 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 --2- 172.21.15.31:0/2296711960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca1c1037d0 0x7fca1c075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:14.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.482+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 >> 172.21.15.31:0/2296711960 conn(0x7fca1c0fdc00 msgr2=0x7fca1c106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:14.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.483+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 shutdown_connections 2023-07-29T16:40:14.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:14.483+0000 7fca22bff700 1 -- 172.21.15.31:0/2296711960 wait complete. 2023-07-29T16:40:15.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:14 smithi031 ceph-mon[95504]: pgmap v1115: 97 pgs: 97 active+clean; 699 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.0 MiB/s wr, 517 op/s 2023-07-29T16:40:15.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:14 smithi039 ceph-mon[128858]: pgmap v1115: 97 pgs: 97 active+clean; 699 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.0 MiB/s wr, 517 op/s 2023-07-29T16:40:15.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:14 smithi156 ceph-mon[129245]: pgmap v1115: 97 pgs: 97 active+clean; 699 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.0 MiB/s wr, 517 op/s 2023-07-29T16:40:15.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 375138 33.46 MB/sec execute 479 sec latency 570.623 ms 2023-07-29T16:40:16.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 375917 33.49 MB/sec execute 480 sec latency 81.183 ms 2023-07-29T16:40:17.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:16 smithi031 ceph-mon[95504]: pgmap v1116: 97 pgs: 97 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 405 op/s 2023-07-29T16:40:17.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:16 smithi039 ceph-mon[128858]: pgmap v1116: 97 pgs: 97 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 405 op/s 2023-07-29T16:40:17.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:16 smithi156 ceph-mon[129245]: pgmap v1116: 97 pgs: 97 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 405 op/s 2023-07-29T16:40:17.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 376462 33.49 MB/sec execute 481 sec latency 153.178 ms 2023-07-29T16:40:18.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 376906 33.47 MB/sec execute 482 sec latency 95.941 ms 2023-07-29T16:40:18.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:18 smithi039 ceph-mon[128858]: pgmap v1117: 97 pgs: 97 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.7 MiB/s wr, 496 op/s 2023-07-29T16:40:18.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:18 smithi156 ceph-mon[129245]: pgmap v1117: 97 pgs: 97 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.7 MiB/s wr, 496 op/s 2023-07-29T16:40:18.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:18 smithi031 ceph-mon[95504]: pgmap v1117: 97 pgs: 97 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.7 MiB/s wr, 496 op/s 2023-07-29T16:40:19.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 377381 33.44 MB/sec execute 483 sec latency 84.556 ms 2023-07-29T16:40:20.398 INFO:tasks.workunit.client.0.smithi031.stdout: 10 377845 33.42 MB/sec execute 484 sec latency 124.424 ms 2023-07-29T16:40:21.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:20 smithi031 ceph-mon[95504]: pgmap v1118: 97 pgs: 97 active+clean; 698 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 559 op/s 2023-07-29T16:40:21.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:20 smithi039 ceph-mon[128858]: pgmap v1118: 97 pgs: 97 active+clean; 698 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 559 op/s 2023-07-29T16:40:21.329 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:20 smithi156 ceph-mon[129245]: pgmap v1118: 97 pgs: 97 active+clean; 698 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 559 op/s 2023-07-29T16:40:21.400 INFO:tasks.workunit.client.0.smithi031.stdout: 10 378249 33.39 MB/sec execute 485 sec latency 468.034 ms 2023-07-29T16:40:22.399 INFO:tasks.workunit.client.0.smithi031.stdout: 10 378313 33.33 MB/sec execute 486 sec latency 1374.610 ms 2023-07-29T16:40:23.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:22 smithi031 ceph-mon[95504]: pgmap v1119: 97 pgs: 97 active+clean; 714 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.8 MiB/s wr, 549 op/s 2023-07-29T16:40:23.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:22 smithi039 ceph-mon[128858]: pgmap v1119: 97 pgs: 97 active+clean; 714 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.8 MiB/s wr, 549 op/s 2023-07-29T16:40:23.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:22 smithi156 ceph-mon[129245]: pgmap v1119: 97 pgs: 97 active+clean; 714 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.8 MiB/s wr, 549 op/s 2023-07-29T16:40:23.399 INFO:tasks.workunit.client.0.smithi031.stdout: 10 378620 33.28 MB/sec execute 487 sec latency 315.607 ms 2023-07-29T16:40:24.400 INFO:tasks.workunit.client.0.smithi031.stdout: 10 378950 33.24 MB/sec execute 488 sec latency 374.715 ms 2023-07-29T16:40:25.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:24 smithi039 ceph-mon[128858]: pgmap v1120: 97 pgs: 97 active+clean; 760 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.8 MiB/s wr, 632 op/s 2023-07-29T16:40:25.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:24 smithi156 ceph-mon[129245]: pgmap v1120: 97 pgs: 97 active+clean; 760 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.8 MiB/s wr, 632 op/s 2023-07-29T16:40:25.400 INFO:tasks.workunit.client.0.smithi031.stdout: 10 379462 33.26 MB/sec execute 489 sec latency 87.324 ms 2023-07-29T16:40:25.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:24 smithi031 ceph-mon[95504]: pgmap v1120: 97 pgs: 97 active+clean; 760 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.8 MiB/s wr, 632 op/s 2023-07-29T16:40:26.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 380139 33.29 MB/sec execute 490 sec latency 72.367 ms 2023-07-29T16:40:27.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:26 smithi039 ceph-mon[128858]: pgmap v1121: 97 pgs: 97 active+clean; 762 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 750 op/s 2023-07-29T16:40:27.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:26 smithi156 ceph-mon[129245]: pgmap v1121: 97 pgs: 97 active+clean; 762 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 750 op/s 2023-07-29T16:40:27.400 INFO:tasks.workunit.client.0.smithi031.stdout: 10 380275 33.24 MB/sec execute 491 sec latency 602.690 ms 2023-07-29T16:40:27.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:26 smithi031 ceph-mon[95504]: pgmap v1121: 97 pgs: 97 active+clean; 762 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 13 MiB/s wr, 750 op/s 2023-07-29T16:40:27.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 -- 172.21.15.31:0/3738674995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568068e80 msgr2=0x7fe568102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3738674995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568068e80 0x7fe568102ed0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fe55800b0a0 tx=0x7fe55800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 -- 172.21.15.31:0/3738674995 shutdown_connections 2023-07-29T16:40:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3738674995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568105f20 0x7fe5681083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3738674995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568068e80 0x7fe568102ed0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3738674995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe568103410 0x7fe568105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.723+0000 7fe56dfff700 1 -- 172.21.15.31:0/3738674995 >> 172.21.15.31:0/3738674995 conn(0x7fe5680fa840 msgr2=0x7fe5680fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.726+0000 7fe56dfff700 1 -- 172.21.15.31:0/3738674995 shutdown_connections 2023-07-29T16:40:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.729+0000 7fe56dfff700 1 -- 172.21.15.31:0/3738674995 wait complete. 2023-07-29T16:40:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.730+0000 7fe56dfff700 1 Processor -- start 2023-07-29T16:40:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 -- start start 2023-07-29T16:40:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 0x7fe56806b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe568103410 0x7fe56806bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568105f20 0x7fe56806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56806d0b0 con 0x7fe568103410 2023-07-29T16:40:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56806d220 con 0x7fe568068e80 2023-07-29T16:40:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe56dfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56806d3e0 con 0x7fe568105f20 2023-07-29T16:40:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe5677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 0x7fe56806b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe5677fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 0x7fe56806b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49716/0 (socket says 172.21.15.31:49716) 2023-07-29T16:40:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.731+0000 7fe5677fe700 1 -- 172.21.15.31:0/3881933097 learned_addr learned my addr 172.21.15.31:0/3881933097 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe55effd700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe568103410 0x7fe56806bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe567fff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568105f20 0x7fe56806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe5677fe700 1 -- 172.21.15.31:0/3881933097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568105f20 msgr2=0x7fe56806ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe5677fe700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568105f20 0x7fe56806ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe5677fe700 1 -- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe568103410 msgr2=0x7fe56806bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe5677fe700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe568103410 0x7fe56806bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe5677fe700 1 -- 172.21.15.31:0/3881933097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558009d00 con 0x7fe568068e80 2023-07-29T16:40:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.732+0000 7fe567fff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568105f20 0x7fe56806ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:40:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.734+0000 7fe5677fe700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 0x7fe56806b640 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe55800b070 tx=0x7fe558007930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.734+0000 7fe5657fa700 1 -- 172.21.15.31:0/3881933097 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55800e040 con 0x7fe568068e80 2023-07-29T16:40:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.735+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56806d660 con 0x7fe568068e80 2023-07-29T16:40:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.735+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe568071e90 con 0x7fe568068e80 2023-07-29T16:40:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.735+0000 7fe5657fa700 1 -- 172.21.15.31:0/3881933097 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe558004c50 con 0x7fe568068e80 2023-07-29T16:40:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.735+0000 7fe5657fa700 1 -- 172.21.15.31:0/3881933097 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55801b6c0 con 0x7fe568068e80 2023-07-29T16:40:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.748+0000 7fe5819efb80 1 Processor -- start 2023-07-29T16:40:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.749+0000 7fe5657fa700 1 -- 172.21.15.31:0/3881933097 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fe55801d470 con 0x7fe568068e80 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.749+0000 7fe5819efb80 1 -- start start 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55836749e6b0 0x5583676538c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558367653e00 0x5583676547c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x558367655170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5819efb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836759a9b0 con 0x558367653e00 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5819efb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367598370 con 0x558367654d00 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5819efb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367593c60 con 0x55836749e6b0 2023-07-29T16:40:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe567fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55836749e6b0 0x5583676538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe55effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x558367655170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe55effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x558367655170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:49722/0 (socket says 172.21.15.31:49722) 2023-07-29T16:40:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe55effd700 1 -- 172.21.15.31:0/2695522620 learned_addr learned my addr 172.21.15.31:0/2695522620 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.750+0000 7fe5677fe700 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558367653e00 0x5583676547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.751+0000 7fe567fff700 1 -- 172.21.15.31:0/2695522620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 msgr2=0x558367655170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.751+0000 7fe567fff700 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x558367655170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.751+0000 7fe567fff700 1 -- 172.21.15.31:0/2695522620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558367653e00 msgr2=0x5583676547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.751+0000 7fe567fff700 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558367653e00 0x5583676547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.751+0000 7fe567fff700 1 -- 172.21.15.31:0/2695522620 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583673c1d80 con 0x55836749e6b0 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.752+0000 7fe567fff700 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55836749e6b0 0x5583676538c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe560009e50 tx=0x7fe5600116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31bfd86d0d350cd6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.752+0000 7fe5657fa700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5540a0860 0x7fe5540a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.752+0000 7fe5657fa700 1 -- 172.21.15.31:0/3881933097 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe5580ccac0 con 0x7fe568068e80 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.752+0000 7fe55dffb700 1 -- 172.21.15.31:0/2695522620 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560011e20 con 0x55836749e6b0 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.752+0000 7fe55dffb700 1 -- 172.21.15.31:0/2695522620 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe560005020 con 0x55836749e6b0 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.752+0000 7fe55dffb700 1 -- 172.21.15.31:0/2695522620 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560012670 con 0x55836749e6b0 2023-07-29T16:40:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.753+0000 7fe5677fe700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5540a0860 0x7fe5540a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.754+0000 7fe5819efb80 1 -- 172.21.15.31:0/2695522620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55836749e6b0 msgr2=0x5583676538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.754+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55836749e6b0 0x5583676538c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fe560009e50 tx=0x7fe5600116c0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.754+0000 7fe5677fe700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5540a0860 0x7fe5540a2d10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe558009fd0 tx=0x7fe5580099e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.756+0000 7fe5819efb80 1 -- 172.21.15.31:0/2695522620 shutdown_connections 2023-07-29T16:40:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.756+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x558367655170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.756+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55836749e6b0 0x5583676538c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.756+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2695522620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558367653e00 0x5583676547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.756+0000 7fe5819efb80 1 -- 172.21.15.31:0/2695522620 >> 172.21.15.31:0/2695522620 conn(0x558367637fd0 msgr2=0x55836763d300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.756+0000 7fe5819efb80 1 -- 172.21.15.31:0/2695522620 shutdown_connections 2023-07-29T16:40:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.757+0000 7fe5819efb80 1 -- 172.21.15.31:0/2695522620 wait complete. 2023-07-29T16:40:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.757+0000 7fe5819efb80 1 -- start start 2023-07-29T16:40:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.758+0000 7fe5819efb80 1 Processor -- start 2023-07-29T16:40:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x558367649b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367653e00 0x55836764a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x5583676510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583675304b0 con 0x55836749e6b0 2023-07-29T16:40:27.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367530150 con 0x558367654d00 2023-07-29T16:40:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583674178c0 con 0x558367653e00 2023-07-29T16:40:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe5819efb80 10 client.0 _ll_register_callbacks cb 0x5583675395c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.759+0000 7fe567fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x558367649b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x558367649b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58178/0 (socket says 172.21.15.31:58178) 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 -- 172.21.15.31:0/3177981295 learned_addr learned my addr 172.21.15.31:0/3177981295 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe55effd700 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367653e00 0x55836764a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 -- 172.21.15.31:0/3177981295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367653e00 msgr2=0x55836764a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367653e00 0x55836764a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 -- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 msgr2=0x5583676510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x5583676510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.760+0000 7fe567fff700 1 -- 172.21.15.31:0/3177981295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe560009a80 con 0x55836749e6b0 2023-07-29T16:40:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.761+0000 7fe567fff700 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x558367649b40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a010 tx=0x7fe560004ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a76347e23402c4 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.761+0000 7fe546ffd700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:40:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.761+0000 7fe546ffd700 1 -- 172.21.15.31:0/3177981295 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600055c0 con 0x55836749e6b0 2023-07-29T16:40:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.761+0000 7fe546ffd700 1 -- 172.21.15.31:0/3177981295 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe560012450 con 0x55836749e6b0 2023-07-29T16:40:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.762+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583675becb0 con 0x55836749e6b0 2023-07-29T16:40:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.762+0000 7fe546ffd700 1 -- 172.21.15.31:0/3177981295 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe560005720 con 0x55836749e6b0 2023-07-29T16:40:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.762+0000 7fe5819efb80 10 client.20379 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:27.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.762+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5583673aac00 con 0x55836749e6b0 2023-07-29T16:40:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.762+0000 7fe546ffd700 1 -- 172.21.15.31:0/3177981295 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5583675becb0 con 0x55836749e6b0 2023-07-29T16:40:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe546ffd700 1 -- 172.21.15.31:0/3177981295 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fe560024780 con 0x55836749e6b0 2023-07-29T16:40:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe5819efb80 10 client.20379 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe5819efb80 10 client.20379 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe5819efb80 20 client.20379 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:27.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe5819efb80 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558367657ce0 0x55836765a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe5819efb80 4 client.20379 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:40:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558367611d10 con 0x558367657ce0 2023-07-29T16:40:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.763+0000 7fe55effd700 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558367657ce0 0x55836765a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.765+0000 7fe55effd700 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558367657ce0 0x55836765a0c0 crc :-1 s=READY pgs=507 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e708f0375b855404 server_cookie=43a0b863cad1833a in_seq=0 out_seq=0 2023-07-29T16:40:27.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.765+0000 7fe546ffd700 10 client.20379 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.766+0000 7fe546ffd700 1 -- 172.21.15.31:0/3177981295 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558367611d10 con 0x558367657ce0 2023-07-29T16:40:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.766+0000 7fe546ffd700 10 client.20379 handle_command_reply: tid=0 2023-07-29T16:40:27.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.766+0000 7fe5819efb80 1 client.20379 shutdown 2023-07-29T16:40:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.772+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 msgr2=0x558367649b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.772+0000 7fe5819efb80 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x558367649b40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fe56000a010 tx=0x7fe560004ef0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.773+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 shutdown_connections 2023-07-29T16:40:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.773+0000 7fe5819efb80 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558367657ce0 0x55836765a0c0 crc :-1 s=READY pgs=507 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.773+0000 7fe5819efb80 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367654d00 0x5583676510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.773+0000 7fe5819efb80 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x558367649b40 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.773+0000 7fe5819efb80 1 --2- 172.21.15.31:0/3177981295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367653e00 0x55836764a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.773+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 >> 172.21.15.31:0/3177981295 conn(0x558367637fd0 msgr2=0x55836763ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.775+0000 7fe546ffd700 0 client.20379 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.775+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 shutdown_connections 2023-07-29T16:40:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.776+0000 7fe5819efb80 1 -- 172.21.15.31:0/3177981295 wait complete. 2023-07-29T16:40:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.776+0000 7fe5819efb80 20 client.20379 trim_cache size 0 max 16384 2023-07-29T16:40:27.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.805+0000 7fe5819efb80 1 Processor -- start 2023-07-29T16:40:27.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.805+0000 7fe5819efb80 1 -- start start 2023-07-29T16:40:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x5583676550d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x558367651080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x55836763ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe5819efb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367655640 con 0x55836749e6b0 2023-07-29T16:40:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe5819efb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367642550 con 0x558367653e00 2023-07-29T16:40:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe5819efb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55836765a650 con 0x558367654d00 2023-07-29T16:40:27.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe567fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x5583676550d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe567fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x5583676550d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58192/0 (socket says 172.21.15.31:58192) 2023-07-29T16:40:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe567fff700 1 -- 172.21.15.31:0/1044528054 learned_addr learned my addr 172.21.15.31:0/1044528054 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.806+0000 7fe567fff700 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x55836763ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.807+0000 7fe567fff700 1 -- 172.21.15.31:0/1044528054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 msgr2=0x55836763ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.807+0000 7fe567fff700 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x55836763ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.807+0000 7fe567fff700 1 -- 172.21.15.31:0/1044528054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 msgr2=0x558367651080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:40:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.807+0000 7fe567fff700 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x558367651080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.807+0000 7fe567fff700 1 -- 172.21.15.31:0/1044528054 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583673e81f0 con 0x55836749e6b0 2023-07-29T16:40:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.807+0000 7fe567fff700 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x5583676550d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe5600020c0 tx=0x7fe560014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436a4df5e2a07536 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.808+0000 7fe55e7fc700 1 -- 172.21.15.31:0/1044528054 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560024610 con 0x55836749e6b0 2023-07-29T16:40:27.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.808+0000 7fe55e7fc700 1 -- 172.21.15.31:0/1044528054 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe560005750 con 0x55836749e6b0 2023-07-29T16:40:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.808+0000 7fe55e7fc700 1 -- 172.21.15.31:0/1044528054 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe560023740 con 0x55836749e6b0 2023-07-29T16:40:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 -- 172.21.15.31:0/1044528054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 msgr2=0x5583676550d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x5583676550d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe5600020c0 tx=0x7fe560014040 comp rx=0 tx=0).stop 2023-07-29T16:40:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 -- 172.21.15.31:0/1044528054 shutdown_connections 2023-07-29T16:40:27.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x55836763ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x5583676550d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 --2- 172.21.15.31:0/1044528054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x558367651080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.809+0000 7fe5819efb80 1 -- 172.21.15.31:0/1044528054 >> 172.21.15.31:0/1044528054 conn(0x558367637fd0 msgr2=0x55836763d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.811+0000 7fe5819efb80 1 -- 172.21.15.31:0/1044528054 shutdown_connections 2023-07-29T16:40:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.811+0000 7fe5819efb80 1 -- 172.21.15.31:0/1044528054 wait complete. 2023-07-29T16:40:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.811+0000 7fe5819efb80 1 -- start start 2023-07-29T16:40:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.812+0000 7fe5819efb80 1 Processor -- start 2023-07-29T16:40:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x55836764d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x55836764d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x558367650590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367650ad0 con 0x55836749e6b0 2023-07-29T16:40:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367650c40 con 0x558367653e00 2023-07-29T16:40:27.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558367650db0 con 0x558367654d00 2023-07-29T16:40:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe5819efb80 10 client.0 _ll_register_callbacks cb 0x5583675395c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.813+0000 7fe567fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x558367650590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x558367650590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:50316/0 (socket says 172.21.15.31:50316) 2023-07-29T16:40:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 -- 172.21.15.31:0/2161215648 learned_addr learned my addr 172.21.15.31:0/2161215648 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:27.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe55effd700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x55836764d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x55836764d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 -- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 msgr2=0x55836764d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x55836764d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 -- 172.21.15.31:0/2161215648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 msgr2=0x55836764d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x55836764d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.814+0000 7fe567fff700 1 -- 172.21.15.31:0/2161215648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583673aac00 con 0x558367654d00 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.815+0000 7fe567fff700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x558367650590 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe560011d00 tx=0x7fe5600097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d208d769bcccea93 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.815+0000 7fe546ffd700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.815+0000 7fe546ffd700 1 -- 172.21.15.31:0/2161215648 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600234e0 con 0x558367654d00 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.815+0000 7fe546ffd700 1 -- 172.21.15.31:0/2161215648 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fe560023640 con 0x558367654d00 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.815+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe560005020 con 0x558367654d00 2023-07-29T16:40:27.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.816+0000 7fe546ffd700 1 -- 172.21.15.31:0/2161215648 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe560023800 con 0x558367654d00 2023-07-29T16:40:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.817+0000 7fe5819efb80 10 client.30700 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.817+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x5583673c0250 con 0x558367654d00 2023-07-29T16:40:27.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe546ffd700 1 -- 172.21.15.31:0/2161215648 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe560005020 con 0x558367654d00 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe546ffd700 1 -- 172.21.15.31:0/2161215648 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fe560012970 con 0x558367654d00 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe5819efb80 10 client.30700 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe5819efb80 10 client.30700 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe5819efb80 20 client.30700 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5583676aa230 0x5583676ac610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe5819efb80 4 client.30700 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:40:27.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.818+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55836758cce0 con 0x5583676aa230 2023-07-29T16:40:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.819+0000 7fe55effd700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5583676aa230 0x5583676ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.819+0000 7fe55effd700 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5583676aa230 0x5583676ac610 crc :-1 s=READY pgs=508 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=214665f6a487422f server_cookie=a2c8956b3273a618 in_seq=0 out_seq=0 2023-07-29T16:40:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:27.819+0000 7fe546ffd700 10 client.30700 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.285+0000 7fe546ffd700 1 -- 172.21.15.31:0/2161215648 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55836758cce0 con 0x5583676aa230 2023-07-29T16:40:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.285+0000 7fe546ffd700 10 client.30700 handle_command_reply: tid=0 2023-07-29T16:40:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.285+0000 7fe5819efb80 1 client.30700 shutdown 2023-07-29T16:40:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 msgr2=0x558367650590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x558367650590 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe560011d00 tx=0x7fe5600097d0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 shutdown_connections 2023-07-29T16:40:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x5583676aa230 0x5583676ac610 crc :-1 s=READY pgs=508 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558367654d00 0x558367650590 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55836749e6b0 0x55836764d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 --2- 172.21.15.31:0/2161215648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558367653e00 0x55836764d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.287+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 >> 172.21.15.31:0/2161215648 conn(0x558367637fd0 msgr2=0x5583676498f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.288+0000 7fe546ffd700 0 client.30700 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.288+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 shutdown_connections 2023-07-29T16:40:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.291+0000 7fe5819efb80 1 -- 172.21.15.31:0/2161215648 wait complete. 2023-07-29T16:40:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.291+0000 7fe5819efb80 20 client.30700 trim_cache size 0 max 16384 2023-07-29T16:40:28.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:40:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:40:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:40:28.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:40:28.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5540a0860 msgr2=0x7fe5540a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5540a0860 0x7fe5540a2d10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe558009fd0 tx=0x7fe5580099e0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 msgr2=0x7fe56806b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 0x7fe56806b640 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe55800b070 tx=0x7fe558007930 comp rx=0 tx=0).stop 2023-07-29T16:40:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 shutdown_connections 2023-07-29T16:40:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fe5540a0860 0x7fe5540a2d10 secure :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe558009fd0 tx=0x7fe5580099e0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe568105f20 0x7fe56806ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe568068e80 0x7fe56806b640 secure :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe55800b070 tx=0x7fe558007930 comp rx=0 tx=0).stop 2023-07-29T16:40:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 --2- 172.21.15.31:0/3881933097 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe568103410 0x7fe56806bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.308+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 >> 172.21.15.31:0/3881933097 conn(0x7fe5680fa840 msgr2=0x7fe568106cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.320+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 shutdown_connections 2023-07-29T16:40:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:28.321+0000 7fe56dfff700 1 -- 172.21.15.31:0/3881933097 wait complete. 2023-07-29T16:40:28.400 INFO:tasks.workunit.client.0.smithi031.stdout: 10 380506 33.21 MB/sec execute 492 sec latency 203.672 ms 2023-07-29T16:40:28.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:28 smithi156 ceph-mon[129245]: pgmap v1122: 97 pgs: 97 active+clean; 727 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 14 MiB/s wr, 760 op/s 2023-07-29T16:40:28.690 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:28 smithi031 ceph-mon[95504]: pgmap v1122: 97 pgs: 97 active+clean; 727 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 14 MiB/s wr, 760 op/s 2023-07-29T16:40:28.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:28 smithi039 ceph-mon[128858]: pgmap v1122: 97 pgs: 97 active+clean; 727 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 14 MiB/s wr, 760 op/s 2023-07-29T16:40:29.400 INFO:tasks.workunit.client.0.smithi031.stdout: 10 380917 33.19 MB/sec execute 493 sec latency 120.771 ms 2023-07-29T16:40:29.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:29 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:40:30.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:29 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:40:30.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:29 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:40:30.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 381445 33.16 MB/sec execute 494 sec latency 87.217 ms 2023-07-29T16:40:30.612 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:30 smithi031 ceph-mon[95504]: pgmap v1123: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 15 MiB/s wr, 731 op/s 2023-07-29T16:40:31.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:30 smithi039 ceph-mon[128858]: pgmap v1123: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 15 MiB/s wr, 731 op/s 2023-07-29T16:40:31.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:30 smithi156 ceph-mon[129245]: pgmap v1123: 97 pgs: 97 active+clean; 684 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 15 MiB/s wr, 731 op/s 2023-07-29T16:40:31.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 382041 33.17 MB/sec execute 495 sec latency 66.130 ms 2023-07-29T16:40:31.938 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:40:31.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:40:31.939 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:40:32.028 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:40:32.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:40:32.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:40:32.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:40:32.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:40:32.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:40:32.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 382871 33.19 MB/sec execute 496 sec latency 66.364 ms 2023-07-29T16:40:32.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:32 smithi031 ceph-mon[95504]: pgmap v1124: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 17 MiB/s wr, 797 op/s 2023-07-29T16:40:33.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:32 smithi039 ceph-mon[128858]: pgmap v1124: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 17 MiB/s wr, 797 op/s 2023-07-29T16:40:33.077 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:32 smithi156 ceph-mon[129245]: pgmap v1124: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 17 MiB/s wr, 797 op/s 2023-07-29T16:40:33.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 383543 33.21 MB/sec execute 497 sec latency 77.597 ms 2023-07-29T16:40:34.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 384247 33.20 MB/sec execute 498 sec latency 105.506 ms 2023-07-29T16:40:35.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:34 smithi031 ceph-mon[95504]: pgmap v1125: 97 pgs: 97 active+clean; 652 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 16 MiB/s wr, 825 op/s 2023-07-29T16:40:35.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:34 smithi039 ceph-mon[128858]: pgmap v1125: 97 pgs: 97 active+clean; 652 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 16 MiB/s wr, 825 op/s 2023-07-29T16:40:35.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:34 smithi156 ceph-mon[129245]: pgmap v1125: 97 pgs: 97 active+clean; 652 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 16 MiB/s wr, 825 op/s 2023-07-29T16:40:35.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 384902 33.22 MB/sec execute 499 sec latency 71.686 ms 2023-07-29T16:40:36.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 385561 33.23 MB/sec execute 500 sec latency 89.462 ms 2023-07-29T16:40:37.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:36 smithi031 ceph-mon[95504]: pgmap v1126: 97 pgs: 97 active+clean; 659 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 10 MiB/s wr, 721 op/s 2023-07-29T16:40:37.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:36 smithi039 ceph-mon[128858]: pgmap v1126: 97 pgs: 97 active+clean; 659 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 10 MiB/s wr, 721 op/s 2023-07-29T16:40:37.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:36 smithi156 ceph-mon[129245]: pgmap v1126: 97 pgs: 97 active+clean; 659 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 10 MiB/s wr, 721 op/s 2023-07-29T16:40:37.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 386147 33.22 MB/sec execute 501 sec latency 95.474 ms 2023-07-29T16:40:38.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 386791 33.23 MB/sec execute 502 sec latency 110.863 ms 2023-07-29T16:40:38.528 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:38 smithi039 ceph-mon[128858]: pgmap v1127: 97 pgs: 97 active+clean; 664 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.8 MiB/s wr, 614 op/s 2023-07-29T16:40:38.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:38 smithi156 ceph-mon[129245]: pgmap v1127: 97 pgs: 97 active+clean; 664 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.8 MiB/s wr, 614 op/s 2023-07-29T16:40:38.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:38 smithi031 ceph-mon[95504]: pgmap v1127: 97 pgs: 97 active+clean; 664 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.8 MiB/s wr, 614 op/s 2023-07-29T16:40:39.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 387403 33.22 MB/sec execute 503 sec latency 74.808 ms 2023-07-29T16:40:40.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 388049 33.22 MB/sec execute 504 sec latency 82.993 ms 2023-07-29T16:40:41.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:40 smithi031 ceph-mon[95504]: pgmap v1128: 97 pgs: 97 active+clean; 665 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.0 MiB/s wr, 578 op/s 2023-07-29T16:40:41.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:40 smithi039 ceph-mon[128858]: pgmap v1128: 97 pgs: 97 active+clean; 665 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.0 MiB/s wr, 578 op/s 2023-07-29T16:40:41.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:40 smithi156 ceph-mon[129245]: pgmap v1128: 97 pgs: 97 active+clean; 665 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 6.0 MiB/s wr, 578 op/s 2023-07-29T16:40:41.403 INFO:tasks.workunit.client.0.smithi031.stdout: 10 388740 33.22 MB/sec execute 505 sec latency 109.224 ms 2023-07-29T16:40:41.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 -- 172.21.15.31:0/1346915658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd860103f70 msgr2=0x7fd8601043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 --2- 172.21.15.31:0/1346915658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd860103f70 0x7fd8601043f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd858009bc0 tx=0x7fd858009ed0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 -- 172.21.15.31:0/1346915658 shutdown_connections 2023-07-29T16:40:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 --2- 172.21.15.31:0/1346915658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd860103f70 0x7fd8601043f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 --2- 172.21.15.31:0/1346915658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8600fec50 0x7fd860101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 --2- 172.21.15.31:0/1346915658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd8601015b0 0x7fd860103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.556+0000 7fd8685ff700 1 -- 172.21.15.31:0/1346915658 >> 172.21.15.31:0/1346915658 conn(0x7fd8600fa840 msgr2=0x7fd8600fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.557+0000 7fd8685ff700 1 -- 172.21.15.31:0/1346915658 shutdown_connections 2023-07-29T16:40:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.557+0000 7fd8685ff700 1 -- 172.21.15.31:0/1346915658 wait complete. 2023-07-29T16:40:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.558+0000 7fd8685ff700 1 Processor -- start 2023-07-29T16:40:41.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.558+0000 7fd8685ff700 1 -- start start 2023-07-29T16:40:41.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd8685ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 0x7fd86010a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd8685ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8601015b0 0x7fd86010a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd86639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 0x7fd86010a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd8685ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860103f70 0x7fd86010b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd8685ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010bbc0 con 0x7fd8601015b0 2023-07-29T16:40:41.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd8685ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010bd30 con 0x7fd860103f70 2023-07-29T16:40:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd8685ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010bef0 con 0x7fd8600fec50 2023-07-29T16:40:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd86639b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 0x7fd86010a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53662/0 (socket says 172.21.15.31:53662) 2023-07-29T16:40:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.559+0000 7fd86639b700 1 -- 172.21.15.31:0/427334761 learned_addr learned my addr 172.21.15.31:0/427334761 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd866b9c700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860103f70 0x7fd86010b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd86639b700 1 -- 172.21.15.31:0/427334761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860103f70 msgr2=0x7fd86010b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd86639b700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860103f70 0x7fd86010b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd86639b700 1 -- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8601015b0 msgr2=0x7fd86010a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd86639b700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8601015b0 0x7fd86010a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd86639b700 1 -- 172.21.15.31:0/427334761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd858009820 con 0x7fd8600fec50 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd866b9c700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860103f70 0x7fd86010b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.560+0000 7fd86639b700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 0x7fd86010a150 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00b9e0 tx=0x7fd85c00bda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:41.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.561+0000 7fd8577fe700 1 -- 172.21.15.31:0/427334761 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00ca30 con 0x7fd8600fec50 2023-07-29T16:40:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.561+0000 7fd8577fe700 1 -- 172.21.15.31:0/427334761 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd85c00cb90 con 0x7fd8600fec50 2023-07-29T16:40:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.561+0000 7fd8577fe700 1 -- 172.21.15.31:0/427334761 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0125a0 con 0x7fd8600fec50 2023-07-29T16:40:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.561+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86010c1d0 con 0x7fd8600fec50 2023-07-29T16:40:41.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.561+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86010ca00 con 0x7fd8600fec50 2023-07-29T16:40:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.567+0000 7fd8577fe700 1 -- 172.21.15.31:0/427334761 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd85c00cd00 con 0x7fd8600fec50 2023-07-29T16:40:41.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.567+0000 7fd87be44b80 1 Processor -- start 2023-07-29T16:40:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 -- start start 2023-07-29T16:40:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 0x55d5351978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535197df0 0x55d5351987b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d535199160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5350ef9b0 con 0x55d535197df0 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5350ed370 con 0x55d535198cf0 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.568+0000 7fd87be44b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5350e8c60 con 0x55d534ff36b0 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 0x55d5351978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 0x55d5351978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:53676/0 (socket says 172.21.15.31:53676) 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 -- 172.21.15.31:0/1733229815 learned_addr learned my addr 172.21.15.31:0/1733229815 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd865b9a700 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d535199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 -- 172.21.15.31:0/1733229815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 msgr2=0x55d535199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d535199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 -- 172.21.15.31:0/1733229815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535197df0 msgr2=0x55d5351987b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:40:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535197df0 0x55d5351987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.569+0000 7fd866b9c700 1 -- 172.21.15.31:0/1733229815 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d534f16d80 con 0x55d534ff36b0 2023-07-29T16:40:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.570+0000 7fd8577fe700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd84c0a08e0 0x7fd84c0a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.570+0000 7fd8577fe700 1 -- 172.21.15.31:0/427334761 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd85c0cb230 con 0x7fd8600fec50 2023-07-29T16:40:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.571+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 0x55d5351978b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd8580057b0 tx=0x7fd8580057e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78ea710c6b9fd763 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.571+0000 7fd86639b700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd84c0a08e0 0x7fd84c0a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.571+0000 7fd854bfe700 1 -- 172.21.15.31:0/1733229815 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85800bb70 con 0x55d534ff36b0 2023-07-29T16:40:41.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.571+0000 7fd854bfe700 1 -- 172.21.15.31:0/1733229815 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd85800f920 con 0x55d534ff36b0 2023-07-29T16:40:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.571+0000 7fd854bfe700 1 -- 172.21.15.31:0/1733229815 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8580177a0 con 0x55d534ff36b0 2023-07-29T16:40:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.573+0000 7fd87be44b80 1 -- 172.21.15.31:0/1733229815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 msgr2=0x55d5351978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.573+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 0x55d5351978b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd8580057b0 tx=0x7fd8580057e0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.573+0000 7fd86639b700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd84c0a08e0 0x7fd84c0a2d90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0953d0 tx=0x7fd85c015f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 -- 172.21.15.31:0/1733229815 shutdown_connections 2023-07-29T16:40:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d535199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d534ff36b0 0x55d5351978b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1733229815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535197df0 0x55d5351987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 -- 172.21.15.31:0/1733229815 >> 172.21.15.31:0/1733229815 conn(0x55d53518cfd0 msgr2=0x55d535192300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:41.578 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 -- 172.21.15.31:0/1733229815 shutdown_connections 2023-07-29T16:40:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.574+0000 7fd87be44b80 1 -- 172.21.15.31:0/1733229815 wait complete. 2023-07-29T16:40:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.575+0000 7fd87be44b80 1 -- start start 2023-07-29T16:40:41.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.575+0000 7fd87be44b80 1 Processor -- start 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 0x55d5351a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d5351a8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5350854b0 con 0x55d534ff36b0 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d535085150 con 0x55d535198cf0 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d534f6c8c0 con 0x55d535197df0 2023-07-29T16:40:41.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd87be44b80 10 client.0 _ll_register_callbacks cb 0x55d53508e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 0x55d5351a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 0x55d5351a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40184/0 (socket says 172.21.15.31:40184) 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.577+0000 7fd866b9c700 1 -- 172.21.15.31:0/2721894519 learned_addr learned my addr 172.21.15.31:0/2721894519 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd865b9a700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd866b9c700 1 -- 172.21.15.31:0/2721894519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 msgr2=0x55d5351a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd866b9c700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd866b9c700 1 -- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 msgr2=0x55d5351a8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd866b9c700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d5351a8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd866b9c700 1 -- 172.21.15.31:0/2721894519 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860104ee0 con 0x55d534ff36b0 2023-07-29T16:40:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.578+0000 7fd865b9a700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:40:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd866b9c700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 0x55d5351a0e90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd858009540 tx=0x7fd85800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b51694b1e6a0d865 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:41.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd849ffb700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:40:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd849ffb700 1 -- 172.21.15.31:0/2721894519 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858017520 con 0x55d534ff36b0 2023-07-29T16:40:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd849ffb700 1 -- 172.21.15.31:0/2721894519 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd858017680 con 0x55d534ff36b0 2023-07-29T16:40:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d535113cb0 con 0x55d534ff36b0 2023-07-29T16:40:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd849ffb700 1 -- 172.21.15.31:0/2721894519 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd858017840 con 0x55d534ff36b0 2023-07-29T16:40:41.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd87be44b80 10 client.20388 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.579+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55d534effc00 con 0x55d534ff36b0 2023-07-29T16:40:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd849ffb700 1 -- 172.21.15.31:0/2721894519 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d535113cb0 con 0x55d534ff36b0 2023-07-29T16:40:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd849ffb700 1 -- 172.21.15.31:0/2721894519 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd858015070 con 0x55d534ff36b0 2023-07-29T16:40:41.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd87be44b80 10 client.20388 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd87be44b80 10 client.20388 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd87be44b80 20 client.20388 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:41.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd87be44b80 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5351bd6e0 0x55d5351bfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd87be44b80 4 client.20388 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:40:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.580+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d535166d10 con 0x55d5351bd6e0 2023-07-29T16:40:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.581+0000 7fd865b9a700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5351bd6e0 0x55d5351bfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.582+0000 7fd849ffb700 10 client.20388 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:41.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.582+0000 7fd865b9a700 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5351bd6e0 0x55d5351bfaf0 crc :-1 s=READY pgs=509 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd345572f8de208 server_cookie=adc645bd31029f12 in_seq=0 out_seq=0 2023-07-29T16:40:41.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.590+0000 7fd849ffb700 1 -- 172.21.15.31:0/2721894519 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55d535166d10 con 0x55d5351bd6e0 2023-07-29T16:40:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.590+0000 7fd849ffb700 10 client.20388 handle_command_reply: tid=0 2023-07-29T16:40:41.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 client.20388 shutdown 2023-07-29T16:40:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 msgr2=0x55d5351a0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 0x55d5351a0e90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd858009540 tx=0x7fd85800f690 comp rx=0 tx=0).stop 2023-07-29T16:40:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 shutdown_connections 2023-07-29T16:40:41.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d5351bd6e0 0x55d5351bfaf0 crc :-1 s=READY pgs=509 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d535198cf0 0x55d5351a8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d534ff36b0 0x55d5351a0e90 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 --2- 172.21.15.31:0/2721894519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.591+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 >> 172.21.15.31:0/2721894519 conn(0x55d53518cfd0 msgr2=0x55d535191a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.592+0000 7fd849ffb700 0 client.20388 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.592+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 shutdown_connections 2023-07-29T16:40:41.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.592+0000 7fd87be44b80 1 -- 172.21.15.31:0/2721894519 wait complete. 2023-07-29T16:40:41.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.592+0000 7fd87be44b80 20 client.20388 trim_cache size 0 max 16384 2023-07-29T16:40:41.619 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.616+0000 7fd87be44b80 1 Processor -- start 2023-07-29T16:40:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 -- start start 2023-07-29T16:40:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d53519f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 0x55d53519bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5351a2080 con 0x55d535198cf0 2023-07-29T16:40:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5351a0340 con 0x55d534ff36b0 2023-07-29T16:40:41.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.617+0000 7fd87be44b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5351a0a00 con 0x55d535197df0 2023-07-29T16:40:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.618+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d53519f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.618+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d53519f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55620/0 (socket says 172.21.15.31:55620) 2023-07-29T16:40:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.618+0000 7fd866b9c700 1 -- 172.21.15.31:0/1499440619 learned_addr learned my addr 172.21.15.31:0/1499440619 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:41.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd866b9c700 1 -- 172.21.15.31:0/1499440619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 msgr2=0x55d5351a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd866b9c700 1 -- 172.21.15.31:0/1499440619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 msgr2=0x55d53519bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 0x55d53519bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd866b9c700 1 -- 172.21.15.31:0/1499440619 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d534f3d1f0 con 0x55d534ff36b0 2023-07-29T16:40:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d53519f070 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b540 tx=0x7fd858009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31cd4c0a737dd2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd8553ff700 1 -- 172.21.15.31:0/1499440619 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858004ae0 con 0x55d534ff36b0 2023-07-29T16:40:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.619+0000 7fd8553ff700 1 -- 172.21.15.31:0/1499440619 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd858005320 con 0x55d534ff36b0 2023-07-29T16:40:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.620+0000 7fd8553ff700 1 -- 172.21.15.31:0/1499440619 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85800eea0 con 0x55d534ff36b0 2023-07-29T16:40:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 -- 172.21.15.31:0/1499440619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 msgr2=0x55d53519f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d53519f070 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fd85800b540 tx=0x7fd858009ed0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 -- 172.21.15.31:0/1499440619 shutdown_connections 2023-07-29T16:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 0x55d53519bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d53519f070 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1499440619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.621+0000 7fd87be44b80 1 -- 172.21.15.31:0/1499440619 >> 172.21.15.31:0/1499440619 conn(0x55d53518cfd0 msgr2=0x55d5351bf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.623+0000 7fd87be44b80 1 -- 172.21.15.31:0/1499440619 shutdown_connections 2023-07-29T16:40:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.623+0000 7fd87be44b80 1 -- 172.21.15.31:0/1499440619 wait complete. 2023-07-29T16:40:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.624+0000 7fd87be44b80 1 -- start start 2023-07-29T16:40:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.624+0000 7fd87be44b80 1 Processor -- start 2023-07-29T16:40:41.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d5351a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 0x55d5351a7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5351a7d50 con 0x55d535198cf0 2023-07-29T16:40:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5351a7ec0 con 0x55d534ff36b0 2023-07-29T16:40:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5351a8080 con 0x55d535197df0 2023-07-29T16:40:41.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd87be44b80 10 client.0 _ll_register_callbacks cb 0x55d53508e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:41.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d5351a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd866b9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d5351a42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:55624/0 (socket says 172.21.15.31:55624) 2023-07-29T16:40:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.625+0000 7fd866b9c700 1 -- 172.21.15.31:0/1594673700 learned_addr learned my addr 172.21.15.31:0/1594673700 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd865b9a700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd866b9c700 1 -- 172.21.15.31:0/1594673700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 msgr2=0x55d5351a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd866b9c700 1 -- 172.21.15.31:0/1594673700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 msgr2=0x55d5351a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 0x55d5351a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd866b9c700 1 -- 172.21.15.31:0/1594673700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d534effc00 con 0x55d534ff36b0 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd865b9a700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.626+0000 7fd866b9c700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d5351a42b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd858006010 tx=0x7fd85800ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d917053b8873d54 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.628+0000 7fd849ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.628+0000 7fd849ffb700 1 -- 172.21.15.31:0/1594673700 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858004ae0 con 0x55d534ff36b0 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.628+0000 7fd849ffb700 1 -- 172.21.15.31:0/1594673700 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd858004c40 con 0x55d534ff36b0 2023-07-29T16:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.628+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd85800f920 con 0x55d534ff36b0 2023-07-29T16:40:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.629+0000 7fd849ffb700 1 -- 172.21.15.31:0/1594673700 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd858004e00 con 0x55d534ff36b0 2023-07-29T16:40:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.629+0000 7fd87be44b80 10 client.30680 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.629+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55d534f15250 con 0x55d534ff36b0 2023-07-29T16:40:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd849ffb700 1 -- 172.21.15.31:0/1594673700 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd85800f920 con 0x55d534ff36b0 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd849ffb700 1 -- 172.21.15.31:0/1594673700 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd858068070 con 0x55d534ff36b0 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd87be44b80 10 client.30680 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd87be44b80 10 client.30680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd87be44b80 20 client.30680 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d53520fd00 0x55d5352120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd87be44b80 4 client.30680 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:40:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d5350e1ce0 con 0x55d53520fd00 2023-07-29T16:40:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.630+0000 7fd865b9a700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d53520fd00 0x55d5352120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.634+0000 7fd865b9a700 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d53520fd00 0x55d5352120e0 crc :-1 s=READY pgs=510 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c08a551871ae9a7 server_cookie=f5123712a976305d in_seq=0 out_seq=0 2023-07-29T16:40:41.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:41.634+0000 7fd849ffb700 10 client.30680 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:42.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.109+0000 7fd849ffb700 1 -- 172.21.15.31:0/1594673700 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d5350e1ce0 con 0x55d53520fd00 2023-07-29T16:40:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.109+0000 7fd849ffb700 10 client.30680 handle_command_reply: tid=0 2023-07-29T16:40:42.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.109+0000 7fd87be44b80 1 client.30680 shutdown 2023-07-29T16:40:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 msgr2=0x55d5351a42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d5351a42b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd858006010 tx=0x7fd85800ef70 comp rx=0 tx=0).stop 2023-07-29T16:40:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 shutdown_connections 2023-07-29T16:40:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55d53520fd00 0x55d5352120e0 crc :-1 s=READY pgs=510 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d535198cf0 0x55d5351a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d534ff36b0 0x55d5351a42b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 --2- 172.21.15.31:0/1594673700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d535197df0 0x55d5351a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 >> 172.21.15.31:0/1594673700 conn(0x55d53518cfd0 msgr2=0x55d53519f070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.110+0000 7fd849ffb700 0 client.30680 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.111+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 shutdown_connections 2023-07-29T16:40:42.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.111+0000 7fd87be44b80 1 -- 172.21.15.31:0/1594673700 wait complete. 2023-07-29T16:40:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.112+0000 7fd87be44b80 20 client.30680 trim_cache size 0 max 16384 2023-07-29T16:40:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:40:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:40:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:40:42.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:40:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd84c0a08e0 msgr2=0x7fd84c0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd84c0a08e0 0x7fd84c0a2d90 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0953d0 tx=0x7fd85c015f90 comp rx=0 tx=0).stop 2023-07-29T16:40:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 msgr2=0x7fd86010a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 0x7fd86010a150 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00b9e0 tx=0x7fd85c00bda0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 shutdown_connections 2023-07-29T16:40:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd84c0a08e0 0x7fd84c0a2d90 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd860103f70 0x7fd86010b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8600fec50 0x7fd86010a150 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 --2- 172.21.15.31:0/427334761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8601015b0 0x7fd86010a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:42.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.113+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 >> 172.21.15.31:0/427334761 conn(0x7fd8600fa840 msgr2=0x7fd8600ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:42.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.114+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 shutdown_connections 2023-07-29T16:40:42.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:42.115+0000 7fd8685ff700 1 -- 172.21.15.31:0/427334761 wait complete. 2023-07-29T16:40:42.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 388988 33.19 MB/sec execute 506 sec latency 628.166 ms 2023-07-29T16:40:43.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:42 smithi039 ceph-mon[128858]: pgmap v1129: 97 pgs: 97 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.2 MiB/s wr, 522 op/s 2023-07-29T16:40:43.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:42 smithi156 ceph-mon[129245]: pgmap v1129: 97 pgs: 97 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.2 MiB/s wr, 522 op/s 2023-07-29T16:40:43.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 389655 33.21 MB/sec execute 507 sec latency 114.214 ms 2023-07-29T16:40:43.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:42 smithi031 ceph-mon[95504]: pgmap v1129: 97 pgs: 97 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.2 MiB/s wr, 522 op/s 2023-07-29T16:40:44.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 390185 33.19 MB/sec execute 508 sec latency 76.213 ms 2023-07-29T16:40:45.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:44 smithi039 ceph-mon[128858]: pgmap v1130: 97 pgs: 97 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2023-07-29T16:40:45.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:44 smithi156 ceph-mon[129245]: pgmap v1130: 97 pgs: 97 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2023-07-29T16:40:45.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 390858 33.20 MB/sec execute 509 sec latency 43.654 ms 2023-07-29T16:40:45.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:44 smithi031 ceph-mon[95504]: pgmap v1130: 97 pgs: 97 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2023-07-29T16:40:46.401 INFO:tasks.workunit.client.0.smithi031.stdout: 10 391587 33.20 MB/sec execute 510 sec latency 60.422 ms 2023-07-29T16:40:47.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:46 smithi039 ceph-mon[128858]: pgmap v1131: 97 pgs: 97 active+clean; 675 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.6 MiB/s wr, 518 op/s 2023-07-29T16:40:47.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:46 smithi156 ceph-mon[129245]: pgmap v1131: 97 pgs: 97 active+clean; 675 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.6 MiB/s wr, 518 op/s 2023-07-29T16:40:47.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 392372 33.22 MB/sec execute 511 sec latency 134.401 ms 2023-07-29T16:40:47.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:46 smithi031 ceph-mon[95504]: pgmap v1131: 97 pgs: 97 active+clean; 675 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.6 MiB/s wr, 518 op/s 2023-07-29T16:40:48.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 392690 33.20 MB/sec execute 512 sec latency 280.542 ms 2023-07-29T16:40:48.580 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:48 smithi156 ceph-mon[129245]: pgmap v1132: 97 pgs: 97 active+clean; 676 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 464 op/s 2023-07-29T16:40:48.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:48 smithi031 ceph-mon[95504]: pgmap v1132: 97 pgs: 97 active+clean; 676 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 464 op/s 2023-07-29T16:40:48.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:48 smithi039 ceph-mon[128858]: pgmap v1132: 97 pgs: 97 active+clean; 676 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.3 MiB/s wr, 464 op/s 2023-07-29T16:40:49.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 393232 33.19 MB/sec execute 513 sec latency 95.112 ms 2023-07-29T16:40:50.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 393875 33.18 MB/sec execute 514 sec latency 71.924 ms 2023-07-29T16:40:51.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:50 smithi031 ceph-mon[95504]: pgmap v1133: 97 pgs: 97 active+clean; 689 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 548 op/s 2023-07-29T16:40:51.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:50 smithi039 ceph-mon[128858]: pgmap v1133: 97 pgs: 97 active+clean; 689 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 548 op/s 2023-07-29T16:40:51.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:50 smithi156 ceph-mon[129245]: pgmap v1133: 97 pgs: 97 active+clean; 689 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 548 op/s 2023-07-29T16:40:51.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 394712 33.20 MB/sec execute 515 sec latency 78.577 ms 2023-07-29T16:40:52.402 INFO:tasks.workunit.client.0.smithi031.stdout: 10 395451 33.22 MB/sec execute 516 sec latency 50.561 ms 2023-07-29T16:40:53.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:52 smithi039 ceph-mon[128858]: pgmap v1134: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 521 op/s 2023-07-29T16:40:53.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:52 smithi156 ceph-mon[129245]: pgmap v1134: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 521 op/s 2023-07-29T16:40:53.403 INFO:tasks.workunit.client.0.smithi031.stdout: 10 395682 33.19 MB/sec execute 517 sec latency 659.861 ms 2023-07-29T16:40:53.445 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:52 smithi031 ceph-mon[95504]: pgmap v1134: 97 pgs: 97 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 521 op/s 2023-07-29T16:40:54.403 INFO:tasks.workunit.client.0.smithi031.stdout: 10 395808 33.14 MB/sec execute 518 sec latency 1250.331 ms 2023-07-29T16:40:55.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:54 smithi039 ceph-mon[128858]: pgmap v1135: 97 pgs: 97 active+clean; 729 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 11 MiB/s wr, 683 op/s 2023-07-29T16:40:55.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:54 smithi156 ceph-mon[129245]: pgmap v1135: 97 pgs: 97 active+clean; 729 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 11 MiB/s wr, 683 op/s 2023-07-29T16:40:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 -- 172.21.15.31:0/896586735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394075710 msgr2=0x7ff394075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/896586735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394075710 0x7ff394075b20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009990 tx=0x7ff38c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 -- 172.21.15.31:0/896586735 shutdown_connections 2023-07-29T16:40:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/896586735 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff394077300 0x7ff394077b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/896586735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394075710 0x7ff394075b20 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/896586735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394076950 0x7ff394076dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 -- 172.21.15.31:0/896586735 >> 172.21.15.31:0/896586735 conn(0x7ff3940fdbd0 msgr2=0x7ff394100020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 -- 172.21.15.31:0/896586735 shutdown_connections 2023-07-29T16:40:55.381 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.376+0000 7ff39a9ff700 1 -- 172.21.15.31:0/896586735 wait complete. 2023-07-29T16:40:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 Processor -- start 2023-07-29T16:40:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 -- start start 2023-07-29T16:40:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff394075710 0x7ff39419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394076950 0x7ff39419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 0x7ff39419dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39419e270 con 0x7ff394075710 2023-07-29T16:40:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39419e3e0 con 0x7ff394077300 2023-07-29T16:40:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.378+0000 7ff39a9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff39419e5a0 con 0x7ff394076950 2023-07-29T16:40:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.379+0000 7ff398f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 0x7ff39419dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.379+0000 7ff398f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 0x7ff39419dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58754/0 (socket says 172.21.15.31:58754) 2023-07-29T16:40:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.379+0000 7ff398f9c700 1 -- 172.21.15.31:0/3390177205 learned_addr learned my addr 172.21.15.31:0/3390177205 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:55.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.381+0000 7ff393fff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff394075710 0x7ff39419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.382+0000 7ff3937fe700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394076950 0x7ff39419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.382+0000 7ff398f9c700 1 -- 172.21.15.31:0/3390177205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394076950 msgr2=0x7ff39419cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.382+0000 7ff398f9c700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394076950 0x7ff39419cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.382+0000 7ff398f9c700 1 -- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff394075710 msgr2=0x7ff39419c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.382+0000 7ff398f9c700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff394075710 0x7ff39419c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.382+0000 7ff398f9c700 1 -- 172.21.15.31:0/3390177205 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c0095f0 con 0x7ff394077300 2023-07-29T16:40:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.383+0000 7ff398f9c700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 0x7ff39419dc10 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ff38400b550 tx=0x7ff38400b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.383+0000 7ff3917fa700 1 -- 172.21.15.31:0/3390177205 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff384011830 con 0x7ff394077300 2023-07-29T16:40:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.383+0000 7ff3917fa700 1 -- 172.21.15.31:0/3390177205 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff384011e50 con 0x7ff394077300 2023-07-29T16:40:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.383+0000 7ff3917fa700 1 -- 172.21.15.31:0/3390177205 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38400f2e0 con 0x7ff394077300 2023-07-29T16:40:55.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.384+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff39419e880 con 0x7ff394077300 2023-07-29T16:40:55.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.384+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff39406a9d0 con 0x7ff394077300 2023-07-29T16:40:55.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.391+0000 7ff3ae23ab80 1 Processor -- start 2023-07-29T16:40:55.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 -- start start 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e0a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e0b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e0c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131d629b0 con 0x55a131e0bcf0 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131d60370 con 0x55a131c666b0 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.392+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131d5bc60 con 0x55a131e0adf0 2023-07-29T16:40:55.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.393+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e0a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.393+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e0a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58764/0 (socket says 172.21.15.31:58764) 2023-07-29T16:40:55.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.393+0000 7ff3937fe700 1 -- 172.21.15.31:0/4004314540 learned_addr learned my addr 172.21.15.31:0/4004314540 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:55.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.395+0000 7ff3937fe700 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e0c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff393fff700 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e0b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff3937fe700 1 -- 172.21.15.31:0/4004314540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 msgr2=0x55a131e0b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff3937fe700 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e0b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff3937fe700 1 -- 172.21.15.31:0/4004314540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 msgr2=0x55a131e0c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff3937fe700 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e0c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff3937fe700 1 -- 172.21.15.31:0/4004314540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a131b89d80 con 0x55a131c666b0 2023-07-29T16:40:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.396+0000 7ff393fff700 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e0b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:40:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.399+0000 7ff3937fe700 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e0a8b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff37c002110 tx=0x7ff37c001a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81030c965a54b53c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:55.403 INFO:tasks.workunit.client.0.smithi031.stdout: 10 396214 33.11 MB/sec execute 519 sec latency 225.424 ms 2023-07-29T16:40:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.400+0000 7ff38a7fc700 1 -- 172.21.15.31:0/4004314540 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0126c0 con 0x55a131c666b0 2023-07-29T16:40:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.400+0000 7ff38a7fc700 1 -- 172.21.15.31:0/4004314540 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37c00e040 con 0x55a131c666b0 2023-07-29T16:40:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.400+0000 7ff38a7fc700 1 -- 172.21.15.31:0/4004314540 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c017a90 con 0x55a131c666b0 2023-07-29T16:40:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.400+0000 7ff3917fa700 1 -- 172.21.15.31:0/3390177205 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7ff3840105a0 con 0x7ff394077300 2023-07-29T16:40:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.401+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/4004314540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 msgr2=0x55a131e0a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.401+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e0a8b0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff37c002110 tx=0x7ff37c001a00 comp rx=0 tx=0).stop 2023-07-29T16:40:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.402+0000 7ff3917fa700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3800a0960 0x7ff3800a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.403+0000 7ff3917fa700 1 -- 172.21.15.31:0/3390177205 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff3840cb440 con 0x7ff394077300 2023-07-29T16:40:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.403+0000 7ff3937fe700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3800a0960 0x7ff3800a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.404+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/4004314540 shutdown_connections 2023-07-29T16:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.404+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e0c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.404+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e0a8b0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.404+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/4004314540 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e0b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.404+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/4004314540 >> 172.21.15.31:0/4004314540 conn(0x55a131dfffd0 msgr2=0x55a131e05300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.404+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/4004314540 shutdown_connections 2023-07-29T16:40:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.409+0000 7ff3937fe700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3800a0960 0x7ff3800a2e10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff37c017550 tx=0x7ff37c004be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.411+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/4004314540 wait complete. 2023-07-29T16:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.412+0000 7ff3ae23ab80 1 -- start start 2023-07-29T16:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.412+0000 7ff3ae23ab80 1 Processor -- start 2023-07-29T16:40:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e13f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0adf0 0x55a131e14470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0bcf0 0x55a131e1b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131cf84b0 con 0x55a131c666b0 2023-07-29T16:40:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131cf8150 con 0x55a131e0adf0 2023-07-29T16:40:55.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131bdf8c0 con 0x55a131e0bcf0 2023-07-29T16:40:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.413+0000 7ff3ae23ab80 10 client.0 _ll_register_callbacks cb 0x55a131d015c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.415+0000 7ff398f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0bcf0 0x55a131e1b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.415+0000 7ff393fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e13f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.415+0000 7ff398f9c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0bcf0 0x55a131e1b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39748/0 (socket says 172.21.15.31:39748) 2023-07-29T16:40:55.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.415+0000 7ff398f9c700 1 -- 172.21.15.31:0/2457870161 learned_addr learned my addr 172.21.15.31:0/2457870161 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.416+0000 7ff393fff700 1 -- 172.21.15.31:0/2457870161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0bcf0 msgr2=0x55a131e1b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.416+0000 7ff393fff700 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0bcf0 0x55a131e1b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.416+0000 7ff393fff700 1 -- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0adf0 msgr2=0x55a131e14470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:40:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.416+0000 7ff393fff700 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0adf0 0x55a131e14470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.416+0000 7ff393fff700 1 -- 172.21.15.31:0/2457870161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c0142d0 con 0x55a131c666b0 2023-07-29T16:40:55.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.417+0000 7ff393fff700 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e13f30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff38c000c00 tx=0x7ff38c005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3afdf0b4ff94f321 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:55.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.425+0000 7ff3737fe700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:40:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.430+0000 7ff3737fe700 1 -- 172.21.15.31:0/2457870161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c005600 con 0x55a131c666b0 2023-07-29T16:40:55.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.430+0000 7ff3737fe700 1 -- 172.21.15.31:0/2457870161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff38c004710 con 0x55a131c666b0 2023-07-29T16:40:55.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.433+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a131d86cb0 con 0x55a131c666b0 2023-07-29T16:40:55.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.434+0000 7ff3737fe700 1 -- 172.21.15.31:0/2457870161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff38c004910 con 0x55a131c666b0 2023-07-29T16:40:55.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:54 smithi031 ceph-mon[95504]: pgmap v1135: 97 pgs: 97 active+clean; 729 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 11 MiB/s wr, 683 op/s 2023-07-29T16:40:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.434+0000 7ff3ae23ab80 10 client.20397 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:55.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.434+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55a131b72c00 con 0x55a131c666b0 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3737fe700 1 -- 172.21.15.31:0/2457870161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a131d86cb0 con 0x55a131c666b0 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3737fe700 1 -- 172.21.15.31:0/2457870161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff38c00fbc0 con 0x55a131c666b0 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3ae23ab80 10 client.20397 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3ae23ab80 10 client.20397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3ae23ab80 20 client.20397 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e30c20 0x55a131e33030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3ae23ab80 4 client.20397 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:40:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.435+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a131dd9d10 con 0x55a131e30c20 2023-07-29T16:40:55.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.437+0000 7ff3937fe700 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e30c20 0x55a131e33030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.442+0000 7ff3937fe700 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e30c20 0x55a131e33030 crc :-1 s=READY pgs=511 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b42ba21868a5664 server_cookie=14ca27a574b97d51 in_seq=0 out_seq=0 2023-07-29T16:40:55.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.442+0000 7ff3737fe700 10 client.20397 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:55.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.451+0000 7ff3737fe700 1 -- 172.21.15.31:0/2457870161 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a131dd9d10 con 0x55a131e30c20 2023-07-29T16:40:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.451+0000 7ff3737fe700 10 client.20397 handle_command_reply: tid=0 2023-07-29T16:40:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.451+0000 7ff3ae23ab80 1 client.20397 shutdown 2023-07-29T16:40:55.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 msgr2=0x55a131e13f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e13f30 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7ff38c000c00 tx=0x7ff38c005c80 comp rx=0 tx=0).stop 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 shutdown_connections 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e30c20 0x55a131e33030 crc :-1 s=READY pgs=511 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0bcf0 0x55a131e1b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e13f30 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/2457870161 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0adf0 0x55a131e14470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.452+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 >> 172.21.15.31:0/2457870161 conn(0x55a131dfffd0 msgr2=0x55a131e04a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.453+0000 7ff3737fe700 0 client.20397 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.453+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 shutdown_connections 2023-07-29T16:40:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.453+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/2457870161 wait complete. 2023-07-29T16:40:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.454+0000 7ff3ae23ab80 20 client.20397 trim_cache size 0 max 16384 2023-07-29T16:40:55.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.486+0000 7ff3ae23ab80 1 Processor -- start 2023-07-29T16:40:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 -- start start 2023-07-29T16:40:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e1b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e32c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131e13b00 con 0x55a131e0bcf0 2023-07-29T16:40:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131dd9d10 con 0x55a131c666b0 2023-07-29T16:40:55.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.487+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131d86cb0 con 0x55a131e0adf0 2023-07-29T16:40:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.488+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e1b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.488+0000 7ff393fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.488+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e1b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:39760/0 (socket says 172.21.15.31:39760) 2023-07-29T16:40:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.488+0000 7ff3937fe700 1 -- 172.21.15.31:0/3445217998 learned_addr learned my addr 172.21.15.31:0/3445217998 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:55.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff398f9c700 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e32c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff393fff700 1 -- 172.21.15.31:0/3445217998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 msgr2=0x55a131e1b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff393fff700 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e1b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff393fff700 1 -- 172.21.15.31:0/3445217998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 msgr2=0x55a131e32c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff393fff700 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e32c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff393fff700 1 -- 172.21.15.31:0/3445217998 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a131bb01f0 con 0x55a131c666b0 2023-07-29T16:40:55.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.489+0000 7ff398f9c700 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e32c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:40:55.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.490+0000 7ff393fff700 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e052d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0098e0 tx=0x7ff38c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e8dbf927e020ae6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.495+0000 7ff38affd700 1 -- 172.21.15.31:0/3445217998 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c023600 con 0x55a131c666b0 2023-07-29T16:40:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.495+0000 7ff38affd700 1 -- 172.21.15.31:0/3445217998 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff38c01e050 con 0x55a131c666b0 2023-07-29T16:40:55.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.495+0000 7ff38affd700 1 -- 172.21.15.31:0/3445217998 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c023bb0 con 0x55a131c666b0 2023-07-29T16:40:55.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/3445217998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 msgr2=0x55a131e052d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e052d0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0098e0 tx=0x7ff38c005e80 comp rx=0 tx=0).stop 2023-07-29T16:40:55.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/3445217998 shutdown_connections 2023-07-29T16:40:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131e0bcf0 0x55a131e32c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131c666b0 0x55a131e052d0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/3445217998 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e1b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/3445217998 >> 172.21.15.31:0/3445217998 conn(0x55a131dfffd0 msgr2=0x55a131e12c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:55.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.496+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/3445217998 shutdown_connections 2023-07-29T16:40:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.497+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/3445217998 wait complete. 2023-07-29T16:40:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.497+0000 7ff3ae23ab80 1 -- start start 2023-07-29T16:40:55.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.498+0000 7ff3ae23ab80 1 Processor -- start 2023-07-29T16:40:55.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e16f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e17460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 0x55a131e1a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131e1a9c0 con 0x55a131c666b0 2023-07-29T16:40:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131e1ab30 con 0x55a131e0bcf0 2023-07-29T16:40:55.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a131e1acf0 con 0x55a131e0adf0 2023-07-29T16:40:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.507+0000 7ff3ae23ab80 10 client.0 _ll_register_callbacks cb 0x55a131d015c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:40:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.508+0000 7ff3937fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e17460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.508+0000 7ff398f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 0x55a131e1a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.508+0000 7ff398f9c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 0x55a131e1a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:58784/0 (socket says 172.21.15.31:58784) 2023-07-29T16:40:55.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.508+0000 7ff398f9c700 1 -- 172.21.15.31:0/309455357 learned_addr learned my addr 172.21.15.31:0/309455357 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:40:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.515+0000 7ff393fff700 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e16f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.515+0000 7ff398f9c700 1 -- 172.21.15.31:0/309455357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 msgr2=0x55a131e17460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.515+0000 7ff398f9c700 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e17460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.515+0000 7ff398f9c700 1 -- 172.21.15.31:0/309455357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 msgr2=0x55a131e16f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:40:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.515+0000 7ff398f9c700 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e16f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:55.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.515+0000 7ff398f9c700 1 -- 172.21.15.31:0/309455357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a131b72c00 con 0x55a131e0bcf0 2023-07-29T16:40:55.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.516+0000 7ff398f9c700 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 0x55a131e1a480 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff384094f90 tx=0x7ff384098ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c949a56b5ef33289 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:40:55.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.516+0000 7ff3737fe700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:40:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.521+0000 7ff3737fe700 1 -- 172.21.15.31:0/309455357 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38409cb70 con 0x55a131e0bcf0 2023-07-29T16:40:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.521+0000 7ff3737fe700 1 -- 172.21.15.31:0/309455357 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7ff3840a0f30 con 0x55a131e0bcf0 2023-07-29T16:40:55.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.521+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff37c00e040 con 0x55a131e0bcf0 2023-07-29T16:40:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.524+0000 7ff3737fe700 1 -- 172.21.15.31:0/309455357 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3840c8f90 con 0x55a131e0bcf0 2023-07-29T16:40:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.525+0000 7ff3ae23ab80 10 client.30692 fetch_fsmap learned FSMap version 43 2023-07-29T16:40:55.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.525+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55a131b88250 con 0x55a131e0bcf0 2023-07-29T16:40:55.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.529+0000 7ff3737fe700 1 -- 172.21.15.31:0/309455357 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff37c00e040 con 0x55a131e0bcf0 2023-07-29T16:40:55.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.529+0000 7ff3737fe700 1 -- 172.21.15.31:0/309455357 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7ff3840a5070 con 0x55a131e0bcf0 2023-07-29T16:40:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.530+0000 7ff3ae23ab80 10 client.30692 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:40:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.530+0000 7ff3ae23ab80 10 client.30692 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:40:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.530+0000 7ff3ae23ab80 20 client.30692 populate_metadata read hostname 'smithi031' 2023-07-29T16:40:55.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.530+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e83450 0x55a131e85860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:40:55.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.530+0000 7ff3ae23ab80 4 client.30692 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:40:55.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.530+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a131d54ce0 con 0x55a131e83450 2023-07-29T16:40:55.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.531+0000 7ff393fff700 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e83450 0x55a131e85860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:40:55.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.537+0000 7ff393fff700 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e83450 0x55a131e85860 crc :-1 s=READY pgs=512 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=587c1b3915cf6ad6 server_cookie=d95e96833c603bf8 in_seq=0 out_seq=0 2023-07-29T16:40:55.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.537+0000 7ff3737fe700 10 client.30692 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.993+0000 7ff3737fe700 1 -- 172.21.15.31:0/309455357 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a131d54ce0 con 0x55a131e83450 2023-07-29T16:40:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.993+0000 7ff3737fe700 10 client.30692 handle_command_reply: tid=0 2023-07-29T16:40:55.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:55.994+0000 7ff3ae23ab80 1 client.30692 shutdown 2023-07-29T16:40:56.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.000+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 msgr2=0x55a131e1a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:56.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.000+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 0x55a131e1a480 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff384094f90 tx=0x7ff384098ae0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:40:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:40:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:40:56.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:40:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.000+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 shutdown_connections 2023-07-29T16:40:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a131e83450 0x55a131e85860 crc :-1 s=READY pgs=512 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a131e0bcf0 0x55a131e1a480 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a131c666b0 0x55a131e16f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 --2- 172.21.15.31:0/309455357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a131e0adf0 0x55a131e17460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 >> 172.21.15.31:0/309455357 conn(0x55a131dfffd0 msgr2=0x55a131e32c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3737fe700 0 client.30692 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:40:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 shutdown_connections 2023-07-29T16:40:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 1 -- 172.21.15.31:0/309455357 wait complete. 2023-07-29T16:40:56.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.001+0000 7ff3ae23ab80 20 client.30692 trim_cache size 0 max 16384 2023-07-29T16:40:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3800a0960 msgr2=0x7ff3800a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3800a0960 0x7ff3800a2e10 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff37c017550 tx=0x7ff37c004be0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 msgr2=0x7ff39419dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:40:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 0x7ff39419dc10 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ff38400b550 tx=0x7ff38400b860 comp rx=0 tx=0).stop 2023-07-29T16:40:56.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 shutdown_connections 2023-07-29T16:40:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7ff3800a0960 0x7ff3800a2e10 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff394077300 0x7ff39419dc10 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff394075710 0x7ff39419c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 --2- 172.21.15.31:0/3390177205 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff394076950 0x7ff39419cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:40:56.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.005+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 >> 172.21.15.31:0/3390177205 conn(0x7ff3940fdbd0 msgr2=0x7ff3940ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:40:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.007+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 shutdown_connections 2023-07-29T16:40:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:40:56.007+0000 7ff39a9ff700 1 -- 172.21.15.31:0/3390177205 wait complete. 2023-07-29T16:40:56.405 INFO:tasks.workunit.client.0.smithi031.stdout: 10 396474 33.07 MB/sec execute 520 sec latency 257.797 ms 2023-07-29T16:40:57.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:56 smithi039 ceph-mon[128858]: pgmap v1136: 97 pgs: 97 active+clean; 750 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 596 op/s 2023-07-29T16:40:57.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:56 smithi156 ceph-mon[129245]: pgmap v1136: 97 pgs: 97 active+clean; 750 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 596 op/s 2023-07-29T16:40:57.406 INFO:tasks.workunit.client.0.smithi031.stdout: 10 396907 33.06 MB/sec execute 521 sec latency 227.804 ms 2023-07-29T16:40:57.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:56 smithi031 ceph-mon[95504]: pgmap v1136: 97 pgs: 97 active+clean; 750 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 12 MiB/s wr, 596 op/s 2023-07-29T16:40:58.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 397228 33.03 MB/sec execute 522 sec latency 468.124 ms 2023-07-29T16:40:58.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:40:58 smithi031 ceph-mon[95504]: pgmap v1137: 97 pgs: 97 active+clean; 750 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 15 MiB/s wr, 661 op/s 2023-07-29T16:40:58.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:40:58 smithi039 ceph-mon[128858]: pgmap v1137: 97 pgs: 97 active+clean; 750 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 15 MiB/s wr, 661 op/s 2023-07-29T16:40:58.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:40:58 smithi156 ceph-mon[129245]: pgmap v1137: 97 pgs: 97 active+clean; 750 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 15 MiB/s wr, 661 op/s 2023-07-29T16:40:59.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 397762 33.01 MB/sec execute 523 sec latency 63.883 ms 2023-07-29T16:41:00.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 398473 33.02 MB/sec execute 524 sec latency 48.551 ms 2023-07-29T16:41:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:00 smithi031 ceph-mon[95504]: pgmap v1138: 97 pgs: 97 active+clean; 706 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 17 MiB/s wr, 816 op/s 2023-07-29T16:41:01.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:00 smithi039 ceph-mon[128858]: pgmap v1138: 97 pgs: 97 active+clean; 706 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 17 MiB/s wr, 816 op/s 2023-07-29T16:41:01.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:00 smithi156 ceph-mon[129245]: pgmap v1138: 97 pgs: 97 active+clean; 706 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 17 MiB/s wr, 816 op/s 2023-07-29T16:41:01.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 398979 33.03 MB/sec execute 525 sec latency 315.382 ms 2023-07-29T16:41:02.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 399555 33.02 MB/sec execute 526 sec latency 80.256 ms 2023-07-29T16:41:03.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:02 smithi031 ceph-mon[95504]: pgmap v1139: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 MiB/s wr, 765 op/s 2023-07-29T16:41:03.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:02 smithi039 ceph-mon[128858]: pgmap v1139: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 MiB/s wr, 765 op/s 2023-07-29T16:41:03.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:02 smithi156 ceph-mon[129245]: pgmap v1139: 97 pgs: 97 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 MiB/s wr, 765 op/s 2023-07-29T16:41:03.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 400126 33.01 MB/sec execute 527 sec latency 89.866 ms 2023-07-29T16:41:04.408 INFO:tasks.workunit.client.0.smithi031.stdout: 10 400681 32.99 MB/sec execute 528 sec latency 176.932 ms 2023-07-29T16:41:05.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:04 smithi031 ceph-mon[95504]: pgmap v1140: 97 pgs: 97 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 17 MiB/s wr, 884 op/s 2023-07-29T16:41:05.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:04 smithi039 ceph-mon[128858]: pgmap v1140: 97 pgs: 97 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 17 MiB/s wr, 884 op/s 2023-07-29T16:41:05.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:04 smithi156 ceph-mon[129245]: pgmap v1140: 97 pgs: 97 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 17 MiB/s wr, 884 op/s 2023-07-29T16:41:05.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 401374 33.02 MB/sec execute 529 sec latency 94.791 ms 2023-07-29T16:41:06.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 402105 33.02 MB/sec execute 530 sec latency 72.874 ms 2023-07-29T16:41:07.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:06 smithi031 ceph-mon[95504]: pgmap v1141: 97 pgs: 97 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 714 op/s 2023-07-29T16:41:07.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:06 smithi039 ceph-mon[128858]: pgmap v1141: 97 pgs: 97 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 714 op/s 2023-07-29T16:41:07.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:06 smithi156 ceph-mon[129245]: pgmap v1141: 97 pgs: 97 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 714 op/s 2023-07-29T16:41:07.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 402696 33.03 MB/sec execute 531 sec latency 89.908 ms 2023-07-29T16:41:08.408 INFO:tasks.workunit.client.0.smithi031.stdout: 10 403222 33.02 MB/sec execute 532 sec latency 123.675 ms 2023-07-29T16:41:08.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:08 smithi156 ceph-mon[129245]: pgmap v1142: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.9 MiB/s wr, 684 op/s 2023-07-29T16:41:08.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:08 smithi031 ceph-mon[95504]: pgmap v1142: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.9 MiB/s wr, 684 op/s 2023-07-29T16:41:08.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:08 smithi039 ceph-mon[128858]: pgmap v1142: 97 pgs: 97 active+clean; 679 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.9 MiB/s wr, 684 op/s 2023-07-29T16:41:09.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 -- 172.21.15.31:0/533219309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8101540 msgr2=0x7f55a8101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 --2- 172.21.15.31:0/533219309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8101540 0x7f55a8101990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f559c009990 tx=0x7f559c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 -- 172.21.15.31:0/533219309 shutdown_connections 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 --2- 172.21.15.31:0/533219309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8101ed0 0x7f55a8108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 --2- 172.21.15.31:0/533219309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100350 0x7f55a8100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 --2- 172.21.15.31:0/533219309 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8101540 0x7f55a8101990 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 -- 172.21.15.31:0/533219309 >> 172.21.15.31:0/533219309 conn(0x7f55a80fb970 msgr2=0x7f55a80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:09.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 -- 172.21.15.31:0/533219309 shutdown_connections 2023-07-29T16:41:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.284+0000 7f55acfff700 1 -- 172.21.15.31:0/533219309 wait complete. 2023-07-29T16:41:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.285+0000 7f55acfff700 1 Processor -- start 2023-07-29T16:41:09.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.285+0000 7f55acfff700 1 -- start start 2023-07-29T16:41:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55acfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 0x7f55a8198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55acfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8101540 0x7f55a8198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55acfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8101ed0 0x7f55a81996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55acfff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a8199d40 con 0x7f55a8101ed0 2023-07-29T16:41:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55acfff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a8199eb0 con 0x7f55a8100350 2023-07-29T16:41:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55acfff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55a819a070 con 0x7f55a8101540 2023-07-29T16:41:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55a659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 0x7f55a8198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55a659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 0x7f55a8198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:37982/0 (socket says 172.21.15.31:37982) 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55a659c700 1 -- 172.21.15.31:0/2194626637 learned_addr learned my addr 172.21.15.31:0/2194626637 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8101ed0 0x7f55a81996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.286+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8101540 0x7f55a8198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a659c700 1 -- 172.21.15.31:0/2194626637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8101540 msgr2=0x7f55a8198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a659c700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8101540 0x7f55a8198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a659c700 1 -- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8101ed0 msgr2=0x7f55a81996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a659c700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8101ed0 0x7f55a81996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a659c700 1 -- 172.21.15.31:0/2194626637 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5598009520 con 0x7f55a8100350 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8101540 0x7f55a8198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55a659c700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 0x7f55a8198320 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f559800ea50 tx=0x7f559800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.287+0000 7f55977fe700 1 -- 172.21.15.31:0/2194626637 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559800cbd0 con 0x7f55a8100350 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.288+0000 7f55977fe700 1 -- 172.21.15.31:0/2194626637 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f559800cd30 con 0x7f55a8100350 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.288+0000 7f55977fe700 1 -- 172.21.15.31:0/2194626637 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5598018710 con 0x7f55a8100350 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.288+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f559c0095f0 con 0x7f55a8100350 2023-07-29T16:41:09.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.288+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a81a0930 con 0x7f55a8100350 2023-07-29T16:41:09.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.293+0000 7f55977fe700 1 -- 172.21.15.31:0/2194626637 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5598010430 con 0x7f55a8100350 2023-07-29T16:41:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.294+0000 7f55c09aab80 1 Processor -- start 2023-07-29T16:41:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.294+0000 7f55c09aab80 1 -- start start 2023-07-29T16:41:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.294+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e4a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.294+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e4afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.294+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e4b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55c09aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839d999b0 con 0x558839e4b500 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55977fe700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55900a0810 0x7f55900a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55c09aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839d97370 con 0x558839c9d6b0 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55c09aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839d92c60 con 0x558839e4a600 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55977fe700 1 -- 172.21.15.31:0/2194626637 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f5598014070 con 0x7f55a8100350 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a6d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e4a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e4afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e4afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55958/0 (socket says 172.21.15.31:55958) 2023-07-29T16:41:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 -- 172.21.15.31:0/324571414 learned_addr learned my addr 172.21.15.31:0/324571414 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55900a0810 0x7f55900a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e4b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 -- 172.21.15.31:0/324571414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 msgr2=0x558839e4a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e4a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 -- 172.21.15.31:0/324571414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 msgr2=0x558839e4b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e4b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a659c700 1 -- 172.21.15.31:0/324571414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558839bc0d80 con 0x558839e4a600 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.295+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e4b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.296+0000 7f55a659c700 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e4afc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5598099760 tx=0x7f55980996d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6190f5b361a34b10 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.296+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55900a0810 0x7f55900a2cc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.296+0000 7f5594ff9700 1 -- 172.21.15.31:0/324571414 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55980bdab0 con 0x558839e4a600 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.296+0000 7f5594ff9700 1 -- 172.21.15.31:0/324571414 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55980997d0 con 0x558839e4a600 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.296+0000 7f5594ff9700 1 -- 172.21.15.31:0/324571414 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5598099a30 con 0x558839e4a600 2023-07-29T16:41:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 -- 172.21.15.31:0/324571414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 msgr2=0x558839e4afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e4afc0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5598099760 tx=0x7f55980996d0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 -- 172.21.15.31:0/324571414 shutdown_connections 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e4b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e4a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 --2- 172.21.15.31:0/324571414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e4afc0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 -- 172.21.15.31:0/324571414 >> 172.21.15.31:0/324571414 conn(0x558839e36fd0 msgr2=0x558839e3c300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 -- 172.21.15.31:0/324571414 shutdown_connections 2023-07-29T16:41:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.297+0000 7f55c09aab80 1 -- 172.21.15.31:0/324571414 wait complete. 2023-07-29T16:41:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.298+0000 7f55c09aab80 1 -- start start 2023-07-29T16:41:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.298+0000 7f55c09aab80 1 Processor -- start 2023-07-29T16:41:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839c9d6b0 0x558839e536d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e53c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4b500 0x558839e5ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839d2f4b0 con 0x558839c9d6b0 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839d2f150 con 0x558839e4b500 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839c168c0 con 0x558839e4a600 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55c09aab80 10 client.0 _ll_register_callbacks cb 0x558839d385c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55a5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e53c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55a5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e53c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55974/0 (socket says 172.21.15.31:55974) 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55a5d9b700 1 -- 172.21.15.31:0/2207394864 learned_addr learned my addr 172.21.15.31:0/2207394864 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:09.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4b500 0x558839e5ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.299+0000 7f55a659c700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839c9d6b0 0x558839e536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f55a6d9d700 1 -- 172.21.15.31:0/2207394864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 msgr2=0x558839e53c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e53c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f55a6d9d700 1 -- 172.21.15.31:0/2207394864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839c9d6b0 msgr2=0x558839e536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839c9d6b0 0x558839e536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f55a6d9d700 1 -- 172.21.15.31:0/2207394864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a81a0930 con 0x558839e4b500 2023-07-29T16:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4b500 0x558839e5ac50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f55a0007be0 tx=0x7f55a0007ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8734ec572a41fe9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.300+0000 7f558dffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.301+0000 7f558dffb700 1 -- 172.21.15.31:0/2207394864 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a0004bb0 con 0x558839e4b500 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.301+0000 7f558dffb700 1 -- 172.21.15.31:0/2207394864 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f55a0004d10 con 0x558839e4b500 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.301+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558839dbdcb0 con 0x558839e4b500 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.301+0000 7f558dffb700 1 -- 172.21.15.31:0/2207394864 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55a0003a00 con 0x558839e4b500 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.301+0000 7f55c09aab80 10 client.30704 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.301+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x558839ba9c00 con 0x558839e4b500 2023-07-29T16:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f558dffb700 1 -- 172.21.15.31:0/2207394864 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558839dbdcb0 con 0x558839e4b500 2023-07-29T16:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f558dffb700 1 -- 172.21.15.31:0/2207394864 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f55a0004050 con 0x558839e4b500 2023-07-29T16:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f55c09aab80 10 client.30704 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f55c09aab80 10 client.30704 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f55c09aab80 20 client.30704 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f55c09aab80 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839e5f4f0 0x558839e61900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f55c09aab80 4 client.30704 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.302+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558839e10d10 con 0x558839e5f4f0 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.303+0000 7f55a659c700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839e5f4f0 0x558839e61900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.303+0000 7f55a659c700 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839e5f4f0 0x558839e61900 crc :-1 s=READY pgs=513 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d0ea59f84fd301f server_cookie=3e23ab5d37012491 in_seq=0 out_seq=0 2023-07-29T16:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.303+0000 7f558dffb700 10 client.30704 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.306+0000 7f558dffb700 1 -- 172.21.15.31:0/2207394864 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x558839e10d10 con 0x558839e5f4f0 2023-07-29T16:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.306+0000 7f558dffb700 10 client.30704 handle_command_reply: tid=0 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 client.30704 shutdown 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4b500 msgr2=0x558839e5ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4b500 0x558839e5ac50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f55a0007be0 tx=0x7f55a0007ef0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 shutdown_connections 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839e5f4f0 0x558839e61900 crc :-1 s=READY pgs=513 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4b500 0x558839e5ac50 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839c9d6b0 0x558839e536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f558dffb700 0 client.30704 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 --2- 172.21.15.31:0/2207394864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e53c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.307+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 >> 172.21.15.31:0/2207394864 conn(0x558839e36fd0 msgr2=0x558839e3ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.308+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 shutdown_connections 2023-07-29T16:41:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.308+0000 7f55c09aab80 1 -- 172.21.15.31:0/2207394864 wait complete. 2023-07-29T16:41:09.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.308+0000 7f55c09aab80 20 client.30704 trim_cache size 0 max 16384 2023-07-29T16:41:09.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.333+0000 7f55c09aab80 1 Processor -- start 2023-07-29T16:41:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.333+0000 7f55c09aab80 1 -- start start 2023-07-29T16:41:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e523e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e5ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e51900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55c09aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839e3ddf0 con 0x558839e4b500 2023-07-29T16:41:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55c09aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839e3d1c0 con 0x558839c9d6b0 2023-07-29T16:41:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55c09aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839e5f2e0 con 0x558839e4a600 2023-07-29T16:41:09.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55a659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55a659c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:38008/0 (socket says 172.21.15.31:38008) 2023-07-29T16:41:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55a659c700 1 -- 172.21.15.31:0/3737183405 learned_addr learned my addr 172.21.15.31:0/3737183405 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e5ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.334+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e51900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55a659c700 1 -- 172.21.15.31:0/3737183405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 msgr2=0x558839e5ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55a659c700 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e5ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55a659c700 1 -- 172.21.15.31:0/3737183405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 msgr2=0x558839e51900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55a659c700 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e51900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55a659c700 1 -- 172.21.15.31:0/3737183405 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558839be71f0 con 0x558839c9d6b0 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55a659c700 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e523e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5598010d00 tx=0x7f5598099650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa83f0bace55c181 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55957fa700 1 -- 172.21.15.31:0/3737183405 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55980bdb30 con 0x558839c9d6b0 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55957fa700 1 -- 172.21.15.31:0/3737183405 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5598010750 con 0x558839c9d6b0 2023-07-29T16:41:09.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.335+0000 7f55957fa700 1 -- 172.21.15.31:0/3737183405 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5598097c70 con 0x558839c9d6b0 2023-07-29T16:41:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 -- 172.21.15.31:0/3737183405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 msgr2=0x558839e523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e523e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5598010d00 tx=0x7f5598099650 comp rx=0 tx=0).stop 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 -- 172.21.15.31:0/3737183405 shutdown_connections 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e51900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839c9d6b0 0x558839e523e0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3737183405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839e4a600 0x558839e5ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 -- 172.21.15.31:0/3737183405 >> 172.21.15.31:0/3737183405 conn(0x558839e36fd0 msgr2=0x558839e3ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.336+0000 7f55c09aab80 1 -- 172.21.15.31:0/3737183405 shutdown_connections 2023-07-29T16:41:09.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.337+0000 7f55c09aab80 1 -- 172.21.15.31:0/3737183405 wait complete. 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.337+0000 7f55c09aab80 1 -- start start 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.337+0000 7f55c09aab80 1 Processor -- start 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 0x558839e5a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4a600 0x558839e5aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e56f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839e574c0 con 0x558839e4b500 2023-07-29T16:41:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839e57630 con 0x558839e4a600 2023-07-29T16:41:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x558839e577f0 con 0x558839c9d6b0 2023-07-29T16:41:09.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55c09aab80 10 client.0 _ll_register_callbacks cb 0x558839d385c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55a659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 0x558839e5a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.338+0000 7f55a659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 0x558839e5a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:56002/0 (socket says 172.21.15.31:56002) 2023-07-29T16:41:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 -- 172.21.15.31:0/3835732746 learned_addr learned my addr 172.21.15.31:0/3835732746 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:09.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a6d9d700 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e56f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 -- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4a600 msgr2=0x558839e5aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4a600 0x558839e5aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 -- 172.21.15.31:0/3835732746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 msgr2=0x558839e56f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e56f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 -- 172.21.15.31:0/3835732746 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558839bc0d80 con 0x558839c9d6b0 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f55a659c700 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 0x558839e5a4e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5598099760 tx=0x7f5598097ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1e34c18bba8947b server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:09.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f558dffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.339+0000 7f558dffb700 1 -- 172.21.15.31:0/3835732746 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55980106e0 con 0x558839c9d6b0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f558dffb700 1 -- 172.21.15.31:0/3835732746 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f559809d800 con 0x558839c9d6b0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558839d8bce0 con 0x558839c9d6b0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f558dffb700 1 -- 172.21.15.31:0/3835732746 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f55980bdb30 con 0x558839c9d6b0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 10 client.30742 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x558839bbf250 con 0x558839c9d6b0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f558dffb700 1 -- 172.21.15.31:0/3835732746 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5598064020 con 0x558839c9d6b0 2023-07-29T16:41:09.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f558dffb700 1 -- 172.21.15.31:0/3835732746 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f5598099a30 con 0x558839c9d6b0 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 10 client.30742 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 10 client.30742 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 20 client.30742 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839eb2060 0x558839eb4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 4 client.30742 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.340+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558839d8ba00 con 0x558839eb2060 2023-07-29T16:41:09.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.341+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839eb2060 0x558839eb4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.341+0000 7f55a5d9b700 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839eb2060 0x558839eb4470 crc :-1 s=READY pgs=514 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=754d861894090d26 server_cookie=2e12c898ee180da in_seq=0 out_seq=0 2023-07-29T16:41:09.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.341+0000 7f558dffb700 10 client.30742 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:09.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 403627 32.99 MB/sec execute 533 sec latency 153.374 ms 2023-07-29T16:41:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.761+0000 7f558dffb700 1 -- 172.21.15.31:0/3835732746 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558839d8ba00 con 0x558839eb2060 2023-07-29T16:41:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.761+0000 7f558dffb700 10 client.30742 handle_command_reply: tid=0 2023-07-29T16:41:09.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.761+0000 7f55c09aab80 1 client.30742 shutdown 2023-07-29T16:41:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.765+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 msgr2=0x558839e5a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.765+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 0x558839e5a4e0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5598099760 tx=0x7f5598097ce0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.766+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 shutdown_connections 2023-07-29T16:41:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.766+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x558839eb2060 0x558839eb4470 crc :-1 s=READY pgs=514 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.766+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558839e4b500 0x558839e56f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.766+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558839c9d6b0 0x558839e5a4e0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.766+0000 7f55c09aab80 1 --2- 172.21.15.31:0/3835732746 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558839e4a600 0x558839e5aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.766+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 >> 172.21.15.31:0/3835732746 conn(0x558839e36fd0 msgr2=0x558839e523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:09.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.767+0000 7f558dffb700 0 client.30742 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.767+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 shutdown_connections 2023-07-29T16:41:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.767+0000 7f55c09aab80 1 -- 172.21.15.31:0/3835732746 wait complete. 2023-07-29T16:41:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.768+0000 7f55c09aab80 20 client.30742 trim_cache size 0 max 16384 2023-07-29T16:41:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:41:09.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:41:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:41:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:41:09.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55900a0810 msgr2=0x7f55900a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55900a0810 0x7f55900a2cc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f559c000c00 tx=0x7f559c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 msgr2=0x7f55a8198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 0x7f55a8198320 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f559800ea50 tx=0x7f559800ed60 comp rx=0 tx=0).stop 2023-07-29T16:41:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 shutdown_connections 2023-07-29T16:41:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f55900a0810 0x7f55900a2cc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8101ed0 0x7f55a81996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55a8100350 0x7f55a8198320 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 --2- 172.21.15.31:0/2194626637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8101540 0x7f55a8198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 >> 172.21.15.31:0/2194626637 conn(0x7f55a80fb970 msgr2=0x7f55a8106e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 shutdown_connections 2023-07-29T16:41:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:09.769+0000 7f55acfff700 1 -- 172.21.15.31:0/2194626637 wait complete. 2023-07-29T16:41:10.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 404060 32.97 MB/sec execute 534 sec latency 248.172 ms 2023-07-29T16:41:11.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:10 smithi031 ceph-mon[95504]: pgmap v1143: 97 pgs: 97 active+clean; 687 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 7.0 MiB/s wr, 630 op/s 2023-07-29T16:41:11.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:10 smithi039 ceph-mon[128858]: pgmap v1143: 97 pgs: 97 active+clean; 687 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 7.0 MiB/s wr, 630 op/s 2023-07-29T16:41:11.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:10 smithi156 ceph-mon[129245]: pgmap v1143: 97 pgs: 97 active+clean; 687 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 7.0 MiB/s wr, 630 op/s 2023-07-29T16:41:11.407 INFO:tasks.workunit.client.0.smithi031.stdout: 10 404671 32.98 MB/sec execute 535 sec latency 84.072 ms 2023-07-29T16:41:12.409 INFO:tasks.workunit.client.0.smithi031.stdout: 10 405313 32.96 MB/sec execute 536 sec latency 136.164 ms 2023-07-29T16:41:13.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:12 smithi031 ceph-mon[95504]: pgmap v1144: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.1 MiB/s wr, 490 op/s 2023-07-29T16:41:13.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:12 smithi039 ceph-mon[128858]: pgmap v1144: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.1 MiB/s wr, 490 op/s 2023-07-29T16:41:13.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:12 smithi156 ceph-mon[129245]: pgmap v1144: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.1 MiB/s wr, 490 op/s 2023-07-29T16:41:13.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 406070 33.00 MB/sec execute 537 sec latency 53.811 ms 2023-07-29T16:41:14.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 406844 33.00 MB/sec execute 538 sec latency 60.987 ms 2023-07-29T16:41:15.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:14 smithi031 ceph-mon[95504]: pgmap v1145: 97 pgs: 97 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2023-07-29T16:41:15.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:14 smithi039 ceph-mon[128858]: pgmap v1145: 97 pgs: 97 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2023-07-29T16:41:15.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:14 smithi156 ceph-mon[129245]: pgmap v1145: 97 pgs: 97 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.0 MiB/s wr, 568 op/s 2023-07-29T16:41:15.409 INFO:tasks.workunit.client.0.smithi031.stdout: 10 407419 33.02 MB/sec execute 539 sec latency 92.473 ms 2023-07-29T16:41:16.409 INFO:tasks.workunit.client.0.smithi031.stdout: 10 407968 33.00 MB/sec execute 540 sec latency 117.101 ms 2023-07-29T16:41:17.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:16 smithi031 ceph-mon[95504]: pgmap v1146: 97 pgs: 97 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.3 MiB/s wr, 474 op/s 2023-07-29T16:41:17.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:16 smithi039 ceph-mon[128858]: pgmap v1146: 97 pgs: 97 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.3 MiB/s wr, 474 op/s 2023-07-29T16:41:17.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:16 smithi156 ceph-mon[129245]: pgmap v1146: 97 pgs: 97 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 3.3 MiB/s wr, 474 op/s 2023-07-29T16:41:17.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 408605 33.01 MB/sec execute 541 sec latency 97.330 ms 2023-07-29T16:41:18.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 409185 32.99 MB/sec execute 542 sec latency 99.212 ms 2023-07-29T16:41:18.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:18 smithi156 ceph-mon[129245]: pgmap v1147: 97 pgs: 97 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.3 MiB/s wr, 476 op/s 2023-07-29T16:41:18.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:18 smithi031 ceph-mon[95504]: pgmap v1147: 97 pgs: 97 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.3 MiB/s wr, 476 op/s 2023-07-29T16:41:18.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:18 smithi039 ceph-mon[128858]: pgmap v1147: 97 pgs: 97 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.3 MiB/s wr, 476 op/s 2023-07-29T16:41:19.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 409844 33.00 MB/sec execute 543 sec latency 51.640 ms 2023-07-29T16:41:20.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 410479 33.00 MB/sec execute 544 sec latency 60.865 ms 2023-07-29T16:41:21.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:20 smithi031 ceph-mon[95504]: pgmap v1148: 97 pgs: 97 active+clean; 688 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.7 MiB/s wr, 526 op/s 2023-07-29T16:41:21.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:20 smithi039 ceph-mon[128858]: pgmap v1148: 97 pgs: 97 active+clean; 688 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.7 MiB/s wr, 526 op/s 2023-07-29T16:41:21.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:20 smithi156 ceph-mon[129245]: pgmap v1148: 97 pgs: 97 active+clean; 688 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 3.7 MiB/s wr, 526 op/s 2023-07-29T16:41:21.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 411236 33.01 MB/sec execute 545 sec latency 75.041 ms 2023-07-29T16:41:22.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 412029 33.04 MB/sec execute 546 sec latency 37.343 ms 2023-07-29T16:41:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.153+0000 7f16673ff700 1 -- 172.21.15.31:0/3608354497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f166010c5a0 msgr2=0x7f166010ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.153+0000 7f16673ff700 1 --2- 172.21.15.31:0/3608354497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f166010c5a0 0x7f166010ea50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f165c009bc0 tx=0x7f165c009ed0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.164+0000 7f16673ff700 1 -- 172.21.15.31:0/3608354497 shutdown_connections 2023-07-29T16:41:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.164+0000 7f16673ff700 1 --2- 172.21.15.31:0/3608354497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f166010c5a0 0x7f166010ea50 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.164+0000 7f16673ff700 1 --2- 172.21.15.31:0/3608354497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1660071740 0x7f1660071b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.164+0000 7f16673ff700 1 --2- 172.21.15.31:0/3608354497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f166010c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.164+0000 7f16673ff700 1 -- 172.21.15.31:0/3608354497 >> 172.21.15.31:0/3608354497 conn(0x7f166006d290 msgr2=0x7f166006f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.185+0000 7f16673ff700 1 -- 172.21.15.31:0/3608354497 shutdown_connections 2023-07-29T16:41:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.185+0000 7f16673ff700 1 -- 172.21.15.31:0/3608354497 wait complete. 2023-07-29T16:41:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 Processor -- start 2023-07-29T16:41:23.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 -- start start 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1660071740 0x7f1660116ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f1660117010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f166010c5a0 0x7f16601151f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1660115730 con 0x7f1660071740 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16601158a0 con 0x7f1660072090 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.187+0000 7f16673ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1660115a60 con 0x7f166010c5a0 2023-07-29T16:41:23.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.188+0000 7f166499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f1660117010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.188+0000 7f166499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f1660117010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39636/0 (socket says 172.21.15.31:39636) 2023-07-29T16:41:23.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.188+0000 7f166499a700 1 -- 172.21.15.31:0/3744199667 learned_addr learned my addr 172.21.15.31:0/3744199667 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.192+0000 7f166519b700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1660071740 0x7f1660116ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.193+0000 7f166599c700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f166010c5a0 0x7f16601151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.203+0000 7f166499a700 1 -- 172.21.15.31:0/3744199667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f166010c5a0 msgr2=0x7f16601151f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.203+0000 7f166499a700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f166010c5a0 0x7f16601151f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.203+0000 7f166499a700 1 -- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1660071740 msgr2=0x7f1660116ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.203+0000 7f166499a700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1660071740 0x7f1660116ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.203+0000 7f166499a700 1 -- 172.21.15.31:0/3744199667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c009820 con 0x7f1660072090 2023-07-29T16:41:23.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.216+0000 7f166499a700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f1660117010 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f165800efc0 tx=0x7f165800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:23.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.231+0000 7f16567fc700 1 -- 172.21.15.31:0/3744199667 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658018070 con 0x7f1660072090 2023-07-29T16:41:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.231+0000 7f16567fc700 1 -- 172.21.15.31:0/3744199667 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f165800f040 con 0x7f1660072090 2023-07-29T16:41:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.231+0000 7f16567fc700 1 -- 172.21.15.31:0/3744199667 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658004280 con 0x7f1660072090 2023-07-29T16:41:23.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.231+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1660115ce0 con 0x7f1660072090 2023-07-29T16:41:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.232+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1660120290 con 0x7f1660072090 2023-07-29T16:41:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.239+0000 7f167ad01b80 1 Processor -- start 2023-07-29T16:41:23.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 -- start start 2023-07-29T16:41:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 0x55c406a47160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c40699d9b0 con 0x55c4068a16b0 2023-07-29T16:41:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c40699b370 con 0x55c406a45df0 2023-07-29T16:41:23.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.240+0000 7f167ad01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406996c60 con 0x55c406a46cf0 2023-07-29T16:41:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.242+0000 7f166499a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 0x55c406a47160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.242+0000 7f166499a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 0x55c406a47160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40634/0 (socket says 172.21.15.31:40634) 2023-07-29T16:41:23.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.242+0000 7f166499a700 1 -- 172.21.15.31:0/559270422 learned_addr learned my addr 172.21.15.31:0/559270422 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:23.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.250+0000 7f166599c700 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.262+0000 7f166499a700 1 -- 172.21.15.31:0/559270422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 msgr2=0x55c406a467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.262+0000 7f166499a700 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.262+0000 7f166499a700 1 -- 172.21.15.31:0/559270422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 msgr2=0x55c406a458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.262+0000 7f166499a700 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.262+0000 7f166499a700 1 -- 172.21.15.31:0/559270422 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4067c4d80 con 0x55c406a46cf0 2023-07-29T16:41:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.265+0000 7f16567fc700 1 -- 172.21.15.31:0/3744199667 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1658007500 con 0x7f1660072090 2023-07-29T16:41:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.268+0000 7f16567fc700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f164c0a0990 0x7f164c0a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.268+0000 7f16567fc700 1 -- 172.21.15.31:0/3744199667 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f16580cce90 con 0x7f1660072090 2023-07-29T16:41:23.272 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.269+0000 7f166499a700 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 0x55c406a47160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1658009430 tx=0x7f1658003d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76c42229b22f2036 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.275+0000 7f166599c700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f164c0a0990 0x7f164c0a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:22 smithi039 ceph-mon[128858]: pgmap v1149: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.0 MiB/s wr, 425 op/s 2023-07-29T16:41:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.289+0000 7f164b7fe700 1 -- 172.21.15.31:0/559270422 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658096a90 con 0x55c406a46cf0 2023-07-29T16:41:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.289+0000 7f164b7fe700 1 -- 172.21.15.31:0/559270422 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f165809a520 con 0x55c406a46cf0 2023-07-29T16:41:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.289+0000 7f164b7fe700 1 -- 172.21.15.31:0/559270422 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16580a2b10 con 0x55c406a46cf0 2023-07-29T16:41:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 -- 172.21.15.31:0/559270422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 msgr2=0x55c406a47160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 0x55c406a47160 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1658009430 tx=0x7f1658003d90 comp rx=0 tx=0).stop 2023-07-29T16:41:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 -- 172.21.15.31:0/559270422 shutdown_connections 2023-07-29T16:41:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a46cf0 0x55c406a47160 secure :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1658009430 tx=0x7f1658003d90 comp rx=0 tx=0).stop 2023-07-29T16:41:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 --2- 172.21.15.31:0/559270422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a467b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.290+0000 7f167ad01b80 1 -- 172.21.15.31:0/559270422 >> 172.21.15.31:0/559270422 conn(0x55c406a3afd0 msgr2=0x55c406a40300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.291+0000 7f167ad01b80 1 -- 172.21.15.31:0/559270422 shutdown_connections 2023-07-29T16:41:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.291+0000 7f167ad01b80 1 -- 172.21.15.31:0/559270422 wait complete. 2023-07-29T16:41:23.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.292+0000 7f167ad01b80 1 -- start start 2023-07-29T16:41:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.292+0000 7f167ad01b80 1 Processor -- start 2023-07-29T16:41:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a4f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a4f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a54660 0x55c406a56a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4069334b0 con 0x55c4068a16b0 2023-07-29T16:41:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406933150 con 0x55c406a45df0 2023-07-29T16:41:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c40681a8c0 con 0x55c406a54660 2023-07-29T16:41:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.293+0000 7f167ad01b80 10 client.0 _ll_register_callbacks cb 0x55c40693c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.294+0000 7f166499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a4f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.294+0000 7f166499a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a4f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39654/0 (socket says 172.21.15.31:39654) 2023-07-29T16:41:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.294+0000 7f166499a700 1 -- 172.21.15.31:0/2309366053 learned_addr learned my addr 172.21.15.31:0/2309366053 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166519b700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a4f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166599c700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a54660 0x55c406a56a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166499a700 1 -- 172.21.15.31:0/2309366053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a54660 msgr2=0x55c406a56a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166499a700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a54660 0x55c406a56a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166499a700 1 -- 172.21.15.31:0/2309366053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 msgr2=0x55c406a4f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166499a700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a4f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.298+0000 7f166499a700 1 -- 172.21.15.31:0/2309366053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1660120290 con 0x55c406a45df0 2023-07-29T16:41:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.315+0000 7f166499a700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a4f630 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1658098aa0 tx=0x7f165809a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9360db6aaa494f00 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:23.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.315+0000 7f1648ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:41:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.318+0000 7f1648ff9700 1 -- 172.21.15.31:0/2309366053 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658003e00 con 0x55c406a45df0 2023-07-29T16:41:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.319+0000 7f1648ff9700 1 -- 172.21.15.31:0/2309366053 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f16580a2b10 con 0x55c406a45df0 2023-07-29T16:41:23.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.319+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4069c1cb0 con 0x55c406a45df0 2023-07-29T16:41:23.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.328+0000 7f1648ff9700 1 -- 172.21.15.31:0/2309366053 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f165809a9b0 con 0x55c406a45df0 2023-07-29T16:41:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.328+0000 7f167ad01b80 10 client.30716 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:23.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.328+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55c4067adc00 con 0x55c406a45df0 2023-07-29T16:41:23.331 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:22 smithi156 ceph-mon[129245]: pgmap v1149: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.0 MiB/s wr, 425 op/s 2023-07-29T16:41:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.331+0000 7f166599c700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f164c0a0990 0x7f164c0a2e40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f165c00b5c0 tx=0x7f165c004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.335+0000 7f1648ff9700 1 -- 172.21.15.31:0/2309366053 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16580c8030 con 0x55c406a45df0 2023-07-29T16:41:23.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.335+0000 7f1648ff9700 1 -- 172.21.15.31:0/2309366053 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f165809bbb0 con 0x55c406a45df0 2023-07-29T16:41:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.336+0000 7f167ad01b80 10 client.30716 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.336+0000 7f167ad01b80 10 client.30716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:23.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.336+0000 7f167ad01b80 20 client.30716 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:23.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.336+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406a5ae10 0x55c406a5d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.336+0000 7f167ad01b80 4 client.30716 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:41:23.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.336+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c406a14d10 con 0x55c406a5ae10 2023-07-29T16:41:23.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.337+0000 7f166519b700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406a5ae10 0x55c406a5d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.373+0000 7f166519b700 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406a5ae10 0x55c406a5d220 crc :-1 s=READY pgs=515 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bad167732891a17 server_cookie=539e7c2fc55dd23f in_seq=0 out_seq=0 2023-07-29T16:41:23.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.373+0000 7f1648ff9700 10 client.30716 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.377+0000 7f1648ff9700 1 -- 172.21.15.31:0/2309366053 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55c406a14d10 con 0x55c406a5ae10 2023-07-29T16:41:23.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.377+0000 7f1648ff9700 10 client.30716 handle_command_reply: tid=0 2023-07-29T16:41:23.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.377+0000 7f167ad01b80 1 client.30716 shutdown 2023-07-29T16:41:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.378+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 msgr2=0x55c406a4f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.378+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a4f630 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1658098aa0 tx=0x7f165809a340 comp rx=0 tx=0).stop 2023-07-29T16:41:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 shutdown_connections 2023-07-29T16:41:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406a5ae10 0x55c406a5d220 crc :-1 s=READY pgs=515 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a54660 0x55c406a56a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a4f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2309366053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a45df0 0x55c406a4f630 secure :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1658098aa0 tx=0x7f165809a340 comp rx=0 tx=0).stop 2023-07-29T16:41:23.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 >> 172.21.15.31:0/2309366053 conn(0x55c406a3afd0 msgr2=0x55c406a3fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:23.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.384+0000 7f1648ff9700 0 client.30716 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.393+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 shutdown_connections 2023-07-29T16:41:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.396+0000 7f167ad01b80 1 -- 172.21.15.31:0/2309366053 wait complete. 2023-07-29T16:41:23.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.404+0000 7f167ad01b80 20 client.30716 trim_cache size 0 max 16384 2023-07-29T16:41:23.410 INFO:tasks.workunit.client.0.smithi031.stdout: 10 412370 33.01 MB/sec execute 547 sec latency 566.527 ms 2023-07-29T16:41:23.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 Processor -- start 2023-07-29T16:41:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 -- start start 2023-07-29T16:41:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a6a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a4d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a3fa40 0x55c406a464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406a4ec60 con 0x55c4068a16b0 2023-07-29T16:41:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406a41e30 con 0x55c406a54660 2023-07-29T16:41:23.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.442+0000 7f167ad01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406a4fd30 con 0x55c406a3fa40 2023-07-29T16:41:23.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:22 smithi031 ceph-mon[95504]: pgmap v1149: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 3.0 MiB/s wr, 425 op/s 2023-07-29T16:41:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.451+0000 7f166599c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a3fa40 0x55c406a464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.451+0000 7f166599c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a3fa40 0x55c406a464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:40652/0 (socket says 172.21.15.31:40652) 2023-07-29T16:41:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.451+0000 7f166599c700 1 -- 172.21.15.31:0/2693159096 learned_addr learned my addr 172.21.15.31:0/2693159096 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.451+0000 7f166519b700 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a6a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.459+0000 7f166499a700 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a4d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.460+0000 7f166519b700 1 -- 172.21.15.31:0/2693159096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a3fa40 msgr2=0x55c406a464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.460+0000 7f166519b700 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a3fa40 0x55c406a464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.460+0000 7f166519b700 1 -- 172.21.15.31:0/2693159096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 msgr2=0x55c406a4d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.460+0000 7f166519b700 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a4d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.460+0000 7f166519b700 1 -- 172.21.15.31:0/2693159096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4067eb1f0 con 0x55c4068a16b0 2023-07-29T16:41:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.460+0000 7f166519b700 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a6a740 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x55c406a5b1e0 tx=0x7f16500097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04f09324890f6ac server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.474+0000 7f164bfff700 1 -- 172.21.15.31:0/2693159096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16500169c0 con 0x55c4068a16b0 2023-07-29T16:41:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.474+0000 7f164bfff700 1 -- 172.21.15.31:0/2693159096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f165001e070 con 0x55c4068a16b0 2023-07-29T16:41:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.474+0000 7f164bfff700 1 -- 172.21.15.31:0/2693159096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1650022d80 con 0x55c4068a16b0 2023-07-29T16:41:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 -- 172.21.15.31:0/2693159096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 msgr2=0x55c406a6a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a6a740 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x55c406a5b1e0 tx=0x7f16500097d0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 -- 172.21.15.31:0/2693159096 shutdown_connections 2023-07-29T16:41:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c406a3fa40 0x55c406a464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c4068a16b0 0x55c406a6a740 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2693159096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a4d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.475+0000 7f167ad01b80 1 -- 172.21.15.31:0/2693159096 >> 172.21.15.31:0/2693159096 conn(0x55c406a3afd0 msgr2=0x55c406a3c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.476+0000 7f167ad01b80 1 -- 172.21.15.31:0/2693159096 shutdown_connections 2023-07-29T16:41:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.476+0000 7f167ad01b80 1 -- 172.21.15.31:0/2693159096 wait complete. 2023-07-29T16:41:23.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.476+0000 7f167ad01b80 1 -- start start 2023-07-29T16:41:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 Processor -- start 2023-07-29T16:41:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4068a16b0 0x55c406a6a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c406a3fa40 0x55c406a68a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a68fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406a694f0 con 0x55c406a3fa40 2023-07-29T16:41:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406a69660 con 0x55c406a54660 2023-07-29T16:41:23.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c406a69820 con 0x55c4068a16b0 2023-07-29T16:41:23.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.477+0000 7f167ad01b80 10 client.0 _ll_register_callbacks cb 0x55c40693c5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.481+0000 7f166599c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a68fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.481+0000 7f166599c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a68fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:39684/0 (socket says 172.21.15.31:39684) 2023-07-29T16:41:23.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.481+0000 7f166599c700 1 -- 172.21.15.31:0/2068239869 learned_addr learned my addr 172.21.15.31:0/2068239869 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:23.484 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.481+0000 7f166519b700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4068a16b0 0x55c406a6a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.492+0000 7f166499a700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c406a3fa40 0x55c406a68a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.496+0000 7f166599c700 1 -- 172.21.15.31:0/2068239869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4068a16b0 msgr2=0x55c406a6a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.496+0000 7f166599c700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4068a16b0 0x55c406a6a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.496+0000 7f166599c700 1 -- 172.21.15.31:0/2068239869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c406a3fa40 msgr2=0x55c406a68a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.496+0000 7f166599c700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c406a3fa40 0x55c406a68a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.496+0000 7f166599c700 1 -- 172.21.15.31:0/2068239869 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1650009a60 con 0x55c406a54660 2023-07-29T16:41:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.503+0000 7f166599c700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a68fb0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f165c009b10 tx=0x7f165c0094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b739fdc823a3be98 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.505+0000 7f1648ff9700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:41:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.508+0000 7f1648ff9700 1 -- 172.21.15.31:0/2068239869 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c030070 con 0x55c406a54660 2023-07-29T16:41:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.508+0000 7f1648ff9700 1 -- 172.21.15.31:0/2068239869 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f165c00baa0 con 0x55c406a54660 2023-07-29T16:41:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.509+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f165809a520 con 0x55c406a54660 2023-07-29T16:41:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.520+0000 7f1648ff9700 1 -- 172.21.15.31:0/2068239869 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f165c00bc60 con 0x55c406a54660 2023-07-29T16:41:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.523+0000 7f167ad01b80 10 client.30722 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.523+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55c4067c3250 con 0x55c406a54660 2023-07-29T16:41:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f1648ff9700 1 -- 172.21.15.31:0/2068239869 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f165809a520 con 0x55c406a54660 2023-07-29T16:41:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f1648ff9700 1 -- 172.21.15.31:0/2068239869 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f165c00f460 con 0x55c406a54660 2023-07-29T16:41:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f167ad01b80 10 client.30722 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f167ad01b80 10 client.30722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f167ad01b80 20 client.30722 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406aae720 0x55c406ab0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f167ad01b80 4 client.30722 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:41:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.530+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c40698fce0 con 0x55c406aae720 2023-07-29T16:41:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.547+0000 7f166519b700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406aae720 0x55c406ab0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.580+0000 7f166519b700 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406aae720 0x55c406ab0b30 crc :-1 s=READY pgs=516 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8602c375a6d3f578 server_cookie=a3b5b9035036b8fb in_seq=0 out_seq=0 2023-07-29T16:41:23.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:23.585+0000 7f1648ff9700 10 client.30722 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:24.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 412382 32.96 MB/sec execute 548 sec latency 1366.314 ms 2023-07-29T16:41:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.192+0000 7f1648ff9700 1 -- 172.21.15.31:0/2068239869 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c40698fce0 con 0x55c406aae720 2023-07-29T16:41:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.192+0000 7f1648ff9700 10 client.30722 handle_command_reply: tid=0 2023-07-29T16:41:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.192+0000 7f167ad01b80 1 client.30722 shutdown 2023-07-29T16:41:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 msgr2=0x55c406a68fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a68fb0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f165c009b10 tx=0x7f165c0094e0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 shutdown_connections 2023-07-29T16:41:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55c406aae720 0x55c406ab0b30 crc :-1 s=READY pgs=516 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c406a54660 0x55c406a68fb0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4068a16b0 0x55c406a6a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 --2- 172.21.15.31:0/2068239869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c406a3fa40 0x55c406a68a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 >> 172.21.15.31:0/2068239869 conn(0x55c406a3afd0 msgr2=0x55c406a4d230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f1648ff9700 0 client.30722 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.193+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 shutdown_connections 2023-07-29T16:41:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.194+0000 7f167ad01b80 1 -- 172.21.15.31:0/2068239869 wait complete. 2023-07-29T16:41:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.199+0000 7f167ad01b80 20 client.30722 trim_cache size 0 max 16384 2023-07-29T16:41:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:41:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:41:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:41:25.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:41:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.201+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f164c0a0990 msgr2=0x7f164c0a2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.201+0000 7f16673ff700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f164c0a0990 0x7f164c0a2e40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f165c00b5c0 tx=0x7f165c004f10 comp rx=0 tx=0).stop 2023-07-29T16:41:25.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.201+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 msgr2=0x7f1660117010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.201+0000 7f16673ff700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f1660117010 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f165800efc0 tx=0x7f165800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.202+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 shutdown_connections 2023-07-29T16:41:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.202+0000 7f16673ff700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f164c0a0990 0x7f164c0a2e40 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.202+0000 7f16673ff700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f166010c5a0 0x7f16601151f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.202+0000 7f16673ff700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1660071740 0x7f1660116ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.202+0000 7f16673ff700 1 --2- 172.21.15.31:0/3744199667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1660072090 0x7f1660117010 secure :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f165800efc0 tx=0x7f165800c3a0 comp rx=0 tx=0).stop 2023-07-29T16:41:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.202+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 >> 172.21.15.31:0/3744199667 conn(0x7f166006d290 msgr2=0x7f166006f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.204+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 shutdown_connections 2023-07-29T16:41:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:25.205+0000 7f16673ff700 1 -- 172.21.15.31:0/3744199667 wait complete. 2023-07-29T16:41:25.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:25 smithi156 ceph-mon[129245]: pgmap v1150: 97 pgs: 97 active+clean; 714 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 5.4 MiB/s wr, 519 op/s 2023-07-29T16:41:25.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 412511 32.91 MB/sec execute 549 sec latency 1757.748 ms 2023-07-29T16:41:25.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:25 smithi031 ceph-mon[95504]: pgmap v1150: 97 pgs: 97 active+clean; 714 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 5.4 MiB/s wr, 519 op/s 2023-07-29T16:41:25.530 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:25 smithi039 ceph-mon[128858]: pgmap v1150: 97 pgs: 97 active+clean; 714 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 5.4 MiB/s wr, 519 op/s 2023-07-29T16:41:26.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 412744 32.87 MB/sec execute 550 sec latency 137.242 ms 2023-07-29T16:41:27.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:27 smithi156 ceph-mon[129245]: pgmap v1151: 97 pgs: 97 active+clean; 755 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.8 MiB/s wr, 503 op/s 2023-07-29T16:41:27.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 413134 32.86 MB/sec execute 551 sec latency 283.425 ms 2023-07-29T16:41:27.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:27 smithi031 ceph-mon[95504]: pgmap v1151: 97 pgs: 97 active+clean; 755 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.8 MiB/s wr, 503 op/s 2023-07-29T16:41:27.529 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:27 smithi039 ceph-mon[128858]: pgmap v1151: 97 pgs: 97 active+clean; 755 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.8 MiB/s wr, 503 op/s 2023-07-29T16:41:28.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 413715 32.83 MB/sec execute 552 sec latency 65.949 ms 2023-07-29T16:41:28.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:28 smithi156 ceph-mon[129245]: pgmap v1152: 97 pgs: 97 active+clean; 756 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 13 MiB/s wr, 645 op/s 2023-07-29T16:41:28.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:28 smithi031 ceph-mon[95504]: pgmap v1152: 97 pgs: 97 active+clean; 756 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 13 MiB/s wr, 645 op/s 2023-07-29T16:41:28.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:28 smithi039 ceph-mon[128858]: pgmap v1152: 97 pgs: 97 active+clean; 756 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 13 MiB/s wr, 645 op/s 2023-07-29T16:41:29.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 414359 32.84 MB/sec execute 553 sec latency 85.095 ms 2023-07-29T16:41:30.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 415349 32.87 MB/sec execute 554 sec latency 45.666 ms 2023-07-29T16:41:31.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:30 smithi031 ceph-mon[95504]: pgmap v1153: 97 pgs: 97 active+clean; 708 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 794 op/s 2023-07-29T16:41:31.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:30 smithi039 ceph-mon[128858]: pgmap v1153: 97 pgs: 97 active+clean; 708 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 794 op/s 2023-07-29T16:41:31.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:30 smithi156 ceph-mon[129245]: pgmap v1153: 97 pgs: 97 active+clean; 708 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 794 op/s 2023-07-29T16:41:31.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 415895 32.88 MB/sec execute 555 sec latency 133.339 ms 2023-07-29T16:41:32.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:31 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:41:32.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:31 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:41:32.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:31 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:41:32.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 416138 32.85 MB/sec execute 556 sec latency 395.448 ms 2023-07-29T16:41:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:32 smithi031 ceph-mon[95504]: pgmap v1154: 97 pgs: 97 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 14 MiB/s wr, 759 op/s 2023-07-29T16:41:33.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:32 smithi039 ceph-mon[128858]: pgmap v1154: 97 pgs: 97 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 14 MiB/s wr, 759 op/s 2023-07-29T16:41:33.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:32 smithi156 ceph-mon[129245]: pgmap v1154: 97 pgs: 97 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 14 MiB/s wr, 759 op/s 2023-07-29T16:41:33.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 416603 32.84 MB/sec execute 557 sec latency 118.961 ms 2023-07-29T16:41:34.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 417003 32.81 MB/sec execute 558 sec latency 79.840 ms 2023-07-29T16:41:35.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:34 smithi031 ceph-mon[95504]: pgmap v1155: 97 pgs: 97 active+clean; 690 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 16 MiB/s wr, 868 op/s 2023-07-29T16:41:35.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:34 smithi156 ceph-mon[129245]: pgmap v1155: 97 pgs: 97 active+clean; 690 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 16 MiB/s wr, 868 op/s 2023-07-29T16:41:35.348 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:34 smithi039 ceph-mon[128858]: pgmap v1155: 97 pgs: 97 active+clean; 690 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 16 MiB/s wr, 868 op/s 2023-07-29T16:41:35.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 417515 32.81 MB/sec execute 559 sec latency 92.798 ms 2023-07-29T16:41:36.416 INFO:tasks.workunit.client.0.smithi031.stdout: 10 418065 32.78 MB/sec execute 560 sec latency 92.463 ms 2023-07-29T16:41:37.191 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:36 smithi031 ceph-mon[95504]: pgmap v1156: 97 pgs: 97 active+clean; 691 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 15 MiB/s wr, 825 op/s 2023-07-29T16:41:37.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:36 smithi039 ceph-mon[128858]: pgmap v1156: 97 pgs: 97 active+clean; 691 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 15 MiB/s wr, 825 op/s 2023-07-29T16:41:37.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:36 smithi156 ceph-mon[129245]: pgmap v1156: 97 pgs: 97 active+clean; 691 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 15 MiB/s wr, 825 op/s 2023-07-29T16:41:37.417 INFO:tasks.workunit.client.0.smithi031.stdout: 10 418697 32.77 MB/sec execute 561 sec latency 68.069 ms 2023-07-29T16:41:38.417 INFO:tasks.workunit.client.0.smithi031.stdout: 10 419438 32.80 MB/sec execute 562 sec latency 65.582 ms 2023-07-29T16:41:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 -- 172.21.15.31:0/3663927422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 msgr2=0x7fba5c100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 --2- 172.21.15.31:0/3663927422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 0x7fba5c100750 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fba58009990 tx=0x7fba58009ca0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 -- 172.21.15.31:0/3663927422 shutdown_connections 2023-07-29T16:41:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 --2- 172.21.15.31:0/3663927422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101ec0 0x7fba5c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 --2- 172.21.15.31:0/3663927422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 0x7fba5c100750 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 --2- 172.21.15.31:0/3663927422 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba5c101530 0x7fba5c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 -- 172.21.15.31:0/3663927422 >> 172.21.15.31:0/3663927422 conn(0x7fba5c0fb920 msgr2=0x7fba5c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.431+0000 7fba641ff700 1 -- 172.21.15.31:0/3663927422 shutdown_connections 2023-07-29T16:41:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.432+0000 7fba641ff700 1 -- 172.21.15.31:0/3663927422 wait complete. 2023-07-29T16:41:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.433+0000 7fba641ff700 1 Processor -- start 2023-07-29T16:41:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 -- start start 2023-07-29T16:41:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 0x7fba5c193f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 0x7fba5c194490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba5c101ec0 0x7fba5c195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c195970 con 0x7fba5c101ec0 2023-07-29T16:41:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c195ae0 con 0x7fba5c101530 2023-07-29T16:41:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.434+0000 7fba641ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c195ca0 con 0x7fba5c100340 2023-07-29T16:41:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 0x7fba5c194490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 0x7fba5c194490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40662/0 (socket says 172.21.15.31:40662) 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 -- 172.21.15.31:0/1503350888 learned_addr learned my addr 172.21.15.31:0/1503350888 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6279c700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba5c101ec0 0x7fba5c195310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba61f9b700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 0x7fba5c193f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 -- 172.21.15.31:0/1503350888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 msgr2=0x7fba5c193f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 0x7fba5c193f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 -- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba5c101ec0 msgr2=0x7fba5c195310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba5c101ec0 0x7fba5c195310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.435+0000 7fba6179a700 1 -- 172.21.15.31:0/1503350888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c009520 con 0x7fba5c101530 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.436+0000 7fba6179a700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 0x7fba5c194490 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00eb30 tx=0x7fba4c00eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.436+0000 7fba52ffd700 1 -- 172.21.15.31:0/1503350888 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c009880 con 0x7fba5c101530 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.436+0000 7fba52ffd700 1 -- 172.21.15.31:0/1503350888 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba4c0044e0 con 0x7fba5c101530 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.436+0000 7fba52ffd700 1 -- 172.21.15.31:0/1503350888 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c010430 con 0x7fba5c101530 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.436+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba580095f0 con 0x7fba5c101530 2023-07-29T16:41:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.436+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c06a770 con 0x7fba5c101530 2023-07-29T16:41:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.443+0000 7fba52ffd700 1 -- 172.21.15.31:0/1503350888 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fba4c00cd40 con 0x7fba5c101530 2023-07-29T16:41:38.446 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.444+0000 7fba77b2bb80 1 Processor -- start 2023-07-29T16:41:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.444+0000 7fba77b2bb80 1 -- start start 2023-07-29T16:41:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27548230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27546e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27547820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb274969b0 con 0x55fb27548770 2023-07-29T16:41:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb27494370 con 0x55fb2739a6b0 2023-07-29T16:41:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb2748fc60 con 0x55fb27547380 2023-07-29T16:41:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba6279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27548230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba6279c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27548230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40668/0 (socket says 172.21.15.31:40668) 2023-07-29T16:41:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba6179a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27547820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.445+0000 7fba6279c700 1 -- 172.21.15.31:0/3820655510 learned_addr learned my addr 172.21.15.31:0/3820655510 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.446+0000 7fba6279c700 1 -- 172.21.15.31:0/3820655510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 msgr2=0x55fb27547820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.446+0000 7fba6279c700 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27547820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.446+0000 7fba6279c700 1 -- 172.21.15.31:0/3820655510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 msgr2=0x55fb27546e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.446+0000 7fba6279c700 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27546e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.446+0000 7fba6279c700 1 -- 172.21.15.31:0/3820655510 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb272bdd80 con 0x55fb2739a6b0 2023-07-29T16:41:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.446+0000 7fba6279c700 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27548230 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fba5400bf60 tx=0x7fba5400bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f388122a86a4cbb9 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.447+0000 7fba47fff700 1 -- 172.21.15.31:0/3820655510 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54012070 con 0x55fb2739a6b0 2023-07-29T16:41:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.447+0000 7fba52ffd700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba480a0890 0x7fba480a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.447+0000 7fba47fff700 1 -- 172.21.15.31:0/3820655510 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba54004bb0 con 0x55fb2739a6b0 2023-07-29T16:41:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.447+0000 7fba47fff700 1 -- 172.21.15.31:0/3820655510 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54008e00 con 0x55fb2739a6b0 2023-07-29T16:41:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.447+0000 7fba52ffd700 1 -- 172.21.15.31:0/1503350888 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fba4c014070 con 0x7fba5c101530 2023-07-29T16:41:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 -- 172.21.15.31:0/3820655510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 msgr2=0x55fb27548230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27548230 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fba5400bf60 tx=0x7fba5400bf90 comp rx=0 tx=0).stop 2023-07-29T16:41:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba61f9b700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba480a0890 0x7fba480a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 -- 172.21.15.31:0/3820655510 shutdown_connections 2023-07-29T16:41:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27547820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27548230 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/3820655510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27546e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.448+0000 7fba77b2bb80 1 -- 172.21.15.31:0/3820655510 >> 172.21.15.31:0/3820655510 conn(0x55fb27533fd0 msgr2=0x55fb27539310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.449+0000 7fba77b2bb80 1 -- 172.21.15.31:0/3820655510 shutdown_connections 2023-07-29T16:41:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.449+0000 7fba77b2bb80 1 -- 172.21.15.31:0/3820655510 wait complete. 2023-07-29T16:41:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.449+0000 7fba61f9b700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba480a0890 0x7fba480a2d40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fba58006010 tx=0x7fba5800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.450+0000 7fba77b2bb80 1 -- start start 2023-07-29T16:41:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 Processor -- start 2023-07-29T16:41:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27550930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27550e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb27557ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb2742c4b0 con 0x55fb27547380 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb2742c150 con 0x55fb2739a6b0 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.451+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb273138c0 con 0x55fb27548770 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.452+0000 7fba77b2bb80 10 client.0 _ll_register_callbacks cb 0x55fb274355c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.452+0000 7fba61f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27550930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.452+0000 7fba61f9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27550930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40682/0 (socket says 172.21.15.31:40682) 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.452+0000 7fba61f9b700 1 -- 172.21.15.31:0/1304290745 learned_addr learned my addr 172.21.15.31:0/1304290745 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.452+0000 7fba6279c700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb27557ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.452+0000 7fba6179a700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27550e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6279c700 1 -- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 msgr2=0x55fb27550930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6279c700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27550930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6279c700 1 -- 172.21.15.31:0/1304290745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 msgr2=0x55fb27550e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6279c700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27550e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6279c700 1 -- 172.21.15.31:0/1304290745 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5400bc10 con 0x55fb27548770 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6179a700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27550e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba6279c700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb27557ca0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b800 tx=0x7fba5400d730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c43de415e59e487f server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba457fa700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.453+0000 7fba457fa700 1 -- 172.21.15.31:0/1304290745 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5401b070 con 0x55fb27548770 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.454+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb274bacb0 con 0x55fb27548770 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.454+0000 7fba457fa700 1 -- 172.21.15.31:0/1304290745 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fba54017680 con 0x55fb27548770 2023-07-29T16:41:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba457fa700 1 -- 172.21.15.31:0/1304290745 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba54012040 con 0x55fb27548770 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 10 client.30769 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55fb272a6c00 con 0x55fb27548770 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba457fa700 1 -- 172.21.15.31:0/1304290745 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fb274bacb0 con 0x55fb27548770 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba457fa700 1 -- 172.21.15.31:0/1304290745 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fba54020670 con 0x55fb27548770 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 10 client.30769 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 10 client.30769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:38.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 20 client.30769 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb2755c640 0x55fb2755ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 4 client.30769 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:41:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.455+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb2750dd10 con 0x55fb2755c640 2023-07-29T16:41:38.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.456+0000 7fba61f9b700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb2755c640 0x55fb2755ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.457+0000 7fba457fa700 10 client.30769 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.457+0000 7fba61f9b700 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb2755c640 0x55fb2755ea50 crc :-1 s=READY pgs=517 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=738a94e9d0448867 server_cookie=671b171c2aa9b419 in_seq=0 out_seq=0 2023-07-29T16:41:38.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.457+0000 7fba457fa700 1 -- 172.21.15.31:0/1304290745 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55fb2750dd10 con 0x55fb2755c640 2023-07-29T16:41:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba457fa700 10 client.30769 handle_command_reply: tid=0 2023-07-29T16:41:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 client.30769 shutdown 2023-07-29T16:41:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 msgr2=0x55fb27557ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb27557ca0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b800 tx=0x7fba5400d730 comp rx=0 tx=0).stop 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 shutdown_connections 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb2755c640 0x55fb2755ea50 crc :-1 s=READY pgs=517 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb27557ca0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27550930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1304290745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27550e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 >> 172.21.15.31:0/1304290745 conn(0x55fb27533fd0 msgr2=0x55fb27538a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba457fa700 0 client.30769 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:38.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.458+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 shutdown_connections 2023-07-29T16:41:38.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.459+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1304290745 wait complete. 2023-07-29T16:41:38.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.459+0000 7fba77b2bb80 20 client.30769 trim_cache size 0 max 16384 2023-07-29T16:41:38.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.485+0000 7fba77b2bb80 1 Processor -- start 2023-07-29T16:41:38.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.485+0000 7fba77b2bb80 1 -- start start 2023-07-29T16:41:38.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.485+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27555bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.485+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27546d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.485+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb2755e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb27551910 con 0x55fb27547380 2023-07-29T16:41:38.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb2753a1f0 con 0x55fb2739a6b0 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb27558280 con 0x55fb27548770 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba6179a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27546d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba6179a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27546d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59170/0 (socket says 172.21.15.31:59170) 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba6179a700 1 -- 172.21.15.31:0/1790353502 learned_addr learned my addr 172.21.15.31:0/1790353502 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba61f9b700 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27555bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.486+0000 7fba6279c700 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb2755e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba6179a700 1 -- 172.21.15.31:0/1790353502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 msgr2=0x55fb2755e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba6179a700 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb2755e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba6179a700 1 -- 172.21.15.31:0/1790353502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 msgr2=0x55fb27555bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba6179a700 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27555bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba6179a700 1 -- 172.21.15.31:0/1790353502 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb272e41f0 con 0x55fb27547380 2023-07-29T16:41:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba6179a700 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27546d60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fba4c095bb0 tx=0x7fba4c099550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4a0a4630ab8826 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.487+0000 7fba50ff9700 1 -- 172.21.15.31:0/1790353502 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c016330 con 0x55fb27547380 2023-07-29T16:41:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.488+0000 7fba50ff9700 1 -- 172.21.15.31:0/1790353502 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba4c0996b0 con 0x55fb27547380 2023-07-29T16:41:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.488+0000 7fba50ff9700 1 -- 172.21.15.31:0/1790353502 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c099910 con 0x55fb27547380 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.488+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1790353502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 msgr2=0x55fb27546d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.488+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27546d60 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fba4c095bb0 tx=0x7fba4c099550 comp rx=0 tx=0).stop 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1790353502 shutdown_connections 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27548770 0x55fb2755e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb27555bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/1790353502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27547380 0x55fb27546d60 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1790353502 >> 172.21.15.31:0/1790353502 conn(0x55fb27533fd0 msgr2=0x55fb2754f490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1790353502 shutdown_connections 2023-07-29T16:41:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 -- 172.21.15.31:0/1790353502 wait complete. 2023-07-29T16:41:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.489+0000 7fba77b2bb80 1 -- start start 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.490+0000 7fba77b2bb80 1 Processor -- start 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb275556d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27553ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27554220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb27555ef0 con 0x55fb27548770 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb27554760 con 0x55fb2739a6b0 2023-07-29T16:41:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb27554920 con 0x55fb27547380 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.491+0000 7fba77b2bb80 10 client.0 _ll_register_callbacks cb 0x55fb274355c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.492+0000 7fba6279c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27554220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.492+0000 7fba6279c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27554220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59186/0 (socket says 172.21.15.31:59186) 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.492+0000 7fba6279c700 1 -- 172.21.15.31:0/355758253 learned_addr learned my addr 172.21.15.31:0/355758253 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.492+0000 7fba6179a700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27553ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.492+0000 7fba61f9b700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb275556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.493+0000 7fba6279c700 1 -- 172.21.15.31:0/355758253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 msgr2=0x55fb27553ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.493+0000 7fba6279c700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27553ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.493+0000 7fba6279c700 1 -- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 msgr2=0x55fb275556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.493+0000 7fba6279c700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb275556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.493+0000 7fba6279c700 1 -- 172.21.15.31:0/355758253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c06a770 con 0x55fb27548770 2023-07-29T16:41:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.493+0000 7fba6179a700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27553ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:41:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.494+0000 7fba61f9b700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb275556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-29T16:41:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.494+0000 7fba6279c700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27554220 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fba54000f80 tx=0x7fba54020be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81505e7d81ceb9c server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.494+0000 7fba457fa700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:41:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.494+0000 7fba457fa700 1 -- 172.21.15.31:0/355758253 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54012030 con 0x55fb27548770 2023-07-29T16:41:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.495+0000 7fba457fa700 1 -- 172.21.15.31:0/355758253 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fba54020ca0 con 0x55fb27548770 2023-07-29T16:41:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.495+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba54004bb0 con 0x55fb27548770 2023-07-29T16:41:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.496+0000 7fba457fa700 1 -- 172.21.15.31:0/355758253 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba5401b040 con 0x55fb27548770 2023-07-29T16:41:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.496+0000 7fba77b2bb80 10 client.20451 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.496+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55fb272bc250 con 0x55fb27548770 2023-07-29T16:41:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba457fa700 1 -- 172.21.15.31:0/355758253 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba54004bb0 con 0x55fb27548770 2023-07-29T16:41:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba457fa700 1 -- 172.21.15.31:0/355758253 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fba5402a420 con 0x55fb27548770 2023-07-29T16:41:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba77b2bb80 10 client.20451 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba77b2bb80 10 client.20451 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba77b2bb80 20 client.20451 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb275af290 0x55fb275b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba77b2bb80 4 client.20451 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:41:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.497+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fb27488ce0 con 0x55fb275af290 2023-07-29T16:41:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.498+0000 7fba61f9b700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb275af290 0x55fb275b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.499+0000 7fba61f9b700 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb275af290 0x55fb275b16a0 crc :-1 s=READY pgs=518 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bed3e4fb978ba7d server_cookie=b70d7b70501308fa in_seq=0 out_seq=0 2023-07-29T16:41:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.499+0000 7fba457fa700 10 client.20451 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:38.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:38 smithi156 ceph-mon[129245]: pgmap v1157: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 762 op/s 2023-07-29T16:41:38.623 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:38 smithi031 ceph-mon[95504]: pgmap v1157: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 762 op/s 2023-07-29T16:41:38.784 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:38 smithi039 ceph-mon[128858]: pgmap v1157: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 762 op/s 2023-07-29T16:41:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.874+0000 7fba457fa700 1 -- 172.21.15.31:0/355758253 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fb27488ce0 con 0x55fb275af290 2023-07-29T16:41:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.874+0000 7fba457fa700 10 client.20451 handle_command_reply: tid=0 2023-07-29T16:41:38.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.874+0000 7fba77b2bb80 1 client.20451 shutdown 2023-07-29T16:41:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 msgr2=0x55fb27554220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27554220 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fba54000f80 tx=0x7fba54020be0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 shutdown_connections 2023-07-29T16:41:38.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55fb275af290 0x55fb275b16a0 crc :-1 s=READY pgs=518 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fb27548770 0x55fb27554220 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fb2739a6b0 0x55fb275556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 --2- 172.21.15.31:0/355758253 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fb27547380 0x55fb27553ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 >> 172.21.15.31:0/355758253 conn(0x55fb27533fd0 msgr2=0x55fb2755e650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.875+0000 7fba457fa700 0 client.20451 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.876+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 shutdown_connections 2023-07-29T16:41:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.876+0000 7fba77b2bb80 1 -- 172.21.15.31:0/355758253 wait complete. 2023-07-29T16:41:38.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.876+0000 7fba77b2bb80 20 client.20451 trim_cache size 0 max 16384 2023-07-29T16:41:38.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:41:38.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:41:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:41:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:41:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba480a0890 msgr2=0x7fba480a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba480a0890 0x7fba480a2d40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fba58006010 tx=0x7fba5800b260 comp rx=0 tx=0).stop 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 msgr2=0x7fba5c194490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 0x7fba5c194490 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00eb30 tx=0x7fba4c00eef0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 shutdown_connections 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fba480a0890 0x7fba480a2d40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba5c101ec0 0x7fba5c195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fba5c100340 0x7fba5c193f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 --2- 172.21.15.31:0/1503350888 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba5c101530 0x7fba5c194490 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:38.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 >> 172.21.15.31:0/1503350888 conn(0x7fba5c0fb920 msgr2=0x7fba5c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:38.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.878+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 shutdown_connections 2023-07-29T16:41:38.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:38.879+0000 7fba641ff700 1 -- 172.21.15.31:0/1503350888 wait complete. 2023-07-29T16:41:39.417 INFO:tasks.workunit.client.0.smithi031.stdout: 10 419914 32.80 MB/sec execute 563 sec latency 261.077 ms 2023-07-29T16:41:40.417 INFO:tasks.workunit.client.0.smithi031.stdout: 10 420372 32.79 MB/sec execute 564 sec latency 113.167 ms 2023-07-29T16:41:40.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:40.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:40 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:40.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:40 smithi031 ceph-mon[95504]: pgmap v1158: 97 pgs: 97 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.6 MiB/s wr, 600 op/s 2023-07-29T16:41:40.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:40.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:40 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:40.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:40 smithi039 ceph-mon[128858]: pgmap v1158: 97 pgs: 97 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.6 MiB/s wr, 600 op/s 2023-07-29T16:41:40.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:40.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:40 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:40.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:40 smithi156 ceph-mon[129245]: pgmap v1158: 97 pgs: 97 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 7.6 MiB/s wr, 600 op/s 2023-07-29T16:41:41.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 420736 32.76 MB/sec execute 565 sec latency 135.673 ms 2023-07-29T16:41:42.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 421181 32.74 MB/sec execute 566 sec latency 144.563 ms 2023-07-29T16:41:42.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:41:42.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:41:42.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:42 smithi031 ceph-mon[95504]: pgmap v1159: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 466 op/s 2023-07-29T16:41:42.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:41:42.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:41:42.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:42 smithi039 ceph-mon[128858]: pgmap v1159: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 466 op/s 2023-07-29T16:41:42.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:41:42.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:41:42.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:41:42.827 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:42 smithi156 ceph-mon[129245]: pgmap v1159: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.0 MiB/s wr, 466 op/s 2023-07-29T16:41:43.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 421825 32.73 MB/sec execute 567 sec latency 69.153 ms 2023-07-29T16:41:44.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 422644 32.75 MB/sec execute 568 sec latency 59.675 ms 2023-07-29T16:41:45.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:44 smithi031 ceph-mon[95504]: pgmap v1160: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.7 MiB/s wr, 552 op/s 2023-07-29T16:41:45.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:44 smithi039 ceph-mon[128858]: pgmap v1160: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.7 MiB/s wr, 552 op/s 2023-07-29T16:41:45.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:44 smithi156 ceph-mon[129245]: pgmap v1160: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 6.7 MiB/s wr, 552 op/s 2023-07-29T16:41:45.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 423244 32.76 MB/sec execute 569 sec latency 105.077 ms 2023-07-29T16:41:46.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 423813 32.77 MB/sec execute 570 sec latency 98.074 ms 2023-07-29T16:41:47.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:46 smithi031 ceph-mon[95504]: pgmap v1161: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 478 op/s 2023-07-29T16:41:47.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:46 smithi039 ceph-mon[128858]: pgmap v1161: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 478 op/s 2023-07-29T16:41:47.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:46 smithi156 ceph-mon[129245]: pgmap v1161: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.0 MiB/s wr, 478 op/s 2023-07-29T16:41:47.418 INFO:tasks.workunit.client.0.smithi031.stdout: 10 424339 32.75 MB/sec execute 571 sec latency 98.199 ms 2023-07-29T16:41:48.419 INFO:tasks.workunit.client.0.smithi031.stdout: 10 424873 32.75 MB/sec execute 572 sec latency 174.817 ms 2023-07-29T16:41:48.575 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:48 smithi156 ceph-mon[129245]: pgmap v1162: 97 pgs: 97 active+clean; 682 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 449 op/s 2023-07-29T16:41:48.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:48 smithi031 ceph-mon[95504]: pgmap v1162: 97 pgs: 97 active+clean; 682 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 449 op/s 2023-07-29T16:41:48.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:48 smithi039 ceph-mon[128858]: pgmap v1162: 97 pgs: 97 active+clean; 682 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 449 op/s 2023-07-29T16:41:49.420 INFO:tasks.workunit.client.0.smithi031.stdout: 10 425498 32.74 MB/sec execute 573 sec latency 60.997 ms 2023-07-29T16:41:50.420 INFO:tasks.workunit.client.0.smithi031.stdout: 10 426144 32.75 MB/sec execute 574 sec latency 43.900 ms 2023-07-29T16:41:51.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:50 smithi031 ceph-mon[95504]: pgmap v1163: 97 pgs: 97 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 406 op/s 2023-07-29T16:41:51.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:50 smithi039 ceph-mon[128858]: pgmap v1163: 97 pgs: 97 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 406 op/s 2023-07-29T16:41:51.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:50 smithi156 ceph-mon[129245]: pgmap v1163: 97 pgs: 97 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.8 MiB/s wr, 406 op/s 2023-07-29T16:41:51.420 INFO:tasks.workunit.client.0.smithi031.stdout: 10 426825 32.75 MB/sec execute 575 sec latency 73.277 ms 2023-07-29T16:41:52.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.126+0000 7fd050dff700 1 -- 172.21.15.31:0/4264386702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c0737b0 msgr2=0x7fd04c073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.126+0000 7fd050dff700 1 --2- 172.21.15.31:0/4264386702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c0737b0 0x7fd04c073c20 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd0400098f0 tx=0x7fd040009c00 comp rx=0 tx=0).stop 2023-07-29T16:41:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 -- 172.21.15.31:0/4264386702 shutdown_connections 2023-07-29T16:41:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 --2- 172.21.15.31:0/4264386702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd04c10c910 0x7fd04c10ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 --2- 172.21.15.31:0/4264386702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c074d90 0x7fd04c0731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 --2- 172.21.15.31:0/4264386702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c0737b0 0x7fd04c073c20 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 -- 172.21.15.31:0/4264386702 >> 172.21.15.31:0/4264386702 conn(0x7fd04c0fb920 msgr2=0x7fd04c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:52.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 -- 172.21.15.31:0/4264386702 shutdown_connections 2023-07-29T16:41:52.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.127+0000 7fd050dff700 1 -- 172.21.15.31:0/4264386702 wait complete. 2023-07-29T16:41:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 Processor -- start 2023-07-29T16:41:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 -- start start 2023-07-29T16:41:52.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c0737b0 0x7fd04c194820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c074d90 0x7fd04c194d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd04c10c910 0x7fd04c195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c195b60 con 0x7fd04c074d90 2023-07-29T16:41:52.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c195cd0 con 0x7fd04c0737b0 2023-07-29T16:41:52.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.129+0000 7fd050dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c195e90 con 0x7fd04c10c910 2023-07-29T16:41:52.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.130+0000 7fd049d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c074d90 0x7fd04c194d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.130+0000 7fd049d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c074d90 0x7fd04c194d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41004/0 (socket says 172.21.15.31:41004) 2023-07-29T16:41:52.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.130+0000 7fd049d9b700 1 -- 172.21.15.31:0/3569352286 learned_addr learned my addr 172.21.15.31:0/3569352286 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:52.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.130+0000 7fd04a59c700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c0737b0 0x7fd04c194820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.130+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd04c10c910 0x7fd04c195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.132+0000 7fd04a59c700 1 -- 172.21.15.31:0/3569352286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd04c10c910 msgr2=0x7fd04c195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.132+0000 7fd04a59c700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd04c10c910 0x7fd04c195370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.132+0000 7fd04a59c700 1 -- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c074d90 msgr2=0x7fd04c194d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.132+0000 7fd04a59c700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c074d90 0x7fd04c194d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.132+0000 7fd04a59c700 1 -- 172.21.15.31:0/3569352286 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04401ad10 con 0x7fd04c0737b0 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.133+0000 7fd04a59c700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c0737b0 0x7fd04c194820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd04c075160 tx=0x7fd03c00b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.133+0000 7fd03b7fe700 1 -- 172.21.15.31:0/3569352286 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c011830 con 0x7fd04c0737b0 2023-07-29T16:41:52.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.133+0000 7fd03b7fe700 1 -- 172.21.15.31:0/3569352286 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd03c011e50 con 0x7fd04c0737b0 2023-07-29T16:41:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.133+0000 7fd03b7fe700 1 -- 172.21.15.31:0/3569352286 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c010db0 con 0x7fd04c0737b0 2023-07-29T16:41:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.135+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd040009520 con 0x7fd04c0737b0 2023-07-29T16:41:52.140 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.135+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c196490 con 0x7fd04c0737b0 2023-07-29T16:41:52.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.148+0000 7fd06470db80 1 Processor -- start 2023-07-29T16:41:52.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.151+0000 7fd03b7fe700 1 -- 172.21.15.31:0/3569352286 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fd03c011990 con 0x7fd04c0737b0 2023-07-29T16:41:52.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.153+0000 7fd06470db80 1 -- start start 2023-07-29T16:41:52.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd06470db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a39f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd06470db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd06470db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3a1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd06470db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a2f79b0 con 0x55cf2a3a0cf0 2023-07-29T16:41:52.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd06470db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a2f5370 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd06470db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a2f0c60 con 0x55cf2a39fdf0 2023-07-29T16:41:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.154+0000 7fd03b7fe700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0340a0890 0x7fd0340a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.155+0000 7fd03b7fe700 1 -- 172.21.15.31:0/3569352286 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd03c0cb360 con 0x7fd04c0737b0 2023-07-29T16:41:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.155+0000 7fd04ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a39f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.155+0000 7fd04ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a39f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59158/0 (socket says 172.21.15.31:59158) 2023-07-29T16:41:52.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.155+0000 7fd04ad9d700 1 -- 172.21.15.31:0/3429611760 learned_addr learned my addr 172.21.15.31:0/3429611760 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:52.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.156+0000 7fd04ad9d700 1 -- 172.21.15.31:0/3429611760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 msgr2=0x55cf2a3a07b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-07-29T16:41:52.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.156+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a07b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.156+0000 7fd04ad9d700 1 -- 172.21.15.31:0/3429611760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a0cf0 msgr2=0x55cf2a3a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:52.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.156+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.156+0000 7fd04ad9d700 1 -- 172.21.15.31:0/3429611760 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf2a11ed80 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.156+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a39f8b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd04401a580 tx=0x7fd0440198c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4636bb024dc4fd server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.157+0000 7fd038ff9700 1 -- 172.21.15.31:0/3429611760 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044028680 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.157+0000 7fd038ff9700 1 -- 172.21.15.31:0/3429611760 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd044020040 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.157+0000 7fd038ff9700 1 -- 172.21.15.31:0/3429611760 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044027ad0 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.158+0000 7fd06470db80 1 -- 172.21.15.31:0/3429611760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 msgr2=0x55cf2a39f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.158+0000 7fd06470db80 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a39f8b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fd04401a580 tx=0x7fd0440198c0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.161+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0340a0890 0x7fd0340a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.161+0000 7fd06470db80 1 -- 172.21.15.31:0/3429611760 shutdown_connections 2023-07-29T16:41:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.161+0000 7fd06470db80 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.161+0000 7fd06470db80 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a39f8b0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.161+0000 7fd06470db80 1 --2- 172.21.15.31:0/3429611760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.161+0000 7fd06470db80 1 -- 172.21.15.31:0/3429611760 >> 172.21.15.31:0/3429611760 conn(0x55cf2a394fd0 msgr2=0x55cf2a39a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:52.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.163+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0340a0890 0x7fd0340a2d40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd04401a580 tx=0x7fd044019f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:52.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.169+0000 7fd06470db80 1 -- 172.21.15.31:0/3429611760 shutdown_connections 2023-07-29T16:41:52.174 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.171+0000 7fd06470db80 1 -- 172.21.15.31:0/3429611760 wait complete. 2023-07-29T16:41:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.172+0000 7fd06470db80 1 -- start start 2023-07-29T16:41:52.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.172+0000 7fd06470db80 1 Processor -- start 2023-07-29T16:41:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a3a8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a9430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a28d4b0 con 0x55cf2a39fdf0 2023-07-29T16:41:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a28d150 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a1748c0 con 0x55cf2a3a0cf0 2023-07-29T16:41:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.173+0000 7fd06470db80 10 client.0 _ll_register_callbacks cb 0x55cf2a2965c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.175+0000 7fd049d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.175+0000 7fd049d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41020/0 (socket says 172.21.15.31:41020) 2023-07-29T16:41:52.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.175+0000 7fd049d9b700 1 -- 172.21.15.31:0/4024024434 learned_addr learned my addr 172.21.15.31:0/4024024434 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:52.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.177+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.177+0000 7fd04a59c700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a3a8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd04a59c700 1 -- 172.21.15.31:0/4024024434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 msgr2=0x55cf2a3b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd04a59c700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd04a59c700 1 -- 172.21.15.31:0/4024024434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 msgr2=0x55cf2a3a9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd04a59c700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a9430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd04a59c700 1 -- 172.21.15.31:0/4024024434 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04401a230 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd04a59c700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a3a8ef0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00ba80 tx=0x7fd03c098850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e02e14a261fa611 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.178+0000 7fd031ffb700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:41:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.179+0000 7fd031ffb700 1 -- 172.21.15.31:0/4024024434 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c0b1220 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.179+0000 7fd031ffb700 1 -- 172.21.15.31:0/4024024434 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd03c098a00 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.180+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf2a31bcb0 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.180+0000 7fd031ffb700 1 -- 172.21.15.31:0/4024024434 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd03c09ca30 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.182+0000 7fd06470db80 10 client.30746 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:52.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.182+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55cf2a107c00 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.182+0000 7fd031ffb700 1 -- 172.21.15.31:0/4024024434 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cf2a31bcb0 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd031ffb700 1 -- 172.21.15.31:0/4024024434 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd03c0a09c0 con 0x55cf2a1fb6b0 2023-07-29T16:41:52.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd06470db80 10 client.30746 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:52.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd06470db80 10 client.30746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:52.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd06470db80 20 client.30746 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:52.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd06470db80 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a3c5bd0 0x55cf2a3c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd06470db80 4 client.30746 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:41:52.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.183+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf2a36ed10 con 0x55cf2a3c5bd0 2023-07-29T16:41:52.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.184+0000 7fd049d9b700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a3c5bd0 0x55cf2a3c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.185+0000 7fd049d9b700 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a3c5bd0 0x55cf2a3c7fe0 crc :-1 s=READY pgs=519 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f73f7ed3f644f508 server_cookie=ccebbbd15d43e1bf in_seq=0 out_seq=0 2023-07-29T16:41:52.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.185+0000 7fd031ffb700 10 client.30746 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:52.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.189+0000 7fd031ffb700 1 -- 172.21.15.31:0/4024024434 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55cf2a36ed10 con 0x55cf2a3c5bd0 2023-07-29T16:41:52.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.189+0000 7fd031ffb700 10 client.30746 handle_command_reply: tid=0 2023-07-29T16:41:52.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.189+0000 7fd06470db80 1 client.30746 shutdown 2023-07-29T16:41:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 msgr2=0x55cf2a3a8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a3a8ef0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00ba80 tx=0x7fd03c098850 comp rx=0 tx=0).stop 2023-07-29T16:41:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 shutdown_connections 2023-07-29T16:41:52.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a3c5bd0 0x55cf2a3c7fe0 crc :-1 s=READY pgs=519 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a1fb6b0 0x55cf2a3a8ef0 secure :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fd03c00ba80 tx=0x7fd03c098850 comp rx=0 tx=0).stop 2023-07-29T16:41:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 --2- 172.21.15.31:0/4024024434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 >> 172.21.15.31:0/4024024434 conn(0x55cf2a394fd0 msgr2=0x55cf2a399a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:52.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.190+0000 7fd031ffb700 0 client.30746 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.196+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 shutdown_connections 2023-07-29T16:41:52.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.197+0000 7fd06470db80 1 -- 172.21.15.31:0/4024024434 wait complete. 2023-07-29T16:41:52.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.201+0000 7fd06470db80 20 client.30746 trim_cache size 0 max 16384 2023-07-29T16:41:52.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.230+0000 7fd06470db80 1 Processor -- start 2023-07-29T16:41:52.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 -- start start 2023-07-29T16:41:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 0x55cf2a3a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a3a8ac0 con 0x55cf2a39fdf0 2023-07-29T16:41:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a3c3170 con 0x55cf2a3a7630 2023-07-29T16:41:52.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.231+0000 7fd06470db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a3a0220 con 0x55cf2a3a0cf0 2023-07-29T16:41:52.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.234+0000 7fd04ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 0x55cf2a3a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.234+0000 7fd04ad9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 0x55cf2a3a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59184/0 (socket says 172.21.15.31:59184) 2023-07-29T16:41:52.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.234+0000 7fd04ad9d700 1 -- 172.21.15.31:0/1772665111 learned_addr learned my addr 172.21.15.31:0/1772665111 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:52.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.234+0000 7fd049d9b700 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.235+0000 7fd04ad9d700 1 -- 172.21.15.31:0/1772665111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 msgr2=0x55cf2a3b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:52.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.235+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.235+0000 7fd04ad9d700 1 -- 172.21.15.31:0/1772665111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 msgr2=0x55cf2a3a70f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-07-29T16:41:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.235+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a70f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.235+0000 7fd04ad9d700 1 -- 172.21.15.31:0/1772665111 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf2a1451f0 con 0x55cf2a3a7630 2023-07-29T16:41:52.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.236+0000 7fd04ad9d700 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 0x55cf2a3a3c40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd0440275b0 tx=0x7fd04401c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e796f1b72f564396 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:52.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.236+0000 7fd0397fa700 1 -- 172.21.15.31:0/1772665111 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044043040 con 0x55cf2a3a7630 2023-07-29T16:41:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.236+0000 7fd0397fa700 1 -- 172.21.15.31:0/1772665111 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd044027e10 con 0x55cf2a3a7630 2023-07-29T16:41:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.236+0000 7fd0397fa700 1 -- 172.21.15.31:0/1772665111 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044036070 con 0x55cf2a3a7630 2023-07-29T16:41:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 -- 172.21.15.31:0/1772665111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 msgr2=0x55cf2a3a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 0x55cf2a3a3c40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd0440275b0 tx=0x7fd04401c040 comp rx=0 tx=0).stop 2023-07-29T16:41:52.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 -- 172.21.15.31:0/1772665111 shutdown_connections 2023-07-29T16:41:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a7630 0x55cf2a3a3c40 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3a70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 --2- 172.21.15.31:0/1772665111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.237+0000 7fd06470db80 1 -- 172.21.15.31:0/1772665111 >> 172.21.15.31:0/1772665111 conn(0x55cf2a394fd0 msgr2=0x55cf2a3a7ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.238+0000 7fd06470db80 1 -- 172.21.15.31:0/1772665111 shutdown_connections 2023-07-29T16:41:52.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.238+0000 7fd06470db80 1 -- 172.21.15.31:0/1772665111 wait complete. 2023-07-29T16:41:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.238+0000 7fd06470db80 1 -- start start 2023-07-29T16:41:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.239+0000 7fd06470db80 1 Processor -- start 2023-07-29T16:41:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a7630 0x55cf2a3ad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a3ad950 con 0x55cf2a3a7630 2023-07-29T16:41:52.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a417750 con 0x55cf2a3a0cf0 2023-07-29T16:41:52.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf2a417890 con 0x55cf2a39fdf0 2023-07-29T16:41:52.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.240+0000 7fd06470db80 10 client.0 _ll_register_callbacks cb 0x55cf2a2965c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:41:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.241+0000 7fd04a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.241+0000 7fd04a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3b0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:55578/0 (socket says 172.21.15.31:55578) 2023-07-29T16:41:52.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.241+0000 7fd04a59c700 1 -- 172.21.15.31:0/1782065038 learned_addr learned my addr 172.21.15.31:0/1782065038 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:41:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.241+0000 7fd04a59c700 1 -- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a0cf0 msgr2=0x55cf2a3acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.241+0000 7fd04a59c700 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd04a59c700 1 -- 172.21.15.31:0/1782065038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a7630 msgr2=0x55cf2a3ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:41:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd04a59c700 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a7630 0x55cf2a3ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd04a59c700 1 -- 172.21.15.31:0/1782065038 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf2a11ed80 con 0x55cf2a39fdf0 2023-07-29T16:41:52.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd04a59c700 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3b0260 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009fd0 tx=0x7fd03c0a0cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e07bd2d840b332d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:41:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd031ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:41:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd031ffb700 1 -- 172.21.15.31:0/1782065038 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c09c850 con 0x55cf2a39fdf0 2023-07-29T16:41:52.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd031ffb700 1 -- 172.21.15.31:0/1782065038 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fd03c098a00 con 0x55cf2a39fdf0 2023-07-29T16:41:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.242+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd044020040 con 0x55cf2a39fdf0 2023-07-29T16:41:52.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.243+0000 7fd031ffb700 1 -- 172.21.15.31:0/1782065038 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd03c098540 con 0x55cf2a39fdf0 2023-07-29T16:41:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.245+0000 7fd06470db80 10 client.30787 fetch_fsmap learned FSMap version 43 2023-07-29T16:41:52.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.245+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55cf2a11d250 con 0x55cf2a39fdf0 2023-07-29T16:41:52.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.251+0000 7fd031ffb700 1 -- 172.21.15.31:0/1782065038 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd044020040 con 0x55cf2a39fdf0 2023-07-29T16:41:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.251+0000 7fd031ffb700 1 -- 172.21.15.31:0/1782065038 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7fd03c097960 con 0x55cf2a39fdf0 2023-07-29T16:41:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.252+0000 7fd06470db80 10 client.30787 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:41:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.252+0000 7fd06470db80 10 client.30787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:41:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.252+0000 7fd06470db80 20 client.30787 populate_metadata read hostname 'smithi031' 2023-07-29T16:41:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.252+0000 7fd06470db80 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a418e70 0x55cf2a41b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:41:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.252+0000 7fd06470db80 4 client.30787 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:41:52.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.252+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cf2a2e9ce0 con 0x55cf2a418e70 2023-07-29T16:41:52.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.257+0000 7fd049d9b700 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a418e70 0x55cf2a41b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:41:52.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.260+0000 7fd049d9b700 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a418e70 0x55cf2a41b280 crc :-1 s=READY pgs=520 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f65992228bafb44d server_cookie=a732b16e45754c7d in_seq=0 out_seq=0 2023-07-29T16:41:52.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.260+0000 7fd031ffb700 10 client.30787 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:52.423 INFO:tasks.workunit.client.0.smithi031.stdout: 10 427389 32.75 MB/sec execute 576 sec latency 110.847 ms 2023-07-29T16:41:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.718+0000 7fd031ffb700 1 -- 172.21.15.31:0/1782065038 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cf2a2e9ce0 con 0x55cf2a418e70 2023-07-29T16:41:52.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.718+0000 7fd031ffb700 10 client.30787 handle_command_reply: tid=0 2023-07-29T16:41:52.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.718+0000 7fd06470db80 1 client.30787 shutdown 2023-07-29T16:41:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.722+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 msgr2=0x55cf2a3b0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.722+0000 7fd06470db80 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3b0260 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009fd0 tx=0x7fd03c0a0cf0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 shutdown_connections 2023-07-29T16:41:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd06470db80 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55cf2a418e70 0x55cf2a41b280 crc :-1 s=READY pgs=520 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd06470db80 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cf2a3a7630 0x55cf2a3ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd06470db80 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cf2a39fdf0 0x55cf2a3b0260 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd06470db80 1 --2- 172.21.15.31:0/1782065038 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf2a3a0cf0 0x55cf2a3acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 >> 172.21.15.31:0/1782065038 conn(0x55cf2a394fd0 msgr2=0x55cf2a3a3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.723+0000 7fd031ffb700 0 client.30787 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:41:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.724+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 shutdown_connections 2023-07-29T16:41:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.724+0000 7fd06470db80 1 -- 172.21.15.31:0/1782065038 wait complete. 2023-07-29T16:41:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.724+0000 7fd06470db80 20 client.30787 trim_cache size 0 max 16384 2023-07-29T16:41:52.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:41:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:41:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:41:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:41:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0340a0890 msgr2=0x7fd0340a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0340a0890 0x7fd0340a2d40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd04401a580 tx=0x7fd044019f10 comp rx=0 tx=0).stop 2023-07-29T16:41:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c0737b0 msgr2=0x7fd04c194820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:41:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c0737b0 0x7fd04c194820 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd04c075160 tx=0x7fd03c00b730 comp rx=0 tx=0).stop 2023-07-29T16:41:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 shutdown_connections 2023-07-29T16:41:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fd0340a0890 0x7fd0340a2d40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd04c10c910 0x7fd04c195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd04c0737b0 0x7fd04c194820 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 --2- 172.21.15.31:0/3569352286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd04c074d90 0x7fd04c194d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:41:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.725+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 >> 172.21.15.31:0/3569352286 conn(0x7fd04c0fb920 msgr2=0x7fd04c0fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:41:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.726+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 shutdown_connections 2023-07-29T16:41:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:41:52.726+0000 7fd050dff700 1 -- 172.21.15.31:0/3569352286 wait complete. 2023-07-29T16:41:53.191 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:52 smithi031 ceph-mon[95504]: pgmap v1164: 97 pgs: 97 active+clean; 688 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.1 MiB/s wr, 442 op/s 2023-07-29T16:41:53.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:52 smithi039 ceph-mon[128858]: pgmap v1164: 97 pgs: 97 active+clean; 688 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.1 MiB/s wr, 442 op/s 2023-07-29T16:41:53.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:52 smithi156 ceph-mon[129245]: pgmap v1164: 97 pgs: 97 active+clean; 688 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.1 MiB/s wr, 442 op/s 2023-07-29T16:41:53.423 INFO:tasks.workunit.client.0.smithi031.stdout: 10 427830 32.74 MB/sec execute 577 sec latency 328.261 ms 2023-07-29T16:41:54.423 INFO:tasks.workunit.client.0.smithi031.stdout: 10 428443 32.75 MB/sec execute 578 sec latency 83.360 ms 2023-07-29T16:41:55.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:54 smithi031 ceph-mon[95504]: pgmap v1165: 97 pgs: 97 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.5 MiB/s wr, 496 op/s 2023-07-29T16:41:55.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:54 smithi039 ceph-mon[128858]: pgmap v1165: 97 pgs: 97 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.5 MiB/s wr, 496 op/s 2023-07-29T16:41:55.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:54 smithi156 ceph-mon[129245]: pgmap v1165: 97 pgs: 97 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 3.5 MiB/s wr, 496 op/s 2023-07-29T16:41:55.424 INFO:tasks.workunit.client.0.smithi031.stdout: 10 429017 32.77 MB/sec execute 579 sec latency 67.242 ms 2023-07-29T16:41:56.424 INFO:tasks.workunit.client.0.smithi031.stdout: 10 429597 32.76 MB/sec execute 580 sec latency 61.801 ms 2023-07-29T16:41:57.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:56 smithi031 ceph-mon[95504]: pgmap v1166: 97 pgs: 97 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.2 MiB/s wr, 446 op/s 2023-07-29T16:41:57.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:56 smithi039 ceph-mon[128858]: pgmap v1166: 97 pgs: 97 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.2 MiB/s wr, 446 op/s 2023-07-29T16:41:57.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:56 smithi156 ceph-mon[129245]: pgmap v1166: 97 pgs: 97 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.2 MiB/s wr, 446 op/s 2023-07-29T16:41:57.423 INFO:tasks.workunit.client.0.smithi031.stdout: 10 430300 32.76 MB/sec execute 581 sec latency 49.983 ms 2023-07-29T16:41:58.424 INFO:tasks.workunit.client.0.smithi031.stdout: 10 431113 32.77 MB/sec execute 582 sec latency 64.968 ms 2023-07-29T16:41:58.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:41:58 smithi031 ceph-mon[95504]: pgmap v1167: 97 pgs: 97 active+clean; 687 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2023-07-29T16:41:58.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:41:58 smithi039 ceph-mon[128858]: pgmap v1167: 97 pgs: 97 active+clean; 687 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2023-07-29T16:41:58.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:41:58 smithi156 ceph-mon[129245]: pgmap v1167: 97 pgs: 97 active+clean; 687 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2023-07-29T16:41:59.423 INFO:tasks.workunit.client.0.smithi031.stdout: 10 431792 32.80 MB/sec execute 583 sec latency 67.692 ms 2023-07-29T16:42:00.424 INFO:tasks.workunit.client.0.smithi031.stdout: 10 432311 32.80 MB/sec execute 584 sec latency 164.179 ms 2023-07-29T16:42:01.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:00 smithi031 ceph-mon[95504]: pgmap v1168: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 504 op/s 2023-07-29T16:42:01.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:00 smithi039 ceph-mon[128858]: pgmap v1168: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 504 op/s 2023-07-29T16:42:01.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:00 smithi156 ceph-mon[129245]: pgmap v1168: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 504 op/s 2023-07-29T16:42:01.424 INFO:tasks.workunit.client.0.smithi031.stdout: 10 432786 32.78 MB/sec execute 585 sec latency 101.763 ms 2023-07-29T16:42:02.425 INFO:tasks.workunit.client.0.smithi031.stdout: 10 433282 32.78 MB/sec execute 586 sec latency 81.763 ms 2023-07-29T16:42:03.205 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:02 smithi031 ceph-mon[95504]: pgmap v1169: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 466 op/s 2023-07-29T16:42:03.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:02 smithi039 ceph-mon[128858]: pgmap v1169: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 466 op/s 2023-07-29T16:42:03.334 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:02 smithi156 ceph-mon[129245]: pgmap v1169: 97 pgs: 97 active+clean; 689 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.3 MiB/s wr, 466 op/s 2023-07-29T16:42:03.429 INFO:tasks.workunit.client.0.smithi031.stdout: 10 433596 32.75 MB/sec execute 587 sec latency 617.118 ms 2023-07-29T16:42:04.429 INFO:tasks.workunit.client.0.smithi031.stdout: 10 433623 32.70 MB/sec execute 588 sec latency 995.010 ms 2023-07-29T16:42:05.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:04 smithi039 ceph-mon[128858]: pgmap v1170: 97 pgs: 97 active+clean; 711 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.1 MiB/s wr, 552 op/s 2023-07-29T16:42:05.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:04 smithi156 ceph-mon[129245]: pgmap v1170: 97 pgs: 97 active+clean; 711 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.1 MiB/s wr, 552 op/s 2023-07-29T16:42:05.429 INFO:tasks.workunit.client.0.smithi031.stdout: 10 433892 32.67 MB/sec execute 589 sec latency 893.767 ms 2023-07-29T16:42:05.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:04 smithi031 ceph-mon[95504]: pgmap v1170: 97 pgs: 97 active+clean; 711 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.1 MiB/s wr, 552 op/s 2023-07-29T16:42:05.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.970+0000 7f58cedff700 1 -- 172.21.15.31:0/1876490880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c81000f0 msgr2=0x7f58c81069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:05.977 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.970+0000 7f58cedff700 1 --2- 172.21.15.31:0/1876490880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c81000f0 0x7f58c81069b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009f90 tx=0x7f58c000b330 comp rx=0 tx=0).stop 2023-07-29T16:42:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.975+0000 7f58cedff700 1 -- 172.21.15.31:0/1876490880 shutdown_connections 2023-07-29T16:42:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.975+0000 7f58cedff700 1 --2- 172.21.15.31:0/1876490880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c81000f0 0x7f58c81069b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.975+0000 7f58cedff700 1 --2- 172.21.15.31:0/1876490880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c80fe570 0x7f58c80fe980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.975+0000 7f58cedff700 1 --2- 172.21.15.31:0/1876490880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c80ffbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.975+0000 7f58cedff700 1 -- 172.21.15.31:0/1876490880 >> 172.21.15.31:0/1876490880 conn(0x7f58c80f9b70 msgr2=0x7f58c80fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:05.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.981+0000 7f58cedff700 1 -- 172.21.15.31:0/1876490880 shutdown_connections 2023-07-29T16:42:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.981+0000 7f58cedff700 1 -- 172.21.15.31:0/1876490880 wait complete. 2023-07-29T16:42:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 Processor -- start 2023-07-29T16:42:05.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 -- start start 2023-07-29T16:42:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c80fe570 0x7f58c806f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c806fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 0x7f58c80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c8071040 con 0x7f58c80fe570 2023-07-29T16:42:05.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c80711b0 con 0x7f58c81000f0 2023-07-29T16:42:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.983+0000 7f58cedff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c8071370 con 0x7f58c80ff760 2023-07-29T16:42:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.984+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c806fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.984+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 0x7f58c80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.984+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 0x7f58c80709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57526/0 (socket says 172.21.15.31:57526) 2023-07-29T16:42:05.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.984+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c806fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59870/0 (socket says 172.21.15.31:59870) 2023-07-29T16:42:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.984+0000 7f58cd39c700 1 -- 172.21.15.31:0/1431905355 learned_addr learned my addr 172.21.15.31:0/1431905355 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58c7fff700 1 -- 172.21.15.31:0/1431905355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 msgr2=0x7f58c80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:05.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58c7fff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 0x7f58c80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58c7fff700 1 -- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c80fe570 msgr2=0x7f58c806f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58c7fff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c80fe570 0x7f58c806f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58c7fff700 1 -- 172.21.15.31:0/1431905355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c0009c40 con 0x7f58c80ff760 2023-07-29T16:42:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58c7fff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c806fb10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f58b400b450 tx=0x7f58b400b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:05.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.985+0000 7f58cd39c700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 0x7f58c80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:42:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.998+0000 7f58c5ffb700 1 -- 172.21.15.31:0/1431905355 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b4011830 con 0x7f58c80ff760 2023-07-29T16:42:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.998+0000 7f58c5ffb700 1 -- 172.21.15.31:0/1431905355 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58b4011e50 con 0x7f58c80ff760 2023-07-29T16:42:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:05.998+0000 7f58c5ffb700 1 -- 172.21.15.31:0/1431905355 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b400f2e0 con 0x7f58c80ff760 2023-07-29T16:42:06.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.003+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c8071650 con 0x7f58c80ff760 2023-07-29T16:42:06.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.003+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c8071e00 con 0x7f58c80ff760 2023-07-29T16:42:06.013 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.010+0000 7f58e26b8b80 1 Processor -- start 2023-07-29T16:42:06.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 -- start start 2023-07-29T16:42:06.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6a3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6a2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f5f19b0 con 0x56431f6a3770 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f5ef370 con 0x56431f6a2380 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.011+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f5eac60 con 0x56431f4f56b0 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.012+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.013+0000 7f58ccb9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.013+0000 7f58ccb9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37874/0 (socket says 172.21.15.31:37874) 2023-07-29T16:42:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.013+0000 7f58ccb9b700 1 -- 172.21.15.31:0/2258567000 learned_addr learned my addr 172.21.15.31:0/2258567000 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.013+0000 7f58c7fff700 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58ccb9b700 1 -- 172.21.15.31:0/2258567000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 msgr2=0x56431f6a3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6a3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58ccb9b700 1 -- 172.21.15.31:0/2258567000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 msgr2=0x56431f6a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58ccb9b700 1 -- 172.21.15.31:0/2258567000 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431f418d80 con 0x56431f6a3770 2023-07-29T16:42:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58c5ffb700 1 -- 172.21.15.31:0/1431905355 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f58b4010560 con 0x7f58c80ff760 2023-07-29T16:42:06.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.017+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6a1e40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00afd0 tx=0x7f58bc00aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d460290698d6102 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:06.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.018+0000 7f58b2ffd700 1 -- 172.21.15.31:0/2258567000 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc018070 con 0x56431f6a3770 2023-07-29T16:42:06.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.018+0000 7f58b2ffd700 1 -- 172.21.15.31:0/2258567000 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58bc010040 con 0x56431f6a3770 2023-07-29T16:42:06.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.018+0000 7f58b2ffd700 1 -- 172.21.15.31:0/2258567000 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc005280 con 0x56431f6a3770 2023-07-29T16:42:06.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.020+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2258567000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 msgr2=0x56431f6a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:06.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.020+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6a1e40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00afd0 tx=0x7f58bc00aa10 comp rx=0 tx=0).stop 2023-07-29T16:42:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.020+0000 7f58c5ffb700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58b80a08e0 0x7f58b80a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.020+0000 7f58c5ffb700 1 -- 172.21.15.31:0/1431905355 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58b40d0e90 con 0x7f58c80ff760 2023-07-29T16:42:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2258567000 shutdown_connections 2023-07-29T16:42:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6a3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2258567000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6a1e40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2258567000 >> 172.21.15.31:0/2258567000 conn(0x56431f68efd0 msgr2=0x56431f694310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2258567000 shutdown_connections 2023-07-29T16:42:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.024+0000 7f58cd39c700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58b80a08e0 0x7f58b80a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.025+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2258567000 wait complete. 2023-07-29T16:42:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.025+0000 7f58e26b8b80 1 -- start start 2023-07-29T16:42:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.026+0000 7f58e26b8b80 1 Processor -- start 2023-07-29T16:42:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58cd39c700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58b80a08e0 0x7f58b80a2d90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f58c000d010 tx=0x7f58c0005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6ab930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6abe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f5874b0 con 0x56431f6a3770 2023-07-29T16:42:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f587150 con 0x56431f6a2380 2023-07-29T16:42:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f46e8c0 con 0x56431f4f56b0 2023-07-29T16:42:06.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.027+0000 7f58e26b8b80 10 client.0 _ll_register_callbacks cb 0x56431f5905c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:42:06.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.028+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.028+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6b2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37882/0 (socket says 172.21.15.31:37882) 2023-07-29T16:42:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.028+0000 7f58cd39c700 1 -- 172.21.15.31:0/2529770946 learned_addr learned my addr 172.21.15.31:0/2529770946 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58c7fff700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58cd39c700 1 -- 172.21.15.31:0/2529770946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 msgr2=0x56431f6ab930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58cd39c700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6ab930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58cd39c700 1 -- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 msgr2=0x56431f6abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58cd39c700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6abe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.029+0000 7f58cd39c700 1 -- 172.21.15.31:0/2529770946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc00a410 con 0x56431f6a3770 2023-07-29T16:42:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.030+0000 7f58cd39c700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6b2ca0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f58c000be00 tx=0x7f58c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64d77bcd6ad1bd2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.030+0000 7f58a7fff700 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2023-07-29T16:42:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.030+0000 7f58a7fff700 1 -- 172.21.15.31:0/2529770946 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c000e040 con 0x56431f6a3770 2023-07-29T16:42:06.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.031+0000 7f58a7fff700 1 -- 172.21.15.31:0/2529770946 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58c00052c0 con 0x56431f6a3770 2023-07-29T16:42:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.031+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56431f615cb0 con 0x56431f6a3770 2023-07-29T16:42:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.031+0000 7f58a7fff700 1 -- 172.21.15.31:0/2529770946 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58c0005480 con 0x56431f6a3770 2023-07-29T16:42:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.031+0000 7f58e26b8b80 10 client.20469 fetch_fsmap learned FSMap version 43 2023-07-29T16:42:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.031+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x56431f401c00 con 0x56431f6a3770 2023-07-29T16:42:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58a7fff700 1 -- 172.21.15.31:0/2529770946 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56431f615cb0 con 0x56431f6a3770 2023-07-29T16:42:06.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58a7fff700 1 -- 172.21.15.31:0/2529770946 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f58c0028b10 con 0x56431f6a3770 2023-07-29T16:42:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58e26b8b80 10 client.20469 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:42:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58e26b8b80 10 client.20469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:42:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58e26b8b80 20 client.20469 populate_metadata read hostname 'smithi031' 2023-07-29T16:42:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f6b76e0 0x56431f6b9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58e26b8b80 4 client.20469 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:42:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.032+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56431f668d10 con 0x56431f6b76e0 2023-07-29T16:42:06.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.033+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f6b76e0 0x56431f6b9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.034+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f6b76e0 0x56431f6b9af0 crc :-1 s=READY pgs=521 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d71365f118d065f server_cookie=e004f2f2c9c40985 in_seq=0 out_seq=0 2023-07-29T16:42:06.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.034+0000 7f58a7fff700 10 client.20469 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.040+0000 7f58a7fff700 1 -- 172.21.15.31:0/2529770946 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x56431f668d10 con 0x56431f6b76e0 2023-07-29T16:42:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.040+0000 7f58a7fff700 10 client.20469 handle_command_reply: tid=0 2023-07-29T16:42:06.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.040+0000 7f58e26b8b80 1 client.20469 shutdown 2023-07-29T16:42:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.040+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 msgr2=0x56431f6b2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.040+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6b2ca0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f58c000be00 tx=0x7f58c000b330 comp rx=0 tx=0).stop 2023-07-29T16:42:06.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 shutdown_connections 2023-07-29T16:42:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f6b76e0 0x56431f6b9af0 crc :-1 s=READY pgs=521 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f6a3770 0x56431f6b2ca0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f4f56b0 0x56431f6ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/2529770946 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6abe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 >> 172.21.15.31:0/2529770946 conn(0x56431f68efd0 msgr2=0x56431f693a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58a7fff700 0 client.20469 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.041+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 shutdown_connections 2023-07-29T16:42:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.042+0000 7f58e26b8b80 1 -- 172.21.15.31:0/2529770946 wait complete. 2023-07-29T16:42:06.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.042+0000 7f58e26b8b80 20 client.20469 trim_cache size 0 max 16384 2023-07-29T16:42:06.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.065+0000 7f58e26b8b80 1 Processor -- start 2023-07-29T16:42:06.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 -- start start 2023-07-29T16:42:06.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 0x56431f6a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a2380 0x56431f6b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 0x56431f6a1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f6b74a0 con 0x56431f4f56b0 2023-07-29T16:42:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f68e9d0 con 0x56431f6a3770 2023-07-29T16:42:06.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.066+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f695b40 con 0x56431f6a2380 2023-07-29T16:42:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.067+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 0x56431f6a1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.067+0000 7f58cd39c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 0x56431f6a1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57544/0 (socket says 172.21.15.31:57544) 2023-07-29T16:42:06.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.067+0000 7f58cd39c700 1 -- 172.21.15.31:0/854449756 learned_addr learned my addr 172.21.15.31:0/854449756 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:06.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.071+0000 7f58cd39c700 1 -- 172.21.15.31:0/854449756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a2380 msgr2=0x56431f6b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.071+0000 7f58cd39c700 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a2380 0x56431f6b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.071+0000 7f58cd39c700 1 -- 172.21.15.31:0/854449756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 msgr2=0x56431f6a31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.071+0000 7f58cd39c700 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 0x56431f6a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.071+0000 7f58cd39c700 1 -- 172.21.15.31:0/854449756 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431f43f1f0 con 0x56431f6a3770 2023-07-29T16:42:06.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.071+0000 7f58cd39c700 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 0x56431f6a1d60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f58c00061e0 tx=0x7f58c0009470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36f09e338282d10a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:06.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.079+0000 7f58b37fe700 1 -- 172.21.15.31:0/854449756 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c003d070 con 0x56431f6a3770 2023-07-29T16:42:06.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.079+0000 7f58b37fe700 1 -- 172.21.15.31:0/854449756 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58c001cd00 con 0x56431f6a3770 2023-07-29T16:42:06.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.079+0000 7f58b37fe700 1 -- 172.21.15.31:0/854449756 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c000e040 con 0x56431f6a3770 2023-07-29T16:42:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.080+0000 7f58e26b8b80 1 -- 172.21.15.31:0/854449756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 msgr2=0x56431f6a1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.080+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 0x56431f6a1d60 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f58c00061e0 tx=0x7f58c0009470 comp rx=0 tx=0).stop 2023-07-29T16:42:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.081+0000 7f58e26b8b80 1 -- 172.21.15.31:0/854449756 shutdown_connections 2023-07-29T16:42:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.081+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a3770 0x56431f6a1d60 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.081+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 0x56431f6a31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.081+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/854449756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a2380 0x56431f6b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.081+0000 7f58e26b8b80 1 -- 172.21.15.31:0/854449756 >> 172.21.15.31:0/854449756 conn(0x56431f68efd0 msgr2=0x56431f6a63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.082+0000 7f58e26b8b80 1 -- 172.21.15.31:0/854449756 shutdown_connections 2023-07-29T16:42:06.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.082+0000 7f58e26b8b80 1 -- 172.21.15.31:0/854449756 wait complete. 2023-07-29T16:42:06.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.082+0000 7f58e26b8b80 1 -- start start 2023-07-29T16:42:06.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 Processor -- start 2023-07-29T16:42:06.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 0x56431f6acde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6ad320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a3770 0x56431f6b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f6b21d0 con 0x56431f4f56b0 2023-07-29T16:42:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f6b2340 con 0x56431f6a2380 2023-07-29T16:42:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.083+0000 7f58e26b8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431f6b2500 con 0x56431f6a3770 2023-07-29T16:42:06.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.084+0000 7f58e26b8b80 10 client.0 _ll_register_callbacks cb 0x56431f5905c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:42:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.084+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.084+0000 7f58c7fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6ad320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:57560/0 (socket says 172.21.15.31:57560) 2023-07-29T16:42:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.084+0000 7f58c7fff700 1 -- 172.21.15.31:0/1392032390 learned_addr learned my addr 172.21.15.31:0/1392032390 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.084+0000 7f58cd39c700 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a3770 0x56431f6b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.086+0000 7f58c7fff700 1 -- 172.21.15.31:0/1392032390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a3770 msgr2=0x56431f6b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.086+0000 7f58c7fff700 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a3770 0x56431f6b1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.086+0000 7f58c7fff700 1 -- 172.21.15.31:0/1392032390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 msgr2=0x56431f6acde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.086+0000 7f58c7fff700 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 0x56431f6acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.086+0000 7f58c7fff700 1 -- 172.21.15.31:0/1392032390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56431f401c00 con 0x56431f6a2380 2023-07-29T16:42:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.087+0000 7f58c7fff700 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6ad320 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f58b400bc10 tx=0x7f58b409e430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1c8097c40b13da2 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.087+0000 7f58a7fff700 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2023-07-29T16:42:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.087+0000 7f58a7fff700 1 -- 172.21.15.31:0/1392032390 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b409dc20 con 0x56431f6a2380 2023-07-29T16:42:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.087+0000 7f58a7fff700 1 -- 172.21.15.31:0/1392032390 <== mon.2 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f58b4010380 con 0x56431f6a2380 2023-07-29T16:42:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.092+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f58bc010040 con 0x56431f6a2380 2023-07-29T16:42:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.093+0000 7f58a7fff700 1 -- 172.21.15.31:0/1392032390 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58b40a2380 con 0x56431f6a2380 2023-07-29T16:42:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.094+0000 7f58e26b8b80 10 client.30767 fetch_fsmap learned FSMap version 43 2023-07-29T16:42:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.094+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x56431f417250 con 0x56431f6a2380 2023-07-29T16:42:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58a7fff700 1 -- 172.21.15.31:0/1392032390 <== mon.2 v2:172.21.15.39:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f58bc010040 con 0x56431f6a2380 2023-07-29T16:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58a7fff700 1 -- 172.21.15.31:0/1392032390 <== mon.2 v2:172.21.15.39:3300/0 5 ==== fsmap(e 43) v1 ==== 7454+0+0 (secure 0 0 0) 0x7f58b40b4070 con 0x56431f6a2380 2023-07-29T16:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58e26b8b80 10 client.30767 fetch_fsmap finished waiting for FSMap version 43 2023-07-29T16:42:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58e26b8b80 10 client.30767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58e26b8b80 20 client.30767 populate_metadata read hostname 'smithi031' 2023-07-29T16:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f699250 0x56431f69b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58e26b8b80 4 client.30767 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:42:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.104+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56431f5e3ce0 con 0x56431f699250 2023-07-29T16:42:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.105+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f699250 0x56431f69b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.113+0000 7f58ccb9b700 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f699250 0x56431f69b660 crc :-1 s=READY pgs=522 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d675b55a39d72822 server_cookie=a5044e0f6732f3ca in_seq=0 out_seq=0 2023-07-29T16:42:06.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.114+0000 7f58a7fff700 10 client.30767 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:06.429 INFO:tasks.workunit.client.0.smithi031.stdout: 10 434122 32.64 MB/sec execute 590 sec latency 175.940 ms 2023-07-29T16:42:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.540+0000 7f58a7fff700 1 -- 172.21.15.31:0/1392032390 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56431f5e3ce0 con 0x56431f699250 2023-07-29T16:42:06.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.540+0000 7f58a7fff700 10 client.30767 handle_command_reply: tid=0 2023-07-29T16:42:06.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.544+0000 7f58e26b8b80 1 client.30767 shutdown 2023-07-29T16:42:06.554 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.551+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 msgr2=0x56431f6ad320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:06.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.551+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6ad320 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f58b400bc10 tx=0x7f58b409e430 comp rx=0 tx=0).stop 2023-07-29T16:42:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 shutdown_connections 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x56431f699250 0x56431f69b660 crc :-1 s=READY pgs=522 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56431f6a3770 0x56431f6b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56431f4f56b0 0x56431f6acde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58e26b8b80 1 --2- 172.21.15.31:0/1392032390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56431f6a2380 0x56431f6ad320 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 >> 172.21.15.31:0/1392032390 conn(0x56431f68efd0 msgr2=0x56431f6a9880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.554+0000 7f58a7fff700 0 client.30767 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.555+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 shutdown_connections 2023-07-29T16:42:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.555+0000 7f58e26b8b80 1 -- 172.21.15.31:0/1392032390 wait complete. 2023-07-29T16:42:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.555+0000 7f58e26b8b80 20 client.30767 trim_cache size 0 max 16384 2023-07-29T16:42:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:42:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:42:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:42:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:42:06.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.560+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58b80a08e0 msgr2=0x7f58b80a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.560+0000 7f58cedff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58b80a08e0 0x7f58b80a2d90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f58c000d010 tx=0x7f58c0005040 comp rx=0 tx=0).stop 2023-07-29T16:42:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.560+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 msgr2=0x7f58c806fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:06.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.560+0000 7f58cedff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c806fb10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f58b400b450 tx=0x7f58b400b810 comp rx=0 tx=0).stop 2023-07-29T16:42:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.561+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 shutdown_connections 2023-07-29T16:42:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.561+0000 7f58cedff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f58b80a08e0 0x7f58b80a2d90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.561+0000 7f58cedff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58c81000f0 0x7f58c80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.561+0000 7f58cedff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f58c80fe570 0x7f58c806f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.561+0000 7f58cedff700 1 --2- 172.21.15.31:0/1431905355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58c80ff760 0x7f58c806fb10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.561+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 >> 172.21.15.31:0/1431905355 conn(0x7f58c80f9b70 msgr2=0x7f58c80fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.562+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 shutdown_connections 2023-07-29T16:42:06.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:06.569+0000 7f58cedff700 1 -- 172.21.15.31:0/1431905355 wait complete. 2023-07-29T16:42:07.278 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:06 smithi039 ceph-mon[128858]: pgmap v1171: 97 pgs: 97 active+clean; 759 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.2 MiB/s wr, 538 op/s 2023-07-29T16:42:07.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:06 smithi156 ceph-mon[129245]: pgmap v1171: 97 pgs: 97 active+clean; 759 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.2 MiB/s wr, 538 op/s 2023-07-29T16:42:07.429 INFO:tasks.workunit.client.0.smithi031.stdout: 10 434669 32.65 MB/sec execute 591 sec latency 177.222 ms 2023-07-29T16:42:07.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:06 smithi031 ceph-mon[95504]: pgmap v1171: 97 pgs: 97 active+clean; 759 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.2 MiB/s wr, 538 op/s 2023-07-29T16:42:08.430 INFO:tasks.workunit.client.0.smithi031.stdout: 10 435432 32.64 MB/sec execute 592 sec latency 66.452 ms 2023-07-29T16:42:08.576 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:08 smithi156 ceph-mon[129245]: pgmap v1172: 97 pgs: 97 active+clean; 737 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 635 op/s 2023-07-29T16:42:08.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:08 smithi031 ceph-mon[95504]: pgmap v1172: 97 pgs: 97 active+clean; 737 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 635 op/s 2023-07-29T16:42:08.778 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:08 smithi039 ceph-mon[128858]: pgmap v1172: 97 pgs: 97 active+clean; 737 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 14 MiB/s wr, 635 op/s 2023-07-29T16:42:09.430 INFO:tasks.workunit.client.0.smithi031.stdout: 10 435793 32.63 MB/sec execute 593 sec latency 297.381 ms 2023-07-29T16:42:10.430 INFO:tasks.workunit.client.0.smithi031.stdout: 10 436391 32.63 MB/sec execute 594 sec latency 103.168 ms 2023-07-29T16:42:11.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:10 smithi031 ceph-mon[95504]: pgmap v1173: 97 pgs: 97 active+clean; 704 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 MiB/s wr, 721 op/s 2023-07-29T16:42:11.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:10 smithi039 ceph-mon[128858]: pgmap v1173: 97 pgs: 97 active+clean; 704 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 MiB/s wr, 721 op/s 2023-07-29T16:42:11.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:10 smithi156 ceph-mon[129245]: pgmap v1173: 97 pgs: 97 active+clean; 704 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 MiB/s wr, 721 op/s 2023-07-29T16:42:11.430 INFO:tasks.workunit.client.0.smithi031.stdout: 10 437088 32.65 MB/sec execute 595 sec latency 56.137 ms 2023-07-29T16:42:12.431 INFO:tasks.workunit.client.0.smithi031.stdout: 10 437674 32.65 MB/sec execute 596 sec latency 121.523 ms 2023-07-29T16:42:13.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:12 smithi031 ceph-mon[95504]: pgmap v1174: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 MiB/s wr, 744 op/s 2023-07-29T16:42:13.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:12 smithi039 ceph-mon[128858]: pgmap v1174: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 MiB/s wr, 744 op/s 2023-07-29T16:42:13.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:12 smithi156 ceph-mon[129245]: pgmap v1174: 97 pgs: 97 active+clean; 678 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 MiB/s wr, 744 op/s 2023-07-29T16:42:13.431 INFO:tasks.workunit.client.0.smithi031.stdout: 10 438197 32.64 MB/sec execute 597 sec latency 135.785 ms 2023-07-29T16:42:14.431 INFO:tasks.workunit.client.0.smithi031.stdout: 10 438798 32.63 MB/sec execute 598 sec latency 47.753 ms 2023-07-29T16:42:15.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:14 smithi031 ceph-mon[95504]: pgmap v1175: 97 pgs: 97 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 18 MiB/s wr, 819 op/s 2023-07-29T16:42:15.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:14 smithi039 ceph-mon[128858]: pgmap v1175: 97 pgs: 97 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 18 MiB/s wr, 819 op/s 2023-07-29T16:42:15.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:14 smithi156 ceph-mon[129245]: pgmap v1175: 97 pgs: 97 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 18 MiB/s wr, 819 op/s 2023-07-29T16:42:15.432 INFO:tasks.workunit.client.0.smithi031.stdout: 10 439372 32.61 MB/sec execute 599 sec latency 205.504 ms 2023-07-29T16:42:16.432 INFO:tasks.workunit.client.0.smithi031.stdout: 10 cleanup 600 sec 2023-07-29T16:42:17.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:16 smithi031 ceph-mon[95504]: pgmap v1176: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 16 MiB/s wr, 780 op/s 2023-07-29T16:42:17.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:16 smithi039 ceph-mon[128858]: pgmap v1176: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 16 MiB/s wr, 780 op/s 2023-07-29T16:42:17.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:16 smithi156 ceph-mon[129245]: pgmap v1176: 97 pgs: 97 active+clean; 685 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 16 MiB/s wr, 780 op/s 2023-07-29T16:42:17.432 INFO:tasks.workunit.client.0.smithi031.stdout: 1 cleanup 601 sec 2023-07-29T16:42:17.502 INFO:tasks.workunit.client.0.smithi031.stdout: 0 cleanup 601 sec 2023-07-29T16:42:17.502 INFO:tasks.workunit.client.0.smithi031.stdout: 2023-07-29T16:42:17.502 INFO:tasks.workunit.client.0.smithi031.stdout: Operation Count AvgLat MaxLat 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: ---------------------------------------- 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: NTCreateX 625319 3.985 1708.449 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: Close 459212 0.006 1.825 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: Rename 26500 16.423 1589.596 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: Unlink 126349 4.980 618.844 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: Deltree 10 168.033 287.116 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: Mkdir 5 0.025 0.026 2023-07-29T16:42:17.503 INFO:tasks.workunit.client.0.smithi031.stdout: Qpathinfo 566872 2.854 1757.733 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: Qfileinfo 99163 0.003 0.076 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: Qfsinfo 103980 0.887 93.917 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: Sfileinfo 50990 0.193 383.102 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: Find 219184 0.371 381.282 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: WriteX 310779 0.072 604.779 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: ReadX 980644 0.037 448.985 2023-07-29T16:42:17.504 INFO:tasks.workunit.client.0.smithi031.stdout: LockX 2038 3.448 63.885 2023-07-29T16:42:17.505 INFO:tasks.workunit.client.0.smithi031.stdout: UnlockX 2038 3.182 392.643 2023-07-29T16:42:17.505 INFO:tasks.workunit.client.0.smithi031.stdout: Flush 43859 12.560 1377.414 2023-07-29T16:42:17.505 INFO:tasks.workunit.client.0.smithi031.stdout: 2023-07-29T16:42:17.505 INFO:tasks.workunit.client.0.smithi031.stdout:Throughput 32.6141 MB/sec 10 clients 10 procs max_latency=1757.748 ms 2023-07-29T16:42:17.514 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-07-29T16:42:17.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-07-29T16:42:18.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:18 smithi031 ceph-mon[95504]: pgmap v1177: 97 pgs: 97 active+clean; 682 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 707 op/s 2023-07-29T16:42:18.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:18 smithi039 ceph-mon[128858]: pgmap v1177: 97 pgs: 97 active+clean; 682 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 707 op/s 2023-07-29T16:42:18.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:18 smithi156 ceph-mon[129245]: pgmap v1177: 97 pgs: 97 active+clean; 682 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 707 op/s 2023-07-29T16:42:19.200 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2023-07-29T16:42:19.200 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-07-29T16:42:19.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:19 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.752+0000 7f06f33ff700 1 -- 172.21.15.31:0/2500184758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec101ec0 msgr2=0x7f06ec108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.752+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2500184758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec101ec0 0x7f06ec108780 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f06e400b0a0 tx=0x7f06e400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 -- 172.21.15.31:0/2500184758 shutdown_connections 2023-07-29T16:42:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2500184758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec101ec0 0x7f06ec108780 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2500184758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec100340 0x7f06ec100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2500184758 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06ec101530 0x7f06ec101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 -- 172.21.15.31:0/2500184758 >> 172.21.15.31:0/2500184758 conn(0x7f06ec0fb920 msgr2=0x7f06ec0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:19.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 -- 172.21.15.31:0/2500184758 shutdown_connections 2023-07-29T16:42:19.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.753+0000 7f06f33ff700 1 -- 172.21.15.31:0/2500184758 wait complete. 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.754+0000 7f06f33ff700 1 Processor -- start 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.754+0000 7f06f33ff700 1 -- start start 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f33ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec100340 0x7f06ec19cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f33ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06ec101530 0x7f06ec19d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f33ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 0x7f06ec19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f33ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec19e2e0 con 0x7f06ec100340 2023-07-29T16:42:19.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f33ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec19e450 con 0x7f06ec101ec0 2023-07-29T16:42:19.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f33ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec19e610 con 0x7f06ec101530 2023-07-29T16:42:19.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f119b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec100340 0x7f06ec19cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 0x7f06ec19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f119b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec100340 0x7f06ec19cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43704/0 (socket says 172.21.15.31:43704) 2023-07-29T16:42:19.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f199c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 0x7f06ec19dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59034/0 (socket says 172.21.15.31:59034) 2023-07-29T16:42:19.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f119b700 1 -- 172.21.15.31:0/2323726023 learned_addr learned my addr 172.21.15.31:0/2323726023 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:19.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.755+0000 7f06f099a700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06ec101530 0x7f06ec19d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f199c700 1 -- 172.21.15.31:0/2323726023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06ec101530 msgr2=0x7f06ec19d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f199c700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06ec101530 0x7f06ec19d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f199c700 1 -- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec100340 msgr2=0x7f06ec19cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f199c700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec100340 0x7f06ec19cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f199c700 1 -- 172.21.15.31:0/2323726023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e8009520 con 0x7f06ec101ec0 2023-07-29T16:42:19.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f199c700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 0x7f06ec19dc00 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f06e400b070 tx=0x7f06e400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:19.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06e27fc700 1 -- 172.21.15.31:0/2323726023 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e400e040 con 0x7f06ec101ec0 2023-07-29T16:42:19.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e4009d00 con 0x7f06ec101ec0 2023-07-29T16:42:19.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06e27fc700 1 -- 172.21.15.31:0/2323726023 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06e4003e10 con 0x7f06ec101ec0 2023-07-29T16:42:19.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06e27fc700 1 -- 172.21.15.31:0/2323726023 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e40052a0 con 0x7f06ec101ec0 2023-07-29T16:42:19.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.756+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec1a4e50 con 0x7f06ec101ec0 2023-07-29T16:42:19.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.761+0000 7f0706c37b80 1 Processor -- start 2023-07-29T16:42:19.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.761+0000 7f06e27fc700 1 -- 172.21.15.31:0/2323726023 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f06e401d480 con 0x7f06ec101ec0 2023-07-29T16:42:19.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 -- start start 2023-07-29T16:42:19.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fa48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fa57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fa6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727efc9b0 con 0x55a727fa5cf0 2023-07-29T16:42:19.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727efa370 con 0x55a727e006b0 2023-07-29T16:42:19.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.762+0000 7f0706c37b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727ef5c60 con 0x55a727fa4df0 2023-07-29T16:42:19.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fa48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fa48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59044/0 (socket says 172.21.15.31:59044) 2023-07-29T16:42:19.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 -- 172.21.15.31:0/3446013476 learned_addr learned my addr 172.21.15.31:0/3446013476 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f119b700 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fa57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fa6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 -- 172.21.15.31:0/3446013476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 msgr2=0x55a727fa57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fa57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 -- 172.21.15.31:0/3446013476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 msgr2=0x55a727fa6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fa6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 -- 172.21.15.31:0/3446013476 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a727d23d80 con 0x55a727e006b0 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f119b700 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fa57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.763+0000 7f06f099a700 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fa48b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x55a727fa4610 tx=0x7f06dc005d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab47e3b14c611b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f06e27fc700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06d80a0790 0x7f06d80a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f06d77fe700 1 -- 172.21.15.31:0/3446013476 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc012dd0 con 0x55a727e006b0 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f06d77fe700 1 -- 172.21.15.31:0/3446013476 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06dc004cf0 con 0x55a727e006b0 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f06d77fe700 1 -- 172.21.15.31:0/3446013476 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc00f3f0 con 0x55a727e006b0 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f06e27fc700 1 -- 172.21.15.31:0/2323726023 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f06e40cce90 con 0x7f06ec101ec0 2023-07-29T16:42:19.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f06f119b700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06d80a0790 0x7f06d80a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f0706c37b80 1 -- 172.21.15.31:0/3446013476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 msgr2=0x55a727fa48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.764+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fa48b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x55a727fa4610 tx=0x7f06dc005d20 comp rx=0 tx=0).stop 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 -- 172.21.15.31:0/3446013476 shutdown_connections 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fa6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fa48b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3446013476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fa57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f06f119b700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06d80a0790 0x7f06d80a2c40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f06e8005950 tx=0x7f06e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 -- 172.21.15.31:0/3446013476 >> 172.21.15.31:0/3446013476 conn(0x55a727f99fd0 msgr2=0x55a727f9f300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 -- 172.21.15.31:0/3446013476 shutdown_connections 2023-07-29T16:42:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 -- 172.21.15.31:0/3446013476 wait complete. 2023-07-29T16:42:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 -- start start 2023-07-29T16:42:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.765+0000 7f0706c37b80 1 Processor -- start 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fadf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa4df0 0x55a727fae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727e924b0 con 0x55a727fa5cf0 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727e92150 con 0x55a727fa4df0 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727d798c0 con 0x55a727e006b0 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f0706c37b80 10 client.0 _ll_register_callbacks cb 0x55a727e9b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:42:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f06f119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fadf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f06f119b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fadf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36720/0 (socket says 172.21.15.31:36720) 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f06f119b700 1 -- 172.21.15.31:0/3290308189 learned_addr learned my addr 172.21.15.31:0/3290308189 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f06f199c700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.766+0000 7f06f099a700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa4df0 0x55a727fae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06f119b700 1 -- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa4df0 msgr2=0x55a727fae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06f119b700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa4df0 0x55a727fae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06f119b700 1 -- 172.21.15.31:0/3290308189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 msgr2=0x55a727fb54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06f119b700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06f119b700 1 -- 172.21.15.31:0/3290308189 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e8009b10 con 0x55a727e006b0 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06f119b700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fadf70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f06e80119f0 tx=0x7f06e80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aff14bc3d587f452 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06d4ff9700 1 -- 172.21.15.31:0/3290308189 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8017070 con 0x55a727e006b0 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f06d4ff9700 1 -- 172.21.15.31:0/3290308189 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f06e8004020 con 0x55a727e006b0 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.767+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a727f20cb0 con 0x55a727e006b0 2023-07-29T16:42:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f06d4ff9700 1 -- 172.21.15.31:0/3290308189 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06e8012330 con 0x55a727e006b0 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 10 client.30811 fetch_fsmap learned FSMap version 44 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x7f06dc01a040 con 0x55a727e006b0 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f06d4ff9700 1 -- 172.21.15.31:0/3290308189 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a727f20cb0 con 0x55a727e006b0 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f06d4ff9700 1 -- 172.21.15.31:0/3290308189 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f06e801f630 con 0x55a727e006b0 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 10 client.30811 fetch_fsmap finished waiting for FSMap version 44 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 10 client.30811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:42:19.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 20 client.30811 populate_metadata read hostname 'smithi031' 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a727fb9de0 0x55a727fbc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 4 client.30811 mds_command: new command op to 24581 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.768+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a727f73d10 con 0x55a727fb9de0 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.769+0000 7f06f099a700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a727fb9de0 0x55a727fbc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.769+0000 7f06f099a700 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a727fb9de0 0x55a727fbc1f0 crc :-1 s=READY pgs=524 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85cce8c00857dc4a server_cookie=b15ff7a1e1783df6 in_seq=0 out_seq=0 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.769+0000 7f06d4ff9700 10 client.30811 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f06d4ff9700 1 -- 172.21.15.31:0/3290308189 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17888 (crc 0 0 0) 0x55a727f73d10 con 0x55a727fb9de0 2023-07-29T16:42:19.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f06d4ff9700 10 client.30811 handle_command_reply: tid=0 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 client.30811 shutdown 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 msgr2=0x55a727fadf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fadf70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f06e80119f0 tx=0x7f06e80047c0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 shutdown_connections 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a727fb9de0 0x55a727fbc1f0 crc :-1 s=READY pgs=524 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fadf70 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 --2- 172.21.15.31:0/3290308189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa4df0 0x55a727fae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f06d4ff9700 0 client.30811 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.770+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 >> 172.21.15.31:0/3290308189 conn(0x55a727f99fd0 msgr2=0x55a727f9ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.771+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 shutdown_connections 2023-07-29T16:42:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.771+0000 7f0706c37b80 1 -- 172.21.15.31:0/3290308189 wait complete. 2023-07-29T16:42:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.771+0000 7f0706c37b80 20 client.30811 trim_cache size 0 max 16384 2023-07-29T16:42:19.782 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:19 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 Processor -- start 2023-07-29T16:42:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 -- start start 2023-07-29T16:42:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fb3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa4df0 0x55a727fa8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa5cf0 0x55a727fa9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727fadb40 con 0x55a727fa4df0 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727f20cb0 con 0x55a727fa5cf0 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f0706c37b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727f99a20 con 0x55a727e006b0 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f06f099a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa4df0 0x55a727fa8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f06f099a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa4df0 0x55a727fa8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43736/0 (socket says 172.21.15.31:43736) 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f06f099a700 1 -- 172.21.15.31:0/532318239 learned_addr learned my addr 172.21.15.31:0/532318239 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f06f119b700 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fb3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.786+0000 7f06f199c700 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa5cf0 0x55a727fa9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06f119b700 1 -- 172.21.15.31:0/532318239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa5cf0 msgr2=0x55a727fa9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06f119b700 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa5cf0 0x55a727fa9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06f119b700 1 -- 172.21.15.31:0/532318239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa4df0 msgr2=0x55a727fa8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06f119b700 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa4df0 0x55a727fa8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06f119b700 1 -- 172.21.15.31:0/532318239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a727d0cc00 con 0x55a727e006b0 2023-07-29T16:42:19.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06f119b700 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fb3470 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f06e801fa10 tx=0x7f06e801f7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d956b2a8b07f8f47 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06d7fff700 1 -- 172.21.15.31:0/532318239 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80121c0 con 0x55a727e006b0 2023-07-29T16:42:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06d7fff700 1 -- 172.21.15.31:0/532318239 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06e8012320 con 0x55a727e006b0 2023-07-29T16:42:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.787+0000 7f06d7fff700 1 -- 172.21.15.31:0/532318239 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80056f0 con 0x55a727e006b0 2023-07-29T16:42:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 -- 172.21.15.31:0/532318239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 msgr2=0x55a727fb3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fb3470 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f06e801fa10 tx=0x7f06e801f7b0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 -- 172.21.15.31:0/532318239 shutdown_connections 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727fa5cf0 0x55a727fa9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727e006b0 0x55a727fb3470 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 --2- 172.21.15.31:0/532318239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa4df0 0x55a727fa8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 -- 172.21.15.31:0/532318239 >> 172.21.15.31:0/532318239 conn(0x55a727f99fd0 msgr2=0x55a727fac0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 -- 172.21.15.31:0/532318239 shutdown_connections 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 -- 172.21.15.31:0/532318239 wait complete. 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.788+0000 7f0706c37b80 1 -- start start 2023-07-29T16:42:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 Processor -- start 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fb1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fb1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727fb2c90 con 0x55a727fa5cf0 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727fb2e00 con 0x55a727e006b0 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a727fb2fc0 con 0x55a727fa4df0 2023-07-29T16:42:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f0706c37b80 10 client.0 _ll_register_callbacks cb 0x55a727e9b5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f199c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fb1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f099a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fb1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f199c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43738/0 (socket says 172.21.15.31:43738) 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f099a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fb1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:36738/0 (socket says 172.21.15.31:36738) 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f199c700 1 -- 172.21.15.31:0/2053543628 learned_addr learned my addr 172.21.15.31:0/2053543628 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.789+0000 7f06f119b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fb1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:59076/0 (socket says 172.21.15.31:59076) 2023-07-29T16:42:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06f099a700 1 -- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 msgr2=0x55a727fb1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06f099a700 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fb1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06f099a700 1 -- 172.21.15.31:0/2053543628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 msgr2=0x55a727fb2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06f099a700 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06f099a700 1 -- 172.21.15.31:0/2053543628 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06dc01a040 con 0x55a727fa4df0 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06f099a700 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fb1b80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0127e0 tx=0x7f06dc005510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85a184ecc7b7c67a server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2023-07-29T16:42:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06d4ff9700 1 -- 172.21.15.31:0/2053543628 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc024040 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06d4ff9700 1 -- 172.21.15.31:0/2053543628 <== mon.1 v2:172.21.15.156:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f06dc017040 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06dc004cf0 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06d4ff9700 1 -- 172.21.15.31:0/2053543628 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06dc005680 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 10 client.30817 fetch_fsmap learned FSMap version 44 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) v3 -- 0x55a727d4a1f0 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06d4ff9700 1 -- 172.21.15.31:0/2053543628 <== mon.1 v2:172.21.15.156:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f06dc004cf0 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f06d4ff9700 1 -- 172.21.15.31:0/2053543628 <== mon.1 v2:172.21.15.156:3300/0 5 ==== fsmap(e 44) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f06dc003680 con 0x55a727fa4df0 2023-07-29T16:42:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 10 client.30817 fetch_fsmap finished waiting for FSMap version 44 2023-07-29T16:42:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 10 client.30817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-07-29T16:42:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 20 client.30817 populate_metadata read hostname 'smithi031' 2023-07-29T16:42:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a72800c7b0 0x55a72800ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 4 client.30817 mds_command: new command op to 24581 tid=0[{"prefix": "flush journal"}] 2023-07-29T16:42:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.791+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 --> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a727eeece0 con 0x55a72800c7b0 2023-07-29T16:42:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.792+0000 7f06f119b700 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a72800c7b0 0x55a72800ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.792+0000 7f06f119b700 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a72800c7b0 0x55a72800ebc0 crc :-1 s=READY pgs=525 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d7e8cc2a89d8dcf server_cookie=8a094ce0a9097387 in_seq=0 out_seq=0 2023-07-29T16:42:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:19.792+0000 7f06d4ff9700 10 client.30817 ms_handle_connect on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:19.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:19 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.137+0000 7f06d4ff9700 1 -- 172.21.15.31:0/2053543628 <== mds.0 v2:172.21.15.39:6838/2751828361 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a727eeece0 con 0x55a72800c7b0 2023-07-29T16:42:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.137+0000 7f06d4ff9700 10 client.30817 handle_command_reply: tid=0 2023-07-29T16:42:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.137+0000 7f0706c37b80 1 client.30817 shutdown 2023-07-29T16:42:20.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.137+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 msgr2=0x55a727fb1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.137+0000 7f0706c37b80 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fb1b80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0127e0 tx=0x7f06dc005510 comp rx=0 tx=0).stop 2023-07-29T16:42:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 shutdown_connections 2023-07-29T16:42:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:6838/2751828361,v1:172.21.15.39:6839/2751828361] conn(0x55a72800c7b0 0x55a72800ebc0 crc :-1 s=READY pgs=525 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a727fa5cf0 0x55a727fb2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a727e006b0 0x55a727fb1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 --2- 172.21.15.31:0/2053543628 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a727fa4df0 0x55a727fb1b80 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f06d4ff9700 0 client.30817 ms_handle_reset on v2:172.21.15.39:6838/2751828361 2023-07-29T16:42:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 >> 172.21.15.31:0/2053543628 conn(0x55a727f99fd0 msgr2=0x55a727fa8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 shutdown_connections 2023-07-29T16:42:20.143 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 1 -- 172.21.15.31:0/2053543628 wait complete. 2023-07-29T16:42:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.138+0000 7f0706c37b80 20 client.30817 trim_cache size 0 max 16384 2023-07-29T16:42:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2023-07-29T16:42:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2023-07-29T16:42:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2023-07-29T16:42:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2023-07-29T16:42:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06d80a0790 msgr2=0x7f06d80a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06d80a0790 0x7f06d80a2c40 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f06e8005950 tx=0x7f06e8009260 comp rx=0 tx=0).stop 2023-07-29T16:42:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 msgr2=0x7f06ec19dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 0x7f06ec19dc00 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f06e400b070 tx=0x7f06e400bc80 comp rx=0 tx=0).stop 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 shutdown_connections 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f06d80a0790 0x7f06d80a2c40 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f06ec101ec0 0x7f06ec19dc00 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f06ec100340 0x7f06ec19cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 --2- 172.21.15.31:0/2323726023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06ec101530 0x7f06ec19d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 >> 172.21.15.31:0/2323726023 conn(0x7f06ec0fb920 msgr2=0x7f06ec0fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:20.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 shutdown_connections 2023-07-29T16:42:20.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2023-07-29T16:42:20.139+0000 7f06f33ff700 1 -- 172.21.15.31:0/2323726023 wait complete. 2023-07-29T16:42:20.384 DEBUG:teuthology.parallel:result is None 2023-07-29T16:42:20.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-07-29T16:42:20.466 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-07-29T16:42:20.466 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-07-29T16:42:20.482 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T16:42:20.494 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2023-07-29T16:42:20.507 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2023-07-29T16:42:20.507 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_61140164d2579c30ffff3970615e0f82e40c6470/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/github.com_ceph_ceph-c_0484097b0572e96f42cf6402cbe0ac8dcb046577/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2023-07-29T16:42:20.508 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-07-29T16:42:20.517 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:42:20.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:20 smithi031 ceph-mon[95504]: pgmap v1178: 97 pgs: 97 active+clean; 644 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.0 MiB/s wr, 576 op/s 2023-07-29T16:42:20.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 -- 172.21.15.31:0/2865081198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 msgr2=0x7f9128100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 --2- 172.21.15.31:0/2865081198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128100750 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f911c009990 tx=0x7f911c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 -- 172.21.15.31:0/2865081198 shutdown_connections 2023-07-29T16:42:20.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 --2- 172.21.15.31:0/2865081198 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128101ec0 0x7f9128108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 --2- 172.21.15.31:0/2865081198 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128100750 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 --2- 172.21.15.31:0/2865081198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9128101530 0x7f9128101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 -- 172.21.15.31:0/2865081198 >> 172.21.15.31:0/2865081198 conn(0x7f91280fb920 msgr2=0x7f91280fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:20.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 -- 172.21.15.31:0/2865081198 shutdown_connections 2023-07-29T16:42:20.740 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.735+0000 7f912efff700 1 -- 172.21.15.31:0/2865081198 wait complete. 2023-07-29T16:42:20.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.736+0000 7f912efff700 1 Processor -- start 2023-07-29T16:42:20.742 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.736+0000 7f912efff700 1 -- start start 2023-07-29T16:42:20.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912efff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:20.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912efff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9128101530 0x7f9128073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:20.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912efff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128101ec0 0x7f9128073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:20.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912efff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128073e20 con 0x7f9128100340 2023-07-29T16:42:20.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912efff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9128073f90 con 0x7f9128101530 2023-07-29T16:42:20.743 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912efff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91281a00c0 con 0x7f9128101ec0 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912cd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43482/0 (socket says 172.21.15.31:43482) 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912cd9b700 1 -- 172.21.15.31:0/372950411 learned_addr learned my addr 172.21.15.31:0/372950411 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f9127fff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9128101530 0x7f9128073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.737+0000 7f912d59c700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128101ec0 0x7f9128073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912cd9b700 1 -- 172.21.15.31:0/372950411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128101ec0 msgr2=0x7f9128073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.744 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912cd9b700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128101ec0 0x7f9128073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912cd9b700 1 -- 172.21.15.31:0/372950411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9128101530 msgr2=0x7f9128073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912cd9b700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9128101530 0x7f9128073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912cd9b700 1 -- 172.21.15.31:0/372950411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c0095f0 con 0x7f9128100340 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912cd9b700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128074b00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f911c00b2e0 tx=0x7f911c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c01d070 con 0x7f9128100340 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f911c005030 con 0x7f9128100340 2023-07-29T16:42:20.745 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91281a0340 con 0x7f9128100340 2023-07-29T16:42:20.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c004bb0 con 0x7f9128100340 2023-07-29T16:42:20.746 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.738+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91281a0700 con 0x7f9128100340 2023-07-29T16:42:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.739+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9128107830 con 0x7f9128100340 2023-07-29T16:42:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.741+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f911c0051a0 con 0x7f9128100340 2023-07-29T16:42:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.743+0000 7f9125ffb700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f91180a07e0 0x7f91180a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.743+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f911c0cd0b0 con 0x7f9128100340 2023-07-29T16:42:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.743+0000 7f9127fff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f91180a07e0 0x7f91180a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:20.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.743+0000 7f9127fff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f91180a07e0 0x7f91180a2c90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9114009c60 tx=0x7f9114009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:20.748 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.745+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f911c096b80 con 0x7f9128100340 2023-07-29T16:42:20.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:20 smithi039 ceph-mon[128858]: pgmap v1178: 97 pgs: 97 active+clean; 644 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.0 MiB/s wr, 576 op/s 2023-07-29T16:42:20.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:20 smithi156 ceph-mon[129245]: pgmap v1178: 97 pgs: 97 active+clean; 644 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 7.0 MiB/s wr, 576 op/s 2023-07-29T16:42:20.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.955+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f912802ce80 con 0x7f9128100340 2023-07-29T16:42:20.960 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.957+0000 7f9125ffb700 1 -- 172.21.15.31:0/372950411 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f911c02f090 con 0x7f9128100340 2023-07-29T16:42:20.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:42:20.961 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:35:01.881611+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:42:20.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f91180a07e0 msgr2=0x7f91180a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f91180a07e0 0x7f91180a2c90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9114009c60 tx=0x7f9114009260 comp rx=0 tx=0).stop 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 msgr2=0x7f9128074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128074b00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f911c00b2e0 tx=0x7f911c009580 comp rx=0 tx=0).stop 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 shutdown_connections 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f91180a07e0 0x7f91180a2c90 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128101ec0 0x7f9128073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9128100340 0x7f9128074b00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 --2- 172.21.15.31:0/372950411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9128101530 0x7f9128073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 >> 172.21.15.31:0/372950411 conn(0x7f91280fb920 msgr2=0x7f9128104970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:20.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 shutdown_connections 2023-07-29T16:42:20.965 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:20.960+0000 7f912efff700 1 -- 172.21.15.31:0/372950411 wait complete. 2023-07-29T16:42:20.983 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:42:20.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:42:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/4074667609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08101510 msgr2=0x7fbe08101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/4074667609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08101510 0x7fbe08101960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009990 tx=0x7fbdf8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/4074667609 shutdown_connections 2023-07-29T16:42:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/4074667609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/4074667609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe08100320 0x7fbe08100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/4074667609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08101510 0x7fbe08101960 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/4074667609 >> 172.21.15.31:0/4074667609 conn(0x7fbe080fb920 msgr2=0x7fbe080fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:21.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/4074667609 shutdown_connections 2023-07-29T16:42:21.191 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.187+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/4074667609 wait complete. 2023-07-29T16:42:21.192 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.188+0000 7fbe0f9ff700 1 Processor -- start 2023-07-29T16:42:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.188+0000 7fbe0f9ff700 1 -- start start 2023-07-29T16:42:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0f9ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08100320 0x7fbe08074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0f9ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe08101510 0x7fbe08073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0f9ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0f9ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe08073e20 con 0x7fbe08101ea0 2023-07-29T16:42:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0f9ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe08073f90 con 0x7fbe08100320 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0f9ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe081a00c0 con 0x7fbe08101510 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0df9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0d79b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08100320 0x7fbe08074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0df9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43502/0 (socket says 172.21.15.31:43502) 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0df9c700 1 -- 172.21.15.31:0/433857924 learned_addr learned my addr 172.21.15.31:0/433857924 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0cf9a700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe08101510 0x7fbe08073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.189+0000 7fbe0d79b700 1 -- 172.21.15.31:0/433857924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe08101510 msgr2=0x7fbe08073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.194 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0d79b700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe08101510 0x7fbe08073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0d79b700 1 -- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 msgr2=0x7fbe08073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0d79b700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0d79b700 1 -- 172.21.15.31:0/433857924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04009520 con 0x7fbe08100320 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0df9c700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0d79b700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08100320 0x7fbe08074b00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400e9c0 tx=0x7fbe0400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400cb70 con 0x7fbe08100320 2023-07-29T16:42:21.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe0400ee60 con 0x7fbe08100320 2023-07-29T16:42:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04018680 con 0x7fbe08100320 2023-07-29T16:42:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf80095f0 con 0x7fbe08100320 2023-07-29T16:42:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.190+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe081a06d0 con 0x7fbe08100320 2023-07-29T16:42:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.191+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0810baf0 con 0x7fbe08100320 2023-07-29T16:42:21.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.193+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fbe04010430 con 0x7fbe08100320 2023-07-29T16:42:21.197 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.195+0000 7fbdfe7fc700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbdf40a0790 0x7fbdf40a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.195+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fbe04014070 con 0x7fbe08100320 2023-07-29T16:42:21.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.195+0000 7fbe0cf9a700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbdf40a0790 0x7fbdf40a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.196+0000 7fbe0cf9a700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbdf40a0790 0x7fbdf40a2c40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800b620 tx=0x7fbdf8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:21.199 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.197+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fbe04095e90 con 0x7fbe08100320 2023-07-29T16:42:21.432 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.429+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe08107260 con 0x7fbe08100320 2023-07-29T16:42:21.433 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.430+0000 7fbdfe7fc700 1 -- 172.21.15.31:0/433857924 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe0400ce60 con 0x7fbe08100320 2023-07-29T16:42:21.433 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:42:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbdf40a0790 msgr2=0x7fbdf40a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbdf40a0790 0x7fbdf40a2c40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800b620 tx=0x7fbdf8005960 comp rx=0 tx=0).stop 2023-07-29T16:42:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08100320 msgr2=0x7fbe08074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.435 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08100320 0x7fbe08074b00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400e9c0 tx=0x7fbe0400ecd0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 shutdown_connections 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fbdf40a0790 0x7fbdf40a2c40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe08101ea0 0x7fbe08073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbe08100320 0x7fbe08074b00 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 --2- 172.21.15.31:0/433857924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbe08101510 0x7fbe08073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 >> 172.21.15.31:0/433857924 conn(0x7fbe080fb920 msgr2=0x7fbe080fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.432+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 shutdown_connections 2023-07-29T16:42:21.437 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.433+0000 7fbe0f9ff700 1 -- 172.21.15.31:0/433857924 wait complete. 2023-07-29T16:42:21.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:42:21.662 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 -- 172.21.15.31:0/1731506953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868100330 msgr2=0x7f1868100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 --2- 172.21.15.31:0/1731506953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868100330 0x7f1868100740 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1858009990 tx=0x7f1858009ca0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 -- 172.21.15.31:0/1731506953 shutdown_connections 2023-07-29T16:42:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 --2- 172.21.15.31:0/1731506953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1868101eb0 0x7f1868108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.663 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 --2- 172.21.15.31:0/1731506953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868100330 0x7f1868100740 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 --2- 172.21.15.31:0/1731506953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868101520 0x7f1868101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 -- 172.21.15.31:0/1731506953 >> 172.21.15.31:0/1731506953 conn(0x7f18680fb970 msgr2=0x7f18680fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:21.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 -- 172.21.15.31:0/1731506953 shutdown_connections 2023-07-29T16:42:21.664 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.660+0000 7f186e5ff700 1 -- 172.21.15.31:0/1731506953 wait complete. 2023-07-29T16:42:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 Processor -- start 2023-07-29T16:42:21.666 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 -- start start 2023-07-29T16:42:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 0x7f1868074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868101520 0x7f1868073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1868101eb0 0x7f1868073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868073e20 con 0x7f1868101eb0 2023-07-29T16:42:21.667 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1868073f90 con 0x7f1868101520 2023-07-29T16:42:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.661+0000 7f186e5ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18681a0120 con 0x7f1868100330 2023-07-29T16:42:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 0x7f1868074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.668 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 0x7f1868074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35104/0 (socket says 172.21.15.31:35104) 2023-07-29T16:42:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 -- 172.21.15.31:0/3801959656 learned_addr learned my addr 172.21.15.31:0/3801959656 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f185ffff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868101520 0x7f1868073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f186cb9c700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1868101eb0 0x7f1868073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 -- 172.21.15.31:0/3801959656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868101520 msgr2=0x7f1868073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868101520 0x7f1868073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 -- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1868101eb0 msgr2=0x7f1868073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1868101eb0 0x7f1868073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 -- 172.21.15.31:0/3801959656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580095f0 con 0x7f1868100330 2023-07-29T16:42:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1867fff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 0x7f1868074b00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1858004730 tx=0x7f1858004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185801d070 con 0x7f1868100330 2023-07-29T16:42:21.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1858021410 con 0x7f1868100330 2023-07-29T16:42:21.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1858017650 con 0x7f1868100330 2023-07-29T16:42:21.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.662+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18681a03a0 con 0x7f1868100330 2023-07-29T16:42:21.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.663+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18681a0770 con 0x7f1868100330 2023-07-29T16:42:21.671 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.663+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186810bb50 con 0x7f1868100330 2023-07-29T16:42:21.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.666+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f1858021580 con 0x7f1868100330 2023-07-29T16:42:21.672 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.667+0000 7f1865ffb700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18540a0790 0x7f18540a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:21.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.667+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f18580cd5d0 con 0x7f1868100330 2023-07-29T16:42:21.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.667+0000 7f185ffff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18540a0790 0x7f18540a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:21.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.668+0000 7f185ffff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18540a0790 0x7f18540a2c40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1850009c60 tx=0x7f1850009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:21.673 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.670+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f1858097040 con 0x7f1868100330 2023-07-29T16:42:21.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:21 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:21.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:21 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/372950411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:42:21.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:21 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:21.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:21 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/372950411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:42:21.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:21 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:21.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:21 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/372950411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:42:21.880 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.877+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1868107220 con 0x7f1868100330 2023-07-29T16:42:21.881 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.878+0000 7f1865ffb700 1 -- 172.21.15.31:0/3801959656 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f185809a660 con 0x7f1868100330 2023-07-29T16:42:21.881 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:42:21.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18540a0790 msgr2=0x7f18540a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18540a0790 0x7f18540a2c40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1850009c60 tx=0x7f1850009260 comp rx=0 tx=0).stop 2023-07-29T16:42:21.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 msgr2=0x7f1868074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:21.883 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 0x7f1868074b00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1858004730 tx=0x7f1858004760 comp rx=0 tx=0).stop 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 shutdown_connections 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f18540a0790 0x7f18540a2c40 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1868101eb0 0x7f1868073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1868100330 0x7f1868074b00 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 --2- 172.21.15.31:0/3801959656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1868101520 0x7f1868073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 >> 172.21.15.31:0/3801959656 conn(0x7f18680fb970 msgr2=0x7f18680fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 shutdown_connections 2023-07-29T16:42:21.884 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:21.880+0000 7f186e5ff700 1 -- 172.21.15.31:0/3801959656 wait complete. 2023-07-29T16:42:21.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:42:22.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 -- 172.21.15.31:0/437628459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 msgr2=0x7fb1ac108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/437628459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac108760 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400b0a0 tx=0x7fb1a400b3b0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.111 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 -- 172.21.15.31:0/437628459 shutdown_connections 2023-07-29T16:42:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/437628459 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac108760 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/437628459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1ac100320 0x7fb1ac100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/437628459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac101510 0x7fb1ac101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 -- 172.21.15.31:0/437628459 >> 172.21.15.31:0/437628459 conn(0x7fb1ac0fb920 msgr2=0x7fb1ac0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 -- 172.21.15.31:0/437628459 shutdown_connections 2023-07-29T16:42:22.112 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.108+0000 7fb1b0fff700 1 -- 172.21.15.31:0/437628459 wait complete. 2023-07-29T16:42:22.115 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.109+0000 7fb1b0fff700 1 Processor -- start 2023-07-29T16:42:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 -- start start 2023-07-29T16:42:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 0x7fb1ac198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1ac101510 0x7fb1ac1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac19fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac19ffe0 con 0x7fb1ac100320 2023-07-29T16:42:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac1a0120 con 0x7fb1ac101ea0 2023-07-29T16:42:22.117 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1b0fff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac1a02b0 con 0x7fb1ac101510 2023-07-29T16:42:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1aa59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 0x7fb1ac198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1aa59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 0x7fb1ac198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43522/0 (socket says 172.21.15.31:43522) 2023-07-29T16:42:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1aa59c700 1 -- 172.21.15.31:0/924128912 learned_addr learned my addr 172.21.15.31:0/924128912 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:22.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.110+0000 7fb1aad9d700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac19fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aa59c700 1 -- 172.21.15.31:0/924128912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1ac101510 msgr2=0x7fb1ac1989a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-07-29T16:42:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aa59c700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1ac101510 0x7fb1ac1989a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aa59c700 1 -- 172.21.15.31:0/924128912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 msgr2=0x7fb1ac19fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aa59c700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac19fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aa59c700 1 -- 172.21.15.31:0/924128912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a4009d00 con 0x7fb1ac100320 2023-07-29T16:42:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aad9d700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:42:22.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1aa59c700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 0x7fb1ac198460 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00b500 tx=0x7fb19c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c011830 con 0x7fb1ac100320 2023-07-29T16:42:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac1a0590 con 0x7fb1ac100320 2023-07-29T16:42:22.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb19c011e50 con 0x7fb1ac100320 2023-07-29T16:42:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c00f2e0 con 0x7fb1ac100320 2023-07-29T16:42:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.111+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac1a0aa0 con 0x7fb1ac100320 2023-07-29T16:42:22.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.112+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac10bbe0 con 0x7fb1ac100320 2023-07-29T16:42:22.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.114+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fb19c010580 con 0x7fb1ac100320 2023-07-29T16:42:22.123 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.116+0000 7fb1a37fe700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb1980a0790 0x7fb1980a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.116+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fb19c0cb560 con 0x7fb1ac100320 2023-07-29T16:42:22.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.116+0000 7fb1a3fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb1980a0790 0x7fb1980a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.117+0000 7fb1a3fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb1980a0790 0x7fb1980a2c40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb194009af0 tx=0x7fb194009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:22.124 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.120+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fb19c094f40 con 0x7fb1ac100320 2023-07-29T16:42:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.331+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb1ac1a1140 con 0x7fb1ac100320 2023-07-29T16:42:22.334 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.332+0000 7fb1a37fe700 1 -- 172.21.15.31:0/924128912 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb19c015020 con 0x7fb1ac100320 2023-07-29T16:42:22.334 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:42:22.336 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb1980a0790 msgr2=0x7fb1980a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb1980a0790 0x7fb1980a2c40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb194009af0 tx=0x7fb194009260 comp rx=0 tx=0).stop 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 msgr2=0x7fb1ac198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 0x7fb1ac198460 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00b500 tx=0x7fb19c00b810 comp rx=0 tx=0).stop 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 shutdown_connections 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fb1980a0790 0x7fb1980a2c40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb1ac101ea0 0x7fb1ac19fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb1ac100320 0x7fb1ac198460 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 --2- 172.21.15.31:0/924128912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1ac101510 0x7fb1ac1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 >> 172.21.15.31:0/924128912 conn(0x7fb1ac0fb920 msgr2=0x7fb1ac0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 shutdown_connections 2023-07-29T16:42:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.334+0000 7fb1b0fff700 1 -- 172.21.15.31:0/924128912 wait complete. 2023-07-29T16:42:22.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-07-29T16:42:22.560 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 -- 172.21.15.31:0/664409333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c100340 msgr2=0x7f659c100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 --2- 172.21.15.31:0/664409333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c100340 0x7f659c100750 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6598009990 tx=0x7f6598009ca0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 -- 172.21.15.31:0/664409333 shutdown_connections 2023-07-29T16:42:22.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 --2- 172.21.15.31:0/664409333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f659c101ec0 0x7f659c108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 --2- 172.21.15.31:0/664409333 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c100340 0x7f659c100750 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.561 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 --2- 172.21.15.31:0/664409333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c101530 0x7f659c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 -- 172.21.15.31:0/664409333 >> 172.21.15.31:0/664409333 conn(0x7f659c0fb920 msgr2=0x7f659c0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:22.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 -- 172.21.15.31:0/664409333 shutdown_connections 2023-07-29T16:42:22.562 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.557+0000 7f65a45ff700 1 -- 172.21.15.31:0/664409333 wait complete. 2023-07-29T16:42:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.558+0000 7f65a45ff700 1 Processor -- start 2023-07-29T16:42:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.558+0000 7f65a45ff700 1 -- start start 2023-07-29T16:42:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a45ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 0x7f659c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a45ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c101530 0x7f659c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a45ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f659c101ec0 0x7f659c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.565 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a45ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c19fe00 con 0x7f659c101530 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a45ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c19ff40 con 0x7f659c101ec0 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a45ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c1a00d0 con 0x7f659c100340 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a239b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 0x7f659c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a239b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 0x7f659c198280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:35126/0 (socket says 172.21.15.31:35126) 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a239b700 1 -- 172.21.15.31:0/1234371245 learned_addr learned my addr 172.21.15.31:0/1234371245 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a1b9a700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c101530 0x7f659c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.566 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a2b9c700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f659c101ec0 0x7f659c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a239b700 1 -- 172.21.15.31:0/1234371245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f659c101ec0 msgr2=0x7f659c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.559+0000 7f65a239b700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f659c101ec0 0x7f659c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a239b700 1 -- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c101530 msgr2=0x7f659c1987c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a239b700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c101530 0x7f659c1987c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a239b700 1 -- 172.21.15.31:0/1234371245 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980095f0 con 0x7f659c100340 2023-07-29T16:42:22.567 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a1b9a700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c101530 0x7f659c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a239b700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 0x7f659c198280 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f659800b620 tx=0x7f6598005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659801d070 con 0x7f659c100340 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6598021410 con 0x7f659c100340 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c1a0350 con 0x7f659c100340 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598017650 con 0x7f659c100340 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.560+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c1a0770 con 0x7f659c100340 2023-07-29T16:42:22.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.561+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c10bb60 con 0x7f659c100340 2023-07-29T16:42:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.564+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f6598021580 con 0x7f659c100340 2023-07-29T16:42:22.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.565+0000 7f65937fe700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f65880a07e0 0x7f65880a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:42:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.565+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f65980cd650 con 0x7f659c100340 2023-07-29T16:42:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.565+0000 7f65a1b9a700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f65880a07e0 0x7f65880a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:42:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.566+0000 7f65a1b9a700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f65880a07e0 0x7f65880a2c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f658c010420 tx=0x7f658c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:42:22.570 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.567+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f65980970c0 con 0x7f659c100340 2023-07-29T16:42:22.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/433857924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:22 smithi031 ceph-mon[95504]: pgmap v1179: 97 pgs: 97 active+clean; 617 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2023-07-29T16:42:22.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3801959656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:22 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/924128912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/433857924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:22 smithi039 ceph-mon[128858]: pgmap v1179: 97 pgs: 97 active+clean; 617 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2023-07-29T16:42:22.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3801959656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.780 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:22 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/924128912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/433857924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:22 smithi156 ceph-mon[129245]: pgmap v1179: 97 pgs: 97 active+clean; 617 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 426 op/s 2023-07-29T16:42:22.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3801959656' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:22 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/924128912' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:42:22.837 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.835+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f659c10bcf0 con 0x7f659c100340 2023-07-29T16:42:22.850 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.847+0000 7f65937fe700 1 -- 172.21.15.31:0/1234371245 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) v1 ==== 81+0+730551 (secure 0 0 0) 0x7f6598026020 con 0x7f659c100340 2023-07-29T16:42:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:42:22.864 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:42:22.864 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 18, 2023-07-29T16:42:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "active_gid": 14152, 2023-07-29T16:42:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "active_name": "x", 2023-07-29T16:42:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addrs": { 2023-07-29T16:42:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2023-07-29T16:42:22.865 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2023-07-29T16:42:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2023-07-29T16:42:22.866 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4163929785 2023-07-29T16:42:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2023-07-29T16:42:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2023-07-29T16:42:22.867 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 4163929785 2023-07-29T16:42:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addr": "172.21.15.31:6801/4163929785", 2023-07-29T16:42:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: "active_change": "2023-07-29T16:05:49.191983+0000", 2023-07-29T16:42:22.868 INFO:teuthology.orchestra.run.smithi031.stdout: "active_mgr_features": 4540138322906710015, 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: "available": true, 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: "standbys": [ 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 24109, 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "z", 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540138322906710015, 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2023-07-29T16:42:22.869 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:22.873 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.876 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:22.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.885 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.885 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2023-07-29T16:42:22.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2023-07-29T16:42:22.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2023-07-29T16:42:22.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.889 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2023-07-29T16:42:22.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2023-07-29T16:42:22.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:22.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2023-07-29T16:42:22.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2023-07-29T16:42:22.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.892 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2023-07-29T16:42:22.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2023-07-29T16:42:22.893 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2023-07-29T16:42:22.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.896 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2023-07-29T16:42:22.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2023-07-29T16:42:22.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:22.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2023-07-29T16:42:22.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.900 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.900 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2023-07-29T16:42:22.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2023-07-29T16:42:22.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:22.902 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2023-07-29T16:42:22.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.906 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.907 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.908 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2023-07-29T16:42:22.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.912 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2023-07-29T16:42:22.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2023-07-29T16:42:22.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2023-07-29T16:42:22.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:22.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-07-29T16:42:22.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:22.919 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:22.924 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.928 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2023-07-29T16:42:22.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap" 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:22.932 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.933 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2023-07-29T16:42:22.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2023-07-29T16:42:22.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:22.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:22.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.941 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.942 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2023-07-29T16:42:22.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2023-07-29T16:42:22.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-07-29T16:42:22.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.944 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2023-07-29T16:42:22.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-07-29T16:42:22.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.946 INFO:teuthology.orchestra.run.smithi031.stdout: "alertmanager_web_password": { 2023-07-29T16:42:22.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alertmanager_web_password", 2023-07-29T16:42:22.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Alertmanager web password", 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "alertmanager_web_user": { 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alertmanager_web_user", 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Alertmanager web user", 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2023-07-29T16:42:22.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2023-07-29T16:42:22.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2023-07-29T16:42:22.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2023-07-29T16:42:22.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:22.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph:v18", 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-07-29T16:42:22.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.966 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2023-07-29T16:42:22.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2023-07-29T16:42:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-07-29T16:42:22.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.975 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2023-07-29T16:42:22.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.979 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:22.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-07-29T16:42:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.984 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2023-07-29T16:42:22.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2023-07-29T16:42:22.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2023-07-29T16:42:22.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:22.999 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.000 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.002 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2023-07-29T16:42:23.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-07-29T16:42:23.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.008 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.009 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2023-07-29T16:42:23.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.012 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-07-29T16:42:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.020 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.021 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_web_password": { 2023-07-29T16:42:23.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_web_password", 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus web password", 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_web_user": { 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_web_user", 2023-07-29T16:42:23.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus web user", 2023-07-29T16:42:23.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.031 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2023-07-29T16:42:23.033 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.034 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2023-07-29T16:42:23.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2023-07-29T16:42:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2023-07-29T16:42:23.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2023-07-29T16:42:23.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.049 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.051 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.054 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.056 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.057 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2023-07-29T16:42:23.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2023-07-29T16:42:23.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2023-07-29T16:42:23.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2023-07-29T16:42:23.063 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-07-29T16:42:23.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2023-07-29T16:42:23.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.070 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.074 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-07-29T16:42:23.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.083 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.084 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-07-29T16:42:23.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2023-07-29T16:42:23.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.094 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.096 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-07-29T16:42:23.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-07-29T16:42:23.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.103 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.104 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.105 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-07-29T16:42:23.108 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.109 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.114 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.115 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-07-29T16:42:23.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.120 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-07-29T16:42:23.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.122 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.123 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-07-29T16:42:23.124 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.126 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.129 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.131 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2023-07-29T16:42:23.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.138 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2023-07-29T16:42:23.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.147 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.150 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.151 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.152 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.155 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.156 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.157 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2023-07-29T16:42:23.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.159 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.162 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.164 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.165 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2023-07-29T16:42:23.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.171 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.174 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2023-07-29T16:42:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2023-07-29T16:42:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.177 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.181 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.183 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.184 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2023-07-29T16:42:23.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.190 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2023-07-29T16:42:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.194 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.195 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.196 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.197 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.198 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.199 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.201 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.205 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.207 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.208 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2023-07-29T16:42:23.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.212 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2023-07-29T16:42:23.214 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2023-07-29T16:42:23.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.216 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2023-07-29T16:42:23.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.219 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.220 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.223 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.225 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.226 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.227 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.228 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.233 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2023-07-29T16:42:23.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2023-07-29T16:42:23.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.241 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.243 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.247 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.248 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.253 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.258 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.259 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.262 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.265 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.271 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.273 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2023-07-29T16:42:23.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.278 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.283 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-07-29T16:42:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2023-07-29T16:42:23.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.291 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2023-07-29T16:42:23.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2023-07-29T16:42:23.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2023-07-29T16:42:23.295 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.297 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.300 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.305 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.307 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.314 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.315 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.321 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.324 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2023-07-29T16:42:23.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.328 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.333 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.336 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2023-07-29T16:42:23.337 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.340 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.342 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.345 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.346 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.348 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.350 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.357 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2023-07-29T16:42:23.358 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.360 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.361 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.364 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "noautoscale": { 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "noautoscale", 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "global autoscale flag", 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.369 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2023-07-29T16:42:23.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.372 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.379 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.383 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2023-07-29T16:42:23.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2023-07-29T16:42:23.390 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.391 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.393 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.394 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.395 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.397 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.398 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.400 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2023-07-29T16:42:23.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2023-07-29T16:42:23.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2023-07-29T16:42:23.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2023-07-29T16:42:23.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.413 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2023-07-29T16:42:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.418 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.423 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.424 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.425 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2023-07-29T16:42:23.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.432 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2023-07-29T16:42:23.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.437 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.439 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.440 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.441 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.442 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.444 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.445 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.446 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.449 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.450 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.451 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.452 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.454 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.455 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.456 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.458 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: "drive_group_interval": { 2023-07-29T16:42:23.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "drive_group_interval", 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300.0", 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.460 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.462 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.467 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.468 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2023-07-29T16:42:23.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.472 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.474 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.477 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.479 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.484 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2023-07-29T16:42:23.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.494 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2023-07-29T16:42:23.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.498 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2023-07-29T16:42:23.504 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.505 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.506 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.508 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.508 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.508 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.508 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.510 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.511 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.512 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.514 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.515 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.516 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.517 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.518 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.519 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.520 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.524 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.525 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.526 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.527 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.528 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.530 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.532 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.534 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.535 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.539 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.541 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.543 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.544 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.547 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.548 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.551 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.552 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2023-07-29T16:42:23.554 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2023-07-29T16:42:23.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-07-29T16:42:23.557 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.559 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.560 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.561 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2023-07-29T16:42:23.563 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.566 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2023-07-29T16:42:23.569 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.570 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.571 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2023-07-29T16:42:23.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.574 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.576 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.577 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.578 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.581 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.584 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2023-07-29T16:42:23.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.589 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.590 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.592 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.594 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.595 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.597 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.599 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2023-07-29T16:42:23.600 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.602 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.603 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.606 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.609 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2023-07-29T16:42:23.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.611 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.612 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.613 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:23.614 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:23.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.621 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.623 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.625 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.626 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.628 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2023-07-29T16:42:23.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.631 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.633 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 24113, 2023-07-29T16:42:23.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "y", 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540138322906710015, 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.635 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.638 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.639 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.641 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.643 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.644 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.645 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2023-07-29T16:42:23.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.651 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.653 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2023-07-29T16:42:23.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2023-07-29T16:42:23.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.660 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.661 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2023-07-29T16:42:23.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2023-07-29T16:42:23.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.666 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2023-07-29T16:42:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.670 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2023-07-29T16:42:23.671 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2023-07-29T16:42:23.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2023-07-29T16:42:23.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.676 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.677 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.678 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.679 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.681 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.683 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2023-07-29T16:42:23.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.687 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap" 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.690 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2023-07-29T16:42:23.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.693 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:23.694 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.695 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.696 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2023-07-29T16:42:23.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.698 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2023-07-29T16:42:23.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.701 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "alertmanager_web_password": { 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alertmanager_web_password", 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Alertmanager web password", 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "alertmanager_web_user": { 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alertmanager_web_user", 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Alertmanager web user", 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2023-07-29T16:42:23.707 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:23.709 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-07-29T16:42:23.710 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.711 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.712 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.713 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.715 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2023-07-29T16:42:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.717 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.718 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph:v18", 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2023-07-29T16:42:23.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.724 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2023-07-29T16:42:23.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2023-07-29T16:42:23.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.728 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-07-29T16:42:23.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.730 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.731 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2023-07-29T16:42:23.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.733 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2023-07-29T16:42:23.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:23.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-07-29T16:42:23.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.739 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2023-07-29T16:42:23.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.742 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.747 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.748 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.749 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2023-07-29T16:42:23.750 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2023-07-29T16:42:23.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.753 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.757 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2023-07-29T16:42:23.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.763 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.764 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-07-29T16:42:23.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.770 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2023-07-29T16:42:23.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.774 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2023-07-29T16:42:23.776 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2023-07-29T16:42:23.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_web_password": { 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_web_password", 2023-07-29T16:42:23.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus web password", 2023-07-29T16:42:23.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_web_user": { 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_web_user", 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus web user", 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.785 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2023-07-29T16:42:23.786 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.787 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.788 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2023-07-29T16:42:23.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.794 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2023-07-29T16:42:23.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-07-29T16:42:23.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2023-07-29T16:42:23.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.804 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.806 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.808 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.809 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.810 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2023-07-29T16:42:23.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-07-29T16:42:23.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2023-07-29T16:42:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.824 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.827 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.828 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-07-29T16:42:23.829 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2023-07-29T16:42:23.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.838 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.841 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.842 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.843 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.844 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-07-29T16:42:23.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.846 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.847 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.848 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.849 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.852 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-07-29T16:42:23.854 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.861 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-07-29T16:42:23.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.868 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.869 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-07-29T16:42:23.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-07-29T16:42:23.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-07-29T16:42:23.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.877 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-07-29T16:42:23.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.885 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2023-07-29T16:42:23.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.887 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2023-07-29T16:42:23.888 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:23.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.893 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2023-07-29T16:42:23.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.900 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2023-07-29T16:42:23.901 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.902 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.903 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.905 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.907 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.908 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.909 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.910 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2023-07-29T16:42:23.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.918 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2023-07-29T16:42:23.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.929 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2023-07-29T16:42:23.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.933 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.934 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.938 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2023-07-29T16:42:23.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.946 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.947 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2023-07-29T16:42:23.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-07-29T16:42:23.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2023-07-29T16:42:23.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.953 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.955 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.960 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.961 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2023-07-29T16:42:23.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2023-07-29T16:42:23.969 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.970 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2023-07-29T16:42:23.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.972 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2023-07-29T16:42:23.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2023-07-29T16:42:23.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.978 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.979 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:23.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.982 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.984 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.985 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.986 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.987 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2023-07-29T16:42:23.989 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.990 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:23.991 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2023-07-29T16:42:23.995 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.996 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:23.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:23.998 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:23.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.003 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.004 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.006 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.008 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.009 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.010 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.013 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.022 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.026 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.028 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.029 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2023-07-29T16:42:24.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.033 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.038 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2023-07-29T16:42:24.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2023-07-29T16:42:24.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.047 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2023-07-29T16:42:24.048 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.055 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.060 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.062 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.063 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.065 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.068 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2023-07-29T16:42:24.070 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.072 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.073 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.076 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.080 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.083 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.088 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.089 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2023-07-29T16:42:24.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.092 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.093 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.094 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.099 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.100 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.105 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.108 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.109 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.110 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.111 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.112 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.113 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.115 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.118 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.119 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.120 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "noautoscale": { 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "noautoscale", 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "global autoscale flag", 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:24.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.126 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.128 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.129 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.134 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.136 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.138 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.140 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.141 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.142 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.145 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.149 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.150 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.153 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.157 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.158 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.159 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2023-07-29T16:42:24.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-07-29T16:42:24.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.164 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2023-07-29T16:42:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.167 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2023-07-29T16:42:24.170 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.171 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.173 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.176 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.177 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.178 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.179 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2023-07-29T16:42:24.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.182 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.186 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.187 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2023-07-29T16:42:24.189 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.190 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.192 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.195 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.197 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.199 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2023-07-29T16:42:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.207 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.213 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "drive_group_interval": { 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "drive_group_interval", 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300.0", 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.219 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.220 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.221 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.224 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.227 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2023-07-29T16:42:24.228 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.229 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.230 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.231 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.233 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.234 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.236 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.239 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2023-07-29T16:42:24.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.242 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.247 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2023-07-29T16:42:24.248 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.253 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2023-07-29T16:42:24.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.259 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.261 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.265 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.268 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.271 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.273 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.281 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.286 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.291 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.300 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.304 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2023-07-29T16:42:24.308 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.315 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2023-07-29T16:42:24.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.319 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.324 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.325 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2023-07-29T16:42:24.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.333 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.336 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.337 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2023-07-29T16:42:24.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.346 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.348 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.349 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.352 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.356 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.358 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.362 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2023-07-29T16:42:24.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.369 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:24.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.380 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.381 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.382 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2023-07-29T16:42:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.386 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.387 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: "modules": [ 2023-07-29T16:42:24.390 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard", 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: "iostat", 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs", 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: "restful" 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.395 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.397 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.398 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.399 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.400 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.401 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2023-07-29T16:42:24.405 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.406 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.407 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2023-07-29T16:42:24.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2023-07-29T16:42:24.409 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.410 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2023-07-29T16:42:24.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.416 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.417 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.419 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:24.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.423 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2023-07-29T16:42:24.424 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2023-07-29T16:42:24.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-07-29T16:42:24.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2023-07-29T16:42:24.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.432 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.435 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.437 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.438 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.439 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.440 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2023-07-29T16:42:24.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.442 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2023-07-29T16:42:24.444 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap" 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.445 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-07-29T16:42:24.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-07-29T16:42:24.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.452 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.453 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.454 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2023-07-29T16:42:24.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-07-29T16:42:24.456 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "alertmanager_web_password": { 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alertmanager_web_password", 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Alertmanager web password", 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "alertmanager_web_user": { 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alertmanager_web_user", 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.461 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Alertmanager web user", 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2023-07-29T16:42:24.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2023-07-29T16:42:24.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2023-07-29T16:42:24.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.469 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.470 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2023-07-29T16:42:24.471 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.472 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.474 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.475 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph:v18", 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.476 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-07-29T16:42:24.477 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.478 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.479 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2023-07-29T16:42:24.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2023-07-29T16:42:24.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-07-29T16:42:24.484 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.485 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2023-07-29T16:42:24.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2023-07-29T16:42:24.494 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2023-07-29T16:42:24.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2023-07-29T16:42:24.497 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.498 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.499 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:24.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2023-07-29T16:42:24.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.504 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.507 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.508 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:24.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.510 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.511 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.513 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.514 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.515 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.518 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.519 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.520 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.521 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.522 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.528 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2023-07-29T16:42:24.530 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2023-07-29T16:42:24.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.533 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.534 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_web_password": { 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_web_password", 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus web password", 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.537 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_web_user": { 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_web_user", 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.538 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus web user", 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2023-07-29T16:42:24.541 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-07-29T16:42:24.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.544 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.544 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.544 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-07-29T16:42:24.544 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.545 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.546 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-07-29T16:42:24.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.547 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2023-07-29T16:42:24.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2023-07-29T16:42:24.548 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.548 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.548 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.549 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.549 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-07-29T16:42:24.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.549 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.550 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2023-07-29T16:42:24.550 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2023-07-29T16:42:24.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.552 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.553 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.554 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.555 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.556 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2023-07-29T16:42:24.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2023-07-29T16:42:24.558 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.559 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.561 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.562 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.563 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.565 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.566 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.569 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.571 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2023-07-29T16:42:24.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.574 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.576 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.577 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.579 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-07-29T16:42:24.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.581 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.586 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.587 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.588 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-07-29T16:42:24.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.592 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.593 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2023-07-29T16:42:24.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.597 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.598 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.599 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.600 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-07-29T16:42:24.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.604 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.605 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.606 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.607 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.610 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.611 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-07-29T16:42:24.612 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.614 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-07-29T16:42:24.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.620 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.621 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.626 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.627 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-07-29T16:42:24.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.629 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.631 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.632 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.633 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.634 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-07-29T16:42:24.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.636 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-07-29T16:42:24.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.642 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2023-07-29T16:42:24.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.650 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-07-29T16:42:24.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.652 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2023-07-29T16:42:24.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.656 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.658 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2023-07-29T16:42:24.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.661 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.663 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.667 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.668 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:24.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.678 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.679 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.680 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.681 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.682 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.684 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.685 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.686 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2023-07-29T16:42:24.688 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2023-07-29T16:42:24.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.691 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.702 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2023-07-29T16:42:24.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2023-07-29T16:42:24.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.713 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.714 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.715 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.718 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.719 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.720 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.721 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.722 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.723 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.724 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2023-07-29T16:42:24.725 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.726 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2023-07-29T16:42:24.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2023-07-29T16:42:24.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-07-29T16:42:24.730 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2023-07-29T16:42:24.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.735 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.736 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.739 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.740 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2023-07-29T16:42:24.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.745 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2023-07-29T16:42:24.746 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.747 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.748 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:24.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.750 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2023-07-29T16:42:24.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.752 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.754 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.756 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.757 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.760 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.761 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.763 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.766 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.770 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.774 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2023-07-29T16:42:24.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.776 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2023-07-29T16:42:24.777 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.778 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.779 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.781 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.784 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.787 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.788 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-07-29T16:42:24.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.791 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.792 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.794 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.795 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2023-07-29T16:42:24.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.803 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2023-07-29T16:42:24.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2023-07-29T16:42:24.805 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.808 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.809 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.810 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.811 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.812 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.814 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.817 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.818 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.820 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.822 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.823 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.826 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.829 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.834 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.837 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.838 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.839 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.840 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.841 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.842 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.844 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.844 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.844 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.844 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.846 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.848 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2023-07-29T16:42:24.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.850 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.851 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.852 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.854 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.855 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.856 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2023-07-29T16:42:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.864 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.866 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.867 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.869 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.872 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.874 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.879 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.881 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "noautoscale": { 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "noautoscale", 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.882 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "global autoscale flag", 2023-07-29T16:42:24.883 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.884 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.885 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.886 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.888 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.889 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2023-07-29T16:42:24.890 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.893 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.894 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.898 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.899 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2023-07-29T16:42:24.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.902 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:24.903 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.904 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.905 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.906 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.909 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.910 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.912 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.914 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.915 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.916 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:24.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2023-07-29T16:42:24.928 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.931 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.932 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.938 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.940 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.941 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.944 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2023-07-29T16:42:24.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.950 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.951 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.953 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.956 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.957 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.958 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.959 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2023-07-29T16:42:24.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.964 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.966 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.971 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.975 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "drive_group_interval": { 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "drive_group_interval", 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300.0", 2023-07-29T16:42:24.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.977 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.978 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.981 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.982 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.983 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.984 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2023-07-29T16:42:24.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2023-07-29T16:42:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2023-07-29T16:42:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.987 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:24.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:24.989 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.992 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:24.995 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:24.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:24.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:24.999 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.002 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.003 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.006 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2023-07-29T16:42:25.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2023-07-29T16:42:25.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2023-07-29T16:42:25.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.010 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2023-07-29T16:42:25.012 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2023-07-29T16:42:25.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.015 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2023-07-29T16:42:25.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2023-07-29T16:42:25.024 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2023-07-29T16:42:25.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.026 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.027 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.032 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.035 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.036 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.037 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.038 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.039 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.040 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.045 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2023-07-29T16:42:25.046 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.047 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.048 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.049 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.052 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.054 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.057 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.058 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:25.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.061 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.063 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.071 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.071 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.071 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.071 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.071 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.073 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2023-07-29T16:42:25.073 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.073 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.073 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.074 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2023-07-29T16:42:25.074 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2023-07-29T16:42:25.074 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2023-07-29T16:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.076 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2023-07-29T16:42:25.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2023-07-29T16:42:25.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:25.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-07-29T16:42:25.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.078 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2023-07-29T16:42:25.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2023-07-29T16:42:25.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2023-07-29T16:42:25.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-07-29T16:42:25.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.086 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-07-29T16:42:25.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:25.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.094 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.094 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.094 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.094 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.096 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.096 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.096 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.097 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2023-07-29T16:42:25.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2023-07-29T16:42:25.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.097 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.098 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.099 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2023-07-29T16:42:25.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2023-07-29T16:42:25.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.099 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.101 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.101 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.101 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.103 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.103 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.103 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.103 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.103 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.104 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.105 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.107 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.107 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.107 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.107 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.108 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.108 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.108 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.108 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.108 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.109 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.109 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.109 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.109 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.109 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.109 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.110 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2023-07-29T16:42:25.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.114 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.115 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.117 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.118 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.119 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.120 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.124 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.124 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.124 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.124 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.126 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.126 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.126 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.126 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.127 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.128 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.129 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2023-07-29T16:42:25.129 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.129 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.129 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.131 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.131 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.131 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.132 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.132 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.132 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.132 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.134 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.135 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.137 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2023-07-29T16:42:25.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.144 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2023-07-29T16:42:25.145 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.148 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.149 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2023-07-29T16:42:25.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2023-07-29T16:42:25.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2023-07-29T16:42:25.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.149 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.152 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.152 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.152 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.152 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.152 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.152 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.153 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.153 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.153 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.153 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.153 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.154 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.154 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.154 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.154 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.154 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.154 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2023-07-29T16:42:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2023-07-29T16:42:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.157 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2023-07-29T16:42:25.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2023-07-29T16:42:25.157 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.157 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.158 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2023-07-29T16:42:25.158 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2023-07-29T16:42:25.159 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2023-07-29T16:42:25.159 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2023-07-29T16:42:25.159 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2023-07-29T16:42:25.159 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2023-07-29T16:42:25.159 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2023-07-29T16:42:25.159 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.160 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.161 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.162 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2023-07-29T16:42:25.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2023-07-29T16:42:25.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.163 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2023-07-29T16:42:25.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.164 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.165 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2023-07-29T16:42:25.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2023-07-29T16:42:25.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.168 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: "services": { 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard": "https://172.21.15.31:8443/" 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: "always_on_modules": { 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: "octopus": [ 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2023-07-29T16:42:25.169 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2023-07-29T16:42:25.170 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "pacific": [ 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2023-07-29T16:42:25.171 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.172 INFO:teuthology.orchestra.run.smithi031.stdout: "quincy": [ 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2023-07-29T16:42:25.173 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "reef": [ 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2023-07-29T16:42:25.174 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2023-07-29T16:42:25.175 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: "last_failure_osd_epoch": 3, 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: "active_clients": [ 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2023-07-29T16:42:25.176 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2384937690 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.177 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "libcephsqlite", 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2401285902 2023-07-29T16:42:25.178 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2023-07-29T16:42:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2023-07-29T16:42:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 1792039579 2023-07-29T16:42:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:25.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2023-07-29T16:42:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2023-07-29T16:42:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: { 2023-07-29T16:42:25.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2023-07-29T16:42:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2023-07-29T16:42:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3868017039 2023-07-29T16:42:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:25.182 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:25.183 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-07-29T16:42:25.183 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:42:25.185 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:23 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1234371245' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-07-29T16:42:25.185 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:24 smithi156 ceph-mon[129245]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:25.185 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:24 smithi156 ceph-mon[129245]: pgmap v1180: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 410 op/s 2023-07-29T16:42:25.185 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:23 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1234371245' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-07-29T16:42:25.186 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:24 smithi039 ceph-mon[128858]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:25.186 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:24 smithi039 ceph-mon[128858]: pgmap v1180: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 410 op/s 2023-07-29T16:42:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f65880a07e0 msgr2=0x7f65880a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f65880a07e0 0x7f65880a2c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f658c010420 tx=0x7f658c0073d0 comp rx=0 tx=0).stop 2023-07-29T16:42:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 msgr2=0x7f659c198280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:42:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 0x7f659c198280 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f659800b620 tx=0x7f6598005740 comp rx=0 tx=0).stop 2023-07-29T16:42:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 shutdown_connections 2023-07-29T16:42:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f65880a07e0 0x7f65880a2c90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:25.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f659c101ec0 0x7f659c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:25.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f659c100340 0x7f659c198280 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:25.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 --2- 172.21.15.31:0/1234371245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f659c101530 0x7f659c1987c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:42:25.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 >> 172.21.15.31:0/1234371245 conn(0x7f659c0fb920 msgr2=0x7f659c106d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:42:25.188 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 shutdown_connections 2023-07-29T16:42:25.189 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:42:22.858+0000 7f65a45ff700 1 -- 172.21.15.31:0/1234371245 wait complete. 2023-07-29T16:42:25.196 DEBUG:tasks.check_counter:Getting stats from a 2023-07-29T16:42:25.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.a perf dump 2023-07-29T16:42:25.198 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:23 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1234371245' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-07-29T16:42:25.198 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:24 smithi031 ceph-mon[95504]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-07-29T16:42:25.198 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:24 smithi031 ceph-mon[95504]: pgmap v1180: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 410 op/s 2023-07-29T16:42:25.946 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:42:27.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:26 smithi031 ceph-mon[95504]: pgmap v1181: 97 pgs: 97 active+clean; 541 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 350 op/s 2023-07-29T16:42:27.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:26 smithi039 ceph-mon[128858]: pgmap v1181: 97 pgs: 97 active+clean; 541 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 350 op/s 2023-07-29T16:42:27.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:26 smithi156 ceph-mon[129245]: pgmap v1181: 97 pgs: 97 active+clean; 541 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.0 MiB/s wr, 350 op/s 2023-07-29T16:42:27.870 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:42:27.870 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2023-07-29T16:42:27.871 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.003810416, 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000592574, 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:27.872 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.873 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 484, 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 481, 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 544288, 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 275920, 2023-07-29T16:42:27.874 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.073628704, 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.033421127, 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.024985919, 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.010854045, 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:27.875 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 480, 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.073311309, 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000152731 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:27.876 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 544288, 2023-07-29T16:42:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 275920 2023-07-29T16:42:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:27.877 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 7, 2023-07-29T16:42:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 370, 2023-07-29T16:42:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3424, 2023-07-29T16:42:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 293488, 2023-07-29T16:42:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2023-07-29T16:42:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2023-07-29T16:42:27.878 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.032234433, 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.024870473, 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.002967754, 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 369, 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.035940480, 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000097399 2023-07-29T16:42:27.879 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:27.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 293488 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.881 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:27.883 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:27.884 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:27.885 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:27.886 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:27.887 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2023-07-29T16:42:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:27.888 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 149735, 2023-07-29T16:42:27.889 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2023-07-29T16:42:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:27.890 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2023-07-29T16:42:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2023-07-29T16:42:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:27.891 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2023-07-29T16:42:27.892 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:27.893 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2023-07-29T16:42:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 487, 2023-07-29T16:42:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 475861, 2023-07-29T16:42:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:27.894 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 487, 2023-07-29T16:42:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2023-07-29T16:42:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2023-07-29T16:42:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 487, 2023-07-29T16:42:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 475861, 2023-07-29T16:42:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:27.895 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:27.896 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:27.896 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:42:28.352 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-07-29T16:42:28.352 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-07-29T16:42:28.353 DEBUG:tasks.check_counter:Getting stats from d 2023-07-29T16:42:28.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.d perf dump 2023-07-29T16:42:28.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:28 smithi031 ceph-mon[95504]: pgmap v1182: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 232 op/s 2023-07-29T16:42:28.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:28 smithi039 ceph-mon[128858]: pgmap v1182: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 232 op/s 2023-07-29T16:42:28.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:28 smithi156 ceph-mon[129245]: pgmap v1182: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 232 op/s 2023-07-29T16:42:29.088 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:42:30.533 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 488, 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 485, 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 545376, 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 278288, 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2023-07-29T16:42:30.534 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.072818543, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.025053510, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.026757749, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.011796950, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 484, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.029094434, 2023-07-29T16:42:30.535 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000060112 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.536 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 545376, 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 278288 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 7, 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 374, 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3424, 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 294192, 2023-07-29T16:42:30.537 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.033537122, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.025524884, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003550999, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 373, 2023-07-29T16:42:30.538 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.053432514, 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000143250 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:30.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 294192 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2023-07-29T16:42:30.540 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001056475, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000823748, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:30.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:30.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2023-07-29T16:42:30.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:30.544 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:30.545 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:30.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:30.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:30.548 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148951, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 28, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:30.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2023-07-29T16:42:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:30.551 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 491, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 476381, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 491, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2023-07-29T16:42:30.552 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 491, 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 476381, 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:30.553 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:30.554 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:42:31.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:30 smithi031 ceph-mon[95504]: pgmap v1183: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 781 KiB/s wr, 207 op/s 2023-07-29T16:42:31.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:30 smithi039 ceph-mon[128858]: pgmap v1183: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 781 KiB/s wr, 207 op/s 2023-07-29T16:42:31.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:30 smithi156 ceph-mon[129245]: pgmap v1183: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 781 KiB/s wr, 207 op/s 2023-07-29T16:42:32.291 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-07-29T16:42:32.292 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-07-29T16:42:32.292 DEBUG:tasks.check_counter:Getting stats from g 2023-07-29T16:42:32.293 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.g perf dump 2023-07-29T16:42:33.028 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:42:33.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:32 smithi031 ceph-mon[95504]: pgmap v1184: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 296 KiB/s wr, 149 op/s 2023-07-29T16:42:33.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:32 smithi039 ceph-mon[128858]: pgmap v1184: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 296 KiB/s wr, 149 op/s 2023-07-29T16:42:33.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:32 smithi156 ceph-mon[129245]: pgmap v1184: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 296 KiB/s wr, 149 op/s 2023-07-29T16:42:34.760 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:42:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 232045, 2023-07-29T16:42:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 198560, 2023-07-29T16:42:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 59792770, 2023-07-29T16:42:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 206455247, 2023-07-29T16:42:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 57, 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 22.122327456, 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 8.408466421, 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 11.662349661, 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 3.984367652, 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:34.762 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 198558, 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 21.921092579, 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000110401 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 335178, 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.063730431, 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000190 2023-07-29T16:42:34.763 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 500264, 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 824985, 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 178074231, 2023-07-29T16:42:34.764 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 934308562, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 30, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 6, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 77.768992923, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 42.879388505, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 49.131665166, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 3.452942068, 2023-07-29T16:42:34.765 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 824982, 2023-07-29T16:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 184.587626881, 2023-07-29T16:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000223747 2023-07-29T16:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:34.766 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1039831, 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.379664598, 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000365 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 164741440, 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 700932928 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:34.767 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 92993, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 89359, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 29445185, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 216026175, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 7, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 9.974800400, 2023-07-29T16:42:34.768 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 3.823884931, 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 3.614734220, 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 2.580799652, 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 89360, 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 16.026679724, 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000179349 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.769 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 81048, 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.016395105, 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000202 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 672224, 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 304192 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.770 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 142774, 2023-07-29T16:42:34.771 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 216.867986282, 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001518959 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 87454, 2023-07-29T16:42:34.772 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 20.134647809, 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000230231 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 324598, 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 311644, 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2023-07-29T16:42:34.773 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 311644, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1503.275187590, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.004823693 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 12954, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2023-07-29T16:42:34.774 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 263, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 25, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 33, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 3597, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 2015, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 950, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 632, 2023-07-29T16:42:34.775 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 717, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 236, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 430, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 430, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 4, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 483447, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 266416, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 12950, 2023-07-29T16:42:34.776 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 49, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 2019, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 26833, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 2, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 6, 2023-07-29T16:42:34.777 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 76, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 6, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 0, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 780, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 4, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 246, 2023-07-29T16:42:34.778 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 780, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 30, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2023-07-29T16:42:34.779 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 30, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 750, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 14, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 13018, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 243743, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2023-07-29T16:42:34.780 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 91411, 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 51931, 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 15897, 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 198609 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 147, 2023-07-29T16:42:34.781 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 176, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 99, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 216, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 164, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 433, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 59807, 2023-07-29T16:42:34.782 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 59374, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2023-07-29T16:42:34.783 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 9, 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 58, 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2023-07-29T16:42:34.784 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 278923, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 167728, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 167728, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 111195, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 1024, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 318, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 189, 2023-07-29T16:42:34.785 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 189, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 129, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 1, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 329497904, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 564403367, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 0, 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2023-07-29T16:42:34.786 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 83580, 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 850.065480171, 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.010170680 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 0 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2023-07-29T16:42:34.787 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 459, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 59933, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 59474, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 45, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 456, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 411, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 3599, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 120189, 2023-07-29T16:42:34.788 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 116590, 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 305, 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 64093, 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 63788, 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 79792, 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 198964 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2023-07-29T16:42:34.789 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 446432, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 2146, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 324598, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 8338, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 14046, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 59542, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 196.427151370, 2023-07-29T16:42:34.790 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.003298967 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 5721, 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 104.769459709, 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.018313137 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2023-07-29T16:42:34.791 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 89343, 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 131.981652756, 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001477246 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.793 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2, 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000282197, 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000141098 2023-07-29T16:42:34.794 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2023-07-29T16:42:34.795 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.799 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 84288, 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 856.855868375, 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.010165810 2023-07-29T16:42:34.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 11169, 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 40.517526269, 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.003627677 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.801 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 30, 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.157589163, 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.005252972 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.802 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.803 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2023-07-29T16:42:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 28, 2023-07-29T16:42:34.804 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.680788337, 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.024313869 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1744, 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 3.041136322, 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001743770 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.806 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.807 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 59777, 2023-07-29T16:42:34.808 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 168.843733092, 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.002824560 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.809 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 1, 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 2, 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 1, 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 1, 2023-07-29T16:42:34.810 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 29131, 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 29131, 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 1715 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:34.811 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:34.812 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:34.813 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:34.814 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:34.815 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 306699, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 553, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 528, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 6, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2023-07-29T16:42:34.816 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 7920, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 167, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:34.817 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 4058202, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 101954, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2023-07-29T16:42:34.818 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 148249, 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 572292141, 2023-07-29T16:42:34.819 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 27, 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 148249, 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 148249, 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1554.941547825, 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.010488715 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2023-07-29T16:42:34.820 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 148249, 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 149164 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 148249, 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 2134, 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 146115, 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2023-07-29T16:42:34.821 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 19, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 21, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 1354, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 84934, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 789, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2023-07-29T16:42:34.822 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 25, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 59792, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 780, 2023-07-29T16:42:34.823 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 42, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2023-07-29T16:42:34.824 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2023-07-29T16:42:34.825 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2023-07-29T16:42:34.826 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 93, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 7, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 12, 2023-07-29T16:42:34.827 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 12, 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 1087, 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 0, 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 321 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2023-07-29T16:42:34.828 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 1, 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 326811 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.829 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 10, 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 10, 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 59374, 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.830 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 825298, 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 172286213, 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:34.831 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 825298, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 825298, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 172286213, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.832 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2023-07-29T16:42:34.833 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 148249, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 578288915, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 148249, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 578288915, 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:34.834 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2023-07-29T16:42:34.835 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 148249, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 148249, 2023-07-29T16:42:34.836 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 148249, 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 148249, 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 59374, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 5996774, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 59374, 2023-07-29T16:42:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1354, 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 5996774, 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.839 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x5567b11320a0": { 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 298298, 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 278923, 2023-07-29T16:42:34.840 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 560209063, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 278923, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 83580, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 559910765, 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:34.841 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.842 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:34.842 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:42:35.135 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:34 smithi031 ceph-mon[95504]: pgmap v1185: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 153 KiB/s wr, 106 op/s 2023-07-29T16:42:35.166 INFO:tasks.check_counter:Daemon mds.g mds.exported=6 2023-07-29T16:42:35.166 INFO:tasks.check_counter:Daemon mds.g mds.imported=6 2023-07-29T16:42:35.167 DEBUG:tasks.check_counter:Getting stats from j 2023-07-29T16:42:35.167 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.j perf dump 2023-07-29T16:42:35.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:34 smithi039 ceph-mon[128858]: pgmap v1185: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 153 KiB/s wr, 106 op/s 2023-07-29T16:42:35.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:34 smithi156 ceph-mon[129245]: pgmap v1185: 97 pgs: 97 active+clean; 540 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 153 KiB/s wr, 106 op/s 2023-07-29T16:42:35.929 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.a/config 2023-07-29T16:42:37.077 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:36 smithi031 ceph-mon[95504]: pgmap v1186: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1.0 MiB/s wr, 50 op/s 2023-07-29T16:42:37.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:36 smithi039 ceph-mon[128858]: pgmap v1186: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1.0 MiB/s wr, 50 op/s 2023-07-29T16:42:37.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:36 smithi156 ceph-mon[129245]: pgmap v1186: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 1.0 MiB/s wr, 50 op/s 2023-07-29T16:42:38.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:38 smithi031 ceph-mon[95504]: pgmap v1187: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 2.2 MiB/s wr, 47 op/s 2023-07-29T16:42:38.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:38 smithi039 ceph-mon[128858]: pgmap v1187: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 2.2 MiB/s wr, 47 op/s 2023-07-29T16:42:38.825 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:38 smithi156 ceph-mon[129245]: pgmap v1187: 97 pgs: 97 active+clean; 549 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 2.2 MiB/s wr, 47 op/s 2023-07-29T16:42:39.693 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2023-07-29T16:42:39.694 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.003412770, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000484413, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:39.695 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.696 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 492, 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 489, 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 546464, 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 280656, 2023-07-29T16:42:39.697 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.077420941, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.035282212, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.026142208, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.011276397, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 488, 2023-07-29T16:42:39.698 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.052216403, 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000107000 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.699 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 546464, 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 280656 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 7, 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 376, 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 3424, 2023-07-29T16:42:39.700 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 294544, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.033628984, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.026013734, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.003134234, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:39.701 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 375, 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.038339786, 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000102239 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:39.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 294544 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.703 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:39.704 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:39.705 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:39.706 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:39.707 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 147920, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 27, 2023-07-29T16:42:39.708 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:39.709 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2023-07-29T16:42:39.710 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2023-07-29T16:42:39.711 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 495, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 476901, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 495, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2023-07-29T16:42:39.712 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 495, 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 476901, 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-07-29T16:42:39.713 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-07-29T16:42:40.148 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-07-29T16:42:40.148 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-07-29T16:42:40.148 DEBUG:tasks.check_counter:Getting stats from b 2023-07-29T16:42:40.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.b perf dump 2023-07-29T16:42:41.016 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:42:41.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:40 smithi031 ceph-mon[95504]: pgmap v1188: 97 pgs: 97 active+clean; 522 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 3.5 MiB/s wr, 85 op/s 2023-07-29T16:42:41.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:40 smithi039 ceph-mon[128858]: pgmap v1188: 97 pgs: 97 active+clean; 522 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 3.5 MiB/s wr, 85 op/s 2023-07-29T16:42:41.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:40 smithi156 ceph-mon[129245]: pgmap v1188: 97 pgs: 97 active+clean; 522 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 3.5 MiB/s wr, 85 op/s 2023-07-29T16:42:42.969 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-07-29T16:42:42.969 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:42.969 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 251117, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 284601, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 63747942, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 427304218, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 504, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 3, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 16.925329971, 2023-07-29T16:42:42.970 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 6.120025085, 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 7.751953731, 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 3.372802279, 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 284602, 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 80.053836657, 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000281283 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.971 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 335837, 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.057236426, 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000170 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 2944, 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 448 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.972 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 339878, 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 347491, 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 87806237, 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 346015722, 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 11, 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:42.973 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 19.860157655, 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 7.030302721, 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 11.139335918, 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 3.004128320, 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 347491, 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 107.570337415, 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000309562 2023-07-29T16:42:42.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 510192, 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.080882842, 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000158 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 480, 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 888624 2023-07-29T16:42:42.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 655914, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 1027895, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 235690197, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 943579669, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 14, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 8, 2023-07-29T16:42:42.976 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 47.102821929, 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 21.455739434, 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 30.966799525, 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 2.324182923, 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1027890, 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 156.820803623, 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000152565 2023-07-29T16:42:42.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1235173, 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.223223299, 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000180 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 220401760, 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 749749456 2023-07-29T16:42:42.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 1, 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-MDSRank": { 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 394, 2023-07-29T16:42:42.979 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 230127, 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 400.621866719, 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.001740872 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-PurgeQueue": { 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2023-07-29T16:42:42.980 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 106323, 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 40.286351490, 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000378905 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "request": 435852, 2023-07-29T16:42:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: "reply": 394733, 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "reply_latency": { 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 394733, 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 2966.869105417, 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.007516141 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_reply": 0, 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "forward": 41119, 2023-07-29T16:42:42.982 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_complete": 15, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_keys": 6, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_commit": 5351, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_split": 24, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_merge": 40, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes": 1251, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_top": 500, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_bottom": 292, 2023-07-29T16:42:42.983 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pin_tail": 459, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pinned": 533, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_expired": 1359, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_with_caps": 89, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": 89, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "subtrees": 4, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse": 637191, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_hit": 388638, 2023-07-29T16:42:42.984 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_forward": 41115, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_discover": 14, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_dir_fetch": 0, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_remote_ino": 0, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_lock": 4034, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "load_cent": 217, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "q": 0, 2023-07-29T16:42:42.985 INFO:teuthology.orchestra.run.smithi039.stdout: "exported": 13, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "exported_inodes": 1040, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "imported": 13, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "imported_inodes": 0, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_dir_fetch": 0, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_backtrace_fetch": 1538, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_peer_discover": 0, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rfiles": 4, 2023-07-29T16:42:42.986 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rbytes": 246, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rsnaps": 0, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_fetch": 1538, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_set_tag": 0, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_repaired": 0, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_inotable_repaired": 0, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_inodes": 586, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_base_inodes": 31, 2023-07-29T16:42:42.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dirfrag_rstats": 555, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_file_inodes": 952, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_inode_file_caps": 41, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_revoke": 45058, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_grant": 269307, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_trunc": 0, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flush_ack": 0, 2023-07-29T16:42:42.988 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps": 126081, 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps_dirty": 58628, 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_cap_release": 14426, 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "process_request_cap_release": 235201 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache": { 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update": 11, 2023-07-29T16:42:42.989 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update_receipt": 15, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_try_discover": 10, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_send_discover": 19, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_handle_discover": 281, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays": 262, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_delayed": 92, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_enqueuing": 0, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_created": 67025, 2023-07-29T16:42:42.990 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_enqueued": 66763, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_reintegrated": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_migrated": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_processing": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_enqueued": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_prioritized": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_started": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_completed": 0, 2023-07-29T16:42:42.991 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_enqueue_scrub": 31, 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_exportdir": 14, 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_flush": 0, 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragmentdir": 64, 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragstats": 0, 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_inodestats": 0 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log": { 2023-07-29T16:42:42.992 INFO:teuthology.orchestra.run.smithi039.stdout: "evadd": 342308, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "evex": 341480, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "evtrm": 341480, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "ev": 828, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "evexg": 0, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "evexd": 3580, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "segadd": 492, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "segex": 491, 2023-07-29T16:42:42.993 INFO:teuthology.orchestra.run.smithi039.stdout: "segtrm": 491, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "seg": 6, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "segexg": 0, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "segexd": 5, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "expos": 721547169, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "wrpos": 722893942, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "rdpos": 0, 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "jlat": { 2023-07-29T16:42:42.994 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 128579, 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1188.192936457, 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.009240956 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: "replayed": 0 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mem": { 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: "ino": 509, 2023-07-29T16:42:42.995 INFO:teuthology.orchestra.run.smithi039.stdout: "ino+": 67709, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "ino-": 67200, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "dir": 27, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "dir+": 390, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "dir-": 363, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "dn": 1282, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "dn+": 141396, 2023-07-29T16:42:42.996 INFO:teuthology.orchestra.run.smithi039.stdout: "dn-": 140114, 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "cap": 89, 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "cap+": 69818, 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "cap-": 69729, 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "rss": 83008, 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "heap": 207156 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_server": { 2023-07-29T16:42:42.997 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_client_request": 600431, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_server_request": 34532, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_request": 435852, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_session": 10056, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_peer_request": 75168, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "req_create_latency": { 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 67409, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 470.229714767, 2023-07-29T16:42:42.998 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.006975770 2023-07-29T16:42:42.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:42.999 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getattr_latency": { 2023-07-29T16:42:42.999 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 42717, 2023-07-29T16:42:42.999 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 403.902128901, 2023-07-29T16:42:42.999 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.009455301 2023-07-29T16:42:42.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getfilelock_latency": { 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: "req_link_latency": { 2023-07-29T16:42:43.000 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookup_latency": { 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 102948, 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 409.474590382, 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.003977489 2023-07-29T16:42:43.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookuphash_latency": { 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupino_latency": { 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.002 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupname_latency": { 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupparent_latency": { 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupsnap_latency": { 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.004 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lssnap_latency": { 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.005 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mkdir_latency": { 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 119, 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1.590213530, 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.013363138 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mknod_latency": { 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mksnap_latency": { 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: "req_open_latency": { 2023-07-29T16:42:43.007 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 9, 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.019835792, 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.002203976 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: "req_readdir_latency": { 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 96807, 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1167.826301229, 2023-07-29T16:42:43.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.012063448 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rename_latency": { 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 15965, 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 94.224334041, 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.005901931 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: "req_renamesnap_latency": { 2023-07-29T16:42:43.009 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmdir_latency": { 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 64, 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.854991454, 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.013359241 2023-07-29T16:42:43.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmsnap_latency": { 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmxattr_latency": { 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.011 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setattr_latency": { 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 157, 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 6.019008633, 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.038337634 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.012 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setdirlayout_latency": { 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setfilelock_latency": { 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1560, 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 7.664289217, 2023-07-29T16:42:43.013 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.004913005 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setlayout_latency": { 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.014 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setxattr_latency": { 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 6, 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.019960026, 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.003326671 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "req_symlink_latency": { 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.015 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: "req_unlink_latency": { 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 66972, 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 405.043737445, 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.006047956 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_revoke_eviction": 0, 2023-07-29T16:42:43.016 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition_throttle": 0, 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getvxattr_latency": { 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessions": { 2023-07-29T16:42:43.017 INFO:teuthology.orchestra.run.smithi039.stdout: "session_count": 1, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "session_add": 2, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "session_remove": 1, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_open": 1, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_stale": 0, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "total_load": 29677, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "average_load": 29677, 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: "avg_session_uptime": 1725 2023-07-29T16:42:43.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:43.019 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:43.021 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:43.022 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 4175259, 2023-07-29T16:42:43.023 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 1558, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 440, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 5, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:43.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 7920, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 167, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 2709907, 2023-07-29T16:42:43.025 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 105009, 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter": { 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "op_active": 0, 2023-07-29T16:42:43.026 INFO:teuthology.orchestra.run.smithi039.stdout: "op_laggy": 0, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send": 241202, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send_bytes": 772855165, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "op_resend": 37, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "op_reply": 241202, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "op_latency": { 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 241202, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 6769.566884638, 2023-07-29T16:42:43.027 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.028065964 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: "op_inflight": 0, 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: "oplen_avg": { 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 241202, 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 319911 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: "op": 241202, 2023-07-29T16:42:43.028 INFO:teuthology.orchestra.run.smithi039.stdout: "op_r": 3316, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "op_w": 237886, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "op_rmw": 0, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "op_pg": 0, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_stat": 4988, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_create": 32621, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_read": 1726, 2023-07-29T16:42:43.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_write": 130274, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writefull": 1260, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writesame": 0, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_append": 0, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_zero": 0, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_truncate": 24, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_delete": 67827, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_mapext": 0, 2023-07-29T16:42:43.030 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_sparse_read": 0, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_clonerange": 0, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_getxattr": 1590, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_setxattr": 65242, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_cmpxattr": 0, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_rmxattr": 0, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_resetxattrs": 0, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_call": 0, 2023-07-29T16:42:43.031 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_watch": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_notify": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_src_cmpxattr": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls_filter": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_other": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_active": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_send": 0, 2023-07-29T16:42:43.032 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_resend": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_ping": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_active": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_send": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_resend": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_active": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_send": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_resend": 0, 2023-07-29T16:42:43.033 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_active": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_send": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_resend": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "command_active": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "command_send": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "command_resend": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "map_epoch": 93, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "map_full": 0, 2023-07-29T16:42:43.034 INFO:teuthology.orchestra.run.smithi039.stdout: "map_inc": 8, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_sessions": 12, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_open": 12, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_close": 0, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_laggy": 0, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_wr": 10851, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_rd": 42, 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_del": 3466 2023-07-29T16:42:43.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: "oft": { 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_objs": 1, 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_kv_pairs": 0, 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_updates": 0, 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_removes": 0 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.036 INFO:teuthology.orchestra.run.smithi039.stdout: "osd-slow-ops": { 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_ops_count": 470628 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "purge_queue": { 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops": 0, 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops_high_water": 10, 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing": 0, 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_high_water": 10, 2023-07-29T16:42:43.037 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed": 66763, 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_item_in_journal": 0 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 1246905, 2023-07-29T16:42:43.038 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 247554823, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 1246905, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 1246905, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 247554823, 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:43.039 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_bytes": { 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:43.040 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 241202, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 779597964, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 241202, 2023-07-29T16:42:43.041 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 779597964, 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.042 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_ops": { 2023-07-29T16:42:43.043 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:43.043 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 1024, 2023-07-29T16:42:43.043 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:43.043 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:43.043 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:43.043 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 241202, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 241202, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 241202, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 241202, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.044 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle": { 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:43.045 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 66763, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 6743063, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 66763, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 1695, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 6743063, 2023-07-29T16:42:43.046 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle-0x55fb70c360a0": { 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:43.047 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 342308, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 718699638, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 342308, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:43.048 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 128579, 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 718699638, 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:43.050 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-07-29T16:42:43.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:42 smithi031 ceph-mon[95504]: pgmap v1189: 97 pgs: 97 active+clean; 511 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.0 MiB/s wr, 103 op/s 2023-07-29T16:42:43.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:42 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:42:43.480 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:42 smithi156 ceph-mon[129245]: pgmap v1189: 97 pgs: 97 active+clean; 511 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.0 MiB/s wr, 103 op/s 2023-07-29T16:42:43.481 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:42 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:42:43.481 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:42 smithi039 ceph-mon[128858]: pgmap v1189: 97 pgs: 97 active+clean; 511 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 4.0 MiB/s wr, 103 op/s 2023-07-29T16:42:43.481 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:42 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-29T16:42:43.482 INFO:tasks.check_counter:Daemon mds.b mds.exported=13 2023-07-29T16:42:43.482 INFO:tasks.check_counter:Daemon mds.b mds.imported=13 2023-07-29T16:42:43.482 DEBUG:tasks.check_counter:Getting stats from e 2023-07-29T16:42:43.483 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.e perf dump 2023-07-29T16:42:44.159 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:42:44.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:43 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.278 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.278 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.278 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.279 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.279 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.279 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.279 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.279 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:43 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:44.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:43 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:45.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:44 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:42:45.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:44 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:42:45.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:44 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:42:45.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:44 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:42:45.029 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:44 smithi039 ceph-mon[128858]: pgmap v1190: 97 pgs: 97 active+clean; 491 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 5.1 MiB/s wr, 157 op/s 2023-07-29T16:42:45.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:44 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:42:45.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:44 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:42:45.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:44 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:42:45.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:44 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:42:45.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:44 smithi031 ceph-mon[95504]: pgmap v1190: 97 pgs: 97 active+clean; 491 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 5.1 MiB/s wr, 157 op/s 2023-07-29T16:42:45.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:44 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi039 to 196.2M 2023-07-29T16:42:45.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:44 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi039 to 205773209: error parsing value: Value '205773209' is below minimum 939524096 2023-07-29T16:42:45.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:44 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi031 to 222.0M 2023-07-29T16:42:45.204 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:44 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi031 to 232786534: error parsing value: Value '232786534' is below minimum 939524096 2023-07-29T16:42:45.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:44 smithi156 ceph-mon[129245]: pgmap v1190: 97 pgs: 97 active+clean; 491 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 5.1 MiB/s wr, 157 op/s 2023-07-29T16:42:45.518 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-07-29T16:42:45.518 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 1030, 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 1032, 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 115236444, 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 320340, 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 8, 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 4, 2023-07-29T16:42:45.519 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.142986923, 2023-07-29T16:42:45.520 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.024625912, 2023-07-29T16:42:45.520 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.077992628, 2023-07-29T16:42:45.520 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.031876272, 2023-07-29T16:42:45.520 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:45.521 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1030, 2023-07-29T16:42:45.521 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.116893147, 2023-07-29T16:42:45.521 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000113488 2023-07-29T16:42:45.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.522 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:45.522 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.522 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.523 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.523 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:45.523 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:45.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.524 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:45.524 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 893, 2023-07-29T16:42:45.524 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 885, 2023-07-29T16:42:45.525 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 75560460, 2023-07-29T16:42:45.525 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 402805, 2023-07-29T16:42:45.525 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 6, 2023-07-29T16:42:45.525 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:45.525 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.141581764, 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.030080443, 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.079556057, 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.025252508, 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 885, 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.120080365, 2023-07-29T16:42:45.526 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000135684 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 574080, 2023-07-29T16:42:45.527 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 282080 2023-07-29T16:42:45.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.528 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:45.528 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 452, 2023-07-29T16:42:45.528 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 823, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 88196135, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 592465, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 8, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.103100731, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.021650731, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.060964996, 2023-07-29T16:42:45.529 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.013854179, 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 821, 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.105505408, 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000128508 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.530 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 480, 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 452816 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2023-07-29T16:42:45.531 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 1, 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-MDSRank": { 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3092, 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.631200394, 2023-07-29T16:42:45.532 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000204139 2023-07-29T16:42:45.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.533 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-PurgeQueue": { 2023-07-29T16:42:45.533 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2023-07-29T16:42:45.533 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2023-07-29T16:42:45.534 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.534 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.534 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.534 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.534 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: "request": 0, 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: "reply": 0, 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: "reply_latency": { 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.536 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_reply": 0, 2023-07-29T16:42:45.536 INFO:teuthology.orchestra.run.smithi039.stdout: "forward": 0, 2023-07-29T16:42:45.536 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_complete": 0, 2023-07-29T16:42:45.536 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_keys": 0, 2023-07-29T16:42:45.537 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_commit": 0, 2023-07-29T16:42:45.537 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_split": 0, 2023-07-29T16:42:45.537 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_merge": 0, 2023-07-29T16:42:45.537 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes": 31532, 2023-07-29T16:42:45.538 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_top": 21828, 2023-07-29T16:42:45.538 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_bottom": 9558, 2023-07-29T16:42:45.538 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pin_tail": 146, 2023-07-29T16:42:45.538 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pinned": 348, 2023-07-29T16:42:45.539 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_expired": 10091, 2023-07-29T16:42:45.539 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_with_caps": 0, 2023-07-29T16:42:45.539 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": 0, 2023-07-29T16:42:45.539 INFO:teuthology.orchestra.run.smithi039.stdout: "subtrees": 2, 2023-07-29T16:42:45.539 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse": 0, 2023-07-29T16:42:45.540 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_hit": 0, 2023-07-29T16:42:45.540 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_forward": 0, 2023-07-29T16:42:45.540 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_discover": 0, 2023-07-29T16:42:45.541 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_dir_fetch": 0, 2023-07-29T16:42:45.541 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_remote_ino": 0, 2023-07-29T16:42:45.541 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_lock": 0, 2023-07-29T16:42:45.541 INFO:teuthology.orchestra.run.smithi039.stdout: "load_cent": 0, 2023-07-29T16:42:45.541 INFO:teuthology.orchestra.run.smithi039.stdout: "q": 0, 2023-07-29T16:42:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "exported": 0, 2023-07-29T16:42:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "exported_inodes": 0, 2023-07-29T16:42:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "imported": 0, 2023-07-29T16:42:45.542 INFO:teuthology.orchestra.run.smithi039.stdout: "imported_inodes": 0, 2023-07-29T16:42:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_dir_fetch": 0, 2023-07-29T16:42:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_backtrace_fetch": 0, 2023-07-29T16:42:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_peer_discover": 0, 2023-07-29T16:42:45.543 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rfiles": 4, 2023-07-29T16:42:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rbytes": 246, 2023-07-29T16:42:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rsnaps": 0, 2023-07-29T16:42:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_fetch": 0, 2023-07-29T16:42:45.544 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_set_tag": 0, 2023-07-29T16:42:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_repaired": 0, 2023-07-29T16:42:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_inotable_repaired": 0, 2023-07-29T16:42:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_inodes": 0, 2023-07-29T16:42:45.545 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_base_inodes": 0, 2023-07-29T16:42:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dirfrag_rstats": 0, 2023-07-29T16:42:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_file_inodes": 0, 2023-07-29T16:42:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_inode_file_caps": 0, 2023-07-29T16:42:45.546 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_revoke": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_grant": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_trunc": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flush_ack": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps_dirty": 0, 2023-07-29T16:42:45.547 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_cap_release": 0, 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "process_request_cap_release": 0 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache": { 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update": 0, 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update_receipt": 0, 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_try_discover": 0, 2023-07-29T16:42:45.548 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_send_discover": 0, 2023-07-29T16:42:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_handle_discover": 0, 2023-07-29T16:42:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays": 0, 2023-07-29T16:42:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_delayed": 0, 2023-07-29T16:42:45.549 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_enqueuing": 0, 2023-07-29T16:42:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_created": 0, 2023-07-29T16:42:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_enqueued": 0, 2023-07-29T16:42:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_reintegrated": 0, 2023-07-29T16:42:45.550 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_migrated": 0, 2023-07-29T16:42:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_processing": 0, 2023-07-29T16:42:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_enqueued": 0, 2023-07-29T16:42:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_prioritized": 0, 2023-07-29T16:42:45.551 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_started": 0, 2023-07-29T16:42:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_completed": 0, 2023-07-29T16:42:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_enqueue_scrub": 0, 2023-07-29T16:42:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_exportdir": 0, 2023-07-29T16:42:45.552 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_flush": 0, 2023-07-29T16:42:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragmentdir": 0, 2023-07-29T16:42:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragstats": 0, 2023-07-29T16:42:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_inodestats": 0 2023-07-29T16:42:45.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.554 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log": { 2023-07-29T16:42:45.554 INFO:teuthology.orchestra.run.smithi039.stdout: "evadd": 0, 2023-07-29T16:42:45.554 INFO:teuthology.orchestra.run.smithi039.stdout: "evex": 0, 2023-07-29T16:42:45.554 INFO:teuthology.orchestra.run.smithi039.stdout: "evtrm": 0, 2023-07-29T16:42:45.555 INFO:teuthology.orchestra.run.smithi039.stdout: "ev": 0, 2023-07-29T16:42:45.555 INFO:teuthology.orchestra.run.smithi039.stdout: "evexg": 0, 2023-07-29T16:42:45.555 INFO:teuthology.orchestra.run.smithi039.stdout: "evexd": 0, 2023-07-29T16:42:45.555 INFO:teuthology.orchestra.run.smithi039.stdout: "segadd": 0, 2023-07-29T16:42:45.556 INFO:teuthology.orchestra.run.smithi039.stdout: "segex": 0, 2023-07-29T16:42:45.556 INFO:teuthology.orchestra.run.smithi039.stdout: "segtrm": 0, 2023-07-29T16:42:45.556 INFO:teuthology.orchestra.run.smithi039.stdout: "seg": 6, 2023-07-29T16:42:45.556 INFO:teuthology.orchestra.run.smithi039.stdout: "segexg": 0, 2023-07-29T16:42:45.557 INFO:teuthology.orchestra.run.smithi039.stdout: "segexd": 0, 2023-07-29T16:42:45.557 INFO:teuthology.orchestra.run.smithi039.stdout: "expos": 721547169, 2023-07-29T16:42:45.557 INFO:teuthology.orchestra.run.smithi039.stdout: "wrpos": 0, 2023-07-29T16:42:45.557 INFO:teuthology.orchestra.run.smithi039.stdout: "rdpos": 723110514, 2023-07-29T16:42:45.558 INFO:teuthology.orchestra.run.smithi039.stdout: "jlat": { 2023-07-29T16:42:45.558 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.558 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.558 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: "replayed": 135169 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mem": { 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: "ino": 211, 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: "ino+": 31574, 2023-07-29T16:42:45.559 INFO:teuthology.orchestra.run.smithi039.stdout: "ino-": 31363, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "dir": 19, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "dir+": 199, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "dir-": 180, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "dn": 21763, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "dn+": 32431, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "dn-": 10668, 2023-07-29T16:42:45.560 INFO:teuthology.orchestra.run.smithi039.stdout: "cap": 0, 2023-07-29T16:42:45.561 INFO:teuthology.orchestra.run.smithi039.stdout: "cap+": 0, 2023-07-29T16:42:45.561 INFO:teuthology.orchestra.run.smithi039.stdout: "cap-": 0, 2023-07-29T16:42:45.561 INFO:teuthology.orchestra.run.smithi039.stdout: "rss": 81116, 2023-07-29T16:42:45.561 INFO:teuthology.orchestra.run.smithi039.stdout: "heap": 207156 2023-07-29T16:42:45.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.562 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_server": { 2023-07-29T16:42:45.562 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_client_request": 0, 2023-07-29T16:42:45.562 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_server_request": 0, 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_request": 0, 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_session": 0, 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_peer_request": 0, 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "req_create_latency": { 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.563 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getattr_latency": { 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getfilelock_latency": { 2023-07-29T16:42:45.564 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: "req_link_latency": { 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookup_latency": { 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookuphash_latency": { 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.566 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupino_latency": { 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.567 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupname_latency": { 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupparent_latency": { 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.568 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupsnap_latency": { 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.569 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lssnap_latency": { 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mkdir_latency": { 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.570 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mknod_latency": { 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mksnap_latency": { 2023-07-29T16:42:45.571 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: "req_open_latency": { 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.572 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: "req_readdir_latency": { 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rename_latency": { 2023-07-29T16:42:45.573 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: "req_renamesnap_latency": { 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmdir_latency": { 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmsnap_latency": { 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.575 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmxattr_latency": { 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setattr_latency": { 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setdirlayout_latency": { 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.577 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setfilelock_latency": { 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.578 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setlayout_latency": { 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setxattr_latency": { 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.579 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: "req_symlink_latency": { 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: "req_unlink_latency": { 2023-07-29T16:42:45.580 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_revoke_eviction": 0, 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition_throttle": 0, 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getvxattr_latency": { 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.581 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessions": { 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: "session_count": 1, 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: "session_add": 1, 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: "session_remove": 0, 2023-07-29T16:42:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_open": 0, 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_stale": 0, 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: "total_load": 0, 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: "average_load": 0, 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: "avg_session_uptime": 3751 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2023-07-29T16:42:45.583 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:45.584 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:45.585 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:45.586 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2023-07-29T16:42:45.587 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 254688, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 555, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:45.588 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 7920, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 167, 2023-07-29T16:42:45.589 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 23394403, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 192205, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2023-07-29T16:42:45.590 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter": { 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "op_active": 0, 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "op_laggy": 0, 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send": 1867, 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send_bytes": 26, 2023-07-29T16:42:45.591 INFO:teuthology.orchestra.run.smithi039.stdout: "op_resend": 4, 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "op_reply": 1867, 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "op_latency": { 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1867, 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 15.678681656, 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.008397794 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "op_inflight": 0, 2023-07-29T16:42:45.592 INFO:teuthology.orchestra.run.smithi039.stdout: "oplen_avg": { 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1867, 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1868 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: "op": 1867, 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: "op_r": 1867, 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: "op_w": 0, 2023-07-29T16:42:45.593 INFO:teuthology.orchestra.run.smithi039.stdout: "op_rmw": 0, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "op_pg": 0, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_stat": 961, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_create": 0, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_read": 903, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_write": 0, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writefull": 0, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writesame": 0, 2023-07-29T16:42:45.594 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_append": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_zero": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_truncate": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_delete": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_mapext": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_sparse_read": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_clonerange": 0, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_getxattr": 2, 2023-07-29T16:42:45.595 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_setxattr": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_cmpxattr": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_rmxattr": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_resetxattrs": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_call": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_watch": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_notify": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_src_cmpxattr": 0, 2023-07-29T16:42:45.596 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls_filter": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_other": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_active": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_send": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_resend": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_ping": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_active": 0, 2023-07-29T16:42:45.597 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_send": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_resend": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_active": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_send": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_resend": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_active": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_send": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_resend": 0, 2023-07-29T16:42:45.598 INFO:teuthology.orchestra.run.smithi039.stdout: "command_active": 0, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "command_send": 0, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "command_resend": 0, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "map_epoch": 0, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "map_full": 0, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "map_inc": 0, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_sessions": 11, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_open": 11, 2023-07-29T16:42:45.599 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_close": 0, 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_laggy": 0, 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_wr": 0, 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_rd": 2, 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_del": 0 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "oft": { 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_objs": 0, 2023-07-29T16:42:45.600 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_kv_pairs": 0, 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_updates": 0, 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_removes": 0 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: "osd-slow-ops": { 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_ops_count": 0 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: "purge_queue": { 2023-07-29T16:42:45.601 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops": 0, 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops_high_water": 0, 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing": 0, 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_high_water": 0, 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed": 0, 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_item_in_journal": 0 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:45.602 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 2371, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 278750797, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 2371, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:45.603 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 2371, 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 278750797, 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_bytes": { 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:45.605 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 1867, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 277905979, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 1867, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 277905979, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.606 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_ops": { 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 1024, 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:45.607 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 1867, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 1867, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 1867, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 1867, 2023-07-29T16:42:45.608 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle": { 2023-07-29T16:42:45.609 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:45.610 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 0, 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 0, 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle-0x55fa15502640": { 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:45.612 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 0, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 0, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:45.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:45.614 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.614 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:45.614 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-07-29T16:42:45.911 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-07-29T16:42:45.911 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-07-29T16:42:45.911 DEBUG:tasks.check_counter:Getting stats from h 2023-07-29T16:42:45.912 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.h perf dump 2023-07-29T16:42:46.479 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:42:47.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: pgmap v1191: 97 pgs: 97 active+clean; 471 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2023-07-29T16:42:47.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.189 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:42:47.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:42:47.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:46 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: pgmap v1191: 97 pgs: 97 active+clean; 471 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2023-07-29T16:42:47.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:42:47.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:42:47.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:46 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: pgmap v1191: 97 pgs: 97 active+clean; 471 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 5.7 MiB/s wr, 197 op/s 2023-07-29T16:42:47.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:47.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-29T16:42:47.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-29T16:42:47.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-29T16:42:47.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:46 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' 2023-07-29T16:42:48.018 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:47 smithi039 ceph-mon[128858]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:42:48.018 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:47 smithi039 ceph-mon[128858]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:42:48.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:47 smithi031 ceph-mon[95504]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:42:48.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:47 smithi031 ceph-mon[95504]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:42:48.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:47 smithi156 ceph-mon[129245]: Adjusting osd_memory_target on smithi156 to 196.2M 2023-07-29T16:42:48.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:47 smithi156 ceph-mon[129245]: Unable to set osd_memory_target on smithi156 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-07-29T16:42:48.561 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-07-29T16:42:48.561 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:48.561 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 3729, 2023-07-29T16:42:48.561 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 3702, 2023-07-29T16:42:48.562 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 194558268, 2023-07-29T16:42:48.562 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 1277934, 2023-07-29T16:42:48.562 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 7, 2023-07-29T16:42:48.563 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 3, 2023-07-29T16:42:48.563 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.421148375, 2023-07-29T16:42:48.563 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.107531627, 2023-07-29T16:42:48.564 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.178394244, 2023-07-29T16:42:48.564 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.107874058, 2023-07-29T16:42:48.564 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:48.565 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3701, 2023-07-29T16:42:48.565 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.221989531, 2023-07-29T16:42:48.565 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000059980 2023-07-29T16:42:48.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.566 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:48.566 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.567 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.567 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.567 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 656400, 2023-07-29T16:42:48.567 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 282112 2023-07-29T16:42:48.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.568 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:48.568 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 1950, 2023-07-29T16:42:48.568 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 2318, 2023-07-29T16:42:48.569 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 168271054, 2023-07-29T16:42:48.569 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 1400543, 2023-07-29T16:42:48.569 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 8, 2023-07-29T16:42:48.569 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 6, 2023-07-29T16:42:48.569 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.242157303, 2023-07-29T16:42:48.569 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.047139829, 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.124137531, 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.056437762, 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 2317, 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.140404223, 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000060597 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.570 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 3264, 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 796208 2023-07-29T16:42:48.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.572 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:48.572 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 1584, 2023-07-29T16:42:48.572 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 1586, 2023-07-29T16:42:48.572 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 200590656, 2023-07-29T16:42:48.572 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 492631, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 7, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.231798767, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.028023862, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.147953363, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.044192327, 2023-07-29T16:42:48.573 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:48.574 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1584, 2023-07-29T16:42:48.574 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.091915855, 2023-07-29T16:42:48.574 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000058027 2023-07-29T16:42:48.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.575 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:48.575 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.575 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.575 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.576 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:48.576 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:48.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.576 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2023-07-29T16:42:48.576 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 1, 2023-07-29T16:42:48.577 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2023-07-29T16:42:48.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.577 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-MDSRank": { 2023-07-29T16:42:48.577 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 1, 2023-07-29T16:42:48.577 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 11479, 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1.061822810, 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000092501 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-PurgeQueue": { 2023-07-29T16:42:48.578 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2023-07-29T16:42:48.579 INFO:teuthology.orchestra.run.smithi039.stdout: "request": 0, 2023-07-29T16:42:48.580 INFO:teuthology.orchestra.run.smithi039.stdout: "reply": 0, 2023-07-29T16:42:48.580 INFO:teuthology.orchestra.run.smithi039.stdout: "reply_latency": { 2023-07-29T16:42:48.580 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.580 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.580 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_reply": 0, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "forward": 0, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_complete": 0, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_keys": 0, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_commit": 0, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_split": 0, 2023-07-29T16:42:48.581 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_merge": 0, 2023-07-29T16:42:48.582 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes": 25953, 2023-07-29T16:42:48.582 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_top": 0, 2023-07-29T16:42:48.582 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_bottom": 0, 2023-07-29T16:42:48.582 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pin_tail": 25953, 2023-07-29T16:42:48.582 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pinned": 25953, 2023-07-29T16:42:48.582 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_expired": 33972, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_with_caps": 0, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": 0, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "subtrees": 1, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse": 0, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_hit": 0, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_forward": 0, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_discover": 0, 2023-07-29T16:42:48.583 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_dir_fetch": 0, 2023-07-29T16:42:48.584 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_remote_ino": 0, 2023-07-29T16:42:48.584 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_lock": 0, 2023-07-29T16:42:48.584 INFO:teuthology.orchestra.run.smithi039.stdout: "load_cent": 0, 2023-07-29T16:42:48.584 INFO:teuthology.orchestra.run.smithi039.stdout: "q": 0, 2023-07-29T16:42:48.584 INFO:teuthology.orchestra.run.smithi039.stdout: "exported": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "exported_inodes": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "imported": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "imported_inodes": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_dir_fetch": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_backtrace_fetch": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_peer_discover": 0, 2023-07-29T16:42:48.585 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rfiles": 0, 2023-07-29T16:42:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rbytes": 0, 2023-07-29T16:42:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rsnaps": 0, 2023-07-29T16:42:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_fetch": 0, 2023-07-29T16:42:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_set_tag": 0, 2023-07-29T16:42:48.586 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_repaired": 0, 2023-07-29T16:42:48.587 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_inotable_repaired": 0, 2023-07-29T16:42:48.587 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_inodes": 0, 2023-07-29T16:42:48.587 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_base_inodes": 0, 2023-07-29T16:42:48.588 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dirfrag_rstats": 0, 2023-07-29T16:42:48.588 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_file_inodes": 0, 2023-07-29T16:42:48.588 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_inode_file_caps": 0, 2023-07-29T16:42:48.588 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_revoke": 0, 2023-07-29T16:42:48.588 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_grant": 0, 2023-07-29T16:42:48.588 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_trunc": 0, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flush_ack": 0, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps": 0, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps_dirty": 0, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_cap_release": 0, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "process_request_cap_release": 0 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.589 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache": { 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update_receipt": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_try_discover": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_send_discover": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_handle_discover": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_delayed": 0, 2023-07-29T16:42:48.590 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_enqueuing": 0, 2023-07-29T16:42:48.591 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_created": 0, 2023-07-29T16:42:48.591 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_enqueued": 0, 2023-07-29T16:42:48.591 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_reintegrated": 0, 2023-07-29T16:42:48.591 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_migrated": 0, 2023-07-29T16:42:48.591 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_processing": 0, 2023-07-29T16:42:48.591 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_enqueued": 0, 2023-07-29T16:42:48.592 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_prioritized": 0, 2023-07-29T16:42:48.592 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_started": 0, 2023-07-29T16:42:48.592 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_completed": 0, 2023-07-29T16:42:48.592 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_enqueue_scrub": 0, 2023-07-29T16:42:48.592 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_exportdir": 0, 2023-07-29T16:42:48.593 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_flush": 0, 2023-07-29T16:42:48.593 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragmentdir": 0, 2023-07-29T16:42:48.593 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragstats": 0, 2023-07-29T16:42:48.593 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_inodestats": 0 2023-07-29T16:42:48.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.594 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log": { 2023-07-29T16:42:48.594 INFO:teuthology.orchestra.run.smithi039.stdout: "evadd": 0, 2023-07-29T16:42:48.594 INFO:teuthology.orchestra.run.smithi039.stdout: "evex": 0, 2023-07-29T16:42:48.594 INFO:teuthology.orchestra.run.smithi039.stdout: "evtrm": 0, 2023-07-29T16:42:48.594 INFO:teuthology.orchestra.run.smithi039.stdout: "ev": 0, 2023-07-29T16:42:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: "evexg": 0, 2023-07-29T16:42:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: "evexd": 0, 2023-07-29T16:42:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: "segadd": 0, 2023-07-29T16:42:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: "segex": 0, 2023-07-29T16:42:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: "segtrm": 0, 2023-07-29T16:42:48.595 INFO:teuthology.orchestra.run.smithi039.stdout: "seg": 128, 2023-07-29T16:42:48.596 INFO:teuthology.orchestra.run.smithi039.stdout: "segexg": 0, 2023-07-29T16:42:48.596 INFO:teuthology.orchestra.run.smithi039.stdout: "segexd": 0, 2023-07-29T16:42:48.596 INFO:teuthology.orchestra.run.smithi039.stdout: "expos": 331353154, 2023-07-29T16:42:48.596 INFO:teuthology.orchestra.run.smithi039.stdout: "wrpos": 0, 2023-07-29T16:42:48.596 INFO:teuthology.orchestra.run.smithi039.stdout: "rdpos": 565087120, 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: "jlat": { 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: "replayed": 279351 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.597 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mem": { 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ino": 304, 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ino+": 59873, 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ino-": 59569, 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "dir": 31, 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "dir+": 291, 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "dir-": 260, 2023-07-29T16:42:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "dn": 25953, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "dn+": 60355, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "dn-": 34402, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "cap": 0, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "cap+": 0, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "cap-": 0, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "rss": 79716, 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: "heap": 207156 2023-07-29T16:42:48.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_server": { 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_client_request": 0, 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_server_request": 0, 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_request": 0, 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_session": 0, 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_peer_request": 0, 2023-07-29T16:42:48.600 INFO:teuthology.orchestra.run.smithi039.stdout: "req_create_latency": { 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getattr_latency": { 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.601 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getfilelock_latency": { 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.602 INFO:teuthology.orchestra.run.smithi039.stdout: "req_link_latency": { 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookup_latency": { 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.603 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookuphash_latency": { 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.604 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupino_latency": { 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupname_latency": { 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.605 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupparent_latency": { 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupsnap_latency": { 2023-07-29T16:42:48.606 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lssnap_latency": { 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mkdir_latency": { 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mknod_latency": { 2023-07-29T16:42:48.608 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mksnap_latency": { 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.609 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.610 INFO:teuthology.orchestra.run.smithi039.stdout: "req_open_latency": { 2023-07-29T16:42:48.610 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.610 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.610 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.611 INFO:teuthology.orchestra.run.smithi039.stdout: "req_readdir_latency": { 2023-07-29T16:42:48.611 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.611 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.612 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.612 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rename_latency": { 2023-07-29T16:42:48.612 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.612 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.612 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.613 INFO:teuthology.orchestra.run.smithi039.stdout: "req_renamesnap_latency": { 2023-07-29T16:42:48.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.613 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.613 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.614 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmdir_latency": { 2023-07-29T16:42:48.614 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.614 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.614 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmsnap_latency": { 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmxattr_latency": { 2023-07-29T16:42:48.615 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.616 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.616 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.616 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setattr_latency": { 2023-07-29T16:42:48.616 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.616 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.617 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.617 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setdirlayout_latency": { 2023-07-29T16:42:48.617 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.617 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.617 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.618 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setfilelock_latency": { 2023-07-29T16:42:48.618 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.618 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.618 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.619 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setlayout_latency": { 2023-07-29T16:42:48.619 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.619 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.619 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.619 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setxattr_latency": { 2023-07-29T16:42:48.620 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.620 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.620 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.620 INFO:teuthology.orchestra.run.smithi039.stdout: "req_symlink_latency": { 2023-07-29T16:42:48.620 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.621 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.621 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.621 INFO:teuthology.orchestra.run.smithi039.stdout: "req_unlink_latency": { 2023-07-29T16:42:48.621 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.621 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.622 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.622 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_revoke_eviction": 0, 2023-07-29T16:42:48.622 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition_throttle": 0, 2023-07-29T16:42:48.622 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getvxattr_latency": { 2023-07-29T16:42:48.622 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.623 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.623 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.623 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessions": { 2023-07-29T16:42:48.623 INFO:teuthology.orchestra.run.smithi039.stdout: "session_count": 1, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "session_add": 2, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "session_remove": 1, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_open": 1, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_stale": 0, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "total_load": 0, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "average_load": 0, 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: "avg_session_uptime": 3757 2023-07-29T16:42:48.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.625 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2023-07-29T16:42:48.625 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:48.625 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:48.625 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:48.625 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:48.625 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:48.626 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:48.627 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:48.628 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:48.628 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:48.628 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:48.628 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:48.628 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:48.628 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:48.629 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 251677, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 375, 2023-07-29T16:42:48.630 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:48.631 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:48.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2023-07-29T16:42:48.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2023-07-29T16:42:48.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:48.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:48.631 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:48.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:48.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 7920, 2023-07-29T16:42:48.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 167, 2023-07-29T16:42:48.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:48.632 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:48.632 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 19020290, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 143926, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2023-07-29T16:42:48.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter": { 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_active": 1, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_laggy": 0, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send": 6730, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send_bytes": 21, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_resend": 3, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_reply": 6729, 2023-07-29T16:42:48.634 INFO:teuthology.orchestra.run.smithi039.stdout: "op_latency": { 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 6729, 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 27.183443843, 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.004039744 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: "op_inflight": 0, 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: "oplen_avg": { 2023-07-29T16:42:48.635 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 6730, 2023-07-29T16:42:48.636 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 6731 2023-07-29T16:42:48.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.636 INFO:teuthology.orchestra.run.smithi039.stdout: "op": 6730, 2023-07-29T16:42:48.636 INFO:teuthology.orchestra.run.smithi039.stdout: "op_r": 6730, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "op_w": 0, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "op_rmw": 0, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "op_pg": 0, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_stat": 3539, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_create": 0, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_read": 3189, 2023-07-29T16:42:48.637 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_write": 0, 2023-07-29T16:42:48.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writefull": 0, 2023-07-29T16:42:48.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writesame": 0, 2023-07-29T16:42:48.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_append": 0, 2023-07-29T16:42:48.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_zero": 0, 2023-07-29T16:42:48.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_truncate": 0, 2023-07-29T16:42:48.638 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_delete": 0, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_mapext": 0, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_sparse_read": 0, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_clonerange": 0, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_getxattr": 1, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_setxattr": 0, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_cmpxattr": 0, 2023-07-29T16:42:48.639 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_rmxattr": 0, 2023-07-29T16:42:48.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_resetxattrs": 0, 2023-07-29T16:42:48.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_call": 0, 2023-07-29T16:42:48.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_watch": 0, 2023-07-29T16:42:48.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_notify": 0, 2023-07-29T16:42:48.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_src_cmpxattr": 0, 2023-07-29T16:42:48.640 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls": 0, 2023-07-29T16:42:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls_filter": 0, 2023-07-29T16:42:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_other": 0, 2023-07-29T16:42:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_active": 0, 2023-07-29T16:42:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_send": 0, 2023-07-29T16:42:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_resend": 0, 2023-07-29T16:42:48.641 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_ping": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_active": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_send": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_resend": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_active": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_send": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_resend": 0, 2023-07-29T16:42:48.642 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_active": 0, 2023-07-29T16:42:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_send": 0, 2023-07-29T16:42:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_resend": 0, 2023-07-29T16:42:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: "command_active": 0, 2023-07-29T16:42:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: "command_send": 0, 2023-07-29T16:42:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: "command_resend": 0, 2023-07-29T16:42:48.643 INFO:teuthology.orchestra.run.smithi039.stdout: "map_epoch": 93, 2023-07-29T16:42:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: "map_full": 0, 2023-07-29T16:42:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: "map_inc": 7, 2023-07-29T16:42:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_sessions": 11, 2023-07-29T16:42:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_open": 12, 2023-07-29T16:42:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_close": 1, 2023-07-29T16:42:48.644 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_laggy": 0, 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_wr": 0, 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_rd": 2, 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_del": 0 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "oft": { 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_objs": 0, 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_kv_pairs": 0, 2023-07-29T16:42:48.645 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_updates": 0, 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_removes": 0 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: "osd-slow-ops": { 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_ops_count": 0 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: "purge_queue": { 2023-07-29T16:42:48.646 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops": 0, 2023-07-29T16:42:48.647 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops_high_water": 0, 2023-07-29T16:42:48.647 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing": 0, 2023-07-29T16:42:48.647 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_high_water": 0, 2023-07-29T16:42:48.647 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed": 0, 2023-07-29T16:42:48.647 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_item_in_journal": 0 2023-07-29T16:42:48.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.648 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:48.648 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:48.648 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:48.648 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:48.648 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 7259, 2023-07-29T16:42:48.648 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 562736576, 2023-07-29T16:42:48.649 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:48.649 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 7259, 2023-07-29T16:42:48.649 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:48.649 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:48.649 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 7259, 2023-07-29T16:42:48.649 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 562736576, 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.650 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_bytes": { 2023-07-29T16:42:48.651 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 170715, 2023-07-29T16:42:48.651 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:48.651 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:48.651 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:48.651 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:48.651 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 6730, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 561079521, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 6729, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 560908806, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.652 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_ops": { 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 1, 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 1024, 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:48.653 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 6730, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 6730, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 6729, 2023-07-29T16:42:48.654 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 6729, 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.655 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle": { 2023-07-29T16:42:48.656 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:48.656 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2023-07-29T16:42:48.656 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:48.656 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:48.656 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:48.656 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.657 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle-0x5577a078e500": { 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:48.658 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2023-07-29T16:42:48.659 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2023-07-29T16:42:48.659 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:48.659 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:48.659 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:48.659 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:48.659 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 0, 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 0, 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:48.660 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-07-29T16:42:48.950 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2023-07-29T16:42:48.951 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2023-07-29T16:42:48.951 DEBUG:tasks.check_counter:Getting stats from k 2023-07-29T16:42:48.951 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.k perf dump 2023-07-29T16:42:49.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:48 smithi039 ceph-mon[128858]: pgmap v1192: 97 pgs: 97 active+clean; 456 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 4.8 MiB/s wr, 200 op/s 2023-07-29T16:42:49.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:48 smithi156 ceph-mon[129245]: pgmap v1192: 97 pgs: 97 active+clean; 456 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 4.8 MiB/s wr, 200 op/s 2023-07-29T16:42:49.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:48 smithi031 ceph-mon[95504]: pgmap v1192: 97 pgs: 97 active+clean; 456 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 4.8 MiB/s wr, 200 op/s 2023-07-29T16:42:49.518 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.b/config 2023-07-29T16:42:50.761 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2023-07-29T16:42:50.762 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:50.762 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 0, 2023-07-29T16:42:50.762 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 2, 2023-07-29T16:42:50.762 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 0, 2023-07-29T16:42:50.762 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 0, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 0, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.001460681, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.000000000, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.000296765, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:50.763 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:50.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 495, 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 492, 2023-07-29T16:42:50.765 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 547280, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 282416, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 2, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 1, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.061974433, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.022678633, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.022656159, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.013047635, 2023-07-29T16:42:50.766 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 491, 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.039519238, 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000080487 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:50.767 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 547280, 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 282416 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 7, 2023-07-29T16:42:50.768 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 379, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 3424, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 295072, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 5, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.017078764, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.011738576, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.001979692, 2023-07-29T16:42:50.769 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 378, 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.021393073, 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000056595 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:50.770 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 295072 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 0, 2023-07-29T16:42:50.771 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:50.772 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:50.773 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:50.773 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:50.773 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:50.773 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:50.773 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:50.773 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:50.774 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:50.775 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:50.776 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 148704, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 29, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:50.777 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:50.778 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 928, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-07-29T16:42:50.779 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 498, 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 477291, 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:50.780 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 498, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 498, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 477291, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2023-07-29T16:42:50.781 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2023-07-29T16:42:50.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:50.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-07-29T16:42:50.782 INFO:teuthology.orchestra.run.smithi039.stdout:} 2023-07-29T16:42:51.143 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:50 smithi039 ceph-mon[128858]: pgmap v1193: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2023-07-29T16:42:51.182 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2023-07-29T16:42:51.183 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2023-07-29T16:42:51.183 DEBUG:tasks.check_counter:Getting stats from c 2023-07-29T16:42:51.183 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.c perf dump 2023-07-29T16:42:51.315 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:50 smithi156 ceph-mon[129245]: pgmap v1193: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2023-07-29T16:42:51.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:50 smithi031 ceph-mon[95504]: pgmap v1193: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 3.6 MiB/s wr, 182 op/s 2023-07-29T16:42:51.863 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:42:53.181 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-07-29T16:42:53.181 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:53.181 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 0, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 0, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 0, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 0, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.001820712, 2023-07-29T16:42:53.182 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.000000000, 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000297669, 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.183 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:53.184 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 98, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 160, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 93104, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 3, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2023-07-29T16:42:53.185 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.004045141, 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.002757659, 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000501278, 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 96, 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.007651461, 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000079702 2023-07-29T16:42:53.186 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 93104 2023-07-29T16:42:53.187 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 133, 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 130, 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 176288, 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 72880, 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 2, 2023-07-29T16:42:53.188 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.014483425, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.004259613, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.006297387, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.003174037, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 130, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.010666686, 2023-07-29T16:42:53.189 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000082051 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.190 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 176288, 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 72880 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 0, 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-07-29T16:42:53.191 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:53.192 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:53.193 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:53.194 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:53.195 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 148801, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 27, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:53.196 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 0, 2023-07-29T16:42:53.197 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 0, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 928, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 2, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:53.198 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-07-29T16:42:53.199 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 130, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 155131, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 130, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 130, 2023-07-29T16:42:53.200 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 155131, 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:53.201 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-07-29T16:42:53.205 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:52 smithi156 ceph-mon[129245]: pgmap v1194: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.3 MiB/s wr, 143 op/s 2023-07-29T16:42:53.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:52 smithi039 ceph-mon[128858]: pgmap v1194: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.3 MiB/s wr, 143 op/s 2023-07-29T16:42:53.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:52 smithi031 ceph-mon[95504]: pgmap v1194: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 2.3 MiB/s wr, 143 op/s 2023-07-29T16:42:53.604 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.c 2023-07-29T16:42:53.605 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.c 2023-07-29T16:42:53.605 DEBUG:tasks.check_counter:Getting stats from f 2023-07-29T16:42:53.605 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.f perf dump 2023-07-29T16:42:54.156 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:42:55.075 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:54 smithi156 ceph-mon[129245]: pgmap v1195: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2023-07-29T16:42:55.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:54 smithi039 ceph-mon[128858]: pgmap v1195: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2023-07-29T16:42:55.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:54 smithi031 ceph-mon[95504]: pgmap v1195: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 1.8 MiB/s wr, 124 op/s 2023-07-29T16:42:55.491 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-07-29T16:42:55.491 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:55.491 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1596, 2023-07-29T16:42:55.491 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1569, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 156120835, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 615420, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.219286600, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.043373036, 2023-07-29T16:42:55.492 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.125412106, 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.039807583, 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1568, 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.064856202, 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000041362 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:55.493 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 657216, 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 283888 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:55.494 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1724, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2095, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 176080249, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 1332399, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 8, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 6, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.228054620, 2023-07-29T16:42:55.495 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.049090424, 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.113622789, 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.051190959, 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2094, 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.140901520, 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000067288 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.496 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3264, 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 798096 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.497 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 3940, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 3942, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 257001671, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 1225398, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 6, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.436574633, 2023-07-29T16:42:55.498 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.081492631, 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.209571055, 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.116927935, 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3940, 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.159128462, 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000040387 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.499 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:55.500 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-07-29T16:42:55.501 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 11483, 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.867367479, 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000075534 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-07-29T16:42:55.502 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2023-07-29T16:42:55.503 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2023-07-29T16:42:55.504 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 23956, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 0, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 0, 2023-07-29T16:42:55.505 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 23956, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 23956, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 44970, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 1, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2023-07-29T16:42:55.506 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2023-07-29T16:42:55.507 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2023-07-29T16:42:55.508 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2023-07-29T16:42:55.509 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2023-07-29T16:42:55.510 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2023-07-29T16:42:55.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2023-07-29T16:42:55.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2023-07-29T16:42:55.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2023-07-29T16:42:55.514 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2023-07-29T16:42:55.515 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 128, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 362952893, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 0, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 590878653, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.516 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 316273 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 17, 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 68817, 2023-07-29T16:42:55.517 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 68800, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 22, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 311, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 289, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 23956, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 69393, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 45437, 2023-07-29T16:42:55.518 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 81044, 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 207156 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2023-07-29T16:42:55.519 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.520 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.521 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.522 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2023-07-29T16:42:55.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.526 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.527 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.528 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.529 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.530 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.531 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2023-07-29T16:42:55.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.534 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.535 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.536 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.538 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2023-07-29T16:42:55.539 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 3817 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:55.540 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:55.541 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:55.542 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:55.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:55.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 241903, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 87, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-07-29T16:42:55.545 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7920, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 167, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:55.546 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 16932527, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 96482, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-07-29T16:42:55.547 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 6724, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 6724, 2023-07-29T16:42:55.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 6724, 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 24.874635327, 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003699380 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 6724, 2023-07-29T16:42:55.549 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 6725 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 6724, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 6724, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 3467, 2023-07-29T16:42:55.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 3255, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2023-07-29T16:42:55.551 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2023-07-29T16:42:55.552 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2023-07-29T16:42:55.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2023-07-29T16:42:55.554 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2023-07-29T16:42:55.555 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 7, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 11, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 11, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2023-07-29T16:42:55.556 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2023-07-29T16:42:55.557 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2023-07-29T16:42:55.558 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-07-29T16:42:55.559 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 7256, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 588519467, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 7256, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:55.560 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 7256, 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 588519467, 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.561 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:55.562 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 6724, 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 586691399, 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 6724, 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 586691399, 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.563 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-07-29T16:42:55.564 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 6724, 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 6724, 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 6724, 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 6724, 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:55.565 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-07-29T16:42:55.566 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2023-07-29T16:42:55.567 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55abae7625a0": { 2023-07-29T16:42:55.568 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:55.569 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.570 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:55.571 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-07-29T16:42:55.912 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-07-29T16:42:55.912 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-07-29T16:42:55.913 DEBUG:tasks.check_counter:Getting stats from i 2023-07-29T16:42:55.913 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.i perf dump 2023-07-29T16:42:56.464 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:42:57.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:56 smithi039 ceph-mon[128858]: pgmap v1196: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 651 KiB/s wr, 71 op/s 2023-07-29T16:42:57.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:56 smithi156 ceph-mon[129245]: pgmap v1196: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 651 KiB/s wr, 71 op/s 2023-07-29T16:42:57.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:56 smithi031 ceph-mon[95504]: pgmap v1196: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 651 KiB/s wr, 71 op/s 2023-07-29T16:42:57.660 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-07-29T16:42:57.660 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:42:57.660 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 83966, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 87544, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 20627933, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 238503172, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 55, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 5.816777037, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 2.075790052, 2023-07-29T16:42:57.661 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 2.184667539, 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 1.580017703, 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 87544, 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 6.195043227, 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000070764 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.662 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 80537, 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.012899256, 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000160 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 673408, 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 314816 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.663 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 543332, 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 915200, 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 196080394, 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 926573672, 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 28, 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2023-07-29T16:42:57.664 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 36.806995086, 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 17.200382588, 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 24.606185085, 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 1.720465163, 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 915198, 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 64.796960154, 2023-07-29T16:42:57.665 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000070801 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1063122, 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.166467899, 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000156 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 183007328, 2023-07-29T16:42:57.666 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 726822848 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 331953, 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 324705, 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 91464367, 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 283321768, 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 12, 2023-07-29T16:42:57.667 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 6, 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 17.890585668, 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 6.648268429, 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 10.398698866, 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 2.388336992, 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 324703, 2023-07-29T16:42:57.668 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 14.283961895, 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000043990 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 508539, 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.068323229, 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000134 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.669 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2023-07-29T16:42:57.670 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 168224, 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 193.654503782, 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001151170 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2023-07-29T16:42:57.671 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 115859, 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 13.352430377, 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000115247 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.672 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 376179, 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 360837, 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 360837, 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1360.860206064, 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003771398 2023-07-29T16:42:57.673 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 15342, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 265, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 27, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 35, 2023-07-29T16:42:57.674 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 3241, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 1976, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 916, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 349, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 411, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 395, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2023-07-29T16:42:57.675 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 4, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 618022, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 336079, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 15341, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 22, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2023-07-29T16:42:57.676 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 4914, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 7, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 128, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 7, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2023-07-29T16:42:57.677 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 851, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 4, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 246, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 851, 2023-07-29T16:42:57.678 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 30, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 30, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 821, 2023-07-29T16:42:57.679 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 10, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 14288, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 260862, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 82599, 2023-07-29T16:42:57.680 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 45748, 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 9890, 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 259568 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 75, 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 316, 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 165, 2023-07-29T16:42:57.681 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 320, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 110, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 4, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 68766, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 68762, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2023-07-29T16:42:57.682 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 10, 2023-07-29T16:42:57.683 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 62, 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 316273, 2023-07-29T16:42:57.684 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 208483, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 208483, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 107790, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 1024, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 371, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 243, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 243, 2023-07-29T16:42:57.685 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 129, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 1, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 362952893, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 590879791, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 92957, 2023-07-29T16:42:57.686 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 759.608202916, 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.008171608 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 27, 2023-07-29T16:42:57.687 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 68902, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 68875, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 33, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 527, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 494, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 3241, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 122199, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 118958, 2023-07-29T16:42:57.688 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 72877, 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 72877, 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 80784, 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 198964 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2023-07-29T16:42:57.689 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 557547, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 3939, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 376179, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 8536, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 32652, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 68538, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 207.731095998, 2023-07-29T16:42:57.690 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003030889 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 6234, 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 165.815848043, 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.026598628 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2023-07-29T16:42:57.691 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.692 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 99199, 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 111.621912538, 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001125232 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2023-07-29T16:42:57.693 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.694 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.694 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.694 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2023-07-29T16:42:57.694 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.695 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.695 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.695 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.695 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2023-07-29T16:42:57.695 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.695 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.696 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2023-07-29T16:42:57.697 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.698 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.699 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2023-07-29T16:42:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.700 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.701 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.701 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2023-07-29T16:42:57.701 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 101390, 2023-07-29T16:42:57.701 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 678.899617818, 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.006695922 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 13705, 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 43.427543117, 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003168737 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.702 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 30, 2023-07-29T16:42:57.703 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.270846185, 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.009028206 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.704 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2023-07-29T16:42:57.705 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 33, 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.730914695, 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.022148930 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.706 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2972, 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2.673797215, 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000899662 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.707 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.708 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 68736, 2023-07-29T16:42:57.709 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 149.688630455, 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002177732 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.710 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2023-07-29T16:42:57.711 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 27643, 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 27643, 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 1471 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-07-29T16:42:57.712 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:42:57.713 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-07-29T16:42:57.714 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:42:57.715 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-07-29T16:42:57.716 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 300687, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 123, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 88, 2023-07-29T16:42:57.717 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 1, 2023-07-29T16:42:57.718 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-07-29T16:42:57.718 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-07-29T16:42:57.718 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:42:57.718 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-07-29T16:42:57.718 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:42:57.718 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7920, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 167, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-07-29T16:42:57.719 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 2561324, 2023-07-29T16:42:57.720 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 35772, 2023-07-29T16:42:57.720 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-07-29T16:42:57.720 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-07-29T16:42:57.720 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-07-29T16:42:57.720 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-07-29T16:42:57.720 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 167739, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 600306720, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 28, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 167739, 2023-07-29T16:42:57.721 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 167739, 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1485.321259917, 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.008854954 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 167739, 2023-07-29T16:42:57.722 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 168642 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 167739, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 2438, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 165301, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 3, 2023-07-29T16:42:57.723 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 21, 2023-07-29T16:42:57.724 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 1587, 2023-07-29T16:42:57.724 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 94544, 2023-07-29T16:42:57.724 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 933, 2023-07-29T16:42:57.724 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2023-07-29T16:42:57.724 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 27, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 69217, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2023-07-29T16:42:57.725 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 851, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 42, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2023-07-29T16:42:57.726 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2023-07-29T16:42:57.727 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2023-07-29T16:42:57.727 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2023-07-29T16:42:57.727 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2023-07-29T16:42:57.727 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2023-07-29T16:42:57.727 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2023-07-29T16:42:57.727 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2023-07-29T16:42:57.728 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2023-07-29T16:42:57.728 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2023-07-29T16:42:57.728 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2023-07-29T16:42:57.728 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2023-07-29T16:42:57.728 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2023-07-29T16:42:57.729 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 93, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 7, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 12, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 12, 2023-07-29T16:42:57.730 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2023-07-29T16:42:57.731 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2023-07-29T16:42:57.731 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 1097, 2023-07-29T16:42:57.731 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2023-07-29T16:42:57.731 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 320 2023-07-29T16:42:57.732 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.732 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2023-07-29T16:42:57.732 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2023-07-29T16:42:57.732 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2023-07-29T16:42:57.733 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2023-07-29T16:42:57.733 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2023-07-29T16:42:57.733 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.733 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2023-07-29T16:42:57.734 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 380188 2023-07-29T16:42:57.734 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.734 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2023-07-29T16:42:57.734 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2023-07-29T16:42:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 10, 2023-07-29T16:42:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2023-07-29T16:42:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 10, 2023-07-29T16:42:57.735 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 68762, 2023-07-29T16:42:57.736 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2023-07-29T16:42:57.736 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.736 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:42:57.736 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:57.737 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-07-29T16:42:57.737 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:57.737 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 959247, 2023-07-29T16:42:57.737 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 199008885, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 959247, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 959247, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 199008885, 2023-07-29T16:42:57.738 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:57.739 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.739 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.739 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.739 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.739 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.739 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:57.740 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 167739, 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 607251682, 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 167739, 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 607251682, 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.741 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:57.742 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 167739, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 167739, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 167739, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 167739, 2023-07-29T16:42:57.743 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:57.744 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.744 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.744 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.744 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.744 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.744 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 68762, 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 6944962, 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:57.745 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 68762, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1587, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 6944962, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.746 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x56152bb220a0": { 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 316273, 2023-07-29T16:42:57.747 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 586685487, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 316273, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 92957, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 586685487, 2023-07-29T16:42:57.748 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:42:57.749 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:42:57.749 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:42:57.749 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:42:57.749 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.749 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:42:57.749 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-07-29T16:42:58.110 INFO:tasks.check_counter:Daemon mds.i mds.exported=7 2023-07-29T16:42:58.110 INFO:tasks.check_counter:Daemon mds.i mds.imported=7 2023-07-29T16:42:58.110 DEBUG:tasks.check_counter:Getting stats from l 2023-07-29T16:42:58.111 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:0484097b0572e96f42cf6402cbe0ac8dcb046577 shell --fsid 858a7894-2e29-11ee-9b36-001a4aab830c -- ceph daemon mds.l perf dump 2023-07-29T16:42:58.662 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/mon.c/config 2023-07-29T16:42:58.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:42:58 smithi031 ceph-mon[95504]: pgmap v1197: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 30 op/s 2023-07-29T16:42:58.779 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:42:58 smithi039 ceph-mon[128858]: pgmap v1197: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 30 op/s 2023-07-29T16:42:58.826 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:42:58 smithi156 ceph-mon[129245]: pgmap v1197: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 30 op/s 2023-07-29T16:43:00.108 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2023-07-29T16:43:00.108 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 0, 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2, 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 0, 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 0, 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 0, 2023-07-29T16:43:00.109 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.001323342, 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.000000000, 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.000333054, 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:43:00.110 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2023-07-29T16:43:00.111 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 7, 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 381, 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 3424, 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 295424, 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2023-07-29T16:43:00.112 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.016977577, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.012155372, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.001535122, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 380, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.025363343, 2023-07-29T16:43:00.113 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000066745 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3424, 2023-07-29T16:43:00.114 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 295424 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 498, 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 495, 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 548096, 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 284192, 2023-07-29T16:43:00.115 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 2, 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 1, 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.060539385, 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.023413120, 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.021654250, 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.012028402, 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2023-07-29T16:43:00.116 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 494, 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.026729865, 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000054109 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:43:00.117 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 548096, 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 284192 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 0, 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.118 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2023-07-29T16:43:00.119 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2023-07-29T16:43:00.120 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2023-07-29T16:43:00.121 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2023-07-29T16:43:00.122 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 148721, 2023-07-29T16:43:00.123 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 30, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2023-07-29T16:43:00.124 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2023-07-29T16:43:00.125 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 928, 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 2, 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-07-29T16:43:00.126 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 501, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 477681, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 501, 2023-07-29T16:43:00.127 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 501, 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 477681, 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2023-07-29T16:43:00.128 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:43:00.129 INFO:teuthology.orchestra.run.smithi156.stdout: } 2023-07-29T16:43:00.129 INFO:teuthology.orchestra.run.smithi156.stdout:} 2023-07-29T16:43:00.521 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-07-29T16:43:00.522 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-07-29T16:43:00.522 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-07-29T16:43:00.536 INFO:tasks.kclient:Unmounting kernel clients... 2023-07-29T16:43:00.536 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:43:00.536 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-29T16:43:00.597 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:43:00.598 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-29T16:43:00.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:43:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 -- 172.21.15.31:0/2010599942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0101ec0 msgr2=0x7f61d0108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 --2- 172.21.15.31:0/2010599942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0101ec0 0x7f61d0108780 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f61c800b0a0 tx=0x7f61c800b3b0 comp rx=0 tx=0).stop 2023-07-29T16:43:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 -- 172.21.15.31:0/2010599942 shutdown_connections 2023-07-29T16:43:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 --2- 172.21.15.31:0/2010599942 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0101ec0 0x7f61d0108780 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 --2- 172.21.15.31:0/2010599942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0100340 0x7f61d0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:00.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 --2- 172.21.15.31:0/2010599942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:00.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 -- 172.21.15.31:0/2010599942 >> 172.21.15.31:0/2010599942 conn(0x7f61d00fb920 msgr2=0x7f61d00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:00.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 -- 172.21.15.31:0/2010599942 shutdown_connections 2023-07-29T16:43:00.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.891+0000 7f61d81ff700 1 -- 172.21.15.31:0/2010599942 wait complete. 2023-07-29T16:43:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.893+0000 7f61d81ff700 1 Processor -- start 2023-07-29T16:43:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.893+0000 7f61d81ff700 1 -- start start 2023-07-29T16:43:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.893+0000 7f61d81ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0100340 0x7f61d0074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.893+0000 7f61d81ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.893+0000 7f61d81ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0101ec0 0x7f61d0073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d81ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0073e20 con 0x7f61d0100340 2023-07-29T16:43:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d81ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d0073f90 con 0x7f61d0101530 2023-07-29T16:43:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d81ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61d01a01a0 con 0x7f61d0101ec0 2023-07-29T16:43:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d5f9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0100340 0x7f61d0074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d679c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0101ec0 0x7f61d0073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40262/0 (socket says 172.21.15.31:40262) 2023-07-29T16:43:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 -- 172.21.15.31:0/3679518151 learned_addr learned my addr 172.21.15.31:0/3679518151 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 -- 172.21.15.31:0/3679518151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0101ec0 msgr2=0x7f61d0073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0101ec0 0x7f61d0073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 -- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0100340 msgr2=0x7f61d0074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0100340 0x7f61d0074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 -- 172.21.15.31:0/3679518151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8009d00 con 0x7f61d0101530 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d679c700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0101ec0 0x7f61d0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:43:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.894+0000 7f61d579a700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0073140 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f61c000e9a0 tx=0x7f61c000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.895+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0009790 con 0x7f61d0101530 2023-07-29T16:43:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.895+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61d01a0480 con 0x7f61d0101530 2023-07-29T16:43:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.895+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61c0010430 con 0x7f61d0101530 2023-07-29T16:43:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.895+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c0018680 con 0x7f61d0101530 2023-07-29T16:43:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.895+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61d01a09b0 con 0x7f61d0101530 2023-07-29T16:43:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.896+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61d0107790 con 0x7f61d0101530 2023-07-29T16:43:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.898+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f61c00105a0 con 0x7f61d0101530 2023-07-29T16:43:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.899+0000 7f61c6ffd700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61bc0a07e0 0x7f61bc0a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.899+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f61c0014070 con 0x7f61d0101530 2023-07-29T16:43:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.899+0000 7f61d5f9b700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61bc0a07e0 0x7f61bc0a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.900+0000 7f61d5f9b700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61bc0a07e0 0x7f61bc0a2c90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f61d0101390 tx=0x7f61cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:00.905 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:00.901+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f61c00959c0 con 0x7f61d0101530 2023-07-29T16:43:01.116 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.112+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61d002cc30 con 0x7f61d0101530 2023-07-29T16:43:01.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.115+0000 7f61c6ffd700 1 -- 172.21.15.31:0/3679518151 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f61c0026070 con 0x7f61d0101530 2023-07-29T16:43:01.119 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:43:01.119 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:35:01.881611+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:43:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61bc0a07e0 msgr2=0x7f61bc0a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61bc0a07e0 0x7f61bc0a2c90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f61d0101390 tx=0x7f61cc009260 comp rx=0 tx=0).stop 2023-07-29T16:43:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 msgr2=0x7f61d0073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0073140 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f61c000e9a0 tx=0x7f61c000ecb0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 shutdown_connections 2023-07-29T16:43:01.121 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f61bc0a07e0 0x7f61bc0a2c90 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61d0101ec0 0x7f61d0073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61d0100340 0x7f61d0074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 --2- 172.21.15.31:0/3679518151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61d0101530 0x7f61d0073140 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 >> 172.21.15.31:0/3679518151 conn(0x7f61d00fb920 msgr2=0x7f61d01048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 shutdown_connections 2023-07-29T16:43:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.117+0000 7f61d81ff700 1 -- 172.21.15.31:0/3679518151 wait complete. 2023-07-29T16:43:01.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:00 smithi031 ceph-mon[95504]: pgmap v1198: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-07-29T16:43:01.141 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-29T16:08:09.787659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-29T16:12:20.183673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-29T16:12:21.310891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-29T16:43:01.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-29T16:43:01.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:00 smithi039 ceph-mon[128858]: pgmap v1198: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-07-29T16:43:01.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:00 smithi156 ceph-mon[129245]: pgmap v1198: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 12 op/s 2023-07-29T16:43:01.349 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.345+0000 7f99719ff700 1 -- 172.21.15.31:0/1936599589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101510 msgr2=0x7f996c101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.345+0000 7f99719ff700 1 --2- 172.21.15.31:0/1936599589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101510 0x7f996c101960 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f9960009990 tx=0x7f9960009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 -- 172.21.15.31:0/1936599589 shutdown_connections 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 --2- 172.21.15.31:0/1936599589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101ea0 0x7f996c108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 --2- 172.21.15.31:0/1936599589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f996c100320 0x7f996c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 --2- 172.21.15.31:0/1936599589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101510 0x7f996c101960 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 -- 172.21.15.31:0/1936599589 >> 172.21.15.31:0/1936599589 conn(0x7f996c0fb920 msgr2=0x7f996c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 -- 172.21.15.31:0/1936599589 shutdown_connections 2023-07-29T16:43:01.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.346+0000 7f99719ff700 1 -- 172.21.15.31:0/1936599589 wait complete. 2023-07-29T16:43:01.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 Processor -- start 2023-07-29T16:43:01.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 -- start start 2023-07-29T16:43:01.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f996c100320 0x7f996c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 0x7f996c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101ea0 0x7f996c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c19fe00 con 0x7f996c101ea0 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c19ff40 con 0x7f996c101510 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.347+0000 7f99719ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996c1a00d0 con 0x7f996c100320 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 0x7f996c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101ea0 0x7f996c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 0x7f996c1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40278/0 (socket says 172.21.15.31:40278) 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 -- 172.21.15.31:0/4075716263 learned_addr learned my addr 172.21.15.31:0/4075716263 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996affd700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f996c100320 0x7f996c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 -- 172.21.15.31:0/4075716263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f996c100320 msgr2=0x7f996c198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f996c100320 0x7f996c198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 -- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101ea0 msgr2=0x7f996c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101ea0 0x7f996c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 -- 172.21.15.31:0/4075716263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600095f0 con 0x7f996c101510 2023-07-29T16:43:01.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.348+0000 7f996a7fc700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 0x7f996c1987c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f996000b620 tx=0x7f9960005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:01.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.349+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996001d070 con 0x7f996c101510 2023-07-29T16:43:01.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.349+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996c1a0350 con 0x7f996c101510 2023-07-29T16:43:01.355 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.349+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c1a07d0 con 0x7f996c101510 2023-07-29T16:43:01.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.349+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9960022470 con 0x7f996c101510 2023-07-29T16:43:01.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.349+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960017650 con 0x7f996c101510 2023-07-29T16:43:01.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.350+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996c10bb50 con 0x7f996c101510 2023-07-29T16:43:01.356 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.352+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f996000fb10 con 0x7f996c101510 2023-07-29T16:43:01.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.353+0000 7f99709fd700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99540a0790 0x7f99540a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.353+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f99600cd0d0 con 0x7f996c101510 2023-07-29T16:43:01.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.354+0000 7f996affd700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99540a0790 0x7f99540a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.354+0000 7f996affd700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99540a0790 0x7f99540a2c40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f996c101370 tx=0x7f995c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:01.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.355+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9960096b40 con 0x7f996c101510 2023-07-29T16:43:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.564+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f996c1a0e60 con 0x7f996c101510 2023-07-29T16:43:01.568 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.565+0000 7f99709fd700 1 -- 172.21.15.31:0/4075716263 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f99600310a0 con 0x7f996c101510 2023-07-29T16:43:01.569 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-07-29T16:43:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99540a0790 msgr2=0x7f99540a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99540a0790 0x7f99540a2c40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f996c101370 tx=0x7f995c009260 comp rx=0 tx=0).stop 2023-07-29T16:43:01.571 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 msgr2=0x7f996c1987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 0x7f996c1987c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f996000b620 tx=0x7f9960005740 comp rx=0 tx=0).stop 2023-07-29T16:43:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 shutdown_connections 2023-07-29T16:43:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f99540a0790 0x7f99540a2c40 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f996c101ea0 0x7f996c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f996c100320 0x7f996c198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.572 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 --2- 172.21.15.31:0/4075716263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f996c101510 0x7f996c1987c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 >> 172.21.15.31:0/4075716263 conn(0x7f996c0fb920 msgr2=0x7f996c106da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:01.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.567+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 shutdown_connections 2023-07-29T16:43:01.573 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.568+0000 7f99719ff700 1 -- 172.21.15.31:0/4075716263 wait complete. 2023-07-29T16:43:01.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-29T16:43:01.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.793+0000 7f06589ff700 1 -- 172.21.15.31:0/4273242715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 msgr2=0x7f0654108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.797 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.793+0000 7f06589ff700 1 --2- 172.21.15.31:0/4273242715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f0654108770 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f064c00b0a0 tx=0x7f064c00b3b0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 -- 172.21.15.31:0/4273242715 shutdown_connections 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 --2- 172.21.15.31:0/4273242715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f0654108770 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 --2- 172.21.15.31:0/4273242715 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 0x7f0654100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 --2- 172.21.15.31:0/4273242715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 0x7f0654101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 -- 172.21.15.31:0/4273242715 >> 172.21.15.31:0/4273242715 conn(0x7f06540fb970 msgr2=0x7f06540fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 -- 172.21.15.31:0/4273242715 shutdown_connections 2023-07-29T16:43:01.798 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.794+0000 7f06589ff700 1 -- 172.21.15.31:0/4273242715 wait complete. 2023-07-29T16:43:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 Processor -- start 2023-07-29T16:43:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 -- start start 2023-07-29T16:43:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 0x7f0654196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 0x7f0654196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f065419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.800 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065419dcc0 con 0x7f0654101520 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065419de00 con 0x7f0654101eb0 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.795+0000 7f06589ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065419df90 con 0x7f0654100330 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f065419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0651d9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 0x7f0654196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f065259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 0x7f0654196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f065419d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40292/0 (socket says 172.21.15.31:40292) 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f065259c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 0x7f0654196140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59132/0 (socket says 172.21.15.31:59132) 2023-07-29T16:43:01.801 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 -- 172.21.15.31:0/897411258 learned_addr learned my addr 172.21.15.31:0/897411258 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 -- 172.21.15.31:0/897411258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 msgr2=0x7f0654196140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 0x7f0654196140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 -- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 msgr2=0x7f0654196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 0x7f0654196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 -- 172.21.15.31:0/897411258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c009d00 con 0x7f0654101eb0 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0651d9b700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 0x7f0654196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:43:01.802 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.796+0000 7f0652d9d700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f065419d6f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f064c00b780 tx=0x7f064c00bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:01.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.797+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c00e040 con 0x7f0654101eb0 2023-07-29T16:43:01.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.797+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065419e210 con 0x7f0654101eb0 2023-07-29T16:43:01.803 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.797+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f064c004c60 con 0x7f0654101eb0 2023-07-29T16:43:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.797+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064c01b6b0 con 0x7f0654101eb0 2023-07-29T16:43:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.797+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065419e690 con 0x7f0654101eb0 2023-07-29T16:43:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.798+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065410bb20 con 0x7f0654101eb0 2023-07-29T16:43:01.804 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.800+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f064c01d480 con 0x7f0654101eb0 2023-07-29T16:43:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.801+0000 7f06437fe700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f063c0a0790 0x7f063c0a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.801+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f064c0cd6a0 con 0x7f0654101eb0 2023-07-29T16:43:01.805 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.802+0000 7f065259c700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f063c0a0790 0x7f063c0a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:01.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.802+0000 7f065259c700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f063c0a0790 0x7f063c0a2c40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0654101380 tx=0x7f0644009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:01.806 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:01.803+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f064c097190 con 0x7f0654101eb0 2023-07-29T16:43:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.014+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06541071b0 con 0x7f0654101eb0 2023-07-29T16:43:02.021 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.015+0000 7f06437fe700 1 -- 172.21.15.31:0/897411258 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f064c0210a0 con 0x7f0654101eb0 2023-07-29T16:43:02.021 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2023-07-29T16:43:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f063c0a0790 msgr2=0x7f063c0a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f063c0a0790 0x7f063c0a2c40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0654101380 tx=0x7f0644009260 comp rx=0 tx=0).stop 2023-07-29T16:43:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 msgr2=0x7f065419d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.022 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f065419d6f0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f064c00b780 tx=0x7f064c00bd00 comp rx=0 tx=0).stop 2023-07-29T16:43:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 shutdown_connections 2023-07-29T16:43:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f063c0a0790 0x7f063c0a2c40 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0654101eb0 0x7f065419d6f0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0654100330 0x7f0654196140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 --2- 172.21.15.31:0/897411258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0654101520 0x7f0654196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 >> 172.21.15.31:0/897411258 conn(0x7f06540fb970 msgr2=0x7f06540fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 shutdown_connections 2023-07-29T16:43:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.018+0000 7f06589ff700 1 -- 172.21.15.31:0/897411258 wait complete. 2023-07-29T16:43:02.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-29T16:43:02.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3679518151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:43:02.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:01 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/4075716263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:43:02.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/891158066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101520 msgr2=0x7fc4c4101970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.251 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/891158066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101520 0x7fc4c4101970 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc4b40098f0 tx=0x7fc4b4009c00 comp rx=0 tx=0).stop 2023-07-29T16:43:02.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/891158066 shutdown_connections 2023-07-29T16:43:02.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/891158066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c4101eb0 0x7fc4c4108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/891158066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4100330 0x7fc4c4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/891158066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101520 0x7fc4c4101970 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/891158066 >> 172.21.15.31:0/891158066 conn(0x7fc4c40fb970 msgr2=0x7fc4c40fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:02.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.247+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/891158066 shutdown_connections 2023-07-29T16:43:02.253 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.248+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/891158066 wait complete. 2023-07-29T16:43:02.254 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.249+0000 7fc4cc3ff700 1 Processor -- start 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.249+0000 7fc4cc3ff700 1 -- start start 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c4100330 0x7fc4c4196100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 0x7fc4c4196640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4cc3ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101eb0 0x7fc4c419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c419dc80 con 0x7fc4c4101eb0 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c419ddc0 con 0x7fc4c4101520 2023-07-29T16:43:02.255 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4cc3ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c419df50 con 0x7fc4c4100330 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 0x7fc4c4196640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 0x7fc4c4196640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40306/0 (socket says 172.21.15.31:40306) 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 -- 172.21.15.31:0/3835802625 learned_addr learned my addr 172.21.15.31:0/3835802625 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4ca19b700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c4100330 0x7fc4c4196100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4ca99c700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101eb0 0x7fc4c419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 -- 172.21.15.31:0/3835802625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c4100330 msgr2=0x7fc4c4196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.256 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c4100330 0x7fc4c4196100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 -- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101eb0 msgr2=0x7fc4c419d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101eb0 0x7fc4c419d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.250+0000 7fc4c999a700 1 -- 172.21.15.31:0/3835802625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c00095f0 con 0x7fc4c4101520 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.251+0000 7fc4c999a700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 0x7fc4c4196640 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005e90 tx=0x7fc4b400b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.251+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b401c070 con 0x7fc4c4101520 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.251+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4009520 con 0x7fc4c4101520 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.251+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc4b400f7f0 con 0x7fc4c4101520 2023-07-29T16:43:02.257 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.251+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40049c0 con 0x7fc4c4101520 2023-07-29T16:43:02.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.251+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c419e4c0 con 0x7fc4c4101520 2023-07-29T16:43:02.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.252+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c410bb60 con 0x7fc4c4101520 2023-07-29T16:43:02.258 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.254+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc4b4020070 con 0x7fc4c4101520 2023-07-29T16:43:02.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.256+0000 7fc4bb7fe700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4b00a0790 0x7fc4b00a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.259 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.256+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7fc4b40ccf50 con 0x7fc4c4101520 2023-07-29T16:43:02.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.256+0000 7fc4ca19b700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4b00a0790 0x7fc4b00a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.260 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.256+0000 7fc4ca19b700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4b00a0790 0x7fc4b00a2c40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4101380 tx=0x7fc4c0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:02.261 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.258+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc4b40969c0 con 0x7fc4c4101520 2023-07-29T16:43:02.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3679518151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:43:02.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:01 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/4075716263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:43:02.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3679518151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:43:02.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:01 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/4075716263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-29T16:43:02.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.468+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4c410bd20 con 0x7fc4c4101520 2023-07-29T16:43:02.473 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.469+0000 7fc4bb7fe700 1 -- 172.21.15.31:0/3835802625 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc4b40270a0 con 0x7fc4c4101520 2023-07-29T16:43:02.473 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2023-07-29T16:43:02.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4b00a0790 msgr2=0x7fc4b00a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4b00a0790 0x7fc4b00a2c40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4101380 tx=0x7fc4c0009310 comp rx=0 tx=0).stop 2023-07-29T16:43:02.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 msgr2=0x7fc4c4196640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.475 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 0x7fc4c4196640 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005e90 tx=0x7fc4b400b730 comp rx=0 tx=0).stop 2023-07-29T16:43:02.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 shutdown_connections 2023-07-29T16:43:02.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc4b00a0790 0x7fc4b00a2c40 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc4c4101eb0 0x7fc4c419d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c4100330 0x7fc4c4196100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.476 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 --2- 172.21.15.31:0/3835802625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4c4101520 0x7fc4c4196640 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.471+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 >> 172.21.15.31:0/3835802625 conn(0x7fc4c40fb970 msgr2=0x7fc4c40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:02.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.472+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 shutdown_connections 2023-07-29T16:43:02.477 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.472+0000 7fc4cc3ff700 1 -- 172.21.15.31:0/3835802625 wait complete. 2023-07-29T16:43:02.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-29T16:43:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 -- 172.21.15.31:0/3332958699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90100320 msgr2=0x7f0a90100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 --2- 172.21.15.31:0/3332958699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90100320 0x7f0a90100730 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009990 tx=0x7f0a8c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 -- 172.21.15.31:0/3332958699 shutdown_connections 2023-07-29T16:43:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 --2- 172.21.15.31:0/3332958699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90101ea0 0x7f0a90108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.701 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 --2- 172.21.15.31:0/3332958699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90100320 0x7f0a90100730 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 --2- 172.21.15.31:0/3332958699 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a90101510 0x7f0a90101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 -- 172.21.15.31:0/3332958699 >> 172.21.15.31:0/3332958699 conn(0x7f0a900fb920 msgr2=0x7f0a900fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 -- 172.21.15.31:0/3332958699 shutdown_connections 2023-07-29T16:43:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.697+0000 7f0a985ff700 1 -- 172.21.15.31:0/3332958699 wait complete. 2023-07-29T16:43:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.698+0000 7f0a985ff700 1 Processor -- start 2023-07-29T16:43:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.698+0000 7f0a985ff700 1 -- start start 2023-07-29T16:43:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.698+0000 7f0a985ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90100320 0x7f0a90074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.698+0000 7f0a985ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a90101510 0x7f0a90073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a985ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 0x7f0a90073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a985ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90073e20 con 0x7f0a90101ea0 2023-07-29T16:43:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a985ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a90073f90 con 0x7f0a90100320 2023-07-29T16:43:02.704 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a985ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a901a0120 con 0x7f0a90101510 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 0x7f0a90073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a9639b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90100320 0x7f0a90074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 0x7f0a90073680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43102/0 (socket says 172.21.15.31:43102) 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a95b9a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a90101510 0x7f0a90073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 -- 172.21.15.31:0/2811002893 learned_addr learned my addr 172.21.15.31:0/2811002893 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 -- 172.21.15.31:0/2811002893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a90101510 msgr2=0x7f0a90073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a90101510 0x7f0a90073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 -- 172.21.15.31:0/2811002893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90100320 msgr2=0x7f0a90074b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90100320 0x7f0a90074b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a96b9c700 1 -- 172.21.15.31:0/2811002893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c0095f0 con 0x7f0a90101ea0 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.699+0000 7f0a9639b700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90100320 0x7f0a90074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.700+0000 7f0a96b9c700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 0x7f0a90073680 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0a88007ee0 tx=0x7f0a8800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.700+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8800dd20 con 0x7f0a90101ea0 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.700+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a901a0400 con 0x7f0a90101ea0 2023-07-29T16:43:02.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.700+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a8800f040 con 0x7f0a90101ea0 2023-07-29T16:43:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.700+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a880048a0 con 0x7f0a90101ea0 2023-07-29T16:43:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.700+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a901a0960 con 0x7f0a90101ea0 2023-07-29T16:43:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.701+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9010bb40 con 0x7f0a90101ea0 2023-07-29T16:43:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.703+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0a88003a00 con 0x7f0a90101ea0 2023-07-29T16:43:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.704+0000 7f0a877fe700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a7c0a0810 0x7f0a7c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.704+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0a880cb290 con 0x7f0a90101ea0 2023-07-29T16:43:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.704+0000 7f0a9639b700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a7c0a0810 0x7f0a7c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.705+0000 7f0a9639b700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a7c0a0810 0x7f0a7c0a2cc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00b2e0 tx=0x7f0a8c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:02.712 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.708+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0a8804e720 con 0x7f0a90101ea0 2023-07-29T16:43:02.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.935+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a9010bd00 con 0x7f0a90101ea0 2023-07-29T16:43:02.941 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.937+0000 7f0a877fe700 1 -- 172.21.15.31:0/2811002893 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) v1 ==== 81+0+585 (secure 0 0 0) 0x7f0a88094c70 con 0x7f0a90101ea0 2023-07-29T16:43:02.941 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3069668148 2023-07-30T16:05:49.191872+0000 2023-07-29T16:43:02.941 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/1090542383 2023-07-30T16:05:49.191872+0000 2023-07-29T16:43:02.941 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/1090542383 2023-07-30T16:05:49.191872+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3115186416 2023-07-30T16:05:49.191872+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/480480667 2023-07-30T16:05:22.573088+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3981163231 2023-07-30T16:05:22.573088+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/890525828 2023-07-30T16:05:22.573088+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/76880357 2023-07-30T16:05:22.573088+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/76880357 2023-07-30T16:05:22.573088+0000 2023-07-29T16:43:02.942 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/538594089 2023-07-30T16:05:49.191872+0000 2023-07-29T16:43:02.943 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a7c0a0810 msgr2=0x7f0a7c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a7c0a0810 0x7f0a7c0a2cc0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00b2e0 tx=0x7f0a8c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 msgr2=0x7f0a90073680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 0x7f0a90073680 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0a88007ee0 tx=0x7f0a8800d3b0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 shutdown_connections 2023-07-29T16:43:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0a7c0a0810 0x7f0a7c0a2cc0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.944 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0a90101ea0 0x7f0a90073680 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a90100320 0x7f0a90074b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 --2- 172.21.15.31:0/2811002893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a90101510 0x7f0a90073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:02.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 >> 172.21.15.31:0/2811002893 conn(0x7f0a900fb920 msgr2=0x7f0a900fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:02.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 shutdown_connections 2023-07-29T16:43:02.945 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:02.940+0000 7f0a985ff700 1 -- 172.21.15.31:0/2811002893 wait complete. 2023-07-29T16:43:02.946 INFO:teuthology.orchestra.run.smithi031.stderr:listed 10 entries 2023-07-29T16:43:02.963 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-07-29T16:43:02.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:02.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-07-29T16:43:03.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:02 smithi031 ceph-mon[95504]: pgmap v1199: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-07-29T16:43:03.239 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:02 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/897411258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:43:03.240 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:02 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3835802625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:43:03.241 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T16:43:03.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:03.242 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-29T16:43:03.286 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:02 smithi039 ceph-mon[128858]: pgmap v1199: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-07-29T16:43:03.287 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:02 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/897411258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:43:03.287 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:02 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3835802625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:43:03.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:02 smithi156 ceph-mon[129245]: pgmap v1199: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-07-29T16:43:03.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:02 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/897411258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-29T16:43:03.329 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:02 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3835802625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-29T16:43:03.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:03.476 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-07-29T16:43:03.696 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-29T16:43:03.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:03.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-29T16:43:03.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:03.812 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2023-07-29T16:43:03.933 INFO:tasks.kclient:Unmounting kernel clients... 2023-07-29T16:43:03.933 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-07-29T16:43:03.934 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-29T16:43:03.981 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=16348708k,nr_inodes=4087177,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,seclabel,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,seclabel,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,seclabel,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,seclabel,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,seclabel,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,seclabel,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,seclabel,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,seclabel,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,seclabel,nosuid,nodev,noexec,relatime,hugetlb 0 0 none /sys/kernel/tracing tracefs rw,seclabel,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275628k,mode=700,uid=1000,gid=1265 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/0d5815d2d32d5fcc067a516e35b317f9f11bd1a9ccee7c6d1b03cc7f76367cd5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/0d5815d2d32d5fcc067a516e35b317f9f11bd1a9ccee7c6d1b03cc7f76367cd5/diff,workdir=/var/lib/containers/storage/overlay/0d5815d2d32d5fcc067a516e35b317f9f11bd1a9ccee7c6d1b03cc7f76367cd5/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/279af28f05b464fc6a3454601bb00cbc7fc943af186692254a9d22b9caf71a39/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/279af28f05b464fc6a3454601bb00cbc7fc943af186692254a9d22b9caf71a39/diff,workdir=/var/lib/containers/storage/overlay/279af28f05b464fc6a3454601bb00cbc7fc943af186692254a9d22b9caf71a39/work,redirect_dir=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3275628k,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 overlay /var/lib/containers/storage/overlay/3818da0e52bf3ce65076da4e027db623b241a67d5bbec2e9171c15cb33aaef5f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/3818da0e52bf3ce65076da4e027db623b241a67d5bbec2e9171c15cb33aaef5f/diff,workdir=/var/lib/containers/storage/overlay/3818da0e52bf3ce65076da4e027db623b241a67d5bbec2e9171c15cb33aaef5f/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0ad7480e52698688b58ef1a6cbca439e2392dbfd2aa6b1691061a3000aad2f98/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/0ad7480e52698688b58ef1a6cbca439e2392dbfd2aa6b1691061a3000aad2f98/diff,workdir=/var/lib/containers/storage/overlay/0ad7480e52698688b58ef1a6cbca439e2392dbfd2aa6b1691061a3000aad2f98/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/1a34e2a9c556be353c88de411e4e895323328fefd7629d4db070631963ffff6e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/1a34e2a9c556be353c88de411e4e895323328fefd7629d4db070631963ffff6e/diff,workdir=/var/lib/containers/storage/overlay/1a34e2a9c556be353c88de411e4e895323328fefd7629d4db070631963ffff6e/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c048d18d117977fd376ef8691e2a77cb79a81d0884d56132d8fc98d39033e145/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/c048d18d117977fd376ef8691e2a77cb79a81d0884d56132d8fc98d39033e145/diff,workdir=/var/lib/containers/storage/overlay/c048d18d117977fd376ef8691e2a77cb79a81d0884d56132d8fc98d39033e145/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a36d8c05ca002ad0243610e217f6ade4799fd936f9a0da8bf7dedc26f76be3f8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/a36d8c05ca002ad0243610e217f6ade4799fd936f9a0da8bf7dedc26f76be3f8/diff,workdir=/var/lib/containers/storage/overlay/a36d8c05ca002ad0243610e217f6ade4799fd936f9a0da8bf7dedc26f76be3f8/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6a8c8374bcb605f92b816d7a891638b452c498a8191231911d89566b07c20a60/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/6a8c8374bcb605f92b816d7a891638b452c498a8191231911d89566b07c20a60/diff,workdir=/var/lib/containers/storage/overlay/6a8c8374bcb605f92b816d7a891638b452c498a8191231911d89566b07c20a60/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6c6b111cfd14ac5f14bdf88651714711296670dc25c78c2e73661f22f7aa779f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/6c6b111cfd14ac5f14bdf88651714711296670dc25c78c2e73661f22f7aa779f/diff,workdir=/var/lib/containers/storage/overlay/6c6b111cfd14ac5f14bdf88651714711296670dc25c78c2e73661f22f7aa779f/work,redirect_dir=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/283d66a14cd62dc55a05dffdce669c9c904fbb7f0aa2afcd1fbf1439d12971f7/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/P3CSEDU3MXXRE2XGYRRBEKAJLT:/var/lib/containers/storage/overlay/l/ISY7476AOWCZOVEUZIXURECQUB,upperdir=/var/lib/containers/storage/overlay/283d66a14cd62dc55a05dffdce669c9c904fbb7f0aa2afcd1fbf1439d12971f7/diff,workdir=/var/lib/containers/storage/overlay/283d66a14cd62dc55a05dffdce669c9c904fbb7f0aa2afcd1fbf1439d12971f7/work,redirect_dir=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0 2023-07-29T16:43:03.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:03.982 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2023-07-29T16:43:04.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-29T16:43:04.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2023-07-29T16:43:04.167 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:03 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2811002893' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-29T16:43:04.255 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2023-07-29T16:43:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:43:04.260 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-29T16:43:04.302 INFO:tasks.cephadm:Teardown begin 2023-07-29T16:43:04.302 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-29T16:43:04.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:43:04.303 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:03 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2811002893' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-29T16:43:04.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:03 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2811002893' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-29T16:43:04.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 -- 172.21.15.31:0/3692035559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520100df0 msgr2=0x7f05201031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:04.508 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 --2- 172.21.15.31:0/3692035559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520100df0 0x7f05201031d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f0510009990 tx=0x7f0510009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 -- 172.21.15.31:0/3692035559 shutdown_connections 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 --2- 172.21.15.31:0/3692035559 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520106030 0x7f0520108430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 --2- 172.21.15.31:0/3692035559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520100df0 0x7f05201031d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 --2- 172.21.15.31:0/3692035559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0520103710 0x7f0520105af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 -- 172.21.15.31:0/3692035559 >> 172.21.15.31:0/3692035559 conn(0x7f05200fa810 msgr2=0x7f05200fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 -- 172.21.15.31:0/3692035559 shutdown_connections 2023-07-29T16:43:04.509 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.504+0000 7f05259ff700 1 -- 172.21.15.31:0/3692035559 wait complete. 2023-07-29T16:43:04.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.506+0000 7f05259ff700 1 Processor -- start 2023-07-29T16:43:04.510 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.506+0000 7f05259ff700 1 -- start start 2023-07-29T16:43:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f05259ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520100df0 0x7f0520100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f05259ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 0x7f05200feda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f05259ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0520106030 0x7f05200ff2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f05259ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05200ffa80 con 0x7f0520103710 2023-07-29T16:43:04.512 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f05259ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05200ffbf0 con 0x7f0520100df0 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f05259ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05201a00c0 con 0x7f0520106030 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f051e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 0x7f05200feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f051e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 0x7f05200feda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43122/0 (socket says 172.21.15.31:43122) 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f051effd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520100df0 0x7f0520100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f051e7fc700 1 -- 172.21.15.31:0/3966184644 learned_addr learned my addr 172.21.15.31:0/3966184644 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.507+0000 7f051f7fe700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0520106030 0x7f05200ff2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:04.513 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f051e7fc700 1 -- 172.21.15.31:0/3966184644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0520106030 msgr2=0x7f05200ff2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f051e7fc700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0520106030 0x7f05200ff2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f051e7fc700 1 -- 172.21.15.31:0/3966184644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520100df0 msgr2=0x7f0520100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f051e7fc700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520100df0 0x7f0520100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f051e7fc700 1 -- 172.21.15.31:0/3966184644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05100095f0 con 0x7f0520103710 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f051e7fc700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 0x7f05200feda0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f051400e9a0 tx=0x7f051400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514009790 con 0x7f0520103710 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0514010430 con 0x7f0520103710 2023-07-29T16:43:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514018680 con 0x7f0520103710 2023-07-29T16:43:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05201a03a0 con 0x7f0520103710 2023-07-29T16:43:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.508+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05201a08b0 con 0x7f0520103710 2023-07-29T16:43:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.509+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052010bb60 con 0x7f0520103710 2023-07-29T16:43:04.515 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.512+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f05140105a0 con 0x7f0520103710 2023-07-29T16:43:04.516 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.513+0000 7f05249fd700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05080a0860 0x7f05080a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.513+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f0514014070 con 0x7f0520103710 2023-07-29T16:43:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.513+0000 7f051effd700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05080a0860 0x7f05080a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.514+0000 7f051effd700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05080a0860 0x7f05080a2d10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f051000b2e0 tx=0x7f0510005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:04.518 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.515+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0514095980 con 0x7f0520103710 2023-07-29T16:43:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.769+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f052010bd50 con 0x7f0520103710 2023-07-29T16:43:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.769+0000 7f05249fd700 1 -- 172.21.15.31:0/3966184644 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+10788 (secure 0 0 0) 0x7f0514026070 con 0x7f0520103710 2023-07-29T16:43:04.773 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:43:04.773 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":20124,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1692916896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1692916896},{"type":"v1","addr":"172.21.15.156:6837","nonce":1692916896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":46,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:42:23.357229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":282,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:43:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05080a0860 msgr2=0x7f05080a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05080a0860 0x7f05080a2d10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f051000b2e0 tx=0x7f0510005fb0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 msgr2=0x7f05200feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 0x7f05200feda0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f051400e9a0 tx=0x7f051400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 shutdown_connections 2023-07-29T16:43:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f05080a0860 0x7f05080a2d10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0520106030 0x7f05200ff2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0520100df0 0x7f0520100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 --2- 172.21.15.31:0/3966184644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0520103710 0x7f05200feda0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 >> 172.21.15.31:0/3966184644 conn(0x7f05200fa810 msgr2=0x7f0520102260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 shutdown_connections 2023-07-29T16:43:04.778 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.772+0000 7f05259ff700 1 -- 172.21.15.31:0/3966184644 wait complete. 2023-07-29T16:43:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2023-07-29T16:43:04.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-29T16:43:05.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 -- 172.21.15.31:0/1218516592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8073730 msgr2=0x7f31f8073ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 --2- 172.21.15.31:0/1218516592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8073730 0x7f31f8073ba0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f31e8009990 tx=0x7f31e8009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 -- 172.21.15.31:0/1218516592 shutdown_connections 2023-07-29T16:43:05.001 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 --2- 172.21.15.31:0/1218516592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f81081f0 0x7f31f810a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 --2- 172.21.15.31:0/1218516592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31f8074d10 0x7f31f8073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 --2- 172.21.15.31:0/1218516592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8073730 0x7f31f8073ba0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 -- 172.21.15.31:0/1218516592 >> 172.21.15.31:0/1218516592 conn(0x7f31f80fb920 msgr2=0x7f31f80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 -- 172.21.15.31:0/1218516592 shutdown_connections 2023-07-29T16:43:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.997+0000 7f32007ff700 1 -- 172.21.15.31:0/1218516592 wait complete. 2023-07-29T16:43:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.999+0000 7f32007ff700 1 Processor -- start 2023-07-29T16:43:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.999+0000 7f32007ff700 1 -- start start 2023-07-29T16:43:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.999+0000 7f32007ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f8073730 0x7f31f8198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:04.999+0000 7f32007ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 0x7f31f8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f32007ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31f81081f0 0x7f31f819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f32007ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f819fd80 con 0x7f31f8074d10 2023-07-29T16:43:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f32007ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f819fec0 con 0x7f31f8073730 2023-07-29T16:43:05.004 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f32007ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f81a0050 con 0x7f31f81081f0 2023-07-29T16:43:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 0x7f31f8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 0x7f31f8198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43152/0 (socket says 172.21.15.31:43152) 2023-07-29T16:43:05.005 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fe59b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f8073730 0x7f31f8198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 -- 172.21.15.31:0/639318668 learned_addr learned my addr 172.21.15.31:0/639318668 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fed9c700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31f81081f0 0x7f31f819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 -- 172.21.15.31:0/639318668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31f81081f0 msgr2=0x7f31f819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31f81081f0 0x7f31f819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 -- 172.21.15.31:0/639318668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f8073730 msgr2=0x7f31f8198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f8073730 0x7f31f8198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.006 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 -- 172.21.15.31:0/639318668 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e80095f0 con 0x7f31f8074d10 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fe59b700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f8073730 0x7f31f8198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.000+0000 7f31fdd9a700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 0x7f31f8198740 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f31e800b680 tx=0x7f31e8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.001+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e801d070 con 0x7f31f8074d10 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.001+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31e8022470 con 0x7f31f8074d10 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.001+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f81a02d0 con 0x7f31f8074d10 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.001+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e8017650 con 0x7f31f8074d10 2023-07-29T16:43:05.007 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.001+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f81a06f0 con 0x7f31f8074d10 2023-07-29T16:43:05.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.002+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f8106930 con 0x7f31f8074d10 2023-07-29T16:43:05.008 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.004+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f31e80225e0 con 0x7f31f8074d10 2023-07-29T16:43:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.005+0000 7f31ef7fe700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31e40a07e0 0x7f31e40a2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.006+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f31e80cd680 con 0x7f31f8074d10 2023-07-29T16:43:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.006+0000 7f31fe59b700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31e40a07e0 0x7f31e40a2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.006+0000 7f31fe59b700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31e40a07e0 0x7f31e40a2c90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f31f4011490 tx=0x7f31f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:05.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.007+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f31e80970f0 con 0x7f31f8074d10 2023-07-29T16:43:05.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:04 smithi031 ceph-mon[95504]: pgmap v1200: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-29T16:43:05.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:04 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3966184644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:43:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.263+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31f802cc30 con 0x7f31f8074d10 2023-07-29T16:43:05.267 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.264+0000 7f31ef7fe700 1 -- 172.21.15.31:0/639318668 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+10788 (secure 0 0 0) 0x7f31e809a710 con 0x7f31f8074d10 2023-07-29T16:43:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:43:05.268 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14499,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/2459121100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":2459121100},{"type":"v1","addr":"172.21.15.31:6835","nonce":2459121100}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14586,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/3775768915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":3775768915},{"type":"v1","addr":"172.21.15.31:6841","nonce":3775768915}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":20124,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/1692916896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":1692916896},{"type":"v1","addr":"172.21.15.156:6837","nonce":1692916896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/2085573271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":2085573271},{"type":"v1","addr":"172.21.15.156:6833","nonce":2085573271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24464,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/3785261631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":3785261631},{"type":"v1","addr":"172.21.15.39:6833","nonce":3785261631}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/3256727678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":3256727678},{"type":"v1","addr":"172.21.15.31:6837","nonce":3256727678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":46,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-29T16:12:21.912364+0000","modified":"2023-07-29T16:42:23.357229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24581,"mds_1":24551,"mds_2":24550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6835/2848213481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2848213481},{"type":"v1","addr":"172.21.15.39:6835","nonce":2848213481}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1642153321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1642153321},{"type":"v1","addr":"172.21.15.156:6835","nonce":1642153321}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6837/1297587845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":1297587845},{"type":"v1","addr":"172.21.15.39:6837","nonce":1297587845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24550":{"gid":24550,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":282,"addr":"172.21.15.156:6839/2888902152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2888902152},{"type":"v1","addr":"172.21.15.156:6839","nonce":2888902152}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24551":{"gid":24551,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6839/3329463675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3329463675},{"type":"v1","addr":"172.21.15.31:6839","nonce":3329463675}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24581":{"gid":24581,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.39:6839/2751828361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":2751828361},{"type":"v1","addr":"172.21.15.39:6839","nonce":2751828361}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-29T16:43:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31e40a07e0 msgr2=0x7f31e40a2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.270 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31e40a07e0 0x7f31e40a2c90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f31f4011490 tx=0x7f31f4009260 comp rx=0 tx=0).stop 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 msgr2=0x7f31f8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 0x7f31f8198740 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f31e800b680 tx=0x7f31e8005740 comp rx=0 tx=0).stop 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 shutdown_connections 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f31e40a07e0 0x7f31e40a2c90 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f31f81081f0 0x7f31f819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31f8073730 0x7f31f8198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.271 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 --2- 172.21.15.31:0/639318668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31f8074d10 0x7f31f8198740 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 >> 172.21.15.31:0/639318668 conn(0x7f31f80fb920 msgr2=0x7f31f810a510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 shutdown_connections 2023-07-29T16:43:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.267+0000 7f32007ff700 1 -- 172.21.15.31:0/639318668 wait complete. 2023-07-29T16:43:05.272 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 46 2023-07-29T16:43:05.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:04 smithi039 ceph-mon[128858]: pgmap v1200: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-29T16:43:05.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:04 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3966184644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:43:05.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-29T16:43:05.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:04 smithi156 ceph-mon[129245]: pgmap v1200: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-29T16:43:05.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:04 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3966184644' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:43:05.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 -- 172.21.15.31:0/1212156187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710100360 msgr2=0x7f6710100770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 --2- 172.21.15.31:0/1212156187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710100360 0x7f6710100770 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f670c009990 tx=0x7f670c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 -- 172.21.15.31:0/1212156187 shutdown_connections 2023-07-29T16:43:05.493 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 --2- 172.21.15.31:0/1212156187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6710101ee0 0x7f67101087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 --2- 172.21.15.31:0/1212156187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710100360 0x7f6710100770 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 --2- 172.21.15.31:0/1212156187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710101550 0x7f67101019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 -- 172.21.15.31:0/1212156187 >> 172.21.15.31:0/1212156187 conn(0x7f67100fb9a0 msgr2=0x7f67100fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:05.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 -- 172.21.15.31:0/1212156187 shutdown_connections 2023-07-29T16:43:05.494 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.489+0000 7f67183ff700 1 -- 172.21.15.31:0/1212156187 wait complete. 2023-07-29T16:43:05.495 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.491+0000 7f67183ff700 1 Processor -- start 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.491+0000 7f67183ff700 1 -- start start 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.491+0000 7f67183ff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 0x7f6710074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.491+0000 7f67183ff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710101550 0x7f6710073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.491+0000 7f67183ff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6710101ee0 0x7f6710073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f67183ff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6710073e20 con 0x7f6710101550 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f67183ff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6710073f90 con 0x7f6710100360 2023-07-29T16:43:05.496 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f67183ff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67101a00c0 con 0x7f6710101ee0 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 0x7f6710074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 0x7f6710074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40370/0 (socket says 172.21.15.31:40370) 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 -- 172.21.15.31:0/2445948903 learned_addr learned my addr 172.21.15.31:0/2445948903 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671599a700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710101550 0x7f6710073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671699c700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6710101ee0 0x7f6710073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 -- 172.21.15.31:0/2445948903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6710101ee0 msgr2=0x7f6710073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6710101ee0 0x7f6710073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 -- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710101550 msgr2=0x7f6710073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710101550 0x7f6710073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 -- 172.21.15.31:0/2445948903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c0095f0 con 0x7f6710100360 2023-07-29T16:43:05.498 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671599a700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710101550 0x7f6710073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-29T16:43:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.492+0000 7f671619b700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 0x7f6710074b00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f670c005740 tx=0x7f670c009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.493+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c01d070 con 0x7f6710100360 2023-07-29T16:43:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.493+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f670c021410 con 0x7f6710100360 2023-07-29T16:43:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.493+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c017650 con 0x7f6710100360 2023-07-29T16:43:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.493+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67101a0340 con 0x7f6710100360 2023-07-29T16:43:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.493+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67101a0760 con 0x7f6710100360 2023-07-29T16:43:05.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.494+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671010bb80 con 0x7f6710100360 2023-07-29T16:43:05.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.496+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f670c021810 con 0x7f6710100360 2023-07-29T16:43:05.500 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.497+0000 7f67077fe700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f66fc0a0860 0x7f66fc0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.497+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f670c0ccf50 con 0x7f6710100360 2023-07-29T16:43:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.497+0000 7f671599a700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f66fc0a0860 0x7f66fc0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.498+0000 7f671599a700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f66fc0a0860 0x7f66fc0a2d10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6700009cb0 tx=0x7f6700009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.501+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f670c0969c0 con 0x7f6710100360 2023-07-29T16:43:05.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.713+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67101a0da0 con 0x7f6710100360 2023-07-29T16:43:05.719 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.716+0000 7f67077fe700 1 -- 172.21.15.31:0/2445948903 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+22731 (secure 0 0 0) 0x7f670c026090 con 0x7f6710100360 2023-07-29T16:43:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-07-29T16:43:05.722 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":93,"fsid":"858a7894-2e29-11ee-9b36-001a4aab830c","created":"2023-07-29T16:04:46.417698+0000","modified":"2023-07-29T16:35:01.881611+0000","last_up_change":"2023-07-29T16:11:37.272686+0000","last_in_change":"2023-07-29T16:11:19.885660+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-29T16:08:09.787659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-29T16:12:20.183673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-29T16:12:21.310891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f598978-c50a-4143-a36d-091f24731a98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6803","nonce":2308009899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6805","nonce":2308009899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6809","nonce":2308009899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2308009899},{"type":"v1","addr":"172.21.15.31:6807","nonce":2308009899}]},"public_addr":"172.21.15.31:6803/2308009899","cluster_addr":"172.21.15.31:6805/2308009899","heartbeat_back_addr":"172.21.15.31:6809/2308009899","heartbeat_front_addr":"172.21.15.31:6807/2308009899","state":["exists","up"]},{"osd":1,"uuid":"2ceb36a3-18a7-4af6-ae87-03d42276da3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6801","nonce":701098917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6803","nonce":701098917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6807","nonce":701098917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":701098917},{"type":"v1","addr":"172.21.15.39:6805","nonce":701098917}]},"public_addr":"172.21.15.39:6801/701098917","cluster_addr":"172.21.15.39:6803/701098917","heartbeat_back_addr":"172.21.15.39:6807/701098917","heartbeat_front_addr":"172.21.15.39:6805/701098917","state":["exists","up"]},{"osd":2,"uuid":"b70de0c9-39ca-4d04-bbe3-532e576fd586","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6801","nonce":3425187698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6803","nonce":3425187698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6807","nonce":3425187698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":3425187698},{"type":"v1","addr":"172.21.15.156:6805","nonce":3425187698}]},"public_addr":"172.21.15.156:6801/3425187698","cluster_addr":"172.21.15.156:6803/3425187698","heartbeat_back_addr":"172.21.15.156:6807/3425187698","heartbeat_front_addr":"172.21.15.156:6805/3425187698","state":["exists","up"]},{"osd":3,"uuid":"de564104-d127-432d-b638-ab1547a64d6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6811","nonce":1855156941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6813","nonce":1855156941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6817","nonce":1855156941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1855156941},{"type":"v1","addr":"172.21.15.31:6815","nonce":1855156941}]},"public_addr":"172.21.15.31:6811/1855156941","cluster_addr":"172.21.15.31:6813/1855156941","heartbeat_back_addr":"172.21.15.31:6817/1855156941","heartbeat_front_addr":"172.21.15.31:6815/1855156941","state":["exists","up"]},{"osd":4,"uuid":"8c0cb642-c0c5-4349-8c45-584df086b71b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6809","nonce":1826445248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6811","nonce":1826445248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6815","nonce":1826445248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1826445248},{"type":"v1","addr":"172.21.15.39:6813","nonce":1826445248}]},"public_addr":"172.21.15.39:6809/1826445248","cluster_addr":"172.21.15.39:6811/1826445248","heartbeat_back_addr":"172.21.15.39:6815/1826445248","heartbeat_front_addr":"172.21.15.39:6813/1826445248","state":["exists","up"]},{"osd":5,"uuid":"37a3e4ce-86d4-41ba-8544-bb8dbc3dd0df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6809","nonce":1995637959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6811","nonce":1995637959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6815","nonce":1995637959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1995637959},{"type":"v1","addr":"172.21.15.156:6813","nonce":1995637959}]},"public_addr":"172.21.15.156:6809/1995637959","cluster_addr":"172.21.15.156:6811/1995637959","heartbeat_back_addr":"172.21.15.156:6815/1995637959","heartbeat_front_addr":"172.21.15.156:6813/1995637959","state":["exists","up"]},{"osd":6,"uuid":"e93cbf91-6144-4ac7-8bb6-1e964c5945d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6819","nonce":2073594680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6821","nonce":2073594680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6825","nonce":2073594680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2073594680},{"type":"v1","addr":"172.21.15.31:6823","nonce":2073594680}]},"public_addr":"172.21.15.31:6819/2073594680","cluster_addr":"172.21.15.31:6821/2073594680","heartbeat_back_addr":"172.21.15.31:6825/2073594680","heartbeat_front_addr":"172.21.15.31:6823/2073594680","state":["exists","up"]},{"osd":7,"uuid":"78eb79f8-f2ac-4b63-a6d7-431e91959528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6817","nonce":4224022326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6819","nonce":4224022326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6823","nonce":4224022326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":4224022326},{"type":"v1","addr":"172.21.15.39:6821","nonce":4224022326}]},"public_addr":"172.21.15.39:6817/4224022326","cluster_addr":"172.21.15.39:6819/4224022326","heartbeat_back_addr":"172.21.15.39:6823/4224022326","heartbeat_front_addr":"172.21.15.39:6821/4224022326","state":["exists","up"]},{"osd":8,"uuid":"fead396d-f980-4122-8ec0-cbb58dd236fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6817","nonce":2938271220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6819","nonce":2938271220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6823","nonce":2938271220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":2938271220},{"type":"v1","addr":"172.21.15.156:6821","nonce":2938271220}]},"public_addr":"172.21.15.156:6817/2938271220","cluster_addr":"172.21.15.156:6819/2938271220","heartbeat_back_addr":"172.21.15.156:6823/2938271220","heartbeat_front_addr":"172.21.15.156:6821/2938271220","state":["exists","up"]},{"osd":9,"uuid":"4646b9f8-76b7-4985-b448-0323ecbe272a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6827","nonce":3232146044}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6829","nonce":3232146044}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6833","nonce":3232146044}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3232146044},{"type":"v1","addr":"172.21.15.31:6831","nonce":3232146044}]},"public_addr":"172.21.15.31:6827/3232146044","cluster_addr":"172.21.15.31:6829/3232146044","heartbeat_back_addr":"172.21.15.31:6833/3232146044","heartbeat_front_addr":"172.21.15.31:6831/3232146044","state":["exists","up"]},{"osd":10,"uuid":"7ced3e9f-39cb-4112-bd93-5a36a3949bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6825","nonce":2960119201}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6827","nonce":2960119201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6831","nonce":2960119201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":2960119201},{"type":"v1","addr":"172.21.15.39:6829","nonce":2960119201}]},"public_addr":"172.21.15.39:6825/2960119201","cluster_addr":"172.21.15.39:6827/2960119201","heartbeat_back_addr":"172.21.15.39:6831/2960119201","heartbeat_front_addr":"172.21.15.39:6829/2960119201","state":["exists","up"]},{"osd":11,"uuid":"8bbdc9b7-5087-4e7d-b6b6-0c55a6144a19","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6825","nonce":2124313764}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6827","nonce":2124313764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6831","nonce":2124313764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2124313764},{"type":"v1","addr":"172.21.15.156:6829","nonce":2124313764}]},"public_addr":"172.21.15.156:6825/2124313764","cluster_addr":"172.21.15.156:6827/2124313764","heartbeat_back_addr":"172.21.15.156:6831/2124313764","heartbeat_front_addr":"172.21.15.156:6829/2124313764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:20.816473+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:07:45.209016+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:06.398748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:31.457105+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:08:55.089353+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:15.898069+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:09:41.177034+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:04.614145+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:25.628436+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:10:50.864665+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:14.686350+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-29T16:11:35.523383+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":4}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1},{"from":5,"to":11}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/3069668148":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6801/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:6800/1090542383":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/3115186416":"2023-07-30T16:05:49.191872+0000","172.21.15.31:0/480480667":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/3981163231":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/890525828":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6801/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:6800/76880357":"2023-07-30T16:05:22.573088+0000","172.21.15.31:0/538594089":"2023-07-30T16:05:49.191872+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-29T16:43:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.718+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f66fc0a0860 msgr2=0x7f66fc0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.718+0000 7f67183ff700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f66fc0a0860 0x7f66fc0a2d10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6700009cb0 tx=0x7f6700009260 comp rx=0 tx=0).stop 2023-07-29T16:43:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.718+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 msgr2=0x7f6710074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.718+0000 7f67183ff700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 0x7f6710074b00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f670c005740 tx=0x7f670c009510 comp rx=0 tx=0).stop 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 shutdown_connections 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f66fc0a0860 0x7f66fc0a2d10 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6710101ee0 0x7f6710073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6710100360 0x7f6710074b00 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 --2- 172.21.15.31:0/2445948903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6710101550 0x7f6710073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 >> 172.21.15.31:0/2445948903 conn(0x7f67100fb9a0 msgr2=0x7f67100fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 shutdown_connections 2023-07-29T16:43:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.719+0000 7f67183ff700 1 -- 172.21.15.31:0/2445948903 wait complete. 2023-07-29T16:43:05.742 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-29T16:43:05.947 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 -- 172.21.15.31:0/209457734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638074d90 msgr2=0x7f96380731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 --2- 172.21.15.31:0/209457734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638074d90 0x7f96380731e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f9628009990 tx=0x7f9628009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 -- 172.21.15.31:0/209457734 shutdown_connections 2023-07-29T16:43:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 --2- 172.21.15.31:0/209457734 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963810a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 --2- 172.21.15.31:0/209457734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9638074d90 0x7f96380731e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.948 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 --2- 172.21.15.31:0/209457734 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f96380737b0 0x7f9638073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 -- 172.21.15.31:0/209457734 >> 172.21.15.31:0/209457734 conn(0x7f96380fb970 msgr2=0x7f96380fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 -- 172.21.15.31:0/209457734 shutdown_connections 2023-07-29T16:43:05.949 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.944+0000 7f963ddff700 1 -- 172.21.15.31:0/209457734 wait complete. 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.945+0000 7f963ddff700 1 Processor -- start 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.945+0000 7f963ddff700 1 -- start start 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f963ddff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380737b0 0x7f9638198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f963ddff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9638074d90 0x7f9638198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f963ddff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f963ddff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963819feb0 con 0x7f96380737b0 2023-07-29T16:43:05.950 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f963ddff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963819fff0 con 0x7f96381081f0 2023-07-29T16:43:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f963ddff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96381a0180 con 0x7f9638074d90 2023-07-29T16:43:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f9637fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f96377fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380737b0 0x7f9638198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.951 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f9636ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9638074d90 0x7f9638198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f9637fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963819f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40396/0 (socket says 172.21.15.31:40396) 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.946+0000 7f9637fff700 1 -- 172.21.15.31:0/1733631249 learned_addr learned my addr 172.21.15.31:0/1733631249 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.947+0000 7f9637fff700 1 -- 172.21.15.31:0/1733631249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9638074d90 msgr2=0x7f9638198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.947+0000 7f9637fff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9638074d90 0x7f9638198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.947+0000 7f9637fff700 1 -- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380737b0 msgr2=0x7f9638198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.947+0000 7f9637fff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380737b0 0x7f9638198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:05.952 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.947+0000 7f9637fff700 1 -- 172.21.15.31:0/1733631249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630009c40 con 0x7f96381081f0 2023-07-29T16:43:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.947+0000 7f9637fff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963819f8e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f963000c370 tx=0x7f963000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.948+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630017070 con 0x7f96381081f0 2023-07-29T16:43:05.953 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.949+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f963000f040 con 0x7f96381081f0 2023-07-29T16:43:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.949+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96280095f0 con 0x7f96381081f0 2023-07-29T16:43:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.949+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963000e050 con 0x7f96381081f0 2023-07-29T16:43:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.949+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96381a07d0 con 0x7f96381081f0 2023-07-29T16:43:05.954 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.950+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96381068a0 con 0x7f96381081f0 2023-07-29T16:43:05.956 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.953+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f9630004bd0 con 0x7f96381081f0 2023-07-29T16:43:05.957 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.954+0000 7f9634ff9700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96200a0810 0x7f96200a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:05.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.955+0000 7f96377fe700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96200a0810 0x7f96200a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:05.958 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.955+0000 7f96377fe700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96200a0810 0x7f96200a2cc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f962800b2e0 tx=0x7f9628005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:05.959 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.956+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8335+0+0 (secure 0 0 0) 0x7f96300092e0 con 0x7f96381081f0 2023-07-29T16:43:05.967 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:05.963+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f9630094da0 con 0x7f96381081f0 2023-07-29T16:43:06.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:05 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/639318668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:43:06.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:05 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/2445948903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:43:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.221+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f963802cc30 con 0x7f96381081f0 2023-07-29T16:43:06.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:05 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/639318668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:43:06.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:05 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/2445948903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:43:06.325 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:05 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/639318668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-29T16:43:06.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:05 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/2445948903' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-29T16:43:06.964 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.960+0000 7f9634ff9700 1 -- 172.21.15.31:0/1733631249 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v47) v1 ==== 161+0+0 (secure 0 0 0) 0x7f96300983c0 con 0x7f96381081f0 2023-07-29T16:43:06.968 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96200a0810 msgr2=0x7f96200a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96200a0810 0x7f96200a2cc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f962800b2e0 tx=0x7f9628005fb0 comp rx=0 tx=0).stop 2023-07-29T16:43:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 msgr2=0x7f963819f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963819f8e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f963000c370 tx=0x7f963000c680 comp rx=0 tx=0).stop 2023-07-29T16:43:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 shutdown_connections 2023-07-29T16:43:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f96200a0810 0x7f96200a2cc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96381081f0 0x7f963819f8e0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96380737b0 0x7f9638198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 --2- 172.21.15.31:0/1733631249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9638074d90 0x7f9638198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 >> 172.21.15.31:0/1733631249 conn(0x7f96380fb970 msgr2=0x7f96380fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 shutdown_connections 2023-07-29T16:43:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:06.965+0000 7f963ddff700 1 -- 172.21.15.31:0/1733631249 wait complete. 2023-07-29T16:43:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-29T16:43:06.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-29T16:43:07.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 -- 172.21.15.31:0/2226049441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50100320 msgr2=0x7f5a50100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:07.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 --2- 172.21.15.31:0/2226049441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50100320 0x7f5a50100730 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009990 tx=0x7f5a40009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.207 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 -- 172.21.15.31:0/2226049441 shutdown_connections 2023-07-29T16:43:07.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 --2- 172.21.15.31:0/2226049441 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50101ea0 0x7f5a50108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 --2- 172.21.15.31:0/2226049441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50100320 0x7f5a50100730 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 --2- 172.21.15.31:0/2226049441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 -- 172.21.15.31:0/2226049441 >> 172.21.15.31:0/2226049441 conn(0x7f5a500fb920 msgr2=0x7f5a500fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:07.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 -- 172.21.15.31:0/2226049441 shutdown_connections 2023-07-29T16:43:07.208 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.203+0000 7f5a54dff700 1 -- 172.21.15.31:0/2226049441 wait complete. 2023-07-29T16:43:07.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.204+0000 7f5a54dff700 1 Processor -- start 2023-07-29T16:43:07.210 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.204+0000 7f5a54dff700 1 -- start start 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a54dff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50100320 0x7f5a501982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a54dff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a54dff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50101ea0 0x7f5a5019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a54dff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5019fe50 con 0x7f5a50101ea0 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a54dff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5019ff90 con 0x7f5a50101510 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a54dff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a501a0120 con 0x7f5a50100320 2023-07-29T16:43:07.211 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.31:40430/0 (socket says 172.21.15.31:40430) 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 -- 172.21.15.31:0/1144233814 learned_addr learned my addr 172.21.15.31:0/1144233814 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4ed9d700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50101ea0 0x7f5a5019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4e59c700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50100320 0x7f5a501982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 -- 172.21.15.31:0/1144233814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50100320 msgr2=0x7f5a501982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50100320 0x7f5a501982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.212 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 -- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50101ea0 msgr2=0x7f5a5019f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:07.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50101ea0 0x7f5a5019f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.205+0000 7f5a4dd9b700 1 -- 172.21.15.31:0/1144233814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a400095f0 con 0x7f5a50101510 2023-07-29T16:43:07.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.206+0000 7f5a4dd9b700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50198810 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9a0 tx=0x7f5a4400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:07.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.206+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a44009790 con 0x7f5a50101510 2023-07-29T16:43:07.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.206+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a501a0400 con 0x7f5a50101510 2023-07-29T16:43:07.213 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.206+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a4400cbd0 con 0x7f5a50101510 2023-07-29T16:43:07.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.206+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a44018680 con 0x7f5a50101510 2023-07-29T16:43:07.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.206+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a501a0960 con 0x7f5a50101510 2023-07-29T16:43:07.214 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.207+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a50192a00 con 0x7f5a50101510 2023-07-29T16:43:07.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.209+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f5a4400cd40 con 0x7f5a50101510 2023-07-29T16:43:07.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.210+0000 7f5a3f7fe700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5a380a0810 0x7f5a380a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:07.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.211+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8593+0+0 (secure 0 0 0) 0x7f5a44014070 con 0x7f5a50101510 2023-07-29T16:43:07.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.211+0000 7f5a4e59c700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5a380a0810 0x7f5a380a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:07.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.211+0000 7f5a4e59c700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5a380a0810 0x7f5a380a2cc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000b2e0 tx=0x7f5a40005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:07.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.212+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f5a44095950 con 0x7f5a50101510 2023-07-29T16:43:07.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:06 smithi039 ceph-mon[128858]: pgmap v1201: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2023-07-29T16:43:07.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:06 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1733631249' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-29T16:43:07.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:06 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-29T16:43:07.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:06 smithi039 ceph-mon[128858]: osdmap e94: 12 total, 12 up, 12 in 2023-07-29T16:43:07.280 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: ignoring --setuser ceph since I am not root 2023-07-29T16:43:07.281 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: ignoring --setgroup ceph since I am not root 2023-07-29T16:43:07.281 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mds[154307]: -- 172.21.15.39:0/3906254430 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5638da5bb1e0 con 0x5638da590800 2023-07-29T16:43:07.281 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: starting mds.b at 2023-07-29T16:43:07.281 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: 2023-07-29T16:43:07.100+0000 7f908a59f700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2023-07-29T16:43:07.281 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e[153602]: ignoring --setuser ceph since I am not root 2023-07-29T16:43:07.282 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e[153602]: ignoring --setgroup ceph since I am not root 2023-07-29T16:43:07.282 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mds[153626]: -- 172.21.15.39:0/1135474674 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a58927d1e0 con 0x55a589252800 2023-07-29T16:43:07.282 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e[153602]: starting mds.e at 2023-07-29T16:43:07.282 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h[153943]: ignoring --setuser ceph since I am not root 2023-07-29T16:43:07.282 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h[153943]: ignoring --setgroup ceph since I am not root 2023-07-29T16:43:07.282 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mds[153967]: -- 172.21.15.39:0/3417426904 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c687ef31e0 con 0x55c687ec8c00 2023-07-29T16:43:07.283 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h[153943]: starting mds.h at 2023-07-29T16:43:07.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:06 smithi156 ceph-mon[129245]: pgmap v1201: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2023-07-29T16:43:07.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:06 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1733631249' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-29T16:43:07.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:06 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-29T16:43:07.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:06 smithi156 ceph-mon[129245]: osdmap e94: 12 total, 12 up, 12 in 2023-07-29T16:43:07.327 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f[154846]: ignoring --setuser ceph since I am not root 2023-07-29T16:43:07.327 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f[154846]: ignoring --setgroup ceph since I am not root 2023-07-29T16:43:07.327 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mds[154870]: -- 172.21.15.156:0/1624982761 <== mon.2 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561871cc11e0 con 0x561871c96800 2023-07-29T16:43:07.327 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f[154846]: starting mds.f at 2023-07-29T16:43:07.328 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: ignoring --setuser ceph since I am not root 2023-07-29T16:43:07.328 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: ignoring --setgroup ceph since I am not root 2023-07-29T16:43:07.328 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mds[155548]: -- 172.21.15.156:0/3670857323 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c48ab471e0 con 0x55c48ab1cc00 2023-07-29T16:43:07.328 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: starting mds.i at 2023-07-29T16:43:07.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:43:06.950+0000 7fa43fcae700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-29T16:43:07.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:06 smithi031 ceph-mon[95504]: pgmap v1201: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 7 op/s 2023-07-29T16:43:07.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:06 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1733631249' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-29T16:43:07.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:06 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-29T16:43:07.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:06 smithi031 ceph-mon[95504]: osdmap e94: 12 total, 12 up, 12 in 2023-07-29T16:43:07.439 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g[130372]: ignoring --setuser ceph since I am not root 2023-07-29T16:43:07.440 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g[130372]: ignoring --setgroup ceph since I am not root 2023-07-29T16:43:07.440 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mds[130412]: -- 172.21.15.31:0/1936678055 <== mon.1 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560a404a51e0 con 0x560a4047b000 2023-07-29T16:43:07.440 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g[130372]: starting mds.g at 2023-07-29T16:43:07.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.467+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5a5002cdc0 con 0x7f5a50101510 2023-07-29T16:43:07.973 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.969+0000 7f5a3f7fe700 1 -- 172.21.15.31:0/1144233814 <== mon.2 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v48) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5a44098f70 con 0x7f5a50101510 2023-07-29T16:43:07.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5a380a0810 msgr2=0x7f5a380a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:07.977 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5a380a0810 0x7f5a380a2cc0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000b2e0 tx=0x7f5a40005fb0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 msgr2=0x7f5a50198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50198810 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5a4400e9a0 tx=0x7f5a4400ecb0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 shutdown_connections 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f5a380a0810 0x7f5a380a2cc0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a50101ea0 0x7f5a5019f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a50100320 0x7f5a501982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 --2- 172.21.15.31:0/1144233814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5a50101510 0x7f5a50198810 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:07.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 >> 172.21.15.31:0/1144233814 conn(0x7f5a500fb920 msgr2=0x7f5a50106df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:07.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 shutdown_connections 2023-07-29T16:43:07.979 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:07.973+0000 7f5a54dff700 1 -- 172.21.15.31:0/1144233814 wait complete. 2023-07-29T16:43:08.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-29T16:43:08.215 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.211+0000 7fc13cdff700 1 -- 172.21.15.31:0/401258218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138101510 msgr2=0x7fc138101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.211+0000 7fc13cdff700 1 --2- 172.21.15.31:0/401258218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138101510 0x7fc138101960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009990 tx=0x7fc12c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 -- 172.21.15.31:0/401258218 shutdown_connections 2023-07-29T16:43:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 --2- 172.21.15.31:0/401258218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc138101ea0 0x7fc138108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 --2- 172.21.15.31:0/401258218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc138100320 0x7fc138100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 --2- 172.21.15.31:0/401258218 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138101510 0x7fc138101960 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:08.216 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 -- 172.21.15.31:0/401258218 >> 172.21.15.31:0/401258218 conn(0x7fc1380fb920 msgr2=0x7fc1380fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:08.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 -- 172.21.15.31:0/401258218 shutdown_connections 2023-07-29T16:43:08.217 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.212+0000 7fc13cdff700 1 -- 172.21.15.31:0/401258218 wait complete. 2023-07-29T16:43:08.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 Processor -- start 2023-07-29T16:43:08.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 -- start start 2023-07-29T16:43:08.218 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 0x7fc138074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc138101510 0x7fc138073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc138101ea0 0x7fc138073680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc138073e20 con 0x7fc138100320 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc138073f90 con 0x7fc138101ea0 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13cdff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1381a0100 con 0x7fc138101510 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 0x7fc138074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:08.219 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 0x7fc138074b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43236/0 (socket says 172.21.15.31:43236) 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.213+0000 7fc13659c700 1 -- 172.21.15.31:0/3257346952 learned_addr learned my addr 172.21.15.31:0/3257346952 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc136d9d700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc138101ea0 0x7fc138073680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc135d9b700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc138101510 0x7fc138073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13659c700 1 -- 172.21.15.31:0/3257346952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc138101510 msgr2=0x7fc138073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13659c700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc138101510 0x7fc138073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13659c700 1 -- 172.21.15.31:0/3257346952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc138101ea0 msgr2=0x7fc138073680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:08.220 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13659c700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc138101ea0 0x7fc138073680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13659c700 1 -- 172.21.15.31:0/3257346952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0095f0 con 0x7fc138100320 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13659c700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 0x7fc138074b00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc12800e970 tx=0x7fc12800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128009790 con 0x7fc138100320 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc12800c7e0 con 0x7fc138100320 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1381a03e0 con 0x7fc138100320 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.214+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128003c70 con 0x7fc138100320 2023-07-29T16:43:08.221 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.215+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1381a0940 con 0x7fc138100320 2023-07-29T16:43:08.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.215+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13810bb40 con 0x7fc138100320 2023-07-29T16:43:08.222 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.217+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7fc12800cc50 con 0x7fc138100320 2023-07-29T16:43:08.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.219+0000 7fc1277fe700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc1200a0810 0x7fc1200a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:08.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.219+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8593+0+0 (secure 0 0 0) 0x7fc128014070 con 0x7fc138100320 2023-07-29T16:43:08.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.219+0000 7fc135d9b700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc1200a0810 0x7fc1200a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:08.223 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.220+0000 7fc135d9b700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc1200a0810 0x7fc1200a2cc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b2e0 tx=0x7fc12c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:08.226 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.223+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7fc128095740 con 0x7fc138100320 2023-07-29T16:43:08.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-29T16:43:08.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-29T16:43:08.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2023-07-29T16:43:08.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-29T16:43:08.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-29T16:43:08.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/1144233814' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:43:08.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:07 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:43:08.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-29T16:43:08.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-29T16:43:08.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2023-07-29T16:43:08.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-29T16:43:08.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-29T16:43:08.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/1144233814' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:43:08.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:07 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:43:08.436 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:08.432+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc138107250 con 0x7fc138100320 2023-07-29T16:43:08.437 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-29T16:43:08.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-29T16:43:08.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2023-07-29T16:43:08.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-29T16:43:08.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-29T16:43:08.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/1144233814' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:43:08.439 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:07 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-29T16:43:09.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: pgmap v1203: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 6 op/s 2023-07-29T16:43:09.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-29T16:43:09.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-29T16:43:09.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: Cluster is now healthy 2023-07-29T16:43:09.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6835/328882271,v1:172.21.15.156:6839/328882271] up:boot 2023-07-29T16:43:09.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.156:6834/22755146,v1:172.21.15.156:6838/22755146] up:boot 2023-07-29T16:43:09.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.31:6838/2534614949,v1:172.21.15.31:6839/2534614949] up:boot 2023-07-29T16:43:09.280 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6835/2440539996,v1:172.21.15.39:6838/2440539996] up:boot 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6834/1686225058,v1:172.21.15.39:6837/1686225058] up:boot 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: mds.? [v2:172.21.15.39:6836/238549551,v1:172.21.15.39:6839/238549551] up:boot 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: fsmap 12 up:standby 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-29T16:43:09.281 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-29T16:43:09.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-29T16:43:09.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-29T16:43:09.282 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:08 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3257346952' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:09.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: pgmap v1203: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 6 op/s 2023-07-29T16:43:09.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-29T16:43:09.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-29T16:43:09.326 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: Cluster is now healthy 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6835/328882271,v1:172.21.15.156:6839/328882271] up:boot 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.156:6834/22755146,v1:172.21.15.156:6838/22755146] up:boot 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.31:6838/2534614949,v1:172.21.15.31:6839/2534614949] up:boot 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6835/2440539996,v1:172.21.15.39:6838/2440539996] up:boot 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6834/1686225058,v1:172.21.15.39:6837/1686225058] up:boot 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: mds.? [v2:172.21.15.39:6836/238549551,v1:172.21.15.39:6839/238549551] up:boot 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-29T16:43:09.327 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: fsmap 12 up:standby 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-29T16:43:09.328 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:08 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3257346952' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:09.346 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.342+0000 7fc1277fe700 1 -- 172.21.15.31:0/3257346952 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc128003dd0 con 0x7fc138100320 2023-07-29T16:43:09.350 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc1200a0810 msgr2=0x7fc1200a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc1200a0810 0x7fc1200a2cc0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00b2e0 tx=0x7fc12c005fb0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 msgr2=0x7fc138074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 0x7fc138074b00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc12800e970 tx=0x7fc12800ec80 comp rx=0 tx=0).stop 2023-07-29T16:43:09.351 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 shutdown_connections 2023-07-29T16:43:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7fc1200a0810 0x7fc1200a2cc0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc138101ea0 0x7fc138073680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc138100320 0x7fc138074b00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 --2- 172.21.15.31:0/3257346952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc138101510 0x7fc138073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 >> 172.21.15.31:0/3257346952 conn(0x7fc1380fb920 msgr2=0x7fc1380fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:09.352 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.347+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 shutdown_connections 2023-07-29T16:43:09.353 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.348+0000 7fc13cdff700 1 -- 172.21.15.31:0/3257346952 wait complete. 2023-07-29T16:43:09.354 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2023-07-29T16:43:09.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: pgmap v1203: 97 pgs: 97 active+clean; 452 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 6 op/s 2023-07-29T16:43:09.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-29T16:43:09.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: Cluster is now healthy 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6835/328882271,v1:172.21.15.156:6839/328882271] up:boot 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.156:6834/22755146,v1:172.21.15.156:6838/22755146] up:boot 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.31:6838/2534614949,v1:172.21.15.31:6839/2534614949] up:boot 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6835/2440539996,v1:172.21.15.39:6838/2440539996] up:boot 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6834/1686225058,v1:172.21.15.39:6837/1686225058] up:boot 2023-07-29T16:43:09.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: mds.? [v2:172.21.15.39:6836/238549551,v1:172.21.15.39:6839/238549551] up:boot 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: fsmap 12 up:standby 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-29T16:43:09.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-29T16:43:09.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='mgr.14152 172.21.15.31:0/2510024747' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-29T16:43:09.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:08 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3257346952' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:09.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-29T16:43:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.630+0000 7f0b40bff700 1 -- 172.21.15.31:0/926915551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c100330 msgr2=0x7f0b3c100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:09.634 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.630+0000 7f0b40bff700 1 --2- 172.21.15.31:0/926915551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c100330 0x7f0b3c100740 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009990 tx=0x7f0b2c009ca0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 -- 172.21.15.31:0/926915551 shutdown_connections 2023-07-29T16:43:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 --2- 172.21.15.31:0/926915551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b3c101eb0 0x7f0b3c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 --2- 172.21.15.31:0/926915551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c100330 0x7f0b3c100740 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 --2- 172.21.15.31:0/926915551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c101520 0x7f0b3c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 -- 172.21.15.31:0/926915551 >> 172.21.15.31:0/926915551 conn(0x7f0b3c0fb970 msgr2=0x7f0b3c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:09.635 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 -- 172.21.15.31:0/926915551 shutdown_connections 2023-07-29T16:43:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.631+0000 7f0b40bff700 1 -- 172.21.15.31:0/926915551 wait complete. 2023-07-29T16:43:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.632+0000 7f0b40bff700 1 Processor -- start 2023-07-29T16:43:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.632+0000 7f0b40bff700 1 -- start start 2023-07-29T16:43:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b40bff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 0x7f0b3c1960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b40bff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c101520 0x7f0b3c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b40bff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b3c101eb0 0x7f0b3c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b40bff700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c19dc40 con 0x7f0b3c101eb0 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b40bff700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c19dd80 con 0x7f0b3c101520 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b40bff700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c19df10 con 0x7f0b3c100330 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b3a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 0x7f0b3c1960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:09.638 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b39d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c101520 0x7f0b3c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b3a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 0x7f0b3c1960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.31:59308/0 (socket says 172.21.15.31:59308) 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b3a59c700 1 -- 172.21.15.31:0/10498042 learned_addr learned my addr 172.21.15.31:0/10498042 (peer_addr_for_me v2:172.21.15.31:0/0) 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.633+0000 7f0b3ad9d700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b3c101eb0 0x7f0b3c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b3a59c700 1 -- 172.21.15.31:0/10498042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c101520 msgr2=0x7f0b3c196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b3a59c700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c101520 0x7f0b3c196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b3a59c700 1 -- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b3c101eb0 msgr2=0x7f0b3c19d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b3a59c700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b3c101eb0 0x7f0b3c19d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b3a59c700 1 -- 172.21.15.31:0/10498042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0095f0 con 0x7f0b3c100330 2023-07-29T16:43:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b39d9b700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c101520 0x7f0b3c196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-29T16:43:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b3a59c700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 0x7f0b3c1960c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c004730 tx=0x7f0b2c004760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c01d070 con 0x7f0b3c100330 2023-07-29T16:43:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c19e190 con 0x7f0b3c100330 2023-07-29T16:43:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b2c021410 con 0x7f0b3c100330 2023-07-29T16:43:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c017650 con 0x7f0b3c100330 2023-07-29T16:43:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.634+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c19e5b0 con 0x7f0b3c100330 2023-07-29T16:43:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.635+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c1078f0 con 0x7f0b3c100330 2023-07-29T16:43:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.638+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) v1 ==== 134473+0+0 (secure 0 0 0) 0x7f0b2c021580 con 0x7f0b3c100330 2023-07-29T16:43:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.639+0000 7f0b2b7fe700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b240a0790 0x7f0b240a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-29T16:43:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.639+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8031+0+0 (secure 0 0 0) 0x7f0b2c0cd490 con 0x7f0b3c100330 2023-07-29T16:43:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.639+0000 7f0b39d9b700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b240a0790 0x7f0b240a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-29T16:43:09.643 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.640+0000 7f0b39d9b700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b240a0790 0x7f0b240a2c40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0b30011490 tx=0x7f0b30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-29T16:43:09.645 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.642+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177933 (secure 0 0 0) 0x7f0b2c097040 con 0x7f0b3c100330 2023-07-29T16:43:09.854 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:09.850+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0b3c02d0b0 con 0x7f0b3c100330 2023-07-29T16:43:10.354 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.350+0000 7f0b2b7fe700 1 -- 172.21.15.31:0/10498042 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0b3c02d0b0 con 0x7f0b3c100330 2023-07-29T16:43:10.357 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b240a0790 msgr2=0x7f0b240a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b240a0790 0x7f0b240a2c40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0b30011490 tx=0x7f0b30009260 comp rx=0 tx=0).stop 2023-07-29T16:43:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 msgr2=0x7f0b3c1960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-29T16:43:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 0x7f0b3c1960c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c004730 tx=0x7f0b2c004760 comp rx=0 tx=0).stop 2023-07-29T16:43:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 shutdown_connections 2023-07-29T16:43:10.358 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:6800/4163929785,v1:172.21.15.31:6801/4163929785] conn(0x7f0b240a0790 0x7f0b240a2c40 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0b3c101eb0 0x7f0b3c19d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0b3c100330 0x7f0b3c1960c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 --2- 172.21.15.31:0/10498042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b3c101520 0x7f0b3c196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-29T16:43:10.359 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.354+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 >> 172.21.15.31:0/10498042 conn(0x7f0b3c0fb970 msgr2=0x7f0b3c104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-29T16:43:10.361 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.357+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 shutdown_connections 2023-07-29T16:43:10.362 INFO:teuthology.orchestra.run.smithi031.stderr:2023-07-29T16:43:10.358+0000 7f0b40bff700 1 -- 172.21.15.31:0/10498042 wait complete. 2023-07-29T16:43:10.366 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2023-07-29T16:43:10.395 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:43:10.573 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:43:10.602 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:10 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/3257346952' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-29T16:43:10.602 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:10 smithi039 ceph-mon[128858]: osdmap e95: 12 total, 12 up, 12 in 2023-07-29T16:43:10.602 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:10 smithi039 ceph-mon[128858]: from='client.? 172.21.15.31:0/10498042' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:10.603 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:10 smithi039 ceph-mon[128858]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:10.603 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:10 smithi039 ceph-mon[128858]: pgmap v1205: 65 pgs: 65 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:43:10.618 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:43:10.645 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:10 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/3257346952' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-29T16:43:10.645 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:10 smithi156 ceph-mon[129245]: osdmap e95: 12 total, 12 up, 12 in 2023-07-29T16:43:10.645 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:10 smithi156 ceph-mon[129245]: from='client.? 172.21.15.31:0/10498042' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:10.645 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:10 smithi156 ceph-mon[129245]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:10.645 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:10 smithi156 ceph-mon[129245]: pgmap v1205: 65 pgs: 65 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:43:10.659 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-29T16:43:10.660 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-29T16:43:10.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/3257346952' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-29T16:43:10.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:10 smithi031 ceph-mon[95504]: osdmap e95: 12 total, 12 up, 12 in 2023-07-29T16:43:10.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:10 smithi031 ceph-mon[95504]: from='client.? 172.21.15.31:0/10498042' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:10.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:10 smithi031 ceph-mon[95504]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-29T16:43:10.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:10 smithi031 ceph-mon[95504]: pgmap v1205: 65 pgs: 65 active+clean; 577 KiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2023-07-29T16:43:10.703 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-29T16:43:10.730 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-29T16:43:10.746 INFO:tasks.cephadm:Stopping all daemons... 2023-07-29T16:43:10.747 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-29T16:43:10.747 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a 2023-07-29T16:43:11.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:10 smithi031 systemd[1]: Stopping Ceph mon.a for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:11.687 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:11 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:43:11.344+0000 7fa442cb4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:11.688 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:11 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a[95480]: 2023-07-29T16:43:11.344+0000 7fa442cb4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-29T16:43:12.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Jul 29 16:43:11 smithi031 bash[172997]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-a 2023-07-29T16:43:12.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.a.service' 2023-07-29T16:43:12.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:12.574 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-29T16:43:12.574 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-29T16:43:12.574 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.b 2023-07-29T16:43:13.014 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:12 smithi039 systemd[1]: Stopping Ceph mon.b for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:13.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-b[128834]: 2023-07-29T16:43:13.010+0000 7f432573a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:13.279 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-b[128834]: 2023-07-29T16:43:13.011+0000 7f432573a700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-29T16:43:13.717 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.b.service' 2023-07-29T16:43:13.744 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:13 smithi039 bash[164204]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-b 2023-07-29T16:43:13.745 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:13 smithi039 systemd[1]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.b.service: Succeeded. 2023-07-29T16:43:13.745 INFO:journalctl@ceph.mon.b.smithi039.stdout:Jul 29 16:43:13 smithi039 systemd[1]: Stopped Ceph mon.b for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:43:13.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:13.769 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-29T16:43:13.770 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-29T16:43:13.770 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.c 2023-07-29T16:43:14.076 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:13 smithi156 systemd[1]: Stopping Ceph mon.c for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:14.462 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-c[129207]: 2023-07-29T16:43:14.179+0000 7f0984794700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:14.463 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-c[129207]: 2023-07-29T16:43:14.179+0000 7f0984794700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-29T16:43:14.739 INFO:journalctl@ceph.mon.c.smithi156.stdout:Jul 29 16:43:14 smithi156 bash[166125]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mon-c 2023-07-29T16:43:14.926 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mon.c.service' 2023-07-29T16:43:14.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:14.982 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-29T16:43:14.982 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-29T16:43:14.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x 2023-07-29T16:43:15.437 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:43:15 smithi031 systemd[1]: Stopping Ceph mgr.x for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:15.829 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:43:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x[95750]: 2023-07-29T16:43:15.485+0000 7f349bfc0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:16.187 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Jul 29 16:43:15 smithi031 bash[173152]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-x 2023-07-29T16:43:16.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.x.service' 2023-07-29T16:43:16.401 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:16.402 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-29T16:43:16.402 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-29T16:43:16.403 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.y 2023-07-29T16:43:16.779 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:43:16 smithi039 systemd[1]: Stopping Ceph mgr.y for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:17.118 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:43:17 smithi039 bash[164359]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-y 2023-07-29T16:43:17.305 INFO:journalctl@ceph.mgr.y.smithi039.stdout:Jul 29 16:43:17 smithi039 systemd[1]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-29T16:43:17.309 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.y.service' 2023-07-29T16:43:17.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:17.346 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-29T16:43:17.346 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-29T16:43:17.347 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.z 2023-07-29T16:43:17.770 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:43:17 smithi156 systemd[1]: Stopping Ceph mgr.z for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:18.056 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:43:18 smithi156 bash[166273]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mgr-z 2023-07-29T16:43:18.236 INFO:journalctl@ceph.mgr.z.smithi156.stdout:Jul 29 16:43:18 smithi156 systemd[1]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-29T16:43:18.240 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mgr.z.service' 2023-07-29T16:43:18.276 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:18.277 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-29T16:43:18.277 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-29T16:43:18.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.0 2023-07-29T16:43:18.687 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:43:18 smithi031 systemd[1]: Stopping Ceph osd.0 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:19.187 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:43:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-0[108638]: 2023-07-29T16:43:18.785+0000 7f3a12c8b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:19.188 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:43:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-0[108638]: 2023-07-29T16:43:18.785+0000 7f3a12c8b700 -1 osd.0 96 *** Got signal Terminated *** 2023-07-29T16:43:19.188 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:43:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-0[108638]: 2023-07-29T16:43:18.785+0000 7f3a12c8b700 -1 osd.0 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:43:24.686 INFO:journalctl@ceph.osd.0.smithi031.stdout:Jul 29 16:43:24 smithi031 bash[173300]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-0 2023-07-29T16:43:26.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.0.service' 2023-07-29T16:43:26.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:26.773 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-29T16:43:26.773 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-29T16:43:26.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.1 2023-07-29T16:43:27.144 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:43:26 smithi039 systemd[1]: Stopping Ceph osd.1 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:27.529 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:43:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1[133963]: 2023-07-29T16:43:27.139+0000 7fd2afb76700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:27.529 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:43:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1[133963]: 2023-07-29T16:43:27.139+0000 7fd2afb76700 -1 osd.1 96 *** Got signal Terminated *** 2023-07-29T16:43:27.529 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:43:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1[133963]: 2023-07-29T16:43:27.139+0000 7fd2afb76700 -1 osd.1 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:43:33.029 INFO:journalctl@ceph.osd.1.smithi039.stdout:Jul 29 16:43:32 smithi039 bash[164511]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-1 2023-07-29T16:43:34.398 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.1.service' 2023-07-29T16:43:34.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:34.441 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-29T16:43:34.441 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-29T16:43:34.441 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.2 2023-07-29T16:43:34.797 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:43:34 smithi156 systemd[1]: Stopping Ceph osd.2 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:35.076 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:43:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2[135103]: 2023-07-29T16:43:34.792+0000 7feb8bb0c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:35.076 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:43:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2[135103]: 2023-07-29T16:43:34.792+0000 7feb8bb0c700 -1 osd.2 96 *** Got signal Terminated *** 2023-07-29T16:43:35.076 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:43:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2[135103]: 2023-07-29T16:43:34.792+0000 7feb8bb0c700 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:43:40.576 INFO:journalctl@ceph.osd.2.smithi156.stdout:Jul 29 16:43:40 smithi156 bash[166427]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-2 2023-07-29T16:43:41.840 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.2.service' 2023-07-29T16:43:41.880 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:41.880 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-29T16:43:41.881 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-29T16:43:41.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.3 2023-07-29T16:43:42.401 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:43:41 smithi031 systemd[1]: Stopping Ceph osd.3 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:42.687 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:43:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3[114237]: 2023-07-29T16:43:42.398+0000 7f33bc30c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:42.688 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:43:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3[114237]: 2023-07-29T16:43:42.398+0000 7f33bc30c700 -1 osd.3 96 *** Got signal Terminated *** 2023-07-29T16:43:42.688 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:43:42 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3[114237]: 2023-07-29T16:43:42.398+0000 7f33bc30c700 -1 osd.3 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:43:45.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:44 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:44.801+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:46.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:45 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:45.846+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:46.326 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:45.989+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:47.188 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:46 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:46.705+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:47.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:46 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:46.865+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:47.326 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:46 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:46.941+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:47.779 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:47 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:47.341+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:47.938 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:47 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:47.661+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:48.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:47 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:47.891+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:48.326 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:48.165+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:48.326 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:47.916+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:48.624 INFO:journalctl@ceph.osd.3.smithi031.stdout:Jul 29 16:43:48 smithi031 bash[173571]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-3 2023-07-29T16:43:48.779 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:48 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:48.339+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:48.937 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:48 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:48.621+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:49.138 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:48.873+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:49.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:48 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:48.886+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:49.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:49 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:49.134+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:49.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:49 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:49.583+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:49.779 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:49 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:49.304+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:50.182 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:49 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:49.847+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:50.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:49 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:49.888+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:50.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.3.service' 2023-07-29T16:43:50.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:50.406 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-29T16:43:50.406 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-29T16:43:50.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.4 2023-07-29T16:43:50.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:50.179+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:50.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:50.206+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:50.598 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:50 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:50.505+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:50.773 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:50 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:50.348+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:50.774 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:50 smithi039 systemd[1]: Stopping Ceph osd.4 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:50.938 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:50 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:50.596+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:51.029 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:50 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:50.887+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:51.030 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:50 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:50.770+0000 7f11f86d4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:51.030 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:50 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:50.770+0000 7f11f86d4700 -1 osd.4 96 *** Got signal Terminated *** 2023-07-29T16:43:51.030 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:50 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:50.770+0000 7f11f86d4700 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:43:51.158 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:50.886+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:51.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:51 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:51.154+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:51.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:51 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:51.186+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:51.779 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:51 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:51.386+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:51.937 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:51 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:51.566+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:51.938 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:51 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:51.458+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:52.172 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:51 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:51.847+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:52.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:51 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:51.893+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:52.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:52 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:52.169+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:52.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:52 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:52.173+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:52.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:52 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:52.533+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:52.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:52 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:52.421+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:52.779 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:52 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:52.424+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:53.167 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:52 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:52.895+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:53.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:52 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:52.896+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:53.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:53.173+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:53.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:53.164+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:53.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:53.164+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:53.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:53 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:53.503+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:53.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:53 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:53.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:53.779 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:53 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:53.396+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:54.212 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:53 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:53.941+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:54.212 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:54.168+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:54.212 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:54.168+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:54.247 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:53 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:53.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:54.529 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:54 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:54.369+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:54.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:43:54 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:43:54.245+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:43:54.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:54.209+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:54.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:54.489+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:54.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:54 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:54.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:55.241 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:54 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:54.960+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:55.241 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:55.171+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:55.241 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:55.171+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:55.270 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:54 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:54.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:55.271 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:54 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:54.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:43:55.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:43:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:43:55.267+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:43:55.529 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4[139424]: 2023-07-29T16:43:55.416+0000 7f11f3cc8700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.817754+0000 front 2023-07-29T16:43:21.817441+0000 (oldest deadline 2023-07-29T16:43:46.517618+0000) 2023-07-29T16:43:55.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:55.238+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:55.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:55.238+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:43:55.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:55.331+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:55.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:55.331+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:43:55.689 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:55.445+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:55.689 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:55 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:55.445+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:43:56.250 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:55.978+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:56.251 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:55 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:55.978+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:43:56.270 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:55 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:55.946+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:56.270 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:56.161+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:56.270 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:56.161+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:56.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:43:56 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:43:56.247+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:43:56.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:56.267+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:56.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:56.267+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:43:56.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:56.446+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:56.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:56.446+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:43:56.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:56.355+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:56.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:56.355+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:43:57.008 INFO:journalctl@ceph.osd.4.smithi039.stdout:Jul 29 16:43:56 smithi039 bash[164780]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-4 2023-07-29T16:43:57.185 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:56 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:56.923+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:57.279 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:43:57 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:43:57.246+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:43:57.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:57 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:57.005+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:57.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:57 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:57.005+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:43:57.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:57.182+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:57.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:57.182+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:57.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:57.233+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:57.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:57.233+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:43:57.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:57 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:57.496+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:57.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:57 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:57.496+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:43:57.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:57 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:57.340+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:57.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:57 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:57.340+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:43:57.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.4.service' 2023-07-29T16:43:57.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:43:57.866 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-29T16:43:57.867 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-29T16:43:57.867 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.5 2023-07-29T16:43:58.150 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:57 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:57.896+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:58.151 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:57 smithi156 systemd[1]: Stopping Ceph osd.5 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:43:58.151 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:58.147+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:58.209 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:58 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:58.021+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:58.209 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:58 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:58.021+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:43:58.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:43:58 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:43:58.206+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:43:58.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:58.184+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:58.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:58.184+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:43:58.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:58.147+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:58.577 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:58.222+0000 7fb503668700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:43:58.577 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:58.222+0000 7fb503668700 -1 osd.5 96 *** Got signal Terminated *** 2023-07-29T16:43:58.577 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:58.222+0000 7fb503668700 -1 osd.5 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:43:58.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:58.504+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:58.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:58.504+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:43:58.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:58.361+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:58.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:58 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:58.361+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:43:59.139 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:58 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:58.884+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:43:59.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:43:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:43:59.253+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:43:59.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:59.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:43:59.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:43:59 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:43:59.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:43:59.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:59 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:59.136+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:43:59.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:43:59 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:43:59.136+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:43:59.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:59 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:59.199+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:43:59.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:43:59 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:43:59.199+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:43:59.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:59.465+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:43:59.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:43:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:43:59.465+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:43:59.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:59.370+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:43:59.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:43:59 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:43:59.370+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:00.146 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:43:59 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:43:59.870+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:00.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:00 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:00.232+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:00.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:00 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:00.090+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:00.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:00 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:00.090+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:00.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:00 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:00.143+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:44:00.577 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:00 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:00.143+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:44:00.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:00 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:00.191+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:00.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:00 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:00.191+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:00.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:00.491+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:00.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:00.491+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:00.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:00.414+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:00.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:00.414+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:01.137 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:00 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:00.839+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:01.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:01.269+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:01.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:01.126+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:01.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:01.126+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:01.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:01 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:01.126+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:01.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:01.134+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:44:01.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:01.134+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:44:01.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:01.224+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:01.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:01.224+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:01.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:01.224+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:01.938 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:01.444+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:01.939 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:01.444+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:01.939 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:01.449+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:01.939 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:01.449+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:01.939 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:01 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:01.449+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:02.133 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:01 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:01.835+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:02.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:02 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:02.306+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:02.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:02 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:02.306+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:02.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:02 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:02.156+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:02.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:02 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:02.156+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:02.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:02 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:02.156+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:02.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:02.130+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:44:02.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:02.130+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:44:02.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:02.130+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.846120+0000 front 2023-07-29T16:43:37.846148+0000 (oldest deadline 2023-07-29T16:44:01.346293+0000) 2023-07-29T16:44:02.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:02.176+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:02.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:02.176+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:02.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:02.176+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:02.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:02 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:02.411+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:02.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:02 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:02.411+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:02.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:02 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:02.408+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:02.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:02 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:02.408+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:02.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:02 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:02.408+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:03.121 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:02 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:02.829+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:03.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:03 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:03.342+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:03.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:03 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:03.342+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:03.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:03 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:03.198+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:03.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:03 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:03.198+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:03.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:03 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:03.198+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:03.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:03.119+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.344388+0000 front 2023-07-29T16:43:21.344405+0000 (oldest deadline 2023-07-29T16:43:47.244317+0000) 2023-07-29T16:44:03.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:03.119+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:27.245069+0000 front 2023-07-29T16:43:27.245049+0000 (oldest deadline 2023-07-29T16:43:52.545042+0000) 2023-07-29T16:44:03.576 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5[140562]: 2023-07-29T16:44:03.119+0000 7fb50045f700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.846120+0000 front 2023-07-29T16:43:37.846148+0000 (oldest deadline 2023-07-29T16:44:01.346293+0000) 2023-07-29T16:44:03.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:03.224+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:03.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:03.224+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:03.577 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:03.224+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:03.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:03 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:03.434+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:03.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:03 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:03.434+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:03.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:03 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:03.377+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:03.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:03 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:03.377+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:03.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:03 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:03.377+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:04.076 INFO:journalctl@ceph.osd.5.smithi156.stdout:Jul 29 16:44:04 smithi156 bash[166687]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-5 2023-07-29T16:44:04.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:03.810+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:04.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:03 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:03.810+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:04.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:04 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:04.362+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:04.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:04 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:04.362+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:04.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:04 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:04.191+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:04.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:04 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:04.191+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:04.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:04 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:04.191+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:04.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:04 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:04.187+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:04.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:04 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:04.187+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:04.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:04 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:04.187+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:04.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:04.415+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:04.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:04.415+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:04.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:04.334+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:04.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:04.334+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:04.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:04.334+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:05.044 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:04 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:04.810+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:05.044 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:04 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:04.810+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:05.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:05.149+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:05.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:05.149+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:05.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:05.149+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:05.363 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.5.service' 2023-07-29T16:44:05.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:05.404 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-29T16:44:05.404 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-29T16:44:05.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.6 2023-07-29T16:44:05.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:05.345+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:05.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:05.345+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:05.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:05.187+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:05.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:05.187+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:05.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:05 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:05.187+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:05.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:05.383+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:05.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:05.383+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:05.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:05.383+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:05.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:05.460+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:05.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:05.460+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:05.689 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:05.460+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:39.210821+0000 front 2023-07-29T16:43:39.210896+0000 (oldest deadline 2023-07-29T16:44:04.510746+0000) 2023-07-29T16:44:05.689 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 systemd[1]: Stopping Ceph osd.6 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:06.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:05.793+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:06.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:05 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:05.793+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:06.187 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:05.918+0000 7fae13118700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:06.188 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:05.918+0000 7fae13118700 -1 osd.6 96 *** Got signal Terminated *** 2023-07-29T16:44:06.188 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:05 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:05.918+0000 7fae13118700 -1 osd.6 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:44:06.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:06 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:06.374+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:06.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:06 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:06.374+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:06.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:06 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:06.209+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:06.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:06 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:06.209+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:06.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:06 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:06.209+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:06.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:06.109+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:06.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:06.109+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:06.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:06.109+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:06.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:06.510+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:06.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:06.510+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:06.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:06.510+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:39.210821+0000 front 2023-07-29T16:43:39.210896+0000 (oldest deadline 2023-07-29T16:44:04.510746+0000) 2023-07-29T16:44:06.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:06.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:06.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:06.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:06.690 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:06 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:06.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:07.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:06.762+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:07.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:06 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:06.762+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:07.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:07.339+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:07.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:07.339+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:07.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:07.211+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:07.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:07.211+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:07.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:07 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:07.211+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:07.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:07.153+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:07.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:07.153+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:07.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:07.153+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:07.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:07.470+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:07.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:07.470+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:07.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:07.470+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:39.210821+0000 front 2023-07-29T16:43:39.210896+0000 (oldest deadline 2023-07-29T16:44:04.510746+0000) 2023-07-29T16:44:07.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:07.327+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:07.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:07.327+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:07.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:07 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:07.327+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:08.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:07.728+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:08.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:07 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:07.728+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:08.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:08.304+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:08.530 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:08.304+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:08.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:08.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:08.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:08.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:08.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:08.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:08.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:08 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:08.107+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:08.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:08 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:08.107+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:08.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:08 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:08.107+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:08.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:08 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:08.493+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:08.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:08 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:08.493+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:08.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:08 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:08.493+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:39.210821+0000 front 2023-07-29T16:43:39.210896+0000 (oldest deadline 2023-07-29T16:44:04.510746+0000) 2023-07-29T16:44:08.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:08 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:08.297+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:08.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:08 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:08.297+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:08.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:08 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:08.297+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:09.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:08 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:08.739+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:09.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:08 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:08.739+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:09.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:09 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:09.323+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:09.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:09 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:09.323+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:09.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:09 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:09.158+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:09.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:09 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:09.158+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:09.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:09 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:09.158+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:09.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:09 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:09.158+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:09.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:09 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:09.091+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:09.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:09 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:09.091+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:09.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:09 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:09.091+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:09.687 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:09.499+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:09.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:09.499+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:09.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:09.499+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:39.210821+0000 front 2023-07-29T16:43:39.210896+0000 (oldest deadline 2023-07-29T16:44:04.510746+0000) 2023-07-29T16:44:09.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:09.499+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.012096+0000 front 2023-07-29T16:43:45.012056+0000 (oldest deadline 2023-07-29T16:44:09.112182+0000) 2023-07-29T16:44:09.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:09.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:09.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:09.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:09.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:09 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:09.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:10.051 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:09 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:09.690+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:10.051 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:09 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:09.690+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:10.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:10 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:10.048+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:10.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:10 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:10.048+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:10.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:10 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:10.048+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:10.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:10 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:10.317+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:10.530 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:10 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:10.317+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:10.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:10 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:10.185+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:10.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:10 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:10.185+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:10.531 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:10 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:10.185+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:10.531 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:10 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:10.185+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:10.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:10.278+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:10.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:10.278+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:10.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:10.278+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:10.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:10.464+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:21.607861+0000 front 2023-07-29T16:43:21.607605+0000 (oldest deadline 2023-07-29T16:43:46.307566+0000) 2023-07-29T16:44:10.688 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:10.464+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:26.308388+0000 front 2023-07-29T16:43:26.308229+0000 (oldest deadline 2023-07-29T16:43:54.509407+0000) 2023-07-29T16:44:10.689 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:10.464+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:39.210821+0000 front 2023-07-29T16:43:39.210896+0000 (oldest deadline 2023-07-29T16:44:04.510746+0000) 2023-07-29T16:44:10.689 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:10 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6[119793]: 2023-07-29T16:44:10.464+0000 7fae0f70e700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.012096+0000 front 2023-07-29T16:43:45.012056+0000 (oldest deadline 2023-07-29T16:44:09.112182+0000) 2023-07-29T16:44:11.003 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:10 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:10.649+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:11.004 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:10 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:10.649+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:11.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:11 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:11.001+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:11.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:11 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:11.001+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:11.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:11 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:11.001+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:11.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:11.308+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:11.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:11.308+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:11.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:11.308+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:11.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:11.175+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:11.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:11.175+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:11.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:11.175+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:11.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:11 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:11.175+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:11.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:11 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:11.281+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:11.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:11 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:11.281+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:11.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:11 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:11.281+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:11.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:11 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:11.281+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:12.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:11 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:11.678+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:12.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:11 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:11.678+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:12.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:11 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:11.678+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:12.187 INFO:journalctl@ceph.osd.6.smithi031.stdout:Jul 29 16:44:11 smithi031 bash[173848]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-6 2023-07-29T16:44:12.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:12.008+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:12.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:12.008+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:12.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:12.008+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:12.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:12.339+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:12.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:12.339+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:12.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:12.339+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:12.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:12.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:12.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:12.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:12.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:12.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:12.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:12.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:12.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:12 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:12.285+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:12.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:12 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:12.285+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:12.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:12 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:12.285+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:12.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:12 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:12.285+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:13.001 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:12.640+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:13.001 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:12.640+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:13.001 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:12.640+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:13.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:12.998+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:13.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:12.998+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:13.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:12 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:12.998+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:13.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:13.296+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:13.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:13.296+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:13.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:13.296+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:13.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:13.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:13.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:13.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:13.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:13.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:13.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:13 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:13.166+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:13.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:13 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:13.322+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:13.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:13 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:13.322+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:13.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:13 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:13.322+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:13.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:13 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:13.322+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:13.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.6.service' 2023-07-29T16:44:13.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:13.927 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-29T16:44:13.927 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-29T16:44:13.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.7 2023-07-29T16:44:14.027 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:13 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:13.637+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:14.028 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:13 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:13.637+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:14.028 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:13 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:13.637+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:14.279 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:13 smithi039 systemd[1]: Stopping Ceph osd.7 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:14.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:14.182+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:14.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:14.182+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:14.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:14.182+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:14.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:14.182+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:14.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:14.025+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:14.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:14.025+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:14.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:14.025+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:14.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:14.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:14.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:14.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:14.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:14.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:14.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:14 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:14.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:14.779 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:14.291+0000 7f4cdbfff700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:14.780 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:14.291+0000 7f4cdbfff700 -1 osd.7 96 *** Got signal Terminated *** 2023-07-29T16:44:14.780 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:14.291+0000 7f4cdbfff700 -1 osd.7 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:44:14.780 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:14.294+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:14.780 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:14.294+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:14.780 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:14 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:14.294+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:15.014 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:14.648+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:15.014 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:14.648+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:15.014 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:14 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:14.648+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:15.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:15 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:15.011+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:15.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:15 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:15.011+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:15.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:15 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:15.011+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:15.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:15.313+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:15.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:15.313+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:15.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:15.313+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:15.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:15.168+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:15.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:15.168+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:15.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:15.168+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:15.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:15 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:15.168+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:15.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:15.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:15.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:15.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:15.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:15.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:15.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:15 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:15.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:16.045 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:15 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:15.634+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:16.046 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:15 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:15.634+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:16.046 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:15 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:15.634+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:16.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:16.043+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:16.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:16.043+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:16.328 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:16.043+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:16.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:16.306+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:16.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:16.306+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:16.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:16.306+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:16.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:16.146+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:16.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:16.146+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:16.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:16.146+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:16.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:16.146+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:16.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:16 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:16.222+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:16.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:16 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:16.222+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:16.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:16 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:16.222+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:16.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:16 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:16.222+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:16.996 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:16.660+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:16.997 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:16.660+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:16.997 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:16.660+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:16.997 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:16.660+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:17.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:16.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:17.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:16.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:17.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:16 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:16.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:17.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:17.333+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:17.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:17.333+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:17.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:17.333+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:17.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:17.186+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:17.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:17.186+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:17.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:17.186+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:17.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:17 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:17.186+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:17.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:17 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:17.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:17.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:17 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:17.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:17.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:17 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:17.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:17.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:17 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:17.257+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:18.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:17 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:17.684+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:18.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:17 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:17.684+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:18.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:17 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:17.684+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:18.011 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:17 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:17.684+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:18.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:18.007+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:18.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:18.007+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:18.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:18.007+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:18.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:18.331+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:30.570820+0000 front 2023-07-29T16:43:30.570775+0000 (oldest deadline 2023-07-29T16:43:53.470935+0000) 2023-07-29T16:44:18.529 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:18.331+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.172137+0000 front 2023-07-29T16:43:38.172187+0000 (oldest deadline 2023-07-29T16:44:02.272350+0000) 2023-07-29T16:44:18.530 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7[144912]: 2023-07-29T16:44:18.331+0000 7f4cd75f3700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:42.773748+0000 front 2023-07-29T16:43:42.773611+0000 (oldest deadline 2023-07-29T16:44:10.374381+0000) 2023-07-29T16:44:18.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:18.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:18.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:18.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:18.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:18.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:18.531 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:18 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:18.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:18.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:18.258+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:18.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:18.258+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:18.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:18.258+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:18.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:18 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:18.258+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:19.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:18.978+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:19.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:18.978+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:19.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:18.978+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:19.077 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:18.731+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:19.077 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:18.731+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:19.077 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:18.731+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:19.077 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:18 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:18.731+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:19.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:19 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:19.213+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:19.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:19 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:19.213+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:19.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:19 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:19.213+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:19.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:19 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:19.213+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:19.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:19 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:19.213+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:19.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:19.287+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:19.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:19.287+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:19.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:19.287+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:19.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:19.287+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:19.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:19 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:19.287+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:20.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:19.742+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:20.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:19.742+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:20.076 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:19.742+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:20.077 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:19.742+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:20.077 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:19.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:20.077 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:19.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:20.077 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:19.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:20.077 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:19 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:19.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:20.273 INFO:journalctl@ceph.osd.7.smithi039.stdout:Jul 29 16:44:20 smithi039 bash[165046]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-7 2023-07-29T16:44:20.274 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:20.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:20.274 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:20.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:20.274 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:20.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:20.274 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:20.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:20.275 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:20.188+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:20.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:20.296+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:20.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:20.296+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:20.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:20.296+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:20.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:20.296+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:20.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:20 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:20.296+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:20.951 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:20.696+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:20.951 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:20.696+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:20.951 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:20.696+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:20.951 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:20.696+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:21.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:21 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:21.138+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:21.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:21 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:21.138+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:21.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:21 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:21.138+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:21.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:21 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:21.138+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:21.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:21 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:21.138+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:21.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:20.947+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:21.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:20.947+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:21.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:20.947+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:21.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:20 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:20.947+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:21.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:21.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:21.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:21.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:21.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:21.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:21.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:21.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:21.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:21 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:21.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:21.742 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.7.service' 2023-07-29T16:44:21.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:21.785 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-29T16:44:21.786 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-29T16:44:21.786 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.8 2023-07-29T16:44:21.953 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:21.690+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:21.954 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:21.690+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:21.954 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:21.690+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:21.954 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:21.690+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:21.954 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:21 smithi156 systemd[1]: Stopping Ceph osd.8 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:22.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:21.949+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:22.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:21.949+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:22.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:21.949+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:22.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:21 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:21.949+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:22.327 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.134+0000 7fe87ef60700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:22.327 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.134+0000 7fe87ef60700 -1 osd.8 96 *** Got signal Terminated *** 2023-07-29T16:44:22.327 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.134+0000 7fe87ef60700 -1 osd.8 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:44:22.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:22 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:22.121+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:22.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:22 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:22.121+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:22.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:22 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:22.121+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:22.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:22 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:22.121+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:22.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:22 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:22.121+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:22.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:22 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:22.293+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:22.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:22 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:22.293+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:22.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:22 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:22.293+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:22.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:22 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:22.293+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:22.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:22 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:22.293+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:22.997 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.679+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:22.997 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.679+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:22.998 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.679+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:22.998 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:22.679+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:23.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:22.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:23.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:22.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:23.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:22.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:23.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:22 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:22.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:23.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:23 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:23.071+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:23.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:23 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:23.071+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:23.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:23 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:23.071+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:23.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:23 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:23.071+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:23.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:23 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:23.071+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:23.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:23 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:23.290+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:23.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:23 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:23.290+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:23.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:23 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:23.290+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:23.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:23 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:23.290+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:23.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:23 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:23.290+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:24.029 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:23 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:23.689+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:24.030 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:23 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:23.689+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:24.030 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:23 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:23.689+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:24.030 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:23 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:23.689+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:24.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:24.026+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:24.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:24.026+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:24.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:24.026+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:24.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:24.026+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:24.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:24 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:24.043+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:24.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:24 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:24.043+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:24.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:24 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:24.043+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:24.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:24 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:24.043+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:24.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:24 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:24.043+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:24.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:24 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:24.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:24.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:24 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:24.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:24.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:24 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:24.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:24.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:24 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:24.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:24.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:24 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:24.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:25.038 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:24.725+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:25.038 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:24.725+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:25.038 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:24.725+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:25.038 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:24.725+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:25.038 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:24.725+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.555900+0000 front 2023-07-29T16:43:58.555918+0000 (oldest deadline 2023-07-29T16:44:23.856153+0000) 2023-07-29T16:44:25.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:25 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:25.022+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:25.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:25 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:25.022+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:25.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:25 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:25.022+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:25.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:25 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:25.022+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:25.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:25 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:25.022+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:25.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:25 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:25.022+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:25.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:25.034+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:25.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:25.034+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:25.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:25.034+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:25.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:25.034+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:25.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:25 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:25.255+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:25.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:25 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:25.255+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:25.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:25 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:25.255+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:25.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:25 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:25.255+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:25.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:25 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:25.255+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:25.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:25 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:25.255+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:26.040 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:25.733+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:26.040 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:25.733+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:26.041 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:25.733+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:26.041 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:25.733+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:26.041 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:25 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:25.733+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.555900+0000 front 2023-07-29T16:43:58.555918+0000 (oldest deadline 2023-07-29T16:44:23.856153+0000) 2023-07-29T16:44:26.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:26.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:26.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:26.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:26.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:26.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:26.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:26.037+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:26.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:26.037+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:26.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:26.037+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:26.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:26.037+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:26.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:26.037+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:26.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:26 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:26.217+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:26.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:26 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:26.217+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:26.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:26 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:26.217+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:26.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:26 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:26.217+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:26.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:26 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:26.217+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:26.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:26 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:26.217+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:27.043 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:26.758+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.048706+0000 front 2023-07-29T16:43:20.048922+0000 (oldest deadline 2023-07-29T16:43:45.948773+0000) 2023-07-29T16:44:27.043 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:26.758+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:37.551892+0000 front 2023-07-29T16:43:37.551916+0000 (oldest deadline 2023-07-29T16:44:03.452093+0000) 2023-07-29T16:44:27.043 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:26.758+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:45.753800+0000 front 2023-07-29T16:43:45.753846+0000 (oldest deadline 2023-07-29T16:44:11.053814+0000) 2023-07-29T16:44:27.043 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:26.758+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:51.555057+0000 front 2023-07-29T16:43:51.555027+0000 (oldest deadline 2023-07-29T16:44:16.255096+0000) 2023-07-29T16:44:27.043 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:26 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8[146024]: 2023-07-29T16:44:26.758+0000 7fe87bd57700 -1 osd.8 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.555900+0000 front 2023-07-29T16:43:58.555918+0000 (oldest deadline 2023-07-29T16:44:23.856153+0000) 2023-07-29T16:44:27.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:27.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:27.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:27.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:27.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:27.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:26 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:26.977+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:27.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.039+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:27.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.039+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:27.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.039+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:27.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.039+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:27.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.039+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:27.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:27.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:27.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:27.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:27.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:27.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:27.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:27.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:27.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:27.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:27.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:27 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:27.264+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:28.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:27.950+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:28.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:27.950+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:28.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:27.950+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:28.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:27.950+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:28.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:27.950+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:28.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:27 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:27.950+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:28.326 INFO:journalctl@ceph.osd.8.smithi156.stdout:Jul 29 16:44:27 smithi156 bash[166946]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-8 2023-07-29T16:44:28.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:28.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:28.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:28.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:28.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:27 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:27.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:28.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:28.239+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:28.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:28.239+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:28.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:28.239+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:28.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:28.239+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:28.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:28.239+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:28.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:28 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:28.239+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:29.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:28 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:28.977+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:29.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:28 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:28.977+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:29.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:28 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:28.977+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:29.076 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:28 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:28.977+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:29.077 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:28 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:28.977+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:29.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:28 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:28.988+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:29.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:28 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:28.988+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:29.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:28 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:28.988+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:29.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:28 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:28.988+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:29.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:28 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:28.988+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:29.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:28 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:28.988+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:29.455 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.8.service' 2023-07-29T16:44:29.495 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:29.495 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-29T16:44:29.495 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-29T16:44:29.496 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.9 2023-07-29T16:44:29.563 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:29.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:29.563 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:29.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:29.564 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:29.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:29.564 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:29.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:29.564 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:29.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:29.564 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:29.272+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:29.938 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:29 smithi031 systemd[1]: Stopping Ceph osd.9 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:30.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:29 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:29.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:30.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:29 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:29.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:30.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:29 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:29.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:30.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:29 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:29.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:30.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:29 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:29.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:30.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:29 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:29.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:30.315 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.007+0000 7feaf7e70700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:30.315 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.007+0000 7feaf7e70700 -1 osd.9 96 *** Got signal Terminated *** 2023-07-29T16:44:30.316 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.007+0000 7feaf7e70700 -1 osd.9 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:44:30.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:29.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:30.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:29.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:30.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:29.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:30.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:29.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:30.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:29.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:30.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:30.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:30.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:30.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:30.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:30.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:30 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:30.312+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:31.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:30 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:30.976+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:31.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:30 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:30.976+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:31.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:30 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:30.976+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:31.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:30 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:30.976+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:31.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:30 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:30.976+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:31.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:30 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:30.976+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:31.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:30.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:31.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:30.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:31.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:30.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:31.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:30.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:31.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:30 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:30.943+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:31.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:31 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:31.308+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:31.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:31 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:31.308+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:31.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:31 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:31.308+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:31.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:31 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:31.308+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:31.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:31 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:31.308+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:31.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:31 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:31.309+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:32.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:31 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:31.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:32.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:31 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:31.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:32.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:31 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:31.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:32.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:31 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:31.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:32.281 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:31 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:31.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:32.281 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:31 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:31.941+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:32.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:31.939+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:32.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:31.939+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:32.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:31.939+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:32.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:31.939+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:32.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:31.939+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:32.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:32.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:32.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:32.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:32.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:32.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:32.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:32 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:32.346+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.305118+0000 front 2023-07-29T16:44:06.305134+0000 (oldest deadline 2023-07-29T16:44:32.205369+0000) 2023-07-29T16:44:33.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:33.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:33.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:33.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:33.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:33.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:33.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:32 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:32.923+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:33.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:32 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:32.898+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:33.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:32 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:32.898+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:33.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:32 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:32.898+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:33.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:32 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:32.898+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:33.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:32 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:32.898+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:33.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:33.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:33.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:33.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:33.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:33.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:33.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:33 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:33.374+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.305118+0000 front 2023-07-29T16:44:06.305134+0000 (oldest deadline 2023-07-29T16:44:32.205369+0000) 2023-07-29T16:44:34.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:34.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:34.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:34.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:34.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:34.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:34.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:33 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:33.903+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:34.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:33 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:33.942+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:34.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:33 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:33.942+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:34.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:33 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:33.942+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:34.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:33 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:33.942+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:34.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:33 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:33.942+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:34.687 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.598197+0000 front 2023-07-29T16:43:20.598456+0000 (oldest deadline 2023-07-29T16:43:49.998935+0000) 2023-07-29T16:44:34.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:29.999665+0000 front 2023-07-29T16:43:29.999600+0000 (oldest deadline 2023-07-29T16:43:55.299548+0000) 2023-07-29T16:44:34.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.300263+0000 front 2023-07-29T16:43:35.300252+0000 (oldest deadline 2023-07-29T16:44:00.600137+0000) 2023-07-29T16:44:34.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.501450+0000 front 2023-07-29T16:43:46.501208+0000 (oldest deadline 2023-07-29T16:44:10.601274+0000) 2023-07-29T16:44:34.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:54.102694+0000 front 2023-07-29T16:43:54.102644+0000 (oldest deadline 2023-07-29T16:44:18.802788+0000) 2023-07-29T16:44:34.688 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:02.904042+0000 front 2023-07-29T16:44:02.904068+0000 (oldest deadline 2023-07-29T16:44:24.603859+0000) 2023-07-29T16:44:34.689 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:34 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9[125409]: 2023-07-29T16:44:34.366+0000 7feaf2c63700 -1 osd.9 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.305118+0000 front 2023-07-29T16:44:06.305134+0000 (oldest deadline 2023-07-29T16:44:32.205369+0000) 2023-07-29T16:44:35.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:35.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:35.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:35.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:35.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:35.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:35.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:34 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:34.953+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:35.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:34.958+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:35.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:34.958+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:35.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:34.958+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:35.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:34.958+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:35.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:34 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:34.958+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:36.188 INFO:journalctl@ceph.osd.9.smithi031.stdout:Jul 29 16:44:35 smithi031 bash[174128]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-9 2023-07-29T16:44:36.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:36.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:36.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:36.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:36.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:36.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:36.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:36 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:36.001+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:36.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:35.984+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:36.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:35.984+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:36.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:35.984+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:36.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:35.984+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:36.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:35.984+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:36.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:35.984+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:37.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:37.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:37.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:37.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:37.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:37.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:37.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:37.026+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:37.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:36.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:37.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:36.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:37.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:36.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:37.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:36.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:37.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:36.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:37.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:36 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:36.994+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:37.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.9.service' 2023-07-29T16:44:37.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:37.797 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-29T16:44:37.797 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-29T16:44:37.798 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.10 2023-07-29T16:44:38.167 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:37 smithi039 systemd[1]: Stopping Ceph osd.10 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:38.167 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:38.167 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:38.167 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.041+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.163+0000 7f3d7966c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.163+0000 7f3d7966c700 -1 osd.10 96 *** Got signal Terminated *** 2023-07-29T16:44:38.168 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:38 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:38.163+0000 7f3d7966c700 -1 osd.10 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:44:38.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:37 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:37.980+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:38.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:37 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:37.980+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:38.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:37 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:37.980+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:38.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:37 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:37.980+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:38.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:37 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:37.980+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:38.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:37 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:37.980+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:39.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:38 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:38.997+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:39.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:38 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:38.997+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:39.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:38 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:38.997+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:39.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:38 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:38.997+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:39.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:38 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:38.997+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:39.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:38 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:38.997+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:39.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:39.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:39.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:39.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:39.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:39.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:39.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:39 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:39.060+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:40.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.019+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:40.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.019+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:40.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.019+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:40.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.019+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:40.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.019+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:40.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.019+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:40.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:40.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:40.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:40.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:40.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:40.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:40.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:40 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:40.094+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:41.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.970+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:41.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.970+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:41.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.971+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:41.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.971+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:41.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.971+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:41.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:40 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:40.971+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:41.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:41.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:41.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:41.529 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:41.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:41.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:41.530 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:41 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:41.059+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:42.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:42.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:42.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:42.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:42.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:42.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:42.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.012+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:42.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:41 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:41.933+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:42.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:41 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:41.933+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:42.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:41 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:41.933+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:42.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:41 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:41.933+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:42.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:41 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:41.933+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:42.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:41 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:41.933+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:43.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:20.050465+0000 front 2023-07-29T16:43:20.050264+0000 (oldest deadline 2023-07-29T16:43:44.750304+0000) 2023-07-29T16:44:43.279 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:28.851509+0000 front 2023-07-29T16:43:28.851591+0000 (oldest deadline 2023-07-29T16:43:54.751749+0000) 2023-07-29T16:44:43.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:38.153893+0000 front 2023-07-29T16:43:38.153842+0000 (oldest deadline 2023-07-29T16:44:01.053873+0000) 2023-07-29T16:44:43.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2023-07-29T16:43:46.856158+0000 front 2023-07-29T16:43:46.856058+0000 (oldest deadline 2023-07-29T16:44:08.556061+0000) 2023-07-29T16:44:43.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:52.657308+0000 front 2023-07-29T16:43:52.657286+0000 (oldest deadline 2023-07-29T16:44:18.557517+0000) 2023-07-29T16:44:43.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:43:58.557999+0000 front 2023-07-29T16:43:58.557900+0000 (oldest deadline 2023-07-29T16:44:24.458032+0000) 2023-07-29T16:44:43.280 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:42 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10[150487]: 2023-07-29T16:44:42.968+0000 7f3d74c60700 -1 osd.10 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:06.659868+0000 front 2023-07-29T16:44:06.659908+0000 (oldest deadline 2023-07-29T16:44:32.559635+0000) 2023-07-29T16:44:43.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:42 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:42.905+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:43.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:42 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:42.905+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:43.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:42 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:42.905+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:43.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:42 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:42.905+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:43.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:42 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:42.906+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:43.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:42 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:42.906+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:44.146 INFO:journalctl@ceph.osd.10.smithi039.stdout:Jul 29 16:44:44 smithi039 bash[165323]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-10 2023-07-29T16:44:44.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:43 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:43.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:44.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:43 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:43.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:44.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:43 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:43.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:44.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:43 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:43.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:44.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:43 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:43.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:44.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:43 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:43.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:45.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.10.service' 2023-07-29T16:44:45.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:45.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:45.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:45.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:45.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:45.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:45.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:44 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:44.955+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6820 osd.7 since back 2023-07-29T16:44:18.853423+0000 front 2023-07-29T16:44:18.853413+0000 (oldest deadline 2023-07-29T16:44:44.753351+0000) 2023-07-29T16:44:45.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:45.362 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-29T16:44:45.363 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-29T16:44:45.363 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.11 2023-07-29T16:44:45.718 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 systemd[1]: Stopping Ceph osd.11 for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:45.990 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.714+0000 7fc6c64a3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:45.991 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.714+0000 7fc6c64a3700 -1 osd.11 96 *** Got signal Terminated *** 2023-07-29T16:44:45.991 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.714+0000 7fc6c64a3700 -1 osd.11 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-29T16:44:45.991 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:46.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:46.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:46.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:46.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:46.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:46.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:45 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:45.986+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6820 osd.7 since back 2023-07-29T16:44:18.853423+0000 front 2023-07-29T16:44:18.853413+0000 (oldest deadline 2023-07-29T16:44:44.753351+0000) 2023-07-29T16:44:47.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:47.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:47.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:47.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:47.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:47.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:47.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.015+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6820 osd.7 since back 2023-07-29T16:44:18.853423+0000 front 2023-07-29T16:44:18.853413+0000 (oldest deadline 2023-07-29T16:44:44.753351+0000) 2023-07-29T16:44:48.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:48.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:48.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:48.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:48.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:48.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:48.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:47 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:47.968+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6820 osd.7 since back 2023-07-29T16:44:18.853423+0000 front 2023-07-29T16:44:18.853413+0000 (oldest deadline 2023-07-29T16:44:44.753351+0000) 2023-07-29T16:44:49.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:49.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:49.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:49.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:49.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:49.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:49.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6820 osd.7 since back 2023-07-29T16:44:18.853423+0000 front 2023-07-29T16:44:18.853413+0000 (oldest deadline 2023-07-29T16:44:44.753351+0000) 2023-07-29T16:44:49.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:48 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:48.972+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2023-07-29T16:44:24.753826+0000 front 2023-07-29T16:44:24.753848+0000 (oldest deadline 2023-07-29T16:44:48.853832+0000) 2023-07-29T16:44:50.326 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-07-29T16:43:23.642786+0000 front 2023-07-29T16:43:23.642729+0000 (oldest deadline 2023-07-29T16:43:50.043326+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6804 osd.1 since back 2023-07-29T16:43:31.144851+0000 front 2023-07-29T16:43:31.144672+0000 (oldest deadline 2023-07-29T16:43:54.644610+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6804 osd.2 since back 2023-07-29T16:43:35.145609+0000 front 2023-07-29T16:43:35.145629+0000 (oldest deadline 2023-07-29T16:44:00.445813+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6812 osd.4 since back 2023-07-29T16:43:50.248749+0000 front 2023-07-29T16:43:50.248763+0000 (oldest deadline 2023-07-29T16:44:19.649395+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6812 osd.5 since back 2023-07-29T16:44:03.150309+0000 front 2023-07-29T16:44:03.150376+0000 (oldest deadline 2023-07-29T16:44:25.450629+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2023-07-29T16:44:10.151657+0000 front 2023-07-29T16:44:10.151619+0000 (oldest deadline 2023-07-29T16:44:35.451526+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.39:6820 osd.7 since back 2023-07-29T16:44:18.853423+0000 front 2023-07-29T16:44:18.853413+0000 (oldest deadline 2023-07-29T16:44:44.753351+0000) 2023-07-29T16:44:50.327 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:50 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11[151488]: 2023-07-29T16:44:50.014+0000 7fc6bf292700 -1 osd.11 96 heartbeat_check: no reply from 172.21.15.156:6820 osd.8 since back 2023-07-29T16:44:24.753826+0000 front 2023-07-29T16:44:24.753848+0000 (oldest deadline 2023-07-29T16:44:48.853832+0000) 2023-07-29T16:44:51.576 INFO:journalctl@ceph.osd.11.smithi156.stdout:Jul 29 16:44:51 smithi156 bash[167225]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-osd-11 2023-07-29T16:44:52.962 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@osd.11.service' 2023-07-29T16:44:53.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:53.002 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-29T16:44:53.002 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-29T16:44:53.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.a 2023-07-29T16:44:53.437 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:44:53 smithi031 systemd[1]: Stopping Ceph mds.a for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:53.937 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:44:53 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-a[130794]: 2023-07-29T16:44:53.512+0000 7f58f8808700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:53.938 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:44:53 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-a[130794]: 2023-07-29T16:44:53.512+0000 7f58f8808700 -1 mds.a *** got signal Terminated *** 2023-07-29T16:44:55.938 INFO:journalctl@ceph.mds.a.smithi031.stdout:Jul 29 16:44:55 smithi031 bash[174402]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-a 2023-07-29T16:44:56.136 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.a.service' 2023-07-29T16:44:56.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:56.223 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-29T16:44:56.224 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-29T16:44:56.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.d 2023-07-29T16:44:56.438 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:44:56 smithi031 systemd[1]: Stopping Ceph mds.d for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:44:57.188 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:44:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-d[129784]: 2023-07-29T16:44:56.710+0000 7fa91a791700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:44:57.188 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:44:56 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-d[129784]: 2023-07-29T16:44:56.710+0000 7fa91a791700 -1 mds.d *** got signal Terminated *** 2023-07-29T16:44:59.576 INFO:journalctl@ceph.mds.d.smithi031.stdout:Jul 29 16:44:59 smithi031 bash[174546]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-d 2023-07-29T16:44:59.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.d.service' 2023-07-29T16:44:59.691 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:44:59.691 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-29T16:44:59.692 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-29T16:44:59.692 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.g 2023-07-29T16:44:59.937 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:44:59 smithi031 systemd[1]: Stopping Ceph mds.g for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:00.688 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:45:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g[130372]: 2023-07-29T16:45:00.183+0000 7f53a7416700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:00.688 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:45:00 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g[130372]: 2023-07-29T16:45:00.183+0000 7f53a7416700 -1 mds.g *** got signal Terminated *** 2023-07-29T16:45:03.687 INFO:journalctl@ceph.mds.g.smithi031.stdout:Jul 29 16:45:03 smithi031 bash[174683]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-g 2023-07-29T16:45:03.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.g.service' 2023-07-29T16:45:03.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:03.863 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-29T16:45:03.864 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-29T16:45:03.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.j 2023-07-29T16:45:04.188 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:45:03 smithi031 systemd[1]: Stopping Ceph mds.j for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:04.687 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:45:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-j[129389]: 2023-07-29T16:45:04.350+0000 7f272ac2e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:04.688 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:45:04 smithi031 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-j[129389]: 2023-07-29T16:45:04.350+0000 7f272ac2e700 -1 mds.j *** got signal Terminated *** 2023-07-29T16:45:08.184 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:45:07 smithi031 bash[174827]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-j 2023-07-29T16:45:08.422 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.j.service' 2023-07-29T16:45:08.437 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:45:08 smithi031 systemd[1]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.j.service: Succeeded. 2023-07-29T16:45:08.438 INFO:journalctl@ceph.mds.j.smithi031.stdout:Jul 29 16:45:08 smithi031 systemd[1]: Stopped Ceph mds.j for 858a7894-2e29-11ee-9b36-001a4aab830c. 2023-07-29T16:45:08.505 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:08.506 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-29T16:45:08.506 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-29T16:45:08.507 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.b 2023-07-29T16:45:08.879 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:45:08 smithi039 systemd[1]: Stopping Ceph mds.b for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:09.279 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:45:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: 2023-07-29T16:45:08.874+0000 7f908859b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:09.279 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:45:08 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b[154283]: 2023-07-29T16:45:08.874+0000 7f908859b700 -1 mds.b *** got signal Terminated *** 2023-07-29T16:45:11.437 INFO:journalctl@ceph.mds.b.smithi039.stdout:Jul 29 16:45:11 smithi039 bash[165584]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-b 2023-07-29T16:45:11.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.b.service' 2023-07-29T16:45:11.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:11.673 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-29T16:45:11.674 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-29T16:45:11.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.e 2023-07-29T16:45:12.029 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:45:11 smithi039 systemd[1]: Stopping Ceph mds.e for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:12.529 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:45:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e[153602]: 2023-07-29T16:45:12.066+0000 7f645bbfd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:12.529 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:45:12 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e[153602]: 2023-07-29T16:45:12.066+0000 7f645bbfd700 -1 mds.e *** got signal Terminated *** 2023-07-29T16:45:15.427 INFO:journalctl@ceph.mds.e.smithi039.stdout:Jul 29 16:45:15 smithi039 bash[165726]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-e 2023-07-29T16:45:15.621 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.e.service' 2023-07-29T16:45:15.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:15.657 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-29T16:45:15.658 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-29T16:45:15.658 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.h 2023-07-29T16:45:16.029 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:45:15 smithi039 systemd[1]: Stopping Ceph mds.h for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:16.529 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:45:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h[153943]: 2023-07-29T16:45:16.040+0000 7fb7fb472700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:16.529 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:45:16 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h[153943]: 2023-07-29T16:45:16.040+0000 7fb7fb472700 -1 mds.h *** got signal Terminated *** 2023-07-29T16:45:19.636 INFO:journalctl@ceph.mds.h.smithi039.stdout:Jul 29 16:45:19 smithi039 bash[165868]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-h 2023-07-29T16:45:19.642 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.h.service' 2023-07-29T16:45:19.676 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:19.676 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-29T16:45:19.676 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-29T16:45:19.677 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.k 2023-07-29T16:45:20.029 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:45:19 smithi039 systemd[1]: Stopping Ceph mds.k for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:20.529 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:45:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-k[153260]: 2023-07-29T16:45:20.066+0000 7f507066c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:20.529 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:45:20 smithi039 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-k[153260]: 2023-07-29T16:45:20.066+0000 7f507066c700 -1 mds.k *** got signal Terminated *** 2023-07-29T16:45:24.529 INFO:journalctl@ceph.mds.k.smithi039.stdout:Jul 29 16:45:24 smithi039 bash[166015]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-k 2023-07-29T16:45:24.543 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.k.service' 2023-07-29T16:45:24.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:24.581 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-29T16:45:24.582 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-29T16:45:24.582 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.c 2023-07-29T16:45:24.986 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:45:24 smithi156 systemd[1]: Stopping Ceph mds.c for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:25.326 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:45:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: 2023-07-29T16:45:24.983+0000 7f4a21017700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:25.326 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:45:24 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c[155186]: 2023-07-29T16:45:24.983+0000 7f4a21017700 -1 mds.c *** got signal Terminated *** 2023-07-29T16:45:29.489 INFO:journalctl@ceph.mds.c.smithi156.stdout:Jul 29 16:45:29 smithi156 bash[167490]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-c 2023-07-29T16:45:29.496 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.c.service' 2023-07-29T16:45:29.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:29.534 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-29T16:45:29.534 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-29T16:45:29.534 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.f 2023-07-29T16:45:29.826 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:45:29 smithi156 systemd[1]: Stopping Ceph mds.f for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:30.326 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:45:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f[154846]: 2023-07-29T16:45:29.898+0000 7fcbc8787700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:30.326 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:45:29 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f[154846]: 2023-07-29T16:45:29.898+0000 7fcbc8787700 -1 mds.f *** got signal Terminated *** 2023-07-29T16:45:31.369 INFO:journalctl@ceph.mds.f.smithi156.stdout:Jul 29 16:45:31 smithi156 bash[167642]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-f 2023-07-29T16:45:31.559 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.f.service' 2023-07-29T16:45:31.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:31.593 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-29T16:45:31.594 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-29T16:45:31.594 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.i 2023-07-29T16:45:31.826 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:45:31 smithi156 systemd[1]: Stopping Ceph mds.i for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:32.326 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:45:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: 2023-07-29T16:45:31.973+0000 7eff47e15700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:32.327 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:45:31 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i[155524]: 2023-07-29T16:45:31.973+0000 7eff47e15700 -1 mds.i *** got signal Terminated *** 2023-07-29T16:45:35.523 INFO:journalctl@ceph.mds.i.smithi156.stdout:Jul 29 16:45:35 smithi156 bash[167781]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-i 2023-07-29T16:45:35.529 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.i.service' 2023-07-29T16:45:35.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:35.563 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-29T16:45:35.563 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-29T16:45:35.564 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.l 2023-07-29T16:45:35.826 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:45:35 smithi156 systemd[1]: Stopping Ceph mds.l for 858a7894-2e29-11ee-9b36-001a4aab830c... 2023-07-29T16:45:36.326 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:45:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-l[154507]: 2023-07-29T16:45:35.929+0000 7fed5ceb9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-29T16:45:36.326 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:45:35 smithi156 ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-l[154507]: 2023-07-29T16:45:35.929+0000 7fed5ceb9700 -1 mds.l *** got signal Terminated *** 2023-07-29T16:45:38.995 INFO:journalctl@ceph.mds.l.smithi156.stdout:Jul 29 16:45:38 smithi156 bash[167923]: ceph-858a7894-2e29-11ee-9b36-001a4aab830c-mds-l 2023-07-29T16:45:39.001 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-858a7894-2e29-11ee-9b36-001a4aab830c@mds.l.service' 2023-07-29T16:45:39.034 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-29T16:45:39.034 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-29T16:45:39.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --force --keep-logs 2023-07-29T16:45:42.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --force --keep-logs 2023-07-29T16:45:44.726 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --force --keep-logs 2023-07-29T16:45:46.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:45:46.700 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:45:46.737 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-29T16:45:46.766 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-29T16:45:46.768 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi031/crash 2023-07-29T16:45:46.769 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash -- . 2023-07-29T16:45:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-29T16:45:46.863 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2023-07-29T16:45:46.867 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi039/crash 2023-07-29T16:45:46.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash -- . 2023-07-29T16:45:46.922 INFO:teuthology.orchestra.run.smithi039.stderr:tar: /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-29T16:45:46.922 INFO:teuthology.orchestra.run.smithi039.stderr:tar: Error is not recoverable: exiting now 2023-07-29T16:45:46.924 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi156/crash 2023-07-29T16:45:46.925 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash -- . 2023-07-29T16:45:46.972 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-29T16:45:46.972 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2023-07-29T16:45:46.973 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-29T16:45:46.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-29T16:45:47.054 INFO:teuthology.orchestra.run.smithi031.stderr:grep: /var/log/ceph/858a7894-2e29-11ee-9b36-001a4aab830c/ceph.log: No such file or directory 2023-07-29T16:45:47.059 INFO:tasks.cephadm:Compressing logs... 2023-07-29T16:45:47.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T16:45:47.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T16:45:47.066 DEBUG:teuthology.orchestra.run.smithi156:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T16:45:47.131 INFO:teuthology.orchestra.run.smithi039.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-29T16:45:47.138 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-29T16:45:47.139 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-29T16:54:55.643 INFO:tasks.cephadm:Archiving logs... 2023-07-29T16:54:55.645 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi031/log 2023-07-29T16:54:55.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T16:55:33.487 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi039/log 2023-07-29T16:55:33.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T16:56:21.919 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi156/log 2023-07-29T16:56:21.920 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-29T16:57:05.402 INFO:tasks.cephadm:Removing cluster... 2023-07-29T16:57:05.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --force 2023-07-29T16:57:07.947 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --force 2023-07-29T16:57:09.352 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 858a7894-2e29-11ee-9b36-001a4aab830c --force 2023-07-29T16:57:10.930 INFO:tasks.cephadm:Removing cephadm ... 2023-07-29T16:57:10.931 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-29T16:57:10.985 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-29T16:57:11.017 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-29T16:57:11.038 INFO:tasks.cephadm:Teardown complete 2023-07-29T16:57:11.039 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-29T16:57:11.050 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-29T16:57:11.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-29T16:57:11.057 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-29T16:57:11.060 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-29T16:57:11.205 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-29T16:57:11.219 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-29T16:57:11.220 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T16:57:11.225 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T16:57:11.229 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-29T16:57:11.245 INFO:teuthology.orchestra.run.smithi156.stderr:bash: ntpq: command not found 2023-07-29T16:57:11.246 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-07-29T16:57:11.263 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-07-29T16:57:11.265 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T16:57:11.266 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-07-29T16:57:11.266 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +2581us[+2564us] +/- 74ms 2023-07-29T16:57:11.266 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -3477us[-3493us] +/- 51ms 2023-07-29T16:57:11.266 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 +2658us[+2658us] +/- 85ms 2023-07-29T16:57:11.266 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T16:57:11.275 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T16:57:11.275 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2023-07-29T16:57:11.275 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +2587us[+2587us] +/- 74ms 2023-07-29T16:57:11.275 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -3452us[-3230us] +/- 51ms 2023-07-29T16:57:11.275 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +2682us[+2904us] +/- 85ms 2023-07-29T16:57:11.276 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T16:57:11.292 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-29T16:57:11.292 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-07-29T16:57:11.292 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2564us[+2567us] +/- 74ms 2023-07-29T16:57:11.293 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -3490us[-3488us] +/- 51ms 2023-07-29T16:57:11.293 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +2653us[+2655us] +/- 85ms 2023-07-29T16:57:11.293 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-07-29T16:57:11.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-29T16:57:11.306 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-29T16:57:11.308 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-29T16:57:11.363 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-29T16:57:11.637 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-29T16:57:11.838 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-29T16:57:12.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T16:57:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:57:12.085 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T16:57:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:57:12.123 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-29T16:57:12.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:57:12.155 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-29T16:57:12.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-29T16:57:12.199 INFO:teuthology.task.internal:Duration was 4358.783881 seconds 2023-07-29T16:57:12.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-29T16:57:12.212 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-29T16:57:12.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T16:57:12.217 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T16:57:12.220 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-29T16:57:12.298 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T16:57:12.305 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T16:57:12.393 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-29T16:57:12.606 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-29T16:57:12.607 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-07-29T16:57:12.607 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T16:57:12.685 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2023-07-29T16:57:12.685 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T16:57:12.725 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2023-07-29T16:57:12.726 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-29T16:57:12.755 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-29T16:57:12.756 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T16:57:12.761 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T16:57:12.767 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-29T16:57:12.861 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-29T16:57:12.861 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T16:57:12.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T16:57:12.869 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-29T16:57:13.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-29T16:57:13.052 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-29T16:57:13.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T16:57:13.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T16:57:13.157 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-29T16:57:13.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-29T16:57:13.213 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T16:57:13.219 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T16:57:13.222 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-29T16:57:13.255 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2023-07-29T16:57:13.279 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2023-07-29T16:57:13.292 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-07-29T16:57:13.328 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T16:57:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:57:13.374 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T16:57:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:57:13.394 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-29T16:57:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-29T16:57:13.412 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-29T16:57:13.424 INFO:teuthology.task.internal:Transferring archived files... 2023-07-29T16:57:13.426 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi031 2023-07-29T16:57:13.427 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T16:57:13.516 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi039 2023-07-29T16:57:13.517 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T16:57:13.566 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830/remote/smithi156 2023-07-29T16:57:13.567 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-29T16:57:13.608 INFO:teuthology.task.internal:Removing archive directory... 2023-07-29T16:57:13.608 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T16:57:13.614 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T16:57:13.616 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-29T16:57:13.656 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-29T16:57:13.672 INFO:teuthology.task.internal:Not uploading archives. 2023-07-29T16:57:13.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-29T16:57:13.685 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-29T16:57:13.685 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T16:57:13.699 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T16:57:13.702 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-29T16:57:13.717 INFO:teuthology.orchestra.run.smithi039.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 16:57 /home/ubuntu/cephtest 2023-07-29T16:57:13.718 INFO:teuthology.orchestra.run.smithi156.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 16:57 /home/ubuntu/cephtest 2023-07-29T16:57:13.737 INFO:teuthology.orchestra.run.smithi031.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 29 16:57 /home/ubuntu/cephtest 2023-07-29T16:57:13.740 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-29T16:57:13.754 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-29T16:57:13.779 INFO:teuthology.nuke:Checking targets against current locks 2023-07-29T16:57:13.811 DEBUG:teuthology.nuke:shortname: smithi031 2023-07-29T16:57:13.814 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T16:57:13.837 DEBUG:teuthology.nuke:shortname: smithi039 2023-07-29T16:57:13.838 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T16:57:13.841 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 15:34:56.813796', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-29T16:57:13.861 DEBUG:teuthology.nuke:shortname: smithi156 2023-07-29T16:57:13.862 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-29T16:57:13.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 15:34:56.814850', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-29T16:57:13.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-07-29_14:02:18-fs-reef-release-distro-default-smithi/7356830', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-29 15:34:56.812591', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-29T16:57:13.895 INFO:teuthology.orchestra.console.smithi031:Power off 2023-07-29T16:57:13.896 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T16:57:13.924 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-07-29T16:57:13.925 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:13.946 INFO:teuthology.orchestra.console.smithi156:Power off 2023-07-29T16:57:13.947 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T16:57:13.966 INFO:teuthology.orchestra.console.smithi039:Power off 2023-07-29T16:57:13.967 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-29T16:57:13.978 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-07-29T16:57:13.979 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2023-07-29T16:57:13.979 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:13.993 DEBUG:teuthology.orchestra.console.smithi039:power off output: Chassis Power Control: Down/Off 2023-07-29T16:57:13.994 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:14.006 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-07-29T16:57:14.024 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2023-07-29T16:57:17.980 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:18.008 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:18.025 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:18.094 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-07-29T16:57:18.120 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2023-07-29T16:57:18.136 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2023-07-29T16:57:22.096 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:22.122 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:22.137 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:22.210 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-07-29T16:57:22.236 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2023-07-29T16:57:22.249 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2023-07-29T16:57:22.312 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-07-29T16:57:22.338 INFO:teuthology.orchestra.console.smithi156:Power off completed 2023-07-29T16:57:26.252 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-29T16:57:26.364 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is off 2023-07-29T16:57:26.465 INFO:teuthology.orchestra.console.smithi039:Power off completed 2023-07-29T16:57:26.570 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 8e3722cc016410c8de834520a714aab9f006b92a description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} duration: 4358.783880710602 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-07-29T16:57:26.571 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-29T16:57:26.681 INFO:teuthology.run:FAIL